# DidiBrain - Index

Knowledge atom storage + claim verification cache + analysis atom cache pentru platforma DIDI. FastAPI service care vorbeste contractul `web-gathering` 1:1 (drop-in replacement / cache layer pentru `/v1/gather`) si in plus expune doua cache-uri proprii: `brain_verification_cache` (claims, din 2026-04-23) si `brain_analysis_atom` (techniques + ai_tampered + claims, NEW v2 din 2026-05-01).

**Stack**: Python 3.11+, FastAPI, Pydantic v2, asyncpg, BGE-M3 1024-dim embeddings (vLLM), BGE-reranker-v2-m3 (vLLM cross-encoder), qwen3.5 (vLLM via OpenAI-compat router)
**Storage**: Atomic (Rust knowledge graph) on Postgres 16 + pgvector. Brain owns its own tables prefixed `brain_*` alongside Atomic's tables.
**URL**: `http://10.11.10.12:8090` (production-local, since brain cutover 2026-05-01 — anterior pe `10.11.10.13:8090`)
**Container**: `didibrain-api` (alongside `didibrain-atomic` and `didibrain-postgres`)

---

## Ce face

DidiBrain sta intre Didi backend (agent-v3) si modulele live de web-gathering. Vorbeste acelasi contract HTTP pentru `/v1/gather`, raspunzand din knowledge atoms pre-ingerate + semantic search + cross-encoder reranking + NLI stance vs query. Tipic 1.5-6s in loc de 20+s pentru un crawl live. Plus detine 2 cache-uri proprii pentru rezultate LLM ale didi-backend: `brain_verification_cache` (per-claim verification, since 2026-04-23) si `brain_analysis_atom` (rezultate full-component pe techniques + ai_tampered + claims, since 2026-05-01 v2).

---

## API endpoints

| Method | Path | Ce face |
|---|---|---|
| GET | `/health` | Liveness probe |
| POST | `/v1/search` | Flat list of doc-level search results |
| POST | `/v1/fetch` | Look up atoms by URL → return extracted text |
| POST | `/v1/gather` | Full claim → ranked evidence pipeline (with optional verification cache lookup) |
| POST | `/v1/image-search` | Stub, always returns empty list |
| POST | `/v1/ingest` | Populate brain from web-module output |
| POST | `/v1/verification_cache` | Write claim verification result (by didi-backend after LLM run) |
| POST | `/v1/analysis_atom/lookup` | NEW v2 — find cached analysis result by `(content_hash, component, prompt_hash)`. Tier-agnostic. Bronze never served. |
| POST | `/v1/analysis_atom` | NEW v2 — write atom (rejects `tier=free`). Auto cache_tier silver/bronze din `llm_confidence` (>=60 silver). Gold preservation in upsert. |
| PATCH | `/v1/analysis_atom/{atom_id}` | NEW v2 — promote to gold (after moderator review). `human_validated=true`, `expires_at=NULL`. |
| GET | `/v1/analysis_atom/stats` | NEW v2 — counts per tier/component, hit rate 24h |

Plus FastAPI auto: `/docs` (Swagger UI), `/redoc`, `/openapi.json`.

---

## Structura fisiere

```
brain_api/
  app.py                  # FastAPI app + lifespan + 11 endpoints
  db.py                   # asyncpg pool + _SCHEMA_SQL (creates brain_verification_cache + brain_analysis_atom tables idempotent)
  schemas.py              # Pydantic v2 models (request/response for all endpoints)
  deps.py                 # AppState (atomic, embed, llm, resolver) singleton
  run.py                  # CLI entry (uvicorn launcher)
  Dockerfile
  requirements.txt
  prompts/                # LLM prompts for verification + extraction (nli_v1.md etc.)
  services/
    fetch.py              # /v1/fetch
    gather.py             # /v1/gather (most complex — semantic search + reranking + NLI + verification cache integration)
    ingest.py             # /v1/ingest
    search.py             # /v1/search
    verification_cache.py # claims cache (since 2026-04-23 v2 schema)
    analysis_atom.py      # NEW 2026-05-01 — techniques+ai_tampered+claims cache, 3 tiers (gold/silver/bronze)
    nli.py                # NLI stance classification
    mapping.py            # taxonomy / tag mapping helpers (atom → EvidenceItem / FetchedPage)

shared/                   # (one level up, used by brain_api)
  atomic_api.py           # AtomicClient (typed REST client for atomic-server)
  embedding_client.py     # BGE-M3 embed + reranker
  llm_client.py           # async OpenAI-compat wrapper
  config.py               # Pydantic Settings, LlmRole, model routing
  taxonomy.py             # canonical TAXONOMY tree + TagResolver
  logging.py              # structlog setup

extractor/                # claim extraction layer (separate package)
  extract.py              # single-doc extraction + substring quote validation
  push.py                 # claim atom creation
  batch.py                # batch orchestrator with state file
  prompts/

scripts/                  # operator CLI tools (numbered 01-11 + bootstrap_deploy.sh)
infra/                    # docker-compose.yml + deploy
lint/                     # custom lint pass (cross-corpus contradiction detection)
reports/                  # generated audit reports
```

---

## Brain-owned PG schema

Doua tabele alaturi de cele ale Atomic, in acelasi Postgres (prefix `brain_*` pentru izolare).

### brain_verification_cache (since 2026-04-23 v2)

Cache pentru verificare claims (one row per `(claim_hash, tier)`, last-wins UPSERT). `evidence_urls` + `evidence_hash` stocate ca METADATA (NU in unique key) pentru ca backend sa poata calcula overlap. TTL 30 zile default.

| Coloana | Tip | Scop |
|---|---|---|
| `id` | uuid PK | `gen_random_uuid()` |
| `claim_hash` | text NOT NULL | `sha256(normalized claim)` |
| `tier` | text CHECK (`free`\|`premium`) | |
| `evidence_hash` | text NOT NULL | `sha256(canonical urls)`, metadata only |
| `evidence_urls` | jsonb NOT NULL | metadata only |
| `model` | text NULL | LLM used |
| `prompt_hash` | text NOT NULL | `sha256(system+user_template)[:12]` |
| `framework_version` | text NULL | sha256 of relevant Redis configs |
| `schema_name` | text default `'didi-v1'` | |
| `verification_raw` | jsonb NULL | raw LLM output |
| `verification_processed` | jsonb NOT NULL | mapped to canonical |
| `created_at` | timestamptz default `now()` | |
| `updated_at` | timestamptz default `now()` | |
| `expires_at` | timestamptz NOT NULL | TTL 30d default |

UNIQUE: `(claim_hash, tier)`. Indexes: `idx_bvc_lookup`, `idx_bvc_expires`, `idx_bvc_prompt`.

**Migration v1 → v2**: vechiul UNIQUE `(claim_hash, evidence_hash, tier)` facea cache-ul practic nereachable (URL-urile la read difera de cele la write). v2 muta `evidence_hash` afara din unique key. Migrarea e idempotenta (rulata la fiecare connect).

### brain_analysis_atom (NEW 2026-05-01)

Cache pentru rezultate LLM full-component (techniques, ai_tampered, claims-aggregator). 3-tier system gold/silver/bronze.

| Coloana | Tip | Scop |
|---|---|---|
| `atom_id` | bigserial PK | |
| `content_hash` | text NOT NULL | `sha256(normalized text)` |
| `content_preview` | text NULL | first 200 chars for debug |
| `component` | text CHECK (`techniques`\|`ai_tampered`\|`claims`) | |
| `tier` | text CHECK (`free`\|`premium`) | informational only on row |
| `prompt_hash` | text NOT NULL | |
| `framework_version` | text NULL | |
| `model_used` | text NULL | |
| `result_processed` | jsonb NOT NULL | canonical mapped output |
| `result_raw` | jsonb NULL | |
| `llm_confidence` | numeric NULL | 0-100, drives silver vs bronze |
| `cache_tier` | text CHECK (`gold`\|`silver`\|`bronze`) default `'silver'` | |
| `human_validated` | bool default false | true after moderator review |
| `human_corrections` | jsonb NULL | diff applied by moderator |
| `validator_user_id` | text NULL | keycloak_id |
| `validated_at` | timestamptz NULL | |
| `hit_count` | integer default 0 | incremented on lookup hit |
| `last_hit_at` | timestamptz NULL | |
| `created_at`, `updated_at` | timestamptz | |
| `expires_at` | timestamptz NULL | NULL=never (gold). Silver=90d, Bronze=30d |

UNIQUE: `(content_hash, component, prompt_hash)` — `tier` OUT of unique key (write only `premium`, read tier-agnostic). Indexes: `idx_baa_lookup`, `idx_baa_gold` (partial WHERE `cache_tier='gold'`), `idx_baa_expires`, `idx_baa_prompt`.

**Cache tier rules**:
- **gold**: `human_validated=true`. Set by didi-backend moderation flow (PATCH endpoint). Survives prompt change. Forever (`expires_at=NULL`).
- **silver**: LLM result, `llm_confidence >= 60`. Default fresh write. TTL 90d.
- **bronze**: LLM result, `llm_confidence < 60`. Stored for audit but **NEVER served on lookup**. TTL 30d.

**Gold preservation**: ON CONFLICT upsert preserves gold (won't downgrade to silver if a fresh write comes in — `result_processed`, `cache_tier`, `llm_confidence`, `expires_at` all CASE-WHEN protected).

**Lookup logic** (`services/analysis_atom.py::lookup`): SELECT WHERE `content_hash=$1 AND component=$2 AND cache_tier IN ('gold','silver') AND (expires_at IS NULL OR expires_at > now())` ORDER BY `(cache_tier='gold') DESC, updated_at DESC` LIMIT 1. Apoi `decide_freshness()` pe baza `prompt_hash` + `framework_version`. Gold = always `fresh`.

---

## How didi-backend (agent-v3) uses brain

3 puncte de integrare:

1. **Claims verification cache** — claims executor cheama `gatherFromBrain()` cu `include_verification=true`. Brain returneaza evidence + cached verification (if any) + staleness. Backend decide fresh / stale_framework / stale_prompt / miss path. Backend scrie via fire-and-forget POST `/v1/verification_cache`. Detalii contract: `CONTRACT_VERIFICATION_CACHE.md`. Helper centralizat: `agent-v3/src/shared/brain/client.ts`.

2. **Analysis atom cache** (NEW 2026-05-01) — techniques si ai_tampered executors cheama `lookupAnalysisAtom()` **inainte** de LLM. Pe hit (gold sau silver+fresh), skip LLM, return cached. Dupa LLM run, fire-and-forget `writeAnalysisAtomAsync()` (only `tier=premium`). Dupa ce moderator rezolva sesiunea cu corectii, `patchAnalysisAtomGold()` promoveaza silver → gold.

3. **Gather pipeline** — agent-v3 inca foloseste `/v1/gather` pentru live web-augmented retrieval (pre-ingerat + semantic search + reranker + NLI). Branch-ul pe `brain_meta.cache_status` = `HIT|PARTIAL|MISS` decide fallback la web module.

---

## Configuration

- Settings via `shared/config.py` cu `pydantic-settings`
- Env vars: `postgres_dsn`, `atomic_url`, `atomic_token`, `llm_router_url`, `embedding_url`, `reranker_url`, `verification_cache_ttl_days` (default 30), `verification_cache_max_payload_kb` (default 64)
- `ATOMIC_TOKEN`: required for atomic-server API calls, auto-generated/auto-populated by `scripts/02_bootstrap_atomic.py` (written back to `.env`)
- Internal Atomic URL: `http://atomic-server:8080` (Docker DNS)
- Upstream LLM/embed/rerank reached over Docker DNS on `didi-network` (`didiAI-llm-api` → 10.11.10.17:14011 router, `didiAI-embeddings-api` → 10.11.10.15:14100 BGE-M3, `didiAI-rerank-api` → 10.11.10.15:14200 reranker)
- **No auth in v1** (binds to internal Docker network)

---

## Deployment

- Docker compose la `infra/docker-compose.yml`
- 4 containere: `didibrain-api` (FastAPI), `didibrain-atomic` (Rust KG), `didibrain-postgres` (PG 16 + pgvector), `didibrain-scheduler` (feeder + auditor + watcher)
- Schema bootstrap on startup (idempotent — `_SCHEMA_SQL` ruleaza la fiecare connect, all DDL is `IF NOT EXISTS` + ALTER guards)
- TTL cleanup: cron job in `scripts/` sterge expired silver/bronze atoms
- Resource footprint: ~200 MB RAM total cross 4 containere; brain-api idle ~6% CPU, spike ~20% during `/v1/gather` cu NLI

**Rebuild**:
```bash
cd /home/admin365/didi_mono/ai_platform/modules/didi_brain
docker compose -f infra/docker-compose.yml --env-file .env up -d --build
```

---

## Bootstrap & lint

- `scripts/01_sanity_full.py` — upstream stack validator (LLM + embed + rerank, 8 checks)
- `scripts/02_bootstrap_atomic.py` — claim atomic instance, configure BGE-M3 provider
- `scripts/03_sanity_atomic.py` — brain end-to-end (create → embed → search → cleanup)
- `scripts/04_seed_taxonomy.py` — seed canonical tag taxonomy (79 tags, 7 namespaces)
- `scripts/05_import_wikipedia_seed.py` — seed-list Wikipedia import
- `scripts/06_validate_queries.py` — smoke test doc-level retrieval
- `scripts/07_run_extraction.py` — claim extraction batch (qwen3.5)
- `scripts/08_validate_claims.py` — smoke test claim-level retrieval
- `scripts/09_brain_api_demo.py` — brain_api contract test (all endpoints)
- `scripts/10_run_lint.py` — Lint pass runner (contradiction detection)
- `scripts/11_show_contradictions.py` — render top contradictions
- `scripts/bootstrap_deploy.sh` — fresh-server deploy orchestrator
- `lint/` — custom lint pass (cross-corpus contradiction detection via NLI)

---

## Ce NU face

- No image storage (image-search endpoint always returns empty)
- No auth in v1 (intern only, binds to Docker internal network)
- No multi-tenant (single brain instance, multiple consumers)
- No long-term archive — atoms expire on TTL (gold = forever, silver = 90d, bronze = 30d)
- No live web crawling (delegata la modulele M17 / web — brain serveste din corpus pre-ingest)

---

## Related docs (in this module)

- `README.md` — project landing, quick start, architecture diagram
- `STATUS.md` — end-of-session snapshot + troubleshooting + session log
- `CONTRACT_VERIFICATION_CACHE.md` — contract backend ↔ brain pentru verification cache (v2)
- `CHANGES_2026-04-23.md` — session changelog cu integrare web-api + verification cache v2
- `ARCHITECTURE.md` — diagrame + pipeline detail
- `AUDIT.md` — initial upstream Atomic audit

---

## Cache Freshness Defense (2026-05-04 → 2026-05-05) — 11/11 piloni live

Whole-system response to "what happens if a cached verdict goes stale because the world changed?". Strict additive — zero breaking changes on the contracts agent-v3 already calls.

### New services in `brain_api/services/`

| Module | Pilon | Role |
|--------|-------|------|
| `classifier.py` | 1 | LLM single-call classify of a claim → `{volatility, topic_codes, entity_bindings, estimated_validity_hours}`. Caps clamped per tier (volatile≤48h, evolving≤720h, stable≤26280h). Conservative defaults on LLM failure. |
| `cache_judge.py` | 2 | NLI judge run on cache hits — given a claim, cached truth direction (TRUE/FALSE/MIXED), and current top-3 evidence, decides KEEP_CACHE / INVALIDATE / NEEDS_FULL_RECHECK. Cheap-path skip for stable+young. Audit-pass bonus raises invalidate threshold for atoms that survived multiple audits. |
| `fact_status.py` | 11 | Versioned (subject, predicate, object) knowledge layer. `register_facts_from_bindings`, `assert_fact_truth` (opens new fact_version when truth flips), `lock_fact`/`unlock_fact` for moderator overrides, `check_fact_validity` for gather-time lookup. Plus admin: `list_facts_admin`, `list_audit_log`. |
| `canonicalizer.py` | 7 | LLM rewrites a claim with relative time markers (`azi`, `alegerile`) anchored to the current date. Same surface text at different times produces different cache keys. Passthrough on failure. |
| `invalidation.py` | 8 | `InvalidateFilter` (topic_codes / entity_canonicals / claim_pattern / since / invalidate_gold / dry_run) → bulk UPDATE on both cache tables. Audit log row written per non-dry-run execution. |
| `topic_volatility.py` | D1 | HTTP polls didiFramework `/api/sensitive-topics?active=true` (cache 60s) for admin-configured per-topic TTL/recency. `reconcile_with_classifier` picks min(LLM_estimate, admin_TTL). |

### DB schema (alongside Atomic in same Postgres, prefix `brain_*`)

```
ALTER brain_analysis_atom + brain_verification_cache
  ADD volatility ('volatile'|'evolving'|'stable')
  ADD topic_codes text[]
  ADD entity_bindings jsonb         -- list of {subject, predicate, object, confidence}
  ADD ttl_hours_used integer
  ADD last_audited_at timestamptz
  ADD audit_history jsonb           -- last-50 JudgeVerdict snapshots
  ADD consecutive_audit_passes integer

CREATE brain_fact_status               -- current truth per (subject, predicate, object)
CREATE brain_fact_version              -- temporal versioning with valid_from/valid_to
CREATE brain_audit_log                 -- mass invalidations + judge decisions + truth changes
```

GIN indexes on `topic_codes`. Partial indexes on `cache_tier IN ('gold','silver') AND volatility != 'stable'` for the auditor's fast candidate query.

### New endpoints (`brain_api/app.py`)

```
POST /v1/canonicalize                — Pilon 7 temporal disambiguation
POST /v1/cache/invalidate            — Pilon 8 mass invalidation (with dry_run)
GET  /v1/cache/audit_log             — paginated audit browser
GET  /v1/fact_status/list            — paginated fact browser (admin)
GET  /v1/fact_status/{fact_id}       — single fact detail
GET  /v1/fact_status/{fact_id}/versions  — full timeline
PATCH /v1/fact_status/{fact_id}      — moderator override (set_truth + lock + notes)
```

Existing upsert paths (`POST /v1/analysis_atom`, `POST /v1/verification_cache`) now run the classifier internally and persist volatility/topic_codes/entity_bindings on every write. Lookup paths (`/v1/gather` with `include_verification=true`) consult `brain_fact_status` and demote staleness to `stale_evidence` when bound facts have flipped.

### `gather.py` recency reranking (Pilon 3+4)

`_apply_recency` blends rerank score with age decay using volatility-driven profiles:

| Volatility | w_recency | half_life_days | hard cut |
|---|---|---|---|
| volatile | 0.50 | 3 | 7 days |
| evolving | 0.30 | 30 | (none) |
| stable | 0.00 | ∞ | (none) |
| (none / hint absent) | 0.15 | 30 | (none) |

`combined = (1 - w_recency) * rerank + w_recency * exp(-age_days/half_life)`. Volatile evidence older than `recency_window_days` is hard-dropped.

### `didibrain-scheduler` container (`scheduler/`)

Single image, four asyncio tasks under `_supervised` (auto-restart with exponential backoff):

- **feeder** — RSS pull at volatility-aware intervals (15 min volatile / 6 h evolving / 24 h stable). Default feeds in `config.py:DEFAULT_*_FEEDS`; override via `FEED_VOLATILE_URLS`/`FEED_EVOLVING_URLS`/`FEED_STABLE_URLS`/`FEED_BREAKING_URLS` env. Feeds POST to `/v1/ingest` with `run_extraction=true`.
- **auditor** — daily sweep of `brain_analysis_atom` rows with `cache_tier IN ('gold','silver') AND volatility != 'stable' AND last_audited_at < now()-1d`. For each: confidence-decay cheap path → judge → `apply_judge_verdict`. Sequential (not parallel) to avoid LLM router saturation.
- **watcher** — 5-min poll on the breaking-news feeds. Per item: classify with LLM → `_build_invalidation_targets` → `POST /v1/cache/invalidate` with `topic_codes` + `entity_canonicals`. In-process LRU of seen URLs (cap 2000). End-to-end target latency: <30s from RSS publish to cache invalidation.
- **heartbeat** — touches `/tmp/scheduler.healthy` every 30s for the docker healthcheck.

Compose entry in `infra/docker-compose.yml` builds via `scheduler/Dockerfile`. Imports `brain_api.*` directly so the auditor uses the same DB pool + `apply_judge_verdict` helper as the API container (separate process, separate pool to the same Postgres).

### Live state at write-time

47 articles ingested in first cycle. Watcher invalidated 4 breaking stories live (Iran/Hormuz, Nigerian elections, Ukraine drones, etc.) within seconds of feed publication. Auditor scheduled 24h after first start.

### Memory pointer

`~/.claude/projects/-home-admin365/memory/MEMORY.md` § "Cache Freshness Defense" — one-liner status + counts.