clossers/didi-lot1-ai
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Livrare LOT 1 - Didi

Browse source
This commit is contained in:
Dezvoltari Evotech 2026-06-25 14:13:25 -07:00
commit 5380c3fc63
990 changed files with 133308 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

51
ai_platform/modules/didi_brain/.dockerignore Normal file
View file

@ -0,0 +1,51 @@
# Keep the brain_api build context minimal and secret-free.
# docker-compose builds from the didibrain/ root so this lives there.
# Local dev
.venv/
venv/
.env
.env.local
.env.*.local
# Python caches
__pycache__/
**/__pycache__/
*.pyc
*.pyo
.pytest_cache/
.mypy_cache/
.ruff_cache/
# Reports / generated state
reports/
logs/
profiles/
archive/
*.log
*.db
*.sqlite
*.sqlite3
# Scripts aren't needed at runtime, and the extractor log may contain
# references to specific atom ids from previous dev runs.
scripts/
extractor/_extracted.json
shared/_tag_ids.json
# VCS / editors / OS
.git/
.gitignore
.vscode/
.idea/
*.swp
.DS_Store
Thumbs.db
# Infra (compose configs aren't part of the image itself)
infra/
# Docs
README.md
STATUS.md
AUDIT.md

54
ai_platform/modules/didi_brain/.env.example Normal file
View file

@ -0,0 +1,54 @@
# =============================================================================
# DidiBrain — environment configuration
# Copy to .env and fill in. .env is gitignored.
# =============================================================================
# -- LLM router (single entry point, picks backend) ---------------------------
LLM_ROUTER_URL=http://10.11.10.17:14011
LLM_ROUTER_API_KEY=
# Direct backends (used only as fallback / health checks)
LLM_VLLM_URL=http://localhost:14001
LLM_LLAMACPP_URLS=http://10.11.10.18:14001,http://10.11.10.19:14001
# -- Model names (as reported by /v1/models) ----------------------------------
# Workhorse for everything critical: extraction, NLI, verdict, wiki, chat
MODEL_REASONING=Qwen3.5-397B-A17B
MODEL_REASONING_BACKEND=llamacpp
# Fast worker for mass processing (currently DISABLED — thinking mode + safety
# alignment block structured output on disinfo topics). Re-enable when fixed.
MODEL_FAST=qwen3.5
MODEL_FAST_BACKEND=vllm
MODEL_FAST_ENABLED=false
# Multimodal / second-opinion (currently DOWN — port 8001 not responding)
MODEL_VISION=gemma-3-27b-it
MODEL_VISION_URL=http://10.11.10.16:8001
MODEL_VISION_ENABLED=false
# -- Embeddings (BGE-M3 via vLLM) ---------------------------------------------
EMBEDDING_URL=http://10.11.10.15:8200
EMBEDDING_API_KEY=
EMBEDDING_MODEL=BAAI/bge-m3
EMBEDDING_DIM=1024
EMBEDDING_MAX_TOKENS=8192
# -- Reranker (BGE-reranker-v2-m3) --------------------------------------------
RERANKER_URL=http://10.11.10.15:8100
RERANKER_API_KEY=
RERANKER_MODEL=BAAI/bge-reranker-v2-m3
# -- Atomic server (the brain) ------------------------------------------------
# When running locally for dev, this is the dockerized atomic-server
ATOMIC_URL=http://localhost:8088
ATOMIC_TOKEN=
# -- Postgres (for atomic-server multi-db scale) ------------------------------
POSTGRES_USER=atomic
POSTGRES_PASSWORD=changeme_in_real_env
POSTGRES_DB=atomic
POSTGRES_PORT=5434
# -- Logging ------------------------------------------------------------------
LOG_LEVEL=INFO

44
ai_platform/modules/didi_brain/.gitignore vendored Normal file
View file

@ -0,0 +1,44 @@
# Secrets
.env
.env.local
.env.*.local
# Python
__pycache__/
*.pyc
*.pyo
.venv/
venv/
.pytest_cache/
.mypy_cache/
.ruff_cache/
*.egg-info/
dist/
build/
# uv
.uv/
# IDE
.vscode/
.idea/
*.swp
# OS
.DS_Store
Thumbs.db
# Data / runtime
data/
logs/
*.log
*.db
*.sqlite
*.sqlite3
profiles/
archive/
# Reports
reports/*.json
reports/*.html
!reports/.gitkeep

645
ai_platform/modules/didi_brain/ARCHITECTURE.md Normal file
View file

@ -0,0 +1,645 @@
# DidiBrain — Architecture & "Who Does What"
This is the **codebase rosetta stone**. If you read this top-to-bottom you
should be able to answer "what does file X do, when does it run, who calls
it, what does it depend on" for any file in the repo.
> **Reading order if you're new**:
> 1. `README.md` — what DidiBrain is in 2 minutes
> 2. **this file** — full mental model of the codebase (15 minutes)
> 3. `STATUS.md` — operational state, deployment, troubleshooting
> 4. `brain_api/schemas.py` — the literal HTTP contract with Didi
> 5. `brain_api/services/gather.py` — the main pipeline in 200 lines
---
## TL;DR — "WHO DOES WHAT" in one table
| Module / File | What it does | When it runs | Who calls it |
|---|---|---|---|
| **`infra/docker-compose.yml`** | Defines the 3-container stack | `docker compose up` | operator |
| **`brain_api/`** *(container)* | HTTP service speaking Didi's contract | every Didi request | Didi backend (HTTP) |
| **`shared/`** *(library)* | Common code: config, clients, taxonomy | imported everywhere | brain_api, extractor, lint, scripts |
| **`extractor/`** *(library + jobs)* | Turns Document atoms into Claim atoms | operator (script 07) OR background after `/v1/ingest` | scripts/07, brain_api ingest |
| **`lint/`** *(library + job)* | Detects contradictions across the corpus | operator overnight (script 10) | scripts/10, scripts/11 |
| **`scripts/`** *(operator CLI)* | One-shot maintenance + bootstrap commands | operator types `python scripts/XX_*.py` | human |
| **Atomic server** *(container)* | Storage + chunking + embedding pipeline + REST | always | brain_api (HTTP), scripts (HTTP) |
| **Postgres + pgvector** *(container)* | Persistent atom + vector storage | always | atomic-server (SQL) |
**External services** (not in our repo, on VPN):
| Where | What | Used by |
|---|---|---|
| `10.11.10.17:14011` | LLM router → Qwen 397B | brain_api NLI, extractor, lint |
| `10.11.10.15:8200` | BGE-M3 embedding server | atomic-server (for embeddings) |
| `10.11.10.15:8100` | BGE-reranker-v2-m3 | brain_api gather (for precision rerank) |
---
## The big picture in one diagram
```
┌─────────────────────────────────┐
│ Didi backend │
│ (somewhere else) │
└─────┬───────────────────────────┘
│ POST /v1/gather, /v1/search,
│ /v1/fetch, /v1/ingest, ...
▼
┌────────────────────────────────────────────────────────────┐
│ brain_api (container :8090) │
│ ┌──────────────────────────────────────────────────────┐ │
│ │ FastAPI app (app.py) │ │
│ │ routes → services/{gather, search, fetch, ingest, │ │
│ │ mapping, nli} │ │
│ └──────────────────────────────────────────────────────┘ │
│ imports → shared/{config, llm_client, embedding_client, │
│ atomic_api, taxonomy, logging} │
└─────┬─────────────┬──────────────┬──────────────────────────┘
│ │ │
│ HTTP │ HTTPS │ HTTPS
▼ ▼ ▼
┌───────────┐ ┌──────────┐ ┌────────────────┐
│ atomic- │ │ BGE-M3 │ │ Qwen 397B │
│ server │ │ + rerank │ │ via LLM router │
│ (cont.) │ │ (VPN) │ │ (VPN) │
└─────┬─────┘ └──────────┘ └────────────────┘
│
▼
┌───────────────┐
│ postgres │
│ + pgvector │
│ (container) │
└───────────────┘
Operator scripts run on the host, talk over HTTP:
┌─────────────────────┐
│ host venv │ ┌──────────┐
│ scripts/01..11.py │──HTTP─▶│ brain │
│ + bootstrap.sh │ │ stack │
└─────────────────────┘ └──────────┘
```
---
# Module deep-dives
Each module section answers:
1. **Purpose** — one paragraph
2. **When it runs** — request-time / startup / background / operator
3. **Files inside** — what each does
4. **Imports** — what it depends on
5. **Imported by** — who depends on it
## `shared/` — the library every other module imports
**Purpose.** Pure-Python utilities and clients with no HTTP surface of their
own. Single source of truth for config, logging, all upstream clients, and
the canonical tag taxonomy. Everything else in the repo treats this as a
read-only library.
**When it runs.** Imported at process startup; no long-lived behavior of
its own.
**Files inside.**
| File | Role |
|---|---|
| `config.py` | Pydantic Settings singleton. Reads `.env` + env vars, validates types, exposes `settings.X` everywhere. Defines `LlmRole` enum and `model_for(role)` for routing. |
| `logging.py` | Sets up structlog once per process. Forces UTF-8 stdout (Windows fix), silences noisy `httpx`/`httpcore` loggers. Call `setup_logging()` once in `main()`. |
| `llm_client.py` | Async OpenAI-compat HTTP client targeting `LLM_ROUTER_URL`. Exposes `chat`, `chat_text`, `chat_json`, `chat_label`. tenacity-based retries on HTTP/timeout errors. |
| `embedding_client.py` | Async client for BGE-M3 embeddings (`/v1/embeddings`) and BGE reranker (`/v1/rerank`). Exposes `embed`, `embed_one`, `rerank`. Has the `cosine()` helper. |
| `atomic_api.py` | Typed REST client for atomic-server. Wraps the endpoints we actually use: `health`, `settings` get/set, `atoms` CRUD + by-source-url, `tags` list/create, `search`, `find_similar`. Returns dataclasses (`SearchHit`, `AtomSummary`) or raw dicts. |
| `taxonomy.py` | Defines the canonical `TAXONOMY` tree (~80 paths), exposes `walk()`, `all_paths()`, `build_path_map_from_tags()`, and the `TagResolver` class for `path → uuid` lookup. Also `load_from_mapping()` for in-memory refresh (used by brain_api at startup). |
**Imports.** Standard library + `httpx` + `pydantic` + `pydantic-settings` + `structlog` + `tenacity` + `rich`. Nothing internal.
**Imported by.** `brain_api/`, `extractor/`, `lint/`, every script in `scripts/`.
---
## `brain_api/` — the HTTP service
**Purpose.** Containerized FastAPI service that exposes the 5-endpoint
contract Didi's backend already speaks. It does NOT contain business logic
that another module reuses; it ties together `shared` + `extractor` (via
`/v1/ingest`) and translates between Didi's HTTP shape and the brain's
internal capabilities.
**When it runs.** Always. One container per deployment, lifespan-managed by
FastAPI/Uvicorn. The lifespan hook initializes the long-lived clients
(`AtomicClient`, `EmbeddingClient`, `LlmClient`) and refreshes the
`TagResolver` from atomic-server at startup so the image is portable.
**Files inside.**
| File | Role |
|---|---|
| `Dockerfile` | python:3.12-slim, non-root user `brain`, COPY shared/extractor/brain_api, curl healthcheck on /health, CMD `python -m brain_api.run`. |
| `requirements.txt` | Pinned runtime deps. |
| `__init__.py` | Module marker + docstring. |
| `app.py` | The FastAPI app. Defines `lifespan` (startup/shutdown), the `/health` route, and the 5 v1 routes. Each route delegates to a function in `services/`. |
| `run.py` | Uvicorn entry. Reads `BRAIN_API_HOST` and `BRAIN_API_PORT` from env (defaults 127.0.0.1:8090; container overrides to 0.0.0.0). |
| `deps.py` | `AppState` dataclass + module-level singleton accessor. Set by lifespan, read by route handlers. |
| `schemas.py` | **The contract.** Pydantic v2 models for every request/response shape the service speaks: `SearchRequest/Response`, `FetchRequest/Response`, `GatherRequest/Response`, `ImageSearchRequest/Response`, `IngestRequest/Response`, plus `EvidenceItem`, `BrainEvidenceMeta`, `BrainMeta`, etc. |
| `prompts/nli_v1.md` | Versioned prompt template for the stance-vs-query NLI step in `/v1/gather`. |
| `services/mapping.py` | Pure functions converting atomic atoms ↔ Didi's `EvidenceItem` / `FetchedPage` / `SearchResultItem`. Includes `parse_claim_atom_body()` (used by lint too) and the credibility-tag → score mapping. |
| `services/gather.py` | The 5-stage pipeline for `/v1/gather`: context → retrieval → rerank → NLI → evidence build. Returns a `GatherResponse`. |
| `services/search.py` | Lighter version of gather: just semantic search across queries, dedup by parent doc, return `SearchResponse`. |
| `services/fetch.py` | Look up atoms by URL via `atomic.get_atom_by_source_url`. Returns `FetchResponse` with HIT pages and `failed_urls=[{url, "not_in_brain"}]` for misses. |
| `services/ingest.py` | Accept `IngestRequest` (typically a web-module result), create `Type/Document` atoms with inferred tags, optionally schedule background extraction via FastAPI `BackgroundTasks`. |
| `services/nli.py` | The stance-vs-query LLM call. `classify_one()` for a single (claim, evidence) pair, `classify_batch()` for parallel with `MAX_PARALLEL=2` (matches llamacpp backend concurrency). |
**Imports.** `shared`, `extractor.batch` (lazily, only when ingest spawns extraction), `fastapi`, `uvicorn`.
**Imported by.** Nothing — it's the leaf consumer. Didi backend talks to it over HTTP, not Python imports.
---
## `extractor/` — Document → Claim transformation
**Purpose.** Reads `Type/Document` atoms from Atomic, asks Qwen 397B to
extract atomic factual claims, validates each claim against the source
text (substring check on the quote), and creates new `Type/Claim` atoms
with canonical hash-based source URLs and inherited tags.
**When it runs.** Two paths:
1. **Operator-triggered**: `python scripts/07_run_extraction.py` runs
`extractor.batch.run_batch()` over every Document atom that hasn't been
processed at the current prompt version.
2. **Background after ingest**: when `POST /v1/ingest` is called with
`run_extraction: true` (the default), the brain_api's ingest service
schedules a `BackgroundTasks` callback that calls
`extractor.batch.run_batch(only_atom_ids=...)` on just the atoms that
were just created. Fire-and-forget, doesn't block the HTTP response.
Both paths share the same idempotency state (`extractor/_extracted.json`).
**Files inside.**
| File | Role |
|---|---|
| `__init__.py` | Module marker. |
| `_state.py` | `DocExtractionRecord` dataclass + `ExtractionState` class with `has`, `get`, `upsert`, `save`, `_load`. JSON file at `extractor/_extracted.json`, host-only. |
| `extract.py` | Single-document logic. `extract_claims_from_atom(llm, title, language, content)` → `ExtractionResult` with `valid: list[ExtractedClaim]` and `rejected: dict[reason → count]`. Includes the substring quote validation. |
| `push.py` | `push_claim(atomic, parent_atom, claim, parent_title, resolver)` creates one `Type/Claim` atom. URL is `{parent_url}#claim={hash8}` for natural dedup. Inherits parent's Country/Topic/SourceType/Credibility/Language tags, adds Type/Claim + Stance/{X}. |
| `batch.py` | The orchestrator. `run_batch(limit, only_atom_ids)` paginates Document atoms, fetches each fully (list_atoms returns summary only — gotcha!), runs extraction sequentially, pushes claims, saves state after every doc so Ctrl-C is recoverable. |
| `prompts/claim_extraction_v1.md` | Versioned prompt that instructs Qwen 397B to return strict JSON with claims + verbatim quotes. Output format pinned. |
**Imports.** `shared`, `httpx` (transitively).
**Imported by.** `scripts/07_run_extraction.py`, `brain_api/services/ingest.py`.
---
## `lint/` — cross-corpus contradiction detection
**Purpose.** Audit job that finds claim atoms in our corpus that
contradict each other (or are paraphrases). For every claim, asks Atomic
for its semantic neighbors, then asks Qwen 397B to classify each
candidate pair as **EQUIVALENT**, **CONTRADICTORY**, or **INCOMPARABLE**.
Stores all verdicts in a JSON ledger so re-runs only process new pairs.
**When it runs.** Operator-triggered, typically as an overnight job
(~1 pair / 1.5s × ~6000 pairs for 513 atoms ≈ 4 hours). Re-runs are
incremental thanks to the ledger.
**Files inside.**
| File | Role |
|---|---|
| `__init__.py` | Module marker. |
| `_state.py` | `PairVerdict` dataclass, `LintState` class (loads/saves the ledger atomically via tmp+rename). `pair_hash(a, b)` is canonical (sorted) so (A,B) and (B,A) hash the same. `LintStats` for per-run counters. |
| `detector.py` | Single-pair classification. `classify_pair(llm, claim_a, claim_b)` → `PairClassification(label, confidence, error)`. Never raises; falls through to INCOMPARABLE on any error so the batch keeps going. |
| `pairs.py` | `generate_candidates(atomic, source_atom_ids)` walks each source atom, calls `atomic.find_similar` with `MIN_PAIR_SIMILARITY=0.55`, filters to Type/Claim neighbors only, returns canonical de-duplicated pairs. |
| `runner.py` | The orchestrator. `run_lint_pass(limit_atoms, force)` does the full flow: load atom ids → generate candidates → drop already-evaluated → bulk-fetch full atom bodies → classify with `MAX_PARALLEL=2` → save ledger every 25 verdicts → ALWAYS save on exit (even crash). |
| `reporter.py` | Pretty printing with `rich`: `render_stats(stats)`, `render_contradictions(state, top_n, min_confidence)`, `render_equivalents(state)`. |
| `prompts/pair_nli_v1.md` | Versioned prompt for pair classification. Same-language and cross-language are judged identically. |
**Imports.** `shared`, `brain_api.services.mapping` (for `parse_claim_atom_body`).
**Imported by.** `scripts/10_run_lint.py`, `scripts/11_show_contradictions.py`.
---
## `scripts/` — operator CLI tools
**Purpose.** One-shot maintenance and bootstrap commands that operators
run from the host venv. Each is self-contained, idempotent, prints a
clean rich-formatted report, and exits with a meaningful code.
**When it runs.** Operator types `python scripts/XX_*.py`.
**Files inside.** All numbered for execution order during a fresh deploy.
| Script | What it does | Idempotent? | Typical runtime |
|---|---|---|---|
| `01_sanity_full.py` | Validates upstream stack: pings router, lists models, runs JSON extraction + NLI + Romanian + cross-lingual + embed cosine + reranker. Writes a JSON report. | yes | ~10 s |
| `02_bootstrap_atomic.py` | Waits for atomic-server, claims the instance via `/api/setup/claim` if needed, configures `provider=openai_compat` pointed at BGE-M3, disables `auto_tagging_enabled`, patches `ATOMIC_TOKEN` into `.env`. | yes | ~5 s |
| `03_sanity_atomic.py` | End-to-end brain check: create test atom → wait for embedding → semantic search retrieves it → cleanup. Catches "atomic + Postgres + BGE-M3 wired" issues. | yes (cleanup deletes the atom) | ~5-10 s |
| `04_seed_taxonomy.py` | Reads `shared.taxonomy.TAXONOMY`, creates any missing tags in atomic-server preserving parent_id structure, writes `shared/_tag_ids.json`. | yes | ~10 s |
| `05_import_wikipedia_seed.py` | Imports a curated seed list of Wikipedia articles (currently vaccines EN+RO) via the MediaWiki action API, dedupes by canonical URL, creates Type/Document atoms with proper tags. | yes (URL dedup) | ~30 s for 19 articles |
| `06_validate_queries.py` | 4 doc-level queries against `/api/search` with reranker, prints top hits. Sanity that retrieval works on the corpus. | yes (read-only) | ~5-10 s |
| `07_run_extraction.py` | Calls `extractor.batch.run_batch()`. With `--limit N` for smoke tests. | yes (state file) | ~1 min/document |
| `08_validate_claims.py` | 4 claim-level queries, splits doc/claim hits, reranks claims. Sanity that claim-level retrieval works. | yes | ~10-20 s |
| `09_brain_api_demo.py` | Hits all 5 brain_api endpoints with realistic inputs, validates schema compliance and HIT/MISS cache_status. | yes | ~30 s |
| `10_run_lint.py` | Calls `lint.runner.run_lint_pass()`. With `--limit N` to cap source atoms, `--force` to re-evaluate cached pairs. | yes (ledger) | ~30 s × N atoms |
| `11_show_contradictions.py` | READ ONLY. Loads `lint/_contradictions.json`, renders top contradictions and (with `--equivalents`) paraphrase clusters. | yes | <1 s |
| `bootstrap_deploy.sh` | Bash orchestrator for fresh-server deploy. Preflight → compose build+up → wait healthy → venv → 02 → 04 → optional 05+07 → smoke test. Idempotent, color-coded output. | yes | ~5 min + 15 min if importing corpus |
**Imports.** Each script `sys.path.insert`s the project root then imports
`shared`, `brain_api`, `extractor`, or `lint` as needed.
**Imported by.** Nothing. They are leaf entry points.
---
## `infra/` — containerization
**Purpose.** Defines the deployment topology.
**Files inside.**
| File | Role |
|---|---|
| `docker-compose.yml` | Three services: `postgres` (pgvector/pgvector:pg16), `atomic-server` (ghcr.io/kenforthewin/atomic-server:latest with overridden entrypoint to use `--data-dir` instead of legacy `--db-path`, and `ATOMIC_STORAGE=postgres` env), `brain-api` (built from `brain_api/Dockerfile`). Two named volumes (`didibrain-pg-data`, `didibrain-atomic-data`), one bridge network (`didibrain`). All services have `restart: unless-stopped` and healthchecks. |
**When it runs.** `docker compose up` from the operator (or via `bootstrap_deploy.sh`).
---
# Data flow — three concrete scenarios
The codebase has exactly three control flows worth memorizing.
## Scenario 1 — request time: Didi calls `POST /v1/gather`
```
Didi backend
│
│ POST http://brain-host:8090/v1/gather
│ {"claim": "vaccinurile cauzeaza autism", "max_evidence": 5, "run_nli": true}
▼
brain_api/app.py: post_gather()
│
├─▶ deps.get_state() (returns shared AppState with clients)
│
├─▶ services/gather.py: gather(req, atomic, embed, llm, resolver)
│
│ stage 1 context → mapping.detect_language_simple(claim)
│ (~0 ms)
│
│ stage 2 retrieval → atomic.search(claim, mode="semantic", limit=100)
│ → filter to Type/Claim only
│ (~300 ms; atomic embeds query via BGE-M3 internally)
│
│ stage 3 rerank → atomic.get_atom(...) for top-15 claim atoms
│ → embed.rerank(claim, claim_texts)
│ (~1000 ms; calls reranker via BGE cross-encoder)
│
│ stage 4 nli → mapping.parse_claim_atom_body() per top result
│ → nli.classify_batch(llm, claim, evidence_texts)
│ (parallel calls to Qwen 397B, ~3-4 sec; one per parent doc)
│
│ stage 5 evidence → atomic.get_atom_by_source_url() per parent doc
│ → mapping.evidence_from_parent() builds EvidenceItem
│ (~400 ms parallel)
│
├─▶ Returns GatherResponse with brain_meta.cache_status set based on top
│ relevance score: >=0.6 HIT, 0.3-0.6 PARTIAL, <0.3 MISS.
│
▼
Didi backend receives JSON, branches on cache_status.
```
**Total**: ~5-6 sec with NLI, ~1.5 sec without.
## Scenario 2 — operator time: fresh-server deployment
```
operator what runs
═══════ ══════════
ssh server, cd ~/didibrain (none)
cp .env.example .env, vim .env (none)
./scripts/bootstrap_deploy.sh
│
├─ preflight docker, python3, curl available
├─ docker compose build brain-api brain_api/Dockerfile builds the image
├─ docker compose up -d starts postgres, atomic-server, brain-api
├─ wait_healthy x3 polls docker inspect for "healthy"
├─ python3 -m venv .venv local Python venv
├─ pip install ... operator deps in venv
├─ python scripts/02_bootstrap_atomic.py
│ │
│ ├─ atomic.health() verify atomic responding
│ ├─ atomic.setup_status() is the instance claimed?
│ ├─ atomic._request POST /api/setup/claim if not, claim and get token
│ ├─ patch_env_token() write ATOMIC_TOKEN= back to .env
│ └─ atomic.set_setting() x8 provider=openai_compat, BGE URLs, dim, etc.
│
├─ python scripts/04_seed_taxonomy.py
│ │
│ ├─ atomic.list_tags() what's already there
│ ├─ for each path in TAXONOMY: atomic.create_tag() if missing
│ └─ resolver.save() write shared/_tag_ids.json (host-only)
│
├─ python scripts/05_import_wikipedia_seed.py (optional, on by default)
│ │
│ ├─ for each title in SEED_EN + SEED_RO:
│ │ fetch_article() → atomic.create_atom() with proper tag_ids
│ └─ Atomic chunks + embeds in background via BGE-M3
│
├─ python scripts/07_run_extraction.py (optional, on by default)
│ │
│ ├─ list Type/Document atoms
│ ├─ for each not in extractor/_extracted.json:
│ │ fetch full atom → extract.extract_claims_from_atom() (Qwen 397B)
│ │ push.push_claim() per valid claim → atomic.create_atom()
│ └─ save state every doc
│
└─ smoke test: curl /health + curl /v1/gather, parse cache_status
```
## Scenario 3 — background time: Didi falls back to web module, posts result back
```
Didi backend
│
│ POST /v1/gather → cache_status=MISS
│
├─ Didi backend calls live web module (expensive)
│
│ POST /v1/ingest with the web-module result
▼
brain_api/app.py: post_ingest()
│
├─▶ services/ingest.py: ingest(req, atomic, resolver, background)
│
│ for each evidence in req.evidence:
│ atomic.get_atom_by_source_url() (dedup check)
│ if not exists:
│ evidence_to_markdown(evidence)
│ build_tag_ids_for_evidence() (Type/Document + Credibility from
│ credibility_score + Language)
│ atomic.create_atom() (synchronous)
│
│ if req.run_extraction and created_ids:
│ background.add_task(_run_extraction_background, created_ids)
│
├─▶ Returns IngestResponse immediately (extraction is fire-and-forget)
│
▼
FastAPI BackgroundTasks: _run_extraction_background(created_ids)
│
└─▶ extractor.batch.run_batch(only_atom_ids=set(created_ids))
├─ runs Qwen 397B claim extraction
├─ pushes Type/Claim atoms
└─ saves extractor/_extracted.json
```
**Effect**: next time Didi asks the same claim, brain returns HIT and Didi
doesn't need to call the web module.
---
# State files reference
DidiBrain has **three host-only JSON state files** (gitignored). They are
all idempotency ledgers — never primary data — so they can be regenerated
from Atomic without loss.
| File | Owner | Purpose | Recreatable? |
|---|---|---|---|
| `shared/_tag_ids.json` | `scripts/04_seed_taxonomy.py` | Maps canonical paths like `Country/Romania` → atom tag UUIDs. Used by every script that creates atoms. | yes — re-run script 04 |
| `extractor/_extracted.json` | `scripts/07_run_extraction.py` (and brain_api ingest) | Per-document extraction log: which docs have been processed, with which prompt version, how many claims came out. Skip already-done docs on rerun. | yes — delete and re-run; will re-extract everything |
| `lint/_contradictions.json` | `scripts/10_run_lint.py` | Per-pair verdict ledger with the EQUIVALENT / CONTRADICTORY / INCOMPARABLE labels. Skip already-done pairs on rerun. | yes — delete and re-run; will re-classify everything |
The brain_api **container** does NOT use any of these files. It calls
`atomic.list_tags()` at startup to refresh `TagResolver` in-memory, so
the image stays portable.
---
# Environment variables — full table
All loaded by `shared/config.py` via Pydantic Settings from `.env` at the
project root. The brain_api container also receives these via
`env_file: ../.env` in docker-compose, with `ATOMIC_URL` overridden to
`http://atomic-server:8080` (Docker DNS).
| Var | Default | What it controls | Required? |
|---|---|---|---|
| `LLM_ROUTER_URL` | `http://localhost:14011` | URL of the unified LLM router (vLLM + llamacpp behind it) | yes (set to `10.11.10.17:14011` in our setup) |
| `LLM_ROUTER_API_KEY` | empty | optional bearer | no |
| `LLM_VLLM_URL` | `http://localhost:14001` | direct vLLM URL (fallback only, currently unused) | no |
| `LLM_LLAMACPP_URLS` | empty | comma-separated direct llamacpp URLs (fallback) | no |
| `MODEL_REASONING` | `Qwen3.5-397B-A17B` | model id for extraction, NLI, gather | yes |
| `MODEL_REASONING_BACKEND` | `llamacpp` | router backend hint | yes |
| `MODEL_FAST` | `qwen3.5` | fast model id (currently disabled) | no |
| `MODEL_FAST_BACKEND` | `vllm` | fast backend hint | no |
| `MODEL_FAST_ENABLED` | `false` | enable fast model in routing | no |
| `MODEL_VISION` | `gemma-3-27b-it` | vision model id (currently down) | no |
| `MODEL_VISION_URL` | empty | direct vision endpoint | no |
| `MODEL_VISION_ENABLED` | `false` | enable vision in routing | no |
| `EMBEDDING_URL` | `http://10.11.10.15:8200` | BGE-M3 vLLM endpoint | yes |
| `EMBEDDING_API_KEY` | empty | optional bearer | no |
| `EMBEDDING_MODEL` | `BAAI/bge-m3` | model id sent in requests | yes |
| `EMBEDDING_DIM` | `1024` | vector dimension (must match Atomic's setting) | yes |
| `EMBEDDING_MAX_TOKENS` | `8192` | input length cap | yes |
| `RERANKER_URL` | `http://10.11.10.15:8100` | BGE-reranker-v2-m3 endpoint | yes |
| `RERANKER_API_KEY` | empty | optional bearer | no |
| `RERANKER_MODEL` | `BAAI/bge-reranker-v2-m3` | model id sent in requests | yes |
| `ATOMIC_URL` | `http://localhost:8088` | atomic-server URL (host scripts use this; brain_api container overrides to docker DNS) | yes |
| `ATOMIC_TOKEN` | empty | API token, set by `02_bootstrap_atomic.py` | yes (auto-populated) |
| `POSTGRES_USER` | `atomic` | Postgres user | yes |
| `POSTGRES_PASSWORD` | `atomic_dev_changeme` | Postgres password (override on server!) | yes |
| `POSTGRES_DB` | `atomic` | Postgres database name | yes |
| `POSTGRES_PORT` | `5434` | host-side port mapping | yes |
| `LOG_LEVEL` | `INFO` | structlog filter level | no |
**Container-only env vars set inside the Dockerfile** (not in `.env`):
| Var | Set to | Why |
|---|---|---|
| `BRAIN_API_HOST` | `0.0.0.0` | so Docker port mapping reaches uvicorn |
| `BRAIN_API_PORT` | `8090` | container internal port |
| `PYTHONIOENCODING` | `utf-8` | UTF-8 logging on Linux base image |
---
# Versioned prompts
Three prompts. Each has its own version number. Bumping a version means
re-running the relevant batch (extractor or lint) to re-process what's
been done.
| Prompt file | Used by | Output | Version |
|---|---|---|---|
| `extractor/prompts/claim_extraction_v1.md` | `extractor.extract.extract_claims_from_atom` | JSON list of `{claim, quote, stance, confidence}` from a document | v1 |
| `brain_api/prompts/nli_v1.md` | `brain_api.services.nli.classify_one` | JSON `{label: SUPPORTS/CONTRADICTS/NEUTRAL, confidence}` | v1 |
| `lint/prompts/pair_nli_v1.md` | `lint.detector.classify_pair` | JSON `{label: EQUIVALENT/CONTRADICTORY/INCOMPARABLE, confidence}` | v1 |
To bump: copy the file, rename to `_v2.md`, update the constant
`PROMPT_VERSION = "v2"` in the Python file, re-run the relevant batch.
The state files (`extractor/_extracted.json`, `lint/_contradictions.json`)
include `prompt_version` per record, so re-runs only re-process items at
the old version.
---
# When something breaks — start here
```
Something with /v1/gather failing?
│
├─ check brain_api/app.py route handler — it just delegates
├─ → services/gather.py — the 5-stage pipeline
├─ → services/mapping.py — atom shape conversion
└─ → shared/atomic_api.py + embedding_client.py + llm_client.py — upstream calls
│
└─ run scripts/01_sanity_full.py first to verify upstream stack
Atomic returning 500?
│
├─ docker logs didibrain-atomic — likely BGE unreachable
└─ curl http://10.11.10.15:8200/v1/models on host — VPN check
Claim extraction acting weird?
│
├─ extractor/_extracted.json — see what's been processed
├─ extractor/extract.py — substring quote validation may be rejecting
└─ extractor/prompts/claim_extraction_v1.md — prompt drift?
Lint pass slow / hanging?
│
├─ lint/runner.py MAX_PARALLEL — must match llamacpp backend count (we have 2)
├─ lint/detector.py PER_CALL_TIMEOUT_S — currently 30s
└─ lint/_contradictions.json saved every 25 — Ctrl-C safe to interrupt
Atomic doesn't have my tag?
│
├─ shared/taxonomy.py — is the path defined in TAXONOMY?
├─ shared/_tag_ids.json — is it in the cache?
└─ run scripts/04_seed_taxonomy.py to refresh both
```
For deeper troubleshooting see `STATUS.md` "Troubleshooting" section.
---
# Verification Cache (added 2026-04-23)
## What
A relational layer **alongside** the knowledge graph (Atomic/atoms) that stores
the LLM verification result produced by didi-backend after it runs NLI + verdict
analysis on evidence. This lets backend skip its expensive LLM call on
subsequent requests for the same `(claim, tier)` pair.
**Backend owns the prompt + model + status logic.** Brain is a pure cache: it
stores the opaque JSON blob backend produces, and returns it 1:1 on read.
Zero prompt sync between the two sides.
## Table: `brain_verification_cache`
Lives in the **same Postgres instance atomic-server uses** — we share the DB
connection with `brain_` prefix for trivial isolation. Asyncpg pool connects
directly (doesn't go through atomic-server).
Column summary:
| Column | Purpose |
|---|---|
| `claim_hash` | `sha256(normalize_claim(claim_text))` — normalization strips diacritics, lowercases, collapses whitespace, trims trailing `.?!` |
| `tier` | `'free'` or `'premium'` — **separate cache per tier** (different LLMs used) |
| `evidence_hash` | `sha256(sorted(lowercased_urls))` — **metadata only**, not part of the key |
| `evidence_urls` | jsonb array of URLs the verification was written for |
| `prompt_hash` | Backend's current prompt hash; used for staleness detection on read |
| `framework_version` | Backend's current thresholds hash; used for staleness on read |
| `verification_processed` | jsonb — UI-ready payload (status + sources + reasoning) |
| `verification_raw` | jsonb — raw LLM output; lets backend recompute when framework changes |
| `expires_at` | TTL, default 30 days from write |
**Unique key:** `(claim_hash, tier)`. UPSERT last-wins.
Why not `(claim_hash, evidence_hash, tier)`? Because the URLs backend has at
write-time rarely match what brain's `/v1/gather` returns at read-time
(brain uses its own semantic retrieval, which may rank differently than
backend's original source list). Including evidence in the key made the
cache unreachable. v2 drops it.
## Flow
### Write (backend → brain, fire-and-forget)
```
backend runs its LLM verification call
│
▼
POST /v1/verification_cache
claim, evidence_urls, tier,
model, prompt_hash, framework_version,
verification_raw, verification_processed
│
▼
brain normalizes + hashes + UPSERTs row (claim_hash, tier)
│
▼
200 OK (cached: true, claim_hash, evidence_hash, expires_at)
```
### Read (via /v1/gather extension)
```
backend wants to check if verification exists for this claim
│
▼
POST /v1/gather
claim, include_verification: true, tier,
prompt_hash, framework_version
│
▼
brain runs its normal gather (retrieval + rerank + evidence)
│
▼
brain queries brain_verification_cache (claim_hash, tier)
│
▼
returns GatherResponse with brain_meta.verification_staleness:
- miss → null; backend runs LLM + writes cache
- stale_prompt → null; backend runs LLM + writes cache
- stale_framework→ returns verification_raw; backend recomputes status locally (zero LLM)
- fresh → returns verification_processed; direct to UI
```
See `CONTRACT_VERIFICATION_CACHE.md` for the full field-by-field spec.
## Files
- `brain_api/db.py` — asyncpg pool + idempotent schema migrations (runs in lifespan)
- `brain_api/services/verification_cache.py` — normalize, hash, upsert, lookup, freshness decider
- `brain_api/schemas.py` — `GatherRequest` extension, `VerificationCacheWriteRequest/Response`, `BrainMeta` extension
- `brain_api/app.py` — `POST /v1/verification_cache` endpoint wired in lifespan
- `brain_api/services/gather.py` — cache lookup after evidence build, attach to `brain_meta`
---
# Glossary
| Term | Meaning |
|---|---|
| **atom** | The unit of storage in Atomic. Markdown content + metadata + tags. Two flavors: `Type/Document` (a full source article) and `Type/Claim` (an atomic factual claim extracted from a document). |
| **Atomic** | The Rust knowledge-graph backend (kenforthewin/atomic). DidiBrain uses its REST API. Runs as `didibrain-atomic` container. |
| **brain_api** | Our FastAPI service. The HTTP face DidiBrain shows to Didi backend. |
| **chunk** | A piece of an atom's content split for embedding. Atomic does this internally; we never see chunks directly. |
| **HIT / PARTIAL / MISS** | `brain_meta.cache_status` returned by `/v1/gather`. Tells Didi backend whether the brain had relevant knowledge or it should fall back to web module. |
| **Lint pass** | Background contradiction-detection job. Reads existing claim atoms, finds contradictory pairs cross-corpus. |
| **NLI** | Natural Language Inference. The "decide if A supports / contradicts / is neutral toward B" classification. We do it twice: stance-vs-query (gather) and pair-equivalence (lint). |
| **prompt_version** | A string like "v1" pinned per prompt file. State files record it so we can bump prompts and re-process selectively. |
| **reranker** | Cross-encoder (BGE-reranker-v2-m3). Takes (query, document) pairs and outputs a precise relevance score. Used after embedding retrieval to refine top-K. |
| **stance_in_source** | What the source itself asserts about a claim (ASSERTS/REPORTS/REFUTES/QUESTIONS/NEUTRAL). Set during extraction. |
| **stance_vs_query** | What an evidence claim says about Didi user's input claim (SUPPORTS/CONTRADICTS/NEUTRAL). Set during gather's NLI stage. |
| **TagResolver** | Helper that maps canonical paths like `Country/Romania` to Atomic tag UUIDs. Loaded from `_tag_ids.json` (scripts) or refreshed from atomic at startup (brain_api container). |
| **TAXONOMY** | The canonical tree in `shared/taxonomy.py`. Single source of truth for what tags exist. |

611
ai_platform/modules/didi_brain/AUDIT.md Normal file
View file

@ -0,0 +1,611 @@
# AUDIT COMPLET — Atomic Knowledge Base
> Audit generat prin analiză paralelă cu 6 agenți specializați.
> Versiune cod auditată: `1.19.2` (clone GitHub `kenforthewin/atomic`)
> Data: 2026-04-10
---
## 0. CE ESTE ATOMIC (TL;DR)
**Atomic** este un **personal knowledge base** care transformă note Markdown ("atoms") într-un **graf semantic AI-augmented**. Fiecare notă este automat:
1. **Chunked** (bucățită markdown-aware)
2. **Embedded** (vectorizată prin LLM)
3. **Tagged** (taguri ierarhice extrase de LLM)
4. **Linked** (edges semantice către alte atoms similare)
Pe baza acestui graf oferă: **căutare semantică**, **wiki articles auto-sintetizate cu citații**, **canvas vizual interactiv**, **chat agentic RAG**, **sincronizare offline**.
Rulează ca:
- **App desktop Tauri** (macOS/Linux/Windows) — sidecar care pornește atomic-server local
- **Server headless Docker/Fly.io** — REST + WebSocket + MCP
- **iOS native SwiftUI** — client thin HTTP
- **Browser extension** (Web Clipper) — capturi web
- **Plugin Obsidian** — sincronizare vault
- **Bot Discord** — capturi mesaje/threads
- **MCP server** — expune knowledge base la Claude Desktop & alte AI tools
---
## 1. ARHITECTURA GENERALĂ — "Core + Thin Wrappers"
```
┌────────────────────┐
│ atomic-core │ ← TOATĂ logica de business
│ (Rust crate, no │ (no actix/no tauri deps)
│ framework deps) │
└─────────┬──────────┘
┌───────────────┼─────────────────┐
▼ ▼ ▼
┌────────────┐ ┌──────────────┐ ┌───────────────┐
│ src-tauri │ │atomic-server │ │ mcp-bridge │
│ (sidecar │ │(REST+WS+MCP) │ │ (stdio→HTTP │
│ launcher) │ │ actix-web │ │ pentru Claude│
└─────┬──────┘ └──────┬───────┘ │ Desktop) │
│ │ └───────┬───────┘
▼ ▼ ▼
┌──────────────────────────────┐ ┌──────────────┐
│ React UI (TypeScript+Vite) │ │ MCP clients │
│ (folosește același │ │ (Claude etc) │
│ HttpTransport în desktop │ └──────────────┘
│ și browser) │
└──────────────────────────────┘
+
┌──────────────┐ ┌─────────────┐ ┌─────────────┐
│ iOS app │ │ Browser ext │ │ Discord bot │
│ SwiftUI │ │ (MV3) │ │ Obsidian │
└──────────────┘ └─────────────┘ └─────────────┘
```
**Principiu cheie:** `atomic-core` este transport-agnostic. Toate evenimentele async sunt livrate prin **callback-uri** (`Fn(EmbeddingEvent)`, `Fn(ChatEvent)`). Fiecare wrapper traduce callback-uri în mecanismul lui (Tauri `app_handle.emit`, sau actix `broadcast::Sender → WebSocket`).
---
## 2. WORKSPACE — STRUCTURA REPO
```
atomic/
├── Cargo.toml # Workspace Rust (5 crate-uri)
├── package.json # Frontend npm (versiune 1.19.2)
├── crates/
│ ├── atomic-core/ # ~16.4k linii Rust — toată logica
│ ├── atomic-server/ # ~7.5k linii — actix-web wrapper
│ ├── mcp-bridge/ # ~285 linii — bridge stdio↔HTTP MCP
│ └── atomic-cloud/ # Control plane SaaS (Stripe + Fly.io)
├── src-tauri/ # Tauri v2 desktop launcher
├── src/ # React 18 + TS + Tailwind v4 + Zustand
├── ios/ # SwiftUI + XcodeGen (Swift 6, iOS 17+)
├── extension/ # Manifest V3 Web Clipper
├── plugins/
│ ├── discord/ # discord.js 14 bot
│ └── obsidian-plugin/ # Obsidian plugin (TS)
├── scripts/ # 15 scripturi Node (build, import, reset)
├── docker/ # nginx, supervisord, litestream
├── docs/ # 5 spec markdown (planuri arhitecturale)
├── docker-compose.{yml,dev,test,build}.yml
├── Dockerfile # Multi-stage: server, web, all-in-one
├── server.dockerfile / web.dockerfile
└── fly.toml.example
```
---
## 3. `atomic-core` — INIMA SISTEMULUI
### 3.1 Module principale
| Modul | Linii | Rol |
|---|---:|---|
| `lib.rs` | 3,399 | Facade `AtomicCore`, orchestrare toate operațiile |
| `db.rs` | 916 | Init SQLite, migrații v0→v10, PRAGMA tuning, schema |
| `manager.rs` | 511 | Multi-database manager (lazy-load instance per DB) |
| `models.rs` | 696 | `Atom`, `Tag`, `WikiArticle`, `ChatMessage`, `SemanticEdge` etc. |
| `registry.rs` | 863 | Multi-database registry (cross-DB settings/tokens) |
| `tokens.rs` | 430 | API tokens (SHA-256 hash + revocable) |
| `settings.rs` | 191 | Key-value settings (provider, modele, thresholds) |
| `executor.rs` | 39 | Background runtime (4 worker threads) + semafoare concurrență |
| `projection.rs` | 287 | PCA/t-SNE pentru canvas (cache poziții) |
| `error.rs` | 70 | `AtomicCoreError` enum |
### 3.2 Pipeline de procesare atom (fire-and-forget)
```
POST /api/atoms (sau core.create_atom)
│
▼ caller primește atom-ul instant (saved)
│
│ background, prin executor + callback Fn(EmbeddingEvent):
│
├─[1]─► CHUNKING (markdown-aware)
│ respectă code blocks, headers, paragraphs
│ token-aware via tiktoken-rs
│
├─[2]─► EMBEDDING (provider configurat)
│ OpenRouter / Ollama / OpenAI-compat
│ inserează în vec_chunks (sqlite-vec virtual table)
│ → emit EmbeddingEvent::Started/Complete/Failed
│
├─[3]─► AUTO-TAGGING (LLM structured outputs)
│ categorii: Topics, People, Locations,
│ Organizations, Events
│ → emit TaggingComplete/Failed/Skipped
│
├─[4]─► SEMANTIC EDGES (cosine similarity)
│ threshold: 0.5 default → tabela semantic_edges
│
└─[5]─► WIKI INCREMENTAL UPDATE (dacă tagged)
LLM integrează atom-ul nou în wiki article-ul existent
```
**Praguri (thresholds):**
- Similaritate edges/related atoms: **0.5**
- Semantic search & wiki chunk selection: **0.3**
- Formula: `similarity = 1.0 - (distance² / 2.0)` (din Euclidean al sqlite-vec pe vectori normalizați)
### 3.3 Storage abstraction (SQLite + Postgres)
`atomic-core` are un trait `StorageBackend` cu **două implementări**:
- **SQLite** (`storage/sqlite/`) — default, prin `rusqlite 0.32` (bundled) + `sqlite-vec 0.1.6` (vector search)
- **Postgres** (`storage/postgres/`) — feature-gated, prin `sqlx 0.8` + `pgvector 0.4`
Dispatch macro `dispatch!` rulează 111 metode peste backend-uri. Există un plan documentat (`docs/plan-async-migration.md`) pentru a face întregul `AtomicCore` async-native (acum Postgres face sync→async bridge prin `PG_RUNTIME.block_on`).
### 3.4 Schema bază date (data DB-uri)
Tabele cheie:
- `atoms` — content, source_url, embedding_status, tagging_status, created/updated_at
- `atom_chunks` — chunks per atom (cu offset-uri în text)
- `vec_chunks` — **virtual table sqlite-vec** (vector index)
- `tags` — ierarhie (parent_id), category
- `atom_tags` — many-to-many
- `semantic_edges` — pereche atom_a, atom_b, similarity
- `atom_clusters` — rezultate clustering
- `atom_positions` — poziții persistate canvas
- `wiki_articles` — content + metadata per tag
- `wiki_proposals` — (M1+) human-in-the-loop updates
- `conversations`, `chat_messages` — chat agentic
- FTS virtual table pentru keyword search
`registry.db` (separat): `settings`, `api_tokens`, `databases` (UUIDs + nume).
### 3.5 AI Provider abstraction
Trait-uri în `providers/traits.rs`:
- `EmbeddingProvider` — generare batch embeddings
- `LlmProvider` — chat completions
- `StreamingLlmProvider` — streaming + tool calling
Implementări:
- `providers/openrouter/` — cloud, OAuth flow, modele separate per capability (embedding/tagging/wiki/chat)
- `providers/ollama/` — local, **auto-discovery modele**
- `providers/openai_compat/` — generic (Azure OpenAI, Groq, Together, etc.)
Selecția runtime: factory întoarce `Arc<dyn Trait>` în funcție de setting `provider_type`.
### 3.6 Wiki synthesis
- **Generation**: LLM produce articol din atomii unui tag, cu citații inline (`[atom_id]`)
- **Centroid mode** (`wiki/centroid.rs`): selecție chunks după centroid embedding al tagului
- **Agentic mode** (`wiki/agentic.rs`): agent care pune query-uri pe baza de cunoștințe
- **Section operations** (`wiki/section_ops.rs`): `WikiSectionOp::{NoChange, AppendToSection, ReplaceSection, InsertSection}` — LLM emite *operații* peste secțiuni, **nu rewrite total** → diff-uri review-abile (M1 wiki proposals)
- **Versioning**: tabela `wiki_versions` păstrează istoric
### 3.7 Chat agentic / RAG
- Conversații pot fi **scoped la tag-uri**
- Agent are tool-uri: search semantic, read atom, citation
- Streaming via `ChatEvent::{Delta, ToolStart, ToolComplete, Complete, CanvasAction, Error}`
- `ChatCanvasAction` permite agentului să declanșeze acțiuni vizuale în UI
### 3.8 Ingestion URL & RSS & Obsidian
- `ingest/fetch.rs` — HTTP GET cu reqwest
- `ingest/extract.rs` — HTML → Markdown via `dom_smoothie 0.15`
- `ingest/obsidian.rs` — vault Obsidian, păstrează `[[wikilinks]]`, extrage tags din foldere + YAML frontmatter
- RSS — `feed-rs 2.3`, polling configurabil per feed (default 60s)
---
## 4. `atomic-server` — REST + WebSocket + MCP
### 4.1 Stack
- **actix-web 4.9** + actix-cors + actix-ws
- **rmcp 0.15** + rmcp-actix-web 0.11 (MCP Streamable HTTP)
- **utoipa 5** + utoipa-scalar (OpenAPI auto-generat la `/api/docs`)
- **clap 4** (CLI cu subcomenzi)
- **tokio broadcast channel** (256 buffer) pentru events
### 4.2 ~78 Endpoints REST (grupate)
| Domeniu | # | Exemple |
|---|---:|---|
| **Atoms** | 10 | `GET/POST/PUT/DELETE /api/atoms`, `/api/atoms/bulk`, `/api/atoms/{id}/embedding-status`, `/api/atoms/by-source-url`, `/api/atoms/sources` |
| **Tags** | 5 | `GET/POST/PUT/DELETE /api/tags`, `/api/tags/{id}/children` |
| **Search** | 2 | `POST /api/search` (modes: keyword/semantic/hybrid), `GET /api/atoms/{id}/similar` |
| **Wiki** | 13 | `/api/wiki/{tag_id}` cu generate, update, versions, suggestions, proposal/{accept,dismiss}, related, links |
| **Embeddings pipeline** | 8 | process-pending, process-tagging, retry/{atom_id}, reembed-all, reset-stuck, status |
| **Canvas** | 5 | positions GET/PUT, atoms-with-embeddings, level, global (PCA proj) |
| **Graph** | 3 | edges, neighborhood/{atom_id}, rebuild-edges |
| **Clustering** | 3 | compute, get clusters, connection-counts |
| **Chat** | 8 | conversations CRUD, scope add/remove tags, send-message |
| **Settings** | 5 | get/set, test-openrouter, test-openai-compat, models, embedding-models |
| **Databases** | 7 | list, create, rename, delete, activate, set-default, stats |
| **Feeds (RSS)** | 6 | list/create/get/update/delete, poll |
| **Ingest/Import** | 3 | ingest URL (single/batch), import Obsidian vault |
| **Ollama** | 5 | test, models (all/embedding/llm), provider verify |
| **Auth tokens** | 3 | create, list, revoke |
| **Setup** | 2 | status, claim instance |
| **OAuth 2.0 (MCP)** | 7 | `.well-known/*`, `/oauth/{register,authorize,token}` cu PKCE + DCR |
| **Utils & logs** | 3 | sqlite-vec check, compact-tags, logs export |
| **Public** | 2 | `/health`, `/api/docs/openapi.json` |
### 4.3 WebSocket
- Endpoint: `GET /ws?token=<api_token>`
- Subscribe la `tokio::sync::broadcast::Sender<ServerEvent>` (buffer 256)
- `ServerEvent` enum (~25 variante) include:
- **Embedding pipeline**: `EmbeddingStarted/Complete/Failed`, `TaggingComplete/Failed/Skipped`, `BatchProgress`
- **Atom lifecycle**: `AtomCreated`
- **Ingestion**: `IngestionFetchStarted/Complete/Failed/Skipped`, `IngestionComplete/Failed`
- **Feeds**: `FeedPollComplete/Failed`
- **Chat streaming**: `ChatStreamDelta`, `ChatToolStart/Complete`, `ChatComplete`, `ChatCanvasAction`, `ChatError`
- **Import**: `ImportProgress`
### 4.4 MCP endpoint `/mcp`
- **Transport**: Streamable HTTP (stateful, SSE keep-alive 30s)
- **Auth**: `McpAuth` middleware → 401 cu `WWW-Authenticate` care indică spre `/.well-known/oauth-protected-resource` (Claude.ai compatible)
- **Multi-DB**: `?db=<uuid>` în query
- **Tools expuse:**
1. `semantic_search(query, limit)` — hybrid search
2. `read_atom(atom_id, limit, offset)` — paginated 500 lines
3. `create_atom(content, source_url)` — broadcastează `AtomCreated`
4. `update_atom(atom_id, content, source_url)`
### 4.5 Auth
- **API tokens**: SHA-256 hash în DB, prefix de 8 chars pentru lookup rapid
- `BearerAuth` middleware pe `/api/*`, `McpAuth` pe `/mcp`
- `last_used_at` updated fire-and-forget
- **OAuth 2.0 + PKCE + DCR** (1139 linii în `routes/oauth.rs`):
- Dynamic Client Registration → genere client_secret (32 bytes random, base64url)
- Authorization endpoint cu consent HTML
- Code → 5 min expiration, hash storage
- Token exchange cu PKCE S256 verification
- Folosit pentru Claude.ai remote MCP
### 4.6 CLI
```bash
atomic-server [--data-dir PATH] serve --port 8080 --bind 127.0.0.1 \
--public-url https://... --storage sqlite|postgres
atomic-server token create --name "..."
atomic-server token list
atomic-server token revoke <id>
```
### 4.7 Startup behavior
1. Init logging (tracing + ring buffer 1000 entries pentru `/api/logs`)
2. `DatabaseManager::new()` (SQLite sau Postgres)
3. Migrate legacy tokens
4. Create broadcast channel(256)
5. **Recovery**: reset atoms blocate în `processing`, process pending embeddings/tagging
6. **Spawn RSS poll loop** (60s tick, all DBs)
7. Bind HTTP server (4 workers), CORS permissive
8. Graceful shutdown → `PRAGMA optimize`
---
## 5. `mcp-bridge` — stdio↔HTTP
Binary mic (~285 linii) care se compilează cross-platform și e embedded ca **sidecar Tauri** + distribuit pentru Claude Desktop.
**Flow:**
1. Citește JSON-RPC din **stdin**
2. POST la `http://127.0.0.1:44380/mcp` cu header `Mcp-Protocol-Version: 2025-03-26`
3. Captează `mcp-session-id` din răspunsul `initialize`, îl folosește pe requesturile următoare
4. Parsează SSE (`text/event-stream`) și emite linii data: ca JSON-RPC pe **stdout**
5. Timeout HTTP: **300s** (operații AI lungi)
Env vars: `ATOMIC_HOST` (127.0.0.1), `ATOMIC_PORT` (44380).
Config Claude Desktop:
```json
{
"mcpServers": {
"atomic": { "url": "http://localhost:44380/mcp" }
}
}
```
---
## 6. `atomic-cloud` — Control Plane SaaS
Aplicație **Actix separată** care gestionează hosting managed pe Fly.io.
**Module:**
- `clients/fly.rs` — Fly Machines API (create_app, allocate_ips, machines, volumes, delete_app)
- `clients/stripe.rs` — Checkout sessions + webhook HMAC verify
- `clients/mailgun.rs` — magic link delivery
- `routes/checkout.rs` — POST /api/checkout creează Stripe session + Fly app
- `routes/webhooks.rs` — `customer.subscription.{created,updated,deleted}`
- `routes/instances.rs` — start/stop/restart/billing_portal (auth via management_token)
- `routes/admin.rs` — list instances, stats (MRR estimate), rollout image
- `routes/auth.rs` — magic link send/verify
- `jobs.rs` — **cleanup background**: după 30 zile cancel → `fly.delete_app()` (machines + volumes + IPs)
**Modele Postgres:** `Customer`, `Subscription`, `Instance`, `Event`.
**Env vars necesare:** `DATABASE_URL`, `STRIPE_SECRET_KEY`, `STRIPE_WEBHOOK_SECRET`, `STRIPE_PRICE_ID`, `FLY_API_TOKEN`, `FLY_ORG`, `FLY_REGION`, `BASE_DOMAIN`, `MAILGUN_API_KEY/DOMAIN/FROM`, `ADMIN_API_KEY`, `PUBLIC_URL`.
---
## 7. `src-tauri` — Desktop wrapper
- **Tauri v2** + plugins: `opener`, `dialog`, `shell`, `fs`
- Pornește `atomic-server` ca **sidecar process** (binar pre-built în `src-tauri/binaries/{target_triple}/`)
- Expune o singură comandă IPC: `get_local_server_config()` → `{ url, token }`
- Frontend folosește apoi același `HttpTransport` ca în browser → conectează la sidecar
- Pe exit, Tauri kill-uiește sidecar-ul
**Database location** auto:
- macOS: `~/Library/Application Support/com.atomic.app/`
- Linux: `~/.local/share/com.atomic.app/`
- Windows: `%APPDATA%/com.atomic.app/`
---
## 8. Frontend React (`src/`)
### 8.1 Stack
- **React 18** + TypeScript strict
- **Vite 6** + Tailwind CSS v4 (`@tailwindcss/vite` + `@tailwindcss/typography`)
- **Zustand 5** (state management modular)
- **CodeMirror 6** (`@uiw/react-codemirror`, lang-markdown, theme-one-dark)
- **react-markdown** + remark-gfm
- **Sigma.js v3** + `@sigma/edge-curve` + **Graphology** (canvas graph)
- **d3-force v3** (physics simulation)
- **react-zoom-pan-pinch**, `@tanstack/react-virtual`
- **sonner** (toasts), **diff** (diff view), **qrcode**
### 8.2 Stores Zustand (`src/stores/`)
`atoms`, `tags`, `ui`, `settings`, `wiki`, `chat`, `databases`, `canvas`, `embedding-progress`.
`ui` store: tag filter selectat, drawer state, view mode (canvas/grid/list — persistat în localStorage), search query.
### 8.3 Componente principale (`src/components/`)
- `atoms/` — `AtomCard`, `AtomGrid`, `AtomList`, `AtomViewer` (markdown render + search), `AtomReader`, `RelatedAtoms`, `FilterBar`
- `canvas/` — `CanvasView`, `SigmaCanvas`, `HierarchicalCanvas`, `LocalGraphView`, `MiniGraphPreview`, `ClusterBubble`, `ClusterVisualization`, `ConnectionLines`, `CanvasControls`, `CanvasBreadcrumb`, `useForceSimulation` hooks
- `wiki/` — `WikiArticleContent`, `CitationPopover`, proposal diff view
- `chat/` — `ChatMessage`, conversations list
- `onboarding/` — wizard cu `AIProviderStep`, OpenRouter OAuth callback
- `settings/`, `tags/`, `search/`
### 8.4 Transport abstraction
Interfața `Transport { invoke(name, args), subscribe(event, handler) }`.
Singura implementare: `HttpTransport` (folosită și în desktop, și în browser):
- Mapează nume comandă → HTTP spec (method, path, body/query transform) printr-un **command map**
- WebSocket pentru events
- În Tauri: înainte cheamă `get_local_server_config` via Tauri IPC → primește URL+token sidecar, apoi totul e HTTP
Codul React e **transport-unaware**.
### 8.5 Vite config (`vite.config.ts`)
- Desktop: fără proxy, stub-uri Tauri în `src/lib/stubs/tauri-*.ts`
- Web (`VITE_BUILD_TARGET=web`): proxy `/api`, `/health`, `/ws` → `http://127.0.0.1:8080`
### 8.6 Design system
Dark theme inspirat Obsidian:
- Backgrounds: `#1e1e1e` / `#252525` / `#2d2d2d`
- Accent purple: `#7c3aed`
- Text: white / `#8c8c8c`
- 3-panel layout: tag tree stânga (fix) | main view (canvas/grid/list) | right drawer (editor/viewer/wiki/chat)
---
## 9. iOS App (`ios/`)
- **SwiftUI** + Swift 6 strict concurrency, iOS 17.0+
- **XcodeGen** (`project.yml` → `.xcodeproj` regenerat)
- **Bundle**: `com.atomic.mobile` + extension `com.atomic.mobile.share` (Share Extension)
- **App Group**: `group.com.atomic.mobile` (shared credentials)
- **Dependencies** (SPM): `MarkdownUI 2.4`, `Runestone 0.5`, `TreeSitterLanguages`
**Fișiere cheie:**
- `AtomicApp.swift` — entry point, QR scanner setup
- `APIClient.swift` — `@Observable` HTTP client (Bearer + opțional `X-Atomic-Database`)
- `AtomStore.swift` — state cu DiskCache (atoms, tags) + OfflineQueue
- `Models.swift` — Codable: `Atom`, `AtomSummary`, `SearchResult`, `TagWithCount`, `DatabaseInfo`
- `OfflineQueue.swift` — `Documents/pending_atoms.json`
- `SharedConfig.swift` — UserDefaults suite pentru extensie
- `Theme.swift` — culori match cu desktop
**Endpoint-uri folosite:** `/api/atoms` (CRUD + sources), `/api/tags`, `/api/search`, `/api/databases/{activate}`.
---
## 10. Browser Extension (`extension/`)
- **Manifest V3** (Chrome/Edge/Brave)
- **Permisiuni**: activeTab, scripting, contextMenus, storage, notifications, alarms
**Structură:**
- `background/service-worker.js` — capture flow, queue offline, sync alarm 30s, badge status
- `content/content-script.js` — extragere via **Readability** (Mozilla) + **Turndown** (HTML→Markdown), suportă full page și selection
- `popup/` — toolbar UI cu Capture Page / Selection / Sync Now
- `options/` — server URL + API token + Test Connection
- `lib/config.js` — `chrome.storage.local` cu cheia `serverConfig`
**Endpoint-uri:** `POST /api/atoms`, `GET /health`, `GET /api/atoms?limit=1` (test).
**Offline queue** în `chrome.storage.local`, drain la fiecare alarm tick (30s).
---
## 11. Plugins externe
### 11.1 Discord bot (`plugins/discord/`)
- **discord.js 14.16** + **better-sqlite3** + **yaml**
- Slash commands: `/atomic-subscribe`, `/atomic-unsubscribe`, `/atomic-config`, `/atomic-save`, `/atomic-search`, `/atomic-status`
- **Reaction-based capture**: emoji custom (sau fallback Unicode) pe orice mesaj → ingestie
- **Settle window** (default 300s): debounce per channel pentru thread-uri active (resetat la fiecare nou mesaj)
- Suportă **Forum channels** + **Voice/Stage** + **Threads** (cu fetch full history)
- **NormalizedMessage** structure unifică text/forum/voice/dm
- Local SQLite tables: `dedup_index` (guild:channel:message → atom_id), `channel_configs`
- Templates de formatting per tip (single message / thread / forum post)
- Tag resolution cu cache + auto-create pentru taguri ierarhice (`discord/<channel>`)
### 11.2 Obsidian plugin (`plugins/obsidian-plugin/`)
- **TypeScript** + Obsidian API (`requestUrl`)
- Comenzi: `semantic-search`, `sync-current-note`, `sync-vault`, `toggle-auto-sync`, `open-similar-notes`, `open-wiki`
- 2 sidebar views: **SimilarView** (related notes), **WikiView** (browse wiki articles)
- **Sync engine**:
- File watcher cu debounce configurabil (default 2s)
- Hash SHA-256 pe content → skip dacă neschimbat
- Source URL: `obsidian://VaultName/path/to/note.md`
- Handle: create/modify/delete/rename
- State în `plugin-data.json`: `Map<path, {atomId, contentHash, lastSynced}>`
- Settings: server URL, token, vault name, auto-sync, debounce, folder→tags, delete-on-remove, exclude patterns
---
## 12. Scripturi Node (`scripts/`)
| Script | Rol |
|---|---|
| `dev-server.js` | Pornește atomic-server + Vite simultan; opțional `--postgres` (docker pgvector) |
| `build-server.js` | Compile `atomic-server` → `src-tauri/binaries/{target}/` |
| `build-mcp-bridge.js` | Compile `mcp-bridge` → același folder pentru sidecar Tauri |
| `build-release.js` | Bump versiune + git tag + tauri build cross-platform + GitHub upload |
| `import/obsidian.js` | Import vault Obsidian (folders→tags, YAML frontmatter, dedup, dry-run) |
| `import-rss.js` | Import feed RSS (turndown HTML→MD) |
| `import-wikipedia.js` | Crawl Wikipedia (BFS din 3 domenii: Computing/Philosophy/History), 100ms rate limit |
| `stress-test-wikipedia.js` | Volume test 1000+ articole |
| `stress-test-summaries.js` | Lightweight: Wikipedia summary endpoint, concurrency 10 |
| `reset-database.js` | Drop completă a DB |
| `reset-tags.js` | Reset tags + remark atoms pentru re-tagging |
| `reset-chunks.js` | Șterge chunks/embeddings/edges/positions, păstrează atoms |
| `drop-database.js` | Delete persistent (registry + databases) |
| `open-in-db-browser.sh` | Quick sqlite3 inspection |
---
## 13. Docker & Deployment
### 13.1 Dockerfile multi-stage
1. **planner** — `cargo-chef prepare` (cache deps)
2. **rust-builder** — mold linker, `cargo chef cook`, build `atomic-server` cu profile `server`
3. **frontend-builder** — `npm ci` + `vite build` web
4. **server** target — `debian:bookworm-slim`, EXPOSE 8080, ENTRYPOINT atomic-server
5. **web** target — `nginx:1.28`, copy dist-web
6. **all-in-one** target — supervisord rulează atomic-server (127.0.0.1:8080) + nginx (8081), VOLUME /data — folosit pentru Fly.io single-machine
### 13.2 docker-compose.yml (production)
Servicii:
- `server` (ghcr.io/kenforthewin/atomic-server)
- `web` (ghcr.io/kenforthewin/atomic-web)
- `proxy` (nginx :8080→80) cu config în `docker/nginx.conf`
- `litestream` (profile backup) — replicare DB → S3 (sync 10s, snapshot 1h)
- volume `atomic-data:/data`
### 13.3 nginx config
- `/api/` → server (proxy_buffering off pentru SSE, read_timeout 300s)
- `/ws` → server (Upgrade headers, 86400s timeout)
- `/.well-known/`, `/oauth/`, `/mcp` → server
- `/` → web frontend
- `/assets/` → cache 1y immutable
- SPA fallback `try_files $uri /index.html`
### 13.4 docker-compose.dev.yml
- `pgvector/pgvector:pg16` pe portul 5434, healthcheck `pg_isready`
- Folosit de `npm run dev:server:pg` și `npm run db:reset:pg`
### 13.5 Fly.io (`fly.toml.example`)
- Target build: `all-in-one`
- Mount volume `atomic_data → /data`
- Internal port 8081 (nginx)
- Auto-stop suspend, min 0 machines
- Health check `/health` 30s
- VM `shared-cpu-1x` 512MB
---
## 14. Documentația din `docs/`
| Fișier | Subiect |
|---|---|
| `foreign-keys.md` | Plan de a activa `PRAGMA foreign_keys` (acum off). 6 probleme + 6 pași: virtual table cleanup, tranzacții, validare tag IDs, stale positions, wiki migration on tag merge. |
| `llm-wiki-gist-analysis.md` | Comparație cu gist-ul Karpathy "LLM Wiki". Top idee nouă: **Lint pass** (flag contradictions, orphan pages) — distinctive feature. |
| `plan-async-migration.md` | Plan în 6 pași pentru a face `AtomicCore` async-native (eliminarea sync→async bridge la Postgres prin `PG_RUNTIME.block_on`). |
| `url-ingestion-improvements.md` | Roadmap inspirat din Obsidian Web Clipper / Defuddle: wire metadata în columns, auto published_at, site-specific extractors (AI chats prioritate), MathML/footnotes/callouts. |
| `wiki-proposal-loop-plan.md` | Spec arhitectural M1+M2+M3 pentru **wiki updates ca propuneri review-abile** în loc de mutații directe. Tabela `wiki_proposals`, `WikiSectionOp`, dirty set, quiet window, supersede budget, daily caps. |
---
## 15. Tech stack — TABEL FINAL
| Layer | Tehnologii |
|---|---|
| **Core (Rust)** | rusqlite 0.32 (bundled) + sqlite-vec 0.1.6, sqlx 0.8 + pgvector 0.4, tokio 1, reqwest 0.12, tiktoken-rs 0.6, pulldown-cmark 0.12, dom_smoothie 0.15, feed-rs 2.3, sha2, uuid, chrono, tracing |
| **Server** | actix-web 4.9, actix-cors, actix-ws, rmcp 0.15 (MCP Streamable HTTP), utoipa 5 + scalar, clap 4 |
| **Desktop** | Tauri v2 + plugins (opener, dialog, shell, fs) |
| **Frontend** | React 18, TypeScript strict, Vite 6, Tailwind v4, Zustand 5, CodeMirror 6, react-markdown + remark-gfm, Sigma.js 3 + Graphology, d3-force 3, react-zoom-pan-pinch, @tanstack/react-virtual, sonner, diff, qrcode |
| **iOS** | SwiftUI, Swift 6, iOS 17+, MarkdownUI, Runestone, TreeSitterLanguages, XcodeGen |
| **Extension** | Manifest V3, Mozilla Readability, Turndown |
| **Discord** | discord.js 14.16, better-sqlite3, yaml |
| **Obsidian** | Obsidian API, Web Crypto SHA-256 |
| **Cloud (SaaS)** | actix-web, sqlx postgres, Stripe API, Fly Machines API, Mailgun, hmac+sha2 (webhook verify) |
| **Containere** | Docker multi-stage (cargo-chef + mold), nginx, supervisord, litestream backup → S3, pgvector/pgvector:pg16 dev |
| **Deploy** | Fly.io single-machine (volume), Docker Compose self-host, GHCR images |
| **AI providers** | OpenRouter (cloud, OAuth), Ollama (local, auto-discover), OpenAI-compatible (Azure/Groq/Together) |
| **MCP** | Streamable HTTP server + stdio bridge pentru Claude Desktop, OAuth 2.0 + PKCE + DCR pentru Claude.ai remote |
---
## 16. Integrări externe (efective)
| Integrare | Cum |
|---|---|
| **OpenRouter** | OAuth callback `public/oauth/openrouter-callback.html`, modele separate per capability, settings persistate |
| **Ollama** | Auto-discovery via `/api/ollama/models` |
| **OpenAI-compatible** | Base URL + API key generic |
| **Stripe** | Cloud only — checkout sessions + webhook signed (HMAC-SHA256) |
| **Fly.io Machines API** | Cloud only — provision/start/stop/destroy machines + volumes + IPs |
| **Mailgun** | Cloud only — magic link auth |
| **Claude Desktop / Claude.ai** | MCP `/mcp` (Streamable HTTP) + `mcp-bridge` (stdio) + OAuth 2.0 PKCE pentru remote |
| **GitHub Releases** | Auto-upload via `build-release.js` |
| **Wikipedia REST API** | Doar import scripts |
| **Obsidian** | Plugin oficial + import script CLI |
| **Discord API** | discord.js gateway + REST |
| **S3 / R2** | Litestream backup opțional |
---
## 17. Tehnical debt & roadmap (din docs)
1. **FK constraints OFF** → plan în `foreign-keys.md` (cleanup virtual tables, tranzacții, validare)
2. **AtomicCore sync→async bridge la Postgres** → plan în `plan-async-migration.md` (6 pași)
3. **Wiki proposals M1/M2/M3** → spec în `wiki-proposal-loop-plan.md` (M1 manual ready, M2 background scheduler planificat)
4. **URL ingestion** → metadata wiring, published_at extraction, site-specific extractors (AI chat transcripts prioritate)
5. **Lint pass** (contradicții/orfani) → idee nouă din analiza gist Karpathy
---
## 18. CONCLUZIE
**Atomic** este o arhitectură excepțional curat designed:
- **Single source of truth** în `atomic-core` (16k+ linii Rust, zero framework deps)
- **Storage abstraction completă** SQLite ↔ Postgres
- **AI provider abstraction** plug-and-play (3 implementări trait-based)
- **Wrappers thin** pentru fiecare transport: Tauri (sidecar IPC), actix-web (REST/WS/MCP), mcp-bridge (stdio)
- **Frontend transport-unaware** (același cod în desktop și browser)
- **Pipeline async fire-and-forget** cu callback eventing → broadcast → WebSocket
- **Multi-database** cu registry separat
- **API token + OAuth 2.0 PKCE + DCR** pentru securitate enterprise
- **6 platforme client**: desktop, web, iOS, Chrome ext, Discord, Obsidian
- **Deployment flexibil**: standalone binary, Docker Compose, Fly.io single-machine, SaaS managed (atomic-cloud)
- **Documentație tehnică matură** (5 spec docs cu planuri executabile)
Este un proiect **production-ready** cu o roadmap clară de tehnical debt și features noi.

398
ai_platform/modules/didi_brain/CONTRACT_VERIFICATION_CACHE.md Normal file
View file

@ -0,0 +1,398 @@
# Verification Cache — Contract didi-brain ↔ didi-backend
Status: **IMPLEMENTAT v2** și live la `http://10.11.10.13:8090`. Backend-side trebuie să
implementeze apelurile descrise mai jos.
**Schimbare majoră față de v1** (2026-04-23): cheia de cache NU mai include
`evidence_hash`. Cheia e acum `(claim_hash, tier)`. URL-urile evidence rămân
stocate ca metadata și sunt returnate în response ca `verification_evidence_urls`
— backend decide overlap cu evidence-ul curent.
**De ce**: URL-urile pe care le avea backend-ul la WRITE-time rareori coincid
cu URL-urile pe care brain le întoarce la READ-time (brain face semantic
search în corpus-ul propriu, backend avea URL-uri din M17/web sau din altă
rulare). Cu evidence_hash în cheie, cache-ul era practic nereachable.
---
## Principiu
- **Backend deține prompt-ul + modelul + logica de status/thresholds.**
- Brain stochează rezultatul LLM ca **blob opac** și îl returnează 1:1.
- Zero sync de prompt între backend și brain. Zero duplicare de lucru LLM.
---
## Ciclul complet per claim
```
┌─ gather (cu verification cache check) ─────────────────────────────┐
│ │
│ backend → POST http://10.11.10.13:8090/v1/gather │
│ body: { claim, max_evidence, include_verification: true, │
│ tier, prompt_hash, framework_version } │
│ │
│ brain → returnează evidence + brain_meta cu: │
│ ├─ verification_staleness: fresh | stale_framework | │
│ │ stale_prompt | miss │
│ └─ verification: { ... } sau null │
│ │
│ backend decide: │
│ fresh → folosește verification DIRECT la UI │
│ stale_framework → ia raw, RECOMPUTĂ status local (zero LLM) │
│ stale_prompt → LLM call + POST /v1/verification_cache │
│ miss → LLM call + POST /v1/verification_cache │
│ │
└────────────────────────────────────────────────────────────────────┘
```
---
## Endpoints
### 1. POST /v1/verification_cache (WRITE)
**URL:** `http://10.11.10.13:8090/v1/verification_cache`
**Request body:**
```json
{
"claim": "Romania a câștigat 9 medalii olimpice la Paris 2024.",
"evidence_urls": [
"https://adevarul.ro/articol1",
"https://somes-tisa.rowater.ro/pdf2"
],
"tier": "free",
"model": "qwen35:Qwen3.5-397B-A17B",
"prompt_hash": "a3f2b1c9d4e7",
"framework_version": "f9e1d2c3b4a5",
"schema_name": "didi-v1",
"verification_raw": {
"sources_analysis": [
{"url": "...", "stance": "SUPPORTS", "reliability": "news", "relevant_quote": "..."}
],
"agreement_score": 60,
"confidence": 75,
"status": "LT",
"reasoning": "..."
},
"verification_processed": {
"id": "claim_1",
"text": "...",
"status": "LT",
"status_name_ro": "Probabil Adevărat",
"confidence": 75,
"agreement_score": 100,
"sources": [...],
"reasoning": "..."
}
}
```
**Required:** `claim`, `evidence_urls` (non-empty), `tier`, `prompt_hash` (≥4 chars),
`verification_processed`.
**Optional:** `model`, `framework_version`, `schema_name` (default `"didi-v1"`),
`verification_raw`.
**Response 200:**
```json
{
"cached": true,
"claim_hash": "9b769fa8caba883...",
"evidence_hash": "d04318eec4bfc01...",
"tier": "free",
"created_at": "2026-04-23T11:01:22.607213Z",
"updated_at": "2026-04-23T11:01:22.607213Z",
"expires_at": "2026-05-23T11:01:22.598217Z"
}
```
**Errors:**
- `413 Payload Too Large` — body > 64 KB
- `422 Unprocessable Entity` — validare pydantic (tier != free/premium, prompt_hash < 4 chars, etc.)
- `503 Service Unavailable` — PG indisponibil, retry mai târziu
- `500 Internal Server Error` — orice alt eșec
**Idempotență:** UPSERT pe cheia `(claim_hash, evidence_hash, tier)`. Same-tuple,
second write → update `verification_processed`/`verification_raw`/`updated_at`/`expires_at`.
Last-writer-wins.
---
### 2. POST /v1/gather (READ extins)
**URL:** `http://10.11.10.13:8090/v1/gather`
**Request body:** exact ca azi, plus 4 câmpuri noi (toate optional; default behavior nu se schimbă):
```json
{
"claim": "...",
"max_evidence": 5,
"run_nli": false,
"include_verification": true, // NEW — opt-in, default false
"tier": "free", // REQUIRED când include_verification=true
"prompt_hash": "a3f2b1c9d4e7", // optional, dar recomandat
"framework_version": "f9e1d2c3b4a5" // optional
}
```
**Response body:** exact ca azi, plus `brain_meta` îmbogățit:
```json
{
"claim": "...",
"evidence": [ ... ], // URL-uri din corpus-ul brain — pot diferi
"brain_meta": { // de cele din verification_evidence_urls
"cache_status": "HIT",
"evidence_sources": 5,
"verification_staleness": "fresh",
"verification": { ... payload stocat ... },
"verification_model": "qwen35:Qwen3.5-397B-A17B",
"verification_tier": "free",
"verification_prompt_hash": "a3f2b1c9d4e7",
"verification_framework_version": "f9e1d2c3b4a5",
"verification_cached_at": "2026-04-23T11:01:22.607213Z",
"verification_expires_at": "2026-05-23T11:01:22.598217Z",
"verification_evidence_urls": [ // URL-urile pe care a fost rulat LLM-ul
"https://adevarul.ro/articol1", // când a fost scris cache-ul.
"https://somes-tisa.rowater.ro/pdf2" // Backend poate compara cu
], // evidence[].url și decide dacă cache-ul
"verification_evidence_hash": "d04318eec4bfc01..." // mai e relevant.
}
}
```
**Important: verificare overlap evidence (backend-side)**
URL-urile din `evidence[]` (ce returnează gather din corpus) pot diferi de
`brain_meta.verification_evidence_urls` (ce a primit LLM-ul la WRITE). Backend
decide ce face:
- **Overlap >= 60%** (recomandat): folosește `verification` direct; `sources`
din verification acoperă cele mai multe evidence din `evidence[]`.
- **Overlap mic/zero**: corpusul brain s-a schimbat; poți trata ca `miss` și
să rulezi LLM din nou. Brain returnează staleness=fresh din perspectiva
prompt/framework, dar tu evaluezi semantic dacă sursele mai sunt relevante.
Exemplu TypeScript:
```typescript
function evidenceOverlap(a: string[], b: string[]): number {
const norm = (u: string) => u.trim().toLowerCase().replace(/\/$/, '');
const setA = new Set(a.map(norm));
const setB = new Set(b.map(norm));
const inter = [...setA].filter(x => setB.has(x)).length;
return inter / Math.max(setA.size, setB.size);
}
const gatherEvidence = data.evidence.map(e => e.url);
const cachedFor = data.brain_meta.verification_evidence_urls || [];
if (evidenceOverlap(gatherEvidence, cachedFor) >= 0.6) {
// use cached verification
} else {
// treat as miss despite staleness=fresh
}
```
---
## Semantica `verification_staleness`
| Staleness | Când apare | `verification` conține | Ce face backend |
|---|---|---|---|
| `fresh` | prompt + framework identice cu cache | `verification_processed` | folosește direct la UI |
| `stale_framework` | prompt identic, framework diferit | `verification_raw` | recomputează status local din raw (zero LLM) |
| `stale_prompt` | prompt diferit | `null` | LLM call + POST cache |
| `miss` | nu există entry sau a expirat | `null` | LLM call + POST cache |
---
## Reguli de hashing (toate calculate server-side de brain)
**Backend trimite plain `claim` și `evidence_urls`.** Brain normalizează și hash-uiește.
Backend nu trebuie să computeze nimic.
### Normalizare `claim`
```python
def normalize_claim(s):
s = unicodedata.normalize("NFKD", s) # decompunere diacritice
s = "".join(c for c in s if not unicodedata.combining(c)) # strip diacritice
s = s.lower() # lowercase
s = " ".join(s.split()) # collapse whitespace
s = s.rstrip(".?!") # trim trailing punctuation
return s
```
**Aceleași hash:**
- `"România a câștigat 9 medalii."`
- `"romania a castigat 9 medalii"`
- `"Romania a câștigat 9 medalii!"`
**Hash-uri diferite (corect):**
- `"Vaccinul e sigur"` vs `"Vaccinul nu e sigur"` (negație contează)
- `"X are 10 mașini"` vs `"X are 11 mașini"` (numere contează)
### Normalizare `evidence_urls`
```python
def hash_evidence_urls(urls):
canonical = sorted({u.strip().lower().rstrip("/") for u in urls if u})
return sha256("\n".join(canonical)).hexdigest()
```
- Ordinea URL-urilor nu contează (sortate).
- Case nu contează (`HTTPS://X.COM` == `https://x.com`).
- Trailing `/` nu contează.
- Duplicate sunt eliminate.
---
## Isolation pe tier
Cache pe `free` și `premium` sunt **complet separate**. Un user `free` nu vede
ce a scris un user `premium` (și invers). Este intenționat: modele diferite =
nuanțe diferite la stance.
Cheia unică în DB: `(claim_hash, evidence_hash, tier)`.
---
## TTL
Default **30 zile** per entry. Reîmprospătat la fiecare UPSERT. După expirare,
rândul există încă în DB dar e invizibil la lookup (filtrat cu `expires_at > now()`).
Pruning ulterior — nu e critic pentru corectitudine.
---
## Cum calculezi `prompt_hash` și `framework_version` backend-side
Recomandarea ta, pe care o respect în brain:
```javascript
// prompt_hash — schimbă la orice editare a prompt-ului (auto-invalidation)
const promptData = await redis.get('didi:config:claims:v1:prompts:verification');
const { system, user_template } = JSON.parse(promptData);
const promptHash = sha256(system + '|' + user_template).slice(0, 12);
// framework_version — hash peste thresholds + scoring config
const frameworkData = await redis.get('didi:framework:claims');
const scoringData = await redis.get('didi:config:claims:v1:scoring_config');
const frameworkVersion = sha256(frameworkData + '|' + scoringData).slice(0, 12);
```
Calculat în `loadConfigs()` și cached în memorie până la următorul reload.
---
## Exemplu complet flow backend-side
```typescript
// În searchEvidence / verifySingleClaim:
async function verifyClaim(claim, evidence, tier) {
const evidenceUrls = evidence.map(e => e.url);
// 1. Check brain cache via gather extension
const brainResp = await fetch(`${DIDI_BRAIN_URL}/v1/gather`, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({
claim,
max_evidence: evidence.length,
run_nli: false,
include_verification: true,
tier,
prompt_hash: await getPromptHash(),
framework_version: await getFrameworkVersion(),
}),
signal: AbortSignal.timeout(5000),
});
const data = await brainResp.json();
const meta = data.brain_meta || {};
if (meta.verification_staleness === 'fresh') {
// Direct la UI — zero LLM, zero overhead
return meta.verification;
}
if (meta.verification_staleness === 'stale_framework') {
// Backend recomputează din raw (thresholds noi)
const raw = meta.verification; // verification_raw
return {
...buildProcessedFromRaw(raw), // aplică calculateStatusFromSources
reasoning: raw.reasoning,
};
}
// 2. miss sau stale_prompt — LLM call propriu
const llmResponse = await callVerificationLLM(claim, evidence, tier);
const processed = buildProcessedFromRaw(llmResponse);
// 3. Fire-and-forget push la brain
fetch(`${DIDI_BRAIN_URL}/v1/verification_cache`, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({
claim,
evidence_urls: evidenceUrls,
tier,
model: modelUsed,
prompt_hash: await getPromptHash(),
framework_version: await getFrameworkVersion(),
verification_raw: llmResponse,
verification_processed: processed,
}),
signal: AbortSignal.timeout(3000),
}).catch(() => {}); // silent failure
return processed;
}
```
---
## Smoke tests (poți rula oricând)
```bash
# Write test
curl -sf -X POST http://10.11.10.13:8090/v1/verification_cache \
-H 'Content-Type: application/json' \
-d '{
"claim": "test claim",
"evidence_urls": ["https://example.com/a"],
"tier": "free",
"prompt_hash": "test_abc_12",
"verification_processed": {"status": "LT", "confidence": 80}
}' | jq
# Read test (should be fresh)
curl -sf -X POST http://10.11.10.13:8090/v1/gather \
-H 'Content-Type: application/json' \
-d '{
"claim": "test claim",
"include_verification": true,
"tier": "free",
"prompt_hash": "test_abc_12"
}' | jq '.brain_meta.verification_staleness, .brain_meta.verification'
```
---
## Config brain-side
Azi:
- `VERIFICATION_CACHE_TTL_DAYS=30`
- `VERIFICATION_CACHE_MAX_PAYLOAD_KB=64`
Ajustabil via `.env` dacă e nevoie.

306
ai_platform/modules/didi_brain/INDEX.md Normal file
View file

@ -0,0 +1,306 @@
# DidiBrain - Index
Knowledge atom storage + claim verification cache + analysis atom cache pentru platforma DIDI. FastAPI service care vorbeste contractul `web-gathering` 1:1 (drop-in replacement / cache layer pentru `/v1/gather`) si in plus expune doua cache-uri proprii: `brain_verification_cache` (claims, din 2026-04-23) si `brain_analysis_atom` (techniques + ai_tampered + claims, NEW v2 din 2026-05-01).
**Stack**: Python 3.11+, FastAPI, Pydantic v2, asyncpg, BGE-M3 1024-dim embeddings (vLLM), BGE-reranker-v2-m3 (vLLM cross-encoder), Qwen3.5-397B-A17B (llama.cpp via OpenAI-compat router)
**Storage**: Atomic (Rust knowledge graph) on Postgres 16 + pgvector. Brain owns its own tables prefixed `brain_*` alongside Atomic's tables.
**URL**: `http://10.11.10.12:8090` (production-local, since brain cutover 2026-05-01 — anterior pe `10.11.10.13:8090`)
**Container**: `didibrain-api` (alongside `didibrain-atomic` and `didibrain-postgres`)
---
## Ce face
DidiBrain sta intre Didi backend (agent-v3) si modulele live de web-gathering. Vorbeste acelasi contract HTTP pentru `/v1/gather`, raspunzand din knowledge atoms pre-ingerate + semantic search + cross-encoder reranking + NLI stance vs query. Tipic 1.5-6s in loc de 20+s pentru un crawl live. Plus detine 2 cache-uri proprii pentru rezultate LLM ale didi-backend: `brain_verification_cache` (per-claim verification, since 2026-04-23) si `brain_analysis_atom` (rezultate full-component pe techniques + ai_tampered + claims, since 2026-05-01 v2).
---
## API endpoints
| Method | Path | Ce face |
|---|---|---|
| GET | `/health` | Liveness probe |
| POST | `/v1/search` | Flat list of doc-level search results |
| POST | `/v1/fetch` | Look up atoms by URL → return extracted text |
| POST | `/v1/gather` | Full claim → ranked evidence pipeline (with optional verification cache lookup) |
| POST | `/v1/image-search` | Stub, always returns empty list |
| POST | `/v1/ingest` | Populate brain from web-module output |
| POST | `/v1/verification_cache` | Write claim verification result (by didi-backend after LLM run) |
| POST | `/v1/analysis_atom/lookup` | NEW v2 — find cached analysis result by `(content_hash, component, prompt_hash)`. Tier-agnostic. Bronze never served. |
| POST | `/v1/analysis_atom` | NEW v2 — write atom (rejects `tier=free`). Auto cache_tier silver/bronze din `llm_confidence` (>=60 silver). Gold preservation in upsert. |
| PATCH | `/v1/analysis_atom/{atom_id}` | NEW v2 — promote to gold (after moderator review). `human_validated=true`, `expires_at=NULL`. |
| GET | `/v1/analysis_atom/stats` | NEW v2 — counts per tier/component, hit rate 24h |
Plus FastAPI auto: `/docs` (Swagger UI), `/redoc`, `/openapi.json`.
---
## Structura fisiere
```
brain_api/
app.py # FastAPI app + lifespan + 11 endpoints
db.py # asyncpg pool + _SCHEMA_SQL (creates brain_verification_cache + brain_analysis_atom tables idempotent)
schemas.py # Pydantic v2 models (request/response for all endpoints)
deps.py # AppState (atomic, embed, llm, resolver) singleton
run.py # CLI entry (uvicorn launcher)
Dockerfile
requirements.txt
prompts/ # LLM prompts for verification + extraction (nli_v1.md etc.)
services/
fetch.py # /v1/fetch
gather.py # /v1/gather (most complex — semantic search + reranking + NLI + verification cache integration)
ingest.py # /v1/ingest
search.py # /v1/search
verification_cache.py # claims cache (since 2026-04-23 v2 schema)
analysis_atom.py # NEW 2026-05-01 — techniques+ai_tampered+claims cache, 3 tiers (gold/silver/bronze)
nli.py # NLI stance classification
mapping.py # taxonomy / tag mapping helpers (atom → EvidenceItem / FetchedPage)
shared/ # (one level up, used by brain_api)
atomic_api.py # AtomicClient (typed REST client for atomic-server)
embedding_client.py # BGE-M3 embed + reranker
llm_client.py # async OpenAI-compat wrapper
config.py # Pydantic Settings, LlmRole, model routing
taxonomy.py # canonical TAXONOMY tree + TagResolver
logging.py # structlog setup
extractor/ # claim extraction layer (separate package)
extract.py # single-doc extraction + substring quote validation
push.py # claim atom creation
batch.py # batch orchestrator with state file
prompts/
scripts/ # operator CLI tools (numbered 01-11 + bootstrap_deploy.sh)
infra/ # docker-compose.yml + deploy
lint/ # custom lint pass (cross-corpus contradiction detection)
reports/ # generated audit reports
```
---
## Brain-owned PG schema
Doua tabele alaturi de cele ale Atomic, in acelasi Postgres (prefix `brain_*` pentru izolare).
### brain_verification_cache (since 2026-04-23 v2)
Cache pentru verificare claims (one row per `(claim_hash, tier)`, last-wins UPSERT). `evidence_urls` + `evidence_hash` stocate ca METADATA (NU in unique key) pentru ca backend sa poata calcula overlap. TTL 30 zile default.
| Coloana | Tip | Scop |
|---|---|---|
| `id` | uuid PK | `gen_random_uuid()` |
| `claim_hash` | text NOT NULL | `sha256(normalized claim)` |
| `tier` | text CHECK (`free`\|`premium`) | |
| `evidence_hash` | text NOT NULL | `sha256(canonical urls)`, metadata only |
| `evidence_urls` | jsonb NOT NULL | metadata only |
| `model` | text NULL | LLM used |
| `prompt_hash` | text NOT NULL | `sha256(system+user_template)[:12]` |
| `framework_version` | text NULL | sha256 of relevant Redis configs |
| `schema_name` | text default `'didi-v1'` | |
| `verification_raw` | jsonb NULL | raw LLM output |
| `verification_processed` | jsonb NOT NULL | mapped to canonical |
| `created_at` | timestamptz default `now()` | |
| `updated_at` | timestamptz default `now()` | |
| `expires_at` | timestamptz NOT NULL | TTL 30d default |
UNIQUE: `(claim_hash, tier)`. Indexes: `idx_bvc_lookup`, `idx_bvc_expires`, `idx_bvc_prompt`.
**Migration v1 → v2**: vechiul UNIQUE `(claim_hash, evidence_hash, tier)` facea cache-ul practic nereachable (URL-urile la read difera de cele la write). v2 muta `evidence_hash` afara din unique key. Migrarea e idempotenta (rulata la fiecare connect).
### brain_analysis_atom (NEW 2026-05-01)
Cache pentru rezultate LLM full-component (techniques, ai_tampered, claims-aggregator). 3-tier system gold/silver/bronze.
| Coloana | Tip | Scop |
|---|---|---|
| `atom_id` | bigserial PK | |
| `content_hash` | text NOT NULL | `sha256(normalized text)` |
| `content_preview` | text NULL | first 200 chars for debug |
| `component` | text CHECK (`techniques`\|`ai_tampered`\|`claims`) | |
| `tier` | text CHECK (`free`\|`premium`) | informational only on row |
| `prompt_hash` | text NOT NULL | |
| `framework_version` | text NULL | |
| `model_used` | text NULL | |
| `result_processed` | jsonb NOT NULL | canonical mapped output |
| `result_raw` | jsonb NULL | |
| `llm_confidence` | numeric NULL | 0-100, drives silver vs bronze |
| `cache_tier` | text CHECK (`gold`\|`silver`\|`bronze`) default `'silver'` | |
| `human_validated` | bool default false | true after moderator review |
| `human_corrections` | jsonb NULL | diff applied by moderator |
| `validator_user_id` | text NULL | keycloak_id |
| `validated_at` | timestamptz NULL | |
| `hit_count` | integer default 0 | incremented on lookup hit |
| `last_hit_at` | timestamptz NULL | |
| `created_at`, `updated_at` | timestamptz | |
| `expires_at` | timestamptz NULL | NULL=never (gold). Silver=90d, Bronze=30d |
UNIQUE: `(content_hash, component, prompt_hash)` — `tier` OUT of unique key (write only `premium`, read tier-agnostic). Indexes: `idx_baa_lookup`, `idx_baa_gold` (partial WHERE `cache_tier='gold'`), `idx_baa_expires`, `idx_baa_prompt`.
**Cache tier rules**:
- **gold**: `human_validated=true`. Set by didi-backend moderation flow (PATCH endpoint). Survives prompt change. Forever (`expires_at=NULL`).
- **silver**: LLM result, `llm_confidence >= 60`. Default fresh write. TTL 90d.
- **bronze**: LLM result, `llm_confidence < 60`. Stored for audit but **NEVER served on lookup**. TTL 30d.
**Gold preservation**: ON CONFLICT upsert preserves gold (won't downgrade to silver if a fresh write comes in — `result_processed`, `cache_tier`, `llm_confidence`, `expires_at` all CASE-WHEN protected).
**Lookup logic** (`services/analysis_atom.py::lookup`): SELECT WHERE `content_hash=$1 AND component=$2 AND cache_tier IN ('gold','silver') AND (expires_at IS NULL OR expires_at > now())` ORDER BY `(cache_tier='gold') DESC, updated_at DESC` LIMIT 1. Apoi `decide_freshness()` pe baza `prompt_hash` + `framework_version`. Gold = always `fresh`.
---
## How didi-backend (agent-v3) uses brain
3 puncte de integrare:
1. **Claims verification cache** — claims executor cheama `gatherFromBrain()` cu `include_verification=true`. Brain returneaza evidence + cached verification (if any) + staleness. Backend decide fresh / stale_framework / stale_prompt / miss path. Backend scrie via fire-and-forget POST `/v1/verification_cache`. Detalii contract: `CONTRACT_VERIFICATION_CACHE.md`. Helper centralizat: `agent-v3/src/shared/brain/client.ts`.
2. **Analysis atom cache** (NEW 2026-05-01) — techniques si ai_tampered executors cheama `lookupAnalysisAtom()` **inainte** de LLM. Pe hit (gold sau silver+fresh), skip LLM, return cached. Dupa LLM run, fire-and-forget `writeAnalysisAtomAsync()` (only `tier=premium`). Dupa ce moderator rezolva sesiunea cu corectii, `patchAnalysisAtomGold()` promoveaza silver → gold.
3. **Gather pipeline** — agent-v3 inca foloseste `/v1/gather` pentru live web-augmented retrieval (pre-ingerat + semantic search + reranker + NLI). Branch-ul pe `brain_meta.cache_status` = `HIT|PARTIAL|MISS` decide fallback la web module.
---
## Configuration
- Settings via `shared/config.py` cu `pydantic-settings`
- Env vars: `postgres_dsn`, `atomic_url`, `llm_router_url`, `embedding_url`, `reranker_url`, `verification_cache_ttl_days` (default 30), `verification_cache_max_payload_kb` (default 64)
- Internal Atomic URL: `http://didibrain-atomic:8080` (Docker DNS)
- Upstream LLM/embed/rerank pe VPN 10.11.10.x (10.11.10.17:14011 router, 10.11.10.15:8200 BGE-M3, 10.11.10.15:8100 reranker)
- **No auth in v1** (binds to internal Docker network)
---
## Deployment
- Docker compose la `infra/docker-compose.yml`
- 3 containere: `didibrain-api` (FastAPI), `didibrain-atomic` (Rust KG), `didibrain-postgres` (PG 16 + pgvector)
- Schema bootstrap on startup (idempotent — `_SCHEMA_SQL` ruleaza la fiecare connect, all DDL is `IF NOT EXISTS` + ALTER guards)
- TTL cleanup: cron job in `scripts/` sterge expired silver/bronze atoms
- Resource footprint: ~200 MB RAM total cross 3 containere; brain-api idle ~6% CPU, spike ~20% during `/v1/gather` cu NLI
**Rebuild**:
```bash
cd /home/admin365/didi_mono/ai_platform/modules/didi_brain
docker compose -f infra/docker-compose.yml --env-file .env up -d --build
```
---
## Bootstrap & lint
- `scripts/01_sanity_full.py` — upstream stack validator (LLM + embed + rerank, 8 checks)
- `scripts/02_bootstrap_atomic.py` — claim atomic instance, configure BGE-M3 provider
- `scripts/03_sanity_atomic.py` — brain end-to-end (create → embed → search → cleanup)
- `scripts/04_seed_taxonomy.py` — seed canonical tag taxonomy (79 tags, 7 namespaces)
- `scripts/05_import_wikipedia_seed.py` — seed-list Wikipedia import
- `scripts/06_validate_queries.py` — smoke test doc-level retrieval
- `scripts/07_run_extraction.py` — claim extraction batch (Qwen 397B)
- `scripts/08_validate_claims.py` — smoke test claim-level retrieval
- `scripts/09_brain_api_demo.py` — brain_api contract test (all endpoints)
- `scripts/10_run_lint.py` — Lint pass runner (contradiction detection)
- `scripts/11_show_contradictions.py` — render top contradictions
- `scripts/bootstrap_deploy.sh` — fresh-server deploy orchestrator
- `lint/` — custom lint pass (cross-corpus contradiction detection via NLI)
---
## Ce NU face
- No image storage (image-search endpoint always returns empty)
- No auth in v1 (intern only, binds to Docker internal network)
- No multi-tenant (single brain instance, multiple consumers)
- No long-term archive — atoms expire on TTL (gold = forever, silver = 90d, bronze = 30d)
- No live web crawling (delegata la modulele M17 / web — brain serveste din corpus pre-ingest)
---
## Related docs (in this module)
- `README.md` — project landing, quick start, architecture diagram
- `STATUS.md` — end-of-session snapshot + troubleshooting + session log
- `CONTRACT_VERIFICATION_CACHE.md` — contract backend ↔ brain pentru verification cache (v2)
- `CHANGES_2026-04-23.md` — session changelog cu integrare web-api + verification cache v2
- `ARCHITECTURE.md` — diagrame + pipeline detail
- `AUDIT.md` — initial upstream Atomic audit
---
## Cache Freshness Defense (2026-05-04 → 2026-05-05) — 11/11 piloni live
Whole-system response to "what happens if a cached verdict goes stale because the world changed?". Strict additive — zero breaking changes on the contracts agent-v3 already calls.
### New services in `brain_api/services/`
| Module | Pilon | Role |
|--------|-------|------|
| `classifier.py` | 1 | LLM single-call classify of a claim → `{volatility, topic_codes, entity_bindings, estimated_validity_hours}`. Caps clamped per tier (volatile≤48h, evolving≤720h, stable≤26280h). Conservative defaults on LLM failure. |
| `cache_judge.py` | 2 | NLI judge run on cache hits — given a claim, cached truth direction (TRUE/FALSE/MIXED), and current top-3 evidence, decides KEEP_CACHE / INVALIDATE / NEEDS_FULL_RECHECK. Cheap-path skip for stable+young. Audit-pass bonus raises invalidate threshold for atoms that survived multiple audits. |
| `fact_status.py` | 11 | Versioned (subject, predicate, object) knowledge layer. `register_facts_from_bindings`, `assert_fact_truth` (opens new fact_version when truth flips), `lock_fact`/`unlock_fact` for moderator overrides, `check_fact_validity` for gather-time lookup. Plus admin: `list_facts_admin`, `list_audit_log`. |
| `canonicalizer.py` | 7 | LLM rewrites a claim with relative time markers (`azi`, `alegerile`) anchored to the current date. Same surface text at different times produces different cache keys. Passthrough on failure. |
| `invalidation.py` | 8 | `InvalidateFilter` (topic_codes / entity_canonicals / claim_pattern / since / invalidate_gold / dry_run) → bulk UPDATE on both cache tables. Audit log row written per non-dry-run execution. |
| `topic_volatility.py` | D1 | HTTP polls didiFramework `/api/sensitive-topics?active=true` (cache 60s) for admin-configured per-topic TTL/recency. `reconcile_with_classifier` picks min(LLM_estimate, admin_TTL). |
### DB schema (alongside Atomic in same Postgres, prefix `brain_*`)
```
ALTER brain_analysis_atom + brain_verification_cache
ADD volatility ('volatile'|'evolving'|'stable')
ADD topic_codes text[]
ADD entity_bindings jsonb -- list of {subject, predicate, object, confidence}
ADD ttl_hours_used integer
ADD last_audited_at timestamptz
ADD audit_history jsonb -- last-50 JudgeVerdict snapshots
ADD consecutive_audit_passes integer
CREATE brain_fact_status -- current truth per (subject, predicate, object)
CREATE brain_fact_version -- temporal versioning with valid_from/valid_to
CREATE brain_audit_log -- mass invalidations + judge decisions + truth changes
```
GIN indexes on `topic_codes`. Partial indexes on `cache_tier IN ('gold','silver') AND volatility != 'stable'` for the auditor's fast candidate query.
### New endpoints (`brain_api/app.py`)
```
POST /v1/canonicalize — Pilon 7 temporal disambiguation
POST /v1/cache/invalidate — Pilon 8 mass invalidation (with dry_run)
GET /v1/cache/audit_log — paginated audit browser
GET /v1/fact_status/list — paginated fact browser (admin)
GET /v1/fact_status/{fact_id} — single fact detail
GET /v1/fact_status/{fact_id}/versions — full timeline
PATCH /v1/fact_status/{fact_id} — moderator override (set_truth + lock + notes)
```
Existing upsert paths (`POST /v1/analysis_atom`, `POST /v1/verification_cache`) now run the classifier internally and persist volatility/topic_codes/entity_bindings on every write. Lookup paths (`/v1/gather` with `include_verification=true`) consult `brain_fact_status` and demote staleness to `stale_evidence` when bound facts have flipped.
### `gather.py` recency reranking (Pilon 3+4)
`_apply_recency` blends rerank score with age decay using volatility-driven profiles:
| Volatility | w_recency | half_life_days | hard cut |
|---|---|---|---|
| volatile | 0.50 | 3 | 7 days |
| evolving | 0.30 | 30 | (none) |
| stable | 0.00 | ∞ | (none) |
| (none / hint absent) | 0.15 | 30 | (none) |
`combined = (1 - w_recency) * rerank + w_recency * exp(-age_days/half_life)`. Volatile evidence older than `recency_window_days` is hard-dropped.
### `didibrain-scheduler` container (`scheduler/`)
Single image, four asyncio tasks under `_supervised` (auto-restart with exponential backoff):
- **feeder** — RSS pull at volatility-aware intervals (15 min volatile / 6 h evolving / 24 h stable). Default feeds in `config.py:DEFAULT_*_FEEDS`; override via `FEED_VOLATILE_URLS`/`FEED_EVOLVING_URLS`/`FEED_STABLE_URLS`/`FEED_BREAKING_URLS` env. Feeds POST to `/v1/ingest` with `run_extraction=true`.
- **auditor** — daily sweep of `brain_analysis_atom` rows with `cache_tier IN ('gold','silver') AND volatility != 'stable' AND last_audited_at < now()-1d`. For each: confidence-decay cheap path → judge → `apply_judge_verdict`. Sequential (not parallel) to avoid LLM router saturation.
- **watcher** — 5-min poll on the breaking-news feeds. Per item: classify with LLM → `_build_invalidation_targets` → `POST /v1/cache/invalidate` with `topic_codes` + `entity_canonicals`. In-process LRU of seen URLs (cap 2000). End-to-end target latency: <30s from RSS publish to cache invalidation.
- **heartbeat** — touches `/tmp/scheduler.healthy` every 30s for the docker healthcheck.
Compose entry in `infra/docker-compose.yml` builds via `scheduler/Dockerfile`. Imports `brain_api.*` directly so the auditor uses the same DB pool + `apply_judge_verdict` helper as the API container (separate process, separate pool to the same Postgres).
### Live state at write-time
47 articles ingested in first cycle. Watcher invalidated 4 breaking stories live (Iran/Hormuz, Nigerian elections, Ukraine drones, etc.) within seconds of feed publication. Auditor scheduled 24h after first start.
### Memory pointer
`~/.claude/projects/-home-admin365/memory/MEMORY.md` § "Cache Freshness Defense" — one-liner status + counts.

236
ai_platform/modules/didi_brain/README.md Normal file
View file

@ -0,0 +1,236 @@
# DidiBrain
A claim-verification knowledge cache for **Didi**, a disinformation analysis
application. DidiBrain sits between Didi's backend and Didi's live
web-gathering module and speaks the same HTTP contract, answering from
pre-ingested atoms + multilingual semantic retrieval + LLM stance
classification — typically in 1.5-6 seconds instead of 20+ seconds for a
live web crawl.
Built on top of [Atomic](https://github.com/kenforthewin/atomic), a Rust
knowledge-graph backend. DidiBrain adds the scraping, claim-level extraction,
reranker integration, NLI stance layer, HTTP contract, and the contradiction
audit job on top.
---
## What it is, in one paragraph
Didi's backend currently verifies claims by calling a live
`POST /v1/gather` endpoint that does web search + fetch + score. That's
slow (~23 s) and expensive on premium engines. DidiBrain exposes the same
HTTP contract but serves responses from a locally-grown **knowledge graph**:
- **Atoms** (documents and atomic claims) are stored in a Postgres +
pgvector backend with multilingual embeddings (BGE-M3, 1024-dim).
- **Claim extraction** runs Qwen 3.5 397B over every ingested document to
pull out verifiable atomic claims with source quotes and stance.
- **Retrieval** uses vector kNN + BGE-reranker-v2-m3 cross-encoder for
precision.
- **NLI stance vs query** classifies each piece of evidence as SUPPORTS,
CONTRADICTS, or NEUTRAL relative to the input claim — giving Didi the
signal it really needs for a disinfo verdict.
- **`brain_meta.cache_status`** tells Didi HIT / PARTIAL / MISS so the
backend can cleanly fall back to the expensive web module when the brain
doesn't have relevant knowledge yet.
The brain is also **self-populating**: whenever Didi falls back to the web
module and gets fresh evidence, it can POST the result to `/v1/ingest`
and DidiBrain will create new atoms + queue claim extraction in the
background. Next time a similar claim arrives, it's a HIT.
## Tech stack
| Layer | Tech |
|---|---|
| **Brain storage** | [Atomic](https://github.com/kenforthewin/atomic) (Rust) on Postgres 16 + pgvector |
| **Embeddings** | `BAAI/bge-m3` via vLLM OpenAI-compat (1024 dim, 8K ctx, multilingual) |
| **Reranker** | `BAAI/bge-reranker-v2-m3` cross-encoder via vllm-rerank-api |
| **LLM (reasoning)** | `Qwen3.5-397B-A17B` (MoE) via llama.cpp through an OpenAI-compat router |
| **Service** | Python 3.12, FastAPI, Uvicorn, Pydantic v2, httpx, structlog, tenacity |
| **Deployment** | Docker + docker-compose (3 services: api, atomic, postgres) |
## Quick start — local dev
Assumes Docker Desktop running and `.env` filled with reachable upstream
endpoints.
```bash
cd didibrain
# 1. Copy env template and fill upstream URLs
cp .env.example .env
# Edit LLM_ROUTER_URL, EMBEDDING_URL, RERANKER_URL to reachable addresses
# 2. Bring up the stack (atomic + postgres + brain_api)
docker compose -f infra/docker-compose.yml --env-file .env up -d --build
# 3. Create a Python venv for operator scripts
python3 -m venv .venv
.venv/bin/pip install httpx pydantic pydantic-settings structlog \
python-dotenv tenacity rich
# 4. Bootstrap Atomic (claim instance, configure BGE-M3 provider)
.venv/bin/python scripts/02_bootstrap_atomic.py
# 5. Seed the canonical tag taxonomy
.venv/bin/python scripts/04_seed_taxonomy.py
# 6. (Optional) Import seed Wikipedia corpus — first topic: vaccines
.venv/bin/python scripts/05_import_wikipedia_seed.py
# 7. (Optional) Extract atomic claims from the imported documents
.venv/bin/python scripts/07_run_extraction.py
# 8. Verify it works
curl -fsS http://localhost:8090/health
.venv/bin/python scripts/09_brain_api_demo.py
```
Open the interactive Swagger UI at **http://localhost:8090/docs** to poke
the 5 endpoints live.
## Production deploy — fresh Linux server
A single `bash` script does the whole thing:
```bash
scp -r didibrain/ user@server:~/
ssh user@server
cd ~/didibrain
cp .env.example .env
vim .env # set LLM/BGE/reranker URLs for this network
./scripts/bootstrap_deploy.sh # preflight → build → up → seed → smoke test
```
The script is idempotent — re-run it any time to pick up from wherever it
was interrupted. See `scripts/bootstrap_deploy.sh` for the exact flow and
optional env flags (`BRAIN_IMPORT_CORPUS=0` to skip the seed import, etc.).
Alternatively, run the numbered `scripts/XX_*.py` files in order by hand —
they are all idempotent and explicit.
## HTTP API — the contract
Brain-API speaks **Didi's existing web-gathering module contract**, 1:1 at
the response-shape level, with additive `brain_meta` fields that older
backends safely ignore.
| Endpoint | Purpose | Typical latency |
|---|---|---|
| `GET /health` | liveness | <10 ms |
| `GET /docs` | interactive Swagger UI | — |
| `GET /redoc` | read-only API docs | — |
| `GET /openapi.json` | machine-readable schema | — |
| `POST /v1/search` | flat list of doc-level search results | 200-400 ms |
| `POST /v1/fetch` | look up atoms by URL → extracted text | 30-100 ms |
| `POST /v1/gather` | claim → ranked evidence with NLI stance | 5-6 s (1.5 s without NLI) |
| `POST /v1/image-search` | stub (always empty list) | <5 ms |
| `POST /v1/ingest` | populate brain from web-module output | variable (async extraction) |
The response from `/v1/gather` matches the existing web-module shape
exactly plus an additive `brain_meta` object on the top level and inside
each `evidence[].provenance`. The key signal to branch on:
```python
resp = requests.post(f"{BRAIN_URL}/v1/gather", json={"claim": text}).json()
if resp["brain_meta"]["cache_status"] == "MISS":
resp = requests.post(f"{WEB_MODULE_URL}/v1/gather", json={"claim": text}).json()
# optional self-populating:
requests.post(f"{BRAIN_URL}/v1/ingest",
json={"claim": text, "evidence": resp["evidence"]})
```
See `STATUS.md` for the full field reference.
## Architecture
```
Didi backend
│
│ POST /v1/gather {claim}
▼
┌────────────────────────────────────────────────────┐
│ brain_api :8090 (FastAPI, Uvicorn) │
│ │
│ stage context → language detection │
│ stage retrieval → Atomic semantic search (top 50) │
│ stage rerank → BGE cross-encoder (top 15) │
│ stage nli → Qwen 397B stance vs query │
│ stage evidence → group by parent doc, shape │
└──────┬──────────────────────────┬──────────────────┘
│ │
│ HTTP (docker DNS) │ HTTPS (VPN)
▼ ▼
┌────────────────┐ ┌───────────────────────┐
│ atomic-server │ │ BGE-M3 embeddings │
│ :8080 (8088) │◀──SQL──│ & BGE reranker │
└──────┬─────────┘ │ (vLLM) │
│ └───────────────────────┘
▼
┌────────────────┐ ┌───────────────────────┐
│ postgres │ │ Qwen 3.5 397B-A17B │
│ + pgvector │ │ via LLM router │
│ :5432 (5434) │ │ (llama.cpp + vLLM) │
└────────────────┘ └───────────────────────┘
```
## Project structure
```
didibrain/
├── infra/docker-compose.yml # 3-service stack
├── brain_api/ # FastAPI service (the main deliverable)
├── shared/ # config, clients, taxonomy (reused everywhere)
├── extractor/ # claim extraction (host jobs + /v1/ingest)
├── lint/ # cross-corpus contradiction detection
├── scripts/ # operator CLI tools (numbered 01-11)
│ └── bootstrap_deploy.sh # fresh-server bootstrap script
├── STATUS.md # end-of-session snapshot + troubleshooting
└── README.md # this file
```
Detailed file-by-file rundown is in `STATUS.md`.
## Status
- [x] Upstream stack validation (LLM + BGE + reranker sanity gate)
- [x] Atomic running on Postgres + pgvector in compose
- [x] Canonical tag taxonomy (79 tags, 7 root namespaces)
- [x] First Wikipedia import (19 documents, EN+RO, vaccines topic)
- [x] Claim extraction (513 atoms, 1.2% hallucination filter)
- [x] Document-level retrieval validated (cross-lingual cosine 0.88-0.94)
- [x] Claim-level retrieval validated
- [x] brain_api HTTP service with Didi contract (5 endpoints)
- [x] brain_api dockerized (self-sufficient, taxonomy auto-refresh)
- [x] NLI stance vs query in `/v1/gather`
- [x] Lint pass contradiction detection (code + smoke test)
- [x] Deployment bootstrap script
- [ ] Corpus expansion — more topics, more sources (next session)
- [ ] Full Lint run on diverse corpus
- [ ] Bearer auth on brain_api (when exposing beyond loopback)
- [ ] Optional `atomic-web` frontend service
## Operational notes
- **Resource footprint**: ~200 MB RAM total across the 3 containers; brain-api
idles at ~6% CPU, spikes to ~20% during a `/v1/gather` with NLI.
- **Image size**: brain-api Docker image is ~253 MB (Python 3.12-slim base).
- **VPN dependency**: BGE and the LLM router live on a VPN-routed 10.11.10.x
network. If the VPN drops, `/v1/gather` returns 500 because Atomic cannot
embed the query. Confirm upstream reachability before debugging anything
else when search starts failing.
- **Self-sufficient startup**: brain_api pulls the current taxonomy from
Atomic at startup, so there is no baked `_tag_ids.json` in the image and
the container is portable across environments.
- **Idempotency**: every operator script (taxonomy seeder, Wikipedia
importer, claim extractor, Lint pass) is idempotent via state files or
URL-based dedup. Re-running is always safe.
## License & upstream
DidiBrain itself is private (not open source). It builds on top of
[Atomic](https://github.com/kenforthewin/atomic) (MIT). The Atomic
upstream clone at `D:\didi_brain\atomic\` is untouched — `git pull`
from upstream is always safe and doesn't conflict with anything in
this repo.

424
ai_platform/modules/didi_brain/STATUS.md Normal file
View file

@ -0,0 +1,424 @@
# DidiBrain — Status Snapshot
**Last updated:** 2026-04-23 (integration session — web-api cache + backend verification cache)
**Working dir:** `/home/admin365/ml-projects/modules/didi_brain/`
**Related docs:**
- `CONTRACT_VERIFICATION_CACHE.md` — contract backend↔brain pentru verification cache (v2, current)
- `../../CHANGES_2026-04-23.md` — session changelog cu lista completă de schimbări
- `../web/BRAIN_INTEGRATION.md` — integrarea web-api ↔ brain
---
## TL;DR
DidiBrain is **SHIP READY** și **INTEGRATED**. Pe lângă contractul inițial HTTP
(web-gathering compatibility), brain servește acum și ca:
- **Cache layer pentru web-api** — premium ingests + read pentru ambele tiers
- **Verification cache pentru didi-backend** — post-LLM stance/verdict storage
cu 4 staleness states (fresh / stale_framework / stale_prompt / miss)
**Trei containere, zero regressions pe funcționalitatea v1.** Tabelul relational
nou `brain_verification_cache` trăiește în același Postgres cu atomic-server,
prefix `brain_` pentru izolare.
## Current state in 30 seconds
```
3 containers running continuously, all healthy
didibrain-api brain_api FastAPI service :8090 52 MB RAM
didibrain-atomic atomic-server (API only) :8088 83 MB RAM
didibrain-postgres pgvector pg16 :5434 60 MB RAM
(+ brain_verification_cache table)
Brain contents (grow continuously via web-api ingest + manual bootstrap)
79 tags (canonical taxonomy, 7 root namespaces)
19+ documents (initial Wikipedia vaccines seed + anything premium adds)
509+ claims (extracted by Qwen 397B, substring validated)
Verification cache (new in v2 — Apr 23)
brain_verification_cache UNIQUE (claim_hash, tier)
TTL 30 days, max payload 64KB, UPSERT last-wins
4 staleness states exposed via brain_meta
Query latency (measured end-to-end)
/v1/gather with NLI ~5-6 s
/v1/gather no NLI ~1.5-2 s
/v1/search ~200-400 ms
/v1/fetch ~30-100 ms
/v1/image-search ~5 ms (stub)
/v1/ingest variable (atom create + optional async extract)
```
## What it does
Brain-API speaks **Didi's existing web-gathering module contract**. Didi's
backend can call it like calling the web module today, and switch between
them transparently based on `brain_meta.cache_status`:
```
Didi backend ┌───────────────────┐
│ │ DidiBrain │
│ POST /v1/gather {claim} │ (this repo) │
├────────────────────────────────▶│ │
│ │ 1 semantic search│
│ │ 2 BGE rerank │
│ │ 3 NLI stance │
│ │ 4 aggregate │
│◀────────────────────────────────┤ │
│ GatherResponse └───────────────────┘
│ (brain_meta.cache_status = HIT | PARTIAL | MISS)
│
if cache_status == MISS:
fallback = http.post(WEB_MODULE, {claim})
# optional: POST /v1/ingest {fallback.evidence}
# so the brain self-populates from the expensive web module
```
## Architecture
```
┌─────────────────────────────────────────────────────────────────┐
│ DidiBrain stack │
│ │
│ ┌──────────────────────┐ │
│ │ brain-api :8090 │ FastAPI + Uvicorn, 5 v1 endpoints │
│ │ (didibrain-api) │ speaks Didi contract 1:1 │
│ └──────────┬───────────┘ │
│ │ HTTP (docker DNS) │
│ ┌──────────▼───────────┐ ┌─────────────────────┐ │
│ │ atomic-server │ │ postgres + pgvector│ │
│ │ :8080 (8088 ext) │◀─┤ :5432 (5434 ext) │ │
│ │ (didibrain-atomic) │ │ (didibrain-postgres│ │
│ └──────┬───────┬───────┘ └─────────────────────┘ │
│ │ │ │
└─────────┼───────┼───────────────────────────────────────────────┘
│ │
│ │ HTTPS
│ ▼
│ ┌────────────────────────────────┐
│ │ BGE-M3 embeddings │ 10.11.10.15:8200
│ │ (vLLM OpenAI-compat) │ 1024 dim, 8K ctx, multilingual
│ └────────────────────────────────┘
│
│ ┌────────────────────────────────┐
│ │ BGE-reranker-v2-m3 │ 10.11.10.15:8100
│ │ (cross-encoder) │ precision boost
│ └────────────────────────────────┘
│
│ ┌────────────────────────────────┐
└─▶│ Qwen3.5-397B-A17B │ 10.11.10.17:14011 (router)
│ via LLM router │ round-robin to .18 and .19
│ (llama.cpp + vLLM backends) │ claim extraction, NLI
└────────────────────────────────┘
```
All upstream endpoints live on a VPN-routed 10.11.10.x network. The Docker
containers reach them through WSL2 NAT (Docker Desktop) or host networking
(Linux server).
## Repo layout
```
didibrain/
├── .env # real endpoints + token (gitignored)
├── .env.example # template
├── .dockerignore
├── .gitignore
├── pyproject.toml
├── README.md # project landing
├── STATUS.md # this file
├── AUDIT.md # initial upstream Atomic audit
│
├── infra/
│ └── docker-compose.yml # 3-service stack (+ optional atomic-web)
│
├── brain_api/ # HTTP service — the main deliverable
│ ├── Dockerfile
│ ├── requirements.txt
│ ├── app.py # FastAPI routes
│ ├── schemas.py # Pydantic v2 models — Didi contract
│ ├── deps.py # app state (clients)
│ ├── run.py # uvicorn entry
│ ├── prompts/
│ │ └── nli_v1.md
│ └── services/
│ ├── mapping.py # atom → EvidenceItem / FetchedPage
│ ├── gather.py # /v1/gather pipeline
│ ├── search.py # /v1/search
│ ├── fetch.py # /v1/fetch
│ ├── ingest.py # /v1/ingest + background extraction
│ └── nli.py # stance vs query classification
│
├── shared/ # libraries used by brain_api + scripts + lint
│ ├── config.py # Pydantic Settings, LlmRole, model routing
│ ├── logging.py # structlog, UTF-8 stdout, httpx silencer
│ ├── llm_client.py # async OpenAI-compat wrapper
│ ├── embedding_client.py # BGE-M3 embed + reranker
│ ├── atomic_api.py # typed REST client for atomic-server
│ └── taxonomy.py # canonical TAXONOMY tree + TagResolver
│
├── extractor/ # claim extraction service (host + background)
│ ├── extract.py # single-doc extraction + substring quote validation
│ ├── push.py # claim atom creation with hash URL fragment
│ ├── batch.py # batch orchestrator with state file
│ ├── _state.py # idempotency log
│ └── prompts/
│ └── claim_extraction_v1.md
│
├── lint/ # background contradiction detection
│ ├── runner.py # orchestrator
│ ├── detector.py # single-pair NLI
│ ├── pairs.py # candidate generation via find_similar
│ ├── reporter.py # rich table output
│ ├── _state.py # PairVerdict ledger (idempotent)
│ └── prompts/
│ └── pair_nli_v1.md
│
└── scripts/ # operator CLI tools (host Python venv)
├── 01_sanity_full.py # upstream stack validator (LLM+embed+rerank)
├── 02_bootstrap_atomic.py # claim instance, configure provider
├── 03_sanity_atomic.py # brain end-to-end (create→embed→search→cleanup)
├── 04_seed_taxonomy.py # taxonomy seeder (idempotent)
├── 05_import_wikipedia_seed.py # seed-list Wikipedia import
├── 06_validate_queries.py # smoke test doc-level retrieval
├── 07_run_extraction.py # claim extraction batch
├── 08_validate_claims.py # smoke test claim-level retrieval
├── 09_brain_api_demo.py # brain_api contract test (all 5 endpoints)
├── 10_run_lint.py # Lint pass runner (--limit --force)
├── 11_show_contradictions.py # read state file, render top contradictions
└── bootstrap_deploy.sh # fresh-server deploy orchestrator
```
## HTTP API summary — Didi contract
All responses include an additive `brain_meta` object with `cache_status`,
`api_version`, `implementation`, `evidence_sources`, and
`total_claim_atoms_matched`. Unknown fields are safely ignored by older
backends.
| Endpoint | Purpose | Latency | Notes |
|---|---|---|---|
| `GET /health` | liveness | <10 ms | container healthcheck |
| `GET /docs` | Swagger UI | — | interactive tester (FastAPI auto) |
| `GET /redoc` | ReDoc | — | read-only doc view |
| `GET /openapi.json` | contract JSON | — | machine-readable schema |
| `POST /v1/search` | flat search results | ~200-400 ms | no rerank, no NLI |
| `POST /v1/fetch` | URL lookup → text | ~30-100 ms | returns `not_in_brain` for misses |
| `POST /v1/gather` | claim → ranked evidence | ~5-6 s with NLI | **the main Didi entry point** |
| `POST /v1/gather` with `run_nli:false` | claim → evidence | ~1.5-2 s | skip stance classification for speed |
| `POST /v1/image-search` | stub | <5 ms | always empty list |
| `POST /v1/ingest` | populate from web module | variable | creates atoms + optional async extraction |
### /v1/gather response shape (key fields)
```json
{
"request_id": "uuid",
"claim": "original claim text",
"evidence": [
{
"url": "https://source.example",
"title": "...",
"publisher": "example.com",
"published_at": "2024-...",
"retrieved_at": "2026-04-11T...",
"summary": "the matching claim text, canonical single sentence",
"full_text": "full parent document content",
"full_text_hash": "sha256",
"relevance_score": 0.98,
"credibility_score": 0.70,
"provenance": {
"extraction_method": "brain",
"fallback_chain": [],
"brain_meta": {
"parent_atom_id": "uuid",
"best_claim_text": "...",
"best_claim_stance_in_source": "ASSERTS|REPORTS|REFUTES|QUESTIONS|NEUTRAL",
"stance_vs_query": "SUPPORTS|CONTRADICTS|NEUTRAL|UNKNOWN",
"nli_confidence": 0.95,
"reranker_score": 0.98,
"embedding_similarity": 0.74
}
}
}
],
"evidence_stats": { ... },
"search_context": { "primary_country": "Global", "detected_language": "en", ... },
"stages": [
{ "stage": "context", "success": true, "duration_ms": 0 },
{ "stage": "retrieval", "success": true, "duration_ms": 300 },
{ "stage": "rerank", "success": true, "duration_ms": 1000 },
{ "stage": "nli", "success": true, "duration_ms": 3800 },
{ "stage": "evidence", "success": true, "duration_ms": 300 }
],
"total_evidence_items": 5,
"execution_time_ms": 5400,
"brain_meta": {
"cache_status": "HIT",
"api_version": "v1",
"implementation": "didibrain",
"evidence_sources": 5,
"total_claim_atoms_matched": 42
}
}
```
The key signal for Didi backend:
```python
response = requests.post(f"{BRAIN_URL}/v1/gather", json={"claim": text})
data = response.json()
if data["brain_meta"]["cache_status"] == "MISS":
# brain does not have relevant knowledge — fall back to live web module
data = requests.post(f"{WEB_MODULE_URL}/v1/gather", json={"claim": text}).json()
# optional: pump it into the brain so next time we HIT
requests.post(f"{BRAIN_URL}/v1/ingest", json={"evidence": data["evidence"]})
```
## Deployment on a server — cheat sheet
Full automated flow:
```bash
scp -r didibrain/ user@server:~/
ssh user@server
cd ~/didibrain
cp .env.example .env
vim .env # fill LLM_ROUTER_URL / EMBEDDING_URL / RERANKER_URL
./scripts/bootstrap_deploy.sh
```
Manual step-by-step is documented in `scripts/bootstrap_deploy.sh` comments
or in README.md under "Production deploy".
### What to change in .env when moving machines
Only the upstream endpoint URLs may need updating:
```bash
LLM_ROUTER_URL=http://10.11.10.17:14011 # if router is on VPN, unchanged
EMBEDDING_URL=http://10.11.10.15:8200 # if BGE is on VPN, unchanged
RERANKER_URL=http://10.11.10.15:8100 # if reranker is on VPN, unchanged
```
Everything else (`ATOMIC_URL`, ports, model names, Postgres creds) is either
handled by Docker DNS automatically or comes from the same file.
## Health checklist — run after any deploy/restart
```bash
# 1. Containers healthy
docker ps --format '{{.Names}} {{.Status}}' | grep didibrain
# expected: 3 lines, all "Up ... (healthy)"
# 2. Liveness
curl -fsS http://localhost:8090/health
# expected: {"status":"ok","service":"didibrain-api","version":"0.1.0"}
# 3. Full stack sanity (LLM + embed + rerank + atomic)
.venv/bin/python scripts/01_sanity_full.py
# expected: 8/8 PASS, total <10s
# 4. Brain end-to-end (create atom → embed → search → cleanup)
.venv/bin/python scripts/03_sanity_atomic.py
# expected: 5/5 PASS
# 5. Real Didi-style query
curl -s -X POST http://localhost:8090/v1/gather \
-H 'Content-Type: application/json' \
-d '{"claim":"vaccines cause autism","max_evidence":3,"run_nli":false}' \
| jq '.brain_meta.cache_status, .total_evidence_items'
# expected: "HIT" and a positive number (if the vaccines corpus is loaded)
```
## Known issues that DO NOT block deploy
1. **`edges_status column does not exist`** — atomic-server logs this warning
at startup due to an upstream Postgres schema mismatch. The semantic_edges
feature in Atomic is partially broken, but **nothing we use depends on it**:
vector search, reranker, claim extraction, and the Didi contract all work.
Cosmetic — safe to ignore until upstream Atomic fixes the migration.
2. **`/api/embeddings/status`** returns 500 — same root cause as #1. We never
call this endpoint; we read `embedding_status` per atom via `list_atoms`
or `get_atom` instead.
3. **Qwen 35B (vLLM on :14001)** — thinking mode stuck ON via the router and
safety alignment refuses disinfo-extraction tasks. Disabled in config
(`MODEL_FAST_ENABLED=false`); the whole pipeline runs on Qwen 397B. If a
future session un-sticks 35B, Lint pass could speed up ~4x by using it.
4. **Gemma 31B endpoint (10.11.10.16:8001)** — port unreachable (host pings
OK). Not used by any feature; listed as optional in `.env`.
5. **Atomic's React UI not running** — we only deploy `atomic-server` (API),
not `atomic-web` (React frontend). `http://localhost:8088/` returns 404
for this reason. Brain_api has its own Swagger UI at `/docs` which covers
dev-testing needs. If visual atom/tag/wiki browsing is needed, add an
`atomic-web` service to compose (5-minute task).
## Troubleshooting — things that have actually gone wrong
| Symptom | Root cause | Fix |
|---|---|---|
| `/health` timeouts | BGE endpoint unreachable (VPN down?) | `curl http://10.11.10.15:8200/v1/models` on host; bring VPN back |
| `/v1/gather` returns 500 silently | Same as above — Atomic can't embed the query | Same fix |
| Post-reboot: brain_api empty reply | uvicorn bound to 127.0.0.1 inside container (not 0.0.0.0) | Rebuild — Dockerfile now sets `BRAIN_API_HOST=0.0.0.0` |
| `LLM_ROUTER_URL` points to localhost but nothing there | Router is actually on 10.11.10.17:14011 (not on the dev box) | Update `.env`; validated in session 2 |
| Claim extraction returns zero claims on a doc | `list_atoms` returns summary without `content`; must call `get_atom` | Fixed in `extractor/batch.py` |
| NLI responses come back as NEUTRAL with confidence 0.0 | MAX_PARALLEL > backend concurrency → timeouts | Set `MAX_PARALLEL=2` to match 2 llama.cpp instances |
| Wikipedia API 403 on every request | Non-compliant User-Agent | Use `DidiBrain/0.1 (https://github.com/didibrain; didibrain@local.test)` |
## What is NOT done (all optional, not blockers)
- **Bearer auth on brain_api** — skipped intentionally. Add when exposing
beyond loopback on a server with a public IP.
- **Full Lint pass on vaccines corpus** — skipped. Wikipedia is too
internally consistent for Lint to surface many contradictions; rerun
after corpus diversification (B).
- **Corpus expansion (B)** — next session. Wikipedia seeds for RO elections
2024, Russia-Ukraine war, climate, COVID general. Also RSS feeds and
possibly Playwright adapters for non-Wiki sources.
- **`/v1/contradictions` endpoint in brain_api** — currently contradictions
are only visible via `scripts/11_show_contradictions.py`. Expose over
HTTP when Didi needs programmatic access.
- **Atomic React UI (`atomic-web`)** — optional visual browsing, add if
useful for operators.
- **Monitoring / metrics** — no Prometheus exporter yet.
- **Upstream schema fix for `edges_status`** — cosmetic, PR to kenforthewin/atomic.
## Session log (condensed)
### Session 1 (2026-04-10, ~6 hours)
- Full audit of upstream Atomic repo
- Validated LLM/embed/rerank stack (8 sanity checks PASS)
- Built `shared/` + `extractor/` + scripts 01-08
- Stood up Atomic + Postgres in compose
- Imported 19 Wikipedia vaccines articles, EN + RO
- Extracted 513 claims via Qwen 397B (1.2% hallucination drop)
- Validated claim-level retrieval (cross-lingual RO↔EN confirmed)
### Session 2 (2026-04-11, ~4 hours)
- Re-onboarded state after VPN restart (fixed LLM router URL)
- Built `brain_api/` FastAPI service with 5-endpoint Didi contract
- Dockerized brain_api (self-sufficient startup, taxonomy refresh from atomic)
- Added NLI stance-vs-query pass in `/v1/gather` (additive fields in `brain_meta`)
- Fixed NLI concurrency (MAX_PARALLEL=2 matches llama.cpp backends)
- Built `lint/` contradiction detection module with idempotent state file
- Smoke-tested Lint (371 pairs, 1 genuine contradiction found at 0.95 confidence)
- Wrote deployment bootstrap script
- Wrote this STATUS.md and the new README.md
## Next session — pick-up plan
Brain is in a stable end-of-session state. To resume:
1. Verify infra is alive (`docker ps | grep didibrain`)
2. If it is not, `docker compose -f infra/docker-compose.yml --env-file .env up -d`
3. Run `.venv/bin/python scripts/01_sanity_full.py` to confirm upstream stack
4. Decide: corpus expansion (B), Lint full run, new feature, or deploy to server
**Natural next step is corpus expansion.** The code paths are complete;
everything else is about feeding the brain more knowledge and then running
the existing scripts over the new data.

60
ai_platform/modules/didi_brain/brain_api/Dockerfile Normal file
View file

@ -0,0 +1,60 @@
# syntax=docker/dockerfile:1.6
# =============================================================================
# brain_api — DidiBrain HTTP service that speaks Didi's web-module contract.
#
# Build context is the didibrain/ project root (one level up), so we can
# COPY shared/ + extractor/ + brain_api/ in one shot:
#
# docker build -t didibrain-api -f brain_api/Dockerfile .
#
# Build via docker-compose at infra/docker-compose.yml (see `brain-api`
# service) for the normal flow.
# =============================================================================
FROM python:3.12-slim-bookworm AS runtime
ENV PYTHONDONTWRITEBYTECODE=1 \
PYTHONUNBUFFERED=1 \
PYTHONIOENCODING=utf-8 \
PIP_NO_CACHE_DIR=1 \
PIP_DISABLE_PIP_VERSION_CHECK=1 \
PIP_ROOT_USER_ACTION=ignore \
BRAIN_API_HOST=0.0.0.0 \
BRAIN_API_PORT=8090
WORKDIR /app
# curl is used by the HEALTHCHECK directive below.
RUN apt-get update \
&& apt-get install -y --no-install-recommends curl \
&& rm -rf /var/lib/apt/lists/*
# Install Python dependencies first so the layer is cached when only app
# code changes (which is the common case while iterating).
COPY brain_api/requirements.txt /tmp/requirements.txt
RUN pip install --upgrade pip \
&& pip install -r /tmp/requirements.txt \
&& rm /tmp/requirements.txt
# App code. We intentionally COPY each top-level package separately so any
# accidental extras (reports/, .venv/, etc.) don't sneak in even if
# .dockerignore is missing.
COPY shared /app/shared
COPY extractor /app/extractor
COPY brain_api /app/brain_api
# Non-root runtime for safety. /app is owned by `brain` so the extractor
# state file (extractor/_extracted.json) can be written if /v1/ingest fires
# a background extraction run.
RUN useradd --system --create-home --shell /bin/false brain \
&& chown -R brain:brain /app
USER brain
EXPOSE 8090
# Liveness — the app exposes /health, which returns {status:"ok"} as soon
# as the lifespan hook finishes (taxonomy refresh included).
HEALTHCHECK --interval=10s --timeout=5s --start-period=20s --retries=5 \
CMD curl -fsS http://localhost:8090/health || exit 1
CMD ["python", "-m", "brain_api.run"]

6
ai_platform/modules/didi_brain/brain_api/__init__.py Normal file
View file

@ -0,0 +1,6 @@
"""DidiBrain HTTP API — speaks the same dialect as Didi's web-gathering module.
Exposes /v1/search, /v1/fetch, /v1/gather, /v1/image-search, /v1/ingest over
FastAPI. Didi's backend treats this service as a drop-in cache/source that
happens to answer from pre-ingested knowledge rather than live web crawling.
"""

1283
ai_platform/modules/didi_brain/brain_api/app.py Normal file
View file

File diff suppressed because it is too large Load diff

352
ai_platform/modules/didi_brain/brain_api/db.py Normal file
View file

@ -0,0 +1,352 @@
"""PostgreSQL connection pool + schema bootstrap for brain-side caches.
Brain owns a small relational layer alongside the Atomic knowledge graph for
things that don't fit as atoms (verification caches keyed by multiple fields
with TTL and per-column indexes). We connect directly to the same Postgres
instance atomic-server uses, but keep our tables in the `public` schema with
a `brain_` prefix so they're easy to spot and never collide with Atomic's.
"""
from __future__ import annotations
import asyncpg
from shared.config import settings
from shared.logging import get_logger
log = get_logger(__name__)
_SCHEMA_SQL = """
CREATE TABLE IF NOT EXISTS brain_verification_cache (
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
claim_hash text NOT NULL,
tier text NOT NULL CHECK (tier IN ('free', 'premium')),
-- evidence_hash and evidence_urls are METADATA only. The URLs brain
-- returns at gather time may differ from what backend wrote cache with
-- (corpus drift, ranker tie-breaks). We still store them so backend can
-- compare overlap and decide if cache is applicable to the current
-- evidence set. Lookup key is (claim_hash, tier) — one row per claim-
-- tier pair, UPSERT last-writer-wins.
evidence_hash text NOT NULL,
evidence_urls jsonb NOT NULL,
model text,
prompt_hash text NOT NULL,
framework_version text,
schema_name text NOT NULL DEFAULT 'didi-v1',
verification_raw jsonb,
verification_processed jsonb NOT NULL,
created_at timestamptz NOT NULL DEFAULT now(),
updated_at timestamptz NOT NULL DEFAULT now(),
expires_at timestamptz NOT NULL
);
-- v1 schema had UNIQUE (claim_hash, evidence_hash, tier). That made the
-- cache unreachable because evidence URLs at read time rarely match write
-- time. v2 = UNIQUE (claim_hash, tier) + new evidence_urls jsonb column.
-- Migrations are idempotent.
ALTER TABLE brain_verification_cache
ADD COLUMN IF NOT EXISTS evidence_urls jsonb NOT NULL DEFAULT '[]'::jsonb;
DO $$
BEGIN
IF EXISTS (
SELECT 1 FROM pg_constraint
WHERE conname = 'brain_verification_cache_claim_hash_evidence_hash_tier_key'
) THEN
ALTER TABLE brain_verification_cache
DROP CONSTRAINT brain_verification_cache_claim_hash_evidence_hash_tier_key;
END IF;
IF NOT EXISTS (
SELECT 1 FROM pg_constraint
WHERE conname = 'brain_verification_cache_claim_hash_tier_key'
) THEN
ALTER TABLE brain_verification_cache
ADD CONSTRAINT brain_verification_cache_claim_hash_tier_key
UNIQUE (claim_hash, tier);
END IF;
END$$;
CREATE INDEX IF NOT EXISTS idx_bvc_lookup
ON brain_verification_cache (claim_hash, tier);
CREATE INDEX IF NOT EXISTS idx_bvc_expires
ON brain_verification_cache (expires_at);
CREATE INDEX IF NOT EXISTS idx_bvc_prompt
ON brain_verification_cache (prompt_hash);
-- Drop the old 3-column lookup index if it exists.
DROP INDEX IF EXISTS idx_bvc_lookup_v1;
-- ============================================================================
-- brain_analysis_atom — cache for full-component LLM results (techniques, ai_tampered).
-- One row per (content_hash, component, prompt_hash). Tier is stored on the row
-- but NOT part of the unique key: write only happens for tier='premium', read
-- is tier-agnostic so free users benefit from premium-cached results.
--
-- 3 cache tiers:
-- gold — human_validated=true (set by didi moderation HIL flow). Survives
-- prompt change. Returned at maximum confidence.
-- silver — LLM result, llm_confidence >= threshold. Default for fresh writes.
-- bronze — LLM result, llm_confidence < threshold. Stored for audit but
-- NEVER served on lookup.
-- ============================================================================
CREATE TABLE IF NOT EXISTS brain_analysis_atom (
atom_id bigserial PRIMARY KEY,
content_hash text NOT NULL,
content_preview text,
component text NOT NULL CHECK (component IN ('techniques', 'ai_tampered', 'claims')),
tier text NOT NULL CHECK (tier IN ('free', 'premium')),
prompt_hash text NOT NULL,
framework_version text,
model_used text,
result_processed jsonb NOT NULL,
result_raw jsonb,
llm_confidence numeric,
cache_tier text NOT NULL DEFAULT 'silver'
CHECK (cache_tier IN ('gold', 'silver', 'bronze')),
human_validated boolean NOT NULL DEFAULT false,
human_corrections jsonb,
validator_user_id text,
validated_at timestamptz,
hit_count integer NOT NULL DEFAULT 0,
last_hit_at timestamptz,
created_at timestamptz NOT NULL DEFAULT now(),
updated_at timestamptz NOT NULL DEFAULT now(),
expires_at timestamptz, -- NULL = never expires (gold)
UNIQUE (content_hash, component, prompt_hash)
);
CREATE INDEX IF NOT EXISTS idx_baa_lookup
ON brain_analysis_atom (content_hash, component);
CREATE INDEX IF NOT EXISTS idx_baa_gold
ON brain_analysis_atom (component, cache_tier)
WHERE cache_tier = 'gold';
CREATE INDEX IF NOT EXISTS idx_baa_expires
ON brain_analysis_atom (expires_at)
WHERE expires_at IS NOT NULL;
CREATE INDEX IF NOT EXISTS idx_baa_prompt
ON brain_analysis_atom (prompt_hash);
-- ============================================================================
-- Volatility-aware caching extension (2026-05-04)
-- ============================================================================
-- Per-row volatility classification + topic tagging + audit history. Driven by
-- LLM classifier at write time (services/classifier.py) and used by:
-- - lookup paths (services/cache_judge.py) → confidence decay, NLI judge
-- - daily auditor (didibrain-auditor) → consecutive_audit_passes tracking
-- - breaking news watcher (didibrain-breaking-watcher) → topic-based mass invalidation
-- All ALTERs are idempotent — safe on every connect.
-- ============================================================================
ALTER TABLE brain_analysis_atom
ADD COLUMN IF NOT EXISTS volatility text
CHECK (volatility IN ('volatile', 'evolving', 'stable')),
ADD COLUMN IF NOT EXISTS topic_codes text[] DEFAULT ARRAY[]::text[],
ADD COLUMN IF NOT EXISTS entity_bindings jsonb DEFAULT '[]'::jsonb,
ADD COLUMN IF NOT EXISTS ttl_hours_used integer,
ADD COLUMN IF NOT EXISTS last_audited_at timestamptz,
ADD COLUMN IF NOT EXISTS audit_history jsonb DEFAULT '[]'::jsonb,
ADD COLUMN IF NOT EXISTS consecutive_audit_passes integer DEFAULT 0;
ALTER TABLE brain_verification_cache
ADD COLUMN IF NOT EXISTS volatility text
CHECK (volatility IN ('volatile', 'evolving', 'stable')),
ADD COLUMN IF NOT EXISTS topic_codes text[] DEFAULT ARRAY[]::text[],
ADD COLUMN IF NOT EXISTS entity_bindings jsonb DEFAULT '[]'::jsonb,
ADD COLUMN IF NOT EXISTS ttl_hours_used integer,
ADD COLUMN IF NOT EXISTS last_audited_at timestamptz,
ADD COLUMN IF NOT EXISTS audit_history jsonb DEFAULT '[]'::jsonb,
ADD COLUMN IF NOT EXISTS consecutive_audit_passes integer DEFAULT 0;
-- GIN indexes on topic_codes for fast topic-scoped invalidation/audit queries.
CREATE INDEX IF NOT EXISTS idx_baa_topics
ON brain_analysis_atom USING GIN (topic_codes);
CREATE INDEX IF NOT EXISTS idx_bvc_topics
ON brain_verification_cache USING GIN (topic_codes);
-- Partial indexes targeting audit-eligible rows (gold+silver, non-stable).
-- Auditor cron queries these to find atoms due for re-verification.
CREATE INDEX IF NOT EXISTS idx_baa_audit_due
ON brain_analysis_atom (last_audited_at NULLS FIRST, volatility)
WHERE cache_tier IN ('gold', 'silver') AND volatility IS NOT NULL AND volatility != 'stable';
CREATE INDEX IF NOT EXISTS idx_bvc_audit_due
ON brain_verification_cache (last_audited_at NULLS FIRST, volatility)
WHERE volatility IS NOT NULL AND volatility != 'stable';
-- ============================================================================
-- brain_fact_status — current truth value for entity-predicate-object triples.
-- Pilon 11 (versioned facts). Populated by:
-- - extractor pipeline (services/fact_status.py::extract_facts_from_claim)
-- when ingesting new claim atoms
-- - moderator overrides (admin endpoint PATCH /v1/fact_status/{id})
-- - breaking news watcher (when LLM detects fact change in fresh article)
-- Read by:
-- - gather lookup → if any bound fact is invalid, treat cache as stale_evidence
-- - dashboard fact browser (Phase D2)
-- ============================================================================
CREATE TABLE IF NOT EXISTS brain_fact_status (
fact_id bigserial PRIMARY KEY,
subject text NOT NULL, -- "Vladimir Putin"
predicate text NOT NULL, -- "is_president_of"
object text NOT NULL, -- "Russia"
canonical_form text NOT NULL, -- "Vladimir Putin is_president_of Russia"
canonical_form_hash text NOT NULL, -- sha256(canonical_form)[:32]
current_truth boolean, -- TRUE / FALSE / NULL=unknown
current_version_id bigint, -- non-FK pointer to brain_fact_version (avoid circular FK)
current_confidence numeric, -- 0-100, from latest LLM judgment
last_verified_at timestamptz,
last_evidence_urls jsonb DEFAULT '[]'::jsonb,
volatility text CHECK (volatility IN ('volatile', 'evolving', 'stable')),
topic_codes text[] DEFAULT ARRAY[]::text[],
-- Scheduling: auditor picks up rows where next_check_at < now()
next_check_at timestamptz NOT NULL DEFAULT now(),
check_interval_hours integer NOT NULL DEFAULT 24,
-- HIL trail
moderator_locked boolean NOT NULL DEFAULT false, -- true = audit cron must NOT auto-update
moderator_user_id text,
moderator_notes text,
created_at timestamptz NOT NULL DEFAULT now(),
updated_at timestamptz NOT NULL DEFAULT now(),
UNIQUE (canonical_form_hash)
);
CREATE INDEX IF NOT EXISTS idx_bfs_subject_predicate
ON brain_fact_status (subject, predicate);
CREATE INDEX IF NOT EXISTS idx_bfs_check_due
ON brain_fact_status (next_check_at)
WHERE moderator_locked = false;
CREATE INDEX IF NOT EXISTS idx_bfs_topics
ON brain_fact_status USING GIN (topic_codes);
CREATE INDEX IF NOT EXISTS idx_bfs_truth
ON brain_fact_status (current_truth)
WHERE current_truth IS NOT NULL;
-- ============================================================================
-- brain_fact_version — temporal versioning for facts.
-- Each row = one truth assertion valid in a [valid_from, valid_to) window.
-- valid_to IS NULL means "currently in force". When a fact changes, the active
-- version gets valid_to=now() and a new version is opened.
-- ============================================================================
CREATE TABLE IF NOT EXISTS brain_fact_version (
version_id bigserial PRIMARY KEY,
fact_id bigint NOT NULL REFERENCES brain_fact_status(fact_id) ON DELETE CASCADE,
truth_value boolean NOT NULL,
confidence numeric, -- 0-100
valid_from timestamptz NOT NULL,
valid_to timestamptz, -- NULL = current
source_atom_ids text[] DEFAULT ARRAY[]::text[], -- Atomic atom IDs supporting this version
evidence_urls jsonb DEFAULT '[]'::jsonb,
llm_reasoning text, -- LLM justification for this assertion
created_by text NOT NULL DEFAULT 'auto', -- 'auto' | 'moderator' | 'breaking_news_watcher'
moderator_user_id text, -- if created_by='moderator'
notes text,
created_at timestamptz NOT NULL DEFAULT now()
);
CREATE INDEX IF NOT EXISTS idx_bfv_fact
ON brain_fact_version (fact_id, valid_from DESC);
-- Partial index for currently-active versions (the most common lookup pattern).
CREATE INDEX IF NOT EXISTS idx_bfv_current
ON brain_fact_version (fact_id)
WHERE valid_to IS NULL;
-- ============================================================================
-- Audit log table (small, for invalidation/promotion/moderator actions).
-- Mirrors the lightweight audit pattern used by didi-admin's audit_log.
-- ============================================================================
CREATE TABLE IF NOT EXISTS brain_audit_log (
log_id bigserial PRIMARY KEY,
action text NOT NULL, -- 'invalidate' | 'promote_gold' | 'fact_override' | 'audit_demote'
target_table text NOT NULL, -- 'brain_analysis_atom' | 'brain_verification_cache' | 'brain_fact_status'
target_id text NOT NULL,
actor text, -- 'auditor' | 'breaking_watcher' | keycloak_id
payload jsonb DEFAULT '{}'::jsonb,
created_at timestamptz NOT NULL DEFAULT now()
);
CREATE INDEX IF NOT EXISTS idx_bal_target
ON brain_audit_log (target_table, target_id);
CREATE INDEX IF NOT EXISTS idx_bal_action_time
ON brain_audit_log (action, created_at DESC);
"""
class Database:
"""Thin wrapper over an asyncpg pool with a migration hook."""
def __init__(self, dsn: str) -> None:
self._dsn = dsn
self._pool: asyncpg.Pool | None = None
async def connect(self) -> None:
if self._pool is not None:
return
self._pool = await asyncpg.create_pool(
self._dsn,
min_size=1,
max_size=5,
command_timeout=10.0,
)
async with self._pool.acquire() as conn:
# pgcrypto for gen_random_uuid() — atomic-server usually enables it
# already via its own migrations, but we do it idempotently just
# in case brain is the first consumer on a fresh volume.
await conn.execute("CREATE EXTENSION IF NOT EXISTS pgcrypto")
await conn.execute(_SCHEMA_SQL)
log.info("brain_db_ready", dsn=self._redacted_dsn())
async def close(self) -> None:
if self._pool is not None:
await self._pool.close()
self._pool = None
@property
def pool(self) -> asyncpg.Pool:
if self._pool is None:
raise RuntimeError("brain_db not connected — call connect() first")
return self._pool
def _redacted_dsn(self) -> str:
# Hide password in logs
import re
return re.sub(r":([^@:/]+)@", ":***@", self._dsn)
# Singleton — wired up in app lifespan
db = Database(settings.postgres_dsn)

42
ai_platform/modules/didi_brain/brain_api/deps.py Normal file
View file

@ -0,0 +1,42 @@
"""Shared clients that live for the lifetime of the FastAPI process.
We keep one AtomicClient, one EmbeddingClient, and one TagResolver in module
state. FastAPI dependencies pull them out so handlers stay clean.
Initialized from app.py's lifespan context manager at startup; closed on
shutdown.
"""
from __future__ import annotations
from dataclasses import dataclass
from shared.atomic_api import AtomicClient
from shared.embedding_client import EmbeddingClient
from shared.llm_client import LlmClient
from shared.taxonomy import TagResolver
@dataclass(slots=True)
class AppState:
atomic: AtomicClient
embed: EmbeddingClient
llm: LlmClient
resolver: TagResolver
# Module-level singleton, populated by the lifespan handler.
_state: AppState | None = None
def set_state(state: AppState) -> None:
global _state
_state = state
def get_state() -> AppState:
if _state is None:
raise RuntimeError(
"brain_api state not initialized — FastAPI lifespan must run first"
)
return _state

0
ai_platform/modules/didi_brain/brain_api/events/__init__.py Normal file
View file

92
ai_platform/modules/didi_brain/brain_api/events/middleware.py Normal file
View file

@ -0,0 +1,92 @@
"""Per-request middleware that emits events to the dashboard sink.
Captures: request_id (X-Request-ID header or generated UUID), endpoint
(URL path), HTTP method, duration, status_code. Skips /health and noisy
internal paths. Endpoint string is normalized to the route template
(no params) so a high-cardinality table doesn't blow up.
"""
from __future__ import annotations
import time
import uuid
from typing import Awaitable, Callable
from starlette.requests import Request
from starlette.responses import Response
from brain_api.events.sink import get_global_sink
# Endpoints we don't want to record — too noisy / not interesting in History.
_SKIP_PREFIXES = (
"/health",
"/docs",
"/redoc",
"/openapi.json",
"/metrics",
)
def _normalize_endpoint(path: str) -> str:
"""Collapse path params so we don't blow up the request_history table.
/v1/analysis_atom/123 → /v1/analysis_atom/{id}
/v1/fact_status/45/versions → /v1/fact_status/{id}/versions
/v1/verification_cache/abc/free → /v1/verification_cache/{hash}/{tier}
"""
parts = path.split("/")
if len(parts) >= 4 and parts[1] == "v1" and parts[2] == "analysis_atom":
if len(parts) == 4 and parts[3].isdigit():
return "/v1/analysis_atom/{id}"
if len(parts) >= 4 and parts[1] == "v1" and parts[2] == "fact_status":
if parts[3].isdigit():
tail = "/" + "/".join(parts[4:]) if len(parts) > 4 else ""
return f"/v1/fact_status/{{id}}{tail}"
if (
len(parts) >= 5
and parts[1] == "v1"
and parts[2] == "verification_cache"
and parts[4] in ("free", "premium")
):
return "/v1/verification_cache/{hash}/{tier}"
return path
async def event_emit_middleware(
request: Request,
call_next: Callable[[Request], Awaitable[Response]],
) -> Response:
"""Starlette HTTP middleware: emit a dashboard event per request."""
path = request.url.path
if any(path.startswith(p) for p in _SKIP_PREFIXES):
return await call_next(request)
request_id = request.headers.get("x-request-id") or uuid.uuid4().hex[:32]
started = time.monotonic()
status_code = 500
error: str | None = None
try:
response = await call_next(request)
status_code = response.status_code
return response
except Exception as e: # noqa: BLE001
error = f"{type(e).__name__}: {e}"
raise
finally:
duration_ms = int((time.monotonic() - started) * 1000)
sink = get_global_sink()
if sink is not None and sink.enabled:
sink.emit(
{
"request_id": request_id,
"tier": "n/a",
"endpoint": _normalize_endpoint(path),
# Brain has no upstream "provider" concept — it resolves
# locally (PG + atomic + LLM router). Leaving null keeps
# the column UI honest (renders as "—").
"provider": None,
"duration_ms": duration_ms,
"status_code": status_code,
"error": error,
}
)

125
ai_platform/modules/didi_brain/brain_api/events/sink.py Normal file
View file

@ -0,0 +1,125 @@
"""Async event sink that forwards brain request events to the AI platform dashboard.
Fire-and-forget — dashboard outages must never affect brain latency or
availability. Pattern mirrors web-api/events/sink.py 1:1, with module='brain'
baked in so events are distinguishable in the unified Insights/History view.
"""
from __future__ import annotations
import asyncio
import contextlib
import logging
from typing import Any
import httpx
logger = logging.getLogger("brain_api.events.sink")
class DashboardEventSink:
"""POSTs request events to the dashboard /api/ingest/event endpoint.
The sink runs all writes through a bounded queue processed by a single
worker task. Overflow drops oldest. Failures are logged but swallowed.
"""
def __init__(
self,
dashboard_url: str | None,
token: str | None = None,
queue_size: int = 1000,
request_timeout: float = 5.0,
) -> None:
self.dashboard_url = dashboard_url.rstrip("/") if dashboard_url else None
self.token = token
self._queue: asyncio.Queue[dict[str, Any]] = asyncio.Queue(maxsize=queue_size)
self._client: httpx.AsyncClient | None = None
self._worker_task: asyncio.Task[None] | None = None
self._request_timeout = request_timeout
self._enabled = bool(dashboard_url)
@property
def enabled(self) -> bool:
return self._enabled
async def start(self) -> None:
if not self._enabled:
logger.info("brain DashboardEventSink disabled (no dashboard URL)")
return
self._client = httpx.AsyncClient(
timeout=httpx.Timeout(
connect=2.0, read=self._request_timeout, write=2.0, pool=5.0
),
limits=httpx.Limits(max_connections=5, max_keepalive_connections=2),
)
self._worker_task = asyncio.create_task(self._worker())
logger.info("brain DashboardEventSink started → %s", self.dashboard_url)
async def stop(self) -> None:
if self._worker_task is not None:
self._worker_task.cancel()
with contextlib.suppress(asyncio.CancelledError, Exception):
await self._worker_task
self._worker_task = None
if self._client is not None and not self._client.is_closed:
await self._client.aclose()
self._client = None
def emit(self, event: dict[str, Any]) -> None:
"""Enqueue an event for async send. Never raises.
The caller does NOT have to set 'module' — we stamp it here so all
events from this process are tagged 'brain' regardless of who called.
"""
if not self._enabled:
return
event = {**event, "module": "brain"}
try:
self._queue.put_nowait(event)
except asyncio.QueueFull:
# Drop the oldest event to make room for the new one.
try:
_ = self._queue.get_nowait()
self._queue.put_nowait(event)
except Exception: # noqa: BLE001
pass
async def _worker(self) -> None:
while True:
try:
event = await self._queue.get()
except asyncio.CancelledError:
raise
try:
await self._send(event)
except Exception as e: # noqa: BLE001
logger.debug("Event dropped (%s): %s", type(e).__name__, e)
finally:
self._queue.task_done()
async def _send(self, event: dict[str, Any]) -> None:
if self._client is None or self.dashboard_url is None:
return
headers = {"Content-Type": "application/json"}
if self.token:
headers["Authorization"] = f"Bearer {self.token}"
resp = await self._client.post(
f"{self.dashboard_url}/api/ingest/event",
json=event,
headers=headers,
)
resp.raise_for_status()
# Module-level singleton — initialized in app.py lifespan.
_sink: DashboardEventSink | None = None
def init_global_sink(sink: DashboardEventSink) -> None:
global _sink
_sink = sink
def get_global_sink() -> DashboardEventSink | None:
return _sink

60
ai_platform/modules/didi_brain/brain_api/prompts/canonicalize_v1.md Normal file
View file

@ -0,0 +1,60 @@
You are a temporal disambiguator for a misinformation detection cache. Your job is to take an ambiguous CLAIM and produce a CANONICAL form that anchors all relative time references and underspecified entities to specific values, so the same claim asked at different times produces different cache keys.
# Why this matters
A user asking "Cine câștigă alegerile?" in 2024 and again in 2026 is asking about *different* elections. If both claims hash to the same cache key, the 2024 verdict gets served in 2026, which is wrong. The job of canonicalization is to expand ambiguous references so the cache key reflects what the user actually means *right now*.
# Task
Given a CLAIM and the CURRENT_DATE, output a JSON object with these fields:
1. **canonical** (string, required) — the rewritten claim with:
- Relative time references resolved to absolute references using CURRENT_DATE.
Examples:
- "azi" / "today" → the actual date (e.g., "în 2026-05-04").
- "ieri" / "yesterday" → CURRENT_DATE - 1.
- "săptămâna asta" / "this week" → "în săptămâna {ISO week}".
- "luna trecută" / "last month" → name of the prior month.
- "anul trecut" / "last year" → CURRENT_DATE.year - 1.
- "acum" / "now" / "currently" / "în prezent" → "în {CURRENT_DATE}".
- Underspecified entities expanded with the most contextually plausible disambiguation, only when context allows (DO NOT invent if truly ambiguous).
Examples:
- "alegerile" → "alegerile prezidențiale din [country] din [year]" if the year is implied by current_date and a clear election cycle exists.
- "războiul" → preserve as-is unless context strongly suggests one specific conflict.
- "președintele" → preserve as-is — adding a name would be unsafe inference.
- Original wording preserved as much as possible. Goal is anchor, not rewrite.
- Same language as the input claim (Romanian → Romanian, English → English).
2. **changed** (boolean, required) — true if the canonical form differs meaningfully from the original; false if no temporal/entity disambiguation was needed (claim was already specific).
3. **anchors_added** (list of strings, required, may be empty) — short labels for what was disambiguated, e.g., `["temporal:today", "year:2026"]` or `["entity:alegerile→alegerile_prezidentiale_2026"]`. Used for audit and debugging.
4. **reasoning** (string, max 200 chars) — one-line explanation of any non-trivial decision.
# Rules
1. **NEVER invent facts.** Adding "Trump" to "the president said" is unsafe — leave it ambiguous. The canonical form must remain truthful about what the user asked.
2. **Always anchor relative time markers** when the claim contains them — this is the primary value of canonicalization.
3. **Be conservative with entity expansion.** Only expand when context (the rest of the claim or current date) makes the disambiguation unambiguous.
4. **Preserve the user's intent.** If they wrote "ieri", don't replace it with "May 3rd 2026" verbatim — write something natural like "în data de 2026-05-03 (ieri)" so the meaning is preserved alongside the anchor.
5. **If the claim is already fully specific** (no relative markers, no ambiguous entities), return it as-is with `changed: false`.
6. **Numbers and named entities stay intact.** Do not normalize "9 medalii" to "9 medals" or "România" to "Romania" — those distinctions matter elsewhere in the pipeline (verification_cache.normalize_claim handles textual normalization separately).
# Output format — STRICT
Respond with ONLY this JSON object. No preamble, no markdown fences, no commentary.
```
{
"canonical": "...",
"changed": true,
"anchors_added": ["..."],
"reasoning": "..."
}
```
# Input
CURRENT_DATE: {current_date}
CLAIM: {claim}

60
ai_platform/modules/didi_brain/brain_api/prompts/classifier_v1.md Normal file
View file

@ -0,0 +1,60 @@
You are a temporal volatility classifier supporting a misinformation detection cache. Your job is to assess how quickly a given CLAIM may become outdated, so the system knows how long to trust a cached verification of it.
# Task
Given a CLAIM and the CURRENT_DATE, output a JSON object with these fields:
1. **volatility** (string, required) — how fast can this claim become outdated?
- `volatile`: minutes-to-days. War updates, casualty counts, breaking news, ongoing crisis, current weather, stock prices, sports scores, current officeholders during active election seasons, ongoing legal proceedings.
- `evolving`: days-to-weeks. Government policies, economic indicators, completed-but-recent trials, employment status of public figures, scientific debates, climate negotiations, recent appointments.
- `stable`: months-to-years. Historical facts, settled science, geographical facts, completed events with no further development possible, biographical facts of deceased historical figures, mathematical truths.
2. **topic_codes** (list of strings, required, may be empty) — short codes for the topics this claim involves. Prefer these canonical codes when applicable: `war`, `armed_conflict`, `elections`, `politics`, `health`, `health_outbreak`, `economy`, `economy_indicators`, `climate`, `science`, `sports`, `entertainment`, `crime`, `disaster`, `breaking_news`, `technology`, `education`, `religion`, `culture`. Add free-form codes only if none of these fit.
3. **entity_bindings** (list of objects, required, may be empty) — every `(subject, predicate, object)` triple this claim depends on. For each:
- `subject`: canonical name of the entity (e.g., `"Vladimir Putin"`, `"Romania"`, `"World Health Organization"`).
- `predicate`: short relation name (e.g., `"is_president_of"`, `"won_election_in"`, `"is_alive"`, `"has_population"`, `"happened_on"`, `"is_ceo_of"`, `"defeated"`, `"signed_treaty_with"`).
- `object`: target value (entity, date, number, country, etc.).
- `confidence`: 0.0-1.0 of your extraction certainty.
4. **estimated_validity_hours** (integer, required) — your best estimate of how many hours from now this verdict can be trusted, given current world state. Reasonable bounds:
- volatile: 1-48 hours
- evolving: 24-720 hours (1-30 days)
- stable: 720-26280 hours (1-36 months)
5. **time_sensitive** (boolean, required) — true if the claim contains relative time markers (`today`, `yesterday`, `now`, `currently`, `azi`, `ieri`, `acum`, `în prezent`, `recently`) or specific recent dates that strongly anchor it to a particular moment.
6. **reasoning** (string, max 200 chars) — one-line explanation of your volatility decision.
# Rules
1. **When in doubt, prefer SHORTER validity** — false-fresh is much worse than false-stale (which just means re-verification).
2. **Currently-in-office officials** → volatile regardless of base topic. "X is the prime minister" can change overnight.
3. **Numerical statistics that update** (deaths, cases, GDP, prices) → volatile or evolving, never stable.
4. **Pure historical/geographical facts** ("Bucharest is the capital of Romania", "WW2 ended in 1945", "Mount Everest is the tallest mountain") → stable.
5. **Be aggressive about extracting entity_bindings** — these are how the system tracks fact changes over time. A claim like "X is president of Y" should yield at least one binding `{subject: X, predicate: is_president_of, object: Y}`.
6. **If the claim is vague or unverifiable** ("the situation is bad"), still classify volatility based on the inferred topic. Default to `evolving`.
7. **Do not include topic codes that aren't actually relevant** to the claim — only the directly applicable ones.
# Output format — STRICT
Respond with ONLY this JSON object. No preamble, no markdown fences, no commentary, no thinking-out-loud.
```
{
"volatility": "volatile|evolving|stable",
"topic_codes": ["..."],
"entity_bindings": [
{"subject": "...", "predicate": "...", "object": "...", "confidence": 0.0-1.0}
],
"estimated_validity_hours": 24,
"time_sensitive": false,
"reasoning": "..."
}
```
# Input
CURRENT_DATE: {current_date}
CLAIM: {claim}

34
ai_platform/modules/didi_brain/brain_api/prompts/nli_v1.md Normal file
View file

@ -0,0 +1,34 @@
You are a natural language inference (NLI) classifier supporting a disinformation analysis pipeline. Your job is to decide how a piece of EVIDENCE relates to a specific CLAIM.
Output exactly ONE of these three labels:
- **SUPPORTS** — the evidence provides information that would make a reasonable person believe the claim is true (or more likely true). The evidence directly or strongly indirectly backs the claim.
- **CONTRADICTS** — the evidence provides information that would make a reasonable person believe the claim is false (or less likely true). This includes explicit debunks, scientific consensus against, or facts that are incompatible with the claim.
- **NEUTRAL** — the evidence is related to the same topic but does not clearly support or contradict the claim. Includes tangential context, definitions, unrelated details about the same entities.
# Important rules
1. Focus ONLY on the truth-value relationship, not on the source's credibility or intent.
2. If the evidence describes someone ASSERTING the claim (without the source endorsing it), but the source's overall framing treats the claim as factual, label SUPPORTS. If the source treats it as debunked, label CONTRADICTS.
3. Scientific consensus statements against a claim count as CONTRADICTS (strong).
4. An evidence item that merely mentions the claim topic without a clear truth-direction is NEUTRAL.
5. If the evidence could be read both ways, pick NEUTRAL.
6. "Confidence" reflects how clean the relationship is:
- 0.9-1.0: unambiguous, single-interpretation
- 0.7-0.9: clear but with minor caveats
- 0.5-0.7: probable but could be argued
- <0.5: you are guessing — prefer NEUTRAL
# Output format — STRICT
Respond with ONLY this JSON object. No preamble, no markdown fences, no commentary.
```
{"label": "SUPPORTS|CONTRADICTS|NEUTRAL", "confidence": 0.0-1.0}
```
# Input
CLAIM: {claim}
EVIDENCE: {evidence}

32
ai_platform/modules/didi_brain/brain_api/requirements.txt Normal file
View file

@ -0,0 +1,32 @@
# Runtime dependencies for the brain_api container.
# Pinned to minor versions that match the local dev venv we validated against.
# If you bump one, rebuild the image and re-run scripts/09_brain_api_demo.py.
# Web framework
fastapi>=0.135,<0.140
uvicorn>=0.44,<0.50
# HTTP client (used by every shared client)
httpx>=0.28,<0.30
# Config / validation
pydantic>=2.12,<3.0
pydantic-settings>=2.13,<3.0
python-dotenv>=1.2,<2.0
# Retries
tenacity>=9.1,<10.0
# Logging + pretty console
structlog>=25.5,<26.0
rich>=14.3,<15.0
# PostgreSQL (verification cache)
asyncpg>=0.30,<0.32
# Observability — Prometheus metrics + OpenTelemetry traces
prometheus-fastapi-instrumentator>=7.0,<8.0
opentelemetry-instrumentation-fastapi>=0.50b0
opentelemetry-instrumentation-asyncpg>=0.50b0
opentelemetry-instrumentation-httpx>=0.50b0
opentelemetry-exporter-otlp-proto-grpc>=1.30.0

35
ai_platform/modules/didi_brain/brain_api/run.py Normal file
View file

@ -0,0 +1,35 @@
"""Entry point for the brain_api FastAPI service.
python -m brain_api.run
Bind host defaults to 127.0.0.1 (local dev, loopback only). Containerized
runs must override with BRAIN_API_HOST=0.0.0.0 so Docker port mapping can
actually forward traffic in from the outside.
Environment variables:
BRAIN_API_HOST bind address (default: 127.0.0.1)
BRAIN_API_PORT port (default: 8090)
"""
from __future__ import annotations
import os
import uvicorn
def main() -> None:
host = os.environ.get("BRAIN_API_HOST", "127.0.0.1")
port = int(os.environ.get("BRAIN_API_PORT", "8090"))
uvicorn.run(
"brain_api.app:app",
host=host,
port=port,
reload=False,
log_level="info",
access_log=True,
)
if __name__ == "__main__":
main()

169
ai_platform/modules/didi_brain/brain_api/runtime_config.py Normal file
View file

@ -0,0 +1,169 @@
"""Runtime config client — fetches config overrides from AI platform dashboard.
Polls the dashboard /api/config endpoint periodically and caches values in
memory. Consumers read keys like `brain.atom.silver_ttl_days` and pass their
own fallback (typically the pydantic Settings value).
Pattern mirrors the one used by the embeddings/rerank/catalog modules so the
behaviour is consistent across the AI platform: same poll interval, same
fail-open semantics, same optional auto-apply for log level.
"""
from __future__ import annotations
import asyncio
import contextlib
import logging
from typing import Any
import httpx
logger = logging.getLogger(__name__)
class RuntimeConfigClient:
"""Polls dashboard /api/config and caches values in-process."""
def __init__(
self,
dashboard_url: str | None,
poll_interval_seconds: int = 30,
request_timeout: float = 5.0,
live_log_logger_name: str | None = None,
live_log_key: str | None = None,
) -> None:
self.dashboard_url = dashboard_url.rstrip("/") if dashboard_url else None
self.poll_interval = poll_interval_seconds
self._timeout = request_timeout
self._cache: dict[str, Any] = {}
self._client: httpx.AsyncClient | None = None
self._task: asyncio.Task[None] | None = None
self._enabled = bool(dashboard_url)
self._live_log_logger_name = live_log_logger_name
self._live_log_key = live_log_key
self._last_log_level: str | None = None
@property
def enabled(self) -> bool:
return self._enabled
def get(self, key: str, default: Any = None) -> Any:
v = self._cache.get(key)
return v if v is not None else default
def get_bool(self, key: str, default: bool = False) -> bool:
v = self._cache.get(key)
return bool(v) if v is not None else default
def get_int(self, key: str, default: int = 0) -> int:
v = self._cache.get(key)
if v is None:
return default
try:
return int(v)
except (TypeError, ValueError):
return default
def get_float(self, key: str, default: float = 0.0) -> float:
v = self._cache.get(key)
if v is None:
return default
try:
return float(v)
except (TypeError, ValueError):
return default
def get_str(self, key: str, default: str = "") -> str:
v = self._cache.get(key)
return str(v) if v is not None else default
async def start(self) -> None:
if not self._enabled:
logger.info("RuntimeConfigClient disabled (no dashboard URL)")
return
self._client = httpx.AsyncClient(
timeout=httpx.Timeout(connect=2.0, read=self._timeout, write=2.0, pool=5.0)
)
await self._refresh()
self._task = asyncio.create_task(self._loop())
logger.info(
"RuntimeConfigClient started (polling %s every %ds, %d keys cached)",
self.dashboard_url,
self.poll_interval,
len(self._cache),
)
async def stop(self) -> None:
if self._task is not None:
self._task.cancel()
with contextlib.suppress(asyncio.CancelledError, Exception):
await self._task
self._task = None
if self._client is not None and not self._client.is_closed:
await self._client.aclose()
self._client = None
async def _loop(self) -> None:
while True:
try:
await asyncio.sleep(self.poll_interval)
await self._refresh()
except asyncio.CancelledError:
raise
except Exception as e: # noqa: BLE001
logger.debug("Config poll failed: %s", e)
async def _refresh(self) -> None:
if self._client is None or self.dashboard_url is None:
return
try:
resp = await self._client.get(f"{self.dashboard_url}/api/config")
resp.raise_for_status()
data = resp.json()
except Exception as e: # noqa: BLE001
logger.debug("Config refresh failed: %s", e)
return
items = data.get("items", {})
new_cache: dict[str, Any] = {}
for key, entry in items.items():
new_cache[key] = entry.get("value")
self._cache = new_cache
self._maybe_apply_log_level()
def _maybe_apply_log_level(self) -> None:
if not self._live_log_key or not self._live_log_logger_name:
return
new_level = self.get_str(self._live_log_key)
if not new_level or new_level == self._last_log_level:
return
try:
level_int = logging.getLevelName(new_level.upper())
if isinstance(level_int, int):
logging.getLogger(self._live_log_logger_name).setLevel(level_int)
self._last_log_level = new_level
logger.info(
"Log level for %s changed to %s (via runtime config)",
self._live_log_logger_name,
new_level,
)
except Exception as e: # noqa: BLE001
logger.warning("Failed to apply log level %s: %s", new_level, e)
# Module-level singleton — initialized in app.py lifespan.
_global_client: RuntimeConfigClient | None = None
def init_global_client(client: RuntimeConfigClient) -> None:
global _global_client
_global_client = client
def get_global_client() -> RuntimeConfigClient | None:
"""Returns the live RuntimeConfigClient if started, else None.
Consumers (e.g. analysis_atom service) call this with a fallback so they
work both before lifespan starts and when no dashboard is configured.
"""
return _global_client

880
ai_platform/modules/didi_brain/brain_api/schemas.py Normal file
View file

@ -0,0 +1,880 @@
"""Pydantic v2 schemas — the exact contract Didi's backend parses.
These match the shape documented for the existing web-gathering module
1:1. Any field the backend checks MUST exist in the response. Where we don't
have meaningful data, we populate with safe non-null defaults (empty list,
"Global", "en", etc.) rather than leaving a field out or null.
Additive fields that are DidiBrain-specific live under `brain_meta` blocks so
an unknowing backend ignores them while a newer one can consume them.
"""
from __future__ import annotations
from datetime import datetime
from typing import Any, Literal
from pydantic import BaseModel, ConfigDict, Field
# =============================================================================
# Request models
# =============================================================================
class SearchRequest(BaseModel):
"""POST /v1/search — simple list-style retrieval."""
model_config = ConfigDict(extra="allow")
queries: list[str] = Field(
...,
min_length=1,
description="One or more query strings. Multiple are executed in parallel.",
)
max_results: int = Field(20, ge=1, le=200)
language: str | None = Field(None, description="Hint for detected query language, optional.")
class FetchRequest(BaseModel):
"""POST /v1/fetch — retrieve full text for a list of URLs."""
model_config = ConfigDict(extra="allow")
urls: list[str] = Field(..., min_length=1)
include_html: bool = Field(False, description="Return raw HTML along with extracted text.")
class GatherRequest(BaseModel):
"""POST /v1/gather — full claim-to-evidence pipeline."""
model_config = ConfigDict(extra="allow")
claim: str = Field(..., min_length=3)
max_evidence: int = Field(15, ge=1, le=100)
include_full_text: bool = Field(True)
summarize: bool = Field(True)
score_relevance: bool = Field(True)
language_hint: str | None = Field(None)
# When true, run an independent NLI pass to decide whether each piece of
# evidence supports, contradicts, or is neutral toward the input claim.
# Adds ~2-4 seconds to the gather call; disable when latency matters more
# than classification detail.
run_nli: bool = Field(True)
# ----- Verification cache extension (agreed contract with didi-backend) -----
include_verification: bool = Field(
False,
description=(
"If true, brain attempts to look up cached verification for "
"(claim, evidence_urls, tier) and attach it under brain_meta."
),
)
tier: Literal["free", "premium"] | None = Field(
None,
description="Required when include_verification=true — isolates cache.",
)
prompt_hash: str | None = Field(
None,
description=(
"Current backend prompt hash. Brain marks entries stale when the "
"cached entry's prompt_hash differs (returns nothing)."
),
)
framework_version: str | None = Field(
None,
description=(
"Current backend framework config hash (thresholds). "
"Differs → return verification_raw so backend can recompute."
),
)
# ----- Phase B4: recency-aware retrieval (Pilon 3+4) -------------------
volatility_hint: Literal["volatile", "evolving", "stable"] | None = Field(
None,
description=(
"Caller's hint about how fast this claim's truth can change. "
"Drives recency boost in ranking and a hard recency filter for "
"volatile claims. Absent → mild defaults applied (no aggression)."
),
)
recency_window_days: int | None = Field(
None,
ge=1,
le=365,
description=(
"When volatility_hint='volatile' and this is set, hard-drop any "
"evidence older than N days. Default behavior (None): 7 days for "
"volatile, no cut for evolving/stable."
),
)
class VerificationCacheWriteRequest(BaseModel):
"""POST /v1/verification_cache — fire-and-forget push from didi-backend."""
model_config = ConfigDict(extra="allow")
claim: str = Field(..., min_length=1)
evidence_urls: list[str] = Field(..., min_length=1)
tier: Literal["free", "premium"]
model: str | None = None
prompt_hash: str = Field(..., min_length=4)
framework_version: str | None = None
schema_name: str = Field("didi-v1")
verification_processed: dict = Field(..., description="Opaque blob; stored 1:1.")
verification_raw: dict | None = Field(
None,
description=(
"Optional — lets brain hand this back when thresholds change "
"(stale_framework response), so backend can recompute status."
),
)
class VerificationCacheWriteResponse(BaseModel):
model_config = ConfigDict(extra="allow")
cached: bool
claim_hash: str
evidence_hash: str
tier: str
created_at: datetime
updated_at: datetime
expires_at: datetime
# -----------------------------------------------------------------------------
# Analysis Atom — cache for techniques + ai_tampered LLM results
# Contract: didi-backend agent-v3 reads/writes via /v1/analysis_atom/{lookup,POST,PATCH}
# -----------------------------------------------------------------------------
AtomComponent = Literal["techniques", "ai_tampered", "claims"]
AtomTier = Literal["free", "premium"]
AtomCacheTier = Literal["gold", "silver", "bronze"]
AtomStaleness = Literal["fresh", "stale_prompt", "stale_framework", "miss"]
class AnalysisAtomLookupRequest(BaseModel):
"""POST /v1/analysis_atom/lookup — find a cached LLM analysis result.
Lookup is keyed on (content_hash, component, prompt_hash). Tier is NOT
part of the key — read tier-agnostic so free users benefit from premium
cached entries.
"""
model_config = ConfigDict(extra="allow")
content_hash: str = Field(..., min_length=8, description="sha256 of normalized content")
component: AtomComponent
tier: AtomTier = Field(..., description="Caller's current tier (informational, not used for lookup key)")
prompt_hash: str = Field(..., min_length=4)
framework_version: str | None = None
class AnalysisAtomData(BaseModel):
model_config = ConfigDict(extra="allow")
atom_id: int
content_hash: str
component: AtomComponent
tier: AtomTier
prompt_hash: str
framework_version: str | None
model_used: str | None
cache_tier: AtomCacheTier
human_validated: bool
result_processed: dict
validator_user_id: str | None
validated_at: datetime | None
hit_count: int
created_at: datetime
updated_at: datetime
expires_at: datetime | None
class AnalysisAtomLookupResponse(BaseModel):
model_config = ConfigDict(extra="allow")
hit: bool
atom: AnalysisAtomData | None = None
staleness: AtomStaleness | None = None
match_type: Literal["exact", "semantic"] | None = None
class AnalysisAtomWriteRequest(BaseModel):
"""POST /v1/analysis_atom — fire-and-forget write from backend after LLM run.
Brain rejects writes for tier='free' (premium-only ingest, by design).
cache_tier is computed from llm_confidence (>= threshold → silver, else bronze).
"""
model_config = ConfigDict(extra="allow")
content_hash: str = Field(..., min_length=8)
content_preview: str | None = Field(None, max_length=500)
component: AtomComponent
tier: AtomTier
prompt_hash: str = Field(..., min_length=4)
framework_version: str | None = None
model_used: str | None = None
result_processed: dict = Field(..., description="Canonical mapped result, stored 1:1")
result_raw: dict | None = None
llm_confidence: float | None = Field(None, ge=0, le=100)
# If None (default), cache_tier is decided server-side from llm_confidence.
# Set explicitly only if caller wants to force a specific tier.
cache_tier: AtomCacheTier | None = Field(None, description="Optional override; None = decide from confidence")
class AnalysisAtomWriteResponse(BaseModel):
model_config = ConfigDict(extra="allow")
cached: bool
atom_id: int | None = None
cache_tier: AtomCacheTier | None = None
skipped_reason: str | None = None # e.g. "tier=free" or "confidence_below_threshold"
class AnalysisAtomPatchRequest(BaseModel):
"""PATCH /v1/analysis_atom/{atom_id} — promote to gold after human review."""
model_config = ConfigDict(extra="allow")
human_validated: bool = True
human_corrections: dict | None = None
validator_user_id: str | None = None
result_processed: dict | None = Field(None, description="Updated result after corrections applied")
cache_tier: AtomCacheTier = "gold"
class AnalysisAtomStatsResponse(BaseModel):
model_config = ConfigDict(extra="allow")
total_atoms: int
by_tier: dict # {gold: N, silver: N, bronze: N}
by_component: dict # {techniques: N, ai_tampered: N, claims: N}
hit_rate_24h: float | None = None
writes_24h: int
# -----------------------------------------------------------------------------
# Admin browser models (consumed by AI platform dashboard reskin)
# -----------------------------------------------------------------------------
class AnalysisAtomListItem(BaseModel):
"""Lightweight row for atom list (no result_raw / result_processed)."""
atom_id: int
content_hash: str
content_preview: str | None = None
component: str
tier: str
cache_tier: str
prompt_hash: str
framework_version: str | None = None
model_used: str | None = None
llm_confidence: float | None = None
human_validated: bool
hit_count: int
last_hit_at: datetime | None = None
created_at: datetime
updated_at: datetime
expires_at: datetime | None = None
class AnalysisAtomListResponse(BaseModel):
items: list[AnalysisAtomListItem]
total: int
page: int
page_size: int
class AnalysisAtomDetailResponse(AnalysisAtomListItem):
"""Full atom row including result payload + corrections."""
result_processed: dict
result_raw: dict | None = None
human_corrections: dict | None = None
validator_user_id: str | None = None
validated_at: datetime | None = None
class VerificationCacheListItem(BaseModel):
claim_hash: str
tier: str
model: str | None = None
prompt_hash: str
framework_version: str | None = None
schema_name: str
evidence_url_count: int
status: str | None = None
volatility: str | None = None
topic_codes: list[str] = []
created_at: datetime
updated_at: datetime
expires_at: datetime
class VerificationCacheListResponse(BaseModel):
items: list[VerificationCacheListItem]
total: int
page: int
page_size: int
class VerificationCacheDetailResponse(VerificationCacheListItem):
evidence_urls: list[str]
verification_processed: dict
verification_raw: dict | None = None
class TaxonomyInfoResponse(BaseModel):
total_tags: int
namespaces: list[str]
by_namespace: dict
class TaxonomyReloadResponse(BaseModel):
ok: bool
before: int | None = None
after: int | None = None
fetched: int | None = None
error: str | None = None
class AnalysisAtomStatsExtendedResponse(BaseModel):
model_config = ConfigDict(extra="allow")
total_atoms: int
by_tier: dict
by_component: dict
hit_rate_24h: float | None = None
hits_24h_gold: int
hits_24h_silver: int
writes_24h: int
gold_promotions_24h: int
class GenericOkResponse(BaseModel):
ok: bool
message: str | None = None
class CacheInvalidateRequest(BaseModel):
"""POST /v1/cache/invalidate — Pilon 8 mass invalidation.
At least one filter field must be set (topic_codes, entity_canonicals,
claim_pattern, or since) — empty filter is rejected to avoid accidental
"flush everything". By default gold atoms are spared; pass
``invalidate_gold=true`` to flush them too (only do this in moderator-
initiated flows).
Use ``dry_run=true`` first to count matches without modifying anything.
"""
model_config = ConfigDict(extra="forbid")
topic_codes: list[str] | None = Field(
default=None,
description="Match rows whose topic_codes overlap with this set.",
)
entity_canonicals: list[str] | None = Field(
default=None,
description=(
"Pre-normalized canonical forms ('subject predicate object' "
"lowercased). Caller computes via fact_status.canonicalize_triple."
),
)
claim_pattern: str | None = Field(
default=None,
max_length=200,
description="ILIKE pattern matched against content_preview.",
)
since: str | None = Field(
default=None,
description="ISO datetime — match rows updated at or after this time.",
)
invalidate_gold: bool = Field(
default=False,
description="If true, also expire gold (human-validated) atoms.",
)
dry_run: bool = Field(
default=False,
description="Count matches without modifying anything.",
)
actor: str | None = Field(
default=None,
max_length=120,
description="Audit-log label (e.g., 'admin:foo@bar', 'breaking_watcher').",
)
reason: str | None = Field(
default=None,
max_length=500,
description="Optional human-readable note for audit trail.",
)
class CacheInvalidateResponse(BaseModel):
"""Output of POST /v1/cache/invalidate."""
model_config = ConfigDict(extra="allow")
invalidated_atoms: int
invalidated_vcache: int
dry_run: bool
filters_applied: dict
executed_at: datetime
# =============================================================================
# Fact Status admin schemas (Phase D2)
# =============================================================================
class FactStatusItem(BaseModel):
"""One brain_fact_status row, flattened for admin browser."""
model_config = ConfigDict(extra="allow")
fact_id: int
subject: str
predicate: str
object: str
canonical_form: str
canonical_form_hash: str
current_truth: bool | None = None
current_version_id: int | None = None
current_confidence: float | None = None
last_verified_at: datetime | None = None
last_evidence_urls: list[str] = Field(default_factory=list)
volatility: str | None = None
topic_codes: list[str] = Field(default_factory=list)
next_check_at: datetime
check_interval_hours: int
moderator_locked: bool
moderator_user_id: str | None = None
moderator_notes: str | None = None
created_at: datetime
updated_at: datetime
class FactStatusListResponse(BaseModel):
items: list[FactStatusItem]
total: int
page: int
page_size: int
class FactStatusVersionItem(BaseModel):
"""One brain_fact_version row."""
model_config = ConfigDict(extra="allow")
version_id: int
fact_id: int
truth_value: bool
confidence: float | None = None
valid_from: datetime
valid_to: datetime | None = None
source_atom_ids: list[str] = Field(default_factory=list)
evidence_urls: list[str] = Field(default_factory=list)
llm_reasoning: str | None = None
created_by: str
moderator_user_id: str | None = None
notes: str | None = None
created_at: datetime
class FactStatusVersionsResponse(BaseModel):
versions: list[FactStatusVersionItem]
fact_id: int
total: int
class FactStatusPatchRequest(BaseModel):
"""PATCH /v1/fact_status/{fact_id} — moderator override.
Three orthogonal operations, all optional:
- ``set_truth``: assert TRUE/FALSE as the moderator's verdict
- ``lock``: prevent the auditor from auto-changing this fact
- ``unlock``: re-enable auditor updates
"""
model_config = ConfigDict(extra="forbid")
set_truth: bool | None = Field(
default=None,
description="Set current_truth (TRUE/FALSE). Omit to leave unchanged.",
)
confidence: float | None = Field(
default=None,
ge=0.0,
le=100.0,
description="Moderator confidence in this assertion, 0-100.",
)
evidence_urls: list[str] = Field(default_factory=list)
notes: str | None = Field(default=None, max_length=2000)
lock: bool | None = Field(
default=None,
description=(
"true → moderator_locked=true (auditor must skip). "
"false → unlock. None → leave as-is."
),
)
moderator_user_id: str = Field(
...,
min_length=1,
max_length=120,
description="Required — keycloak_id of the moderator making the change.",
)
class AuditLogItem(BaseModel):
"""One brain_audit_log row."""
model_config = ConfigDict(extra="allow")
log_id: int
action: str
target_table: str
target_id: str
actor: str | None = None
payload: dict
created_at: datetime
class AuditLogResponse(BaseModel):
items: list[AuditLogItem]
total: int
page: int
page_size: int
class CanonicalizeRequest(BaseModel):
"""POST /v1/canonicalize — Pilon 7 temporal disambiguation.
Caller (typically agent-v3) sends the raw user claim plus an optional
explicit current_date (ISO). Brain returns the rewritten claim with
relative time markers and ambiguous entities anchored. The caller hashes
the canonical form for cache lookups.
"""
model_config = ConfigDict(extra="forbid")
claim: str = Field(..., min_length=1, max_length=2000)
current_date: str | None = Field(
default=None,
description=(
"ISO date (YYYY-MM-DD) used as 'now' for relative-marker "
"resolution. Defaults to UTC today."
),
)
class CanonicalizeResponse(BaseModel):
"""Output of POST /v1/canonicalize.
On LLM failure the canonical equals the original and ``error`` is set.
Caller can still proceed (no regression).
"""
model_config = ConfigDict(extra="allow")
canonical: str
original: str
changed: bool
anchors_added: list[str] = Field(default_factory=list)
reasoning: str = ""
error: str | None = None
class ImageSearchRequest(BaseModel):
"""POST /v1/image-search — stub, returns empty list."""
model_config = ConfigDict(extra="allow")
queries: list[str] = Field(..., min_length=1)
max_results: int = Field(20, ge=1, le=200)
class IngestRequest(BaseModel):
"""POST /v1/ingest — populate brain from Didi's web-gathering results.
Body shape is intentionally liberal: we accept either a full GatherResponse
(as emitted by the web module) or a thinned envelope with just evidence[].
"""
model_config = ConfigDict(extra="allow")
claim: str | None = Field(None, description="Original query that produced this evidence.")
evidence: list[EvidenceItem] = Field(default_factory=list)
default_tags: list[str] = Field(
default_factory=list,
description="Canonical tag paths to apply to every ingested atom.",
)
run_extraction: bool = Field(
True,
description="If true, queue claim extraction on the newly ingested documents.",
)
# =============================================================================
# Shared sub-schemas
# =============================================================================
class Provenance(BaseModel):
"""Where this evidence came from and how it was produced."""
model_config = ConfigDict(extra="allow")
extraction_method: str = Field("http", description="http | browse | vision | brain")
fallback_chain: list[str] = Field(default_factory=list)
# Additive DidiBrain-specific metadata — safe to ignore if unknown.
brain_meta: "BrainEvidenceMeta | None" = None
class BrainEvidenceMeta(BaseModel):
"""Additive fields specific to DidiBrain that a consumer MAY use."""
model_config = ConfigDict(extra="allow")
parent_atom_id: str
matching_claim_atom_ids: list[str] = Field(default_factory=list)
best_claim_text: str = ""
best_claim_stance_in_source: str = "NEUTRAL"
best_claim_hash: str = ""
claim_count: int = 0
reranker_score: float = 0.0
embedding_similarity: float = 0.0
# NLI stance of the evidence AGAINST the user's query claim. Populated
# when GatherRequest.run_nli is true (default). Unlike stance_in_source,
# this is the direction the backend actually needs for disinfo verdicts.
stance_vs_query: str = "UNKNOWN" # SUPPORTS / CONTRADICTS / NEUTRAL / UNKNOWN
nli_confidence: float = 0.0 # 0..1
nli_error: str | None = None # populated on timeout / bad response
class Entities(BaseModel):
model_config = ConfigDict(extra="allow")
persons: list[str] = Field(default_factory=list)
institutions: list[str] = Field(default_factory=list)
locations: list[str] = Field(default_factory=list)
class SearchContext(BaseModel):
model_config = ConfigDict(extra="allow")
primary_country: str = "Global"
secondary_countries: list[str] = Field(default_factory=list)
entities: Entities = Field(default_factory=Entities)
detected_language: str = "en"
search_queries: list[str] = Field(default_factory=list)
class SearchResultItem(BaseModel):
model_config = ConfigDict(extra="allow")
query: str
url: str
title: str
snippet: str = ""
rank: int
site: str = ""
published_at: datetime | None = None
class StageRecord(BaseModel):
model_config = ConfigDict(extra="allow")
stage: str
success: bool
items_processed: int = 0
items_failed: int = 0
duration_ms: float = 0.0
error: str | None = None
class EvidenceStats(BaseModel):
model_config = ConfigDict(extra="allow")
input_items: int = 0
after_dedup: int = 0
output_items: int = 0
duplicates_removed: int = 0
tokens_used: int = 0
class EvidenceItem(BaseModel):
"""One piece of evidence — always at the DOCUMENT level, not chunk level."""
model_config = ConfigDict(extra="allow")
url: str
canonical_url: str | None = None
title: str
publisher: str = ""
published_at: datetime | None = None
retrieved_at: datetime
snippet: str | None = None
summary: str | None = None
full_text: str | None = None
full_text_hash: str = ""
provenance: Provenance = Field(default_factory=Provenance)
relevance_score: float = 0.0
credibility_score: float = 0.5
# =============================================================================
# Response models
# =============================================================================
class FailedUrl(BaseModel):
model_config = ConfigDict(extra="allow")
url: str
error: str
class FetchedPage(BaseModel):
model_config = ConfigDict(extra="allow")
url: str
canonical_url: str | None = None
title: str = ""
text: str = ""
text_hash: str = ""
html: str | None = None
extraction_method: str = "brain"
fallback_chain: list[str] = Field(default_factory=list)
published_at: datetime | None = None
retrieved_at: datetime
extraction_time_ms: float = 0.0
warnings: list[str] = Field(default_factory=list)
needs_fallback: bool = False
status_code: int = 200
content_type: str = "text/markdown"
class BrainMeta(BaseModel):
"""Top-level meta about the brain response — additive, safe to ignore."""
model_config = ConfigDict(extra="allow")
cache_status: Literal["HIT", "PARTIAL", "MISS"] = "MISS"
api_version: str = "v1"
implementation: str = "didibrain"
evidence_sources: int = 0
total_claim_atoms_matched: int = 0
# ---- Verification cache (populated only when request included the flag) ----
verification: dict | None = Field(
None,
description=(
"Cached verification payload when (claim, tier) is a HIT. Shape "
"is opaque — returned 1:1 as stored by backend."
),
)
verification_staleness: (
Literal[
"fresh",
"stale_framework",
"stale_prompt",
"stale_evidence", # Pilon 11: bound facts have flipped
"miss",
]
| None
) = None
verification_model: str | None = None
verification_tier: str | None = None
verification_prompt_hash: str | None = None
verification_framework_version: str | None = None
verification_cached_at: datetime | None = None
verification_expires_at: datetime | None = None
# URLs the cache was written for — surfaced so backend can detect corpus
# drift and decide whether the cached verification still applies to the
# current evidence set (e.g. compute URL overlap %).
verification_evidence_urls: list[str] | None = None
verification_evidence_hash: str | None = None
# Pilon 11: when staleness=stale_evidence, lists each entity binding whose
# current_truth in brain_fact_status contradicts what the cache assumed.
# Each item: {subject, predicate, object, canonical_form, cached_assumes,
# current_truth}.
verification_facts_invalidated: list[dict] | None = None
class SearchResponse(BaseModel):
model_config = ConfigDict(extra="allow")
request_id: str
results: list[SearchResultItem] = Field(default_factory=list)
total_results: int = 0
execution_time_ms: float = 0.0
queries_processed: int = 0
brain_meta: BrainMeta | None = None
class FetchResponse(BaseModel):
model_config = ConfigDict(extra="allow")
request_id: str
pages: list[FetchedPage] = Field(default_factory=list)
total_fetched: int = 0
total_failed: int = 0
execution_time_ms: float = 0.0
failed_urls: list[FailedUrl] = Field(default_factory=list)
brain_meta: BrainMeta | None = None
class GatherResponse(BaseModel):
model_config = ConfigDict(extra="allow")
request_id: str
claim: str
evidence: list[EvidenceItem] = Field(default_factory=list)
evidence_stats: EvidenceStats = Field(default_factory=EvidenceStats)
search_context: SearchContext = Field(default_factory=SearchContext)
search_results: list[SearchResultItem] = Field(default_factory=list)
stages: list[StageRecord] = Field(default_factory=list)
total_urls_found: int = 0
total_pages_fetched: int = 0
total_evidence_items: int = 0
execution_time_ms: float = 0.0
brain_meta: BrainMeta | None = None
class ImageSearchResponse(BaseModel):
model_config = ConfigDict(extra="allow")
request_id: str
results: list[Any] = Field(default_factory=list)
total_results: int = 0
execution_time_ms: float = 0.0
queries_processed: int = 0
brain_meta: BrainMeta | None = None
class IngestResponse(BaseModel):
model_config = ConfigDict(extra="allow")
request_id: str
accepted: int = 0
skipped_duplicate: int = 0
errors: int = 0
created_atom_ids: list[str] = Field(default_factory=list)
extraction_queued: bool = False
execution_time_ms: float = 0.0
warnings: list[str] = Field(default_factory=list)
# Rebuild forward-ref
Provenance.model_rebuild()

0
ai_platform/modules/didi_brain/brain_api/services/__init__.py Normal file
View file

532
ai_platform/modules/didi_brain/brain_api/services/admin.py Normal file
View file

@ -0,0 +1,532 @@
"""Admin operations for Brain — used by AI platform dashboard.
Provides:
- paginated atom + verification cache browsers with filters
- manual expire (atom) / hard delete (verification cache)
- taxonomy snapshot + reload trigger
Read paths return lightweight rows (no result_raw, no full evidence_urls
arrays) to keep DataGrid responses small. Detail endpoints expose the full
JSON payload.
"""
from __future__ import annotations
import json
from dataclasses import dataclass
from datetime import datetime, timezone
from typing import Any, Literal
from brain_api.db import db
from shared.logging import get_logger
log = get_logger(__name__)
# ============================================================================
# Atom admin
# ============================================================================
@dataclass(slots=True)
class AtomListRow:
atom_id: int
content_hash: str
content_preview: str | None
component: str
tier: str
cache_tier: str
prompt_hash: str
framework_version: str | None
model_used: str | None
llm_confidence: float | None
human_validated: bool
hit_count: int
last_hit_at: datetime | None
created_at: datetime
updated_at: datetime
expires_at: datetime | None
@dataclass(slots=True)
class AtomDetailRow(AtomListRow):
result_processed: dict
result_raw: dict | None
human_corrections: dict | None
validator_user_id: str | None
validated_at: datetime | None
@dataclass(slots=True)
class AtomListPage:
items: list[AtomListRow]
total: int
page: int
page_size: int
_FRESHNESS_OPTIONS = ("fresh", "expiring", "expired", "all")
def _atom_list_filters(
component: str | None,
tier: str | None,
freshness: str | None,
q: str | None,
) -> tuple[str, list]:
"""Build WHERE clause + params (1-indexed)."""
where = ["1=1"]
params: list[Any] = []
idx = 1
if component and component != "all":
where.append(f"component = ${idx}")
params.append(component)
idx += 1
if tier and tier != "all":
# 'tier' here = cache_tier (gold/silver/bronze) — that's what users want to filter by
where.append(f"cache_tier = ${idx}")
params.append(tier)
idx += 1
# Freshness on expires_at:
# fresh = not yet expired (expires_at IS NULL or > now())
# expiring = expires within 7 days
# expired = expires_at <= now()
if freshness == "fresh":
where.append("(expires_at IS NULL OR expires_at > now())")
elif freshness == "expiring":
where.append("expires_at IS NOT NULL AND expires_at > now() AND expires_at <= now() + interval '7 days'")
elif freshness == "expired":
where.append("expires_at IS NOT NULL AND expires_at <= now()")
# else "all" or None → no filter
if q:
where.append(f"(content_preview ILIKE ${idx} OR content_hash ILIKE ${idx})")
params.append(f"%{q}%")
idx += 1
return " AND ".join(where), params
async def list_atoms(
*,
component: str | None = None,
tier: str | None = None,
freshness: str | None = None,
q: str | None = None,
page: int = 1,
page_size: int = 25,
) -> AtomListPage:
page = max(1, page)
page_size = min(max(1, page_size), 100)
offset = (page - 1) * page_size
where_sql, params = _atom_list_filters(component, tier, freshness, q)
count_sql = f"SELECT COUNT(*) AS n FROM brain_analysis_atom WHERE {where_sql}"
list_sql = f"""
SELECT atom_id, content_hash, content_preview, component, tier, cache_tier,
prompt_hash, framework_version, model_used, llm_confidence,
human_validated, hit_count, last_hit_at,
created_at, updated_at, expires_at
FROM brain_analysis_atom
WHERE {where_sql}
ORDER BY (cache_tier = 'gold') DESC, updated_at DESC
LIMIT ${len(params)+1} OFFSET ${len(params)+2}
"""
async with db.pool.acquire() as conn:
count_row = await conn.fetchrow(count_sql, *params)
rows = await conn.fetch(list_sql, *params, page_size, offset)
items = [
AtomListRow(
atom_id=r["atom_id"],
content_hash=r["content_hash"],
content_preview=r["content_preview"],
component=r["component"],
tier=r["tier"],
cache_tier=r["cache_tier"],
prompt_hash=r["prompt_hash"],
framework_version=r["framework_version"],
model_used=r["model_used"],
llm_confidence=float(r["llm_confidence"]) if r["llm_confidence"] is not None else None,
human_validated=r["human_validated"],
hit_count=r["hit_count"] or 0,
last_hit_at=r["last_hit_at"],
created_at=r["created_at"],
updated_at=r["updated_at"],
expires_at=r["expires_at"],
)
for r in rows
]
return AtomListPage(
items=items,
total=count_row["n"] or 0,
page=page,
page_size=page_size,
)
async def get_atom(atom_id: int) -> AtomDetailRow | None:
sql = """
SELECT atom_id, content_hash, content_preview, component, tier, cache_tier,
prompt_hash, framework_version, model_used, llm_confidence,
human_validated, hit_count, last_hit_at,
created_at, updated_at, expires_at,
result_processed, result_raw, human_corrections,
validator_user_id, validated_at
FROM brain_analysis_atom
WHERE atom_id = $1
"""
async with db.pool.acquire() as conn:
r = await conn.fetchrow(sql, atom_id)
if not r:
return None
rp = r["result_processed"]
rr = r["result_raw"]
hc = r["human_corrections"]
if isinstance(rp, str):
rp = json.loads(rp)
if isinstance(rr, str):
rr = json.loads(rr)
if isinstance(hc, str):
hc = json.loads(hc)
return AtomDetailRow(
atom_id=r["atom_id"],
content_hash=r["content_hash"],
content_preview=r["content_preview"],
component=r["component"],
tier=r["tier"],
cache_tier=r["cache_tier"],
prompt_hash=r["prompt_hash"],
framework_version=r["framework_version"],
model_used=r["model_used"],
llm_confidence=float(r["llm_confidence"]) if r["llm_confidence"] is not None else None,
human_validated=r["human_validated"],
hit_count=r["hit_count"] or 0,
last_hit_at=r["last_hit_at"],
created_at=r["created_at"],
updated_at=r["updated_at"],
expires_at=r["expires_at"],
result_processed=rp or {},
result_raw=rr,
human_corrections=hc,
validator_user_id=r["validator_user_id"],
validated_at=r["validated_at"],
)
async def expire_atom(atom_id: int) -> bool:
"""Mark atom as expired immediately (soft delete — keeps row for audit).
Returns True if a row was updated, False if not found.
"""
sql = """
UPDATE brain_analysis_atom
SET expires_at = now(), updated_at = now()
WHERE atom_id = $1
"""
async with db.pool.acquire() as conn:
result = await conn.execute(sql, atom_id)
# asyncpg execute() returns string like "UPDATE 1"
return result.endswith("1")
# ============================================================================
# Verification cache admin
# ============================================================================
@dataclass(slots=True)
class VerificationListRow:
claim_hash: str
tier: str
model: str | None
prompt_hash: str
framework_version: str | None
schema_name: str
evidence_url_count: int
created_at: datetime
updated_at: datetime
expires_at: datetime
status: str | None # extracted from verification_processed.status if present
volatility: str | None
topic_codes: list[str]
@dataclass(slots=True)
class VerificationDetailRow(VerificationListRow):
evidence_urls: list[str]
verification_processed: dict
verification_raw: dict | None
@dataclass(slots=True)
class VerificationListPage:
items: list[VerificationListRow]
total: int
page: int
page_size: int
def _verif_filters(
tier: str | None,
q: str | None,
) -> tuple[str, list]:
where = ["1=1"]
params: list[Any] = []
idx = 1
if tier and tier != "all":
where.append(f"tier = ${idx}")
params.append(tier)
idx += 1
if q:
# Search on claim_hash prefix or model name
where.append(f"(claim_hash ILIKE ${idx} OR model ILIKE ${idx})")
params.append(f"%{q}%")
idx += 1
return " AND ".join(where), params
async def list_verifications(
*,
tier: str | None = None,
q: str | None = None,
page: int = 1,
page_size: int = 25,
) -> VerificationListPage:
page = max(1, page)
page_size = min(max(1, page_size), 100)
offset = (page - 1) * page_size
where_sql, params = _verif_filters(tier, q)
count_sql = f"SELECT COUNT(*) AS n FROM brain_verification_cache WHERE {where_sql}"
list_sql = f"""
SELECT claim_hash, tier, model, prompt_hash, framework_version, schema_name,
jsonb_array_length(evidence_urls) AS evidence_url_count,
verification_processed,
volatility, topic_codes,
created_at, updated_at, expires_at
FROM brain_verification_cache
WHERE {where_sql}
ORDER BY updated_at DESC
LIMIT ${len(params)+1} OFFSET ${len(params)+2}
"""
async with db.pool.acquire() as conn:
count_row = await conn.fetchrow(count_sql, *params)
rows = await conn.fetch(list_sql, *params, page_size, offset)
items: list[VerificationListRow] = []
for r in rows:
vp = r["verification_processed"]
if isinstance(vp, str):
try:
vp = json.loads(vp)
except Exception:
vp = {}
status = vp.get("status") if isinstance(vp, dict) else None
items.append(
VerificationListRow(
claim_hash=r["claim_hash"],
tier=r["tier"],
model=r["model"],
prompt_hash=r["prompt_hash"],
framework_version=r["framework_version"],
schema_name=r["schema_name"],
evidence_url_count=r["evidence_url_count"] or 0,
created_at=r["created_at"],
updated_at=r["updated_at"],
expires_at=r["expires_at"],
status=status,
volatility=r["volatility"],
topic_codes=list(r["topic_codes"]) if r["topic_codes"] else [],
)
)
return VerificationListPage(
items=items,
total=count_row["n"] or 0,
page=page,
page_size=page_size,
)
async def get_verification(
claim_hash: str, tier: Literal["free", "premium"]
) -> VerificationDetailRow | None:
sql = """
SELECT claim_hash, tier, evidence_hash, evidence_urls, model, prompt_hash,
framework_version, schema_name, verification_processed, verification_raw,
volatility, topic_codes,
created_at, updated_at, expires_at
FROM brain_verification_cache
WHERE claim_hash = $1 AND tier = $2
"""
async with db.pool.acquire() as conn:
r = await conn.fetchrow(sql, claim_hash, tier)
if not r:
return None
ev = r["evidence_urls"]
vp = r["verification_processed"]
vr = r["verification_raw"]
if isinstance(ev, str):
ev = json.loads(ev)
if isinstance(vp, str):
vp = json.loads(vp)
if isinstance(vr, str):
vr = json.loads(vr)
status = vp.get("status") if isinstance(vp, dict) else None
return VerificationDetailRow(
claim_hash=r["claim_hash"],
tier=r["tier"],
model=r["model"],
prompt_hash=r["prompt_hash"],
framework_version=r["framework_version"],
schema_name=r["schema_name"],
evidence_url_count=len(ev) if ev else 0,
created_at=r["created_at"],
updated_at=r["updated_at"],
expires_at=r["expires_at"],
status=status,
volatility=r["volatility"],
topic_codes=list(r["topic_codes"]) if r["topic_codes"] else [],
evidence_urls=list(ev) if ev else [],
verification_processed=vp or {},
verification_raw=vr,
)
async def delete_verification(
claim_hash: str, tier: Literal["free", "premium"]
) -> bool:
sql = "DELETE FROM brain_verification_cache WHERE claim_hash = $1 AND tier = $2"
async with db.pool.acquire() as conn:
result = await conn.execute(sql, claim_hash, tier)
return result.endswith("1")
# ============================================================================
# Taxonomy admin
# ============================================================================
async def get_taxonomy_info(resolver) -> dict:
"""Snapshot of currently loaded taxonomy.
`resolver.all` is a dict {path: tag_id}, so iterating it yields the path
strings directly (e.g. "Country", "Country/France", "Topics/Health/COVID").
"""
all_paths: list[str] = list(resolver.all) if hasattr(resolver, "all") else []
by_namespace: dict[str, int] = {}
for path in all_paths:
ns = path.split("/", 1)[0] if "/" in path else (path or "(root)")
by_namespace[ns] = by_namespace.get(ns, 0) + 1
return {
"total_tags": len(all_paths),
"by_namespace": by_namespace,
"namespaces": sorted(by_namespace.keys()),
}
async def reload_taxonomy(resolver, atomic) -> dict:
"""Re-fetch tags from atomic and replace the in-process resolver state.
Returns a small status dict.
"""
from shared.taxonomy import build_path_map_from_tags
try:
live_tags = await atomic.list_tags()
path_map = build_path_map_from_tags(live_tags)
before = len(resolver.all) if hasattr(resolver, "all") else 0
if path_map:
resolver.load_from_mapping(path_map)
after = len(resolver.all) if hasattr(resolver, "all") else 0
return {
"ok": True,
"before": before,
"after": after,
"fetched": len(path_map),
}
except Exception as e: # noqa: BLE001
log.exception("taxonomy_reload_failed")
return {"ok": False, "error": f"{type(e).__name__}: {e}"}
# ============================================================================
# Extended stats (existing /v1/analysis_atom/stats kept; this returns more)
# ============================================================================
async def get_stats_extended() -> dict:
"""Extended stats: per-tier hit rates, top components, recent activity."""
sql = """
WITH base AS (
SELECT
cache_tier,
component,
hit_count,
last_hit_at,
created_at,
validated_at,
human_validated
FROM brain_analysis_atom
)
SELECT
COUNT(*) AS total,
COUNT(*) FILTER (WHERE cache_tier='gold') AS gold,
COUNT(*) FILTER (WHERE cache_tier='silver') AS silver,
COUNT(*) FILTER (WHERE cache_tier='bronze') AS bronze,
COUNT(*) FILTER (WHERE component='techniques') AS c_tech,
COUNT(*) FILTER (WHERE component='ai_tampered') AS c_ai,
COUNT(*) FILTER (WHERE component='claims') AS c_claims,
COUNT(*) FILTER (WHERE created_at > now() - interval '24 hours') AS writes_24h,
SUM(hit_count) FILTER (WHERE last_hit_at > now() - interval '24 hours') AS hits_24h,
SUM(hit_count) FILTER (WHERE cache_tier='gold' AND last_hit_at > now() - interval '24 hours') AS hits_24h_gold,
SUM(hit_count) FILTER (WHERE cache_tier='silver' AND last_hit_at > now() - interval '24 hours') AS hits_24h_silver,
-- Count promotions by when the moderator validated, not when the atom was first written.
COUNT(*) FILTER (WHERE human_validated = true AND validated_at > now() - interval '24 hours') AS gold_promotions_24h
FROM base
"""
async with db.pool.acquire() as conn:
row = await conn.fetchrow(sql)
total = row["total"] or 0
hits_24h = int(row["hits_24h"] or 0)
writes_24h = int(row["writes_24h"] or 0)
hits_24h_gold = int(row["hits_24h_gold"] or 0)
hits_24h_silver = int(row["hits_24h_silver"] or 0)
denom = hits_24h + writes_24h
hit_rate_24h = (hits_24h / max(denom, 1)) if denom > 0 else None
return {
"total_atoms": total,
"by_tier": {
"gold": row["gold"] or 0,
"silver": row["silver"] or 0,
"bronze": row["bronze"] or 0,
},
"by_component": {
"techniques": row["c_tech"] or 0,
"ai_tampered": row["c_ai"] or 0,
"claims": row["c_claims"] or 0,
},
"hit_rate_24h": hit_rate_24h,
"hits_24h_gold": hits_24h_gold,
"hits_24h_silver": hits_24h_silver,
"writes_24h": writes_24h,
"gold_promotions_24h": int(row["gold_promotions_24h"] or 0),
}

761
ai_platform/modules/didi_brain/brain_api/services/analysis_atom.py Normal file
View file

@ -0,0 +1,761 @@
"""Analysis atom cache — store full-component LLM results for techniques/ai_tampered/claims.
Contract with didi-backend agent-v3:
- Before LLM run, agent-v3 calls POST /v1/analysis_atom/lookup with
(content_hash, component, prompt_hash). On gold or silver+fresh hit,
backend skips LLM and uses cached result.
- After LLM run (only if tier='premium'), agent-v3 fires POST /v1/analysis_atom
to cache the result. cache_tier is silver by default; bronze if llm_confidence
is below the configured threshold.
- After moderator resolves a session with corrections, agent-v3 calls PATCH
/v1/analysis_atom/{atom_id} with human_validated=true to promote silver→gold.
Storage rules:
- Lookup key: (content_hash, component, prompt_hash) — tier excluded so
free users benefit from premium cached entries.
- Write: rejected if tier='free' (only premium runs ingest).
- Bronze atoms (low confidence) are stored for audit but NEVER served on
lookup. They can be promoted to silver if a future run produces higher
confidence on the same content.
- Gold atoms have expires_at=NULL (forever). Silver/bronze get TTL.
"""
from __future__ import annotations
import asyncio
import hashlib
import json
from dataclasses import dataclass, field
from datetime import datetime, timedelta, timezone
from typing import Any, Literal
from brain_api.db import db
from brain_api.services.classifier import (
ClaimVolatility,
classify_claim_volatility,
)
from shared.llm_client import LlmClient
from shared.logging import get_logger
log = get_logger(__name__)
# ----- legacy fallback TTLs when no classification is available -----
# These are now the *startup defaults* — at runtime they can be overridden
# live by the AI platform dashboard via RuntimeConfigClient (keys
# `brain.atom.silver_ttl_days`, `brain.atom.bronze_ttl_days`,
# `brain.atom.confidence_silver_threshold`). The helpers below read live
# values with these as fallback. Direct constant access is kept for tests
# and code paths that don't yet plumb through Settings.
SILVER_TTL_HOURS = 90 * 24 # 2160h, ~3 months
BRONZE_TTL_HOURS = 30 * 24 # 720h, ~1 month
DEFAULT_CONFIDENCE_THRESHOLD = 60.0 # below → bronze, at/above → silver
def _live_silver_ttl_hours() -> int:
"""Read silver TTL from runtime config (or fall back to settings/constant).
Priority: dashboard live value > Settings default > constant. Settings is
cached so this is essentially free; runtime_config falls back gracefully
if the dashboard is unreachable.
"""
from brain_api.runtime_config import get_global_client
from shared.config import get_settings
rc = get_global_client()
fallback_days = get_settings().atom_silver_ttl_days
if rc is not None and rc.enabled:
return rc.get_int("brain.atom.silver_ttl_days", fallback_days) * 24
return fallback_days * 24
def _live_bronze_ttl_hours() -> int:
from brain_api.runtime_config import get_global_client
from shared.config import get_settings
rc = get_global_client()
fallback_days = get_settings().atom_bronze_ttl_days
if rc is not None and rc.enabled:
return rc.get_int("brain.atom.bronze_ttl_days", fallback_days) * 24
return fallback_days * 24
def _live_confidence_threshold() -> float:
from brain_api.runtime_config import get_global_client
from shared.config import get_settings
rc = get_global_client()
fallback = get_settings().atom_confidence_silver_threshold
if rc is not None and rc.enabled:
return rc.get_float("brain.atom.confidence_silver_threshold", fallback)
return fallback
AtomComponent = Literal["techniques", "ai_tampered", "claims"]
AtomTier = Literal["free", "premium"]
AtomCacheTier = Literal["gold", "silver", "bronze"]
StalenessStatus = Literal["fresh", "stale_framework", "stale_prompt", "miss"]
# --------------------------------------------------------------------------- DTO
@dataclass(slots=True)
class AtomEntry:
atom_id: int
content_hash: str
component: str
tier: str
prompt_hash: str
framework_version: str | None
model_used: str | None
cache_tier: str
human_validated: bool
human_corrections: dict | None
validator_user_id: str | None
validated_at: datetime | None
result_processed: dict
result_raw: dict | None
llm_confidence: float | None
hit_count: int
last_hit_at: datetime | None
created_at: datetime
updated_at: datetime
expires_at: datetime | None
content_preview: str | None = None
# --------------------------------------------------------------------- helpers
def normalize_content_hash(content_hash: str) -> str:
"""Backend computes content_hash already; this is just a passthrough/sanity check.
Convention: backend sends sha256(text)[:16] or sha256(text). We store as-is.
"""
return content_hash.strip().lower()
def _decide_cache_tier(llm_confidence: float | None, override: str | None) -> str:
if override in ("gold", "silver", "bronze"):
return override
if llm_confidence is None:
# No confidence info → assume good enough (silver)
return "silver"
threshold = _live_confidence_threshold()
return "silver" if llm_confidence >= threshold else "bronze"
def _resolve_ttl_hours(
*,
cache_tier: str,
classification: ClaimVolatility | None,
) -> int | None:
"""Pick the effective TTL in hours, combining cache_tier + classification.
Rules:
- gold → None (forever, regardless of classification)
- silver/bronze with classification → use classifier estimate (already
capped per volatility tier in classifier.py)
- silver/bronze without classification → fall back to legacy fixed TTLs
The classification's estimate is *already* clamped to per-tier hard caps
(volatile≤48h, evolving≤720h, stable≤26280h) by the classifier, so we
just trust it here.
"""
if cache_tier == "gold":
return None
if classification is not None and not classification.degraded:
return classification.estimated_validity_hours
return _live_silver_ttl_hours() if cache_tier == "silver" else _live_bronze_ttl_hours()
def _expires_at_from_hours(ttl_hours: int | None) -> datetime | None:
"""Convert TTL hours → expires_at timestamptz. None → no expiry (gold)."""
if ttl_hours is None:
return None
return datetime.now(tz=timezone.utc) + timedelta(hours=ttl_hours)
async def _get_or_compute_classification(
*,
classification: ClaimVolatility | None,
llm: LlmClient | None,
content_preview: str | None,
) -> ClaimVolatility | None:
"""Use caller-provided classification, else compute via LLM if possible.
Returns None if neither path is available — caller falls back to legacy
behavior (no volatility, fixed TTL).
"""
if classification is not None:
return classification
if llm is None or not content_preview:
return None
try:
return await classify_claim_volatility(llm, claim=content_preview)
except Exception as e: # noqa: BLE001
log.warning(
"atom_upsert_classifier_failed",
error=f"{type(e).__name__}:{e}",
)
return None
async def _register_facts_async(
classification: ClaimVolatility | None,
*,
source_atom_id: int,
) -> None:
"""Best-effort fact registration after a successful upsert.
Imports lazily to avoid a circular import (fact_status imports classifier
types). Errors are swallowed — fact registration is enrichment, not core.
"""
if classification is None or not classification.entity_bindings:
return
try:
# Local import: services.fact_status imports classifier types, so
# importing it at module load would create a cycle.
from brain_api.services.fact_status import register_facts_from_bindings
await register_facts_from_bindings(
classification.entity_bindings,
volatility=classification.volatility,
topic_codes=classification.topic_codes,
source_atom_id=str(source_atom_id),
)
except Exception as e: # noqa: BLE001
log.warning(
"fact_registration_failed",
atom_id=source_atom_id,
error=f"{type(e).__name__}:{e}",
)
def _row_to_entry(row) -> AtomEntry:
rp = row["result_processed"]
rr = row["result_raw"]
hc = row["human_corrections"]
if isinstance(rp, str):
rp = json.loads(rp)
if isinstance(rr, str):
rr = json.loads(rr)
if isinstance(hc, str):
hc = json.loads(hc)
return AtomEntry(
atom_id=row["atom_id"],
content_hash=row["content_hash"],
component=row["component"],
tier=row["tier"],
prompt_hash=row["prompt_hash"],
framework_version=row["framework_version"],
model_used=row["model_used"],
cache_tier=row["cache_tier"],
human_validated=row["human_validated"],
human_corrections=hc,
validator_user_id=row["validator_user_id"],
validated_at=row["validated_at"],
result_processed=rp or {},
result_raw=rr,
llm_confidence=float(row["llm_confidence"]) if row["llm_confidence"] is not None else None,
hit_count=row["hit_count"],
last_hit_at=row["last_hit_at"],
created_at=row["created_at"],
updated_at=row["updated_at"],
expires_at=row["expires_at"],
content_preview=row.get("content_preview") if hasattr(row, "get") else None,
)
def decide_freshness(
entry: AtomEntry | None,
current_prompt_hash: str | None,
current_framework_version: str | None,
) -> StalenessStatus:
"""fresh | stale_prompt | stale_framework | miss.
Gold atoms are ALWAYS fresh — human-validated answers don't depend on prompt.
"""
if entry is None:
return "miss"
if entry.cache_tier == "gold":
return "fresh"
if current_prompt_hash and entry.prompt_hash != current_prompt_hash:
return "stale_prompt"
if (
current_framework_version
and entry.framework_version
and entry.framework_version != current_framework_version
):
return "stale_framework"
return "fresh"
# ------------------------------------------------------------------------- IO
async def lookup(
*,
content_hash: str,
component: AtomComponent,
prompt_hash: str,
framework_version: str | None = None,
) -> tuple[AtomEntry | None, StalenessStatus]:
"""Lookup an atom by (content_hash, component) — tier-agnostic.
Returns (entry_or_None, staleness). Bronze atoms are filtered out (NEVER
served). For multiple matches with different prompt_hash, prefers gold.
"""
ch = normalize_content_hash(content_hash)
sql = """
SELECT atom_id, content_hash, component, tier, prompt_hash, framework_version,
model_used, cache_tier, human_validated, human_corrections, validator_user_id,
validated_at, result_processed, result_raw, llm_confidence, hit_count,
last_hit_at, created_at, updated_at, expires_at, content_preview
FROM brain_analysis_atom
WHERE content_hash = $1
AND component = $2
AND cache_tier IN ('gold', 'silver')
AND (expires_at IS NULL OR expires_at > now())
ORDER BY (cache_tier = 'gold') DESC, updated_at DESC
LIMIT 1
"""
async with db.pool.acquire() as conn:
row = await conn.fetchrow(sql, ch, component)
if not row:
return None, "miss"
entry = _row_to_entry(row)
staleness = decide_freshness(entry, prompt_hash, framework_version)
# Increment hit_count (best-effort)
if staleness == "fresh":
try:
async with db.pool.acquire() as conn:
await conn.execute(
"UPDATE brain_analysis_atom SET hit_count = hit_count + 1, last_hit_at = now() WHERE atom_id = $1",
entry.atom_id,
)
except Exception: # noqa: BLE001
log.debug("hit_count_update_skipped", atom_id=entry.atom_id)
return entry, staleness
async def upsert(
*,
content_hash: str,
content_preview: str | None,
component: AtomComponent,
tier: AtomTier,
prompt_hash: str,
framework_version: str | None,
model_used: str | None,
result_processed: dict,
result_raw: dict | None,
llm_confidence: float | None,
cache_tier_override: str | None = None,
classification: ClaimVolatility | None = None,
llm: LlmClient | None = None,
) -> tuple[AtomEntry | None, str | None]:
"""Insert or update an atom row, with volatility classification.
Pipeline:
1. Reject tier='free' silently (premium-only ingest).
2. If no ``classification`` provided and an ``llm`` client is, run the
volatility classifier on ``content_preview`` to derive volatility,
topic_codes, entity_bindings, and the recommended TTL.
3. UPSERT the row with the new metadata columns. Gold rows are
preserved on every soft field (truth-preserving).
4. After successful write, schedule fact_status registration as a
background task (best-effort, errors swallowed).
Args:
content_hash, content_preview, component, tier, prompt_hash,
framework_version, model_used, result_processed, result_raw,
llm_confidence, cache_tier_override: same as before.
classification: Pre-computed ClaimVolatility from caller. If None,
attempts to compute via ``llm``.
llm: LLM client for classifier. Optional — passing None disables
classification (legacy fixed-TTL behavior).
Returns:
``(entry, skip_reason)``. Skip cases (entry=None):
- tier='free' → reject silently (premium-only ingest)
- SQL error → propagated to caller
"""
if tier == "free":
return None, "tier=free (premium-only ingest)"
ch = normalize_content_hash(content_hash)
cache_tier = _decide_cache_tier(llm_confidence, cache_tier_override)
# Volatility classification — caller-provided or LLM-derived.
classification = await _get_or_compute_classification(
classification=classification,
llm=llm,
content_preview=content_preview,
)
ttl_hours = _resolve_ttl_hours(
cache_tier=cache_tier, classification=classification
)
expires_at = _expires_at_from_hours(ttl_hours)
volatility = classification.volatility if classification else None
topic_codes = classification.topic_codes if classification else []
entity_bindings_json = (
json.dumps(classification.entity_bindings_jsonb())
if classification
else "[]"
)
rp_json = json.dumps(result_processed)
rr_json = json.dumps(result_raw) if result_raw is not None else None
sql = """
INSERT INTO brain_analysis_atom (
content_hash, content_preview, component, tier, prompt_hash, framework_version,
model_used, result_processed, result_raw, llm_confidence,
cache_tier, expires_at,
volatility, topic_codes, entity_bindings, ttl_hours_used
)
VALUES (
$1, $2, $3, $4, $5, $6, $7, $8::jsonb, $9::jsonb, $10, $11, $12,
$13, $14, $15::jsonb, $16
)
ON CONFLICT (content_hash, component, prompt_hash) DO UPDATE SET
-- DO NOT downgrade gold to silver — preserve human validation
result_processed = CASE
WHEN brain_analysis_atom.cache_tier = 'gold' THEN brain_analysis_atom.result_processed
ELSE EXCLUDED.result_processed
END,
result_raw = CASE
WHEN brain_analysis_atom.cache_tier = 'gold' THEN brain_analysis_atom.result_raw
ELSE EXCLUDED.result_raw
END,
cache_tier = CASE
WHEN brain_analysis_atom.cache_tier = 'gold' THEN brain_analysis_atom.cache_tier
ELSE EXCLUDED.cache_tier
END,
llm_confidence = CASE
WHEN brain_analysis_atom.cache_tier = 'gold' THEN brain_analysis_atom.llm_confidence
ELSE EXCLUDED.llm_confidence
END,
model_used = COALESCE(EXCLUDED.model_used, brain_analysis_atom.model_used),
framework_version = COALESCE(EXCLUDED.framework_version, brain_analysis_atom.framework_version),
content_preview = COALESCE(EXCLUDED.content_preview, brain_analysis_atom.content_preview),
tier = EXCLUDED.tier,
updated_at = now(),
expires_at = CASE
WHEN brain_analysis_atom.cache_tier = 'gold' THEN brain_analysis_atom.expires_at
ELSE EXCLUDED.expires_at
END,
-- Volatility metadata: prefer fresh values when present (a re-write
-- with classifier may have better data than the original write).
volatility = COALESCE(EXCLUDED.volatility, brain_analysis_atom.volatility),
topic_codes = CASE
WHEN array_length(EXCLUDED.topic_codes, 1) > 0
THEN EXCLUDED.topic_codes
ELSE brain_analysis_atom.topic_codes
END,
entity_bindings = CASE
WHEN jsonb_array_length(EXCLUDED.entity_bindings) > 0
THEN EXCLUDED.entity_bindings
ELSE brain_analysis_atom.entity_bindings
END,
ttl_hours_used = COALESCE(EXCLUDED.ttl_hours_used, brain_analysis_atom.ttl_hours_used)
RETURNING atom_id, content_hash, component, tier, prompt_hash, framework_version,
model_used, cache_tier, human_validated, human_corrections, validator_user_id,
validated_at, result_processed, result_raw, llm_confidence, hit_count,
last_hit_at, created_at, updated_at, expires_at, content_preview
"""
async with db.pool.acquire() as conn:
row = await conn.fetchrow(
sql,
ch, content_preview, component, tier, prompt_hash, framework_version,
model_used, rp_json, rr_json, llm_confidence, cache_tier, expires_at,
volatility, topic_codes, entity_bindings_json, ttl_hours,
)
assert row is not None
entry = _row_to_entry(row)
# Fire-and-forget fact registration (best-effort, never blocks the write).
if classification and classification.entity_bindings:
asyncio.create_task(
_register_facts_async(classification, source_atom_id=entry.atom_id)
)
log.info(
"atom_upsert_ok",
atom_id=entry.atom_id,
component=component,
tier=tier,
cache_tier=entry.cache_tier,
volatility=volatility,
ttl_hours=ttl_hours,
topic_codes=topic_codes,
binding_count=len(classification.entity_bindings) if classification else 0,
)
return entry, None
async def patch_to_gold(
*,
atom_id: int,
human_validated: bool = True,
human_corrections: dict | None = None,
validator_user_id: str | None = None,
result_processed: dict | None = None,
) -> AtomEntry | None:
"""Promote an atom to gold after human review.
If result_processed is provided (corrections applied), it replaces the LLM
result. Otherwise the existing result is kept (e.g. moderator approved as is).
"""
sets = [
"human_validated = $2",
"validator_user_id = $3",
"validated_at = now()",
"cache_tier = 'gold'",
"expires_at = NULL",
"updated_at = now()",
]
params: list[Any] = [atom_id, human_validated, validator_user_id]
next_idx = 4
if human_corrections is not None:
sets.append(f"human_corrections = ${next_idx}::jsonb")
params.append(json.dumps(human_corrections))
next_idx += 1
else:
sets.append("human_corrections = NULL")
if result_processed is not None:
sets.append(f"result_processed = ${next_idx}::jsonb")
params.append(json.dumps(result_processed))
next_idx += 1
sql = f"""
UPDATE brain_analysis_atom
SET {", ".join(sets)}
WHERE atom_id = $1
RETURNING atom_id, content_hash, component, tier, prompt_hash, framework_version,
model_used, cache_tier, human_validated, human_corrections, validator_user_id,
validated_at, result_processed, result_raw, llm_confidence, hit_count,
last_hit_at, created_at, updated_at, expires_at, content_preview
"""
async with db.pool.acquire() as conn:
row = await conn.fetchrow(sql, *params)
if not row:
return None
return _row_to_entry(row)
async def get_stats() -> dict: # noqa: PLR0915 (kept compact; flake later)
return await _get_stats_impl()
# ============================================================================
# Phase B2: confidence decay + judge integration
# ============================================================================
# These helpers run on cache HITS to decide whether the cached verdict is
# still trustworthy. They do not own the lookup query itself — callers
# (gather.py, the lookup endpoint, the daily auditor) call ``lookup()`` first,
# then optionally call ``judge_and_update`` if they have fresh evidence.
# Decay half-lives (hours) per volatility tier. Beyond half-life, the cached
# confidence is halved; at 2× half-life, quartered; etc. Stable claims do
# not decay.
DECAY_HALF_LIVES_HOURS: dict[str, float] = {
"volatile": 24.0, # ~half confidence after 1 day
"evolving": 168.0, # ~half after 1 week
"stable": float("inf"),
}
# Cap on audit_history length kept on each row — older entries are trimmed.
AUDIT_HISTORY_MAX = 50
# Minimum hours between audit-pass increments triggered by lookup judges.
# Without this, popular content hits the auditor 100×/day and consecutive_-
# audit_passes rockets, defeating the purpose. The daily auditor cron is
# the authoritative source of audit passes; lookups only nudge.
AUDIT_PASS_MIN_INTERVAL_HOURS = 6.0
# Confidence floor below which a "fresh" cache entry is treated as miss.
EFFECTIVE_CONFIDENCE_FLOOR = 60.0
def compute_effective_confidence(
*,
base_confidence: float | None,
volatility: str | None,
age_hours: float,
consecutive_audit_passes: int = 0,
) -> float | None:
"""Decay base confidence by age, modulated by volatility and audit history.
Stable rows do not decay. Volatile/evolving rows lose confidence with
exponential half-life. Atoms that survived many audits get a multiplier
boost (max +30% over base).
Returns:
Decayed confidence value, or None if base was None.
"""
if base_confidence is None:
return None
half = DECAY_HALF_LIVES_HOURS.get(volatility or "evolving", 168.0)
if half == float("inf") or age_hours <= 0:
decay = 1.0
else:
# Exponential decay: each half-life halves the confidence.
decay = 0.5 ** (age_hours / half)
audit_boost = min(0.3, 0.03 * max(0, consecutive_audit_passes))
return float(base_confidence) * decay * (1.0 + audit_boost)
def is_effectively_fresh(
*,
base_confidence: float | None,
volatility: str | None,
age_hours: float,
consecutive_audit_passes: int = 0,
floor: float = EFFECTIVE_CONFIDENCE_FLOOR,
) -> bool:
"""True if the decayed confidence is above the freshness floor.
Callers can use this *in addition* to ``decide_freshness`` to drop
entries that are technically not stale but have decayed below usable
confidence.
"""
eff = compute_effective_confidence(
base_confidence=base_confidence,
volatility=volatility,
age_hours=age_hours,
consecutive_audit_passes=consecutive_audit_passes,
)
if eff is None:
# No base confidence stored → trust the freshness flag from the SQL
# path; we have no other signal.
return True
return eff >= floor
async def apply_judge_verdict(
atom_id: int,
verdict: object, # JudgeVerdict — typed loosely to avoid circular import
) -> None:
"""Persist a JudgeVerdict to brain_analysis_atom.
Updates audit_history (append, cap at AUDIT_HISTORY_MAX), last_audited_at,
consecutive_audit_passes (incremented only when KEEP_CACHE and last
increment was >AUDIT_PASS_MIN_INTERVAL_HOURS ago), and expires_at on
INVALIDATE (sets to now() so the row is treated as expired).
Also writes a brain_audit_log row for global telemetry.
"""
if not db.pool:
raise RuntimeError("brain_db not connected")
# Lazy import to avoid circular: cache_judge ← nli ← (transitively) us.
from brain_api.services.cache_judge import JudgeVerdict
if not isinstance(verdict, JudgeVerdict):
raise TypeError(
f"apply_judge_verdict: expected JudgeVerdict, got {type(verdict).__name__}"
)
audit_entry = verdict.to_audit_entry()
audit_json = json.dumps(audit_entry)
sql = """
UPDATE brain_analysis_atom
SET
audit_history = (
-- Append new entry, then keep only the last AUDIT_HISTORY_MAX.
SELECT jsonb_agg(elem)
FROM (
SELECT elem
FROM jsonb_array_elements(
COALESCE(audit_history, '[]'::jsonb) || $2::jsonb
) WITH ORDINALITY AS t(elem, ord)
ORDER BY ord DESC
LIMIT $3
) recent
),
last_audited_at = now(),
consecutive_audit_passes = CASE
WHEN $4 = 'KEEP_CACHE' AND (
last_audited_at IS NULL
OR last_audited_at < now() - ($5 || ' hours')::interval
)
THEN consecutive_audit_passes + 1
WHEN $4 = 'INVALIDATE' THEN 0
ELSE consecutive_audit_passes
END,
expires_at = CASE
WHEN $4 = 'INVALIDATE' AND cache_tier <> 'gold' THEN now()
ELSE expires_at
END,
updated_at = now()
WHERE atom_id = $1
"""
async with db.pool.acquire() as conn:
await conn.execute(
sql,
atom_id,
json.dumps([audit_entry]), # wrap as JSONB array for concat
AUDIT_HISTORY_MAX,
verdict.decision,
str(int(AUDIT_PASS_MIN_INTERVAL_HOURS)),
)
# Audit log entry for cross-table telemetry / dashboards.
await conn.execute(
"""
INSERT INTO brain_audit_log (action, target_table, target_id, actor, payload)
VALUES ($1, 'brain_analysis_atom', $2, 'cache_judge', $3::jsonb)
""",
f"judge_{verdict.decision.lower()}",
str(atom_id),
audit_json,
)
async def _get_stats_impl() -> dict:
"""Return aggregated counts for monitoring."""
sql = """
SELECT
COUNT(*) AS total_atoms,
COUNT(*) FILTER (WHERE cache_tier = 'gold') AS gold,
COUNT(*) FILTER (WHERE cache_tier = 'silver') AS silver,
COUNT(*) FILTER (WHERE cache_tier = 'bronze') AS bronze,
COUNT(*) FILTER (WHERE component = 'techniques') AS c_techniques,
COUNT(*) FILTER (WHERE component = 'ai_tampered') AS c_ai_tampered,
COUNT(*) FILTER (WHERE component = 'claims') AS c_claims,
COUNT(*) FILTER (WHERE created_at > now() - interval '24 hours') AS writes_24h,
SUM(hit_count) FILTER (WHERE last_hit_at > now() - interval '24 hours') AS hits_24h
FROM brain_analysis_atom
"""
async with db.pool.acquire() as conn:
row = await conn.fetchrow(sql)
total = row["total_atoms"] or 0
hits_24h = int(row["hits_24h"] or 0)
writes_24h = int(row["writes_24h"] or 0)
hit_rate = None
if hits_24h + writes_24h > 0:
hit_rate = hits_24h / max(hits_24h + writes_24h, 1)
return {
"total_atoms": total,
"by_tier": {
"gold": row["gold"] or 0,
"silver": row["silver"] or 0,
"bronze": row["bronze"] or 0,
},
"by_component": {
"techniques": row["c_techniques"] or 0,
"ai_tampered": row["c_ai_tampered"] or 0,
"claims": row["c_claims"] or 0,
},
"hit_rate_24h": hit_rate,
"writes_24h": writes_24h,
}

315
ai_platform/modules/didi_brain/brain_api/services/cache_judge.py Normal file
View file

@ -0,0 +1,315 @@
"""Cache Judge — Pilon 2 of the cache freshness defense.
On every cache hit (verification_cache or analysis_atom), the judge runs NLI
between the cached truth direction and current top-K fresh evidence. If fresh
sources contradict the cached verdict, the cache is invalidated and the
caller is forced to recompute with current data.
Decision rules:
- **KEEP_CACHE** — fresh evidence supports the cached verdict (or volatility
is stable and age is below threshold, where we skip NLI entirely).
- **INVALIDATE** — fresh evidence contradicts the cached verdict above the
threshold; caller must treat this as a miss and recompute.
- **NEEDS_FULL_RECHECK** — evidence is mostly neutral or split; caller may
still serve the cache but should mark it as low-confidence.
Cheap path: stable claims younger than ``STABLE_NLI_SKIP_HOURS`` skip NLI
entirely (no LLM call) — pure cache hit.
Confidence boost: rows that passed many consecutive audits get a higher
contradiction threshold (we trust them more). A row that survived 10 daily
audits requires more contradicting evidence to invalidate than a fresh write.
Audit logging: callers should append the JudgeVerdict to brain_analysis_atom.
audit_history (or brain_verification_cache.audit_history) so we have a
running record of why a cache was kept or invalidated.
"""
from __future__ import annotations
from dataclasses import dataclass, field
from datetime import datetime, timezone
from typing import Literal
from brain_api.services.nli import NliResult, classify_batch
from shared.llm_client import LlmClient
from shared.logging import get_logger
log = get_logger(__name__)
Decision = Literal["KEEP_CACHE", "INVALIDATE", "NEEDS_FULL_RECHECK"]
CachedTruth = Literal["TRUE", "FALSE", "MIXED", "UNVERIFIED"]
# Skip NLI entirely for stable claims younger than this. Pure cache hit, zero
# LLM cost. Stable + old → still run NLI (rare facts can change).
STABLE_NLI_SKIP_HOURS = 720.0 # 30 days
# Truncate evidence text before sending to NLI — matches nli.MAX_EVIDENCE_CHARS.
MAX_EVIDENCE_CHARS = 1500
MAX_EVIDENCE_PIECES = 3 # only judge against top-3 fresh sources
# Decision thresholds. These are the fractions of NLI calls that determine
# the verdict.
#
# When cached truth is TRUE:
# contradicts_fraction >= INVALIDATE_THRESHOLD → INVALIDATE
# supports_fraction >= KEEP_THRESHOLD → KEEP_CACHE
# else → NEEDS_FULL_RECHECK
#
# When cached truth is FALSE: roles flip — supports invalidates, contradicts keeps.
INVALIDATE_THRESHOLD = 0.50 # 50% disagreeing → invalidate
KEEP_THRESHOLD = 0.50 # 50% agreeing → keep
MIN_CONFIDENCE = 0.50 # NLI calls below this confidence are ignored
# Confidence boost from consecutive audit passes — each pass nudges the
# invalidate threshold up by this much (so trusted atoms are harder to flip).
AUDIT_PASS_BONUS = 0.03 # +3% per pass
MAX_AUDIT_BONUS = 0.30 # cap at +30%
@dataclass(slots=True, frozen=True)
class EvidenceSnippet:
"""A single piece of fresh evidence to judge cached verdicts against.
Attributes:
url: Canonical source URL (used in audit log).
text: The text excerpt from the source. Will be truncated to
``MAX_EVIDENCE_CHARS`` before NLI.
published_at: When the source was published (ISO string or None).
Used by callers to filter out stale evidence before passing here.
"""
url: str
text: str
published_at: str | None = None
@dataclass(slots=True)
class JudgeVerdict:
"""Output of the cache judge — drives KEEP/INVALIDATE/RECHECK decision.
Attributes:
decision: One of "KEEP_CACHE", "INVALIDATE", "NEEDS_FULL_RECHECK".
nli_results: Per-evidence NLI labels (for audit_history).
nli_skipped: True if we took the cheap path and skipped LLM entirely.
supports_fraction: Fraction of high-confidence NLI calls labelled SUPPORTS.
contradicts_fraction: Fraction labelled CONTRADICTS.
neutral_fraction: Fraction labelled NEUTRAL (or low-confidence).
effective_invalidate_threshold: Threshold actually applied (after audit bonus).
reasoning: Short human-readable explanation, suitable for audit_history.
evaluated_at: When the judgment was made (UTC ISO).
"""
decision: Decision
nli_results: list[NliResult] = field(default_factory=list)
nli_skipped: bool = False
supports_fraction: float = 0.0
contradicts_fraction: float = 0.0
neutral_fraction: float = 0.0
effective_invalidate_threshold: float = INVALIDATE_THRESHOLD
reasoning: str = ""
evaluated_at: str = ""
def to_audit_entry(self) -> dict[str, object]:
"""Serialize for appending to brain_analysis_atom.audit_history."""
return {
"evaluated_at": self.evaluated_at,
"decision": self.decision,
"nli_skipped": self.nli_skipped,
"supports": round(self.supports_fraction, 3),
"contradicts": round(self.contradicts_fraction, 3),
"neutral": round(self.neutral_fraction, 3),
"threshold": round(self.effective_invalidate_threshold, 3),
"reasoning": self.reasoning[:200],
"evidence_count": len(self.nli_results),
}
def _compute_invalidate_threshold(consecutive_audit_passes: int) -> float:
"""Audit-pass bonus: trusted atoms are harder to invalidate.
Each consecutive daily audit that judged KEEP_CACHE increments the
threshold so a single contradicting source can't overturn an atom that's
been stable for weeks.
"""
bonus = min(
MAX_AUDIT_BONUS,
AUDIT_PASS_BONUS * max(0, consecutive_audit_passes),
)
return min(0.95, INVALIDATE_THRESHOLD + bonus)
def _aggregate_nli(results: list[NliResult]) -> tuple[float, float, float]:
"""Compute (supports, contradicts, neutral) fractions over high-confidence calls.
Low-confidence (< MIN_CONFIDENCE) and errored calls count as NEUTRAL — we
don't want noisy signals to invalidate cache.
"""
if not results:
return 0.0, 0.0, 1.0
supports = 0
contradicts = 0
neutral = 0
for r in results:
if r.error or r.confidence < MIN_CONFIDENCE:
neutral += 1
elif r.label == "SUPPORTS":
supports += 1
elif r.label == "CONTRADICTS":
contradicts += 1
else:
neutral += 1
total = float(len(results))
return supports / total, contradicts / total, neutral / total
def _decide_for_truth_direction(
*,
cached_truth: CachedTruth,
supports_fraction: float,
contradicts_fraction: float,
invalidate_threshold: float,
) -> tuple[Decision, str]:
"""Map NLI aggregates to KEEP/INVALIDATE/RECHECK based on cached truth direction."""
if cached_truth == "TRUE":
# We expect SUPPORTS. CONTRADICTS is the danger signal.
if contradicts_fraction >= invalidate_threshold:
return "INVALIDATE", (
f"cached=TRUE but {contradicts_fraction:.0%} of fresh evidence "
f"contradicts (threshold {invalidate_threshold:.0%})"
)
if supports_fraction >= KEEP_THRESHOLD:
return "KEEP_CACHE", (
f"cached=TRUE confirmed by {supports_fraction:.0%} fresh evidence"
)
return "NEEDS_FULL_RECHECK", (
f"cached=TRUE but evidence is split: "
f"{supports_fraction:.0%}/{contradicts_fraction:.0%}"
)
if cached_truth == "FALSE":
# We expect CONTRADICTS. SUPPORTS is the danger signal (claim now true).
if supports_fraction >= invalidate_threshold:
return "INVALIDATE", (
f"cached=FALSE but {supports_fraction:.0%} of fresh evidence "
f"supports (threshold {invalidate_threshold:.0%})"
)
if contradicts_fraction >= KEEP_THRESHOLD:
return "KEEP_CACHE", (
f"cached=FALSE confirmed by {contradicts_fraction:.0%} fresh evidence"
)
return "NEEDS_FULL_RECHECK", (
f"cached=FALSE but evidence is split: "
f"{supports_fraction:.0%}/{contradicts_fraction:.0%}"
)
# MIXED / UNVERIFIED — caller couldn't decide originally either; if fresh
# evidence is now decisive in either direction, force a full recheck so
# a stronger verdict can be issued.
if supports_fraction >= KEEP_THRESHOLD or contradicts_fraction >= KEEP_THRESHOLD:
return "NEEDS_FULL_RECHECK", (
f"cached={cached_truth} but fresh evidence has shifted "
f"({supports_fraction:.0%}/{contradicts_fraction:.0%})"
)
return "KEEP_CACHE", (
f"cached={cached_truth}, fresh evidence still inconclusive"
)
async def judge_cache_validity(
llm: LlmClient,
*,
claim: str,
cached_truth: CachedTruth,
current_evidence: list[EvidenceSnippet],
volatility: str,
age_hours: float,
consecutive_audit_passes: int = 0,
) -> JudgeVerdict:
"""Decide whether a cached verdict still holds against current evidence.
Args:
llm: LLM client (used by NLI).
claim: The original claim text — what the cache was written for.
cached_truth: The truth direction the cache claims (TRUE/FALSE/MIXED/UNVERIFIED).
current_evidence: Top-K fresh evidence snippets from /v1/gather. Caller
should already have applied recency filtering for volatile topics.
volatility: One of "volatile", "evolving", "stable" — controls the
cheap-path skip and influences logging.
age_hours: How old the cache row is (for cheap-path eligibility).
consecutive_audit_passes: How many prior audits the cache survived.
Increases invalidation resistance.
Returns:
JudgeVerdict — never raises. On NLI failure, individual evidence calls
return NEUTRAL with error set; aggregation handles it gracefully.
"""
now_iso = datetime.now(tz=timezone.utc).isoformat()
# Cheap path: stable + young → trust the cache without LLM.
if volatility == "stable" and age_hours < STABLE_NLI_SKIP_HOURS:
return JudgeVerdict(
decision="KEEP_CACHE",
nli_skipped=True,
reasoning=(
f"stable + age {age_hours:.0f}h < {STABLE_NLI_SKIP_HOURS:.0f}h "
f"(cheap path)"
),
evaluated_at=now_iso,
)
# No fresh evidence to check against → can't make a decision; let the
# caller treat as a recheck so they go and gather some.
if not current_evidence:
return JudgeVerdict(
decision="NEEDS_FULL_RECHECK",
reasoning="no fresh evidence available to judge against",
evaluated_at=now_iso,
)
# Truncate + cap evidence count.
snippets = current_evidence[:MAX_EVIDENCE_PIECES]
evidence_texts = [s.text[:MAX_EVIDENCE_CHARS] for s in snippets]
nli_results = await classify_batch(
llm,
claim=claim,
evidence_texts=evidence_texts,
)
supports, contradicts, neutral = _aggregate_nli(nli_results)
threshold = _compute_invalidate_threshold(consecutive_audit_passes)
decision, reasoning = _decide_for_truth_direction(
cached_truth=cached_truth,
supports_fraction=supports,
contradicts_fraction=contradicts,
invalidate_threshold=threshold,
)
log.info(
"cache_judge_done",
decision=decision,
cached_truth=cached_truth,
volatility=volatility,
age_hours=round(age_hours, 1),
supports=round(supports, 2),
contradicts=round(contradicts, 2),
neutral=round(neutral, 2),
threshold=round(threshold, 2),
audit_passes=consecutive_audit_passes,
)
return JudgeVerdict(
decision=decision,
nli_results=nli_results,
nli_skipped=False,
supports_fraction=supports,
contradicts_fraction=contradicts,
neutral_fraction=neutral,
effective_invalidate_threshold=threshold,
reasoning=reasoning,
evaluated_at=now_iso,
)

189
ai_platform/modules/didi_brain/brain_api/services/canonicalizer.py Normal file
View file

@ -0,0 +1,189 @@
"""Temporal canonicalizer — Pilon 7 of the cache freshness defense.
Resolves relative time markers ("azi", "today", "săptămâna asta") and
underspecified entities ("alegerile") in a claim against the current date,
so the same surface text asked at different times produces different cache
keys. This prevents the most insidious form of cache staleness: a claim
phrased identically in 2024 and 2026 silently serving the 2024 verdict.
Pipeline position:
1. agent-v3 receives a user claim
2. agent-v3 calls /v1/canonicalize with claim + current_date
3. agent-v3 hashes the *canonical* form (not the original) for cache lookups
4. brain receives the same canonical form on subsequent identical-text
requests, but only if the same time horizon yields the same canonical
Failure mode: if LLM fails or returns invalid JSON, we return the original
claim verbatim with ``changed=false``. The cache then behaves as today
(no temporal disambiguation, but no regression either).
"""
from __future__ import annotations
import asyncio
from dataclasses import dataclass
from datetime import datetime, timezone
from pathlib import Path
from shared.config import LlmRole
from shared.llm_client import LlmClient, LlmError
from shared.logging import get_logger
log = get_logger(__name__)
PROMPT_VERSION = "v1"
_PROMPT_PATH = (
Path(__file__).resolve().parent.parent
/ "prompts"
/ f"canonicalize_{PROMPT_VERSION}.md"
)
PER_CALL_TIMEOUT_S = 15.0
MAX_CLAIM_CHARS = 2000
_PROMPT_TEMPLATE: str | None = None
def _load_prompt() -> str:
global _PROMPT_TEMPLATE
if _PROMPT_TEMPLATE is None:
_PROMPT_TEMPLATE = _PROMPT_PATH.read_text(encoding="utf-8")
return _PROMPT_TEMPLATE
@dataclass(slots=True, frozen=True)
class Canonicalization:
"""Result of canonicalize_claim_temporal.
Attributes:
canonical: The rewritten claim with anchors applied.
original: The input claim, verbatim (for audit).
changed: Whether canonical differs meaningfully from original.
anchors_added: Short labels of what was disambiguated.
reasoning: One-line LLM justification.
error: Populated if LLM failed and we fell back to original.
"""
canonical: str
original: str
changed: bool
anchors_added: list[str]
reasoning: str
error: str | None = None
@property
def degraded(self) -> bool:
return self.error is not None
def _passthrough(claim: str, error: str | None = None) -> Canonicalization:
"""Build a no-op canonicalization (claim unchanged)."""
return Canonicalization(
canonical=claim,
original=claim,
changed=False,
anchors_added=[],
reasoning="passthrough" if error is None else "fallback_passthrough",
error=error,
)
def _parse_response(data: object, original: str) -> Canonicalization | None:
"""Validate the LLM JSON response. Returns None on bad shape."""
if not isinstance(data, dict):
return None
canonical = str(data.get("canonical", "")).strip()
if not canonical:
return None
changed = bool(data.get("changed", False))
anchors_raw = data.get("anchors_added") or []
if not isinstance(anchors_raw, list):
return None
anchors = [str(a).strip() for a in anchors_raw if str(a).strip()]
reasoning = str(data.get("reasoning", "")).strip()[:200]
return Canonicalization(
canonical=canonical,
original=original,
changed=changed,
anchors_added=anchors,
reasoning=reasoning,
)
async def canonicalize_claim_temporal(
llm: LlmClient,
*,
claim: str,
current_date: datetime | None = None,
) -> Canonicalization:
"""Resolve temporal markers and ambiguous entities in a claim.
On any failure, returns a passthrough Canonicalization (original=canonical,
changed=False) with ``error`` populated. The caller can log telemetry but
the cache lookup proceeds with the original text — no regression.
Args:
llm: LLM client (uses REASONING role).
claim: User claim, possibly containing relative time markers.
current_date: Reference "now". Defaults to UTC now.
Returns:
Canonicalization with the rewritten claim or a passthrough on failure.
"""
if not claim or not claim.strip():
return _passthrough(claim, error="empty_claim")
truncated = claim.strip()[:MAX_CLAIM_CHARS]
today = (current_date or datetime.now(tz=timezone.utc)).date().isoformat()
prompt = (
_load_prompt()
.replace("{current_date}", today)
.replace("{claim}", truncated)
)
try:
result, _usage = await asyncio.wait_for(
llm.chat_json(
role=LlmRole.REASONING,
system=(
"You are a temporal disambiguator. Respond with strictly "
"valid JSON only, no commentary."
),
user=prompt,
max_tokens=400,
temperature=0.0,
),
timeout=PER_CALL_TIMEOUT_S,
)
except asyncio.TimeoutError:
log.warning("canonicalize_timeout", claim_preview=truncated[:80])
return _passthrough(truncated, error="timeout")
except LlmError as e:
log.warning("canonicalize_llm_error", error=str(e)[:200])
return _passthrough(truncated, error=f"llm:{e}")
except Exception as e: # noqa: BLE001
log.warning(
"canonicalize_unexpected_error",
error=f"{type(e).__name__}:{e}",
)
return _passthrough(truncated, error=f"{type(e).__name__}:{e}")
parsed = _parse_response(result, truncated)
if parsed is None:
log.warning("canonicalize_bad_response_shape", got=type(result).__name__)
return _passthrough(truncated, error="bad_response_shape")
if parsed.changed:
log.info(
"canonicalize_anchored",
anchors=parsed.anchors_added,
preview_in=truncated[:80],
preview_out=parsed.canonical[:80],
)
return parsed

350
ai_platform/modules/didi_brain/brain_api/services/classifier.py Normal file
View file

@ -0,0 +1,350 @@
"""Volatility classifier — Pilon 1 of the cache freshness defense.
Single LLM call returns the temporal characteristics of a claim:
- how fast it can become outdated (volatility: volatile|evolving|stable)
- which topics it touches (topic_codes)
- which entity-predicate-object triples it binds to (entity_bindings)
- how many hours from now its verification can be trusted
This is invoked BEFORE writing to brain_analysis_atom or brain_verification_cache,
so the resulting metadata becomes part of the cache row and drives:
- TTL (expires_at = now() + estimated_validity_hours, capped per tier)
- audit scheduling (volatile rows get audited daily by didibrain-auditor)
- mass invalidation by topic (didibrain-breaking-watcher)
- fact-status registration (entity_bindings → brain_fact_status, Pilon 11)
Failure mode: if the LLM call fails or returns invalid JSON, we degrade
gracefully to a conservative fallback (volatility="evolving",
estimated_validity_hours=168) with `error` populated. The caller still gets a
usable classification and the cache write proceeds. The auditor picks up
non-stable rows on its next sweep and corrects misclassifications over time.
"""
from __future__ import annotations
import asyncio
from dataclasses import dataclass
from datetime import datetime, timezone
from pathlib import Path
from typing import Literal
from shared.config import LlmRole
from shared.llm_client import LlmClient, LlmError
from shared.logging import get_logger
log = get_logger(__name__)
PROMPT_VERSION = "v1"
_PROMPT_PATH = (
Path(__file__).resolve().parent.parent
/ "prompts"
/ f"classifier_{PROMPT_VERSION}.md"
)
ALLOWED_VOLATILITY: set[str] = {"volatile", "evolving", "stable"}
# Hard caps (hours) per volatility level — LLM estimate is clamped to these
# upper bounds. Even if the LLM says "this is stable for 5 years", we cap.
HARD_CAPS_HOURS: dict[str, int] = {
"volatile": 48, # max 2 days
"evolving": 720, # max 30 days
"stable": 26280, # max ~3 years
}
# Sensible floors — clamp from below so we never get a 0-hour TTL.
HARD_FLOORS_HOURS: dict[str, int] = {
"volatile": 1,
"evolving": 24,
"stable": 720,
}
# Conservative defaults applied when classification fails.
DEFAULT_VOLATILITY: Literal["volatile", "evolving", "stable"] = "evolving"
DEFAULT_VALIDITY_HOURS = 168 # 7 days
PER_CALL_TIMEOUT_S = 20.0
MAX_CLAIM_CHARS = 2000
_PROMPT_TEMPLATE: str | None = None
def _load_prompt() -> str:
"""Load and cache the classifier prompt template."""
global _PROMPT_TEMPLATE
if _PROMPT_TEMPLATE is None:
_PROMPT_TEMPLATE = _PROMPT_PATH.read_text(encoding="utf-8")
return _PROMPT_TEMPLATE
@dataclass(slots=True, frozen=True)
class EntityBinding:
"""A single (subject, predicate, object) triple extracted from a claim.
Attributes:
subject: Canonical name of the entity (e.g., "Vladimir Putin").
predicate: Short relation name (e.g., "is_president_of").
obj: Target value of the relation. Named ``obj`` instead of ``object``
to avoid shadowing the Python builtin in callers.
confidence: LLM's certainty about this extraction, 0.0-1.0.
"""
subject: str
predicate: str
obj: str
confidence: float
def to_dict(self) -> dict[str, object]:
"""Serialize for JSONB storage in brain_fact_status."""
return {
"subject": self.subject,
"predicate": self.predicate,
"object": self.obj,
"confidence": self.confidence,
}
@dataclass(slots=True, frozen=True)
class ClaimVolatility:
"""Output of the volatility classifier — drives all caching decisions.
Attributes:
volatility: One of "volatile", "evolving", "stable".
topic_codes: List of topic identifiers this claim touches.
entity_bindings: Subject-predicate-object triples to track in
brain_fact_status for temporal versioning.
estimated_validity_hours: How many hours the cached verdict can be
trusted (already clamped by per-tier caps and floors).
time_sensitive: True if the claim has relative time markers.
reasoning: One-line LLM explanation of the volatility decision.
error: Populated only when classification fell back to defaults.
"""
volatility: Literal["volatile", "evolving", "stable"]
topic_codes: list[str]
entity_bindings: list[EntityBinding]
estimated_validity_hours: int
time_sensitive: bool
reasoning: str
error: str | None = None
@property
def degraded(self) -> bool:
"""True if classification fell back to defaults (LLM failed)."""
return self.error is not None
def entity_bindings_jsonb(self) -> list[dict[str, object]]:
"""Serialize entity_bindings for JSONB storage."""
return [b.to_dict() for b in self.entity_bindings]
def _conservative_default(error_msg: str) -> ClaimVolatility:
"""Build a safe-default classification on LLM failure.
The auditor will pick this up on its next sweep (since volatility is
"evolving", not "stable") and may correct it.
"""
return ClaimVolatility(
volatility=DEFAULT_VOLATILITY,
topic_codes=[],
entity_bindings=[],
estimated_validity_hours=DEFAULT_VALIDITY_HOURS,
time_sensitive=False,
reasoning="classifier_fallback",
error=error_msg,
)
def _clamp_validity_hours(raw: int, volatility: str) -> int:
"""Apply hard caps + floors per volatility tier."""
cap = HARD_CAPS_HOURS.get(volatility, DEFAULT_VALIDITY_HOURS)
floor = HARD_FLOORS_HOURS.get(volatility, 1)
try:
value = int(raw)
except (TypeError, ValueError):
value = DEFAULT_VALIDITY_HOURS
return max(floor, min(value, cap))
def _parse_response(data: object) -> ClaimVolatility | None:
"""Validate the LLM JSON response. Returns None on bad shape.
Strictly checks volatility label, list types, and binding structure.
Silently drops malformed entity_bindings rather than rejecting the whole
response.
"""
if not isinstance(data, dict):
return None
raw_vol = (data.get("volatility") or "").strip().lower()
if raw_vol not in ALLOWED_VOLATILITY:
return None
topic_codes_raw = data.get("topic_codes") or []
if not isinstance(topic_codes_raw, list):
return None
topic_codes = [
str(t).strip() for t in topic_codes_raw if str(t).strip()
]
bindings_raw = data.get("entity_bindings") or []
if not isinstance(bindings_raw, list):
return None
bindings: list[EntityBinding] = []
for item in bindings_raw:
if not isinstance(item, dict):
continue
subj = str(item.get("subject", "")).strip()
pred = str(item.get("predicate", "")).strip()
obj_ = str(item.get("object", "")).strip()
try:
conf = float(item.get("confidence", 0.5))
except (TypeError, ValueError):
conf = 0.5
if not (subj and pred and obj_):
continue
bindings.append(
EntityBinding(
subject=subj,
predicate=pred,
obj=obj_,
confidence=max(0.0, min(1.0, conf)),
)
)
validity = _clamp_validity_hours(
data.get("estimated_validity_hours", DEFAULT_VALIDITY_HOURS),
raw_vol,
)
time_sensitive = bool(data.get("time_sensitive", False))
reasoning = str(data.get("reasoning", "")).strip()[:200]
return ClaimVolatility(
volatility=raw_vol, # type: ignore[arg-type]
topic_codes=topic_codes,
entity_bindings=bindings,
estimated_validity_hours=validity,
time_sensitive=time_sensitive,
reasoning=reasoning,
)
async def classify_claim_volatility(
llm: LlmClient,
*,
claim: str,
current_date: datetime | None = None,
) -> ClaimVolatility:
"""Classify a claim's temporal characteristics with one LLM call.
Always returns a ClaimVolatility. On any failure (timeout, LLM error,
bad JSON), returns a conservative default with ``error`` populated so the
caller can log telemetry but still proceed with the cache write.
Args:
llm: Configured LLM client. Uses LlmRole.REASONING internally.
claim: The claim text to classify. Truncated at ``MAX_CLAIM_CHARS``.
current_date: The "now" reference for the classifier (used for
relative time resolution). Defaults to UTC now.
Returns:
ClaimVolatility with the parsed classification, or a conservative
fallback (volatility="evolving", validity=168h) on failure.
"""
if not claim or not claim.strip():
return _conservative_default("empty_claim")
truncated = claim.strip()[:MAX_CLAIM_CHARS]
today = (current_date or datetime.now(tz=timezone.utc)).date().isoformat()
prompt = (
_load_prompt()
.replace("{current_date}", today)
.replace("{claim}", truncated)
)
try:
result, _usage = await asyncio.wait_for(
llm.chat_json(
role=LlmRole.REASONING,
system=(
"You are a temporal volatility classifier. Respond with "
"strictly valid JSON only, no commentary."
),
user=prompt,
max_tokens=600,
temperature=0.0,
),
timeout=PER_CALL_TIMEOUT_S,
)
except asyncio.TimeoutError:
log.warning("classifier_timeout", claim_preview=truncated[:80])
return _conservative_default("timeout")
except LlmError as e:
log.warning("classifier_llm_error", error=str(e)[:200])
return _conservative_default(f"llm:{e}")
except Exception as e: # noqa: BLE001
log.warning(
"classifier_unexpected_error",
error=f"{type(e).__name__}:{e}",
)
return _conservative_default(f"{type(e).__name__}:{e}")
parsed = _parse_response(result)
if parsed is None:
log.warning(
"classifier_bad_response_shape",
got=type(result).__name__,
)
return _conservative_default("bad_response_shape")
# D1 — apply admin-configured topic overrides on top of LLM judgment.
# Lazy import to avoid a circular dependency if topic_volatility ever
# grows to import classifier types.
try:
from brain_api.services.topic_volatility import (
get_topic_overrides,
reconcile_with_classifier,
)
overrides = await get_topic_overrides()
eff_vol, eff_ttl = reconcile_with_classifier(
classifier_volatility=parsed.volatility,
classifier_validity_hours=parsed.estimated_validity_hours,
classifier_topics=parsed.topic_codes,
overrides=overrides,
)
if eff_vol != parsed.volatility or eff_ttl != parsed.estimated_validity_hours:
log.info(
"classifier_admin_override",
llm_volatility=parsed.volatility,
llm_ttl=parsed.estimated_validity_hours,
final_volatility=eff_vol,
final_ttl=eff_ttl,
topics=parsed.topic_codes,
)
# Re-clamp the final TTL against the per-tier hard caps.
eff_ttl = _clamp_validity_hours(eff_ttl, eff_vol)
parsed = ClaimVolatility(
volatility=eff_vol, # type: ignore[arg-type]
topic_codes=parsed.topic_codes,
entity_bindings=parsed.entity_bindings,
estimated_validity_hours=eff_ttl,
time_sensitive=parsed.time_sensitive,
reasoning=parsed.reasoning,
)
except Exception as e: # noqa: BLE001
# Override layer is best-effort — never block on it.
log.debug(
"classifier_override_skipped",
error=f"{type(e).__name__}:{e}",
)
log.info(
"classifier_ok",
volatility=parsed.volatility,
topics=parsed.topic_codes,
validity_hours=parsed.estimated_validity_hours,
bindings=len(parsed.entity_bindings),
)
return parsed

758
ai_platform/modules/didi_brain/brain_api/services/fact_status.py Normal file
View file

@ -0,0 +1,758 @@
"""Fact Status — Pilon 11 of the cache freshness defense.
Versioned knowledge layer for entity-predicate-object triples extracted from
claims. Each fact has:
- a current truth value (TRUE / FALSE / NULL=unknown), in brain_fact_status
- a chronological history of (truth, valid_from, valid_to) windows in
brain_fact_version
When the world changes (a president loses an election, an official dies, a
ceasefire is signed), the active fact_version gets ``valid_to = now()`` and a
new version opens with the new truth value. The cache invalidation pipeline
queries this table at lookup time to decide whether any cached verdict
depends on a fact whose current truth no longer matches what the cache
assumed.
Population:
- extractor pipeline (services/ingest.py background task) → upsert at
ingestion of new claim atoms, with truth=NULL until verified
- classifier (services/classifier.py) returns entity_bindings for every
classified claim → these are upserted lazily on first cache write
- moderator override (admin endpoint) → ``moderator_locked=true`` prevents
the auditor from reverting the moderator's decision
- breaking news watcher → close current version + open new one with the
fresh truth value derived from the breaking story
Read by:
- cache lookups (services/cache_judge.py) — if any binding is known-FALSE,
treat cache as INVALIDATE without running NLI
- admin dashboard (Phase D2) — fact browser with timeline
This module is pure persistence + canonicalization. Truth detection (the
"is X currently true?" decision) lives in services/cache_judge.py + the
auditor cron — they call upsert_fact_truth here when they have an answer.
"""
from __future__ import annotations
import hashlib
import json
import unicodedata
from dataclasses import dataclass
from datetime import datetime, timedelta, timezone
from typing import Any, Literal
from brain_api.db import db
from brain_api.services.classifier import EntityBinding
from shared.logging import get_logger
log = get_logger(__name__)
# Default re-check intervals per volatility tier (hours). Auditor uses these
# to schedule next_check_at when no per-fact override is set.
DEFAULT_CHECK_INTERVAL_HOURS: dict[str, int] = {
"volatile": 6,
"evolving": 168, # 7 days
"stable": 2160, # 90 days
}
CreatedBy = Literal[
"auto",
"moderator",
"breaking_news_watcher",
"auditor",
"extractor",
]
# --------------------------------------------------------------------- DTOs
@dataclass(slots=True)
class FactRecord:
"""Current state of a fact (one row in brain_fact_status)."""
fact_id: int
subject: str
predicate: str
obj: str
canonical_form: str
canonical_form_hash: str
current_truth: bool | None
current_version_id: int | None
current_confidence: float | None
last_verified_at: datetime | None
last_evidence_urls: list[str]
volatility: str | None
topic_codes: list[str]
next_check_at: datetime
check_interval_hours: int
moderator_locked: bool
moderator_user_id: str | None
moderator_notes: str | None
created_at: datetime
updated_at: datetime
@dataclass(slots=True)
class FactVersion:
"""One historical version of a fact (one row in brain_fact_version)."""
version_id: int
fact_id: int
truth_value: bool
confidence: float | None
valid_from: datetime
valid_to: datetime | None
source_atom_ids: list[str]
evidence_urls: list[str]
llm_reasoning: str | None
created_by: str
moderator_user_id: str | None
notes: str | None
created_at: datetime
# --------------------------------------------------------------- canonicalization
def _normalize_token(s: str) -> str:
"""Strip diacritics + lowercase + collapse whitespace.
Matches verification_cache.normalize_claim style so subject "România" and
"Romania" hash to the same fact.
"""
s = unicodedata.normalize("NFKD", s)
s = "".join(c for c in s if not unicodedata.combining(c))
s = s.lower().strip()
return " ".join(s.split())
def canonicalize_triple(subject: str, predicate: str, obj: str) -> str:
"""Build a canonical "subject predicate object" string.
Predicate is normalized to ``snake_case`` (already conventional in the
classifier prompt). Subject and object are lowercased, diacritic-stripped,
whitespace-collapsed.
"""
s = _normalize_token(subject)
p = _normalize_token(predicate).replace(" ", "_")
o = _normalize_token(obj)
return f"{s} {p} {o}"
def hash_canonical(canonical_form: str) -> str:
"""sha256[:32] of the canonical form. Matches the UNIQUE constraint width."""
return hashlib.sha256(canonical_form.encode("utf-8")).hexdigest()[:32]
# --------------------------------------------------------------------- writes
async def register_facts_from_bindings(
bindings: list[EntityBinding],
*,
volatility: str | None = None,
topic_codes: list[str] | None = None,
source_atom_id: str | None = None,
) -> list[int]:
"""Upsert fact_status rows from classifier-extracted bindings.
No truth value is asserted here — bindings are recorded with
``current_truth=NULL`` until something verifies them (the auditor, a
breaking-news event, or a moderator). This is the lazy-registration
path called from the cache write hooks.
Returns:
List of fact_ids touched (one per binding, in the same order).
"""
if not bindings:
return []
if not db.pool:
raise RuntimeError("brain_db not connected")
interval_hours = DEFAULT_CHECK_INTERVAL_HOURS.get(volatility or "evolving", 168)
next_check = datetime.now(tz=timezone.utc) + timedelta(hours=interval_hours)
topics = topic_codes or []
sql = """
INSERT INTO brain_fact_status (
subject, predicate, object, canonical_form, canonical_form_hash,
volatility, topic_codes, next_check_at, check_interval_hours
)
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9)
ON CONFLICT (canonical_form_hash) DO UPDATE SET
topic_codes = (
SELECT ARRAY(
SELECT DISTINCT t FROM unnest(
brain_fact_status.topic_codes || EXCLUDED.topic_codes
) AS t
)
),
volatility = COALESCE(EXCLUDED.volatility, brain_fact_status.volatility),
check_interval_hours = LEAST(
brain_fact_status.check_interval_hours,
EXCLUDED.check_interval_hours
),
updated_at = now()
RETURNING fact_id
"""
fact_ids: list[int] = []
async with db.pool.acquire() as conn:
async with conn.transaction():
for b in bindings:
canonical = canonicalize_triple(b.subject, b.predicate, b.obj)
ch = hash_canonical(canonical)
row = await conn.fetchrow(
sql,
b.subject,
b.predicate,
b.obj,
canonical,
ch,
volatility,
topics,
next_check,
interval_hours,
)
if row:
fact_ids.append(row["fact_id"])
if source_atom_id and fact_ids:
log.debug(
"fact_status_registered",
atom_id=source_atom_id,
fact_count=len(fact_ids),
)
return fact_ids
async def assert_fact_truth(
*,
canonical_form_hash: str,
truth_value: bool,
confidence: float | None,
evidence_urls: list[str],
source_atom_ids: list[str] | None = None,
llm_reasoning: str | None = None,
created_by: CreatedBy = "auto",
moderator_user_id: str | None = None,
notes: str | None = None,
) -> tuple[FactRecord, bool] | None:
"""Set a fact's current truth value, opening a new version if it changed.
If the new truth_value matches the current truth (same boolean), the
existing version is touched (its evidence list is augmented) but no new
version is opened. If the value differs (or the fact had no truth yet),
the active version gets ``valid_to=now()`` and a new version opens.
Skipped silently if ``moderator_locked`` is set on the row — moderator
overrides win until explicitly unlocked.
Args:
canonical_form_hash: The hash from ``hash_canonical``.
truth_value: TRUE or FALSE; pass through ``assert_fact_unknown`` if
you want to clear back to NULL.
confidence: 0-100 LLM confidence (or moderator confidence).
evidence_urls: Sources backing this assertion.
source_atom_ids: Atomic atom IDs that contributed to this assertion.
llm_reasoning: One-line LLM justification.
created_by: Provenance tag for the version.
moderator_user_id: Required if ``created_by='moderator'``.
notes: Free-form notes (especially useful for moderator overrides).
Returns:
``(FactRecord, version_changed)`` — version_changed=True when a new
version was opened. Returns None if the fact is moderator_locked
and the caller is not a moderator.
"""
if not db.pool:
raise RuntimeError("brain_db not connected")
now = datetime.now(tz=timezone.utc)
async with db.pool.acquire() as conn:
async with conn.transaction():
# 1. Lock-and-load the fact row.
row = await conn.fetchrow(
"""
SELECT * FROM brain_fact_status
WHERE canonical_form_hash = $1
FOR UPDATE
""",
canonical_form_hash,
)
if row is None:
log.warning(
"assert_fact_truth_unknown_fact",
canonical_form_hash=canonical_form_hash,
)
return None
if row["moderator_locked"] and created_by != "moderator":
log.info(
"assert_fact_truth_locked",
fact_id=row["fact_id"],
canonical=row["canonical_form"][:80],
)
return _row_to_fact(row), False
old_truth = row["current_truth"]
value_changed = old_truth is None or bool(old_truth) != truth_value
new_version_id: int | None = None
if value_changed:
# Close the active version (if any).
await conn.execute(
"""
UPDATE brain_fact_version
SET valid_to = $1
WHERE fact_id = $2 AND valid_to IS NULL
""",
now,
row["fact_id"],
)
# Open a new version.
inserted = await conn.fetchrow(
"""
INSERT INTO brain_fact_version (
fact_id, truth_value, confidence,
valid_from, valid_to,
source_atom_ids, evidence_urls, llm_reasoning,
created_by, moderator_user_id, notes
)
VALUES ($1, $2, $3, $4, NULL, $5, $6::jsonb, $7, $8, $9, $10)
RETURNING version_id
""",
row["fact_id"],
truth_value,
confidence,
now,
source_atom_ids or [],
json.dumps(list(evidence_urls)),
llm_reasoning,
created_by,
moderator_user_id,
notes,
)
new_version_id = inserted["version_id"] if inserted else None
# 2. Update brain_fact_status (always — even on no-change we
# bump last_verified_at and merge evidence URLs).
interval_h = row["check_interval_hours"]
next_check = now + timedelta(hours=interval_h)
updated = await conn.fetchrow(
"""
UPDATE brain_fact_status
SET current_truth = $2,
current_version_id = COALESCE($3, current_version_id),
current_confidence = $4,
last_verified_at = $5,
last_evidence_urls = $6::jsonb,
next_check_at = $7,
moderator_user_id = COALESCE($8, moderator_user_id),
moderator_notes = COALESCE($9, moderator_notes),
updated_at = now()
WHERE fact_id = $1
RETURNING *
""",
row["fact_id"],
truth_value,
new_version_id,
confidence,
now,
json.dumps(list(evidence_urls)),
next_check,
moderator_user_id,
notes,
)
# 3. Audit log.
await conn.execute(
"""
INSERT INTO brain_audit_log (action, target_table, target_id, actor, payload)
VALUES ($1, 'brain_fact_status', $2, $3, $4::jsonb)
""",
"fact_truth_set" if not value_changed else "fact_truth_changed",
str(row["fact_id"]),
created_by if created_by != "moderator" else (moderator_user_id or "moderator"),
json.dumps({
"old_truth": old_truth,
"new_truth": truth_value,
"confidence": confidence,
"evidence_count": len(evidence_urls),
}),
)
return _row_to_fact(updated), value_changed # type: ignore[arg-type]
async def lock_fact(
*,
canonical_form_hash: str,
moderator_user_id: str,
moderator_notes: str | None = None,
) -> FactRecord | None:
"""Moderator override: prevent auditor from changing this fact.
Use when a human has decided the truth and machine judgment is unreliable
for the topic.
"""
if not db.pool:
raise RuntimeError("brain_db not connected")
async with db.pool.acquire() as conn:
row = await conn.fetchrow(
"""
UPDATE brain_fact_status
SET moderator_locked = true,
moderator_user_id = $2,
moderator_notes = COALESCE($3, moderator_notes),
updated_at = now()
WHERE canonical_form_hash = $1
RETURNING *
""",
canonical_form_hash,
moderator_user_id,
moderator_notes,
)
return _row_to_fact(row) if row else None
async def unlock_fact(
*, canonical_form_hash: str, moderator_user_id: str
) -> FactRecord | None:
"""Re-enable auditor updates on a previously locked fact."""
if not db.pool:
raise RuntimeError("brain_db not connected")
async with db.pool.acquire() as conn:
row = await conn.fetchrow(
"""
UPDATE brain_fact_status
SET moderator_locked = false,
moderator_user_id = $2,
updated_at = now()
WHERE canonical_form_hash = $1
RETURNING *
""",
canonical_form_hash,
moderator_user_id,
)
return _row_to_fact(row) if row else None
# --------------------------------------------------------------------- reads
async def check_fact_validity(
bindings: list[EntityBinding],
) -> dict[str, bool | None]:
"""For each binding, return current_truth from brain_fact_status.
Returns:
Dict keyed by canonical_form_hash. Values:
- True → fact is currently TRUE (cache-aligned if cache assumed TRUE)
- False → fact is currently FALSE (cache-aligned if cache assumed FALSE)
- None → unknown / not registered yet
Caller (typically gather.py) compares the cache's assumption against this
dict and invalidates if any binding flipped against the cache.
"""
if not bindings:
return {}
if not db.pool:
raise RuntimeError("brain_db not connected")
hashes = [hash_canonical(canonicalize_triple(b.subject, b.predicate, b.obj)) for b in bindings]
sql = """
SELECT canonical_form_hash, current_truth
FROM brain_fact_status
WHERE canonical_form_hash = ANY($1::text[])
"""
out: dict[str, bool | None] = {h: None for h in hashes}
async with db.pool.acquire() as conn:
rows = await conn.fetch(sql, hashes)
for r in rows:
out[r["canonical_form_hash"]] = r["current_truth"]
return out
async def get_fact(canonical_form_hash: str) -> FactRecord | None:
"""Fetch one fact_status row by hash."""
if not db.pool:
raise RuntimeError("brain_db not connected")
async with db.pool.acquire() as conn:
row = await conn.fetchrow(
"SELECT * FROM brain_fact_status WHERE canonical_form_hash = $1",
canonical_form_hash,
)
return _row_to_fact(row) if row else None
async def get_fact_by_id(fact_id: int) -> FactRecord | None:
if not db.pool:
raise RuntimeError("brain_db not connected")
async with db.pool.acquire() as conn:
row = await conn.fetchrow(
"SELECT * FROM brain_fact_status WHERE fact_id = $1",
fact_id,
)
return _row_to_fact(row) if row else None
async def list_versions(fact_id: int) -> list[FactVersion]:
"""All versions for one fact, newest first."""
if not db.pool:
raise RuntimeError("brain_db not connected")
async with db.pool.acquire() as conn:
rows = await conn.fetch(
"""
SELECT * FROM brain_fact_version
WHERE fact_id = $1
ORDER BY valid_from DESC
""",
fact_id,
)
return [_row_to_version(r) for r in rows]
async def list_facts_due_for_check(
*, limit: int = 100, volatility: str | None = None
) -> list[FactRecord]:
"""Auditor entry point: facts whose ``next_check_at`` has passed.
Excludes moderator-locked facts (those are managed by humans).
"""
if not db.pool:
raise RuntimeError("brain_db not connected")
sql = """
SELECT * FROM brain_fact_status
WHERE moderator_locked = false
AND next_check_at <= now()
AND ($1::text IS NULL OR volatility = $1)
ORDER BY next_check_at ASC
LIMIT $2
"""
async with db.pool.acquire() as conn:
rows = await conn.fetch(sql, volatility, limit)
return [_row_to_fact(r) for r in rows]
# --------------------------------------------------------------------- admin
@dataclass(slots=True)
class FactPage:
"""Paginated fact_status listing."""
items: list[FactRecord]
total: int
page: int
page_size: int
async def list_facts_admin(
*,
entity: str | None = None,
predicate: str | None = None,
current_truth: bool | None = None,
locked_only: bool = False,
topic: str | None = None,
page: int = 1,
page_size: int = 25,
) -> FactPage:
"""Admin browser query — paginated, ILIKE search on subject/object.
Args:
entity: ILIKE search on subject OR object (e.g., "putin").
predicate: Exact match on predicate (e.g., "is_president_of").
current_truth: Filter to TRUE / FALSE only when set.
locked_only: Show only moderator-locked facts.
topic: Filter to facts whose topic_codes contain this code.
page, page_size: Pagination (1-based).
"""
if not db.pool:
raise RuntimeError("brain_db not connected")
page = max(1, page)
page_size = max(1, min(100, page_size))
where: list[str] = ["1=1"]
params: list[Any] = []
if entity:
params.append(f"%{entity}%")
where.append(f"(subject ILIKE ${len(params)} OR object ILIKE ${len(params)})")
if predicate:
params.append(predicate)
where.append(f"predicate = ${len(params)}")
if current_truth is not None:
params.append(current_truth)
where.append(f"current_truth = ${len(params)}")
if locked_only:
where.append("moderator_locked = true")
if topic:
params.append([topic])
where.append(f"topic_codes && ${len(params)}::text[]")
where_sql = " AND ".join(where)
async with db.pool.acquire() as conn:
total_row = await conn.fetchrow(
f"SELECT COUNT(*) AS c FROM brain_fact_status WHERE {where_sql}",
*params,
)
total = int(total_row["c"]) if total_row else 0
params.append(page_size)
params.append((page - 1) * page_size)
rows = await conn.fetch(
f"""SELECT * FROM brain_fact_status
WHERE {where_sql}
ORDER BY updated_at DESC
LIMIT ${len(params) - 1} OFFSET ${len(params)}""",
*params,
)
return FactPage(
items=[_row_to_fact(r) for r in rows],
total=total,
page=page,
page_size=page_size,
)
async def list_audit_log(
*,
action: str | None = None,
target_table: str | None = None,
actor: str | None = None,
since: datetime | None = None,
page: int = 1,
page_size: int = 50,
) -> tuple[list[dict[str, Any]], int]:
"""Browse brain_audit_log entries (paginated, newest first).
Returns ``(items, total)``.
"""
if not db.pool:
raise RuntimeError("brain_db not connected")
page = max(1, page)
page_size = max(1, min(200, page_size))
where: list[str] = ["1=1"]
params: list[Any] = []
if action:
params.append(f"{action}%")
where.append(f"action ILIKE ${len(params)}")
if target_table:
params.append(target_table)
where.append(f"target_table = ${len(params)}")
if actor:
params.append(f"%{actor}%")
where.append(f"actor ILIKE ${len(params)}")
if since is not None:
params.append(since)
where.append(f"created_at >= ${len(params)}")
where_sql = " AND ".join(where)
async with db.pool.acquire() as conn:
total_row = await conn.fetchrow(
f"SELECT COUNT(*) AS c FROM brain_audit_log WHERE {where_sql}",
*params,
)
total = int(total_row["c"]) if total_row else 0
params.append(page_size)
params.append((page - 1) * page_size)
rows = await conn.fetch(
f"""SELECT log_id, action, target_table, target_id, actor,
payload, created_at
FROM brain_audit_log
WHERE {where_sql}
ORDER BY created_at DESC
LIMIT ${len(params) - 1} OFFSET ${len(params)}""",
*params,
)
items: list[dict[str, Any]] = []
for r in rows:
payload = r["payload"]
if isinstance(payload, str):
try:
payload = json.loads(payload)
except (TypeError, ValueError):
payload = {}
items.append({
"log_id": r["log_id"],
"action": r["action"],
"target_table": r["target_table"],
"target_id": r["target_id"],
"actor": r["actor"],
"payload": payload,
"created_at": r["created_at"],
})
return items, total
# --------------------------------------------------------------------- helpers
def _row_to_fact(row: Any) -> FactRecord:
last_evidence = row["last_evidence_urls"]
if isinstance(last_evidence, str):
last_evidence = json.loads(last_evidence)
return FactRecord(
fact_id=row["fact_id"],
subject=row["subject"],
predicate=row["predicate"],
obj=row["object"],
canonical_form=row["canonical_form"],
canonical_form_hash=row["canonical_form_hash"],
current_truth=row["current_truth"],
current_version_id=row["current_version_id"],
current_confidence=(
float(row["current_confidence"])
if row["current_confidence"] is not None
else None
),
last_verified_at=row["last_verified_at"],
last_evidence_urls=list(last_evidence) if last_evidence else [],
volatility=row["volatility"],
topic_codes=list(row["topic_codes"]) if row["topic_codes"] else [],
next_check_at=row["next_check_at"],
check_interval_hours=row["check_interval_hours"],
moderator_locked=row["moderator_locked"],
moderator_user_id=row["moderator_user_id"],
moderator_notes=row["moderator_notes"],
created_at=row["created_at"],
updated_at=row["updated_at"],
)
def _row_to_version(row: Any) -> FactVersion:
ev_urls = row["evidence_urls"]
if isinstance(ev_urls, str):
ev_urls = json.loads(ev_urls)
return FactVersion(
version_id=row["version_id"],
fact_id=row["fact_id"],
truth_value=row["truth_value"],
confidence=(
float(row["confidence"]) if row["confidence"] is not None else None
),
valid_from=row["valid_from"],
valid_to=row["valid_to"],
source_atom_ids=(
list(row["source_atom_ids"]) if row["source_atom_ids"] else []
),
evidence_urls=list(ev_urls) if ev_urls else [],
llm_reasoning=row["llm_reasoning"],
created_by=row["created_by"],
moderator_user_id=row["moderator_user_id"],
notes=row["notes"],
created_at=row["created_at"],
)

62
ai_platform/modules/didi_brain/brain_api/services/fetch.py Normal file
View file

@ -0,0 +1,62 @@
"""POST /v1/fetch — look up stored atoms by URL and return their content.
Unlike the web module's /v1/fetch which fetches URLs live, ours just checks
if we already have an atom for each URL. URLs we don't have go into
`failed_urls` with reason "not_in_brain" — Didi's backend can then fall back
to the live web module for those.
"""
from __future__ import annotations
import asyncio
import time
import uuid
from brain_api.schemas import (
BrainMeta,
FailedUrl,
FetchedPage,
FetchRequest,
FetchResponse,
)
from brain_api.services.mapping import doc_to_fetched_page
from shared.atomic_api import AtomicClient
from shared.logging import get_logger
log = get_logger(__name__)
async def fetch(req: FetchRequest, *, atomic: AtomicClient) -> FetchResponse:
t0 = time.perf_counter()
request_id = str(uuid.uuid4())
tasks = [atomic.get_atom_by_source_url(u) for u in req.urls]
results = await asyncio.gather(*tasks, return_exceptions=True)
pages: list[FetchedPage] = []
failed: list[FailedUrl] = []
for url, result in zip(req.urls, results, strict=True):
if isinstance(result, Exception):
failed.append(FailedUrl(url=url, error=f"brain_lookup_error: {result}"))
continue
if result is None:
failed.append(FailedUrl(url=url, error="not_in_brain"))
continue
pages.append(doc_to_fetched_page(result, include_html=req.include_html))
total_ms = round((time.perf_counter() - t0) * 1000, 1)
return FetchResponse(
request_id=request_id,
pages=pages,
total_fetched=len(pages),
total_failed=len(failed),
execution_time_ms=total_ms,
failed_urls=failed,
brain_meta=BrainMeta(
cache_status="HIT" if pages else "MISS",
api_version="v1",
implementation="didibrain",
evidence_sources=len(pages),
total_claim_atoms_matched=0,
),
)

665
ai_platform/modules/didi_brain/brain_api/services/gather.py Normal file
View file

@ -0,0 +1,665 @@
"""POST /v1/gather — the main claim-to-evidence pipeline.
Flow:
1. semantic search with Type/Claim filter, pull top-K candidates
2. rerank with BGE cross-encoder against the input claim (precision)
3. group hits by parent document URL
4. fetch full parent doc + top claim atom bodies in parallel
5. emit one EvidenceItem per parent doc, sorted by best rerank score
6. shape the full GatherResponse with stages, stats, context
"""
from __future__ import annotations
import asyncio
import math
import time
import uuid
from datetime import datetime, timezone
from brain_api.schemas import (
BrainMeta,
EvidenceStats,
GatherRequest,
GatherResponse,
SearchContext,
SearchResultItem,
StageRecord,
)
from brain_api.services.mapping import (
detect_language_simple,
doc_to_search_result,
evidence_from_parent,
group_hits_by_parent,
parent_url_of,
parse_claim_atom_body,
)
from brain_api.services.nli import classify_batch
from brain_api.services import verification_cache as vcache
from brain_api.services import fact_status as fact_svc
from brain_api.services.classifier import EntityBinding
from shared.atomic_api import AtomicClient, SearchHit
from shared.embedding_client import EmbeddingClient
from shared.llm_client import LlmClient
from shared.logging import get_logger
from shared.taxonomy import TagResolver
log = get_logger(__name__)
FIRST_STAGE_LIMIT = 50 # how many claim hits to pull before reranking
SIMILARITY_FLOOR = 0.20 # below this we don't even consider a hit
RERANK_TOP_N = 15
# Phase B4 — recency boost configuration. Half-life is in days.
# Output of _combined_score = (1 - w_recency) * rerank + w_recency * recency,
# where recency = exp(-age_days / half_life). Stable claims skip the recency
# pass entirely; volatile claims weight recency heavily.
RECENCY_PROFILES: dict[str, tuple[float, float]] = {
# volatility → (w_recency, half_life_days)
"volatile": (0.50, 3.0), # heavy recency, fast decay
"evolving": (0.30, 30.0), # moderate recency, monthly half-life
"stable": (0.00, 9999.0), # ignore recency
}
# Default profile when no hint provided — light recency boost so older
# articles can't fully dominate even on stable topics, without harming them.
RECENCY_DEFAULT: tuple[float, float] = (0.15, 30.0)
# Hard recency filter (days) — only applied when volatility_hint=='volatile'.
DEFAULT_VOLATILE_RECENCY_DAYS = 7
async def gather(
req: GatherRequest,
*,
atomic: AtomicClient,
embed: EmbeddingClient,
llm: LlmClient,
resolver: TagResolver,
) -> GatherResponse:
t0 = time.perf_counter()
request_id = str(uuid.uuid4())
stages: list[StageRecord] = []
# ---- stage 1: context (very lightweight — no LLM for now) --------------
s1_t0 = time.perf_counter()
context = SearchContext(
primary_country="Global",
secondary_countries=[],
detected_language=req.language_hint or detect_language_simple(req.claim),
search_queries=[req.claim],
)
stages.append(
StageRecord(
stage="context",
success=True,
items_processed=1,
items_failed=0,
duration_ms=round((time.perf_counter() - s1_t0) * 1000, 1),
)
)
# ---- stage 2: first-stage retrieval against Type/Claim ----------------
s2_t0 = time.perf_counter()
type_claim_id = resolver.get("Type/Claim")
try:
# Atomic's /api/search currently does not accept a tag_id filter in the
# body — we filter client-side after the call. Pull enough results.
raw_hits = await atomic.search(
req.claim,
mode="semantic",
limit=FIRST_STAGE_LIMIT * 2,
threshold=SIMILARITY_FLOOR,
)
claim_hits: list[SearchHit] = [
h
for h in raw_hits
if any(t.get("name") == "Claim" for t in h.tags)
][:FIRST_STAGE_LIMIT]
stages.append(
StageRecord(
stage="retrieval",
success=True,
items_processed=len(claim_hits),
items_failed=0,
duration_ms=round((time.perf_counter() - s2_t0) * 1000, 1),
)
)
except Exception as e: # noqa: BLE001
log.error("gather_retrieval_failed", error=str(e))
stages.append(
StageRecord(
stage="retrieval",
success=False,
items_processed=0,
items_failed=1,
duration_ms=round((time.perf_counter() - s2_t0) * 1000, 1),
error=str(e),
)
)
return _empty_response(
request_id=request_id,
claim=req.claim,
context=context,
stages=stages,
started_at=t0,
)
if not claim_hits:
return _empty_response(
request_id=request_id,
claim=req.claim,
context=context,
stages=stages,
started_at=t0,
)
# ---- stage 3: rerank top-N with BGE cross-encoder --------------------
s3_t0 = time.perf_counter()
rerank_scores: dict[str, float] = {}
try:
# We need the CONTENT of each claim atom to rerank it; fetch in parallel.
full_claim_atoms = await _fetch_full_atoms(
atomic,
[h.atom_id for h in claim_hits[:RERANK_TOP_N]],
)
rerank_docs: list[str] = []
rerank_atom_ids: list[str] = []
for h in claim_hits[:RERANK_TOP_N]:
full = full_claim_atoms.get(h.atom_id) or {}
text = (full.get("content") or "")[:2000]
if text:
rerank_docs.append(text)
rerank_atom_ids.append(h.atom_id)
if rerank_docs:
reranked = await embed.rerank(req.claim, rerank_docs)
for r in reranked:
if 0 <= r.index < len(rerank_atom_ids):
rerank_scores[rerank_atom_ids[r.index]] = r.score
stages.append(
StageRecord(
stage="rerank",
success=True,
items_processed=len(rerank_docs),
items_failed=0,
duration_ms=round((time.perf_counter() - s3_t0) * 1000, 1),
)
)
except Exception as e: # noqa: BLE001
log.warning("gather_rerank_failed", error=str(e))
full_claim_atoms = {}
stages.append(
StageRecord(
stage="rerank",
success=False,
items_processed=0,
items_failed=len(claim_hits),
duration_ms=round((time.perf_counter() - s3_t0) * 1000, 1),
error=str(e),
)
)
# ---- stage 4: group by parent doc --------------------------------------
s4_t0 = time.perf_counter()
buckets = group_hits_by_parent(claim_hits, rerank_scores)
# Phase B4 — fetch a wider parent pool so the recency reranker has
# candidates to choose from. We oversample by 2x then trim to max_evidence
# AFTER recency reranking. For volatile claims we also need the broader
# pool so the hard recency filter doesn't leave us empty.
fetch_count = min(len(buckets), max(req.max_evidence * 2, req.max_evidence))
parent_urls = [url for url, _ in buckets][:fetch_count]
parent_atoms = await _fetch_parents(atomic, parent_urls)
# Apply recency reranking + filter, trim to max_evidence.
buckets = _apply_recency(
buckets=buckets,
parent_atoms=parent_atoms,
volatility_hint=req.volatility_hint,
recency_window_days=req.recency_window_days,
max_evidence=req.max_evidence,
)
# Fetch any missing full claim atom bodies (for non-reranked but still emitted)
need_more_claim_atoms = [
h.atom_id
for _, hs in buckets[: req.max_evidence]
for (h, _) in hs
if h.atom_id not in full_claim_atoms
]
if need_more_claim_atoms:
extra = await _fetch_full_atoms(atomic, need_more_claim_atoms)
full_claim_atoms.update(extra)
# ---- stage 5: NLI stance vs query (optional) --------------------------
# For each bucket that will become an evidence item, run NLI on the best
# matching claim atom (the one we surface as `summary`). Parallelized so
# ~10 calls complete in a couple of seconds rather than seconds per call.
nli_by_atom_id: dict[str, tuple[str, float, str | None]] = {}
if req.run_nli:
s5_t0 = time.perf_counter()
best_per_bucket: list[tuple[str, str]] = []
for parent_url, hits in buckets[: req.max_evidence]:
if parent_url not in parent_atoms:
continue
best_hit, _ = hits[0]
full = full_claim_atoms.get(best_hit.atom_id) or {}
claim_text, _stance, _parent_id = parse_claim_atom_body(
full.get("content") or ""
)
if claim_text:
best_per_bucket.append((best_hit.atom_id, claim_text))
if best_per_bucket:
nli_results = await classify_batch(
llm,
claim=req.claim,
evidence_texts=[text for _, text in best_per_bucket],
)
for (atom_id, _text), result in zip(
best_per_bucket, nli_results, strict=True
):
nli_by_atom_id[atom_id] = (
result.label,
result.confidence,
result.error,
)
failed = sum(1 for v in nli_by_atom_id.values() if v[2] is not None)
stages.append(
StageRecord(
stage="nli",
success=True,
items_processed=len(nli_by_atom_id),
items_failed=failed,
duration_ms=round((time.perf_counter() - s5_t0) * 1000, 1),
)
)
# ---- stage 6: build evidence list with NLI attached -------------------
evidence_items = []
for parent_url, hits in buckets[: req.max_evidence]:
parent_atom = parent_atoms.get(parent_url)
if not parent_atom:
continue
item = evidence_from_parent(
parent_atom=parent_atom,
claim_hits=hits,
parent_full_atoms=full_claim_atoms,
include_full_text=req.include_full_text,
nli_by_atom_id=nli_by_atom_id or None,
)
evidence_items.append(item)
stages.append(
StageRecord(
stage="evidence",
success=True,
items_processed=len(evidence_items),
items_failed=0,
duration_ms=round((time.perf_counter() - s4_t0) * 1000, 1),
)
)
# ---- shape response --------------------------------------------------
total_ms = round((time.perf_counter() - t0) * 1000, 1)
search_results: list[SearchResultItem] = []
for i, item in enumerate(evidence_items, 1):
search_results.append(
SearchResultItem(
query=req.claim,
url=item.url,
title=item.title,
snippet=item.snippet or "",
rank=i,
site=item.publisher,
published_at=item.published_at,
)
)
stats = EvidenceStats(
input_items=len(claim_hits),
after_dedup=len(buckets),
output_items=len(evidence_items),
duplicates_removed=max(0, len(claim_hits) - len(buckets)),
tokens_used=0,
)
# Cache status reflects BOTH presence and quality:
# - MISS if no evidence at all, or the best match is weak (< 0.3 rerank)
# - PARTIAL if we have evidence but best rerank is between 0.3 and 0.6
# - HIT when the brain actually has a strong, direct match (>= 0.6)
if not evidence_items:
cache_status = "MISS"
else:
top_relevance = max(
(e.relevance_score for e in evidence_items), default=0.0
)
if top_relevance < 0.30:
cache_status = "MISS"
elif top_relevance < 0.60:
cache_status = "PARTIAL"
else:
cache_status = "HIT"
brain_meta = BrainMeta(
cache_status=cache_status,
api_version="v1",
implementation="didibrain",
evidence_sources=len({e.url for e in evidence_items}),
total_claim_atoms_matched=len(claim_hits),
)
# ---- optional verification cache lookup ---------------------------------
# Lookup key is (claim_hash, tier) only. The evidence URLs the cache was
# written for may differ from `evidence_items` (different runs, different
# corpora). We surface the original URLs in metadata so backend can
# decide whether the cached verification applies to its current view.
if req.include_verification and req.tier:
try:
entry = await vcache.lookup(claim=req.claim, tier=req.tier)
except Exception as e: # noqa: BLE001
log.warning("verification_lookup_failed", error=f"{type(e).__name__}: {e}")
entry = None
staleness = vcache.decide_freshness(
entry,
current_prompt_hash=req.prompt_hash,
current_framework_version=req.framework_version,
)
# Pilon 11 — fact-status check. If the cache is otherwise fresh but
# one of its bound facts has flipped (e.g., "X is president of Y"
# was TRUE when cached, but brain_fact_status now says FALSE), demote
# to stale_evidence so the caller recomputes. We only run this when
# the cache would otherwise be served (fresh / stale_framework — the
# other states already force recompute).
if entry is not None and staleness in ("fresh", "stale_framework"):
try:
flipped = await _detect_flipped_facts(entry)
if flipped:
log.info(
"verification_facts_flipped",
flipped_count=len(flipped),
was=staleness,
)
staleness = "stale_evidence"
brain_meta.verification_facts_invalidated = flipped
except Exception as e: # noqa: BLE001
log.warning(
"verification_fact_check_failed",
error=f"{type(e).__name__}: {e}",
)
brain_meta.verification_staleness = staleness
if entry is not None:
brain_meta.verification_model = entry.model
brain_meta.verification_tier = entry.tier
brain_meta.verification_prompt_hash = entry.prompt_hash
brain_meta.verification_framework_version = entry.framework_version
brain_meta.verification_cached_at = entry.updated_at
brain_meta.verification_expires_at = entry.expires_at
brain_meta.verification_evidence_urls = entry.evidence_urls
brain_meta.verification_evidence_hash = entry.evidence_hash
if staleness == "fresh":
brain_meta.verification = entry.verification_processed
elif staleness == "stale_framework":
# Backend can recompute status from raw locally — no LLM call.
brain_meta.verification = entry.verification_raw
# stale_evidence / stale_prompt / miss → don't expose verification
# so caller is forced to recompute.
return GatherResponse(
request_id=request_id,
claim=req.claim,
evidence=evidence_items,
evidence_stats=stats,
search_context=context,
search_results=search_results,
stages=stages,
total_urls_found=len(claim_hits),
total_pages_fetched=len(evidence_items),
total_evidence_items=len(evidence_items),
execution_time_ms=total_ms,
brain_meta=brain_meta,
)
# --- helpers ---------------------------------------------------------------
def _published_at_of(parent_atom: dict | None) -> datetime | None:
"""Best-effort published_at extractor for a parent Document atom."""
if not parent_atom:
return None
raw = parent_atom.get("published_at") or parent_atom.get("created_at")
if not raw:
return None
if isinstance(raw, datetime):
return raw if raw.tzinfo else raw.replace(tzinfo=timezone.utc)
try:
s = str(raw).rstrip("Z")
# Tolerate trailing Z by using fromisoformat with tz-aware handling.
dt = datetime.fromisoformat(s)
return dt if dt.tzinfo else dt.replace(tzinfo=timezone.utc)
except (TypeError, ValueError):
return None
def _combined_score(
*,
rerank_score: float,
age_days: float | None,
volatility_hint: str | None,
) -> float:
"""Blend rerank with age-based recency. Stable / no-published_at = pure rerank.
For each volatility profile we use:
combined = (1 - w_recency) * rerank + w_recency * exp(-age_days/half_life)
rerank_score is on [0, 1] from the cross-encoder; recency is also [0, 1].
"""
if age_days is None or age_days < 0:
return rerank_score
profile = RECENCY_PROFILES.get(volatility_hint or "", RECENCY_DEFAULT)
w_recency, half_life = profile
if w_recency <= 0:
return rerank_score
recency = math.exp(-age_days / half_life)
return (1.0 - w_recency) * rerank_score + w_recency * recency
def _apply_recency(
*,
buckets: list[tuple[str, list[tuple[SearchHit, float]]]],
parent_atoms: dict[str, dict],
volatility_hint: str | None,
recency_window_days: int | None,
max_evidence: int,
) -> list[tuple[str, list[tuple[SearchHit, float]]]]:
"""Rerank buckets by combined (rerank + recency) score, trim to max_evidence.
For volatility_hint='volatile', also drops parents older than
recency_window_days (default 7) — caller will see fewer evidence items
and can fall back to live web search.
Buckets without a fetched parent_atom (i.e., not in parent_atoms) drop
out entirely — they were just placeholders for a wider fetch.
"""
now = datetime.now(tz=timezone.utc)
# Hard recency cut for volatile (or any volatility when caller pinned a
# window).
if volatility_hint == "volatile" or recency_window_days is not None:
window = recency_window_days or DEFAULT_VOLATILE_RECENCY_DAYS
def _within_window(parent_url: str) -> bool:
atom = parent_atoms.get(parent_url)
pub = _published_at_of(atom)
if pub is None:
# No publish date → keep only when no hard cut requested
# (volatile defaults to dropping unknowns to be safe).
return volatility_hint != "volatile"
return (now - pub).days <= window
buckets = [(u, hs) for (u, hs) in buckets if _within_window(u)]
# Score each surviving bucket using its top hit's rerank score and the
# parent's age, then re-sort. Buckets without parent_atoms are dropped.
scored: list[tuple[float, str, list[tuple[SearchHit, float]]]] = []
for parent_url, hits in buckets:
atom = parent_atoms.get(parent_url)
if not atom or not hits:
continue
top_rerank = float(hits[0][1])
pub = _published_at_of(atom)
age_days = (now - pub).days if pub else None
score = _combined_score(
rerank_score=top_rerank,
age_days=age_days,
volatility_hint=volatility_hint,
)
scored.append((score, parent_url, hits))
scored.sort(key=lambda t: t[0], reverse=True)
return [(url, hits) for (_score, url, hits) in scored[:max_evidence]]
def _extract_cached_truth(processed: dict) -> bool | None:
"""Map a cached verification_processed dict to a binary truth direction.
DIDI v1 schema uses "status": "TRUE" | "FALSE" | "UV" | "OP" | "MIXED".
Anything other than TRUE/FALSE returns None — the cache didn't commit
to a direction so we can't compare it against fact_status.
"""
if not isinstance(processed, dict):
return None
status = processed.get("status")
if isinstance(status, str):
s = status.strip().upper()
if s in ("TRUE", "VERIFIED_TRUE", "VT"):
return True
if s in ("FALSE", "VERIFIED_FALSE", "VF"):
return False
return None
async def _detect_flipped_facts(entry: vcache.CacheEntry) -> list[dict]:
"""Return entity bindings whose current_truth contradicts the cached verdict.
Each returned dict mirrors the binding shape stored on the row, with
extra fields ``cached_assumes`` and ``current_truth`` so the caller
(admin dashboard, downstream backend) can show what changed.
Returns [] if the cache had no bindings, or if no current truth could be
extracted from verification_processed, or if no bound fact disagrees.
"""
if not entry.entity_bindings:
return []
cached_truth = _extract_cached_truth(entry.verification_processed)
if cached_truth is None:
return []
# Reconstruct EntityBinding instances from the JSONB row.
bindings: list[EntityBinding] = []
for raw in entry.entity_bindings:
if not isinstance(raw, dict):
continue
subj = str(raw.get("subject", "")).strip()
pred = str(raw.get("predicate", "")).strip()
obj_ = str(raw.get("object", "")).strip()
try:
conf = float(raw.get("confidence", 0.5))
except (TypeError, ValueError):
conf = 0.5
if subj and pred and obj_:
bindings.append(
EntityBinding(
subject=subj, predicate=pred, obj=obj_, confidence=conf
)
)
if not bindings:
return []
truth_map = await fact_svc.check_fact_validity(bindings)
flipped: list[dict] = []
for b in bindings:
canonical = fact_svc.canonicalize_triple(b.subject, b.predicate, b.obj)
ch = fact_svc.hash_canonical(canonical)
current = truth_map.get(ch)
# We only flag explicit disagreement; current=None means we have no
# opinion (yet) and falls through to the existing freshness checks.
if current is None:
continue
if current != cached_truth:
flipped.append({
"subject": b.subject,
"predicate": b.predicate,
"object": b.obj,
"canonical_form": canonical,
"cached_assumes": cached_truth,
"current_truth": current,
})
return flipped
async def _fetch_full_atoms(
atomic: AtomicClient, atom_ids: list[str]
) -> dict[str, dict]:
"""Parallel get_atom for a list of atom IDs. Missing atoms are dropped."""
if not atom_ids:
return {}
tasks = [atomic.get_atom(a) for a in atom_ids]
results = await asyncio.gather(*tasks, return_exceptions=True)
out: dict[str, dict] = {}
for a, r in zip(atom_ids, results, strict=True):
if isinstance(r, dict):
out[a] = r
return out
async def _fetch_parents(
atomic: AtomicClient, parent_urls: list[str]
) -> dict[str, dict]:
"""Parallel get_atom_by_source_url for parent document URLs."""
if not parent_urls:
return {}
tasks = [atomic.get_atom_by_source_url(u) for u in parent_urls]
results = await asyncio.gather(*tasks, return_exceptions=True)
out: dict[str, dict] = {}
for u, r in zip(parent_urls, results, strict=True):
if isinstance(r, dict):
out[u] = r
return out
def _empty_response(
*,
request_id: str,
claim: str,
context: SearchContext,
stages: list[StageRecord],
started_at: float,
) -> GatherResponse:
return GatherResponse(
request_id=request_id,
claim=claim,
evidence=[],
evidence_stats=EvidenceStats(),
search_context=context,
search_results=[],
stages=stages,
total_urls_found=0,
total_pages_fetched=0,
total_evidence_items=0,
execution_time_ms=round((time.perf_counter() - started_at) * 1000, 1),
brain_meta=BrainMeta(
cache_status="MISS",
api_version="v1",
implementation="didibrain",
evidence_sources=0,
total_claim_atoms_matched=0,
),
)

268
ai_platform/modules/didi_brain/brain_api/services/ingest.py Normal file
View file

@ -0,0 +1,268 @@
"""POST /v1/ingest — populate brain from Didi's web-module results.
When Didi's backend calls the live (expensive) web module and gets a fresh
GatherResponse, it can POST the same body here. We:
1. For each evidence item, dedup against existing atoms by canonical URL
2. Build proper Type/Document atoms with inferred tags:
- Credibility from credibility_score bucket
- Language from search_context.detected_language
- Country/Global by default (future: infer from publisher TLD)
- Any `default_tags` provided by the caller
3. Create atoms synchronously (returns quickly)
4. Optionally queue claim extraction in background via FastAPI BackgroundTasks
Response returns counts + the created atom IDs so the caller can correlate.
"""
from __future__ import annotations
import time
import uuid
from typing import Any
from fastapi import BackgroundTasks
from brain_api.schemas import (
EvidenceItem,
IngestRequest,
IngestResponse,
)
from brain_api.services.classifier import (
ClaimVolatility,
classify_claim_volatility,
)
from shared.atomic_api import AtomicApiError, AtomicClient
from shared.llm_client import LlmClient
from shared.logging import get_logger
from shared.taxonomy import TagResolver
log = get_logger(__name__)
# Credibility bucket boundaries mirror mapping.CREDIBILITY_SCORES inverse.
def credibility_score_to_tag_path(score: float) -> str:
if score >= 0.85:
return "Credibility/Tier1"
if score >= 0.60:
return "Credibility/Tier2"
if score >= 0.40:
return "Credibility/Tier3"
if score >= 0.25:
return "Credibility/StateAffiliated"
if score >= 0.01:
return "Credibility/KnownDisinfo"
return "Credibility/Unknown"
def detected_language_to_tag_path(lang: str | None) -> str:
if not lang:
return "Language/EN"
code = lang.strip().upper()[:2]
mapping = {
"RO": "Language/RO",
"EN": "Language/EN",
"RU": "Language/RU",
"UA": "Language/UA",
"FR": "Language/FR",
"DE": "Language/DE",
"ES": "Language/ES",
"IT": "Language/IT",
"PL": "Language/PL",
}
return mapping.get(code, "Language/EN")
def build_tag_ids_for_evidence(
*,
ev: EvidenceItem,
detected_language: str,
default_tags: list[str],
resolver: TagResolver,
) -> list[str]:
paths: list[str] = [
"Type/Document",
"SourceType/MainstreamMedia", # default — callers can override via default_tags
credibility_score_to_tag_path(ev.credibility_score),
detected_language_to_tag_path(detected_language),
"Country/Global",
]
# Append any caller-provided canonical tags, deduped
for p in default_tags:
if p and p not in paths:
paths.append(p)
return resolver.ids_for(paths, ignore_missing=True)
def evidence_to_markdown(ev: EvidenceItem) -> str:
"""Build the markdown body for a Type/Document atom from an EvidenceItem."""
title = ev.title or ev.url
body = ev.full_text or ev.summary or ev.snippet or ""
header_lines = [f"# {title}", ""]
if ev.published_at:
header_lines.append(f"**Published:** {ev.published_at.isoformat()}")
if ev.publisher:
header_lines.append(f"**Publisher:** {ev.publisher}")
if ev.published_at or ev.publisher:
header_lines.append("")
return "\n".join(header_lines) + body.strip() + "\n"
async def _classify_and_register_facts_async(
*, claim: str, llm: LlmClient, source_label: str
) -> None:
"""Classify the claim and register entity bindings in brain_fact_status.
Best-effort: any failure is logged and swallowed. Runs as a background
task triggered by FastAPI's BackgroundTasks queue, after the ingest
response has been returned to the caller.
"""
try:
classification = await classify_claim_volatility(llm, claim=claim)
except Exception as e: # noqa: BLE001
log.warning(
"ingest_classifier_failed",
error=f"{type(e).__name__}:{e}",
)
return
if not classification.entity_bindings:
return
try:
from brain_api.services.fact_status import register_facts_from_bindings
await register_facts_from_bindings(
classification.entity_bindings,
volatility=classification.volatility,
topic_codes=classification.topic_codes,
source_atom_id=source_label,
)
log.info(
"ingest_facts_registered",
source=source_label,
volatility=classification.volatility,
bindings=len(classification.entity_bindings),
)
except Exception as e: # noqa: BLE001
log.warning(
"ingest_fact_registration_failed",
error=f"{type(e).__name__}:{e}",
)
async def ingest(
req: IngestRequest,
*,
atomic: AtomicClient,
resolver: TagResolver,
background: BackgroundTasks | None = None,
llm: LlmClient | None = None,
) -> IngestResponse:
t0 = time.perf_counter()
request_id = str(uuid.uuid4())
accepted = 0
skipped = 0
errors = 0
warnings: list[str] = []
created_ids: list[str] = []
detected_language = "en"
# The caller may pass language hints inside default_tags or a nested context;
# we support both. Fall back to English.
for p in req.default_tags:
if p.startswith("Language/"):
detected_language = p.split("/", 1)[-1]
break
for ev in req.evidence:
if not ev.url:
warnings.append("evidence item missing url — skipped")
continue
# Dedup on canonical URL
try:
existing = await atomic.get_atom_by_source_url(ev.url)
except AtomicApiError as e:
errors += 1
warnings.append(f"dedup check failed for {ev.url}: {e.status}")
continue
if existing:
skipped += 1
continue
content = evidence_to_markdown(ev)
tag_ids = build_tag_ids_for_evidence(
ev=ev,
detected_language=detected_language,
default_tags=req.default_tags,
resolver=resolver,
)
published_iso = ev.published_at.isoformat() if ev.published_at else None
try:
atom = await atomic.create_atom(
content=content,
source_url=ev.url,
tag_ids=tag_ids,
published_at=published_iso,
)
atom_id = atom.get("id")
if atom_id:
created_ids.append(atom_id)
accepted += 1
else:
errors += 1
warnings.append(f"create_atom returned no id for {ev.url}")
except AtomicApiError as e:
errors += 1
warnings.append(f"create_atom failed for {ev.url}: {e.status} {e.body[:120]}")
extraction_queued = False
if req.run_extraction and created_ids and background is not None:
background.add_task(_run_extraction_background, created_ids)
extraction_queued = True
# Pilon 11: classify req.claim once and register entity bindings into
# brain_fact_status. Lazy (current_truth=NULL) until verified.
if (
req.claim
and req.claim.strip()
and llm is not None
and background is not None
):
background.add_task(
_classify_and_register_facts_async,
claim=req.claim,
llm=llm,
source_label=f"ingest:{request_id[:12]}",
)
return IngestResponse(
request_id=request_id,
accepted=accepted,
skipped_duplicate=skipped,
errors=errors,
created_atom_ids=created_ids,
extraction_queued=extraction_queued,
execution_time_ms=round((time.perf_counter() - t0) * 1000, 1),
warnings=warnings[:20],
)
async def _run_extraction_background(atom_ids: list[str]) -> None:
"""Fire-and-forget extraction for newly ingested atoms."""
from extractor.batch import run_batch
log.info("brain_ingest_extraction_start", count=len(atom_ids))
try:
stats = await run_batch(only_atom_ids=set(atom_ids))
log.info(
"brain_ingest_extraction_done",
processed=stats.docs_processed,
claims=stats.claims_created,
failed=stats.docs_failed,
)
except Exception as e: # noqa: BLE001
log.error("brain_ingest_extraction_failed", error=str(e))

282
ai_platform/modules/didi_brain/brain_api/services/invalidation.py Normal file
View file

@ -0,0 +1,282 @@
"""Cache invalidation service — Pilon 8 of the cache freshness defense.
Mass-invalidates rows in brain_analysis_atom and brain_verification_cache
based on filters (topic, entity, since, content pattern). Used by:
- Admin dashboard "Flush topic" button (manual ops)
- didibrain-breaking-watcher (real-time, when a breaking story affects
a topic or entity)
- Daily auditor (when gold demotion cascades to dependent rows)
Invalidation = set expires_at = now() (soft delete, preserves row for audit).
Gold atoms in brain_analysis_atom are NOT invalidated by topic/entity filters
unless ``invalidate_gold=True`` is passed — gold rows reflect human moderator
decisions and shouldn't be flushed by automated breaking news. Operators who
need to flush them must opt in explicitly.
Every invalidation logs a row in brain_audit_log with the filter spec and
counts so the admin dashboard can show recent flush operations.
"""
from __future__ import annotations
import json
from dataclasses import dataclass
from datetime import datetime, timezone
from typing import Any
from brain_api.db import db
from shared.logging import get_logger
log = get_logger(__name__)
@dataclass(slots=True)
class InvalidateFilter:
"""Selection criteria for a mass invalidation operation.
At least one of topic_codes / entity_canonicals / claim_pattern / since
must be non-empty — calling with all-empty filters is rejected to avoid
accidental "flush everything".
Attributes:
topic_codes: Match rows whose ``topic_codes && this`` is true (any
overlap). E.g., ``["war", "elections"]``.
entity_canonicals: Match rows whose ``entity_bindings`` JSONB
includes a triple that lower-cases to one of these
``"<subject> <predicate> <object>"`` canonical strings.
Caller is responsible for normalizing (lowercasing, predicate
snake_case'd, etc.) — see ``fact_status.canonicalize_triple``.
claim_pattern: ILIKE pattern on content_preview / claim text.
since: Match rows created or updated AFTER this timestamp.
invalidate_gold: If true, also expire gold rows. Default false.
dry_run: If true, count matches without modifying anything.
"""
topic_codes: list[str] | None = None
entity_canonicals: list[str] | None = None
claim_pattern: str | None = None
since: datetime | None = None
invalidate_gold: bool = False
dry_run: bool = False
def is_empty(self) -> bool:
"""True if no filter criteria are set — caller must reject."""
return not (
self.topic_codes
or self.entity_canonicals
or self.claim_pattern
or self.since
)
@dataclass(slots=True)
class InvalidateResult:
"""Counts and metadata returned by ``invalidate_caches``."""
invalidated_atoms: int
invalidated_vcache: int
dry_run: bool
filters_applied: dict[str, Any]
executed_at: datetime
def _entity_match_clause(idx: int) -> str:
"""JSONB existence clause matching any binding's canonical form.
Caller pre-normalizes inputs to lowercase
``"<subject> <predicate_snake> <object>"`` and passes them as a text[].
PG just concatenates the binding fields with the same shape and
compares — no extensions needed.
"""
return (
"EXISTS ("
" SELECT 1 FROM jsonb_array_elements(entity_bindings) AS b "
" WHERE lower(coalesce(b->>'subject','')) || ' ' || "
" lower(replace(coalesce(b->>'predicate',''), ' ', '_')) || ' ' || "
" lower(coalesce(b->>'object','')) "
f" = ANY(${idx}::text[])"
")"
)
def _build_atom_where_clause(
f: InvalidateFilter, params: list[Any]
) -> str:
"""Compose WHERE clause + side-effects on ``params`` for analysis atoms.
Returns a SQL fragment starting with ``WHERE`` (always non-empty since
is_empty() is checked upstream).
"""
clauses: list[str] = ["(expires_at IS NULL OR expires_at > now())"]
if not f.invalidate_gold:
clauses.append("cache_tier <> 'gold'")
if f.topic_codes:
params.append(f.topic_codes)
clauses.append(f"topic_codes && ${len(params)}::text[]")
if f.entity_canonicals:
# Caller-side canonicalization (lowercased "subject predicate object")
# — see fact_status.canonicalize_triple. PG just does string match
# against entity_bindings JSONB without needing unaccent/digest.
params.append(f.entity_canonicals)
clauses.append(_entity_match_clause(len(params)))
if f.claim_pattern:
params.append(f"%{f.claim_pattern}%")
clauses.append(f"content_preview ILIKE ${len(params)}")
if f.since is not None:
params.append(f.since)
clauses.append(f"updated_at >= ${len(params)}")
return "WHERE " + " AND ".join(clauses)
def _build_vcache_where_clause(
f: InvalidateFilter, params: list[Any]
) -> str:
"""Compose WHERE clause for verification cache (no cache_tier here)."""
clauses: list[str] = ["expires_at > now()"]
if f.topic_codes:
params.append(f.topic_codes)
clauses.append(f"topic_codes && ${len(params)}::text[]")
if f.entity_canonicals:
params.append(f.entity_canonicals)
clauses.append(_entity_match_clause(len(params)))
if f.claim_pattern:
params.append(f"%{f.claim_pattern}%")
# vcache stores the claim text only as a hash — match against the
# processed verification payload as a fallback.
clauses.append(
f"verification_processed::text ILIKE ${len(params)}"
)
if f.since is not None:
params.append(f.since)
clauses.append(f"updated_at >= ${len(params)}")
return "WHERE " + " AND ".join(clauses)
async def invalidate_caches(
f: InvalidateFilter,
*,
actor: str = "admin",
reason: str | None = None,
) -> InvalidateResult:
"""Invalidate rows in both cache tables matching the filter.
Args:
f: The selection criteria. Must not be empty (raises ValueError).
actor: Free-form label for the audit log (e.g., 'admin:foo@bar',
'breaking_watcher', 'auditor').
reason: Optional human-readable note for audit trail.
Returns:
InvalidateResult with counts and the filters that were applied.
Raises:
ValueError: If the filter is empty (no criteria set).
RuntimeError: If brain_db is not connected.
"""
if f.is_empty():
raise ValueError(
"invalidate filter is empty — refusing to flush everything"
)
if not db.pool:
raise RuntimeError("brain_db not connected")
now = datetime.now(tz=timezone.utc)
# Count first (always — even non-dry-run runs the count for the audit log).
atom_params: list[Any] = []
atom_where = _build_atom_where_clause(f, atom_params)
vcache_params: list[Any] = []
vcache_where = _build_vcache_where_clause(f, vcache_params)
async with db.pool.acquire() as conn:
# COUNT pre-update so we know how many rows we'll touch.
atom_count_row = await conn.fetchrow(
f"SELECT COUNT(*) AS c FROM brain_analysis_atom {atom_where}",
*atom_params,
)
vcache_count_row = await conn.fetchrow(
f"SELECT COUNT(*) AS c FROM brain_verification_cache {vcache_where}",
*vcache_params,
)
atom_count = int(atom_count_row["c"]) if atom_count_row else 0
vcache_count = int(vcache_count_row["c"]) if vcache_count_row else 0
if not f.dry_run and (atom_count > 0 or vcache_count > 0):
async with conn.transaction():
if atom_count > 0:
await conn.execute(
f"UPDATE brain_analysis_atom SET expires_at = now(), "
f"updated_at = now() {atom_where}",
*atom_params,
)
if vcache_count > 0:
await conn.execute(
f"UPDATE brain_verification_cache SET expires_at = now(), "
f"updated_at = now() {vcache_where}",
*vcache_params,
)
payload = {
"filter": {
"topic_codes": f.topic_codes,
"entity_canonicals_count": (
len(f.entity_canonicals)
if f.entity_canonicals
else 0
),
"claim_pattern": f.claim_pattern,
"since": f.since.isoformat() if f.since else None,
"invalidate_gold": f.invalidate_gold,
},
"counts": {
"atoms": atom_count,
"vcache": vcache_count,
},
"reason": reason,
}
await conn.execute(
"""
INSERT INTO brain_audit_log (action, target_table, target_id, actor, payload)
VALUES ('invalidate', 'multi', 'mass', $1, $2::jsonb)
""",
actor,
json.dumps(payload),
)
log.info(
"cache_invalidated",
atom_count=atom_count,
vcache_count=vcache_count,
dry_run=f.dry_run,
actor=actor,
topic_codes=f.topic_codes,
invalidate_gold=f.invalidate_gold,
)
return InvalidateResult(
invalidated_atoms=atom_count,
invalidated_vcache=vcache_count,
dry_run=f.dry_run,
filters_applied={
"topic_codes": f.topic_codes,
"entity_canonicals_count": (
len(f.entity_canonicals) if f.entity_canonicals else 0
),
"claim_pattern": f.claim_pattern,
"since": f.since.isoformat() if f.since else None,
"invalidate_gold": f.invalidate_gold,
},
executed_at=now,
)

321
ai_platform/modules/didi_brain/brain_api/services/mapping.py Normal file
View file

@ -0,0 +1,321 @@
"""Translate DidiBrain atoms into Didi's EvidenceItem / FetchedPage shapes.
The trick here is that the brain stores TWO kinds of atoms:
- Type/Document → the full source article (parent)
- Type/Claim → an atomic factual claim extracted from a parent
Didi's response shape expects evidence AT THE DOCUMENT LEVEL (url, title,
full_text). So we:
1. Run semantic + rerank at claim level (precision)
2. Group hits by parent document URL
3. Emit one EvidenceItem per distinct parent, with the best-scoring claim
attached as `summary` and supporting data in `brain_meta`
Credibility tags in our taxonomy (Tier1/Tier2/Tier3/StateAffiliated/KnownDisinfo)
map to numeric scores that mirror Didi's web-module output range (0..1).
"""
from __future__ import annotations
import hashlib
import re
from collections import defaultdict
from datetime import datetime, timezone
from typing import Any
from urllib.parse import unquote, urlparse
from brain_api.schemas import (
BrainEvidenceMeta,
EvidenceItem,
FetchedPage,
Provenance,
SearchResultItem,
)
from shared.atomic_api import SearchHit
# --- credibility mapping ---------------------------------------------------
CREDIBILITY_SCORES: dict[str, float] = {
"Tier1": 0.90,
"Tier2": 0.70,
"Tier3": 0.50,
"StateAffiliated": 0.40,
"KnownDisinfo": 0.15,
"Unknown": 0.50,
}
def tag_to_credibility_score(tags: list[dict[str, Any]]) -> float:
"""Pick the highest-priority credibility tag and map to a score."""
for t in tags:
name = (t.get("name") or "").strip()
if name in CREDIBILITY_SCORES:
return CREDIBILITY_SCORES[name]
return CREDIBILITY_SCORES["Unknown"]
# --- parent URL / publisher ------------------------------------------------
def parent_url_of(source_url: str | None) -> str:
"""Strip `#claim=...` fragment from a claim atom's source_url."""
if not source_url:
return ""
return source_url.split("#", 1)[0]
def publisher_of(url: str) -> str:
try:
host = urlparse(url).hostname or ""
except ValueError:
return ""
if host.startswith("www."):
host = host[4:]
return host
def title_from_url(url: str) -> str:
if not url:
return ""
slug = url.rstrip("/").rsplit("/", 1)[-1]
return unquote(slug).replace("_", " ")
# --- claim atom body parsing -----------------------------------------------
_CLAIM_BODY_RE = re.compile(r"^# Claim\s*\n+(.+?)\n+##", re.DOTALL | re.MULTILINE)
_STANCE_RE = re.compile(r"Stance in source:\s*(\w+)", re.IGNORECASE)
_PARENT_ID_RE = re.compile(r"Parent atom:\s*`([^`]+)`")
def parse_claim_atom_body(content: str) -> tuple[str, str, str]:
"""Return (claim_text, stance, parent_atom_id) from a Type/Claim markdown body."""
claim_text = ""
stance = "NEUTRAL"
parent_id = ""
m = _CLAIM_BODY_RE.search(content)
if m:
claim_text = m.group(1).strip()
s = _STANCE_RE.search(content)
if s:
stance = s.group(1).strip().upper()
p = _PARENT_ID_RE.search(content)
if p:
parent_id = p.group(1).strip()
return claim_text, stance, parent_id
# --- document title from content header -----------------------------------
def title_from_content(content: str | None, fallback_url: str = "") -> str:
if not content:
return title_from_url(fallback_url)
for line in content.lstrip().splitlines():
stripped = line.strip()
if stripped.startswith("# "):
return stripped[2:].strip()
if stripped:
break
return title_from_url(fallback_url)
def sha256_hex(text: str) -> str:
return hashlib.sha256(text.encode("utf-8", errors="replace")).hexdigest()
# --- aggregated per-parent ------------------------------------------------
def group_hits_by_parent(
hits: list[SearchHit], rerank_scores: dict[str, float]
) -> list[tuple[str, list[tuple[SearchHit, float]]]]:
"""Return ordered list of (parent_url, [(hit, rerank_score), ...]).
Order is by the best rerank_score within each parent, descending.
If a hit has no rerank score (wasn't in top-N), it falls to the end.
"""
buckets: dict[str, list[tuple[SearchHit, float]]] = defaultdict(list)
for h in hits:
parent = parent_url_of(h.source_url)
if not parent:
continue
rr = rerank_scores.get(h.atom_id, 0.0)
buckets[parent].append((h, rr))
# sort each bucket: highest rerank first, then highest embedding sim
for parent in buckets:
buckets[parent].sort(key=lambda x: (-x[1], -x[0].similarity))
# sort buckets by their top entry's rerank score, descending
ordered = sorted(
buckets.items(),
key=lambda kv: (-kv[1][0][1], -kv[1][0][0].similarity),
)
return ordered
# --- build EvidenceItem from a parent doc + claim matches ------------------
def evidence_from_parent(
*,
parent_atom: dict[str, Any],
claim_hits: list[tuple[SearchHit, float]],
parent_full_atoms: dict[str, dict[str, Any]],
include_full_text: bool,
nli_by_atom_id: dict[str, tuple[str, float, str | None]] | None = None,
) -> EvidenceItem:
"""Build an EvidenceItem given one parent document and its best-matching claim atoms.
`parent_atom` is the parent Type/Document atom (with full content).
`claim_hits` are (SearchHit, rerank_score) for claims belonging to this parent,
pre-sorted descending.
`parent_full_atoms` is an already-fetched map of full atom bodies so we can
pull the claim text from each matching claim atom.
"""
parent_url = parent_atom.get("source_url") or ""
parent_id = parent_atom.get("id") or ""
parent_content = parent_atom.get("content") or ""
title = title_from_content(parent_content, parent_url)
# Best matching claim (for summary + brain_meta)
best_hit, best_rerank = claim_hits[0]
best_full = parent_full_atoms.get(best_hit.atom_id) or {}
best_claim_text, best_stance, _best_parent = parse_claim_atom_body(
best_full.get("content") or ""
)
# Snippet: first paragraph of the parent, trimmed
snippet = (
parent_content.strip().split("\n\n", 1)[0][:300]
if parent_content
else None
)
# Dates — prefer published_at from the parent; fall back to created_at; never null
published_at = _parse_dt(parent_atom.get("published_at"))
retrieved_at = _parse_dt(parent_atom.get("created_at")) or datetime.now(timezone.utc)
# Credibility from parent's tag set
credibility = tag_to_credibility_score(parent_atom.get("tags") or [])
# NLI stance vs query — only attached to the BEST claim (the one we
# already surface as `summary`), since that's the one Didi will show.
nli_label = "UNKNOWN"
nli_conf = 0.0
nli_err: str | None = None
if nli_by_atom_id is not None:
entry = nli_by_atom_id.get(best_hit.atom_id)
if entry is not None:
nli_label, nli_conf, nli_err = entry
# Brain meta: one per evidence item, holds every matching claim's info
brain_meta = BrainEvidenceMeta(
parent_atom_id=parent_id,
matching_claim_atom_ids=[h.atom_id for h, _ in claim_hits],
best_claim_text=best_claim_text,
best_claim_stance_in_source=best_stance,
best_claim_hash=(best_hit.source_url or "").split("#claim=", 1)[-1][:16],
claim_count=len(claim_hits),
reranker_score=best_rerank,
embedding_similarity=best_hit.similarity,
stance_vs_query=nli_label,
nli_confidence=nli_conf,
nli_error=nli_err,
)
full_text = parent_content if include_full_text else None
return EvidenceItem(
url=parent_url,
canonical_url=parent_url or None,
title=title,
publisher=publisher_of(parent_url),
published_at=published_at,
retrieved_at=retrieved_at,
snippet=snippet,
summary=best_claim_text or None,
full_text=full_text,
full_text_hash=sha256_hex(parent_content) if parent_content else "",
provenance=Provenance(
extraction_method="brain",
fallback_chain=[],
brain_meta=brain_meta,
),
relevance_score=round(best_rerank or best_hit.similarity, 4),
credibility_score=credibility,
)
def _parse_dt(value: Any) -> datetime | None:
if not value:
return None
if isinstance(value, datetime):
return value
try:
text = str(value).replace("Z", "+00:00")
return datetime.fromisoformat(text)
except (ValueError, TypeError):
return None
# --- doc atom → FetchedPage ------------------------------------------------
def doc_to_fetched_page(
full_atom: dict[str, Any], *, include_html: bool = False
) -> FetchedPage:
url = full_atom.get("source_url") or ""
content = full_atom.get("content") or ""
return FetchedPage(
url=url,
canonical_url=url or None,
title=title_from_content(content, url),
text=content,
text_hash=sha256_hex(content),
html=None if not include_html else content,
extraction_method="brain",
fallback_chain=[],
published_at=_parse_dt(full_atom.get("published_at")),
retrieved_at=_parse_dt(full_atom.get("created_at")) or datetime.now(timezone.utc),
extraction_time_ms=0.0,
warnings=[],
needs_fallback=False,
status_code=200,
content_type="text/markdown",
)
# --- doc atom → SearchResultItem -------------------------------------------
def doc_to_search_result(
atom: dict[str, Any], *, query: str, rank: int
) -> SearchResultItem:
url = atom.get("source_url") or ""
title = title_from_content(atom.get("content"), url)
snippet = atom.get("snippet") or ""
if not snippet and atom.get("content"):
snippet = (atom["content"] or "").strip().split("\n\n", 1)[0][:200]
return SearchResultItem(
query=query,
url=url,
title=title,
snippet=snippet,
rank=rank,
site=publisher_of(url),
published_at=_parse_dt(atom.get("published_at")),
)
# --- language detection (tiny heuristic) ----------------------------------
_RO_CHARS = set("ăâîșțĂÂÎȘȚşţŞŢ")
def detect_language_simple(text: str) -> str:
if not text:
return "en"
if any(c in _RO_CHARS for c in text):
return "ro"
return "en"

145
ai_platform/modules/didi_brain/brain_api/services/nli.py Normal file
View file

@ -0,0 +1,145 @@
"""NLI stance classification: is this evidence supporting, contradicting,
or neutral relative to the user's claim?
This is separate from `stance_in_source` (what the original source asserts
about itself). For disinfo analysis, the question Didi's backend really
needs answered is: "does this evidence back the user's claim or refute it?"
Implementation:
- one LLM call per (claim, evidence) pair
- async + parallel across top-N evidence items
- returns a stance label and a confidence
- uses the versioned prompt at brain_api/prompts/nli_v1.md
"""
from __future__ import annotations
import asyncio
from dataclasses import dataclass
from pathlib import Path
from shared.config import LlmRole
from shared.llm_client import LlmClient, LlmError
from shared.logging import get_logger
log = get_logger(__name__)
PROMPT_VERSION = "v1"
_PROMPT_PATH = Path(__file__).resolve().parent.parent / "prompts" / f"nli_{PROMPT_VERSION}.md"
ALLOWED_LABELS = {"SUPPORTS", "CONTRADICTS", "NEUTRAL"}
MAX_EVIDENCE_CHARS = 1500 # truncate long evidence before sending to the NLI model
# Match the effective llama.cpp backend concurrency: we have two instances
# behind the router (10.11.10.18 and 10.11.10.19), each serves one request
# at a time. Flooding with more parallel calls just queues them on the
# backend and hits our per-call timeout.
MAX_PARALLEL = 2
PER_CALL_TIMEOUT_S = 30.0 # generous; queuing + generation
TOTAL_TIMEOUT_S = 60.0 # wall-clock for the whole batch
_PROMPT_TEMPLATE: str | None = None
def _load_prompt() -> str:
global _PROMPT_TEMPLATE
if _PROMPT_TEMPLATE is None:
_PROMPT_TEMPLATE = _PROMPT_PATH.read_text(encoding="utf-8")
return _PROMPT_TEMPLATE
@dataclass(slots=True, frozen=True)
class NliResult:
label: str # SUPPORTS / CONTRADICTS / NEUTRAL
confidence: float # 0.0 - 1.0
error: str | None = None
async def classify_one(
llm: LlmClient, *, claim: str, evidence: str
) -> NliResult:
"""Classify a single (claim, evidence) pair. Never raises — returns
NeutralResult with error populated on failure so the caller can still
produce a response for that evidence item."""
if not evidence.strip():
return NliResult(label="NEUTRAL", confidence=0.0, error="empty_evidence")
truncated = evidence[:MAX_EVIDENCE_CHARS]
prompt = _load_prompt().replace("{claim}", claim).replace("{evidence}", truncated)
try:
result, _usage = await asyncio.wait_for(
llm.chat_json(
role=LlmRole.REASONING,
system=(
"You are an NLI classifier. Respond with strictly valid "
"JSON only, no commentary."
),
user=prompt,
max_tokens=120,
temperature=0.0,
),
timeout=PER_CALL_TIMEOUT_S,
)
except asyncio.TimeoutError:
return NliResult(label="NEUTRAL", confidence=0.0, error="timeout")
except LlmError as e:
return NliResult(label="NEUTRAL", confidence=0.0, error=f"llm:{e}")
except Exception as e: # noqa: BLE001
return NliResult(label="NEUTRAL", confidence=0.0, error=f"{type(e).__name__}:{e}")
if not isinstance(result, dict):
return NliResult(label="NEUTRAL", confidence=0.0, error="non_dict_response")
raw_label = (result.get("label") or "").strip().upper()
try:
conf = float(result.get("confidence", 0))
except (TypeError, ValueError):
conf = 0.0
conf = max(0.0, min(1.0, conf))
if raw_label not in ALLOWED_LABELS:
return NliResult(
label="NEUTRAL",
confidence=0.0,
error=f"bad_label:{raw_label[:40]}",
)
return NliResult(label=raw_label, confidence=conf)
async def classify_batch(
llm: LlmClient,
*,
claim: str,
evidence_texts: list[str],
max_parallel: int = MAX_PARALLEL,
) -> list[NliResult]:
"""Classify many evidence items in parallel, preserving input order.
Concurrency is bounded by `max_parallel` so we don't hammer the LLM
router. Individual failures produce a NeutralResult with an error field
(never raises to the caller).
"""
if not evidence_texts:
return []
sem = asyncio.Semaphore(max_parallel)
async def _guarded(ev: str) -> NliResult:
async with sem:
return await classify_one(llm, claim=claim, evidence=ev)
try:
results = await asyncio.wait_for(
asyncio.gather(*[_guarded(ev) for ev in evidence_texts]),
timeout=TOTAL_TIMEOUT_S,
)
except asyncio.TimeoutError:
log.warning("nli_batch_total_timeout", count=len(evidence_texts))
return [
NliResult(label="NEUTRAL", confidence=0.0, error="batch_timeout")
for _ in evidence_texts
]
return list(results)

93
ai_platform/modules/didi_brain/brain_api/services/search.py Normal file
View file

@ -0,0 +1,93 @@
"""POST /v1/search — thin retrieval that returns a flat list of results.
Unlike /v1/gather, we do NOT rerank or group — just semantic search the brain
and convert each document-level hit into a SearchResultItem. This is the
equivalent of a search engine result list; callers that want ranked evidence
should hit /v1/gather instead.
"""
from __future__ import annotations
import asyncio
import time
import uuid
from brain_api.schemas import (
BrainMeta,
SearchRequest,
SearchResponse,
SearchResultItem,
)
from brain_api.services.mapping import doc_to_search_result, parent_url_of
from shared.atomic_api import AtomicClient
from shared.logging import get_logger
log = get_logger(__name__)
async def search(
req: SearchRequest, *, atomic: AtomicClient
) -> SearchResponse:
t0 = time.perf_counter()
request_id = str(uuid.uuid4())
# Run queries in parallel; merge into a flat ranked list.
tasks = [
atomic.search(q, mode="semantic", limit=req.max_results, threshold=0.2)
for q in req.queries
]
per_query_hits = await asyncio.gather(*tasks, return_exceptions=True)
# We need full document atoms (parents, de-duped by URL) to render results
seen_urls: set[str] = set()
results: list[SearchResultItem] = []
parent_atom_cache: dict[str, dict] = {}
for query_str, query_hits in zip(req.queries, per_query_hits, strict=True):
if isinstance(query_hits, Exception):
log.warning("search_query_failed", query=query_str, error=str(query_hits))
continue
# Collect parent URLs from this query in order
ordered_parents: list[str] = []
for h in query_hits:
parent = parent_url_of(h.source_url)
if not parent or parent in seen_urls:
continue
seen_urls.add(parent)
ordered_parents.append(parent)
if len(ordered_parents) >= req.max_results:
break
# Fetch any parent docs we haven't seen yet, in parallel
need = [p for p in ordered_parents if p not in parent_atom_cache]
if need:
atoms = await asyncio.gather(
*[atomic.get_atom_by_source_url(u) for u in need],
return_exceptions=True,
)
for url, atom in zip(need, atoms, strict=True):
if isinstance(atom, dict):
parent_atom_cache[url] = atom
for rank, url in enumerate(ordered_parents, start=len(results) + 1):
atom = parent_atom_cache.get(url)
if not atom:
continue
results.append(doc_to_search_result(atom, query=query_str, rank=rank))
total_ms = round((time.perf_counter() - t0) * 1000, 1)
return SearchResponse(
request_id=request_id,
results=results,
total_results=len(results),
execution_time_ms=total_ms,
queries_processed=len(req.queries),
brain_meta=BrainMeta(
cache_status="HIT" if results else "MISS",
api_version="v1",
implementation="didibrain",
evidence_sources=len({r.url for r in results}),
total_claim_atoms_matched=0,
),
)

186
ai_platform/modules/didi_brain/brain_api/services/topic_volatility.py Normal file
View file

@ -0,0 +1,186 @@
"""Topic volatility overrides — Phase D1.
Reads admin-configured volatility/TTL/recency per topic from didiFramework's
sensitive_topic table (proxied via HTTP to keep brain free of a Redis
dependency). The classifier consults this map AFTER its LLM call: if any of
the LLM-derived topic_codes matches an admin-configured topic, the admin's
values override the LLM estimates for that topic.
Resolution order for a claim's effective TTL:
1. classifier returns volatility + estimated_validity_hours (LLM judgment)
2. for each LLM-detected topic_code, fetch admin override from this module
3. if admin override exists, use the more conservative of (LLM, admin) — i.e.
pick the SHORTER TTL; admins can tighten brain's own estimate but never
loosen it (a stable claim that touches an admin-tagged 'volatile' topic
gets the volatile TTL)
Cache: in-process, 60s TTL. Failures (didiFramework down, HTTP timeout) leave
the cache empty so callers fall through to LLM-only behavior — never blocks.
"""
from __future__ import annotations
import asyncio
import os
import time
from dataclasses import dataclass
import httpx
from shared.logging import get_logger
log = get_logger(__name__)
DIDI_FRAMEWORK_URL = os.environ.get(
"DIDI_FRAMEWORK_URL", "http://didi-framework:3005"
).rstrip("/")
CACHE_TTL_S = 60.0
HTTP_TIMEOUT_S = 5.0
@dataclass(slots=True, frozen=True)
class TopicConfig:
"""Admin-configured policy for one topic.
Attributes:
topic_code: Canonical code (matches classifier's topic_codes output).
volatility: One of "volatile", "evolving", "stable".
cache_ttl_hours: Hard cap on cache TTL for verdicts touching this topic.
recency_window_days: For volatile/evolving topics, drop evidence older
than this in /v1/gather.
half_life_days: Recency-boost half-life used in combined ranking.
"""
topic_code: str
volatility: str
cache_ttl_hours: int
recency_window_days: int
half_life_days: float
_cache: dict[str, TopicConfig] | None = None
_cache_loaded_at: float = 0.0
_cache_lock = asyncio.Lock()
async def _fetch_from_framework() -> dict[str, TopicConfig]:
"""Pull active topics from didiFramework. Empty dict on any failure."""
url = f"{DIDI_FRAMEWORK_URL}/api/sensitive-topics?active=true"
try:
async with httpx.AsyncClient(timeout=HTTP_TIMEOUT_S) as client:
resp = await client.get(url)
if resp.status_code >= 400:
log.debug(
"topic_overrides_http_error",
status=resp.status_code,
)
return {}
payload = resp.json()
except Exception as e: # noqa: BLE001
log.debug(
"topic_overrides_fetch_failed",
error=f"{type(e).__name__}:{e}",
)
return {}
if not isinstance(payload, dict) or not payload.get("success"):
return {}
rows = payload.get("data") or []
if not isinstance(rows, list):
return {}
out: dict[str, TopicConfig] = {}
for row in rows:
if not isinstance(row, dict):
continue
code = row.get("topic_code")
vol = row.get("volatility")
if not isinstance(code, str) or vol not in (
"volatile",
"evolving",
"stable",
):
continue
try:
out[code] = TopicConfig(
topic_code=code,
volatility=vol,
cache_ttl_hours=int(row.get("cache_ttl_hours") or 720),
recency_window_days=int(row.get("recency_window_days") or 30),
half_life_days=float(row.get("half_life_days") or 30.0),
)
except (TypeError, ValueError):
continue
return out
async def get_topic_overrides() -> dict[str, TopicConfig]:
"""Return current admin-configured topic policies (cached 60s).
Always returns a dict — empty if didiFramework is unreachable or
sensitive_topic doesn't have the volatility columns yet (migration 012
not run). Callers can iterate over it freely.
"""
global _cache, _cache_loaded_at
now = time.time()
if _cache is not None and (now - _cache_loaded_at) < CACHE_TTL_S:
return _cache
async with _cache_lock:
# Double-check inside the lock.
if _cache is not None and (time.time() - _cache_loaded_at) < CACHE_TTL_S:
return _cache
fresh = await _fetch_from_framework()
_cache = fresh
_cache_loaded_at = time.time()
return fresh
def invalidate_cache() -> None:
"""Force a refetch on next ``get_topic_overrides()`` call.
Called by admin endpoints after a topic is mutated in didiFramework so
the change propagates without waiting for the 60s cache window.
"""
global _cache
_cache = None
def reconcile_with_classifier(
*,
classifier_volatility: str,
classifier_validity_hours: int,
classifier_topics: list[str],
overrides: dict[str, TopicConfig],
) -> tuple[str, int]:
"""Combine LLM classifier output with admin overrides.
Picks the MORE conservative (shorter) TTL when admin override exists.
Volatility ranking: volatile < evolving < stable (volatile = shorter
"shelf life"). If admin says 'volatile' for any matching topic, the
final volatility is 'volatile' regardless of what the classifier said.
Returns:
(effective_volatility, effective_ttl_hours).
"""
if not overrides or not classifier_topics:
return classifier_volatility, classifier_validity_hours
rank = {"volatile": 0, "evolving": 1, "stable": 2}
eff_vol = classifier_volatility
eff_ttl = classifier_validity_hours
for topic in classifier_topics:
cfg = overrides.get(topic)
if cfg is None:
continue
# Pick the more conservative volatility (lower rank wins).
if rank.get(cfg.volatility, 1) < rank.get(eff_vol, 1):
eff_vol = cfg.volatility
# Pick the shorter TTL.
if cfg.cache_ttl_hours < eff_ttl:
eff_ttl = cfg.cache_ttl_hours
return eff_vol, eff_ttl

566
ai_platform/modules/didi_brain/brain_api/services/verification_cache.py Normal file
View file

@ -0,0 +1,566 @@
"""Verification cache — store the LLM verification result per (claim, tier).
Contract agreed with didi-backend: backend runs its own LLM verification call
(prompt + model are owned by backend side via Redis config), then POSTs the
result here fire-and-forget. On the next /v1/gather for the same claim + tier,
brain returns the cached payload verbatim in brain_meta.
Schema v2 change:
- v1 keyed on (claim_hash, evidence_hash, tier) — unreachable because
evidence URLs at gather read-time rarely match those at write-time
(brain's live search ranks/filters differently than backend's original
source list).
- v2 keys on (claim_hash, tier). evidence_hash + evidence_urls are kept
as metadata on the stored row; backend uses them at read-time to
decide overlap with its current evidence set.
This module owns:
- normalization + hashing (claim + urls; urls hash is metadata now)
- Upsert writes with TTL
- Staleness detection via prompt_hash and framework_version
"""
from __future__ import annotations
import asyncio
import hashlib
import json
import unicodedata
from dataclasses import dataclass, field
from datetime import datetime, timedelta, timezone
from typing import Any, Literal
from brain_api.db import db
from brain_api.services.classifier import (
ClaimVolatility,
classify_claim_volatility,
)
from shared.config import settings
from shared.llm_client import LlmClient
from shared.logging import get_logger
log = get_logger(__name__)
# ----------------------------------------------------------------------- hashing
def normalize_claim(s: str) -> str:
"""Hash-input normalization agreed with backend.
Same variations collapse to one bucket:
- "România a câștigat 9 medalii." (RO, punctuated)
- "romania a castigat 9 medalii" (stripped diacritics)
- "Romania a câștigat 9 medalii!" (extra whitespace, exclam)
Different intents stay separate:
- negation ("nu e sigur" vs "e sigur")
- numbers ("9 medalii" vs "10 medalii")
- middle punctuation ("X, ironic")
"""
s = unicodedata.normalize("NFKD", s)
s = "".join(c for c in s if not unicodedata.combining(c))
s = s.lower()
s = " ".join(s.split())
s = s.rstrip(".?!")
return s
def hash_claim(claim: str) -> str:
return hashlib.sha256(normalize_claim(claim).encode("utf-8")).hexdigest()
def hash_evidence_urls(urls: list[str]) -> str:
"""Order-independent, case-insensitive canonical hash over a URL set.
Used purely as metadata now (not part of the cache key) so backend can
detect corpus drift without needing to recompute locally.
"""
canonical = sorted({u.strip().lower().rstrip("/") for u in urls if u})
return hashlib.sha256("\n".join(canonical).encode("utf-8")).hexdigest()
# --------------------------------------------------------------------- dataclass
@dataclass(slots=True)
class CacheEntry:
claim_hash: str
tier: str
evidence_hash: str
evidence_urls: list[str]
model: str | None
prompt_hash: str
framework_version: str | None
schema_name: str
verification_raw: dict[str, Any] | None
verification_processed: dict[str, Any]
created_at: datetime
updated_at: datetime
expires_at: datetime
# Phase B1+B2 metadata (may be missing for legacy rows written before
# the volatility migration — defaults are conservative).
volatility: str | None = None
topic_codes: list[str] = field(default_factory=list)
entity_bindings: list[dict[str, Any]] = field(default_factory=list)
consecutive_audit_passes: int = 0
last_audited_at: datetime | None = None
# -------------------------------------------------------------------------- IO
async def _register_facts_async(
classification: ClaimVolatility | None, *, source_label: str
) -> None:
"""Best-effort fact registration after a successful verification cache write.
Lazy import avoids a circular dependency (fact_status imports classifier).
"""
if classification is None or not classification.entity_bindings:
return
try:
from brain_api.services.fact_status import register_facts_from_bindings
await register_facts_from_bindings(
classification.entity_bindings,
volatility=classification.volatility,
topic_codes=classification.topic_codes,
source_atom_id=source_label,
)
except Exception as e: # noqa: BLE001
log.warning(
"vcache_fact_registration_failed",
error=f"{type(e).__name__}:{e}",
)
def _resolve_ttl_days(
ttl_days: int | None, classification: ClaimVolatility | None
) -> int:
"""Pick the effective TTL in days, prefering classifier estimate.
When classification is fresh (not degraded), its hour estimate is
converted to whole days (rounded up) so a volatile 6h claim yields ttl=1
day, never 30 days. Otherwise we fall back to either the explicit
``ttl_days`` argument or the global verification_cache_ttl_days setting.
"""
if classification is not None and not classification.degraded:
# Round up to whole days, but never below 1 day.
days_from_classifier = max(
1, (classification.estimated_validity_hours + 23) // 24
)
return days_from_classifier
if ttl_days is not None:
return ttl_days
return settings.verification_cache_ttl_days
async def upsert(
*,
claim: str,
evidence_urls: list[str],
tier: Literal["free", "premium"],
prompt_hash: str,
verification_processed: dict[str, Any],
verification_raw: dict[str, Any] | None = None,
model: str | None = None,
framework_version: str | None = None,
schema_name: str = "didi-v1",
ttl_days: int | None = None,
classification: ClaimVolatility | None = None,
llm: LlmClient | None = None,
) -> CacheEntry:
"""Last-wins upsert on (claim_hash, tier), with volatility classification.
Pipeline:
1. If no ``classification`` provided and an ``llm`` client is, run the
volatility classifier on the claim text. This drives TTL and adds
topic_codes / entity_bindings metadata for invalidation by topic
and for fact_status registration.
2. UPSERT row, including the new metadata columns.
3. Fire-and-forget fact registration after successful write.
Multiple verification runs for the same claim+tier (different evidence
sets, rerun on model fallback) all write into the same row; the latest
successful verification wins.
Args:
claim: Original user claim text. Hashed for the cache key and also
passed to the classifier.
evidence_urls: URLs the LLM verification ran over (metadata only).
tier: free | premium.
prompt_hash: sha256[:12] of the verification prompt template.
verification_processed: Canonical mapped verdict (status, confidence,
etc.) — what callers serve from cache.
verification_raw: Raw LLM response (used by stale_framework recompute).
model: LLM model identifier.
framework_version: sha256[:12] of relevant framework configs.
schema_name: Versioned schema label, default "didi-v1".
ttl_days: Caller-provided TTL override; ignored if a classification
with non-degraded estimate is available.
classification: Pre-computed ClaimVolatility from caller.
llm: LLM client for classifier. None disables classification.
"""
classification = await _maybe_classify(
classification=classification, llm=llm, claim=claim
)
ttl = _resolve_ttl_days(ttl_days, classification)
expires_at = datetime.now(tz=timezone.utc) + timedelta(days=ttl)
ch = hash_claim(claim)
eh = hash_evidence_urls(evidence_urls)
ev_urls_json = json.dumps(list(evidence_urls))
raw_json = (
json.dumps(verification_raw) if verification_raw is not None else None
)
processed_json = json.dumps(verification_processed)
volatility = classification.volatility if classification else None
topic_codes = classification.topic_codes if classification else []
entity_bindings_json = (
json.dumps(classification.entity_bindings_jsonb())
if classification
else "[]"
)
ttl_hours_used = ttl * 24
sql = """
INSERT INTO brain_verification_cache (
claim_hash, tier,
evidence_hash, evidence_urls,
model, prompt_hash, framework_version, schema_name,
verification_raw, verification_processed,
expires_at,
volatility, topic_codes, entity_bindings, ttl_hours_used
)
VALUES (
$1, $2, $3, $4::jsonb, $5, $6, $7, $8, $9::jsonb, $10::jsonb, $11,
$12, $13, $14::jsonb, $15
)
ON CONFLICT (claim_hash, tier) DO UPDATE SET
evidence_hash = EXCLUDED.evidence_hash,
evidence_urls = EXCLUDED.evidence_urls,
model = EXCLUDED.model,
prompt_hash = EXCLUDED.prompt_hash,
framework_version = EXCLUDED.framework_version,
schema_name = EXCLUDED.schema_name,
verification_raw = EXCLUDED.verification_raw,
verification_processed = EXCLUDED.verification_processed,
updated_at = now(),
expires_at = EXCLUDED.expires_at,
volatility = COALESCE(EXCLUDED.volatility, brain_verification_cache.volatility),
topic_codes = CASE
WHEN array_length(EXCLUDED.topic_codes, 1) > 0
THEN EXCLUDED.topic_codes
ELSE brain_verification_cache.topic_codes
END,
entity_bindings = CASE
WHEN jsonb_array_length(EXCLUDED.entity_bindings) > 0
THEN EXCLUDED.entity_bindings
ELSE brain_verification_cache.entity_bindings
END,
ttl_hours_used = EXCLUDED.ttl_hours_used
RETURNING
claim_hash, tier,
evidence_hash, evidence_urls,
model, prompt_hash, framework_version, schema_name,
verification_raw, verification_processed,
created_at, updated_at, expires_at
"""
async with db.pool.acquire() as conn:
row = await conn.fetchrow(
sql,
ch,
tier,
eh,
ev_urls_json,
model,
prompt_hash,
framework_version,
schema_name,
raw_json,
processed_json,
expires_at,
volatility,
topic_codes,
entity_bindings_json,
ttl_hours_used,
)
assert row is not None # UPSERT with RETURNING always yields a row
entry = _row_to_entry(row)
# Best-effort fact registration after the write succeeds.
if classification and classification.entity_bindings:
asyncio.create_task(
_register_facts_async(
classification, source_label=f"vcache:{ch[:12]}"
)
)
log.info(
"vcache_upsert_ok",
claim_hash=ch[:12],
tier=tier,
volatility=volatility,
ttl_days=ttl,
topic_codes=topic_codes,
binding_count=len(classification.entity_bindings) if classification else 0,
)
return entry
async def _maybe_classify(
*,
classification: ClaimVolatility | None,
llm: LlmClient | None,
claim: str,
) -> ClaimVolatility | None:
"""Use caller's classification or compute one. Never raises."""
if classification is not None:
return classification
if llm is None or not claim.strip():
return None
try:
return await classify_claim_volatility(llm, claim=claim)
except Exception as e: # noqa: BLE001
log.warning(
"vcache_classifier_failed",
error=f"{type(e).__name__}:{e}",
)
return None
# ============================================================================
# Phase B2: confidence decay + judge integration for verification_cache
# ============================================================================
# Same decay model as analysis_atom — keeps the two caches behaviorally
# consistent so callers don't have to special-case.
DECAY_HALF_LIVES_HOURS: dict[str, float] = {
"volatile": 24.0,
"evolving": 168.0,
"stable": float("inf"),
}
AUDIT_HISTORY_MAX = 50
AUDIT_PASS_MIN_INTERVAL_HOURS = 6.0
def compute_effective_confidence(
*,
base_confidence: float | None,
volatility: str | None,
age_hours: float,
consecutive_audit_passes: int = 0,
) -> float | None:
"""Decay base confidence by age, modulated by volatility and audit history.
Mirrors analysis_atom.compute_effective_confidence so the two cache
paths share the same model and admin tooling can reuse formulas.
"""
if base_confidence is None:
return None
half = DECAY_HALF_LIVES_HOURS.get(volatility or "evolving", 168.0)
if half == float("inf") or age_hours <= 0:
decay = 1.0
else:
decay = 0.5 ** (age_hours / half)
audit_boost = min(0.3, 0.03 * max(0, consecutive_audit_passes))
return float(base_confidence) * decay * (1.0 + audit_boost)
async def apply_judge_verdict(
*,
claim_hash: str,
tier: Literal["free", "premium"],
verdict: object, # JudgeVerdict — typed loosely to avoid circular import
) -> None:
"""Persist a JudgeVerdict to the verification_cache row.
Updates audit_history (capped), last_audited_at, consecutive_audit_passes
(rate-limited), and expires_at on INVALIDATE. Also writes a brain_audit_log
entry for telemetry.
"""
if not db.pool:
raise RuntimeError("brain_db not connected")
from brain_api.services.cache_judge import JudgeVerdict
if not isinstance(verdict, JudgeVerdict):
raise TypeError(
f"apply_judge_verdict: expected JudgeVerdict, got {type(verdict).__name__}"
)
audit_entry = verdict.to_audit_entry()
audit_json = json.dumps(audit_entry)
audit_array_json = json.dumps([audit_entry])
sql = """
UPDATE brain_verification_cache
SET
audit_history = (
SELECT jsonb_agg(elem)
FROM (
SELECT elem
FROM jsonb_array_elements(
COALESCE(audit_history, '[]'::jsonb) || $3::jsonb
) WITH ORDINALITY AS t(elem, ord)
ORDER BY ord DESC
LIMIT $4
) recent
),
last_audited_at = now(),
consecutive_audit_passes = CASE
WHEN $5 = 'KEEP_CACHE' AND (
last_audited_at IS NULL
OR last_audited_at < now() - ($6 || ' hours')::interval
)
THEN consecutive_audit_passes + 1
WHEN $5 = 'INVALIDATE' THEN 0
ELSE consecutive_audit_passes
END,
expires_at = CASE
WHEN $5 = 'INVALIDATE' THEN now()
ELSE expires_at
END,
updated_at = now()
WHERE claim_hash = $1 AND tier = $2
"""
async with db.pool.acquire() as conn:
await conn.execute(
sql,
claim_hash,
tier,
audit_array_json,
AUDIT_HISTORY_MAX,
verdict.decision,
str(int(AUDIT_PASS_MIN_INTERVAL_HOURS)),
)
await conn.execute(
"""
INSERT INTO brain_audit_log (action, target_table, target_id, actor, payload)
VALUES ($1, 'brain_verification_cache', $2, 'cache_judge', $3::jsonb)
""",
f"judge_{verdict.decision.lower()}",
f"{claim_hash[:12]}/{tier}",
audit_json,
)
async def lookup(
*,
claim: str,
tier: Literal["free", "premium"],
) -> CacheEntry | None:
"""Fetch the cached entry for (claim, tier) — evidence is metadata only."""
ch = hash_claim(claim)
sql = """
SELECT
claim_hash, tier,
evidence_hash, evidence_urls,
model, prompt_hash, framework_version, schema_name,
verification_raw, verification_processed,
created_at, updated_at, expires_at,
volatility, topic_codes, entity_bindings,
consecutive_audit_passes, last_audited_at
FROM brain_verification_cache
WHERE claim_hash = $1 AND tier = $2
AND expires_at > now()
"""
async with db.pool.acquire() as conn:
row = await conn.fetchrow(sql, ch, tier)
if not row:
return None
return _row_to_entry(row)
# ------------------------------------------------------------- staleness decider
StalenessStatus = Literal[
"fresh",
"stale_framework",
"stale_prompt",
"stale_evidence", # bound facts have flipped — caller must recompute
"miss",
]
def decide_freshness(
entry: CacheEntry | None,
current_prompt_hash: str | None,
current_framework_version: str | None,
) -> StalenessStatus:
"""Given a cached entry + the caller's current prompt/framework, decide.
- miss: no entry at all (or expired in DB)
- stale_prompt: prompt changed since cache was written (verification_raw
stances may differ semantically) — caller should NOT use cache
- stale_framework: prompt unchanged but threshold config changed — caller
CAN use verification_raw and recompute status locally
- fresh: everything matches; return verification_processed directly
"""
if entry is None:
return "miss"
if current_prompt_hash and entry.prompt_hash != current_prompt_hash:
return "stale_prompt"
if (
current_framework_version
and entry.framework_version
and entry.framework_version != current_framework_version
):
return "stale_framework"
return "fresh"
# ------------------------------------------------------------------- internal
def _row_to_entry(row) -> CacheEntry:
raw = row["verification_raw"]
processed = row["verification_processed"]
ev_urls = row["evidence_urls"]
# asyncpg decodes jsonb as str; json.loads needed
if isinstance(raw, str):
raw = json.loads(raw)
if isinstance(processed, str):
processed = json.loads(processed)
if isinstance(ev_urls, str):
ev_urls = json.loads(ev_urls)
# Optional B1 metadata — these may not be present on legacy rows or in
# callers that select an older column set.
def _opt(key: str, default: Any = None) -> Any:
try:
return row[key]
except (KeyError, IndexError):
return default
bindings = _opt("entity_bindings", [])
if isinstance(bindings, str):
bindings = json.loads(bindings)
topic_codes = _opt("topic_codes", []) or []
return CacheEntry(
claim_hash=row["claim_hash"],
tier=row["tier"],
evidence_hash=row["evidence_hash"],
evidence_urls=list(ev_urls) if ev_urls else [],
model=row["model"],
prompt_hash=row["prompt_hash"],
framework_version=row["framework_version"],
schema_name=row["schema_name"],
verification_raw=raw,
verification_processed=processed,
created_at=row["created_at"],
updated_at=row["updated_at"],
expires_at=row["expires_at"],
volatility=_opt("volatility"),
topic_codes=list(topic_codes) if topic_codes else [],
entity_bindings=list(bindings) if bindings else [],
consecutive_audit_passes=int(_opt("consecutive_audit_passes", 0) or 0),
last_audited_at=_opt("last_audited_at"),
)

6
ai_platform/modules/didi_brain/extractor/__init__.py Normal file
View file

@ -0,0 +1,6 @@
"""Claim extractor — turns Type/Document atoms into Type/Claim atoms.
Public API:
from extractor.extract import extract_claims_from_atom, ExtractedClaim
from extractor.batch import run_batch_extraction
"""

398
ai_platform/modules/didi_brain/extractor/_extracted.json Normal file
View file

@ -0,0 +1,398 @@
{
"1d7be08c-9fc9-4063-a13b-d20a01b1f24f": {
"atom_id": "1d7be08c-9fc9-4063-a13b-d20a01b1f24f",
"extracted_at": "2026-04-30T17:39:20.830644+00:00",
"prompt_version": "v1",
"pushed_count": 29,
"raw_count": 30,
"rejected": {
"quote_not_in_source": 1
},
"source_url": "https://en.wikipedia.org/wiki/Moderna_COVID-19_vaccine",
"valid_count": 29
},
"221df1cf-8361-4a07-b720-b7660a062da7": {
"atom_id": "221df1cf-8361-4a07-b720-b7660a062da7",
"extracted_at": "2026-04-22T08:20:58.548047+00:00",
"prompt_version": "v1",
"pushed_count": 29,
"raw_count": 30,
"rejected": {
"quote_not_in_source": 1
},
"source_url": "https://en.wikipedia.org/wiki/Plandemic",
"valid_count": 29
},
"3c332c68-8908-44a3-8b6f-3e76e83da111": {
"atom_id": "3c332c68-8908-44a3-8b6f-3e76e83da111",
"extracted_at": "2026-04-22T08:13:47.644503+00:00",
"prompt_version": "v1",
"pushed_count": 15,
"raw_count": 15,
"rejected": {},
"source_url": "https://ro.wikipedia.org/wiki/Tiomersal",
"valid_count": 15
},
"3c90743e-04e2-4f2b-a099-8e4c8d7d83b6": {
"atom_id": "3c90743e-04e2-4f2b-a099-8e4c8d7d83b6",
"extracted_at": "2026-04-30T17:27:01.973084+00:00",
"prompt_version": "v1",
"pushed_count": 15,
"raw_count": 15,
"rejected": {},
"source_url": "https://ro.wikipedia.org/wiki/Tiomersal",
"valid_count": 15
},
"5094dfc9-9b8b-42e9-a8f7-b3ad16c39713": {
"atom_id": "5094dfc9-9b8b-42e9-a8f7-b3ad16c39713",
"extracted_at": "2026-04-30T17:32:40.173050+00:00",
"prompt_version": "v1",
"pushed_count": 30,
"raw_count": 30,
"rejected": {},
"source_url": "https://en.wikipedia.org/wiki/Children%27s_Health_Defense",
"valid_count": 30
},
"5857c5f6-e90e-4028-9643-efd0228d8bdc": {
"atom_id": "5857c5f6-e90e-4028-9643-efd0228d8bdc",
"extracted_at": "2026-04-22T08:27:25.131781+00:00",
"prompt_version": "v1",
"pushed_count": 29,
"raw_count": 30,
"rejected": {
"quote_not_in_source": 1
},
"source_url": "https://en.wikipedia.org/wiki/Moderna_COVID-19_vaccine",
"valid_count": 29
},
"6169e3a9-3574-4e48-8318-3515ae35eb11": {
"atom_id": "6169e3a9-3574-4e48-8318-3515ae35eb11",
"extracted_at": "2026-04-22T08:34:28.188203+00:00",
"prompt_version": "v1",
"pushed_count": 29,
"raw_count": 30,
"rejected": {
"quote_not_in_source": 1
},
"source_url": "https://en.wikipedia.org/wiki/MMR_vaccine_and_autism",
"valid_count": 29
},
"61f690f3-0b6a-45a5-8074-798d4afa1453": {
"atom_id": "61f690f3-0b6a-45a5-8074-798d4afa1453",
"extracted_at": "2026-04-22T08:30:34.956251+00:00",
"prompt_version": "v1",
"pushed_count": 28,
"raw_count": 28,
"rejected": {},
"source_url": "https://en.wikipedia.org/wiki/Vaccine_adverse_event",
"valid_count": 28
},
"685eedb4-62e9-437d-887f-ec81743f6bb8": {
"atom_id": "685eedb4-62e9-437d-887f-ec81743f6bb8",
"extracted_at": "2026-04-22T08:24:36.031727+00:00",
"prompt_version": "v1",
"pushed_count": 29,
"raw_count": 30,
"rejected": {
"quote_not_in_source": 1
},
"source_url": "https://en.wikipedia.org/wiki/Robert_F._Kennedy_Jr.",
"valid_count": 29
},
"69f16450-60e7-40fd-ae50-dd607514352b": {
"atom_id": "69f16450-60e7-40fd-ae50-dd607514352b",
"extracted_at": "2026-04-22T08:37:13.179879+00:00",
"prompt_version": "v1",
"pushed_count": 29,
"raw_count": 30,
"rejected": {
"quote_not_in_source": 1
},
"source_url": "https://en.wikipedia.org/wiki/Vaccination",
"valid_count": 29
},
"6b86489d-fc88-4f40-8c81-371e82ace952": {
"atom_id": "6b86489d-fc88-4f40-8c81-371e82ace952",
"extracted_at": "2026-04-30T17:51:07.975837+00:00",
"prompt_version": "v1",
"pushed_count": 30,
"raw_count": 30,
"rejected": {},
"source_url": "https://en.wikipedia.org/wiki/Vaccine",
"valid_count": 30
},
"6e13b816-50c9-4cdd-a9e4-a67b2d65dca0": {
"atom_id": "6e13b816-50c9-4cdd-a9e4-a67b2d65dca0",
"extracted_at": "2026-04-22T08:38:16.576393+00:00",
"prompt_version": "v1",
"pushed_count": 30,
"raw_count": 30,
"rejected": {},
"source_url": "https://en.wikipedia.org/wiki/Vaccine",
"valid_count": 30
},
"6e791e83-8bfd-4aaf-9111-9e77072517d8": {
"atom_id": "6e791e83-8bfd-4aaf-9111-9e77072517d8",
"extracted_at": "2026-04-22T08:14:35.928834+00:00",
"prompt_version": "v1",
"pushed_count": 27,
"raw_count": 27,
"rejected": {},
"source_url": "https://ro.wikipedia.org/wiki/Andrew_Wakefield",
"valid_count": 27
},
"79c1010d-1406-4b37-8ab9-c8e3e6d054ef": {
"atom_id": "79c1010d-1406-4b37-8ab9-c8e3e6d054ef",
"extracted_at": "2026-04-30T17:41:23.814310+00:00",
"prompt_version": "v1",
"pushed_count": 28,
"raw_count": 28,
"rejected": {},
"source_url": "https://en.wikipedia.org/wiki/Vaccine_adverse_event",
"valid_count": 28
},
"8182687d-9350-4870-818f-5e6e226def88": {
"atom_id": "8182687d-9350-4870-818f-5e6e226def88",
"extracted_at": "2026-04-22T08:16:30.831673+00:00",
"prompt_version": "v1",
"pushed_count": 29,
"raw_count": 30,
"rejected": {
"quote_not_in_source": 1
},
"source_url": "https://ro.wikipedia.org/wiki/Pandemia_de_COVID-19_%C3%AEn_Rom%C3%A2nia",
"valid_count": 29
},
"8516163b-fe3e-448c-9fda-f48155a05327": {
"atom_id": "8516163b-fe3e-448c-9fda-f48155a05327",
"extracted_at": "2026-04-30T17:37:46.813140+00:00",
"prompt_version": "v1",
"pushed_count": 30,
"raw_count": 30,
"rejected": {},
"source_url": "https://en.wikipedia.org/wiki/COVID-19_vaccine_misinformation_and_hesitancy",
"valid_count": 30
},
"86de36c3-7b60-4120-ab0f-018dfa1c8ba9": {
"atom_id": "86de36c3-7b60-4120-ab0f-018dfa1c8ba9",
"extracted_at": "2026-04-22T08:32:07.787823+00:00",
"prompt_version": "v1",
"pushed_count": 30,
"raw_count": 30,
"rejected": {},
"source_url": "https://en.wikipedia.org/wiki/Anti-vaccine_activism",
"valid_count": 30
},
"8d87dd49-6a8e-4d6d-b15a-1631db0c7153": {
"atom_id": "8d87dd49-6a8e-4d6d-b15a-1631db0c7153",
"extracted_at": "2026-04-30T17:31:49.206015+00:00",
"prompt_version": "v1",
"pushed_count": 29,
"raw_count": 30,
"rejected": {
"quote_not_in_source": 1
},
"source_url": "https://ro.wikipedia.org/wiki/Vaccin",
"valid_count": 29
},
"9272802d-211c-42e4-b064-cb8acc143e1c": {
"atom_id": "9272802d-211c-42e4-b064-cb8acc143e1c",
"extracted_at": "2026-04-30T17:46:26.293983+00:00",
"prompt_version": "v1",
"pushed_count": 29,
"raw_count": 30,
"rejected": {
"quote_not_in_source": 1
},
"source_url": "https://en.wikipedia.org/wiki/MMR_vaccine_and_autism",
"valid_count": 29
},
"9c12182c-f2fe-487f-b43d-f97fd8f84984": {
"atom_id": "9c12182c-f2fe-487f-b43d-f97fd8f84984",
"extracted_at": "2026-04-30T17:43:18.929395+00:00",
"prompt_version": "v1",
"pushed_count": 30,
"raw_count": 30,
"rejected": {},
"source_url": "https://en.wikipedia.org/wiki/Anti-vaccine_activism",
"valid_count": 30
},
"9c2c5fbb-cb32-40ab-b6fd-6819c8e49fd2": {
"atom_id": "9c2c5fbb-cb32-40ab-b6fd-6819c8e49fd2",
"extracted_at": "2026-04-30T17:30:19.630562+00:00",
"prompt_version": "v1",
"pushed_count": 6,
"raw_count": 6,
"rejected": {},
"source_url": "https://ro.wikipedia.org/wiki/Vaccinare",
"valid_count": 6
},
"a5d2ce82-0249-4253-9c2d-bddd6853d311": {
"atom_id": "a5d2ce82-0249-4253-9c2d-bddd6853d311",
"extracted_at": "2026-04-30T17:44:43.113762+00:00",
"prompt_version": "v1",
"pushed_count": 30,
"raw_count": 30,
"rejected": {},
"source_url": "https://en.wikipedia.org/wiki/Andrew_Wakefield",
"valid_count": 30
},
"ab56b518-9d13-4de0-bf01-1c89abd087e0": {
"atom_id": "ab56b518-9d13-4de0-bf01-1c89abd087e0",
"extracted_at": "2026-04-22T08:12:44.236406+00:00",
"prompt_version": "v1",
"pushed_count": 21,
"raw_count": 21,
"rejected": {},
"source_url": "https://ro.wikipedia.org/wiki/Variol%C4%83",
"valid_count": 21
},
"abc36af6-e4fe-4c75-b98d-57738f9406ca": {
"atom_id": "abc36af6-e4fe-4c75-b98d-57738f9406ca",
"extracted_at": "2026-04-22T08:26:19.603769+00:00",
"prompt_version": "v1",
"pushed_count": 30,
"raw_count": 30,
"rejected": {},
"source_url": "https://en.wikipedia.org/wiki/COVID-19_vaccine_misinformation_and_hesitancy",
"valid_count": 30
},
"aef7f81f-c6a8-44e0-859d-812b144380e2": {
"atom_id": "aef7f81f-c6a8-44e0-859d-812b144380e2",
"extracted_at": "2026-04-30T17:34:27.380347+00:00",
"prompt_version": "v1",
"pushed_count": 29,
"raw_count": 30,
"rejected": {
"quote_not_in_source": 1
},
"source_url": "https://en.wikipedia.org/wiki/Plandemic",
"valid_count": 29
},
"b09275c4-d325-4aec-b730-8fa23e374284": {
"atom_id": "b09275c4-d325-4aec-b730-8fa23e374284",
"extracted_at": "2026-04-30T17:28:03.929162+00:00",
"prompt_version": "v1",
"pushed_count": 27,
"raw_count": 27,
"rejected": {},
"source_url": "https://ro.wikipedia.org/wiki/Andrew_Wakefield",
"valid_count": 27
},
"b569c570-6d19-42a7-b13d-41039c8b4391": {
"atom_id": "b569c570-6d19-42a7-b13d-41039c8b4391",
"extracted_at": "2026-04-30T17:36:11.186401+00:00",
"prompt_version": "v1",
"pushed_count": 25,
"raw_count": 30,
"rejected": {
"quote_not_in_source": 5
},
"source_url": "https://en.wikipedia.org/wiki/Robert_F._Kennedy_Jr.",
"valid_count": 25
},
"b7b39518-eb7b-4d7d-afb4-d6e9ef815094": {
"atom_id": "b7b39518-eb7b-4d7d-afb4-d6e9ef815094",
"extracted_at": "2026-04-22T08:35:55.179684+00:00",
"prompt_version": "v1",
"pushed_count": 30,
"raw_count": 30,
"rejected": {},
"source_url": "https://en.wikipedia.org/wiki/Vaccine_hesitancy",
"valid_count": 30
},
"c7651882-6731-438a-933c-efb827344495": {
"atom_id": "c7651882-6731-438a-933c-efb827344495",
"extracted_at": "2026-04-22T08:29:18.331921+00:00",
"prompt_version": "v1",
"pushed_count": 30,
"raw_count": 30,
"rejected": {},
"source_url": "https://en.wikipedia.org/wiki/Pfizer%E2%80%93BioNTech_COVID-19_vaccine",
"valid_count": 30
},
"c97398f4-819b-46a9-b6fc-f25f6e192340": {
"atom_id": "c97398f4-819b-46a9-b6fc-f25f6e192340",
"extracted_at": "2026-04-22T08:16:44.511818+00:00",
"prompt_version": "v1",
"pushed_count": 6,
"raw_count": 6,
"rejected": {},
"source_url": "https://ro.wikipedia.org/wiki/Vaccinare",
"valid_count": 6
},
"d933d1f5-03a5-4238-b0ea-352d72f0699c": {
"atom_id": "d933d1f5-03a5-4238-b0ea-352d72f0699c",
"extracted_at": "2026-04-30T17:49:08.873214+00:00",
"prompt_version": "v1",
"pushed_count": 29,
"raw_count": 30,
"rejected": {
"quote_not_in_source": 1
},
"source_url": "https://en.wikipedia.org/wiki/Vaccination",
"valid_count": 29
},
"e5e5aac1-addb-4586-a18b-2e004ac8f50f": {
"atom_id": "e5e5aac1-addb-4586-a18b-2e004ac8f50f",
"extracted_at": "2026-04-30T17:48:08.258151+00:00",
"prompt_version": "v1",
"pushed_count": 30,
"raw_count": 30,
"rejected": {},
"source_url": "https://en.wikipedia.org/wiki/Vaccine_hesitancy",
"valid_count": 30
},
"e7e388e8-6b1c-4d48-8565-635fde082c75": {
"atom_id": "e7e388e8-6b1c-4d48-8565-635fde082c75",
"extracted_at": "2026-04-22T08:33:25.352110+00:00",
"prompt_version": "v1",
"pushed_count": 30,
"raw_count": 30,
"rejected": {},
"source_url": "https://en.wikipedia.org/wiki/Andrew_Wakefield",
"valid_count": 30
},
"e9a1c988-01ac-48c8-8da3-a5f3827e0594": {
"atom_id": "e9a1c988-01ac-48c8-8da3-a5f3827e0594",
"extracted_at": "2026-04-22T08:19:10.267888+00:00",
"prompt_version": "v1",
"pushed_count": 30,
"raw_count": 30,
"rejected": {},
"source_url": "https://en.wikipedia.org/wiki/Children%27s_Health_Defense",
"valid_count": 30
},
"e9e72b6f-909c-400a-8d59-0c103c2fac47": {
"atom_id": "e9e72b6f-909c-400a-8d59-0c103c2fac47",
"extracted_at": "2026-04-30T17:40:36.770904+00:00",
"prompt_version": "v1",
"pushed_count": 30,
"raw_count": 30,
"rejected": {},
"source_url": "https://en.wikipedia.org/wiki/Pfizer%E2%80%93BioNTech_COVID-19_vaccine",
"valid_count": 30
},
"ecb20b8f-4141-4831-b88a-10ad4c415305": {
"atom_id": "ecb20b8f-4141-4831-b88a-10ad4c415305",
"extracted_at": "2026-04-22T08:18:19.056060+00:00",
"prompt_version": "v1",
"pushed_count": 28,
"raw_count": 30,
"rejected": {
"quote_not_in_source": 2
},
"source_url": "https://ro.wikipedia.org/wiki/Vaccin",
"valid_count": 28
},
"f0434eae-41f4-42bf-8459-54bbc32e9b29": {
"atom_id": "f0434eae-41f4-42bf-8459-54bbc32e9b29",
"extracted_at": "2026-04-30T17:26:38.082088+00:00",
"prompt_version": "v1",
"pushed_count": 22,
"raw_count": 22,
"rejected": {},
"source_url": "https://ro.wikipedia.org/wiki/Variol%C4%83",
"valid_count": 22
}
}

67
ai_platform/modules/didi_brain/extractor/_state.py Normal file
View file

@ -0,0 +1,67 @@
"""Persistent state for the claim extractor.
We track which document atoms have already been processed so re-runs are
idempotent. Stored as a single JSON file under extractor/_extracted.json.
Each entry records the run timestamp, prompt version, and a small summary
of what came out, so we can audit later or selectively re-extract if a
prompt version changes.
"""
from __future__ import annotations
import json
from dataclasses import asdict, dataclass, field
from datetime import datetime, timezone
from pathlib import Path
_STATE_FILE = Path(__file__).resolve().parent / "_extracted.json"
@dataclass(slots=True)
class DocExtractionRecord:
atom_id: str
source_url: str
extracted_at: str
prompt_version: str
raw_count: int # how many claims the LLM returned
valid_count: int # how many passed validation
pushed_count: int # how many were created in Atomic (excludes dedup hits)
rejected: dict[str, int] = field(default_factory=dict)
class ExtractionState:
"""Loads / saves the extraction log file."""
def __init__(self, path: Path | None = None):
self._path = path or _STATE_FILE
self._records: dict[str, DocExtractionRecord] = {}
if self._path.exists():
data = json.loads(self._path.read_text(encoding="utf-8"))
for atom_id, raw in data.items():
self._records[atom_id] = DocExtractionRecord(**raw)
def has(self, atom_id: str, prompt_version: str) -> bool:
rec = self._records.get(atom_id)
return rec is not None and rec.prompt_version == prompt_version
def get(self, atom_id: str) -> DocExtractionRecord | None:
return self._records.get(atom_id)
def upsert(self, record: DocExtractionRecord) -> None:
self._records[record.atom_id] = record
def save(self) -> None:
out = {k: asdict(v) for k, v in self._records.items()}
self._path.write_text(
json.dumps(out, indent=2, ensure_ascii=False, sort_keys=True),
encoding="utf-8",
)
@property
def all(self) -> dict[str, DocExtractionRecord]:
return dict(self._records)
def now_iso() -> str:
return datetime.now(timezone.utc).isoformat()

233
ai_platform/modules/didi_brain/extractor/batch.py Normal file
View file

@ -0,0 +1,233 @@
"""Batch orchestrator for claim extraction.
Walks all Type/Document atoms in Atomic, runs extraction on each, and pushes
the resulting claims as new Type/Claim atoms. Idempotent across runs via the
state file in extractor/_extracted.json.
"""
from __future__ import annotations
import asyncio
from collections.abc import AsyncIterator
from dataclasses import dataclass, field
from typing import Any
from urllib.parse import unquote
from shared.atomic_api import AtomicClient
from shared.config import settings
from shared.embedding_client import EmbeddingClient # noqa: F401 (future use)
from shared.llm_client import LlmClient, LlmError
from shared.logging import get_logger
from shared.taxonomy import TagResolver
from extractor._state import DocExtractionRecord, ExtractionState, now_iso
from extractor.extract import (
PROMPT_VERSION,
ExtractionResult,
extract_claims_from_atom,
)
from extractor.push import push_claim
log = get_logger(__name__)
@dataclass(slots=True)
class BatchStats:
docs_seen: int = 0
docs_skipped_already_done: int = 0
docs_processed: int = 0
docs_failed: int = 0
claims_raw: int = 0
claims_valid: int = 0
claims_created: int = 0
claims_duplicate: int = 0
claims_error: int = 0
rejected_reasons: dict[str, int] = field(default_factory=dict)
# ====================================================== document selection
async def _list_documents_to_process(
atomic: AtomicClient, resolver: TagResolver, *, limit: int = 1000
) -> list[dict[str, Any]]:
"""Return all atoms tagged Type/Document, with their tags inlined.
We page through /api/atoms?tag_id=<Type/Document> and pull metadata for
each, since the list endpoint already returns tags inline.
"""
type_doc_id = resolver.require("Type/Document")
page_size = 50
out: list[dict[str, Any]] = []
offset = 0
while True:
result = await atomic.list_atoms(
limit=page_size, offset=offset, tag_id=type_doc_id
)
atoms = result.get("atoms") or result.get("data") or (result if isinstance(result, list) else [])
if not atoms:
break
for a in atoms:
out.append(a)
if len(out) >= limit:
return out
if len(atoms) < page_size:
break
offset += page_size
return out
def _title_from_atom(atom: dict[str, Any]) -> str:
"""Best-effort title: prefer the Markdown H1 in content, fall back to URL slug."""
content = atom.get("content") or ""
# Look for the first '# ...' line at the start
for line in content.lstrip().splitlines():
line = line.strip()
if line.startswith("# "):
return line[2:].strip()
if line:
break # first non-empty isn't a header → fall through to URL
url = atom.get("source_url") or ""
if url:
last = url.rstrip("/").rsplit("/", 1)[-1]
return unquote(last).replace("_", " ")
return atom.get("id", "?")[:8]
def _language_from_atom(atom: dict[str, Any]) -> str:
"""Read the Language/<X> tag if present, default 'EN'."""
for tag in atom.get("tags") or []:
name = tag.get("name", "")
# The tag list returns just `name`, not the full path. Languages are
# short codes (RO/EN/RU/...) so direct match works.
if name in {"RO", "EN", "RU", "UA", "FR", "DE", "ES", "IT", "PL"}:
return name
return "EN"
# ============================================================== one document
async def process_one(
*,
llm: LlmClient,
atomic: AtomicClient,
resolver: TagResolver,
state: ExtractionState,
atom: dict[str, Any],
stats: BatchStats,
) -> None:
atom_id = atom["id"]
if state.has(atom_id, PROMPT_VERSION):
stats.docs_skipped_already_done += 1
return
# /api/atoms (list) returns summary objects WITHOUT full content. We have
# to fetch the full atom individually to get the body for extraction.
full_atom = await atomic.get_atom(atom_id)
content = full_atom.get("content") or ""
title = _title_from_atom(full_atom)
language = _language_from_atom(full_atom)
if not content:
log.warning("doc_no_content", atom_id=atom_id)
stats.docs_failed += 1
return
log.info("extracting", atom_id=atom_id[:8], title=title, lang=language, chars=len(content))
try:
result: ExtractionResult = await extract_claims_from_atom(
llm,
title=title,
language=language,
content=content,
)
except LlmError as e:
log.error("extraction_failed", atom_id=atom_id, error=str(e), body=(e.body or "")[:300])
stats.docs_failed += 1
return
except Exception as e: # noqa: BLE001
log.error("extraction_crashed", atom_id=atom_id, error=f"{type(e).__name__}: {e}")
stats.docs_failed += 1
return
stats.docs_processed += 1
stats.claims_raw += result.raw_count
stats.claims_valid += len(result.valid)
for k, v in result.rejected.items():
stats.rejected_reasons[k] = stats.rejected_reasons.get(k, 0) + v
pushed = 0
duplicates = 0
errors = 0
for c in result.valid:
_, status = await push_claim(
atomic,
parent_atom=full_atom,
claim=c,
parent_title=title,
resolver=resolver,
)
if status == "created":
pushed += 1
elif status == "duplicate":
duplicates += 1
else:
errors += 1
stats.claims_created += pushed
stats.claims_duplicate += duplicates
stats.claims_error += errors
state.upsert(
DocExtractionRecord(
atom_id=atom_id,
source_url=full_atom.get("source_url", ""),
extracted_at=now_iso(),
prompt_version=PROMPT_VERSION,
raw_count=result.raw_count,
valid_count=len(result.valid),
pushed_count=pushed,
rejected=dict(result.rejected),
)
)
state.save() # save after each doc so a crash doesn't lose progress
# ============================================================ batch entry
async def run_batch(
*,
limit: int | None = None,
only_atom_ids: set[str] | None = None,
) -> BatchStats:
if not settings.atomic_token:
raise RuntimeError("ATOMIC_TOKEN missing")
resolver = TagResolver()
if "Type/Claim" not in resolver:
raise RuntimeError("taxonomy not seeded — run scripts/04_seed_taxonomy.py")
state = ExtractionState()
stats = BatchStats()
async with AtomicClient() as atomic, LlmClient() as llm:
docs = await _list_documents_to_process(atomic, resolver, limit=limit or 1000)
if only_atom_ids:
docs = [d for d in docs if d["id"] in only_atom_ids]
stats.docs_seen = len(docs)
log.info("batch_start", docs=len(docs), prompt_version=PROMPT_VERSION)
for atom in docs:
await process_one(
llm=llm,
atomic=atomic,
resolver=resolver,
state=state,
atom=atom,
stats=stats,
)
return stats

204
ai_platform/modules/didi_brain/extractor/extract.py Normal file
View file

@ -0,0 +1,204 @@
"""Single-document claim extraction logic.
Given one Type/Document atom, this module:
1. Builds the extraction prompt from the versioned template
2. Calls Qwen 397B (REASONING role) for structured JSON output
3. Parses + validates each claim:
- Required fields present and right types
- Stance is in the allowed enum
- Confidence above threshold
- Quote is a verbatim substring of the source (programmatic check —
this is the cheap defense against LLM hallucination)
4. Returns a list of ExtractedClaim dataclasses ready for the pusher.
The pusher (push.py) takes ExtractedClaim and creates a Type/Claim atom in
Atomic, with the proper tag inheritance and a stable hash-based source_url.
"""
from __future__ import annotations
import hashlib
import re
from dataclasses import dataclass
from pathlib import Path
from typing import Any
from shared.config import LlmRole
from shared.llm_client import LlmClient, LlmError
from shared.logging import get_logger
log = get_logger(__name__)
PROMPT_VERSION = "v1"
PROMPT_PATH = Path(__file__).resolve().parent / "prompts" / f"claim_extraction_{PROMPT_VERSION}.md"
ALLOWED_STANCES = {"ASSERTS", "REPORTS", "REFUTES", "QUESTIONS", "NEUTRAL"}
MIN_CONFIDENCE = 0.7
MIN_CLAIM_CHARS = 20
MIN_QUOTE_CHARS = 10
MAX_QUOTE_CHARS = 600
MAX_INPUT_CHARS = 120_000 # ~30K tokens, well under Qwen's 262K context
_PROMPT_TEMPLATE: str | None = None
def _load_prompt() -> str:
global _PROMPT_TEMPLATE
if _PROMPT_TEMPLATE is None:
_PROMPT_TEMPLATE = PROMPT_PATH.read_text(encoding="utf-8")
return _PROMPT_TEMPLATE
# ============================================================ data structures
@dataclass(slots=True, frozen=True)
class ExtractedClaim:
claim: str
quote: str
stance: str # uppercase: ASSERTS / REPORTS / REFUTES / QUESTIONS / NEUTRAL
confidence: float
def stable_hash(self) -> str:
"""8-char SHA-1 of canonicalized claim text — used in source_url fragment."""
canonical = " ".join(self.claim.lower().strip().split())
return hashlib.sha1(canonical.encode("utf-8")).hexdigest()[:8]
@dataclass(slots=True)
class ExtractionResult:
"""What `extract_claims_from_atom` returns."""
raw_count: int # how many items the LLM returned
valid: list[ExtractedClaim]
rejected: dict[str, int] # reason → count
backend: str # which backend served the request
# ============================================================ extraction core
async def extract_claims_from_atom(
llm: LlmClient,
*,
title: str,
language: str,
content: str,
max_tokens_out: int = 6000,
) -> ExtractionResult:
"""Run the extraction LLM call and validate the results.
Raises LlmError on transport / JSON parse failure (caller decides whether
to retry or skip the document).
"""
prompt = _load_prompt().replace("{title}", title)
prompt = prompt.replace("{language}", language)
truncated = content[:MAX_INPUT_CHARS]
if len(content) > MAX_INPUT_CHARS:
truncated += "\n\n[document truncated for extraction]"
prompt = prompt.replace("{content}", truncated)
result, usage = await llm.chat_json(
role=LlmRole.REASONING,
system=(
"You are a precise information extractor. Output strictly valid "
"JSON, no commentary, no markdown fences."
),
user=prompt,
max_tokens=max_tokens_out,
temperature=0.0,
)
if not isinstance(result, dict) or "claims" not in result:
raise LlmError(
f"unexpected response shape: keys={list(result.keys()) if isinstance(result, dict) else type(result).__name__}"
)
raw_claims = result.get("claims") or []
if not isinstance(raw_claims, list):
raise LlmError(f"`claims` is not a list: {type(raw_claims).__name__}")
valid: list[ExtractedClaim] = []
rejected: dict[str, int] = {}
norm_source = _normalize_for_match(content)
for raw in raw_claims:
outcome = _parse_one(raw, norm_source)
if isinstance(outcome, ExtractedClaim):
valid.append(outcome)
else:
rejected[outcome] = rejected.get(outcome, 0) + 1
# Dedup within this batch by stable hash
seen: set[str] = set()
deduped: list[ExtractedClaim] = []
for c in valid:
h = c.stable_hash()
if h in seen:
rejected["intra_batch_duplicate"] = rejected.get("intra_batch_duplicate", 0) + 1
continue
seen.add(h)
deduped.append(c)
return ExtractionResult(
raw_count=len(raw_claims),
valid=deduped,
rejected=rejected,
backend=str(usage.get("backend", "")),
)
# ============================================================ validation
def _parse_one(raw: Any, norm_source: str) -> ExtractedClaim | str:
"""Validate one raw item from the LLM. Returns ExtractedClaim or error reason str."""
if not isinstance(raw, dict):
return "not_a_dict"
claim = (raw.get("claim") or "").strip()
quote = (raw.get("quote") or "").strip()
stance = (raw.get("stance") or "").strip().upper()
try:
confidence = float(raw.get("confidence", 0))
except (TypeError, ValueError):
return "bad_confidence_type"
if len(claim) < MIN_CLAIM_CHARS:
return "claim_too_short"
if len(quote) < MIN_QUOTE_CHARS:
return "quote_too_short"
if len(quote) > MAX_QUOTE_CHARS:
return "quote_too_long"
if stance not in ALLOWED_STANCES:
return "bad_stance"
if confidence < MIN_CONFIDENCE:
return "low_confidence"
# The critical anti-hallucination check: the quote must actually appear
# in the source document (after whitespace normalization).
if not _quote_in_source(quote, norm_source):
return "quote_not_in_source"
return ExtractedClaim(
claim=claim,
quote=quote,
stance=stance,
confidence=confidence,
)
_WHITESPACE_RE = re.compile(r"\s+")
def _normalize_for_match(s: str) -> str:
"""Collapse runs of whitespace and normalize quote chars for substring match."""
s = s.replace("\u2018", "'").replace("\u2019", "'")
s = s.replace("\u201c", '"').replace("\u201d", '"')
s = s.replace("\u2013", "-").replace("\u2014", "-")
return _WHITESPACE_RE.sub(" ", s).strip()
def _quote_in_source(quote: str, norm_source: str) -> bool:
return _normalize_for_match(quote) in norm_source

65
ai_platform/modules/didi_brain/extractor/prompts/claim_extraction_v1.md Normal file
View file

@ -0,0 +1,65 @@
You are an expert fact extractor for a knowledge graph that supports
disinformation analysis. Your job is to read one source document and extract
every distinct, verifiable factual claim it makes.
# What counts as a "claim"
A claim is a **specific, self-contained, verifiable proposition**. It must be:
- Concrete enough that someone could check it against evidence
- Self-contained: understandable without reading surrounding text
- About facts, events, findings, or attributed statements — not opinions
A claim is **NOT**:
- An opinion or value judgment ("the policy was misguided")
- A vague generality ("many people worry", "some scientists think")
- A definition or terminology explanation
- A question or recommendation
- Pure background description without a specific assertion
# Stance classification
For each claim, decide how the SOURCE itself treats it:
- **ASSERTS** — the source presents the claim as factual / a finding it stands behind
- **REPORTS** — the source describes someone else's claim without endorsing or refuting it
- **REFUTES** — the source explicitly disagrees with, debunks, or corrects the claim
- **QUESTIONS** — the source raises doubts or uncertainty without fully refuting
- **NEUTRAL** — purely descriptive context with no editorial stance
# Output rules — STRICT
You MUST output ONLY a single JSON object with this exact shape:
```json
{
"claims": [
{
"claim": "<canonical claim text, in the SAME LANGUAGE as the source>",
"quote": "<EXACT verbatim substring from the source where this claim appears>",
"stance": "ASSERTS|REPORTS|REFUTES|QUESTIONS|NEUTRAL",
"confidence": 0.0-1.0
}
]
}
```
Hard rules:
1. The `quote` MUST be a character-perfect substring of the source. Copy-paste it. Do not paraphrase, translate, or add ellipses inside it.
2. The `claim` must be in the SAME language as the source document. If the source is Romanian, the claim must be in Romanian. Do NOT translate.
3. Quote at minimum one full sentence; quote at most ~300 characters.
4. Extract up to **30** claims, prioritizing the most consequential and contestable ones. Skip duplicates and trivia.
5. Confidence reflects how clean and verifiable the claim is. Use 0.7-0.85 for ordinary factual claims, 0.85-0.95 for well-supported specific findings, below 0.7 for claims you are unsure about (these will be filtered out).
6. Do NOT add any text before or after the JSON. No markdown fences. No commentary. Just the JSON object.
7. If the document has no extractable factual claims, return `{"claims": []}`.
# Source
Title: {title}
Language: {language}
# Source content
{content}

124
ai_platform/modules/didi_brain/extractor/push.py Normal file
View file

@ -0,0 +1,124 @@
"""Push validated ExtractedClaim objects into Atomic as Type/Claim atoms.
Each claim becomes a tiny atom with:
- source_url = `{parent_url}#claim={hash8}` so it dedups idempotently and
so `parent_url = source_url.split("#")[0]` is trivial to recover later
- tag inheritance from the parent (Country, Topic, SourceType, Credibility,
Language) plus our two new tags: Type/Claim and Stance/<X>
"""
from __future__ import annotations
from typing import Any
from shared.atomic_api import AtomicApiError, AtomicClient
from shared.logging import get_logger
from shared.taxonomy import TagResolver
from extractor.extract import ExtractedClaim
log = get_logger(__name__)
_STANCE_PATH = {
"ASSERTS": "Stance/Asserts",
"REPORTS": "Stance/Reports",
"REFUTES": "Stance/Refutes",
"QUESTIONS": "Stance/Questions",
"NEUTRAL": "Stance/Neutral",
}
def build_claim_markdown(
claim: ExtractedClaim,
*,
parent_title: str,
parent_url: str,
parent_atom_id: str,
) -> str:
"""Render a claim atom's body as Markdown.
The structure is intentionally consistent so it can be parsed back later
by Didi or by re-indexing scripts.
"""
return (
f"# Claim\n\n"
f"{claim.claim}\n\n"
f"## Quote\n"
f"> {claim.quote}\n\n"
f"## Source\n"
f"- Document: [{parent_title}]({parent_url})\n"
f"- Parent atom: `{parent_atom_id}`\n"
f"- Stance in source: {claim.stance}\n"
f"- Extraction confidence: {claim.confidence:.2f}\n"
)
def build_claim_url(parent_url: str, claim: ExtractedClaim) -> str:
"""Stable hash-based URL fragment so re-extraction dedupes naturally."""
base = parent_url.split("#", 1)[0]
return f"{base}#claim={claim.stable_hash()}"
def inherit_tag_ids(
parent_atom: dict[str, Any],
claim: ExtractedClaim,
resolver: TagResolver,
) -> list[str]:
"""Build the tag-id list for a new claim atom.
Inherits all parent tags except Type/Document, and adds Type/Claim plus
the appropriate Stance/<X>.
"""
type_doc_id = resolver.require("Type/Document")
type_claim_id = resolver.require("Type/Claim")
stance_id = resolver.require(_STANCE_PATH[claim.stance])
parent_tag_ids = [
t["id"] for t in (parent_atom.get("tags") or []) if t.get("id") != type_doc_id
]
return parent_tag_ids + [type_claim_id, stance_id]
async def push_claim(
atomic: AtomicClient,
*,
parent_atom: dict[str, Any],
claim: ExtractedClaim,
parent_title: str,
resolver: TagResolver,
) -> tuple[dict[str, Any] | None, str]:
"""Create one claim atom in Atomic. Returns (atom_dict, status).
Status is one of:
- "created": new atom was created
- "duplicate": same hash already exists, skipped
- "error": creation failed (atom_dict is None)
"""
parent_url = parent_atom.get("source_url") or ""
parent_id = parent_atom.get("id") or ""
claim_url = build_claim_url(parent_url, claim)
# Idempotency: same canonical hash → skip
existing = await atomic.get_atom_by_source_url(claim_url)
if existing:
return existing, "duplicate"
md = build_claim_markdown(
claim,
parent_title=parent_title,
parent_url=parent_url,
parent_atom_id=parent_id,
)
tag_ids = inherit_tag_ids(parent_atom, claim, resolver)
try:
atom = await atomic.create_atom(
content=md,
source_url=claim_url,
tag_ids=tag_ids,
)
return atom, "created"
except AtomicApiError as e:
log.error("push_claim_failed", url=claim_url, status=e.status, body=e.body[:200])
return None, "error"

183
ai_platform/modules/didi_brain/infra/docker-compose.yml Normal file
View file

@ -0,0 +1,183 @@
# =============================================================================
# DidiBrain — Atomic + Postgres pgvector
# =============================================================================
# This compose stack runs:
# - postgres (pgvector/pgvector:pg16) on port 5434
# - atomic-server (kenforthewin/atomic-server:latest) on port 8080
#
# Atomic is configured to use Postgres as the data backend (atoms, embeddings,
# tags, etc.) while keeping the SQLite registry for tokens & global settings
# in the local volume at /data.
#
# AI provider config (BGE-M3 endpoint, Qwen 397B router) is NOT set here —
# it lives in Atomic's settings table and is bootstrapped post-startup by
# `scripts/02_bootstrap_atomic.py` which calls PUT /api/settings.
# =============================================================================
services:
postgres:
image: pgvector/pgvector:pg16
container_name: didibrain-postgres
environment:
POSTGRES_USER: ${POSTGRES_USER:-atomic}
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:-atomic_dev_changeme}
POSTGRES_DB: ${POSTGRES_DB:-atomic}
ports:
- "${POSTGRES_PORT:-5434}:5432"
volumes:
- didibrain-pg-data:/var/lib/postgresql/data
healthcheck:
test: ["CMD-SHELL", "pg_isready -U ${POSTGRES_USER:-atomic} -d ${POSTGRES_DB:-atomic}"]
interval: 3s
timeout: 5s
retries: 10
start_period: 5s
networks:
- didi-network
restart: unless-stopped
atomic-server:
image: ghcr.io/kenforthewin/atomic-server:latest
container_name: didibrain-atomic
# Override the upstream image's hard-coded `--db-path` (legacy SQLite mode)
# so we can use the modern `--data-dir` + `--storage postgres` flow.
entrypoint: ["atomic-server", "--data-dir", "/data"]
command: ["serve", "--bind", "0.0.0.0", "--port", "8080"]
environment:
ATOMIC_STORAGE: postgres
ATOMIC_DATABASE_URL: postgres://${POSTGRES_USER:-atomic}:${POSTGRES_PASSWORD:-atomic_dev_changeme}@postgres:5432/${POSTGRES_DB:-atomic}
RUST_LOG: "atomic_core=info,atomic_server=info,warn"
# PUBLIC_URL is required for OAuth/MCP discovery — fine to leave empty
# for local dev (we'll set it on the real server).
PUBLIC_URL: ""
ports:
# 8080 is taken on this dev box (aria-frontend); use 8088 externally,
# the container itself still listens on 8080 internally.
- "8088:8080"
volumes:
- didibrain-atomic-data:/data
depends_on:
postgres:
condition: service_healthy
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:8080/health"]
interval: 5s
timeout: 3s
retries: 10
start_period: 15s
networks:
- didi-network
restart: unless-stopped
# Atomic-server makes outbound HTTPS calls to the LLM router and BGE
# endpoint. Since those live on the host network (10.11.10.x and
# localhost:14011), Linux containers need to reach them. Two options:
# 1. Use host.docker.internal:14011 for the router (Docker Desktop
# maps this to the host on Win/Mac).
# 2. Use the actual host IP for 10.11.10.x targets (Docker Desktop
# routes through WSL2's NAT, so 10.x is reachable directly).
# We rely on (2) being true for the BGE endpoint and (1) for the LLM
# router. The bootstrap script will configure both.
extra_hosts:
- "host.docker.internal:host-gateway"
brain-api:
# DidiBrain's HTTP service (speaks Didi's web-module contract).
# Build context is the didibrain/ root one level up, so shared/ and
# extractor/ get COPY'd in by the Dockerfile.
build:
context: ..
dockerfile: brain_api/Dockerfile
image: didibrain-api:latest
container_name: didibrain-api
# Load the full host .env (single source of truth for endpoints),
# then override the atomic URL to use Docker's internal DNS. Everything
# else — BGE, LLM router, model names — stays exactly as on the host.
env_file:
- ../.env
environment:
# Inside the compose network, atomic-server is reachable by its
# service name on its internal port (8080), NOT the host-mapped 8088.
ATOMIC_URL: http://atomic-server:8080
# brain_api connects directly to postgres for the verification cache.
# Inside the network, postgres is at 'postgres:5432' (not 5434/localhost).
POSTGRES_HOST: postgres
POSTGRES_INTERNAL_PORT: "5432"
# AI platform dashboard — RuntimeConfigClient polls /api/config every 30s
# for live overrides on atom_* and log_level. Empty/unset = polling
# disabled (settings/constants used as-is).
DASHBOARD_URL: ${DASHBOARD_URL:-http://didiAI-dashboard:51300}
# OTel — traces to OTel Collector → Jaeger
OTEL_EXPORTER_OTLP_ENDPOINT: ${OTEL_EXPORTER_OTLP_ENDPOINT:-http://didi-otel-collector:4317}
OTEL_SERVICE_NAME: didibrain-api
volumes:
# Mount the host-seeded taxonomy map so background extraction (which
# instantiates TagResolver directly, bypassing the app lifespan
# refresh) can find canonical tag UUIDs.
- ../shared/_tag_ids.json:/app/shared/_tag_ids.json:ro
ports:
- "8090:8090"
depends_on:
postgres:
condition: service_healthy
atomic-server:
condition: service_healthy
healthcheck:
test: ["CMD", "curl", "-fsS", "http://localhost:8090/health"]
interval: 10s
timeout: 5s
retries: 5
start_period: 20s
networks:
- didi-network
restart: unless-stopped
# BGE endpoint (10.11.10.15) and LLM router (10.11.10.17) live on the
# VPN-routed internal network. Docker Desktop on Windows routes these
# through WSL2's NAT just like the atomic-server container already does,
# so no special extra_hosts entries are needed for them.
scheduler:
# Phase C — feeder (RSS hourly), auditor (daily LLM audit), and breaking
# news watcher in a single container. Imports brain_api.* directly for
# DB access (same Postgres pool, separate process).
build:
context: ..
dockerfile: scheduler/Dockerfile
image: didibrain-scheduler:latest
container_name: didibrain-scheduler
env_file:
- ../.env
environment:
# Same as brain-api — keeps the auditor's DB connection consistent.
ATOMIC_URL: http://atomic-server:8080
POSTGRES_HOST: postgres
POSTGRES_INTERNAL_PORT: "5432"
# Default brain target uses Docker DNS to reach the brain-api service
# in this same compose stack. Override per environment if needed.
SCHED_BRAIN_API_URL: http://brain-api:8090
SCHED_LOG_LEVEL: INFO
depends_on:
postgres:
condition: service_healthy
brain-api:
condition: service_healthy
healthcheck:
test: ["CMD-SHELL",
"test -f /tmp/scheduler.healthy && test \"$$(($$(date +%s) - $$(stat -c %Y /tmp/scheduler.healthy)))\" -lt 300"]
interval: 60s
timeout: 10s
retries: 3
start_period: 60s
networks:
- didi-network
restart: unless-stopped
volumes:
didibrain-pg-data:
name: didibrain-pg-data
didibrain-atomic-data:
name: didibrain-atomic-data
networks:
didi-network:
external: true # single shared network for all DIDI + AI platform stacks

13
ai_platform/modules/didi_brain/lint/__init__.py Normal file
View file

@ -0,0 +1,13 @@
"""Lint pass — cross-corpus contradiction detection.
For each claim atom in the brain, find semantic neighbors and classify each
candidate pair as EQUIVALENT, CONTRADICTORY, or INCOMPARABLE via Qwen 397B.
Contradictions are stored in `lint/_contradictions.json` with idempotency
markers so reruns only process new pairs.
Entry points:
python scripts/10_run_lint.py # full corpus
python scripts/10_run_lint.py --limit N # cap source claims
python scripts/10_run_lint.py --force # re-evaluate cached pairs
python scripts/11_show_contradictions.py # read state + render
"""

143
ai_platform/modules/didi_brain/lint/_state.py Normal file
View file

@ -0,0 +1,143 @@
"""Persistent state for the Lint pass.
A single JSON file at `lint/_contradictions.json` stores every pair we have
already evaluated, keyed by a stable pair_hash. Re-runs skip any pair already
evaluated at the current prompt version.
Only CONTRADICTORY verdicts are the "interesting output" but we keep
EQUIVALENT / INCOMPARABLE too because:
- EQUIVALENT pairs are paraphrase clusters (useful later for canonicalization)
- All labels matter for the idempotency ledger
"""
from __future__ import annotations
import hashlib
import json
from dataclasses import asdict, dataclass, field
from datetime import datetime, timezone
from pathlib import Path
_STATE_FILE = Path(__file__).resolve().parent / "_contradictions.json"
STATE_VERSION = "v1"
def pair_hash(atom_a_id: str, atom_b_id: str) -> str:
"""Canonical hash of an (a, b) pair, invariant under swap."""
lo, hi = sorted((atom_a_id, atom_b_id))
return hashlib.sha1(f"{lo}|{hi}".encode("utf-8")).hexdigest()[:16]
@dataclass(slots=True, frozen=True)
class PairVerdict:
pair_hash: str
atom_a_id: str
atom_b_id: str
atom_a_url: str
atom_b_url: str
atom_a_claim: str
atom_b_claim: str
label: str # EQUIVALENT / CONTRADICTORY / INCOMPARABLE
confidence: float
similarity: float # the first-stage embedding similarity that selected this pair
detected_at: str
prompt_version: str
error: str | None = None
@dataclass(slots=True)
class LintStats:
atoms_seen: int = 0
candidates_generated: int = 0
pairs_evaluated: int = 0
pairs_skipped_cached: int = 0
contradictory: int = 0
equivalent: int = 0
incomparable: int = 0
errors: int = 0
class LintState:
"""Loads/saves the contradiction ledger JSON with idempotency support."""
def __init__(self, path: Path | None = None):
self._path = path or _STATE_FILE
self._evaluated: dict[str, PairVerdict] = {}
self._prompt_version: str = "v1"
self._last_run: str | None = None
self._load()
def _load(self) -> None:
if not self._path.exists():
return
try:
data = json.loads(self._path.read_text(encoding="utf-8"))
except (json.JSONDecodeError, OSError):
# Corrupt or unreadable state — start fresh rather than crash.
return
self._prompt_version = data.get("prompt_version", "v1")
self._last_run = data.get("last_run")
for raw in data.get("evaluated", []):
try:
pv = PairVerdict(**raw)
except TypeError:
continue
self._evaluated[pv.pair_hash] = pv
def set_prompt_version(self, pv: str) -> None:
self._prompt_version = pv
def already_evaluated(self, h: str, prompt_version: str) -> bool:
pv = self._evaluated.get(h)
return pv is not None and pv.prompt_version == prompt_version
def get(self, h: str) -> PairVerdict | None:
return self._evaluated.get(h)
def upsert(self, verdict: PairVerdict) -> None:
self._evaluated[verdict.pair_hash] = verdict
@property
def all_verdicts(self) -> list[PairVerdict]:
return list(self._evaluated.values())
@property
def all_contradictions(self) -> list[PairVerdict]:
return [v for v in self._evaluated.values() if v.label == "CONTRADICTORY"]
@property
def all_equivalents(self) -> list[PairVerdict]:
return [v for v in self._evaluated.values() if v.label == "EQUIVALENT"]
def save(self) -> None:
"""Write the full ledger to disk atomically (write-temp + rename)."""
self._path.parent.mkdir(parents=True, exist_ok=True)
contras = self.all_contradictions
equivs = self.all_equivalents
body = {
"version": STATE_VERSION,
"prompt_version": self._prompt_version,
"last_run": datetime.now(timezone.utc).isoformat(),
"stats": {
"total_pairs": len(self._evaluated),
"contradictory": len(contras),
"equivalent": len(equivs),
"incomparable": len(self._evaluated) - len(contras) - len(equivs),
},
"evaluated": [asdict(v) for v in self._evaluated.values()],
}
tmp = self._path.with_suffix(self._path.suffix + ".tmp")
tmp.write_text(
json.dumps(body, indent=2, ensure_ascii=False),
encoding="utf-8",
)
tmp.replace(self._path)
@property
def path(self) -> Path:
return self._path
def now_iso() -> str:
return datetime.now(timezone.utc).isoformat()

114
ai_platform/modules/didi_brain/lint/detector.py Normal file
View file

@ -0,0 +1,114 @@
"""Single-pair NLI classification for the Lint pass.
Takes two claim texts and asks Qwen 397B whether they are EQUIVALENT,
CONTRADICTORY, or INCOMPARABLE. Returns a dataclass with label +
confidence + optional error. Never raises — all failure paths fall
through to INCOMPARABLE with an error string.
"""
from __future__ import annotations
import asyncio
from dataclasses import dataclass
from pathlib import Path
from shared.config import LlmRole
from shared.llm_client import LlmClient, LlmError
from shared.logging import get_logger
log = get_logger(__name__)
PROMPT_VERSION = "v1"
_PROMPT_PATH = (
Path(__file__).resolve().parent / "prompts" / f"pair_nli_{PROMPT_VERSION}.md"
)
ALLOWED_LABELS = {"EQUIVALENT", "CONTRADICTORY", "INCOMPARABLE"}
MAX_CLAIM_CHARS = 1500 # truncate long evidence before sending to LLM
PER_CALL_TIMEOUT_S = 30.0
_PROMPT: str | None = None
def _load_prompt() -> str:
global _PROMPT
if _PROMPT is None:
_PROMPT = _PROMPT_PATH.read_text(encoding="utf-8")
return _PROMPT
@dataclass(slots=True, frozen=True)
class PairClassification:
label: str
confidence: float
error: str | None = None
async def classify_pair(
llm: LlmClient,
*,
claim_a: str,
claim_b: str,
timeout_s: float = PER_CALL_TIMEOUT_S,
) -> PairClassification:
if not claim_a.strip() or not claim_b.strip():
return PairClassification(
label="INCOMPARABLE", confidence=0.0, error="empty_claim"
)
prompt = (
_load_prompt()
.replace("{claim_a}", claim_a[:MAX_CLAIM_CHARS])
.replace("{claim_b}", claim_b[:MAX_CLAIM_CHARS])
)
try:
result, _usage = await asyncio.wait_for(
llm.chat_json(
role=LlmRole.REASONING,
system=(
"You are a precise claim comparison classifier. "
"Respond with strictly valid JSON only, no commentary."
),
user=prompt,
max_tokens=120,
temperature=0.0,
),
timeout=timeout_s,
)
except asyncio.TimeoutError:
return PairClassification(
label="INCOMPARABLE", confidence=0.0, error="timeout"
)
except LlmError as e:
return PairClassification(
label="INCOMPARABLE", confidence=0.0, error=f"llm:{e}"
)
except Exception as e: # noqa: BLE001
return PairClassification(
label="INCOMPARABLE",
confidence=0.0,
error=f"{type(e).__name__}:{e}",
)
if not isinstance(result, dict):
return PairClassification(
label="INCOMPARABLE", confidence=0.0, error="non_dict_response"
)
raw_label = (result.get("label") or "").strip().upper()
try:
conf = float(result.get("confidence", 0))
except (TypeError, ValueError):
conf = 0.0
conf = max(0.0, min(1.0, conf))
if raw_label not in ALLOWED_LABELS:
return PairClassification(
label="INCOMPARABLE",
confidence=0.0,
error=f"bad_label:{raw_label[:40]}",
)
return PairClassification(label=raw_label, confidence=conf)

93
ai_platform/modules/didi_brain/lint/pairs.py Normal file
View file

@ -0,0 +1,93 @@
"""Candidate pair generation for the Lint pass.
For each source claim atom, we pull its nearest neighbors via the Atomic
/api/atoms/{id}/similar endpoint (which uses pgvector kNN under the hood).
Only Type/Claim neighbors are kept; Type/Document neighbors are dropped so
we only compare claim-to-claim.
Pairs are canonicalized as (lower_id, higher_id) so (A, B) and (B, A)
produce the same CandidatePair and don't get evaluated twice.
"""
from __future__ import annotations
from dataclasses import dataclass
from shared.atomic_api import AtomicClient
from shared.logging import get_logger
log = get_logger(__name__)
# Atoms with similarity below this are considered too unrelated to be worth
# an NLI call. Between 0.55 and 0.95 is the interesting band — below is
# "probably different topic", above is "almost certainly the same text".
MIN_PAIR_SIMILARITY = 0.55
NEIGHBORS_PER_CLAIM = 15
@dataclass(slots=True, frozen=True)
class CandidatePair:
"""A pair of claim atom ids to evaluate, canonicalized lo<hi."""
atom_a_id: str
atom_b_id: str
similarity: float
def _is_claim_hit(hit_tags: list[dict]) -> bool:
return any(t.get("name") == "Claim" for t in hit_tags)
async def generate_candidates(
atomic: AtomicClient,
*,
source_atom_ids: list[str],
min_similarity: float = MIN_PAIR_SIMILARITY,
neighbors_per: int = NEIGHBORS_PER_CLAIM,
) -> list[CandidatePair]:
"""Walk each source atom and collect canonical candidate pairs.
Limitation: Atomic's /similar endpoint returns Document and Claim atoms
mixed together. We filter client-side for Type/Claim. It might be worth
upstream adding a tag filter later, but for this corpus size the current
approach is fine.
"""
seen: set[tuple[str, str]] = set()
out: list[CandidatePair] = []
for idx, src_id in enumerate(source_atom_ids):
try:
hits = await atomic.find_similar(
src_id, threshold=min_similarity, limit=neighbors_per
)
except Exception as e: # noqa: BLE001
log.warning(
"find_similar_failed", atom_id=src_id, error=f"{type(e).__name__}:{e}"
)
continue
for h in hits:
if h.atom_id == src_id:
continue
if not _is_claim_hit(h.tags):
continue
lo, hi = sorted((src_id, h.atom_id))
key = (lo, hi)
if key in seen:
continue
seen.add(key)
out.append(
CandidatePair(
atom_a_id=lo, atom_b_id=hi, similarity=round(h.similarity, 4)
)
)
if (idx + 1) % 50 == 0:
log.info(
"candidates_progress",
processed=idx + 1,
total=len(source_atom_ids),
pairs=len(out),
)
return out

37
ai_platform/modules/didi_brain/lint/prompts/pair_nli_v1.md Normal file
View file

@ -0,0 +1,37 @@
You are a claim comparison classifier for a knowledge graph auditor.
Given two factual claims extracted from different sources in the same knowledge base, decide their logical relationship. Both claims are typically about the same general topic.
# Labels
- **EQUIVALENT** — A and B assert the same factual proposition, just in different words (or different languages). If one is a paraphrase, translation, or near-restatement of the other, the label is EQUIVALENT.
- **CONTRADICTORY** — A and B make claims that cannot both be true. If one asserts X and the other asserts NOT-X (or something logically incompatible), the label is CONTRADICTORY.
- **INCOMPARABLE** — A and B are on the same topic but make independent, non-overlapping assertions. One is not entailed or denied by the other. Use this as your default when unsure.
# Rules
1. Focus ONLY on the logical/factual relationship between the two claims themselves. Do not consider the sources' credibility or intent.
2. Same-language and cross-language pairs are judged identically — meaning matters, not wording.
3. Minor numerical differences (e.g., "14 cases" vs "15 cases") count as CONTRADICTORY only when the difference is clearly factual and specific, not approximate reporting.
4. A claim about X that is a SUBSET of a claim about X is EQUIVALENT if it means the same thing; otherwise INCOMPARABLE.
5. A retraction/correction claim ("study was found to be fraudulent") is CONTRADICTORY to the original claim it retracts.
6. If either claim is too vague, too broad, or you cannot decide — use INCOMPARABLE.
7. Confidence reflects how clean the relationship is:
- 0.9-1.0: unambiguous, single-interpretation
- 0.7-0.9: clear but with minor caveats
- 0.5-0.7: probable but could be argued
- below 0.5: you are guessing — prefer INCOMPARABLE with a low confidence
# Output format — STRICT
Respond with ONLY this JSON object. No preamble, no markdown fences, no commentary.
```
{"label": "EQUIVALENT|CONTRADICTORY|INCOMPARABLE", "confidence": 0.0-1.0}
```
# Input
CLAIM A: {claim_a}
CLAIM B: {claim_b}

92
ai_platform/modules/didi_brain/lint/reporter.py Normal file
View file

@ -0,0 +1,92 @@
"""Pretty printing for the Lint pass output.
Called from scripts/10_run_lint.py after a run, and from
scripts/11_show_contradictions.py for ad-hoc inspection of the state file
without re-running classification.
"""
from __future__ import annotations
from rich.console import Console
from rich.panel import Panel
from rich.table import Table
from lint._state import LintState, LintStats, PairVerdict
console = Console()
def render_stats(stats: LintStats) -> None:
t = Table(title="Lint pass stats", show_lines=False)
t.add_column("Metric", style="bold")
t.add_column("Count", justify="right")
t.add_row("atoms seen", str(stats.atoms_seen))
t.add_row("candidate pairs", str(stats.candidates_generated))
t.add_row("[dim]skipped (cached)[/dim]", str(stats.pairs_skipped_cached))
t.add_row("pairs evaluated", str(stats.pairs_evaluated))
t.add_row("[red]CONTRADICTORY[/red]", str(stats.contradictory))
t.add_row("[yellow]EQUIVALENT[/yellow]", str(stats.equivalent))
t.add_row("[dim]INCOMPARABLE[/dim]", str(stats.incomparable))
t.add_row("errors", str(stats.errors))
console.print(t)
def render_contradictions(
state: LintState,
*,
top_n: int = 10,
min_confidence: float = 0.7,
) -> None:
contras = [v for v in state.all_contradictions if v.confidence >= min_confidence]
contras.sort(key=lambda v: (-v.confidence, -v.similarity))
header = (
f"[bold red]CONTRADICTIONS[/bold red] "
f"({len(contras)} with confidence >= {min_confidence:.2f}, showing top {top_n})"
)
console.print(Panel.fit(header))
if not contras:
console.print(
"[dim]no contradictions above threshold. "
"The current Wikipedia-only corpus is self-consistent by design, "
"which is expected for a single well-curated source. "
"Add diverse sources to surface real disagreement.[/dim]"
)
return
for i, v in enumerate(contras[:top_n], 1):
console.print()
console.print(
f"[bold]#{i}[/bold] "
f"confidence=[red]{v.confidence:.2f}[/red] "
f"(embed sim {v.similarity:.2f})"
)
console.print(f" [green]A:[/green] {v.atom_a_claim[:240]}")
console.print(f" [dim]→ {v.atom_a_url}[/dim]")
console.print(f" [red]B:[/red] {v.atom_b_claim[:240]}")
console.print(f" [dim]→ {v.atom_b_url}[/dim]")
def render_equivalents(
state: LintState, *, top_n: int = 10, min_confidence: float = 0.85
) -> None:
equivs = [v for v in state.all_equivalents if v.confidence >= min_confidence]
equivs.sort(key=lambda v: (-v.confidence, -v.similarity))
header = (
f"[bold yellow]PARAPHRASE CLUSTERS[/bold yellow] "
f"({len(equivs)} with confidence >= {min_confidence:.2f}, showing top {top_n})"
)
console.print()
console.print(Panel.fit(header))
if not equivs:
console.print("[dim]no paraphrase clusters above threshold[/dim]")
return
for i, v in enumerate(equivs[:top_n], 1):
console.print()
console.print(f"[bold]#{i}[/bold] confidence=[yellow]{v.confidence:.2f}[/yellow]")
console.print(f" · {v.atom_a_claim[:200]}")
console.print(f" · {v.atom_b_claim[:200]}")

258
ai_platform/modules/didi_brain/lint/runner.py Normal file
View file

@ -0,0 +1,258 @@
"""Orchestrator for the Lint pass.
Top-level flow:
1. Pull every Type/Claim atom id from Atomic
2. For each claim, ask Atomic for its nearest neighbors → candidate pairs
3. Drop pairs we've already evaluated at the current prompt version
4. Fetch the full body of each unique atom referenced by the new pairs
5. Run Qwen NLI pair classification with bounded parallelism
6. Record every verdict in the state file; save every PERIODIC_SAVE_EVERY
pairs so a crash mid-run doesn't lose everything
The runner is safe to re-run at any time — the state file makes it fully
idempotent, and we ALWAYS save on exit even on exception.
"""
from __future__ import annotations
import asyncio
import time
from shared.atomic_api import AtomicClient
from shared.config import settings
from shared.llm_client import LlmClient
from shared.logging import get_logger
from shared.taxonomy import TagResolver
# Cross-module reuse: the same function the extractor/brain_api use to
# pull (claim_text, stance, parent_id) out of a Type/Claim atom body.
from brain_api.services.mapping import parse_claim_atom_body
from lint._state import LintState, LintStats, PairVerdict, now_iso, pair_hash
from lint.detector import PROMPT_VERSION, classify_pair
from lint.pairs import CandidatePair, generate_candidates
log = get_logger(__name__)
# Bounded parallelism for NLI calls. We have two llama.cpp backends behind
# the router; anything more than that just queues at the backend and eats
# our per-call timeout. See brain_api/services/nli.py for the same reasoning.
MAX_PARALLEL = 2
# Write the state file every N verdicts so a crash or Ctrl-C doesn't erase
# the whole run. Saving is cheap (small JSON file).
PERIODIC_SAVE_EVERY = 25
# Chunked atom fetch to avoid hammering /api/atoms/{id} with one huge gather.
FETCH_CHUNK = 20
async def _load_claim_atom_ids(
atomic: AtomicClient,
*,
type_claim_id: str,
limit: int | None,
) -> list[str]:
"""Page through /api/atoms?tag_id=<Type/Claim> and collect ids."""
page_size = 100
offset = 0
out: list[str] = []
while True:
result = await atomic.list_atoms(
limit=page_size, offset=offset, tag_id=type_claim_id
)
atoms = result.get("atoms") or (result if isinstance(result, list) else [])
if not atoms:
break
for a in atoms:
aid = a.get("id")
if aid:
out.append(aid)
if limit and len(out) >= limit:
return out
if len(atoms) < page_size:
break
offset += page_size
return out
async def _fetch_atoms_bulk(
atomic: AtomicClient, atom_ids: list[str], *, chunk: int = FETCH_CHUNK
) -> dict[str, dict]:
"""Fetch full atom bodies in parallel chunks. Missing atoms are dropped."""
out: dict[str, dict] = {}
total = len(atom_ids)
for i in range(0, total, chunk):
ids = atom_ids[i : i + chunk]
results = await asyncio.gather(
*[atomic.get_atom(a) for a in ids], return_exceptions=True
)
for a, r in zip(ids, results, strict=True):
if isinstance(r, dict):
out[a] = r
if (i + chunk) % 200 == 0 or (i + chunk) >= total:
log.info("fetch_progress", fetched=len(out), total=total)
return out
async def run_lint_pass(
*,
limit_atoms: int | None = None,
force: bool = False,
) -> LintStats:
if not settings.atomic_token:
raise RuntimeError("ATOMIC_TOKEN missing — can't talk to brain")
resolver = TagResolver()
type_claim_id = resolver.require("Type/Claim")
state = LintState()
state.set_prompt_version(PROMPT_VERSION)
stats = LintStats()
async with AtomicClient() as atomic, LlmClient() as llm:
# ------------------------------------------------------ 1. load atoms
t0 = time.perf_counter()
claim_ids = await _load_claim_atom_ids(
atomic, type_claim_id=type_claim_id, limit=limit_atoms
)
stats.atoms_seen = len(claim_ids)
log.info(
"lint_atoms_loaded",
count=stats.atoms_seen,
elapsed_s=round(time.perf_counter() - t0, 1),
)
if not claim_ids:
return stats
# ------------------------------------------ 2. candidate pair generation
t0 = time.perf_counter()
candidates = await generate_candidates(atomic, source_atom_ids=claim_ids)
stats.candidates_generated = len(candidates)
log.info(
"lint_candidates_built",
pairs=stats.candidates_generated,
elapsed_s=round(time.perf_counter() - t0, 1),
)
if not candidates:
return stats
# ----------------------------------- 3. filter out already-evaluated pairs
to_eval: list[CandidatePair] = []
for c in candidates:
h = pair_hash(c.atom_a_id, c.atom_b_id)
if not force and state.already_evaluated(h, PROMPT_VERSION):
stats.pairs_skipped_cached += 1
continue
to_eval.append(c)
log.info(
"lint_filter_done",
new_pairs=len(to_eval),
cached=stats.pairs_skipped_cached,
)
if not to_eval:
state.save()
return stats
# -------------------------------------- 4. pre-fetch full atom bodies
needed_ids: set[str] = set()
for c in to_eval:
needed_ids.add(c.atom_a_id)
needed_ids.add(c.atom_b_id)
t0 = time.perf_counter()
full_atoms = await _fetch_atoms_bulk(atomic, list(needed_ids))
log.info(
"atoms_fetched",
count=len(full_atoms),
needed=len(needed_ids),
elapsed_s=round(time.perf_counter() - t0, 1),
)
# Cache parsed claim bodies so each one is parsed once, not per-pair
parsed_by_id: dict[str, tuple[str, str]] = {}
for atom_id, full in full_atoms.items():
content = full.get("content") or ""
text, _stance, _parent_id = parse_claim_atom_body(content)
parent_url = (full.get("source_url") or "").split("#", 1)[0]
if text:
parsed_by_id[atom_id] = (text, parent_url)
# -------------------------------------- 5. classify pairs (parallel)
sem = asyncio.Semaphore(MAX_PARALLEL)
total = len(to_eval)
# Mutable counters so we can log inside the coroutine
progress = {"completed": 0}
async def _worker(pair: CandidatePair) -> PairVerdict | None:
a = parsed_by_id.get(pair.atom_a_id)
b = parsed_by_id.get(pair.atom_b_id)
if not a or not b:
return None
text_a, url_a = a
text_b, url_b = b
async with sem:
result = await classify_pair(llm, claim_a=text_a, claim_b=text_b)
progress["completed"] += 1
if progress["completed"] % 20 == 0:
log.info(
"lint_progress",
done=progress["completed"],
total=total,
pct=round(progress["completed"] / total * 100, 1),
)
return PairVerdict(
pair_hash=pair_hash(pair.atom_a_id, pair.atom_b_id),
atom_a_id=pair.atom_a_id,
atom_b_id=pair.atom_b_id,
atom_a_url=url_a,
atom_b_url=url_b,
atom_a_claim=text_a,
atom_b_claim=text_b,
label=result.label,
confidence=result.confidence,
similarity=pair.similarity,
detected_at=now_iso(),
prompt_version=PROMPT_VERSION,
error=result.error,
)
t0 = time.perf_counter()
tasks = [_worker(c) for c in to_eval]
# Process results as they arrive so we can save periodically and
# keep the ledger fresh even during long runs.
try:
for fut in asyncio.as_completed(tasks):
verdict = await fut
if verdict is None:
stats.errors += 1
continue
state.upsert(verdict)
stats.pairs_evaluated += 1
if verdict.error is not None:
stats.errors += 1
elif verdict.label == "CONTRADICTORY":
stats.contradictory += 1
elif verdict.label == "EQUIVALENT":
stats.equivalent += 1
else:
stats.incomparable += 1
if stats.pairs_evaluated % PERIODIC_SAVE_EVERY == 0:
state.save()
finally:
# ALWAYS save — even on Ctrl-C / exception, we keep what we got
state.save()
log.info(
"lint_classify_done",
evaluated=stats.pairs_evaluated,
contradictory=stats.contradictory,
equivalent=stats.equivalent,
incomparable=stats.incomparable,
errors=stats.errors,
elapsed_s=round(time.perf_counter() - t0, 1),
)
return stats

50
ai_platform/modules/didi_brain/pyproject.toml Normal file
View file

@ -0,0 +1,50 @@
[project]
name = "didibrain"
version = "0.1.0"
description = "Knowledge brain for Didi disinformation analysis — Atomic + multilingual RAG + Lint pass"
requires-python = ">=3.11"
dependencies = [
"httpx>=0.27",
"pydantic>=2.6",
"pydantic-settings>=2.2",
"structlog>=24.1",
"python-dotenv>=1.0",
"tenacity>=8.2",
"rich>=13.7",
"tomli>=2.0",
]
[dependency-groups]
dev = [
"ruff>=0.4",
"mypy>=1.10",
"pytest>=8.0",
"pytest-asyncio>=0.23",
]
scrape = [
"patchright>=1.40",
"playwright>=1.40",
"feedparser>=6.0",
"trafilatura>=1.12",
"beautifulsoup4>=4.12",
]
[tool.ruff]
line-length = 100
target-version = "py311"
[tool.ruff.lint]
select = ["E", "F", "I", "N", "UP", "B", "SIM", "RET"]
ignore = ["E501"]
[tool.mypy]
python_version = "3.11"
strict = true
warn_unused_ignores = true
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"
[tool.hatch.build.targets.wheel]
packages = ["shared", "scraper", "extractor", "lint", "didi_client"]

49
ai_platform/modules/didi_brain/scheduler/Dockerfile Normal file
View file

@ -0,0 +1,49 @@
# syntax=docker/dockerfile:1.6
# =============================================================================
# didibrain-scheduler — feeder + auditor + watcher in a single container.
#
# Imports brain_api.* directly (same Python package) so the auditor can use
# DB pool + cache_judge + apply_judge_verdict without re-implementing them.
# Build context is the didibrain/ project root (one level up).
#
# docker build -t didibrain-scheduler -f scheduler/Dockerfile .
# =============================================================================
FROM python:3.12-slim-bookworm AS runtime
ENV PYTHONDONTWRITEBYTECODE=1 \
PYTHONUNBUFFERED=1 \
PYTHONIOENCODING=utf-8 \
PIP_NO_CACHE_DIR=1 \
PIP_DISABLE_PIP_VERSION_CHECK=1 \
PIP_ROOT_USER_ACTION=ignore
WORKDIR /app
RUN apt-get update \
&& apt-get install -y --no-install-recommends curl \
&& rm -rf /var/lib/apt/lists/*
COPY scheduler/requirements.txt /tmp/requirements.txt
RUN pip install --upgrade pip \
&& pip install -r /tmp/requirements.txt \
&& rm /tmp/requirements.txt
# Brain modules (auditor imports them directly).
COPY shared /app/shared
COPY extractor /app/extractor
COPY brain_api /app/brain_api
COPY scheduler /app/scheduler
RUN useradd --system --create-home --shell /bin/false brain \
&& chown -R brain:brain /app
USER brain
# Healthcheck: main.py touches /tmp/scheduler.healthy every 30s. If the
# file is older than 5 min, the container is considered unhealthy.
HEALTHCHECK --interval=60s --timeout=10s --start-period=60s --retries=3 \
CMD test -f /tmp/scheduler.healthy && \
test "$(( $(date +%s) - $(stat -c %Y /tmp/scheduler.healthy) ))" -lt 300 \
|| exit 1
CMD ["python", "-m", "scheduler.main"]

21
ai_platform/modules/didi_brain/scheduler/__init__.py Normal file
View file

@ -0,0 +1,21 @@
"""didibrain-scheduler — Phase C orchestrator.
Single container running three independent asyncio tasks for cache freshness
defense:
- **feeder** (Pilon 4) — pulls RSS feeds at volatility-aware intervals
(15min for volatile topics, 6h for evolving, 24h for stable) and POSTs
new articles to /v1/ingest so the brain corpus stays current.
- **auditor** (Pilon 5) — daily sweep of gold + silver atoms whose
last_audited_at is older than the audit interval, judging each against
fresh evidence and applying KEEP/INVALIDATE decisions.
- **watcher** (Pilon 9) — fast RSS pull on a small breaking-news feed set,
LLM-classifies each item to identify affected topics/entities, and
triggers /v1/cache/invalidate when the news affects cached verdicts.
Each task is independent and self-restarting — a failure in one does not
stop the others. All HTTP calls go to the brain-api container by Docker DNS
(``BRAIN_API_URL``, default ``http://brain-api:8090``).
"""
__version__ = "0.1.0"

311
ai_platform/modules/didi_brain/scheduler/auditor.py Normal file
View file

@ -0,0 +1,311 @@
"""Daily auditor — Pilon 5 of the cache freshness defense.
Sweeps non-stable cache rows whose ``last_audited_at`` is older than the
audit interval, runs the cache_judge against fresh evidence from a /v1/gather
call, and applies KEEP/INVALIDATE decisions. Atoms that survive consecutive
audits accumulate ``consecutive_audit_passes`` so the judge becomes harder
to flip them — trusted gold/silver gain inertia over time.
Pipeline per atom:
1. SELECT candidate atom from brain_analysis_atom
2. POST /v1/gather to brain to get fresh top-3 evidence for the claim
3. Extract cached_truth from result_processed (TRUE / FALSE / MIXED / UV)
4. cache_judge.judge_cache_validity(claim, cached_truth, evidence, ...)
5. analysis_atom.apply_judge_verdict(atom_id, verdict)
— INVALIDATE → expires_at = now()
— KEEP_CACHE → bump consecutive_audit_passes (rate-limited)
— NEEDS_FULL_RECHECK → no DB change, just logged
This task imports brain_api directly (same Python code) for DB pool access
and helper functions; HTTP is used only for gather (which orchestrates
search + rerank + NLI inside brain itself).
"""
from __future__ import annotations
import asyncio
import json
from datetime import datetime, timezone
from typing import Any
from brain_api.db import db
from brain_api.services.analysis_atom import (
apply_judge_verdict,
is_effectively_fresh,
)
from brain_api.services.cache_judge import (
EvidenceSnippet,
JudgeVerdict,
judge_cache_validity,
)
from scheduler.brain_client import BrainClient
from scheduler.config import settings
from shared.llm_client import LlmClient
from shared.logging import get_logger
log = get_logger(__name__)
def _extract_cached_truth(processed: dict | None) -> str:
"""Pull a cached_truth label from result_processed.
DIDI v1 schema for the claims component uses status: TRUE/FALSE/UV/OP/MIXED.
For techniques + ai_tampered the result is a structured score object
rather than a truth direction — those components don't benefit from the
NLI judge anyway (their result depends on the input text, not the world),
so we return UNVERIFIED to make the judge degrade to NEEDS_FULL_RECHECK
or KEEP_CACHE depending on evidence.
"""
if not isinstance(processed, dict):
return "UNVERIFIED"
raw = processed.get("status") or processed.get("verdict")
if not isinstance(raw, str):
return "UNVERIFIED"
s = raw.strip().upper()
if s in ("TRUE", "VT", "VERIFIED_TRUE"):
return "TRUE"
if s in ("FALSE", "VF", "VERIFIED_FALSE"):
return "FALSE"
if s in ("MIXED",):
return "MIXED"
return "UNVERIFIED"
async def _select_candidates(limit: int) -> list[dict[str, Any]]:
"""Pick atoms eligible for audit, oldest-first.
Eligibility:
- cache_tier IN ('gold','silver')
- volatility IS NOT NULL AND volatility != 'stable'
- last_audited_at IS NULL OR < now() - auditor_interval
- created_at < now() - min_age_hours (let new writes settle)
- expires_at IS NULL OR > now() (don't audit dead rows)
"""
if not db.pool:
raise RuntimeError("brain_db not connected")
sql = """
SELECT atom_id, content_hash, content_preview, component, tier,
cache_tier, volatility, topic_codes, llm_confidence,
result_processed, created_at, last_audited_at,
consecutive_audit_passes
FROM brain_analysis_atom
WHERE cache_tier IN ('gold', 'silver')
AND volatility IS NOT NULL
AND volatility <> 'stable'
AND (expires_at IS NULL OR expires_at > now())
AND created_at < now() - ($1 || ' hours')::interval
AND (
last_audited_at IS NULL
OR last_audited_at < now() - ($2 || ' seconds')::interval
)
ORDER BY last_audited_at NULLS FIRST, created_at ASC
LIMIT $3
"""
async with db.pool.acquire() as conn:
rows = await conn.fetch(
sql,
str(settings.auditor_min_age_hours),
str(settings.auditor_interval_s),
limit,
)
out: list[dict[str, Any]] = []
for r in rows:
result_processed = r["result_processed"]
if isinstance(result_processed, str):
result_processed = json.loads(result_processed)
out.append({
"atom_id": r["atom_id"],
"content_hash": r["content_hash"],
"content_preview": r["content_preview"] or "",
"component": r["component"],
"tier": r["tier"],
"cache_tier": r["cache_tier"],
"volatility": r["volatility"],
"topic_codes": list(r["topic_codes"]) if r["topic_codes"] else [],
"llm_confidence": (
float(r["llm_confidence"])
if r["llm_confidence"] is not None
else None
),
"result_processed": result_processed or {},
"created_at": r["created_at"],
"last_audited_at": r["last_audited_at"],
"consecutive_audit_passes": r["consecutive_audit_passes"],
})
return out
def _evidence_from_gather_response(resp: dict | None) -> list[EvidenceSnippet]:
"""Convert a brain /v1/gather response payload into EvidenceSnippet list."""
if not resp:
return []
items = resp.get("evidence") or []
out: list[EvidenceSnippet] = []
for it in items[:3]: # judge uses top 3 anyway
url = (it.get("url") or "").strip()
text = (
it.get("full_text")
or it.get("summary")
or it.get("snippet")
or ""
)
published = it.get("published_at")
if url and text:
out.append(EvidenceSnippet(url=url, text=text, published_at=published))
return out
async def _gather_fresh_evidence(
brain: BrainClient, *, claim: str, volatility: str
) -> list[EvidenceSnippet]:
"""Call /v1/gather over HTTP to get fresh evidence for the claim.
Disables NLI on this internal call (we run our own NLI via cache_judge
afterwards, so doing it twice would be wasteful).
"""
body = {
"claim": claim[:1500],
"max_evidence": 5,
"include_full_text": True,
"run_nli": False,
"volatility_hint": volatility,
}
try:
resp = await brain._http.post("/v1/gather", json=body)
if resp.status_code >= 400:
log.debug(
"auditor_gather_http_error",
status=resp.status_code,
)
return []
return _evidence_from_gather_response(resp.json())
except Exception as e: # noqa: BLE001
log.debug("auditor_gather_failed", error=f"{type(e).__name__}:{e}")
return []
async def _audit_one(
*,
candidate: dict[str, Any],
brain: BrainClient,
llm: LlmClient,
) -> str:
"""Run a single audit. Returns the decision label for telemetry."""
atom_id = candidate["atom_id"]
claim = candidate["content_preview"]
volatility = candidate["volatility"] or "evolving"
if not claim:
# Can't judge without a claim text — touch last_audited_at to defer
# this row and move on.
log.debug("auditor_no_claim_text", atom_id=atom_id)
return "SKIPPED"
cached_truth = _extract_cached_truth(candidate["result_processed"])
age_hours = (
datetime.now(tz=timezone.utc) - candidate["created_at"]
).total_seconds() / 3600.0
# Cheap path: if effective confidence is still high we don't even need
# to gather. Bumps consecutive_audit_passes via apply_judge_verdict.
eff_fresh = is_effectively_fresh(
base_confidence=candidate["llm_confidence"],
volatility=volatility,
age_hours=age_hours,
consecutive_audit_passes=candidate["consecutive_audit_passes"],
)
if not eff_fresh:
# Confidence has decayed below floor — auto-invalidate without
# spending an LLM call.
verdict = JudgeVerdict(
decision="INVALIDATE",
nli_skipped=True,
reasoning=(
f"effective confidence below floor "
f"(volatility={volatility}, age={age_hours:.0f}h)"
),
evaluated_at=datetime.now(tz=timezone.utc).isoformat(),
)
await apply_judge_verdict(atom_id, verdict)
log.info(
"auditor_invalidated_decay",
atom_id=atom_id,
volatility=volatility,
age_hours=round(age_hours, 1),
)
return "INVALIDATE_DECAY"
# Active path: gather + judge.
evidence = await _gather_fresh_evidence(
brain, claim=claim, volatility=volatility
)
verdict = await judge_cache_validity(
llm,
claim=claim,
cached_truth=cached_truth,
current_evidence=evidence,
volatility=volatility,
age_hours=age_hours,
consecutive_audit_passes=candidate["consecutive_audit_passes"],
)
await apply_judge_verdict(atom_id, verdict)
log.info(
"auditor_decision",
atom_id=atom_id,
decision=verdict.decision,
volatility=volatility,
cached_truth=cached_truth,
)
return verdict.decision
async def _run_one_cycle(brain: BrainClient, llm: LlmClient) -> dict[str, int]:
"""Pull candidates and audit them sequentially.
Sequential rather than parallel because each judge call hits the LLM
router — running 200 in parallel would saturate it. The local Qwen
handles ~2-4 concurrent requests well, so we could batch with a
semaphore later if cycle duration becomes an issue.
"""
candidates = await _select_candidates(settings.auditor_batch_limit)
if not candidates:
return {"audited": 0, "candidates": 0}
counts: dict[str, int] = {}
for c in candidates:
try:
decision = await _audit_one(candidate=c, brain=brain, llm=llm)
counts[decision] = counts.get(decision, 0) + 1
except Exception as e: # noqa: BLE001
log.warning(
"auditor_one_failed",
atom_id=c.get("atom_id"),
error=f"{type(e).__name__}:{e}",
)
counts["ERROR"] = counts.get("ERROR", 0) + 1
log.info(
"auditor_cycle_done",
audited=len(candidates),
breakdown=counts,
)
return {"audited": len(candidates), **counts}
async def run_auditor(brain: BrainClient, llm: LlmClient) -> None:
"""Long-running task — sweeps daily by default."""
if not settings.auditor_enabled:
log.info("auditor_disabled")
return
log.info(
"auditor_loop_start",
interval_s=settings.auditor_interval_s,
batch_limit=settings.auditor_batch_limit,
)
while True:
try:
await _run_one_cycle(brain, llm)
except Exception: # noqa: BLE001
log.exception("auditor_cycle_error")
await asyncio.sleep(settings.auditor_interval_s)

145
ai_platform/modules/didi_brain/scheduler/brain_client.py Normal file
View file

@ -0,0 +1,145 @@
"""HTTP client for the brain API.
Thin wrapper that the three scheduler tasks (feeder/auditor/watcher) use to
talk to didibrain-api. All calls have generous timeouts (some operations
internally trigger LLM calls + DB queries) and return None on failure so
the caller can decide whether to retry or just skip.
"""
from __future__ import annotations
from typing import Any
import httpx
from scheduler.config import settings
from shared.logging import get_logger
log = get_logger(__name__)
class BrainClient:
"""Async client for the brain HTTP API.
Reuse one instance per task — httpx.AsyncClient pools connections.
"""
def __init__(self, *, base_url: str | None = None) -> None:
self._base = (base_url or settings.brain_api_url).rstrip("/")
self._http = httpx.AsyncClient(
base_url=self._base,
timeout=httpx.Timeout(
settings.brain_api_timeout_s, connect=10.0
),
headers={"Content-Type": "application/json"},
)
async def aclose(self) -> None:
await self._http.aclose()
async def __aenter__(self) -> BrainClient:
return self
async def __aexit__(self, *args: Any) -> None:
await self.aclose()
# --------------------------------------------------------------- ingest
async def ingest(
self,
*,
claim: str | None,
evidence: list[dict],
default_tags: list[str] | None = None,
run_extraction: bool = True,
) -> dict | None:
"""POST /v1/ingest. Returns response dict or None on error.
Each evidence item should have: url, title, summary, full_text (opt),
publisher (opt), published_at (ISO str, opt).
"""
body = {
"claim": claim,
"evidence": evidence,
"default_tags": default_tags or [],
"run_extraction": run_extraction,
}
try:
resp = await self._http.post("/v1/ingest", json=body)
if resp.status_code >= 400:
log.warning(
"brain_ingest_http_error",
status=resp.status_code,
body=resp.text[:200],
)
return None
return resp.json()
except httpx.HTTPError as e:
log.warning("brain_ingest_failed", error=str(e))
return None
# ----------------------------------------------------------- invalidate
async def invalidate(
self,
*,
topic_codes: list[str] | None = None,
entity_canonicals: list[str] | None = None,
claim_pattern: str | None = None,
since_iso: str | None = None,
invalidate_gold: bool = False,
dry_run: bool = False,
actor: str = "scheduler",
reason: str | None = None,
) -> dict | None:
"""POST /v1/cache/invalidate. Returns counts dict or None on error."""
body: dict[str, Any] = {
"invalidate_gold": invalidate_gold,
"dry_run": dry_run,
"actor": actor,
}
if topic_codes:
body["topic_codes"] = topic_codes
if entity_canonicals:
body["entity_canonicals"] = entity_canonicals
if claim_pattern:
body["claim_pattern"] = claim_pattern
if since_iso:
body["since"] = since_iso
if reason:
body["reason"] = reason
try:
resp = await self._http.post("/v1/cache/invalidate", json=body)
if resp.status_code >= 400:
log.warning(
"brain_invalidate_http_error",
status=resp.status_code,
body=resp.text[:200],
)
return None
return resp.json()
except httpx.HTTPError as e:
log.warning("brain_invalidate_failed", error=str(e))
return None
# ------------------------------------------------------------ canonicalize
async def canonicalize(
self, *, claim: str, current_date: str | None = None
) -> dict | None:
"""POST /v1/canonicalize. Returns response dict or None on error."""
body: dict[str, Any] = {"claim": claim}
if current_date:
body["current_date"] = current_date
try:
resp = await self._http.post("/v1/canonicalize", json=body)
if resp.status_code >= 400:
return None
return resp.json()
except httpx.HTTPError:
return None
# ------------------------------------------------------------------ misc
async def health(self) -> bool:
try:
resp = await self._http.get("/health")
return resp.status_code == 200
except httpx.HTTPError:
return False

124
ai_platform/modules/didi_brain/scheduler/config.py Normal file
View file

@ -0,0 +1,124 @@
"""Configuration for the scheduler container.
All settings flow from env vars (prefix ``SCHED_``) so deployment can tune
intervals + feed lists without rebuilds. Reasonable defaults for a typical
DIDI deployment are baked in, but ``brain_api_url`` is required to fail-fast
if misconfigured.
"""
from __future__ import annotations
from functools import lru_cache
from pathlib import Path
from pydantic import Field
from pydantic_settings import BaseSettings, SettingsConfigDict
class SchedulerSettings(BaseSettings):
"""Top-level scheduler config — validated at startup."""
model_config = SettingsConfigDict(
env_file=Path(__file__).parent.parent / ".env",
env_file_encoding="utf-8",
extra="ignore",
env_prefix="SCHED_",
)
# ---- Brain API target -------------------------------------------------
brain_api_url: str = Field(
default="http://brain-api:8090",
description="Base URL of the brain HTTP API. Docker DNS by default.",
)
brain_api_timeout_s: float = Field(default=120.0)
# ---- Feeder (Pilon 4) -------------------------------------------------
feeder_enabled: bool = Field(default=True)
# Intervals (seconds) per volatility — controls how often we poll RSS
# feeds in each tier.
feeder_volatile_interval_s: int = Field(default=900) # 15 min
feeder_evolving_interval_s: int = Field(default=21600) # 6 h
feeder_stable_interval_s: int = Field(default=86400) # 24 h
feeder_max_items_per_run: int = Field(default=20)
feeder_min_published_age_s: int = Field(default=3600) # skip <1h items
feeder_max_published_age_s: int = Field(default=259200) # skip >3d items
# ---- Auditor (Pilon 5) ------------------------------------------------
auditor_enabled: bool = Field(default=True)
auditor_interval_s: int = Field(default=86400) # 24h sweep
auditor_batch_limit: int = Field(default=200)
auditor_min_age_hours: float = Field(
default=24.0,
description=(
"Don't audit atoms younger than this — they were just written, "
"judging them yields no new signal."
),
)
# ---- Breaking-news watcher (Pilon 9) ----------------------------------
watcher_enabled: bool = Field(default=True)
watcher_poll_interval_s: int = Field(default=300) # 5 min
watcher_max_items_per_run: int = Field(default=10)
watcher_min_published_age_s: int = Field(default=60) # >1min old
watcher_max_published_age_s: int = Field(default=3600) # <1h old
# ---- Logging ----------------------------------------------------------
log_level: str = Field(default="INFO")
log_json: bool = Field(default=False)
@lru_cache(maxsize=1)
def get_settings() -> SchedulerSettings:
return SchedulerSettings()
settings = get_settings()
# ----------------------------------------------------------------------------
# Default feed lists per volatility — overridable via env (FEED_VOLATILE_URLS,
# FEED_EVOLVING_URLS, FEED_STABLE_URLS, FEED_BREAKING_URLS as
# comma-separated strings) for ops flexibility.
# ----------------------------------------------------------------------------
# Volatile: fast-moving news (war, breaking events, daily politics).
# Note: Reuters retired their public RSS feeds. Operators with paid Reuters
# access can add their feed URLs via the FEED_VOLATILE_URLS env override.
DEFAULT_VOLATILE_FEEDS: list[str] = [
"https://feeds.bbci.co.uk/news/world/rss.xml",
"https://www.aljazeera.com/xml/rss/all.xml",
"https://feeds.npr.org/1004/rss.xml",
"https://www.theguardian.com/world/rss",
# Romanian
"https://www.digi24.ro/rss",
"https://www.hotnews.ro/rss",
"https://www.g4media.ro/feed",
]
# Evolving: weekly-stable topics (economy, climate, science debates).
DEFAULT_EVOLVING_FEEDS: list[str] = [
"https://feeds.bbci.co.uk/news/business/rss.xml",
"https://feeds.bbci.co.uk/news/health/rss.xml",
"https://rss.nytimes.com/services/xml/rss/nyt/Science.xml",
]
# Stable: long-cycle topics (basic science, history, settled facts).
DEFAULT_STABLE_FEEDS: list[str] = [
"https://feeds.bbci.co.uk/news/science_and_environment/rss.xml",
]
# Breaking: small set polled fast (5 min) — only the highest-credibility
# real-time wires. Each item runs through the LLM classifier to decide
# which topics/entities to invalidate caches for.
DEFAULT_BREAKING_FEEDS: list[str] = [
"https://www.theguardian.com/world/rss",
"https://feeds.bbci.co.uk/news/world/rss.xml",
"https://www.aljazeera.com/xml/rss/all.xml",
]
def parse_csv(value: str | None, default: list[str]) -> list[str]:
"""Parse a comma-separated env var into a list, falling back to default."""
if not value:
return list(default)
return [v.strip() for v in value.split(",") if v.strip()]

197
ai_platform/modules/didi_brain/scheduler/feed_parser.py Normal file
View file

@ -0,0 +1,197 @@
"""RSS feed parsing + normalization.
Thin wrapper over feedparser that returns a clean list of FeedItem records.
We deliberately keep field extraction conservative — every downstream task
works with the same minimal shape.
"""
from __future__ import annotations
import asyncio
from dataclasses import dataclass
from datetime import datetime, timedelta, timezone
from typing import Any
import feedparser
import httpx
from shared.logging import get_logger
log = get_logger(__name__)
FETCH_TIMEOUT_S = 30.0
USER_AGENT = "didibrain-scheduler/0.1 (+https://didi365.eu)"
@dataclass(slots=True, frozen=True)
class FeedItem:
"""One article retrieved from an RSS feed.
Attributes:
title: Article title (cleaned).
url: Canonical URL (link tag).
summary: Short summary or description, may be empty.
full_text: Full article body if the feed exposes it (RSS rarely does;
most feeds only have summaries — caller may fetch the URL
separately to enrich).
publisher: Hostname of the source URL.
published_at: When the article was published. UTC.
feed_url: Source feed URL (for traceability).
"""
title: str
url: str
summary: str
full_text: str
publisher: str
published_at: datetime
feed_url: str
def _parse_published(entry: Any) -> datetime | None:
"""Best-effort parser for feedparser's various date fields.
Falls back to None if the entry has no usable date.
"""
for field in ("published_parsed", "updated_parsed", "created_parsed"):
struct = getattr(entry, field, None) or entry.get(field)
if struct:
try:
# struct_time is naive; treat as UTC (most feeds are).
return datetime(*struct[:6], tzinfo=timezone.utc)
except (TypeError, ValueError):
continue
return None
def _publisher_of(url: str) -> str:
"""Extract host from URL — used as the EvidenceItem.publisher field."""
try:
from urllib.parse import urlparse
host = urlparse(url).hostname or ""
return host.lower().lstrip("www.")
except Exception: # noqa: BLE001
return ""
async def fetch_feed(feed_url: str) -> list[FeedItem]:
"""Fetch and parse a single RSS feed.
Returns an empty list on any error (logged) — the caller iterates over
many feeds and shouldn't be derailed by one bad source.
"""
try:
async with httpx.AsyncClient(
timeout=FETCH_TIMEOUT_S,
headers={"User-Agent": USER_AGENT},
follow_redirects=True,
) as client:
resp = await client.get(feed_url)
if resp.status_code >= 400:
log.warning(
"feed_fetch_http_error",
feed=feed_url,
status=resp.status_code,
)
return []
body = resp.text
except httpx.HTTPError as e:
log.warning("feed_fetch_failed", feed=feed_url, error=str(e))
return []
except Exception as e: # noqa: BLE001
log.warning(
"feed_fetch_unexpected",
feed=feed_url,
error=f"{type(e).__name__}:{e}",
)
return []
# feedparser is synchronous + CPU-bound on parse; offload to a thread so
# we don't block the event loop.
parsed = await asyncio.to_thread(feedparser.parse, body)
if parsed.bozo and not parsed.entries:
log.debug(
"feed_bozo",
feed=feed_url,
error=str(parsed.bozo_exception)[:120],
)
return []
items: list[FeedItem] = []
for entry in parsed.entries:
url = (entry.get("link") or "").strip()
if not url:
continue
title = (entry.get("title") or "").strip()
summary = (entry.get("summary") or entry.get("description") or "").strip()
# full_text rarely present — feedparser exposes 'content' on some
# feeds. Take the first content block when available.
full_text = ""
contents = entry.get("content") or []
if contents and isinstance(contents, list):
first = contents[0]
if isinstance(first, dict):
full_text = (first.get("value") or "").strip()
published_at = _parse_published(entry) or datetime.now(tz=timezone.utc)
items.append(
FeedItem(
title=title,
url=url,
summary=summary,
full_text=full_text,
publisher=_publisher_of(url),
published_at=published_at,
feed_url=feed_url,
)
)
return items
async def fetch_feeds(feed_urls: list[str]) -> list[FeedItem]:
"""Fetch many feeds in parallel, flatten results."""
if not feed_urls:
return []
tasks = [fetch_feed(u) for u in feed_urls]
results = await asyncio.gather(*tasks, return_exceptions=True)
flat: list[FeedItem] = []
for r in results:
if isinstance(r, list):
flat.extend(r)
return flat
def filter_by_age(
items: list[FeedItem],
*,
min_age_s: int,
max_age_s: int,
) -> list[FeedItem]:
"""Keep items whose age is within ``[min_age_s, max_age_s]``.
The min bound exists because some feeds publish before the article
body is fully crawlable; we'd rather wait a bit. The max bound prevents
re-ingesting old items that were already in the corpus.
"""
now = datetime.now(tz=timezone.utc)
out: list[FeedItem] = []
for item in items:
age = (now - item.published_at).total_seconds()
if age < min_age_s or age > max_age_s:
continue
out.append(item)
return out
def dedup_by_url(items: list[FeedItem]) -> list[FeedItem]:
"""Drop duplicates within a batch (same URL across multiple feeds)."""
seen: set[str] = set()
out: list[FeedItem] = []
for item in items:
key = item.url.split("?")[0].rstrip("/").lower()
if key in seen:
continue
seen.add(key)
out.append(item)
return out

204
ai_platform/modules/didi_brain/scheduler/feeder.py Normal file
View file

@ -0,0 +1,204 @@
"""Fresh feeder — Pilon 4 of the cache freshness defense.
Polls RSS feeds at volatility-aware intervals and POSTs new articles to
``/v1/ingest`` so the brain corpus stays current. The intervals are:
- volatile feeds → every 15 min (war, breaking, daily politics)
- evolving feeds → every 6 h (economy, climate, science debates)
- stable feeds → every 24 h (settled science, history)
Each cycle:
1. Fetch all feeds in the tier (parallel httpx).
2. Drop items outside the freshness window (default: ≥1h old, ≤3 days old).
3. Dedup by URL within the batch.
4. POST to /v1/ingest with run_extraction=true so brain extracts claim
atoms in the background.
Brain handles deduplication against existing atoms via canonical URL match
(see brain_api/services/ingest.py), so resending the same article is
idempotent — already-ingested items are skipped silently.
"""
from __future__ import annotations
import asyncio
import os
from datetime import datetime, timezone # noqa: F401 (timezone used below)
from scheduler.brain_client import BrainClient
from scheduler.config import (
DEFAULT_EVOLVING_FEEDS,
DEFAULT_STABLE_FEEDS,
DEFAULT_VOLATILE_FEEDS,
parse_csv,
settings,
)
from scheduler.feed_parser import (
FeedItem,
dedup_by_url,
fetch_feeds,
filter_by_age,
)
from shared.logging import get_logger
log = get_logger(__name__)
def _feed_item_to_evidence(item: FeedItem) -> dict:
"""Convert a FeedItem to the EvidenceItem shape that /v1/ingest expects.
EvidenceItem requires ``retrieved_at`` (when we fetched it) plus optional
``published_at`` (the source's publish date). Ingest.evidence_to_markdown
assembles the Document atom body from these fields.
"""
body = item.full_text or item.summary or ""
return {
"url": item.url,
"title": item.title,
"summary": item.summary,
"full_text": body,
"publisher": item.publisher,
"published_at": item.published_at.isoformat(),
"retrieved_at": datetime.now(tz=timezone.utc).isoformat(),
"credibility_score": _publisher_credibility_score(item.publisher),
"relevance_score": 0.5, # neutral; real ranking happens at gather time
}
# Heuristic credibility tiers — keeps brain_api/services/ingest.py's
# credibility_score_to_tag_path mapping coherent.
_TIER_1_PUBLISHERS = {
"reuters.com", "ap.org", "apnews.com", "bbc.co.uk", "bbc.com",
"afp.com", "npr.org", "aljazeera.com", "scientificamerican.com",
}
_TIER_2_PUBLISHERS = {
"digi24.ro", "hotnews.ro", "g4media.ro",
}
def _publisher_credibility_score(publisher: str) -> float:
"""Map publisher hostname to a coarse credibility score in [0,1].
Used by brain's credibility_score_to_tag_path to assign Credibility/Tier
tags to ingested Document atoms. Conservative defaults — anything we
don't recognize gets the middle tier.
"""
p = publisher.lower().lstrip("www.")
if p in _TIER_1_PUBLISHERS:
return 0.90
if p in _TIER_2_PUBLISHERS:
return 0.70
return 0.50
def _feeds_for_volatility(volatility: str) -> list[str]:
"""Resolve env override → default for a volatility tier."""
env_var = f"FEED_{volatility.upper()}_URLS"
defaults = {
"volatile": DEFAULT_VOLATILE_FEEDS,
"evolving": DEFAULT_EVOLVING_FEEDS,
"stable": DEFAULT_STABLE_FEEDS,
}[volatility]
return parse_csv(os.environ.get(env_var), defaults)
async def _run_one_cycle(
*, brain: BrainClient, volatility: str, max_items: int
) -> tuple[int, int]:
"""Pull feeds for one volatility tier and ingest fresh items.
Returns ``(fetched, ingested)`` — fetched is items that passed the age
filter, ingested is what brain accepted (could be lower if some were
duplicates of existing atoms).
"""
feeds = _feeds_for_volatility(volatility)
if not feeds:
return 0, 0
items = await fetch_feeds(feeds)
items = filter_by_age(
items,
min_age_s=settings.feeder_min_published_age_s,
max_age_s=settings.feeder_max_published_age_s,
)
items = dedup_by_url(items)
# Newest first, cap at max_items per cycle so a single feed flood doesn't
# overwhelm the LLM-backed extraction pipeline downstream.
items.sort(key=lambda i: i.published_at, reverse=True)
items = items[:max_items]
if not items:
return 0, 0
evidence = [_feed_item_to_evidence(i) for i in items]
# Tag each batch with its volatility so brain's classifier has a hint
# already and the resulting Document atoms can be invalidated by
# topic + volatility later.
response = await brain.ingest(
claim=None, # this is corpus refresh, not a specific user claim
evidence=evidence,
default_tags=[
f"Volatility/{volatility.capitalize()}",
],
run_extraction=True,
)
if response is None:
log.warning(
"feeder_ingest_failed",
volatility=volatility,
attempted=len(items),
)
return len(items), 0
accepted = int(response.get("accepted") or 0)
skipped = int(response.get("skipped_duplicate") or 0)
log.info(
"feeder_cycle_done",
volatility=volatility,
feeds=len(feeds),
fetched=len(items),
accepted=accepted,
skipped_dup=skipped,
errors=int(response.get("errors") or 0),
)
return len(items), accepted
async def feeder_loop(brain: BrainClient, volatility: str) -> None:
"""Long-running task — one per volatility tier.
Runs forever; each cycle catches its own exceptions so a single failure
doesn't kill the loop.
"""
interval = {
"volatile": settings.feeder_volatile_interval_s,
"evolving": settings.feeder_evolving_interval_s,
"stable": settings.feeder_stable_interval_s,
}[volatility]
log.info(
"feeder_loop_start",
volatility=volatility,
interval_s=interval,
)
while True:
try:
await _run_one_cycle(
brain=brain,
volatility=volatility,
max_items=settings.feeder_max_items_per_run,
)
except Exception as e: # noqa: BLE001
log.exception("feeder_cycle_error", volatility=volatility)
await asyncio.sleep(interval)
async def run_feeder(brain: BrainClient) -> None:
"""Launch one loop per volatility tier — runs forever."""
if not settings.feeder_enabled:
log.info("feeder_disabled")
return
await asyncio.gather(
feeder_loop(brain, "volatile"),
feeder_loop(brain, "evolving"),
feeder_loop(brain, "stable"),
)

146
ai_platform/modules/didi_brain/scheduler/main.py Normal file
View file

@ -0,0 +1,146 @@
"""Scheduler entry point — orchestrates feeder + auditor + watcher.
Runs as a single container with three independent asyncio tasks. Each task
is supervisor-style: catches its own exceptions and continues, so a failure
in one (e.g., RSS feed temporarily down) doesn't kill the others.
Health check: writes ``/tmp/scheduler.healthy`` periodically. Docker
healthcheck probes the file's mtime to detect a stuck loop.
"""
from __future__ import annotations
import asyncio
import signal
import time
from pathlib import Path
from brain_api.db import db
from scheduler.auditor import run_auditor
from scheduler.brain_client import BrainClient
from scheduler.config import settings
from scheduler.feeder import run_feeder
from scheduler.watcher import run_watcher
from shared.llm_client import LlmClient
from shared.logging import setup_logging, get_logger
log = get_logger(__name__)
HEALTH_FILE = Path("/tmp/scheduler.healthy")
HEALTH_INTERVAL_S = 30.0
async def _heartbeat() -> None:
"""Periodically touch the health file so healthcheck sees activity."""
while True:
try:
HEALTH_FILE.write_text(str(time.time()))
except Exception: # noqa: BLE001
pass
await asyncio.sleep(HEALTH_INTERVAL_S)
async def _wait_brain_ready(brain: BrainClient, *, max_attempts: int = 60) -> None:
"""Spin until brain-api answers /health, with bounded retries.
Compose dependencies don't always order brain-api before scheduler at
runtime; we'd rather wait than crash on first call.
"""
for attempt in range(max_attempts):
if await brain.health():
log.info("brain_api_reachable", attempts=attempt + 1)
return
await asyncio.sleep(2.0)
log.warning("brain_api_unreachable_after_retries", attempts=max_attempts)
async def _supervised(name: str, coro_factory) -> None:
"""Wrap a long-running task so a crash inside doesn't unwind the rest.
The task never returns under normal operation; if it raises, we log and
restart after a short backoff.
"""
backoff = 5.0
while True:
try:
await coro_factory()
log.warning("supervised_task_returned", name=name)
except asyncio.CancelledError:
log.info("supervised_task_cancelled", name=name)
raise
except Exception: # noqa: BLE001
log.exception("supervised_task_crashed", name=name)
await asyncio.sleep(backoff)
backoff = min(backoff * 2, 300.0) # cap at 5 min
async def main() -> None:
setup_logging()
log.info(
"scheduler_starting",
feeder=settings.feeder_enabled,
auditor=settings.auditor_enabled,
watcher=settings.watcher_enabled,
brain_url=settings.brain_api_url,
)
# Connect to PG (auditor needs direct DB access for SELECT/UPDATE).
try:
await db.connect()
except Exception: # noqa: BLE001
log.exception("scheduler_db_connect_failed")
# Without DB the auditor can't run, but feeder + watcher only need
# HTTP — degrade gracefully rather than exit.
pass
brain = BrainClient()
llm = LlmClient()
await _wait_brain_ready(brain)
# Graceful shutdown: cancel all tasks on SIGTERM/SIGINT.
loop = asyncio.get_running_loop()
stop_event = asyncio.Event()
def _trigger_stop() -> None:
stop_event.set()
for sig in (signal.SIGTERM, signal.SIGINT):
try:
loop.add_signal_handler(sig, _trigger_stop)
except NotImplementedError:
# Windows / restricted env — ignore.
pass
tasks = [
asyncio.create_task(_heartbeat(), name="heartbeat"),
asyncio.create_task(
_supervised("feeder", lambda: run_feeder(brain)),
name="feeder",
),
asyncio.create_task(
_supervised("auditor", lambda: run_auditor(brain, llm)),
name="auditor",
),
asyncio.create_task(
_supervised("watcher", lambda: run_watcher(brain, llm)),
name="watcher",
),
]
try:
await stop_event.wait()
finally:
log.info("scheduler_stopping")
for t in tasks:
t.cancel()
# Give tasks a moment to clean up.
await asyncio.gather(*tasks, return_exceptions=True)
await brain.aclose()
await llm.aclose()
await db.close()
log.info("scheduler_stopped")
if __name__ == "__main__":
asyncio.run(main())

18
ai_platform/modules/didi_brain/scheduler/requirements.txt Normal file
View file

@ -0,0 +1,18 @@
# Scheduler container — feeder + auditor + watcher.
# Inherits brain_api dependencies (we import brain_api.* directly) plus
# RSS parsing.
# Inherit the brain_api stack
fastapi>=0.135,<0.140
uvicorn>=0.44,<0.50
httpx>=0.28,<0.30
pydantic>=2.12,<3.0
pydantic-settings>=2.13,<3.0
python-dotenv>=1.2,<2.0
tenacity>=9.1,<10.0
structlog>=25.5,<26.0
rich>=14.3,<15.0
asyncpg>=0.30,<0.32
# RSS feed parsing (the only scheduler-specific dep)
feedparser>=6.0,<7.0

228
ai_platform/modules/didi_brain/scheduler/watcher.py Normal file
View file

@ -0,0 +1,228 @@
"""Breaking-news watcher — Pilon 9 of the cache freshness defense.
Polls a small set of high-credibility breaking-news RSS feeds every 5 min.
For each new item, runs the volatility classifier (LLM) to identify the
affected topics + entities, then triggers ``/v1/cache/invalidate`` on those
topics so cached verdicts that depend on the just-changed reality are
flushed within minutes of the breaking story going live.
End-to-end latency target: under 30s from RSS publish to cache invalidation
(network + LLM ≈ 5-15s in practice with local Qwen).
Memory: keeps an in-process LRU of recently-seen URLs to avoid re-classifying
the same article on every poll. Restart-safe: brain's ingest dedup also
skips already-known URLs.
"""
from __future__ import annotations
import asyncio
import os
from collections import deque
from datetime import datetime, timezone
from brain_api.services.classifier import (
ClaimVolatility,
classify_claim_volatility,
)
from brain_api.services.fact_status import canonicalize_triple
from scheduler.brain_client import BrainClient
from scheduler.config import (
DEFAULT_BREAKING_FEEDS,
parse_csv,
settings,
)
from scheduler.feed_parser import (
FeedItem,
dedup_by_url,
fetch_feeds,
filter_by_age,
)
from shared.llm_client import LlmClient
from shared.logging import get_logger
log = get_logger(__name__)
# In-process LRU of seen URLs — capped so restart doesn't accumulate forever.
SEEN_LRU_MAX = 2000
_seen_urls: deque[str] = deque(maxlen=SEEN_LRU_MAX)
_seen_set: set[str] = set()
def _mark_seen(url: str) -> None:
"""Track URL as seen, evicting oldest if at capacity."""
if url in _seen_set:
return
if len(_seen_urls) == SEEN_LRU_MAX:
# deque.append at maxlen drops the oldest; reflect in the set.
oldest = _seen_urls[0]
_seen_set.discard(oldest)
_seen_urls.append(url)
_seen_set.add(url)
def _is_seen(url: str) -> bool:
return url in _seen_set
def _build_invalidation_targets(
classification: ClaimVolatility,
) -> tuple[list[str], list[str]]:
"""Decide what to invalidate based on classifier output.
Only volatile and evolving classifications trigger invalidation —
stable items (background pieces, historical recap) shouldn't flush
anything.
Returns ``(topic_codes, entity_canonicals)``:
- topic_codes: pass through directly to /v1/cache/invalidate
- entity_canonicals: canonicalize each binding so PG can match
against cached entity_bindings JSONB
"""
if classification.volatility not in ("volatile", "evolving"):
return [], []
topics = list(classification.topic_codes or [])
canonicals: list[str] = []
for b in classification.entity_bindings or []:
if b.confidence < 0.6:
# Low-confidence extractions are noisy — skip to avoid
# accidental mass invalidation.
continue
canonicals.append(
canonicalize_triple(b.subject, b.predicate, b.obj)
)
return topics, canonicals
async def _process_one_item(
*,
item: FeedItem,
brain: BrainClient,
llm: LlmClient,
) -> str:
"""Classify one breaking item and trigger invalidation if applicable.
Returns a short label for telemetry: 'classified_no_action' /
'invalidated' / 'ignored_low_confidence' / 'classifier_failed'.
"""
# Title + summary is what we feed the classifier — full article body
# would be expensive and the headline carries the signal we need.
text = (item.title or "")
if item.summary:
text = f"{text}. {item.summary}"
if not text.strip():
return "no_text"
classification = await classify_claim_volatility(llm, claim=text)
if classification.degraded:
return "classifier_failed"
topics, canonicals = _build_invalidation_targets(classification)
if not topics and not canonicals:
return "classified_no_action"
# Dry-run first to count, then real invalidate. We tolerate partial
# successes — if a network blip kills the real call, the next watcher
# cycle will catch the same item again.
result = await brain.invalidate(
topic_codes=topics or None,
entity_canonicals=canonicals or None,
# Only invalidate verdicts written before this breaking story —
# avoids racing with concurrent writes that may have used fresh
# information already.
since_iso=None,
invalidate_gold=False,
actor=f"breaking_watcher:{item.publisher}",
reason=(
f"breaking story: {item.title[:120]} "
f"(vol={classification.volatility})"
),
)
if result is None:
return "invalidate_http_error"
log.info(
"watcher_invalidated",
title=item.title[:80],
publisher=item.publisher,
volatility=classification.volatility,
topics=topics,
entities=len(canonicals),
atoms=result.get("invalidated_atoms"),
vcache=result.get("invalidated_vcache"),
)
return "invalidated"
async def _run_one_cycle(brain: BrainClient, llm: LlmClient) -> dict[str, int]:
"""Pull breaking feeds, classify novel items, invalidate as needed."""
feed_urls = parse_csv(
os.environ.get("FEED_BREAKING_URLS"), DEFAULT_BREAKING_FEEDS
)
if not feed_urls:
return {"checked": 0, "novel": 0}
items = await fetch_feeds(feed_urls)
items = filter_by_age(
items,
min_age_s=settings.watcher_min_published_age_s,
max_age_s=settings.watcher_max_published_age_s,
)
items = dedup_by_url(items)
# Drop items already processed in a previous cycle.
novel = [i for i in items if not _is_seen(i.url)]
novel.sort(key=lambda i: i.published_at, reverse=True)
novel = novel[: settings.watcher_max_items_per_run]
if not novel:
return {"checked": len(items), "novel": 0}
results: dict[str, int] = {}
for item in novel:
try:
label = await _process_one_item(
item=item, brain=brain, llm=llm
)
results[label] = results.get(label, 0) + 1
except Exception as e: # noqa: BLE001
log.warning(
"watcher_item_failed",
url=item.url,
error=f"{type(e).__name__}:{e}",
)
results["item_error"] = results.get("item_error", 0) + 1
finally:
_mark_seen(item.url)
log.info(
"watcher_cycle_done",
feeds=len(feed_urls),
items_total=len(items),
novel=len(novel),
breakdown=results,
)
return {"checked": len(items), "novel": len(novel), **results}
async def run_watcher(brain: BrainClient, llm: LlmClient) -> None:
"""Long-running task — polls breaking-news feeds at watcher_poll_interval_s."""
if not settings.watcher_enabled:
log.info("watcher_disabled")
return
log.info(
"watcher_loop_start",
interval_s=settings.watcher_poll_interval_s,
feeds=len(
parse_csv(
os.environ.get("FEED_BREAKING_URLS"), DEFAULT_BREAKING_FEEDS
)
),
)
while True:
try:
await _run_one_cycle(brain, llm)
except Exception: # noqa: BLE001
log.exception("watcher_cycle_error")
await asyncio.sleep(settings.watcher_poll_interval_s)

433
ai_platform/modules/didi_brain/scripts/01_sanity_full.py Normal file
View file

@ -0,0 +1,433 @@
"""Critical-gate sanity check for the entire upstream stack.
Runs every check that must pass before we trust the brain to do real work:
1. LLM router reachable, models loaded
2. LLM JSON extraction (claim extraction primitive) on a disinfo topic
3. LLM NLI single-label classification (SUPPORT/CONTRADICT/NEUTRAL)
4. LLM Romanian native generation
5. LLM cross-lingual semantic understanding (RO claim ↔ EN claim)
6. Embeddings reachable, model name correct, vectors 1024-dim
7. Embedding cross-lingual cosine RO↔EN > 0.80 on disinfo test pairs
8. Embedding discriminates unrelated topics (cosine < 0.60)
9. Reranker reachable, top-1 is the most relevant document with > 100x gap
Idempotent. Safe to rerun any time. Exit code 0 = all passed, 1 = any failed.
A JSON report is written to reports/sanity_<timestamp>.json for diffing.
"""
from __future__ import annotations
import asyncio
import json
import sys
import time
from dataclasses import asdict, dataclass, field
from datetime import datetime, timezone
from pathlib import Path
from typing import Any
# Allow running as `python scripts/01_sanity_full.py` from project root
sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
from rich.console import Console # noqa: E402
from rich.table import Table # noqa: E402
from shared.config import LlmRole, settings # noqa: E402
from shared.embedding_client import EmbeddingClient, cosine # noqa: E402
from shared.llm_client import LlmClient # noqa: E402
from shared.logging import setup_logging # noqa: E402
console = Console()
@dataclass(slots=True)
class CheckResult:
name: str
passed: bool
duration_ms: int
detail: str = ""
metrics: dict[str, Any] = field(default_factory=dict)
error: str | None = None
@dataclass(slots=True)
class Report:
started_at: str
finished_at: str = ""
all_passed: bool = False
checks: list[CheckResult] = field(default_factory=list)
# ============================================================ individual checks
async def check_llm_models(llm: LlmClient) -> CheckResult:
t0 = time.perf_counter()
try:
models = await llm.list_models()
names = [m.get("id", "?") for m in models]
wanted = settings.model_reasoning
passed = wanted in names
return CheckResult(
name="LLM router /v1/models",
passed=passed,
duration_ms=int((time.perf_counter() - t0) * 1000),
detail=f"loaded={names}",
metrics={"models": names, "wanted": wanted},
error=None if passed else f"required model {wanted!r} not in router",
)
except Exception as e:
return CheckResult(
name="LLM router /v1/models",
passed=False,
duration_ms=int((time.perf_counter() - t0) * 1000),
error=f"{type(e).__name__}: {e}",
)
async def check_llm_json_extraction(llm: LlmClient) -> CheckResult:
t0 = time.perf_counter()
system = (
"You extract verifiable factual claims from text. "
"Respond ONLY with valid JSON, no commentary."
)
user = (
"Extract claims from: \"WHO data shows that the Pfizer vaccine caused "
"1,200 myocarditis cases in 2023.\" "
'Return JSON: {"claims": [{"claim": "...", "quote": "..."}]}'
)
try:
result, usage = await llm.chat_json(
role=LlmRole.REASONING, system=system, user=user, max_tokens=300
)
claims = result.get("claims", []) if isinstance(result, dict) else []
passed = len(claims) >= 1 and all("claim" in c and "quote" in c for c in claims)
return CheckResult(
name="LLM JSON claim extraction",
passed=passed,
duration_ms=int((time.perf_counter() - t0) * 1000),
detail=f"extracted {len(claims)} claim(s) — backend={usage.get('backend')}",
metrics={"claim_count": len(claims), "usage": usage, "first": claims[0] if claims else None},
error=None if passed else "no valid claim objects returned",
)
except Exception as e:
return CheckResult(
name="LLM JSON claim extraction",
passed=False,
duration_ms=int((time.perf_counter() - t0) * 1000),
error=f"{type(e).__name__}: {e}",
)
async def check_llm_nli(llm: LlmClient) -> CheckResult:
t0 = time.perf_counter()
system = "Reply with exactly one word: SUPPORT, CONTRADICT, or NEUTRAL."
user = (
"CLAIM: Childhood vaccines cause autism.\n"
"EVIDENCE: A 2019 Danish cohort study of 657,461 children found no "
"association between MMR vaccination and autism.\n\nStance:"
)
try:
label, usage = await llm.chat_label(
role=LlmRole.REASONING,
system=system,
user=user,
allowed=["SUPPORT", "CONTRADICT", "NEUTRAL"],
max_tokens=20,
)
passed = label == "CONTRADICT"
return CheckResult(
name="LLM NLI stance classification",
passed=passed,
duration_ms=int((time.perf_counter() - t0) * 1000),
detail=f"label={label} — backend={usage.get('backend')}",
metrics={"label": label, "usage": usage},
error=None if passed else f"expected CONTRADICT, got {label}",
)
except Exception as e:
return CheckResult(
name="LLM NLI stance classification",
passed=False,
duration_ms=int((time.perf_counter() - t0) * 1000),
error=f"{type(e).__name__}: {e}",
)
async def check_llm_romanian(llm: LlmClient) -> CheckResult:
t0 = time.perf_counter()
try:
text, usage = await llm.chat_text(
role=LlmRole.REASONING,
system="Respond only in Romanian, in one short sentence.",
user="Care este capitala Romaniei si ce populatie are aproximativ?",
max_tokens=100,
)
# Heuristic: response must contain Romanian-specific characters or words
ro_markers = ["București", "Bucuresti", "milion", "România", "Romania", "este"]
hits = [m for m in ro_markers if m.lower() in text.lower()]
passed = len(hits) >= 2
return CheckResult(
name="LLM Romanian native generation",
passed=passed,
duration_ms=int((time.perf_counter() - t0) * 1000),
detail=text.strip()[:200],
metrics={"hits": hits, "usage": usage},
error=None if passed else f"text does not look Romanian: {text[:100]}",
)
except Exception as e:
return CheckResult(
name="LLM Romanian native generation",
passed=False,
duration_ms=int((time.perf_counter() - t0) * 1000),
error=f"{type(e).__name__}: {e}",
)
async def check_llm_crosslingual(llm: LlmClient) -> CheckResult:
"""Sanity that the LLM understands RO↔EN claim equivalence (separate from BGE)."""
t0 = time.perf_counter()
system = (
"You analyze whether two statements express the same factual claim, "
"possibly in different languages. Respond ONLY with JSON."
)
user = (
'A: "Vaccinurile pediatrice cauzeaza autism la copii"\n'
'B: "Childhood vaccines are linked to autism"\n\n'
'Return: {"same_claim": true/false, "confidence": 0-100}'
)
try:
result, usage = await llm.chat_json(
role=LlmRole.REASONING, system=system, user=user, max_tokens=200
)
same = result.get("same_claim") if isinstance(result, dict) else None
conf = result.get("confidence", 0) if isinstance(result, dict) else 0
passed = same is True and conf >= 70
return CheckResult(
name="LLM cross-lingual claim equivalence",
passed=passed,
duration_ms=int((time.perf_counter() - t0) * 1000),
detail=f"same={same} confidence={conf} — backend={usage.get('backend')}",
metrics={"result": result, "usage": usage},
error=None if passed else f"got same={same} conf={conf}",
)
except Exception as e:
return CheckResult(
name="LLM cross-lingual claim equivalence",
passed=False,
duration_ms=int((time.perf_counter() - t0) * 1000),
error=f"{type(e).__name__}: {e}",
)
async def check_embedding_basic(embed: EmbeddingClient) -> CheckResult:
t0 = time.perf_counter()
try:
vec = await embed.embed_one("hello world")
passed = len(vec) == settings.embedding_dim
return CheckResult(
name="Embedding basic call",
passed=passed,
duration_ms=int((time.perf_counter() - t0) * 1000),
detail=f"dim={len(vec)} (wanted {settings.embedding_dim})",
metrics={"dim": len(vec)},
error=None if passed else f"dim mismatch: {len(vec)}",
)
except Exception as e:
return CheckResult(
name="Embedding basic call",
passed=False,
duration_ms=int((time.perf_counter() - t0) * 1000),
error=f"{type(e).__name__}: {e}",
)
# Test pairs for cross-lingual gate. Must hit cosine >= CROSSLINGUAL_THRESHOLD.
CROSSLINGUAL_PAIRS: list[tuple[str, str, str]] = [
("vax", "vaccinurile pediatrice cauzeaza autism", "childhood vaccines cause autism"),
("covid", "covidul a fost o pandemie globala", "COVID was a global pandemic"),
("elect", "alegerile prezidentiale din Romania 2024", "Romanian presidential elections 2024"),
("war", "razboiul din Ucraina a inceput in 2022", "the war in Ukraine started in 2022"),
]
UNRELATED_PAIRS: list[tuple[str, str, str]] = [
("vax-weather", "vaccinurile pediatrice cauzeaza autism", "the weather is nice today"),
("vax-elect", "childhood vaccines cause autism", "Romanian presidential elections 2024"),
]
CROSSLINGUAL_THRESHOLD = 0.80
UNRELATED_MAX = 0.60
async def check_embedding_crosslingual(embed: EmbeddingClient) -> CheckResult:
t0 = time.perf_counter()
try:
all_texts: list[str] = []
for _, ro, en in CROSSLINGUAL_PAIRS:
all_texts.extend([ro, en])
for _, a, b in UNRELATED_PAIRS:
all_texts.extend([a, b])
vecs = await embed.embed(all_texts)
idx = 0
cross_scores: dict[str, float] = {}
for label, _, _ in CROSSLINGUAL_PAIRS:
cross_scores[label] = cosine(vecs[idx], vecs[idx + 1])
idx += 2
unrelated_scores: dict[str, float] = {}
for label, _, _ in UNRELATED_PAIRS:
unrelated_scores[label] = cosine(vecs[idx], vecs[idx + 1])
idx += 2
cross_ok = all(s >= CROSSLINGUAL_THRESHOLD for s in cross_scores.values())
unrelated_ok = all(s <= UNRELATED_MAX for s in unrelated_scores.values())
passed = cross_ok and unrelated_ok
detail_lines = [f"{k}={v:.3f}" for k, v in cross_scores.items()]
detail_lines += [f"!{k}={v:.3f}" for k, v in unrelated_scores.items()]
return CheckResult(
name="Embedding cross-lingual cosine",
passed=passed,
duration_ms=int((time.perf_counter() - t0) * 1000),
detail=" ".join(detail_lines),
metrics={
"crosslingual": cross_scores,
"unrelated": unrelated_scores,
"threshold_cross": CROSSLINGUAL_THRESHOLD,
"threshold_unrelated": UNRELATED_MAX,
},
error=None
if passed
else f"cross_ok={cross_ok} unrelated_ok={unrelated_ok}",
)
except Exception as e:
return CheckResult(
name="Embedding cross-lingual cosine",
passed=False,
duration_ms=int((time.perf_counter() - t0) * 1000),
error=f"{type(e).__name__}: {e}",
)
async def check_reranker(embed: EmbeddingClient) -> CheckResult:
t0 = time.perf_counter()
query = "Does the Pfizer vaccine cause myocarditis?"
docs = [
"A 2022 study found rare myocarditis cases in young men after mRNA vaccination, mostly mild.",
"The Kremlin announced new sanctions on European imports yesterday.",
"Pfizer reported strong Q3 2023 earnings driven by COVID antiviral sales.",
"Danish cohort study of 657,461 children found no link between MMR vaccine and autism.",
"A case report described acute pericarditis 4 days after second Pfizer-BioNTech dose in a 17-year-old male.",
]
relevant_indices = {0, 4} # the two docs that actually answer the query
try:
results = await embed.rerank(query, docs)
top2 = {r.index for r in results[:2]}
# Top-1 must be relevant; top-2 should both be relevant; gap to #3 must be large
top1_relevant = results[0].index in relevant_indices
top2_relevant = top2 == relevant_indices
# ratio of top-1 score to score of best irrelevant
irrelevant = [r for r in results if r.index not in relevant_indices]
gap = (results[0].score / irrelevant[0].score) if irrelevant and irrelevant[0].score > 0 else float("inf")
passed = top1_relevant and top2_relevant and gap >= 100
return CheckResult(
name="Reranker top-K precision",
passed=passed,
duration_ms=int((time.perf_counter() - t0) * 1000),
detail=f"top1={results[0].index}({results[0].score:.4f}) top2_set={top2} gap≈{gap:.0f}x",
metrics={
"ranked": [{"index": r.index, "score": r.score} for r in results],
"gap_top1_vs_best_irrelevant": gap,
},
error=None
if passed
else f"top1_relevant={top1_relevant} top2_relevant={top2_relevant} gap={gap:.1f}",
)
except Exception as e:
return CheckResult(
name="Reranker top-K precision",
passed=False,
duration_ms=int((time.perf_counter() - t0) * 1000),
error=f"{type(e).__name__}: {e}",
)
# =============================================================== orchestration
async def run_all() -> Report:
report = Report(started_at=datetime.now(timezone.utc).isoformat())
async with LlmClient() as llm, EmbeddingClient() as embed:
# Run independent checks concurrently. Each check is self-contained.
coros = [
check_llm_models(llm),
check_llm_json_extraction(llm),
check_llm_nli(llm),
check_llm_romanian(llm),
check_llm_crosslingual(llm),
check_embedding_basic(embed),
check_embedding_crosslingual(embed),
check_reranker(embed),
]
results = await asyncio.gather(*coros, return_exceptions=False)
report.checks.extend(results)
report.finished_at = datetime.now(timezone.utc).isoformat()
report.all_passed = all(c.passed for c in report.checks)
return report
def render(report: Report) -> None:
table = Table(title="DidiBrain — Sanity Check", show_lines=False)
table.add_column("Check", style="bold")
table.add_column("Status", justify="center")
table.add_column("Time", justify="right")
table.add_column("Detail", overflow="fold")
for c in report.checks:
status = "[green]PASS[/green]" if c.passed else "[red]FAIL[/red]"
detail = c.detail or (c.error or "")
table.add_row(c.name, status, f"{c.duration_ms} ms", detail)
console.print(table)
if report.all_passed:
console.print("\n[bold green]ALL CHECKS PASSED[/bold green] — go ahead.\n")
else:
console.print("\n[bold red]GATE FAILED[/bold red] — fix before continuing.\n")
for c in report.checks:
if not c.passed and c.error:
console.print(f" • [red]{c.name}[/red]: {c.error}")
def save_report(report: Report) -> Path:
reports_dir = Path(__file__).resolve().parent.parent / "reports"
reports_dir.mkdir(exist_ok=True)
ts = report.started_at.replace(":", "").replace("-", "")[:15]
path = reports_dir / f"sanity_{ts}.json"
path.write_text(
json.dumps(asdict(report), indent=2, ensure_ascii=False, default=str),
encoding="utf-8",
)
return path
def main() -> int:
setup_logging()
console.print(f"[dim]Router: {settings.llm_router_url}[/dim]")
console.print(f"[dim]Embed: {settings.embedding_url} ({settings.embedding_model})[/dim]")
console.print(f"[dim]Rerank: {settings.reranker_url} ({settings.reranker_model})[/dim]\n")
try:
report = asyncio.run(run_all())
except KeyboardInterrupt:
console.print("\n[yellow]interrupted[/yellow]")
return 130
render(report)
path = save_report(report)
console.print(f"[dim]report → {path}[/dim]")
return 0 if report.all_passed else 1
if __name__ == "__main__":
raise SystemExit(main())

209
ai_platform/modules/didi_brain/scripts/02_bootstrap_atomic.py Normal file
View file

@ -0,0 +1,209 @@
"""Bootstrap a fresh atomic-server instance for DidiBrain.
Run this ONCE after `docker compose up -d` against a fresh stack. It will:
1. Wait for atomic-server to be reachable on /health
2. Check setup status; if no token exists yet, claim the instance to create one
- If a token already exists locally in .env, skip claiming and just verify
3. Configure provider settings to use BGE-M3 (via openai_compat) for embeddings
4. Disable auto_tagging (we control tagging from extractor service)
5. Sanity-poke /api/settings to confirm everything stuck
6. Patch the local .env file with ATOMIC_TOKEN if it changed
Idempotent. Safe to rerun. Will not overwrite an existing valid token.
"""
from __future__ import annotations
import asyncio
import re
import sys
import time
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
from rich.console import Console # noqa: E402
from shared.atomic_api import AtomicApiError, AtomicClient # noqa: E402
from shared.config import settings # noqa: E402
from shared.logging import setup_logging # noqa: E402
console = Console()
ENV_FILE = Path(__file__).resolve().parent.parent / ".env"
# Provider settings to push into Atomic. These tell atomic-core to use the
# vLLM-served BGE-M3 endpoint via the OpenAI-compatible interface.
PROVIDER_SETTINGS: dict[str, str] = {
"provider": "openai_compat",
"openai_compat_base_url": settings.embedding_url,
"openai_compat_embedding_model": settings.embedding_model,
"openai_compat_embedding_dimension": str(settings.embedding_dim),
"openai_compat_context_length": str(settings.embedding_max_tokens),
"openai_compat_llm_model": settings.model_reasoning, # placeholder, not called
"openai_compat_timeout_secs": "300",
"openai_compat_api_key": settings.embedding_api_key,
# We do tagging ourselves with explicit taxonomy in extractor — disable
# the built-in LLM auto-tagging to avoid Atomic trying to call the BGE
# endpoint as if it were an LLM (which would fail).
"auto_tagging_enabled": "false",
}
async def wait_for_atomic(timeout_s: int = 60) -> None:
"""Poll /health until 200 OK or timeout."""
deadline = time.monotonic() + timeout_s
last_err: str = ""
async with AtomicClient(token="") as client: # no token needed for health
while time.monotonic() < deadline:
try:
await client.health()
console.print("[green]✓[/green] atomic-server /health responding")
return
except Exception as e: # noqa: BLE001
last_err = f"{type(e).__name__}: {e}"
await asyncio.sleep(2)
raise RuntimeError(f"atomic-server not healthy after {timeout_s}s — {last_err}")
async def ensure_token() -> str:
"""Return a valid API token, creating one via /api/setup/claim if needed.
Tries (in order):
1. The token already in .env (verify it works by calling /api/settings)
2. /api/setup/status — if needs_setup, /api/setup/claim to mint one
3. Fall back to advising the user to run `docker exec ... token create`
"""
# 1. Existing token works?
if settings.atomic_token:
async with AtomicClient(token=settings.atomic_token) as c:
try:
await c.get_settings()
console.print("[green]✓[/green] existing ATOMIC_TOKEN is valid")
return settings.atomic_token
except AtomicApiError as e:
if e.status == 401:
console.print(
"[yellow]![/yellow] existing ATOMIC_TOKEN rejected (401), reclaiming"
)
else:
raise
# 2. Need to claim
async with AtomicClient(token="") as c:
status = await c.setup_status()
needs_setup = status.get("needs_setup", True)
if needs_setup:
console.print(
"[cyan]→[/cyan] instance not yet claimed, calling /api/setup/claim"
)
result = await c._request( # type: ignore[attr-defined]
"POST",
"/api/setup/claim",
json={"name": "didibrain-bootstrap"},
)
token = result.get("token") or result.get("api_token")
if not token:
raise RuntimeError(f"claim succeeded but no token in response: {result}")
console.print("[green]✓[/green] new token claimed")
return token
# 3. Already claimed but we have no token — user must mint one manually
raise RuntimeError(
"Instance is already claimed but ATOMIC_TOKEN is empty in .env.\n"
"Run: docker exec didibrain-atomic atomic-server "
"--data-dir /data token create --name didibrain\n"
"Then paste the token into .env as ATOMIC_TOKEN=..."
)
def patch_env_token(new_token: str) -> None:
"""Update ATOMIC_TOKEN= line in .env in place. Creates the line if missing."""
if not ENV_FILE.exists():
console.print(f"[red]![/red] .env not found at {ENV_FILE}, skipping write")
return
text = ENV_FILE.read_text(encoding="utf-8")
pattern = re.compile(r"^ATOMIC_TOKEN=.*$", re.MULTILINE)
if pattern.search(text):
new_text = pattern.sub(f"ATOMIC_TOKEN={new_token}", text)
else:
new_text = text.rstrip() + f"\nATOMIC_TOKEN={new_token}\n"
if new_text != text:
ENV_FILE.write_text(new_text, encoding="utf-8")
console.print(f"[green]✓[/green] wrote ATOMIC_TOKEN to {ENV_FILE.name}")
else:
console.print("[dim]·[/dim] ATOMIC_TOKEN already current in .env")
async def configure_provider(token: str) -> None:
async with AtomicClient(token=token) as c:
before = await c.get_settings()
before_provider = before.get("provider", "?")
console.print(f"[dim]current provider:[/dim] {before_provider}")
# Push our settings
applied: list[str] = []
for key, value in PROVIDER_SETTINGS.items():
current = before.get(key)
if current == value:
continue
await c.set_setting(key, value)
applied.append(key)
console.print(f" [cyan]·[/cyan] {key} = {value}")
if not applied:
console.print("[dim]· all provider settings already current[/dim]")
else:
console.print(f"[green]✓[/green] applied {len(applied)} setting(s)")
# Verify
after = await c.get_settings()
if after.get("provider") != "openai_compat":
raise RuntimeError(
f"provider did not stick: got {after.get('provider')!r}"
)
if after.get("openai_compat_base_url") != settings.embedding_url:
raise RuntimeError(
f"base_url did not stick: got {after.get('openai_compat_base_url')!r}"
)
console.print("[green]✓[/green] settings verified post-write")
async def main_async() -> int:
setup_logging()
console.print(f"[bold]Bootstrap atomic-server[/bold] @ {settings.atomic_url}")
console.print(f"[dim]target embedding endpoint:[/dim] {settings.embedding_url}\n")
try:
await wait_for_atomic(timeout_s=120)
except Exception as e: # noqa: BLE001
console.print(f"[red]✗ atomic not reachable:[/red] {e}")
return 2
try:
token = await ensure_token()
except Exception as e: # noqa: BLE001
console.print(f"[red]✗ token bootstrap failed:[/red] {e}")
return 3
patch_env_token(token)
try:
await configure_provider(token)
except Exception as e: # noqa: BLE001
console.print(f"[red]✗ provider config failed:[/red] {e}")
return 4
console.print("\n[bold green]bootstrap complete[/bold green] — ready for sanity check")
return 0
def main() -> int:
try:
return asyncio.run(main_async())
except KeyboardInterrupt:
return 130
if __name__ == "__main__":
raise SystemExit(main())

386
ai_platform/modules/didi_brain/scripts/03_sanity_atomic.py Normal file
View file

@ -0,0 +1,386 @@
"""End-to-end brain sanity check.
Validates that atomic-server + Postgres + BGE-M3 work together by:
1. Hitting /health
2. Reading /api/settings and confirming provider is openai_compat → BGE-M3
3. Creating a known test atom with a unique source_url marker
4. Polling /api/atoms/{id}/embedding-status until 'completed' (or fail at timeout)
5. Issuing a semantic search for a paraphrase that should match the test atom
6. Verifying the test atom appears in the top results with similarity above threshold
7. Cleaning up the test atom (delete) so reruns are clean
Idempotent. Cleans up on success AND on most failure paths.
Exit 0 = brain works. Exit non-zero = something's broken.
"""
from __future__ import annotations
import asyncio
import sys
import time
import uuid
from dataclasses import asdict, dataclass, field
from datetime import datetime, timezone
from pathlib import Path
from typing import Any
sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
from rich.console import Console # noqa: E402
from rich.table import Table # noqa: E402
from shared.atomic_api import AtomicApiError, AtomicClient # noqa: E402
from shared.config import settings # noqa: E402
from shared.logging import setup_logging # noqa: E402
console = Console()
# Test atom content. The text is bilingual on purpose: it lets us prove that
# BGE-M3 cross-lingual works end-to-end through the brain (not just at the
# embedding endpoint level we already validated in 01_sanity_full).
TEST_MARKER = f"didibrain-sanity-{uuid.uuid4().hex[:8]}"
TEST_SOURCE_URL = f"https://didibrain.test/sanity/{TEST_MARKER}"
TEST_ATOM_CONTENT = """# Sanity test atom
This is a synthetic atom created by the DidiBrain sanity script.
The 2019 Danish cohort study of 657,461 children found no association
between MMR vaccination and autism. The Pfizer-BioNTech mRNA vaccine has
been linked in rare cases to mild myocarditis in young men, but the
benefits clearly outweigh the risks for most populations.
In Romanian: Studiul danez din 2019 nu a gasit nicio legatura intre
vaccinul MMR si autism la copii.
"""
# Cross-lingual paraphrase used to query the brain. If BGE-M3 is wired up
# correctly, this RO query should retrieve the test atom (which mixes EN+RO).
TEST_QUERY = "studii care arata ca vaccinurile nu cauzeaza autism"
SIMILARITY_MIN = 0.4 # generous; chunked match should easily exceed this
@dataclass(slots=True)
class CheckResult:
name: str
passed: bool
duration_ms: int
detail: str = ""
metrics: dict[str, Any] = field(default_factory=dict)
error: str | None = None
@dataclass(slots=True)
class Report:
started_at: str
finished_at: str = ""
all_passed: bool = False
checks: list[CheckResult] = field(default_factory=list)
test_atom_id: str | None = None
# ============================================================ orchestration
async def check_health(client: AtomicClient) -> CheckResult:
t0 = time.perf_counter()
try:
h = await client.health()
return CheckResult(
name="atomic-server /health",
passed=True,
duration_ms=int((time.perf_counter() - t0) * 1000),
detail=str(h)[:120],
)
except Exception as e: # noqa: BLE001
return CheckResult(
name="atomic-server /health",
passed=False,
duration_ms=int((time.perf_counter() - t0) * 1000),
error=f"{type(e).__name__}: {e}",
)
async def check_provider_settings(client: AtomicClient) -> CheckResult:
t0 = time.perf_counter()
try:
s = await client.get_settings()
provider = s.get("provider")
base_url = s.get("openai_compat_base_url")
emb_model = s.get("openai_compat_embedding_model")
emb_dim = s.get("openai_compat_embedding_dimension")
ok = (
provider == "openai_compat"
and base_url == settings.embedding_url
and emb_model == settings.embedding_model
and str(emb_dim) == str(settings.embedding_dim)
)
return CheckResult(
name="provider settings (BGE-M3 wired)",
passed=ok,
duration_ms=int((time.perf_counter() - t0) * 1000),
detail=f"provider={provider} base_url={base_url} model={emb_model} dim={emb_dim}",
metrics={
"provider": provider,
"base_url": base_url,
"embedding_model": emb_model,
"embedding_dimension": emb_dim,
},
error=None if ok else "settings do not match expected BGE-M3 config",
)
except Exception as e: # noqa: BLE001
return CheckResult(
name="provider settings (BGE-M3 wired)",
passed=False,
duration_ms=int((time.perf_counter() - t0) * 1000),
error=f"{type(e).__name__}: {e}",
)
async def check_create_atom(
client: AtomicClient, report: Report
) -> CheckResult:
t0 = time.perf_counter()
try:
# First, defensive cleanup: if a previous run left an orphan, drop it.
existing = await client.get_atom_by_source_url(TEST_SOURCE_URL)
if existing:
try:
await client.delete_atom(existing["id"])
except Exception: # noqa: BLE001, S110
pass
atom = await client.create_atom(
content=TEST_ATOM_CONTENT,
source_url=TEST_SOURCE_URL,
)
atom_id = atom.get("id") or atom.get("atom_id")
if not atom_id:
return CheckResult(
name="create test atom",
passed=False,
duration_ms=int((time.perf_counter() - t0) * 1000),
error=f"no id in response: {str(atom)[:300]}",
)
report.test_atom_id = atom_id
return CheckResult(
name="create test atom",
passed=True,
duration_ms=int((time.perf_counter() - t0) * 1000),
detail=f"id={atom_id}",
metrics={"atom_id": atom_id},
)
except Exception as e: # noqa: BLE001
return CheckResult(
name="create test atom",
passed=False,
duration_ms=int((time.perf_counter() - t0) * 1000),
error=f"{type(e).__name__}: {e}",
)
async def check_embedding_pipeline(
client: AtomicClient, atom_id: str, *, timeout_s: int = 60
) -> CheckResult:
t0 = time.perf_counter()
deadline = time.monotonic() + timeout_s
last_status: str = "?"
try:
while time.monotonic() < deadline:
status = await client.get_embedding_status(atom_id)
last_status = (
status.get("embedding_status")
or status.get("status")
or "unknown"
)
# Atomic uses 'complete' (not 'completed'); accept both defensively.
if last_status in ("complete", "completed"):
return CheckResult(
name="embedding pipeline (BGE-M3 → atom)",
passed=True,
duration_ms=int((time.perf_counter() - t0) * 1000),
detail=f"{last_status} in {int(time.perf_counter() - t0)}s",
metrics={"final_status": last_status},
)
if last_status == "failed":
return CheckResult(
name="embedding pipeline (BGE-M3 → atom)",
passed=False,
duration_ms=int((time.perf_counter() - t0) * 1000),
error=f"pipeline reported failed: {status}",
)
await asyncio.sleep(1.5)
return CheckResult(
name="embedding pipeline (BGE-M3 → atom)",
passed=False,
duration_ms=int((time.perf_counter() - t0) * 1000),
error=f"timed out after {timeout_s}s — last status={last_status}",
)
except Exception as e: # noqa: BLE001
return CheckResult(
name="embedding pipeline (BGE-M3 → atom)",
passed=False,
duration_ms=int((time.perf_counter() - t0) * 1000),
error=f"{type(e).__name__}: {e}",
)
async def check_semantic_search(
client: AtomicClient, atom_id: str
) -> CheckResult:
t0 = time.perf_counter()
try:
hits = await client.search(TEST_QUERY, mode="semantic", limit=20)
if not hits:
return CheckResult(
name="semantic search retrieves test atom",
passed=False,
duration_ms=int((time.perf_counter() - t0) * 1000),
error="search returned 0 hits",
)
# Find our test atom in the hits
match = next((h for h in hits if h.atom_id == atom_id), None)
if not match:
top_ids = [h.atom_id for h in hits[:5]]
return CheckResult(
name="semantic search retrieves test atom",
passed=False,
duration_ms=int((time.perf_counter() - t0) * 1000),
error=f"test atom {atom_id} not in top {len(hits)} hits, top5={top_ids}",
)
passed = match.similarity >= SIMILARITY_MIN
return CheckResult(
name="semantic search retrieves test atom",
passed=passed,
duration_ms=int((time.perf_counter() - t0) * 1000),
detail=f"sim={match.similarity:.3f} (≥{SIMILARITY_MIN}) of {len(hits)} hits",
metrics={
"test_atom_similarity": match.similarity,
"total_hits": len(hits),
},
error=None
if passed
else f"similarity {match.similarity:.3f} below {SIMILARITY_MIN}",
)
except Exception as e: # noqa: BLE001
return CheckResult(
name="semantic search retrieves test atom",
passed=False,
duration_ms=int((time.perf_counter() - t0) * 1000),
error=f"{type(e).__name__}: {e}",
)
async def cleanup_test_atom(client: AtomicClient, atom_id: str | None) -> None:
if not atom_id:
return
try:
await client.delete_atom(atom_id)
console.print(f"[dim]· cleaned up test atom {atom_id}[/dim]")
except AtomicApiError as e:
console.print(f"[yellow]! cleanup failed:[/yellow] {e}")
async def run_all() -> Report:
report = Report(started_at=datetime.now(timezone.utc).isoformat())
if not settings.atomic_token:
console.print(
"[red]ATOMIC_TOKEN is empty in .env — run 02_bootstrap_atomic.py first[/red]"
)
report.checks.append(
CheckResult(name="precondition", passed=False, duration_ms=0,
error="ATOMIC_TOKEN missing")
)
report.finished_at = datetime.now(timezone.utc).isoformat()
return report
async with AtomicClient() as client:
report.checks.append(await check_health(client))
if not report.checks[-1].passed:
report.finished_at = datetime.now(timezone.utc).isoformat()
return report
report.checks.append(await check_provider_settings(client))
if not report.checks[-1].passed:
report.finished_at = datetime.now(timezone.utc).isoformat()
return report
report.checks.append(await check_create_atom(client, report))
if not report.checks[-1].passed or not report.test_atom_id:
report.finished_at = datetime.now(timezone.utc).isoformat()
return report
try:
report.checks.append(
await check_embedding_pipeline(client, report.test_atom_id)
)
if not report.checks[-1].passed:
return report
report.checks.append(
await check_semantic_search(client, report.test_atom_id)
)
finally:
await cleanup_test_atom(client, report.test_atom_id)
report.finished_at = datetime.now(timezone.utc).isoformat()
report.all_passed = all(c.passed for c in report.checks)
return report
# ====================================================================== render
def render(report: Report) -> None:
table = Table(title="DidiBrain — Atomic Sanity", show_lines=False)
table.add_column("Check", style="bold")
table.add_column("Status", justify="center")
table.add_column("Time", justify="right")
table.add_column("Detail", overflow="fold")
for c in report.checks:
status = "[green]PASS[/green]" if c.passed else "[red]FAIL[/red]"
detail = c.detail or (c.error or "")
table.add_row(c.name, status, f"{c.duration_ms} ms", detail)
console.print(table)
if report.all_passed:
console.print("\n[bold green]BRAIN OK[/bold green] — atomic + Postgres + BGE-M3 fully wired.\n")
else:
console.print("\n[bold red]BRAIN FAILED[/bold red]\n")
for c in report.checks:
if not c.passed and c.error:
console.print(f" • [red]{c.name}[/red]: {c.error}")
def save_report(report: Report) -> Path:
import json
reports_dir = Path(__file__).resolve().parent.parent / "reports"
reports_dir.mkdir(exist_ok=True)
ts = report.started_at.replace(":", "").replace("-", "")[:15]
path = reports_dir / f"sanity_atomic_{ts}.json"
path.write_text(
json.dumps(asdict(report), indent=2, ensure_ascii=False, default=str),
encoding="utf-8",
)
return path
def main() -> int:
setup_logging()
console.print(f"[dim]Atomic: {settings.atomic_url}[/dim]")
console.print(f"[dim]Token: {'set' if settings.atomic_token else 'NOT SET'}[/dim]\n")
try:
report = asyncio.run(run_all())
except KeyboardInterrupt:
return 130
render(report)
path = save_report(report)
console.print(f"[dim]report → {path}[/dim]")
return 0 if report.all_passed else 1
if __name__ == "__main__":
raise SystemExit(main())

125
ai_platform/modules/didi_brain/scripts/04_seed_taxonomy.py Normal file
View file

@ -0,0 +1,125 @@
"""Seed the canonical tag taxonomy into Atomic.
Reads `shared.taxonomy.TAXONOMY` and creates any missing tags in Atomic,
preserving the parent-child structure. Reuses Atomic's default root tags
(Topics, People, Locations, Organizations, Events) where they already exist.
After seeding, writes the complete `path → tag_id` map to
`shared/_tag_ids.json` for use by all downstream scripts.
Idempotent: re-running it will only create tags that don't yet exist by
(name, parent_id). Safe to invoke at any time, e.g. after pulling a newer
TAXONOMY definition.
"""
from __future__ import annotations
import asyncio
import sys
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
from rich.console import Console # noqa: E402
from rich.table import Table # noqa: E402
from shared.atomic_api import AtomicClient # noqa: E402
from shared.config import settings # noqa: E402
from shared.logging import setup_logging # noqa: E402
from shared.taxonomy import ( # noqa: E402
TAXONOMY,
TagResolver,
build_path_map_from_tags,
walk,
)
console = Console()
async def seed() -> dict[str, int]:
"""Walk TAXONOMY and create missing tags. Return summary counts."""
if not settings.atomic_token:
raise RuntimeError("ATOMIC_TOKEN missing — run 02_bootstrap_atomic.py first")
counts = {"created": 0, "existing": 0, "errors": 0}
async with AtomicClient() as client:
# 1. Pull current state
existing_tags = await client.list_tags(min_count=0)
path_map = build_path_map_from_tags(existing_tags)
console.print(
f"[dim]Atomic currently has [bold]{len(path_map)}[/bold] tags "
f"({len([p for p in path_map if '/' not in p])} roots)[/dim]"
)
# 2. Walk taxonomy depth-first; create what's missing
for path, parent_path, name in walk(TAXONOMY):
if path in path_map:
counts["existing"] += 1
continue
parent_id = path_map.get(parent_path) if parent_path else None
if parent_path and not parent_id:
console.print(
f"[red]✗[/red] cannot create {path!r}: parent {parent_path!r} "
f"missing — should have been created earlier"
)
counts["errors"] += 1
continue
try:
created = await client.create_tag(name=name, parent_id=parent_id)
new_id = created.get("id")
if not new_id:
counts["errors"] += 1
console.print(f"[red]✗[/red] {path}: no id in response: {created}")
continue
path_map[path] = new_id
counts["created"] += 1
console.print(f"[green]+[/green] {path}")
except Exception as e: # noqa: BLE001
counts["errors"] += 1
console.print(f"[red]✗[/red] {path}: {type(e).__name__}: {e}")
# 3. Re-read full state to make sure path_map is fresh and complete
final_tags = await client.list_tags(min_count=0)
final_map = build_path_map_from_tags(final_tags)
# 4. Persist the map
resolver = TagResolver()
resolver.save(final_map)
console.print(
f"\n[dim]wrote [bold]{len(final_map)}[/bold] entries to "
f"shared/_tag_ids.json[/dim]"
)
return counts
def render_summary(counts: dict[str, int]) -> None:
table = Table(title="Taxonomy seeding")
table.add_column("Status", style="bold")
table.add_column("Count", justify="right")
style = {"created": "green", "existing": "dim", "errors": "red"}
for k, v in counts.items():
table.add_row(f"[{style[k]}]{k}[/{style[k]}]", str(v))
console.print(table)
def main() -> int:
setup_logging()
console.print(f"[dim]Atomic: {settings.atomic_url}[/dim]\n")
try:
counts = asyncio.run(seed())
except KeyboardInterrupt:
return 130
except Exception as e: # noqa: BLE001
console.print(f"[red]seeding failed:[/red] {e}")
return 1
render_summary(counts)
if counts["errors"]:
console.print("\n[red]some tags failed[/red]")
return 1
console.print("\n[bold green]taxonomy seeded[/bold green]")
return 0
if __name__ == "__main__":
raise SystemExit(main())

332
ai_platform/modules/didi_brain/scripts/05_import_wikipedia_seed.py Normal file
View file

@ -0,0 +1,332 @@
"""Import a curated seed of Wikipedia articles about vaccines into the brain.
This is the FIRST real ingestion script. Goal: get ~20-30 high-quality
articles in EN and RO about vaccines / vaccine misinformation into Atomic,
properly tagged, so we can run real Didi-style queries against the brain.
Why a curated seed and not category-crawl?
- Quality > quantity for first validation
- Avoids legal gray areas of mass scraping
- Wikipedia categories are noisy (stub pages, redirects, lists)
- 20-30 well-chosen articles cover all the disinfo claims we want to test
Pipeline per article:
1. MediaWiki action API: action=query&prop=extracts (plain text, full)
2. Build clean markdown: "# Title\n\n{extract}"
3. Dedup by source_url against existing atoms
4. POST to Atomic with explicit taxonomy tags
5. (Atomic embeds it in background; we don't wait)
Idempotent: re-running skips articles already present (by canonical URL).
"""
from __future__ import annotations
import asyncio
import sys
import time
from dataclasses import dataclass
from pathlib import Path
import httpx
sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
from rich.console import Console # noqa: E402
from rich.table import Table # noqa: E402
from shared.atomic_api import AtomicApiError, AtomicClient # noqa: E402
from shared.config import settings # noqa: E402
from shared.logging import setup_logging # noqa: E402
from shared.taxonomy import TagResolver # noqa: E402
console = Console()
# Per-language seed lists. Keys are MediaWiki page titles. Order = priority.
# These are chosen to span the full disinfo landscape on vaccines:
# - factual baseline (Vaccine, Vaccination)
# - the canonical false claim (MMR vaccine and autism)
# - the perpetrator (Andrew Wakefield)
# - movements (Anti-vaccinationism, Vaccine hesitancy)
# - COVID-era specifics (Pfizer-BioNTech vaccine, COVID-19 vaccine misinformation)
# - relevant figures and incidents
SEED_EN: list[str] = [
"Vaccine",
"Vaccination",
"Vaccine hesitancy",
"MMR vaccine and autism",
"Andrew Wakefield",
"Anti-vaccinationism",
"Vaccine controversies",
"Vaccine injury",
"Pfizer–BioNTech COVID-19 vaccine",
"Moderna COVID-19 vaccine",
"COVID-19 vaccine misinformation",
"Robert F. Kennedy Jr.",
"Plandemic",
"Children's Health Defense",
]
SEED_RO: list[str] = [
"Vaccin",
"Vaccinare",
"Mișcarea antivaccinare",
"Pandemia de COVID-19 în România",
"Vaccin împotriva COVID-19",
"Vaccinare împotriva COVID-19 în România",
"Andrew Wakefield",
"Vaccin ROR",
"Tiomersal",
"Variolă",
]
# Wikimedia rejects vague User-Agents and Mozilla-like impersonations.
# Compliant format per their policy:
# https://meta.wikimedia.org/wiki/User-Agent_policy
# "AppName/Version (URL or email contact) optional-libraries"
USER_AGENT = (
"DidiBrain/0.1 (https://github.com/didibrain; didibrain@local.test) httpx/0.28"
)
RATE_LIMIT_SECONDS = 0.5 # be a good Wikipedia citizen
@dataclass(slots=True)
class WikiArticle:
title: str
extract: str
canonical_url: str
page_id: int
language: str # "EN" / "RO"
@dataclass(slots=True)
class ImportStats:
fetched: int = 0
skipped_missing: int = 0
skipped_duplicate: int = 0
created: int = 0
errors: int = 0
# ============================================================ wikipedia client
async def fetch_article(
http: httpx.AsyncClient, lang: str, title: str
) -> WikiArticle | None:
"""Fetch one Wikipedia page via the MediaWiki action API.
Returns None if the page doesn't exist (missing) or has no extract.
"""
base = f"https://{lang.lower()}.wikipedia.org/w/api.php"
params = {
"action": "query",
"format": "json",
"prop": "extracts|info",
"explaintext": "1",
"exsectionformat": "plain",
"exlimit": "1",
"inprop": "url",
"redirects": "1",
"titles": title,
"formatversion": "2",
}
try:
resp = await http.get(base, params=params)
resp.raise_for_status()
except httpx.HTTPError as e:
console.print(f"[red]✗[/red] HTTP {lang}/{title}: {e}")
return None
data = resp.json()
pages = (data.get("query") or {}).get("pages") or []
if not pages:
return None
page = pages[0]
if page.get("missing"):
return None
extract = (page.get("extract") or "").strip()
if not extract or len(extract) < 200:
# too short to be useful
return None
return WikiArticle(
title=page.get("title", title),
extract=extract,
canonical_url=page.get("canonicalurl") or page.get("fullurl") or "",
page_id=int(page.get("pageid", 0)),
language=lang.upper(),
)
def article_to_markdown(article: WikiArticle) -> str:
"""Convert a Wikipedia plain-text extract to Atomic-friendly markdown.
The action API extract uses plain text section breaks like:
Title
First paragraph.
Section name
Section content.
We can't reliably distinguish section headers from short paragraphs from
the plain text alone. So we just preserve the structure with the page
title as a single H1, then the body verbatim. Atomic's chunker is smart
enough to chunk on paragraph boundaries; we don't need section markers.
"""
return f"# {article.title}\n\n{article.extract}\n"
# ============================================================== atomic push
def tag_ids_for_language(resolver: TagResolver, language: str) -> list[str]:
"""Return the canonical tag-id list for a Wikipedia article in a given lang."""
return resolver.ids_for(
[
"Topics/Health/Vaccines",
"SourceType/Wikipedia",
"Credibility/Tier2",
f"Language/{language}",
"Type/Document",
"Country/Global",
]
)
async def import_one(
client: AtomicClient,
http: httpx.AsyncClient,
resolver: TagResolver,
lang: str,
title: str,
stats: ImportStats,
) -> None:
article = await fetch_article(http, lang, title)
if not article:
stats.skipped_missing += 1
console.print(f" [dim]·[/dim] {lang}/{title}: missing/empty")
return
stats.fetched += 1
# Dedup
existing = await client.get_atom_by_source_url(article.canonical_url)
if existing:
stats.skipped_duplicate += 1
console.print(
f" [dim]·[/dim] {lang}/{article.title}: already in brain"
)
return
# Push
md = article_to_markdown(article)
tag_ids = tag_ids_for_language(resolver, article.language)
try:
atom = await client.create_atom(
content=md,
source_url=article.canonical_url,
tag_ids=tag_ids,
)
stats.created += 1
size_kb = len(md) / 1024
console.print(
f" [green]+[/green] {lang}/{article.title} "
f"[dim]({size_kb:.1f} KB → {atom.get('id', '?')[:8]}...)[/dim]"
)
except AtomicApiError as e:
stats.errors += 1
console.print(
f" [red]✗[/red] {lang}/{article.title}: {e.status} {e.body[:200]}"
)
# ================================================================== main flow
async def import_seeds(
client: AtomicClient, resolver: TagResolver
) -> ImportStats:
stats = ImportStats()
headers = {"User-Agent": USER_AGENT, "Accept": "application/json"}
async with httpx.AsyncClient(timeout=30.0, headers=headers) as http:
for lang, seeds in (("EN", SEED_EN), ("RO", SEED_RO)):
console.print(f"\n[bold]{lang}[/bold] — {len(seeds)} seed articles")
for title in seeds:
await import_one(client, http, resolver, lang, title, stats)
await asyncio.sleep(RATE_LIMIT_SECONDS)
return stats
async def main_async() -> int:
setup_logging()
if not settings.atomic_token:
console.print(
"[red]ATOMIC_TOKEN missing — run 02_bootstrap_atomic.py first[/red]"
)
return 2
resolver = TagResolver()
if not resolver.all:
console.print(
"[red]tag id cache empty — run 04_seed_taxonomy.py first[/red]"
)
return 3
# Sanity: required tags exist
required = [
"Topics/Health/Vaccines",
"SourceType/Wikipedia",
"Credibility/Tier2",
"Language/EN",
"Language/RO",
"Type/Document",
"Country/Global",
]
missing = [p for p in required if p not in resolver]
if missing:
console.print(f"[red]missing required tags: {missing}[/red]")
return 4
console.print(
f"[dim]Atomic: {settings.atomic_url} "
f"| taxonomy: {len(resolver.all)} tags loaded[/dim]"
)
started = time.perf_counter()
async with AtomicClient() as client:
stats = await import_seeds(client, resolver)
elapsed = time.perf_counter() - started
table = Table(title=f"Wikipedia seed import (took {elapsed:.1f}s)")
table.add_column("Status", style="bold")
table.add_column("Count", justify="right")
table.add_row("[green]created[/green]", str(stats.created))
table.add_row("[dim]duplicate (skipped)[/dim]", str(stats.skipped_duplicate))
table.add_row("[dim]missing on Wikipedia[/dim]", str(stats.skipped_missing))
table.add_row("[red]errors[/red]", str(stats.errors))
table.add_row("fetched total", str(stats.fetched))
console.print(table)
if stats.errors:
return 1
if stats.created == 0 and stats.skipped_duplicate == 0:
console.print("\n[yellow]nothing imported — Wikipedia returned nothing[/yellow]")
return 5
console.print(
f"\n[bold green]ok[/bold green] — brain now has {stats.created} new "
f"atom(s) (embedding processes async, check status with sanity)"
)
return 0
def main() -> int:
try:
return asyncio.run(main_async())
except KeyboardInterrupt:
return 130
if __name__ == "__main__":
raise SystemExit(main())

175
ai_platform/modules/didi_brain/scripts/06_validate_queries.py Normal file
View file

@ -0,0 +1,175 @@
"""Run real Didi-style queries against the brain and show retrieval quality.
This is the FIRST end-to-end validation on real corpus content. It runs
three claim-style queries (mix of EN and RO) and shows:
- Atomic semantic search (top 10 hits with similarity)
- BGE reranker rescoring (top 5 with cross-encoder scores)
- Title + URL of each hit so you can eyeball relevance
This is how Didi's retrieval layer will work in production: first-stage
embedding retrieval, then cross-encoder rerank for precision.
"""
from __future__ import annotations
import asyncio
import sys
from dataclasses import dataclass
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
from rich.console import Console # noqa: E402
from rich.table import Table # noqa: E402
from shared.atomic_api import AtomicClient, SearchHit # noqa: E402
from shared.config import settings # noqa: E402
from shared.embedding_client import EmbeddingClient # noqa: E402
from shared.logging import setup_logging # noqa: E402
console = Console()
@dataclass(slots=True, frozen=True)
class QuerySpec:
label: str
query: str
expect_titles: list[str] # substrings we hope to see at the top
QUERIES: list[QuerySpec] = [
QuerySpec(
label="EN claim: MMR autism",
query="Does the MMR vaccine cause autism in children?",
expect_titles=["MMR", "autism", "Wakefield"],
),
QuerySpec(
label="RO query → expects EN+RO hits",
query="Cine este Andrew Wakefield si ce a facut cu studiul despre vaccinul MMR?",
expect_titles=["Wakefield", "MMR"],
),
QuerySpec(
label="RO claim: COVID vax danger",
query="Vaccinurile COVID-19 sunt periculoase pentru tineri si cauzeaza miocardita?",
expect_titles=["Pfizer", "COVID", "BioNTech", "misinformation"],
),
QuerySpec(
label="EN claim: RFK Jr disinfo",
query="Robert F Kennedy Jr anti-vaccine claims",
expect_titles=["Kennedy", "Children's Health"],
),
]
async def query_with_rerank(
atomic: AtomicClient,
embed: EmbeddingClient,
spec: QuerySpec,
) -> None:
console.print(f"\n[bold cyan]── {spec.label} ──[/bold cyan]")
console.print(f"[dim]query:[/dim] {spec.query}")
# Stage 1: semantic search via Atomic (which uses BGE-M3 internally)
hits = await atomic.search(spec.query, mode="semantic", limit=20, threshold=0.2)
if not hits:
console.print("[yellow]no hits[/yellow]")
return
# Stage 2: rerank with BGE-reranker-v2-m3 cross-encoder
docs = [_doc_for_rerank(h) for h in hits]
rerank_results = await embed.rerank(spec.query, docs, top_n=5)
# Build display table
table = Table(show_lines=False, expand=False)
table.add_column("#", justify="right", width=3)
table.add_column("emb_sim", justify="right", width=8)
table.add_column("rerank", justify="right", width=10)
table.add_column("source")
table.add_column("preview", overflow="fold")
# Map back: rerank result.index → original hit
rank_position: dict[int, int] = {r.index: i for i, r in enumerate(rerank_results)}
for i, hit in enumerate(hits[:10]):
rerank_idx = rank_position.get(i)
rerank_str = (
f"#{rerank_idx + 1}: {rerank_results[rerank_idx].score:.3f}"
if rerank_idx is not None
else "[dim]-[/dim]"
)
title = _extract_title(hit)
preview = (hit.matching_chunk_content or hit.snippet or "")[:120]
table.add_row(
str(i + 1),
f"{hit.similarity:.3f}",
rerank_str,
title,
preview.replace("\n", " "),
)
console.print(table)
# Did we get what we expected?
matched = [
e
for e in spec.expect_titles
if any(e.lower() in _extract_title(h).lower() for h in hits[:5])
]
if len(matched) == len(spec.expect_titles):
console.print(
f"[green]✓ all expected hit substrings present in top 5: {matched}[/green]"
)
else:
missing = [e for e in spec.expect_titles if e not in matched]
console.print(
f"[yellow]partial: matched {matched} missing {missing}[/yellow]"
)
def _doc_for_rerank(hit: SearchHit) -> str:
"""Build the text payload to send to the reranker for one hit."""
title = _extract_title(hit)
body = hit.matching_chunk_content or hit.snippet or ""
return f"{title}\n\n{body}"[:2000] # cap to keep reranker fast
def _extract_title(hit: SearchHit) -> str:
"""Best-effort title from source URL or chunk content."""
if hit.source_url:
# https://en.wikipedia.org/wiki/Andrew_Wakefield → Andrew Wakefield
from urllib.parse import unquote
last = hit.source_url.rstrip("/").rsplit("/", 1)[-1]
return unquote(last).replace("_", " ")
return (hit.matching_chunk_content or "")[:60]
async def main_async() -> int:
setup_logging()
if not settings.atomic_token:
console.print("[red]ATOMIC_TOKEN missing[/red]")
return 2
console.print(
f"[dim]Atomic: {settings.atomic_url} | Reranker: {settings.reranker_url}[/dim]"
)
async with AtomicClient() as atomic, EmbeddingClient() as embed:
for spec in QUERIES:
try:
await query_with_rerank(atomic, embed, spec)
except Exception as e: # noqa: BLE001
console.print(f"[red]× error on {spec.label}: {e}[/red]")
console.print("\n[bold]done[/bold]")
return 0
def main() -> int:
try:
return asyncio.run(main_async())
except KeyboardInterrupt:
return 130
if __name__ == "__main__":
raise SystemExit(main())

94
ai_platform/modules/didi_brain/scripts/07_run_extraction.py Normal file
View file

@ -0,0 +1,94 @@
"""Run claim extraction over all Type/Document atoms in the brain.
Reads from extractor.batch.run_batch() and renders a summary table.
Idempotent: documents already processed at the current prompt version are
skipped automatically (state file at extractor/_extracted.json).
Usage:
python scripts/07_run_extraction.py # process everything new
python scripts/07_run_extraction.py --limit 5 # cap docs (smoke test)
"""
from __future__ import annotations
import argparse
import asyncio
import sys
import time
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
from rich.console import Console # noqa: E402
from rich.table import Table # noqa: E402
from extractor.batch import run_batch # noqa: E402
from shared.config import settings # noqa: E402
from shared.logging import setup_logging # noqa: E402
console = Console()
def parse_args() -> argparse.Namespace:
p = argparse.ArgumentParser(description="Run claim extraction batch")
p.add_argument("--limit", type=int, default=None, help="cap number of docs")
return p.parse_args()
async def main_async(args: argparse.Namespace) -> int:
setup_logging()
console.print(
f"[dim]Atomic: {settings.atomic_url} "
f"| LLM router: {settings.llm_router_url} "
f"| Reasoning model: {settings.model_reasoning}[/dim]\n"
)
started = time.perf_counter()
try:
stats = await run_batch(limit=args.limit)
except Exception as e: # noqa: BLE001
console.print(f"[red]batch failed:[/red] {e}")
return 1
elapsed = time.perf_counter() - started
table = Table(title=f"Claim extraction (took {elapsed:.1f}s)", show_lines=False)
table.add_column("Metric", style="bold")
table.add_column("Count", justify="right")
table.add_row("documents seen", str(stats.docs_seen))
table.add_row("[dim]skipped (already done)[/dim]", str(stats.docs_skipped_already_done))
table.add_row("[green]processed[/green]", str(stats.docs_processed))
table.add_row("[red]failed[/red]", str(stats.docs_failed))
table.add_row("", "")
table.add_row("LLM raw claims", str(stats.claims_raw))
table.add_row("[green]valid claims[/green]", str(stats.claims_valid))
table.add_row("[green]created in brain[/green]", str(stats.claims_created))
table.add_row("[dim]duplicate (skipped)[/dim]", str(stats.claims_duplicate))
table.add_row("[red]push errors[/red]", str(stats.claims_error))
console.print(table)
if stats.rejected_reasons:
rt = Table(title="Validation rejections", show_lines=False)
rt.add_column("Reason")
rt.add_column("Count", justify="right")
for k, v in sorted(stats.rejected_reasons.items(), key=lambda x: -x[1]):
rt.add_row(k, str(v))
console.print(rt)
if stats.docs_failed > 0 and stats.docs_processed == 0:
return 2
if stats.docs_processed == 0 and stats.docs_skipped_already_done == 0:
console.print("[yellow]no documents found to process[/yellow]")
return 3
console.print("\n[bold green]extraction complete[/bold green]")
return 0
def main() -> int:
try:
return asyncio.run(main_async(parse_args()))
except KeyboardInterrupt:
return 130
if __name__ == "__main__":
raise SystemExit(main())

248
ai_platform/modules/didi_brain/scripts/08_validate_claims.py Normal file
View file

@ -0,0 +1,248 @@
"""Demonstrate claim-level retrieval — the payoff of script 07.
For each test claim, this script:
1. Queries Atomic semantic search (no filter — gets BOTH docs and claims)
2. Splits results into Type/Document hits and Type/Claim hits
3. For the claim hits: shows the actual claim text, stance, and source language
4. Reranks the claim hits with BGE-reranker-v2-m3 for precision
5. Aggregates by stance (asserts/reports/refutes) and language
This is the verdict-packet primitive that Didi will call. After D, the brain
returns ATOMIC FACTUAL CLAIMS, not paragraphs of articles to read.
"""
from __future__ import annotations
import asyncio
import re
import sys
from collections import Counter
from dataclasses import dataclass
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
from rich.console import Console # noqa: E402
from rich.table import Table # noqa: E402
from shared.atomic_api import AtomicClient, SearchHit # noqa: E402
from shared.config import settings # noqa: E402
from shared.embedding_client import EmbeddingClient # noqa: E402
from shared.logging import setup_logging # noqa: E402
console = Console()
@dataclass(slots=True, frozen=True)
class ClaimQuery:
label: str
query: str
QUERIES: list[ClaimQuery] = [
ClaimQuery(
label="EN — Wakefield fraud",
query="Andrew Wakefield falsified medical records in his 1998 study",
),
ClaimQuery(
label="RO — vaccinurile cauzeaza autism",
query="vaccinurile pediatrice cauzeaza autism la copii",
),
ClaimQuery(
label="EN — RFK Jr COVID lies",
query="Robert F Kennedy Jr promoted COVID-19 vaccine misinformation",
),
ClaimQuery(
label="RO — Pfizer myocarditis",
query="vaccinul Pfizer cauzeaza miocardita la tineri",
),
]
# ============================================================ rendering
# Match the canonical claim text from a Type/Claim atom's markdown body
_CLAIM_BODY_RE = re.compile(
r"^# Claim\s*\n+(.+?)\n+##",
re.DOTALL | re.MULTILINE,
)
# Match the stance line in the markdown body
_STANCE_RE = re.compile(r"Stance in source:\s*(\w+)")
def parse_claim_atom(content: str) -> tuple[str, str | None]:
"""Pull (claim_text, stance) out of a claim atom's markdown body."""
m = _CLAIM_BODY_RE.search(content)
claim_text = m.group(1).strip() if m else content[:200]
s = _STANCE_RE.search(content)
stance = s.group(1) if s else None
return claim_text, stance
def is_claim_hit(hit: SearchHit) -> bool:
"""A hit is a claim if it has the Type/Claim tag."""
return any(t.get("name") == "Claim" for t in hit.tags)
def language_of(hit: SearchHit) -> str | None:
for t in hit.tags:
if t.get("name") in {"RO", "EN", "RU", "UA", "FR", "DE"}:
return t["name"]
return None
def credibility_of(hit: SearchHit) -> str | None:
for t in hit.tags:
n = t.get("name", "")
if n.startswith("Tier") or n in {"StateAffiliated", "KnownDisinfo"}:
return n
return None
def parent_url_of(hit: SearchHit) -> str:
if not hit.source_url:
return ""
return hit.source_url.split("#", 1)[0]
# ====================================================== orchestration per query
async def evaluate_query(
spec: ClaimQuery,
atomic: AtomicClient,
embed: EmbeddingClient,
) -> None:
console.print(f"\n[bold cyan]── {spec.label} ──[/bold cyan]")
console.print(f"[dim]query:[/dim] {spec.query}\n")
# Pull a wide net of semantic hits — we'll split docs vs claims after
raw_hits = await atomic.search(spec.query, mode="semantic", limit=50, threshold=0.2)
if not raw_hits:
console.print("[yellow]no hits[/yellow]")
return
# Fetch full content for the claim hits so we can extract claim text + stance
claim_hits = [h for h in raw_hits if is_claim_hit(h)]
doc_hits = [h for h in raw_hits if not is_claim_hit(h)]
# Resolve full atom content for top-15 claim hits in one async batch
top_claim_hits = claim_hits[:15]
full_atoms = await asyncio.gather(
*(atomic.get_atom(h.atom_id) for h in top_claim_hits)
)
# Build (claim_text, stance, hit, parent_url) tuples
parsed: list[tuple[SearchHit, str, str | None, str]] = []
for hit, full in zip(top_claim_hits, full_atoms, strict=True):
text, stance = parse_claim_atom(full.get("content") or "")
parent = parent_url_of(hit)
parsed.append((hit, text, stance, parent))
if not parsed:
console.print("[yellow]no claim hits — only documents matched[/yellow]")
_render_doc_table(doc_hits[:5])
return
# Rerank the claims with cross-encoder for precision
docs_for_rerank = [text for _, text, _, _ in parsed]
reranked = await embed.rerank(spec.query, docs_for_rerank, top_n=10)
rank_position = {r.index: (i, r.score) for i, r in enumerate(reranked)}
# Pretty print top claims
table = Table(show_lines=False)
table.add_column("#", width=3, justify="right")
table.add_column("emb", width=6, justify="right")
table.add_column("rerank", width=8, justify="right")
table.add_column("stance", width=10)
table.add_column("lang", width=4)
table.add_column("claim", overflow="fold")
# Show top 8 by reranker order
sorted_by_rerank = sorted(
enumerate(parsed),
key=lambda x: rank_position.get(x[0], (999, 0))[0],
)
for display_i, (orig_idx, (hit, text, stance, parent)) in enumerate(sorted_by_rerank[:8], 1):
rerank_info = rank_position.get(orig_idx)
rerank_str = f"{rerank_info[1]:.3f}" if rerank_info else "-"
lang = language_of(hit) or "?"
stance_color = {
"ASSERTS": "green",
"REPORTS": "blue",
"REFUTES": "red",
"QUESTIONS": "yellow",
"NEUTRAL": "dim",
}.get(stance or "", "dim")
table.add_row(
str(display_i),
f"{hit.similarity:.3f}",
rerank_str,
f"[{stance_color}]{stance or '?'}[/{stance_color}]",
lang,
text[:200],
)
console.print(table)
# Aggregations: stance, language, parent doc count, credibility
stance_counts = Counter(stance for _, _, stance, _ in parsed if stance)
lang_counts = Counter(language_of(h) for h, _, _, _ in parsed if language_of(h))
parent_docs = {parent for _, _, _, parent in parsed if parent}
summary = Table(show_header=False, show_lines=False, padding=(0, 2))
summary.add_column(style="bold dim")
summary.add_column()
summary.add_row("total claim hits", str(len(claim_hits)))
summary.add_row("distinct parent docs", str(len(parent_docs)))
summary.add_row("by stance (top 15)", " ".join(f"{k}={v}" for k, v in stance_counts.most_common()))
summary.add_row("by language (top 15)", " ".join(f"{k}={v}" for k, v in lang_counts.most_common()))
summary.add_row("doc-level matches also", str(len(doc_hits)))
console.print(summary)
def _render_doc_table(hits: list[SearchHit]) -> None:
if not hits:
return
t = Table(title="Document fallback", show_lines=False)
t.add_column("#", width=3, justify="right")
t.add_column("sim", width=6, justify="right")
t.add_column("doc")
t.add_column("preview", overflow="fold")
for i, h in enumerate(hits, 1):
from urllib.parse import unquote
slug = unquote((h.source_url or "").rsplit("/", 1)[-1])
t.add_row(str(i), f"{h.similarity:.3f}", slug, (h.matching_chunk_content or "")[:120])
console.print(t)
async def main_async() -> int:
setup_logging()
if not settings.atomic_token:
console.print("[red]ATOMIC_TOKEN missing[/red]")
return 2
console.print(
f"[dim]Atomic: {settings.atomic_url} | Reranker: {settings.reranker_url}[/dim]"
)
async with AtomicClient() as atomic, EmbeddingClient() as embed:
for spec in QUERIES:
try:
await evaluate_query(spec, atomic, embed)
except Exception as e: # noqa: BLE001
console.print(f"[red]× error on {spec.label}: {type(e).__name__}: {e}[/red]")
console.print("\n[bold]done[/bold]")
return 0
def main() -> int:
try:
return asyncio.run(main_async())
except KeyboardInterrupt:
return 130
if __name__ == "__main__":
raise SystemExit(main())

296
ai_platform/modules/didi_brain/scripts/09_brain_api_demo.py Normal file
View file

@ -0,0 +1,296 @@
"""End-to-end demo of the brain_api HTTP service.
Assumes the API is already running (`python -m brain_api.run` in another
terminal, or via the helper flag --spawn here).
Hits all five endpoints in sequence with realistic inputs and prints the
key fields so you can eyeball both schema compliance and retrieval quality.
"""
from __future__ import annotations
import argparse
import asyncio
import json
import subprocess
import sys
import time
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
import httpx # noqa: E402
from rich.console import Console # noqa: E402
from rich.panel import Panel # noqa: E402
from rich.table import Table # noqa: E402
from shared.logging import setup_logging # noqa: E402
console = Console()
BASE = "http://127.0.0.1:8090"
TIMEOUT = 120.0
GATHER_CLAIMS = [
"vaccinurile pediatrice cauzeaza autism la copii",
"Andrew Wakefield falsified medical records in his 1998 study",
"Robert F Kennedy Jr promoted COVID-19 vaccine misinformation",
"HPV vaccines cause infertility in teenage girls", # expected MISS / weak
]
SEARCH_QUERIES = [
"MMR vaccine autism controversy",
"Pfizer BioNTech myocarditis",
]
FETCH_URLS_KNOWN = [
"https://en.wikipedia.org/wiki/Andrew_Wakefield",
"https://en.wikipedia.org/wiki/Vaccine_hesitancy",
"https://en.wikipedia.org/wiki/MMR_vaccine_and_autism",
]
FETCH_URLS_UNKNOWN = [
"https://example.com/totally-not-in-brain",
]
async def wait_for_api(client: httpx.AsyncClient, timeout: float = 30.0) -> bool:
deadline = time.monotonic() + timeout
while time.monotonic() < deadline:
try:
r = await client.get(f"{BASE}/health")
if r.status_code == 200:
return True
except httpx.HTTPError:
pass
await asyncio.sleep(0.5)
return False
async def demo_gather(client: httpx.AsyncClient) -> None:
console.print(Panel.fit("[bold cyan]POST /v1/gather[/bold cyan]"))
for claim in GATHER_CLAIMS:
t0 = time.perf_counter()
r = await client.post(
f"{BASE}/v1/gather",
json={
"claim": claim,
"max_evidence": 8,
"include_full_text": False,
"summarize": True,
"score_relevance": True,
},
)
elapsed = (time.perf_counter() - t0) * 1000
if r.status_code != 200:
console.print(f"[red]HTTP {r.status_code}[/red]: {r.text[:200]}")
continue
data = r.json()
brain_meta = data.get("brain_meta") or {}
cache = brain_meta.get("cache_status", "?")
ev_count = data.get("total_evidence_items", 0)
sources = brain_meta.get("evidence_sources", 0)
color = {
"HIT": "green",
"PARTIAL": "yellow",
"MISS": "dim",
}.get(cache, "white")
console.print(
f"\n[bold]{claim}[/bold]\n"
f" [{color}]cache={cache}[/{color}] "
f"evidence={ev_count} "
f"sources={sources} "
f"time={elapsed:.0f}ms "
f"server_reported={data.get('execution_time_ms', 0):.0f}ms"
)
if not data.get("evidence"):
console.print(" [dim]· no evidence[/dim]")
continue
t = Table(show_header=True, header_style="bold", show_lines=False, padding=(0, 1))
t.add_column("#", width=3)
t.add_column("rel", width=6, justify="right")
t.add_column("cred", width=6, justify="right")
t.add_column("publisher", width=20)
t.add_column("summary (best claim)", overflow="fold")
for i, ev in enumerate(data["evidence"][:5], 1):
summary = (ev.get("summary") or ev.get("snippet") or "")[:180]
t.add_row(
str(i),
f"{ev.get('relevance_score', 0):.3f}",
f"{ev.get('credibility_score', 0):.2f}",
(ev.get("publisher") or "")[:20],
summary,
)
console.print(t)
# Show the stages timing
stages = data.get("stages") or []
stage_line = " ".join(
f"{s['stage']}={s.get('duration_ms', 0):.0f}ms"
for s in stages
)
console.print(f" [dim]stages: {stage_line}[/dim]")
async def demo_search(client: httpx.AsyncClient) -> None:
console.print(Panel.fit("[bold cyan]POST /v1/search[/bold cyan]"))
r = await client.post(
f"{BASE}/v1/search",
json={"queries": SEARCH_QUERIES, "max_results": 5},
)
if r.status_code != 200:
console.print(f"[red]HTTP {r.status_code}[/red]: {r.text[:200]}")
return
data = r.json()
console.print(
f"total={data['total_results']} "
f"queries_processed={data['queries_processed']} "
f"time={data['execution_time_ms']:.0f}ms"
)
t = Table(show_header=True)
t.add_column("rank", width=4)
t.add_column("query", width=20)
t.add_column("site", width=20)
t.add_column("title", overflow="fold")
for res in data.get("results", []):
t.add_row(
str(res.get("rank", 0)),
(res.get("query") or "")[:18],
(res.get("site") or "")[:18],
(res.get("title") or "")[:60],
)
console.print(t)
async def demo_fetch(client: httpx.AsyncClient) -> None:
console.print(Panel.fit("[bold cyan]POST /v1/fetch[/bold cyan]"))
urls = FETCH_URLS_KNOWN + FETCH_URLS_UNKNOWN
r = await client.post(f"{BASE}/v1/fetch", json={"urls": urls})
if r.status_code != 200:
console.print(f"[red]HTTP {r.status_code}[/red]: {r.text[:200]}")
return
data = r.json()
console.print(
f"total_fetched={data['total_fetched']} "
f"total_failed={data['total_failed']} "
f"time={data['execution_time_ms']:.0f}ms"
)
for p in data.get("pages", []):
console.print(
f" [green]HIT[/green] {p.get('title', '?')[:60]} "
f"({len(p.get('text') or '')} chars)"
)
for f in data.get("failed_urls", []):
console.print(f" [dim]MISS[/dim] {f['url']} ({f['error']})")
async def demo_image_search(client: httpx.AsyncClient) -> None:
console.print(Panel.fit("[bold cyan]POST /v1/image-search[/bold cyan]"))
r = await client.post(
f"{BASE}/v1/image-search",
json={"queries": ["Andrew Wakefield"], "max_results": 10},
)
console.print(
f"status={r.status_code} "
f"body={r.json() if r.status_code == 200 else r.text[:200]}"
)
async def demo_ingest(client: httpx.AsyncClient) -> None:
console.print(Panel.fit("[bold cyan]POST /v1/ingest (dry smoke)[/bold cyan]"))
body = {
"claim": "Sample unit-test claim for brain_api ingest smoke",
"default_tags": [
"Topics/Health/Vaccines",
"Language/EN",
],
"run_extraction": False,
"evidence": [
{
"url": "https://example.test/brain-ingest-smoke-1",
"title": "Brain ingest smoke test 1",
"publisher": "example.test",
"retrieved_at": "2026-04-11T00:00:00+00:00",
"full_text": (
"# Brain ingest smoke 1\n\n"
"This is a synthetic document created by the brain_api "
"demo script to verify /v1/ingest. It does not represent "
"any real factual claim."
),
"relevance_score": 0.75,
"credibility_score": 0.70,
}
],
}
r = await client.post(f"{BASE}/v1/ingest", json=body)
if r.status_code != 200:
console.print(f"[red]HTTP {r.status_code}[/red]: {r.text[:300]}")
return
data = r.json()
console.print(
f"accepted={data['accepted']} "
f"skipped_duplicate={data['skipped_duplicate']} "
f"errors={data['errors']} "
f"ids={data.get('created_atom_ids', [])}"
)
if data.get("warnings"):
for w in data["warnings"]:
console.print(f" [yellow]![/yellow] {w}")
async def main_async(args: argparse.Namespace) -> int:
setup_logging()
spawned: subprocess.Popen | None = None
if args.spawn:
console.print("[dim]spawning brain_api in background...[/dim]")
spawned = subprocess.Popen(
[sys.executable, "-m", "brain_api.run"],
cwd=str(Path(__file__).resolve().parent.parent),
stdout=subprocess.DEVNULL,
stderr=subprocess.DEVNULL,
)
try:
async with httpx.AsyncClient(timeout=TIMEOUT) as client:
if not await wait_for_api(client):
console.print("[red]brain_api did not come up within 30s[/red]")
return 1
console.print("[green]✓[/green] brain_api /health OK\n")
await demo_gather(client)
console.print()
await demo_search(client)
console.print()
await demo_fetch(client)
console.print()
await demo_image_search(client)
console.print()
if args.ingest:
await demo_ingest(client)
console.print("\n[bold green]demo complete[/bold green]")
return 0
finally:
if spawned is not None:
spawned.terminate()
def main() -> int:
p = argparse.ArgumentParser()
p.add_argument("--spawn", action="store_true", help="start uvicorn ourselves")
p.add_argument("--ingest", action="store_true", help="also exercise /v1/ingest (writes to brain)")
args = p.parse_args()
try:
return asyncio.run(main_async(args))
except KeyboardInterrupt:
return 130
if __name__ == "__main__":
raise SystemExit(main())

85
ai_platform/modules/didi_brain/scripts/10_run_lint.py Normal file
View file

@ -0,0 +1,85 @@
"""Run the Lint pass over the current claim atoms in the brain.
python scripts/10_run_lint.py # full corpus
python scripts/10_run_lint.py --limit 30 # smoke test
python scripts/10_run_lint.py --force # re-evaluate cached pairs
The state file at lint/_contradictions.json is always updated idempotently.
Interrupting the run with Ctrl-C will still save whatever has been classified
so far, and a subsequent run will pick up from there.
"""
from __future__ import annotations
import argparse
import asyncio
import sys
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
from rich.console import Console # noqa: E402
from lint._state import LintState # noqa: E402
from lint.reporter import ( # noqa: E402
render_contradictions,
render_equivalents,
render_stats,
)
from lint.runner import run_lint_pass # noqa: E402
from shared.config import settings # noqa: E402
from shared.logging import setup_logging # noqa: E402
console = Console()
def parse_args() -> argparse.Namespace:
p = argparse.ArgumentParser(description="Run cross-corpus contradiction detection")
p.add_argument("--limit", type=int, default=None, help="cap number of source claim atoms")
p.add_argument("--force", action="store_true", help="re-evaluate cached pairs")
p.add_argument(
"--show-equivalents",
action="store_true",
help="also print paraphrase clusters (not just contradictions)",
)
return p.parse_args()
async def main_async(args: argparse.Namespace) -> int:
setup_logging()
console.print(
f"[dim]Atomic: {settings.atomic_url} | "
f"LLM: {settings.llm_router_url} | "
f"model: {settings.model_reasoning}[/dim]\n"
)
try:
stats = await run_lint_pass(limit_atoms=args.limit, force=args.force)
except KeyboardInterrupt:
console.print("\n[yellow]interrupted — state file preserved[/yellow]")
return 130
except Exception as e: # noqa: BLE001
console.print(f"[red]lint failed:[/red] {type(e).__name__}: {e}")
return 1
console.print()
render_stats(stats)
state = LintState()
render_contradictions(state, top_n=10, min_confidence=0.7)
if args.show_equivalents:
render_equivalents(state, top_n=10, min_confidence=0.85)
console.print(f"\n[dim]ledger → {state.path}[/dim]")
return 0
def main() -> int:
try:
return asyncio.run(main_async(parse_args()))
except KeyboardInterrupt:
return 130
if __name__ == "__main__":
raise SystemExit(main())

61
ai_platform/modules/didi_brain/scripts/11_show_contradictions.py Normal file
View file

@ -0,0 +1,61 @@
"""Read the Lint pass state file and render detected contradictions.
Does NOT run any classification — it's a read-only view over whatever the
last run of scripts/10_run_lint.py produced.
python scripts/11_show_contradictions.py # top 20, >= 0.7 conf
python scripts/11_show_contradictions.py --min 0.85 # stricter
python scripts/11_show_contradictions.py --top 50 # wider net
python scripts/11_show_contradictions.py --equivalents # also paraphrases
"""
from __future__ import annotations
import argparse
import sys
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
from rich.console import Console # noqa: E402
from lint._state import LintState # noqa: E402
from lint.reporter import render_contradictions, render_equivalents # noqa: E402
console = Console()
def main() -> int:
p = argparse.ArgumentParser()
p.add_argument("--min", type=float, default=0.7, help="minimum confidence")
p.add_argument("--top", type=int, default=20, help="top N to display")
p.add_argument("--equivalents", action="store_true", help="also show paraphrases")
args = p.parse_args()
state = LintState()
if not state.all_verdicts:
console.print(
f"[yellow]no verdicts in state file at {state.path}[/yellow]\n"
"Run scripts/10_run_lint.py first."
)
return 2
total = len(state.all_verdicts)
contras_total = len(state.all_contradictions)
equivs_total = len(state.all_equivalents)
console.print(
f"[dim]state file: {state.path}[/dim]\n"
f"[dim]total pairs: {total} | "
f"contradictory: {contras_total} | "
f"equivalent: {equivs_total} | "
f"incomparable: {total - contras_total - equivs_total}[/dim]\n"
)
render_contradictions(state, top_n=args.top, min_confidence=args.min)
if args.equivalents:
render_equivalents(state, top_n=args.top, min_confidence=max(args.min, 0.85))
return 0
if __name__ == "__main__":
raise SystemExit(main())

0
ai_platform/modules/didi_brain/scripts/__init__.py Normal file
View file

301
ai_platform/modules/didi_brain/scripts/bootstrap_deploy.sh Normal file
View file

@ -0,0 +1,301 @@
#!/usr/bin/env bash
# =============================================================================
# DidiBrain — fresh-server bootstrap
# =============================================================================
#
# End-to-end deploy of the DidiBrain stack on a clean Linux server:
#
# 1. Preflight (docker, compose, python3, curl)
# 2. .env validation
# 3. docker compose build + up
# 4. Wait for all three containers to become healthy
# 5. Create a local Python venv for operator scripts
# 6. Install operator dependencies (httpx, pydantic, etc.)
# 7. Claim the Atomic instance + configure the BGE-M3 provider
# 8. Seed the canonical tag taxonomy
# 9. (Optional) Import the seed Wikipedia corpus
# 10. (Optional) Run the claim extraction batch
# 11. Final smoke test against /v1/gather
#
# Every step is idempotent — you can re-run this script after a crash or
# after editing .env, and it will only do what still needs doing. No step
# is destructive (no `down -v`, no volume deletions).
#
# Environment flags you can set before running:
#
# BRAIN_IMPORT_CORPUS 1 to import the Wikipedia seed (default 1)
# BRAIN_RUN_EXTRACTION 1 to run claim extraction after import (default 1)
# BRAIN_SKIP_VENV 1 to skip venv creation / reuse existing .venv
# BRAIN_SKIP_SANITY 1 to skip the final smoke test (save a couple sec)
#
# Usage:
# cd ~/didibrain
# cp .env.example .env # then edit LLM_ROUTER_URL / EMBEDDING_URL / ...
# ./scripts/bootstrap_deploy.sh
#
# =============================================================================
set -euo pipefail
# ----------------------------------------------------------------- paths
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
PROJECT_ROOT="$(cd "${SCRIPT_DIR}/.." && pwd)"
INFRA_DIR="${PROJECT_ROOT}/infra"
COMPOSE_FILE="${INFRA_DIR}/docker-compose.yml"
ENV_FILE="${PROJECT_ROOT}/.env"
ENV_EXAMPLE="${PROJECT_ROOT}/.env.example"
VENV_DIR="${PROJECT_ROOT}/.venv"
cd "${PROJECT_ROOT}"
# ----------------------------------------------------------------- colors
RED=$'\033[31m'
GREEN=$'\033[32m'
YELLOW=$'\033[33m'
BLUE=$'\033[34m'
DIM=$'\033[2m'
BOLD=$'\033[1m'
RESET=$'\033[0m'
step() { echo "${BLUE}${BOLD}== $* ==${RESET}"; }
info() { echo "${DIM} · $*${RESET}"; }
ok() { echo "${GREEN} ✓ $*${RESET}"; }
warn() { echo "${YELLOW} ! $*${RESET}"; }
fail() { echo "${RED}${BOLD} ✗ $*${RESET}" >&2; exit 1; }
# ------------------------------------------------------------- step 1 preflight
step "1. Preflight checks"
require_cmd() {
command -v "$1" >/dev/null 2>&1 || fail "required command not found: $1"
ok "$1 found"
}
require_cmd docker
require_cmd python3
require_cmd curl
# Docker Compose can be `docker compose` (v2) or `docker-compose` (v1). Prefer v2.
if docker compose version >/dev/null 2>&1; then
DC="docker compose"
ok "docker compose (v2) found"
elif command -v docker-compose >/dev/null 2>&1; then
DC="docker-compose"
warn "using legacy docker-compose (v1) — v2 recommended"
else
fail "neither 'docker compose' (v2) nor 'docker-compose' (v1) found"
fi
# Docker daemon reachable?
docker info >/dev/null 2>&1 || fail "docker daemon not reachable — is Docker running and your user in the docker group?"
ok "docker daemon reachable"
# ------------------------------------------------------------- step 2 .env
step "2. Environment file"
if [[ ! -f "${ENV_FILE}" ]]; then
if [[ -f "${ENV_EXAMPLE}" ]]; then
warn ".env missing — copying from .env.example"
warn "REVIEW IT AND FILL IN LLM_ROUTER_URL / EMBEDDING_URL / RERANKER_URL"
cp "${ENV_EXAMPLE}" "${ENV_FILE}"
fail ".env was just created from template. Edit it, then rerun this script."
else
fail "no .env and no .env.example in ${PROJECT_ROOT}"
fi
fi
ok ".env present at ${ENV_FILE}"
# Minimal sanity on required variables
check_env_var() {
local key="$1"
if ! grep -E "^${key}=" "${ENV_FILE}" >/dev/null 2>&1; then
fail "${key} is missing from .env"
fi
local value
value="$(grep -E "^${key}=" "${ENV_FILE}" | head -1 | cut -d= -f2-)"
if [[ -z "${value}" ]]; then
warn "${key} is empty in .env (may be filled by bootstrap — continuing)"
fi
}
check_env_var LLM_ROUTER_URL
check_env_var EMBEDDING_URL
check_env_var RERANKER_URL
check_env_var POSTGRES_USER
check_env_var POSTGRES_PASSWORD
ok ".env keys look structurally correct"
# ------------------------------------------------------ step 3 docker compose
step "3. Build and start containers"
info "building brain-api image (cached layers reused where possible)..."
${DC} -f "${COMPOSE_FILE}" --env-file "${ENV_FILE}" build brain-api
info "starting the full stack (postgres, atomic-server, brain-api)..."
${DC} -f "${COMPOSE_FILE}" --env-file "${ENV_FILE}" up -d
ok "containers started"
# ------------------------------------------------------ step 4 wait for healthy
step "4. Wait for all three containers to become healthy"
wait_healthy() {
local name="$1"
local deadline=$(( $(date +%s) + 180 ))
while (( $(date +%s) < deadline )); do
local status
status=$(docker inspect --format='{{.State.Health.Status}}' "${name}" 2>/dev/null || echo "missing")
case "${status}" in
healthy)
ok "${name} healthy"
return 0
;;
unhealthy)
fail "${name} reports unhealthy — check 'docker logs ${name}'"
;;
starting)
info "${name} starting..."
;;
missing)
info "${name} not yet visible to docker..."
;;
*)
info "${name} status: ${status}"
;;
esac
sleep 3
done
fail "${name} did not reach healthy within 180 seconds"
}
wait_healthy didibrain-postgres
wait_healthy didibrain-atomic
wait_healthy didibrain-api
# -------------------------------------------------------------- step 5 venv
step "5. Python operator venv"
if [[ "${BRAIN_SKIP_VENV:-0}" == "1" ]]; then
warn "BRAIN_SKIP_VENV=1 — skipping venv creation"
elif [[ -d "${VENV_DIR}" ]]; then
ok "venv already exists at ${VENV_DIR}"
else
info "creating venv at ${VENV_DIR}..."
python3 -m venv "${VENV_DIR}"
ok "venv created"
fi
if [[ "${BRAIN_SKIP_VENV:-0}" != "1" ]]; then
info "installing operator dependencies..."
"${VENV_DIR}/bin/pip" install --quiet --upgrade pip
"${VENV_DIR}/bin/pip" install --quiet \
"httpx>=0.28,<0.30" \
"pydantic>=2.12,<3.0" \
"pydantic-settings>=2.13,<3.0" \
"structlog>=25.5,<26.0" \
"python-dotenv>=1.2,<2.0" \
"tenacity>=9.1,<10.0" \
"rich>=14.3,<15.0"
ok "operator dependencies installed"
fi
PY="${VENV_DIR}/bin/python"
export PYTHONIOENCODING=utf-8
# -------------------------------------------------------------- step 6 bootstrap
step "6. Atomic bootstrap (claim instance + provider config)"
info "running scripts/02_bootstrap_atomic.py (idempotent)..."
"${PY}" "${PROJECT_ROOT}/scripts/02_bootstrap_atomic.py"
ok "atomic bootstrapped"
# -------------------------------------------------------------- step 7 taxonomy
step "7. Seed canonical taxonomy"
info "running scripts/04_seed_taxonomy.py (idempotent)..."
"${PY}" "${PROJECT_ROOT}/scripts/04_seed_taxonomy.py"
ok "taxonomy seeded"
# -------------------------------------------------------- step 8 corpus import
if [[ "${BRAIN_IMPORT_CORPUS:-1}" == "1" ]]; then
step "8. Import the Wikipedia seed corpus"
info "running scripts/05_import_wikipedia_seed.py..."
"${PY}" "${PROJECT_ROOT}/scripts/05_import_wikipedia_seed.py"
ok "seed corpus imported"
else
warn "BRAIN_IMPORT_CORPUS=0 — skipping Wikipedia seed import"
fi
# -------------------------------------------------------- step 9 extraction
if [[ "${BRAIN_RUN_EXTRACTION:-1}" == "1" && "${BRAIN_IMPORT_CORPUS:-1}" == "1" ]]; then
step "9. Run claim extraction (may take ~15-20 min for the seed)"
info "running scripts/07_run_extraction.py..."
"${PY}" "${PROJECT_ROOT}/scripts/07_run_extraction.py"
ok "claim extraction complete"
else
warn "skipping claim extraction"
fi
# ---------------------------------------------------------- step 10 smoke test
if [[ "${BRAIN_SKIP_SANITY:-0}" == "1" ]]; then
warn "BRAIN_SKIP_SANITY=1 — skipping final smoke test"
else
step "10. Final smoke test — /v1/gather against brain_api"
HEALTH_JSON="$(curl -fsS http://localhost:8090/health)"
info "brain_api /health → ${HEALTH_JSON}"
GATHER_JSON="$(curl -fsS -X POST http://localhost:8090/v1/gather \
-H 'Content-Type: application/json' \
-d '{"claim":"vaccines cause autism","max_evidence":3,"include_full_text":false,"run_nli":false}')"
CACHE_STATUS=$(echo "${GATHER_JSON}" | "${PY}" -c \
"import sys,json; print(json.load(sys.stdin).get('brain_meta',{}).get('cache_status','?'))")
ITEM_COUNT=$(echo "${GATHER_JSON}" | "${PY}" -c \
"import sys,json; print(json.load(sys.stdin).get('total_evidence_items',0))")
info "gather: cache_status=${CACHE_STATUS} evidence_items=${ITEM_COUNT}"
if [[ "${CACHE_STATUS}" == "HIT" ]]; then
ok "brain returned HIT with ${ITEM_COUNT} items — end-to-end working"
elif [[ "${CACHE_STATUS}" == "PARTIAL" ]]; then
ok "brain returned PARTIAL (${ITEM_COUNT} items) — end-to-end working, partial coverage"
elif [[ "${CACHE_STATUS}" == "MISS" ]]; then
warn "brain returned MISS — this is expected if the corpus was not imported"
warn "(re-run with BRAIN_IMPORT_CORPUS=1 BRAIN_RUN_EXTRACTION=1 to populate)"
else
fail "unexpected cache_status: ${CACHE_STATUS}"
fi
fi
# ---------------------------------------------------------------- done
echo
step "DONE"
echo
echo " brain_api: http://localhost:8090"
echo " Swagger UI: http://localhost:8090/docs"
echo " ReDoc: http://localhost:8090/redoc"
echo " OpenAPI spec: http://localhost:8090/openapi.json"
echo " atomic-server API: http://localhost:8088"
echo " atomic API docs: http://localhost:8088/api/docs"
echo " postgres: localhost:5434"
echo
echo "${DIM}Next steps:${RESET}"
echo " · Point Didi backend at http://<this-host>:8090/v1/gather"
echo " · Run ${BOLD}${PY} scripts/10_run_lint.py${RESET} overnight for the first contradiction audit"
echo " · When corpus needs to grow, extend scripts/05 seed lists or feed"
echo " web-module output back via ${BOLD}POST /v1/ingest${RESET}"
echo

1
ai_platform/modules/didi_brain/shared/__init__.py Normal file
View file

@ -0,0 +1 @@
"""Shared infrastructure: config, LLM client, embedding client, Atomic client."""

81
ai_platform/modules/didi_brain/shared/_tag_ids.json Normal file
View file

@ -0,0 +1,81 @@
{
"ClaimStatus": "a4bdf09e-52f2-4b85-8a56-e22c3b05f835",
"ClaimStatus/Confirmed": "fd25a0b8-bfb1-4cc5-a5c1-225070f49d5c",
"ClaimStatus/Debunked": "50b9e623-144c-4093-8a3c-0ded59d533e8",
"ClaimStatus/Disputed": "3f0e90a5-f64a-41a6-b430-e08bfb326d90",
"ClaimStatus/PartiallyTrue": "554a7f3f-63f3-4de5-bb26-83d9e5eb8495",
"ClaimStatus/Unverified": "fb5ecbf5-7237-4ed7-b694-2a871c1c3038",
"Country": "9b22cf38-0175-4cc4-952c-318ae735ecec",
"Country/France": "b7ae6946-b2ae-474d-906e-f25c2b5f08ce",
"Country/Germany": "986f37e0-be60-4e1c-b966-0b2273358f57",
"Country/Global": "ef8938f9-8866-4ebf-8412-1603b4220c5e",
"Country/Italy": "fb724b1f-7b66-4567-912a-bf7b17013921",
"Country/Moldova": "35e44ad8-31a2-43be-8fdc-149a1d48fc1c",
"Country/Poland": "57d3023b-165c-4d3c-bdc6-8203d938ab94",
"Country/Romania": "6e5657a5-47d8-4602-82ee-b9ef27e05b6f",
"Country/Russia": "b045c4cd-b346-4dba-815f-8bdb86ac5aff",
"Country/Spain": "9037debf-6108-4634-9955-6e3ab7803c24",
"Country/UK": "52200de8-841a-46f1-be12-39ee7d5df275",
"Country/USA": "a86f71dd-ab0c-4b26-8f0c-e6370f41ab14",
"Country/Ukraine": "f4d6b5ef-3114-43ad-9349-a2215765babf",
"Credibility": "e1d6214c-4e8d-4f55-81ff-c51130638108",
"Credibility/KnownDisinfo": "bbcbbacd-cbd8-4da8-b08e-1fa420bffe8a",
"Credibility/StateAffiliated": "75bdaecb-61e3-4f52-b459-fd9d35584a88",
"Credibility/Tier1": "a6fd68db-7d24-4155-b038-a354d7aa90a7",
"Credibility/Tier2": "ca4b2bad-7f9b-44f4-86e1-9fa6cc460807",
"Credibility/Tier3": "b180c0ed-0488-4098-8b36-5bc9eddb095e",
"Credibility/Unknown": "e3122bf0-de42-473e-acc4-06b29cfd58b0",
"Events": "acf1d4c9-6e7b-4dc0-8d4e-f0db324e038d",
"Language": "d7a7b7fa-d7e1-465d-95f4-fe242afa1da4",
"Language/DE": "5fbcb199-e5b2-409f-86a4-c267a64609e8",
"Language/EN": "1f901309-ed11-4235-aa7b-39953cbe74da",
"Language/ES": "209c6c0d-efac-481b-bb3f-69345c1f0f8d",
"Language/FR": "6e0ac309-9b3d-4ed0-b4f8-d26f89c7a940",
"Language/IT": "b9ddebbf-db73-4d08-a82c-90c726d2b61b",
"Language/PL": "8375f071-fa08-4b09-a135-74c288a8d293",
"Language/RO": "5b8e7fa9-42ae-40f0-ab02-8959e3703700",
"Language/RU": "69d1b480-2bac-447c-9cd0-845a6dc04f24",
"Language/UA": "8b976668-cd2f-4374-b251-dcf35bb40e92",
"Locations": "b6383171-7b06-4761-ae9d-95e2a6c4de31",
"Organizations": "1b0eca4b-1e21-425c-9813-07337ae4af83",
"People": "f8864b48-8ecc-482a-846f-6f2adfa67b82",
"SourceType": "de854162-f5e3-4430-8af1-77727344617b",
"SourceType/Blog": "bfdc7f83-e900-4972-b728-06a53fb4c40d",
"SourceType/FactCheck": "2a2c40cf-d4db-4a26-a347-2377ad11acaf",
"SourceType/Forum": "052fa147-ba51-4fde-89da-be9a88170a2d",
"SourceType/Government": "03dbd479-939d-4e8a-8d3f-8e9a54e9eb4b",
"SourceType/MainstreamMedia": "783ca275-165d-4669-ad09-2ee98d0f85ce",
"SourceType/ScientificJournal": "0b5d65a0-fcc5-46f6-8e29-9fb5564e728b",
"SourceType/SocialMedia": "46e6b7a6-d8fd-48f9-a5c2-db0fe6f149ec",
"SourceType/StateMedia": "3b95b904-c240-4841-9c59-5abe699f983d",
"SourceType/TabloidMedia": "1eff4e9c-e4f5-4439-b10c-d0feb189d25e",
"SourceType/Wikipedia": "ea109ab3-b709-4199-9833-47283b862ba0",
"Stance": "615ddf04-28eb-42a4-9ea1-8ce0c1c5f346",
"Stance/Asserts": "45f71063-e0dc-48ab-b80b-5b26cf68cc3d",
"Stance/Neutral": "e8d52343-0062-472c-ab4e-c5b473f267db",
"Stance/Questions": "d6eb0f8e-e633-41b1-ba4e-483451fe9e75",
"Stance/Refutes": "590afc4c-beff-4973-b8c3-c2f93fe5be13",
"Stance/Reports": "e09f46d5-2d84-4934-916a-9bdb9e1fbb1d",
"Topics": "8efc2a64-3ccb-4e7e-a100-a6d83de031f0",
"Topics/Climate": "047490e0-c848-4bc9-bedb-2a4e7c5d2f3d",
"Topics/Economy": "7bb131b0-99b8-4c3a-bf72-25c781f0b4f4",
"Topics/Health": "3c4151fa-8fcc-48d3-a367-277c204f23e2",
"Topics/Health/COVID": "a14e59fa-a7c3-4cec-b35a-67e1455335bf",
"Topics/Health/Disease": "4187f778-a1cf-4baa-9686-ce48e556f841",
"Topics/Health/Medicine": "fd011a07-e373-4a85-ab2a-ec6da89945b7",
"Topics/Health/PublicHealth": "773c7f9d-b9e8-478f-821e-6daf59e53e9a",
"Topics/Health/Vaccines": "3e0c54b9-1554-4b2e-8030-15417c47c04d",
"Topics/Politics": "f714e9ce-78d6-4739-be5f-b273a7ab7445",
"Topics/Politics/Diplomacy": "0d946f6e-0b9a-4dc2-8244-cdb7fbf741e6",
"Topics/Politics/Elections": "e3d4764b-49cb-47bb-ae5a-24a75f6ea845",
"Topics/Politics/Government": "da76c072-682f-4e1e-b481-2f805bdb8d8a",
"Topics/Politics/War": "f3dd2d86-054d-4cc2-9757-0337af2e90c5",
"Topics/Society": "af362549-35e0-4a3e-9b53-485f46376a52",
"Topics/Technology": "5d2b4444-2b4f-4494-a673-0904bbf801f2",
"Type": "656dac2b-c77c-4e21-b779-78a029b797f7",
"Type/Annotation": "af20bc10-8142-44ad-a826-b9214755fe8d",
"Type/Claim": "a56f9dce-0ba5-4504-9ff7-38ee005da084",
"Type/Document": "c80cbebd-4fae-4395-8992-b510335872ab",
"Type/Quote": "f4fdae65-293e-4d9a-8336-9b8838894f4d",
"Type/Summary": "6899e2ed-62f7-4336-9c60-2d1a8d1e0418"
}

307
ai_platform/modules/didi_brain/shared/atomic_api.py Normal file
View file

@ -0,0 +1,307 @@
"""Typed async client for the Atomic REST API.
Atomic is the brain — this client is how every other DidiBrain component
(bootstrap, scraper, extractor, lint, didi_client) talks to it.
Design notes:
- One AtomicClient instance per long-running process. Reusable httpx.AsyncClient.
- Auth: Bearer token loaded from `settings.atomic_token`. If empty, public
endpoints (/health, setup) still work.
- Errors: any non-2xx is wrapped in AtomicApiError with the response body.
- Pagination: helpers like `iter_atoms()` (added later) yield pages.
Only the endpoints we actually need are wrapped here. Add more as required;
do not pre-emptively wrap the full ~78-route surface.
"""
from __future__ import annotations
from collections.abc import AsyncIterator, Sequence
from contextlib import asynccontextmanager
from dataclasses import dataclass
from typing import Any
import httpx
from tenacity import (
AsyncRetrying,
retry_if_exception_type,
stop_after_attempt,
wait_exponential,
)
from shared.config import settings
from shared.logging import get_logger
log = get_logger(__name__)
class AtomicApiError(Exception):
"""Raised on any non-2xx response from atomic-server."""
def __init__(self, message: str, *, status: int, body: str = "", url: str = ""):
super().__init__(message)
self.status = status
self.body = body
self.url = url
@dataclass(slots=True)
class AtomSummary:
"""Minimal atom view returned by list/search endpoints."""
id: str
content: str
source_url: str | None
embedding_status: str
tagging_status: str
created_at: str
updated_at: str
tags: list[dict[str, Any]]
@classmethod
def from_dict(cls, d: dict[str, Any]) -> AtomSummary:
return cls(
id=d["id"],
content=d.get("content", ""),
source_url=d.get("source_url"),
embedding_status=d.get("embedding_status", "unknown"),
tagging_status=d.get("tagging_status", "unknown"),
created_at=d.get("created_at", ""),
updated_at=d.get("updated_at", ""),
tags=d.get("tags", []),
)
@dataclass(slots=True)
class SearchHit:
"""One result from POST /api/search."""
atom_id: str
similarity: float
matching_chunk_content: str | None
snippet: str | None
source_url: str | None
tags: list[dict[str, Any]]
@classmethod
def from_dict(cls, d: dict[str, Any]) -> SearchHit:
return cls(
atom_id=d.get("id") or d.get("atom_id", ""),
similarity=float(d.get("similarity_score") or d.get("similarity", 0.0)),
matching_chunk_content=d.get("matching_chunk_content"),
snippet=d.get("snippet"),
source_url=d.get("source_url"),
tags=d.get("tags", []),
)
class AtomicClient:
"""Async REST client for atomic-server. Use as `async with AtomicClient() as a:`."""
def __init__(
self,
*,
base_url: str | None = None,
token: str | None = None,
timeout: float = 60.0,
max_attempts: int = 3,
):
self._base_url = (base_url or settings.atomic_url).rstrip("/")
self._token = token if token is not None else settings.atomic_token
self._max_attempts = max_attempts
headers = {"Content-Type": "application/json"}
if self._token:
headers["Authorization"] = f"Bearer {self._token}"
self._http = httpx.AsyncClient(
base_url=self._base_url,
headers=headers,
timeout=httpx.Timeout(timeout, connect=10.0),
)
async def aclose(self) -> None:
await self._http.aclose()
async def __aenter__(self) -> AtomicClient:
return self
async def __aexit__(self, *args: Any) -> None:
await self.aclose()
@property
def base_url(self) -> str:
return self._base_url
@property
def has_token(self) -> bool:
return bool(self._token)
# ---------------------------------------------------------------- internals
async def _request(
self,
method: str,
path: str,
*,
json: Any = None,
params: dict[str, Any] | None = None,
retry_on_5xx: bool = True,
) -> Any:
async for attempt in AsyncRetrying(
stop=stop_after_attempt(self._max_attempts),
wait=wait_exponential(multiplier=1, min=1, max=10),
retry=retry_if_exception_type((httpx.HTTPError, httpx.TimeoutException)),
reraise=True,
):
with attempt:
resp = await self._http.request(method, path, json=json, params=params)
if resp.status_code >= 500 and retry_on_5xx:
raise httpx.HTTPError(f"5xx from {path}: {resp.status_code}")
if resp.status_code >= 400:
raise AtomicApiError(
f"HTTP {resp.status_code} {method} {path}",
status=resp.status_code,
body=resp.text[:1000],
url=str(resp.url),
)
if resp.status_code == 204 or not resp.content:
return None
try:
return resp.json()
except ValueError:
return resp.text
raise AtomicApiError("retry loop exited without result", status=0) # unreachable
# ----------------------------------------------------------------- health
async def health(self) -> dict[str, Any]:
return await self._request("GET", "/health")
async def setup_status(self) -> dict[str, Any]:
"""Whether the instance still needs initial token claim."""
return await self._request("GET", "/api/setup/status")
# ----------------------------------------------------------------- settings
async def get_settings(self) -> dict[str, Any]:
return await self._request("GET", "/api/settings")
async def set_setting(self, key: str, value: str) -> Any:
"""Set a single setting. Use set_settings() for bulk."""
return await self._request(
"PUT", f"/api/settings/{key}", json={"value": value}
)
async def set_settings(self, items: dict[str, str]) -> dict[str, Any]:
"""Apply many settings sequentially. Returns map of key → response."""
out: dict[str, Any] = {}
for k, v in items.items():
out[k] = await self.set_setting(k, v)
return out
# -------------------------------------------------------------------- tags
async def list_tags(self, *, min_count: int = 0) -> list[dict[str, Any]]:
return await self._request("GET", "/api/tags", params={"min_count": min_count})
async def create_tag(
self, name: str, *, parent_id: str | None = None
) -> dict[str, Any]:
body: dict[str, Any] = {"name": name}
if parent_id:
body["parent_id"] = parent_id
return await self._request("POST", "/api/tags", json=body)
# ------------------------------------------------------------------- atoms
async def create_atom(
self,
*,
content: str,
source_url: str | None = None,
tag_ids: Sequence[str] | None = None,
published_at: str | None = None,
) -> dict[str, Any]:
body: dict[str, Any] = {"content": content, "tag_ids": list(tag_ids or [])}
if source_url:
body["source_url"] = source_url
if published_at:
body["published_at"] = published_at
return await self._request("POST", "/api/atoms", json=body)
async def get_atom(self, atom_id: str) -> dict[str, Any]:
return await self._request("GET", f"/api/atoms/{atom_id}")
async def delete_atom(self, atom_id: str) -> None:
await self._request("DELETE", f"/api/atoms/{atom_id}")
async def get_atom_by_source_url(self, source_url: str) -> dict[str, Any] | None:
try:
# Atomic's GetAtomBySourceUrlQuery uses `url` not `source_url`.
return await self._request(
"GET", "/api/atoms/by-source-url", params={"url": source_url}
)
except AtomicApiError as e:
if e.status == 404:
return None
raise
async def get_embedding_status(self, atom_id: str) -> dict[str, Any]:
return await self._request("GET", f"/api/atoms/{atom_id}/embedding-status")
async def list_atoms(
self,
*,
limit: int = 50,
offset: int = 0,
tag_id: str | None = None,
) -> dict[str, Any]:
params: dict[str, Any] = {"limit": limit, "offset": offset}
if tag_id:
params["tag_id"] = tag_id
return await self._request("GET", "/api/atoms", params=params)
# ------------------------------------------------------------------ search
async def search(
self,
query: str,
*,
mode: str = "semantic",
limit: int = 20,
threshold: float | None = None,
) -> list[SearchHit]:
body: dict[str, Any] = {"query": query, "mode": mode, "limit": limit}
if threshold is not None:
body["threshold"] = threshold
result = await self._request("POST", "/api/search", json=body)
# Atomic returns either a list directly or {"results": [...]}
if isinstance(result, dict):
items = result.get("results") or result.get("data") or []
else:
items = result or []
return [SearchHit.from_dict(item) for item in items]
async def find_similar(
self, atom_id: str, *, threshold: float = 0.5, limit: int = 20
) -> list[SearchHit]:
params = {"threshold": threshold, "limit": limit}
result = await self._request(
"GET", f"/api/atoms/{atom_id}/similar", params=params
)
items = result if isinstance(result, list) else result.get("results", [])
return [SearchHit.from_dict(item) for item in items]
# ------------------------------------------------------------- embeddings
async def get_pipeline_status(self) -> dict[str, Any]:
return await self._request("GET", "/api/embeddings/status")
async def process_pending_embeddings(self) -> dict[str, Any]:
return await self._request("POST", "/api/embeddings/process-pending")
@asynccontextmanager
async def atomic_client(
*, token: str | None = None
) -> AsyncIterator[AtomicClient]:
"""`async with atomic_client() as a:` for short-lived scripts."""
client = AtomicClient(token=token)
try:
yield client
finally:
await client.aclose()

162
ai_platform/modules/didi_brain/shared/config.py Normal file
View file

@ -0,0 +1,162 @@
"""Centralized configuration via Pydantic Settings.
All env vars are loaded from .env once at import time and validated.
Import the singleton `settings` everywhere — never read os.environ directly.
"""
from __future__ import annotations
from enum import Enum
from functools import lru_cache
from pathlib import Path
from pydantic import Field, HttpUrl, field_validator
from pydantic_settings import BaseSettings, SettingsConfigDict
class LlmRole(str, Enum):
"""Logical role a caller asks for. Routing decides which model serves it."""
REASONING = "reasoning" # critical: extraction, NLI, verdict, wiki
FAST = "fast" # mass processing (currently disabled)
VISION = "vision" # multimodal (currently disabled)
class Settings(BaseSettings):
"""Top-level config. Validated at startup, immutable thereafter."""
model_config = SettingsConfigDict(
env_file=Path(__file__).parent.parent / ".env",
env_file_encoding="utf-8",
extra="ignore",
)
# ---- LLM router ---------------------------------------------------------
llm_router_url: str = Field(default="http://localhost:14011")
llm_router_api_key: str = Field(default="")
llm_vllm_url: str = Field(default="http://localhost:14001")
llm_llamacpp_urls: str = Field(default="") # comma-separated
# ---- Models -------------------------------------------------------------
model_reasoning: str = Field(default="Qwen3.5-397B-A17B")
model_reasoning_backend: str = Field(default="llamacpp")
model_fast: str = Field(default="qwen3.5")
model_fast_backend: str = Field(default="vllm")
model_fast_enabled: bool = Field(default=False)
model_vision: str = Field(default="gemma-3-27b-it")
model_vision_url: str = Field(default="")
model_vision_enabled: bool = Field(default=False)
# ---- Embeddings ---------------------------------------------------------
embedding_url: str = Field(default="http://10.11.10.15:8200")
embedding_api_key: str = Field(default="")
embedding_model: str = Field(default="BAAI/bge-m3")
embedding_dim: int = Field(default=1024)
embedding_max_tokens: int = Field(default=8192)
# ---- Reranker -----------------------------------------------------------
reranker_url: str = Field(default="http://10.11.10.15:8100")
reranker_api_key: str = Field(default="")
reranker_model: str = Field(default="BAAI/bge-reranker-v2-m3")
# ---- Atomic -------------------------------------------------------------
atomic_url: str = Field(default="http://localhost:8080")
atomic_token: str = Field(default="")
# ---- Postgres -----------------------------------------------------------
postgres_user: str = Field(default="atomic")
postgres_password: str = Field(default="atomic_dev_changeme")
postgres_db: str = Field(default="atomic")
postgres_port: int = Field(default=5434)
postgres_host: str = Field(
default="postgres",
description="Hostname for direct PG connection (Docker: 'postgres', host: 'localhost')",
)
postgres_internal_port: int = Field(
default=5432,
description="Port inside the Docker network (external is postgres_port)",
)
# ---- Verification cache --------------------------------------------------
verification_cache_ttl_days: int = Field(
default=30,
description="How long cached verification entries live before auto-expiry",
)
verification_cache_max_payload_kb: int = Field(
default=64,
description="Reject POST /v1/verification_cache with payloads above this cap",
)
# ---- Analysis atom tier policy ------------------------------------------
# These are read live from the AI platform dashboard via RuntimeConfigClient
# (keys: brain.atom.silver_ttl_days, brain.atom.bronze_ttl_days,
# brain.atom.confidence_silver_threshold). The values below are fallbacks
# used at startup until the first dashboard poll completes (~30s).
atom_silver_ttl_days: int = Field(
default=90,
description="TTL for LLM-cached analysis atoms (silver tier)",
)
atom_bronze_ttl_days: int = Field(
default=30,
description="TTL for low-confidence atoms (never served, kept for audit)",
)
atom_confidence_silver_threshold: float = Field(
default=60.0,
description="LLM confidence ≥ this stores atom as silver, else bronze",
)
# ---- Logging ------------------------------------------------------------
log_level: str = Field(default="INFO")
# ---- Runtime config (live polling from AI platform dashboard) -----------
dashboard_url: str | None = Field(
default=None,
description=(
"Optional dashboard base URL (e.g. http://didiAI-dashboard:51300). "
"When set, RuntimeConfigClient polls /api/config every 30s for live "
"overrides on atom_* and log_level."
),
)
# ---- Computed -----------------------------------------------------------
@property
def postgres_dsn(self) -> str:
"""Async-compatible DSN for direct asyncpg connections."""
return (
f"postgresql://{self.postgres_user}:{self.postgres_password}"
f"@{self.postgres_host}:{self.postgres_internal_port}/{self.postgres_db}"
)
@property
def llamacpp_urls_list(self) -> list[str]:
return [u.strip() for u in self.llm_llamacpp_urls.split(",") if u.strip()]
def model_for(self, role: LlmRole) -> tuple[str, str] | None:
"""Return (model_id, backend_hint) for a logical role, or None if disabled."""
if role == LlmRole.REASONING:
return (self.model_reasoning, self.model_reasoning_backend)
if role == LlmRole.FAST and self.model_fast_enabled:
return (self.model_fast, self.model_fast_backend)
if role == LlmRole.VISION and self.model_vision_enabled:
return (self.model_vision, "external")
return None
@field_validator("log_level")
@classmethod
def _validate_log_level(cls, v: str) -> str:
v = v.upper()
if v not in {"DEBUG", "INFO", "WARNING", "ERROR", "CRITICAL"}:
raise ValueError(f"invalid log_level: {v}")
return v
@lru_cache(maxsize=1)
def get_settings() -> Settings:
"""Singleton accessor. Cached so .env is parsed only once per process."""
return Settings()
# Convenience: most code can `from shared.config import settings`
settings = get_settings()

203
ai_platform/modules/didi_brain/shared/embedding_client.py Normal file
View file

@ -0,0 +1,203 @@
"""Embedding + reranking client.
- Embeddings: BAAI/bge-m3 via vLLM OpenAI-compat endpoint, 1024-dim, 8K context.
- Reranker: BAAI/bge-reranker-v2-m3 cross-encoder via vllm-rerank-api endpoint.
Both run on the same host (10.11.10.15) on different ports (8200 / 8100).
"""
from __future__ import annotations
import math
from collections.abc import AsyncIterator, Sequence
from contextlib import asynccontextmanager
from dataclasses import dataclass
from typing import Any
import httpx
from tenacity import (
AsyncRetrying,
retry_if_exception_type,
stop_after_attempt,
wait_exponential,
)
from shared.config import settings
from shared.logging import get_logger
log = get_logger(__name__)
class EmbeddingError(Exception):
pass
@dataclass(slots=True, frozen=True)
class RerankResult:
"""One reranked document, sorted by score descending."""
index: int # original position in the input documents list
score: float # cross-encoder score (higher = more relevant)
document: str # the text of that document
class EmbeddingClient:
"""Async client for BGE-M3 embeddings + BGE-reranker-v2-m3 reranking."""
def __init__(
self,
*,
embed_url: str | None = None,
rerank_url: str | None = None,
timeout: float = 60.0,
max_attempts: int = 3,
):
self._embed_base = (embed_url or settings.embedding_url).rstrip("/")
self._rerank_base = (rerank_url or settings.reranker_url).rstrip("/")
self._max_attempts = max_attempts
embed_headers = {"Content-Type": "application/json"}
if settings.embedding_api_key:
embed_headers["Authorization"] = f"Bearer {settings.embedding_api_key}"
rerank_headers = {"Content-Type": "application/json"}
if settings.reranker_api_key:
rerank_headers["Authorization"] = f"Bearer {settings.reranker_api_key}"
self._embed_http = httpx.AsyncClient(
base_url=self._embed_base,
headers=embed_headers,
timeout=httpx.Timeout(timeout, connect=10.0),
)
self._rerank_http = httpx.AsyncClient(
base_url=self._rerank_base,
headers=rerank_headers,
timeout=httpx.Timeout(timeout, connect=10.0),
)
async def aclose(self) -> None:
await self._embed_http.aclose()
await self._rerank_http.aclose()
async def __aenter__(self) -> EmbeddingClient:
return self
async def __aexit__(self, *args: Any) -> None:
await self.aclose()
# ------------------------------------------------------------------ embed
async def embed(self, texts: Sequence[str]) -> list[list[float]]:
"""Embed a batch of texts. Returns 1024-dim vectors in input order.
BGE-M3 max input is 8192 tokens; we don't truncate here — caller is
responsible for chunking long inputs (Atomic does this server-side).
Empty input returns empty list.
"""
if not texts:
return []
payload = {"model": settings.embedding_model, "input": list(texts)}
async for attempt in AsyncRetrying(
stop=stop_after_attempt(self._max_attempts),
wait=wait_exponential(multiplier=1, min=1, max=10),
retry=retry_if_exception_type((httpx.HTTPError, httpx.TimeoutException)),
reraise=True,
):
with attempt:
resp = await self._embed_http.post("/v1/embeddings", json=payload)
if resp.status_code >= 400:
raise EmbeddingError(
f"HTTP {resp.status_code} from embedding endpoint: "
f"{resp.text[:500]}"
)
data = resp.json()
items = data.get("data")
if not items or len(items) != len(texts):
raise EmbeddingError(
f"Embedding count mismatch: got {len(items or [])} for "
f"{len(texts)} inputs"
)
vecs = [item["embedding"] for item in items]
if vecs and len(vecs[0]) != settings.embedding_dim:
raise EmbeddingError(
f"Embedding dim mismatch: got {len(vecs[0])}, "
f"expected {settings.embedding_dim}"
)
return vecs
raise EmbeddingError("retry loop exited without result") # unreachable
async def embed_one(self, text: str) -> list[float]:
"""Convenience for single-text embedding."""
result = await self.embed([text])
return result[0]
# ----------------------------------------------------------------- rerank
async def rerank(
self,
query: str,
documents: Sequence[str],
*,
top_n: int | None = None,
) -> list[RerankResult]:
"""Cross-encoder rerank query × documents. Returns sorted by score desc.
Use this AFTER first-stage embedding retrieval to dramatically boost
precision on the top-N candidates (typical pattern: embed retrieves 50,
rerank picks 10).
"""
if not documents:
return []
payload: dict[str, Any] = {
"model": settings.reranker_model,
"query": query,
"documents": list(documents),
}
if top_n is not None:
payload["top_n"] = top_n
async for attempt in AsyncRetrying(
stop=stop_after_attempt(self._max_attempts),
wait=wait_exponential(multiplier=1, min=1, max=10),
retry=retry_if_exception_type((httpx.HTTPError, httpx.TimeoutException)),
reraise=True,
):
with attempt:
resp = await self._rerank_http.post("/v1/rerank", json=payload)
if resp.status_code >= 400:
raise EmbeddingError(
f"HTTP {resp.status_code} from reranker: {resp.text[:500]}"
)
data = resp.json()
results = data.get("results", [])
return [
RerankResult(
index=int(r["index"]),
score=float(r["score"]),
document=r.get("document") or documents[int(r["index"])],
)
for r in results
]
raise EmbeddingError("retry loop exited without result") # unreachable
# --------------------------------------------------------------------- math util
def cosine(a: Sequence[float], b: Sequence[float]) -> float:
"""Cosine similarity. BGE-M3 vectors are NOT pre-normalized; we compute it."""
if len(a) != len(b):
raise ValueError(f"vector dim mismatch: {len(a)} vs {len(b)}")
dot = sum(x * y for x, y in zip(a, b, strict=True))
na = math.sqrt(sum(x * x for x in a))
nb = math.sqrt(sum(x * x for x in b))
if na == 0 or nb == 0:
return 0.0
return dot / (na * nb)
@asynccontextmanager
async def embedding_client() -> AsyncIterator[EmbeddingClient]:
client = EmbeddingClient()
try:
yield client
finally:
await client.aclose()

260
ai_platform/modules/didi_brain/shared/llm_client.py Normal file
View file

@ -0,0 +1,260 @@
"""LLM client — async wrapper over the OpenAI-compat router on :14011.
Design:
- One LlmClient instance per long-running process (reusable httpx.AsyncClient).
- Roles (REASONING / FAST / VISION) decide model + backend automatically.
- Hard timeouts and retries are enforced; no LLM call hangs forever.
- Helpers for the two patterns we use most: structured JSON and single-token NLI.
"""
from __future__ import annotations
import json
import re
from collections.abc import AsyncIterator
from contextlib import asynccontextmanager
from typing import Any
import httpx
from tenacity import (
AsyncRetrying,
retry_if_exception_type,
stop_after_attempt,
wait_exponential,
)
from shared.config import LlmRole, settings
from shared.logging import get_logger
log = get_logger(__name__)
class LlmError(Exception):
"""Wraps any failure from the LLM stack with enough context to debug."""
def __init__(self, message: str, *, status: int | None = None, body: str | None = None):
super().__init__(message)
self.status = status
self.body = body
class RoleNotAvailable(LlmError):
"""The requested role has no enabled model behind it."""
class LlmClient:
"""Async OpenAI-compat client targeting the unified router."""
def __init__(
self,
*,
base_url: str | None = None,
timeout: float = 120.0,
max_attempts: int = 3,
):
self._base_url = (base_url or settings.llm_router_url).rstrip("/")
self._timeout = timeout
self._max_attempts = max_attempts
headers = {"Content-Type": "application/json"}
if settings.llm_router_api_key:
headers["Authorization"] = f"Bearer {settings.llm_router_api_key}"
self._http = httpx.AsyncClient(
base_url=self._base_url,
headers=headers,
timeout=httpx.Timeout(timeout, connect=10.0),
)
async def aclose(self) -> None:
await self._http.aclose()
async def __aenter__(self) -> LlmClient:
return self
async def __aexit__(self, *args: Any) -> None:
await self.aclose()
# ------------------------------------------------------------------ core
async def chat(
self,
*,
role: LlmRole = LlmRole.REASONING,
messages: list[dict[str, str]],
temperature: float = 0.1,
max_tokens: int = 1024,
stop: list[str] | None = None,
extra: dict[str, Any] | None = None,
) -> dict[str, Any]:
"""Raw chat completion. Returns the parsed response dict.
The response shape is OpenAI-compatible:
{"choices": [{"message": {"content": "..."}}], "usage": {...}, "backend": "..."}
"""
model_info = settings.model_for(role)
if not model_info:
raise RoleNotAvailable(f"Role {role.value} has no enabled model")
model_id, backend = model_info
payload: dict[str, Any] = {
"model": model_id,
"backend": backend,
"messages": messages,
"temperature": temperature,
"max_tokens": max_tokens,
}
if stop:
payload["stop"] = stop
if extra:
payload.update(extra)
async for attempt in AsyncRetrying(
stop=stop_after_attempt(self._max_attempts),
wait=wait_exponential(multiplier=1, min=1, max=10),
retry=retry_if_exception_type((httpx.HTTPError, httpx.TimeoutException)),
reraise=True,
):
with attempt:
resp = await self._http.post("/v1/chat/completions", json=payload)
if resp.status_code >= 400:
raise LlmError(
f"HTTP {resp.status_code} from LLM router",
status=resp.status_code,
body=resp.text[:1000],
)
data = resp.json()
if "choices" not in data:
raise LlmError(
"LLM response missing 'choices'",
body=json.dumps(data)[:1000],
)
return data
raise LlmError("retry loop exited without result") # unreachable
# ------------------------------------------------------------------ helpers
async def chat_text(
self,
*,
role: LlmRole = LlmRole.REASONING,
system: str | None = None,
user: str,
temperature: float = 0.1,
max_tokens: int = 1024,
stop: list[str] | None = None,
) -> tuple[str, dict[str, Any]]:
"""Convenience: send (system?, user) → return (text, usage_dict)."""
msgs: list[dict[str, str]] = []
if system:
msgs.append({"role": "system", "content": system})
msgs.append({"role": "user", "content": user})
data = await self.chat(
role=role,
messages=msgs,
temperature=temperature,
max_tokens=max_tokens,
stop=stop,
)
text = data["choices"][0]["message"]["content"]
usage = data.get("usage", {}) | {"backend": data.get("backend", "")}
return text, usage
async def chat_json(
self,
*,
role: LlmRole = LlmRole.REASONING,
system: str | None = None,
user: str,
max_tokens: int = 2048,
temperature: float = 0.0,
) -> tuple[dict[str, Any] | list[Any], dict[str, Any]]:
"""Send a request expected to return JSON. Strips fences if present.
Raises LlmError if the response is not parseable as JSON.
"""
text, usage = await self.chat_text(
role=role,
system=system,
user=user,
temperature=temperature,
max_tokens=max_tokens,
)
cleaned = _extract_json(text)
try:
return json.loads(cleaned), usage
except json.JSONDecodeError as e:
raise LlmError(
f"LLM did not return valid JSON: {e}",
body=text[:1000],
) from e
async def chat_label(
self,
*,
role: LlmRole = LlmRole.REASONING,
system: str,
user: str,
allowed: list[str],
max_tokens: int = 16,
) -> tuple[str, dict[str, Any]]:
"""Single-label classification. Returns the matched label uppercased.
Useful for NLI (SUPPORT/CONTRADICT/NEUTRAL), credibility tiers, etc.
Raises LlmError if no allowed label is found in the response.
"""
text, usage = await self.chat_text(
role=role,
system=system,
user=user,
temperature=0.0,
max_tokens=max_tokens,
)
upper = text.upper()
for label in allowed:
if label.upper() in upper:
return label.upper(), usage
raise LlmError(
f"LLM response did not contain any allowed label {allowed}",
body=text[:500],
)
# ------------------------------------------------------------------ health
async def list_models(self) -> list[dict[str, Any]]:
resp = await self._http.get("/v1/models")
resp.raise_for_status()
data = resp.json()
return data.get("data", [])
async def list_backends(self) -> list[str]:
try:
resp = await self._http.get("/v1/backends")
resp.raise_for_status()
return resp.json().get("backends", [])
except httpx.HTTPError:
return []
# --------------------------------------------------------------------- utilities
_FENCE_RE = re.compile(r"```(?:json)?\s*(.*?)\s*```", re.DOTALL)
def _extract_json(text: str) -> str:
"""Strip markdown code fences and isolate the JSON object/array if needed."""
text = text.strip()
m = _FENCE_RE.search(text)
if m:
return m.group(1).strip()
# Find first { or [ and last matching close
starts = [text.find("{"), text.find("[")]
starts = [s for s in starts if s >= 0]
if not starts:
return text
start = min(starts)
return text[start:].strip()
@asynccontextmanager
async def llm_client() -> AsyncIterator[LlmClient]:
"""`async with llm_client() as llm:` for short-lived scripts."""
client = LlmClient()
try:
yield client
finally:
await client.aclose()

50
ai_platform/modules/didi_brain/shared/logging.py Normal file
View file

@ -0,0 +1,50 @@
"""Structured logging via structlog. Single setup function called from main scripts."""
from __future__ import annotations
import logging
import sys
import structlog
from shared.config import settings
def setup_logging(level: str | None = None) -> structlog.stdlib.BoundLogger:
"""Configure structlog + stdlib logging once. Returns a base logger."""
log_level = (level or settings.log_level).upper()
# Force UTF-8 stdout on Windows so Romanian/Cyrillic/etc. don't crash rich.
try:
sys.stdout.reconfigure(encoding="utf-8") # type: ignore[union-attr]
sys.stderr.reconfigure(encoding="utf-8") # type: ignore[union-attr]
except (AttributeError, OSError):
pass
logging.basicConfig(
format="%(message)s",
stream=sys.stdout,
level=getattr(logging, log_level),
)
# Silence overly chatty third-party loggers (httpx prints every request).
for noisy in ("httpx", "httpcore", "urllib3"):
logging.getLogger(noisy).setLevel(logging.WARNING)
structlog.configure(
processors=[
structlog.contextvars.merge_contextvars,
structlog.processors.add_log_level,
structlog.processors.TimeStamper(fmt="iso", utc=True),
structlog.dev.ConsoleRenderer(colors=False),
],
wrapper_class=structlog.make_filtering_bound_logger(getattr(logging, log_level)),
cache_logger_on_first_use=True,
)
return structlog.get_logger()
def get_logger(name: str | None = None) -> structlog.stdlib.BoundLogger:
"""Get a logger; setup_logging() must have been called once first."""
return structlog.get_logger(name) if name else structlog.get_logger()

262
ai_platform/modules/didi_brain/shared/taxonomy.py Normal file
View file

@ -0,0 +1,262 @@
"""Canonical tag taxonomy for DidiBrain.
This module is the SINGLE SOURCE OF TRUTH for what tags exist in the brain.
All scrapers, extractors, and importers reference these names — never make up
ad-hoc tags.
The structure is a nested dict where each key is either:
- a leaf (value is None), or
- a subtree (value is another dict)
The seeder script (scripts/04_seed_taxonomy.py) walks this tree and creates
any missing tags in Atomic. It is idempotent.
Atomic creates 5 default root tags at first boot: Topics, People, Locations,
Organizations, Events. We REUSE the "Topics" root by extending it with our
own children — the rest stay as-is for compatibility with Atomic's optional
auto-tagging (which we keep disabled but won't fight).
After seeding, scripts/04 writes a flat map of "Path/Like/This → tag_uuid"
to shared/_tag_ids.json, which downstream code reads via TagResolver.
"""
from __future__ import annotations
import json
from collections.abc import Iterator
from pathlib import Path
from typing import Any
# A node is either a dict (subtree) or None (leaf).
TaxonomyTree = dict[str, "TaxonomyTree | None"]
# ---------------------------------------------------------------- the spec
# Order in this dict is the order tags will be created in Atomic.
TAXONOMY: TaxonomyTree = {
# Reuse Atomic's existing "Topics" root and extend it with our hierarchy.
"Topics": {
"Health": {
"Vaccines": None,
"COVID": None,
"Disease": None,
"Medicine": None,
"PublicHealth": None,
},
"Politics": {
"Elections": None,
"Diplomacy": None,
"War": None,
"Government": None,
},
"Climate": None,
"Technology": None,
"Economy": None,
"Society": None,
},
# Country of origin for the source / event.
"Country": {
"Romania": None,
"USA": None,
"Russia": None,
"Ukraine": None,
"UK": None,
"France": None,
"Germany": None,
"Spain": None,
"Italy": None,
"Poland": None,
"Moldova": None,
"Global": None, # for transnational / multi-country items
},
# What kind of source the atom came from.
"SourceType": {
"Wikipedia": None,
"MainstreamMedia": None,
"StateMedia": None, # state-affiliated outlets (TASS, RT, Sputnik, Xinhua...)
"TabloidMedia": None,
"FactCheck": None, # Snopes, PolitiFact, AFP FC, Veridica, Funky...
"Government": None, # gov.ro, whitehouse.gov, who.int...
"ScientificJournal": None, # peer-reviewed
"SocialMedia": None,
"Blog": None,
"Forum": None,
},
# Editorial credibility tier — applied by scraper from a static registry.
"Credibility": {
"Tier1": None, # Reuters/AP/BBC class
"Tier2": None, # major mainstream
"Tier3": None, # weaker mainstream / tabloid
"StateAffiliated": None,
"KnownDisinfo": None, # known disinfo outlets (we do still ingest these)
"Unknown": None,
},
# Primary language of the atom content.
"Language": {
"RO": None,
"EN": None,
"RU": None,
"UA": None,
"FR": None,
"DE": None,
"ES": None,
"IT": None,
"PL": None,
},
# What KIND of atom this is (vs "what topic" — that's Topics).
"Type": {
"Document": None, # full scraped article
"Claim": None, # extracted atomic claim
"Quote": None, # verbatim quote/excerpt
"Summary": None, # synthesized summary
"Annotation": None, # human/AI annotation about another atom
},
# Stance the source itself takes toward the central claim of the document.
"Stance": {
"Asserts": None, # source presents it as fact
"Reports": None, # source describes it as someone else's claim
"Refutes": None, # source disagrees / debunks
"Questions": None, # source raises doubts but doesn't refute
"Neutral": None, # purely informational, no stance
},
# Verification status of a Claim atom — populated by Didi after analysis,
# not at ingest. Documents stay un-tagged here.
"ClaimStatus": {
"Confirmed": None,
"Disputed": None,
"Debunked": None,
"Unverified": None,
"PartiallyTrue": None,
},
}
# ============================================================ flatten helpers
def walk(tree: TaxonomyTree, parent_path: str = "") -> Iterator[tuple[str, str | None, str]]:
"""Yield (full_path, parent_path_or_None, name) for every node, depth-first.
Example output for {"A": {"B": None}}:
("A", None, "A")
("A/B", "A", "B")
"""
for name, children in tree.items():
path = f"{parent_path}/{name}" if parent_path else name
yield (path, parent_path or None, name)
if children:
yield from walk(children, path)
def all_paths(tree: TaxonomyTree | None = None) -> list[str]:
"""All canonical paths in the taxonomy, in creation order."""
return [p for p, _, _ in walk(tree if tree is not None else TAXONOMY)]
# ============================================================ tag id resolver
_DEFAULT_CACHE = Path(__file__).resolve().parent / "_tag_ids.json"
class TagResolver:
"""Resolves canonical tag paths to Atomic UUIDs.
Loads from a JSON cache file written by the seeder. If the file is
missing or stale, callers should re-run scripts/04_seed_taxonomy.py.
"""
def __init__(self, cache_path: Path | None = None):
self._cache_path = cache_path or _DEFAULT_CACHE
self._map: dict[str, str] = {}
if self._cache_path.exists():
self._map = json.loads(self._cache_path.read_text(encoding="utf-8"))
def __contains__(self, path: str) -> bool:
return path in self._map
def get(self, path: str) -> str | None:
return self._map.get(path)
def require(self, path: str) -> str:
v = self._map.get(path)
if not v:
raise KeyError(
f"Tag path {path!r} not in resolver cache at {self._cache_path}. "
f"Run scripts/04_seed_taxonomy.py."
)
return v
def ids_for(self, paths: list[str], *, ignore_missing: bool = False) -> list[str]:
ids: list[str] = []
missing: list[str] = []
for p in paths:
v = self._map.get(p)
if v:
ids.append(v)
else:
missing.append(p)
if missing and not ignore_missing:
raise KeyError(f"Missing tag paths: {missing}")
return ids
@property
def all(self) -> dict[str, str]:
return dict(self._map)
def save(self, mapping: dict[str, str]) -> None:
self._cache_path.write_text(
json.dumps(mapping, indent=2, ensure_ascii=False, sort_keys=True),
encoding="utf-8",
)
self._map = mapping
def load_from_mapping(self, mapping: dict[str, str]) -> None:
"""Replace the in-memory map without touching disk.
Used by long-running services (e.g. containerized brain_api) that
refresh the resolver from Atomic at startup, so they don't need
the _tag_ids.json file baked into the image.
"""
self._map = dict(mapping)
def build_path_map_from_tags(tags: list[dict[str, Any]]) -> dict[str, str]:
"""Convert Atomic's flat tag list (each with parent_id) into path → id map.
Atomic /api/tags returns each tag with id, name, parent_id, and a nested
children list. We don't trust the children list (depth may be limited)
and instead walk parent_id chains ourselves.
"""
by_id: dict[str, dict[str, Any]] = {}
def collect(items: list[dict[str, Any]]) -> None:
for t in items:
tid = t.get("id")
if not tid:
continue
by_id[tid] = t
kids = t.get("children") or []
if kids:
collect(kids)
collect(tags)
def path_for(tid: str) -> str:
parts: list[str] = []
cur: str | None = tid
seen: set[str] = set()
while cur and cur not in seen:
seen.add(cur)
t = by_id.get(cur)
if not t:
break
parts.append(t.get("name", "?"))
cur = t.get("parent_id")
return "/".join(reversed(parts))
return {path_for(tid): tid for tid in by_id}