clossers/didi-lot1-ai
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
didi-lot1-ai/ai_platform/modules/didi_brain/INDEX.md

19 KiB
Raw Blame History

DidiBrain - Index

Knowledge atom storage + claim verification cache + analysis atom cache pentru platforma DIDI. FastAPI service care vorbeste contractul web-gathering 1:1 (drop-in replacement / cache layer pentru /v1/gather) si in plus expune doua cache-uri proprii: brain_verification_cache (claims, din 2026-04-23) si brain_analysis_atom (techniques + ai_tampered + claims, NEW v2 din 2026-05-01).

Stack: Python 3.11+, FastAPI, Pydantic v2, asyncpg, BGE-M3 1024-dim embeddings (vLLM), BGE-reranker-v2-m3 (vLLM cross-encoder), Qwen3.5-397B-A17B (llama.cpp via OpenAI-compat router) Storage: Atomic (Rust knowledge graph) on Postgres 16 + pgvector. Brain owns its own tables prefixed brain_* alongside Atomic's tables. URL: http://10.11.10.12:8090 (production-local, since brain cutover 2026-05-01 — anterior pe 10.11.10.13:8090) Container: didibrain-api (alongside didibrain-atomic and didibrain-postgres)

Ce face

DidiBrain sta intre Didi backend (agent-v3) si modulele live de web-gathering. Vorbeste acelasi contract HTTP pentru /v1/gather, raspunzand din knowledge atoms pre-ingerate + semantic search + cross-encoder reranking + NLI stance vs query. Tipic 1.5-6s in loc de 20+s pentru un crawl live. Plus detine 2 cache-uri proprii pentru rezultate LLM ale didi-backend: brain_verification_cache (per-claim verification, since 2026-04-23) si brain_analysis_atom (rezultate full-component pe techniques + ai_tampered + claims, since 2026-05-01 v2).

API endpoints

Method Path Ce face
GET /health Liveness probe
POST /v1/search Flat list of doc-level search results
POST /v1/fetch Look up atoms by URL → return extracted text
POST /v1/gather Full claim → ranked evidence pipeline (with optional verification cache lookup)
POST /v1/image-search Stub, always returns empty list
POST /v1/ingest Populate brain from web-module output
POST /v1/verification_cache Write claim verification result (by didi-backend after LLM run)
POST /v1/analysis_atom/lookup NEW v2 — find cached analysis result by (content_hash, component, prompt_hash). Tier-agnostic. Bronze never served.
POST /v1/analysis_atom NEW v2 — write atom (rejects tier=free). Auto cache_tier silver/bronze din llm_confidence (>=60 silver). Gold preservation in upsert.
PATCH /v1/analysis_atom/{atom_id} NEW v2 — promote to gold (after moderator review). human_validated=true, expires_at=NULL.
GET /v1/analysis_atom/stats NEW v2 — counts per tier/component, hit rate 24h

Plus FastAPI auto: /docs (Swagger UI), /redoc, /openapi.json.

Structura fisiere

brain_api/
  app.py                  # FastAPI app + lifespan + 11 endpoints
  db.py                   # asyncpg pool + _SCHEMA_SQL (creates brain_verification_cache + brain_analysis_atom tables idempotent)
  schemas.py              # Pydantic v2 models (request/response for all endpoints)
  deps.py                 # AppState (atomic, embed, llm, resolver) singleton
  run.py                  # CLI entry (uvicorn launcher)
  Dockerfile
  requirements.txt
  prompts/                # LLM prompts for verification + extraction (nli_v1.md etc.)
  services/
    fetch.py              # /v1/fetch
    gather.py             # /v1/gather (most complex — semantic search + reranking + NLI + verification cache integration)
    ingest.py             # /v1/ingest
    search.py             # /v1/search
    verification_cache.py # claims cache (since 2026-04-23 v2 schema)
    analysis_atom.py      # NEW 2026-05-01 — techniques+ai_tampered+claims cache, 3 tiers (gold/silver/bronze)
    nli.py                # NLI stance classification
    mapping.py            # taxonomy / tag mapping helpers (atom → EvidenceItem / FetchedPage)

shared/                   # (one level up, used by brain_api)
  atomic_api.py           # AtomicClient (typed REST client for atomic-server)
  embedding_client.py     # BGE-M3 embed + reranker
  llm_client.py           # async OpenAI-compat wrapper
  config.py               # Pydantic Settings, LlmRole, model routing
  taxonomy.py             # canonical TAXONOMY tree + TagResolver
  logging.py              # structlog setup

extractor/                # claim extraction layer (separate package)
  extract.py              # single-doc extraction + substring quote validation
  push.py                 # claim atom creation
  batch.py                # batch orchestrator with state file
  prompts/

scripts/                  # operator CLI tools (numbered 01-11 + bootstrap_deploy.sh)
infra/                    # docker-compose.yml + deploy
lint/                     # custom lint pass (cross-corpus contradiction detection)
reports/                  # generated audit reports

Brain-owned PG schema

Doua tabele alaturi de cele ale Atomic, in acelasi Postgres (prefix brain_* pentru izolare).

brain_verification_cache (since 2026-04-23 v2)

Cache pentru verificare claims (one row per (claim_hash, tier), last-wins UPSERT). evidence_urls + evidence_hash stocate ca METADATA (NU in unique key) pentru ca backend sa poata calcula overlap. TTL 30 zile default.

Coloana Tip Scop
id uuid PK gen_random_uuid()
claim_hash text NOT NULL sha256(normalized claim)
tier text CHECK (free|premium)
evidence_hash text NOT NULL sha256(canonical urls), metadata only
evidence_urls jsonb NOT NULL metadata only
model text NULL LLM used
prompt_hash text NOT NULL sha256(system+user_template)[:12]
framework_version text NULL sha256 of relevant Redis configs
schema_name text default 'didi-v1'
verification_raw jsonb NULL raw LLM output
verification_processed jsonb NOT NULL mapped to canonical
created_at timestamptz default now()
updated_at timestamptz default now()
expires_at timestamptz NOT NULL TTL 30d default

UNIQUE: (claim_hash, tier). Indexes: idx_bvc_lookup, idx_bvc_expires, idx_bvc_prompt.

Migration v1 → v2: vechiul UNIQUE (claim_hash, evidence_hash, tier) facea cache-ul practic nereachable (URL-urile la read difera de cele la write). v2 muta evidence_hash afara din unique key. Migrarea e idempotenta (rulata la fiecare connect).

brain_analysis_atom (NEW 2026-05-01)

Cache pentru rezultate LLM full-component (techniques, ai_tampered, claims-aggregator). 3-tier system gold/silver/bronze.

Coloana Tip Scop
atom_id bigserial PK
content_hash text NOT NULL sha256(normalized text)
content_preview text NULL first 200 chars for debug
component text CHECK (techniques|ai_tampered|claims)
tier text CHECK (free|premium) informational only on row
prompt_hash text NOT NULL
framework_version text NULL
model_used text NULL
result_processed jsonb NOT NULL canonical mapped output
result_raw jsonb NULL
llm_confidence numeric NULL 0-100, drives silver vs bronze
cache_tier text CHECK (gold|silver|bronze) default 'silver'
human_validated bool default false true after moderator review
human_corrections jsonb NULL diff applied by moderator
validator_user_id text NULL keycloak_id
validated_at timestamptz NULL
hit_count integer default 0 incremented on lookup hit
last_hit_at timestamptz NULL
created_at, updated_at timestamptz
expires_at timestamptz NULL NULL=never (gold). Silver=90d, Bronze=30d

UNIQUE: (content_hash, component, prompt_hash) — tier OUT of unique key (write only premium, read tier-agnostic). Indexes: idx_baa_lookup, idx_baa_gold (partial WHERE cache_tier='gold'), idx_baa_expires, idx_baa_prompt.

Cache tier rules:

  • gold: human_validated=true. Set by didi-backend moderation flow (PATCH endpoint). Survives prompt change. Forever (expires_at=NULL).
  • silver: LLM result, llm_confidence >= 60. Default fresh write. TTL 90d.
  • bronze: LLM result, llm_confidence < 60. Stored for audit but NEVER served on lookup. TTL 30d.

Gold preservation: ON CONFLICT upsert preserves gold (won't downgrade to silver if a fresh write comes in — result_processed, cache_tier, llm_confidence, expires_at all CASE-WHEN protected).

Lookup logic (services/analysis_atom.py::lookup): SELECT WHERE content_hash=$1 AND component=$2 AND cache_tier IN ('gold','silver') AND (expires_at IS NULL OR expires_at > now()) ORDER BY (cache_tier='gold') DESC, updated_at DESC LIMIT 1. Apoi decide_freshness() pe baza prompt_hash + framework_version. Gold = always fresh.

How didi-backend (agent-v3) uses brain

3 puncte de integrare:

  1. Claims verification cache — claims executor cheama gatherFromBrain() cu include_verification=true. Brain returneaza evidence + cached verification (if any) + staleness. Backend decide fresh / stale_framework / stale_prompt / miss path. Backend scrie via fire-and-forget POST /v1/verification_cache. Detalii contract: CONTRACT_VERIFICATION_CACHE.md. Helper centralizat: agent-v3/src/shared/brain/client.ts.

  2. Analysis atom cache (NEW 2026-05-01) — techniques si ai_tampered executors cheama lookupAnalysisAtom() inainte de LLM. Pe hit (gold sau silver+fresh), skip LLM, return cached. Dupa LLM run, fire-and-forget writeAnalysisAtomAsync() (only tier=premium). Dupa ce moderator rezolva sesiunea cu corectii, patchAnalysisAtomGold() promoveaza silver → gold.

  3. Gather pipeline — agent-v3 inca foloseste /v1/gather pentru live web-augmented retrieval (pre-ingerat + semantic search + reranker + NLI). Branch-ul pe brain_meta.cache_status = HIT|PARTIAL|MISS decide fallback la web module.

Configuration

  • Settings via shared/config.py cu pydantic-settings
  • Env vars: postgres_dsn, atomic_url, llm_router_url, embedding_url, reranker_url, verification_cache_ttl_days (default 30), verification_cache_max_payload_kb (default 64)
  • Internal Atomic URL: http://didibrain-atomic:8080 (Docker DNS)
  • Upstream LLM/embed/rerank pe VPN 10.11.10.x (10.11.10.17:14011 router, 10.11.10.15:8200 BGE-M3, 10.11.10.15:8100 reranker)
  • No auth in v1 (binds to internal Docker network)

Deployment

  • Docker compose la infra/docker-compose.yml
  • 3 containere: didibrain-api (FastAPI), didibrain-atomic (Rust KG), didibrain-postgres (PG 16 + pgvector)
  • Schema bootstrap on startup (idempotent — _SCHEMA_SQL ruleaza la fiecare connect, all DDL is IF NOT EXISTS + ALTER guards)
  • TTL cleanup: cron job in scripts/ sterge expired silver/bronze atoms
  • Resource footprint: ~200 MB RAM total cross 3 containere; brain-api idle ~6% CPU, spike ~20% during /v1/gather cu NLI

Rebuild:

cd /home/admin365/didi_mono/ai_platform/modules/didi_brain
docker compose -f infra/docker-compose.yml --env-file .env up -d --build

Bootstrap & lint

  • scripts/01_sanity_full.py — upstream stack validator (LLM + embed + rerank, 8 checks)
  • scripts/02_bootstrap_atomic.py — claim atomic instance, configure BGE-M3 provider
  • scripts/03_sanity_atomic.py — brain end-to-end (create → embed → search → cleanup)
  • scripts/04_seed_taxonomy.py — seed canonical tag taxonomy (79 tags, 7 namespaces)
  • scripts/05_import_wikipedia_seed.py — seed-list Wikipedia import
  • scripts/06_validate_queries.py — smoke test doc-level retrieval
  • scripts/07_run_extraction.py — claim extraction batch (Qwen 397B)
  • scripts/08_validate_claims.py — smoke test claim-level retrieval
  • scripts/09_brain_api_demo.py — brain_api contract test (all endpoints)
  • scripts/10_run_lint.py — Lint pass runner (contradiction detection)
  • scripts/11_show_contradictions.py — render top contradictions
  • scripts/bootstrap_deploy.sh — fresh-server deploy orchestrator
  • lint/ — custom lint pass (cross-corpus contradiction detection via NLI)

Ce NU face

  • No image storage (image-search endpoint always returns empty)
  • No auth in v1 (intern only, binds to Docker internal network)
  • No multi-tenant (single brain instance, multiple consumers)
  • No long-term archive — atoms expire on TTL (gold = forever, silver = 90d, bronze = 30d)
  • No live web crawling (delegata la modulele M17 / web — brain serveste din corpus pre-ingest)

Related docs (in this module)

  • README.md — project landing, quick start, architecture diagram
  • STATUS.md — end-of-session snapshot + troubleshooting + session log
  • CONTRACT_VERIFICATION_CACHE.md — contract backend ↔ brain pentru verification cache (v2)
  • CHANGES_2026-04-23.md — session changelog cu integrare web-api + verification cache v2
  • ARCHITECTURE.md — diagrame + pipeline detail
  • AUDIT.md — initial upstream Atomic audit

Cache Freshness Defense (2026-05-04 → 2026-05-05) — 11/11 piloni live

Whole-system response to "what happens if a cached verdict goes stale because the world changed?". Strict additive — zero breaking changes on the contracts agent-v3 already calls.

New services in brain_api/services/

Module Pilon Role
classifier.py 1 LLM single-call classify of a claim → {volatility, topic_codes, entity_bindings, estimated_validity_hours}. Caps clamped per tier (volatile≤48h, evolving≤720h, stable≤26280h). Conservative defaults on LLM failure.
cache_judge.py 2 NLI judge run on cache hits — given a claim, cached truth direction (TRUE/FALSE/MIXED), and current top-3 evidence, decides KEEP_CACHE / INVALIDATE / NEEDS_FULL_RECHECK. Cheap-path skip for stable+young. Audit-pass bonus raises invalidate threshold for atoms that survived multiple audits.
fact_status.py 11 Versioned (subject, predicate, object) knowledge layer. register_facts_from_bindings, assert_fact_truth (opens new fact_version when truth flips), lock_fact/unlock_fact for moderator overrides, check_fact_validity for gather-time lookup. Plus admin: list_facts_admin, list_audit_log.
canonicalizer.py 7 LLM rewrites a claim with relative time markers (azi, alegerile) anchored to the current date. Same surface text at different times produces different cache keys. Passthrough on failure.
invalidation.py 8 InvalidateFilter (topic_codes / entity_canonicals / claim_pattern / since / invalidate_gold / dry_run) → bulk UPDATE on both cache tables. Audit log row written per non-dry-run execution.
topic_volatility.py D1 HTTP polls didiFramework /api/sensitive-topics?active=true (cache 60s) for admin-configured per-topic TTL/recency. reconcile_with_classifier picks min(LLM_estimate, admin_TTL).

DB schema (alongside Atomic in same Postgres, prefix brain_*)

ALTER brain_analysis_atom + brain_verification_cache
  ADD volatility ('volatile'|'evolving'|'stable')
  ADD topic_codes text[]
  ADD entity_bindings jsonb         -- list of {subject, predicate, object, confidence}
  ADD ttl_hours_used integer
  ADD last_audited_at timestamptz
  ADD audit_history jsonb           -- last-50 JudgeVerdict snapshots
  ADD consecutive_audit_passes integer

CREATE brain_fact_status               -- current truth per (subject, predicate, object)
CREATE brain_fact_version              -- temporal versioning with valid_from/valid_to
CREATE brain_audit_log                 -- mass invalidations + judge decisions + truth changes

GIN indexes on topic_codes. Partial indexes on cache_tier IN ('gold','silver') AND volatility != 'stable' for the auditor's fast candidate query.

New endpoints (brain_api/app.py)

POST /v1/canonicalize                — Pilon 7 temporal disambiguation
POST /v1/cache/invalidate            — Pilon 8 mass invalidation (with dry_run)
GET  /v1/cache/audit_log             — paginated audit browser
GET  /v1/fact_status/list            — paginated fact browser (admin)
GET  /v1/fact_status/{fact_id}       — single fact detail
GET  /v1/fact_status/{fact_id}/versions  — full timeline
PATCH /v1/fact_status/{fact_id}      — moderator override (set_truth + lock + notes)

Existing upsert paths (POST /v1/analysis_atom, POST /v1/verification_cache) now run the classifier internally and persist volatility/topic_codes/entity_bindings on every write. Lookup paths (/v1/gather with include_verification=true) consult brain_fact_status and demote staleness to stale_evidence when bound facts have flipped.

gather.py recency reranking (Pilon 3+4)

_apply_recency blends rerank score with age decay using volatility-driven profiles:

Volatility w_recency half_life_days hard cut
volatile 0.50 3 7 days
evolving 0.30 30 (none)
stable 0.00 ∞ (none)
(none / hint absent) 0.15 30 (none)

combined = (1 - w_recency) * rerank + w_recency * exp(-age_days/half_life). Volatile evidence older than recency_window_days is hard-dropped.

didibrain-scheduler container (scheduler/)

Single image, four asyncio tasks under _supervised (auto-restart with exponential backoff):

  • feeder — RSS pull at volatility-aware intervals (15 min volatile / 6 h evolving / 24 h stable). Default feeds in config.py:DEFAULT_*_FEEDS; override via FEED_VOLATILE_URLS/FEED_EVOLVING_URLS/FEED_STABLE_URLS/FEED_BREAKING_URLS env. Feeds POST to /v1/ingest with run_extraction=true.
  • auditor — daily sweep of brain_analysis_atom rows with cache_tier IN ('gold','silver') AND volatility != 'stable' AND last_audited_at < now()-1d. For each: confidence-decay cheap path → judge → apply_judge_verdict. Sequential (not parallel) to avoid LLM router saturation.
  • watcher — 5-min poll on the breaking-news feeds. Per item: classify with LLM → _build_invalidation_targets → POST /v1/cache/invalidate with topic_codes + entity_canonicals. In-process LRU of seen URLs (cap 2000). End-to-end target latency: <30s from RSS publish to cache invalidation.
  • heartbeat — touches /tmp/scheduler.healthy every 30s for the docker healthcheck.

Compose entry in infra/docker-compose.yml builds via scheduler/Dockerfile. Imports brain_api.* directly so the auditor uses the same DB pool + apply_judge_verdict helper as the API container (separate process, separate pool to the same Postgres).

Live state at write-time

47 articles ingested in first cycle. Watcher invalidated 4 breaking stories live (Iran/Hormuz, Nigerian elections, Ukraine drones, etc.) within seconds of feed publication. Auditor scheduled 24h after first start.

Memory pointer

~/.claude/projects/-home-admin365/memory/MEMORY.md § "Cache Freshness Defense" — one-liner status + counts.