clossers/didi-lot1-ai
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
didi-lot1-ai/ai_platform/modules/didi_brain/STATUS.md

21 KiB
Raw Blame History

DidiBrain — Status Snapshot

Last updated: 2026-04-23 (integration session — web-api cache + backend verification cache) Working dir: /home/admin365/ml-projects/modules/didi_brain/ Related docs:

  • CONTRACT_VERIFICATION_CACHE.md — contract backend↔brain pentru verification cache (v2, current)
  • ../../CHANGES_2026-04-23.md — session changelog cu lista completă de schimbări
  • ../web/BRAIN_INTEGRATION.md — integrarea web-api ↔ brain

TL;DR

DidiBrain is SHIP READY și INTEGRATED. Pe lângă contractul inițial HTTP (web-gathering compatibility), brain servește acum și ca:

  • Cache layer pentru web-api — premium ingests + read pentru ambele tiers
  • Verification cache pentru didi-backend — post-LLM stance/verdict storage cu 4 staleness states (fresh / stale_framework / stale_prompt / miss)

Trei containere, zero regressions pe funcționalitatea v1. Tabelul relational nou brain_verification_cache trăiește în același Postgres cu atomic-server, prefix brain_ pentru izolare.

Current state in 30 seconds

3 containers running continuously, all healthy

  didibrain-api       brain_api FastAPI service     :8090   52 MB RAM
  didibrain-atomic    atomic-server (API only)      :8088   83 MB RAM
  didibrain-postgres  pgvector pg16                 :5434   60 MB RAM
                                                    (+ brain_verification_cache table)

Brain contents (grow continuously via web-api ingest + manual bootstrap)
  79 tags         (canonical taxonomy, 7 root namespaces)
  19+ documents   (initial Wikipedia vaccines seed + anything premium adds)
  509+ claims     (extracted by Qwen 397B, substring validated)

Verification cache (new in v2 — Apr 23)
  brain_verification_cache  UNIQUE (claim_hash, tier)
  TTL 30 days, max payload 64KB, UPSERT last-wins
  4 staleness states exposed via brain_meta

Query latency (measured end-to-end)
  /v1/gather  with  NLI  ~5-6 s
  /v1/gather  no    NLI  ~1.5-2 s
  /v1/search                 ~200-400 ms
  /v1/fetch                  ~30-100 ms
  /v1/image-search           ~5 ms (stub)
  /v1/ingest                 variable (atom create + optional async extract)

What it does

Brain-API speaks Didi's existing web-gathering module contract. Didi's backend can call it like calling the web module today, and switch between them transparently based on brain_meta.cache_status:

Didi backend                         ┌───────────────────┐
   │                                 │   DidiBrain       │
   │  POST /v1/gather {claim}        │   (this repo)     │
   ├────────────────────────────────▶│                   │
   │                                 │  1 semantic search│
   │                                 │  2 BGE rerank     │
   │                                 │  3 NLI stance     │
   │                                 │  4 aggregate      │
   │◀────────────────────────────────┤                   │
   │  GatherResponse                 └───────────────────┘
   │  (brain_meta.cache_status = HIT | PARTIAL | MISS)
   │
   if cache_status == MISS:
       fallback = http.post(WEB_MODULE, {claim})
       # optional: POST /v1/ingest {fallback.evidence}
       # so the brain self-populates from the expensive web module

Architecture

┌─────────────────────────────────────────────────────────────────┐
│                       DidiBrain stack                            │
│                                                                   │
│  ┌──────────────────────┐                                        │
│  │   brain-api :8090    │  FastAPI + Uvicorn, 5 v1 endpoints     │
│  │   (didibrain-api)    │  speaks Didi contract 1:1              │
│  └──────────┬───────────┘                                        │
│             │ HTTP (docker DNS)                                  │
│  ┌──────────▼───────────┐  ┌─────────────────────┐              │
│  │   atomic-server      │  │   postgres + pgvector│              │
│  │   :8080 (8088 ext)   │◀─┤   :5432 (5434 ext)   │              │
│  │   (didibrain-atomic) │  │   (didibrain-postgres│              │
│  └──────┬───────┬───────┘  └─────────────────────┘              │
│         │       │                                                 │
└─────────┼───────┼───────────────────────────────────────────────┘
          │       │
          │       │ HTTPS
          │       ▼
          │  ┌────────────────────────────────┐
          │  │  BGE-M3 embeddings             │ 10.11.10.15:8200
          │  │  (vLLM OpenAI-compat)          │ 1024 dim, 8K ctx, multilingual
          │  └────────────────────────────────┘
          │
          │  ┌────────────────────────────────┐
          │  │  BGE-reranker-v2-m3            │ 10.11.10.15:8100
          │  │  (cross-encoder)                │ precision boost
          │  └────────────────────────────────┘
          │
          │  ┌────────────────────────────────┐
          └─▶│  Qwen3.5-397B-A17B             │ 10.11.10.17:14011 (router)
             │  via LLM router                 │ round-robin to .18 and .19
             │  (llama.cpp + vLLM backends)    │ claim extraction, NLI
             └────────────────────────────────┘

All upstream endpoints live on a VPN-routed 10.11.10.x network. The Docker containers reach them through WSL2 NAT (Docker Desktop) or host networking (Linux server).

Repo layout

didibrain/
├── .env                           # real endpoints + token (gitignored)
├── .env.example                   # template
├── .dockerignore
├── .gitignore
├── pyproject.toml
├── README.md                      # project landing
├── STATUS.md                      # this file
├── AUDIT.md                       # initial upstream Atomic audit
│
├── infra/
│   └── docker-compose.yml         # 3-service stack (+ optional atomic-web)
│
├── brain_api/                     # HTTP service — the main deliverable
│   ├── Dockerfile
│   ├── requirements.txt
│   ├── app.py                     # FastAPI routes
│   ├── schemas.py                 # Pydantic v2 models — Didi contract
│   ├── deps.py                    # app state (clients)
│   ├── run.py                     # uvicorn entry
│   ├── prompts/
│   │   └── nli_v1.md
│   └── services/
│       ├── mapping.py             # atom → EvidenceItem / FetchedPage
│       ├── gather.py              # /v1/gather pipeline
│       ├── search.py              # /v1/search
│       ├── fetch.py               # /v1/fetch
│       ├── ingest.py              # /v1/ingest + background extraction
│       └── nli.py                 # stance vs query classification
│
├── shared/                        # libraries used by brain_api + scripts + lint
│   ├── config.py                  # Pydantic Settings, LlmRole, model routing
│   ├── logging.py                 # structlog, UTF-8 stdout, httpx silencer
│   ├── llm_client.py              # async OpenAI-compat wrapper
│   ├── embedding_client.py        # BGE-M3 embed + reranker
│   ├── atomic_api.py              # typed REST client for atomic-server
│   └── taxonomy.py                # canonical TAXONOMY tree + TagResolver
│
├── extractor/                     # claim extraction service (host + background)
│   ├── extract.py                 # single-doc extraction + substring quote validation
│   ├── push.py                    # claim atom creation with hash URL fragment
│   ├── batch.py                   # batch orchestrator with state file
│   ├── _state.py                  # idempotency log
│   └── prompts/
│       └── claim_extraction_v1.md
│
├── lint/                          # background contradiction detection
│   ├── runner.py                  # orchestrator
│   ├── detector.py                # single-pair NLI
│   ├── pairs.py                   # candidate generation via find_similar
│   ├── reporter.py                # rich table output
│   ├── _state.py                  # PairVerdict ledger (idempotent)
│   └── prompts/
│       └── pair_nli_v1.md
│
└── scripts/                       # operator CLI tools (host Python venv)
    ├── 01_sanity_full.py          # upstream stack validator (LLM+embed+rerank)
    ├── 02_bootstrap_atomic.py     # claim instance, configure provider
    ├── 03_sanity_atomic.py        # brain end-to-end (create→embed→search→cleanup)
    ├── 04_seed_taxonomy.py        # taxonomy seeder (idempotent)
    ├── 05_import_wikipedia_seed.py # seed-list Wikipedia import
    ├── 06_validate_queries.py     # smoke test doc-level retrieval
    ├── 07_run_extraction.py       # claim extraction batch
    ├── 08_validate_claims.py      # smoke test claim-level retrieval
    ├── 09_brain_api_demo.py       # brain_api contract test (all 5 endpoints)
    ├── 10_run_lint.py             # Lint pass runner (--limit --force)
    ├── 11_show_contradictions.py  # read state file, render top contradictions
    └── bootstrap_deploy.sh        # fresh-server deploy orchestrator

HTTP API summary — Didi contract

All responses include an additive brain_meta object with cache_status, api_version, implementation, evidence_sources, and total_claim_atoms_matched. Unknown fields are safely ignored by older backends.

Endpoint Purpose Latency Notes
GET /health liveness <10 ms container healthcheck
GET /docs Swagger UI — interactive tester (FastAPI auto)
GET /redoc ReDoc — read-only doc view
GET /openapi.json contract JSON — machine-readable schema
POST /v1/search flat search results ~200-400 ms no rerank, no NLI
POST /v1/fetch URL lookup → text ~30-100 ms returns not_in_brain for misses
POST /v1/gather claim → ranked evidence ~5-6 s with NLI the main Didi entry point
POST /v1/gather with run_nli:false claim → evidence ~1.5-2 s skip stance classification for speed
POST /v1/image-search stub <5 ms always empty list
POST /v1/ingest populate from web module variable creates atoms + optional async extraction

/v1/gather response shape (key fields)

{
  "request_id": "uuid",
  "claim": "original claim text",
  "evidence": [
    {
      "url": "https://source.example",
      "title": "...",
      "publisher": "example.com",
      "published_at": "2024-...",
      "retrieved_at": "2026-04-11T...",
      "summary": "the matching claim text, canonical single sentence",
      "full_text": "full parent document content",
      "full_text_hash": "sha256",
      "relevance_score": 0.98,
      "credibility_score": 0.70,
      "provenance": {
        "extraction_method": "brain",
        "fallback_chain": [],
        "brain_meta": {
          "parent_atom_id": "uuid",
          "best_claim_text": "...",
          "best_claim_stance_in_source": "ASSERTS|REPORTS|REFUTES|QUESTIONS|NEUTRAL",
          "stance_vs_query": "SUPPORTS|CONTRADICTS|NEUTRAL|UNKNOWN",
          "nli_confidence": 0.95,
          "reranker_score": 0.98,
          "embedding_similarity": 0.74
        }
      }
    }
  ],
  "evidence_stats": { ... },
  "search_context": { "primary_country": "Global", "detected_language": "en", ... },
  "stages": [
    { "stage": "context", "success": true, "duration_ms": 0 },
    { "stage": "retrieval", "success": true, "duration_ms": 300 },
    { "stage": "rerank", "success": true, "duration_ms": 1000 },
    { "stage": "nli", "success": true, "duration_ms": 3800 },
    { "stage": "evidence", "success": true, "duration_ms": 300 }
  ],
  "total_evidence_items": 5,
  "execution_time_ms": 5400,
  "brain_meta": {
    "cache_status": "HIT",
    "api_version": "v1",
    "implementation": "didibrain",
    "evidence_sources": 5,
    "total_claim_atoms_matched": 42
  }
}

The key signal for Didi backend:

response = requests.post(f"{BRAIN_URL}/v1/gather", json={"claim": text})
data = response.json()
if data["brain_meta"]["cache_status"] == "MISS":
    # brain does not have relevant knowledge — fall back to live web module
    data = requests.post(f"{WEB_MODULE_URL}/v1/gather", json={"claim": text}).json()
    # optional: pump it into the brain so next time we HIT
    requests.post(f"{BRAIN_URL}/v1/ingest", json={"evidence": data["evidence"]})

Deployment on a server — cheat sheet

Full automated flow:

scp -r didibrain/ user@server:~/
ssh user@server
cd ~/didibrain
cp .env.example .env
vim .env                # fill LLM_ROUTER_URL / EMBEDDING_URL / RERANKER_URL
./scripts/bootstrap_deploy.sh

Manual step-by-step is documented in scripts/bootstrap_deploy.sh comments or in README.md under "Production deploy".

What to change in .env when moving machines

Only the upstream endpoint URLs may need updating:

LLM_ROUTER_URL=http://10.11.10.17:14011       # if router is on VPN, unchanged
EMBEDDING_URL=http://10.11.10.15:8200         # if BGE is on VPN, unchanged
RERANKER_URL=http://10.11.10.15:8100          # if reranker is on VPN, unchanged

Everything else (ATOMIC_URL, ports, model names, Postgres creds) is either handled by Docker DNS automatically or comes from the same file.

Health checklist — run after any deploy/restart

# 1. Containers healthy
docker ps --format '{{.Names}} {{.Status}}' | grep didibrain
# expected: 3 lines, all "Up ... (healthy)"

# 2. Liveness
curl -fsS http://localhost:8090/health
# expected: {"status":"ok","service":"didibrain-api","version":"0.1.0"}

# 3. Full stack sanity (LLM + embed + rerank + atomic)
.venv/bin/python scripts/01_sanity_full.py
# expected: 8/8 PASS, total <10s

# 4. Brain end-to-end (create atom → embed → search → cleanup)
.venv/bin/python scripts/03_sanity_atomic.py
# expected: 5/5 PASS

# 5. Real Didi-style query
curl -s -X POST http://localhost:8090/v1/gather \
  -H 'Content-Type: application/json' \
  -d '{"claim":"vaccines cause autism","max_evidence":3,"run_nli":false}' \
  | jq '.brain_meta.cache_status, .total_evidence_items'
# expected: "HIT" and a positive number (if the vaccines corpus is loaded)

Known issues that DO NOT block deploy

  1. edges_status column does not exist — atomic-server logs this warning at startup due to an upstream Postgres schema mismatch. The semantic_edges feature in Atomic is partially broken, but nothing we use depends on it: vector search, reranker, claim extraction, and the Didi contract all work. Cosmetic — safe to ignore until upstream Atomic fixes the migration.

  2. /api/embeddings/status returns 500 — same root cause as #1. We never call this endpoint; we read embedding_status per atom via list_atoms or get_atom instead.

  3. Qwen 35B (vLLM on :14001) — thinking mode stuck ON via the router and safety alignment refuses disinfo-extraction tasks. Disabled in config (MODEL_FAST_ENABLED=false); the whole pipeline runs on Qwen 397B. If a future session un-sticks 35B, Lint pass could speed up ~4x by using it.

  4. Gemma 31B endpoint (10.11.10.16:8001) — port unreachable (host pings OK). Not used by any feature; listed as optional in .env.

  5. Atomic's React UI not running — we only deploy atomic-server (API), not atomic-web (React frontend). http://localhost:8088/ returns 404 for this reason. Brain_api has its own Swagger UI at /docs which covers dev-testing needs. If visual atom/tag/wiki browsing is needed, add an atomic-web service to compose (5-minute task).

Troubleshooting — things that have actually gone wrong

Symptom Root cause Fix
/health timeouts BGE endpoint unreachable (VPN down?) curl http://10.11.10.15:8200/v1/models on host; bring VPN back
/v1/gather returns 500 silently Same as above — Atomic can't embed the query Same fix
Post-reboot: brain_api empty reply uvicorn bound to 127.0.0.1 inside container (not 0.0.0.0) Rebuild — Dockerfile now sets BRAIN_API_HOST=0.0.0.0
LLM_ROUTER_URL points to localhost but nothing there Router is actually on 10.11.10.17:14011 (not on the dev box) Update .env; validated in session 2
Claim extraction returns zero claims on a doc list_atoms returns summary without content; must call get_atom Fixed in extractor/batch.py
NLI responses come back as NEUTRAL with confidence 0.0 MAX_PARALLEL > backend concurrency → timeouts Set MAX_PARALLEL=2 to match 2 llama.cpp instances
Wikipedia API 403 on every request Non-compliant User-Agent Use DidiBrain/0.1 (https://github.com/didibrain; didibrain@local.test)

What is NOT done (all optional, not blockers)

  • Bearer auth on brain_api — skipped intentionally. Add when exposing beyond loopback on a server with a public IP.
  • Full Lint pass on vaccines corpus — skipped. Wikipedia is too internally consistent for Lint to surface many contradictions; rerun after corpus diversification (B).
  • Corpus expansion (B) — next session. Wikipedia seeds for RO elections 2024, Russia-Ukraine war, climate, COVID general. Also RSS feeds and possibly Playwright adapters for non-Wiki sources.
  • /v1/contradictions endpoint in brain_api — currently contradictions are only visible via scripts/11_show_contradictions.py. Expose over HTTP when Didi needs programmatic access.
  • Atomic React UI (atomic-web) — optional visual browsing, add if useful for operators.
  • Monitoring / metrics — no Prometheus exporter yet.
  • Upstream schema fix for edges_status — cosmetic, PR to kenforthewin/atomic.

Session log (condensed)

Session 1 (2026-04-10, ~6 hours)

  • Full audit of upstream Atomic repo
  • Validated LLM/embed/rerank stack (8 sanity checks PASS)
  • Built shared/ + extractor/ + scripts 01-08
  • Stood up Atomic + Postgres in compose
  • Imported 19 Wikipedia vaccines articles, EN + RO
  • Extracted 513 claims via Qwen 397B (1.2% hallucination drop)
  • Validated claim-level retrieval (cross-lingual RO↔EN confirmed)

Session 2 (2026-04-11, ~4 hours)

  • Re-onboarded state after VPN restart (fixed LLM router URL)
  • Built brain_api/ FastAPI service with 5-endpoint Didi contract
  • Dockerized brain_api (self-sufficient startup, taxonomy refresh from atomic)
  • Added NLI stance-vs-query pass in /v1/gather (additive fields in brain_meta)
  • Fixed NLI concurrency (MAX_PARALLEL=2 matches llama.cpp backends)
  • Built lint/ contradiction detection module with idempotent state file
  • Smoke-tested Lint (371 pairs, 1 genuine contradiction found at 0.95 confidence)
  • Wrote deployment bootstrap script
  • Wrote this STATUS.md and the new README.md

Next session — pick-up plan

Brain is in a stable end-of-session state. To resume:

  1. Verify infra is alive (docker ps | grep didibrain)
  2. If it is not, docker compose -f infra/docker-compose.yml --env-file .env up -d
  3. Run .venv/bin/python scripts/01_sanity_full.py to confirm upstream stack
  4. Decide: corpus expansion (B), Lint full run, new feature, or deploy to server

Natural next step is corpus expansion. The code paths are complete; everything else is about feeding the brain more knowledge and then running the existing scripts over the new data.