Dezvoltari Evotech 42ff22bf85 LOT 1 - Optimizare script build -Instalare mono comanda

2026-06-27 06:42:02 -07:00

22 KiB

Raw Permalink Blame History

DidiBrain — Status Snapshot

Last updated: 2026-04-23 (integration session — web-api cache + backend verification cache)

Note (current): the platform is LIVE and functional. Since this snapshot the stack grew to four containers (added didibrain-scheduler), the analysis-atom cache (v2, gold/silver/bronze) and the Cache Freshness Defense layer (feeder + auditor + watcher) shipped, and the live working model is qwen3.5 (vLLM, MODEL_FAST enabled). Sections below that still say "3 containers / 5 endpoints / 397B" are corrected inline. Working dir: /home/admin365/ml-projects/modules/didi_brain/ Related docs:

CONTRACT_VERIFICATION_CACHE.md — contract backend↔brain pentru verification cache (v2, current)
../../CHANGES_2026-04-23.md — session changelog cu lista completă de schimbări
../web/BRAIN_INTEGRATION.md — integrarea web-api ↔ brain

TL;DR

DidiBrain is SHIP READY și INTEGRATED. Pe lângă contractul inițial HTTP (web-gathering compatibility), brain servește acum și ca:

Cache layer pentru web-api — premium ingests + read pentru ambele tiers
Verification cache pentru didi-backend — post-LLM stance/verdict storage cu 4 staleness states (fresh / stale_framework / stale_prompt / miss)
Analysis atom cache (v2) — techniques + ai_tampered + claims full-component results, 3-tier gold/silver/bronze (since 2026-05-01)
Cache Freshness Defense — didibrain-scheduler (feeder + auditor + watcher) keeps cached verdicts fresh

Patru containere, zero regressions pe funcționalitatea v1. Tabelele relationale noi brain_verification_cache + brain_analysis_atom (și layerul fact_status) trăiesc în același Postgres cu atomic-server, prefix brain_ pentru izolare.

Current state in 30 seconds

4 containers running continuously, all healthy

  didibrain-api        brain_api FastAPI service     :8090   52 MB RAM
  didibrain-atomic     atomic-server (API only)      :8088 (8080 internal)  83 MB RAM
  didibrain-postgres   pgvector pg16                 :5434   60 MB RAM
                                                     (+ brain_verification_cache
                                                      + brain_analysis_atom tables)
  didibrain-scheduler  feeder + auditor + watcher    (no port)

Brain contents (grow continuously via web-api ingest + manual bootstrap)
  79 tags         (canonical taxonomy, 7 root namespaces)
  19+ documents   (initial Wikipedia vaccines seed + anything premium adds)
  509+ claims     (extracted by qwen3.5, substring validated)

Verification cache (new in v2 — Apr 23)
  brain_verification_cache  UNIQUE (claim_hash, tier)
  TTL 30 days, max payload 64KB, UPSERT last-wins
  4 staleness states exposed via brain_meta

Query latency (measured end-to-end)
  /v1/gather  with  NLI  ~5-6 s
  /v1/gather  no    NLI  ~1.5-2 s
  /v1/search                 ~200-400 ms
  /v1/fetch                  ~30-100 ms
  /v1/image-search           ~5 ms (stub)
  /v1/ingest                 variable (atom create + optional async extract)

What it does

Brain-API speaks Didi's existing web-gathering module contract. Didi's backend can call it like calling the web module today, and switch between them transparently based on brain_meta.cache_status:

Didi backend                         ┌───────────────────┐
   │                                 │   DidiBrain       │
   │  POST /v1/gather {claim}        │   (this repo)     │
   ├────────────────────────────────▶│                   │
   │                                 │  1 semantic search│
   │                                 │  2 BGE rerank     │
   │                                 │  3 NLI stance     │
   │                                 │  4 aggregate      │
   │◀────────────────────────────────┤                   │
   │  GatherResponse                 └───────────────────┘
   │  (brain_meta.cache_status = HIT | PARTIAL | MISS)
   │
   if cache_status == MISS:
       fallback = http.post(WEB_MODULE, {claim})
       # optional: POST /v1/ingest {fallback.evidence}
       # so the brain self-populates from the expensive web module

Architecture

┌─────────────────────────────────────────────────────────────────┐
│                       DidiBrain stack                            │
│                                                                   │
│  ┌──────────────────────┐                                        │
│  │   brain-api :8090    │  FastAPI + Uvicorn, full route set     │
│  │   (didibrain-api)    │  speaks Didi contract 1:1              │
│  └──────────┬───────────┘                                        │
│             │ HTTP (docker DNS)                                  │
│  ┌──────────▼───────────┐  ┌─────────────────────┐              │
│  │   atomic-server      │  │   postgres + pgvector│              │
│  │   :8080 (8088 ext)   │◀─┤   :5432 (5434 ext)   │              │
│  │   (didibrain-atomic) │  │   (didibrain-postgres│              │
│  └──────┬───────┬───────┘  └─────────────────────┘              │
│         │       │                                                 │
└─────────┼───────┼───────────────────────────────────────────────┘
          │       │
          │       │ HTTPS
          │       ▼
          │  ┌────────────────────────────────┐
          │  │  BGE-M3 embeddings             │ 10.11.10.15:14100
          │  │  (vLLM OpenAI-compat)          │ 1024 dim, 8K ctx, multilingual
          │  └────────────────────────────────┘
          │
          │  ┌────────────────────────────────┐
          │  │  BGE-reranker-v2-m3            │ 10.11.10.15:14200
          │  │  (cross-encoder)                │ precision boost
          │  └────────────────────────────────┘
          │
          │  ┌────────────────────────────────┐
          └─▶│  qwen3.5                       │ 10.11.10.17:14011 (router)
             │  via LLM router                 │ round-robin to .18 and .19
             │  (vLLM backends)                │ claim extraction, NLI
             └────────────────────────────────┘

All upstream endpoints (LLM router, BGE-M3, reranker) are reached over Docker DNS on the shared external didi-network as didiAI-llm-api:14011, didiAI-embeddings-api:14100, and didiAI-rerank-api:14200 (the 10.11.10.x addresses above are the host-side equivalents).

Repo layout

didibrain/
├── .env                           # real endpoints + token (gitignored)
├── .env.example                   # template
├── .dockerignore
├── .gitignore
├── pyproject.toml
├── README.md                      # project landing
├── STATUS.md                      # this file
├── AUDIT.md                       # initial upstream Atomic audit
│
├── infra/
│   └── docker-compose.yml         # 4-service stack (api, atomic, postgres, scheduler)
│
├── brain_api/                     # HTTP service — the main deliverable
│   ├── Dockerfile
│   ├── requirements.txt
│   ├── app.py                     # FastAPI routes
│   ├── schemas.py                 # Pydantic v2 models — Didi contract
│   ├── deps.py                    # app state (clients)
│   ├── run.py                     # uvicorn entry
│   ├── prompts/
│   │   └── nli_v1.md
│   └── services/
│       ├── mapping.py             # atom → EvidenceItem / FetchedPage
│       ├── gather.py              # /v1/gather pipeline
│       ├── search.py              # /v1/search
│       ├── fetch.py               # /v1/fetch
│       ├── ingest.py              # /v1/ingest + background extraction
│       └── nli.py                 # stance vs query classification
│
├── shared/                        # libraries used by brain_api + scripts + lint
│   ├── config.py                  # Pydantic Settings, LlmRole, model routing
│   ├── logging.py                 # structlog, UTF-8 stdout, httpx silencer
│   ├── llm_client.py              # async OpenAI-compat wrapper
│   ├── embedding_client.py        # BGE-M3 embed + reranker
│   ├── atomic_api.py              # typed REST client for atomic-server
│   └── taxonomy.py                # canonical TAXONOMY tree + TagResolver
│
├── extractor/                     # claim extraction service (host + background)
│   ├── extract.py                 # single-doc extraction + substring quote validation
│   ├── push.py                    # claim atom creation with hash URL fragment
│   ├── batch.py                   # batch orchestrator with state file
│   ├── _state.py                  # idempotency log
│   └── prompts/
│       └── claim_extraction_v1.md
│
├── lint/                          # background contradiction detection
│   ├── runner.py                  # orchestrator
│   ├── detector.py                # single-pair NLI
│   ├── pairs.py                   # candidate generation via find_similar
│   ├── reporter.py                # rich table output
│   ├── _state.py                  # PairVerdict ledger (idempotent)
│   └── prompts/
│       └── pair_nli_v1.md
│
└── scripts/                       # operator CLI tools (host Python venv)
    ├── 01_sanity_full.py          # upstream stack validator (LLM+embed+rerank)
    ├── 02_bootstrap_atomic.py     # claim instance, configure provider
    ├── 03_sanity_atomic.py        # brain end-to-end (create→embed→search→cleanup)
    ├── 04_seed_taxonomy.py        # taxonomy seeder (idempotent)
    ├── 05_import_wikipedia_seed.py # seed-list Wikipedia import
    ├── 06_validate_queries.py     # smoke test doc-level retrieval
    ├── 07_run_extraction.py       # claim extraction batch
    ├── 08_validate_claims.py      # smoke test claim-level retrieval
    ├── 09_brain_api_demo.py       # brain_api contract test (core web-gathering endpoints)
    ├── 10_run_lint.py             # Lint pass runner (--limit --force)
    ├── 11_show_contradictions.py  # read state file, render top contradictions
    └── bootstrap_deploy.sh        # fresh-server deploy orchestrator

HTTP API summary — Didi contract

All responses include an additive brain_meta object with cache_status, api_version, implementation, evidence_sources, and total_claim_atoms_matched. Unknown fields are safely ignored by older backends.

Endpoint	Purpose	Latency	Notes
`GET /health`	liveness	<10 ms	container healthcheck
`GET /docs`	Swagger UI	—	interactive tester (FastAPI auto)
`GET /redoc`	ReDoc	—	read-only doc view
`GET /openapi.json`	contract JSON	—	machine-readable schema
`POST /v1/search`	flat search results	~200-400 ms	no rerank, no NLI
`POST /v1/fetch`	URL lookup → text	~30-100 ms	returns `not_in_brain` for misses
`POST /v1/gather`	claim → ranked evidence	~5-6 s with NLI	the main Didi entry point
`POST /v1/gather` with `run_nli:false`	claim → evidence	~1.5-2 s	skip stance classification for speed
`POST /v1/image-search`	stub	<5 ms	always empty list
`POST /v1/ingest`	populate from web module	variable	creates atoms + optional async extraction
`POST /v1/verification_cache`	write claim verification result	<50 ms	claims cache (v2)
`POST /v1/analysis_atom/lookup`	read cached analysis	<50 ms	techniques / ai_tampered / claims
`POST /v1/analysis_atom`	write analysis atom	<50 ms	silver/bronze by `llm_confidence`
`PATCH /v1/analysis_atom/{id}`	promote to gold	<50 ms	moderator review
`GET /v1/analysis_atom/stats`	per-tier/component counts	<20 ms	24h hit rate
`POST /v1/canonicalize`	temporal claim disambiguation	LLM-bound	Pilon 7
`POST /v1/cache/invalidate`	mass invalidation (`dry_run`)	variable	Pilon 8
`GET /v1/cache/audit_log`	audit browser	<50 ms	paginated
`GET/PATCH /v1/fact_status/*`	versioned fact-status layer	<50 ms	list/detail/versions/override

(Full set ~30 routes incl. FastAPI auto docs; see INDEX.md for the canonical list.)

/v1/gather response shape (key fields)

{
  "request_id": "uuid",
  "claim": "original claim text",
  "evidence": [
    {
      "url": "https://source.example",
      "title": "...",
      "publisher": "example.com",
      "published_at": "2024-...",
      "retrieved_at": "2026-04-11T...",
      "summary": "the matching claim text, canonical single sentence",
      "full_text": "full parent document content",
      "full_text_hash": "sha256",
      "relevance_score": 0.98,
      "credibility_score": 0.70,
      "provenance": {
        "extraction_method": "brain",
        "fallback_chain": [],
        "brain_meta": {
          "parent_atom_id": "uuid",
          "best_claim_text": "...",
          "best_claim_stance_in_source": "ASSERTS|REPORTS|REFUTES|QUESTIONS|NEUTRAL",
          "stance_vs_query": "SUPPORTS|CONTRADICTS|NEUTRAL|UNKNOWN",
          "nli_confidence": 0.95,
          "reranker_score": 0.98,
          "embedding_similarity": 0.74
        }
      }
    }
  ],
  "evidence_stats": { ... },
  "search_context": { "primary_country": "Global", "detected_language": "en", ... },
  "stages": [
    { "stage": "context", "success": true, "duration_ms": 0 },
    { "stage": "retrieval", "success": true, "duration_ms": 300 },
    { "stage": "rerank", "success": true, "duration_ms": 1000 },
    { "stage": "nli", "success": true, "duration_ms": 3800 },
    { "stage": "evidence", "success": true, "duration_ms": 300 }
  ],
  "total_evidence_items": 5,
  "execution_time_ms": 5400,
  "brain_meta": {
    "cache_status": "HIT",
    "api_version": "v1",
    "implementation": "didibrain",
    "evidence_sources": 5,
    "total_claim_atoms_matched": 42
  }
}

The key signal for Didi backend:

response = requests.post(f"{BRAIN_URL}/v1/gather", json={"claim": text})
data = response.json()
if data["brain_meta"]["cache_status"] == "MISS":
    # brain does not have relevant knowledge — fall back to live web module
    data = requests.post(f"{WEB_MODULE_URL}/v1/gather", json={"claim": text}).json()
    # optional: pump it into the brain so next time we HIT
    requests.post(f"{BRAIN_URL}/v1/ingest", json={"evidence": data["evidence"]})

Deployment on a server — cheat sheet

Full automated flow:

scp -r didibrain/ user@server:~/
ssh user@server
cd ~/didibrain
cp .env.example .env
vim .env                # fill LLM_ROUTER_URL / EMBEDDING_URL / RERANKER_URL
./scripts/bootstrap_deploy.sh

Manual step-by-step is documented in scripts/bootstrap_deploy.sh comments or in README.md under "Production deploy".

What to change in .env when moving machines

Only the upstream endpoint URLs may need updating:

LLM_ROUTER_URL=http://10.11.10.17:14011       # didiAI-llm-api, unchanged
EMBEDDING_URL=http://10.11.10.15:14100        # didiAI-embeddings-api, unchanged
RERANKER_URL=http://10.11.10.15:14200         # didiAI-rerank-api, unchanged

Everything else (ATOMIC_URL, ports, model names, Postgres creds) is either handled by Docker DNS automatically or comes from the same file.

Health checklist — run after any deploy/restart

# 1. Containers healthy
docker ps --format '{{.Names}} {{.Status}}' | grep didibrain
# expected: 3 lines, all "Up ... (healthy)"

# 2. Liveness
curl -fsS http://localhost:8090/health
# expected: {"status":"ok","service":"didibrain-api","version":"0.1.0"}

# 3. Full stack sanity (LLM + embed + rerank + atomic)
.venv/bin/python scripts/01_sanity_full.py
# expected: 8/8 PASS, total <10s

# 4. Brain end-to-end (create atom → embed → search → cleanup)
.venv/bin/python scripts/03_sanity_atomic.py
# expected: 5/5 PASS

# 5. Real Didi-style query
curl -s -X POST http://localhost:8090/v1/gather \
  -H 'Content-Type: application/json' \
  -d '{"claim":"vaccines cause autism","max_evidence":3,"run_nli":false}' \
  | jq '.brain_meta.cache_status, .total_evidence_items'
# expected: "HIT" and a positive number (if the vaccines corpus is loaded)

Known issues that DO NOT block deploy

edges_status column does not exist — atomic-server logs this warning at startup due to an upstream Postgres schema mismatch. The semantic_edges feature in Atomic is partially broken, but nothing we use depends on it: vector search, reranker, claim extraction, and the Didi contract all work. Cosmetic — safe to ignore until upstream Atomic fixes the migration.
/api/embeddings/status returns 500 — same root cause as #1. We never call this endpoint; we read embedding_status per atom via list_atoms or get_atom instead.
Fast model qwen3.5 — now enabled (MODEL_FAST_ENABLED=true) and is the live working model for the whole pipeline (extraction, NLI, gather) via the LLM router at didiAI-llm-api:14011 (vLLM). (Historically the fast model was disabled and the pipeline ran on a separate reasoning model; that is no longer the case.)
Atomic's React UI not running — we only deploy atomic-server (API), not atomic-web (React frontend). http://localhost:8088/ returns 404 for this reason. Brain_api has its own Swagger UI at /docs which covers dev-testing needs. If visual atom/tag/wiki browsing is needed, add an atomic-web service to compose (5-minute task).

Troubleshooting — things that have actually gone wrong

Symptom	Root cause	Fix
`/health` timeouts	BGE endpoint unreachable	`curl http://10.11.10.15:14100/v1/models` on host; restore upstream reachability
`/v1/gather` returns 500 silently	Same as above — Atomic can't embed the query	Same fix
Post-reboot: brain_api empty reply	uvicorn bound to 127.0.0.1 inside container (not 0.0.0.0)	Rebuild — Dockerfile now sets `BRAIN_API_HOST=0.0.0.0`
`LLM_ROUTER_URL` points to localhost but nothing there	Router is actually on 10.11.10.17:14011 (not on the dev box)	Update `.env`; validated in session 2
Claim extraction returns zero claims on a doc	`list_atoms` returns summary without `content`; must call `get_atom`	Fixed in `extractor/batch.py`
NLI responses come back as NEUTRAL with confidence 0.0	MAX_PARALLEL > backend concurrency → timeouts	Set `MAX_PARALLEL=2` to match 2 llama.cpp instances
Wikipedia API 403 on every request	Non-compliant User-Agent	Use `DidiBrain/0.1 (https://github.com/didibrain; didibrain@local.test)`

What is NOT done (all optional, not blockers)

Bearer auth on brain_api — skipped intentionally. Add when exposing beyond loopback on a server with a public IP.
Full Lint pass on vaccines corpus — skipped. Wikipedia is too internally consistent for Lint to surface many contradictions; rerun after corpus diversification (B).
Corpus expansion (B) — next session. Wikipedia seeds for RO elections 2024, Russia-Ukraine war, climate, COVID general. Also RSS feeds and possibly Playwright adapters for non-Wiki sources.
/v1/contradictions endpoint in brain_api — currently contradictions are only visible via scripts/11_show_contradictions.py. Expose over HTTP when Didi needs programmatic access.
Atomic React UI (atomic-web) — optional visual browsing, add if useful for operators.
Monitoring / metrics — no Prometheus exporter yet.
Upstream schema fix for edges_status — cosmetic, PR to kenforthewin/atomic.

Session log (condensed)

Session 1 (2026-04-10, ~6 hours)

Full audit of upstream Atomic repo
Validated LLM/embed/rerank stack (8 sanity checks PASS)
Built shared/ + extractor/ + scripts 01-08
Stood up Atomic + Postgres in compose
Imported 19 Wikipedia vaccines articles, EN + RO
Extracted 513 claims via qwen3.5 (1.2% hallucination drop)
Validated claim-level retrieval (cross-lingual RO↔EN confirmed)

Session 2 (2026-04-11, ~4 hours)

Re-onboarded state after VPN restart (fixed LLM router URL)
Built brain_api/ FastAPI service with the Didi web-gathering contract
Dockerized brain_api (self-sufficient startup, taxonomy refresh from atomic)
Added NLI stance-vs-query pass in /v1/gather (additive fields in brain_meta)
Fixed NLI concurrency (MAX_PARALLEL=2 matches llama.cpp backends)
Built lint/ contradiction detection module with idempotent state file
Smoke-tested Lint (371 pairs, 1 genuine contradiction found at 0.95 confidence)
Wrote deployment bootstrap script
Wrote this STATUS.md and the new README.md

Next session — pick-up plan

Brain is in a stable end-of-session state. To resume:

Verify infra is alive (docker ps | grep didibrain)
If it is not, docker compose -f infra/docker-compose.yml --env-file .env up -d
Run .venv/bin/python scripts/01_sanity_full.py to confirm upstream stack
Decide: corpus expansion (B), Lint full run, new feature, or deploy to server

Natural next step is corpus expansion. The code paths are complete; everything else is about feeding the brain more knowledge and then running the existing scripts over the new data.

22 KiB Raw Permalink Blame History