clossers/didi-lot1-ai
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

LOT 1 - Optimizare script build -Instalare mono comanda

Browse source
This commit is contained in:
Dezvoltari Evotech 2026-06-27 06:42:02 -07:00
parent 5380c3fc63
commit 42ff22bf85
127 changed files with 16163 additions and 532 deletions
Download patch file Download diff file Expand all files Collapse all files

64
ai_platform/modules/didi_brain/ARCHITECTURE.md
View file

@ -17,7 +17,7 @@ it, what does it depend on" for any file in the repo.
| Module / File | What it does | When it runs | Who calls it |
|---|---|---|---|
| **`infra/docker-compose.yml`** | Defines the 3-container stack | `docker compose up` | operator |
| **`infra/docker-compose.yml`** | Defines the 4-container stack (api, atomic, postgres, scheduler) | `docker compose up` | operator |
| **`brain_api/`** *(container)* | HTTP service speaking Didi's contract | every Didi request | Didi backend (HTTP) |
| **`shared/`** *(library)* | Common code: config, clients, taxonomy | imported everywhere | brain_api, extractor, lint, scripts |
| **`extractor/`** *(library + jobs)* | Turns Document atoms into Claim atoms | operator (script 07) OR background after `/v1/ingest` | scripts/07, brain_api ingest |
@ -26,12 +26,12 @@ it, what does it depend on" for any file in the repo.
| **Atomic server** *(container)* | Storage + chunking + embedding pipeline + REST | always | brain_api (HTTP), scripts (HTTP) |
| **Postgres + pgvector** *(container)* | Persistent atom + vector storage | always | atomic-server (SQL) |
**External services** (not in our repo, on VPN):
**External services** (not in our repo, reached over Docker DNS on `didi-network`):
| Where | What | Used by |
|---|---|---|
| `10.11.10.17:14011` | LLM router → Qwen 397B | brain_api NLI, extractor, lint |
| `10.11.10.15:8200` | BGE-M3 embedding server | atomic-server (for embeddings) |
| `10.11.10.15:8100` | BGE-reranker-v2-m3 | brain_api gather (for precision rerank) |
| `didiAI-llm-api` → `10.11.10.17:14011` | LLM router → qwen3.5 | brain_api NLI, extractor, lint |
| `didiAI-embeddings-api` → `10.11.10.15:14100` | BGE-M3 embedding server | atomic-server (for embeddings) |
| `didiAI-rerank-api` → `10.11.10.15:14200` | BGE-reranker-v2-m3 | brain_api gather (for precision rerank) |
---
@ -56,12 +56,12 @@ it, what does it depend on" for any file in the repo.
│ atomic_api, taxonomy, logging} │
└─────┬─────────────┬──────────────┬──────────────────────────┘
│ │ │
│ HTTP │ HTTPS │ HTTPS
│ HTTP │ HTTP │ HTTP
▼ ▼ ▼
┌───────────┐ ┌──────────┐ ┌────────────────┐
│ atomic- │ │ BGE-M3 │ │ Qwen 397B │
│ atomic- │ │ BGE-M3 │ │ qwen3.5 │
│ server │ │ + rerank │ │ via LLM router │
│ (cont.) │ │ (VPN) │ │ (VPN) │
│ (cont.) │ │ (DNS) │ │ (DNS) │
└─────┬─────┘ └──────────┘ └────────────────┘
│
▼
@ -121,8 +121,9 @@ its own.
## `brain_api/` — the HTTP service
**Purpose.** Containerized FastAPI service that exposes the 5-endpoint
contract Didi's backend already speaks. It does NOT contain business logic
**Purpose.** Containerized FastAPI service that exposes the full route set
(~30 routes: the web-gathering contract + brain-owned cache/freshness layers)
that Didi's backend already speaks. It does NOT contain business logic
that another module reuses; it ties together `shared` + `extractor` (via
`/v1/ingest`) and translates between Didi's HTTP shape and the brain's
internal capabilities.
@ -139,7 +140,7 @@ FastAPI/Uvicorn. The lifespan hook initializes the long-lived clients
| `Dockerfile` | python:3.12-slim, non-root user `brain`, COPY shared/extractor/brain_api, curl healthcheck on /health, CMD `python -m brain_api.run`. |
| `requirements.txt` | Pinned runtime deps. |
| `__init__.py` | Module marker + docstring. |
| `app.py` | The FastAPI app. Defines `lifespan` (startup/shutdown), the `/health` route, and the 5 v1 routes. Each route delegates to a function in `services/`. |
| `app.py` | The FastAPI app. Defines `lifespan` (startup/shutdown), the `/health` route, and the full set of v1 routes (gather/search/fetch/image-search/ingest + verification_cache + analysis_atom* + canonicalize + cache/invalidate + cache/audit_log + fact_status*). Each route delegates to a function in `services/`. |
| `run.py` | Uvicorn entry. Reads `BRAIN_API_HOST` and `BRAIN_API_PORT` from env (defaults 127.0.0.1:8090; container overrides to 0.0.0.0). |
| `deps.py` | `AppState` dataclass + module-level singleton accessor. Set by lifespan, read by route handlers. |
| `schemas.py` | **The contract.** Pydantic v2 models for every request/response shape the service speaks: `SearchRequest/Response`, `FetchRequest/Response`, `GatherRequest/Response`, `ImageSearchRequest/Response`, `IngestRequest/Response`, plus `EvidenceItem`, `BrainEvidenceMeta`, `BrainMeta`, etc. |
@ -159,7 +160,7 @@ FastAPI/Uvicorn. The lifespan hook initializes the long-lived clients
## `extractor/` — Document → Claim transformation
**Purpose.** Reads `Type/Document` atoms from Atomic, asks Qwen 397B to
**Purpose.** Reads `Type/Document` atoms from Atomic, asks qwen3.5 to
extract atomic factual claims, validates each claim against the source
text (substring check on the quote), and creates new `Type/Claim` atoms
with canonical hash-based source URLs and inherited tags.
@ -186,7 +187,7 @@ Both paths share the same idempotency state (`extractor/_extracted.json`).
| `extract.py` | Single-document logic. `extract_claims_from_atom(llm, title, language, content)` → `ExtractionResult` with `valid: list[ExtractedClaim]` and `rejected: dict[reason → count]`. Includes the substring quote validation. |
| `push.py` | `push_claim(atomic, parent_atom, claim, parent_title, resolver)` creates one `Type/Claim` atom. URL is `{parent_url}#claim={hash8}` for natural dedup. Inherits parent's Country/Topic/SourceType/Credibility/Language tags, adds Type/Claim + Stance/{X}. |
| `batch.py` | The orchestrator. `run_batch(limit, only_atom_ids)` paginates Document atoms, fetches each fully (list_atoms returns summary only — gotcha!), runs extraction sequentially, pushes claims, saves state after every doc so Ctrl-C is recoverable. |
| `prompts/claim_extraction_v1.md` | Versioned prompt that instructs Qwen 397B to return strict JSON with claims + verbatim quotes. Output format pinned. |
| `prompts/claim_extraction_v1.md` | Versioned prompt that instructs qwen3.5 to return strict JSON with claims + verbatim quotes. Output format pinned. |
**Imports.** `shared`, `httpx` (transitively).
@ -198,7 +199,7 @@ Both paths share the same idempotency state (`extractor/_extracted.json`).
**Purpose.** Audit job that finds claim atoms in our corpus that
contradict each other (or are paraphrases). For every claim, asks Atomic
for its semantic neighbors, then asks Qwen 397B to classify each
for its semantic neighbors, then asks qwen3.5 to classify each
candidate pair as **EQUIVALENT**, **CONTRADICTORY**, or **INCOMPARABLE**.
Stores all verdicts in a JSON ledger so re-runs only process new pairs.
@ -264,7 +265,7 @@ clean rich-formatted report, and exits with a meaningful code.
| File | Role |
|---|---|
| `docker-compose.yml` | Three services: `postgres` (pgvector/pgvector:pg16), `atomic-server` (ghcr.io/kenforthewin/atomic-server:latest with overridden entrypoint to use `--data-dir` instead of legacy `--db-path`, and `ATOMIC_STORAGE=postgres` env), `brain-api` (built from `brain_api/Dockerfile`). Two named volumes (`didibrain-pg-data`, `didibrain-atomic-data`), one bridge network (`didibrain`). All services have `restart: unless-stopped` and healthchecks. |
| `docker-compose.yml` | Four services: `postgres` (pgvector/pgvector:pg16), `atomic-server` (ghcr.io/kenforthewin/atomic-server:latest with overridden entrypoint to use `--data-dir` instead of legacy `--db-path`, and `ATOMIC_STORAGE=postgres` env), `brain-api` (built from `brain_api/Dockerfile`), and `scheduler` (built from `scheduler/Dockerfile` — feeder + auditor + watcher + heartbeat). Two named volumes (`didibrain-pg-data`, `didibrain-atomic-data`), and the external shared network `didi-network`. All services have `restart: unless-stopped` and healthchecks. |
**When it runs.** `docker compose up` from the operator (or via `bootstrap_deploy.sh`).
@ -301,7 +302,7 @@ brain_api/app.py: post_gather()
│
│ stage 4 nli → mapping.parse_claim_atom_body() per top result
│ → nli.classify_batch(llm, claim, evidence_texts)
│ (parallel calls to Qwen 397B, ~3-4 sec; one per parent doc)
│ (parallel calls to qwen3.5, ~3-4 sec; one per parent doc)
│
│ stage 5 evidence → atomic.get_atom_by_source_url() per parent doc
│ → mapping.evidence_from_parent() builds EvidenceItem
@ -355,7 +356,7 @@ cp .env.example .env, vim .env (none)
│ │
│ ├─ list Type/Document atoms
│ ├─ for each not in extractor/_extracted.json:
│ │ fetch full atom → extract.extract_claims_from_atom() (Qwen 397B)
│ │ fetch full atom → extract.extract_claims_from_atom() (qwen3.5)
│ │ push.push_claim() per valid claim → atomic.create_atom()
│ └─ save state every doc
│
@ -394,7 +395,7 @@ brain_api/app.py: post_ingest()
FastAPI BackgroundTasks: _run_extraction_background(created_ids)
│
└─▶ extractor.batch.run_batch(only_atom_ids=set(created_ids))
├─ runs Qwen 397B claim extraction
├─ runs qwen3.5 claim extraction
├─ pushes Type/Claim atoms
└─ saves extractor/_extracted.json
```
@ -416,9 +417,14 @@ from Atomic without loss.
| `extractor/_extracted.json` | `scripts/07_run_extraction.py` (and brain_api ingest) | Per-document extraction log: which docs have been processed, with which prompt version, how many claims came out. Skip already-done docs on rerun. | yes — delete and re-run; will re-extract everything |
| `lint/_contradictions.json` | `scripts/10_run_lint.py` | Per-pair verdict ledger with the EQUIVALENT / CONTRADICTORY / INCOMPARABLE labels. Skip already-done pairs on rerun. | yes — delete and re-run; will re-classify everything |
The brain_api **container** does NOT use any of these files. It calls
`atomic.list_tags()` at startup to refresh `TagResolver` in-memory, so
the image stays portable.
The brain_api **container** refreshes `TagResolver` in-memory at startup by
calling `atomic.list_tags()`, so it does not bake any of these files into the
image. It does, however, **mount** `shared/_tag_ids.json` read-only via the
compose `volumes:` entry — background extraction (triggered by `/v1/ingest`)
instantiates `TagResolver` directly, bypassing the lifespan refresh, so it
relies on that mounted file to resolve canonical tag UUIDs. The other two
ledgers (`extractor/_extracted.json`, `lint/_contradictions.json`) are
host-only and not used by the container.
---
@ -435,20 +441,20 @@ project root. The brain_api container also receives these via
| `LLM_ROUTER_API_KEY` | empty | optional bearer | no |
| `LLM_VLLM_URL` | `http://localhost:14001` | direct vLLM URL (fallback only, currently unused) | no |
| `LLM_LLAMACPP_URLS` | empty | comma-separated direct llamacpp URLs (fallback) | no |
| `MODEL_REASONING` | `Qwen3.5-397B-A17B` | model id for extraction, NLI, gather | yes |
| `MODEL_REASONING_BACKEND` | `llamacpp` | router backend hint | yes |
| `MODEL_FAST` | `qwen3.5` | fast model id (currently disabled) | no |
| `MODEL_REASONING` | `qwen3.5` | model id for extraction, NLI, gather (live working model) | yes |
| `MODEL_REASONING_BACKEND` | `vllm` | router backend hint | yes |
| `MODEL_FAST` | `qwen3.5` | fast model id (enabled — the live working model) | no |
| `MODEL_FAST_BACKEND` | `vllm` | fast backend hint | no |
| `MODEL_FAST_ENABLED` | `false` | enable fast model in routing | no |
| `MODEL_VISION` | `gemma-3-27b-it` | vision model id (currently down) | no |
| `MODEL_FAST_ENABLED` | `true` | fast model enabled in routing (live: `true`, qwen3.5 active) | no |
| `MODEL_VISION` | empty | vision model id (vision disabled; no vision model deployed) | no |
| `MODEL_VISION_URL` | empty | direct vision endpoint | no |
| `MODEL_VISION_ENABLED` | `false` | enable vision in routing | no |
| `EMBEDDING_URL` | `http://10.11.10.15:8200` | BGE-M3 vLLM endpoint | yes |
| `EMBEDDING_URL` | `http://10.11.10.15:14100` | BGE-M3 vLLM endpoint (`didiAI-embeddings-api`) | yes |
| `EMBEDDING_API_KEY` | empty | optional bearer | no |
| `EMBEDDING_MODEL` | `BAAI/bge-m3` | model id sent in requests | yes |
| `EMBEDDING_DIM` | `1024` | vector dimension (must match Atomic's setting) | yes |
| `EMBEDDING_MAX_TOKENS` | `8192` | input length cap | yes |
| `RERANKER_URL` | `http://10.11.10.15:8100` | BGE-reranker-v2-m3 endpoint | yes |
| `RERANKER_URL` | `http://10.11.10.15:14200` | BGE-reranker-v2-m3 endpoint (`didiAI-rerank-api`) | yes |
| `RERANKER_API_KEY` | empty | optional bearer | no |
| `RERANKER_MODEL` | `BAAI/bge-reranker-v2-m3` | model id sent in requests | yes |
| `ATOMIC_URL` | `http://localhost:8088` | atomic-server URL (host scripts use this; brain_api container overrides to docker DNS) | yes |
@ -504,7 +510,7 @@ Something with /v1/gather failing?
Atomic returning 500?
│
├─ docker logs didibrain-atomic — likely BGE unreachable
└─ curl http://10.11.10.15:8200/v1/models on host — VPN check
└─ curl http://10.11.10.15:14100/v1/models on host — upstream reachability check
Claim extraction acting weird?
│