# INDEX - Documentatie tehnica completa ml-projects Ultima actualizare: 2026-03-18 Acest fisier descrie fiecare modul, fiecare fisier, fiecare clasa, fiecare functie, fiecare ruta, fiecare container si fiecare port din acest monorepo. Nimic nu este omis. --- ## Cuprins 1. [Arhitectura generala](#1-arhitectura-generala) 2. [Gateway (nginx)](#2-gateway-nginx---punctul-unic-de-intrare) 3. [Catalog API](#3-catalog-api---agregator-de-servicii) 4. [LLM Inference](#4-llm-inference---router-llm-unificat) 5. [Embeddings](#5-embeddings---api-de-embeddings) 6. [Rerank](#6-rerank---api-de-reranking) 7. [Audio](#7-audio---transcriere-audio) 8. [Video Analysis](#8-video-analysis---analiza-video) 9. [Web](#9-web---cautare-web-si-fact-checking) 10. [Harta porturilor](#10-harta-completa-a-porturilor) 11. [Harta GPU](#11-alocare-gpu) 12. [Retea Docker](#12-retea-docker) --- ## 1. Arhitectura generala Monorepo cu 13 module independente (audio, catalog-api, cloak, dashboard, didi_brain, embeddings, extractors, forensic_features, gateway, llm-inference, rerank, video-analysis, web). Fiecare modul este un pachet Python instalabil cu FastAPI, containerizat in Docker, conectat pe reteaua comuna `didi-network`. Toate comunica prin HTTP intern. Singurul port expus extern este 11000 (gateway nginx). Flux tipic de request extern: ``` Client -> Gateway (nginx :11000) -> Serviciu intern (llm/audio/web/catalog) ``` Flux intern (intre servicii): ``` Web API -> LLM Inference API -> vLLM (Qwen3.5) Video API -> vLLM (BusterX) Catalog API -> (interogheaza toate celelalte servicii pe /v1/info) ``` Modele ML servite (7 modele, 13 module): - Qwen3.5-35B-A3B (text + vision + OCR, MoE) - vLLM pe GPU 0, servit ca `qwen3.5`. Nu exista model vision separat: OCR/vision se face prin acest gateway LLM. - BAAI/bge-m3 (embeddings) - vLLM sau llama.cpp, GPU 1 - BAAI/bge-reranker-v2-m3 (reranking) - vLLM sau llama.cpp, GPU 1 - Whisper large-v3-turbo (speech-to-text) - faster-whisper pe GPU 0 - BusterX (Qwen2.5-VL-7B fine-tuned, deepfake detection) - vLLM pe GPU 1 - GLiNER (backbone mdeberta, NER) + YOLOv8n (object detection) - extractors - MediaPipe (rPPG / lip-sync / landmarks) - forensic_features --- ## 2. Gateway (nginx) - punctul unic de intrare **Locatie:** `modules/gateway/` **Container:** `didiAI-gateway` **Port extern:** 11000 **Imagine:** `nginx:1.27-alpine` Gateway-ul este un reverse proxy nginx care ruteaza toate request-urile catre serviciile interne. Toate rutele (in afara de /health) necesita autentificare Bearer token. ### Fisiere **deploy/nginx.conf.template** - Template nginx cu variabile de mediu Defineste 4 upstream-uri: - `llm` -> `didiAI-llm-api:14011` - `audio` -> `didiAI-audio:54300` - `web` -> `didiAI-web-api:51100` - `catalog` -> `didiAI-catalog-api:11000` Autentificarea: nginx `map` compara header-ul `Authorization` cu `Bearer ${GATEWAY_API_TOKEN}`. Daca nu coincide, returneaza 401 JSON. Rute: - `GET /health` - fara autentificare, returneaza `{"status":"ok"}` direct din nginx - `/llm/` -> proxy catre llm upstream, cu SSE streaming (proxy_buffering off, chunked transfer) - `/audio/` -> proxy catre audio upstream, cu body buffer 10M pentru upload-uri mari - `/web/` -> proxy catre web upstream - `/catalog/` -> proxy catre catalog upstream - `/` (orice altceva) -> 404 cu lista rutelor disponibile Timeout-uri proxy: connect 60s, send 300s, read 600s. Upload maxim: 500MB (client_max_body_size). Toate request-urile primesc header X-Request-ID generat de nginx. **deploy/docker-compose.yml** - Un singur serviciu `gateway` Container `didiAI-gateway` pe imaginea `nginx:1.27-alpine`. La pornire, ruleaza `envsubst` care inlocuieste `${GATEWAY_API_TOKEN}` in template si genereaza `nginx.conf` final. Healthcheck cu `wget` pe `/health`. **deploy/deploy.sh** - Script bash Valideaza variabila `GATEWAY_API_TOKEN` (fail-fast). Suporta actiunile `up`, `down`, `logs`. Incarca `.env` din directorul `deploy/`. **deploy/.env.example** - O singura variabila required: `GATEWAY_API_TOKEN`. --- ## 3. Catalog API - agregator de servicii **Locatie:** `modules/catalog-api/` **Container:** `didiAI-catalog-api` **Port intern:** 11000 (accesat prin gateway la `/catalog/`) ### Ce face Interogheaza periodic sau la cerere endpoint-ul `/v1/info` de pe fiecare serviciu intern (LLM, Audio, Video, Web). Colecteaza metadata (modele disponibile, functii, starea de sanatate) si le expune intr-un singur loc. Genereaza si un OpenAPI spec agregat care combina spec-urile tuturor componentelor. ### Fisiere sursa **src/catalog_api/settings.py** Clasa `CatalogSettings(BaseSettings)` cu prefix `CATALOG_`: - `external_url: str` (REQUIRED, fara default) - URL extern pentru OpenAPI - `host`, `port`, `log_level` - setari server (cu default-uri) - `llm_url`, `audio_url`, `video_url`, `web_url` - URL-uri interne Docker ale serviciilor - `llm_external_port`, `audio_external_port`, `video_external_port`, `web_external_port` - porturi externe Clasa `Component(BaseSettings)`: - `id`, `url`, `external_url`, `timeout` - metadata despre un serviciu Metoda `get_components() -> list[Component]` - construieste lista componentelor configurate, omitand cele cu URL gol (ex: video_url="" dezactiveaza video). Instanta globala `settings = CatalogSettings()`. **src/catalog_api/app.py** Aplicatia FastAPI principala. Constanta `AGGREGATED_OPENAPI_VERSION = "0.1.0"`. Rute: - `GET /health` -> `health()` - returneaza `{"status": "ok"}` - `GET /v1/components` -> `list_components()` - face fetch async la `/v1/info` pe fiecare component din `settings.get_components()`. Adauga `component_id` si `base_url` la fiecare raspuns. Returneaza `{components: [...], total: N, errors: [...]}`. Gestioneaza timeout-uri si erori HTTP per component. - `GET /v1/components/{component_id}` -> `get_component(component_id)` - cauta componenta dupa ID, face fetch la `/v1/info`, returneaza metadata. 404 daca ID-ul nu exista. - `GET /v1/models` -> `list_models()` - colecteaza array-ul `models` din `/v1/info` al fiecarei componente. Adauga `component_id` si `component_name` la fiecare model. Ignora silentios componentele cu erori. - `GET /v1/functions` -> `list_functions()` - la fel ca models, dar extrage array-ul `functions`. - `GET /v1/status` -> `get_status()` - verifica conectivitatea cu fiecare componenta. Statusul general: "healthy" (toate ok), "degraded" (unele ok), "unhealthy" (niciuna). Returneaza `{status, components: [...], healthy_count, total_count}`. - `GET /v1/openapi` -> `get_aggregated_openapi()` - face fetch la `/openapi.json` de pe fiecare componenta. Combina spec-urile intr-un singur OpenAPI 3.1.0 cu titlul "didiAI - Aggregated ML Services API". Prefixeaza path-urile cu `/{component_id}` si schema-urile cu `{component_id}_` pentru a evita coliziuni. - `GET /v1/docs` -> `get_aggregated_swagger_ui()` - pagina HTML cu Swagger UI care incarca `/v1/openapi`. - `GET /v1/redoc` -> `get_aggregated_redoc()` - pagina HTML cu ReDoc. - `GET /v1/openapi/component/{component_id}` -> `get_component_openapi(component_id)` - returneaza OpenAPI spec-ul brut al unei componente specifice. Functii helper: - `_merge_openapi_schemas(base_spec, component_spec, component_id, component_url)` - combina spec-ul unei componente in spec-ul agregat. Prefixeaza path-urile, schema-urile, operationId-urile. - `_update_refs(obj, component_id)` - actualizeaza recursiv referintele `$ref` din obiectele OpenAPI. La final, `uvicorn.run()` porneste serverul. ### Deploy **deploy/Dockerfile** - Multi-stage build pe python:3.11-slim cu uv. Port default 11000. CMD: `uvicorn catalog_api.app:app`. **deploy/docker-compose.yml** - Serviciu `catalog-api`, container `didiAI-catalog-api`, port 11000, retea `didi-network`. Healthcheck pe `/health`. --- ## 4. LLM Inference - router LLM unificat **Locatie:** `modules/llm-inference/` **Containere:** `didiAI-llm-api` (port 14011), `didiAI-vllm-qwen3.5` (port 14001) ### Ce face Gateway unificat pentru inferenta LLM. Primeste cereri OpenAI-compatibile si le ruteaza catre unul din 3 backend-uri: LiteLLM (100+ provideri cloud), vLLM (GPU local), llama.cpp (CPU local). Suporta streaming SSE, retry cu backoff exponential, rate limiting, concurrency limiting, autentificare Bearer, load balancing pentru llama.cpp. ### Fisiere sursa **src/llm_inference/types.py** - Tipuri de baza - `BackendType(str, Enum)` - LITELLM, VLLM, LLAMACPP - `ChatMessage(BaseModel)` - role (system/user/assistant/function/tool), content (str sau list multimodal), name optional - `Usage(BaseModel)` - prompt_tokens, completion_tokens, total_tokens - `Choice(BaseModel)` - index, message, finish_reason (raspuns non-streaming) - `Delta(BaseModel)` - role, content (raspuns streaming) - `StreamChoice(BaseModel)` - index, delta, finish_reason - `ModelInfo(BaseModel)` - id, backend, loaded, context_length, capabilities **src/llm_inference/schemas.py** - Scheme API - `CompletionRequest(BaseModel)` - messages (1-1000), model, temperature (0-2, default 0.7), max_tokens, stream (bool), backend (override optional), top_p, frequency_penalty, presence_penalty, stop. Validator: non-assistant messages trebuie sa aiba content. - `CompletionResponse(BaseModel)` - id, object="chat.completion", created (unix timestamp), model, choices, usage, backend - `CompletionChunk(BaseModel)` - id, object="chat.completion.chunk", created, model, choices (StreamChoice) - `ModelListResponse`, `ModelLoadRequest`, `ModelLoadResponse` - management modele - `BackendHealth`, `HealthResponse`, `ReadinessResponse`, `BackendListResponse` - monitoring **src/llm_inference/config.py** - Configurare Pydantic Settings Clasa `LLMSettings(BaseSettings)` cu prefix `LLM_`: Campuri REQUIRED (fara default): - `default_backend: Literal["litellm", "vllm", "llamacpp"]` - `enable_vllm: bool` - `enable_llamacpp: bool` - `external_url: str` Campuri cu default: - `host="0.0.0.0"`, `port=14011` - `default_model="gpt-3.5-turbo"` - `openrouter_api_key`, `openai_api_key`, `anthropic_api_key` - chei API optionale - `vllm_base_url="http://localhost:14001"`, `vllm_api_key` - `llamacpp_base_url="http://localhost:8102"`, `llamacpp_base_urls` (lista, comma-separated, pt load balancing) - `llamacpp_health_check_interval=30` (interval verificare sanatate servere) - `models_dir="/models"` - `request_timeout=120.0`, `connect_timeout=10.0` - `max_retries=3`, `retry_min_wait=1.0`, `retry_max_wait=60.0` - `rate_limit_rps=10.0`, `rate_limit_burst=20` - `max_concurrent_completions=10` - `api_tokens: frozenset[str] | None` - tokeni Bearer (comma-separated) - `log_level="INFO"`, `log_json=False` Proprietati: `auth_enabled`, `llamacpp_urls`. Validatori: `parse_llamacpp_base_urls()` (parseaza string comma-separated in lista), `parse_api_tokens()`, `validate_api_keys()` (avertizeaza daca litellm fara chei API). Clasa `SettingsCache` - singleton thread-safe cu `get()`, `clear()`, `set()`. **src/llm_inference/exceptions.py** - Exceptii custom Baza: `LLMInferenceError(Exception)`. Derivate: - `AuthenticationError` - autentificare esuata - `BackendNotAvailableError(backend, reason)` - backend indisponibil - `BackendNotEnabledError(backend)` - backend neactivat in config - `ModelNotFoundError(model, backend)` - model negasit - `CompletionError(message, backend, model)` - eroare la generare - `ModelLoadError(model, backend, reason)` - eroare la incarcare model - `ModelListError(backend, reason)` - eroare la listare modele - `LLMRateLimitError(message, backend, retry_after)` - rate limit atins - `LLMTimeoutError(message, backend, timeout)` - timeout - `LLMConnectionError(backend, reason)` - conexiune esuata **src/llm_inference/retry.py** - Logica de retry Importa conditionat exceptiile din `litellm`, `openai`, `httpx`. Defineste doua tupluri globale: - `RETRYABLE_EXCEPTIONS` - RateLimitError, Timeout, ServiceUnavailableError, TimeoutException, ConnectError - `NON_RETRYABLE_EXCEPTIONS` - AuthenticationError, BadRequestError, NotFoundError Functii: - `is_retryable_exception(exc)` - verifica daca exceptia e retryable - `extract_retry_after(exc)` - extrage headerul Retry-After din exceptii de rate limit - `translate_exception(exc, backend, model)` - traduce exceptii provider-specifice in exceptii custom - `retry_with_backoff(func, max_retries, min_wait, max_wait, backend, model)` - executa functie async cu retry exponential. Formula: `min_wait * 2^attempt`, capped la max_wait, cu jitter 0-25%. Respecta Retry-After headers. **src/llm_inference/logging.py** - Logging structurat - `request_id_ctx: ContextVar[str | None]` - propagare request ID prin context - `RequestIdFilter(logging.Filter)` - adauga request_id la log records - `JsonFormatter(logging.Formatter)` - formatare JSON cu timestamp, level, logger, message, request_id, exceptie - `configure_logging(level, json_format)` - configureaza root logger "llm_inference" - `get_logger(name)` - returneaza logger cu prefix "llm_inference." - `set_request_id(request_id)` / `get_request_id()` - management context **src/llm_inference/utils.py** - `safe_close_stream(stream, logger)` - inchide sigur un stream async (incearca aclose(), fallback pe close()) **src/llm_inference/image_processing.py** - Procesare imagini multimodale Client HTTP global partajat `_http_client` pentru download. Functii: - `_get_http_client()` - returneaza/creeaza clientul HTTP partajat - `_guess_mime_type(url, content_type)` - determina MIME type din URL sau Content-Type - `_download_and_encode(url)` - descarca imagine si returneaza ca data URI base64 - `_process_content_item(item)` - proceseaza un element de continut (descarca imagini HTTP, lasa base64 si non-HTTP neschimbate) - `_process_text_with_urls(text)` - detecteaza URL-uri de imagini in text plain si le converteste in format multimodal `[{type: "text"}, {type: "image_url"}]` - `process_messages(messages: list[ChatMessage])` - proceseaza toate mesajele, descarcand URL-urile de imagini si convertindu-le in base64 **src/llm_inference/client.py** - Client de nivel inalt Clasa `LLMClient`: - `__init__(settings)` - initializeaza cu settings si BackendRegistry - `complete(messages, model, backend, **kwargs)` - generare chat completion. Proceseaza imagini, rezolva backend-ul, apeleaza backend.complete() - `stream(messages, model, backend, **kwargs)` - generare streaming. Yield-uieste CompletionChunk - `list_models(backend)` - listeaza modele de la un backend sau toate - `load_model(model, backend)` - incarca model pe backend local - `unload_model(model, backend)` - descarca model - `list_backends()` - listeaza tipurile de backend disponibile - `health_check()` - returneaza starea de sanatate per backend - `_resolve_backend_for_model(model)` - interogheaza fiecare backend local sa vada care serveste modelul - `_parse_messages(messages)` - converteste dict-uri in ChatMessage **src/llm_inference/cli.py** - Punct de intrare CLI Clasa `GracefulShutdown`: - Inregistreaza handlere SIGTERM/SIGINT - Primul semnal: shutdown graceful - Al doilea semnal: exit fortat Functia `main()`: - Argumente: `--host`, `--port`, `--workers`, `--reload`, `--graceful-timeout` - Porneste uvicorn cu factory mode `llm_inference.api.app:create_app` ### Backend-uri **src/llm_inference/backends/base.py** - Clasa abstracta `LLMBackend` Metode abstracte: `name` (property), `complete()`, `stream()`, `list_models()` Metode concrete (cu default): `load_model()` (NotImplementedError), `unload_model()` (NotImplementedError), `health_check()` (True) **src/llm_inference/backends/registry.py** - Registru de backend-uri Clasa `BackendRegistry`: - `__init__(settings)` - initializeaza backend-urile activate (LiteLLM mereu, vLLM/llama.cpp optional) - `get(backend_type)` - returneaza instanta backend (default daca None) - `list_backends()` - lista tipurilor disponibile - `is_available(backend_type)` - verifica disponibilitatea **src/llm_inference/backends/litellm_backend.py** - Backend LiteLLM Clasa `LiteLLMBackend(LLMBackend)`: - Interfata cu 100+ provideri cloud (OpenAI, Anthropic, OpenRouter, Azure, Google, AWS) - `_configure_litellm()` - seteaza cheile API pe modulul litellm - `complete()` - apeleaza `litellm.acompletion()` cu retry_with_backoff - `stream()` - streaming prin `litellm.acompletion(stream=True)`, yield-uieste CompletionChunk - `list_models()` - cache TTL 3600s. Fetcheaza dinamic de la OpenAI si OpenRouter API, fallback pe liste curate (hardcoded) - `health_check()` - verifica conectivitatea la cel putin un provider - Liste fallback: OpenAI (gpt-4o, gpt-4o-mini, gpt-4-turbo, gpt-3.5-turbo), Anthropic (Claude 3.5 Sonnet, 3 Opus, 3 Haiku) **src/llm_inference/backends/vllm_backend.py** - Backend vLLM Clasa `VLLMBackend(LLMBackend)`: - Conectare la server vLLM prin API OpenAI-compatible (pachetul `openai`) - Creeaza `AsyncOpenAI` client care pointeaza la `vllm_base_url` - `complete()` - `client.chat.completions.create()` cu retry - `stream()` - streaming prin acelasi client, cu safe_close_stream la erori - `list_models()` - `client.models.list()` - `health_check()` - incearca `models.list()`, True daca reuseste **src/llm_inference/backends/llamacpp_backend.py** - Backend llama.cpp cu load balancing Dataclass `LlamaCppServer`: - `url`, `healthy`, `request_count`, `error_count`, `last_check`, `response_times` (ultimele 50) - Property: `avg_response_ms`, `short_name` Clasa `LlamaCppBackend(LLMBackend)`: - Creeaza obiecte `LlamaCppServer` si clienti `AsyncOpenAI` pentru fiecare URL din configurare - Round-robin cu failover automat - `_get_server()` - returneaza urmatorul server sanatos (round-robin). Daca toate sunt nesanatoase, incearca pe toate. - `_mark_unhealthy(server)` / `_mark_healthy(server)` - actualizeaza starea serverului - `_periodic_health_check(interval)` - task async care verifica periodic toate serverele - `_complete_on_server(server, client, messages, model)` - executa completare pe un server specific, masoara timp de raspuns - `complete()` - incearca servere sanatoase round-robin, failover la urmatorul pe erori de conexiune/timeout - `stream()` - streaming cu failover - `list_models()` - interogheaza primul server sanatos - `health_check()` - True daca orice server e disponibil - `get_servers_status()` - returneaza starea tuturor serverelor (URL, health, request_count, error_count, avg_response_ms) ### API (FastAPI) **src/llm_inference/api/app.py** - Factory aplicatie Functia `create_app()`: - Creeaza FastAPI cu titlu "LLM Inference API", versiune 0.1.0 - Lifespan manager: la startup configureaza logging, initializeaza ConcurrencyLimiter si LLMClient. La shutdown logheaza. - Middleware (ordinea conteaza, primul adaugat = cel mai exterior): 1. RateLimitMiddleware - token bucket, exclude /health si /ready 2. RequestIdMiddleware - genereaza/extrage X-Request-ID - Routere montate: health, completions (/v1), models (/v1), info (/v1) **src/llm_inference/api/dependencies.py** - Dependinte FastAPI - `get_client(request)` - returneaza LLMClient din app.state - `get_settings(request)` - returneaza settings din app.state - `verify_bearer_token(request, authorization)` - valideaza token Bearer. Comparatie constant-time cu `hmac.compare_digest`. 401 daca invalid. Clasa `ConcurrencyLimiter`: - Limiteaza numarul de completari concurente cu asyncio.Semaphore - `acquire(blocking=False)` - context manager async. Non-blocking (default): 503 imediat daca nu sunt sloturi. Blocking: asteapta slot. - `current_count`, `available` - proprietati de monitorizare Functii globale: `init_concurrency_limiter()`, `get_concurrency_limiter()`, `require_completion_slot()` (dependinta FastAPI). **src/llm_inference/api/middleware.py** - Middleware Clasa `RequestIdMiddleware(BaseHTTPMiddleware)`: - Extrage X-Request-ID din header sau genereaza UUID - Seteaza in context (pt logging), in request.state, si in response headers Clasa `TokenBucket`: - Algoritm token bucket pentru rate limiting - `acquire()` - incearca sa consume un token. Returneaza True/False. - `retry_after()` - secunde pana cand un token e disponibil. Clasa `RateLimitMiddleware(BaseHTTPMiddleware)`: - Foloseste TokenBucket. Exclude /health si /ready. - Returneaza 429 Too Many Requests cu header Retry-After cand limita e atinsa. - ATENTIE: rate limiting per-proces, nu distribuit. Fiecare replica are limita proprie. **src/llm_inference/api/routes/completions.py** - Ruta de completari Endpoint `POST /v1/chat/completions`: - Necesita Bearer token (daca auth activat) - Pentru streaming: creeaza generator SSE care yield-uieste chunks JSON, tine slot de concurenta pe intreaga durata stream-ului, ping la fiecare 15s - Pentru non-streaming: achizitioneaza slot, ruleaza completare, elibereaza slot - Erori: BackendNotAvailableError/BackendNotEnabledError -> 400, CompletionError -> 500 Functii helper: - `_stream_generator(client, request)` - generator SSE, trimite `[DONE]` la final, eroare ca eveniment SSE - `_stream_with_slot(generator, limiter)` - wrapper care tine slot-ul pe durata stream-ului - `_build_completion_kwargs(request)` - construieste kwargs din CompletionRequest **src/llm_inference/api/routes/models.py** - Management modele - `GET /v1/models` - query param optional `backend` pentru filtrare - `POST /v1/models/load` - incarca model pe backend local (body: ModelLoadRequest) - `POST /v1/models/unload` - descarca model - `GET /v1/backends` - listeaza backend-urile disponibile **src/llm_inference/api/routes/health.py** - Health checks - `GET /health` - status per-backend, status general (healthy/degraded/unhealthy) - `GET /ready` - probe Kubernetes, verifica backend-ul default **src/llm_inference/api/routes/info.py** - Informatii component - `GET /v1/info` - returneaza metadata completa pentru catalog: resource (name, slug, config, auth, rate_limits, tags), models (cu capabilities, provider, endpoint), functions (Chat Completions, List Models, List Backends, Load Model, Unload Model cu input/output schema) ### Deploy **deploy/docker-compose.yml** - Defineste 3 servicii: 1. `llm-api` (didiAI-llm-api, port 14011) - API-ul FastAPI. Profile: api, vllm. Variabile: backend config, chei API, concurrency. 2. `vllm-qwen3.5` (didiAI-vllm-qwen3.5, port 14001) - Server vLLM cu Qwen/Qwen3.5-35B-A3B. Profile: vllm. GPU 0. Image: `vllm/vllm-openai:qwen3_5`. Parametri: max-model-len 32000, gpu-memory-utilization 0.65, enable-prefix-caching, enable-auto-tool-choice (Hermes parser). Healthcheck cu 600s start_period (modelul se incarca lent). 3. `llamacpp` (optional) - Image: `ghcr.io/ggml-org/llama.cpp:server`. Profile: llamacpp, full. GGUF model din MODELS_DIR. ctx-size 4096. **deploy/Dockerfile** - Multi-stage: python:3.11-slim cu uv. Port intern 14011. CMD: `python -m llm_inference.cli`. --- ## 5. Embeddings - API de embeddings **Locatie:** `modules/embeddings/` **Containere:** `didiAI-embeddings-api` (port 14100/54100), `didiAI-embeddings-vllm` (port 14101/54101), `didiAI-embeddings-llamacpp` (port 14110/54110) ### Ce face API OpenAI-compatibil de embeddings cu suport pentru doua backend-uri: vLLM (GPU) si llama.cpp (CPU/GGUF). Modelul principal: BAAI/bge-m3 (max 8192 tokeni). ### Fisiere sursa Structura e identica cu llm-inference (acelasi tipar arhitectural). Diferentele principale: **src/embeddings/types.py** - `BackendType(str, Enum)` - VLLM, LLAMACPP (fara LITELLM) - `EmbeddingUsage(BaseModel)` - prompt_tokens, total_tokens - `EmbeddingData(BaseModel)` - object="embedding", index, embedding (list[float]) - `ModelInfo(BaseModel)` - id, backend, loaded, dimensions, max_input_tokens **src/embeddings/schemas.py** - `EmbeddingRequest` - input (list[str] sau str), model, encoding_format ("float"/"base64"), dimensions (optional), backend (override). Validatori: ensure_list() converteste str in list, validate_input() verifica ca input-ul nu e gol. - `EmbeddingResponse` - object="list", data (list[EmbeddingData]), model, usage, backend - `encode_embedding_base64(embedding)` - encodeaza vector embedding ca base64 (little-endian floats) **src/embeddings/config.py** - `EmbeddingSettings` cu prefix `EMB_`: - Required: `default_backend`, `enable_vllm`, `enable_llamacpp`, `external_url` - Default-uri: port=54100, vllm_base_url="http://localhost:54101", llamacpp_base_url="http://localhost:54110" - Rate limiting: 20 RPS, burst 40, max 20 concurrent **src/embeddings/backends/base.py** - `EmbeddingBackend(ABC)`: - `embed(texts, model, dimensions) -> tuple[list[list[float]], EmbeddingUsage]` - `list_models()`, `health_check()` **src/embeddings/backends/vllm_backend.py** - `VLLMEmbeddingBackend`: - Foloseste `AsyncOpenAI` client catre serverul vLLM - `embed()` - `client.embeddings.create()`, returneaza vectori si usage **src/embeddings/backends/llamacpp_backend.py** - `LlamaCppEmbeddingBackend`: - Identic cu vLLM dar pointeaza la serverul llama.cpp **src/embeddings/client.py** - `EmbeddingClient`: - `embed(texts, model, backend, dimensions)` - genereaza embeddings - `list_models(backend)`, `list_backends()`, `health_check()` **src/embeddings/api/routes/embeddings.py** - `POST /v1/embeddings`: - Primeste EmbeddingRequest, apeleaza client.embed() - Erori: 400 (backend invalid), 429 (rate limit), 504 (timeout), 503 (conexiune), 500 (eroare generala) Celelalte fisiere (cli.py, logging.py, exceptions.py, middleware.py, dependencies.py, routes/models.py, routes/health.py) sunt structurate identic cu llm-inference, adaptate pentru embeddings. ### Deploy **deploy/docker-compose.yml** - 3 servicii: 1. `embeddings-api` (didiAI-embeddings-api) - FastAPI API. Profile: api, vllm, llamacpp. 2. `vllm-embed` (didiAI-embeddings-vllm) - Image: `vllm/vllm-openai:v0.8.5`. Model: BAAI/bge-m3 (configurabil). Task: embed. gpu-memory-utilization configurable (default 0.50). max-model-len configurable (default 8192). Profile: vllm. 3. `llamacpp-embed` (didiAI-embeddings-llamacpp) - Image: `ghcr.io/ggml-org/llama.cpp:server`. Model GGUF. Mod embedding activat. ctx-size 8192, threads 4, parallel 4. Profile: llamacpp. --- ## 6. Rerank - API de reranking **Locatie:** `modules/rerank/` **Containere:** `didiAI-rerank-api` (port 14200/54200), `didiAI-rerank-vllm` (port 14201/54201), `didiAI-rerank-llamacpp` (port 14210/54210) ### Ce face API compatibil Cohere/Jina pentru reranking documente. Primeste un query si o lista de documente, returneaza documentele sortate dupa relevanta cu scoruri. Doua backend-uri: vLLM (GPU) si llama.cpp (CPU). ### Fisiere sursa Structura identica cu embeddings. Diferente specifice: **src/rerank/types.py** - `RerankUsage(BaseModel)` - total_tokens - `RerankResult(BaseModel)` - index (pozitia originala), relevance_score, document (optional) - `ModelInfo` - id, backend, loaded, max_input_tokens **src/rerank/schemas.py** - `RerankRequest` - model, query (min_length=1), documents (1-1000, fara stringuri goale), top_n (optional), return_documents (bool, default False), backend (override) - `RerankResponse` - id (generat: "rerank-{uuid12}"), model, results (list[RerankResult]), usage, backend **src/rerank/config.py** - `RerankSettings` cu prefix `RERANK_`: - Required: `default_backend`, `enable_vllm`, `enable_llamacpp`, `external_url` - Default-uri: port=54200, vllm_base_url="http://localhost:54201", llamacpp_base_url="http://localhost:54210" - Rate limiting: 20 RPS, burst 50, max 20 concurrent **src/rerank/backends/vllm_backend.py** - `VLLMRerankBackend`: - Foloseste `httpx.AsyncClient` pentru POST la `/rerank` (nu API OpenAI) - `rerank(query, documents, model, top_n)` - trimite cerere, parseaza rezultatele, returneaza `[(index, score), ...]` si usage **src/rerank/backends/llamacpp_backend.py** - `LlamaCppRerankBackend`: - POST la `/rerank`. Gestioneaza field-uri alternative: "relevance_score" sau "score". - Sorteaza descrescator dupa scor, aplica top_n. **src/rerank/client.py** - `RerankClient`: - `rerank(query, documents, model, backend, top_n, return_documents)` - obtine backend, apeleaza rerank, construieste RerankResult-uri - `list_models()`, `list_backends()`, `health_check()` **src/rerank/api/routes/rerank.py** - Doua routere: - `POST /v1/rerank` si `POST /v2/rerank` (alias) - ambele apeleaza `_handle_rerank()` care achizitioneaza slot de concurenta, apeleaza client.rerank() ### Deploy **deploy/docker-compose.yml** - 3 servicii: 1. `rerank-api` (didiAI-rerank-api). Profile: api, vllm, llamacpp. 2. `vllm-rerank` (didiAI-rerank-vllm) - Image: `vllm/vllm-openai:v0.8.5`. Task: score. Model: BAAI/bge-reranker-v2-m3. Profile: vllm. GPU configurable. 300s start_period. 3. `llamacpp-rerank` (didiAI-rerank-llamacpp) - Image: `ghcr.io/ggml-org/llama.cpp:server`. Model GGUF. Mod reranking activat. Profile: llamacpp. --- ## 7. Audio - transcriere audio **Locatie:** `modules/audio/` **Container:** `didiAI-audio` (port 54300) ### Ce face Serviciu speech-to-text folosind faster-whisper (de 4x mai rapid decat Whisper original). API OpenAI-compatibil. Suporta 99+ limbi, detectie automata limba, VAD filtering. ### Fisiere sursa **src/audio/settings.py** - `Settings(BaseSettings)` cu prefix `AUDIO_`: - `model="large-v3-turbo"` - modelul Whisper - `device="cuda"` - cuda sau cpu - `compute_type="int8"` - tip de cuantizare (int8, float16, int8_float16) - `cache_dir="/root/.cache/huggingface"` - `beam_size=5`, `best_of=5`, `temperature=0.0` - `host`, `port=8200`, `log_level`, `external_url` (REQUIRED) - `max_file_size_mb=500` **src/audio/schemas.py** - `TranscriptionSegment` - id, seek, start, end, text, tokens, temperature, avg_logprob, compression_ratio, no_speech_prob - `TranscriptionResponse` - text, language, duration, segments (optional, doar pt verbose_json) - `TranscriptionRequest` - model, language, prompt, response_format ("json"/"text"/"verbose_json"), temperature **src/audio/transcriber.py** Clasa `Transcriber`: - `__init__()` - incarca `WhisperModel` cu model, device, compute_type, cache_dir din settings - `transcribe(audio_path, language, initial_prompt, temperature)` - apeleaza `self.model.transcribe()` cu beam_size, best_of, VAD filter (min_silence 500ms). Colecteaza segmente. Returneaza (text_complet, metadata). - Metadata: language, language_probability, duration, duration_after_vad, all_language_probs, segments Functia `get_transcriber()` - singleton, instantiaza Transcriber la primul apel. **src/audio/app.py** Aplicatie FastAPI "Audio Transcription API". La startup (`startup_event`) incarca modelul Whisper in memorie. Rute: - `GET /health` -> `{"status": "ok"}` - `GET /v1/models` -> lista cu un singur model (cel configurat), format OpenAI-compatibil - `POST /v1/audio/transcriptions` -> endpoint principal de transcriere - Parametri form: file (UploadFile) SAU url (str), model, language, prompt, response_format, temperature - `_get_audio_content(file, url)` - obtine continut audio din upload sau URL. Valideaza dimensiune contra max_file_size_mb. - `_download_url(url)` - descarca audio de la URL cu httpx - Flux: obtine audio -> salveaza in fisier temporar -> transcrie -> formateaza raspuns -> sterge fisier temp - Formate raspuns: "text" (PlainTextResponse), "json" (TranscriptionResponse), "verbose_json" (cu segmente detaliate) - `GET /v1/info` -> metadata pentru catalog (resource, models, functions) ### Deploy **deploy/Dockerfile** - Bazat pe `nvidia/cuda:12.1.0-runtime-ubuntu22.04`. Instaleaza Python 3.10, ffmpeg. Nu foloseste uv, ci pip direct. Port 54300. **deploy/docker-compose.yml** - Serviciu `audio-api`, container `didiAI-audio`. GPU 0 (CUDA_VISIBLE_DEVICES=0). Volum pentru cache modele. Profile: api. Start period 60s. **deploy/deploy.sh** - Valideaza AUDIO_MODEL, AUDIO_DEVICE, AUDIO_CACHE_DIR. Suporta profile `api` si `api-nginx`. --- ## 8. Video Analysis - analiza video **Locatie:** `modules/video-analysis/` **Containere:** `didiAI-video-api` (port 54600), `didiAI-video-vllm-buster` (port 54500) ### Ce face Doua functionalitati: 1. Detectie deepfake - extrage 16 frame-uri uniforme, le trimite la BusterX (model fine-tuned pe Qwen2.5-VL-7B), obtine verdict REAL/FAKE/UNCERTAIN 2. Analiza semantica - divide video-ul in chunk-uri temporale (default 10s), extrage 24 frame-uri/chunk, descrie fiecare chunk cu LLM vision, optional agrega intr-un summary final ### Fisiere sursa **src/video_analysis/settings.py** - `Settings(BaseSettings)` cu prefix `VIDEO_ANALYSIS_`: Required: - `vllm_base_url` - URL server vLLM pt deepfake (ex: http://vllm-buster:8000) - `vllm_model` - nume model (ex: "busterx") - `runs_dir` - director artefacte - `external_url` - URL extern OpenAPI Optional: - `semantic_vllm_base_url`, `semantic_vllm_model` - vLLM separat pt analiza semantica - `frames=16` - nr frame-uri pt sampling uniform (1-64) - `max_side=960` - dimensiune maxima frame (100-2048) - `jpeg_quality=85` - calitate JPEG (1-100) - `max_tokens=750`, `temperature=1e-6`, `repetition_penalty=1.05` - `analysis_prompt` - prompt pt deepfake ("analyze whether...") - `semantic_prompt` - prompt pt descriere chunk - `aggregation_prompt_template` - template pt agregare - `semantic_chunk_duration_s=10.0`, `semantic_frames_per_chunk=24` - `semantic_enable_aggregation=True` - `semantic_aggregation_model="qwen3.5"` - modelul pt agregare (LLM text, nu vision) - `semantic_llm_base_url` - URL LLM text pt agregare Suporta configurare din `deploy/config.yaml` (YAML), cu override din variabile de mediu. **src/video_analysis/video_sampling.py** - Utilitare pentru sampling frame-uri - `get_video_props(cap)` - extrage total_frames, fps, duration_s, width, height din cv2.VideoCapture - `compute_uniform_indices(total, num_frames)` - calculeaza indici uniformi. Formula: `round(i * (total-1) / (num_frames-1))` - `sample_frames_uniform(video_path, num_frames=16)` - deschide video cu OpenCV, selecteaza frame-uri uniform. Daca total_frames necunoscut, citeste pana la 2000 frame-uri si subsampleaza. Returneaza (liste frame-uri, metadata cu timpi si indici) - `sample_frames_chunked(video_path, chunk_duration_s=10.0, frames_per_chunk=24)` - divide video in chunk-uri temporale. Calculeaza nr chunk-uri = ceil(duration/chunk_duration). Pentru fiecare chunk: calculeaza interval temporal, converteste in indici frame, extrage frame-uri (uniform daca chunk > frames_per_chunk). Returneaza (lista de liste de frame-uri, metadata) **src/video_analysis/buster_client.py** - Client vision LLM - `frame_to_data_url_b64jpeg(frame_bgr, max_side, jpeg_quality)` - converteste frame BGR la RGB PIL Image, scaleaza la max_side, encodeaza JPEG, returneaza data URI base64 - `call_vllm_chat(base_url, model, data_urls, prompt, max_tokens, temperature, repetition_penalty, timeout_s=180)` - construieste payload cu imagini + text, POST la `/v1/chat/completions`, masoara timpul. Returneaza (response JSON, elapsed_seconds) - `parse_verdict_and_explanation(model_text)` - verifica primele 20 caractere (uppercase) pt prefix verdict. REAL/FAKE/altceva=UNCERTAIN. **src/video_analysis/schemas.py** - `Verdict = Literal["REAL", "FAKE", "UNCERTAIN"]` - `Usage` - prompt_tokens, completion_tokens, total_tokens - `LatencyS` - sampling_time_s, encode_time_s, model_inference_time_s - `Meta` - fps, total_frames, duration_s, sampled, indices, timestamps_s - `AnalyzeResponse` - request_id (UUID), run_dir, verdict, explanation, usage, latency_s, meta - `ChunkResult` - chunk_idx, time_range, description, frames_analyzed, inference_time_s, usage - `SemanticMeta` - fps, total_frames, duration_s, chunk_duration_s, frames_per_chunk, total_frames_sampled - `SemanticAnalysisResponse` - request_id, run_dir, analysis_type="semantic", video_duration_s, num_chunks, chunk_results, final_summary, aggregation_time_s, total_latency_s, meta **src/video_analysis/app.py** Rute: - `GET /health` -> `{"status": "ok"}` - `POST /analyze/video` -> deepfake detection 1. Genereaza UUID, creeaza run_dir 2. Salveaza video, calculeaza SHA256 3. `sample_frames_uniform()` cu settings.frames 4. `frame_to_data_url_b64jpeg()` pt fiecare frame 5. Salveaza request metadata in JSON 6. `call_vllm_chat()` cu data_urls + analysis_prompt 7. `parse_verdict_and_explanation()` 8. Salveaza result in JSON, returneaza AnalyzeResponse - `POST /analyze/video/semantic` -> analiza semantica 1. Parametri form: file, chunk_duration_s, frames_per_chunk, enable_aggregation 2. Selecteaza semantic vLLM daca configurat, altfel fallback la vLLM principal 3. `sample_frames_chunked()` - divide in chunk-uri 4. Per chunk: encodeaza frame-uri, call vLLM chat, extrage text, creeaza ChunkResult 5. Daca aggregation activat si >1 chunk: construieste prompt cu descrierile chunk-urilor, apeleaza LLM text (semantic_llm_base_url) pt summary final 6. Returneaza SemanticAnalysisResponse - `GET /v1/info` -> metadata catalog Functii helper: `safe_mkdir()`, `write_json()`, `sha256_file()`. ### Deploy **deploy/docker-compose.yml** - 2 servicii: 1. `vllm-buster` (didiAI-video-vllm-buster, port 54500) - Image: `vllm/vllm-openai:qwen3_5`. Model: `l8cv/BusterX_plusplus` (served as "busterx"). GPU 1. max-model-len 32768, gpu-memory-utilization 0.25, prefix caching activat. Profile: api-vllm. 600s start_period. 2. `video-analysis-api` (didiAI-video-api, port 54600) - FastAPI. Volum `../runs` montat la `/app/runs`. Profile: api, api-vllm. **deploy/Dockerfile** - Multi-stage python:3.11-slim cu uv. Port 54600. --- ## 9. Web - cautare web si fact-checking **Locatie:** `modules/web/` **Container:** `didiAI-web-api` (port 51100) ### Ce face Modul complex de fact-checking cu pipeline complet: detectie context -> cautare web (SearXNG) -> extragere continut (HTTP/Playwright/Vision) -> impachetare dovezi (deduplicare, extragere snippete cu LLM, scoring relevanta). Pipeline cu fallback automat si cautare multi-round bazata pe context. ### Fisiere sursa **src/web/config.py** - `WebSettings(BaseSettings)` cu prefix `WEB_`: Required: - `searxng_base_url` - URL SearXNG (ex: http://didiAI-web-searxng:8080) - `llm_base_url` - URL LLM inference server - `external_url` - URL extern OpenAPI Campuri cu default (selectie principala): - `port=51100`, `host="0.0.0.0"` - `vision_model="qwen-vl"`, `text_model="qwen3-235b"` - modele LLM - `llm_api_key`, `openai_api_key`, `anthropic_api_key` - chei API - `fetch_timeout=30`, `fetch_user_agent` - setari HTTP - `browse_timeout=30000`, `browse_viewport_width=1280` - setari Playwright - `vision_max_tokens=2000`, `vision_concurrency=3` - setari Vision - `evidence_max_items=30`, `evidence_dedup_threshold=0.9` - setari Evidence - `rate_limit_rps=10.0`, `rate_limit_burst=20` - `api_tokens: frozenset[str] | None` - auth - `context_detection_enabled=True` - detectie context activata/dezactivata **src/web/exceptions.py** - Exceptii custom: - `WebError`, `AuthenticationError`, `ProviderError`, `ProviderNotAvailableError`, `SearchError`, `RateLimitError`, `WebTimeoutError`, `WebConnectionError` **src/web/orchestrator.py** - Orchestratorul principal Clasa `Orchestrator`: - Proprietati lazy-loaded: `search_client`, `fetch_client`, `browse_client`, `vision_client`, `evidence_packer`, `context_detector` Metoda `gather(request, request_id)`: - Ruleaza pipeline-ul complet cu timeout global - Inregistreaza duratele fiecarui stage Metoda `_run_pipeline(request, request_id, stages)`: Stage 0 - Context Detection (`_run_context_stage`): - Analizeaza claim-ul cu LLM-ul local - Detecteaza tara, limba, entitati, genereaza query-uri optimizate Stage 1 - Search (`_run_search_stage`): - Cautare multi-round (cand contextul e disponibil): - Round 1: surse oficiale + media din tara detectata - Round 2: surse internationale de fact-checking - Round 3: cautare normala nerestrictata - Rezultatele se combina si se deduplica Stage 2 - Fetch (`_run_fetch_stage`): - Lant de fallback: HTTP fetch -> Browse (Playwright) -> Vision (screenshot + LLM) - Conditii de escaladare: - Text extras prea scurt (< `fetch_min_text_length`) - Pagina necesita JavaScript (detectat prin indicatori SPA) - Erori HTTP 401/403 - URL-uri PDF sunt sarite complet Stage 3 - Evidence (`_run_evidence_stage`): - Deduplicare, extragere snippete, scoring relevanta **src/web/search/searxng.py** - Client SearXNG Clasa `SearXNGClient`: - `search(request, request_id)` - executa query-uri in paralel, combina rezultatele - `_search_single(query, ...)` - cautare singura cu rate limiting - `_build_query(query, site_allowlist, site_blocklist)` - adauga filtre de site (format `site:example.com`) - `_execute_request(params, ...)` - cu retry si backoff exponential - `_parse_results(data, query)` - extrage SearchResult din raspunsul SearXNG - `image_search(request, request_id)` - cautare imagini prin SearXNG - `health_check()` - probe /healthz **src/web/fetch/client.py** - Client HTTP Clasa `FetchClient`: - `MAX_PAGE_SIZE = 5MB` - `fetch(request, request_id)` - fetch paralel pe URL-uri - `_fetch_single(url, ...)` - fetch cu extragere continut - `_extract_content(html, url)` - extragere text cu 3 nivele de fallback: 1. readability-lxml (calitate cea mai buna) 2. BeautifulSoup4 (fallback) 3. Regex (ultima sansa) - `_detect_javascript_required(html, text)` - detecteaza pagini JS-heavy: "enable javascript", `