960 lines
49 KiB
Markdown
960 lines
49 KiB
Markdown
# INDEX - Documentatie tehnica completa ml-projects
|
|
|
|
Ultima actualizare: 2026-03-18
|
|
|
|
Acest fisier descrie fiecare modul, fiecare fisier, fiecare clasa, fiecare functie, fiecare ruta, fiecare container si fiecare port din acest monorepo. Nimic nu este omis.
|
|
|
|
---
|
|
|
|
## Cuprins
|
|
|
|
1. [Arhitectura generala](#1-arhitectura-generala)
|
|
2. [Gateway (nginx)](#2-gateway-nginx---punctul-unic-de-intrare)
|
|
3. [Catalog API](#3-catalog-api---agregator-de-servicii)
|
|
4. [LLM Inference](#4-llm-inference---router-llm-unificat)
|
|
5. [Embeddings](#5-embeddings---api-de-embeddings)
|
|
6. [Rerank](#6-rerank---api-de-reranking)
|
|
7. [Audio](#7-audio---transcriere-audio)
|
|
8. [Video Analysis](#8-video-analysis---analiza-video)
|
|
9. [Web](#9-web---cautare-web-si-fact-checking)
|
|
10. [Harta porturilor](#10-harta-completa-a-porturilor)
|
|
11. [Harta GPU](#11-alocare-gpu)
|
|
12. [Retea Docker](#12-retea-docker)
|
|
|
|
---
|
|
|
|
## 1. Arhitectura generala
|
|
|
|
Monorepo cu 8 module independente. Fiecare modul este un pachet Python instalabil cu FastAPI, containerizat in Docker, conectat pe reteaua comuna `didi-network`. Toate comunica prin HTTP intern. Singurul port expus extern este 11000 (gateway nginx).
|
|
|
|
Flux tipic de request extern:
|
|
```
|
|
Client -> Gateway (nginx :11000) -> Serviciu intern (llm/audio/web/catalog)
|
|
```
|
|
|
|
Flux intern (intre servicii):
|
|
```
|
|
Web API -> LLM Inference API -> vLLM (Qwen3.5)
|
|
Video API -> vLLM (BusterX)
|
|
Catalog API -> (interogheaza toate celelalte servicii pe /v1/info)
|
|
```
|
|
|
|
Modele ML servite:
|
|
- Qwen3.5-35B-A3B (text + vision, MoE) - vLLM pe GPU 0
|
|
- BAAI/bge-m3 (embeddings) - vLLM sau llama.cpp
|
|
- BAAI/bge-reranker-v2-m3 (reranking) - vLLM sau llama.cpp
|
|
- Whisper large-v3-turbo (speech-to-text) - faster-whisper pe GPU 0
|
|
- BusterX / Qwen2.5-VL-7B (deepfake detection) - vLLM pe GPU 1
|
|
|
|
---
|
|
|
|
## 2. Gateway (nginx) - punctul unic de intrare
|
|
|
|
**Locatie:** `modules/gateway/`
|
|
**Container:** `didiAI-gateway`
|
|
**Port extern:** 11000
|
|
**Imagine:** `nginx:1.27-alpine`
|
|
|
|
Gateway-ul este un reverse proxy nginx care ruteaza toate request-urile catre serviciile interne. Toate rutele (in afara de /health) necesita autentificare Bearer token.
|
|
|
|
### Fisiere
|
|
|
|
**deploy/nginx.conf.template** - Template nginx cu variabile de mediu
|
|
|
|
Defineste 4 upstream-uri:
|
|
- `llm` -> `didiAI-llm-api:14011`
|
|
- `audio` -> `didiAI-audio-api:54300`
|
|
- `web` -> `didiAI-web-api:51100`
|
|
- `catalog` -> `didiAI-catalog-api:11000`
|
|
|
|
Autentificarea: nginx `map` compara header-ul `Authorization` cu `Bearer ${GATEWAY_API_TOKEN}`. Daca nu coincide, returneaza 401 JSON.
|
|
|
|
Rute:
|
|
- `GET /health` - fara autentificare, returneaza `{"status":"ok"}` direct din nginx
|
|
- `/llm/` -> proxy catre llm upstream, cu SSE streaming (proxy_buffering off, chunked transfer)
|
|
- `/audio/` -> proxy catre audio upstream, cu body buffer 10M pentru upload-uri mari
|
|
- `/web/` -> proxy catre web upstream
|
|
- `/catalog/` -> proxy catre catalog upstream
|
|
- `/` (orice altceva) -> 404 cu lista rutelor disponibile
|
|
|
|
Timeout-uri proxy: connect 60s, send 300s, read 600s.
|
|
Upload maxim: 500MB (client_max_body_size).
|
|
Toate request-urile primesc header X-Request-ID generat de nginx.
|
|
|
|
**deploy/docker-compose.yml** - Un singur serviciu `gateway`
|
|
|
|
Container `didiAI-gateway` pe imaginea `nginx:1.27-alpine`. La pornire, ruleaza `envsubst` care inlocuieste `${GATEWAY_API_TOKEN}` in template si genereaza `nginx.conf` final. Healthcheck cu `wget` pe `/health`.
|
|
|
|
**deploy/deploy.sh** - Script bash
|
|
|
|
Valideaza variabila `GATEWAY_API_TOKEN` (fail-fast). Suporta actiunile `up`, `down`, `logs`. Incarca `.env` din directorul `deploy/`.
|
|
|
|
**deploy/.env.example** - O singura variabila required: `GATEWAY_API_TOKEN`.
|
|
|
|
---
|
|
|
|
## 3. Catalog API - agregator de servicii
|
|
|
|
**Locatie:** `modules/catalog-api/`
|
|
**Container:** `didiAI-catalog-api`
|
|
**Port intern:** 11000 (accesat prin gateway la `/catalog/`)
|
|
|
|
### Ce face
|
|
|
|
Interogheaza periodic sau la cerere endpoint-ul `/v1/info` de pe fiecare serviciu intern (LLM, Audio, Video, Web). Colecteaza metadata (modele disponibile, functii, starea de sanatate) si le expune intr-un singur loc. Genereaza si un OpenAPI spec agregat care combina spec-urile tuturor componentelor.
|
|
|
|
### Fisiere sursa
|
|
|
|
**src/catalog_api/settings.py**
|
|
|
|
Clasa `CatalogSettings(BaseSettings)` cu prefix `CATALOG_`:
|
|
- `external_url: str` (REQUIRED, fara default) - URL extern pentru OpenAPI
|
|
- `host`, `port`, `log_level` - setari server (cu default-uri)
|
|
- `llm_url`, `audio_url`, `video_url`, `web_url` - URL-uri interne Docker ale serviciilor
|
|
- `llm_external_port`, `audio_external_port`, `video_external_port`, `web_external_port` - porturi externe
|
|
|
|
Clasa `Component(BaseSettings)`:
|
|
- `id`, `url`, `external_url`, `timeout` - metadata despre un serviciu
|
|
|
|
Metoda `get_components() -> list[Component]` - construieste lista componentelor configurate, omitand cele cu URL gol (ex: video_url="" dezactiveaza video).
|
|
|
|
Instanta globala `settings = CatalogSettings()`.
|
|
|
|
**src/catalog_api/app.py**
|
|
|
|
Aplicatia FastAPI principala. Constanta `AGGREGATED_OPENAPI_VERSION = "0.1.0"`.
|
|
|
|
Rute:
|
|
|
|
- `GET /health` -> `health()` - returneaza `{"status": "ok"}`
|
|
|
|
- `GET /v1/components` -> `list_components()` - face fetch async la `/v1/info` pe fiecare component din `settings.get_components()`. Adauga `component_id` si `base_url` la fiecare raspuns. Returneaza `{components: [...], total: N, errors: [...]}`. Gestioneaza timeout-uri si erori HTTP per component.
|
|
|
|
- `GET /v1/components/{component_id}` -> `get_component(component_id)` - cauta componenta dupa ID, face fetch la `/v1/info`, returneaza metadata. 404 daca ID-ul nu exista.
|
|
|
|
- `GET /v1/models` -> `list_models()` - colecteaza array-ul `models` din `/v1/info` al fiecarei componente. Adauga `component_id` si `component_name` la fiecare model. Ignora silentios componentele cu erori.
|
|
|
|
- `GET /v1/functions` -> `list_functions()` - la fel ca models, dar extrage array-ul `functions`.
|
|
|
|
- `GET /v1/status` -> `get_status()` - verifica conectivitatea cu fiecare componenta. Statusul general: "healthy" (toate ok), "degraded" (unele ok), "unhealthy" (niciuna). Returneaza `{status, components: [...], healthy_count, total_count}`.
|
|
|
|
- `GET /v1/openapi` -> `get_aggregated_openapi()` - face fetch la `/openapi.json` de pe fiecare componenta. Combina spec-urile intr-un singur OpenAPI 3.1.0 cu titlul "didiAI - Aggregated ML Services API". Prefixeaza path-urile cu `/{component_id}` si schema-urile cu `{component_id}_` pentru a evita coliziuni.
|
|
|
|
- `GET /v1/docs` -> `get_aggregated_swagger_ui()` - pagina HTML cu Swagger UI care incarca `/v1/openapi`.
|
|
|
|
- `GET /v1/redoc` -> `get_aggregated_redoc()` - pagina HTML cu ReDoc.
|
|
|
|
- `GET /v1/openapi/component/{component_id}` -> `get_component_openapi(component_id)` - returneaza OpenAPI spec-ul brut al unei componente specifice.
|
|
|
|
Functii helper:
|
|
- `_merge_openapi_schemas(base_spec, component_spec, component_id, component_url)` - combina spec-ul unei componente in spec-ul agregat. Prefixeaza path-urile, schema-urile, operationId-urile.
|
|
- `_update_refs(obj, component_id)` - actualizeaza recursiv referintele `$ref` din obiectele OpenAPI.
|
|
|
|
La final, `uvicorn.run()` porneste serverul.
|
|
|
|
### Deploy
|
|
|
|
**deploy/Dockerfile** - Multi-stage build pe python:3.11-slim cu uv. Port default 11000. CMD: `uvicorn catalog_api.app:app`.
|
|
|
|
**deploy/docker-compose.yml** - Serviciu `catalog-api`, container `didiAI-catalog-api`, port 11000, retea `didi-network`. Healthcheck pe `/health`.
|
|
|
|
---
|
|
|
|
## 4. LLM Inference - router LLM unificat
|
|
|
|
**Locatie:** `modules/llm-inference/`
|
|
**Containere:** `didiAI-llm-api` (port 14011), `didiAI-vllm-qwen3.5` (port 14001)
|
|
|
|
### Ce face
|
|
|
|
Gateway unificat pentru inferenta LLM. Primeste cereri OpenAI-compatibile si le ruteaza catre unul din 3 backend-uri: LiteLLM (100+ provideri cloud), vLLM (GPU local), llama.cpp (CPU local). Suporta streaming SSE, retry cu backoff exponential, rate limiting, concurrency limiting, autentificare Bearer, load balancing pentru llama.cpp.
|
|
|
|
### Fisiere sursa
|
|
|
|
**src/llm_inference/types.py** - Tipuri de baza
|
|
|
|
- `BackendType(str, Enum)` - LITELLM, VLLM, LLAMACPP
|
|
- `ChatMessage(BaseModel)` - role (system/user/assistant/function/tool), content (str sau list multimodal), name optional
|
|
- `Usage(BaseModel)` - prompt_tokens, completion_tokens, total_tokens
|
|
- `Choice(BaseModel)` - index, message, finish_reason (raspuns non-streaming)
|
|
- `Delta(BaseModel)` - role, content (raspuns streaming)
|
|
- `StreamChoice(BaseModel)` - index, delta, finish_reason
|
|
- `ModelInfo(BaseModel)` - id, backend, loaded, context_length, capabilities
|
|
|
|
**src/llm_inference/schemas.py** - Scheme API
|
|
|
|
- `CompletionRequest(BaseModel)` - messages (1-1000), model, temperature (0-2, default 0.7), max_tokens, stream (bool), backend (override optional), top_p, frequency_penalty, presence_penalty, stop. Validator: non-assistant messages trebuie sa aiba content.
|
|
- `CompletionResponse(BaseModel)` - id, object="chat.completion", created (unix timestamp), model, choices, usage, backend
|
|
- `CompletionChunk(BaseModel)` - id, object="chat.completion.chunk", created, model, choices (StreamChoice)
|
|
- `ModelListResponse`, `ModelLoadRequest`, `ModelLoadResponse` - management modele
|
|
- `BackendHealth`, `HealthResponse`, `ReadinessResponse`, `BackendListResponse` - monitoring
|
|
|
|
**src/llm_inference/config.py** - Configurare Pydantic Settings
|
|
|
|
Clasa `LLMSettings(BaseSettings)` cu prefix `LLM_`:
|
|
|
|
Campuri REQUIRED (fara default):
|
|
- `default_backend: Literal["litellm", "vllm", "llamacpp"]`
|
|
- `enable_vllm: bool`
|
|
- `enable_llamacpp: bool`
|
|
- `external_url: str`
|
|
|
|
Campuri cu default:
|
|
- `host="0.0.0.0"`, `port=14011`
|
|
- `default_model="gpt-3.5-turbo"`
|
|
- `openrouter_api_key`, `openai_api_key`, `anthropic_api_key` - chei API optionale
|
|
- `vllm_base_url="http://localhost:14001"`, `vllm_api_key`
|
|
- `llamacpp_base_url="http://localhost:8102"`, `llamacpp_base_urls` (lista, comma-separated, pt load balancing)
|
|
- `llamacpp_health_check_interval=30` (interval verificare sanatate servere)
|
|
- `models_dir="/models"`
|
|
- `request_timeout=120.0`, `connect_timeout=10.0`
|
|
- `max_retries=3`, `retry_min_wait=1.0`, `retry_max_wait=60.0`
|
|
- `rate_limit_rps=10.0`, `rate_limit_burst=20`
|
|
- `max_concurrent_completions=10`
|
|
- `api_tokens: frozenset[str] | None` - tokeni Bearer (comma-separated)
|
|
- `log_level="INFO"`, `log_json=False`
|
|
|
|
Proprietati: `auth_enabled`, `llamacpp_urls`.
|
|
Validatori: `parse_llamacpp_base_urls()` (parseaza string comma-separated in lista), `parse_api_tokens()`, `validate_api_keys()` (avertizeaza daca litellm fara chei API).
|
|
|
|
Clasa `SettingsCache` - singleton thread-safe cu `get()`, `clear()`, `set()`.
|
|
|
|
**src/llm_inference/exceptions.py** - Exceptii custom
|
|
|
|
Baza: `LLMInferenceError(Exception)`. Derivate:
|
|
- `AuthenticationError` - autentificare esuata
|
|
- `BackendNotAvailableError(backend, reason)` - backend indisponibil
|
|
- `BackendNotEnabledError(backend)` - backend neactivat in config
|
|
- `ModelNotFoundError(model, backend)` - model negasit
|
|
- `CompletionError(message, backend, model)` - eroare la generare
|
|
- `ModelLoadError(model, backend, reason)` - eroare la incarcare model
|
|
- `ModelListError(backend, reason)` - eroare la listare modele
|
|
- `LLMRateLimitError(message, backend, retry_after)` - rate limit atins
|
|
- `LLMTimeoutError(message, backend, timeout)` - timeout
|
|
- `LLMConnectionError(backend, reason)` - conexiune esuata
|
|
|
|
**src/llm_inference/retry.py** - Logica de retry
|
|
|
|
Importa conditionat exceptiile din `litellm`, `openai`, `httpx`. Defineste doua tupluri globale:
|
|
- `RETRYABLE_EXCEPTIONS` - RateLimitError, Timeout, ServiceUnavailableError, TimeoutException, ConnectError
|
|
- `NON_RETRYABLE_EXCEPTIONS` - AuthenticationError, BadRequestError, NotFoundError
|
|
|
|
Functii:
|
|
- `is_retryable_exception(exc)` - verifica daca exceptia e retryable
|
|
- `extract_retry_after(exc)` - extrage headerul Retry-After din exceptii de rate limit
|
|
- `translate_exception(exc, backend, model)` - traduce exceptii provider-specifice in exceptii custom
|
|
- `retry_with_backoff(func, max_retries, min_wait, max_wait, backend, model)` - executa functie async cu retry exponential. Formula: `min_wait * 2^attempt`, capped la max_wait, cu jitter 0-25%. Respecta Retry-After headers.
|
|
|
|
**src/llm_inference/logging.py** - Logging structurat
|
|
|
|
- `request_id_ctx: ContextVar[str | None]` - propagare request ID prin context
|
|
- `RequestIdFilter(logging.Filter)` - adauga request_id la log records
|
|
- `JsonFormatter(logging.Formatter)` - formatare JSON cu timestamp, level, logger, message, request_id, exceptie
|
|
- `configure_logging(level, json_format)` - configureaza root logger "llm_inference"
|
|
- `get_logger(name)` - returneaza logger cu prefix "llm_inference."
|
|
- `set_request_id(request_id)` / `get_request_id()` - management context
|
|
|
|
**src/llm_inference/utils.py**
|
|
|
|
- `safe_close_stream(stream, logger)` - inchide sigur un stream async (incearca aclose(), fallback pe close())
|
|
|
|
**src/llm_inference/image_processing.py** - Procesare imagini multimodale
|
|
|
|
Client HTTP global partajat `_http_client` pentru download.
|
|
|
|
Functii:
|
|
- `_get_http_client()` - returneaza/creeaza clientul HTTP partajat
|
|
- `_guess_mime_type(url, content_type)` - determina MIME type din URL sau Content-Type
|
|
- `_download_and_encode(url)` - descarca imagine si returneaza ca data URI base64
|
|
- `_process_content_item(item)` - proceseaza un element de continut (descarca imagini HTTP, lasa base64 si non-HTTP neschimbate)
|
|
- `_process_text_with_urls(text)` - detecteaza URL-uri de imagini in text plain si le converteste in format multimodal `[{type: "text"}, {type: "image_url"}]`
|
|
- `process_messages(messages: list[ChatMessage])` - proceseaza toate mesajele, descarcand URL-urile de imagini si convertindu-le in base64
|
|
|
|
**src/llm_inference/client.py** - Client de nivel inalt
|
|
|
|
Clasa `LLMClient`:
|
|
- `__init__(settings)` - initializeaza cu settings si BackendRegistry
|
|
- `complete(messages, model, backend, **kwargs)` - generare chat completion. Proceseaza imagini, rezolva backend-ul, apeleaza backend.complete()
|
|
- `stream(messages, model, backend, **kwargs)` - generare streaming. Yield-uieste CompletionChunk
|
|
- `list_models(backend)` - listeaza modele de la un backend sau toate
|
|
- `load_model(model, backend)` - incarca model pe backend local
|
|
- `unload_model(model, backend)` - descarca model
|
|
- `list_backends()` - listeaza tipurile de backend disponibile
|
|
- `health_check()` - returneaza starea de sanatate per backend
|
|
- `_resolve_backend_for_model(model)` - interogheaza fiecare backend local sa vada care serveste modelul
|
|
- `_parse_messages(messages)` - converteste dict-uri in ChatMessage
|
|
|
|
**src/llm_inference/cli.py** - Punct de intrare CLI
|
|
|
|
Clasa `GracefulShutdown`:
|
|
- Inregistreaza handlere SIGTERM/SIGINT
|
|
- Primul semnal: shutdown graceful
|
|
- Al doilea semnal: exit fortat
|
|
|
|
Functia `main()`:
|
|
- Argumente: `--host`, `--port`, `--workers`, `--reload`, `--graceful-timeout`
|
|
- Porneste uvicorn cu factory mode `llm_inference.api.app:create_app`
|
|
|
|
### Backend-uri
|
|
|
|
**src/llm_inference/backends/base.py** - Clasa abstracta `LLMBackend`
|
|
|
|
Metode abstracte: `name` (property), `complete()`, `stream()`, `list_models()`
|
|
Metode concrete (cu default): `load_model()` (NotImplementedError), `unload_model()` (NotImplementedError), `health_check()` (True)
|
|
|
|
**src/llm_inference/backends/registry.py** - Registru de backend-uri
|
|
|
|
Clasa `BackendRegistry`:
|
|
- `__init__(settings)` - initializeaza backend-urile activate (LiteLLM mereu, vLLM/llama.cpp optional)
|
|
- `get(backend_type)` - returneaza instanta backend (default daca None)
|
|
- `list_backends()` - lista tipurilor disponibile
|
|
- `is_available(backend_type)` - verifica disponibilitatea
|
|
|
|
**src/llm_inference/backends/litellm_backend.py** - Backend LiteLLM
|
|
|
|
Clasa `LiteLLMBackend(LLMBackend)`:
|
|
- Interfata cu 100+ provideri cloud (OpenAI, Anthropic, OpenRouter, Azure, Google, AWS)
|
|
- `_configure_litellm()` - seteaza cheile API pe modulul litellm
|
|
- `complete()` - apeleaza `litellm.acompletion()` cu retry_with_backoff
|
|
- `stream()` - streaming prin `litellm.acompletion(stream=True)`, yield-uieste CompletionChunk
|
|
- `list_models()` - cache TTL 3600s. Fetcheaza dinamic de la OpenAI si OpenRouter API, fallback pe liste curate (hardcoded)
|
|
- `health_check()` - verifica conectivitatea la cel putin un provider
|
|
- Liste fallback: OpenAI (gpt-4o, gpt-4o-mini, gpt-4-turbo, gpt-3.5-turbo), Anthropic (Claude 3.5 Sonnet, 3 Opus, 3 Haiku)
|
|
|
|
**src/llm_inference/backends/vllm_backend.py** - Backend vLLM
|
|
|
|
Clasa `VLLMBackend(LLMBackend)`:
|
|
- Conectare la server vLLM prin API OpenAI-compatible (pachetul `openai`)
|
|
- Creeaza `AsyncOpenAI` client care pointeaza la `vllm_base_url`
|
|
- `complete()` - `client.chat.completions.create()` cu retry
|
|
- `stream()` - streaming prin acelasi client, cu safe_close_stream la erori
|
|
- `list_models()` - `client.models.list()`
|
|
- `health_check()` - incearca `models.list()`, True daca reuseste
|
|
|
|
**src/llm_inference/backends/llamacpp_backend.py** - Backend llama.cpp cu load balancing
|
|
|
|
Dataclass `LlamaCppServer`:
|
|
- `url`, `healthy`, `request_count`, `error_count`, `last_check`, `response_times` (ultimele 50)
|
|
- Property: `avg_response_ms`, `short_name`
|
|
|
|
Clasa `LlamaCppBackend(LLMBackend)`:
|
|
- Creeaza obiecte `LlamaCppServer` si clienti `AsyncOpenAI` pentru fiecare URL din configurare
|
|
- Round-robin cu failover automat
|
|
- `_get_server()` - returneaza urmatorul server sanatos (round-robin). Daca toate sunt nesanatoase, incearca pe toate.
|
|
- `_mark_unhealthy(server)` / `_mark_healthy(server)` - actualizeaza starea serverului
|
|
- `_periodic_health_check(interval)` - task async care verifica periodic toate serverele
|
|
- `_complete_on_server(server, client, messages, model)` - executa completare pe un server specific, masoara timp de raspuns
|
|
- `complete()` - incearca servere sanatoase round-robin, failover la urmatorul pe erori de conexiune/timeout
|
|
- `stream()` - streaming cu failover
|
|
- `list_models()` - interogheaza primul server sanatos
|
|
- `health_check()` - True daca orice server e disponibil
|
|
- `get_servers_status()` - returneaza starea tuturor serverelor (URL, health, request_count, error_count, avg_response_ms)
|
|
|
|
### API (FastAPI)
|
|
|
|
**src/llm_inference/api/app.py** - Factory aplicatie
|
|
|
|
Functia `create_app()`:
|
|
- Creeaza FastAPI cu titlu "LLM Inference API", versiune 0.1.0
|
|
- Lifespan manager: la startup configureaza logging, initializeaza ConcurrencyLimiter si LLMClient. La shutdown logheaza.
|
|
- Middleware (ordinea conteaza, primul adaugat = cel mai exterior):
|
|
1. RateLimitMiddleware - token bucket, exclude /health si /ready
|
|
2. RequestIdMiddleware - genereaza/extrage X-Request-ID
|
|
- Routere montate: health, completions (/v1), models (/v1), info (/v1)
|
|
|
|
**src/llm_inference/api/dependencies.py** - Dependinte FastAPI
|
|
|
|
- `get_client(request)` - returneaza LLMClient din app.state
|
|
- `get_settings(request)` - returneaza settings din app.state
|
|
- `verify_bearer_token(request, authorization)` - valideaza token Bearer. Comparatie constant-time cu `hmac.compare_digest`. 401 daca invalid.
|
|
|
|
Clasa `ConcurrencyLimiter`:
|
|
- Limiteaza numarul de completari concurente cu asyncio.Semaphore
|
|
- `acquire(blocking=False)` - context manager async. Non-blocking (default): 503 imediat daca nu sunt sloturi. Blocking: asteapta slot.
|
|
- `current_count`, `available` - proprietati de monitorizare
|
|
|
|
Functii globale: `init_concurrency_limiter()`, `get_concurrency_limiter()`, `require_completion_slot()` (dependinta FastAPI).
|
|
|
|
**src/llm_inference/api/middleware.py** - Middleware
|
|
|
|
Clasa `RequestIdMiddleware(BaseHTTPMiddleware)`:
|
|
- Extrage X-Request-ID din header sau genereaza UUID
|
|
- Seteaza in context (pt logging), in request.state, si in response headers
|
|
|
|
Clasa `TokenBucket`:
|
|
- Algoritm token bucket pentru rate limiting
|
|
- `acquire()` - incearca sa consume un token. Returneaza True/False.
|
|
- `retry_after()` - secunde pana cand un token e disponibil.
|
|
|
|
Clasa `RateLimitMiddleware(BaseHTTPMiddleware)`:
|
|
- Foloseste TokenBucket. Exclude /health si /ready.
|
|
- Returneaza 429 Too Many Requests cu header Retry-After cand limita e atinsa.
|
|
- ATENTIE: rate limiting per-proces, nu distribuit. Fiecare replica are limita proprie.
|
|
|
|
**src/llm_inference/api/routes/completions.py** - Ruta de completari
|
|
|
|
Endpoint `POST /v1/chat/completions`:
|
|
- Necesita Bearer token (daca auth activat)
|
|
- Pentru streaming: creeaza generator SSE care yield-uieste chunks JSON, tine slot de concurenta pe intreaga durata stream-ului, ping la fiecare 15s
|
|
- Pentru non-streaming: achizitioneaza slot, ruleaza completare, elibereaza slot
|
|
- Erori: BackendNotAvailableError/BackendNotEnabledError -> 400, CompletionError -> 500
|
|
|
|
Functii helper:
|
|
- `_stream_generator(client, request)` - generator SSE, trimite `[DONE]` la final, eroare ca eveniment SSE
|
|
- `_stream_with_slot(generator, limiter)` - wrapper care tine slot-ul pe durata stream-ului
|
|
- `_build_completion_kwargs(request)` - construieste kwargs din CompletionRequest
|
|
|
|
**src/llm_inference/api/routes/models.py** - Management modele
|
|
|
|
- `GET /v1/models` - query param optional `backend` pentru filtrare
|
|
- `POST /v1/models/load` - incarca model pe backend local (body: ModelLoadRequest)
|
|
- `POST /v1/models/unload` - descarca model
|
|
- `GET /v1/backends` - listeaza backend-urile disponibile
|
|
|
|
**src/llm_inference/api/routes/health.py** - Health checks
|
|
|
|
- `GET /health` - status per-backend, status general (healthy/degraded/unhealthy)
|
|
- `GET /ready` - probe Kubernetes, verifica backend-ul default
|
|
|
|
**src/llm_inference/api/routes/info.py** - Informatii component
|
|
|
|
- `GET /v1/info` - returneaza metadata completa pentru catalog: resource (name, slug, config, auth, rate_limits, tags), models (cu capabilities, provider, endpoint), functions (Chat Completions, List Models, List Backends, Load Model, Unload Model cu input/output schema)
|
|
|
|
### Deploy
|
|
|
|
**deploy/docker-compose.yml** - Defineste 3 servicii:
|
|
|
|
1. `llm-api` (didiAI-llm-api, port 14011) - API-ul FastAPI. Profile: api, vllm. Variabile: backend config, chei API, concurrency.
|
|
2. `vllm-qwen3.5` (didiAI-vllm-qwen3.5, port 14001) - Server vLLM cu Qwen/Qwen3.5-35B-A3B. Profile: vllm. GPU 0. Image: `vllm/vllm-openai:qwen3_5`. Parametri: max-model-len 32000, gpu-memory-utilization 0.65, enable-prefix-caching, enable-auto-tool-choice (Hermes parser). Healthcheck cu 600s start_period (modelul se incarca lent).
|
|
3. `llamacpp` (optional) - Image: `ghcr.io/ggml-org/llama.cpp:server`. Profile: llamacpp, full. GGUF model din MODELS_DIR. ctx-size 4096.
|
|
|
|
**deploy/Dockerfile** - Multi-stage: python:3.11-slim cu uv. Port intern 14011. CMD: `python -m llm_inference.cli`.
|
|
|
|
---
|
|
|
|
## 5. Embeddings - API de embeddings
|
|
|
|
**Locatie:** `modules/embeddings/`
|
|
**Containere:** `didiAI-embeddings-api` (port 14100/54100), `didiAI-embeddings-vllm` (port 14101/54101), `didiAI-embeddings-llamacpp` (port 14110/54110)
|
|
|
|
### Ce face
|
|
|
|
API OpenAI-compatibil de embeddings cu suport pentru doua backend-uri: vLLM (GPU) si llama.cpp (CPU/GGUF). Modelul principal: BAAI/bge-m3 (max 8192 tokeni).
|
|
|
|
### Fisiere sursa
|
|
|
|
Structura e identica cu llm-inference (acelasi tipar arhitectural). Diferentele principale:
|
|
|
|
**src/embeddings/types.py**
|
|
- `BackendType(str, Enum)` - VLLM, LLAMACPP (fara LITELLM)
|
|
- `EmbeddingUsage(BaseModel)` - prompt_tokens, total_tokens
|
|
- `EmbeddingData(BaseModel)` - object="embedding", index, embedding (list[float])
|
|
- `ModelInfo(BaseModel)` - id, backend, loaded, dimensions, max_input_tokens
|
|
|
|
**src/embeddings/schemas.py**
|
|
- `EmbeddingRequest` - input (list[str] sau str), model, encoding_format ("float"/"base64"), dimensions (optional), backend (override). Validatori: ensure_list() converteste str in list, validate_input() verifica ca input-ul nu e gol.
|
|
- `EmbeddingResponse` - object="list", data (list[EmbeddingData]), model, usage, backend
|
|
- `encode_embedding_base64(embedding)` - encodeaza vector embedding ca base64 (little-endian floats)
|
|
|
|
**src/embeddings/config.py** - `EmbeddingSettings` cu prefix `EMB_`:
|
|
- Required: `default_backend`, `enable_vllm`, `enable_llamacpp`, `external_url`
|
|
- Default-uri: port=54100, vllm_base_url="http://localhost:54101", llamacpp_base_url="http://localhost:54110"
|
|
- Rate limiting: 20 RPS, burst 40, max 20 concurrent
|
|
|
|
**src/embeddings/backends/base.py** - `EmbeddingBackend(ABC)`:
|
|
- `embed(texts, model, dimensions) -> tuple[list[list[float]], EmbeddingUsage]`
|
|
- `list_models()`, `health_check()`
|
|
|
|
**src/embeddings/backends/vllm_backend.py** - `VLLMEmbeddingBackend`:
|
|
- Foloseste `AsyncOpenAI` client catre serverul vLLM
|
|
- `embed()` - `client.embeddings.create()`, returneaza vectori si usage
|
|
|
|
**src/embeddings/backends/llamacpp_backend.py** - `LlamaCppEmbeddingBackend`:
|
|
- Identic cu vLLM dar pointeaza la serverul llama.cpp
|
|
|
|
**src/embeddings/client.py** - `EmbeddingClient`:
|
|
- `embed(texts, model, backend, dimensions)` - genereaza embeddings
|
|
- `list_models(backend)`, `list_backends()`, `health_check()`
|
|
|
|
**src/embeddings/api/routes/embeddings.py** - `POST /v1/embeddings`:
|
|
- Primeste EmbeddingRequest, apeleaza client.embed()
|
|
- Erori: 400 (backend invalid), 429 (rate limit), 504 (timeout), 503 (conexiune), 500 (eroare generala)
|
|
|
|
Celelalte fisiere (cli.py, logging.py, exceptions.py, middleware.py, dependencies.py, routes/models.py, routes/health.py) sunt structurate identic cu llm-inference, adaptate pentru embeddings.
|
|
|
|
### Deploy
|
|
|
|
**deploy/docker-compose.yml** - 3 servicii:
|
|
|
|
1. `embeddings-api` (didiAI-embeddings-api) - FastAPI API. Profile: api, vllm, llamacpp.
|
|
2. `vllm-embed` (didiAI-embeddings-vllm) - Image: `vllm/vllm-openai:v0.8.5`. Model: BAAI/bge-m3 (configurabil). Task: embed. gpu-memory-utilization configurable (default 0.50). max-model-len configurable (default 8192). Profile: vllm.
|
|
3. `llamacpp-embed` (didiAI-embeddings-llamacpp) - Image: `ghcr.io/ggml-org/llama.cpp:server`. Model GGUF. Mod embedding activat. ctx-size 8192, threads 4, parallel 4. Profile: llamacpp.
|
|
|
|
---
|
|
|
|
## 6. Rerank - API de reranking
|
|
|
|
**Locatie:** `modules/rerank/`
|
|
**Containere:** `didiAI-rerank-api` (port 14200/54200), `didiAI-rerank-vllm` (port 14201/54201), `didiAI-rerank-llamacpp` (port 14210/54210)
|
|
|
|
### Ce face
|
|
|
|
API compatibil Cohere/Jina pentru reranking documente. Primeste un query si o lista de documente, returneaza documentele sortate dupa relevanta cu scoruri. Doua backend-uri: vLLM (GPU) si llama.cpp (CPU).
|
|
|
|
### Fisiere sursa
|
|
|
|
Structura identica cu embeddings. Diferente specifice:
|
|
|
|
**src/rerank/types.py**
|
|
- `RerankUsage(BaseModel)` - total_tokens
|
|
- `RerankResult(BaseModel)` - index (pozitia originala), relevance_score, document (optional)
|
|
- `ModelInfo` - id, backend, loaded, max_input_tokens
|
|
|
|
**src/rerank/schemas.py**
|
|
- `RerankRequest` - model, query (min_length=1), documents (1-1000, fara stringuri goale), top_n (optional), return_documents (bool, default False), backend (override)
|
|
- `RerankResponse` - id (generat: "rerank-{uuid12}"), model, results (list[RerankResult]), usage, backend
|
|
|
|
**src/rerank/config.py** - `RerankSettings` cu prefix `RERANK_`:
|
|
- Required: `default_backend`, `enable_vllm`, `enable_llamacpp`, `external_url`
|
|
- Default-uri: port=54200, vllm_base_url="http://localhost:54201", llamacpp_base_url="http://localhost:54210"
|
|
- Rate limiting: 20 RPS, burst 50, max 20 concurrent
|
|
|
|
**src/rerank/backends/vllm_backend.py** - `VLLMRerankBackend`:
|
|
- Foloseste `httpx.AsyncClient` pentru POST la `/rerank` (nu API OpenAI)
|
|
- `rerank(query, documents, model, top_n)` - trimite cerere, parseaza rezultatele, returneaza `[(index, score), ...]` si usage
|
|
|
|
**src/rerank/backends/llamacpp_backend.py** - `LlamaCppRerankBackend`:
|
|
- POST la `/rerank`. Gestioneaza field-uri alternative: "relevance_score" sau "score".
|
|
- Sorteaza descrescator dupa scor, aplica top_n.
|
|
|
|
**src/rerank/client.py** - `RerankClient`:
|
|
- `rerank(query, documents, model, backend, top_n, return_documents)` - obtine backend, apeleaza rerank, construieste RerankResult-uri
|
|
- `list_models()`, `list_backends()`, `health_check()`
|
|
|
|
**src/rerank/api/routes/rerank.py** - Doua routere:
|
|
- `POST /v1/rerank` si `POST /v2/rerank` (alias) - ambele apeleaza `_handle_rerank()` care achizitioneaza slot de concurenta, apeleaza client.rerank()
|
|
|
|
### Deploy
|
|
|
|
**deploy/docker-compose.yml** - 3 servicii:
|
|
|
|
1. `rerank-api` (didiAI-rerank-api). Profile: api, vllm, llamacpp.
|
|
2. `vllm-rerank` (didiAI-rerank-vllm) - Image: `vllm/vllm-openai:v0.8.5`. Task: score. Model: BAAI/bge-reranker-v2-m3. Profile: vllm. GPU configurable. 300s start_period.
|
|
3. `llamacpp-rerank` (didiAI-rerank-llamacpp) - Image: `ghcr.io/ggml-org/llama.cpp:server`. Model GGUF. Mod reranking activat. Profile: llamacpp.
|
|
|
|
---
|
|
|
|
## 7. Audio - transcriere audio
|
|
|
|
**Locatie:** `modules/audio/`
|
|
**Container:** `didiAI-audio-api` (port 54300)
|
|
|
|
### Ce face
|
|
|
|
Serviciu speech-to-text folosind faster-whisper (de 4x mai rapid decat Whisper original). API OpenAI-compatibil. Suporta 99+ limbi, detectie automata limba, VAD filtering.
|
|
|
|
### Fisiere sursa
|
|
|
|
**src/audio/settings.py** - `Settings(BaseSettings)` cu prefix `AUDIO_`:
|
|
- `model="large-v3-turbo"` - modelul Whisper
|
|
- `device="cuda"` - cuda sau cpu
|
|
- `compute_type="int8"` - tip de cuantizare (int8, float16, int8_float16)
|
|
- `cache_dir="/root/.cache/huggingface"`
|
|
- `beam_size=5`, `best_of=5`, `temperature=0.0`
|
|
- `host`, `port=8200`, `log_level`, `external_url` (REQUIRED)
|
|
- `max_file_size_mb=500`
|
|
|
|
**src/audio/schemas.py**
|
|
- `TranscriptionSegment` - id, seek, start, end, text, tokens, temperature, avg_logprob, compression_ratio, no_speech_prob
|
|
- `TranscriptionResponse` - text, language, duration, segments (optional, doar pt verbose_json)
|
|
- `TranscriptionRequest` - model, language, prompt, response_format ("json"/"text"/"verbose_json"), temperature
|
|
|
|
**src/audio/transcriber.py**
|
|
|
|
Clasa `Transcriber`:
|
|
- `__init__()` - incarca `WhisperModel` cu model, device, compute_type, cache_dir din settings
|
|
- `transcribe(audio_path, language, initial_prompt, temperature)` - apeleaza `self.model.transcribe()` cu beam_size, best_of, VAD filter (min_silence 500ms). Colecteaza segmente. Returneaza (text_complet, metadata).
|
|
- Metadata: language, language_probability, duration, duration_after_vad, all_language_probs, segments
|
|
|
|
Functia `get_transcriber()` - singleton, instantiaza Transcriber la primul apel.
|
|
|
|
**src/audio/app.py**
|
|
|
|
Aplicatie FastAPI "Audio Transcription API".
|
|
|
|
La startup (`startup_event`) incarca modelul Whisper in memorie.
|
|
|
|
Rute:
|
|
- `GET /health` -> `{"status": "ok"}`
|
|
- `GET /v1/models` -> lista cu un singur model (cel configurat), format OpenAI-compatibil
|
|
- `POST /v1/audio/transcriptions` -> endpoint principal de transcriere
|
|
- Parametri form: file (UploadFile) SAU url (str), model, language, prompt, response_format, temperature
|
|
- `_get_audio_content(file, url)` - obtine continut audio din upload sau URL. Valideaza dimensiune contra max_file_size_mb.
|
|
- `_download_url(url)` - descarca audio de la URL cu httpx
|
|
- Flux: obtine audio -> salveaza in fisier temporar -> transcrie -> formateaza raspuns -> sterge fisier temp
|
|
- Formate raspuns: "text" (PlainTextResponse), "json" (TranscriptionResponse), "verbose_json" (cu segmente detaliate)
|
|
- `GET /v1/info` -> metadata pentru catalog (resource, models, functions)
|
|
|
|
### Deploy
|
|
|
|
**deploy/Dockerfile** - Bazat pe `nvidia/cuda:12.1.0-runtime-ubuntu22.04`. Instaleaza Python 3.10, ffmpeg. Nu foloseste uv, ci pip direct. Port 54300.
|
|
|
|
**deploy/docker-compose.yml** - Serviciu `audio-api`, container `didiAI-audio-api`. GPU 0 (CUDA_VISIBLE_DEVICES=0). Volum pentru cache modele. Profile: api. Start period 60s.
|
|
|
|
**deploy/deploy.sh** - Valideaza AUDIO_MODEL, AUDIO_DEVICE, AUDIO_CACHE_DIR. Suporta profile `api` si `api-nginx`.
|
|
|
|
---
|
|
|
|
## 8. Video Analysis - analiza video
|
|
|
|
**Locatie:** `modules/video-analysis/`
|
|
**Containere:** `didiAI-video-api` (port 54600), `didiAI-video-vllm-buster` (port 54500)
|
|
|
|
### Ce face
|
|
|
|
Doua functionalitati:
|
|
1. Detectie deepfake - extrage 16 frame-uri uniforme, le trimite la BusterX (model fine-tuned pe Qwen2.5-VL-7B), obtine verdict REAL/FAKE/INCONCLUSIVE
|
|
2. Analiza semantica - divide video-ul in chunk-uri temporale (default 10s), extrage 24 frame-uri/chunk, descrie fiecare chunk cu LLM vision, optional agrega intr-un summary final
|
|
|
|
### Fisiere sursa
|
|
|
|
**src/video_analysis/settings.py** - `Settings(BaseSettings)` cu prefix `VIDEO_ANALYSIS_`:
|
|
|
|
Required:
|
|
- `vllm_base_url` - URL server vLLM pt deepfake (ex: http://vllm-buster:8000)
|
|
- `vllm_model` - nume model (ex: "busterx")
|
|
- `runs_dir` - director artefacte
|
|
- `external_url` - URL extern OpenAPI
|
|
|
|
Optional:
|
|
- `semantic_vllm_base_url`, `semantic_vllm_model` - vLLM separat pt analiza semantica
|
|
- `frames=16` - nr frame-uri pt sampling uniform (1-64)
|
|
- `max_side=960` - dimensiune maxima frame (100-2048)
|
|
- `jpeg_quality=85` - calitate JPEG (1-100)
|
|
- `max_tokens=750`, `temperature=1e-6`, `repetition_penalty=1.05`
|
|
- `analysis_prompt` - prompt pt deepfake ("analyze whether...")
|
|
- `semantic_prompt` - prompt pt descriere chunk
|
|
- `aggregation_prompt_template` - template pt agregare
|
|
- `semantic_chunk_duration_s=10.0`, `semantic_frames_per_chunk=24`
|
|
- `semantic_enable_aggregation=True`
|
|
- `semantic_aggregation_model="qwen3.5"` - modelul pt agregare (LLM text, nu vision)
|
|
- `semantic_llm_base_url` - URL LLM text pt agregare
|
|
|
|
Suporta configurare din `deploy/config.yaml` (YAML), cu override din variabile de mediu.
|
|
|
|
**src/video_analysis/video_sampling.py** - Utilitare pentru sampling frame-uri
|
|
|
|
- `get_video_props(cap)` - extrage total_frames, fps, duration_s, width, height din cv2.VideoCapture
|
|
- `compute_uniform_indices(total, num_frames)` - calculeaza indici uniformi. Formula: `round(i * (total-1) / (num_frames-1))`
|
|
- `sample_frames_uniform(video_path, num_frames=16)` - deschide video cu OpenCV, selecteaza frame-uri uniform. Daca total_frames necunoscut, citeste pana la 2000 frame-uri si subsampleaza. Returneaza (liste frame-uri, metadata cu timpi si indici)
|
|
- `sample_frames_chunked(video_path, chunk_duration_s=10.0, frames_per_chunk=24)` - divide video in chunk-uri temporale. Calculeaza nr chunk-uri = ceil(duration/chunk_duration). Pentru fiecare chunk: calculeaza interval temporal, converteste in indici frame, extrage frame-uri (uniform daca chunk > frames_per_chunk). Returneaza (lista de liste de frame-uri, metadata)
|
|
|
|
**src/video_analysis/buster_client.py** - Client vision LLM
|
|
|
|
- `frame_to_data_url_b64jpeg(frame_bgr, max_side, jpeg_quality)` - converteste frame BGR la RGB PIL Image, scaleaza la max_side, encodeaza JPEG, returneaza data URI base64
|
|
- `call_vllm_chat(base_url, model, data_urls, prompt, max_tokens, temperature, repetition_penalty, timeout_s=180)` - construieste payload cu imagini + text, POST la `/v1/chat/completions`, masoara timpul. Returneaza (response JSON, elapsed_seconds)
|
|
- `parse_verdict_and_explanation(model_text)` - verifica primele 20 caractere (uppercase) pt prefix verdict. REAL/FAKE/altceva=INCONCLUSIVE.
|
|
|
|
**src/video_analysis/schemas.py**
|
|
- `Verdict = Literal["REAL", "FAKE", "INCONCLUSIVE"]`
|
|
- `Usage` - prompt_tokens, completion_tokens, total_tokens
|
|
- `LatencyS` - sampling_time_s, encode_time_s, model_inference_time_s
|
|
- `Meta` - fps, total_frames, duration_s, sampled, indices, timestamps_s
|
|
- `AnalyzeResponse` - request_id (UUID), run_dir, verdict, explanation, usage, latency_s, meta
|
|
- `ChunkResult` - chunk_idx, time_range, description, frames_analyzed, inference_time_s, usage
|
|
- `SemanticMeta` - fps, total_frames, duration_s, chunk_duration_s, frames_per_chunk, total_frames_sampled
|
|
- `SemanticAnalysisResponse` - request_id, run_dir, analysis_type="semantic", video_duration_s, num_chunks, chunk_results, final_summary, aggregation_time_s, total_latency_s, meta
|
|
|
|
**src/video_analysis/app.py**
|
|
|
|
Rute:
|
|
- `GET /health` -> `{"status": "ok"}`
|
|
|
|
- `POST /analyze/video` -> deepfake detection
|
|
1. Genereaza UUID, creeaza run_dir
|
|
2. Salveaza video, calculeaza SHA256
|
|
3. `sample_frames_uniform()` cu settings.frames
|
|
4. `frame_to_data_url_b64jpeg()` pt fiecare frame
|
|
5. Salveaza request metadata in JSON
|
|
6. `call_vllm_chat()` cu data_urls + analysis_prompt
|
|
7. `parse_verdict_and_explanation()`
|
|
8. Salveaza result in JSON, returneaza AnalyzeResponse
|
|
|
|
- `POST /analyze/video/semantic` -> analiza semantica
|
|
1. Parametri form: file, chunk_duration_s, frames_per_chunk, enable_aggregation
|
|
2. Selecteaza semantic vLLM daca configurat, altfel fallback la vLLM principal
|
|
3. `sample_frames_chunked()` - divide in chunk-uri
|
|
4. Per chunk: encodeaza frame-uri, call vLLM chat, extrage text, creeaza ChunkResult
|
|
5. Daca aggregation activat si >1 chunk: construieste prompt cu descrierile chunk-urilor, apeleaza LLM text (semantic_llm_base_url) pt summary final
|
|
6. Returneaza SemanticAnalysisResponse
|
|
|
|
- `GET /v1/info` -> metadata catalog
|
|
|
|
Functii helper: `safe_mkdir()`, `write_json()`, `sha256_file()`.
|
|
|
|
### Deploy
|
|
|
|
**deploy/docker-compose.yml** - 2 servicii:
|
|
|
|
1. `vllm-buster` (didiAI-video-vllm-buster, port 54500) - Image: `vllm/vllm-openai:latest`. Model: `l8cv/BusterX_plusplus` (served as "busterx"). GPU 1. max-model-len 32768, gpu-memory-utilization 0.25, prefix caching activat. Profile: api-vllm. 600s start_period.
|
|
2. `video-analysis-api` (didiAI-video-api, port 54600) - FastAPI. Volum `../runs` montat la `/app/runs`. Profile: api, api-vllm.
|
|
|
|
**deploy/Dockerfile** - Multi-stage python:3.11-slim cu uv. Port 54600.
|
|
|
|
---
|
|
|
|
## 9. Web - cautare web si fact-checking
|
|
|
|
**Locatie:** `modules/web/`
|
|
**Container:** `didiAI-web-api` (port 51100)
|
|
|
|
### Ce face
|
|
|
|
Modul complex de fact-checking cu pipeline complet: detectie context -> cautare web (SearXNG) -> extragere continut (HTTP/Playwright/Vision) -> impachetare dovezi (deduplicare, extragere snippete cu LLM, scoring relevanta). Pipeline cu fallback automat si cautare multi-round bazata pe context.
|
|
|
|
### Fisiere sursa
|
|
|
|
**src/web/config.py** - `WebSettings(BaseSettings)` cu prefix `WEB_`:
|
|
|
|
Required:
|
|
- `searxng_base_url` - URL SearXNG (ex: http://localhost:55100)
|
|
- `llm_base_url` - URL LLM inference server
|
|
- `external_url` - URL extern OpenAPI
|
|
|
|
Campuri cu default (selectie principala):
|
|
- `port=51100`, `host="0.0.0.0"`
|
|
- `vision_model="qwen-vl"`, `text_model="qwen3-235b"` - modele LLM
|
|
- `llm_api_key`, `openai_api_key`, `anthropic_api_key` - chei API
|
|
- `fetch_timeout=30`, `fetch_user_agent` - setari HTTP
|
|
- `browse_timeout=30000`, `browse_viewport_width=1280` - setari Playwright
|
|
- `vision_max_tokens=2000`, `vision_concurrency=3` - setari Vision
|
|
- `evidence_max_items=30`, `evidence_dedup_threshold=0.9` - setari Evidence
|
|
- `rate_limit_rps=10.0`, `rate_limit_burst=20`
|
|
- `api_tokens: frozenset[str] | None` - auth
|
|
- `context_detection_enabled=True` - detectie context activata/dezactivata
|
|
|
|
**src/web/exceptions.py** - Exceptii custom:
|
|
- `WebError`, `AuthenticationError`, `ProviderError`, `ProviderNotAvailableError`, `SearchError`, `RateLimitError`, `WebTimeoutError`, `WebConnectionError`
|
|
|
|
**src/web/orchestrator.py** - Orchestratorul principal
|
|
|
|
Clasa `Orchestrator`:
|
|
- Proprietati lazy-loaded: `search_client`, `fetch_client`, `browse_client`, `vision_client`, `evidence_packer`, `context_detector`
|
|
|
|
Metoda `gather(request, request_id)`:
|
|
- Ruleaza pipeline-ul complet cu timeout global
|
|
- Inregistreaza duratele fiecarui stage
|
|
|
|
Metoda `_run_pipeline(request, request_id, stages)`:
|
|
|
|
Stage 0 - Context Detection (`_run_context_stage`):
|
|
- Analizeaza claim-ul cu LLM-ul local
|
|
- Detecteaza tara, limba, entitati, genereaza query-uri optimizate
|
|
|
|
Stage 1 - Search (`_run_search_stage`):
|
|
- Cautare multi-round (cand contextul e disponibil):
|
|
- Round 1: surse oficiale + media din tara detectata
|
|
- Round 2: surse internationale de fact-checking
|
|
- Round 3: cautare normala nerestrictata
|
|
- Rezultatele se combina si se deduplica
|
|
|
|
Stage 2 - Fetch (`_run_fetch_stage`):
|
|
- Lant de fallback: HTTP fetch -> Browse (Playwright) -> Vision (screenshot + LLM)
|
|
- Conditii de escaladare:
|
|
- Text extras prea scurt (< `fetch_min_text_length`)
|
|
- Pagina necesita JavaScript (detectat prin indicatori SPA)
|
|
- Erori HTTP 401/403
|
|
- URL-uri PDF sunt sarite complet
|
|
|
|
Stage 3 - Evidence (`_run_evidence_stage`):
|
|
- Deduplicare, extragere snippete, scoring relevanta
|
|
|
|
**src/web/search/searxng.py** - Client SearXNG
|
|
|
|
Clasa `SearXNGClient`:
|
|
- `search(request, request_id)` - executa query-uri in paralel, combina rezultatele
|
|
- `_search_single(query, ...)` - cautare singura cu rate limiting
|
|
- `_build_query(query, site_allowlist, site_blocklist)` - adauga filtre de site (format `site:example.com`)
|
|
- `_execute_request(params, ...)` - cu retry si backoff exponential
|
|
- `_parse_results(data, query)` - extrage SearchResult din raspunsul SearXNG
|
|
- `image_search(request, request_id)` - cautare imagini prin SearXNG
|
|
- `health_check()` - probe /healthz
|
|
|
|
**src/web/fetch/client.py** - Client HTTP
|
|
|
|
Clasa `FetchClient`:
|
|
- `MAX_PAGE_SIZE = 5MB`
|
|
- `fetch(request, request_id)` - fetch paralel pe URL-uri
|
|
- `_fetch_single(url, ...)` - fetch cu extragere continut
|
|
- `_extract_content(html, url)` - extragere text cu 3 nivele de fallback:
|
|
1. readability-lxml (calitate cea mai buna)
|
|
2. BeautifulSoup4 (fallback)
|
|
3. Regex (ultima sansa)
|
|
- `_detect_javascript_required(html, text)` - detecteaza pagini JS-heavy: "enable javascript", `<noscript>`, indicatori SPA (react-root, ng-app, __next)
|
|
|
|
**src/web/browse/client.py** - Client Playwright
|
|
|
|
Clasa `BrowseClient`:
|
|
- `browse(request, request_id)` - navigare paralela cu browser headless
|
|
- `_browse_single(url, ...)` - navigare cu asteptare continut dinamic (networkidle, selectori custom), screenshot optional
|
|
- `_extract_content(page)` - manipulare DOM + extragere text
|
|
- Suport: data publicare, URL canonic, screenshot base64
|
|
|
|
**src/web/vision/client.py** - Client Vision LLM
|
|
|
|
Clasa `VisionClient`:
|
|
- Foloseste `LLMProviderChain` pt fallback provider (local -> OpenAI -> Anthropic)
|
|
- `extract(request, request_id)` - extragere paralela
|
|
- `_extract_single(url, ...)` - screenshot + apel vision LLM
|
|
- `_call_vision_llm(messages, provider, ...)` - apeleaza modelul vision cu imagini
|
|
- `_extract_images(page)` - analizeaza imaginile de pe pagina
|
|
|
|
**src/web/evidence/packer.py** - Impachetare dovezi
|
|
|
|
Clasa `EvidencePacker`:
|
|
|
|
Algoritm de deduplicare:
|
|
- SimHash fingerprinting (64-bit) pentru comparare rapida O(n)
|
|
- Distanta Hamming ca prag de candidati
|
|
- SequenceMatcher pentru comparare precisa
|
|
- Multi-nivel: hash exact SHA256, SimHash, similaritate precisa (ratio lungime, prefix/sufix, shingles pt texte lungi)
|
|
|
|
Metode:
|
|
- `pack(request, request_id)` - pipeline complet: deduplicare -> extragere snippete -> scoring
|
|
- `_deduplicate(pages)` - deduplicare pe baza de SimHash
|
|
- `_create_evidence_item(page, claim)` - creeaza EvidenceItem din PageContent
|
|
- `_extract_snippet_llm(text, claim)` - extrage snippet relevant cu LLM
|
|
- `_extract_snippet_and_score_llm(text, claim)` - snippet + scor relevanta intr-un singur apel
|
|
- `_score_relevance(text, claim)` - scoring relevanta cu LLM
|
|
- `_score_credibility_simple(url)` - scoring credibilitate pe baza de domeniu (Reuters, BBC, Nature etc. primesc scor mare)
|
|
- `_extract_snippets_batch(pages, claim)` - grupuri de 2-3 pagini per apel LLM
|
|
- `_summarize_batch(pages, claim)` - sumarizare in batch
|
|
|
|
Tratament special: elimina taguri `<think>` din output-ul modelelor de reasoning.
|
|
Circuit breaker: cache negativ de 60s daca LLM-ul nu e disponibil.
|
|
|
|
**src/web/llm/provider.py** - Lant de provideri LLM
|
|
|
|
Clasa `LLMProviderChain`:
|
|
- 3 provideri in ordine: local (vLLM), OpenAI, Anthropic
|
|
- `call_chat(messages, provider, max_tokens, temperature)` - apeleaza providerul specificat
|
|
- `_call_local(messages, ...)` - apel HTTP direct la vLLM-ul local
|
|
- `_call_openai(messages, ...)` - API OpenAI
|
|
- `_call_anthropic(messages, ...)` - API Anthropic (converteste formatul mesajelor)
|
|
- Suport multimodal: data URI-uri si URL-uri de imagini
|
|
|
|
**src/web/context/detector.py** - Detectie context
|
|
|
|
Clasa `ContextDetector`:
|
|
- `detect(claim)` - analizeaza claim-ul inainte de cautare
|
|
- `_detect_with_llm(claim)` - apeleaza LLM local pt a extrage: tara principala (ISO 3166-1), limba, entitati (persoane, institutii, locatii), query-uri optimizate de cautare
|
|
- `_is_llm_available()` - verifica disponibilitatea LLM-ului cu probe la `/v1/models`, cache negativ 60s
|
|
- Fallback: returneaza SearchContext gol daca LLM indisponibil
|
|
|
|
**src/web/context/sources.py** - Surse pe tari
|
|
|
|
Surse predefinite per tara:
|
|
- RO (Romania): gov.ro, cdep.ro, senat.ro, digi24.ro, hotnews.ro etc.
|
|
- US (SUA): whitehouse.gov, congress.gov, nytimes.com, apnews.com etc.
|
|
- Surse fact-check internationale: Reuters, Snopes, PolitiFact, FactCheck.org, FullFact, BBC, AFP, Veridica.ro
|
|
|
|
**src/web/validation.py** - Validare URL (protectie SSRF)
|
|
- `validate_url(url)` - verificari sincrone: schema (http/https), hostname blocklist (localhost, metadata.google.internal, 169.254.169.254), IP-uri private
|
|
- `validate_url_dns(url)` - verificare DNS async
|
|
- `validate_urls_async(urls)` - validare DNS in paralel
|
|
- Blocate: 127.0.0.0/8, 10.0.0.0/8, 192.168.0.0/16, 172.16.0.0/12
|
|
|
|
### Scheme (schemas/)
|
|
|
|
- `common.py` - FailedUrl, PageImage, PageContent (url, title, text, hash, extraction_method, timestamps), ProviderHealth, HealthResponse, ErrorDetail, ErrorResponse
|
|
- `search.py` - SearchResult (query, url, title, snippet, rank, site, published_at), SearchRequest (queries, max_results, site_allowlist/blocklist, language, country, freshness, safe_search), SearchResponse
|
|
- `fetch.py` - FetchRequest (urls, auto_fallback, method, min_text_length, parallel_fetches), FetchPageResult (status_code, content_type, needs_fallback), FetchResponse
|
|
- `browse.py` - BrowseRequest (urls, wait, timeout, screenshot), BrowsePageResult (final_url, screenshot_base64, viewport), BrowseResponse
|
|
- `vision.py` - ImageContext, VisionExtractRequest (urls, context_query, model, screenshots, provider), VisionPageResult (extracted_text, images, tokens_used), VisionExtractResponse
|
|
- `evidence.py` - EvidenceItem (url, title, publisher, snippet, summary, full_text, relevance_score, credibility_score, provenance), EvidencePackRequest (pages, claim, dedup, LLM flags, limits), EvidenceStats, EvidencePackResponse
|
|
- `gather.py` - GatherRequest (claim 10-1000 chars, search/fetch/evidence options, context detection, timeout), GatherStageResult (stage, success, counts, duration, error), GatherResponse (evidence, stats, search_results, stages, execution_time)
|
|
- `image_search.py` - ImageSearchResult (image_url, thumbnail, source_url, title, dimensions), ImageSearchRequest, ImageSearchResponse
|
|
- `context.py` - EntitySet (persons, institutions, locations), SearchContext (primary_country, entities, detected_language, search_queries)
|
|
|
|
### API
|
|
|
|
Rute:
|
|
- `GET /health` - starea serviciului
|
|
- `GET /ready` - readiness probe
|
|
- `POST /v1/search` - cautare web prin SearXNG
|
|
- `POST /v1/image-search` - cautare imagini
|
|
- `POST /v1/fetch` - fetch HTTP cu extragere continut
|
|
- `POST /v1/gather` - pipeline complet de fact-checking (endpointul principal)
|
|
- `GET /v1/info` - metadata catalog
|
|
|
|
### Deploy
|
|
|
|
**deploy/docker-compose.yml** - Serviciu `didiAI-web-api`, port 51100. SHM 2GB (pt Playwright browsers). Profile: api. Depinde de SearXNG si llm-inference.
|
|
|
|
**deploy/Dockerfile** - Multi-stage cu python:3.11-slim. Instaleaza browsere Playwright. Creeaza user non-root. Port 51100.
|
|
|
|
---
|
|
|
|
## 10. Harta completa a porturilor
|
|
|
|
```
|
|
PRODUCTION (1xxxx):
|
|
11000 Gateway (nginx) - singurul port expus extern
|
|
14001 vLLM Qwen3.5-35B-A3B - server LLM text+vision
|
|
14011 LLM Inference API - router LLM unificat
|
|
14100 Embeddings API - API embeddings
|
|
14101 vLLM Embed Server - backend GPU embeddings
|
|
14110 llama.cpp Embed Server - backend CPU embeddings
|
|
14200 Rerank API - API reranking
|
|
14201 vLLM Rerank Server - backend GPU reranking
|
|
14210 llama.cpp Rerank Server - backend CPU reranking
|
|
|
|
DEVELOPMENT (5xxxx):
|
|
51100 Web API - fact-checking + cautare web
|
|
54100 Embeddings API Dev
|
|
54101 vLLM Embed Server Dev
|
|
54110 llama.cpp Embed Server Dev
|
|
54200 Rerank API Dev
|
|
54201 vLLM Rerank Server Dev
|
|
54210 llama.cpp Rerank Server Dev
|
|
54300 Audio API - transcriere Whisper
|
|
54500 BusterX vLLM - server vision deepfake
|
|
54600 Video Analysis API - analiza video
|
|
```
|
|
|
|
Schema porturi: 5 cifre. Prima cifra: 1=prod, 5=dev. A doua cifra: 1=API/Gateway, 4=LLM/AI.
|
|
|
|
---
|
|
|
|
## 11. Alocare GPU
|
|
|
|
| GPU | Ce ruleaza | VRAM folosit | VRAM total |
|
|
|-----|-----------|-------------|------------|
|
|
| GPU 0 | Qwen3.5-35B-A3B (~57GB) + Whisper large-v3-turbo (~2GB) | ~59GB | 143GB |
|
|
| GPU 1 | BusterX / Qwen2.5-VL-7B (~22GB) | ~22GB | 143GB |
|
|
|
|
---
|
|
|
|
## 12. Retea Docker
|
|
|
|
Toate containerele sunt pe reteaua externa `didi-network`. Comunicarea interna se face prin DNS Docker (nume containere):
|
|
|
|
```
|
|
didiAI-gateway -> didiAI-llm-api, didiAI-audio-api, didiAI-web-api, didiAI-catalog-api
|
|
didiAI-llm-api -> didiAI-vllm-qwen3.5
|
|
didiAI-catalog-api -> didiAI-llm-api, didiAI-audio-api, didiAI-video-api, didiAI-web-api
|
|
didiAI-web-api -> SearXNG, didiAI-llm-api
|
|
didiAI-video-api -> didiAI-video-vllm-buster, didiAI-llm-api (pt agregare semantica)
|
|
didiAI-embeddings-api -> didiAI-embeddings-vllm, didiAI-embeddings-llamacpp
|
|
didiAI-rerank-api -> didiAI-rerank-vllm, didiAI-rerank-llamacpp
|
|
```
|
|
|
|
Naming convention containere: `didiAI-{modul}-{serviciu}`.
|
|
|
|
## Recent Changes (2026-05-05)
|
|
|
|
- **Login Keycloak SSO functional la `/admin-ai/`**: realm `didi-admins` (mutat din `didi-clients`), client `ai-platform-dashboard` (creat in didi-admins ca clona), required role `admin`. SSO comun cu admin-dashboard backend (1 login = ambele dashboard-uri).
|
|
- **AI dashboard env**: `VITE_KEYCLOAK_URL=https://sso.clossers.com`, `VITE_KEYCLOAK_REALM=didi-admins`, `VITE_KEYCLOAK_CLIENT_ID=ai-platform-dashboard`, `VITE_KEYCLOAK_REQUIRED_ROLE=admin`. Dual var pentru build (VITE_*) + runtime (DASHBOARD_*).
|
|
- **Schema config DB-overridable**: tabel nou `config_schema_override` (auto-creat la startup), helper `_merged_schema(session)` in `routes/config.py`, endpoint-uri admin `GET /api/config/schema/_overrides`, `PUT /api/config/schema/{key}`, `DELETE /api/config/schema/{key}`. Audit trail (action `config.schema.upsert/delete/seed`).
|
|
- **Migrare automata 98 chei -> DB**: la primul startup, `seed_schema_if_empty()` populeaza tabelul din `KNOWN_KEYS` (idempotent). Codul KNOWN_KEYS ramane fallback daca DB e sters. DB = single source of truth pentru schema acum.
|
|
- **didi_brain endpoint nou**: `GET /v1/fact_status/due_for_recheck?limit=N&volatility=X` (facts cu next_check_at <= now, nelocked). Plus rate-limit pe `POST /v1/cache/invalidate` (10/h per actor, dry-run free).
|