# Agent V3 - Index Serviciul principal al platformei DIDI. Analizeaza continut (text, URL, imagine, audio, video) pentru detectarea dezinformarii. Produce scoruri 0-100 pe 4 componente si un verdict agregat. **Port**: 24803 **Framework**: Express 5 + TypeScript **Container**: didi-agent-v3 --- ## Infrastructura data-layer Data layer rulează **local pe mașina de deployment**, ca containere Docker pe `didi-network`. Nu există cluster extern — serviciile se adresează prin **numele de container Docker**, nu prin IP de mașină. Config real din env-ul `didi-agent-v3`: | Serviciu | Container (target) | Config | Credentials | |---|---|---|---| | **PostgreSQL 17** | `didi-postgres:5432` | DB `DIDI`, schema `bos_analysis` | user `bos_interface` / `interface` | | **Redis 7** | `didi-cache:6379` | DB 0 | parolă `redis123` | | **RabbitMQ 3.12** | `staging-dataLayer-rabbitmq:5672` | vhost `/` | user `admin` / `rabbitmq123` | **Clienți ioredis centralizați** via `src/shared/redis/connection.ts` — `createRedisConnection(label)`. Zero `new Redis({...})` inline. Auto-reconnect + reconnectOnError. **RabbitMQ client** via `src/queue/connection.ts` + `src/shared/queue/constants.ts`. Vhost `/` (`encodeURIComponent()` la construirea URI-ului AMQP). **Bootstrap auto-populate**: la startup, didi-framework detectează dacă `didi:framework:manifest` lipsește în Redis și auto-rulează `POST /api/sync-redis`. Zero intervenție manuală pe fresh deploy. --- ## Integrare cu platforma AI (Lot 1) Backend-ul (Lot 2) **apelează** serviciile platformei AI (Lot 1) prin HTTP — nu implementează niciunul intern. Fiecare are un URL **configurabil din env**, cu **fail-open**: orice eroare (timeout, 5xx, network, feature flag off) → analiza continuă fără serviciul respectiv (fallback la calea locală sau skip). Cum funcționează intern brain/RAG, atom cache, extractorii, forensic, buster etc. e documentat în platforma AI (Lot 1); aici descriem doar contractul de integrare. | Serviciu Lot 1 | Env (default) | Folosit pentru | Fail-open | |---|---|---|---| | didi_brain | `DIDI_BRAIN_URL` (`http://10.11.10.12:8090`) | Verification cache + analysis atom cache pentru claims/techniques/ai-tampered | → LLM direct | | extractors | `EXTRACTORS_URL` (`http://:54400`) | Extragere conținut/metadata | → skip | | video analysis (buster) | `VIDEO_ANALYSIS_URL` (`http://:54600`) | Analiză video | → procesare locală | | forensic | `FORENSIC_API_URL` (`http://forensic-features-api:8080`) | Forensic features imagine | → skip | | whisper (audio) | `M17_WHISPER_URL` (`http://10.11.10.17:54300`) | Transcriere audio | → Groq → OpenAI | | web search | `M17_WEB_API_URL` (`http://10.11.10.13:51100`) | Căutare web claims/source | fallback când brain MISS | | LLM router | `LLM_ROUTER_URL` (`http://10.11.10.17:14011`) | Inferență LLM primară (Qwen local) | → OpenRouter | | vision | `VISION_LLM_URL` (`http://10.11.10.17:14011`) | OCR + AI detection imagine | → Gemini / GPT-4o | | domain check | `DOMAIN_CHECK_API_URL` (`http://:11000`) | WHOIS/DNS/SSL/blacklist domeniu | → skip | IP-urile GPU `10.11.10.17:14011` (LLM/vision) și `10.11.10.17:54300` (whisper) sunt **default**, configurabile din env. **Client brain (cod backend, `src/shared/brain/client.ts`)**: `gatherFromBrain()` (verification cache lookup + evidence pentru claims), `writeVerificationCacheAsync()`, `lookupAnalysisAtom()` / `writeAnalysisAtomAsync()` / `patchAnalysisAtomGold()` (analysis atom cache pentru techniques/ai-tampered), plus helpers de hashing. Detaliul acestei integrări backend e la secțiunea „Funcționalități suplimentare". --- ## Tier-aware execution (premium vs free) Agent-v3 ruleaza chain-uri diferite de modele LLM in functie de **planul de abonament** al utilizatorului. ### Cum se determina tier-ul 1. Fiecare request face `checkCredits()` la didiFramework, care returneaza `planType` (1-6) in raspuns 2. `getSearchTier(planType)` din `src/shared/credits.ts` mapeaza: - `plan_type` 1-3 → tier `'free'` - `plan_type` 4-6 → tier `'premium'` 3. `tier` (variabila `searchTier` in cod) e propagat prin toate straturile: - **Sync routes**: `pipeline-routes.ts`, `ai-tampered-routes.ts` → `PipelineInput.searchTier` - **Async via queue**: `component-worker.ts` deriva din `taskMessage.planType` - `ComponentRunner.runAll()` → `AnalysisInput.searchTier` - Fiecare executor primeste `tier` ca parametru la `execute()` - `verdict-explanation.ts`: primeste `tier` la `generate()` - `vision.ts` / `callVision()`: primeste `tier` - `transcription.ts` / `transcribe()`: primeste `tier` in options - M17 search (claims/source-assessment): trimite `X-Search-Tier: premium` header ### Ce se schimba per tier | Component/Stage | Free primary | Premium primary | |-----------------|--------------|-----------------| | Techniques screening | Qwen 3.5 397B local | Gemini 3 Flash Preview (OR) | | Techniques deep | Qwen 3.5 397B local | Claude Sonnet 4.6 (OR) | | AI-Tampered screening | Qwen 3.5 397B local | Gemini 3 Flash Preview (OR) | | AI-Tampered deep | Qwen 3.5 397B local | Claude Sonnet 4.6 (OR) | | Claims extraction | Qwen 3.5 397B local | Gemini 3 Flash Preview (OR) | | Claims verification | Qwen 3.5 397B local | Claude Sonnet 4.6 (OR) | | Source extraction | Qwen 3.5 397B local | Gemini 3 Flash Preview (OR) | | Source evaluation | Qwen 3.5 397B local | Claude Sonnet 4.6 (OR) | | Verdict reviewer | Qwen 3.5 397B local | Gemini 3 Flash Preview (OR) | | Vision (OCR + AI detect) | Qwen Vision local | Gemini 3 Flash Preview (OR) | | Transcription (audio) | M17-Whisper local | Groq-Whisper | | Web search (claims/source) | M17 SearXNG | M17 + Brave + Tavily (via X-Search-Tier header) | Fallback chain pentru premium termina pe Qwen local (safety net, 0 cost) — zero erori de rutare daca cloud crapa. ### Cum se incarca stage assignments tier-based Executoarele citesc cheia Redis `didi:config:{component}:v1:stage_assignments` cu structura **tier-nested**: ```json { "techniques_screening": { "free": { "models": [{ order:1, model_key: "qwen35:Qwen3.5-397B-A17B", ...}, ...] }, "premium": { "models": [{ order:1, model_key: "openrouter:google/gemini-3-flash-preview", ...}, ...] } }, "techniques_deep": { "free": {...}, "premium": {...} } } ``` Helper `resolveStageAssignment(assignments, stageCode, tier)` returneaza chain-ul corect, cu **fallback automat la 'free'** daca 'premium' lipseste. Asta protejeaza utilizatorii premium daca configul nu e complet. --- ## Componente de analiza Serviciul ruleaza 4 componente independente + un calculator de verdict: | Componenta | Ce face | Scor produs | |----------------|--------------------------------------------------------|--------------------| | Techniques | Detecteaza tehnici de manipulare (propaganda, apel emotional, etc.) | manipulation_score 0-100 | | AI-Tampered | Detecteaza continut generat/modificat de AI | ai_probability 0-100 | | Claims | Extrage afirmatii si le verifica prin cautare web | credibility_score 0-100 | | Domain | Analizeaza domeniul sursa (varsta, SSL, blacklist, reputatie) | trust_score 0-100 | | Verdict | Agrega scorurile componentelor intr-un scor final de risc | risk_score 0-100 | Fiecare componenta (in afara de Domain) foloseste un pipeline in 2 etape: - **Screening**: analiza rapida cu un model LLM rapid - **Deep Analysis**: analiza detaliata pe fiecare dimensiune/categorie detectata Fiecare etapa are 3 nivele de fallback (primary -> fallback_1 -> fallback_2 -> fallback_3). --- ## Structura fisierelor ``` src/ index.ts -- Server Express, middleware, montare rute api/ -- Rute organizate pe subdirectoare (fiecare *-routes.ts la root e shim de re-export) routes.ts / techniques.ts -- Endpoint-uri Techniques (+ /health general) ai-tampered/ -- Endpoint-uri AI-Tampered claims/ -- Endpoint-uri Claims (analyze, config, results) source-assessment/ -- Endpoint-uri Source Assessment (config, models, stage-assignments, analyze) domain.ts -- Endpoint-uri Domain (legacy) media.ts -- Endpoint-uri Media/Fisiere (upload/download MinIO) pipeline/ -- Pipeline complet + Istoric + Extensie + dry-run.ts + cancel.ts (cancel/resume) + async.ts moderation/ -- HIL Moderation (queue.ts, flag-stats.ts, _gold-promotion.ts, _middleware.ts) components/ component-runner.ts -- Orchestrator direct (invocare executori fara HTTP intern) techniques/ executor.ts -- Executor Techniques: screening -> deep analysis -> scoring (params din Redis) ai-tampered/ executor.ts -- Executor AI-Tampered: disclosure check -> screening -> deep -> scoring (params din Redis) claims/ executor.ts -- Executor Claims: extractie afirmatii -> verificare web -> scoring (params din Redis) source-assessment/ executor.ts -- Executor Source Assessment: extractie sursa -> M17 search -> evaluare (params din Redis) moderation/ triage.ts -- Decide daca o sesiune intra in coada review (config Redis, feature-flag `triage_enabled`) queue-manager.ts -- Operatii PG pe `bos_analysis.moderation_queue` (enqueue/list/get/claim/resolve/stats) pipeline/ executor.ts -- Orchestrator pipeline: ruleaza toate componentele + verdict verdict-calculator.ts -- Calcul verdict cu input profiles per tip input (params din Redis) verdict-explanation.ts -- Generare explicatie bilingva (RO+EN) via LLM virality-calculator.ts -- Calcul scor viralitate (factori: emotie, urgenta, reach) types.ts -- Tipuri interne pipeline shared/ media/ transcription.ts -- Transcriere audio: M17-Whisper -> Groq -> OpenAI (fallback chain) vision.ts -- Analiza imagine: Qwen Local -> Gemini Flash -> GPT-4o (fallback chain) video-processor.ts -- Pipeline video: download -> extrage cadre -> transcriere + viziune media-service.ts -- Interfata unificata media (download, format, cleanup) persistence/ persist-service.ts -- Coordonare Redis (cache) + PostgreSQL (permanent) pg-adapter.ts -- Adaptor PostgreSQL: save/load/history/delete pe schema bos_analysis pg-pool.ts -- Pool conexiuni PostgreSQL (singleton) redis/ connection.ts -- Factory centralizat ioredis (ACL ready, retry, reconnectOnError) keys.ts -- Registru central chei Redis session-store.ts -- Stocare sesiuni in Redis ca JSON (TTL 7 zile) brain/ client.ts -- Client didi-brain: gather cu verification lookup + write cache + analysis atom lookup/write/patch (gold/silver/bronze tiers, fire-and-forget writes) queue/ constants.ts -- Topologie cozi: exchange, routing keys, prioritati types/ component-results.ts -- Definitii tipuri pentru toate cele 5 rezultate componente analysis-session.ts -- Tipul AnalysisSession (wrapper root al unei analize) api-responses.ts -- Tipuri raspunsuri API (success/error envelope) test-utils/ fixtures.ts -- Date de test (sesiuni, rezultate componente) mock-pg.ts -- Mock PostgreSQL pool pentru teste mock-redis.ts -- Mock Redis pentru teste helpers/ empty-media-response.ts -- Raspuns standardizat cand media nu are continut text auth/ jwt-verify.ts -- Verificare criptografica JWT RS256 via Keycloak JWKS (Modul 7) guards.ts -- Route guards (resolveUserId, authRequired, requireRole, ROLES_ADMIN) observability/ metrics.ts -- Metrici Prometheus business + OTel tracing + /metrics server workeri (Modul 8) queue-depth-poller.ts -- Poller RabbitMQ Management API → gauge didi_queue_depth / didi_active_workers credits.ts -- Verificare si deducere credite utilizator (via didiFramework) config/ analysisLimits.ts -- Limite text: min 20, max 50000 caractere + validare UTF-8 queue/ dispatcher.ts -- Dispatch mesaje catre RabbitMQ (async) connection.ts -- Manager conexiune RabbitMQ aggregator.ts -- Agregator verdict (consuma rezultate componente) types.ts -- Tipuri mesaje coada (dispatch, result, etc.) workers/ component-worker.ts -- Worker generic (consuma din coada, ruleaza componenta) techniques-worker.ts -- Worker Techniques (config) ai-tampered-worker.ts -- Worker AI-Tampered (config) claims-worker.ts -- Worker Claims (config) worker-entrypoints/ techniques.ts -- Entry point Docker: worker techniques (2 replici) ai-tampered.ts -- Entry point Docker: worker ai-tampered (2 replici) claims.ts -- Entry point Docker: worker claims (3 replici) domain.ts -- Entry point Docker: worker domain (2 replici) aggregator.ts -- Entry point Docker: verdict aggregator (2 replici) scripts/ export-schemas.ts -- Export scheme Zod ca JSON Schema init-priority-queues.ts -- Initializare cozi RabbitMQ cu prioritati ``` **Nota**: `queue/constants.ts` (topologie cozi: exchange, routing keys, prioritati) se afla la `shared/queue/constants.ts`. --- ## API - Toate endpoint-urile ### General (`/api/v3`) | Metoda | Path | Ce face | Logica in fisier | |--------|------|---------|------------------| | GET | /health | Health check (raspunde cu service name, version, timestamp) | api/routes.ts | **Nota**: `/health` este montat pe routerul `/api/v3`, deci path-ul complet este `/api/v3/health`. ### Techniques (`/api/v3/techniques`) | Metoda | Path | Ce face | Logica in fisier | |--------|------|---------|------------------| | GET | /definitions | Ierarhie completa tehnici din Redis (dimensiuni -> subdimensiuni -> tehnici) | api/routes.ts | | GET | /config | Configurare completa din Redis | api/routes.ts | | GET | /models | Lista modele LLM disponibile | api/routes.ts | | GET | /stage-assignments | Configurare etape (screening/deep) | api/routes.ts | | PUT | /stage-assignments | Actualizeaza configurare etape | api/routes.ts | | POST | /test-model | Testeaza conexiunea la un model LLM | api/routes.ts | | POST | /test-openrouter | Testeaza OpenRouter cu provider routing | api/routes.ts | | POST | /analyze | Analiza text sincrona | api/routes.ts -> components/techniques/executor.ts | | POST | /analyze-media | Analiza media sincrona (imagine/audio/video) | api/routes.ts -> shared/media/* -> components/techniques/executor.ts | | POST | /analyze-async | Analiza asincrona via RabbitMQ | api/routes.ts -> queue/dispatcher.ts | | GET | /results/:sessionId | Rezultate intermediare din Redis | api/routes.ts | ### AI-Tampered (`/api/v3/ai-tampered`) | Metoda | Path | Ce face | Logica in fisier | |--------|------|---------|------------------| | GET | /health | Health check | api/ai-tampered-routes.ts | | GET | /config | Configurare completa din Redis | api/ai-tampered-routes.ts | | GET | /models | Lista modele LLM | api/ai-tampered-routes.ts | | GET | /categories | Categorii + ierarhie indicatori AI | api/ai-tampered-routes.ts | | GET | /stage-assignments | Configurare etape | api/ai-tampered-routes.ts | | PUT | /stage-assignments | Actualizeaza configurare etape | api/ai-tampered-routes.ts | | POST | /quick | Analiza rapida fara LLM (doar euristici) | api/ai-tampered-routes.ts -> components/ai-tampered/executor.ts | | POST | /analyze | Analiza text completa cu LLM | api/ai-tampered-routes.ts -> components/ai-tampered/executor.ts | | POST | /analyze-image | Analiza imagine pentru AI (doar viziune) | api/ai-tampered-routes.ts -> shared/media/vision.ts | | POST | /analyze-media | Analiza media sincrona | api/ai-tampered-routes.ts -> shared/media/* -> components/ai-tampered/executor.ts | | POST | /analyze-async | Analiza asincrona via RabbitMQ | api/ai-tampered-routes.ts -> queue/dispatcher.ts | | GET | /results/:sessionId | Rezultate intermediare din Redis | api/ai-tampered-routes.ts | | POST | /test-model | Testeaza model LLM | api/ai-tampered-routes.ts | ### Claims (`/api/v3/claims`) | Metoda | Path | Ce face | Logica in fisier | |--------|------|---------|------------------| | GET | /health | Health check | api/claims-routes.ts | | GET | /config | Configurare + tipuri/statusuri din framework | api/claims-routes.ts | | GET | /types | Tipuri de afirmatii (VF, EF, RE, etc.) | api/claims-routes.ts | | GET | /statuses | Statusuri verificare (VT, LT, UV, etc.) | api/claims-routes.ts | | GET | /models | Lista modele LLM | api/claims-routes.ts | | GET | /stage-assignments | Configurare etape | api/claims-routes.ts | | PUT | /stage-assignments | Actualizeaza configurare etape | api/claims-routes.ts | | POST | /analyze | Analiza text: extractie + verificare afirmatii | api/claims-routes.ts -> components/claims/executor.ts | | POST | /analyze-media | Analiza media sincrona | api/claims-routes.ts -> shared/media/* -> components/claims/executor.ts | | POST | /analyze-async | Analiza asincrona via RabbitMQ | api/claims-routes.ts -> queue/dispatcher.ts | | GET | /results/:sessionId | Rezultate intermediare din Redis | api/claims-routes.ts | | POST | /test-model | Testeaza model LLM | api/claims-routes.ts | ### Pipeline complet (`/api/v3/pipeline`) | Metoda | Path | Ce face | Logica in fisier | |--------|------|---------|------------------| | POST | /analyze | Analiza completa sincrona (toate componentele) | api/pipeline-routes.ts -> components/pipeline/executor.ts | | POST | /analyze-url | Analiza URL inteligenta (detecteaza YouTube, imagine, articol) | api/pipeline-routes.ts -> shared/media/video-processor.ts | | POST | /analyze-media | Analiza media sincrona (shortcut) | api/pipeline-routes.ts -> components/pipeline/executor.ts | | POST | /analyze-async | Analiza completa asincrona via RabbitMQ | api/pipeline-routes.ts -> queue/dispatcher.ts | | GET | /:sessionId/status | Status sesiune (polling) | api/pipeline-routes.ts -> shared/redis/session-store.ts | | GET | /:sessionId/component/:name | Rezultat o singura componenta | api/pipeline-routes.ts -> shared/redis/session-store.ts | | GET | /:sessionId/result | Rezultat complet sesiune | api/pipeline-routes.ts -> shared/persistence/persist-service.ts | | GET | /:sessionId/queue-status | Status coada async (progres %) | api/pipeline-routes.ts -> queue/dispatcher.ts | | GET | /queue-health | Sanatate RabbitMQ | api/pipeline-routes.ts -> queue/connection.ts | | GET | /verdict-config | Configurare verdict (categorii, ponderi, override-uri din Redis) | api/pipeline-routes.ts | | POST | /dry-run | Rezolvă planul complet de execuție fără dispatch (ce componente rulează/skip + de ce, chain-uri model/tier, profil verdict). Read-only: fără session id, fără credite, fără mesaje. Modul 1. | api/pipeline/dry-run.ts | | POST | /:sessionId/cancel | Anulează o sesiune în execuție (flag Redis; workerii short-circuit, aggregator finalizează `canceled`). Idempotent, owner/admin. | api/pipeline/cancel.ts | | POST | /:sessionId/resume | Resume din checkpoint — re-dispatch DOAR componentele necompletate (`dispatcher.resumeDispatch`). Owner/admin. | api/pipeline/cancel.ts | ### Istoric (`/api/v3/pipeline`) | Metoda | Path | Ce face | Logica in fisier | |--------|------|---------|------------------| | GET | /history | Istoric utilizator paginat | api/pipeline-routes.ts -> shared/persistence/pg-adapter.ts | | GET | /history/:id | O intrare din istoric | api/pipeline-routes.ts -> shared/persistence/persist-service.ts | | DELETE | /history/:id | Sterge intrare din istoric | api/pipeline-routes.ts -> shared/persistence/persist-service.ts | | GET | /history/admin | Istoric admin (toti utilizatorii, filtre) | api/pipeline-routes.ts -> shared/persistence/pg-adapter.ts | | GET | /history/admin/:id | O intrare admin | api/pipeline-routes.ts -> shared/persistence/persist-service.ts | | DELETE | /history/admin/:id | Sterge intrare admin | api/pipeline-routes.ts -> shared/persistence/persist-service.ts | ### Extensie browser (`/api/v3/pipeline/extension`) | Metoda | Path | Ce face | Logica in fisier | |--------|------|---------|------------------| | POST | /analyze | Analiza via API key (nu JWT) | api/pipeline-routes.ts -> components/pipeline/executor.ts | | POST | /keys | Creeaza API key pentru extensie | api/pipeline-routes.ts (proxy -> didiFramework) | | GET | /keys | Lista API keys | api/pipeline-routes.ts (proxy -> didiFramework) | | DELETE | /keys/:id | Sterge API key | api/pipeline-routes.ts (proxy -> didiFramework) | ### Media/Fisiere (`/api/v3/media`) | Metoda | Path | Ce face | Logica in fisier | |--------|------|---------|------------------| | POST | /upload-url | Obtine URL presemnat pentru upload in MinIO | api/routes.ts | | POST | /upload | Upload direct fisier (multipart, max 50MB) | api/routes.ts | | POST | /download-url | Obtine URL presemnat pentru download din MinIO | api/routes.ts | | GET | /file/:bucket/{*objectKey} | Proxy servire fisiere din MinIO (suporta range requests, Express 5 wildcard) | api/routes.ts | ### Source Assessment (`/api/v3/source-assessment`) | Metoda | Path | Ce face | Logica in fisier | |--------|------|---------|------------------| | GET | /health | Health check | api/source-assessment-routes.ts | | GET | /config | Configurare (scoring_config + framework sources) | api/source-assessment-routes.ts | | GET | /models | Modele LLM disponibile | api/source-assessment-routes.ts | | GET | /stage-assignments | Configurare etape | api/source-assessment-routes.ts | | PUT | /stage-assignments | Actualizeaza etape | api/source-assessment-routes.ts | | POST | /test-model | Test conectivitate model | api/source-assessment-routes.ts | | POST | /analyze | Analiza sursa (text + optional URL) | api/source-assessment-routes.ts -> components/source-assessment/executor.ts | | POST | /analyze-media | Analiza sursa din media | api/source-assessment-routes.ts | ### Domain (`/api/v3/domain`) — LEGACY, inlocuit de Source Assessment | Metoda | Path | Ce face | Logica in fisier | |--------|------|---------|------------------| | POST | /analyze | Analiza domeniu sincrona (apeleaza Domain Check API extern) | api/routes.ts | | POST | /analyze-async | Analiza domeniu asincrona | api/routes.ts -> queue/dispatcher.ts | ### HIL Moderation (`/api/v3/moderation`) | Metoda | Path | Roluri permise | Ce face | Logica in fisier | |--------|------|----------------|---------|------------------| | GET | /queue | admin, moderator, senior_moderator | Listeaza coada review (filtre: `status` CSV, `priority` CSV, `assigned_to=me\|user`) | api/moderation-routes.ts -> components/moderation/queue-manager.ts | | GET | /queue/:queueId | admin, moderator, senior_moderator | Intrare coada + sesiunea analiza JOIN-ed | api/moderation-routes.ts -> components/moderation/queue-manager.ts | | POST | /queue/:queueId/claim | moderator, senior_moderator | Claim atomic (UPDATE conditionat pe pending) — assigneaza la user-ul JWT | api/moderation-routes.ts -> components/moderation/queue-manager.ts | | PUT | /queue/:queueId/resolve | moderator, senior_moderator | Resolve cu actiune `approved` \| `corrected` \| `rejected` + corrections JSONB diff | api/moderation-routes.ts -> components/moderation/queue-manager.ts -> shared/brain/client.ts (PATCH atom gold) | | POST | /flag | orice user autenticat | Flag user (extensie browser) — enqueue review | api/moderation-routes.ts -> components/moderation/queue-manager.ts | | GET | /stats | admin, moderator, senior_moderator | Counts pe tier/component + `hit_rate_24h` | api/moderation-routes.ts -> components/moderation/queue-manager.ts | **Auth (Faza D done 2026-05-02)**: `requireRole()` și `requireAuth()` din `moderation-routes.ts` aplică role check **strict** când `STAGING_MODE=false` (production). În staging (`STAGING_MODE=true`, default actual): GET-urile de read trec fără JWT (legacy), `/flag` cere body `user_id` sau JWT. Vezi `IMPLEMENTATION_PLAN_HIL_BRAIN.md` Faza D pentru permission matrix completa. Constants in `moderation-routes.ts`: - `ROLES_READ = ['admin', 'moderator', 'senior_moderator']` — citire queue/stats - `ROLES_WRITE = ['moderator', 'senior_moderator']` — claim/resolve (admin nu modifică queue, doar vede) - `STAGING = process.env.STAGING_MODE === 'true'` — controller pentru soft auth fallback --- ## Autentificare JWT (Modul 7) Fișier: `src/shared/auth/jwt-verify.ts` (+ guards în `src/shared/auth/guards.ts`). Verificare **criptografică** a token-urilor JWT — nu doar la gateway, ci și în backend, deci un acces direct pe `:24803` (bypass Kong) sau prin nginx-ul admin-dashboard este la fel de protejat. - **RS256 via Keycloak JWKS**: fiecare Bearer JWT e verificat împotriva JWKS-ului realm-ului emitent (`didi-clients` / `didi-admins`, allow-list `JWT_ALLOWED_REALMS`). JWKS descărcat remote și cache-uit per realm (`createRemoteJWKSet` din `jose`). - **Respinge cu 401** (`error_code: JWT_INVALID`): token forjat (semnătură invalidă pe cheile noastre), expirat/`nbf` (validare `exp`/`nbf`), sau emis de un realm foreign (issuer în afara allow-list). Un token cu issuer `didi-*` spoofat tot pică pe semnătură. - Fără header `Authorization` → request rămâne anonim (route guards decid). Bearer non-JWT (API key opac) trece netransformat. - `jwtIdentityMiddleware()` populează `req.jwtUserId` / `req.jwtEmail` / `req.jwtRoles` din payload-ul **verificat** (`realm_access.roles`). - Env: `KEYCLOAK_URL` (default `http://didi-keycloak:8080/auth`), `JWT_ALLOWED_REALMS`, `JWT_VERIFY_ENABLED` (`false` → decode-only, doar dev, cu warning zgomotos). ## Observabilitate (Modul 8) Fișiere: `src/shared/observability/metrics.ts` + `src/shared/observability/queue-depth-poller.ts`. **Metrici Prometheus** (client `prom-client`), inclusiv metrici business DiDi: - `didi_analyses_completed_total` / `didi_analyses_failed_total` (labels: component, tier, media_type, verdict/reason) - `didi_pipeline_duration_seconds` (histogram, per component+tier) - `didi_llm_calls_total` + `didi_llm_tokens_total` (provider, model, component) - `didi_brain_cache_hits_total` / `..._misses_total` (verification + atom) - `didi_queue_depth` + `didi_active_workers` (gauge, per component+plan) — alimentate de `queue-depth-poller.ts` care interoghează RabbitMQ Management API la 30s (fail-open) - `didi_dlq_messages_total`, `didi_http_requests_total`, `didi_http_request_duration_seconds` + default Node/process metrics **Server `/metrics`**: expus de agent-v3 (Express) și de fiecare container worker via `startWorkerMetricsServer()` (workerii n-au app Express, deci server HTTP minimal dedicat pentru scrape). **OpenTelemetry tracing**: `initObservability()` pornește OTel NodeSDK cu auto-instrumentations (Express, http, redis, amqplib, pg) când `OTEL_EXPORTER_OTLP_ENDPOINT` e setat (default `http://didi-otel-collector:4317`); altfel disabled. ## Spec OpenAPI Fișier: `openapi.yaml` la root-ul serviciului (OpenAPI 3.0.3, Modulele 1-3). Descrie toate endpoint-urile (async 202 + poll, pipeline cu dependențe explicite, dry-run/cancel/resume, istoric, moderare HIL). Security: `bearerAuth` (JWT Keycloak RS256 verificat în backend, nu doar la gateway). ## Fluxul de date ### Analiza text (sincrona) ``` Request POST /api/v3/pipeline/analyze { text } | v Validare text (20-50000 caractere) + Verificare credite | v PipelineExecutor.execute() | v ComponentRunner.runAll() -- ruleaza in paralel: |-- TechniquesV3Executor: screening -> deep analysis -> manipulation_score |-- AITamperedExecutor: disclosure check -> screening -> deep -> ai_probability |-- ClaimsExecutor: extractie afirmatii -> brain /v1/gather (cache check + evidence) -> fresh HIT skip LLM / miss: LLM verificare + fire-and-forget cache write -> credibility_score |-- analyzeDomain(): analiza locala (daca exista URL) -> trust_score | v VerdictCalculator.calculate() -- functie pura, fara I/O |-- Selecteaza INPUT PROFILE per input_type (din Redis didi:config:pipeline:v1:input_profiles) | Profiluri: text_no_url, text_with_url, image, audio, video, url | Fiecare profil defineste: ponderi, override-uri active, reguli INCONCLUSIVE |-- Extrage scoruri componente (0-100) |-- Aplica ponderi din profil (ex: image: ai=50%, tech=20%, claims=15%, source=15%) | Fallback la ponderi globale daca profilul nu exista |-- Aplica multiplicator topic INAINTE de override (nu amplifica bonusurile) |-- Aplica override-uri din profil (doar cele activate per input type): | - synergy_bonus: 2+ componente cu risc >= threshold (+bonus/componenta, max cap) | - false_claims: afirmatii false (+bonus/claim, max cap) | - unverified_verifiable: claims verificabile neverificate (bonus fix) | - severe_techniques: tehnici severe (bonus fix) | - undisclosed_ai: AI nedeclarat (bonus fix) | - untrusted_source: sursa neincrezuta/suspecta (bonus fix, din source_assessment) | - blacklisted_source: sursa pe lista neagra (bonus fix, din source_assessment) | - source_red_flags: red flags sursa (+bonus/flag, max cap) | - override cap total per profil (default 50) |-- Rotunjire inainte de mapare la categorii |-- Mapare la risk_category, risk_level, severity (din Redis) |-- Calcul confidence (bonusuri/penalitati din profil) |-- INCONCLUSIVE check din profil (componenta primara crapat, min_components, required_any) |-- Virality: delegat la virality-calculator.ts (scor + factori + nivel) | v PersistService.persist() -- salveaza in Redis + PostgreSQL | v Response: AnalysisSession cu toate componentele ``` ### Analiza media (audio/video) ``` Request POST /api/v3/techniques/analyze-media { media_url, media_type } | v Verificare credite | v Procesare media: |-- Audio: transcribe() via M17-Whisper -> Groq -> OpenAI |-- Video: download (yt-dlp) -> extrage cadre (ffmpeg) -> transcriere + viziune |-- Imagine: callVision() OCR -> text extras | v Text extras -> TechniquesV3Executor (la fel ca text) | v PersistService.persist() | v Response: AnalysisSession cu metadata extractie (durata, provider, cadre analizate) ``` ### Analiza asincrona ``` Request POST /api/v3/pipeline/analyze-async { text/media_url, plan_type } | v Validare + Creditare | v Dispatcher.dispatch() -> publica in RabbitMQ | v Response 202: { session_id, poll_url, result_url } --- In fundal (workeri Docker separati) --- ComponentWorker consuma din coada -> ruleaza componenta -> publica rezultat in analysis.results | v Aggregator consuma din analysis.results -> asteapta toate componentele -> VerdictCalculator | v PersistService.persist() -> sesiune finala in Redis + PostgreSQL --- Clientul polleaza --- GET /api/v3/pipeline/{sessionId}/queue-status -> progres + componente finalizate GET /api/v3/pipeline/{sessionId}/result -> sesiune completa cand status=completed ``` --- ## Servicii externe apelate | Serviciu | Scop | Unde in cod | |----------|------|-------------| | **Redis** local (`didi-cache:6379`, DB 0, parolă redis123) | Config framework + cache sesiuni (TTL 7 zile), acces prin `createRedisConnection()` | shared/redis/* | | **PostgreSQL 17** local (`didi-postgres:5432`, DB `DIDI`, user `bos_interface`) | Stocare permanenta (schema bos_analysis) | shared/persistence/* | | **RabbitMQ** local (`staging-dataLayer-rabbitmq:5672`, vhost `/`, user admin) | Cozi async pentru workeri | queue/* | | MinIO (9000) | Stocare fisiere media | api/routes.ts (media endpoints) | | OpenRouter API | Modele LLM (Gemini, GPT-4o, Claude, etc.) | components/*/executor.ts | | OpenAI API | LLM fallback + Whisper transcriere | components/*/executor.ts, shared/media/transcription.ts | | Groq API | LLM fallback + Whisper transcriere | components/*/executor.ts, shared/media/transcription.ts | | M17-Whisper (`M17_WHISPER_URL`, default `http://10.11.10.17:54300`, configurabil din env) | Transcriere audio primara | shared/media/transcription.ts | | M17 Web API (10.11.10.13:51100) | Fallback cautare web — folosit cand brain dezactivat sau MISS fara evidence | components/claims/executor.ts, components/source-assessment/executor.ts | | **didi-brain** (Lot 1, `DIDI_BRAIN_URL` default `http://10.11.10.12:8090`) | Verification cache claims (`/v1/gather`, `/v1/verification_cache`) + analysis atom cache techniques/ai-tampered (`/v1/analysis_atom*`) — vezi „Integrare cu platforma AI (Lot 1)" | shared/brain/client.ts, components/*/executor.ts | | Vision LLM (Lot 1, `VISION_LLM_URL` default `http://10.11.10.17:14011`) | Analiza imagine (OCR + AI detection) | shared/media/vision.ts | | Domain Check API (:11000) | Analiza domeniu (WHOIS, DNS, SSL, blacklist) | api/routes.ts (domain/analyze) | | didiFramework (didi-framework:3005) | Verificare/deducere credite + API keys extensie + sync framework → Redis local | shared/credits.ts, api/pipeline-routes.ts | --- ## Baza de date PostgreSQL Schema: `bos_analysis` | Tabel | Ce stocheaza | Cheie | |-------|-------------|-------| | analysis_session | Sesiunea root: user, input, status, timing, risk_score | session_id (UUID) | | analysis_techniques | Rezultat componenta Techniques (JSONB: techniques_detected, coupling_context) | session_id FK | | analysis_ai_tampered | Rezultat componenta AI-Tampered (JSONB: indicators_detected, image_analysis) | session_id FK | | analysis_claims | Rezultat componenta Claims (JSONB: claims_verified, claims_by_status) | session_id FK | | analysis_domain | Rezultat componenta Domain (trust_score, red_flags TEXT[], warnings) | session_id FK | | analysis_verdict | Verdict final (risk_score, JSONB: applied_weights, context_summary) | session_id FK | | moderation_queue | Coada review HIL (FK → analysis_session.session_id UUID): tier, priority, status, assigned_to, corrections JSONB | queue_id (UUID) | Tabelul `analysis_session` are 6 coloane suplimentare pentru HIL: `review_status`, `human_corrected`, `human_corrections` (JSONB), `verified_by`, `verified_at`, `review_notes` (adaugate via migration `011_add_moderation.sql` rulata de didiFramework). In schema `bos_parammgmt`, didiFramework owneste 3 tabele de config citite de agent-v3 prin Redis: `moderation_config` (single row), `sensitive_topic`, `moderation_role`. Salvarea este tranzactionala (all-or-nothing). Ce produce componenta = ce se salveaza (zero transformari). --- ## Chei Redis importante ### Framework (FrameworkKeys) — permanente, scrise de didiFramework sync-redis | Cheie | Scop | |-------|------| | didi:framework:manifest | Index categorii config | | didi:framework:techniques | Ierarhie tehnici: dimensiuni -> subdimensiuni -> tehnici -> indicatori | | didi:framework:sources | Parametri credibilitate sursa | | didi:framework:claims | Parametri claims (tipuri, statusuri, confidence) | | didi:framework:verdicts | Categorii verdict, risk mappings, severity assessments | | didi:framework:weights | Ponderi componente, scenarii, multiplicatori | | didi:framework:providers | Config provideri LLM (optional) | | didi:framework:dimensions_compact | Lista compacta dimensiuni pentru prompts screening | ### Config componente (ConfigKeys) — permanente, scrise de didiFramework sync-redis **Toate `stage_assignments` sunt tier-nested** (structura `{stage_code: {free: {...}, premium: {...}}}`) dupa migration 006 (tier column). Chain-urile de modele sunt diferite per tier — vezi sectiunea "Tier-aware execution" de mai sus. | Cheie | Scop | |-------|------| | didi:config:pipeline:v1:component_config | Config executor pipeline + video track weights | | didi:config:pipeline:v1:session_config | Config sesiune pipeline | | didi:config:pipeline:v1:verdict_config | Override-uri, synergy, confidence (globale, fallback) | | didi:config:pipeline:v1:input_profiles | **Profiluri verdict per input type (6 profile cu ponderi, override-uri, INCONCLUSIVE rules)** | | didi:config:techniques:v3:available_models | Modele LLM Techniques (union free + premium) | | didi:config:techniques:v3:stage_assignments | **Tier-nested** asignari etape Techniques | | didi:config:techniques:v3:scoring_config | **Parametri scoring: count_scaler, severe_threshold, intensity bonus, bonuses** | | didi:config:techniques:v3:dimensions_compact | Lista compacta dimensiuni Techniques | | didi:config:claims:v1:available_models | Modele LLM Claims (union free + premium) | | didi:config:claims:v1:stage_assignments | **Tier-nested** asignari etape Claims | | didi:config:claims:v1:scoring_config | **Parametri scoring: status_weights, unverified behavior, claim_type_weights** | | didi:config:ai-tampered:v1:available_models | Modele LLM AI-Tampered (union free + premium) | | didi:config:ai-tampered:v1:stage_assignments | **Tier-nested** asignari etape AI-Tampered | | didi:config:ai-tampered:v1:scoring_config | **Parametri scoring: blend_weights, disclosure_impact, thresholds, undisclosed_threshold** | | didi:config:ai-tampered:v1:vision_models | Legacy flat vision config (fallback — nou e `didi:config:vision:v1:stage_assignments`) | | didi:config:source-assessment:v1:available_models | Modele LLM Source Assessment (union free + premium) | | didi:config:source-assessment:v1:stage_assignments | **Tier-nested** asignari etape Source Assessment | | didi:config:source-assessment:v1:scoring_config | **Parametri scoring: axis_weights, verdict_thresholds** | | didi:config:source-assessment:v1:prompts:extraction | Prompt extractie metadata sursa | | didi:config:source-assessment:v1:prompts:evaluation | Prompt evaluare sursa cu categorii framework | | didi:config:verdict:v1:stage_assignments | **Tier-nested** asignari Verdict reviewer (stage: verdict_review) | | didi:config:verdict:v1:available_models | Legacy fallback pentru verdict-explanation.ts | | didi:config:vision:v1:stage_assignments | **Tier-nested** asignari Vision (stage: image_analysis — OCR + AI detection + video frames) | | didi:config:vision:v1:prompts:extraction | Prompt viziune: extractie text din imagine | | didi:config:vision:v1:prompts:video_frames | Prompt viziune: analiza cadre video | | didi:config:vision:v1:prompts:ai_detection | Prompt viziune: detectie AI in imagine | | didi:config:moderation:v1:settings | Setari moderation + brain client (thresholds: `confidence_low`, `risk_grey_min/max`, feature flags: `triage_enabled`, `brain_enabled`, `brain_url`, `brain_lookup_timeout_ms`, `brain_per_component`) — citit cu cache 60s | | didi:config:moderation:v1:sensitive_topics | Lista topice sensibile pentru triage (auto-enqueue daca match) | | didi:config:moderation:v1:roles | Map roluri moderation (folosit la role check soft) | **Toate cheile config au corespondent in PG** (`bos_parammgmt.component_config` + `component_prompt` + `component_stage_assignment` + `moderation_config` + `sensitive_topic` + `moderation_role`). Sync-redis le citeste din PG si le scrie in Redis. Admin dashboard editeaza PG, apoi sync to Redis. ### Sesiuni pipeline (SessionKeys) — TTL 7 zile | Pattern | Scop | |---------|------| | didi:pipeline:{sessionId}:status | Status executie pipeline (JSON: PipelineStatus) | | didi:pipeline:{sessionId}:{component} | Rezultat componenta (JSON: TechniquesResult etc.) | | didi:pipeline:{sessionId}:verdict | Verdict final (JSON: VerdictResult) | ### Istoric (HistoryKeys) — TTL 7 zile | Pattern | Scop | |---------|------| | didi:pipeline:history:entry:{sessionId} | Date intrare istoric (JSON: input + metadata) | | didi:pipeline:history:user:{userId} | Sorted set istoric utilizator (score=timestamp, member=sessionId) | ### Coada (QueueKeys) — TTL scurt (30s-5min) | Pattern | Scop | |---------|------| | didi:queue:session:{sessionId} | Stare fan-in agregare (JSON: partial AnalysisSession) | | didi:queue:lock:{sessionId}:{component} | Lock executie componenta (previne duplicare worker) | | didi:queue:aggregator:{sessionId} | Lock agregator (previne calcul verdict concurent) | ### Rezultate intermediare (AgentKeys) — TTL 7 zile | Pattern | Scop | |---------|------| | agent:result:{sessionId}:{component}:{stage} | Rezultat intermediar etapa (screening, deep_analysis) | | agent:result:{sessionId}:ai-tampered:visual | Rezultat analiza vizuala AI | | agent:result:{sessionId}:ai-tampered:final | Rezultat final AI tampered (media merged) | | agent:result:{sessionId}:ai-tampered:media | Rezultat intermediar AI tampered media | ### Media cache (MediaCacheKeys) — TTL 1 ora Scrise de media-preprocess worker. Citite de component workers. | Pattern | Scop | |---------|------| | agent:media:{sessionId}:transcript | Transcript audio (Whisper) | | agent:media:{sessionId}:vision:misinformation | Output Vision frames — analiza manipulare | | agent:media:{sessionId}:vision:ai_detection | Output Vision frames — detectie AI | | agent:media:{sessionId}:merged_text | Transcript + vision misinformation (pentru techniques + claims) | | agent:media:{sessionId}:lock | Lock procesare media (previne duplicare) | | agent:media:{sessionId}:ready | Semnal finalizare pre-procesare ("1") | ### Extensie (ExtensionKeys) — permanente | Pattern | Scop | |---------|------| | didi:extension:key:{apiKey} | Cache API key extensie browser | --- ## Docker Compose - Servicii | Serviciu | Replici | Rol | |----------|---------|-----| | agent-v3 | 1 | Server API principal (port 24803) | | worker-media-preprocess | 2 | Pre-procesare media: 1× download, 1× ffmpeg, 1× transcriptie, 2× Vision. Cache in Redis. Dispatche-aza component workers. | | worker-techniques | 2 | Consuma din coada techniques. Citeste media din Redis cache. | | worker-ai-tampered | 2 | Consuma din coada ai-tampered. Citeste media din Redis cache. 2-track video scoring (text 40% + visual 60%). | | worker-claims | 3 | Consuma din coada claims (mai lent, mai multe replici). Citeste merged_text din Redis cache (transcript + text vizual). | | worker-domain | 2 | Consuma din coada domain | | verdict-aggregator | 2 | Colecteaza rezultate si calculeaza verdict | Dependinte sistem in container: ffmpeg, yt-dlp, python3 (pentru video download si procesare). ### Fluxul async media (video/audio/image) ``` POST /api/v3/pipeline/analyze-async { media_type: "video" } | v Dispatcher → analysis.media_preprocess.{plan} (1 task) | v MediaPreprocessWorker: 1. Download video (1×) 2. ffmpeg: extract frames uniform (1×) + extract audio (1×) 3. Whisper: transcriere audio (1×) 4. Vision misinformation: analiza frames (1×) 5. Vision ai_detection: analiza frames (1×) 6. Salveaza in Redis (TTL 1h): - agent:media:{sid}:transcript - agent:media:{sid}:vision:misinformation - agent:media:{sid}:vision:ai_detection - agent:media:{sid}:merged_text - agent:media:{sid}:ready 7. Dispatche-aza: techniques + ai_tampered + claims (+ domain daca URL) | v Component Workers (citesc din Redis cache, zero download/ffmpeg): - techniques: primeste merged_text (transcript + vizual) - ai_tampered: primeste transcript (Track 1 text) + vision:ai_detection (Track 2 visual) - claims: primeste merged_text (transcript + text vizibil de pe ecran) - domain: primeste doar URL (skip pe media fara URL) | v Verdict Aggregator → verdict (neschimbat) ``` Fallback: daca media preprocess esueaza, component workers proceseaza media local (safety net). ### Fluxul async text/URL (neschimbat) Dispatcher → direct la component workers (fara media preprocess). --- ## Timeout-uri pe rute | Ruta | Timeout | Motiv | |------|---------|-------| | /api/v3/pipeline/* | 660s (11 min) | Download video + transcriere + analiza completa | | /api/v3/techniques/analyze-media | 300s (5 min) | Transcriere media + analiza | | /api/v3/ai-tampered/analyze-media | 300s (5 min) | Procesare media + analiza | | /api/v3/claims/analyze-media | 300s (5 min) | Transcriere + extractie + verificare | | Toate celelalte | 180s (3 min) | Analiza text standard (server.timeout) | | keepAliveTimeout | 665s | Usor mai mare decat max per-route timeout | | headersTimeout | 670s | Usor mai mare decat keepAliveTimeout | --- ## Tipuri de raspuns ### AnalysisSession (raspuns standard pentru toate analizele) ``` { session_id, user_id, user_email, input_type (text/url/image/audio/video), input_text, input_url, input_media_url, input_hash, // SHA256 status (pending/running/completed/failed), components_run[], components_skipped[], risk_score, risk_category, risk_level, confidence, confidence_level, started_at, completed_at, total_duration_ms, scenario_applied, topic_applied, source_app, api_version: "v3", created_at, techniques: { manipulation_score, total_severity, dimensions_affected[], techniques_count, techniques_detected[], coupling_context, llm_screening, llm_deep, screening_duration_ms, deep_analysis_duration_ms, total_duration_ms, fallbacks_screening, fallbacks_deep }, ai_tampered: { ai_probability, verdict, risk_score, categories_affected[], indicators_count, disclosure_detected, disclosure_explicit, disclosure_text, indicators_detected[], coupling_context, llm_screening, llm_deep, screening_duration_ms, deep_analysis_duration_ms, total_duration_ms, fallbacks_screening, fallbacks_deep, content_type, image_analysis }, claims: { total_claims, verified_true, verified_false, unverified, opinions, credibility_score, interpretation, claims_by_status, claims_by_type, claims_verified[], llm_extraction, llm_verification, extraction_duration_ms, verification_duration_ms, total_duration_ms, web_searches_made }, domain: { domain, trust_score, verdict, risk_level, red_flags[], warnings[], duration_ms, ... }, verdict: { risk_score, risk_category, risk_category_color, risk_level, risk_level_color, severity, recommended_action, confidence, confidence_level, score_manipulation, score_claims, score_ai, score_source, score_context, applied_weights, override_applied, override_type, override_reason, override_adjustment, context_summary, components_used[], weights_source, duration_ms, explanation_ro, explanation_en, virality_score, virality_level, virality_factors[] } } ``` ### Raspuns async (202 Accepted) ``` { success: true, async: true, data: { session_id, status: "processing", queued_components[], plan_type, poll_url, result_url } } ``` ### Raspuns media gol (cand nu s-a extras text din media) ``` { success: true, data: { session_id, status: "completed", input_type, input_media_url, media_type, skipped: true, skip_reason: "no_text_content" | "transcription_failed" | "vision_failed" | "extraction_failed", message: "...", techniques: null, ai_tampered: null, claims: null, domain: null, verdict: null, components_run: [], components_skipped: ["techniques", "ai_tampered", "claims"], risk_score: null, risk_category: null, metadata: { extraction_duration_ms, provider_tried } } } ``` --- ## Schimbari de logica (2026-03-20) ### Verdict INCONCLUSIVE determinist pe media fara claims Fisier: `components/pipeline/verdict-calculator.ts` Daca claims nu a produs rezultat (null) pe input video/audio/image → forteaza `risk_category = INCONCLUSIVE`. Inainte, LLM reviewer-ul decidea (inconsistent: +14 pe un video, +45 pe altul identic). Acum e determinist in cod. LLM reviewer-ul (`verdict-explanation.ts`) nu mai penalizeaza pentru componente lipsa — instructiune explicita in prompt. ### Claims primeste text vizual din video frames Fisier: `queue/workers/component-worker.ts` Claims primeste `merged_text` (transcript audio + text vizibil de pe ecran) in loc de doar transcript. Permite verificarea afirmatiilor scrise pe ecran (titluri, statistici, URL-uri). Prompt-ul claims extraction (Redis `didi:config:claims:v1:prompts:extraction`) instruieste LLM-ul sa distinga intre text citat de pe ecran (extrage ca claim) si observatii Vision AI (ignora). ### AI Tampered video: ponderi 40% text + 60% vizual Fisier: `queue/workers/component-worker.ts` Ponderile 2-track video scoring inversate: text 40% + vizual 60% (era text 60% + vizual 40%). Deepfake-uri cu voce reala dar fete AI primeau scor mic (34%) — vizualul trebuie sa domine pe video. Prag minim text track: 200 chars. Sub 200 chars, text track e skip-uit si vizualul primeste 100% weight. ### Prompt AI detection Vision — 3 tiere forensic Redis: `didi:config:vision:v1:prompts:ai_detection` Tier 1: artefacte grosolane (warping, extra fingers) → AI_CONFIDENCE 75-95 Tier 2: indicii subtile deepfake modern (face-background mismatch, skin uniformity, hair boundary, eye reflections) → AI_CONFIDENCE 45-75 Tier 3: evaluare context (face closeup TV = minim 30, nu 0) → AI_CONFIDENCE 30-40 AI_CONFIDENCE: 0 interzis pe face closeup video. ### Techniques: intensity bonus diminishing returns Fisier: `components/techniques/executor.ts` Formula veche (liniara): `(avgIntensity - 1) * 0.05` → intensity 8 = 0.35 (35pp bonus) Formula noua (sqrt + cap): `min(0.15, sqrt(avgIntensity - 1) * 0.05)` → intensity 8 = 0.13 (13pp bonus) Previne scor 100% pe editoriale/jurnalism de investigatie cu ton puternic dar fara manipulare reala. ### Limite durata media Fisiere: `api/pipeline-routes.ts` (API check), `shared/media/video-processor.ts` (ffprobe check) - Video: max 180s (3 minute) - Audio: max 420s (7 minute) Doua guard-uri: API (instant, daca frontend trimite `media_duration_sec`) + ffprobe (dupa download, safety net). ### Optimizari claims Fisiere: `components/claims/executor.ts`, `components/component-runner.ts` - Claims global timeout: 300s → 600s - Claims Qwen timeout: 60s → 120s (Redis config) - Claims cap: 10 → 7 (skip low priority) - Chunks paralele: 3 → 5 - Transcript trunchiat la 4000 chars pentru claims pe media ### Video frame sampling uniform Fisier: `shared/media/video-processor.ts` Inainte: `fps=1/5` secvential → frames doar din primele 50s. Video 5 min = 83% ignorat. Acum: interval dinamic = `durata / maxFrames` → acoperire uniforma 100%. Minim 3 frames pe orice video. --- ## Funcționalități suplimentare (în afara celor 9 module de caiet) Secțiunile de mai jos sunt **cod backend real**, dar reprezintă funcționalitate în plus față de cele 9 module ale caietului de sarcini. Tot aici se încadrează și **Tier-aware execution (premium vs free)**, **Virality** (`virality-calculator.ts`) și **HIL Moderation** (secțiunea următoare). Logica internă a brain-ului (RAG, tiering atom pe brain-side) aparține platformei AI (Lot 1); aici e descrisă doar integrarea client-side din agent-v3. ### Brain v2 Atom Cache (integrare client-side) Brain v2 introduce un cache de **rezultate de analiza per componenta** (analysis atoms), in plus fata de verification cache-ul existent pentru claims. Salveaza output-ul deep analysis pentru `techniques` si `ai_tampered` indexat pe `(content_hash, component, tier, prompt_hash, framework_version)`. Hit fresh + tier potrivit (gold/silver) → skip LLM complet. ### Tier-uri atom | Tier | Cum se obtine | Cum se foloseste | |------|--------------|------------------| | `gold` | Promovat manual via PATCH din HIL Moderation (action: approved sau corrected) | Hit fresh `gold` → folosit cu prioritate maxima | | `silver` | Auto-scris de agent-v3 dupa fiecare analiza reusita pe tier `premium` | Hit fresh `silver` → folosit (overridable de gold) | | `bronze` | Tier intermediar pe brain side (rezervat) | Ignorat de agent-v3 (skip LLM doar pe gold OR silver) | Brain refuza scrieri pentru tier `free` — atoms scrise doar din premium analyses (calitate consistenta). ### Client extensie (`src/shared/brain/client.ts`) Pe langa helperele existente (`gatherFromBrain`, `writeVerificationCacheAsync`, `evidenceOverlap`), clientul expune: - `lookupAnalysisAtom({ content_hash, component, tier, prompt_hash, framework_version })` → `{ atom_id, data: AnalysisAtomData, cache_tier: AtomCacheTier, staleness: AtomStaleness } | null` - `writeAnalysisAtomAsync(payload)` — fire-and-forget POST `/v1/analysis_atom`, doar pentru `tier = premium` - `patchAnalysisAtomGold(atomId, corrections)` — promoveaza atom la `gold` cu corectiile umane (apelat din `moderation-routes.ts` pe resolve) - helpers: `computeContentHash(text)`, `computePromptHash(prompt)`, `computeFrameworkVersion(...)` - types: `AnalysisAtomData`, `AtomCacheTier` (`'gold' | 'silver' | 'bronze'`), `AtomStaleness` (`'fresh' | 'stale_framework' | 'stale_prompt' | 'miss'`) ### Config & feature flags (Redis: `didi:config:moderation:v1:settings`, cache 60s) | Cheie | Rol | |-------|-----| | `brain_enabled` | Master switch — false → toate apelurile atom skip-uite, fallback direct LLM | | `brain_url` | Override per environment (default `DIDI_BRAIN_URL` env) | | `brain_lookup_timeout_ms` | Timeout per lookup (default tipic 800ms) | | `brain_per_component` | Granular: `{techniques: bool, ai_tampered: bool, claims: bool}` | Fail-open in toate cazurile: orice eroare brain (timeout, 5xx, network, feature flag off) → continua cu LLM. Atom cache nu poate bloca o analiza. ### Integrare in executoare Atat `components/techniques/executor.ts` cat si `components/ai-tampered/executor.ts`: ``` execute(input): // START — atom lookup if (brain_enabled && brain_per_component[component]) { atom = await lookupAnalysisAtom({ content_hash, component, tier, prompt_hash, framework_version }) if (atom && atom.staleness === 'fresh' && (atom.cache_tier === 'gold' || atom.cache_tier === 'silver')) { return atomToComponentResult(atom.data) // skip LLM complet } } // ... screening + deep analysis (LLM normal flow) ... // END — atom write (happy path AND early-exit branches) writeAnalysisAtomAsync({ ... }) // fire-and-forget, doar tier=premium return result ``` Atom write e plasat pe **ambele** ramuri (happy path si early-exit pe disclosure detected / no techniques) — orice rezultat reprezentativ se cache-uieste. ### Promovare la gold via HIL In `moderation-routes.ts` handler-ul `PUT /queue/:queueId/resolve`: - `action = 'approved'` → patch atomii existenti per componenta (`techniques`, `ai_tampered`, `claims`) la `gold` (semnal: omul confirma rezultatul automat) - `action = 'corrected'` → patch atomii la `gold` cu `corrections` aplicate (omul a editat output-ul automat — varianta corectata devine fontul de adevar) - `action = 'rejected'` → nu touch-uie atoms (rezultatul a fost gresit, dar nu avem corectie validata) PATCH-ul foloseste `(content_hash, component, tier, prompt_hash, framework_version)` pentru a localiza atomii potriviti pe brain side. ### Endpoint-uri brain v2 folosite | Metoda | Path | Folosit de | |--------|------|-----------| | POST | /v1/analysis_atom/lookup | `lookupAnalysisAtom()` la START in techniques + ai-tampered | | POST | /v1/analysis_atom | `writeAnalysisAtomAsync()` la END in techniques + ai-tampered (premium only) | | PATCH | /v1/analysis_atom/{id} | `patchAnalysisAtomGold()` din HIL resolve | | GET | /v1/analysis_atom/stats | Endpoint diagnostic (folosit la nevoie) | | POST | /v1/gather | Existent — claims (verification cache lookup + evidence) | | POST | /v1/verification_cache | Existent — claims write | --- ## HIL Moderation Human-in-the-loop pentru sesiuni cu confidence scazut, scor in zona gri, sau topice sensibile. Triage-ul ruleaza dupa persistenta (NU blocheaza analiza) si decide daca sesiunea intra in coada de review uman. Moderatorii claim & resolve via REST API; rezolutiile pozitive (approved/corrected) promoveaza atomii la `gold` in brain. ### Triage (`src/components/moderation/triage.ts`) Citeste config din Redis (`didi:config:moderation:v1:settings`, cache 60s): | Threshold | Rol | |-----------|-----| | `confidence_low` | Sub aceasta valoare → enqueue review (model nesigur) | | `risk_grey_min/max` | Banda gri risk_score (ex. 40-60) → enqueue review (rezultat ambiguu) | | `sensitive_topics` | Match in continut → enqueue review automat | Feature-flag protected: `triage_enabled = false` → triage no-op total. ### Queue manager (`src/components/moderation/queue-manager.ts`) Operatii directe pe `bos_analysis.moderation_queue` via `shared/persistence/pg-pool.ts`: | Functie | Ce face | |---------|---------| | `enqueueForReview(session, reason, priority)` | Idempotent — skip daca exista deja entry pending pentru acest `session_id` | | `listQueue(filters)` | Listare cu filtre `status` CSV, `priority` CSV, `assigned_to` | | `getQueueEntry(queueId)` | Entry + JOIN pe `analysis_session` (sesiunea full) | | `claimQueueEntry(queueId, userId)` | UPDATE atomic conditionat pe `status = 'pending'` | | `resolveQueueEntry(queueId, action, corrections, notes)` | UPDATE status + scrie `human_corrected` / `human_corrections` / `verified_by` / `verified_at` pe `analysis_session` | | `getQueueStats()` | Counts pe tier/component + `hit_rate_24h` | ### Integrare in pipeline Triage hook apelat **dupa** `persistService.persist(session)` in: - `src/components/pipeline/executor.ts` (sync path) - `src/queue/aggregator.ts` (async path) Wrapping try/catch — orice eroare in triage e log-uita, NU se propaga. Analiza nu se blocheaza niciodata din cauza moderation. ### Role check soft Middleware-ul JWT (`src/index.ts`) extrage `req.jwtRoles` din `realm_access.roles`. Endpoint-urile `/api/v3/moderation/*`: - **Staging fara JWT roles** (Keycloak nu trimite role): permit (pentru testing) - **Productie cu JWT**: require rol `moderator` (sau echivalent din `didi:config:moderation:v1:roles`) Politica e soft (permit pe absent) ca sa nu blocheze flows-uri de testare interne. ### `userFlagged` flow `POST /api/v3/moderation/flag` permite oricarui user autenticat (sau extensiei browser) sa raporteze o sesiune. Cand sesiunea e creata cu `user_flagged = true` (request body field), flag-ul e plumbed prin: - `PipelineInput.userFlagged` (`components/pipeline/types.ts`) - `SessionState.userFlagged` (`queue/types.ts`) - `api/pipeline-routes.ts` extrage din body si forward-eaza Triage citeste flag-ul si forteaza enqueue indiferent de threshold-uri (semnal uman = priority).