clossers/didi-lot2-backend
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
didi-lot2-backend/backend/services/orchestration-layer/agent-v3/INDEX.md
EVOTECH IT SRL 8ecc78e729 livrare lot 2
2026-07-10 03:39:53 -07:00

58 KiB
Raw Permalink Blame History

Agent V3 - Index

Serviciul principal al platformei DIDI. Analizeaza continut (text, URL, imagine, audio, video) pentru detectarea dezinformarii. Produce scoruri 0-100 pe 4 componente si un verdict agregat.

Port: 24803 Framework: Express 5 + TypeScript Container: didi-agent-v3

Infrastructura data-layer

Data layer rulează local pe mașina de deployment, ca containere Docker pe didi-network. Nu există cluster extern — serviciile se adresează prin numele de container Docker, nu prin IP de mașină. Config real din env-ul didi-agent-v3:

Serviciu Container (target) Config Credentials
PostgreSQL 17 didi-postgres:5432 DB DIDI, schema bos_analysis user bos_interface / interface
Redis 7 didi-cache:6379 DB 0 parolă redis123
RabbitMQ 3.12 staging-dataLayer-rabbitmq:5672 vhost / user admin / rabbitmq123

Clienți ioredis centralizați via src/shared/redis/connection.ts — createRedisConnection(label). Zero new Redis({...}) inline. Auto-reconnect + reconnectOnError.

RabbitMQ client via src/queue/connection.ts + src/shared/queue/constants.ts. Vhost / (encodeURIComponent() la construirea URI-ului AMQP).

Bootstrap auto-populate: la startup, didi-framework detectează dacă didi:framework:manifest lipsește în Redis și auto-rulează POST /api/sync-redis. Zero intervenție manuală pe fresh deploy.

Integrare cu platforma AI (Lot 1)

Backend-ul (Lot 2) apelează serviciile platformei AI (Lot 1) prin HTTP — nu implementează niciunul intern. Fiecare are un URL configurabil din env, cu fail-open: orice eroare (timeout, 5xx, network, feature flag off) → analiza continuă fără serviciul respectiv (fallback la calea locală sau skip). Cum funcționează intern brain/RAG, atom cache, extractorii, forensic, buster etc. e documentat în platforma AI (Lot 1); aici descriem doar contractul de integrare.

Serviciu Lot 1 Env (default) Folosit pentru Fail-open
didi_brain DIDI_BRAIN_URL (http://10.11.10.12:8090) Verification cache + analysis atom cache pentru claims/techniques/ai-tampered → LLM direct
extractors EXTRACTORS_URL (http://<HOST_IP>:54400) Extragere conținut/metadata → skip
video analysis (buster) VIDEO_ANALYSIS_URL (http://<HOST_IP>:54600) Analiză video → procesare locală
forensic FORENSIC_API_URL (http://forensic-features-api:8080) Forensic features imagine → skip
whisper (audio) M17_WHISPER_URL (http://10.11.10.17:54300) Transcriere audio → Groq → OpenAI
web search M17_WEB_API_URL (http://10.11.10.13:51100) Căutare web claims/source fallback când brain MISS
LLM router LLM_ROUTER_URL (http://10.11.10.17:14011) Inferență LLM primară (Qwen local) → OpenRouter
vision VISION_LLM_URL (http://10.11.10.17:14011) OCR + AI detection imagine → Gemini / GPT-4o
domain check DOMAIN_CHECK_API_URL (http://<domain-check-host>:11000) WHOIS/DNS/SSL/blacklist domeniu → skip

IP-urile GPU 10.11.10.17:14011 (LLM/vision) și 10.11.10.17:54300 (whisper) sunt default, configurabile din env.

Client brain (cod backend, src/shared/brain/client.ts): gatherFromBrain() (verification cache lookup + evidence pentru claims), writeVerificationCacheAsync(), lookupAnalysisAtom() / writeAnalysisAtomAsync() / patchAnalysisAtomGold() (analysis atom cache pentru techniques/ai-tampered), plus helpers de hashing. Detaliul acestei integrări backend e la secțiunea „Funcționalități suplimentare".

Tier-aware execution (premium vs free)

Agent-v3 ruleaza chain-uri diferite de modele LLM in functie de planul de abonament al utilizatorului.

Cum se determina tier-ul

  1. Fiecare request face checkCredits() la didiFramework, care returneaza planType (1-6) in raspuns
  2. getSearchTier(planType) din src/shared/credits.ts mapeaza:
    • plan_type 1-3 → tier 'free'
    • plan_type 4-6 → tier 'premium'
  3. tier (variabila searchTier in cod) e propagat prin toate straturile:
    • Sync routes: pipeline-routes.ts, ai-tampered-routes.ts → PipelineInput.searchTier
    • Async via queue: component-worker.ts deriva din taskMessage.planType
    • ComponentRunner.runAll() → AnalysisInput.searchTier
    • Fiecare executor primeste tier ca parametru la execute()
    • verdict-explanation.ts: primeste tier la generate()
    • vision.ts / callVision(): primeste tier
    • transcription.ts / transcribe(): primeste tier in options
    • M17 search (claims/source-assessment): trimite X-Search-Tier: premium header

Ce se schimba per tier

Component/Stage Free primary Premium primary
Techniques screening Qwen 3.5 397B local Gemini 3 Flash Preview (OR)
Techniques deep Qwen 3.5 397B local Claude Sonnet 4.6 (OR)
AI-Tampered screening Qwen 3.5 397B local Gemini 3 Flash Preview (OR)
AI-Tampered deep Qwen 3.5 397B local Claude Sonnet 4.6 (OR)
Claims extraction Qwen 3.5 397B local Gemini 3 Flash Preview (OR)
Claims verification Qwen 3.5 397B local Claude Sonnet 4.6 (OR)
Source extraction Qwen 3.5 397B local Gemini 3 Flash Preview (OR)
Source evaluation Qwen 3.5 397B local Claude Sonnet 4.6 (OR)
Verdict reviewer Qwen 3.5 397B local Gemini 3 Flash Preview (OR)
Vision (OCR + AI detect) Qwen Vision local Gemini 3 Flash Preview (OR)
Transcription (audio) M17-Whisper local Groq-Whisper
Web search (claims/source) M17 SearXNG M17 + Brave + Tavily (via X-Search-Tier header)

Fallback chain pentru premium termina pe Qwen local (safety net, 0 cost) — zero erori de rutare daca cloud crapa.

Cum se incarca stage assignments tier-based

Executoarele citesc cheia Redis didi:config:{component}:v1:stage_assignments cu structura tier-nested:

{
  "techniques_screening": {
    "free":    { "models": [{ order:1, model_key: "qwen35:Qwen3.5-397B-A17B", ...}, ...] },
    "premium": { "models": [{ order:1, model_key: "openrouter:google/gemini-3-flash-preview", ...}, ...] }
  },
  "techniques_deep": { "free": {...}, "premium": {...} }
}

Helper resolveStageAssignment(assignments, stageCode, tier) returneaza chain-ul corect, cu fallback automat la 'free' daca 'premium' lipseste. Asta protejeaza utilizatorii premium daca configul nu e complet.

Componente de analiza

Serviciul ruleaza 4 componente independente + un calculator de verdict:

Componenta Ce face Scor produs
Techniques Detecteaza tehnici de manipulare (propaganda, apel emotional, etc.) manipulation_score 0-100
AI-Tampered Detecteaza continut generat/modificat de AI ai_probability 0-100
Claims Extrage afirmatii si le verifica prin cautare web credibility_score 0-100
Domain Analizeaza domeniul sursa (varsta, SSL, blacklist, reputatie) trust_score 0-100
Verdict Agrega scorurile componentelor intr-un scor final de risc risk_score 0-100

Fiecare componenta (in afara de Domain) foloseste un pipeline in 2 etape:

  • Screening: analiza rapida cu un model LLM rapid
  • Deep Analysis: analiza detaliata pe fiecare dimensiune/categorie detectata

Fiecare etapa are 3 nivele de fallback (primary -> fallback_1 -> fallback_2 -> fallback_3).

Structura fisierelor

src/
  index.ts                              -- Server Express, middleware, montare rute
  api/                                  -- Rute organizate pe subdirectoare (fiecare *-routes.ts la root e shim de re-export)
    routes.ts / techniques.ts           -- Endpoint-uri Techniques (+ /health general)
    ai-tampered/                        -- Endpoint-uri AI-Tampered
    claims/                             -- Endpoint-uri Claims (analyze, config, results)
    source-assessment/                  -- Endpoint-uri Source Assessment (config, models, stage-assignments, analyze)
    domain.ts                           -- Endpoint-uri Domain (legacy)
    media.ts                            -- Endpoint-uri Media/Fisiere (upload/download MinIO)
    pipeline/                           -- Pipeline complet + Istoric + Extensie + dry-run.ts + cancel.ts (cancel/resume) + async.ts
    moderation/                         -- HIL Moderation (queue.ts, flag-stats.ts, _gold-promotion.ts, _middleware.ts)
  components/
    component-runner.ts                 -- Orchestrator direct (invocare executori fara HTTP intern)
    techniques/
      executor.ts                       -- Executor Techniques: screening -> deep analysis -> scoring (params din Redis)
    ai-tampered/
      executor.ts                       -- Executor AI-Tampered: disclosure check -> screening -> deep -> scoring (params din Redis)
    claims/
      executor.ts                       -- Executor Claims: extractie afirmatii -> verificare web -> scoring (params din Redis)
    source-assessment/
      executor.ts                       -- Executor Source Assessment: extractie sursa -> M17 search -> evaluare (params din Redis)
    moderation/
      triage.ts                         -- Decide daca o sesiune intra in coada review (config Redis, feature-flag `triage_enabled`)
      queue-manager.ts                  -- Operatii PG pe `bos_analysis.moderation_queue` (enqueue/list/get/claim/resolve/stats)
    pipeline/
      executor.ts                       -- Orchestrator pipeline: ruleaza toate componentele + verdict
      verdict-calculator.ts             -- Calcul verdict cu input profiles per tip input (params din Redis)
      verdict-explanation.ts            -- Generare explicatie bilingva (RO+EN) via LLM
      virality-calculator.ts            -- Calcul scor viralitate (factori: emotie, urgenta, reach)
      types.ts                          -- Tipuri interne pipeline
  shared/
    media/
      transcription.ts                  -- Transcriere audio: M17-Whisper -> Groq -> OpenAI (fallback chain)
      vision.ts                         -- Analiza imagine: Qwen Local -> Gemini Flash -> GPT-4o (fallback chain)
      video-processor.ts               -- Pipeline video: download -> extrage cadre -> transcriere + viziune
      media-service.ts                  -- Interfata unificata media (download, format, cleanup)
    persistence/
      persist-service.ts                -- Coordonare Redis (cache) + PostgreSQL (permanent)
      pg-adapter.ts                     -- Adaptor PostgreSQL: save/load/history/delete pe schema bos_analysis
      pg-pool.ts                        -- Pool conexiuni PostgreSQL (singleton)
    redis/
      connection.ts                     -- Factory centralizat ioredis (ACL ready, retry, reconnectOnError)
      keys.ts                           -- Registru central chei Redis
      session-store.ts                  -- Stocare sesiuni in Redis ca JSON (TTL 7 zile)
    brain/
      client.ts                         -- Client didi-brain: gather cu verification lookup + write cache + analysis atom lookup/write/patch (gold/silver/bronze tiers, fire-and-forget writes)
    queue/
      constants.ts                      -- Topologie cozi: exchange, routing keys, prioritati
    types/
      component-results.ts              -- Definitii tipuri pentru toate cele 5 rezultate componente
      analysis-session.ts               -- Tipul AnalysisSession (wrapper root al unei analize)
      api-responses.ts                  -- Tipuri raspunsuri API (success/error envelope)
    test-utils/
      fixtures.ts                       -- Date de test (sesiuni, rezultate componente)
      mock-pg.ts                        -- Mock PostgreSQL pool pentru teste
      mock-redis.ts                     -- Mock Redis pentru teste
    helpers/
      empty-media-response.ts           -- Raspuns standardizat cand media nu are continut text
    auth/
      jwt-verify.ts                     -- Verificare criptografica JWT RS256 via Keycloak JWKS (Modul 7)
      guards.ts                         -- Route guards (resolveUserId, authRequired, requireRole, ROLES_ADMIN)
    observability/
      metrics.ts                        -- Metrici Prometheus business + OTel tracing + /metrics server workeri (Modul 8)
      queue-depth-poller.ts             -- Poller RabbitMQ Management API → gauge didi_queue_depth / didi_active_workers
    credits.ts                          -- Verificare si deducere credite utilizator (via didiFramework)
  config/
    analysisLimits.ts                   -- Limite text: min 20, max 50000 caractere + validare UTF-8
  queue/
    dispatcher.ts                       -- Dispatch mesaje catre RabbitMQ (async)
    connection.ts                       -- Manager conexiune RabbitMQ
    aggregator.ts                       -- Agregator verdict (consuma rezultate componente)
    types.ts                            -- Tipuri mesaje coada (dispatch, result, etc.)
    workers/
      component-worker.ts              -- Worker generic (consuma din coada, ruleaza componenta)
      techniques-worker.ts             -- Worker Techniques (config)
      ai-tampered-worker.ts            -- Worker AI-Tampered (config)
      claims-worker.ts                 -- Worker Claims (config)
  worker-entrypoints/
    techniques.ts                       -- Entry point Docker: worker techniques (2 replici)
    ai-tampered.ts                      -- Entry point Docker: worker ai-tampered (2 replici)
    claims.ts                           -- Entry point Docker: worker claims (3 replici)
    domain.ts                           -- Entry point Docker: worker domain (2 replici)
    aggregator.ts                       -- Entry point Docker: verdict aggregator (2 replici)
  scripts/
    export-schemas.ts                   -- Export scheme Zod ca JSON Schema
    init-priority-queues.ts             -- Initializare cozi RabbitMQ cu prioritati

Nota: queue/constants.ts (topologie cozi: exchange, routing keys, prioritati) se afla la shared/queue/constants.ts.

API - Toate endpoint-urile

General (/api/v3)

Metoda Path Ce face Logica in fisier
GET /health Health check (raspunde cu service name, version, timestamp) api/routes.ts

Nota: /health este montat pe routerul /api/v3, deci path-ul complet este /api/v3/health.

Techniques (/api/v3/techniques)

Metoda Path Ce face Logica in fisier
GET /definitions Ierarhie completa tehnici din Redis (dimensiuni -> subdimensiuni -> tehnici) api/routes.ts
GET /config Configurare completa din Redis api/routes.ts
GET /models Lista modele LLM disponibile api/routes.ts
GET /stage-assignments Configurare etape (screening/deep) api/routes.ts
PUT /stage-assignments Actualizeaza configurare etape api/routes.ts
POST /test-model Testeaza conexiunea la un model LLM api/routes.ts
POST /test-openrouter Testeaza OpenRouter cu provider routing api/routes.ts
POST /analyze Analiza text sincrona api/routes.ts -> components/techniques/executor.ts
POST /analyze-media Analiza media sincrona (imagine/audio/video) api/routes.ts -> shared/media/* -> components/techniques/executor.ts
POST /analyze-async Analiza asincrona via RabbitMQ api/routes.ts -> queue/dispatcher.ts
GET /results/:sessionId Rezultate intermediare din Redis api/routes.ts

AI-Tampered (/api/v3/ai-tampered)

Metoda Path Ce face Logica in fisier
GET /health Health check api/ai-tampered-routes.ts
GET /config Configurare completa din Redis api/ai-tampered-routes.ts
GET /models Lista modele LLM api/ai-tampered-routes.ts
GET /categories Categorii + ierarhie indicatori AI api/ai-tampered-routes.ts
GET /stage-assignments Configurare etape api/ai-tampered-routes.ts
PUT /stage-assignments Actualizeaza configurare etape api/ai-tampered-routes.ts
POST /quick Analiza rapida fara LLM (doar euristici) api/ai-tampered-routes.ts -> components/ai-tampered/executor.ts
POST /analyze Analiza text completa cu LLM api/ai-tampered-routes.ts -> components/ai-tampered/executor.ts
POST /analyze-image Analiza imagine pentru AI (doar viziune) api/ai-tampered-routes.ts -> shared/media/vision.ts
POST /analyze-media Analiza media sincrona api/ai-tampered-routes.ts -> shared/media/* -> components/ai-tampered/executor.ts
POST /analyze-async Analiza asincrona via RabbitMQ api/ai-tampered-routes.ts -> queue/dispatcher.ts
GET /results/:sessionId Rezultate intermediare din Redis api/ai-tampered-routes.ts
POST /test-model Testeaza model LLM api/ai-tampered-routes.ts

Claims (/api/v3/claims)

Metoda Path Ce face Logica in fisier
GET /health Health check api/claims-routes.ts
GET /config Configurare + tipuri/statusuri din framework api/claims-routes.ts
GET /types Tipuri de afirmatii (VF, EF, RE, etc.) api/claims-routes.ts
GET /statuses Statusuri verificare (VT, LT, UV, etc.) api/claims-routes.ts
GET /models Lista modele LLM api/claims-routes.ts
GET /stage-assignments Configurare etape api/claims-routes.ts
PUT /stage-assignments Actualizeaza configurare etape api/claims-routes.ts
POST /analyze Analiza text: extractie + verificare afirmatii api/claims-routes.ts -> components/claims/executor.ts
POST /analyze-media Analiza media sincrona api/claims-routes.ts -> shared/media/* -> components/claims/executor.ts
POST /analyze-async Analiza asincrona via RabbitMQ api/claims-routes.ts -> queue/dispatcher.ts
GET /results/:sessionId Rezultate intermediare din Redis api/claims-routes.ts
POST /test-model Testeaza model LLM api/claims-routes.ts

Pipeline complet (/api/v3/pipeline)

Metoda Path Ce face Logica in fisier
POST /analyze Analiza completa sincrona (toate componentele) api/pipeline-routes.ts -> components/pipeline/executor.ts
POST /analyze-url Analiza URL inteligenta (detecteaza YouTube, imagine, articol) api/pipeline-routes.ts -> shared/media/video-processor.ts
POST /analyze-media Analiza media sincrona (shortcut) api/pipeline-routes.ts -> components/pipeline/executor.ts
POST /analyze-async Analiza completa asincrona via RabbitMQ api/pipeline-routes.ts -> queue/dispatcher.ts
GET /:sessionId/status Status sesiune (polling) api/pipeline-routes.ts -> shared/redis/session-store.ts
GET /:sessionId/component/:name Rezultat o singura componenta api/pipeline-routes.ts -> shared/redis/session-store.ts
GET /:sessionId/result Rezultat complet sesiune api/pipeline-routes.ts -> shared/persistence/persist-service.ts
GET /:sessionId/queue-status Status coada async (progres %) api/pipeline-routes.ts -> queue/dispatcher.ts
GET /queue-health Sanatate RabbitMQ api/pipeline-routes.ts -> queue/connection.ts
GET /verdict-config Configurare verdict (categorii, ponderi, override-uri din Redis) api/pipeline-routes.ts
POST /dry-run Rezolvă planul complet de execuție fără dispatch (ce componente rulează/skip + de ce, chain-uri model/tier, profil verdict). Read-only: fără session id, fără credite, fără mesaje. Modul 1. api/pipeline/dry-run.ts
POST /:sessionId/cancel Anulează o sesiune în execuție (flag Redis; workerii short-circuit, aggregator finalizează canceled). Idempotent, owner/admin. api/pipeline/cancel.ts
POST /:sessionId/resume Resume din checkpoint — re-dispatch DOAR componentele necompletate (dispatcher.resumeDispatch). Owner/admin. api/pipeline/cancel.ts

Istoric (/api/v3/pipeline)

Metoda Path Ce face Logica in fisier
GET /history Istoric utilizator paginat api/pipeline-routes.ts -> shared/persistence/pg-adapter.ts
GET /history/:id O intrare din istoric api/pipeline-routes.ts -> shared/persistence/persist-service.ts
DELETE /history/:id Sterge intrare din istoric api/pipeline-routes.ts -> shared/persistence/persist-service.ts
GET /history/admin Istoric admin (toti utilizatorii, filtre) api/pipeline-routes.ts -> shared/persistence/pg-adapter.ts
GET /history/admin/:id O intrare admin api/pipeline-routes.ts -> shared/persistence/persist-service.ts
DELETE /history/admin/:id Sterge intrare admin api/pipeline-routes.ts -> shared/persistence/persist-service.ts

Extensie browser (/api/v3/pipeline/extension)

Metoda Path Ce face Logica in fisier
POST /analyze Analiza via API key (nu JWT) api/pipeline-routes.ts -> components/pipeline/executor.ts
POST /keys Creeaza API key pentru extensie api/pipeline-routes.ts (proxy -> didiFramework)
GET /keys Lista API keys api/pipeline-routes.ts (proxy -> didiFramework)
DELETE /keys/:id Sterge API key api/pipeline-routes.ts (proxy -> didiFramework)

Media/Fisiere (/api/v3/media)

Metoda Path Ce face Logica in fisier
POST /upload-url Obtine URL presemnat pentru upload in MinIO api/routes.ts
POST /upload Upload direct fisier (multipart, max 50MB) api/routes.ts
POST /download-url Obtine URL presemnat pentru download din MinIO api/routes.ts
GET /file/:bucket/{*objectKey} Proxy servire fisiere din MinIO (suporta range requests, Express 5 wildcard) api/routes.ts

Source Assessment (/api/v3/source-assessment)

Metoda Path Ce face Logica in fisier
GET /health Health check api/source-assessment-routes.ts
GET /config Configurare (scoring_config + framework sources) api/source-assessment-routes.ts
GET /models Modele LLM disponibile api/source-assessment-routes.ts
GET /stage-assignments Configurare etape api/source-assessment-routes.ts
PUT /stage-assignments Actualizeaza etape api/source-assessment-routes.ts
POST /test-model Test conectivitate model api/source-assessment-routes.ts
POST /analyze Analiza sursa (text + optional URL) api/source-assessment-routes.ts -> components/source-assessment/executor.ts
POST /analyze-media Analiza sursa din media api/source-assessment-routes.ts

Domain (/api/v3/domain) — LEGACY, inlocuit de Source Assessment

Metoda Path Ce face Logica in fisier
POST /analyze Analiza domeniu sincrona (apeleaza Domain Check API extern) api/routes.ts
POST /analyze-async Analiza domeniu asincrona api/routes.ts -> queue/dispatcher.ts

HIL Moderation (/api/v3/moderation)

Metoda Path Roluri permise Ce face Logica in fisier
GET /queue admin, moderator, senior_moderator Listeaza coada review (filtre: status CSV, priority CSV, assigned_to=me|user) api/moderation-routes.ts -> components/moderation/queue-manager.ts
GET /queue/:queueId admin, moderator, senior_moderator Intrare coada + sesiunea analiza JOIN-ed api/moderation-routes.ts -> components/moderation/queue-manager.ts
POST /queue/:queueId/claim moderator, senior_moderator Claim atomic (UPDATE conditionat pe pending) — assigneaza la user-ul JWT api/moderation-routes.ts -> components/moderation/queue-manager.ts
PUT /queue/:queueId/resolve moderator, senior_moderator Resolve cu actiune approved | corrected | rejected + corrections JSONB diff api/moderation-routes.ts -> components/moderation/queue-manager.ts -> shared/brain/client.ts (PATCH atom gold)
POST /flag orice user autenticat Flag user (extensie browser) — enqueue review api/moderation-routes.ts -> components/moderation/queue-manager.ts
GET /stats admin, moderator, senior_moderator Counts pe tier/component + hit_rate_24h api/moderation-routes.ts -> components/moderation/queue-manager.ts

Auth (Faza D done 2026-05-02): requireRole() și requireAuth() din moderation-routes.ts aplică role check strict când STAGING_MODE=false (production). În staging (STAGING_MODE=true, default actual): GET-urile de read trec fără JWT (legacy), /flag cere body user_id sau JWT. Vezi IMPLEMENTATION_PLAN_HIL_BRAIN.md Faza D pentru permission matrix completa.

Constants in moderation-routes.ts:

  • ROLES_READ = ['admin', 'moderator', 'senior_moderator'] — citire queue/stats
  • ROLES_WRITE = ['moderator', 'senior_moderator'] — claim/resolve (admin nu modifică queue, doar vede)
  • STAGING = process.env.STAGING_MODE === 'true' — controller pentru soft auth fallback

Autentificare JWT (Modul 7)

Fișier: src/shared/auth/jwt-verify.ts (+ guards în src/shared/auth/guards.ts).

Verificare criptografică a token-urilor JWT — nu doar la gateway, ci și în backend, deci un acces direct pe :24803 (bypass Kong) sau prin nginx-ul admin-dashboard este la fel de protejat.

  • RS256 via Keycloak JWKS: fiecare Bearer JWT e verificat împotriva JWKS-ului realm-ului emitent (didi-clients / didi-admins, allow-list JWT_ALLOWED_REALMS). JWKS descărcat remote și cache-uit per realm (createRemoteJWKSet din jose).
  • Respinge cu 401 (error_code: JWT_INVALID): token forjat (semnătură invalidă pe cheile noastre), expirat/nbf (validare exp/nbf), sau emis de un realm foreign (issuer în afara allow-list). Un token cu issuer didi-* spoofat tot pică pe semnătură.
  • Fără header Authorization → request rămâne anonim (route guards decid). Bearer non-JWT (API key opac) trece netransformat.
  • jwtIdentityMiddleware() populează req.jwtUserId / req.jwtEmail / req.jwtRoles din payload-ul verificat (realm_access.roles).
  • Env: KEYCLOAK_URL (default http://didi-keycloak:8080/auth), JWT_ALLOWED_REALMS, JWT_VERIFY_ENABLED (false → decode-only, doar dev, cu warning zgomotos).

Observabilitate (Modul 8)

Fișiere: src/shared/observability/metrics.ts + src/shared/observability/queue-depth-poller.ts.

Metrici Prometheus (client prom-client), inclusiv metrici business DiDi:

  • didi_analyses_completed_total / didi_analyses_failed_total (labels: component, tier, media_type, verdict/reason)
  • didi_pipeline_duration_seconds (histogram, per component+tier)
  • didi_llm_calls_total + didi_llm_tokens_total (provider, model, component)
  • didi_brain_cache_hits_total / ..._misses_total (verification + atom)
  • didi_queue_depth + didi_active_workers (gauge, per component+plan) — alimentate de queue-depth-poller.ts care interoghează RabbitMQ Management API la 30s (fail-open)
  • didi_dlq_messages_total, didi_http_requests_total, didi_http_request_duration_seconds + default Node/process metrics

Server /metrics: expus de agent-v3 (Express) și de fiecare container worker via startWorkerMetricsServer() (workerii n-au app Express, deci server HTTP minimal dedicat pentru scrape).

OpenTelemetry tracing: initObservability() pornește OTel NodeSDK cu auto-instrumentations (Express, http, redis, amqplib, pg) când OTEL_EXPORTER_OTLP_ENDPOINT e setat (default http://didi-otel-collector:4317); altfel disabled.

Spec OpenAPI

Fișier: openapi.yaml la root-ul serviciului (OpenAPI 3.0.3, Modulele 1-3). Descrie toate endpoint-urile (async 202 + poll, pipeline cu dependențe explicite, dry-run/cancel/resume, istoric, moderare HIL). Security: bearerAuth (JWT Keycloak RS256 verificat în backend, nu doar la gateway).

Fluxul de date

Analiza text (sincrona)

Request POST /api/v3/pipeline/analyze { text }
  |
  v
Validare text (20-50000 caractere) + Verificare credite
  |
  v
PipelineExecutor.execute()
  |
  v
ComponentRunner.runAll() -- ruleaza in paralel:
  |-- TechniquesV3Executor: screening -> deep analysis -> manipulation_score
  |-- AITamperedExecutor:   disclosure check -> screening -> deep -> ai_probability
  |-- ClaimsExecutor:       extractie afirmatii -> brain /v1/gather (cache check + evidence) -> fresh HIT skip LLM / miss: LLM verificare + fire-and-forget cache write -> credibility_score
  |-- analyzeDomain():      analiza locala (daca exista URL) -> trust_score
  |
  v
VerdictCalculator.calculate() -- functie pura, fara I/O
  |-- Selecteaza INPUT PROFILE per input_type (din Redis didi:config:pipeline:v1:input_profiles)
  |     Profiluri: text_no_url, text_with_url, image, audio, video, url
  |     Fiecare profil defineste: ponderi, override-uri active, reguli INCONCLUSIVE
  |-- Extrage scoruri componente (0-100)
  |-- Aplica ponderi din profil (ex: image: ai=50%, tech=20%, claims=15%, source=15%)
  |     Fallback la ponderi globale daca profilul nu exista
  |-- Aplica multiplicator topic INAINTE de override (nu amplifica bonusurile)
  |-- Aplica override-uri din profil (doar cele activate per input type):
  |     - synergy_bonus: 2+ componente cu risc >= threshold (+bonus/componenta, max cap)
  |     - false_claims: afirmatii false (+bonus/claim, max cap)
  |     - unverified_verifiable: claims verificabile neverificate (bonus fix)
  |     - severe_techniques: tehnici severe (bonus fix)
  |     - undisclosed_ai: AI nedeclarat (bonus fix)
  |     - untrusted_source: sursa neincrezuta/suspecta (bonus fix, din source_assessment)
  |     - blacklisted_source: sursa pe lista neagra (bonus fix, din source_assessment)
  |     - source_red_flags: red flags sursa (+bonus/flag, max cap)
  |     - override cap total per profil (default 50)
  |-- Rotunjire inainte de mapare la categorii
  |-- Mapare la risk_category, risk_level, severity (din Redis)
  |-- Calcul confidence (bonusuri/penalitati din profil)
  |-- INCONCLUSIVE check din profil (componenta primara crapat, min_components, required_any)
  |-- Virality: delegat la virality-calculator.ts (scor + factori + nivel)
  |
  v
PersistService.persist() -- salveaza in Redis + PostgreSQL
  |
  v
Response: AnalysisSession cu toate componentele

Analiza media (audio/video)

Request POST /api/v3/techniques/analyze-media { media_url, media_type }
  |
  v
Verificare credite
  |
  v
Procesare media:
  |-- Audio: transcribe() via M17-Whisper -> Groq -> OpenAI
  |-- Video: download (yt-dlp) -> extrage cadre (ffmpeg) -> transcriere + viziune
  |-- Imagine: callVision() OCR -> text extras
  |
  v
Text extras -> TechniquesV3Executor (la fel ca text)
  |
  v
PersistService.persist()
  |
  v
Response: AnalysisSession cu metadata extractie (durata, provider, cadre analizate)

Analiza asincrona

Request POST /api/v3/pipeline/analyze-async { text/media_url, plan_type }
  |
  v
Validare + Creditare
  |
  v
Dispatcher.dispatch() -> publica in RabbitMQ
  |
  v
Response 202: { session_id, poll_url, result_url }

--- In fundal (workeri Docker separati) ---

ComponentWorker consuma din coada -> ruleaza componenta -> publica rezultat in analysis.results
  |
  v
Aggregator consuma din analysis.results -> asteapta toate componentele -> VerdictCalculator
  |
  v
PersistService.persist() -> sesiune finala in Redis + PostgreSQL

--- Clientul polleaza ---

GET /api/v3/pipeline/{sessionId}/queue-status -> progres + componente finalizate
GET /api/v3/pipeline/{sessionId}/result -> sesiune completa cand status=completed

Servicii externe apelate

Serviciu Scop Unde in cod
Redis local (didi-cache:6379, DB 0, parolă redis123) Config framework + cache sesiuni (TTL 7 zile), acces prin createRedisConnection() shared/redis/*
PostgreSQL 17 local (didi-postgres:5432, DB DIDI, user bos_interface) Stocare permanenta (schema bos_analysis) shared/persistence/*
RabbitMQ local (staging-dataLayer-rabbitmq:5672, vhost /, user admin) Cozi async pentru workeri queue/*
MinIO (9000) Stocare fisiere media api/routes.ts (media endpoints)
OpenRouter API Modele LLM (Gemini, GPT-4o, Claude, etc.) components/*/executor.ts
OpenAI API LLM fallback + Whisper transcriere components/*/executor.ts, shared/media/transcription.ts
Groq API LLM fallback + Whisper transcriere components/*/executor.ts, shared/media/transcription.ts
M17-Whisper (M17_WHISPER_URL, default http://10.11.10.17:54300, configurabil din env) Transcriere audio primara shared/media/transcription.ts
M17 Web API (10.11.10.13:51100) Fallback cautare web — folosit cand brain dezactivat sau MISS fara evidence components/claims/executor.ts, components/source-assessment/executor.ts
didi-brain (Lot 1, DIDI_BRAIN_URL default http://10.11.10.12:8090) Verification cache claims (/v1/gather, /v1/verification_cache) + analysis atom cache techniques/ai-tampered (/v1/analysis_atom*) — vezi „Integrare cu platforma AI (Lot 1)" shared/brain/client.ts, components/*/executor.ts
Vision LLM (Lot 1, VISION_LLM_URL default http://10.11.10.17:14011) Analiza imagine (OCR + AI detection) shared/media/vision.ts
Domain Check API (:11000) Analiza domeniu (WHOIS, DNS, SSL, blacklist) api/routes.ts (domain/analyze)
didiFramework (didi-framework:3005) Verificare/deducere credite + API keys extensie + sync framework → Redis local shared/credits.ts, api/pipeline-routes.ts

Baza de date PostgreSQL

Schema: bos_analysis

Tabel Ce stocheaza Cheie
analysis_session Sesiunea root: user, input, status, timing, risk_score session_id (UUID)
analysis_techniques Rezultat componenta Techniques (JSONB: techniques_detected, coupling_context) session_id FK
analysis_ai_tampered Rezultat componenta AI-Tampered (JSONB: indicators_detected, image_analysis) session_id FK
analysis_claims Rezultat componenta Claims (JSONB: claims_verified, claims_by_status) session_id FK
analysis_domain Rezultat componenta Domain (trust_score, red_flags TEXT[], warnings) session_id FK
analysis_verdict Verdict final (risk_score, JSONB: applied_weights, context_summary) session_id FK
moderation_queue Coada review HIL (FK → analysis_session.session_id UUID): tier, priority, status, assigned_to, corrections JSONB queue_id (UUID)

Tabelul analysis_session are 6 coloane suplimentare pentru HIL: review_status, human_corrected, human_corrections (JSONB), verified_by, verified_at, review_notes (adaugate via migration 011_add_moderation.sql rulata de didiFramework).

In schema bos_parammgmt, didiFramework owneste 3 tabele de config citite de agent-v3 prin Redis: moderation_config (single row), sensitive_topic, moderation_role.

Salvarea este tranzactionala (all-or-nothing). Ce produce componenta = ce se salveaza (zero transformari).

Chei Redis importante

Framework (FrameworkKeys) — permanente, scrise de didiFramework sync-redis

Cheie Scop
didi:framework:manifest Index categorii config
didi:framework:techniques Ierarhie tehnici: dimensiuni -> subdimensiuni -> tehnici -> indicatori
didi:framework:sources Parametri credibilitate sursa
didi:framework:claims Parametri claims (tipuri, statusuri, confidence)
didi:framework:verdicts Categorii verdict, risk mappings, severity assessments
didi:framework:weights Ponderi componente, scenarii, multiplicatori
didi:framework:providers Config provideri LLM (optional)
didi:framework:dimensions_compact Lista compacta dimensiuni pentru prompts screening

Config componente (ConfigKeys) — permanente, scrise de didiFramework sync-redis

Toate stage_assignments sunt tier-nested (structura {stage_code: {free: {...}, premium: {...}}}) dupa migration 006 (tier column). Chain-urile de modele sunt diferite per tier — vezi sectiunea "Tier-aware execution" de mai sus.

Cheie Scop
didi:config:pipeline:v1:component_config Config executor pipeline + video track weights
didi:config:pipeline:v1:session_config Config sesiune pipeline
didi:config:pipeline:v1:verdict_config Override-uri, synergy, confidence (globale, fallback)
didi:config:pipeline:v1:input_profiles Profiluri verdict per input type (6 profile cu ponderi, override-uri, INCONCLUSIVE rules)
didi:config:techniques:v3:available_models Modele LLM Techniques (union free + premium)
didi:config:techniques:v3:stage_assignments Tier-nested asignari etape Techniques
didi:config:techniques:v3:scoring_config Parametri scoring: count_scaler, severe_threshold, intensity bonus, bonuses
didi:config:techniques:v3:dimensions_compact Lista compacta dimensiuni Techniques
didi:config:claims:v1:available_models Modele LLM Claims (union free + premium)
didi:config:claims:v1:stage_assignments Tier-nested asignari etape Claims
didi:config:claims:v1:scoring_config Parametri scoring: status_weights, unverified behavior, claim_type_weights
didi:config:ai-tampered:v1:available_models Modele LLM AI-Tampered (union free + premium)
didi:config:ai-tampered:v1:stage_assignments Tier-nested asignari etape AI-Tampered
didi:config:ai-tampered:v1:scoring_config Parametri scoring: blend_weights, disclosure_impact, thresholds, undisclosed_threshold
didi:config:ai-tampered:v1:vision_models Legacy flat vision config (fallback — nou e didi:config:vision:v1:stage_assignments)
didi:config:source-assessment:v1:available_models Modele LLM Source Assessment (union free + premium)
didi:config:source-assessment:v1:stage_assignments Tier-nested asignari etape Source Assessment
didi:config:source-assessment:v1:scoring_config Parametri scoring: axis_weights, verdict_thresholds
didi:config:source-assessment:v1:prompts:extraction Prompt extractie metadata sursa
didi:config:source-assessment:v1:prompts:evaluation Prompt evaluare sursa cu categorii framework
didi:config:verdict:v1:stage_assignments Tier-nested asignari Verdict reviewer (stage: verdict_review)
didi:config:verdict:v1:available_models Legacy fallback pentru verdict-explanation.ts
didi:config:vision:v1:stage_assignments Tier-nested asignari Vision (stage: image_analysis — OCR + AI detection + video frames)
didi:config:vision:v1:prompts:extraction Prompt viziune: extractie text din imagine
didi:config:vision:v1:prompts:video_frames Prompt viziune: analiza cadre video
didi:config:vision:v1:prompts:ai_detection Prompt viziune: detectie AI in imagine
didi:config:moderation:v1:settings Setari moderation + brain client (thresholds: confidence_low, risk_grey_min/max, feature flags: triage_enabled, brain_enabled, brain_url, brain_lookup_timeout_ms, brain_per_component) — citit cu cache 60s
didi:config:moderation:v1:sensitive_topics Lista topice sensibile pentru triage (auto-enqueue daca match)
didi:config:moderation:v1:roles Map roluri moderation (folosit la role check soft)

Toate cheile config au corespondent in PG (bos_parammgmt.component_config + component_prompt + component_stage_assignment + moderation_config + sensitive_topic + moderation_role). Sync-redis le citeste din PG si le scrie in Redis. Admin dashboard editeaza PG, apoi sync to Redis.

Sesiuni pipeline (SessionKeys) — TTL 7 zile

Pattern Scop
didi:pipeline:{sessionId}:status Status executie pipeline (JSON: PipelineStatus)
didi:pipeline:{sessionId}:{component} Rezultat componenta (JSON: TechniquesResult etc.)
didi:pipeline:{sessionId}:verdict Verdict final (JSON: VerdictResult)

Istoric (HistoryKeys) — TTL 7 zile

Pattern Scop
didi:pipeline:history:entry:{sessionId} Date intrare istoric (JSON: input + metadata)
didi:pipeline:history:user:{userId} Sorted set istoric utilizator (score=timestamp, member=sessionId)

Coada (QueueKeys) — TTL scurt (30s-5min)

Pattern Scop
didi:queue:session:{sessionId} Stare fan-in agregare (JSON: partial AnalysisSession)
didi:queue:lock:{sessionId}:{component} Lock executie componenta (previne duplicare worker)
didi:queue:aggregator:{sessionId} Lock agregator (previne calcul verdict concurent)

Rezultate intermediare (AgentKeys) — TTL 7 zile

Pattern Scop
agent:result:{sessionId}:{component}:{stage} Rezultat intermediar etapa (screening, deep_analysis)
agent:result:{sessionId}:ai-tampered:visual Rezultat analiza vizuala AI
agent:result:{sessionId}:ai-tampered:final Rezultat final AI tampered (media merged)
agent:result:{sessionId}:ai-tampered:media Rezultat intermediar AI tampered media

Media cache (MediaCacheKeys) — TTL 1 ora

Scrise de media-preprocess worker. Citite de component workers.

Pattern Scop
agent:media:{sessionId}:transcript Transcript audio (Whisper)
agent:media:{sessionId}:vision:misinformation Output Vision frames — analiza manipulare
agent:media:{sessionId}:vision:ai_detection Output Vision frames — detectie AI
agent:media:{sessionId}:merged_text Transcript + vision misinformation (pentru techniques + claims)
agent:media:{sessionId}:lock Lock procesare media (previne duplicare)
agent:media:{sessionId}:ready Semnal finalizare pre-procesare ("1")

Extensie (ExtensionKeys) — permanente

Pattern Scop
didi:extension:key:{apiKey} Cache API key extensie browser

Docker Compose - Servicii

Serviciu Replici Rol
agent-v3 1 Server API principal (port 24803)
worker-media-preprocess 2 Pre-procesare media: 1× download, 1× ffmpeg, 1× transcriptie, 2× Vision. Cache in Redis. Dispatche-aza component workers.
worker-techniques 2 Consuma din coada techniques. Citeste media din Redis cache.
worker-ai-tampered 2 Consuma din coada ai-tampered. Citeste media din Redis cache. 2-track video scoring (text 40% + visual 60%).
worker-claims 3 Consuma din coada claims (mai lent, mai multe replici). Citeste merged_text din Redis cache (transcript + text vizual).
worker-domain 2 Consuma din coada domain
verdict-aggregator 2 Colecteaza rezultate si calculeaza verdict

Dependinte sistem in container: ffmpeg, yt-dlp, python3 (pentru video download si procesare).

Fluxul async media (video/audio/image)

POST /api/v3/pipeline/analyze-async { media_type: "video" }
  |
  v
Dispatcher → analysis.media_preprocess.{plan} (1 task)
  |
  v
MediaPreprocessWorker:
  1. Download video (1×)
  2. ffmpeg: extract frames uniform (1×) + extract audio (1×)
  3. Whisper: transcriere audio (1×)
  4. Vision misinformation: analiza frames (1×)
  5. Vision ai_detection: analiza frames (1×)
  6. Salveaza in Redis (TTL 1h):
     - agent:media:{sid}:transcript
     - agent:media:{sid}:vision:misinformation
     - agent:media:{sid}:vision:ai_detection
     - agent:media:{sid}:merged_text
     - agent:media:{sid}:ready
  7. Dispatche-aza: techniques + ai_tampered + claims (+ domain daca URL)
  |
  v
Component Workers (citesc din Redis cache, zero download/ffmpeg):
  - techniques: primeste merged_text (transcript + vizual)
  - ai_tampered: primeste transcript (Track 1 text) + vision:ai_detection (Track 2 visual)
  - claims: primeste merged_text (transcript + text vizibil de pe ecran)
  - domain: primeste doar URL (skip pe media fara URL)
  |
  v
Verdict Aggregator → verdict (neschimbat)

Fallback: daca media preprocess esueaza, component workers proceseaza media local (safety net).

Fluxul async text/URL (neschimbat)

Dispatcher → direct la component workers (fara media preprocess).

Timeout-uri pe rute

Ruta Timeout Motiv
/api/v3/pipeline/* 660s (11 min) Download video + transcriere + analiza completa
/api/v3/techniques/analyze-media 300s (5 min) Transcriere media + analiza
/api/v3/ai-tampered/analyze-media 300s (5 min) Procesare media + analiza
/api/v3/claims/analyze-media 300s (5 min) Transcriere + extractie + verificare
Toate celelalte 180s (3 min) Analiza text standard (server.timeout)
keepAliveTimeout 665s Usor mai mare decat max per-route timeout
headersTimeout 670s Usor mai mare decat keepAliveTimeout

Tipuri de raspuns

AnalysisSession (raspuns standard pentru toate analizele)

{
  session_id, user_id, user_email,
  input_type (text/url/image/audio/video),
  input_text, input_url, input_media_url,
  input_hash,                               // SHA256
  status (pending/running/completed/failed),
  components_run[], components_skipped[],
  risk_score, risk_category, risk_level,
  confidence, confidence_level,
  started_at, completed_at, total_duration_ms,
  scenario_applied, topic_applied,
  source_app, api_version: "v3",
  created_at,

  techniques: {
    manipulation_score, total_severity, dimensions_affected[], techniques_count,
    techniques_detected[], coupling_context,
    llm_screening, llm_deep,
    screening_duration_ms, deep_analysis_duration_ms, total_duration_ms,
    fallbacks_screening, fallbacks_deep
  },
  ai_tampered: {
    ai_probability, verdict, risk_score, categories_affected[], indicators_count,
    disclosure_detected, disclosure_explicit, disclosure_text,
    indicators_detected[], coupling_context,
    llm_screening, llm_deep,
    screening_duration_ms, deep_analysis_duration_ms, total_duration_ms,
    fallbacks_screening, fallbacks_deep,
    content_type, image_analysis
  },
  claims: {
    total_claims, verified_true, verified_false, unverified, opinions,
    credibility_score, interpretation,
    claims_by_status, claims_by_type, claims_verified[],
    llm_extraction, llm_verification,
    extraction_duration_ms, verification_duration_ms, total_duration_ms,
    web_searches_made
  },
  domain: { domain, trust_score, verdict, risk_level, red_flags[], warnings[], duration_ms, ... },
  verdict: {
    risk_score, risk_category, risk_category_color,
    risk_level, risk_level_color,
    severity, recommended_action,
    confidence, confidence_level,
    score_manipulation, score_claims, score_ai, score_source, score_context,
    applied_weights, override_applied, override_type, override_reason, override_adjustment,
    context_summary, components_used[], weights_source, duration_ms,
    explanation_ro, explanation_en,
    virality_score, virality_level, virality_factors[]
  }
}

Raspuns async (202 Accepted)

{
  success: true, async: true,
  data: { session_id, status: "processing", queued_components[], plan_type, poll_url, result_url }
}

Raspuns media gol (cand nu s-a extras text din media)

{
  success: true,
  data: {
    session_id, status: "completed",
    input_type, input_media_url, media_type,
    skipped: true, skip_reason: "no_text_content" | "transcription_failed" | "vision_failed" | "extraction_failed",
    message: "...",
    techniques: null, ai_tampered: null, claims: null, domain: null, verdict: null,
    components_run: [], components_skipped: ["techniques", "ai_tampered", "claims"],
    risk_score: null, risk_category: null,
    metadata: { extraction_duration_ms, provider_tried }
  }
}

Schimbari de logica (2026-03-20)

Verdict INCONCLUSIVE determinist pe media fara claims

Fisier: components/pipeline/verdict-calculator.ts

Daca claims nu a produs rezultat (null) pe input video/audio/image → forteaza risk_category = INCONCLUSIVE. Inainte, LLM reviewer-ul decidea (inconsistent: +14 pe un video, +45 pe altul identic). Acum e determinist in cod.

LLM reviewer-ul (verdict-explanation.ts) nu mai penalizeaza pentru componente lipsa — instructiune explicita in prompt.

Claims primeste text vizual din video frames

Fisier: queue/workers/component-worker.ts

Claims primeste merged_text (transcript audio + text vizibil de pe ecran) in loc de doar transcript. Permite verificarea afirmatiilor scrise pe ecran (titluri, statistici, URL-uri).

Prompt-ul claims extraction (Redis didi:config:claims:v1:prompts:extraction) instruieste LLM-ul sa distinga intre text citat de pe ecran (extrage ca claim) si observatii Vision AI (ignora).

AI Tampered video: ponderi 40% text + 60% vizual

Fisier: queue/workers/component-worker.ts

Ponderile 2-track video scoring inversate: text 40% + vizual 60% (era text 60% + vizual 40%). Deepfake-uri cu voce reala dar fete AI primeau scor mic (34%) — vizualul trebuie sa domine pe video.

Prag minim text track: 200 chars. Sub 200 chars, text track e skip-uit si vizualul primeste 100% weight.

Prompt AI detection Vision — 3 tiere forensic

Redis: didi:config:vision:v1:prompts:ai_detection

Tier 1: artefacte grosolane (warping, extra fingers) → AI_CONFIDENCE 75-95 Tier 2: indicii subtile deepfake modern (face-background mismatch, skin uniformity, hair boundary, eye reflections) → AI_CONFIDENCE 45-75 Tier 3: evaluare context (face closeup TV = minim 30, nu 0) → AI_CONFIDENCE 30-40

AI_CONFIDENCE: 0 interzis pe face closeup video.

Techniques: intensity bonus diminishing returns

Fisier: components/techniques/executor.ts

Formula veche (liniara): (avgIntensity - 1) * 0.05 → intensity 8 = 0.35 (35pp bonus) Formula noua (sqrt + cap): min(0.15, sqrt(avgIntensity - 1) * 0.05) → intensity 8 = 0.13 (13pp bonus)

Previne scor 100% pe editoriale/jurnalism de investigatie cu ton puternic dar fara manipulare reala.

Limite durata media

Fisiere: api/pipeline-routes.ts (API check), shared/media/video-processor.ts (ffprobe check)

  • Video: max 180s (3 minute)
  • Audio: max 420s (7 minute)

Doua guard-uri: API (instant, daca frontend trimite media_duration_sec) + ffprobe (dupa download, safety net).

Optimizari claims

Fisiere: components/claims/executor.ts, components/component-runner.ts

  • Claims global timeout: 300s → 600s
  • Claims Qwen timeout: 60s → 120s (Redis config)
  • Claims cap: 10 → 7 (skip low priority)
  • Chunks paralele: 3 → 5
  • Transcript trunchiat la 4000 chars pentru claims pe media

Video frame sampling uniform

Fisier: shared/media/video-processor.ts

Inainte: fps=1/5 secvential → frames doar din primele 50s. Video 5 min = 83% ignorat. Acum: interval dinamic = durata / maxFrames → acoperire uniforma 100%. Minim 3 frames pe orice video.

Funcționalități suplimentare (în afara celor 9 module de caiet)

Secțiunile de mai jos sunt cod backend real, dar reprezintă funcționalitate în plus față de cele 9 module ale caietului de sarcini. Tot aici se încadrează și Tier-aware execution (premium vs free), Virality (virality-calculator.ts) și HIL Moderation (secțiunea următoare). Logica internă a brain-ului (RAG, tiering atom pe brain-side) aparține platformei AI (Lot 1); aici e descrisă doar integrarea client-side din agent-v3.

Brain v2 Atom Cache (integrare client-side)

Brain v2 introduce un cache de rezultate de analiza per componenta (analysis atoms), in plus fata de verification cache-ul existent pentru claims. Salveaza output-ul deep analysis pentru techniques si ai_tampered indexat pe (content_hash, component, tier, prompt_hash, framework_version). Hit fresh + tier potrivit (gold/silver) → skip LLM complet.

Tier-uri atom

Tier Cum se obtine Cum se foloseste
gold Promovat manual via PATCH din HIL Moderation (action: approved sau corrected) Hit fresh gold → folosit cu prioritate maxima
silver Auto-scris de agent-v3 dupa fiecare analiza reusita pe tier premium Hit fresh silver → folosit (overridable de gold)
bronze Tier intermediar pe brain side (rezervat) Ignorat de agent-v3 (skip LLM doar pe gold OR silver)

Brain refuza scrieri pentru tier free — atoms scrise doar din premium analyses (calitate consistenta).

Client extensie (src/shared/brain/client.ts)

Pe langa helperele existente (gatherFromBrain, writeVerificationCacheAsync, evidenceOverlap), clientul expune:

  • lookupAnalysisAtom({ content_hash, component, tier, prompt_hash, framework_version }) → { atom_id, data: AnalysisAtomData, cache_tier: AtomCacheTier, staleness: AtomStaleness } | null
  • writeAnalysisAtomAsync(payload) — fire-and-forget POST /v1/analysis_atom, doar pentru tier = premium
  • patchAnalysisAtomGold(atomId, corrections) — promoveaza atom la gold cu corectiile umane (apelat din moderation-routes.ts pe resolve)
  • helpers: computeContentHash(text), computePromptHash(prompt), computeFrameworkVersion(...)
  • types: AnalysisAtomData, AtomCacheTier ('gold' | 'silver' | 'bronze'), AtomStaleness ('fresh' | 'stale_framework' | 'stale_prompt' | 'miss')

Config & feature flags (Redis: didi:config:moderation:v1:settings, cache 60s)

Cheie Rol
brain_enabled Master switch — false → toate apelurile atom skip-uite, fallback direct LLM
brain_url Override per environment (default DIDI_BRAIN_URL env)
brain_lookup_timeout_ms Timeout per lookup (default tipic 800ms)
brain_per_component Granular: {techniques: bool, ai_tampered: bool, claims: bool}

Fail-open in toate cazurile: orice eroare brain (timeout, 5xx, network, feature flag off) → continua cu LLM. Atom cache nu poate bloca o analiza.

Integrare in executoare

Atat components/techniques/executor.ts cat si components/ai-tampered/executor.ts:

execute(input):
  // START — atom lookup
  if (brain_enabled && brain_per_component[component]) {
    atom = await lookupAnalysisAtom({ content_hash, component, tier, prompt_hash, framework_version })
    if (atom && atom.staleness === 'fresh' && (atom.cache_tier === 'gold' || atom.cache_tier === 'silver')) {
      return atomToComponentResult(atom.data)   // skip LLM complet
    }
  }

  // ... screening + deep analysis (LLM normal flow) ...

  // END — atom write (happy path AND early-exit branches)
  writeAnalysisAtomAsync({ ... })   // fire-and-forget, doar tier=premium
  return result

Atom write e plasat pe ambele ramuri (happy path si early-exit pe disclosure detected / no techniques) — orice rezultat reprezentativ se cache-uieste.

Promovare la gold via HIL

In moderation-routes.ts handler-ul PUT /queue/:queueId/resolve:

  • action = 'approved' → patch atomii existenti per componenta (techniques, ai_tampered, claims) la gold (semnal: omul confirma rezultatul automat)
  • action = 'corrected' → patch atomii la gold cu corrections aplicate (omul a editat output-ul automat — varianta corectata devine fontul de adevar)
  • action = 'rejected' → nu touch-uie atoms (rezultatul a fost gresit, dar nu avem corectie validata)

PATCH-ul foloseste (content_hash, component, tier, prompt_hash, framework_version) pentru a localiza atomii potriviti pe brain side.

Endpoint-uri brain v2 folosite

Metoda Path Folosit de
POST /v1/analysis_atom/lookup lookupAnalysisAtom() la START in techniques + ai-tampered
POST /v1/analysis_atom writeAnalysisAtomAsync() la END in techniques + ai-tampered (premium only)
PATCH /v1/analysis_atom/{id} patchAnalysisAtomGold() din HIL resolve
GET /v1/analysis_atom/stats Endpoint diagnostic (folosit la nevoie)
POST /v1/gather Existent — claims (verification cache lookup + evidence)
POST /v1/verification_cache Existent — claims write

HIL Moderation

Human-in-the-loop pentru sesiuni cu confidence scazut, scor in zona gri, sau topice sensibile. Triage-ul ruleaza dupa persistenta (NU blocheaza analiza) si decide daca sesiunea intra in coada de review uman. Moderatorii claim & resolve via REST API; rezolutiile pozitive (approved/corrected) promoveaza atomii la gold in brain.

Triage (src/components/moderation/triage.ts)

Citeste config din Redis (didi:config:moderation:v1:settings, cache 60s):

Threshold Rol
confidence_low Sub aceasta valoare → enqueue review (model nesigur)
risk_grey_min/max Banda gri risk_score (ex. 40-60) → enqueue review (rezultat ambiguu)
sensitive_topics Match in continut → enqueue review automat

Feature-flag protected: triage_enabled = false → triage no-op total.

Queue manager (src/components/moderation/queue-manager.ts)

Operatii directe pe bos_analysis.moderation_queue via shared/persistence/pg-pool.ts:

Functie Ce face
enqueueForReview(session, reason, priority) Idempotent — skip daca exista deja entry pending pentru acest session_id
listQueue(filters) Listare cu filtre status CSV, priority CSV, assigned_to
getQueueEntry(queueId) Entry + JOIN pe analysis_session (sesiunea full)
claimQueueEntry(queueId, userId) UPDATE atomic conditionat pe status = 'pending'
resolveQueueEntry(queueId, action, corrections, notes) UPDATE status + scrie human_corrected / human_corrections / verified_by / verified_at pe analysis_session
getQueueStats() Counts pe tier/component + hit_rate_24h

Integrare in pipeline

Triage hook apelat dupa persistService.persist(session) in:

  • src/components/pipeline/executor.ts (sync path)
  • src/queue/aggregator.ts (async path)

Wrapping try/catch — orice eroare in triage e log-uita, NU se propaga. Analiza nu se blocheaza niciodata din cauza moderation.

Role check soft

Middleware-ul JWT (src/index.ts) extrage req.jwtRoles din realm_access.roles. Endpoint-urile /api/v3/moderation/*:

  • Staging fara JWT roles (Keycloak nu trimite role): permit (pentru testing)
  • Productie cu JWT: require rol moderator (sau echivalent din didi:config:moderation:v1:roles)

Politica e soft (permit pe absent) ca sa nu blocheze flows-uri de testare interne.

userFlagged flow

POST /api/v3/moderation/flag permite oricarui user autenticat (sau extensiei browser) sa raporteze o sesiune. Cand sesiunea e creata cu user_flagged = true (request body field), flag-ul e plumbed prin:

  • PipelineInput.userFlagged (components/pipeline/types.ts)
  • SessionState.userFlagged (queue/types.ts)
  • api/pipeline-routes.ts extrage din body si forward-eaza

Triage citeste flag-ul si forteaza enqueue indiferent de threshold-uri (semnal uman = priority).