video-analysis — INDEX

Video analysis service for DIDI. Performs deepfake detection and semantic temporal analysis on video content via uniform / chunked frame extraction plus a vision-language model backend (vLLM). Exposes a small FastAPI surface used by agent-v3 for video media sessions.

Stack: Python 3.11+, FastAPI, uvicorn, OpenCV (headless), Pillow, NumPy, httpx/requests, ffmpeg toolchain (via OpenCV), pydantic-settings + YAML
URL (Dev): http://10.11.10.12:54600
Container: runs on GPU host, attached to the external bridge network didi-network (see deploy/docker-compose.yml); the service itself is CPU-only — GPU is consumed by the upstream vLLM server
Vision backend: external vLLM server (BusterX 7B @ port 54500); the same BusterX endpoint drives both deepfake detection and semantic per-chunk analysis. Semantic narratives are produced by aggregating the per-chunk descriptions with the DIDI text LLM (Qwen3.5) via http://didiAI-llm-api:14011. The wider DIDI vision cascade (Qwen Vision local → Gemini Flash → GPT-4o) lives in agent-v3; this service talks only to the BusterX vLLM.

Ce face

Receives a video file via multipart/form-data upload (no URL/path indirection — file bytes are POSTed)
Extracts frames via OpenCV (video_sampling.py) with two strategies:
- Uniform sampling for fast deepfake check (default 16 frames over the whole video)
- Temporal chunking for semantic analysis (default 10s chunks × 24 frames per chunk)
Encodes frames to JPEG (configurable max_side, jpeg_quality) and ships them to the vLLM server as base64 image payloads
Runs vision-language model inference and returns:
- verdict (REAL / FAKE / UNCERTAIN) plus explanation for the deepfake endpoint
- per-chunk description array + optional aggregated final_summary (narrative) for the semantic endpoint
Persists request artifacts (frames, prompts, responses) to a runs/<request_id>/ directory for reproducibility/debugging
Used by didi-backend agent-v3 for video media sessions (techniques + ai_tampered components)

API endpoints

Defined in src/video_analysis/app.py:

Endpoint	Method	Description
`/health`	GET	Liveness probe — `{"status":"ok"}`
`/analyze/video`	POST	Deepfake detection. Form field `file` (video). Fast path, ~16 frames. Returns `verdict`, `explanation`, `usage`, `latency_s`, `meta`.
`/analyze/video/semantic`	POST	Semantic temporal analysis. Form fields: `file`, `chunk_duration_s` (default 10.0, range 1–60), `frames_per_chunk` (default 24, range 4–64), `enable_aggregation` (default true). Returns `chunk_results[]`, optional `final_summary`, `meta`.

Auth: none (called over private network / through Kong upstream by agent-v3).

How didi-backend uses it

agent-v3 video pipeline calls /analyze/video for fast deepfake screening and (where enabled) /analyze/video/semantic for chunked scene narration
Output feeds the ai_tampered component (verdict + explanation) and contributes visual cues to the techniques component
Frame metadata (fps, duration_s, sampled, indices, timestamps_s) is surfaced upstream so the agent can correlate detections with timestamps
Long inference times (~77s for 60s video, semantic mode) are the reason agent-v3 routes video through the async session path (not the sync pipeline)

Frame extraction logic

Deepfake path: uniform interval sampling — interval = total_frames / frames where frames defaults to 16; min effectively 1 frame, capped by video length. Tunable via VIDEO_ANALYSIS_FRAMES.
Semantic path: temporal chunking — video is split into chunk_duration_s slices, each slice gets frames_per_chunk uniformly sampled frames; total frames analyzed scales with duration (e.g. 60s @ defaults → 6 chunks × 24 = 144 frames, ~8% of source).
Frames are downscaled so the longer side ≤ max_side (default 960 px), encoded JPEG at jpeg_quality (default 85), then base64-embedded into the chat-completions request.
All sampled frames + indices + timestamps are returned in meta and persisted under runs/<request_id>/.

Vision cascade

This service does not implement a multi-provider cascade. It is a thin client over a single vLLM endpoint configured at startup:

Deepfake + semantic (vision): BusterX (Qwen2.5-VL-7B fine-tune, l8cv/BusterX_plusplus, served as busterx) at VIDEO_ANALYSIS_VLLM_BASE_URL — typically http://didiAI-video-vllm-buster:54500 on the GPU host. The same endpoint handles both the deepfake verdict and the per-chunk semantic descriptions. BusterX is self-contained (Qwen2.5-VL is bundled inside the fine-tune) — it does not load a separate Qwen base model.
Semantic aggregation (text): per-chunk descriptions are merged into a narrative final_summary by the DIDI text LLM (Qwen3.5) via http://didiAI-llm-api:14011 — set through VIDEO_ANALYSIS_SEMANTIC_LLM_BASE_URL.
DIDI-wide cascade (Qwen Vision local → OpenRouter Gemini Flash → GPT-4o) is implemented in agent-v3, NOT here. This service is a leaf node in that chain — agent-v3 calls it as one of several vision options.
Service refuses to start if VIDEO_ANALYSIS_VLLM_BASE_URL is not set or the vLLM endpoint is unreachable (see buster_client.py, settings.py).

Configuration

Env vars (prefix VIDEO_ANALYSIS_), loaded from deploy/.env:

Variable	Required	Description
`VIDEO_ANALYSIS_VLLM_BASE_URL`	yes	Upstream vLLM server URL
`VIDEO_ANALYSIS_VLLM_MODEL`	yes	Served model name passed to vLLM (`busterx`; underlying weights `l8cv/BusterX_plusplus`)
`VIDEO_ANALYSIS_RUNS_DIR`	yes	Where to drop per-request artifacts (default `/app/runs` in container)
`VIDEO_ANALYSIS_EXTERNAL_URL`	yes	External URL embedded in the OpenAPI spec
`HF_TOKEN`, `HF_CACHE_DIR`	yes (when running bundled vLLM)	HuggingFace creds + shared cache for the vLLM container
`VIDEO_ANALYSIS_FRAMES`	no (default 16)	Uniform-sampling frame count
`VIDEO_ANALYSIS_MAX_SIDE`	no (default 960)	Frame downscale cap
`VIDEO_ANALYSIS_JPEG_QUALITY`	no (default 85)	JPEG quality 1–100
`VIDEO_ANALYSIS_MAX_TOKENS`	no (default 750)	Model response cap
`VIDEO_ANALYSIS_TEMPERATURE`	no (default 0.000001)	Near-deterministic decoding
`VIDEO_ANALYSIS_REPETITION_PENALTY`	no (default 1.05)	Repetition penalty
`NGINX_CONNECT_TIMEOUT` / `_SEND_TIMEOUT` / `_READ_TIMEOUT`	no	nginx upstream timeouts (only `api-nginx` profile)

Tuning defaults are baked into settings.py and overridden via the VIDEO_ANALYSIS_* env vars above (a deploy/config.yaml may optionally be supplied to override defaults, but none ships with the module).

Deployment

GPU host required for the vLLM upstream (NVIDIA driver 535+, NVIDIA Container Toolkit, ≥16 GB VRAM). The video-analysis container itself is CPU-only.
Compose lives in deploy/:
- deploy/docker-compose.yml — services + profiles (api, api-nginx)
- deploy/Dockerfile — Python 3.11 + OpenCV-headless + uv
- deploy/deploy.sh — wrapper around docker compose (loads env strictly from deploy/.env)
- deploy/nginx.conf / nginx.conf.template — optional reverse proxy
Typical bring-up:
- cp .env.example deploy/.env && $EDITOR deploy/.env
- cd deploy && ./deploy.sh --profile api --detach
Port allocation: 54600 video-analysis API, 54500 BusterX vLLM (Dev).
Restart: cd deploy && docker compose restart video-analysis-api (or docker restart video_analysis).

agent-v3 video pipeline — /home/admin365/didi_mono/backend/services/orchestration-layer/agent-v3 is the consumer; orchestrates async video sessions and merges this service's verdict into ai_tampered + techniques results
BusterX vLLM (port 54500) — sibling service in the AI platform; the actual GPU-backed model that this service queries for both deepfake and semantic chunks (referenced in main CLAUDE.md ports section)
DIDI text LLM API (http://didiAI-llm-api:14011) — Qwen3.5 endpoint used to aggregate semantic chunk descriptions into the final narrative
AI platform shared assets — ../../README.md, ../../ruff.toml
Internal package layout: src/video_analysis/{app.py, buster_client.py, schemas.py, settings.py, video_sampling.py}
Sibling docs: README.md, API.md, TESTING.md in this folder

8.3 KiB Raw Blame History Unescape Escape