Livrare LOT 1 - Didi
This commit is contained in:
commit
5380c3fc63
990 changed files with 133308 additions and 0 deletions
43
ai_platform/modules/video-analysis/.env.example
Normal file
43
ai_platform/modules/video-analysis/.env.example
Normal file
|
|
@ -0,0 +1,43 @@
|
|||
# video-analysis configuration
|
||||
# Copy to deploy/.env and fill ALL required values.
|
||||
# Service must FAIL to start if required vars are missing.
|
||||
# Tuning parameters are in deploy/config.yaml (not here)
|
||||
# =============================================================================
|
||||
# REQUIRED (no defaults)
|
||||
# =============================================================================
|
||||
# Hugging Face token for downloading BusterX model
|
||||
HF_TOKEN=your_huggingface_token_here
|
||||
|
||||
# Hugging Face cache directory (shared with llm-inference)
|
||||
HF_CACHE_DIR=/cai2_ds_storage/hf_cache
|
||||
|
||||
# Directory for storing analysis artifacts (created at runtime)
|
||||
VIDEO_ANALYSIS_RUNS_DIR=/app/runs
|
||||
|
||||
# =============================================================================
|
||||
# vLLM Connection (auto-configured for docker-compose profiles)
|
||||
# =============================================================================
|
||||
# For profile 'api-vllm' or 'full': uses internal vllm-buster container
|
||||
VIDEO_ANALYSIS_VLLM_BASE_URL=http://vllm-buster:8000
|
||||
VIDEO_ANALYSIS_VLLM_MODEL=busterx
|
||||
|
||||
# For profile 'api' with external vLLM: point to your external vLLM server
|
||||
# VIDEO_ANALYSIS_VLLM_BASE_URL=http://external-host:8008
|
||||
# VIDEO_ANALYSIS_VLLM_MODEL=l8cv/BusterX_plusplus
|
||||
|
||||
# =============================================================================
|
||||
# Optional Tuning (override config.yaml defaults)
|
||||
# =============================================================================
|
||||
# VIDEO_ANALYSIS_FRAMES=16
|
||||
# VIDEO_ANALYSIS_MAX_SIDE=960
|
||||
# VIDEO_ANALYSIS_JPEG_QUALITY=85
|
||||
# VIDEO_ANALYSIS_MAX_TOKENS=750
|
||||
# VIDEO_ANALYSIS_TEMPERATURE=0.000001
|
||||
# VIDEO_ANALYSIS_REPETITION_PENALTY=1.05
|
||||
|
||||
# =============================================================================
|
||||
# Optional Nginx Timeouts (for api-nginx or full profile)
|
||||
# =============================================================================
|
||||
# NGINX_CONNECT_TIMEOUT=60s
|
||||
# NGINX_SEND_TIMEOUT=120s
|
||||
# NGINX_READ_TIMEOUT=600s
|
||||
403
ai_platform/modules/video-analysis/API.md
Normal file
403
ai_platform/modules/video-analysis/API.md
Normal file
|
|
@ -0,0 +1,403 @@
|
|||
# Video Analysis API Documentation
|
||||
|
||||
REST API for video deepfake detection using semantic analysis via vLLM backends.
|
||||
Note: This API depends on an external vLLM server configured via `VIDEO_ANALYSIS_VLLM_BASE_URL`.
|
||||
|
||||
---
|
||||
|
||||
## Base URL
|
||||
|
||||
`{BASE_URL}`
|
||||
|
||||
Common configurations:
|
||||
- **Local development:** `http://localhost:54600`
|
||||
- **Docker (internal):** `http://didiAI-video-api:54600`
|
||||
- **Production:** Use your configured hostname
|
||||
|
||||
---
|
||||
|
||||
## Authentication
|
||||
|
||||
No authentication required.
|
||||
|
||||
---
|
||||
|
||||
## ⚠️ TESTING REMINDER: Alternative Vision Models
|
||||
|
||||
**Current Configuration:**
|
||||
- **Deepfake Detection:** Uses BusterX (Qwen2.5-VL-7B fine-tuned) @ port 54500
|
||||
- **Semantic Analysis:** Uses BusterX (7B parameters)
|
||||
|
||||
**TODO - Test with Qwen3-VL-30B for Better Semantic Analysis:**
|
||||
|
||||
The semantic analysis endpoint can be configured to use **Qwen3-VL-30B** (already running @ port 14002) instead of BusterX for potentially better results:
|
||||
|
||||
| Model | Size | Port | Best For |
|
||||
|-------|------|------|----------|
|
||||
| **BusterX** | 7B | 54500 | Deepfake detection (specialized) |
|
||||
| **Qwen3-VL-30B** | 30B | 14002 | General semantic understanding |
|
||||
|
||||
**To test with Qwen3-VL-30B:**
|
||||
|
||||
1. Update `.env`:
|
||||
```bash
|
||||
VIDEO_ANALYSIS_VLLM_BASE_URL=http://didiAI-llm-vllm-vision:14002 # Use Qwen3-VL instead of BusterX
|
||||
VIDEO_ANALYSIS_VLLM_MODEL=qwen3-vl # Change from busterx
|
||||
```
|
||||
|
||||
2. Rebuild container:
|
||||
```bash
|
||||
cd deploy/
|
||||
docker compose build video-analysis-api
|
||||
docker compose up -d video-analysis-api
|
||||
```
|
||||
|
||||
3. Test semantic analysis:
|
||||
```bash
|
||||
curl -X POST http://localhost:54600/analyze/video/semantic \
|
||||
-F "file=@test_video.mp4"
|
||||
```
|
||||
|
||||
**Expected Benefits:**
|
||||
- More detailed scene descriptions (30B vs 7B parameters)
|
||||
- Better context understanding
|
||||
- More coherent narrative flow
|
||||
- Higher accuracy for complex scenes
|
||||
|
||||
**Note:** Deepfake detection should continue using BusterX (specialized model).
|
||||
|
||||
---
|
||||
|
||||
## Endpoints
|
||||
|
||||
### Health Check
|
||||
|
||||
Check if the service is running.
|
||||
|
||||
**GET** `/health`
|
||||
|
||||
**Response**
|
||||
|
||||
```json
|
||||
{
|
||||
"status": "ok"
|
||||
}
|
||||
```
|
||||
|
||||
**Example**
|
||||
|
||||
```bash
|
||||
curl http://localhost:54600/health
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Analyze Video
|
||||
|
||||
Upload a video for deepfake analysis.
|
||||
|
||||
**POST** `/analyze/video`
|
||||
|
||||
#### Request
|
||||
|
||||
**Content-Type:** `multipart/form-data`
|
||||
|
||||
| Field | Type | Required | Description |
|
||||
|-------|------|----------|-------------|
|
||||
| `file` | file | Yes | Video file to analyze |
|
||||
|
||||
**Example**
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:54600/analyze/video \
|
||||
-F "file=@/path/to/video.mp4"
|
||||
```
|
||||
|
||||
#### Response
|
||||
|
||||
```json
|
||||
{
|
||||
"request_id": "550e8400-e29b-41d4-a716-446655440000",
|
||||
"run_dir": "runs/550e8400-e29b-41d4-a716-446655440000",
|
||||
"verdict": "FAKE",
|
||||
"explanation": "The video shows clear signs of manipulation...",
|
||||
"frames_analyzed": 16,
|
||||
"evidence": [
|
||||
{"frame_index": 0, "timestamp_s": 0.0},
|
||||
{"frame_index": 30, "timestamp_s": 1.0}
|
||||
],
|
||||
"usage": {
|
||||
"prompt_tokens": 1250,
|
||||
"completion_tokens": 150,
|
||||
"total_tokens": 1400
|
||||
},
|
||||
"latency_s": {
|
||||
"sampling_time_s": 0.234,
|
||||
"encode_time_s": 0.567,
|
||||
"model_inference_time_s": 12.345
|
||||
},
|
||||
"meta": {
|
||||
"fps": 30.0,
|
||||
"total_frames": 450,
|
||||
"duration_s": 15.0,
|
||||
"sampled": 16,
|
||||
"indices": [0, 30, 60, 90, 120, 150, 180, 210, 240, 270, 300, 330, 360, 390, 420, 449],
|
||||
"timestamps_s": [0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0, 10.0, 11.0, 12.0, 13.0, 14.0, 14.97]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
#### Response Fields
|
||||
|
||||
| Field | Description |
|
||||
|-------|-------------|
|
||||
| `request_id` | UUID identifying the analysis request |
|
||||
| `run_dir` | Directory where analysis artifacts are stored |
|
||||
| `verdict` | Classification result: `REAL`, `FAKE`, or `UNCERTAIN` |
|
||||
| `explanation` | Model explanation for the verdict |
|
||||
| `usage` | Token usage statistics from the vLLM backend |
|
||||
| `latency_s` | Timing breakdown in seconds |
|
||||
| `meta` | Video metadata and frame sampling information |
|
||||
|
||||
---
|
||||
|
||||
### Semantic Video Analysis
|
||||
|
||||
Upload a video for deep semantic analysis with temporal chunking.
|
||||
|
||||
**POST** `/analyze/video/semantic`
|
||||
|
||||
This endpoint performs comprehensive content understanding by:
|
||||
1. Dividing the video into temporal chunks (default: 10s each)
|
||||
2. Densely sampling each chunk (default: 24 frames per chunk)
|
||||
3. Analyzing each chunk with the vision LLM
|
||||
4. Optionally aggregating chunk descriptions into a coherent narrative
|
||||
|
||||
**Use cases:**
|
||||
- Content description and understanding
|
||||
- Action recognition and tracking
|
||||
- Scene analysis
|
||||
- Narrative extraction from video
|
||||
|
||||
#### Request
|
||||
|
||||
**Content-Type:** `multipart/form-data`
|
||||
|
||||
| Field | Type | Required | Default | Description |
|
||||
|-------|------|----------|---------|-------------|
|
||||
| `file` | file | Yes | - | Video file to analyze |
|
||||
| `chunk_duration_s` | float | No | 10.0 | Duration of each chunk in seconds (1-60) |
|
||||
| `frames_per_chunk` | int | No | 24 | Number of frames to sample per chunk (4-64) |
|
||||
| `enable_aggregation` | bool | No | true | Whether to aggregate chunks into final summary |
|
||||
|
||||
**Examples**
|
||||
|
||||
Basic semantic analysis (default settings):
|
||||
```bash
|
||||
curl -X POST http://localhost:54600/analyze/video/semantic \
|
||||
-F "file=@meeting.mp4"
|
||||
```
|
||||
|
||||
Custom chunk settings:
|
||||
```bash
|
||||
curl -X POST http://localhost:54600/analyze/video/semantic \
|
||||
-F "file=@meeting.mp4" \
|
||||
-F "chunk_duration_s=5.0" \
|
||||
-F "frames_per_chunk=32"
|
||||
```
|
||||
|
||||
Without aggregation (get only chunk descriptions):
|
||||
```bash
|
||||
curl -X POST http://localhost:54600/analyze/video/semantic \
|
||||
-F "file=@meeting.mp4" \
|
||||
-F "enable_aggregation=false"
|
||||
```
|
||||
|
||||
#### Response
|
||||
|
||||
```json
|
||||
{
|
||||
"request_id": "abc-123-def-456",
|
||||
"run_dir": "/app/runs/abc-123-def-456",
|
||||
"analysis_type": "semantic",
|
||||
"video_duration_s": 60.0,
|
||||
"num_chunks": 6,
|
||||
"chunk_results": [
|
||||
{
|
||||
"chunk_idx": 0,
|
||||
"time_range": "0.0s - 10.0s",
|
||||
"description": "A person in business attire enters a conference room, walks to the head of the table, and places a laptop down. The room has white walls and a large window showing daylight outside.",
|
||||
"frames_analyzed": 24,
|
||||
"inference_time_s": 12.3,
|
||||
"usage": {
|
||||
"prompt_tokens": 5234,
|
||||
"completion_tokens": 87,
|
||||
"total_tokens": 5321
|
||||
}
|
||||
},
|
||||
{
|
||||
"chunk_idx": 1,
|
||||
"time_range": "10.0s - 20.0s",
|
||||
"description": "The person opens the laptop and begins gesturing while speaking. Two other people enter the room and take seats at the conference table. One person carries a notebook.",
|
||||
"frames_analyzed": 24,
|
||||
"inference_time_s": 12.1,
|
||||
"usage": {
|
||||
"prompt_tokens": 5198,
|
||||
"completion_tokens": 92,
|
||||
"total_tokens": 5290
|
||||
}
|
||||
},
|
||||
{
|
||||
"chunk_idx": 2,
|
||||
"time_range": "20.0s - 30.0s",
|
||||
"description": "The presenter is now showing content on the laptop screen to the group. All three people are focused on the screen. One person is taking notes.",
|
||||
"frames_analyzed": 24,
|
||||
"inference_time_s": 11.8,
|
||||
"usage": {
|
||||
"prompt_tokens": 5201,
|
||||
"completion_tokens": 78,
|
||||
"total_tokens": 5279
|
||||
}
|
||||
},
|
||||
{
|
||||
"chunk_idx": 3,
|
||||
"time_range": "30.0s - 40.0s",
|
||||
"description": "Discussion is ongoing. The presenter is gesturing toward the screen. One attendee raises their hand and appears to ask a question.",
|
||||
"frames_analyzed": 24,
|
||||
"inference_time_s": 12.0,
|
||||
"usage": {
|
||||
"prompt_tokens": 5187,
|
||||
"completion_tokens": 71,
|
||||
"total_tokens": 5258
|
||||
}
|
||||
},
|
||||
{
|
||||
"chunk_idx": 4,
|
||||
"time_range": "40.0s - 50.0s",
|
||||
"description": "The presenter responds to the question with gestures. All participants are engaged in the discussion. Papers are visible on the table.",
|
||||
"frames_analyzed": 24,
|
||||
"inference_time_s": 11.9,
|
||||
"usage": {
|
||||
"prompt_tokens": 5209,
|
||||
"completion_tokens": 68,
|
||||
"total_tokens": 5277
|
||||
}
|
||||
},
|
||||
{
|
||||
"chunk_idx": 5,
|
||||
"time_range": "50.0s - 60.0s",
|
||||
"description": "The meeting appears to be concluding. Participants are gathering their belongings. The presenter closes the laptop and people begin standing up.",
|
||||
"frames_analyzed": 24,
|
||||
"inference_time_s": 12.2,
|
||||
"usage": {
|
||||
"prompt_tokens": 5223,
|
||||
"completion_tokens": 75,
|
||||
"total_tokens": 5298
|
||||
}
|
||||
}
|
||||
],
|
||||
"final_summary": "The video captures a business meeting in a conference room. It begins with a presenter setting up and two colleagues joining. The presenter delivers a presentation using a laptop, with the group discussing the content. One attendee asks questions and takes notes throughout. The meeting concludes with participants gathering their items and preparing to leave. The entire sequence lasts approximately 60 seconds.",
|
||||
"aggregation_time_s": 4.5,
|
||||
"total_latency_s": 76.8,
|
||||
"meta": {
|
||||
"fps": 30.0,
|
||||
"total_frames": 1800,
|
||||
"duration_s": 60.0,
|
||||
"chunk_duration_s": 10.0,
|
||||
"frames_per_chunk": 24,
|
||||
"total_frames_sampled": 144
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
#### Response Fields
|
||||
|
||||
| Field | Description |
|
||||
|-------|-------------|
|
||||
| `request_id` | UUID identifying the analysis request |
|
||||
| `run_dir` | Directory where analysis artifacts are stored |
|
||||
| `analysis_type` | Always "semantic" for this endpoint |
|
||||
| `video_duration_s` | Total video duration in seconds |
|
||||
| `num_chunks` | Number of temporal chunks processed |
|
||||
| `chunk_results` | Array of results for each chunk (see below) |
|
||||
| `final_summary` | Aggregated narrative (null if aggregation disabled) |
|
||||
| `aggregation_time_s` | Time spent on aggregation (null if disabled) |
|
||||
| `total_latency_s` | Total processing time in seconds |
|
||||
| `meta` | Video metadata and sampling configuration |
|
||||
|
||||
**ChunkResult Fields:**
|
||||
|
||||
| Field | Description |
|
||||
|-------|-------------|
|
||||
| `chunk_idx` | Zero-based chunk index |
|
||||
| `time_range` | Temporal range of this chunk (e.g., "0.0s - 10.0s") |
|
||||
| `description` | Semantic description of what happens in this chunk |
|
||||
| `frames_analyzed` | Number of frames analyzed for this chunk |
|
||||
| `inference_time_s` | Time spent on VLM inference for this chunk |
|
||||
| `usage` | Token usage statistics for this chunk |
|
||||
|
||||
#### Performance Characteristics
|
||||
|
||||
**For a 60-second video with default settings:**
|
||||
|
||||
- **Chunks:** 6 (10s each)
|
||||
- **Total frames analyzed:** 144 (24 per chunk)
|
||||
- **Coverage:** ~8% of all frames (vs 0.89% for deepfake detection)
|
||||
- **Latency:** ~77s total
|
||||
- Chunk processing: ~72s (6 × 12s per chunk)
|
||||
- Aggregation: ~5s
|
||||
- Overhead (sampling, encoding): <1s
|
||||
|
||||
**Optimization options:**
|
||||
|
||||
| Setting | Fast | Balanced | Detailed |
|
||||
|---------|------|----------|----------|
|
||||
| `chunk_duration_s` | 20.0 | 10.0 | 5.0 |
|
||||
| `frames_per_chunk` | 16 | 24 | 32 |
|
||||
| Coverage (60s video) | ~3% | ~8% | ~21% |
|
||||
| Latency estimate | ~40s | ~77s | ~150s |
|
||||
|
||||
---
|
||||
|
||||
## Error Responses
|
||||
|
||||
| Status | Description |
|
||||
|--------|-------------|
|
||||
| `400` | Bad Request (missing or invalid file) |
|
||||
| `500` | Internal Server Error |
|
||||
|
||||
### Possible 500 Error Causes
|
||||
|
||||
- vLLM backend unreachable
|
||||
- `VIDEO_ANALYSIS_VLLM_BASE_URL` not set
|
||||
- Model inference failed
|
||||
|
||||
**Example**
|
||||
|
||||
```json
|
||||
{
|
||||
"detail": "VIDEO_ANALYSIS_VLLM_BASE_URL is not set"
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Minimal Python Client Example
|
||||
|
||||
```python
|
||||
import httpx
|
||||
|
||||
def analyze_video(video_path: str) -> dict:
|
||||
with open(video_path, "rb") as f:
|
||||
response = httpx.post(
|
||||
"http://localhost:54600/analyze/video",
|
||||
files={"file": f},
|
||||
timeout=180.0,
|
||||
)
|
||||
response.raise_for_status()
|
||||
return response.json()
|
||||
|
||||
result = analyze_video("suspect_video.mp4")
|
||||
print(f"Verdict: {result['verdict']}")
|
||||
print(f"Explanation: {result['explanation']}")
|
||||
```
|
||||
100
ai_platform/modules/video-analysis/INDEX.md
Normal file
100
ai_platform/modules/video-analysis/INDEX.md
Normal file
|
|
@ -0,0 +1,100 @@
|
|||
# video-analysis — INDEX
|
||||
|
||||
Video analysis service for DIDI. Performs deepfake detection and semantic temporal analysis on video content via uniform / chunked frame extraction plus a vision-language model backend (vLLM). Exposes a small FastAPI surface used by `agent-v3` for video media sessions.
|
||||
|
||||
- **Stack:** Python 3.11+, FastAPI, uvicorn, OpenCV (headless), Pillow, NumPy, httpx/requests, ffmpeg toolchain (via OpenCV), pydantic-settings + YAML
|
||||
- **URL (Dev):** `http://10.11.10.12:54600`
|
||||
- **Container:** runs on GPU host (`network_mode: host` in `deploy/docker-compose.yml`); the service itself is CPU-only — GPU is consumed by the upstream vLLM server
|
||||
- **Vision backend:** external vLLM server (default: BusterX 7B @ port `54500`); same endpoint also drives semantic analysis. Optional alternative: Qwen3-VL-30B @ port `14002` for richer semantic narratives. The wider DIDI vision cascade (Qwen Vision local → Gemini Flash → GPT-4o) lives in `agent-v3`; this service only talks to one vLLM at a time.
|
||||
|
||||
## Ce face
|
||||
|
||||
- Receives a video file via `multipart/form-data` upload (no URL/path indirection — file bytes are POSTed)
|
||||
- Extracts frames via OpenCV (`video_sampling.py`) with two strategies:
|
||||
- **Uniform sampling** for fast deepfake check (default 16 frames over the whole video)
|
||||
- **Temporal chunking** for semantic analysis (default 10s chunks × 24 frames per chunk)
|
||||
- Encodes frames to JPEG (configurable `max_side`, `jpeg_quality`) and ships them to the vLLM server as base64 image payloads
|
||||
- Runs vision-language model inference and returns:
|
||||
- `verdict` (REAL / FAKE / UNCERTAIN) plus `explanation` for the deepfake endpoint
|
||||
- per-chunk `description` array + optional aggregated `final_summary` (narrative) for the semantic endpoint
|
||||
- Persists request artifacts (frames, prompts, responses) to a `runs/<request_id>/` directory for reproducibility/debugging
|
||||
- Used by `didi-backend` agent-v3 for video media sessions (techniques + ai_tampered components)
|
||||
|
||||
## API endpoints
|
||||
|
||||
Defined in `src/video_analysis/app.py`:
|
||||
|
||||
| Endpoint | Method | Description |
|
||||
|---|---|---|
|
||||
| `/health` | GET | Liveness probe — `{"status":"ok"}` |
|
||||
| `/analyze/video` | POST | Deepfake detection. Form field `file` (video). Fast path, ~16 frames. Returns `verdict`, `explanation`, `usage`, `latency_s`, `meta`. |
|
||||
| `/analyze/video/semantic` | POST | Semantic temporal analysis. Form fields: `file`, `chunk_duration_s` (default 10.0, range 1–60), `frames_per_chunk` (default 24, range 4–64), `enable_aggregation` (default true). Returns `chunk_results[]`, optional `final_summary`, `meta`. |
|
||||
|
||||
Auth: none (called over private network / through Kong upstream by agent-v3).
|
||||
|
||||
## How didi-backend uses it
|
||||
|
||||
- agent-v3 video pipeline calls `/analyze/video` for fast deepfake screening and (where enabled) `/analyze/video/semantic` for chunked scene narration
|
||||
- Output feeds the `ai_tampered` component (verdict + explanation) and contributes visual cues to the `techniques` component
|
||||
- Frame metadata (`fps`, `duration_s`, `sampled`, `indices`, `timestamps_s`) is surfaced upstream so the agent can correlate detections with timestamps
|
||||
- Long inference times (~77s for 60s video, semantic mode) are the reason agent-v3 routes video through the **async** session path (not the sync pipeline)
|
||||
|
||||
## Frame extraction logic
|
||||
|
||||
- **Deepfake path:** uniform interval sampling — `interval = total_frames / frames` where `frames` defaults to 16; min effectively 1 frame, capped by video length. Tunable via `VIDEO_ANALYSIS_FRAMES`.
|
||||
- **Semantic path:** temporal chunking — video is split into `chunk_duration_s` slices, each slice gets `frames_per_chunk` uniformly sampled frames; total frames analyzed scales with duration (e.g. 60s @ defaults → 6 chunks × 24 = 144 frames, ~8% of source).
|
||||
- Frames are downscaled so the longer side ≤ `max_side` (default 960 px), encoded JPEG at `jpeg_quality` (default 85), then base64-embedded into the chat-completions request.
|
||||
- All sampled frames + indices + timestamps are returned in `meta` and persisted under `runs/<request_id>/`.
|
||||
|
||||
## Vision cascade
|
||||
|
||||
This service does **not** implement a multi-provider cascade. It is a thin client over a single vLLM endpoint configured at startup:
|
||||
|
||||
- **Primary (deepfake):** BusterX (Qwen2.5-VL-7B fine-tune, `l8cv/BusterX_plusplus`) at `VIDEO_ANALYSIS_VLLM_BASE_URL` — typically `http://didiAI-video-vllm-buster:54500` on the GPU host
|
||||
- **Optional (semantic):** Qwen3-VL-30B at `http://didiAI-llm-vllm-vision:14002` — swap by editing `deploy/.env` and rebuilding
|
||||
- **DIDI-wide cascade** (Qwen Vision local → OpenRouter Gemini Flash → GPT-4o) is implemented in agent-v3, NOT here. This service is a leaf node in that chain — agent-v3 calls it as one of several vision options.
|
||||
- Service refuses to start if `VIDEO_ANALYSIS_VLLM_BASE_URL` is not set or the vLLM endpoint is unreachable (see `buster_client.py`, `settings.py`).
|
||||
|
||||
## Configuration
|
||||
|
||||
Env vars (prefix `VIDEO_ANALYSIS_`), loaded from `deploy/.env`:
|
||||
|
||||
| Variable | Required | Description |
|
||||
|---|---|---|
|
||||
| `VIDEO_ANALYSIS_VLLM_BASE_URL` | yes | Upstream vLLM server URL |
|
||||
| `VIDEO_ANALYSIS_VLLM_MODEL` | yes | Model name passed to vLLM (`busterx`, `qwen3-vl`, `l8cv/BusterX_plusplus`, …) |
|
||||
| `VIDEO_ANALYSIS_RUNS_DIR` | yes | Where to drop per-request artifacts (default `/app/runs` in container) |
|
||||
| `VIDEO_ANALYSIS_EXTERNAL_URL` | yes | External URL embedded in the OpenAPI spec |
|
||||
| `HF_TOKEN`, `HF_CACHE_DIR` | yes (when running bundled vLLM) | HuggingFace creds + shared cache for the vLLM container |
|
||||
| `VIDEO_ANALYSIS_FRAMES` | no (default 16) | Uniform-sampling frame count |
|
||||
| `VIDEO_ANALYSIS_MAX_SIDE` | no (default 960) | Frame downscale cap |
|
||||
| `VIDEO_ANALYSIS_JPEG_QUALITY` | no (default 85) | JPEG quality 1–100 |
|
||||
| `VIDEO_ANALYSIS_MAX_TOKENS` | no (default 750) | Model response cap |
|
||||
| `VIDEO_ANALYSIS_TEMPERATURE` | no (default 0.000001) | Near-deterministic decoding |
|
||||
| `VIDEO_ANALYSIS_REPETITION_PENALTY` | no (default 1.05) | Repetition penalty |
|
||||
| `NGINX_CONNECT_TIMEOUT` / `_SEND_TIMEOUT` / `_READ_TIMEOUT` | no | nginx upstream timeouts (only `api-nginx` profile) |
|
||||
|
||||
Tuning defaults live in `deploy/config.yaml`; env vars override YAML.
|
||||
|
||||
## Deployment
|
||||
|
||||
- GPU host required for the vLLM upstream (NVIDIA driver 535+, NVIDIA Container Toolkit, ≥16 GB VRAM). The video-analysis container itself is CPU-only.
|
||||
- Compose lives in `deploy/`:
|
||||
- `deploy/docker-compose.yml` — services + profiles (`api`, `api-nginx`)
|
||||
- `deploy/Dockerfile` — Python 3.11 + OpenCV-headless + uv
|
||||
- `deploy/deploy.sh` — wrapper around `docker compose` (loads env strictly from `deploy/.env`)
|
||||
- `deploy/nginx.conf` / `nginx.conf.template` — optional reverse proxy
|
||||
- Typical bring-up:
|
||||
- `cp .env.example deploy/.env && $EDITOR deploy/.env`
|
||||
- `cd deploy && ./deploy.sh --profile api --detach`
|
||||
- Port allocation: `54600` video-analysis API, `54500` BusterX vLLM (Dev).
|
||||
- Restart: `cd deploy && docker compose restart video-analysis-api` (or `docker restart video_analysis`).
|
||||
|
||||
## Related
|
||||
|
||||
- **agent-v3 video pipeline** — `/home/admin365/didi_mono/backend/services/orchestration-layer/agent-v3` is the consumer; orchestrates async video sessions and merges this service's verdict into `ai_tampered` + `techniques` results
|
||||
- **BusterX vLLM** (port `54500`) — sibling service in the AI platform; the actual GPU-backed model that this service queries (referenced in main `CLAUDE.md` ports section)
|
||||
- **Qwen3-VL vision vLLM** (port `14002`) — alternative semantic backend (`didiAI-llm-vllm-vision`)
|
||||
- **AI platform shared assets** — `../../README.md`, `../../ruff.toml`
|
||||
- **Internal package layout:** `src/video_analysis/{app.py, buster_client.py, schemas.py, settings.py, video_sampling.py}`
|
||||
- **Sibling docs:** `README.md`, `API.md`, `TESTING.md` in this folder
|
||||
275
ai_platform/modules/video-analysis/README.md
Normal file
275
ai_platform/modules/video-analysis/README.md
Normal file
|
|
@ -0,0 +1,275 @@
|
|||
# Video Analysis
|
||||
|
||||
Video analysis service for deepfake detection using semantic analysis via vLLM backends.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
**vLLM Server Requirement:**
|
||||
This service **does NOT start or manage a vLLM instance**.
|
||||
You must have a **vLLM-compatible server already running and accessible**
|
||||
from the machine where this service is deployed.
|
||||
|
||||
- By default, the service expects vLLM at the URL set in VIDEO_ANALYSIS_VLLM_BASE_URL
|
||||
- This can be changed via: VIDEO_ANALYSIS_VLLM_BASE_URL=http://<host>:<port>
|
||||
|
||||
**Important:**
|
||||
- If you run `video-analysis` in Docker with `network_mode: host`,
|
||||
the vLLM server must be reachable from the host network.
|
||||
- vLLM may run:
|
||||
- locally on the same machine, **or**
|
||||
- on another machine, as long as the URL is reachable.
|
||||
|
||||
The service will **fail to start** if the vLLM endpoint is not reachable.
|
||||
|
||||
**Required:**
|
||||
- All global prerequisites (see main [README.md](../../README.md))
|
||||
- A running vLLM server with a vision-language model (e.g., BusterX)
|
||||
|
||||
**vLLM Server Requirements:**
|
||||
- GPU required; VRAM depends on model (recommend ≥16GB, may require more).
|
||||
- NVIDIA Driver 535+
|
||||
- NVIDIA Container Toolkit
|
||||
|
||||
> **Note:** `video-analysis` itself runs on CPU. The GPU is only needed for the vLLM server.
|
||||
|
||||
## Features
|
||||
|
||||
- **Deepfake Detection**: Analyzes videos for signs of manipulation
|
||||
- **Uniform Frame Sampling**: Extracts representative frames from videos
|
||||
- **vLLM Integration**: Uses vision-language models for semantic analysis
|
||||
- **Reproducibility Artifacts (optional)**: Saves request/response artifacts to a configurable runs directory for debugging and reproducibility
|
||||
(this directory is created/used at runtime; it is not meant to be committed to the repo)
|
||||
|
||||
## Installation
|
||||
|
||||
```bash
|
||||
cd modules/video-analysis
|
||||
|
||||
# Install dependencies
|
||||
uv sync
|
||||
|
||||
# Install with dev dependencies
|
||||
uv sync --extra dev
|
||||
```
|
||||
|
||||
## Quick Start
|
||||
|
||||
### As an API Server
|
||||
|
||||
```bash
|
||||
cd deploy/
|
||||
|
||||
# Create deployment env file
|
||||
# NOTE: deploy.sh loads env ONLY from deploy/.env
|
||||
cp ../.env.example .env
|
||||
# Edit deploy/.env with your vLLM server URL and model
|
||||
|
||||
# Start the server
|
||||
./deploy.sh --profile api --detach
|
||||
|
||||
# Or with nginx reverse proxy
|
||||
./deploy.sh --profile api-nginx --detach
|
||||
```
|
||||
|
||||
### API Endpoints
|
||||
|
||||
| Endpoint | Method | Description |
|
||||
|----------|--------|-------------|
|
||||
| `/health` | GET | Health check |
|
||||
| `/analyze/video` | POST | Deepfake detection (fast, 16 frames) |
|
||||
| `/analyze/video/semantic` | POST | Semantic analysis (detailed, 144+ frames) |
|
||||
|
||||
### Example API Request
|
||||
|
||||
```bash
|
||||
# Health check
|
||||
curl http://localhost:54600/health
|
||||
|
||||
# Analyze video
|
||||
VIDEO="/path/to/video.mp4"
|
||||
curl -sS -X POST "http://localhost:54600/analyze/video" -F "file=@${VIDEO}"
|
||||
```
|
||||
|
||||
### Example Response
|
||||
|
||||
```json
|
||||
{
|
||||
"request_id": "550e8400-e29b-41d4-a716-446655440000",
|
||||
"run_dir": "runs/550e8400-e29b-41d4-a716-446655440000",
|
||||
"verdict": "FAKE",
|
||||
"explanation": "The video shows clear signs of manipulation...",
|
||||
"usage": {
|
||||
"prompt_tokens": 1250,
|
||||
"completion_tokens": 150,
|
||||
"total_tokens": 1400
|
||||
},
|
||||
"latency_s": {
|
||||
"sampling_time_s": 0.234,
|
||||
"encode_time_s": 0.567,
|
||||
"model_inference_time_s": 12.345
|
||||
},
|
||||
"meta": {
|
||||
"fps": 30.0,
|
||||
"total_frames": 450,
|
||||
"duration_s": 15.0,
|
||||
"sampled": 16
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
### Required Environment Variables
|
||||
|
||||
Configured via environment variables (prefix: `VIDEO_ANALYSIS_`). These are typically set in `deploy/.env`:
|
||||
|
||||
| Variable | Description |
|
||||
|----------|-------------|
|
||||
| `VIDEO_ANALYSIS_VLLM_BASE_URL` | vLLM server URL (e.g., `http://didiAI-video-vllm-buster:8000`) |
|
||||
| `VIDEO_ANALYSIS_VLLM_MODEL` | Model name (e.g., `l8cv/BusterX_plusplus`) |
|
||||
| `VIDEO_ANALYSIS_RUNS_DIR` | Directory for storing analysis artifacts (created/used at runtime) |
|
||||
|
||||
| `VIDEO_ANALYSIS_EXTERNAL_URL` | External URL for OpenAPI spec (e.g., `http://localhost:54600`) |
|
||||
|
||||
|
||||
### Optional Nginx Environment Variables (api-nginx profile)
|
||||
|
||||
If you use the `api-nginx` profile, the nginx container can read these optional variables from `deploy/.env`:
|
||||
|
||||
| Variable | Example | Description |
|
||||
|----------|---------|-------------|
|
||||
| `NGINX_CONNECT_TIMEOUT` | `60s` | Upstream connect timeout |
|
||||
| `NGINX_SEND_TIMEOUT` | `120s` | Upstream send timeout |
|
||||
| `NGINX_READ_TIMEOUT` | `600s` | Upstream read timeout |
|
||||
|
||||
### Optional Tuning Parameters
|
||||
|
||||
Configured via `deploy/config.yaml` (env vars override YAML):
|
||||
|
||||
| Parameter | Default | Description |
|
||||
|-----------|---------|-------------|
|
||||
| `frames` | `16` | Number of frames to sample |
|
||||
| `max_side` | `960` | Max image dimension (pixels) |
|
||||
| `jpeg_quality` | `85` | JPEG encoding quality (1-100) |
|
||||
| `max_tokens` | `750` | Max tokens for model response |
|
||||
| `temperature` | `0.000001` | Sampling temperature |
|
||||
| `repetition_penalty` | `1.05` | Repetition penalty |
|
||||
|
||||
## ⚠️ Testing Recommendations
|
||||
|
||||
### Model Selection for Semantic Analysis
|
||||
|
||||
**Current Setup:**
|
||||
- Both deepfake and semantic analysis use **BusterX** (7B parameters)
|
||||
- BusterX is optimized for deepfake detection
|
||||
|
||||
**TODO: Test Semantic Analysis with Qwen3-VL-30B**
|
||||
|
||||
For better semantic understanding, consider testing with the larger **Qwen3-VL-30B** model (already running @ port 8102):
|
||||
|
||||
```bash
|
||||
# Current (BusterX 7B)
|
||||
VIDEO_ANALYSIS_VLLM_BASE_URL=http://didiAI-video-vllm-buster:54500
|
||||
VIDEO_ANALYSIS_VLLM_MODEL=busterx
|
||||
|
||||
# Alternative (Qwen3-VL 30B) - Better for semantic analysis
|
||||
VIDEO_ANALYSIS_VLLM_BASE_URL=http://didiAI-llm-vllm-vision:14002
|
||||
VIDEO_ANALYSIS_VLLM_MODEL=qwen3-vl
|
||||
```
|
||||
|
||||
**Expected Improvements:**
|
||||
- ✅ More detailed scene descriptions (30B vs 7B)
|
||||
- ✅ Better understanding of complex actions
|
||||
- ✅ More coherent narrative synthesis
|
||||
- ✅ Higher quality semantic annotations
|
||||
|
||||
**Trade-offs:**
|
||||
- ⏱️ Slightly higher latency (~15-20s per chunk vs ~12s)
|
||||
- 📊 Better for semantic analysis, but keep BusterX for deepfake detection
|
||||
|
||||
**Recommendation:**
|
||||
- **Deepfake endpoint:** Keep using BusterX (specialized for forgery detection)
|
||||
- **Semantic endpoint:** Test with Qwen3-VL-30B for better results
|
||||
|
||||
## Deployment
|
||||
|
||||
```bash
|
||||
cd deploy/
|
||||
|
||||
# Create deployment env file (REQUIRED)
|
||||
cp ../.env.example .env
|
||||
# Edit deploy/.env with your settings
|
||||
|
||||
# API only
|
||||
./deploy.sh --profile api --detach
|
||||
|
||||
# API with nginx reverse proxy
|
||||
./deploy.sh --profile api-nginx --detach
|
||||
|
||||
# View logs
|
||||
./deploy.sh --profile api --logs
|
||||
|
||||
# Stop services
|
||||
./deploy.sh --profile api --down
|
||||
```
|
||||
|
||||
### Common Docker Commands
|
||||
|
||||
```bash
|
||||
cd deploy/
|
||||
|
||||
# Restart the API service (compose service name)
|
||||
docker compose restart video-analysis-api
|
||||
|
||||
# (Optional) restart by container name
|
||||
docker restart video_analysis
|
||||
```
|
||||
|
||||
### Port Allocation
|
||||
|
||||
| Port | Service | Environment |
|
||||
|------|---------|-------------|
|
||||
| `54600` | Video Analysis API | Development |
|
||||
| `54500` | BusterX vLLM Server | Development |
|
||||
|
||||
## Development
|
||||
|
||||
```bash
|
||||
# Install dev dependencies
|
||||
uv sync --extra dev
|
||||
|
||||
# Run tests
|
||||
uv run pytest
|
||||
|
||||
# Run tests with coverage
|
||||
uv run pytest --cov=src/video_analysis --cov-report=term-missing
|
||||
|
||||
# Lint and format
|
||||
uv run ruff check .
|
||||
uv run ruff format .
|
||||
```
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
modules/video-analysis/
|
||||
├── deploy/
|
||||
│ ├── deploy.sh # Deployment script
|
||||
│ ├── docker-compose.yml # Docker services
|
||||
│ ├── Dockerfile # Container image
|
||||
│ ├── nginx.conf # Nginx reverse proxy config (optional)
|
||||
│ ├── nginx.conf.template # Template-based nginx config (optional)
|
||||
│ └── config.yaml # Tuning parameters
|
||||
├── src/video_analysis/
|
||||
│ ├── __init__.py
|
||||
│ ├── app.py # FastAPI application
|
||||
│ ├── buster_client.py # vLLM client
|
||||
│ ├── schemas.py # Response schemas
|
||||
│ ├── settings.py # Configuration (env + yaml)
|
||||
│ └── video_sampling.py # Frame extraction
|
||||
├── tests/
|
||||
├── .env.example # Environment template
|
||||
├── API.md # API documentation
|
||||
├── pyproject.toml # Dependencies
|
||||
└── README.md # This file
|
||||
```
|
||||
252
ai_platform/modules/video-analysis/TESTING.md
Normal file
252
ai_platform/modules/video-analysis/TESTING.md
Normal file
|
|
@ -0,0 +1,252 @@
|
|||
# Video Analysis - Testing Checklist
|
||||
|
||||
## ⚠️ HIGH PRIORITY: Model Comparison for Semantic Analysis
|
||||
|
||||
### Background
|
||||
|
||||
Currently both deepfake detection and semantic analysis use **BusterX** (Qwen2.5-VL-7B, 7B parameters). However, we have access to a much larger model **Qwen3-VL-30B** (30B parameters) that could provide significantly better semantic understanding.
|
||||
|
||||
### Hypothesis
|
||||
|
||||
Semantic analysis (content understanding, scene description, narrative) would benefit from the larger Qwen3-VL-30B model, while deepfake detection should continue using the specialized BusterX model.
|
||||
|
||||
---
|
||||
|
||||
## Test Plan: Qwen3-VL-30B for Semantic Analysis
|
||||
|
||||
### Current Configuration
|
||||
|
||||
```bash
|
||||
# modules/video-analysis/deploy/.env
|
||||
VIDEO_ANALYSIS_VLLM_BASE_URL=http://didiAI-video-vllm-buster:54500 # Port 54500
|
||||
VIDEO_ANALYSIS_VLLM_MODEL=busterx # 7B parameters
|
||||
```
|
||||
|
||||
### Test Configuration
|
||||
|
||||
```bash
|
||||
# modules/video-analysis/deploy/.env
|
||||
VIDEO_ANALYSIS_VLLM_BASE_URL=http://didiAI-llm-vllm-vision:14002 # Port 14002
|
||||
VIDEO_ANALYSIS_VLLM_MODEL=qwen3-vl # 30B parameters
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Testing Procedure
|
||||
|
||||
### Step 1: Baseline Test (BusterX 7B)
|
||||
|
||||
```bash
|
||||
# Current configuration - no changes needed
|
||||
cd /home/vasi/ml-projects/modules/video-analysis/deploy
|
||||
|
||||
# Test semantic analysis with BusterX
|
||||
curl -X POST http://localhost:8007/analyze/video/semantic \
|
||||
-F "file=@test_video_60s.mp4" \
|
||||
-F "chunk_duration_s=10.0" \
|
||||
-F "frames_per_chunk=24" \
|
||||
-o baseline_busterx.json
|
||||
|
||||
# Review results
|
||||
jq '{
|
||||
model: "BusterX-7B",
|
||||
num_chunks: .num_chunks,
|
||||
chunk_descriptions: [.chunk_results[].description],
|
||||
final_summary: .final_summary,
|
||||
total_latency_s: .total_latency_s
|
||||
}' baseline_busterx.json
|
||||
```
|
||||
|
||||
### Step 2: Test with Qwen3-VL-30B
|
||||
|
||||
```bash
|
||||
cd /home/vasi/ml-projects/modules/video-analysis/deploy
|
||||
|
||||
# Backup current config
|
||||
cp .env .env.backup
|
||||
|
||||
# Update to use Qwen3-VL
|
||||
cat > .env << 'ENVFILE'
|
||||
HF_TOKEN=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
|
||||
HF_CACHE_DIR=/cai2_ds_storage/hf_cache
|
||||
VIDEO_ANALYSIS_RUNS_DIR=/app/runs
|
||||
|
||||
# Switch to Qwen3-VL for testing
|
||||
VIDEO_ANALYSIS_VLLM_BASE_URL=http://didiAI-llm-vllm-vision:14002
|
||||
VIDEO_ANALYSIS_VLLM_MODEL=qwen3-vl
|
||||
|
||||
# Semantic analysis settings
|
||||
VIDEO_ANALYSIS_SEMANTIC_CHUNK_DURATION_S=10.0
|
||||
VIDEO_ANALYSIS_SEMANTIC_FRAMES_PER_CHUNK=24
|
||||
VIDEO_ANALYSIS_SEMANTIC_ENABLE_AGGREGATION=true
|
||||
VIDEO_ANALYSIS_SEMANTIC_AGGREGATION_MODEL=gpt-oss-120b
|
||||
VIDEO_ANALYSIS_SEMANTIC_LLM_BASE_URL=http://deploy-llm-api-1:8100
|
||||
ENVFILE
|
||||
|
||||
# Rebuild and restart
|
||||
docker compose build video-analysis-api
|
||||
docker compose up -d video-analysis-api
|
||||
|
||||
# Wait for startup
|
||||
sleep 10
|
||||
|
||||
# Test with same video
|
||||
curl -X POST http://localhost:8007/analyze/video/semantic \
|
||||
-F "file=@test_video_60s.mp4" \
|
||||
-F "chunk_duration_s=10.0" \
|
||||
-F "frames_per_chunk=24" \
|
||||
-o test_qwen3vl.json
|
||||
|
||||
# Review results
|
||||
jq '{
|
||||
model: "Qwen3-VL-30B",
|
||||
num_chunks: .num_chunks,
|
||||
chunk_descriptions: [.chunk_results[].description],
|
||||
final_summary: .final_summary,
|
||||
total_latency_s: .total_latency_s
|
||||
}' test_qwen3vl.json
|
||||
```
|
||||
|
||||
### Step 3: Compare Results
|
||||
|
||||
```bash
|
||||
# Side-by-side comparison
|
||||
echo "=== BusterX 7B ==="
|
||||
jq -r '.chunk_results[0].description' baseline_busterx.json
|
||||
echo ""
|
||||
echo "=== Qwen3-VL 30B ==="
|
||||
jq -r '.chunk_results[0].description' test_qwen3vl.json
|
||||
echo ""
|
||||
|
||||
echo "=== Final Summaries ==="
|
||||
echo "BusterX: $(jq -r '.final_summary' baseline_busterx.json)"
|
||||
echo ""
|
||||
echo "Qwen3-VL: $(jq -r '.final_summary' test_qwen3vl.json)"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Evaluation Criteria
|
||||
|
||||
### Quality Metrics
|
||||
|
||||
1. **Description Detail**
|
||||
- [ ] More specific object identification
|
||||
- [ ] Better action recognition
|
||||
- [ ] More context understanding
|
||||
|
||||
2. **Narrative Coherence**
|
||||
- [ ] Logical flow between segments
|
||||
- [ ] Temporal consistency
|
||||
- [ ] Better story understanding
|
||||
|
||||
3. **Accuracy**
|
||||
- [ ] Correct identification of people/objects
|
||||
- [ ] Accurate scene descriptions
|
||||
- [ ] Proper action sequences
|
||||
|
||||
### Performance Metrics
|
||||
|
||||
| Metric | BusterX 7B | Qwen3-VL 30B | Difference |
|
||||
|--------|-----------|--------------|------------|
|
||||
| Latency per chunk | ~12s | ~??s | ?? |
|
||||
| Total latency (60s video) | ~77s | ~??s | ?? |
|
||||
| Token usage per chunk | ~5200 | ~????? | ?? |
|
||||
|
||||
---
|
||||
|
||||
## Expected Outcomes
|
||||
|
||||
### If Qwen3-VL is Better:
|
||||
|
||||
**Action:** Update default configuration to use Qwen3-VL for semantic analysis
|
||||
|
||||
```bash
|
||||
# Keep two separate configs:
|
||||
# 1. Deepfake endpoint → BusterX (specialized)
|
||||
# 2. Semantic endpoint → Qwen3-VL (better understanding)
|
||||
```
|
||||
|
||||
**Implementation:**
|
||||
- Add `VIDEO_ANALYSIS_DEEPFAKE_MODEL` and `VIDEO_ANALYSIS_SEMANTIC_MODEL` settings
|
||||
- Configure different models per endpoint
|
||||
|
||||
### If BusterX is Sufficient:
|
||||
|
||||
**Action:** Keep current configuration, document findings
|
||||
|
||||
**Rationale:**
|
||||
- Latency advantage (30B model is slower)
|
||||
- VRAM savings
|
||||
- BusterX might be sufficient for semantic tasks
|
||||
|
||||
---
|
||||
|
||||
## Test Videos
|
||||
|
||||
Suggested test scenarios:
|
||||
|
||||
1. **Meeting/Conference** (60s)
|
||||
- Multiple people
|
||||
- Complex interactions
|
||||
- Scene changes
|
||||
|
||||
2. **Action Sequence** (60s)
|
||||
- Fast movements
|
||||
- Object manipulation
|
||||
- Environmental changes
|
||||
|
||||
3. **Indoor/Outdoor Transition** (60s)
|
||||
- Lighting changes
|
||||
- Multiple scenes
|
||||
- Context shifts
|
||||
|
||||
---
|
||||
|
||||
## Rollback Procedure
|
||||
|
||||
If issues occur:
|
||||
|
||||
```bash
|
||||
cd /home/vasi/ml-projects/modules/video-analysis/deploy
|
||||
|
||||
# Restore original config
|
||||
cp .env.backup .env
|
||||
|
||||
# Rebuild
|
||||
docker compose build video-analysis-api
|
||||
docker compose up -d video-analysis-api
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Status
|
||||
|
||||
- [ ] Baseline test completed (BusterX)
|
||||
- [ ] Test with Qwen3-VL completed
|
||||
- [ ] Results compared
|
||||
- [ ] Decision made
|
||||
- [ ] Configuration updated (if needed)
|
||||
- [ ] Documentation updated
|
||||
|
||||
---
|
||||
|
||||
## Notes
|
||||
|
||||
Add observations here during testing:
|
||||
|
||||
```
|
||||
Date: ___________
|
||||
Tester: _________
|
||||
|
||||
Observations:
|
||||
|
||||
|
||||
|
||||
|
||||
Recommendation:
|
||||
|
||||
|
||||
|
||||
```
|
||||
|
||||
41
ai_platform/modules/video-analysis/deploy/Dockerfile
Normal file
41
ai_platform/modules/video-analysis/deploy/Dockerfile
Normal file
|
|
@ -0,0 +1,41 @@
|
|||
# Video Analysis API Server
|
||||
# Multi-stage build for smaller final image
|
||||
|
||||
FROM python:3.11.12-slim AS builder
|
||||
|
||||
# Install uv
|
||||
COPY --from=ghcr.io/astral-sh/uv:0.10 /uv /usr/local/bin/uv
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
# Copy project files
|
||||
COPY pyproject.toml uv.lock README.md ./
|
||||
COPY src/ ./src/
|
||||
|
||||
# Install dependencies (allow resolving to pick up new deps)
|
||||
RUN uv sync --no-dev
|
||||
|
||||
# Production image
|
||||
FROM python:3.11.12-slim
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
# Copy virtual environment from builder
|
||||
COPY --from=builder /app/.venv /app/.venv
|
||||
|
||||
# Copy source code
|
||||
COPY src/ ./src/
|
||||
|
||||
ENV PATH="/app/.venv/bin:$PATH"
|
||||
ENV PYTHONUNBUFFERED=1
|
||||
ENV VIDEO_ANALYSIS_PORT=54600
|
||||
|
||||
# Health check (python urllib; no curl in slim)
|
||||
HEALTHCHECK --interval=30s --timeout=10s --retries=3 \
|
||||
CMD python -c "import urllib.request; import os; urllib.request.urlopen(f'http://localhost:{os.environ.get(\"VIDEO_ANALYSIS_PORT\", 54600)}/health')" || exit 1
|
||||
|
||||
EXPOSE 54600
|
||||
|
||||
CMD uvicorn video_analysis.app:app --host 0.0.0.0 --port ${VIDEO_ANALYSIS_PORT}
|
||||
|
||||
|
||||
116
ai_platform/modules/video-analysis/deploy/deploy.sh
Normal file
116
ai_platform/modules/video-analysis/deploy/deploy.sh
Normal file
|
|
@ -0,0 +1,116 @@
|
|||
#!/usr/bin/env bash
|
||||
#
|
||||
# Docker Compose Startup Script for Video Analysis
|
||||
#
|
||||
# Usage: ./deploy/deploy.sh [OPTIONS]
|
||||
#
|
||||
# Options:
|
||||
# --profile <api|api-vllm> Docker compose profile
|
||||
# --detach Run in detached mode
|
||||
# --down Stop and remove containers
|
||||
# --logs Show logs
|
||||
# --help Show this help message
|
||||
#
|
||||
# Required: Set environment variables in deploy/.env file or export them before running.
|
||||
# See ../.env.example (module root) for the full list of variables.
|
||||
#
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
# Load .env file ONLY from deploy/ directory
|
||||
if [[ -f "$SCRIPT_DIR/.env" ]]; then
|
||||
echo "Loading environment from: $SCRIPT_DIR/.env"
|
||||
set -a
|
||||
source "$SCRIPT_DIR/.env"
|
||||
set +a
|
||||
fi
|
||||
|
||||
PROFILE=""
|
||||
DETACH=""
|
||||
ACTION="up"
|
||||
|
||||
show_help() {
|
||||
sed -n '2,18p' "$0" | sed 's/^# //' | sed 's/^#//'
|
||||
exit 0
|
||||
}
|
||||
|
||||
check_required_var() {
|
||||
local var_name="$1"
|
||||
if [[ -z "${!var_name:-}" ]]; then
|
||||
echo "ERROR: Required environment variable $var_name is not set"
|
||||
echo "Set it in deploy/.env file or export it before running this script"
|
||||
exit 1
|
||||
fi
|
||||
}
|
||||
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case $1 in
|
||||
--profile)
|
||||
PROFILE="$2"
|
||||
shift 2
|
||||
;;
|
||||
--detach|-d)
|
||||
DETACH="-d"
|
||||
shift
|
||||
;;
|
||||
--down)
|
||||
ACTION="down"
|
||||
shift
|
||||
;;
|
||||
--logs)
|
||||
ACTION="logs"
|
||||
shift
|
||||
;;
|
||||
--help|-h)
|
||||
show_help
|
||||
;;
|
||||
*)
|
||||
echo "Unknown option: $1"
|
||||
echo "Use --help for usage information"
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
done
|
||||
|
||||
# Require profile when bringing up
|
||||
if [[ -z "$PROFILE" && "$ACTION" == "up" ]]; then
|
||||
echo "ERROR: --profile is required"
|
||||
echo "Options: api, api-vllm"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Fail-fast required vars for wrapper <-> vLLM wiring
|
||||
check_required_var "VIDEO_ANALYSIS_VLLM_BASE_URL"
|
||||
check_required_var "VIDEO_ANALYSIS_VLLM_MODEL"
|
||||
check_required_var "VIDEO_ANALYSIS_RUNS_DIR"
|
||||
check_required_var "VIDEO_ANALYSIS_EXTERNAL_URL"
|
||||
|
||||
cd "$SCRIPT_DIR"
|
||||
|
||||
case $ACTION in
|
||||
up)
|
||||
echo "Starting Video Analysis with profile: $PROFILE"
|
||||
echo " vLLM base URL: $VIDEO_ANALYSIS_VLLM_BASE_URL"
|
||||
echo " vLLM model: $VIDEO_ANALYSIS_VLLM_MODEL"
|
||||
echo " runs dir: $VIDEO_ANALYSIS_RUNS_DIR"
|
||||
echo ""
|
||||
# shellcheck disable=SC2086
|
||||
exec docker compose --profile "$PROFILE" up $DETACH
|
||||
;;
|
||||
down)
|
||||
if [[ -z "$PROFILE" ]]; then
|
||||
echo "ERROR: --profile is required with --down"
|
||||
exit 1
|
||||
fi
|
||||
echo "Stopping Video Analysis containers..."
|
||||
exec docker compose --profile "$PROFILE" down
|
||||
;;
|
||||
logs)
|
||||
if [[ -z "$PROFILE" ]]; then
|
||||
echo "ERROR: --profile is required with --logs"
|
||||
exit 1
|
||||
fi
|
||||
exec docker compose --profile "$PROFILE" logs -f
|
||||
;;
|
||||
esac
|
||||
130
ai_platform/modules/video-analysis/deploy/docker-compose.yml
Normal file
130
ai_platform/modules/video-analysis/deploy/docker-compose.yml
Normal file
|
|
@ -0,0 +1,130 @@
|
|||
# Video Analysis Module - Docker Compose Configuration
|
||||
#
|
||||
# Port Allocation (Dev AI Vision/Video: 54500-54600):
|
||||
# 54500 - vLLM BusterX (deepfake detection model)
|
||||
# 54600 - Video Analysis API
|
||||
#
|
||||
# Profiles:
|
||||
# api - API server only (requires external vLLM)
|
||||
# api-vllm - API + vLLM BusterX (GPU required)
|
||||
#
|
||||
# Required environment variables (set in deploy/.env file):
|
||||
# HF_TOKEN - Hugging Face token for model downloads
|
||||
# HF_CACHE_DIR - Cache directory for models
|
||||
# VIDEO_ANALYSIS_RUNS_DIR - Directory for analysis artifacts
|
||||
#
|
||||
# Naming Convention: didiAI-{module}-{service}
|
||||
#
|
||||
# Network:
|
||||
# Uses deploy_default network (shared with other modules)
|
||||
|
||||
networks:
|
||||
deploy_default:
|
||||
external: true
|
||||
|
||||
services:
|
||||
# ==========================================================================
|
||||
# vLLM Server - BusterX (Deepfake Detection Model)
|
||||
# ==========================================================================
|
||||
# Vision-language model for video deepfake detection
|
||||
# Runs on GPU 1 (Qwen3.5-35B-A3B uses GPU 0 via llm-inference module)
|
||||
# BusterX is based on Qwen2.5-VL-7B (~17GB VRAM with optimizations)
|
||||
vllm-buster:
|
||||
container_name: didiAI-video-vllm-buster
|
||||
image: vllm/vllm-openai:v0.8.5
|
||||
ports:
|
||||
- "54500:54500"
|
||||
networks:
|
||||
- deploy_default
|
||||
volumes:
|
||||
- ${HF_CACHE_DIR:-/cai2_ds_storage/hf_cache}:/root/.cache/huggingface
|
||||
environment:
|
||||
- HF_HOME=/root/.cache/huggingface
|
||||
- HUGGING_FACE_HUB_TOKEN=${HF_TOKEN}
|
||||
- CUDA_VISIBLE_DEVICES=1
|
||||
command: >
|
||||
--model l8cv/BusterX_plusplus
|
||||
--host 0.0.0.0
|
||||
--port 54500
|
||||
--served-model-name busterx
|
||||
--tensor-parallel-size 1
|
||||
--max-model-len 32768
|
||||
--gpu-memory-utilization 0.25
|
||||
--trust-remote-code
|
||||
--enable-prefix-caching
|
||||
--disable-log-requests
|
||||
deploy:
|
||||
resources:
|
||||
reservations:
|
||||
devices:
|
||||
- driver: nvidia
|
||||
device_ids: ['1']
|
||||
capabilities: [gpu]
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "-f", "http://localhost:54500/health"]
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 10
|
||||
start_period: 600s
|
||||
restart: unless-stopped
|
||||
profiles:
|
||||
- api-vllm
|
||||
|
||||
# ==========================================================================
|
||||
# Video Analysis API Server
|
||||
# ==========================================================================
|
||||
video-analysis-api:
|
||||
container_name: didiAI-video-api
|
||||
image: didiai-video-api
|
||||
build:
|
||||
context: ..
|
||||
dockerfile: deploy/Dockerfile
|
||||
|
||||
ports:
|
||||
- "54600:54600"
|
||||
networks:
|
||||
- deploy_default
|
||||
|
||||
environment:
|
||||
# Server settings
|
||||
- VIDEO_ANALYSIS_HOST=0.0.0.0
|
||||
- VIDEO_ANALYSIS_PORT=54600
|
||||
# External URL for OpenAPI spec (REQUIRED)
|
||||
- VIDEO_ANALYSIS_EXTERNAL_URL=${VIDEO_ANALYSIS_EXTERNAL_URL}
|
||||
# vLLM connection (points to vllm-buster container)
|
||||
- VIDEO_ANALYSIS_VLLM_BASE_URL=${VIDEO_ANALYSIS_VLLM_BASE_URL:-http://didiAI-video-vllm-buster:54500}
|
||||
- VIDEO_ANALYSIS_VLLM_MODEL=${VIDEO_ANALYSIS_VLLM_MODEL:-busterx}
|
||||
- VIDEO_ANALYSIS_RUNS_DIR=${VIDEO_ANALYSIS_RUNS_DIR:-/app/runs}
|
||||
|
||||
# Optional tuning
|
||||
- VIDEO_ANALYSIS_FRAMES=${VIDEO_ANALYSIS_FRAMES:-16}
|
||||
- VIDEO_ANALYSIS_MAX_SIDE=${VIDEO_ANALYSIS_MAX_SIDE:-960}
|
||||
- VIDEO_ANALYSIS_JPEG_QUALITY=${VIDEO_ANALYSIS_JPEG_QUALITY:-85}
|
||||
- VIDEO_ANALYSIS_MAX_TOKENS=${VIDEO_ANALYSIS_MAX_TOKENS:-750}
|
||||
- VIDEO_ANALYSIS_TEMPERATURE=${VIDEO_ANALYSIS_TEMPERATURE:-0.000001}
|
||||
- VIDEO_ANALYSIS_REPETITION_PENALTY=${VIDEO_ANALYSIS_REPETITION_PENALTY:-1.05}
|
||||
|
||||
# Runtime config polling
|
||||
- VIDEO_ANALYSIS_DASHBOARD_URL=${VIDEO_ANALYSIS_DASHBOARD_URL:-http://didiAI-dashboard:51300}
|
||||
|
||||
# Semantic analysis settings
|
||||
- VIDEO_ANALYSIS_SEMANTIC_LLM_BASE_URL=${VIDEO_ANALYSIS_SEMANTIC_LLM_BASE_URL}
|
||||
- VIDEO_ANALYSIS_SEMANTIC_AGGREGATION_MODEL=${VIDEO_ANALYSIS_SEMANTIC_AGGREGATION_MODEL}
|
||||
- VIDEO_ANALYSIS_SEMANTIC_LLM_API_KEY=${VIDEO_ANALYSIS_SEMANTIC_LLM_API_KEY}
|
||||
|
||||
volumes:
|
||||
# Persist artifacts on the host (module-root runs/ folder)
|
||||
- ../runs:/app/runs
|
||||
|
||||
healthcheck:
|
||||
test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:54600/health')"]
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 3
|
||||
start_period: 20s
|
||||
|
||||
restart: unless-stopped
|
||||
profiles:
|
||||
- api
|
||||
- api-vllm
|
||||
|
||||
52
ai_platform/modules/video-analysis/deploy/nginx.conf
Normal file
52
ai_platform/modules/video-analysis/deploy/nginx.conf
Normal file
|
|
@ -0,0 +1,52 @@
|
|||
upstream video_analysis_api {
|
||||
# With host networking, the API binds host:8007
|
||||
server 127.0.0.1:8007;
|
||||
keepalive 32;
|
||||
}
|
||||
|
||||
server {
|
||||
listen 80;
|
||||
server_name _;
|
||||
|
||||
# Timeouts (video processing + vLLM may take time)
|
||||
proxy_connect_timeout 60s;
|
||||
proxy_send_timeout 300s;
|
||||
proxy_read_timeout 600s;
|
||||
|
||||
# Allow large video uploads
|
||||
client_max_body_size 500M;
|
||||
|
||||
# Disable buffering for large uploads / streaming
|
||||
proxy_buffering off;
|
||||
proxy_request_buffering off;
|
||||
|
||||
location /health {
|
||||
access_log off;
|
||||
proxy_pass http://video_analysis_api/health;
|
||||
proxy_http_version 1.1;
|
||||
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header Connection "";
|
||||
}
|
||||
|
||||
location / {
|
||||
proxy_pass http://video_analysis_api;
|
||||
proxy_http_version 1.1;
|
||||
|
||||
# Forward client info
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
|
||||
# Streaming friendliness (even if not currently used)
|
||||
proxy_set_header Connection "";
|
||||
proxy_set_header X-Accel-Buffering no;
|
||||
|
||||
proxy_cache off;
|
||||
chunked_transfer_encoding off;
|
||||
|
||||
# Request correlation
|
||||
proxy_set_header X-Request-ID $request_id;
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,41 @@
|
|||
upstream video_analysis_api {
|
||||
server didiAI-video-api:8011;
|
||||
keepalive 32;
|
||||
}
|
||||
|
||||
server {
|
||||
listen 80;
|
||||
server_name _;
|
||||
|
||||
proxy_connect_timeout ${NGINX_CONNECT_TIMEOUT};
|
||||
proxy_send_timeout ${NGINX_SEND_TIMEOUT};
|
||||
proxy_read_timeout ${NGINX_READ_TIMEOUT};
|
||||
|
||||
# Larger uploads (videos)
|
||||
client_max_body_size 500M;
|
||||
|
||||
location /health {
|
||||
access_log off;
|
||||
proxy_pass http://video_analysis_api/health;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
}
|
||||
|
||||
location / {
|
||||
proxy_pass http://video_analysis_api;
|
||||
proxy_http_version 1.1;
|
||||
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
|
||||
# SSE-friendly (safe even if you don't stream yet)
|
||||
proxy_set_header Connection '';
|
||||
proxy_set_header X-Accel-Buffering no;
|
||||
proxy_buffering off;
|
||||
proxy_cache off;
|
||||
chunked_transfer_encoding off;
|
||||
|
||||
proxy_set_header X-Request-ID $request_id;
|
||||
}
|
||||
}
|
||||
46
ai_platform/modules/video-analysis/pyproject.toml
Normal file
46
ai_platform/modules/video-analysis/pyproject.toml
Normal file
|
|
@ -0,0 +1,46 @@
|
|||
[project]
|
||||
name = "video-analysis"
|
||||
version = "0.1.0"
|
||||
description = "Video analysis service for deepfake detection (semantic analysis via vLLM backends)."
|
||||
requires-python = ">=3.11"
|
||||
readme = "README.md"
|
||||
|
||||
dependencies = [
|
||||
"fastapi>=0.115.0,<1.0",
|
||||
"uvicorn[standard]>=0.32.0,<1.0",
|
||||
"pydantic>=2.0,<3.0",
|
||||
"pydantic-settings>=2.0,<3.0",
|
||||
"python-multipart>=0.0.9,<1.0",
|
||||
"opencv-python-headless>=4.8",
|
||||
"pillow>=10.0",
|
||||
"requests>=2.31,<3.0",
|
||||
"httpx>=0.27,<1.0",
|
||||
"numpy>=1.24,<2.0",
|
||||
"pyyaml>=6.0,<7.0",
|
||||
"prometheus-fastapi-instrumentator>=7.0.0",
|
||||
"opentelemetry-instrumentation-fastapi>=0.50b0",
|
||||
"opentelemetry-exporter-otlp-proto-grpc>=1.30.0",
|
||||
]
|
||||
|
||||
[project.optional-dependencies]
|
||||
dev = [
|
||||
"pytest>=8.0,<9.0",
|
||||
"pytest-cov>=4.0,<6.0",
|
||||
"ruff>=0.8,<1.0",
|
||||
"httpx>=0.27,<1.0",
|
||||
]
|
||||
|
||||
[build-system]
|
||||
requires = ["hatchling"]
|
||||
build-backend = "hatchling.build"
|
||||
|
||||
[tool.hatch.build.targets.wheel]
|
||||
packages = ["src/video_analysis"]
|
||||
|
||||
[tool.ruff]
|
||||
extend = "../../ruff.toml"
|
||||
|
||||
[tool.pytest.ini_options]
|
||||
pythonpath = ["src"]
|
||||
testpaths = ["tests"]
|
||||
|
||||
772
ai_platform/modules/video-analysis/src/video_analysis/app.py
Normal file
772
ai_platform/modules/video-analysis/src/video_analysis/app.py
Normal file
|
|
@ -0,0 +1,772 @@
|
|||
import hashlib
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import time
|
||||
import uuid
|
||||
from collections.abc import AsyncGenerator
|
||||
from contextlib import asynccontextmanager
|
||||
from datetime import datetime, timezone
|
||||
|
||||
import httpx
|
||||
from fastapi import FastAPI, File, Form, HTTPException, UploadFile
|
||||
from fastapi.responses import JSONResponse
|
||||
|
||||
from .buster_client import (
|
||||
build_frame_evidence,
|
||||
call_vllm_chat,
|
||||
frame_to_data_url_b64jpeg,
|
||||
parse_verdict_and_explanation,
|
||||
)
|
||||
from .runtime_config import RuntimeConfigClient
|
||||
from .schemas import AnalyzeResponse, ChunkResult, SemanticAnalysisResponse, SemanticMeta, Usage
|
||||
from .settings import settings
|
||||
from .video_sampling import sample_frames_chunked, sample_frames_uniform
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Runtime config (reachable from handlers via app.state.runtime_config)
|
||||
runtime_config = RuntimeConfigClient(
|
||||
dashboard_url=settings.dashboard_url,
|
||||
live_log_logger_name="video_analysis",
|
||||
live_log_key="video.log.level",
|
||||
)
|
||||
|
||||
|
||||
@asynccontextmanager
|
||||
async def lifespan(app: FastAPI) -> AsyncGenerator[None, None]:
|
||||
"""App lifespan: start runtime config polling."""
|
||||
logger.info("Starting Video Analysis API")
|
||||
await runtime_config.start()
|
||||
app.state.runtime_config = runtime_config
|
||||
yield
|
||||
logger.info("Shutting down Video Analysis API")
|
||||
await runtime_config.stop()
|
||||
|
||||
|
||||
app = FastAPI(
|
||||
title="Video Analysis API",
|
||||
description="Deepfake detection and semantic video analysis",
|
||||
version="0.1.0",
|
||||
servers=[{"url": settings.external_url, "description": "Video Analysis API"}],
|
||||
lifespan=lifespan,
|
||||
)
|
||||
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- observability
|
||||
|
||||
|
||||
# Prometheus /metrics + OTel tracing (no-op if deps missing or OTEL endpoint unset)
|
||||
|
||||
|
||||
try:
|
||||
|
||||
|
||||
from prometheus_fastapi_instrumentator import Instrumentator as _Inst # type: ignore
|
||||
|
||||
|
||||
_Inst(should_group_status_codes=True).instrument(app).expose(app, endpoint="/metrics", include_in_schema=False)
|
||||
|
||||
|
||||
except ImportError:
|
||||
|
||||
|
||||
pass
|
||||
|
||||
|
||||
import os as _os # noqa: E402
|
||||
|
||||
|
||||
_otel_ep = _os.environ.get("OTEL_EXPORTER_OTLP_ENDPOINT")
|
||||
|
||||
|
||||
if _otel_ep:
|
||||
|
||||
|
||||
try:
|
||||
|
||||
|
||||
from opentelemetry import trace as _trace # type: ignore
|
||||
|
||||
|
||||
from opentelemetry.sdk.resources import Resource as _R # type: ignore
|
||||
|
||||
|
||||
from opentelemetry.sdk.trace import TracerProvider as _TP # type: ignore
|
||||
|
||||
|
||||
from opentelemetry.sdk.trace.export import BatchSpanProcessor as _BSP # type: ignore
|
||||
|
||||
|
||||
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter as _Exp # type: ignore
|
||||
|
||||
|
||||
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor as _FInst # type: ignore
|
||||
|
||||
|
||||
_provider = _TP(resource=_R.create({"service.name": _os.environ.get("OTEL_SERVICE_NAME", "didiAI-video-api")}))
|
||||
|
||||
|
||||
_provider.add_span_processor(_BSP(_Exp(endpoint=_otel_ep, insecure=True)))
|
||||
|
||||
|
||||
_trace.set_tracer_provider(_provider)
|
||||
|
||||
|
||||
_FInst.instrument_app(app)
|
||||
|
||||
|
||||
print(f"[otel] didiAI-video-api instrumented -> {_otel_ep}")
|
||||
|
||||
|
||||
except ImportError as _e:
|
||||
|
||||
|
||||
print(f"[otel] skip: {_e}")
|
||||
|
||||
|
||||
|
||||
def safe_mkdir(path: str) -> None:
|
||||
os.makedirs(path, exist_ok=True)
|
||||
|
||||
|
||||
def write_json(path: str, obj) -> None:
|
||||
with open(path, "w", encoding="utf-8") as f:
|
||||
json.dump(obj, f, ensure_ascii=False, indent=2)
|
||||
|
||||
|
||||
def sha256_file(path: str, chunk_size: int = 8 * 1024 * 1024) -> str:
|
||||
h = hashlib.sha256()
|
||||
with open(path, "rb") as f:
|
||||
while True:
|
||||
b = f.read(chunk_size)
|
||||
if not b:
|
||||
break
|
||||
h.update(b)
|
||||
return h.hexdigest()
|
||||
|
||||
|
||||
@app.get("/health")
|
||||
def health():
|
||||
return {"status": "ok"}
|
||||
|
||||
|
||||
@app.post("/analyze/video", response_model=AnalyzeResponse)
|
||||
def analyze_video(file: UploadFile = File(...)):
|
||||
# Defensive (Settings already requires it)
|
||||
if not settings.vllm_base_url:
|
||||
raise HTTPException(status_code=500, detail="VIDEO_ANALYSIS_VLLM_BASE_URL is not set")
|
||||
|
||||
request_id = str(uuid.uuid4())
|
||||
started_at = datetime.now(timezone.utc).isoformat()
|
||||
|
||||
# Make run_dir absolute under runs_dir (inside container: /app/runs/<uuid>)
|
||||
run_dir = os.path.join(settings.runs_dir, request_id)
|
||||
safe_mkdir(run_dir)
|
||||
|
||||
# Save upload to disk for reproducibility
|
||||
video_path = os.path.join(run_dir, file.filename or "upload.mp4")
|
||||
t0 = time.time()
|
||||
with open(video_path, "wb") as out:
|
||||
out.write(file.file.read())
|
||||
save_time_s = round(time.time() - t0, 4)
|
||||
|
||||
video_hash = sha256_file(video_path)
|
||||
prompt = settings.analysis_prompt
|
||||
|
||||
# Sampling
|
||||
frames, meta = sample_frames_uniform(video_path, num_frames=settings.frames)
|
||||
|
||||
# Encode frames
|
||||
t_enc0 = time.time()
|
||||
data_urls = [
|
||||
frame_to_data_url_b64jpeg(f, max_side=settings.max_side, jpeg_quality=settings.jpeg_quality)
|
||||
for f in frames
|
||||
]
|
||||
encode_time_s = round(time.time() - t_enc0, 4)
|
||||
|
||||
request_record = {
|
||||
"request_id": request_id,
|
||||
"started_at_utc": started_at,
|
||||
"video": {
|
||||
"path": video_path,
|
||||
"sha256": video_hash,
|
||||
"fps": meta.get("fps"),
|
||||
"total_frames": meta.get("total_frames"),
|
||||
"duration_s": meta.get("duration_s"),
|
||||
"width": meta.get("width"),
|
||||
"height": meta.get("height"),
|
||||
},
|
||||
"sampling_policy": {
|
||||
"frames_requested": meta.get("frames_requested"),
|
||||
"sampled": meta.get("sampled"),
|
||||
"indices": meta.get("indices"),
|
||||
"timestamps_s": meta.get("timestamps_s"),
|
||||
"sampling_method": meta.get("sampling_method"),
|
||||
"max_side": settings.max_side,
|
||||
"jpeg_quality": settings.jpeg_quality,
|
||||
},
|
||||
"runtime": {
|
||||
"vllm_base_url": settings.vllm_base_url,
|
||||
"model": settings.vllm_model,
|
||||
},
|
||||
"generation_params": {
|
||||
"max_tokens": settings.max_tokens,
|
||||
"temperature": settings.temperature,
|
||||
"repetition_penalty": settings.repetition_penalty,
|
||||
},
|
||||
"latency_s": {"upload_save_time_s": save_time_s},
|
||||
"prompt": prompt,
|
||||
}
|
||||
write_json(os.path.join(run_dir, "request.json"), request_record)
|
||||
|
||||
# Call vLLM
|
||||
raw_resp, infer_time_s = call_vllm_chat(
|
||||
base_url=settings.vllm_base_url,
|
||||
model=settings.vllm_model,
|
||||
data_urls=data_urls,
|
||||
prompt=prompt,
|
||||
max_tokens=settings.max_tokens,
|
||||
temperature=settings.temperature,
|
||||
repetition_penalty=settings.repetition_penalty,
|
||||
)
|
||||
write_json(os.path.join(run_dir, "raw_response.json"), raw_resp)
|
||||
|
||||
try:
|
||||
model_text = raw_resp["choices"][0]["message"]["content"]
|
||||
except Exception:
|
||||
model_text = ""
|
||||
|
||||
parsed = parse_verdict_and_explanation(model_text)
|
||||
usage = raw_resp.get("usage") or {}
|
||||
|
||||
indices = meta.get("indices") or []
|
||||
timestamps = meta.get("timestamps_s") or []
|
||||
evidence = build_frame_evidence(indices, timestamps)
|
||||
|
||||
result_record = {
|
||||
"request_id": request_id,
|
||||
"run_dir": run_dir,
|
||||
"verdict": parsed["verdict"],
|
||||
"explanation": parsed["explanation"],
|
||||
"frames_analyzed": int(meta.get("sampled") or len(indices)),
|
||||
"evidence": evidence,
|
||||
"usage": {
|
||||
"prompt_tokens": int(usage.get("prompt_tokens") or 0),
|
||||
"completion_tokens": int(usage.get("completion_tokens") or 0),
|
||||
"total_tokens": int(usage.get("total_tokens") or 0),
|
||||
},
|
||||
"latency_s": {
|
||||
"sampling_time_s": float(meta.get("sampling_time_s") or 0.0),
|
||||
"encode_time_s": encode_time_s,
|
||||
"model_inference_time_s": round(infer_time_s, 4),
|
||||
},
|
||||
"meta": {
|
||||
"fps": float(meta.get("fps") or 0.0),
|
||||
"total_frames": int(meta.get("total_frames") or 0),
|
||||
"duration_s": meta.get("duration_s"),
|
||||
"sampled": int(meta.get("sampled") or 0),
|
||||
"indices": meta.get("indices") or [],
|
||||
"timestamps_s": meta.get("timestamps_s") or [],
|
||||
},
|
||||
}
|
||||
write_json(os.path.join(run_dir, "result.json"), result_record)
|
||||
|
||||
return JSONResponse(content=result_record)
|
||||
|
||||
|
||||
@app.post("/analyze/video/semantic", response_model=SemanticAnalysisResponse)
|
||||
async def analyze_video_semantic(
|
||||
file: UploadFile = File(...),
|
||||
chunk_duration_s: float = Form(default=None),
|
||||
frames_per_chunk: int = Form(default=None),
|
||||
enable_aggregation: bool = Form(default=None),
|
||||
):
|
||||
"""
|
||||
Semantic video analysis with dense temporal sampling.
|
||||
|
||||
This endpoint performs deep content understanding by:
|
||||
1. Dividing video into temporal chunks
|
||||
2. Densely sampling each chunk (24 frames per 10s default)
|
||||
3. Analyzing each chunk with vision LLM
|
||||
4. Optionally aggregating into coherent narrative
|
||||
|
||||
Use this for:
|
||||
- Content understanding and description
|
||||
- Action recognition
|
||||
- Scene analysis
|
||||
- Narrative extraction
|
||||
|
||||
Args:
|
||||
file: Video file to analyze
|
||||
chunk_duration_s: Duration of each chunk in seconds (default from settings)
|
||||
frames_per_chunk: Frames to sample per chunk (default from settings)
|
||||
enable_aggregation: Whether to aggregate chunks into summary (default from settings)
|
||||
|
||||
Returns:
|
||||
SemanticAnalysisResponse with chunk descriptions and optional summary
|
||||
"""
|
||||
# Use settings defaults if not provided
|
||||
chunk_duration_s = chunk_duration_s or settings.semantic_chunk_duration_s
|
||||
frames_per_chunk = frames_per_chunk or settings.semantic_frames_per_chunk
|
||||
enable_aggregation = enable_aggregation if enable_aggregation is not None else settings.semantic_enable_aggregation
|
||||
|
||||
# Use dedicated semantic vLLM if configured, otherwise fall back to deepfake vLLM
|
||||
semantic_vllm_url = settings.semantic_vllm_base_url or settings.vllm_base_url
|
||||
semantic_vllm_model = settings.semantic_vllm_model or settings.vllm_model
|
||||
|
||||
if not semantic_vllm_url:
|
||||
raise HTTPException(status_code=500, detail="VIDEO_ANALYSIS_SEMANTIC_VLLM_BASE_URL or VIDEO_ANALYSIS_VLLM_BASE_URL is not set")
|
||||
|
||||
request_id = str(uuid.uuid4())
|
||||
started_at = datetime.now(timezone.utc).isoformat()
|
||||
overall_start = time.time()
|
||||
|
||||
# Create run directory
|
||||
run_dir = os.path.join(settings.runs_dir, request_id)
|
||||
safe_mkdir(run_dir)
|
||||
|
||||
# Save uploaded video
|
||||
video_path = os.path.join(run_dir, file.filename or "upload.mp4")
|
||||
with open(video_path, "wb") as out:
|
||||
out.write(file.file.read())
|
||||
|
||||
video_hash = sha256_file(video_path)
|
||||
|
||||
# Sample video in chunks
|
||||
try:
|
||||
chunks, meta = sample_frames_chunked(
|
||||
video_path,
|
||||
chunk_duration_s=chunk_duration_s,
|
||||
frames_per_chunk=frames_per_chunk,
|
||||
)
|
||||
except Exception as e:
|
||||
raise HTTPException(status_code=400, detail=f"Failed to sample video: {str(e)}")
|
||||
|
||||
# Save request metadata
|
||||
request_record = {
|
||||
"request_id": request_id,
|
||||
"started_at_utc": started_at,
|
||||
"analysis_type": "semantic",
|
||||
"video": {
|
||||
"path": video_path,
|
||||
"sha256": video_hash,
|
||||
"fps": meta.get("fps"),
|
||||
"total_frames": meta.get("total_frames"),
|
||||
"duration_s": meta.get("duration_s"),
|
||||
"width": meta.get("width"),
|
||||
"height": meta.get("height"),
|
||||
},
|
||||
"sampling_policy": {
|
||||
"chunk_duration_s": chunk_duration_s,
|
||||
"frames_per_chunk": frames_per_chunk,
|
||||
"num_chunks": meta["num_chunks"],
|
||||
"total_frames_sampled": meta["total_frames_sampled"],
|
||||
"chunks": meta["chunks"],
|
||||
},
|
||||
"runtime": {
|
||||
"vllm_base_url": semantic_vllm_url,
|
||||
"model": semantic_vllm_model,
|
||||
},
|
||||
}
|
||||
write_json(os.path.join(run_dir, "semantic_request.json"), request_record)
|
||||
|
||||
# Process each chunk with vision LLM
|
||||
chunk_results = []
|
||||
prompt = settings.semantic_prompt
|
||||
|
||||
for chunk_idx, chunk_frames in enumerate(chunks):
|
||||
chunk_meta = meta["chunks"][chunk_idx]
|
||||
|
||||
# Encode frames for this chunk
|
||||
t_enc = time.time()
|
||||
data_urls = [
|
||||
frame_to_data_url_b64jpeg(f, max_side=settings.max_side, jpeg_quality=settings.jpeg_quality)
|
||||
for f in chunk_frames
|
||||
]
|
||||
encode_time = time.time() - t_enc
|
||||
|
||||
# Call vLLM for this chunk (uses semantic vLLM - Qwen3-VL)
|
||||
try:
|
||||
raw_resp, infer_time = call_vllm_chat(
|
||||
base_url=semantic_vllm_url,
|
||||
model=semantic_vllm_model,
|
||||
data_urls=data_urls,
|
||||
prompt=prompt,
|
||||
max_tokens=settings.max_tokens,
|
||||
temperature=settings.temperature,
|
||||
repetition_penalty=settings.repetition_penalty,
|
||||
)
|
||||
except Exception as e:
|
||||
raise HTTPException(status_code=500, detail=f"VLM inference failed for chunk {chunk_idx}: {str(e)}")
|
||||
|
||||
# Extract description
|
||||
try:
|
||||
chunk_description = raw_resp["choices"][0]["message"]["content"]
|
||||
except Exception:
|
||||
chunk_description = ""
|
||||
|
||||
usage_data = raw_resp.get("usage", {})
|
||||
|
||||
chunk_result = ChunkResult(
|
||||
chunk_idx=chunk_idx,
|
||||
time_range=f"{chunk_meta['start_time_s']}s - {chunk_meta['end_time_s']}s",
|
||||
description=chunk_description,
|
||||
frames_analyzed=len(chunk_frames),
|
||||
inference_time_s=round(infer_time, 4),
|
||||
usage=Usage(
|
||||
prompt_tokens=int(usage_data.get("prompt_tokens", 0)),
|
||||
completion_tokens=int(usage_data.get("completion_tokens", 0)),
|
||||
total_tokens=int(usage_data.get("total_tokens", 0)),
|
||||
),
|
||||
)
|
||||
chunk_results.append(chunk_result)
|
||||
|
||||
# Save intermediate chunk result
|
||||
write_json(
|
||||
os.path.join(run_dir, f"chunk_{chunk_idx:03d}_response.json"),
|
||||
{
|
||||
"chunk_idx": chunk_idx,
|
||||
"raw_response": raw_resp,
|
||||
"description": chunk_description,
|
||||
"encode_time_s": round(encode_time, 4),
|
||||
"inference_time_s": round(infer_time, 4),
|
||||
},
|
||||
)
|
||||
|
||||
# Aggregate chunks into final summary (if enabled)
|
||||
final_summary = None
|
||||
aggregation_time = None
|
||||
|
||||
if enable_aggregation and len(chunk_results) > 1:
|
||||
t_agg = time.time()
|
||||
|
||||
# Build aggregation prompt
|
||||
chunks_text = "\n\n".join(
|
||||
[f"Segment {r.chunk_idx + 1} ({r.time_range}):\n{r.description}" for r in chunk_results]
|
||||
)
|
||||
|
||||
aggregation_prompt = settings.aggregation_prompt_template.format(
|
||||
num_chunks=len(chunk_results),
|
||||
duration=meta.get("duration_s", 0),
|
||||
chunks_text=chunks_text,
|
||||
)
|
||||
|
||||
# Call text LLM for aggregation
|
||||
try:
|
||||
headers = {"Content-Type": "application/json"}
|
||||
if settings.semantic_llm_api_key:
|
||||
headers["Authorization"] = f"Bearer {settings.semantic_llm_api_key}"
|
||||
|
||||
async with httpx.AsyncClient(timeout=120.0) as client:
|
||||
llm_resp = await client.post(
|
||||
f"{settings.semantic_llm_base_url}/v1/chat/completions",
|
||||
json={
|
||||
"model": settings.semantic_aggregation_model,
|
||||
"messages": [{"role": "user", "content": aggregation_prompt}],
|
||||
"max_tokens": 1500,
|
||||
"temperature": 0.3,
|
||||
},
|
||||
headers=headers,
|
||||
)
|
||||
llm_resp.raise_for_status()
|
||||
llm_data = llm_resp.json()
|
||||
final_summary = llm_data["choices"][0]["message"]["content"]
|
||||
except Exception as e:
|
||||
# Non-fatal: log warning but continue
|
||||
final_summary = f"[Aggregation failed: {str(e)}]"
|
||||
|
||||
aggregation_time = round(time.time() - t_agg, 4)
|
||||
|
||||
# Save aggregation result
|
||||
write_json(
|
||||
os.path.join(run_dir, "aggregation.json"),
|
||||
{
|
||||
"prompt": aggregation_prompt,
|
||||
"summary": final_summary,
|
||||
"aggregation_time_s": aggregation_time,
|
||||
},
|
||||
)
|
||||
|
||||
total_latency = round(time.time() - overall_start, 4)
|
||||
|
||||
# Build final response
|
||||
result = SemanticAnalysisResponse(
|
||||
request_id=uuid.UUID(request_id),
|
||||
run_dir=run_dir,
|
||||
analysis_type="semantic",
|
||||
video_duration_s=meta.get("duration_s"),
|
||||
num_chunks=len(chunk_results),
|
||||
chunk_results=chunk_results,
|
||||
final_summary=final_summary,
|
||||
aggregation_time_s=aggregation_time,
|
||||
total_latency_s=total_latency,
|
||||
meta=SemanticMeta(
|
||||
fps=float(meta.get("fps", 0)),
|
||||
total_frames=int(meta.get("total_frames", 0)),
|
||||
duration_s=meta.get("duration_s"),
|
||||
chunk_duration_s=chunk_duration_s,
|
||||
frames_per_chunk=frames_per_chunk,
|
||||
total_frames_sampled=meta["total_frames_sampled"],
|
||||
),
|
||||
)
|
||||
|
||||
# Save final result
|
||||
write_json(os.path.join(run_dir, "semantic_result.json"), result.model_dump(mode="json"))
|
||||
|
||||
return result
|
||||
|
||||
|
||||
@app.get("/v1/info")
|
||||
def get_component_info():
|
||||
"""
|
||||
Get component information for service catalog.
|
||||
|
||||
Returns complete metadata about this service including:
|
||||
- Resource information (component metadata)
|
||||
- Available models (BusterX for deepfake detection)
|
||||
- Available functions (API endpoints)
|
||||
|
||||
This endpoint is used by the catalog-api to aggregate service information
|
||||
and by backend systems to populate the catalog database.
|
||||
|
||||
Returns:
|
||||
dict: Component information matching catalog.resources, catalog.models,
|
||||
and catalog.functions schemas.
|
||||
"""
|
||||
# Build resource information (maps to catalog.resources)
|
||||
resource = {
|
||||
"name": "Video Analysis Service",
|
||||
"slug": "video-analysis",
|
||||
"resource_type": "api_service",
|
||||
"provider": "internal",
|
||||
"base_url": f"http://didiAI-video-api:54600",
|
||||
"configuration": {
|
||||
"version": "0.1.0",
|
||||
"port": 54600,
|
||||
"external_url": settings.external_url,
|
||||
"vllm_base_url": settings.vllm_base_url,
|
||||
"vllm_model": settings.vllm_model,
|
||||
"frames": settings.frames,
|
||||
"max_side": settings.max_side,
|
||||
},
|
||||
"authentication": {
|
||||
"type": "none",
|
||||
"required": False,
|
||||
},
|
||||
"headers": {
|
||||
"Content-Type": "multipart/form-data",
|
||||
"Accept": "application/json",
|
||||
},
|
||||
"rate_limits": {
|
||||
"enabled": False,
|
||||
},
|
||||
"cost_tracking": {
|
||||
"enabled": False,
|
||||
},
|
||||
"tags": ["video", "deepfake", "semantic-analysis", "vision", "computer-vision"],
|
||||
"is_active": True,
|
||||
"metadata": {
|
||||
"category": "computer-vision",
|
||||
"gpu_required": True,
|
||||
"status": "healthy",
|
||||
"runs_dir": settings.runs_dir,
|
||||
},
|
||||
}
|
||||
|
||||
# Define models (maps to catalog.models)
|
||||
models = [
|
||||
{
|
||||
"name": "BusterX Deepfake Detector",
|
||||
"slug": "busterx",
|
||||
"provider": "l8cv",
|
||||
"model_type": "vision",
|
||||
"capabilities": [
|
||||
"deepfake-detection",
|
||||
"video-analysis",
|
||||
"multimodal",
|
||||
],
|
||||
"configuration": {
|
||||
"backend": "vllm",
|
||||
"base_model": "Qwen2.5-VL-7B",
|
||||
"max_tokens": settings.max_tokens,
|
||||
"temperature": settings.temperature,
|
||||
},
|
||||
"endpoint": settings.vllm_base_url,
|
||||
"api_key_ref": None,
|
||||
"tags": ["deepfake", "vision", "busterx", "video", "7b"],
|
||||
"is_active": True,
|
||||
"metadata": {
|
||||
"model_id": settings.vllm_model,
|
||||
"vllm_base_url": settings.vllm_base_url,
|
||||
"gpu_id": 1,
|
||||
"vram_gb": 22,
|
||||
"specialized": "deepfake-detection",
|
||||
},
|
||||
}
|
||||
]
|
||||
|
||||
# Define available functions (maps to catalog.functions)
|
||||
functions = [
|
||||
{
|
||||
"name": "Video Deepfake Analysis",
|
||||
"slug": "video-deepfake-analysis",
|
||||
"category": "deepfake-detection",
|
||||
"description": (
|
||||
"Analyze video for deepfake manipulation using BusterX model. "
|
||||
"Samples 16 frames uniformly and returns verdict: REAL/FAKE/UNCERTAIN."
|
||||
),
|
||||
"input_schema": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"file": {
|
||||
"type": "file",
|
||||
"description": "Video file to analyze (mp4, avi, mov, etc.)",
|
||||
},
|
||||
},
|
||||
"required": ["file"],
|
||||
},
|
||||
"output_schema": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"request_id": {"type": "string", "format": "uuid"},
|
||||
"run_dir": {"type": "string"},
|
||||
"verdict": {
|
||||
"type": "string",
|
||||
"enum": ["REAL", "FAKE", "UNCERTAIN"],
|
||||
},
|
||||
"explanation": {"type": "string"},
|
||||
"frames_analyzed": {"type": "integer"},
|
||||
"evidence": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"frame_index": {"type": "integer"},
|
||||
"timestamp_s": {"type": "number"},
|
||||
},
|
||||
},
|
||||
},
|
||||
"usage": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"prompt_tokens": {"type": "integer"},
|
||||
"completion_tokens": {"type": "integer"},
|
||||
"total_tokens": {"type": "integer"},
|
||||
},
|
||||
},
|
||||
"latency_s": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"sampling_time_s": {"type": "number"},
|
||||
"encode_time_s": {"type": "number"},
|
||||
"model_inference_time_s": {"type": "number"},
|
||||
},
|
||||
},
|
||||
"meta": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"fps": {"type": "number"},
|
||||
"total_frames": {"type": "integer"},
|
||||
"duration_s": {"type": "number"},
|
||||
"sampled": {"type": "integer"},
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
"implementation": {
|
||||
"method": "POST",
|
||||
"path": "/analyze/video",
|
||||
"content_type": "multipart/form-data",
|
||||
"timeout": 120,
|
||||
},
|
||||
"endpoint": "http://localhost:8007/analyze/video",
|
||||
"tags": ["video", "deepfake", "detection", "fast"],
|
||||
"is_active": True,
|
||||
"metadata": {
|
||||
"average_latency_s": 13,
|
||||
"frames_analyzed": 16,
|
||||
"coverage": "~0.89% of frames",
|
||||
},
|
||||
},
|
||||
{
|
||||
"name": "Video Semantic Analysis",
|
||||
"slug": "video-semantic-analysis",
|
||||
"category": "semantic-analysis",
|
||||
"description": (
|
||||
"Deep semantic analysis of video content with temporal chunking. "
|
||||
"Divides video into chunks, analyzes each with vision LLM, and synthesizes narrative."
|
||||
),
|
||||
"input_schema": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"file": {
|
||||
"type": "file",
|
||||
"description": "Video file to analyze",
|
||||
},
|
||||
"chunk_duration_s": {
|
||||
"type": "number",
|
||||
"minimum": 1.0,
|
||||
"maximum": 60.0,
|
||||
"default": 10.0,
|
||||
"description": "Duration of each chunk in seconds",
|
||||
},
|
||||
"frames_per_chunk": {
|
||||
"type": "integer",
|
||||
"minimum": 4,
|
||||
"maximum": 64,
|
||||
"default": 24,
|
||||
"description": "Frames to sample per chunk",
|
||||
},
|
||||
"enable_aggregation": {
|
||||
"type": "boolean",
|
||||
"default": True,
|
||||
"description": "Aggregate chunks into final summary",
|
||||
},
|
||||
},
|
||||
"required": ["file"],
|
||||
},
|
||||
"output_schema": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"request_id": {"type": "string"},
|
||||
"num_chunks": {"type": "integer"},
|
||||
"chunk_results": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"chunk_idx": {"type": "integer"},
|
||||
"time_range": {"type": "string"},
|
||||
"description": {"type": "string"},
|
||||
"frames_analyzed": {"type": "integer"},
|
||||
},
|
||||
},
|
||||
},
|
||||
"final_summary": {"type": "string"},
|
||||
"total_latency_s": {"type": "number"},
|
||||
},
|
||||
},
|
||||
"implementation": {
|
||||
"method": "POST",
|
||||
"path": "/analyze/video/semantic",
|
||||
"content_type": "multipart/form-data",
|
||||
"timeout": 300,
|
||||
},
|
||||
"endpoint": "http://localhost:8007/analyze/video/semantic",
|
||||
"tags": ["video", "semantic", "analysis", "detailed"],
|
||||
"is_active": True,
|
||||
"metadata": {
|
||||
"average_latency_s": 77,
|
||||
"coverage": "~8% of frames",
|
||||
"use_case": "content-description",
|
||||
},
|
||||
},
|
||||
]
|
||||
|
||||
return {
|
||||
"resource": resource,
|
||||
"models": models,
|
||||
"functions": functions,
|
||||
}
|
||||
|
|
@ -0,0 +1,91 @@
|
|||
import base64
|
||||
import time
|
||||
from io import BytesIO
|
||||
from typing import Any, Dict, List, Tuple
|
||||
|
||||
import cv2
|
||||
import requests
|
||||
from PIL import Image
|
||||
|
||||
|
||||
def frame_to_data_url_b64jpeg(frame_bgr, max_side: int, jpeg_quality: int) -> str:
|
||||
frame_rgb = cv2.cvtColor(frame_bgr, cv2.COLOR_BGR2RGB)
|
||||
im = Image.fromarray(frame_rgb)
|
||||
|
||||
w, h = im.size
|
||||
scale = min(1.0, float(max_side) / float(max(w, h)))
|
||||
if scale < 1.0:
|
||||
im = im.resize((int(w * scale), int(h * scale)), Image.BICUBIC)
|
||||
|
||||
buf = BytesIO()
|
||||
im.save(buf, format="JPEG", quality=jpeg_quality, optimize=True)
|
||||
b64 = base64.b64encode(buf.getvalue()).decode("utf-8")
|
||||
return f"data:image/jpeg;base64,{b64}"
|
||||
|
||||
|
||||
def call_vllm_chat(
|
||||
base_url: str,
|
||||
model: str,
|
||||
data_urls: List[str],
|
||||
prompt: str,
|
||||
max_tokens: int,
|
||||
temperature: float,
|
||||
repetition_penalty: float,
|
||||
timeout_s: int = 180,
|
||||
) -> Tuple[Dict[str, Any], float]:
|
||||
url = base_url.rstrip("/") + "/v1/chat/completions"
|
||||
|
||||
content = [{"type": "image_url", "image_url": {"url": u}} for u in data_urls]
|
||||
content.append({"type": "text", "text": prompt})
|
||||
|
||||
payload = {
|
||||
"model": model,
|
||||
"messages": [{"role": "user", "content": content}],
|
||||
"max_tokens": max_tokens,
|
||||
"temperature": temperature,
|
||||
"repetition_penalty": repetition_penalty,
|
||||
}
|
||||
|
||||
t0 = time.time()
|
||||
r = requests.post(url, json=payload, timeout=timeout_s)
|
||||
dt = time.time() - t0
|
||||
r.raise_for_status()
|
||||
return r.json(), dt
|
||||
|
||||
|
||||
def parse_verdict_and_explanation(model_text: str) -> Dict[str, Any]:
|
||||
text = (model_text or "").strip()
|
||||
text_start = text[:20].upper()
|
||||
|
||||
if text_start.startswith("REAL"):
|
||||
verdict = "REAL"
|
||||
elif text_start.startswith("FAKE"):
|
||||
verdict = "FAKE"
|
||||
else:
|
||||
verdict = "UNCERTAIN"
|
||||
|
||||
return {"verdict": verdict, "explanation": text}
|
||||
|
||||
|
||||
def build_frame_evidence(
|
||||
indices: List[int], timestamps: List[float | None]
|
||||
) -> List[Dict[str, Any]]:
|
||||
"""Map analyzed frame indices to temporal evidence entries.
|
||||
|
||||
The model yields one holistic verdict over all sampled frames, so the honest
|
||||
per-frame evidence is the set of frames analyzed and their timestamps. Numeric
|
||||
confidence and cross-signal fusion are the backend aggregator's job.
|
||||
|
||||
Args:
|
||||
indices: Source frame indices that were sampled and sent to the model.
|
||||
timestamps: Per-frame timestamps in seconds (aligned with ``indices``).
|
||||
|
||||
Returns:
|
||||
List of ``{"frame_index", "timestamp_s"}`` dicts.
|
||||
"""
|
||||
|
||||
evidence: List[Dict[str, Any]] = []
|
||||
for pos, idx in enumerate(indices):
|
||||
ts = timestamps[pos] if pos < len(timestamps) else None
|
||||
evidence.append({"frame_index": int(idx), "timestamp_s": ts})
|
||||
return evidence
|
||||
|
|
@ -0,0 +1,157 @@
|
|||
"""Runtime config client — fetches config overrides from dashboard.
|
||||
|
||||
Polls the dashboard /api/config endpoint periodically and caches values
|
||||
in memory. All accessors fall back to caller-supplied defaults if the
|
||||
dashboard is unreachable or the key is missing.
|
||||
|
||||
Single source of truth = dashboard `KNOWN_KEYS`. This client carries no
|
||||
local default registry; consumers pass their own fallback (typically the
|
||||
pydantic settings value).
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import contextlib
|
||||
import logging
|
||||
from typing import Any
|
||||
|
||||
import httpx
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class RuntimeConfigClient:
|
||||
"""Polls dashboard /api/config and caches values in-process."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
dashboard_url: str | None,
|
||||
poll_interval_seconds: int = 30,
|
||||
request_timeout: float = 5.0,
|
||||
live_log_logger_name: str | None = None,
|
||||
live_log_key: str | None = None,
|
||||
) -> None:
|
||||
self.dashboard_url = dashboard_url.rstrip("/") if dashboard_url else None
|
||||
self.poll_interval = poll_interval_seconds
|
||||
self._timeout = request_timeout
|
||||
self._cache: dict[str, Any] = {}
|
||||
self._client: httpx.AsyncClient | None = None
|
||||
self._task: asyncio.Task[None] | None = None
|
||||
self._enabled = bool(dashboard_url)
|
||||
# Optional auto-apply: re-set log level on the named logger when the
|
||||
# configured key changes value (e.g., `llm.log.level`).
|
||||
self._live_log_logger_name = live_log_logger_name
|
||||
self._live_log_key = live_log_key
|
||||
self._last_log_level: str | None = None
|
||||
|
||||
@property
|
||||
def enabled(self) -> bool:
|
||||
return self._enabled
|
||||
|
||||
def get(self, key: str, default: Any = None) -> Any:
|
||||
v = self._cache.get(key)
|
||||
return v if v is not None else default
|
||||
|
||||
def get_bool(self, key: str, default: bool = False) -> bool:
|
||||
v = self._cache.get(key)
|
||||
return bool(v) if v is not None else default
|
||||
|
||||
def get_int(self, key: str, default: int = 0) -> int:
|
||||
v = self._cache.get(key)
|
||||
if v is None:
|
||||
return default
|
||||
try:
|
||||
return int(v)
|
||||
except (TypeError, ValueError):
|
||||
return default
|
||||
|
||||
def get_float(self, key: str, default: float = 0.0) -> float:
|
||||
v = self._cache.get(key)
|
||||
if v is None:
|
||||
return default
|
||||
try:
|
||||
return float(v)
|
||||
except (TypeError, ValueError):
|
||||
return default
|
||||
|
||||
def get_str(self, key: str, default: str = "") -> str:
|
||||
v = self._cache.get(key)
|
||||
return str(v) if v is not None else default
|
||||
|
||||
async def start(self) -> None:
|
||||
if not self._enabled:
|
||||
logger.info("RuntimeConfigClient disabled (no dashboard URL)")
|
||||
return
|
||||
self._client = httpx.AsyncClient(
|
||||
timeout=httpx.Timeout(
|
||||
connect=2.0, read=self._timeout, write=2.0, pool=5.0
|
||||
)
|
||||
)
|
||||
# Fetch once synchronously so first requests already have overrides
|
||||
await self._refresh()
|
||||
self._task = asyncio.create_task(self._loop())
|
||||
logger.info(
|
||||
"RuntimeConfigClient started (polling %s every %ds, %d keys cached)",
|
||||
self.dashboard_url,
|
||||
self.poll_interval,
|
||||
len(self._cache),
|
||||
)
|
||||
|
||||
async def stop(self) -> None:
|
||||
if self._task is not None:
|
||||
self._task.cancel()
|
||||
with contextlib.suppress(asyncio.CancelledError, Exception):
|
||||
await self._task
|
||||
self._task = None
|
||||
if self._client is not None and not self._client.is_closed:
|
||||
await self._client.aclose()
|
||||
self._client = None
|
||||
|
||||
async def _loop(self) -> None:
|
||||
while True:
|
||||
try:
|
||||
await asyncio.sleep(self.poll_interval)
|
||||
await self._refresh()
|
||||
except asyncio.CancelledError:
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.debug("Config poll failed: %s", e)
|
||||
|
||||
async def _refresh(self) -> None:
|
||||
if self._client is None or self.dashboard_url is None:
|
||||
return
|
||||
try:
|
||||
resp = await self._client.get(f"{self.dashboard_url}/api/config")
|
||||
resp.raise_for_status()
|
||||
data = resp.json()
|
||||
except Exception as e:
|
||||
logger.debug("Config refresh failed: %s", e)
|
||||
return
|
||||
|
||||
items = data.get("items", {})
|
||||
new_cache: dict[str, Any] = {}
|
||||
for key, entry in items.items():
|
||||
new_cache[key] = entry.get("value")
|
||||
self._cache = new_cache
|
||||
self._maybe_apply_log_level()
|
||||
|
||||
def _maybe_apply_log_level(self) -> None:
|
||||
"""Re-apply log level live if configured key changed."""
|
||||
if not self._live_log_key or not self._live_log_logger_name:
|
||||
return
|
||||
new_level = self.get_str(self._live_log_key)
|
||||
if not new_level:
|
||||
return
|
||||
if new_level == self._last_log_level:
|
||||
return
|
||||
try:
|
||||
level_int = logging.getLevelName(new_level.upper())
|
||||
if isinstance(level_int, int):
|
||||
logging.getLogger(self._live_log_logger_name).setLevel(level_int)
|
||||
self._last_log_level = new_level
|
||||
logger.info(
|
||||
"Log level for %s changed to %s (via runtime config)",
|
||||
self._live_log_logger_name,
|
||||
new_level,
|
||||
)
|
||||
except Exception as e:
|
||||
logger.warning("Failed to apply log level %s: %s", new_level, e)
|
||||
120
ai_platform/modules/video-analysis/src/video_analysis/schemas.py
Normal file
120
ai_platform/modules/video-analysis/src/video_analysis/schemas.py
Normal file
|
|
@ -0,0 +1,120 @@
|
|||
from __future__ import annotations
|
||||
|
||||
from typing import Literal
|
||||
from uuid import UUID
|
||||
|
||||
from pydantic import BaseModel, ConfigDict, Field
|
||||
|
||||
|
||||
Verdict = Literal["REAL", "FAKE", "UNCERTAIN"]
|
||||
|
||||
|
||||
class Usage(BaseModel):
|
||||
"""Token usage statistics."""
|
||||
|
||||
model_config = ConfigDict(extra="forbid")
|
||||
|
||||
prompt_tokens: int = Field(..., ge=0, description="Tokens in prompt")
|
||||
completion_tokens: int = Field(..., ge=0, description="Tokens in completion")
|
||||
total_tokens: int = Field(..., ge=0, description="Total tokens")
|
||||
|
||||
|
||||
class LatencyS(BaseModel):
|
||||
"""Latency breakdown in seconds."""
|
||||
|
||||
model_config = ConfigDict(extra="forbid")
|
||||
|
||||
sampling_time_s: float = Field(..., ge=0, description="Frame sampling time")
|
||||
encode_time_s: float = Field(..., ge=0, description="Encoding time")
|
||||
model_inference_time_s: float = Field(..., ge=0, description="Model inference time")
|
||||
|
||||
|
||||
class Meta(BaseModel):
|
||||
"""Video metadata and sampling info."""
|
||||
|
||||
model_config = ConfigDict(extra="forbid")
|
||||
|
||||
fps: float = Field(..., ge=0, description="Video frames per second (0 if unknown)")
|
||||
total_frames: int = Field(..., ge=0, description="Total frames in video")
|
||||
duration_s: float | None = Field(None, ge=0, description="Video duration in seconds (None if unknown)")
|
||||
sampled: int = Field(..., ge=1, description="Number of frames sampled")
|
||||
indices: list[int] = Field(..., description="Frame indices sampled")
|
||||
timestamps_s: list[float | None] = Field(..., description="Timestamps for sampled frames (None if unknown)")
|
||||
|
||||
|
||||
class FrameEvidence(BaseModel):
|
||||
"""One analyzed frame that informed the verdict (temporal evidence).
|
||||
|
||||
The model returns a single holistic verdict over all sampled frames, so the
|
||||
honest per-frame evidence is which frames were analyzed and at what time.
|
||||
Numeric per-signal confidence is produced by the backend aggregation layer
|
||||
(fusing BusterX + forensic + extractor signals), not by this service.
|
||||
"""
|
||||
|
||||
model_config = ConfigDict(extra="forbid")
|
||||
|
||||
frame_index: int = Field(..., ge=0, description="Source frame index")
|
||||
timestamp_s: float | None = Field(
|
||||
None, ge=0, description="Frame timestamp in seconds (None if unknown)"
|
||||
)
|
||||
|
||||
|
||||
class AnalyzeResponse(BaseModel):
|
||||
"""Response schema for video analysis."""
|
||||
|
||||
model_config = ConfigDict(extra="forbid")
|
||||
|
||||
request_id: UUID = Field(..., description="Request/run ID (UUID)")
|
||||
run_dir: str = Field(..., min_length=1, description="Run directory (e.g. /app/runs/<uuid>)")
|
||||
verdict: Verdict = Field(..., description="Classification verdict (REAL/FAKE/UNCERTAIN)")
|
||||
explanation: str = Field(..., min_length=1, description="Model explanation for verdict")
|
||||
frames_analyzed: int = Field(..., ge=1, description="Number of frames the verdict is based on")
|
||||
evidence: list[FrameEvidence] = Field(
|
||||
default_factory=list, description="Analyzed frames (temporal evidence)"
|
||||
)
|
||||
usage: Usage = Field(..., description="Token usage statistics")
|
||||
latency_s: LatencyS = Field(..., description="Latency breakdown")
|
||||
meta: Meta = Field(..., description="Video metadata")
|
||||
|
||||
|
||||
class ChunkResult(BaseModel):
|
||||
"""Result for a single video chunk."""
|
||||
|
||||
model_config = ConfigDict(extra="forbid")
|
||||
|
||||
chunk_idx: int = Field(..., ge=0, description="Chunk index (0-based)")
|
||||
time_range: str = Field(..., description="Time range (e.g. '0.0s - 10.0s')")
|
||||
description: str = Field(..., min_length=1, description="Semantic description of chunk")
|
||||
frames_analyzed: int = Field(..., ge=1, description="Number of frames analyzed")
|
||||
inference_time_s: float = Field(..., ge=0, description="Inference time for this chunk")
|
||||
usage: Usage = Field(..., description="Token usage for this chunk")
|
||||
|
||||
|
||||
class SemanticMeta(BaseModel):
|
||||
"""Metadata for semantic analysis."""
|
||||
|
||||
model_config = ConfigDict(extra="forbid")
|
||||
|
||||
fps: float = Field(..., ge=0, description="Video frames per second")
|
||||
total_frames: int = Field(..., ge=0, description="Total frames in video")
|
||||
duration_s: float | None = Field(None, ge=0, description="Video duration in seconds")
|
||||
chunk_duration_s: float = Field(..., ge=0, description="Duration of each chunk")
|
||||
frames_per_chunk: int = Field(..., ge=1, description="Frames sampled per chunk")
|
||||
total_frames_sampled: int = Field(..., ge=1, description="Total frames analyzed across all chunks")
|
||||
|
||||
|
||||
class SemanticAnalysisResponse(BaseModel):
|
||||
"""Response schema for semantic video analysis."""
|
||||
|
||||
model_config = ConfigDict(extra="forbid")
|
||||
|
||||
request_id: UUID = Field(..., description="Request/run ID (UUID)")
|
||||
run_dir: str = Field(..., min_length=1, description="Run directory (e.g. /app/runs/<uuid>)")
|
||||
analysis_type: Literal["semantic"] = Field(..., description="Analysis type identifier")
|
||||
video_duration_s: float | None = Field(None, ge=0, description="Total video duration in seconds")
|
||||
num_chunks: int = Field(..., ge=1, description="Number of chunks processed")
|
||||
chunk_results: list[ChunkResult] = Field(..., min_items=1, description="Results for each chunk")
|
||||
final_summary: str | None = Field(None, description="Aggregated summary (if enabled)")
|
||||
aggregation_time_s: float | None = Field(None, ge=0, description="Time spent on aggregation")
|
||||
total_latency_s: float = Field(..., ge=0, description="Total processing time")
|
||||
meta: SemanticMeta = Field(..., description="Video and sampling metadata")
|
||||
|
|
@ -0,0 +1,100 @@
|
|||
from pathlib import Path
|
||||
|
||||
import yaml
|
||||
from pydantic import Field
|
||||
from pydantic_settings import BaseSettings, SettingsConfigDict
|
||||
|
||||
|
||||
def load_yaml_config() -> dict:
|
||||
"""Load tuning config from yaml file if it exists."""
|
||||
yaml_paths = [
|
||||
Path(__file__).parent.parent.parent / "deploy" / "config.yaml",
|
||||
Path(__file__).parent.parent.parent / "config.yaml",
|
||||
]
|
||||
|
||||
for path in yaml_paths:
|
||||
if path.exists():
|
||||
return yaml.safe_load(path.read_text()) or {}
|
||||
|
||||
return {}
|
||||
|
||||
|
||||
class Settings(BaseSettings):
|
||||
model_config = SettingsConfigDict(
|
||||
env_prefix="VIDEO_ANALYSIS_",
|
||||
extra="ignore",
|
||||
)
|
||||
|
||||
# REQUIRED (no defaults): wiring to vLLM for deepfake detection (BusterX)
|
||||
vllm_base_url: str
|
||||
vllm_model: str
|
||||
|
||||
# REQUIRED (no defaults): evidence packs
|
||||
runs_dir: str
|
||||
|
||||
# External URL for OpenAPI spec (e.g., http://10.11.10.42:54600)
|
||||
external_url: str = Field(
|
||||
description="External URL for OpenAPI spec. REQUIRED.",
|
||||
)
|
||||
|
||||
# Semantic vision vLLM (Qwen3.5) - separate from deepfake
|
||||
semantic_vllm_base_url: str = Field(default="")
|
||||
semantic_vllm_model: str = Field(default="")
|
||||
|
||||
# OPTIONAL (tuning - loaded from yaml, env vars override)
|
||||
frames: int = Field(default=16, ge=1, le=64)
|
||||
max_side: int = Field(default=960, ge=100, le=2048)
|
||||
jpeg_quality: int = Field(default=85, ge=1, le=100)
|
||||
|
||||
max_tokens: int = Field(default=750, ge=1, le=8192)
|
||||
temperature: float = Field(default=1e-6, ge=0.0, le=2.0)
|
||||
repetition_penalty: float = Field(default=1.05, ge=1.0, le=2.0)
|
||||
|
||||
analysis_prompt: str = (
|
||||
"Please analyze whether there are any inconsistencies or obvious signs of forgery in the video, "
|
||||
"and finally come to a conclusion: Is this video real or fake?\n\n"
|
||||
"Return your final answer clearly as one of: REAL / FAKE / UNCERTAIN, then explain briefly why."
|
||||
)
|
||||
|
||||
# Semantic analysis settings
|
||||
semantic_chunk_duration_s: float = Field(default=10.0, ge=1.0, le=60.0)
|
||||
semantic_frames_per_chunk: int = Field(default=24, ge=4, le=64)
|
||||
semantic_enable_aggregation: bool = Field(default=True)
|
||||
semantic_aggregation_model: str = Field(default="qwen3.5")
|
||||
semantic_llm_base_url: str = Field(default="http://didiAI-llm-api:14011")
|
||||
semantic_llm_api_key: str | None = Field(default=None)
|
||||
|
||||
semantic_prompt: str = (
|
||||
"Describe in detail what is happening in this video segment. Include:\n"
|
||||
"- Main actions and activities\n"
|
||||
"- Objects and people present\n"
|
||||
"- Scene location and setting\n"
|
||||
"- Any significant events or changes\n\n"
|
||||
"Be concise but comprehensive."
|
||||
)
|
||||
|
||||
aggregation_prompt_template: str = (
|
||||
"Below are descriptions of {num_chunks} sequential video segments totaling {duration}s. "
|
||||
"Synthesize them into a coherent narrative describing the entire video.\n\n"
|
||||
"{chunks_text}\n\n"
|
||||
"Provide a comprehensive summary that captures the full context and flow of the video."
|
||||
)
|
||||
|
||||
# Runtime config (dashboard polling)
|
||||
dashboard_url: str | None = Field(
|
||||
default=None,
|
||||
description=(
|
||||
"Optional dashboard base URL. When set, runtime_config polls "
|
||||
"/api/config every 30s for live overrides."
|
||||
),
|
||||
)
|
||||
|
||||
def __init__(self, **kwargs):
|
||||
yaml_config = load_yaml_config()
|
||||
# yaml values are defaults, kwargs (env vars) override
|
||||
super().__init__(**{**yaml_config, **kwargs})
|
||||
|
||||
|
||||
settings = Settings()
|
||||
|
||||
|
||||
|
|
@ -0,0 +1,180 @@
|
|||
import time
|
||||
from typing import Any, Dict, List, Tuple
|
||||
|
||||
import cv2
|
||||
import numpy as np
|
||||
|
||||
|
||||
def get_video_props(cap: cv2.VideoCapture) -> Dict[str, Any]:
|
||||
total_frames = int(cap.get(cv2.CAP_PROP_FRAME_COUNT)) or 0
|
||||
fps = float(cap.get(cv2.CAP_PROP_FPS)) or 0.0
|
||||
duration_s = (total_frames / fps) if (total_frames > 0 and fps > 0) else None
|
||||
w = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH)) or None
|
||||
h = int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT)) or None
|
||||
return {"total_frames": total_frames, "fps": fps, "duration_s": duration_s, "width": w, "height": h}
|
||||
|
||||
|
||||
def compute_uniform_indices(total: int, num_frames: int) -> List[int]:
|
||||
if num_frames <= 1:
|
||||
return [0]
|
||||
return [int(round(i * (total - 1) / (num_frames - 1))) for i in range(num_frames)]
|
||||
|
||||
|
||||
def sample_frames_uniform(video_path: str, num_frames: int = 16) -> Tuple[List[Any], Dict[str, Any]]:
|
||||
t0 = time.time()
|
||||
cap = cv2.VideoCapture(video_path)
|
||||
if not cap.isOpened():
|
||||
raise RuntimeError(f"Could not open video: {video_path}")
|
||||
|
||||
props = get_video_props(cap)
|
||||
total = props["total_frames"]
|
||||
fps = props["fps"]
|
||||
|
||||
frames = []
|
||||
idxs: List[int] = []
|
||||
|
||||
if total > 0:
|
||||
idxs = compute_uniform_indices(total, num_frames)
|
||||
for idx in idxs:
|
||||
cap.set(cv2.CAP_PROP_POS_FRAMES, idx)
|
||||
ok, frame = cap.read()
|
||||
if not ok:
|
||||
break
|
||||
frames.append(frame)
|
||||
else:
|
||||
raw = []
|
||||
for _ in range(2000):
|
||||
ok, frame = cap.read()
|
||||
if not ok:
|
||||
break
|
||||
raw.append(frame)
|
||||
if not raw:
|
||||
raise RuntimeError("No frames could be read from video.")
|
||||
step = max(1, len(raw) // num_frames)
|
||||
frames = raw[::step][:num_frames]
|
||||
idxs = list(range(0, step * len(frames), step))
|
||||
|
||||
cap.release()
|
||||
if not frames:
|
||||
raise RuntimeError("No frames sampled from video.")
|
||||
|
||||
timestamps = [(idx / fps) if fps > 0 else None for idx in idxs[: len(frames)]]
|
||||
|
||||
meta = {
|
||||
**props,
|
||||
"frames_requested": num_frames,
|
||||
"sampled": len(frames),
|
||||
"indices": idxs[: len(frames)],
|
||||
"timestamps_s": timestamps,
|
||||
"sampling_method": "uniform_indices" if total > 0 else "sequential_fallback",
|
||||
"sampling_time_s": round(time.time() - t0, 4),
|
||||
}
|
||||
return frames, meta
|
||||
|
||||
|
||||
def sample_frames_chunked(
|
||||
video_path: str,
|
||||
chunk_duration_s: float = 10.0,
|
||||
frames_per_chunk: int = 24,
|
||||
) -> Tuple[List[List[Any]], Dict[str, Any]]:
|
||||
"""
|
||||
Sample video in temporal chunks with dense sampling.
|
||||
|
||||
For semantic analysis that requires understanding the full video context,
|
||||
this function divides the video into chunks and samples each chunk densely.
|
||||
|
||||
Args:
|
||||
video_path: Path to video file
|
||||
chunk_duration_s: Duration of each chunk in seconds (default: 10.0)
|
||||
frames_per_chunk: Number of frames to sample per chunk (default: 24)
|
||||
|
||||
Returns:
|
||||
chunks: List of frame lists [[chunk1_frames], [chunk2_frames], ...]
|
||||
meta: Metadata with chunk information
|
||||
|
||||
Example:
|
||||
For a 60s video with chunk_duration_s=10 and frames_per_chunk=24:
|
||||
- Creates 6 chunks (0-10s, 10-20s, ..., 50-60s)
|
||||
- Samples 24 frames per chunk (2.4 fps within each chunk)
|
||||
- Total: 144 frames analyzed (vs 16 in uniform sampling)
|
||||
"""
|
||||
t0 = time.time()
|
||||
cap = cv2.VideoCapture(video_path)
|
||||
if not cap.isOpened():
|
||||
raise RuntimeError(f"Could not open video: {video_path}")
|
||||
|
||||
props = get_video_props(cap)
|
||||
fps = props["fps"]
|
||||
duration_s = props["duration_s"]
|
||||
|
||||
if not duration_s or fps <= 0:
|
||||
raise RuntimeError("Cannot determine video duration/fps for chunked sampling")
|
||||
|
||||
# Calculate number of chunks
|
||||
num_chunks = int(np.ceil(duration_s / chunk_duration_s))
|
||||
|
||||
chunks = []
|
||||
chunk_metadata = []
|
||||
|
||||
for chunk_idx in range(num_chunks):
|
||||
# Temporal interval for this chunk
|
||||
start_time = chunk_idx * chunk_duration_s
|
||||
end_time = min((chunk_idx + 1) * chunk_duration_s, duration_s)
|
||||
|
||||
# Frame indices for this interval
|
||||
start_frame = int(start_time * fps)
|
||||
end_frame = int(end_time * fps)
|
||||
|
||||
# Sample uniformly within this chunk
|
||||
chunk_total_frames = end_frame - start_frame
|
||||
if chunk_total_frames <= 0:
|
||||
continue
|
||||
|
||||
# Calculate uniform indices within this chunk
|
||||
if frames_per_chunk >= chunk_total_frames:
|
||||
# Take all frames if chunk is small
|
||||
indices = list(range(start_frame, end_frame))
|
||||
else:
|
||||
# Uniform sampling within chunk
|
||||
indices = compute_uniform_indices(chunk_total_frames, frames_per_chunk)
|
||||
# Offset by start_frame to get absolute indices
|
||||
indices = [start_frame + idx for idx in indices]
|
||||
|
||||
# Extract frames
|
||||
chunk_frames = []
|
||||
for idx in indices:
|
||||
cap.set(cv2.CAP_PROP_POS_FRAMES, idx)
|
||||
ok, frame = cap.read()
|
||||
if ok:
|
||||
chunk_frames.append(frame)
|
||||
|
||||
if chunk_frames:
|
||||
chunks.append(chunk_frames)
|
||||
chunk_metadata.append({
|
||||
"chunk_idx": chunk_idx,
|
||||
"start_time_s": round(start_time, 2),
|
||||
"end_time_s": round(end_time, 2),
|
||||
"start_frame": start_frame,
|
||||
"end_frame": end_frame,
|
||||
"sampled": len(chunk_frames),
|
||||
"indices": indices[: len(chunk_frames)],
|
||||
})
|
||||
|
||||
cap.release()
|
||||
|
||||
if not chunks:
|
||||
raise RuntimeError("No chunks sampled from video")
|
||||
|
||||
meta = {
|
||||
**props,
|
||||
"num_chunks": len(chunks),
|
||||
"chunk_duration_s": chunk_duration_s,
|
||||
"frames_per_chunk": frames_per_chunk,
|
||||
"chunks": chunk_metadata,
|
||||
"total_frames_sampled": sum(len(c) for c in chunks),
|
||||
"sampling_method": "temporal_chunked",
|
||||
"sampling_time_s": round(time.time() - t0, 4),
|
||||
}
|
||||
|
||||
return chunks, meta
|
||||
|
||||
0
ai_platform/modules/video-analysis/tests/__init__.py
Normal file
0
ai_platform/modules/video-analysis/tests/__init__.py
Normal file
48
ai_platform/modules/video-analysis/tests/test_verdict.py
Normal file
48
ai_platform/modules/video-analysis/tests/test_verdict.py
Normal file
|
|
@ -0,0 +1,48 @@
|
|||
"""Tests for verdict parsing and per-frame evidence mapping."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from video_analysis.buster_client import (
|
||||
build_frame_evidence,
|
||||
parse_verdict_and_explanation,
|
||||
)
|
||||
|
||||
|
||||
def test_parse_real():
|
||||
out = parse_verdict_and_explanation("REAL — no signs of manipulation.")
|
||||
assert out["verdict"] == "REAL"
|
||||
assert out["explanation"].startswith("REAL")
|
||||
|
||||
|
||||
def test_parse_fake():
|
||||
assert parse_verdict_and_explanation("FAKE, visible artifacts")["verdict"] == "FAKE"
|
||||
|
||||
|
||||
def test_parse_unclear_is_uncertain():
|
||||
# Anything that is not clearly REAL/FAKE maps to UNCERTAIN (offer enum).
|
||||
out = parse_verdict_and_explanation("Hard to tell, mixed signals.")
|
||||
assert out["verdict"] == "UNCERTAIN"
|
||||
|
||||
|
||||
def test_parse_empty_is_uncertain():
|
||||
assert parse_verdict_and_explanation("")["verdict"] == "UNCERTAIN"
|
||||
|
||||
|
||||
def test_build_frame_evidence_aligns_indices_and_timestamps():
|
||||
ev = build_frame_evidence([0, 30, 60], [0.0, 1.0, 2.0])
|
||||
assert ev == [
|
||||
{"frame_index": 0, "timestamp_s": 0.0},
|
||||
{"frame_index": 30, "timestamp_s": 1.0},
|
||||
{"frame_index": 60, "timestamp_s": 2.0},
|
||||
]
|
||||
|
||||
|
||||
def test_build_frame_evidence_handles_missing_timestamps():
|
||||
ev = build_frame_evidence([5, 10], [None])
|
||||
assert ev[0] == {"frame_index": 5, "timestamp_s": None}
|
||||
# Second frame has no timestamp entry → defaults to None.
|
||||
assert ev[1] == {"frame_index": 10, "timestamp_s": None}
|
||||
|
||||
|
||||
def test_build_frame_evidence_empty():
|
||||
assert build_frame_evidence([], []) == []
|
||||
Loading…
Add table
Add a link
Reference in a new issue