didi-lot1-ai/ai_platform/modules/video-analysis/API.md

# Video Analysis API Documentation

REST API for video deepfake detection using semantic analysis via vLLM backends.
Note: This API depends on an external vLLM server configured via `VIDEO_ANALYSIS_VLLM_BASE_URL`.

---

## Base URL

`{BASE_URL}`

Common configurations:
- **Local development:** `http://localhost:54600`
- **Docker (internal):** `http://didiAI-video-api:54600`
- **Production:** Use your configured hostname

---

## Authentication

No authentication required.

---

## ⚠️ TESTING REMINDER: Alternative Vision Models

**Current Configuration:**
- **Deepfake Detection:** Uses BusterX (Qwen2.5-VL-7B fine-tuned) @ port 54500
- **Semantic Analysis:** Uses BusterX (7B parameters)

**TODO - Test with Qwen3-VL-30B for Better Semantic Analysis:**

The semantic analysis endpoint can be configured to use **Qwen3-VL-30B** (already running @ port 14002) instead of BusterX for potentially better results:

| Model | Size | Port | Best For |
|-------|------|------|----------|
| **BusterX** | 7B | 54500 | Deepfake detection (specialized) |
| **Qwen3-VL-30B** | 30B | 14002 | General semantic understanding |

**To test with Qwen3-VL-30B:**

1. Update `.env`:
```bash
VIDEO_ANALYSIS_VLLM_BASE_URL=http://didiAI-llm-vllm-vision:14002  # Use Qwen3-VL instead of BusterX
VIDEO_ANALYSIS_VLLM_MODEL=qwen3-vl                     # Change from busterx
```

2. Rebuild container:
```bash
cd deploy/
docker compose build video-analysis-api
docker compose up -d video-analysis-api
```

3. Test semantic analysis:
```bash
curl -X POST http://localhost:54600/analyze/video/semantic \
  -F "file=@test_video.mp4"
```

**Expected Benefits:**
- More detailed scene descriptions (30B vs 7B parameters)
- Better context understanding
- More coherent narrative flow
- Higher accuracy for complex scenes

**Note:** Deepfake detection should continue using BusterX (specialized model).

---

## Endpoints

### Health Check

Check if the service is running.

**GET** `/health`

**Response**

```json
{
  "status": "ok"
}
```

**Example**

```bash
curl http://localhost:54600/health
```

---

### Analyze Video

Upload a video for deepfake analysis.

**POST** `/analyze/video`

#### Request

**Content-Type:** `multipart/form-data`

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `file` | file | Yes | Video file to analyze |

**Example**

```bash
curl -X POST http://localhost:54600/analyze/video \
  -F "file=@/path/to/video.mp4"
```

#### Response

```json
{
  "request_id": "550e8400-e29b-41d4-a716-446655440000",
  "run_dir": "runs/550e8400-e29b-41d4-a716-446655440000",
  "verdict": "FAKE",
  "explanation": "The video shows clear signs of manipulation...",
  "frames_analyzed": 16,
  "evidence": [
    {"frame_index": 0, "timestamp_s": 0.0},
    {"frame_index": 30, "timestamp_s": 1.0}
  ],
  "usage": {
    "prompt_tokens": 1250,
    "completion_tokens": 150,
    "total_tokens": 1400
  },
  "latency_s": {
    "sampling_time_s": 0.234,
    "encode_time_s": 0.567,
    "model_inference_time_s": 12.345
  },
  "meta": {
    "fps": 30.0,
    "total_frames": 450,
    "duration_s": 15.0,
    "sampled": 16,
    "indices": [0, 30, 60, 90, 120, 150, 180, 210, 240, 270, 300, 330, 360, 390, 420, 449],
    "timestamps_s": [0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0, 10.0, 11.0, 12.0, 13.0, 14.0, 14.97]
  }
}
```

#### Response Fields

| Field | Description |
|-------|-------------|
| `request_id` | UUID identifying the analysis request |
| `run_dir` | Directory where analysis artifacts are stored |
| `verdict` | Classification result: `REAL`, `FAKE`, or `UNCERTAIN` |
| `explanation` | Model explanation for the verdict |
| `usage` | Token usage statistics from the vLLM backend |
| `latency_s` | Timing breakdown in seconds |
| `meta` | Video metadata and frame sampling information |

---

### Semantic Video Analysis

Upload a video for deep semantic analysis with temporal chunking.

**POST** `/analyze/video/semantic`

This endpoint performs comprehensive content understanding by:
1. Dividing the video into temporal chunks (default: 10s each)
2. Densely sampling each chunk (default: 24 frames per chunk)
3. Analyzing each chunk with the vision LLM
4. Optionally aggregating chunk descriptions into a coherent narrative

**Use cases:**
- Content description and understanding
- Action recognition and tracking
- Scene analysis
- Narrative extraction from video

#### Request

**Content-Type:** `multipart/form-data`

| Field | Type | Required | Default | Description |
|-------|------|----------|---------|-------------|
| `file` | file | Yes | - | Video file to analyze |
| `chunk_duration_s` | float | No | 10.0 | Duration of each chunk in seconds (1-60) |
| `frames_per_chunk` | int | No | 24 | Number of frames to sample per chunk (4-64) |
| `enable_aggregation` | bool | No | true | Whether to aggregate chunks into final summary |

**Examples**

Basic semantic analysis (default settings):
```bash
curl -X POST http://localhost:54600/analyze/video/semantic \
  -F "file=@meeting.mp4"
```

Custom chunk settings:
```bash
curl -X POST http://localhost:54600/analyze/video/semantic \
  -F "file=@meeting.mp4" \
  -F "chunk_duration_s=5.0" \
  -F "frames_per_chunk=32"
```

Without aggregation (get only chunk descriptions):
```bash
curl -X POST http://localhost:54600/analyze/video/semantic \
  -F "file=@meeting.mp4" \
  -F "enable_aggregation=false"
```

#### Response

```json
{
  "request_id": "abc-123-def-456",
  "run_dir": "/app/runs/abc-123-def-456",
  "analysis_type": "semantic",
  "video_duration_s": 60.0,
  "num_chunks": 6,
  "chunk_results": [
    {
      "chunk_idx": 0,
      "time_range": "0.0s - 10.0s",
      "description": "A person in business attire enters a conference room, walks to the head of the table, and places a laptop down. The room has white walls and a large window showing daylight outside.",
      "frames_analyzed": 24,
      "inference_time_s": 12.3,
      "usage": {
        "prompt_tokens": 5234,
        "completion_tokens": 87,
        "total_tokens": 5321
      }
    },
    {
      "chunk_idx": 1,
      "time_range": "10.0s - 20.0s",
      "description": "The person opens the laptop and begins gesturing while speaking. Two other people enter the room and take seats at the conference table. One person carries a notebook.",
      "frames_analyzed": 24,
      "inference_time_s": 12.1,
      "usage": {
        "prompt_tokens": 5198,
        "completion_tokens": 92,
        "total_tokens": 5290
      }
    },
    {
      "chunk_idx": 2,
      "time_range": "20.0s - 30.0s",
      "description": "The presenter is now showing content on the laptop screen to the group. All three people are focused on the screen. One person is taking notes.",
      "frames_analyzed": 24,
      "inference_time_s": 11.8,
      "usage": {
        "prompt_tokens": 5201,
        "completion_tokens": 78,
        "total_tokens": 5279
      }
    },
    {
      "chunk_idx": 3,
      "time_range": "30.0s - 40.0s",
      "description": "Discussion is ongoing. The presenter is gesturing toward the screen. One attendee raises their hand and appears to ask a question.",
      "frames_analyzed": 24,
      "inference_time_s": 12.0,
      "usage": {
        "prompt_tokens": 5187,
        "completion_tokens": 71,
        "total_tokens": 5258
      }
    },
    {
      "chunk_idx": 4,
      "time_range": "40.0s - 50.0s",
      "description": "The presenter responds to the question with gestures. All participants are engaged in the discussion. Papers are visible on the table.",
      "frames_analyzed": 24,
      "inference_time_s": 11.9,
      "usage": {
        "prompt_tokens": 5209,
        "completion_tokens": 68,
        "total_tokens": 5277
      }
    },
    {
      "chunk_idx": 5,
      "time_range": "50.0s - 60.0s",
      "description": "The meeting appears to be concluding. Participants are gathering their belongings. The presenter closes the laptop and people begin standing up.",
      "frames_analyzed": 24,
      "inference_time_s": 12.2,
      "usage": {
        "prompt_tokens": 5223,
        "completion_tokens": 75,
        "total_tokens": 5298
      }
    }
  ],
  "final_summary": "The video captures a business meeting in a conference room. It begins with a presenter setting up and two colleagues joining. The presenter delivers a presentation using a laptop, with the group discussing the content. One attendee asks questions and takes notes throughout. The meeting concludes with participants gathering their items and preparing to leave. The entire sequence lasts approximately 60 seconds.",
  "aggregation_time_s": 4.5,
  "total_latency_s": 76.8,
  "meta": {
    "fps": 30.0,
    "total_frames": 1800,
    "duration_s": 60.0,
    "chunk_duration_s": 10.0,
    "frames_per_chunk": 24,
    "total_frames_sampled": 144
  }
}
```

#### Response Fields

| Field | Description |
|-------|-------------|
| `request_id` | UUID identifying the analysis request |
| `run_dir` | Directory where analysis artifacts are stored |
| `analysis_type` | Always "semantic" for this endpoint |
| `video_duration_s` | Total video duration in seconds |
| `num_chunks` | Number of temporal chunks processed |
| `chunk_results` | Array of results for each chunk (see below) |
| `final_summary` | Aggregated narrative (null if aggregation disabled) |
| `aggregation_time_s` | Time spent on aggregation (null if disabled) |
| `total_latency_s` | Total processing time in seconds |
| `meta` | Video metadata and sampling configuration |

**ChunkResult Fields:**

| Field | Description |
|-------|-------------|
| `chunk_idx` | Zero-based chunk index |
| `time_range` | Temporal range of this chunk (e.g., "0.0s - 10.0s") |
| `description` | Semantic description of what happens in this chunk |
| `frames_analyzed` | Number of frames analyzed for this chunk |
| `inference_time_s` | Time spent on VLM inference for this chunk |
| `usage` | Token usage statistics for this chunk |

#### Performance Characteristics

**For a 60-second video with default settings:**

- **Chunks:** 6 (10s each)
- **Total frames analyzed:** 144 (24 per chunk)
- **Coverage:** ~8% of all frames (vs 0.89% for deepfake detection)
- **Latency:** ~77s total
  - Chunk processing: ~72s (6 × 12s per chunk)
  - Aggregation: ~5s
  - Overhead (sampling, encoding): <1s

**Optimization options:**

| Setting | Fast | Balanced | Detailed |
|---------|------|----------|----------|
| `chunk_duration_s` | 20.0 | 10.0 | 5.0 |
| `frames_per_chunk` | 16 | 24 | 32 |
| Coverage (60s video) | ~3% | ~8% | ~21% |
| Latency estimate | ~40s | ~77s | ~150s |

---

## Error Responses

| Status | Description |
|--------|-------------|
| `400` | Bad Request (missing or invalid file) |
| `500` | Internal Server Error |

### Possible 500 Error Causes

- vLLM backend unreachable
- `VIDEO_ANALYSIS_VLLM_BASE_URL` not set
- Model inference failed

**Example**

```json
{
  "detail": "VIDEO_ANALYSIS_VLLM_BASE_URL is not set"
}
```

---

## Minimal Python Client Example

```python
import httpx

def analyze_video(video_path: str) -> dict:
    with open(video_path, "rb") as f:
        response = httpx.post(
            "http://localhost:54600/analyze/video",
            files={"file": f},
            timeout=180.0,
        )
    response.raise_for_status()
    return response.json()

result = analyze_video("suspect_video.mp4")
print(f"Verdict: {result['verdict']}")
print(f"Explanation: {result['explanation']}")
```