Livrare LOT 1 - Didi
This commit is contained in:
commit
5380c3fc63
990 changed files with 133308 additions and 0 deletions
275
ai_platform/modules/video-analysis/README.md
Normal file
275
ai_platform/modules/video-analysis/README.md
Normal file
|
|
@ -0,0 +1,275 @@
|
|||
# Video Analysis
|
||||
|
||||
Video analysis service for deepfake detection using semantic analysis via vLLM backends.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
**vLLM Server Requirement:**
|
||||
This service **does NOT start or manage a vLLM instance**.
|
||||
You must have a **vLLM-compatible server already running and accessible**
|
||||
from the machine where this service is deployed.
|
||||
|
||||
- By default, the service expects vLLM at the URL set in VIDEO_ANALYSIS_VLLM_BASE_URL
|
||||
- This can be changed via: VIDEO_ANALYSIS_VLLM_BASE_URL=http://<host>:<port>
|
||||
|
||||
**Important:**
|
||||
- If you run `video-analysis` in Docker with `network_mode: host`,
|
||||
the vLLM server must be reachable from the host network.
|
||||
- vLLM may run:
|
||||
- locally on the same machine, **or**
|
||||
- on another machine, as long as the URL is reachable.
|
||||
|
||||
The service will **fail to start** if the vLLM endpoint is not reachable.
|
||||
|
||||
**Required:**
|
||||
- All global prerequisites (see main [README.md](../../README.md))
|
||||
- A running vLLM server with a vision-language model (e.g., BusterX)
|
||||
|
||||
**vLLM Server Requirements:**
|
||||
- GPU required; VRAM depends on model (recommend ≥16GB, may require more).
|
||||
- NVIDIA Driver 535+
|
||||
- NVIDIA Container Toolkit
|
||||
|
||||
> **Note:** `video-analysis` itself runs on CPU. The GPU is only needed for the vLLM server.
|
||||
|
||||
## Features
|
||||
|
||||
- **Deepfake Detection**: Analyzes videos for signs of manipulation
|
||||
- **Uniform Frame Sampling**: Extracts representative frames from videos
|
||||
- **vLLM Integration**: Uses vision-language models for semantic analysis
|
||||
- **Reproducibility Artifacts (optional)**: Saves request/response artifacts to a configurable runs directory for debugging and reproducibility
|
||||
(this directory is created/used at runtime; it is not meant to be committed to the repo)
|
||||
|
||||
## Installation
|
||||
|
||||
```bash
|
||||
cd modules/video-analysis
|
||||
|
||||
# Install dependencies
|
||||
uv sync
|
||||
|
||||
# Install with dev dependencies
|
||||
uv sync --extra dev
|
||||
```
|
||||
|
||||
## Quick Start
|
||||
|
||||
### As an API Server
|
||||
|
||||
```bash
|
||||
cd deploy/
|
||||
|
||||
# Create deployment env file
|
||||
# NOTE: deploy.sh loads env ONLY from deploy/.env
|
||||
cp ../.env.example .env
|
||||
# Edit deploy/.env with your vLLM server URL and model
|
||||
|
||||
# Start the server
|
||||
./deploy.sh --profile api --detach
|
||||
|
||||
# Or with nginx reverse proxy
|
||||
./deploy.sh --profile api-nginx --detach
|
||||
```
|
||||
|
||||
### API Endpoints
|
||||
|
||||
| Endpoint | Method | Description |
|
||||
|----------|--------|-------------|
|
||||
| `/health` | GET | Health check |
|
||||
| `/analyze/video` | POST | Deepfake detection (fast, 16 frames) |
|
||||
| `/analyze/video/semantic` | POST | Semantic analysis (detailed, 144+ frames) |
|
||||
|
||||
### Example API Request
|
||||
|
||||
```bash
|
||||
# Health check
|
||||
curl http://localhost:54600/health
|
||||
|
||||
# Analyze video
|
||||
VIDEO="/path/to/video.mp4"
|
||||
curl -sS -X POST "http://localhost:54600/analyze/video" -F "file=@${VIDEO}"
|
||||
```
|
||||
|
||||
### Example Response
|
||||
|
||||
```json
|
||||
{
|
||||
"request_id": "550e8400-e29b-41d4-a716-446655440000",
|
||||
"run_dir": "runs/550e8400-e29b-41d4-a716-446655440000",
|
||||
"verdict": "FAKE",
|
||||
"explanation": "The video shows clear signs of manipulation...",
|
||||
"usage": {
|
||||
"prompt_tokens": 1250,
|
||||
"completion_tokens": 150,
|
||||
"total_tokens": 1400
|
||||
},
|
||||
"latency_s": {
|
||||
"sampling_time_s": 0.234,
|
||||
"encode_time_s": 0.567,
|
||||
"model_inference_time_s": 12.345
|
||||
},
|
||||
"meta": {
|
||||
"fps": 30.0,
|
||||
"total_frames": 450,
|
||||
"duration_s": 15.0,
|
||||
"sampled": 16
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
### Required Environment Variables
|
||||
|
||||
Configured via environment variables (prefix: `VIDEO_ANALYSIS_`). These are typically set in `deploy/.env`:
|
||||
|
||||
| Variable | Description |
|
||||
|----------|-------------|
|
||||
| `VIDEO_ANALYSIS_VLLM_BASE_URL` | vLLM server URL (e.g., `http://didiAI-video-vllm-buster:8000`) |
|
||||
| `VIDEO_ANALYSIS_VLLM_MODEL` | Model name (e.g., `l8cv/BusterX_plusplus`) |
|
||||
| `VIDEO_ANALYSIS_RUNS_DIR` | Directory for storing analysis artifacts (created/used at runtime) |
|
||||
|
||||
| `VIDEO_ANALYSIS_EXTERNAL_URL` | External URL for OpenAPI spec (e.g., `http://localhost:54600`) |
|
||||
|
||||
|
||||
### Optional Nginx Environment Variables (api-nginx profile)
|
||||
|
||||
If you use the `api-nginx` profile, the nginx container can read these optional variables from `deploy/.env`:
|
||||
|
||||
| Variable | Example | Description |
|
||||
|----------|---------|-------------|
|
||||
| `NGINX_CONNECT_TIMEOUT` | `60s` | Upstream connect timeout |
|
||||
| `NGINX_SEND_TIMEOUT` | `120s` | Upstream send timeout |
|
||||
| `NGINX_READ_TIMEOUT` | `600s` | Upstream read timeout |
|
||||
|
||||
### Optional Tuning Parameters
|
||||
|
||||
Configured via `deploy/config.yaml` (env vars override YAML):
|
||||
|
||||
| Parameter | Default | Description |
|
||||
|-----------|---------|-------------|
|
||||
| `frames` | `16` | Number of frames to sample |
|
||||
| `max_side` | `960` | Max image dimension (pixels) |
|
||||
| `jpeg_quality` | `85` | JPEG encoding quality (1-100) |
|
||||
| `max_tokens` | `750` | Max tokens for model response |
|
||||
| `temperature` | `0.000001` | Sampling temperature |
|
||||
| `repetition_penalty` | `1.05` | Repetition penalty |
|
||||
|
||||
## ⚠️ Testing Recommendations
|
||||
|
||||
### Model Selection for Semantic Analysis
|
||||
|
||||
**Current Setup:**
|
||||
- Both deepfake and semantic analysis use **BusterX** (7B parameters)
|
||||
- BusterX is optimized for deepfake detection
|
||||
|
||||
**TODO: Test Semantic Analysis with Qwen3-VL-30B**
|
||||
|
||||
For better semantic understanding, consider testing with the larger **Qwen3-VL-30B** model (already running @ port 8102):
|
||||
|
||||
```bash
|
||||
# Current (BusterX 7B)
|
||||
VIDEO_ANALYSIS_VLLM_BASE_URL=http://didiAI-video-vllm-buster:54500
|
||||
VIDEO_ANALYSIS_VLLM_MODEL=busterx
|
||||
|
||||
# Alternative (Qwen3-VL 30B) - Better for semantic analysis
|
||||
VIDEO_ANALYSIS_VLLM_BASE_URL=http://didiAI-llm-vllm-vision:14002
|
||||
VIDEO_ANALYSIS_VLLM_MODEL=qwen3-vl
|
||||
```
|
||||
|
||||
**Expected Improvements:**
|
||||
- ✅ More detailed scene descriptions (30B vs 7B)
|
||||
- ✅ Better understanding of complex actions
|
||||
- ✅ More coherent narrative synthesis
|
||||
- ✅ Higher quality semantic annotations
|
||||
|
||||
**Trade-offs:**
|
||||
- ⏱️ Slightly higher latency (~15-20s per chunk vs ~12s)
|
||||
- 📊 Better for semantic analysis, but keep BusterX for deepfake detection
|
||||
|
||||
**Recommendation:**
|
||||
- **Deepfake endpoint:** Keep using BusterX (specialized for forgery detection)
|
||||
- **Semantic endpoint:** Test with Qwen3-VL-30B for better results
|
||||
|
||||
## Deployment
|
||||
|
||||
```bash
|
||||
cd deploy/
|
||||
|
||||
# Create deployment env file (REQUIRED)
|
||||
cp ../.env.example .env
|
||||
# Edit deploy/.env with your settings
|
||||
|
||||
# API only
|
||||
./deploy.sh --profile api --detach
|
||||
|
||||
# API with nginx reverse proxy
|
||||
./deploy.sh --profile api-nginx --detach
|
||||
|
||||
# View logs
|
||||
./deploy.sh --profile api --logs
|
||||
|
||||
# Stop services
|
||||
./deploy.sh --profile api --down
|
||||
```
|
||||
|
||||
### Common Docker Commands
|
||||
|
||||
```bash
|
||||
cd deploy/
|
||||
|
||||
# Restart the API service (compose service name)
|
||||
docker compose restart video-analysis-api
|
||||
|
||||
# (Optional) restart by container name
|
||||
docker restart video_analysis
|
||||
```
|
||||
|
||||
### Port Allocation
|
||||
|
||||
| Port | Service | Environment |
|
||||
|------|---------|-------------|
|
||||
| `54600` | Video Analysis API | Development |
|
||||
| `54500` | BusterX vLLM Server | Development |
|
||||
|
||||
## Development
|
||||
|
||||
```bash
|
||||
# Install dev dependencies
|
||||
uv sync --extra dev
|
||||
|
||||
# Run tests
|
||||
uv run pytest
|
||||
|
||||
# Run tests with coverage
|
||||
uv run pytest --cov=src/video_analysis --cov-report=term-missing
|
||||
|
||||
# Lint and format
|
||||
uv run ruff check .
|
||||
uv run ruff format .
|
||||
```
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
modules/video-analysis/
|
||||
├── deploy/
|
||||
│ ├── deploy.sh # Deployment script
|
||||
│ ├── docker-compose.yml # Docker services
|
||||
│ ├── Dockerfile # Container image
|
||||
│ ├── nginx.conf # Nginx reverse proxy config (optional)
|
||||
│ ├── nginx.conf.template # Template-based nginx config (optional)
|
||||
│ └── config.yaml # Tuning parameters
|
||||
├── src/video_analysis/
|
||||
│ ├── __init__.py
|
||||
│ ├── app.py # FastAPI application
|
||||
│ ├── buster_client.py # vLLM client
|
||||
│ ├── schemas.py # Response schemas
|
||||
│ ├── settings.py # Configuration (env + yaml)
|
||||
│ └── video_sampling.py # Frame extraction
|
||||
├── tests/
|
||||
├── .env.example # Environment template
|
||||
├── API.md # API documentation
|
||||
├── pyproject.toml # Dependencies
|
||||
└── README.md # This file
|
||||
```
|
||||
Loading…
Add table
Add a link
Reference in a new issue