137 lines
7.4 KiB
Markdown
137 lines
7.4 KiB
Markdown
# Audio Transcription Service (M17-Whisper)
|
||
|
||
Audio transcription service for DIDI media analysis. Whisper-based (M17-Whisper, faster-whisper backend), used by agent-v3 media-preprocess worker for video/audio session pipelines (techniques, ai-tampered, claims). OpenAI-compatible API — drop-in for `client.audio.transcriptions.create()`.
|
||
|
||
## Stack
|
||
- Python 3.10+ / FastAPI / Uvicorn
|
||
- Backend: `faster-whisper` >= 1.0.0 (CTranslate2 optimized inference)
|
||
- Default model: `large-v3-turbo` (809M params, ~6GB VRAM int8)
|
||
- GPU: CUDA (shared GPU 0 with Qwen3.5-35B-A3B)
|
||
- URL (Dev): `http://10.11.10.17:54300/v1/audio/transcriptions`
|
||
- Container: `didiAI-audio` (GPU host)
|
||
- Auth: none on the service itself; agent-v3 uses bearer token via `M17_WHISPER_TOKEN` (enforced by gateway/nginx if configured)
|
||
|
||
## Ce face
|
||
Speech-to-text transcription pe fişiere audio sau URL-uri. agent-v3 trimite fie audio buffer (multipart upload), fie URL public (din MinIO), primeşte text + metadata (lang detect, duration, optional segments). Folosit în:
|
||
- **Video analysis pipeline** — agent-v3 extrage track-ul audio cu ffmpeg, trimite la M17-Whisper, foloseşte transcript pentru `claims` + `techniques` + `ai-tampered`
|
||
- **Audio-only sessions** — direct upload, transcripted, apoi pipeline normal de analiză text
|
||
- **Cascade fallback** — agent-v3 `transcription.ts` are cascadă: M17-Whisper (local GPU) → Groq Whisper → OpenAI Whisper. Dacă local pică sau întoarce string gol, trece la următorul provider.
|
||
|
||
## API endpoints
|
||
|
||
| Method | Path | Purpose |
|
||
|--------|------|---------|
|
||
| `GET` | `/health` | Health probe (returns `{"status":"ok"}`) |
|
||
| `GET` | `/v1/models` | List models, OpenAI-compatible |
|
||
| `POST` | `/v1/audio/transcriptions` | Transcribe audio (OpenAI-compatible) |
|
||
| `GET` | `/v1/info` | Service catalog metadata (used by didi catalog-api) |
|
||
|
||
### `POST /v1/audio/transcriptions`
|
||
Content-Type: `multipart/form-data`
|
||
|
||
| Param | Type | Default | Notes |
|
||
|-------|------|---------|-------|
|
||
| `file` | file | — | Audio file (MP3/WAV/M4A/FLAC/OGG/MP4 audio track…). XOR with `url`. |
|
||
| `url` | string | — | URL to download audio from (DIDI extension over OpenAI). XOR with `file`. |
|
||
| `model` | string | configured | Ignored — server uses `AUDIO_MODEL` env. |
|
||
| `language` | string | auto | ISO-639-1 code (`en`, `ro`, `es`, …). Auto-detect if omitted. |
|
||
| `prompt` | string | none | Optional initial prompt to bias style/vocab. |
|
||
| `response_format` | string | `json` | `json` \| `text` \| `verbose_json` (segments). |
|
||
| `temperature` | float | `0.0` | Sampling temperature 0.0–1.0. |
|
||
|
||
Response (json): `{ text, language, duration }` — verbose_json adds `segments[]` with `start/end/text/tokens/avg_logprob/no_speech_prob`.
|
||
|
||
Errors: `400` invalid params / download failed, `413` payload too large (default 500MB), `422` corrupted audio, `500` transcription failed, `503` model not loaded.
|
||
|
||
## Backends / Models
|
||
- Production: `large-v3-turbo` (8x faster than `large-v3`, similar quality, ~6GB VRAM int8)
|
||
- Available: `tiny`, `base`, `small`, `medium`, `large-v3`, `large-v3-turbo`
|
||
- VAD (Voice Activity Detection) enabled by default — skip silence
|
||
- Quantization: `int8` (recommended), `float16`, `int8_float16`
|
||
|
||
## How didi-backend uses it
|
||
- agent-v3 cascade: `backend/services/orchestration-layer/agent-v3/src/shared/media/transcription.ts`
|
||
- Order: **M17-Whisper (local GPU)** → Groq Whisper → OpenAI Whisper
|
||
- Empty-string from local triggers retry on next provider (Faza 1 din SAFETY_NETS_PLAN)
|
||
- Configured în agent-v3 docker-compose env:
|
||
- `M17_WHISPER_URL=http://10.11.10.17:54300`
|
||
- `M17_WHISPER_TOKEN=<gateway-token>` (optional, dacă există proxy auth)
|
||
- Apelat din **media-preprocess worker** la sesiuni cu `media_type` în `{audio, video}` — transcript devine input pentru `claims-routes.ts`, `routes.ts` (techniques), `ai-tampered-routes.ts`.
|
||
- Video flow: ffmpeg extract → upload temp / pass buffer → POST `/v1/audio/transcriptions` → text → pipeline analysis.
|
||
|
||
## Configuration
|
||
|
||
Env vars (prefix `AUDIO_`, set in `deploy/.env`):
|
||
|
||
**Required**
|
||
- `AUDIO_MODEL` — Whisper model (default `large-v3-turbo`)
|
||
- `AUDIO_DEVICE` — `cuda` | `cpu`
|
||
- `AUDIO_CACHE_DIR` — HuggingFace model cache path (`/cai2_ds_storage/hf_cache` typical)
|
||
- `AUDIO_EXTERNAL_URL` — used in OpenAPI servers spec
|
||
|
||
**Optional**
|
||
- `AUDIO_COMPUTE_TYPE` — `int8` (default), `float16`, `int8_float16`
|
||
- `AUDIO_BEAM_SIZE` — beam search width (default 5)
|
||
- `AUDIO_BEST_OF` — sampling candidates (default 5)
|
||
- `AUDIO_TEMPERATURE` — default 0.0
|
||
- `AUDIO_HOST` / `AUDIO_PORT` — default `0.0.0.0:54300`
|
||
- `AUDIO_LOG_LEVEL` — default `INFO`
|
||
- `AUDIO_MAX_FILE_SIZE_MB` — upload cap (default 500)
|
||
- `CUDA_VISIBLE_DEVICES=0` — pinned to GPU 0
|
||
|
||
## Deployment
|
||
|
||
Compose dir: `/home/admin365/didi_mono/ai_platform/modules/audio/deploy/`
|
||
|
||
```bash
|
||
cd modules/audio/deploy
|
||
cp ../.env.example .env # edit values
|
||
./deploy.sh --profile api --detach # API only
|
||
./deploy.sh --profile api-nginx --detach # with nginx reverse proxy
|
||
./deploy.sh --profile api --logs # tail logs
|
||
docker compose restart audio-api # quick restart
|
||
```
|
||
|
||
- Container name: `didiAI-audio`
|
||
- Image: `didiai-audio:audit`
|
||
- Network: `didi-network` (external, shared with other AI modules)
|
||
- GPU reservation: NVIDIA driver, device `0`
|
||
- Healthcheck: HTTP `GET /health` every 30s, 60s start period (model load)
|
||
- Model cache mount: `${AUDIO_CACHE_DIR}:/root/.cache/huggingface`
|
||
|
||
## Performance
|
||
- Real-Time Factor on H200 GPU + int8 + `large-v3-turbo`: **~0.05–0.08x** (10s audio = 0.5–1s transcription, 1h audio = 3–5min)
|
||
- VRAM: ~6GB for large-v3-turbo int8, ~8GB float16
|
||
- Throughput: sequential — process files one at a time per GPU
|
||
- File size cap: 500MB default (configurable)
|
||
- Recommended: pre-split audio > 1h cu ffmpeg segments
|
||
|
||
## Architecture (source layout)
|
||
```
|
||
modules/audio/
|
||
├── deploy/
|
||
│ ├── deploy.sh # CLI wrapper
|
||
│ ├── docker-compose.yml # didiAI-audio service
|
||
│ ├── Dockerfile # CUDA + faster-whisper image
|
||
│ └── .env # runtime config
|
||
├── src/audio/
|
||
│ ├── app.py # FastAPI routes + /v1/info catalog
|
||
│ ├── transcriber.py # faster-whisper wrapper (singleton)
|
||
│ ├── schemas.py # TranscriptionResponse / Segment
|
||
│ └── settings.py # pydantic-settings (AUDIO_* env)
|
||
├── tests/
|
||
├── pyproject.toml # fastapi, faster-whisper, httpx
|
||
├── README.md # full operator guide
|
||
├── API.md # full HTTP API reference
|
||
└── INDEX.md # this file
|
||
```
|
||
|
||
## Related
|
||
- agent-v3 transcription cascade — `backend/services/orchestration-layer/agent-v3/src/shared/media/transcription.ts`
|
||
- agent-v3 media routes — `routes.ts` / `ai-tampered-routes.ts` / `claims-routes.ts`
|
||
- media-preprocess worker (in agent-v3) — invokes this service
|
||
- Test files in MinIO (used during refactor verification):
|
||
- Audio: `https://didi365.eu/api/v3/media/file/uploads/test-user/1771883851173-audio_with_voice.mp3`
|
||
- Video: `https://didi365.eu/api/v3/media/file/uploads/test-user/1771883852186-voice_video.mp4`
|
||
- Faza 1 safety net (transcription retry on empty) — `agent-v3/SAFETY_NETS_PLAN.md`
|
||
- Upstream refs: [faster-whisper](https://github.com/SYSTRAN/faster-whisper), [OpenAI Whisper](https://github.com/openai/whisper)
|