168 lines
13 KiB
Markdown
168 lines
13 KiB
Markdown
# llm-inference
|
|
|
|
OpenAI-compatible LLM router for DIDI. Routes requests to the local `Qwen/Qwen3.5-35B-A3B` model (served as `qwen3.5`, via vLLM / llama.cpp backends) for the free tier and proxies to OpenRouter cloud (Claude Sonnet 4.6, Gemini Flash 3, GPT-4o, etc.) for premium models. All endpoints are OpenAI-compatible (`/v1/chat/completions`), so callers can use the OpenAI SDK or plain `httpx` interchangeably.
|
|
|
|
- **Stack**: Python 3.10+, FastAPI, uvicorn, httpx, LiteLLM, Pydantic v2, sse-starlette
|
|
- **URL**: `http://10.11.10.17:14011` (LLM router on GPU host) — referenced as `LLM_ROUTER_URL` in DIDI services
|
|
- **Container**: `didiAI-llm-api` (image `didiai-llm-api`), runs on GPU host (typically `10.11.10.17`)
|
|
- **Local model**: `Qwen/Qwen3.5-35B-A3B` (MoE **reasoning** model; thinking can be toggled on/off — the gateway disables it by default, see `LLM_VLLM_DISABLE_THINKING`) served as `qwen3.5` via vLLM (`didiAI-vllm-qwen3.5`, internal port 14001) or via the llama.cpp pool
|
|
- **Entry point**: `llm-inference` console script -> `src/llm_inference/cli.py:main` -> uvicorn factory `llm_inference.api.app:create_app`
|
|
|
|
## Ce face
|
|
|
|
Single OpenAI-compatible endpoint (`/v1/chat/completions`) that selects a backend based on the request's `model` field and request-time `backend` override:
|
|
|
|
- **Auto backend resolution** (`LLMClient._resolve_backend_for_model` in `src/llm_inference/client.py`): the requested `model` is first normalized through `LLM_MODEL_ALIASES` (e.g. `qwen3.5` → `Qwen/Qwen3.5-35B-A3B`), then each enabled local backend's `list_models()` is probed. If the model is served locally, route there; otherwise fall back to `LLM_DEFAULT_BACKEND`.
|
|
- **Cross-backend fallback cascade** (`LLM_ENABLE_FALLBACK`, default `true`): on a backend failure the request is retried on the next enabled backend in `LLM_FALLBACK_ORDER` (default `[vllm, llamacpp, litellm]`). The cascade is skipped when the caller pins an explicit `backend`.
|
|
- **Explicit backend override**: clients may pass `"backend": "litellm" | "vllm" | "llamacpp"` in the JSON body to force routing.
|
|
- **Streaming + non-streaming**: same endpoint; `"stream": true` returns SSE chunks (`text/event-stream` with `data: [DONE]` terminator).
|
|
- **Multimodal**: messages are pre-processed by `image_processing.process_messages` so image URLs/base64 attachments work the same across backends.
|
|
- **Retries**: transient failures (`LLMTimeoutError`, `LLMRateLimitError`, `LLMConnectionError`) wrapped by `retry.retry_with_backoff` with exponential backoff (`max_retries`, `retry_min_wait`, `retry_max_wait`).
|
|
- **Provider keys**: read from env (`OPENROUTER_API_KEY`, `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`); LiteLLM dispatches to OpenRouter / OpenAI / Anthropic / Gemini / Bedrock / 100+ providers using its own model-id parsing (`openrouter/...`, `claude-3-...`, `gpt-4o`, etc.).
|
|
- **Optional Bearer auth**: when `LLM_API_TOKENS` is set, all `/v1/*` endpoints require `Authorization: Bearer <token>`; `/health` and `/ready` are always public.
|
|
|
|
## API endpoints
|
|
|
|
All paths are mounted by `src/llm_inference/api/app.py`:
|
|
|
|
| Method | Path | Purpose |
|
|
|--------|------|---------|
|
|
| POST | `/v1/chat/completions` | OpenAI-compat chat completion (streaming + non-streaming) |
|
|
| POST | `/v1/completions` | OpenAI-compat legacy text completion (non-streaming; 501 from backends that don't support it) |
|
|
| GET | `/v1/models` | List models from all (or specific via `?backend=`) backends |
|
|
| POST | `/v1/models/load` | Load a model on a local backend (vLLM / llama.cpp) |
|
|
| POST | `/v1/models/unload` | Unload a model from a local backend |
|
|
| GET | `/v1/backends` | List enabled backends |
|
|
| GET | `/health` | Per-backend health (overall: `healthy` / `degraded` / `unhealthy`) |
|
|
| GET | `/ready` | K8s-style readiness probe (200 if default backend healthy, 503 otherwise) |
|
|
| GET | `/v1/info` *(catalog router)* | Service metadata for cross-module catalog integration |
|
|
|
|
Source files for the routers: `src/llm_inference/api/routes/{completions,models,health,info}.py`.
|
|
|
|
## Model routing logic
|
|
|
|
The backend dispatch lives in `LLMClient._resolve_backend_for_model` (auto) and `BackendRegistry.get` (explicit). Model identifier conventions:
|
|
|
|
- **Local vLLM** (`backend=vllm` or auto): the vLLM container exposes Qwen with `--served-model-name qwen3.5`. Requests with `model="qwen3.5"` (or whatever Hugging Face id is loaded) are forwarded to `LLM_VLLM_BASE_URL` (default `http://didiAI-vllm-qwen3.5:14001`) using OpenAI SDK.
|
|
- **Local llama.cpp** (`backend=llamacpp` or auto): one or more llama.cpp servers (`LLM_LLAMACPP_BASE_URLS=http://10.11.10.43:14001,http://10.11.10.18:14001`) load-balanced round-robin with periodic health checks (`LLM_LLAMACPP_HEALTH_CHECK_INTERVAL`). See `src/llm_inference/backends/llamacpp_backend.py` (`LlamaCppBackend`, `LlamaCppServer` dataclass).
|
|
- **Cloud / LiteLLM** (default): `model` is passed as-is to LiteLLM. Examples:
|
|
- `openrouter/google/gemini-2.5-flash`, `openrouter/anthropic/claude-sonnet-4-6`, `openrouter/meta-llama/llama-3-70b` -> OpenRouter API (`OPENROUTER_API_KEY`)
|
|
- `gpt-4o`, `gpt-4-turbo`, `gpt-3.5-turbo` -> OpenAI (`OPENAI_API_KEY`)
|
|
- `claude-3-opus`, `claude-3-sonnet`, `claude-3-haiku` -> Anthropic (`ANTHROPIC_API_KEY`)
|
|
- LiteLLM also handles Azure / Bedrock / Gemini natively — fallback chains can be implemented at the caller level by retrying with the next model id.
|
|
|
|
There is no Groq backend in the current code; cloud routing is consolidated through LiteLLM.
|
|
|
|
## How didi-brain + agent-v3 use it
|
|
|
|
This service is the single point of LLM accounting for the platform:
|
|
|
|
- **agent-v3** (`backend/services/orchestration-layer/agent-v3`): the `LLMClient.call()` helper inside `src/components/component-runner.ts` and the per-component executors (techniques, ai-tampered, claims, pipeline) call `${LLM_ROUTER_URL}/v1/chat/completions` for every analysis prompt. The `model` field is taken from the moderation config (Redis-synced from `didiFramework`) so admins can swap models without redeploying agent-v3.
|
|
- **didi-brain** (`ai_platform/modules/didi_brain/brain_api`): the claim extractor and verification pipeline use the same router for both extraction prompts and rationale generation.
|
|
- **All LLM traffic** from DIDI flows through this router — so any cost-tracking middleware added here gives a unified billing view.
|
|
|
|
## Structura fisiere
|
|
|
|
```
|
|
src/llm_inference/
|
|
__init__.py
|
|
cli.py # `llm-inference` console script (uvicorn launcher + graceful shutdown)
|
|
client.py # LLMClient (high-level Python API used as library)
|
|
config.py # LLMSettings (Pydantic Settings, LLM_* env vars) + SettingsCache
|
|
schemas.py # OpenAI-shaped CompletionResponse / CompletionChunk request/response models
|
|
types.py # ChatMessage, BackendType enum, ModelInfo, Choice, Delta, Usage
|
|
exceptions.py # CompletionError, LLMTimeoutError, LLMRateLimitError, LLMConnectionError, ModelLoadError
|
|
retry.py # retry_with_backoff helper
|
|
image_processing.py # process_messages — fetches/encodes image content for multimodal calls
|
|
logging.py # configure_logging, get_logger (JSON + plain modes)
|
|
utils.py # safe_close_stream, etc.
|
|
backends/
|
|
base.py # Abstract LLMBackend (complete / stream / list_models / health_check / load_model / unload_model)
|
|
registry.py # BackendRegistry — instantiates enabled backends from settings
|
|
litellm_backend.py # LiteLLMBackend (default; OpenRouter / OpenAI / Anthropic / 100+ via litellm)
|
|
vllm_backend.py # vLLMBackend (OpenAI SDK -> internal vLLM server)
|
|
llamacpp_backend.py # LlamaCppBackend (round-robin pool with health checks + failover)
|
|
api/
|
|
app.py # create_app() — FastAPI factory, lifespan, middleware wiring
|
|
dependencies.py # init_concurrency_limiter, get_client, auth dependency
|
|
middleware.py # RateLimitMiddleware (token bucket), RequestIdMiddleware
|
|
routes/
|
|
completions.py # POST /v1/chat/completions
|
|
models.py # /v1/models, /v1/models/load, /v1/models/unload, /v1/backends
|
|
health.py # /health, /ready
|
|
info.py # /v1/info (catalog metadata)
|
|
deploy/
|
|
Dockerfile # Builds `didiai-llm-api` image
|
|
docker-compose.yml # Profiles: api, vllm, llamacpp, full
|
|
deploy.sh # Profile launcher
|
|
nginx.conf # Optional reverse proxy
|
|
tests/ # pytest (asyncio_mode=auto, markers: e2e, slow)
|
|
API.md # OpenAPI-style endpoint docs
|
|
README.md # Usage / install / deployment
|
|
.env.example # Full env reference
|
|
pyproject.toml # uv/hatchling project (llm-inference v0.1.0)
|
|
```
|
|
|
|
## Configuration
|
|
|
|
All env vars use the `LLM_` prefix (Pydantic Settings, `extra="forbid"` so typos fail loudly). Required (no default):
|
|
|
|
- `LLM_DEFAULT_BACKEND` — `litellm` | `vllm` | `llamacpp`
|
|
- `LLM_ENABLE_VLLM` — `true` / `false`
|
|
- `LLM_ENABLE_LLAMACPP` — `true` / `false`
|
|
- `LLM_EXTERNAL_URL` — public URL (used in OpenAPI `servers[]` and `/v1/info`)
|
|
|
|
Common optional:
|
|
|
|
- `OPENROUTER_API_KEY`, `OPENAI_API_KEY`, `ANTHROPIC_API_KEY` — provider keys for LiteLLM
|
|
- `LLM_DEFAULT_MODEL` (default `qwen3.5`) — alias used when a request omits `model`
|
|
- `LLM_MODEL_ALIASES` (default `{}`) — JSON map of friendly alias → real served id, e.g. `{"qwen3.5":"Qwen/Qwen3.5-35B-A3B"}`
|
|
- `LLM_ENABLE_FALLBACK` (default `true`), `LLM_FALLBACK_ORDER` (default `[vllm,llamacpp,litellm]`) — cross-backend fallback cascade
|
|
- `LLM_HOST` (default `0.0.0.0`), `LLM_PORT` (default `14011`)
|
|
- `LLM_VLLM_BASE_URL` (default `http://localhost:14001`), `LLM_VLLM_API_KEY`
|
|
- `LLM_VLLM_DISABLE_THINKING` (default `true`) — injects `chat_template_kwargs={'enable_thinking': false}` on vLLM chat requests so the Qwen3.5 reasoning model returns the final answer directly (no `thinking` preamble); key behavior for JSON-parsing callers. Callers may override per request.
|
|
- `LLM_LLAMACPP_BASE_URL` (single-server legacy) **or** `LLM_LLAMACPP_BASE_URLS` (comma-separated pool, overrides single)
|
|
- `LLM_LLAMACPP_HEALTH_CHECK_INTERVAL` (default `30s`)
|
|
- `LLM_REQUEST_TIMEOUT` (default `120s`), `LLM_CONNECT_TIMEOUT` (default `10s`)
|
|
- `LLM_MAX_RETRIES` (default `3`), `LLM_RETRY_MIN_WAIT`, `LLM_RETRY_MAX_WAIT`
|
|
- `LLM_RATE_LIMIT_RPS` (default `10`), `LLM_RATE_LIMIT_BURST` (default `20`), `LLM_MAX_CONCURRENT_COMPLETIONS` (default `10`)
|
|
- `LLM_API_TOKENS` — comma-separated; if set, `/v1/*` requires `Authorization: Bearer <token>`
|
|
- `LLM_LOG_LEVEL`, `LLM_LOG_JSON`
|
|
|
|
Full reference: `.env.example`. Settings class: `src/llm_inference/config.py:LLMSettings`.
|
|
|
|
## Deployment
|
|
|
|
GPU host `10.11.10.17` (typically). The vLLM container needs an NVIDIA H200 NVL (~143 GB VRAM); Qwen 3.5-35B-A3B occupies ~57 GB BF16 with `--gpu-memory-utilization 0.65` and `--max-model-len 32000`, leaving room on GPU 0 for Whisper.
|
|
|
|
Compose profiles (in `deploy/docker-compose.yml`):
|
|
|
|
- `api` — only the FastAPI router (uses external/cloud providers)
|
|
- `vllm` — router + vLLM Qwen 3.5 (GPU)
|
|
- `llamacpp` — router + a local llama.cpp server (CPU)
|
|
- `full` — everything
|
|
|
|
Networking: shares the external Docker network `didi-network` with sibling AI modules (whisper, embeddings, etc.). Internal address used by the router for vLLM is `http://didiAI-vllm-qwen3.5:14001`.
|
|
|
|
```bash
|
|
cd deploy/
|
|
cp ../.env.example .env # fill keys + URLs
|
|
./deploy.sh --profile api -d
|
|
./deploy.sh --profile vllm -d
|
|
./deploy.sh --profile api --logs
|
|
./deploy.sh --profile api --down
|
|
```
|
|
|
|
Healthcheck for the API container: `python -c "import urllib.request; urllib.request.urlopen('http://localhost:14011/health')"` every 30s.
|
|
|
|
## Cost tracking
|
|
|
|
The current code base does not yet ship a per-request cost-accounting middleware. The wiring point exists (`api/middleware.py` and the `RequestIdMiddleware` propagates a `X-Request-ID` to every call) and prompt/completion token counts are returned in `Usage` (`types.py`) on every non-streaming completion and in the final stream chunk. To plug into the AI dashboard, intercept at `routes/completions.py` (or a new middleware) and persist `(request_id, backend, model, prompt_tokens, completion_tokens, latency_ms)` — the dashboard's `cost.html` consumes that schema.
|
|
|
|
## Related
|
|
|
|
- **agent-v3 client**: `backend/services/orchestration-layer/agent-v3/src/components/component-runner.ts` — `LLMClient.call()` posts to `${LLM_ROUTER_URL}/v1/chat/completions`. Model id comes from Redis moderation config synced by `didiFramework`.
|
|
- **didi-brain**: `ai_platform/modules/didi_brain/brain_api` — extractor + verification calls hit the same router.
|
|
- **AI dashboard**: `cost.html` (when accounting middleware is added) consumes per-request token usage.
|
|
- **Sibling AI modules** (sharing `didi-network` network): whisper / transcription, embeddings, vision, etc., living under `ai_platform/modules/`.
|
|
- **OpenAI SDK compatibility**: clients can simply do `OpenAI(base_url="http://10.11.10.17:14011/v1", api_key=<token-or-not-needed>)`.
|