clossers/didi-lot1-ai
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
didi-lot1-ai/ai_platform/modules/llm-inference/INDEX.md

13 KiB
Raw Permalink Blame History

llm-inference

OpenAI-compatible LLM router for DIDI. Routes requests to the local Qwen/Qwen3.5-35B-A3B model (served as qwen3.5, via vLLM / llama.cpp backends) for the free tier and proxies to OpenRouter cloud (Claude Sonnet 4.6, Gemini Flash 3, GPT-4o, etc.) for premium models. All endpoints are OpenAI-compatible (/v1/chat/completions), so callers can use the OpenAI SDK or plain httpx interchangeably.

  • Stack: Python 3.10+, FastAPI, uvicorn, httpx, LiteLLM, Pydantic v2, sse-starlette
  • URL: http://10.11.10.17:14011 (LLM router on GPU host) — referenced as LLM_ROUTER_URL in DIDI services
  • Container: didiAI-llm-api (image didiai-llm-api), runs on GPU host (typically 10.11.10.17)
  • Local model: Qwen/Qwen3.5-35B-A3B (MoE reasoning model; thinking can be toggled on/off — the gateway disables it by default, see LLM_VLLM_DISABLE_THINKING) served as qwen3.5 via vLLM (didiAI-vllm-qwen3.5, internal port 14001) or via the llama.cpp pool
  • Entry point: llm-inference console script -> src/llm_inference/cli.py:main -> uvicorn factory llm_inference.api.app:create_app

Ce face

Single OpenAI-compatible endpoint (/v1/chat/completions) that selects a backend based on the request's model field and request-time backend override:

  • Auto backend resolution (LLMClient._resolve_backend_for_model in src/llm_inference/client.py): the requested model is first normalized through LLM_MODEL_ALIASES (e.g. qwen3.5 → Qwen/Qwen3.5-35B-A3B), then each enabled local backend's list_models() is probed. If the model is served locally, route there; otherwise fall back to LLM_DEFAULT_BACKEND.
  • Cross-backend fallback cascade (LLM_ENABLE_FALLBACK, default true): on a backend failure the request is retried on the next enabled backend in LLM_FALLBACK_ORDER (default [vllm, llamacpp, litellm]). The cascade is skipped when the caller pins an explicit backend.
  • Explicit backend override: clients may pass "backend": "litellm" | "vllm" | "llamacpp" in the JSON body to force routing.
  • Streaming + non-streaming: same endpoint; "stream": true returns SSE chunks (text/event-stream with data: [DONE] terminator).
  • Multimodal: messages are pre-processed by image_processing.process_messages so image URLs/base64 attachments work the same across backends.
  • Retries: transient failures (LLMTimeoutError, LLMRateLimitError, LLMConnectionError) wrapped by retry.retry_with_backoff with exponential backoff (max_retries, retry_min_wait, retry_max_wait).
  • Provider keys: read from env (OPENROUTER_API_KEY, OPENAI_API_KEY, ANTHROPIC_API_KEY); LiteLLM dispatches to OpenRouter / OpenAI / Anthropic / Gemini / Bedrock / 100+ providers using its own model-id parsing (openrouter/..., claude-3-..., gpt-4o, etc.).
  • Optional Bearer auth: when LLM_API_TOKENS is set, all /v1/* endpoints require Authorization: Bearer <token>; /health and /ready are always public.

API endpoints

All paths are mounted by src/llm_inference/api/app.py:

Method Path Purpose
POST /v1/chat/completions OpenAI-compat chat completion (streaming + non-streaming)
POST /v1/completions OpenAI-compat legacy text completion (non-streaming; 501 from backends that don't support it)
GET /v1/models List models from all (or specific via ?backend=) backends
POST /v1/models/load Load a model on a local backend (vLLM / llama.cpp)
POST /v1/models/unload Unload a model from a local backend
GET /v1/backends List enabled backends
GET /health Per-backend health (overall: healthy / degraded / unhealthy)
GET /ready K8s-style readiness probe (200 if default backend healthy, 503 otherwise)
GET /v1/info (catalog router) Service metadata for cross-module catalog integration

Source files for the routers: src/llm_inference/api/routes/{completions,models,health,info}.py.

Model routing logic

The backend dispatch lives in LLMClient._resolve_backend_for_model (auto) and BackendRegistry.get (explicit). Model identifier conventions:

  • Local vLLM (backend=vllm or auto): the vLLM container exposes Qwen with --served-model-name qwen3.5. Requests with model="qwen3.5" (or whatever Hugging Face id is loaded) are forwarded to LLM_VLLM_BASE_URL (default http://didiAI-vllm-qwen3.5:14001) using OpenAI SDK.
  • Local llama.cpp (backend=llamacpp or auto): one or more llama.cpp servers (LLM_LLAMACPP_BASE_URLS=http://10.11.10.43:14001,http://10.11.10.18:14001) load-balanced round-robin with periodic health checks (LLM_LLAMACPP_HEALTH_CHECK_INTERVAL). See src/llm_inference/backends/llamacpp_backend.py (LlamaCppBackend, LlamaCppServer dataclass).
  • Cloud / LiteLLM (default): model is passed as-is to LiteLLM. Examples:
    • openrouter/google/gemini-2.5-flash, openrouter/anthropic/claude-sonnet-4-6, openrouter/meta-llama/llama-3-70b -> OpenRouter API (OPENROUTER_API_KEY)
    • gpt-4o, gpt-4-turbo, gpt-3.5-turbo -> OpenAI (OPENAI_API_KEY)
    • claude-3-opus, claude-3-sonnet, claude-3-haiku -> Anthropic (ANTHROPIC_API_KEY)
    • LiteLLM also handles Azure / Bedrock / Gemini natively — fallback chains can be implemented at the caller level by retrying with the next model id.

There is no Groq backend in the current code; cloud routing is consolidated through LiteLLM.

How didi-brain + agent-v3 use it

This service is the single point of LLM accounting for the platform:

  • agent-v3 (backend/services/orchestration-layer/agent-v3): the LLMClient.call() helper inside src/components/component-runner.ts and the per-component executors (techniques, ai-tampered, claims, pipeline) call ${LLM_ROUTER_URL}/v1/chat/completions for every analysis prompt. The model field is taken from the moderation config (Redis-synced from didiFramework) so admins can swap models without redeploying agent-v3.
  • didi-brain (ai_platform/modules/didi_brain/brain_api): the claim extractor and verification pipeline use the same router for both extraction prompts and rationale generation.
  • All LLM traffic from DIDI flows through this router — so any cost-tracking middleware added here gives a unified billing view.

Structura fisiere

src/llm_inference/
  __init__.py
  cli.py                 # `llm-inference` console script (uvicorn launcher + graceful shutdown)
  client.py              # LLMClient (high-level Python API used as library)
  config.py              # LLMSettings (Pydantic Settings, LLM_* env vars) + SettingsCache
  schemas.py             # OpenAI-shaped CompletionResponse / CompletionChunk request/response models
  types.py               # ChatMessage, BackendType enum, ModelInfo, Choice, Delta, Usage
  exceptions.py          # CompletionError, LLMTimeoutError, LLMRateLimitError, LLMConnectionError, ModelLoadError
  retry.py               # retry_with_backoff helper
  image_processing.py    # process_messages — fetches/encodes image content for multimodal calls
  logging.py             # configure_logging, get_logger (JSON + plain modes)
  utils.py               # safe_close_stream, etc.
  backends/
    base.py              # Abstract LLMBackend (complete / stream / list_models / health_check / load_model / unload_model)
    registry.py          # BackendRegistry — instantiates enabled backends from settings
    litellm_backend.py   # LiteLLMBackend (default; OpenRouter / OpenAI / Anthropic / 100+ via litellm)
    vllm_backend.py      # vLLMBackend (OpenAI SDK -> internal vLLM server)
    llamacpp_backend.py  # LlamaCppBackend (round-robin pool with health checks + failover)
  api/
    app.py               # create_app() — FastAPI factory, lifespan, middleware wiring
    dependencies.py      # init_concurrency_limiter, get_client, auth dependency
    middleware.py        # RateLimitMiddleware (token bucket), RequestIdMiddleware
    routes/
      completions.py     # POST /v1/chat/completions
      models.py          # /v1/models, /v1/models/load, /v1/models/unload, /v1/backends
      health.py          # /health, /ready
      info.py            # /v1/info (catalog metadata)
deploy/
  Dockerfile             # Builds `didiai-llm-api` image
  docker-compose.yml     # Profiles: api, vllm, llamacpp, full
  deploy.sh              # Profile launcher
  nginx.conf             # Optional reverse proxy
tests/                   # pytest (asyncio_mode=auto, markers: e2e, slow)
API.md                   # OpenAPI-style endpoint docs
README.md                # Usage / install / deployment
.env.example             # Full env reference
pyproject.toml           # uv/hatchling project (llm-inference v0.1.0)

Configuration

All env vars use the LLM_ prefix (Pydantic Settings, extra="forbid" so typos fail loudly). Required (no default):

  • LLM_DEFAULT_BACKEND — litellm | vllm | llamacpp
  • LLM_ENABLE_VLLM — true / false
  • LLM_ENABLE_LLAMACPP — true / false
  • LLM_EXTERNAL_URL — public URL (used in OpenAPI servers[] and /v1/info)

Common optional:

  • OPENROUTER_API_KEY, OPENAI_API_KEY, ANTHROPIC_API_KEY — provider keys for LiteLLM
  • LLM_DEFAULT_MODEL (default qwen3.5) — alias used when a request omits model
  • LLM_MODEL_ALIASES (default {}) — JSON map of friendly alias → real served id, e.g. {"qwen3.5":"Qwen/Qwen3.5-35B-A3B"}
  • LLM_ENABLE_FALLBACK (default true), LLM_FALLBACK_ORDER (default [vllm,llamacpp,litellm]) — cross-backend fallback cascade
  • LLM_HOST (default 0.0.0.0), LLM_PORT (default 14011)
  • LLM_VLLM_BASE_URL (default http://localhost:14001), LLM_VLLM_API_KEY
  • LLM_VLLM_DISABLE_THINKING (default true) — injects chat_template_kwargs={'enable_thinking': false} on vLLM chat requests so the Qwen3.5 reasoning model returns the final answer directly (no thinking preamble); key behavior for JSON-parsing callers. Callers may override per request.
  • LLM_LLAMACPP_BASE_URL (single-server legacy) or LLM_LLAMACPP_BASE_URLS (comma-separated pool, overrides single)
  • LLM_LLAMACPP_HEALTH_CHECK_INTERVAL (default 30s)
  • LLM_REQUEST_TIMEOUT (default 120s), LLM_CONNECT_TIMEOUT (default 10s)
  • LLM_MAX_RETRIES (default 3), LLM_RETRY_MIN_WAIT, LLM_RETRY_MAX_WAIT
  • LLM_RATE_LIMIT_RPS (default 10), LLM_RATE_LIMIT_BURST (default 20), LLM_MAX_CONCURRENT_COMPLETIONS (default 10)
  • LLM_API_TOKENS — comma-separated; if set, /v1/* requires Authorization: Bearer <token>
  • LLM_LOG_LEVEL, LLM_LOG_JSON

Full reference: .env.example. Settings class: src/llm_inference/config.py:LLMSettings.

Deployment

GPU host 10.11.10.17 (typically). The vLLM container needs an NVIDIA H200 NVL (~143 GB VRAM); Qwen 3.5-35B-A3B occupies ~57 GB BF16 with --gpu-memory-utilization 0.65 and --max-model-len 32000, leaving room on GPU 0 for Whisper.

Compose profiles (in deploy/docker-compose.yml):

  • api — only the FastAPI router (uses external/cloud providers)
  • vllm — router + vLLM Qwen 3.5 (GPU)
  • llamacpp — router + a local llama.cpp server (CPU)
  • full — everything

Networking: shares the external Docker network didi-network with sibling AI modules (whisper, embeddings, etc.). Internal address used by the router for vLLM is http://didiAI-vllm-qwen3.5:14001.

cd deploy/
cp ../.env.example .env       # fill keys + URLs
./deploy.sh --profile api -d
./deploy.sh --profile vllm -d
./deploy.sh --profile api --logs
./deploy.sh --profile api --down

Healthcheck for the API container: python -c "import urllib.request; urllib.request.urlopen('http://localhost:14011/health')" every 30s.

Cost tracking

The current code base does not yet ship a per-request cost-accounting middleware. The wiring point exists (api/middleware.py and the RequestIdMiddleware propagates a X-Request-ID to every call) and prompt/completion token counts are returned in Usage (types.py) on every non-streaming completion and in the final stream chunk. To plug into the AI dashboard, intercept at routes/completions.py (or a new middleware) and persist (request_id, backend, model, prompt_tokens, completion_tokens, latency_ms) — the dashboard's cost.html consumes that schema.

Related

  • agent-v3 client: backend/services/orchestration-layer/agent-v3/src/components/component-runner.ts — LLMClient.call() posts to ${LLM_ROUTER_URL}/v1/chat/completions. Model id comes from Redis moderation config synced by didiFramework.
  • didi-brain: ai_platform/modules/didi_brain/brain_api — extractor + verification calls hit the same router.
  • AI dashboard: cost.html (when accounting middleware is added) consumes per-request token usage.
  • Sibling AI modules (sharing didi-network network): whisper / transcription, embeddings, vision, etc., living under ai_platform/modules/.
  • OpenAI SDK compatibility: clients can simply do OpenAI(base_url="http://10.11.10.17:14011/v1", api_key=<token-or-not-needed>).