clossers/didi-lot1-ai
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Livrare LOT 1 - Didi

Browse source
This commit is contained in:
Dezvoltari Evotech 2026-06-25 14:13:25 -07:00
commit 5380c3fc63
990 changed files with 133308 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

163
ai_platform/modules/llm-inference/INDEX.md Normal file
View file

@ -0,0 +1,163 @@
# llm-inference
OpenAI-compatible LLM router for DIDI. Routes requests to local Qwen 3.5 397B (llama.cpp / vLLM backends) for free tier and proxies to OpenRouter cloud (Claude Sonnet 4.6, Gemini Flash 3, GPT-4o, etc.) for premium models. All endpoints are OpenAI-compatible (`/v1/chat/completions`), so callers can use the OpenAI SDK or plain `httpx` interchangeably.
- **Stack**: Python 3.10+, FastAPI, uvicorn, httpx, LiteLLM, Pydantic v2, sse-starlette
- **URL**: `http://10.11.10.17:14011` (LLM router on GPU host) — referenced as `LLM_ROUTER_URL` in DIDI services
- **Container**: `didiAI-llm-api` (image `didiai-llm-api`), runs on GPU host (typically `10.11.10.17`)
- **Local model**: Qwen 3.5-35B-A3B (MoE, native multimodal text+vision) via vLLM (`didiAI-vllm-qwen3.5`, internal port 14001) or Qwen 3.5 397B-A17B variant via llama.cpp pool
- **Entry point**: `llm-inference` console script -> `src/llm_inference/cli.py:main` -> uvicorn factory `llm_inference.api.app:create_app`
## Ce face
Single OpenAI-compatible endpoint (`/v1/chat/completions`) that selects a backend based on the request's `model` field and request-time `backend` override:
- **Auto backend resolution** (`LLMClient._resolve_backend_for_model` in `src/llm_inference/client.py`): probes each enabled local backend's `list_models()`. If the requested model is served locally, route to that backend; otherwise fall back to the configured default (litellm).
- **Explicit backend override**: clients may pass `"backend": "litellm" | "vllm" | "llamacpp"` in the JSON body to force routing.
- **Streaming + non-streaming**: same endpoint; `"stream": true` returns SSE chunks (`text/event-stream` with `data: [DONE]` terminator).
- **Multimodal**: messages are pre-processed by `image_processing.process_messages` so image URLs/base64 attachments work the same across backends.
- **Retries**: transient failures (`LLMTimeoutError`, `LLMRateLimitError`, `LLMConnectionError`) wrapped by `retry.retry_with_backoff` with exponential backoff (`max_retries`, `retry_min_wait`, `retry_max_wait`).
- **Provider keys**: read from env (`OPENROUTER_API_KEY`, `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`); LiteLLM dispatches to OpenRouter / OpenAI / Anthropic / Gemini / Bedrock / 100+ providers using its own model-id parsing (`openrouter/...`, `claude-3-...`, `gpt-4o`, etc.).
- **Optional Bearer auth**: when `LLM_API_TOKENS` is set, all `/v1/*` endpoints require `Authorization: Bearer <token>`; `/health` and `/ready` are always public.
## API endpoints
All paths are mounted by `src/llm_inference/api/app.py`:
| Method | Path | Purpose |
|--------|------|---------|
| POST | `/v1/chat/completions` | OpenAI-compat chat completion (streaming + non-streaming) |
| GET | `/v1/models` | List models from all (or specific via `?backend=`) backends |
| POST | `/v1/models/load` | Load a model on a local backend (vLLM / llama.cpp) |
| POST | `/v1/models/unload` | Unload a model from a local backend |
| GET | `/v1/backends` | List enabled backends |
| GET | `/health` | Per-backend health (overall: `healthy` / `degraded` / `unhealthy`) |
| GET | `/ready` | K8s-style readiness probe (200 if default backend healthy, 503 otherwise) |
| GET | `/v1/info` *(catalog router)* | Service metadata for cross-module catalog integration |
Source files for the routers: `src/llm_inference/api/routes/{completions,models,health,info}.py`.
## Model routing logic
The backend dispatch lives in `LLMClient._resolve_backend_for_model` (auto) and `BackendRegistry.get` (explicit). Model identifier conventions:
- **Local vLLM** (`backend=vllm` or auto): the vLLM container exposes Qwen with `--served-model-name qwen3.5`. Requests with `model="qwen3.5"` (or whatever Hugging Face id is loaded) are forwarded to `LLM_VLLM_BASE_URL` (default `http://didiAI-vllm-qwen3.5:14001`) using OpenAI SDK.
- **Local llama.cpp** (`backend=llamacpp` or auto): one or more llama.cpp servers (`LLM_LLAMACPP_BASE_URLS=http://10.11.10.43:14001,http://10.11.10.18:14001`) load-balanced round-robin with periodic health checks (`LLM_LLAMACPP_HEALTH_CHECK_INTERVAL`). See `src/llm_inference/backends/llamacpp_backend.py` (`LlamaCppBackend`, `LlamaCppServer` dataclass).
- **Cloud / LiteLLM** (default): `model` is passed as-is to LiteLLM. Examples:
- `openrouter/google/gemini-2.5-flash`, `openrouter/anthropic/claude-sonnet-4-6`, `openrouter/meta-llama/llama-3-70b` -> OpenRouter API (`OPENROUTER_API_KEY`)
- `gpt-4o`, `gpt-4-turbo`, `gpt-3.5-turbo` -> OpenAI (`OPENAI_API_KEY`)
- `claude-3-opus`, `claude-3-sonnet`, `claude-3-haiku` -> Anthropic (`ANTHROPIC_API_KEY`)
- LiteLLM also handles Azure / Bedrock / Gemini natively — fallback chains can be implemented at the caller level by retrying with the next model id.
There is no Groq backend in the current code; cloud routing is consolidated through LiteLLM.
## How didi-brain + agent-v3 use it
This service is the single point of LLM accounting for the platform:
- **agent-v3** (`backend/services/orchestration-layer/agent-v3`): the `LLMClient.call()` helper inside `src/components/component-runner.ts` and the per-component executors (techniques, ai-tampered, claims, pipeline) call `${LLM_ROUTER_URL}/v1/chat/completions` for every analysis prompt. The `model` field is taken from the moderation config (Redis-synced from `didiFramework`) so admins can swap models without redeploying agent-v3.
- **didi-brain** (`ai_platform/modules/didi_brain/brain_api`): the claim extractor and verification pipeline use the same router for both extraction prompts and rationale generation.
- **All LLM traffic** from DIDI flows through this router — so any cost-tracking middleware added here gives a unified billing view.
## Structura fisiere
```
src/llm_inference/
__init__.py
cli.py # `llm-inference` console script (uvicorn launcher + graceful shutdown)
client.py # LLMClient (high-level Python API used as library)
config.py # LLMSettings (Pydantic Settings, LLM_* env vars) + SettingsCache
schemas.py # OpenAI-shaped CompletionResponse / CompletionChunk request/response models
types.py # ChatMessage, BackendType enum, ModelInfo, Choice, Delta, Usage
exceptions.py # CompletionError, LLMTimeoutError, LLMRateLimitError, LLMConnectionError, ModelLoadError
retry.py # retry_with_backoff helper
image_processing.py # process_messages — fetches/encodes image content for multimodal calls
logging.py # configure_logging, get_logger (JSON + plain modes)
utils.py # safe_close_stream, etc.
backends/
base.py # Abstract LLMBackend (complete / stream / list_models / health_check / load_model / unload_model)
registry.py # BackendRegistry — instantiates enabled backends from settings
litellm_backend.py # LiteLLMBackend (default; OpenRouter / OpenAI / Anthropic / 100+ via litellm)
vllm_backend.py # vLLMBackend (OpenAI SDK -> internal vLLM server)
llamacpp_backend.py # LlamaCppBackend (round-robin pool with health checks + failover)
api/
app.py # create_app() — FastAPI factory, lifespan, middleware wiring
dependencies.py # init_concurrency_limiter, get_client, auth dependency
middleware.py # RateLimitMiddleware (token bucket), RequestIdMiddleware
routes/
completions.py # POST /v1/chat/completions
models.py # /v1/models, /v1/models/load, /v1/models/unload, /v1/backends
health.py # /health, /ready
info.py # /v1/info (catalog metadata)
deploy/
Dockerfile # Builds `didiai-llm-api` image
docker-compose.yml # Profiles: api, vllm, llamacpp, full
deploy.sh # Profile launcher
nginx.conf # Optional reverse proxy
tests/ # pytest (asyncio_mode=auto, markers: e2e, slow)
API.md # OpenAPI-style endpoint docs
README.md # Usage / install / deployment
.env.example # Full env reference
pyproject.toml # uv/hatchling project (llm-inference v0.1.0)
```
## Configuration
All env vars use the `LLM_` prefix (Pydantic Settings, `extra="forbid"` so typos fail loudly). Required (no default):
- `LLM_DEFAULT_BACKEND` — `litellm` | `vllm` | `llamacpp`
- `LLM_ENABLE_VLLM` — `true` / `false`
- `LLM_ENABLE_LLAMACPP` — `true` / `false`
- `LLM_EXTERNAL_URL` — public URL (used in OpenAPI `servers[]` and `/v1/info`)
Common optional:
- `OPENROUTER_API_KEY`, `OPENAI_API_KEY`, `ANTHROPIC_API_KEY` — provider keys for LiteLLM
- `LLM_DEFAULT_MODEL` (default `gpt-3.5-turbo`)
- `LLM_HOST` (default `0.0.0.0`), `LLM_PORT` (default `14011`)
- `LLM_VLLM_BASE_URL` (default `http://localhost:14001`), `LLM_VLLM_API_KEY`
- `LLM_LLAMACPP_BASE_URL` (single-server legacy) **or** `LLM_LLAMACPP_BASE_URLS` (comma-separated pool, overrides single)
- `LLM_LLAMACPP_HEALTH_CHECK_INTERVAL` (default `30s`)
- `LLM_REQUEST_TIMEOUT` (default `120s`), `LLM_CONNECT_TIMEOUT` (default `10s`)
- `LLM_MAX_RETRIES` (default `3`), `LLM_RETRY_MIN_WAIT`, `LLM_RETRY_MAX_WAIT`
- `LLM_RATE_LIMIT_RPS` (default `10`), `LLM_RATE_LIMIT_BURST` (default `20`), `LLM_MAX_CONCURRENT_COMPLETIONS` (default `10`)
- `LLM_API_TOKENS` — comma-separated; if set, `/v1/*` requires `Authorization: Bearer <token>`
- `LLM_LOG_LEVEL`, `LLM_LOG_JSON`
Full reference: `.env.example`. Settings class: `src/llm_inference/config.py:LLMSettings`.
## Deployment
GPU host `10.11.10.17` (typically). The vLLM container needs an NVIDIA H200 NVL (~143 GB VRAM); Qwen 3.5-35B-A3B occupies ~57 GB BF16 with `--gpu-memory-utilization 0.65` and `--max-model-len 32000`, leaving room on GPU 0 for Whisper.
Compose profiles (in `deploy/docker-compose.yml`):
- `api` — only the FastAPI router (uses external/cloud providers)
- `vllm` — router + vLLM Qwen 3.5 (GPU)
- `llamacpp` — router + a local llama.cpp server (CPU)
- `full` — everything
Networking: shares the external Docker network `didi-network` with sibling AI modules (whisper, embeddings, etc.). Internal address used by the router for vLLM is `http://didiAI-vllm-qwen3.5:14001`.
```bash
cd deploy/
cp ../.env.example .env # fill keys + URLs
./deploy.sh --profile api -d
./deploy.sh --profile vllm -d
./deploy.sh --profile api --logs
./deploy.sh --profile api --down
```
Healthcheck for the API container: `python -c "import urllib.request; urllib.request.urlopen('http://localhost:14011/health')"` every 30s.
## Cost tracking
The current code base does not yet ship a per-request cost-accounting middleware. The wiring point exists (`api/middleware.py` and the `RequestIdMiddleware` propagates a `X-Request-ID` to every call) and prompt/completion token counts are returned in `Usage` (`types.py`) on every non-streaming completion and in the final stream chunk. To plug into the AI dashboard, intercept at `routes/completions.py` (or a new middleware) and persist `(request_id, backend, model, prompt_tokens, completion_tokens, latency_ms)` — the dashboard's `cost.html` consumes that schema.
## Related
- **agent-v3 client**: `backend/services/orchestration-layer/agent-v3/src/components/component-runner.ts` — `LLMClient.call()` posts to `${LLM_ROUTER_URL}/v1/chat/completions`. Model id comes from Redis moderation config synced by `didiFramework`.
- **didi-brain**: `ai_platform/modules/didi_brain/brain_api` — extractor + verification calls hit the same router.
- **AI dashboard**: `cost.html` (when accounting middleware is added) consumes per-request token usage.
- **Sibling AI modules** (sharing `didi-network` network): whisper / transcription, embeddings, vision, etc., living under `ai_platform/modules/`.
- **OpenAI SDK compatibility**: clients can simply do `OpenAI(base_url="http://10.11.10.17:14011/v1", api_key=<token-or-not-needed>)`.