clossers/didi-lot1-ai
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

LOT 1 - Optimizare script build -Instalare mono comanda

Browse source
This commit is contained in:
Dezvoltari Evotech 2026-06-27 06:42:02 -07:00
parent 5380c3fc63
commit 42ff22bf85
127 changed files with 16163 additions and 532 deletions
Download patch file Download diff file Expand all files Collapse all files

13
ai_platform/modules/llm-inference/INDEX.md
View file

@ -1,18 +1,19 @@
# llm-inference
OpenAI-compatible LLM router for DIDI. Routes requests to local Qwen 3.5 397B (llama.cpp / vLLM backends) for free tier and proxies to OpenRouter cloud (Claude Sonnet 4.6, Gemini Flash 3, GPT-4o, etc.) for premium models. All endpoints are OpenAI-compatible (`/v1/chat/completions`), so callers can use the OpenAI SDK or plain `httpx` interchangeably.
OpenAI-compatible LLM router for DIDI. Routes requests to the local `Qwen/Qwen3.5-35B-A3B` model (served as `qwen3.5`, via vLLM / llama.cpp backends) for the free tier and proxies to OpenRouter cloud (Claude Sonnet 4.6, Gemini Flash 3, GPT-4o, etc.) for premium models. All endpoints are OpenAI-compatible (`/v1/chat/completions`), so callers can use the OpenAI SDK or plain `httpx` interchangeably.
- **Stack**: Python 3.10+, FastAPI, uvicorn, httpx, LiteLLM, Pydantic v2, sse-starlette
- **URL**: `http://10.11.10.17:14011` (LLM router on GPU host) — referenced as `LLM_ROUTER_URL` in DIDI services
- **Container**: `didiAI-llm-api` (image `didiai-llm-api`), runs on GPU host (typically `10.11.10.17`)
- **Local model**: Qwen 3.5-35B-A3B (MoE, native multimodal text+vision) via vLLM (`didiAI-vllm-qwen3.5`, internal port 14001) or Qwen 3.5 397B-A17B variant via llama.cpp pool
- **Local model**: `Qwen/Qwen3.5-35B-A3B` (MoE **reasoning** model; thinking can be toggled on/off — the gateway disables it by default, see `LLM_VLLM_DISABLE_THINKING`) served as `qwen3.5` via vLLM (`didiAI-vllm-qwen3.5`, internal port 14001) or via the llama.cpp pool
- **Entry point**: `llm-inference` console script -> `src/llm_inference/cli.py:main` -> uvicorn factory `llm_inference.api.app:create_app`
## Ce face
Single OpenAI-compatible endpoint (`/v1/chat/completions`) that selects a backend based on the request's `model` field and request-time `backend` override:
- **Auto backend resolution** (`LLMClient._resolve_backend_for_model` in `src/llm_inference/client.py`): probes each enabled local backend's `list_models()`. If the requested model is served locally, route to that backend; otherwise fall back to the configured default (litellm).
- **Auto backend resolution** (`LLMClient._resolve_backend_for_model` in `src/llm_inference/client.py`): the requested `model` is first normalized through `LLM_MODEL_ALIASES` (e.g. `qwen3.5` → `Qwen/Qwen3.5-35B-A3B`), then each enabled local backend's `list_models()` is probed. If the model is served locally, route there; otherwise fall back to `LLM_DEFAULT_BACKEND`.
- **Cross-backend fallback cascade** (`LLM_ENABLE_FALLBACK`, default `true`): on a backend failure the request is retried on the next enabled backend in `LLM_FALLBACK_ORDER` (default `[vllm, llamacpp, litellm]`). The cascade is skipped when the caller pins an explicit `backend`.
- **Explicit backend override**: clients may pass `"backend": "litellm" | "vllm" | "llamacpp"` in the JSON body to force routing.
- **Streaming + non-streaming**: same endpoint; `"stream": true` returns SSE chunks (`text/event-stream` with `data: [DONE]` terminator).
- **Multimodal**: messages are pre-processed by `image_processing.process_messages` so image URLs/base64 attachments work the same across backends.
@ -27,6 +28,7 @@ All paths are mounted by `src/llm_inference/api/app.py`:
| Method | Path | Purpose |
|--------|------|---------|
| POST | `/v1/chat/completions` | OpenAI-compat chat completion (streaming + non-streaming) |
| POST | `/v1/completions` | OpenAI-compat legacy text completion (non-streaming; 501 from backends that don't support it) |
| GET | `/v1/models` | List models from all (or specific via `?backend=`) backends |
| POST | `/v1/models/load` | Load a model on a local backend (vLLM / llama.cpp) |
| POST | `/v1/models/unload` | Unload a model from a local backend |
@ -113,9 +115,12 @@ All env vars use the `LLM_` prefix (Pydantic Settings, `extra="forbid"` so typos
Common optional:
- `OPENROUTER_API_KEY`, `OPENAI_API_KEY`, `ANTHROPIC_API_KEY` — provider keys for LiteLLM
- `LLM_DEFAULT_MODEL` (default `gpt-3.5-turbo`)
- `LLM_DEFAULT_MODEL` (default `qwen3.5`) — alias used when a request omits `model`
- `LLM_MODEL_ALIASES` (default `{}`) — JSON map of friendly alias → real served id, e.g. `{"qwen3.5":"Qwen/Qwen3.5-35B-A3B"}`
- `LLM_ENABLE_FALLBACK` (default `true`), `LLM_FALLBACK_ORDER` (default `[vllm,llamacpp,litellm]`) — cross-backend fallback cascade
- `LLM_HOST` (default `0.0.0.0`), `LLM_PORT` (default `14011`)
- `LLM_VLLM_BASE_URL` (default `http://localhost:14001`), `LLM_VLLM_API_KEY`
- `LLM_VLLM_DISABLE_THINKING` (default `true`) — injects `chat_template_kwargs={'enable_thinking': false}` on vLLM chat requests so the Qwen3.5 reasoning model returns the final answer directly (no `thinking` preamble); key behavior for JSON-parsing callers. Callers may override per request.
- `LLM_LLAMACPP_BASE_URL` (single-server legacy) **or** `LLM_LLAMACPP_BASE_URLS` (comma-separated pool, overrides single)
- `LLM_LLAMACPP_HEALTH_CHECK_INTERVAL` (default `30s`)
- `LLM_REQUEST_TIMEOUT` (default `120s`), `LLM_CONNECT_TIMEOUT` (default `10s`)