clossers/didi-lot1-ai
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

LOT 1 - Optimizare script build -Instalare mono comanda

Browse source
This commit is contained in:
Dezvoltari Evotech 2026-06-27 06:42:02 -07:00
parent 5380c3fc63
commit 42ff22bf85
127 changed files with 16163 additions and 532 deletions
Download patch file Download diff file Expand all files Collapse all files

83
ai_platform/modules/llm-inference/API.md
View file

@ -36,6 +36,7 @@ Authorization: Bearer <your-token>
**Protected endpoints (require auth when enabled):**
- `POST /v1/chat/completions`
- `POST /v1/completions`
- `GET /v1/models`
- `POST /v1/models/load`
- `POST /v1/models/unload`
@ -59,6 +60,8 @@ Backend services require their own API keys:
- **LiteLLM**: `OPENROUTER_API_KEY`, `OPENAI_API_KEY`, or `ANTHROPIC_API_KEY`
- **vLLM**: Optional `LLM_VLLM_API_KEY` for vLLM server
> **Reasoning model note (`LLM_VLLM_DISABLE_THINKING`, default `true`):** the local model `Qwen/Qwen3.5-35B-A3B` (served as `qwen3.5`) is a reasoning model. By default the gateway injects `chat_template_kwargs={"enable_thinking": false}` on vLLM chat requests so the model returns the final answer directly instead of a `thinking` preamble — important for callers that parse JSON. Callers may override by passing their own `chat_template_kwargs` in the request body.
## Rate Limiting
The API uses token bucket rate limiting:
@ -193,6 +196,66 @@ curl -X POST http://localhost:14011/v1/chat/completions \
---
### Text Completions (legacy)
Create a non-streaming text completion from a raw prompt. OpenAI-compatible legacy `/v1/completions`. Routes to the model's backend (vLLM primary, cloud via LiteLLM). Backends that do not support text completion return `501`.
```
POST /v1/completions
```
#### Request Body
| Field | Type | Required | Default | Description |
|-------|------|----------|---------|-------------|
| `prompt` | string/array | Yes | - | Prompt(s) to complete |
| `model` | string | No | server default | Model (alias) to use |
| `temperature` | float | No | 0.7 | Sampling temperature (0.0-2.0) |
| `max_tokens` | integer | No | null | Maximum tokens to generate (1-1,000,000) |
| `backend` | string | No | null | Backend override: `litellm`, `vllm`, `llamacpp` |
| `top_p` | float | No | null | Top-p sampling (0.0-1.0) |
| `frequency_penalty` | float | No | null | Frequency penalty (-2.0 to 2.0) |
| `presence_penalty` | float | No | null | Presence penalty (-2.0 to 2.0) |
| `stop` | string/array | No | null | Stop sequences |
#### Response
```json
{
"id": "cmpl-abc123",
"object": "text_completion",
"created": 1704067200,
"model": "qwen3.5",
"choices": [
{
"index": 0,
"text": "Paris is the capital of France.",
"finish_reason": "stop"
}
],
"usage": {
"prompt_tokens": 8,
"completion_tokens": 7,
"total_tokens": 15
},
"backend": "vllm"
}
```
#### Example Request
```bash
curl -X POST http://localhost:14011/v1/completions \
-H "Content-Type: application/json" \
-d '{
"prompt": "The capital of France is",
"model": "qwen3.5",
"max_tokens": 16
}'
```
---
### List Models
List available models across all or specific backends.
@ -433,6 +496,22 @@ curl http://localhost:14011/ready
---
### Service Info (catalog)
Service metadata for cross-module catalog integration. Returns `resource`, `models` (from all enabled backends), and `functions` (API endpoint descriptors). Consumed by the catalog-api.
```
GET /v1/info
```
#### Example Request
```bash
curl http://localhost:14011/v1/info
```
---
## Error Responses
All errors follow a consistent format:
@ -507,9 +586,9 @@ Response includes `Retry-After` header with seconds to wait.
## Supported Backends
### LiteLLM (Default)
### LiteLLM (cloud)
Supports 100+ LLM providers through a unified interface.
Supports 100+ LLM providers through a unified interface. (`LLM_DEFAULT_BACKEND` is required and has no default; the DIDI deployment runs `vllm` with the local `qwen3.5` model and uses LiteLLM for cloud/premium models.)
**Popular models:**
- `gpt-3.5-turbo`, `gpt-4`, `gpt-4-turbo` (OpenAI)

13
ai_platform/modules/llm-inference/INDEX.md
View file

@ -1,18 +1,19 @@
# llm-inference
OpenAI-compatible LLM router for DIDI. Routes requests to local Qwen 3.5 397B (llama.cpp / vLLM backends) for free tier and proxies to OpenRouter cloud (Claude Sonnet 4.6, Gemini Flash 3, GPT-4o, etc.) for premium models. All endpoints are OpenAI-compatible (`/v1/chat/completions`), so callers can use the OpenAI SDK or plain `httpx` interchangeably.
OpenAI-compatible LLM router for DIDI. Routes requests to the local `Qwen/Qwen3.5-35B-A3B` model (served as `qwen3.5`, via vLLM / llama.cpp backends) for the free tier and proxies to OpenRouter cloud (Claude Sonnet 4.6, Gemini Flash 3, GPT-4o, etc.) for premium models. All endpoints are OpenAI-compatible (`/v1/chat/completions`), so callers can use the OpenAI SDK or plain `httpx` interchangeably.
- **Stack**: Python 3.10+, FastAPI, uvicorn, httpx, LiteLLM, Pydantic v2, sse-starlette
- **URL**: `http://10.11.10.17:14011` (LLM router on GPU host) — referenced as `LLM_ROUTER_URL` in DIDI services
- **Container**: `didiAI-llm-api` (image `didiai-llm-api`), runs on GPU host (typically `10.11.10.17`)
- **Local model**: Qwen 3.5-35B-A3B (MoE, native multimodal text+vision) via vLLM (`didiAI-vllm-qwen3.5`, internal port 14001) or Qwen 3.5 397B-A17B variant via llama.cpp pool
- **Local model**: `Qwen/Qwen3.5-35B-A3B` (MoE **reasoning** model; thinking can be toggled on/off — the gateway disables it by default, see `LLM_VLLM_DISABLE_THINKING`) served as `qwen3.5` via vLLM (`didiAI-vllm-qwen3.5`, internal port 14001) or via the llama.cpp pool
- **Entry point**: `llm-inference` console script -> `src/llm_inference/cli.py:main` -> uvicorn factory `llm_inference.api.app:create_app`
## Ce face
Single OpenAI-compatible endpoint (`/v1/chat/completions`) that selects a backend based on the request's `model` field and request-time `backend` override:
- **Auto backend resolution** (`LLMClient._resolve_backend_for_model` in `src/llm_inference/client.py`): probes each enabled local backend's `list_models()`. If the requested model is served locally, route to that backend; otherwise fall back to the configured default (litellm).
- **Auto backend resolution** (`LLMClient._resolve_backend_for_model` in `src/llm_inference/client.py`): the requested `model` is first normalized through `LLM_MODEL_ALIASES` (e.g. `qwen3.5` → `Qwen/Qwen3.5-35B-A3B`), then each enabled local backend's `list_models()` is probed. If the model is served locally, route there; otherwise fall back to `LLM_DEFAULT_BACKEND`.
- **Cross-backend fallback cascade** (`LLM_ENABLE_FALLBACK`, default `true`): on a backend failure the request is retried on the next enabled backend in `LLM_FALLBACK_ORDER` (default `[vllm, llamacpp, litellm]`). The cascade is skipped when the caller pins an explicit `backend`.
- **Explicit backend override**: clients may pass `"backend": "litellm" | "vllm" | "llamacpp"` in the JSON body to force routing.
- **Streaming + non-streaming**: same endpoint; `"stream": true` returns SSE chunks (`text/event-stream` with `data: [DONE]` terminator).
- **Multimodal**: messages are pre-processed by `image_processing.process_messages` so image URLs/base64 attachments work the same across backends.
@ -27,6 +28,7 @@ All paths are mounted by `src/llm_inference/api/app.py`:
| Method | Path | Purpose |
|--------|------|---------|
| POST | `/v1/chat/completions` | OpenAI-compat chat completion (streaming + non-streaming) |
| POST | `/v1/completions` | OpenAI-compat legacy text completion (non-streaming; 501 from backends that don't support it) |
| GET | `/v1/models` | List models from all (or specific via `?backend=`) backends |
| POST | `/v1/models/load` | Load a model on a local backend (vLLM / llama.cpp) |
| POST | `/v1/models/unload` | Unload a model from a local backend |
@ -113,9 +115,12 @@ All env vars use the `LLM_` prefix (Pydantic Settings, `extra="forbid"` so typos
Common optional:
- `OPENROUTER_API_KEY`, `OPENAI_API_KEY`, `ANTHROPIC_API_KEY` — provider keys for LiteLLM
- `LLM_DEFAULT_MODEL` (default `gpt-3.5-turbo`)
- `LLM_DEFAULT_MODEL` (default `qwen3.5`) — alias used when a request omits `model`
- `LLM_MODEL_ALIASES` (default `{}`) — JSON map of friendly alias → real served id, e.g. `{"qwen3.5":"Qwen/Qwen3.5-35B-A3B"}`
- `LLM_ENABLE_FALLBACK` (default `true`), `LLM_FALLBACK_ORDER` (default `[vllm,llamacpp,litellm]`) — cross-backend fallback cascade
- `LLM_HOST` (default `0.0.0.0`), `LLM_PORT` (default `14011`)
- `LLM_VLLM_BASE_URL` (default `http://localhost:14001`), `LLM_VLLM_API_KEY`
- `LLM_VLLM_DISABLE_THINKING` (default `true`) — injects `chat_template_kwargs={'enable_thinking': false}` on vLLM chat requests so the Qwen3.5 reasoning model returns the final answer directly (no `thinking` preamble); key behavior for JSON-parsing callers. Callers may override per request.
- `LLM_LLAMACPP_BASE_URL` (single-server legacy) **or** `LLM_LLAMACPP_BASE_URLS` (comma-separated pool, overrides single)
- `LLM_LLAMACPP_HEALTH_CHECK_INTERVAL` (default `30s`)
- `LLM_REQUEST_TIMEOUT` (default `120s`), `LLM_CONNECT_TIMEOUT` (default `10s`)

14
ai_platform/modules/llm-inference/README.md
View file

@ -91,10 +91,12 @@ cp ../.env.example .env
| Endpoint | Method | Description |
|----------|--------|-------------|
| `/v1/chat/completions` | POST | Chat completion (supports streaming) |
| `/v1/completions` | POST | Legacy text completion (non-streaming; 501 if backend unsupported) |
| `/v1/models` | GET | List available models |
| `/v1/models/load` | POST | Load a model (local backends) |
| `/v1/models/unload` | POST | Unload a model (local backends) |
| `/v1/backends` | GET | List available backends |
| `/v1/info` | GET | Service/catalog metadata |
| `/health` | GET | Health check |
| `/ready` | GET | Readiness probe |
@ -125,13 +127,17 @@ Configure via environment variables (prefix: `LLM_`):
| Variable | Default | Description |
|----------|---------|-------------|
| `LLM_DEFAULT_BACKEND` | `litellm` | Default backend (litellm, vllm, llamacpp) |
| `LLM_DEFAULT_MODEL` | `gpt-3.5-turbo` | Default model |
| `LLM_DEFAULT_BACKEND` | _required_ | Default backend (`litellm`, `vllm`, `llamacpp`) — no default, must be set; the DIDI deployment runs `vllm` |
| `LLM_DEFAULT_MODEL` | `qwen3.5` | Default model alias used when a request omits `model` |
| `LLM_MODEL_ALIASES` | `{}` | JSON map of alias → served id, e.g. `{"qwen3.5":"Qwen/Qwen3.5-35B-A3B"}` |
| `LLM_ENABLE_FALLBACK` | `true` | Cross-backend fallback cascade on backend failure |
| `LLM_FALLBACK_ORDER` | `[vllm,llamacpp,litellm]` | Backend order tried in the fallback cascade |
| `LLM_PORT` | `14011` | API server port |
| `LLM_HOST` | `0.0.0.0` | API server host |
| `LLM_ENABLE_VLLM` | `false` | Enable vLLM backend |
| `LLM_ENABLE_LLAMACPP` | `false` | Enable llama.cpp backend |
| `LLM_ENABLE_VLLM` | _required_ | Enable vLLM backend (no default) |
| `LLM_ENABLE_LLAMACPP` | _required_ | Enable llama.cpp backend (no default) |
| `LLM_VLLM_BASE_URL` | `http://localhost:14001` | vLLM server URL |
| `LLM_VLLM_DISABLE_THINKING` | `true` | Inject `enable_thinking=false` for the Qwen3.5 reasoning model so it returns the final answer directly (no thinking preamble) |
| `LLM_LLAMACPP_BASE_URL` | `http://localhost:8080` | llama.cpp server URL |
| `OPENROUTER_API_KEY` | - | OpenRouter API key |
| `OPENAI_API_KEY` | - | OpenAI API key |

6
ai_platform/modules/llm-inference/deploy/Dockerfile
View file

@ -8,12 +8,12 @@ COPY --from=ghcr.io/astral-sh/uv:0.10 /uv /usr/local/bin/uv
WORKDIR /app
# Copy project files
COPY pyproject.toml uv.lock README.md ./
# Copy project files (uv.lock not committed; resolved at build time)
COPY pyproject.toml README.md ./
COPY src/ ./src/
# Install dependencies
RUN uv sync --frozen --no-dev
RUN uv sync --no-dev
# Production image
FROM python:3.11.12-slim

8
ai_platform/modules/llm-inference/deploy/docker-compose.yml
View file

@ -18,10 +18,10 @@
# Naming Convention: didiAI-{module}-{service}
#
# Network:
# Uses deploy_default network (shared with other modules)
# Uses didi-network (shared with all DIDI + AI platform stacks)
networks:
deploy_default:
didi-network:
external: true
services:
@ -37,7 +37,7 @@ services:
ports:
- "14011:14011"
networks:
- deploy_default
- didi-network
environment:
- LLM_PORT=14011
- LLM_EXTERNAL_URL=${LLM_EXTERNAL_URL}
@ -72,7 +72,7 @@ services:
ports:
- "14001:14001"
networks:
- deploy_default
- didi-network
volumes:
- ${HF_CACHE_DIR}:/root/.cache/huggingface
environment:

18
ai_platform/modules/llm-inference/src/llm_inference/backends/vllm_backend.py
View file

@ -105,6 +105,21 @@ class VLLMBackend(LLMBackend):
"""Backend identifier name."""
return "vllm"
def _apply_thinking_default(self, kwargs: dict[str, object]) -> dict[str, object]:
"""Default reasoning models to non-thinking output for clean answers.
Injects ``extra_body={"chat_template_kwargs": {"enable_thinking": False}}``
unless the caller already supplied ``chat_template_kwargs``. Lets JSON-parsing
consumers (extractors, video semantic, brain) get the final answer directly.
"""
if not getattr(self._settings, "vllm_disable_thinking", True):
return kwargs
extra = dict(kwargs.get("extra_body") or {}) # type: ignore[arg-type]
ctk = dict(extra.get("chat_template_kwargs") or {})
ctk.setdefault("enable_thinking", False)
extra["chat_template_kwargs"] = ctk
return {**kwargs, "extra_body": extra}
async def complete(
self,
messages: list[ChatMessage],
@ -127,6 +142,8 @@ class VLLMBackend(LLMBackend):
LLMTimeoutError: If the request times out.
"""
kwargs = self._apply_thinking_default(kwargs)
async def _do_complete() -> CompletionResponse:
response = await self._client.chat.completions.create(
model=model,
@ -240,6 +257,7 @@ class VLLMBackend(LLMBackend):
LLMTimeoutError: If the request times out.
"""
stream = None
kwargs = self._apply_thinking_default(kwargs)
try:
stream = await self._client.chat.completions.create(
model=model,

10
ai_platform/modules/llm-inference/src/llm_inference/config.py
View file

@ -111,6 +111,16 @@ class LLMSettings(BaseSettings):
default=None,
description="API key for vLLM server (if required)",
)
vllm_disable_thinking: bool = Field(
default=True,
description=(
"Inject chat_template_kwargs={'enable_thinking': False} on vLLM chat "
"requests so reasoning models (e.g. Qwen3.5) return the final answer "
"directly instead of a 'thinking' preamble — required for callers that "
"parse JSON (extractors sentiment/OCR, video semantic, brain). Callers "
"may override by passing their own chat_template_kwargs."
),
)
# llama.cpp settings
llamacpp_base_url: str = Field(