clossers/didi-lot1-ai
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

LOT 1 - Optimizare script build -Instalare mono comanda

Browse source
This commit is contained in:
Dezvoltari Evotech 2026-06-27 06:42:02 -07:00
parent 5380c3fc63
commit 42ff22bf85
127 changed files with 16163 additions and 532 deletions
Download patch file Download diff file Expand all files Collapse all files

14
ai_platform/modules/llm-inference/README.md
View file

@ -91,10 +91,12 @@ cp ../.env.example .env
| Endpoint | Method | Description |
|----------|--------|-------------|
| `/v1/chat/completions` | POST | Chat completion (supports streaming) |
| `/v1/completions` | POST | Legacy text completion (non-streaming; 501 if backend unsupported) |
| `/v1/models` | GET | List available models |
| `/v1/models/load` | POST | Load a model (local backends) |
| `/v1/models/unload` | POST | Unload a model (local backends) |
| `/v1/backends` | GET | List available backends |
| `/v1/info` | GET | Service/catalog metadata |
| `/health` | GET | Health check |
| `/ready` | GET | Readiness probe |
@ -125,13 +127,17 @@ Configure via environment variables (prefix: `LLM_`):
| Variable | Default | Description |
|----------|---------|-------------|
| `LLM_DEFAULT_BACKEND` | `litellm` | Default backend (litellm, vllm, llamacpp) |
| `LLM_DEFAULT_MODEL` | `gpt-3.5-turbo` | Default model |
| `LLM_DEFAULT_BACKEND` | _required_ | Default backend (`litellm`, `vllm`, `llamacpp`) — no default, must be set; the DIDI deployment runs `vllm` |
| `LLM_DEFAULT_MODEL` | `qwen3.5` | Default model alias used when a request omits `model` |
| `LLM_MODEL_ALIASES` | `{}` | JSON map of alias → served id, e.g. `{"qwen3.5":"Qwen/Qwen3.5-35B-A3B"}` |
| `LLM_ENABLE_FALLBACK` | `true` | Cross-backend fallback cascade on backend failure |
| `LLM_FALLBACK_ORDER` | `[vllm,llamacpp,litellm]` | Backend order tried in the fallback cascade |
| `LLM_PORT` | `14011` | API server port |
| `LLM_HOST` | `0.0.0.0` | API server host |
| `LLM_ENABLE_VLLM` | `false` | Enable vLLM backend |
| `LLM_ENABLE_LLAMACPP` | `false` | Enable llama.cpp backend |
| `LLM_ENABLE_VLLM` | _required_ | Enable vLLM backend (no default) |
| `LLM_ENABLE_LLAMACPP` | _required_ | Enable llama.cpp backend (no default) |
| `LLM_VLLM_BASE_URL` | `http://localhost:14001` | vLLM server URL |
| `LLM_VLLM_DISABLE_THINKING` | `true` | Inject `enable_thinking=false` for the Qwen3.5 reasoning model so it returns the final answer directly (no thinking preamble) |
| `LLM_LLAMACPP_BASE_URL` | `http://localhost:8080` | llama.cpp server URL |
| `OPENROUTER_API_KEY` | - | OpenRouter API key |
| `OPENAI_API_KEY` | - | OpenAI API key |