LOT 1 - Optimizare script build -Instalare mono comanda
This commit is contained in:
parent
5380c3fc63
commit
42ff22bf85
127 changed files with 16163 additions and 532 deletions
|
|
@ -20,7 +20,7 @@ Generate text embeddings for downstream DIDI consumers:
|
|||
|
||||
- **didi-brain semantic atom storage** — text claims/atoms get embedded once at ingestion, stored as pgvector columns.
|
||||
- **didi-brain `/v1/gather` query embedding** — incoming claim text gets embedded, used as the kNN probe vector.
|
||||
- **Cross-encoder reranking** — separate `rerank` module (port 54200) handles re-scoring; this module only does dense embeddings.
|
||||
- **Cross-encoder reranking** — separate `rerank` module (port 14200) handles re-scoring; this module only does dense embeddings.
|
||||
- **Catalog / retrieval consumers** — anything that needs a vector representation hits this single endpoint.
|
||||
|
||||
The module is OpenAI-compatible: drop-in for `openai` SDK code that uses `client.embeddings.create()`.
|
||||
|
|
@ -68,7 +68,6 @@ embeddings/
|
|||
├── README.md quick-start + config table
|
||||
├── API.md full HTTP API reference + SDK examples
|
||||
├── pyproject.toml package def, optional extras: vllm, llamacpp, all, dev
|
||||
├── uv.lock
|
||||
├── .env.example all EMB_* env vars documented
|
||||
├── deploy/
|
||||
│ ├── Dockerfile multi-stage, python 3.11-slim + uv
|
||||
|
|
@ -111,7 +110,7 @@ embeddings/
|
|||
|
||||
- **Ingestion path** — when atoms/claims are written, brain calls `EmbeddingClient.embed(text)` to get a 1024-dim vector and stores it in pgvector alongside the row. One round-trip per batch.
|
||||
- **Query path** — for `/v1/gather`, the incoming claim text is embedded the same way, then the resulting vector is used as the probe in a pgvector `<=> ` (cosine) kNN search to retrieve candidate atoms.
|
||||
- **Reranking** — top-k candidates from the dense search are forwarded to the separate `rerank` module (cross-encoder, port 54200) for fine-grained scoring. That module does not call this one — they're parallel concerns.
|
||||
- **Reranking** — top-k candidates from the dense search are forwarded to the separate `rerank` module (cross-encoder, port 14200) for fine-grained scoring. That module does not call this one — they're parallel concerns.
|
||||
- **Catalog API** — also retrieves via the same embedding pipeline (text → vector → kNN), reusing this single endpoint.
|
||||
|
||||
Because the wrapper is OpenAI-compatible, `EmbeddingClient` can be a vanilla `openai.OpenAI(base_url=..., api_key=...)` instance — no DIDI-specific client code needed in brain.
|
||||
|
|
@ -138,16 +137,9 @@ All env vars use the `EMB_` prefix. Required vars have no defaults — the app r
|
|||
| `EMB_PORT` | `14100` (Prod) / `54100` (Dev) | API server port |
|
||||
| `EMB_HOST` | `0.0.0.0` | API bind address |
|
||||
| `EMB_API_TOKENS` | unset | Comma-separated bearer tokens; auth disabled if unset |
|
||||
| `EMB_VLLM_BASE_URL` | `http://localhost:54101` | Where vLLM listens |
|
||||
| `EMB_VLLM_MODEL` | `BAAI/bge-m3` | HF model id loaded by vLLM |
|
||||
| `EMB_VLLM_GPU` | `0` | `CUDA_VISIBLE_DEVICES` for vLLM |
|
||||
| `EMB_VLLM_GPU_UTIL` | `0.50` | vLLM `--gpu-memory-utilization` |
|
||||
| `EMB_VLLM_MAX_LEN` | `8192` | vLLM `--max-model-len` |
|
||||
| `EMB_LLAMACPP_BASE_URL` | `http://localhost:54110` | Where llama.cpp listens |
|
||||
| `EMB_LLAMACPP_MODEL` | `bge-m3-q4_k_m.gguf` | GGUF filename inside `MODELS_DIR` |
|
||||
| `EMB_LLAMACPP_CTX` | `8192` | llama.cpp context size |
|
||||
| `EMB_LLAMACPP_THREADS` | `4` | llama.cpp threads |
|
||||
| `EMB_LLAMACPP_PARALLEL` | `4` | llama.cpp parallel slots |
|
||||
| `EMB_VLLM_BASE_URL` | `http://localhost:54101` | Where the API wrapper reaches vLLM |
|
||||
| `EMB_VLLM_API_KEY` | unset | Optional API key for the vLLM server |
|
||||
| `EMB_LLAMACPP_BASE_URL` | `http://localhost:54110` | Where the API wrapper reaches llama.cpp |
|
||||
| `EMB_REQUEST_TIMEOUT` | `120.0` | Per-request backend timeout (s) |
|
||||
| `EMB_CONNECT_TIMEOUT` | `10.0` | TCP connect timeout (s) |
|
||||
| `EMB_RATE_LIMIT_RPS` | `20.0` | Token-bucket rate (req/s) |
|
||||
|
|
@ -155,7 +147,24 @@ All env vars use the `EMB_` prefix. Required vars have no defaults — the app r
|
|||
| `EMB_MAX_CONCURRENT_REQUESTS` | `20` | In-flight cap |
|
||||
| `EMB_LOG_LEVEL` | `INFO` | DEBUG / INFO / WARNING / ERROR |
|
||||
| `EMB_LOG_JSON` | `false` | Emit JSON-formatted log lines |
|
||||
| `HF_CACHE_DIR` | `/cai2_ds_storage/hf_cache` | Mounted into vLLM container |
|
||||
|
||||
> The API app reads only the variables above (fields on `EmbeddingSettings` in `config.py`). `EMB_VLLM_BASE_URL` / `EMB_LLAMACPP_BASE_URL` just tell the wrapper where to reach the backend servers.
|
||||
|
||||
### Backend service variables (docker-compose only)
|
||||
|
||||
These are **not** fields on `EmbeddingSettings` — they are consumed by `deploy/docker-compose.yml` to launch the vLLM / llama.cpp containers (model id, GPU pinning, memory, context). They configure the backend server, not the API app.
|
||||
|
||||
| Variable | Default | Description |
|
||||
|----------|---------|-------------|
|
||||
| `EMB_VLLM_MODEL` | `BAAI/bge-m3` | HF model id loaded by the vLLM container |
|
||||
| `EMB_VLLM_GPU` | `0` | `CUDA_VISIBLE_DEVICES` for the vLLM container |
|
||||
| `EMB_VLLM_GPU_UTIL` | `0.50` | vLLM `--gpu-memory-utilization` |
|
||||
| `EMB_VLLM_MAX_LEN` | `8192` | vLLM `--max-model-len` |
|
||||
| `EMB_LLAMACPP_MODEL` | `bge-m3-q4_k_m.gguf` | GGUF filename inside `MODELS_DIR` |
|
||||
| `EMB_LLAMACPP_CTX` | `8192` | llama.cpp context size |
|
||||
| `EMB_LLAMACPP_THREADS` | `4` | llama.cpp threads |
|
||||
| `EMB_LLAMACPP_PARALLEL` | `4` | llama.cpp parallel slots |
|
||||
| `HF_CACHE_DIR` | `/cai2_ds_storage/hf_cache` | Mounted into the vLLM container |
|
||||
| `HF_TOKEN` | unset | Forwarded as `HUGGING_FACE_HUB_TOKEN` |
|
||||
| `MODELS_DIR` | `/cai2_ds_storage/models` | GGUF model directory for llama.cpp |
|
||||
|
||||
|
|
@ -208,7 +217,7 @@ Compose profiles:
|
|||
## Related
|
||||
|
||||
- **`didi-brain`** consumes via `shared/embedding_client.py` for both ingestion (text → pgvector storage) and `/v1/gather` query embedding.
|
||||
- **`rerank` module** (separate, port 54200) handles cross-encoder scoring on top of dense kNN candidates from this module — they're complementary, not chained inside this service.
|
||||
- **`rerank` module** (separate, port 14200) handles cross-encoder scoring on top of dense kNN candidates from this module — they're complementary, not chained inside this service.
|
||||
- **Catalog API** uses the same endpoint for retrieval-side embeddings.
|
||||
- **pgvector** in the brain Postgres stores the resulting 1024-dim vectors (cosine distance index).
|
||||
- **Local Python use**: `from embeddings import EmbeddingClient` (works without the HTTP wrapper if you want in-process inference and have the `vllm`/`llamacpp` extras installed).
|
||||
|
|
|
|||
|
|
@ -14,10 +14,10 @@
|
|||
# Naming Convention: didiAI-{module}-{service}
|
||||
#
|
||||
# Network:
|
||||
# Uses deploy_default network (shared with other modules)
|
||||
# Uses didi-network (shared with all DIDI + AI platform stacks)
|
||||
|
||||
networks:
|
||||
deploy_default:
|
||||
didi-network:
|
||||
external: true
|
||||
|
||||
services:
|
||||
|
|
@ -33,7 +33,7 @@ services:
|
|||
ports:
|
||||
- "${EMB_PORT:-14100}:${EMB_PORT:-14100}"
|
||||
networks:
|
||||
- deploy_default
|
||||
- didi-network
|
||||
environment:
|
||||
- EMB_PORT=${EMB_PORT:-14100}
|
||||
- EMB_EXTERNAL_URL=${EMB_EXTERNAL_URL}
|
||||
|
|
@ -66,7 +66,7 @@ services:
|
|||
ports:
|
||||
- "${EMB_VLLM_PORT:-14101}:14101"
|
||||
networks:
|
||||
- deploy_default
|
||||
- didi-network
|
||||
volumes:
|
||||
- ${HF_CACHE_DIR:-/cai2_ds_storage/hf_cache}:/root/.cache/huggingface
|
||||
environment:
|
||||
|
|
@ -109,7 +109,7 @@ services:
|
|||
ports:
|
||||
- "${EMB_LLAMACPP_PORT:-14110}:8080"
|
||||
networks:
|
||||
- deploy_default
|
||||
- didi-network
|
||||
volumes:
|
||||
- ${MODELS_DIR:-/cai2_ds_storage/models}:/models:ro
|
||||
command: >
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue