474 lines
18 KiB
Markdown
474 lines
18 KiB
Markdown
# Deployment Guide
|
|
|
|
> **Note — delivered deployment:** the delivered stack runs **single-host on the GPU machine** via `ai_platform/local_gpu_stack/deploy.sh` (see `local_gpu_stack/README.md`). The guide below describes the **reference multi-host variant** (CPU host + separate GPU host), kept intentionally for portability.
|
|
|
|
This guide covers bootstrapping the **web + dashboard** stack on a fresh CPU-only host. All GPU-dependent services (`llm-inference`, `embeddings`, `rerank`, `audio`, `video-analysis`) remain on a separate GPU machine and are called over HTTP.
|
|
|
|
---
|
|
|
|
## Architecture
|
|
|
|
```
|
|
┌─────────────────────────── CPU-only host (this machine) ──────────────────────────────┐
|
|
│ │
|
|
│ ┌────────────────────┐ ┌───────────────────────────────────────────────────┐ │
|
|
│ │ Backend clients │────▶ │ Web API (:51100) │ │
|
|
│ │ (your app / UI) │ │ ├─ free tier → SearXNG │ │
|
|
│ └────────────────────┘ │ └─ premium tier → SerpAPI/Tavily/Brave/LinkUp │ │
|
|
│ │ + OpenRouter for LLM calls │ │
|
|
│ └──────────────┬────────────────────────────────────┘ │
|
|
│ │ fires events │
|
|
│ ▼ │
|
|
│ ┌─────────────────────────────────────────────────────────────────────────────┐ │
|
|
│ │ Dashboard (:51300) ◀── PostgreSQL (:15432 internal) │ │
|
|
│ │ · overview / providers / history │ │
|
|
│ │ · /config (runtime toggles) │ │
|
|
│ │ · /archive (promote gathers for future claims-api) │ │
|
|
│ │ · /cost (spend + projections) │ │
|
|
│ └─────────────────────────────────────────────────────────────────────────────┘ │
|
|
│ │
|
|
│ ┌───────────────────┐ │
|
|
│ │ SearXNG │ (existing — free tier source of truth) │
|
|
│ │ :8080 (intern) │ │
|
|
│ └───────────────────┘ │
|
|
│ │
|
|
└──────────┬────────────────────────────────────────────────────────────────────────────┘
|
|
│ HTTP calls (over VPN / LAN)
|
|
▼
|
|
┌─────────────────────────── GPU host (separate machine) ──────────────────────────────┐
|
|
│ │
|
|
│ llm-inference (:14011) ◀── Web API calls this for context + evidence LLM │
|
|
│ └─ vLLM Qwen3.5-35B (:14001) │
|
|
│ └─ llama.cpp servers (optional, on separate GPU nodes) │
|
|
│ │
|
|
│ embeddings (:14100) ◀── reserved for future semantic search │
|
|
│ rerank (:14200) ◀── reserved │
|
|
│ audio (:54300) ◀── whisper transcription │
|
|
│ video-api (:54600) ◀── deepfake detection │
|
|
│ │
|
|
└──────────────────────────────────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
### What's deployed where
|
|
|
|
| Service | This host (CPU) | GPU host |
|
|
|-----------------|:---------------:|:--------------:|
|
|
| SearXNG | ✓ | |
|
|
| web-api | ✓ | |
|
|
| dashboard | ✓ | |
|
|
| dashboard-db | ✓ | |
|
|
| llm-inference | | ✓ |
|
|
| embeddings | | ✓ |
|
|
| rerank | | ✓ |
|
|
| audio | | ✓ |
|
|
| video-analysis | | ✓ |
|
|
|
|
---
|
|
|
|
## Prerequisites
|
|
|
|
- **OS:** Linux (Ubuntu 22.04+ or similar)
|
|
- **Docker:** 24.0+ with Docker Compose V2 (`docker compose` subcommand)
|
|
- **Disk:** ~2 GB free for images + DB volume
|
|
- **RAM:** 2 GB+ available for containers
|
|
- **Network:**
|
|
- Port 51100 (web-api) + 51300 (dashboard) free on the host
|
|
- 15432 free if you want the dashboard DB exposed for debugging
|
|
- Outbound HTTPS to SerpAPI, Tavily, Brave, LinkUp, OpenRouter (if using premium)
|
|
- LAN/VPN reachability to your GPU host for `WEB_LLM_BASE_URL`
|
|
|
|
Not required on this host: NVIDIA driver, CUDA, nvidia-container-toolkit.
|
|
|
|
---
|
|
|
|
## Quickstart (one command)
|
|
|
|
```bash
|
|
git clone https://git.finesynergy.eu/ml/ml-projects.git
|
|
cd ml-projects
|
|
./bootstrap.sh
|
|
```
|
|
|
|
The script will:
|
|
|
|
1. Verify Docker + Compose.
|
|
2. Create the `didi-network` network.
|
|
3. Prompt for:
|
|
- GPU host address (for LLM URL)
|
|
- SearXNG container name / URL
|
|
- Paid provider keys (SerpAPI, Tavily, Brave, LinkUp, OpenRouter — leave blank to skip)
|
|
- Generate a random Postgres password
|
|
4. Render `modules/web/deploy/.env` and `modules/dashboard/deploy/.env`.
|
|
5. Deploy dashboard + web-api.
|
|
6. Create an admin user and print the Bearer token.
|
|
7. Run smoke tests.
|
|
|
|
### Script flags
|
|
|
|
| Flag | Purpose |
|
|
|------|---------|
|
|
| _(none)_ | Interactive — prompts for everything |
|
|
| `--non-interactive` | Use existing `.env` files without prompting |
|
|
| `--deploy-searxng` | Also deploy the bundled SearXNG stack from `modules/web/deploy/metasearch` |
|
|
| `--skip-smoke` | Skip post-deploy smoke tests |
|
|
| `--help` | Show usage |
|
|
|
|
---
|
|
|
|
## Manual deployment (step by step)
|
|
|
|
If the bootstrap script fails or you prefer manual control:
|
|
|
|
### 1. Create the network
|
|
|
|
```bash
|
|
docker network inspect didi-network >/dev/null 2>&1 \
|
|
|| docker network create didi-network
|
|
```
|
|
|
|
### 2. Ensure SearXNG is reachable
|
|
|
|
If SearXNG is already running on this host, connect it to the network:
|
|
|
|
```bash
|
|
docker network connect didi-network <searxng-container-name>
|
|
```
|
|
|
|
Otherwise, deploy the bundled SearXNG:
|
|
|
|
```bash
|
|
cd modules/web/deploy/metasearch
|
|
docker compose up -d
|
|
cd -
|
|
```
|
|
|
|
### 3. Configure and deploy the dashboard
|
|
|
|
```bash
|
|
cp modules/dashboard/.env.example modules/dashboard/deploy/.env
|
|
# Edit modules/dashboard/deploy/.env — set:
|
|
# DASHBOARD_DB_PASSWORD=<strong-random-password>
|
|
# DASHBOARD_LLM_API_URL=http://<your-gpu-host>:14011
|
|
# DASHBOARD_VLLM_QWEN_URL=http://<your-gpu-host>:14001
|
|
# (paid provider keys if you want live quota readouts)
|
|
|
|
cd modules/dashboard/deploy
|
|
./deploy.sh up
|
|
|
|
# Wait for health
|
|
curl http://localhost:51300/health
|
|
```
|
|
|
|
### 4. Configure and deploy web-api
|
|
|
|
```bash
|
|
cp modules/web/.env.example modules/web/deploy/.env
|
|
# Edit modules/web/deploy/.env — set:
|
|
# WEB_SEARXNG_BASE_URL=http://<searxng-container>:8080
|
|
# WEB_LLM_BASE_URL=http://<your-gpu-host>:14011
|
|
# WEB_VISION_BASE_URL=http://<your-gpu-host>:14011
|
|
# WEB_DASHBOARD_URL=http://didiAI-dashboard:51300
|
|
# (paid provider keys for premium tier)
|
|
# (WEB_OPENROUTER_API_KEY for premium LLM)
|
|
|
|
cd modules/web/deploy
|
|
docker compose --profile api up -d --build
|
|
|
|
# Wait for health
|
|
curl http://localhost:51100/health
|
|
```
|
|
|
|
### 5. Create an admin user on the dashboard
|
|
|
|
```bash
|
|
docker exec -it didiAI-dashboard python -m dashboard.cli \
|
|
create-user <your-username> \
|
|
--email <your-email> \
|
|
--role admin
|
|
```
|
|
|
|
Save the Bearer token it prints — it's not stored in plain text and can't be recovered.
|
|
|
|
### 6. Verify everything works
|
|
|
|
```bash
|
|
# Dashboard
|
|
curl http://localhost:51300/health
|
|
curl http://localhost:51300/api/stats/providers | jq '.providers | length'
|
|
|
|
# Free tier search (uses SearXNG)
|
|
curl -X POST http://localhost:51100/v1/search \
|
|
-H "Content-Type: application/json" \
|
|
-d '{"queries":["hello world"],"max_results":3}'
|
|
|
|
# Premium tier search (uses paid rotation)
|
|
curl -X POST http://localhost:51100/v1/search \
|
|
-H "Content-Type: application/json" \
|
|
-H "X-Search-Tier: premium" \
|
|
-d '{"queries":["hello world"],"max_results":3}'
|
|
```
|
|
|
|
Then browse to <http://localhost:51300> for the UI.
|
|
|
|
---
|
|
|
|
## Configuration reference
|
|
|
|
### Required values for this host
|
|
|
|
| File | Variable | Purpose |
|
|
|------|----------|---------|
|
|
| `modules/web/deploy/.env` | `WEB_SEARXNG_BASE_URL` | SearXNG internal URL |
|
|
| `modules/web/deploy/.env` | `WEB_LLM_BASE_URL` | Remote LLM inference endpoint |
|
|
| `modules/web/deploy/.env` | `WEB_VISION_BASE_URL` | Usually same as LLM URL |
|
|
| `modules/web/deploy/.env` | `WEB_EXTERNAL_URL` | Public-facing web-api URL |
|
|
| `modules/dashboard/deploy/.env` | `DASHBOARD_DB_PASSWORD` | Random strong password |
|
|
| `modules/dashboard/deploy/.env` | `DASHBOARD_EXTERNAL_URL` | Public-facing dashboard URL |
|
|
| `modules/dashboard/deploy/.env` | `DASHBOARD_LLM_API_URL` | Remote LLM (for health check card) |
|
|
|
|
### Optional — enable premium tier
|
|
|
|
Set **at least one** of these in `modules/web/deploy/.env`. If all are blank, premium tier returns empty results.
|
|
|
|
```
|
|
WEB_SERPAPI_API_KEY=...
|
|
WEB_TAVILY_API_KEY=...
|
|
WEB_BRAVE_API_KEY=...
|
|
WEB_LINKUP_API_KEY=...
|
|
WEB_OPENROUTER_API_KEY=...
|
|
WEB_OPENROUTER_MODEL=google/gemini-3.1-flash-lite-preview
|
|
```
|
|
|
|
Also mirror these into `modules/dashboard/deploy/.env` (`DASHBOARD_*` prefix) so the dashboard can show live quota.
|
|
|
|
### Dashboard ↔ Web wiring
|
|
|
|
The web-api polls the dashboard for runtime config every 30s and fires events back. This requires:
|
|
|
|
```
|
|
# modules/web/deploy/.env
|
|
WEB_DASHBOARD_URL=http://didiAI-dashboard:51300
|
|
```
|
|
|
|
Both containers must be on the `didi-network` network for internal hostname resolution.
|
|
|
|
---
|
|
|
|
## Sending traffic
|
|
|
|
Send `X-Search-Tier` to select the pipeline:
|
|
|
|
```bash
|
|
# Free — SearXNG only + remote Qwen LLM (cheap / slow / open)
|
|
curl -X POST http://localhost:51100/v1/gather \
|
|
-H "Content-Type: application/json" \
|
|
-d '{"claim":"...","max_search_results":10}'
|
|
|
|
# Premium — paid rotation + OpenRouter (fast / higher quality)
|
|
curl -X POST http://localhost:51100/v1/gather \
|
|
-H "Content-Type: application/json" \
|
|
-H "X-Search-Tier: premium" \
|
|
-d '{"claim":"...","max_search_results":10}'
|
|
```
|
|
|
|
Response schema is identical between tiers. Only the upstream sources differ.
|
|
|
|
---
|
|
|
|
## Operations
|
|
|
|
### Tailing logs
|
|
|
|
```bash
|
|
docker logs -f didiAI-web-api
|
|
docker logs -f didiAI-dashboard
|
|
docker logs -f didiAI-dashboard-db
|
|
```
|
|
|
|
### Restarting a single container
|
|
|
|
```bash
|
|
docker restart didiAI-web-api
|
|
# or for a full rebuild:
|
|
cd modules/web/deploy && docker compose --profile api up -d --build
|
|
```
|
|
|
|
### Managing users
|
|
|
|
```bash
|
|
# Create
|
|
docker exec didiAI-dashboard python -m dashboard.cli create-user alice \
|
|
--email alice@example.com --role admin
|
|
|
|
# List
|
|
docker exec didiAI-dashboard python -m dashboard.cli list-users
|
|
|
|
# Delete
|
|
docker exec didiAI-dashboard python -m dashboard.cli delete-user alice
|
|
```
|
|
|
|
### Runtime configuration (no restart needed)
|
|
|
|
Open `http://localhost:51300/config` for the UI, or use the API:
|
|
|
|
```bash
|
|
TOKEN="<your admin token>"
|
|
|
|
# Disable a provider
|
|
curl -X PUT http://localhost:51300/api/config/web.providers.serpapi.enabled \
|
|
-H "Authorization: Bearer $TOKEN" \
|
|
-H "Content-Type: application/json" \
|
|
-d '{"value": false}'
|
|
|
|
# Switch premium rotation strategy
|
|
curl -X PUT http://localhost:51300/api/config/web.premium.strategy \
|
|
-H "Authorization: Bearer $TOKEN" \
|
|
-H "Content-Type: application/json" \
|
|
-d '{"value": "parallel"}'
|
|
|
|
# Swap OpenRouter model
|
|
curl -X PUT http://localhost:51300/api/config/web.openrouter.model \
|
|
-H "Authorization: Bearer $TOKEN" \
|
|
-H "Content-Type: application/json" \
|
|
-d '{"value": "anthropic/claude-haiku-4.5"}'
|
|
```
|
|
|
|
Changes propagate within 30 seconds (next poll cycle) — or faster with `docker restart didiAI-web-api`.
|
|
|
|
Audit trail at `http://localhost:51300/audit`.
|
|
|
|
---
|
|
|
|
## Backup & restore
|
|
|
|
### Dashboard database
|
|
|
|
The dashboard stores request history, runtime config, audit log, users, and archived claims in PostgreSQL volume `deploy_dashboard_db_data`.
|
|
|
|
Nightly backup:
|
|
|
|
```bash
|
|
docker exec didiAI-dashboard-db pg_dump -U dashboard -d dashboard \
|
|
| gzip > "backup-$(date +%Y%m%d).sql.gz"
|
|
```
|
|
|
|
Restore on a fresh install:
|
|
|
|
```bash
|
|
# Make sure the dashboard DB container is running but the app is down
|
|
docker stop didiAI-dashboard
|
|
gunzip < backup-YYYYMMDD.sql.gz | \
|
|
docker exec -i didiAI-dashboard-db psql -U dashboard -d dashboard
|
|
docker start didiAI-dashboard
|
|
```
|
|
|
|
### Config files
|
|
|
|
`.env` files under `modules/*/deploy/.env` are gitignored — they are the only stateful config on the host outside PostgreSQL. Back them up securely (they contain provider API keys and the DB password).
|
|
|
|
---
|
|
|
|
## Troubleshooting
|
|
|
|
### `docker network didi-network not found`
|
|
|
|
```bash
|
|
docker network create didi-network
|
|
```
|
|
|
|
### SearXNG unreachable from web-api
|
|
|
|
Check that SearXNG is on the shared network:
|
|
|
|
```bash
|
|
docker network inspect didi-network | grep -A2 searxng
|
|
# If missing:
|
|
docker network connect didi-network <searxng-container>
|
|
```
|
|
|
|
Then verify the hostname/port in `WEB_SEARXNG_BASE_URL`.
|
|
|
|
### Dashboard shows "LLM unreachable"
|
|
|
|
This host has no GPU. `WEB_LLM_BASE_URL` + `DASHBOARD_LLM_API_URL` must point at your GPU machine:
|
|
|
|
```
|
|
WEB_LLM_BASE_URL=http://10.x.x.x:14011
|
|
```
|
|
|
|
Verify from inside the web-api container:
|
|
|
|
```bash
|
|
docker exec didiAI-web-api curl -sf http://10.x.x.x:14011/v1/models
|
|
```
|
|
|
|
### Premium tier returns empty results
|
|
|
|
Open `http://localhost:51300/providers` and check each paid provider's status:
|
|
|
|
- **Down / 401** → invalid API key in `.env`. Fix and redeploy web-api.
|
|
- **429 Too Many Requests** → monthly quota exhausted. Either disable the provider runtime on `/config` or top up the plan.
|
|
|
|
### `pg_isready` failing on dashboard-db
|
|
|
|
First start takes ~15 s to initialize the database cluster. If it's still failing after 60 s:
|
|
|
|
```bash
|
|
docker logs didiAI-dashboard-db --tail 50
|
|
```
|
|
|
|
Common cause: stale volume with different credentials. To wipe and restart:
|
|
|
|
```bash
|
|
cd modules/dashboard/deploy
|
|
docker compose --profile dashboard down
|
|
docker volume rm deploy_dashboard_db_data
|
|
docker compose --profile dashboard up -d
|
|
```
|
|
|
|
**Warning:** this deletes all request history, config overrides, archive, and users. Back up first if needed.
|
|
|
|
### Port already in use
|
|
|
|
Edit the exposed port mapping in `modules/<module>/deploy/docker-compose.yml` and update `*_EXTERNAL_URL` in the matching `.env`.
|
|
|
|
### "Invalid token" on `/api/config` PUT
|
|
|
|
Use a user token created via `docker exec ... create-user`, not an env-based token. Legacy static tokens in `DASHBOARD_API_TOKENS` still work but only if no users exist in the DB.
|
|
|
|
---
|
|
|
|
## Upgrade
|
|
|
|
Pull latest code and redeploy affected modules:
|
|
|
|
```bash
|
|
git pull
|
|
# Dashboard changes:
|
|
cd modules/dashboard/deploy && docker compose --profile dashboard up -d --build
|
|
# Web changes:
|
|
cd modules/web/deploy && docker compose --profile api up -d --build
|
|
```
|
|
|
|
Schema migrations happen automatically on startup (the dashboard creates missing tables idempotently). For destructive schema changes, see release notes.
|
|
|
|
---
|
|
|
|
## Security notes
|
|
|
|
- All services assume they run on a trusted internal network (VPN / LAN). The dashboard's read endpoints are unauthenticated by design.
|
|
- Only mutations (`PUT /api/config/*`, `POST /api/archive/promote/*`) require a Bearer token.
|
|
- If you need to expose any of these on the public internet, put them behind an authenticated reverse proxy and set `WEB_API_TOKENS` / `DASHBOARD_API_TOKENS` to enforce auth on read endpoints too.
|
|
- Provider API keys live in `.env` files (mode 0644 on disk). Rotate them periodically via each provider's dashboard.
|
|
- The admin Bearer token is printed once. Store it in a password manager; rotate with `delete-user` + `create-user`.
|
|
|
|
---
|
|
|
|
## Related docs
|
|
|
|
- `CLAUDE.md` — stable coding conventions
|
|
- `ENDPOINTS.md` — endpoint catalog across modules
|
|
- `INDEX.md` — module directory
|
|
- `STATUS.md` — living notes on the active work stream
|
|
- `modules/web/README.md` — web module details
|
|
- `modules/dashboard/README.md` — dashboard module details
|