18 KiB
Deployment Guide
Note — delivered deployment: the delivered stack runs single-host on the GPU machine via
ai_platform/local_gpu_stack/deploy.sh(seelocal_gpu_stack/README.md). The guide below describes the reference multi-host variant (CPU host + separate GPU host), kept intentionally for portability.
This guide covers bootstrapping the web + dashboard stack on a fresh CPU-only host. All GPU-dependent services (llm-inference, embeddings, rerank, audio, video-analysis) remain on a separate GPU machine and are called over HTTP.
Architecture
┌─────────────────────────── CPU-only host (this machine) ──────────────────────────────┐
│ │
│ ┌────────────────────┐ ┌───────────────────────────────────────────────────┐ │
│ │ Backend clients │────▶ │ Web API (:51100) │ │
│ │ (your app / UI) │ │ ├─ free tier → SearXNG │ │
│ └────────────────────┘ │ └─ premium tier → SerpAPI/Tavily/Brave/LinkUp │ │
│ │ + OpenRouter for LLM calls │ │
│ └──────────────┬────────────────────────────────────┘ │
│ │ fires events │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────────────────────┐ │
│ │ Dashboard (:51300) ◀── PostgreSQL (:15432 internal) │ │
│ │ · overview / providers / history │ │
│ │ · /config (runtime toggles) │ │
│ │ · /archive (promote gathers for future claims-api) │ │
│ │ · /cost (spend + projections) │ │
│ └─────────────────────────────────────────────────────────────────────────────┘ │
│ │
│ ┌───────────────────┐ │
│ │ SearXNG │ (existing — free tier source of truth) │
│ │ :8080 (intern) │ │
│ └───────────────────┘ │
│ │
└──────────┬────────────────────────────────────────────────────────────────────────────┘
│ HTTP calls (over VPN / LAN)
▼
┌─────────────────────────── GPU host (separate machine) ──────────────────────────────┐
│ │
│ llm-inference (:14011) ◀── Web API calls this for context + evidence LLM │
│ └─ vLLM Qwen3.5-35B (:14001) │
│ └─ llama.cpp servers (optional, on separate GPU nodes) │
│ │
│ embeddings (:14100) ◀── reserved for future semantic search │
│ rerank (:14200) ◀── reserved │
│ audio (:54300) ◀── whisper transcription │
│ video-api (:54600) ◀── deepfake detection │
│ │
└──────────────────────────────────────────────────────────────────────────────────────┘
What's deployed where
| Service | This host (CPU) | GPU host |
|---|---|---|
| SearXNG | ✓ | |
| web-api | ✓ | |
| dashboard | ✓ | |
| dashboard-db | ✓ | |
| llm-inference | ✓ | |
| embeddings | ✓ | |
| rerank | ✓ | |
| audio | ✓ | |
| video-analysis | ✓ |
Prerequisites
- OS: Linux (Ubuntu 22.04+ or similar)
- Docker: 24.0+ with Docker Compose V2 (
docker composesubcommand) - Disk: ~2 GB free for images + DB volume
- RAM: 2 GB+ available for containers
- Network:
- Port 51100 (web-api) + 51300 (dashboard) free on the host
- 15432 free if you want the dashboard DB exposed for debugging
- Outbound HTTPS to SerpAPI, Tavily, Brave, LinkUp, OpenRouter (if using premium)
- LAN/VPN reachability to your GPU host for
WEB_LLM_BASE_URL
Not required on this host: NVIDIA driver, CUDA, nvidia-container-toolkit.
Quickstart (one command)
git clone https://git.finesynergy.eu/ml/ml-projects.git
cd ml-projects
./bootstrap.sh
The script will:
- Verify Docker + Compose.
- Create the
didi-networknetwork. - Prompt for:
- GPU host address (for LLM URL)
- SearXNG container name / URL
- Paid provider keys (SerpAPI, Tavily, Brave, LinkUp, OpenRouter — leave blank to skip)
- Generate a random Postgres password
- Render
modules/web/deploy/.envandmodules/dashboard/deploy/.env. - Deploy dashboard + web-api.
- Create an admin user and print the Bearer token.
- Run smoke tests.
Script flags
| Flag | Purpose |
|---|---|
| (none) | Interactive — prompts for everything |
--non-interactive |
Use existing .env files without prompting |
--deploy-searxng |
Also deploy the bundled SearXNG stack from modules/web/deploy/metasearch |
--skip-smoke |
Skip post-deploy smoke tests |
--help |
Show usage |
Manual deployment (step by step)
If the bootstrap script fails or you prefer manual control:
1. Create the network
docker network inspect didi-network >/dev/null 2>&1 \
|| docker network create didi-network
2. Ensure SearXNG is reachable
If SearXNG is already running on this host, connect it to the network:
docker network connect didi-network <searxng-container-name>
Otherwise, deploy the bundled SearXNG:
cd modules/web/deploy/metasearch
docker compose up -d
cd -
3. Configure and deploy the dashboard
cp modules/dashboard/.env.example modules/dashboard/deploy/.env
# Edit modules/dashboard/deploy/.env — set:
# DASHBOARD_DB_PASSWORD=<strong-random-password>
# DASHBOARD_LLM_API_URL=http://<your-gpu-host>:14011
# DASHBOARD_VLLM_QWEN_URL=http://<your-gpu-host>:14001
# (paid provider keys if you want live quota readouts)
cd modules/dashboard/deploy
./deploy.sh up
# Wait for health
curl http://localhost:51300/health
4. Configure and deploy web-api
cp modules/web/.env.example modules/web/deploy/.env
# Edit modules/web/deploy/.env — set:
# WEB_SEARXNG_BASE_URL=http://<searxng-container>:8080
# WEB_LLM_BASE_URL=http://<your-gpu-host>:14011
# WEB_VISION_BASE_URL=http://<your-gpu-host>:14011
# WEB_DASHBOARD_URL=http://didiAI-dashboard:51300
# (paid provider keys for premium tier)
# (WEB_OPENROUTER_API_KEY for premium LLM)
cd modules/web/deploy
docker compose --profile api up -d --build
# Wait for health
curl http://localhost:51100/health
5. Create an admin user on the dashboard
docker exec -it didiAI-dashboard python -m dashboard.cli \
create-user <your-username> \
--email <your-email> \
--role admin
Save the Bearer token it prints — it's not stored in plain text and can't be recovered.
6. Verify everything works
# Dashboard
curl http://localhost:51300/health
curl http://localhost:51300/api/stats/providers | jq '.providers | length'
# Free tier search (uses SearXNG)
curl -X POST http://localhost:51100/v1/search \
-H "Content-Type: application/json" \
-d '{"queries":["hello world"],"max_results":3}'
# Premium tier search (uses paid rotation)
curl -X POST http://localhost:51100/v1/search \
-H "Content-Type: application/json" \
-H "X-Search-Tier: premium" \
-d '{"queries":["hello world"],"max_results":3}'
Then browse to http://localhost:51300 for the UI.
Configuration reference
Required values for this host
| File | Variable | Purpose |
|---|---|---|
modules/web/deploy/.env |
WEB_SEARXNG_BASE_URL |
SearXNG internal URL |
modules/web/deploy/.env |
WEB_LLM_BASE_URL |
Remote LLM inference endpoint |
modules/web/deploy/.env |
WEB_VISION_BASE_URL |
Usually same as LLM URL |
modules/web/deploy/.env |
WEB_EXTERNAL_URL |
Public-facing web-api URL |
modules/dashboard/deploy/.env |
DASHBOARD_DB_PASSWORD |
Random strong password |
modules/dashboard/deploy/.env |
DASHBOARD_EXTERNAL_URL |
Public-facing dashboard URL |
modules/dashboard/deploy/.env |
DASHBOARD_LLM_API_URL |
Remote LLM (for health check card) |
Optional — enable premium tier
Set at least one of these in modules/web/deploy/.env. If all are blank, premium tier returns empty results.
WEB_SERPAPI_API_KEY=...
WEB_TAVILY_API_KEY=...
WEB_BRAVE_API_KEY=...
WEB_LINKUP_API_KEY=...
WEB_OPENROUTER_API_KEY=...
WEB_OPENROUTER_MODEL=google/gemini-3.1-flash-lite-preview
Also mirror these into modules/dashboard/deploy/.env (DASHBOARD_* prefix) so the dashboard can show live quota.
Dashboard ↔ Web wiring
The web-api polls the dashboard for runtime config every 30s and fires events back. This requires:
# modules/web/deploy/.env
WEB_DASHBOARD_URL=http://didiAI-dashboard:51300
Both containers must be on the didi-network network for internal hostname resolution.
Sending traffic
Send X-Search-Tier to select the pipeline:
# Free — SearXNG only + remote Qwen LLM (cheap / slow / open)
curl -X POST http://localhost:51100/v1/gather \
-H "Content-Type: application/json" \
-d '{"claim":"...","max_search_results":10}'
# Premium — paid rotation + OpenRouter (fast / higher quality)
curl -X POST http://localhost:51100/v1/gather \
-H "Content-Type: application/json" \
-H "X-Search-Tier: premium" \
-d '{"claim":"...","max_search_results":10}'
Response schema is identical between tiers. Only the upstream sources differ.
Operations
Tailing logs
docker logs -f didiAI-web-api
docker logs -f didiAI-dashboard
docker logs -f didiAI-dashboard-db
Restarting a single container
docker restart didiAI-web-api
# or for a full rebuild:
cd modules/web/deploy && docker compose --profile api up -d --build
Managing users
# Create
docker exec didiAI-dashboard python -m dashboard.cli create-user alice \
--email alice@example.com --role admin
# List
docker exec didiAI-dashboard python -m dashboard.cli list-users
# Delete
docker exec didiAI-dashboard python -m dashboard.cli delete-user alice
Runtime configuration (no restart needed)
Open http://localhost:51300/config for the UI, or use the API:
TOKEN="<your admin token>"
# Disable a provider
curl -X PUT http://localhost:51300/api/config/web.providers.serpapi.enabled \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
-d '{"value": false}'
# Switch premium rotation strategy
curl -X PUT http://localhost:51300/api/config/web.premium.strategy \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
-d '{"value": "parallel"}'
# Swap OpenRouter model
curl -X PUT http://localhost:51300/api/config/web.openrouter.model \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
-d '{"value": "anthropic/claude-haiku-4.5"}'
Changes propagate within 30 seconds (next poll cycle) — or faster with docker restart didiAI-web-api.
Audit trail at http://localhost:51300/audit.
Backup & restore
Dashboard database
The dashboard stores request history, runtime config, audit log, users, and archived claims in PostgreSQL volume deploy_dashboard_db_data.
Nightly backup:
docker exec didiAI-dashboard-db pg_dump -U dashboard -d dashboard \
| gzip > "backup-$(date +%Y%m%d).sql.gz"
Restore on a fresh install:
# Make sure the dashboard DB container is running but the app is down
docker stop didiAI-dashboard
gunzip < backup-YYYYMMDD.sql.gz | \
docker exec -i didiAI-dashboard-db psql -U dashboard -d dashboard
docker start didiAI-dashboard
Config files
.env files under modules/*/deploy/.env are gitignored — they are the only stateful config on the host outside PostgreSQL. Back them up securely (they contain provider API keys and the DB password).
Troubleshooting
docker network didi-network not found
docker network create didi-network
SearXNG unreachable from web-api
Check that SearXNG is on the shared network:
docker network inspect didi-network | grep -A2 searxng
# If missing:
docker network connect didi-network <searxng-container>
Then verify the hostname/port in WEB_SEARXNG_BASE_URL.
Dashboard shows "LLM unreachable"
This host has no GPU. WEB_LLM_BASE_URL + DASHBOARD_LLM_API_URL must point at your GPU machine:
WEB_LLM_BASE_URL=http://10.x.x.x:14011
Verify from inside the web-api container:
docker exec didiAI-web-api curl -sf http://10.x.x.x:14011/v1/models
Premium tier returns empty results
Open http://localhost:51300/providers and check each paid provider's status:
- Down / 401 → invalid API key in
.env. Fix and redeploy web-api. - 429 Too Many Requests → monthly quota exhausted. Either disable the provider runtime on
/configor top up the plan.
pg_isready failing on dashboard-db
First start takes ~15 s to initialize the database cluster. If it's still failing after 60 s:
docker logs didiAI-dashboard-db --tail 50
Common cause: stale volume with different credentials. To wipe and restart:
cd modules/dashboard/deploy
docker compose --profile dashboard down
docker volume rm deploy_dashboard_db_data
docker compose --profile dashboard up -d
Warning: this deletes all request history, config overrides, archive, and users. Back up first if needed.
Port already in use
Edit the exposed port mapping in modules/<module>/deploy/docker-compose.yml and update *_EXTERNAL_URL in the matching .env.
"Invalid token" on /api/config PUT
Use a user token created via docker exec ... create-user, not an env-based token. Legacy static tokens in DASHBOARD_API_TOKENS still work but only if no users exist in the DB.
Upgrade
Pull latest code and redeploy affected modules:
git pull
# Dashboard changes:
cd modules/dashboard/deploy && docker compose --profile dashboard up -d --build
# Web changes:
cd modules/web/deploy && docker compose --profile api up -d --build
Schema migrations happen automatically on startup (the dashboard creates missing tables idempotently). For destructive schema changes, see release notes.
Security notes
- All services assume they run on a trusted internal network (VPN / LAN). The dashboard's read endpoints are unauthenticated by design.
- Only mutations (
PUT /api/config/*,POST /api/archive/promote/*) require a Bearer token. - If you need to expose any of these on the public internet, put them behind an authenticated reverse proxy and set
WEB_API_TOKENS/DASHBOARD_API_TOKENSto enforce auth on read endpoints too. - Provider API keys live in
.envfiles (mode 0644 on disk). Rotate them periodically via each provider's dashboard. - The admin Bearer token is printed once. Store it in a password manager; rotate with
delete-user+create-user.
Related docs
CLAUDE.md— stable coding conventionsENDPOINTS.md— endpoint catalog across modulesINDEX.md— module directorySTATUS.md— living notes on the active work streammodules/web/README.md— web module detailsmodules/dashboard/README.md— dashboard module details