clossers/didi-lot1-ai
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
didi-lot1-ai/ai_platform/DEPLOYMENT.md

17 KiB
Raw Blame History

Deployment Guide

This guide covers bootstrapping the web + dashboard stack on a fresh CPU-only host. All GPU-dependent services (llm-inference, embeddings, rerank, audio, video-analysis) remain on a separate GPU machine and are called over HTTP.

Architecture

┌─────────────────────────── CPU-only host (this machine) ──────────────────────────────┐
│                                                                                       │
│   ┌────────────────────┐      ┌───────────────────────────────────────────────────┐   │
│   │  Backend clients   │────▶ │  Web API  (:51100)                                │   │
│   │  (your app / UI)   │      │    ├─ free tier    → SearXNG                      │   │
│   └────────────────────┘      │    └─ premium tier → SerpAPI/Tavily/Brave/LinkUp  │   │
│                               │         + OpenRouter for LLM calls                │   │
│                               └──────────────┬────────────────────────────────────┘   │
│                                              │ fires events                           │
│                                              ▼                                        │
│   ┌─────────────────────────────────────────────────────────────────────────────┐    │
│   │  Dashboard  (:51300)   ◀──  PostgreSQL  (:15432 internal)                   │    │
│   │    · overview / providers / history                                         │    │
│   │    · /config (runtime toggles)                                              │    │
│   │    · /archive (promote gathers for future claims-api)                       │    │
│   │    · /cost (spend + projections)                                            │    │
│   └─────────────────────────────────────────────────────────────────────────────┘    │
│                                                                                       │
│   ┌───────────────────┐                                                               │
│   │  SearXNG          │  (existing — free tier source of truth)                       │
│   │  :55100/8080      │                                                               │
│   └───────────────────┘                                                               │
│                                                                                       │
└──────────┬────────────────────────────────────────────────────────────────────────────┘
           │ HTTP calls (over VPN / LAN)
           ▼
┌─────────────────────────── GPU host (separate machine) ──────────────────────────────┐
│                                                                                      │
│   llm-inference  (:14011)  ◀── Web API calls this for context + evidence LLM         │
│     └─ vLLM Qwen3.5-35B  (:14001)                                                    │
│     └─ llama.cpp servers (optional, on separate GPU nodes)                           │
│                                                                                      │
│   embeddings  (:14100)    ◀── reserved for future semantic search                    │
│   rerank      (:14200)    ◀── reserved                                               │
│   audio       (:54300)    ◀── whisper transcription                                  │
│   video-api   (:54600)    ◀── deepfake detection                                     │
│                                                                                      │
└──────────────────────────────────────────────────────────────────────────────────────┘

What's deployed where

Service This host (CPU) GPU host
SearXNG ✓
web-api ✓
dashboard ✓
dashboard-db ✓
llm-inference ✓
embeddings ✓
rerank ✓
audio ✓
video-analysis ✓

Prerequisites

  • OS: Linux (Ubuntu 22.04+ or similar)
  • Docker: 24.0+ with Docker Compose V2 (docker compose subcommand)
  • Disk: ~2 GB free for images + DB volume
  • RAM: 2 GB+ available for containers
  • Network:
    • Port 51100 (web-api) + 51300 (dashboard) free on the host
    • 15432 free if you want the dashboard DB exposed for debugging
    • Outbound HTTPS to SerpAPI, Tavily, Brave, LinkUp, OpenRouter (if using premium)
    • LAN/VPN reachability to your GPU host for WEB_LLM_BASE_URL

Not required on this host: NVIDIA driver, CUDA, nvidia-container-toolkit.

Quickstart (one command)

git clone https://git.finesynergy.eu/ml/ml-projects.git
cd ml-projects
./bootstrap.sh

The script will:

  1. Verify Docker + Compose.
  2. Create the deploy_default network.
  3. Prompt for:
    • GPU host address (for LLM URL)
    • SearXNG container name / URL
    • Paid provider keys (SerpAPI, Tavily, Brave, LinkUp, OpenRouter — leave blank to skip)
    • Generate a random Postgres password
  4. Render modules/web/deploy/.env and modules/dashboard/deploy/.env.
  5. Deploy dashboard + web-api.
  6. Create an admin user and print the Bearer token.
  7. Run smoke tests.

Script flags

Flag Purpose
(none) Interactive — prompts for everything
--non-interactive Use existing .env files without prompting
--deploy-searxng Also deploy the bundled SearXNG stack from modules/web/deploy/metasearch
--skip-smoke Skip post-deploy smoke tests
--help Show usage

Manual deployment (step by step)

If the bootstrap script fails or you prefer manual control:

1. Create the network

docker network inspect deploy_default >/dev/null 2>&1 \
  || docker network create deploy_default

2. Ensure SearXNG is reachable

If SearXNG is already running on this host, connect it to the network:

docker network connect deploy_default <searxng-container-name>

Otherwise, deploy the bundled SearXNG:

cd modules/web/deploy/metasearch
docker compose up -d
cd -

3. Configure and deploy the dashboard

cp modules/dashboard/.env.example modules/dashboard/deploy/.env
# Edit modules/dashboard/deploy/.env — set:
#   DASHBOARD_DB_PASSWORD=<strong-random-password>
#   DASHBOARD_LLM_API_URL=http://<your-gpu-host>:14011
#   DASHBOARD_VLLM_QWEN_URL=http://<your-gpu-host>:14001
#   (paid provider keys if you want live quota readouts)

cd modules/dashboard/deploy
./deploy.sh up

# Wait for health
curl http://localhost:51300/health

4. Configure and deploy web-api

cp modules/web/.env.example modules/web/deploy/.env
# Edit modules/web/deploy/.env — set:
#   WEB_SEARXNG_BASE_URL=http://<searxng-container>:8080
#   WEB_LLM_BASE_URL=http://<your-gpu-host>:14011
#   WEB_VISION_BASE_URL=http://<your-gpu-host>:14011
#   WEB_DASHBOARD_URL=http://didiAI-dashboard:51300
#   (paid provider keys for premium tier)
#   (WEB_OPENROUTER_API_KEY for premium LLM)

cd modules/web/deploy
docker compose --profile api up -d --build

# Wait for health
curl http://localhost:51100/health

5. Create an admin user on the dashboard

docker exec -it didiAI-dashboard python -m dashboard.cli \
  create-user <your-username> \
  --email <your-email> \
  --role admin

Save the Bearer token it prints — it's not stored in plain text and can't be recovered.

6. Verify everything works

# Dashboard
curl http://localhost:51300/health
curl http://localhost:51300/api/stats/providers | jq '.providers | length'

# Free tier search (uses SearXNG)
curl -X POST http://localhost:51100/v1/search \
  -H "Content-Type: application/json" \
  -d '{"queries":["hello world"],"max_results":3}'

# Premium tier search (uses paid rotation)
curl -X POST http://localhost:51100/v1/search \
  -H "Content-Type: application/json" \
  -H "X-Search-Tier: premium" \
  -d '{"queries":["hello world"],"max_results":3}'

Then browse to http://localhost:51300 for the UI.

Configuration reference

Required values for this host

File Variable Purpose
modules/web/deploy/.env WEB_SEARXNG_BASE_URL SearXNG internal URL
modules/web/deploy/.env WEB_LLM_BASE_URL Remote LLM inference endpoint
modules/web/deploy/.env WEB_VISION_BASE_URL Usually same as LLM URL
modules/web/deploy/.env WEB_EXTERNAL_URL Public-facing web-api URL
modules/dashboard/deploy/.env DASHBOARD_DB_PASSWORD Random strong password
modules/dashboard/deploy/.env DASHBOARD_EXTERNAL_URL Public-facing dashboard URL
modules/dashboard/deploy/.env DASHBOARD_LLM_API_URL Remote LLM (for health check card)

Optional — enable premium tier

Set at least one of these in modules/web/deploy/.env. If all are blank, premium tier returns empty results.

WEB_SERPAPI_API_KEY=...
WEB_TAVILY_API_KEY=...
WEB_BRAVE_API_KEY=...
WEB_LINKUP_API_KEY=...
WEB_OPENROUTER_API_KEY=...
WEB_OPENROUTER_MODEL=google/gemini-3.1-flash-lite-preview

Also mirror these into modules/dashboard/deploy/.env (DASHBOARD_* prefix) so the dashboard can show live quota.

Dashboard ↔ Web wiring

The web-api polls the dashboard for runtime config every 30s and fires events back. This requires:

# modules/web/deploy/.env
WEB_DASHBOARD_URL=http://didiAI-dashboard:51300

Both containers must be on the deploy_default network for internal hostname resolution.

Sending traffic

Send X-Search-Tier to select the pipeline:

# Free — SearXNG only + remote Qwen LLM (cheap / slow / open)
curl -X POST http://localhost:51100/v1/gather \
  -H "Content-Type: application/json" \
  -d '{"claim":"...","max_search_results":10}'

# Premium — paid rotation + OpenRouter (fast / higher quality)
curl -X POST http://localhost:51100/v1/gather \
  -H "Content-Type: application/json" \
  -H "X-Search-Tier: premium" \
  -d '{"claim":"...","max_search_results":10}'

Response schema is identical between tiers. Only the upstream sources differ.

Operations

Tailing logs

docker logs -f didiAI-web-api
docker logs -f didiAI-dashboard
docker logs -f didiAI-dashboard-db

Restarting a single container

docker restart didiAI-web-api
# or for a full rebuild:
cd modules/web/deploy && docker compose --profile api up -d --build

Managing users

# Create
docker exec didiAI-dashboard python -m dashboard.cli create-user alice \
  --email alice@example.com --role admin

# List
docker exec didiAI-dashboard python -m dashboard.cli list-users

# Delete
docker exec didiAI-dashboard python -m dashboard.cli delete-user alice

Runtime configuration (no restart needed)

Open http://localhost:51300/config for the UI, or use the API:

TOKEN="<your admin token>"

# Disable a provider
curl -X PUT http://localhost:51300/api/config/web.providers.serpapi.enabled \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"value": false}'

# Switch premium rotation strategy
curl -X PUT http://localhost:51300/api/config/web.premium.strategy \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"value": "parallel"}'

# Swap OpenRouter model
curl -X PUT http://localhost:51300/api/config/web.openrouter.model \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"value": "anthropic/claude-haiku-4.5"}'

Changes propagate within 30 seconds (next poll cycle) — or faster with docker restart didiAI-web-api.

Audit trail at http://localhost:51300/audit.

Backup & restore

Dashboard database

The dashboard stores request history, runtime config, audit log, users, and archived claims in PostgreSQL volume deploy_dashboard_db_data.

Nightly backup:

docker exec didiAI-dashboard-db pg_dump -U dashboard -d dashboard \
  | gzip > "backup-$(date +%Y%m%d).sql.gz"

Restore on a fresh install:

# Make sure the dashboard DB container is running but the app is down
docker stop didiAI-dashboard
gunzip < backup-YYYYMMDD.sql.gz | \
  docker exec -i didiAI-dashboard-db psql -U dashboard -d dashboard
docker start didiAI-dashboard

Config files

.env files under modules/*/deploy/.env are gitignored — they are the only stateful config on the host outside PostgreSQL. Back them up securely (they contain provider API keys and the DB password).

Troubleshooting

docker network deploy_default not found

docker network create deploy_default

SearXNG unreachable from web-api

Check that SearXNG is on the shared network:

docker network inspect deploy_default | grep -A2 searxng
# If missing:
docker network connect deploy_default <searxng-container>

Then verify the hostname/port in WEB_SEARXNG_BASE_URL.

Dashboard shows "LLM unreachable"

This host has no GPU. WEB_LLM_BASE_URL + DASHBOARD_LLM_API_URL must point at your GPU machine:

WEB_LLM_BASE_URL=http://10.x.x.x:14011

Verify from inside the web-api container:

docker exec didiAI-web-api curl -sf http://10.x.x.x:14011/v1/models

Premium tier returns empty results

Open http://localhost:51300/providers and check each paid provider's status:

  • Down / 401 → invalid API key in .env. Fix and redeploy web-api.
  • 429 Too Many Requests → monthly quota exhausted. Either disable the provider runtime on /config or top up the plan.

pg_isready failing on dashboard-db

First start takes ~15 s to initialize the database cluster. If it's still failing after 60 s:

docker logs didiAI-dashboard-db --tail 50

Common cause: stale volume with different credentials. To wipe and restart:

cd modules/dashboard/deploy
docker compose --profile dashboard down
docker volume rm deploy_dashboard_db_data
docker compose --profile dashboard up -d

Warning: this deletes all request history, config overrides, archive, and users. Back up first if needed.

Port already in use

Edit the exposed port mapping in modules/<module>/deploy/docker-compose.yml and update *_EXTERNAL_URL in the matching .env.

"Invalid token" on /api/config PUT

Use a user token created via docker exec ... create-user, not an env-based token. Legacy static tokens in DASHBOARD_API_TOKENS still work but only if no users exist in the DB.

Upgrade

Pull latest code and redeploy affected modules:

git pull
# Dashboard changes:
cd modules/dashboard/deploy && docker compose --profile dashboard up -d --build
# Web changes:
cd modules/web/deploy && docker compose --profile api up -d --build

Schema migrations happen automatically on startup (the dashboard creates missing tables idempotently). For destructive schema changes, see release notes.

Security notes

  • All services assume they run on a trusted internal network (VPN / LAN). The dashboard's read endpoints are unauthenticated by design.
  • Only mutations (PUT /api/config/*, POST /api/archive/promote/*) require a Bearer token.
  • If you need to expose any of these on the public internet, put them behind an authenticated reverse proxy and set WEB_API_TOKENS / DASHBOARD_API_TOKENS to enforce auth on read endpoints too.
  • Provider API keys live in .env files (mode 0644 on disk). Rotate them periodically via each provider's dashboard.
  • The admin Bearer token is printed once. Store it in a password manager; rotate with delete-user + create-user.

Related docs

  • CLAUDE.md — stable coding conventions
  • ENDPOINTS.md — endpoint catalog across modules
  • INDEX.md — module directory
  • STATUS.md — living notes on the active work stream
  • modules/web/README.md — web module details
  • modules/dashboard/README.md — dashboard module details