153 lines
10 KiB
Markdown
153 lines
10 KiB
Markdown
# Web Module
|
|
|
|
Web search service for DIDI claim verification. Routes between free (SearXNG meta-search) and premium providers (Brave, Tavily, SerpAPI, Linkup, Exa) based on the `X-Search-Tier` header. Provides a unified gather pipeline (search -> fetch -> evidence extraction) used by agent-v3 as the fallback path when the brain knowledge base returns no evidence for a claim.
|
|
|
|
- **Stack:** Python 3.10+, FastAPI, Pydantic v2, httpx (HTTP/2), Uvicorn
|
|
- **URL:** `http://10.11.10.12:51100` (Dev) / `http://10.11.10.13:51100` (Prod-style reference per agent-v3 default)
|
|
- **Container:** `didiAI-web-api`
|
|
- **SearXNG:** single container `didiAI-web-searxng` (internal `:8080`, not published on the host) — see `local_gpu_stack`. The multi-replica cluster (`deploy/metasearch`) is an optional scaling variant, not part of the delivery.
|
|
|
|
## Ce face
|
|
|
|
The module powers the fact-checking pipeline:
|
|
|
|
- **Free search** via SearXNG metasearch (the delivered deployment uses a single `didiAI-web-searxng` container on internal `:8080`). Local Qwen LLM is used for context detection + evidence snippet extraction.
|
|
- **Premium search** via paid APIs — Brave, Tavily, SerpAPI, Linkup, Exa — selected by `PaidSearchClient` rotation (read keys from `WEB_*_API_KEY` envs). Uses OpenRouter (configurable model) for LLM steps.
|
|
- **URL fetching** with readability-lxml extraction; auto-fallback chain `HTTP -> Playwright (browse) -> Vision LLM screenshot OCR` for JS-heavy or protected pages. PDF URLs are skipped.
|
|
- **Brain cache** integration: both tiers READ from the brain cache on `/v1/gather`; only premium WRITES quality results back via the `BrainIngestSink` (fire-and-forget). Free users effectively get the paid knowledge base for free.
|
|
- **Dashboard event sink:** every request emits a structured event (tier, provider, duration, status, results_count, raw response snapshot for `/v1/gather`) to the configured dashboard.
|
|
- **Runtime config:** `RuntimeConfigClient` polls the dashboard for tier overrides (e.g. `web.tier.free.max_search_results`).
|
|
|
|
## API endpoints
|
|
|
|
All `/v1/*` routes are protected by Bearer auth (`verify_bearer_token`) and concurrency-limited. Tier-aware routes read the `X-Search-Tier: free | premium` header (default `free`).
|
|
|
|
| Method | Path | Body schema | Description |
|
|
|--------|------|-------------|-------------|
|
|
| `POST` | `/v1/gather` | `GatherRequest` | **Main endpoint** — full pipeline (cache check -> search -> fetch -> evidence) returning `GatherResponse`. Tier-aware (free/premium orchestrators). Premium populates the brain cache when quality threshold is met. |
|
|
| `POST` | `/v1/search` | `SearchRequest` | Multi-query web search. Tier-aware (`searxng` for free, `paid-rotation` for premium). |
|
|
| `POST` | `/v1/image-search` | `ImageSearchRequest` | Image search (tier-aware, same routing as text search). |
|
|
| `POST` | `/v1/fetch` | `FetchRequest` | Fetch URL list with readability extraction. |
|
|
| `GET` | `/v1/info` | — | Service catalog metadata (resource + functions, JSON schemas) for the catalog-api. |
|
|
| `GET` | `/health` | — | Liveness + provider health (`searxng` reachability). |
|
|
| `GET` | `/ready` | — | Readiness probe (verifies `search_client_free` + `orchestrator_free` are wired). |
|
|
|
|
Errors map to standard HTTP codes via `make_error_detail()`: 429 rate-limit, 502 connection/provider, 504 timeout, 500 generic.
|
|
|
|
## How didi-backend uses it
|
|
|
|
- **agent-v3** calls this service via `M17_WEB_API_URL` (default `http://10.11.10.13:51100` per agent-v3 `docker-compose.yml`).
|
|
- The brain client in agent-v3 first queries the DIDI brain knowledge base; on a MISS (or low-quality hit), agent-v3 falls through to `POST /v1/gather` here.
|
|
- Agent-v3 sets the `X-Search-Tier` header to choose providers: `free` for normal sessions, `premium` for paid/priority queries. Premium runs are what populate the shared brain cache for everyone.
|
|
|
|
## Structura fisiere
|
|
|
|
```
|
|
src/web/
|
|
├── __init__.py
|
|
├── cli.py # `web` CLI entry point
|
|
├── config.py # WebSettings (pydantic-settings, WEB_* prefix) + SettingsCache
|
|
├── exceptions.py # WebError hierarchy (Provider/RateLimit/Timeout/Connection/Search)
|
|
├── logging.py # Structured logging + request_id contextvar
|
|
├── orchestrator.py # Orchestrator: search -> fetch -> evidence pack pipeline
|
|
├── runtime_config.py # RuntimeConfigClient (polls dashboard /api/config)
|
|
├── validation.py # Schema validators
|
|
│
|
|
├── api/
|
|
│ ├── app.py # FastAPI factory, lifespan (clients + free/premium orchestrators)
|
|
│ ├── dependencies.py # Bearer auth, concurrency limiter, tier resolver
|
|
│ ├── middleware.py # CombinedMiddleware (request_id + rate limit)
|
|
│ └── routes/
|
|
│ ├── gather.py # POST /v1/gather (main pipeline + brain cache read/write)
|
|
│ ├── search.py # POST /v1/search
|
|
│ ├── image_search.py # POST /v1/image-search
|
|
│ ├── fetch.py # POST /v1/fetch
|
|
│ ├── health.py # GET /health, /ready
|
|
│ └── info.py # GET /v1/info (catalog metadata)
|
|
│
|
|
├── schemas/ # Pydantic request/response models
|
|
│ ├── common.py # PageContent, ProviderHealth, error helpers
|
|
│ ├── search.py # SearchRequest/Response
|
|
│ ├── image_search.py # ImageSearchRequest/Response
|
|
│ ├── fetch.py # FetchRequest/Response
|
|
│ ├── browse.py # BrowseRequest/Response
|
|
│ ├── vision.py # VisionExtractRequest/Response
|
|
│ ├── evidence.py # EvidencePackRequest/Response
|
|
│ ├── context.py # Context detection schemas
|
|
│ └── gather.py # GatherRequest/Response (unified)
|
|
│
|
|
├── search/ # Provider implementations
|
|
│ ├── protocol.py # SearchProvider Protocol
|
|
│ ├── multi.py # MultiSearchClient (default, backwards-compat)
|
|
│ ├── paid.py # PaidSearchClient (rotation across paid providers)
|
|
│ ├── brave.py # Brave Search API
|
|
│ ├── tavily.py # Tavily API
|
|
│ ├── serpapi.py # SerpAPI
|
|
│ ├── linkup.py # Linkup API
|
|
│ └── exa.py # Exa API
|
|
│
|
|
├── metasearch/
|
|
│ └── client.py # SearXNGClient (free tier)
|
|
│
|
|
├── fetch/ # HTTP + readability extraction
|
|
├── browse/ # Playwright browser automation
|
|
├── vision/ # Screenshot + Vision LLM extraction
|
|
├── evidence/ # EvidencePacker (dedupe, snippets, scoring)
|
|
├── llm/ # LLMProviderChain (local -> OpenRouter -> OpenAI -> Anthropic)
|
|
├── context/ # Claim context detection helpers
|
|
├── events/
|
|
│ └── sink.py # DashboardEventSink (fire-and-forget telemetry)
|
|
└── brain/
|
|
├── client.py # BrainClient (gather + ingest HTTP client)
|
|
├── adapter.py # brain <-> web schema converters
|
|
├── quality.py # quality_ok_for_cache, brain_hit_acceptable
|
|
└── sink.py # BrainIngestSink (premium-only cache writer)
|
|
```
|
|
|
|
## SearXNG
|
|
|
|
**Delivered deployment:** a single SearXNG container `didiAI-web-searxng` (`docker.io/searxng/searxng:latest`) on internal port `:8080`, wired via `local_gpu_stack`. It is **not** published on the host, and there is no Redis/Valkey cache, no Tor proxy, and no nginx load balancer in the current infrastructure. The web-api reaches it at `WEB_SEARXNG_BASE_URL=http://didiAI-web-searxng:8080`.
|
|
|
|
Article extraction in the API uses readability-lxml; JS-heavy pages fall through to Playwright (chromium) and finally to a vision-LLM screenshot pass.
|
|
|
|
**Optional scaling variant** (not part of the delivery), located at `deploy/metasearch/`:
|
|
|
|
- Multiple SearXNG instances round-robin behind an nginx LB.
|
|
- Per-instance config, Caddyfile + reset script (`searxng-reset.sh`) included for ops.
|
|
|
|
## Configuration
|
|
|
|
All envs use the `WEB_` prefix (loaded via pydantic-settings). Highlights:
|
|
|
|
| Variable | Purpose |
|
|
|----------|---------|
|
|
| `WEB_SEARXNG_BASE_URL` | SearXNG LB endpoint (default `http://didiAI-web-searxng:8080`) — REQUIRED |
|
|
| `WEB_LLM_BASE_URL` / `WEB_VISION_BASE_URL` | Local LLM router (Qwen) for free tier |
|
|
| `WEB_TEXT_MODEL` / `WEB_VISION_MODEL` | Model names served by the LLM endpoint |
|
|
| `WEB_LLM_API_KEY` | Optional auth for the LLM endpoint |
|
|
| `WEB_BRAVE_API_KEY` / `WEB_TAVILY_API_KEY` / `WEB_SERPAPI_API_KEY` / `WEB_LINKUP_API_KEY` / `WEB_EXA_API_KEY` | Premium-tier provider keys (any subset) |
|
|
| `WEB_OPENROUTER_API_KEY` / `WEB_OPENROUTER_MODEL` | Premium-tier LLM provider |
|
|
| `WEB_OPENAI_API_KEY` / `WEB_ANTHROPIC_API_KEY` | Optional fallback LLM providers |
|
|
| `WEB_DASHBOARD_URL` / `WEB_DASHBOARD_TOKEN` | Dashboard event sink + runtime config polling |
|
|
| `WEB_EXTERNAL_URL` | Public URL advertised in OpenAPI/catalog |
|
|
| `WEB_HOST` / `WEB_PORT` | Bind (defaults `0.0.0.0:51100`) |
|
|
| `WEB_RATE_LIMIT_RPS` / `WEB_RATE_LIMIT_BURST` / `WEB_MAX_CONCURRENT_REQUESTS` | Throttling |
|
|
| `WEB_API_TOKENS` | Comma-separated bearer tokens for `/v1/*`; blank = open (VPN deploys) |
|
|
| `WEB_FETCH_*` / `WEB_BROWSE_*` / `WEB_VISION_*` / `WEB_EVIDENCE_*` | Tuning knobs (timeouts, viewport, dedupe threshold, snippet length) |
|
|
|
|
Full reference: `.env.example`.
|
|
|
|
## Deployment
|
|
|
|
- **API:** `cd deploy/ && docker compose --profile api up -d` (builds `didiai-web-api` from `deploy/Dockerfile`, joins networks `didi-network` + `didibrain`, exposes `51100:51100`).
|
|
- **SearXNG (delivered):** single `didiAI-web-searxng` container on internal `:8080`, brought up via `local_gpu_stack`. Optional scaling variant: `cd deploy/metasearch/ && docker compose up -d` (LB + multiple SearXNG replicas).
|
|
- **Healthcheck:** `curl http://localhost:51100/health`.
|
|
- Compose files: `deploy/docker-compose.yml`, `deploy/metasearch/docker-compose.yaml`.
|
|
|
|
## Related docs
|
|
|
|
- Module README: `README.md` (quick-start, gather request schema, fallback chain diagram).
|
|
- API reference: `API.md`.
|
|
- Brain cache integration: `BRAIN_INTEGRATION.md`.
|
|
- AI platform CLAUDE.md (parent module conventions).
|
|
- Backend integration: agent-v3 brain client (`backend/services/orchestration-layer/agent-v3`) falls through to `POST /v1/gather` here when the brain returns a MISS or low-quality hit.
|