Web Module

Web search service for DIDI claim verification. Routes between free (SearXNG meta-search) and premium providers (Brave, Tavily, SerpAPI, Linkup, Exa) based on the X-Search-Tier header. Provides a unified gather pipeline (search -> fetch -> evidence extraction) used by agent-v3 as the fallback path when the brain knowledge base returns no evidence for a claim.

Stack: Python 3.10+, FastAPI, Pydantic v2, httpx (HTTP/2), Uvicorn
URL: http://10.11.10.12:51100 (Dev) / http://10.11.10.13:51100 (Prod-style reference per agent-v3 default)
Container: didiAI-web-api
SearXNG: single container didiAI-web-searxng (internal :8080, not published on the host) — see local_gpu_stack. The multi-replica cluster (deploy/metasearch) is an optional scaling variant, not part of the delivery.

Ce face

The module powers the fact-checking pipeline:

Free search via SearXNG metasearch (the delivered deployment uses a single didiAI-web-searxng container on internal :8080). Local Qwen LLM is used for context detection + evidence snippet extraction.
Premium search via paid APIs — Brave, Tavily, SerpAPI, Linkup, Exa — selected by PaidSearchClient rotation (read keys from WEB_*_API_KEY envs). Uses OpenRouter (configurable model) for LLM steps.
URL fetching with readability-lxml extraction; auto-fallback chain HTTP -> Playwright (browse) -> Vision LLM screenshot OCR for JS-heavy or protected pages. PDF URLs are skipped.
Brain cache integration: both tiers READ from the brain cache on /v1/gather; only premium WRITES quality results back via the BrainIngestSink (fire-and-forget). Free users effectively get the paid knowledge base for free.
Dashboard event sink: every request emits a structured event (tier, provider, duration, status, results_count, raw response snapshot for /v1/gather) to the configured dashboard.
Runtime config: RuntimeConfigClient polls the dashboard for tier overrides (e.g. web.tier.free.max_search_results).

API endpoints

All /v1/* routes are protected by Bearer auth (verify_bearer_token) and concurrency-limited. Tier-aware routes read the X-Search-Tier: free | premium header (default free).

Method	Path	Body schema	Description
`POST`	`/v1/gather`	`GatherRequest`	Main endpoint — full pipeline (cache check -> search -> fetch -> evidence) returning `GatherResponse`. Tier-aware (free/premium orchestrators). Premium populates the brain cache when quality threshold is met.
`POST`	`/v1/search`	`SearchRequest`	Multi-query web search. Tier-aware (`searxng` for free, `paid-rotation` for premium).
`POST`	`/v1/image-search`	`ImageSearchRequest`	Image search (tier-aware, same routing as text search).
`POST`	`/v1/fetch`	`FetchRequest`	Fetch URL list with readability extraction.
`GET`	`/v1/info`	—	Service catalog metadata (resource + functions, JSON schemas) for the catalog-api.
`GET`	`/health`	—	Liveness + provider health (`searxng` reachability).
`GET`	`/ready`	—	Readiness probe (verifies `search_client_free` + `orchestrator_free` are wired).

Errors map to standard HTTP codes via make_error_detail(): 429 rate-limit, 502 connection/provider, 504 timeout, 500 generic.

How didi-backend uses it

agent-v3 calls this service via M17_WEB_API_URL (default http://10.11.10.13:51100 per agent-v3 docker-compose.yml).
The brain client in agent-v3 first queries the DIDI brain knowledge base; on a MISS (or low-quality hit), agent-v3 falls through to POST /v1/gather here.
Agent-v3 sets the X-Search-Tier header to choose providers: free for normal sessions, premium for paid/priority queries. Premium runs are what populate the shared brain cache for everyone.

Structura fisiere

src/web/
├── __init__.py
├── cli.py                    # `web` CLI entry point
├── config.py                 # WebSettings (pydantic-settings, WEB_* prefix) + SettingsCache
├── exceptions.py             # WebError hierarchy (Provider/RateLimit/Timeout/Connection/Search)
├── logging.py                # Structured logging + request_id contextvar
├── orchestrator.py           # Orchestrator: search -> fetch -> evidence pack pipeline
├── runtime_config.py         # RuntimeConfigClient (polls dashboard /api/config)
├── validation.py             # Schema validators
│
├── api/
│   ├── app.py                # FastAPI factory, lifespan (clients + free/premium orchestrators)
│   ├── dependencies.py       # Bearer auth, concurrency limiter, tier resolver
│   ├── middleware.py         # CombinedMiddleware (request_id + rate limit)
│   └── routes/
│       ├── gather.py         # POST /v1/gather (main pipeline + brain cache read/write)
│       ├── search.py         # POST /v1/search
│       ├── image_search.py   # POST /v1/image-search
│       ├── fetch.py          # POST /v1/fetch
│       ├── health.py         # GET /health, /ready
│       └── info.py           # GET /v1/info (catalog metadata)
│
├── schemas/                  # Pydantic request/response models
│   ├── common.py             # PageContent, ProviderHealth, error helpers
│   ├── search.py             # SearchRequest/Response
│   ├── image_search.py       # ImageSearchRequest/Response
│   ├── fetch.py              # FetchRequest/Response
│   ├── browse.py             # BrowseRequest/Response
│   ├── vision.py             # VisionExtractRequest/Response
│   ├── evidence.py           # EvidencePackRequest/Response
│   ├── context.py            # Context detection schemas
│   └── gather.py             # GatherRequest/Response (unified)
│
├── search/                   # Provider implementations
│   ├── protocol.py           # SearchProvider Protocol
│   ├── multi.py              # MultiSearchClient (default, backwards-compat)
│   ├── paid.py               # PaidSearchClient (rotation across paid providers)
│   ├── brave.py              # Brave Search API
│   ├── tavily.py             # Tavily API
│   ├── serpapi.py            # SerpAPI
│   ├── linkup.py             # Linkup API
│   └── exa.py                # Exa API
│
├── metasearch/
│   └── client.py             # SearXNGClient (free tier)
│
├── fetch/                    # HTTP + readability extraction
├── browse/                   # Playwright browser automation
├── vision/                   # Screenshot + Vision LLM extraction
├── evidence/                 # EvidencePacker (dedupe, snippets, scoring)
├── llm/                      # LLMProviderChain (local -> OpenRouter -> OpenAI -> Anthropic)
├── context/                  # Claim context detection helpers
├── events/
│   └── sink.py               # DashboardEventSink (fire-and-forget telemetry)
└── brain/
    ├── client.py             # BrainClient (gather + ingest HTTP client)
    ├── adapter.py            # brain <-> web schema converters
    ├── quality.py            # quality_ok_for_cache, brain_hit_acceptable
    └── sink.py               # BrainIngestSink (premium-only cache writer)

SearXNG

Delivered deployment: a single SearXNG container didiAI-web-searxng (docker.io/searxng/searxng:latest) on internal port :8080, wired via local_gpu_stack. It is not published on the host, and there is no Redis/Valkey cache, no Tor proxy, and no nginx load balancer in the current infrastructure. The web-api reaches it at WEB_SEARXNG_BASE_URL=http://didiAI-web-searxng:8080.

Article extraction in the API uses readability-lxml; JS-heavy pages fall through to Playwright (chromium) and finally to a vision-LLM screenshot pass.

Optional scaling variant (not part of the delivery), located at deploy/metasearch/:

Multiple SearXNG instances round-robin behind an nginx LB.
Per-instance config, Caddyfile + reset script (searxng-reset.sh) included for ops.

Configuration

All envs use the WEB_ prefix (loaded via pydantic-settings). Highlights:

Variable	Purpose
`WEB_SEARXNG_BASE_URL`	SearXNG LB endpoint (default `http://didiAI-web-searxng:8080`) — REQUIRED
`WEB_LLM_BASE_URL` / `WEB_VISION_BASE_URL`	Local LLM router (Qwen) for free tier
`WEB_TEXT_MODEL` / `WEB_VISION_MODEL`	Model names served by the LLM endpoint
`WEB_LLM_API_KEY`	Optional auth for the LLM endpoint
`WEB_BRAVE_API_KEY` / `WEB_TAVILY_API_KEY` / `WEB_SERPAPI_API_KEY` / `WEB_LINKUP_API_KEY` / `WEB_EXA_API_KEY`	Premium-tier provider keys (any subset)
`WEB_OPENROUTER_API_KEY` / `WEB_OPENROUTER_MODEL`	Premium-tier LLM provider
`WEB_OPENAI_API_KEY` / `WEB_ANTHROPIC_API_KEY`	Optional fallback LLM providers
`WEB_DASHBOARD_URL` / `WEB_DASHBOARD_TOKEN`	Dashboard event sink + runtime config polling
`WEB_EXTERNAL_URL`	Public URL advertised in OpenAPI/catalog
`WEB_HOST` / `WEB_PORT`	Bind (defaults `0.0.0.0:51100`)
`WEB_RATE_LIMIT_RPS` / `WEB_RATE_LIMIT_BURST` / `WEB_MAX_CONCURRENT_REQUESTS`	Throttling
`WEB_API_TOKENS`	Comma-separated bearer tokens for `/v1/*`; blank = open (VPN deploys)
`WEB_FETCH_` / `WEB_BROWSE_` / `WEB_VISION_` / `WEB_EVIDENCE_`	Tuning knobs (timeouts, viewport, dedupe threshold, snippet length)

Full reference: .env.example.

Deployment

API: cd deploy/ && docker compose --profile api up -d (builds didiai-web-api from deploy/Dockerfile, joins networks didi-network + didibrain, exposes 51100:51100).
SearXNG (delivered): single didiAI-web-searxng container on internal :8080, brought up via local_gpu_stack. Optional scaling variant: cd deploy/metasearch/ && docker compose up -d (LB + multiple SearXNG replicas).
Healthcheck: curl http://localhost:51100/health.
Compose files: deploy/docker-compose.yml, deploy/metasearch/docker-compose.yaml.

Module README: README.md (quick-start, gather request schema, fallback chain diagram).
API reference: API.md.
Brain cache integration: BRAIN_INTEGRATION.md.
AI platform CLAUDE.md (parent module conventions).
Backend integration: agent-v3 brain client (backend/services/orchestration-layer/agent-v3) falls through to POST /v1/gather here when the brain returns a MISS or low-quality hit.

10 KiB Raw Blame History