# Web Module

Web search service for DIDI claim verification. Routes between free (SearXNG meta-search) and premium providers (Brave, Tavily, SerpAPI, Linkup, Exa) based on the `X-Search-Tier` header. Provides a unified gather pipeline (search -> fetch -> evidence extraction) used by agent-v3 as the fallback path when the brain knowledge base returns no evidence for a claim.

- **Stack:** Python 3.10+, FastAPI, Pydantic v2, httpx (HTTP/2), Uvicorn
- **URL:** `http://10.11.10.12:51100` (Dev) / `http://10.11.10.13:51100` (Prod-style reference per agent-v3 default)
- **Container:** `didiAI-web-api`
- **SearXNG:** single container `didiAI-web-searxng` (internal `:8080`, not published on the host) — see `local_gpu_stack`. The multi-replica cluster (`deploy/metasearch`) is an optional scaling variant, not part of the delivery.

## Ce face

The module powers the fact-checking pipeline:

- **Free search** via SearXNG metasearch (the delivered deployment uses a single `didiAI-web-searxng` container on internal `:8080`). Local Qwen LLM is used for context detection + evidence snippet extraction.
- **Premium search** via paid APIs — Brave, Tavily, SerpAPI, Linkup, Exa — selected by `PaidSearchClient` rotation (read keys from `WEB_*_API_KEY` envs). Uses OpenRouter (configurable model) for LLM steps.
- **URL fetching** with readability-lxml extraction; auto-fallback chain `HTTP -> Playwright (browse) -> Vision LLM screenshot OCR` for JS-heavy or protected pages. PDF URLs are skipped.
- **Brain cache** integration: both tiers READ from the brain cache on `/v1/gather`; only premium WRITES quality results back via the `BrainIngestSink` (fire-and-forget). Free users effectively get the paid knowledge base for free.
- **Dashboard event sink:** every request emits a structured event (tier, provider, duration, status, results_count, raw response snapshot for `/v1/gather`) to the configured dashboard.
- **Runtime config:** `RuntimeConfigClient` polls the dashboard for tier overrides (e.g. `web.tier.free.max_search_results`).

## API endpoints

All `/v1/*` routes are protected by Bearer auth (`verify_bearer_token`) and concurrency-limited. Tier-aware routes read the `X-Search-Tier: free | premium` header (default `free`).

| Method | Path | Body schema | Description |
|--------|------|-------------|-------------|
| `POST` | `/v1/gather` | `GatherRequest` | **Main endpoint** — full pipeline (cache check -> search -> fetch -> evidence) returning `GatherResponse`. Tier-aware (free/premium orchestrators). Premium populates the brain cache when quality threshold is met. |
| `POST` | `/v1/search` | `SearchRequest` | Multi-query web search. Tier-aware (`searxng` for free, `paid-rotation` for premium). |
| `POST` | `/v1/image-search` | `ImageSearchRequest` | Image search (tier-aware, same routing as text search). |
| `POST` | `/v1/fetch` | `FetchRequest` | Fetch URL list with readability extraction. |
| `GET`  | `/v1/info` | — | Service catalog metadata (resource + functions, JSON schemas) for the catalog-api. |
| `GET`  | `/health` | — | Liveness + provider health (`searxng` reachability). |
| `GET`  | `/ready` | — | Readiness probe (verifies `search_client_free` + `orchestrator_free` are wired). |

Errors map to standard HTTP codes via `make_error_detail()`: 429 rate-limit, 502 connection/provider, 504 timeout, 500 generic.

## How didi-backend uses it

- **agent-v3** calls this service via `M17_WEB_API_URL` (default `http://10.11.10.13:51100` per agent-v3 `docker-compose.yml`).
- The brain client in agent-v3 first queries the DIDI brain knowledge base; on a MISS (or low-quality hit), agent-v3 falls through to `POST /v1/gather` here.
- Agent-v3 sets the `X-Search-Tier` header to choose providers: `free` for normal sessions, `premium` for paid/priority queries. Premium runs are what populate the shared brain cache for everyone.

## Structura fisiere

```
src/web/
├── __init__.py
├── cli.py                    # `web` CLI entry point
├── config.py                 # WebSettings (pydantic-settings, WEB_* prefix) + SettingsCache
├── exceptions.py             # WebError hierarchy (Provider/RateLimit/Timeout/Connection/Search)
├── logging.py                # Structured logging + request_id contextvar
├── orchestrator.py           # Orchestrator: search -> fetch -> evidence pack pipeline
├── runtime_config.py         # RuntimeConfigClient (polls dashboard /api/config)
├── validation.py             # Schema validators
│
├── api/
│   ├── app.py                # FastAPI factory, lifespan (clients + free/premium orchestrators)
│   ├── dependencies.py       # Bearer auth, concurrency limiter, tier resolver
│   ├── middleware.py         # CombinedMiddleware (request_id + rate limit)
│   └── routes/
│       ├── gather.py         # POST /v1/gather (main pipeline + brain cache read/write)
│       ├── search.py         # POST /v1/search
│       ├── image_search.py   # POST /v1/image-search
│       ├── fetch.py          # POST /v1/fetch
│       ├── health.py         # GET /health, /ready
│       └── info.py           # GET /v1/info (catalog metadata)
│
├── schemas/                  # Pydantic request/response models
│   ├── common.py             # PageContent, ProviderHealth, error helpers
│   ├── search.py             # SearchRequest/Response
│   ├── image_search.py       # ImageSearchRequest/Response
│   ├── fetch.py              # FetchRequest/Response
│   ├── browse.py             # BrowseRequest/Response
│   ├── vision.py             # VisionExtractRequest/Response
│   ├── evidence.py           # EvidencePackRequest/Response
│   ├── context.py            # Context detection schemas
│   └── gather.py             # GatherRequest/Response (unified)
│
├── search/                   # Provider implementations
│   ├── protocol.py           # SearchProvider Protocol
│   ├── multi.py              # MultiSearchClient (default, backwards-compat)
│   ├── paid.py               # PaidSearchClient (rotation across paid providers)
│   ├── brave.py              # Brave Search API
│   ├── tavily.py             # Tavily API
│   ├── serpapi.py            # SerpAPI
│   ├── linkup.py             # Linkup API
│   └── exa.py                # Exa API
│
├── metasearch/
│   └── client.py             # SearXNGClient (free tier)
│
├── fetch/                    # HTTP + readability extraction
├── browse/                   # Playwright browser automation
├── vision/                   # Screenshot + Vision LLM extraction
├── evidence/                 # EvidencePacker (dedupe, snippets, scoring)
├── llm/                      # LLMProviderChain (local -> OpenRouter -> OpenAI -> Anthropic)
├── context/                  # Claim context detection helpers
├── events/
│   └── sink.py               # DashboardEventSink (fire-and-forget telemetry)
└── brain/
    ├── client.py             # BrainClient (gather + ingest HTTP client)
    ├── adapter.py            # brain <-> web schema converters
    ├── quality.py            # quality_ok_for_cache, brain_hit_acceptable
    └── sink.py               # BrainIngestSink (premium-only cache writer)
```

## SearXNG

**Delivered deployment:** a single SearXNG container `didiAI-web-searxng` (`docker.io/searxng/searxng:latest`) on internal port `:8080`, wired via `local_gpu_stack`. It is **not** published on the host, and there is no Redis/Valkey cache, no Tor proxy, and no nginx load balancer in the current infrastructure. The web-api reaches it at `WEB_SEARXNG_BASE_URL=http://didiAI-web-searxng:8080`.

Article extraction in the API uses readability-lxml; JS-heavy pages fall through to Playwright (chromium) and finally to a vision-LLM screenshot pass.

**Optional scaling variant** (not part of the delivery), located at `deploy/metasearch/`:

- Multiple SearXNG instances round-robin behind an nginx LB.
- Per-instance config, Caddyfile + reset script (`searxng-reset.sh`) included for ops.

## Configuration

All envs use the `WEB_` prefix (loaded via pydantic-settings). Highlights:

| Variable | Purpose |
|----------|---------|
| `WEB_SEARXNG_BASE_URL` | SearXNG LB endpoint (default `http://didiAI-web-searxng:8080`) — REQUIRED |
| `WEB_LLM_BASE_URL` / `WEB_VISION_BASE_URL` | Local LLM router (Qwen) for free tier |
| `WEB_TEXT_MODEL` / `WEB_VISION_MODEL` | Model names served by the LLM endpoint |
| `WEB_LLM_API_KEY` | Optional auth for the LLM endpoint |
| `WEB_BRAVE_API_KEY` / `WEB_TAVILY_API_KEY` / `WEB_SERPAPI_API_KEY` / `WEB_LINKUP_API_KEY` / `WEB_EXA_API_KEY` | Premium-tier provider keys (any subset) |
| `WEB_OPENROUTER_API_KEY` / `WEB_OPENROUTER_MODEL` | Premium-tier LLM provider |
| `WEB_OPENAI_API_KEY` / `WEB_ANTHROPIC_API_KEY` | Optional fallback LLM providers |
| `WEB_DASHBOARD_URL` / `WEB_DASHBOARD_TOKEN` | Dashboard event sink + runtime config polling |
| `WEB_EXTERNAL_URL` | Public URL advertised in OpenAPI/catalog |
| `WEB_HOST` / `WEB_PORT` | Bind (defaults `0.0.0.0:51100`) |
| `WEB_RATE_LIMIT_RPS` / `WEB_RATE_LIMIT_BURST` / `WEB_MAX_CONCURRENT_REQUESTS` | Throttling |
| `WEB_API_TOKENS` | Comma-separated bearer tokens for `/v1/*`; blank = open (VPN deploys) |
| `WEB_FETCH_*` / `WEB_BROWSE_*` / `WEB_VISION_*` / `WEB_EVIDENCE_*` | Tuning knobs (timeouts, viewport, dedupe threshold, snippet length) |

Full reference: `.env.example`.

## Deployment

- **API:** `cd deploy/ && docker compose --profile api up -d` (builds `didiai-web-api` from `deploy/Dockerfile`, joins networks `didi-network` + `didibrain`, exposes `51100:51100`).
- **SearXNG (delivered):** single `didiAI-web-searxng` container on internal `:8080`, brought up via `local_gpu_stack`. Optional scaling variant: `cd deploy/metasearch/ && docker compose up -d` (LB + multiple SearXNG replicas).
- **Healthcheck:** `curl http://localhost:51100/health`.
- Compose files: `deploy/docker-compose.yml`, `deploy/metasearch/docker-compose.yaml`.

## Related docs

- Module README: `README.md` (quick-start, gather request schema, fallback chain diagram).
- API reference: `API.md`.
- Brain cache integration: `BRAIN_INTEGRATION.md`.
- AI platform CLAUDE.md (parent module conventions).
- Backend integration: agent-v3 brain client (`backend/services/orchestration-layer/agent-v3`) falls through to `POST /v1/gather` here when the brain returns a MISS or low-quality hit.