clossers/didi-lot1-ai
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
didi-lot1-ai/ai_platform/modules/web/INDEX.md

10 KiB
Raw Blame History

Web Module

Web search service for DIDI claim verification. Routes between free (SearXNG meta-search) and premium providers (Brave, Tavily, SerpAPI, Linkup, Exa) based on the X-Search-Tier header. Provides a unified gather pipeline (search -> fetch -> evidence extraction) used by agent-v3 as the fallback path when the brain knowledge base returns no evidence for a claim.

  • Stack: Python 3.10+, FastAPI, Pydantic v2, httpx (HTTP/2), Uvicorn
  • URL: http://10.11.10.12:51100 (Dev) / http://10.11.10.13:51100 (Prod-style reference per agent-v3 default)
  • Container: didiAI-web-api
  • SearXNG cluster: 3 replicas (didiAI-web-searxng-1/2/3) behind nginx LB (didiAI-web-searxng, port 55100)
  • SearXNG cache: 3 Valkey/Redis replicas (didiAI-web-searxng-redis-1/2/3)
  • Anonymity proxy: didiAI-web-tor (Tor SOCKS5 shared by all SearXNG instances)

Ce face

The module powers the fact-checking pipeline:

  • Free search via SearXNG metasearch (3 round-robin instances, each with a dedicated Valkey for isolated cache/state). Local Qwen LLM is used for context detection + evidence snippet extraction.
  • Premium search via paid APIs — Brave, Tavily, SerpAPI, Linkup, Exa — selected by PaidSearchClient rotation (read keys from WEB_*_API_KEY envs). Uses OpenRouter (configurable model) for LLM steps.
  • URL fetching with readability-lxml extraction; auto-fallback chain HTTP -> Playwright (browse) -> Vision LLM screenshot OCR for JS-heavy or protected pages. PDF URLs are skipped.
  • Tor proxy for sensitive/anonymized SearXNG queries.
  • Brain cache integration: both tiers READ from the brain cache on /v1/gather; only premium WRITES quality results back via the BrainIngestSink (fire-and-forget). Free users effectively get the paid knowledge base for free.
  • Dashboard event sink: every request emits a structured event (tier, provider, duration, status, results_count, raw response snapshot for /v1/gather) to the configured dashboard.
  • Runtime config: RuntimeConfigClient polls the dashboard for tier overrides (e.g. web.tier.free.max_search_results).

API endpoints

All /v1/* routes are protected by Bearer auth (verify_bearer_token) and concurrency-limited. Tier-aware routes read the X-Search-Tier: free | premium header (default free).

Method Path Body schema Description
POST /v1/gather GatherRequest Main endpoint — full pipeline (cache check -> search -> fetch -> evidence) returning GatherResponse. Tier-aware (free/premium orchestrators). Premium populates the brain cache when quality threshold is met.
POST /v1/search SearchRequest Multi-query web search. Tier-aware (searxng for free, paid-rotation for premium).
POST /v1/image-search ImageSearchRequest Image search (tier-aware, same routing as text search).
POST /v1/fetch FetchRequest Fetch URL list with readability extraction.
GET /v1/info — Service catalog metadata (resource + functions, JSON schemas) for the catalog-api.
GET /health — Liveness + provider health (searxng reachability).
GET /ready — Readiness probe (verifies search_client_free + orchestrator_free are wired).

Errors map to standard HTTP codes via make_error_detail(): 429 rate-limit, 502 connection/provider, 504 timeout, 500 generic.

How didi-backend uses it

  • agent-v3 calls this service via M17_WEB_API_URL (default http://10.11.10.13:51100 per agent-v3 docker-compose.yml).
  • The brain client in agent-v3 first queries the DIDI brain knowledge base; on a MISS (or low-quality hit), agent-v3 falls through to POST /v1/gather here.
  • Agent-v3 sets the X-Search-Tier header to choose providers: free for normal sessions, premium for paid/priority queries. Premium runs are what populate the shared brain cache for everyone.

Structura fisiere

src/web/
├── __init__.py
├── cli.py                    # `web` CLI entry point
├── config.py                 # WebSettings (pydantic-settings, WEB_* prefix) + SettingsCache
├── exceptions.py             # WebError hierarchy (Provider/RateLimit/Timeout/Connection/Search)
├── logging.py                # Structured logging + request_id contextvar
├── orchestrator.py           # Orchestrator: search -> fetch -> evidence pack pipeline
├── runtime_config.py         # RuntimeConfigClient (polls dashboard /api/config)
├── validation.py             # Schema validators
│
├── api/
│   ├── app.py                # FastAPI factory, lifespan (clients + free/premium orchestrators)
│   ├── dependencies.py       # Bearer auth, concurrency limiter, tier resolver
│   ├── middleware.py         # CombinedMiddleware (request_id + rate limit)
│   └── routes/
│       ├── gather.py         # POST /v1/gather (main pipeline + brain cache read/write)
│       ├── search.py         # POST /v1/search
│       ├── image_search.py   # POST /v1/image-search
│       ├── fetch.py          # POST /v1/fetch
│       ├── health.py         # GET /health, /ready
│       └── info.py           # GET /v1/info (catalog metadata)
│
├── schemas/                  # Pydantic request/response models
│   ├── common.py             # PageContent, ProviderHealth, error helpers
│   ├── search.py             # SearchRequest/Response
│   ├── image_search.py       # ImageSearchRequest/Response
│   ├── fetch.py              # FetchRequest/Response
│   ├── browse.py             # BrowseRequest/Response
│   ├── vision.py             # VisionExtractRequest/Response
│   ├── evidence.py           # EvidencePackRequest/Response
│   ├── context.py            # Context detection schemas
│   └── gather.py             # GatherRequest/Response (unified)
│
├── search/                   # Provider implementations
│   ├── protocol.py           # SearchProvider Protocol
│   ├── multi.py              # MultiSearchClient (default, backwards-compat)
│   ├── paid.py               # PaidSearchClient (rotation across paid providers)
│   ├── brave.py              # Brave Search API
│   ├── tavily.py             # Tavily API
│   ├── serpapi.py            # SerpAPI
│   ├── linkup.py             # Linkup API
│   └── exa.py                # Exa API
│
├── metasearch/
│   └── client.py             # SearXNGClient (free tier)
│
├── fetch/                    # HTTP + readability extraction
├── browse/                   # Playwright browser automation
├── vision/                   # Screenshot + Vision LLM extraction
├── evidence/                 # EvidencePacker (dedupe, snippets, scoring)
├── llm/                      # LLMProviderChain (local -> OpenRouter -> OpenAI -> Anthropic)
├── context/                  # Claim context detection helpers
├── events/
│   └── sink.py               # DashboardEventSink (fire-and-forget telemetry)
└── brain/
    ├── client.py             # BrainClient (gather + ingest HTTP client)
    ├── adapter.py            # brain <-> web schema converters
    ├── quality.py            # quality_ok_for_cache, brain_hit_acceptable
    └── sink.py               # BrainIngestSink (premium-only cache writer)

SearXNG cluster

Located at deploy/metasearch/:

  • 3 SearXNG instances (docker.io/searxng/searxng:latest) round-robin behind an nginx LB (searxng-lb -> port 55100).
  • Each SearXNG instance has its own dedicated Valkey 8 cache (searxng-redis-data-{1,2,3}) for isolated state.
  • Shared Tor SOCKS5 proxy (dperson/torproxy) for queries needing anonymity.
  • Per-instance config under deploy/metasearch/searxng-{1,2,3}/.
  • Caddyfile + reset script (searxng-reset.sh) included for ops.
  • Article extraction in the API uses readability-lxml; JS-heavy pages fall through to Playwright (chromium) and finally to a vision-LLM screenshot pass.

Configuration

All envs use the WEB_ prefix (loaded via pydantic-settings). Highlights:

Variable Purpose
WEB_SEARXNG_BASE_URL SearXNG LB endpoint (default http://didiAI-web-searxng:8080) — REQUIRED
WEB_LLM_BASE_URL / WEB_VISION_BASE_URL Local LLM router (Qwen) for free tier
WEB_TEXT_MODEL / WEB_VISION_MODEL Model names served by the LLM endpoint
WEB_LLM_API_KEY Optional auth for the LLM endpoint
WEB_BRAVE_API_KEY / WEB_TAVILY_API_KEY / WEB_SERPAPI_API_KEY / WEB_LINKUP_API_KEY / WEB_EXA_API_KEY Premium-tier provider keys (any subset)
WEB_OPENROUTER_API_KEY / WEB_OPENROUTER_MODEL Premium-tier LLM provider
WEB_OPENAI_API_KEY / WEB_ANTHROPIC_API_KEY Optional fallback LLM providers
WEB_DASHBOARD_URL / WEB_DASHBOARD_TOKEN Dashboard event sink + runtime config polling
WEB_EXTERNAL_URL Public URL advertised in OpenAPI/catalog
WEB_HOST / WEB_PORT Bind (defaults 0.0.0.0:51100)
WEB_RATE_LIMIT_RPS / WEB_RATE_LIMIT_BURST / WEB_MAX_CONCURRENT_REQUESTS Throttling
WEB_API_TOKENS Comma-separated bearer tokens for /v1/*; blank = open (VPN deploys)
WEB_FETCH_* / WEB_BROWSE_* / WEB_VISION_* / WEB_EVIDENCE_* Tuning knobs (timeouts, viewport, dedupe threshold, snippet length)

Full reference: .env.example.

Deployment

  • API: cd deploy/ && docker compose --profile api up -d (builds didiai-web-api from deploy/Dockerfile, joins networks didi-network + didibrain, exposes 51100:51100).
  • SearXNG cluster: cd deploy/metasearch/ && docker compose up -d (LB + 3x SearXNG + 3x Valkey + Tor).
  • Healthcheck: curl http://localhost:51100/health.
  • Compose files: deploy/docker-compose.yml, deploy/metasearch/docker-compose.yaml.

Related docs

  • Module README: README.md (quick-start, gather request schema, fallback chain diagram).
  • API reference: API.md.
  • Brain cache integration: BRAIN_INTEGRATION.md.
  • AI platform CLAUDE.md (parent module conventions).
  • Backend integration: agent-v3 brain client (backend/services/orchestration-layer/agent-v3) falls through to POST /v1/gather here when the brain returns a MISS or low-quality hit.