clossers/didi-lot1-ai
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Livrare LOT 1 - Didi

Browse source
This commit is contained in:
Dezvoltari Evotech 2026-06-25 14:13:25 -07:00
commit 5380c3fc63
990 changed files with 133308 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

156
ai_platform/modules/web/INDEX.md Normal file
View file

@ -0,0 +1,156 @@
# Web Module
Web search service for DIDI claim verification. Routes between free (SearXNG meta-search) and premium providers (Brave, Tavily, SerpAPI, Linkup, Exa) based on the `X-Search-Tier` header. Provides a unified gather pipeline (search -> fetch -> evidence extraction) used by agent-v3 as the fallback path when the brain knowledge base returns no evidence for a claim.
- **Stack:** Python 3.10+, FastAPI, Pydantic v2, httpx (HTTP/2), Uvicorn
- **URL:** `http://10.11.10.12:51100` (Dev) / `http://10.11.10.13:51100` (Prod-style reference per agent-v3 default)
- **Container:** `didiAI-web-api`
- **SearXNG cluster:** 3 replicas (`didiAI-web-searxng-1/2/3`) behind nginx LB (`didiAI-web-searxng`, port 55100)
- **SearXNG cache:** 3 Valkey/Redis replicas (`didiAI-web-searxng-redis-1/2/3`)
- **Anonymity proxy:** `didiAI-web-tor` (Tor SOCKS5 shared by all SearXNG instances)
## Ce face
The module powers the fact-checking pipeline:
- **Free search** via SearXNG metasearch (3 round-robin instances, each with a dedicated Valkey for isolated cache/state). Local Qwen LLM is used for context detection + evidence snippet extraction.
- **Premium search** via paid APIs — Brave, Tavily, SerpAPI, Linkup, Exa — selected by `PaidSearchClient` rotation (read keys from `WEB_*_API_KEY` envs). Uses OpenRouter (configurable model) for LLM steps.
- **URL fetching** with readability-lxml extraction; auto-fallback chain `HTTP -> Playwright (browse) -> Vision LLM screenshot OCR` for JS-heavy or protected pages. PDF URLs are skipped.
- **Tor proxy** for sensitive/anonymized SearXNG queries.
- **Brain cache** integration: both tiers READ from the brain cache on `/v1/gather`; only premium WRITES quality results back via the `BrainIngestSink` (fire-and-forget). Free users effectively get the paid knowledge base for free.
- **Dashboard event sink:** every request emits a structured event (tier, provider, duration, status, results_count, raw response snapshot for `/v1/gather`) to the configured dashboard.
- **Runtime config:** `RuntimeConfigClient` polls the dashboard for tier overrides (e.g. `web.tier.free.max_search_results`).
## API endpoints
All `/v1/*` routes are protected by Bearer auth (`verify_bearer_token`) and concurrency-limited. Tier-aware routes read the `X-Search-Tier: free | premium` header (default `free`).
| Method | Path | Body schema | Description |
|--------|------|-------------|-------------|
| `POST` | `/v1/gather` | `GatherRequest` | **Main endpoint** — full pipeline (cache check -> search -> fetch -> evidence) returning `GatherResponse`. Tier-aware (free/premium orchestrators). Premium populates the brain cache when quality threshold is met. |
| `POST` | `/v1/search` | `SearchRequest` | Multi-query web search. Tier-aware (`searxng` for free, `paid-rotation` for premium). |
| `POST` | `/v1/image-search` | `ImageSearchRequest` | Image search (tier-aware, same routing as text search). |
| `POST` | `/v1/fetch` | `FetchRequest` | Fetch URL list with readability extraction. |
| `GET` | `/v1/info` | — | Service catalog metadata (resource + functions, JSON schemas) for the catalog-api. |
| `GET` | `/health` | — | Liveness + provider health (`searxng` reachability). |
| `GET` | `/ready` | — | Readiness probe (verifies `search_client_free` + `orchestrator_free` are wired). |
Errors map to standard HTTP codes via `make_error_detail()`: 429 rate-limit, 502 connection/provider, 504 timeout, 500 generic.
## How didi-backend uses it
- **agent-v3** calls this service via `M17_WEB_API_URL` (default `http://10.11.10.13:51100` per agent-v3 `docker-compose.yml`).
- The brain client in agent-v3 first queries the DIDI brain knowledge base; on a MISS (or low-quality hit), agent-v3 falls through to `POST /v1/gather` here.
- Agent-v3 sets the `X-Search-Tier` header to choose providers: `free` for normal sessions, `premium` for paid/priority queries. Premium runs are what populate the shared brain cache for everyone.
## Structura fisiere
```
src/web/
├── __init__.py
├── cli.py # `web` CLI entry point
├── config.py # WebSettings (pydantic-settings, WEB_* prefix) + SettingsCache
├── exceptions.py # WebError hierarchy (Provider/RateLimit/Timeout/Connection/Search)
├── logging.py # Structured logging + request_id contextvar
├── orchestrator.py # Orchestrator: search -> fetch -> evidence pack pipeline
├── runtime_config.py # RuntimeConfigClient (polls dashboard /api/config)
├── validation.py # Schema validators
│
├── api/
│ ├── app.py # FastAPI factory, lifespan (clients + free/premium orchestrators)
│ ├── dependencies.py # Bearer auth, concurrency limiter, tier resolver
│ ├── middleware.py # CombinedMiddleware (request_id + rate limit)
│ └── routes/
│ ├── gather.py # POST /v1/gather (main pipeline + brain cache read/write)
│ ├── search.py # POST /v1/search
│ ├── image_search.py # POST /v1/image-search
│ ├── fetch.py # POST /v1/fetch
│ ├── health.py # GET /health, /ready
│ └── info.py # GET /v1/info (catalog metadata)
│
├── schemas/ # Pydantic request/response models
│ ├── common.py # PageContent, ProviderHealth, error helpers
│ ├── search.py # SearchRequest/Response
│ ├── image_search.py # ImageSearchRequest/Response
│ ├── fetch.py # FetchRequest/Response
│ ├── browse.py # BrowseRequest/Response
│ ├── vision.py # VisionExtractRequest/Response
│ ├── evidence.py # EvidencePackRequest/Response
│ ├── context.py # Context detection schemas
│ └── gather.py # GatherRequest/Response (unified)
│
├── search/ # Provider implementations
│ ├── protocol.py # SearchProvider Protocol
│ ├── multi.py # MultiSearchClient (default, backwards-compat)
│ ├── paid.py # PaidSearchClient (rotation across paid providers)
│ ├── brave.py # Brave Search API
│ ├── tavily.py # Tavily API
│ ├── serpapi.py # SerpAPI
│ ├── linkup.py # Linkup API
│ └── exa.py # Exa API
│
├── metasearch/
│ └── client.py # SearXNGClient (free tier)
│
├── fetch/ # HTTP + readability extraction
├── browse/ # Playwright browser automation
├── vision/ # Screenshot + Vision LLM extraction
├── evidence/ # EvidencePacker (dedupe, snippets, scoring)
├── llm/ # LLMProviderChain (local -> OpenRouter -> OpenAI -> Anthropic)
├── context/ # Claim context detection helpers
├── events/
│ └── sink.py # DashboardEventSink (fire-and-forget telemetry)
└── brain/
├── client.py # BrainClient (gather + ingest HTTP client)
├── adapter.py # brain <-> web schema converters
├── quality.py # quality_ok_for_cache, brain_hit_acceptable
└── sink.py # BrainIngestSink (premium-only cache writer)
```
## SearXNG cluster
Located at `deploy/metasearch/`:
- 3 SearXNG instances (`docker.io/searxng/searxng:latest`) round-robin behind an nginx LB (`searxng-lb` -> port 55100).
- Each SearXNG instance has its own dedicated Valkey 8 cache (`searxng-redis-data-{1,2,3}`) for isolated state.
- Shared Tor SOCKS5 proxy (`dperson/torproxy`) for queries needing anonymity.
- Per-instance config under `deploy/metasearch/searxng-{1,2,3}/`.
- Caddyfile + reset script (`searxng-reset.sh`) included for ops.
- Article extraction in the API uses readability-lxml; JS-heavy pages fall through to Playwright (chromium) and finally to a vision-LLM screenshot pass.
## Configuration
All envs use the `WEB_` prefix (loaded via pydantic-settings). Highlights:
| Variable | Purpose |
|----------|---------|
| `WEB_SEARXNG_BASE_URL` | SearXNG LB endpoint (default `http://didiAI-web-searxng:8080`) — REQUIRED |
| `WEB_LLM_BASE_URL` / `WEB_VISION_BASE_URL` | Local LLM router (Qwen) for free tier |
| `WEB_TEXT_MODEL` / `WEB_VISION_MODEL` | Model names served by the LLM endpoint |
| `WEB_LLM_API_KEY` | Optional auth for the LLM endpoint |
| `WEB_BRAVE_API_KEY` / `WEB_TAVILY_API_KEY` / `WEB_SERPAPI_API_KEY` / `WEB_LINKUP_API_KEY` / `WEB_EXA_API_KEY` | Premium-tier provider keys (any subset) |
| `WEB_OPENROUTER_API_KEY` / `WEB_OPENROUTER_MODEL` | Premium-tier LLM provider |
| `WEB_OPENAI_API_KEY` / `WEB_ANTHROPIC_API_KEY` | Optional fallback LLM providers |
| `WEB_DASHBOARD_URL` / `WEB_DASHBOARD_TOKEN` | Dashboard event sink + runtime config polling |
| `WEB_EXTERNAL_URL` | Public URL advertised in OpenAPI/catalog |
| `WEB_HOST` / `WEB_PORT` | Bind (defaults `0.0.0.0:51100`) |
| `WEB_RATE_LIMIT_RPS` / `WEB_RATE_LIMIT_BURST` / `WEB_MAX_CONCURRENT_REQUESTS` | Throttling |
| `WEB_API_TOKENS` | Comma-separated bearer tokens for `/v1/*`; blank = open (VPN deploys) |
| `WEB_FETCH_*` / `WEB_BROWSE_*` / `WEB_VISION_*` / `WEB_EVIDENCE_*` | Tuning knobs (timeouts, viewport, dedupe threshold, snippet length) |
Full reference: `.env.example`.
## Deployment
- **API:** `cd deploy/ && docker compose --profile api up -d` (builds `didiai-web-api` from `deploy/Dockerfile`, joins networks `didi-network` + `didibrain`, exposes `51100:51100`).
- **SearXNG cluster:** `cd deploy/metasearch/ && docker compose up -d` (LB + 3x SearXNG + 3x Valkey + Tor).
- **Healthcheck:** `curl http://localhost:51100/health`.
- Compose files: `deploy/docker-compose.yml`, `deploy/metasearch/docker-compose.yaml`.
## Related docs
- Module README: `README.md` (quick-start, gather request schema, fallback chain diagram).
- API reference: `API.md`.
- Brain cache integration: `BRAIN_INTEGRATION.md`.
- AI platform CLAUDE.md (parent module conventions).
- Backend integration: agent-v3 brain client (`backend/services/orchestration-layer/agent-v3`) falls through to `POST /v1/gather` here when the brain returns a MISS or low-quality hit.