didi-lot1-ai/ai_platform/modules/web/README.md

Web Module

Unified web module for search, fetch, browse, vision, and evidence gathering. Provides REST APIs for web content extraction with automatic fallback between methods.

What It Does

This module is designed for fact-checking pipelines. Given a claim, it:

1. Searches the web for relevant sources (SearXNG metasearch)
2. Fetches page content using the best method available
3. Extracts relevant evidence snippets using LLM
4. Returns structured evidence for verification

Prerequisites

Required:

• All global prerequisites (see main README.md)
• SearXNG instance (deploy with cd deploy/metasearch && docker compose up -d)
• LLM Inference server (llm-inference module at port 14011)

Optional:

• OpenAI API key (for vision/LLM fallback)
• Anthropic API key (for vision/LLM fallback)

Features

Component	Description
Search	SearXNG metasearch client with site filtering, freshness, language
Fetch	HTTP content extraction with readability-lxml
Browse	Playwright-based JavaScript rendering for dynamic pages
Vision	Screenshot + Vision LLM for complex/protected pages
Evidence	Deduplicate + LLM snippet extraction + relevance scoring
Orchestrator	Auto-fallback chain: HTTP → Playwright → Vision LLM

Smart Features

• PDF Filtering: Automatically skips direct PDF URLs (can't extract text)
• Thinking Mode Handling: Strips <think> tags from reasoning models (Qwen3)
• Auto-Fallback: Escalates to more powerful methods when content is poor
• Deduplication: Removes duplicate content based on text similarity

Installation

cd modules/web

# Install base dependencies
uv sync

# Install with all extras (fetch, browse, vision)
uv sync --all-extras

# Install Playwright browsers
uv run playwright install chromium

# Install with dev dependencies
uv sync --extra dev --all-extras

Quick Start

Docker Deployment (Recommended)

cd deploy/

# Configure environment
cat > .env << 'EOF'
WEB_SEARXNG_BASE_URL=http://localhost:55100
WEB_LLM_BASE_URL=http://didiAI-llm-api:14011
WEB_LLM_API_KEY=your-llm-api-key
WEB_VISION_MODEL=qwen-vl
WEB_TEXT_MODEL=qwen3-235b
WEB_LOG_LEVEL=INFO
EOF

# Start the server
docker compose --profile api up -d

# Check health
curl http://localhost:51100/health

Example: Fact-Check a Claim

curl -X POST http://localhost:51100/v1/gather \
  -H "Content-Type: application/json" \
  -d '{
    "claim": "Romania had the highest economic growth in the EU in 2024",
    "search_queries": ["Romania GDP growth 2024 EU"],
    "max_search_results": 5,
    "extract_snippets": true,
    "max_evidence_items": 5
  }'

Response:

{
  "claim": "Romania had the highest economic growth in the EU in 2024",
  "evidence": [
    {
      "url": "https://en.wikipedia.org/wiki/Economy_of_Romania",
      "title": "Economy of Romania - Wikipedia",
      "snippet": "Romania's nominal GDP reached approximately $423 billion in 2024, reflecting real growth of 0.9% that year...",
      "relevance_score": 0.6
    }
  ],
  "stages": [
    {"stage": "search", "success": true, "items_processed": 5},
    {"stage": "fetch", "success": true, "items_processed": 4},
    {"stage": "evidence", "success": true, "items_processed": 3}
  ],
  "execution_time_ms": 40643.68
}

API Endpoints

Endpoint	Method	Description
/v1/gather	POST	Main endpoint - unified pipeline
/v1/search	POST	Execute web search only
/v1/image-search	POST	Search for images
/v1/fetch	POST	Fetch URLs (HTTP + readability)
/health	GET	Health check
/ready	GET	Readiness probe

Gather Request Schema

{
  "claim": "The claim to verify",
  "search_queries": ["optional", "custom", "queries"],
  "max_search_results": 20,
  "site_allowlist": ["reuters.com", "bbc.com"],
  "site_blocklist": ["spam-site.com"],
  "fetch_method": "auto",
  "auto_fallback": true,
  "extract_snippets": true,
  "max_evidence_items": 15,
  "dedupe": true
}

Field	Type	Default	Description
claim	string	required	The claim to gather evidence for
search_queries	array	null	Custom search queries (auto-generated if not provided)
max_search_results	int	20	Max search results (5-50)
site_allowlist	array	null	Only search these domains
site_blocklist	array	null	Exclude these domains
fetch_method	string	"auto"	"auto", "http", "browse", "vision"
auto_fallback	bool	true	Escalate on fetch failure
extract_snippets	bool	true	Use LLM for snippet extraction
max_evidence_items	int	15	Max items in final pack
dedupe	bool	true	Deduplicate evidence

Configuration

Configure via environment variables (prefix: WEB_):

Variable	Default	Description
WEB_SEARXNG_BASE_URL	-	Required SearXNG instance URL
WEB_LLM_BASE_URL	-	Required LLM inference server URL
WEB_LLM_API_KEY	-	LLM API authentication token
WEB_VISION_MODEL	qwen-vl	Vision model for screenshots
WEB_TEXT_MODEL	qwen3-235b	Text model for snippets
WEB_PORT	51100	API server port
WEB_OPENAI_API_KEY	-	OpenAI fallback (optional)
WEB_ANTHROPIC_API_KEY	-	Anthropic fallback (optional)

See .env.example for full configuration options.

Auto-Fallback Logic

The module uses intelligent fallback to get the best content:

┌─────────────────────────────────────────────────────────┐
│  URL Input                                              │
└─────────────────┬───────────────────────────────────────┘
                  ▼
┌─────────────────────────────────────────────────────────┐
│  1. PDF Check - Skip .pdf URLs (can't extract text)     │
└─────────────────┬───────────────────────────────────────┘
                  ▼
┌─────────────────────────────────────────────────────────┐
│  2. HTTP Fetch (fastest, cheapest)                      │
│     - Uses readability-lxml for content extraction      │
│     - Detects JS-heavy pages and insufficient content   │
└─────────────────┬───────────────────────────────────────┘
                  ▼ if text < min_length or JS detected
┌─────────────────────────────────────────────────────────┐
│  3. Playwright Browse (renders JavaScript)              │
│     - Waits for networkidle                             │
│     - Extracts rendered content                         │
└─────────────────┬───────────────────────────────────────┘
                  ▼ if content still poor quality
┌─────────────────────────────────────────────────────────┐
│  4. Vision LLM (screenshot → extract)                   │
│     - Takes full-page screenshot                        │
│     - Uses vision model to extract text                 │
└─────────────────────────────────────────────────────────┘

Enable with fetch_method: "auto" and auto_fallback: true.

Architecture

src/web/
├── __init__.py           # Package exports
├── config.py             # Unified WebSettings
├── exceptions.py         # Custom exceptions
├── logging.py            # Structured logging
├── orchestrator.py       # Auto-fallback pipeline orchestration
├── cli.py                # CLI entry point
│
├── schemas/              # All request/response schemas
│   ├── common.py         # PageContent, FailedUrl, PageImage, shared types
│   ├── search.py         # SearchRequest/Response
│   ├── fetch.py          # FetchRequest/Response
│   ├── browse.py         # BrowseRequest/Response
│   ├── vision.py         # VisionExtractRequest/Response
│   ├── evidence.py       # EvidencePackRequest/Response
│   └── gather.py         # GatherRequest/Response (unified)
│
├── llm/                  # LLM provider abstraction
│   ├── __init__.py
│   └── provider.py       # LLMProviderChain (local→OpenAI→Anthropic)
│
├── search/               # Search providers
│   ├── __init__.py
│   ├── protocol.py       # SearchProvider Protocol
│   └── searxng.py        # SearXNGClient
│
├── fetch/                # HTTP + readability extraction
│   └── client.py         # FetchClient
│
├── browse/               # Playwright browser automation
│   └── client.py         # BrowseClient
│
├── vision/               # Screenshot + Vision LLM
│   └── client.py         # VisionClient (local + OpenAI + Anthropic)
│
├── evidence/             # Evidence processing
│   └── packer.py         # EvidencePacker (dedupe, snippets, scoring)
│
└── api/
    ├── app.py            # FastAPI app factory
    ├── dependencies.py   # DI (auth, rate limiter)
    ├── middleware.py     # RequestId, RateLimit
    └── routes/
        ├── search.py     # POST /v1/search
        ├── fetch.py      # POST /v1/fetch
        ├── gather.py     # POST /v1/gather (main endpoint)
        └── health.py     # /health, /ready

Deployment

cd deploy/

# Configure environment (REQUIRED)
# Edit .env with your API keys

# Start server
docker compose --profile api up -d

# View logs
docker compose --profile api logs -f

# Stop server
docker compose --profile api down

Port Allocation

Port	Service
51100	Web API

Network

The container joins the deploy_default network to communicate with:

• didiAI-llm-api:14011 - LLM inference server

Development

# Install dev dependencies
uv sync --extra dev --all-extras

# Run tests
uv run pytest

# Run tests with coverage
uv run pytest --cov=src/web --cov-report=term-missing

# Lint and format
uv run ruff check .
uv run ruff format .

Dependencies on Other Modules

Module	Purpose	Required
llm-inference	LLM for snippet extraction & relevance scoring	Yes (for extract_snippets)

License

MIT