didi-lot1-ai/ai_platform/modules/web/README.md

# Web Module

Unified web module for search, fetch, browse, vision, and evidence gathering. Provides REST APIs for web content extraction with automatic fallback between methods.

## What It Does

This module is designed for **fact-checking pipelines**. Given a claim, it:
1. **Searches** the web for relevant sources (SearXNG metasearch)
2. **Fetches** page content using the best method available
3. **Extracts** relevant evidence snippets using LLM
4. Returns structured evidence for verification

## Prerequisites

**Required:**
- All global prerequisites (see main [README.md](../../README.md))
- SearXNG instance (deploy with `cd deploy/metasearch && docker compose up -d`)
- LLM Inference server (llm-inference module at port 14011)

**Optional:**
- OpenAI API key (for vision/LLM fallback)
- Anthropic API key (for vision/LLM fallback)

## Features

| Component | Description |
|-----------|-------------|
| **Search** | SearXNG metasearch client with site filtering, freshness, language |
| **Fetch** | HTTP content extraction with readability-lxml |
| **Browse** | Playwright-based JavaScript rendering for dynamic pages |
| **Vision** | Screenshot + Vision LLM for complex/protected pages |
| **Evidence** | Deduplicate + LLM snippet extraction + relevance scoring |
| **Orchestrator** | Auto-fallback chain: HTTP → Playwright → Vision LLM |

### Smart Features

- **PDF Filtering**: Automatically skips direct PDF URLs (can't extract text)
- **Thinking Mode Handling**: Strips `<think>` tags from reasoning models (Qwen3)
- **Auto-Fallback**: Escalates to more powerful methods when content is poor
- **Deduplication**: Removes duplicate content based on text similarity

## Installation

```bash
cd modules/web

# Install base dependencies
uv sync

# Install with all extras (fetch, browse, vision)
uv sync --all-extras

# Install Playwright browsers
uv run playwright install chromium

# Install with dev dependencies
uv sync --extra dev --all-extras
```

## Quick Start

### Docker Deployment (Recommended)

```bash
cd deploy/

# Configure environment
cat > .env << 'EOF'
WEB_SEARXNG_BASE_URL=http://localhost:55100
WEB_LLM_BASE_URL=http://didiAI-llm-api:14011
WEB_LLM_API_KEY=your-llm-api-key
WEB_VISION_MODEL=qwen-vl
WEB_TEXT_MODEL=qwen3-235b
WEB_LOG_LEVEL=INFO
EOF

# Start the server
docker compose --profile api up -d

# Check health
curl http://localhost:51100/health
```

### Example: Fact-Check a Claim

```bash
curl -X POST http://localhost:51100/v1/gather \
  -H "Content-Type: application/json" \
  -d '{
    "claim": "Romania had the highest economic growth in the EU in 2024",
    "search_queries": ["Romania GDP growth 2024 EU"],
    "max_search_results": 5,
    "extract_snippets": true,
    "max_evidence_items": 5
  }'
```

**Response:**
```json
{
  "claim": "Romania had the highest economic growth in the EU in 2024",
  "evidence": [
    {
      "url": "https://en.wikipedia.org/wiki/Economy_of_Romania",
      "title": "Economy of Romania - Wikipedia",
      "snippet": "Romania's nominal GDP reached approximately $423 billion in 2024, reflecting real growth of 0.9% that year...",
      "relevance_score": 0.6
    }
  ],
  "stages": [
    {"stage": "search", "success": true, "items_processed": 5},
    {"stage": "fetch", "success": true, "items_processed": 4},
    {"stage": "evidence", "success": true, "items_processed": 3}
  ],
  "execution_time_ms": 40643.68
}
```

## API Endpoints

| Endpoint | Method | Description |
|----------|--------|-------------|
| `/v1/gather` | POST | **Main endpoint** - unified pipeline |
| `/v1/search` | POST | Execute web search only |
| `/v1/image-search` | POST | Search for images |
| `/v1/fetch` | POST | Fetch URLs (HTTP + readability) |
| `/health` | GET | Health check |
| `/ready` | GET | Readiness probe |

### Gather Request Schema

```json
{
  "claim": "The claim to verify",
  "search_queries": ["optional", "custom", "queries"],
  "max_search_results": 20,
  "site_allowlist": ["reuters.com", "bbc.com"],
  "site_blocklist": ["spam-site.com"],
  "fetch_method": "auto",
  "auto_fallback": true,
  "extract_snippets": true,
  "max_evidence_items": 15,
  "dedupe": true
}
```

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `claim` | string | required | The claim to gather evidence for |
| `search_queries` | array | null | Custom search queries (auto-generated if not provided) |
| `max_search_results` | int | 20 | Max search results (5-50) |
| `site_allowlist` | array | null | Only search these domains |
| `site_blocklist` | array | null | Exclude these domains |
| `fetch_method` | string | "auto" | "auto", "http", "browse", "vision" |
| `auto_fallback` | bool | true | Escalate on fetch failure |
| `extract_snippets` | bool | true | Use LLM for snippet extraction |
| `max_evidence_items` | int | 15 | Max items in final pack |
| `dedupe` | bool | true | Deduplicate evidence |

## Configuration

Configure via environment variables (prefix: `WEB_`):

| Variable | Default | Description |
|----------|---------|-------------|
| `WEB_SEARXNG_BASE_URL` | - | **Required** SearXNG instance URL |
| `WEB_LLM_BASE_URL` | - | **Required** LLM inference server URL |
| `WEB_LLM_API_KEY` | - | LLM API authentication token |
| `WEB_VISION_MODEL` | `qwen-vl` | Vision model for screenshots |
| `WEB_TEXT_MODEL` | `qwen3-235b` | Text model for snippets |
| `WEB_PORT` | `51100` | API server port |
| `WEB_OPENAI_API_KEY` | - | OpenAI fallback (optional) |
| `WEB_ANTHROPIC_API_KEY` | - | Anthropic fallback (optional) |

See `.env.example` for full configuration options.

## Auto-Fallback Logic

The module uses intelligent fallback to get the best content:

```
┌─────────────────────────────────────────────────────────┐
│  URL Input                                              │
└─────────────────┬───────────────────────────────────────┘
                  ▼
┌─────────────────────────────────────────────────────────┐
│  1. PDF Check - Skip .pdf URLs (can't extract text)     │
└─────────────────┬───────────────────────────────────────┘
                  ▼
┌─────────────────────────────────────────────────────────┐
│  2. HTTP Fetch (fastest, cheapest)                      │
│     - Uses readability-lxml for content extraction      │
│     - Detects JS-heavy pages and insufficient content   │
└─────────────────┬───────────────────────────────────────┘
                  ▼ if text < min_length or JS detected
┌─────────────────────────────────────────────────────────┐
│  3. Playwright Browse (renders JavaScript)              │
│     - Waits for networkidle                             │
│     - Extracts rendered content                         │
└─────────────────┬───────────────────────────────────────┘
                  ▼ if content still poor quality
┌─────────────────────────────────────────────────────────┐
│  4. Vision LLM (screenshot → extract)                   │
│     - Takes full-page screenshot                        │
│     - Uses vision model to extract text                 │
└─────────────────────────────────────────────────────────┘
```

Enable with `fetch_method: "auto"` and `auto_fallback: true`.

## Architecture

```
src/web/
├── __init__.py           # Package exports
├── config.py             # Unified WebSettings
├── exceptions.py         # Custom exceptions
├── logging.py            # Structured logging
├── orchestrator.py       # Auto-fallback pipeline orchestration
├── cli.py                # CLI entry point
│
├── schemas/              # All request/response schemas
│   ├── common.py         # PageContent, FailedUrl, PageImage, shared types
│   ├── search.py         # SearchRequest/Response
│   ├── fetch.py          # FetchRequest/Response
│   ├── browse.py         # BrowseRequest/Response
│   ├── vision.py         # VisionExtractRequest/Response
│   ├── evidence.py       # EvidencePackRequest/Response
│   └── gather.py         # GatherRequest/Response (unified)
│
├── llm/                  # LLM provider abstraction
│   ├── __init__.py
│   └── provider.py       # LLMProviderChain (local→OpenAI→Anthropic)
│
├── search/               # Search providers
│   ├── __init__.py
│   ├── protocol.py       # SearchProvider Protocol
│   └── searxng.py        # SearXNGClient
│
├── fetch/                # HTTP + readability extraction
│   └── client.py         # FetchClient
│
├── browse/               # Playwright browser automation
│   └── client.py         # BrowseClient
│
├── vision/               # Screenshot + Vision LLM
│   └── client.py         # VisionClient (local + OpenAI + Anthropic)
│
├── evidence/             # Evidence processing
│   └── packer.py         # EvidencePacker (dedupe, snippets, scoring)
│
└── api/
    ├── app.py            # FastAPI app factory
    ├── dependencies.py   # DI (auth, rate limiter)
    ├── middleware.py     # RequestId, RateLimit
    └── routes/
        ├── search.py     # POST /v1/search
        ├── fetch.py      # POST /v1/fetch
        ├── gather.py     # POST /v1/gather (main endpoint)
        └── health.py     # /health, /ready
```

## Deployment

```bash
cd deploy/

# Configure environment (REQUIRED)
# Edit .env with your API keys

# Start server
docker compose --profile api up -d

# View logs
docker compose --profile api logs -f

# Stop server
docker compose --profile api down
```

### Port Allocation

| Port | Service |
|------|---------|
| 51100 | Web API |

### Network

The container joins the `deploy_default` network to communicate with:
- `didiAI-llm-api:14011` - LLM inference server

## Development

```bash
# Install dev dependencies
uv sync --extra dev --all-extras

# Run tests
uv run pytest

# Run tests with coverage
uv run pytest --cov=src/web --cov-report=term-missing

# Lint and format
uv run ruff check .
uv run ruff format .
```

## Dependencies on Other Modules

| Module | Purpose | Required |
|--------|---------|----------|
| llm-inference | LLM for snippet extraction & relevance scoring | Yes (for extract_snippets) |

## License

MIT