Livrare LOT 1 - Didi
This commit is contained in:
commit
5380c3fc63
990 changed files with 133308 additions and 0 deletions
317
ai_platform/modules/web/README.md
Normal file
317
ai_platform/modules/web/README.md
Normal file
|
|
@ -0,0 +1,317 @@
|
|||
# Web Module
|
||||
|
||||
Unified web module for search, fetch, browse, vision, and evidence gathering. Provides REST APIs for web content extraction with automatic fallback between methods.
|
||||
|
||||
## What It Does
|
||||
|
||||
This module is designed for **fact-checking pipelines**. Given a claim, it:
|
||||
1. **Searches** the web for relevant sources (SearXNG metasearch)
|
||||
2. **Fetches** page content using the best method available
|
||||
3. **Extracts** relevant evidence snippets using LLM
|
||||
4. Returns structured evidence for verification
|
||||
|
||||
## Prerequisites
|
||||
|
||||
**Required:**
|
||||
- All global prerequisites (see main [README.md](../../README.md))
|
||||
- SearXNG instance (deploy with `cd deploy/metasearch && docker compose up -d`)
|
||||
- LLM Inference server (llm-inference module at port 14011)
|
||||
|
||||
**Optional:**
|
||||
- OpenAI API key (for vision/LLM fallback)
|
||||
- Anthropic API key (for vision/LLM fallback)
|
||||
|
||||
## Features
|
||||
|
||||
| Component | Description |
|
||||
|-----------|-------------|
|
||||
| **Search** | SearXNG metasearch client with site filtering, freshness, language |
|
||||
| **Fetch** | HTTP content extraction with readability-lxml |
|
||||
| **Browse** | Playwright-based JavaScript rendering for dynamic pages |
|
||||
| **Vision** | Screenshot + Vision LLM for complex/protected pages |
|
||||
| **Evidence** | Deduplicate + LLM snippet extraction + relevance scoring |
|
||||
| **Orchestrator** | Auto-fallback chain: HTTP → Playwright → Vision LLM |
|
||||
|
||||
### Smart Features
|
||||
|
||||
- **PDF Filtering**: Automatically skips direct PDF URLs (can't extract text)
|
||||
- **Thinking Mode Handling**: Strips `<think>` tags from reasoning models (Qwen3)
|
||||
- **Auto-Fallback**: Escalates to more powerful methods when content is poor
|
||||
- **Deduplication**: Removes duplicate content based on text similarity
|
||||
|
||||
## Installation
|
||||
|
||||
```bash
|
||||
cd modules/web
|
||||
|
||||
# Install base dependencies
|
||||
uv sync
|
||||
|
||||
# Install with all extras (fetch, browse, vision)
|
||||
uv sync --all-extras
|
||||
|
||||
# Install Playwright browsers
|
||||
uv run playwright install chromium
|
||||
|
||||
# Install with dev dependencies
|
||||
uv sync --extra dev --all-extras
|
||||
```
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Docker Deployment (Recommended)
|
||||
|
||||
```bash
|
||||
cd deploy/
|
||||
|
||||
# Configure environment
|
||||
cat > .env << 'EOF'
|
||||
WEB_SEARXNG_BASE_URL=http://localhost:55100
|
||||
WEB_LLM_BASE_URL=http://didiAI-llm-api:14011
|
||||
WEB_LLM_API_KEY=your-llm-api-key
|
||||
WEB_VISION_MODEL=qwen-vl
|
||||
WEB_TEXT_MODEL=qwen3-235b
|
||||
WEB_LOG_LEVEL=INFO
|
||||
EOF
|
||||
|
||||
# Start the server
|
||||
docker compose --profile api up -d
|
||||
|
||||
# Check health
|
||||
curl http://localhost:51100/health
|
||||
```
|
||||
|
||||
### Example: Fact-Check a Claim
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:51100/v1/gather \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"claim": "Romania had the highest economic growth in the EU in 2024",
|
||||
"search_queries": ["Romania GDP growth 2024 EU"],
|
||||
"max_search_results": 5,
|
||||
"extract_snippets": true,
|
||||
"max_evidence_items": 5
|
||||
}'
|
||||
```
|
||||
|
||||
**Response:**
|
||||
```json
|
||||
{
|
||||
"claim": "Romania had the highest economic growth in the EU in 2024",
|
||||
"evidence": [
|
||||
{
|
||||
"url": "https://en.wikipedia.org/wiki/Economy_of_Romania",
|
||||
"title": "Economy of Romania - Wikipedia",
|
||||
"snippet": "Romania's nominal GDP reached approximately $423 billion in 2024, reflecting real growth of 0.9% that year...",
|
||||
"relevance_score": 0.6
|
||||
}
|
||||
],
|
||||
"stages": [
|
||||
{"stage": "search", "success": true, "items_processed": 5},
|
||||
{"stage": "fetch", "success": true, "items_processed": 4},
|
||||
{"stage": "evidence", "success": true, "items_processed": 3}
|
||||
],
|
||||
"execution_time_ms": 40643.68
|
||||
}
|
||||
```
|
||||
|
||||
## API Endpoints
|
||||
|
||||
| Endpoint | Method | Description |
|
||||
|----------|--------|-------------|
|
||||
| `/v1/gather` | POST | **Main endpoint** - unified pipeline |
|
||||
| `/v1/search` | POST | Execute web search only |
|
||||
| `/v1/image-search` | POST | Search for images |
|
||||
| `/v1/fetch` | POST | Fetch URLs (HTTP + readability) |
|
||||
| `/health` | GET | Health check |
|
||||
| `/ready` | GET | Readiness probe |
|
||||
|
||||
### Gather Request Schema
|
||||
|
||||
```json
|
||||
{
|
||||
"claim": "The claim to verify",
|
||||
"search_queries": ["optional", "custom", "queries"],
|
||||
"max_search_results": 20,
|
||||
"site_allowlist": ["reuters.com", "bbc.com"],
|
||||
"site_blocklist": ["spam-site.com"],
|
||||
"fetch_method": "auto",
|
||||
"auto_fallback": true,
|
||||
"extract_snippets": true,
|
||||
"max_evidence_items": 15,
|
||||
"dedupe": true
|
||||
}
|
||||
```
|
||||
|
||||
| Field | Type | Default | Description |
|
||||
|-------|------|---------|-------------|
|
||||
| `claim` | string | required | The claim to gather evidence for |
|
||||
| `search_queries` | array | null | Custom search queries (auto-generated if not provided) |
|
||||
| `max_search_results` | int | 20 | Max search results (5-50) |
|
||||
| `site_allowlist` | array | null | Only search these domains |
|
||||
| `site_blocklist` | array | null | Exclude these domains |
|
||||
| `fetch_method` | string | "auto" | "auto", "http", "browse", "vision" |
|
||||
| `auto_fallback` | bool | true | Escalate on fetch failure |
|
||||
| `extract_snippets` | bool | true | Use LLM for snippet extraction |
|
||||
| `max_evidence_items` | int | 15 | Max items in final pack |
|
||||
| `dedupe` | bool | true | Deduplicate evidence |
|
||||
|
||||
## Configuration
|
||||
|
||||
Configure via environment variables (prefix: `WEB_`):
|
||||
|
||||
| Variable | Default | Description |
|
||||
|----------|---------|-------------|
|
||||
| `WEB_SEARXNG_BASE_URL` | - | **Required** SearXNG instance URL |
|
||||
| `WEB_LLM_BASE_URL` | - | **Required** LLM inference server URL |
|
||||
| `WEB_LLM_API_KEY` | - | LLM API authentication token |
|
||||
| `WEB_VISION_MODEL` | `qwen-vl` | Vision model for screenshots |
|
||||
| `WEB_TEXT_MODEL` | `qwen3-235b` | Text model for snippets |
|
||||
| `WEB_PORT` | `51100` | API server port |
|
||||
| `WEB_OPENAI_API_KEY` | - | OpenAI fallback (optional) |
|
||||
| `WEB_ANTHROPIC_API_KEY` | - | Anthropic fallback (optional) |
|
||||
|
||||
See `.env.example` for full configuration options.
|
||||
|
||||
## Auto-Fallback Logic
|
||||
|
||||
The module uses intelligent fallback to get the best content:
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────┐
|
||||
│ URL Input │
|
||||
└─────────────────┬───────────────────────────────────────┘
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────┐
|
||||
│ 1. PDF Check - Skip .pdf URLs (can't extract text) │
|
||||
└─────────────────┬───────────────────────────────────────┘
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────┐
|
||||
│ 2. HTTP Fetch (fastest, cheapest) │
|
||||
│ - Uses readability-lxml for content extraction │
|
||||
│ - Detects JS-heavy pages and insufficient content │
|
||||
└─────────────────┬───────────────────────────────────────┘
|
||||
▼ if text < min_length or JS detected
|
||||
┌─────────────────────────────────────────────────────────┐
|
||||
│ 3. Playwright Browse (renders JavaScript) │
|
||||
│ - Waits for networkidle │
|
||||
│ - Extracts rendered content │
|
||||
└─────────────────┬───────────────────────────────────────┘
|
||||
▼ if content still poor quality
|
||||
┌─────────────────────────────────────────────────────────┐
|
||||
│ 4. Vision LLM (screenshot → extract) │
|
||||
│ - Takes full-page screenshot │
|
||||
│ - Uses vision model to extract text │
|
||||
└─────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
Enable with `fetch_method: "auto"` and `auto_fallback: true`.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
src/web/
|
||||
├── __init__.py # Package exports
|
||||
├── config.py # Unified WebSettings
|
||||
├── exceptions.py # Custom exceptions
|
||||
├── logging.py # Structured logging
|
||||
├── orchestrator.py # Auto-fallback pipeline orchestration
|
||||
├── cli.py # CLI entry point
|
||||
│
|
||||
├── schemas/ # All request/response schemas
|
||||
│ ├── common.py # PageContent, FailedUrl, PageImage, shared types
|
||||
│ ├── search.py # SearchRequest/Response
|
||||
│ ├── fetch.py # FetchRequest/Response
|
||||
│ ├── browse.py # BrowseRequest/Response
|
||||
│ ├── vision.py # VisionExtractRequest/Response
|
||||
│ ├── evidence.py # EvidencePackRequest/Response
|
||||
│ └── gather.py # GatherRequest/Response (unified)
|
||||
│
|
||||
├── llm/ # LLM provider abstraction
|
||||
│ ├── __init__.py
|
||||
│ └── provider.py # LLMProviderChain (local→OpenAI→Anthropic)
|
||||
│
|
||||
├── search/ # Search providers
|
||||
│ ├── __init__.py
|
||||
│ ├── protocol.py # SearchProvider Protocol
|
||||
│ └── searxng.py # SearXNGClient
|
||||
│
|
||||
├── fetch/ # HTTP + readability extraction
|
||||
│ └── client.py # FetchClient
|
||||
│
|
||||
├── browse/ # Playwright browser automation
|
||||
│ └── client.py # BrowseClient
|
||||
│
|
||||
├── vision/ # Screenshot + Vision LLM
|
||||
│ └── client.py # VisionClient (local + OpenAI + Anthropic)
|
||||
│
|
||||
├── evidence/ # Evidence processing
|
||||
│ └── packer.py # EvidencePacker (dedupe, snippets, scoring)
|
||||
│
|
||||
└── api/
|
||||
├── app.py # FastAPI app factory
|
||||
├── dependencies.py # DI (auth, rate limiter)
|
||||
├── middleware.py # RequestId, RateLimit
|
||||
└── routes/
|
||||
├── search.py # POST /v1/search
|
||||
├── fetch.py # POST /v1/fetch
|
||||
├── gather.py # POST /v1/gather (main endpoint)
|
||||
└── health.py # /health, /ready
|
||||
```
|
||||
|
||||
## Deployment
|
||||
|
||||
```bash
|
||||
cd deploy/
|
||||
|
||||
# Configure environment (REQUIRED)
|
||||
# Edit .env with your API keys
|
||||
|
||||
# Start server
|
||||
docker compose --profile api up -d
|
||||
|
||||
# View logs
|
||||
docker compose --profile api logs -f
|
||||
|
||||
# Stop server
|
||||
docker compose --profile api down
|
||||
```
|
||||
|
||||
### Port Allocation
|
||||
|
||||
| Port | Service |
|
||||
|------|---------|
|
||||
| 51100 | Web API |
|
||||
|
||||
### Network
|
||||
|
||||
The container joins the `deploy_default` network to communicate with:
|
||||
- `didiAI-llm-api:14011` - LLM inference server
|
||||
|
||||
## Development
|
||||
|
||||
```bash
|
||||
# Install dev dependencies
|
||||
uv sync --extra dev --all-extras
|
||||
|
||||
# Run tests
|
||||
uv run pytest
|
||||
|
||||
# Run tests with coverage
|
||||
uv run pytest --cov=src/web --cov-report=term-missing
|
||||
|
||||
# Lint and format
|
||||
uv run ruff check .
|
||||
uv run ruff format .
|
||||
```
|
||||
|
||||
## Dependencies on Other Modules
|
||||
|
||||
| Module | Purpose | Required |
|
||||
|--------|---------|----------|
|
||||
| llm-inference | LLM for snippet extraction & relevance scoring | Yes (for extract_snippets) |
|
||||
|
||||
## License
|
||||
|
||||
MIT
|
||||
Loading…
Add table
Add a link
Reference in a new issue