didi-lot1-ai/ai_platform/modules/web/API.md

# Web API Documentation

## Base URL

```
{BASE_URL}
```

- **Local development:** `http://localhost:51100`
- **Docker (internal):** `http://web-api:51100`
- **Production:** Use your configured hostname

## Authentication

Authentication is optional. If `WEB_API_TOKENS` is set, Bearer token authentication is required.

```
Authorization: Bearer <token>
```

If authentication is disabled (default), no Authorization header is needed.

## Endpoints

### Gather (Main Endpoint)

Execute the full evidence-gathering pipeline: search → fetch → evidence.

```
POST /v1/gather
```

#### Request Body

```json
{
  "claim": "The claim to verify",
  "search_queries": ["optional custom queries"],
  "max_search_results": 20,
  "site_allowlist": ["reuters.com"],
  "site_blocklist": ["spam-site.com"],
  "fetch_method": "auto",
  "auto_fallback": true,
  "extract_snippets": true,
  "max_evidence_items": 15,
  "dedupe": true,
  "timeout_seconds": 60.0
}
```

| Field | Type | Required | Default | Description |
|-------|------|----------|---------|-------------|
| `claim` | string | Yes | - | Claim to gather evidence for (10-1000 chars) |
| `search_queries` | array[string] | No | null | Custom search queries (auto-generated if not provided) |
| `max_search_results` | integer | No | 20 | Max search results (5-50) |
| `site_allowlist` | array[string] | No | null | Only search these domains |
| `site_blocklist` | array[string] | No | null | Exclude these domains |
| `fetch_method` | string | No | "auto" | "auto", "http", "browse", "vision" |
| `auto_fallback` | boolean | No | true | Escalate on fetch failure |
| `extract_snippets` | boolean | No | true | Use LLM for snippet extraction |
| `max_evidence_items` | integer | No | 15 | Max items in final pack (1-50) |
| `dedupe` | boolean | No | true | Deduplicate evidence |
| `parallel_fetches` | integer | No | 5 | Concurrent fetch operations (1-10) |
| `timeout_seconds` | number | No | 60.0 | Total pipeline timeout (10-300) |

#### Response

```json
{
  "request_id": "uuid",
  "claim": "The claim",
  "evidence": [
    {
      "url": "https://example.com/article",
      "title": "Article Title",
      "publisher": "example.com",
      "snippet": "Relevant excerpt...",
      "relevance_score": 0.8,
      "credibility_score": null,
      "published_at": "2024-01-15",
      "retrieved_at": "2025-01-20T10:00:00Z",
      "full_text_hash": "sha256...",
      "provenance": {
        "extraction_method": "http",
        "fallback_chain": []
      }
    }
  ],
  "evidence_stats": {
    "input_items": 10,
    "after_dedup": 8,
    "output_items": 8,
    "duplicates_removed": 2,
    "tokens_used": 1500
  },
  "stages": [
    {"stage": "search", "success": true, "items_processed": 10, "items_failed": 0, "duration_ms": 500.0},
    {"stage": "fetch", "success": true, "items_processed": 8, "items_failed": 2, "duration_ms": 5000.0},
    {"stage": "evidence", "success": true, "items_processed": 8, "items_failed": 0, "duration_ms": 3000.0}
  ],
  "total_urls_found": 10,
  "total_pages_fetched": 8,
  "total_evidence_items": 8,
  "execution_time_ms": 8500.0
}
```

#### Example

```bash
curl -X POST http://localhost:51100/v1/gather \
  -H "Content-Type: application/json" \
  -d '{
    "claim": "Romania had the highest economic growth in the EU in 2024",
    "max_search_results": 5,
    "extract_snippets": true,
    "max_evidence_items": 5
  }'
```

---

### Search

Execute web search queries.

```
POST /v1/search
```

#### Request Body

```json
{
  "queries": ["string"],
  "max_results": 10,
  "site_allowlist": ["string"],
  "site_blocklist": ["string"],
  "language": "en",
  "country": "US",
  "freshness": "month",
  "safe_search": "moderate"
}
```

| Field | Type | Required | Default | Description |
|-------|------|----------|---------|-------------|
| `queries` | array[string] | Yes | - | Search queries (1-10) |
| `max_results` | integer | No | 10 | Results per query (1-100) |
| `site_allowlist` | array[string] | No | null | Only include these domains |
| `site_blocklist` | array[string] | No | null | Exclude these domains |
| `language` | string | No | "en" | Search language (ISO 639-1) |
| `country` | string | No | "US" | Search country (ISO 3166-1) |
| `freshness` | string | No | null | Filter by age: day, week, month, year |
| `safe_search` | string | No | "moderate" | off, moderate, strict |

#### Response

```json
{
  "request_id": "uuid",
  "results": [
    {
      "query": "string",
      "url": "string",
      "title": "string",
      "snippet": "string",
      "rank": 1,
      "site": "string",
      "published_at": "string|null"
    }
  ],
  "total_results": 10,
  "execution_time_ms": 150.5,
  "queries_processed": 1
}
```

#### Example

```bash
curl -X POST http://localhost:51100/v1/search \
  -H "Content-Type: application/json" \
  -d '{
    "queries": ["climate change effects", "renewable energy"],
    "max_results": 5,
    "site_allowlist": ["reuters.com", "bbc.com"],
    "freshness": "month"
  }'
```

---

### Image Search

Search for images using SearXNG image search.

```
POST /v1/image-search
```

#### Request Body

```json
{
  "queries": ["solar eclipse"],
  "max_results": 50,
  "language": "en",
  "country": "US",
  "safe_search": "strict",
  "spellcheck": true
}
```

| Field | Type | Required | Default | Description |
|-------|------|----------|---------|-------------|
| `queries` | array[string] | Yes | - | Search queries (1-10) |
| `max_results` | integer | No | 50 | Results per query (1-200) |
| `language` | string | No | "en" | Search language (ISO 639-1) |
| `country` | string | No | "US" | Search country (ISO 3166-1) |
| `safe_search` | string | No | "strict" | off or strict |
| `spellcheck` | boolean | No | true | Enable spellcheck |

#### Response

```json
{
  "request_id": "uuid",
  "results": [
    {
      "query": "solar eclipse",
      "image_url": "https://example.com/eclipse.jpg",
      "thumbnail_url": "https://example.com/eclipse_thumb.jpg",
      "source_url": "https://example.com/article",
      "title": "Solar Eclipse Photo",
      "description": "A total solar eclipse captured in 2024",
      "width": 1920,
      "height": 1080,
      "publisher": "example.com",
      "rank": 1
    }
  ],
  "total_results": 50,
  "execution_time_ms": 200.5,
  "queries_processed": 1
}
```

#### Example

```bash
curl -X POST http://localhost:51100/v1/image-search \
  -H "Content-Type: application/json" \
  -d '{
    "queries": ["solar eclipse", "northern lights"],
    "max_results": 5,
    "safe_search": "strict"
  }'
```

---

### Fetch

Fetch and extract content from URLs with automatic fallback (HTTP → Playwright → Vision LLM).

```
POST /v1/fetch
```

#### Request Body

```json
{
  "urls": ["https://example.com/article"],
  "extract_text": true,
  "include_html": false,
  "extract_metadata": true,
  "auto_fallback": true,
  "method": "auto",
  "timeout_seconds": 30.0,
  "min_text_length": 200,
  "parallel_fetches": 5
}
```

| Field | Type | Required | Default | Description |
|-------|------|----------|---------|-------------|
| `urls` | array[string] | Yes | - | URLs to fetch (1-50) |
| `extract_text` | boolean | No | true | Extract main text content |
| `include_html` | boolean | No | false | Include raw HTML |
| `extract_metadata` | boolean | No | true | Extract metadata |
| `auto_fallback` | boolean | No | true | Auto-escalate on failure |
| `method` | string | No | "auto" | "auto", "http", "browse", "vision" |
| `timeout_seconds` | number | No | 30.0 | Timeout per URL (1-120) |
| `min_text_length` | integer | No | 200 | Min text before fallback |
| `parallel_fetches` | integer | No | 5 | Concurrent fetches (1-20) |

#### Response

```json
{
  "request_id": "uuid",
  "pages": [
    {
      "url": "https://example.com/article",
      "title": "Article Title",
      "text": "Extracted text content...",
      "text_hash": "sha256...",
      "extraction_method": "http",
      "fallback_chain": [],
      "retrieved_at": "2025-01-20T10:00:00Z",
      "extraction_time_ms": 500.0,
      "status_code": 200,
      "content_type": "text/html"
    }
  ],
  "total_fetched": 1,
  "total_failed": 0,
  "execution_time_ms": 600.0,
  "failed_urls": []
}
```

#### Example

```bash
curl -X POST http://localhost:51100/v1/fetch \
  -H "Content-Type: application/json" \
  -d '{
    "urls": ["https://en.wikipedia.org/wiki/Climate_change"],
    "method": "auto",
    "auto_fallback": true
  }'
```

---

### Info

Get component information for service catalog.

```
GET /v1/info
```

#### Response

Returns complete metadata about this service including resource information, available functions, and their schemas. Used by the catalog-api for service discovery.

#### Example

```bash
curl http://localhost:51100/v1/info
```

---

### Health Check

Check service health and provider status.

```
GET /health
```

#### Response

```json
{
  "status": "healthy",
  "providers": [
    {
      "name": "searxng",
      "healthy": true,
      "message": null
    }
  ]
}
```

| Status | Description |
|--------|-------------|
| `healthy` | All providers operational |
| `degraded` | Some providers unavailable |
| `unhealthy` | All providers down |

---

### Readiness Probe

Check if service is ready to accept requests.

```
GET /ready
```

#### Response

```json
{
  "ready": true
}
```

---

## Error Responses

### Error Format

```json
{
  "detail": {
    "error": "error_code",
    "message": "Human readable message",
    "provider": "searxng"
  }
}
```

### HTTP Status Codes

| Code | Error | Description |
|------|-------|-------------|
| 400 | `validation_error` | Invalid request body |
| 401 | `authentication_required` | Missing or invalid auth token |
| 422 | `validation_error` | Request validation failed |
| 429 | `rate_limit_exceeded` | Too many requests |
| 502 | `provider_error` | Search provider error |
| 503 | `service_unavailable` | Concurrency limit reached |
| 504 | `timeout` | Request timeout |

---

## Request Headers

| Header | Required | Description |
|--------|----------|-------------|
| `Content-Type` | Yes | Must be `application/json` |
| `Authorization` | Conditional | Bearer token (if auth enabled) |
| `X-Request-ID` | No | Custom request ID for tracing |

---

## Rate Limiting

The API implements token bucket rate limiting:

- **Default:** 10 requests/second with burst of 20
- **429 Response:** Includes `Retry-After` header

**Note:** Rate limiting is per-process. In multi-replica deployments, use external rate limiting.

---

## SDK Example (Python)

```python
import httpx

async def search(query: str, base_url: str = "http://localhost:51100"):
    async with httpx.AsyncClient() as client:
        response = await client.post(
            f"{base_url}/v1/search",
            json={
                "queries": [query],
                "max_results": 10,
            },
        )
        response.raise_for_status()
        return response.json()

# Usage
results = await search("climate change")
for r in results["results"]:
    print(f"{r['title']}: {r['url']}")
```