# cloak — stealth-Chromium scraping service

Standalone HTTP service that scrapes Google / Bing / DuckDuckGo SERPs through a
warm pool of CloakBrowser (patched stealth Chromium) instances.

Designed as **tier-3 search fallback** for the AI platform `web` module: when
SearXNG + the paid rotation return thin results (often the case for very niche
or recent queries), `cloak` provides results scraped directly from Google,
Bing and DuckDuckGo's HTML SERPs.

## Why a separate service?

- **Isolated lifecycle.** Browser pool restarts don't take down the rest of the
  AI platform.
- **Bounded footprint.** A small fixed pool (default 3 instances ≈ 1.2 GB RAM)
  versus N pools spreading across every web worker.
- **Same deployment pattern** as `audio`, `embeddings`, `video-analysis` etc.

## API

```
POST /v1/search
{
  "queries": ["BNR confiscare conturi 10000 euro"],
  "engines": ["google", "bing", "ddg"],
  "max_results_per_engine": 10,
  "language": "ro"            // optional hint
}

200 OK
{
  "results": [
    {"url": "...", "title": "...", "snippet": "...",
     "engine": "google", "query": "...", "rank": 1},
    ...
  ],
  "stats": [
    {"engine": "google", "query": "...", "results_count": 10,
     "blocked": false, "captcha": false, "elapsed_ms": 2750, "error": null},
    ...
  ],
  "total_elapsed_ms": 2900
}

GET /health
{
  "status": "healthy" | "degraded" | "unhealthy",
  "pool_size": 3, "pool_available": 3, "version": "0.1.0"
}
```

Auth is optional via `Authorization: Bearer <CLOAK_AUTH_TOKEN>`; when the env
var is unset (default), all requests are accepted (intra-cluster service —
should never be reachable from the internet).

## Configuration (env)

| Variable | Default | Notes |
|---|---|---|
| `CLOAK_HOST` | `0.0.0.0` | Bind address |
| `CLOAK_PORT` | `8770` | HTTP port |
| `CLOAK_POOL_SIZE` | `3` | Number of warm browsers (~400 MB each) |
| `CLOAK_SEARCH_TIMEOUT_SEC` | `20` | Hard timeout for entire `/v1/search` call |
| `CLOAK_PAGE_TIMEOUT_MS` | `18000` | Per-engine page load timeout |
| `CLOAK_MAX_ENGINES` | `3` | Cap on engines per request |
| `CLOAK_MAX_QUERIES` | `5` | Cap on queries per request |
| `CLOAK_DEFAULT_MAX_RESULTS` | `10` | Default per-engine result cap |
| `CLOAK_MAX_RESULTS_CAP` | `30` | Hard cap regardless of input |
| `CLOAK_ENGINE_MIN_INTERVAL_MS` | `200` | Throttle between successive scrapes per engine |
| `CLOAK_HUMANIZE` | `false` | Human-like mouse/keyboard timing (slower, better for behavioral anti-bot) |
| `CLOAK_AUTH_TOKEN` | `""` | Optional bearer token. Empty = no auth |
| `CLOAK_LOG_LEVEL` | `INFO` | `DEBUG`, `INFO`, `WARNING`, `ERROR` |

## Deploy

```bash
cd ai_platform/modules/cloak/deploy
docker compose up -d --build
docker logs -f didiAI-cloak
# Health
curl -s http://127.0.0.1:8770/health | jq
# Smoke
curl -sS -X POST http://127.0.0.1:8770/v1/search \
  -H 'Content-Type: application/json' \
  -d '{"queries":["NYTimes climate report 2026"],"engines":["google","bing","ddg"]}' | jq '.stats'
```

## Where it fits

```
agent-v3 / claims-verifier
        │
        ▼
didiAI-web-api  ──► SearXNG (free, local)        [tier 1 — always]
                   + Brave/Tavily/etc rotation   [tier 2 — one paid per call]
                   + cloak (this service)         [tier 3 — only if tier 1+2 thin]
```

The web module's orchestrator decides when to invoke `cloak` based on the
number of unique results returned from tiers 1+2. agent-v3 and didi-framework
do not call `cloak` directly.

## Operational notes

- **Selectors break.** Google rotates its result-DOM classes every 6–12 months.
  The scraper has fallback selectors but the primary path will eventually need
  re-tuning. Monitor `stats.blocked` / `stats.results_count` over time.
- **Rate limits.** No formal limit on the SERP endpoints, but bursts trigger
  captcha. Default `CLOAK_ENGINE_MIN_INTERVAL_MS=200` paces requests; tune up
  if you see captcha rates rise.
- **CPU/RAM.** Each browser instance uses ~400 MB RAM and is single-CPU for
  most of a page load. The default `CLOAK_POOL_SIZE=3` is sized for didi12 ≤ 5k
  scrape ops/day; raise to 5–8 if pool starves the request queue.