didi-lot1-ai/ai_platform/modules/cloak

cloak — stealth-Chromium scraping service

Standalone HTTP service that scrapes Google / Bing / DuckDuckGo SERPs through a warm pool of CloakBrowser (patched stealth Chromium) instances.

Designed as tier-3 search fallback for the AI platform web module: when SearXNG + the paid rotation return thin results (often the case for very niche or recent queries), cloak provides results scraped directly from Google, Bing and DuckDuckGo's HTML SERPs.

Why a separate service?

• Isolated lifecycle. Browser pool restarts don't take down the rest of the AI platform.
• Bounded footprint. A small fixed pool (default 3 instances ≈ 1.2 GB RAM) versus N pools spreading across every web worker.
• Same deployment pattern as audio, embeddings, video-analysis etc.

API

POST /v1/search
{
  "queries": ["BNR confiscare conturi 10000 euro"],
  "engines": ["google", "bing", "ddg"],
  "max_results_per_engine": 10,
  "language": "ro"            // optional hint
}

200 OK
{
  "results": [
    {"url": "...", "title": "...", "snippet": "...",
     "engine": "google", "query": "...", "rank": 1},
    ...
  ],
  "stats": [
    {"engine": "google", "query": "...", "results_count": 10,
     "blocked": false, "captcha": false, "elapsed_ms": 2750, "error": null},
    ...
  ],
  "total_elapsed_ms": 2900
}

GET /health
{
  "status": "healthy" | "degraded" | "unhealthy",
  "pool_size": 3, "pool_available": 3, "version": "0.1.0"
}

Auth is optional via Authorization: Bearer <CLOAK_AUTH_TOKEN>; when the env var is unset (default), all requests are accepted (intra-cluster service — should never be reachable from the internet).

Configuration (env)

Variable	Default	Notes
CLOAK_HOST	0.0.0.0	Bind address
CLOAK_PORT	8770	HTTP port
CLOAK_POOL_SIZE	3	Number of warm browsers (~400 MB each)
CLOAK_SEARCH_TIMEOUT_SEC	20	Hard timeout for entire /v1/search call
CLOAK_PAGE_TIMEOUT_MS	18000	Per-engine page load timeout
CLOAK_MAX_ENGINES	3	Cap on engines per request
CLOAK_MAX_QUERIES	5	Cap on queries per request
CLOAK_DEFAULT_MAX_RESULTS	10	Default per-engine result cap
CLOAK_MAX_RESULTS_CAP	30	Hard cap regardless of input
CLOAK_ENGINE_MIN_INTERVAL_MS	200	Throttle between successive scrapes per engine
CLOAK_HUMANIZE	false	Human-like mouse/keyboard timing (slower, better for behavioral anti-bot)
CLOAK_AUTH_TOKEN	""	Optional bearer token. Empty = no auth
CLOAK_LOG_LEVEL	INFO	DEBUG, INFO, WARNING, ERROR

Deploy

cd ai_platform/modules/cloak/deploy
docker compose up -d --build
docker logs -f didiAI-cloak
# Health
curl -s http://127.0.0.1:8770/health | jq
# Smoke
curl -sS -X POST http://127.0.0.1:8770/v1/search \
  -H 'Content-Type: application/json' \
  -d '{"queries":["NYTimes climate report 2026"],"engines":["google","bing","ddg"]}' | jq '.stats'

Where it fits

agent-v3 / claims-verifier
        │
        ▼
didiAI-web-api  ──► SearXNG (free, local)        [tier 1 — always]
                   + Brave/Tavily/etc rotation   [tier 2 — one paid per call]
                   + cloak (this service)         [tier 3 — only if tier 1+2 thin]

The web module's orchestrator decides when to invoke cloak based on the number of unique results returned from tiers 1+2. agent-v3 and didi-framework do not call cloak directly.

Operational notes

• Selectors break. Google rotates its result-DOM classes every 6–12 months. The scraper has fallback selectors but the primary path will eventually need re-tuning. Monitor stats.blocked / stats.results_count over time.
• Rate limits. No formal limit on the SERP endpoints, but bursts trigger captcha. Default CLOAK_ENGINE_MIN_INTERVAL_MS=200 paces requests; tune up if you see captcha rates rise.
• CPU/RAM. Each browser instance uses ~400 MB RAM and is single-CPU for most of a page load. The default CLOAK_POOL_SIZE=3 is sized for didi12 ≤ 5k scrape ops/day; raise to 5–8 if pool starves the request queue.