Livrare LOT 1 - Didi

ai_platform/modules/cloak/INDEX.md

@@ -0,0 +1,56 @@
+# cloak — module index
+
+## Purpose
+
+HTTP wrapper around CloakBrowser. Scrapes Google / Bing / DDG SERPs and exposes
+the parsed organic results as JSON. Used as the third-tier fallback for the AI
+platform `web` module when SearXNG + paid rotation return thin results.
+
+## Files
+
+```
+cloak/
+├── pyproject.toml            FastAPI + uvicorn + cloakbrowser deps
+├── README.md                 User-facing docs
+├── INDEX.md                  This file (module map)
+├── src/cloak/
+│   ├── __init__.py
+│   ├── config.py             Env-driven pydantic settings (CLOAK_* prefix)
+│   ├── schemas.py            Request/Response/Stats pydantic models
+│   ├── scraper.py            Per-engine HTML scrapers (google/bing/ddg)
+│   ├── browser_pool.py       Bounded async pool of CloakBrowser instances
+│   └── server.py             FastAPI app — POST /v1/search, GET /health
+├── tests/                    pytest test suite
+└── deploy/
+    ├── Dockerfile            FROM cloakhq/cloakbrowser:latest + FastAPI
+    └── docker-compose.yml    didiAI-cloak on didi-network, port 8770
+```
+
+## External contracts
+
+| Surface | Path | Method |
+|---|---|---|
+| Search | `/v1/search` | POST |
+| Health | `/health` | GET |
+
+Both reachable inside the cluster at `http://didiAI-cloak:8770/`. Host-port
+`127.0.0.1:8770` is exposed only for local debugging on didi12.
+
+## State
+
+Stateless. No database, no Redis. The browser pool is in-process memory.
+Cold-start (warm 3 browsers) ≈ 8–12 s; from then on each search is 2–4 s
+end-to-end.
+
+## Dependencies
+
+- `cloakhq/cloakbrowser:latest` Docker base image (bundles stealth Chromium + Xvfb)
+- Outbound TCP to `google.com`, `bing.com`, `html.duckduckgo.com`
+
+## Where it's consumed
+
+- `ai_platform/modules/web/src/web/search/cloak.py` (CloakHTTPClient) — pending
+- `ai_platform/modules/web/src/web/orchestrator.py::_run_search_stage` — pending tier-3 hook
+
+Backend services (`agent-v3`, `didi-framework`, admin-dashboard) do NOT call
+this service directly.

ai_platform/modules/cloak/README.md

@@ -0,0 +1,113 @@
+# cloak — stealth-Chromium scraping service
+
+Standalone HTTP service that scrapes Google / Bing / DuckDuckGo SERPs through a
+warm pool of CloakBrowser (patched stealth Chromium) instances.
+
+Designed as **tier-3 search fallback** for the AI platform `web` module: when
+SearXNG + the paid rotation return thin results (often the case for very niche
+or recent queries), `cloak` provides results scraped directly from Google,
+Bing and DuckDuckGo's HTML SERPs.
+
+## Why a separate service?
+
+- **Isolated lifecycle.** Browser pool restarts don't take down the rest of the
+  AI platform.
+- **Bounded footprint.** A small fixed pool (default 3 instances ≈ 1.2 GB RAM)
+  versus N pools spreading across every web worker.
+- **Same deployment pattern** as `audio`, `embeddings`, `video-analysis` etc.
+
+## API
+
+```
+POST /v1/search
+{
+  "queries": ["BNR confiscare conturi 10000 euro"],
+  "engines": ["google", "bing", "ddg"],
+  "max_results_per_engine": 10,
+  "language": "ro"            // optional hint
+}
+
+200 OK
+{
+  "results": [
+    {"url": "...", "title": "...", "snippet": "...",
+     "engine": "google", "query": "...", "rank": 1},
+    ...
+  ],
+  "stats": [
+    {"engine": "google", "query": "...", "results_count": 10,
+     "blocked": false, "captcha": false, "elapsed_ms": 2750, "error": null},
+    ...
+  ],
+  "total_elapsed_ms": 2900
+}
+
+GET /health
+{
+  "status": "healthy" | "degraded" | "unhealthy",
+  "pool_size": 3, "pool_available": 3, "version": "0.1.0"
+}
+```
+
+Auth is optional via `Authorization: Bearer <CLOAK_AUTH_TOKEN>`; when the env
+var is unset (default), all requests are accepted (intra-cluster service —
+should never be reachable from the internet).
+
+## Configuration (env)
+
+| Variable | Default | Notes |
+|---|---|---|
+| `CLOAK_HOST` | `0.0.0.0` | Bind address |
+| `CLOAK_PORT` | `8770` | HTTP port |
+| `CLOAK_POOL_SIZE` | `3` | Number of warm browsers (~400 MB each) |
+| `CLOAK_SEARCH_TIMEOUT_SEC` | `20` | Hard timeout for entire `/v1/search` call |
+| `CLOAK_PAGE_TIMEOUT_MS` | `18000` | Per-engine page load timeout |
+| `CLOAK_MAX_ENGINES` | `3` | Cap on engines per request |
+| `CLOAK_MAX_QUERIES` | `5` | Cap on queries per request |
+| `CLOAK_DEFAULT_MAX_RESULTS` | `10` | Default per-engine result cap |
+| `CLOAK_MAX_RESULTS_CAP` | `30` | Hard cap regardless of input |
+| `CLOAK_ENGINE_MIN_INTERVAL_MS` | `200` | Throttle between successive scrapes per engine |
+| `CLOAK_HUMANIZE` | `false` | Human-like mouse/keyboard timing (slower, better for behavioral anti-bot) |
+| `CLOAK_AUTH_TOKEN` | `""` | Optional bearer token. Empty = no auth |
+| `CLOAK_LOG_LEVEL` | `INFO` | `DEBUG`, `INFO`, `WARNING`, `ERROR` |
+
+## Deploy
+
+```bash
+cd ai_platform/modules/cloak/deploy
+docker compose up -d --build
+docker logs -f didiAI-cloak
+# Health
+curl -s http://127.0.0.1:8770/health | jq
+# Smoke
+curl -sS -X POST http://127.0.0.1:8770/v1/search \
+  -H 'Content-Type: application/json' \
+  -d '{"queries":["NYTimes climate report 2026"],"engines":["google","bing","ddg"]}' | jq '.stats'
+```
+
+## Where it fits
+
+```
+agent-v3 / claims-verifier
+        │
+        ▼
+didiAI-web-api  ──► SearXNG (free, local)        [tier 1 — always]
+                   + Brave/Tavily/etc rotation   [tier 2 — one paid per call]
+                   + cloak (this service)         [tier 3 — only if tier 1+2 thin]
+```
+
+The web module's orchestrator decides when to invoke `cloak` based on the
+number of unique results returned from tiers 1+2. agent-v3 and didi-framework
+do not call `cloak` directly.
+
+## Operational notes
+
+- **Selectors break.** Google rotates its result-DOM classes every 6–12 months.
+  The scraper has fallback selectors but the primary path will eventually need
+  re-tuning. Monitor `stats.blocked` / `stats.results_count` over time.
+- **Rate limits.** No formal limit on the SERP endpoints, but bursts trigger
+  captcha. Default `CLOAK_ENGINE_MIN_INTERVAL_MS=200` paces requests; tune up
+  if you see captcha rates rise.
+- **CPU/RAM.** Each browser instance uses ~400 MB RAM and is single-CPU for
+  most of a page load. The default `CLOAK_POOL_SIZE=3` is sized for didi12 ≤ 5k
+  scrape ops/day; raise to 5–8 if pool starves the request queue.

ai_platform/modules/cloak/deploy/Dockerfile

@@ -0,0 +1,22 @@
+# cloak service — FastAPI wrapper around CloakBrowser
+#
+# Base image already has stealth Chromium, Xvfb, system fonts, Node 20.
+# We just add the FastAPI app + uvicorn on top.
+
+FROM cloakhq/cloakbrowser:latest
+
+WORKDIR /app
+
+COPY pyproject.toml ./
+COPY src/ ./src/
+RUN pip install --no-cache-dir \
+    "fastapi>=0.115.0" \
+    "uvicorn[standard]>=0.32.0" \
+    "pydantic>=2.0" \
+    "pydantic-settings>=2.0" \
+    && pip install --no-cache-dir -e .
+
+EXPOSE 8770
+
+# Base image already runs Xvfb via /entrypoint.sh — we override the CMD only.
+CMD ["python", "-m", "cloak.server"]

ai_platform/modules/cloak/deploy/docker-compose.yml

@@ -0,0 +1,40 @@
+services:
+  cloak:
+    build:
+      context: ..
+      dockerfile: deploy/Dockerfile
+    image: didi-ai/cloak:latest
+    container_name: didiAI-cloak
+    restart: unless-stopped
+    networks:
+      - didi-network
+    environment:
+      CLOAK_HOST: 0.0.0.0
+      CLOAK_PORT: 8770
+      CLOAK_POOL_SIZE: ${CLOAK_POOL_SIZE:-3}
+      CLOAK_SEARCH_TIMEOUT_SEC: ${CLOAK_SEARCH_TIMEOUT_SEC:-20}
+      CLOAK_PAGE_TIMEOUT_MS: ${CLOAK_PAGE_TIMEOUT_MS:-18000}
+      CLOAK_MAX_ENGINES: ${CLOAK_MAX_ENGINES:-3}
+      CLOAK_MAX_QUERIES: ${CLOAK_MAX_QUERIES:-5}
+      CLOAK_DEFAULT_MAX_RESULTS: ${CLOAK_DEFAULT_MAX_RESULTS:-10}
+      CLOAK_MAX_RESULTS_CAP: ${CLOAK_MAX_RESULTS_CAP:-30}
+      CLOAK_ENGINE_MIN_INTERVAL_MS: ${CLOAK_ENGINE_MIN_INTERVAL_MS:-200}
+      CLOAK_HUMANIZE: ${CLOAK_HUMANIZE:-false}
+      CLOAK_AUTH_TOKEN: ${CLOAK_AUTH_TOKEN:-}
+      CLOAK_LOG_LEVEL: ${CLOAK_LOG_LEVEL:-INFO}
+      DISPLAY: ":99"
+    # Bind only to localhost on the host — intra-cluster access via DNS name
+    # `didiAI-cloak` on didi-network. Public exposure is undesirable (scraping
+    # endpoint should never be reachable from the internet).
+    ports:
+      - "127.0.0.1:8770:8770"
+    healthcheck:
+      test: ["CMD-SHELL", "python -c \"import urllib.request,sys; r=urllib.request.urlopen('http://127.0.0.1:8770/health', timeout=3); sys.exit(0 if r.status==200 else 1)\""]
+      interval: 30s
+      timeout: 5s
+      retries: 3
+      start_period: 30s
+
+networks:
+  didi-network:
+    external: true

ai_platform/modules/cloak/pyproject.toml

@@ -0,0 +1,33 @@
+[project]
+name = "cloak"
+version = "0.1.0"
+description = "Stealth Chromium scraping service — Google/Bing/DDG search via CloakBrowser, exposed as HTTP API."
+requires-python = ">=3.10"
+dependencies = [
+  "fastapi>=0.115.0",
+  "uvicorn[standard]>=0.32.0",
+  "pydantic>=2.0",
+  "pydantic-settings>=2.0",
+  # cloakbrowser is provided by the base Docker image (cloakhq/cloakbrowser:latest).
+  # Listed in optional-dependencies for local dev only.
+]
+
+[project.optional-dependencies]
+dev = [
+  "pytest>=8.0",
+  "pytest-asyncio>=0.24",
+  "httpx>=0.27.0",
+  "ruff>=0.8",
+  "cloakbrowser",
+]
+
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[tool.hatch.build.targets.wheel]
+packages = ["src/cloak"]
+
+[tool.ruff]
+line-length = 100
+target-version = "py310"

ai_platform/modules/cloak/src/cloak/__init__.py

@@ -0,0 +1,3 @@
+"""Cloak — stealth scraping service for Google / Bing / DuckDuckGo search results."""
+
+__version__ = "0.1.0"

ai_platform/modules/cloak/src/cloak/browser_pool.py

@@ -0,0 +1,96 @@
+"""Async pool of CloakBrowser instances kept warm.
+
+The pool is a bounded asyncio.Queue of running Browser objects. acquire() and
+release() check out an instance for the lifetime of one search call. If a
+browser raises during use, we discard it and lazily replace it on the next
+acquire so a single bad page doesn't permanently shrink the pool.
+"""
+from __future__ import annotations
+
+import asyncio
+import logging
+from contextlib import asynccontextmanager
+
+logger = logging.getLogger("cloak.pool")
+
+
+class BrowserPool:
+    """Bounded pool of CloakBrowser instances."""
+
+    def __init__(self, size: int, *, humanize: bool):
+        self._size = size
+        self._humanize = humanize
+        self._queue: asyncio.Queue = asyncio.Queue(maxsize=size)
+        self._created = 0
+        self._lock = asyncio.Lock()
+        self._closed = False
+
+    @property
+    def size(self) -> int:
+        return self._size
+
+    @property
+    def available(self) -> int:
+        return self._queue.qsize()
+
+    async def _create(self):
+        """Create one fresh browser. Imported lazily so tests can stub it."""
+        from cloakbrowser import launch_async  # type: ignore
+
+        browser = await launch_async(headless=True, humanize=self._humanize)
+        self._created += 1
+        logger.info("BrowserPool: created instance %d/%d", self._created, self._size)
+        return browser
+
+    async def start(self) -> None:
+        """Pre-create all instances up-front so first calls don't pay launch cost."""
+        async with self._lock:
+            for _ in range(self._size):
+                b = await self._create()
+                await self._queue.put(b)
+        logger.info("BrowserPool: warmed up with %d instances", self._size)
+
+    async def stop(self) -> None:
+        """Close all browsers. Safe to call multiple times."""
+        self._closed = True
+        while not self._queue.empty():
+            try:
+                b = self._queue.get_nowait()
+            except asyncio.QueueEmpty:
+                break
+            try:
+                await b.close()
+            except Exception as e:  # noqa: BLE001
+                logger.warning("BrowserPool: close failed: %s", e)
+        logger.info("BrowserPool: stopped")
+
+    @asynccontextmanager
+    async def acquire(self):
+        """Check out one browser for the duration of the `async with` block.
+
+        If the browser dies during use (any exception inside the block), we
+        close it and replace with a fresh one on release.
+        """
+        if self._closed:
+            raise RuntimeError("Pool is closed")
+        browser = await self._queue.get()
+        broken = False
+        try:
+            yield browser
+        except Exception:
+            broken = True
+            raise
+        finally:
+            if broken:
+                try:
+                    await browser.close()
+                except Exception:
+                    pass
+                try:
+                    fresh = await self._create()
+                    await self._queue.put(fresh)
+                except Exception as e:  # noqa: BLE001
+                    logger.error("BrowserPool: failed to replace dead browser: %s", e)
+                    # Pool shrinks until next successful recreate
+            else:
+                await self._queue.put(browser)

ai_platform/modules/cloak/src/cloak/config.py

@@ -0,0 +1,50 @@
+"""Cloak service configuration — env-driven via pydantic-settings."""
+from __future__ import annotations
+
+from pydantic import Field
+from pydantic_settings import BaseSettings, SettingsConfigDict
+
+
+class CloakSettings(BaseSettings):
+    """Runtime configuration for the cloak service."""
+
+    model_config = SettingsConfigDict(env_prefix="CLOAK_", case_sensitive=False)
+
+    # HTTP server
+    host: str = "0.0.0.0"
+    port: int = 8770
+
+    # Browser pool — number of CloakBrowser instances kept warm.
+    # Each instance uses ~400 MB RAM. Default 3 keeps ~1.2 GB footprint
+    # which fits comfortably alongside the other AI platform services.
+    pool_size: int = 3
+
+    # Per-search timeout (entire search across engines).
+    search_timeout_sec: int = 20
+
+    # Per-engine page load timeout.
+    page_timeout_ms: int = 18000
+
+    # Max engines per call (cap to avoid abuse).
+    max_engines: int = 3
+
+    # Max queries per call (cap to avoid abuse).
+    max_queries: int = 5
+
+    # Default max results per (engine, query).
+    default_max_results: int = 10
+    max_results_cap: int = 30
+
+    # Throttling — minimum delay between successive scrapes on the same engine.
+    # Helps avoid tripping rate-limits when called in bursts.
+    engine_min_interval_ms: int = 200
+
+    # Humanize input (mouse/keyboard timing) — slows requests slightly but
+    # improves bot-detection scores. Default off for bulk throughput.
+    humanize: bool = False
+
+    # Optional shared bearer token. Empty = no auth (intra-cluster only).
+    auth_token: str = ""
+
+    # Log level
+    log_level: str = "INFO"

ai_platform/modules/cloak/src/cloak/schemas.py

@@ -0,0 +1,81 @@
+"""Pydantic schemas — request / response contracts for the cloak service."""
+from __future__ import annotations
+
+from typing import Literal
+
+from pydantic import BaseModel, ConfigDict, Field
+
+Engine = Literal["google", "bing", "ddg"]
+
+
+class SearchRequest(BaseModel):
+    """One search call across one or more engines."""
+
+    model_config = ConfigDict(extra="forbid")
+
+    queries: list[str] = Field(
+        ...,
+        min_length=1,
+        description="One or more text queries; each is scraped on every engine in `engines`.",
+    )
+    engines: list[Engine] = Field(
+        default=["google", "bing", "ddg"],
+        description="Search engines to scrape. Order is independent (parallel execution).",
+    )
+    max_results_per_engine: int = Field(
+        default=10,
+        ge=1,
+        description="Max organic results returned per (engine, query).",
+    )
+    language: str | None = Field(
+        default=None,
+        description="Preferred language hint (e.g. 'en', 'ro'). Engine-specific behavior.",
+    )
+
+
+class SearchResult(BaseModel):
+    """One organic search result."""
+
+    model_config = ConfigDict(extra="forbid")
+
+    url: str
+    title: str
+    snippet: str = ""
+    engine: Engine
+    query: str
+    rank: int = Field(..., description="1-based rank within the engine's result list.")
+
+
+class EngineStats(BaseModel):
+    """Per-engine breakdown for diagnostics."""
+
+    model_config = ConfigDict(extra="forbid")
+
+    engine: Engine
+    query: str
+    results_count: int
+    blocked: bool = False
+    captcha: bool = False
+    elapsed_ms: int
+    error: str | None = None
+
+
+class SearchResponse(BaseModel):
+    """Aggregate response across all (engine × query) pairs."""
+
+    model_config = ConfigDict(extra="forbid")
+
+    results: list[SearchResult]
+    stats: list[EngineStats]
+    total_elapsed_ms: int
+
+
+class HealthResponse(BaseModel):
+    """Health-probe payload."""
+
+    model_config = ConfigDict(extra="forbid")
+
+    status: Literal["healthy", "degraded", "unhealthy"]
+    pool_size: int
+    pool_available: int
+    version: str

ai_platform/modules/cloak/src/cloak/scraper.py

@@ -0,0 +1,254 @@
+"""HTML scrapers for Google / Bing / DuckDuckGo search result pages.
+
+Each scraper accepts a Playwright Page (CloakBrowser-backed) and returns a
+list of (url, title, snippet) tuples plus block/captcha flags. Selectors are
+intentionally redundant — Google in particular rotates result-DOM classes
+periodically. If both primary and fallback selectors fail, returns empty list.
+"""
+from __future__ import annotations
+
+import logging
+from dataclasses import dataclass
+from urllib.parse import quote_plus, unquote, urlparse, parse_qs
+
+logger = logging.getLogger("cloak.scraper")
+
+
+@dataclass
+class ScrapeOutput:
+    results: list[tuple[str, str, str]]  # (url, title, snippet)
+    blocked: bool = False
+    captcha: bool = False
+    error: str | None = None
+
+
+# ─── URLs ─────────────────────────────────────────────────────────────────────
+
+def search_url(engine: str, query: str, language: str | None = None) -> str:
+    q = quote_plus(query)
+    if engine == "google":
+        hl = language or "en"
+        return f"https://www.google.com/search?q={q}&hl={hl}&num=20"
+    if engine == "bing":
+        return f"https://www.bing.com/search?q={q}&count=20"
+    if engine == "ddg":
+        # HTML endpoint is more scraper-friendly than the JS-driven SPA.
+        return f"https://html.duckduckgo.com/html/?q={q}"
+    raise ValueError(f"Unsupported engine: {engine}")
+
+
+# ─── Block / captcha detection ────────────────────────────────────────────────
+
+def _detect_block(html: str, title: str) -> tuple[bool, bool]:
+    """Return (blocked, captcha) flags by inspecting page content."""
+    t = title.lower()
+    h = html.lower()
+    captcha_markers = [
+        "unusual traffic",
+        "before you continue to google search",
+        "/sorry/index",
+        "recaptcha",
+        "are you a robot",
+    ]
+    block_markers = [
+        "access denied",
+        "<title>just a moment...</title>",
+        "checking your browser",
+        "blocked",
+    ]
+    captcha = any(m in h for m in captcha_markers) or "sorry" in t
+    blocked = False
+    # Only flag block if we ALSO see no result containers (avoids false positives
+    # on pages that legitimately mention "blocked" in editorial content).
+    return blocked, captcha
+
+
+def _clean_google_redirect(href: str) -> str:
+    """Google sometimes wraps result URLs in /url?q=...&sa=...; strip it."""
+    if href.startswith("/url?"):
+        try:
+            qs = parse_qs(urlparse("http://x" + href).query)
+            target = qs.get("q", [None])[0]
+            if target:
+                return unquote(target)
+        except Exception:
+            pass
+    return href
+
+
+# ─── Per-engine scrapers ──────────────────────────────────────────────────────
+
+async def _scrape_google(page, max_results: int) -> ScrapeOutput:
+    """Google organic results. Multiple selector strategies for resilience.
+
+    Captcha detection happens AFTER extraction: if we got 0 results AND markers
+    are present, it's a real captcha. The "before you continue" cookies banner
+    and "/sorry/index" footer links are present on every normal Google SERP,
+    so checking markers up front gives massive false-positives.
+    """
+    selectors_to_try = [
+        # Modern (2025-2026) class names
+        "div.MjjYud:has(a h3)",
+        # Legacy: any div containing an h3 inside a link
+        "div.g a:has(h3)",
+        # Last resort
+        "a:has(h3)",
+    ]
+
+    title = await page.title()
+    html = await page.content()
+
+    for sel in selectors_to_try:
+        try:
+            locator = page.locator(sel)
+            count = await locator.count()
+            if count == 0:
+                continue
+            results: list[tuple[str, str, str]] = []
+            for i in range(min(count, max_results * 2)):
+                el = locator.nth(i)
+                try:
+                    # In modern Google each result has h3 inside an anchor.
+                    a = el if (await el.evaluate("e => e.tagName")) == "A" else el.locator("a:has(h3)").first
+                    href = await a.get_attribute("href")
+                    if not href:
+                        continue
+                    href = _clean_google_redirect(href)
+                    if not href.startswith("http"):
+                        continue
+                    try:
+                        h3 = a.locator("h3").first
+                        title_text = (await h3.inner_text()).strip()
+                    except Exception:
+                        title_text = (await a.inner_text()).strip()[:120]
+                    if not title_text:
+                        continue
+                    # Snippet — best-effort, optional
+                    snippet = ""
+                    try:
+                        # Look for any sibling or descendant span with text content.
+                        snippet = await el.evaluate(
+                            "e => { const t = e.innerText || ''; const lines = t.split('\\n'); return lines.slice(1, 4).join(' '); }"
+                        )
+                        snippet = (snippet or "").strip()[:300]
+                    except Exception:
+                        pass
+                    results.append((href, title_text, snippet))
+                    if len(results) >= max_results:
+                        break
+                except Exception:
+                    continue
+            if results:
+                return ScrapeOutput(results=results)
+        except Exception as e:
+            logger.debug("Google selector '%s' failed: %s", sel, e)
+
+    # Zero results extracted → now check markers to distinguish captcha vs DOM rotation
+    _, captcha = _detect_block(html, title)
+    if captcha or "/sorry/index" in html or "unusual traffic" in html.lower():
+        return ScrapeOutput(results=[], captcha=True, error="captcha_detected_after_zero_results")
+    return ScrapeOutput(results=[], blocked=True, error="no_results_no_selectors_matched")
+
+
+async def _scrape_bing(page, max_results: int) -> ScrapeOutput:
+    """Bing organic results — `li.b_algo` is stable since ~2010.
+
+    Same captcha-after-extraction strategy as Google.
+    """
+    title = await page.title()
+    html = await page.content()
+
+    try:
+        locator = page.locator("li.b_algo")
+        count = await locator.count()
+        results: list[tuple[str, str, str]] = []
+        for i in range(min(count, max_results)):
+            li = locator.nth(i)
+            try:
+                a = li.locator("h2 a").first
+                href = await a.get_attribute("href")
+                title_text = (await a.inner_text()).strip()
+                snippet = ""
+                try:
+                    snippet_locator = li.locator(".b_caption p, .b_lineclamp2, .b_paractl")
+                    if await snippet_locator.count() > 0:
+                        snippet = (await snippet_locator.first.inner_text()).strip()[:300]
+                except Exception:
+                    pass
+                if href and title_text:
+                    results.append((href, title_text, snippet))
+            except Exception:
+                continue
+        if results:
+            return ScrapeOutput(results=results)
+        # Zero results — check captcha markers
+        _, captcha = _detect_block(html, title)
+        if captcha:
+            return ScrapeOutput(results=[], captcha=True, error="captcha_after_zero_results")
+        return ScrapeOutput(results=[], blocked=True, error="no_li_b_algo_or_empty")
+    except Exception as e:
+        return ScrapeOutput(results=[], blocked=True, error=f"bing_scrape_error: {e}")
+
+
+async def _scrape_ddg(page, max_results: int) -> ScrapeOutput:
+    """DuckDuckGo HTML endpoint (html.duckduckgo.com/html). Captcha-after-extraction."""
+    title = await page.title()
+    html = await page.content()
+
+    selectors_to_try = ["div.result", "div.web-result"]
+    for sel in selectors_to_try:
+        try:
+            locator = page.locator(sel)
+            count = await locator.count()
+            if count == 0:
+                continue
+            results: list[tuple[str, str, str]] = []
+            for i in range(min(count, max_results)):
+                el = locator.nth(i)
+                try:
+                    a = el.locator("a.result__a, h2 a").first
+                    href = await a.get_attribute("href")
+                    title_text = (await a.inner_text()).strip()
+                    # DDG html sometimes wraps href in a redirect; resolve.
+                    if href and "uddg=" in href:
+                        try:
+                            qs = parse_qs(urlparse(href).query)
+                            real = qs.get("uddg", [None])[0]
+                            if real:
+                                href = unquote(real)
+                        except Exception:
+                            pass
+                    snippet = ""
+                    try:
+                        s = el.locator(".result__snippet").first
+                        if await s.count() > 0:
+                            snippet = (await s.inner_text()).strip()[:300]
+                    except Exception:
+                        pass
+                    if href and title_text:
+                        results.append((href, title_text, snippet))
+                except Exception:
+                    continue
+            if results:
+                return ScrapeOutput(results=results)
+        except Exception as e:
+            logger.debug("DDG selector '%s' failed: %s", sel, e)
+    _, captcha = _detect_block(html, title)
+    if captcha:
+        return ScrapeOutput(results=[], captcha=True, error="captcha_after_zero_results")
+    return ScrapeOutput(results=[], blocked=True, error="no_results_no_selectors_matched")
+
+
+_SCRAPERS = {
+    "google": _scrape_google,
+    "bing": _scrape_bing,
+    "ddg": _scrape_ddg,
+}
+
+
+async def scrape(engine: str, page, max_results: int) -> ScrapeOutput:
+    """Dispatch to the engine-specific scraper."""
+    fn = _SCRAPERS.get(engine)
+    if not fn:
+        return ScrapeOutput(results=[], error=f"unsupported_engine:{engine}")
+    return await fn(page, max_results)

ai_platform/modules/cloak/src/cloak/server.py

@@ -0,0 +1,208 @@
+"""FastAPI server — POST /v1/search, GET /health."""
+from __future__ import annotations
+
+import asyncio
+import logging
+import time
+from contextlib import asynccontextmanager
+
+from fastapi import Depends, FastAPI, Header, HTTPException, Request, status
+from fastapi.responses import JSONResponse
+
+from .browser_pool import BrowserPool
+from .config import CloakSettings
+from .schemas import (
+    EngineStats,
+    HealthResponse,
+    SearchRequest,
+    SearchResponse,
+    SearchResult,
+)
+from .scraper import scrape, search_url
+
+logging.basicConfig(
+    level=logging.INFO,
+    format="%(asctime)s %(levelname)s %(name)s %(message)s",
+)
+logger = logging.getLogger("cloak.server")
+
+
+settings = CloakSettings()
+logger.setLevel(settings.log_level.upper())
+
+pool: BrowserPool | None = None
+_engine_last_call: dict[str, float] = {}
+_engine_lock = asyncio.Lock()
+
+
+@asynccontextmanager
+async def lifespan(app: FastAPI):
+    """Spin up the browser pool on startup, tear down on shutdown."""
+    global pool
+    pool = BrowserPool(size=settings.pool_size, humanize=settings.humanize)
+    try:
+        await pool.start()
+    except Exception as e:  # noqa: BLE001
+        logger.error("Failed to warm browser pool: %s", e)
+        # Continue running — pool may recover via lazy creation on next acquire
+    logger.info("cloak service ready on port %d (pool=%d)", settings.port, settings.pool_size)
+    try:
+        yield
+    finally:
+        if pool:
+            await pool.stop()
+
+
+app = FastAPI(
+    title="cloak",
+    version="0.1.0",
+    description="Stealth Chromium scraping service. Scrapes Google/Bing/DuckDuckGo SERPs.",
+    lifespan=lifespan,
+)
+
+
+def _require_auth(authorization: str | None = Header(default=None)) -> None:
+    """Optional bearer-token check (skipped when CLOAK_AUTH_TOKEN is empty)."""
+    if not settings.auth_token:
+        return
+    if not authorization or not authorization.startswith("Bearer "):
+        raise HTTPException(status_code=status.HTTP_401_UNAUTHORIZED, detail="Missing bearer token")
+    if authorization.removeprefix("Bearer ").strip() != settings.auth_token:
+        raise HTTPException(status_code=status.HTTP_403_FORBIDDEN, detail="Invalid token")
+
+
+@app.get("/health", response_model=HealthResponse, dependencies=[Depends(_require_auth)] if False else [])
+async def health() -> HealthResponse:
+    p = pool
+    if p is None:
+        return HealthResponse(status="unhealthy", pool_size=0, pool_available=0, version="0.1.0")
+    status_label = "healthy" if p.available > 0 else "degraded"
+    return HealthResponse(
+        status=status_label,
+        pool_size=p.size,
+        pool_available=p.available,
+        version="0.1.0",
+    )
+
+
+async def _throttle(engine: str) -> None:
+    """Enforce a minimum interval between requests to the same engine."""
+    if settings.engine_min_interval_ms <= 0:
+        return
+    async with _engine_lock:
+        now = time.monotonic()
+        last = _engine_last_call.get(engine, 0.0)
+        gap = (now - last) * 1000
+        wait = settings.engine_min_interval_ms - gap
+        if wait > 0:
+            await asyncio.sleep(wait / 1000)
+        _engine_last_call[engine] = time.monotonic()
+
+
+async def _scrape_one(
+    engine: str,
+    query: str,
+    max_results: int,
+    language: str | None,
+) -> tuple[list[SearchResult], EngineStats]:
+    """Run one (engine, query) scrape and return parsed results + stats."""
+    assert pool is not None
+    t0 = time.monotonic()
+    url = search_url(engine, query, language)
+    out_results: list[SearchResult] = []
+    stat = EngineStats(
+        engine=engine, query=query, results_count=0,
+        blocked=False, captcha=False, elapsed_ms=0, error=None,
+    )
+    try:
+        await _throttle(engine)
+        async with pool.acquire() as browser:
+            page = await browser.new_page()
+            try:
+                await page.goto(url, timeout=settings.page_timeout_ms, wait_until="domcontentloaded")
+                await page.wait_for_timeout(800)  # let JS settle
+                output = await scrape(engine, page, max_results)
+                stat.blocked = output.blocked
+                stat.captcha = output.captcha
+                if output.error:
+                    stat.error = output.error
+                for rank, (u, t, s) in enumerate(output.results, start=1):
+                    out_results.append(SearchResult(
+                        url=u, title=t, snippet=s,
+                        engine=engine, query=query, rank=rank,
+                    ))
+            finally:
+                try:
+                    await page.close()
+                except Exception:
+                    pass
+    except asyncio.TimeoutError:
+        stat.error = "timeout"
+        stat.blocked = True
+    except Exception as e:  # noqa: BLE001
+        stat.error = f"{type(e).__name__}: {str(e)[:100]}"
+        stat.blocked = True
+
+    stat.results_count = len(out_results)
+    stat.elapsed_ms = int((time.monotonic() - t0) * 1000)
+    return out_results, stat
+
+
+@app.post("/v1/search", response_model=SearchResponse, dependencies=[Depends(_require_auth)])
+async def search(req: SearchRequest, request: Request) -> SearchResponse:
+    """Run search across all (engine × query) pairs in parallel."""
+    if pool is None:
+        raise HTTPException(status_code=503, detail="Pool not ready")
+
+    # Apply caps
+    queries = req.queries[: settings.max_queries]
+    engines = req.engines[: settings.max_engines]
+    per_engine = min(req.max_results_per_engine, settings.max_results_cap)
+
+    t0 = time.monotonic()
+
+    async def _runner():
+        tasks = [
+            _scrape_one(e, q, per_engine, req.language)
+            for e in engines for q in queries
+        ]
+        return await asyncio.gather(*tasks, return_exceptions=False)
+
+    try:
+        gathered = await asyncio.wait_for(_runner(), timeout=settings.search_timeout_sec)
+    except asyncio.TimeoutError:
+        raise HTTPException(status_code=504, detail=f"Search exceeded {settings.search_timeout_sec}s")
+
+    all_results: list[SearchResult] = []
+    stats: list[EngineStats] = []
+    for results, stat in gathered:
+        all_results.extend(results)
+        stats.append(stat)
+
+    elapsed = int((time.monotonic() - t0) * 1000)
+    logger.info(
+        "search: q=%d e=%d -> results=%d in %dms",
+        len(queries), len(engines), len(all_results), elapsed,
+    )
+    return SearchResponse(results=all_results, stats=stats, total_elapsed_ms=elapsed)
+
+
+@app.exception_handler(Exception)
+async def _generic(_: Request, exc: Exception):  # noqa: ARG001
+    logger.exception("Unhandled error")
+    return JSONResponse(status_code=500, content={"error": str(exc)[:200]})
+
+
+def main() -> None:
+    import uvicorn
+
+    uvicorn.run(
+        "cloak.server:app",
+        host=settings.host,
+        port=settings.port,
+        log_level=settings.log_level.lower(),
+    )
+
+
+if __name__ == "__main__":
+    main()

ai_platform/modules/cloak/tests/test_schemas.py

@@ -0,0 +1,50 @@
+"""Quick schema validation tests — run with `pytest tests/`."""
+from __future__ import annotations
+
+import pytest
+from pydantic import ValidationError
+
+from cloak.schemas import SearchRequest, SearchResponse, EngineStats, SearchResult
+
+
+def test_search_request_minimal():
+    req = SearchRequest(queries=["BNR confiscare"])
+    assert req.queries == ["BNR confiscare"]
+    assert req.engines == ["google", "bing", "ddg"]
+    assert req.max_results_per_engine == 10
+
+
+def test_search_request_rejects_unknown_engine():
+    with pytest.raises(ValidationError):
+        SearchRequest(queries=["x"], engines=["yahoo"])
+
+
+def test_search_request_rejects_empty_queries():
+    with pytest.raises(ValidationError):
+        SearchRequest(queries=[])
+
+
+def test_search_response_round_trip():
+    resp = SearchResponse(
+        results=[
+            SearchResult(
+                url="https://example.com",
+                title="Example",
+                snippet="...",
+                engine="google",
+                query="test",
+                rank=1,
+            ),
+        ],
+        stats=[
+            EngineStats(
+                engine="google", query="test",
+                results_count=1, elapsed_ms=2000,
+            ),
+        ],
+        total_elapsed_ms=2050,
+    )
+    j = resp.model_dump_json()
+    parsed = SearchResponse.model_validate_json(j)
+    assert len(parsed.results) == 1
+    assert parsed.stats[0].engine == "google"