clossers/didi-lot1-ai
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

LOT 1 - Optimizare script build -Instalare mono comanda

Browse source
This commit is contained in:
Dezvoltari Evotech 2026-06-27 06:42:02 -07:00
parent 5380c3fc63
commit 42ff22bf85
127 changed files with 16163 additions and 532 deletions
Download patch file Download diff file Expand all files Collapse all files

25
ai_platform/modules/web/INDEX.md
View file

@ -5,18 +5,15 @@ Web search service for DIDI claim verification. Routes between free (SearXNG met
- **Stack:** Python 3.10+, FastAPI, Pydantic v2, httpx (HTTP/2), Uvicorn
- **URL:** `http://10.11.10.12:51100` (Dev) / `http://10.11.10.13:51100` (Prod-style reference per agent-v3 default)
- **Container:** `didiAI-web-api`
- **SearXNG cluster:** 3 replicas (`didiAI-web-searxng-1/2/3`) behind nginx LB (`didiAI-web-searxng`, port 55100)
- **SearXNG cache:** 3 Valkey/Redis replicas (`didiAI-web-searxng-redis-1/2/3`)
- **Anonymity proxy:** `didiAI-web-tor` (Tor SOCKS5 shared by all SearXNG instances)
- **SearXNG:** single container `didiAI-web-searxng` (internal `:8080`, not published on the host) — see `local_gpu_stack`. The multi-replica cluster (`deploy/metasearch`) is an optional scaling variant, not part of the delivery.
## Ce face
The module powers the fact-checking pipeline:
- **Free search** via SearXNG metasearch (3 round-robin instances, each with a dedicated Valkey for isolated cache/state). Local Qwen LLM is used for context detection + evidence snippet extraction.
- **Free search** via SearXNG metasearch (the delivered deployment uses a single `didiAI-web-searxng` container on internal `:8080`). Local Qwen LLM is used for context detection + evidence snippet extraction.
- **Premium search** via paid APIs — Brave, Tavily, SerpAPI, Linkup, Exa — selected by `PaidSearchClient` rotation (read keys from `WEB_*_API_KEY` envs). Uses OpenRouter (configurable model) for LLM steps.
- **URL fetching** with readability-lxml extraction; auto-fallback chain `HTTP -> Playwright (browse) -> Vision LLM screenshot OCR` for JS-heavy or protected pages. PDF URLs are skipped.
- **Tor proxy** for sensitive/anonymized SearXNG queries.
- **Brain cache** integration: both tiers READ from the brain cache on `/v1/gather`; only premium WRITES quality results back via the `BrainIngestSink` (fire-and-forget). Free users effectively get the paid knowledge base for free.
- **Dashboard event sink:** every request emits a structured event (tier, provider, duration, status, results_count, raw response snapshot for `/v1/gather`) to the configured dashboard.
- **Runtime config:** `RuntimeConfigClient` polls the dashboard for tier overrides (e.g. `web.tier.free.max_search_results`).
@ -107,16 +104,16 @@ src/web/
└── sink.py # BrainIngestSink (premium-only cache writer)
```
## SearXNG cluster
## SearXNG
Located at `deploy/metasearch/`:
**Delivered deployment:** a single SearXNG container `didiAI-web-searxng` (`docker.io/searxng/searxng:latest`) on internal port `:8080`, wired via `local_gpu_stack`. It is **not** published on the host, and there is no Redis/Valkey cache, no Tor proxy, and no nginx load balancer in the current infrastructure. The web-api reaches it at `WEB_SEARXNG_BASE_URL=http://didiAI-web-searxng:8080`.
- 3 SearXNG instances (`docker.io/searxng/searxng:latest`) round-robin behind an nginx LB (`searxng-lb` -> port 55100).
- Each SearXNG instance has its own dedicated Valkey 8 cache (`searxng-redis-data-{1,2,3}`) for isolated state.
- Shared Tor SOCKS5 proxy (`dperson/torproxy`) for queries needing anonymity.
- Per-instance config under `deploy/metasearch/searxng-{1,2,3}/`.
- Caddyfile + reset script (`searxng-reset.sh`) included for ops.
- Article extraction in the API uses readability-lxml; JS-heavy pages fall through to Playwright (chromium) and finally to a vision-LLM screenshot pass.
Article extraction in the API uses readability-lxml; JS-heavy pages fall through to Playwright (chromium) and finally to a vision-LLM screenshot pass.
**Optional scaling variant** (not part of the delivery), located at `deploy/metasearch/`:
- Multiple SearXNG instances round-robin behind an nginx LB.
- Per-instance config, Caddyfile + reset script (`searxng-reset.sh`) included for ops.
## Configuration
@ -143,7 +140,7 @@ Full reference: `.env.example`.
## Deployment
- **API:** `cd deploy/ && docker compose --profile api up -d` (builds `didiai-web-api` from `deploy/Dockerfile`, joins networks `didi-network` + `didibrain`, exposes `51100:51100`).
- **SearXNG cluster:** `cd deploy/metasearch/ && docker compose up -d` (LB + 3x SearXNG + 3x Valkey + Tor).
- **SearXNG (delivered):** single `didiAI-web-searxng` container on internal `:8080`, brought up via `local_gpu_stack`. Optional scaling variant: `cd deploy/metasearch/ && docker compose up -d` (LB + multiple SearXNG replicas).
- **Healthcheck:** `curl http://localhost:51100/health`.
- Compose files: `deploy/docker-compose.yml`, `deploy/metasearch/docker-compose.yaml`.