clossers/didi-lot1-ai
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

LOT 1 - Optimizare script build -Instalare mono comanda

Browse source
This commit is contained in:
Dezvoltari Evotech 2026-06-27 06:42:02 -07:00
parent 5380c3fc63
commit 42ff22bf85
127 changed files with 16163 additions and 532 deletions
Download patch file Download diff file Expand all files Collapse all files

16
ai_platform/modules/web/API.md
View file

@ -36,15 +36,15 @@ POST /v1/gather
{
"claim": "The claim to verify",
"search_queries": ["optional custom queries"],
"max_search_results": 20,
"max_search_results": 10,
"site_allowlist": ["reuters.com"],
"site_blocklist": ["spam-site.com"],
"fetch_method": "auto",
"auto_fallback": true,
"extract_snippets": true,
"extract_snippets": false,
"max_evidence_items": 15,
"dedupe": true,
"timeout_seconds": 60.0
"timeout_seconds": 90.0
}
```
@ -52,16 +52,16 @@ POST /v1/gather
|-------|------|----------|---------|-------------|
| `claim` | string | Yes | - | Claim to gather evidence for (10-1000 chars) |
| `search_queries` | array[string] | No | null | Custom search queries (auto-generated if not provided) |
| `max_search_results` | integer | No | 20 | Max search results (5-50) |
| `max_search_results` | integer | No | 10 | Max search results (5-50) |
| `site_allowlist` | array[string] | No | null | Only search these domains |
| `site_blocklist` | array[string] | No | null | Exclude these domains |
| `fetch_method` | string | No | "auto" | "auto", "http", "browse", "vision" |
| `auto_fallback` | boolean | No | true | Escalate on fetch failure |
| `extract_snippets` | boolean | No | true | Use LLM for snippet extraction |
| `extract_snippets` | boolean | No | False | Deprecated — LLM snippet extraction (no longer used by default) |
| `max_evidence_items` | integer | No | 15 | Max items in final pack (1-50) |
| `dedupe` | boolean | No | true | Deduplicate evidence |
| `parallel_fetches` | integer | No | 5 | Concurrent fetch operations (1-10) |
| `timeout_seconds` | number | No | 60.0 | Total pipeline timeout (10-300) |
| `timeout_seconds` | number | No | 90.0 | Total pipeline timeout (10-300) |
#### Response
@ -136,7 +136,7 @@ POST /v1/search
"max_results": 10,
"site_allowlist": ["string"],
"site_blocklist": ["string"],
"language": "en",
"language": "auto",
"country": "US",
"freshness": "month",
"safe_search": "moderate"
@ -149,7 +149,7 @@ POST /v1/search
| `max_results` | integer | No | 10 | Results per query (1-100) |
| `site_allowlist` | array[string] | No | null | Only include these domains |
| `site_blocklist` | array[string] | No | null | Exclude these domains |
| `language` | string | No | "en" | Search language (ISO 639-1) |
| `language` | string | No | "auto" | Search language (ISO 639-1, or "auto") |
| `country` | string | No | "US" | Search country (ISO 3166-1) |
| `freshness` | string | No | null | Filter by age: day, week, month, year |
| `safe_search` | string | No | "moderate" | off, moderate, strict |

6
ai_platform/modules/web/BRAIN_INTEGRATION.md
View file

@ -121,14 +121,14 @@ complet (fallback la orchestrator normal pentru tot).
## Networking Docker
Web-api trăiește pe rețeaua `deploy_default` (cu dashboard, video, audio,
Web-api trăiește pe rețeaua `didi-network` (cu dashboard, video, audio,
searxng). Brain trăiește pe rețeaua proprie `didibrain`.
Web-api e atașat la **ambele** rețele în `deploy/docker-compose.yml`:
```yaml
networks:
deploy_default:
didi-network:
external: true
didibrain:
external: true
@ -136,7 +136,7 @@ networks:
services:
web-api:
networks:
- deploy_default
- didi-network
- didibrain
```

25
ai_platform/modules/web/INDEX.md
View file

@ -5,18 +5,15 @@ Web search service for DIDI claim verification. Routes between free (SearXNG met
- **Stack:** Python 3.10+, FastAPI, Pydantic v2, httpx (HTTP/2), Uvicorn
- **URL:** `http://10.11.10.12:51100` (Dev) / `http://10.11.10.13:51100` (Prod-style reference per agent-v3 default)
- **Container:** `didiAI-web-api`
- **SearXNG cluster:** 3 replicas (`didiAI-web-searxng-1/2/3`) behind nginx LB (`didiAI-web-searxng`, port 55100)
- **SearXNG cache:** 3 Valkey/Redis replicas (`didiAI-web-searxng-redis-1/2/3`)
- **Anonymity proxy:** `didiAI-web-tor` (Tor SOCKS5 shared by all SearXNG instances)
- **SearXNG:** single container `didiAI-web-searxng` (internal `:8080`, not published on the host) — see `local_gpu_stack`. The multi-replica cluster (`deploy/metasearch`) is an optional scaling variant, not part of the delivery.
## Ce face
The module powers the fact-checking pipeline:
- **Free search** via SearXNG metasearch (3 round-robin instances, each with a dedicated Valkey for isolated cache/state). Local Qwen LLM is used for context detection + evidence snippet extraction.
- **Free search** via SearXNG metasearch (the delivered deployment uses a single `didiAI-web-searxng` container on internal `:8080`). Local Qwen LLM is used for context detection + evidence snippet extraction.
- **Premium search** via paid APIs — Brave, Tavily, SerpAPI, Linkup, Exa — selected by `PaidSearchClient` rotation (read keys from `WEB_*_API_KEY` envs). Uses OpenRouter (configurable model) for LLM steps.
- **URL fetching** with readability-lxml extraction; auto-fallback chain `HTTP -> Playwright (browse) -> Vision LLM screenshot OCR` for JS-heavy or protected pages. PDF URLs are skipped.
- **Tor proxy** for sensitive/anonymized SearXNG queries.
- **Brain cache** integration: both tiers READ from the brain cache on `/v1/gather`; only premium WRITES quality results back via the `BrainIngestSink` (fire-and-forget). Free users effectively get the paid knowledge base for free.
- **Dashboard event sink:** every request emits a structured event (tier, provider, duration, status, results_count, raw response snapshot for `/v1/gather`) to the configured dashboard.
- **Runtime config:** `RuntimeConfigClient` polls the dashboard for tier overrides (e.g. `web.tier.free.max_search_results`).
@ -107,16 +104,16 @@ src/web/
└── sink.py # BrainIngestSink (premium-only cache writer)
```
## SearXNG cluster
## SearXNG
Located at `deploy/metasearch/`:
**Delivered deployment:** a single SearXNG container `didiAI-web-searxng` (`docker.io/searxng/searxng:latest`) on internal port `:8080`, wired via `local_gpu_stack`. It is **not** published on the host, and there is no Redis/Valkey cache, no Tor proxy, and no nginx load balancer in the current infrastructure. The web-api reaches it at `WEB_SEARXNG_BASE_URL=http://didiAI-web-searxng:8080`.
- 3 SearXNG instances (`docker.io/searxng/searxng:latest`) round-robin behind an nginx LB (`searxng-lb` -> port 55100).
- Each SearXNG instance has its own dedicated Valkey 8 cache (`searxng-redis-data-{1,2,3}`) for isolated state.
- Shared Tor SOCKS5 proxy (`dperson/torproxy`) for queries needing anonymity.
- Per-instance config under `deploy/metasearch/searxng-{1,2,3}/`.
- Caddyfile + reset script (`searxng-reset.sh`) included for ops.
- Article extraction in the API uses readability-lxml; JS-heavy pages fall through to Playwright (chromium) and finally to a vision-LLM screenshot pass.
Article extraction in the API uses readability-lxml; JS-heavy pages fall through to Playwright (chromium) and finally to a vision-LLM screenshot pass.
**Optional scaling variant** (not part of the delivery), located at `deploy/metasearch/`:
- Multiple SearXNG instances round-robin behind an nginx LB.
- Per-instance config, Caddyfile + reset script (`searxng-reset.sh`) included for ops.
## Configuration
@ -143,7 +140,7 @@ Full reference: `.env.example`.
## Deployment
- **API:** `cd deploy/ && docker compose --profile api up -d` (builds `didiai-web-api` from `deploy/Dockerfile`, joins networks `didi-network` + `didibrain`, exposes `51100:51100`).
- **SearXNG cluster:** `cd deploy/metasearch/ && docker compose up -d` (LB + 3x SearXNG + 3x Valkey + Tor).
- **SearXNG (delivered):** single `didiAI-web-searxng` container on internal `:8080`, brought up via `local_gpu_stack`. Optional scaling variant: `cd deploy/metasearch/ && docker compose up -d` (LB + multiple SearXNG replicas).
- **Healthcheck:** `curl http://localhost:51100/health`.
- Compose files: `deploy/docker-compose.yml`, `deploy/metasearch/docker-compose.yaml`.

30
ai_platform/modules/web/README.md
View file

@ -14,7 +14,7 @@ This module is designed for **fact-checking pipelines**. Given a claim, it:
**Required:**
- All global prerequisites (see main [README.md](../../README.md))
- SearXNG instance (deploy with `cd deploy/metasearch && docker compose up -d`)
- SearXNG instance. The delivered deployment runs a **single** SearXNG container (`didiAI-web-searxng`, internal `:8080`) wired via `local_gpu_stack`. The `deploy/metasearch` cluster (multiple replicas) is an **optional** scaling variant, not the default.
- LLM Inference server (llm-inference module at port 14011)
**Optional:**
@ -66,7 +66,7 @@ cd deploy/
# Configure environment
cat > .env << 'EOF'
WEB_SEARXNG_BASE_URL=http://localhost:55100
WEB_SEARXNG_BASE_URL=http://didiAI-web-searxng:8080
WEB_LLM_BASE_URL=http://didiAI-llm-api:14011
WEB_LLM_API_KEY=your-llm-api-key
WEB_VISION_MODEL=qwen-vl
@ -133,12 +133,12 @@ curl -X POST http://localhost:51100/v1/gather \
{
"claim": "The claim to verify",
"search_queries": ["optional", "custom", "queries"],
"max_search_results": 20,
"max_search_results": 10,
"site_allowlist": ["reuters.com", "bbc.com"],
"site_blocklist": ["spam-site.com"],
"fetch_method": "auto",
"auto_fallback": true,
"extract_snippets": true,
"extract_snippets": false,
"max_evidence_items": 15,
"dedupe": true
}
@ -148,12 +148,12 @@ curl -X POST http://localhost:51100/v1/gather \
|-------|------|---------|-------------|
| `claim` | string | required | The claim to gather evidence for |
| `search_queries` | array | null | Custom search queries (auto-generated if not provided) |
| `max_search_results` | int | 20 | Max search results (5-50) |
| `max_search_results` | int | 10 | Max search results (5-50) |
| `site_allowlist` | array | null | Only search these domains |
| `site_blocklist` | array | null | Exclude these domains |
| `fetch_method` | string | "auto" | "auto", "http", "browse", "vision" |
| `auto_fallback` | bool | true | Escalate on fetch failure |
| `extract_snippets` | bool | true | Use LLM for snippet extraction |
| `extract_snippets` | bool | False | Deprecated — LLM snippet extraction (no longer used by default) |
| `max_evidence_items` | int | 15 | Max items in final pack |
| `dedupe` | bool | true | Deduplicate evidence |
@ -234,8 +234,10 @@ src/web/
│
├── search/ # Search providers
│ ├── __init__.py
│ ├── protocol.py # SearchProvider Protocol
│ └── searxng.py # SearXNGClient
│ └── protocol.py # SearchProvider Protocol
│
├── metasearch/ # SearXNG (free tier)
│ └── client.py # SearXNGClient
│
├── fetch/ # HTTP + readability extraction
│ └── client.py # FetchClient
@ -254,10 +256,12 @@ src/web/
├── dependencies.py # DI (auth, rate limiter)
├── middleware.py # RequestId, RateLimit
└── routes/
├── search.py # POST /v1/search
├── fetch.py # POST /v1/fetch
├── gather.py # POST /v1/gather (main endpoint)
└── health.py # /health, /ready
├── search.py # POST /v1/search
├── image_search.py # POST /v1/image-search
├── fetch.py # POST /v1/fetch
├── gather.py # POST /v1/gather (main endpoint)
├── info.py # GET /v1/info (catalog metadata)
└── health.py # /health, /ready
```
## Deployment
@ -286,7 +290,7 @@ docker compose --profile api down
### Network
The container joins the `deploy_default` network to communicate with:
The container joins the `didi-network` network to communicate with:
- `didiAI-llm-api:14011` - LLM inference server
## Development

3
ai_platform/modules/web/deploy/metasearch/searxng/settings.yml
View file

@ -36,9 +36,6 @@ outgoing:
useragent_suffix: "didiAI-metasearch"
pool_connections: 100
pool_maxsize: 20
proxies:
all://:
- http://smart-ouowetdhvcnw:ctAFve4tpQkTyLtB@proxy.smartproxy.net:3120
redis:
url: redis://didiAI-web-searxng-redis:6379/0