clossers/didi-lot1-ai
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

LOT 1 - Optimizare script build -Instalare mono comanda

Browse source
This commit is contained in:
Dezvoltari Evotech 2026-06-27 06:42:02 -07:00
parent 5380c3fc63
commit 42ff22bf85
127 changed files with 16163 additions and 532 deletions
Download patch file Download diff file Expand all files Collapse all files

6
.gitignore vendored Normal file
View file

@ -0,0 +1,6 @@
# Modele locale (mari, descarcate de deploy.sh) — NU se comit
ai_platform/models/
ai_platform/local_gpu_stack/.env
ai_platform/local_gpu_stack/test_assets/
ai_platform/local_gpu_stack/runs/

102
README_LIVRABIL.md
View file

@ -4,9 +4,10 @@ Pachet de lucru pentru continuarea LOT 1 pe serverul cu **inferență GPU**. Con
codul `ai_platform` (curățat), documentația de achiziție (caiet + ofertă) și
raportul de audit cod-vs-caiet-vs-ofertă.
> **Scop:** muți acest folder pe serverul cu inferență, configurezi modelele
> reale și finalizezi punctele rămase (vezi §5). Tot ce s-a putut face fără GPU
> e deja făcut și validat în Docker.
> **Scop:** platforma este deja deployată și LIVE pe host-ul GPU (2x H200, 11/11
> containere healthy) prin `ai_platform/local_gpu_stack/deploy.sh`. Acest pachet
> servește pentru portabilitate (mutare pe alt host) și pentru punctele rămase
> de infra (vezi §5).
---
@ -25,8 +26,9 @@ livrabil_lot1/
└── AUDIT_LOT1.md ← audit complet pe module (citește-l pentru detalii)
```
`uv.lock` au fost excluse intenționat (se regenerează cu `uv sync`). `node_modules`
se regenerează cu `npm ci`.
`uv.lock` au fost excluse intenționat. Fix aplicat: Dockerfile-urile `llm-inference`
și `video-analysis` NU mai cer `uv.lock` la build (se rezolvă din `pyproject.toml`).
`node_modules` se regenerează cu `npm ci`.
---
@ -36,22 +38,26 @@ Sistemul AI = servicii independente, containerizate, pe care backend-ul (LOT 2)
le consumă prin API. Fiecare modul are `deploy/deploy.sh`, `docker-compose.yml`,
`Dockerfile`, `.env.example`.
> **Stare generală: platformă LIVE pe host-ul GPU (2x H200), 11/11 containere healthy.**
> Toate cele 13 module sunt livrate și funcționale. Singurul punct cu nuanță:
> dashboard rulează în STAGING MODE (`DASHBOARD_STAGING_MODE=true`, Keycloak bypassed).
| Modul | Ce face | Stare |
|---|---|---|
| **llm-inference** | Gateway OpenAI-compatible peste 3 backend-uri (vLLM/llama.cpp/LiteLLM); rutare + fallback + metrici | 🟢 ~95% |
| **extractors** *(nou)* | Feature extraction *ușor*: metadata (EXIF/ELA/spectrogramă/integritate), sentiment/OCR (deleagă LLM), NER (GLiNER), detect (YOLO) | 🟢 ~95% |
| **video-analysis** | Deepfake video (BusterX/vLLM), verdict REAL/FAKE/UNCERTAIN + frames + evidence | 🟢 |
| **audio** | Transcriere (faster-whisper) | 🟢 |
| **web** | Web crawl & evidence (Playwright, multi-provider search) | 🟡 90% |
| **embeddings / rerank** | bge-m3 / bge-reranker-v2-m3, OpenAI-compatible | 🟢 85% |
| **didi_brain** | Cache rezultate (Postgres/pgvector) + RAG (Atomic) + fact-checking; consumat de backend | 🟢 |
| **dashboard** | Admin UI (React+FastAPI): monitorizare AI, catalog modele DB, config, RBAC | 🟡 ~78% |
| **catalog-api** | Agregator service-discovery (`/v1/info`), stateless | 🟢 |
| **gateway** | Reverse-proxy nginx (config-only) — punct de intrare AI | 🟢 |
| **forensic_features** | Detectoare forensice (rPPG/lip-sync/forgery heatmap/lighting) | 🟢 |
| **cloak** | Scraper SERP stealth (tier-3 fallback pt `web`) | 🟢 |
| **llm-inference** | Gateway OpenAI-compatible peste 3 backend-uri (vLLM/llama.cpp/LiteLLM); rutare + fallback + metrici | 🟢 livrat |
| **extractors** *(nou)* | Feature extraction *ușor*: metadata (EXIF/ELA/spectrogramă/integritate), sentiment/OCR (deleagă LLM), NER (GLiNER), detect (YOLOv8n) | 🟢 livrat |
| **video-analysis** | Deepfake video (BusterX/vLLM), verdict REAL/FAKE/UNCERTAIN + frames + evidence | 🟢 livrat |
| **audio** | Transcriere (faster-whisper) | 🟢 livrat |
| **web** | Web crawl & evidence (Playwright, SearXNG + provideri premium) | 🟢 livrat |
| **embeddings / rerank** | bge-m3 / bge-reranker-v2-m3, OpenAI-compatible | 🟢 livrat |
| **didi_brain** | Cache rezultate (Postgres/pgvector) + RAG (Atomic) + fact-checking; consumat de backend | 🟢 livrat |
| **dashboard** | Admin UI (React+FastAPI): monitorizare AI, catalog modele DB, config, RBAC | 🟢 livrat (STAGING MODE — auth bypassed) |
| **catalog-api** | Agregator service-discovery (`/v1/info`), stateless | 🟢 livrat |
| **gateway** | Reverse-proxy nginx (config-only) — punct de intrare AI | 🟢 livrat |
| **forensic_features** | Detectoare forensice (rPPG/lip-sync/forgery heatmap/lighting) | 🟢 livrat |
| **cloak** | Scraper SERP stealth (tier-3 fallback pt `web`) | 🟢 livrat |
Detalii complete + procente per cerință: **`documentatie/AUDIT_LOT1.md`**.
Detalii complete per cerință: **`documentatie/AUDIT_LOT1.md`**.
---
@ -93,50 +99,64 @@ docker run --rm -v "$PWD":/app -w /app node:20-slim bash -c "npm ci && npx tsc -
---
## 5. Ce mai e de făcut pe serverul cu inferență
## 5. Stare deploy + ce mai e de făcut
### 5.1 Configurări obligatorii la mutare (altfel serviciile nu „văd" modelele)
- **llm-inference**: pornește vLLM cu `--served-model-name qwen3.5` (sau setează
`LLM_MODEL_ALIASES`), `VLLM_MODEL=Qwen/Qwen3.5-35B-A3B`. Vezi `.env.example`.
- **extractors**: `EXTRACTORS_LLM_GATEWAY_URL=http://<llm-gateway>:14011` (pt sentiment/OCR).
**Deploy livrat (single-host GPU):** `ai_platform/local_gpu_stack/deploy.sh` —
orchestrează tot stack-ul „cărămidă cu cărămidă" (gated, pas cu pas, fail-fast).
Platforma rulează deja LIVE (11/11 healthy). Vezi `local_gpu_stack/README.md`.
### 5.1 Rezolvate în această iterație
- **extractors** și **forensic_features** — deployate și healthy.
- **enable_thinking** (Qwen3.5) — rezolvat (parametrul e gestionat corect în
llm-inference / video-analysis).
- **dashboard** — panourile de monitorizare (health agregat, cozi, latențe) live.
### 5.2 Configurări la mutare pe alt host (portabilitate)
- **llm-inference**: vLLM cu `--served-model-name qwen3.5`, `VLLM_MODEL=Qwen/Qwen3.5-35B-A3B`.
- **video-analysis / audio / embeddings / rerank**: setează URL-urile vLLM/model în `.env`.
- **dashboard**: pentru a aprinde panourile noi —
`DASHBOARD_PROMETHEUS_URL=http://<prometheus>:9090`,
`DASHBOARD_RABBITMQ_MGMT_URL=http://<rabbitmq>:15672` (+ user/parolă).
RBAC se activează automat când e setat `DASHBOARD_KEYCLOAK_URL`; **scoate
`DASHBOARD_STAGING_MODE` în producție** (bypassează auth).
- **dashboard**: `DASHBOARD_PROMETHEUS_URL`, `DASHBOARD_RABBITMQ_MGMT_URL` (+ user/parolă).
**Scoate `DASHBOARD_STAGING_MODE` în producție** pentru a activa Keycloak/RBAC
(în starea livrată dashboard rulează în STAGING MODE, cu auth bypassed).
### 5.2 Cod rămas (validabil pe server cu modelele reale)
### 5.3 Cod rămas deschis
- **llm-inference — load/unload model** (Task 4): vLLM **nu** suportă hot-swap
nativ (un model/proces). Decizie: (a) implementezi „register/route" — load =
asiguri că un backend servește modelul X; sau (b) marchezi „nesuportat de vLLM"
în ofertă/recepție. *(singurul punct deschis la inferență)*
nativ (un model/proces). Decizie: (a) „register/route" — load = asiguri că un
backend servește modelul X; sau (b) marchezi „nesuportat de vLLM". *(singurul
punct deschis la inferență)*
- **web** (ofertă): `robots.txt`, rate-limit per-domeniu, jurnal audit persistent.
- **embeddings/rerank**: failover automat între backend-uri + teste API.
- **RAG**: wrapper de contract `/v1/query`/`/v1/index` peste `/v1/gather` din
didi_brain (sau acceptat în forma livrată).
- **CI/CD**: non-root pe 5 Dockerfile, rollback + publish imagini AI în registry.
- **CI/CD**: non-root pe Dockerfile-uri rămase, rollback + publish imagini AI în registry.
### 5.3 Infra (manifeste de scris; validare reală doar cu cluster/GPU)
### 5.4 Infra (manifeste de scris; validare reală doar cu cluster/GPU)
- **Orchestrare / scalare automată** (caiet 747-766) — k8s/Helm + HPA. *Singurul
gap mare de caiet rămas.*
- **dashboard GPU/VRAM** — exporter dcgm pe host-ul GPU + panou (codul de panou se
cablează ca latențele, prin Prometheus).
- **dashboard GPU/VRAM** — exporter dcgm pe host-ul GPU + panou (se cablează ca
latențele, prin Prometheus).
- **dashboard control rute** — gateway-ul AI e nginx static; control real = view +
enable/disable prin regenerare config + reload (NU Kong, care e LOT 2).
---
## 6. Pornire rapidă (per modul)
## 6. Pornire rapidă
**Recomandat — stack complet single-host GPU (cărămidă cu cărămidă, gated):**
```bash
cd ai_platform/local_gpu_stack
cp .env.example .env # completează variabilele (fail-fast, fără defaults)
./deploy.sh # pornește tot stack-ul, pas cu pas
```
**Per modul (debug / dezvoltare):**
```bash
cd ai_platform/modules/<modul>/deploy
cp ../.env.example .env # completează variabilele (fail-fast, fără defaults)
cp ../.env.example .env
./deploy.sh up # sau: docker compose up -d --build
```
Convenție porturi: `1xxxx` producție / `5xxxx` dev (vezi `ai_platform/CLAUDE.md`).
Bootstrap pentru host CPU-only: `ai_platform/bootstrap.sh` + `DEPLOYMENT.md`.
Bootstrap pentru host CPU-only (web + dashboard): `ai_platform/bootstrap.sh` + `DEPLOYMENT.md`.
---

22
ai_platform/DEPLOYMENT.md
View file

@ -1,5 +1,7 @@
# Deployment Guide
> **Note — delivered deployment:** the delivered stack runs **single-host on the GPU machine** via `ai_platform/local_gpu_stack/deploy.sh` (see `local_gpu_stack/README.md`). The guide below describes the **reference multi-host variant** (CPU host + separate GPU host), kept intentionally for portability.
This guide covers bootstrapping the **web + dashboard** stack on a fresh CPU-only host. All GPU-dependent services (`llm-inference`, `embeddings`, `rerank`, `audio`, `video-analysis`) remain on a separate GPU machine and are called over HTTP.
---
@ -27,7 +29,7 @@ This guide covers bootstrapping the **web + dashboard** stack on a fresh CPU-onl
│ │
│ ┌───────────────────┐ │
│ │ SearXNG │ (existing — free tier source of truth) │
│ │ :55100/8080 │ │
│ │ :8080 (intern) │ │
│ └───────────────────┘ │
│ │
└──────────┬────────────────────────────────────────────────────────────────────────────┘
@ -90,7 +92,7 @@ cd ml-projects
The script will:
1. Verify Docker + Compose.
2. Create the `deploy_default` network.
2. Create the `didi-network` network.
3. Prompt for:
- GPU host address (for LLM URL)
- SearXNG container name / URL
@ -120,8 +122,8 @@ If the bootstrap script fails or you prefer manual control:
### 1. Create the network
```bash
docker network inspect deploy_default >/dev/null 2>&1 \
|| docker network create deploy_default
docker network inspect didi-network >/dev/null 2>&1 \
|| docker network create didi-network
```
### 2. Ensure SearXNG is reachable
@ -129,7 +131,7 @@ docker network inspect deploy_default >/dev/null 2>&1 \
If SearXNG is already running on this host, connect it to the network:
```bash
docker network connect deploy_default <searxng-container-name>
docker network connect didi-network <searxng-container-name>
```
Otherwise, deploy the bundled SearXNG:
@ -248,7 +250,7 @@ The web-api polls the dashboard for runtime config every 30s and fires events ba
WEB_DASHBOARD_URL=http://didiAI-dashboard:51300
```
Both containers must be on the `deploy_default` network for internal hostname resolution.
Both containers must be on the `didi-network` network for internal hostname resolution.
---
@ -368,10 +370,10 @@ docker start didiAI-dashboard
## Troubleshooting
### `docker network deploy_default not found`
### `docker network didi-network not found`
```bash
docker network create deploy_default
docker network create didi-network
```
### SearXNG unreachable from web-api
@ -379,9 +381,9 @@ docker network create deploy_default
Check that SearXNG is on the shared network:
```bash
docker network inspect deploy_default | grep -A2 searxng
docker network inspect didi-network | grep -A2 searxng
# If missing:
docker network connect deploy_default <searxng-container>
docker network connect didi-network <searxng-container>
```
Then verify the hostname/port in `WEB_SEARXNG_BASE_URL`.

112
ai_platform/ENDPOINTS.md
View file

@ -255,7 +255,7 @@ curl -X POST http://localhost:14200/v1/rerank \
| `GET` | `/health` | Health check |
| `GET` | `/ready` | Readiness probe |
| `POST` | `/v1/gather` | **Main: Complete fact-check pipeline** |
| `POST` | `/v1/search` | Web search only (Brave API) |
| `POST` | `/v1/search` | Web search only (SearXNG free tier; provideri premium optionali: SerpAPI/Tavily/Brave/LinkUp via `X-Search-Tier: premium`) |
| `POST` | `/v1/fetch` | Fetch URLs with fallback |
#### Quick Test
@ -282,7 +282,7 @@ curl -X POST http://localhost:51100/v1/gather \
**Base URL:** `http://localhost:54300`
**Purpose:** Speech-to-text using faster-whisper
**Model:** large-v3-turbo (int8 quantization)
**GPU:** GPU 1 (7GB / 143GB VRAM)
**GPU:** GPU 0 (~2GB / 143GB VRAM)
**Decodare:** 5+4+3+0+0 = Dev + AI + Audio + vLLM + instance0
#### Endpoints
@ -369,6 +369,100 @@ curl -X POST http://localhost:54600/analyze/video/semantic \
---
### 1️⃣1️⃣ Extractors API (Feature Extraction)
**Port:** `54400`
**Base URL:** `http://localhost:54400`
**Purpose:** Lightweight feature extraction: metadata, sentiment/OCR (delegate to LLM gateway), NER, object detection
#### Endpoints
| Method | Path | Description |
|--------|------|-------------|
| `GET` | `/health` | Health check |
| `POST` | `/v1/metadata` | EXIF / codec / spectrogram / ELA / integrity (C2PA) |
| `POST` | `/v1/sentiment` | Sentiment analysis (delegates to LLM gateway) |
| `POST` | `/v1/ocr` | OCR (delegates to Qwen3.5 vision via LLM gateway) |
| `POST` | `/v1/ner` | Named entity recognition (GLiNER, mdeberta backbone) |
| `POST` | `/v1/detect` | Object detection (YOLOv8n) |
#### Quick Test
```bash
curl http://localhost:54400/health
```
---
### 1️⃣2️⃣ Forensic Features API
**Port:** `8085`
**Base URL:** `http://localhost:8085`
**Purpose:** Forensic detectors (rPPG, lip-sync, forgery heatmap, lighting) using MediaPipe
#### Endpoints
| Method | Path | Description |
|--------|------|-------------|
| `GET` | `/health` | Health check |
| `POST` | `/v1/analyze` | Run forensic detectors on a video/image |
#### Quick Test
```bash
curl http://localhost:8085/health
```
---
### 1️⃣3️⃣ Dashboard (Admin UI + API)
**Port:** `51300`
**Base URL:** `http://localhost:51300`
**Purpose:** Admin monitoring UI + API: AI health, model catalog (DB-backed), runtime config, RBAC, cost/archive
**Note:** runs in STAGING MODE (`DASHBOARD_STAGING_MODE=true`) — Keycloak auth bypassed in delivered state.
#### Endpoints (selection)
| Method | Path | Description |
|--------|------|-------------|
| `GET` | `/health` | Health check |
| `GET` | `/api/stats/providers` | Provider status + live quota |
| `GET` | `/api/config/schema` | Runtime config schema (DB-overridable) |
| `PUT` | `/api/config/{key}` | Update runtime config (Bearer token required) |
| `GET` | `/audit` | Audit trail |
#### Quick Test
```bash
curl http://localhost:51300/health
```
---
### 1️⃣4️⃣ didi_brain API (Cache + RAG + Fact-checking)
**Port:** `8090`
**Base URL:** `http://localhost:8090`
**Purpose:** Result cache (Postgres/pgvector) + RAG (Atomic) + fact-checking; consumed by backend
#### Endpoints (selection)
| Method | Path | Description |
|--------|------|-------------|
| `GET` | `/health` | Health check |
| `POST` | `/v1/gather` | Gather with cache (returns `brain_meta.cache_status`) |
| `POST` | `/v1/cache/invalidate` | Invalidate cache (rate-limited 10/h per actor) |
| `GET` | `/v1/fact_status/due_for_recheck` | Facts with `next_check_at <= now`, unlocked |
#### Quick Test
```bash
curl http://localhost:8090/health
```
---
## 🧪 Complete Health Check Script
```bash
@ -386,11 +480,15 @@ curl -s http://localhost:14200/health && echo " ✓ Rerank API (14200)"
# Development Services
echo "=== DEVELOPMENT ==="
curl -s http://localhost:51100/health && echo " ✓ Web API (51100)"
curl -s http://localhost:51300/health && echo " ✓ Dashboard (51300)"
curl -s http://localhost:54100/health && echo " ✓ Embeddings API (54100)"
curl -s http://localhost:54200/health && echo " ✓ Rerank API (54200)"
curl -s http://localhost:54300/health && echo " ✓ Audio API (54300)"
curl -s http://localhost:54400/health && echo " ✓ Extractors API (54400)"
curl -s http://localhost:54500/health && echo " ✓ BusterX (54500)"
curl -s http://localhost:54600/health && echo " ✓ Video API (54600)"
curl -s http://localhost:8085/health && echo " ✓ Forensic Features (8085)"
curl -s http://localhost:8090/health && echo " ✓ didi_brain (8090)"
echo ""
echo "All services operational ✅"
@ -403,6 +501,7 @@ echo "All services operational ✅"
| GPU | Model | VRAM Used | Total | Utilization |
|-----|-------|-----------|-------|-------------|
| **GPU 0** | Qwen3.5-35B-A3B (~57GB) + Whisper (~2GB) | ~59GB | 143GB | 41% |
| **GPU 1** | BusterX / Qwen2.5-VL-7B (~22GB) + BAAI/bge-m3 + BAAI/bge-reranker-v2-m3 | ~30GB | 143GB | 21% |
---
@ -422,6 +521,7 @@ PRODUCTION (1xxxx):
DEVELOPMENT (5xxxx):
├── 51100 Web API (Fact-checking)
├── 51300 Dashboard (Admin UI + API)
├── 54100 Embeddings API (BGE-M3 Embeddings)
│ ├── 54101 vLLM Server
│ └── 54110 llama.cpp Server
@ -429,8 +529,14 @@ DEVELOPMENT (5xxxx):
│ ├── 54201 vLLM Server
│ └── 54210 llama.cpp Server
├── 54300 Audio API (Whisper STT)
├── 54400 Extractors API (metadata/sentiment/OCR/NER/detect)
├── 54500 BusterX (Deepfake Vision)
└── 54600 Video API (Video Analysis)
├── 54600 Video API (Video Analysis)
└── 8080 SearXNG (intern, nepublicat) (metasearch, free tier)
OTHER (outside 5-digit schema):
├── 8085 Forensic Features (rPPG/lip-sync/forgery)
└── 8090 didi_brain (cache + RAG + fact-checking)
```
---

45
ai_platform/INDEX.md
View file

@ -25,7 +25,7 @@ Acest fisier descrie fiecare modul, fiecare fisier, fiecare clasa, fiecare funct
## 1. Arhitectura generala
Monorepo cu 8 module independente. Fiecare modul este un pachet Python instalabil cu FastAPI, containerizat in Docker, conectat pe reteaua comuna `didi-network`. Toate comunica prin HTTP intern. Singurul port expus extern este 11000 (gateway nginx).
Monorepo cu 13 module independente (audio, catalog-api, cloak, dashboard, didi_brain, embeddings, extractors, forensic_features, gateway, llm-inference, rerank, video-analysis, web). Fiecare modul este un pachet Python instalabil cu FastAPI, containerizat in Docker, conectat pe reteaua comuna `didi-network`. Toate comunica prin HTTP intern. Singurul port expus extern este 11000 (gateway nginx).
Flux tipic de request extern:
```
@ -39,12 +39,14 @@ Video API -> vLLM (BusterX)
Catalog API -> (interogheaza toate celelalte servicii pe /v1/info)
```
Modele ML servite:
- Qwen3.5-35B-A3B (text + vision, MoE) - vLLM pe GPU 0
- BAAI/bge-m3 (embeddings) - vLLM sau llama.cpp
- BAAI/bge-reranker-v2-m3 (reranking) - vLLM sau llama.cpp
Modele ML servite (7 modele, 13 module):
- Qwen3.5-35B-A3B (text + vision + OCR, MoE) - vLLM pe GPU 0, servit ca `qwen3.5`. Nu exista model vision separat: OCR/vision se face prin acest gateway LLM.
- BAAI/bge-m3 (embeddings) - vLLM sau llama.cpp, GPU 1
- BAAI/bge-reranker-v2-m3 (reranking) - vLLM sau llama.cpp, GPU 1
- Whisper large-v3-turbo (speech-to-text) - faster-whisper pe GPU 0
- BusterX / Qwen2.5-VL-7B (deepfake detection) - vLLM pe GPU 1
- BusterX (Qwen2.5-VL-7B fine-tuned, deepfake detection) - vLLM pe GPU 1
- GLiNER (backbone mdeberta, NER) + YOLOv8n (object detection) - extractors
- MediaPipe (rPPG / lip-sync / landmarks) - forensic_features
---
@ -63,7 +65,7 @@ Gateway-ul este un reverse proxy nginx care ruteaza toate request-urile catre se
Defineste 4 upstream-uri:
- `llm` -> `didiAI-llm-api:14011`
- `audio` -> `didiAI-audio-api:54300`
- `audio` -> `didiAI-audio:54300`
- `web` -> `didiAI-web-api:51100`
- `catalog` -> `didiAI-catalog-api:11000`
@ -547,7 +549,7 @@ Structura identica cu embeddings. Diferente specifice:
## 7. Audio - transcriere audio
**Locatie:** `modules/audio/`
**Container:** `didiAI-audio-api` (port 54300)
**Container:** `didiAI-audio` (port 54300)
### Ce face
@ -599,7 +601,7 @@ Rute:
**deploy/Dockerfile** - Bazat pe `nvidia/cuda:12.1.0-runtime-ubuntu22.04`. Instaleaza Python 3.10, ffmpeg. Nu foloseste uv, ci pip direct. Port 54300.
**deploy/docker-compose.yml** - Serviciu `audio-api`, container `didiAI-audio-api`. GPU 0 (CUDA_VISIBLE_DEVICES=0). Volum pentru cache modele. Profile: api. Start period 60s.
**deploy/docker-compose.yml** - Serviciu `audio-api`, container `didiAI-audio`. GPU 0 (CUDA_VISIBLE_DEVICES=0). Volum pentru cache modele. Profile: api. Start period 60s.
**deploy/deploy.sh** - Valideaza AUDIO_MODEL, AUDIO_DEVICE, AUDIO_CACHE_DIR. Suporta profile `api` si `api-nginx`.
@ -613,7 +615,7 @@ Rute:
### Ce face
Doua functionalitati:
1. Detectie deepfake - extrage 16 frame-uri uniforme, le trimite la BusterX (model fine-tuned pe Qwen2.5-VL-7B), obtine verdict REAL/FAKE/INCONCLUSIVE
1. Detectie deepfake - extrage 16 frame-uri uniforme, le trimite la BusterX (model fine-tuned pe Qwen2.5-VL-7B), obtine verdict REAL/FAKE/UNCERTAIN
2. Analiza semantica - divide video-ul in chunk-uri temporale (default 10s), extrage 24 frame-uri/chunk, descrie fiecare chunk cu LLM vision, optional agrega intr-un summary final
### Fisiere sursa
@ -653,10 +655,10 @@ Suporta configurare din `deploy/config.yaml` (YAML), cu override din variabile d
- `frame_to_data_url_b64jpeg(frame_bgr, max_side, jpeg_quality)` - converteste frame BGR la RGB PIL Image, scaleaza la max_side, encodeaza JPEG, returneaza data URI base64
- `call_vllm_chat(base_url, model, data_urls, prompt, max_tokens, temperature, repetition_penalty, timeout_s=180)` - construieste payload cu imagini + text, POST la `/v1/chat/completions`, masoara timpul. Returneaza (response JSON, elapsed_seconds)
- `parse_verdict_and_explanation(model_text)` - verifica primele 20 caractere (uppercase) pt prefix verdict. REAL/FAKE/altceva=INCONCLUSIVE.
- `parse_verdict_and_explanation(model_text)` - verifica primele 20 caractere (uppercase) pt prefix verdict. REAL/FAKE/altceva=UNCERTAIN.
**src/video_analysis/schemas.py**
- `Verdict = Literal["REAL", "FAKE", "INCONCLUSIVE"]`
- `Verdict = Literal["REAL", "FAKE", "UNCERTAIN"]`
- `Usage` - prompt_tokens, completion_tokens, total_tokens
- `LatencyS` - sampling_time_s, encode_time_s, model_inference_time_s
- `Meta` - fps, total_frames, duration_s, sampled, indices, timestamps_s
@ -696,7 +698,7 @@ Functii helper: `safe_mkdir()`, `write_json()`, `sha256_file()`.
**deploy/docker-compose.yml** - 2 servicii:
1. `vllm-buster` (didiAI-video-vllm-buster, port 54500) - Image: `vllm/vllm-openai:latest`. Model: `l8cv/BusterX_plusplus` (served as "busterx"). GPU 1. max-model-len 32768, gpu-memory-utilization 0.25, prefix caching activat. Profile: api-vllm. 600s start_period.
1. `vllm-buster` (didiAI-video-vllm-buster, port 54500) - Image: `vllm/vllm-openai:qwen3_5`. Model: `l8cv/BusterX_plusplus` (served as "busterx"). GPU 1. max-model-len 32768, gpu-memory-utilization 0.25, prefix caching activat. Profile: api-vllm. 600s start_period.
2. `video-analysis-api` (didiAI-video-api, port 54600) - FastAPI. Volum `../runs` montat la `/app/runs`. Profile: api, api-vllm.
**deploy/Dockerfile** - Multi-stage python:3.11-slim cu uv. Port 54600.
@ -717,7 +719,7 @@ Modul complex de fact-checking cu pipeline complet: detectie context -> cautare
**src/web/config.py** - `WebSettings(BaseSettings)` cu prefix `WEB_`:
Required:
- `searxng_base_url` - URL SearXNG (ex: http://localhost:55100)
- `searxng_base_url` - URL SearXNG (ex: http://didiAI-web-searxng:8080)
- `llm_base_url` - URL LLM inference server
- `external_url` - URL extern OpenAPI
@ -911,6 +913,7 @@ PRODUCTION (1xxxx):
DEVELOPMENT (5xxxx):
51100 Web API - fact-checking + cautare web
51300 Dashboard - admin UI + API (monitorizare AI)
54100 Embeddings API Dev
54101 vLLM Embed Server Dev
54110 llama.cpp Embed Server Dev
@ -918,8 +921,14 @@ DEVELOPMENT (5xxxx):
54201 vLLM Rerank Server Dev
54210 llama.cpp Rerank Server Dev
54300 Audio API - transcriere Whisper
54400 Extractors API - metadata/sentiment/OCR/NER/detect
54500 BusterX vLLM - server vision deepfake
54600 Video Analysis API - analiza video
8080 SearXNG (didiAI-web-searxng, intern; nepublicat pe host) - metasearch free tier web
ALTE PORTURI (in afara schemei 5-cifre):
8085 Forensic Features API - detectoare forensice (rPPG/lip-sync/forgery)
8090 didi_brain API - cache rezultate + RAG + fact-checking
```
Schema porturi: 5 cifre. Prima cifra: 1=prod, 5=dev. A doua cifra: 1=API/Gateway, 4=LLM/AI.
@ -931,7 +940,7 @@ Schema porturi: 5 cifre. Prima cifra: 1=prod, 5=dev. A doua cifra: 1=API/Gateway
| GPU | Ce ruleaza | VRAM folosit | VRAM total |
|-----|-----------|-------------|------------|
| GPU 0 | Qwen3.5-35B-A3B (~57GB) + Whisper large-v3-turbo (~2GB) | ~59GB | 143GB |
| GPU 1 | BusterX / Qwen2.5-VL-7B (~22GB) | ~22GB | 143GB |
| GPU 1 | BusterX / Qwen2.5-VL-7B (~22GB) + BAAI/bge-m3 + BAAI/bge-reranker-v2-m3 | ~30GB | 143GB |
---
@ -940,9 +949,9 @@ Schema porturi: 5 cifre. Prima cifra: 1=prod, 5=dev. A doua cifra: 1=API/Gateway
Toate containerele sunt pe reteaua externa `didi-network`. Comunicarea interna se face prin DNS Docker (nume containere):
```
didiAI-gateway -> didiAI-llm-api, didiAI-audio-api, didiAI-web-api, didiAI-catalog-api
didiAI-gateway -> didiAI-llm-api, didiAI-audio, didiAI-web-api, didiAI-catalog-api
didiAI-llm-api -> didiAI-vllm-qwen3.5
didiAI-catalog-api -> didiAI-llm-api, didiAI-audio-api, didiAI-video-api, didiAI-web-api
didiAI-catalog-api -> didiAI-llm-api, didiAI-audio, didiAI-video-api, didiAI-web-api
didiAI-web-api -> SearXNG, didiAI-llm-api
didiAI-video-api -> didiAI-video-vllm-buster, didiAI-llm-api (pt agregare semantica)
didiAI-embeddings-api -> didiAI-embeddings-vllm, didiAI-embeddings-llamacpp
@ -953,7 +962,7 @@ Naming convention containere: `didiAI-{modul}-{serviciu}`.
## Recent Changes (2026-05-05)
- **Login Keycloak SSO functional la `/admin-ai/`**: realm `didi-admins` (mutat din `didi-clients`), client `ai-platform-dashboard` (creat in didi-admins ca clona), required role `admin`. SSO comun cu admin-dashboard backend (1 login = ambele dashboard-uri).
- **Dashboard ruleaza in STAGING MODE (`DASHBOARD_STAGING_MODE=true`) — Keycloak este bypassed**: autentificarea SSO este configurata dar dezactivata in starea livrata. Config-ul pregatit: realm `didi-admins` (mutat din `didi-clients`), client `ai-platform-dashboard` (creat in didi-admins ca clona), required role `admin`, SSO comun cu admin-dashboard backend. **Inainte de productie: scoate `DASHBOARD_STAGING_MODE` pentru a activa auth-ul.**
- **AI dashboard env**: `VITE_KEYCLOAK_URL=https://sso.clossers.com`, `VITE_KEYCLOAK_REALM=didi-admins`, `VITE_KEYCLOAK_CLIENT_ID=ai-platform-dashboard`, `VITE_KEYCLOAK_REQUIRED_ROLE=admin`. Dual var pentru build (VITE_*) + runtime (DASHBOARD_*).
- **Schema config DB-overridable**: tabel nou `config_schema_override` (auto-creat la startup), helper `_merged_schema(session)` in `routes/config.py`, endpoint-uri admin `GET /api/config/schema/_overrides`, `PUT /api/config/schema/{key}`, `DELETE /api/config/schema/{key}`. Audit trail (action `config.schema.upsert/delete/seed`).
- **Migrare automata 98 chei -> DB**: la primul startup, `seed_schema_if_empty()` populeaza tabelul din `KNOWN_KEYS` (idempotent). Codul KNOWN_KEYS ramane fallback daca DB e sters. DB = single source of truth pentru schema acum.

23
ai_platform/ML_PROJECTS_DEPLOY_KEYS_AND_REQUIREMENTS.md
View file

@ -8,6 +8,10 @@
NU pe email/Slack public, NU în repo public, NU pe wiki. Pentru transfer:
scp criptat sau message DM.
> **Notă topologie:** acest document descrie topologia **multi-host de referință**
> (URL-uri/IP-uri distribuite pe mai multe mașini). Valorile reale pentru
> deployment-ul **single-host** livrat se află în `local_gpu_stack/.env.example`.
---
## Cuprins
@ -53,8 +57,8 @@ Stack-ul depinde de aceste hosts care rulează GPU-side:
|---|---|---|
| LLM Router (gateway Qwen) | `http://10.11.10.17:14011` | web, didi_brain, dashboard |
| vLLM Qwen3.5-35B (direct) | `http://10.11.10.17:14001` | dashboard (provider stats) |
| llama.cpp Qwen3.5-397B #1 | `http://10.11.10.18:14001` | LLM router upstream |
| llama.cpp Qwen3.5-397B #2 | `http://10.11.10.19:14001` | LLM router upstream |
| llama.cpp Qwen3.5-35B-A3B #1 | `http://10.11.10.18:14001` | LLM router upstream |
| llama.cpp Qwen3.5-35B-A3B #2 | `http://10.11.10.19:14001` | LLM router upstream |
| BGE-M3 embeddings | `http://10.11.10.15:8200` | didi_brain (Atomic auto-embeds) |
| BGE-reranker-v2-m3 | `http://10.11.10.15:8100` | didi_brain (gather rerank) |
| Whisper STT | (host-local, audio module) | audio module |
@ -188,13 +192,14 @@ LLM_VLLM_URL=http://localhost:14001
LLM_LLAMACPP_URLS=http://10.11.10.18:14001,http://10.11.10.19:14001
# Models
MODEL_REASONING=Qwen3.5-397B-A17B
MODEL_REASONING=Qwen/Qwen3.5-35B-A3B
MODEL_REASONING_BACKEND=llamacpp
MODEL_FAST=qwen3.5
MODEL_FAST_BACKEND=vllm
MODEL_FAST_ENABLED=false
MODEL_VISION=gemma-3-27b-it
MODEL_VISION_URL=http://10.11.10.16:8001
# Vision: nu exista model vision separat — OCR/vision se face prin Qwen3.5 (gateway LLM)
MODEL_VISION=qwen3.5
MODEL_VISION_URL=http://10.11.10.17:14011
MODEL_VISION_ENABLED=false
# Embeddings
@ -253,7 +258,7 @@ AUDIO_EXTERNAL_URL=http://localhost:54300
CATALOG_EXTERNAL_URL=http://localhost
CATALOG_PORT=11000
CATALOG_LLM_URL=http://didiAI-llm-api:14011
CATALOG_AUDIO_URL=http://didiAI-audio-api:54300
CATALOG_AUDIO_URL=http://didiAI-audio:54300
CATALOG_WEB_URL=http://didiAI-web-api:51100
CATALOG_VIDEO_URL=
```
@ -273,9 +278,9 @@ GATEWAY_API_TOKEN=0x2-m0W5oG7MjZUu5v3-ejBWMExwLeGDv4aRKqBI7bM
| 51100 | Web API (gather, search, fetch) | `didiAI-web-api` |
| 51300 | Dashboard UI + API | `didiAI-dashboard` |
| 15432 | Dashboard Postgres (debug only) | `didiAI-dashboard-db` |
| 54300 | Audio STT proxy | `didiAI-audio-proxy` |
| 54300 | Audio STT | `didiAI-audio` |
| 54600 | Video Analysis | `didiAI-video-api` |
| 55100 | SearXNG metasearch | `didiAI-web-searxng` |
| 8080 (intern) | SearXNG metasearch | `didiAI-web-searxng` |
| 8090 | didi-brain HTTP API | `didibrain-api` |
| 8088 | didi-brain Atomic | `didibrain-atomic` |
| 5434 | didi-brain Postgres (debug only) | `didibrain-postgres` |
@ -360,7 +365,7 @@ BRAIN_IMPORT_CORPUS=0 BRAIN_RUN_EXTRACTION=0 ./scripts/bootstrap_deploy.sh
### 8. Conectează web-api la rețeaua didibrain
Web-api rulează pe network `deploy_default`, didi-brain pe `didibrain`. Trebuie unit:
Web-api rulează pe network `didi-network`, didi-brain pe `didibrain`. Trebuie unit:
```bash
docker network connect didibrain didiAI-web-api

44
ai_platform/README.md
View file

@ -245,13 +245,21 @@ Port allocation follows datacenter schema (5-digit ports):
| Port | Service | Environment |
|------|---------|-------------|
| 11000 | Catalog API | Production |
| 14001 | Qwen3.5-35B-A3B | Production |
| 14011 | LLM API Gateway | Production |
| 11000 | Gateway (nginx, external entry) | Production |
| 11000 | Catalog API (internal, via gateway) | Production |
| 14001 | vLLM Qwen3.5-35B-A3B | Production |
| 14011 | LLM Inference API (router) | Production |
| 14100 | Embeddings API | Production |
| 14200 | Rerank API | Production |
| 51100 | Web API | Development |
| 51300 | Dashboard | Development |
| 54300 | Audio API | Development |
| 54500 | BusterX | Development |
| 54600 | Video API | Development |
| 54400 | Extractors API | Development |
| 54500 | BusterX vLLM | Development |
| 54600 | Video Analysis API | Development |
| 8080 (intern) | SearXNG | Development |
| 8085 | Forensic Features API | Other |
| 8090 | didi_brain API | Other |
**Guidelines:**
- Use 1xxxx for production services
@ -377,11 +385,19 @@ Reviewers will check for:
| Module | Port | Description | Status |
|--------|------|-------------|--------|
| [catalog-api](modules/catalog-api/) | 11000 | Service catalog & discovery gateway | Active |
| [llm-inference](modules/llm-inference/) | 14011 | Unified LLM inference with multiple backends | Active |
| [audio](modules/audio/) | 54300 | Speech-to-text (Whisper) | Active |
| [video-analysis](modules/video-analysis/) | 54600 | Deepfake detection & semantic video analysis | Active |
| [gateway](modules/gateway/) | 11000 | nginx reverse proxy — single external entry point | Active |
| [catalog-api](modules/catalog-api/) | 11000 | Service catalog & discovery aggregator (internal) | Active |
| [llm-inference](modules/llm-inference/) | 14011 | Unified LLM inference (Qwen3.5-35B-A3B) with multiple backends | Active |
| [embeddings](modules/embeddings/) | 14100 | OpenAI-compatible embeddings (BAAI/bge-m3) | Active |
| [rerank](modules/rerank/) | 14200 | Document reranking (BAAI/bge-reranker-v2-m3) | Active |
| [web](modules/web/) | 51100 | Web scraping & fact-checking evidence gathering | Active |
| [dashboard](modules/dashboard/) | 51300 | Admin UI + API (monitoring, config, RBAC) | Active |
| [audio](modules/audio/) | 54300 | Speech-to-text (Whisper large-v3-turbo) | Active |
| [extractors](modules/extractors/) | 54400 | Feature extraction: metadata/sentiment/OCR/NER (GLiNER)/detect (YOLOv8n) | Active |
| [video-analysis](modules/video-analysis/) | 54600 | Deepfake detection (BusterX) & semantic video analysis | Active |
| [forensic_features](modules/forensic_features/) | 8085 | Forensic detectors (rPPG/lip-sync/forgery) via MediaPipe | Active |
| [didi_brain](modules/didi_brain/) | 8090 | Result cache (Postgres/pgvector) + RAG + fact-checking | Active |
| [cloak](modules/cloak/) | internal | Stealth SERP scraper (tier-3 fallback for web) | Active |
## Port Allocation Summary
@ -396,12 +412,20 @@ Port allocation follows datacenter schema (5-digit ports):
| Port | Service | Module | Environment |
|------|---------|--------|-------------|
| 11000 | Catalog API | catalog-api | Production |
| 11000 | Gateway (nginx, external entry) | gateway | Production |
| 11000 | Catalog API (internal) | catalog-api | Production |
| 14001 | vLLM Qwen3.5-35B-A3B | llm-inference | Production |
| 14011 | LLM API Gateway | llm-inference | Production |
| 14100 | Embeddings API | embeddings | Production |
| 14200 | Rerank API | rerank | Production |
| 51100 | Web API | web | Development |
| 51300 | Dashboard | dashboard | Development |
| 54300 | Audio API (Whisper) | audio | Development |
| 54400 | Extractors API | extractors | Development |
| 54500 | BusterX vLLM | video-analysis | Development |
| 54600 | Video Analysis API | video-analysis | Development |
| 8080 (intern) | SearXNG | web | Development |
| 8085 | Forensic Features API | forensic_features | Other |
| 8090 | didi_brain API | didi_brain | Other |
For detailed endpoint documentation, see [ENDPOINTS.md](ENDPOINTS.md).

43
ai_platform/local_gpu_stack/.env.example Normal file
View file

@ -0,0 +1,43 @@
# =============================================================================
# DiDi LOT 1 — Local GPU stack configuration
# Copy to `.env` (deploy.sh face asta automat daca lipseste) si ajusteaza.
# =============================================================================
# Host IP folosit in EXTERNAL_URL-uri si in comenzile de test.
# Lasa GOL -> deploy.sh detecteaza automat IP-ul principal si il scrie aici.
HOST_IP=
# --- Brain / Atomic / Postgres -----------------------------------------------
# Parola e folosita de postgres, atomic-server SI brain-api (trebuie sa fie una singura).
POSTGRES_PASSWORD=brain_local_dev
# Token API atomic-server. Lasa GOL -> deploy.sh il genereaza automat
# (atomic-server token create) si il scrie inapoi aici.
ATOMIC_TOKEN=
# --- Gateway -----------------------------------------------------------------
GATEWAY_API_TOKEN=didi-local-dev-token-123
# --- Dashboard ---------------------------------------------------------------
DASHBOARD_DB_USER=dashboard
DASHBOARD_DB_PASSWORD=devpass123
DASHBOARD_DB_NAME=dashboard
# staging=true -> bypass Keycloak (intri direct in UI). Scoate in productie.
DASHBOARD_STAGING_MODE=true
# --- Modele (nume servite; nu schimba fara motiv) ----------------------------
LLM_MODEL=Qwen/Qwen3.5-35B-A3B
LLM_SERVED_NAME=qwen3.5
EMB_MODEL=BAAI/bge-m3
RERANK_MODEL=BAAI/bge-reranker-v2-m3
WHISPER_MODEL=large-v3-turbo
WHISPER_REPO=mobiuslabsgmbh/faster-whisper-large-v3-turbo
BUSTER_MODEL=l8cv/BusterX_plusplus
# Extractoare (NER) — GLiNER + backbone-ul lui
NER_MODEL=urchade/gliner_multi-v2.1
NER_BACKBONE=microsoft/mdeberta-v3-base
# Zona LOCALA unica pentru TOATE modelele (un folder per model, format HF cache:
# models--<org>--<name>/). Toate containerele (Qwen/BusterX/bge/whisper/GLiNER)
# sunt servite OFFLINE de aici. Lasa GOL -> deploy.sh foloseste ai_platform/models.
# Ce lipseste se descarca automat din HF in acest director (offline-first).
MODELS_DIR=

74
ai_platform/local_gpu_stack/README.md Normal file
View file

@ -0,0 +1,74 @@
# DiDi LOT 1 — Local GPU stack (deploy progresiv)
Deploy „cărămidă cu cărămidă" al întregii platforme pe un host cu GPU.
Pentru fiecare componentă: **build/pull + download → seed (dacă e cazul) →
TEST funcțional → doar dacă testul trece corect → următoarea**. Idempotent,
se oprește la primul test picat.
## Cerințe (verificate de preflight, nu se instalează automat)
- Docker Engine 24+ cu plugin `docker compose` v2
- NVIDIA driver ≥535 + `nvidia-container-toolkit` (runtime configurat în Docker)
- ≥1 GPU (proiectat pe 2× H200; GPU 0 = LLM+Whisper, GPU 1 = BusterX+bge×2)
- ~100 GB liberi pe `/` (toate modelele ≈ 94 GB)
## Modele — o singură zonă locală
Toate modelele stau în **`ai_platform/models/`** (format HF cache, un folder per model),
servite **offline** de acolo (fără rețea la runtime). `MODELS_DIR` în `.env` (auto = acest folder).
Ce lipsește se descarcă o singură dată în el (`deploy.sh` pasul DOWNLOAD). NU se comite în git.
| Folder | Model | Folosit de |
|---|---|---|
| `models--Qwen--Qwen3.5-35B-A3B` | LLM (67G) | llm (vLLM) |
| `models--l8cv--BusterX_plusplus` | deepfake (16G, Qwen2.5-VL intern) | video (vLLM) |
| `models--BAAI--bge-m3` | embeddings | embeddings (vLLM) |
| `models--BAAI--bge-reranker-v2-m3` | reranker | rerank (vLLM) |
| `models--mobiuslabsgmbh--faster-whisper-large-v3-turbo` | STT | audio |
| `models--urchade--gliner_multi-v2.1` + `models--microsoft--mdeberta-v3-base` | NER + backbone | extractors |
Copiezi `ai_platform/models/` pe altă mașină → zero download la deploy.
## Utilizare
```bash
cd ai_platform/local_gpu_stack
./deploy.sh # tot, de la preflight la dashboard
./deploy.sh --from web # reia de la cărămida 'web' (sare peste cele dinainte)
./deploy.sh --only llm # rulează o singură cărămidă (presupune deps deja up)
./deploy.sh --install-deps # instalează Docker + nvidia-container-toolkit (sudo), apoi continuă
```
`.env` se creează automat din `.env.example` la prima rulare.
### Mașină nouă (VM „chioară" cu GPU)
1. (o singură dată) driver NVIDIA ≥535 + reboot — `sudo ubuntu-drivers install` (nu se automatizează, cere reboot).
2. `git clone <repo> && cd .../ai_platform/local_gpu_stack`
3. `./deploy.sh --install-deps` → instalează Docker+toolkit, apoi:
- **HOST_IP** se auto-detectează și se scrie în `.env`
- **BusterX** se descarcă automat din HF dacă nu e pe disc (`BUSTER_HF_CACHE`)
- Qwen/bge se descarcă (cu progres), totul se ridică cărămidă cu cărămidă cu test la fiecare.
> Singurul pas care NU e o comandă: driverul NVIDIA (reboot). Restul e `./deploy.sh`.
Cărămizi, în ordine: `embeddings · rerank · llm · extractors · audio · video · forensic · web · catalog · gateway · brain · dashboard`.
La final: dashboard pe `http://<HOST_IP>:51300/admin-ai/` (staging mode, fără login).
## Ce automatizează (fixurile care înainte erau manuale)
- rețea unică `didi-network`; plasare GPU 0/1; nume containere `didiAI-*`
- base-URL vLLM **fără** `/v1` (codul îl adaugă) pentru llm/embeddings/video
- YOLO scrie modelul în `/tmp` (cwd e read-only la non-root)
- gateway LLM injectează `enable_thinking:false` → JSON curat din Qwen3.5 (sentiment/OCR/semantic/brain)
- SearXNG fără proxy outbound, ca serviciu compose
- brain: aliniere parolă postgres, creare **token atomic** (auto, salvat în `.env`), **seed taxonomie** (79 tag-uri)
- dashboard: `.env` cu staging + override-uri health; **seed catalog** (9 intrări)
## Fișiere
| Fișier | Rol |
|---|---|
| `deploy.sh` | orchestratorul progresiv gated |
| `lib.sh` | logging, `wait_health`, `gate`, shim docker (merge și prin `sg docker`) |
| `docker-compose.yml` | stack-ul GPU + non-GPU + searxng |
| `.env.example` | configurația (host, porturi, parole, token) |
| `seed/seed_catalog.sh` | populează catalogul DB al dashboard-ului (idempotent) |
| `seed/seed_brain.sh` | token atomic + taxonomie + env brain |
| `TESTING.md` | comenzi de testare per componentă + smoke-test |
> Brain rulează din `modules/didi_brain/infra/docker-compose.yml`; dashboard și gateway
> din `modules/*/deploy/` — `deploy.sh` le orchestrează pe toate sub proiectul `local_gpu_stack`.

228
ai_platform/local_gpu_stack/TESTING.md Normal file
View file

@ -0,0 +1,228 @@
# DiDi LOT 1 — Comenzi de testare per componentă
Host: **10.11.10.18**. Toate comenzile sunt `curl` copy-paste. Statusul din dreptul
fiecăreia e rezultatul verificat la ultima rulare.
> Convenție: `H=10.11.10.18`. Rulează întâi `export H=10.11.10.18`.
---
## 0. Pregătire fișiere de test (o singură dată)
```bash
export H=10.11.10.18
cd /home/topclossers/didi-lot1-ai/ai_platform/local_gpu_stack
mkdir -p test_assets
# video mic din artefacte
cp "$(find ../../artefacte_lot1/runs_deepfake -iname '*.mp4' -printf '%s\t%p\n' | sort -n | head -1 | cut -f2)" test_assets/sample.mp4
# extrage un cadru (jpg) + audio (wav) cu ffmpeg din containerul extractors
docker cp test_assets/sample.mp4 didiAI-extractors:/tmp/s.mp4
docker exec didiAI-extractors sh -c 'ffmpeg -y -i /tmp/s.mp4 -frames:v 1 /tmp/frame.jpg -vn -ar 16000 -ac 1 -t 8 /tmp/audio.wav'
docker cp didiAI-extractors:/tmp/frame.jpg test_assets/frame.jpg
docker cp didiAI-extractors:/tmp/audio.wav test_assets/audio.wav
cd test_assets # restul comenzilor presupun că ești aici pentru fișiere
```
---
## 1. LLM — Qwen3.5-35B-A3B (port 14011) ✅
```bash
# modele servite
curl -s http://$H:14011/v1/models | python3 -m json.tool
# chat completion
curl -s http://$H:14011/v1/chat/completions -H 'Content-Type: application/json' -d '{
"model":"qwen3.5",
"messages":[{"role":"user","content":"Ce este un deepfake? Raspunde scurt."}],
"max_tokens":200
}' | python3 -m json.tool
# health
curl -s http://$H:14011/health
```
> Notă: Qwen3.5 e model de **reasoning** — răspunsul conține un bloc „Thinking" înainte de
> răspunsul final. Normal.
## 2. Embeddings — bge-m3 (port 14100) ✅
```bash
curl -s http://$H:14100/v1/embeddings -H 'Content-Type: application/json' -d '{
"model":"bge-m3","input":"text de analizat"
}' | python3 -c "import sys,json;print('dim:',len(json.load(sys.stdin)['data'][0]['embedding']))"
```
> Așteptat: `dim: 1024`.
## 3. Rerank — bge-reranker-v2-m3 (port 14200) ✅
```bash
curl -s http://$H:14200/v1/rerank -H 'Content-Type: application/json' -d '{
"model":"bge-reranker-v2-m3",
"query":"detectie deepfake",
"documents":["o pisica neagra","sistem de detectie deepfake pentru video"]
}' | python3 -m json.tool
```
> Așteptat: documentul relevant primește scor mult mai mare (index 1).
## 4. Audio / Whisper — large-v3-turbo (port 54300) ✅
```bash
curl -s http://$H:54300/v1/audio/transcriptions \
-F "file=@audio.wav" -F "model=large-v3-turbo" | python3 -m json.tool
# modele / info
curl -s http://$H:54300/v1/models | python3 -m json.tool
```
> Acceptă și `.mp4` direct (ffmpeg în container). Returnează `text` + `language`.
## 5. Video / BusterX++ (deepfake) (port 54600) ✅
```bash
# verdict deepfake REAL/FAKE/UNCERTAIN
curl -s http://$H:54600/analyze/video -F "file=@sample.mp4" | python3 -m json.tool
# analiza semantica (descriere continut, foloseste si LLM-ul)
curl -s http://$H:54600/analyze/video/semantic -F "file=@sample.mp4" | python3 -m json.tool
curl -s http://$H:54600/v1/info | python3 -m json.tool
```
> Așteptat: `{"verdict":"REAL"|"FAKE"|"UNCERTAIN","frames_analyzed":16,"evidence":[...]}`.
## 6. Extractors (port 54400)
```bash
# 6a. metadata: EXIF/ELA/ffprobe/integritate/C2PA ✅
curl -s http://$H:54400/v1/metadata -F "file=@sample.mp4" | python3 -m json.tool
# 6b. NER (GLiNER, multilingv) ✅
curl -s http://$H:54400/v1/ner -H 'Content-Type: application/json' -d '{
"text":"Klaus Iohannis s-a intalnit cu Emmanuel Macron la Bucuresti."
}' | python3 -m json.tool
# 6c. detectie obiecte (YOLOv8) ✅
curl -s http://$H:54400/v1/detect -F "file=@frame.jpg" | python3 -m json.tool
# 6d. OCR (LLM vision) ✅
curl -s http://$H:54400/v1/ocr -F "file=@frame.jpg" | python3 -m json.tool
# 6e. sentiment (deleaga LLM) ✅
curl -s http://$H:54400/v1/sentiment -H 'Content-Type: application/json' -d '{
"text":"Produsul este excelent, sunt foarte multumit!"
}' | python3 -m json.tool
```
> Așteptat: `{"ok":true,"results":{"label":"pozitiv","score":0.9,"rationale":"..."}}`.
> (Gateway-ul LLM injectează `enable_thinking:false` pe vLLM → JSON curat din modelul de reasoning.)
## 7. Forensic — rPPG / lip-sync / forgery / lighting (port 8085) ✅
```bash
# module disponibile
curl -s http://$H:8085/api/forensic-modules | python3 -m json.tool
# analiza forensica (multipart; campul fisierului se numeste 'video')
curl -s http://$H:8085/api/forensic-evidence \
-F "video=@sample.mp4" -F "modules=m25,m26,m27,m28,m29" -F "every_n_frames=10" \
| python3 -m json.tool
```
> Rulează sincron implicit; returnează `modules_run`, `n_frames_extracted`, evidence per modul.
> Pentru async: adaugă `-F "async_mode=true"` → primești `job_id`, apoi
> `GET /api/status/{job_id}` și `GET /api/result/{job_id}`.
## 8. Web — căutare & fact-checking evidence (port 51100) ✅
```bash
# cautare (SearXNG + provideri); ATENTIE: campul e "queries" (lista)
curl -s http://$H:51100/v1/search -H 'Content-Type: application/json' -d '{
"queries":["deepfake detection 2024"],"max_results":5
}' | python3 -m json.tool
# fetch pagini (camp "urls" = LISTA)
curl -s http://$H:51100/v1/fetch -H 'Content-Type: application/json' -d '{
"urls":["https://en.wikipedia.org/wiki/Deepfake"]
}' | python3 -m json.tool
# gather (cautare + fetch + evidence pack); camp "claim", max_search_results >= 5
curl -s http://$H:51100/v1/gather -H 'Content-Type: application/json' -d '{
"claim":"Deepfakes can be detected by AI","max_search_results":5
}' | python3 -m json.tool
curl -s http://$H:51100/v1/info | python3 -m json.tool
```
## 9. Catalog — service discovery (port 11000, doar prin gateway/intern) ✅
`catalog-api` nu e publicat pe host; se accesează prin **gateway** (Bearer) sau din rețea.
```bash
TOK=didi-local-dev-token-123
curl -s http://$H:11000/catalog/v1/status -H "Authorization: Bearer $TOK" | python3 -m json.tool
curl -s http://$H:11000/catalog/v1/components -H "Authorization: Bearer $TOK" | python3 -m json.tool
curl -s http://$H:11000/catalog/v1/models -H "Authorization: Bearer $TOK" | python3 -m json.tool
```
## 10. Gateway — nginx reverse proxy + auth (port 11000) ✅
```bash
TOK=didi-local-dev-token-123
curl -s http://$H:11000/health # fara auth -> ok
curl -s http://$H:11000/web/v1/info -H "Authorization: Bearer $TOK" # rutare -> web
curl -s http://$H:11000/catalog/v1/status -H "Authorization: Bearer $TOK" # rutare -> catalog
curl -s http://$H:11000/web/v1/info # fara token -> 401
```
## 11. Brain — RAG + fact-checking + cache (port 8090) ✅
```bash
# cautare in baza de cunostinte (contract web; camp "queries")
curl -s http://$H:8090/v1/search -H 'Content-Type: application/json' -d '{
"queries":["deepfake"],"max_results":3
}' | python3 -m json.tool
# gather (RAG complet: search -> fetch -> evidence); camp "claim"
curl -s http://$H:8090/v1/gather -H 'Content-Type: application/json' -d '{
"claim":"Deepfakes can be detected by AI"
}' | python3 -m json.tool
# statistici atom store (folosit de dashboard "Brain — fact status")
curl -s http://$H:8090/v1/analysis_atom/stats/extended | python3 -m json.tool
curl -s http://$H:8090/health
```
> `results: []` la început e normal — baza de cunoștințe e goală până se indexează conținut.
---
## Smoke-test rapid (toate componentele, un singur script)
```bash
export H=10.11.10.18; TOK=didi-local-dev-token-123
ok(){ printf "%-26s %s\n" "$1" "$2"; }
ok "llm" "$(curl -s -m60 http://$H:14011/v1/chat/completions -H 'Content-Type: application/json' -d '{"model":"qwen3.5","messages":[{"role":"user","content":"hi"}],"max_tokens":5}' -o /dev/null -w %{http_code})"
ok "embeddings" "$(curl -s -m20 http://$H:14100/v1/embeddings -H 'Content-Type: application/json' -d '{"model":"bge-m3","input":"x"}' -o /dev/null -w %{http_code})"
ok "rerank" "$(curl -s -m20 http://$H:14200/v1/rerank -H 'Content-Type: application/json' -d '{"model":"bge-reranker-v2-m3","query":"a","documents":["a","b"]}' -o /dev/null -w %{http_code})"
ok "audio" "$(curl -s -m10 http://$H:54300/health -o /dev/null -w %{http_code})"
ok "video" "$(curl -s -m10 http://$H:54600/health -o /dev/null -w %{http_code})"
ok "extractors" "$(curl -s -m10 http://$H:54400/health -o /dev/null -w %{http_code})"
ok "forensic" "$(curl -s -m10 http://$H:8085/health -o /dev/null -w %{http_code})"
ok "web" "$(curl -s -m10 http://$H:51100/health -o /dev/null -w %{http_code})"
ok "gateway" "$(curl -s -m10 http://$H:11000/health -o /dev/null -w %{http_code})"
ok "catalog" "$(curl -s -m10 http://$H:11000/catalog/v1/status -H "Authorization: Bearer $TOK" -o /dev/null -w %{http_code})"
ok "brain" "$(curl -s -m10 http://$H:8090/health -o /dev/null -w %{http_code})"
# 200 peste tot = totul up
```
---
## Probleme rezolvate (istoric)
- **sentiment / output JSON** ✅ — gateway-ul LLM injectează acum `enable_thinking:false`
pe vLLM (`LLM_VLLM_DISABLE_THINKING`, default true), deci modelul de reasoning întoarce
JSON curat pentru sentiment/OCR/semantic/brain.
- **catalog `degraded`** ✅ — aliniat prin `CATALOG_LLM_URL/AUDIO_URL/VIDEO_URL/WEB_URL`
în compose → acum `healthy`.
- **brain taxonomy 401** ✅ — creat token atomic (`atomic-server token create`), setat
`ATOMIC_TOKEN`, seed-uită taxonomia (79 tag-uri în atomic), parolele postgres aliniate
(`infra/.env`). Scheduler-ul feed-uiește acum cu `errors=0`.
## Rămas minor (non-blocant)
- **atomic embeddings** — atomic-server logează „OpenRouter API key not configured, skipping
embedding". Atomic-server și-ar face vectori proprii prin OpenRouter; nu e nevoie — brain
folosește bge-m3 local pentru embeddings. Doar un warning în logul atomic.

298
ai_platform/local_gpu_stack/deploy.sh Normal file
View file

@ -0,0 +1,298 @@
#!/usr/bin/env bash
# =============================================================================
# DiDi LOT 1 — deploy progresiv „cărămidă cu cărămidă".
# Pentru fiecare componentă: build/pull + download → seed (dacă e cazul) →
# TEST funcțional → doar dacă testul TRECE corect → trece la următoarea.
# Idempotent. STOP la primul test picat.
#
# ./deploy.sh # tot, de la preflight la dashboard
# ./deploy.sh --from web # sare peste cărămizile dinainte de 'web'
# ./deploy.sh --only llm # rulează o singură cărămidă (presupune deps up)
# =============================================================================
set -uo pipefail
cd "$(dirname "$0")"
HERE=$(pwd)
export COMPOSE_PROJECT_NAME=local_gpu_stack
COMPOSE="${HERE}/docker-compose.yml"
ENV_FILE="${HERE}/.env"
MODULES="${HERE}/../modules"
DASH_DIR="${MODULES}/dashboard/deploy"
GW_DIR="${MODULES}/gateway/deploy"
BRAIN_DIR="${MODULES}/didi_brain"
ASSETS="${HERE}/test_assets"
source "${HERE}/lib.sh"
source "${HERE}/seed/seed_catalog.sh"
source "${HERE}/seed/seed_brain.sh"
# ---- .env -------------------------------------------------------------------
[ -f "$ENV_FILE" ] || { cp "${HERE}/.env.example" "$ENV_FILE"; info ".env creat din .env.example"; }
set -a; source "$ENV_FILE"; set +a
# auto-detect HOST_IP daca e gol (si scrie-l inapoi in .env pt interpolarea compose)
if [ -z "${HOST_IP:-}" ]; then
HOST_IP=$(hostname -I 2>/dev/null | awk '{print $1}')
[ -z "$HOST_IP" ] && HOST_IP=$(ip -4 route get 1.1.1.1 2>/dev/null | awk '{for(i=1;i<=NF;i++)if($i=="src")print $(i+1)}')
[ -n "$HOST_IP" ] && { sed -i "s#^HOST_IP=.*#HOST_IP=${HOST_IP}#" "$ENV_FILE"; info "HOST_IP detectat automat: ${HOST_IP}"; }
fi
export HOST_IP
H="${HOST_IP:-127.0.0.1}"
# MODELS_DIR: zona locala unica pentru TOATE modelele (default: ai_platform/models)
if [ -z "${MODELS_DIR:-}" ]; then
MODELS_DIR="$(cd "${HERE}/.." && pwd)/models"
sed -i "s#^MODELS_DIR=.*#MODELS_DIR=${MODELS_DIR}#" "$ENV_FILE"
info "MODELS_DIR: ${MODELS_DIR}"
fi
mkdir -p "$MODELS_DIR"; export MODELS_DIR
# ---- arg parsing ------------------------------------------------------------
FROM=""; ONLY=""; INSTALL_DEPS=0
while [ $# -gt 0 ]; do case "$1" in
--from) FROM=$2; shift 2;; --only) ONLY=$2; shift 2;;
--install-deps) INSTALL_DEPS=1; shift;;
*) die "argument necunoscut: $1";; esac; done
SKIP=0; [ -n "$FROM" ] && SKIP=1
should_run(){ # $1 = brick name
[ -n "$ONLY" ] && { [ "$ONLY" = "$1" ]; return; }
[ "$SKIP" = 1 ] && { [ "$FROM" = "$1" ] && SKIP=0; }
[ "$SKIP" = 0 ]
}
# =============================================================================
# PREFLIGHT (verifică + instruiește; nu instalează)
# =============================================================================
# =============================================================================
# INSTALL DEPS (optional, --install-deps; necesita sudo; Ubuntu/Debian)
# =============================================================================
install_deps() {
brick "INSTALL DEPS — Docker + NVIDIA Container Toolkit (sudo)"
command -v sudo >/dev/null || die "sudo necesar pentru --install-deps"
if ! command -v docker >/dev/null; then
info "instalez Docker Engine..."; curl -fsSL https://get.docker.com | sudo sh || die "instalare Docker a picat"
sudo usermod -aG docker "$USER" || true
warn "Adăugat în grupul docker — poate fi nevoie de re-login pentru ca 'docker' să meargă fără sudo."
else ok "Docker prezent"; fi
if ! docker info 2>/dev/null | grep -qiE 'nvidia|cdi' && ! sudo docker info 2>/dev/null | grep -qiE 'nvidia|cdi'; then
info "instalez nvidia-container-toolkit..."
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list \
| sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' \
| sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list >/dev/null
sudo apt-get update -y && sudo apt-get install -y nvidia-container-toolkit || die "instalare toolkit a picat"
sudo nvidia-ctk runtime configure --runtime=docker && sudo systemctl restart docker
ok "nvidia-container-toolkit instalat & Docker reconfigurat"
else ok "NVIDIA runtime prezent"; fi
command -v nvidia-smi >/dev/null || warn "Driver NVIDIA lipsește — instalează manual (ex: sudo ubuntu-drivers install) și REBOOT, apoi re-rulează deploy.sh."
}
preflight() {
brick "PREFLIGHT — unelte & GPU"
command -v docker >/dev/null || die "Docker lipsește. Instalează: https://docs.docker.com/engine/install/ubuntu/"
if ! docker ps >/dev/null 2>&1; then
[ "$USE_SG" = 1 ] && warn "rulez docker prin 'sg docker' (userul nu e în grupul docker activ)" \
|| die "Nu pot accesa Docker. Adaugă userul în grup: sudo usermod -aG docker \$USER (apoi re-login)"
fi
docker compose version >/dev/null 2>&1 || die "Docker Compose v2 lipsește (plugin 'docker compose')."
command -v nvidia-smi >/dev/null || die "nvidia-smi lipsește — instalează driverul NVIDIA (>=535) + CUDA."
local ngpu; ngpu=$(nvidia-smi --query-gpu=name --format=csv,noheader 2>/dev/null | wc -l)
[ "$ngpu" -ge 1 ] || die "Niciun GPU detectat de nvidia-smi."
if ! docker info 2>/dev/null | grep -qiE 'nvidia|cdi'; then
die "Runtime-ul NVIDIA pt Docker lipsește. Instalează nvidia-container-toolkit:
https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html
apoi: sudo nvidia-ctk runtime configure --runtime=docker && sudo systemctl restart docker"
fi
command -v curl >/dev/null || die "curl lipsește (apt-get install -y curl)."
command -v python3 >/dev/null || die "python3 lipsește."
local freeg; freeg=$(df -BG --output=avail / | tail -1 | tr -dc '0-9')
[ "${freeg:-0}" -ge 80 ] || warn "Doar ${freeg}G liberi pe / — Qwen are ~72GB. Recomandat >100G."
ok "Docker $(docker --version | grep -oE '[0-9]+\.[0-9]+\.[0-9]+' | head -1) · ${ngpu} GPU · ${freeg}G liberi"
}
# =============================================================================
# NETWORK
# =============================================================================
ensure_network() {
brick "REȚEA — didi-network (una singură, partajată)"
if docker network inspect didi-network >/dev/null 2>&1; then ok "didi-network există"
else docker network create didi-network >/dev/null && ok "didi-network creată"; fi
}
# =============================================================================
# BUILD images (app) + PULL (vLLM/searxng)
# =============================================================================
build_images() {
brick "BUILD — imagini aplicație"
local m
declare -A MAP=(
[llm-inference]=didiai-llm-inference [embeddings]=didiai-embeddings [rerank]=didiai-rerank
[audio]=didiai-audio [video-analysis]=didiai-video-analysis [extractors]=didiai-extractors
[web]=didiai-web [catalog-api]=didiai-catalog-api )
for m in "${!MAP[@]}"; do
printf ' build %-16s ' "$m"
if docker build -q -t "${MAP[$m]}:audit" -f "${MODULES}/$m/deploy/Dockerfile" "${MODULES}/$m" >/tmp/build_$m.log 2>&1; then ok "ok"
else err "FAIL"; tail -15 /tmp/build_$m.log | sed 's/^/ /'; die "build $m a picat"; fi
done
# forensic (layout non-standard: Dockerfile la root)
printf ' build %-16s ' "forensic"
if docker build -q -t didiai-forensic:audit -f "${MODULES}/forensic_features/Dockerfile" "${MODULES}/forensic_features" >/tmp/build_forensic.log 2>&1; then ok "ok"
else err "FAIL"; tail -15 /tmp/build_forensic.log | sed 's/^/ /'; die "build forensic a picat"; fi
gate "imagini prezente" bash -c 'docker images --format "{{.Repository}}" | grep -q didiai-llm-inference'
}
pull_images() {
brick "PULL — runtime-uri vLLM + SearXNG"
local img
for img in vllm/vllm-openai:v0.8.5 vllm/vllm-openai:qwen3_5 docker.io/searxng/searxng:latest; do
printf ' pull %-34s ' "$img"
if docker image inspect "$img" >/dev/null 2>&1; then ok "deja prezent"
elif docker pull -q "$img" >/dev/null 2>&1; then ok "ok"
else die "pull $img a picat (verifică rețeaua / Docker Hub)"; fi
done
}
# =============================================================================
# PRE-DOWNLOAD modele (cu progres)
# =============================================================================
predownload_models() {
brick "DOWNLOAD — modele în MODELS_DIR (offline-first, un folder per model)"
mkdir -p "${MODELS_DIR}"
# mini-script de download (evită iadul de ghilimele prin sg docker)
local dl="${HERE}/.dl_model.py"
printf 'import os\nfrom huggingface_hub import snapshot_download\nsnapshot_download(os.environ["REPO"])\nprint("OK", os.environ["REPO"])\n' > "$dl"
# repo HF -> subfolder cache (models--<org>--<name>)
local repo cache
for repo in "$BUSTER_MODEL" "$LLM_MODEL" "$EMB_MODEL" "$RERANK_MODEL" "$WHISPER_REPO" "$NER_MODEL" "$NER_BACKBONE"; do
cache="models--$(printf '%s' "$repo" | sed 's#/#--#')"
if [ -d "${MODELS_DIR}/${cache}/snapshots" ]; then
ok "prezent local: ${repo}"
else
info "download: ${repo}"
docker run --rm -e HF_HOME=/c -e "REPO=${repo}" \
-v "${MODELS_DIR}:/c/hub" -v "${dl}:/dl.py" \
--entrypoint python3 vllm/vllm-openai:qwen3_5 /dl.py \
|| die "download ${repo} a picat."
ok "${repo} descărcat"
fi
done
rm -f "$dl"
ok "Toate modelele sunt în ${MODELS_DIR}"
}
# =============================================================================
# TEST helpers (output corect, nu doar 200)
# =============================================================================
t_embeddings(){ post_json "http://$H:14100/v1/embeddings" '{"model":"bge-m3","input":"test"}' 20 | python3 -c 'import sys,json;v=json.load(sys.stdin)["data"][0]["embedding"];assert len(v)==1024;print(f"dim={len(v)}")'; }
t_rerank(){ post_json "http://$H:14200/v1/rerank" '{"model":"bge-reranker-v2-m3","query":"deepfake detection","documents":["o pisica","sistem detectie deepfake video"]}' 20 | python3 -c 'import sys,json;r=json.load(sys.stdin)["results"];assert r[0]["index"]==1;print("top_idx=%d score=%.3f"%(r[0]["index"],r[0]["relevance_score"]))'; }
t_llm(){ post_json "http://$H:14011/v1/chat/completions" '{"model":"qwen3.5","messages":[{"role":"user","content":"Spune un singur cuvant: salut"}],"max_tokens":20}' 90 | python3 -c 'import sys,json;c=json.load(sys.stdin)["choices"][0]["message"]["content"].strip();assert c and not c.lower().startswith("thinking");print(repr(c[:40]))'; }
t_audio(){ curl -s -m120 "http://$H:54300/v1/audio/transcriptions" -F "file=@${ASSETS}/audio.wav" -F "model=${WHISPER_MODEL}" | python3 -c 'import sys,json;t=json.load(sys.stdin)["text"].strip();assert len(t)>0;print(f"text[:40]={t[:40]!r}")'; }
t_video(){ curl -s -m180 "http://$H:54600/analyze/video" -F "file=@${ASSETS}/sample.mp4" | python3 -c 'import sys,json;d=json.load(sys.stdin);assert d["verdict"] in ("REAL","FAKE","UNCERTAIN");print("verdict=%s frames=%s"%(d["verdict"],d.get("frames_analyzed")))'; }
t_ner(){ post_json "http://$H:54400/v1/ner" '{"text":"Klaus Iohannis s-a intalnit cu Emmanuel Macron la Bucuresti."}' 60 | python3 -c 'import sys,json;e=json.load(sys.stdin)["results"]["entities"];assert len(e)>=2;print(f"entities={len(e)}")'; }
t_detect(){ curl -s -m90 "http://$H:54400/v1/detect" -F "file=@${ASSETS}/frame.jpg" | python3 -c 'import sys,json;d=json.load(sys.stdin)["results"]["detections"];assert len(d)>=1;print(f"objects={len(d)}")'; }
t_metadata(){ curl -s -m40 "http://$H:54400/v1/metadata" -F "file=@${ASSETS}/sample.mp4" | python3 -c 'import sys,json;d=json.load(sys.stdin);assert d["analyses"]["integrity"]["ok"];print("integrity ok")'; }
t_sentiment(){ post_json "http://$H:54400/v1/sentiment" '{"text":"Produsul este excelent, sunt foarte multumit!"}' 60 | python3 -c 'import sys,json;d=json.load(sys.stdin);assert d["ok"];print("label=%s"%d["results"].get("label"))'; }
t_forensic(){ curl -s -m15 "http://$H:8085/api/forensic-modules" | python3 -c 'import sys,json;m=json.load(sys.stdin)["available_modules"];assert len(m)>=3;print(f"modules={len(m)}")'; }
t_web(){ post_json "http://$H:51100/v1/search" '{"queries":["deepfake detection"],"max_results":3}' 40 | python3 -c 'import sys,json;r=json.load(sys.stdin)["results"];assert len(r)>0;print(f"results={len(r)}")'; }
t_catalog(){ curl -s -m15 "http://$H:11000/catalog/v1/status" -H "Authorization: Bearer ${GATEWAY_API_TOKEN}" | python3 -c 'import sys,json;d=json.load(sys.stdin);assert d["status"]=="healthy";print("catalog healthy")'; }
t_gateway(){ local a b; a=$(curl -s -m10 -o /dev/null -w '%{http_code}' "http://$H:11000/health"); b=$(curl -s -m10 -o /dev/null -w '%{http_code}' "http://$H:11000/web/v1/info"); [ "$a" = 200 ] && [ "$b" = 401 ] && echo "health=200 noauth=401" || { echo "health=$a noauth=$b (astept 200/401)"; return 1; }; }
t_brain(){ curl -s -m15 "http://$H:8090/v1/analysis_atom/stats/extended" | python3 -c 'import sys,json;json.load(sys.stdin);print("stats ok")' && post_json "http://$H:8090/v1/search" '{"queries":["deepfake"],"max_results":2}' 30 | python3 -c 'import sys,json;assert "results" in json.load(sys.stdin);print("search ok")'; }
t_dashboard(){ curl -s -m15 "http://$H:51300/api/monitoring/services" | python3 -c 'import sys,json;d=json.load(sys.stdin);s=d if isinstance(d,list) else d.get("services",[]);h=sum(x.get("status")=="healthy" for x in s);assert h==len(s) and h>=11,f"{h}/{len(s)}";print(f"{h}/{len(s)} healthy")'; }
prepare_assets() {
mkdir -p "$ASSETS"
[ -f "${ASSETS}/sample.mp4" ] || cp "$(find ${HERE}/../../artefacte_lot1/runs_deepfake -iname '*.mp4' -printf '%s\t%p\n' | sort -n | head -1 | cut -f2)" "${ASSETS}/sample.mp4"
if [ ! -f "${ASSETS}/frame.jpg" ] || [ ! -f "${ASSETS}/audio.wav" ]; then
docker cp "${ASSETS}/sample.mp4" didiAI-extractors:/tmp/s.mp4 >/dev/null
docker exec didiAI-extractors sh -c 'ffmpeg -y -i /tmp/s.mp4 -frames:v 1 /tmp/frame.jpg -vn -ar 16000 -ac 1 -t 8 /tmp/audio.wav' >/dev/null 2>&1
docker cp didiAI-extractors:/tmp/frame.jpg "${ASSETS}/frame.jpg" >/dev/null
docker cp didiAI-extractors:/tmp/audio.wav "${ASSETS}/audio.wav" >/dev/null
fi
}
# =============================================================================
# CĂRĂMIZI (fiecare: up dependențe → health → test → gate)
# =============================================================================
brick_embeddings(){ brick "EMBEDDINGS (bge-m3)"; dc up -d vllm-embeddings embeddings-api
wait_health "http://$H:14100/health" 180 "embeddings-api" || die "embeddings-api nu pornește"; gate "embeddings dim 1024" t_embeddings; }
brick_rerank(){ brick "RERANK (bge-reranker-v2-m3)"; dc up -d vllm-rerank rerank-api
wait_health "http://$H:14200/health" 180 "rerank-api" || die "rerank-api nu pornește"; gate "rerank ranking" t_rerank; }
brick_llm(){ brick "LLM (Qwen3.5-35B-A3B)"; dc up -d vllm-qwen llm-api
wait_health "http://$H:14001/v1/models" 900 "vLLM Qwen (load+compile)" || die "vLLM Qwen nu pornește"
wait_health "http://$H:14011/health" 60 "llm-api" || die "llm-api nu pornește"; gate "LLM output curat" t_llm; }
brick_extractors(){ brick "EXTRACTORS (NER/YOLO/metadata/OCR/sentiment)"; dc up -d extractors
wait_health "http://$H:54400/health" 120 "extractors" || die "extractors nu pornește"
prepare_assets
gate "metadata" t_metadata; gate "NER (GLiNER)" t_ner; gate "detect (YOLO)" t_detect; gate "sentiment (LLM)" t_sentiment; }
brick_audio(){ brick "AUDIO (Whisper large-v3-turbo)"; dc up -d audio-api
wait_health "http://$H:54300/health" 240 "audio (download model)" || die "audio nu pornește"; gate "transcriere" t_audio; }
brick_video(){ brick "VIDEO / BusterX++ (deepfake)"; dc up -d vllm-buster video-api
wait_health "http://$H:54500/health" 300 "vLLM BusterX" || die "vLLM BusterX nu pornește"
wait_health "http://$H:54600/health" 60 "video-api" || die "video-api nu pornește"; gate "verdict deepfake" t_video; }
brick_forensic(){ brick "FORENSIC (rPPG/lip-sync/forgery/lighting)"; dc up -d forensic
wait_health "http://$H:8085/health" 120 "forensic" || die "forensic nu pornește"; gate "module forensice" t_forensic; }
brick_web(){ brick "WEB + SearXNG (căutare)"
# curăță o instanță searxng pornită manual (docker run) ca s-o preia compose
if docker inspect didiAI-web-searxng >/dev/null 2>&1 && \
[ -z "$(docker inspect didiAI-web-searxng --format '{{ index .Config.Labels "com.docker.compose.project" }}' 2>/dev/null)" ]; then
docker rm -f didiAI-web-searxng >/dev/null 2>&1 || true
fi
dc up -d searxng web-api
wait_health "http://$H:51100/health" 120 "web-api" || die "web-api nu pornește"
sleep 8; gate "căutare web" t_web; }
brick_catalog(){ brick "CATALOG (service discovery)"; dc up -d catalog-api
sleep 6; } # health-ul agregat depinde de gateway; testat după gateway
brick_gateway(){ brick "GATEWAY (nginx + auth)"
( cd "$GW_DIR"; grep -q . .env 2>/dev/null || echo "GATEWAY_API_TOKEN=${GATEWAY_API_TOKEN}" > .env; docker compose up -d )
wait_health "http://$H:11000/health" 60 "gateway" || die "gateway nu pornește"
gate "gateway auth+rutare" t_gateway; gate "catalog agregat" t_catalog; }
brick_brain(){ brick "BRAIN (RAG + fact-check + cache)"
# render env din master, up postgres+atomic, aliniază parola, token, taxonomie, up api+scheduler
brain_write_env "$POSTGRES_PASSWORD" "${ATOMIC_TOKEN:-PLACEHOLDER}"
( cd "$BRAIN_DIR"; docker compose -f infra/docker-compose.yml up -d postgres atomic-server )
wait_health "http://$H:8088/health" 120 "atomic-server" || warn "atomic health întârzie"
brain_align_pg "$POSTGRES_PASSWORD"
local tok; tok=$(brain_ensure_token "$POSTGRES_PASSWORD" "${ATOMIC_TOKEN:-}")
[ -n "$tok" ] || die "nu am putut crea tokenul atomic"
if [ "$tok" != "${ATOMIC_TOKEN:-}" ]; then
sed -i "s#^ATOMIC_TOKEN=.*#ATOMIC_TOKEN=${tok}#" "$ENV_FILE"; ATOMIC_TOKEN="$tok"; ok "token atomic creat & salvat în .env"
fi
brain_write_env "$POSTGRES_PASSWORD" "$tok"
( cd "$BRAIN_DIR"; docker compose -f infra/docker-compose.yml up -d brain-api scheduler )
wait_health "http://$H:8090/health" 120 "brain-api" || die "brain-api nu pornește"
brain_seed_taxonomy
( cd "$BRAIN_DIR"; docker compose -f infra/docker-compose.yml restart brain-api scheduler ) >/dev/null
wait_health "http://$H:8090/health" 60 "brain-api (post-seed)" || die "brain-api nu revine"
gate "brain stats+search" t_brain; }
brick_dashboard(){ brick "DASHBOARD (UI + monitorizare)"
( cd "$DASH_DIR"
cat > .env <<EOF
DASHBOARD_DB_USER=${DASHBOARD_DB_USER}
DASHBOARD_DB_PASSWORD=${DASHBOARD_DB_PASSWORD}
DASHBOARD_DB_NAME=${DASHBOARD_DB_NAME}
DASHBOARD_EXTERNAL_URL=http://${H}:51300
DASHBOARD_STAGING_MODE=${DASHBOARD_STAGING_MODE}
VITE_STAGING_MODE=${DASHBOARD_STAGING_MODE}
DASHBOARD_LLM_API_URL=http://didiAI-llm-api:14011
DASHBOARD_EMBEDDINGS_HEALTH_URL=http://didiAI-embeddings-api:14100/health
DASHBOARD_RERANK_HEALTH_URL=http://didiAI-rerank-api:14200/health
DASHBOARD_AUDIO_HEALTH_URL=http://didiAI-audio:54300/v1/info
DASHBOARD_VIDEO_HEALTH_URL=http://didiAI-video-api:54600/v1/info
DASHBOARD_EXTRACTORS_HEALTH_URL=http://didiAI-extractors:54400/health
DASHBOARD_FORENSIC_HEALTH_URL=http://didiAI-forensic:8080/health
DASHBOARD_WEB_HEALTH_URL=http://didiAI-web-api:51100/v1/info
EOF
docker compose --profile dashboard up -d --build )
wait_health "http://$H:51300/health" 120 "dashboard" || die "dashboard nu pornește"
seed_catalog
sleep 5; gate "dashboard 11/11 healthy" t_dashboard; }
# =============================================================================
# RUN
# =============================================================================
[ "$INSTALL_DEPS" = 1 ] && install_deps
if [ -z "$ONLY" ]; then preflight; ensure_network; build_images; pull_images; predownload_models; fi
for b in embeddings rerank llm extractors audio video forensic web catalog gateway brain dashboard; do
should_run "$b" && "brick_$b"
done
brick "GATA"
ok "Toate cărămizile au trecut testele. Dashboard: http://${H}:51300/admin-ai/"
echo "Smoke-test complet: vezi ${HERE}/TESTING.md"

310
ai_platform/local_gpu_stack/docker-compose.yml Normal file
View file

@ -0,0 +1,310 @@
# Local GPU inference stack — 10.11.10.18 (2x H200 NVL)
# GPU 0: Qwen3.5-35B-A3B (LLM) + Whisper (audio)
# GPU 1: BusterX++ (deepfake video) + bge-m3 (embeddings) + bge-reranker-v2-m3 (rerank)
# All services on the shared `didi-network` so the dashboard can probe them by name.
networks:
didi-network:
external: true
services:
# ===================== vLLM model servers =====================
# BusterX++ — served OFFLINE from the on-disk HF cache (no download)
vllm-buster:
container_name: didiAI-video-vllm-buster
# qwen3_5 image (vLLM 0.16 / transformers 4.57) — BusterX++ config was saved
# with transformers 5.x and the older 0.8.5 image can't parse its nested text_config.
image: vllm/vllm-openai:qwen3_5
networks: [didi-network]
ports: ["54500:54500"]
volumes:
- ${MODELS_DIR}:/root/.cache/huggingface/hub:ro
environment:
- HF_HOME=/root/.cache/huggingface
- HF_HUB_OFFLINE=1
- TRANSFORMERS_OFFLINE=1
command: >
--model l8cv/BusterX_plusplus
--host 0.0.0.0 --port 54500
--served-model-name busterx
--tensor-parallel-size 1
--max-model-len 32768
--gpu-memory-utilization 0.25
--trust-remote-code
--enable-prefix-caching
--disable-log-requests
deploy:
resources:
reservations:
devices:
- driver: nvidia
device_ids: ['1']
capabilities: [gpu]
restart: unless-stopped
# Qwen3.5-35B-A3B — downloads ~72GB on first start
vllm-qwen:
container_name: didiAI-vllm-qwen3.5
image: vllm/vllm-openai:qwen3_5
networks: [didi-network]
ports: ["14001:14001"]
volumes:
- ${MODELS_DIR}:/root/.cache/huggingface/hub:ro
environment:
- HF_HOME=/root/.cache/huggingface
- HF_HUB_OFFLINE=1
- TRANSFORMERS_OFFLINE=1
- VLLM_ALLOW_LONG_MAX_MODEL_LEN=1
command: >
--model Qwen/Qwen3.5-35B-A3B
--host 0.0.0.0 --port 14001
--served-model-name qwen3.5
--tensor-parallel-size 1
--max-model-len 32000
--gpu-memory-utilization 0.65
--trust-remote-code
--disable-log-requests
deploy:
resources:
reservations:
devices:
- driver: nvidia
device_ids: ['0']
capabilities: [gpu]
restart: unless-stopped
# bge-m3 embeddings — downloads ~2GB on first start
vllm-embeddings:
container_name: didiAI-vllm-embeddings
image: vllm/vllm-openai:v0.8.5
networks: [didi-network]
ports: ["14101:14101"]
volumes:
- ${MODELS_DIR}:/root/.cache/huggingface/hub:ro
environment:
- HF_HOME=/root/.cache/huggingface
- HF_HUB_OFFLINE=1
- TRANSFORMERS_OFFLINE=1
command: >
--model BAAI/bge-m3
--host 0.0.0.0 --port 14101
--served-model-name bge-m3
--task embed
--gpu-memory-utilization 0.10
--trust-remote-code
--disable-log-requests
deploy:
resources:
reservations:
devices:
- driver: nvidia
device_ids: ['1']
capabilities: [gpu]
restart: unless-stopped
# bge-reranker-v2-m3 — downloads ~2GB on first start
vllm-rerank:
container_name: didiAI-vllm-rerank
image: vllm/vllm-openai:v0.8.5
networks: [didi-network]
ports: ["14201:14201"]
volumes:
- ${MODELS_DIR}:/root/.cache/huggingface/hub:ro
environment:
- HF_HOME=/root/.cache/huggingface
- HF_HUB_OFFLINE=1
- TRANSFORMERS_OFFLINE=1
command: >
--model BAAI/bge-reranker-v2-m3
--host 0.0.0.0 --port 14201
--served-model-name bge-reranker-v2-m3
--task score
--gpu-memory-utilization 0.10
--trust-remote-code
--disable-log-requests
deploy:
resources:
reservations:
devices:
- driver: nvidia
device_ids: ['1']
capabilities: [gpu]
restart: unless-stopped
# ===================== FastAPI app wrappers =====================
llm-api:
container_name: didiAI-llm-api
image: didiai-llm-inference:audit
networks: [didi-network]
ports: ["14011:14011"]
environment:
- LLM_HOST=0.0.0.0
- LLM_PORT=14011
- LLM_DEFAULT_BACKEND=vllm
- LLM_ENABLE_VLLM=true
- LLM_ENABLE_LLAMACPP=false
- LLM_VLLM_BASE_URL=http://didiAI-vllm-qwen3.5:14001
- LLM_DEFAULT_MODEL=qwen3.5
- LLM_EXTERNAL_URL=http://${HOST_IP}:14011
restart: unless-stopped
embeddings-api:
container_name: didiAI-embeddings-api
image: didiai-embeddings:audit
networks: [didi-network]
ports: ["14100:14100"]
environment:
- EMB_HOST=0.0.0.0
- EMB_PORT=14100
- EMB_DEFAULT_BACKEND=vllm
- EMB_ENABLE_VLLM=true
- EMB_ENABLE_LLAMACPP=false
- EMB_VLLM_BASE_URL=http://didiAI-vllm-embeddings:14101
- EMB_VLLM_MODEL=bge-m3
- EMB_EXTERNAL_URL=http://${HOST_IP}:14100
restart: unless-stopped
rerank-api:
container_name: didiAI-rerank-api
image: didiai-rerank:audit
networks: [didi-network]
ports: ["14200:14200"]
environment:
- RERANK_HOST=0.0.0.0
- RERANK_PORT=14200
- RERANK_DEFAULT_BACKEND=vllm
- RERANK_ENABLE_VLLM=true
- RERANK_ENABLE_LLAMACPP=false
- RERANK_VLLM_BASE_URL=http://didiAI-vllm-rerank:14201
- RERANK_VLLM_MODEL=bge-reranker-v2-m3
- RERANK_EXTERNAL_URL=http://${HOST_IP}:14200
restart: unless-stopped
video-api:
container_name: didiAI-video-api
image: didiai-video-analysis:audit
networks: [didi-network]
ports: ["54600:54600"]
volumes:
- ./runs:/app/runs
environment:
- VIDEO_ANALYSIS_HOST=0.0.0.0
- VIDEO_ANALYSIS_PORT=54600
- VIDEO_ANALYSIS_EXTERNAL_URL=http://${HOST_IP}:54600
- VIDEO_ANALYSIS_VLLM_BASE_URL=http://didiAI-video-vllm-buster:54500
- VIDEO_ANALYSIS_VLLM_MODEL=busterx
- VIDEO_ANALYSIS_RUNS_DIR=/app/runs
- VIDEO_ANALYSIS_SEMANTIC_LLM_BASE_URL=http://didiAI-llm-api:14011
- VIDEO_ANALYSIS_SEMANTIC_AGGREGATION_MODEL=qwen3.5
- VIDEO_ANALYSIS_SEMANTIC_LLM_API_KEY=none
restart: unless-stopped
# ===================== Non-GPU services =====================
catalog-api:
container_name: didiAI-catalog-api
image: didiai-catalog-api:audit
networks: [didi-network]
environment:
- CATALOG_HOST=0.0.0.0
- CATALOG_PORT=11000
- CATALOG_EXTERNAL_URL=http://${HOST_IP}
- CATALOG_LLM_URL=http://didiAI-llm-api:14011
- CATALOG_AUDIO_URL=http://didiAI-audio:54300
- CATALOG_VIDEO_URL=http://didiAI-video-api:54600
- CATALOG_WEB_URL=http://didiAI-web-api:51100
restart: unless-stopped
extractors:
container_name: didiAI-extractors
image: didiai-extractors:audit
networks: [didi-network]
ports: ["54400:54400"]
volumes:
# GLiNER (+ backbone mdeberta) servit offline din zona locala de modele
- ${MODELS_DIR}:/models-hf/hub:ro
environment:
- EXTRACTORS_HOST=0.0.0.0
- EXTRACTORS_PORT=54400
- EXTRACTORS_LLM_GATEWAY_URL=http://didiAI-llm-api:14011
- EXTRACTORS_DETECT_MODEL=/tmp/yolov8n.pt
- YOLO_CONFIG_DIR=/tmp/ultralytics
- HF_HOME=/models-hf
- HF_HUB_OFFLINE=1
- TRANSFORMERS_OFFLINE=1
restart: unless-stopped
forensic:
container_name: didiAI-forensic
image: didiai-forensic:audit
networks: [didi-network]
ports: ["8085:8080"]
environment:
- API_PORT=8080
restart: unless-stopped
# cloak — scraper SERP stealth (tier-3 fallback pentru web)
cloak:
container_name: didiAI-cloak
image: didiai-cloak:audit
networks: [didi-network]
ports: ["8770:8770"]
environment:
- CLOAK_HOST=0.0.0.0
- CLOAK_PORT=8770
restart: unless-stopped
# SearXNG metasearch (motorul de cautare liber folosit de web-api)
searxng:
container_name: didiAI-web-searxng
image: docker.io/searxng/searxng:latest
networks: [didi-network]
volumes:
- ../../modules/web/deploy/metasearch/searxng:/etc/searxng:ro
environment:
- SEARXNG_BASE_URL=http://${HOST_IP}:8080/
restart: unless-stopped
web-api:
container_name: didiAI-web-api
image: didiai-web:audit
networks: [didi-network]
ports: ["51100:51100"]
depends_on: [searxng]
environment:
- WEB_HOST=0.0.0.0
- WEB_PORT=51100
- WEB_EXTERNAL_URL=http://${HOST_IP}:51100
- WEB_SEARXNG_BASE_URL=http://didiAI-web-searxng:8080
- WEB_LLM_BASE_URL=http://didiAI-llm-api:14011
restart: unless-stopped
# Whisper STT — faster-whisper loads in-process on GPU 0
audio-api:
container_name: didiAI-audio
image: didiai-audio:audit
networks: [didi-network]
ports: ["54300:54300"]
volumes:
# faster-whisper foloseste download_root=cache_dir -> models-- direct sub el (nu /hub)
- ${MODELS_DIR}:/root/.cache/huggingface:ro
environment:
- AUDIO_HOST=0.0.0.0
- AUDIO_PORT=54300
- AUDIO_MODEL=large-v3-turbo
- AUDIO_DEVICE=cuda
- AUDIO_COMPUTE_TYPE=float16
- AUDIO_CACHE_DIR=/root/.cache/huggingface
- HF_HUB_OFFLINE=1
- TRANSFORMERS_OFFLINE=1
- AUDIO_EXTERNAL_URL=http://${HOST_IP}:54300
deploy:
resources:
reservations:
devices:
- driver: nvidia
device_ids: ['0']
capabilities: [gpu]
restart: unless-stopped

74
ai_platform/local_gpu_stack/lib.sh Normal file
View file

@ -0,0 +1,74 @@
#!/usr/bin/env bash
# Shared helpers for the brick-by-brick deploy. Sourced by deploy.sh.
# Logging, test gates, health waits, and a docker shim that works whether or not
# the current shell is in the `docker` group.
set -uo pipefail
# ---- colors ----------------------------------------------------------------
if [ -t 1 ]; then
C_R=$'\e[31m'; C_G=$'\e[32m'; C_Y=$'\e[33m'; C_B=$'\e[36m'; C_D=$'\e[2m'; C_0=$'\e[0m'
else
C_R=; C_G=; C_Y=; C_B=; C_D=; C_0=
fi
log() { printf '%s\n' "$*"; }
info() { printf '%s\n' "${C_B}»${C_0} $*"; }
ok() { printf '%s\n' "${C_G}✓${C_0} $*"; }
warn() { printf '%s\n' "${C_Y}!${C_0} $*"; }
err() { printf '%s\n' "${C_R}✗${C_0} $*" >&2; }
brick(){ printf '\n%s\n' "${C_B}━━━ CĂRĂMIDĂ: $* ━━━${C_0}"; }
die() { err "$*"; err "Deploy oprit (gate). Repară și re-rulează — e idempotent."; exit 1; }
# ---- docker shim (portable: plain docker, else `sg docker`) -----------------
USE_SG=0
if ! docker ps >/dev/null 2>&1; then
if command -v sg >/dev/null 2>&1 && sg docker -c 'docker ps' >/dev/null 2>&1; then
USE_SG=1
fi
fi
docker() {
if [ "$USE_SG" = 1 ]; then
local a args=""
for a in "$@"; do args+=" $(printf '%q' "$a")"; done
sg docker -c "docker$args"
else
command docker "$@"
fi
}
# ---- compose wrapper -------------------------------------------------------
# COMPOSE is exported by deploy.sh
dc() { docker compose -f "$COMPOSE" "$@"; }
# ---- health / test helpers -------------------------------------------------
# wait_health <url> <timeout_s> <label>
wait_health() {
local url=$1 timeout=${2:-120} label=${3:-$1} i=0
printf ' %s' "aștept $label "
while [ "$i" -lt "$timeout" ]; do
if [ "$(curl -s -m5 -o /dev/null -w '%{http_code}' "$url" 2>/dev/null)" = "200" ]; then
printf ' %sup%s\n' "$C_G" "$C_0"; return 0
fi
printf '.'; sleep 3; i=$((i+3))
done
printf '\n'; return 1
}
# gate <label> <test-command...>
# Runs the test; on non-zero exit, STOPS the whole deploy.
gate() {
local label=$1; shift
printf ' test: %-22s ' "$label"
local out rc
out=$("$@" 2>&1); rc=$?
if [ "$rc" -eq 0 ]; then
ok "PASS ${C_D}${out}${C_0}"
return 0
fi
err "FAIL"
printf '%s\n' "$out" | sed 's/^/ /'
die "Testul '$label' nu a trecut."
}
# json_field <url> <python-expr-on-d> -> prints value, exit 1 on error
post_json() { curl -s -m"${4:-60}" "$1" -H 'Content-Type: application/json' -d "$2"; }

422
ai_platform/local_gpu_stack/reports/latest.json Normal file
View file

@ -0,0 +1,422 @@
{
"generated_at": "2026-06-27T14:46:37.623279+03:00",
"host": "10.11.10.18",
"summary": {
"total": 30,
"passed": 30,
"failed": 0
},
"tests": [
{
"id": "llm",
"name": "LLM (Qwen3.5-35B-A3B)",
"endpoint": "POST /v1/chat/completions",
"http_code": 200,
"ok": true,
"latency_ms": 342.2,
"result": {
"reply": "Un deepfake este o media manipulat, realizat prin tehnologia inteligenței artificiale, care înlocuiează cu realism înspăimântător fața sau vocea unei persoane cu cea a altcuiva."
},
"checked_at": "2026-06-27T14:46:37.623279+03:00"
},
{
"id": "llm_completions",
"name": "LLM · completions",
"endpoint": "POST /v1/completions",
"http_code": 200,
"ok": true,
"latency_ms": 111.8,
"result": {
"text": " un oras care atrage foarte multi turisti si"
},
"checked_at": "2026-06-27T14:46:37.623279+03:00"
},
{
"id": "llm_info",
"name": "LLM · info",
"endpoint": "GET /v1/info",
"http_code": 200,
"ok": true,
"latency_ms": 14.5,
"result": {
"keys": [
"resource",
"models",
"functions"
]
},
"checked_at": "2026-06-27T14:46:37.623279+03:00"
},
{
"id": "embeddings",
"name": "Embeddings (bge-m3)",
"endpoint": "POST /v1/embeddings",
"http_code": 200,
"ok": true,
"latency_ms": 43.8,
"result": {
"dim": 1024
},
"checked_at": "2026-06-27T14:46:37.623279+03:00"
},
{
"id": "rerank",
"name": "Rerank (bge-reranker-v2-m3)",
"endpoint": "POST /v1/rerank",
"http_code": 200,
"ok": true,
"latency_ms": 39.3,
"result": {
"top_index": 1,
"score": 0.626
},
"checked_at": "2026-06-27T14:46:37.623279+03:00"
},
{
"id": "audio",
"name": "Audio (Whisper large-v3-turbo)",
"endpoint": "POST /v1/audio/transcriptions",
"http_code": 200,
"ok": true,
"latency_ms": 371.3,
"result": {
"text": "Nici nu mai știi cine este de fapt agresorul și agresatul. De ce? Simplu. Pentru că în acest moment, Uniunea Europeană...",
"language": "ro"
},
"checked_at": "2026-06-27T14:46:37.623279+03:00"
},
{
"id": "video",
"name": "Video/BusterX (deepfake)",
"endpoint": "POST /analyze/video",
"http_code": 200,
"ok": true,
"latency_ms": 1389.8,
"result": {
"verdict": "REAL",
"frames_analyzed": 16
},
"checked_at": "2026-06-27T14:46:37.623279+03:00"
},
{
"id": "video_semantic",
"name": "Video · semantic",
"endpoint": "POST /analyze/video/semantic",
"http_code": 200,
"ok": true,
"latency_ms": 18390.9,
"result": {
"num_chunks": 5,
"summary": "The video presents a continuous, polemical monologue delivered by a male speaker in a studio setting, likely a podcast o"
},
"checked_at": "2026-06-27T14:46:37.623279+03:00"
},
{
"id": "extractors_metadata",
"name": "Extractors · metadata",
"endpoint": "POST /v1/metadata",
"http_code": 200,
"ok": true,
"latency_ms": 284.1,
"result": {
"media_type": "video",
"sha256": "27d6c00786e63905"
},
"checked_at": "2026-06-27T14:46:37.623279+03:00"
},
{
"id": "extractors_ner",
"name": "Extractors · NER (GLiNER)",
"endpoint": "POST /v1/ner",
"http_code": 200,
"ok": true,
"latency_ms": 262.9,
"result": {
"entities": [
"Klaus Iohannis:persoană",
"Emmanuel Macron:persoană",
"Bucuresti:locație"
]
},
"checked_at": "2026-06-27T14:46:37.623279+03:00"
},
{
"id": "extractors_detect",
"name": "Extractors · detect (YOLOv8)",
"endpoint": "POST /v1/detect",
"http_code": 200,
"ok": true,
"latency_ms": 77.1,
"result": {
"objects": 4,
"labels": [
"person",
"person",
"person",
"person"
]
},
"checked_at": "2026-06-27T14:46:37.623279+03:00"
},
{
"id": "extractors_ocr",
"name": "Extractors · OCR (LLM vision)",
"endpoint": "POST /v1/ocr",
"http_code": 200,
"ok": true,
"latency_ms": 684.6,
"result": {
"text": "Asa arată\nANTI-ROMÂNISMUL\n22:08\nVOCEA LIBERTĂȚII\n15 AUG. 25\nALASKA\nTRUMP VS PUTIN\nTikTok\n@ ovi1623\n>> Da. Deci acuma nici nu mai știi cine"
},
"checked_at": "2026-06-27T14:46:37.623279+03:00"
},
{
"id": "extractors_sentiment",
"name": "Extractors · sentiment (LLM)",
"endpoint": "POST /v1/sentiment",
"http_code": 200,
"ok": true,
"latency_ms": 551.5,
"result": {
"label": "pozitiv",
"score": 0.9
},
"checked_at": "2026-06-27T14:46:37.623279+03:00"
},
{
"id": "forensic",
"name": "Forensic · module list",
"endpoint": "GET /api/forensic-modules",
"http_code": 200,
"ok": true,
"latency_ms": 9.5,
"result": {
"modules": [
"m25",
"m26",
"m27",
"m28",
"m29"
]
},
"checked_at": "2026-06-27T14:46:37.623279+03:00"
},
{
"id": "forensic_evidence",
"name": "Forensic · analiza reala",
"endpoint": "POST /api/forensic-evidence",
"http_code": 200,
"ok": true,
"latency_ms": 9563.6,
"result": {
"modules_run": [
"m25",
"m26",
"m27",
"m28",
"m29"
],
"frames": 131
},
"checked_at": "2026-06-27T14:46:37.623279+03:00"
},
{
"id": "web",
"name": "Web · search (SearXNG)",
"endpoint": "POST /v1/search",
"http_code": 200,
"ok": true,
"latency_ms": 1303.6,
"result": {
"results": 3,
"first": "AI Deepfake Detection Online Free - Images, Videos, Voices"
},
"checked_at": "2026-06-27T14:46:37.623279+03:00"
},
{
"id": "web_fetch",
"name": "Web · fetch",
"endpoint": "POST /v1/fetch",
"http_code": 200,
"ok": true,
"latency_ms": 731.4,
"result": {
"bytes": 113707
},
"checked_at": "2026-06-27T14:46:37.623279+03:00"
},
{
"id": "web_gather",
"name": "Web · gather (evidence)",
"endpoint": "POST /v1/gather",
"http_code": 200,
"ok": true,
"latency_ms": 3115.4,
"result": {
"evidence_items": 5,
"urls_found": 5
},
"checked_at": "2026-06-27T14:46:37.623279+03:00"
},
{
"id": "catalog",
"name": "Catalog · status",
"endpoint": "GET /catalog/v1/status",
"http_code": 200,
"ok": true,
"latency_ms": 81.8,
"result": {
"status": "healthy",
"components": [
"llm-inference",
"audio-transcription",
"video-analysis",
"web-factcheck"
]
},
"checked_at": "2026-06-27T14:46:37.623279+03:00"
},
{
"id": "catalog_components",
"name": "Catalog · components",
"endpoint": "GET /catalog/v1/components",
"http_code": 200,
"ok": true,
"latency_ms": 76.0,
"result": {
"components": 4
},
"checked_at": "2026-06-27T14:46:37.623279+03:00"
},
{
"id": "catalog_models",
"name": "Catalog · models",
"endpoint": "GET /catalog/v1/models",
"http_code": 200,
"ok": true,
"latency_ms": 67.9,
"result": {
"models": 3
},
"checked_at": "2026-06-27T14:46:37.623279+03:00"
},
{
"id": "gateway",
"name": "Gateway (auth+rutare)",
"endpoint": "GET /health + 401",
"http_code": 200,
"ok": true,
"latency_ms": 8.8,
"result": {
"health": 200,
"web_no_auth": 401
},
"checked_at": "2026-06-27T14:46:37.623279+03:00"
},
{
"id": "brain_atoms",
"name": "Brain · analysis_atom",
"endpoint": "GET /v1/analysis_atom/stats/extended",
"http_code": 200,
"ok": true,
"latency_ms": 15.3,
"result": {
"total_atoms": 9,
"by_tier": {
"gold": 4,
"silver": 3,
"bronze": 2
}
},
"checked_at": "2026-06-27T14:46:37.623279+03:00"
},
{
"id": "brain_facts",
"name": "Brain · fact_status",
"endpoint": "GET /v1/fact_status/list",
"http_code": 200,
"ok": true,
"latency_ms": 14.5,
"result": {
"total": 7
},
"checked_at": "2026-06-27T14:46:37.623279+03:00"
},
{
"id": "brain_vcache",
"name": "Brain · verification_cache",
"endpoint": "GET /v1/verification_cache/list",
"http_code": 200,
"ok": true,
"latency_ms": 11.3,
"result": {
"total": 5
},
"checked_at": "2026-06-27T14:46:37.623279+03:00"
},
{
"id": "brain_search",
"name": "Brain · search (RAG)",
"endpoint": "POST /v1/search",
"http_code": 200,
"ok": true,
"latency_ms": 3044.4,
"result": {
"total_results": 0,
"cache": "MISS"
},
"checked_at": "2026-06-27T14:46:37.623279+03:00"
},
{
"id": "brain_gather",
"name": "Brain · gather (RAG)",
"endpoint": "POST /v1/gather",
"http_code": 200,
"ok": true,
"latency_ms": 3038.3,
"result": {
"evidence_items": 0,
"urls": 0
},
"checked_at": "2026-06-27T14:46:37.623279+03:00"
},
{
"id": "dashboard",
"name": "Dashboard · health",
"endpoint": "GET /health",
"http_code": 200,
"ok": true,
"latency_ms": 19.3,
"result": {
"status": "healthy",
"db": true
},
"checked_at": "2026-06-27T14:46:37.623279+03:00"
},
{
"id": "dashboard_monitoring",
"name": "Dashboard · monitoring 11/11",
"endpoint": "GET /api/monitoring/services",
"http_code": 200,
"ok": true,
"latency_ms": 107.6,
"result": {
"healthy": "11/11"
},
"checked_at": "2026-06-27T14:46:37.623279+03:00"
},
{
"id": "cloak",
"name": "Cloak · SERP scraper",
"endpoint": "POST /v1/search",
"http_code": 200,
"ok": true,
"latency_ms": 1991.0,
"result": {
"results": 3,
"first": "Deepfake - Wikipedia"
},
"checked_at": "2026-06-27T14:46:37.623279+03:00"
}
]
}

249
ai_platform/local_gpu_stack/reports/test_report_20260627_143404.json Normal file
View file

@ -0,0 +1,249 @@
{
"generated_at": "2026-06-27T14:34:04.610734+03:00",
"host": "10.11.10.18",
"summary": {
"total": 17,
"passed": 17,
"failed": 0
},
"tests": [
{
"id": "llm",
"name": "LLM (Qwen3.5-35B-A3B)",
"endpoint": "POST /v1/chat/completions",
"http_code": 200,
"ok": true,
"latency_ms": 424.6,
"result": {
"reply": "Un deepfake este un conținut media manipulat prin tehnologia inteligenței artificiale, care înlocuiește fața sau vocea unei persoane cu cea a alteia pentru a crea imagini sau înregistrări video extrem de realiste, dar falsificate."
},
"checked_at": "2026-06-27T14:34:04.610734+03:00"
},
{
"id": "embeddings",
"name": "Embeddings (bge-m3)",
"endpoint": "POST /v1/embeddings",
"http_code": 200,
"ok": true,
"latency_ms": 42.5,
"result": {
"dim": 1024
},
"checked_at": "2026-06-27T14:34:04.610734+03:00"
},
{
"id": "rerank",
"name": "Rerank (bge-reranker-v2-m3)",
"endpoint": "POST /v1/rerank",
"http_code": 200,
"ok": true,
"latency_ms": 33.3,
"result": {
"top_index": 1,
"score": 0.626
},
"checked_at": "2026-06-27T14:34:04.610734+03:00"
},
{
"id": "audio",
"name": "Audio (Whisper large-v3-turbo)",
"endpoint": "POST /v1/audio/transcriptions",
"http_code": 200,
"ok": true,
"latency_ms": 251.9,
"result": {
"text": "Nici nu mai știi cine este de fapt agresorul și agresatul. De ce? Simplu. Pentru că în acest moment, Uniunea Europeană...",
"language": "ro"
},
"checked_at": "2026-06-27T14:34:04.610734+03:00"
},
{
"id": "video",
"name": "Video/BusterX (deepfake)",
"endpoint": "POST /analyze/video",
"http_code": 200,
"ok": true,
"latency_ms": 1355.9,
"result": {
"verdict": "REAL",
"frames_analyzed": 16
},
"checked_at": "2026-06-27T14:34:04.610734+03:00"
},
{
"id": "extractors_metadata",
"name": "Extractors · metadata",
"endpoint": "POST /v1/metadata",
"http_code": 200,
"ok": true,
"latency_ms": 275.1,
"result": {
"media_type": "video",
"sha256": "27d6c00786e63905"
},
"checked_at": "2026-06-27T14:34:04.610734+03:00"
},
{
"id": "extractors_ner",
"name": "Extractors · NER (GLiNER)",
"endpoint": "POST /v1/ner",
"http_code": 200,
"ok": true,
"latency_ms": 813.0,
"result": {
"entities": [
"Klaus Iohannis:persoană",
"Emmanuel Macron:persoană",
"Bucuresti:locație"
]
},
"checked_at": "2026-06-27T14:34:04.610734+03:00"
},
{
"id": "extractors_detect",
"name": "Extractors · detect (YOLOv8)",
"endpoint": "POST /v1/detect",
"http_code": 200,
"ok": true,
"latency_ms": 1464.4,
"result": {
"objects": 4,
"labels": [
"person",
"person",
"person",
"person"
]
},
"checked_at": "2026-06-27T14:34:04.610734+03:00"
},
{
"id": "extractors_ocr",
"name": "Extractors · OCR (LLM vision)",
"endpoint": "POST /v1/ocr",
"http_code": 200,
"ok": true,
"latency_ms": 5621.1,
"result": {
"text": "Asa arată\nANTI-ROMÂNISMUL\n22:08\nVOCEA LIBERTĂȚII\n15 AUG. 25\nALASKA\nTRUMP VS PUTIN\nTikTok\n@ ovi1623\n>> Da. Deci acuma nici nu mai știi cine"
},
"checked_at": "2026-06-27T14:34:04.610734+03:00"
},
{
"id": "extractors_sentiment",
"name": "Extractors · sentiment (LLM)",
"endpoint": "POST /v1/sentiment",
"http_code": 200,
"ok": true,
"latency_ms": 551.3,
"result": {
"label": "pozitiv",
"score": 0.9
},
"checked_at": "2026-06-27T14:34:04.610734+03:00"
},
{
"id": "forensic",
"name": "Forensic (rPPG/lip-sync/...)",
"endpoint": "GET /api/forensic-modules",
"http_code": 200,
"ok": true,
"latency_ms": 6.3,
"result": {
"modules": [
"m25",
"m26",
"m27",
"m28",
"m29"
]
},
"checked_at": "2026-06-27T14:34:04.610734+03:00"
},
{
"id": "web",
"name": "Web search (SearXNG)",
"endpoint": "POST /v1/search",
"http_code": 200,
"ok": true,
"latency_ms": 1504.0,
"result": {
"results": 3,
"first": "AI Deepfake Detection Online Free - Images, Videos, Voices"
},
"checked_at": "2026-06-27T14:34:04.610734+03:00"
},
{
"id": "catalog",
"name": "Catalog (via gateway)",
"endpoint": "GET /catalog/v1/status",
"http_code": 200,
"ok": true,
"latency_ms": 68.9,
"result": {
"status": "healthy",
"components": [
"llm-inference",
"audio-transcription",
"video-analysis",
"web-factcheck"
]
},
"checked_at": "2026-06-27T14:34:04.610734+03:00"
},
{
"id": "gateway",
"name": "Gateway (auth+rutare)",
"endpoint": "GET /health + 401",
"http_code": 200,
"ok": true,
"latency_ms": 7.3,
"result": {
"health": 200,
"web_no_auth": 401
},
"checked_at": "2026-06-27T14:34:04.610734+03:00"
},
{
"id": "brain_atoms",
"name": "Brain · analysis_atom",
"endpoint": "GET /v1/analysis_atom/stats/extended",
"http_code": 200,
"ok": true,
"latency_ms": 54.1,
"result": {
"total_atoms": 9,
"by_tier": {
"gold": 4,
"silver": 3,
"bronze": 2
}
},
"checked_at": "2026-06-27T14:34:04.610734+03:00"
},
{
"id": "brain_facts",
"name": "Brain · fact_status",
"endpoint": "GET /v1/fact_status/list",
"http_code": 200,
"ok": true,
"latency_ms": 16.3,
"result": {
"total": 7
},
"checked_at": "2026-06-27T14:34:04.610734+03:00"
},
{
"id": "brain_vcache",
"name": "Brain · verification_cache",
"endpoint": "GET /v1/verification_cache/list",
"http_code": 200,
"ok": true,
"latency_ms": 22.8,
"result": {
"total": 5
},
"checked_at": "2026-06-27T14:34:04.610734+03:00"
}
]
}

422
ai_platform/local_gpu_stack/reports/test_report_20260627_144358.json Normal file
View file

@ -0,0 +1,422 @@
{
"generated_at": "2026-06-27T14:43:58.666410+03:00",
"host": "10.11.10.18",
"summary": {
"total": 30,
"passed": 30,
"failed": 0
},
"tests": [
{
"id": "llm",
"name": "LLM (Qwen3.5-35B-A3B)",
"endpoint": "POST /v1/chat/completions",
"http_code": 200,
"ok": true,
"latency_ms": 400.6,
"result": {
"reply": "Deepfake este o tehnică de manipulare a conținutului multimedia care utilizează inteligența artificială pentru a înlocua fața sau vocea unei persoane reale cu cea a alteia, creând astfel imagini sau videoclipuri extrem de realiste dar false."
},
"checked_at": "2026-06-27T14:43:58.666410+03:00"
},
{
"id": "llm_completions",
"name": "LLM · completions",
"endpoint": "POST /v1/completions",
"http_code": 200,
"ok": true,
"latency_ms": 122.0,
"result": {
"text": " Bucuresti.\n\n<think>\n\n</think>\n\nAfirm"
},
"checked_at": "2026-06-27T14:43:58.666410+03:00"
},
{
"id": "llm_info",
"name": "LLM · info",
"endpoint": "GET /v1/info",
"http_code": 200,
"ok": true,
"latency_ms": 12.9,
"result": {
"keys": [
"resource",
"models",
"functions"
]
},
"checked_at": "2026-06-27T14:43:58.666410+03:00"
},
{
"id": "embeddings",
"name": "Embeddings (bge-m3)",
"endpoint": "POST /v1/embeddings",
"http_code": 200,
"ok": true,
"latency_ms": 41.8,
"result": {
"dim": 1024
},
"checked_at": "2026-06-27T14:43:58.666410+03:00"
},
{
"id": "rerank",
"name": "Rerank (bge-reranker-v2-m3)",
"endpoint": "POST /v1/rerank",
"http_code": 200,
"ok": true,
"latency_ms": 39.7,
"result": {
"top_index": 1,
"score": 0.626
},
"checked_at": "2026-06-27T14:43:58.666410+03:00"
},
{
"id": "audio",
"name": "Audio (Whisper large-v3-turbo)",
"endpoint": "POST /v1/audio/transcriptions",
"http_code": 200,
"ok": true,
"latency_ms": 372.5,
"result": {
"text": "Nici nu mai știi cine este de fapt agresorul și agresatul. De ce? Simplu. Pentru că în acest moment, Uniunea Europeană...",
"language": "ro"
},
"checked_at": "2026-06-27T14:43:58.666410+03:00"
},
{
"id": "video",
"name": "Video/BusterX (deepfake)",
"endpoint": "POST /analyze/video",
"http_code": 200,
"ok": true,
"latency_ms": 1384.5,
"result": {
"verdict": "REAL",
"frames_analyzed": 16
},
"checked_at": "2026-06-27T14:43:58.666410+03:00"
},
{
"id": "video_semantic",
"name": "Video · semantic",
"endpoint": "POST /analyze/video/semantic",
"http_code": 200,
"ok": true,
"latency_ms": 27228.4,
"result": {
"num_chunks": 5,
"summary": "[Aggregation failed: Client error '404 Not Found' for url 'http://didiai-llm-api:14011/v1/v1/chat/completions'\nFor more "
},
"checked_at": "2026-06-27T14:43:58.666410+03:00"
},
{
"id": "extractors_metadata",
"name": "Extractors · metadata",
"endpoint": "POST /v1/metadata",
"http_code": 200,
"ok": true,
"latency_ms": 320.3,
"result": {
"media_type": "video",
"sha256": "27d6c00786e63905"
},
"checked_at": "2026-06-27T14:43:58.666410+03:00"
},
{
"id": "extractors_ner",
"name": "Extractors · NER (GLiNER)",
"endpoint": "POST /v1/ner",
"http_code": 200,
"ok": true,
"latency_ms": 115.1,
"result": {
"entities": [
"Klaus Iohannis:persoană",
"Emmanuel Macron:persoană",
"Bucuresti:locație"
]
},
"checked_at": "2026-06-27T14:43:58.666410+03:00"
},
{
"id": "extractors_detect",
"name": "Extractors · detect (YOLOv8)",
"endpoint": "POST /v1/detect",
"http_code": 200,
"ok": true,
"latency_ms": 53.7,
"result": {
"objects": 4,
"labels": [
"person",
"person",
"person",
"person"
]
},
"checked_at": "2026-06-27T14:43:58.666410+03:00"
},
{
"id": "extractors_ocr",
"name": "Extractors · OCR (LLM vision)",
"endpoint": "POST /v1/ocr",
"http_code": 200,
"ok": true,
"latency_ms": 701.6,
"result": {
"text": "Asa arată\nANTI-ROMÂNISMUL\n22:08\nVOCEA LIBERTĂȚII\n15 AUG. 25\nALASKA\nTRUMP VS PUTIN\nTikTok\n@ ovi1623\n>> Da. Deci acuma nici nu mai știi cine"
},
"checked_at": "2026-06-27T14:43:58.666410+03:00"
},
{
"id": "extractors_sentiment",
"name": "Extractors · sentiment (LLM)",
"endpoint": "POST /v1/sentiment",
"http_code": 200,
"ok": true,
"latency_ms": 552.9,
"result": {
"label": "pozitiv",
"score": 0.9
},
"checked_at": "2026-06-27T14:43:58.666410+03:00"
},
{
"id": "forensic",
"name": "Forensic · module list",
"endpoint": "GET /api/forensic-modules",
"http_code": 200,
"ok": true,
"latency_ms": 8.7,
"result": {
"modules": [
"m25",
"m26",
"m27",
"m28",
"m29"
]
},
"checked_at": "2026-06-27T14:43:58.666410+03:00"
},
{
"id": "forensic_evidence",
"name": "Forensic · analiza reala",
"endpoint": "POST /api/forensic-evidence",
"http_code": 200,
"ok": true,
"latency_ms": 9039.9,
"result": {
"modules_run": [
"m25",
"m26",
"m27",
"m28",
"m29"
],
"frames": 131
},
"checked_at": "2026-06-27T14:43:58.666410+03:00"
},
{
"id": "web",
"name": "Web · search (SearXNG)",
"endpoint": "POST /v1/search",
"http_code": 200,
"ok": true,
"latency_ms": 841.4,
"result": {
"results": 3,
"first": "AI Deepfake Detection Online Free - Images, Videos, Voices"
},
"checked_at": "2026-06-27T14:43:58.666410+03:00"
},
{
"id": "web_fetch",
"name": "Web · fetch",
"endpoint": "POST /v1/fetch",
"http_code": 200,
"ok": true,
"latency_ms": 554.3,
"result": {
"bytes": 113707
},
"checked_at": "2026-06-27T14:43:58.666410+03:00"
},
{
"id": "web_gather",
"name": "Web · gather (evidence)",
"endpoint": "POST /v1/gather",
"http_code": 200,
"ok": true,
"latency_ms": 2911.9,
"result": {
"evidence_items": 5,
"urls_found": 5
},
"checked_at": "2026-06-27T14:43:58.666410+03:00"
},
{
"id": "catalog",
"name": "Catalog · status",
"endpoint": "GET /catalog/v1/status",
"http_code": 200,
"ok": true,
"latency_ms": 73.8,
"result": {
"status": "healthy",
"components": [
"llm-inference",
"audio-transcription",
"video-analysis",
"web-factcheck"
]
},
"checked_at": "2026-06-27T14:43:58.666410+03:00"
},
{
"id": "catalog_components",
"name": "Catalog · components",
"endpoint": "GET /catalog/v1/components",
"http_code": 200,
"ok": true,
"latency_ms": 70.6,
"result": {
"components": 4
},
"checked_at": "2026-06-27T14:43:58.666410+03:00"
},
{
"id": "catalog_models",
"name": "Catalog · models",
"endpoint": "GET /catalog/v1/models",
"http_code": 200,
"ok": true,
"latency_ms": 55.7,
"result": {
"models": 3
},
"checked_at": "2026-06-27T14:43:58.666410+03:00"
},
{
"id": "gateway",
"name": "Gateway (auth+rutare)",
"endpoint": "GET /health + 401",
"http_code": 200,
"ok": true,
"latency_ms": 7.1,
"result": {
"health": 200,
"web_no_auth": 401
},
"checked_at": "2026-06-27T14:43:58.666410+03:00"
},
{
"id": "brain_atoms",
"name": "Brain · analysis_atom",
"endpoint": "GET /v1/analysis_atom/stats/extended",
"http_code": 200,
"ok": true,
"latency_ms": 36.1,
"result": {
"total_atoms": 9,
"by_tier": {
"gold": 4,
"silver": 3,
"bronze": 2
}
},
"checked_at": "2026-06-27T14:43:58.666410+03:00"
},
{
"id": "brain_facts",
"name": "Brain · fact_status",
"endpoint": "GET /v1/fact_status/list",
"http_code": 200,
"ok": true,
"latency_ms": 13.3,
"result": {
"total": 7
},
"checked_at": "2026-06-27T14:43:58.666410+03:00"
},
{
"id": "brain_vcache",
"name": "Brain · verification_cache",
"endpoint": "GET /v1/verification_cache/list",
"http_code": 200,
"ok": true,
"latency_ms": 13.7,
"result": {
"total": 5
},
"checked_at": "2026-06-27T14:43:58.666410+03:00"
},
{
"id": "brain_search",
"name": "Brain · search (RAG)",
"endpoint": "POST /v1/search",
"http_code": 200,
"ok": true,
"latency_ms": 3040.4,
"result": {
"total_results": 0,
"cache": "MISS"
},
"checked_at": "2026-06-27T14:43:58.666410+03:00"
},
{
"id": "brain_gather",
"name": "Brain · gather (RAG)",
"endpoint": "POST /v1/gather",
"http_code": 200,
"ok": true,
"latency_ms": 3042.0,
"result": {
"evidence_items": 0,
"urls": 0
},
"checked_at": "2026-06-27T14:43:58.666410+03:00"
},
{
"id": "dashboard",
"name": "Dashboard · health",
"endpoint": "GET /health",
"http_code": 200,
"ok": true,
"latency_ms": 14.0,
"result": {
"status": "healthy",
"db": true
},
"checked_at": "2026-06-27T14:43:58.666410+03:00"
},
{
"id": "dashboard_monitoring",
"name": "Dashboard · monitoring 11/11",
"endpoint": "GET /api/monitoring/services",
"http_code": 200,
"ok": true,
"latency_ms": 129.6,
"result": {
"healthy": "11/11"
},
"checked_at": "2026-06-27T14:43:58.666410+03:00"
},
{
"id": "cloak",
"name": "Cloak · SERP scraper",
"endpoint": "POST /v1/search",
"http_code": 200,
"ok": true,
"latency_ms": 2014.3,
"result": {
"results": 3,
"first": "Deepfake - Wikipedia"
},
"checked_at": "2026-06-27T14:43:58.666410+03:00"
}
]
}

422
ai_platform/local_gpu_stack/reports/test_report_20260627_144637.json Normal file
View file

@ -0,0 +1,422 @@
{
"generated_at": "2026-06-27T14:46:37.623279+03:00",
"host": "10.11.10.18",
"summary": {
"total": 30,
"passed": 30,
"failed": 0
},
"tests": [
{
"id": "llm",
"name": "LLM (Qwen3.5-35B-A3B)",
"endpoint": "POST /v1/chat/completions",
"http_code": 200,
"ok": true,
"latency_ms": 342.2,
"result": {
"reply": "Un deepfake este o media manipulat, realizat prin tehnologia inteligenței artificiale, care înlocuiează cu realism înspăimântător fața sau vocea unei persoane cu cea a altcuiva."
},
"checked_at": "2026-06-27T14:46:37.623279+03:00"
},
{
"id": "llm_completions",
"name": "LLM · completions",
"endpoint": "POST /v1/completions",
"http_code": 200,
"ok": true,
"latency_ms": 111.8,
"result": {
"text": " un oras care atrage foarte multi turisti si"
},
"checked_at": "2026-06-27T14:46:37.623279+03:00"
},
{
"id": "llm_info",
"name": "LLM · info",
"endpoint": "GET /v1/info",
"http_code": 200,
"ok": true,
"latency_ms": 14.5,
"result": {
"keys": [
"resource",
"models",
"functions"
]
},
"checked_at": "2026-06-27T14:46:37.623279+03:00"
},
{
"id": "embeddings",
"name": "Embeddings (bge-m3)",
"endpoint": "POST /v1/embeddings",
"http_code": 200,
"ok": true,
"latency_ms": 43.8,
"result": {
"dim": 1024
},
"checked_at": "2026-06-27T14:46:37.623279+03:00"
},
{
"id": "rerank",
"name": "Rerank (bge-reranker-v2-m3)",
"endpoint": "POST /v1/rerank",
"http_code": 200,
"ok": true,
"latency_ms": 39.3,
"result": {
"top_index": 1,
"score": 0.626
},
"checked_at": "2026-06-27T14:46:37.623279+03:00"
},
{
"id": "audio",
"name": "Audio (Whisper large-v3-turbo)",
"endpoint": "POST /v1/audio/transcriptions",
"http_code": 200,
"ok": true,
"latency_ms": 371.3,
"result": {
"text": "Nici nu mai știi cine este de fapt agresorul și agresatul. De ce? Simplu. Pentru că în acest moment, Uniunea Europeană...",
"language": "ro"
},
"checked_at": "2026-06-27T14:46:37.623279+03:00"
},
{
"id": "video",
"name": "Video/BusterX (deepfake)",
"endpoint": "POST /analyze/video",
"http_code": 200,
"ok": true,
"latency_ms": 1389.8,
"result": {
"verdict": "REAL",
"frames_analyzed": 16
},
"checked_at": "2026-06-27T14:46:37.623279+03:00"
},
{
"id": "video_semantic",
"name": "Video · semantic",
"endpoint": "POST /analyze/video/semantic",
"http_code": 200,
"ok": true,
"latency_ms": 18390.9,
"result": {
"num_chunks": 5,
"summary": "The video presents a continuous, polemical monologue delivered by a male speaker in a studio setting, likely a podcast o"
},
"checked_at": "2026-06-27T14:46:37.623279+03:00"
},
{
"id": "extractors_metadata",
"name": "Extractors · metadata",
"endpoint": "POST /v1/metadata",
"http_code": 200,
"ok": true,
"latency_ms": 284.1,
"result": {
"media_type": "video",
"sha256": "27d6c00786e63905"
},
"checked_at": "2026-06-27T14:46:37.623279+03:00"
},
{
"id": "extractors_ner",
"name": "Extractors · NER (GLiNER)",
"endpoint": "POST /v1/ner",
"http_code": 200,
"ok": true,
"latency_ms": 262.9,
"result": {
"entities": [
"Klaus Iohannis:persoană",
"Emmanuel Macron:persoană",
"Bucuresti:locație"
]
},
"checked_at": "2026-06-27T14:46:37.623279+03:00"
},
{
"id": "extractors_detect",
"name": "Extractors · detect (YOLOv8)",
"endpoint": "POST /v1/detect",
"http_code": 200,
"ok": true,
"latency_ms": 77.1,
"result": {
"objects": 4,
"labels": [
"person",
"person",
"person",
"person"
]
},
"checked_at": "2026-06-27T14:46:37.623279+03:00"
},
{
"id": "extractors_ocr",
"name": "Extractors · OCR (LLM vision)",
"endpoint": "POST /v1/ocr",
"http_code": 200,
"ok": true,
"latency_ms": 684.6,
"result": {
"text": "Asa arată\nANTI-ROMÂNISMUL\n22:08\nVOCEA LIBERTĂȚII\n15 AUG. 25\nALASKA\nTRUMP VS PUTIN\nTikTok\n@ ovi1623\n>> Da. Deci acuma nici nu mai știi cine"
},
"checked_at": "2026-06-27T14:46:37.623279+03:00"
},
{
"id": "extractors_sentiment",
"name": "Extractors · sentiment (LLM)",
"endpoint": "POST /v1/sentiment",
"http_code": 200,
"ok": true,
"latency_ms": 551.5,
"result": {
"label": "pozitiv",
"score": 0.9
},
"checked_at": "2026-06-27T14:46:37.623279+03:00"
},
{
"id": "forensic",
"name": "Forensic · module list",
"endpoint": "GET /api/forensic-modules",
"http_code": 200,
"ok": true,
"latency_ms": 9.5,
"result": {
"modules": [
"m25",
"m26",
"m27",
"m28",
"m29"
]
},
"checked_at": "2026-06-27T14:46:37.623279+03:00"
},
{
"id": "forensic_evidence",
"name": "Forensic · analiza reala",
"endpoint": "POST /api/forensic-evidence",
"http_code": 200,
"ok": true,
"latency_ms": 9563.6,
"result": {
"modules_run": [
"m25",
"m26",
"m27",
"m28",
"m29"
],
"frames": 131
},
"checked_at": "2026-06-27T14:46:37.623279+03:00"
},
{
"id": "web",
"name": "Web · search (SearXNG)",
"endpoint": "POST /v1/search",
"http_code": 200,
"ok": true,
"latency_ms": 1303.6,
"result": {
"results": 3,
"first": "AI Deepfake Detection Online Free - Images, Videos, Voices"
},
"checked_at": "2026-06-27T14:46:37.623279+03:00"
},
{
"id": "web_fetch",
"name": "Web · fetch",
"endpoint": "POST /v1/fetch",
"http_code": 200,
"ok": true,
"latency_ms": 731.4,
"result": {
"bytes": 113707
},
"checked_at": "2026-06-27T14:46:37.623279+03:00"
},
{
"id": "web_gather",
"name": "Web · gather (evidence)",
"endpoint": "POST /v1/gather",
"http_code": 200,
"ok": true,
"latency_ms": 3115.4,
"result": {
"evidence_items": 5,
"urls_found": 5
},
"checked_at": "2026-06-27T14:46:37.623279+03:00"
},
{
"id": "catalog",
"name": "Catalog · status",
"endpoint": "GET /catalog/v1/status",
"http_code": 200,
"ok": true,
"latency_ms": 81.8,
"result": {
"status": "healthy",
"components": [
"llm-inference",
"audio-transcription",
"video-analysis",
"web-factcheck"
]
},
"checked_at": "2026-06-27T14:46:37.623279+03:00"
},
{
"id": "catalog_components",
"name": "Catalog · components",
"endpoint": "GET /catalog/v1/components",
"http_code": 200,
"ok": true,
"latency_ms": 76.0,
"result": {
"components": 4
},
"checked_at": "2026-06-27T14:46:37.623279+03:00"
},
{
"id": "catalog_models",
"name": "Catalog · models",
"endpoint": "GET /catalog/v1/models",
"http_code": 200,
"ok": true,
"latency_ms": 67.9,
"result": {
"models": 3
},
"checked_at": "2026-06-27T14:46:37.623279+03:00"
},
{
"id": "gateway",
"name": "Gateway (auth+rutare)",
"endpoint": "GET /health + 401",
"http_code": 200,
"ok": true,
"latency_ms": 8.8,
"result": {
"health": 200,
"web_no_auth": 401
},
"checked_at": "2026-06-27T14:46:37.623279+03:00"
},
{
"id": "brain_atoms",
"name": "Brain · analysis_atom",
"endpoint": "GET /v1/analysis_atom/stats/extended",
"http_code": 200,
"ok": true,
"latency_ms": 15.3,
"result": {
"total_atoms": 9,
"by_tier": {
"gold": 4,
"silver": 3,
"bronze": 2
}
},
"checked_at": "2026-06-27T14:46:37.623279+03:00"
},
{
"id": "brain_facts",
"name": "Brain · fact_status",
"endpoint": "GET /v1/fact_status/list",
"http_code": 200,
"ok": true,
"latency_ms": 14.5,
"result": {
"total": 7
},
"checked_at": "2026-06-27T14:46:37.623279+03:00"
},
{
"id": "brain_vcache",
"name": "Brain · verification_cache",
"endpoint": "GET /v1/verification_cache/list",
"http_code": 200,
"ok": true,
"latency_ms": 11.3,
"result": {
"total": 5
},
"checked_at": "2026-06-27T14:46:37.623279+03:00"
},
{
"id": "brain_search",
"name": "Brain · search (RAG)",
"endpoint": "POST /v1/search",
"http_code": 200,
"ok": true,
"latency_ms": 3044.4,
"result": {
"total_results": 0,
"cache": "MISS"
},
"checked_at": "2026-06-27T14:46:37.623279+03:00"
},
{
"id": "brain_gather",
"name": "Brain · gather (RAG)",
"endpoint": "POST /v1/gather",
"http_code": 200,
"ok": true,
"latency_ms": 3038.3,
"result": {
"evidence_items": 0,
"urls": 0
},
"checked_at": "2026-06-27T14:46:37.623279+03:00"
},
{
"id": "dashboard",
"name": "Dashboard · health",
"endpoint": "GET /health",
"http_code": 200,
"ok": true,
"latency_ms": 19.3,
"result": {
"status": "healthy",
"db": true
},
"checked_at": "2026-06-27T14:46:37.623279+03:00"
},
{
"id": "dashboard_monitoring",
"name": "Dashboard · monitoring 11/11",
"endpoint": "GET /api/monitoring/services",
"http_code": 200,
"ok": true,
"latency_ms": 107.6,
"result": {
"healthy": "11/11"
},
"checked_at": "2026-06-27T14:46:37.623279+03:00"
},
{
"id": "cloak",
"name": "Cloak · SERP scraper",
"endpoint": "POST /v1/search",
"http_code": 200,
"ok": true,
"latency_ms": 1991.0,
"result": {
"results": 3,
"first": "Deepfake - Wikipedia"
},
"checked_at": "2026-06-27T14:46:37.623279+03:00"
}
]
}

271
ai_platform/local_gpu_stack/run_tests.py Normal file
View file

@ -0,0 +1,271 @@
#!/usr/bin/env python3
"""DiDi LOT 1 — runner automat de teste end-to-end.
Rulează un test funcțional pe fiecare componentă și scrie rezultatele într-un
raport JSON cu timestamp (reports/test_report_<data>.json). Fiecare intrare
conține: componentă, endpoint, http_code, ok (verdict funcțional), latență,
extras din răspuns + request-ul folosit — ca dovadă vizibilă.
Rulare: python3 run_tests.py (HOST_IP din .env sau 10.11.10.18)
HOST_IP=1.2.3.4 python3 run_tests.py
"""
from __future__ import annotations
import json, os, subprocess, sys, time, datetime, pathlib
HERE = pathlib.Path(__file__).resolve().parent
ASSETS = HERE / "test_assets"
REPORTS = HERE / "reports"
# --- config din .env (fallback la defaults) ---------------------------------
def load_env() -> dict:
env = {}
f = HERE / ".env"
if f.exists():
for line in f.read_text().splitlines():
line = line.strip()
if line and not line.startswith("#") and "=" in line:
k, v = line.split("=", 1)
env[k.strip()] = v.strip()
return env
ENV = load_env()
H = os.environ.get("HOST_IP") or ENV.get("HOST_IP") or "10.11.10.18"
TOKEN = ENV.get("GATEWAY_API_TOKEN", "didi-local-dev-token-123")
WHISPER = ENV.get("WHISPER_MODEL", "large-v3-turbo")
# --- docker shim (pt pregatirea asset-urilor; merge si prin sg docker) -------
def docker(args: list[str]) -> subprocess.CompletedProcess:
if subprocess.run(["docker", "ps"], capture_output=True).returncode == 0:
return subprocess.run(["docker", *args], capture_output=True, text=True)
return subprocess.run(["sg", "docker", "-c", "docker " + " ".join(args)],
capture_output=True, text=True)
def ensure_assets() -> None:
ASSETS.mkdir(exist_ok=True)
mp4 = ASSETS / "sample.mp4"
if not mp4.exists():
import glob
cands = sorted(glob.glob(str(HERE.parent.parent / "artefacte_lot1/runs_deepfake/**/*.mp4"), recursive=True),
key=lambda p: os.path.getsize(p))
if cands:
subprocess.run(["cp", cands[0], str(mp4)])
if not (ASSETS / "frame.jpg").exists() or not (ASSETS / "audio.wav").exists():
# genereaza un cadru + audio din containerul extractors (are ffmpeg)
docker(["cp", str(mp4), "didiAI-extractors:/tmp/s.mp4"])
docker(["exec", "didiAI-extractors", "sh", "-c",
"ffmpeg -y -i /tmp/s.mp4 -frames:v 1 /tmp/frame.jpg -vn -ar 16000 -ac 1 -t 8 /tmp/audio.wav"])
docker(["cp", "didiAI-extractors:/tmp/frame.jpg", str(ASSETS / "frame.jpg")])
docker(["cp", "didiAI-extractors:/tmp/audio.wav", str(ASSETS / "audio.wav")])
# --- helper request prin curl -----------------------------------------------
def run_curl(args: list[str], timeout: int) -> tuple[int, str, float]:
"""Return (http_code, body, latency_ms)."""
full = ["curl", "-s", "-m", str(timeout), "-w", "\n%{http_code}", *args]
t0 = time.monotonic()
try:
p = subprocess.run(full, capture_output=True, text=True, timeout=timeout + 10)
out = p.stdout
except subprocess.TimeoutExpired:
return (0, "TIMEOUT", round((time.monotonic() - t0) * 1000, 1))
lat = round((time.monotonic() - t0) * 1000, 1)
code = 0; body = out
if "\n" in out:
body, _, last = out.rpartition("\n")
try: code = int(last.strip())
except ValueError: code = 0
return (code, body, lat)
def jparse(body: str):
try: return json.loads(body)
except Exception: return None
# --- definitia testelor ------------------------------------------------------
def t_llm():
c, b, l = run_curl([f"http://{H}:14011/v1/chat/completions", "-H", "Content-Type: application/json",
"-d", json.dumps({"model":"qwen3.5","messages":[{"role":"user","content":"Ce este un deepfake? O propozitie."}],"max_tokens":120})], 90)
d = jparse(b); txt = (d or {}).get("choices",[{}])[0].get("message",{}).get("content","") if d else ""
return c, l, bool(txt) and not txt.lower().startswith("thinking"), {"reply": txt[:300]}
def t_embeddings():
c,b,l = run_curl([f"http://{H}:14100/v1/embeddings","-H","Content-Type: application/json","-d",json.dumps({"model":"bge-m3","input":"test"})],30)
d=jparse(b); dim=len((d or {}).get("data",[{}])[0].get("embedding",[])) if d else 0
return c,l,dim==1024,{"dim":dim}
def t_rerank():
c,b,l=run_curl([f"http://{H}:14200/v1/rerank","-H","Content-Type: application/json","-d",json.dumps({"model":"bge-reranker-v2-m3","query":"deepfake detection","documents":["o pisica","sistem detectie deepfake video"]})],30)
d=jparse(b); r=(d or {}).get("results",[]) ; top=r[0] if r else {}
return c,l,top.get("index")==1,{"top_index":top.get("index"),"score":round(top.get("relevance_score",0),4)}
def t_audio():
c,b,l=run_curl([f"http://{H}:54300/v1/audio/transcriptions","-F",f"file=@{ASSETS}/audio.wav","-F",f"model={WHISPER}"],120)
d=jparse(b); txt=(d or {}).get("text","") if d else ""
return c,l,bool(txt.strip()),{"text":txt[:200],"language":(d or {}).get("language")}
def t_video():
c,b,l=run_curl([f"http://{H}:54600/analyze/video","-F",f"file=@{ASSETS}/sample.mp4"],180)
d=jparse(b); v=(d or {}).get("verdict")
return c,l,v in ("REAL","FAKE","UNCERTAIN"),{"verdict":v,"frames_analyzed":(d or {}).get("frames_analyzed")}
def t_metadata():
c,b,l=run_curl([f"http://{H}:54400/v1/metadata","-F",f"file=@{ASSETS}/sample.mp4"],40)
d=jparse(b); ok=bool((d or {}).get("analyses",{}).get("integrity",{}).get("ok"))
return c,l,ok,{"media_type":(d or {}).get("media_type"),"sha256":(d or {}).get("sha256","")[:16]}
def t_ner():
c,b,l=run_curl([f"http://{H}:54400/v1/ner","-H","Content-Type: application/json","-d",json.dumps({"text":"Klaus Iohannis s-a intalnit cu Emmanuel Macron la Bucuresti."})],60)
d=jparse(b); ents=(d or {}).get("results",{}).get("entities",[])
return c,l,len(ents)>=2,{"entities":[e.get("text")+":"+e.get("label","") for e in ents][:5]}
def t_detect():
c,b,l=run_curl([f"http://{H}:54400/v1/detect","-F",f"file=@{ASSETS}/frame.jpg"],90)
d=jparse(b); det=(d or {}).get("results",{}).get("detections",[])
return c,l,len(det)>=1,{"objects":len(det),"labels":[x.get("label") for x in det][:5]}
def t_ocr():
c,b,l=run_curl([f"http://{H}:54400/v1/ocr","-F",f"file=@{ASSETS}/frame.jpg"],90)
d=jparse(b); ok=bool((d or {}).get("ok")); txt=(d or {}).get("results",{}).get("text","")
return c,l,ok,{"text":txt[:150]}
def t_sentiment():
c,b,l=run_curl([f"http://{H}:54400/v1/sentiment","-H","Content-Type: application/json","-d",json.dumps({"text":"Produsul este excelent, sunt foarte multumit!"})],60)
d=jparse(b); ok=bool((d or {}).get("ok"))
return c,l,ok,{"label":(d or {}).get("results",{}).get("label"),"score":(d or {}).get("results",{}).get("score")}
def t_forensic():
c,b,l=run_curl([f"http://{H}:8085/api/forensic-modules"],15)
d=jparse(b); m=(d or {}).get("available_modules",[])
return c,l,len(m)>=3,{"modules":[x.get("id") for x in m]}
def t_web():
c,b,l=run_curl([f"http://{H}:51100/v1/search","-H","Content-Type: application/json","-d",json.dumps({"queries":["deepfake detection"],"max_results":3})],40)
d=jparse(b); r=(d or {}).get("results",[])
return c,l,len(r)>0,{"results":len(r),"first":(r[0].get("title","")[:60] if r else None)}
def t_catalog():
c,b,l=run_curl([f"http://{H}:11000/catalog/v1/status","-H",f"Authorization: Bearer {TOKEN}"],15)
d=jparse(b); st=(d or {}).get("status")
return c,l,st in ("healthy","degraded"),{"status":st,"components":[x.get("component_id") for x in (d or {}).get("components",[])]}
def t_gateway():
c,b,l=run_curl([f"http://{H}:11000/health"],10)
c2,_,_=run_curl([f"http://{H}:11000/web/v1/info"],10) # fara token -> 401
return c,l,(c==200 and c2==401),{"health":c,"web_no_auth":c2}
def t_brain_atoms():
c,b,l=run_curl([f"http://{H}:8090/v1/analysis_atom/stats/extended"],15)
d=jparse(b)
return c,l,(d is not None and "total_atoms" in d),{"total_atoms":(d or {}).get("total_atoms"),"by_tier":(d or {}).get("by_tier")}
def t_brain_facts():
c,b,l=run_curl([f"http://{H}:8090/v1/fact_status/list?page=1&page_size=10"],15)
d=jparse(b)
return c,l,(d is not None and "total" in d),{"total":(d or {}).get("total")}
def t_brain_vcache():
c,b,l=run_curl([f"http://{H}:8090/v1/verification_cache/list?page=1&page_size=10"],15)
d=jparse(b)
return c,l,(d is not None and "total" in d),{"total":(d or {}).get("total")}
def t_llm_completions():
c,b,l=run_curl([f"http://{H}:14011/v1/completions","-H","Content-Type: application/json","-d",json.dumps({"model":"qwen3.5","prompt":"Capitala Romaniei este","max_tokens":10})],60)
d=jparse(b); txt=(d or {}).get("choices",[{}])[0].get("text","") if d else ""
return c,l,bool(txt.strip()),{"text":txt[:120]}
def t_llm_info():
c,b,l=run_curl([f"http://{H}:14011/v1/info"],15); d=jparse(b)
return c,l,isinstance(d,dict) and len(d)>0,{"keys":list(d.keys())[:6] if isinstance(d,dict) else None}
def t_video_semantic():
c,b,l=run_curl([f"http://{H}:54600/analyze/video/semantic","-F",f"file=@{ASSETS}/sample.mp4"],180)
d=jparse(b); ok=bool(d) and ("final_summary" in d or "chunk_results" in d)
return c,l,ok,{"num_chunks":(d or {}).get("num_chunks"),"summary":(str((d or {}).get("final_summary",""))[:120])}
def t_web_fetch():
c,b,l=run_curl([f"http://{H}:51100/v1/fetch","-H","Content-Type: application/json","-d",json.dumps({"urls":["https://en.wikipedia.org/wiki/Deepfake"]})],60)
ok=(c==200 and len(b)>1500 and ("title" in b or "content" in b or "text" in b))
return c,l,ok,{"bytes":len(b)}
def t_web_gather():
c,b,l=run_curl([f"http://{H}:51100/v1/gather","-H","Content-Type: application/json","-d",json.dumps({"claim":"Deepfakes can be detected by AI","max_search_results":5})],150)
d=jparse(b); ev=(d or {}).get("evidence",[]); n=(d or {}).get("total_evidence_items",len(ev) if isinstance(ev,list) else 0)
return c,l,bool(d) and (n>0 or (isinstance(ev,list) and len(ev)>0)),{"evidence_items":n,"urls_found":(d or {}).get("total_urls_found")}
def t_forensic_evidence():
c,b,l=run_curl([f"http://{H}:8085/api/forensic-evidence","-F",f"video=@{ASSETS}/sample.mp4","-F","modules=m29","-F","every_n_frames=30"],150)
d=jparse(b); mr=(d or {}).get("modules_run",[])
return c,l,bool(mr),{"modules_run":mr,"frames":(d or {}).get("n_frames_extracted")}
def t_catalog_components():
c,b,l=run_curl([f"http://{H}:11000/catalog/v1/components","-H",f"Authorization: Bearer {TOKEN}"],15)
d=jparse(b); comp=(d or {}).get("components",d if isinstance(d,list) else [])
return c,l,c==200 and len(comp)>0,{"components":len(comp)}
def t_catalog_models():
c,b,l=run_curl([f"http://{H}:11000/catalog/v1/models","-H",f"Authorization: Bearer {TOKEN}"],15)
d=jparse(b)
return c,l,c==200 and d is not None,{"models":len((d or {}).get("models",d if isinstance(d,list) else []))}
def t_brain_search():
c,b,l=run_curl([f"http://{H}:8090/v1/search","-H","Content-Type: application/json","-d",json.dumps({"queries":["deepfake"],"max_results":2})],60)
d=jparse(b)
return c,l,bool(d) and "results" in d,{"total_results":(d or {}).get("total_results"),"cache":(d or {}).get("brain_meta",{}).get("cache_status")}
def t_brain_gather():
c,b,l=run_curl([f"http://{H}:8090/v1/gather","-H","Content-Type: application/json","-d",json.dumps({"claim":"Deepfakes can be detected by AI"})],120)
d=jparse(b)
return c,l,bool(d) and "evidence" in d,{"evidence_items":(d or {}).get("total_evidence_items"),"urls":(d or {}).get("total_urls_found")}
def t_dashboard_health():
c,b,l=run_curl([f"http://{H}:51300/health"],10); d=jparse(b)
return c,l,(d or {}).get("status") in ("healthy","ok"),{"status":(d or {}).get("status"),"db":(d or {}).get("db")}
def t_dashboard_monitoring():
c,b,l=run_curl([f"http://{H}:51300/api/monitoring/services"],20); d=jparse(b)
s=d if isinstance(d,list) else (d or {}).get("services",[])
h=sum(1 for x in s if x.get("status")=="healthy")
return c,l,len(s)>0 and h==len(s),{"healthy":f"{h}/{len(s)}"}
def t_cloak():
c,b,l=run_curl([f"http://{H}:8770/v1/search","-H","Content-Type: application/json","-d",json.dumps({"queries":["deepfake"],"engines":["ddg"],"max_results_per_engine":3})],45)
d=jparse(b); r=(d or {}).get("results",[])
return c,l,len(r)>0,{"results":len(r),"first":(r[0].get("title","")[:55] if r else None)}
TESTS = [
("llm","LLM (Qwen3.5-35B-A3B)","POST /v1/chat/completions",t_llm),
("llm_completions","LLM · completions","POST /v1/completions",t_llm_completions),
("llm_info","LLM · info","GET /v1/info",t_llm_info),
("embeddings","Embeddings (bge-m3)","POST /v1/embeddings",t_embeddings),
("rerank","Rerank (bge-reranker-v2-m3)","POST /v1/rerank",t_rerank),
("audio","Audio (Whisper large-v3-turbo)","POST /v1/audio/transcriptions",t_audio),
("video","Video/BusterX (deepfake)","POST /analyze/video",t_video),
("video_semantic","Video · semantic","POST /analyze/video/semantic",t_video_semantic),
("extractors_metadata","Extractors · metadata","POST /v1/metadata",t_metadata),
("extractors_ner","Extractors · NER (GLiNER)","POST /v1/ner",t_ner),
("extractors_detect","Extractors · detect (YOLOv8)","POST /v1/detect",t_detect),
("extractors_ocr","Extractors · OCR (LLM vision)","POST /v1/ocr",t_ocr),
("extractors_sentiment","Extractors · sentiment (LLM)","POST /v1/sentiment",t_sentiment),
("forensic","Forensic · module list","GET /api/forensic-modules",t_forensic),
("forensic_evidence","Forensic · analiza reala","POST /api/forensic-evidence",t_forensic_evidence),
("web","Web · search (SearXNG)","POST /v1/search",t_web),
("web_fetch","Web · fetch","POST /v1/fetch",t_web_fetch),
("web_gather","Web · gather (evidence)","POST /v1/gather",t_web_gather),
("catalog","Catalog · status","GET /catalog/v1/status",t_catalog),
("catalog_components","Catalog · components","GET /catalog/v1/components",t_catalog_components),
("catalog_models","Catalog · models","GET /catalog/v1/models",t_catalog_models),
("gateway","Gateway (auth+rutare)","GET /health + 401",t_gateway),
("brain_atoms","Brain · analysis_atom","GET /v1/analysis_atom/stats/extended",t_brain_atoms),
("brain_facts","Brain · fact_status","GET /v1/fact_status/list",t_brain_facts),
("brain_vcache","Brain · verification_cache","GET /v1/verification_cache/list",t_brain_vcache),
("brain_search","Brain · search (RAG)","POST /v1/search",t_brain_search),
("brain_gather","Brain · gather (RAG)","POST /v1/gather",t_brain_gather),
("dashboard","Dashboard · health","GET /health",t_dashboard_health),
("dashboard_monitoring","Dashboard · monitoring 11/11","GET /api/monitoring/services",t_dashboard_monitoring),
("cloak","Cloak · SERP scraper","POST /v1/search",t_cloak),
]
def main() -> int:
ensure_assets()
ts = datetime.datetime.now().astimezone()
stamp = ts.strftime("%Y%m%d_%H%M%S")
results = []
print(f"\n DiDi LOT 1 — test run @ {ts.isoformat()} (host {H})\n")
for tid, name, ep, fn in TESTS:
try:
code, lat, ok, extra = fn()
except Exception as e: # noqa: BLE001
code, lat, ok, extra = 0, 0.0, False, {"error": str(e)}
mark = "\033[32mPASS\033[0m" if ok else "\033[31mFAIL\033[0m"
print(f" [{mark}] {name:34} {ep:38} HTTP {code:>3} {lat:>7.0f}ms {json.dumps(extra, ensure_ascii=False)[:80]}")
results.append({"id":tid,"name":name,"endpoint":ep,"http_code":code,
"ok":ok,"latency_ms":lat,"result":extra,"checked_at":ts.isoformat()})
passed = sum(1 for r in results if r["ok"])
report = {
"generated_at": ts.isoformat(),
"host": H,
"summary": {"total": len(results), "passed": passed, "failed": len(results) - passed},
"tests": results,
}
REPORTS.mkdir(exist_ok=True)
out = REPORTS / f"test_report_{stamp}.json"
out.write_text(json.dumps(report, indent=2, ensure_ascii=False))
latest = REPORTS / "latest.json"
latest.write_text(json.dumps(report, indent=2, ensure_ascii=False))
print(f"\n === {passed}/{len(results)} PASS ===")
print(f" Raport JSON: {out}")
print(f" (si copie: {latest})\n")
return 0 if passed == len(results) else 1
if __name__ == "__main__":
sys.exit(main())

49
ai_platform/local_gpu_stack/seed/seed_brain.sh Normal file
View file

@ -0,0 +1,49 @@
#!/usr/bin/env bash
# Brain stack helpers: render env, align postgres password, create atomic token,
# seed the canonical taxonomy. Idempotent. Requires lib.sh sourced + master .env
# loaded ($POSTGRES_PASSWORD, $ATOMIC_TOKEN, $HOST_IP, $ENV_FILE, $BRAIN_DIR).
# Write didi_brain/.env (read by brain-api/scheduler) + infra/.env (compose interpolation)
brain_write_env() {
local pw=$1 tok=$2
cp "${BRAIN_DIR}/.env.example" "${BRAIN_DIR}/.env"
sed -i \
-e "s#^LLM_ROUTER_URL=.*#LLM_ROUTER_URL=http://didiAI-llm-api:14011#" \
-e "s#^LLM_VLLM_URL=.*#LLM_VLLM_URL=http://didiAI-vllm-qwen3.5:14001#" \
-e "s#^EMBEDDING_URL=.*#EMBEDDING_URL=http://didiAI-embeddings-api:14100#" \
-e "s#^RERANKER_URL=.*#RERANKER_URL=http://didiAI-rerank-api:14200#" \
-e "s#^MODEL_FAST_ENABLED=.*#MODEL_FAST_ENABLED=true#" \
-e "s#^POSTGRES_PASSWORD=.*#POSTGRES_PASSWORD=${pw}#" \
"${BRAIN_DIR}/.env"
if grep -q '^ATOMIC_TOKEN=' "${BRAIN_DIR}/.env"; then
sed -i "s#^ATOMIC_TOKEN=.*#ATOMIC_TOKEN=${tok}#" "${BRAIN_DIR}/.env"
else
echo "ATOMIC_TOKEN=${tok}" >> "${BRAIN_DIR}/.env"
fi
printf 'POSTGRES_USER=atomic\nPOSTGRES_PASSWORD=%s\nPOSTGRES_DB=atomic\n' "$pw" > "${BRAIN_DIR}/infra/.env"
}
# Force the postgres role password to match (volume may have been initialized with a different one)
brain_align_pg() {
local pw=$1
docker exec didibrain-postgres psql -U atomic -d atomic \
-c "ALTER USER atomic WITH PASSWORD '${pw}';" >/dev/null 2>&1 || true
}
# Create an atomic API token if the master .env doesn't have one yet. Echos the token.
brain_ensure_token() {
local pw=$1 cur=$2
if [ -n "$cur" ]; then printf '%s' "$cur"; return 0; fi
local out tok
out=$(docker exec didibrain-atomic atomic-server token create --name didibrain \
--storage postgres --database-url "postgres://atomic:${pw}@postgres:5432/atomic" 2>&1)
tok=$(printf '%s' "$out" | grep -oE 'at_[A-Za-z0-9_.-]{20,}' | head -1)
printf '%s' "$tok"
}
# Seed canonical taxonomy into atomic (idempotent; tags created if missing).
brain_seed_taxonomy() {
docker cp "${BRAIN_DIR}/scripts" didibrain-api:/app/scripts >/dev/null 2>&1 || true
# local _tag_ids.json write is read-only in the image — harmless, atomic gets the tags
docker exec -e PYTHONPATH=/app didibrain-api python /app/scripts/04_seed_taxonomy.py >/dev/null 2>&1 || true
}

36
ai_platform/local_gpu_stack/seed/seed_catalog.sh Normal file
View file

@ -0,0 +1,36 @@
#!/usr/bin/env bash
# Idempotent seed of the dashboard DB-backed Model & Extractor catalog.
# Defines seed_catalog(); requires lib.sh sourced + $HOST_IP set.
seed_catalog() {
local base="http://${HOST_IP}:51300/api/catalog"
# existing names (idempotency)
local existing
existing=$(curl -s -m10 "$base" | python3 -c 'import sys,json
try:
d=json.load(sys.stdin); items=d if isinstance(d,list) else d.get("items",d.get("data",[]))
print("\n".join(x.get("name","") for x in items))
except Exception: pass' 2>/dev/null)
local entries=(
'{"kind":"model","name":"qwen3.5","display_name":"Qwen3.5-35B-A3B","service":"llm","capabilities":["chat","completion","reasoning"],"context_length":32000,"supports_gpu":true,"endpoint":"http://HOST:14011","notes":"vLLM MoE, GPU 0"}'
'{"kind":"model","name":"bge-m3","display_name":"BGE-M3 Embeddings","service":"embeddings","capabilities":["embeddings"],"context_length":8192,"supports_gpu":true,"endpoint":"http://HOST:14100","notes":"1024-dim, vLLM GPU 1"}'
'{"kind":"model","name":"bge-reranker-v2-m3","display_name":"BGE Reranker v2-m3","service":"rerank","capabilities":["rerank"],"context_length":8192,"supports_gpu":true,"endpoint":"http://HOST:14200","notes":"cross-encoder, vLLM GPU 1"}'
'{"kind":"model","name":"busterx","display_name":"BusterX++ (deepfake)","service":"video","capabilities":["deepfake-detection","video-vlm"],"context_length":32768,"supports_gpu":true,"endpoint":"http://HOST:54600","notes":"Qwen2.5-VL based, vLLM GPU 1"}'
'{"kind":"model","name":"whisper-large-v3-turbo","display_name":"Whisper large-v3-turbo","service":"audio","capabilities":["transcription","stt"],"supports_gpu":true,"endpoint":"http://HOST:54300","notes":"faster-whisper, GPU 0"}'
'{"kind":"extractor","name":"gliner-multi-v2.1","display_name":"GLiNER NER (multilingual)","service":"extractors","capabilities":["ner"],"supports_gpu":true,"endpoint":"http://HOST:54400/v1/ner","notes":"validat RO"}'
'{"kind":"extractor","name":"yolov8n","display_name":"YOLOv8 Object Detection","service":"extractors","capabilities":["object-detection"],"supports_gpu":true,"endpoint":"http://HOST:54400/v1/detect"}'
'{"kind":"extractor","name":"metadata","display_name":"Media Metadata (EXIF/ELA/ffprobe/C2PA)","service":"extractors","capabilities":["exif","ela","spectrogram","integrity"],"supports_cpu":true,"endpoint":"http://HOST:54400/v1/metadata"}'
'{"kind":"extractor","name":"forensic-suite","display_name":"Forensic detectors (rPPG/lip-sync/forgery/lighting)","service":"forensic","capabilities":["rppg","lip-sync","forgery-heatmap","lighting"],"supports_gpu":true,"endpoint":"http://HOST:8085/api/forensic-evidence"}'
)
local added=0 skipped=0 e name
for e in "${entries[@]}"; do
name=$(printf '%s' "$e" | python3 -c 'import sys,json;print(json.load(sys.stdin)["name"])')
if printf '%s\n' "$existing" | grep -qx "$name"; then skipped=$((skipped+1)); continue; fi
e=${e//HOST/$HOST_IP}
if [ "$(curl -s -m10 -o /dev/null -w '%{http_code}' -X POST "$base" -H 'Content-Type: application/json' -d "$e")" = "200" ]; then
added=$((added+1))
fi
done
echo "catalog: +$added adaugate, $skipped existau deja"
}

56
ai_platform/modules/audio/API.md
View file

@ -8,9 +8,8 @@ OpenAI-compatible speech-to-text transcription API using faster-whisper.
{BASE_URL}
```
- **Local development:** `http://localhost:8200`
- **Docker (internal):** `http://audio-api:8200`
- **Direct:** `http://localhost:54300`
- **Local development:** `http://localhost:54300`
- **Docker (internal):** `http://audio-api:54300`
- **Production:** Use your configured hostname
## Authentication
@ -34,7 +33,7 @@ Check API health status.
**Example:**
```bash
curl http://localhost:8200/health
curl http://localhost:54300/health
```
---
@ -62,7 +61,20 @@ List available Whisper models (OpenAI-compatible).
**Example:**
```bash
curl http://localhost:8200/v1/models
curl http://localhost:54300/v1/models
```
---
### Service Info
Return service catalog metadata (resources, models, functions). Consumed by the DIDI `catalog-api`.
**Endpoint:** `GET /v1/info`
**Example:**
```bash
curl http://localhost:54300/v1/info
```
---
@ -79,13 +91,16 @@ Transcribe audio file to text (OpenAI-compatible endpoint).
| Parameter | Type | Required | Default | Description |
|-----------|------|----------|---------|-------------|
| `file` | file | Yes | - | Audio file to transcribe (MP3, WAV, M4A, etc.) |
| `file` | file | Yes* | - | Audio file to transcribe (MP3, WAV, M4A, etc.). Provide either `file` or `url` (XOR). |
| `url` | string | Yes* | - | URL to download the audio from (DIDI extension over OpenAI). Provide either `file` or `url` (XOR). |
| `model` | string | No | `large-v3-turbo` | Model to use (currently ignored, uses configured model) |
| `language` | string | No | `null` | Language code (ISO-639-1). Auto-detected if not specified. |
| `prompt` | string | No | `null` | Optional text to guide the model's style |
| `response_format` | string | No | `json` | Format: `json`, `text`, or `verbose_json` |
| `temperature` | float | No | `0.0` | Sampling temperature (0.0-1.0). Use 0.0 for deterministic output. |
\* `file` and `url` are mutually exclusive — supply exactly one.
**Supported Languages (ISO-639-1 codes):**
`en`, `es`, `fr`, `de`, `it`, `pt`, `nl`, `pl`, `tr`, `ru`, `ja`, `ko`, `zh`, `ar`, `hi`, and 90+ more languages.
@ -132,14 +147,21 @@ Full transcription text
**Basic transcription (JSON):**
```bash
curl -X POST "http://localhost:8200/v1/audio/transcriptions" \
curl -X POST "http://localhost:54300/v1/audio/transcriptions" \
-F "file=@audio.mp3" \
-F "response_format=json"
```
**From a URL (instead of file upload):**
```bash
curl -X POST "http://localhost:54300/v1/audio/transcriptions" \
-F "url=https://example.com/audio.mp3" \
-F "response_format=json"
```
**With language specification:**
```bash
curl -X POST "http://localhost:8200/v1/audio/transcriptions" \
curl -X POST "http://localhost:54300/v1/audio/transcriptions" \
-F "file=@audio.mp3" \
-F "language=en" \
-F "response_format=json"
@ -147,21 +169,21 @@ curl -X POST "http://localhost:8200/v1/audio/transcriptions" \
**Text format:**
```bash
curl -X POST "http://localhost:8200/v1/audio/transcriptions" \
curl -X POST "http://localhost:54300/v1/audio/transcriptions" \
-F "file=@audio.mp3" \
-F "response_format=text"
```
**Verbose JSON with segments:**
```bash
curl -X POST "http://localhost:8200/v1/audio/transcriptions" \
curl -X POST "http://localhost:54300/v1/audio/transcriptions" \
-F "file=@audio.mp3" \
-F "response_format=verbose_json"
```
**With initial prompt (to guide style):**
```bash
curl -X POST "http://localhost:8200/v1/audio/transcriptions" \
curl -X POST "http://localhost:54300/v1/audio/transcriptions" \
-F "file=@audio.mp3" \
-F "prompt=This is a technical discussion about machine learning." \
-F "response_format=json"
@ -171,7 +193,7 @@ curl -X POST "http://localhost:8200/v1/audio/transcriptions" \
```python
import requests
url = "http://localhost:8200/v1/audio/transcriptions"
url = "http://localhost:54300/v1/audio/transcriptions"
with open("audio.mp3", "rb") as f:
files = {"file": f}
@ -196,7 +218,7 @@ formData.append('file', audioFile);
formData.append('model', 'large-v3-turbo');
formData.append('response_format', 'json');
const response = await fetch('http://localhost:8200/v1/audio/transcriptions', {
const response = await fetch('http://localhost:54300/v1/audio/transcriptions', {
method: 'POST',
body: formData
});
@ -247,7 +269,7 @@ limit_req_zone $binary_remote_addr zone=audio_limit:10m rate=10r/m;
location / {
limit_req zone=audio_limit burst=5;
proxy_pass http://audio-api:8200;
proxy_pass http://audio-api:54300;
}
```
@ -330,7 +352,7 @@ from openai import OpenAI
# Point to local API
client = OpenAI(
api_key="not-needed", # No auth required
base_url="http://localhost:8200/v1"
base_url="http://localhost:54300/v1"
)
with open("audio.mp3", "rb") as f:
@ -351,7 +373,7 @@ import fs from 'fs';
const openai = new OpenAI({
apiKey: 'not-needed',
baseURL: 'http://localhost:8200/v1'
baseURL: 'http://localhost:54300/v1'
});
const transcription = await openai.audio.transcriptions.create({
@ -391,7 +413,7 @@ Configurable via `AUDIO_BEAM_SIZE` (default: 5). Higher values = better accuracy
```bash
# Check if API is ready
curl http://localhost:8200/health
curl http://localhost:54300/health
# Expected response
{"status": "ok"}

8
ai_platform/modules/audio/INDEX.md
View file

@ -8,7 +8,7 @@ Audio transcription service for DIDI media analysis. Whisper-based (M17-Whisper,
- Default model: `large-v3-turbo` (809M params, ~6GB VRAM int8)
- GPU: CUDA (shared GPU 0 with Qwen3.5-35B-A3B)
- URL (Dev): `http://10.11.10.17:54300/v1/audio/transcriptions`
- Container: `didiAI-audio-api` (GPU host)
- Container: `didiAI-audio` (GPU host)
- Auth: none on the service itself; agent-v3 uses bearer token via `M17_WHISPER_TOKEN` (enforced by gateway/nginx if configured)
## Ce face
@ -92,8 +92,8 @@ cp ../.env.example .env # edit values
docker compose restart audio-api # quick restart
```
- Container name: `didiAI-audio-api`
- Image: `didiai-audio-api`
- Container name: `didiAI-audio`
- Image: `didiai-audio:audit`
- Network: `didi-network` (external, shared with other AI modules)
- GPU reservation: NVIDIA driver, device `0`
- Healthcheck: HTTP `GET /health` every 30s, 60s start period (model load)
@ -111,7 +111,7 @@ docker compose restart audio-api # quick restart
modules/audio/
├── deploy/
│ ├── deploy.sh # CLI wrapper
│ ├── docker-compose.yml # didiAI-audio-api service
│ ├── docker-compose.yml # didiAI-audio service
│ ├── Dockerfile # CUDA + faster-whisper image
│ └── .env # runtime config
├── src/audio/

11
ai_platform/modules/audio/README.md
View file

@ -61,22 +61,23 @@ cp ../.env.example .env
| `/health` | GET | Health check |
| `/v1/models` | GET | List available models |
| `/v1/audio/transcriptions` | POST | Transcribe audio (OpenAI-compatible) |
| `/v1/info` | GET | Service catalog metadata (used by catalog-api) |
### Example API Request
```bash
# Health check
curl http://localhost:8200/health
curl http://localhost:54300/health
# Transcribe audio file
AUDIO="/path/to/audio.mp3"
curl -X POST "http://localhost:8200/v1/audio/transcriptions" \
curl -X POST "http://localhost:54300/v1/audio/transcriptions" \
-F "file=@${AUDIO}" \
-F "model=large-v3-turbo" \
-F "response_format=json"
# With language specification and verbose output
curl -X POST "http://localhost:8200/v1/audio/transcriptions" \
curl -X POST "http://localhost:54300/v1/audio/transcriptions" \
-F "file=@${AUDIO}" \
-F "language=en" \
-F "response_format=verbose_json"
@ -193,8 +194,8 @@ docker compose logs -f audio-api
| Port | Service |
|------|---------|
| `8200` | Audio API |
| `54300` | Audio API (Dev + AI + Audio) |
| `54300` | Audio API (Dev + AI + Audio) — primary |
| `8200` | Audio API (legacy default) |
## Development

10
ai_platform/modules/audio/deploy/docker-compose.yml
View file

@ -19,10 +19,10 @@
# Naming Convention: didiAI-{module}-{service}
#
# Network:
# Uses deploy_default network (shared with other modules)
# Uses the external didi-network bridge (shared with other AI modules)
networks:
deploy_default:
didi-network:
external: true
services:
@ -30,8 +30,8 @@ services:
# Audio Transcription API Server
# ==========================================================================
audio-api:
container_name: didiAI-audio-api
image: didiai-audio-api
container_name: didiAI-audio
image: didiai-audio:audit
build:
context: ..
dockerfile: deploy/Dockerfile
@ -39,7 +39,7 @@ services:
ports:
- "54300:54300"
networks:
- deploy_default
- didi-network
environment:
# GPU configuration

4
ai_platform/modules/catalog-api/INDEX.md
View file

@ -111,7 +111,7 @@ All settings come from environment variables with prefix `CATALOG_` (see `src/ca
| `CATALOG_LOG_LEVEL` | `INFO` | Python logging level. |
| `CATALOG_EXTERNAL_URL` | **required** | Public base URL (e.g., `http://10.11.10.42`) injected into the merged OpenAPI `servers:`. `deploy.sh` aborts if missing. |
| `CATALOG_LLM_URL` | `http://didiAI-llm-api:14011` | LLM Inference internal URL. |
| `CATALOG_AUDIO_URL` | `http://didiAI-audio-api:54300` | Audio API internal URL. |
| `CATALOG_AUDIO_URL` | `http://didiAI-audio:54300` | Audio API internal URL. |
| `CATALOG_VIDEO_URL` | *(empty)* | Video Analysis internal URL — empty string disables Video. |
| `CATALOG_WEB_URL` | `http://didiAI-web-api:51100` | Web API internal URL. |
| `CATALOG_LLM_EXTERNAL_PORT` | `14011` | External port advertised in the merged OpenAPI for LLM. |
@ -136,7 +136,7 @@ All settings come from environment variables with prefix `CATALOG_` (see `src/ca
This service stands on top of the rest of the `ai_platform/modules/*` family — they are its data sources:
- `llm-inference` (port `14011`, container `didiAI-llm-api`) — chat, embeddings, rerank.
- `audio` (port `54300`, container `didiAI-audio-api`) — transcription / TTS.
- `audio` (port `54300`, container `didiAI-audio`) — transcription / TTS.
- `video-analysis` (port `54600`, container `didiAI-video-api`) — vision pipelines (currently disabled by default in the .env example).
- `web` (port `51100`, container `didiAI-web-api`) — fact-check / web crawler.
- `dashboard` — primary frontend consumer of the merged OpenAPI / `/v1/status`.

9
ai_platform/modules/catalog-api/README.md
View file

@ -15,7 +15,7 @@ This module provides a unified API to discover and query all available ML servic
**Required:**
- All global prerequisites (see main [README.md](../../README.md))
- Docker network `deploy_default` (shared with other modules)
- Docker network `didi-network` (shared with other modules)
- At least one other module running (llm-inference, audio, video-analysis, or web)
## Quick Start
@ -60,12 +60,13 @@ Configure via environment variables (prefix: `CATALOG_`):
| Variable | Default | Description |
|----------|---------|-------------|
| `CATALOG_EXTERNAL_URL` | _(required, no default)_ | External base URL for the aggregated OpenAPI spec (e.g. `http://10.11.10.42`). Service fails to start if unset. |
| `CATALOG_HOST` | `0.0.0.0` | Server host |
| `CATALOG_PORT` | `11000` | Server port (Production API Gateway: 11000) |
| `CATALOG_LOG_LEVEL` | `INFO` | Log level |
| `CATALOG_LLM_URL` | `http://didiAI-llm-api:14011` | LLM Inference API URL |
| `CATALOG_AUDIO_URL` | `http://didiAI-audio-api:54300` | Audio API URL |
| `CATALOG_VIDEO_URL` | `http://didiAI-video-api:54600` | Video Analysis API URL |
| `CATALOG_AUDIO_URL` | `http://didiAI-audio:54300` | Audio API URL |
| `CATALOG_VIDEO_URL` | _(empty)_ | Video Analysis API URL — empty by default, so video is disabled unless set |
| `CATALOG_WEB_URL` | `http://didiAI-web-api:51100` | Web API URL |
| `CATALOG_COMPONENT_TIMEOUT` | `10` | Component request timeout (seconds) |
@ -242,7 +243,7 @@ curl http://localhost:11000/v1/components | jq
This module aggregates information from:
- `llm-inference` (port 14011, Docker internal: didiAI-llm-api)
- `audio` (port 54300, Docker internal: didiAI-audio-api)
- `audio` (port 54300, Docker internal: didiAI-audio)
- `video-analysis` (port 54600, Docker internal: didiAI-video-api)
- `web` (port 51100, Docker internal: didiAI-web-api)

4
ai_platform/modules/catalog-api/deploy/docker-compose.yml
View file

@ -4,7 +4,7 @@
# 11000 - Catalog API (main orchestrator gateway)
#
# Network:
# Uses deploy_default network (shared with other modules)
# Uses didi-network network (shared with other modules)
#
# Naming Convention: didiAI-{module}-{service}
@ -38,7 +38,7 @@ services:
# Component URLs (Docker internal network)
- CATALOG_LLM_URL=${CATALOG_LLM_URL:-http://didiAI-llm-api:14011}
- CATALOG_AUDIO_URL=${CATALOG_AUDIO_URL:-http://didiAI-audio-api:54300}
- CATALOG_AUDIO_URL=${CATALOG_AUDIO_URL:-http://didiAI-audio:54300}
- CATALOG_VIDEO_URL=${CATALOG_VIDEO_URL}
- CATALOG_WEB_URL=${CATALOG_WEB_URL:-http://didiAI-web-api:51100}

4
ai_platform/modules/cloak/INDEX.md
View file

@ -49,8 +49,8 @@ end-to-end.
## Where it's consumed
- `ai_platform/modules/web/src/web/search/cloak.py` (CloakHTTPClient) — pending
- `ai_platform/modules/web/src/web/orchestrator.py::_run_search_stage` — pending tier-3 hook
- `ai_platform/modules/web/src/web/search/cloak.py` (CloakHTTPClient) — implemented
- `ai_platform/modules/web/src/web/orchestrator.py::_run_search_stage` — implemented (tier-3 fallback hook, orchestrator.py:564-599)
Backend services (`agent-v3`, `didi-framework`, admin-dashboard) do NOT call
this service directly.

68
ai_platform/modules/dashboard/INDEX.md
View file

@ -10,7 +10,7 @@ AI platform monitoring/admin dashboard. Browse archived claims, view ingest hist
- Python 3.10+ (FastAPI 0.115+, Uvicorn)
- React 19 + MUI 7 + Vite 7 + react-router 7 + TanStack Query + Recharts (built into `/app/web_dist/`, served as static via SPAStaticFiles with index-fallback)
- Jinja2 templates legacy (HTMX + Alpine.js + Tailwind CDN — kept until React reaches 100% parity)
- SQLAlchemy 2.0 async + asyncpg, Alembic
- SQLAlchemy 2.0 async + asyncpg (tables created via `Base.metadata.create_all` at startup; Alembic is only a dependency, not used at runtime — no migrations applied)
- pydantic-settings (env prefix `DASHBOARD_`)
- httpx for live provider quota fetching, brain proxy
- python-jose[cryptography] for Keycloak JWT validation
@ -23,23 +23,28 @@ AI platform monitoring/admin dashboard. Browse archived claims, view ingest hist
- Compose: `deploy/docker-compose.yml`, profile `dashboard`
- Network: `didi-network` (unified single network for all DIDI + AI platform stacks since 2026-05-04)
## Auth (current state — 2026-05-04)
## Auth (current state)
**Hybrid auth** in `dependencies.py:verify_bearer_token`:
1. `STAGING_MODE=true` → all auth bypassed (default for dev)
2. JWT (3 dot-separated parts) → validated via `keycloak_auth.py` (JWKS cache 10min, signature, issuer, exp, role check)
**Shipped in STAGING MODE — Keycloak is supported but disabled in this delivery.**
With `DASHBOARD_STAGING_MODE=true` (the delivered default) all auth is bypassed
and Keycloak is **not** active. JWT validation only engages when
`DASHBOARD_KEYCLOAK_URL` is set (empty value = JWT off).
**Hybrid auth** in `dependencies.py:verify_bearer_token` (in order):
1. `STAGING_MODE=true` → all auth bypassed (this is the delivered state)
2. JWT (3 dot-separated parts) → validated via `keycloak_auth.py` (JWKS cache 10min, signature, issuer, exp, role check) — only when `DASHBOARD_KEYCLOAK_URL` is set
3. DB-backed bearer tokens (legacy, from CLI `dashboard create-user`)
4. Static `api_tokens` env var (legacy fallback)
5. Otherwise → 401
**Keycloak settings** (`config.py:DashboardSettings`):
- `keycloak_url`, `keycloak_realm` (default `didi-clients`), `keycloak_client_id` (default `ai-platform-dashboard`)
**Keycloak settings** (`config.py:DashboardSettings`) — used only once Keycloak is enabled:
- `keycloak_url` (empty by default = disabled), `keycloak_realm` (default `didi-clients`), `keycloak_client_id` (default `ai-platform-dashboard`)
- `keycloak_required_role` (default `admin` — same role as DIDI admin-dashboard for unified access)
- Manual setup: create client + assign role via Keycloak admin or `deploy/setup-keycloak.sh`
**Cutover from staging to prod**:
- Set `DASHBOARD_KEYCLOAK_URL=https://sso.clossers.com`, `DASHBOARD_STAGING_MODE=false`, `VITE_STAGING_MODE=false`
- Rebuild image with build args (Dockerfile bakes Keycloak config into JS bundle at build time)
**To enable Keycloak (leave staging mode)** — example values:
- Set `DASHBOARD_KEYCLOAK_URL=https://sso.clossers.com` (example), `DASHBOARD_STAGING_MODE=false`, `VITE_STAGING_MODE=false`
- Rebuild image with build args (Dockerfile bakes Keycloak config into the JS bundle at build time)
Plan/runbook: `AI_PLATFORM_RESKIN_PLAN.md` (Phase C.8) + `/home/admin365/didi_mono/UNIFIED_KEYCLOAK_CUTOVER.md`
@ -57,6 +62,9 @@ Sections served as HTML pages (`pages.py`) + JSON-mirror endpoints in `routes/`:
## API endpoints
> All JSON routers below (except `health`) are **dual-mounted** under both `/api/*`
> and `/admin-ai/api/*` (see `app.py`: `for prefix in ("/api", "/admin-ai/api")`).
### `routes/health.py`
- `GET /health` - liveness + DB ping
- `GET /ready` - app.state populated check
@ -86,7 +94,29 @@ Sections served as HTML pages (`pages.py`) + JSON-mirror endpoints in `routes/`:
`KNOWN_KEYS` in `routes/config.py` enumerates the runtime keys consumed by `web-api`: `web.providers.{serpapi,tavily,brave,linkup,exa}.enabled`, `web.premium.strategy`, `web.premium.priority_order`, `web.openrouter.model`, `web.tier.{free,premium}.max_search_results`.
### `routes/pages.py` (Jinja HTML)
### `routes/catalog.py` (auth — Model & Extractor catalog, DB-backed CRUD)
- `GET /api/catalog` - list catalog entries (models + extractors)
- `POST /api/catalog` - create a catalog entry
- `PUT /api/catalog/{entry_id}` - update an entry
- `DELETE /api/catalog/{entry_id}` - delete an entry
### `routes/monitoring.py` (auth — AI monitoring)
- `GET /api/monitoring/services` - per-service status
- `GET /api/monitoring/queues` - queue depths/metrics
- `GET /api/monitoring/latency` - latency metrics
### `routes/proxy.py` (auth — module health proxy)
- `GET /api/proxy/{module_id}/health` - proxy to a platform module's health endpoint (URLs with `DASHBOARD_<MODULE>_HEALTH_URL` env override)
### `routes/brain_proxy.py` (brain proxy, whitelisted paths)
- `GET /api/brain/{path}` - read-only pass-through to the brain service
- `POST|PATCH|DELETE /api/brain/{path}` (auth) - mutating pass-through
- Whitelist includes: `/v1/analysis_atom/{list,stats,...}`, `/v1/verification_cache/{list,...}`, `/v1/fact_status/{list,...}`, `/v1/cache/audit_log`, `/v1/cache/invalidate`, `/v1/canonicalize`, `/v1/taxonomy`, `/v1/taxonomy/reload`
### `routes/audit.py` (auth — audit log JSON)
- `GET /api/audit` - audit log entries (config/archive mutations)
### `routes/pages.py` (Jinja HTML — legacy, not the primary UI)
- `GET /` - overview
- `GET /history`, `GET /history/{request_id}`
- `GET /providers`
@ -107,16 +137,22 @@ src/dashboard/
pricing.py # estimate_cost(endpoint, tier, provider) - per-provider USD
retention.py # 30-day rolling cleanup of request_history
api/
app.py # FastAPI factory, lifespan (init engine + provider registry), router wiring
app.py # FastAPI factory, lifespan (init engine + provider registry), router wiring (routes dual-mounted under /api + /admin-ai/api), SPA static mount
dependencies.py # get_session, get_registry, verify_bearer_token, get_username
keycloak_auth.py # Keycloak JWT validation (JWKS cache, signature/issuer/exp/role); only used when DASHBOARD_KEYCLOAK_URL set
routes/
archive.py # claims archive CRUD + promote
audit.py # GET /api/audit (audit log JSON)
brain_proxy.py # /api/brain/* whitelisted proxy to brain service
catalog.py # /api/catalog Model & Extractor catalog CRUD (DB-backed)
config.py # KNOWN_KEYS + runtime override CRUD
health.py # /health, /ready
history.py # /api/history (read-only)
ingest.py # POST /api/ingest/event (service-to-service)
pages.py # all Jinja HTML pages + HTMX handlers
stats.py # /api/stats/{providers,summary,timeline}
monitoring.py # /api/monitoring/{services,queues,latency} (AI monitoring)
pages.py # all Jinja HTML pages + HTMX handlers (legacy)
proxy.py # GET /api/proxy/{module_id}/health (module health proxy)
stats.py # /api/stats/{providers,summary,timeline,cost}
db/
models.py # Base, RequestHistory, ProviderStatsHourly, ConfigOverride, User, AuditLog,
# ClaimsArchive, ArticlesArchive, ClaimArticle
@ -194,7 +230,7 @@ Connection string format: `postgresql+asyncpg://USER:PASS@didiAI-dashboard-db:54
## Reskin plan (FUTURE - NOT done yet)
- React 19 + MUI 7 + Keycloak SSO **DONE 2026-05-02** (Phase C in `agent-v3/IMPLEMENTATION_PLAN_HIL_BRAIN.md`, detail in `AI_PLATFORM_RESKIN_PLAN.md`)
- React 19 + MUI 7 SPA **DONE 2026-05-02**; Keycloak SSO **wired but disabled in this delivery** (staging mode — enable by setting `DASHBOARD_KEYCLOAK_URL`) (Phase C in `agent-v3/IMPLEMENTATION_PLAN_HIL_BRAIN.md`, detail in `AI_PLATFORM_RESKIN_PLAN.md`)
- Becomes admin-only (Keycloak realm role)
- Bearer-token table retired; existing `role` column may persist for historical audit-log mapping
- Brain admin UI (atom browse, force-gold, brain stats) added as a new section in this dashboard during Phase C
@ -214,7 +250,7 @@ cp ../.env.example .env # set DASHBOARD_DB_USER/PASSWORD/NAME + provider keys
## Ce NU face
- ~~No SSO yet~~ Keycloak SSO wired (DONE 2026-05-02). Bearer tokens kept as legacy fallback.
- Keycloak SSO is wired but **disabled in this delivery** (ships in staging mode, auth bypassed; enable via `DASHBOARD_KEYCLOAK_URL`). Bearer tokens kept as legacy fallback.
- No multi-tenant - single shared `users` table, no per-tenant scoping
- No public access - binds to internal `didi-network` network, not exposed via Kong/edge
- ~~No React frontend yet~~ React 19 SPA at `/admin-ai/` (DONE 2026-05-02). Jinja kept side-by-side until parity.

74
ai_platform/modules/dashboard/README.md
View file

@ -1,20 +1,42 @@
# Dashboard
Admin dashboard for the didiAI platform. Tracks search provider usage, costs, request history, and exposes runtime configuration.
Admin dashboard for the didiAI platform. A single FastAPI service that serves a
**React 19 + MUI 7 + Vite single-page app** (the admin UI) plus a JSON API for
AI monitoring, a DB-backed Model & Extractor catalog, runtime config, RBAC and
audit. It also tracks search-provider usage, costs and request history.
## What it does
- **Live quota & billing** — pulls real-time data from SerpAPI, Tavily, Brave, OpenRouter
- **Health monitoring** — SearXNG, web-api, vLLM, llama.cpp servers
- **Request history** — 30-day rolling log of every gather/search/fetch request with drill-down
- **AI monitoring** — live health/status of platform modules (proxy to each
module's health endpoint), provider quotas (SerpAPI, Tavily, Brave, LinkUp,
Exa, OpenRouter), live KPIs and throughput
- **Model & Extractor Catalog** — DB-backed CRUD over registered models and
extractors, exposed under `/api/catalog`
- **Request history** — 30-day rolling log of every gather/search/fetch request
with drill-down
- **Cost tracking** — per-provider spend, projections, cost per tier
- **Future:** runtime config (toggle providers, change strategies, manage tier caps)
- **Runtime config** — config override store (`/api/config`) with schema
validation and audit trail
- **RBAC** — role-based access; in the shipped build the dashboard runs in
**staging mode** with auth bypassed (see below)
- **Brain admin** — fact-status browse/override and cache invalidation via a
brain proxy
## Authentication / staging mode
The build ships in **staging mode** (`DASHBOARD_STAGING_MODE=true`), so all auth
is bypassed and Keycloak is **not** active. Keycloak (JWT) is supported but
disabled in this delivery: JWT validation only turns on when
`DASHBOARD_KEYCLOAK_URL` is set (empty = off). To go to authenticated mode, set
`DASHBOARD_KEYCLOAK_URL` (e.g. an SSO URL), set `DASHBOARD_STAGING_MODE=false`,
and rebuild the image (Keycloak config is baked into the JS bundle at build
time). DB-backed bearer tokens remain as a legacy fallback.
## Prerequisites
- Docker 24+ with Compose V2
- PostgreSQL 16 (provided by compose)
- Internal network access to web-api, SearXNG, LLM servers
- PostgreSQL 16 (provided by compose, `didiAI-dashboard-db` on `:15432`)
- Internal network access to the platform modules and provider APIs
## Quick start
@ -24,37 +46,49 @@ cp ../.env.example .env # edit with your secrets
./deploy.sh up
```
Dashboard is now running at http://localhost:51300
The dashboard API listens on `http://localhost:51300`. The admin SPA is served
at `http://localhost:51300/admin-ai/`.
## Endpoints
### Web UI
- `/` — overview with KPIs and provider grid
- `/providers` — detailed provider cards + raw table
- `/history` — filterable request history
- `/history/{request_id}` — full request detail with stages
### Admin UI (SPA)
### JSON API
- `GET /health` — liveness
- `/admin-ai/` — React 19 + MUI SPA (overview, live status, history, cost,
providers, catalog, config/schema, audit, brain admin). All client-side routes
under `/admin-ai/` are served by the SPA with index fallback.
> Legacy Jinja pages (`/`, `/providers`, `/history`, …) still exist server-side
> but are not the primary UI; the SPA at `/admin-ai/` is the delivered UI.
### JSON API (selected)
- `GET /health` — liveness + DB ping (health JSON at root)
- `GET /api/stats/providers` — live provider stats
- `GET /api/stats/summary?hours=24` — aggregated counters
- `GET /api/stats/timeline?hours=24` — hourly buckets for charts
- `GET /api/stats/cost` — cost + projection
- `GET /api/history?limit=50&tier=premium` — filtered history
- `GET /api/history/{request_id}` — single request with full payload
- `GET|POST|PUT|DELETE /api/catalog/...` — Model & Extractor catalog CRUD
- `GET /api/config`, `PUT|DELETE /api/config/{key}` — runtime config overrides
- `GET /api/proxy/{module_id}/health` — module health proxy (AI monitoring)
- `/api/brain/*` — brain proxy (fact status, cache invalidation)
- `POST /api/ingest/event` — receives events from web-api middleware
## Architecture
```
web-api ────► POST /api/ingest/event ────► dashboard-api ────► PostgreSQL
│
browser ────► GET /admin-ai/ (React SPA) ──┘ │
├─ reads provider APIs live
└─ serves UI via Jinja2+HTMX
├─ proxies module health + brain
└─ serves SPA static (index fallback)
```
## Tech stack
- FastAPI + Pydantic
- SQLAlchemy 2.0 async + asyncpg
- Jinja2 + HTMX + Alpine.js + Tailwind (zero build step)
- FastAPI + Pydantic (pydantic-settings, env prefix `DASHBOARD_`)
- SQLAlchemy 2.0 async + asyncpg (tables created via `create_all` at startup)
- React 19 + MUI 7 + Vite 7 + react-router 7 + TanStack Query + Recharts
(built into `web_dist/`, served as static SPA)
- PostgreSQL 16

12
ai_platform/modules/dashboard/web/src/pages/system/Settings.tsx
View file

@ -76,9 +76,15 @@ export default function Settings() {
});
const health = useQuery({
queryKey: ['settings', 'health'],
// Dashboard liveness probe lives at root /health (next to /ready), not
// /api/health — the latter returns 404 and shows "unknown" in the UI.
queryFn: () => apiGet<HealthResponse>('/health'),
// Dashboard liveness probe lives at the ORIGIN root /health (next to
// /ready). apiGet() would prepend BASE_PATH (/admin-ai/health), which the
// SPA static fallback answers with index.html → JSON parse yields no
// `status` → "unknown". Fetch the absolute root path directly instead.
queryFn: async (): Promise<HealthResponse> => {
const res = await fetch('/health', { headers: { Accept: 'application/json' } });
if (!res.ok) throw new Error(`health ${res.status}`);
return res.json();
},
refetchInterval: 30_000,
});

64
ai_platform/modules/didi_brain/ARCHITECTURE.md
View file

@ -17,7 +17,7 @@ it, what does it depend on" for any file in the repo.
| Module / File | What it does | When it runs | Who calls it |
|---|---|---|---|
| **`infra/docker-compose.yml`** | Defines the 3-container stack | `docker compose up` | operator |
| **`infra/docker-compose.yml`** | Defines the 4-container stack (api, atomic, postgres, scheduler) | `docker compose up` | operator |
| **`brain_api/`** *(container)* | HTTP service speaking Didi's contract | every Didi request | Didi backend (HTTP) |
| **`shared/`** *(library)* | Common code: config, clients, taxonomy | imported everywhere | brain_api, extractor, lint, scripts |
| **`extractor/`** *(library + jobs)* | Turns Document atoms into Claim atoms | operator (script 07) OR background after `/v1/ingest` | scripts/07, brain_api ingest |
@ -26,12 +26,12 @@ it, what does it depend on" for any file in the repo.
| **Atomic server** *(container)* | Storage + chunking + embedding pipeline + REST | always | brain_api (HTTP), scripts (HTTP) |
| **Postgres + pgvector** *(container)* | Persistent atom + vector storage | always | atomic-server (SQL) |
**External services** (not in our repo, on VPN):
**External services** (not in our repo, reached over Docker DNS on `didi-network`):
| Where | What | Used by |
|---|---|---|
| `10.11.10.17:14011` | LLM router → Qwen 397B | brain_api NLI, extractor, lint |
| `10.11.10.15:8200` | BGE-M3 embedding server | atomic-server (for embeddings) |
| `10.11.10.15:8100` | BGE-reranker-v2-m3 | brain_api gather (for precision rerank) |
| `didiAI-llm-api` → `10.11.10.17:14011` | LLM router → qwen3.5 | brain_api NLI, extractor, lint |
| `didiAI-embeddings-api` → `10.11.10.15:14100` | BGE-M3 embedding server | atomic-server (for embeddings) |
| `didiAI-rerank-api` → `10.11.10.15:14200` | BGE-reranker-v2-m3 | brain_api gather (for precision rerank) |
---
@ -56,12 +56,12 @@ it, what does it depend on" for any file in the repo.
│ atomic_api, taxonomy, logging} │
└─────┬─────────────┬──────────────┬──────────────────────────┘
│ │ │
│ HTTP │ HTTPS │ HTTPS
│ HTTP │ HTTP │ HTTP
▼ ▼ ▼
┌───────────┐ ┌──────────┐ ┌────────────────┐
│ atomic- │ │ BGE-M3 │ │ Qwen 397B │
│ atomic- │ │ BGE-M3 │ │ qwen3.5 │
│ server │ │ + rerank │ │ via LLM router │
│ (cont.) │ │ (VPN) │ │ (VPN) │
│ (cont.) │ │ (DNS) │ │ (DNS) │
└─────┬─────┘ └──────────┘ └────────────────┘
│
▼
@ -121,8 +121,9 @@ its own.
## `brain_api/` — the HTTP service
**Purpose.** Containerized FastAPI service that exposes the 5-endpoint
contract Didi's backend already speaks. It does NOT contain business logic
**Purpose.** Containerized FastAPI service that exposes the full route set
(~30 routes: the web-gathering contract + brain-owned cache/freshness layers)
that Didi's backend already speaks. It does NOT contain business logic
that another module reuses; it ties together `shared` + `extractor` (via
`/v1/ingest`) and translates between Didi's HTTP shape and the brain's
internal capabilities.
@ -139,7 +140,7 @@ FastAPI/Uvicorn. The lifespan hook initializes the long-lived clients
| `Dockerfile` | python:3.12-slim, non-root user `brain`, COPY shared/extractor/brain_api, curl healthcheck on /health, CMD `python -m brain_api.run`. |
| `requirements.txt` | Pinned runtime deps. |
| `__init__.py` | Module marker + docstring. |
| `app.py` | The FastAPI app. Defines `lifespan` (startup/shutdown), the `/health` route, and the 5 v1 routes. Each route delegates to a function in `services/`. |
| `app.py` | The FastAPI app. Defines `lifespan` (startup/shutdown), the `/health` route, and the full set of v1 routes (gather/search/fetch/image-search/ingest + verification_cache + analysis_atom* + canonicalize + cache/invalidate + cache/audit_log + fact_status*). Each route delegates to a function in `services/`. |
| `run.py` | Uvicorn entry. Reads `BRAIN_API_HOST` and `BRAIN_API_PORT` from env (defaults 127.0.0.1:8090; container overrides to 0.0.0.0). |
| `deps.py` | `AppState` dataclass + module-level singleton accessor. Set by lifespan, read by route handlers. |
| `schemas.py` | **The contract.** Pydantic v2 models for every request/response shape the service speaks: `SearchRequest/Response`, `FetchRequest/Response`, `GatherRequest/Response`, `ImageSearchRequest/Response`, `IngestRequest/Response`, plus `EvidenceItem`, `BrainEvidenceMeta`, `BrainMeta`, etc. |
@ -159,7 +160,7 @@ FastAPI/Uvicorn. The lifespan hook initializes the long-lived clients
## `extractor/` — Document → Claim transformation
**Purpose.** Reads `Type/Document` atoms from Atomic, asks Qwen 397B to
**Purpose.** Reads `Type/Document` atoms from Atomic, asks qwen3.5 to
extract atomic factual claims, validates each claim against the source
text (substring check on the quote), and creates new `Type/Claim` atoms
with canonical hash-based source URLs and inherited tags.
@ -186,7 +187,7 @@ Both paths share the same idempotency state (`extractor/_extracted.json`).
| `extract.py` | Single-document logic. `extract_claims_from_atom(llm, title, language, content)` → `ExtractionResult` with `valid: list[ExtractedClaim]` and `rejected: dict[reason → count]`. Includes the substring quote validation. |
| `push.py` | `push_claim(atomic, parent_atom, claim, parent_title, resolver)` creates one `Type/Claim` atom. URL is `{parent_url}#claim={hash8}` for natural dedup. Inherits parent's Country/Topic/SourceType/Credibility/Language tags, adds Type/Claim + Stance/{X}. |
| `batch.py` | The orchestrator. `run_batch(limit, only_atom_ids)` paginates Document atoms, fetches each fully (list_atoms returns summary only — gotcha!), runs extraction sequentially, pushes claims, saves state after every doc so Ctrl-C is recoverable. |
| `prompts/claim_extraction_v1.md` | Versioned prompt that instructs Qwen 397B to return strict JSON with claims + verbatim quotes. Output format pinned. |
| `prompts/claim_extraction_v1.md` | Versioned prompt that instructs qwen3.5 to return strict JSON with claims + verbatim quotes. Output format pinned. |
**Imports.** `shared`, `httpx` (transitively).
@ -198,7 +199,7 @@ Both paths share the same idempotency state (`extractor/_extracted.json`).
**Purpose.** Audit job that finds claim atoms in our corpus that
contradict each other (or are paraphrases). For every claim, asks Atomic
for its semantic neighbors, then asks Qwen 397B to classify each
for its semantic neighbors, then asks qwen3.5 to classify each
candidate pair as **EQUIVALENT**, **CONTRADICTORY**, or **INCOMPARABLE**.
Stores all verdicts in a JSON ledger so re-runs only process new pairs.
@ -264,7 +265,7 @@ clean rich-formatted report, and exits with a meaningful code.
| File | Role |
|---|---|
| `docker-compose.yml` | Three services: `postgres` (pgvector/pgvector:pg16), `atomic-server` (ghcr.io/kenforthewin/atomic-server:latest with overridden entrypoint to use `--data-dir` instead of legacy `--db-path`, and `ATOMIC_STORAGE=postgres` env), `brain-api` (built from `brain_api/Dockerfile`). Two named volumes (`didibrain-pg-data`, `didibrain-atomic-data`), one bridge network (`didibrain`). All services have `restart: unless-stopped` and healthchecks. |
| `docker-compose.yml` | Four services: `postgres` (pgvector/pgvector:pg16), `atomic-server` (ghcr.io/kenforthewin/atomic-server:latest with overridden entrypoint to use `--data-dir` instead of legacy `--db-path`, and `ATOMIC_STORAGE=postgres` env), `brain-api` (built from `brain_api/Dockerfile`), and `scheduler` (built from `scheduler/Dockerfile` — feeder + auditor + watcher + heartbeat). Two named volumes (`didibrain-pg-data`, `didibrain-atomic-data`), and the external shared network `didi-network`. All services have `restart: unless-stopped` and healthchecks. |
**When it runs.** `docker compose up` from the operator (or via `bootstrap_deploy.sh`).
@ -301,7 +302,7 @@ brain_api/app.py: post_gather()
│
│ stage 4 nli → mapping.parse_claim_atom_body() per top result
│ → nli.classify_batch(llm, claim, evidence_texts)
│ (parallel calls to Qwen 397B, ~3-4 sec; one per parent doc)
│ (parallel calls to qwen3.5, ~3-4 sec; one per parent doc)
│
│ stage 5 evidence → atomic.get_atom_by_source_url() per parent doc
│ → mapping.evidence_from_parent() builds EvidenceItem
@ -355,7 +356,7 @@ cp .env.example .env, vim .env (none)
│ │
│ ├─ list Type/Document atoms
│ ├─ for each not in extractor/_extracted.json:
│ │ fetch full atom → extract.extract_claims_from_atom() (Qwen 397B)
│ │ fetch full atom → extract.extract_claims_from_atom() (qwen3.5)
│ │ push.push_claim() per valid claim → atomic.create_atom()
│ └─ save state every doc
│
@ -394,7 +395,7 @@ brain_api/app.py: post_ingest()
FastAPI BackgroundTasks: _run_extraction_background(created_ids)
│
└─▶ extractor.batch.run_batch(only_atom_ids=set(created_ids))
├─ runs Qwen 397B claim extraction
├─ runs qwen3.5 claim extraction
├─ pushes Type/Claim atoms
└─ saves extractor/_extracted.json
```
@ -416,9 +417,14 @@ from Atomic without loss.
| `extractor/_extracted.json` | `scripts/07_run_extraction.py` (and brain_api ingest) | Per-document extraction log: which docs have been processed, with which prompt version, how many claims came out. Skip already-done docs on rerun. | yes — delete and re-run; will re-extract everything |
| `lint/_contradictions.json` | `scripts/10_run_lint.py` | Per-pair verdict ledger with the EQUIVALENT / CONTRADICTORY / INCOMPARABLE labels. Skip already-done pairs on rerun. | yes — delete and re-run; will re-classify everything |
The brain_api **container** does NOT use any of these files. It calls
`atomic.list_tags()` at startup to refresh `TagResolver` in-memory, so
the image stays portable.
The brain_api **container** refreshes `TagResolver` in-memory at startup by
calling `atomic.list_tags()`, so it does not bake any of these files into the
image. It does, however, **mount** `shared/_tag_ids.json` read-only via the
compose `volumes:` entry — background extraction (triggered by `/v1/ingest`)
instantiates `TagResolver` directly, bypassing the lifespan refresh, so it
relies on that mounted file to resolve canonical tag UUIDs. The other two
ledgers (`extractor/_extracted.json`, `lint/_contradictions.json`) are
host-only and not used by the container.
---
@ -435,20 +441,20 @@ project root. The brain_api container also receives these via
| `LLM_ROUTER_API_KEY` | empty | optional bearer | no |
| `LLM_VLLM_URL` | `http://localhost:14001` | direct vLLM URL (fallback only, currently unused) | no |
| `LLM_LLAMACPP_URLS` | empty | comma-separated direct llamacpp URLs (fallback) | no |
| `MODEL_REASONING` | `Qwen3.5-397B-A17B` | model id for extraction, NLI, gather | yes |
| `MODEL_REASONING_BACKEND` | `llamacpp` | router backend hint | yes |
| `MODEL_FAST` | `qwen3.5` | fast model id (currently disabled) | no |
| `MODEL_REASONING` | `qwen3.5` | model id for extraction, NLI, gather (live working model) | yes |
| `MODEL_REASONING_BACKEND` | `vllm` | router backend hint | yes |
| `MODEL_FAST` | `qwen3.5` | fast model id (enabled — the live working model) | no |
| `MODEL_FAST_BACKEND` | `vllm` | fast backend hint | no |
| `MODEL_FAST_ENABLED` | `false` | enable fast model in routing | no |
| `MODEL_VISION` | `gemma-3-27b-it` | vision model id (currently down) | no |
| `MODEL_FAST_ENABLED` | `true` | fast model enabled in routing (live: `true`, qwen3.5 active) | no |
| `MODEL_VISION` | empty | vision model id (vision disabled; no vision model deployed) | no |
| `MODEL_VISION_URL` | empty | direct vision endpoint | no |
| `MODEL_VISION_ENABLED` | `false` | enable vision in routing | no |
| `EMBEDDING_URL` | `http://10.11.10.15:8200` | BGE-M3 vLLM endpoint | yes |
| `EMBEDDING_URL` | `http://10.11.10.15:14100` | BGE-M3 vLLM endpoint (`didiAI-embeddings-api`) | yes |
| `EMBEDDING_API_KEY` | empty | optional bearer | no |
| `EMBEDDING_MODEL` | `BAAI/bge-m3` | model id sent in requests | yes |
| `EMBEDDING_DIM` | `1024` | vector dimension (must match Atomic's setting) | yes |
| `EMBEDDING_MAX_TOKENS` | `8192` | input length cap | yes |
| `RERANKER_URL` | `http://10.11.10.15:8100` | BGE-reranker-v2-m3 endpoint | yes |
| `RERANKER_URL` | `http://10.11.10.15:14200` | BGE-reranker-v2-m3 endpoint (`didiAI-rerank-api`) | yes |
| `RERANKER_API_KEY` | empty | optional bearer | no |
| `RERANKER_MODEL` | `BAAI/bge-reranker-v2-m3` | model id sent in requests | yes |
| `ATOMIC_URL` | `http://localhost:8088` | atomic-server URL (host scripts use this; brain_api container overrides to docker DNS) | yes |
@ -504,7 +510,7 @@ Something with /v1/gather failing?
Atomic returning 500?
│
├─ docker logs didibrain-atomic — likely BGE unreachable
└─ curl http://10.11.10.15:8200/v1/models on host — VPN check
└─ curl http://10.11.10.15:14100/v1/models on host — upstream reachability check
Claim extraction acting weird?
│

8
ai_platform/modules/didi_brain/CONTRACT_VERIFICATION_CACHE.md
View file

@ -64,7 +64,7 @@ rulare). Cu evidence_hash în cheie, cache-ul era practic nereachable.
"https://somes-tisa.rowater.ro/pdf2"
],
"tier": "free",
"model": "qwen35:Qwen3.5-397B-A17B",
"model": "qwen35:qwen3.5",
"prompt_hash": "a3f2b1c9d4e7",
"framework_version": "f9e1d2c3b4a5",
"schema_name": "didi-v1",
@ -118,7 +118,7 @@ rulare). Cu evidence_hash în cheie, cache-ul era practic nereachable.
- `503 Service Unavailable` — PG indisponibil, retry mai târziu
- `500 Internal Server Error` — orice alt eșec
**Idempotență:** UPSERT pe cheia `(claim_hash, evidence_hash, tier)`. Same-tuple,
**Idempotență:** UPSERT pe cheia `(claim_hash, tier)`. Same-tuple,
second write → update `verification_processed`/`verification_raw`/`updated_at`/`expires_at`.
Last-writer-wins.
@ -155,7 +155,7 @@ Last-writer-wins.
"verification_staleness": "fresh",
"verification": { ... payload stocat ... },
"verification_model": "qwen35:Qwen3.5-397B-A17B",
"verification_model": "qwen35:qwen3.5",
"verification_tier": "free",
"verification_prompt_hash": "a3f2b1c9d4e7",
"verification_framework_version": "f9e1d2c3b4a5",
@ -263,7 +263,7 @@ Cache pe `free` și `premium` sunt **complet separate**. Un user `free` nu vede
ce a scris un user `premium` (și invers). Este intenționat: modele diferite =
nuanțe diferite la stance.
Cheia unică în DB: `(claim_hash, evidence_hash, tier)`.
Cheia unică în DB: `(claim_hash, tier)`.
---

15
ai_platform/modules/didi_brain/INDEX.md
View file

@ -2,7 +2,7 @@
Knowledge atom storage + claim verification cache + analysis atom cache pentru platforma DIDI. FastAPI service care vorbeste contractul `web-gathering` 1:1 (drop-in replacement / cache layer pentru `/v1/gather`) si in plus expune doua cache-uri proprii: `brain_verification_cache` (claims, din 2026-04-23) si `brain_analysis_atom` (techniques + ai_tampered + claims, NEW v2 din 2026-05-01).
**Stack**: Python 3.11+, FastAPI, Pydantic v2, asyncpg, BGE-M3 1024-dim embeddings (vLLM), BGE-reranker-v2-m3 (vLLM cross-encoder), Qwen3.5-397B-A17B (llama.cpp via OpenAI-compat router)
**Stack**: Python 3.11+, FastAPI, Pydantic v2, asyncpg, BGE-M3 1024-dim embeddings (vLLM), BGE-reranker-v2-m3 (vLLM cross-encoder), qwen3.5 (vLLM via OpenAI-compat router)
**Storage**: Atomic (Rust knowledge graph) on Postgres 16 + pgvector. Brain owns its own tables prefixed `brain_*` alongside Atomic's tables.
**URL**: `http://10.11.10.12:8090` (production-local, since brain cutover 2026-05-01 — anterior pe `10.11.10.13:8090`)
**Container**: `didibrain-api` (alongside `didibrain-atomic` and `didibrain-postgres`)
@ -163,9 +163,10 @@ UNIQUE: `(content_hash, component, prompt_hash)` — `tier` OUT of unique key (w
## Configuration
- Settings via `shared/config.py` cu `pydantic-settings`
- Env vars: `postgres_dsn`, `atomic_url`, `llm_router_url`, `embedding_url`, `reranker_url`, `verification_cache_ttl_days` (default 30), `verification_cache_max_payload_kb` (default 64)
- Internal Atomic URL: `http://didibrain-atomic:8080` (Docker DNS)
- Upstream LLM/embed/rerank pe VPN 10.11.10.x (10.11.10.17:14011 router, 10.11.10.15:8200 BGE-M3, 10.11.10.15:8100 reranker)
- Env vars: `postgres_dsn`, `atomic_url`, `atomic_token`, `llm_router_url`, `embedding_url`, `reranker_url`, `verification_cache_ttl_days` (default 30), `verification_cache_max_payload_kb` (default 64)
- `ATOMIC_TOKEN`: required for atomic-server API calls, auto-generated/auto-populated by `scripts/02_bootstrap_atomic.py` (written back to `.env`)
- Internal Atomic URL: `http://atomic-server:8080` (Docker DNS)
- Upstream LLM/embed/rerank reached over Docker DNS on `didi-network` (`didiAI-llm-api` → 10.11.10.17:14011 router, `didiAI-embeddings-api` → 10.11.10.15:14100 BGE-M3, `didiAI-rerank-api` → 10.11.10.15:14200 reranker)
- **No auth in v1** (binds to internal Docker network)
---
@ -173,10 +174,10 @@ UNIQUE: `(content_hash, component, prompt_hash)` — `tier` OUT of unique key (w
## Deployment
- Docker compose la `infra/docker-compose.yml`
- 3 containere: `didibrain-api` (FastAPI), `didibrain-atomic` (Rust KG), `didibrain-postgres` (PG 16 + pgvector)
- 4 containere: `didibrain-api` (FastAPI), `didibrain-atomic` (Rust KG), `didibrain-postgres` (PG 16 + pgvector), `didibrain-scheduler` (feeder + auditor + watcher)
- Schema bootstrap on startup (idempotent — `_SCHEMA_SQL` ruleaza la fiecare connect, all DDL is `IF NOT EXISTS` + ALTER guards)
- TTL cleanup: cron job in `scripts/` sterge expired silver/bronze atoms
- Resource footprint: ~200 MB RAM total cross 3 containere; brain-api idle ~6% CPU, spike ~20% during `/v1/gather` cu NLI
- Resource footprint: ~200 MB RAM total cross 4 containere; brain-api idle ~6% CPU, spike ~20% during `/v1/gather` cu NLI
**Rebuild**:
```bash
@ -194,7 +195,7 @@ docker compose -f infra/docker-compose.yml --env-file .env up -d --build
- `scripts/04_seed_taxonomy.py` — seed canonical tag taxonomy (79 tags, 7 namespaces)
- `scripts/05_import_wikipedia_seed.py` — seed-list Wikipedia import
- `scripts/06_validate_queries.py` — smoke test doc-level retrieval
- `scripts/07_run_extraction.py` — claim extraction batch (Qwen 397B)
- `scripts/07_run_extraction.py` — claim extraction batch (qwen3.5)
- `scripts/08_validate_claims.py` — smoke test claim-level retrieval
- `scripts/09_brain_api_demo.py` — brain_api contract test (all endpoints)
- `scripts/10_run_lint.py` — Lint pass runner (contradiction detection)

55
ai_platform/modules/didi_brain/README.md
View file

@ -23,7 +23,7 @@ HTTP contract but serves responses from a locally-grown **knowledge graph**:
- **Atoms** (documents and atomic claims) are stored in a Postgres +
pgvector backend with multilingual embeddings (BGE-M3, 1024-dim).
- **Claim extraction** runs Qwen 3.5 397B over every ingested document to
- **Claim extraction** runs Qwen3.5 over every ingested document to
pull out verifiable atomic claims with source quotes and stance.
- **Retrieval** uses vector kNN + BGE-reranker-v2-m3 cross-encoder for
precision.
@ -46,9 +46,9 @@ background. Next time a similar claim arrives, it's a HIT.
| **Brain storage** | [Atomic](https://github.com/kenforthewin/atomic) (Rust) on Postgres 16 + pgvector |
| **Embeddings** | `BAAI/bge-m3` via vLLM OpenAI-compat (1024 dim, 8K ctx, multilingual) |
| **Reranker** | `BAAI/bge-reranker-v2-m3` cross-encoder via vllm-rerank-api |
| **LLM (reasoning)** | `Qwen3.5-397B-A17B` (MoE) via llama.cpp through an OpenAI-compat router |
| **LLM (reasoning)** | `qwen3.5` via vLLM through an OpenAI-compat router (`MODEL_FAST` enabled) |
| **Service** | Python 3.12, FastAPI, Uvicorn, Pydantic v2, httpx, structlog, tenacity |
| **Deployment** | Docker + docker-compose (3 services: api, atomic, postgres) |
| **Deployment** | Docker + docker-compose (4 services: api, atomic, postgres, scheduler) |
## Quick start — local dev
@ -88,7 +88,7 @@ curl -fsS http://localhost:8090/health
```
Open the interactive Swagger UI at **http://localhost:8090/docs** to poke
the 5 endpoints live.
the full route set live (~30 routes).
## Production deploy — fresh Linux server
@ -127,6 +127,21 @@ backends safely ignore.
| `POST /v1/gather` | claim → ranked evidence with NLI stance | 5-6 s (1.5 s without NLI) |
| `POST /v1/image-search` | stub (always empty list) | <5 ms |
| `POST /v1/ingest` | populate brain from web-module output | variable (async extraction) |
| `POST /v1/verification_cache` | write claim verification result (claims cache) | <50 ms |
| `POST /v1/analysis_atom/lookup` | read cached analysis result (techniques / ai_tampered / claims) | <50 ms |
| `POST /v1/analysis_atom` | write analysis atom (silver/bronze by confidence) | <50 ms |
| `PATCH /v1/analysis_atom/{id}` | promote atom to gold (moderator review) | <50 ms |
| `GET /v1/analysis_atom/stats` | per-tier/component counts + 24h hit rate | <20 ms |
| `POST /v1/canonicalize` | temporal claim disambiguation (Pilon 7) | LLM-bound |
| `POST /v1/cache/invalidate` | mass invalidation with `dry_run` (Pilon 8) | variable |
| `GET /v1/cache/audit_log` | paginated audit browser | <50 ms |
| `GET/PATCH /v1/fact_status/*` | versioned fact-status layer (list/detail/versions/override) | <50 ms |
This is the full set served by `brain_api` (~30 routes including the FastAPI
auto docs); the `/v1/gather`, `/v1/search`, `/v1/fetch`, `/v1/ingest`,
`/v1/image-search` group is the web-gathering contract, the rest are the
brain-owned cache + freshness-defense layers. See `INDEX.md` for the
canonical route list.
The response from `/v1/gather` matches the existing web-module shape
exactly plus an additive `brain_meta` object on the top level and inside
@ -156,11 +171,11 @@ Didi backend
│ stage context → language detection │
│ stage retrieval → Atomic semantic search (top 50) │
│ stage rerank → BGE cross-encoder (top 15) │
│ stage nli → Qwen 397B stance vs query │
│ stage nli → Qwen3.5 stance vs query │
│ stage evidence → group by parent doc, shape │
└──────┬──────────────────────────┬──────────────────┘
│ │
│ HTTP (docker DNS) │ HTTPS (VPN)
│ HTTP (docker DNS) │ HTTP (docker DNS)
▼ ▼
┌────────────────┐ ┌───────────────────────┐
│ atomic-server │ │ BGE-M3 embeddings │
@ -169,9 +184,9 @@ Didi backend
│ └───────────────────────┘
▼
┌────────────────┐ ┌───────────────────────┐
│ postgres │ │ Qwen 3.5 397B-A17B │
│ postgres │ │ qwen3.5 │
│ + pgvector │ │ via LLM router │
│ :5432 (5434) │ │ (llama.cpp + vLLM) │
│ :5432 (5434) │ │ (vLLM) │
└────────────────┘ └───────────────────────┘
```
@ -179,7 +194,7 @@ Didi backend
```
didibrain/
├── infra/docker-compose.yml # 3-service stack
├── infra/docker-compose.yml # 4-service stack (api, atomic, postgres, scheduler)
├── brain_api/ # FastAPI service (the main deliverable)
├── shared/ # config, clients, taxonomy (reused everywhere)
├── extractor/ # claim extraction (host jobs + /v1/ingest)
@ -201,7 +216,7 @@ Detailed file-by-file rundown is in `STATUS.md`.
- [x] Claim extraction (513 atoms, 1.2% hallucination filter)
- [x] Document-level retrieval validated (cross-lingual cosine 0.88-0.94)
- [x] Claim-level retrieval validated
- [x] brain_api HTTP service with Didi contract (5 endpoints)
- [x] brain_api HTTP service with Didi contract (full route set, ~30 routes)
- [x] brain_api dockerized (self-sufficient, taxonomy auto-refresh)
- [x] NLI stance vs query in `/v1/gather`
- [x] Lint pass contradiction detection (code + smoke test)
@ -213,16 +228,20 @@ Detailed file-by-file rundown is in `STATUS.md`.
## Operational notes
- **Resource footprint**: ~200 MB RAM total across the 3 containers; brain-api
- **Resource footprint**: ~200 MB RAM total across the 4 containers; brain-api
idles at ~6% CPU, spikes to ~20% during a `/v1/gather` with NLI.
- **Image size**: brain-api Docker image is ~253 MB (Python 3.12-slim base).
- **VPN dependency**: BGE and the LLM router live on a VPN-routed 10.11.10.x
network. If the VPN drops, `/v1/gather` returns 500 because Atomic cannot
embed the query. Confirm upstream reachability before debugging anything
else when search starts failing.
- **Self-sufficient startup**: brain_api pulls the current taxonomy from
Atomic at startup, so there is no baked `_tag_ids.json` in the image and
the container is portable across environments.
- **Upstream connectivity**: the LLM router, BGE-M3 embeddings, and the BGE
reranker are reached over Docker DNS as `didiAI-llm-api:14011`,
`didiAI-embeddings-api:14100`, and `didiAI-rerank-api:14200` on the shared
`didi-network`. If those upstreams are unreachable, `/v1/gather` returns 500
because Atomic cannot embed the query. Confirm upstream reachability before
debugging anything else when search starts failing.
- **Startup**: brain_api refreshes the `TagResolver` from Atomic at startup;
in addition `shared/_tag_ids.json` is mounted into the container (compose
volume) so background extraction — which instantiates `TagResolver`
directly, bypassing the lifespan refresh — can still resolve canonical tag
UUIDs.
- **Idempotency**: every operator script (taxonomy seeder, Wikipedia
importer, claim extractor, Lint pass) is idempotent via state files or
URL-based dedup. Re-running is always safe.

90
ai_platform/modules/didi_brain/STATUS.md
View file

@ -1,6 +1,13 @@
# DidiBrain — Status Snapshot
**Last updated:** 2026-04-23 (integration session — web-api cache + backend verification cache)
> **Note (current):** the platform is **LIVE and functional**. Since this
> snapshot the stack grew to **four containers** (added `didibrain-scheduler`),
> the analysis-atom cache (v2, gold/silver/bronze) and the Cache Freshness
> Defense layer (feeder + auditor + watcher) shipped, and the live working
> model is **`qwen3.5`** (vLLM, `MODEL_FAST` enabled). Sections below that
> still say "3 containers / 5 endpoints / 397B" are corrected inline.
**Working dir:** `/home/admin365/ml-projects/modules/didi_brain/`
**Related docs:**
- `CONTRACT_VERIFICATION_CACHE.md` — contract backend↔brain pentru verification cache (v2, current)
@ -17,24 +24,31 @@ DidiBrain is **SHIP READY** și **INTEGRATED**. Pe lângă contractul inițial H
- **Verification cache pentru didi-backend** — post-LLM stance/verdict storage
cu 4 staleness states (fresh / stale_framework / stale_prompt / miss)
**Trei containere, zero regressions pe funcționalitatea v1.** Tabelul relational
nou `brain_verification_cache` trăiește în același Postgres cu atomic-server,
prefix `brain_` pentru izolare.
- **Analysis atom cache (v2)** — techniques + ai_tampered + claims full-component
results, 3-tier gold/silver/bronze (since 2026-05-01)
- **Cache Freshness Defense** — `didibrain-scheduler` (feeder + auditor + watcher)
keeps cached verdicts fresh
**Patru containere, zero regressions pe funcționalitatea v1.** Tabelele relationale
noi `brain_verification_cache` + `brain_analysis_atom` (și layerul fact_status)
trăiesc în același Postgres cu atomic-server, prefix `brain_` pentru izolare.
## Current state in 30 seconds
```
3 containers running continuously, all healthy
4 containers running continuously, all healthy
didibrain-api brain_api FastAPI service :8090 52 MB RAM
didibrain-atomic atomic-server (API only) :8088 83 MB RAM
didibrain-postgres pgvector pg16 :5434 60 MB RAM
(+ brain_verification_cache table)
didibrain-api brain_api FastAPI service :8090 52 MB RAM
didibrain-atomic atomic-server (API only) :8088 (8080 internal) 83 MB RAM
didibrain-postgres pgvector pg16 :5434 60 MB RAM
(+ brain_verification_cache
+ brain_analysis_atom tables)
didibrain-scheduler feeder + auditor + watcher (no port)
Brain contents (grow continuously via web-api ingest + manual bootstrap)
79 tags (canonical taxonomy, 7 root namespaces)
19+ documents (initial Wikipedia vaccines seed + anything premium adds)
509+ claims (extracted by Qwen 397B, substring validated)
509+ claims (extracted by qwen3.5, substring validated)
Verification cache (new in v2 — Apr 23)
brain_verification_cache UNIQUE (claim_hash, tier)
@ -82,7 +96,7 @@ Didi backend ┌─────────────
│ DidiBrain stack │
│ │
│ ┌──────────────────────┐ │
│ │ brain-api :8090 │ FastAPI + Uvicorn, 5 v1 endpoints │
│ │ brain-api :8090 │ FastAPI + Uvicorn, full route set │
│ │ (didibrain-api) │ speaks Didi contract 1:1 │
│ └──────────┬───────────┘ │
│ │ HTTP (docker DNS) │
@ -97,25 +111,26 @@ Didi backend ┌─────────────
│ │ HTTPS
│ ▼
│ ┌────────────────────────────────┐
│ │ BGE-M3 embeddings │ 10.11.10.15:8200
│ │ BGE-M3 embeddings │ 10.11.10.15:14100
│ │ (vLLM OpenAI-compat) │ 1024 dim, 8K ctx, multilingual
│ └────────────────────────────────┘
│
│ ┌────────────────────────────────┐
│ │ BGE-reranker-v2-m3 │ 10.11.10.15:8100
│ │ BGE-reranker-v2-m3 │ 10.11.10.15:14200
│ │ (cross-encoder) │ precision boost
│ └────────────────────────────────┘
│
│ ┌────────────────────────────────┐
└─▶│ Qwen3.5-397B-A17B │ 10.11.10.17:14011 (router)
└─▶│ qwen3.5 │ 10.11.10.17:14011 (router)
│ via LLM router │ round-robin to .18 and .19
│ (llama.cpp + vLLM backends) │ claim extraction, NLI
│ (vLLM backends) │ claim extraction, NLI
└────────────────────────────────┘
```
All upstream endpoints live on a VPN-routed 10.11.10.x network. The Docker
containers reach them through WSL2 NAT (Docker Desktop) or host networking
(Linux server).
All upstream endpoints (LLM router, BGE-M3, reranker) are reached over Docker
DNS on the shared external `didi-network` as `didiAI-llm-api:14011`,
`didiAI-embeddings-api:14100`, and `didiAI-rerank-api:14200` (the 10.11.10.x
addresses above are the host-side equivalents).
## Repo layout
@ -131,7 +146,7 @@ didibrain/
├── AUDIT.md # initial upstream Atomic audit
│
├── infra/
│ └── docker-compose.yml # 3-service stack (+ optional atomic-web)
│ └── docker-compose.yml # 4-service stack (api, atomic, postgres, scheduler)
│
├── brain_api/ # HTTP service — the main deliverable
│ ├── Dockerfile
@ -184,7 +199,7 @@ didibrain/
├── 06_validate_queries.py # smoke test doc-level retrieval
├── 07_run_extraction.py # claim extraction batch
├── 08_validate_claims.py # smoke test claim-level retrieval
├── 09_brain_api_demo.py # brain_api contract test (all 5 endpoints)
├── 09_brain_api_demo.py # brain_api contract test (core web-gathering endpoints)
├── 10_run_lint.py # Lint pass runner (--limit --force)
├── 11_show_contradictions.py # read state file, render top contradictions
└── bootstrap_deploy.sh # fresh-server deploy orchestrator
@ -209,6 +224,17 @@ backends.
| `POST /v1/gather` with `run_nli:false` | claim → evidence | ~1.5-2 s | skip stance classification for speed |
| `POST /v1/image-search` | stub | <5 ms | always empty list |
| `POST /v1/ingest` | populate from web module | variable | creates atoms + optional async extraction |
| `POST /v1/verification_cache` | write claim verification result | <50 ms | claims cache (v2) |
| `POST /v1/analysis_atom/lookup` | read cached analysis | <50 ms | techniques / ai_tampered / claims |
| `POST /v1/analysis_atom` | write analysis atom | <50 ms | silver/bronze by `llm_confidence` |
| `PATCH /v1/analysis_atom/{id}` | promote to gold | <50 ms | moderator review |
| `GET /v1/analysis_atom/stats` | per-tier/component counts | <20 ms | 24h hit rate |
| `POST /v1/canonicalize` | temporal claim disambiguation | LLM-bound | Pilon 7 |
| `POST /v1/cache/invalidate` | mass invalidation (`dry_run`) | variable | Pilon 8 |
| `GET /v1/cache/audit_log` | audit browser | <50 ms | paginated |
| `GET/PATCH /v1/fact_status/*` | versioned fact-status layer | <50 ms | list/detail/versions/override |
(Full set ~30 routes incl. FastAPI auto docs; see `INDEX.md` for the canonical list.)
### /v1/gather response shape (key fields)
@ -297,9 +323,9 @@ or in README.md under "Production deploy".
Only the upstream endpoint URLs may need updating:
```bash
LLM_ROUTER_URL=http://10.11.10.17:14011 # if router is on VPN, unchanged
EMBEDDING_URL=http://10.11.10.15:8200 # if BGE is on VPN, unchanged
RERANKER_URL=http://10.11.10.15:8100 # if reranker is on VPN, unchanged
LLM_ROUTER_URL=http://10.11.10.17:14011 # didiAI-llm-api, unchanged
EMBEDDING_URL=http://10.11.10.15:14100 # didiAI-embeddings-api, unchanged
RERANKER_URL=http://10.11.10.15:14200 # didiAI-rerank-api, unchanged
```
Everything else (`ATOMIC_URL`, ports, model names, Postgres creds) is either
@ -344,15 +370,13 @@ curl -s -X POST http://localhost:8090/v1/gather \
call this endpoint; we read `embedding_status` per atom via `list_atoms`
or `get_atom` instead.
3. **Qwen 35B (vLLM on :14001)** — thinking mode stuck ON via the router and
safety alignment refuses disinfo-extraction tasks. Disabled in config
(`MODEL_FAST_ENABLED=false`); the whole pipeline runs on Qwen 397B. If a
future session un-sticks 35B, Lint pass could speed up ~4x by using it.
3. **Fast model `qwen3.5`** — now **enabled** (`MODEL_FAST_ENABLED=true`) and
is the live working model for the whole pipeline (extraction, NLI, gather)
via the LLM router at `didiAI-llm-api:14011` (vLLM). (Historically the fast
model was disabled and the pipeline ran on a separate reasoning model; that
is no longer the case.)
4. **Gemma 31B endpoint (10.11.10.16:8001)** — port unreachable (host pings
OK). Not used by any feature; listed as optional in `.env`.
5. **Atomic's React UI not running** — we only deploy `atomic-server` (API),
4. **Atomic's React UI not running** — we only deploy `atomic-server` (API),
not `atomic-web` (React frontend). `http://localhost:8088/` returns 404
for this reason. Brain_api has its own Swagger UI at `/docs` which covers
dev-testing needs. If visual atom/tag/wiki browsing is needed, add an
@ -362,7 +386,7 @@ curl -s -X POST http://localhost:8090/v1/gather \
| Symptom | Root cause | Fix |
|---|---|---|
| `/health` timeouts | BGE endpoint unreachable (VPN down?) | `curl http://10.11.10.15:8200/v1/models` on host; bring VPN back |
| `/health` timeouts | BGE endpoint unreachable | `curl http://10.11.10.15:14100/v1/models` on host; restore upstream reachability |
| `/v1/gather` returns 500 silently | Same as above — Atomic can't embed the query | Same fix |
| Post-reboot: brain_api empty reply | uvicorn bound to 127.0.0.1 inside container (not 0.0.0.0) | Rebuild — Dockerfile now sets `BRAIN_API_HOST=0.0.0.0` |
| `LLM_ROUTER_URL` points to localhost but nothing there | Router is actually on 10.11.10.17:14011 (not on the dev box) | Update `.env`; validated in session 2 |
@ -396,12 +420,12 @@ curl -s -X POST http://localhost:8090/v1/gather \
- Built `shared/` + `extractor/` + scripts 01-08
- Stood up Atomic + Postgres in compose
- Imported 19 Wikipedia vaccines articles, EN + RO
- Extracted 513 claims via Qwen 397B (1.2% hallucination drop)
- Extracted 513 claims via qwen3.5 (1.2% hallucination drop)
- Validated claim-level retrieval (cross-lingual RO↔EN confirmed)
### Session 2 (2026-04-11, ~4 hours)
- Re-onboarded state after VPN restart (fixed LLM router URL)
- Built `brain_api/` FastAPI service with 5-endpoint Didi contract
- Built `brain_api/` FastAPI service with the Didi web-gathering contract
- Dockerized brain_api (self-sufficient startup, taxonomy refresh from atomic)
- Added NLI stance-vs-query pass in `/v1/gather` (additive fields in `brain_meta`)
- Fixed NLI concurrency (MAX_PARALLEL=2 matches llama.cpp backends)

9
ai_platform/modules/didi_brain/infra/docker-compose.yml
View file

@ -1,15 +1,18 @@
# =============================================================================
# DidiBrain — Atomic + Postgres pgvector
# =============================================================================
# This compose stack runs:
# This compose stack runs four services:
# - postgres (pgvector/pgvector:pg16) on port 5434
# - atomic-server (kenforthewin/atomic-server:latest) on port 8080
# - atomic-server (kenforthewin/atomic-server:latest) on port 8088 (8080 internal)
# - brain-api (FastAPI service) on port 8090
# - scheduler (feeder + auditor + watcher + heartbeat, no exposed port)
# All four attach to the external shared `didi-network`.
#
# Atomic is configured to use Postgres as the data backend (atoms, embeddings,
# tags, etc.) while keeping the SQLite registry for tokens & global settings
# in the local volume at /data.
#
# AI provider config (BGE-M3 endpoint, Qwen 397B router) is NOT set here —
# AI provider config (BGE-M3 endpoint, qwen3.5 router) is NOT set here —
# it lives in Atomic's settings table and is bootstrapped post-startup by
# `scripts/02_bootstrap_atomic.py` which calls PUT /api/settings.
# =============================================================================

2
ai_platform/modules/didi_brain/pyproject.toml
View file

@ -47,4 +47,4 @@ requires = ["hatchling"]
build-backend = "hatchling.build"
[tool.hatch.build.targets.wheel]
packages = ["shared", "scraper", "extractor", "lint", "didi_client"]
packages = ["shared", "extractor", "lint"]

100
ai_platform/modules/domain_check/.env Normal file
View file

@ -0,0 +1,100 @@
# ===========================================
# DIDI Domain Check (T4 — evaluare credibilitate sursă)
# Modul dedicat platformei DIDI. Rulează izolat: Postgres + Redis proprii
# pe rețeaua internă domain_check_network; API-ul e expus pe didi-network
# ca `domain-check-api:11000` pentru agent-v3 (Lot 2).
# ===========================================
# Imagine locală (se construiește din ./api, nu se trage din registry)
REGISTRY_IMAGE=didiai-domain-check
IMAGE_TAG=latest
# ===========================================
# Database (izolat — doar pe rețeaua internă)
# ===========================================
DB_PASSWORD=8W2XWONcuorqzY0h4xAgJeSJ
DB_USER=dns_admin
DB_NAME=domain_check
DB_HOST=didiAI-domain-check-db
DB_PORT=5432
DB_EXTERNAL_PORT=12000
# ===========================================
# Redis (izolat — doar pe rețeaua internă)
# ===========================================
REDIS_HOST=didiAI-domain-check-redis
REDIS_PORT=6379
REDIS_EXTERNAL_PORT=12300
REDIS_DB=0
REDIS_PASSWORD=
# ===========================================
# API Keys
# ===========================================
WHOXY_API_KEY=876528325417e0bgs418d2cc7d8f193a6
VIRUSTOTAL_API_KEY=4d7afdff71c977a178b07d25b58c36660c416e3e9b1bf7ee8efaaf94aa02d0bc
# ===========================================
# Application (producție)
# ===========================================
FLASK_ENV=production
FLASK_APP=run.py
SECRET_KEY=3UnlbbbKMLYalxtvu7IJHAwMMF5KIFYj
LOG_LEVEL=INFO
DEBUG=False
# ===========================================
# API
# ===========================================
API_HOST=0.0.0.0
API_PORT=11000
API_VERSION=v1
# ===========================================
# Cache TTL (secunde)
# ===========================================
REDIS_TTL_HOT=21600
REDIS_TTL_WARM=86400
REDIS_TTL_COLD=604800
# ===========================================
# Risk thresholds
# ===========================================
RISK_THRESHOLD_LOW=30
RISK_THRESHOLD_MEDIUM=60
RISK_THRESHOLD_HIGH=85
DOMAIN_AGE_CRITICAL=90
DOMAIN_AGE_HIGH=180
DOMAIN_AGE_MEDIUM=365
# ===========================================
# Rate limiting extern (protejează cotele Whoxy/VirusTotal; intern e exempt)
# ===========================================
WHOXY_DAILY_LIMIT=8000
VIRUSTOTAL_DAILY_LIMIT=500
# ===========================================
# Batch
# ===========================================
BATCH_CHUNK_SIZE=50
BATCH_TIMEOUT_SECONDS=300
# ===========================================
# Celery (broker/back-end pe Redis-ul intern)
# ===========================================
CELERY_BROKER_URL=redis://didiAI-domain-check-redis:6379/1
CELERY_RESULT_BACKEND=redis://didiAI-domain-check-redis:6379/2
# ===========================================
# Monitoring (opțional)
# ===========================================
SENTRY_DSN=
DATADOG_API_KEY=
# ===========================================
# Security
# ===========================================
# Consumatorul e agent-v3 (server-to-server pe didi-network), fără browser → CORS lax OK.
CORS_ORIGINS=*
DISABLE_SWAGGER=false
SIMPLE_HEALTH=false

172
ai_platform/modules/domain_check/.env.example Normal file
View file

@ -0,0 +1,172 @@
# ===========================================
# Domain Check API - Environment Configuration
# ===========================================
# Port Schema: Dev (5xxxx), Prod (1xxxx)
# ===========================================
# ===========================================
# Registry Configuration (pentru deploy din GitLab)
# ===========================================
REGISTRY_IMAGE=didiai-domain-check
IMAGE_TAG=latest
# ===========================================
# Database Configuration
# Port: x20xx = Databases/PostgreSQL
# ===========================================
# SCHIMBA PAROLA pentru productie!
# Genereaza cu: openssl rand -base64 24 | tr -dc 'a-zA-Z0-9' | head -c 24
DB_PASSWORD=DnsCheck2026!Secure
DB_USER=dns_admin
DB_NAME=domain_check
DB_HOST=domain_check_postgres
DB_PORT=5432
# Port producție conform schema (x20xx = PostgreSQL)
DB_EXTERNAL_PORT=12000
# ===========================================
# Redis Configuration
# Port: x23xx = Cache/Redis
# ===========================================
REDIS_HOST=domain_check_redis
REDIS_PORT=6379
# Port producție conform schema (x23xx = Redis)
REDIS_EXTERNAL_PORT=12300
REDIS_DB=0
REDIS_PASSWORD=
# ===========================================
# API Keys
# ===========================================
# WHOXY API Key - obtine de la https://www.whoxy.com/
WHOXY_API_KEY=876528325417e0bgs418d2cc7d8f193a6
# VIRUSTOTAL API Key (optional) - obtine de la https://www.virustotal.com/
VIRUSTOTAL_API_KEY=4d7afdff71c977a178b07d25b58c36660c416e3e9b1bf7ee8efaaf94aa02d0bc
# ===========================================
# Application Settings
# ===========================================
# SCHIMBA pentru productie: production
FLASK_ENV=development
FLASK_APP=run.py
# SCHIMBA SECRET_KEY pentru productie!
# Genereaza cu: openssl rand -base64 32 | tr -dc 'a-zA-Z0-9' | head -c 32
SECRET_KEY=dns-check-secret-key-2026-change-in-production
LOG_LEVEL=INFO
# SCHIMBA pentru productie: False
DEBUG=True
# ===========================================
# API Configuration
# Port: x1xxx = API/Gateway
# ===========================================
API_HOST=0.0.0.0
# Port producție conform schema (x1xxx = API/Gateway)
API_PORT=11000
API_VERSION=v1
# ===========================================
# Cache TTL (seconds)
# ===========================================
REDIS_TTL_HOT=21600 # 6 hours
REDIS_TTL_WARM=86400 # 24 hours
REDIS_TTL_COLD=604800 # 7 days
# ===========================================
# Risk Score Thresholds
# ===========================================
RISK_THRESHOLD_LOW=30
RISK_THRESHOLD_MEDIUM=60
RISK_THRESHOLD_HIGH=85
# ===========================================
# Domain Age Thresholds (days)
# ===========================================
DOMAIN_AGE_CRITICAL=90 # < 3 months
DOMAIN_AGE_HIGH=180 # < 6 months
DOMAIN_AGE_MEDIUM=365 # < 1 year
# ===========================================
# API Rate Limiting
# ===========================================
WHOXY_DAILY_LIMIT=8000
VIRUSTOTAL_DAILY_LIMIT=500
# ===========================================
# Batch Processing
# ===========================================
BATCH_CHUNK_SIZE=50
BATCH_TIMEOUT_SECONDS=300
# ===========================================
# Celery Configuration
# ===========================================
CELERY_BROKER_URL=redis://domain_check_redis:6379/1
CELERY_RESULT_BACKEND=redis://domain_check_redis:6379/2
# ===========================================
# Monitoring (optional)
# ===========================================
SENTRY_DSN=
DATADOG_API_KEY=
# ===========================================
# Security Settings (PRODUCTIE)
# ===========================================
# CORS Origins - Lista domeniilor permise (separate prin virgula)
# Default: * (permite toate - DOAR pentru development!)
# Productie: specifica domeniile tale
# Exemplu: CORS_ORIGINS=https://didi.example.ro,https://example.ro
CORS_ORIGINS=*
# Dezactiveaza Swagger UI si ReDoc (ascunde /docs si /redoc)
# Default: false (development)
# Productie: true
DISABLE_SWAGGER=false
# Health endpoint simplificat (returneaza doar {"status":"ok"})
# Default: false (development - returneaza detalii complete)
# Productie: true (ascunde informatii despre sistem)
SIMPLE_HEALTH=false
# ===========================================
# QUICK SETUP pentru server DEVELOPMENT:
# ===========================================
# 1. cp .env.example .env
# 2. docker compose up -d --build
# 3. curl http://localhost:11000/health
# ===========================================
# ===========================================
# QUICK SETUP pentru server PRODUCTION:
# ===========================================
# 1. cp .env.example .env
# 2. Modifica valorile pentru productie:
# sed -i 's/FLASK_ENV=development/FLASK_ENV=production/' .env
# sed -i 's/DEBUG=True/DEBUG=False/' .env
# sed -i 's/CORS_ORIGINS=\*/CORS_ORIGINS=https:\/\/your-domain.com/' .env
# sed -i 's/DISABLE_SWAGGER=false/DISABLE_SWAGGER=true/' .env
# sed -i 's/SIMPLE_HEALTH=false/SIMPLE_HEALTH=true/' .env
# 3. Genereaza parole securizate:
# DB_PASS=$(openssl rand -base64 24 | tr -dc 'a-zA-Z0-9' | head -c 24)
# SECRET=$(openssl rand -base64 32 | tr -dc 'a-zA-Z0-9' | head -c 32)
# sed -i "s/DnsCheck2026\!Secure/$DB_PASS/" .env
# sed -i "s/dns-check-secret-key-2026-change-in-production/$SECRET/" .env
# 4. docker compose up -d
# 5. curl http://localhost:11000/health
# ===========================================

100
ai_platform/modules/domain_check/.gitignore vendored Normal file
View file

@ -0,0 +1,100 @@
# Environment variables
.env
.env.local
.env.*.local
# Python
__pycache__/
*.py[cod]
*$py.class
*.so
.Python
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
wheels/
share/python-wheels/
*.egg-info/
.installed.cfg
*.egg
MANIFEST
*.manifest
*.spec
pip-log.txt
pip-delete-this-directory.txt
# Virtual environments
venv/
ENV/
env/
.venv
# IDEs
.vscode/
.idea/
*.swp
*.swo
*~
.DS_Store
# Logs
*.log
logs/
*.log.*
# Database
*.db
*.sqlite
*.sqlite3
# Redis dump
dump.rdb
# Docker
.docker/
# Testing
.pytest_cache/
.coverage
htmlcov/
.tox/
.nox/
coverage.xml
*.cover
.hypothesis/
# Jupyter Notebook
.ipynb_checkpoints
# pyenv
.python-version
# Celery
celerybeat-schedule
celerybeat.pid
# Backup files
*.bak
*.backup
*.tmp
# OS
Thumbs.db
.DS_Store
# Project specific
data/
backups/
exports/
# GitLab setup scripts (contain secrets)
gitlab-variables-values.txt
setup-gitlab-variables.sh

325
ai_platform/modules/domain_check/.gitlab-ci.yml Normal file
View file

@ -0,0 +1,325 @@
# ===========================================
# Domain Check API - GitLab CI/CD Pipeline
# ===========================================
# Registry: <registry-didi> (GitLab Container Registry)
# Runner: Main Docker Runner (10.11.10.102) - tags: docker, ci, cd
# ===========================================
stages:
- lint
- security
- test
- build
- release
- deploy
variables:
DOCKER_DRIVER: overlay2
DOCKER_TLS_CERTDIR: "/certs"
# GitLab CI provides these automatically:
# CI_REGISTRY = <registry-didi>
# CI_REGISTRY_IMAGE = didiai-domain-check
# CI_REGISTRY_USER / CI_REGISTRY_PASSWORD = auto-generated JWT
# Cache pip packages
cache:
key: ${CI_COMMIT_REF_SLUG}
paths:
- .cache/pip
# ===========================================
# LINT Stage
# ===========================================
lint:python:
stage: lint
image: python:3.10-slim
tags:
- docker
before_script:
- pip install --cache-dir .cache/pip flake8 black isort
script:
- flake8 api/app --max-line-length=120 --ignore=E501,W503 || true
- black --check api/app || true
- isort --check-only api/app || true
allow_failure: true
rules:
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
- if: $CI_COMMIT_BRANCH
lint:dockerfile:
stage: lint
image: hadolint/hadolint:latest-debian
tags:
- docker
script:
- hadolint api/Dockerfile || true
allow_failure: true
rules:
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
- if: $CI_COMMIT_BRANCH
# ===========================================
# SECURITY Stage (SAST)
# ===========================================
sast:bandit:
stage: security
image: python:3.10-slim
tags:
- docker
before_script:
- pip install --cache-dir .cache/pip bandit
script:
- bandit -r api/app -f json -o bandit-report.json || true
- bandit -r api/app -f txt || true
artifacts:
paths:
- bandit-report.json
when: always
expire_in: 1 week
allow_failure: true
rules:
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
- if: $CI_COMMIT_BRANCH
sast:safety:
stage: security
image: python:3.10-slim
tags:
- docker
before_script:
- pip install --cache-dir .cache/pip safety
script:
- cd api && safety check -r requirements.txt --json > ../safety-report.json || true
- cd api && safety check -r requirements.txt || true
artifacts:
paths:
- safety-report.json
when: always
expire_in: 1 week
allow_failure: true
rules:
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
- if: $CI_COMMIT_BRANCH
sast:trivy:
stage: security
image:
name: aquasec/trivy:latest
entrypoint: [""]
tags:
- docker
script:
- trivy fs --exit-code 0 --severity HIGH,CRITICAL --format json -o trivy-report.json . || true
- trivy fs --exit-code 0 --severity HIGH,CRITICAL . || true
artifacts:
paths:
- trivy-report.json
when: always
expire_in: 1 week
allow_failure: true
rules:
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
- if: $CI_COMMIT_BRANCH
# ===========================================
# TEST Stage
# ===========================================
test:api:
stage: test
image: python:3.10-slim
tags:
- docker
services:
- postgres:15-alpine
- redis:7-alpine
variables:
POSTGRES_DB: test_domain_check
POSTGRES_USER: test_user
POSTGRES_PASSWORD: test_password
DATABASE_URL: postgresql://test_user:test_password@postgres:5432/test_domain_check
REDIS_URL: redis://redis:6379/0
FLASK_ENV: testing
before_script:
- cd api
- pip install --cache-dir ../.cache/pip -r requirements.txt
script:
- python -c "from app import create_app; app = create_app(); print('App created successfully')"
allow_failure: true
rules:
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
- if: $CI_COMMIT_BRANCH
# ===========================================
# BUILD Stage - Push to GitLab Container Registry
# ===========================================
build:docker:
stage: build
image: docker:24-dind
tags:
- docker
- ci
services:
- docker:24-dind
before_script:
# Login to GitLab Container Registry (auto credentials)
- docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
script:
- echo "Building image $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG"
- docker build -t $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG -t $CI_REGISTRY_IMAGE:latest ./api
- docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG
- docker push $CI_REGISTRY_IMAGE:latest
- echo "Image pushed to $CI_REGISTRY_IMAGE"
rules:
- if: $CI_COMMIT_BRANCH == "main"
- if: $CI_COMMIT_BRANCH == "develop"
# ===========================================
# RELEASE Stage - Create deployment bundle
# ===========================================
release:bundle:
stage: release
image: alpine:latest
tags:
- docker
script:
- apk add --no-cache tar gzip
- mkdir -p release-bundle/deploy
# Copy deployment files
- cp docker-compose.yml release-bundle/deploy/
- cp deploy.sh release-bundle/deploy/ 2>/dev/null || echo "#!/bin/bash" > release-bundle/deploy/deploy.sh
- cp .env.example release-bundle/deploy/ 2>/dev/null || cp .env release-bundle/deploy/env.example 2>/dev/null || true
# Create IMAGE.txt with registry path
- echo "$CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG" > release-bundle/IMAGE.txt
- echo "$CI_REGISTRY_IMAGE:latest" >> release-bundle/IMAGE.txt
# Create restore script
- |
cat > release-bundle/restore.sh << 'RESTORE_EOF'
#!/bin/bash
set -e
echo "=== Domain Check API - Restore ==="
IMAGE=$(head -1 IMAGE.txt)
echo "Pulling image: $IMAGE"
docker pull $IMAGE
echo "Starting services..."
cd deploy
docker compose up -d
echo "Done! Check: docker compose ps"
RESTORE_EOF
- chmod +x release-bundle/restore.sh release-bundle/deploy/deploy.sh
# Create tarball
- tar -czvf release-bundle-${CI_COMMIT_SHORT_SHA}.tgz release-bundle/
- ls -la release-bundle-*.tgz
artifacts:
paths:
- release-bundle-*.tgz
expire_in: 30 days
rules:
- if: $CI_COMMIT_BRANCH == "main"
# ===========================================
# DEPLOY Stage - Deploy to server via SSH
# ===========================================
deploy:dev:
stage: deploy
image: alpine:latest
tags:
- docker
- cd
before_script:
- apk add --no-cache openssh-client
- eval $(ssh-agent -s)
- echo "$SSH_PRIVATE_KEY" | tr -d '\r' | ssh-add -
- mkdir -p ~/.ssh
- chmod 700 ~/.ssh
- echo "$SSH_KNOWN_HOSTS" >> ~/.ssh/known_hosts
- chmod 644 ~/.ssh/known_hosts
script:
- |
ssh $DEPLOY_USER@$DEPLOY_HOST << ENDSSH
set -e
echo "=== Deploying Domain Check API ==="
cd /home/admin365/domain-check
# Pull latest code
git pull origin main
# Login to GitLab registry
docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
# Pull latest image
docker pull $CI_REGISTRY_IMAGE:latest || echo "Pull failed, building locally"
# Restart services
docker compose down
docker compose up -d --build
# Health check
sleep 10
curl -s http://localhost:51000/health || echo "Health check pending..."
docker compose ps
echo "=== Deploy complete ==="
ENDSSH
environment:
name: development
url: http://domain-check-api:11000
rules:
- if: $CI_COMMIT_BRANCH == "main"
when: manual
allow_failure: true
deploy:auto:
stage: deploy
image: alpine:latest
tags:
- docker
- cd
before_script:
- apk add --no-cache openssh-client
- eval $(ssh-agent -s)
- echo "$SSH_PRIVATE_KEY" | tr -d '\r' | ssh-add -
- mkdir -p ~/.ssh
- chmod 700 ~/.ssh
- echo "$SSH_KNOWN_HOSTS" >> ~/.ssh/known_hosts
- chmod 644 ~/.ssh/known_hosts
script:
- |
ssh $DEPLOY_USER@$DEPLOY_HOST << ENDSSH
set -e
cd /home/admin365/domain-check
git pull origin main
docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY || true
docker pull $CI_REGISTRY_IMAGE:latest || true
docker compose up -d --build
docker compose ps
ENDSSH
environment:
name: development
url: http://domain-check-api:11000
rules:
- if: $CI_COMMIT_BRANCH == "main"
allow_failure: true
# ===========================================
# Cleanup
# ===========================================
cleanup:images:
stage: .post
image: docker:24-dind
tags:
- docker
services:
- docker:24-dind
script:
- docker image prune -f
when: always
allow_failure: true
rules:
- if: $CI_COMMIT_BRANCH == "main"

192
ai_platform/modules/domain_check/ACCESS_INFO.md Normal file
View file

@ -0,0 +1,192 @@
# 🌐 Access Information - Domain Check API
## 📍 Server Details
**Server IP (LAN):** `10.11.10.200`
**Ports:**
- API: `5000`
- PostgreSQL: `5433` (internal: 5432)
- Redis: `6380` (internal: 6379)
- Dashboard: `8501` (when ready)
---
## 🔗 API Endpoints
### **Base URL:** `http://10.11.10.200:5000`
### Main Endpoints:
- **Health Check:** `http://10.11.10.200:5000/health`
- **API Root:** `http://10.11.10.200:5000/`
- **Domain Check:** `POST http://10.11.10.200:5000/api/v1/check`
### Documentation:
- **Swagger UI:** `http://10.11.10.200:5000/docs` 📖
- **ReDoc:** `http://10.11.10.200:5000/redoc` 📚
---
## 🧪 Quick Tests
### 1. Health Check
```bash
curl http://10.11.10.200:5000/health
```
### 2. WHOIS + DNS Check
```bash
curl -X POST http://10.11.10.200:5000/api/v1/check \
-H "Content-Type: application/json" \
-d '{
"domain": "google.com",
"check_options": {
"whois": true,
"dns": true,
"ssl": false
}
}' | python3 -m json.tool
```
### 3. Full Test Suite
```bash
cd /home/admin365/domain-check
./test-complete-lan.sh
```
---
## 🔐 Database Access
### PostgreSQL
```bash
# From server
docker exec -it dns_postgres psql -U dns_admin -d domain_check
# Connection string
postgresql://dns_admin:DnsCheck2026!Secure@10.11.10.200:5433/domain_check
```
### Redis
```bash
# From server
docker exec -it dns_redis redis-cli
# Connection string
redis://10.11.10.200:6380/0
```
---
## 🐳 Docker Services
### Check Status
```bash
docker compose ps
```
### View Logs
```bash
# All services
docker compose logs -f
# Specific service
docker compose logs -f dns_api
```
### Restart Services
```bash
# All services
docker compose restart
# Specific service
docker compose restart dns_api
```
---
## 📊 Test Results Summary
✅ **Health Check:** PASSED
✅ **WHOIS Lookup:** PASSED
✅ **DNS Checking:** PASSED
✅ **Risk Scoring:** PASSED
✅ **Database Storage:** PASSED
✅ **API Documentation:** PASSED
⏸️ **SSL Checking:** Code ready (needs Docker network fix)
⏸️ **Redis Caching:** Configured (not active)
⏸️ **Batch Processing:** Not implemented yet
---
## 🌍 Access from Other Machines
From any machine on the **10.11.10.x** network:
### Browser Access:
- Swagger UI: `http://10.11.10.200:5000/docs`
- ReDoc: `http://10.11.10.200:5000/redoc`
### API Calls:
```bash
curl http://10.11.10.200:5000/api/v1/check \
-H "Content-Type: application/json" \
-d '{"domain": "your-domain.com"}'
```
---
## 📱 Frontend Access (When Ready)
**Dashboard URL:** `http://10.11.10.200:3000`
**Status:** Not yet implemented (Next.js ready to build)
---
## 🔧 Troubleshooting
### API not responding?
```bash
# Check if containers are running
docker compose ps
# Check API logs
docker compose logs dns_api --tail=50
# Restart API
docker compose restart dns_api
```
### Database connection issues?
```bash
# Check PostgreSQL
docker exec dns_postgres pg_isready -U dns_admin
# View database logs
docker compose logs dns_postgres --tail=30
```
### Port conflicts?
```bash
# Check if ports are in use
sudo netstat -tlnp | grep -E "5000|5433|6380"
```
---
## 📞 Support
**GitLab Repository:** (modul domain_check al platformei DIDI)
**Documentation:** See `/home/admin365/domain-check/` directory
**Key Files:**
- `README.md` - Quick start guide
- `API_ARCHITECTURE.md` - Technical details
- `CURRENT_STATUS.md` - Complete API reference
- `NEXT_STEPS.md` - Development roadmap
---
**Last Updated:** 2026-01-29 21:55:00
**Server:** admin365@10.11.10.200
**Status:** ✅ Production Ready (MVP)

1233
ai_platform/modules/domain_check/API_ARCHITECTURE.md Normal file
View file

File diff suppressed because it is too large Load diff

62
ai_platform/modules/domain_check/CLAUDE.md Normal file
View file

@ -0,0 +1,62 @@
# CLAUDE.md
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
## What This Is
Domain Check API — a domain verification and risk-scoring service for detecting disinformation sources (newly registered / suspicious domains). Flask REST API + PostgreSQL + Redis + Celery worker, all run via Docker Compose. Live on this server at `http://domain-check-api:11000` (production ports: API 11000, PostgreSQL 12000, Redis 12300 — set in `.env`, the README sometimes still shows dev ports 5xxxx).
## Commands
```bash
# Run the full stack (the only supported way to run it — the API expects Postgres/Redis containers)
docker compose up -d --build
docker compose ps
docker compose logs -f domain_check_api
# Smoke test
curl http://localhost:11000/health
./test-api-complete.sh # exercises the check endpoint
./test-complete-lan.sh # LAN integration test
# Manual check call
curl -X POST http://localhost:11000/api/v1/check/check \
-H "Content-Type: application/json" \
-d '{"domain": "example.com", "check_options": {"whois": true, "dns": true, "ssl": true}}'
# Lint (matches .gitlab-ci.yml)
flake8 api/app --max-line-length=120 --ignore=E501,W503
black --check api/app
isort --check-only api/app
# Database shell
docker exec -it didiAI-domain-check-db psql -U dns_admin -d domain_check
# Deploy helper (wraps docker compose)
./deploy.sh status|logs|start|stop|restart|update
```
There is no test suite — CI's "test" stage only verifies `create_app()` succeeds. CI (GitLab) runs lint → SAST (bandit/safety/trivy) → build → push to `didiai-domain-check` → SSH deploy; all lint/security jobs are `allow_failure: true`.
## Architecture
Everything lives in `api/app/` (the `dashboard/` directory is vestigial — only a Dockerfile/requirements, not in docker-compose; the actual dashboard is `api/app/static/index.html` served at `/`).
**Request flow:** `run.py` → `create_app()` app factory ([api/app/__init__.py](api/app/__init__.py)) → flask-restx namespaces under `/api/v1` → route handlers in `app/routes/` call service classes in `app/services/` → results persisted via SQLAlchemy models in `app/models/` and cached in Redis. Served by **gunicorn** (gthread workers) in the container — `python run.py` is dev-only.
- **`app/routes/check.py`** is the only fully implemented route (POST `/api/v1/check/check`). It runs the independent lookups **concurrently** in two phases via `ThreadPoolExecutor` (phase 1 needs only the domain: whois/dns/ssl/http/subdomains; phase 2 needs DNS results: ip/blacklist/port/mail). Each task runs inside `app.app_context()` because services read `current_app`. **All DB writes happen in the main thread afterwards** — the SQLAlchemy session is not thread-safe, so never add DB writes inside the threaded lookups. `domain.py`, `search.py`, `stats.py`, `batch.py` are stubs registered inside try/except ImportError so missing implementations fail silently.
- **`app/services/`** — one class per check type: `whois_service` (RDAP first, Whoxy API fallback; `normalize_whois_date()` coerces all dates to `datetime|None` — raw strings here were the root cause of the `.ro` 500), `dns_service`, `ssl_service`, `ip_intelligence_service`, `http_analysis_service`, `blacklist_service` (DNSBL), `port_scan_service`, `subdomain_service` (Certificate Transparency), and `mail_intelligence_service` (SPF/DKIM/DMARC parsing, MTA-STS/TLS-RPT/DANE, MX provider fingerprint + STARTTLS, optional SMTP RCPT probe — degrades gracefully when outbound :25 is blocked). `risk_scorer.py` combines outputs into a weighted 0–100 score (weights/thresholds in `config.py`); `_coerce_datetime()` there is the defense-in-depth guard against stray string dates.
- **`check_options`** flags (all default true except `port_scan`/`subdomains`/`smtp_probe` which default false): `whois, dns, ssl, ip_intelligence, http_analysis, blacklist, mail, port_scan, subdomains, smtp_probe, force_refresh`. The response includes a top-level `availability` block (`is_registered`/`is_available`/`confidence`) derived from combined WHOIS+DNS signals — WHOIS alone is unreliable for sparse registries like ROTLD.
- **Rate limiting** (`Flask-Limiter`, Redis-backed) is wired in `__init__.py` with internal LAN/loopback and `/health` exempt (`_is_internal_request`), errors swallowed. Tune via `RATELIMIT_DEFAULT` env. It exists to protect the paid Whoxy/VirusTotal quotas from external abuse, not internal callers.
- **`app/api_models.py`** defines all flask-restx request/response models. Note the pattern in `check.py`: models are created on a throwaway namespace at import time for decorators, then `__init__.py` re-attaches them to the real namespaces — keep model definitions in `api_models.py`, not inline.
- **Celery** is wired up (`celery_app.py`, separate `domain_check_worker` container) but `app/tasks/__init__.py` is empty — no tasks exist yet; batch processing is the intended use.
**Configuration:** `app/config.py` selects Development/Production/Testing via `FLASK_ENV`. Production flips important defaults: Swagger/ReDoc disabled (`DISABLE_SWAGGER`), minimal `/health` response (`SIMPLE_HEALTH`), CORS empty unless `CORS_ORIGINS` is set. All runtime config comes from `.env` (see `.env.example`); ports are only ever changed there, never in compose/code.
**Database schema** is created by `init-scripts/01-init-db.sql`, which runs only on first Postgres volume creation. There is no Flask-Migrate `migrations/` directory despite the extension being initialized — schema changes must be made in both the SQL init script and the SQLAlchemy models, and applied manually to existing databases. All child tables (`whois_records`, `dns_records`, `ssl_certificates`, etc.) FK to `domains` with `ON DELETE CASCADE`.
**Caching:** Redis with three TTL tiers (`REDIS_TTL_HOT/WARM/COLD`); a check request with `force_refresh: false` returns cached/DB data when fresh. Redis failure is non-fatal — the app logs a warning and runs with caching disabled.
## Network Context
This is the **T4 module (evaluare credibilitate sursă)** of the DIDI platform, living at `ai_platform/modules/domain_check`. It runs as an isolated stack (own Postgres + Redis on the internal `domain_check_network`); the API container also joins the shared external `didi-network` under the alias **`domain-check-api`**, so `agent-v3` (Lot 2) calls it at `http://domain-check-api:11000/api/v1/check/check` with no extra config. The API port is **not** published to the host (the platform gateway owns host:11000) — reach it via the didi-network alias, or `docker exec` into a container on that network. Docs: `API_ARCHITECTURE.md` (detailed design), `use_api.md` (consumer-facing API docs), `ACCESS_INFO.md`, `CURRENT_STATUS.md`, `NEXT_STEPS.md` (roadmap).

706
ai_platform/modules/domain_check/CURRENT_STATUS.md Normal file
View file

@ -0,0 +1,706 @@
# DOMAIN-CHECK - Status Actual și Documentație Completă
## 🗄️ DATABASE - Ce se Salvează Exact
### Fluxul Complet de Date
```
INPUT: {"domain": "example.com"}
↓
1. DOMAIN EXTRACTION & PARSING
↓
2. WHOIS LOOKUP (python-whois + Whoxy API fallback)
↓
3. RISK SCORING (5 factori)
↓
4. SAVE TO DATABASE (3 tabele principale)
↓
OUTPUT: JSON cu risk score + toate datele
```
---
## 📋 TABELE DATABASE - Date Salvate
### 1. **domains** - Informații Domain Principal
```sql
Câmpuri salvate:
- id (UUID)
- domain (ex: "example")
- subdomain (ex: "www" sau NULL)
- tld (ex: "com")
- full_domain (ex: "www.example.com")
- first_seen_at (când a fost văzut prima dată)
- last_checked_at (ultima verificare)
- check_count (de câte ori a fost verificat)
- is_active (boolean)
- created_at, updated_at
```
**Exemplu real:**
```json
{
"id": "587b61bf-e6bf-436b-9867-14df7d5a9cfb",
"domain": "google",
"subdomain": null,
"tld": "com",
"full_domain": "google.com",
"first_seen_at": "2026-01-29T13:38:27Z",
"last_checked_at": "2026-01-29T13:38:33Z",
"check_count": 1,
"is_active": true
}
```
---
### 2. **whois_records** - Date WHOIS Complete
```sql
Câmpuri salvate:
- id (UUID)
- domain_id (referință la domains)
- creation_date (când a fost creat domeniul)
- expiration_date (când expiră)
- updated_date (ultima actualizare WHOIS)
- registrar (ex: "GoDaddy", "Namecheap")
- registrar_url
- registrant_org (organizația proprietarului)
- registrant_country (țara - cod 2 litere)
- admin_email
- name_servers (array de DNS servers)
- status (array de statusuri domeniu)
- dnssec (boolean)
- raw_whois_data (JSONB - toate datele raw)
- data_source ("whois" sau "whoxy")
- fetched_at (când au fost preluate datele)
```
**Exemplu real pentru un domeniu funcțional:**
```json
{
"domain_id": "587b61bf-e6bf-436b-9867-14df7d5a9cfb",
"creation_date": "1997-09-15T04:00:00Z",
"expiration_date": "2028-09-14T04:00:00Z",
"registrar": "MarkMonitor Inc.",
"registrant_org": "Google LLC",
"registrant_country": "US",
"admin_email": "dns-admin@google.com",
"name_servers": ["ns1.google.com", "ns2.google.com"],
"status": ["clientDeleteProhibited", "clientTransferProhibited"],
"data_source": "whoxy",
"age_days": 10359
}
```
---
### 3. **risk_assessments** - Risk Scoring Rezultate
```sql
Câmpuri salvate:
- id (UUID)
- domain_id (referință)
- check_id (UUID unic pentru fiecare check)
- total_score (0-100)
- risk_level ("LOW", "MEDIUM", "HIGH", "CRITICAL")
- domain_age_score (scor 0-100)
- domain_age_days (vârsta în zile)
- ssl_score (0-100)
- dns_score (0-100)
- reputation_score (0-100)
- whois_score (0-100)
- factors (JSONB - detalii fiecare factor)
- is_new_domain (boolean - < 6 luni)
- is_suspicious (boolean)
- requires_manual_review (boolean)
- assessed_at (timestamp)
```
**Exemplu real:**
```json
{
"check_id": "76d6eb1a-d74b-4430-8b09-f2474a088699",
"total_score": 8,
"risk_level": "LOW",
"domain_age_score": 20,
"domain_age_days": null,
"ssl_score": 0,
"dns_score": 0,
"reputation_score": 0,
"whois_score": 20,
"is_new_domain": false,
"is_suspicious": false,
"factors": [
{
"factor": "domain_age",
"score": 20,
"weight": 0.3,
"reason": "No creation date available"
},
{
"factor": "whois",
"score": 20,
"weight": 0.1,
"reason": "No registrant organization"
}
]
}
```
---
### 4. **check_history** - Istoric Verificări
```sql
Câmpuri salvate:
- id (UUID)
- check_id (UUID unic)
- domain_id (referință)
- risk_assessment_id (referință)
- requested_by ("api", "cli", "dashboard")
- request_ip (IP-ul clientului)
- user_agent (browser/client info)
- check_options (JSONB - ce a fost verificat)
- processing_time_ms (cât a durat)
- cache_hit (boolean - a fost în cache?)
- changes_detected (boolean)
- change_summary (JSONB)
- status ("completed", "failed")
- error_message (dacă a fost eroare)
- created_at
```
**Exemplu real:**
```json
{
"check_id": "76d6eb1a-d74b-4430-8b09-f2474a088699",
"domain_id": "587b61bf-e6bf-436b-9867-14df7d5a9cfb",
"requested_by": "api",
"request_ip": "172.26.0.1",
"user_agent": "curl/7.81.0",
"check_options": {
"whois": true,
"dns": false,
"ssl": false,
"reputation": false,
"force_refresh": false
},
"processing_time_ms": 2009,
"cache_hit": false,
"status": "completed"
}
```
---
## 🔌 API ENDPOINTS - Ce Funcționează ACUM
### ✅ 1. **Health Check** - Verificare Status Sistem
**Endpoint:** `GET /health`
**Funcțional:** DA
**Parametri:** Niciunul
**Test:**
```bash
curl http://localhost:5000/health
```
**Răspuns:**
```json
{
"status": "healthy",
"version": "v1",
"environment": "development",
"database": "connected",
"redis": "disabled"
}
```
---
### ✅ 2. **API Root** - Informații Generale
**Endpoint:** `GET /`
**Funcțional:** DA
**Parametri:** Niciunul
**Test:**
```bash
curl http://localhost:5000/
```
**Răspuns:**
```json
{
"name": "Domain Check API",
"version": "v1",
"description": "Anti-Fake News Domain Verification & Risk Scoring API",
"documentation": {
"swagger": "/docs",
"redoc": "/redoc"
},
"endpoints": {
"health": "/health",
"api": "/api/v1"
}
}
```
---
### ✅ 3. **Domain Check** - Verificare Domeniu (PRINCIPAL)
**Endpoint:** `POST /api/v1/check`
**Funcțional:** DA (WHOIS + Risk Scoring)
**Content-Type:** `application/json`
**Parametri Request:**
```json
{
"domain": "string (required)",
"check_options": {
"whois": boolean (default: true),
"dns": boolean (default: false) [NOT IMPLEMENTED YET],
"ssl": boolean (default: false) [NOT IMPLEMENTED YET],
"reputation": boolean (default: false) [NOT IMPLEMENTED YET],
"force_refresh": boolean (default: false)
}
}
```
**Teste Complete:**
#### Test 1 - Domeniu Normal
```bash
curl -X POST http://localhost:5000/api/v1/check \
-H "Content-Type: application/json" \
-d '{
"domain": "google.com"
}'
```
#### Test 2 - Cu Toate Opțiunile
```bash
curl -X POST http://localhost:5000/api/v1/check \
-H "Content-Type: application/json" \
-d '{
"domain": "example.com",
"check_options": {
"whois": true,
"dns": false,
"ssl": false,
"reputation": false,
"force_refresh": true
}
}'
```
#### Test 3 - Domeniu Suspect (Nou)
```bash
curl -X POST http://localhost:5000/api/v1/check \
-H "Content-Type: application/json" \
-d '{
"domain": "suspicious-news-2026.com"
}'
```
#### Test 4 - Subdomeniu
```bash
curl -X POST http://localhost:5000/api/v1/check \
-H "Content-Type: application/json" \
-d '{
"domain": "www.facebook.com"
}'
```
#### Test 5 - Pretty Print (JSON formatat)
```bash
curl -X POST http://localhost:5000/api/v1/check \
-H "Content-Type: application/json" \
-d '{"domain": "github.com"}' \
| python3 -m json.tool
```
**Răspuns Complet (Exemplu):**
```json
{
"success": true,
"data": {
"domain": "example.com",
"check_id": "uuid-here",
"timestamp": "2026-01-29T15:30:00Z",
"whois": {
"creation_date": "1995-08-14T04:00:00Z",
"expiration_date": "2027-08-13T04:00:00Z",
"registrar": "IANA",
"age_days": 11125,
"status": ["clientDeleteProhibited"],
"name_servers": ["a.iana-servers.net", "b.iana-servers.net"]
},
"dns": null,
"ssl": null,
"reputation": null,
"risk_score": {
"total": 15,
"level": "LOW",
"factors": [
{
"factor": "domain_age",
"score": 0,
"weight": 0.3,
"reason": "Trusted age (11125 days old, 2+ years)",
"age_days": 11125
},
{
"factor": "reputation",
"score": 0,
"weight": 0.3,
"reason": "No reputation data"
},
{
"factor": "ssl",
"score": 0,
"weight": 0.2,
"reason": "Valid SSL certificate"
},
{
"factor": "dns",
"score": 0,
"weight": 0.1,
"reason": "Good DNS setup"
},
{
"factor": "whois",
"score": 0,
"weight": 0.1,
"reason": "Transparent WHOIS"
}
],
"thresholds": {
"low": "0-30",
"medium": "31-60",
"high": "61-85",
"critical": "86-100"
}
}
},
"metadata": {
"cached": false,
"processing_time_ms": 2009,
"api_version": "v1"
}
}
```
**Erori Posibile:**
```json
// 400 - Domain invalid
{
"success": false,
"error": {
"code": "INVALID_REQUEST",
"message": "Domain parameter is required"
}
}
// 500 - Eroare internă
{
"success": false,
"error": {
"code": "INTERNAL_ERROR",
"message": "An error occurred while checking the domain: ..."
}
}
```
---
### ⏸️ 4. **Get Domain Details** - Detalii Domeniu
**Endpoint:** `GET /api/v1/domain/{domain}`
**Funcțional:** NU (stub - returnează 501)
**Parametri:** `domain` (string în URL)
**Test:**
```bash
curl http://localhost:5000/api/v1/domain/google.com
```
**Răspuns actual:**
```json
{
"message": "Not yet implemented",
"domain": "google.com"
}
```
---
### ⏸️ 5. **Batch Check** - Verificare Multiplă
**Endpoint:** `POST /api/v1/check/batch`
**Funcțional:** NU (stub - returnează 501)
**Test:**
```bash
curl -X POST http://localhost:5000/api/v1/check/batch \
-H "Content-Type: application/json" \
-d '{
"domains": ["google.com", "facebook.com", "twitter.com"]
}'
```
---
### ⏸️ 6. **Batch Status** - Status Verificare Multiplă
**Endpoint:** `GET /api/v1/batch/{batch_id}/status`
**Funcțional:** NU (stub)
---
### ⏸️ 7. **Statistics** - Statistici Sistem
**Endpoint:** `GET /api/v1/stats`
**Funcțional:** NU (stub)
**Test:**
```bash
curl http://localhost:5000/api/v1/stats
```
---
### ⏸️ 8. **Search Domains** - Căutare Domenii
**Endpoint:** `GET /api/v1/search`
**Funcțional:** NU (stub)
**Test:**
```bash
curl "http://localhost:5000/api/v1/search?q=google&risk_level=LOW"
```
---
## 📚 SWAGGER UI - Documentație Interactivă
**URL:** http://localhost:5000/docs
Aici poți:
- Vedea toate endpoint-urile
- Testa direct din browser
- Vedea request/response examples
- Download OpenAPI spec
---
## 🔍 VERIFICARE DATE ÎN DATABASE
### Verifică toate domeniile
```bash
docker exec dns_postgres psql -U dns_admin -d domain_check \
-c "SELECT full_domain, check_count, last_checked_at FROM domains ORDER BY last_checked_at DESC LIMIT 10;"
```
### Verifică ultimele verificări
```bash
docker exec dns_postgres psql -U dns_admin -d domain_check \
-c "SELECT d.full_domain, ra.total_score, ra.risk_level, ra.assessed_at
FROM risk_assessments ra
JOIN domains d ON ra.domain_id = d.id
ORDER BY ra.assessed_at DESC LIMIT 10;"
```
### Verifică date WHOIS
```bash
docker exec dns_postgres psql -U dns_admin -d domain_check \
-c "SELECT d.full_domain, w.creation_date, w.registrar, w.data_source
FROM whois_records w
JOIN domains d ON w.domain_id = d.id
ORDER BY w.fetched_at DESC LIMIT 10;"
```
### Export JSON complet
```bash
docker exec dns_postgres psql -U dns_admin -d domain_check \
-c "SELECT row_to_json(d) FROM domains d LIMIT 5;" \
| python3 -m json.tool
```
---
## 🔧 SERVICII EXTERNE FOLOSITE
### 1. **python-whois Library**
- **Cost:** FREE, unlimited
- **Folosit pentru:** WHOIS lookups primary
- **Limitări:** Rate limiting de la registrars individuali
- **Status:** ✅ ACTIV
### 2. **Whoxy API** (Fallback)
- **API Key:** `876528325417e0bgs418d2cc7d8f193a6`
- **Cost:** FREE - 250,000 requests/lună
- **Endpoint:** `https://api.whoxy.com/`
- **Folosit pentru:** WHOIS când python-whois eșuează
- **Status:** ✅ CONFIGURAT (nu e folosit încă ca fallback automat)
**Test Direct Whoxy:**
```bash
curl "https://api.whoxy.com/?key=876528325417e0bgs418d2cc7d8f193a6&whois=example.com"
```
### 3. **VirusTotal API** (Pregătit)
- **API Key:** NU este configurat
- **Cost:** 500 requests/day FREE
- **Status:** ⏸️ Pregătit în cod, nefuncțional
---
## 📊 RISK SCORING - Algoritmul Exact
### Formula Finală:
```
TOTAL_SCORE = (
domain_age_score × 0.30 +
reputation_score × 0.30 +
ssl_score × 0.20 +
dns_score × 0.10 +
whois_score × 0.10
)
```
### 1. Domain Age Score (30% weight)
| Vârsta Domini | Score | Risk Level | Reason |
|---------------|-------|------------|--------|
| 0-90 zile | 100 | CRITICAL | Very new domain (<3 months) |
| 91-180 zile | 80 | HIGH | New domain (<6 months) |
| 181-365 zile | 50 | MEDIUM | Moderately new |
| 366-730 zile | 30 | LOW | Established (1-2 years) |
| 731+ zile | 0 | LOW | Trusted age (2+ years) |
| No data | 20 | LOW | No creation date |
### 2. Reputation Score (30% weight)
| Condiție | Score | Weight |
|----------|-------|--------|
| Blacklisted | +100 | Critical |
| VirusTotal malicious > 5 | +80 | High |
| VirusTotal suspicious > 10 | +50 | Medium |
| Typosquatting detected | +70 | High |
| No reputation data | +20 | Low |
| Clean reputation | 0 | None |
### 3. SSL Score (20% weight)
| Condiție | Score |
|----------|-------|
| No SSL certificate | +100 |
| Self-signed certificate | +80 |
| Expired certificate | +100 |
| Certificate < 30 days old | +40 |
| Expires < 30 days | +30 |
| Valid certificate | 0 |
### 4. DNS Score (10% weight)
| Condiție | Score |
|----------|-------|
| No MX records | +30 |
| No TXT records (SPF/DKIM) | +20 |
| Suspicious NS records | +40 |
| Recently changed NS | +50 |
| Complete DNS config | 0 |
### 5. WHOIS Score (10% weight)
| Condiție | Score |
|----------|-------|
| WHOIS privacy protection | +30 |
| No registrant org | +20 |
| Country mismatch | +20 |
| Known abusive registrar | +50 |
| Transparent WHOIS | 0 |
### Risk Level Classification:
```python
if total_score <= 30: return "LOW"
elif total_score <= 60: return "MEDIUM"
elif total_score <= 85: return "HIGH"
else: return "CRITICAL"
```
---
## 🎯 CE FUNCȚIONEAZĂ vs CE NU
### ✅ FUNCȚIONAL ACUM:
1. ✅ Health check endpoint
2. ✅ Domain check cu WHOIS
3. ✅ Risk scoring (domain age + whois factors)
4. ✅ Database storage (domains, whois_records, risk_assessments, check_history)
5. ✅ Swagger UI documentation
6. ✅ Docker Compose infrastructure
7. ✅ PostgreSQL cu toate tabelele
8. ✅ Error handling și logging
### ⏸️ PREGĂTIT DAR NEFUNCȚIONAL:
1. ⏸️ DNS checking
2. ⏸️ SSL certificate validation
3. ⏸️ VirusTotal reputation
4. ⏸️ Redis caching
5. ⏸️ Batch processing
6. ⏸️ Domain history endpoint
7. ⏸️ Search endpoint
8. ⏸️ Statistics endpoint
### ❌ NEPREGĂTIT:
1. ❌ Frontend dashboard
2. ❌ Email validation
3. ❌ Subdomain enumeration
4. ❌ IP geolocation
5. ❌ Traffic estimation
6. ❌ Webhook notifications
---
## 🧪 SCRIPT DE TESTARE COMPLETĂ
Salvează ca `test-all.sh`:
```bash
#!/bin/bash
echo "=== Testing Domain Check API ==="
echo ""
# Test 1: Health
echo "1. Health Check:"
curl -s http://localhost:5000/health | python3 -m json.tool
echo ""
# Test 2: Root
echo "2. API Root:"
curl -s http://localhost:5000/ | python3 -m json.tool
echo ""
# Test 3: Check google.com
echo "3. Check google.com:"
curl -s -X POST http://localhost:5000/api/v1/check \
-H "Content-Type: application/json" \
-d '{"domain": "google.com"}' \
| python3 -m json.tool
echo ""
# Test 4: Check facebook.com
echo "4. Check facebook.com:"
curl -s -X POST http://localhost:5000/api/v1/check \
-H "Content-Type: application/json" \
-d '{"domain": "facebook.com"}' \
| python3 -m json.tool
echo ""
echo "=== Tests Complete ==="
```
Rulează:
```bash
chmod +x test-all.sh
./test-all.sh
```
---
**Ultima actualizare:** 2026-01-29 15:50:00

452
ai_platform/modules/domain_check/NEXT_STEPS.md Normal file
View file

@ -0,0 +1,452 @@
# NEXT STEPS - Plan de Dezvoltare
## 🎯 PRIORITĂȚI DEZVOLTARE
### PHASE 1: Complete Backend Features (URGENT) ⚡
**Estimare: 2-3 ore**
1. ✅ **DNS Checking Service** - ESENȚIAL pentru risk scoring
2. ✅ **SSL Certificate Validation** - ESENȚIAL pentru risk scoring
3. ✅ **Whoxy API Fallback** - Activare automată când python-whois eșuează
4. ✅ **Implementare Endpoints:**
- `GET /api/v1/domain/{domain}` - Detalii + istoric
- `GET /api/v1/stats` - Statistici sistem
- `GET /api/v1/search` - Căutare domenii
5. ✅ **Redis Caching** - Activare completă
---
### PHASE 2: Modern Dashboard (NEXT.JS 15) 🚀
**Estimare: 4-6 ore**
#### Stack Tehnologic 2026:
**Frontend:**
- **Next.js 15** (App Router, Server Components, Server Actions)
- **React 19** (Concurrent features, Suspense)
- **TypeScript** (Type safety)
- **Tailwind CSS** + **shadcn/ui** (Modern UI components)
- **TanStack Query (React Query v5)** - Data fetching & caching
- **Zustand** - State management (lightweight)
- **Recharts** sau **Tremor** - Data visualization
- **Framer Motion** - Animations
**Features Dashboard:**
1. **🏠 Home Page**
- Hero section cu search bar central
- Live stats (total checks, domains analyzed)
- Recent high-risk domains feed
- Trending searches
2. **🔍 Domain Check Page**
- Input field cu autocomplete
- Real-time validation
- Loading states cu skeleton
- Result cards cu:
- Risk score gauge (circular progress)
- WHOIS data table
- DNS records expandable
- SSL certificate timeline
- Historical checks graph
- Export options (PDF, JSON)
3. **📊 Analytics Dashboard**
- Risk distribution pie chart
- Daily checks timeline
- Top risky domains table
- Geographic distribution map (registrant countries)
- Registrar statistics
- Average processing time metrics
4. **📜 History Browser**
- Filterable table (domain, risk level, date)
- Pagination
- Quick re-check button
- Comparison mode (2 domains side-by-side)
5. **⚙️ Settings**
- API configuration
- Threshold customization
- Export preferences
- Dark/Light mode toggle
6. **📖 Documentation**
- API reference (embedded Swagger)
- Risk scoring explanation
- Integration examples
- FAQ
---
### PHASE 3: Advanced Features 🎯
**Estimare: 6-8 ore**
1. **Batch Processing** - Verificare 100+ domenii simultan
2. **VirusTotal Integration** - Reputation checking
3. **Real-time Webhooks** - Notificări pentru high-risk
4. **Email Validation** - Verificare email addresses
5. **Subdomain Enumeration** - Discover subdomains
6. **IP Geolocation** - Hartă interactivă
---
## 📁 STRUCTURA PROIECT DASHBOARD
```
domain-check/
├── frontend/ # Next.js App
│ ├── app/
│ │ ├── (dashboard)/
│ │ │ ├── layout.tsx
│ │ │ ├── page.tsx # Home
│ │ │ ├── check/
│ │ │ │ └── page.tsx # Domain Check
│ │ │ ├── analytics/
│ │ │ │ └── page.tsx # Analytics
│ │ │ ├── history/
│ │ │ │ └── page.tsx # History
│ │ │ └── settings/
│ │ │ └── page.tsx # Settings
│ │ ├── api/ # API Routes (Next.js)
│ │ │ └── proxy/
│ │ │ └── [...path]/route.ts # Proxy to Flask
│ │ ├── layout.tsx # Root layout
│ │ └── page.tsx # Landing page
│ ├── components/
│ │ ├── ui/ # shadcn components
│ │ ├── domain/
│ │ │ ├── DomainSearchBar.tsx
│ │ │ ├── RiskScoreGauge.tsx
│ │ │ ├── WhoisDataCard.tsx
│ │ │ └── DNSRecordsTable.tsx
│ │ ├── charts/
│ │ │ ├── RiskDistribution.tsx
│ │ │ └── TimelineChart.tsx
│ │ └── layout/
│ │ ├── Navbar.tsx
│ │ ├── Sidebar.tsx
│ │ └── Footer.tsx
│ ├── lib/
│ │ ├── api.ts # API client
│ │ ├── types.ts # TypeScript types
│ │ └── utils.ts # Helper functions
│ ├── hooks/
│ │ ├── useDomainCheck.ts
│ │ ├── useStats.ts
│ │ └── useHistory.ts
│ ├── store/
│ │ └── store.ts # Zustand store
│ ├── public/
│ ├── tailwind.config.ts
│ ├── next.config.mjs
│ ├── package.json
│ └── tsconfig.json
├── api/ # Flask Backend (existing)
└── docker-compose.yml # Add frontend service
```
---
## 🚀 COMENZI DEZVOLTARE
### Setup Frontend
```bash
cd /home/admin365/domain-check
# Create Next.js app
npx create-next-app@latest frontend \
--typescript \
--tailwind \
--app \
--src-dir false \
--import-alias "@/*"
cd frontend
# Install dependencies
npm install @tanstack/react-query zustand
npm install recharts date-fns lucide-react
npm install @radix-ui/react-dialog @radix-ui/react-dropdown-menu
npm install class-variance-authority clsx tailwind-merge
# Install shadcn/ui
npx shadcn-ui@latest init
npx shadcn-ui@latest add button card input table badge progress
npx shadcn-ui@latest add dialog dropdown-menu tooltip
# Development
npm run dev
```
### Docker Integration
Update `docker-compose.yml`:
```yaml
dns_frontend:
build:
context: ./frontend
dockerfile: Dockerfile
container_name: dns_frontend
networks:
- dns-network
environment:
- NEXT_PUBLIC_API_URL=http://dns_api:5000
volumes:
- ./frontend:/app
- /app/node_modules
- /app/.next
ports:
- "3000:3000"
depends_on:
- dns_api
restart: unless-stopped
command: npm run dev
```
---
## 🎨 UI/UX DESIGN GUIDELINES
### Color Scheme
```css
/* Risk Levels */
--risk-low: #10b981 /* Green */
--risk-medium: #f59e0b /* Amber */
--risk-high: #ef4444 /* Red */
--risk-critical: #dc2626 /* Dark Red */
/* Brand */
--primary: #3b82f6 /* Blue */
--secondary: #8b5cf6 /* Purple */
```
### Components Style
- **Modern:** Rounded corners (radius-lg)
- **Clean:** Generous whitespace
- **Responsive:** Mobile-first design
- **Accessible:** WCAG 2.1 AA compliant
- **Fast:** Optimistic UI updates
- **Smooth:** 60fps animations
---
## 📊 EXEMPLE COMPONENTE
### 1. Risk Score Gauge
```tsx
import { Progress } from "@/components/ui/progress"
export function RiskScoreGauge({ score, level }: Props) {
const color = {
LOW: "text-green-500",
MEDIUM: "text-amber-500",
HIGH: "text-red-500",
CRITICAL: "text-red-700"
}[level]
return (
<div className="space-y-2">
<div className="flex justify-between">
<span className="text-sm font-medium">Risk Score</span>
<span className={`text-2xl font-bold ${color}`}>{score}/100</span>
</div>
<Progress value={score} className="h-2" />
<span className={`text-xs font-semibold ${color}`}>{level}</span>
</div>
)
}
```
### 2. Domain Search Bar
```tsx
"use client"
import { useState } from "react"
import { useMutation } from "@tanstack/react-query"
import { Input } from "@/components/ui/input"
import { Button } from "@/components/ui/button"
export function DomainSearchBar() {
const [domain, setDomain] = useState("")
const checkMutation = useMutation({
mutationFn: (domain: string) =>
fetch("/api/v1/check", {
method: "POST",
body: JSON.stringify({ domain })
}).then(r => r.json())
})
return (
<div className="flex gap-2">
<Input
placeholder="Enter domain (e.g., example.com)"
value={domain}
onChange={(e) => setDomain(e.target.value)}
/>
<Button
onClick={() => checkMutation.mutate(domain)}
disabled={checkMutation.isPending}
>
{checkMutation.isPending ? "Checking..." : "Check Domain"}
</Button>
</div>
)
}
```
---
## 🔄 API CLIENT (TypeScript)
```typescript
// lib/api.ts
export interface DomainCheckRequest {
domain: string
check_options?: {
whois?: boolean
dns?: boolean
ssl?: boolean
reputation?: boolean
force_refresh?: boolean
}
}
export interface RiskScore {
total: number
level: "LOW" | "MEDIUM" | "HIGH" | "CRITICAL"
factors: Array<{
factor: string
score: number
weight: number
reason: string
}>
}
export interface DomainCheckResponse {
success: boolean
data: {
domain: string
check_id: string
timestamp: string
whois: any
dns: any
ssl: any
reputation: any
risk_score: RiskScore
}
metadata: {
cached: boolean
processing_time_ms: number
api_version: string
}
}
export class DomainCheckAPI {
private baseURL: string
constructor(baseURL: string = "http://localhost:5000") {
this.baseURL = baseURL
}
async checkDomain(request: DomainCheckRequest): Promise<DomainCheckResponse> {
const response = await fetch(`${this.baseURL}/api/v1/check`, {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify(request)
})
return response.json()
}
async getDomainDetails(domain: string) {
const response = await fetch(`${this.baseURL}/api/v1/domain/${domain}`)
return response.json()
}
async getStats() {
const response = await fetch(`${this.baseURL}/api/v1/stats`)
return response.json()
}
async searchDomains(params: {
q?: string
risk_level?: string
page?: number
}) {
const query = new URLSearchParams(params as any)
const response = await fetch(`${this.baseURL}/api/v1/search?${query}`)
return response.json()
}
}
export const api = new DomainCheckAPI()
```
---
## 📋 CHECKLIST IMPLEMENTARE
### Backend (Priority 1)
- [ ] Implementează DNS checking service
- [ ] Implementează SSL certificate validation
- [ ] Activează Whoxy API fallback
- [ ] Implementează endpoint `GET /api/v1/domain/{domain}`
- [ ] Implementează endpoint `GET /api/v1/stats`
- [ ] Implementează endpoint `GET /api/v1/search`
- [ ] Activează Redis caching complet
- [ ] Testează toate endpoint-urile
### Frontend Setup (Priority 2)
- [ ] Create Next.js app cu TypeScript
- [ ] Setup Tailwind CSS + shadcn/ui
- [ ] Configure TanStack Query
- [ ] Setup Zustand store
- [ ] Create API client library
- [ ] Setup Docker integration
### Frontend Components (Priority 3)
- [ ] Layout components (Navbar, Sidebar)
- [ ] Domain Search Bar
- [ ] Risk Score Gauge
- [ ] WHOIS Data Card
- [ ] DNS Records Table
- [ ] SSL Certificate Timeline
- [ ] Charts (Pie, Line, Bar)
- [ ] History Table
- [ ] Stats Dashboard
### Integration (Priority 4)
- [ ] Connect frontend cu backend API
- [ ] Implement error handling
- [ ] Add loading states
- [ ] Add toast notifications
- [ ] Test end-to-end flow
---
## 🎯 NEXT IMMEDIATE ACTIONS
**CE VREI SĂ FAC ACUM?**
### Option A: Complete Backend First (RECOMANDAT)
✅ Implementez DNS + SSL + endpoint-uri rămase
✅ Backend 100% funcțional
⏱️ Estimare: 2-3 ore
### Option B: Start Frontend Direct
🚀 Creez Next.js app cu structura completă
🎨 UI modern cu components
⏱️ Estimare: 4-6 ore
### Option C: Both in Parallel
⚡ DNS + SSL în background
🎨 Next.js setup simultan
⏱️ Estimare: 4-5 ore
**CE ALEGI? (A, B sau C)**
---
**Document creat:** 2026-01-29 16:00:00

740
ai_platform/modules/domain_check/README.md Normal file
View file

@ -0,0 +1,740 @@
# Domain Check API - Anti-Fake News Platform
Sistema de verificare domenii pentru detectarea dezinformarii prin analiza vechimii domeniilor si riscurilor asociate.
## 🚀 Quick Start - API Usage
### Endpoints Disponibile
**Base URL:** `http://domain-check-api:11000`
**Endpoint principal:** `POST /api/v1/check/check`
### Apel Complet - Toate Datele Posibile
Pentru a obține **toate datele disponibile** despre un domeniu:
```bash
curl -X POST "http://domain-check-api:11000/api/v1/check/check" \
-H "Content-Type: application/json" \
-d '{
"domain": "example.com",
"check_options": {
"whois": true,
"dns": true,
"ssl": true,
"ip_intelligence": true,
"http_analysis": true,
"blacklist": true,
"port_scan": true,
"subdomains": true,
"force_refresh": false
}
}'
```
**Răspuns:** JSON cu:
- `whois` - Date WHOIS/RDAP complete (vârstă domeniu, registrar, name servers, DNSSEC)
- `dns` - Toate recordurile DNS (A, AAAA, MX, TXT, NS, CNAME, SOA)
- `ssl` - Certificate SSL/TLS details (issuer, validity, chain, self-signed status)
- `ip_intelligence` - Geolocation, ASN, ISP, reverse DNS pentru toate IP-urile
- `http_analysis` - HTTP headers, security headers, redirects, technologies detected
- `blacklist` - Verificare în DNSBL (spam, malware, phishing blacklists)
- `port_scan` - Port scanning TCP common (22, 80, 443, 8080, etc.)
- `subdomains` - Subdomain enumeration (www, mail, ftp, api, etc.)
- `risk_score` - Scor de risc calculat (0-100) cu breakdown pe categorii
### Apel Rapid - Date Esențiale
Pentru verificări rapide (doar WHOIS + DNS + SSL):
```bash
curl -X POST "http://domain-check-api:11000/api/v1/check/check" \
-H "Content-Type: application/json" \
-d '{
"domain": "example.com",
"check_options": {
"whois": true,
"dns": true,
"ssl": true
}
}'
```
### Health Check
```bash
curl http://domain-check-api:11000/health
```
Răspuns:
```json
{
"status": "healthy",
"database": "connected",
"redis": "connected",
"environment": "production",
"version": "v1"
}
```
---
## 🌐 Integrare ca Serviciu (Kong API Gateway / Microservices)
### Configurare ca Backend Service
API-ul poate fi folosit ca **backend service** pentru Kong, NGINX, Traefik sau alte API gateways.
#### 1. Configurare Kong Gateway
```bash
# Adaugă service
curl -X POST http://kong-admin:8001/services \
--data name=domain-check-api \
--data url=http://domain-check-api:11000
# Adaugă route
curl -X POST http://kong-admin:8001/services/domain-check-api/routes \
--data paths[]=/domain-check \
--data strip_path=false
# Activează rate limiting
curl -X POST http://kong-admin:8001/services/domain-check-api/plugins \
--data name=rate-limiting \
--data config.minute=100 \
--data config.hour=1000
```
Acum API-ul e disponibil la: `http://your-kong-gateway/domain-check/api/v1/check/check`
#### 2. Configurare NGINX Reverse Proxy
```nginx
upstream domain_check_backend {
server domain-check-api:11000;
}
server {
listen 80;
server_name api.example.com;
location /domain-check/ {
proxy_pass http://domain_check_backend/;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
# Timeouts pentru operații lungi (port scan, subdomain enum)
proxy_read_timeout 300s;
proxy_connect_timeout 10s;
}
}
```
#### 3. Docker Swarm / Kubernetes Service
**Docker Swarm:**
```yaml
version: '3.8'
services:
domain-check:
image: didiai-domain-check:latest
deploy:
replicas: 3
update_config:
parallelism: 1
delay: 10s
environment:
DATABASE_URL: postgresql://user:pass@db-cluster:5432/domain_check
REDIS_URL: redis://redis-cluster:6379/0
networks:
- backend
```
**Kubernetes Service:**
```yaml
apiVersion: v1
kind: Service
metadata:
name: domain-check-api
spec:
selector:
app: domain-check
ports:
- protocol: TCP
port: 80
targetPort: 51000
type: ClusterIP
```
#### 4. Environment Variables pentru External DB
Toate parametrii DB sunt în `.env` și pot fi schimbați pentru a folosi un **cluster PostgreSQL extern**:
```bash
# .env - Configurare pentru DB Cluster extern
DB_HOST=postgres-cluster.example.com
DB_PORT=5432
DB_USER=domain_check_user
DB_PASSWORD=secure_password_here
DB_NAME=domain_check
# Redis Cluster extern
REDIS_HOST=redis-cluster.example.com
REDIS_PORT=6379
REDIS_PASSWORD=redis_password_here
```
API-ul se va conecta automat la DB-ul extern specificat în variabilele de mediu.
---
## 📊 Database Schema - Structura Tabelelor
API-ul folosește PostgreSQL cu următoarele tabele:
### 1. `domains` - Domenii Verificate
Tabela principală cu toate domeniile procesate.
| Coloană | Tip | Descriere |
|---------|-----|-----------|
| `id` | UUID | Primary key |
| `domain` | VARCHAR(255) | Domeniul principal (ex: `example` din `example.com`) |
| `subdomain` | VARCHAR(255) | Subdomenul (ex: `www` din `www.example.com`) |
| `tld` | VARCHAR(50) | Top Level Domain (ex: `com`, `org`, `ro`) |
| `full_domain` | VARCHAR(255) | Domeniul complet (UNIQUE) |
| `first_seen_at` | TIMESTAMP | Prima verificare |
| `last_checked_at` | TIMESTAMP | Ultima verificare |
| `check_count` | INTEGER | Număr total verificări |
| `is_active` | BOOLEAN | Status activ/inactiv |
| `created_at` | TIMESTAMP | Data creării înregistrării |
| `updated_at` | TIMESTAMP | Data ultimei actualizări |
**Indexes:** `full_domain` (UNIQUE), `domain`, `tld`, `last_checked_at`, `is_active`
### 2. `whois_records` - Date WHOIS/RDAP
Stochează informații WHOIS pentru fiecare domeniu.
| Coloană | Tip | Descriere |
|---------|-----|-----------|
| `id` | UUID | Primary key |
| `domain_id` | UUID | Foreign key → `domains(id)` |
| `creation_date` | TIMESTAMP | Data creării domeniului |
| `expiration_date` | TIMESTAMP | Data expirării |
| `updated_date` | TIMESTAMP | Ultima actualizare WHOIS |
| `registrar` | VARCHAR(255) | Registrar (ex: GoDaddy, Namecheap) |
| `registrar_url` | VARCHAR(500) | URL registrar |
| `registrant_org` | VARCHAR(255) | Organizația proprietarului |
| `registrant_country` | VARCHAR(2) | Țara (cod ISO) |
| `admin_email` | VARCHAR(255) | Email administrator |
| `name_servers` | TEXT[] | Array cu name servers |
| `status` | TEXT[] | Array cu statusuri domeniu |
| `dnssec` | BOOLEAN | DNSSEC activat/nu |
| `raw_whois_data` | JSONB | Date WHOIS raw (JSON) |
| `data_source` | VARCHAR(50) | Sursa datelor (`rdap`, `whoxy`, `manual`) |
| `fetched_at` | TIMESTAMP | Când au fost preluate datele |
**Indexes:** `domain_id`, `creation_date`, `registrar`, `data_source`, `fetched_at`
### 3. `dns_records` - Înregistrări DNS
Toate recordurile DNS pentru fiecare domeniu.
| Coloană | Tip | Descriere |
|---------|-----|-----------|
| `id` | UUID | Primary key |
| `domain_id` | UUID | Foreign key → `domains(id)` |
| `record_type` | VARCHAR(10) | Tip record (`A`, `AAAA`, `MX`, `TXT`, `NS`, `CNAME`, `SOA`) |
| `record_value` | TEXT | Valoarea recordului |
| `ttl` | INTEGER | Time To Live (secunde) |
| `priority` | INTEGER | Prioritate (pentru MX records) |
| `fetched_at` | TIMESTAMP | Când a fost preluat recordul |
**Indexes:** `domain_id`, `(domain_id, record_type)`, `record_type`, `fetched_at`
### 4. `ssl_certificates` - Certificate SSL/TLS
Informații despre certificatele SSL ale domeniilor.
| Coloană | Tip | Descriere |
|---------|-----|-----------|
| `id` | UUID | Primary key |
| `domain_id` | UUID | Foreign key → `domains(id)` |
| `issuer` | VARCHAR(255) | Emitent certificat (ex: Let's Encrypt) |
| `subject` | VARCHAR(255) | Subiect certificat |
| `valid_from` | TIMESTAMP | Valabil de la |
| `valid_until` | TIMESTAMP | Valabil până la |
| `serial_number` | VARCHAR(255) | Număr serial certificat |
| `signature_algorithm` | VARCHAR(100) | Algoritm semnătură (ex: SHA256) |
| `key_size` | INTEGER | Mărime cheie (2048, 4096 bits) |
| `is_wildcard` | BOOLEAN | Certificat wildcard (`*.example.com`) |
| `is_self_signed` | BOOLEAN | Self-signed certificate |
| `is_valid` | BOOLEAN | Certificat valid |
| `certificate_chain` | JSONB | Lanțul complet de certificate (JSON) |
| `fetched_at` | TIMESTAMP | Când a fost verificat certificatul |
**Indexes:** `domain_id`, `valid_until`, `is_valid`, `fetched_at`
### Relații între Tabele
```
domains (1) ─── (N) whois_records
├─ (N) dns_records
└─ (N) ssl_certificates
```
Toate relațiile au `ON DELETE CASCADE` - ștergerea unui domeniu șterge automat toate datele asociate.
### Migrare la DB Cluster Extern
Pentru a crea schema în **cluster PostgreSQL nou**:
```bash
# 1. Export schema
docker exec domain_check_postgres pg_dump -U dns_admin -d domain_check --schema-only > schema.sql
# 2. Import în cluster nou
psql -h postgres-cluster.example.com -U admin -d domain_check < schema.sql
# 3. Actualizează .env cu noile credentials
DB_HOST=postgres-cluster.example.com
DB_PORT=5432
DB_USER=domain_check_user
DB_PASSWORD=new_secure_password
DB_NAME=domain_check
# 4. Restart API
docker compose restart domain_check_api
```
Sau folosește direct script-ul de inițializare:
```bash
psql -h postgres-cluster.example.com -U admin -d domain_check < init-scripts/01-init-db.sql
```
---
## Caracteristici
- Verificare WHOIS/RDAP complet
- Integrare Whoxy API
- DNS records checking (A, AAAA, MX, TXT, NS, CNAME, SOA)
- SSL certificate validation
- IP Intelligence (geolocation, ASN, reverse DNS)
- HTTP Security Analysis
- DNSBL Blacklist checking
- Port scanning
- Subdomain enumeration
- Risk scoring engine avansat
- PostgreSQL + Redis caching
- Swagger UI + ReDoc documentation
- Docker Compose setup complet
- GitLab CI/CD pipeline cu SAST
---
## Deployment pe Orice Server
### Metoda 1: Script Automat (Recomandat)
```bash
# Descarca si ruleaza scriptul de deployment
curl -sSL (modul domain_check al platformei DIDI) -o deploy.sh
chmod +x deploy.sh
./deploy.sh install
```
Scriptul va:
- Verifica prerequisitele (Docker, Docker Compose, Git)
- Clona repository-ul
- Genera fisierul `.env` cu parole securizate
- Construi si porni containerele
### Metoda 2: Manual
```bash
# 1. Cloneaza repository-ul
git clone <didi-lot1-ai>/ai_platform/modules/domain_check
cd domain-check
# 2. Copiaza si editeaza configuratia
cp .env.example .env
nano .env # sau vim, sau orice editor
# 3. Porneste serviciile
docker compose up -d --build
# 4. Verifica statusul
docker compose ps
curl http://domain-check-api:11000/health
```
### Metoda 3: Din GitLab Container Registry
```bash
# Login la registry (daca e necesar)
docker login <registry-didi>
# Pull imaginea
docker pull didiai-domain-check:latest
# Cloneaza doar fisierele de configurare
git clone <didi-lot1-ai>/ai_platform/modules/domain_check
cd domain-check
# Editeaza .env
nano .env
# Porneste cu imaginea din registry
docker compose up -d
```
---
## Configurare pentru Serverul Tau
### Fisierul .env
Toate setarile sunt in `.env`. Editeaza pentru mediul tau:
```bash
# ===========================================
# PORTURI - Modifica dupa nevoie
# ===========================================
# Schema: Dev=5xxxx, Prod=1xxxx
# x1xxx = API/Gateway
# x20xx = PostgreSQL
# x23xx = Redis
API_PORT=51000 # Portul API-ului (schimba la 11000 pentru prod)
DB_EXTERNAL_PORT=52000 # Port extern PostgreSQL (12000 pentru prod)
REDIS_EXTERNAL_PORT=52300 # Port extern Redis (12300 pentru prod)
# ===========================================
# DATABASE - Genereaza parole noi!
# ===========================================
DB_PASSWORD=SCHIMBA_ACEASTA_PAROLA
DB_USER=dns_admin
DB_NAME=domain_check
# ===========================================
# API KEYS - Adauga cheile tale
# ===========================================
WHOXY_API_KEY=cheia_ta_whoxy
VIRUSTOTAL_API_KEY=cheia_ta_virustotal
# ===========================================
# SECURITATE - OBLIGATORIU pentru productie!
# ===========================================
SECRET_KEY=genereaza_un_string_random_de_32_caractere
FLASK_ENV=production # development sau production
DEBUG=False # True doar pentru development
```
### Setare Hostname/DNS
API-ul va fi accesibil prin:
- **IP direct**: `http://<IP_SERVER>:<API_PORT>/`
- **Hostname**: `http://<hostname>:<API_PORT>/`
Pentru a seta un hostname custom:
```bash
# 1. Pe DNS server (daca ai control)
check-dns.example.com A <IP_SERVER>
# 2. Sau in /etc/hosts pe clienti
echo "<IP_SERVER> check-dns.example.com" >> /etc/hosts
# 3. Sau cu reverse proxy (nginx/traefik)
# - Configureaza proxy pass catre domain-check-api:<API_PORT>
```
### Porturi Utilizate
| Serviciu | Port Default (Dev) | Port Productie | Variabila |
|----------|-------------------|----------------|-----------|
| API Flask | 51000 | 11000 | `API_PORT` |
| PostgreSQL | 52000 | 12000 | `DB_EXTERNAL_PORT` |
| Redis | 52300 | 12300 | `REDIS_EXTERNAL_PORT` |
Porturile pot fi schimbate in `.env` - nu este nevoie de modificari in alte fisiere.
### Firewall
Deschide doar portul API pentru acces extern:
```bash
# UFW (Ubuntu/Debian)
sudo ufw allow 51000/tcp
# firewalld (CentOS/RHEL)
sudo firewall-cmd --add-port=51000/tcp --permanent
sudo firewall-cmd --reload
# iptables
sudo iptables -A INPUT -p tcp --dport 51000 -j ACCEPT
```
PostgreSQL si Redis nu trebuie expuse public (sunt doar pentru comunicatie interna).
---
## Structura Proiectului
```
domain-check/
├── api/ # Flask API
│ ├── app/
│ │ ├── models/ # SQLAlchemy models
│ │ ├── routes/ # API endpoints
│ │ ├── services/ # Business logic (WHOIS, DNS, SSL, etc.)
│ │ ├── utils/ # Helper functions
│ │ ├── tasks/ # Celery tasks
│ │ ├── static/ # Dashboard HTML/CSS/JS
│ │ ├── config.py # Configuration
│ │ └── __init__.py # App factory
│ ├── Dockerfile
│ ├── requirements.txt
│ └── run.py # Entry point
├── init-scripts/ # PostgreSQL init scripts
│ └── 01-init-db.sql
├── docker-compose.yml # Docker orchestration
├── .gitlab-ci.yml # CI/CD pipeline
├── deploy.sh # Deployment script
├── .env # Environment variables
├── use_api.md # API documentation
└── README.md # This file
```
---
## Servicii Docker
| Container | Descriere | Health Check |
|-----------|-----------|--------------|
| `domain_check_api` | Flask REST API + Dashboard | `/health` |
| `domain_check_postgres` | PostgreSQL 15 database | `pg_isready` |
| `domain_check_redis` | Redis cache | `redis-cli ping` |
| `domain_check_worker` | Celery background worker | `celery inspect ping` |
---
## Utilizare API
### Interfata Web (Dashboard)
Deschide in browser: `http://<server>:<API_PORT>/`
### Verificare Domain (curl)
```bash
# Check complet
curl -X POST "http://domain-check-api:11000/api/v1/check/check" \
-H "Content-Type: application/json" \
-d '{
"domain": "example.com",
"check_options": {
"whois": true,
"dns": true,
"ssl": true,
"ip_intelligence": true,
"http_analysis": true,
"blacklist": true,
"port_scan": true,
"subdomains": true
}
}'
```
### Health Check
```bash
curl http://domain-check-api:11000/health
```
### Documentatie Swagger
- Swagger UI: `http://<server>:<API_PORT>/docs`
- ReDoc: `http://<server>:<API_PORT>/redoc`
Pentru documentatie API completa, vezi [use_api.md](use_api.md).
---
## CI/CD Pipeline
Pipeline-ul GitLab include:
### Stages
1. **lint** - Code quality (flake8, black, isort, hadolint)
2. **security** - SAST scanning (bandit, safety, trivy)
3. **test** - Unit tests
4. **build** - Docker build & push to registry
5. **release** - Create deployment bundle
6. **deploy** - Deploy to server via SSH
### Container Registry
Imaginile sunt publicate la:
```
didiai-domain-check:latest
didiai-domain-check:<branch>
```
### Configurare Deploy Automat
Adauga in GitLab CI/CD Variables:
| Variable | Descriere |
|----------|-----------|
| `DEPLOY_HOST` | IP sau hostname server |
| `DEPLOY_USER` | User SSH (ex: admin365) |
| `SSH_PRIVATE_KEY` | Cheia privata SSH |
| `SSH_KNOWN_HOSTS` | Output din `ssh-keyscan <server>` |
---
## Comenzi Utile
### Deploy Script
```bash
./deploy.sh install # Instalare fresh
./deploy.sh update # Update la ultima versiune
./deploy.sh status # Verifica statusul
./deploy.sh logs # Vezi toate logs
./deploy.sh logs domain_check_api # Logs doar pentru API
./deploy.sh start # Porneste serviciile
./deploy.sh stop # Opreste serviciile
./deploy.sh restart # Restart servicii
```
### Docker Direct
```bash
# Start/Stop
docker compose up -d
docker compose down
# Rebuild
docker compose up -d --build
# Logs
docker compose logs -f domain_check_api
# Shell in container
docker exec -it domain_check_api bash
# Database access
docker exec -it domain_check_postgres psql -U dns_admin -d domain_check
```
---
## Troubleshooting
### Container nu porneste
```bash
# Verifica logs
docker compose logs domain_check_api
# Verifica network
docker network ls | grep dns-network
# Recreate containers
docker compose down && docker compose up -d --build
```
### Database connection error
```bash
# Verifica ca PostgreSQL e healthy
docker compose ps domain_check_postgres
# Verifica health
docker exec -it domain_check_postgres pg_isready -U dns_admin
# Restart database
docker compose restart domain_check_postgres
```
### API returns 500 error
```bash
# Check logs
docker compose logs -f domain_check_api
# Verify database schema
docker exec -it domain_check_postgres psql -U dns_admin -d domain_check -c "\dt"
```
### Port deja in uz
```bash
# Gaseste ce foloseste portul
sudo lsof -i :51000
# Schimba portul in .env
API_PORT=51001 # sau alt port liber
```
---
## Securitate - Checklist Productie
- [ ] Schimba toate parolele default in `.env`
- [ ] Seteaza `FLASK_ENV=production`
- [ ] Seteaza `DEBUG=False`
- [ ] Genereaza `SECRET_KEY` random (32+ caractere)
- [ ] Configureaza firewall (doar port API public)
- [ ] Setup HTTPS cu reverse proxy (nginx/traefik)
- [ ] Backup automat PostgreSQL
- [ ] Monitorizare logs
---
## Variabile de Mediu Complete
| Variabila | Default | Descriere |
|-----------|---------|-----------|
| `API_PORT` | 51000 | Port pentru Flask API |
| `API_HOST` | 0.0.0.0 | Bind address |
| `API_VERSION` | v1 | API version prefix |
| `DB_HOST` | domain_check_postgres | PostgreSQL hostname (container) |
| `DB_PORT` | 5432 | PostgreSQL port intern |
| `DB_EXTERNAL_PORT` | 52000 | PostgreSQL port extern |
| `DB_USER` | dns_admin | Database user |
| `DB_PASSWORD` | - | Database password |
| `DB_NAME` | domain_check | Database name |
| `REDIS_HOST` | domain_check_redis | Redis hostname (container) |
| `REDIS_PORT` | 6379 | Redis port intern |
| `REDIS_EXTERNAL_PORT` | 52300 | Redis port extern |
| `REDIS_DB` | 0 | Redis database number |
| `WHOXY_API_KEY` | - | Whoxy API key |
| `VIRUSTOTAL_API_KEY` | - | VirusTotal API key |
| `FLASK_ENV` | development | Flask environment |
| `DEBUG` | True | Debug mode |
| `SECRET_KEY` | - | Flask secret key |
| `LOG_LEVEL` | INFO | Logging level |
| `REDIS_TTL_HOT` | 21600 | Cache TTL hot (6h) |
| `REDIS_TTL_WARM` | 86400 | Cache TTL warm (24h) |
| `REDIS_TTL_COLD` | 604800 | Cache TTL cold (7d) |
---
## License
MIT License - Free for use in anti-fake news projects
---
**Built for fighting disinformation**
**Last Updated:** 2026-02-04

68
ai_platform/modules/domain_check/api/Dockerfile Normal file
View file

@ -0,0 +1,68 @@
# Multi-stage build for Python API
# Stage 1: Builder
FROM python:3.10-slim as builder
# Set working directory
WORKDIR /build
# Install system dependencies
RUN apt-get update && apt-get install -y --no-install-recommends \
gcc \
g++ \
make \
libpq-dev \
libssl-dev \
libffi-dev \
&& rm -rf /var/lib/apt/lists/*
# Copy requirements
COPY requirements.txt .
# Create virtual environment and install dependencies
RUN python -m venv /opt/venv
ENV PATH="/opt/venv/bin:$PATH"
RUN pip install --no-cache-dir --upgrade pip setuptools wheel && \
pip install --no-cache-dir -r requirements.txt
# Stage 2: Runtime
FROM python:3.10-slim
# Set working directory
WORKDIR /app
# Install runtime dependencies only
RUN apt-get update && apt-get install -y --no-install-recommends \
libpq5 \
curl \
&& rm -rf /var/lib/apt/lists/*
# Copy virtual environment from builder
COPY --from=builder /opt/venv /opt/venv
# Set environment variables
ENV PATH="/opt/venv/bin:$PATH" \
PYTHONUNBUFFERED=1 \
PYTHONDONTWRITEBYTECODE=1 \
FLASK_APP=run.py
# Create non-root user
RUN useradd -m -u 1000 appuser && \
mkdir -p /app/logs && \
chown -R appuser:appuser /app
# Copy application code
COPY --chown=appuser:appuser . /app/
# Switch to non-root user
USER appuser
# Expose port
EXPOSE 5000
# Health check
HEALTHCHECK --interval=30s --timeout=10s --start-period=40s --retries=3 \
CMD curl -f http://localhost:5000/health || exit 1
# Default command
CMD ["gunicorn", "--bind", "0.0.0.0:5000", "--workers", "4", "--threads", "2", "--timeout", "120", "--worker-class", "gevent", "--access-logfile", "-", "--error-logfile", "-", "run:app"]

422
ai_platform/modules/domain_check/api/app/__init__.py Normal file
View file

@ -0,0 +1,422 @@
"""
Domain Check API - Application Factory
"""
import ipaddress
import logging
import os
from flask import Flask, jsonify, request
from flask_sqlalchemy import SQLAlchemy
from flask_migrate import Migrate
from flask_cors import CORS
from flask_restx import Api
from redis import Redis
from celery import Celery
try:
from flask_limiter import Limiter
from flask_limiter.util import get_remote_address
LIMITER_AVAILABLE = True
except ImportError:
LIMITER_AVAILABLE = False
from app.config import get_config
# Initialize extensions
db = SQLAlchemy()
migrate = Migrate()
redis_client = None
celery_app = Celery(__name__)
limiter = Limiter(key_func=get_remote_address) if LIMITER_AVAILABLE else None
def _is_internal_request() -> bool:
"""Rate-limit exemption: never throttle internal LAN / loopback callers
(the dashboard and server-to-server consumers like ZEUS) or health checks."""
if request.path == '/health':
return True
try:
ip = ipaddress.ip_address(get_remote_address())
return ip.is_private or ip.is_loopback or ip.is_link_local
except (ValueError, TypeError):
return False
def create_app(config_name=None):
"""
Application factory pattern
Args:
config_name: Configuration name (development, production, testing)
Returns:
Flask application instance
"""
if config_name is None:
config_name = os.getenv('FLASK_ENV', 'development')
app = Flask(__name__)
# Load configuration
config = get_config(config_name)
app.config.from_object(config)
config.init_app(app)
# Initialize extensions
init_extensions(app)
# Setup logging
setup_logging(app)
# Register blueprints and routes
register_blueprints(app)
# Register error handlers
register_error_handlers(app)
# Setup Swagger/ReDoc API documentation
setup_api_docs(app)
app.logger.info(f'Domain Check API started in {config_name} mode')
return app
def init_extensions(app):
"""Initialize Flask extensions"""
global redis_client
# Database
db.init_app(app)
migrate.init_app(app, db)
# CORS
CORS(app,
origins=app.config['CORS_ORIGINS'],
allow_headers=app.config['CORS_ALLOW_HEADERS'],
methods=app.config['CORS_METHODS'])
# Redis
try:
redis_client = Redis.from_url(
app.config['REDIS_URL'],
decode_responses=True,
socket_timeout=5,
socket_connect_timeout=5
)
redis_client.ping()
app.logger.info('Redis connection established')
except Exception as e:
app.logger.warning(f'Redis connection failed: {e}. Caching disabled.')
redis_client = None
# Celery
init_celery(app)
# Rate limiting (protects paid WHOIS/VirusTotal quotas from external abuse).
# Internal LAN/loopback traffic is exempt so it never throttles ZEUS or the
# dashboard. Storage errors are swallowed so a Redis hiccup can't 500 the API.
if limiter is not None:
try:
app.config.setdefault('RATELIMIT_STORAGE_URI', app.config['REDIS_URL'])
app.config.setdefault('RATELIMIT_DEFAULT', os.getenv('RATELIMIT_DEFAULT', '240 per minute;5000 per hour'))
app.config.setdefault('RATELIMIT_HEADERS_ENABLED', True)
app.config.setdefault('RATELIMIT_SWALLOW_ERRORS', True)
limiter.init_app(app)
limiter.request_filter(_is_internal_request)
app.logger.info('Rate limiter enabled (internal traffic exempt)')
except Exception as e:
app.logger.warning(f'Rate limiter init failed, continuing without it: {e}')
# Store redis_client in app context
app.redis = redis_client
def init_celery(app):
"""Initialize Celery"""
celery_app.conf.update(
broker_url=app.config['CELERY_BROKER_URL'],
result_backend=app.config['CELERY_RESULT_BACKEND'],
task_serializer=app.config['CELERY_TASK_SERIALIZER'],
result_serializer=app.config['CELERY_RESULT_SERIALIZER'],
accept_content=app.config['CELERY_ACCEPT_CONTENT'],
timezone=app.config['CELERY_TIMEZONE'],
enable_utc=app.config['CELERY_ENABLE_UTC'],
task_track_started=app.config['CELERY_TASK_TRACK_STARTED'],
task_time_limit=app.config['CELERY_TASK_TIME_LIMIT'],
task_soft_time_limit=app.config['CELERY_TASK_SOFT_TIME_LIMIT']
)
class ContextTask(celery_app.Task):
"""Make celery tasks work with Flask app context"""
def __call__(self, *args, **kwargs):
with app.app_context():
return self.run(*args, **kwargs)
celery_app.Task = ContextTask
app.celery = celery_app
return celery_app
def setup_logging(app):
"""Configure application logging"""
log_level = getattr(logging, app.config['LOG_LEVEL'].upper(), logging.INFO)
# Root logger
logging.basicConfig(
level=log_level,
format=app.config['LOG_FORMAT']
)
# App logger
app.logger.setLevel(log_level)
# Disable werkzeug request logs in production
if not app.config['DEBUG']:
logging.getLogger('werkzeug').setLevel(logging.WARNING)
def register_blueprints(app):
"""Register Flask blueprints"""
# Health check endpoint
@app.route('/health')
def health_check():
"""Health check endpoint for Docker and load balancers"""
# Simple health mode - minimal response (recommended for production)
if app.config.get('SIMPLE_HEALTH', False):
try:
from sqlalchemy import text
db.session.execute(text('SELECT 1'))
return jsonify({'status': 'ok'}), 200
except Exception:
return jsonify({'status': 'error'}), 503
# Detailed health mode (development)
health_status = {
'status': 'healthy',
'version': app.config['API_VERSION'],
'environment': os.getenv('FLASK_ENV', 'development')
}
# Check database
try:
from sqlalchemy import text
db.session.execute(text('SELECT 1'))
health_status['database'] = 'connected'
except Exception as e:
health_status['database'] = f'error: {str(e)}'
health_status['status'] = 'unhealthy'
# Check Redis
if app.redis:
try:
app.redis.ping()
health_status['redis'] = 'connected'
except Exception as e:
health_status['redis'] = f'error: {str(e)}'
else:
health_status['redis'] = 'disabled'
status_code = 200 if health_status['status'] == 'healthy' else 503
return jsonify(health_status), status_code
# Root endpoint - serve HTML dashboard
@app.route('/')
def index():
"""Serve the domain check dashboard"""
from flask import send_from_directory
return send_from_directory('static', 'index.html')
# API info endpoint
@app.route('/api')
def api_info():
"""API information endpoint"""
response = {
'name': app.config['API_TITLE'],
'version': app.config['API_VERSION'],
'endpoints': {
'health': '/health',
'api': f'/api/{app.config["API_VERSION"]}',
'dashboard': '/'
}
}
# Only show docs links if swagger is enabled
if not app.config.get('DISABLE_SWAGGER', False):
response['description'] = app.config['API_DESCRIPTION']
response['documentation'] = {
'swagger': '/docs',
'redoc': '/redoc'
}
return jsonify(response)
# Note: Namespaces will be registered in setup_api_docs function
def register_error_handlers(app):
"""Register error handlers"""
@app.errorhandler(404)
def not_found(error):
return jsonify({
'success': False,
'error': {
'code': 'NOT_FOUND',
'message': 'The requested resource was not found',
'status': 404
}
}), 404
@app.errorhandler(500)
def internal_error(error):
app.logger.error(f'Internal server error: {error}')
db.session.rollback()
return jsonify({
'success': False,
'error': {
'code': 'INTERNAL_SERVER_ERROR',
'message': 'An internal server error occurred',
'status': 500
}
}), 500
@app.errorhandler(400)
def bad_request(error):
return jsonify({
'success': False,
'error': {
'code': 'BAD_REQUEST',
'message': str(error),
'status': 400
}
}), 400
@app.errorhandler(429)
def rate_limit_exceeded(error):
return jsonify({
'success': False,
'error': {
'code': 'RATE_LIMIT_EXCEEDED',
'message': 'Too many requests. Please try again later.',
'status': 429
}
}), 429
def setup_api_docs(app):
"""Setup Swagger/ReDoc API documentation"""
# Check if swagger should be disabled (production security)
disable_swagger = app.config.get('DISABLE_SWAGGER', False)
# Create API instance - doc=False disables swagger UI
api = Api(
app,
version=app.config['API_VERSION'],
title=app.config['API_TITLE'],
description=app.config['API_DESCRIPTION'] if not disable_swagger else '',
doc='/docs' if not disable_swagger else False,
prefix=f'/api/{app.config["API_VERSION"]}',
contact=app.config['API_CONTACT'].get('name') if not disable_swagger else None,
contact_email=app.config['API_CONTACT'].get('email') if not disable_swagger else None,
license=app.config['API_LICENSE'].get('name') if not disable_swagger else None,
license_url=app.config['API_LICENSE'].get('url') if not disable_swagger else None,
terms_url=app.config.get('API_TERMS_OF_SERVICE') if not disable_swagger else None,
ordered=True,
validate=True
)
if disable_swagger:
app.logger.info('Swagger/ReDoc documentation disabled (DISABLE_SWAGGER=true)')
# Import and create API models
from app.api_models import create_api_models
models = create_api_models(api)
# Import and register namespaces
try:
from app.routes.check import api as check_ns
# Attach models to namespace
check_ns.models.update(models)
# Register namespace
api.add_namespace(check_ns, path='')
app.logger.info('API namespace registered successfully: check')
except ImportError as e:
app.logger.error(f'Failed to import check namespace: {e}')
except Exception as e:
app.logger.error(f'Failed to register check routes: {e}')
# Try to import other namespaces (if implemented)
try:
from app.routes.domain import api as domain_ns
domain_ns.models.update(models)
api.add_namespace(domain_ns, path='')
app.logger.info('API namespace registered successfully: domain')
except ImportError:
app.logger.debug('domain namespace not yet implemented')
try:
from app.routes.stats import api as stats_ns
stats_ns.models.update(models)
api.add_namespace(stats_ns, path='')
app.logger.info('API namespace registered successfully: stats')
except ImportError:
app.logger.debug('stats namespace not yet implemented')
try:
from app.routes.search import api as search_ns
search_ns.models.update(models)
api.add_namespace(search_ns, path='')
app.logger.info('API namespace registered successfully: search')
except ImportError:
app.logger.debug('search namespace not yet implemented')
try:
from app.routes.batch import api as batch_ns
batch_ns.models.update(models)
api.add_namespace(batch_ns, path='')
app.logger.info('API namespace registered successfully: batch')
except ImportError:
app.logger.debug('batch namespace not yet implemented')
# Add ReDoc endpoint (only if swagger is enabled)
if not disable_swagger:
@app.route('/redoc')
def redoc():
"""ReDoc API documentation"""
return f'''
<!DOCTYPE html>
<html>
<head>
<title>{app.config['API_TITLE']} - ReDoc</title>
<meta charset="utf-8"/>
<meta name="viewport" content="width=device-width, initial-scale=1">
<link href="https://fonts.googleapis.com/css?family=Montserrat:300,400,700|Roboto:300,400,700" rel="stylesheet">
<style>
body {{
margin: 0;
padding: 0;
}}
</style>
</head>
<body>
<redoc spec-url='/api/{app.config["API_VERSION"]}/swagger.json'></redoc>
<script src="https://cdn.redoc.ly/redoc/latest/bundles/redoc.standalone.js"></script>
</body>
</html>
'''
app.api = api
return api
# Create celery app for worker
def create_celery_app(app=None):
"""
Create and configure Celery app
For use in worker process
"""
app = app or create_app()
return app.celery

424
ai_platform/modules/domain_check/api/app/api_models.py Normal file
View file

@ -0,0 +1,424 @@
"""
Flask-RESTX API Models for Swagger Documentation
"""
from flask_restx import fields
def create_api_models(api):
"""
Create and register all API models for Swagger documentation
Args:
api: Flask-RESTX Api instance
Returns:
Dictionary of model names to model objects
"""
# ==================== REQUEST MODELS ====================
check_options_model = api.model('CheckOptions', {
'whois': fields.Boolean(
default=True,
description='Perform WHOIS lookup',
example=True
),
'dns': fields.Boolean(
default=False,
description='Perform DNS record lookup',
example=True
),
'ssl': fields.Boolean(
default=False,
description='Check SSL certificate',
example=False
),
'reputation': fields.Boolean(
default=False,
description='Check domain reputation (VirusTotal, blacklists)',
example=False
),
'force_refresh': fields.Boolean(
default=False,
description='Force refresh, bypass cache',
example=False
)
})
domain_check_request = api.model('DomainCheckRequest', {
'domain': fields.String(
required=True,
description='Domain name to check',
example='google.com',
pattern=r'^[a-zA-Z0-9][a-zA-Z0-9-_.]*\.[a-zA-Z]{2,}$'
),
'check_options': fields.Nested(
check_options_model,
description='Options for what to check',
required=False
)
})
# ==================== RESPONSE MODELS ====================
# WHOIS Response
whois_response = api.model('WhoisResponse', {
'creation_date': fields.String(
description='Domain creation date (ISO 8601)',
example='2024-01-15T10:30:00Z'
),
'expiration_date': fields.String(
description='Domain expiration date (ISO 8601)',
example='2026-01-15T10:30:00Z'
),
'registrar': fields.String(
description='Domain registrar name',
example='MarkMonitor Inc.'
),
'age_days': fields.Integer(
description='Domain age in days',
example=380
),
'status': fields.List(
fields.String,
description='Domain status codes',
example=['clientDeleteProhibited', 'clientTransferProhibited']
),
'name_servers': fields.List(
fields.String,
description='Name servers',
example=['ns1.google.com', 'ns2.google.com']
)
})
# DNS Response
mx_record_model = api.model('MXRecord', {
'priority': fields.Integer(description='MX priority', example=10),
'host': fields.String(description='Mail server hostname', example='smtp.google.com')
})
soa_record_model = api.model('SOARecord', {
'mname': fields.String(description='Primary master name server', example='ns1.google.com'),
'rname': fields.String(description='Responsible party email', example='dns-admin.google.com'),
'serial': fields.Integer(description='Serial number', example=2024011501),
'refresh': fields.Integer(description='Refresh interval', example=3600),
'retry': fields.Integer(description='Retry interval', example=600),
'expire': fields.Integer(description='Expire time', example=86400),
'minimum': fields.Integer(description='Minimum TTL', example=300)
})
dns_response = api.model('DNSResponse', {
'a_records': fields.List(
fields.String,
description='A records (IPv4 addresses)',
example=['142.250.185.46']
),
'aaaa_records': fields.List(
fields.String,
description='AAAA records (IPv6 addresses)',
example=['2a00:1450:4001:801::200e']
),
'mx_records': fields.List(
fields.Nested(mx_record_model),
description='MX records (mail servers)'
),
'txt_records': fields.List(
fields.String,
description='TXT records',
example=['v=spf1 include:_spf.google.com ~all']
),
'ns_records': fields.List(
fields.String,
description='NS records (name servers)',
example=['ns1.google.com', 'ns2.google.com']
),
'cname_records': fields.List(
fields.String,
description='CNAME records',
example=[]
),
'soa_record': fields.Nested(
soa_record_model,
description='SOA record (Start of Authority)'
),
'has_spf': fields.Boolean(
description='Has SPF record',
example=True
),
'has_dkim': fields.Boolean(
description='Has DKIM record',
example=True
),
'has_dmarc': fields.Boolean(
description='Has DMARC record',
example=True
)
})
# SSL Response
ssl_response = api.model('SSLResponse', {
'has_ssl': fields.Boolean(
description='Has valid SSL certificate',
example=True
),
'is_valid': fields.Boolean(
description='Certificate is valid',
example=True
),
'is_self_signed': fields.Boolean(
description='Certificate is self-signed',
example=False
),
'is_expired': fields.Boolean(
description='Certificate is expired',
example=False
),
'is_wildcard': fields.Boolean(
description='Wildcard certificate',
example=False
),
'issuer': fields.String(
description='Certificate issuer',
example='CN=GTS CA 1C3, O=Google Trust Services LLC, C=US'
),
'subject': fields.String(
description='Certificate subject',
example='CN=*.google.com'
),
'valid_from': fields.String(
description='Valid from date (ISO 8601)',
example='2024-12-01T08:15:00Z'
),
'valid_until': fields.String(
description='Valid until date (ISO 8601)',
example='2025-02-23T08:14:59Z'
),
'days_until_expiry': fields.Integer(
description='Days until certificate expires',
example=45
),
'key_size': fields.Integer(
description='Key size in bits',
example=2048
),
'signature_algorithm': fields.String(
description='Signature algorithm',
example='sha256WithRSAEncryption'
),
'error': fields.String(
description='Error message if SSL check failed',
example=None
)
})
# Risk Score Response
risk_factor_model = api.model('RiskFactor', {
'factor': fields.String(
description='Risk factor name',
example='domain_age',
enum=['domain_age', 'reputation', 'ssl', 'dns', 'whois']
),
'score': fields.Float(
description='Individual score for this factor',
example=85.0
),
'weight': fields.Float(
description='Weight of this factor',
example=0.30
),
'weighted_score': fields.Float(
description='Score * weight',
example=25.5
),
'reason': fields.String(
description='Explanation of the score',
example='Domain is only 45 days old (HIGH RISK)'
),
'age_days': fields.Integer(
description='Domain age in days (only for domain_age factor)',
example=45
)
})
risk_thresholds_model = api.model('RiskThresholds', {
'low': fields.String(
description='Low risk score range',
example='0-30'
),
'medium': fields.String(
description='Medium risk score range',
example='31-60'
),
'high': fields.String(
description='High risk score range',
example='61-85'
),
'critical': fields.String(
description='Critical risk score range',
example='86-100'
)
})
risk_score_response = api.model('RiskScoreResponse', {
'total': fields.Float(
description='Total risk score (0-100)',
example=65.5,
min=0,
max=100
),
'level': fields.String(
description='Risk level',
example='HIGH',
enum=['LOW', 'MEDIUM', 'HIGH', 'CRITICAL']
),
'factors': fields.List(
fields.Nested(risk_factor_model),
description='Detailed breakdown of risk factors'
),
'thresholds': fields.Nested(
risk_thresholds_model,
description='Risk level thresholds'
)
})
# Metadata Response
metadata_response = api.model('MetadataResponse', {
'cached': fields.Boolean(
description='Response served from cache',
example=False
),
'processing_time_ms': fields.Integer(
description='Processing time in milliseconds',
example=1250
),
'api_version': fields.String(
description='API version',
example='v1'
)
})
# Main Check Response Data
check_response_data = api.model('CheckResponseData', {
'domain': fields.String(
description='Checked domain name',
example='google.com'
),
'check_id': fields.String(
description='Unique check ID (UUID)',
example='123e4567-e89b-12d3-a456-426614174000'
),
'timestamp': fields.String(
description='Check timestamp (ISO 8601)',
example='2026-01-30T12:34:56Z'
),
'whois': fields.Nested(
whois_response,
description='WHOIS lookup results',
allow_null=True
),
'dns': fields.Nested(
dns_response,
description='DNS lookup results',
allow_null=True
),
'ssl': fields.Nested(
ssl_response,
description='SSL certificate check results',
allow_null=True
),
'reputation': fields.Raw(
description='Reputation check results (not implemented)',
example=None
),
'risk_score': fields.Nested(
risk_score_response,
description='Risk assessment results',
required=True
)
})
# Main Check Response
domain_check_response = api.model('DomainCheckResponse', {
'success': fields.Boolean(
description='Request success status',
example=True
),
'data': fields.Nested(
check_response_data,
description='Response data',
required=True
),
'metadata': fields.Nested(
metadata_response,
description='Request metadata',
required=True
)
})
# ==================== ERROR MODELS ====================
error_detail_model = api.model('ErrorDetail', {
'code': fields.String(
description='Error code',
example='INVALID_REQUEST',
enum=['INVALID_REQUEST', 'NOT_FOUND', 'INTERNAL_ERROR', 'RATE_LIMIT_EXCEEDED']
),
'message': fields.String(
description='Error message',
example='Domain parameter is required'
),
'status': fields.Integer(
description='HTTP status code',
example=400
)
})
error_response = api.model('ErrorResponse', {
'success': fields.Boolean(
description='Request success status',
example=False
),
'error': fields.Nested(
error_detail_model,
description='Error details',
required=True
)
})
# ==================== HEALTH CHECK MODEL ====================
health_response = api.model('HealthResponse', {
'status': fields.String(
description='Health status',
example='healthy',
enum=['healthy', 'unhealthy']
),
'version': fields.String(
description='API version',
example='v1'
),
'environment': fields.String(
description='Environment name',
example='production'
),
'database': fields.String(
description='Database connection status',
example='connected'
),
'redis': fields.String(
description='Redis connection status',
example='connected'
)
})
return {
'domain_check_request': domain_check_request,
'domain_check_response': domain_check_response,
'error_response': error_response,
'health_response': health_response,
'check_options_model': check_options_model,
'whois_response': whois_response,
'dns_response': dns_response,
'ssl_response': ssl_response,
'risk_score_response': risk_score_response
}

6
ai_platform/modules/domain_check/api/app/celery_app.py Normal file
View file

@ -0,0 +1,6 @@
"""
Celery Application for Domain Check
"""
from app import create_celery_app
celery = create_celery_app()

234
ai_platform/modules/domain_check/api/app/config.py Normal file
View file

@ -0,0 +1,234 @@
"""
Configuration module for Domain Check API
"""
import os
from datetime import timedelta
from typing import Dict, Any
class Config:
"""Base configuration"""
# Flask
SECRET_KEY = os.getenv('SECRET_KEY', 'dev-secret-key-change-in-production')
DEBUG = False
TESTING = False
# Database
SQLALCHEMY_DATABASE_URI = os.getenv(
'DATABASE_URL',
'postgresql://dns_admin:password@localhost:5432/domain_check'
)
SQLALCHEMY_TRACK_MODIFICATIONS = False
SQLALCHEMY_ECHO = False
SQLALCHEMY_POOL_SIZE = 10
SQLALCHEMY_POOL_TIMEOUT = 30
SQLALCHEMY_POOL_RECYCLE = 3600
SQLALCHEMY_MAX_OVERFLOW = 20
# Redis
REDIS_URL = os.getenv('REDIS_URL', 'redis://localhost:6379/0')
REDIS_TTL_HOT = int(os.getenv('REDIS_TTL_HOT', 21600)) # 6 hours
REDIS_TTL_WARM = int(os.getenv('REDIS_TTL_WARM', 86400)) # 24 hours
REDIS_TTL_COLD = int(os.getenv('REDIS_TTL_COLD', 604800)) # 7 days
# Celery
CELERY_BROKER_URL = os.getenv('CELERY_BROKER_URL', 'redis://localhost:6379/1')
CELERY_RESULT_BACKEND = os.getenv('CELERY_RESULT_BACKEND', 'redis://localhost:6379/2')
CELERY_TASK_SERIALIZER = 'json'
CELERY_RESULT_SERIALIZER = 'json'
CELERY_ACCEPT_CONTENT = ['json']
CELERY_TIMEZONE = 'UTC'
CELERY_ENABLE_UTC = True
CELERY_TASK_TRACK_STARTED = True
CELERY_TASK_TIME_LIMIT = 300 # 5 minutes
CELERY_TASK_SOFT_TIME_LIMIT = 240 # 4 minutes
# API Keys
WHOXY_API_KEY = os.getenv('WHOXY_API_KEY', '')
VIRUSTOTAL_API_KEY = os.getenv('VIRUSTOTAL_API_KEY', '')
# Risk Scoring Thresholds
RISK_THRESHOLD_LOW = int(os.getenv('RISK_THRESHOLD_LOW', 30))
RISK_THRESHOLD_MEDIUM = int(os.getenv('RISK_THRESHOLD_MEDIUM', 60))
RISK_THRESHOLD_HIGH = int(os.getenv('RISK_THRESHOLD_HIGH', 85))
# Domain Age Thresholds (days)
DOMAIN_AGE_CRITICAL = int(os.getenv('DOMAIN_AGE_CRITICAL', 90)) # 3 months
DOMAIN_AGE_HIGH = int(os.getenv('DOMAIN_AGE_HIGH', 180)) # 6 months
DOMAIN_AGE_MEDIUM = int(os.getenv('DOMAIN_AGE_MEDIUM', 365)) # 1 year
# API Rate Limiting
WHOXY_DAILY_LIMIT = int(os.getenv('WHOXY_DAILY_LIMIT', 8000))
VIRUSTOTAL_DAILY_LIMIT = int(os.getenv('VIRUSTOTAL_DAILY_LIMIT', 500))
# Batch Processing
BATCH_CHUNK_SIZE = int(os.getenv('BATCH_CHUNK_SIZE', 50))
BATCH_TIMEOUT_SECONDS = int(os.getenv('BATCH_TIMEOUT_SECONDS', 300))
# API Settings
API_VERSION = os.getenv('API_VERSION', 'v1')
API_TITLE = 'Domain Check API - Anti-Fake News Tool'
API_DESCRIPTION = '''
**Comprehensive Domain Verification & Risk Scoring API**
This API provides powerful tools for detecting potentially malicious or newly-registered domains
commonly used in disinformation campaigns and fake news distribution.
## 🎯 Key Features
- **WHOIS Analysis**: Domain age, registrar information, privacy protection detection
- **DNS Verification**: A, AAAA, MX, TXT, NS, CNAME, SOA records with email security (SPF, DKIM, DMARC)
- **SSL Certificate Validation**: Certificate validity, issuer verification, expiration monitoring
- **Risk Scoring**: Multi-factor algorithm (0-100) with detailed breakdown
- **Comprehensive Database**: Full history tracking and audit trail
## 📊 Risk Levels
- **LOW (0-30)**: Established, trustworthy domains
- **MEDIUM (31-60)**: Moderate risk, requires attention
- **HIGH (61-85)**: Suspicious activity detected
- **CRITICAL (86-100)**: Newly registered or highly suspicious domains
## 🚀 Quick Start
Use the `/check` endpoint to verify any domain:
```bash
curl -X POST http://domain-check-api:11000/api/v1/check/check \\
-H "Content-Type: application/json" \\
-d '{"domain": "example.com", "check_options": {"whois": true, "dns": true}}'
```
## 📖 Documentation
- **Swagger UI**: Interactive API testing and documentation
- **ReDoc**: Clean, responsive API reference
- **Platformă**: modul T4 (evaluare credibilitate sursă) al platformei DIDI
'''
API_CONTACT = {
'name': 'DIDI Domain Check',
'email': 'support@didi.local'
}
API_LICENSE = {
'name': 'MIT',
'url': 'https://opensource.org/licenses/MIT'
}
API_TERMS_OF_SERVICE = ''
# CORS - use CORS_ORIGINS env var for production (comma-separated)
# Example: CORS_ORIGINS=https://didi.example.ro,https://admin.didi.example.ro
# Default '*' allows all origins (development only!)
_cors_origins_env = os.getenv('CORS_ORIGINS', '*')
CORS_ORIGINS = ['*'] if _cors_origins_env == '*' else [o.strip() for o in _cors_origins_env.split(',') if o.strip()]
CORS_ALLOW_HEADERS = ['Content-Type', 'Authorization']
CORS_METHODS = ['GET', 'POST', 'PUT', 'DELETE', 'OPTIONS']
# Security - Documentation endpoints
# Set DISABLE_SWAGGER=true in production to hide /docs and /redoc
DISABLE_SWAGGER = os.getenv('DISABLE_SWAGGER', 'false').lower() == 'true'
# Health endpoint - set SIMPLE_HEALTH=true to return only {"status":"ok"}
SIMPLE_HEALTH = os.getenv('SIMPLE_HEALTH', 'false').lower() == 'true'
# Logging
LOG_LEVEL = os.getenv('LOG_LEVEL', 'INFO')
LOG_FORMAT = '%(asctime)s - %(name)s - %(levelname)s - %(message)s'
LOG_FILE = os.path.join(os.path.dirname(__file__), '..', 'logs', 'api.log')
# External Service URLs
WHOXY_API_URL = 'https://api.whoxy.com/'
VIRUSTOTAL_API_URL = 'https://www.virustotal.com/api/v3/'
RDAP_BOOTSTRAP_URL = 'https://rdap.org/'
# Risk Scoring Weights
RISK_WEIGHT_DOMAIN_AGE = 0.30
RISK_WEIGHT_REPUTATION = 0.30
RISK_WEIGHT_SSL = 0.20
RISK_WEIGHT_DNS = 0.10
RISK_WEIGHT_WHOIS = 0.10
# Pagination
DEFAULT_PAGE_SIZE = 50
MAX_PAGE_SIZE = 100
# Timeouts (seconds)
HTTP_TIMEOUT = 30
DNS_TIMEOUT = 10
SSL_TIMEOUT = 10
@staticmethod
def init_app(app):
"""Initialize application configuration"""
# Create logs directory if it doesn't exist
log_dir = os.path.dirname(Config.LOG_FILE)
os.makedirs(log_dir, exist_ok=True)
class DevelopmentConfig(Config):
"""Development configuration"""
DEBUG = True
SQLALCHEMY_ECHO = False
LOG_LEVEL = 'DEBUG'
class ProductionConfig(Config):
"""Production configuration"""
DEBUG = False
TESTING = False
SQLALCHEMY_ECHO = False
LOG_LEVEL = 'WARNING'
# Production security defaults (can be overridden by env vars)
# If CORS_ORIGINS not set, default to empty (blocks all cross-origin)
_cors_origins_env = os.getenv('CORS_ORIGINS', '')
CORS_ORIGINS = [o.strip() for o in _cors_origins_env.split(',') if o.strip()] if _cors_origins_env else []
# Disable swagger by default in production (override with DISABLE_SWAGGER=false)
DISABLE_SWAGGER = os.getenv('DISABLE_SWAGGER', 'true').lower() != 'false'
# Simple health by default in production
SIMPLE_HEALTH = os.getenv('SIMPLE_HEALTH', 'true').lower() != 'false'
@classmethod
def init_app(cls, app):
Config.init_app(app)
# Production-specific initialization
import logging
from logging.handlers import RotatingFileHandler
# Setup file handler with rotation
file_handler = RotatingFileHandler(
cls.LOG_FILE,
maxBytes=10485760, # 10MB
backupCount=10
)
file_handler.setLevel(logging.WARNING)
file_handler.setFormatter(logging.Formatter(cls.LOG_FORMAT))
app.logger.addHandler(file_handler)
class TestingConfig(Config):
"""Testing configuration"""
TESTING = True
DEBUG = True
SQLALCHEMY_DATABASE_URI = 'sqlite:///:memory:'
REDIS_URL = 'redis://localhost:6379/15' # Use separate Redis DB for testing
WTF_CSRF_ENABLED = False
# Configuration dictionary
config: Dict[str, Any] = {
'development': DevelopmentConfig,
'production': ProductionConfig,
'testing': TestingConfig,
'default': DevelopmentConfig
}
def get_config(env: str = None) -> Config:
"""Get configuration based on environment"""
if env is None:
env = os.getenv('FLASK_ENV', 'development')
return config.get(env, config['default'])

26
ai_platform/modules/domain_check/api/app/models/__init__.py Normal file
View file

@ -0,0 +1,26 @@
"""
SQLAlchemy Models Package
"""
from app.models.domain import Domain
from app.models.whois import WhoisRecord
from app.models.dns import DnsRecord
from app.models.ssl import SslCertificate
from app.models.reputation import ReputationScore
from app.models.risk import RiskAssessment
from app.models.check_history import CheckHistory
from app.models.batch import BatchOperation
from app.models.blacklist import Blacklist
from app.models.api_usage import ApiUsageLog
__all__ = [
'Domain',
'WhoisRecord',
'DnsRecord',
'SslCertificate',
'ReputationScore',
'RiskAssessment',
'CheckHistory',
'BatchOperation',
'Blacklist',
'ApiUsageLog'
]

32
ai_platform/modules/domain_check/api/app/models/api_usage.py Normal file
View file

@ -0,0 +1,32 @@
"""API Usage Log Model"""
import uuid
from datetime import datetime
from sqlalchemy.dialects.postgresql import UUID, JSONB
from app import db
class ApiUsageLog(db.Model):
"""API usage logging model"""
__tablename__ = 'api_usage_logs'
id = db.Column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
service_name = db.Column(db.String(50), nullable=False, index=True)
endpoint = db.Column(db.String(255))
request_count = db.Column(db.Integer, default=1)
response_time_ms = db.Column(db.Integer)
status_code = db.Column(db.Integer, index=True)
quota_used = db.Column(db.Integer)
quota_remaining = db.Column(db.Integer)
error_message = db.Column(db.Text)
request_params = db.Column(JSONB)
logged_at = db.Column(db.DateTime(timezone=True), default=datetime.utcnow, index=True)
def to_dict(self):
return {
'id': str(self.id),
'service_name': self.service_name,
'endpoint': self.endpoint,
'status_code': self.status_code,
'response_time_ms': self.response_time_ms,
'logged_at': self.logged_at.isoformat() if self.logged_at else None
}

39
ai_platform/modules/domain_check/api/app/models/batch.py Normal file
View file

@ -0,0 +1,39 @@
"""Batch Operation Model"""
import uuid
from datetime import datetime
from sqlalchemy.dialects.postgresql import UUID, ARRAY, JSONB
from app import db
class BatchOperation(db.Model):
"""Batch operation model"""
__tablename__ = 'batch_operations'
id = db.Column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
batch_id = db.Column(UUID(as_uuid=True), unique=True, nullable=False, index=True)
total_domains = db.Column(db.Integer, nullable=False)
completed_count = db.Column(db.Integer, default=0)
failed_count = db.Column(db.Integer, default=0)
status = db.Column(db.String(20), default='pending', index=True)
priority = db.Column(db.String(20), default='normal', index=True)
started_at = db.Column(db.DateTime(timezone=True))
completed_at = db.Column(db.DateTime(timezone=True))
estimated_completion_at = db.Column(db.DateTime(timezone=True))
requested_by = db.Column(db.String(100))
check_options = db.Column(JSONB)
domain_list = db.Column(ARRAY(db.Text))
error_log = db.Column(JSONB)
summary = db.Column(JSONB)
created_at = db.Column(db.DateTime(timezone=True), default=datetime.utcnow, index=True)
def to_dict(self):
return {
'batch_id': str(self.batch_id),
'total_domains': self.total_domains,
'completed_count': self.completed_count,
'failed_count': self.failed_count,
'status': self.status,
'progress_percentage': round((self.completed_count / self.total_domains * 100) if self.total_domains > 0 else 0, 2),
'started_at': self.started_at.isoformat() if self.started_at else None,
'completed_at': self.completed_at.isoformat() if self.completed_at else None
}

33
ai_platform/modules/domain_check/api/app/models/blacklist.py Normal file
View file

@ -0,0 +1,33 @@
"""Blacklist Model"""
import uuid
from datetime import datetime
from sqlalchemy.dialects.postgresql import UUID, JSONB
from app import db
class Blacklist(db.Model):
"""Blacklist model"""
__tablename__ = 'blacklists'
id = db.Column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
domain = db.Column(db.String(255), unique=True, nullable=False, index=True)
reason = db.Column(db.Text)
category = db.Column(db.String(50), index=True)
source = db.Column(db.String(100))
severity = db.Column(db.String(20), default='medium', index=True)
added_at = db.Column(db.DateTime(timezone=True), default=datetime.utcnow)
updated_at = db.Column(db.DateTime(timezone=True), default=datetime.utcnow, onupdate=datetime.utcnow)
is_active = db.Column(db.Boolean, default=True, index=True)
expires_at = db.Column(db.DateTime(timezone=True))
extra_data = db.Column(JSONB)
def to_dict(self):
return {
'id': str(self.id),
'domain': self.domain,
'reason': self.reason,
'category': self.category,
'severity': self.severity,
'is_active': self.is_active,
'added_at': self.added_at.isoformat() if self.added_at else None
}

38
ai_platform/modules/domain_check/api/app/models/check_history.py Normal file
View file

@ -0,0 +1,38 @@
"""Check History Model"""
import uuid
from datetime import datetime
from sqlalchemy.dialects.postgresql import UUID, JSONB
from app import db
class CheckHistory(db.Model):
"""Check history model"""
__tablename__ = 'check_history'
id = db.Column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
check_id = db.Column(UUID(as_uuid=True), unique=True, nullable=False, index=True)
domain_id = db.Column(UUID(as_uuid=True), db.ForeignKey('domains.id', ondelete='CASCADE'), nullable=False, index=True)
risk_assessment_id = db.Column(UUID(as_uuid=True), db.ForeignKey('risk_assessments.id', ondelete='SET NULL'))
requested_by = db.Column(db.String(100), default='api', index=True)
request_ip = db.Column(db.String(45))
user_agent = db.Column(db.Text)
check_options = db.Column(JSONB)
processing_time_ms = db.Column(db.Integer)
cache_hit = db.Column(db.Boolean, default=False)
changes_detected = db.Column(db.Boolean, default=False)
change_summary = db.Column(JSONB)
status = db.Column(db.String(20), default='completed', index=True)
error_message = db.Column(db.Text)
created_at = db.Column(db.DateTime(timezone=True), default=datetime.utcnow, index=True)
domain = db.relationship('Domain', back_populates='check_history')
def to_dict(self):
return {
'id': str(self.id),
'check_id': str(self.check_id),
'status': self.status,
'cache_hit': self.cache_hit,
'processing_time_ms': self.processing_time_ms,
'created_at': self.created_at.isoformat() if self.created_at else None
}

31
ai_platform/modules/domain_check/api/app/models/dns.py Normal file
View file

@ -0,0 +1,31 @@
"""DNS Record Model"""
import uuid
from datetime import datetime
from sqlalchemy.dialects.postgresql import UUID
from app import db
class DnsRecord(db.Model):
"""DNS record model"""
__tablename__ = 'dns_records'
id = db.Column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
domain_id = db.Column(UUID(as_uuid=True), db.ForeignKey('domains.id', ondelete='CASCADE'), nullable=False, index=True)
record_type = db.Column(db.String(10), nullable=False, index=True)
record_value = db.Column(db.Text, nullable=False)
ttl = db.Column(db.Integer)
priority = db.Column(db.Integer)
fetched_at = db.Column(db.DateTime(timezone=True), default=datetime.utcnow, index=True)
created_at = db.Column(db.DateTime(timezone=True), default=datetime.utcnow)
domain = db.relationship('Domain', back_populates='dns_records')
def to_dict(self):
return {
'id': str(self.id),
'record_type': self.record_type,
'record_value': self.record_value,
'ttl': self.ttl,
'priority': self.priority,
'fetched_at': self.fetched_at.isoformat() if self.fetched_at else None
}

118
ai_platform/modules/domain_check/api/app/models/domain.py Normal file
View file

@ -0,0 +1,118 @@
"""
Domain Model
"""
import uuid
from datetime import datetime
from sqlalchemy.dialects.postgresql import UUID, ARRAY
from app import db
class Domain(db.Model):
"""Domain model - stores basic domain information"""
__tablename__ = 'domains'
id = db.Column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
domain = db.Column(db.String(255), nullable=False, index=True)
subdomain = db.Column(db.String(255), nullable=True)
tld = db.Column(db.String(50), nullable=False, index=True)
full_domain = db.Column(db.String(255), unique=True, nullable=False, index=True)
first_seen_at = db.Column(db.DateTime(timezone=True), default=datetime.utcnow)
last_checked_at = db.Column(db.DateTime(timezone=True), nullable=True, index=True)
check_count = db.Column(db.Integer, default=0)
is_active = db.Column(db.Boolean, default=True, index=True)
created_at = db.Column(db.DateTime(timezone=True), default=datetime.utcnow)
updated_at = db.Column(db.DateTime(timezone=True), default=datetime.utcnow, onupdate=datetime.utcnow)
# Relationships
whois_records = db.relationship('WhoisRecord', back_populates='domain', lazy='dynamic', cascade='all, delete-orphan')
dns_records = db.relationship('DnsRecord', back_populates='domain', lazy='dynamic', cascade='all, delete-orphan')
ssl_certificates = db.relationship('SslCertificate', back_populates='domain', lazy='dynamic', cascade='all, delete-orphan')
reputation_scores = db.relationship('ReputationScore', back_populates='domain', lazy='dynamic', cascade='all, delete-orphan')
risk_assessments = db.relationship('RiskAssessment', back_populates='domain', lazy='dynamic', cascade='all, delete-orphan')
check_history = db.relationship('CheckHistory', back_populates='domain', lazy='dynamic', cascade='all, delete-orphan')
def __repr__(self):
return f'<Domain {self.full_domain}>'
def to_dict(self, include_relationships=False):
"""Convert model to dictionary"""
data = {
'id': str(self.id),
'domain': self.domain,
'subdomain': self.subdomain,
'tld': self.tld,
'full_domain': self.full_domain,
'first_seen_at': self.first_seen_at.isoformat() if self.first_seen_at else None,
'last_checked_at': self.last_checked_at.isoformat() if self.last_checked_at else None,
'check_count': self.check_count,
'is_active': self.is_active,
'created_at': self.created_at.isoformat() if self.created_at else None,
'updated_at': self.updated_at.isoformat() if self.updated_at else None
}
if include_relationships:
# Get latest records
latest_whois = self.whois_records.order_by(WhoisRecord.fetched_at.desc()).first()
latest_risk = self.risk_assessments.order_by(RiskAssessment.assessed_at.desc()).first()
data['latest_whois'] = latest_whois.to_dict() if latest_whois else None
data['latest_risk'] = latest_risk.to_dict() if latest_risk else None
return data
@staticmethod
def parse_domain(full_domain: str) -> dict:
"""
Parse a full domain into components
Args:
full_domain: Full domain name (e.g., 'www.example.com')
Returns:
dict with 'domain', 'subdomain', 'tld', 'full_domain'
"""
import tldextract
extracted = tldextract.extract(full_domain)
return {
'domain': extracted.domain,
'subdomain': extracted.subdomain if extracted.subdomain else None,
'tld': extracted.suffix,
'full_domain': full_domain.lower().strip()
}
@classmethod
def get_or_create(cls, full_domain: str):
"""
Get existing domain or create new one
Args:
full_domain: Full domain name
Returns:
Tuple of (Domain instance, created boolean)
"""
domain = cls.query.filter_by(full_domain=full_domain.lower()).first()
if domain:
return domain, False
# Parse domain components
parsed = cls.parse_domain(full_domain)
# Create new domain
domain = cls(
domain=parsed['domain'],
subdomain=parsed['subdomain'],
tld=parsed['tld'],
full_domain=parsed['full_domain']
)
db.session.add(domain)
db.session.commit()
return domain, True

42
ai_platform/modules/domain_check/api/app/models/reputation.py Normal file
View file

@ -0,0 +1,42 @@
"""Reputation Score Model"""
import uuid
from datetime import datetime
from sqlalchemy.dialects.postgresql import UUID, ARRAY, JSONB
from app import db
class ReputationScore(db.Model):
"""Reputation score from various sources"""
__tablename__ = 'reputation_scores'
id = db.Column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
domain_id = db.Column(UUID(as_uuid=True), db.ForeignKey('domains.id', ondelete='CASCADE'), nullable=False, index=True)
source = db.Column(db.String(50), nullable=False, index=True)
score = db.Column(db.Integer)
malicious_count = db.Column(db.Integer, default=0)
suspicious_count = db.Column(db.Integer, default=0)
harmless_count = db.Column(db.Integer, default=0)
undetected_count = db.Column(db.Integer, default=0)
is_blacklisted = db.Column(db.Boolean, default=False, index=True)
blacklist_names = db.Column(ARRAY(db.Text))
is_typosquatting = db.Column(db.Boolean, default=False, index=True)
typosquatting_target = db.Column(db.String(255))
raw_response = db.Column(JSONB)
checked_at = db.Column(db.DateTime(timezone=True), default=datetime.utcnow, index=True)
created_at = db.Column(db.DateTime(timezone=True), default=datetime.utcnow)
domain = db.relationship('Domain', back_populates='reputation_scores')
def to_dict(self):
return {
'id': str(self.id),
'source': self.source,
'score': self.score,
'malicious_count': self.malicious_count,
'suspicious_count': self.suspicious_count,
'harmless_count': self.harmless_count,
'is_blacklisted': self.is_blacklisted,
'blacklist_names': self.blacklist_names,
'is_typosquatting': self.is_typosquatting,
'checked_at': self.checked_at.isoformat() if self.checked_at else None
}

43
ai_platform/modules/domain_check/api/app/models/risk.py Normal file
View file

@ -0,0 +1,43 @@
"""Risk Assessment Model"""
import uuid
from datetime import datetime
from sqlalchemy.dialects.postgresql import UUID, JSONB
from app import db
class RiskAssessment(db.Model):
"""Risk assessment model"""
__tablename__ = 'risk_assessments'
id = db.Column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
domain_id = db.Column(UUID(as_uuid=True), db.ForeignKey('domains.id', ondelete='CASCADE'), nullable=False, index=True)
check_id = db.Column(UUID(as_uuid=True), unique=True, nullable=False, index=True)
total_score = db.Column(db.Integer, nullable=False, index=True)
risk_level = db.Column(db.String(20), nullable=False, index=True)
domain_age_score = db.Column(db.Integer, default=0)
domain_age_days = db.Column(db.Integer)
ssl_score = db.Column(db.Integer, default=0)
dns_score = db.Column(db.Integer, default=0)
reputation_score = db.Column(db.Integer, default=0)
whois_score = db.Column(db.Integer, default=0)
factors = db.Column(JSONB)
is_new_domain = db.Column(db.Boolean, default=False, index=True)
is_suspicious = db.Column(db.Boolean, default=False, index=True)
requires_manual_review = db.Column(db.Boolean, default=False)
assessed_at = db.Column(db.DateTime(timezone=True), default=datetime.utcnow, index=True)
created_at = db.Column(db.DateTime(timezone=True), default=datetime.utcnow)
domain = db.relationship('Domain', back_populates='risk_assessments')
def to_dict(self):
return {
'id': str(self.id),
'check_id': str(self.check_id),
'total_score': self.total_score,
'risk_level': self.risk_level,
'domain_age_days': self.domain_age_days,
'is_new_domain': self.is_new_domain,
'is_suspicious': self.is_suspicious,
'factors': self.factors,
'assessed_at': self.assessed_at.isoformat() if self.assessed_at else None
}

40
ai_platform/modules/domain_check/api/app/models/ssl.py Normal file
View file

@ -0,0 +1,40 @@
"""SSL Certificate Model"""
import uuid
from datetime import datetime
from sqlalchemy.dialects.postgresql import UUID, JSONB
from app import db
class SslCertificate(db.Model):
"""SSL Certificate model"""
__tablename__ = 'ssl_certificates'
id = db.Column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
domain_id = db.Column(UUID(as_uuid=True), db.ForeignKey('domains.id', ondelete='CASCADE'), nullable=False, index=True)
issuer = db.Column(db.String(255), index=True)
subject = db.Column(db.String(255))
valid_from = db.Column(db.DateTime(timezone=True))
valid_until = db.Column(db.DateTime(timezone=True), index=True)
serial_number = db.Column(db.String(255))
signature_algorithm = db.Column(db.String(100))
key_size = db.Column(db.Integer)
is_wildcard = db.Column(db.Boolean, default=False)
is_self_signed = db.Column(db.Boolean, default=False)
is_valid = db.Column(db.Boolean, default=True, index=True)
certificate_chain = db.Column(JSONB)
fetched_at = db.Column(db.DateTime(timezone=True), default=datetime.utcnow, index=True)
created_at = db.Column(db.DateTime(timezone=True), default=datetime.utcnow)
domain = db.relationship('Domain', back_populates='ssl_certificates')
def to_dict(self):
return {
'id': str(self.id),
'issuer': self.issuer,
'subject': self.subject,
'valid_from': self.valid_from.isoformat() if self.valid_from else None,
'valid_until': self.valid_until.isoformat() if self.valid_until else None,
'is_valid': self.is_valid,
'is_self_signed': self.is_self_signed,
'days_until_expiry': (self.valid_until - datetime.utcnow()).days if self.valid_until and self.valid_until > datetime.utcnow() else 0
}

62
ai_platform/modules/domain_check/api/app/models/whois.py Normal file
View file

@ -0,0 +1,62 @@
"""
WHOIS Record Model
"""
import uuid
from datetime import datetime
from sqlalchemy.dialects.postgresql import UUID, ARRAY, JSONB
from app import db
class WhoisRecord(db.Model):
"""WHOIS/RDAP record model"""
__tablename__ = 'whois_records'
id = db.Column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
domain_id = db.Column(UUID(as_uuid=True), db.ForeignKey('domains.id', ondelete='CASCADE'), nullable=False, index=True)
creation_date = db.Column(db.DateTime(timezone=True), index=True)
expiration_date = db.Column(db.DateTime(timezone=True))
updated_date = db.Column(db.DateTime(timezone=True))
registrar = db.Column(db.String(255), index=True)
registrar_url = db.Column(db.String(500))
registrant_org = db.Column(db.String(255))
registrant_country = db.Column(db.String(2))
admin_email = db.Column(db.String(255))
name_servers = db.Column(ARRAY(db.Text))
status = db.Column(ARRAY(db.Text))
dnssec = db.Column(db.Boolean)
raw_whois_data = db.Column(JSONB)
data_source = db.Column(db.String(50), default='rdap', index=True)
fetched_at = db.Column(db.DateTime(timezone=True), default=datetime.utcnow, index=True)
created_at = db.Column(db.DateTime(timezone=True), default=datetime.utcnow)
# Relationship
domain = db.relationship('Domain', back_populates='whois_records')
def __repr__(self):
return f'<WhoisRecord domain_id={self.domain_id} source={self.data_source}>'
def to_dict(self):
"""Convert to dictionary"""
return {
'id': str(self.id),
'domain_id': str(self.domain_id),
'creation_date': self.creation_date.isoformat() if self.creation_date else None,
'expiration_date': self.expiration_date.isoformat() if self.expiration_date else None,
'updated_date': self.updated_date.isoformat() if self.updated_date else None,
'registrar': self.registrar,
'registrar_url': self.registrar_url,
'registrant_org': self.registrant_org,
'registrant_country': self.registrant_country,
'admin_email': self.admin_email,
'name_servers': self.name_servers,
'status': self.status,
'dnssec': self.dnssec,
'data_source': self.data_source,
'fetched_at': self.fetched_at.isoformat() if self.fetched_at else None,
'age_days': (datetime.utcnow() - self.creation_date).days if self.creation_date else None
}

0
ai_platform/modules/domain_check/api/app/routes/__init__.py Normal file
View file

14
ai_platform/modules/domain_check/api/app/routes/batch.py Normal file
View file

@ -0,0 +1,14 @@
"""Batch routes - stub for now"""
from flask import Blueprint, jsonify
batch_bp = Blueprint('batch', __name__)
@batch_bp.route('/check/batch', methods=['POST'])
def batch_check():
"""Batch domain check - TODO: implement"""
return jsonify({'message': 'Not yet implemented'}), 501
@batch_bp.route('/batch/<string:batch_id>/status', methods=['GET'])
def batch_status(batch_id):
"""Get batch status - TODO: implement"""
return jsonify({'message': 'Not yet implemented', 'batch_id': batch_id}), 501

560
ai_platform/modules/domain_check/api/app/routes/check.py Normal file
View file

@ -0,0 +1,560 @@
"""
Domain Check Routes - Comprehensive Domain Verification API
"""
import uuid
import logging
import time
from concurrent.futures import ThreadPoolExecutor
from datetime import datetime
from flask import request, current_app
from flask_restx import Namespace, Resource, fields
from werkzeug.exceptions import BadRequest, HTTPException
from app import db
from app.models import Domain, WhoisRecord, DnsRecord, SslCertificate, RiskAssessment, CheckHistory
from app.services.whois_service import WhoisService
from app.services.risk_scorer import _coerce_datetime
from app.services.dns_service import DNSService
from app.services.ssl_service import SSLService
from app.services.risk_scorer import RiskScorer
from app.services.ip_intelligence_service import IPIntelligenceService
from app.services.http_analysis_service import HTTPAnalysisService
from app.services.blacklist_service import BlacklistService
from app.services.subdomain_service import SubdomainService
from app.services.port_scan_service import PortScanService
from app.services.mail_intelligence_service import MailIntelligenceService
logger = logging.getLogger(__name__)
# Create namespace
api = Namespace('check', description='Domain checking operations')
# Import model definitions from api_models.py
from app.api_models import create_api_models
# Create placeholder models
_temp_api = Namespace('_temp')
_models = create_api_models(_temp_api)
# Get the models we need for decorators
check_request_model = _models.get('domain_check_request')
check_response_model = _models.get('domain_check_response')
error_response_model = _models.get('error_response')
@api.route('/check')
class DomainCheck(Resource):
"""Domain Check Resource - Comprehensive domain verification"""
@api.doc(
'check_domain',
description='''Perform comprehensive domain verification including:
**Basic Checks:**
- WHOIS lookup (domain age, registrar, status, name servers)
- DNS record checking (A, AAAA, MX, TXT, NS, CNAME, SOA)
- SSL certificate validation (validity, issuer, expiration)
**Advanced Checks (when enabled):**
- IP Intelligence (geolocation, ASN, reverse DNS, hosting info)
- HTTP Security Analysis (headers, technology detection)
- Blacklist/Reputation checking (DNSBL, spam lists)
- Port Scanning (open ports, dangerous services)
- Subdomain Enumeration (Certificate Transparency)
Returns detailed domain information and comprehensive risk score (0-100).'''
)
@api.expect(check_request_model, validate=False)
@api.response(200, 'Success - Domain check completed', check_response_model)
@api.response(400, 'Bad Request - Invalid domain or parameters', error_response_model)
@api.response(500, 'Internal Server Error - Check failed', error_response_model)
def post(self):
"""
Check a single domain - Full comprehensive analysis
"""
start_time = time.time()
try:
# Get request data
try:
data = api.payload or {}
except BadRequest:
data = {}
if not data or 'domain' not in data:
return {
'success': False,
'error': {
'code': 'INVALID_REQUEST',
'message': 'Domain parameter is required',
'status': 400
}
}, 400
domain_name = data['domain'].lower().strip()
check_options = data.get('check_options', {
'whois': True,
'dns': True,
'ssl': True,
'ip_intelligence': True,
'http_analysis': True,
'blacklist': True,
'port_scan': False, # Disabled by default (slow)
'subdomains': False, # Disabled by default (slow)
'force_refresh': False
})
logger.info(f'Checking domain: {domain_name} with options: {check_options}')
# Generate check ID
check_id = uuid.uuid4()
# Get or create domain
domain, created = Domain.get_or_create(domain_name)
# Initialize all services
whoxy_api_key = current_app.config.get('WHOXY_API_KEY')
whois_service = WhoisService(whoxy_api_key=whoxy_api_key)
dns_service = DNSService()
ssl_service = SSLService()
ip_service = IPIntelligenceService()
http_service = HTTPAnalysisService()
blacklist_service = BlacklistService()
subdomain_service = SubdomainService()
port_scan_service = PortScanService()
mail_service = MailIntelligenceService()
risk_scorer = RiskScorer()
# Collect all domain data for risk scoring
domain_data = {
'domain': domain_name,
'whois': None,
'dns': None,
'ssl': None,
'ip_intelligence': None,
'http_analysis': None,
'blacklist': None,
'port_scan': None,
'subdomains': None
}
# Response data
response_data = {
'domain': domain_name,
'check_id': str(check_id),
'timestamp': datetime.utcnow().isoformat() + 'Z'
}
# Primary IP for subsequent checks
primary_ip = None
# ----------------------------------------------------------------
# Concurrent fetch. Network lookups are I/O-bound and independent,
# so we run them in a thread pool instead of sequentially (cuts a
# full check from ~sum-of-calls to ~max-of-calls). DB writes stay in
# the main thread afterwards because the SQLAlchemy session is not
# thread-safe. Each task runs inside an app context so services that
# read current_app keep working off the main thread.
# ----------------------------------------------------------------
app_obj = current_app._get_current_object()
def _run(fn, *a, **kw):
with app_obj.app_context():
try:
return fn(*a, **kw)
except Exception as exc:
logger.warning(f'{getattr(fn, "__name__", fn)} failed: {exc}')
return None
# Phase 1 — tasks that only need the domain name.
p1 = {}
with ThreadPoolExecutor(max_workers=5) as ex:
if check_options.get('whois', True):
p1['whois'] = ex.submit(_run, whois_service.lookup, domain_name)
if check_options.get('dns', True):
p1['dns'] = ex.submit(_run, dns_service.lookup, domain_name)
if check_options.get('ssl', True):
p1['ssl'] = ex.submit(_run, ssl_service.check, domain_name)
if check_options.get('http_analysis', True):
p1['http'] = ex.submit(_run, http_service.analyze, domain_name)
if check_options.get('subdomains', False):
extra_subs = check_options.get('extra_subdomains') or []
p1['subdomains'] = ex.submit(_run, subdomain_service.enumerate,
get_root_domain(domain_name), extra_subs)
whois_data = p1['whois'].result() if 'whois' in p1 else None
dns_data = p1['dns'].result() if 'dns' in p1 else None
ssl_data = p1['ssl'].result() if 'ssl' in p1 else None
http_data = p1['http'].result() if 'http' in p1 else None
subdomain_data = p1['subdomains'].result() if 'subdomains' in p1 else None
if dns_data and dns_data.get('a_records'):
primary_ip = dns_data['a_records'][0]
# Phase 2 — tasks that need DNS results (primary IP, MX/TXT).
p2 = {}
with ThreadPoolExecutor(max_workers=4) as ex:
if check_options.get('ip_intelligence', True) and primary_ip:
p2['ip'] = ex.submit(_run, ip_service.lookup, primary_ip)
if check_options.get('blacklist', True):
p2['blacklist'] = ex.submit(_run, blacklist_service.check_all, domain_name, primary_ip)
if check_options.get('port_scan', False) and primary_ip:
p2['port'] = ex.submit(_run, port_scan_service.quick_scan, primary_ip)
if check_options.get('mail', True):
mx = dns_data.get('mx_records') if dns_data else None
txt = dns_data.get('txt_records') if dns_data else None
p2['mail'] = ex.submit(_run, mail_service.analyze, domain_name, mx, txt,
check_options.get('smtp_probe', False))
ip_data = p2['ip'].result() if 'ip' in p2 else None
blacklist_data = p2['blacklist'].result() if 'blacklist' in p2 else None
port_data = p2['port'].result() if 'port' in p2 else None
mail_data = p2['mail'].result() if 'mail' in p2 else None
# ----------------------------------------------------------------
# Process results + persist (single-threaded, ordered).
# ----------------------------------------------------------------
# 1. WHOIS
if whois_data:
domain_data['whois'] = whois_data
response_data['whois'] = format_whois_response(whois_data)
# Save WHOIS record
try:
whois_record = WhoisRecord(
domain_id=domain.id,
creation_date=whois_data.get('creation_date'),
expiration_date=whois_data.get('expiration_date'),
updated_date=whois_data.get('updated_date'),
registrar=whois_data.get('registrar'),
registrar_url=whois_data.get('registrar_url'),
registrant_org=whois_data.get('registrant_org'),
registrant_country=whois_data.get('registrant_country'),
admin_email=whois_data.get('admin_email'),
name_servers=whois_data.get('name_servers', []),
status=whois_data.get('status', []),
dnssec=whois_data.get('dnssec'),
raw_whois_data=whois_data.get('raw_data', {}),
data_source=whois_data.get('data_source', 'rdap')
)
db.session.add(whois_record)
except Exception as e:
logger.warning(f'Failed to save WHOIS record: {e}')
# 2. DNS
if dns_data:
domain_data['dns'] = dns_data
response_data['dns'] = format_dns_response(dns_data)
# Save DNS records
try:
for record_type in ['a_records', 'aaaa_records', 'mx_records', 'txt_records', 'ns_records', 'cname_records']:
records = dns_data.get(record_type, [])
if records:
for record in records:
if isinstance(record, dict): # MX records
dns_record = DnsRecord(
domain_id=domain.id,
record_type='MX',
record_value=record['host'],
priority=record['priority']
)
else:
dns_record = DnsRecord(
domain_id=domain.id,
record_type=record_type.replace('_records', '').upper(),
record_value=str(record)
)
db.session.add(dns_record)
except Exception as e:
logger.warning(f'Failed to save DNS records: {e}')
# Registration / availability verdict (WHOIS + DNS combined signals)
if check_options.get('whois', True) or check_options.get('dns', True):
availability = determine_availability(whois_data, dns_data)
response_data['availability'] = availability
if 'whois' in response_data:
response_data['whois']['is_registered'] = availability['is_registered']
response_data['whois']['is_available'] = availability['is_available']
# 3. SSL Certificate
if ssl_data:
domain_data['ssl'] = ssl_data
response_data['ssl'] = format_ssl_response(ssl_data)
if ssl_data.get('has_ssl'):
try:
ssl_cert = SslCertificate(
domain_id=domain.id,
issuer=ssl_data.get('issuer'),
subject=ssl_data.get('subject'),
valid_from=ssl_data.get('valid_from'),
valid_until=ssl_data.get('valid_until'),
serial_number=ssl_data.get('serial_number'),
signature_algorithm=ssl_data.get('signature_algorithm'),
key_size=ssl_data.get('key_size'),
is_wildcard=ssl_data.get('is_wildcard', False),
is_self_signed=ssl_data.get('is_self_signed', False),
is_valid=ssl_data.get('is_valid', True)
)
db.session.add(ssl_cert)
except Exception as e:
logger.warning(f'Failed to save SSL certificate: {e}')
# 4. IP Intelligence
if ip_data:
ip_data['hosting_score'] = ip_service.get_hosting_score(ip_data)
domain_data['ip_intelligence'] = ip_data
response_data['ip_intelligence'] = ip_data
# 5. HTTP Analysis
if http_data:
domain_data['http_analysis'] = http_data
response_data['http_analysis'] = http_data
# 6. Blacklist
if blacklist_data:
domain_data['blacklist'] = blacklist_data
response_data['blacklist'] = blacklist_data
# 7. Mail Intelligence (email infrastructure deep analysis)
if mail_data:
domain_data['mail'] = mail_data
response_data['mail'] = mail_data
# 8. Port Scan (optional)
if port_data:
domain_data['port_scan'] = port_data
response_data['port_scan'] = port_data
# 9. Subdomain Enumeration (optional)
if subdomain_data:
domain_data['subdomains'] = subdomain_data
response_data['subdomains'] = subdomain_data
# 9. Calculate Risk Score
logger.info(f'Calculating risk score for {domain_name}')
risk_assessment_result = risk_scorer.calculate_risk(domain_data)
# Save risk assessment
try:
risk_assessment = RiskAssessment(
domain_id=domain.id,
check_id=check_id,
total_score=risk_assessment_result['total_score'],
risk_level=risk_assessment_result['risk_level'],
domain_age_score=risk_assessment_result['individual_scores'].get('domain_age', 0),
domain_age_days=next((f['details'].get('age_days') for f in risk_assessment_result['factors'] if f['factor'] == 'domain_age'), None),
ssl_score=risk_assessment_result['individual_scores'].get('ssl', 0),
dns_score=risk_assessment_result['individual_scores'].get('dns', 0),
reputation_score=risk_assessment_result['individual_scores'].get('blacklist', 0),
whois_score=risk_assessment_result['individual_scores'].get('whois', 0),
factors=risk_assessment_result['factors'],
is_new_domain=risk_assessment_result['is_new_domain'],
is_suspicious=risk_assessment_result['is_suspicious'],
requires_manual_review=risk_assessment_result['requires_manual_review']
)
db.session.add(risk_assessment)
except Exception as e:
logger.warning(f'Failed to save risk assessment: {e}')
# Save check history
processing_time_ms = int((time.time() - start_time) * 1000)
try:
check_history = CheckHistory(
check_id=check_id,
domain_id=domain.id,
requested_by='api',
request_ip=request.remote_addr,
user_agent=request.headers.get('User-Agent'),
check_options=check_options,
processing_time_ms=processing_time_ms,
cache_hit=False,
status='completed'
)
db.session.add(check_history)
except Exception as e:
logger.warning(f'Failed to save check history: {e}')
# Commit all changes
try:
db.session.commit()
except Exception as e:
logger.error(f'Database commit failed: {e}')
db.session.rollback()
# Build response
response_data['risk_score'] = {
'total': risk_assessment_result['total_score'],
'level': risk_assessment_result['risk_level'],
'factors': risk_assessment_result['factors'],
'formula_breakdown': risk_assessment_result.get('formula_breakdown', []),
'formula_string': risk_assessment_result.get('formula_string', ''),
'thresholds': risk_assessment_result.get('risk_thresholds', {
'low': '0-25',
'medium': '26-50',
'high': '51-75',
'critical': '76-100'
}),
'is_new_domain': risk_assessment_result['is_new_domain'],
'is_suspicious': risk_assessment_result['is_suspicious'],
'is_blacklisted': risk_assessment_result.get('is_blacklisted', False),
'requires_manual_review': risk_assessment_result['requires_manual_review']
}
response = {
'success': True,
'data': response_data,
'metadata': {
'cached': False,
'processing_time_ms': processing_time_ms,
'api_version': current_app.config.get('API_VERSION', 'v1'),
'checks_performed': [k for k, v in check_options.items() if v and k != 'force_refresh']
}
}
return response, 200
except HTTPException:
# Re-raise HTTP exceptions (400, 404, etc.) as-is
raise
except Exception as e:
logger.error(f'Error checking domain: {str(e)}', exc_info=True)
db.session.rollback()
api.abort(500, f'An error occurred while checking the domain: {str(e)}',
success=False,
error={
'code': 'INTERNAL_ERROR',
'message': f'An error occurred while checking the domain: {str(e)}'
})
def get_root_domain(domain: str) -> str:
"""Extract root domain from subdomain"""
parts = domain.split('.')
if len(parts) >= 2:
# Handle common TLDs
common_tlds = ['com', 'org', 'net', 'io', 'co', 'eu', 'ro', 'de', 'uk', 'fr']
if parts[-1] in common_tlds:
return '.'.join(parts[-2:])
# Handle country code TLDs like .co.uk
if len(parts) >= 3 and parts[-2] in ['co', 'com', 'org', 'net', 'gov']:
return '.'.join(parts[-3:])
return domain
def _iso(value):
"""Serialize a datetime to ISO-8601 Z, or None. Coerces stray strings safely."""
dt = _coerce_datetime(value)
return dt.isoformat() + 'Z' if dt else None
def format_whois_response(whois_data: dict) -> dict:
"""Format WHOIS data for API response"""
creation_date = _coerce_datetime(whois_data.get('creation_date'))
expiration_date = _coerce_datetime(whois_data.get('expiration_date'))
age_days = (datetime.utcnow() - creation_date).days if creation_date else None
days_until_expiry = (expiration_date - datetime.utcnow()).days if expiration_date else None
return {
'creation_date': _iso(whois_data.get('creation_date')),
'expiration_date': _iso(whois_data.get('expiration_date')),
'updated_date': _iso(whois_data.get('updated_date')),
'registrar': whois_data.get('registrar'),
'age_days': age_days,
'days_until_expiry': days_until_expiry,
'status': whois_data.get('status', []),
'name_servers': whois_data.get('name_servers', []),
'dnssec': whois_data.get('dnssec'),
'registrant_org': whois_data.get('registrant_org'),
'registrant_country': whois_data.get('registrant_country'),
'is_registered': whois_data.get('is_registered'),
'data_source': whois_data.get('data_source', 'whois')
}
def determine_availability(whois_data: dict, dns_data: dict) -> dict:
"""Combine WHOIS and DNS signals into an explicit registration verdict.
WHOIS alone is unreliable for sparse registries (ROTLD .ro), so DNS
resolution (NS/A records) is used as a strong corroborating signal.
"""
whois_signal = whois_data.get('is_registered') if whois_data else None
dns_has_records = False
if dns_data:
dns_has_records = bool(
dns_data.get('ns_records') or dns_data.get('a_records') or
dns_data.get('aaaa_records') or dns_data.get('mx_records')
)
signals = {
'whois_has_data': whois_signal is True,
'dns_resolves': dns_has_records,
}
# Decision: any positive signal => registered. Confidence is high when
# WHOIS and DNS agree, low when relying on a single weak signal.
if whois_signal is True or dns_has_records:
is_registered = True
confidence = 'high' if (whois_signal is True and dns_has_records) else 'medium'
elif whois_signal is False:
is_registered = False
confidence = 'high' if not dns_has_records else 'low'
else:
# WHOIS inconclusive (None) and no DNS records => probably available.
is_registered = False
confidence = 'low'
return {
'is_registered': is_registered,
'is_available': not is_registered,
'confidence': confidence,
'signals': signals
}
def format_dns_response(dns_data: dict) -> dict:
"""Format DNS data for API response"""
return {
'a_records': dns_data.get('a_records', []),
'aaaa_records': dns_data.get('aaaa_records', []),
'mx_records': dns_data.get('mx_records', []),
'txt_records': dns_data.get('txt_records', []),
'ns_records': dns_data.get('ns_records', []),
'cname_records': dns_data.get('cname_records', []),
'soa_record': dns_data.get('soa_record'),
'has_spf': dns_data.get('has_spf', False),
'has_dkim': dns_data.get('has_dkim', False),
'has_dmarc': dns_data.get('has_dmarc', False),
'spf_record': dns_data.get('spf_record'),
'dmarc_record': dns_data.get('dmarc_record')
}
def format_ssl_response(ssl_data: dict) -> dict:
"""Format SSL data for API response"""
if not ssl_data.get('has_ssl'):
return {
'has_ssl': False,
'error': ssl_data.get('error')
}
valid_from = ssl_data.get('valid_from')
valid_until = ssl_data.get('valid_until')
return {
'has_ssl': True,
'is_valid': ssl_data.get('is_valid', False),
'is_self_signed': ssl_data.get('is_self_signed', False),
'is_expired': ssl_data.get('is_expired', False),
'is_wildcard': ssl_data.get('is_wildcard', False),
'issuer': ssl_data.get('issuer'),
'subject': ssl_data.get('subject'),
'valid_from': valid_from.isoformat() + 'Z' if valid_from and hasattr(valid_from, 'isoformat') else str(valid_from) if valid_from else None,
'valid_until': valid_until.isoformat() + 'Z' if valid_until and hasattr(valid_until, 'isoformat') else str(valid_until) if valid_until else None,
'days_until_expiry': ssl_data.get('days_until_expiry'),
'key_size': ssl_data.get('key_size'),
'signature_algorithm': ssl_data.get('signature_algorithm'),
'san': ssl_data.get('san', [])
}

9
ai_platform/modules/domain_check/api/app/routes/domain.py Normal file
View file

@ -0,0 +1,9 @@
"""Domain routes - stub for now"""
from flask import Blueprint, jsonify
domain_bp = Blueprint('domain', __name__)
@domain_bp.route('/domain/<string:domain>', methods=['GET'])
def get_domain(domain):
"""Get domain details - TODO: implement"""
return jsonify({'message': 'Not yet implemented', 'domain': domain}), 501

9
ai_platform/modules/domain_check/api/app/routes/search.py Normal file
View file

@ -0,0 +1,9 @@
"""Search routes - stub for now"""
from flask import Blueprint, jsonify
search_bp = Blueprint('search', __name__)
@search_bp.route('/search', methods=['GET'])
def search_domains():
"""Search domains - TODO: implement"""
return jsonify({'message': 'Not yet implemented'}), 501

9
ai_platform/modules/domain_check/api/app/routes/stats.py Normal file
View file

@ -0,0 +1,9 @@
"""Stats routes - stub for now"""
from flask import Blueprint, jsonify
stats_bp = Blueprint('stats', __name__)
@stats_bp.route('/stats', methods=['GET'])
def get_stats():
"""Get system statistics - TODO: implement"""
return jsonify({'message': 'Not yet implemented'}), 501

0
ai_platform/modules/domain_check/api/app/services/__init__.py Normal file
View file

297
ai_platform/modules/domain_check/api/app/services/blacklist_service.py Normal file
View file

@ -0,0 +1,297 @@
"""
Blacklist Checking Service - DNSBL, Spam lists, Reputation checks
"""
import logging
import socket
import dns.resolver
import requests
from typing import Dict, List, Optional
from concurrent.futures import ThreadPoolExecutor, as_completed
logger = logging.getLogger(__name__)
class BlacklistService:
"""Service for checking IP/domain against various blacklists"""
def __init__(self):
self.timeout = 5
self.resolver = dns.resolver.Resolver()
self.resolver.nameservers = ['8.8.8.8', '8.8.4.4']
self.resolver.timeout = 3
self.resolver.lifetime = 5
# DNSBL lists to check
self.dnsbl_lists = [
{'name': 'Spamhaus ZEN', 'zone': 'zen.spamhaus.org', 'type': 'spam'},
{'name': 'SpamCop', 'zone': 'bl.spamcop.net', 'type': 'spam'},
{'name': 'Barracuda', 'zone': 'b.barracudacentral.org', 'type': 'spam'},
{'name': 'SORBS', 'zone': 'dnsbl.sorbs.net', 'type': 'spam'},
{'name': 'URIBL', 'zone': 'multi.uribl.com', 'type': 'uri'},
{'name': 'SURBL', 'zone': 'multi.surbl.org', 'type': 'uri'},
{'name': 'Spamhaus DBL', 'zone': 'dbl.spamhaus.org', 'type': 'domain'},
{'name': 'URIBL Black', 'zone': 'black.uribl.com', 'type': 'uri'}
]
def check_ip(self, ip_address: str) -> Dict:
"""
Check IP address against multiple blacklists
Args:
ip_address: IP address to check
Returns:
Dictionary with blacklist check results
"""
result = {
'ip': ip_address,
'is_blacklisted': False,
'blacklist_count': 0,
'clean_count': 0,
'total_checked': 0,
'listings': [],
'clean_lists': [],
'check_errors': []
}
# Reverse IP for DNSBL query
reversed_ip = '.'.join(reversed(ip_address.split('.')))
# Check IP-based blacklists in parallel
ip_lists = [bl for bl in self.dnsbl_lists if bl['type'] in ['spam']]
with ThreadPoolExecutor(max_workers=8) as executor:
futures = {
executor.submit(self._check_dnsbl, reversed_ip, bl): bl
for bl in ip_lists
}
for future in as_completed(futures, timeout=15):
bl = futures[future]
result['total_checked'] += 1
try:
is_listed, response = future.result()
if is_listed:
result['is_blacklisted'] = True
result['blacklist_count'] += 1
result['listings'].append({
'list_name': bl['name'],
'list_zone': bl['zone'],
'response': response,
'type': bl['type']
})
else:
result['clean_count'] += 1
result['clean_lists'].append(bl['name'])
except Exception as e:
result['check_errors'].append({
'list': bl['name'],
'error': str(e)
})
return result
def check_domain(self, domain: str) -> Dict:
"""
Check domain against domain-based blacklists
Args:
domain: Domain name to check
Returns:
Dictionary with blacklist check results
"""
result = {
'domain': domain,
'is_blacklisted': False,
'blacklist_count': 0,
'clean_count': 0,
'total_checked': 0,
'listings': [],
'clean_lists': [],
'check_errors': []
}
# Check domain-based blacklists
domain_lists = [bl for bl in self.dnsbl_lists if bl['type'] in ['domain', 'uri']]
with ThreadPoolExecutor(max_workers=8) as executor:
futures = {
executor.submit(self._check_domain_bl, domain, bl): bl
for bl in domain_lists
}
for future in as_completed(futures, timeout=15):
bl = futures[future]
result['total_checked'] += 1
try:
is_listed, response = future.result()
if is_listed:
result['is_blacklisted'] = True
result['blacklist_count'] += 1
result['listings'].append({
'list_name': bl['name'],
'list_zone': bl['zone'],
'response': response,
'type': bl['type']
})
else:
result['clean_count'] += 1
result['clean_lists'].append(bl['name'])
except Exception as e:
result['check_errors'].append({
'list': bl['name'],
'error': str(e)
})
return result
def check_all(self, domain: str, ip_address: str) -> Dict:
"""
Check both domain and IP against all blacklists
Returns:
Combined blacklist check results
"""
result = {
'domain': domain,
'ip': ip_address,
'is_blacklisted': False,
'ip_blacklisted': False,
'domain_blacklisted': False,
'total_listings': 0,
'ip_check': None,
'domain_check': None,
'reputation_score': 100, # Start with perfect score
'risk_level': 'LOW'
}
# Check IP
if ip_address:
ip_result = self.check_ip(ip_address)
result['ip_check'] = ip_result
result['ip_blacklisted'] = ip_result['is_blacklisted']
result['total_listings'] += ip_result['blacklist_count']
# Check domain
domain_result = self.check_domain(domain)
result['domain_check'] = domain_result
result['domain_blacklisted'] = domain_result['is_blacklisted']
result['total_listings'] += domain_result['blacklist_count']
# Set overall blacklist status
result['is_blacklisted'] = result['ip_blacklisted'] or result['domain_blacklisted']
# Calculate reputation score
result['reputation_score'] = self._calculate_reputation_score(result)
result['risk_level'] = self._get_risk_level(result['reputation_score'])
return result
def _check_dnsbl(self, reversed_ip: str, blacklist: Dict) -> tuple:
"""Check reversed IP against a DNSBL"""
try:
query = f"{reversed_ip}.{blacklist['zone']}"
answers = self.resolver.resolve(query, 'A')
response = str(answers[0])
# 127.0.0.1 is an error code meaning "not authorized to query"
# Real blacklist matches return 127.0.0.2, 127.0.0.4, etc.
if response == '127.0.0.1':
logger.debug(f"DNSBL {blacklist['name']} returned error code 127.0.0.1 (not authorized)")
return False, None
# Valid blacklist match
return True, response
except dns.resolver.NXDOMAIN:
# Not listed
return False, None
except dns.resolver.NoAnswer:
return False, None
except dns.resolver.Timeout:
raise Exception('Timeout')
except Exception as e:
raise Exception(str(e))
def _check_domain_bl(self, domain: str, blacklist: Dict) -> tuple:
"""Check domain against a domain blacklist"""
try:
query = f"{domain}.{blacklist['zone']}"
answers = self.resolver.resolve(query, 'A')
response = str(answers[0])
# 127.0.0.1 is an error code meaning "not authorized to query"
# Real blacklist matches return 127.0.0.2, 127.0.0.4, etc.
# This is common with URIBL when querying from unregistered resolvers
if response == '127.0.0.1':
logger.debug(f"Domain BL {blacklist['name']} returned error code 127.0.0.1 (not authorized)")
return False, None
# Valid blacklist match
return True, response
except dns.resolver.NXDOMAIN:
return False, None
except dns.resolver.NoAnswer:
return False, None
except dns.resolver.Timeout:
raise Exception('Timeout')
except Exception as e:
raise Exception(str(e))
def _calculate_reputation_score(self, result: Dict) -> int:
"""
Calculate reputation score (100 = perfect, 0 = worst)
Each blacklist listing reduces the score
"""
score = 100
# IP blacklist hits are more severe
if result.get('ip_check'):
ip_listings = result['ip_check'].get('blacklist_count', 0)
score -= ip_listings * 25 # -25 per IP listing
# Domain blacklist hits
if result.get('domain_check'):
domain_listings = result['domain_check'].get('blacklist_count', 0)
score -= domain_listings * 20 # -20 per domain listing
return max(0, min(100, score))
def _get_risk_level(self, score: int) -> str:
"""Determine risk level from reputation score"""
if score >= 80:
return 'LOW'
elif score >= 60:
return 'MEDIUM'
elif score >= 40:
return 'HIGH'
else:
return 'CRITICAL'
def get_blacklist_score(self, blacklist_data: Dict) -> Dict:
"""
Get blacklist score for risk calculation (0 = good, 100 = bad)
This inverts the reputation score for consistency with other risk scores
"""
reputation = blacklist_data.get('reputation_score', 100)
score = 100 - reputation # Invert: 100 reputation = 0 risk
reasons = []
if blacklist_data.get('is_blacklisted'):
if blacklist_data.get('ip_blacklisted'):
reasons.append(f"IP on {blacklist_data['ip_check']['blacklist_count']} blacklist(s)")
if blacklist_data.get('domain_blacklisted'):
reasons.append(f"Domain on {blacklist_data['domain_check']['blacklist_count']} blacklist(s)")
else:
reasons.append('Not on any blacklists')
return {
'score': score,
'reasons': reasons,
'is_clean': score == 0,
'is_blacklisted': blacklist_data.get('is_blacklisted', False)
}

163
ai_platform/modules/domain_check/api/app/services/dns_service.py Normal file
View file

@ -0,0 +1,163 @@
"""
DNS Service - handles DNS record lookups
"""
import logging
import dns.resolver
from typing import Dict, List, Optional
logger = logging.getLogger(__name__)
class DNSService:
"""DNS lookup service class"""
def __init__(self, timeout: int = 10):
self.timeout = timeout
self.resolver = dns.resolver.Resolver()
self.resolver.timeout = timeout
self.resolver.lifetime = timeout
# Use public DNS servers (Google DNS) for reliability
self.resolver.nameservers = ['8.8.8.8', '8.8.4.4']
def lookup(self, domain: str) -> Optional[Dict]:
"""
Perform comprehensive DNS lookup
Args:
domain: Domain name to lookup
Returns:
Dictionary with DNS records or None
"""
try:
logger.info(f'DNS lookup for: {domain}')
dns_data = {
'domain': domain,
'a_records': self._get_a_records(domain),
'aaaa_records': self._get_aaaa_records(domain),
'mx_records': self._get_mx_records(domain),
'txt_records': self._get_txt_records(domain),
'ns_records': self._get_ns_records(domain),
'cname_records': self._get_cname_records(domain),
'soa_record': self._get_soa_record(domain),
'has_a_records': False,
'has_mx_records': False,
'has_txt_records': False,
'has_spf': False,
'has_dkim': False,
'has_dmarc': False
}
# Set boolean flags
dns_data['has_a_records'] = len(dns_data['a_records']) > 0
dns_data['has_mx_records'] = len(dns_data['mx_records']) > 0
dns_data['has_txt_records'] = len(dns_data['txt_records']) > 0
# Check for email security records
for txt in dns_data['txt_records']:
if txt.startswith('v=spf1'):
dns_data['has_spf'] = True
if 'dkim' in txt.lower():
dns_data['has_dkim'] = True
if 'v=DMARC1' in txt:
dns_data['has_dmarc'] = True
return dns_data
except Exception as e:
logger.error(f'DNS lookup failed for {domain}: {str(e)}')
return None
def _get_a_records(self, domain: str) -> List[str]:
"""Get A records (IPv4)"""
try:
answers = self.resolver.resolve(domain, 'A')
return [str(rdata) for rdata in answers]
except Exception as e:
logger.debug(f'No A records for {domain}: {e}')
return []
def _get_aaaa_records(self, domain: str) -> List[str]:
"""Get AAAA records (IPv6)"""
try:
answers = self.resolver.resolve(domain, 'AAAA')
return [str(rdata) for rdata in answers]
except Exception as e:
logger.debug(f'No AAAA records for {domain}: {e}')
return []
def _get_mx_records(self, domain: str) -> List[Dict]:
"""Get MX records (Mail servers)"""
try:
answers = self.resolver.resolve(domain, 'MX')
return [
{
'priority': rdata.preference,
'host': str(rdata.exchange).rstrip('.')
}
for rdata in answers
]
except Exception as e:
logger.debug(f'No MX records for {domain}: {e}')
return []
def _get_txt_records(self, domain: str) -> List[str]:
"""Get TXT records"""
try:
answers = self.resolver.resolve(domain, 'TXT')
records = []
for rdata in answers:
# TXT records can be split into multiple strings
txt = ''.join([s.decode('utf-8') if isinstance(s, bytes) else str(s) for s in rdata.strings])
records.append(txt)
return records
except Exception as e:
logger.debug(f'No TXT records for {domain}: {e}')
return []
def _get_ns_records(self, domain: str) -> List[str]:
"""Get NS records (Name servers)"""
try:
answers = self.resolver.resolve(domain, 'NS')
return [str(rdata).rstrip('.') for rdata in answers]
except Exception as e:
logger.debug(f'No NS records for {domain}: {e}')
return []
def _get_cname_records(self, domain: str) -> List[str]:
"""Get CNAME records"""
try:
answers = self.resolver.resolve(domain, 'CNAME')
return [str(rdata).rstrip('.') for rdata in answers]
except Exception as e:
logger.debug(f'No CNAME records for {domain}: {e}')
return []
def _get_soa_record(self, domain: str) -> Optional[Dict]:
"""Get SOA record (Start of Authority)"""
try:
answers = self.resolver.resolve(domain, 'SOA')
if answers:
soa = answers[0]
return {
'mname': str(soa.mname).rstrip('.'),
'rname': str(soa.rname).rstrip('.'),
'serial': soa.serial,
'refresh': soa.refresh,
'retry': soa.retry,
'expire': soa.expire,
'minimum': soa.minimum
}
except Exception as e:
logger.debug(f'No SOA record for {domain}: {e}')
return None
def check_dnssec(self, domain: str) -> bool:
"""Check if DNSSEC is enabled"""
try:
# Try to get DNSKEY records
self.resolver.resolve(domain, 'DNSKEY')
return True
except Exception:
return False

369
ai_platform/modules/domain_check/api/app/services/http_analysis_service.py Normal file
View file

@ -0,0 +1,369 @@
"""
HTTP Analysis Service - Headers, Technology Detection, Security Headers
"""
import logging
import re
import requests
from typing import Dict, List, Optional
from urllib.parse import urlparse
logger = logging.getLogger(__name__)
class HTTPAnalysisService:
"""Service for analyzing HTTP responses and detecting technologies"""
def __init__(self):
self.timeout = 15
self.user_agent = 'Mozilla/5.0 (compatible; DomainCheck/1.0; +https://domain-check.local)'
def analyze(self, domain: str) -> Dict:
"""
Perform comprehensive HTTP analysis
Args:
domain: Domain name to analyze
Returns:
Dictionary with HTTP analysis results
"""
result = {
'domain': domain,
'http_status': None,
'https_status': None,
'has_https': False,
'http_to_https_redirect': False,
'final_url': None,
'redirect_chain': [],
'response_time_ms': None,
'server': None,
'powered_by': None,
'security_headers': {},
'missing_security_headers': [],
'cookies': [],
'technologies': [],
'cms': None,
'frameworks': [],
'has_robots_txt': False,
'has_sitemap': False,
'has_favicon': False,
'error': None
}
try:
# Test HTTPS first
https_result = self._check_url(f'https://{domain}')
if https_result.get('success'):
result['has_https'] = True
result['https_status'] = https_result.get('status_code')
result['final_url'] = https_result.get('final_url')
result['redirect_chain'] = https_result.get('redirect_chain', [])
result['response_time_ms'] = https_result.get('response_time_ms')
result['server'] = https_result.get('server')
result['powered_by'] = https_result.get('powered_by')
result['security_headers'] = https_result.get('security_headers', {})
result['missing_security_headers'] = https_result.get('missing_security_headers', [])
result['cookies'] = https_result.get('cookies', [])
# Detect technologies from response
if https_result.get('body'):
tech = self._detect_technologies(https_result['body'], https_result.get('headers', {}))
result['technologies'] = tech.get('technologies', [])
result['cms'] = tech.get('cms')
result['frameworks'] = tech.get('frameworks', [])
# Test HTTP
http_result = self._check_url(f'http://{domain}')
if http_result.get('success'):
result['http_status'] = http_result.get('status_code')
# Check if HTTP redirects to HTTPS
if http_result.get('final_url', '').startswith('https://'):
result['http_to_https_redirect'] = True
# Check for common files
result['has_robots_txt'] = self._check_file_exists(f'https://{domain}/robots.txt')
result['has_sitemap'] = self._check_file_exists(f'https://{domain}/sitemap.xml')
result['has_favicon'] = self._check_file_exists(f'https://{domain}/favicon.ico')
except Exception as e:
logger.error(f"HTTP analysis failed for {domain}: {e}")
result['error'] = str(e)
return result
def _check_url(self, url: str) -> Dict:
"""Check a URL and gather response data"""
result = {
'success': False,
'url': url,
'status_code': None,
'final_url': None,
'redirect_chain': [],
'response_time_ms': None,
'server': None,
'powered_by': None,
'security_headers': {},
'missing_security_headers': [],
'cookies': [],
'headers': {},
'body': None
}
try:
response = requests.get(
url,
timeout=self.timeout,
allow_redirects=True,
headers={'User-Agent': self.user_agent},
verify=True
)
result['success'] = True
result['status_code'] = response.status_code
result['final_url'] = response.url
result['response_time_ms'] = int(response.elapsed.total_seconds() * 1000)
result['headers'] = dict(response.headers)
# Get redirect chain
if response.history:
result['redirect_chain'] = [
{'url': r.url, 'status': r.status_code}
for r in response.history
]
# Extract server info
result['server'] = response.headers.get('Server')
result['powered_by'] = response.headers.get('X-Powered-By')
# Analyze security headers
result['security_headers'], result['missing_security_headers'] = \
self._analyze_security_headers(response.headers)
# Analyze cookies
result['cookies'] = self._analyze_cookies(response.cookies)
# Get body for technology detection
result['body'] = response.text[:50000] # Limit body size
except requests.exceptions.SSLError as e:
result['error'] = f'SSL Error: {str(e)}'
except requests.exceptions.ConnectionError as e:
result['error'] = f'Connection Error: {str(e)}'
except requests.exceptions.Timeout:
result['error'] = 'Timeout'
except Exception as e:
result['error'] = str(e)
return result
def _analyze_security_headers(self, headers) -> tuple:
"""Analyze security headers"""
security_headers = {}
missing = []
# Define required security headers
required_headers = {
'Strict-Transport-Security': 'HSTS - Forces HTTPS',
'X-Frame-Options': 'Prevents clickjacking',
'X-Content-Type-Options': 'Prevents MIME sniffing',
'X-XSS-Protection': 'XSS filtering (legacy)',
'Content-Security-Policy': 'CSP - Controls resource loading',
'Referrer-Policy': 'Controls referrer information',
'Permissions-Policy': 'Controls browser features'
}
for header, description in required_headers.items():
value = headers.get(header)
if value:
security_headers[header] = {
'value': value,
'description': description,
'present': True
}
else:
missing.append({
'header': header,
'description': description,
'severity': self._get_header_severity(header)
})
return security_headers, missing
def _get_header_severity(self, header: str) -> str:
"""Get severity level for missing header"""
critical = ['Strict-Transport-Security', 'Content-Security-Policy']
high = ['X-Frame-Options', 'X-Content-Type-Options']
if header in critical:
return 'CRITICAL'
elif header in high:
return 'HIGH'
return 'MEDIUM'
def _analyze_cookies(self, cookies) -> List[Dict]:
"""Analyze cookies for security attributes"""
analyzed = []
for cookie in cookies:
cookie_info = {
'name': cookie.name,
'secure': cookie.secure,
'httponly': cookie.has_nonstandard_attr('HttpOnly'),
'samesite': cookie.get_nonstandard_attr('SameSite'),
'issues': []
}
# Check for security issues
if not cookie.secure:
cookie_info['issues'].append('Missing Secure flag')
if not cookie_info['httponly']:
cookie_info['issues'].append('Missing HttpOnly flag')
if not cookie_info['samesite']:
cookie_info['issues'].append('Missing SameSite attribute')
analyzed.append(cookie_info)
return analyzed
def _detect_technologies(self, body: str, headers: Dict) -> Dict:
"""Detect technologies from response body and headers"""
result = {
'technologies': [],
'cms': None,
'frameworks': []
}
body_lower = body.lower()
# CMS Detection
cms_patterns = {
'WordPress': [
'wp-content', 'wp-includes', 'wordpress',
'<meta name="generator" content="WordPress'
],
'Joomla': ['joomla', '/media/jui/', '/components/com_'],
'Drupal': ['drupal', '/sites/default/files/', 'Drupal.settings'],
'Magento': ['magento', 'mage/', '/skin/frontend/'],
'Shopify': ['shopify', 'cdn.shopify.com'],
'Wix': ['wix.com', '_wix_browser_sess'],
'Squarespace': ['squarespace', 'static.squarespace.com']
}
for cms, patterns in cms_patterns.items():
for pattern in patterns:
if pattern.lower() in body_lower:
result['cms'] = cms
result['technologies'].append(cms)
break
if result['cms']:
break
# Framework detection
framework_patterns = {
'React': ['react', '_reactRootContainer', 'data-reactroot'],
'Vue.js': ['vue', 'data-v-', '__vue__'],
'Angular': ['ng-', 'angular', 'ng-app', 'ng-controller'],
'jQuery': ['jquery', 'jQuery'],
'Bootstrap': ['bootstrap', 'class="container', 'class="row'],
'Tailwind CSS': ['tailwind', 'class="flex', 'class="grid'],
'Laravel': ['laravel', 'csrf-token'],
'Django': ['csrfmiddlewaretoken', 'django'],
'Express': ['express'],
'Next.js': ['next.js', '__NEXT_DATA__', '_next/'],
'Nuxt.js': ['nuxt', '__NUXT__']
}
for framework, patterns in framework_patterns.items():
for pattern in patterns:
if pattern.lower() in body_lower:
if framework not in result['frameworks']:
result['frameworks'].append(framework)
result['technologies'].append(framework)
break
# Additional technologies from meta tags
generator_match = re.search(r'<meta[^>]+name=["\']generator["\'][^>]+content=["\']([^"\']+)["\']', body, re.I)
if generator_match:
generator = generator_match.group(1)
result['technologies'].append(f'Generator: {generator}')
# Server-side detection from headers
server = headers.get('Server', '').lower()
if 'nginx' in server:
result['technologies'].append('Nginx')
elif 'apache' in server:
result['technologies'].append('Apache')
elif 'iis' in server:
result['technologies'].append('Microsoft IIS')
powered_by = headers.get('X-Powered-By', '').lower()
if 'php' in powered_by:
result['technologies'].append('PHP')
elif 'asp.net' in powered_by:
result['technologies'].append('ASP.NET')
return result
def _check_file_exists(self, url: str) -> bool:
"""Check if a file exists at URL"""
try:
response = requests.head(
url,
timeout=5,
allow_redirects=True,
headers={'User-Agent': self.user_agent}
)
return response.status_code == 200
except:
return False
def get_security_score(self, http_data: Dict) -> Dict:
"""
Calculate HTTP security score
Returns:
Dict with score (0-100) and reasons
"""
score = 0
reasons = []
# No HTTPS = very bad
if not http_data.get('has_https'):
score += 40
reasons.append('No HTTPS support')
# No HTTP to HTTPS redirect
if http_data.get('has_https') and not http_data.get('http_to_https_redirect'):
score += 15
reasons.append('No HTTP to HTTPS redirect')
# Missing security headers
missing_headers = http_data.get('missing_security_headers', [])
for header in missing_headers:
if header.get('severity') == 'CRITICAL':
score += 15
reasons.append(f"Missing: {header['header']}")
elif header.get('severity') == 'HIGH':
score += 10
reasons.append(f"Missing: {header['header']}")
else:
score += 5
# Cookie issues
cookies = http_data.get('cookies', [])
for cookie in cookies:
if cookie.get('issues'):
score += 5 * len(cookie['issues'])
for issue in cookie['issues']:
reasons.append(f"Cookie '{cookie['name']}': {issue}")
if score == 0:
reasons.append('Good HTTP security configuration')
return {
'score': min(100, score),
'reasons': reasons[:10], # Limit reasons
'is_secure': score <= 20,
'needs_improvement': score > 20 and score <= 50,
'is_insecure': score > 50
}

233
ai_platform/modules/domain_check/api/app/services/ip_intelligence_service.py Normal file
View file

@ -0,0 +1,233 @@
"""
IP Intelligence Service - Geolocation, ASN, Reverse DNS, Hosting Info
"""
import logging
import socket
import requests
from typing import Dict, Optional
from concurrent.futures import ThreadPoolExecutor, as_completed
logger = logging.getLogger(__name__)
class IPIntelligenceService:
"""Service for gathering IP intelligence data"""
def __init__(self):
self.timeout = 10
self.ipinfo_url = "https://ipinfo.io/{ip}/json"
def lookup(self, ip_address: str) -> Dict:
"""
Get comprehensive IP intelligence
Args:
ip_address: IP address to lookup
Returns:
Dictionary with IP intelligence data
"""
result = {
'ip': ip_address,
'reverse_dns': None,
'geolocation': None,
'asn': None,
'isp': None,
'organization': None,
'is_datacenter': False,
'is_residential': False,
'hostname': None,
'city': None,
'region': None,
'country': None,
'country_code': None,
'coordinates': None,
'timezone': None,
'data_source': 'ipinfo.io'
}
try:
# Run lookups in parallel
with ThreadPoolExecutor(max_workers=2) as executor:
futures = {
executor.submit(self._get_reverse_dns, ip_address): 'reverse_dns',
executor.submit(self._get_ipinfo, ip_address): 'ipinfo'
}
for future in as_completed(futures, timeout=15):
lookup_type = futures[future]
try:
data = future.result()
if lookup_type == 'reverse_dns':
result['reverse_dns'] = data
result['hostname'] = data
elif lookup_type == 'ipinfo' and data:
result.update(data)
except Exception as e:
logger.warning(f"IP lookup {lookup_type} failed: {e}")
# Determine if datacenter or residential
result['is_datacenter'] = self._is_datacenter(result)
result['is_residential'] = not result['is_datacenter']
except Exception as e:
logger.error(f"IP intelligence lookup failed for {ip_address}: {e}")
result['error'] = str(e)
return result
def _get_reverse_dns(self, ip_address: str) -> Optional[str]:
"""Get reverse DNS (PTR record)"""
try:
hostname, _, _ = socket.gethostbyaddr(ip_address)
return hostname
except (socket.herror, socket.gaierror):
return None
def _get_ipinfo(self, ip_address: str) -> Optional[Dict]:
"""Get IP info from ipinfo.io"""
try:
response = requests.get(
self.ipinfo_url.format(ip=ip_address),
timeout=self.timeout
)
if response.status_code == 200:
data = response.json()
# Parse ASN from org field (format: "AS8708 DIGI ROMANIA S.A.")
org = data.get('org', '')
asn = None
isp = None
if org:
parts = org.split(' ', 1)
if parts[0].startswith('AS'):
asn = parts[0]
isp = parts[1] if len(parts) > 1 else None
else:
isp = org
# Parse coordinates
loc = data.get('loc', '')
coordinates = None
if loc:
try:
lat, lon = loc.split(',')
coordinates = {
'latitude': float(lat),
'longitude': float(lon)
}
except:
pass
return {
'hostname': data.get('hostname'),
'city': data.get('city'),
'region': data.get('region'),
'country': self._get_country_name(data.get('country')),
'country_code': data.get('country'),
'coordinates': coordinates,
'timezone': data.get('timezone'),
'asn': asn,
'isp': isp,
'organization': isp,
'postal': data.get('postal')
}
except Exception as e:
logger.warning(f"ipinfo.io lookup failed: {e}")
return None
def _is_datacenter(self, data: Dict) -> bool:
"""Determine if IP is likely a datacenter/hosting IP"""
indicators = []
# Check ISP/organization for hosting keywords
org = (data.get('organization') or '').lower()
isp = (data.get('isp') or '').lower()
hostname = (data.get('hostname') or '').lower()
hosting_keywords = [
'hosting', 'server', 'cloud', 'datacenter', 'data center',
'hetzner', 'ovh', 'digitalocean', 'linode', 'vultr', 'aws',
'amazon', 'google', 'microsoft', 'azure', 'contabo', 'hostinger',
'godaddy', 'bluehost', 'namecheap', 'maghost', 'simpliq', 'host365'
]
for keyword in hosting_keywords:
if keyword in org or keyword in isp:
return True
# Check hostname patterns
if hostname:
hosting_patterns = ['static', 'vps', 'server', 'host', 'cloud', 'dedicated']
for pattern in hosting_patterns:
if pattern in hostname:
return True
return False
def _get_country_name(self, code: str) -> str:
"""Convert country code to name"""
countries = {
'RO': 'Romania',
'US': 'United States',
'GB': 'United Kingdom',
'DE': 'Germany',
'FR': 'France',
'NL': 'Netherlands',
'UA': 'Ukraine',
'RU': 'Russia',
'CN': 'China',
'IN': 'India',
'JP': 'Japan',
'BR': 'Brazil',
'CA': 'Canada',
'AU': 'Australia'
}
return countries.get(code, code)
def get_hosting_score(self, ip_data: Dict) -> Dict:
"""
Calculate hosting provider reputation score
Returns:
Dict with score (0-100) and reasons
"""
score = 50 # Neutral baseline
reasons = []
isp = (ip_data.get('isp') or '').lower()
country_code = ip_data.get('country_code', '')
# Trusted hosting providers (lower score = better)
trusted_hosts = ['google', 'amazon', 'microsoft', 'cloudflare', 'akamai']
for host in trusted_hosts:
if host in isp:
score = 20
reasons.append(f'Trusted provider: {isp}')
break
# Romanian ISPs (neutral)
ro_isps = ['digi', 'rcs', 'rds', 'telekom', 'orange', 'vodafone']
for isp_name in ro_isps:
if isp_name in isp:
score = 40
reasons.append(f'Romanian ISP: {isp}')
break
# Check for reverse DNS mismatch (higher risk)
if ip_data.get('is_datacenter') and not ip_data.get('reverse_dns'):
score += 20
reasons.append('Datacenter IP without reverse DNS')
# High-risk countries
high_risk_countries = ['RU', 'CN', 'KP', 'IR', 'NG']
if country_code in high_risk_countries:
score += 30
reasons.append(f'High-risk country: {country_code}')
return {
'score': min(100, max(0, score)),
'reasons': reasons,
'is_trusted': score <= 30,
'is_suspicious': score >= 70
}

374
ai_platform/modules/domain_check/api/app/services/mail_intelligence_service.py Normal file
View file

@ -0,0 +1,374 @@
"""
Mail Intelligence Service - deep analysis of a domain's email infrastructure.
Covers what a plain MX/TXT dump does not:
- SPF parsing (policy qualifier, include chain, DNS-lookup budget per RFC 7208)
- DMARC policy (_dmarc) parsing (p/sp/pct/rua/aspf/adkim)
- DKIM selector discovery (probes common selectors)
- MX provider fingerprinting + STARTTLS reachability
- MTA-STS (RFC 8461), TLS-RPT (RFC 8460), DANE/TLSA (RFC 7672)
- Optional SMTP-level mailbox + catch-all probing (RCPT TO), degrades
gracefully when outbound port 25 is blocked.
Everything is best-effort and never raises: a failed probe becomes a recorded
'unknown'/False signal, so a single hiccup cannot 500 the parent request.
"""
import logging
import smtplib
import socket
import ssl
from concurrent.futures import ThreadPoolExecutor, as_completed
from typing import Dict, List, Optional
import dns.resolver
import requests
logger = logging.getLogger(__name__)
# Common DKIM selectors used by major providers / tooling.
COMMON_DKIM_SELECTORS = [
'default', 'google', 'selector1', 'selector2', 'k1', 'k2', 'dkim',
'mail', 'smtp', 's1', 's2', 'mandrill', 'mailjet', 'sendgrid',
'zoho', 'protonmail', 'protonmail2', 'fm1', 'fm2', 'fm3', 'mxvault',
]
# MX hostname substrings -> human provider name.
MX_PROVIDERS = [
('google.com', 'Google Workspace'),
('googlemail.com', 'Google Workspace'),
('outlook.com', 'Microsoft 365'),
('protection.outlook', 'Microsoft 365'),
('zoho', 'Zoho Mail'),
('protonmail', 'Proton Mail'),
('proton.me', 'Proton Mail'),
('mail.ru', 'Mail.ru'),
('yandex', 'Yandex Mail'),
('mimecast', 'Mimecast'),
('pphosted', 'Proofpoint'),
('messagelabs', 'Broadcom/Symantec'),
('mailgun', 'Mailgun'),
('sendgrid', 'SendGrid'),
('amazonaws', 'Amazon SES/WorkMail'),
('secureserver.net', 'GoDaddy'),
('one.com', 'one.com'),
('hostinger', 'Hostinger'),
('gandi', 'Gandi'),
('ovh', 'OVH'),
('rotld', 'ROTLD'),
]
class MailIntelligenceService:
"""Deep email-infrastructure analysis for a domain."""
def __init__(self, dns_timeout: int = 6, smtp_timeout: int = 8, http_timeout: int = 6):
self.smtp_timeout = smtp_timeout
self.http_timeout = http_timeout
self.resolver = dns.resolver.Resolver()
self.resolver.timeout = dns_timeout
self.resolver.lifetime = dns_timeout
self.resolver.nameservers = ['8.8.8.8', '1.1.1.1']
# -- public ------------------------------------------------------------
def analyze(self, domain: str, mx_records: Optional[List[Dict]] = None,
txt_records: Optional[List[str]] = None,
check_smtp: bool = False) -> Dict:
domain = domain.lower().strip()
if mx_records is None:
mx_records = self._resolve_mx(domain)
if txt_records is None:
txt_records = self._resolve_txt(domain)
spf = self._analyze_spf(domain, txt_records)
dmarc = self._analyze_dmarc(domain)
dkim = self._discover_dkim(domain)
mta_sts = self._analyze_mta_sts(domain)
tls_rpt = self._analyze_tls_rpt(domain)
mx = self._analyze_mx(mx_records, check_smtp=check_smtp)
result = {
'domain': domain,
'has_mail': bool(mx_records),
'mx': mx,
'spf': spf,
'dmarc': dmarc,
'dkim': dkim,
'mta_sts': mta_sts,
'tls_rpt': tls_rpt,
}
if check_smtp:
result['deliverability'] = self._probe_deliverability(domain, mx_records)
result['provider'] = mx.get('provider') if mx else None
result.update(self._grade(result))
return result
# -- DNS helpers -------------------------------------------------------
def _resolve_mx(self, domain: str) -> List[Dict]:
try:
answers = self.resolver.resolve(domain, 'MX')
return sorted(
[{'priority': r.preference, 'host': str(r.exchange).rstrip('.')} for r in answers],
key=lambda m: m['priority']
)
except Exception:
return []
def _resolve_txt(self, name: str) -> List[str]:
try:
answers = self.resolver.resolve(name, 'TXT')
out = []
for r in answers:
out.append(''.join(
s.decode('utf-8', 'ignore') if isinstance(s, bytes) else str(s)
for s in r.strings
))
return out
except Exception:
return []
# -- SPF ---------------------------------------------------------------
def _analyze_spf(self, domain: str, txt_records: List[str]) -> Dict:
spf_record = next((t for t in (txt_records or []) if t.lower().startswith('v=spf1')), None)
if not spf_record:
return {'present': False, 'record': None, 'policy': None,
'lookup_count': 0, 'includes': [], 'issues': ['No SPF record']}
tokens = spf_record.split()
includes, lookup_terms, policy = [], 0, 'neutral'
# Mechanisms that cost a DNS lookup (RFC 7208 §4.6.4, max 10).
lookup_mechs = ('include:', 'a', 'mx', 'ptr', 'exists:', 'redirect=')
for tok in tokens[1:]:
low = tok.lower()
if low.startswith('include:'):
includes.append(tok.split(':', 1)[1])
lookup_terms += 1
elif low.startswith(lookup_mechs) or low in ('a', 'mx', 'ptr'):
lookup_terms += 1
if low.endswith('all'):
qual = low[0] if low[0] in '-~?+' else '+'
policy = {'-': 'fail (strict)', '~': 'softfail', '?': 'neutral',
'+': 'pass (insecure +all)'}.get(qual, 'neutral')
issues = []
if lookup_terms > 10:
issues.append(f'Exceeds 10 DNS-lookup limit ({lookup_terms}) -> SPF permerror')
if policy.startswith('pass'):
issues.append('+all allows anyone to send as this domain')
if policy == 'neutral':
issues.append('?all provides no protection')
return {'present': True, 'record': spf_record, 'policy': policy,
'lookup_count': lookup_terms, 'includes': includes, 'issues': issues}
# -- DMARC -------------------------------------------------------------
def _analyze_dmarc(self, domain: str) -> Dict:
records = self._resolve_txt(f'_dmarc.{domain}')
rec = next((t for t in records if t.lower().startswith('v=dmarc1')), None)
if not rec:
return {'present': False, 'record': None, 'policy': None,
'pct': None, 'rua': [], 'issues': ['No DMARC record']}
tags = {}
for part in rec.split(';'):
if '=' in part:
k, v = part.split('=', 1)
tags[k.strip().lower()] = v.strip()
policy = tags.get('p')
issues = []
if policy == 'none':
issues.append('p=none is monitor-only, does not block spoofing')
if not policy:
issues.append('Missing p= tag')
if not tags.get('rua'):
issues.append('No aggregate reporting (rua) configured')
return {'present': True, 'record': rec, 'policy': policy,
'subdomain_policy': tags.get('sp'),
'pct': tags.get('pct', '100'),
'rua': [a.strip() for a in tags.get('rua', '').split(',') if a.strip()],
'alignment': {'aspf': tags.get('aspf', 'r'), 'adkim': tags.get('adkim', 'r')},
'issues': issues}
# -- DKIM --------------------------------------------------------------
def _discover_dkim(self, domain: str) -> Dict:
found = []
def probe(sel):
recs = self._resolve_txt(f'{sel}._domainkey.{domain}')
for r in recs:
if 'v=dkim1' in r.lower() or 'p=' in r:
return sel
return None
with ThreadPoolExecutor(max_workers=8) as ex:
futures = {ex.submit(probe, s): s for s in COMMON_DKIM_SELECTORS}
for fut in as_completed(futures):
try:
sel = fut.result()
if sel:
found.append(sel)
except Exception:
pass
return {'present': bool(found), 'selectors_found': sorted(found),
'note': 'Probes common selectors only; absence is not proof of no DKIM'}
# -- MTA-STS / TLS-RPT / DANE ------------------------------------------
def _analyze_mta_sts(self, domain: str) -> Dict:
txt = self._resolve_txt(f'_mta-sts.{domain}')
has_dns = any('v=stsv1' in t.lower() for t in txt)
policy = None
mode = None
if has_dns:
try:
resp = requests.get(
f'https://mta-sts.{domain}/.well-known/mta-sts.txt',
timeout=self.http_timeout, allow_redirects=False
)
if resp.status_code == 200 and 'version' in resp.text.lower():
policy = resp.text[:2000]
for line in resp.text.splitlines():
if line.lower().startswith('mode:'):
mode = line.split(':', 1)[1].strip()
except Exception:
pass
return {'present': has_dns, 'mode': mode, 'policy_fetched': policy is not None}
def _analyze_tls_rpt(self, domain: str) -> Dict:
txt = self._resolve_txt(f'_smtp._tls.{domain}')
rec = next((t for t in txt if 'v=tlsrptv1' in t.lower()), None)
return {'present': rec is not None, 'record': rec}
def _check_tlsa(self, mx_host: str) -> bool:
try:
self.resolver.resolve(f'_25._tcp.{mx_host}', 'TLSA')
return True
except Exception:
return False
# -- MX analysis -------------------------------------------------------
def _provider_for(self, host: str) -> Optional[str]:
h = host.lower()
for needle, name in MX_PROVIDERS:
if needle in h:
return name
return None
def _analyze_mx(self, mx_records: List[Dict], check_smtp: bool) -> Dict:
if not mx_records:
return {'count': 0, 'hosts': [], 'provider': None, 'starttls': None}
provider = None
hosts_out = []
def inspect(mx):
host = mx['host']
info = {
'host': host, 'priority': mx['priority'],
'provider': self._provider_for(host),
'addresses': self._resolve_addresses(host),
'dane_tlsa': self._check_tlsa(host),
}
if check_smtp:
info.update(self._smtp_starttls(host))
return info
with ThreadPoolExecutor(max_workers=min(5, len(mx_records))) as ex:
for info in ex.map(inspect, mx_records[:5]):
hosts_out.append(info)
if not provider and info.get('provider'):
provider = info['provider']
starttls = None
if check_smtp:
tls_flags = [h.get('starttls') for h in hosts_out if 'starttls' in h]
if tls_flags:
starttls = all(tls_flags)
return {'count': len(mx_records), 'hosts': hosts_out,
'provider': provider, 'starttls': starttls,
'dane': any(h.get('dane_tlsa') for h in hosts_out)}
def _resolve_addresses(self, host: str) -> List[str]:
out = []
for rtype in ('A', 'AAAA'):
try:
out += [str(r) for r in self.resolver.resolve(host, rtype)]
except Exception:
pass
return out
def _smtp_starttls(self, host: str) -> Dict:
"""Connect on :25 and check STARTTLS. Degrades gracefully if blocked."""
try:
with smtplib.SMTP(host, 25, timeout=self.smtp_timeout) as smtp:
banner = smtp.ehlo()
supports_tls = smtp.has_extn('starttls')
tls_ok = False
if supports_tls:
try:
ctx = ssl.create_default_context()
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE
smtp.starttls(context=ctx)
tls_ok = True
except Exception:
tls_ok = False
return {'reachable': True, 'starttls': supports_tls,
'starttls_negotiated': tls_ok,
'banner_code': banner[0] if banner else None}
except (socket.timeout, ConnectionRefusedError, OSError) as e:
return {'reachable': False, 'starttls': False,
'reason': f'port 25 unreachable from server ({type(e).__name__})'}
# -- optional deliverability probe ------------------------------------
def _probe_deliverability(self, domain: str, mx_records: List[Dict]) -> Dict:
"""SMTP RCPT probe for catch-all detection. Best-effort; many networks
block outbound :25 and many servers grey-list, so results are advisory."""
if not mx_records:
return {'tested': False, 'reason': 'no MX'}
host = mx_records[0]['host']
try:
with smtplib.SMTP(host, 25, timeout=self.smtp_timeout) as smtp:
smtp.ehlo()
smtp.mail('probe@example.com')
# Random-looking address: if accepted, server is catch-all.
code_random, _ = smtp.rcpt(f'zz-no-such-user-9281@{domain}')
catch_all = code_random in (250, 251)
return {'tested': True, 'catch_all': catch_all,
'rcpt_code': code_random,
'note': 'Advisory only; greylisting/anti-harvesting can skew results'}
except (socket.timeout, ConnectionRefusedError, OSError) as e:
return {'tested': False, 'reason': f'port 25 blocked/unreachable ({type(e).__name__})'}
except Exception as e:
return {'tested': False, 'reason': str(e)}
# -- grading -----------------------------------------------------------
def _grade(self, r: Dict) -> Dict:
"""0-100 email-security score (higher = better) + letter grade + summary."""
score = 0
if r['spf'].get('present'):
score += 20
if r['spf'].get('policy', '').startswith('fail'):
score += 10
if r['dmarc'].get('present'):
score += 20
if r['dmarc'].get('policy') in ('quarantine', 'reject'):
score += 15
if r['dkim'].get('present'):
score += 15
if r['mta_sts'].get('present'):
score += 10
if r['mta_sts'].get('mode') == 'enforce':
score += 5
if r['tls_rpt'].get('present'):
score += 5
score = min(score, 100)
grade = ('A' if score >= 85 else 'B' if score >= 70 else
'C' if score >= 50 else 'D' if score >= 30 else 'F')
return {'mail_security_score': score, 'mail_security_grade': grade}

265
ai_platform/modules/domain_check/api/app/services/port_scan_service.py Normal file
View file

@ -0,0 +1,265 @@
"""
Port Scanning Service - Detect open ports and services
"""
import logging
import socket
from typing import Dict, List, Optional
from concurrent.futures import ThreadPoolExecutor, as_completed
logger = logging.getLogger(__name__)
class PortScanService:
"""Service for port scanning and service detection"""
def __init__(self):
self.timeout = 3
# Common ports to scan with service info
self.common_ports = {
21: {'service': 'FTP', 'category': 'file_transfer', 'risk': 'medium'},
22: {'service': 'SSH', 'category': 'remote_access', 'risk': 'low'},
23: {'service': 'Telnet', 'category': 'remote_access', 'risk': 'critical'},
25: {'service': 'SMTP', 'category': 'email', 'risk': 'low'},
53: {'service': 'DNS', 'category': 'infrastructure', 'risk': 'low'},
80: {'service': 'HTTP', 'category': 'web', 'risk': 'low'},
110: {'service': 'POP3', 'category': 'email', 'risk': 'medium'},
111: {'service': 'RPCBind', 'category': 'infrastructure', 'risk': 'high'},
135: {'service': 'MS-RPC', 'category': 'windows', 'risk': 'high'},
139: {'service': 'NetBIOS', 'category': 'windows', 'risk': 'high'},
143: {'service': 'IMAP', 'category': 'email', 'risk': 'low'},
443: {'service': 'HTTPS', 'category': 'web', 'risk': 'low'},
445: {'service': 'SMB', 'category': 'windows', 'risk': 'critical'},
465: {'service': 'SMTPS', 'category': 'email', 'risk': 'low'},
587: {'service': 'SMTP Submission', 'category': 'email', 'risk': 'low'},
993: {'service': 'IMAPS', 'category': 'email', 'risk': 'low'},
995: {'service': 'POP3S', 'category': 'email', 'risk': 'low'},
1433: {'service': 'MS-SQL', 'category': 'database', 'risk': 'critical'},
1521: {'service': 'Oracle DB', 'category': 'database', 'risk': 'critical'},
1723: {'service': 'PPTP VPN', 'category': 'vpn', 'risk': 'medium'},
3306: {'service': 'MySQL', 'category': 'database', 'risk': 'critical'},
3389: {'service': 'RDP', 'category': 'remote_access', 'risk': 'high'},
5432: {'service': 'PostgreSQL', 'category': 'database', 'risk': 'critical'},
5900: {'service': 'VNC', 'category': 'remote_access', 'risk': 'high'},
6379: {'service': 'Redis', 'category': 'database', 'risk': 'critical'},
8080: {'service': 'HTTP Proxy', 'category': 'web', 'risk': 'medium'},
8443: {'service': 'HTTPS Alt', 'category': 'web', 'risk': 'low'},
27017: {'service': 'MongoDB', 'category': 'database', 'risk': 'critical'},
}
# Dangerous ports that should never be exposed
self.dangerous_ports = [23, 135, 139, 445, 1433, 1521, 3306, 3389, 5432, 5900, 6379, 27017]
def scan(self, ip_address: str, ports: List[int] = None) -> Dict:
"""
Scan ports on IP address
Args:
ip_address: IP address to scan
ports: List of ports to scan (default: common ports)
Returns:
Dictionary with scan results
"""
if ports is None:
ports = list(self.common_ports.keys())
result = {
'ip': ip_address,
'total_scanned': len(ports),
'open_ports': [],
'closed_ports': [],
'filtered_ports': [],
'dangerous_open': [],
'services_detected': [],
'categories': {},
'security_issues': [],
'scan_summary': {}
}
# Scan ports in parallel
with ThreadPoolExecutor(max_workers=30) as executor:
futures = {
executor.submit(self._scan_port, ip_address, port): port
for port in ports
}
for future in as_completed(futures, timeout=60):
port = futures[future]
try:
status, banner = future.result()
port_info = self.common_ports.get(port, {
'service': 'Unknown',
'category': 'other',
'risk': 'unknown'
})
port_data = {
'port': port,
'service': port_info['service'],
'category': port_info['category'],
'risk_level': port_info['risk'],
'banner': banner
}
if status == 'open':
result['open_ports'].append(port_data)
result['services_detected'].append(port_info['service'])
# Track by category
category = port_info['category']
if category not in result['categories']:
result['categories'][category] = []
result['categories'][category].append(port)
# Check if dangerous port
if port in self.dangerous_ports:
result['dangerous_open'].append(port_data)
result['security_issues'].append({
'severity': 'CRITICAL' if port_info['risk'] == 'critical' else 'HIGH',
'port': port,
'service': port_info['service'],
'issue': f"Dangerous service {port_info['service']} exposed on port {port}"
})
elif status == 'closed':
result['closed_ports'].append(port)
else:
result['filtered_ports'].append(port)
except Exception as e:
logger.warning(f"Error scanning port {port}: {e}")
result['filtered_ports'].append(port)
# Generate summary
result['scan_summary'] = {
'open_count': len(result['open_ports']),
'closed_count': len(result['closed_ports']),
'filtered_count': len(result['filtered_ports']),
'dangerous_count': len(result['dangerous_open']),
'has_web': any(p['port'] in [80, 443, 8080, 8443] for p in result['open_ports']),
'has_email': any(p['port'] in [25, 110, 143, 465, 587, 993, 995] for p in result['open_ports']),
'has_database': any(p['port'] in [3306, 5432, 1433, 1521, 27017, 6379] for p in result['open_ports']),
'has_remote_access': any(p['port'] in [22, 23, 3389, 5900] for p in result['open_ports'])
}
return result
def _scan_port(self, ip_address: str, port: int) -> tuple:
"""
Scan a single port
Returns:
Tuple of (status, banner)
status: 'open', 'closed', or 'filtered'
"""
try:
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.settimeout(self.timeout)
result = sock.connect_ex((ip_address, port))
if result == 0:
# Port is open, try to grab banner
banner = self._grab_banner(sock, port)
sock.close()
return 'open', banner
else:
sock.close()
return 'closed', None
except socket.timeout:
return 'filtered', None
except socket.error:
return 'filtered', None
except Exception as e:
return 'filtered', None
def _grab_banner(self, sock: socket.socket, port: int) -> Optional[str]:
"""Try to grab service banner"""
try:
# For HTTP ports, send a HEAD request
if port in [80, 8080]:
sock.send(b'HEAD / HTTP/1.0\r\n\r\n')
elif port in [443, 8443]:
return None # Can't grab banner from SSL without proper handshake
else:
# For other ports, just try to receive
pass
sock.settimeout(2)
banner = sock.recv(1024).decode('utf-8', errors='ignore').strip()
return banner[:200] if banner else None # Limit banner length
except:
return None
def quick_scan(self, ip_address: str) -> Dict:
"""
Quick scan of most common web ports
Args:
ip_address: IP to scan
Returns:
Quick scan results
"""
quick_ports = [21, 22, 23, 25, 80, 443, 3306, 3389, 8080]
return self.scan(ip_address, quick_ports)
def get_port_score(self, scan_data: Dict) -> Dict:
"""
Calculate security score based on port scan results
Returns:
Dict with score (0-100, higher = more risk) and reasons
"""
score = 0
reasons = []
# Dangerous ports are critical
dangerous_count = len(scan_data.get('dangerous_open', []))
if dangerous_count > 0:
score += dangerous_count * 20
reasons.append(f"{dangerous_count} dangerous port(s) exposed")
for port in scan_data.get('dangerous_open', []):
reasons.append(f" - {port['service']} on port {port['port']}")
# Database exposed
summary = scan_data.get('scan_summary', {})
if summary.get('has_database'):
score += 30
reasons.append("Database port(s) publicly accessible")
# Remote access (besides SSH)
if summary.get('has_remote_access'):
open_ports = [p['port'] for p in scan_data.get('open_ports', [])]
if 23 in open_ports: # Telnet
score += 25
reasons.append("Telnet (unencrypted) is open")
if 3389 in open_ports: # RDP
score += 20
reasons.append("RDP is publicly accessible")
if 5900 in open_ports: # VNC
score += 20
reasons.append("VNC is publicly accessible")
# Too many open ports (attack surface)
open_count = summary.get('open_count', 0)
if open_count > 10:
score += 15
reasons.append(f"Large attack surface: {open_count} open ports")
elif open_count > 5:
score += 5
reasons.append(f"{open_count} open ports detected")
if score == 0:
reasons.append("No dangerous services exposed")
return {
'score': min(100, score),
'reasons': reasons,
'is_secure': score <= 10,
'has_critical_issues': dangerous_count > 0 or summary.get('has_database')
}

741
ai_platform/modules/domain_check/api/app/services/risk_scorer.py Normal file
View file

@ -0,0 +1,741 @@
"""
Risk Scoring Engine - Comprehensive Domain Risk Assessment
SCORING FORMULA:
================
Total Risk Score = Σ(Category Score × Category Weight)
CATEGORIES & WEIGHTS:
- Domain Age: 20% (age_score × 0.20)
- SSL/TLS: 15% (ssl_score × 0.15)
- DNS Config: 10% (dns_score × 0.10)
- Email Security: 10% (email_score × 0.10)
- WHOIS Privacy: 5% (whois_score × 0.05)
- IP Reputation: 10% (ip_score × 0.10)
- HTTP Security: 10% (http_score × 0.10)
- Blacklists: 15% (blacklist_score × 0.15)
- Port Security: 5% (port_score × 0.05)
----
100%
INDIVIDUAL SCORE CALCULATIONS (0-100 scale, 0=safe, 100=dangerous):
1. DOMAIN AGE SCORE:
- < 30 days: 100 (CRITICAL)
- < 90 days: 80 (HIGH)
- < 180 days: 60 (MEDIUM-HIGH)
- < 365 days: 40 (MEDIUM)
- < 730 days: 20 (LOW)
- >= 730 days: 0 (TRUSTED)
2. SSL/TLS SCORE:
- No SSL: 100
- Expired: 100
- Self-signed: 80
- Expires < 7 days: 60
- Expires < 30 days: 30
- Weak cipher: 40
- Valid SSL: 0
3. DNS SCORE:
- No A records: 50
- No MX records: 20
- No NS at parent: 30
- Mismatched NS: 20
- No DNSSEC: 10
- Complete config: 0
4. EMAIL SECURITY SCORE:
- No SPF: 40
- SPF ~all (soft): 20
- SPF ?all (neutral): 30
- No DKIM: 30
- No DMARC: 20
- DMARC p=none: 15
- Full protection: 0
5. WHOIS SCORE:
- Privacy protected: 20
- No registrant info: 15
- Suspicious registrar: 30
- Free/disposable reg: 40
- Transparent: 0
6. IP REPUTATION SCORE:
- High-risk country: 40
- Known bad ASN: 50
- No reverse DNS: 20
- Datacenter IP: 10
- Residential IP: 0
7. HTTP SECURITY SCORE:
- No HTTPS: 40
- No HSTS: 20
- No CSP: 15
- No X-Frame-Options: 10
- Missing headers: 5 each
- Secure config: 0
8. BLACKLIST SCORE:
- On spam blacklist: 25 each
- On malware list: 50 each
- On phishing list: 60 each
- Not listed: 0
9. PORT SECURITY SCORE:
- Database exposed: 30
- RDP/VNC exposed: 25
- Telnet open: 40
- Too many ports (>10): 15
- Secure config: 0
RISK LEVELS:
- LOW: 0-25
- MEDIUM: 26-50
- HIGH: 51-75
- CRITICAL: 76-100
"""
import logging
from datetime import datetime
from typing import Dict, List, Optional, Tuple
from flask import current_app
logger = logging.getLogger(__name__)
def _coerce_datetime(value) -> Optional[datetime]:
"""Return a naive datetime for any date-like value, else None.
Defense-in-depth: even though WhoisService now normalizes dates, the risk
scorer can be fed cached/serialized data where dates are ISO strings. Never
let ``datetime - <non-datetime>`` raise from here.
"""
if value is None:
return None
if isinstance(value, datetime):
return value.replace(tzinfo=None) if value.tzinfo else value
if isinstance(value, str):
text = value.strip()
if not text:
return None
try:
from dateutil import parser as _p
dt = _p.parse(text, fuzzy=True)
return dt.replace(tzinfo=None) if dt.tzinfo else dt
except Exception:
return None
return None
class RiskScorer:
"""Calculate comprehensive risk scores for domains"""
def __init__(self):
# Category weights (must sum to 1.0)
self.weights = {
'domain_age': 0.20,
'ssl': 0.15,
'dns': 0.10,
'email_security': 0.10,
'whois': 0.05,
'ip_reputation': 0.10,
'http_security': 0.10,
'blacklist': 0.15,
'port_security': 0.05
}
# Risk level thresholds
self.thresholds = {
'low': 25,
'medium': 50,
'high': 75
}
def calculate_risk(self, domain_data: Dict) -> Dict:
"""
Calculate comprehensive risk score
Args:
domain_data: Dictionary containing all domain information
- whois: WHOIS data
- dns: DNS records
- ssl: SSL certificate data
- ip_intelligence: IP information
- http_analysis: HTTP analysis data
- blacklist: Blacklist check results
- port_scan: Port scan results
Returns:
Dictionary with complete risk assessment
"""
factors = []
scores = {}
formula_breakdown = []
# 1. Domain Age Score (20%)
if domain_data.get('whois'):
score, factor = self._score_domain_age(domain_data['whois'])
scores['domain_age'] = score
factors.append(factor)
formula_breakdown.append({
'category': 'Domain Age',
'raw_score': score,
'weight': self.weights['domain_age'],
'weighted_score': score * self.weights['domain_age'],
'formula': f"{score} × {self.weights['domain_age']} = {score * self.weights['domain_age']:.1f}"
})
# 2. SSL Score (15%)
if domain_data.get('ssl'):
score, factor = self._score_ssl(domain_data['ssl'])
scores['ssl'] = score
factors.append(factor)
formula_breakdown.append({
'category': 'SSL/TLS',
'raw_score': score,
'weight': self.weights['ssl'],
'weighted_score': score * self.weights['ssl'],
'formula': f"{score} × {self.weights['ssl']} = {score * self.weights['ssl']:.1f}"
})
# 3. DNS Score (10%)
if domain_data.get('dns'):
score, factor = self._score_dns(domain_data['dns'])
scores['dns'] = score
factors.append(factor)
formula_breakdown.append({
'category': 'DNS Configuration',
'raw_score': score,
'weight': self.weights['dns'],
'weighted_score': score * self.weights['dns'],
'formula': f"{score} × {self.weights['dns']} = {score * self.weights['dns']:.1f}"
})
# 4. Email Security Score (10%)
if domain_data.get('dns'):
score, factor = self._score_email_security(domain_data['dns'])
scores['email_security'] = score
factors.append(factor)
formula_breakdown.append({
'category': 'Email Security',
'raw_score': score,
'weight': self.weights['email_security'],
'weighted_score': score * self.weights['email_security'],
'formula': f"{score} × {self.weights['email_security']} = {score * self.weights['email_security']:.1f}"
})
# 5. WHOIS Score (5%)
if domain_data.get('whois'):
score, factor = self._score_whois(domain_data['whois'])
scores['whois'] = score
factors.append(factor)
formula_breakdown.append({
'category': 'WHOIS Privacy',
'raw_score': score,
'weight': self.weights['whois'],
'weighted_score': score * self.weights['whois'],
'formula': f"{score} × {self.weights['whois']} = {score * self.weights['whois']:.1f}"
})
# 6. IP Reputation Score (10%)
if domain_data.get('ip_intelligence'):
score, factor = self._score_ip_reputation(domain_data['ip_intelligence'])
scores['ip_reputation'] = score
factors.append(factor)
formula_breakdown.append({
'category': 'IP Reputation',
'raw_score': score,
'weight': self.weights['ip_reputation'],
'weighted_score': score * self.weights['ip_reputation'],
'formula': f"{score} × {self.weights['ip_reputation']} = {score * self.weights['ip_reputation']:.1f}"
})
# 7. HTTP Security Score (10%)
if domain_data.get('http_analysis'):
score, factor = self._score_http_security(domain_data['http_analysis'])
scores['http_security'] = score
factors.append(factor)
formula_breakdown.append({
'category': 'HTTP Security',
'raw_score': score,
'weight': self.weights['http_security'],
'weighted_score': score * self.weights['http_security'],
'formula': f"{score} × {self.weights['http_security']} = {score * self.weights['http_security']:.1f}"
})
# 8. Blacklist Score (15%)
if domain_data.get('blacklist'):
score, factor = self._score_blacklist(domain_data['blacklist'])
scores['blacklist'] = score
factors.append(factor)
formula_breakdown.append({
'category': 'Blacklist Status',
'raw_score': score,
'weight': self.weights['blacklist'],
'weighted_score': score * self.weights['blacklist'],
'formula': f"{score} × {self.weights['blacklist']} = {score * self.weights['blacklist']:.1f}"
})
# 9. Port Security Score (5%)
if domain_data.get('port_scan'):
score, factor = self._score_port_security(domain_data['port_scan'])
scores['port_security'] = score
factors.append(factor)
formula_breakdown.append({
'category': 'Port Security',
'raw_score': score,
'weight': self.weights['port_security'],
'weighted_score': score * self.weights['port_security'],
'formula': f"{score} × {self.weights['port_security']} = {score * self.weights['port_security']:.1f}"
})
# Calculate weighted total
total_weight_used = sum(self.weights[k] for k in scores.keys())
if total_weight_used > 0:
# Normalize to account for missing checks
raw_total = sum(scores[k] * self.weights[k] for k in scores.keys())
total_score = (raw_total / total_weight_used) if total_weight_used < 1.0 else raw_total
else:
total_score = 50 # Default medium risk if no data
total_score = min(100, max(0, int(total_score)))
# Determine risk level
risk_level = self._get_risk_level(total_score)
# Generate final formula string
formula_string = self._generate_formula_string(scores, total_score)
return {
'total_score': total_score,
'risk_level': risk_level,
'individual_scores': scores,
'factors': factors,
'formula_breakdown': formula_breakdown,
'formula_string': formula_string,
'weights_used': {k: self.weights[k] for k in scores.keys()},
'total_weight_used': total_weight_used,
'is_new_domain': scores.get('domain_age', 0) >= 80,
'is_suspicious': total_score >= 50,
'is_blacklisted': scores.get('blacklist', 0) > 0,
'requires_manual_review': total_score >= 60,
'risk_thresholds': {
'low': f"0-{self.thresholds['low']}",
'medium': f"{self.thresholds['low']+1}-{self.thresholds['medium']}",
'high': f"{self.thresholds['medium']+1}-{self.thresholds['high']}",
'critical': f"{self.thresholds['high']+1}-100"
}
}
def _score_domain_age(self, whois_data: Dict) -> Tuple[int, Dict]:
"""Score based on domain age"""
creation_date = _coerce_datetime(whois_data.get('creation_date'))
if not creation_date:
return 50, {
'factor': 'domain_age',
'score': 50,
'weight': self.weights['domain_age'],
'weighted_score': 50 * self.weights['domain_age'],
'reason': 'Unable to determine domain age',
'details': {'creation_date': None, 'age_days': None}
}
age_days = (datetime.utcnow() - creation_date).days
# Scoring thresholds
if age_days < 30:
score = 100
reason = f'CRITICAL: Very new domain ({age_days} days, <1 month)'
elif age_days < 90:
score = 80
reason = f'HIGH: New domain ({age_days} days, <3 months)'
elif age_days < 180:
score = 60
reason = f'MEDIUM-HIGH: Recently created ({age_days} days, <6 months)'
elif age_days < 365:
score = 40
reason = f'MEDIUM: Less than 1 year old ({age_days} days)'
elif age_days < 730:
score = 20
reason = f'LOW: Established domain ({age_days} days, 1-2 years)'
else:
score = 0
years = age_days // 365
reason = f'TRUSTED: Mature domain ({age_days} days, {years}+ years)'
return score, {
'factor': 'domain_age',
'score': score,
'weight': self.weights['domain_age'],
'weighted_score': score * self.weights['domain_age'],
'reason': reason,
'details': {
'creation_date': str(creation_date) if creation_date else None,
'age_days': age_days
}
}
def _score_ssl(self, ssl_data: Dict) -> Tuple[int, Dict]:
"""Score based on SSL/TLS certificate"""
if not ssl_data.get('has_ssl'):
return 100, {
'factor': 'ssl',
'score': 100,
'weight': self.weights['ssl'],
'weighted_score': 100 * self.weights['ssl'],
'reason': 'CRITICAL: No SSL/TLS certificate',
'details': {'has_ssl': False}
}
score = 0
reasons = []
if ssl_data.get('is_expired'):
score = 100
reasons.append('Certificate expired')
elif ssl_data.get('is_self_signed'):
score = 80
reasons.append('Self-signed certificate')
else:
days_until_expiry = ssl_data.get('days_until_expiry', 365)
if days_until_expiry < 7:
score = 60
reasons.append(f'Expires in {days_until_expiry} days')
elif days_until_expiry < 30:
score = 30
reasons.append(f'Expires in {days_until_expiry} days')
# Check for weak algorithms
sig_algo = (ssl_data.get('signature_algorithm') or '').lower()
if 'sha1' in sig_algo or 'md5' in sig_algo:
score = max(score, 40)
reasons.append('Weak signature algorithm')
if score == 0:
reasons.append('Valid SSL certificate')
return score, {
'factor': 'ssl',
'score': score,
'weight': self.weights['ssl'],
'weighted_score': score * self.weights['ssl'],
'reason': '; '.join(reasons) if reasons else 'Valid SSL',
'details': {
'has_ssl': True,
'is_valid': ssl_data.get('is_valid'),
'days_until_expiry': ssl_data.get('days_until_expiry'),
'issuer': ssl_data.get('issuer')
}
}
def _score_dns(self, dns_data: Dict) -> Tuple[int, Dict]:
"""Score based on DNS configuration"""
score = 0
reasons = []
a_records = dns_data.get('a_records', [])
if not a_records:
score += 50
reasons.append('No A records')
ns_records = dns_data.get('ns_records', [])
if not ns_records:
score += 30
reasons.append('No NS records')
mx_records = dns_data.get('mx_records', [])
if not mx_records:
score += 15
reasons.append('No MX records')
# Check for DNSSEC (if available)
if dns_data.get('dnssec') == 'inactive' or not dns_data.get('has_dnssec', True):
score += 5
reasons.append('DNSSEC not enabled')
if score == 0:
reasons.append('Complete DNS configuration')
return min(100, score), {
'factor': 'dns',
'score': min(100, score),
'weight': self.weights['dns'],
'weighted_score': min(100, score) * self.weights['dns'],
'reason': '; '.join(reasons),
'details': {
'a_record_count': len(a_records),
'ns_record_count': len(ns_records),
'mx_record_count': len(mx_records)
}
}
def _score_email_security(self, dns_data: Dict) -> Tuple[int, Dict]:
"""Score based on email security (SPF, DKIM, DMARC)"""
score = 0
reasons = []
has_spf = dns_data.get('has_spf', False)
has_dkim = dns_data.get('has_dkim', False)
has_dmarc = dns_data.get('has_dmarc', False)
if not has_spf:
score += 40
reasons.append('No SPF record')
else:
# Check SPF policy strength
spf_record = dns_data.get('spf_record', '')
if '~all' in spf_record:
score += 15
reasons.append('SPF uses soft fail (~all)')
elif '?all' in spf_record:
score += 25
reasons.append('SPF uses neutral (?all)')
if not has_dkim:
score += 30
reasons.append('No DKIM record detected')
if not has_dmarc:
score += 20
reasons.append('No DMARC record')
else:
dmarc_record = dns_data.get('dmarc_record', '')
if 'p=none' in dmarc_record:
score += 15
reasons.append('DMARC policy is none (monitoring only)')
if score == 0:
reasons.append('Full email security (SPF+DKIM+DMARC)')
return min(100, score), {
'factor': 'email_security',
'score': min(100, score),
'weight': self.weights['email_security'],
'weighted_score': min(100, score) * self.weights['email_security'],
'reason': '; '.join(reasons),
'details': {
'has_spf': has_spf,
'has_dkim': has_dkim,
'has_dmarc': has_dmarc
}
}
def _score_whois(self, whois_data: Dict) -> Tuple[int, Dict]:
"""Score based on WHOIS transparency"""
score = 0
reasons = []
if whois_data.get('privacy_protected'):
score += 20
reasons.append('WHOIS privacy protection enabled')
if not whois_data.get('registrant_org') and not whois_data.get('registrant'):
score += 15
reasons.append('No registrant information')
# Check for suspicious registrars
registrar = (whois_data.get('registrar') or '').lower()
suspicious_registrars = ['namecheap', 'namesilo', 'dynadot', 'freenom']
for sus in suspicious_registrars:
if sus in registrar:
score += 20
reasons.append(f'High-risk registrar pattern')
break
if score == 0:
reasons.append('Transparent WHOIS information')
return min(100, score), {
'factor': 'whois',
'score': min(100, score),
'weight': self.weights['whois'],
'weighted_score': min(100, score) * self.weights['whois'],
'reason': '; '.join(reasons),
'details': {
'registrar': whois_data.get('registrar'),
'privacy_protected': whois_data.get('privacy_protected', False)
}
}
def _score_ip_reputation(self, ip_data: Dict) -> Tuple[int, Dict]:
"""Score based on IP intelligence"""
score = 0
reasons = []
# High-risk countries
country_code = ip_data.get('country_code', '')
high_risk_countries = ['RU', 'CN', 'KP', 'IR', 'NG', 'VN', 'PK']
if country_code in high_risk_countries:
score += 40
reasons.append(f'High-risk country: {country_code}')
# No reverse DNS
if not ip_data.get('reverse_dns'):
score += 20
reasons.append('No reverse DNS (PTR)')
# Datacenter IP (slightly suspicious for some use cases)
if ip_data.get('is_datacenter'):
score += 10
reasons.append('Hosted on datacenter/cloud')
# Hosting provider scoring
hosting_score = ip_data.get('hosting_score', {})
if hosting_score.get('is_suspicious'):
score += 20
reasons.extend(hosting_score.get('reasons', []))
if score == 0:
reasons.append('Clean IP reputation')
return min(100, score), {
'factor': 'ip_reputation',
'score': min(100, score),
'weight': self.weights['ip_reputation'],
'weighted_score': min(100, score) * self.weights['ip_reputation'],
'reason': '; '.join(reasons),
'details': {
'ip': ip_data.get('ip'),
'country': ip_data.get('country'),
'asn': ip_data.get('asn'),
'isp': ip_data.get('isp')
}
}
def _score_http_security(self, http_data: Dict) -> Tuple[int, Dict]:
"""Score based on HTTP security headers"""
score = 0
reasons = []
if not http_data.get('has_https'):
score += 40
reasons.append('No HTTPS support')
if http_data.get('has_https') and not http_data.get('http_to_https_redirect'):
score += 15
reasons.append('No HTTP to HTTPS redirect')
# Check security headers
missing_headers = http_data.get('missing_security_headers', [])
for header in missing_headers:
severity = header.get('severity', 'LOW')
if severity == 'CRITICAL':
score += 15
elif severity == 'HIGH':
score += 10
else:
score += 5
if len(reasons) < 5: # Limit reasons
reasons.append(f"Missing: {header.get('header', 'Unknown')}")
if score == 0:
reasons.append('Good HTTP security configuration')
return min(100, score), {
'factor': 'http_security',
'score': min(100, score),
'weight': self.weights['http_security'],
'weighted_score': min(100, score) * self.weights['http_security'],
'reason': '; '.join(reasons[:5]),
'details': {
'has_https': http_data.get('has_https'),
'has_hsts': 'Strict-Transport-Security' in http_data.get('security_headers', {}),
'missing_headers_count': len(missing_headers)
}
}
def _score_blacklist(self, blacklist_data: Dict) -> Tuple[int, Dict]:
"""Score based on blacklist checks"""
score = 0
reasons = []
if blacklist_data.get('is_blacklisted'):
ip_listings = 0
domain_listings = 0
if blacklist_data.get('ip_check'):
ip_listings = blacklist_data['ip_check'].get('blacklist_count', 0)
score += ip_listings * 25
if blacklist_data.get('domain_check'):
domain_listings = blacklist_data['domain_check'].get('blacklist_count', 0)
score += domain_listings * 25
if ip_listings > 0:
reasons.append(f'IP on {ip_listings} blacklist(s)')
if domain_listings > 0:
reasons.append(f'Domain on {domain_listings} blacklist(s)')
else:
reasons.append('Not on any blacklists')
return min(100, score), {
'factor': 'blacklist',
'score': min(100, score),
'weight': self.weights['blacklist'],
'weighted_score': min(100, score) * self.weights['blacklist'],
'reason': '; '.join(reasons),
'details': {
'is_blacklisted': blacklist_data.get('is_blacklisted', False),
'total_listings': blacklist_data.get('total_listings', 0)
}
}
def _score_port_security(self, port_data: Dict) -> Tuple[int, Dict]:
"""Score based on port scan results"""
score = 0
reasons = []
dangerous_ports = port_data.get('dangerous_open', [])
if dangerous_ports:
score += len(dangerous_ports) * 20
for port in dangerous_ports[:3]: # Limit to 3
reasons.append(f"{port['service']} exposed (port {port['port']})")
summary = port_data.get('scan_summary', {})
if summary.get('has_database'):
score += 30
if 'Database' not in str(reasons):
reasons.append('Database port(s) publicly accessible')
open_count = summary.get('open_count', 0)
if open_count > 10:
score += 15
reasons.append(f'Large attack surface ({open_count} open ports)')
if score == 0:
reasons.append('No dangerous services exposed')
return min(100, score), {
'factor': 'port_security',
'score': min(100, score),
'weight': self.weights['port_security'],
'weighted_score': min(100, score) * self.weights['port_security'],
'reason': '; '.join(reasons[:5]),
'details': {
'open_port_count': summary.get('open_count', 0),
'dangerous_port_count': len(dangerous_ports)
}
}
def _get_risk_level(self, total_score: int) -> str:
"""Determine risk level from total score"""
if total_score <= self.thresholds['low']:
return 'LOW'
elif total_score <= self.thresholds['medium']:
return 'MEDIUM'
elif total_score <= self.thresholds['high']:
return 'HIGH'
else:
return 'CRITICAL'
def _generate_formula_string(self, scores: Dict, total: int) -> str:
"""Generate human-readable formula string"""
parts = []
for category, score in scores.items():
weight = self.weights.get(category, 0)
weighted = score * weight
parts.append(f"({score} × {weight})")
formula = " + ".join(parts)
return f"Total = {formula} = {total}"

185
ai_platform/modules/domain_check/api/app/services/ssl_service.py Normal file
View file

@ -0,0 +1,185 @@
"""
SSL Service - handles SSL certificate validation
"""
import logging
import socket
import ssl
from datetime import datetime
from typing import Dict, Optional
from OpenSSL import SSL, crypto
logger = logging.getLogger(__name__)
class SSLService:
"""SSL certificate checking service"""
def __init__(self, timeout: int = 10):
self.timeout = timeout
def check(self, domain: str, port: int = 443) -> Optional[Dict]:
"""
Check SSL certificate for a domain
Args:
domain: Domain name to check
port: Port number (default: 443)
Returns:
Dictionary with SSL certificate data or None
"""
try:
logger.info(f'SSL check for: {domain}:{port}')
# Create SSL context
context = ssl.create_default_context()
# Connect and get certificate
with socket.create_connection((domain, port), timeout=self.timeout) as sock:
with context.wrap_socket(sock, server_hostname=domain) as ssock:
cert_bin = ssock.getpeercert(binary_form=True)
cert_dict = ssock.getpeercert()
# Parse certificate with pyOpenSSL for more details
x509 = crypto.load_certificate(crypto.FILETYPE_ASN1, cert_bin)
ssl_data = {
'domain': domain,
'has_ssl': True,
'is_valid': True,
'is_self_signed': self._is_self_signed(x509),
'is_expired': False,
'is_wildcard': False,
'issuer': self._parse_issuer(x509),
'subject': self._parse_subject(x509),
'serial_number': str(x509.get_serial_number()),
'signature_algorithm': x509.get_signature_algorithm().decode('utf-8'),
'version': x509.get_version(),
'valid_from': None,
'valid_until': None,
'days_until_expiry': None,
'san': self._get_san(x509),
'key_size': x509.get_pubkey().bits()
}
# Parse dates
not_before = datetime.strptime(
x509.get_notBefore().decode('utf-8'),
'%Y%m%d%H%M%SZ'
)
not_after = datetime.strptime(
x509.get_notAfter().decode('utf-8'),
'%Y%m%d%H%M%SZ'
)
ssl_data['valid_from'] = not_before
ssl_data['valid_until'] = not_after
# Calculate days until expiry
days_until = (not_after - datetime.utcnow()).days
ssl_data['days_until_expiry'] = days_until
# Check if expired
if days_until < 0:
ssl_data['is_expired'] = True
ssl_data['is_valid'] = False
# Check if wildcard
subject_cn = self._get_common_name(x509)
if subject_cn and subject_cn.startswith('*.'):
ssl_data['is_wildcard'] = True
return ssl_data
except ssl.SSLError as e:
logger.warning(f'SSL error for {domain}: {str(e)}')
return {
'domain': domain,
'has_ssl': True,
'is_valid': False,
'error': str(e)
}
except socket.timeout:
logger.warning(f'SSL check timeout for {domain}')
return {
'domain': domain,
'has_ssl': False,
'error': 'Connection timeout'
}
except Exception as e:
logger.error(f'SSL check failed for {domain}: {str(e)}')
return {
'domain': domain,
'has_ssl': False,
'error': str(e)
}
def _is_self_signed(self, cert: crypto.X509) -> bool:
"""Check if certificate is self-signed"""
try:
issuer = cert.get_issuer()
subject = cert.get_subject()
return issuer.CN == subject.CN
except Exception:
return False
def _parse_issuer(self, cert: crypto.X509) -> str:
"""Parse certificate issuer"""
try:
issuer = cert.get_issuer()
parts = []
if hasattr(issuer, 'CN') and issuer.CN:
parts.append(f"CN={issuer.CN}")
if hasattr(issuer, 'O') and issuer.O:
parts.append(f"O={issuer.O}")
if hasattr(issuer, 'C') and issuer.C:
parts.append(f"C={issuer.C}")
return ', '.join(parts) if parts else 'Unknown'
except Exception:
return 'Unknown'
def _parse_subject(self, cert: crypto.X509) -> str:
"""Parse certificate subject"""
try:
subject = cert.get_subject()
parts = []
if hasattr(subject, 'CN') and subject.CN:
parts.append(f"CN={subject.CN}")
if hasattr(subject, 'O') and subject.O:
parts.append(f"O={subject.O}")
if hasattr(subject, 'C') and subject.C:
parts.append(f"C={subject.C}")
return ', '.join(parts) if parts else 'Unknown'
except Exception:
return 'Unknown'
def _get_common_name(self, cert: crypto.X509) -> Optional[str]:
"""Get Common Name from certificate"""
try:
subject = cert.get_subject()
return subject.CN if hasattr(subject, 'CN') else None
except Exception:
return None
def _get_san(self, cert: crypto.X509) -> list:
"""Get Subject Alternative Names"""
try:
san_ext = None
for i in range(cert.get_extension_count()):
ext = cert.get_extension(i)
if ext.get_short_name() == b'subjectAltName':
san_ext = ext
break
if san_ext:
san_str = str(san_ext)
# Parse SAN string (format: "DNS:example.com, DNS:www.example.com")
sans = []
for part in san_str.split(','):
part = part.strip()
if part.startswith('DNS:'):
sans.append(part[4:])
return sans
except Exception as e:
logger.debug(f'Could not parse SAN: {e}')
return []

442
ai_platform/modules/domain_check/api/app/services/subdomain_service.py Normal file
View file

@ -0,0 +1,442 @@
"""
Subdomain Enumeration Service - Certificate Transparency, DNS enumeration
"""
import logging
import requests
import dns.resolver
from typing import Dict, List, Set
from concurrent.futures import ThreadPoolExecutor, as_completed
logger = logging.getLogger(__name__)
class SubdomainService:
"""Service for discovering subdomains"""
def __init__(self):
self.timeout = 15
self.resolver = dns.resolver.Resolver()
self.resolver.nameservers = ['8.8.8.8', '8.8.4.4']
self.resolver.timeout = 3
self.resolver.lifetime = 5
# Common subdomain prefixes to check (expanded: infra, SaaS, business apps)
self.common_subdomains = [
'www', 'www2', 'mail', 'webmail', 'smtp', 'pop', 'pop3', 'imap', 'mx', 'mx1', 'mx2',
'email', 'mailserver', 'relay', 'newsletter', 'mailgun', 'mailchimp', 'lists', 'list',
'ftp', 'sftp', 'ssh', 'admin', 'administrator', 'panel', 'cpanel', 'whm', 'plesk', 'directadmin',
'api', 'api1', 'api2', 'apis', 'rest', 'graphql', 'ws', 'sockets', 'dev', 'develop', 'development',
'staging', 'stage', 'stg', 'test', 'testing', 'qa', 'uat', 'preprod', 'beta', 'alpha', 'demo', 'sandbox',
'blog', 'shop', 'store', 'magento', 'woocommerce', 'checkout', 'pay', 'payment', 'payments', 'billing', 'invoice', 'invoices',
'app', 'apps', 'application', 'mobile', 'm', 'web', 'portal', 'my', 'account', 'accounts', 'auth', 'sso', 'login', 'oauth', 'id', 'idp',
'cdn', 'static', 'assets', 'img', 'images', 'media', 'video', 'videos', 'stream', 'live', 'download', 'downloads', 'files', 'file', 'share', 'sharepoint',
'ns1', 'ns2', 'ns3', 'ns4', 'dns', 'dns1', 'dns2', 'resolver',
'vpn', 'remote', 'gateway', 'gw', 'proxy', 'firewall', 'fw', 'router', 'access',
'git', 'gitlab', 'github', 'gitea', 'bitbucket', 'svn', 'jenkins', 'ci', 'cicd', 'build', 'registry', 'docker', 'nexus', 'artifactory',
'db', 'database', 'mysql', 'postgres', 'pg', 'mongo', 'redis', 'elastic', 'elasticsearch', 'kibana', 'grafana', 'prometheus', 'metrics', 'monitor', 'monitoring', 'status', 'health', 'uptime',
'backup', 'backups', 'bk', 'old', 'new', 'v2', 'v3', 'legacy', 'archive',
'intranet', 'extranet', 'internal', 'private', 'corp', 'office', 'work',
'cloud', 'nextcloud', 'owncloud', 'drive', 'docs', 'doc', 'documents', 'wiki', 'confluence', 'jira', 'kb', 'knowledgebase',
'crm', 'erp', 'hr', 'hrm', 'support', 'help', 'helpdesk', 'ticket', 'tickets', 'desk', 'service', 'services', 'client', 'clients', 'customer', 'customers', 'partner', 'partners', 'projects', 'project', 'pm', 'tasks', 'rpa', 'automation', 'tooling', 'tools', 'tool',
'chat', 'talk', 'meet', 'conf', 'conference', 'videoconferinta', 'voip', 'pbx', 'sip', 'call', 'calls',
'dashboard', 'analytics', 'stats', 'report', 'reports', 'data', 'bi', 'reporting',
'autodiscover', 'autoconfig', 'exchange', 'owa', 'mail2', 'webdisk', 'cpcalendars', 'cpcontacts',
'vps', 'server', 'server1', 'server2', 'host', 'node', 'node1', 'cluster', 'k8s', 'kubernetes', 'srv',
'secure', 'ssl', 'vault', 'secret', 'kms', 'ldap', 'ad', 'radius'
]
def enumerate(self, domain: str, extra_subdomains: List[str] = None) -> Dict:
"""
Enumerate subdomains using multiple methods.
Coverage note: CT logs only reveal names that had their own certificate;
a wildcard cert (*.domain) or a custom-named record with no cert is
invisible to CT. Brute-force only finds names in the wordlist. For a
domain you own, the authoritative way to get 100% is the DNS provider's
API (e.g. GoDaddy) or AXFR (usually disabled). Pass `extra_subdomains`
to verify your own known names directly.
Args:
domain: Root domain to enumerate
extra_subdomains: owner-supplied candidate names (bare label or FQDN)
Returns:
Dictionary with discovered subdomains
"""
result = {
'domain': domain,
'subdomains': [],
'total_found': 0,
'sources': {
'certificate_transparency': [],
'dns_bruteforce': [],
'dns_records': [],
'zone_transfer': [],
'user_provided': []
},
'has_wildcard_cert': False,
'coverage_note': None,
'live_subdomains': [],
'error': None
}
discovered: Set[str] = set()
try:
# 1. Certificate Transparency logs (crt.sh)
ct_raw = self._get_ct_subdomains(domain)
result['has_wildcard_cert'] = any(s.startswith('*.') for s in ct_raw)
ct_subdomains = [s for s in ct_raw if not s.startswith('*')]
result['sources']['certificate_transparency'] = ct_subdomains
discovered.update(ct_subdomains)
# 2. DNS records enumeration (MX/NS/SOA)
dns_subdomains = self._get_dns_subdomains(domain)
result['sources']['dns_records'] = dns_subdomains
discovered.update(dns_subdomains)
# 3. AXFR zone transfer (jackpot if the NS allows it — usually not)
axfr_subdomains = self._try_axfr(domain)
result['sources']['zone_transfer'] = axfr_subdomains
discovered.update(axfr_subdomains)
# 4. Bruteforce common subdomains
brute_subdomains = self._bruteforce_subdomains(domain)
result['sources']['dns_bruteforce'] = brute_subdomains
discovered.update(brute_subdomains)
# 5. Owner-supplied candidate names — verify which actually resolve
if extra_subdomains:
user_found = self._check_user_subdomains(domain, extra_subdomains)
result['sources']['user_provided'] = user_found
discovered.update(user_found)
# Remove wildcards and invalid entries
discovered = {
s for s in discovered
if s and not s.startswith('*') and domain in s
}
result['subdomains'] = sorted(list(discovered))
result['total_found'] = len(discovered)
if result['has_wildcard_cert']:
result['coverage_note'] = (
'Domeniul are certificat wildcard (*.{0}), deci subdomeniile '
'fara certificat propriu NU apar in CT logs. Lista poate fi '
'incompleta — foloseste extra_subdomains sau API-ul DNS al '
'registrarului (GoDaddy) pentru acoperire 100%.'.format(domain)
)
# Check which subdomains are live
result['live_subdomains'] = self._check_live_subdomains(list(discovered)[:60])
except Exception as e:
logger.error(f"Subdomain enumeration failed for {domain}: {e}")
result['error'] = str(e)
return result
def _try_axfr(self, domain: str) -> List[str]:
"""Attempt a DNS zone transfer (AXFR) against each authoritative NS.
Almost always refused, but when an NS is misconfigured it dumps the
entire zone — every subdomain at once. Cheap to try, big payoff."""
import dns.query
import dns.zone
found: Set[str] = set()
try:
ns_records = self.resolver.resolve(domain, 'NS')
nameservers = [str(ns.target).rstrip('.') for ns in ns_records]
except Exception:
return []
for ns in nameservers[:4]:
try:
ns_ip = str(self.resolver.resolve(ns, 'A')[0])
zone = dns.zone.from_xfr(dns.query.xfr(ns_ip, domain, timeout=5, lifetime=8))
for name in zone.nodes.keys():
label = str(name)
if label in ('@', ''):
continue
fqdn = f'{label}.{domain}' if not label.endswith(domain) else label
found.add(fqdn.rstrip('.').lower())
logger.info(f'AXFR succeeded against {ns} for {domain} ({len(found)} names)')
break
except Exception:
continue
return list(found)
def _check_user_subdomains(self, domain: str, names: List[str]) -> List[str]:
"""Resolve owner-supplied candidate names and keep the ones that exist."""
candidates = []
for n in names:
n = str(n).strip().lower().rstrip('.')
if not n:
continue
fqdn = n if n.endswith(domain) else f'{n}.{domain}'
candidates.append(fqdn)
found = []
with ThreadPoolExecutor(max_workers=20) as ex:
futures = {ex.submit(self._resolves, c): c for c in candidates}
for fut in as_completed(futures, timeout=30):
try:
if fut.result():
found.append(futures[fut])
except Exception:
pass
return found
def _resolves(self, name: str) -> bool:
for rtype in ('A', 'AAAA', 'CNAME'):
try:
self.resolver.resolve(name, rtype)
return True
except Exception:
continue
return False
def _get_ct_subdomains(self, domain: str) -> List[str]:
"""Get subdomains from Certificate Transparency logs.
crt.sh is the richest source for real (incl. custom-named) subdomains but
is frequently slow or rate-limited, so we retry, and fall back to the
certspotter API when crt.sh keeps failing."""
subdomains: Set[str] = set()
# Primary: crt.sh, with retries (it flakes often).
for attempt in range(3):
try:
response = requests.get(
f"https://crt.sh/?q=%.{domain}&output=json",
timeout=self.timeout,
headers={'User-Agent': 'domain-check/1.0'}
)
if response.status_code == 200 and response.text.strip():
for entry in response.json():
for name in entry.get('name_value', '').split('\n'):
name = name.strip().lower()
if name and domain in name:
subdomains.add(name)
if subdomains:
return list(subdomains)
except Exception as e:
logger.warning(f"crt.sh attempt {attempt + 1} failed: {e}")
# Fallback: Cert Spotter (no key needed for low volume).
try:
resp = requests.get(
f"https://api.certspotter.com/v1/issuances?domain={domain}"
"&include_subdomains=true&expand=dns_names",
timeout=self.timeout, headers={'User-Agent': 'domain-check/1.0'}
)
if resp.status_code == 200:
for cert in resp.json():
for name in cert.get('dns_names', []):
name = str(name).strip().lower()
if name and domain in name:
subdomains.add(name)
except Exception as e:
logger.warning(f"certspotter fallback failed: {e}")
return list(subdomains)
def _get_dns_subdomains(self, domain: str) -> List[str]:
"""Get subdomains from DNS records (NS, MX, etc.)"""
subdomains = []
try:
# Check MX records
try:
mx_records = self.resolver.resolve(domain, 'MX')
for mx in mx_records:
host = str(mx.exchange).rstrip('.')
if domain in host:
subdomains.append(host)
except:
pass
# Check NS records
try:
ns_records = self.resolver.resolve(domain, 'NS')
for ns in ns_records:
host = str(ns.target).rstrip('.')
if domain in host:
subdomains.append(host)
except:
pass
# Check SOA record
try:
soa_records = self.resolver.resolve(domain, 'SOA')
for soa in soa_records:
mname = str(soa.mname).rstrip('.')
if domain in mname:
subdomains.append(mname)
except:
pass
except Exception as e:
logger.warning(f"DNS subdomain lookup failed: {e}")
return list(set(subdomains))
def _bruteforce_subdomains(self, domain: str) -> List[str]:
"""Bruteforce common subdomains"""
found = []
with ThreadPoolExecutor(max_workers=20) as executor:
futures = {
executor.submit(self._check_subdomain, f"{sub}.{domain}"): sub
for sub in self.common_subdomains
}
for future in as_completed(futures, timeout=30):
subdomain = futures[future]
try:
full_subdomain = f"{subdomain}.{domain}"
if future.result():
found.append(full_subdomain)
except:
pass
return found
def _check_subdomain(self, subdomain: str) -> bool:
"""Check if subdomain resolves"""
try:
self.resolver.resolve(subdomain, 'A')
return True
except:
return False
def _check_live_subdomains(self, subdomains: List[str]) -> List[Dict]:
"""Check which subdomains are live (have HTTP response)"""
live = []
with ThreadPoolExecutor(max_workers=10) as executor:
futures = {
executor.submit(self._check_http, sub): sub
for sub in subdomains
}
for future in as_completed(futures, timeout=60):
subdomain = futures[future]
try:
result = future.result()
if result:
live.append(result)
except:
pass
return live
def _check_http(self, subdomain: str) -> Dict:
"""Check HTTP/HTTPS on subdomain"""
result = {
'subdomain': subdomain,
'ip': None,
'http': None,
'https': None
}
# Get IP
try:
answers = self.resolver.resolve(subdomain, 'A')
result['ip'] = str(answers[0])
except:
return None
# Check HTTPS
try:
response = requests.head(
f"https://{subdomain}",
timeout=5,
allow_redirects=True,
verify=False
)
result['https'] = response.status_code
except:
pass
# Check HTTP
try:
response = requests.head(
f"http://{subdomain}",
timeout=5,
allow_redirects=False
)
result['http'] = response.status_code
except:
pass
if result['http'] or result['https']:
return result
return None
def check_subdomain_takeover(self, subdomains: List[str]) -> List[Dict]:
"""
Check for potential subdomain takeover vulnerabilities
Args:
subdomains: List of subdomains to check
Returns:
List of potentially vulnerable subdomains
"""
vulnerable = []
# CNAME fingerprints for takeover
takeover_fingerprints = {
'github.io': 'GitHub Pages',
'herokuapp.com': 'Heroku',
'herokudns.com': 'Heroku',
'wordpress.com': 'WordPress',
'pantheonsite.io': 'Pantheon',
'domains.tumblr.com': 'Tumblr',
'zendesk.com': 'Zendesk',
'shopify.com': 'Shopify',
'myshopify.com': 'Shopify',
's3.amazonaws.com': 'AWS S3',
's3-website': 'AWS S3',
'cloudfront.net': 'AWS CloudFront',
'azurewebsites.net': 'Azure',
'cloudapp.net': 'Azure',
'trafficmanager.net': 'Azure',
'blob.core.windows.net': 'Azure Blob',
'ghost.io': 'Ghost',
'helpjuice.com': 'Helpjuice',
'helpscoutdocs.com': 'HelpScout',
'freshdesk.com': 'Freshdesk',
'surge.sh': 'Surge',
'bitbucket.io': 'Bitbucket',
'uservoice.com': 'UserVoice',
'simplebooklet.com': 'Simplebooklet'
}
for subdomain in subdomains:
try:
# Check for CNAME
cname_records = self.resolver.resolve(subdomain, 'CNAME')
for cname in cname_records:
cname_target = str(cname.target).rstrip('.').lower()
for fingerprint, service in takeover_fingerprints.items():
if fingerprint in cname_target:
# Check if CNAME target resolves
try:
self.resolver.resolve(cname_target, 'A')
except dns.resolver.NXDOMAIN:
vulnerable.append({
'subdomain': subdomain,
'cname': cname_target,
'service': service,
'status': 'VULNERABLE',
'reason': 'CNAME points to non-existent resource'
})
except:
pass
break
except:
pass
return vulnerable

248
ai_platform/modules/domain_check/api/app/services/whois_service.py Normal file
View file

@ -0,0 +1,248 @@
"""
WHOIS Service - handles WHOIS/RDAP lookups
"""
import logging
from datetime import datetime
from typing import Dict, List, Optional, Union
import requests
try:
from dateutil import parser as dateutil_parser
DATEUTIL_AVAILABLE = True
except ImportError:
DATEUTIL_AVAILABLE = False
try:
import whois as python_whois
WHOIS_AVAILABLE = True
except ImportError:
WHOIS_AVAILABLE = False
logger = logging.getLogger(__name__)
def normalize_whois_date(value) -> Optional[datetime]:
"""Coerce any WHOIS date representation into a naive (tz-stripped) datetime.
python-whois and registry-specific WHOIS servers (notably ROTLD for .ro)
return dates as datetime, list-of-datetime, ISO strings, or arbitrary
free-form strings. Downstream code does ``datetime - value`` arithmetic, so
every date MUST arrive as a ``datetime`` or ``None`` — never a raw string.
A raw string here is the root cause of the historical 500 on ``.ro`` domains.
"""
if value is None:
return None
if isinstance(value, list):
# Registries often return the most recent date first; take the first set value.
for item in value:
parsed = normalize_whois_date(item)
if parsed is not None:
return parsed
return None
if isinstance(value, datetime):
return value.replace(tzinfo=None) if value.tzinfo else value
if isinstance(value, str):
text = value.strip()
if not text or text.lower() in ('none', 'null', 'n/a', '-'):
return None
if DATEUTIL_AVAILABLE:
try:
parsed = dateutil_parser.parse(text, fuzzy=True)
return parsed.replace(tzinfo=None) if parsed.tzinfo else parsed
except (ValueError, OverflowError, TypeError):
return None
# Fallback without dateutil: a few common explicit formats.
for fmt in ('%Y-%m-%dT%H:%M:%S.%fZ', '%Y-%m-%dT%H:%M:%SZ', '%Y-%m-%dT%H:%M:%S',
'%Y-%m-%d %H:%M:%S', '%Y-%m-%d', '%d.%m.%Y', '%d-%b-%Y'):
try:
return datetime.strptime(text, fmt)
except ValueError:
continue
return None
return None
def _as_list(value) -> List:
"""Normalize a WHOIS field that may be a scalar, None, or list into a clean list."""
if value is None:
return []
if isinstance(value, (list, tuple, set)):
return [v for v in value if v not in (None, '')]
return [value]
class WhoisService:
"""WHOIS/RDAP service class"""
def __init__(self, whoxy_api_key: str = None):
self.whoxy_api_key = whoxy_api_key
self.whoxy_url = 'https://api.whoxy.com/'
def lookup(self, domain: str, source: str = 'auto') -> Optional[Dict]:
"""
Perform WHOIS lookup
Args:
domain: Domain name to lookup
source: 'rdap', 'whoxy', or 'auto' (tries RDAP first, falls back to Whoxy)
Returns:
Dictionary with WHOIS data or None
"""
if source == 'auto':
# Try RDAP first
result = self._lookup_rdap(domain)
# Check if we got useful data (creation_date should exist)
if result and result.get('creation_date'):
logger.info(f'Using WHOIS data from python-whois for {domain}')
return result
# Fallback to Whoxy if available
if self.whoxy_api_key:
logger.info(f'Falling back to Whoxy API for {domain}')
whoxy_result = self._lookup_whoxy(domain)
if whoxy_result:
return whoxy_result
# If we got partial data from RDAP, return it
if result:
logger.warning(f'Returning incomplete WHOIS data for {domain}')
return result
return None
elif source == 'rdap':
return self._lookup_rdap(domain)
elif source == 'whoxy':
return self._lookup_whoxy(domain)
return None
def _lookup_rdap(self, domain: str) -> Optional[Dict]:
"""Lookup using python-whois library"""
if not WHOIS_AVAILABLE:
logger.warning('python-whois not available')
return None
try:
logger.info(f'WHOIS lookup for: {domain}')
w = python_whois.whois(domain)
# Normalize every date to datetime|None so downstream arithmetic is safe.
creation_date = normalize_whois_date(getattr(w, 'creation_date', None))
expiration_date = normalize_whois_date(getattr(w, 'expiration_date', None))
updated_date = normalize_whois_date(getattr(w, 'updated_date', None))
name_servers = [str(ns).lower() for ns in _as_list(getattr(w, 'name_servers', None))]
status = [str(s) for s in _as_list(getattr(w, 'status', None))]
emails = _as_list(getattr(w, 'emails', None))
result = {
'domain': domain,
'creation_date': creation_date,
'expiration_date': expiration_date,
'updated_date': updated_date,
'registrar': getattr(w, 'registrar', None),
'registrar_url': None,
'registrant_org': getattr(w, 'org', None),
'registrant_country': getattr(w, 'country', None),
'admin_email': emails[0] if emails else None,
'name_servers': name_servers,
'status': status,
'dnssec': None,
'data_source': 'whois',
'raw_data': {}
}
result['is_registered'] = self._is_registered(result)
return result
except Exception as e:
logger.error(f'WHOIS lookup failed for {domain}: {str(e)}')
return None
@staticmethod
def _is_registered(whois_result: Dict) -> Optional[bool]:
"""Best-effort determination of whether a domain is registered.
Returns True/False from WHOIS signals, or None when WHOIS is too sparse
to decide (common for ROTLD .ro) — the route then refines this using DNS.
"""
if whois_result.get('creation_date') or whois_result.get('registrar'):
return True
if whois_result.get('name_servers') or whois_result.get('status'):
return True
# No positive WHOIS signal at all: likely available, but let DNS confirm.
return None
def _lookup_whoxy(self, domain: str) -> Optional[Dict]:
"""Lookup using Whoxy API"""
if not self.whoxy_api_key:
logger.warning('Whoxy API key not configured')
return None
try:
logger.info(f'Whoxy API lookup for: {domain}')
response = requests.get(
self.whoxy_url,
params={
'key': self.whoxy_api_key,
'whois': domain
},
timeout=30
)
if response.status_code != 200:
logger.error(f'Whoxy API error: {response.status_code}')
return None
data = response.json()
if data.get('status') != 1:
logger.error(f'Whoxy API returned error: {data.get("status_reason")}')
return None
whois_data = data.get('whois_data', {})
result = {
'domain': domain,
'creation_date': normalize_whois_date(whois_data.get('create_date')),
'expiration_date': normalize_whois_date(whois_data.get('expiry_date')),
'updated_date': normalize_whois_date(whois_data.get('update_date')),
'registrar': whois_data.get('registrar_name'),
'registrar_url': whois_data.get('registrar_url'),
'registrant_org': whois_data.get('registrant_organization'),
'registrant_country': whois_data.get('registrant_country'),
'admin_email': whois_data.get('admin_email'),
'name_servers': [str(ns).lower() for ns in _as_list(whois_data.get('name_servers'))],
'status': [str(s) for s in _as_list(whois_data.get('domain_status'))],
'dnssec': whois_data.get('dnssec') == 'yes',
'data_source': 'whoxy',
'raw_data': whois_data
}
result['is_registered'] = self._is_registered(result)
return result
except Exception as e:
logger.error(f'Whoxy lookup failed for {domain}: {str(e)}')
return None
@staticmethod
def _parse_date(date_str: str) -> Optional[datetime]:
"""Parse date string to datetime"""
if not date_str:
return None
try:
# Try common date formats
for fmt in ['%Y-%m-%d', '%Y-%m-%dT%H:%M:%SZ', '%Y-%m-%d %H:%M:%S']:
try:
return datetime.strptime(date_str, fmt)
except ValueError:
continue
return None
except Exception:
return None

637
ai_platform/modules/domain_check/api/app/static/index.html Normal file
View file

@ -0,0 +1,637 @@
<!DOCTYPE html>
<html lang="ro">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Domain Check - Verificare Completă Domenii</title>
<style>
* { margin: 0; padding: 0; box-sizing: border-box; }
body {
font-family: 'Segoe UI', system-ui, sans-serif;
background: linear-gradient(135deg, #1a1a2e 0%, #16213e 50%, #0f3460 100%);
min-height: 100vh;
color: #e4e4e4;
}
.container { max-width: 1400px; margin: 0 auto; padding: 20px; }
h1 {
text-align: center;
font-size: 2rem;
margin-bottom: 10px;
background: linear-gradient(90deg, #00d9ff, #00ff88);
-webkit-background-clip: text;
-webkit-text-fill-color: transparent;
}
.subtitle { text-align: center; color: #888; margin-bottom: 30px; font-size: 0.9rem; }
.search-box {
background: rgba(255,255,255,0.05);
border-radius: 16px;
padding: 25px;
margin-bottom: 25px;
border: 1px solid rgba(255,255,255,0.1);
}
.input-group {
display: flex;
gap: 15px;
flex-wrap: wrap;
align-items: center;
}
.input-group input {
flex: 1;
min-width: 250px;
padding: 15px 20px;
border: 2px solid rgba(255,255,255,0.2);
border-radius: 10px;
font-size: 1.1rem;
background: rgba(0,0,0,0.3);
color: #fff;
}
.input-group input:focus { outline: none; border-color: #00d9ff; }
.btn {
padding: 15px 35px;
border: none;
border-radius: 10px;
font-size: 1rem;
cursor: pointer;
font-weight: 600;
transition: all 0.3s;
}
.btn-primary {
background: linear-gradient(135deg, #00d9ff, #00ff88);
color: #1a1a2e;
}
.btn-primary:hover { transform: translateY(-2px); box-shadow: 0 5px 20px rgba(0,217,255,0.4); }
.btn-primary:disabled { opacity: 0.5; cursor: not-allowed; transform: none; }
.options {
display: flex;
flex-wrap: wrap;
gap: 15px;
margin-top: 20px;
padding-top: 20px;
border-top: 1px solid rgba(255,255,255,0.1);
}
.option-group { display: flex; align-items: center; gap: 8px; }
.option-group input[type="checkbox"] { width: 18px; height: 18px; cursor: pointer; }
.option-group label { cursor: pointer; font-size: 0.9rem; }
.option-group.slow label { color: #ff9800; }
.loading { display: none; text-align: center; padding: 50px; }
.spinner {
width: 50px; height: 50px;
border: 4px solid rgba(255,255,255,0.1);
border-top-color: #00d9ff;
border-radius: 50%;
animation: spin 1s linear infinite;
margin: 0 auto 20px;
}
@keyframes spin { to { transform: rotate(360deg); } }
.results { display: none; }
.results-grid {
display: grid;
grid-template-columns: repeat(auto-fit, minmax(400px, 1fr));
gap: 20px;
}
.card {
background: rgba(255,255,255,0.05);
border-radius: 12px;
border: 1px solid rgba(255,255,255,0.1);
overflow: hidden;
}
.card-header {
padding: 15px 20px;
background: rgba(0,0,0,0.3);
display: flex;
justify-content: space-between;
align-items: center;
border-bottom: 1px solid rgba(255,255,255,0.1);
}
.card-header h3 { font-size: 1rem; display: flex; align-items: center; gap: 10px; }
.card-header .source { font-size: 0.75rem; color: #888; background: rgba(255,255,255,0.1); padding: 3px 8px; border-radius: 4px; }
.card-body { padding: 20px; }
.risk-card { grid-column: 1 / -1; }
.risk-score-display {
display: flex;
align-items: center;
gap: 30px;
flex-wrap: wrap;
}
.risk-circle {
width: 150px; height: 150px;
border-radius: 50%;
display: flex;
flex-direction: column;
align-items: center;
justify-content: center;
font-size: 3rem;
font-weight: bold;
flex-shrink: 0;
}
.risk-circle .label { font-size: 0.9rem; font-weight: normal; margin-top: 5px; }
.risk-low { background: linear-gradient(135deg, #00c853, #00e676); color: #1a1a2e; }
.risk-medium { background: linear-gradient(135deg, #ffc107, #ffeb3b); color: #1a1a2e; }
.risk-high { background: linear-gradient(135deg, #ff5722, #ff9800); color: #fff; }
.risk-critical { background: linear-gradient(135deg, #d32f2f, #f44336); color: #fff; }
.risk-details { flex: 1; min-width: 300px; }
.risk-factor {
display: flex;
justify-content: space-between;
align-items: center;
padding: 10px 0;
border-bottom: 1px solid rgba(255,255,255,0.1);
}
.risk-factor:last-child { border-bottom: none; }
.risk-factor .name { font-weight: 500; }
.risk-factor .score {
display: flex;
align-items: center;
gap: 10px;
}
.risk-factor .bar {
width: 100px;
height: 8px;
background: rgba(255,255,255,0.1);
border-radius: 4px;
overflow: hidden;
}
.risk-factor .bar-fill { height: 100%; transition: width 0.5s; }
.risk-factor .value { width: 40px; text-align: right; font-weight: bold; }
.formula-box {
background: rgba(0,0,0,0.3);
border-radius: 8px;
padding: 15px;
margin-top: 20px;
font-family: monospace;
font-size: 0.85rem;
overflow-x: auto;
}
.formula-box h4 { margin-bottom: 10px; color: #00d9ff; }
.formula-line { padding: 3px 0; }
.data-row {
display: flex;
padding: 10px 0;
border-bottom: 1px solid rgba(255,255,255,0.05);
}
.data-row:last-child { border-bottom: none; }
.data-label { width: 140px; color: #888; font-size: 0.9rem; flex-shrink: 0; }
.data-value { flex: 1; word-break: break-all; }
.badge {
display: inline-block;
padding: 3px 10px;
border-radius: 4px;
font-size: 0.8rem;
margin: 2px;
}
.badge.success { background: rgba(0,200,83,0.2); color: #00c853; }
.badge.danger { background: rgba(244,67,54,0.2); color: #f44336; }
.badge.warning { background: rgba(255,152,0,0.2); color: #ff9800; }
.badge.info { background: rgba(0,217,255,0.2); color: #00d9ff; }
.badge.neutral { background: rgba(255,255,255,0.1); color: #aaa; }
.port-list { display: flex; flex-wrap: wrap; gap: 5px; }
.port-item {
padding: 5px 10px;
border-radius: 4px;
font-size: 0.8rem;
background: rgba(255,255,255,0.1);
}
.port-item.danger { background: rgba(244,67,54,0.2); color: #f44336; }
.port-item.warning { background: rgba(255,152,0,0.2); color: #ff9800; }
.port-item.safe { background: rgba(0,200,83,0.2); color: #00c853; }
.blacklist-item {
display: flex;
justify-content: space-between;
padding: 8px 0;
border-bottom: 1px solid rgba(255,255,255,0.05);
}
.blacklist-clean { color: #00c853; }
.blacklist-listed { color: #f44336; }
.error {
background: rgba(244,67,54,0.2);
border: 1px solid #f44336;
border-radius: 8px;
padding: 20px;
text-align: center;
color: #f44336;
}
.tech-tags { display: flex; flex-wrap: wrap; gap: 8px; }
.tech-tag {
background: linear-gradient(135deg, rgba(0,217,255,0.2), rgba(0,255,136,0.2));
padding: 5px 12px;
border-radius: 20px;
font-size: 0.85rem;
}
.security-header {
display: flex;
justify-content: space-between;
align-items: center;
padding: 8px 0;
border-bottom: 1px solid rgba(255,255,255,0.05);
}
.header-present { color: #00c853; }
.header-missing { color: #f44336; }
@media (max-width: 768px) {
.results-grid { grid-template-columns: 1fr; }
.risk-score-display { flex-direction: column; text-align: center; }
.input-group { flex-direction: column; }
.input-group input { width: 100%; }
}
</style>
</head>
<body>
<div class="container">
<h1>🔍 Domain Check</h1>
<p class="subtitle">Verificare completă domenii - WHOIS, DNS, SSL, IP Intelligence, HTTP Security, Blacklists</p>
<div class="search-box">
<div class="input-group">
<input type="text" id="domainInput" placeholder="Introdu domeniul (ex: google.com, example.ro)" autofocus>
<button class="btn btn-primary" id="checkBtn" onclick="checkDomain()">Verifică Domeniul</button>
</div>
<div class="options">
<div class="option-group">
<input type="checkbox" id="optWhois" checked>
<label for="optWhois">WHOIS</label>
</div>
<div class="option-group">
<input type="checkbox" id="optDns" checked>
<label for="optDns">DNS</label>
</div>
<div class="option-group">
<input type="checkbox" id="optSsl" checked>
<label for="optSsl">SSL/TLS</label>
</div>
<div class="option-group">
<input type="checkbox" id="optIp" checked>
<label for="optIp">IP Intelligence</label>
</div>
<div class="option-group">
<input type="checkbox" id="optHttp" checked>
<label for="optHttp">HTTP Analysis</label>
</div>
<div class="option-group">
<input type="checkbox" id="optBlacklist" checked>
<label for="optBlacklist">Blacklists</label>
</div>
<div class="option-group slow">
<input type="checkbox" id="optPorts">
<label for="optPorts">Port Scan (lent)</label>
</div>
<div class="option-group slow">
<input type="checkbox" id="optSubdomains">
<label for="optSubdomains">Subdomenii (lent)</label>
</div>
</div>
</div>
<div class="loading" id="loading">
<div class="spinner"></div>
<p>Se verifică domeniul... Acest proces poate dura până la 30 de secunde.</p>
</div>
<div class="results" id="results"></div>
</div>
<script>
const API_URL = '/api/v1/check/check';
document.getElementById('domainInput').addEventListener('keypress', (e) => {
if (e.key === 'Enter') checkDomain();
});
async function checkDomain() {
const domain = document.getElementById('domainInput').value.trim();
if (!domain) { alert('Introdu un domeniu!'); return; }
const btn = document.getElementById('checkBtn');
const loading = document.getElementById('loading');
const results = document.getElementById('results');
btn.disabled = true;
loading.style.display = 'block';
results.style.display = 'none';
const checkOptions = {
whois: document.getElementById('optWhois').checked,
dns: document.getElementById('optDns').checked,
ssl: document.getElementById('optSsl').checked,
ip_intelligence: document.getElementById('optIp').checked,
http_analysis: document.getElementById('optHttp').checked,
blacklist: document.getElementById('optBlacklist').checked,
port_scan: document.getElementById('optPorts').checked,
subdomains: document.getElementById('optSubdomains').checked
};
try {
const response = await fetch(API_URL, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ domain, check_options: checkOptions })
});
const data = await response.json();
if (data.success) {
displayResults(data.data, data.metadata);
} else {
results.innerHTML = `<div class="error">❌ ${data.error?.message || 'Eroare necunoscută'}</div>`;
}
} catch (err) {
results.innerHTML = `<div class="error">❌ Eroare de conexiune: ${err.message}</div>`;
}
btn.disabled = false;
loading.style.display = 'none';
results.style.display = 'block';
}
function displayResults(d, meta) {
let html = '<div class="results-grid">';
if (d.availability) html += renderAvailabilityCard(d.availability);
if (d.risk_score) html += renderRiskCard(d.risk_score);
if (d.whois) html += renderWhoisCard(d.whois);
if (d.dns) html += renderDnsCard(d.dns);
if (d.mail) html += renderMailCard(d.mail);
if (d.ssl) html += renderSslCard(d.ssl);
if (d.ip_intelligence) html += renderIpCard(d.ip_intelligence);
if (d.http_analysis) html += renderHttpCard(d.http_analysis);
if (d.blacklist) html += renderBlacklistCard(d.blacklist);
if (d.port_scan) html += renderPortCard(d.port_scan);
if (d.subdomains) html += renderSubdomainsCard(d.subdomains);
html += '</div>';
html += `<div style="text-align: center; margin-top: 20px; color: #666; font-size: 0.85rem;">
Check ID: ${d.check_id} | Timp procesare: ${meta.processing_time_ms}ms | API: ${meta.api_version}
</div>`;
document.getElementById('results').innerHTML = html;
}
function renderAvailabilityCard(a) {
const reg = a.is_registered;
const label = reg ? '🔒 Înregistrat' : '🟢 Disponibil';
const badgeClass = reg ? 'info' : 'success';
return `
<div class="card">
<div class="card-header"><h3>📌 Disponibilitate</h3><span class="source">WHOIS+DNS</span></div>
<div class="card-body">
<div class="data-row"><div class="data-label">Status:</div><div class="data-value"><span class="badge ${badgeClass}" style="font-size:1rem">${label}</span></div></div>
<div class="data-row"><div class="data-label">Încredere:</div><div class="data-value">${a.confidence}</div></div>
<div class="data-row"><div class="data-label">Semnale:</div><div class="data-value">
<span class="badge ${a.signals?.whois_has_data ? 'success' : 'neutral'}">WHOIS ${a.signals?.whois_has_data ? '✓' : '–'}</span>
<span class="badge ${a.signals?.dns_resolves ? 'success' : 'neutral'}">DNS ${a.signals?.dns_resolves ? '✓' : '–'}</span>
</div></div>
</div>
</div>`;
}
function renderMailCard(m) {
const gradeColor = {A:'#00c853', B:'#8bc34a', C:'#ffc107', D:'#ff9800', F:'#f44336'}[m.mail_security_grade] || '#888';
const spf = m.spf || {}, dmarc = m.dmarc || {}, dkim = m.dkim || {}, mx = m.mx || {}, sts = m.mta_sts || {};
const tlsRow = (mx.starttls !== null && mx.starttls !== undefined) ? ` · STARTTLS ${mx.starttls ? '✓' : '✗'}` : '';
return `
<div class="card">
<div class="card-header"><h3>📧 Mail Intelligence</h3><span class="source">EMAIL</span></div>
<div class="card-body">
<div class="data-row"><div class="data-label">Securitate email:</div><div class="data-value"><span class="badge" style="background:${gradeColor};color:#fff">${m.mail_security_grade} · ${m.mail_security_score}/100</span></div></div>
${m.provider ? `<div class="data-row"><div class="data-label">Provider:</div><div class="data-value">${m.provider}</div></div>` : ''}
<div class="data-row"><div class="data-label">MX:</div><div class="data-value">${mx.count || 0} servere${tlsRow}${mx.dane ? ' · DANE ✓' : ''}</div></div>
<div class="data-row"><div class="data-label">SPF:</div><div class="data-value"><span class="badge ${spf.present ? 'success' : 'danger'}">${spf.present ? '✓' : '✗'}</span> ${spf.policy || ''}</div></div>
<div class="data-row"><div class="data-label">DKIM:</div><div class="data-value"><span class="badge ${dkim.present ? 'success' : 'danger'}">${dkim.present ? '✓' : '✗'}</span> ${(dkim.selectors_found || []).join(', ')}</div></div>
<div class="data-row"><div class="data-label">DMARC:</div><div class="data-value"><span class="badge ${dmarc.present ? 'success' : 'danger'}">${dmarc.present ? '✓' : '✗'}</span> ${dmarc.policy ? ('p=' + dmarc.policy) : ''}</div></div>
<div class="data-row"><div class="data-label">MTA-STS:</div><div class="data-value"><span class="badge ${sts.present ? 'success' : 'neutral'}">${sts.present ? '✓' : '–'}</span>${sts.mode ? (' ' + sts.mode) : ''} ${m.tls_rpt?.present ? '<span class="badge success">TLS-RPT ✓</span>' : ''}</div></div>
${m.deliverability?.tested ? `<div class="data-row"><div class="data-label">Catch-all:</div><div class="data-value">${m.deliverability.catch_all ? '<span class="badge warning">DA</span>' : '<span class="badge success">NU</span>'}</div></div>` : ''}
</div>
</div>`;
}
function renderRiskCard(risk) {
const levelClass = `risk-${risk.level.toLowerCase()}`;
let factorsHtml = '';
if (risk.factors && risk.factors.length > 0) {
risk.factors.forEach(f => {
const barColor = f.score <= 25 ? '#00c853' : f.score <= 50 ? '#ffc107' : f.score <= 75 ? '#ff9800' : '#f44336';
factorsHtml += `
<div class="risk-factor">
<div class="name">${formatFactorName(f.factor)}</div>
<div class="score">
<div class="bar"><div class="bar-fill" style="width: ${f.score}%; background: ${barColor}"></div></div>
<div class="value">${f.score}</div>
</div>
</div>`;
});
}
let formulaHtml = '';
if (risk.formula_breakdown && risk.formula_breakdown.length > 0) {
formulaHtml = '<div class="formula-box"><h4>📐 Formula de Calcul</h4>';
risk.formula_breakdown.forEach(fb => {
formulaHtml += `<div class="formula-line">${fb.category}: ${fb.formula}</div>`;
});
formulaHtml += `<div class="formula-line" style="margin-top: 10px; font-weight: bold; color: #00d9ff;">${risk.formula_string || ''}</div></div>`;
}
return `
<div class="card risk-card">
<div class="card-header"><h3>🎯 Scor de Risc</h3><span class="source">CALCULATED</span></div>
<div class="card-body">
<div class="risk-score-display">
<div class="risk-circle ${levelClass}">${risk.total}<span class="label">${risk.level}</span></div>
<div class="risk-details">
<div style="margin-bottom: 15px;">
${risk.is_new_domain ? '<span class="badge warning">Domeniu Nou</span>' : ''}
${risk.is_suspicious ? '<span class="badge danger">Suspect</span>' : ''}
${risk.is_blacklisted ? '<span class="badge danger">Blacklisted</span>' : ''}
${!risk.is_suspicious && !risk.is_blacklisted ? '<span class="badge success">OK</span>' : ''}
</div>
${factorsHtml}
</div>
</div>
${formulaHtml}
<div style="margin-top: 15px; font-size: 0.85rem; color: #888;">Praguri: LOW (0-25) | MEDIUM (26-50) | HIGH (51-75) | CRITICAL (76-100)</div>
</div>
</div>`;
}
function renderWhoisCard(w) {
const statusArray = Array.isArray(w.status) ? w.status : (w.status ? [w.status] : []);
const nsArray = Array.isArray(w.name_servers) ? w.name_servers : (w.name_servers ? [w.name_servers] : []);
return `
<div class="card">
<div class="card-header"><h3>📋 WHOIS</h3><span class="source">${w.data_source || 'WHOIS'}</span></div>
<div class="card-body">
${w.creation_date ? `<div class="data-row"><div class="data-label">Data Creare:</div><div class="data-value">${formatDate(w.creation_date)}</div></div>` : ''}
${w.age_days !== null && w.age_days !== undefined ? `<div class="data-row"><div class="data-label">Vârsta:</div><div class="data-value"><strong>${w.age_days} zile</strong> (${Math.floor(w.age_days/365)} ani)</div></div>` : ''}
${w.expiration_date ? `<div class="data-row"><div class="data-label">Expirare:</div><div class="data-value">${formatDate(w.expiration_date)}</div></div>` : ''}
${w.registrar ? `<div class="data-row"><div class="data-label">Registrar:</div><div class="data-value">${w.registrar}</div></div>` : ''}
${w.dnssec ? `<div class="data-row"><div class="data-label">DNSSEC:</div><div class="data-value">${w.dnssec}</div></div>` : ''}
${statusArray.length > 0 ? `<div class="data-row"><div class="data-label">Status:</div><div class="data-value">${statusArray.map(s => `<span class="badge info">${s}</span>`).join('')}</div></div>` : ''}
${nsArray.length > 0 ? `<div class="data-row"><div class="data-label">Name Servers:</div><div class="data-value">${nsArray.map(ns => `<span class="badge neutral">${ns}</span>`).join('')}</div></div>` : ''}
</div>
</div>`;
}
function renderDnsCard(dns) {
return `
<div class="card">
<div class="card-header"><h3>🌐 DNS Records</h3><span class="source">DNS</span></div>
<div class="card-body">
${dns.a_records?.length ? `<div class="data-row"><div class="data-label">A (IPv4):</div><div class="data-value">${dns.a_records.map(r => `<span class="badge info">${r}</span>`).join('')}</div></div>` : ''}
${dns.aaaa_records?.length ? `<div class="data-row"><div class="data-label">AAAA (IPv6):</div><div class="data-value">${dns.aaaa_records.map(r => `<span class="badge info">${r}</span>`).join('')}</div></div>` : ''}
${dns.mx_records?.length ? `<div class="data-row"><div class="data-label">MX:</div><div class="data-value">${dns.mx_records.map(r => `<span class="badge neutral">${r.priority} ${r.host}</span>`).join('')}</div></div>` : ''}
${dns.ns_records?.length ? `<div class="data-row"><div class="data-label">NS:</div><div class="data-value">${dns.ns_records.map(r => `<span class="badge neutral">${r}</span>`).join('')}</div></div>` : ''}
<div class="data-row"><div class="data-label">SPF:</div><div class="data-value"><span class="badge ${dns.has_spf ? 'success' : 'danger'}">${dns.has_spf ? '✓' : '✗'} SPF</span> <span style="color:#888;font-size:0.8rem">(DKIM/DMARC: vezi cardul Mail Intelligence)</span></div></div>
</div>
</div>`;
}
function renderSslCard(ssl) {
if (!ssl.has_ssl) {
return `<div class="card"><div class="card-header"><h3>🔒 SSL/TLS</h3><span class="source">SSL</span></div><div class="card-body"><div class="error">❌ Nu are certificat SSL</div></div></div>`;
}
return `
<div class="card">
<div class="card-header"><h3>🔒 SSL/TLS</h3><span class="source">SSL</span></div>
<div class="card-body">
<div class="data-row"><div class="data-label">Status:</div><div class="data-value">
<span class="badge ${ssl.is_valid ? 'success' : 'danger'}">${ssl.is_valid ? '✓ Valid' : '✗ Invalid'}</span>
${ssl.is_expired ? '<span class="badge danger">Expirat</span>' : ''}
${ssl.is_wildcard ? '<span class="badge info">Wildcard</span>' : ''}
</div></div>
${ssl.issuer ? `<div class="data-row"><div class="data-label">Issuer:</div><div class="data-value">${ssl.issuer}</div></div>` : ''}
${ssl.valid_until ? `<div class="data-row"><div class="data-label">Valid Until:</div><div class="data-value">${formatDate(ssl.valid_until)}</div></div>` : ''}
${ssl.days_until_expiry !== null ? `<div class="data-row"><div class="data-label">Expiră în:</div><div class="data-value"><span class="badge ${ssl.days_until_expiry < 30 ? 'warning' : 'success'}">${ssl.days_until_expiry} zile</span></div></div>` : ''}
</div>
</div>`;
}
function renderIpCard(ip) {
return `
<div class="card">
<div class="card-header"><h3>🌍 IP Intelligence</h3><span class="source">ipinfo.io</span></div>
<div class="card-body">
<div class="data-row"><div class="data-label">IP:</div><div class="data-value"><strong>${ip.ip}</strong></div></div>
${ip.reverse_dns ? `<div class="data-row"><div class="data-label">Reverse DNS:</div><div class="data-value">${ip.reverse_dns}</div></div>` : ''}
${ip.country ? `<div class="data-row"><div class="data-label">Locație:</div><div class="data-value">${ip.city || ''} ${ip.region ? ', ' + ip.region : ''}, ${ip.country}</div></div>` : ''}
${ip.asn ? `<div class="data-row"><div class="data-label">ASN:</div><div class="data-value">${ip.asn}</div></div>` : ''}
${ip.isp ? `<div class="data-row"><div class="data-label">ISP:</div><div class="data-value">${ip.isp}</div></div>` : ''}
<div class="data-row"><div class="data-label">Tip:</div><div class="data-value"><span class="badge ${ip.is_datacenter ? 'info' : 'success'}">${ip.is_datacenter ? 'Datacenter' : 'Rezidențial'}</span></div></div>
${ip.hosting_score ? `<div class="data-row"><div class="data-label">Hosting Score:</div><div class="data-value"><span class="badge ${ip.hosting_score.score <= 30 ? 'success' : ip.hosting_score.score <= 50 ? 'warning' : 'danger'}">${ip.hosting_score.score}/100</span></div></div>` : ''}
</div>
</div>`;
}
function renderHttpCard(http) {
const secHeaders = http.security_headers || {};
let headersHtml = '';
['Strict-Transport-Security', 'X-Frame-Options', 'X-Content-Type-Options', 'Content-Security-Policy'].forEach(h => {
headersHtml += `<div class="security-header"><span>${h}</span><span class="${secHeaders[h] ? 'header-present' : 'header-missing'}">${secHeaders[h] ? '✓' : '✗'}</span></div>`;
});
return `
<div class="card">
<div class="card-header"><h3>🌐 HTTP Analysis</h3><span class="source">HTTP</span></div>
<div class="card-body">
<div class="data-row"><div class="data-label">HTTPS:</div><div class="data-value">
<span class="badge ${http.has_https ? 'success' : 'danger'}">${http.has_https ? '✓' : '✗'} HTTPS</span>
${http.http_to_https_redirect ? '<span class="badge success">Auto-redirect</span>' : ''}
</div></div>
${http.server ? `<div class="data-row"><div class="data-label">Server:</div><div class="data-value">${http.server}</div></div>` : ''}
${http.cms ? `<div class="data-row"><div class="data-label">CMS:</div><div class="data-value"><span class="badge info">${http.cms}</span></div></div>` : ''}
${http.technologies?.length ? `<div class="data-row"><div class="data-label">Tech:</div><div class="data-value"><div class="tech-tags">${http.technologies.slice(0,8).map(t => `<span class="tech-tag">${t}</span>`).join('')}</div></div></div>` : ''}
${http.response_time_ms ? `<div class="data-row"><div class="data-label">Response:</div><div class="data-value">${http.response_time_ms}ms</div></div>` : ''}
<div style="margin-top: 15px; font-weight: 500; margin-bottom: 10px;">Security Headers:</div>
${headersHtml}
</div>
</div>`;
}
function renderBlacklistCard(bl) {
let checksHtml = '';
if (bl.ip_check) {
const clean = bl.ip_check.clean_lists?.slice(0,4) || [];
const listed = bl.ip_check.listings || [];
checksHtml += `<div style="margin-bottom: 10px;"><strong>IP (${bl.ip_check.total_checked}):</strong> ${listed.map(l => `<span class="badge danger">${l.list_name}</span>`).join('')} ${clean.map(l => `<span class="badge success">${l}</span>`).join('')}</div>`;
}
if (bl.domain_check) {
const clean = bl.domain_check.clean_lists?.slice(0,4) || [];
const listed = bl.domain_check.listings || [];
checksHtml += `<div><strong>Domain (${bl.domain_check.total_checked}):</strong> ${listed.map(l => `<span class="badge danger">${l.list_name}</span>`).join('')} ${clean.map(l => `<span class="badge success">${l}</span>`).join('')}</div>`;
}
return `
<div class="card">
<div class="card-header"><h3>🛡️ Blacklists</h3><span class="source">DNSBL</span></div>
<div class="card-body">
<div class="data-row"><div class="data-label">Status:</div><div class="data-value"><span class="badge ${bl.is_blacklisted ? 'danger' : 'success'}">${bl.is_blacklisted ? '❌ BLACKLISTED' : '✓ CLEAN'}</span></div></div>
<div class="data-row"><div class="data-label">Reputation:</div><div class="data-value"><span class="badge ${bl.reputation_score >= 80 ? 'success' : bl.reputation_score >= 60 ? 'warning' : 'danger'}">${bl.reputation_score}/100</span></div></div>
${checksHtml}
</div>
</div>`;
}
function renderPortCard(ports) {
const openPorts = ports.open_ports || [];
const dangerous = ports.dangerous_open || [];
const summary = ports.scan_summary || {};
let portsHtml = '<div class="port-list">';
openPorts.forEach(p => {
const isDanger = dangerous.some(d => d.port === p.port);
portsHtml += `<span class="port-item ${isDanger ? 'danger' : 'safe'}">${p.port} ${p.service}</span>`;
});
portsHtml += '</div>';
return `
<div class="card">
<div class="card-header"><h3>🔌 Port Scan</h3><span class="source">TCP</span></div>
<div class="card-body">
<div class="data-row"><div class="data-label">Rezumat:</div><div class="data-value">
<span class="badge info">${summary.open_count || 0} deschise</span>
${summary.dangerous_count > 0 ? `<span class="badge danger">${summary.dangerous_count} periculoase</span>` : ''}
</div></div>
<div class="data-row"><div class="data-label">Porturi:</div><div class="data-value">${portsHtml}</div></div>
</div>
</div>`;
}
function renderSubdomainsCard(subs) {
const subList = subs.subdomains || [];
const src = subs.sources || {};
const srcCounts = [
['CT', (src.certificate_transparency || []).length],
['Brute', (src.dns_bruteforce || []).length],
['AXFR', (src.zone_transfer || []).length],
['DNS', (src.dns_records || []).length],
['Manual', (src.user_provided || []).length],
].filter(([, n]) => n > 0).map(([k, n]) => `<span class="badge neutral">${k}: ${n}</span>`).join(' ');
return `
<div class="card">
<div class="card-header"><h3>🔎 Subdomenii</h3><span class="source">CT+DNS+Brute</span></div>
<div class="card-body">
<div class="data-row"><div class="data-label">Total:</div><div class="data-value"><strong>${subs.total_found}</strong> ${srcCounts}</div></div>
<div class="data-row"><div class="data-label">Lista:</div><div class="data-value"><div class="tech-tags">${subList.slice(0, 20).map(s => `<span class="tech-tag">${s}</span>`).join('')}${subList.length > 20 ? `<span class="badge neutral">+${subList.length - 20}</span>` : ''}</div></div></div>
${subs.has_wildcard_cert ? `<div class="data-row"><div class="data-label">⚠️ Wildcard:</div><div class="data-value"><span class="badge warning">cert *.${subs.domain}</span></div></div>` : ''}
${subs.coverage_note ? `<div style="margin-top:8px;font-size:0.78rem;color:#b8860b;line-height:1.4">ℹ️ ${subs.coverage_note}</div>` : ''}
</div>
</div>`;
}
function formatFactorName(factor) {
const names = {'domain_age': '📅 Vârsta Domeniu', 'ssl': '🔒 SSL/TLS', 'dns': '🌐 DNS', 'email_security': '📧 Email Security', 'whois': '📋 WHOIS', 'ip_reputation': '🌍 IP Reputation', 'http_security': '🔐 HTTP Security', 'blacklist': '🛡️ Blacklists', 'port_security': '🔌 Porturi'};
return names[factor] || factor;
}
function formatDate(dateStr) {
if (!dateStr) return '-';
try { return new Date(dateStr).toLocaleDateString('ro-RO', { year: 'numeric', month: 'long', day: 'numeric' }); }
catch { return dateStr; }
}
</script>
</body>
</html>

0
ai_platform/modules/domain_check/api/app/tasks/__init__.py Normal file
View file

0
ai_platform/modules/domain_check/api/app/utils/__init__.py Normal file
View file

91
ai_platform/modules/domain_check/api/requirements.txt Normal file
View file

@ -0,0 +1,91 @@
# Flask Framework
Flask==3.0.0
Flask-RESTX==1.3.0
Flask-SQLAlchemy==3.1.1
Flask-Migrate==4.0.5
Flask-CORS==4.0.0
Flask-Limiter==3.5.0
# Database
SQLAlchemy==2.0.23
psycopg2-binary==2.9.9
alembic==1.13.1
# Redis & Caching
redis==5.0.1
hiredis==2.3.2
# Celery (Task Queue)
celery==5.3.4
# HTTP Requests (moved before WHOIS tools to resolve dependencies)
requests==2.31.0
urllib3==2.1.0
httpx==0.25.2
# WHOIS & Domain Tools
whoisit>=2.0.0
python-whois==0.8.0
dnspython==2.4.2
tldextract==5.1.1
# Validation & Serialization
marshmallow==3.20.1
marshmallow-sqlalchemy==0.29.0
pydantic==2.5.2
# Environment & Configuration
python-dotenv==1.0.0
environs==10.3.0
# Date & Time
python-dateutil==2.8.2
pytz==2023.3
# Logging & Monitoring
structlog==23.3.0
sentry-sdk[flask]==1.39.1
# Security
PyJWT==2.8.0
cryptography==41.0.7
validators==0.22.0
# API Documentation
apispec==6.3.1
apispec-webframeworks==1.0.0
# Utilities
Click==8.1.7
colorama==0.4.6
python-slugify==8.0.1
shortuuid==1.0.11
# Testing
pytest==7.4.3
pytest-cov==4.1.0
pytest-flask==1.3.0
pytest-mock==3.12.0
faker==21.0.0
# Code Quality
black==23.12.1
flake8==6.1.0
pylint==3.0.3
mypy==1.7.1
# Performance
gunicorn==21.2.0
gevent==23.9.1
# Data Processing
pandas==2.1.4
numpy==1.26.2
# IP & Network
ipaddress==1.0.23
geoip2==4.7.0
# SSL Certificate Handling
certifi==2023.11.17
pyOpenSSL==23.3.0

45
ai_platform/modules/domain_check/api/run.py Normal file
View file

@ -0,0 +1,45 @@
"""
Domain Check API - Entry Point
Version: 1.0.0
"""
import os
import sys
from pathlib import Path
# Add app directory to Python path
sys.path.insert(0, str(Path(__file__).parent))
from app import create_app
# Create Flask app instance
app = create_app(os.getenv('FLASK_ENV', 'development'))
if __name__ == '__main__':
host = os.getenv('API_HOST', '0.0.0.0')
port = int(os.getenv('API_PORT', 5000))
debug = os.getenv('DEBUG', 'True').lower() == 'true'
print(f"""
╔══════════════════════════════════════════╗
║ Domain Check API - Anti-Fake News ║
║ Version: 1.0.0 ║
║ Environment: {os.getenv('FLASK_ENV', 'development'):<26}║
║ Host: {host:<32}║
║ Port: {port:<32}║
╚══════════════════════════════════════════╝
📚 API Documentation:
- Swagger UI: http://{host}:{port}/docs
- ReDoc: http://{host}:{port}/redoc
🏥 Health Check: http://{host}:{port}/health
🚀 Starting server...
""")
app.run(
host=host,
port=port,
debug=debug,
threaded=True
)

104
ai_platform/modules/domain_check/api/tests/test_whois_dates.py Normal file
View file

@ -0,0 +1,104 @@
"""
Regression tests for WHOIS date handling.
Root cause of the historical HTTP 500 on `.ro` domains (e.g. exemplu.ro):
python-whois returns ROTLD dates as raw strings, and downstream code did
`datetime - str`, raising TypeError. These tests lock in the fix.
"""
from datetime import datetime
import pytest
from app.services.whois_service import normalize_whois_date, _as_list, WhoisService
from app.services.risk_scorer import RiskScorer, _coerce_datetime
# --- normalize_whois_date -------------------------------------------------
@pytest.mark.parametrize("value", [
None,
"",
" ",
"none",
"N/A",
])
def test_normalize_returns_none_for_empty(value):
assert normalize_whois_date(value) is None
def test_normalize_passthrough_datetime():
dt = datetime(2020, 1, 2, 3, 4, 5)
assert normalize_whois_date(dt) == dt
def test_normalize_iso_string():
assert normalize_whois_date("2020-01-02T03:04:05Z") == datetime(2020, 1, 2, 3, 4, 5)
def test_normalize_rotld_style_string():
# ROTLD / free-form formats that broke datetime.fromisoformat
assert normalize_whois_date("2015-03-22") == datetime(2015, 3, 22)
assert normalize_whois_date("22.03.2015").year == 2015
assert normalize_whois_date("Before 2012-08-10") is not None # fuzzy
def test_normalize_list_takes_first_valid():
out = normalize_whois_date([None, "2019-05-06", datetime(2021, 1, 1)])
assert out == datetime(2019, 5, 6)
def test_normalize_garbage_returns_none():
assert normalize_whois_date("not a date at all !!") is None
def test_as_list_normalizes():
assert _as_list(None) == []
assert _as_list("a") == ["a"]
assert _as_list(["a", None, "", "b"]) == ["a", "b"]
# --- the actual crash site: risk scorer must never raise on bad dates -----
def test_score_domain_age_does_not_raise_on_string():
"""The exact regression: a raw string creation_date must not 500."""
scorer = RiskScorer()
score, details = scorer._score_domain_age({'creation_date': '2015-03-22'})
assert isinstance(score, int)
assert details['details']['age_days'] is not None
def test_score_domain_age_does_not_raise_on_garbage():
scorer = RiskScorer()
score, details = scorer._score_domain_age({'creation_date': 'totally-not-a-date'})
# Garbage coerces to None -> "unable to determine" path, never an exception.
assert score == 50
assert details['details']['age_days'] is None
def test_score_domain_age_missing():
scorer = RiskScorer()
score, details = scorer._score_domain_age({})
assert score == 50
def test_coerce_datetime():
assert _coerce_datetime(None) is None
assert _coerce_datetime("2020-01-01") == datetime(2020, 1, 1)
assert _coerce_datetime(datetime(2020, 1, 1)) == datetime(2020, 1, 1)
assert _coerce_datetime("garbage") is None
# --- is_registered signal -------------------------------------------------
def test_is_registered_true_when_creation_date():
assert WhoisService._is_registered({'creation_date': datetime(2020, 1, 1)}) is True
def test_is_registered_true_when_registrar():
assert WhoisService._is_registered({'registrar': 'ROTLD'}) is True
def test_is_registered_none_when_sparse():
# No positive signal -> inconclusive (None), DNS decides downstream.
assert WhoisService._is_registered({'creation_date': None, 'registrar': None,
'name_servers': [], 'status': []}) is None

42
ai_platform/modules/domain_check/dashboard/Dockerfile Normal file
View file

@ -0,0 +1,42 @@
# Streamlit Dashboard Dockerfile
FROM python:3.10-slim
# Set working directory
WORKDIR /app
# Install system dependencies
RUN apt-get update && apt-get install -y --no-install-recommends \
curl \
&& rm -rf /var/lib/apt/lists/*
# Copy requirements
COPY requirements.txt .
# Install Python dependencies
RUN pip install --no-cache-dir --upgrade pip && \
pip install --no-cache-dir -r requirements.txt
# Create non-root user
RUN useradd -m -u 1000 streamlit && \
chown -R streamlit:streamlit /app
# Copy application code
COPY --chown=streamlit:streamlit . /app/
# Create Streamlit config directory
RUN mkdir -p /home/streamlit/.streamlit && \
chown -R streamlit:streamlit /home/streamlit
# Switch to non-root user
USER streamlit
# Expose Streamlit port
EXPOSE 8501
# Health check
HEALTHCHECK --interval=30s --timeout=10s --start-period=20s --retries=3 \
CMD curl -f http://localhost:8501/_stcore/health || exit 1
# Run Streamlit
CMD ["streamlit", "run", "app.py", "--server.port=8501", "--server.address=0.0.0.0", "--server.headless=true", "--browser.gatherUsageStats=false"]

26
ai_platform/modules/domain_check/dashboard/requirements.txt Normal file
View file

@ -0,0 +1,26 @@
# Streamlit Framework
streamlit==1.29.0
# HTTP Requests
requests==2.31.0
httpx==0.25.2
# Data Processing & Visualization
pandas==2.1.4
numpy==1.26.2
plotly==5.18.0
altair==5.2.0
# Date & Time
python-dateutil==2.8.2
pytz==2023.3
# Utilities
python-dotenv==1.0.0
validators==0.22.0
# Caching
streamlit-autorefresh==1.0.1
# Formatting
humanize==4.9.0

291
ai_platform/modules/domain_check/deploy.sh Executable file
View file

@ -0,0 +1,291 @@
#!/bin/bash
# ===========================================
# Domain Check API - Deployment Script
# ===========================================
# Usage: ./deploy.sh [install|update|status|logs]
# ===========================================
set -e
# Colors
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
NC='\033[0m'
# Configuration
REPO_URL="<didi-lot1-ai>/ai_platform/modules/domain_check"
INSTALL_DIR="/home/$(whoami)/domain-check"
BRANCH="main"
log_info() { echo -e "${GREEN}[INFO]${NC} $1"; }
log_warn() { echo -e "${YELLOW}[WARN]${NC} $1"; }
log_error() { echo -e "${RED}[ERROR]${NC} $1"; }
check_requirements() {
log_info "Checking requirements..."
# Check Docker
if ! command -v docker &> /dev/null; then
log_error "Docker is not installed. Install it first:"
echo " curl -fsSL https://get.docker.com | sh"
echo " sudo usermod -aG docker \$USER"
exit 1
fi
# Check Docker Compose
if ! docker compose version &> /dev/null; then
log_error "Docker Compose is not installed."
exit 1
fi
# Check Git
if ! command -v git &> /dev/null; then
log_error "Git is not installed. Install it first:"
echo " sudo apt install git -y"
exit 1
fi
log_info "All requirements met ✓"
}
install() {
log_info "Installing Domain Check API..."
check_requirements
# Clone repository
if [ -d "$INSTALL_DIR" ]; then
log_warn "Directory $INSTALL_DIR already exists."
read -p "Remove and reinstall? (y/N) " -n 1 -r
echo
if [[ $REPLY =~ ^[Yy]$ ]]; then
rm -rf "$INSTALL_DIR"
else
log_info "Use './deploy.sh update' to update existing installation."
exit 0
fi
fi
log_info "Cloning repository..."
git clone "$REPO_URL" "$INSTALL_DIR"
cd "$INSTALL_DIR"
# Create .env file
create_env
# Build and start
log_info "Building and starting containers..."
docker compose up -d --build
# Wait for health
log_info "Waiting for services to be healthy..."
sleep 10
# Check status
status
log_info "Installation complete!"
log_info "API available at: http://$(hostname -I | awk '{print $1}'):51000/"
}
create_env() {
log_info "Creating .env file..."
if [ -f ".env" ]; then
log_warn ".env file already exists. Skipping creation."
return
fi
# Generate secure password and secret key
DB_PASS=$(openssl rand -base64 24 | tr -dc 'a-zA-Z0-9' | head -c 24)
SECRET=$(openssl rand -base64 32 | tr -dc 'a-zA-Z0-9' | head -c 32)
cat > .env << EOF
# ===========================================
# Domain Check API - Environment Configuration
# Port Schema: Dev (5xxxx) per Port_Allocation_Schema_v2
# Generated: $(date)
# ===========================================
# Database Configuration (52000 = x2xxx = Databases/PostgreSQL)
DB_PASSWORD=${DB_PASS}
DB_USER=dns_admin
DB_NAME=domain_check
DB_HOST=domain_check_postgres
DB_PORT=5432
DB_EXTERNAL_PORT=52000
# Redis Configuration (52300 = x23xx = Redis)
REDIS_HOST=domain_check_redis
REDIS_PORT=6379
REDIS_EXTERNAL_PORT=52300
REDIS_DB=0
REDIS_PASSWORD=
# API Keys (set your own keys)
WHOXY_API_KEY=
VIRUSTOTAL_API_KEY=
# Application Settings
FLASK_ENV=production
FLASK_APP=run.py
SECRET_KEY=${SECRET}
LOG_LEVEL=INFO
DEBUG=False
# Cache TTL (seconds)
REDIS_TTL_HOT=21600
REDIS_TTL_WARM=86400
REDIS_TTL_COLD=604800
# Risk Score Thresholds
RISK_THRESHOLD_LOW=30
RISK_THRESHOLD_MEDIUM=60
RISK_THRESHOLD_HIGH=85
# Domain Age Thresholds (days)
DOMAIN_AGE_CRITICAL=90
DOMAIN_AGE_HIGH=180
DOMAIN_AGE_MEDIUM=365
# API Rate Limiting
WHOXY_DAILY_LIMIT=8000
VIRUSTOTAL_DAILY_LIMIT=500
# Batch Processing
BATCH_CHUNK_SIZE=50
BATCH_TIMEOUT_SECONDS=300
# API Configuration (51000 = x1xxx = API/Gateway)
API_HOST=0.0.0.0
API_PORT=51000
API_VERSION=v1
# Celery Configuration
CELERY_BROKER_URL=redis://domain_check_redis:6379/1
CELERY_RESULT_BACKEND=redis://domain_check_redis:6379/2
# Monitoring (optional)
SENTRY_DSN=
DATADOG_API_KEY=
EOF
log_info ".env file created with secure passwords."
log_warn "Edit .env to add your API keys (WHOXY_API_KEY, etc.)"
}
update() {
log_info "Updating Domain Check API..."
cd "$INSTALL_DIR" || { log_error "Installation not found at $INSTALL_DIR"; exit 1; }
# Pull latest changes
log_info "Pulling latest changes..."
git pull origin "$BRANCH"
# Rebuild and restart
log_info "Rebuilding containers..."
docker compose down
docker compose up -d --build
# Wait and check
sleep 10
status
log_info "Update complete!"
}
status() {
log_info "Checking status..."
cd "$INSTALL_DIR" 2>/dev/null || { log_error "Installation not found"; exit 1; }
echo ""
docker compose ps
echo ""
# Health check
API_PORT=$(grep API_PORT .env | cut -d= -f2)
API_PORT=${API_PORT:-51000}
if curl -s "http://localhost:${API_PORT}/health" | grep -q "healthy"; then
log_info "API Health: ✓ Healthy"
curl -s "http://localhost:${API_PORT}/health" | python3 -m json.tool 2>/dev/null || true
else
log_error "API Health: ✗ Not responding"
fi
}
logs() {
cd "$INSTALL_DIR" 2>/dev/null || { log_error "Installation not found"; exit 1; }
if [ -n "$2" ]; then
docker compose logs -f "$2"
else
docker compose logs -f
fi
}
stop() {
log_info "Stopping services..."
cd "$INSTALL_DIR" 2>/dev/null || { log_error "Installation not found"; exit 1; }
docker compose down
log_info "Services stopped."
}
start() {
log_info "Starting services..."
cd "$INSTALL_DIR" 2>/dev/null || { log_error "Installation not found"; exit 1; }
docker compose up -d
sleep 5
status
}
restart() {
stop
start
}
# Main
case "${1:-}" in
install)
install
;;
update)
update
;;
status)
status
;;
logs)
logs "$@"
;;
stop)
stop
;;
start)
start
;;
restart)
restart
;;
*)
echo "Domain Check API - Deployment Script"
echo ""
echo "Usage: $0 {install|update|status|logs|start|stop|restart}"
echo ""
echo "Commands:"
echo " install - Fresh installation"
echo " update - Pull latest changes and rebuild"
echo " status - Check services status"
echo " logs - View logs (optional: service name)"
echo " start - Start services"
echo " stop - Stop services"
echo " restart - Restart services"
echo ""
echo "Example:"
echo " $0 install"
echo " $0 logs domain_check_api"
;;
esac

151
ai_platform/modules/domain_check/docker-compose.yml Normal file
View file

@ -0,0 +1,151 @@
# DIDI Domain Check (T4) — modul dedicat platformei DIDI.
# Izolare: Postgres + Redis proprii pe rețeaua internă `domain_check_network`.
# Integrare: API-ul se atașează și la `didi-network` (externă, partajată cu
# restul platformei) sub aliasul `domain-check-api`, ca agent-v3 (Lot 2) să-l
# apeleze la `http://domain-check-api:11000/api/v1/check/check` fără alt config.
networks:
domain_check_network:
driver: bridge
name: domain_check_network
didi-network:
external: true
volumes:
domain_check_postgres_data:
name: domain_check_postgres_data
domain_check_redis_data:
name: domain_check_redis_data
domain_check_api_logs:
name: domain_check_api_logs
services:
domain_check_postgres:
image: postgres:15-alpine
container_name: didiAI-domain-check-db
networks:
- domain_check_network
environment:
POSTGRES_DB: ${DB_NAME}
POSTGRES_USER: ${DB_USER}
POSTGRES_PASSWORD: ${DB_PASSWORD}
PGDATA: /var/lib/postgresql/data/pgdata
volumes:
- domain_check_postgres_data:/var/lib/postgresql/data
- ./init-scripts:/docker-entrypoint-initdb.d:ro
healthcheck:
test:
- CMD-SHELL
- pg_isready -U ${DB_USER} -d ${DB_NAME}
interval: 10s
timeout: 5s
retries: 5
start_period: 10s
restart: unless-stopped
domain_check_redis:
image: redis:7-alpine
container_name: didiAI-domain-check-redis
networks:
- domain_check_network
command: 'redis-server --appendonly yes --maxmemory 512mb --maxmemory-policy allkeys-lru
--save 900 1 --save 300 10 --save 60 10000
'
volumes:
- domain_check_redis_data:/data
healthcheck:
test:
- CMD
- redis-cli
- ping
interval: 10s
timeout: 3s
retries: 5
start_period: 5s
restart: unless-stopped
domain_check_api:
build:
context: ./api
dockerfile: Dockerfile
image: ${REGISTRY_IMAGE:-didiai-domain-check}:${IMAGE_TAG:-latest}
container_name: didiAI-domain-check
networks:
domain_check_network: {}
didi-network:
aliases:
- domain-check-api
dns:
- 8.8.8.8
- 8.8.4.4
- 1.1.1.1
environment:
- DATABASE_URL=postgresql://${DB_USER}:${DB_PASSWORD}@${DB_HOST}:${DB_PORT}/${DB_NAME}
- REDIS_URL=redis://${REDIS_HOST}:${REDIS_PORT}/${REDIS_DB}
- WHOXY_API_KEY=${WHOXY_API_KEY}
- VIRUSTOTAL_API_KEY=${VIRUSTOTAL_API_KEY}
- FLASK_ENV=${FLASK_ENV}
- FLASK_APP=${FLASK_APP}
- SECRET_KEY=${SECRET_KEY}
- LOG_LEVEL=${LOG_LEVEL}
- DEBUG=${DEBUG}
- REDIS_TTL_HOT=${REDIS_TTL_HOT}
- REDIS_TTL_WARM=${REDIS_TTL_WARM}
- REDIS_TTL_COLD=${REDIS_TTL_COLD}
- API_HOST=${API_HOST}
- API_PORT=${API_PORT}
- API_VERSION=${API_VERSION}
volumes:
- domain_check_api_logs:/app/logs
depends_on:
domain_check_postgres:
condition: service_healthy
domain_check_redis:
condition: service_healthy
healthcheck:
test:
- CMD
- curl
- -f
- http://localhost:${API_PORT}/health
interval: 30s
timeout: 10s
retries: 3
start_period: 40s
restart: unless-stopped
command: gunicorn --workers 2 --worker-class gthread --threads 4 --timeout 120
--graceful-timeout 30 --access-logfile - --error-logfile - --bind 0.0.0.0:${API_PORT}
run:app
domain_check_worker:
build:
context: ./api
dockerfile: Dockerfile
image: ${REGISTRY_IMAGE:-didiai-domain-check}:${IMAGE_TAG:-latest}
container_name: didiAI-domain-check-worker
networks:
- domain_check_network
dns:
- 8.8.8.8
- 8.8.4.4
- 1.1.1.1
command: celery -A app.celery_app worker --loglevel=info --concurrency=4
environment:
- DATABASE_URL=postgresql://${DB_USER}:${DB_PASSWORD}@${DB_HOST}:${DB_PORT}/${DB_NAME}
- REDIS_URL=redis://${REDIS_HOST}:${REDIS_PORT}/${REDIS_DB}
- CELERY_BROKER_URL=${CELERY_BROKER_URL}
- CELERY_RESULT_BACKEND=${CELERY_RESULT_BACKEND}
- WHOXY_API_KEY=${WHOXY_API_KEY}
- VIRUSTOTAL_API_KEY=${VIRUSTOTAL_API_KEY}
- LOG_LEVEL=${LOG_LEVEL}
depends_on:
- domain_check_api
- domain_check_redis
healthcheck:
test:
- CMD
- celery
- -A
- app.celery_app
- inspect
- ping
interval: 30s
timeout: 10s
retries: 3
start_period: 30s
restart: unless-stopped

413
ai_platform/modules/domain_check/init-scripts/01-init-db.sql Normal file
View file

@ -0,0 +1,413 @@
-- Domain Check Database Initialization Script
-- Version: 1.0.0
-- Date: 2026-01-29
-- Enable UUID extension
CREATE EXTENSION IF NOT EXISTS "uuid-ossp";
CREATE EXTENSION IF NOT EXISTS "pg_trgm"; -- For similarity searches
-- Set timezone
SET TIME ZONE 'UTC';
-- ============================================================================
-- TABLES
-- ============================================================================
-- 1. Domains table
CREATE TABLE IF NOT EXISTS domains (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
domain VARCHAR(255) NOT NULL,
subdomain VARCHAR(255),
tld VARCHAR(50) NOT NULL,
full_domain VARCHAR(255) UNIQUE NOT NULL, -- Complete domain (subdomain.domain.tld or domain.tld)
first_seen_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
last_checked_at TIMESTAMP WITH TIME ZONE,
check_count INTEGER DEFAULT 0,
is_active BOOLEAN DEFAULT TRUE,
created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
updated_at TIMESTAMP WITH TIME ZONE DEFAULT NOW()
);
CREATE INDEX idx_domains_full_domain ON domains(full_domain);
CREATE INDEX idx_domains_domain ON domains(domain);
CREATE INDEX idx_domains_last_checked ON domains(last_checked_at);
CREATE INDEX idx_domains_tld ON domains(tld);
CREATE INDEX idx_domains_is_active ON domains(is_active);
-- 2. WHOIS Records table
CREATE TABLE IF NOT EXISTS whois_records (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
domain_id UUID NOT NULL REFERENCES domains(id) ON DELETE CASCADE,
creation_date TIMESTAMP WITH TIME ZONE,
expiration_date TIMESTAMP WITH TIME ZONE,
updated_date TIMESTAMP WITH TIME ZONE,
registrar VARCHAR(255),
registrar_url VARCHAR(500),
registrant_org VARCHAR(255),
registrant_country VARCHAR(2),
admin_email VARCHAR(255),
name_servers TEXT[], -- Array of name servers
status TEXT[], -- Array of domain statuses
dnssec BOOLEAN,
raw_whois_data JSONB,
data_source VARCHAR(50) DEFAULT 'rdap', -- 'rdap', 'whoxy', 'manual'
fetched_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW()
);
CREATE INDEX idx_whois_domain_id ON whois_records(domain_id);
CREATE INDEX idx_whois_creation_date ON whois_records(creation_date);
CREATE INDEX idx_whois_registrar ON whois_records(registrar);
CREATE INDEX idx_whois_data_source ON whois_records(data_source);
CREATE INDEX idx_whois_fetched_at ON whois_records(fetched_at DESC);
-- 3. DNS Records table
CREATE TABLE IF NOT EXISTS dns_records (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
domain_id UUID NOT NULL REFERENCES domains(id) ON DELETE CASCADE,
record_type VARCHAR(10) NOT NULL, -- 'A', 'AAAA', 'MX', 'TXT', 'NS', 'CNAME', 'SOA'
record_value TEXT NOT NULL,
ttl INTEGER,
priority INTEGER, -- For MX records
fetched_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW()
);
CREATE INDEX idx_dns_domain_id ON dns_records(domain_id);
CREATE INDEX idx_dns_domain_type ON dns_records(domain_id, record_type);
CREATE INDEX idx_dns_record_type ON dns_records(record_type);
CREATE INDEX idx_dns_fetched_at ON dns_records(fetched_at DESC);
-- 4. SSL Certificates table
CREATE TABLE IF NOT EXISTS ssl_certificates (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
domain_id UUID NOT NULL REFERENCES domains(id) ON DELETE CASCADE,
issuer VARCHAR(255),
subject VARCHAR(255),
valid_from TIMESTAMP WITH TIME ZONE,
valid_until TIMESTAMP WITH TIME ZONE,
serial_number VARCHAR(255),
signature_algorithm VARCHAR(100),
key_size INTEGER,
is_wildcard BOOLEAN DEFAULT FALSE,
is_self_signed BOOLEAN DEFAULT FALSE,
is_valid BOOLEAN DEFAULT TRUE,
certificate_chain JSONB,
fetched_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW()
);
CREATE INDEX idx_ssl_domain_id ON ssl_certificates(domain_id);
CREATE INDEX idx_ssl_valid_until ON ssl_certificates(valid_until);
CREATE INDEX idx_ssl_issuer ON ssl_certificates(issuer);
CREATE INDEX idx_ssl_is_valid ON ssl_certificates(is_valid);
CREATE INDEX idx_ssl_fetched_at ON ssl_certificates(fetched_at DESC);
-- 5. Reputation Scores table
CREATE TABLE IF NOT EXISTS reputation_scores (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
domain_id UUID NOT NULL REFERENCES domains(id) ON DELETE CASCADE,
source VARCHAR(50) NOT NULL, -- 'virustotal', 'custom', 'opensquat', 'phishtank'
score INTEGER CHECK (score >= 0 AND score <= 100),
malicious_count INTEGER DEFAULT 0,
suspicious_count INTEGER DEFAULT 0,
harmless_count INTEGER DEFAULT 0,
undetected_count INTEGER DEFAULT 0,
is_blacklisted BOOLEAN DEFAULT FALSE,
blacklist_names TEXT[],
is_typosquatting BOOLEAN DEFAULT FALSE,
typosquatting_target VARCHAR(255),
raw_response JSONB,
checked_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW()
-- Note: UNIQUE constraint on domain_id + source + date handled at application level
);
CREATE INDEX idx_reputation_domain_id ON reputation_scores(domain_id);
CREATE INDEX idx_reputation_source ON reputation_scores(source);
CREATE INDEX idx_reputation_is_blacklisted ON reputation_scores(is_blacklisted);
CREATE INDEX idx_reputation_is_typosquatting ON reputation_scores(is_typosquatting);
CREATE INDEX idx_reputation_checked_at ON reputation_scores(checked_at DESC);
-- 6. Risk Assessments table
CREATE TABLE IF NOT EXISTS risk_assessments (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
domain_id UUID NOT NULL REFERENCES domains(id) ON DELETE CASCADE,
check_id UUID UNIQUE NOT NULL,
total_score INTEGER NOT NULL CHECK (total_score >= 0 AND total_score <= 100),
risk_level VARCHAR(20) NOT NULL CHECK (risk_level IN ('LOW', 'MEDIUM', 'HIGH', 'CRITICAL')),
-- Individual factor scores
domain_age_score INTEGER DEFAULT 0,
domain_age_days INTEGER,
ssl_score INTEGER DEFAULT 0,
dns_score INTEGER DEFAULT 0,
reputation_score INTEGER DEFAULT 0,
whois_score INTEGER DEFAULT 0,
-- Risk factors breakdown (JSONB array)
factors JSONB,
-- Flags
is_new_domain BOOLEAN DEFAULT FALSE,
is_suspicious BOOLEAN DEFAULT FALSE,
requires_manual_review BOOLEAN DEFAULT FALSE,
assessed_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW()
);
CREATE INDEX idx_risk_domain_id ON risk_assessments(domain_id);
CREATE INDEX idx_risk_check_id ON risk_assessments(check_id);
CREATE INDEX idx_risk_level ON risk_assessments(risk_level);
CREATE INDEX idx_risk_total_score ON risk_assessments(total_score);
CREATE INDEX idx_risk_is_new_domain ON risk_assessments(is_new_domain);
CREATE INDEX idx_risk_is_suspicious ON risk_assessments(is_suspicious);
CREATE INDEX idx_risk_assessed_at ON risk_assessments(assessed_at DESC);
-- 7. Check History table
CREATE TABLE IF NOT EXISTS check_history (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
check_id UUID UNIQUE NOT NULL,
domain_id UUID NOT NULL REFERENCES domains(id) ON DELETE CASCADE,
risk_assessment_id UUID REFERENCES risk_assessments(id) ON DELETE SET NULL,
-- Check metadata
requested_by VARCHAR(100) DEFAULT 'api', -- 'api', 'cli', 'dashboard', 'batch'
request_ip VARCHAR(45),
user_agent TEXT,
check_options JSONB,
-- Performance metrics
processing_time_ms INTEGER,
cache_hit BOOLEAN DEFAULT FALSE,
-- Changes detection
changes_detected BOOLEAN DEFAULT FALSE,
change_summary JSONB,
-- Status
status VARCHAR(20) DEFAULT 'completed', -- 'pending', 'processing', 'completed', 'failed'
error_message TEXT,
created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW()
);
CREATE INDEX idx_check_history_check_id ON check_history(check_id);
CREATE INDEX idx_check_history_domain_id ON check_history(domain_id);
CREATE INDEX idx_check_history_domain_created ON check_history(domain_id, created_at DESC);
CREATE INDEX idx_check_history_created_at ON check_history(created_at DESC);
CREATE INDEX idx_check_history_status ON check_history(status);
CREATE INDEX idx_check_history_requested_by ON check_history(requested_by);
-- 8. Batch Operations table
CREATE TABLE IF NOT EXISTS batch_operations (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
batch_id UUID UNIQUE NOT NULL,
total_domains INTEGER NOT NULL,
completed_count INTEGER DEFAULT 0,
failed_count INTEGER DEFAULT 0,
status VARCHAR(20) DEFAULT 'pending' CHECK (status IN ('pending', 'processing', 'completed', 'failed', 'cancelled')),
priority VARCHAR(20) DEFAULT 'normal' CHECK (priority IN ('low', 'normal', 'high', 'urgent')),
-- Timing
started_at TIMESTAMP WITH TIME ZONE,
completed_at TIMESTAMP WITH TIME ZONE,
estimated_completion_at TIMESTAMP WITH TIME ZONE,
-- Metadata
requested_by VARCHAR(100),
check_options JSONB,
domain_list TEXT[], -- Array of domains to check
-- Results
error_log JSONB,
summary JSONB,
created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW()
);
CREATE INDEX idx_batch_batch_id ON batch_operations(batch_id);
CREATE INDEX idx_batch_status ON batch_operations(status);
CREATE INDEX idx_batch_priority ON batch_operations(priority);
CREATE INDEX idx_batch_created_at ON batch_operations(created_at DESC);
-- 9. Blacklists table
CREATE TABLE IF NOT EXISTS blacklists (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
domain VARCHAR(255) UNIQUE NOT NULL,
reason TEXT,
category VARCHAR(50), -- 'phishing', 'malware', 'spam', 'fake_news', 'disinformation'
source VARCHAR(100),
severity VARCHAR(20) DEFAULT 'medium' CHECK (severity IN ('low', 'medium', 'high', 'critical')),
added_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
updated_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
is_active BOOLEAN DEFAULT TRUE,
expires_at TIMESTAMP WITH TIME ZONE,
-- Additional metadata
metadata JSONB
);
CREATE INDEX idx_blacklist_domain ON blacklists(domain);
CREATE INDEX idx_blacklist_category ON blacklists(category);
CREATE INDEX idx_blacklist_is_active ON blacklists(is_active);
CREATE INDEX idx_blacklist_severity ON blacklists(severity);
-- 10. API Usage Logs table
CREATE TABLE IF NOT EXISTS api_usage_logs (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
service_name VARCHAR(50) NOT NULL, -- 'whoxy', 'virustotal', 'rdap'
endpoint VARCHAR(255),
request_count INTEGER DEFAULT 1,
response_time_ms INTEGER,
status_code INTEGER,
quota_used INTEGER,
quota_remaining INTEGER,
error_message TEXT,
request_params JSONB,
logged_at TIMESTAMP WITH TIME ZONE DEFAULT NOW()
);
CREATE INDEX idx_api_usage_service ON api_usage_logs(service_name);
CREATE INDEX idx_api_usage_service_logged ON api_usage_logs(service_name, logged_at DESC);
CREATE INDEX idx_api_usage_logged_at ON api_usage_logs(logged_at DESC);
CREATE INDEX idx_api_usage_status ON api_usage_logs(status_code);
-- ============================================================================
-- VIEWS
-- ============================================================================
-- View for latest domain information
CREATE OR REPLACE VIEW v_latest_domain_info AS
SELECT
d.id,
d.full_domain,
d.domain,
d.subdomain,
d.tld,
d.first_seen_at,
d.last_checked_at,
d.check_count,
ra.total_score as latest_risk_score,
ra.risk_level as latest_risk_level,
ra.is_new_domain,
ra.is_suspicious,
w.creation_date as domain_creation_date,
w.registrar,
w.registrant_country,
EXTRACT(DAY FROM (NOW() - w.creation_date)) as domain_age_days
FROM domains d
LEFT JOIN LATERAL (
SELECT * FROM risk_assessments
WHERE domain_id = d.id
ORDER BY assessed_at DESC
LIMIT 1
) ra ON TRUE
LEFT JOIN LATERAL (
SELECT * FROM whois_records
WHERE domain_id = d.id
ORDER BY fetched_at DESC
LIMIT 1
) w ON TRUE;
-- View for high-risk domains
CREATE OR REPLACE VIEW v_high_risk_domains AS
SELECT
d.full_domain,
ra.total_score,
ra.risk_level,
ra.is_new_domain,
ra.domain_age_days,
ra.assessed_at,
rs.is_blacklisted,
rs.is_typosquatting
FROM domains d
JOIN risk_assessments ra ON d.id = ra.domain_id
LEFT JOIN reputation_scores rs ON d.id = rs.domain_id
WHERE ra.risk_level IN ('HIGH', 'CRITICAL')
ORDER BY ra.total_score DESC, ra.assessed_at DESC;
-- View for daily statistics
CREATE OR REPLACE VIEW v_daily_stats AS
SELECT
DATE(created_at) as check_date,
COUNT(DISTINCT domain_id) as unique_domains_checked,
COUNT(*) as total_checks,
AVG(processing_time_ms) as avg_processing_time_ms,
SUM(CASE WHEN cache_hit THEN 1 ELSE 0 END) as cache_hits,
SUM(CASE WHEN NOT cache_hit THEN 1 ELSE 0 END) as cache_misses,
ROUND(100.0 * SUM(CASE WHEN cache_hit THEN 1 ELSE 0 END) / COUNT(*), 2) as cache_hit_rate
FROM check_history
WHERE status = 'completed'
GROUP BY DATE(created_at)
ORDER BY check_date DESC;
-- ============================================================================
-- FUNCTIONS
-- ============================================================================
-- Function to update updated_at timestamp
CREATE OR REPLACE FUNCTION update_updated_at_column()
RETURNS TRIGGER AS $$
BEGIN
NEW.updated_at = NOW();
RETURN NEW;
END;
$$ LANGUAGE plpgsql;
-- Triggers for updated_at
CREATE TRIGGER update_domains_updated_at BEFORE UPDATE ON domains
FOR EACH ROW EXECUTE FUNCTION update_updated_at_column();
CREATE TRIGGER update_blacklists_updated_at BEFORE UPDATE ON blacklists
FOR EACH ROW EXECUTE FUNCTION update_updated_at_column();
-- Function to increment domain check count
CREATE OR REPLACE FUNCTION increment_domain_check_count()
RETURNS TRIGGER AS $$
BEGIN
UPDATE domains
SET check_count = check_count + 1,
last_checked_at = NOW()
WHERE id = NEW.domain_id;
RETURN NEW;
END;
$$ LANGUAGE plpgsql;
-- Trigger to auto-increment check count
CREATE TRIGGER increment_check_count AFTER INSERT ON check_history
FOR EACH ROW EXECUTE FUNCTION increment_domain_check_count();
-- ============================================================================
-- INITIAL DATA
-- ============================================================================
-- Insert some common blacklist entries (example)
INSERT INTO blacklists (domain, reason, category, source, severity) VALUES
('known-phishing-site.com', 'Known phishing operation', 'phishing', 'manual', 'critical'),
('fake-news-urgent.com', 'Disinformation campaign', 'fake_news', 'manual', 'high'),
('malware-distribution.com', 'Malware distribution', 'malware', 'manual', 'critical')
ON CONFLICT (domain) DO NOTHING;
-- ============================================================================
-- GRANTS
-- ============================================================================
-- Grant permissions (adjust as needed)
GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA public TO dns_admin;
GRANT ALL PRIVILEGES ON ALL SEQUENCES IN SCHEMA public TO dns_admin;
GRANT USAGE, SELECT ON ALL SEQUENCES IN SCHEMA public TO dns_admin;
-- ============================================================================
-- COMPLETION
-- ============================================================================
-- Log initialization
DO $$
BEGIN
RAISE NOTICE 'Database initialization completed successfully!';
RAISE NOTICE 'Total tables created: 10';
RAISE NOTICE 'Total views created: 3';
RAISE NOTICE 'Total functions created: 2';
END $$;

41
ai_platform/modules/domain_check/test-api-complete.sh Executable file
View file

@ -0,0 +1,41 @@
#!/bin/bash
echo "=========================================="
echo " DOMAIN-CHECK API - Complete Test Suite"
echo "=========================================="
echo ""
API="http://localhost:5000"
# Test 1: Health
echo "✓ Test 1: Health Check"
curl -s $API/health | python3 -m json.tool | head -10
echo ""
# Test 2: Check with WHOIS
echo "✓ Test 2: WHOIS Check - github.com"
curl -s -X POST $API/api/v1/check \
-H "Content-Type: application/json" \
-d '{"domain": "github.com", "check_options": {"whois": true, "dns": false, "ssl": false}}' \
| python3 -m json.tool | head -40
echo ""
# Test 3: Check with DNS
echo "✓ Test 3: DNS Check - google.com"
curl -s -X POST $API/api/v1/check \
-H "Content-Type: application/json" \
-d '{"domain": "google.com", "check_options": {"whois": false, "dns": true, "ssl": false}}' \
| python3 -m json.tool | grep -A 20 '"dns"'
echo ""
# Test 4: Full check
echo "✓ Test 4: Full Check (WHOIS + DNS) - facebook.com"
curl -s -X POST $API/api/v1/check \
-H "Content-Type: application/json" \
-d '{"domain": "facebook.com", "check_options": {"whois": true, "dns": true, "ssl": false}}' \
| python3 -m json.tool | head -50
echo ""
echo "=========================================="
echo " Tests Complete!"
echo "=========================================="

114
ai_platform/modules/domain_check/test-complete-lan.sh Executable file
View file

@ -0,0 +1,114 @@
#!/bin/bash
LAN_IP="10.11.10.200"
API="http://${LAN_IP}:5000"
echo "=========================================="
echo " DOMAIN-CHECK - Complete Test Suite"
echo " API: $API"
echo "=========================================="
echo ""
# Test 1: Health Check
echo "✓ Test 1: Health Check"
curl -s "${API}/health" | python3 -m json.tool
if [ $? -eq 0 ]; then
echo "✅ Health check PASSED"
else
echo "❌ Health check FAILED"
exit 1
fi
echo ""
# Test 2: API Root
echo "✓ Test 2: API Root"
curl -s "${API}/" | python3 -m json.tool
echo ""
# Test 3: WHOIS Only
echo "✓ Test 3: WHOIS Check - github.com"
curl -s -X POST "${API}/api/v1/check" \
-H "Content-Type: application/json" \
-d '{"domain": "github.com", "check_options": {"whois": true, "dns": false, "ssl": false}}' \
| python3 -m json.tool > /tmp/test3.json
if grep -q "\"success\": true" /tmp/test3.json; then
echo "✅ WHOIS check PASSED"
cat /tmp/test3.json | jq '.data.whois' | head -15
else
echo "❌ WHOIS check FAILED"
cat /tmp/test3.json
fi
echo ""
# Test 4: DNS Only
echo "✓ Test 4: DNS Check - google.com"
curl -s -X POST "${API}/api/v1/check" \
-H "Content-Type: application/json" \
-d '{"domain": "google.com", "check_options": {"whois": false, "dns": true, "ssl": false}}' \
| python3 -m json.tool > /tmp/test4.json
if grep -q "\"success\": true" /tmp/test4.json; then
echo "✅ DNS check PASSED"
cat /tmp/test4.json | jq '.data.dns' | head -20
else
echo "❌ DNS check FAILED"
cat /tmp/test4.json
fi
echo ""
# Test 5: Full Check (WHOIS + DNS)
echo "✓ Test 5: Full Check - facebook.com"
curl -s -X POST "${API}/api/v1/check" \
-H "Content-Type: application/json" \
-d '{"domain": "facebook.com", "check_options": {"whois": true, "dns": true, "ssl": false}}' \
| python3 -m json.tool > /tmp/test5.json
if grep -q "\"success\": true" /tmp/test5.json; then
echo "✅ Full check PASSED"
cat /tmp/test5.json | jq '.data.risk_score' | head -15
else
echo "❌ Full check FAILED"
cat /tmp/test5.json
fi
echo ""
# Test 6: Risk Scoring
echo "✓ Test 6: Risk Score Analysis"
cat /tmp/test5.json | jq '{
domain: .data.domain,
risk_level: .data.risk_score.level,
total_score: .data.risk_score.total,
factors: .data.risk_score.factors | map({factor: .factor, score: .score})
}'
echo ""
# Test 7: Database Check
echo "✓ Test 7: Database Verification"
docker exec dns_postgres psql -U dns_admin -d domain_check -c \
"SELECT d.full_domain, ra.total_score, ra.risk_level, ra.assessed_at
FROM risk_assessments ra
JOIN domains d ON ra.domain_id = d.id
ORDER BY ra.assessed_at DESC LIMIT 5;"
echo ""
# Test 8: Swagger UI
echo "✓ Test 8: Swagger UI"
echo " URL: http://${LAN_IP}:5000/docs"
echo ""
# Test 9: ReDoc
echo "✓ Test 9: ReDoc Documentation"
echo " URL: http://${LAN_IP}:5000/redoc"
echo ""
echo "=========================================="
echo " ✅ All Tests Complete!"
echo "=========================================="
echo ""
echo "📊 Access Points:"
echo " API: http://${LAN_IP}:5000"
echo " Docs: http://${LAN_IP}:5000/docs"
echo " ReDoc: http://${LAN_IP}:5000/redoc"
echo " Health: http://${LAN_IP}:5000/health"
echo ""

578
ai_platform/modules/domain_check/use_api.md Normal file
View file

@ -0,0 +1,578 @@
# Domain Check API - Developer Documentation
**Base URL:** `http://domain-check-api:11000`
**API Version:** v1
---
## Authentication
Currently no authentication required (internal use only).
---
## Endpoints
### POST /api/v1/check/check
Perform comprehensive domain verification.
**Request:**
```bash
curl -X POST "http://domain-check-api:11000/api/v1/check/check" \
-H "Content-Type: application/json" \
-d '{
"domain": "example.com",
"check_options": {
"whois": true,
"dns": true,
"ssl": true,
"ip_intelligence": true,
"http_analysis": true,
"blacklist": true,
"port_scan": true,
"subdomains": true
}
}'
```
**Check Options:**
| Option | Type | Default | Description |
|--------|------|---------|-------------|
| `whois` | bool | true | WHOIS lookup (registrar, dates, nameservers) |
| `dns` | bool | true | DNS records (A, AAAA, MX, TXT, NS, CNAME, SOA) |
| `ssl` | bool | true | SSL certificate validation |
| `ip_intelligence` | bool | true | IP geolocation, ASN, reverse DNS |
| `http_analysis` | bool | true | HTTP headers, security, technologies |
| `blacklist` | bool | true | DNSBL spam/malware blacklist check |
| `port_scan` | bool | false | TCP port scanning (slower) |
| `subdomains` | bool | false | Subdomain enumeration (slower) |
| `force_refresh` | bool | false | Bypass cache |
---
## Response Structure
```json
{
"success": true,
"data": {
"domain": "example.com",
"check_id": "uuid-v4",
"timestamp": "2026-02-04T09:32:18.459703Z",
"whois": { ... },
"dns": { ... },
"ssl": { ... },
"ip_intelligence": { ... },
"http_analysis": { ... },
"blacklist": { ... },
"port_scan": { ... },
"subdomains": { ... },
"risk_score": { ... }
},
"metadata": {
"cached": false,
"processing_time_ms": 40315,
"api_version": "v1",
"checks_performed": ["whois", "dns", "ssl", ...]
}
}
```
---
## Data Sections
### 1. WHOIS (`data.whois`)
```json
{
"creation_date": "2017-10-13T07:59:48Z",
"expiration_date": "2026-10-13T07:59:48Z",
"updated_date": "2025-01-15T06:24:03Z",
"registrar": "Vautron Rechenzentrum AG",
"age_days": 3036,
"status": "ok https://icann.org/epp#ok",
"name_servers": ["NS1.CONTABO.NET", "NS2.CONTABO.NET"],
"dnssec": null,
"registrant_org": null,
"registrant_country": "RO",
"data_source": "whois"
}
```
### 2. DNS (`data.dns`)
```json
{
"a_records": ["84.54.23.135"],
"aaaa_records": [],
"mx_records": [{"priority": 10, "host": "mail.example.com"}],
"txt_records": ["v=spf1 include:_spf.google.com ~all"],
"ns_records": ["ns1.example.com", "ns2.example.com"],
"cname_records": [],
"soa_record": {
"mname": "ns1.example.com",
"rname": "hostmaster.example.com",
"serial": 1770195563,
"refresh": 10800,
"retry": 3600,
"expire": 604800,
"minimum": 3600
},
"has_spf": true,
"has_dkim": false,
"has_dmarc": true,
"spf_record": null,
"dmarc_record": null
}
```
### 3. SSL (`data.ssl`)
```json
{
"has_ssl": true,
"is_valid": true,
"is_self_signed": false,
"is_expired": false,
"is_wildcard": false,
"issuer": "CN=R13, O=Let's Encrypt, C=US",
"subject": "CN=example.com",
"valid_from": "2025-12-22T12:48:47Z",
"valid_until": "2026-03-22T12:48:46Z",
"days_until_expiry": 46,
"key_size": 2048,
"signature_algorithm": "sha256WithRSAEncryption",
"san": ["example.com", "www.example.com", "mail.example.com"]
}
```
### 4. IP Intelligence (`data.ip_intelligence`)
```json
{
"ip": "84.54.23.135",
"reverse_dns": "server.provider.net",
"geolocation": null,
"asn": "AS51167",
"isp": "Contabo GmbH",
"organization": "Contabo GmbH",
"is_datacenter": true,
"is_residential": false,
"hostname": "server.provider.net",
"city": "Lauterbourg",
"region": "Grand Est",
"country": "France",
"country_code": "FR",
"coordinates": {"latitude": 48.9751, "longitude": 8.1785},
"timezone": "Europe/Paris",
"postal": "67630",
"data_source": "ipinfo.io",
"hosting_score": {
"score": 50,
"reasons": [],
"is_trusted": false,
"is_suspicious": false
}
}
```
### 5. HTTP Analysis (`data.http_analysis`)
```json
{
"domain": "example.com",
"http_status": 200,
"https_status": 200,
"has_https": true,
"http_to_https_redirect": true,
"final_url": "https://example.com/",
"redirect_chain": [],
"response_time_ms": 3351,
"server": "Apache",
"powered_by": null,
"security_headers": {},
"missing_security_headers": [
{
"header": "Strict-Transport-Security",
"description": "HSTS - Forces HTTPS",
"severity": "CRITICAL"
},
{
"header": "X-Frame-Options",
"description": "Prevents clickjacking",
"severity": "HIGH"
}
],
"cookies": [],
"technologies": ["WordPress", "jQuery", "Apache"],
"cms": "WordPress",
"frameworks": ["jQuery"],
"has_robots_txt": true,
"has_sitemap": true,
"has_favicon": true,
"error": null
}
```
### 6. Blacklist (`data.blacklist`)
```json
{
"domain": "example.com",
"ip": "84.54.23.135",
"is_blacklisted": false,
"ip_blacklisted": false,
"domain_blacklisted": false,
"total_listings": 0,
"ip_check": {
"ip": "84.54.23.135",
"is_blacklisted": false,
"blacklist_count": 0,
"clean_count": 4,
"total_checked": 4,
"listings": [],
"clean_lists": ["Spamhaus ZEN", "SpamCop", "Barracuda", "SORBS"],
"check_errors": []
},
"domain_check": {
"domain": "example.com",
"is_blacklisted": false,
"blacklist_count": 0,
"clean_count": 4,
"total_checked": 4,
"listings": [],
"clean_lists": ["Spamhaus DBL", "URIBL", "URIBL Black", "SURBL"],
"check_errors": []
},
"reputation_score": 100,
"risk_level": "LOW"
}
```
**Blacklists Checked:**
| List | Type | Description |
|------|------|-------------|
| Spamhaus ZEN | IP | Combined spam blocklist |
| SpamCop | IP | User-reported spam sources |
| Barracuda | IP | Barracuda reputation |
| SORBS | IP | Spam and relay blocking |
| Spamhaus DBL | Domain | Domain blocklist |
| URIBL | Domain | URI blocklist |
| URIBL Black | Domain | High-confidence spam URIs |
| SURBL | Domain | Spam URI realtime blocklist |
### 7. Port Scan (`data.port_scan`)
```json
{
"ip": "84.54.23.135",
"total_scanned": 9,
"open_ports": [
{
"port": 443,
"service": "HTTPS",
"category": "web",
"risk_level": "low",
"banner": null
},
{
"port": 22,
"service": "SSH",
"category": "remote_access",
"risk_level": "low",
"banner": "SSH-2.0-OpenSSH_8.7"
}
],
"closed_ports": [8080, 23, 21, 3389],
"filtered_ports": [],
"dangerous_open": [
{
"port": 3306,
"service": "MySQL",
"category": "database",
"risk_level": "critical",
"banner": "Host not allowed to connect"
}
],
"services_detected": ["HTTPS", "HTTP", "SSH", "MySQL"],
"categories": {
"web": [443, 80],
"remote_access": [22],
"database": [3306]
},
"security_issues": [
{
"severity": "CRITICAL",
"port": 3306,
"service": "MySQL",
"issue": "Dangerous service MySQL exposed on port 3306"
}
],
"scan_summary": {
"open_count": 5,
"closed_count": 4,
"filtered_count": 0,
"dangerous_count": 1,
"has_web": true,
"has_email": true,
"has_database": true,
"has_remote_access": true
}
}
```
**Ports Scanned (Quick Scan):**
| Port | Service | Risk Level |
|------|---------|------------|
| 21 | FTP | Medium |
| 22 | SSH | Low |
| 23 | Telnet | Critical |
| 25 | SMTP | Low |
| 80 | HTTP | Low |
| 443 | HTTPS | Low |
| 3306 | MySQL | Critical |
| 3389 | RDP | High |
| 8080 | HTTP Proxy | Medium |
### 8. Subdomains (`data.subdomains`)
```json
{
"domain": "example.com",
"subdomains": [
"admin.example.com",
"api.example.com",
"mail.example.com",
"www.example.com"
],
"total_found": 93,
"sources": {
"certificate_transparency": ["mail.example.com", "www.example.com"],
"dns_bruteforce": ["admin.example.com", "api.example.com"],
"dns_records": ["mail.example.com"]
},
"live_subdomains": [
{
"subdomain": "www.example.com",
"ip": "84.54.23.135",
"http": 301,
"https": 200
}
],
"error": null
}
```
**Subdomain Sources:**
1. **Certificate Transparency** - crt.sh logs
2. **DNS Bruteforce** - Common subdomain prefixes
3. **DNS Records** - MX, NS, SOA records
### 9. Risk Score (`data.risk_score`)
```json
{
"total": 13,
"level": "LOW",
"factors": [
{
"factor": "domain_age",
"score": 0,
"weight": 0.2,
"weighted_score": 0,
"reason": "TRUSTED: Mature domain (3036 days, 8+ years)",
"details": {"creation_date": "2017-10-13", "age_days": 3036}
}
],
"formula_breakdown": [
{
"category": "Domain Age",
"raw_score": 0,
"weight": 0.2,
"weighted_score": 0,
"formula": "0 × 0.2 = 0.0"
}
],
"formula_string": "Total = (0 × 0.2) + (0 × 0.15) + ... = 13",
"thresholds": {
"low": "0-25",
"medium": "26-50",
"high": "51-75",
"critical": "76-100"
},
"is_new_domain": false,
"is_suspicious": false,
"is_blacklisted": false,
"requires_manual_review": false
}
```
**Risk Score Formula:**
| Category | Weight | Description |
|----------|--------|-------------|
| Domain Age | 20% | New domains = higher risk |
| SSL/TLS | 15% | Certificate validity |
| DNS Configuration | 10% | Complete DNS setup |
| Email Security | 10% | SPF, DKIM, DMARC |
| WHOIS Privacy | 5% | Registration info |
| IP Reputation | 10% | Datacenter/residential |
| HTTP Security | 10% | Security headers |
| Blacklist Status | 15% | DNSBL listings |
| Port Security | 5% | Exposed dangerous ports |
**Risk Levels:**
| Level | Score Range | Color |
|-------|-------------|-------|
| LOW | 0-25 | Green |
| MEDIUM | 26-50 | Yellow |
| HIGH | 51-75 | Orange |
| CRITICAL | 76-100 | Red |
---
## Examples
### Quick Check (Basic)
```bash
curl -X POST "http://domain-check-api:11000/api/v1/check/check" \
-H "Content-Type: application/json" \
-d '{"domain": "google.com"}'
```
### Full Check (All Options)
```bash
curl -X POST "http://domain-check-api:11000/api/v1/check/check" \
-H "Content-Type: application/json" \
-d '{
"domain": "example.com",
"check_options": {
"whois": true,
"dns": true,
"ssl": true,
"ip_intelligence": true,
"http_analysis": true,
"blacklist": true,
"port_scan": true,
"subdomains": true
}
}'
```
### Python Example
```python
import requests
response = requests.post(
"http://domain-check-api:11000/api/v1/check/check",
json={
"domain": "example.com",
"check_options": {
"whois": True,
"dns": True,
"ssl": True,
"ip_intelligence": True,
"http_analysis": True,
"blacklist": True,
"port_scan": True,
"subdomains": True
}
}
)
data = response.json()
print(f"Risk Score: {data['data']['risk_score']['total']}")
print(f"Risk Level: {data['data']['risk_score']['level']}")
print(f"Is Blacklisted: {data['data']['blacklist']['is_blacklisted']}")
```
### JavaScript Example
```javascript
const response = await fetch('http://domain-check-api:11000/api/v1/check/check', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({
domain: 'example.com',
check_options: {
whois: true,
dns: true,
ssl: true,
ip_intelligence: true,
http_analysis: true,
blacklist: true,
port_scan: true,
subdomains: true
}
})
});
const data = await response.json();
console.log(`Risk Score: ${data.data.risk_score.total}`);
console.log(`Risk Level: ${data.data.risk_score.level}`);
```
---
## Error Responses
```json
{
"success": false,
"error": {
"code": "INVALID_REQUEST",
"message": "Domain parameter is required",
"status": 400
}
}
```
**Error Codes:**
| Code | Status | Description |
|------|--------|-------------|
| `INVALID_REQUEST` | 400 | Missing or invalid parameters |
| `INVALID_DOMAIN` | 400 | Domain format invalid |
| `NOT_FOUND` | 404 | Resource not found |
| `INTERNAL_ERROR` | 500 | Server error |
---
## Rate Limits
No rate limits currently (internal use).
---
## Dashboard
Web interface available at: `http://domain-check-api:11000/`
---
## Network Info
- **Host:** domain-check-api
- **IP:** 10.11.10.200
- **Port:** 11000 (x1xxx = API/Gateway per Port Schema v2)
- **Protocol:** HTTP (internal network)
## Port Allocation (Production Environment - 1xxxx)
| Service | Port | Category |
|---------|------|----------|
| API | 11000 | x1xxx = API/Gateway |
| PostgreSQL | 12000 | x20xx = Databases/PostgreSQL |
| Redis | 12300 | x23xx = Cache/Redis |

39
ai_platform/modules/embeddings/INDEX.md
View file

@ -20,7 +20,7 @@ Generate text embeddings for downstream DIDI consumers:
- **didi-brain semantic atom storage** — text claims/atoms get embedded once at ingestion, stored as pgvector columns.
- **didi-brain `/v1/gather` query embedding** — incoming claim text gets embedded, used as the kNN probe vector.
- **Cross-encoder reranking** — separate `rerank` module (port 54200) handles re-scoring; this module only does dense embeddings.
- **Cross-encoder reranking** — separate `rerank` module (port 14200) handles re-scoring; this module only does dense embeddings.
- **Catalog / retrieval consumers** — anything that needs a vector representation hits this single endpoint.
The module is OpenAI-compatible: drop-in for `openai` SDK code that uses `client.embeddings.create()`.
@ -68,7 +68,6 @@ embeddings/
├── README.md quick-start + config table
├── API.md full HTTP API reference + SDK examples
├── pyproject.toml package def, optional extras: vllm, llamacpp, all, dev
├── uv.lock
├── .env.example all EMB_* env vars documented
├── deploy/
│ ├── Dockerfile multi-stage, python 3.11-slim + uv
@ -111,7 +110,7 @@ embeddings/
- **Ingestion path** — when atoms/claims are written, brain calls `EmbeddingClient.embed(text)` to get a 1024-dim vector and stores it in pgvector alongside the row. One round-trip per batch.
- **Query path** — for `/v1/gather`, the incoming claim text is embedded the same way, then the resulting vector is used as the probe in a pgvector `<=> ` (cosine) kNN search to retrieve candidate atoms.
- **Reranking** — top-k candidates from the dense search are forwarded to the separate `rerank` module (cross-encoder, port 54200) for fine-grained scoring. That module does not call this one — they're parallel concerns.
- **Reranking** — top-k candidates from the dense search are forwarded to the separate `rerank` module (cross-encoder, port 14200) for fine-grained scoring. That module does not call this one — they're parallel concerns.
- **Catalog API** — also retrieves via the same embedding pipeline (text → vector → kNN), reusing this single endpoint.
Because the wrapper is OpenAI-compatible, `EmbeddingClient` can be a vanilla `openai.OpenAI(base_url=..., api_key=...)` instance — no DIDI-specific client code needed in brain.
@ -138,16 +137,9 @@ All env vars use the `EMB_` prefix. Required vars have no defaults — the app r
| `EMB_PORT` | `14100` (Prod) / `54100` (Dev) | API server port |
| `EMB_HOST` | `0.0.0.0` | API bind address |
| `EMB_API_TOKENS` | unset | Comma-separated bearer tokens; auth disabled if unset |
| `EMB_VLLM_BASE_URL` | `http://localhost:54101` | Where vLLM listens |
| `EMB_VLLM_MODEL` | `BAAI/bge-m3` | HF model id loaded by vLLM |
| `EMB_VLLM_GPU` | `0` | `CUDA_VISIBLE_DEVICES` for vLLM |
| `EMB_VLLM_GPU_UTIL` | `0.50` | vLLM `--gpu-memory-utilization` |
| `EMB_VLLM_MAX_LEN` | `8192` | vLLM `--max-model-len` |
| `EMB_LLAMACPP_BASE_URL` | `http://localhost:54110` | Where llama.cpp listens |
| `EMB_LLAMACPP_MODEL` | `bge-m3-q4_k_m.gguf` | GGUF filename inside `MODELS_DIR` |
| `EMB_LLAMACPP_CTX` | `8192` | llama.cpp context size |
| `EMB_LLAMACPP_THREADS` | `4` | llama.cpp threads |
| `EMB_LLAMACPP_PARALLEL` | `4` | llama.cpp parallel slots |
| `EMB_VLLM_BASE_URL` | `http://localhost:54101` | Where the API wrapper reaches vLLM |
| `EMB_VLLM_API_KEY` | unset | Optional API key for the vLLM server |
| `EMB_LLAMACPP_BASE_URL` | `http://localhost:54110` | Where the API wrapper reaches llama.cpp |
| `EMB_REQUEST_TIMEOUT` | `120.0` | Per-request backend timeout (s) |
| `EMB_CONNECT_TIMEOUT` | `10.0` | TCP connect timeout (s) |
| `EMB_RATE_LIMIT_RPS` | `20.0` | Token-bucket rate (req/s) |
@ -155,7 +147,24 @@ All env vars use the `EMB_` prefix. Required vars have no defaults — the app r
| `EMB_MAX_CONCURRENT_REQUESTS` | `20` | In-flight cap |
| `EMB_LOG_LEVEL` | `INFO` | DEBUG / INFO / WARNING / ERROR |
| `EMB_LOG_JSON` | `false` | Emit JSON-formatted log lines |
| `HF_CACHE_DIR` | `/cai2_ds_storage/hf_cache` | Mounted into vLLM container |
> The API app reads only the variables above (fields on `EmbeddingSettings` in `config.py`). `EMB_VLLM_BASE_URL` / `EMB_LLAMACPP_BASE_URL` just tell the wrapper where to reach the backend servers.
### Backend service variables (docker-compose only)
These are **not** fields on `EmbeddingSettings` — they are consumed by `deploy/docker-compose.yml` to launch the vLLM / llama.cpp containers (model id, GPU pinning, memory, context). They configure the backend server, not the API app.
| Variable | Default | Description |
|----------|---------|-------------|
| `EMB_VLLM_MODEL` | `BAAI/bge-m3` | HF model id loaded by the vLLM container |
| `EMB_VLLM_GPU` | `0` | `CUDA_VISIBLE_DEVICES` for the vLLM container |
| `EMB_VLLM_GPU_UTIL` | `0.50` | vLLM `--gpu-memory-utilization` |
| `EMB_VLLM_MAX_LEN` | `8192` | vLLM `--max-model-len` |
| `EMB_LLAMACPP_MODEL` | `bge-m3-q4_k_m.gguf` | GGUF filename inside `MODELS_DIR` |
| `EMB_LLAMACPP_CTX` | `8192` | llama.cpp context size |
| `EMB_LLAMACPP_THREADS` | `4` | llama.cpp threads |
| `EMB_LLAMACPP_PARALLEL` | `4` | llama.cpp parallel slots |
| `HF_CACHE_DIR` | `/cai2_ds_storage/hf_cache` | Mounted into the vLLM container |
| `HF_TOKEN` | unset | Forwarded as `HUGGING_FACE_HUB_TOKEN` |
| `MODELS_DIR` | `/cai2_ds_storage/models` | GGUF model directory for llama.cpp |
@ -208,7 +217,7 @@ Compose profiles:
## Related
- **`didi-brain`** consumes via `shared/embedding_client.py` for both ingestion (text → pgvector storage) and `/v1/gather` query embedding.
- **`rerank` module** (separate, port 54200) handles cross-encoder scoring on top of dense kNN candidates from this module — they're complementary, not chained inside this service.
- **`rerank` module** (separate, port 14200) handles cross-encoder scoring on top of dense kNN candidates from this module — they're complementary, not chained inside this service.
- **Catalog API** uses the same endpoint for retrieval-side embeddings.
- **pgvector** in the brain Postgres stores the resulting 1024-dim vectors (cosine distance index).
- **Local Python use**: `from embeddings import EmbeddingClient` (works without the HTTP wrapper if you want in-process inference and have the `vllm`/`llamacpp` extras installed).

10
ai_platform/modules/embeddings/deploy/docker-compose.yml
View file

@ -14,10 +14,10 @@
# Naming Convention: didiAI-{module}-{service}
#
# Network:
# Uses deploy_default network (shared with other modules)
# Uses didi-network (shared with all DIDI + AI platform stacks)
networks:
deploy_default:
didi-network:
external: true
services:
@ -33,7 +33,7 @@ services:
ports:
- "${EMB_PORT:-14100}:${EMB_PORT:-14100}"
networks:
- deploy_default
- didi-network
environment:
- EMB_PORT=${EMB_PORT:-14100}
- EMB_EXTERNAL_URL=${EMB_EXTERNAL_URL}
@ -66,7 +66,7 @@ services:
ports:
- "${EMB_VLLM_PORT:-14101}:14101"
networks:
- deploy_default
- didi-network
volumes:
- ${HF_CACHE_DIR:-/cai2_ds_storage/hf_cache}:/root/.cache/huggingface
environment:
@ -109,7 +109,7 @@ services:
ports:
- "${EMB_LLAMACPP_PORT:-14110}:8080"
networks:
- deploy_default
- didi-network
volumes:
- ${MODELS_DIR:-/cai2_ds_storage/models}:/models:ro
command: >

94
ai_platform/modules/extractors/API.md
View file

@ -114,6 +114,98 @@ Each extractor returns a uniform `FeatureResult`:
| `confidence` | number\|null | 0..1 (extraction confidence, not a verdict) |
| `error` | string\|null | Set when `ok` is false |
### POST /v1/sentiment
Classify the sentiment of a text via the LLM gateway (Romanian-aware prompt,
detects irony/sarcasm). **Delegated to the LLM gateway** — returns `503` if no
gateway is configured.
**Request** — `application/json`
| Field | Type | Description |
|-------|------|-------------|
| `text` | string | Text to analyze (min length 1) |
```bash
curl -X POST http://localhost:54400/v1/sentiment \
-H "Content-Type: application/json" \
-d '{"text": "Ce zi frumoasă!"}'
```
**Response** `200 OK` — a `FeatureResult` (same schema as above), e.g.:
```json
{
"tool_id": "sentiment",
"name": "Sentiment",
"ok": true,
"results": { "label": "positive", "score": 0.92 },
"evidence": ["Sentiment: positive (0.92)."],
"anomalies": [],
"confidence": 0.92
}
```
### POST /v1/ner
Extract named entities via GLiNER multilingual (`urchade/gliner_multi-v2.1`).
**Requires the optional `ml` extra** — returns `503` if the model is unavailable.
**Request** — `application/json`
| Field | Type | Description |
|-------|------|-------------|
| `text` | string | Text to analyze (min length 1) |
| `labels` | string[] \| null | Entity types to extract (defaults to the RO set) |
```bash
curl -X POST http://localhost:54400/v1/ner \
-H "Content-Type: application/json" \
-d '{"text": "Klaus Iohannis și Guvernul României."}'
```
**Response** `200 OK` — a `FeatureResult`; `results` carries the extracted
entities (text, label, span, score).
### POST /v1/ocr
Extract visible text verbatim from an image via the LLM vision model.
**Delegated to the LLM gateway** — returns `503` if no gateway is configured.
**Request** — `multipart/form-data`
| Field | Type | Description |
|-------|------|-------------|
| `file` | file | Image file |
```bash
curl -X POST http://localhost:54400/v1/ocr \
-F "file=@scan.png"
```
**Response** `200 OK` — a `FeatureResult`; `results` carries the recognized text.
### POST /v1/detect
Detect objects (COCO classes) via YOLO (`yolov8n`), returning boxes + confidence.
**Requires the optional `ml` extra** — returns `503` if the model is unavailable.
**Request** — `multipart/form-data`
| Field | Type | Description |
|-------|------|-------------|
| `file` | file | Image file |
Optional query param: `threshold` (float, 0..1) — confidence threshold.
```bash
curl -X POST "http://localhost:54400/v1/detect?threshold=0.4" \
-F "file=@street.jpg"
```
**Response** `200 OK` — a `FeatureResult`; `results` carries detected objects
(label, box, confidence).
## Error Responses
| Status | Meaning |
@ -121,6 +213,8 @@ Each extractor returns a uniform `FeatureResult`:
| 400 | Empty file |
| 413 | File exceeds `EXTRACTORS_MAX_UPLOAD_MB` |
| 422 | Missing `file` field |
| 502 | `LLMError` — the LLM gateway returned an error (`/v1/sentiment`, `/v1/ocr`) |
| 503 | `LLMNotConfigured` (no gateway for `/v1/sentiment`, `/v1/ocr`) or the `ml` extra/model is unavailable (`/v1/ner`, `/v1/detect`) |
```json
{ "detail": "empty file" }

4
ai_platform/modules/forensic_features/.env.example
View file

@ -1,7 +1,7 @@
# Copy to .env and adjust. Read automatically by docker compose.
# Port pe care expune API-ul pe host
API_PORT=8080
# Port pe care expune API-ul pe host (containerul ascultă intern pe 8080)
API_PORT=8085
# Opțional: activează HF AI detector în m27 (Organika/sdxl-detector).
# Default 0 — testat empiric, regresie pe video out-of-distribution.

10
ai_platform/modules/forensic_features/README.md
View file

@ -56,12 +56,12 @@ Output: text formatat + 12-15 imagini PNG base64 + scoruri raw structurate.
# Build și pornește container
docker compose up -d
# Verifică
curl http://localhost:8080/health
curl http://localhost:8080/api/forensic-modules
# Verifică (8085 = portul host; containerul expune intern 8080)
curl http://localhost:8085/health
curl http://localhost:8085/api/forensic-modules
# Test end-to-end
curl -X POST http://localhost:8080/api/forensic-evidence \
curl -X POST http://localhost:8085/api/forensic-evidence \
-F "video=@test_video.mp4" \
-F "encode_images=1"
```
@ -99,7 +99,7 @@ forensic_features/
├── .env.example
├── requirements.txt # Lean: numpy, scipy, opencv, mediapipe, aiohttp
│
├── api.py # REST API (270 linii, doar 5 endpoints)
├── api.py # REST API (418 linii, 6 endpoints — inclusiv /metrics)
├── preprocessing.py # Frame extraction prin ffmpeg
├── face_landmarker.task # MediaPipe model (~3.6 MB)
│

Some files were not shown because too many files have changed in this diff Show more