clossers/didi-lot1-ai
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

LOT 1 - Optimizare script build -Instalare mono comanda

Browse source
This commit is contained in:
Dezvoltari Evotech 2026-06-27 06:42:02 -07:00
parent 5380c3fc63
commit 42ff22bf85
127 changed files with 16163 additions and 532 deletions
Download patch file Download diff file Expand all files Collapse all files

56
ai_platform/modules/audio/API.md
View file

@ -8,9 +8,8 @@ OpenAI-compatible speech-to-text transcription API using faster-whisper.
{BASE_URL}
```
- **Local development:** `http://localhost:8200`
- **Docker (internal):** `http://audio-api:8200`
- **Direct:** `http://localhost:54300`
- **Local development:** `http://localhost:54300`
- **Docker (internal):** `http://audio-api:54300`
- **Production:** Use your configured hostname
## Authentication
@ -34,7 +33,7 @@ Check API health status.
**Example:**
```bash
curl http://localhost:8200/health
curl http://localhost:54300/health
```
---
@ -62,7 +61,20 @@ List available Whisper models (OpenAI-compatible).
**Example:**
```bash
curl http://localhost:8200/v1/models
curl http://localhost:54300/v1/models
```
---
### Service Info
Return service catalog metadata (resources, models, functions). Consumed by the DIDI `catalog-api`.
**Endpoint:** `GET /v1/info`
**Example:**
```bash
curl http://localhost:54300/v1/info
```
---
@ -79,13 +91,16 @@ Transcribe audio file to text (OpenAI-compatible endpoint).
| Parameter | Type | Required | Default | Description |
|-----------|------|----------|---------|-------------|
| `file` | file | Yes | - | Audio file to transcribe (MP3, WAV, M4A, etc.) |
| `file` | file | Yes* | - | Audio file to transcribe (MP3, WAV, M4A, etc.). Provide either `file` or `url` (XOR). |
| `url` | string | Yes* | - | URL to download the audio from (DIDI extension over OpenAI). Provide either `file` or `url` (XOR). |
| `model` | string | No | `large-v3-turbo` | Model to use (currently ignored, uses configured model) |
| `language` | string | No | `null` | Language code (ISO-639-1). Auto-detected if not specified. |
| `prompt` | string | No | `null` | Optional text to guide the model's style |
| `response_format` | string | No | `json` | Format: `json`, `text`, or `verbose_json` |
| `temperature` | float | No | `0.0` | Sampling temperature (0.0-1.0). Use 0.0 for deterministic output. |
\* `file` and `url` are mutually exclusive — supply exactly one.
**Supported Languages (ISO-639-1 codes):**
`en`, `es`, `fr`, `de`, `it`, `pt`, `nl`, `pl`, `tr`, `ru`, `ja`, `ko`, `zh`, `ar`, `hi`, and 90+ more languages.
@ -132,14 +147,21 @@ Full transcription text
**Basic transcription (JSON):**
```bash
curl -X POST "http://localhost:8200/v1/audio/transcriptions" \
curl -X POST "http://localhost:54300/v1/audio/transcriptions" \
-F "file=@audio.mp3" \
-F "response_format=json"
```
**From a URL (instead of file upload):**
```bash
curl -X POST "http://localhost:54300/v1/audio/transcriptions" \
-F "url=https://example.com/audio.mp3" \
-F "response_format=json"
```
**With language specification:**
```bash
curl -X POST "http://localhost:8200/v1/audio/transcriptions" \
curl -X POST "http://localhost:54300/v1/audio/transcriptions" \
-F "file=@audio.mp3" \
-F "language=en" \
-F "response_format=json"
@ -147,21 +169,21 @@ curl -X POST "http://localhost:8200/v1/audio/transcriptions" \
**Text format:**
```bash
curl -X POST "http://localhost:8200/v1/audio/transcriptions" \
curl -X POST "http://localhost:54300/v1/audio/transcriptions" \
-F "file=@audio.mp3" \
-F "response_format=text"
```
**Verbose JSON with segments:**
```bash
curl -X POST "http://localhost:8200/v1/audio/transcriptions" \
curl -X POST "http://localhost:54300/v1/audio/transcriptions" \
-F "file=@audio.mp3" \
-F "response_format=verbose_json"
```
**With initial prompt (to guide style):**
```bash
curl -X POST "http://localhost:8200/v1/audio/transcriptions" \
curl -X POST "http://localhost:54300/v1/audio/transcriptions" \
-F "file=@audio.mp3" \
-F "prompt=This is a technical discussion about machine learning." \
-F "response_format=json"
@ -171,7 +193,7 @@ curl -X POST "http://localhost:8200/v1/audio/transcriptions" \
```python
import requests
url = "http://localhost:8200/v1/audio/transcriptions"
url = "http://localhost:54300/v1/audio/transcriptions"
with open("audio.mp3", "rb") as f:
files = {"file": f}
@ -196,7 +218,7 @@ formData.append('file', audioFile);
formData.append('model', 'large-v3-turbo');
formData.append('response_format', 'json');
const response = await fetch('http://localhost:8200/v1/audio/transcriptions', {
const response = await fetch('http://localhost:54300/v1/audio/transcriptions', {
method: 'POST',
body: formData
});
@ -247,7 +269,7 @@ limit_req_zone $binary_remote_addr zone=audio_limit:10m rate=10r/m;
location / {
limit_req zone=audio_limit burst=5;
proxy_pass http://audio-api:8200;
proxy_pass http://audio-api:54300;
}
```
@ -330,7 +352,7 @@ from openai import OpenAI
# Point to local API
client = OpenAI(
api_key="not-needed", # No auth required
base_url="http://localhost:8200/v1"
base_url="http://localhost:54300/v1"
)
with open("audio.mp3", "rb") as f:
@ -351,7 +373,7 @@ import fs from 'fs';
const openai = new OpenAI({
apiKey: 'not-needed',
baseURL: 'http://localhost:8200/v1'
baseURL: 'http://localhost:54300/v1'
});
const transcription = await openai.audio.transcriptions.create({
@ -391,7 +413,7 @@ Configurable via `AUDIO_BEAM_SIZE` (default: 5). Higher values = better accuracy
```bash
# Check if API is ready
curl http://localhost:8200/health
curl http://localhost:54300/health
# Expected response
{"status": "ok"}

8
ai_platform/modules/audio/INDEX.md
View file

@ -8,7 +8,7 @@ Audio transcription service for DIDI media analysis. Whisper-based (M17-Whisper,
- Default model: `large-v3-turbo` (809M params, ~6GB VRAM int8)
- GPU: CUDA (shared GPU 0 with Qwen3.5-35B-A3B)
- URL (Dev): `http://10.11.10.17:54300/v1/audio/transcriptions`
- Container: `didiAI-audio-api` (GPU host)
- Container: `didiAI-audio` (GPU host)
- Auth: none on the service itself; agent-v3 uses bearer token via `M17_WHISPER_TOKEN` (enforced by gateway/nginx if configured)
## Ce face
@ -92,8 +92,8 @@ cp ../.env.example .env # edit values
docker compose restart audio-api # quick restart
```
- Container name: `didiAI-audio-api`
- Image: `didiai-audio-api`
- Container name: `didiAI-audio`
- Image: `didiai-audio:audit`
- Network: `didi-network` (external, shared with other AI modules)
- GPU reservation: NVIDIA driver, device `0`
- Healthcheck: HTTP `GET /health` every 30s, 60s start period (model load)
@ -111,7 +111,7 @@ docker compose restart audio-api # quick restart
modules/audio/
├── deploy/
│ ├── deploy.sh # CLI wrapper
│ ├── docker-compose.yml # didiAI-audio-api service
│ ├── docker-compose.yml # didiAI-audio service
│ ├── Dockerfile # CUDA + faster-whisper image
│ └── .env # runtime config
├── src/audio/

11
ai_platform/modules/audio/README.md
View file

@ -61,22 +61,23 @@ cp ../.env.example .env
| `/health` | GET | Health check |
| `/v1/models` | GET | List available models |
| `/v1/audio/transcriptions` | POST | Transcribe audio (OpenAI-compatible) |
| `/v1/info` | GET | Service catalog metadata (used by catalog-api) |
### Example API Request
```bash
# Health check
curl http://localhost:8200/health
curl http://localhost:54300/health
# Transcribe audio file
AUDIO="/path/to/audio.mp3"
curl -X POST "http://localhost:8200/v1/audio/transcriptions" \
curl -X POST "http://localhost:54300/v1/audio/transcriptions" \
-F "file=@${AUDIO}" \
-F "model=large-v3-turbo" \
-F "response_format=json"
# With language specification and verbose output
curl -X POST "http://localhost:8200/v1/audio/transcriptions" \
curl -X POST "http://localhost:54300/v1/audio/transcriptions" \
-F "file=@${AUDIO}" \
-F "language=en" \
-F "response_format=verbose_json"
@ -193,8 +194,8 @@ docker compose logs -f audio-api
| Port | Service |
|------|---------|
| `8200` | Audio API |
| `54300` | Audio API (Dev + AI + Audio) |
| `54300` | Audio API (Dev + AI + Audio) — primary |
| `8200` | Audio API (legacy default) |
## Development

10
ai_platform/modules/audio/deploy/docker-compose.yml
View file

@ -19,10 +19,10 @@
# Naming Convention: didiAI-{module}-{service}
#
# Network:
# Uses deploy_default network (shared with other modules)
# Uses the external didi-network bridge (shared with other AI modules)
networks:
deploy_default:
didi-network:
external: true
services:
@ -30,8 +30,8 @@ services:
# Audio Transcription API Server
# ==========================================================================
audio-api:
container_name: didiAI-audio-api
image: didiai-audio-api
container_name: didiAI-audio
image: didiai-audio:audit
build:
context: ..
dockerfile: deploy/Dockerfile
@ -39,7 +39,7 @@ services:
ports:
- "54300:54300"
networks:
- deploy_default
- didi-network
environment:
# GPU configuration

4
ai_platform/modules/catalog-api/INDEX.md
View file

@ -111,7 +111,7 @@ All settings come from environment variables with prefix `CATALOG_` (see `src/ca
| `CATALOG_LOG_LEVEL` | `INFO` | Python logging level. |
| `CATALOG_EXTERNAL_URL` | **required** | Public base URL (e.g., `http://10.11.10.42`) injected into the merged OpenAPI `servers:`. `deploy.sh` aborts if missing. |
| `CATALOG_LLM_URL` | `http://didiAI-llm-api:14011` | LLM Inference internal URL. |
| `CATALOG_AUDIO_URL` | `http://didiAI-audio-api:54300` | Audio API internal URL. |
| `CATALOG_AUDIO_URL` | `http://didiAI-audio:54300` | Audio API internal URL. |
| `CATALOG_VIDEO_URL` | *(empty)* | Video Analysis internal URL — empty string disables Video. |
| `CATALOG_WEB_URL` | `http://didiAI-web-api:51100` | Web API internal URL. |
| `CATALOG_LLM_EXTERNAL_PORT` | `14011` | External port advertised in the merged OpenAPI for LLM. |
@ -136,7 +136,7 @@ All settings come from environment variables with prefix `CATALOG_` (see `src/ca
This service stands on top of the rest of the `ai_platform/modules/*` family — they are its data sources:
- `llm-inference` (port `14011`, container `didiAI-llm-api`) — chat, embeddings, rerank.
- `audio` (port `54300`, container `didiAI-audio-api`) — transcription / TTS.
- `audio` (port `54300`, container `didiAI-audio`) — transcription / TTS.
- `video-analysis` (port `54600`, container `didiAI-video-api`) — vision pipelines (currently disabled by default in the .env example).
- `web` (port `51100`, container `didiAI-web-api`) — fact-check / web crawler.
- `dashboard` — primary frontend consumer of the merged OpenAPI / `/v1/status`.

9
ai_platform/modules/catalog-api/README.md
View file

@ -15,7 +15,7 @@ This module provides a unified API to discover and query all available ML servic
**Required:**
- All global prerequisites (see main [README.md](../../README.md))
- Docker network `deploy_default` (shared with other modules)
- Docker network `didi-network` (shared with other modules)
- At least one other module running (llm-inference, audio, video-analysis, or web)
## Quick Start
@ -60,12 +60,13 @@ Configure via environment variables (prefix: `CATALOG_`):
| Variable | Default | Description |
|----------|---------|-------------|
| `CATALOG_EXTERNAL_URL` | _(required, no default)_ | External base URL for the aggregated OpenAPI spec (e.g. `http://10.11.10.42`). Service fails to start if unset. |
| `CATALOG_HOST` | `0.0.0.0` | Server host |
| `CATALOG_PORT` | `11000` | Server port (Production API Gateway: 11000) |
| `CATALOG_LOG_LEVEL` | `INFO` | Log level |
| `CATALOG_LLM_URL` | `http://didiAI-llm-api:14011` | LLM Inference API URL |
| `CATALOG_AUDIO_URL` | `http://didiAI-audio-api:54300` | Audio API URL |
| `CATALOG_VIDEO_URL` | `http://didiAI-video-api:54600` | Video Analysis API URL |
| `CATALOG_AUDIO_URL` | `http://didiAI-audio:54300` | Audio API URL |
| `CATALOG_VIDEO_URL` | _(empty)_ | Video Analysis API URL — empty by default, so video is disabled unless set |
| `CATALOG_WEB_URL` | `http://didiAI-web-api:51100` | Web API URL |
| `CATALOG_COMPONENT_TIMEOUT` | `10` | Component request timeout (seconds) |
@ -242,7 +243,7 @@ curl http://localhost:11000/v1/components | jq
This module aggregates information from:
- `llm-inference` (port 14011, Docker internal: didiAI-llm-api)
- `audio` (port 54300, Docker internal: didiAI-audio-api)
- `audio` (port 54300, Docker internal: didiAI-audio)
- `video-analysis` (port 54600, Docker internal: didiAI-video-api)
- `web` (port 51100, Docker internal: didiAI-web-api)

4
ai_platform/modules/catalog-api/deploy/docker-compose.yml
View file

@ -4,7 +4,7 @@
# 11000 - Catalog API (main orchestrator gateway)
#
# Network:
# Uses deploy_default network (shared with other modules)
# Uses didi-network network (shared with other modules)
#
# Naming Convention: didiAI-{module}-{service}
@ -38,7 +38,7 @@ services:
# Component URLs (Docker internal network)
- CATALOG_LLM_URL=${CATALOG_LLM_URL:-http://didiAI-llm-api:14011}
- CATALOG_AUDIO_URL=${CATALOG_AUDIO_URL:-http://didiAI-audio-api:54300}
- CATALOG_AUDIO_URL=${CATALOG_AUDIO_URL:-http://didiAI-audio:54300}
- CATALOG_VIDEO_URL=${CATALOG_VIDEO_URL}
- CATALOG_WEB_URL=${CATALOG_WEB_URL:-http://didiAI-web-api:51100}

4
ai_platform/modules/cloak/INDEX.md
View file

@ -49,8 +49,8 @@ end-to-end.
## Where it's consumed
- `ai_platform/modules/web/src/web/search/cloak.py` (CloakHTTPClient) — pending
- `ai_platform/modules/web/src/web/orchestrator.py::_run_search_stage` — pending tier-3 hook
- `ai_platform/modules/web/src/web/search/cloak.py` (CloakHTTPClient) — implemented
- `ai_platform/modules/web/src/web/orchestrator.py::_run_search_stage` — implemented (tier-3 fallback hook, orchestrator.py:564-599)
Backend services (`agent-v3`, `didi-framework`, admin-dashboard) do NOT call
this service directly.

68
ai_platform/modules/dashboard/INDEX.md
View file

@ -10,7 +10,7 @@ AI platform monitoring/admin dashboard. Browse archived claims, view ingest hist
- Python 3.10+ (FastAPI 0.115+, Uvicorn)
- React 19 + MUI 7 + Vite 7 + react-router 7 + TanStack Query + Recharts (built into `/app/web_dist/`, served as static via SPAStaticFiles with index-fallback)
- Jinja2 templates legacy (HTMX + Alpine.js + Tailwind CDN — kept until React reaches 100% parity)
- SQLAlchemy 2.0 async + asyncpg, Alembic
- SQLAlchemy 2.0 async + asyncpg (tables created via `Base.metadata.create_all` at startup; Alembic is only a dependency, not used at runtime — no migrations applied)
- pydantic-settings (env prefix `DASHBOARD_`)
- httpx for live provider quota fetching, brain proxy
- python-jose[cryptography] for Keycloak JWT validation
@ -23,23 +23,28 @@ AI platform monitoring/admin dashboard. Browse archived claims, view ingest hist
- Compose: `deploy/docker-compose.yml`, profile `dashboard`
- Network: `didi-network` (unified single network for all DIDI + AI platform stacks since 2026-05-04)
## Auth (current state — 2026-05-04)
## Auth (current state)
**Hybrid auth** in `dependencies.py:verify_bearer_token`:
1. `STAGING_MODE=true` → all auth bypassed (default for dev)
2. JWT (3 dot-separated parts) → validated via `keycloak_auth.py` (JWKS cache 10min, signature, issuer, exp, role check)
**Shipped in STAGING MODE — Keycloak is supported but disabled in this delivery.**
With `DASHBOARD_STAGING_MODE=true` (the delivered default) all auth is bypassed
and Keycloak is **not** active. JWT validation only engages when
`DASHBOARD_KEYCLOAK_URL` is set (empty value = JWT off).
**Hybrid auth** in `dependencies.py:verify_bearer_token` (in order):
1. `STAGING_MODE=true` → all auth bypassed (this is the delivered state)
2. JWT (3 dot-separated parts) → validated via `keycloak_auth.py` (JWKS cache 10min, signature, issuer, exp, role check) — only when `DASHBOARD_KEYCLOAK_URL` is set
3. DB-backed bearer tokens (legacy, from CLI `dashboard create-user`)
4. Static `api_tokens` env var (legacy fallback)
5. Otherwise → 401
**Keycloak settings** (`config.py:DashboardSettings`):
- `keycloak_url`, `keycloak_realm` (default `didi-clients`), `keycloak_client_id` (default `ai-platform-dashboard`)
**Keycloak settings** (`config.py:DashboardSettings`) — used only once Keycloak is enabled:
- `keycloak_url` (empty by default = disabled), `keycloak_realm` (default `didi-clients`), `keycloak_client_id` (default `ai-platform-dashboard`)
- `keycloak_required_role` (default `admin` — same role as DIDI admin-dashboard for unified access)
- Manual setup: create client + assign role via Keycloak admin or `deploy/setup-keycloak.sh`
**Cutover from staging to prod**:
- Set `DASHBOARD_KEYCLOAK_URL=https://sso.clossers.com`, `DASHBOARD_STAGING_MODE=false`, `VITE_STAGING_MODE=false`
- Rebuild image with build args (Dockerfile bakes Keycloak config into JS bundle at build time)
**To enable Keycloak (leave staging mode)** — example values:
- Set `DASHBOARD_KEYCLOAK_URL=https://sso.clossers.com` (example), `DASHBOARD_STAGING_MODE=false`, `VITE_STAGING_MODE=false`
- Rebuild image with build args (Dockerfile bakes Keycloak config into the JS bundle at build time)
Plan/runbook: `AI_PLATFORM_RESKIN_PLAN.md` (Phase C.8) + `/home/admin365/didi_mono/UNIFIED_KEYCLOAK_CUTOVER.md`
@ -57,6 +62,9 @@ Sections served as HTML pages (`pages.py`) + JSON-mirror endpoints in `routes/`:
## API endpoints
> All JSON routers below (except `health`) are **dual-mounted** under both `/api/*`
> and `/admin-ai/api/*` (see `app.py`: `for prefix in ("/api", "/admin-ai/api")`).
### `routes/health.py`
- `GET /health` - liveness + DB ping
- `GET /ready` - app.state populated check
@ -86,7 +94,29 @@ Sections served as HTML pages (`pages.py`) + JSON-mirror endpoints in `routes/`:
`KNOWN_KEYS` in `routes/config.py` enumerates the runtime keys consumed by `web-api`: `web.providers.{serpapi,tavily,brave,linkup,exa}.enabled`, `web.premium.strategy`, `web.premium.priority_order`, `web.openrouter.model`, `web.tier.{free,premium}.max_search_results`.
### `routes/pages.py` (Jinja HTML)
### `routes/catalog.py` (auth — Model & Extractor catalog, DB-backed CRUD)
- `GET /api/catalog` - list catalog entries (models + extractors)
- `POST /api/catalog` - create a catalog entry
- `PUT /api/catalog/{entry_id}` - update an entry
- `DELETE /api/catalog/{entry_id}` - delete an entry
### `routes/monitoring.py` (auth — AI monitoring)
- `GET /api/monitoring/services` - per-service status
- `GET /api/monitoring/queues` - queue depths/metrics
- `GET /api/monitoring/latency` - latency metrics
### `routes/proxy.py` (auth — module health proxy)
- `GET /api/proxy/{module_id}/health` - proxy to a platform module's health endpoint (URLs with `DASHBOARD_<MODULE>_HEALTH_URL` env override)
### `routes/brain_proxy.py` (brain proxy, whitelisted paths)
- `GET /api/brain/{path}` - read-only pass-through to the brain service
- `POST|PATCH|DELETE /api/brain/{path}` (auth) - mutating pass-through
- Whitelist includes: `/v1/analysis_atom/{list,stats,...}`, `/v1/verification_cache/{list,...}`, `/v1/fact_status/{list,...}`, `/v1/cache/audit_log`, `/v1/cache/invalidate`, `/v1/canonicalize`, `/v1/taxonomy`, `/v1/taxonomy/reload`
### `routes/audit.py` (auth — audit log JSON)
- `GET /api/audit` - audit log entries (config/archive mutations)
### `routes/pages.py` (Jinja HTML — legacy, not the primary UI)
- `GET /` - overview
- `GET /history`, `GET /history/{request_id}`
- `GET /providers`
@ -107,16 +137,22 @@ src/dashboard/
pricing.py # estimate_cost(endpoint, tier, provider) - per-provider USD
retention.py # 30-day rolling cleanup of request_history
api/
app.py # FastAPI factory, lifespan (init engine + provider registry), router wiring
app.py # FastAPI factory, lifespan (init engine + provider registry), router wiring (routes dual-mounted under /api + /admin-ai/api), SPA static mount
dependencies.py # get_session, get_registry, verify_bearer_token, get_username
keycloak_auth.py # Keycloak JWT validation (JWKS cache, signature/issuer/exp/role); only used when DASHBOARD_KEYCLOAK_URL set
routes/
archive.py # claims archive CRUD + promote
audit.py # GET /api/audit (audit log JSON)
brain_proxy.py # /api/brain/* whitelisted proxy to brain service
catalog.py # /api/catalog Model & Extractor catalog CRUD (DB-backed)
config.py # KNOWN_KEYS + runtime override CRUD
health.py # /health, /ready
history.py # /api/history (read-only)
ingest.py # POST /api/ingest/event (service-to-service)
pages.py # all Jinja HTML pages + HTMX handlers
stats.py # /api/stats/{providers,summary,timeline}
monitoring.py # /api/monitoring/{services,queues,latency} (AI monitoring)
pages.py # all Jinja HTML pages + HTMX handlers (legacy)
proxy.py # GET /api/proxy/{module_id}/health (module health proxy)
stats.py # /api/stats/{providers,summary,timeline,cost}
db/
models.py # Base, RequestHistory, ProviderStatsHourly, ConfigOverride, User, AuditLog,
# ClaimsArchive, ArticlesArchive, ClaimArticle
@ -194,7 +230,7 @@ Connection string format: `postgresql+asyncpg://USER:PASS@didiAI-dashboard-db:54
## Reskin plan (FUTURE - NOT done yet)
- React 19 + MUI 7 + Keycloak SSO **DONE 2026-05-02** (Phase C in `agent-v3/IMPLEMENTATION_PLAN_HIL_BRAIN.md`, detail in `AI_PLATFORM_RESKIN_PLAN.md`)
- React 19 + MUI 7 SPA **DONE 2026-05-02**; Keycloak SSO **wired but disabled in this delivery** (staging mode — enable by setting `DASHBOARD_KEYCLOAK_URL`) (Phase C in `agent-v3/IMPLEMENTATION_PLAN_HIL_BRAIN.md`, detail in `AI_PLATFORM_RESKIN_PLAN.md`)
- Becomes admin-only (Keycloak realm role)
- Bearer-token table retired; existing `role` column may persist for historical audit-log mapping
- Brain admin UI (atom browse, force-gold, brain stats) added as a new section in this dashboard during Phase C
@ -214,7 +250,7 @@ cp ../.env.example .env # set DASHBOARD_DB_USER/PASSWORD/NAME + provider keys
## Ce NU face
- ~~No SSO yet~~ Keycloak SSO wired (DONE 2026-05-02). Bearer tokens kept as legacy fallback.
- Keycloak SSO is wired but **disabled in this delivery** (ships in staging mode, auth bypassed; enable via `DASHBOARD_KEYCLOAK_URL`). Bearer tokens kept as legacy fallback.
- No multi-tenant - single shared `users` table, no per-tenant scoping
- No public access - binds to internal `didi-network` network, not exposed via Kong/edge
- ~~No React frontend yet~~ React 19 SPA at `/admin-ai/` (DONE 2026-05-02). Jinja kept side-by-side until parity.

74
ai_platform/modules/dashboard/README.md
View file

@ -1,20 +1,42 @@
# Dashboard
Admin dashboard for the didiAI platform. Tracks search provider usage, costs, request history, and exposes runtime configuration.
Admin dashboard for the didiAI platform. A single FastAPI service that serves a
**React 19 + MUI 7 + Vite single-page app** (the admin UI) plus a JSON API for
AI monitoring, a DB-backed Model & Extractor catalog, runtime config, RBAC and
audit. It also tracks search-provider usage, costs and request history.
## What it does
- **Live quota & billing** — pulls real-time data from SerpAPI, Tavily, Brave, OpenRouter
- **Health monitoring** — SearXNG, web-api, vLLM, llama.cpp servers
- **Request history** — 30-day rolling log of every gather/search/fetch request with drill-down
- **AI monitoring** — live health/status of platform modules (proxy to each
module's health endpoint), provider quotas (SerpAPI, Tavily, Brave, LinkUp,
Exa, OpenRouter), live KPIs and throughput
- **Model & Extractor Catalog** — DB-backed CRUD over registered models and
extractors, exposed under `/api/catalog`
- **Request history** — 30-day rolling log of every gather/search/fetch request
with drill-down
- **Cost tracking** — per-provider spend, projections, cost per tier
- **Future:** runtime config (toggle providers, change strategies, manage tier caps)
- **Runtime config** — config override store (`/api/config`) with schema
validation and audit trail
- **RBAC** — role-based access; in the shipped build the dashboard runs in
**staging mode** with auth bypassed (see below)
- **Brain admin** — fact-status browse/override and cache invalidation via a
brain proxy
## Authentication / staging mode
The build ships in **staging mode** (`DASHBOARD_STAGING_MODE=true`), so all auth
is bypassed and Keycloak is **not** active. Keycloak (JWT) is supported but
disabled in this delivery: JWT validation only turns on when
`DASHBOARD_KEYCLOAK_URL` is set (empty = off). To go to authenticated mode, set
`DASHBOARD_KEYCLOAK_URL` (e.g. an SSO URL), set `DASHBOARD_STAGING_MODE=false`,
and rebuild the image (Keycloak config is baked into the JS bundle at build
time). DB-backed bearer tokens remain as a legacy fallback.
## Prerequisites
- Docker 24+ with Compose V2
- PostgreSQL 16 (provided by compose)
- Internal network access to web-api, SearXNG, LLM servers
- PostgreSQL 16 (provided by compose, `didiAI-dashboard-db` on `:15432`)
- Internal network access to the platform modules and provider APIs
## Quick start
@ -24,37 +46,49 @@ cp ../.env.example .env # edit with your secrets
./deploy.sh up
```
Dashboard is now running at http://localhost:51300
The dashboard API listens on `http://localhost:51300`. The admin SPA is served
at `http://localhost:51300/admin-ai/`.
## Endpoints
### Web UI
- `/` — overview with KPIs and provider grid
- `/providers` — detailed provider cards + raw table
- `/history` — filterable request history
- `/history/{request_id}` — full request detail with stages
### Admin UI (SPA)
### JSON API
- `GET /health` — liveness
- `/admin-ai/` — React 19 + MUI SPA (overview, live status, history, cost,
providers, catalog, config/schema, audit, brain admin). All client-side routes
under `/admin-ai/` are served by the SPA with index fallback.
> Legacy Jinja pages (`/`, `/providers`, `/history`, …) still exist server-side
> but are not the primary UI; the SPA at `/admin-ai/` is the delivered UI.
### JSON API (selected)
- `GET /health` — liveness + DB ping (health JSON at root)
- `GET /api/stats/providers` — live provider stats
- `GET /api/stats/summary?hours=24` — aggregated counters
- `GET /api/stats/timeline?hours=24` — hourly buckets for charts
- `GET /api/stats/cost` — cost + projection
- `GET /api/history?limit=50&tier=premium` — filtered history
- `GET /api/history/{request_id}` — single request with full payload
- `GET|POST|PUT|DELETE /api/catalog/...` — Model & Extractor catalog CRUD
- `GET /api/config`, `PUT|DELETE /api/config/{key}` — runtime config overrides
- `GET /api/proxy/{module_id}/health` — module health proxy (AI monitoring)
- `/api/brain/*` — brain proxy (fact status, cache invalidation)
- `POST /api/ingest/event` — receives events from web-api middleware
## Architecture
```
web-api ────► POST /api/ingest/event ────► dashboard-api ────► PostgreSQL
│
browser ────► GET /admin-ai/ (React SPA) ──┘ │
├─ reads provider APIs live
└─ serves UI via Jinja2+HTMX
├─ proxies module health + brain
└─ serves SPA static (index fallback)
```
## Tech stack
- FastAPI + Pydantic
- SQLAlchemy 2.0 async + asyncpg
- Jinja2 + HTMX + Alpine.js + Tailwind (zero build step)
- FastAPI + Pydantic (pydantic-settings, env prefix `DASHBOARD_`)
- SQLAlchemy 2.0 async + asyncpg (tables created via `create_all` at startup)
- React 19 + MUI 7 + Vite 7 + react-router 7 + TanStack Query + Recharts
(built into `web_dist/`, served as static SPA)
- PostgreSQL 16

12
ai_platform/modules/dashboard/web/src/pages/system/Settings.tsx
View file

@ -76,9 +76,15 @@ export default function Settings() {
});
const health = useQuery({
queryKey: ['settings', 'health'],
// Dashboard liveness probe lives at root /health (next to /ready), not
// /api/health — the latter returns 404 and shows "unknown" in the UI.
queryFn: () => apiGet<HealthResponse>('/health'),
// Dashboard liveness probe lives at the ORIGIN root /health (next to
// /ready). apiGet() would prepend BASE_PATH (/admin-ai/health), which the
// SPA static fallback answers with index.html → JSON parse yields no
// `status` → "unknown". Fetch the absolute root path directly instead.
queryFn: async (): Promise<HealthResponse> => {
const res = await fetch('/health', { headers: { Accept: 'application/json' } });
if (!res.ok) throw new Error(`health ${res.status}`);
return res.json();
},
refetchInterval: 30_000,
});

64
ai_platform/modules/didi_brain/ARCHITECTURE.md
View file

@ -17,7 +17,7 @@ it, what does it depend on" for any file in the repo.
| Module / File | What it does | When it runs | Who calls it |
|---|---|---|---|
| **`infra/docker-compose.yml`** | Defines the 3-container stack | `docker compose up` | operator |
| **`infra/docker-compose.yml`** | Defines the 4-container stack (api, atomic, postgres, scheduler) | `docker compose up` | operator |
| **`brain_api/`** *(container)* | HTTP service speaking Didi's contract | every Didi request | Didi backend (HTTP) |
| **`shared/`** *(library)* | Common code: config, clients, taxonomy | imported everywhere | brain_api, extractor, lint, scripts |
| **`extractor/`** *(library + jobs)* | Turns Document atoms into Claim atoms | operator (script 07) OR background after `/v1/ingest` | scripts/07, brain_api ingest |
@ -26,12 +26,12 @@ it, what does it depend on" for any file in the repo.
| **Atomic server** *(container)* | Storage + chunking + embedding pipeline + REST | always | brain_api (HTTP), scripts (HTTP) |
| **Postgres + pgvector** *(container)* | Persistent atom + vector storage | always | atomic-server (SQL) |
**External services** (not in our repo, on VPN):
**External services** (not in our repo, reached over Docker DNS on `didi-network`):
| Where | What | Used by |
|---|---|---|
| `10.11.10.17:14011` | LLM router → Qwen 397B | brain_api NLI, extractor, lint |
| `10.11.10.15:8200` | BGE-M3 embedding server | atomic-server (for embeddings) |
| `10.11.10.15:8100` | BGE-reranker-v2-m3 | brain_api gather (for precision rerank) |
| `didiAI-llm-api` → `10.11.10.17:14011` | LLM router → qwen3.5 | brain_api NLI, extractor, lint |
| `didiAI-embeddings-api` → `10.11.10.15:14100` | BGE-M3 embedding server | atomic-server (for embeddings) |
| `didiAI-rerank-api` → `10.11.10.15:14200` | BGE-reranker-v2-m3 | brain_api gather (for precision rerank) |
---
@ -56,12 +56,12 @@ it, what does it depend on" for any file in the repo.
│ atomic_api, taxonomy, logging} │
└─────┬─────────────┬──────────────┬──────────────────────────┘
│ │ │
│ HTTP │ HTTPS │ HTTPS
│ HTTP │ HTTP │ HTTP
▼ ▼ ▼
┌───────────┐ ┌──────────┐ ┌────────────────┐
│ atomic- │ │ BGE-M3 │ │ Qwen 397B │
│ atomic- │ │ BGE-M3 │ │ qwen3.5 │
│ server │ │ + rerank │ │ via LLM router │
│ (cont.) │ │ (VPN) │ │ (VPN) │
│ (cont.) │ │ (DNS) │ │ (DNS) │
└─────┬─────┘ └──────────┘ └────────────────┘
│
▼
@ -121,8 +121,9 @@ its own.
## `brain_api/` — the HTTP service
**Purpose.** Containerized FastAPI service that exposes the 5-endpoint
contract Didi's backend already speaks. It does NOT contain business logic
**Purpose.** Containerized FastAPI service that exposes the full route set
(~30 routes: the web-gathering contract + brain-owned cache/freshness layers)
that Didi's backend already speaks. It does NOT contain business logic
that another module reuses; it ties together `shared` + `extractor` (via
`/v1/ingest`) and translates between Didi's HTTP shape and the brain's
internal capabilities.
@ -139,7 +140,7 @@ FastAPI/Uvicorn. The lifespan hook initializes the long-lived clients
| `Dockerfile` | python:3.12-slim, non-root user `brain`, COPY shared/extractor/brain_api, curl healthcheck on /health, CMD `python -m brain_api.run`. |
| `requirements.txt` | Pinned runtime deps. |
| `__init__.py` | Module marker + docstring. |
| `app.py` | The FastAPI app. Defines `lifespan` (startup/shutdown), the `/health` route, and the 5 v1 routes. Each route delegates to a function in `services/`. |
| `app.py` | The FastAPI app. Defines `lifespan` (startup/shutdown), the `/health` route, and the full set of v1 routes (gather/search/fetch/image-search/ingest + verification_cache + analysis_atom* + canonicalize + cache/invalidate + cache/audit_log + fact_status*). Each route delegates to a function in `services/`. |
| `run.py` | Uvicorn entry. Reads `BRAIN_API_HOST` and `BRAIN_API_PORT` from env (defaults 127.0.0.1:8090; container overrides to 0.0.0.0). |
| `deps.py` | `AppState` dataclass + module-level singleton accessor. Set by lifespan, read by route handlers. |
| `schemas.py` | **The contract.** Pydantic v2 models for every request/response shape the service speaks: `SearchRequest/Response`, `FetchRequest/Response`, `GatherRequest/Response`, `ImageSearchRequest/Response`, `IngestRequest/Response`, plus `EvidenceItem`, `BrainEvidenceMeta`, `BrainMeta`, etc. |
@ -159,7 +160,7 @@ FastAPI/Uvicorn. The lifespan hook initializes the long-lived clients
## `extractor/` — Document → Claim transformation
**Purpose.** Reads `Type/Document` atoms from Atomic, asks Qwen 397B to
**Purpose.** Reads `Type/Document` atoms from Atomic, asks qwen3.5 to
extract atomic factual claims, validates each claim against the source
text (substring check on the quote), and creates new `Type/Claim` atoms
with canonical hash-based source URLs and inherited tags.
@ -186,7 +187,7 @@ Both paths share the same idempotency state (`extractor/_extracted.json`).
| `extract.py` | Single-document logic. `extract_claims_from_atom(llm, title, language, content)` → `ExtractionResult` with `valid: list[ExtractedClaim]` and `rejected: dict[reason → count]`. Includes the substring quote validation. |
| `push.py` | `push_claim(atomic, parent_atom, claim, parent_title, resolver)` creates one `Type/Claim` atom. URL is `{parent_url}#claim={hash8}` for natural dedup. Inherits parent's Country/Topic/SourceType/Credibility/Language tags, adds Type/Claim + Stance/{X}. |
| `batch.py` | The orchestrator. `run_batch(limit, only_atom_ids)` paginates Document atoms, fetches each fully (list_atoms returns summary only — gotcha!), runs extraction sequentially, pushes claims, saves state after every doc so Ctrl-C is recoverable. |
| `prompts/claim_extraction_v1.md` | Versioned prompt that instructs Qwen 397B to return strict JSON with claims + verbatim quotes. Output format pinned. |
| `prompts/claim_extraction_v1.md` | Versioned prompt that instructs qwen3.5 to return strict JSON with claims + verbatim quotes. Output format pinned. |
**Imports.** `shared`, `httpx` (transitively).
@ -198,7 +199,7 @@ Both paths share the same idempotency state (`extractor/_extracted.json`).
**Purpose.** Audit job that finds claim atoms in our corpus that
contradict each other (or are paraphrases). For every claim, asks Atomic
for its semantic neighbors, then asks Qwen 397B to classify each
for its semantic neighbors, then asks qwen3.5 to classify each
candidate pair as **EQUIVALENT**, **CONTRADICTORY**, or **INCOMPARABLE**.
Stores all verdicts in a JSON ledger so re-runs only process new pairs.
@ -264,7 +265,7 @@ clean rich-formatted report, and exits with a meaningful code.
| File | Role |
|---|---|
| `docker-compose.yml` | Three services: `postgres` (pgvector/pgvector:pg16), `atomic-server` (ghcr.io/kenforthewin/atomic-server:latest with overridden entrypoint to use `--data-dir` instead of legacy `--db-path`, and `ATOMIC_STORAGE=postgres` env), `brain-api` (built from `brain_api/Dockerfile`). Two named volumes (`didibrain-pg-data`, `didibrain-atomic-data`), one bridge network (`didibrain`). All services have `restart: unless-stopped` and healthchecks. |
| `docker-compose.yml` | Four services: `postgres` (pgvector/pgvector:pg16), `atomic-server` (ghcr.io/kenforthewin/atomic-server:latest with overridden entrypoint to use `--data-dir` instead of legacy `--db-path`, and `ATOMIC_STORAGE=postgres` env), `brain-api` (built from `brain_api/Dockerfile`), and `scheduler` (built from `scheduler/Dockerfile` — feeder + auditor + watcher + heartbeat). Two named volumes (`didibrain-pg-data`, `didibrain-atomic-data`), and the external shared network `didi-network`. All services have `restart: unless-stopped` and healthchecks. |
**When it runs.** `docker compose up` from the operator (or via `bootstrap_deploy.sh`).
@ -301,7 +302,7 @@ brain_api/app.py: post_gather()
│
│ stage 4 nli → mapping.parse_claim_atom_body() per top result
│ → nli.classify_batch(llm, claim, evidence_texts)
│ (parallel calls to Qwen 397B, ~3-4 sec; one per parent doc)
│ (parallel calls to qwen3.5, ~3-4 sec; one per parent doc)
│
│ stage 5 evidence → atomic.get_atom_by_source_url() per parent doc
│ → mapping.evidence_from_parent() builds EvidenceItem
@ -355,7 +356,7 @@ cp .env.example .env, vim .env (none)
│ │
│ ├─ list Type/Document atoms
│ ├─ for each not in extractor/_extracted.json:
│ │ fetch full atom → extract.extract_claims_from_atom() (Qwen 397B)
│ │ fetch full atom → extract.extract_claims_from_atom() (qwen3.5)
│ │ push.push_claim() per valid claim → atomic.create_atom()
│ └─ save state every doc
│
@ -394,7 +395,7 @@ brain_api/app.py: post_ingest()
FastAPI BackgroundTasks: _run_extraction_background(created_ids)
│
└─▶ extractor.batch.run_batch(only_atom_ids=set(created_ids))
├─ runs Qwen 397B claim extraction
├─ runs qwen3.5 claim extraction
├─ pushes Type/Claim atoms
└─ saves extractor/_extracted.json
```
@ -416,9 +417,14 @@ from Atomic without loss.
| `extractor/_extracted.json` | `scripts/07_run_extraction.py` (and brain_api ingest) | Per-document extraction log: which docs have been processed, with which prompt version, how many claims came out. Skip already-done docs on rerun. | yes — delete and re-run; will re-extract everything |
| `lint/_contradictions.json` | `scripts/10_run_lint.py` | Per-pair verdict ledger with the EQUIVALENT / CONTRADICTORY / INCOMPARABLE labels. Skip already-done pairs on rerun. | yes — delete and re-run; will re-classify everything |
The brain_api **container** does NOT use any of these files. It calls
`atomic.list_tags()` at startup to refresh `TagResolver` in-memory, so
the image stays portable.
The brain_api **container** refreshes `TagResolver` in-memory at startup by
calling `atomic.list_tags()`, so it does not bake any of these files into the
image. It does, however, **mount** `shared/_tag_ids.json` read-only via the
compose `volumes:` entry — background extraction (triggered by `/v1/ingest`)
instantiates `TagResolver` directly, bypassing the lifespan refresh, so it
relies on that mounted file to resolve canonical tag UUIDs. The other two
ledgers (`extractor/_extracted.json`, `lint/_contradictions.json`) are
host-only and not used by the container.
---
@ -435,20 +441,20 @@ project root. The brain_api container also receives these via
| `LLM_ROUTER_API_KEY` | empty | optional bearer | no |
| `LLM_VLLM_URL` | `http://localhost:14001` | direct vLLM URL (fallback only, currently unused) | no |
| `LLM_LLAMACPP_URLS` | empty | comma-separated direct llamacpp URLs (fallback) | no |
| `MODEL_REASONING` | `Qwen3.5-397B-A17B` | model id for extraction, NLI, gather | yes |
| `MODEL_REASONING_BACKEND` | `llamacpp` | router backend hint | yes |
| `MODEL_FAST` | `qwen3.5` | fast model id (currently disabled) | no |
| `MODEL_REASONING` | `qwen3.5` | model id for extraction, NLI, gather (live working model) | yes |
| `MODEL_REASONING_BACKEND` | `vllm` | router backend hint | yes |
| `MODEL_FAST` | `qwen3.5` | fast model id (enabled — the live working model) | no |
| `MODEL_FAST_BACKEND` | `vllm` | fast backend hint | no |
| `MODEL_FAST_ENABLED` | `false` | enable fast model in routing | no |
| `MODEL_VISION` | `gemma-3-27b-it` | vision model id (currently down) | no |
| `MODEL_FAST_ENABLED` | `true` | fast model enabled in routing (live: `true`, qwen3.5 active) | no |
| `MODEL_VISION` | empty | vision model id (vision disabled; no vision model deployed) | no |
| `MODEL_VISION_URL` | empty | direct vision endpoint | no |
| `MODEL_VISION_ENABLED` | `false` | enable vision in routing | no |
| `EMBEDDING_URL` | `http://10.11.10.15:8200` | BGE-M3 vLLM endpoint | yes |
| `EMBEDDING_URL` | `http://10.11.10.15:14100` | BGE-M3 vLLM endpoint (`didiAI-embeddings-api`) | yes |
| `EMBEDDING_API_KEY` | empty | optional bearer | no |
| `EMBEDDING_MODEL` | `BAAI/bge-m3` | model id sent in requests | yes |
| `EMBEDDING_DIM` | `1024` | vector dimension (must match Atomic's setting) | yes |
| `EMBEDDING_MAX_TOKENS` | `8192` | input length cap | yes |
| `RERANKER_URL` | `http://10.11.10.15:8100` | BGE-reranker-v2-m3 endpoint | yes |
| `RERANKER_URL` | `http://10.11.10.15:14200` | BGE-reranker-v2-m3 endpoint (`didiAI-rerank-api`) | yes |
| `RERANKER_API_KEY` | empty | optional bearer | no |
| `RERANKER_MODEL` | `BAAI/bge-reranker-v2-m3` | model id sent in requests | yes |
| `ATOMIC_URL` | `http://localhost:8088` | atomic-server URL (host scripts use this; brain_api container overrides to docker DNS) | yes |
@ -504,7 +510,7 @@ Something with /v1/gather failing?
Atomic returning 500?
│
├─ docker logs didibrain-atomic — likely BGE unreachable
└─ curl http://10.11.10.15:8200/v1/models on host — VPN check
└─ curl http://10.11.10.15:14100/v1/models on host — upstream reachability check
Claim extraction acting weird?
│

8
ai_platform/modules/didi_brain/CONTRACT_VERIFICATION_CACHE.md
View file

@ -64,7 +64,7 @@ rulare). Cu evidence_hash în cheie, cache-ul era practic nereachable.
"https://somes-tisa.rowater.ro/pdf2"
],
"tier": "free",
"model": "qwen35:Qwen3.5-397B-A17B",
"model": "qwen35:qwen3.5",
"prompt_hash": "a3f2b1c9d4e7",
"framework_version": "f9e1d2c3b4a5",
"schema_name": "didi-v1",
@ -118,7 +118,7 @@ rulare). Cu evidence_hash în cheie, cache-ul era practic nereachable.
- `503 Service Unavailable` — PG indisponibil, retry mai târziu
- `500 Internal Server Error` — orice alt eșec
**Idempotență:** UPSERT pe cheia `(claim_hash, evidence_hash, tier)`. Same-tuple,
**Idempotență:** UPSERT pe cheia `(claim_hash, tier)`. Same-tuple,
second write → update `verification_processed`/`verification_raw`/`updated_at`/`expires_at`.
Last-writer-wins.
@ -155,7 +155,7 @@ Last-writer-wins.
"verification_staleness": "fresh",
"verification": { ... payload stocat ... },
"verification_model": "qwen35:Qwen3.5-397B-A17B",
"verification_model": "qwen35:qwen3.5",
"verification_tier": "free",
"verification_prompt_hash": "a3f2b1c9d4e7",
"verification_framework_version": "f9e1d2c3b4a5",
@ -263,7 +263,7 @@ Cache pe `free` și `premium` sunt **complet separate**. Un user `free` nu vede
ce a scris un user `premium` (și invers). Este intenționat: modele diferite =
nuanțe diferite la stance.
Cheia unică în DB: `(claim_hash, evidence_hash, tier)`.
Cheia unică în DB: `(claim_hash, tier)`.
---

15
ai_platform/modules/didi_brain/INDEX.md
View file

@ -2,7 +2,7 @@
Knowledge atom storage + claim verification cache + analysis atom cache pentru platforma DIDI. FastAPI service care vorbeste contractul `web-gathering` 1:1 (drop-in replacement / cache layer pentru `/v1/gather`) si in plus expune doua cache-uri proprii: `brain_verification_cache` (claims, din 2026-04-23) si `brain_analysis_atom` (techniques + ai_tampered + claims, NEW v2 din 2026-05-01).
**Stack**: Python 3.11+, FastAPI, Pydantic v2, asyncpg, BGE-M3 1024-dim embeddings (vLLM), BGE-reranker-v2-m3 (vLLM cross-encoder), Qwen3.5-397B-A17B (llama.cpp via OpenAI-compat router)
**Stack**: Python 3.11+, FastAPI, Pydantic v2, asyncpg, BGE-M3 1024-dim embeddings (vLLM), BGE-reranker-v2-m3 (vLLM cross-encoder), qwen3.5 (vLLM via OpenAI-compat router)
**Storage**: Atomic (Rust knowledge graph) on Postgres 16 + pgvector. Brain owns its own tables prefixed `brain_*` alongside Atomic's tables.
**URL**: `http://10.11.10.12:8090` (production-local, since brain cutover 2026-05-01 — anterior pe `10.11.10.13:8090`)
**Container**: `didibrain-api` (alongside `didibrain-atomic` and `didibrain-postgres`)
@ -163,9 +163,10 @@ UNIQUE: `(content_hash, component, prompt_hash)` — `tier` OUT of unique key (w
## Configuration
- Settings via `shared/config.py` cu `pydantic-settings`
- Env vars: `postgres_dsn`, `atomic_url`, `llm_router_url`, `embedding_url`, `reranker_url`, `verification_cache_ttl_days` (default 30), `verification_cache_max_payload_kb` (default 64)
- Internal Atomic URL: `http://didibrain-atomic:8080` (Docker DNS)
- Upstream LLM/embed/rerank pe VPN 10.11.10.x (10.11.10.17:14011 router, 10.11.10.15:8200 BGE-M3, 10.11.10.15:8100 reranker)
- Env vars: `postgres_dsn`, `atomic_url`, `atomic_token`, `llm_router_url`, `embedding_url`, `reranker_url`, `verification_cache_ttl_days` (default 30), `verification_cache_max_payload_kb` (default 64)
- `ATOMIC_TOKEN`: required for atomic-server API calls, auto-generated/auto-populated by `scripts/02_bootstrap_atomic.py` (written back to `.env`)
- Internal Atomic URL: `http://atomic-server:8080` (Docker DNS)
- Upstream LLM/embed/rerank reached over Docker DNS on `didi-network` (`didiAI-llm-api` → 10.11.10.17:14011 router, `didiAI-embeddings-api` → 10.11.10.15:14100 BGE-M3, `didiAI-rerank-api` → 10.11.10.15:14200 reranker)
- **No auth in v1** (binds to internal Docker network)
---
@ -173,10 +174,10 @@ UNIQUE: `(content_hash, component, prompt_hash)` — `tier` OUT of unique key (w
## Deployment
- Docker compose la `infra/docker-compose.yml`
- 3 containere: `didibrain-api` (FastAPI), `didibrain-atomic` (Rust KG), `didibrain-postgres` (PG 16 + pgvector)
- 4 containere: `didibrain-api` (FastAPI), `didibrain-atomic` (Rust KG), `didibrain-postgres` (PG 16 + pgvector), `didibrain-scheduler` (feeder + auditor + watcher)
- Schema bootstrap on startup (idempotent — `_SCHEMA_SQL` ruleaza la fiecare connect, all DDL is `IF NOT EXISTS` + ALTER guards)
- TTL cleanup: cron job in `scripts/` sterge expired silver/bronze atoms
- Resource footprint: ~200 MB RAM total cross 3 containere; brain-api idle ~6% CPU, spike ~20% during `/v1/gather` cu NLI
- Resource footprint: ~200 MB RAM total cross 4 containere; brain-api idle ~6% CPU, spike ~20% during `/v1/gather` cu NLI
**Rebuild**:
```bash
@ -194,7 +195,7 @@ docker compose -f infra/docker-compose.yml --env-file .env up -d --build
- `scripts/04_seed_taxonomy.py` — seed canonical tag taxonomy (79 tags, 7 namespaces)
- `scripts/05_import_wikipedia_seed.py` — seed-list Wikipedia import
- `scripts/06_validate_queries.py` — smoke test doc-level retrieval
- `scripts/07_run_extraction.py` — claim extraction batch (Qwen 397B)
- `scripts/07_run_extraction.py` — claim extraction batch (qwen3.5)
- `scripts/08_validate_claims.py` — smoke test claim-level retrieval
- `scripts/09_brain_api_demo.py` — brain_api contract test (all endpoints)
- `scripts/10_run_lint.py` — Lint pass runner (contradiction detection)

55
ai_platform/modules/didi_brain/README.md
View file

@ -23,7 +23,7 @@ HTTP contract but serves responses from a locally-grown **knowledge graph**:
- **Atoms** (documents and atomic claims) are stored in a Postgres +
pgvector backend with multilingual embeddings (BGE-M3, 1024-dim).
- **Claim extraction** runs Qwen 3.5 397B over every ingested document to
- **Claim extraction** runs Qwen3.5 over every ingested document to
pull out verifiable atomic claims with source quotes and stance.
- **Retrieval** uses vector kNN + BGE-reranker-v2-m3 cross-encoder for
precision.
@ -46,9 +46,9 @@ background. Next time a similar claim arrives, it's a HIT.
| **Brain storage** | [Atomic](https://github.com/kenforthewin/atomic) (Rust) on Postgres 16 + pgvector |
| **Embeddings** | `BAAI/bge-m3` via vLLM OpenAI-compat (1024 dim, 8K ctx, multilingual) |
| **Reranker** | `BAAI/bge-reranker-v2-m3` cross-encoder via vllm-rerank-api |
| **LLM (reasoning)** | `Qwen3.5-397B-A17B` (MoE) via llama.cpp through an OpenAI-compat router |
| **LLM (reasoning)** | `qwen3.5` via vLLM through an OpenAI-compat router (`MODEL_FAST` enabled) |
| **Service** | Python 3.12, FastAPI, Uvicorn, Pydantic v2, httpx, structlog, tenacity |
| **Deployment** | Docker + docker-compose (3 services: api, atomic, postgres) |
| **Deployment** | Docker + docker-compose (4 services: api, atomic, postgres, scheduler) |
## Quick start — local dev
@ -88,7 +88,7 @@ curl -fsS http://localhost:8090/health
```
Open the interactive Swagger UI at **http://localhost:8090/docs** to poke
the 5 endpoints live.
the full route set live (~30 routes).
## Production deploy — fresh Linux server
@ -127,6 +127,21 @@ backends safely ignore.
| `POST /v1/gather` | claim → ranked evidence with NLI stance | 5-6 s (1.5 s without NLI) |
| `POST /v1/image-search` | stub (always empty list) | <5 ms |
| `POST /v1/ingest` | populate brain from web-module output | variable (async extraction) |
| `POST /v1/verification_cache` | write claim verification result (claims cache) | <50 ms |
| `POST /v1/analysis_atom/lookup` | read cached analysis result (techniques / ai_tampered / claims) | <50 ms |
| `POST /v1/analysis_atom` | write analysis atom (silver/bronze by confidence) | <50 ms |
| `PATCH /v1/analysis_atom/{id}` | promote atom to gold (moderator review) | <50 ms |
| `GET /v1/analysis_atom/stats` | per-tier/component counts + 24h hit rate | <20 ms |
| `POST /v1/canonicalize` | temporal claim disambiguation (Pilon 7) | LLM-bound |
| `POST /v1/cache/invalidate` | mass invalidation with `dry_run` (Pilon 8) | variable |
| `GET /v1/cache/audit_log` | paginated audit browser | <50 ms |
| `GET/PATCH /v1/fact_status/*` | versioned fact-status layer (list/detail/versions/override) | <50 ms |
This is the full set served by `brain_api` (~30 routes including the FastAPI
auto docs); the `/v1/gather`, `/v1/search`, `/v1/fetch`, `/v1/ingest`,
`/v1/image-search` group is the web-gathering contract, the rest are the
brain-owned cache + freshness-defense layers. See `INDEX.md` for the
canonical route list.
The response from `/v1/gather` matches the existing web-module shape
exactly plus an additive `brain_meta` object on the top level and inside
@ -156,11 +171,11 @@ Didi backend
│ stage context → language detection │
│ stage retrieval → Atomic semantic search (top 50) │
│ stage rerank → BGE cross-encoder (top 15) │
│ stage nli → Qwen 397B stance vs query │
│ stage nli → Qwen3.5 stance vs query │
│ stage evidence → group by parent doc, shape │
└──────┬──────────────────────────┬──────────────────┘
│ │
│ HTTP (docker DNS) │ HTTPS (VPN)
│ HTTP (docker DNS) │ HTTP (docker DNS)
▼ ▼
┌────────────────┐ ┌───────────────────────┐
│ atomic-server │ │ BGE-M3 embeddings │
@ -169,9 +184,9 @@ Didi backend
│ └───────────────────────┘
▼
┌────────────────┐ ┌───────────────────────┐
│ postgres │ │ Qwen 3.5 397B-A17B │
│ postgres │ │ qwen3.5 │
│ + pgvector │ │ via LLM router │
│ :5432 (5434) │ │ (llama.cpp + vLLM) │
│ :5432 (5434) │ │ (vLLM) │
└────────────────┘ └───────────────────────┘
```
@ -179,7 +194,7 @@ Didi backend
```
didibrain/
├── infra/docker-compose.yml # 3-service stack
├── infra/docker-compose.yml # 4-service stack (api, atomic, postgres, scheduler)
├── brain_api/ # FastAPI service (the main deliverable)
├── shared/ # config, clients, taxonomy (reused everywhere)
├── extractor/ # claim extraction (host jobs + /v1/ingest)
@ -201,7 +216,7 @@ Detailed file-by-file rundown is in `STATUS.md`.
- [x] Claim extraction (513 atoms, 1.2% hallucination filter)
- [x] Document-level retrieval validated (cross-lingual cosine 0.88-0.94)
- [x] Claim-level retrieval validated
- [x] brain_api HTTP service with Didi contract (5 endpoints)
- [x] brain_api HTTP service with Didi contract (full route set, ~30 routes)
- [x] brain_api dockerized (self-sufficient, taxonomy auto-refresh)
- [x] NLI stance vs query in `/v1/gather`
- [x] Lint pass contradiction detection (code + smoke test)
@ -213,16 +228,20 @@ Detailed file-by-file rundown is in `STATUS.md`.
## Operational notes
- **Resource footprint**: ~200 MB RAM total across the 3 containers; brain-api
- **Resource footprint**: ~200 MB RAM total across the 4 containers; brain-api
idles at ~6% CPU, spikes to ~20% during a `/v1/gather` with NLI.
- **Image size**: brain-api Docker image is ~253 MB (Python 3.12-slim base).
- **VPN dependency**: BGE and the LLM router live on a VPN-routed 10.11.10.x
network. If the VPN drops, `/v1/gather` returns 500 because Atomic cannot
embed the query. Confirm upstream reachability before debugging anything
else when search starts failing.
- **Self-sufficient startup**: brain_api pulls the current taxonomy from
Atomic at startup, so there is no baked `_tag_ids.json` in the image and
the container is portable across environments.
- **Upstream connectivity**: the LLM router, BGE-M3 embeddings, and the BGE
reranker are reached over Docker DNS as `didiAI-llm-api:14011`,
`didiAI-embeddings-api:14100`, and `didiAI-rerank-api:14200` on the shared
`didi-network`. If those upstreams are unreachable, `/v1/gather` returns 500
because Atomic cannot embed the query. Confirm upstream reachability before
debugging anything else when search starts failing.
- **Startup**: brain_api refreshes the `TagResolver` from Atomic at startup;
in addition `shared/_tag_ids.json` is mounted into the container (compose
volume) so background extraction — which instantiates `TagResolver`
directly, bypassing the lifespan refresh — can still resolve canonical tag
UUIDs.
- **Idempotency**: every operator script (taxonomy seeder, Wikipedia
importer, claim extractor, Lint pass) is idempotent via state files or
URL-based dedup. Re-running is always safe.

90
ai_platform/modules/didi_brain/STATUS.md
View file

@ -1,6 +1,13 @@
# DidiBrain — Status Snapshot
**Last updated:** 2026-04-23 (integration session — web-api cache + backend verification cache)
> **Note (current):** the platform is **LIVE and functional**. Since this
> snapshot the stack grew to **four containers** (added `didibrain-scheduler`),
> the analysis-atom cache (v2, gold/silver/bronze) and the Cache Freshness
> Defense layer (feeder + auditor + watcher) shipped, and the live working
> model is **`qwen3.5`** (vLLM, `MODEL_FAST` enabled). Sections below that
> still say "3 containers / 5 endpoints / 397B" are corrected inline.
**Working dir:** `/home/admin365/ml-projects/modules/didi_brain/`
**Related docs:**
- `CONTRACT_VERIFICATION_CACHE.md` — contract backend↔brain pentru verification cache (v2, current)
@ -17,24 +24,31 @@ DidiBrain is **SHIP READY** și **INTEGRATED**. Pe lângă contractul inițial H
- **Verification cache pentru didi-backend** — post-LLM stance/verdict storage
cu 4 staleness states (fresh / stale_framework / stale_prompt / miss)
**Trei containere, zero regressions pe funcționalitatea v1.** Tabelul relational
nou `brain_verification_cache` trăiește în același Postgres cu atomic-server,
prefix `brain_` pentru izolare.
- **Analysis atom cache (v2)** — techniques + ai_tampered + claims full-component
results, 3-tier gold/silver/bronze (since 2026-05-01)
- **Cache Freshness Defense** — `didibrain-scheduler` (feeder + auditor + watcher)
keeps cached verdicts fresh
**Patru containere, zero regressions pe funcționalitatea v1.** Tabelele relationale
noi `brain_verification_cache` + `brain_analysis_atom` (și layerul fact_status)
trăiesc în același Postgres cu atomic-server, prefix `brain_` pentru izolare.
## Current state in 30 seconds
```
3 containers running continuously, all healthy
4 containers running continuously, all healthy
didibrain-api brain_api FastAPI service :8090 52 MB RAM
didibrain-atomic atomic-server (API only) :8088 83 MB RAM
didibrain-postgres pgvector pg16 :5434 60 MB RAM
(+ brain_verification_cache table)
didibrain-api brain_api FastAPI service :8090 52 MB RAM
didibrain-atomic atomic-server (API only) :8088 (8080 internal) 83 MB RAM
didibrain-postgres pgvector pg16 :5434 60 MB RAM
(+ brain_verification_cache
+ brain_analysis_atom tables)
didibrain-scheduler feeder + auditor + watcher (no port)
Brain contents (grow continuously via web-api ingest + manual bootstrap)
79 tags (canonical taxonomy, 7 root namespaces)
19+ documents (initial Wikipedia vaccines seed + anything premium adds)
509+ claims (extracted by Qwen 397B, substring validated)
509+ claims (extracted by qwen3.5, substring validated)
Verification cache (new in v2 — Apr 23)
brain_verification_cache UNIQUE (claim_hash, tier)
@ -82,7 +96,7 @@ Didi backend ┌─────────────
│ DidiBrain stack │
│ │
│ ┌──────────────────────┐ │
│ │ brain-api :8090 │ FastAPI + Uvicorn, 5 v1 endpoints │
│ │ brain-api :8090 │ FastAPI + Uvicorn, full route set │
│ │ (didibrain-api) │ speaks Didi contract 1:1 │
│ └──────────┬───────────┘ │
│ │ HTTP (docker DNS) │
@ -97,25 +111,26 @@ Didi backend ┌─────────────
│ │ HTTPS
│ ▼
│ ┌────────────────────────────────┐
│ │ BGE-M3 embeddings │ 10.11.10.15:8200
│ │ BGE-M3 embeddings │ 10.11.10.15:14100
│ │ (vLLM OpenAI-compat) │ 1024 dim, 8K ctx, multilingual
│ └────────────────────────────────┘
│
│ ┌────────────────────────────────┐
│ │ BGE-reranker-v2-m3 │ 10.11.10.15:8100
│ │ BGE-reranker-v2-m3 │ 10.11.10.15:14200
│ │ (cross-encoder) │ precision boost
│ └────────────────────────────────┘
│
│ ┌────────────────────────────────┐
└─▶│ Qwen3.5-397B-A17B │ 10.11.10.17:14011 (router)
└─▶│ qwen3.5 │ 10.11.10.17:14011 (router)
│ via LLM router │ round-robin to .18 and .19
│ (llama.cpp + vLLM backends) │ claim extraction, NLI
│ (vLLM backends) │ claim extraction, NLI
└────────────────────────────────┘
```
All upstream endpoints live on a VPN-routed 10.11.10.x network. The Docker
containers reach them through WSL2 NAT (Docker Desktop) or host networking
(Linux server).
All upstream endpoints (LLM router, BGE-M3, reranker) are reached over Docker
DNS on the shared external `didi-network` as `didiAI-llm-api:14011`,
`didiAI-embeddings-api:14100`, and `didiAI-rerank-api:14200` (the 10.11.10.x
addresses above are the host-side equivalents).
## Repo layout
@ -131,7 +146,7 @@ didibrain/
├── AUDIT.md # initial upstream Atomic audit
│
├── infra/
│ └── docker-compose.yml # 3-service stack (+ optional atomic-web)
│ └── docker-compose.yml # 4-service stack (api, atomic, postgres, scheduler)
│
├── brain_api/ # HTTP service — the main deliverable
│ ├── Dockerfile
@ -184,7 +199,7 @@ didibrain/
├── 06_validate_queries.py # smoke test doc-level retrieval
├── 07_run_extraction.py # claim extraction batch
├── 08_validate_claims.py # smoke test claim-level retrieval
├── 09_brain_api_demo.py # brain_api contract test (all 5 endpoints)
├── 09_brain_api_demo.py # brain_api contract test (core web-gathering endpoints)
├── 10_run_lint.py # Lint pass runner (--limit --force)
├── 11_show_contradictions.py # read state file, render top contradictions
└── bootstrap_deploy.sh # fresh-server deploy orchestrator
@ -209,6 +224,17 @@ backends.
| `POST /v1/gather` with `run_nli:false` | claim → evidence | ~1.5-2 s | skip stance classification for speed |
| `POST /v1/image-search` | stub | <5 ms | always empty list |
| `POST /v1/ingest` | populate from web module | variable | creates atoms + optional async extraction |
| `POST /v1/verification_cache` | write claim verification result | <50 ms | claims cache (v2) |
| `POST /v1/analysis_atom/lookup` | read cached analysis | <50 ms | techniques / ai_tampered / claims |
| `POST /v1/analysis_atom` | write analysis atom | <50 ms | silver/bronze by `llm_confidence` |
| `PATCH /v1/analysis_atom/{id}` | promote to gold | <50 ms | moderator review |
| `GET /v1/analysis_atom/stats` | per-tier/component counts | <20 ms | 24h hit rate |
| `POST /v1/canonicalize` | temporal claim disambiguation | LLM-bound | Pilon 7 |
| `POST /v1/cache/invalidate` | mass invalidation (`dry_run`) | variable | Pilon 8 |
| `GET /v1/cache/audit_log` | audit browser | <50 ms | paginated |
| `GET/PATCH /v1/fact_status/*` | versioned fact-status layer | <50 ms | list/detail/versions/override |
(Full set ~30 routes incl. FastAPI auto docs; see `INDEX.md` for the canonical list.)
### /v1/gather response shape (key fields)
@ -297,9 +323,9 @@ or in README.md under "Production deploy".
Only the upstream endpoint URLs may need updating:
```bash
LLM_ROUTER_URL=http://10.11.10.17:14011 # if router is on VPN, unchanged
EMBEDDING_URL=http://10.11.10.15:8200 # if BGE is on VPN, unchanged
RERANKER_URL=http://10.11.10.15:8100 # if reranker is on VPN, unchanged
LLM_ROUTER_URL=http://10.11.10.17:14011 # didiAI-llm-api, unchanged
EMBEDDING_URL=http://10.11.10.15:14100 # didiAI-embeddings-api, unchanged
RERANKER_URL=http://10.11.10.15:14200 # didiAI-rerank-api, unchanged
```
Everything else (`ATOMIC_URL`, ports, model names, Postgres creds) is either
@ -344,15 +370,13 @@ curl -s -X POST http://localhost:8090/v1/gather \
call this endpoint; we read `embedding_status` per atom via `list_atoms`
or `get_atom` instead.
3. **Qwen 35B (vLLM on :14001)** — thinking mode stuck ON via the router and
safety alignment refuses disinfo-extraction tasks. Disabled in config
(`MODEL_FAST_ENABLED=false`); the whole pipeline runs on Qwen 397B. If a
future session un-sticks 35B, Lint pass could speed up ~4x by using it.
3. **Fast model `qwen3.5`** — now **enabled** (`MODEL_FAST_ENABLED=true`) and
is the live working model for the whole pipeline (extraction, NLI, gather)
via the LLM router at `didiAI-llm-api:14011` (vLLM). (Historically the fast
model was disabled and the pipeline ran on a separate reasoning model; that
is no longer the case.)
4. **Gemma 31B endpoint (10.11.10.16:8001)** — port unreachable (host pings
OK). Not used by any feature; listed as optional in `.env`.
5. **Atomic's React UI not running** — we only deploy `atomic-server` (API),
4. **Atomic's React UI not running** — we only deploy `atomic-server` (API),
not `atomic-web` (React frontend). `http://localhost:8088/` returns 404
for this reason. Brain_api has its own Swagger UI at `/docs` which covers
dev-testing needs. If visual atom/tag/wiki browsing is needed, add an
@ -362,7 +386,7 @@ curl -s -X POST http://localhost:8090/v1/gather \
| Symptom | Root cause | Fix |
|---|---|---|
| `/health` timeouts | BGE endpoint unreachable (VPN down?) | `curl http://10.11.10.15:8200/v1/models` on host; bring VPN back |
| `/health` timeouts | BGE endpoint unreachable | `curl http://10.11.10.15:14100/v1/models` on host; restore upstream reachability |
| `/v1/gather` returns 500 silently | Same as above — Atomic can't embed the query | Same fix |
| Post-reboot: brain_api empty reply | uvicorn bound to 127.0.0.1 inside container (not 0.0.0.0) | Rebuild — Dockerfile now sets `BRAIN_API_HOST=0.0.0.0` |
| `LLM_ROUTER_URL` points to localhost but nothing there | Router is actually on 10.11.10.17:14011 (not on the dev box) | Update `.env`; validated in session 2 |
@ -396,12 +420,12 @@ curl -s -X POST http://localhost:8090/v1/gather \
- Built `shared/` + `extractor/` + scripts 01-08
- Stood up Atomic + Postgres in compose
- Imported 19 Wikipedia vaccines articles, EN + RO
- Extracted 513 claims via Qwen 397B (1.2% hallucination drop)
- Extracted 513 claims via qwen3.5 (1.2% hallucination drop)
- Validated claim-level retrieval (cross-lingual RO↔EN confirmed)
### Session 2 (2026-04-11, ~4 hours)
- Re-onboarded state after VPN restart (fixed LLM router URL)
- Built `brain_api/` FastAPI service with 5-endpoint Didi contract
- Built `brain_api/` FastAPI service with the Didi web-gathering contract
- Dockerized brain_api (self-sufficient startup, taxonomy refresh from atomic)
- Added NLI stance-vs-query pass in `/v1/gather` (additive fields in `brain_meta`)
- Fixed NLI concurrency (MAX_PARALLEL=2 matches llama.cpp backends)

9
ai_platform/modules/didi_brain/infra/docker-compose.yml
View file

@ -1,15 +1,18 @@
# =============================================================================
# DidiBrain — Atomic + Postgres pgvector
# =============================================================================
# This compose stack runs:
# This compose stack runs four services:
# - postgres (pgvector/pgvector:pg16) on port 5434
# - atomic-server (kenforthewin/atomic-server:latest) on port 8080
# - atomic-server (kenforthewin/atomic-server:latest) on port 8088 (8080 internal)
# - brain-api (FastAPI service) on port 8090
# - scheduler (feeder + auditor + watcher + heartbeat, no exposed port)
# All four attach to the external shared `didi-network`.
#
# Atomic is configured to use Postgres as the data backend (atoms, embeddings,
# tags, etc.) while keeping the SQLite registry for tokens & global settings
# in the local volume at /data.
#
# AI provider config (BGE-M3 endpoint, Qwen 397B router) is NOT set here —
# AI provider config (BGE-M3 endpoint, qwen3.5 router) is NOT set here —
# it lives in Atomic's settings table and is bootstrapped post-startup by
# `scripts/02_bootstrap_atomic.py` which calls PUT /api/settings.
# =============================================================================

2
ai_platform/modules/didi_brain/pyproject.toml
View file

@ -47,4 +47,4 @@ requires = ["hatchling"]
build-backend = "hatchling.build"
[tool.hatch.build.targets.wheel]
packages = ["shared", "scraper", "extractor", "lint", "didi_client"]
packages = ["shared", "extractor", "lint"]

100
ai_platform/modules/domain_check/.env Normal file
View file

@ -0,0 +1,100 @@
# ===========================================
# DIDI Domain Check (T4 — evaluare credibilitate sursă)
# Modul dedicat platformei DIDI. Rulează izolat: Postgres + Redis proprii
# pe rețeaua internă domain_check_network; API-ul e expus pe didi-network
# ca `domain-check-api:11000` pentru agent-v3 (Lot 2).
# ===========================================
# Imagine locală (se construiește din ./api, nu se trage din registry)
REGISTRY_IMAGE=didiai-domain-check
IMAGE_TAG=latest
# ===========================================
# Database (izolat — doar pe rețeaua internă)
# ===========================================
DB_PASSWORD=8W2XWONcuorqzY0h4xAgJeSJ
DB_USER=dns_admin
DB_NAME=domain_check
DB_HOST=didiAI-domain-check-db
DB_PORT=5432
DB_EXTERNAL_PORT=12000
# ===========================================
# Redis (izolat — doar pe rețeaua internă)
# ===========================================
REDIS_HOST=didiAI-domain-check-redis
REDIS_PORT=6379
REDIS_EXTERNAL_PORT=12300
REDIS_DB=0
REDIS_PASSWORD=
# ===========================================
# API Keys
# ===========================================
WHOXY_API_KEY=876528325417e0bgs418d2cc7d8f193a6
VIRUSTOTAL_API_KEY=4d7afdff71c977a178b07d25b58c36660c416e3e9b1bf7ee8efaaf94aa02d0bc
# ===========================================
# Application (producție)
# ===========================================
FLASK_ENV=production
FLASK_APP=run.py
SECRET_KEY=3UnlbbbKMLYalxtvu7IJHAwMMF5KIFYj
LOG_LEVEL=INFO
DEBUG=False
# ===========================================
# API
# ===========================================
API_HOST=0.0.0.0
API_PORT=11000
API_VERSION=v1
# ===========================================
# Cache TTL (secunde)
# ===========================================
REDIS_TTL_HOT=21600
REDIS_TTL_WARM=86400
REDIS_TTL_COLD=604800
# ===========================================
# Risk thresholds
# ===========================================
RISK_THRESHOLD_LOW=30
RISK_THRESHOLD_MEDIUM=60
RISK_THRESHOLD_HIGH=85
DOMAIN_AGE_CRITICAL=90
DOMAIN_AGE_HIGH=180
DOMAIN_AGE_MEDIUM=365
# ===========================================
# Rate limiting extern (protejează cotele Whoxy/VirusTotal; intern e exempt)
# ===========================================
WHOXY_DAILY_LIMIT=8000
VIRUSTOTAL_DAILY_LIMIT=500
# ===========================================
# Batch
# ===========================================
BATCH_CHUNK_SIZE=50
BATCH_TIMEOUT_SECONDS=300
# ===========================================
# Celery (broker/back-end pe Redis-ul intern)
# ===========================================
CELERY_BROKER_URL=redis://didiAI-domain-check-redis:6379/1
CELERY_RESULT_BACKEND=redis://didiAI-domain-check-redis:6379/2
# ===========================================
# Monitoring (opțional)
# ===========================================
SENTRY_DSN=
DATADOG_API_KEY=
# ===========================================
# Security
# ===========================================
# Consumatorul e agent-v3 (server-to-server pe didi-network), fără browser → CORS lax OK.
CORS_ORIGINS=*
DISABLE_SWAGGER=false
SIMPLE_HEALTH=false

172
ai_platform/modules/domain_check/.env.example Normal file
View file

@ -0,0 +1,172 @@
# ===========================================
# Domain Check API - Environment Configuration
# ===========================================
# Port Schema: Dev (5xxxx), Prod (1xxxx)
# ===========================================
# ===========================================
# Registry Configuration (pentru deploy din GitLab)
# ===========================================
REGISTRY_IMAGE=didiai-domain-check
IMAGE_TAG=latest
# ===========================================
# Database Configuration
# Port: x20xx = Databases/PostgreSQL
# ===========================================
# SCHIMBA PAROLA pentru productie!
# Genereaza cu: openssl rand -base64 24 | tr -dc 'a-zA-Z0-9' | head -c 24
DB_PASSWORD=DnsCheck2026!Secure
DB_USER=dns_admin
DB_NAME=domain_check
DB_HOST=domain_check_postgres
DB_PORT=5432
# Port producție conform schema (x20xx = PostgreSQL)
DB_EXTERNAL_PORT=12000
# ===========================================
# Redis Configuration
# Port: x23xx = Cache/Redis
# ===========================================
REDIS_HOST=domain_check_redis
REDIS_PORT=6379
# Port producție conform schema (x23xx = Redis)
REDIS_EXTERNAL_PORT=12300
REDIS_DB=0
REDIS_PASSWORD=
# ===========================================
# API Keys
# ===========================================
# WHOXY API Key - obtine de la https://www.whoxy.com/
WHOXY_API_KEY=876528325417e0bgs418d2cc7d8f193a6
# VIRUSTOTAL API Key (optional) - obtine de la https://www.virustotal.com/
VIRUSTOTAL_API_KEY=4d7afdff71c977a178b07d25b58c36660c416e3e9b1bf7ee8efaaf94aa02d0bc
# ===========================================
# Application Settings
# ===========================================
# SCHIMBA pentru productie: production
FLASK_ENV=development
FLASK_APP=run.py
# SCHIMBA SECRET_KEY pentru productie!
# Genereaza cu: openssl rand -base64 32 | tr -dc 'a-zA-Z0-9' | head -c 32
SECRET_KEY=dns-check-secret-key-2026-change-in-production
LOG_LEVEL=INFO
# SCHIMBA pentru productie: False
DEBUG=True
# ===========================================
# API Configuration
# Port: x1xxx = API/Gateway
# ===========================================
API_HOST=0.0.0.0
# Port producție conform schema (x1xxx = API/Gateway)
API_PORT=11000
API_VERSION=v1
# ===========================================
# Cache TTL (seconds)
# ===========================================
REDIS_TTL_HOT=21600 # 6 hours
REDIS_TTL_WARM=86400 # 24 hours
REDIS_TTL_COLD=604800 # 7 days
# ===========================================
# Risk Score Thresholds
# ===========================================
RISK_THRESHOLD_LOW=30
RISK_THRESHOLD_MEDIUM=60
RISK_THRESHOLD_HIGH=85
# ===========================================
# Domain Age Thresholds (days)
# ===========================================
DOMAIN_AGE_CRITICAL=90 # < 3 months
DOMAIN_AGE_HIGH=180 # < 6 months
DOMAIN_AGE_MEDIUM=365 # < 1 year
# ===========================================
# API Rate Limiting
# ===========================================
WHOXY_DAILY_LIMIT=8000
VIRUSTOTAL_DAILY_LIMIT=500
# ===========================================
# Batch Processing
# ===========================================
BATCH_CHUNK_SIZE=50
BATCH_TIMEOUT_SECONDS=300
# ===========================================
# Celery Configuration
# ===========================================
CELERY_BROKER_URL=redis://domain_check_redis:6379/1
CELERY_RESULT_BACKEND=redis://domain_check_redis:6379/2
# ===========================================
# Monitoring (optional)
# ===========================================
SENTRY_DSN=
DATADOG_API_KEY=
# ===========================================
# Security Settings (PRODUCTIE)
# ===========================================
# CORS Origins - Lista domeniilor permise (separate prin virgula)
# Default: * (permite toate - DOAR pentru development!)
# Productie: specifica domeniile tale
# Exemplu: CORS_ORIGINS=https://didi.example.ro,https://example.ro
CORS_ORIGINS=*
# Dezactiveaza Swagger UI si ReDoc (ascunde /docs si /redoc)
# Default: false (development)
# Productie: true
DISABLE_SWAGGER=false
# Health endpoint simplificat (returneaza doar {"status":"ok"})
# Default: false (development - returneaza detalii complete)
# Productie: true (ascunde informatii despre sistem)
SIMPLE_HEALTH=false
# ===========================================
# QUICK SETUP pentru server DEVELOPMENT:
# ===========================================
# 1. cp .env.example .env
# 2. docker compose up -d --build
# 3. curl http://localhost:11000/health
# ===========================================
# ===========================================
# QUICK SETUP pentru server PRODUCTION:
# ===========================================
# 1. cp .env.example .env
# 2. Modifica valorile pentru productie:
# sed -i 's/FLASK_ENV=development/FLASK_ENV=production/' .env
# sed -i 's/DEBUG=True/DEBUG=False/' .env
# sed -i 's/CORS_ORIGINS=\*/CORS_ORIGINS=https:\/\/your-domain.com/' .env
# sed -i 's/DISABLE_SWAGGER=false/DISABLE_SWAGGER=true/' .env
# sed -i 's/SIMPLE_HEALTH=false/SIMPLE_HEALTH=true/' .env
# 3. Genereaza parole securizate:
# DB_PASS=$(openssl rand -base64 24 | tr -dc 'a-zA-Z0-9' | head -c 24)
# SECRET=$(openssl rand -base64 32 | tr -dc 'a-zA-Z0-9' | head -c 32)
# sed -i "s/DnsCheck2026\!Secure/$DB_PASS/" .env
# sed -i "s/dns-check-secret-key-2026-change-in-production/$SECRET/" .env
# 4. docker compose up -d
# 5. curl http://localhost:11000/health
# ===========================================

100
ai_platform/modules/domain_check/.gitignore vendored Normal file
View file

@ -0,0 +1,100 @@
# Environment variables
.env
.env.local
.env.*.local
# Python
__pycache__/
*.py[cod]
*$py.class
*.so
.Python
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
wheels/
share/python-wheels/
*.egg-info/
.installed.cfg
*.egg
MANIFEST
*.manifest
*.spec
pip-log.txt
pip-delete-this-directory.txt
# Virtual environments
venv/
ENV/
env/
.venv
# IDEs
.vscode/
.idea/
*.swp
*.swo
*~
.DS_Store
# Logs
*.log
logs/
*.log.*
# Database
*.db
*.sqlite
*.sqlite3
# Redis dump
dump.rdb
# Docker
.docker/
# Testing
.pytest_cache/
.coverage
htmlcov/
.tox/
.nox/
coverage.xml
*.cover
.hypothesis/
# Jupyter Notebook
.ipynb_checkpoints
# pyenv
.python-version
# Celery
celerybeat-schedule
celerybeat.pid
# Backup files
*.bak
*.backup
*.tmp
# OS
Thumbs.db
.DS_Store
# Project specific
data/
backups/
exports/
# GitLab setup scripts (contain secrets)
gitlab-variables-values.txt
setup-gitlab-variables.sh

325
ai_platform/modules/domain_check/.gitlab-ci.yml Normal file
View file

@ -0,0 +1,325 @@
# ===========================================
# Domain Check API - GitLab CI/CD Pipeline
# ===========================================
# Registry: <registry-didi> (GitLab Container Registry)
# Runner: Main Docker Runner (10.11.10.102) - tags: docker, ci, cd
# ===========================================
stages:
- lint
- security
- test
- build
- release
- deploy
variables:
DOCKER_DRIVER: overlay2
DOCKER_TLS_CERTDIR: "/certs"
# GitLab CI provides these automatically:
# CI_REGISTRY = <registry-didi>
# CI_REGISTRY_IMAGE = didiai-domain-check
# CI_REGISTRY_USER / CI_REGISTRY_PASSWORD = auto-generated JWT
# Cache pip packages
cache:
key: ${CI_COMMIT_REF_SLUG}
paths:
- .cache/pip
# ===========================================
# LINT Stage
# ===========================================
lint:python:
stage: lint
image: python:3.10-slim
tags:
- docker
before_script:
- pip install --cache-dir .cache/pip flake8 black isort
script:
- flake8 api/app --max-line-length=120 --ignore=E501,W503 || true
- black --check api/app || true
- isort --check-only api/app || true
allow_failure: true
rules:
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
- if: $CI_COMMIT_BRANCH
lint:dockerfile:
stage: lint
image: hadolint/hadolint:latest-debian
tags:
- docker
script:
- hadolint api/Dockerfile || true
allow_failure: true
rules:
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
- if: $CI_COMMIT_BRANCH
# ===========================================
# SECURITY Stage (SAST)
# ===========================================
sast:bandit:
stage: security
image: python:3.10-slim
tags:
- docker
before_script:
- pip install --cache-dir .cache/pip bandit
script:
- bandit -r api/app -f json -o bandit-report.json || true
- bandit -r api/app -f txt || true
artifacts:
paths:
- bandit-report.json
when: always
expire_in: 1 week
allow_failure: true
rules:
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
- if: $CI_COMMIT_BRANCH
sast:safety:
stage: security
image: python:3.10-slim
tags:
- docker
before_script:
- pip install --cache-dir .cache/pip safety
script:
- cd api && safety check -r requirements.txt --json > ../safety-report.json || true
- cd api && safety check -r requirements.txt || true
artifacts:
paths:
- safety-report.json
when: always
expire_in: 1 week
allow_failure: true
rules:
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
- if: $CI_COMMIT_BRANCH
sast:trivy:
stage: security
image:
name: aquasec/trivy:latest
entrypoint: [""]
tags:
- docker
script:
- trivy fs --exit-code 0 --severity HIGH,CRITICAL --format json -o trivy-report.json . || true
- trivy fs --exit-code 0 --severity HIGH,CRITICAL . || true
artifacts:
paths:
- trivy-report.json
when: always
expire_in: 1 week
allow_failure: true
rules:
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
- if: $CI_COMMIT_BRANCH
# ===========================================
# TEST Stage
# ===========================================
test:api:
stage: test
image: python:3.10-slim
tags:
- docker
services:
- postgres:15-alpine
- redis:7-alpine
variables:
POSTGRES_DB: test_domain_check
POSTGRES_USER: test_user
POSTGRES_PASSWORD: test_password
DATABASE_URL: postgresql://test_user:test_password@postgres:5432/test_domain_check
REDIS_URL: redis://redis:6379/0
FLASK_ENV: testing
before_script:
- cd api
- pip install --cache-dir ../.cache/pip -r requirements.txt
script:
- python -c "from app import create_app; app = create_app(); print('App created successfully')"
allow_failure: true
rules:
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
- if: $CI_COMMIT_BRANCH
# ===========================================
# BUILD Stage - Push to GitLab Container Registry
# ===========================================
build:docker:
stage: build
image: docker:24-dind
tags:
- docker
- ci
services:
- docker:24-dind
before_script:
# Login to GitLab Container Registry (auto credentials)
- docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
script:
- echo "Building image $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG"
- docker build -t $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG -t $CI_REGISTRY_IMAGE:latest ./api
- docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG
- docker push $CI_REGISTRY_IMAGE:latest
- echo "Image pushed to $CI_REGISTRY_IMAGE"
rules:
- if: $CI_COMMIT_BRANCH == "main"
- if: $CI_COMMIT_BRANCH == "develop"
# ===========================================
# RELEASE Stage - Create deployment bundle
# ===========================================
release:bundle:
stage: release
image: alpine:latest
tags:
- docker
script:
- apk add --no-cache tar gzip
- mkdir -p release-bundle/deploy
# Copy deployment files
- cp docker-compose.yml release-bundle/deploy/
- cp deploy.sh release-bundle/deploy/ 2>/dev/null || echo "#!/bin/bash" > release-bundle/deploy/deploy.sh
- cp .env.example release-bundle/deploy/ 2>/dev/null || cp .env release-bundle/deploy/env.example 2>/dev/null || true
# Create IMAGE.txt with registry path
- echo "$CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG" > release-bundle/IMAGE.txt
- echo "$CI_REGISTRY_IMAGE:latest" >> release-bundle/IMAGE.txt
# Create restore script
- |
cat > release-bundle/restore.sh << 'RESTORE_EOF'
#!/bin/bash
set -e
echo "=== Domain Check API - Restore ==="
IMAGE=$(head -1 IMAGE.txt)
echo "Pulling image: $IMAGE"
docker pull $IMAGE
echo "Starting services..."
cd deploy
docker compose up -d
echo "Done! Check: docker compose ps"
RESTORE_EOF
- chmod +x release-bundle/restore.sh release-bundle/deploy/deploy.sh
# Create tarball
- tar -czvf release-bundle-${CI_COMMIT_SHORT_SHA}.tgz release-bundle/
- ls -la release-bundle-*.tgz
artifacts:
paths:
- release-bundle-*.tgz
expire_in: 30 days
rules:
- if: $CI_COMMIT_BRANCH == "main"
# ===========================================
# DEPLOY Stage - Deploy to server via SSH
# ===========================================
deploy:dev:
stage: deploy
image: alpine:latest
tags:
- docker
- cd
before_script:
- apk add --no-cache openssh-client
- eval $(ssh-agent -s)
- echo "$SSH_PRIVATE_KEY" | tr -d '\r' | ssh-add -
- mkdir -p ~/.ssh
- chmod 700 ~/.ssh
- echo "$SSH_KNOWN_HOSTS" >> ~/.ssh/known_hosts
- chmod 644 ~/.ssh/known_hosts
script:
- |
ssh $DEPLOY_USER@$DEPLOY_HOST << ENDSSH
set -e
echo "=== Deploying Domain Check API ==="
cd /home/admin365/domain-check
# Pull latest code
git pull origin main
# Login to GitLab registry
docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
# Pull latest image
docker pull $CI_REGISTRY_IMAGE:latest || echo "Pull failed, building locally"
# Restart services
docker compose down
docker compose up -d --build
# Health check
sleep 10
curl -s http://localhost:51000/health || echo "Health check pending..."
docker compose ps
echo "=== Deploy complete ==="
ENDSSH
environment:
name: development
url: http://domain-check-api:11000
rules:
- if: $CI_COMMIT_BRANCH == "main"
when: manual
allow_failure: true
deploy:auto:
stage: deploy
image: alpine:latest
tags:
- docker
- cd
before_script:
- apk add --no-cache openssh-client
- eval $(ssh-agent -s)
- echo "$SSH_PRIVATE_KEY" | tr -d '\r' | ssh-add -
- mkdir -p ~/.ssh
- chmod 700 ~/.ssh
- echo "$SSH_KNOWN_HOSTS" >> ~/.ssh/known_hosts
- chmod 644 ~/.ssh/known_hosts
script:
- |
ssh $DEPLOY_USER@$DEPLOY_HOST << ENDSSH
set -e
cd /home/admin365/domain-check
git pull origin main
docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY || true
docker pull $CI_REGISTRY_IMAGE:latest || true
docker compose up -d --build
docker compose ps
ENDSSH
environment:
name: development
url: http://domain-check-api:11000
rules:
- if: $CI_COMMIT_BRANCH == "main"
allow_failure: true
# ===========================================
# Cleanup
# ===========================================
cleanup:images:
stage: .post
image: docker:24-dind
tags:
- docker
services:
- docker:24-dind
script:
- docker image prune -f
when: always
allow_failure: true
rules:
- if: $CI_COMMIT_BRANCH == "main"

192
ai_platform/modules/domain_check/ACCESS_INFO.md Normal file
View file

@ -0,0 +1,192 @@
# 🌐 Access Information - Domain Check API
## 📍 Server Details
**Server IP (LAN):** `10.11.10.200`
**Ports:**
- API: `5000`
- PostgreSQL: `5433` (internal: 5432)
- Redis: `6380` (internal: 6379)
- Dashboard: `8501` (when ready)
---
## 🔗 API Endpoints
### **Base URL:** `http://10.11.10.200:5000`
### Main Endpoints:
- **Health Check:** `http://10.11.10.200:5000/health`
- **API Root:** `http://10.11.10.200:5000/`
- **Domain Check:** `POST http://10.11.10.200:5000/api/v1/check`
### Documentation:
- **Swagger UI:** `http://10.11.10.200:5000/docs` 📖
- **ReDoc:** `http://10.11.10.200:5000/redoc` 📚
---
## 🧪 Quick Tests
### 1. Health Check
```bash
curl http://10.11.10.200:5000/health
```
### 2. WHOIS + DNS Check
```bash
curl -X POST http://10.11.10.200:5000/api/v1/check \
-H "Content-Type: application/json" \
-d '{
"domain": "google.com",
"check_options": {
"whois": true,
"dns": true,
"ssl": false
}
}' | python3 -m json.tool
```
### 3. Full Test Suite
```bash
cd /home/admin365/domain-check
./test-complete-lan.sh
```
---
## 🔐 Database Access
### PostgreSQL
```bash
# From server
docker exec -it dns_postgres psql -U dns_admin -d domain_check
# Connection string
postgresql://dns_admin:DnsCheck2026!Secure@10.11.10.200:5433/domain_check
```
### Redis
```bash
# From server
docker exec -it dns_redis redis-cli
# Connection string
redis://10.11.10.200:6380/0
```
---
## 🐳 Docker Services
### Check Status
```bash
docker compose ps
```
### View Logs
```bash
# All services
docker compose logs -f
# Specific service
docker compose logs -f dns_api
```
### Restart Services
```bash
# All services
docker compose restart
# Specific service
docker compose restart dns_api
```
---
## 📊 Test Results Summary
✅ **Health Check:** PASSED
✅ **WHOIS Lookup:** PASSED
✅ **DNS Checking:** PASSED
✅ **Risk Scoring:** PASSED
✅ **Database Storage:** PASSED
✅ **API Documentation:** PASSED
⏸️ **SSL Checking:** Code ready (needs Docker network fix)
⏸️ **Redis Caching:** Configured (not active)
⏸️ **Batch Processing:** Not implemented yet
---
## 🌍 Access from Other Machines
From any machine on the **10.11.10.x** network:
### Browser Access:
- Swagger UI: `http://10.11.10.200:5000/docs`
- ReDoc: `http://10.11.10.200:5000/redoc`
### API Calls:
```bash
curl http://10.11.10.200:5000/api/v1/check \
-H "Content-Type: application/json" \
-d '{"domain": "your-domain.com"}'
```
---
## 📱 Frontend Access (When Ready)
**Dashboard URL:** `http://10.11.10.200:3000`
**Status:** Not yet implemented (Next.js ready to build)
---
## 🔧 Troubleshooting
### API not responding?
```bash
# Check if containers are running
docker compose ps
# Check API logs
docker compose logs dns_api --tail=50
# Restart API
docker compose restart dns_api
```
### Database connection issues?
```bash
# Check PostgreSQL
docker exec dns_postgres pg_isready -U dns_admin
# View database logs
docker compose logs dns_postgres --tail=30
```
### Port conflicts?
```bash
# Check if ports are in use
sudo netstat -tlnp | grep -E "5000|5433|6380"
```
---
## 📞 Support
**GitLab Repository:** (modul domain_check al platformei DIDI)
**Documentation:** See `/home/admin365/domain-check/` directory
**Key Files:**
- `README.md` - Quick start guide
- `API_ARCHITECTURE.md` - Technical details
- `CURRENT_STATUS.md` - Complete API reference
- `NEXT_STEPS.md` - Development roadmap
---
**Last Updated:** 2026-01-29 21:55:00
**Server:** admin365@10.11.10.200
**Status:** ✅ Production Ready (MVP)

1233
ai_platform/modules/domain_check/API_ARCHITECTURE.md Normal file
View file

File diff suppressed because it is too large Load diff

62
ai_platform/modules/domain_check/CLAUDE.md Normal file
View file

@ -0,0 +1,62 @@
# CLAUDE.md
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
## What This Is
Domain Check API — a domain verification and risk-scoring service for detecting disinformation sources (newly registered / suspicious domains). Flask REST API + PostgreSQL + Redis + Celery worker, all run via Docker Compose. Live on this server at `http://domain-check-api:11000` (production ports: API 11000, PostgreSQL 12000, Redis 12300 — set in `.env`, the README sometimes still shows dev ports 5xxxx).
## Commands
```bash
# Run the full stack (the only supported way to run it — the API expects Postgres/Redis containers)
docker compose up -d --build
docker compose ps
docker compose logs -f domain_check_api
# Smoke test
curl http://localhost:11000/health
./test-api-complete.sh # exercises the check endpoint
./test-complete-lan.sh # LAN integration test
# Manual check call
curl -X POST http://localhost:11000/api/v1/check/check \
-H "Content-Type: application/json" \
-d '{"domain": "example.com", "check_options": {"whois": true, "dns": true, "ssl": true}}'
# Lint (matches .gitlab-ci.yml)
flake8 api/app --max-line-length=120 --ignore=E501,W503
black --check api/app
isort --check-only api/app
# Database shell
docker exec -it didiAI-domain-check-db psql -U dns_admin -d domain_check
# Deploy helper (wraps docker compose)
./deploy.sh status|logs|start|stop|restart|update
```
There is no test suite — CI's "test" stage only verifies `create_app()` succeeds. CI (GitLab) runs lint → SAST (bandit/safety/trivy) → build → push to `didiai-domain-check` → SSH deploy; all lint/security jobs are `allow_failure: true`.
## Architecture
Everything lives in `api/app/` (the `dashboard/` directory is vestigial — only a Dockerfile/requirements, not in docker-compose; the actual dashboard is `api/app/static/index.html` served at `/`).
**Request flow:** `run.py` → `create_app()` app factory ([api/app/__init__.py](api/app/__init__.py)) → flask-restx namespaces under `/api/v1` → route handlers in `app/routes/` call service classes in `app/services/` → results persisted via SQLAlchemy models in `app/models/` and cached in Redis. Served by **gunicorn** (gthread workers) in the container — `python run.py` is dev-only.
- **`app/routes/check.py`** is the only fully implemented route (POST `/api/v1/check/check`). It runs the independent lookups **concurrently** in two phases via `ThreadPoolExecutor` (phase 1 needs only the domain: whois/dns/ssl/http/subdomains; phase 2 needs DNS results: ip/blacklist/port/mail). Each task runs inside `app.app_context()` because services read `current_app`. **All DB writes happen in the main thread afterwards** — the SQLAlchemy session is not thread-safe, so never add DB writes inside the threaded lookups. `domain.py`, `search.py`, `stats.py`, `batch.py` are stubs registered inside try/except ImportError so missing implementations fail silently.
- **`app/services/`** — one class per check type: `whois_service` (RDAP first, Whoxy API fallback; `normalize_whois_date()` coerces all dates to `datetime|None` — raw strings here were the root cause of the `.ro` 500), `dns_service`, `ssl_service`, `ip_intelligence_service`, `http_analysis_service`, `blacklist_service` (DNSBL), `port_scan_service`, `subdomain_service` (Certificate Transparency), and `mail_intelligence_service` (SPF/DKIM/DMARC parsing, MTA-STS/TLS-RPT/DANE, MX provider fingerprint + STARTTLS, optional SMTP RCPT probe — degrades gracefully when outbound :25 is blocked). `risk_scorer.py` combines outputs into a weighted 0–100 score (weights/thresholds in `config.py`); `_coerce_datetime()` there is the defense-in-depth guard against stray string dates.
- **`check_options`** flags (all default true except `port_scan`/`subdomains`/`smtp_probe` which default false): `whois, dns, ssl, ip_intelligence, http_analysis, blacklist, mail, port_scan, subdomains, smtp_probe, force_refresh`. The response includes a top-level `availability` block (`is_registered`/`is_available`/`confidence`) derived from combined WHOIS+DNS signals — WHOIS alone is unreliable for sparse registries like ROTLD.
- **Rate limiting** (`Flask-Limiter`, Redis-backed) is wired in `__init__.py` with internal LAN/loopback and `/health` exempt (`_is_internal_request`), errors swallowed. Tune via `RATELIMIT_DEFAULT` env. It exists to protect the paid Whoxy/VirusTotal quotas from external abuse, not internal callers.
- **`app/api_models.py`** defines all flask-restx request/response models. Note the pattern in `check.py`: models are created on a throwaway namespace at import time for decorators, then `__init__.py` re-attaches them to the real namespaces — keep model definitions in `api_models.py`, not inline.
- **Celery** is wired up (`celery_app.py`, separate `domain_check_worker` container) but `app/tasks/__init__.py` is empty — no tasks exist yet; batch processing is the intended use.
**Configuration:** `app/config.py` selects Development/Production/Testing via `FLASK_ENV`. Production flips important defaults: Swagger/ReDoc disabled (`DISABLE_SWAGGER`), minimal `/health` response (`SIMPLE_HEALTH`), CORS empty unless `CORS_ORIGINS` is set. All runtime config comes from `.env` (see `.env.example`); ports are only ever changed there, never in compose/code.
**Database schema** is created by `init-scripts/01-init-db.sql`, which runs only on first Postgres volume creation. There is no Flask-Migrate `migrations/` directory despite the extension being initialized — schema changes must be made in both the SQL init script and the SQLAlchemy models, and applied manually to existing databases. All child tables (`whois_records`, `dns_records`, `ssl_certificates`, etc.) FK to `domains` with `ON DELETE CASCADE`.
**Caching:** Redis with three TTL tiers (`REDIS_TTL_HOT/WARM/COLD`); a check request with `force_refresh: false` returns cached/DB data when fresh. Redis failure is non-fatal — the app logs a warning and runs with caching disabled.
## Network Context
This is the **T4 module (evaluare credibilitate sursă)** of the DIDI platform, living at `ai_platform/modules/domain_check`. It runs as an isolated stack (own Postgres + Redis on the internal `domain_check_network`); the API container also joins the shared external `didi-network` under the alias **`domain-check-api`**, so `agent-v3` (Lot 2) calls it at `http://domain-check-api:11000/api/v1/check/check` with no extra config. The API port is **not** published to the host (the platform gateway owns host:11000) — reach it via the didi-network alias, or `docker exec` into a container on that network. Docs: `API_ARCHITECTURE.md` (detailed design), `use_api.md` (consumer-facing API docs), `ACCESS_INFO.md`, `CURRENT_STATUS.md`, `NEXT_STEPS.md` (roadmap).

706
ai_platform/modules/domain_check/CURRENT_STATUS.md Normal file
View file

@ -0,0 +1,706 @@
# DOMAIN-CHECK - Status Actual și Documentație Completă
## 🗄️ DATABASE - Ce se Salvează Exact
### Fluxul Complet de Date
```
INPUT: {"domain": "example.com"}
↓
1. DOMAIN EXTRACTION & PARSING
↓
2. WHOIS LOOKUP (python-whois + Whoxy API fallback)
↓
3. RISK SCORING (5 factori)
↓
4. SAVE TO DATABASE (3 tabele principale)
↓
OUTPUT: JSON cu risk score + toate datele
```
---
## 📋 TABELE DATABASE - Date Salvate
### 1. **domains** - Informații Domain Principal
```sql
Câmpuri salvate:
- id (UUID)
- domain (ex: "example")
- subdomain (ex: "www" sau NULL)
- tld (ex: "com")
- full_domain (ex: "www.example.com")
- first_seen_at (când a fost văzut prima dată)
- last_checked_at (ultima verificare)
- check_count (de câte ori a fost verificat)
- is_active (boolean)
- created_at, updated_at
```
**Exemplu real:**
```json
{
"id": "587b61bf-e6bf-436b-9867-14df7d5a9cfb",
"domain": "google",
"subdomain": null,
"tld": "com",
"full_domain": "google.com",
"first_seen_at": "2026-01-29T13:38:27Z",
"last_checked_at": "2026-01-29T13:38:33Z",
"check_count": 1,
"is_active": true
}
```
---
### 2. **whois_records** - Date WHOIS Complete
```sql
Câmpuri salvate:
- id (UUID)
- domain_id (referință la domains)
- creation_date (când a fost creat domeniul)
- expiration_date (când expiră)
- updated_date (ultima actualizare WHOIS)
- registrar (ex: "GoDaddy", "Namecheap")
- registrar_url
- registrant_org (organizația proprietarului)
- registrant_country (țara - cod 2 litere)
- admin_email
- name_servers (array de DNS servers)
- status (array de statusuri domeniu)
- dnssec (boolean)
- raw_whois_data (JSONB - toate datele raw)
- data_source ("whois" sau "whoxy")
- fetched_at (când au fost preluate datele)
```
**Exemplu real pentru un domeniu funcțional:**
```json
{
"domain_id": "587b61bf-e6bf-436b-9867-14df7d5a9cfb",
"creation_date": "1997-09-15T04:00:00Z",
"expiration_date": "2028-09-14T04:00:00Z",
"registrar": "MarkMonitor Inc.",
"registrant_org": "Google LLC",
"registrant_country": "US",
"admin_email": "dns-admin@google.com",
"name_servers": ["ns1.google.com", "ns2.google.com"],
"status": ["clientDeleteProhibited", "clientTransferProhibited"],
"data_source": "whoxy",
"age_days": 10359
}
```
---
### 3. **risk_assessments** - Risk Scoring Rezultate
```sql
Câmpuri salvate:
- id (UUID)
- domain_id (referință)
- check_id (UUID unic pentru fiecare check)
- total_score (0-100)
- risk_level ("LOW", "MEDIUM", "HIGH", "CRITICAL")
- domain_age_score (scor 0-100)
- domain_age_days (vârsta în zile)
- ssl_score (0-100)
- dns_score (0-100)
- reputation_score (0-100)
- whois_score (0-100)
- factors (JSONB - detalii fiecare factor)
- is_new_domain (boolean - < 6 luni)
- is_suspicious (boolean)
- requires_manual_review (boolean)
- assessed_at (timestamp)
```
**Exemplu real:**
```json
{
"check_id": "76d6eb1a-d74b-4430-8b09-f2474a088699",
"total_score": 8,
"risk_level": "LOW",
"domain_age_score": 20,
"domain_age_days": null,
"ssl_score": 0,
"dns_score": 0,
"reputation_score": 0,
"whois_score": 20,
"is_new_domain": false,
"is_suspicious": false,
"factors": [
{
"factor": "domain_age",
"score": 20,
"weight": 0.3,
"reason": "No creation date available"
},
{
"factor": "whois",
"score": 20,
"weight": 0.1,
"reason": "No registrant organization"
}
]
}
```
---
### 4. **check_history** - Istoric Verificări
```sql
Câmpuri salvate:
- id (UUID)
- check_id (UUID unic)
- domain_id (referință)
- risk_assessment_id (referință)
- requested_by ("api", "cli", "dashboard")
- request_ip (IP-ul clientului)
- user_agent (browser/client info)
- check_options (JSONB - ce a fost verificat)
- processing_time_ms (cât a durat)
- cache_hit (boolean - a fost în cache?)
- changes_detected (boolean)
- change_summary (JSONB)
- status ("completed", "failed")
- error_message (dacă a fost eroare)
- created_at
```
**Exemplu real:**
```json
{
"check_id": "76d6eb1a-d74b-4430-8b09-f2474a088699",
"domain_id": "587b61bf-e6bf-436b-9867-14df7d5a9cfb",
"requested_by": "api",
"request_ip": "172.26.0.1",
"user_agent": "curl/7.81.0",
"check_options": {
"whois": true,
"dns": false,
"ssl": false,
"reputation": false,
"force_refresh": false
},
"processing_time_ms": 2009,
"cache_hit": false,
"status": "completed"
}
```
---
## 🔌 API ENDPOINTS - Ce Funcționează ACUM
### ✅ 1. **Health Check** - Verificare Status Sistem
**Endpoint:** `GET /health`
**Funcțional:** DA
**Parametri:** Niciunul
**Test:**
```bash
curl http://localhost:5000/health
```
**Răspuns:**
```json
{
"status": "healthy",
"version": "v1",
"environment": "development",
"database": "connected",
"redis": "disabled"
}
```
---
### ✅ 2. **API Root** - Informații Generale
**Endpoint:** `GET /`
**Funcțional:** DA
**Parametri:** Niciunul
**Test:**
```bash
curl http://localhost:5000/
```
**Răspuns:**
```json
{
"name": "Domain Check API",
"version": "v1",
"description": "Anti-Fake News Domain Verification & Risk Scoring API",
"documentation": {
"swagger": "/docs",
"redoc": "/redoc"
},
"endpoints": {
"health": "/health",
"api": "/api/v1"
}
}
```
---
### ✅ 3. **Domain Check** - Verificare Domeniu (PRINCIPAL)
**Endpoint:** `POST /api/v1/check`
**Funcțional:** DA (WHOIS + Risk Scoring)
**Content-Type:** `application/json`
**Parametri Request:**
```json
{
"domain": "string (required)",
"check_options": {
"whois": boolean (default: true),
"dns": boolean (default: false) [NOT IMPLEMENTED YET],
"ssl": boolean (default: false) [NOT IMPLEMENTED YET],
"reputation": boolean (default: false) [NOT IMPLEMENTED YET],
"force_refresh": boolean (default: false)
}
}
```
**Teste Complete:**
#### Test 1 - Domeniu Normal
```bash
curl -X POST http://localhost:5000/api/v1/check \
-H "Content-Type: application/json" \
-d '{
"domain": "google.com"
}'
```
#### Test 2 - Cu Toate Opțiunile
```bash
curl -X POST http://localhost:5000/api/v1/check \
-H "Content-Type: application/json" \
-d '{
"domain": "example.com",
"check_options": {
"whois": true,
"dns": false,
"ssl": false,
"reputation": false,
"force_refresh": true
}
}'
```
#### Test 3 - Domeniu Suspect (Nou)
```bash
curl -X POST http://localhost:5000/api/v1/check \
-H "Content-Type: application/json" \
-d '{
"domain": "suspicious-news-2026.com"
}'
```
#### Test 4 - Subdomeniu
```bash
curl -X POST http://localhost:5000/api/v1/check \
-H "Content-Type: application/json" \
-d '{
"domain": "www.facebook.com"
}'
```
#### Test 5 - Pretty Print (JSON formatat)
```bash
curl -X POST http://localhost:5000/api/v1/check \
-H "Content-Type: application/json" \
-d '{"domain": "github.com"}' \
| python3 -m json.tool
```
**Răspuns Complet (Exemplu):**
```json
{
"success": true,
"data": {
"domain": "example.com",
"check_id": "uuid-here",
"timestamp": "2026-01-29T15:30:00Z",
"whois": {
"creation_date": "1995-08-14T04:00:00Z",
"expiration_date": "2027-08-13T04:00:00Z",
"registrar": "IANA",
"age_days": 11125,
"status": ["clientDeleteProhibited"],
"name_servers": ["a.iana-servers.net", "b.iana-servers.net"]
},
"dns": null,
"ssl": null,
"reputation": null,
"risk_score": {
"total": 15,
"level": "LOW",
"factors": [
{
"factor": "domain_age",
"score": 0,
"weight": 0.3,
"reason": "Trusted age (11125 days old, 2+ years)",
"age_days": 11125
},
{
"factor": "reputation",
"score": 0,
"weight": 0.3,
"reason": "No reputation data"
},
{
"factor": "ssl",
"score": 0,
"weight": 0.2,
"reason": "Valid SSL certificate"
},
{
"factor": "dns",
"score": 0,
"weight": 0.1,
"reason": "Good DNS setup"
},
{
"factor": "whois",
"score": 0,
"weight": 0.1,
"reason": "Transparent WHOIS"
}
],
"thresholds": {
"low": "0-30",
"medium": "31-60",
"high": "61-85",
"critical": "86-100"
}
}
},
"metadata": {
"cached": false,
"processing_time_ms": 2009,
"api_version": "v1"
}
}
```
**Erori Posibile:**
```json
// 400 - Domain invalid
{
"success": false,
"error": {
"code": "INVALID_REQUEST",
"message": "Domain parameter is required"
}
}
// 500 - Eroare internă
{
"success": false,
"error": {
"code": "INTERNAL_ERROR",
"message": "An error occurred while checking the domain: ..."
}
}
```
---
### ⏸️ 4. **Get Domain Details** - Detalii Domeniu
**Endpoint:** `GET /api/v1/domain/{domain}`
**Funcțional:** NU (stub - returnează 501)
**Parametri:** `domain` (string în URL)
**Test:**
```bash
curl http://localhost:5000/api/v1/domain/google.com
```
**Răspuns actual:**
```json
{
"message": "Not yet implemented",
"domain": "google.com"
}
```
---
### ⏸️ 5. **Batch Check** - Verificare Multiplă
**Endpoint:** `POST /api/v1/check/batch`
**Funcțional:** NU (stub - returnează 501)
**Test:**
```bash
curl -X POST http://localhost:5000/api/v1/check/batch \
-H "Content-Type: application/json" \
-d '{
"domains": ["google.com", "facebook.com", "twitter.com"]
}'
```
---
### ⏸️ 6. **Batch Status** - Status Verificare Multiplă
**Endpoint:** `GET /api/v1/batch/{batch_id}/status`
**Funcțional:** NU (stub)
---
### ⏸️ 7. **Statistics** - Statistici Sistem
**Endpoint:** `GET /api/v1/stats`
**Funcțional:** NU (stub)
**Test:**
```bash
curl http://localhost:5000/api/v1/stats
```
---
### ⏸️ 8. **Search Domains** - Căutare Domenii
**Endpoint:** `GET /api/v1/search`
**Funcțional:** NU (stub)
**Test:**
```bash
curl "http://localhost:5000/api/v1/search?q=google&risk_level=LOW"
```
---
## 📚 SWAGGER UI - Documentație Interactivă
**URL:** http://localhost:5000/docs
Aici poți:
- Vedea toate endpoint-urile
- Testa direct din browser
- Vedea request/response examples
- Download OpenAPI spec
---
## 🔍 VERIFICARE DATE ÎN DATABASE
### Verifică toate domeniile
```bash
docker exec dns_postgres psql -U dns_admin -d domain_check \
-c "SELECT full_domain, check_count, last_checked_at FROM domains ORDER BY last_checked_at DESC LIMIT 10;"
```
### Verifică ultimele verificări
```bash
docker exec dns_postgres psql -U dns_admin -d domain_check \
-c "SELECT d.full_domain, ra.total_score, ra.risk_level, ra.assessed_at
FROM risk_assessments ra
JOIN domains d ON ra.domain_id = d.id
ORDER BY ra.assessed_at DESC LIMIT 10;"
```
### Verifică date WHOIS
```bash
docker exec dns_postgres psql -U dns_admin -d domain_check \
-c "SELECT d.full_domain, w.creation_date, w.registrar, w.data_source
FROM whois_records w
JOIN domains d ON w.domain_id = d.id
ORDER BY w.fetched_at DESC LIMIT 10;"
```
### Export JSON complet
```bash
docker exec dns_postgres psql -U dns_admin -d domain_check \
-c "SELECT row_to_json(d) FROM domains d LIMIT 5;" \
| python3 -m json.tool
```
---
## 🔧 SERVICII EXTERNE FOLOSITE
### 1. **python-whois Library**
- **Cost:** FREE, unlimited
- **Folosit pentru:** WHOIS lookups primary
- **Limitări:** Rate limiting de la registrars individuali
- **Status:** ✅ ACTIV
### 2. **Whoxy API** (Fallback)
- **API Key:** `876528325417e0bgs418d2cc7d8f193a6`
- **Cost:** FREE - 250,000 requests/lună
- **Endpoint:** `https://api.whoxy.com/`
- **Folosit pentru:** WHOIS când python-whois eșuează
- **Status:** ✅ CONFIGURAT (nu e folosit încă ca fallback automat)
**Test Direct Whoxy:**
```bash
curl "https://api.whoxy.com/?key=876528325417e0bgs418d2cc7d8f193a6&whois=example.com"
```
### 3. **VirusTotal API** (Pregătit)
- **API Key:** NU este configurat
- **Cost:** 500 requests/day FREE
- **Status:** ⏸️ Pregătit în cod, nefuncțional
---
## 📊 RISK SCORING - Algoritmul Exact
### Formula Finală:
```
TOTAL_SCORE = (
domain_age_score × 0.30 +
reputation_score × 0.30 +
ssl_score × 0.20 +
dns_score × 0.10 +
whois_score × 0.10
)
```
### 1. Domain Age Score (30% weight)
| Vârsta Domini | Score | Risk Level | Reason |
|---------------|-------|------------|--------|
| 0-90 zile | 100 | CRITICAL | Very new domain (<3 months) |
| 91-180 zile | 80 | HIGH | New domain (<6 months) |
| 181-365 zile | 50 | MEDIUM | Moderately new |
| 366-730 zile | 30 | LOW | Established (1-2 years) |
| 731+ zile | 0 | LOW | Trusted age (2+ years) |
| No data | 20 | LOW | No creation date |
### 2. Reputation Score (30% weight)
| Condiție | Score | Weight |
|----------|-------|--------|
| Blacklisted | +100 | Critical |
| VirusTotal malicious > 5 | +80 | High |
| VirusTotal suspicious > 10 | +50 | Medium |
| Typosquatting detected | +70 | High |
| No reputation data | +20 | Low |
| Clean reputation | 0 | None |
### 3. SSL Score (20% weight)
| Condiție | Score |
|----------|-------|
| No SSL certificate | +100 |
| Self-signed certificate | +80 |
| Expired certificate | +100 |
| Certificate < 30 days old | +40 |
| Expires < 30 days | +30 |
| Valid certificate | 0 |
### 4. DNS Score (10% weight)
| Condiție | Score |
|----------|-------|
| No MX records | +30 |
| No TXT records (SPF/DKIM) | +20 |
| Suspicious NS records | +40 |
| Recently changed NS | +50 |
| Complete DNS config | 0 |
### 5. WHOIS Score (10% weight)
| Condiție | Score |
|----------|-------|
| WHOIS privacy protection | +30 |
| No registrant org | +20 |
| Country mismatch | +20 |
| Known abusive registrar | +50 |
| Transparent WHOIS | 0 |
### Risk Level Classification:
```python
if total_score <= 30: return "LOW"
elif total_score <= 60: return "MEDIUM"
elif total_score <= 85: return "HIGH"
else: return "CRITICAL"
```
---
## 🎯 CE FUNCȚIONEAZĂ vs CE NU
### ✅ FUNCȚIONAL ACUM:
1. ✅ Health check endpoint
2. ✅ Domain check cu WHOIS
3. ✅ Risk scoring (domain age + whois factors)
4. ✅ Database storage (domains, whois_records, risk_assessments, check_history)
5. ✅ Swagger UI documentation
6. ✅ Docker Compose infrastructure
7. ✅ PostgreSQL cu toate tabelele
8. ✅ Error handling și logging
### ⏸️ PREGĂTIT DAR NEFUNCȚIONAL:
1. ⏸️ DNS checking
2. ⏸️ SSL certificate validation
3. ⏸️ VirusTotal reputation
4. ⏸️ Redis caching
5. ⏸️ Batch processing
6. ⏸️ Domain history endpoint
7. ⏸️ Search endpoint
8. ⏸️ Statistics endpoint
### ❌ NEPREGĂTIT:
1. ❌ Frontend dashboard
2. ❌ Email validation
3. ❌ Subdomain enumeration
4. ❌ IP geolocation
5. ❌ Traffic estimation
6. ❌ Webhook notifications
---
## 🧪 SCRIPT DE TESTARE COMPLETĂ
Salvează ca `test-all.sh`:
```bash
#!/bin/bash
echo "=== Testing Domain Check API ==="
echo ""
# Test 1: Health
echo "1. Health Check:"
curl -s http://localhost:5000/health | python3 -m json.tool
echo ""
# Test 2: Root
echo "2. API Root:"
curl -s http://localhost:5000/ | python3 -m json.tool
echo ""
# Test 3: Check google.com
echo "3. Check google.com:"
curl -s -X POST http://localhost:5000/api/v1/check \
-H "Content-Type: application/json" \
-d '{"domain": "google.com"}' \
| python3 -m json.tool
echo ""
# Test 4: Check facebook.com
echo "4. Check facebook.com:"
curl -s -X POST http://localhost:5000/api/v1/check \
-H "Content-Type: application/json" \
-d '{"domain": "facebook.com"}' \
| python3 -m json.tool
echo ""
echo "=== Tests Complete ==="
```
Rulează:
```bash
chmod +x test-all.sh
./test-all.sh
```
---
**Ultima actualizare:** 2026-01-29 15:50:00

452
ai_platform/modules/domain_check/NEXT_STEPS.md Normal file
View file

@ -0,0 +1,452 @@
# NEXT STEPS - Plan de Dezvoltare
## 🎯 PRIORITĂȚI DEZVOLTARE
### PHASE 1: Complete Backend Features (URGENT) ⚡
**Estimare: 2-3 ore**
1. ✅ **DNS Checking Service** - ESENȚIAL pentru risk scoring
2. ✅ **SSL Certificate Validation** - ESENȚIAL pentru risk scoring
3. ✅ **Whoxy API Fallback** - Activare automată când python-whois eșuează
4. ✅ **Implementare Endpoints:**
- `GET /api/v1/domain/{domain}` - Detalii + istoric
- `GET /api/v1/stats` - Statistici sistem
- `GET /api/v1/search` - Căutare domenii
5. ✅ **Redis Caching** - Activare completă
---
### PHASE 2: Modern Dashboard (NEXT.JS 15) 🚀
**Estimare: 4-6 ore**
#### Stack Tehnologic 2026:
**Frontend:**
- **Next.js 15** (App Router, Server Components, Server Actions)
- **React 19** (Concurrent features, Suspense)
- **TypeScript** (Type safety)
- **Tailwind CSS** + **shadcn/ui** (Modern UI components)
- **TanStack Query (React Query v5)** - Data fetching & caching
- **Zustand** - State management (lightweight)
- **Recharts** sau **Tremor** - Data visualization
- **Framer Motion** - Animations
**Features Dashboard:**
1. **🏠 Home Page**
- Hero section cu search bar central
- Live stats (total checks, domains analyzed)
- Recent high-risk domains feed
- Trending searches
2. **🔍 Domain Check Page**
- Input field cu autocomplete
- Real-time validation
- Loading states cu skeleton
- Result cards cu:
- Risk score gauge (circular progress)
- WHOIS data table
- DNS records expandable
- SSL certificate timeline
- Historical checks graph
- Export options (PDF, JSON)
3. **📊 Analytics Dashboard**
- Risk distribution pie chart
- Daily checks timeline
- Top risky domains table
- Geographic distribution map (registrant countries)
- Registrar statistics
- Average processing time metrics
4. **📜 History Browser**
- Filterable table (domain, risk level, date)
- Pagination
- Quick re-check button
- Comparison mode (2 domains side-by-side)
5. **⚙️ Settings**
- API configuration
- Threshold customization
- Export preferences
- Dark/Light mode toggle
6. **📖 Documentation**
- API reference (embedded Swagger)
- Risk scoring explanation
- Integration examples
- FAQ
---
### PHASE 3: Advanced Features 🎯
**Estimare: 6-8 ore**
1. **Batch Processing** - Verificare 100+ domenii simultan
2. **VirusTotal Integration** - Reputation checking
3. **Real-time Webhooks** - Notificări pentru high-risk
4. **Email Validation** - Verificare email addresses
5. **Subdomain Enumeration** - Discover subdomains
6. **IP Geolocation** - Hartă interactivă
---
## 📁 STRUCTURA PROIECT DASHBOARD
```
domain-check/
├── frontend/ # Next.js App
│ ├── app/
│ │ ├── (dashboard)/
│ │ │ ├── layout.tsx
│ │ │ ├── page.tsx # Home
│ │ │ ├── check/
│ │ │ │ └── page.tsx # Domain Check
│ │ │ ├── analytics/
│ │ │ │ └── page.tsx # Analytics
│ │ │ ├── history/
│ │ │ │ └── page.tsx # History
│ │ │ └── settings/
│ │ │ └── page.tsx # Settings
│ │ ├── api/ # API Routes (Next.js)
│ │ │ └── proxy/
│ │ │ └── [...path]/route.ts # Proxy to Flask
│ │ ├── layout.tsx # Root layout
│ │ └── page.tsx # Landing page
│ ├── components/
│ │ ├── ui/ # shadcn components
│ │ ├── domain/
│ │ │ ├── DomainSearchBar.tsx
│ │ │ ├── RiskScoreGauge.tsx
│ │ │ ├── WhoisDataCard.tsx
│ │ │ └── DNSRecordsTable.tsx
│ │ ├── charts/
│ │ │ ├── RiskDistribution.tsx
│ │ │ └── TimelineChart.tsx
│ │ └── layout/
│ │ ├── Navbar.tsx
│ │ ├── Sidebar.tsx
│ │ └── Footer.tsx
│ ├── lib/
│ │ ├── api.ts # API client
│ │ ├── types.ts # TypeScript types
│ │ └── utils.ts # Helper functions
│ ├── hooks/
│ │ ├── useDomainCheck.ts
│ │ ├── useStats.ts
│ │ └── useHistory.ts
│ ├── store/
│ │ └── store.ts # Zustand store
│ ├── public/
│ ├── tailwind.config.ts
│ ├── next.config.mjs
│ ├── package.json
│ └── tsconfig.json
├── api/ # Flask Backend (existing)
└── docker-compose.yml # Add frontend service
```
---
## 🚀 COMENZI DEZVOLTARE
### Setup Frontend
```bash
cd /home/admin365/domain-check
# Create Next.js app
npx create-next-app@latest frontend \
--typescript \
--tailwind \
--app \
--src-dir false \
--import-alias "@/*"
cd frontend
# Install dependencies
npm install @tanstack/react-query zustand
npm install recharts date-fns lucide-react
npm install @radix-ui/react-dialog @radix-ui/react-dropdown-menu
npm install class-variance-authority clsx tailwind-merge
# Install shadcn/ui
npx shadcn-ui@latest init
npx shadcn-ui@latest add button card input table badge progress
npx shadcn-ui@latest add dialog dropdown-menu tooltip
# Development
npm run dev
```
### Docker Integration
Update `docker-compose.yml`:
```yaml
dns_frontend:
build:
context: ./frontend
dockerfile: Dockerfile
container_name: dns_frontend
networks:
- dns-network
environment:
- NEXT_PUBLIC_API_URL=http://dns_api:5000
volumes:
- ./frontend:/app
- /app/node_modules
- /app/.next
ports:
- "3000:3000"
depends_on:
- dns_api
restart: unless-stopped
command: npm run dev
```
---
## 🎨 UI/UX DESIGN GUIDELINES
### Color Scheme
```css
/* Risk Levels */
--risk-low: #10b981 /* Green */
--risk-medium: #f59e0b /* Amber */
--risk-high: #ef4444 /* Red */
--risk-critical: #dc2626 /* Dark Red */
/* Brand */
--primary: #3b82f6 /* Blue */
--secondary: #8b5cf6 /* Purple */
```
### Components Style
- **Modern:** Rounded corners (radius-lg)
- **Clean:** Generous whitespace
- **Responsive:** Mobile-first design
- **Accessible:** WCAG 2.1 AA compliant
- **Fast:** Optimistic UI updates
- **Smooth:** 60fps animations
---
## 📊 EXEMPLE COMPONENTE
### 1. Risk Score Gauge
```tsx
import { Progress } from "@/components/ui/progress"
export function RiskScoreGauge({ score, level }: Props) {
const color = {
LOW: "text-green-500",
MEDIUM: "text-amber-500",
HIGH: "text-red-500",
CRITICAL: "text-red-700"
}[level]
return (
<div className="space-y-2">
<div className="flex justify-between">
<span className="text-sm font-medium">Risk Score</span>
<span className={`text-2xl font-bold ${color}`}>{score}/100</span>
</div>
<Progress value={score} className="h-2" />
<span className={`text-xs font-semibold ${color}`}>{level}</span>
</div>
)
}
```
### 2. Domain Search Bar
```tsx
"use client"
import { useState } from "react"
import { useMutation } from "@tanstack/react-query"
import { Input } from "@/components/ui/input"
import { Button } from "@/components/ui/button"
export function DomainSearchBar() {
const [domain, setDomain] = useState("")
const checkMutation = useMutation({
mutationFn: (domain: string) =>
fetch("/api/v1/check", {
method: "POST",
body: JSON.stringify({ domain })
}).then(r => r.json())
})
return (
<div className="flex gap-2">
<Input
placeholder="Enter domain (e.g., example.com)"
value={domain}
onChange={(e) => setDomain(e.target.value)}
/>
<Button
onClick={() => checkMutation.mutate(domain)}
disabled={checkMutation.isPending}
>
{checkMutation.isPending ? "Checking..." : "Check Domain"}
</Button>
</div>
)
}
```
---
## 🔄 API CLIENT (TypeScript)
```typescript
// lib/api.ts
export interface DomainCheckRequest {
domain: string
check_options?: {
whois?: boolean
dns?: boolean
ssl?: boolean
reputation?: boolean
force_refresh?: boolean
}
}
export interface RiskScore {
total: number
level: "LOW" | "MEDIUM" | "HIGH" | "CRITICAL"
factors: Array<{
factor: string
score: number
weight: number
reason: string
}>
}
export interface DomainCheckResponse {
success: boolean
data: {
domain: string
check_id: string
timestamp: string
whois: any
dns: any
ssl: any
reputation: any
risk_score: RiskScore
}
metadata: {
cached: boolean
processing_time_ms: number
api_version: string
}
}
export class DomainCheckAPI {
private baseURL: string
constructor(baseURL: string = "http://localhost:5000") {
this.baseURL = baseURL
}
async checkDomain(request: DomainCheckRequest): Promise<DomainCheckResponse> {
const response = await fetch(`${this.baseURL}/api/v1/check`, {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify(request)
})
return response.json()
}
async getDomainDetails(domain: string) {
const response = await fetch(`${this.baseURL}/api/v1/domain/${domain}`)
return response.json()
}
async getStats() {
const response = await fetch(`${this.baseURL}/api/v1/stats`)
return response.json()
}
async searchDomains(params: {
q?: string
risk_level?: string
page?: number
}) {
const query = new URLSearchParams(params as any)
const response = await fetch(`${this.baseURL}/api/v1/search?${query}`)
return response.json()
}
}
export const api = new DomainCheckAPI()
```
---
## 📋 CHECKLIST IMPLEMENTARE
### Backend (Priority 1)
- [ ] Implementează DNS checking service
- [ ] Implementează SSL certificate validation
- [ ] Activează Whoxy API fallback
- [ ] Implementează endpoint `GET /api/v1/domain/{domain}`
- [ ] Implementează endpoint `GET /api/v1/stats`
- [ ] Implementează endpoint `GET /api/v1/search`
- [ ] Activează Redis caching complet
- [ ] Testează toate endpoint-urile
### Frontend Setup (Priority 2)
- [ ] Create Next.js app cu TypeScript
- [ ] Setup Tailwind CSS + shadcn/ui
- [ ] Configure TanStack Query
- [ ] Setup Zustand store
- [ ] Create API client library
- [ ] Setup Docker integration
### Frontend Components (Priority 3)
- [ ] Layout components (Navbar, Sidebar)
- [ ] Domain Search Bar
- [ ] Risk Score Gauge
- [ ] WHOIS Data Card
- [ ] DNS Records Table
- [ ] SSL Certificate Timeline
- [ ] Charts (Pie, Line, Bar)
- [ ] History Table
- [ ] Stats Dashboard
### Integration (Priority 4)
- [ ] Connect frontend cu backend API
- [ ] Implement error handling
- [ ] Add loading states
- [ ] Add toast notifications
- [ ] Test end-to-end flow
---
## 🎯 NEXT IMMEDIATE ACTIONS
**CE VREI SĂ FAC ACUM?**
### Option A: Complete Backend First (RECOMANDAT)
✅ Implementez DNS + SSL + endpoint-uri rămase
✅ Backend 100% funcțional
⏱️ Estimare: 2-3 ore
### Option B: Start Frontend Direct
🚀 Creez Next.js app cu structura completă
🎨 UI modern cu components
⏱️ Estimare: 4-6 ore
### Option C: Both in Parallel
⚡ DNS + SSL în background
🎨 Next.js setup simultan
⏱️ Estimare: 4-5 ore
**CE ALEGI? (A, B sau C)**
---
**Document creat:** 2026-01-29 16:00:00

740
ai_platform/modules/domain_check/README.md Normal file
View file

@ -0,0 +1,740 @@
# Domain Check API - Anti-Fake News Platform
Sistema de verificare domenii pentru detectarea dezinformarii prin analiza vechimii domeniilor si riscurilor asociate.
## 🚀 Quick Start - API Usage
### Endpoints Disponibile
**Base URL:** `http://domain-check-api:11000`
**Endpoint principal:** `POST /api/v1/check/check`
### Apel Complet - Toate Datele Posibile
Pentru a obține **toate datele disponibile** despre un domeniu:
```bash
curl -X POST "http://domain-check-api:11000/api/v1/check/check" \
-H "Content-Type: application/json" \
-d '{
"domain": "example.com",
"check_options": {
"whois": true,
"dns": true,
"ssl": true,
"ip_intelligence": true,
"http_analysis": true,
"blacklist": true,
"port_scan": true,
"subdomains": true,
"force_refresh": false
}
}'
```
**Răspuns:** JSON cu:
- `whois` - Date WHOIS/RDAP complete (vârstă domeniu, registrar, name servers, DNSSEC)
- `dns` - Toate recordurile DNS (A, AAAA, MX, TXT, NS, CNAME, SOA)
- `ssl` - Certificate SSL/TLS details (issuer, validity, chain, self-signed status)
- `ip_intelligence` - Geolocation, ASN, ISP, reverse DNS pentru toate IP-urile
- `http_analysis` - HTTP headers, security headers, redirects, technologies detected
- `blacklist` - Verificare în DNSBL (spam, malware, phishing blacklists)
- `port_scan` - Port scanning TCP common (22, 80, 443, 8080, etc.)
- `subdomains` - Subdomain enumeration (www, mail, ftp, api, etc.)
- `risk_score` - Scor de risc calculat (0-100) cu breakdown pe categorii
### Apel Rapid - Date Esențiale
Pentru verificări rapide (doar WHOIS + DNS + SSL):
```bash
curl -X POST "http://domain-check-api:11000/api/v1/check/check" \
-H "Content-Type: application/json" \
-d '{
"domain": "example.com",
"check_options": {
"whois": true,
"dns": true,
"ssl": true
}
}'
```
### Health Check
```bash
curl http://domain-check-api:11000/health
```
Răspuns:
```json
{
"status": "healthy",
"database": "connected",
"redis": "connected",
"environment": "production",
"version": "v1"
}
```
---
## 🌐 Integrare ca Serviciu (Kong API Gateway / Microservices)
### Configurare ca Backend Service
API-ul poate fi folosit ca **backend service** pentru Kong, NGINX, Traefik sau alte API gateways.
#### 1. Configurare Kong Gateway
```bash
# Adaugă service
curl -X POST http://kong-admin:8001/services \
--data name=domain-check-api \
--data url=http://domain-check-api:11000
# Adaugă route
curl -X POST http://kong-admin:8001/services/domain-check-api/routes \
--data paths[]=/domain-check \
--data strip_path=false
# Activează rate limiting
curl -X POST http://kong-admin:8001/services/domain-check-api/plugins \
--data name=rate-limiting \
--data config.minute=100 \
--data config.hour=1000
```
Acum API-ul e disponibil la: `http://your-kong-gateway/domain-check/api/v1/check/check`
#### 2. Configurare NGINX Reverse Proxy
```nginx
upstream domain_check_backend {
server domain-check-api:11000;
}
server {
listen 80;
server_name api.example.com;
location /domain-check/ {
proxy_pass http://domain_check_backend/;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
# Timeouts pentru operații lungi (port scan, subdomain enum)
proxy_read_timeout 300s;
proxy_connect_timeout 10s;
}
}
```
#### 3. Docker Swarm / Kubernetes Service
**Docker Swarm:**
```yaml
version: '3.8'
services:
domain-check:
image: didiai-domain-check:latest
deploy:
replicas: 3
update_config:
parallelism: 1
delay: 10s
environment:
DATABASE_URL: postgresql://user:pass@db-cluster:5432/domain_check
REDIS_URL: redis://redis-cluster:6379/0
networks:
- backend
```
**Kubernetes Service:**
```yaml
apiVersion: v1
kind: Service
metadata:
name: domain-check-api
spec:
selector:
app: domain-check
ports:
- protocol: TCP
port: 80
targetPort: 51000
type: ClusterIP
```
#### 4. Environment Variables pentru External DB
Toate parametrii DB sunt în `.env` și pot fi schimbați pentru a folosi un **cluster PostgreSQL extern**:
```bash
# .env - Configurare pentru DB Cluster extern
DB_HOST=postgres-cluster.example.com
DB_PORT=5432
DB_USER=domain_check_user
DB_PASSWORD=secure_password_here
DB_NAME=domain_check
# Redis Cluster extern
REDIS_HOST=redis-cluster.example.com
REDIS_PORT=6379
REDIS_PASSWORD=redis_password_here
```
API-ul se va conecta automat la DB-ul extern specificat în variabilele de mediu.
---
## 📊 Database Schema - Structura Tabelelor
API-ul folosește PostgreSQL cu următoarele tabele:
### 1. `domains` - Domenii Verificate
Tabela principală cu toate domeniile procesate.
| Coloană | Tip | Descriere |
|---------|-----|-----------|
| `id` | UUID | Primary key |
| `domain` | VARCHAR(255) | Domeniul principal (ex: `example` din `example.com`) |
| `subdomain` | VARCHAR(255) | Subdomenul (ex: `www` din `www.example.com`) |
| `tld` | VARCHAR(50) | Top Level Domain (ex: `com`, `org`, `ro`) |
| `full_domain` | VARCHAR(255) | Domeniul complet (UNIQUE) |
| `first_seen_at` | TIMESTAMP | Prima verificare |
| `last_checked_at` | TIMESTAMP | Ultima verificare |
| `check_count` | INTEGER | Număr total verificări |
| `is_active` | BOOLEAN | Status activ/inactiv |
| `created_at` | TIMESTAMP | Data creării înregistrării |
| `updated_at` | TIMESTAMP | Data ultimei actualizări |
**Indexes:** `full_domain` (UNIQUE), `domain`, `tld`, `last_checked_at`, `is_active`
### 2. `whois_records` - Date WHOIS/RDAP
Stochează informații WHOIS pentru fiecare domeniu.
| Coloană | Tip | Descriere |
|---------|-----|-----------|
| `id` | UUID | Primary key |
| `domain_id` | UUID | Foreign key → `domains(id)` |
| `creation_date` | TIMESTAMP | Data creării domeniului |
| `expiration_date` | TIMESTAMP | Data expirării |
| `updated_date` | TIMESTAMP | Ultima actualizare WHOIS |
| `registrar` | VARCHAR(255) | Registrar (ex: GoDaddy, Namecheap) |
| `registrar_url` | VARCHAR(500) | URL registrar |
| `registrant_org` | VARCHAR(255) | Organizația proprietarului |
| `registrant_country` | VARCHAR(2) | Țara (cod ISO) |
| `admin_email` | VARCHAR(255) | Email administrator |
| `name_servers` | TEXT[] | Array cu name servers |
| `status` | TEXT[] | Array cu statusuri domeniu |
| `dnssec` | BOOLEAN | DNSSEC activat/nu |
| `raw_whois_data` | JSONB | Date WHOIS raw (JSON) |
| `data_source` | VARCHAR(50) | Sursa datelor (`rdap`, `whoxy`, `manual`) |
| `fetched_at` | TIMESTAMP | Când au fost preluate datele |
**Indexes:** `domain_id`, `creation_date`, `registrar`, `data_source`, `fetched_at`
### 3. `dns_records` - Înregistrări DNS
Toate recordurile DNS pentru fiecare domeniu.
| Coloană | Tip | Descriere |
|---------|-----|-----------|
| `id` | UUID | Primary key |
| `domain_id` | UUID | Foreign key → `domains(id)` |
| `record_type` | VARCHAR(10) | Tip record (`A`, `AAAA`, `MX`, `TXT`, `NS`, `CNAME`, `SOA`) |
| `record_value` | TEXT | Valoarea recordului |
| `ttl` | INTEGER | Time To Live (secunde) |
| `priority` | INTEGER | Prioritate (pentru MX records) |
| `fetched_at` | TIMESTAMP | Când a fost preluat recordul |
**Indexes:** `domain_id`, `(domain_id, record_type)`, `record_type`, `fetched_at`
### 4. `ssl_certificates` - Certificate SSL/TLS
Informații despre certificatele SSL ale domeniilor.
| Coloană | Tip | Descriere |
|---------|-----|-----------|
| `id` | UUID | Primary key |
| `domain_id` | UUID | Foreign key → `domains(id)` |
| `issuer` | VARCHAR(255) | Emitent certificat (ex: Let's Encrypt) |
| `subject` | VARCHAR(255) | Subiect certificat |
| `valid_from` | TIMESTAMP | Valabil de la |
| `valid_until` | TIMESTAMP | Valabil până la |
| `serial_number` | VARCHAR(255) | Număr serial certificat |
| `signature_algorithm` | VARCHAR(100) | Algoritm semnătură (ex: SHA256) |
| `key_size` | INTEGER | Mărime cheie (2048, 4096 bits) |
| `is_wildcard` | BOOLEAN | Certificat wildcard (`*.example.com`) |
| `is_self_signed` | BOOLEAN | Self-signed certificate |
| `is_valid` | BOOLEAN | Certificat valid |
| `certificate_chain` | JSONB | Lanțul complet de certificate (JSON) |
| `fetched_at` | TIMESTAMP | Când a fost verificat certificatul |
**Indexes:** `domain_id`, `valid_until`, `is_valid`, `fetched_at`
### Relații între Tabele
```
domains (1) ─── (N) whois_records
├─ (N) dns_records
└─ (N) ssl_certificates
```
Toate relațiile au `ON DELETE CASCADE` - ștergerea unui domeniu șterge automat toate datele asociate.
### Migrare la DB Cluster Extern
Pentru a crea schema în **cluster PostgreSQL nou**:
```bash
# 1. Export schema
docker exec domain_check_postgres pg_dump -U dns_admin -d domain_check --schema-only > schema.sql
# 2. Import în cluster nou
psql -h postgres-cluster.example.com -U admin -d domain_check < schema.sql
# 3. Actualizează .env cu noile credentials
DB_HOST=postgres-cluster.example.com
DB_PORT=5432
DB_USER=domain_check_user
DB_PASSWORD=new_secure_password
DB_NAME=domain_check
# 4. Restart API
docker compose restart domain_check_api
```
Sau folosește direct script-ul de inițializare:
```bash
psql -h postgres-cluster.example.com -U admin -d domain_check < init-scripts/01-init-db.sql
```
---
## Caracteristici
- Verificare WHOIS/RDAP complet
- Integrare Whoxy API
- DNS records checking (A, AAAA, MX, TXT, NS, CNAME, SOA)
- SSL certificate validation
- IP Intelligence (geolocation, ASN, reverse DNS)
- HTTP Security Analysis
- DNSBL Blacklist checking
- Port scanning
- Subdomain enumeration
- Risk scoring engine avansat
- PostgreSQL + Redis caching
- Swagger UI + ReDoc documentation
- Docker Compose setup complet
- GitLab CI/CD pipeline cu SAST
---
## Deployment pe Orice Server
### Metoda 1: Script Automat (Recomandat)
```bash
# Descarca si ruleaza scriptul de deployment
curl -sSL (modul domain_check al platformei DIDI) -o deploy.sh
chmod +x deploy.sh
./deploy.sh install
```
Scriptul va:
- Verifica prerequisitele (Docker, Docker Compose, Git)
- Clona repository-ul
- Genera fisierul `.env` cu parole securizate
- Construi si porni containerele
### Metoda 2: Manual
```bash
# 1. Cloneaza repository-ul
git clone <didi-lot1-ai>/ai_platform/modules/domain_check
cd domain-check
# 2. Copiaza si editeaza configuratia
cp .env.example .env
nano .env # sau vim, sau orice editor
# 3. Porneste serviciile
docker compose up -d --build
# 4. Verifica statusul
docker compose ps
curl http://domain-check-api:11000/health
```
### Metoda 3: Din GitLab Container Registry
```bash
# Login la registry (daca e necesar)
docker login <registry-didi>
# Pull imaginea
docker pull didiai-domain-check:latest
# Cloneaza doar fisierele de configurare
git clone <didi-lot1-ai>/ai_platform/modules/domain_check
cd domain-check
# Editeaza .env
nano .env
# Porneste cu imaginea din registry
docker compose up -d
```
---
## Configurare pentru Serverul Tau
### Fisierul .env
Toate setarile sunt in `.env`. Editeaza pentru mediul tau:
```bash
# ===========================================
# PORTURI - Modifica dupa nevoie
# ===========================================
# Schema: Dev=5xxxx, Prod=1xxxx
# x1xxx = API/Gateway
# x20xx = PostgreSQL
# x23xx = Redis
API_PORT=51000 # Portul API-ului (schimba la 11000 pentru prod)
DB_EXTERNAL_PORT=52000 # Port extern PostgreSQL (12000 pentru prod)
REDIS_EXTERNAL_PORT=52300 # Port extern Redis (12300 pentru prod)
# ===========================================
# DATABASE - Genereaza parole noi!
# ===========================================
DB_PASSWORD=SCHIMBA_ACEASTA_PAROLA
DB_USER=dns_admin
DB_NAME=domain_check
# ===========================================
# API KEYS - Adauga cheile tale
# ===========================================
WHOXY_API_KEY=cheia_ta_whoxy
VIRUSTOTAL_API_KEY=cheia_ta_virustotal
# ===========================================
# SECURITATE - OBLIGATORIU pentru productie!
# ===========================================
SECRET_KEY=genereaza_un_string_random_de_32_caractere
FLASK_ENV=production # development sau production
DEBUG=False # True doar pentru development
```
### Setare Hostname/DNS
API-ul va fi accesibil prin:
- **IP direct**: `http://<IP_SERVER>:<API_PORT>/`
- **Hostname**: `http://<hostname>:<API_PORT>/`
Pentru a seta un hostname custom:
```bash
# 1. Pe DNS server (daca ai control)
check-dns.example.com A <IP_SERVER>
# 2. Sau in /etc/hosts pe clienti
echo "<IP_SERVER> check-dns.example.com" >> /etc/hosts
# 3. Sau cu reverse proxy (nginx/traefik)
# - Configureaza proxy pass catre domain-check-api:<API_PORT>
```
### Porturi Utilizate
| Serviciu | Port Default (Dev) | Port Productie | Variabila |
|----------|-------------------|----------------|-----------|
| API Flask | 51000 | 11000 | `API_PORT` |
| PostgreSQL | 52000 | 12000 | `DB_EXTERNAL_PORT` |
| Redis | 52300 | 12300 | `REDIS_EXTERNAL_PORT` |
Porturile pot fi schimbate in `.env` - nu este nevoie de modificari in alte fisiere.
### Firewall
Deschide doar portul API pentru acces extern:
```bash
# UFW (Ubuntu/Debian)
sudo ufw allow 51000/tcp
# firewalld (CentOS/RHEL)
sudo firewall-cmd --add-port=51000/tcp --permanent
sudo firewall-cmd --reload
# iptables
sudo iptables -A INPUT -p tcp --dport 51000 -j ACCEPT
```
PostgreSQL si Redis nu trebuie expuse public (sunt doar pentru comunicatie interna).
---
## Structura Proiectului
```
domain-check/
├── api/ # Flask API
│ ├── app/
│ │ ├── models/ # SQLAlchemy models
│ │ ├── routes/ # API endpoints
│ │ ├── services/ # Business logic (WHOIS, DNS, SSL, etc.)
│ │ ├── utils/ # Helper functions
│ │ ├── tasks/ # Celery tasks
│ │ ├── static/ # Dashboard HTML/CSS/JS
│ │ ├── config.py # Configuration
│ │ └── __init__.py # App factory
│ ├── Dockerfile
│ ├── requirements.txt
│ └── run.py # Entry point
├── init-scripts/ # PostgreSQL init scripts
│ └── 01-init-db.sql
├── docker-compose.yml # Docker orchestration
├── .gitlab-ci.yml # CI/CD pipeline
├── deploy.sh # Deployment script
├── .env # Environment variables
├── use_api.md # API documentation
└── README.md # This file
```
---
## Servicii Docker
| Container | Descriere | Health Check |
|-----------|-----------|--------------|
| `domain_check_api` | Flask REST API + Dashboard | `/health` |
| `domain_check_postgres` | PostgreSQL 15 database | `pg_isready` |
| `domain_check_redis` | Redis cache | `redis-cli ping` |
| `domain_check_worker` | Celery background worker | `celery inspect ping` |
---
## Utilizare API
### Interfata Web (Dashboard)
Deschide in browser: `http://<server>:<API_PORT>/`
### Verificare Domain (curl)
```bash
# Check complet
curl -X POST "http://domain-check-api:11000/api/v1/check/check" \
-H "Content-Type: application/json" \
-d '{
"domain": "example.com",
"check_options": {
"whois": true,
"dns": true,
"ssl": true,
"ip_intelligence": true,
"http_analysis": true,
"blacklist": true,
"port_scan": true,
"subdomains": true
}
}'
```
### Health Check
```bash
curl http://domain-check-api:11000/health
```
### Documentatie Swagger
- Swagger UI: `http://<server>:<API_PORT>/docs`
- ReDoc: `http://<server>:<API_PORT>/redoc`
Pentru documentatie API completa, vezi [use_api.md](use_api.md).
---
## CI/CD Pipeline
Pipeline-ul GitLab include:
### Stages
1. **lint** - Code quality (flake8, black, isort, hadolint)
2. **security** - SAST scanning (bandit, safety, trivy)
3. **test** - Unit tests
4. **build** - Docker build & push to registry
5. **release** - Create deployment bundle
6. **deploy** - Deploy to server via SSH
### Container Registry
Imaginile sunt publicate la:
```
didiai-domain-check:latest
didiai-domain-check:<branch>
```
### Configurare Deploy Automat
Adauga in GitLab CI/CD Variables:
| Variable | Descriere |
|----------|-----------|
| `DEPLOY_HOST` | IP sau hostname server |
| `DEPLOY_USER` | User SSH (ex: admin365) |
| `SSH_PRIVATE_KEY` | Cheia privata SSH |
| `SSH_KNOWN_HOSTS` | Output din `ssh-keyscan <server>` |
---
## Comenzi Utile
### Deploy Script
```bash
./deploy.sh install # Instalare fresh
./deploy.sh update # Update la ultima versiune
./deploy.sh status # Verifica statusul
./deploy.sh logs # Vezi toate logs
./deploy.sh logs domain_check_api # Logs doar pentru API
./deploy.sh start # Porneste serviciile
./deploy.sh stop # Opreste serviciile
./deploy.sh restart # Restart servicii
```
### Docker Direct
```bash
# Start/Stop
docker compose up -d
docker compose down
# Rebuild
docker compose up -d --build
# Logs
docker compose logs -f domain_check_api
# Shell in container
docker exec -it domain_check_api bash
# Database access
docker exec -it domain_check_postgres psql -U dns_admin -d domain_check
```
---
## Troubleshooting
### Container nu porneste
```bash
# Verifica logs
docker compose logs domain_check_api
# Verifica network
docker network ls | grep dns-network
# Recreate containers
docker compose down && docker compose up -d --build
```
### Database connection error
```bash
# Verifica ca PostgreSQL e healthy
docker compose ps domain_check_postgres
# Verifica health
docker exec -it domain_check_postgres pg_isready -U dns_admin
# Restart database
docker compose restart domain_check_postgres
```
### API returns 500 error
```bash
# Check logs
docker compose logs -f domain_check_api
# Verify database schema
docker exec -it domain_check_postgres psql -U dns_admin -d domain_check -c "\dt"
```
### Port deja in uz
```bash
# Gaseste ce foloseste portul
sudo lsof -i :51000
# Schimba portul in .env
API_PORT=51001 # sau alt port liber
```
---
## Securitate - Checklist Productie
- [ ] Schimba toate parolele default in `.env`
- [ ] Seteaza `FLASK_ENV=production`
- [ ] Seteaza `DEBUG=False`
- [ ] Genereaza `SECRET_KEY` random (32+ caractere)
- [ ] Configureaza firewall (doar port API public)
- [ ] Setup HTTPS cu reverse proxy (nginx/traefik)
- [ ] Backup automat PostgreSQL
- [ ] Monitorizare logs
---
## Variabile de Mediu Complete
| Variabila | Default | Descriere |
|-----------|---------|-----------|
| `API_PORT` | 51000 | Port pentru Flask API |
| `API_HOST` | 0.0.0.0 | Bind address |
| `API_VERSION` | v1 | API version prefix |
| `DB_HOST` | domain_check_postgres | PostgreSQL hostname (container) |
| `DB_PORT` | 5432 | PostgreSQL port intern |
| `DB_EXTERNAL_PORT` | 52000 | PostgreSQL port extern |
| `DB_USER` | dns_admin | Database user |
| `DB_PASSWORD` | - | Database password |
| `DB_NAME` | domain_check | Database name |
| `REDIS_HOST` | domain_check_redis | Redis hostname (container) |
| `REDIS_PORT` | 6379 | Redis port intern |
| `REDIS_EXTERNAL_PORT` | 52300 | Redis port extern |
| `REDIS_DB` | 0 | Redis database number |
| `WHOXY_API_KEY` | - | Whoxy API key |
| `VIRUSTOTAL_API_KEY` | - | VirusTotal API key |
| `FLASK_ENV` | development | Flask environment |
| `DEBUG` | True | Debug mode |
| `SECRET_KEY` | - | Flask secret key |
| `LOG_LEVEL` | INFO | Logging level |
| `REDIS_TTL_HOT` | 21600 | Cache TTL hot (6h) |
| `REDIS_TTL_WARM` | 86400 | Cache TTL warm (24h) |
| `REDIS_TTL_COLD` | 604800 | Cache TTL cold (7d) |
---
## License
MIT License - Free for use in anti-fake news projects
---
**Built for fighting disinformation**
**Last Updated:** 2026-02-04

68
ai_platform/modules/domain_check/api/Dockerfile Normal file
View file

@ -0,0 +1,68 @@
# Multi-stage build for Python API
# Stage 1: Builder
FROM python:3.10-slim as builder
# Set working directory
WORKDIR /build
# Install system dependencies
RUN apt-get update && apt-get install -y --no-install-recommends \
gcc \
g++ \
make \
libpq-dev \
libssl-dev \
libffi-dev \
&& rm -rf /var/lib/apt/lists/*
# Copy requirements
COPY requirements.txt .
# Create virtual environment and install dependencies
RUN python -m venv /opt/venv
ENV PATH="/opt/venv/bin:$PATH"
RUN pip install --no-cache-dir --upgrade pip setuptools wheel && \
pip install --no-cache-dir -r requirements.txt
# Stage 2: Runtime
FROM python:3.10-slim
# Set working directory
WORKDIR /app
# Install runtime dependencies only
RUN apt-get update && apt-get install -y --no-install-recommends \
libpq5 \
curl \
&& rm -rf /var/lib/apt/lists/*
# Copy virtual environment from builder
COPY --from=builder /opt/venv /opt/venv
# Set environment variables
ENV PATH="/opt/venv/bin:$PATH" \
PYTHONUNBUFFERED=1 \
PYTHONDONTWRITEBYTECODE=1 \
FLASK_APP=run.py
# Create non-root user
RUN useradd -m -u 1000 appuser && \
mkdir -p /app/logs && \
chown -R appuser:appuser /app
# Copy application code
COPY --chown=appuser:appuser . /app/
# Switch to non-root user
USER appuser
# Expose port
EXPOSE 5000
# Health check
HEALTHCHECK --interval=30s --timeout=10s --start-period=40s --retries=3 \
CMD curl -f http://localhost:5000/health || exit 1
# Default command
CMD ["gunicorn", "--bind", "0.0.0.0:5000", "--workers", "4", "--threads", "2", "--timeout", "120", "--worker-class", "gevent", "--access-logfile", "-", "--error-logfile", "-", "run:app"]

422
ai_platform/modules/domain_check/api/app/__init__.py Normal file
View file

@ -0,0 +1,422 @@
"""
Domain Check API - Application Factory
"""
import ipaddress
import logging
import os
from flask import Flask, jsonify, request
from flask_sqlalchemy import SQLAlchemy
from flask_migrate import Migrate
from flask_cors import CORS
from flask_restx import Api
from redis import Redis
from celery import Celery
try:
from flask_limiter import Limiter
from flask_limiter.util import get_remote_address
LIMITER_AVAILABLE = True
except ImportError:
LIMITER_AVAILABLE = False
from app.config import get_config
# Initialize extensions
db = SQLAlchemy()
migrate = Migrate()
redis_client = None
celery_app = Celery(__name__)
limiter = Limiter(key_func=get_remote_address) if LIMITER_AVAILABLE else None
def _is_internal_request() -> bool:
"""Rate-limit exemption: never throttle internal LAN / loopback callers
(the dashboard and server-to-server consumers like ZEUS) or health checks."""
if request.path == '/health':
return True
try:
ip = ipaddress.ip_address(get_remote_address())
return ip.is_private or ip.is_loopback or ip.is_link_local
except (ValueError, TypeError):
return False
def create_app(config_name=None):
"""
Application factory pattern
Args:
config_name: Configuration name (development, production, testing)
Returns:
Flask application instance
"""
if config_name is None:
config_name = os.getenv('FLASK_ENV', 'development')
app = Flask(__name__)
# Load configuration
config = get_config(config_name)
app.config.from_object(config)
config.init_app(app)
# Initialize extensions
init_extensions(app)
# Setup logging
setup_logging(app)
# Register blueprints and routes
register_blueprints(app)
# Register error handlers
register_error_handlers(app)
# Setup Swagger/ReDoc API documentation
setup_api_docs(app)
app.logger.info(f'Domain Check API started in {config_name} mode')
return app
def init_extensions(app):
"""Initialize Flask extensions"""
global redis_client
# Database
db.init_app(app)
migrate.init_app(app, db)
# CORS
CORS(app,
origins=app.config['CORS_ORIGINS'],
allow_headers=app.config['CORS_ALLOW_HEADERS'],
methods=app.config['CORS_METHODS'])
# Redis
try:
redis_client = Redis.from_url(
app.config['REDIS_URL'],
decode_responses=True,
socket_timeout=5,
socket_connect_timeout=5
)
redis_client.ping()
app.logger.info('Redis connection established')
except Exception as e:
app.logger.warning(f'Redis connection failed: {e}. Caching disabled.')
redis_client = None
# Celery
init_celery(app)
# Rate limiting (protects paid WHOIS/VirusTotal quotas from external abuse).
# Internal LAN/loopback traffic is exempt so it never throttles ZEUS or the
# dashboard. Storage errors are swallowed so a Redis hiccup can't 500 the API.
if limiter is not None:
try:
app.config.setdefault('RATELIMIT_STORAGE_URI', app.config['REDIS_URL'])
app.config.setdefault('RATELIMIT_DEFAULT', os.getenv('RATELIMIT_DEFAULT', '240 per minute;5000 per hour'))
app.config.setdefault('RATELIMIT_HEADERS_ENABLED', True)
app.config.setdefault('RATELIMIT_SWALLOW_ERRORS', True)
limiter.init_app(app)
limiter.request_filter(_is_internal_request)
app.logger.info('Rate limiter enabled (internal traffic exempt)')
except Exception as e:
app.logger.warning(f'Rate limiter init failed, continuing without it: {e}')
# Store redis_client in app context
app.redis = redis_client
def init_celery(app):
"""Initialize Celery"""
celery_app.conf.update(
broker_url=app.config['CELERY_BROKER_URL'],
result_backend=app.config['CELERY_RESULT_BACKEND'],
task_serializer=app.config['CELERY_TASK_SERIALIZER'],
result_serializer=app.config['CELERY_RESULT_SERIALIZER'],
accept_content=app.config['CELERY_ACCEPT_CONTENT'],
timezone=app.config['CELERY_TIMEZONE'],
enable_utc=app.config['CELERY_ENABLE_UTC'],
task_track_started=app.config['CELERY_TASK_TRACK_STARTED'],
task_time_limit=app.config['CELERY_TASK_TIME_LIMIT'],
task_soft_time_limit=app.config['CELERY_TASK_SOFT_TIME_LIMIT']
)
class ContextTask(celery_app.Task):
"""Make celery tasks work with Flask app context"""
def __call__(self, *args, **kwargs):
with app.app_context():
return self.run(*args, **kwargs)
celery_app.Task = ContextTask
app.celery = celery_app
return celery_app
def setup_logging(app):
"""Configure application logging"""
log_level = getattr(logging, app.config['LOG_LEVEL'].upper(), logging.INFO)
# Root logger
logging.basicConfig(
level=log_level,
format=app.config['LOG_FORMAT']
)
# App logger
app.logger.setLevel(log_level)
# Disable werkzeug request logs in production
if not app.config['DEBUG']:
logging.getLogger('werkzeug').setLevel(logging.WARNING)
def register_blueprints(app):
"""Register Flask blueprints"""
# Health check endpoint
@app.route('/health')
def health_check():
"""Health check endpoint for Docker and load balancers"""
# Simple health mode - minimal response (recommended for production)
if app.config.get('SIMPLE_HEALTH', False):
try:
from sqlalchemy import text
db.session.execute(text('SELECT 1'))
return jsonify({'status': 'ok'}), 200
except Exception:
return jsonify({'status': 'error'}), 503
# Detailed health mode (development)
health_status = {
'status': 'healthy',
'version': app.config['API_VERSION'],
'environment': os.getenv('FLASK_ENV', 'development')
}
# Check database
try:
from sqlalchemy import text
db.session.execute(text('SELECT 1'))
health_status['database'] = 'connected'
except Exception as e:
health_status['database'] = f'error: {str(e)}'
health_status['status'] = 'unhealthy'
# Check Redis
if app.redis:
try:
app.redis.ping()
health_status['redis'] = 'connected'
except Exception as e:
health_status['redis'] = f'error: {str(e)}'
else:
health_status['redis'] = 'disabled'
status_code = 200 if health_status['status'] == 'healthy' else 503
return jsonify(health_status), status_code
# Root endpoint - serve HTML dashboard
@app.route('/')
def index():
"""Serve the domain check dashboard"""
from flask import send_from_directory
return send_from_directory('static', 'index.html')
# API info endpoint
@app.route('/api')
def api_info():
"""API information endpoint"""
response = {
'name': app.config['API_TITLE'],
'version': app.config['API_VERSION'],
'endpoints': {
'health': '/health',
'api': f'/api/{app.config["API_VERSION"]}',
'dashboard': '/'
}
}
# Only show docs links if swagger is enabled
if not app.config.get('DISABLE_SWAGGER', False):
response['description'] = app.config['API_DESCRIPTION']
response['documentation'] = {
'swagger': '/docs',
'redoc': '/redoc'
}
return jsonify(response)
# Note: Namespaces will be registered in setup_api_docs function
def register_error_handlers(app):
"""Register error handlers"""
@app.errorhandler(404)
def not_found(error):
return jsonify({
'success': False,
'error': {
'code': 'NOT_FOUND',
'message': 'The requested resource was not found',
'status': 404
}
}), 404
@app.errorhandler(500)
def internal_error(error):
app.logger.error(f'Internal server error: {error}')
db.session.rollback()
return jsonify({
'success': False,
'error': {
'code': 'INTERNAL_SERVER_ERROR',
'message': 'An internal server error occurred',
'status': 500
}
}), 500
@app.errorhandler(400)
def bad_request(error):
return jsonify({
'success': False,
'error': {
'code': 'BAD_REQUEST',
'message': str(error),
'status': 400
}
}), 400
@app.errorhandler(429)
def rate_limit_exceeded(error):
return jsonify({
'success': False,
'error': {
'code': 'RATE_LIMIT_EXCEEDED',
'message': 'Too many requests. Please try again later.',
'status': 429
}
}), 429
def setup_api_docs(app):
"""Setup Swagger/ReDoc API documentation"""
# Check if swagger should be disabled (production security)
disable_swagger = app.config.get('DISABLE_SWAGGER', False)
# Create API instance - doc=False disables swagger UI
api = Api(
app,
version=app.config['API_VERSION'],
title=app.config['API_TITLE'],
description=app.config['API_DESCRIPTION'] if not disable_swagger else '',
doc='/docs' if not disable_swagger else False,
prefix=f'/api/{app.config["API_VERSION"]}',
contact=app.config['API_CONTACT'].get('name') if not disable_swagger else None,
contact_email=app.config['API_CONTACT'].get('email') if not disable_swagger else None,
license=app.config['API_LICENSE'].get('name') if not disable_swagger else None,
license_url=app.config['API_LICENSE'].get('url') if not disable_swagger else None,
terms_url=app.config.get('API_TERMS_OF_SERVICE') if not disable_swagger else None,
ordered=True,
validate=True
)
if disable_swagger:
app.logger.info('Swagger/ReDoc documentation disabled (DISABLE_SWAGGER=true)')
# Import and create API models
from app.api_models import create_api_models
models = create_api_models(api)
# Import and register namespaces
try:
from app.routes.check import api as check_ns
# Attach models to namespace
check_ns.models.update(models)
# Register namespace
api.add_namespace(check_ns, path='')
app.logger.info('API namespace registered successfully: check')
except ImportError as e:
app.logger.error(f'Failed to import check namespace: {e}')
except Exception as e:
app.logger.error(f'Failed to register check routes: {e}')
# Try to import other namespaces (if implemented)
try:
from app.routes.domain import api as domain_ns
domain_ns.models.update(models)
api.add_namespace(domain_ns, path='')
app.logger.info('API namespace registered successfully: domain')
except ImportError:
app.logger.debug('domain namespace not yet implemented')
try:
from app.routes.stats import api as stats_ns
stats_ns.models.update(models)
api.add_namespace(stats_ns, path='')
app.logger.info('API namespace registered successfully: stats')
except ImportError:
app.logger.debug('stats namespace not yet implemented')
try:
from app.routes.search import api as search_ns
search_ns.models.update(models)
api.add_namespace(search_ns, path='')
app.logger.info('API namespace registered successfully: search')
except ImportError:
app.logger.debug('search namespace not yet implemented')
try:
from app.routes.batch import api as batch_ns
batch_ns.models.update(models)
api.add_namespace(batch_ns, path='')
app.logger.info('API namespace registered successfully: batch')
except ImportError:
app.logger.debug('batch namespace not yet implemented')
# Add ReDoc endpoint (only if swagger is enabled)
if not disable_swagger:
@app.route('/redoc')
def redoc():
"""ReDoc API documentation"""
return f'''
<!DOCTYPE html>
<html>
<head>
<title>{app.config['API_TITLE']} - ReDoc</title>
<meta charset="utf-8"/>
<meta name="viewport" content="width=device-width, initial-scale=1">
<link href="https://fonts.googleapis.com/css?family=Montserrat:300,400,700|Roboto:300,400,700" rel="stylesheet">
<style>
body {{
margin: 0;
padding: 0;
}}
</style>
</head>
<body>
<redoc spec-url='/api/{app.config["API_VERSION"]}/swagger.json'></redoc>
<script src="https://cdn.redoc.ly/redoc/latest/bundles/redoc.standalone.js"></script>
</body>
</html>
'''
app.api = api
return api
# Create celery app for worker
def create_celery_app(app=None):
"""
Create and configure Celery app
For use in worker process
"""
app = app or create_app()
return app.celery

424
ai_platform/modules/domain_check/api/app/api_models.py Normal file
View file

@ -0,0 +1,424 @@
"""
Flask-RESTX API Models for Swagger Documentation
"""
from flask_restx import fields
def create_api_models(api):
"""
Create and register all API models for Swagger documentation
Args:
api: Flask-RESTX Api instance
Returns:
Dictionary of model names to model objects
"""
# ==================== REQUEST MODELS ====================
check_options_model = api.model('CheckOptions', {
'whois': fields.Boolean(
default=True,
description='Perform WHOIS lookup',
example=True
),
'dns': fields.Boolean(
default=False,
description='Perform DNS record lookup',
example=True
),
'ssl': fields.Boolean(
default=False,
description='Check SSL certificate',
example=False
),
'reputation': fields.Boolean(
default=False,
description='Check domain reputation (VirusTotal, blacklists)',
example=False
),
'force_refresh': fields.Boolean(
default=False,
description='Force refresh, bypass cache',
example=False
)
})
domain_check_request = api.model('DomainCheckRequest', {
'domain': fields.String(
required=True,
description='Domain name to check',
example='google.com',
pattern=r'^[a-zA-Z0-9][a-zA-Z0-9-_.]*\.[a-zA-Z]{2,}$'
),
'check_options': fields.Nested(
check_options_model,
description='Options for what to check',
required=False
)
})
# ==================== RESPONSE MODELS ====================
# WHOIS Response
whois_response = api.model('WhoisResponse', {
'creation_date': fields.String(
description='Domain creation date (ISO 8601)',
example='2024-01-15T10:30:00Z'
),
'expiration_date': fields.String(
description='Domain expiration date (ISO 8601)',
example='2026-01-15T10:30:00Z'
),
'registrar': fields.String(
description='Domain registrar name',
example='MarkMonitor Inc.'
),
'age_days': fields.Integer(
description='Domain age in days',
example=380
),
'status': fields.List(
fields.String,
description='Domain status codes',
example=['clientDeleteProhibited', 'clientTransferProhibited']
),
'name_servers': fields.List(
fields.String,
description='Name servers',
example=['ns1.google.com', 'ns2.google.com']
)
})
# DNS Response
mx_record_model = api.model('MXRecord', {
'priority': fields.Integer(description='MX priority', example=10),
'host': fields.String(description='Mail server hostname', example='smtp.google.com')
})
soa_record_model = api.model('SOARecord', {
'mname': fields.String(description='Primary master name server', example='ns1.google.com'),
'rname': fields.String(description='Responsible party email', example='dns-admin.google.com'),
'serial': fields.Integer(description='Serial number', example=2024011501),
'refresh': fields.Integer(description='Refresh interval', example=3600),
'retry': fields.Integer(description='Retry interval', example=600),
'expire': fields.Integer(description='Expire time', example=86400),
'minimum': fields.Integer(description='Minimum TTL', example=300)
})
dns_response = api.model('DNSResponse', {
'a_records': fields.List(
fields.String,
description='A records (IPv4 addresses)',
example=['142.250.185.46']
),
'aaaa_records': fields.List(
fields.String,
description='AAAA records (IPv6 addresses)',
example=['2a00:1450:4001:801::200e']
),
'mx_records': fields.List(
fields.Nested(mx_record_model),
description='MX records (mail servers)'
),
'txt_records': fields.List(
fields.String,
description='TXT records',
example=['v=spf1 include:_spf.google.com ~all']
),
'ns_records': fields.List(
fields.String,
description='NS records (name servers)',
example=['ns1.google.com', 'ns2.google.com']
),
'cname_records': fields.List(
fields.String,
description='CNAME records',
example=[]
),
'soa_record': fields.Nested(
soa_record_model,
description='SOA record (Start of Authority)'
),
'has_spf': fields.Boolean(
description='Has SPF record',
example=True
),
'has_dkim': fields.Boolean(
description='Has DKIM record',
example=True
),
'has_dmarc': fields.Boolean(
description='Has DMARC record',
example=True
)
})
# SSL Response
ssl_response = api.model('SSLResponse', {
'has_ssl': fields.Boolean(
description='Has valid SSL certificate',
example=True
),
'is_valid': fields.Boolean(
description='Certificate is valid',
example=True
),
'is_self_signed': fields.Boolean(
description='Certificate is self-signed',
example=False
),
'is_expired': fields.Boolean(
description='Certificate is expired',
example=False
),
'is_wildcard': fields.Boolean(
description='Wildcard certificate',
example=False
),
'issuer': fields.String(
description='Certificate issuer',
example='CN=GTS CA 1C3, O=Google Trust Services LLC, C=US'
),
'subject': fields.String(
description='Certificate subject',
example='CN=*.google.com'
),
'valid_from': fields.String(
description='Valid from date (ISO 8601)',
example='2024-12-01T08:15:00Z'
),
'valid_until': fields.String(
description='Valid until date (ISO 8601)',
example='2025-02-23T08:14:59Z'
),
'days_until_expiry': fields.Integer(
description='Days until certificate expires',
example=45
),
'key_size': fields.Integer(
description='Key size in bits',
example=2048
),
'signature_algorithm': fields.String(
description='Signature algorithm',
example='sha256WithRSAEncryption'
),
'error': fields.String(
description='Error message if SSL check failed',
example=None
)
})
# Risk Score Response
risk_factor_model = api.model('RiskFactor', {
'factor': fields.String(
description='Risk factor name',
example='domain_age',
enum=['domain_age', 'reputation', 'ssl', 'dns', 'whois']
),
'score': fields.Float(
description='Individual score for this factor',
example=85.0
),
'weight': fields.Float(
description='Weight of this factor',
example=0.30
),
'weighted_score': fields.Float(
description='Score * weight',
example=25.5
),
'reason': fields.String(
description='Explanation of the score',
example='Domain is only 45 days old (HIGH RISK)'
),
'age_days': fields.Integer(
description='Domain age in days (only for domain_age factor)',
example=45
)
})
risk_thresholds_model = api.model('RiskThresholds', {
'low': fields.String(
description='Low risk score range',
example='0-30'
),
'medium': fields.String(
description='Medium risk score range',
example='31-60'
),
'high': fields.String(
description='High risk score range',
example='61-85'
),
'critical': fields.String(
description='Critical risk score range',
example='86-100'
)
})
risk_score_response = api.model('RiskScoreResponse', {
'total': fields.Float(
description='Total risk score (0-100)',
example=65.5,
min=0,
max=100
),
'level': fields.String(
description='Risk level',
example='HIGH',
enum=['LOW', 'MEDIUM', 'HIGH', 'CRITICAL']
),
'factors': fields.List(
fields.Nested(risk_factor_model),
description='Detailed breakdown of risk factors'
),
'thresholds': fields.Nested(
risk_thresholds_model,
description='Risk level thresholds'
)
})
# Metadata Response
metadata_response = api.model('MetadataResponse', {
'cached': fields.Boolean(
description='Response served from cache',
example=False
),
'processing_time_ms': fields.Integer(
description='Processing time in milliseconds',
example=1250
),
'api_version': fields.String(
description='API version',
example='v1'
)
})
# Main Check Response Data
check_response_data = api.model('CheckResponseData', {
'domain': fields.String(
description='Checked domain name',
example='google.com'
),
'check_id': fields.String(
description='Unique check ID (UUID)',
example='123e4567-e89b-12d3-a456-426614174000'
),
'timestamp': fields.String(
description='Check timestamp (ISO 8601)',
example='2026-01-30T12:34:56Z'
),
'whois': fields.Nested(
whois_response,
description='WHOIS lookup results',
allow_null=True
),
'dns': fields.Nested(
dns_response,
description='DNS lookup results',
allow_null=True
),
'ssl': fields.Nested(
ssl_response,
description='SSL certificate check results',
allow_null=True
),
'reputation': fields.Raw(
description='Reputation check results (not implemented)',
example=None
),
'risk_score': fields.Nested(
risk_score_response,
description='Risk assessment results',
required=True
)
})
# Main Check Response
domain_check_response = api.model('DomainCheckResponse', {
'success': fields.Boolean(
description='Request success status',
example=True
),
'data': fields.Nested(
check_response_data,
description='Response data',
required=True
),
'metadata': fields.Nested(
metadata_response,
description='Request metadata',
required=True
)
})
# ==================== ERROR MODELS ====================
error_detail_model = api.model('ErrorDetail', {
'code': fields.String(
description='Error code',
example='INVALID_REQUEST',
enum=['INVALID_REQUEST', 'NOT_FOUND', 'INTERNAL_ERROR', 'RATE_LIMIT_EXCEEDED']
),
'message': fields.String(
description='Error message',
example='Domain parameter is required'
),
'status': fields.Integer(
description='HTTP status code',
example=400
)
})
error_response = api.model('ErrorResponse', {
'success': fields.Boolean(
description='Request success status',
example=False
),
'error': fields.Nested(
error_detail_model,
description='Error details',
required=True
)
})
# ==================== HEALTH CHECK MODEL ====================
health_response = api.model('HealthResponse', {
'status': fields.String(
description='Health status',
example='healthy',
enum=['healthy', 'unhealthy']
),
'version': fields.String(
description='API version',
example='v1'
),
'environment': fields.String(
description='Environment name',
example='production'
),
'database': fields.String(
description='Database connection status',
example='connected'
),
'redis': fields.String(
description='Redis connection status',
example='connected'
)
})
return {
'domain_check_request': domain_check_request,
'domain_check_response': domain_check_response,
'error_response': error_response,
'health_response': health_response,
'check_options_model': check_options_model,
'whois_response': whois_response,
'dns_response': dns_response,
'ssl_response': ssl_response,
'risk_score_response': risk_score_response
}

6
ai_platform/modules/domain_check/api/app/celery_app.py Normal file
View file

@ -0,0 +1,6 @@
"""
Celery Application for Domain Check
"""
from app import create_celery_app
celery = create_celery_app()

234
ai_platform/modules/domain_check/api/app/config.py Normal file
View file

@ -0,0 +1,234 @@
"""
Configuration module for Domain Check API
"""
import os
from datetime import timedelta
from typing import Dict, Any
class Config:
"""Base configuration"""
# Flask
SECRET_KEY = os.getenv('SECRET_KEY', 'dev-secret-key-change-in-production')
DEBUG = False
TESTING = False
# Database
SQLALCHEMY_DATABASE_URI = os.getenv(
'DATABASE_URL',
'postgresql://dns_admin:password@localhost:5432/domain_check'
)
SQLALCHEMY_TRACK_MODIFICATIONS = False
SQLALCHEMY_ECHO = False
SQLALCHEMY_POOL_SIZE = 10
SQLALCHEMY_POOL_TIMEOUT = 30
SQLALCHEMY_POOL_RECYCLE = 3600
SQLALCHEMY_MAX_OVERFLOW = 20
# Redis
REDIS_URL = os.getenv('REDIS_URL', 'redis://localhost:6379/0')
REDIS_TTL_HOT = int(os.getenv('REDIS_TTL_HOT', 21600)) # 6 hours
REDIS_TTL_WARM = int(os.getenv('REDIS_TTL_WARM', 86400)) # 24 hours
REDIS_TTL_COLD = int(os.getenv('REDIS_TTL_COLD', 604800)) # 7 days
# Celery
CELERY_BROKER_URL = os.getenv('CELERY_BROKER_URL', 'redis://localhost:6379/1')
CELERY_RESULT_BACKEND = os.getenv('CELERY_RESULT_BACKEND', 'redis://localhost:6379/2')
CELERY_TASK_SERIALIZER = 'json'
CELERY_RESULT_SERIALIZER = 'json'
CELERY_ACCEPT_CONTENT = ['json']
CELERY_TIMEZONE = 'UTC'
CELERY_ENABLE_UTC = True
CELERY_TASK_TRACK_STARTED = True
CELERY_TASK_TIME_LIMIT = 300 # 5 minutes
CELERY_TASK_SOFT_TIME_LIMIT = 240 # 4 minutes
# API Keys
WHOXY_API_KEY = os.getenv('WHOXY_API_KEY', '')
VIRUSTOTAL_API_KEY = os.getenv('VIRUSTOTAL_API_KEY', '')
# Risk Scoring Thresholds
RISK_THRESHOLD_LOW = int(os.getenv('RISK_THRESHOLD_LOW', 30))
RISK_THRESHOLD_MEDIUM = int(os.getenv('RISK_THRESHOLD_MEDIUM', 60))
RISK_THRESHOLD_HIGH = int(os.getenv('RISK_THRESHOLD_HIGH', 85))
# Domain Age Thresholds (days)
DOMAIN_AGE_CRITICAL = int(os.getenv('DOMAIN_AGE_CRITICAL', 90)) # 3 months
DOMAIN_AGE_HIGH = int(os.getenv('DOMAIN_AGE_HIGH', 180)) # 6 months
DOMAIN_AGE_MEDIUM = int(os.getenv('DOMAIN_AGE_MEDIUM', 365)) # 1 year
# API Rate Limiting
WHOXY_DAILY_LIMIT = int(os.getenv('WHOXY_DAILY_LIMIT', 8000))
VIRUSTOTAL_DAILY_LIMIT = int(os.getenv('VIRUSTOTAL_DAILY_LIMIT', 500))
# Batch Processing
BATCH_CHUNK_SIZE = int(os.getenv('BATCH_CHUNK_SIZE', 50))
BATCH_TIMEOUT_SECONDS = int(os.getenv('BATCH_TIMEOUT_SECONDS', 300))
# API Settings
API_VERSION = os.getenv('API_VERSION', 'v1')
API_TITLE = 'Domain Check API - Anti-Fake News Tool'
API_DESCRIPTION = '''
**Comprehensive Domain Verification & Risk Scoring API**
This API provides powerful tools for detecting potentially malicious or newly-registered domains
commonly used in disinformation campaigns and fake news distribution.
## 🎯 Key Features
- **WHOIS Analysis**: Domain age, registrar information, privacy protection detection
- **DNS Verification**: A, AAAA, MX, TXT, NS, CNAME, SOA records with email security (SPF, DKIM, DMARC)
- **SSL Certificate Validation**: Certificate validity, issuer verification, expiration monitoring
- **Risk Scoring**: Multi-factor algorithm (0-100) with detailed breakdown
- **Comprehensive Database**: Full history tracking and audit trail
## 📊 Risk Levels
- **LOW (0-30)**: Established, trustworthy domains
- **MEDIUM (31-60)**: Moderate risk, requires attention
- **HIGH (61-85)**: Suspicious activity detected
- **CRITICAL (86-100)**: Newly registered or highly suspicious domains
## 🚀 Quick Start
Use the `/check` endpoint to verify any domain:
```bash
curl -X POST http://domain-check-api:11000/api/v1/check/check \\
-H "Content-Type: application/json" \\
-d '{"domain": "example.com", "check_options": {"whois": true, "dns": true}}'
```
## 📖 Documentation
- **Swagger UI**: Interactive API testing and documentation
- **ReDoc**: Clean, responsive API reference
- **Platformă**: modul T4 (evaluare credibilitate sursă) al platformei DIDI
'''
API_CONTACT = {
'name': 'DIDI Domain Check',
'email': 'support@didi.local'
}
API_LICENSE = {
'name': 'MIT',
'url': 'https://opensource.org/licenses/MIT'
}
API_TERMS_OF_SERVICE = ''
# CORS - use CORS_ORIGINS env var for production (comma-separated)
# Example: CORS_ORIGINS=https://didi.example.ro,https://admin.didi.example.ro
# Default '*' allows all origins (development only!)
_cors_origins_env = os.getenv('CORS_ORIGINS', '*')
CORS_ORIGINS = ['*'] if _cors_origins_env == '*' else [o.strip() for o in _cors_origins_env.split(',') if o.strip()]
CORS_ALLOW_HEADERS = ['Content-Type', 'Authorization']
CORS_METHODS = ['GET', 'POST', 'PUT', 'DELETE', 'OPTIONS']
# Security - Documentation endpoints
# Set DISABLE_SWAGGER=true in production to hide /docs and /redoc
DISABLE_SWAGGER = os.getenv('DISABLE_SWAGGER', 'false').lower() == 'true'
# Health endpoint - set SIMPLE_HEALTH=true to return only {"status":"ok"}
SIMPLE_HEALTH = os.getenv('SIMPLE_HEALTH', 'false').lower() == 'true'
# Logging
LOG_LEVEL = os.getenv('LOG_LEVEL', 'INFO')
LOG_FORMAT = '%(asctime)s - %(name)s - %(levelname)s - %(message)s'
LOG_FILE = os.path.join(os.path.dirname(__file__), '..', 'logs', 'api.log')
# External Service URLs
WHOXY_API_URL = 'https://api.whoxy.com/'
VIRUSTOTAL_API_URL = 'https://www.virustotal.com/api/v3/'
RDAP_BOOTSTRAP_URL = 'https://rdap.org/'
# Risk Scoring Weights
RISK_WEIGHT_DOMAIN_AGE = 0.30
RISK_WEIGHT_REPUTATION = 0.30
RISK_WEIGHT_SSL = 0.20
RISK_WEIGHT_DNS = 0.10
RISK_WEIGHT_WHOIS = 0.10
# Pagination
DEFAULT_PAGE_SIZE = 50
MAX_PAGE_SIZE = 100
# Timeouts (seconds)
HTTP_TIMEOUT = 30
DNS_TIMEOUT = 10
SSL_TIMEOUT = 10
@staticmethod
def init_app(app):
"""Initialize application configuration"""
# Create logs directory if it doesn't exist
log_dir = os.path.dirname(Config.LOG_FILE)
os.makedirs(log_dir, exist_ok=True)
class DevelopmentConfig(Config):
"""Development configuration"""
DEBUG = True
SQLALCHEMY_ECHO = False
LOG_LEVEL = 'DEBUG'
class ProductionConfig(Config):
"""Production configuration"""
DEBUG = False
TESTING = False
SQLALCHEMY_ECHO = False
LOG_LEVEL = 'WARNING'
# Production security defaults (can be overridden by env vars)
# If CORS_ORIGINS not set, default to empty (blocks all cross-origin)
_cors_origins_env = os.getenv('CORS_ORIGINS', '')
CORS_ORIGINS = [o.strip() for o in _cors_origins_env.split(',') if o.strip()] if _cors_origins_env else []
# Disable swagger by default in production (override with DISABLE_SWAGGER=false)
DISABLE_SWAGGER = os.getenv('DISABLE_SWAGGER', 'true').lower() != 'false'
# Simple health by default in production
SIMPLE_HEALTH = os.getenv('SIMPLE_HEALTH', 'true').lower() != 'false'
@classmethod
def init_app(cls, app):
Config.init_app(app)
# Production-specific initialization
import logging
from logging.handlers import RotatingFileHandler
# Setup file handler with rotation
file_handler = RotatingFileHandler(
cls.LOG_FILE,
maxBytes=10485760, # 10MB
backupCount=10
)
file_handler.setLevel(logging.WARNING)
file_handler.setFormatter(logging.Formatter(cls.LOG_FORMAT))
app.logger.addHandler(file_handler)
class TestingConfig(Config):
"""Testing configuration"""
TESTING = True
DEBUG = True
SQLALCHEMY_DATABASE_URI = 'sqlite:///:memory:'
REDIS_URL = 'redis://localhost:6379/15' # Use separate Redis DB for testing
WTF_CSRF_ENABLED = False
# Configuration dictionary
config: Dict[str, Any] = {
'development': DevelopmentConfig,
'production': ProductionConfig,
'testing': TestingConfig,
'default': DevelopmentConfig
}
def get_config(env: str = None) -> Config:
"""Get configuration based on environment"""
if env is None:
env = os.getenv('FLASK_ENV', 'development')
return config.get(env, config['default'])

26
ai_platform/modules/domain_check/api/app/models/__init__.py Normal file
View file

@ -0,0 +1,26 @@
"""
SQLAlchemy Models Package
"""
from app.models.domain import Domain
from app.models.whois import WhoisRecord
from app.models.dns import DnsRecord
from app.models.ssl import SslCertificate
from app.models.reputation import ReputationScore
from app.models.risk import RiskAssessment
from app.models.check_history import CheckHistory
from app.models.batch import BatchOperation
from app.models.blacklist import Blacklist
from app.models.api_usage import ApiUsageLog
__all__ = [
'Domain',
'WhoisRecord',
'DnsRecord',
'SslCertificate',
'ReputationScore',
'RiskAssessment',
'CheckHistory',
'BatchOperation',
'Blacklist',
'ApiUsageLog'
]

32
ai_platform/modules/domain_check/api/app/models/api_usage.py Normal file
View file

@ -0,0 +1,32 @@
"""API Usage Log Model"""
import uuid
from datetime import datetime
from sqlalchemy.dialects.postgresql import UUID, JSONB
from app import db
class ApiUsageLog(db.Model):
"""API usage logging model"""
__tablename__ = 'api_usage_logs'
id = db.Column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
service_name = db.Column(db.String(50), nullable=False, index=True)
endpoint = db.Column(db.String(255))
request_count = db.Column(db.Integer, default=1)
response_time_ms = db.Column(db.Integer)
status_code = db.Column(db.Integer, index=True)
quota_used = db.Column(db.Integer)
quota_remaining = db.Column(db.Integer)
error_message = db.Column(db.Text)
request_params = db.Column(JSONB)
logged_at = db.Column(db.DateTime(timezone=True), default=datetime.utcnow, index=True)
def to_dict(self):
return {
'id': str(self.id),
'service_name': self.service_name,
'endpoint': self.endpoint,
'status_code': self.status_code,
'response_time_ms': self.response_time_ms,
'logged_at': self.logged_at.isoformat() if self.logged_at else None
}

39
ai_platform/modules/domain_check/api/app/models/batch.py Normal file
View file

@ -0,0 +1,39 @@
"""Batch Operation Model"""
import uuid
from datetime import datetime
from sqlalchemy.dialects.postgresql import UUID, ARRAY, JSONB
from app import db
class BatchOperation(db.Model):
"""Batch operation model"""
__tablename__ = 'batch_operations'
id = db.Column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
batch_id = db.Column(UUID(as_uuid=True), unique=True, nullable=False, index=True)
total_domains = db.Column(db.Integer, nullable=False)
completed_count = db.Column(db.Integer, default=0)
failed_count = db.Column(db.Integer, default=0)
status = db.Column(db.String(20), default='pending', index=True)
priority = db.Column(db.String(20), default='normal', index=True)
started_at = db.Column(db.DateTime(timezone=True))
completed_at = db.Column(db.DateTime(timezone=True))
estimated_completion_at = db.Column(db.DateTime(timezone=True))
requested_by = db.Column(db.String(100))
check_options = db.Column(JSONB)
domain_list = db.Column(ARRAY(db.Text))
error_log = db.Column(JSONB)
summary = db.Column(JSONB)
created_at = db.Column(db.DateTime(timezone=True), default=datetime.utcnow, index=True)
def to_dict(self):
return {
'batch_id': str(self.batch_id),
'total_domains': self.total_domains,
'completed_count': self.completed_count,
'failed_count': self.failed_count,
'status': self.status,
'progress_percentage': round((self.completed_count / self.total_domains * 100) if self.total_domains > 0 else 0, 2),
'started_at': self.started_at.isoformat() if self.started_at else None,
'completed_at': self.completed_at.isoformat() if self.completed_at else None
}

33
ai_platform/modules/domain_check/api/app/models/blacklist.py Normal file
View file

@ -0,0 +1,33 @@
"""Blacklist Model"""
import uuid
from datetime import datetime
from sqlalchemy.dialects.postgresql import UUID, JSONB
from app import db
class Blacklist(db.Model):
"""Blacklist model"""
__tablename__ = 'blacklists'
id = db.Column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
domain = db.Column(db.String(255), unique=True, nullable=False, index=True)
reason = db.Column(db.Text)
category = db.Column(db.String(50), index=True)
source = db.Column(db.String(100))
severity = db.Column(db.String(20), default='medium', index=True)
added_at = db.Column(db.DateTime(timezone=True), default=datetime.utcnow)
updated_at = db.Column(db.DateTime(timezone=True), default=datetime.utcnow, onupdate=datetime.utcnow)
is_active = db.Column(db.Boolean, default=True, index=True)
expires_at = db.Column(db.DateTime(timezone=True))
extra_data = db.Column(JSONB)
def to_dict(self):
return {
'id': str(self.id),
'domain': self.domain,
'reason': self.reason,
'category': self.category,
'severity': self.severity,
'is_active': self.is_active,
'added_at': self.added_at.isoformat() if self.added_at else None
}

38
ai_platform/modules/domain_check/api/app/models/check_history.py Normal file
View file

@ -0,0 +1,38 @@
"""Check History Model"""
import uuid
from datetime import datetime
from sqlalchemy.dialects.postgresql import UUID, JSONB
from app import db
class CheckHistory(db.Model):
"""Check history model"""
__tablename__ = 'check_history'
id = db.Column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
check_id = db.Column(UUID(as_uuid=True), unique=True, nullable=False, index=True)
domain_id = db.Column(UUID(as_uuid=True), db.ForeignKey('domains.id', ondelete='CASCADE'), nullable=False, index=True)
risk_assessment_id = db.Column(UUID(as_uuid=True), db.ForeignKey('risk_assessments.id', ondelete='SET NULL'))
requested_by = db.Column(db.String(100), default='api', index=True)
request_ip = db.Column(db.String(45))
user_agent = db.Column(db.Text)
check_options = db.Column(JSONB)
processing_time_ms = db.Column(db.Integer)
cache_hit = db.Column(db.Boolean, default=False)
changes_detected = db.Column(db.Boolean, default=False)
change_summary = db.Column(JSONB)
status = db.Column(db.String(20), default='completed', index=True)
error_message = db.Column(db.Text)
created_at = db.Column(db.DateTime(timezone=True), default=datetime.utcnow, index=True)
domain = db.relationship('Domain', back_populates='check_history')
def to_dict(self):
return {
'id': str(self.id),
'check_id': str(self.check_id),
'status': self.status,
'cache_hit': self.cache_hit,
'processing_time_ms': self.processing_time_ms,
'created_at': self.created_at.isoformat() if self.created_at else None
}

31
ai_platform/modules/domain_check/api/app/models/dns.py Normal file
View file

@ -0,0 +1,31 @@
"""DNS Record Model"""
import uuid
from datetime import datetime
from sqlalchemy.dialects.postgresql import UUID
from app import db
class DnsRecord(db.Model):
"""DNS record model"""
__tablename__ = 'dns_records'
id = db.Column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
domain_id = db.Column(UUID(as_uuid=True), db.ForeignKey('domains.id', ondelete='CASCADE'), nullable=False, index=True)
record_type = db.Column(db.String(10), nullable=False, index=True)
record_value = db.Column(db.Text, nullable=False)
ttl = db.Column(db.Integer)
priority = db.Column(db.Integer)
fetched_at = db.Column(db.DateTime(timezone=True), default=datetime.utcnow, index=True)
created_at = db.Column(db.DateTime(timezone=True), default=datetime.utcnow)
domain = db.relationship('Domain', back_populates='dns_records')
def to_dict(self):
return {
'id': str(self.id),
'record_type': self.record_type,
'record_value': self.record_value,
'ttl': self.ttl,
'priority': self.priority,
'fetched_at': self.fetched_at.isoformat() if self.fetched_at else None
}

118
ai_platform/modules/domain_check/api/app/models/domain.py Normal file
View file

@ -0,0 +1,118 @@
"""
Domain Model
"""
import uuid
from datetime import datetime
from sqlalchemy.dialects.postgresql import UUID, ARRAY
from app import db
class Domain(db.Model):
"""Domain model - stores basic domain information"""
__tablename__ = 'domains'
id = db.Column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
domain = db.Column(db.String(255), nullable=False, index=True)
subdomain = db.Column(db.String(255), nullable=True)
tld = db.Column(db.String(50), nullable=False, index=True)
full_domain = db.Column(db.String(255), unique=True, nullable=False, index=True)
first_seen_at = db.Column(db.DateTime(timezone=True), default=datetime.utcnow)
last_checked_at = db.Column(db.DateTime(timezone=True), nullable=True, index=True)
check_count = db.Column(db.Integer, default=0)
is_active = db.Column(db.Boolean, default=True, index=True)
created_at = db.Column(db.DateTime(timezone=True), default=datetime.utcnow)
updated_at = db.Column(db.DateTime(timezone=True), default=datetime.utcnow, onupdate=datetime.utcnow)
# Relationships
whois_records = db.relationship('WhoisRecord', back_populates='domain', lazy='dynamic', cascade='all, delete-orphan')
dns_records = db.relationship('DnsRecord', back_populates='domain', lazy='dynamic', cascade='all, delete-orphan')
ssl_certificates = db.relationship('SslCertificate', back_populates='domain', lazy='dynamic', cascade='all, delete-orphan')
reputation_scores = db.relationship('ReputationScore', back_populates='domain', lazy='dynamic', cascade='all, delete-orphan')
risk_assessments = db.relationship('RiskAssessment', back_populates='domain', lazy='dynamic', cascade='all, delete-orphan')
check_history = db.relationship('CheckHistory', back_populates='domain', lazy='dynamic', cascade='all, delete-orphan')
def __repr__(self):
return f'<Domain {self.full_domain}>'
def to_dict(self, include_relationships=False):
"""Convert model to dictionary"""
data = {
'id': str(self.id),
'domain': self.domain,
'subdomain': self.subdomain,
'tld': self.tld,
'full_domain': self.full_domain,
'first_seen_at': self.first_seen_at.isoformat() if self.first_seen_at else None,
'last_checked_at': self.last_checked_at.isoformat() if self.last_checked_at else None,
'check_count': self.check_count,
'is_active': self.is_active,
'created_at': self.created_at.isoformat() if self.created_at else None,
'updated_at': self.updated_at.isoformat() if self.updated_at else None
}
if include_relationships:
# Get latest records
latest_whois = self.whois_records.order_by(WhoisRecord.fetched_at.desc()).first()
latest_risk = self.risk_assessments.order_by(RiskAssessment.assessed_at.desc()).first()
data['latest_whois'] = latest_whois.to_dict() if latest_whois else None
data['latest_risk'] = latest_risk.to_dict() if latest_risk else None
return data
@staticmethod
def parse_domain(full_domain: str) -> dict:
"""
Parse a full domain into components
Args:
full_domain: Full domain name (e.g., 'www.example.com')
Returns:
dict with 'domain', 'subdomain', 'tld', 'full_domain'
"""
import tldextract
extracted = tldextract.extract(full_domain)
return {
'domain': extracted.domain,
'subdomain': extracted.subdomain if extracted.subdomain else None,
'tld': extracted.suffix,
'full_domain': full_domain.lower().strip()
}
@classmethod
def get_or_create(cls, full_domain: str):
"""
Get existing domain or create new one
Args:
full_domain: Full domain name
Returns:
Tuple of (Domain instance, created boolean)
"""
domain = cls.query.filter_by(full_domain=full_domain.lower()).first()
if domain:
return domain, False
# Parse domain components
parsed = cls.parse_domain(full_domain)
# Create new domain
domain = cls(
domain=parsed['domain'],
subdomain=parsed['subdomain'],
tld=parsed['tld'],
full_domain=parsed['full_domain']
)
db.session.add(domain)
db.session.commit()
return domain, True

42
ai_platform/modules/domain_check/api/app/models/reputation.py Normal file
View file

@ -0,0 +1,42 @@
"""Reputation Score Model"""
import uuid
from datetime import datetime
from sqlalchemy.dialects.postgresql import UUID, ARRAY, JSONB
from app import db
class ReputationScore(db.Model):
"""Reputation score from various sources"""
__tablename__ = 'reputation_scores'
id = db.Column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
domain_id = db.Column(UUID(as_uuid=True), db.ForeignKey('domains.id', ondelete='CASCADE'), nullable=False, index=True)
source = db.Column(db.String(50), nullable=False, index=True)
score = db.Column(db.Integer)
malicious_count = db.Column(db.Integer, default=0)
suspicious_count = db.Column(db.Integer, default=0)
harmless_count = db.Column(db.Integer, default=0)
undetected_count = db.Column(db.Integer, default=0)
is_blacklisted = db.Column(db.Boolean, default=False, index=True)
blacklist_names = db.Column(ARRAY(db.Text))
is_typosquatting = db.Column(db.Boolean, default=False, index=True)
typosquatting_target = db.Column(db.String(255))
raw_response = db.Column(JSONB)
checked_at = db.Column(db.DateTime(timezone=True), default=datetime.utcnow, index=True)
created_at = db.Column(db.DateTime(timezone=True), default=datetime.utcnow)
domain = db.relationship('Domain', back_populates='reputation_scores')
def to_dict(self):
return {
'id': str(self.id),
'source': self.source,
'score': self.score,
'malicious_count': self.malicious_count,
'suspicious_count': self.suspicious_count,
'harmless_count': self.harmless_count,
'is_blacklisted': self.is_blacklisted,
'blacklist_names': self.blacklist_names,
'is_typosquatting': self.is_typosquatting,
'checked_at': self.checked_at.isoformat() if self.checked_at else None
}

43
ai_platform/modules/domain_check/api/app/models/risk.py Normal file
View file

@ -0,0 +1,43 @@
"""Risk Assessment Model"""
import uuid
from datetime import datetime
from sqlalchemy.dialects.postgresql import UUID, JSONB
from app import db
class RiskAssessment(db.Model):
"""Risk assessment model"""
__tablename__ = 'risk_assessments'
id = db.Column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
domain_id = db.Column(UUID(as_uuid=True), db.ForeignKey('domains.id', ondelete='CASCADE'), nullable=False, index=True)
check_id = db.Column(UUID(as_uuid=True), unique=True, nullable=False, index=True)
total_score = db.Column(db.Integer, nullable=False, index=True)
risk_level = db.Column(db.String(20), nullable=False, index=True)
domain_age_score = db.Column(db.Integer, default=0)
domain_age_days = db.Column(db.Integer)
ssl_score = db.Column(db.Integer, default=0)
dns_score = db.Column(db.Integer, default=0)
reputation_score = db.Column(db.Integer, default=0)
whois_score = db.Column(db.Integer, default=0)
factors = db.Column(JSONB)
is_new_domain = db.Column(db.Boolean, default=False, index=True)
is_suspicious = db.Column(db.Boolean, default=False, index=True)
requires_manual_review = db.Column(db.Boolean, default=False)
assessed_at = db.Column(db.DateTime(timezone=True), default=datetime.utcnow, index=True)
created_at = db.Column(db.DateTime(timezone=True), default=datetime.utcnow)
domain = db.relationship('Domain', back_populates='risk_assessments')
def to_dict(self):
return {
'id': str(self.id),
'check_id': str(self.check_id),
'total_score': self.total_score,
'risk_level': self.risk_level,
'domain_age_days': self.domain_age_days,
'is_new_domain': self.is_new_domain,
'is_suspicious': self.is_suspicious,
'factors': self.factors,
'assessed_at': self.assessed_at.isoformat() if self.assessed_at else None
}

40
ai_platform/modules/domain_check/api/app/models/ssl.py Normal file
View file

@ -0,0 +1,40 @@
"""SSL Certificate Model"""
import uuid
from datetime import datetime
from sqlalchemy.dialects.postgresql import UUID, JSONB
from app import db
class SslCertificate(db.Model):
"""SSL Certificate model"""
__tablename__ = 'ssl_certificates'
id = db.Column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
domain_id = db.Column(UUID(as_uuid=True), db.ForeignKey('domains.id', ondelete='CASCADE'), nullable=False, index=True)
issuer = db.Column(db.String(255), index=True)
subject = db.Column(db.String(255))
valid_from = db.Column(db.DateTime(timezone=True))
valid_until = db.Column(db.DateTime(timezone=True), index=True)
serial_number = db.Column(db.String(255))
signature_algorithm = db.Column(db.String(100))
key_size = db.Column(db.Integer)
is_wildcard = db.Column(db.Boolean, default=False)
is_self_signed = db.Column(db.Boolean, default=False)
is_valid = db.Column(db.Boolean, default=True, index=True)
certificate_chain = db.Column(JSONB)
fetched_at = db.Column(db.DateTime(timezone=True), default=datetime.utcnow, index=True)
created_at = db.Column(db.DateTime(timezone=True), default=datetime.utcnow)
domain = db.relationship('Domain', back_populates='ssl_certificates')
def to_dict(self):
return {
'id': str(self.id),
'issuer': self.issuer,
'subject': self.subject,
'valid_from': self.valid_from.isoformat() if self.valid_from else None,
'valid_until': self.valid_until.isoformat() if self.valid_until else None,
'is_valid': self.is_valid,
'is_self_signed': self.is_self_signed,
'days_until_expiry': (self.valid_until - datetime.utcnow()).days if self.valid_until and self.valid_until > datetime.utcnow() else 0
}

62
ai_platform/modules/domain_check/api/app/models/whois.py Normal file
View file

@ -0,0 +1,62 @@
"""
WHOIS Record Model
"""
import uuid
from datetime import datetime
from sqlalchemy.dialects.postgresql import UUID, ARRAY, JSONB
from app import db
class WhoisRecord(db.Model):
"""WHOIS/RDAP record model"""
__tablename__ = 'whois_records'
id = db.Column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
domain_id = db.Column(UUID(as_uuid=True), db.ForeignKey('domains.id', ondelete='CASCADE'), nullable=False, index=True)
creation_date = db.Column(db.DateTime(timezone=True), index=True)
expiration_date = db.Column(db.DateTime(timezone=True))
updated_date = db.Column(db.DateTime(timezone=True))
registrar = db.Column(db.String(255), index=True)
registrar_url = db.Column(db.String(500))
registrant_org = db.Column(db.String(255))
registrant_country = db.Column(db.String(2))
admin_email = db.Column(db.String(255))
name_servers = db.Column(ARRAY(db.Text))
status = db.Column(ARRAY(db.Text))
dnssec = db.Column(db.Boolean)
raw_whois_data = db.Column(JSONB)
data_source = db.Column(db.String(50), default='rdap', index=True)
fetched_at = db.Column(db.DateTime(timezone=True), default=datetime.utcnow, index=True)
created_at = db.Column(db.DateTime(timezone=True), default=datetime.utcnow)
# Relationship
domain = db.relationship('Domain', back_populates='whois_records')
def __repr__(self):
return f'<WhoisRecord domain_id={self.domain_id} source={self.data_source}>'
def to_dict(self):
"""Convert to dictionary"""
return {
'id': str(self.id),
'domain_id': str(self.domain_id),
'creation_date': self.creation_date.isoformat() if self.creation_date else None,
'expiration_date': self.expiration_date.isoformat() if self.expiration_date else None,
'updated_date': self.updated_date.isoformat() if self.updated_date else None,
'registrar': self.registrar,
'registrar_url': self.registrar_url,
'registrant_org': self.registrant_org,
'registrant_country': self.registrant_country,
'admin_email': self.admin_email,
'name_servers': self.name_servers,
'status': self.status,
'dnssec': self.dnssec,
'data_source': self.data_source,
'fetched_at': self.fetched_at.isoformat() if self.fetched_at else None,
'age_days': (datetime.utcnow() - self.creation_date).days if self.creation_date else None
}

0
ai_platform/modules/domain_check/api/app/routes/__init__.py Normal file
View file

14
ai_platform/modules/domain_check/api/app/routes/batch.py Normal file
View file

@ -0,0 +1,14 @@
"""Batch routes - stub for now"""
from flask import Blueprint, jsonify
batch_bp = Blueprint('batch', __name__)
@batch_bp.route('/check/batch', methods=['POST'])
def batch_check():
"""Batch domain check - TODO: implement"""
return jsonify({'message': 'Not yet implemented'}), 501
@batch_bp.route('/batch/<string:batch_id>/status', methods=['GET'])
def batch_status(batch_id):
"""Get batch status - TODO: implement"""
return jsonify({'message': 'Not yet implemented', 'batch_id': batch_id}), 501

560
ai_platform/modules/domain_check/api/app/routes/check.py Normal file
View file

@ -0,0 +1,560 @@
"""
Domain Check Routes - Comprehensive Domain Verification API
"""
import uuid
import logging
import time
from concurrent.futures import ThreadPoolExecutor
from datetime import datetime
from flask import request, current_app
from flask_restx import Namespace, Resource, fields
from werkzeug.exceptions import BadRequest, HTTPException
from app import db
from app.models import Domain, WhoisRecord, DnsRecord, SslCertificate, RiskAssessment, CheckHistory
from app.services.whois_service import WhoisService
from app.services.risk_scorer import _coerce_datetime
from app.services.dns_service import DNSService
from app.services.ssl_service import SSLService
from app.services.risk_scorer import RiskScorer
from app.services.ip_intelligence_service import IPIntelligenceService
from app.services.http_analysis_service import HTTPAnalysisService
from app.services.blacklist_service import BlacklistService
from app.services.subdomain_service import SubdomainService
from app.services.port_scan_service import PortScanService
from app.services.mail_intelligence_service import MailIntelligenceService
logger = logging.getLogger(__name__)
# Create namespace
api = Namespace('check', description='Domain checking operations')
# Import model definitions from api_models.py
from app.api_models import create_api_models
# Create placeholder models
_temp_api = Namespace('_temp')
_models = create_api_models(_temp_api)
# Get the models we need for decorators
check_request_model = _models.get('domain_check_request')
check_response_model = _models.get('domain_check_response')
error_response_model = _models.get('error_response')
@api.route('/check')
class DomainCheck(Resource):
"""Domain Check Resource - Comprehensive domain verification"""
@api.doc(
'check_domain',
description='''Perform comprehensive domain verification including:
**Basic Checks:**
- WHOIS lookup (domain age, registrar, status, name servers)
- DNS record checking (A, AAAA, MX, TXT, NS, CNAME, SOA)
- SSL certificate validation (validity, issuer, expiration)
**Advanced Checks (when enabled):**
- IP Intelligence (geolocation, ASN, reverse DNS, hosting info)
- HTTP Security Analysis (headers, technology detection)
- Blacklist/Reputation checking (DNSBL, spam lists)
- Port Scanning (open ports, dangerous services)
- Subdomain Enumeration (Certificate Transparency)
Returns detailed domain information and comprehensive risk score (0-100).'''
)
@api.expect(check_request_model, validate=False)
@api.response(200, 'Success - Domain check completed', check_response_model)
@api.response(400, 'Bad Request - Invalid domain or parameters', error_response_model)
@api.response(500, 'Internal Server Error - Check failed', error_response_model)
def post(self):
"""
Check a single domain - Full comprehensive analysis
"""
start_time = time.time()
try:
# Get request data
try:
data = api.payload or {}
except BadRequest:
data = {}
if not data or 'domain' not in data:
return {
'success': False,
'error': {
'code': 'INVALID_REQUEST',
'message': 'Domain parameter is required',
'status': 400
}
}, 400
domain_name = data['domain'].lower().strip()
check_options = data.get('check_options', {
'whois': True,
'dns': True,
'ssl': True,
'ip_intelligence': True,
'http_analysis': True,
'blacklist': True,
'port_scan': False, # Disabled by default (slow)
'subdomains': False, # Disabled by default (slow)
'force_refresh': False
})
logger.info(f'Checking domain: {domain_name} with options: {check_options}')
# Generate check ID
check_id = uuid.uuid4()
# Get or create domain
domain, created = Domain.get_or_create(domain_name)
# Initialize all services
whoxy_api_key = current_app.config.get('WHOXY_API_KEY')
whois_service = WhoisService(whoxy_api_key=whoxy_api_key)
dns_service = DNSService()
ssl_service = SSLService()
ip_service = IPIntelligenceService()
http_service = HTTPAnalysisService()
blacklist_service = BlacklistService()
subdomain_service = SubdomainService()
port_scan_service = PortScanService()
mail_service = MailIntelligenceService()
risk_scorer = RiskScorer()
# Collect all domain data for risk scoring
domain_data = {
'domain': domain_name,
'whois': None,
'dns': None,
'ssl': None,
'ip_intelligence': None,
'http_analysis': None,
'blacklist': None,
'port_scan': None,
'subdomains': None
}
# Response data
response_data = {
'domain': domain_name,
'check_id': str(check_id),
'timestamp': datetime.utcnow().isoformat() + 'Z'
}
# Primary IP for subsequent checks
primary_ip = None
# ----------------------------------------------------------------
# Concurrent fetch. Network lookups are I/O-bound and independent,
# so we run them in a thread pool instead of sequentially (cuts a
# full check from ~sum-of-calls to ~max-of-calls). DB writes stay in
# the main thread afterwards because the SQLAlchemy session is not
# thread-safe. Each task runs inside an app context so services that
# read current_app keep working off the main thread.
# ----------------------------------------------------------------
app_obj = current_app._get_current_object()
def _run(fn, *a, **kw):
with app_obj.app_context():
try:
return fn(*a, **kw)
except Exception as exc:
logger.warning(f'{getattr(fn, "__name__", fn)} failed: {exc}')
return None
# Phase 1 — tasks that only need the domain name.
p1 = {}
with ThreadPoolExecutor(max_workers=5) as ex:
if check_options.get('whois', True):
p1['whois'] = ex.submit(_run, whois_service.lookup, domain_name)
if check_options.get('dns', True):
p1['dns'] = ex.submit(_run, dns_service.lookup, domain_name)
if check_options.get('ssl', True):
p1['ssl'] = ex.submit(_run, ssl_service.check, domain_name)
if check_options.get('http_analysis', True):
p1['http'] = ex.submit(_run, http_service.analyze, domain_name)
if check_options.get('subdomains', False):
extra_subs = check_options.get('extra_subdomains') or []
p1['subdomains'] = ex.submit(_run, subdomain_service.enumerate,
get_root_domain(domain_name), extra_subs)
whois_data = p1['whois'].result() if 'whois' in p1 else None
dns_data = p1['dns'].result() if 'dns' in p1 else None
ssl_data = p1['ssl'].result() if 'ssl' in p1 else None
http_data = p1['http'].result() if 'http' in p1 else None
subdomain_data = p1['subdomains'].result() if 'subdomains' in p1 else None
if dns_data and dns_data.get('a_records'):
primary_ip = dns_data['a_records'][0]
# Phase 2 — tasks that need DNS results (primary IP, MX/TXT).
p2 = {}
with ThreadPoolExecutor(max_workers=4) as ex:
if check_options.get('ip_intelligence', True) and primary_ip:
p2['ip'] = ex.submit(_run, ip_service.lookup, primary_ip)
if check_options.get('blacklist', True):
p2['blacklist'] = ex.submit(_run, blacklist_service.check_all, domain_name, primary_ip)
if check_options.get('port_scan', False) and primary_ip:
p2['port'] = ex.submit(_run, port_scan_service.quick_scan, primary_ip)
if check_options.get('mail', True):
mx = dns_data.get('mx_records') if dns_data else None
txt = dns_data.get('txt_records') if dns_data else None
p2['mail'] = ex.submit(_run, mail_service.analyze, domain_name, mx, txt,
check_options.get('smtp_probe', False))
ip_data = p2['ip'].result() if 'ip' in p2 else None
blacklist_data = p2['blacklist'].result() if 'blacklist' in p2 else None
port_data = p2['port'].result() if 'port' in p2 else None
mail_data = p2['mail'].result() if 'mail' in p2 else None
# ----------------------------------------------------------------
# Process results + persist (single-threaded, ordered).
# ----------------------------------------------------------------
# 1. WHOIS
if whois_data:
domain_data['whois'] = whois_data
response_data['whois'] = format_whois_response(whois_data)
# Save WHOIS record
try:
whois_record = WhoisRecord(
domain_id=domain.id,
creation_date=whois_data.get('creation_date'),
expiration_date=whois_data.get('expiration_date'),
updated_date=whois_data.get('updated_date'),
registrar=whois_data.get('registrar'),
registrar_url=whois_data.get('registrar_url'),
registrant_org=whois_data.get('registrant_org'),
registrant_country=whois_data.get('registrant_country'),
admin_email=whois_data.get('admin_email'),
name_servers=whois_data.get('name_servers', []),
status=whois_data.get('status', []),
dnssec=whois_data.get('dnssec'),
raw_whois_data=whois_data.get('raw_data', {}),
data_source=whois_data.get('data_source', 'rdap')
)
db.session.add(whois_record)
except Exception as e:
logger.warning(f'Failed to save WHOIS record: {e}')
# 2. DNS
if dns_data:
domain_data['dns'] = dns_data
response_data['dns'] = format_dns_response(dns_data)
# Save DNS records
try:
for record_type in ['a_records', 'aaaa_records', 'mx_records', 'txt_records', 'ns_records', 'cname_records']:
records = dns_data.get(record_type, [])
if records:
for record in records:
if isinstance(record, dict): # MX records
dns_record = DnsRecord(
domain_id=domain.id,
record_type='MX',
record_value=record['host'],
priority=record['priority']
)
else:
dns_record = DnsRecord(
domain_id=domain.id,
record_type=record_type.replace('_records', '').upper(),
record_value=str(record)
)
db.session.add(dns_record)
except Exception as e:
logger.warning(f'Failed to save DNS records: {e}')
# Registration / availability verdict (WHOIS + DNS combined signals)
if check_options.get('whois', True) or check_options.get('dns', True):
availability = determine_availability(whois_data, dns_data)
response_data['availability'] = availability
if 'whois' in response_data:
response_data['whois']['is_registered'] = availability['is_registered']
response_data['whois']['is_available'] = availability['is_available']
# 3. SSL Certificate
if ssl_data:
domain_data['ssl'] = ssl_data
response_data['ssl'] = format_ssl_response(ssl_data)
if ssl_data.get('has_ssl'):
try:
ssl_cert = SslCertificate(
domain_id=domain.id,
issuer=ssl_data.get('issuer'),
subject=ssl_data.get('subject'),
valid_from=ssl_data.get('valid_from'),
valid_until=ssl_data.get('valid_until'),
serial_number=ssl_data.get('serial_number'),
signature_algorithm=ssl_data.get('signature_algorithm'),
key_size=ssl_data.get('key_size'),
is_wildcard=ssl_data.get('is_wildcard', False),
is_self_signed=ssl_data.get('is_self_signed', False),
is_valid=ssl_data.get('is_valid', True)
)
db.session.add(ssl_cert)
except Exception as e:
logger.warning(f'Failed to save SSL certificate: {e}')
# 4. IP Intelligence
if ip_data:
ip_data['hosting_score'] = ip_service.get_hosting_score(ip_data)
domain_data['ip_intelligence'] = ip_data
response_data['ip_intelligence'] = ip_data
# 5. HTTP Analysis
if http_data:
domain_data['http_analysis'] = http_data
response_data['http_analysis'] = http_data
# 6. Blacklist
if blacklist_data:
domain_data['blacklist'] = blacklist_data
response_data['blacklist'] = blacklist_data
# 7. Mail Intelligence (email infrastructure deep analysis)
if mail_data:
domain_data['mail'] = mail_data
response_data['mail'] = mail_data
# 8. Port Scan (optional)
if port_data:
domain_data['port_scan'] = port_data
response_data['port_scan'] = port_data
# 9. Subdomain Enumeration (optional)
if subdomain_data:
domain_data['subdomains'] = subdomain_data
response_data['subdomains'] = subdomain_data
# 9. Calculate Risk Score
logger.info(f'Calculating risk score for {domain_name}')
risk_assessment_result = risk_scorer.calculate_risk(domain_data)
# Save risk assessment
try:
risk_assessment = RiskAssessment(
domain_id=domain.id,
check_id=check_id,
total_score=risk_assessment_result['total_score'],
risk_level=risk_assessment_result['risk_level'],
domain_age_score=risk_assessment_result['individual_scores'].get('domain_age', 0),
domain_age_days=next((f['details'].get('age_days') for f in risk_assessment_result['factors'] if f['factor'] == 'domain_age'), None),
ssl_score=risk_assessment_result['individual_scores'].get('ssl', 0),
dns_score=risk_assessment_result['individual_scores'].get('dns', 0),
reputation_score=risk_assessment_result['individual_scores'].get('blacklist', 0),
whois_score=risk_assessment_result['individual_scores'].get('whois', 0),
factors=risk_assessment_result['factors'],
is_new_domain=risk_assessment_result['is_new_domain'],
is_suspicious=risk_assessment_result['is_suspicious'],
requires_manual_review=risk_assessment_result['requires_manual_review']
)
db.session.add(risk_assessment)
except Exception as e:
logger.warning(f'Failed to save risk assessment: {e}')
# Save check history
processing_time_ms = int((time.time() - start_time) * 1000)
try:
check_history = CheckHistory(
check_id=check_id,
domain_id=domain.id,
requested_by='api',
request_ip=request.remote_addr,
user_agent=request.headers.get('User-Agent'),
check_options=check_options,
processing_time_ms=processing_time_ms,
cache_hit=False,
status='completed'
)
db.session.add(check_history)
except Exception as e:
logger.warning(f'Failed to save check history: {e}')
# Commit all changes
try:
db.session.commit()
except Exception as e:
logger.error(f'Database commit failed: {e}')
db.session.rollback()
# Build response
response_data['risk_score'] = {
'total': risk_assessment_result['total_score'],
'level': risk_assessment_result['risk_level'],
'factors': risk_assessment_result['factors'],
'formula_breakdown': risk_assessment_result.get('formula_breakdown', []),
'formula_string': risk_assessment_result.get('formula_string', ''),
'thresholds': risk_assessment_result.get('risk_thresholds', {
'low': '0-25',
'medium': '26-50',
'high': '51-75',
'critical': '76-100'
}),
'is_new_domain': risk_assessment_result['is_new_domain'],
'is_suspicious': risk_assessment_result['is_suspicious'],
'is_blacklisted': risk_assessment_result.get('is_blacklisted', False),
'requires_manual_review': risk_assessment_result['requires_manual_review']
}
response = {
'success': True,
'data': response_data,
'metadata': {
'cached': False,
'processing_time_ms': processing_time_ms,
'api_version': current_app.config.get('API_VERSION', 'v1'),
'checks_performed': [k for k, v in check_options.items() if v and k != 'force_refresh']
}
}
return response, 200
except HTTPException:
# Re-raise HTTP exceptions (400, 404, etc.) as-is
raise
except Exception as e:
logger.error(f'Error checking domain: {str(e)}', exc_info=True)
db.session.rollback()
api.abort(500, f'An error occurred while checking the domain: {str(e)}',
success=False,
error={
'code': 'INTERNAL_ERROR',
'message': f'An error occurred while checking the domain: {str(e)}'
})
def get_root_domain(domain: str) -> str:
"""Extract root domain from subdomain"""
parts = domain.split('.')
if len(parts) >= 2:
# Handle common TLDs
common_tlds = ['com', 'org', 'net', 'io', 'co', 'eu', 'ro', 'de', 'uk', 'fr']
if parts[-1] in common_tlds:
return '.'.join(parts[-2:])
# Handle country code TLDs like .co.uk
if len(parts) >= 3 and parts[-2] in ['co', 'com', 'org', 'net', 'gov']:
return '.'.join(parts[-3:])
return domain
def _iso(value):
"""Serialize a datetime to ISO-8601 Z, or None. Coerces stray strings safely."""
dt = _coerce_datetime(value)
return dt.isoformat() + 'Z' if dt else None
def format_whois_response(whois_data: dict) -> dict:
"""Format WHOIS data for API response"""
creation_date = _coerce_datetime(whois_data.get('creation_date'))
expiration_date = _coerce_datetime(whois_data.get('expiration_date'))
age_days = (datetime.utcnow() - creation_date).days if creation_date else None
days_until_expiry = (expiration_date - datetime.utcnow()).days if expiration_date else None
return {
'creation_date': _iso(whois_data.get('creation_date')),
'expiration_date': _iso(whois_data.get('expiration_date')),
'updated_date': _iso(whois_data.get('updated_date')),
'registrar': whois_data.get('registrar'),
'age_days': age_days,
'days_until_expiry': days_until_expiry,
'status': whois_data.get('status', []),
'name_servers': whois_data.get('name_servers', []),
'dnssec': whois_data.get('dnssec'),
'registrant_org': whois_data.get('registrant_org'),
'registrant_country': whois_data.get('registrant_country'),
'is_registered': whois_data.get('is_registered'),
'data_source': whois_data.get('data_source', 'whois')
}
def determine_availability(whois_data: dict, dns_data: dict) -> dict:
"""Combine WHOIS and DNS signals into an explicit registration verdict.
WHOIS alone is unreliable for sparse registries (ROTLD .ro), so DNS
resolution (NS/A records) is used as a strong corroborating signal.
"""
whois_signal = whois_data.get('is_registered') if whois_data else None
dns_has_records = False
if dns_data:
dns_has_records = bool(
dns_data.get('ns_records') or dns_data.get('a_records') or
dns_data.get('aaaa_records') or dns_data.get('mx_records')
)
signals = {
'whois_has_data': whois_signal is True,
'dns_resolves': dns_has_records,
}
# Decision: any positive signal => registered. Confidence is high when
# WHOIS and DNS agree, low when relying on a single weak signal.
if whois_signal is True or dns_has_records:
is_registered = True
confidence = 'high' if (whois_signal is True and dns_has_records) else 'medium'
elif whois_signal is False:
is_registered = False
confidence = 'high' if not dns_has_records else 'low'
else:
# WHOIS inconclusive (None) and no DNS records => probably available.
is_registered = False
confidence = 'low'
return {
'is_registered': is_registered,
'is_available': not is_registered,
'confidence': confidence,
'signals': signals
}
def format_dns_response(dns_data: dict) -> dict:
"""Format DNS data for API response"""
return {
'a_records': dns_data.get('a_records', []),
'aaaa_records': dns_data.get('aaaa_records', []),
'mx_records': dns_data.get('mx_records', []),
'txt_records': dns_data.get('txt_records', []),
'ns_records': dns_data.get('ns_records', []),
'cname_records': dns_data.get('cname_records', []),
'soa_record': dns_data.get('soa_record'),
'has_spf': dns_data.get('has_spf', False),
'has_dkim': dns_data.get('has_dkim', False),
'has_dmarc': dns_data.get('has_dmarc', False),
'spf_record': dns_data.get('spf_record'),
'dmarc_record': dns_data.get('dmarc_record')
}
def format_ssl_response(ssl_data: dict) -> dict:
"""Format SSL data for API response"""
if not ssl_data.get('has_ssl'):
return {
'has_ssl': False,
'error': ssl_data.get('error')
}
valid_from = ssl_data.get('valid_from')
valid_until = ssl_data.get('valid_until')
return {
'has_ssl': True,
'is_valid': ssl_data.get('is_valid', False),
'is_self_signed': ssl_data.get('is_self_signed', False),
'is_expired': ssl_data.get('is_expired', False),
'is_wildcard': ssl_data.get('is_wildcard', False),
'issuer': ssl_data.get('issuer'),
'subject': ssl_data.get('subject'),
'valid_from': valid_from.isoformat() + 'Z' if valid_from and hasattr(valid_from, 'isoformat') else str(valid_from) if valid_from else None,
'valid_until': valid_until.isoformat() + 'Z' if valid_until and hasattr(valid_until, 'isoformat') else str(valid_until) if valid_until else None,
'days_until_expiry': ssl_data.get('days_until_expiry'),
'key_size': ssl_data.get('key_size'),
'signature_algorithm': ssl_data.get('signature_algorithm'),
'san': ssl_data.get('san', [])
}

9
ai_platform/modules/domain_check/api/app/routes/domain.py Normal file
View file

@ -0,0 +1,9 @@
"""Domain routes - stub for now"""
from flask import Blueprint, jsonify
domain_bp = Blueprint('domain', __name__)
@domain_bp.route('/domain/<string:domain>', methods=['GET'])
def get_domain(domain):
"""Get domain details - TODO: implement"""
return jsonify({'message': 'Not yet implemented', 'domain': domain}), 501

9
ai_platform/modules/domain_check/api/app/routes/search.py Normal file
View file

@ -0,0 +1,9 @@
"""Search routes - stub for now"""
from flask import Blueprint, jsonify
search_bp = Blueprint('search', __name__)
@search_bp.route('/search', methods=['GET'])
def search_domains():
"""Search domains - TODO: implement"""
return jsonify({'message': 'Not yet implemented'}), 501

9
ai_platform/modules/domain_check/api/app/routes/stats.py Normal file
View file

@ -0,0 +1,9 @@
"""Stats routes - stub for now"""
from flask import Blueprint, jsonify
stats_bp = Blueprint('stats', __name__)
@stats_bp.route('/stats', methods=['GET'])
def get_stats():
"""Get system statistics - TODO: implement"""
return jsonify({'message': 'Not yet implemented'}), 501

0
ai_platform/modules/domain_check/api/app/services/__init__.py Normal file
View file

297
ai_platform/modules/domain_check/api/app/services/blacklist_service.py Normal file
View file

@ -0,0 +1,297 @@
"""
Blacklist Checking Service - DNSBL, Spam lists, Reputation checks
"""
import logging
import socket
import dns.resolver
import requests
from typing import Dict, List, Optional
from concurrent.futures import ThreadPoolExecutor, as_completed
logger = logging.getLogger(__name__)
class BlacklistService:
"""Service for checking IP/domain against various blacklists"""
def __init__(self):
self.timeout = 5
self.resolver = dns.resolver.Resolver()
self.resolver.nameservers = ['8.8.8.8', '8.8.4.4']
self.resolver.timeout = 3
self.resolver.lifetime = 5
# DNSBL lists to check
self.dnsbl_lists = [
{'name': 'Spamhaus ZEN', 'zone': 'zen.spamhaus.org', 'type': 'spam'},
{'name': 'SpamCop', 'zone': 'bl.spamcop.net', 'type': 'spam'},
{'name': 'Barracuda', 'zone': 'b.barracudacentral.org', 'type': 'spam'},
{'name': 'SORBS', 'zone': 'dnsbl.sorbs.net', 'type': 'spam'},
{'name': 'URIBL', 'zone': 'multi.uribl.com', 'type': 'uri'},
{'name': 'SURBL', 'zone': 'multi.surbl.org', 'type': 'uri'},
{'name': 'Spamhaus DBL', 'zone': 'dbl.spamhaus.org', 'type': 'domain'},
{'name': 'URIBL Black', 'zone': 'black.uribl.com', 'type': 'uri'}
]
def check_ip(self, ip_address: str) -> Dict:
"""
Check IP address against multiple blacklists
Args:
ip_address: IP address to check
Returns:
Dictionary with blacklist check results
"""
result = {
'ip': ip_address,
'is_blacklisted': False,
'blacklist_count': 0,
'clean_count': 0,
'total_checked': 0,
'listings': [],
'clean_lists': [],
'check_errors': []
}
# Reverse IP for DNSBL query
reversed_ip = '.'.join(reversed(ip_address.split('.')))
# Check IP-based blacklists in parallel
ip_lists = [bl for bl in self.dnsbl_lists if bl['type'] in ['spam']]
with ThreadPoolExecutor(max_workers=8) as executor:
futures = {
executor.submit(self._check_dnsbl, reversed_ip, bl): bl
for bl in ip_lists
}
for future in as_completed(futures, timeout=15):
bl = futures[future]
result['total_checked'] += 1
try:
is_listed, response = future.result()
if is_listed:
result['is_blacklisted'] = True
result['blacklist_count'] += 1
result['listings'].append({
'list_name': bl['name'],
'list_zone': bl['zone'],
'response': response,
'type': bl['type']
})
else:
result['clean_count'] += 1
result['clean_lists'].append(bl['name'])
except Exception as e:
result['check_errors'].append({
'list': bl['name'],
'error': str(e)
})
return result
def check_domain(self, domain: str) -> Dict:
"""
Check domain against domain-based blacklists
Args:
domain: Domain name to check
Returns:
Dictionary with blacklist check results
"""
result = {
'domain': domain,
'is_blacklisted': False,
'blacklist_count': 0,
'clean_count': 0,
'total_checked': 0,
'listings': [],
'clean_lists': [],
'check_errors': []
}
# Check domain-based blacklists
domain_lists = [bl for bl in self.dnsbl_lists if bl['type'] in ['domain', 'uri']]
with ThreadPoolExecutor(max_workers=8) as executor:
futures = {
executor.submit(self._check_domain_bl, domain, bl): bl
for bl in domain_lists
}
for future in as_completed(futures, timeout=15):
bl = futures[future]
result['total_checked'] += 1
try:
is_listed, response = future.result()
if is_listed:
result['is_blacklisted'] = True
result['blacklist_count'] += 1
result['listings'].append({
'list_name': bl['name'],
'list_zone': bl['zone'],
'response': response,
'type': bl['type']
})
else:
result['clean_count'] += 1
result['clean_lists'].append(bl['name'])
except Exception as e:
result['check_errors'].append({
'list': bl['name'],
'error': str(e)
})
return result
def check_all(self, domain: str, ip_address: str) -> Dict:
"""
Check both domain and IP against all blacklists
Returns:
Combined blacklist check results
"""
result = {
'domain': domain,
'ip': ip_address,
'is_blacklisted': False,
'ip_blacklisted': False,
'domain_blacklisted': False,
'total_listings': 0,
'ip_check': None,
'domain_check': None,
'reputation_score': 100, # Start with perfect score
'risk_level': 'LOW'
}
# Check IP
if ip_address:
ip_result = self.check_ip(ip_address)
result['ip_check'] = ip_result
result['ip_blacklisted'] = ip_result['is_blacklisted']
result['total_listings'] += ip_result['blacklist_count']
# Check domain
domain_result = self.check_domain(domain)
result['domain_check'] = domain_result
result['domain_blacklisted'] = domain_result['is_blacklisted']
result['total_listings'] += domain_result['blacklist_count']
# Set overall blacklist status
result['is_blacklisted'] = result['ip_blacklisted'] or result['domain_blacklisted']
# Calculate reputation score
result['reputation_score'] = self._calculate_reputation_score(result)
result['risk_level'] = self._get_risk_level(result['reputation_score'])
return result
def _check_dnsbl(self, reversed_ip: str, blacklist: Dict) -> tuple:
"""Check reversed IP against a DNSBL"""
try:
query = f"{reversed_ip}.{blacklist['zone']}"
answers = self.resolver.resolve(query, 'A')
response = str(answers[0])
# 127.0.0.1 is an error code meaning "not authorized to query"
# Real blacklist matches return 127.0.0.2, 127.0.0.4, etc.
if response == '127.0.0.1':
logger.debug(f"DNSBL {blacklist['name']} returned error code 127.0.0.1 (not authorized)")
return False, None
# Valid blacklist match
return True, response
except dns.resolver.NXDOMAIN:
# Not listed
return False, None
except dns.resolver.NoAnswer:
return False, None
except dns.resolver.Timeout:
raise Exception('Timeout')
except Exception as e:
raise Exception(str(e))
def _check_domain_bl(self, domain: str, blacklist: Dict) -> tuple:
"""Check domain against a domain blacklist"""
try:
query = f"{domain}.{blacklist['zone']}"
answers = self.resolver.resolve(query, 'A')
response = str(answers[0])
# 127.0.0.1 is an error code meaning "not authorized to query"
# Real blacklist matches return 127.0.0.2, 127.0.0.4, etc.
# This is common with URIBL when querying from unregistered resolvers
if response == '127.0.0.1':
logger.debug(f"Domain BL {blacklist['name']} returned error code 127.0.0.1 (not authorized)")
return False, None
# Valid blacklist match
return True, response
except dns.resolver.NXDOMAIN:
return False, None
except dns.resolver.NoAnswer:
return False, None
except dns.resolver.Timeout:
raise Exception('Timeout')
except Exception as e:
raise Exception(str(e))
def _calculate_reputation_score(self, result: Dict) -> int:
"""
Calculate reputation score (100 = perfect, 0 = worst)
Each blacklist listing reduces the score
"""
score = 100
# IP blacklist hits are more severe
if result.get('ip_check'):
ip_listings = result['ip_check'].get('blacklist_count', 0)
score -= ip_listings * 25 # -25 per IP listing
# Domain blacklist hits
if result.get('domain_check'):
domain_listings = result['domain_check'].get('blacklist_count', 0)
score -= domain_listings * 20 # -20 per domain listing
return max(0, min(100, score))
def _get_risk_level(self, score: int) -> str:
"""Determine risk level from reputation score"""
if score >= 80:
return 'LOW'
elif score >= 60:
return 'MEDIUM'
elif score >= 40:
return 'HIGH'
else:
return 'CRITICAL'
def get_blacklist_score(self, blacklist_data: Dict) -> Dict:
"""
Get blacklist score for risk calculation (0 = good, 100 = bad)
This inverts the reputation score for consistency with other risk scores
"""
reputation = blacklist_data.get('reputation_score', 100)
score = 100 - reputation # Invert: 100 reputation = 0 risk
reasons = []
if blacklist_data.get('is_blacklisted'):
if blacklist_data.get('ip_blacklisted'):
reasons.append(f"IP on {blacklist_data['ip_check']['blacklist_count']} blacklist(s)")
if blacklist_data.get('domain_blacklisted'):
reasons.append(f"Domain on {blacklist_data['domain_check']['blacklist_count']} blacklist(s)")
else:
reasons.append('Not on any blacklists')
return {
'score': score,
'reasons': reasons,
'is_clean': score == 0,
'is_blacklisted': blacklist_data.get('is_blacklisted', False)
}

163
ai_platform/modules/domain_check/api/app/services/dns_service.py Normal file
View file

@ -0,0 +1,163 @@
"""
DNS Service - handles DNS record lookups
"""
import logging
import dns.resolver
from typing import Dict, List, Optional
logger = logging.getLogger(__name__)
class DNSService:
"""DNS lookup service class"""
def __init__(self, timeout: int = 10):
self.timeout = timeout
self.resolver = dns.resolver.Resolver()
self.resolver.timeout = timeout
self.resolver.lifetime = timeout
# Use public DNS servers (Google DNS) for reliability
self.resolver.nameservers = ['8.8.8.8', '8.8.4.4']
def lookup(self, domain: str) -> Optional[Dict]:
"""
Perform comprehensive DNS lookup
Args:
domain: Domain name to lookup
Returns:
Dictionary with DNS records or None
"""
try:
logger.info(f'DNS lookup for: {domain}')
dns_data = {
'domain': domain,
'a_records': self._get_a_records(domain),
'aaaa_records': self._get_aaaa_records(domain),
'mx_records': self._get_mx_records(domain),
'txt_records': self._get_txt_records(domain),
'ns_records': self._get_ns_records(domain),
'cname_records': self._get_cname_records(domain),
'soa_record': self._get_soa_record(domain),
'has_a_records': False,
'has_mx_records': False,
'has_txt_records': False,
'has_spf': False,
'has_dkim': False,
'has_dmarc': False
}
# Set boolean flags
dns_data['has_a_records'] = len(dns_data['a_records']) > 0
dns_data['has_mx_records'] = len(dns_data['mx_records']) > 0
dns_data['has_txt_records'] = len(dns_data['txt_records']) > 0
# Check for email security records
for txt in dns_data['txt_records']:
if txt.startswith('v=spf1'):
dns_data['has_spf'] = True
if 'dkim' in txt.lower():
dns_data['has_dkim'] = True
if 'v=DMARC1' in txt:
dns_data['has_dmarc'] = True
return dns_data
except Exception as e:
logger.error(f'DNS lookup failed for {domain}: {str(e)}')
return None
def _get_a_records(self, domain: str) -> List[str]:
"""Get A records (IPv4)"""
try:
answers = self.resolver.resolve(domain, 'A')
return [str(rdata) for rdata in answers]
except Exception as e:
logger.debug(f'No A records for {domain}: {e}')
return []
def _get_aaaa_records(self, domain: str) -> List[str]:
"""Get AAAA records (IPv6)"""
try:
answers = self.resolver.resolve(domain, 'AAAA')
return [str(rdata) for rdata in answers]
except Exception as e:
logger.debug(f'No AAAA records for {domain}: {e}')
return []
def _get_mx_records(self, domain: str) -> List[Dict]:
"""Get MX records (Mail servers)"""
try:
answers = self.resolver.resolve(domain, 'MX')
return [
{
'priority': rdata.preference,
'host': str(rdata.exchange).rstrip('.')
}
for rdata in answers
]
except Exception as e:
logger.debug(f'No MX records for {domain}: {e}')
return []
def _get_txt_records(self, domain: str) -> List[str]:
"""Get TXT records"""
try:
answers = self.resolver.resolve(domain, 'TXT')
records = []
for rdata in answers:
# TXT records can be split into multiple strings
txt = ''.join([s.decode('utf-8') if isinstance(s, bytes) else str(s) for s in rdata.strings])
records.append(txt)
return records
except Exception as e:
logger.debug(f'No TXT records for {domain}: {e}')
return []
def _get_ns_records(self, domain: str) -> List[str]:
"""Get NS records (Name servers)"""
try:
answers = self.resolver.resolve(domain, 'NS')
return [str(rdata).rstrip('.') for rdata in answers]
except Exception as e:
logger.debug(f'No NS records for {domain}: {e}')
return []
def _get_cname_records(self, domain: str) -> List[str]:
"""Get CNAME records"""
try:
answers = self.resolver.resolve(domain, 'CNAME')
return [str(rdata).rstrip('.') for rdata in answers]
except Exception as e:
logger.debug(f'No CNAME records for {domain}: {e}')
return []
def _get_soa_record(self, domain: str) -> Optional[Dict]:
"""Get SOA record (Start of Authority)"""
try:
answers = self.resolver.resolve(domain, 'SOA')
if answers:
soa = answers[0]
return {
'mname': str(soa.mname).rstrip('.'),
'rname': str(soa.rname).rstrip('.'),
'serial': soa.serial,
'refresh': soa.refresh,
'retry': soa.retry,
'expire': soa.expire,
'minimum': soa.minimum
}
except Exception as e:
logger.debug(f'No SOA record for {domain}: {e}')
return None
def check_dnssec(self, domain: str) -> bool:
"""Check if DNSSEC is enabled"""
try:
# Try to get DNSKEY records
self.resolver.resolve(domain, 'DNSKEY')
return True
except Exception:
return False

369
ai_platform/modules/domain_check/api/app/services/http_analysis_service.py Normal file
View file

@ -0,0 +1,369 @@
"""
HTTP Analysis Service - Headers, Technology Detection, Security Headers
"""
import logging
import re
import requests
from typing import Dict, List, Optional
from urllib.parse import urlparse
logger = logging.getLogger(__name__)
class HTTPAnalysisService:
"""Service for analyzing HTTP responses and detecting technologies"""
def __init__(self):
self.timeout = 15
self.user_agent = 'Mozilla/5.0 (compatible; DomainCheck/1.0; +https://domain-check.local)'
def analyze(self, domain: str) -> Dict:
"""
Perform comprehensive HTTP analysis
Args:
domain: Domain name to analyze
Returns:
Dictionary with HTTP analysis results
"""
result = {
'domain': domain,
'http_status': None,
'https_status': None,
'has_https': False,
'http_to_https_redirect': False,
'final_url': None,
'redirect_chain': [],
'response_time_ms': None,
'server': None,
'powered_by': None,
'security_headers': {},
'missing_security_headers': [],
'cookies': [],
'technologies': [],
'cms': None,
'frameworks': [],
'has_robots_txt': False,
'has_sitemap': False,
'has_favicon': False,
'error': None
}
try:
# Test HTTPS first
https_result = self._check_url(f'https://{domain}')
if https_result.get('success'):
result['has_https'] = True
result['https_status'] = https_result.get('status_code')
result['final_url'] = https_result.get('final_url')
result['redirect_chain'] = https_result.get('redirect_chain', [])
result['response_time_ms'] = https_result.get('response_time_ms')
result['server'] = https_result.get('server')
result['powered_by'] = https_result.get('powered_by')
result['security_headers'] = https_result.get('security_headers', {})
result['missing_security_headers'] = https_result.get('missing_security_headers', [])
result['cookies'] = https_result.get('cookies', [])
# Detect technologies from response
if https_result.get('body'):
tech = self._detect_technologies(https_result['body'], https_result.get('headers', {}))
result['technologies'] = tech.get('technologies', [])
result['cms'] = tech.get('cms')
result['frameworks'] = tech.get('frameworks', [])
# Test HTTP
http_result = self._check_url(f'http://{domain}')
if http_result.get('success'):
result['http_status'] = http_result.get('status_code')
# Check if HTTP redirects to HTTPS
if http_result.get('final_url', '').startswith('https://'):
result['http_to_https_redirect'] = True
# Check for common files
result['has_robots_txt'] = self._check_file_exists(f'https://{domain}/robots.txt')
result['has_sitemap'] = self._check_file_exists(f'https://{domain}/sitemap.xml')
result['has_favicon'] = self._check_file_exists(f'https://{domain}/favicon.ico')
except Exception as e:
logger.error(f"HTTP analysis failed for {domain}: {e}")
result['error'] = str(e)
return result
def _check_url(self, url: str) -> Dict:
"""Check a URL and gather response data"""
result = {
'success': False,
'url': url,
'status_code': None,
'final_url': None,
'redirect_chain': [],
'response_time_ms': None,
'server': None,
'powered_by': None,
'security_headers': {},
'missing_security_headers': [],
'cookies': [],
'headers': {},
'body': None
}
try:
response = requests.get(
url,
timeout=self.timeout,
allow_redirects=True,
headers={'User-Agent': self.user_agent},
verify=True
)
result['success'] = True
result['status_code'] = response.status_code
result['final_url'] = response.url
result['response_time_ms'] = int(response.elapsed.total_seconds() * 1000)
result['headers'] = dict(response.headers)
# Get redirect chain
if response.history:
result['redirect_chain'] = [
{'url': r.url, 'status': r.status_code}
for r in response.history
]
# Extract server info
result['server'] = response.headers.get('Server')
result['powered_by'] = response.headers.get('X-Powered-By')
# Analyze security headers
result['security_headers'], result['missing_security_headers'] = \
self._analyze_security_headers(response.headers)
# Analyze cookies
result['cookies'] = self._analyze_cookies(response.cookies)
# Get body for technology detection
result['body'] = response.text[:50000] # Limit body size
except requests.exceptions.SSLError as e:
result['error'] = f'SSL Error: {str(e)}'
except requests.exceptions.ConnectionError as e:
result['error'] = f'Connection Error: {str(e)}'
except requests.exceptions.Timeout:
result['error'] = 'Timeout'
except Exception as e:
result['error'] = str(e)
return result
def _analyze_security_headers(self, headers) -> tuple:
"""Analyze security headers"""
security_headers = {}
missing = []
# Define required security headers
required_headers = {
'Strict-Transport-Security': 'HSTS - Forces HTTPS',
'X-Frame-Options': 'Prevents clickjacking',
'X-Content-Type-Options': 'Prevents MIME sniffing',
'X-XSS-Protection': 'XSS filtering (legacy)',
'Content-Security-Policy': 'CSP - Controls resource loading',
'Referrer-Policy': 'Controls referrer information',
'Permissions-Policy': 'Controls browser features'
}
for header, description in required_headers.items():
value = headers.get(header)
if value:
security_headers[header] = {
'value': value,
'description': description,
'present': True
}
else:
missing.append({
'header': header,
'description': description,
'severity': self._get_header_severity(header)
})
return security_headers, missing
def _get_header_severity(self, header: str) -> str:
"""Get severity level for missing header"""
critical = ['Strict-Transport-Security', 'Content-Security-Policy']
high = ['X-Frame-Options', 'X-Content-Type-Options']
if header in critical:
return 'CRITICAL'
elif header in high:
return 'HIGH'
return 'MEDIUM'
def _analyze_cookies(self, cookies) -> List[Dict]:
"""Analyze cookies for security attributes"""
analyzed = []
for cookie in cookies:
cookie_info = {
'name': cookie.name,
'secure': cookie.secure,
'httponly': cookie.has_nonstandard_attr('HttpOnly'),
'samesite': cookie.get_nonstandard_attr('SameSite'),
'issues': []
}
# Check for security issues
if not cookie.secure:
cookie_info['issues'].append('Missing Secure flag')
if not cookie_info['httponly']:
cookie_info['issues'].append('Missing HttpOnly flag')
if not cookie_info['samesite']:
cookie_info['issues'].append('Missing SameSite attribute')
analyzed.append(cookie_info)
return analyzed
def _detect_technologies(self, body: str, headers: Dict) -> Dict:
"""Detect technologies from response body and headers"""
result = {
'technologies': [],
'cms': None,
'frameworks': []
}
body_lower = body.lower()
# CMS Detection
cms_patterns = {
'WordPress': [
'wp-content', 'wp-includes', 'wordpress',
'<meta name="generator" content="WordPress'
],
'Joomla': ['joomla', '/media/jui/', '/components/com_'],
'Drupal': ['drupal', '/sites/default/files/', 'Drupal.settings'],
'Magento': ['magento', 'mage/', '/skin/frontend/'],
'Shopify': ['shopify', 'cdn.shopify.com'],
'Wix': ['wix.com', '_wix_browser_sess'],
'Squarespace': ['squarespace', 'static.squarespace.com']
}
for cms, patterns in cms_patterns.items():
for pattern in patterns:
if pattern.lower() in body_lower:
result['cms'] = cms
result['technologies'].append(cms)
break
if result['cms']:
break
# Framework detection
framework_patterns = {
'React': ['react', '_reactRootContainer', 'data-reactroot'],
'Vue.js': ['vue', 'data-v-', '__vue__'],
'Angular': ['ng-', 'angular', 'ng-app', 'ng-controller'],
'jQuery': ['jquery', 'jQuery'],
'Bootstrap': ['bootstrap', 'class="container', 'class="row'],
'Tailwind CSS': ['tailwind', 'class="flex', 'class="grid'],
'Laravel': ['laravel', 'csrf-token'],
'Django': ['csrfmiddlewaretoken', 'django'],
'Express': ['express'],
'Next.js': ['next.js', '__NEXT_DATA__', '_next/'],
'Nuxt.js': ['nuxt', '__NUXT__']
}
for framework, patterns in framework_patterns.items():
for pattern in patterns:
if pattern.lower() in body_lower:
if framework not in result['frameworks']:
result['frameworks'].append(framework)
result['technologies'].append(framework)
break
# Additional technologies from meta tags
generator_match = re.search(r'<meta[^>]+name=["\']generator["\'][^>]+content=["\']([^"\']+)["\']', body, re.I)
if generator_match:
generator = generator_match.group(1)
result['technologies'].append(f'Generator: {generator}')
# Server-side detection from headers
server = headers.get('Server', '').lower()
if 'nginx' in server:
result['technologies'].append('Nginx')
elif 'apache' in server:
result['technologies'].append('Apache')
elif 'iis' in server:
result['technologies'].append('Microsoft IIS')
powered_by = headers.get('X-Powered-By', '').lower()
if 'php' in powered_by:
result['technologies'].append('PHP')
elif 'asp.net' in powered_by:
result['technologies'].append('ASP.NET')
return result
def _check_file_exists(self, url: str) -> bool:
"""Check if a file exists at URL"""
try:
response = requests.head(
url,
timeout=5,
allow_redirects=True,
headers={'User-Agent': self.user_agent}
)
return response.status_code == 200
except:
return False
def get_security_score(self, http_data: Dict) -> Dict:
"""
Calculate HTTP security score
Returns:
Dict with score (0-100) and reasons
"""
score = 0
reasons = []
# No HTTPS = very bad
if not http_data.get('has_https'):
score += 40
reasons.append('No HTTPS support')
# No HTTP to HTTPS redirect
if http_data.get('has_https') and not http_data.get('http_to_https_redirect'):
score += 15
reasons.append('No HTTP to HTTPS redirect')
# Missing security headers
missing_headers = http_data.get('missing_security_headers', [])
for header in missing_headers:
if header.get('severity') == 'CRITICAL':
score += 15
reasons.append(f"Missing: {header['header']}")
elif header.get('severity') == 'HIGH':
score += 10
reasons.append(f"Missing: {header['header']}")
else:
score += 5
# Cookie issues
cookies = http_data.get('cookies', [])
for cookie in cookies:
if cookie.get('issues'):
score += 5 * len(cookie['issues'])
for issue in cookie['issues']:
reasons.append(f"Cookie '{cookie['name']}': {issue}")
if score == 0:
reasons.append('Good HTTP security configuration')
return {
'score': min(100, score),
'reasons': reasons[:10], # Limit reasons
'is_secure': score <= 20,
'needs_improvement': score > 20 and score <= 50,
'is_insecure': score > 50
}

233
ai_platform/modules/domain_check/api/app/services/ip_intelligence_service.py Normal file
View file

@ -0,0 +1,233 @@
"""
IP Intelligence Service - Geolocation, ASN, Reverse DNS, Hosting Info
"""
import logging
import socket
import requests
from typing import Dict, Optional
from concurrent.futures import ThreadPoolExecutor, as_completed
logger = logging.getLogger(__name__)
class IPIntelligenceService:
"""Service for gathering IP intelligence data"""
def __init__(self):
self.timeout = 10
self.ipinfo_url = "https://ipinfo.io/{ip}/json"
def lookup(self, ip_address: str) -> Dict:
"""
Get comprehensive IP intelligence
Args:
ip_address: IP address to lookup
Returns:
Dictionary with IP intelligence data
"""
result = {
'ip': ip_address,
'reverse_dns': None,
'geolocation': None,
'asn': None,
'isp': None,
'organization': None,
'is_datacenter': False,
'is_residential': False,
'hostname': None,
'city': None,
'region': None,
'country': None,
'country_code': None,
'coordinates': None,
'timezone': None,
'data_source': 'ipinfo.io'
}
try:
# Run lookups in parallel
with ThreadPoolExecutor(max_workers=2) as executor:
futures = {
executor.submit(self._get_reverse_dns, ip_address): 'reverse_dns',
executor.submit(self._get_ipinfo, ip_address): 'ipinfo'
}
for future in as_completed(futures, timeout=15):
lookup_type = futures[future]
try:
data = future.result()
if lookup_type == 'reverse_dns':
result['reverse_dns'] = data
result['hostname'] = data
elif lookup_type == 'ipinfo' and data:
result.update(data)
except Exception as e:
logger.warning(f"IP lookup {lookup_type} failed: {e}")
# Determine if datacenter or residential
result['is_datacenter'] = self._is_datacenter(result)
result['is_residential'] = not result['is_datacenter']
except Exception as e:
logger.error(f"IP intelligence lookup failed for {ip_address}: {e}")
result['error'] = str(e)
return result
def _get_reverse_dns(self, ip_address: str) -> Optional[str]:
"""Get reverse DNS (PTR record)"""
try:
hostname, _, _ = socket.gethostbyaddr(ip_address)
return hostname
except (socket.herror, socket.gaierror):
return None
def _get_ipinfo(self, ip_address: str) -> Optional[Dict]:
"""Get IP info from ipinfo.io"""
try:
response = requests.get(
self.ipinfo_url.format(ip=ip_address),
timeout=self.timeout
)
if response.status_code == 200:
data = response.json()
# Parse ASN from org field (format: "AS8708 DIGI ROMANIA S.A.")
org = data.get('org', '')
asn = None
isp = None
if org:
parts = org.split(' ', 1)
if parts[0].startswith('AS'):
asn = parts[0]
isp = parts[1] if len(parts) > 1 else None
else:
isp = org
# Parse coordinates
loc = data.get('loc', '')
coordinates = None
if loc:
try:
lat, lon = loc.split(',')
coordinates = {
'latitude': float(lat),
'longitude': float(lon)
}
except:
pass
return {
'hostname': data.get('hostname'),
'city': data.get('city'),
'region': data.get('region'),
'country': self._get_country_name(data.get('country')),
'country_code': data.get('country'),
'coordinates': coordinates,
'timezone': data.get('timezone'),
'asn': asn,
'isp': isp,
'organization': isp,
'postal': data.get('postal')
}
except Exception as e:
logger.warning(f"ipinfo.io lookup failed: {e}")
return None
def _is_datacenter(self, data: Dict) -> bool:
"""Determine if IP is likely a datacenter/hosting IP"""
indicators = []
# Check ISP/organization for hosting keywords
org = (data.get('organization') or '').lower()
isp = (data.get('isp') or '').lower()
hostname = (data.get('hostname') or '').lower()
hosting_keywords = [
'hosting', 'server', 'cloud', 'datacenter', 'data center',
'hetzner', 'ovh', 'digitalocean', 'linode', 'vultr', 'aws',
'amazon', 'google', 'microsoft', 'azure', 'contabo', 'hostinger',
'godaddy', 'bluehost', 'namecheap', 'maghost', 'simpliq', 'host365'
]
for keyword in hosting_keywords:
if keyword in org or keyword in isp:
return True
# Check hostname patterns
if hostname:
hosting_patterns = ['static', 'vps', 'server', 'host', 'cloud', 'dedicated']
for pattern in hosting_patterns:
if pattern in hostname:
return True
return False
def _get_country_name(self, code: str) -> str:
"""Convert country code to name"""
countries = {
'RO': 'Romania',
'US': 'United States',
'GB': 'United Kingdom',
'DE': 'Germany',
'FR': 'France',
'NL': 'Netherlands',
'UA': 'Ukraine',
'RU': 'Russia',
'CN': 'China',
'IN': 'India',
'JP': 'Japan',
'BR': 'Brazil',
'CA': 'Canada',
'AU': 'Australia'
}
return countries.get(code, code)
def get_hosting_score(self, ip_data: Dict) -> Dict:
"""
Calculate hosting provider reputation score
Returns:
Dict with score (0-100) and reasons
"""
score = 50 # Neutral baseline
reasons = []
isp = (ip_data.get('isp') or '').lower()
country_code = ip_data.get('country_code', '')
# Trusted hosting providers (lower score = better)
trusted_hosts = ['google', 'amazon', 'microsoft', 'cloudflare', 'akamai']
for host in trusted_hosts:
if host in isp:
score = 20
reasons.append(f'Trusted provider: {isp}')
break
# Romanian ISPs (neutral)
ro_isps = ['digi', 'rcs', 'rds', 'telekom', 'orange', 'vodafone']
for isp_name in ro_isps:
if isp_name in isp:
score = 40
reasons.append(f'Romanian ISP: {isp}')
break
# Check for reverse DNS mismatch (higher risk)
if ip_data.get('is_datacenter') and not ip_data.get('reverse_dns'):
score += 20
reasons.append('Datacenter IP without reverse DNS')
# High-risk countries
high_risk_countries = ['RU', 'CN', 'KP', 'IR', 'NG']
if country_code in high_risk_countries:
score += 30
reasons.append(f'High-risk country: {country_code}')
return {
'score': min(100, max(0, score)),
'reasons': reasons,
'is_trusted': score <= 30,
'is_suspicious': score >= 70
}

374
ai_platform/modules/domain_check/api/app/services/mail_intelligence_service.py Normal file
View file

@ -0,0 +1,374 @@
"""
Mail Intelligence Service - deep analysis of a domain's email infrastructure.
Covers what a plain MX/TXT dump does not:
- SPF parsing (policy qualifier, include chain, DNS-lookup budget per RFC 7208)
- DMARC policy (_dmarc) parsing (p/sp/pct/rua/aspf/adkim)
- DKIM selector discovery (probes common selectors)
- MX provider fingerprinting + STARTTLS reachability
- MTA-STS (RFC 8461), TLS-RPT (RFC 8460), DANE/TLSA (RFC 7672)
- Optional SMTP-level mailbox + catch-all probing (RCPT TO), degrades
gracefully when outbound port 25 is blocked.
Everything is best-effort and never raises: a failed probe becomes a recorded
'unknown'/False signal, so a single hiccup cannot 500 the parent request.
"""
import logging
import smtplib
import socket
import ssl
from concurrent.futures import ThreadPoolExecutor, as_completed
from typing import Dict, List, Optional
import dns.resolver
import requests
logger = logging.getLogger(__name__)
# Common DKIM selectors used by major providers / tooling.
COMMON_DKIM_SELECTORS = [
'default', 'google', 'selector1', 'selector2', 'k1', 'k2', 'dkim',
'mail', 'smtp', 's1', 's2', 'mandrill', 'mailjet', 'sendgrid',
'zoho', 'protonmail', 'protonmail2', 'fm1', 'fm2', 'fm3', 'mxvault',
]
# MX hostname substrings -> human provider name.
MX_PROVIDERS = [
('google.com', 'Google Workspace'),
('googlemail.com', 'Google Workspace'),
('outlook.com', 'Microsoft 365'),
('protection.outlook', 'Microsoft 365'),
('zoho', 'Zoho Mail'),
('protonmail', 'Proton Mail'),
('proton.me', 'Proton Mail'),
('mail.ru', 'Mail.ru'),
('yandex', 'Yandex Mail'),
('mimecast', 'Mimecast'),
('pphosted', 'Proofpoint'),
('messagelabs', 'Broadcom/Symantec'),
('mailgun', 'Mailgun'),
('sendgrid', 'SendGrid'),
('amazonaws', 'Amazon SES/WorkMail'),
('secureserver.net', 'GoDaddy'),
('one.com', 'one.com'),
('hostinger', 'Hostinger'),
('gandi', 'Gandi'),
('ovh', 'OVH'),
('rotld', 'ROTLD'),
]
class MailIntelligenceService:
"""Deep email-infrastructure analysis for a domain."""
def __init__(self, dns_timeout: int = 6, smtp_timeout: int = 8, http_timeout: int = 6):
self.smtp_timeout = smtp_timeout
self.http_timeout = http_timeout
self.resolver = dns.resolver.Resolver()
self.resolver.timeout = dns_timeout
self.resolver.lifetime = dns_timeout
self.resolver.nameservers = ['8.8.8.8', '1.1.1.1']
# -- public ------------------------------------------------------------
def analyze(self, domain: str, mx_records: Optional[List[Dict]] = None,
txt_records: Optional[List[str]] = None,
check_smtp: bool = False) -> Dict:
domain = domain.lower().strip()
if mx_records is None:
mx_records = self._resolve_mx(domain)
if txt_records is None:
txt_records = self._resolve_txt(domain)
spf = self._analyze_spf(domain, txt_records)
dmarc = self._analyze_dmarc(domain)
dkim = self._discover_dkim(domain)
mta_sts = self._analyze_mta_sts(domain)
tls_rpt = self._analyze_tls_rpt(domain)
mx = self._analyze_mx(mx_records, check_smtp=check_smtp)
result = {
'domain': domain,
'has_mail': bool(mx_records),
'mx': mx,
'spf': spf,
'dmarc': dmarc,
'dkim': dkim,
'mta_sts': mta_sts,
'tls_rpt': tls_rpt,
}
if check_smtp:
result['deliverability'] = self._probe_deliverability(domain, mx_records)
result['provider'] = mx.get('provider') if mx else None
result.update(self._grade(result))
return result
# -- DNS helpers -------------------------------------------------------
def _resolve_mx(self, domain: str) -> List[Dict]:
try:
answers = self.resolver.resolve(domain, 'MX')
return sorted(
[{'priority': r.preference, 'host': str(r.exchange).rstrip('.')} for r in answers],
key=lambda m: m['priority']
)
except Exception:
return []
def _resolve_txt(self, name: str) -> List[str]:
try:
answers = self.resolver.resolve(name, 'TXT')
out = []
for r in answers:
out.append(''.join(
s.decode('utf-8', 'ignore') if isinstance(s, bytes) else str(s)
for s in r.strings
))
return out
except Exception:
return []
# -- SPF ---------------------------------------------------------------
def _analyze_spf(self, domain: str, txt_records: List[str]) -> Dict:
spf_record = next((t for t in (txt_records or []) if t.lower().startswith('v=spf1')), None)
if not spf_record:
return {'present': False, 'record': None, 'policy': None,
'lookup_count': 0, 'includes': [], 'issues': ['No SPF record']}
tokens = spf_record.split()
includes, lookup_terms, policy = [], 0, 'neutral'
# Mechanisms that cost a DNS lookup (RFC 7208 §4.6.4, max 10).
lookup_mechs = ('include:', 'a', 'mx', 'ptr', 'exists:', 'redirect=')
for tok in tokens[1:]:
low = tok.lower()
if low.startswith('include:'):
includes.append(tok.split(':', 1)[1])
lookup_terms += 1
elif low.startswith(lookup_mechs) or low in ('a', 'mx', 'ptr'):
lookup_terms += 1
if low.endswith('all'):
qual = low[0] if low[0] in '-~?+' else '+'
policy = {'-': 'fail (strict)', '~': 'softfail', '?': 'neutral',
'+': 'pass (insecure +all)'}.get(qual, 'neutral')
issues = []
if lookup_terms > 10:
issues.append(f'Exceeds 10 DNS-lookup limit ({lookup_terms}) -> SPF permerror')
if policy.startswith('pass'):
issues.append('+all allows anyone to send as this domain')
if policy == 'neutral':
issues.append('?all provides no protection')
return {'present': True, 'record': spf_record, 'policy': policy,
'lookup_count': lookup_terms, 'includes': includes, 'issues': issues}
# -- DMARC -------------------------------------------------------------
def _analyze_dmarc(self, domain: str) -> Dict:
records = self._resolve_txt(f'_dmarc.{domain}')
rec = next((t for t in records if t.lower().startswith('v=dmarc1')), None)
if not rec:
return {'present': False, 'record': None, 'policy': None,
'pct': None, 'rua': [], 'issues': ['No DMARC record']}
tags = {}
for part in rec.split(';'):
if '=' in part:
k, v = part.split('=', 1)
tags[k.strip().lower()] = v.strip()
policy = tags.get('p')
issues = []
if policy == 'none':
issues.append('p=none is monitor-only, does not block spoofing')
if not policy:
issues.append('Missing p= tag')
if not tags.get('rua'):
issues.append('No aggregate reporting (rua) configured')
return {'present': True, 'record': rec, 'policy': policy,
'subdomain_policy': tags.get('sp'),
'pct': tags.get('pct', '100'),
'rua': [a.strip() for a in tags.get('rua', '').split(',') if a.strip()],
'alignment': {'aspf': tags.get('aspf', 'r'), 'adkim': tags.get('adkim', 'r')},
'issues': issues}
# -- DKIM --------------------------------------------------------------
def _discover_dkim(self, domain: str) -> Dict:
found = []
def probe(sel):
recs = self._resolve_txt(f'{sel}._domainkey.{domain}')
for r in recs:
if 'v=dkim1' in r.lower() or 'p=' in r:
return sel
return None
with ThreadPoolExecutor(max_workers=8) as ex:
futures = {ex.submit(probe, s): s for s in COMMON_DKIM_SELECTORS}
for fut in as_completed(futures):
try:
sel = fut.result()
if sel:
found.append(sel)
except Exception:
pass
return {'present': bool(found), 'selectors_found': sorted(found),
'note': 'Probes common selectors only; absence is not proof of no DKIM'}
# -- MTA-STS / TLS-RPT / DANE ------------------------------------------
def _analyze_mta_sts(self, domain: str) -> Dict:
txt = self._resolve_txt(f'_mta-sts.{domain}')
has_dns = any('v=stsv1' in t.lower() for t in txt)
policy = None
mode = None
if has_dns:
try:
resp = requests.get(
f'https://mta-sts.{domain}/.well-known/mta-sts.txt',
timeout=self.http_timeout, allow_redirects=False
)
if resp.status_code == 200 and 'version' in resp.text.lower():
policy = resp.text[:2000]
for line in resp.text.splitlines():
if line.lower().startswith('mode:'):
mode = line.split(':', 1)[1].strip()
except Exception:
pass
return {'present': has_dns, 'mode': mode, 'policy_fetched': policy is not None}
def _analyze_tls_rpt(self, domain: str) -> Dict:
txt = self._resolve_txt(f'_smtp._tls.{domain}')
rec = next((t for t in txt if 'v=tlsrptv1' in t.lower()), None)
return {'present': rec is not None, 'record': rec}
def _check_tlsa(self, mx_host: str) -> bool:
try:
self.resolver.resolve(f'_25._tcp.{mx_host}', 'TLSA')
return True
except Exception:
return False
# -- MX analysis -------------------------------------------------------
def _provider_for(self, host: str) -> Optional[str]:
h = host.lower()
for needle, name in MX_PROVIDERS:
if needle in h:
return name
return None
def _analyze_mx(self, mx_records: List[Dict], check_smtp: bool) -> Dict:
if not mx_records:
return {'count': 0, 'hosts': [], 'provider': None, 'starttls': None}
provider = None
hosts_out = []
def inspect(mx):
host = mx['host']
info = {
'host': host, 'priority': mx['priority'],
'provider': self._provider_for(host),
'addresses': self._resolve_addresses(host),
'dane_tlsa': self._check_tlsa(host),
}
if check_smtp:
info.update(self._smtp_starttls(host))
return info
with ThreadPoolExecutor(max_workers=min(5, len(mx_records))) as ex:
for info in ex.map(inspect, mx_records[:5]):
hosts_out.append(info)
if not provider and info.get('provider'):
provider = info['provider']
starttls = None
if check_smtp:
tls_flags = [h.get('starttls') for h in hosts_out if 'starttls' in h]
if tls_flags:
starttls = all(tls_flags)
return {'count': len(mx_records), 'hosts': hosts_out,
'provider': provider, 'starttls': starttls,
'dane': any(h.get('dane_tlsa') for h in hosts_out)}
def _resolve_addresses(self, host: str) -> List[str]:
out = []
for rtype in ('A', 'AAAA'):
try:
out += [str(r) for r in self.resolver.resolve(host, rtype)]
except Exception:
pass
return out
def _smtp_starttls(self, host: str) -> Dict:
"""Connect on :25 and check STARTTLS. Degrades gracefully if blocked."""
try:
with smtplib.SMTP(host, 25, timeout=self.smtp_timeout) as smtp:
banner = smtp.ehlo()
supports_tls = smtp.has_extn('starttls')
tls_ok = False
if supports_tls:
try:
ctx = ssl.create_default_context()
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE
smtp.starttls(context=ctx)
tls_ok = True
except Exception:
tls_ok = False
return {'reachable': True, 'starttls': supports_tls,
'starttls_negotiated': tls_ok,
'banner_code': banner[0] if banner else None}
except (socket.timeout, ConnectionRefusedError, OSError) as e:
return {'reachable': False, 'starttls': False,
'reason': f'port 25 unreachable from server ({type(e).__name__})'}
# -- optional deliverability probe ------------------------------------
def _probe_deliverability(self, domain: str, mx_records: List[Dict]) -> Dict:
"""SMTP RCPT probe for catch-all detection. Best-effort; many networks
block outbound :25 and many servers grey-list, so results are advisory."""
if not mx_records:
return {'tested': False, 'reason': 'no MX'}
host = mx_records[0]['host']
try:
with smtplib.SMTP(host, 25, timeout=self.smtp_timeout) as smtp:
smtp.ehlo()
smtp.mail('probe@example.com')
# Random-looking address: if accepted, server is catch-all.
code_random, _ = smtp.rcpt(f'zz-no-such-user-9281@{domain}')
catch_all = code_random in (250, 251)
return {'tested': True, 'catch_all': catch_all,
'rcpt_code': code_random,
'note': 'Advisory only; greylisting/anti-harvesting can skew results'}
except (socket.timeout, ConnectionRefusedError, OSError) as e:
return {'tested': False, 'reason': f'port 25 blocked/unreachable ({type(e).__name__})'}
except Exception as e:
return {'tested': False, 'reason': str(e)}
# -- grading -----------------------------------------------------------
def _grade(self, r: Dict) -> Dict:
"""0-100 email-security score (higher = better) + letter grade + summary."""
score = 0
if r['spf'].get('present'):
score += 20
if r['spf'].get('policy', '').startswith('fail'):
score += 10
if r['dmarc'].get('present'):
score += 20
if r['dmarc'].get('policy') in ('quarantine', 'reject'):
score += 15
if r['dkim'].get('present'):
score += 15
if r['mta_sts'].get('present'):
score += 10
if r['mta_sts'].get('mode') == 'enforce':
score += 5
if r['tls_rpt'].get('present'):
score += 5
score = min(score, 100)
grade = ('A' if score >= 85 else 'B' if score >= 70 else
'C' if score >= 50 else 'D' if score >= 30 else 'F')
return {'mail_security_score': score, 'mail_security_grade': grade}

265
ai_platform/modules/domain_check/api/app/services/port_scan_service.py Normal file
View file

@ -0,0 +1,265 @@
"""
Port Scanning Service - Detect open ports and services
"""
import logging
import socket
from typing import Dict, List, Optional
from concurrent.futures import ThreadPoolExecutor, as_completed
logger = logging.getLogger(__name__)
class PortScanService:
"""Service for port scanning and service detection"""
def __init__(self):
self.timeout = 3
# Common ports to scan with service info
self.common_ports = {
21: {'service': 'FTP', 'category': 'file_transfer', 'risk': 'medium'},
22: {'service': 'SSH', 'category': 'remote_access', 'risk': 'low'},
23: {'service': 'Telnet', 'category': 'remote_access', 'risk': 'critical'},
25: {'service': 'SMTP', 'category': 'email', 'risk': 'low'},
53: {'service': 'DNS', 'category': 'infrastructure', 'risk': 'low'},
80: {'service': 'HTTP', 'category': 'web', 'risk': 'low'},
110: {'service': 'POP3', 'category': 'email', 'risk': 'medium'},
111: {'service': 'RPCBind', 'category': 'infrastructure', 'risk': 'high'},
135: {'service': 'MS-RPC', 'category': 'windows', 'risk': 'high'},
139: {'service': 'NetBIOS', 'category': 'windows', 'risk': 'high'},
143: {'service': 'IMAP', 'category': 'email', 'risk': 'low'},
443: {'service': 'HTTPS', 'category': 'web', 'risk': 'low'},
445: {'service': 'SMB', 'category': 'windows', 'risk': 'critical'},
465: {'service': 'SMTPS', 'category': 'email', 'risk': 'low'},
587: {'service': 'SMTP Submission', 'category': 'email', 'risk': 'low'},
993: {'service': 'IMAPS', 'category': 'email', 'risk': 'low'},
995: {'service': 'POP3S', 'category': 'email', 'risk': 'low'},
1433: {'service': 'MS-SQL', 'category': 'database', 'risk': 'critical'},
1521: {'service': 'Oracle DB', 'category': 'database', 'risk': 'critical'},
1723: {'service': 'PPTP VPN', 'category': 'vpn', 'risk': 'medium'},
3306: {'service': 'MySQL', 'category': 'database', 'risk': 'critical'},
3389: {'service': 'RDP', 'category': 'remote_access', 'risk': 'high'},
5432: {'service': 'PostgreSQL', 'category': 'database', 'risk': 'critical'},
5900: {'service': 'VNC', 'category': 'remote_access', 'risk': 'high'},
6379: {'service': 'Redis', 'category': 'database', 'risk': 'critical'},
8080: {'service': 'HTTP Proxy', 'category': 'web', 'risk': 'medium'},
8443: {'service': 'HTTPS Alt', 'category': 'web', 'risk': 'low'},
27017: {'service': 'MongoDB', 'category': 'database', 'risk': 'critical'},
}
# Dangerous ports that should never be exposed
self.dangerous_ports = [23, 135, 139, 445, 1433, 1521, 3306, 3389, 5432, 5900, 6379, 27017]
def scan(self, ip_address: str, ports: List[int] = None) -> Dict:
"""
Scan ports on IP address
Args:
ip_address: IP address to scan
ports: List of ports to scan (default: common ports)
Returns:
Dictionary with scan results
"""
if ports is None:
ports = list(self.common_ports.keys())
result = {
'ip': ip_address,
'total_scanned': len(ports),
'open_ports': [],
'closed_ports': [],
'filtered_ports': [],
'dangerous_open': [],
'services_detected': [],
'categories': {},
'security_issues': [],
'scan_summary': {}
}
# Scan ports in parallel
with ThreadPoolExecutor(max_workers=30) as executor:
futures = {
executor.submit(self._scan_port, ip_address, port): port
for port in ports
}
for future in as_completed(futures, timeout=60):
port = futures[future]
try:
status, banner = future.result()
port_info = self.common_ports.get(port, {
'service': 'Unknown',
'category': 'other',
'risk': 'unknown'
})
port_data = {
'port': port,
'service': port_info['service'],
'category': port_info['category'],
'risk_level': port_info['risk'],
'banner': banner
}
if status == 'open':
result['open_ports'].append(port_data)
result['services_detected'].append(port_info['service'])
# Track by category
category = port_info['category']
if category not in result['categories']:
result['categories'][category] = []
result['categories'][category].append(port)
# Check if dangerous port
if port in self.dangerous_ports:
result['dangerous_open'].append(port_data)
result['security_issues'].append({
'severity': 'CRITICAL' if port_info['risk'] == 'critical' else 'HIGH',
'port': port,
'service': port_info['service'],
'issue': f"Dangerous service {port_info['service']} exposed on port {port}"
})
elif status == 'closed':
result['closed_ports'].append(port)
else:
result['filtered_ports'].append(port)
except Exception as e:
logger.warning(f"Error scanning port {port}: {e}")
result['filtered_ports'].append(port)
# Generate summary
result['scan_summary'] = {
'open_count': len(result['open_ports']),
'closed_count': len(result['closed_ports']),
'filtered_count': len(result['filtered_ports']),
'dangerous_count': len(result['dangerous_open']),
'has_web': any(p['port'] in [80, 443, 8080, 8443] for p in result['open_ports']),
'has_email': any(p['port'] in [25, 110, 143, 465, 587, 993, 995] for p in result['open_ports']),
'has_database': any(p['port'] in [3306, 5432, 1433, 1521, 27017, 6379] for p in result['open_ports']),
'has_remote_access': any(p['port'] in [22, 23, 3389, 5900] for p in result['open_ports'])
}
return result
def _scan_port(self, ip_address: str, port: int) -> tuple:
"""
Scan a single port
Returns:
Tuple of (status, banner)
status: 'open', 'closed', or 'filtered'
"""
try:
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.settimeout(self.timeout)
result = sock.connect_ex((ip_address, port))
if result == 0:
# Port is open, try to grab banner
banner = self._grab_banner(sock, port)
sock.close()
return 'open', banner
else:
sock.close()
return 'closed', None
except socket.timeout:
return 'filtered', None
except socket.error:
return 'filtered', None
except Exception as e:
return 'filtered', None
def _grab_banner(self, sock: socket.socket, port: int) -> Optional[str]:
"""Try to grab service banner"""
try:
# For HTTP ports, send a HEAD request
if port in [80, 8080]:
sock.send(b'HEAD / HTTP/1.0\r\n\r\n')
elif port in [443, 8443]:
return None # Can't grab banner from SSL without proper handshake
else:
# For other ports, just try to receive
pass
sock.settimeout(2)
banner = sock.recv(1024).decode('utf-8', errors='ignore').strip()
return banner[:200] if banner else None # Limit banner length
except:
return None
def quick_scan(self, ip_address: str) -> Dict:
"""
Quick scan of most common web ports
Args:
ip_address: IP to scan
Returns:
Quick scan results
"""
quick_ports = [21, 22, 23, 25, 80, 443, 3306, 3389, 8080]
return self.scan(ip_address, quick_ports)
def get_port_score(self, scan_data: Dict) -> Dict:
"""
Calculate security score based on port scan results
Returns:
Dict with score (0-100, higher = more risk) and reasons
"""
score = 0
reasons = []
# Dangerous ports are critical
dangerous_count = len(scan_data.get('dangerous_open', []))
if dangerous_count > 0:
score += dangerous_count * 20
reasons.append(f"{dangerous_count} dangerous port(s) exposed")
for port in scan_data.get('dangerous_open', []):
reasons.append(f" - {port['service']} on port {port['port']}")
# Database exposed
summary = scan_data.get('scan_summary', {})
if summary.get('has_database'):
score += 30
reasons.append("Database port(s) publicly accessible")
# Remote access (besides SSH)
if summary.get('has_remote_access'):
open_ports = [p['port'] for p in scan_data.get('open_ports', [])]
if 23 in open_ports: # Telnet
score += 25
reasons.append("Telnet (unencrypted) is open")
if 3389 in open_ports: # RDP
score += 20
reasons.append("RDP is publicly accessible")
if 5900 in open_ports: # VNC
score += 20
reasons.append("VNC is publicly accessible")
# Too many open ports (attack surface)
open_count = summary.get('open_count', 0)
if open_count > 10:
score += 15
reasons.append(f"Large attack surface: {open_count} open ports")
elif open_count > 5:
score += 5
reasons.append(f"{open_count} open ports detected")
if score == 0:
reasons.append("No dangerous services exposed")
return {
'score': min(100, score),
'reasons': reasons,
'is_secure': score <= 10,
'has_critical_issues': dangerous_count > 0 or summary.get('has_database')
}

741
ai_platform/modules/domain_check/api/app/services/risk_scorer.py Normal file
View file

@ -0,0 +1,741 @@
"""
Risk Scoring Engine - Comprehensive Domain Risk Assessment
SCORING FORMULA:
================
Total Risk Score = Σ(Category Score × Category Weight)
CATEGORIES & WEIGHTS:
- Domain Age: 20% (age_score × 0.20)
- SSL/TLS: 15% (ssl_score × 0.15)
- DNS Config: 10% (dns_score × 0.10)
- Email Security: 10% (email_score × 0.10)
- WHOIS Privacy: 5% (whois_score × 0.05)
- IP Reputation: 10% (ip_score × 0.10)
- HTTP Security: 10% (http_score × 0.10)
- Blacklists: 15% (blacklist_score × 0.15)
- Port Security: 5% (port_score × 0.05)
----
100%
INDIVIDUAL SCORE CALCULATIONS (0-100 scale, 0=safe, 100=dangerous):
1. DOMAIN AGE SCORE:
- < 30 days: 100 (CRITICAL)
- < 90 days: 80 (HIGH)
- < 180 days: 60 (MEDIUM-HIGH)
- < 365 days: 40 (MEDIUM)
- < 730 days: 20 (LOW)
- >= 730 days: 0 (TRUSTED)
2. SSL/TLS SCORE:
- No SSL: 100
- Expired: 100
- Self-signed: 80
- Expires < 7 days: 60
- Expires < 30 days: 30
- Weak cipher: 40
- Valid SSL: 0
3. DNS SCORE:
- No A records: 50
- No MX records: 20
- No NS at parent: 30
- Mismatched NS: 20
- No DNSSEC: 10
- Complete config: 0
4. EMAIL SECURITY SCORE:
- No SPF: 40
- SPF ~all (soft): 20
- SPF ?all (neutral): 30
- No DKIM: 30
- No DMARC: 20
- DMARC p=none: 15
- Full protection: 0
5. WHOIS SCORE:
- Privacy protected: 20
- No registrant info: 15
- Suspicious registrar: 30
- Free/disposable reg: 40
- Transparent: 0
6. IP REPUTATION SCORE:
- High-risk country: 40
- Known bad ASN: 50
- No reverse DNS: 20
- Datacenter IP: 10
- Residential IP: 0
7. HTTP SECURITY SCORE:
- No HTTPS: 40
- No HSTS: 20
- No CSP: 15
- No X-Frame-Options: 10
- Missing headers: 5 each
- Secure config: 0
8. BLACKLIST SCORE:
- On spam blacklist: 25 each
- On malware list: 50 each
- On phishing list: 60 each
- Not listed: 0
9. PORT SECURITY SCORE:
- Database exposed: 30
- RDP/VNC exposed: 25
- Telnet open: 40
- Too many ports (>10): 15
- Secure config: 0
RISK LEVELS:
- LOW: 0-25
- MEDIUM: 26-50
- HIGH: 51-75
- CRITICAL: 76-100
"""
import logging
from datetime import datetime
from typing import Dict, List, Optional, Tuple
from flask import current_app
logger = logging.getLogger(__name__)
def _coerce_datetime(value) -> Optional[datetime]:
"""Return a naive datetime for any date-like value, else None.
Defense-in-depth: even though WhoisService now normalizes dates, the risk
scorer can be fed cached/serialized data where dates are ISO strings. Never
let ``datetime - <non-datetime>`` raise from here.
"""
if value is None:
return None
if isinstance(value, datetime):
return value.replace(tzinfo=None) if value.tzinfo else value
if isinstance(value, str):
text = value.strip()
if not text:
return None
try:
from dateutil import parser as _p
dt = _p.parse(text, fuzzy=True)
return dt.replace(tzinfo=None) if dt.tzinfo else dt
except Exception:
return None
return None
class RiskScorer:
"""Calculate comprehensive risk scores for domains"""
def __init__(self):
# Category weights (must sum to 1.0)
self.weights = {
'domain_age': 0.20,
'ssl': 0.15,
'dns': 0.10,
'email_security': 0.10,
'whois': 0.05,
'ip_reputation': 0.10,
'http_security': 0.10,
'blacklist': 0.15,
'port_security': 0.05
}
# Risk level thresholds
self.thresholds = {
'low': 25,
'medium': 50,
'high': 75
}
def calculate_risk(self, domain_data: Dict) -> Dict:
"""
Calculate comprehensive risk score
Args:
domain_data: Dictionary containing all domain information
- whois: WHOIS data
- dns: DNS records
- ssl: SSL certificate data
- ip_intelligence: IP information
- http_analysis: HTTP analysis data
- blacklist: Blacklist check results
- port_scan: Port scan results
Returns:
Dictionary with complete risk assessment
"""
factors = []
scores = {}
formula_breakdown = []
# 1. Domain Age Score (20%)
if domain_data.get('whois'):
score, factor = self._score_domain_age(domain_data['whois'])
scores['domain_age'] = score
factors.append(factor)
formula_breakdown.append({
'category': 'Domain Age',
'raw_score': score,
'weight': self.weights['domain_age'],
'weighted_score': score * self.weights['domain_age'],
'formula': f"{score} × {self.weights['domain_age']} = {score * self.weights['domain_age']:.1f}"
})
# 2. SSL Score (15%)
if domain_data.get('ssl'):
score, factor = self._score_ssl(domain_data['ssl'])
scores['ssl'] = score
factors.append(factor)
formula_breakdown.append({
'category': 'SSL/TLS',
'raw_score': score,
'weight': self.weights['ssl'],
'weighted_score': score * self.weights['ssl'],
'formula': f"{score} × {self.weights['ssl']} = {score * self.weights['ssl']:.1f}"
})
# 3. DNS Score (10%)
if domain_data.get('dns'):
score, factor = self._score_dns(domain_data['dns'])
scores['dns'] = score
factors.append(factor)
formula_breakdown.append({
'category': 'DNS Configuration',
'raw_score': score,
'weight': self.weights['dns'],
'weighted_score': score * self.weights['dns'],
'formula': f"{score} × {self.weights['dns']} = {score * self.weights['dns']:.1f}"
})
# 4. Email Security Score (10%)
if domain_data.get('dns'):
score, factor = self._score_email_security(domain_data['dns'])
scores['email_security'] = score
factors.append(factor)
formula_breakdown.append({
'category': 'Email Security',
'raw_score': score,
'weight': self.weights['email_security'],
'weighted_score': score * self.weights['email_security'],
'formula': f"{score} × {self.weights['email_security']} = {score * self.weights['email_security']:.1f}"
})
# 5. WHOIS Score (5%)
if domain_data.get('whois'):
score, factor = self._score_whois(domain_data['whois'])
scores['whois'] = score
factors.append(factor)
formula_breakdown.append({
'category': 'WHOIS Privacy',
'raw_score': score,
'weight': self.weights['whois'],
'weighted_score': score * self.weights['whois'],
'formula': f"{score} × {self.weights['whois']} = {score * self.weights['whois']:.1f}"
})
# 6. IP Reputation Score (10%)
if domain_data.get('ip_intelligence'):
score, factor = self._score_ip_reputation(domain_data['ip_intelligence'])
scores['ip_reputation'] = score
factors.append(factor)
formula_breakdown.append({
'category': 'IP Reputation',
'raw_score': score,
'weight': self.weights['ip_reputation'],
'weighted_score': score * self.weights['ip_reputation'],
'formula': f"{score} × {self.weights['ip_reputation']} = {score * self.weights['ip_reputation']:.1f}"
})
# 7. HTTP Security Score (10%)
if domain_data.get('http_analysis'):
score, factor = self._score_http_security(domain_data['http_analysis'])
scores['http_security'] = score
factors.append(factor)
formula_breakdown.append({
'category': 'HTTP Security',
'raw_score': score,
'weight': self.weights['http_security'],
'weighted_score': score * self.weights['http_security'],
'formula': f"{score} × {self.weights['http_security']} = {score * self.weights['http_security']:.1f}"
})
# 8. Blacklist Score (15%)
if domain_data.get('blacklist'):
score, factor = self._score_blacklist(domain_data['blacklist'])
scores['blacklist'] = score
factors.append(factor)
formula_breakdown.append({
'category': 'Blacklist Status',
'raw_score': score,
'weight': self.weights['blacklist'],
'weighted_score': score * self.weights['blacklist'],
'formula': f"{score} × {self.weights['blacklist']} = {score * self.weights['blacklist']:.1f}"
})
# 9. Port Security Score (5%)
if domain_data.get('port_scan'):
score, factor = self._score_port_security(domain_data['port_scan'])
scores['port_security'] = score
factors.append(factor)
formula_breakdown.append({
'category': 'Port Security',
'raw_score': score,
'weight': self.weights['port_security'],
'weighted_score': score * self.weights['port_security'],
'formula': f"{score} × {self.weights['port_security']} = {score * self.weights['port_security']:.1f}"
})
# Calculate weighted total
total_weight_used = sum(self.weights[k] for k in scores.keys())
if total_weight_used > 0:
# Normalize to account for missing checks
raw_total = sum(scores[k] * self.weights[k] for k in scores.keys())
total_score = (raw_total / total_weight_used) if total_weight_used < 1.0 else raw_total
else:
total_score = 50 # Default medium risk if no data
total_score = min(100, max(0, int(total_score)))
# Determine risk level
risk_level = self._get_risk_level(total_score)
# Generate final formula string
formula_string = self._generate_formula_string(scores, total_score)
return {
'total_score': total_score,
'risk_level': risk_level,
'individual_scores': scores,
'factors': factors,
'formula_breakdown': formula_breakdown,
'formula_string': formula_string,
'weights_used': {k: self.weights[k] for k in scores.keys()},
'total_weight_used': total_weight_used,
'is_new_domain': scores.get('domain_age', 0) >= 80,
'is_suspicious': total_score >= 50,
'is_blacklisted': scores.get('blacklist', 0) > 0,
'requires_manual_review': total_score >= 60,
'risk_thresholds': {
'low': f"0-{self.thresholds['low']}",
'medium': f"{self.thresholds['low']+1}-{self.thresholds['medium']}",
'high': f"{self.thresholds['medium']+1}-{self.thresholds['high']}",
'critical': f"{self.thresholds['high']+1}-100"
}
}
def _score_domain_age(self, whois_data: Dict) -> Tuple[int, Dict]:
"""Score based on domain age"""
creation_date = _coerce_datetime(whois_data.get('creation_date'))
if not creation_date:
return 50, {
'factor': 'domain_age',
'score': 50,
'weight': self.weights['domain_age'],
'weighted_score': 50 * self.weights['domain_age'],
'reason': 'Unable to determine domain age',
'details': {'creation_date': None, 'age_days': None}
}
age_days = (datetime.utcnow() - creation_date).days
# Scoring thresholds
if age_days < 30:
score = 100
reason = f'CRITICAL: Very new domain ({age_days} days, <1 month)'
elif age_days < 90:
score = 80
reason = f'HIGH: New domain ({age_days} days, <3 months)'
elif age_days < 180:
score = 60
reason = f'MEDIUM-HIGH: Recently created ({age_days} days, <6 months)'
elif age_days < 365:
score = 40
reason = f'MEDIUM: Less than 1 year old ({age_days} days)'
elif age_days < 730:
score = 20
reason = f'LOW: Established domain ({age_days} days, 1-2 years)'
else:
score = 0
years = age_days // 365
reason = f'TRUSTED: Mature domain ({age_days} days, {years}+ years)'
return score, {
'factor': 'domain_age',
'score': score,
'weight': self.weights['domain_age'],
'weighted_score': score * self.weights['domain_age'],
'reason': reason,
'details': {
'creation_date': str(creation_date) if creation_date else None,
'age_days': age_days
}
}
def _score_ssl(self, ssl_data: Dict) -> Tuple[int, Dict]:
"""Score based on SSL/TLS certificate"""
if not ssl_data.get('has_ssl'):
return 100, {
'factor': 'ssl',
'score': 100,
'weight': self.weights['ssl'],
'weighted_score': 100 * self.weights['ssl'],
'reason': 'CRITICAL: No SSL/TLS certificate',
'details': {'has_ssl': False}
}
score = 0
reasons = []
if ssl_data.get('is_expired'):
score = 100
reasons.append('Certificate expired')
elif ssl_data.get('is_self_signed'):
score = 80
reasons.append('Self-signed certificate')
else:
days_until_expiry = ssl_data.get('days_until_expiry', 365)
if days_until_expiry < 7:
score = 60
reasons.append(f'Expires in {days_until_expiry} days')
elif days_until_expiry < 30:
score = 30
reasons.append(f'Expires in {days_until_expiry} days')
# Check for weak algorithms
sig_algo = (ssl_data.get('signature_algorithm') or '').lower()
if 'sha1' in sig_algo or 'md5' in sig_algo:
score = max(score, 40)
reasons.append('Weak signature algorithm')
if score == 0:
reasons.append('Valid SSL certificate')
return score, {
'factor': 'ssl',
'score': score,
'weight': self.weights['ssl'],
'weighted_score': score * self.weights['ssl'],
'reason': '; '.join(reasons) if reasons else 'Valid SSL',
'details': {
'has_ssl': True,
'is_valid': ssl_data.get('is_valid'),
'days_until_expiry': ssl_data.get('days_until_expiry'),
'issuer': ssl_data.get('issuer')
}
}
def _score_dns(self, dns_data: Dict) -> Tuple[int, Dict]:
"""Score based on DNS configuration"""
score = 0
reasons = []
a_records = dns_data.get('a_records', [])
if not a_records:
score += 50
reasons.append('No A records')
ns_records = dns_data.get('ns_records', [])
if not ns_records:
score += 30
reasons.append('No NS records')
mx_records = dns_data.get('mx_records', [])
if not mx_records:
score += 15
reasons.append('No MX records')
# Check for DNSSEC (if available)
if dns_data.get('dnssec') == 'inactive' or not dns_data.get('has_dnssec', True):
score += 5
reasons.append('DNSSEC not enabled')
if score == 0:
reasons.append('Complete DNS configuration')
return min(100, score), {
'factor': 'dns',
'score': min(100, score),
'weight': self.weights['dns'],
'weighted_score': min(100, score) * self.weights['dns'],
'reason': '; '.join(reasons),
'details': {
'a_record_count': len(a_records),
'ns_record_count': len(ns_records),
'mx_record_count': len(mx_records)
}
}
def _score_email_security(self, dns_data: Dict) -> Tuple[int, Dict]:
"""Score based on email security (SPF, DKIM, DMARC)"""
score = 0
reasons = []
has_spf = dns_data.get('has_spf', False)
has_dkim = dns_data.get('has_dkim', False)
has_dmarc = dns_data.get('has_dmarc', False)
if not has_spf:
score += 40
reasons.append('No SPF record')
else:
# Check SPF policy strength
spf_record = dns_data.get('spf_record', '')
if '~all' in spf_record:
score += 15
reasons.append('SPF uses soft fail (~all)')
elif '?all' in spf_record:
score += 25
reasons.append('SPF uses neutral (?all)')
if not has_dkim:
score += 30
reasons.append('No DKIM record detected')
if not has_dmarc:
score += 20
reasons.append('No DMARC record')
else:
dmarc_record = dns_data.get('dmarc_record', '')
if 'p=none' in dmarc_record:
score += 15
reasons.append('DMARC policy is none (monitoring only)')
if score == 0:
reasons.append('Full email security (SPF+DKIM+DMARC)')
return min(100, score), {
'factor': 'email_security',
'score': min(100, score),
'weight': self.weights['email_security'],
'weighted_score': min(100, score) * self.weights['email_security'],
'reason': '; '.join(reasons),
'details': {
'has_spf': has_spf,
'has_dkim': has_dkim,
'has_dmarc': has_dmarc
}
}
def _score_whois(self, whois_data: Dict) -> Tuple[int, Dict]:
"""Score based on WHOIS transparency"""
score = 0
reasons = []
if whois_data.get('privacy_protected'):
score += 20
reasons.append('WHOIS privacy protection enabled')
if not whois_data.get('registrant_org') and not whois_data.get('registrant'):
score += 15
reasons.append('No registrant information')
# Check for suspicious registrars
registrar = (whois_data.get('registrar') or '').lower()
suspicious_registrars = ['namecheap', 'namesilo', 'dynadot', 'freenom']
for sus in suspicious_registrars:
if sus in registrar:
score += 20
reasons.append(f'High-risk registrar pattern')
break
if score == 0:
reasons.append('Transparent WHOIS information')
return min(100, score), {
'factor': 'whois',
'score': min(100, score),
'weight': self.weights['whois'],
'weighted_score': min(100, score) * self.weights['whois'],
'reason': '; '.join(reasons),
'details': {
'registrar': whois_data.get('registrar'),
'privacy_protected': whois_data.get('privacy_protected', False)
}
}
def _score_ip_reputation(self, ip_data: Dict) -> Tuple[int, Dict]:
"""Score based on IP intelligence"""
score = 0
reasons = []
# High-risk countries
country_code = ip_data.get('country_code', '')
high_risk_countries = ['RU', 'CN', 'KP', 'IR', 'NG', 'VN', 'PK']
if country_code in high_risk_countries:
score += 40
reasons.append(f'High-risk country: {country_code}')
# No reverse DNS
if not ip_data.get('reverse_dns'):
score += 20
reasons.append('No reverse DNS (PTR)')
# Datacenter IP (slightly suspicious for some use cases)
if ip_data.get('is_datacenter'):
score += 10
reasons.append('Hosted on datacenter/cloud')
# Hosting provider scoring
hosting_score = ip_data.get('hosting_score', {})
if hosting_score.get('is_suspicious'):
score += 20
reasons.extend(hosting_score.get('reasons', []))
if score == 0:
reasons.append('Clean IP reputation')
return min(100, score), {
'factor': 'ip_reputation',
'score': min(100, score),
'weight': self.weights['ip_reputation'],
'weighted_score': min(100, score) * self.weights['ip_reputation'],
'reason': '; '.join(reasons),
'details': {
'ip': ip_data.get('ip'),
'country': ip_data.get('country'),
'asn': ip_data.get('asn'),
'isp': ip_data.get('isp')
}
}
def _score_http_security(self, http_data: Dict) -> Tuple[int, Dict]:
"""Score based on HTTP security headers"""
score = 0
reasons = []
if not http_data.get('has_https'):
score += 40
reasons.append('No HTTPS support')
if http_data.get('has_https') and not http_data.get('http_to_https_redirect'):
score += 15
reasons.append('No HTTP to HTTPS redirect')
# Check security headers
missing_headers = http_data.get('missing_security_headers', [])
for header in missing_headers:
severity = header.get('severity', 'LOW')
if severity == 'CRITICAL':
score += 15
elif severity == 'HIGH':
score += 10
else:
score += 5
if len(reasons) < 5: # Limit reasons
reasons.append(f"Missing: {header.get('header', 'Unknown')}")
if score == 0:
reasons.append('Good HTTP security configuration')
return min(100, score), {
'factor': 'http_security',
'score': min(100, score),
'weight': self.weights['http_security'],
'weighted_score': min(100, score) * self.weights['http_security'],
'reason': '; '.join(reasons[:5]),
'details': {
'has_https': http_data.get('has_https'),
'has_hsts': 'Strict-Transport-Security' in http_data.get('security_headers', {}),
'missing_headers_count': len(missing_headers)
}
}
def _score_blacklist(self, blacklist_data: Dict) -> Tuple[int, Dict]:
"""Score based on blacklist checks"""
score = 0
reasons = []
if blacklist_data.get('is_blacklisted'):
ip_listings = 0
domain_listings = 0
if blacklist_data.get('ip_check'):
ip_listings = blacklist_data['ip_check'].get('blacklist_count', 0)
score += ip_listings * 25
if blacklist_data.get('domain_check'):
domain_listings = blacklist_data['domain_check'].get('blacklist_count', 0)
score += domain_listings * 25
if ip_listings > 0:
reasons.append(f'IP on {ip_listings} blacklist(s)')
if domain_listings > 0:
reasons.append(f'Domain on {domain_listings} blacklist(s)')
else:
reasons.append('Not on any blacklists')
return min(100, score), {
'factor': 'blacklist',
'score': min(100, score),
'weight': self.weights['blacklist'],
'weighted_score': min(100, score) * self.weights['blacklist'],
'reason': '; '.join(reasons),
'details': {
'is_blacklisted': blacklist_data.get('is_blacklisted', False),
'total_listings': blacklist_data.get('total_listings', 0)
}
}
def _score_port_security(self, port_data: Dict) -> Tuple[int, Dict]:
"""Score based on port scan results"""
score = 0
reasons = []
dangerous_ports = port_data.get('dangerous_open', [])
if dangerous_ports:
score += len(dangerous_ports) * 20
for port in dangerous_ports[:3]: # Limit to 3
reasons.append(f"{port['service']} exposed (port {port['port']})")
summary = port_data.get('scan_summary', {})
if summary.get('has_database'):
score += 30
if 'Database' not in str(reasons):
reasons.append('Database port(s) publicly accessible')
open_count = summary.get('open_count', 0)
if open_count > 10:
score += 15
reasons.append(f'Large attack surface ({open_count} open ports)')
if score == 0:
reasons.append('No dangerous services exposed')
return min(100, score), {
'factor': 'port_security',
'score': min(100, score),
'weight': self.weights['port_security'],
'weighted_score': min(100, score) * self.weights['port_security'],
'reason': '; '.join(reasons[:5]),
'details': {
'open_port_count': summary.get('open_count', 0),
'dangerous_port_count': len(dangerous_ports)
}
}
def _get_risk_level(self, total_score: int) -> str:
"""Determine risk level from total score"""
if total_score <= self.thresholds['low']:
return 'LOW'
elif total_score <= self.thresholds['medium']:
return 'MEDIUM'
elif total_score <= self.thresholds['high']:
return 'HIGH'
else:
return 'CRITICAL'
def _generate_formula_string(self, scores: Dict, total: int) -> str:
"""Generate human-readable formula string"""
parts = []
for category, score in scores.items():
weight = self.weights.get(category, 0)
weighted = score * weight
parts.append(f"({score} × {weight})")
formula = " + ".join(parts)
return f"Total = {formula} = {total}"

185
ai_platform/modules/domain_check/api/app/services/ssl_service.py Normal file
View file

@ -0,0 +1,185 @@
"""
SSL Service - handles SSL certificate validation
"""
import logging
import socket
import ssl
from datetime import datetime
from typing import Dict, Optional
from OpenSSL import SSL, crypto
logger = logging.getLogger(__name__)
class SSLService:
"""SSL certificate checking service"""
def __init__(self, timeout: int = 10):
self.timeout = timeout
def check(self, domain: str, port: int = 443) -> Optional[Dict]:
"""
Check SSL certificate for a domain
Args:
domain: Domain name to check
port: Port number (default: 443)
Returns:
Dictionary with SSL certificate data or None
"""
try:
logger.info(f'SSL check for: {domain}:{port}')
# Create SSL context
context = ssl.create_default_context()
# Connect and get certificate
with socket.create_connection((domain, port), timeout=self.timeout) as sock:
with context.wrap_socket(sock, server_hostname=domain) as ssock:
cert_bin = ssock.getpeercert(binary_form=True)
cert_dict = ssock.getpeercert()
# Parse certificate with pyOpenSSL for more details
x509 = crypto.load_certificate(crypto.FILETYPE_ASN1, cert_bin)
ssl_data = {
'domain': domain,
'has_ssl': True,
'is_valid': True,
'is_self_signed': self._is_self_signed(x509),
'is_expired': False,
'is_wildcard': False,
'issuer': self._parse_issuer(x509),
'subject': self._parse_subject(x509),
'serial_number': str(x509.get_serial_number()),
'signature_algorithm': x509.get_signature_algorithm().decode('utf-8'),
'version': x509.get_version(),
'valid_from': None,
'valid_until': None,
'days_until_expiry': None,
'san': self._get_san(x509),
'key_size': x509.get_pubkey().bits()
}
# Parse dates
not_before = datetime.strptime(
x509.get_notBefore().decode('utf-8'),
'%Y%m%d%H%M%SZ'
)
not_after = datetime.strptime(
x509.get_notAfter().decode('utf-8'),
'%Y%m%d%H%M%SZ'
)
ssl_data['valid_from'] = not_before
ssl_data['valid_until'] = not_after
# Calculate days until expiry
days_until = (not_after - datetime.utcnow()).days
ssl_data['days_until_expiry'] = days_until
# Check if expired
if days_until < 0:
ssl_data['is_expired'] = True
ssl_data['is_valid'] = False
# Check if wildcard
subject_cn = self._get_common_name(x509)
if subject_cn and subject_cn.startswith('*.'):
ssl_data['is_wildcard'] = True
return ssl_data
except ssl.SSLError as e:
logger.warning(f'SSL error for {domain}: {str(e)}')
return {
'domain': domain,
'has_ssl': True,
'is_valid': False,
'error': str(e)
}
except socket.timeout:
logger.warning(f'SSL check timeout for {domain}')
return {
'domain': domain,
'has_ssl': False,
'error': 'Connection timeout'
}
except Exception as e:
logger.error(f'SSL check failed for {domain}: {str(e)}')
return {
'domain': domain,
'has_ssl': False,
'error': str(e)
}
def _is_self_signed(self, cert: crypto.X509) -> bool:
"""Check if certificate is self-signed"""
try:
issuer = cert.get_issuer()
subject = cert.get_subject()
return issuer.CN == subject.CN
except Exception:
return False
def _parse_issuer(self, cert: crypto.X509) -> str:
"""Parse certificate issuer"""
try:
issuer = cert.get_issuer()
parts = []
if hasattr(issuer, 'CN') and issuer.CN:
parts.append(f"CN={issuer.CN}")
if hasattr(issuer, 'O') and issuer.O:
parts.append(f"O={issuer.O}")
if hasattr(issuer, 'C') and issuer.C:
parts.append(f"C={issuer.C}")
return ', '.join(parts) if parts else 'Unknown'
except Exception:
return 'Unknown'
def _parse_subject(self, cert: crypto.X509) -> str:
"""Parse certificate subject"""
try:
subject = cert.get_subject()
parts = []
if hasattr(subject, 'CN') and subject.CN:
parts.append(f"CN={subject.CN}")
if hasattr(subject, 'O') and subject.O:
parts.append(f"O={subject.O}")
if hasattr(subject, 'C') and subject.C:
parts.append(f"C={subject.C}")
return ', '.join(parts) if parts else 'Unknown'
except Exception:
return 'Unknown'
def _get_common_name(self, cert: crypto.X509) -> Optional[str]:
"""Get Common Name from certificate"""
try:
subject = cert.get_subject()
return subject.CN if hasattr(subject, 'CN') else None
except Exception:
return None
def _get_san(self, cert: crypto.X509) -> list:
"""Get Subject Alternative Names"""
try:
san_ext = None
for i in range(cert.get_extension_count()):
ext = cert.get_extension(i)
if ext.get_short_name() == b'subjectAltName':
san_ext = ext
break
if san_ext:
san_str = str(san_ext)
# Parse SAN string (format: "DNS:example.com, DNS:www.example.com")
sans = []
for part in san_str.split(','):
part = part.strip()
if part.startswith('DNS:'):
sans.append(part[4:])
return sans
except Exception as e:
logger.debug(f'Could not parse SAN: {e}')
return []

442
ai_platform/modules/domain_check/api/app/services/subdomain_service.py Normal file
View file

@ -0,0 +1,442 @@
"""
Subdomain Enumeration Service - Certificate Transparency, DNS enumeration
"""
import logging
import requests
import dns.resolver
from typing import Dict, List, Set
from concurrent.futures import ThreadPoolExecutor, as_completed
logger = logging.getLogger(__name__)
class SubdomainService:
"""Service for discovering subdomains"""
def __init__(self):
self.timeout = 15
self.resolver = dns.resolver.Resolver()
self.resolver.nameservers = ['8.8.8.8', '8.8.4.4']
self.resolver.timeout = 3
self.resolver.lifetime = 5
# Common subdomain prefixes to check (expanded: infra, SaaS, business apps)
self.common_subdomains = [
'www', 'www2', 'mail', 'webmail', 'smtp', 'pop', 'pop3', 'imap', 'mx', 'mx1', 'mx2',
'email', 'mailserver', 'relay', 'newsletter', 'mailgun', 'mailchimp', 'lists', 'list',
'ftp', 'sftp', 'ssh', 'admin', 'administrator', 'panel', 'cpanel', 'whm', 'plesk', 'directadmin',
'api', 'api1', 'api2', 'apis', 'rest', 'graphql', 'ws', 'sockets', 'dev', 'develop', 'development',
'staging', 'stage', 'stg', 'test', 'testing', 'qa', 'uat', 'preprod', 'beta', 'alpha', 'demo', 'sandbox',
'blog', 'shop', 'store', 'magento', 'woocommerce', 'checkout', 'pay', 'payment', 'payments', 'billing', 'invoice', 'invoices',
'app', 'apps', 'application', 'mobile', 'm', 'web', 'portal', 'my', 'account', 'accounts', 'auth', 'sso', 'login', 'oauth', 'id', 'idp',
'cdn', 'static', 'assets', 'img', 'images', 'media', 'video', 'videos', 'stream', 'live', 'download', 'downloads', 'files', 'file', 'share', 'sharepoint',
'ns1', 'ns2', 'ns3', 'ns4', 'dns', 'dns1', 'dns2', 'resolver',
'vpn', 'remote', 'gateway', 'gw', 'proxy', 'firewall', 'fw', 'router', 'access',
'git', 'gitlab', 'github', 'gitea', 'bitbucket', 'svn', 'jenkins', 'ci', 'cicd', 'build', 'registry', 'docker', 'nexus', 'artifactory',
'db', 'database', 'mysql', 'postgres', 'pg', 'mongo', 'redis', 'elastic', 'elasticsearch', 'kibana', 'grafana', 'prometheus', 'metrics', 'monitor', 'monitoring', 'status', 'health', 'uptime',
'backup', 'backups', 'bk', 'old', 'new', 'v2', 'v3', 'legacy', 'archive',
'intranet', 'extranet', 'internal', 'private', 'corp', 'office', 'work',
'cloud', 'nextcloud', 'owncloud', 'drive', 'docs', 'doc', 'documents', 'wiki', 'confluence', 'jira', 'kb', 'knowledgebase',
'crm', 'erp', 'hr', 'hrm', 'support', 'help', 'helpdesk', 'ticket', 'tickets', 'desk', 'service', 'services', 'client', 'clients', 'customer', 'customers', 'partner', 'partners', 'projects', 'project', 'pm', 'tasks', 'rpa', 'automation', 'tooling', 'tools', 'tool',
'chat', 'talk', 'meet', 'conf', 'conference', 'videoconferinta', 'voip', 'pbx', 'sip', 'call', 'calls',
'dashboard', 'analytics', 'stats', 'report', 'reports', 'data', 'bi', 'reporting',
'autodiscover', 'autoconfig', 'exchange', 'owa', 'mail2', 'webdisk', 'cpcalendars', 'cpcontacts',
'vps', 'server', 'server1', 'server2', 'host', 'node', 'node1', 'cluster', 'k8s', 'kubernetes', 'srv',
'secure', 'ssl', 'vault', 'secret', 'kms', 'ldap', 'ad', 'radius'
]
def enumerate(self, domain: str, extra_subdomains: List[str] = None) -> Dict:
"""
Enumerate subdomains using multiple methods.
Coverage note: CT logs only reveal names that had their own certificate;
a wildcard cert (*.domain) or a custom-named record with no cert is
invisible to CT. Brute-force only finds names in the wordlist. For a
domain you own, the authoritative way to get 100% is the DNS provider's
API (e.g. GoDaddy) or AXFR (usually disabled). Pass `extra_subdomains`
to verify your own known names directly.
Args:
domain: Root domain to enumerate
extra_subdomains: owner-supplied candidate names (bare label or FQDN)
Returns:
Dictionary with discovered subdomains
"""
result = {
'domain': domain,
'subdomains': [],
'total_found': 0,
'sources': {
'certificate_transparency': [],
'dns_bruteforce': [],
'dns_records': [],
'zone_transfer': [],
'user_provided': []
},
'has_wildcard_cert': False,
'coverage_note': None,
'live_subdomains': [],
'error': None
}
discovered: Set[str] = set()
try:
# 1. Certificate Transparency logs (crt.sh)
ct_raw = self._get_ct_subdomains(domain)
result['has_wildcard_cert'] = any(s.startswith('*.') for s in ct_raw)
ct_subdomains = [s for s in ct_raw if not s.startswith('*')]
result['sources']['certificate_transparency'] = ct_subdomains
discovered.update(ct_subdomains)
# 2. DNS records enumeration (MX/NS/SOA)
dns_subdomains = self._get_dns_subdomains(domain)
result['sources']['dns_records'] = dns_subdomains
discovered.update(dns_subdomains)
# 3. AXFR zone transfer (jackpot if the NS allows it — usually not)
axfr_subdomains = self._try_axfr(domain)
result['sources']['zone_transfer'] = axfr_subdomains
discovered.update(axfr_subdomains)
# 4. Bruteforce common subdomains
brute_subdomains = self._bruteforce_subdomains(domain)
result['sources']['dns_bruteforce'] = brute_subdomains
discovered.update(brute_subdomains)
# 5. Owner-supplied candidate names — verify which actually resolve
if extra_subdomains:
user_found = self._check_user_subdomains(domain, extra_subdomains)
result['sources']['user_provided'] = user_found
discovered.update(user_found)
# Remove wildcards and invalid entries
discovered = {
s for s in discovered
if s and not s.startswith('*') and domain in s
}
result['subdomains'] = sorted(list(discovered))
result['total_found'] = len(discovered)
if result['has_wildcard_cert']:
result['coverage_note'] = (
'Domeniul are certificat wildcard (*.{0}), deci subdomeniile '
'fara certificat propriu NU apar in CT logs. Lista poate fi '
'incompleta — foloseste extra_subdomains sau API-ul DNS al '
'registrarului (GoDaddy) pentru acoperire 100%.'.format(domain)
)
# Check which subdomains are live
result['live_subdomains'] = self._check_live_subdomains(list(discovered)[:60])
except Exception as e:
logger.error(f"Subdomain enumeration failed for {domain}: {e}")
result['error'] = str(e)
return result
def _try_axfr(self, domain: str) -> List[str]:
"""Attempt a DNS zone transfer (AXFR) against each authoritative NS.
Almost always refused, but when an NS is misconfigured it dumps the
entire zone — every subdomain at once. Cheap to try, big payoff."""
import dns.query
import dns.zone
found: Set[str] = set()
try:
ns_records = self.resolver.resolve(domain, 'NS')
nameservers = [str(ns.target).rstrip('.') for ns in ns_records]
except Exception:
return []
for ns in nameservers[:4]:
try:
ns_ip = str(self.resolver.resolve(ns, 'A')[0])
zone = dns.zone.from_xfr(dns.query.xfr(ns_ip, domain, timeout=5, lifetime=8))
for name in zone.nodes.keys():
label = str(name)
if label in ('@', ''):
continue
fqdn = f'{label}.{domain}' if not label.endswith(domain) else label
found.add(fqdn.rstrip('.').lower())
logger.info(f'AXFR succeeded against {ns} for {domain} ({len(found)} names)')
break
except Exception:
continue
return list(found)
def _check_user_subdomains(self, domain: str, names: List[str]) -> List[str]:
"""Resolve owner-supplied candidate names and keep the ones that exist."""
candidates = []
for n in names:
n = str(n).strip().lower().rstrip('.')
if not n:
continue
fqdn = n if n.endswith(domain) else f'{n}.{domain}'
candidates.append(fqdn)
found = []
with ThreadPoolExecutor(max_workers=20) as ex:
futures = {ex.submit(self._resolves, c): c for c in candidates}
for fut in as_completed(futures, timeout=30):
try:
if fut.result():
found.append(futures[fut])
except Exception:
pass
return found
def _resolves(self, name: str) -> bool:
for rtype in ('A', 'AAAA', 'CNAME'):
try:
self.resolver.resolve(name, rtype)
return True
except Exception:
continue
return False
def _get_ct_subdomains(self, domain: str) -> List[str]:
"""Get subdomains from Certificate Transparency logs.
crt.sh is the richest source for real (incl. custom-named) subdomains but
is frequently slow or rate-limited, so we retry, and fall back to the
certspotter API when crt.sh keeps failing."""
subdomains: Set[str] = set()
# Primary: crt.sh, with retries (it flakes often).
for attempt in range(3):
try:
response = requests.get(
f"https://crt.sh/?q=%.{domain}&output=json",
timeout=self.timeout,
headers={'User-Agent': 'domain-check/1.0'}
)
if response.status_code == 200 and response.text.strip():
for entry in response.json():
for name in entry.get('name_value', '').split('\n'):
name = name.strip().lower()
if name and domain in name:
subdomains.add(name)
if subdomains:
return list(subdomains)
except Exception as e:
logger.warning(f"crt.sh attempt {attempt + 1} failed: {e}")
# Fallback: Cert Spotter (no key needed for low volume).
try:
resp = requests.get(
f"https://api.certspotter.com/v1/issuances?domain={domain}"
"&include_subdomains=true&expand=dns_names",
timeout=self.timeout, headers={'User-Agent': 'domain-check/1.0'}
)
if resp.status_code == 200:
for cert in resp.json():
for name in cert.get('dns_names', []):
name = str(name).strip().lower()
if name and domain in name:
subdomains.add(name)
except Exception as e:
logger.warning(f"certspotter fallback failed: {e}")
return list(subdomains)
def _get_dns_subdomains(self, domain: str) -> List[str]:
"""Get subdomains from DNS records (NS, MX, etc.)"""
subdomains = []
try:
# Check MX records
try:
mx_records = self.resolver.resolve(domain, 'MX')
for mx in mx_records:
host = str(mx.exchange).rstrip('.')
if domain in host:
subdomains.append(host)
except:
pass
# Check NS records
try:
ns_records = self.resolver.resolve(domain, 'NS')
for ns in ns_records:
host = str(ns.target).rstrip('.')
if domain in host:
subdomains.append(host)
except:
pass
# Check SOA record
try:
soa_records = self.resolver.resolve(domain, 'SOA')
for soa in soa_records:
mname = str(soa.mname).rstrip('.')
if domain in mname:
subdomains.append(mname)
except:
pass
except Exception as e:
logger.warning(f"DNS subdomain lookup failed: {e}")
return list(set(subdomains))
def _bruteforce_subdomains(self, domain: str) -> List[str]:
"""Bruteforce common subdomains"""
found = []
with ThreadPoolExecutor(max_workers=20) as executor:
futures = {
executor.submit(self._check_subdomain, f"{sub}.{domain}"): sub
for sub in self.common_subdomains
}
for future in as_completed(futures, timeout=30):
subdomain = futures[future]
try:
full_subdomain = f"{subdomain}.{domain}"
if future.result():
found.append(full_subdomain)
except:
pass
return found
def _check_subdomain(self, subdomain: str) -> bool:
"""Check if subdomain resolves"""
try:
self.resolver.resolve(subdomain, 'A')
return True
except:
return False
def _check_live_subdomains(self, subdomains: List[str]) -> List[Dict]:
"""Check which subdomains are live (have HTTP response)"""
live = []
with ThreadPoolExecutor(max_workers=10) as executor:
futures = {
executor.submit(self._check_http, sub): sub
for sub in subdomains
}
for future in as_completed(futures, timeout=60):
subdomain = futures[future]
try:
result = future.result()
if result:
live.append(result)
except:
pass
return live
def _check_http(self, subdomain: str) -> Dict:
"""Check HTTP/HTTPS on subdomain"""
result = {
'subdomain': subdomain,
'ip': None,
'http': None,
'https': None
}
# Get IP
try:
answers = self.resolver.resolve(subdomain, 'A')
result['ip'] = str(answers[0])
except:
return None
# Check HTTPS
try:
response = requests.head(
f"https://{subdomain}",
timeout=5,
allow_redirects=True,
verify=False
)
result['https'] = response.status_code
except:
pass
# Check HTTP
try:
response = requests.head(
f"http://{subdomain}",
timeout=5,
allow_redirects=False
)
result['http'] = response.status_code
except:
pass
if result['http'] or result['https']:
return result
return None
def check_subdomain_takeover(self, subdomains: List[str]) -> List[Dict]:
"""
Check for potential subdomain takeover vulnerabilities
Args:
subdomains: List of subdomains to check
Returns:
List of potentially vulnerable subdomains
"""
vulnerable = []
# CNAME fingerprints for takeover
takeover_fingerprints = {
'github.io': 'GitHub Pages',
'herokuapp.com': 'Heroku',
'herokudns.com': 'Heroku',
'wordpress.com': 'WordPress',
'pantheonsite.io': 'Pantheon',
'domains.tumblr.com': 'Tumblr',
'zendesk.com': 'Zendesk',
'shopify.com': 'Shopify',
'myshopify.com': 'Shopify',
's3.amazonaws.com': 'AWS S3',
's3-website': 'AWS S3',
'cloudfront.net': 'AWS CloudFront',
'azurewebsites.net': 'Azure',
'cloudapp.net': 'Azure',
'trafficmanager.net': 'Azure',
'blob.core.windows.net': 'Azure Blob',
'ghost.io': 'Ghost',
'helpjuice.com': 'Helpjuice',
'helpscoutdocs.com': 'HelpScout',
'freshdesk.com': 'Freshdesk',
'surge.sh': 'Surge',
'bitbucket.io': 'Bitbucket',
'uservoice.com': 'UserVoice',
'simplebooklet.com': 'Simplebooklet'
}
for subdomain in subdomains:
try:
# Check for CNAME
cname_records = self.resolver.resolve(subdomain, 'CNAME')
for cname in cname_records:
cname_target = str(cname.target).rstrip('.').lower()
for fingerprint, service in takeover_fingerprints.items():
if fingerprint in cname_target:
# Check if CNAME target resolves
try:
self.resolver.resolve(cname_target, 'A')
except dns.resolver.NXDOMAIN:
vulnerable.append({
'subdomain': subdomain,
'cname': cname_target,
'service': service,
'status': 'VULNERABLE',
'reason': 'CNAME points to non-existent resource'
})
except:
pass
break
except:
pass
return vulnerable

248
ai_platform/modules/domain_check/api/app/services/whois_service.py Normal file
View file

@ -0,0 +1,248 @@
"""
WHOIS Service - handles WHOIS/RDAP lookups
"""
import logging
from datetime import datetime
from typing import Dict, List, Optional, Union
import requests
try:
from dateutil import parser as dateutil_parser
DATEUTIL_AVAILABLE = True
except ImportError:
DATEUTIL_AVAILABLE = False
try:
import whois as python_whois
WHOIS_AVAILABLE = True
except ImportError:
WHOIS_AVAILABLE = False
logger = logging.getLogger(__name__)
def normalize_whois_date(value) -> Optional[datetime]:
"""Coerce any WHOIS date representation into a naive (tz-stripped) datetime.
python-whois and registry-specific WHOIS servers (notably ROTLD for .ro)
return dates as datetime, list-of-datetime, ISO strings, or arbitrary
free-form strings. Downstream code does ``datetime - value`` arithmetic, so
every date MUST arrive as a ``datetime`` or ``None`` — never a raw string.
A raw string here is the root cause of the historical 500 on ``.ro`` domains.
"""
if value is None:
return None
if isinstance(value, list):
# Registries often return the most recent date first; take the first set value.
for item in value:
parsed = normalize_whois_date(item)
if parsed is not None:
return parsed
return None
if isinstance(value, datetime):
return value.replace(tzinfo=None) if value.tzinfo else value
if isinstance(value, str):
text = value.strip()
if not text or text.lower() in ('none', 'null', 'n/a', '-'):
return None
if DATEUTIL_AVAILABLE:
try:
parsed = dateutil_parser.parse(text, fuzzy=True)
return parsed.replace(tzinfo=None) if parsed.tzinfo else parsed
except (ValueError, OverflowError, TypeError):
return None
# Fallback without dateutil: a few common explicit formats.
for fmt in ('%Y-%m-%dT%H:%M:%S.%fZ', '%Y-%m-%dT%H:%M:%SZ', '%Y-%m-%dT%H:%M:%S',
'%Y-%m-%d %H:%M:%S', '%Y-%m-%d', '%d.%m.%Y', '%d-%b-%Y'):
try:
return datetime.strptime(text, fmt)
except ValueError:
continue
return None
return None
def _as_list(value) -> List:
"""Normalize a WHOIS field that may be a scalar, None, or list into a clean list."""
if value is None:
return []
if isinstance(value, (list, tuple, set)):
return [v for v in value if v not in (None, '')]
return [value]
class WhoisService:
"""WHOIS/RDAP service class"""
def __init__(self, whoxy_api_key: str = None):
self.whoxy_api_key = whoxy_api_key
self.whoxy_url = 'https://api.whoxy.com/'
def lookup(self, domain: str, source: str = 'auto') -> Optional[Dict]:
"""
Perform WHOIS lookup
Args:
domain: Domain name to lookup
source: 'rdap', 'whoxy', or 'auto' (tries RDAP first, falls back to Whoxy)
Returns:
Dictionary with WHOIS data or None
"""
if source == 'auto':
# Try RDAP first
result = self._lookup_rdap(domain)
# Check if we got useful data (creation_date should exist)
if result and result.get('creation_date'):
logger.info(f'Using WHOIS data from python-whois for {domain}')
return result
# Fallback to Whoxy if available
if self.whoxy_api_key:
logger.info(f'Falling back to Whoxy API for {domain}')
whoxy_result = self._lookup_whoxy(domain)
if whoxy_result:
return whoxy_result
# If we got partial data from RDAP, return it
if result:
logger.warning(f'Returning incomplete WHOIS data for {domain}')
return result
return None
elif source == 'rdap':
return self._lookup_rdap(domain)
elif source == 'whoxy':
return self._lookup_whoxy(domain)
return None
def _lookup_rdap(self, domain: str) -> Optional[Dict]:
"""Lookup using python-whois library"""
if not WHOIS_AVAILABLE:
logger.warning('python-whois not available')
return None
try:
logger.info(f'WHOIS lookup for: {domain}')
w = python_whois.whois(domain)
# Normalize every date to datetime|None so downstream arithmetic is safe.
creation_date = normalize_whois_date(getattr(w, 'creation_date', None))
expiration_date = normalize_whois_date(getattr(w, 'expiration_date', None))
updated_date = normalize_whois_date(getattr(w, 'updated_date', None))
name_servers = [str(ns).lower() for ns in _as_list(getattr(w, 'name_servers', None))]
status = [str(s) for s in _as_list(getattr(w, 'status', None))]
emails = _as_list(getattr(w, 'emails', None))
result = {
'domain': domain,
'creation_date': creation_date,
'expiration_date': expiration_date,
'updated_date': updated_date,
'registrar': getattr(w, 'registrar', None),
'registrar_url': None,
'registrant_org': getattr(w, 'org', None),
'registrant_country': getattr(w, 'country', None),
'admin_email': emails[0] if emails else None,
'name_servers': name_servers,
'status': status,
'dnssec': None,
'data_source': 'whois',
'raw_data': {}
}
result['is_registered'] = self._is_registered(result)
return result
except Exception as e:
logger.error(f'WHOIS lookup failed for {domain}: {str(e)}')
return None
@staticmethod
def _is_registered(whois_result: Dict) -> Optional[bool]:
"""Best-effort determination of whether a domain is registered.
Returns True/False from WHOIS signals, or None when WHOIS is too sparse
to decide (common for ROTLD .ro) — the route then refines this using DNS.
"""
if whois_result.get('creation_date') or whois_result.get('registrar'):
return True
if whois_result.get('name_servers') or whois_result.get('status'):
return True
# No positive WHOIS signal at all: likely available, but let DNS confirm.
return None
def _lookup_whoxy(self, domain: str) -> Optional[Dict]:
"""Lookup using Whoxy API"""
if not self.whoxy_api_key:
logger.warning('Whoxy API key not configured')
return None
try:
logger.info(f'Whoxy API lookup for: {domain}')
response = requests.get(
self.whoxy_url,
params={
'key': self.whoxy_api_key,
'whois': domain
},
timeout=30
)
if response.status_code != 200:
logger.error(f'Whoxy API error: {response.status_code}')
return None
data = response.json()
if data.get('status') != 1:
logger.error(f'Whoxy API returned error: {data.get("status_reason")}')
return None
whois_data = data.get('whois_data', {})
result = {
'domain': domain,
'creation_date': normalize_whois_date(whois_data.get('create_date')),
'expiration_date': normalize_whois_date(whois_data.get('expiry_date')),
'updated_date': normalize_whois_date(whois_data.get('update_date')),
'registrar': whois_data.get('registrar_name'),
'registrar_url': whois_data.get('registrar_url'),
'registrant_org': whois_data.get('registrant_organization'),
'registrant_country': whois_data.get('registrant_country'),
'admin_email': whois_data.get('admin_email'),
'name_servers': [str(ns).lower() for ns in _as_list(whois_data.get('name_servers'))],
'status': [str(s) for s in _as_list(whois_data.get('domain_status'))],
'dnssec': whois_data.get('dnssec') == 'yes',
'data_source': 'whoxy',
'raw_data': whois_data
}
result['is_registered'] = self._is_registered(result)
return result
except Exception as e:
logger.error(f'Whoxy lookup failed for {domain}: {str(e)}')
return None
@staticmethod
def _parse_date(date_str: str) -> Optional[datetime]:
"""Parse date string to datetime"""
if not date_str:
return None
try:
# Try common date formats
for fmt in ['%Y-%m-%d', '%Y-%m-%dT%H:%M:%SZ', '%Y-%m-%d %H:%M:%S']:
try:
return datetime.strptime(date_str, fmt)
except ValueError:
continue
return None
except Exception:
return None

637
ai_platform/modules/domain_check/api/app/static/index.html Normal file
View file

@ -0,0 +1,637 @@
<!DOCTYPE html>
<html lang="ro">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Domain Check - Verificare Completă Domenii</title>
<style>
* { margin: 0; padding: 0; box-sizing: border-box; }
body {
font-family: 'Segoe UI', system-ui, sans-serif;
background: linear-gradient(135deg, #1a1a2e 0%, #16213e 50%, #0f3460 100%);
min-height: 100vh;
color: #e4e4e4;
}
.container { max-width: 1400px; margin: 0 auto; padding: 20px; }
h1 {
text-align: center;
font-size: 2rem;
margin-bottom: 10px;
background: linear-gradient(90deg, #00d9ff, #00ff88);
-webkit-background-clip: text;
-webkit-text-fill-color: transparent;
}
.subtitle { text-align: center; color: #888; margin-bottom: 30px; font-size: 0.9rem; }
.search-box {
background: rgba(255,255,255,0.05);
border-radius: 16px;
padding: 25px;
margin-bottom: 25px;
border: 1px solid rgba(255,255,255,0.1);
}
.input-group {
display: flex;
gap: 15px;
flex-wrap: wrap;
align-items: center;
}
.input-group input {
flex: 1;
min-width: 250px;
padding: 15px 20px;
border: 2px solid rgba(255,255,255,0.2);
border-radius: 10px;
font-size: 1.1rem;
background: rgba(0,0,0,0.3);
color: #fff;
}
.input-group input:focus { outline: none; border-color: #00d9ff; }
.btn {
padding: 15px 35px;
border: none;
border-radius: 10px;
font-size: 1rem;
cursor: pointer;
font-weight: 600;
transition: all 0.3s;
}
.btn-primary {
background: linear-gradient(135deg, #00d9ff, #00ff88);
color: #1a1a2e;
}
.btn-primary:hover { transform: translateY(-2px); box-shadow: 0 5px 20px rgba(0,217,255,0.4); }
.btn-primary:disabled { opacity: 0.5; cursor: not-allowed; transform: none; }
.options {
display: flex;
flex-wrap: wrap;
gap: 15px;
margin-top: 20px;
padding-top: 20px;
border-top: 1px solid rgba(255,255,255,0.1);
}
.option-group { display: flex; align-items: center; gap: 8px; }
.option-group input[type="checkbox"] { width: 18px; height: 18px; cursor: pointer; }
.option-group label { cursor: pointer; font-size: 0.9rem; }
.option-group.slow label { color: #ff9800; }
.loading { display: none; text-align: center; padding: 50px; }
.spinner {
width: 50px; height: 50px;
border: 4px solid rgba(255,255,255,0.1);
border-top-color: #00d9ff;
border-radius: 50%;
animation: spin 1s linear infinite;
margin: 0 auto 20px;
}
@keyframes spin { to { transform: rotate(360deg); } }
.results { display: none; }
.results-grid {
display: grid;
grid-template-columns: repeat(auto-fit, minmax(400px, 1fr));
gap: 20px;
}
.card {
background: rgba(255,255,255,0.05);
border-radius: 12px;
border: 1px solid rgba(255,255,255,0.1);
overflow: hidden;
}
.card-header {
padding: 15px 20px;
background: rgba(0,0,0,0.3);
display: flex;
justify-content: space-between;
align-items: center;
border-bottom: 1px solid rgba(255,255,255,0.1);
}
.card-header h3 { font-size: 1rem; display: flex; align-items: center; gap: 10px; }
.card-header .source { font-size: 0.75rem; color: #888; background: rgba(255,255,255,0.1); padding: 3px 8px; border-radius: 4px; }
.card-body { padding: 20px; }
.risk-card { grid-column: 1 / -1; }
.risk-score-display {
display: flex;
align-items: center;
gap: 30px;
flex-wrap: wrap;
}
.risk-circle {
width: 150px; height: 150px;
border-radius: 50%;
display: flex;
flex-direction: column;
align-items: center;
justify-content: center;
font-size: 3rem;
font-weight: bold;
flex-shrink: 0;
}
.risk-circle .label { font-size: 0.9rem; font-weight: normal; margin-top: 5px; }
.risk-low { background: linear-gradient(135deg, #00c853, #00e676); color: #1a1a2e; }
.risk-medium { background: linear-gradient(135deg, #ffc107, #ffeb3b); color: #1a1a2e; }
.risk-high { background: linear-gradient(135deg, #ff5722, #ff9800); color: #fff; }
.risk-critical { background: linear-gradient(135deg, #d32f2f, #f44336); color: #fff; }
.risk-details { flex: 1; min-width: 300px; }
.risk-factor {
display: flex;
justify-content: space-between;
align-items: center;
padding: 10px 0;
border-bottom: 1px solid rgba(255,255,255,0.1);
}
.risk-factor:last-child { border-bottom: none; }
.risk-factor .name { font-weight: 500; }
.risk-factor .score {
display: flex;
align-items: center;
gap: 10px;
}
.risk-factor .bar {
width: 100px;
height: 8px;
background: rgba(255,255,255,0.1);
border-radius: 4px;
overflow: hidden;
}
.risk-factor .bar-fill { height: 100%; transition: width 0.5s; }
.risk-factor .value { width: 40px; text-align: right; font-weight: bold; }
.formula-box {
background: rgba(0,0,0,0.3);
border-radius: 8px;
padding: 15px;
margin-top: 20px;
font-family: monospace;
font-size: 0.85rem;
overflow-x: auto;
}
.formula-box h4 { margin-bottom: 10px; color: #00d9ff; }
.formula-line { padding: 3px 0; }
.data-row {
display: flex;
padding: 10px 0;
border-bottom: 1px solid rgba(255,255,255,0.05);
}
.data-row:last-child { border-bottom: none; }
.data-label { width: 140px; color: #888; font-size: 0.9rem; flex-shrink: 0; }
.data-value { flex: 1; word-break: break-all; }
.badge {
display: inline-block;
padding: 3px 10px;
border-radius: 4px;
font-size: 0.8rem;
margin: 2px;
}
.badge.success { background: rgba(0,200,83,0.2); color: #00c853; }
.badge.danger { background: rgba(244,67,54,0.2); color: #f44336; }
.badge.warning { background: rgba(255,152,0,0.2); color: #ff9800; }
.badge.info { background: rgba(0,217,255,0.2); color: #00d9ff; }
.badge.neutral { background: rgba(255,255,255,0.1); color: #aaa; }
.port-list { display: flex; flex-wrap: wrap; gap: 5px; }
.port-item {
padding: 5px 10px;
border-radius: 4px;
font-size: 0.8rem;
background: rgba(255,255,255,0.1);
}
.port-item.danger { background: rgba(244,67,54,0.2); color: #f44336; }
.port-item.warning { background: rgba(255,152,0,0.2); color: #ff9800; }
.port-item.safe { background: rgba(0,200,83,0.2); color: #00c853; }
.blacklist-item {
display: flex;
justify-content: space-between;
padding: 8px 0;
border-bottom: 1px solid rgba(255,255,255,0.05);
}
.blacklist-clean { color: #00c853; }
.blacklist-listed { color: #f44336; }
.error {
background: rgba(244,67,54,0.2);
border: 1px solid #f44336;
border-radius: 8px;
padding: 20px;
text-align: center;
color: #f44336;
}
.tech-tags { display: flex; flex-wrap: wrap; gap: 8px; }
.tech-tag {
background: linear-gradient(135deg, rgba(0,217,255,0.2), rgba(0,255,136,0.2));
padding: 5px 12px;
border-radius: 20px;
font-size: 0.85rem;
}
.security-header {
display: flex;
justify-content: space-between;
align-items: center;
padding: 8px 0;
border-bottom: 1px solid rgba(255,255,255,0.05);
}
.header-present { color: #00c853; }
.header-missing { color: #f44336; }
@media (max-width: 768px) {
.results-grid { grid-template-columns: 1fr; }
.risk-score-display { flex-direction: column; text-align: center; }
.input-group { flex-direction: column; }
.input-group input { width: 100%; }
}
</style>
</head>
<body>
<div class="container">
<h1>🔍 Domain Check</h1>
<p class="subtitle">Verificare completă domenii - WHOIS, DNS, SSL, IP Intelligence, HTTP Security, Blacklists</p>
<div class="search-box">
<div class="input-group">
<input type="text" id="domainInput" placeholder="Introdu domeniul (ex: google.com, example.ro)" autofocus>
<button class="btn btn-primary" id="checkBtn" onclick="checkDomain()">Verifică Domeniul</button>
</div>
<div class="options">
<div class="option-group">
<input type="checkbox" id="optWhois" checked>
<label for="optWhois">WHOIS</label>
</div>
<div class="option-group">
<input type="checkbox" id="optDns" checked>
<label for="optDns">DNS</label>
</div>
<div class="option-group">
<input type="checkbox" id="optSsl" checked>
<label for="optSsl">SSL/TLS</label>
</div>
<div class="option-group">
<input type="checkbox" id="optIp" checked>
<label for="optIp">IP Intelligence</label>
</div>
<div class="option-group">
<input type="checkbox" id="optHttp" checked>
<label for="optHttp">HTTP Analysis</label>
</div>
<div class="option-group">
<input type="checkbox" id="optBlacklist" checked>
<label for="optBlacklist">Blacklists</label>
</div>
<div class="option-group slow">
<input type="checkbox" id="optPorts">
<label for="optPorts">Port Scan (lent)</label>
</div>
<div class="option-group slow">
<input type="checkbox" id="optSubdomains">
<label for="optSubdomains">Subdomenii (lent)</label>
</div>
</div>
</div>
<div class="loading" id="loading">
<div class="spinner"></div>
<p>Se verifică domeniul... Acest proces poate dura până la 30 de secunde.</p>
</div>
<div class="results" id="results"></div>
</div>
<script>
const API_URL = '/api/v1/check/check';
document.getElementById('domainInput').addEventListener('keypress', (e) => {
if (e.key === 'Enter') checkDomain();
});
async function checkDomain() {
const domain = document.getElementById('domainInput').value.trim();
if (!domain) { alert('Introdu un domeniu!'); return; }
const btn = document.getElementById('checkBtn');
const loading = document.getElementById('loading');
const results = document.getElementById('results');
btn.disabled = true;
loading.style.display = 'block';
results.style.display = 'none';
const checkOptions = {
whois: document.getElementById('optWhois').checked,
dns: document.getElementById('optDns').checked,
ssl: document.getElementById('optSsl').checked,
ip_intelligence: document.getElementById('optIp').checked,
http_analysis: document.getElementById('optHttp').checked,
blacklist: document.getElementById('optBlacklist').checked,
port_scan: document.getElementById('optPorts').checked,
subdomains: document.getElementById('optSubdomains').checked
};
try {
const response = await fetch(API_URL, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ domain, check_options: checkOptions })
});
const data = await response.json();
if (data.success) {
displayResults(data.data, data.metadata);
} else {
results.innerHTML = `<div class="error">❌ ${data.error?.message || 'Eroare necunoscută'}</div>`;
}
} catch (err) {
results.innerHTML = `<div class="error">❌ Eroare de conexiune: ${err.message}</div>`;
}
btn.disabled = false;
loading.style.display = 'none';
results.style.display = 'block';
}
function displayResults(d, meta) {
let html = '<div class="results-grid">';
if (d.availability) html += renderAvailabilityCard(d.availability);
if (d.risk_score) html += renderRiskCard(d.risk_score);
if (d.whois) html += renderWhoisCard(d.whois);
if (d.dns) html += renderDnsCard(d.dns);
if (d.mail) html += renderMailCard(d.mail);
if (d.ssl) html += renderSslCard(d.ssl);
if (d.ip_intelligence) html += renderIpCard(d.ip_intelligence);
if (d.http_analysis) html += renderHttpCard(d.http_analysis);
if (d.blacklist) html += renderBlacklistCard(d.blacklist);
if (d.port_scan) html += renderPortCard(d.port_scan);
if (d.subdomains) html += renderSubdomainsCard(d.subdomains);
html += '</div>';
html += `<div style="text-align: center; margin-top: 20px; color: #666; font-size: 0.85rem;">
Check ID: ${d.check_id} | Timp procesare: ${meta.processing_time_ms}ms | API: ${meta.api_version}
</div>`;
document.getElementById('results').innerHTML = html;
}
function renderAvailabilityCard(a) {
const reg = a.is_registered;
const label = reg ? '🔒 Înregistrat' : '🟢 Disponibil';
const badgeClass = reg ? 'info' : 'success';
return `
<div class="card">
<div class="card-header"><h3>📌 Disponibilitate</h3><span class="source">WHOIS+DNS</span></div>
<div class="card-body">
<div class="data-row"><div class="data-label">Status:</div><div class="data-value"><span class="badge ${badgeClass}" style="font-size:1rem">${label}</span></div></div>
<div class="data-row"><div class="data-label">Încredere:</div><div class="data-value">${a.confidence}</div></div>
<div class="data-row"><div class="data-label">Semnale:</div><div class="data-value">
<span class="badge ${a.signals?.whois_has_data ? 'success' : 'neutral'}">WHOIS ${a.signals?.whois_has_data ? '✓' : '–'}</span>
<span class="badge ${a.signals?.dns_resolves ? 'success' : 'neutral'}">DNS ${a.signals?.dns_resolves ? '✓' : '–'}</span>
</div></div>
</div>
</div>`;
}
function renderMailCard(m) {
const gradeColor = {A:'#00c853', B:'#8bc34a', C:'#ffc107', D:'#ff9800', F:'#f44336'}[m.mail_security_grade] || '#888';
const spf = m.spf || {}, dmarc = m.dmarc || {}, dkim = m.dkim || {}, mx = m.mx || {}, sts = m.mta_sts || {};
const tlsRow = (mx.starttls !== null && mx.starttls !== undefined) ? ` · STARTTLS ${mx.starttls ? '✓' : '✗'}` : '';
return `
<div class="card">
<div class="card-header"><h3>📧 Mail Intelligence</h3><span class="source">EMAIL</span></div>
<div class="card-body">
<div class="data-row"><div class="data-label">Securitate email:</div><div class="data-value"><span class="badge" style="background:${gradeColor};color:#fff">${m.mail_security_grade} · ${m.mail_security_score}/100</span></div></div>
${m.provider ? `<div class="data-row"><div class="data-label">Provider:</div><div class="data-value">${m.provider}</div></div>` : ''}
<div class="data-row"><div class="data-label">MX:</div><div class="data-value">${mx.count || 0} servere${tlsRow}${mx.dane ? ' · DANE ✓' : ''}</div></div>
<div class="data-row"><div class="data-label">SPF:</div><div class="data-value"><span class="badge ${spf.present ? 'success' : 'danger'}">${spf.present ? '✓' : '✗'}</span> ${spf.policy || ''}</div></div>
<div class="data-row"><div class="data-label">DKIM:</div><div class="data-value"><span class="badge ${dkim.present ? 'success' : 'danger'}">${dkim.present ? '✓' : '✗'}</span> ${(dkim.selectors_found || []).join(', ')}</div></div>
<div class="data-row"><div class="data-label">DMARC:</div><div class="data-value"><span class="badge ${dmarc.present ? 'success' : 'danger'}">${dmarc.present ? '✓' : '✗'}</span> ${dmarc.policy ? ('p=' + dmarc.policy) : ''}</div></div>
<div class="data-row"><div class="data-label">MTA-STS:</div><div class="data-value"><span class="badge ${sts.present ? 'success' : 'neutral'}">${sts.present ? '✓' : '–'}</span>${sts.mode ? (' ' + sts.mode) : ''} ${m.tls_rpt?.present ? '<span class="badge success">TLS-RPT ✓</span>' : ''}</div></div>
${m.deliverability?.tested ? `<div class="data-row"><div class="data-label">Catch-all:</div><div class="data-value">${m.deliverability.catch_all ? '<span class="badge warning">DA</span>' : '<span class="badge success">NU</span>'}</div></div>` : ''}
</div>
</div>`;
}
function renderRiskCard(risk) {
const levelClass = `risk-${risk.level.toLowerCase()}`;
let factorsHtml = '';
if (risk.factors && risk.factors.length > 0) {
risk.factors.forEach(f => {
const barColor = f.score <= 25 ? '#00c853' : f.score <= 50 ? '#ffc107' : f.score <= 75 ? '#ff9800' : '#f44336';
factorsHtml += `
<div class="risk-factor">
<div class="name">${formatFactorName(f.factor)}</div>
<div class="score">
<div class="bar"><div class="bar-fill" style="width: ${f.score}%; background: ${barColor}"></div></div>
<div class="value">${f.score}</div>
</div>
</div>`;
});
}
let formulaHtml = '';
if (risk.formula_breakdown && risk.formula_breakdown.length > 0) {
formulaHtml = '<div class="formula-box"><h4>📐 Formula de Calcul</h4>';
risk.formula_breakdown.forEach(fb => {
formulaHtml += `<div class="formula-line">${fb.category}: ${fb.formula}</div>`;
});
formulaHtml += `<div class="formula-line" style="margin-top: 10px; font-weight: bold; color: #00d9ff;">${risk.formula_string || ''}</div></div>`;
}
return `
<div class="card risk-card">
<div class="card-header"><h3>🎯 Scor de Risc</h3><span class="source">CALCULATED</span></div>
<div class="card-body">
<div class="risk-score-display">
<div class="risk-circle ${levelClass}">${risk.total}<span class="label">${risk.level}</span></div>
<div class="risk-details">
<div style="margin-bottom: 15px;">
${risk.is_new_domain ? '<span class="badge warning">Domeniu Nou</span>' : ''}
${risk.is_suspicious ? '<span class="badge danger">Suspect</span>' : ''}
${risk.is_blacklisted ? '<span class="badge danger">Blacklisted</span>' : ''}
${!risk.is_suspicious && !risk.is_blacklisted ? '<span class="badge success">OK</span>' : ''}
</div>
${factorsHtml}
</div>
</div>
${formulaHtml}
<div style="margin-top: 15px; font-size: 0.85rem; color: #888;">Praguri: LOW (0-25) | MEDIUM (26-50) | HIGH (51-75) | CRITICAL (76-100)</div>
</div>
</div>`;
}
function renderWhoisCard(w) {
const statusArray = Array.isArray(w.status) ? w.status : (w.status ? [w.status] : []);
const nsArray = Array.isArray(w.name_servers) ? w.name_servers : (w.name_servers ? [w.name_servers] : []);
return `
<div class="card">
<div class="card-header"><h3>📋 WHOIS</h3><span class="source">${w.data_source || 'WHOIS'}</span></div>
<div class="card-body">
${w.creation_date ? `<div class="data-row"><div class="data-label">Data Creare:</div><div class="data-value">${formatDate(w.creation_date)}</div></div>` : ''}
${w.age_days !== null && w.age_days !== undefined ? `<div class="data-row"><div class="data-label">Vârsta:</div><div class="data-value"><strong>${w.age_days} zile</strong> (${Math.floor(w.age_days/365)} ani)</div></div>` : ''}
${w.expiration_date ? `<div class="data-row"><div class="data-label">Expirare:</div><div class="data-value">${formatDate(w.expiration_date)}</div></div>` : ''}
${w.registrar ? `<div class="data-row"><div class="data-label">Registrar:</div><div class="data-value">${w.registrar}</div></div>` : ''}
${w.dnssec ? `<div class="data-row"><div class="data-label">DNSSEC:</div><div class="data-value">${w.dnssec}</div></div>` : ''}
${statusArray.length > 0 ? `<div class="data-row"><div class="data-label">Status:</div><div class="data-value">${statusArray.map(s => `<span class="badge info">${s}</span>`).join('')}</div></div>` : ''}
${nsArray.length > 0 ? `<div class="data-row"><div class="data-label">Name Servers:</div><div class="data-value">${nsArray.map(ns => `<span class="badge neutral">${ns}</span>`).join('')}</div></div>` : ''}
</div>
</div>`;
}
function renderDnsCard(dns) {
return `
<div class="card">
<div class="card-header"><h3>🌐 DNS Records</h3><span class="source">DNS</span></div>
<div class="card-body">
${dns.a_records?.length ? `<div class="data-row"><div class="data-label">A (IPv4):</div><div class="data-value">${dns.a_records.map(r => `<span class="badge info">${r}</span>`).join('')}</div></div>` : ''}
${dns.aaaa_records?.length ? `<div class="data-row"><div class="data-label">AAAA (IPv6):</div><div class="data-value">${dns.aaaa_records.map(r => `<span class="badge info">${r}</span>`).join('')}</div></div>` : ''}
${dns.mx_records?.length ? `<div class="data-row"><div class="data-label">MX:</div><div class="data-value">${dns.mx_records.map(r => `<span class="badge neutral">${r.priority} ${r.host}</span>`).join('')}</div></div>` : ''}
${dns.ns_records?.length ? `<div class="data-row"><div class="data-label">NS:</div><div class="data-value">${dns.ns_records.map(r => `<span class="badge neutral">${r}</span>`).join('')}</div></div>` : ''}
<div class="data-row"><div class="data-label">SPF:</div><div class="data-value"><span class="badge ${dns.has_spf ? 'success' : 'danger'}">${dns.has_spf ? '✓' : '✗'} SPF</span> <span style="color:#888;font-size:0.8rem">(DKIM/DMARC: vezi cardul Mail Intelligence)</span></div></div>
</div>
</div>`;
}
function renderSslCard(ssl) {
if (!ssl.has_ssl) {
return `<div class="card"><div class="card-header"><h3>🔒 SSL/TLS</h3><span class="source">SSL</span></div><div class="card-body"><div class="error">❌ Nu are certificat SSL</div></div></div>`;
}
return `
<div class="card">
<div class="card-header"><h3>🔒 SSL/TLS</h3><span class="source">SSL</span></div>
<div class="card-body">
<div class="data-row"><div class="data-label">Status:</div><div class="data-value">
<span class="badge ${ssl.is_valid ? 'success' : 'danger'}">${ssl.is_valid ? '✓ Valid' : '✗ Invalid'}</span>
${ssl.is_expired ? '<span class="badge danger">Expirat</span>' : ''}
${ssl.is_wildcard ? '<span class="badge info">Wildcard</span>' : ''}
</div></div>
${ssl.issuer ? `<div class="data-row"><div class="data-label">Issuer:</div><div class="data-value">${ssl.issuer}</div></div>` : ''}
${ssl.valid_until ? `<div class="data-row"><div class="data-label">Valid Until:</div><div class="data-value">${formatDate(ssl.valid_until)}</div></div>` : ''}
${ssl.days_until_expiry !== null ? `<div class="data-row"><div class="data-label">Expiră în:</div><div class="data-value"><span class="badge ${ssl.days_until_expiry < 30 ? 'warning' : 'success'}">${ssl.days_until_expiry} zile</span></div></div>` : ''}
</div>
</div>`;
}
function renderIpCard(ip) {
return `
<div class="card">
<div class="card-header"><h3>🌍 IP Intelligence</h3><span class="source">ipinfo.io</span></div>
<div class="card-body">
<div class="data-row"><div class="data-label">IP:</div><div class="data-value"><strong>${ip.ip}</strong></div></div>
${ip.reverse_dns ? `<div class="data-row"><div class="data-label">Reverse DNS:</div><div class="data-value">${ip.reverse_dns}</div></div>` : ''}
${ip.country ? `<div class="data-row"><div class="data-label">Locație:</div><div class="data-value">${ip.city || ''} ${ip.region ? ', ' + ip.region : ''}, ${ip.country}</div></div>` : ''}
${ip.asn ? `<div class="data-row"><div class="data-label">ASN:</div><div class="data-value">${ip.asn}</div></div>` : ''}
${ip.isp ? `<div class="data-row"><div class="data-label">ISP:</div><div class="data-value">${ip.isp}</div></div>` : ''}
<div class="data-row"><div class="data-label">Tip:</div><div class="data-value"><span class="badge ${ip.is_datacenter ? 'info' : 'success'}">${ip.is_datacenter ? 'Datacenter' : 'Rezidențial'}</span></div></div>
${ip.hosting_score ? `<div class="data-row"><div class="data-label">Hosting Score:</div><div class="data-value"><span class="badge ${ip.hosting_score.score <= 30 ? 'success' : ip.hosting_score.score <= 50 ? 'warning' : 'danger'}">${ip.hosting_score.score}/100</span></div></div>` : ''}
</div>
</div>`;
}
function renderHttpCard(http) {
const secHeaders = http.security_headers || {};
let headersHtml = '';
['Strict-Transport-Security', 'X-Frame-Options', 'X-Content-Type-Options', 'Content-Security-Policy'].forEach(h => {
headersHtml += `<div class="security-header"><span>${h}</span><span class="${secHeaders[h] ? 'header-present' : 'header-missing'}">${secHeaders[h] ? '✓' : '✗'}</span></div>`;
});
return `
<div class="card">
<div class="card-header"><h3>🌐 HTTP Analysis</h3><span class="source">HTTP</span></div>
<div class="card-body">
<div class="data-row"><div class="data-label">HTTPS:</div><div class="data-value">
<span class="badge ${http.has_https ? 'success' : 'danger'}">${http.has_https ? '✓' : '✗'} HTTPS</span>
${http.http_to_https_redirect ? '<span class="badge success">Auto-redirect</span>' : ''}
</div></div>
${http.server ? `<div class="data-row"><div class="data-label">Server:</div><div class="data-value">${http.server}</div></div>` : ''}
${http.cms ? `<div class="data-row"><div class="data-label">CMS:</div><div class="data-value"><span class="badge info">${http.cms}</span></div></div>` : ''}
${http.technologies?.length ? `<div class="data-row"><div class="data-label">Tech:</div><div class="data-value"><div class="tech-tags">${http.technologies.slice(0,8).map(t => `<span class="tech-tag">${t}</span>`).join('')}</div></div></div>` : ''}
${http.response_time_ms ? `<div class="data-row"><div class="data-label">Response:</div><div class="data-value">${http.response_time_ms}ms</div></div>` : ''}
<div style="margin-top: 15px; font-weight: 500; margin-bottom: 10px;">Security Headers:</div>
${headersHtml}
</div>
</div>`;
}
function renderBlacklistCard(bl) {
let checksHtml = '';
if (bl.ip_check) {
const clean = bl.ip_check.clean_lists?.slice(0,4) || [];
const listed = bl.ip_check.listings || [];
checksHtml += `<div style="margin-bottom: 10px;"><strong>IP (${bl.ip_check.total_checked}):</strong> ${listed.map(l => `<span class="badge danger">${l.list_name}</span>`).join('')} ${clean.map(l => `<span class="badge success">${l}</span>`).join('')}</div>`;
}
if (bl.domain_check) {
const clean = bl.domain_check.clean_lists?.slice(0,4) || [];
const listed = bl.domain_check.listings || [];
checksHtml += `<div><strong>Domain (${bl.domain_check.total_checked}):</strong> ${listed.map(l => `<span class="badge danger">${l.list_name}</span>`).join('')} ${clean.map(l => `<span class="badge success">${l}</span>`).join('')}</div>`;
}
return `
<div class="card">
<div class="card-header"><h3>🛡️ Blacklists</h3><span class="source">DNSBL</span></div>
<div class="card-body">
<div class="data-row"><div class="data-label">Status:</div><div class="data-value"><span class="badge ${bl.is_blacklisted ? 'danger' : 'success'}">${bl.is_blacklisted ? '❌ BLACKLISTED' : '✓ CLEAN'}</span></div></div>
<div class="data-row"><div class="data-label">Reputation:</div><div class="data-value"><span class="badge ${bl.reputation_score >= 80 ? 'success' : bl.reputation_score >= 60 ? 'warning' : 'danger'}">${bl.reputation_score}/100</span></div></div>
${checksHtml}
</div>
</div>`;
}
function renderPortCard(ports) {
const openPorts = ports.open_ports || [];
const dangerous = ports.dangerous_open || [];
const summary = ports.scan_summary || {};
let portsHtml = '<div class="port-list">';
openPorts.forEach(p => {
const isDanger = dangerous.some(d => d.port === p.port);
portsHtml += `<span class="port-item ${isDanger ? 'danger' : 'safe'}">${p.port} ${p.service}</span>`;
});
portsHtml += '</div>';
return `
<div class="card">
<div class="card-header"><h3>🔌 Port Scan</h3><span class="source">TCP</span></div>
<div class="card-body">
<div class="data-row"><div class="data-label">Rezumat:</div><div class="data-value">
<span class="badge info">${summary.open_count || 0} deschise</span>
${summary.dangerous_count > 0 ? `<span class="badge danger">${summary.dangerous_count} periculoase</span>` : ''}
</div></div>
<div class="data-row"><div class="data-label">Porturi:</div><div class="data-value">${portsHtml}</div></div>
</div>
</div>`;
}
function renderSubdomainsCard(subs) {
const subList = subs.subdomains || [];
const src = subs.sources || {};
const srcCounts = [
['CT', (src.certificate_transparency || []).length],
['Brute', (src.dns_bruteforce || []).length],
['AXFR', (src.zone_transfer || []).length],
['DNS', (src.dns_records || []).length],
['Manual', (src.user_provided || []).length],
].filter(([, n]) => n > 0).map(([k, n]) => `<span class="badge neutral">${k}: ${n}</span>`).join(' ');
return `
<div class="card">
<div class="card-header"><h3>🔎 Subdomenii</h3><span class="source">CT+DNS+Brute</span></div>
<div class="card-body">
<div class="data-row"><div class="data-label">Total:</div><div class="data-value"><strong>${subs.total_found}</strong> ${srcCounts}</div></div>
<div class="data-row"><div class="data-label">Lista:</div><div class="data-value"><div class="tech-tags">${subList.slice(0, 20).map(s => `<span class="tech-tag">${s}</span>`).join('')}${subList.length > 20 ? `<span class="badge neutral">+${subList.length - 20}</span>` : ''}</div></div></div>
${subs.has_wildcard_cert ? `<div class="data-row"><div class="data-label">⚠️ Wildcard:</div><div class="data-value"><span class="badge warning">cert *.${subs.domain}</span></div></div>` : ''}
${subs.coverage_note ? `<div style="margin-top:8px;font-size:0.78rem;color:#b8860b;line-height:1.4">ℹ️ ${subs.coverage_note}</div>` : ''}
</div>
</div>`;
}
function formatFactorName(factor) {
const names = {'domain_age': '📅 Vârsta Domeniu', 'ssl': '🔒 SSL/TLS', 'dns': '🌐 DNS', 'email_security': '📧 Email Security', 'whois': '📋 WHOIS', 'ip_reputation': '🌍 IP Reputation', 'http_security': '🔐 HTTP Security', 'blacklist': '🛡️ Blacklists', 'port_security': '🔌 Porturi'};
return names[factor] || factor;
}
function formatDate(dateStr) {
if (!dateStr) return '-';
try { return new Date(dateStr).toLocaleDateString('ro-RO', { year: 'numeric', month: 'long', day: 'numeric' }); }
catch { return dateStr; }
}
</script>
</body>
</html>

0
ai_platform/modules/domain_check/api/app/tasks/__init__.py Normal file
View file

0
ai_platform/modules/domain_check/api/app/utils/__init__.py Normal file
View file

91
ai_platform/modules/domain_check/api/requirements.txt Normal file
View file

@ -0,0 +1,91 @@
# Flask Framework
Flask==3.0.0
Flask-RESTX==1.3.0
Flask-SQLAlchemy==3.1.1
Flask-Migrate==4.0.5
Flask-CORS==4.0.0
Flask-Limiter==3.5.0
# Database
SQLAlchemy==2.0.23
psycopg2-binary==2.9.9
alembic==1.13.1
# Redis & Caching
redis==5.0.1
hiredis==2.3.2
# Celery (Task Queue)
celery==5.3.4
# HTTP Requests (moved before WHOIS tools to resolve dependencies)
requests==2.31.0
urllib3==2.1.0
httpx==0.25.2
# WHOIS & Domain Tools
whoisit>=2.0.0
python-whois==0.8.0
dnspython==2.4.2
tldextract==5.1.1
# Validation & Serialization
marshmallow==3.20.1
marshmallow-sqlalchemy==0.29.0
pydantic==2.5.2
# Environment & Configuration
python-dotenv==1.0.0
environs==10.3.0
# Date & Time
python-dateutil==2.8.2
pytz==2023.3
# Logging & Monitoring
structlog==23.3.0
sentry-sdk[flask]==1.39.1
# Security
PyJWT==2.8.0
cryptography==41.0.7
validators==0.22.0
# API Documentation
apispec==6.3.1
apispec-webframeworks==1.0.0
# Utilities
Click==8.1.7
colorama==0.4.6
python-slugify==8.0.1
shortuuid==1.0.11
# Testing
pytest==7.4.3
pytest-cov==4.1.0
pytest-flask==1.3.0
pytest-mock==3.12.0
faker==21.0.0
# Code Quality
black==23.12.1
flake8==6.1.0
pylint==3.0.3
mypy==1.7.1
# Performance
gunicorn==21.2.0
gevent==23.9.1
# Data Processing
pandas==2.1.4
numpy==1.26.2
# IP & Network
ipaddress==1.0.23
geoip2==4.7.0
# SSL Certificate Handling
certifi==2023.11.17
pyOpenSSL==23.3.0

45
ai_platform/modules/domain_check/api/run.py Normal file
View file

@ -0,0 +1,45 @@
"""
Domain Check API - Entry Point
Version: 1.0.0
"""
import os
import sys
from pathlib import Path
# Add app directory to Python path
sys.path.insert(0, str(Path(__file__).parent))
from app import create_app
# Create Flask app instance
app = create_app(os.getenv('FLASK_ENV', 'development'))
if __name__ == '__main__':
host = os.getenv('API_HOST', '0.0.0.0')
port = int(os.getenv('API_PORT', 5000))
debug = os.getenv('DEBUG', 'True').lower() == 'true'
print(f"""
╔══════════════════════════════════════════╗
║ Domain Check API - Anti-Fake News ║
║ Version: 1.0.0 ║
║ Environment: {os.getenv('FLASK_ENV', 'development'):<26}║
║ Host: {host:<32}║
║ Port: {port:<32}║
╚══════════════════════════════════════════╝
📚 API Documentation:
- Swagger UI: http://{host}:{port}/docs
- ReDoc: http://{host}:{port}/redoc
🏥 Health Check: http://{host}:{port}/health
🚀 Starting server...
""")
app.run(
host=host,
port=port,
debug=debug,
threaded=True
)

104
ai_platform/modules/domain_check/api/tests/test_whois_dates.py Normal file
View file

@ -0,0 +1,104 @@
"""
Regression tests for WHOIS date handling.
Root cause of the historical HTTP 500 on `.ro` domains (e.g. exemplu.ro):
python-whois returns ROTLD dates as raw strings, and downstream code did
`datetime - str`, raising TypeError. These tests lock in the fix.
"""
from datetime import datetime
import pytest
from app.services.whois_service import normalize_whois_date, _as_list, WhoisService
from app.services.risk_scorer import RiskScorer, _coerce_datetime
# --- normalize_whois_date -------------------------------------------------
@pytest.mark.parametrize("value", [
None,
"",
" ",
"none",
"N/A",
])
def test_normalize_returns_none_for_empty(value):
assert normalize_whois_date(value) is None
def test_normalize_passthrough_datetime():
dt = datetime(2020, 1, 2, 3, 4, 5)
assert normalize_whois_date(dt) == dt
def test_normalize_iso_string():
assert normalize_whois_date("2020-01-02T03:04:05Z") == datetime(2020, 1, 2, 3, 4, 5)
def test_normalize_rotld_style_string():
# ROTLD / free-form formats that broke datetime.fromisoformat
assert normalize_whois_date("2015-03-22") == datetime(2015, 3, 22)
assert normalize_whois_date("22.03.2015").year == 2015
assert normalize_whois_date("Before 2012-08-10") is not None # fuzzy
def test_normalize_list_takes_first_valid():
out = normalize_whois_date([None, "2019-05-06", datetime(2021, 1, 1)])
assert out == datetime(2019, 5, 6)
def test_normalize_garbage_returns_none():
assert normalize_whois_date("not a date at all !!") is None
def test_as_list_normalizes():
assert _as_list(None) == []
assert _as_list("a") == ["a"]
assert _as_list(["a", None, "", "b"]) == ["a", "b"]
# --- the actual crash site: risk scorer must never raise on bad dates -----
def test_score_domain_age_does_not_raise_on_string():
"""The exact regression: a raw string creation_date must not 500."""
scorer = RiskScorer()
score, details = scorer._score_domain_age({'creation_date': '2015-03-22'})
assert isinstance(score, int)
assert details['details']['age_days'] is not None
def test_score_domain_age_does_not_raise_on_garbage():
scorer = RiskScorer()
score, details = scorer._score_domain_age({'creation_date': 'totally-not-a-date'})
# Garbage coerces to None -> "unable to determine" path, never an exception.
assert score == 50
assert details['details']['age_days'] is None
def test_score_domain_age_missing():
scorer = RiskScorer()
score, details = scorer._score_domain_age({})
assert score == 50
def test_coerce_datetime():
assert _coerce_datetime(None) is None
assert _coerce_datetime("2020-01-01") == datetime(2020, 1, 1)
assert _coerce_datetime(datetime(2020, 1, 1)) == datetime(2020, 1, 1)
assert _coerce_datetime("garbage") is None
# --- is_registered signal -------------------------------------------------
def test_is_registered_true_when_creation_date():
assert WhoisService._is_registered({'creation_date': datetime(2020, 1, 1)}) is True
def test_is_registered_true_when_registrar():
assert WhoisService._is_registered({'registrar': 'ROTLD'}) is True
def test_is_registered_none_when_sparse():
# No positive signal -> inconclusive (None), DNS decides downstream.
assert WhoisService._is_registered({'creation_date': None, 'registrar': None,
'name_servers': [], 'status': []}) is None

42
ai_platform/modules/domain_check/dashboard/Dockerfile Normal file
View file

@ -0,0 +1,42 @@
# Streamlit Dashboard Dockerfile
FROM python:3.10-slim
# Set working directory
WORKDIR /app
# Install system dependencies
RUN apt-get update && apt-get install -y --no-install-recommends \
curl \
&& rm -rf /var/lib/apt/lists/*
# Copy requirements
COPY requirements.txt .
# Install Python dependencies
RUN pip install --no-cache-dir --upgrade pip && \
pip install --no-cache-dir -r requirements.txt
# Create non-root user
RUN useradd -m -u 1000 streamlit && \
chown -R streamlit:streamlit /app
# Copy application code
COPY --chown=streamlit:streamlit . /app/
# Create Streamlit config directory
RUN mkdir -p /home/streamlit/.streamlit && \
chown -R streamlit:streamlit /home/streamlit
# Switch to non-root user
USER streamlit
# Expose Streamlit port
EXPOSE 8501
# Health check
HEALTHCHECK --interval=30s --timeout=10s --start-period=20s --retries=3 \
CMD curl -f http://localhost:8501/_stcore/health || exit 1
# Run Streamlit
CMD ["streamlit", "run", "app.py", "--server.port=8501", "--server.address=0.0.0.0", "--server.headless=true", "--browser.gatherUsageStats=false"]

26
ai_platform/modules/domain_check/dashboard/requirements.txt Normal file
View file

@ -0,0 +1,26 @@
# Streamlit Framework
streamlit==1.29.0
# HTTP Requests
requests==2.31.0
httpx==0.25.2
# Data Processing & Visualization
pandas==2.1.4
numpy==1.26.2
plotly==5.18.0
altair==5.2.0
# Date & Time
python-dateutil==2.8.2
pytz==2023.3
# Utilities
python-dotenv==1.0.0
validators==0.22.0
# Caching
streamlit-autorefresh==1.0.1
# Formatting
humanize==4.9.0

291
ai_platform/modules/domain_check/deploy.sh Executable file
View file

@ -0,0 +1,291 @@
#!/bin/bash
# ===========================================
# Domain Check API - Deployment Script
# ===========================================
# Usage: ./deploy.sh [install|update|status|logs]
# ===========================================
set -e
# Colors
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
NC='\033[0m'
# Configuration
REPO_URL="<didi-lot1-ai>/ai_platform/modules/domain_check"
INSTALL_DIR="/home/$(whoami)/domain-check"
BRANCH="main"
log_info() { echo -e "${GREEN}[INFO]${NC} $1"; }
log_warn() { echo -e "${YELLOW}[WARN]${NC} $1"; }
log_error() { echo -e "${RED}[ERROR]${NC} $1"; }
check_requirements() {
log_info "Checking requirements..."
# Check Docker
if ! command -v docker &> /dev/null; then
log_error "Docker is not installed. Install it first:"
echo " curl -fsSL https://get.docker.com | sh"
echo " sudo usermod -aG docker \$USER"
exit 1
fi
# Check Docker Compose
if ! docker compose version &> /dev/null; then
log_error "Docker Compose is not installed."
exit 1
fi
# Check Git
if ! command -v git &> /dev/null; then
log_error "Git is not installed. Install it first:"
echo " sudo apt install git -y"
exit 1
fi
log_info "All requirements met ✓"
}
install() {
log_info "Installing Domain Check API..."
check_requirements
# Clone repository
if [ -d "$INSTALL_DIR" ]; then
log_warn "Directory $INSTALL_DIR already exists."
read -p "Remove and reinstall? (y/N) " -n 1 -r
echo
if [[ $REPLY =~ ^[Yy]$ ]]; then
rm -rf "$INSTALL_DIR"
else
log_info "Use './deploy.sh update' to update existing installation."
exit 0
fi
fi
log_info "Cloning repository..."
git clone "$REPO_URL" "$INSTALL_DIR"
cd "$INSTALL_DIR"
# Create .env file
create_env
# Build and start
log_info "Building and starting containers..."
docker compose up -d --build
# Wait for health
log_info "Waiting for services to be healthy..."
sleep 10
# Check status
status
log_info "Installation complete!"
log_info "API available at: http://$(hostname -I | awk '{print $1}'):51000/"
}
create_env() {
log_info "Creating .env file..."
if [ -f ".env" ]; then
log_warn ".env file already exists. Skipping creation."
return
fi
# Generate secure password and secret key
DB_PASS=$(openssl rand -base64 24 | tr -dc 'a-zA-Z0-9' | head -c 24)
SECRET=$(openssl rand -base64 32 | tr -dc 'a-zA-Z0-9' | head -c 32)
cat > .env << EOF
# ===========================================
# Domain Check API - Environment Configuration
# Port Schema: Dev (5xxxx) per Port_Allocation_Schema_v2
# Generated: $(date)
# ===========================================
# Database Configuration (52000 = x2xxx = Databases/PostgreSQL)
DB_PASSWORD=${DB_PASS}
DB_USER=dns_admin
DB_NAME=domain_check
DB_HOST=domain_check_postgres
DB_PORT=5432
DB_EXTERNAL_PORT=52000
# Redis Configuration (52300 = x23xx = Redis)
REDIS_HOST=domain_check_redis
REDIS_PORT=6379
REDIS_EXTERNAL_PORT=52300
REDIS_DB=0
REDIS_PASSWORD=
# API Keys (set your own keys)
WHOXY_API_KEY=
VIRUSTOTAL_API_KEY=
# Application Settings
FLASK_ENV=production
FLASK_APP=run.py
SECRET_KEY=${SECRET}
LOG_LEVEL=INFO
DEBUG=False
# Cache TTL (seconds)
REDIS_TTL_HOT=21600
REDIS_TTL_WARM=86400
REDIS_TTL_COLD=604800
# Risk Score Thresholds
RISK_THRESHOLD_LOW=30
RISK_THRESHOLD_MEDIUM=60
RISK_THRESHOLD_HIGH=85
# Domain Age Thresholds (days)
DOMAIN_AGE_CRITICAL=90
DOMAIN_AGE_HIGH=180
DOMAIN_AGE_MEDIUM=365
# API Rate Limiting
WHOXY_DAILY_LIMIT=8000
VIRUSTOTAL_DAILY_LIMIT=500
# Batch Processing
BATCH_CHUNK_SIZE=50
BATCH_TIMEOUT_SECONDS=300
# API Configuration (51000 = x1xxx = API/Gateway)
API_HOST=0.0.0.0
API_PORT=51000
API_VERSION=v1
# Celery Configuration
CELERY_BROKER_URL=redis://domain_check_redis:6379/1
CELERY_RESULT_BACKEND=redis://domain_check_redis:6379/2
# Monitoring (optional)
SENTRY_DSN=
DATADOG_API_KEY=
EOF
log_info ".env file created with secure passwords."
log_warn "Edit .env to add your API keys (WHOXY_API_KEY, etc.)"
}
update() {
log_info "Updating Domain Check API..."
cd "$INSTALL_DIR" || { log_error "Installation not found at $INSTALL_DIR"; exit 1; }
# Pull latest changes
log_info "Pulling latest changes..."
git pull origin "$BRANCH"
# Rebuild and restart
log_info "Rebuilding containers..."
docker compose down
docker compose up -d --build
# Wait and check
sleep 10
status
log_info "Update complete!"
}
status() {
log_info "Checking status..."
cd "$INSTALL_DIR" 2>/dev/null || { log_error "Installation not found"; exit 1; }
echo ""
docker compose ps
echo ""
# Health check
API_PORT=$(grep API_PORT .env | cut -d= -f2)
API_PORT=${API_PORT:-51000}
if curl -s "http://localhost:${API_PORT}/health" | grep -q "healthy"; then
log_info "API Health: ✓ Healthy"
curl -s "http://localhost:${API_PORT}/health" | python3 -m json.tool 2>/dev/null || true
else
log_error "API Health: ✗ Not responding"
fi
}
logs() {
cd "$INSTALL_DIR" 2>/dev/null || { log_error "Installation not found"; exit 1; }
if [ -n "$2" ]; then
docker compose logs -f "$2"
else
docker compose logs -f
fi
}
stop() {
log_info "Stopping services..."
cd "$INSTALL_DIR" 2>/dev/null || { log_error "Installation not found"; exit 1; }
docker compose down
log_info "Services stopped."
}
start() {
log_info "Starting services..."
cd "$INSTALL_DIR" 2>/dev/null || { log_error "Installation not found"; exit 1; }
docker compose up -d
sleep 5
status
}
restart() {
stop
start
}
# Main
case "${1:-}" in
install)
install
;;
update)
update
;;
status)
status
;;
logs)
logs "$@"
;;
stop)
stop
;;
start)
start
;;
restart)
restart
;;
*)
echo "Domain Check API - Deployment Script"
echo ""
echo "Usage: $0 {install|update|status|logs|start|stop|restart}"
echo ""
echo "Commands:"
echo " install - Fresh installation"
echo " update - Pull latest changes and rebuild"
echo " status - Check services status"
echo " logs - View logs (optional: service name)"
echo " start - Start services"
echo " stop - Stop services"
echo " restart - Restart services"
echo ""
echo "Example:"
echo " $0 install"
echo " $0 logs domain_check_api"
;;
esac

151
ai_platform/modules/domain_check/docker-compose.yml Normal file
View file

@ -0,0 +1,151 @@
# DIDI Domain Check (T4) — modul dedicat platformei DIDI.
# Izolare: Postgres + Redis proprii pe rețeaua internă `domain_check_network`.
# Integrare: API-ul se atașează și la `didi-network` (externă, partajată cu
# restul platformei) sub aliasul `domain-check-api`, ca agent-v3 (Lot 2) să-l
# apeleze la `http://domain-check-api:11000/api/v1/check/check` fără alt config.
networks:
domain_check_network:
driver: bridge
name: domain_check_network
didi-network:
external: true
volumes:
domain_check_postgres_data:
name: domain_check_postgres_data
domain_check_redis_data:
name: domain_check_redis_data
domain_check_api_logs:
name: domain_check_api_logs
services:
domain_check_postgres:
image: postgres:15-alpine
container_name: didiAI-domain-check-db
networks:
- domain_check_network
environment:
POSTGRES_DB: ${DB_NAME}
POSTGRES_USER: ${DB_USER}
POSTGRES_PASSWORD: ${DB_PASSWORD}
PGDATA: /var/lib/postgresql/data/pgdata
volumes:
- domain_check_postgres_data:/var/lib/postgresql/data
- ./init-scripts:/docker-entrypoint-initdb.d:ro
healthcheck:
test:
- CMD-SHELL
- pg_isready -U ${DB_USER} -d ${DB_NAME}
interval: 10s
timeout: 5s
retries: 5
start_period: 10s
restart: unless-stopped
domain_check_redis:
image: redis:7-alpine
container_name: didiAI-domain-check-redis
networks:
- domain_check_network
command: 'redis-server --appendonly yes --maxmemory 512mb --maxmemory-policy allkeys-lru
--save 900 1 --save 300 10 --save 60 10000
'
volumes:
- domain_check_redis_data:/data
healthcheck:
test:
- CMD
- redis-cli
- ping
interval: 10s
timeout: 3s
retries: 5
start_period: 5s
restart: unless-stopped
domain_check_api:
build:
context: ./api
dockerfile: Dockerfile
image: ${REGISTRY_IMAGE:-didiai-domain-check}:${IMAGE_TAG:-latest}
container_name: didiAI-domain-check
networks:
domain_check_network: {}
didi-network:
aliases:
- domain-check-api
dns:
- 8.8.8.8
- 8.8.4.4
- 1.1.1.1
environment:
- DATABASE_URL=postgresql://${DB_USER}:${DB_PASSWORD}@${DB_HOST}:${DB_PORT}/${DB_NAME}
- REDIS_URL=redis://${REDIS_HOST}:${REDIS_PORT}/${REDIS_DB}
- WHOXY_API_KEY=${WHOXY_API_KEY}
- VIRUSTOTAL_API_KEY=${VIRUSTOTAL_API_KEY}
- FLASK_ENV=${FLASK_ENV}
- FLASK_APP=${FLASK_APP}
- SECRET_KEY=${SECRET_KEY}
- LOG_LEVEL=${LOG_LEVEL}
- DEBUG=${DEBUG}
- REDIS_TTL_HOT=${REDIS_TTL_HOT}
- REDIS_TTL_WARM=${REDIS_TTL_WARM}
- REDIS_TTL_COLD=${REDIS_TTL_COLD}
- API_HOST=${API_HOST}
- API_PORT=${API_PORT}
- API_VERSION=${API_VERSION}
volumes:
- domain_check_api_logs:/app/logs
depends_on:
domain_check_postgres:
condition: service_healthy
domain_check_redis:
condition: service_healthy
healthcheck:
test:
- CMD
- curl
- -f
- http://localhost:${API_PORT}/health
interval: 30s
timeout: 10s
retries: 3
start_period: 40s
restart: unless-stopped
command: gunicorn --workers 2 --worker-class gthread --threads 4 --timeout 120
--graceful-timeout 30 --access-logfile - --error-logfile - --bind 0.0.0.0:${API_PORT}
run:app
domain_check_worker:
build:
context: ./api
dockerfile: Dockerfile
image: ${REGISTRY_IMAGE:-didiai-domain-check}:${IMAGE_TAG:-latest}
container_name: didiAI-domain-check-worker
networks:
- domain_check_network
dns:
- 8.8.8.8
- 8.8.4.4
- 1.1.1.1
command: celery -A app.celery_app worker --loglevel=info --concurrency=4
environment:
- DATABASE_URL=postgresql://${DB_USER}:${DB_PASSWORD}@${DB_HOST}:${DB_PORT}/${DB_NAME}
- REDIS_URL=redis://${REDIS_HOST}:${REDIS_PORT}/${REDIS_DB}
- CELERY_BROKER_URL=${CELERY_BROKER_URL}
- CELERY_RESULT_BACKEND=${CELERY_RESULT_BACKEND}
- WHOXY_API_KEY=${WHOXY_API_KEY}
- VIRUSTOTAL_API_KEY=${VIRUSTOTAL_API_KEY}
- LOG_LEVEL=${LOG_LEVEL}
depends_on:
- domain_check_api
- domain_check_redis
healthcheck:
test:
- CMD
- celery
- -A
- app.celery_app
- inspect
- ping
interval: 30s
timeout: 10s
retries: 3
start_period: 30s
restart: unless-stopped

413
ai_platform/modules/domain_check/init-scripts/01-init-db.sql Normal file
View file

@ -0,0 +1,413 @@
-- Domain Check Database Initialization Script
-- Version: 1.0.0
-- Date: 2026-01-29
-- Enable UUID extension
CREATE EXTENSION IF NOT EXISTS "uuid-ossp";
CREATE EXTENSION IF NOT EXISTS "pg_trgm"; -- For similarity searches
-- Set timezone
SET TIME ZONE 'UTC';
-- ============================================================================
-- TABLES
-- ============================================================================
-- 1. Domains table
CREATE TABLE IF NOT EXISTS domains (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
domain VARCHAR(255) NOT NULL,
subdomain VARCHAR(255),
tld VARCHAR(50) NOT NULL,
full_domain VARCHAR(255) UNIQUE NOT NULL, -- Complete domain (subdomain.domain.tld or domain.tld)
first_seen_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
last_checked_at TIMESTAMP WITH TIME ZONE,
check_count INTEGER DEFAULT 0,
is_active BOOLEAN DEFAULT TRUE,
created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
updated_at TIMESTAMP WITH TIME ZONE DEFAULT NOW()
);
CREATE INDEX idx_domains_full_domain ON domains(full_domain);
CREATE INDEX idx_domains_domain ON domains(domain);
CREATE INDEX idx_domains_last_checked ON domains(last_checked_at);
CREATE INDEX idx_domains_tld ON domains(tld);
CREATE INDEX idx_domains_is_active ON domains(is_active);
-- 2. WHOIS Records table
CREATE TABLE IF NOT EXISTS whois_records (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
domain_id UUID NOT NULL REFERENCES domains(id) ON DELETE CASCADE,
creation_date TIMESTAMP WITH TIME ZONE,
expiration_date TIMESTAMP WITH TIME ZONE,
updated_date TIMESTAMP WITH TIME ZONE,
registrar VARCHAR(255),
registrar_url VARCHAR(500),
registrant_org VARCHAR(255),
registrant_country VARCHAR(2),
admin_email VARCHAR(255),
name_servers TEXT[], -- Array of name servers
status TEXT[], -- Array of domain statuses
dnssec BOOLEAN,
raw_whois_data JSONB,
data_source VARCHAR(50) DEFAULT 'rdap', -- 'rdap', 'whoxy', 'manual'
fetched_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW()
);
CREATE INDEX idx_whois_domain_id ON whois_records(domain_id);
CREATE INDEX idx_whois_creation_date ON whois_records(creation_date);
CREATE INDEX idx_whois_registrar ON whois_records(registrar);
CREATE INDEX idx_whois_data_source ON whois_records(data_source);
CREATE INDEX idx_whois_fetched_at ON whois_records(fetched_at DESC);
-- 3. DNS Records table
CREATE TABLE IF NOT EXISTS dns_records (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
domain_id UUID NOT NULL REFERENCES domains(id) ON DELETE CASCADE,
record_type VARCHAR(10) NOT NULL, -- 'A', 'AAAA', 'MX', 'TXT', 'NS', 'CNAME', 'SOA'
record_value TEXT NOT NULL,
ttl INTEGER,
priority INTEGER, -- For MX records
fetched_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW()
);
CREATE INDEX idx_dns_domain_id ON dns_records(domain_id);
CREATE INDEX idx_dns_domain_type ON dns_records(domain_id, record_type);
CREATE INDEX idx_dns_record_type ON dns_records(record_type);
CREATE INDEX idx_dns_fetched_at ON dns_records(fetched_at DESC);
-- 4. SSL Certificates table
CREATE TABLE IF NOT EXISTS ssl_certificates (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
domain_id UUID NOT NULL REFERENCES domains(id) ON DELETE CASCADE,
issuer VARCHAR(255),
subject VARCHAR(255),
valid_from TIMESTAMP WITH TIME ZONE,
valid_until TIMESTAMP WITH TIME ZONE,
serial_number VARCHAR(255),
signature_algorithm VARCHAR(100),
key_size INTEGER,
is_wildcard BOOLEAN DEFAULT FALSE,
is_self_signed BOOLEAN DEFAULT FALSE,
is_valid BOOLEAN DEFAULT TRUE,
certificate_chain JSONB,
fetched_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW()
);
CREATE INDEX idx_ssl_domain_id ON ssl_certificates(domain_id);
CREATE INDEX idx_ssl_valid_until ON ssl_certificates(valid_until);
CREATE INDEX idx_ssl_issuer ON ssl_certificates(issuer);
CREATE INDEX idx_ssl_is_valid ON ssl_certificates(is_valid);
CREATE INDEX idx_ssl_fetched_at ON ssl_certificates(fetched_at DESC);
-- 5. Reputation Scores table
CREATE TABLE IF NOT EXISTS reputation_scores (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
domain_id UUID NOT NULL REFERENCES domains(id) ON DELETE CASCADE,
source VARCHAR(50) NOT NULL, -- 'virustotal', 'custom', 'opensquat', 'phishtank'
score INTEGER CHECK (score >= 0 AND score <= 100),
malicious_count INTEGER DEFAULT 0,
suspicious_count INTEGER DEFAULT 0,
harmless_count INTEGER DEFAULT 0,
undetected_count INTEGER DEFAULT 0,
is_blacklisted BOOLEAN DEFAULT FALSE,
blacklist_names TEXT[],
is_typosquatting BOOLEAN DEFAULT FALSE,
typosquatting_target VARCHAR(255),
raw_response JSONB,
checked_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW()
-- Note: UNIQUE constraint on domain_id + source + date handled at application level
);
CREATE INDEX idx_reputation_domain_id ON reputation_scores(domain_id);
CREATE INDEX idx_reputation_source ON reputation_scores(source);
CREATE INDEX idx_reputation_is_blacklisted ON reputation_scores(is_blacklisted);
CREATE INDEX idx_reputation_is_typosquatting ON reputation_scores(is_typosquatting);
CREATE INDEX idx_reputation_checked_at ON reputation_scores(checked_at DESC);
-- 6. Risk Assessments table
CREATE TABLE IF NOT EXISTS risk_assessments (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
domain_id UUID NOT NULL REFERENCES domains(id) ON DELETE CASCADE,
check_id UUID UNIQUE NOT NULL,
total_score INTEGER NOT NULL CHECK (total_score >= 0 AND total_score <= 100),
risk_level VARCHAR(20) NOT NULL CHECK (risk_level IN ('LOW', 'MEDIUM', 'HIGH', 'CRITICAL')),
-- Individual factor scores
domain_age_score INTEGER DEFAULT 0,
domain_age_days INTEGER,
ssl_score INTEGER DEFAULT 0,
dns_score INTEGER DEFAULT 0,
reputation_score INTEGER DEFAULT 0,
whois_score INTEGER DEFAULT 0,
-- Risk factors breakdown (JSONB array)
factors JSONB,
-- Flags
is_new_domain BOOLEAN DEFAULT FALSE,
is_suspicious BOOLEAN DEFAULT FALSE,
requires_manual_review BOOLEAN DEFAULT FALSE,
assessed_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW()
);
CREATE INDEX idx_risk_domain_id ON risk_assessments(domain_id);
CREATE INDEX idx_risk_check_id ON risk_assessments(check_id);
CREATE INDEX idx_risk_level ON risk_assessments(risk_level);
CREATE INDEX idx_risk_total_score ON risk_assessments(total_score);
CREATE INDEX idx_risk_is_new_domain ON risk_assessments(is_new_domain);
CREATE INDEX idx_risk_is_suspicious ON risk_assessments(is_suspicious);
CREATE INDEX idx_risk_assessed_at ON risk_assessments(assessed_at DESC);
-- 7. Check History table
CREATE TABLE IF NOT EXISTS check_history (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
check_id UUID UNIQUE NOT NULL,
domain_id UUID NOT NULL REFERENCES domains(id) ON DELETE CASCADE,
risk_assessment_id UUID REFERENCES risk_assessments(id) ON DELETE SET NULL,
-- Check metadata
requested_by VARCHAR(100) DEFAULT 'api', -- 'api', 'cli', 'dashboard', 'batch'
request_ip VARCHAR(45),
user_agent TEXT,
check_options JSONB,
-- Performance metrics
processing_time_ms INTEGER,
cache_hit BOOLEAN DEFAULT FALSE,
-- Changes detection
changes_detected BOOLEAN DEFAULT FALSE,
change_summary JSONB,
-- Status
status VARCHAR(20) DEFAULT 'completed', -- 'pending', 'processing', 'completed', 'failed'
error_message TEXT,
created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW()
);
CREATE INDEX idx_check_history_check_id ON check_history(check_id);
CREATE INDEX idx_check_history_domain_id ON check_history(domain_id);
CREATE INDEX idx_check_history_domain_created ON check_history(domain_id, created_at DESC);
CREATE INDEX idx_check_history_created_at ON check_history(created_at DESC);
CREATE INDEX idx_check_history_status ON check_history(status);
CREATE INDEX idx_check_history_requested_by ON check_history(requested_by);
-- 8. Batch Operations table
CREATE TABLE IF NOT EXISTS batch_operations (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
batch_id UUID UNIQUE NOT NULL,
total_domains INTEGER NOT NULL,
completed_count INTEGER DEFAULT 0,
failed_count INTEGER DEFAULT 0,
status VARCHAR(20) DEFAULT 'pending' CHECK (status IN ('pending', 'processing', 'completed', 'failed', 'cancelled')),
priority VARCHAR(20) DEFAULT 'normal' CHECK (priority IN ('low', 'normal', 'high', 'urgent')),
-- Timing
started_at TIMESTAMP WITH TIME ZONE,
completed_at TIMESTAMP WITH TIME ZONE,
estimated_completion_at TIMESTAMP WITH TIME ZONE,
-- Metadata
requested_by VARCHAR(100),
check_options JSONB,
domain_list TEXT[], -- Array of domains to check
-- Results
error_log JSONB,
summary JSONB,
created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW()
);
CREATE INDEX idx_batch_batch_id ON batch_operations(batch_id);
CREATE INDEX idx_batch_status ON batch_operations(status);
CREATE INDEX idx_batch_priority ON batch_operations(priority);
CREATE INDEX idx_batch_created_at ON batch_operations(created_at DESC);
-- 9. Blacklists table
CREATE TABLE IF NOT EXISTS blacklists (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
domain VARCHAR(255) UNIQUE NOT NULL,
reason TEXT,
category VARCHAR(50), -- 'phishing', 'malware', 'spam', 'fake_news', 'disinformation'
source VARCHAR(100),
severity VARCHAR(20) DEFAULT 'medium' CHECK (severity IN ('low', 'medium', 'high', 'critical')),
added_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
updated_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
is_active BOOLEAN DEFAULT TRUE,
expires_at TIMESTAMP WITH TIME ZONE,
-- Additional metadata
metadata JSONB
);
CREATE INDEX idx_blacklist_domain ON blacklists(domain);
CREATE INDEX idx_blacklist_category ON blacklists(category);
CREATE INDEX idx_blacklist_is_active ON blacklists(is_active);
CREATE INDEX idx_blacklist_severity ON blacklists(severity);
-- 10. API Usage Logs table
CREATE TABLE IF NOT EXISTS api_usage_logs (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
service_name VARCHAR(50) NOT NULL, -- 'whoxy', 'virustotal', 'rdap'
endpoint VARCHAR(255),
request_count INTEGER DEFAULT 1,
response_time_ms INTEGER,
status_code INTEGER,
quota_used INTEGER,
quota_remaining INTEGER,
error_message TEXT,
request_params JSONB,
logged_at TIMESTAMP WITH TIME ZONE DEFAULT NOW()
);
CREATE INDEX idx_api_usage_service ON api_usage_logs(service_name);
CREATE INDEX idx_api_usage_service_logged ON api_usage_logs(service_name, logged_at DESC);
CREATE INDEX idx_api_usage_logged_at ON api_usage_logs(logged_at DESC);
CREATE INDEX idx_api_usage_status ON api_usage_logs(status_code);
-- ============================================================================
-- VIEWS
-- ============================================================================
-- View for latest domain information
CREATE OR REPLACE VIEW v_latest_domain_info AS
SELECT
d.id,
d.full_domain,
d.domain,
d.subdomain,
d.tld,
d.first_seen_at,
d.last_checked_at,
d.check_count,
ra.total_score as latest_risk_score,
ra.risk_level as latest_risk_level,
ra.is_new_domain,
ra.is_suspicious,
w.creation_date as domain_creation_date,
w.registrar,
w.registrant_country,
EXTRACT(DAY FROM (NOW() - w.creation_date)) as domain_age_days
FROM domains d
LEFT JOIN LATERAL (
SELECT * FROM risk_assessments
WHERE domain_id = d.id
ORDER BY assessed_at DESC
LIMIT 1
) ra ON TRUE
LEFT JOIN LATERAL (
SELECT * FROM whois_records
WHERE domain_id = d.id
ORDER BY fetched_at DESC
LIMIT 1
) w ON TRUE;
-- View for high-risk domains
CREATE OR REPLACE VIEW v_high_risk_domains AS
SELECT
d.full_domain,
ra.total_score,
ra.risk_level,
ra.is_new_domain,
ra.domain_age_days,
ra.assessed_at,
rs.is_blacklisted,
rs.is_typosquatting
FROM domains d
JOIN risk_assessments ra ON d.id = ra.domain_id
LEFT JOIN reputation_scores rs ON d.id = rs.domain_id
WHERE ra.risk_level IN ('HIGH', 'CRITICAL')
ORDER BY ra.total_score DESC, ra.assessed_at DESC;
-- View for daily statistics
CREATE OR REPLACE VIEW v_daily_stats AS
SELECT
DATE(created_at) as check_date,
COUNT(DISTINCT domain_id) as unique_domains_checked,
COUNT(*) as total_checks,
AVG(processing_time_ms) as avg_processing_time_ms,
SUM(CASE WHEN cache_hit THEN 1 ELSE 0 END) as cache_hits,
SUM(CASE WHEN NOT cache_hit THEN 1 ELSE 0 END) as cache_misses,
ROUND(100.0 * SUM(CASE WHEN cache_hit THEN 1 ELSE 0 END) / COUNT(*), 2) as cache_hit_rate
FROM check_history
WHERE status = 'completed'
GROUP BY DATE(created_at)
ORDER BY check_date DESC;
-- ============================================================================
-- FUNCTIONS
-- ============================================================================
-- Function to update updated_at timestamp
CREATE OR REPLACE FUNCTION update_updated_at_column()
RETURNS TRIGGER AS $$
BEGIN
NEW.updated_at = NOW();
RETURN NEW;
END;
$$ LANGUAGE plpgsql;
-- Triggers for updated_at
CREATE TRIGGER update_domains_updated_at BEFORE UPDATE ON domains
FOR EACH ROW EXECUTE FUNCTION update_updated_at_column();
CREATE TRIGGER update_blacklists_updated_at BEFORE UPDATE ON blacklists
FOR EACH ROW EXECUTE FUNCTION update_updated_at_column();
-- Function to increment domain check count
CREATE OR REPLACE FUNCTION increment_domain_check_count()
RETURNS TRIGGER AS $$
BEGIN
UPDATE domains
SET check_count = check_count + 1,
last_checked_at = NOW()
WHERE id = NEW.domain_id;
RETURN NEW;
END;
$$ LANGUAGE plpgsql;
-- Trigger to auto-increment check count
CREATE TRIGGER increment_check_count AFTER INSERT ON check_history
FOR EACH ROW EXECUTE FUNCTION increment_domain_check_count();
-- ============================================================================
-- INITIAL DATA
-- ============================================================================
-- Insert some common blacklist entries (example)
INSERT INTO blacklists (domain, reason, category, source, severity) VALUES
('known-phishing-site.com', 'Known phishing operation', 'phishing', 'manual', 'critical'),
('fake-news-urgent.com', 'Disinformation campaign', 'fake_news', 'manual', 'high'),
('malware-distribution.com', 'Malware distribution', 'malware', 'manual', 'critical')
ON CONFLICT (domain) DO NOTHING;
-- ============================================================================
-- GRANTS
-- ============================================================================
-- Grant permissions (adjust as needed)
GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA public TO dns_admin;
GRANT ALL PRIVILEGES ON ALL SEQUENCES IN SCHEMA public TO dns_admin;
GRANT USAGE, SELECT ON ALL SEQUENCES IN SCHEMA public TO dns_admin;
-- ============================================================================
-- COMPLETION
-- ============================================================================
-- Log initialization
DO $$
BEGIN
RAISE NOTICE 'Database initialization completed successfully!';
RAISE NOTICE 'Total tables created: 10';
RAISE NOTICE 'Total views created: 3';
RAISE NOTICE 'Total functions created: 2';
END $$;

41
ai_platform/modules/domain_check/test-api-complete.sh Executable file
View file

@ -0,0 +1,41 @@
#!/bin/bash
echo "=========================================="
echo " DOMAIN-CHECK API - Complete Test Suite"
echo "=========================================="
echo ""
API="http://localhost:5000"
# Test 1: Health
echo "✓ Test 1: Health Check"
curl -s $API/health | python3 -m json.tool | head -10
echo ""
# Test 2: Check with WHOIS
echo "✓ Test 2: WHOIS Check - github.com"
curl -s -X POST $API/api/v1/check \
-H "Content-Type: application/json" \
-d '{"domain": "github.com", "check_options": {"whois": true, "dns": false, "ssl": false}}' \
| python3 -m json.tool | head -40
echo ""
# Test 3: Check with DNS
echo "✓ Test 3: DNS Check - google.com"
curl -s -X POST $API/api/v1/check \
-H "Content-Type: application/json" \
-d '{"domain": "google.com", "check_options": {"whois": false, "dns": true, "ssl": false}}' \
| python3 -m json.tool | grep -A 20 '"dns"'
echo ""
# Test 4: Full check
echo "✓ Test 4: Full Check (WHOIS + DNS) - facebook.com"
curl -s -X POST $API/api/v1/check \
-H "Content-Type: application/json" \
-d '{"domain": "facebook.com", "check_options": {"whois": true, "dns": true, "ssl": false}}' \
| python3 -m json.tool | head -50
echo ""
echo "=========================================="
echo " Tests Complete!"
echo "=========================================="

114
ai_platform/modules/domain_check/test-complete-lan.sh Executable file
View file

@ -0,0 +1,114 @@
#!/bin/bash
LAN_IP="10.11.10.200"
API="http://${LAN_IP}:5000"
echo "=========================================="
echo " DOMAIN-CHECK - Complete Test Suite"
echo " API: $API"
echo "=========================================="
echo ""
# Test 1: Health Check
echo "✓ Test 1: Health Check"
curl -s "${API}/health" | python3 -m json.tool
if [ $? -eq 0 ]; then
echo "✅ Health check PASSED"
else
echo "❌ Health check FAILED"
exit 1
fi
echo ""
# Test 2: API Root
echo "✓ Test 2: API Root"
curl -s "${API}/" | python3 -m json.tool
echo ""
# Test 3: WHOIS Only
echo "✓ Test 3: WHOIS Check - github.com"
curl -s -X POST "${API}/api/v1/check" \
-H "Content-Type: application/json" \
-d '{"domain": "github.com", "check_options": {"whois": true, "dns": false, "ssl": false}}' \
| python3 -m json.tool > /tmp/test3.json
if grep -q "\"success\": true" /tmp/test3.json; then
echo "✅ WHOIS check PASSED"
cat /tmp/test3.json | jq '.data.whois' | head -15
else
echo "❌ WHOIS check FAILED"
cat /tmp/test3.json
fi
echo ""
# Test 4: DNS Only
echo "✓ Test 4: DNS Check - google.com"
curl -s -X POST "${API}/api/v1/check" \
-H "Content-Type: application/json" \
-d '{"domain": "google.com", "check_options": {"whois": false, "dns": true, "ssl": false}}' \
| python3 -m json.tool > /tmp/test4.json
if grep -q "\"success\": true" /tmp/test4.json; then
echo "✅ DNS check PASSED"
cat /tmp/test4.json | jq '.data.dns' | head -20
else
echo "❌ DNS check FAILED"
cat /tmp/test4.json
fi
echo ""
# Test 5: Full Check (WHOIS + DNS)
echo "✓ Test 5: Full Check - facebook.com"
curl -s -X POST "${API}/api/v1/check" \
-H "Content-Type: application/json" \
-d '{"domain": "facebook.com", "check_options": {"whois": true, "dns": true, "ssl": false}}' \
| python3 -m json.tool > /tmp/test5.json
if grep -q "\"success\": true" /tmp/test5.json; then
echo "✅ Full check PASSED"
cat /tmp/test5.json | jq '.data.risk_score' | head -15
else
echo "❌ Full check FAILED"
cat /tmp/test5.json
fi
echo ""
# Test 6: Risk Scoring
echo "✓ Test 6: Risk Score Analysis"
cat /tmp/test5.json | jq '{
domain: .data.domain,
risk_level: .data.risk_score.level,
total_score: .data.risk_score.total,
factors: .data.risk_score.factors | map({factor: .factor, score: .score})
}'
echo ""
# Test 7: Database Check
echo "✓ Test 7: Database Verification"
docker exec dns_postgres psql -U dns_admin -d domain_check -c \
"SELECT d.full_domain, ra.total_score, ra.risk_level, ra.assessed_at
FROM risk_assessments ra
JOIN domains d ON ra.domain_id = d.id
ORDER BY ra.assessed_at DESC LIMIT 5;"
echo ""
# Test 8: Swagger UI
echo "✓ Test 8: Swagger UI"
echo " URL: http://${LAN_IP}:5000/docs"
echo ""
# Test 9: ReDoc
echo "✓ Test 9: ReDoc Documentation"
echo " URL: http://${LAN_IP}:5000/redoc"
echo ""
echo "=========================================="
echo " ✅ All Tests Complete!"
echo "=========================================="
echo ""
echo "📊 Access Points:"
echo " API: http://${LAN_IP}:5000"
echo " Docs: http://${LAN_IP}:5000/docs"
echo " ReDoc: http://${LAN_IP}:5000/redoc"
echo " Health: http://${LAN_IP}:5000/health"
echo ""

578
ai_platform/modules/domain_check/use_api.md Normal file
View file

@ -0,0 +1,578 @@
# Domain Check API - Developer Documentation
**Base URL:** `http://domain-check-api:11000`
**API Version:** v1
---
## Authentication
Currently no authentication required (internal use only).
---
## Endpoints
### POST /api/v1/check/check
Perform comprehensive domain verification.
**Request:**
```bash
curl -X POST "http://domain-check-api:11000/api/v1/check/check" \
-H "Content-Type: application/json" \
-d '{
"domain": "example.com",
"check_options": {
"whois": true,
"dns": true,
"ssl": true,
"ip_intelligence": true,
"http_analysis": true,
"blacklist": true,
"port_scan": true,
"subdomains": true
}
}'
```
**Check Options:**
| Option | Type | Default | Description |
|--------|------|---------|-------------|
| `whois` | bool | true | WHOIS lookup (registrar, dates, nameservers) |
| `dns` | bool | true | DNS records (A, AAAA, MX, TXT, NS, CNAME, SOA) |
| `ssl` | bool | true | SSL certificate validation |
| `ip_intelligence` | bool | true | IP geolocation, ASN, reverse DNS |
| `http_analysis` | bool | true | HTTP headers, security, technologies |
| `blacklist` | bool | true | DNSBL spam/malware blacklist check |
| `port_scan` | bool | false | TCP port scanning (slower) |
| `subdomains` | bool | false | Subdomain enumeration (slower) |
| `force_refresh` | bool | false | Bypass cache |
---
## Response Structure
```json
{
"success": true,
"data": {
"domain": "example.com",
"check_id": "uuid-v4",
"timestamp": "2026-02-04T09:32:18.459703Z",
"whois": { ... },
"dns": { ... },
"ssl": { ... },
"ip_intelligence": { ... },
"http_analysis": { ... },
"blacklist": { ... },
"port_scan": { ... },
"subdomains": { ... },
"risk_score": { ... }
},
"metadata": {
"cached": false,
"processing_time_ms": 40315,
"api_version": "v1",
"checks_performed": ["whois", "dns", "ssl", ...]
}
}
```
---
## Data Sections
### 1. WHOIS (`data.whois`)
```json
{
"creation_date": "2017-10-13T07:59:48Z",
"expiration_date": "2026-10-13T07:59:48Z",
"updated_date": "2025-01-15T06:24:03Z",
"registrar": "Vautron Rechenzentrum AG",
"age_days": 3036,
"status": "ok https://icann.org/epp#ok",
"name_servers": ["NS1.CONTABO.NET", "NS2.CONTABO.NET"],
"dnssec": null,
"registrant_org": null,
"registrant_country": "RO",
"data_source": "whois"
}
```
### 2. DNS (`data.dns`)
```json
{
"a_records": ["84.54.23.135"],
"aaaa_records": [],
"mx_records": [{"priority": 10, "host": "mail.example.com"}],
"txt_records": ["v=spf1 include:_spf.google.com ~all"],
"ns_records": ["ns1.example.com", "ns2.example.com"],
"cname_records": [],
"soa_record": {
"mname": "ns1.example.com",
"rname": "hostmaster.example.com",
"serial": 1770195563,
"refresh": 10800,
"retry": 3600,
"expire": 604800,
"minimum": 3600
},
"has_spf": true,
"has_dkim": false,
"has_dmarc": true,
"spf_record": null,
"dmarc_record": null
}
```
### 3. SSL (`data.ssl`)
```json
{
"has_ssl": true,
"is_valid": true,
"is_self_signed": false,
"is_expired": false,
"is_wildcard": false,
"issuer": "CN=R13, O=Let's Encrypt, C=US",
"subject": "CN=example.com",
"valid_from": "2025-12-22T12:48:47Z",
"valid_until": "2026-03-22T12:48:46Z",
"days_until_expiry": 46,
"key_size": 2048,
"signature_algorithm": "sha256WithRSAEncryption",
"san": ["example.com", "www.example.com", "mail.example.com"]
}
```
### 4. IP Intelligence (`data.ip_intelligence`)
```json
{
"ip": "84.54.23.135",
"reverse_dns": "server.provider.net",
"geolocation": null,
"asn": "AS51167",
"isp": "Contabo GmbH",
"organization": "Contabo GmbH",
"is_datacenter": true,
"is_residential": false,
"hostname": "server.provider.net",
"city": "Lauterbourg",
"region": "Grand Est",
"country": "France",
"country_code": "FR",
"coordinates": {"latitude": 48.9751, "longitude": 8.1785},
"timezone": "Europe/Paris",
"postal": "67630",
"data_source": "ipinfo.io",
"hosting_score": {
"score": 50,
"reasons": [],
"is_trusted": false,
"is_suspicious": false
}
}
```
### 5. HTTP Analysis (`data.http_analysis`)
```json
{
"domain": "example.com",
"http_status": 200,
"https_status": 200,
"has_https": true,
"http_to_https_redirect": true,
"final_url": "https://example.com/",
"redirect_chain": [],
"response_time_ms": 3351,
"server": "Apache",
"powered_by": null,
"security_headers": {},
"missing_security_headers": [
{
"header": "Strict-Transport-Security",
"description": "HSTS - Forces HTTPS",
"severity": "CRITICAL"
},
{
"header": "X-Frame-Options",
"description": "Prevents clickjacking",
"severity": "HIGH"
}
],
"cookies": [],
"technologies": ["WordPress", "jQuery", "Apache"],
"cms": "WordPress",
"frameworks": ["jQuery"],
"has_robots_txt": true,
"has_sitemap": true,
"has_favicon": true,
"error": null
}
```
### 6. Blacklist (`data.blacklist`)
```json
{
"domain": "example.com",
"ip": "84.54.23.135",
"is_blacklisted": false,
"ip_blacklisted": false,
"domain_blacklisted": false,
"total_listings": 0,
"ip_check": {
"ip": "84.54.23.135",
"is_blacklisted": false,
"blacklist_count": 0,
"clean_count": 4,
"total_checked": 4,
"listings": [],
"clean_lists": ["Spamhaus ZEN", "SpamCop", "Barracuda", "SORBS"],
"check_errors": []
},
"domain_check": {
"domain": "example.com",
"is_blacklisted": false,
"blacklist_count": 0,
"clean_count": 4,
"total_checked": 4,
"listings": [],
"clean_lists": ["Spamhaus DBL", "URIBL", "URIBL Black", "SURBL"],
"check_errors": []
},
"reputation_score": 100,
"risk_level": "LOW"
}
```
**Blacklists Checked:**
| List | Type | Description |
|------|------|-------------|
| Spamhaus ZEN | IP | Combined spam blocklist |
| SpamCop | IP | User-reported spam sources |
| Barracuda | IP | Barracuda reputation |
| SORBS | IP | Spam and relay blocking |
| Spamhaus DBL | Domain | Domain blocklist |
| URIBL | Domain | URI blocklist |
| URIBL Black | Domain | High-confidence spam URIs |
| SURBL | Domain | Spam URI realtime blocklist |
### 7. Port Scan (`data.port_scan`)
```json
{
"ip": "84.54.23.135",
"total_scanned": 9,
"open_ports": [
{
"port": 443,
"service": "HTTPS",
"category": "web",
"risk_level": "low",
"banner": null
},
{
"port": 22,
"service": "SSH",
"category": "remote_access",
"risk_level": "low",
"banner": "SSH-2.0-OpenSSH_8.7"
}
],
"closed_ports": [8080, 23, 21, 3389],
"filtered_ports": [],
"dangerous_open": [
{
"port": 3306,
"service": "MySQL",
"category": "database",
"risk_level": "critical",
"banner": "Host not allowed to connect"
}
],
"services_detected": ["HTTPS", "HTTP", "SSH", "MySQL"],
"categories": {
"web": [443, 80],
"remote_access": [22],
"database": [3306]
},
"security_issues": [
{
"severity": "CRITICAL",
"port": 3306,
"service": "MySQL",
"issue": "Dangerous service MySQL exposed on port 3306"
}
],
"scan_summary": {
"open_count": 5,
"closed_count": 4,
"filtered_count": 0,
"dangerous_count": 1,
"has_web": true,
"has_email": true,
"has_database": true,
"has_remote_access": true
}
}
```
**Ports Scanned (Quick Scan):**
| Port | Service | Risk Level |
|------|---------|------------|
| 21 | FTP | Medium |
| 22 | SSH | Low |
| 23 | Telnet | Critical |
| 25 | SMTP | Low |
| 80 | HTTP | Low |
| 443 | HTTPS | Low |
| 3306 | MySQL | Critical |
| 3389 | RDP | High |
| 8080 | HTTP Proxy | Medium |
### 8. Subdomains (`data.subdomains`)
```json
{
"domain": "example.com",
"subdomains": [
"admin.example.com",
"api.example.com",
"mail.example.com",
"www.example.com"
],
"total_found": 93,
"sources": {
"certificate_transparency": ["mail.example.com", "www.example.com"],
"dns_bruteforce": ["admin.example.com", "api.example.com"],
"dns_records": ["mail.example.com"]
},
"live_subdomains": [
{
"subdomain": "www.example.com",
"ip": "84.54.23.135",
"http": 301,
"https": 200
}
],
"error": null
}
```
**Subdomain Sources:**
1. **Certificate Transparency** - crt.sh logs
2. **DNS Bruteforce** - Common subdomain prefixes
3. **DNS Records** - MX, NS, SOA records
### 9. Risk Score (`data.risk_score`)
```json
{
"total": 13,
"level": "LOW",
"factors": [
{
"factor": "domain_age",
"score": 0,
"weight": 0.2,
"weighted_score": 0,
"reason": "TRUSTED: Mature domain (3036 days, 8+ years)",
"details": {"creation_date": "2017-10-13", "age_days": 3036}
}
],
"formula_breakdown": [
{
"category": "Domain Age",
"raw_score": 0,
"weight": 0.2,
"weighted_score": 0,
"formula": "0 × 0.2 = 0.0"
}
],
"formula_string": "Total = (0 × 0.2) + (0 × 0.15) + ... = 13",
"thresholds": {
"low": "0-25",
"medium": "26-50",
"high": "51-75",
"critical": "76-100"
},
"is_new_domain": false,
"is_suspicious": false,
"is_blacklisted": false,
"requires_manual_review": false
}
```
**Risk Score Formula:**
| Category | Weight | Description |
|----------|--------|-------------|
| Domain Age | 20% | New domains = higher risk |
| SSL/TLS | 15% | Certificate validity |
| DNS Configuration | 10% | Complete DNS setup |
| Email Security | 10% | SPF, DKIM, DMARC |
| WHOIS Privacy | 5% | Registration info |
| IP Reputation | 10% | Datacenter/residential |
| HTTP Security | 10% | Security headers |
| Blacklist Status | 15% | DNSBL listings |
| Port Security | 5% | Exposed dangerous ports |
**Risk Levels:**
| Level | Score Range | Color |
|-------|-------------|-------|
| LOW | 0-25 | Green |
| MEDIUM | 26-50 | Yellow |
| HIGH | 51-75 | Orange |
| CRITICAL | 76-100 | Red |
---
## Examples
### Quick Check (Basic)
```bash
curl -X POST "http://domain-check-api:11000/api/v1/check/check" \
-H "Content-Type: application/json" \
-d '{"domain": "google.com"}'
```
### Full Check (All Options)
```bash
curl -X POST "http://domain-check-api:11000/api/v1/check/check" \
-H "Content-Type: application/json" \
-d '{
"domain": "example.com",
"check_options": {
"whois": true,
"dns": true,
"ssl": true,
"ip_intelligence": true,
"http_analysis": true,
"blacklist": true,
"port_scan": true,
"subdomains": true
}
}'
```
### Python Example
```python
import requests
response = requests.post(
"http://domain-check-api:11000/api/v1/check/check",
json={
"domain": "example.com",
"check_options": {
"whois": True,
"dns": True,
"ssl": True,
"ip_intelligence": True,
"http_analysis": True,
"blacklist": True,
"port_scan": True,
"subdomains": True
}
}
)
data = response.json()
print(f"Risk Score: {data['data']['risk_score']['total']}")
print(f"Risk Level: {data['data']['risk_score']['level']}")
print(f"Is Blacklisted: {data['data']['blacklist']['is_blacklisted']}")
```
### JavaScript Example
```javascript
const response = await fetch('http://domain-check-api:11000/api/v1/check/check', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({
domain: 'example.com',
check_options: {
whois: true,
dns: true,
ssl: true,
ip_intelligence: true,
http_analysis: true,
blacklist: true,
port_scan: true,
subdomains: true
}
})
});
const data = await response.json();
console.log(`Risk Score: ${data.data.risk_score.total}`);
console.log(`Risk Level: ${data.data.risk_score.level}`);
```
---
## Error Responses
```json
{
"success": false,
"error": {
"code": "INVALID_REQUEST",
"message": "Domain parameter is required",
"status": 400
}
}
```
**Error Codes:**
| Code | Status | Description |
|------|--------|-------------|
| `INVALID_REQUEST` | 400 | Missing or invalid parameters |
| `INVALID_DOMAIN` | 400 | Domain format invalid |
| `NOT_FOUND` | 404 | Resource not found |
| `INTERNAL_ERROR` | 500 | Server error |
---
## Rate Limits
No rate limits currently (internal use).
---
## Dashboard
Web interface available at: `http://domain-check-api:11000/`
---
## Network Info
- **Host:** domain-check-api
- **IP:** 10.11.10.200
- **Port:** 11000 (x1xxx = API/Gateway per Port Schema v2)
- **Protocol:** HTTP (internal network)
## Port Allocation (Production Environment - 1xxxx)
| Service | Port | Category |
|---------|------|----------|
| API | 11000 | x1xxx = API/Gateway |
| PostgreSQL | 12000 | x20xx = Databases/PostgreSQL |
| Redis | 12300 | x23xx = Cache/Redis |

39
ai_platform/modules/embeddings/INDEX.md
View file

@ -20,7 +20,7 @@ Generate text embeddings for downstream DIDI consumers:
- **didi-brain semantic atom storage** — text claims/atoms get embedded once at ingestion, stored as pgvector columns.
- **didi-brain `/v1/gather` query embedding** — incoming claim text gets embedded, used as the kNN probe vector.
- **Cross-encoder reranking** — separate `rerank` module (port 54200) handles re-scoring; this module only does dense embeddings.
- **Cross-encoder reranking** — separate `rerank` module (port 14200) handles re-scoring; this module only does dense embeddings.
- **Catalog / retrieval consumers** — anything that needs a vector representation hits this single endpoint.
The module is OpenAI-compatible: drop-in for `openai` SDK code that uses `client.embeddings.create()`.
@ -68,7 +68,6 @@ embeddings/
├── README.md quick-start + config table
├── API.md full HTTP API reference + SDK examples
├── pyproject.toml package def, optional extras: vllm, llamacpp, all, dev
├── uv.lock
├── .env.example all EMB_* env vars documented
├── deploy/
│ ├── Dockerfile multi-stage, python 3.11-slim + uv
@ -111,7 +110,7 @@ embeddings/
- **Ingestion path** — when atoms/claims are written, brain calls `EmbeddingClient.embed(text)` to get a 1024-dim vector and stores it in pgvector alongside the row. One round-trip per batch.
- **Query path** — for `/v1/gather`, the incoming claim text is embedded the same way, then the resulting vector is used as the probe in a pgvector `<=> ` (cosine) kNN search to retrieve candidate atoms.
- **Reranking** — top-k candidates from the dense search are forwarded to the separate `rerank` module (cross-encoder, port 54200) for fine-grained scoring. That module does not call this one — they're parallel concerns.
- **Reranking** — top-k candidates from the dense search are forwarded to the separate `rerank` module (cross-encoder, port 14200) for fine-grained scoring. That module does not call this one — they're parallel concerns.
- **Catalog API** — also retrieves via the same embedding pipeline (text → vector → kNN), reusing this single endpoint.
Because the wrapper is OpenAI-compatible, `EmbeddingClient` can be a vanilla `openai.OpenAI(base_url=..., api_key=...)` instance — no DIDI-specific client code needed in brain.
@ -138,16 +137,9 @@ All env vars use the `EMB_` prefix. Required vars have no defaults — the app r
| `EMB_PORT` | `14100` (Prod) / `54100` (Dev) | API server port |
| `EMB_HOST` | `0.0.0.0` | API bind address |
| `EMB_API_TOKENS` | unset | Comma-separated bearer tokens; auth disabled if unset |
| `EMB_VLLM_BASE_URL` | `http://localhost:54101` | Where vLLM listens |
| `EMB_VLLM_MODEL` | `BAAI/bge-m3` | HF model id loaded by vLLM |
| `EMB_VLLM_GPU` | `0` | `CUDA_VISIBLE_DEVICES` for vLLM |
| `EMB_VLLM_GPU_UTIL` | `0.50` | vLLM `--gpu-memory-utilization` |
| `EMB_VLLM_MAX_LEN` | `8192` | vLLM `--max-model-len` |
| `EMB_LLAMACPP_BASE_URL` | `http://localhost:54110` | Where llama.cpp listens |
| `EMB_LLAMACPP_MODEL` | `bge-m3-q4_k_m.gguf` | GGUF filename inside `MODELS_DIR` |
| `EMB_LLAMACPP_CTX` | `8192` | llama.cpp context size |
| `EMB_LLAMACPP_THREADS` | `4` | llama.cpp threads |
| `EMB_LLAMACPP_PARALLEL` | `4` | llama.cpp parallel slots |
| `EMB_VLLM_BASE_URL` | `http://localhost:54101` | Where the API wrapper reaches vLLM |
| `EMB_VLLM_API_KEY` | unset | Optional API key for the vLLM server |
| `EMB_LLAMACPP_BASE_URL` | `http://localhost:54110` | Where the API wrapper reaches llama.cpp |
| `EMB_REQUEST_TIMEOUT` | `120.0` | Per-request backend timeout (s) |
| `EMB_CONNECT_TIMEOUT` | `10.0` | TCP connect timeout (s) |
| `EMB_RATE_LIMIT_RPS` | `20.0` | Token-bucket rate (req/s) |
@ -155,7 +147,24 @@ All env vars use the `EMB_` prefix. Required vars have no defaults — the app r
| `EMB_MAX_CONCURRENT_REQUESTS` | `20` | In-flight cap |
| `EMB_LOG_LEVEL` | `INFO` | DEBUG / INFO / WARNING / ERROR |
| `EMB_LOG_JSON` | `false` | Emit JSON-formatted log lines |
| `HF_CACHE_DIR` | `/cai2_ds_storage/hf_cache` | Mounted into vLLM container |
> The API app reads only the variables above (fields on `EmbeddingSettings` in `config.py`). `EMB_VLLM_BASE_URL` / `EMB_LLAMACPP_BASE_URL` just tell the wrapper where to reach the backend servers.
### Backend service variables (docker-compose only)
These are **not** fields on `EmbeddingSettings` — they are consumed by `deploy/docker-compose.yml` to launch the vLLM / llama.cpp containers (model id, GPU pinning, memory, context). They configure the backend server, not the API app.
| Variable | Default | Description |
|----------|---------|-------------|
| `EMB_VLLM_MODEL` | `BAAI/bge-m3` | HF model id loaded by the vLLM container |
| `EMB_VLLM_GPU` | `0` | `CUDA_VISIBLE_DEVICES` for the vLLM container |
| `EMB_VLLM_GPU_UTIL` | `0.50` | vLLM `--gpu-memory-utilization` |
| `EMB_VLLM_MAX_LEN` | `8192` | vLLM `--max-model-len` |
| `EMB_LLAMACPP_MODEL` | `bge-m3-q4_k_m.gguf` | GGUF filename inside `MODELS_DIR` |
| `EMB_LLAMACPP_CTX` | `8192` | llama.cpp context size |
| `EMB_LLAMACPP_THREADS` | `4` | llama.cpp threads |
| `EMB_LLAMACPP_PARALLEL` | `4` | llama.cpp parallel slots |
| `HF_CACHE_DIR` | `/cai2_ds_storage/hf_cache` | Mounted into the vLLM container |
| `HF_TOKEN` | unset | Forwarded as `HUGGING_FACE_HUB_TOKEN` |
| `MODELS_DIR` | `/cai2_ds_storage/models` | GGUF model directory for llama.cpp |
@ -208,7 +217,7 @@ Compose profiles:
## Related
- **`didi-brain`** consumes via `shared/embedding_client.py` for both ingestion (text → pgvector storage) and `/v1/gather` query embedding.
- **`rerank` module** (separate, port 54200) handles cross-encoder scoring on top of dense kNN candidates from this module — they're complementary, not chained inside this service.
- **`rerank` module** (separate, port 14200) handles cross-encoder scoring on top of dense kNN candidates from this module — they're complementary, not chained inside this service.
- **Catalog API** uses the same endpoint for retrieval-side embeddings.
- **pgvector** in the brain Postgres stores the resulting 1024-dim vectors (cosine distance index).
- **Local Python use**: `from embeddings import EmbeddingClient` (works without the HTTP wrapper if you want in-process inference and have the `vllm`/`llamacpp` extras installed).

10
ai_platform/modules/embeddings/deploy/docker-compose.yml
View file

@ -14,10 +14,10 @@
# Naming Convention: didiAI-{module}-{service}
#
# Network:
# Uses deploy_default network (shared with other modules)
# Uses didi-network (shared with all DIDI + AI platform stacks)
networks:
deploy_default:
didi-network:
external: true
services:
@ -33,7 +33,7 @@ services:
ports:
- "${EMB_PORT:-14100}:${EMB_PORT:-14100}"
networks:
- deploy_default
- didi-network
environment:
- EMB_PORT=${EMB_PORT:-14100}
- EMB_EXTERNAL_URL=${EMB_EXTERNAL_URL}
@ -66,7 +66,7 @@ services:
ports:
- "${EMB_VLLM_PORT:-14101}:14101"
networks:
- deploy_default
- didi-network
volumes:
- ${HF_CACHE_DIR:-/cai2_ds_storage/hf_cache}:/root/.cache/huggingface
environment:
@ -109,7 +109,7 @@ services:
ports:
- "${EMB_LLAMACPP_PORT:-14110}:8080"
networks:
- deploy_default
- didi-network
volumes:
- ${MODELS_DIR:-/cai2_ds_storage/models}:/models:ro
command: >

94
ai_platform/modules/extractors/API.md
View file

@ -114,6 +114,98 @@ Each extractor returns a uniform `FeatureResult`:
| `confidence` | number\|null | 0..1 (extraction confidence, not a verdict) |
| `error` | string\|null | Set when `ok` is false |
### POST /v1/sentiment
Classify the sentiment of a text via the LLM gateway (Romanian-aware prompt,
detects irony/sarcasm). **Delegated to the LLM gateway** — returns `503` if no
gateway is configured.
**Request** — `application/json`
| Field | Type | Description |
|-------|------|-------------|
| `text` | string | Text to analyze (min length 1) |
```bash
curl -X POST http://localhost:54400/v1/sentiment \
-H "Content-Type: application/json" \
-d '{"text": "Ce zi frumoasă!"}'
```
**Response** `200 OK` — a `FeatureResult` (same schema as above), e.g.:
```json
{
"tool_id": "sentiment",
"name": "Sentiment",
"ok": true,
"results": { "label": "positive", "score": 0.92 },
"evidence": ["Sentiment: positive (0.92)."],
"anomalies": [],
"confidence": 0.92
}
```
### POST /v1/ner
Extract named entities via GLiNER multilingual (`urchade/gliner_multi-v2.1`).
**Requires the optional `ml` extra** — returns `503` if the model is unavailable.
**Request** — `application/json`
| Field | Type | Description |
|-------|------|-------------|
| `text` | string | Text to analyze (min length 1) |
| `labels` | string[] \| null | Entity types to extract (defaults to the RO set) |
```bash
curl -X POST http://localhost:54400/v1/ner \
-H "Content-Type: application/json" \
-d '{"text": "Klaus Iohannis și Guvernul României."}'
```
**Response** `200 OK` — a `FeatureResult`; `results` carries the extracted
entities (text, label, span, score).
### POST /v1/ocr
Extract visible text verbatim from an image via the LLM vision model.
**Delegated to the LLM gateway** — returns `503` if no gateway is configured.
**Request** — `multipart/form-data`
| Field | Type | Description |
|-------|------|-------------|
| `file` | file | Image file |
```bash
curl -X POST http://localhost:54400/v1/ocr \
-F "file=@scan.png"
```
**Response** `200 OK` — a `FeatureResult`; `results` carries the recognized text.
### POST /v1/detect
Detect objects (COCO classes) via YOLO (`yolov8n`), returning boxes + confidence.
**Requires the optional `ml` extra** — returns `503` if the model is unavailable.
**Request** — `multipart/form-data`
| Field | Type | Description |
|-------|------|-------------|
| `file` | file | Image file |
Optional query param: `threshold` (float, 0..1) — confidence threshold.
```bash
curl -X POST "http://localhost:54400/v1/detect?threshold=0.4" \
-F "file=@street.jpg"
```
**Response** `200 OK` — a `FeatureResult`; `results` carries detected objects
(label, box, confidence).
## Error Responses
| Status | Meaning |
@ -121,6 +213,8 @@ Each extractor returns a uniform `FeatureResult`:
| 400 | Empty file |
| 413 | File exceeds `EXTRACTORS_MAX_UPLOAD_MB` |
| 422 | Missing `file` field |
| 502 | `LLMError` — the LLM gateway returned an error (`/v1/sentiment`, `/v1/ocr`) |
| 503 | `LLMNotConfigured` (no gateway for `/v1/sentiment`, `/v1/ocr`) or the `ml` extra/model is unavailable (`/v1/ner`, `/v1/detect`) |
```json
{ "detail": "empty file" }

4
ai_platform/modules/forensic_features/.env.example
View file

@ -1,7 +1,7 @@
# Copy to .env and adjust. Read automatically by docker compose.
# Port pe care expune API-ul pe host
API_PORT=8080
# Port pe care expune API-ul pe host (containerul ascultă intern pe 8080)
API_PORT=8085
# Opțional: activează HF AI detector în m27 (Organika/sdxl-detector).
# Default 0 — testat empiric, regresie pe video out-of-distribution.

10
ai_platform/modules/forensic_features/README.md
View file

@ -56,12 +56,12 @@ Output: text formatat + 12-15 imagini PNG base64 + scoruri raw structurate.
# Build și pornește container
docker compose up -d
# Verifică
curl http://localhost:8080/health
curl http://localhost:8080/api/forensic-modules
# Verifică (8085 = portul host; containerul expune intern 8080)
curl http://localhost:8085/health
curl http://localhost:8085/api/forensic-modules
# Test end-to-end
curl -X POST http://localhost:8080/api/forensic-evidence \
curl -X POST http://localhost:8085/api/forensic-evidence \
-F "video=@test_video.mp4" \
-F "encode_images=1"
```
@ -99,7 +99,7 @@ forensic_features/
├── .env.example
├── requirements.txt # Lean: numpy, scipy, opencv, mediapipe, aiohttp
│
├── api.py # REST API (270 linii, doar 5 endpoints)
├── api.py # REST API (418 linii, 6 endpoints — inclusiv /metrics)
├── preprocessing.py # Frame extraction prin ffmpeg
├── face_landmarker.task # MediaPipe model (~3.6 MB)
│

39
ai_platform/modules/forensic_features/docs/API.md
View file

@ -5,9 +5,12 @@ Toate endpoint-urile, parametrii, status codes, exemple curl.
## Base URL
```
http://localhost:8080
http://localhost:8085
```
> `8085` este portul publicat pe host. În interiorul containerului serviciul
> ascultă pe `8080` (vezi `docker-compose.yml`: `127.0.0.1:${API_PORT:-8085}:8080`).
## Endpoints
### `POST /api/forensic-evidence`
@ -113,13 +116,13 @@ Apoi polling pe `/api/status/{job_id}` și preluare cu `/api/result/{job_id}`.
**Sync, toate modulele**:
```bash
curl -X POST http://localhost:8080/api/forensic-evidence \
curl -X POST http://localhost:8085/api/forensic-evidence \
-F "video=@suspicious_video.mp4"
```
**Sync, doar 2 module + fără base64 (mai rapid)**:
```bash
curl -X POST http://localhost:8080/api/forensic-evidence \
curl -X POST http://localhost:8085/api/forensic-evidence \
-F "video=@image.jpg" \
-F "modules=m27,m28" \
-F "encode_images=0"
@ -128,18 +131,18 @@ curl -X POST http://localhost:8080/api/forensic-evidence \
**Async, polling**:
```bash
# Submit
JOB=$(curl -s -X POST http://localhost:8080/api/forensic-evidence \
JOB=$(curl -s -X POST http://localhost:8085/api/forensic-evidence \
-F "video=@long_video.mp4" -F "async_mode=1" \
| python -c "import sys,json; print(json.load(sys.stdin)['job_id'])")
# Wait
while [ "$(curl -s http://localhost:8080/api/status/$JOB \
while [ "$(curl -s http://localhost:8085/api/status/$JOB \
| python -c "import sys,json; print(json.load(sys.stdin)['status'])")" != "done" ]; do
sleep 5
done
# Get result
curl http://localhost:8080/api/result/$JOB | jq .summary
curl http://localhost:8085/api/result/$JOB | jq .summary
```
---
@ -149,7 +152,7 @@ curl http://localhost:8080/api/result/$JOB | jq .summary
Listează modulele disponibile, pentru introspection.
```bash
curl http://localhost:8080/api/forensic-modules
curl http://localhost:8085/api/forensic-modules
```
```json
@ -177,7 +180,7 @@ curl http://localhost:8080/api/forensic-modules
Polling pentru cereri async. Returns 200 cu status string.
```bash
curl http://localhost:8080/api/status/abc123def456789a
curl http://localhost:8085/api/status/abc123def456789a
```
```json
@ -205,7 +208,7 @@ Preluare rezultat job async. Comportament:
| `error` | 500 cu mesajul de eroare |
```bash
curl http://localhost:8080/api/result/abc123def456789a
curl http://localhost:8085/api/result/abc123def456789a
```
Error code 404 dacă job_id nu există.
@ -217,7 +220,7 @@ Error code 404 dacă job_id nu există.
Healthcheck pentru Docker / load balancer / monitoring.
```bash
curl http://localhost:8080/health
curl http://localhost:8085/health
```
```json
@ -228,6 +231,20 @@ curl http://localhost:8080/health
}
```
---
### `GET /metrics`
Expune metrici Prometheus (contor + histogramă durată per rută) pentru
monitoring. Format text Prometheus.
```bash
curl http://localhost:8085/metrics
```
Returnează `503` cu `prometheus_client not installed` dacă dependența opțională
`prometheus_client` nu e instalată.
## Notes
### Limita upload
@ -241,7 +258,7 @@ Job store este **in-memory** (Python dict). Asta înseamnă:
- Joburile se pierd la restart container
- Multi-replica fără sticky sessions = nu funcționează
Pentru producție serioasă, înlocuiește `_jobs` din `api.py:30` cu un store
Pentru producție serioasă, înlocuiește `_jobs` din `api.py:46` cu un store
persistent (Redis recomandat).
### CORS

8
ai_platform/modules/forensic_features/docs/ARCHITECTURE.md
View file

@ -75,13 +75,13 @@ LLM-ul integrează totul și decide singur. **Noi nu decidem nimic.**
Pentru `POST /api/forensic-evidence` cu un video:
### 1. Upload + validare (api.py:106-180)
### 1. Upload + validare (api.py:152-211, în `handle_forensic_evidence`)
- Multipart streaming în chunks 64KB (nu buffer tot fișierul)
- Validare extensie: mp4/mov/avi/mkv/webm/jpg/png
- Asignare `job_id` UUID hex 16 chars
- Salvare temporară la `/app/data/inference/{job_id}/`
### 2. Orchestrator setup (forensic/orchestrator.py:75-110)
### 2. Orchestrator setup (forensic/orchestrator.py:220-270, în `run_forensic_pipeline`)
- **Adaptive `every_n_frames`** bazat pe durata video:
- < 5s → every_n=1 (toate cadrele)
- < 30s → every_n=3 (~10 fps efectiv)
@ -91,7 +91,7 @@ Pentru `POST /api/forensic-evidence` cu un video:
- Extracție cadre cu ffmpeg via `preprocessing.extract_frames()`
- Scriere `_meta.json` cu fps efectiv (m25 îl folosește pentru rPPG)
### 3. Rulare module (forensic/orchestrator.py:120-145)
### 3. Rulare module (forensic/orchestrator.py:283-296, via `_run_single_module` la :77)
Fiecare modul rulează **secvențial** (default), izolat în try/except:
- Crash într-un modul NU oprește restul
- Modulul eșuat returnează `empty_response()` cu primary_score=None
@ -117,7 +117,7 @@ Construiește:
- **summary** — obiect cu scoruri agregate pentru parsing programatic
- **instruction_for_llm** — text fix care explică LLM-ului cum să folosească datele
### 6. Răspuns (api.py:225-265)
### 6. Răspuns (api.py:235-256, ramura sync din `handle_forensic_evidence`)
- Sync (default): JSON imediat
- Async (`async_mode=1`): 202 cu job_id, polling pe `/api/status/{id}`

10
ai_platform/modules/forensic_features/docs/INTEGRATION.md
View file

@ -35,7 +35,7 @@ import requests
import base64
from pathlib import Path
FORENSIC_API = "http://localhost:8080"
FORENSIC_API = "http://localhost:8085"
QWEN_API = "http://your-qwen-host:14011/v1/chat/completions"
def analyze_video(video_path: str, your_typologies: list[str]) -> dict:
@ -97,7 +97,7 @@ def analyze_video(video_path: str, your_typologies: list[str]) -> dict:
# ── Pas 4: Apelează LLM-ul tău ───────────────────────────────────
qwen_payload = {
"model": "Qwen3.5-397B-A17B",
"model": "qwen3.5",
"messages": [
{"role": "system", "content": "Ești analist forensic. Răspunzi în JSON valid."},
{"role": "user", "content": content},
@ -138,7 +138,7 @@ async function analyzeWithForensic(videoFile) {
fd.append("video", videoFile);
fd.append("encode_images", "1");
const forensicResp = await fetch("http://localhost:8080/api/forensic-evidence", {
const forensicResp = await fetch("http://localhost:8085/api/forensic-evidence", {
method: "POST",
body: fd,
});
@ -351,10 +351,10 @@ else:
```bash
# Verifică serviciu
curl http://localhost:8080/health
curl http://localhost:8085/health
# Test cu un video real
curl -X POST http://localhost:8080/api/forensic-evidence \
curl -X POST http://localhost:8085/api/forensic-evidence \
-F "video=@your_test_video.mp4" \
-o response.json

44
ai_platform/modules/gateway/INDEX.md
View file

@ -1,6 +1,6 @@
# Gateway (didiAI-gateway)
API gateway for the AI platform: a thin **Nginx reverse proxy** that acts as the single externally exposed entry point for all didiAI internal services (LLM inference, audio transcription, web fact-checking, catalog, embeddings, rerank). Every request except `/health` requires Bearer token authentication enforced via an `nginx map` block. Currently in a **restart loop** because several upstream containers it references do not exist on this host (see "Current status" below).
API gateway for the AI platform: a thin **Nginx reverse proxy** that acts as the single externally exposed entry point for all didiAI internal services (LLM inference, audio transcription, web fact-checking, catalog, embeddings, rerank). Every request except `/health` requires Bearer token authentication enforced via an `nginx map` block. It currently starts cleanly: only the upstreams whose containers exist on this host (`web`, `catalog`) are active; the upstreams for services not deployed here are commented out in `nginx.conf.template` (see "Current status" below).
- **Stack**: Nginx 1.27-alpine, no Python, no custom code (config-only module).
- **URL** (intended): `http://<host>:11000` — listens on a single port, `11000`, mapped 1:1 to the host. No TLS at this layer.
@ -22,36 +22,34 @@ There is no rate limiting, no request body inspection, no JWT/Keycloak — just
All routes from `deploy/nginx.conf.template`:
| Path prefix | Auth | Upstream | Notes |
|-------------|------|----------|-------|
| `GET /health` | none | nginx direct (returns `{"status":"ok","service":"didiAI-gateway"}`) | for healthcheck |
| `/llm/*` | Bearer | `didiAI-llm-api:14011` | SSE streaming: `proxy_buffering off`, `chunked_transfer_encoding on`, HTTP/1.1, `Connection: ''` |
| `/audio/*` | Bearer | `didiAI-audio-api:54300` | `client_body_buffer_size 10M` for large uploads |
| `/web/*` | Bearer | `didiAI-web-api:51100` | plain proxy_pass |
| `/catalog/*` | Bearer | `didiAI-catalog-api:11000` | plain proxy_pass; note: catalog also listens on 11000 internally |
| `/embeddings/*` | Bearer | `didiAI-embeddings-api:14100` | OpenAI-compatible API |
| `/rerank/*` | Bearer | `didiAI-rerank-api:14200` | Cohere/Jina-compatible API |
| `/` (anything else) | none | nginx direct | returns `404 {"error":"not_found","routes":[...]}` listing the valid prefixes |
| Path prefix | Auth | Upstream | Status | Notes |
|-------------|------|----------|--------|-------|
| `GET /health` | none | nginx direct (returns `{"status":"ok","service":"didiAI-gateway"}`) | active | for healthcheck |
| `/web/*` | Bearer | `didiAI-web-api:51100` | active | plain proxy_pass |
| `/catalog/*` | Bearer | `didiAI-catalog-api:11000` | active | plain proxy_pass; note: catalog also listens on 11000 internally |
| `/llm/*` | Bearer | `didiAI-llm-api:14011` | disabled (upstream + location commented) | SSE streaming: `proxy_buffering off`, `chunked_transfer_encoding on`, HTTP/1.1, `Connection: ''` |
| `/audio/*` | Bearer | `didiAI-audio:54300` | disabled (upstream + location commented) | `client_body_buffer_size 10M` for large uploads |
| `/embeddings/*` | Bearer | `didiAI-embeddings-api:14100` | disabled (upstream + location commented) | OpenAI-compatible API |
| `/rerank/*` | Bearer | `didiAI-rerank-api:14200` | disabled (upstream + location commented) | Cohere/Jina-compatible API |
| `/` (anything else) | none | nginx direct | active | returns `404 {"error":"not_found","routes":["/web/","/catalog/","/health"]}` listing the active prefixes |
The disabled routes are kept commented in `nginx.conf.template`; re-enable the matching `upstream` + `location` blocks once those containers run on this host.
Trailing slash on `proxy_pass http://upstream/;` strips the `/<service>/` prefix when forwarding (so `/llm/v1/chat` becomes `/v1/chat` upstream).
## Current status
**BROKEN / restart loop as of 2026-05-01.** `docker logs didiAI-gateway` shows nginx failing to start with:
**RESOLVED — gateway starts cleanly.** The earlier restart loop (`[emerg] host not found in upstream "didiAI-llm-api:14011"`) was caused by nginx resolving all upstream hostnames at config load (not per-request), so any missing upstream container aborted startup.
```
[emerg] host not found in upstream "didiAI-llm-api:14011" in /etc/nginx/nginx.conf:35
```
Fix applied: in `nginx.conf.template` the `upstream` + `location` blocks for the services not deployed on this host are commented out (option 2 below). The gateway now starts with only the active upstreams:
Root cause: the nginx upstreams resolve hostnames at config load (not per-request), so any missing upstream container kills the whole gateway. On this host only **2 of the 6 upstreams** are running:
- Active: `didiAI-web-api` (`/web/`), `didiAI-catalog-api` (`/catalog/`), plus the `/health` and `/` (404) direct locations.
- Commented out (re-enable when their containers run here): `didiAI-llm-api`, `didiAI-audio`, `didiAI-embeddings-api`, `didiAI-rerank-api`.
- Running: `didiAI-catalog-api`, `didiAI-web-api`.
- Missing: `didiAI-llm-api`, `didiAI-audio-api`, `didiAI-embeddings-api`, `didiAI-rerank-api`.
Fix options (pick one before redeploying):
Other ways the same problem could be addressed if you prefer to keep all blocks listed:
1. Start the missing service modules (`llm-inference`, `audio`, `embeddings`, `rerank`) on this host so the names resolve.
2. Edit `nginx.conf.template` and remove (or comment out) the `upstream` blocks + `location` blocks for services not deployed locally.
2. (applied) Comment out the `upstream` + `location` blocks for services not deployed locally.
3. Switch the upstream definitions to lazy-resolution form (`set $upstream "didiAI-llm-api:14011"; proxy_pass http://$upstream/;` plus a `resolver` directive) so missing names fail per-request instead of bringing the whole gateway down.
Investigation commands:
@ -68,7 +66,7 @@ modules/gateway/
├── INDEX.md # This file
└── deploy/
├── docker-compose.yml # nginx:1.27-alpine, port 11000:11000, didi-network network
├── nginx.conf.template # 175 lines: map auth, 6 upstreams, 7 locations + 404 fallback
├── nginx.conf.template # ~169 lines: map auth, 2 active upstreams (4 commented), 3 active locations (/web/, /catalog/, /health) + 404 fallback (4 locations commented)
├── deploy.sh # Bash wrapper: --detach / --down / --logs, fail-fast on missing GATEWAY_API_TOKEN
├── .env.example # Template (only GATEWAY_API_TOKEN)
└── .env # Active config (GATEWAY_API_TOKEN value)
@ -111,7 +109,7 @@ The AI platform mixes two patterns; this gateway is **opt-in aggregation**, not
| didiAI-llm-api | `:14011` | YES (`/llm/`) — required for SSE | LLM router; gateway adds streaming-friendly proxy settings |
| didiAI-embeddings-api | `:14100` | YES (`/embeddings/`) and direct | OpenAI-compatible |
| didiAI-rerank-api | `:14200` | YES (`/rerank/`) and direct | Cohere/Jina-compatible |
| didiAI-audio-api | `:54300` | YES (`/audio/`) — recommended for large uploads | gateway sets a 10 MB body buffer |
| didiAI-audio | `:54300` | YES (`/audio/`) — recommended for large uploads | gateway sets a 10 MB body buffer |
| didiAI-catalog-api | `:11000` | YES (`/catalog/`) | Catalog also listens on 11000 internally — same number as gateway, different network endpoint |
So the gateway aggregates the **inference/IO services** (LLM, embeddings, rerank, audio, web, catalog) under one host:port, while observability/orchestration components (brain, dashboard) stay on their own ports.

40
ai_platform/modules/gateway/README.md
View file

@ -5,20 +5,25 @@ Nginx reverse proxy that serves as the **single entry point** for all didiAI ser
## Prerequisites
- All global prerequisites (see main [README.md](../../README.md))
- Docker network `deploy_default` (shared with other modules)
- Docker network `didi-network` (shared with other modules)
- At least one backend service running (llm-inference, audio, web, catalog-api)
## Routes
| Route | Upstream | Description |
|-------|----------|-------------|
| `/health` | (nginx direct) | Health check, no auth required |
| `/llm/` | `didiAI-llm-api:14011` | LLM Inference API (SSE streaming enabled) |
| `/audio/` | `didiAI-audio-api:54300` | Audio Transcription API (10M body buffer) |
| `/web/` | `didiAI-web-api:51100` | Web Fact-checking API |
| `/catalog/` | `didiAI-catalog-api:11000` | Catalog API (service discovery) |
| `/embeddings/` | `didiAI-embeddings-api:14100` | Embeddings API (OpenAI-compatible) |
| `/rerank/` | `didiAI-rerank-api:14200` | Rerank API (Cohere/Jina-compatible) |
Only the routes whose upstream containers exist on this host are active. The
others are present in `nginx.conf.template` but commented out (upstream + location
blocks), because nginx resolves upstream hostnames at config load and a missing
name aborts startup.
| Route | Upstream | Status | Description |
|-------|----------|--------|-------------|
| `/health` | (nginx direct) | active | Health check, no auth required |
| `/web/` | `didiAI-web-api:51100` | active | Web Fact-checking API |
| `/catalog/` | `didiAI-catalog-api:11000` | active | Catalog API (service discovery) |
| `/llm/` | `didiAI-llm-api:14011` | disabled (upstream commented) | LLM Inference API (SSE streaming) — re-enable when deployed |
| `/audio/` | `didiAI-audio:54300` | disabled (upstream commented) | Audio Transcription API (10M body buffer) — re-enable when deployed |
| `/embeddings/` | `didiAI-embeddings-api:14100` | disabled (upstream commented) | Embeddings API (OpenAI-compatible) — re-enable when deployed |
| `/rerank/` | `didiAI-rerank-api:14200` | disabled (upstream commented) | Rerank API (Cohere/Jina-compatible) — re-enable when deployed |
## Authentication
@ -53,7 +58,7 @@ docker compose up -d
# Test
curl http://localhost:11000/health
curl -H "Authorization: Bearer <token>" http://localhost:11000/llm/health
curl -H "Authorization: Bearer <token>" http://localhost:11000/catalog/health
```
## Port
@ -78,10 +83,11 @@ Client
v
Gateway (nginx :11000) ---> Bearer token check
|
+-- /llm/ --> didiAI-llm-api:14011 (SSE streaming)
+-- /audio/ --> didiAI-audio-api:54300 (large uploads)
+-- /web/ --> didiAI-web-api:51100
+-- /catalog/ --> didiAI-catalog-api:11000
+-- /embeddings/ --> didiAI-embeddings-api:14100
+-- /rerank/ --> didiAI-rerank-api:14200
+-- /web/ --> didiAI-web-api:51100 (active)
+-- /catalog/ --> didiAI-catalog-api:11000 (active)
|
+-- /llm/ --> didiAI-llm-api:14011 (disabled — upstream commented)
+-- /audio/ --> didiAI-audio:54300 (disabled — upstream commented)
+-- /embeddings/ --> didiAI-embeddings-api:14100 (disabled — upstream commented)
+-- /rerank/ --> didiAI-rerank-api:14200 (disabled — upstream commented)
```

2
ai_platform/modules/gateway/deploy/docker-compose.yml
View file

@ -18,7 +18,7 @@
# Naming Convention: didiAI-{module}-{service}
#
# Network:
# Uses deploy_default network (shared with other modules)
# Uses didi-network network (shared with other modules)
networks:
didi-network:

83
ai_platform/modules/llm-inference/API.md
View file

@ -36,6 +36,7 @@ Authorization: Bearer <your-token>
**Protected endpoints (require auth when enabled):**
- `POST /v1/chat/completions`
- `POST /v1/completions`
- `GET /v1/models`
- `POST /v1/models/load`
- `POST /v1/models/unload`
@ -59,6 +60,8 @@ Backend services require their own API keys:
- **LiteLLM**: `OPENROUTER_API_KEY`, `OPENAI_API_KEY`, or `ANTHROPIC_API_KEY`
- **vLLM**: Optional `LLM_VLLM_API_KEY` for vLLM server
> **Reasoning model note (`LLM_VLLM_DISABLE_THINKING`, default `true`):** the local model `Qwen/Qwen3.5-35B-A3B` (served as `qwen3.5`) is a reasoning model. By default the gateway injects `chat_template_kwargs={"enable_thinking": false}` on vLLM chat requests so the model returns the final answer directly instead of a `thinking` preamble — important for callers that parse JSON. Callers may override by passing their own `chat_template_kwargs` in the request body.
## Rate Limiting
The API uses token bucket rate limiting:
@ -193,6 +196,66 @@ curl -X POST http://localhost:14011/v1/chat/completions \
---
### Text Completions (legacy)
Create a non-streaming text completion from a raw prompt. OpenAI-compatible legacy `/v1/completions`. Routes to the model's backend (vLLM primary, cloud via LiteLLM). Backends that do not support text completion return `501`.
```
POST /v1/completions
```
#### Request Body
| Field | Type | Required | Default | Description |
|-------|------|----------|---------|-------------|
| `prompt` | string/array | Yes | - | Prompt(s) to complete |
| `model` | string | No | server default | Model (alias) to use |
| `temperature` | float | No | 0.7 | Sampling temperature (0.0-2.0) |
| `max_tokens` | integer | No | null | Maximum tokens to generate (1-1,000,000) |
| `backend` | string | No | null | Backend override: `litellm`, `vllm`, `llamacpp` |
| `top_p` | float | No | null | Top-p sampling (0.0-1.0) |
| `frequency_penalty` | float | No | null | Frequency penalty (-2.0 to 2.0) |
| `presence_penalty` | float | No | null | Presence penalty (-2.0 to 2.0) |
| `stop` | string/array | No | null | Stop sequences |
#### Response
```json
{
"id": "cmpl-abc123",
"object": "text_completion",
"created": 1704067200,
"model": "qwen3.5",
"choices": [
{
"index": 0,
"text": "Paris is the capital of France.",
"finish_reason": "stop"
}
],
"usage": {
"prompt_tokens": 8,
"completion_tokens": 7,
"total_tokens": 15
},
"backend": "vllm"
}
```
#### Example Request
```bash
curl -X POST http://localhost:14011/v1/completions \
-H "Content-Type: application/json" \
-d '{
"prompt": "The capital of France is",
"model": "qwen3.5",
"max_tokens": 16
}'
```
---
### List Models
List available models across all or specific backends.
@ -433,6 +496,22 @@ curl http://localhost:14011/ready
---
### Service Info (catalog)
Service metadata for cross-module catalog integration. Returns `resource`, `models` (from all enabled backends), and `functions` (API endpoint descriptors). Consumed by the catalog-api.
```
GET /v1/info
```
#### Example Request
```bash
curl http://localhost:14011/v1/info
```
---
## Error Responses
All errors follow a consistent format:
@ -507,9 +586,9 @@ Response includes `Retry-After` header with seconds to wait.
## Supported Backends
### LiteLLM (Default)
### LiteLLM (cloud)
Supports 100+ LLM providers through a unified interface.
Supports 100+ LLM providers through a unified interface. (`LLM_DEFAULT_BACKEND` is required and has no default; the DIDI deployment runs `vllm` with the local `qwen3.5` model and uses LiteLLM for cloud/premium models.)
**Popular models:**
- `gpt-3.5-turbo`, `gpt-4`, `gpt-4-turbo` (OpenAI)

13
ai_platform/modules/llm-inference/INDEX.md
View file

@ -1,18 +1,19 @@
# llm-inference
OpenAI-compatible LLM router for DIDI. Routes requests to local Qwen 3.5 397B (llama.cpp / vLLM backends) for free tier and proxies to OpenRouter cloud (Claude Sonnet 4.6, Gemini Flash 3, GPT-4o, etc.) for premium models. All endpoints are OpenAI-compatible (`/v1/chat/completions`), so callers can use the OpenAI SDK or plain `httpx` interchangeably.
OpenAI-compatible LLM router for DIDI. Routes requests to the local `Qwen/Qwen3.5-35B-A3B` model (served as `qwen3.5`, via vLLM / llama.cpp backends) for the free tier and proxies to OpenRouter cloud (Claude Sonnet 4.6, Gemini Flash 3, GPT-4o, etc.) for premium models. All endpoints are OpenAI-compatible (`/v1/chat/completions`), so callers can use the OpenAI SDK or plain `httpx` interchangeably.
- **Stack**: Python 3.10+, FastAPI, uvicorn, httpx, LiteLLM, Pydantic v2, sse-starlette
- **URL**: `http://10.11.10.17:14011` (LLM router on GPU host) — referenced as `LLM_ROUTER_URL` in DIDI services
- **Container**: `didiAI-llm-api` (image `didiai-llm-api`), runs on GPU host (typically `10.11.10.17`)
- **Local model**: Qwen 3.5-35B-A3B (MoE, native multimodal text+vision) via vLLM (`didiAI-vllm-qwen3.5`, internal port 14001) or Qwen 3.5 397B-A17B variant via llama.cpp pool
- **Local model**: `Qwen/Qwen3.5-35B-A3B` (MoE **reasoning** model; thinking can be toggled on/off — the gateway disables it by default, see `LLM_VLLM_DISABLE_THINKING`) served as `qwen3.5` via vLLM (`didiAI-vllm-qwen3.5`, internal port 14001) or via the llama.cpp pool
- **Entry point**: `llm-inference` console script -> `src/llm_inference/cli.py:main` -> uvicorn factory `llm_inference.api.app:create_app`
## Ce face
Single OpenAI-compatible endpoint (`/v1/chat/completions`) that selects a backend based on the request's `model` field and request-time `backend` override:
- **Auto backend resolution** (`LLMClient._resolve_backend_for_model` in `src/llm_inference/client.py`): probes each enabled local backend's `list_models()`. If the requested model is served locally, route to that backend; otherwise fall back to the configured default (litellm).
- **Auto backend resolution** (`LLMClient._resolve_backend_for_model` in `src/llm_inference/client.py`): the requested `model` is first normalized through `LLM_MODEL_ALIASES` (e.g. `qwen3.5` → `Qwen/Qwen3.5-35B-A3B`), then each enabled local backend's `list_models()` is probed. If the model is served locally, route there; otherwise fall back to `LLM_DEFAULT_BACKEND`.
- **Cross-backend fallback cascade** (`LLM_ENABLE_FALLBACK`, default `true`): on a backend failure the request is retried on the next enabled backend in `LLM_FALLBACK_ORDER` (default `[vllm, llamacpp, litellm]`). The cascade is skipped when the caller pins an explicit `backend`.
- **Explicit backend override**: clients may pass `"backend": "litellm" | "vllm" | "llamacpp"` in the JSON body to force routing.
- **Streaming + non-streaming**: same endpoint; `"stream": true` returns SSE chunks (`text/event-stream` with `data: [DONE]` terminator).
- **Multimodal**: messages are pre-processed by `image_processing.process_messages` so image URLs/base64 attachments work the same across backends.
@ -27,6 +28,7 @@ All paths are mounted by `src/llm_inference/api/app.py`:
| Method | Path | Purpose |
|--------|------|---------|
| POST | `/v1/chat/completions` | OpenAI-compat chat completion (streaming + non-streaming) |
| POST | `/v1/completions` | OpenAI-compat legacy text completion (non-streaming; 501 from backends that don't support it) |
| GET | `/v1/models` | List models from all (or specific via `?backend=`) backends |
| POST | `/v1/models/load` | Load a model on a local backend (vLLM / llama.cpp) |
| POST | `/v1/models/unload` | Unload a model from a local backend |
@ -113,9 +115,12 @@ All env vars use the `LLM_` prefix (Pydantic Settings, `extra="forbid"` so typos
Common optional:
- `OPENROUTER_API_KEY`, `OPENAI_API_KEY`, `ANTHROPIC_API_KEY` — provider keys for LiteLLM
- `LLM_DEFAULT_MODEL` (default `gpt-3.5-turbo`)
- `LLM_DEFAULT_MODEL` (default `qwen3.5`) — alias used when a request omits `model`
- `LLM_MODEL_ALIASES` (default `{}`) — JSON map of friendly alias → real served id, e.g. `{"qwen3.5":"Qwen/Qwen3.5-35B-A3B"}`
- `LLM_ENABLE_FALLBACK` (default `true`), `LLM_FALLBACK_ORDER` (default `[vllm,llamacpp,litellm]`) — cross-backend fallback cascade
- `LLM_HOST` (default `0.0.0.0`), `LLM_PORT` (default `14011`)
- `LLM_VLLM_BASE_URL` (default `http://localhost:14001`), `LLM_VLLM_API_KEY`
- `LLM_VLLM_DISABLE_THINKING` (default `true`) — injects `chat_template_kwargs={'enable_thinking': false}` on vLLM chat requests so the Qwen3.5 reasoning model returns the final answer directly (no `thinking` preamble); key behavior for JSON-parsing callers. Callers may override per request.
- `LLM_LLAMACPP_BASE_URL` (single-server legacy) **or** `LLM_LLAMACPP_BASE_URLS` (comma-separated pool, overrides single)
- `LLM_LLAMACPP_HEALTH_CHECK_INTERVAL` (default `30s`)
- `LLM_REQUEST_TIMEOUT` (default `120s`), `LLM_CONNECT_TIMEOUT` (default `10s`)

14
ai_platform/modules/llm-inference/README.md
View file

@ -91,10 +91,12 @@ cp ../.env.example .env
| Endpoint | Method | Description |
|----------|--------|-------------|
| `/v1/chat/completions` | POST | Chat completion (supports streaming) |
| `/v1/completions` | POST | Legacy text completion (non-streaming; 501 if backend unsupported) |
| `/v1/models` | GET | List available models |
| `/v1/models/load` | POST | Load a model (local backends) |
| `/v1/models/unload` | POST | Unload a model (local backends) |
| `/v1/backends` | GET | List available backends |
| `/v1/info` | GET | Service/catalog metadata |
| `/health` | GET | Health check |
| `/ready` | GET | Readiness probe |
@ -125,13 +127,17 @@ Configure via environment variables (prefix: `LLM_`):
| Variable | Default | Description |
|----------|---------|-------------|
| `LLM_DEFAULT_BACKEND` | `litellm` | Default backend (litellm, vllm, llamacpp) |
| `LLM_DEFAULT_MODEL` | `gpt-3.5-turbo` | Default model |
| `LLM_DEFAULT_BACKEND` | _required_ | Default backend (`litellm`, `vllm`, `llamacpp`) — no default, must be set; the DIDI deployment runs `vllm` |
| `LLM_DEFAULT_MODEL` | `qwen3.5` | Default model alias used when a request omits `model` |
| `LLM_MODEL_ALIASES` | `{}` | JSON map of alias → served id, e.g. `{"qwen3.5":"Qwen/Qwen3.5-35B-A3B"}` |
| `LLM_ENABLE_FALLBACK` | `true` | Cross-backend fallback cascade on backend failure |
| `LLM_FALLBACK_ORDER` | `[vllm,llamacpp,litellm]` | Backend order tried in the fallback cascade |
| `LLM_PORT` | `14011` | API server port |
| `LLM_HOST` | `0.0.0.0` | API server host |
| `LLM_ENABLE_VLLM` | `false` | Enable vLLM backend |
| `LLM_ENABLE_LLAMACPP` | `false` | Enable llama.cpp backend |
| `LLM_ENABLE_VLLM` | _required_ | Enable vLLM backend (no default) |
| `LLM_ENABLE_LLAMACPP` | _required_ | Enable llama.cpp backend (no default) |
| `LLM_VLLM_BASE_URL` | `http://localhost:14001` | vLLM server URL |
| `LLM_VLLM_DISABLE_THINKING` | `true` | Inject `enable_thinking=false` for the Qwen3.5 reasoning model so it returns the final answer directly (no thinking preamble) |
| `LLM_LLAMACPP_BASE_URL` | `http://localhost:8080` | llama.cpp server URL |
| `OPENROUTER_API_KEY` | - | OpenRouter API key |
| `OPENAI_API_KEY` | - | OpenAI API key |

6
ai_platform/modules/llm-inference/deploy/Dockerfile
View file

@ -8,12 +8,12 @@ COPY --from=ghcr.io/astral-sh/uv:0.10 /uv /usr/local/bin/uv
WORKDIR /app
# Copy project files
COPY pyproject.toml uv.lock README.md ./
# Copy project files (uv.lock not committed; resolved at build time)
COPY pyproject.toml README.md ./
COPY src/ ./src/
# Install dependencies
RUN uv sync --frozen --no-dev
RUN uv sync --no-dev
# Production image
FROM python:3.11.12-slim

8
ai_platform/modules/llm-inference/deploy/docker-compose.yml
View file

@ -18,10 +18,10 @@
# Naming Convention: didiAI-{module}-{service}
#
# Network:
# Uses deploy_default network (shared with other modules)
# Uses didi-network (shared with all DIDI + AI platform stacks)
networks:
deploy_default:
didi-network:
external: true
services:
@ -37,7 +37,7 @@ services:
ports:
- "14011:14011"
networks:
- deploy_default
- didi-network
environment:
- LLM_PORT=14011
- LLM_EXTERNAL_URL=${LLM_EXTERNAL_URL}
@ -72,7 +72,7 @@ services:
ports:
- "14001:14001"
networks:
- deploy_default
- didi-network
volumes:
- ${HF_CACHE_DIR}:/root/.cache/huggingface
environment:

18
ai_platform/modules/llm-inference/src/llm_inference/backends/vllm_backend.py
View file

@ -105,6 +105,21 @@ class VLLMBackend(LLMBackend):
"""Backend identifier name."""
return "vllm"
def _apply_thinking_default(self, kwargs: dict[str, object]) -> dict[str, object]:
"""Default reasoning models to non-thinking output for clean answers.
Injects ``extra_body={"chat_template_kwargs": {"enable_thinking": False}}``
unless the caller already supplied ``chat_template_kwargs``. Lets JSON-parsing
consumers (extractors, video semantic, brain) get the final answer directly.
"""
if not getattr(self._settings, "vllm_disable_thinking", True):
return kwargs
extra = dict(kwargs.get("extra_body") or {}) # type: ignore[arg-type]
ctk = dict(extra.get("chat_template_kwargs") or {})
ctk.setdefault("enable_thinking", False)
extra["chat_template_kwargs"] = ctk
return {**kwargs, "extra_body": extra}
async def complete(
self,
messages: list[ChatMessage],
@ -127,6 +142,8 @@ class VLLMBackend(LLMBackend):
LLMTimeoutError: If the request times out.
"""
kwargs = self._apply_thinking_default(kwargs)
async def _do_complete() -> CompletionResponse:
response = await self._client.chat.completions.create(
model=model,
@ -240,6 +257,7 @@ class VLLMBackend(LLMBackend):
LLMTimeoutError: If the request times out.
"""
stream = None
kwargs = self._apply_thinking_default(kwargs)
try:
stream = await self._client.chat.completions.create(
model=model,

10
ai_platform/modules/llm-inference/src/llm_inference/config.py
View file

@ -111,6 +111,16 @@ class LLMSettings(BaseSettings):
default=None,
description="API key for vLLM server (if required)",
)
vllm_disable_thinking: bool = Field(
default=True,
description=(
"Inject chat_template_kwargs={'enable_thinking': False} on vLLM chat "
"requests so reasoning models (e.g. Qwen3.5) return the final answer "
"directly instead of a 'thinking' preamble — required for callers that "
"parse JSON (extractors sentiment/OCR, video semantic, brain). Callers "
"may override by passing their own chat_template_kwargs."
),
)
# llama.cpp settings
llamacpp_base_url: str = Field(

46
ai_platform/modules/rerank/INDEX.md
View file

@ -132,7 +132,6 @@ rerank/
├── API.md # Full HTTP API reference + curl/Python examples
├── INDEX.md # This file
├── pyproject.toml # Package metadata, deps (fastapi, httpx, openai SDK)
├── uv.lock # uv lockfile
├── .env.example # All RERANK_* env vars documented
├── deploy/
│ ├── Dockerfile # API wrapper image
@ -196,30 +195,35 @@ All env vars use the `RERANK_` prefix and are read via Pydantic Settings (`confi
| `RERANK_LOG_LEVEL` | `INFO` | |
| `RERANK_LOG_JSON` | `false` | |
**vLLM backend:**
**Backend base URLs (app settings — fields on `RerankSettings`):**
| Var | Default |
|-----|---------|
| `RERANK_VLLM_BASE_URL` | `http://localhost:54201` |
| `RERANK_VLLM_MODEL` | `BAAI/bge-reranker-v2-m3` |
| `RERANK_VLLM_PORT` | `14201` |
| `RERANK_VLLM_GPU` | `0` |
| `RERANK_VLLM_GPU_UTIL` | `0.50` |
| `RERANK_VLLM_MAX_LEN` | `8192` |
| Var | Default | Notes |
|-----|---------|-------|
| `RERANK_VLLM_BASE_URL` | `http://localhost:54201` | Where the API wrapper reaches vLLM |
| `RERANK_VLLM_API_KEY` | (empty) | Optional API key for the vLLM server |
| `RERANK_LLAMACPP_BASE_URL` | `http://localhost:54210` | Where the API wrapper reaches llama.cpp |
**llama.cpp backend:**
> Only `*_BASE_URL` / `*_API_KEY` are read by the API app (`config.py`). The model id, port, GPU and context knobs below are **not** `RerankSettings` fields.
| Var | Default |
|-----|---------|
| `RERANK_LLAMACPP_BASE_URL` | `http://localhost:54210` |
| `RERANK_LLAMACPP_MODEL` | `bge-reranker-v2-m3-q4_k_m.gguf` |
| `RERANK_LLAMACPP_PORT` | `14210` |
| `RERANK_LLAMACPP_CTX` | `8192` |
| `RERANK_LLAMACPP_THREADS` | `4` |
| `RERANK_LLAMACPP_PARALLEL` | `4` |
| `MODELS_DIR` | `/cai2_ds_storage/models` |
**Backend service variables (docker-compose only):**
**Shared HF:** `HF_CACHE_DIR` (default `/cai2_ds_storage/hf_cache`), `HF_TOKEN`.
These are consumed by `deploy/docker-compose.yml` to launch the vLLM / llama.cpp containers — they configure the backend server, not the API app.
| Var | Default | Description |
|-----|---------|-------------|
| `RERANK_VLLM_MODEL` | `BAAI/bge-reranker-v2-m3` | HF model id loaded by the vLLM container |
| `RERANK_VLLM_PORT` | `14201` | Host port mapped to the vLLM container |
| `RERANK_VLLM_GPU` | `0` | `CUDA_VISIBLE_DEVICES` for the vLLM container |
| `RERANK_VLLM_GPU_UTIL` | `0.50` | vLLM `--gpu-memory-utilization` |
| `RERANK_VLLM_MAX_LEN` | `8192` | vLLM `--max-model-len` |
| `RERANK_LLAMACPP_MODEL` | `bge-reranker-v2-m3-q4_k_m.gguf` | GGUF filename inside `MODELS_DIR` |
| `RERANK_LLAMACPP_PORT` | `14210` | Host port mapped to the llama.cpp container |
| `RERANK_LLAMACPP_CTX` | `8192` | llama.cpp context size |
| `RERANK_LLAMACPP_THREADS` | `4` | llama.cpp threads |
| `RERANK_LLAMACPP_PARALLEL` | `4` | llama.cpp parallel slots |
| `MODELS_DIR` | `/cai2_ds_storage/models` | GGUF model directory for llama.cpp |
| `HF_CACHE_DIR` | `/cai2_ds_storage/hf_cache` | Mounted into the vLLM container |
| `HF_TOKEN` | (unset) | Forwarded as `HUGGING_FACE_HUB_TOKEN` |
Full list with comments in `.env.example`.

10
ai_platform/modules/rerank/deploy/docker-compose.yml
View file

@ -14,10 +14,10 @@
# Naming Convention: didiAI-{module}-{service}
#
# Network:
# Uses deploy_default network (shared with other modules)
# Uses didi-network (shared with all DIDI + AI platform stacks)
networks:
deploy_default:
didi-network:
external: true
services:
@ -33,7 +33,7 @@ services:
ports:
- "${RERANK_PORT:-14200}:${RERANK_PORT:-14200}"
networks:
- deploy_default
- didi-network
environment:
- RERANK_PORT=${RERANK_PORT:-14200}
- RERANK_EXTERNAL_URL=${RERANK_EXTERNAL_URL}
@ -66,7 +66,7 @@ services:
ports:
- "${RERANK_VLLM_PORT:-14201}:14201"
networks:
- deploy_default
- didi-network
volumes:
- ${HF_CACHE_DIR:-/cai2_ds_storage/hf_cache}:/root/.cache/huggingface
environment:
@ -109,7 +109,7 @@ services:
ports:
- "${RERANK_LLAMACPP_PORT:-14210}:8080"
networks:
- deploy_default
- didi-network
volumes:
- ${MODELS_DIR:-/cai2_ds_storage/models}:/models:ro
command: >

6
ai_platform/modules/video-analysis/.env.example
View file

@ -1,7 +1,7 @@
# video-analysis configuration
# Copy to deploy/.env and fill ALL required values.
# Service must FAIL to start if required vars are missing.
# Tuning parameters are in deploy/config.yaml (not here)
# Tuning parameters default in settings.py; override via the VIDEO_ANALYSIS_* vars below
# =============================================================================
# REQUIRED (no defaults)
# =============================================================================
@ -18,7 +18,7 @@ VIDEO_ANALYSIS_RUNS_DIR=/app/runs
# vLLM Connection (auto-configured for docker-compose profiles)
# =============================================================================
# For profile 'api-vllm' or 'full': uses internal vllm-buster container
VIDEO_ANALYSIS_VLLM_BASE_URL=http://vllm-buster:8000
VIDEO_ANALYSIS_VLLM_BASE_URL=http://didiAI-video-vllm-buster:54500
VIDEO_ANALYSIS_VLLM_MODEL=busterx
# For profile 'api' with external vLLM: point to your external vLLM server
@ -26,7 +26,7 @@ VIDEO_ANALYSIS_VLLM_MODEL=busterx
# VIDEO_ANALYSIS_VLLM_MODEL=l8cv/BusterX_plusplus
# =============================================================================
# Optional Tuning (override config.yaml defaults)
# Optional Tuning (override settings.py defaults)
# =============================================================================
# VIDEO_ANALYSIS_FRAMES=16
# VIDEO_ANALYSIS_MAX_SIDE=960

62
ai_platform/modules/video-analysis/API.md
View file

@ -22,49 +22,17 @@ No authentication required.
---
## ⚠️ TESTING REMINDER: Alternative Vision Models
## Vision & Aggregation Models
**Current Configuration:**
- **Deepfake Detection:** Uses BusterX (Qwen2.5-VL-7B fine-tuned) @ port 54500
- **Semantic Analysis:** Uses BusterX (7B parameters)
Both endpoints are backed by a single vision model — **BusterX** (`l8cv/BusterX_plusplus`, served as `busterx`) @ port `54500`:
**TODO - Test with Qwen3-VL-30B for Better Semantic Analysis:**
| Stage | Model | Endpoint | Role |
|-------|-------|----------|------|
| Deepfake verdict | BusterX (Qwen2.5-VL-7B fine-tune) | `http://didiAI-video-vllm-buster:54500` | `REAL` / `FAKE` / `UNCERTAIN` + explanation |
| Semantic chunk descriptions | BusterX (same endpoint) | `http://didiAI-video-vllm-buster:54500` | per-chunk `description` |
| Semantic aggregation | DIDI text LLM (Qwen3.5) | `http://didiAI-llm-api:14011` | merges chunk descriptions into `final_summary` |
The semantic analysis endpoint can be configured to use **Qwen3-VL-30B** (already running @ port 14002) instead of BusterX for potentially better results:
| Model | Size | Port | Best For |
|-------|------|------|----------|
| **BusterX** | 7B | 54500 | Deepfake detection (specialized) |
| **Qwen3-VL-30B** | 30B | 14002 | General semantic understanding |
**To test with Qwen3-VL-30B:**
1. Update `.env`:
```bash
VIDEO_ANALYSIS_VLLM_BASE_URL=http://didiAI-llm-vllm-vision:14002 # Use Qwen3-VL instead of BusterX
VIDEO_ANALYSIS_VLLM_MODEL=qwen3-vl # Change from busterx
```
2. Rebuild container:
```bash
cd deploy/
docker compose build video-analysis-api
docker compose up -d video-analysis-api
```
3. Test semantic analysis:
```bash
curl -X POST http://localhost:54600/analyze/video/semantic \
-F "file=@test_video.mp4"
```
**Expected Benefits:**
- More detailed scene descriptions (30B vs 7B parameters)
- Better context understanding
- More coherent narrative flow
- Higher accuracy for complex scenes
**Note:** Deepfake detection should continue using BusterX (specialized model).
BusterX is self-contained — Qwen2.5-VL is bundled inside the fine-tune, so no separate vision base model is loaded. There is no separate Qwen3-VL vision backend in this deployment.
---
@ -359,6 +327,20 @@ curl -X POST http://localhost:54600/analyze/video/semantic \
---
### Service Info
Return service catalog metadata (resources, models, functions). Consumed by the DIDI `catalog-api`.
**GET** `/v1/info`
**Example**
```bash
curl http://localhost:54600/v1/info
```
---
## Error Responses
| Status | Description |

16
ai_platform/modules/video-analysis/INDEX.md
View file

@ -4,8 +4,8 @@ Video analysis service for DIDI. Performs deepfake detection and semantic tempor
- **Stack:** Python 3.11+, FastAPI, uvicorn, OpenCV (headless), Pillow, NumPy, httpx/requests, ffmpeg toolchain (via OpenCV), pydantic-settings + YAML
- **URL (Dev):** `http://10.11.10.12:54600`
- **Container:** runs on GPU host (`network_mode: host` in `deploy/docker-compose.yml`); the service itself is CPU-only — GPU is consumed by the upstream vLLM server
- **Vision backend:** external vLLM server (default: BusterX 7B @ port `54500`); same endpoint also drives semantic analysis. Optional alternative: Qwen3-VL-30B @ port `14002` for richer semantic narratives. The wider DIDI vision cascade (Qwen Vision local → Gemini Flash → GPT-4o) lives in `agent-v3`; this service only talks to one vLLM at a time.
- **Container:** runs on GPU host, attached to the external bridge network `didi-network` (see `deploy/docker-compose.yml`); the service itself is CPU-only — GPU is consumed by the upstream vLLM server
- **Vision backend:** external vLLM server (BusterX 7B @ port `54500`); the **same** BusterX endpoint drives both deepfake detection and semantic per-chunk analysis. Semantic narratives are produced by aggregating the per-chunk descriptions with the DIDI text LLM (Qwen3.5) via `http://didiAI-llm-api:14011`. The wider DIDI vision cascade (Qwen Vision local → Gemini Flash → GPT-4o) lives in `agent-v3`; this service talks only to the BusterX vLLM.
## Ce face
@ -50,8 +50,8 @@ Auth: none (called over private network / through Kong upstream by agent-v3).
This service does **not** implement a multi-provider cascade. It is a thin client over a single vLLM endpoint configured at startup:
- **Primary (deepfake):** BusterX (Qwen2.5-VL-7B fine-tune, `l8cv/BusterX_plusplus`) at `VIDEO_ANALYSIS_VLLM_BASE_URL` — typically `http://didiAI-video-vllm-buster:54500` on the GPU host
- **Optional (semantic):** Qwen3-VL-30B at `http://didiAI-llm-vllm-vision:14002` — swap by editing `deploy/.env` and rebuilding
- **Deepfake + semantic (vision):** BusterX (Qwen2.5-VL-7B fine-tune, `l8cv/BusterX_plusplus`, served as `busterx`) at `VIDEO_ANALYSIS_VLLM_BASE_URL` — typically `http://didiAI-video-vllm-buster:54500` on the GPU host. The same endpoint handles both the deepfake verdict and the per-chunk semantic descriptions. BusterX is self-contained (Qwen2.5-VL is bundled inside the fine-tune) — it does not load a separate Qwen base model.
- **Semantic aggregation (text):** per-chunk descriptions are merged into a narrative `final_summary` by the DIDI text LLM (Qwen3.5) via `http://didiAI-llm-api:14011` — set through `VIDEO_ANALYSIS_SEMANTIC_LLM_BASE_URL`.
- **DIDI-wide cascade** (Qwen Vision local → OpenRouter Gemini Flash → GPT-4o) is implemented in agent-v3, NOT here. This service is a leaf node in that chain — agent-v3 calls it as one of several vision options.
- Service refuses to start if `VIDEO_ANALYSIS_VLLM_BASE_URL` is not set or the vLLM endpoint is unreachable (see `buster_client.py`, `settings.py`).
@ -62,7 +62,7 @@ Env vars (prefix `VIDEO_ANALYSIS_`), loaded from `deploy/.env`:
| Variable | Required | Description |
|---|---|---|
| `VIDEO_ANALYSIS_VLLM_BASE_URL` | yes | Upstream vLLM server URL |
| `VIDEO_ANALYSIS_VLLM_MODEL` | yes | Model name passed to vLLM (`busterx`, `qwen3-vl`, `l8cv/BusterX_plusplus`, …) |
| `VIDEO_ANALYSIS_VLLM_MODEL` | yes | Served model name passed to vLLM (`busterx`; underlying weights `l8cv/BusterX_plusplus`) |
| `VIDEO_ANALYSIS_RUNS_DIR` | yes | Where to drop per-request artifacts (default `/app/runs` in container) |
| `VIDEO_ANALYSIS_EXTERNAL_URL` | yes | External URL embedded in the OpenAPI spec |
| `HF_TOKEN`, `HF_CACHE_DIR` | yes (when running bundled vLLM) | HuggingFace creds + shared cache for the vLLM container |
@ -74,7 +74,7 @@ Env vars (prefix `VIDEO_ANALYSIS_`), loaded from `deploy/.env`:
| `VIDEO_ANALYSIS_REPETITION_PENALTY` | no (default 1.05) | Repetition penalty |
| `NGINX_CONNECT_TIMEOUT` / `_SEND_TIMEOUT` / `_READ_TIMEOUT` | no | nginx upstream timeouts (only `api-nginx` profile) |
Tuning defaults live in `deploy/config.yaml`; env vars override YAML.
Tuning defaults are baked into `settings.py` and overridden via the `VIDEO_ANALYSIS_*` env vars above (a `deploy/config.yaml` may optionally be supplied to override defaults, but none ships with the module).
## Deployment
@ -93,8 +93,8 @@ Tuning defaults live in `deploy/config.yaml`; env vars override YAML.
## Related
- **agent-v3 video pipeline** — `/home/admin365/didi_mono/backend/services/orchestration-layer/agent-v3` is the consumer; orchestrates async video sessions and merges this service's verdict into `ai_tampered` + `techniques` results
- **BusterX vLLM** (port `54500`) — sibling service in the AI platform; the actual GPU-backed model that this service queries (referenced in main `CLAUDE.md` ports section)
- **Qwen3-VL vision vLLM** (port `14002`) — alternative semantic backend (`didiAI-llm-vllm-vision`)
- **BusterX vLLM** (port `54500`) — sibling service in the AI platform; the actual GPU-backed model that this service queries for both deepfake and semantic chunks (referenced in main `CLAUDE.md` ports section)
- **DIDI text LLM API** (`http://didiAI-llm-api:14011`) — Qwen3.5 endpoint used to aggregate semantic chunk descriptions into the final narrative
- **AI platform shared assets** — `../../README.md`, `../../ruff.toml`
- **Internal package layout:** `src/video_analysis/{app.py, buster_client.py, schemas.py, settings.py, video_sampling.py}`
- **Sibling docs:** `README.md`, `API.md`, `TESTING.md` in this folder

44
ai_platform/modules/video-analysis/README.md
View file

@ -78,6 +78,7 @@ cp ../.env.example .env
| `/health` | GET | Health check |
| `/analyze/video` | POST | Deepfake detection (fast, 16 frames) |
| `/analyze/video/semantic` | POST | Semantic analysis (detailed, 144+ frames) |
| `/v1/info` | GET | Service catalog metadata (used by catalog-api) |
### Example API Request
@ -125,8 +126,8 @@ Configured via environment variables (prefix: `VIDEO_ANALYSIS_`). These are typi
| Variable | Description |
|----------|-------------|
| `VIDEO_ANALYSIS_VLLM_BASE_URL` | vLLM server URL (e.g., `http://didiAI-video-vllm-buster:8000`) |
| `VIDEO_ANALYSIS_VLLM_MODEL` | Model name (e.g., `l8cv/BusterX_plusplus`) |
| `VIDEO_ANALYSIS_VLLM_BASE_URL` | vLLM server URL (e.g., `http://didiAI-video-vllm-buster:54500`) |
| `VIDEO_ANALYSIS_VLLM_MODEL` | Served model name (e.g., `busterx`) |
| `VIDEO_ANALYSIS_RUNS_DIR` | Directory for storing analysis artifacts (created/used at runtime) |
| `VIDEO_ANALYSIS_EXTERNAL_URL` | External URL for OpenAPI spec (e.g., `http://localhost:54600`) |
@ -144,7 +145,7 @@ If you use the `api-nginx` profile, the nginx container can read these optional
### Optional Tuning Parameters
Configured via `deploy/config.yaml` (env vars override YAML):
These default to the values below in `settings.py` and are overridden via the matching `VIDEO_ANALYSIS_*` environment variables (a `deploy/config.yaml` may optionally be supplied to override defaults, but none ships with the module):
| Parameter | Default | Description |
|-----------|---------|-------------|
@ -155,42 +156,24 @@ Configured via `deploy/config.yaml` (env vars override YAML):
| `temperature` | `0.000001` | Sampling temperature |
| `repetition_penalty` | `1.05` | Repetition penalty |
## ⚠️ Testing Recommendations
## Semantic Analysis Pipeline
### Model Selection for Semantic Analysis
Both endpoints use the **same** BusterX vLLM (`busterx` @ port `54500`):
**Current Setup:**
- Both deepfake and semantic analysis use **BusterX** (7B parameters)
- BusterX is optimized for deepfake detection
- **Deepfake endpoint** — BusterX returns the `REAL` / `FAKE` / `UNCERTAIN` verdict + explanation.
- **Semantic endpoint** — BusterX produces a per-chunk `description` for each temporal chunk. When `enable_aggregation=true`, those chunk descriptions are merged into a single narrative `final_summary` by the DIDI text LLM (Qwen3.5) via `http://didiAI-llm-api:14011` (set through `VIDEO_ANALYSIS_SEMANTIC_LLM_BASE_URL`).
**TODO: Test Semantic Analysis with Qwen3-VL-30B**
For better semantic understanding, consider testing with the larger **Qwen3-VL-30B** model (already running @ port 8102):
BusterX is self-contained — Qwen2.5-VL is bundled inside the `l8cv/BusterX_plusplus` fine-tune, so no separate vision base model is loaded. There is no separate Qwen3-VL vision backend in this deployment; the only vision model the service talks to is BusterX.
```bash
# Current (BusterX 7B)
# Vision backend (deepfake + semantic chunk descriptions)
VIDEO_ANALYSIS_VLLM_BASE_URL=http://didiAI-video-vllm-buster:54500
VIDEO_ANALYSIS_VLLM_MODEL=busterx
# Alternative (Qwen3-VL 30B) - Better for semantic analysis
VIDEO_ANALYSIS_VLLM_BASE_URL=http://didiAI-llm-vllm-vision:14002
VIDEO_ANALYSIS_VLLM_MODEL=qwen3-vl
# Text LLM used only to aggregate semantic chunks into a narrative summary
VIDEO_ANALYSIS_SEMANTIC_LLM_BASE_URL=http://didiAI-llm-api:14011
```
**Expected Improvements:**
- ✅ More detailed scene descriptions (30B vs 7B)
- ✅ Better understanding of complex actions
- ✅ More coherent narrative synthesis
- ✅ Higher quality semantic annotations
**Trade-offs:**
- ⏱️ Slightly higher latency (~15-20s per chunk vs ~12s)
- 📊 Better for semantic analysis, but keep BusterX for deepfake detection
**Recommendation:**
- **Deepfake endpoint:** Keep using BusterX (specialized for forgery detection)
- **Semantic endpoint:** Test with Qwen3-VL-30B for better results
## Deployment
```bash
@ -258,8 +241,7 @@ modules/video-analysis/
│ ├── docker-compose.yml # Docker services
│ ├── Dockerfile # Container image
│ ├── nginx.conf # Nginx reverse proxy config (optional)
│ ├── nginx.conf.template # Template-based nginx config (optional)
│ └── config.yaml # Tuning parameters
│ └── nginx.conf.template # Template-based nginx config (optional)
├── src/video_analysis/
│ ├── __init__.py
│ ├── app.py # FastAPI application

19
ai_platform/modules/video-analysis/TESTING.md
View file

@ -1,14 +1,23 @@
# Video Analysis - Testing Checklist
## ⚠️ HIGH PRIORITY: Model Comparison for Semantic Analysis
> ⚠️ **NOTĂ:** Acest fișier este un plan **EXPLORATORIU / R&D**, NU configurația livrată.
> Modelul `Qwen3-VL-30B` (port 14002 / `gpt-oss-120b` / `deploy-llm-api-1`) **nu există** în
> deployment-ul livrat. Pipeline-ul REAL: deepfake + semantic folosesc **BusterX** (`busterx`,
> vLLM `didiAI-video-vllm-buster:54500`), iar agregarea semantică pe text folosește **Qwen3.5**
> (`didiAI-llm-api:14011`). Pentru testarea sistemului livrat vezi `ai_platform/local_gpu_stack/TESTING.md`.
## R&D (opțional): Model Comparison for Semantic Analysis
### Background
Currently both deepfake detection and semantic analysis use **BusterX** (Qwen2.5-VL-7B, 7B parameters). However, we have access to a much larger model **Qwen3-VL-30B** (30B parameters) that could provide significantly better semantic understanding.
Sistemul livrat folosește **BusterX** (Qwen2.5-VL-7B) atât pentru deepfake cât și pentru cadrele
din analiza semantică, cu agregare text pe **Qwen3.5**. Ca direcție de cercetare, s-ar putea
evalua un model vision mai mare pentru partea semantică (dacă va fi disponibil în viitor).
### Hypothesis
Semantic analysis (content understanding, scene description, narrative) would benefit from the larger Qwen3-VL-30B model, while deepfake detection should continue using the specialized BusterX model.
Analiza semantică (descriere scenă, narativ) ar putea beneficia de un model vision mai mare,
în timp ce detecția deepfake rămâne pe modelul specializat BusterX.
---
@ -41,7 +50,7 @@ VIDEO_ANALYSIS_VLLM_MODEL=qwen3-vl # 30B parameters
cd /home/vasi/ml-projects/modules/video-analysis/deploy
# Test semantic analysis with BusterX
curl -X POST http://localhost:8007/analyze/video/semantic \
curl -X POST http://localhost:54600/analyze/video/semantic \
-F "file=@test_video_60s.mp4" \
-F "chunk_duration_s=10.0" \
-F "frames_per_chunk=24" \
@ -91,7 +100,7 @@ docker compose up -d video-analysis-api
sleep 10
# Test with same video
curl -X POST http://localhost:8007/analyze/video/semantic \
curl -X POST http://localhost:54600/analyze/video/semantic \
-F "file=@test_video_60s.mp4" \
-F "chunk_duration_s=10.0" \
-F "frames_per_chunk=24" \

Some files were not shown because too many files have changed in this diff Show more