clossers/didi-lot1-ai
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
didi-lot1-ai/ai_platform/modules/domain_check/README.md

20 KiB
Raw Blame History

Domain Check API - Anti-Fake News Platform

Sistema de verificare domenii pentru detectarea dezinformarii prin analiza vechimii domeniilor si riscurilor asociate.

🚀 Quick Start - API Usage

Endpoints Disponibile

Base URL: http://domain-check-api:11000

Endpoint principal: POST /api/v1/check/check

Apel Complet - Toate Datele Posibile

Pentru a obține toate datele disponibile despre un domeniu:

curl -X POST "http://domain-check-api:11000/api/v1/check/check" \
  -H "Content-Type: application/json" \
  -d '{
    "domain": "example.com",
    "check_options": {
      "whois": true,
      "dns": true,
      "ssl": true,
      "ip_intelligence": true,
      "http_analysis": true,
      "blacklist": true,
      "port_scan": true,
      "subdomains": true,
      "force_refresh": false
    }
  }'

Răspuns: JSON cu:

  • whois - Date WHOIS/RDAP complete (vârstă domeniu, registrar, name servers, DNSSEC)
  • dns - Toate recordurile DNS (A, AAAA, MX, TXT, NS, CNAME, SOA)
  • ssl - Certificate SSL/TLS details (issuer, validity, chain, self-signed status)
  • ip_intelligence - Geolocation, ASN, ISP, reverse DNS pentru toate IP-urile
  • http_analysis - HTTP headers, security headers, redirects, technologies detected
  • blacklist - Verificare în DNSBL (spam, malware, phishing blacklists)
  • port_scan - Port scanning TCP common (22, 80, 443, 8080, etc.)
  • subdomains - Subdomain enumeration (www, mail, ftp, api, etc.)
  • risk_score - Scor de risc calculat (0-100) cu breakdown pe categorii

Apel Rapid - Date Esențiale

Pentru verificări rapide (doar WHOIS + DNS + SSL):

curl -X POST "http://domain-check-api:11000/api/v1/check/check" \
  -H "Content-Type: application/json" \
  -d '{
    "domain": "example.com",
    "check_options": {
      "whois": true,
      "dns": true,
      "ssl": true
    }
  }'

Health Check

curl http://domain-check-api:11000/health

Răspuns:

{
  "status": "healthy",
  "database": "connected",
  "redis": "connected",
  "environment": "production",
  "version": "v1"
}

🌐 Integrare ca Serviciu (Kong API Gateway / Microservices)

Configurare ca Backend Service

API-ul poate fi folosit ca backend service pentru Kong, NGINX, Traefik sau alte API gateways.

1. Configurare Kong Gateway

# Adaugă service
curl -X POST http://kong-admin:8001/services \
  --data name=domain-check-api \
  --data url=http://domain-check-api:11000

# Adaugă route
curl -X POST http://kong-admin:8001/services/domain-check-api/routes \
  --data paths[]=/domain-check \
  --data strip_path=false

# Activează rate limiting
curl -X POST http://kong-admin:8001/services/domain-check-api/plugins \
  --data name=rate-limiting \
  --data config.minute=100 \
  --data config.hour=1000

Acum API-ul e disponibil la: http://your-kong-gateway/domain-check/api/v1/check/check

2. Configurare NGINX Reverse Proxy

upstream domain_check_backend {
    server domain-check-api:11000;
}

server {
    listen 80;
    server_name api.example.com;

    location /domain-check/ {
        proxy_pass http://domain_check_backend/;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        
        # Timeouts pentru operații lungi (port scan, subdomain enum)
        proxy_read_timeout 300s;
        proxy_connect_timeout 10s;
    }
}

3. Docker Swarm / Kubernetes Service

Docker Swarm:

version: '3.8'
services:
  domain-check:
    image: didiai-domain-check:latest
    deploy:
      replicas: 3
      update_config:
        parallelism: 1
        delay: 10s
    environment:
      DATABASE_URL: postgresql://user:pass@db-cluster:5432/domain_check
      REDIS_URL: redis://redis-cluster:6379/0
    networks:
      - backend

Kubernetes Service:

apiVersion: v1
kind: Service
metadata:
  name: domain-check-api
spec:
  selector:
    app: domain-check
  ports:
    - protocol: TCP
      port: 80
      targetPort: 51000
  type: ClusterIP

4. Environment Variables pentru External DB

Toate parametrii DB sunt în .env și pot fi schimbați pentru a folosi un cluster PostgreSQL extern:

# .env - Configurare pentru DB Cluster extern
DB_HOST=postgres-cluster.example.com
DB_PORT=5432
DB_USER=domain_check_user
DB_PASSWORD=secure_password_here
DB_NAME=domain_check

# Redis Cluster extern
REDIS_HOST=redis-cluster.example.com
REDIS_PORT=6379
REDIS_PASSWORD=redis_password_here

API-ul se va conecta automat la DB-ul extern specificat în variabilele de mediu.

📊 Database Schema - Structura Tabelelor

API-ul folosește PostgreSQL cu următoarele tabele:

1. domains - Domenii Verificate

Tabela principală cu toate domeniile procesate.

Coloană Tip Descriere
id UUID Primary key
domain VARCHAR(255) Domeniul principal (ex: example din example.com)
subdomain VARCHAR(255) Subdomenul (ex: www din www.example.com)
tld VARCHAR(50) Top Level Domain (ex: com, org, ro)
full_domain VARCHAR(255) Domeniul complet (UNIQUE)
first_seen_at TIMESTAMP Prima verificare
last_checked_at TIMESTAMP Ultima verificare
check_count INTEGER Număr total verificări
is_active BOOLEAN Status activ/inactiv
created_at TIMESTAMP Data creării înregistrării
updated_at TIMESTAMP Data ultimei actualizări

Indexes: full_domain (UNIQUE), domain, tld, last_checked_at, is_active

2. whois_records - Date WHOIS/RDAP

Stochează informații WHOIS pentru fiecare domeniu.

Coloană Tip Descriere
id UUID Primary key
domain_id UUID Foreign key → domains(id)
creation_date TIMESTAMP Data creării domeniului
expiration_date TIMESTAMP Data expirării
updated_date TIMESTAMP Ultima actualizare WHOIS
registrar VARCHAR(255) Registrar (ex: GoDaddy, Namecheap)
registrar_url VARCHAR(500) URL registrar
registrant_org VARCHAR(255) Organizația proprietarului
registrant_country VARCHAR(2) Țara (cod ISO)
admin_email VARCHAR(255) Email administrator
name_servers TEXT[] Array cu name servers
status TEXT[] Array cu statusuri domeniu
dnssec BOOLEAN DNSSEC activat/nu
raw_whois_data JSONB Date WHOIS raw (JSON)
data_source VARCHAR(50) Sursa datelor (rdap, whoxy, manual)
fetched_at TIMESTAMP Când au fost preluate datele

Indexes: domain_id, creation_date, registrar, data_source, fetched_at

3. dns_records - Înregistrări DNS

Toate recordurile DNS pentru fiecare domeniu.

Coloană Tip Descriere
id UUID Primary key
domain_id UUID Foreign key → domains(id)
record_type VARCHAR(10) Tip record (A, AAAA, MX, TXT, NS, CNAME, SOA)
record_value TEXT Valoarea recordului
ttl INTEGER Time To Live (secunde)
priority INTEGER Prioritate (pentru MX records)
fetched_at TIMESTAMP Când a fost preluat recordul

Indexes: domain_id, (domain_id, record_type), record_type, fetched_at

4. ssl_certificates - Certificate SSL/TLS

Informații despre certificatele SSL ale domeniilor.

Coloană Tip Descriere
id UUID Primary key
domain_id UUID Foreign key → domains(id)
issuer VARCHAR(255) Emitent certificat (ex: Let's Encrypt)
subject VARCHAR(255) Subiect certificat
valid_from TIMESTAMP Valabil de la
valid_until TIMESTAMP Valabil până la
serial_number VARCHAR(255) Număr serial certificat
signature_algorithm VARCHAR(100) Algoritm semnătură (ex: SHA256)
key_size INTEGER Mărime cheie (2048, 4096 bits)
is_wildcard BOOLEAN Certificat wildcard (*.example.com)
is_self_signed BOOLEAN Self-signed certificate
is_valid BOOLEAN Certificat valid
certificate_chain JSONB Lanțul complet de certificate (JSON)
fetched_at TIMESTAMP Când a fost verificat certificatul

Indexes: domain_id, valid_until, is_valid, fetched_at

Relații între Tabele

domains (1) ─── (N) whois_records
              ├─ (N) dns_records
              └─ (N) ssl_certificates

Toate relațiile au ON DELETE CASCADE - ștergerea unui domeniu șterge automat toate datele asociate.

Migrare la DB Cluster Extern

Pentru a crea schema în cluster PostgreSQL nou:

# 1. Export schema
docker exec domain_check_postgres pg_dump -U dns_admin -d domain_check --schema-only > schema.sql

# 2. Import în cluster nou
psql -h postgres-cluster.example.com -U admin -d domain_check < schema.sql

# 3. Actualizează .env cu noile credentials
DB_HOST=postgres-cluster.example.com
DB_PORT=5432
DB_USER=domain_check_user
DB_PASSWORD=new_secure_password
DB_NAME=domain_check

# 4. Restart API
docker compose restart domain_check_api

Sau folosește direct script-ul de inițializare:

psql -h postgres-cluster.example.com -U admin -d domain_check < init-scripts/01-init-db.sql

Caracteristici

  • Verificare WHOIS/RDAP complet
  • Integrare Whoxy API
  • DNS records checking (A, AAAA, MX, TXT, NS, CNAME, SOA)
  • SSL certificate validation
  • IP Intelligence (geolocation, ASN, reverse DNS)
  • HTTP Security Analysis
  • DNSBL Blacklist checking
  • Port scanning
  • Subdomain enumeration
  • Risk scoring engine avansat
  • PostgreSQL + Redis caching
  • Swagger UI + ReDoc documentation
  • Docker Compose setup complet
  • GitLab CI/CD pipeline cu SAST

Deployment pe Orice Server

Metoda 1: Script Automat (Recomandat)

# Descarca si ruleaza scriptul de deployment
curl -sSL (modul domain_check al platformei DIDI) -o deploy.sh
chmod +x deploy.sh
./deploy.sh install

Scriptul va:

  • Verifica prerequisitele (Docker, Docker Compose, Git)
  • Clona repository-ul
  • Genera fisierul .env cu parole securizate
  • Construi si porni containerele

Metoda 2: Manual

# 1. Cloneaza repository-ul
git clone <didi-lot1-ai>/ai_platform/modules/domain_check
cd domain-check

# 2. Copiaza si editeaza configuratia
cp .env.example .env
nano .env  # sau vim, sau orice editor

# 3. Porneste serviciile
docker compose up -d --build

# 4. Verifica statusul
docker compose ps
curl http://domain-check-api:11000/health

Metoda 3: Din GitLab Container Registry

# Login la registry (daca e necesar)
docker login <registry-didi>

# Pull imaginea
docker pull didiai-domain-check:latest

# Cloneaza doar fisierele de configurare
git clone <didi-lot1-ai>/ai_platform/modules/domain_check
cd domain-check

# Editeaza .env
nano .env

# Porneste cu imaginea din registry
docker compose up -d

Configurare pentru Serverul Tau

Fisierul .env

Toate setarile sunt in .env. Editeaza pentru mediul tau:

# ===========================================
# PORTURI - Modifica dupa nevoie
# ===========================================
# Schema: Dev=5xxxx, Prod=1xxxx
# x1xxx = API/Gateway
# x20xx = PostgreSQL
# x23xx = Redis

API_PORT=51000              # Portul API-ului (schimba la 11000 pentru prod)
DB_EXTERNAL_PORT=52000      # Port extern PostgreSQL (12000 pentru prod)
REDIS_EXTERNAL_PORT=52300   # Port extern Redis (12300 pentru prod)

# ===========================================
# DATABASE - Genereaza parole noi!
# ===========================================
DB_PASSWORD=SCHIMBA_ACEASTA_PAROLA
DB_USER=dns_admin
DB_NAME=domain_check

# ===========================================
# API KEYS - Adauga cheile tale
# ===========================================
WHOXY_API_KEY=cheia_ta_whoxy
VIRUSTOTAL_API_KEY=cheia_ta_virustotal

# ===========================================
# SECURITATE - OBLIGATORIU pentru productie!
# ===========================================
SECRET_KEY=genereaza_un_string_random_de_32_caractere
FLASK_ENV=production        # development sau production
DEBUG=False                 # True doar pentru development

Setare Hostname/DNS

API-ul va fi accesibil prin:

  • IP direct: http://<IP_SERVER>:<API_PORT>/
  • Hostname: http://<hostname>:<API_PORT>/

Pentru a seta un hostname custom:

# 1. Pe DNS server (daca ai control)
check-dns.example.com  A  <IP_SERVER>

# 2. Sau in /etc/hosts pe clienti
echo "<IP_SERVER>  check-dns.example.com" >> /etc/hosts

# 3. Sau cu reverse proxy (nginx/traefik)
# - Configureaza proxy pass catre domain-check-api:<API_PORT>

Porturi Utilizate

Serviciu Port Default (Dev) Port Productie Variabila
API Flask 51000 11000 API_PORT
PostgreSQL 52000 12000 DB_EXTERNAL_PORT
Redis 52300 12300 REDIS_EXTERNAL_PORT

Porturile pot fi schimbate in .env - nu este nevoie de modificari in alte fisiere.

Firewall

Deschide doar portul API pentru acces extern:

# UFW (Ubuntu/Debian)
sudo ufw allow 51000/tcp

# firewalld (CentOS/RHEL)
sudo firewall-cmd --add-port=51000/tcp --permanent
sudo firewall-cmd --reload

# iptables
sudo iptables -A INPUT -p tcp --dport 51000 -j ACCEPT

PostgreSQL si Redis nu trebuie expuse public (sunt doar pentru comunicatie interna).

Structura Proiectului

domain-check/
├── api/                        # Flask API
│   ├── app/
│   │   ├── models/            # SQLAlchemy models
│   │   ├── routes/            # API endpoints
│   │   ├── services/          # Business logic (WHOIS, DNS, SSL, etc.)
│   │   ├── utils/             # Helper functions
│   │   ├── tasks/             # Celery tasks
│   │   ├── static/            # Dashboard HTML/CSS/JS
│   │   ├── config.py          # Configuration
│   │   └── __init__.py        # App factory
│   ├── Dockerfile
│   ├── requirements.txt
│   └── run.py                 # Entry point
├── init-scripts/              # PostgreSQL init scripts
│   └── 01-init-db.sql
├── docker-compose.yml         # Docker orchestration
├── .gitlab-ci.yml             # CI/CD pipeline
├── deploy.sh                  # Deployment script
├── .env                       # Environment variables
├── use_api.md                 # API documentation
└── README.md                  # This file

Servicii Docker

Container Descriere Health Check
domain_check_api Flask REST API + Dashboard /health
domain_check_postgres PostgreSQL 15 database pg_isready
domain_check_redis Redis cache redis-cli ping
domain_check_worker Celery background worker celery inspect ping

Utilizare API

Interfata Web (Dashboard)

Deschide in browser: http://<server>:<API_PORT>/

Verificare Domain (curl)

# Check complet
curl -X POST "http://domain-check-api:11000/api/v1/check/check" \
  -H "Content-Type: application/json" \
  -d '{
    "domain": "example.com",
    "check_options": {
      "whois": true,
      "dns": true,
      "ssl": true,
      "ip_intelligence": true,
      "http_analysis": true,
      "blacklist": true,
      "port_scan": true,
      "subdomains": true
    }
  }'

Health Check

curl http://domain-check-api:11000/health

Documentatie Swagger

  • Swagger UI: http://<server>:<API_PORT>/docs
  • ReDoc: http://<server>:<API_PORT>/redoc

Pentru documentatie API completa, vezi use_api.md.

CI/CD Pipeline

Pipeline-ul GitLab include:

Stages

  1. lint - Code quality (flake8, black, isort, hadolint)
  2. security - SAST scanning (bandit, safety, trivy)
  3. test - Unit tests
  4. build - Docker build & push to registry
  5. release - Create deployment bundle
  6. deploy - Deploy to server via SSH

Container Registry

Imaginile sunt publicate la:

didiai-domain-check:latest
didiai-domain-check:<branch>

Configurare Deploy Automat

Adauga in GitLab CI/CD Variables:

Variable Descriere
DEPLOY_HOST IP sau hostname server
DEPLOY_USER User SSH (ex: admin365)
SSH_PRIVATE_KEY Cheia privata SSH
SSH_KNOWN_HOSTS Output din ssh-keyscan <server>

Comenzi Utile

Deploy Script

./deploy.sh install   # Instalare fresh
./deploy.sh update    # Update la ultima versiune
./deploy.sh status    # Verifica statusul
./deploy.sh logs      # Vezi toate logs
./deploy.sh logs domain_check_api  # Logs doar pentru API
./deploy.sh start     # Porneste serviciile
./deploy.sh stop      # Opreste serviciile
./deploy.sh restart   # Restart servicii

Docker Direct

# Start/Stop
docker compose up -d
docker compose down

# Rebuild
docker compose up -d --build

# Logs
docker compose logs -f domain_check_api

# Shell in container
docker exec -it domain_check_api bash

# Database access
docker exec -it domain_check_postgres psql -U dns_admin -d domain_check

Troubleshooting

Container nu porneste

# Verifica logs
docker compose logs domain_check_api

# Verifica network
docker network ls | grep dns-network

# Recreate containers
docker compose down && docker compose up -d --build

Database connection error

# Verifica ca PostgreSQL e healthy
docker compose ps domain_check_postgres

# Verifica health
docker exec -it domain_check_postgres pg_isready -U dns_admin

# Restart database
docker compose restart domain_check_postgres

API returns 500 error

# Check logs
docker compose logs -f domain_check_api

# Verify database schema
docker exec -it domain_check_postgres psql -U dns_admin -d domain_check -c "\dt"

Port deja in uz

# Gaseste ce foloseste portul
sudo lsof -i :51000

# Schimba portul in .env
API_PORT=51001  # sau alt port liber

Securitate - Checklist Productie

  • Schimba toate parolele default in .env
  • Seteaza FLASK_ENV=production
  • Seteaza DEBUG=False
  • Genereaza SECRET_KEY random (32+ caractere)
  • Configureaza firewall (doar port API public)
  • Setup HTTPS cu reverse proxy (nginx/traefik)
  • Backup automat PostgreSQL
  • Monitorizare logs

Variabile de Mediu Complete

Variabila Default Descriere
API_PORT 51000 Port pentru Flask API
API_HOST 0.0.0.0 Bind address
API_VERSION v1 API version prefix
DB_HOST domain_check_postgres PostgreSQL hostname (container)
DB_PORT 5432 PostgreSQL port intern
DB_EXTERNAL_PORT 52000 PostgreSQL port extern
DB_USER dns_admin Database user
DB_PASSWORD - Database password
DB_NAME domain_check Database name
REDIS_HOST domain_check_redis Redis hostname (container)
REDIS_PORT 6379 Redis port intern
REDIS_EXTERNAL_PORT 52300 Redis port extern
REDIS_DB 0 Redis database number
WHOXY_API_KEY - Whoxy API key
VIRUSTOTAL_API_KEY - VirusTotal API key
FLASK_ENV development Flask environment
DEBUG True Debug mode
SECRET_KEY - Flask secret key
LOG_LEVEL INFO Logging level
REDIS_TTL_HOT 21600 Cache TTL hot (6h)
REDIS_TTL_WARM 86400 Cache TTL warm (24h)
REDIS_TTL_COLD 604800 Cache TTL cold (7d)

License

MIT License - Free for use in anti-fake news projects

Built for fighting disinformation

Last Updated: 2026-02-04