LOT 1 - Optimizare script build -Instalare mono comanda

This commit is contained in:
Dezvoltari Evotech 2026-06-27 06:42:02 -07:00
parent 5380c3fc63
commit 42ff22bf85
127 changed files with 16163 additions and 532 deletions

View file

@ -0,0 +1,740 @@
# Domain Check API - Anti-Fake News Platform
Sistema de verificare domenii pentru detectarea dezinformarii prin analiza vechimii domeniilor si riscurilor asociate.
## 🚀 Quick Start - API Usage
### Endpoints Disponibile
**Base URL:** `http://domain-check-api:11000`
**Endpoint principal:** `POST /api/v1/check/check`
### Apel Complet - Toate Datele Posibile
Pentru a obține **toate datele disponibile** despre un domeniu:
```bash
curl -X POST "http://domain-check-api:11000/api/v1/check/check" \
-H "Content-Type: application/json" \
-d '{
"domain": "example.com",
"check_options": {
"whois": true,
"dns": true,
"ssl": true,
"ip_intelligence": true,
"http_analysis": true,
"blacklist": true,
"port_scan": true,
"subdomains": true,
"force_refresh": false
}
}'
```
**Răspuns:** JSON cu:
- `whois` - Date WHOIS/RDAP complete (vârstă domeniu, registrar, name servers, DNSSEC)
- `dns` - Toate recordurile DNS (A, AAAA, MX, TXT, NS, CNAME, SOA)
- `ssl` - Certificate SSL/TLS details (issuer, validity, chain, self-signed status)
- `ip_intelligence` - Geolocation, ASN, ISP, reverse DNS pentru toate IP-urile
- `http_analysis` - HTTP headers, security headers, redirects, technologies detected
- `blacklist` - Verificare în DNSBL (spam, malware, phishing blacklists)
- `port_scan` - Port scanning TCP common (22, 80, 443, 8080, etc.)
- `subdomains` - Subdomain enumeration (www, mail, ftp, api, etc.)
- `risk_score` - Scor de risc calculat (0-100) cu breakdown pe categorii
### Apel Rapid - Date Esențiale
Pentru verificări rapide (doar WHOIS + DNS + SSL):
```bash
curl -X POST "http://domain-check-api:11000/api/v1/check/check" \
-H "Content-Type: application/json" \
-d '{
"domain": "example.com",
"check_options": {
"whois": true,
"dns": true,
"ssl": true
}
}'
```
### Health Check
```bash
curl http://domain-check-api:11000/health
```
Răspuns:
```json
{
"status": "healthy",
"database": "connected",
"redis": "connected",
"environment": "production",
"version": "v1"
}
```
---
## 🌐 Integrare ca Serviciu (Kong API Gateway / Microservices)
### Configurare ca Backend Service
API-ul poate fi folosit ca **backend service** pentru Kong, NGINX, Traefik sau alte API gateways.
#### 1. Configurare Kong Gateway
```bash
# Adaugă service
curl -X POST http://kong-admin:8001/services \
--data name=domain-check-api \
--data url=http://domain-check-api:11000
# Adaugă route
curl -X POST http://kong-admin:8001/services/domain-check-api/routes \
--data paths[]=/domain-check \
--data strip_path=false
# Activează rate limiting
curl -X POST http://kong-admin:8001/services/domain-check-api/plugins \
--data name=rate-limiting \
--data config.minute=100 \
--data config.hour=1000
```
Acum API-ul e disponibil la: `http://your-kong-gateway/domain-check/api/v1/check/check`
#### 2. Configurare NGINX Reverse Proxy
```nginx
upstream domain_check_backend {
server domain-check-api:11000;
}
server {
listen 80;
server_name api.example.com;
location /domain-check/ {
proxy_pass http://domain_check_backend/;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
# Timeouts pentru operații lungi (port scan, subdomain enum)
proxy_read_timeout 300s;
proxy_connect_timeout 10s;
}
}
```
#### 3. Docker Swarm / Kubernetes Service
**Docker Swarm:**
```yaml
version: '3.8'
services:
domain-check:
image: didiai-domain-check:latest
deploy:
replicas: 3
update_config:
parallelism: 1
delay: 10s
environment:
DATABASE_URL: postgresql://user:pass@db-cluster:5432/domain_check
REDIS_URL: redis://redis-cluster:6379/0
networks:
- backend
```
**Kubernetes Service:**
```yaml
apiVersion: v1
kind: Service
metadata:
name: domain-check-api
spec:
selector:
app: domain-check
ports:
- protocol: TCP
port: 80
targetPort: 51000
type: ClusterIP
```
#### 4. Environment Variables pentru External DB
Toate parametrii DB sunt în `.env` și pot fi schimbați pentru a folosi un **cluster PostgreSQL extern**:
```bash
# .env - Configurare pentru DB Cluster extern
DB_HOST=postgres-cluster.example.com
DB_PORT=5432
DB_USER=domain_check_user
DB_PASSWORD=secure_password_here
DB_NAME=domain_check
# Redis Cluster extern
REDIS_HOST=redis-cluster.example.com
REDIS_PORT=6379
REDIS_PASSWORD=redis_password_here
```
API-ul se va conecta automat la DB-ul extern specificat în variabilele de mediu.
---
## 📊 Database Schema - Structura Tabelelor
API-ul folosește PostgreSQL cu următoarele tabele:
### 1. `domains` - Domenii Verificate
Tabela principală cu toate domeniile procesate.
| Coloană | Tip | Descriere |
|---------|-----|-----------|
| `id` | UUID | Primary key |
| `domain` | VARCHAR(255) | Domeniul principal (ex: `example` din `example.com`) |
| `subdomain` | VARCHAR(255) | Subdomenul (ex: `www` din `www.example.com`) |
| `tld` | VARCHAR(50) | Top Level Domain (ex: `com`, `org`, `ro`) |
| `full_domain` | VARCHAR(255) | Domeniul complet (UNIQUE) |
| `first_seen_at` | TIMESTAMP | Prima verificare |
| `last_checked_at` | TIMESTAMP | Ultima verificare |
| `check_count` | INTEGER | Număr total verificări |
| `is_active` | BOOLEAN | Status activ/inactiv |
| `created_at` | TIMESTAMP | Data creării înregistrării |
| `updated_at` | TIMESTAMP | Data ultimei actualizări |
**Indexes:** `full_domain` (UNIQUE), `domain`, `tld`, `last_checked_at`, `is_active`
### 2. `whois_records` - Date WHOIS/RDAP
Stochează informații WHOIS pentru fiecare domeniu.
| Coloană | Tip | Descriere |
|---------|-----|-----------|
| `id` | UUID | Primary key |
| `domain_id` | UUID | Foreign key → `domains(id)` |
| `creation_date` | TIMESTAMP | Data creării domeniului |
| `expiration_date` | TIMESTAMP | Data expirării |
| `updated_date` | TIMESTAMP | Ultima actualizare WHOIS |
| `registrar` | VARCHAR(255) | Registrar (ex: GoDaddy, Namecheap) |
| `registrar_url` | VARCHAR(500) | URL registrar |
| `registrant_org` | VARCHAR(255) | Organizația proprietarului |
| `registrant_country` | VARCHAR(2) | Țara (cod ISO) |
| `admin_email` | VARCHAR(255) | Email administrator |
| `name_servers` | TEXT[] | Array cu name servers |
| `status` | TEXT[] | Array cu statusuri domeniu |
| `dnssec` | BOOLEAN | DNSSEC activat/nu |
| `raw_whois_data` | JSONB | Date WHOIS raw (JSON) |
| `data_source` | VARCHAR(50) | Sursa datelor (`rdap`, `whoxy`, `manual`) |
| `fetched_at` | TIMESTAMP | Când au fost preluate datele |
**Indexes:** `domain_id`, `creation_date`, `registrar`, `data_source`, `fetched_at`
### 3. `dns_records` - Înregistrări DNS
Toate recordurile DNS pentru fiecare domeniu.
| Coloană | Tip | Descriere |
|---------|-----|-----------|
| `id` | UUID | Primary key |
| `domain_id` | UUID | Foreign key → `domains(id)` |
| `record_type` | VARCHAR(10) | Tip record (`A`, `AAAA`, `MX`, `TXT`, `NS`, `CNAME`, `SOA`) |
| `record_value` | TEXT | Valoarea recordului |
| `ttl` | INTEGER | Time To Live (secunde) |
| `priority` | INTEGER | Prioritate (pentru MX records) |
| `fetched_at` | TIMESTAMP | Când a fost preluat recordul |
**Indexes:** `domain_id`, `(domain_id, record_type)`, `record_type`, `fetched_at`
### 4. `ssl_certificates` - Certificate SSL/TLS
Informații despre certificatele SSL ale domeniilor.
| Coloană | Tip | Descriere |
|---------|-----|-----------|
| `id` | UUID | Primary key |
| `domain_id` | UUID | Foreign key → `domains(id)` |
| `issuer` | VARCHAR(255) | Emitent certificat (ex: Let's Encrypt) |
| `subject` | VARCHAR(255) | Subiect certificat |
| `valid_from` | TIMESTAMP | Valabil de la |
| `valid_until` | TIMESTAMP | Valabil până la |
| `serial_number` | VARCHAR(255) | Număr serial certificat |
| `signature_algorithm` | VARCHAR(100) | Algoritm semnătură (ex: SHA256) |
| `key_size` | INTEGER | Mărime cheie (2048, 4096 bits) |
| `is_wildcard` | BOOLEAN | Certificat wildcard (`*.example.com`) |
| `is_self_signed` | BOOLEAN | Self-signed certificate |
| `is_valid` | BOOLEAN | Certificat valid |
| `certificate_chain` | JSONB | Lanțul complet de certificate (JSON) |
| `fetched_at` | TIMESTAMP | Când a fost verificat certificatul |
**Indexes:** `domain_id`, `valid_until`, `is_valid`, `fetched_at`
### Relații între Tabele
```
domains (1) ─── (N) whois_records
├─ (N) dns_records
└─ (N) ssl_certificates
```
Toate relațiile au `ON DELETE CASCADE` - ștergerea unui domeniu șterge automat toate datele asociate.
### Migrare la DB Cluster Extern
Pentru a crea schema în **cluster PostgreSQL nou**:
```bash
# 1. Export schema
docker exec domain_check_postgres pg_dump -U dns_admin -d domain_check --schema-only > schema.sql
# 2. Import în cluster nou
psql -h postgres-cluster.example.com -U admin -d domain_check < schema.sql
# 3. Actualizează .env cu noile credentials
DB_HOST=postgres-cluster.example.com
DB_PORT=5432
DB_USER=domain_check_user
DB_PASSWORD=new_secure_password
DB_NAME=domain_check
# 4. Restart API
docker compose restart domain_check_api
```
Sau folosește direct script-ul de inițializare:
```bash
psql -h postgres-cluster.example.com -U admin -d domain_check < init-scripts/01-init-db.sql
```
---
## Caracteristici
- Verificare WHOIS/RDAP complet
- Integrare Whoxy API
- DNS records checking (A, AAAA, MX, TXT, NS, CNAME, SOA)
- SSL certificate validation
- IP Intelligence (geolocation, ASN, reverse DNS)
- HTTP Security Analysis
- DNSBL Blacklist checking
- Port scanning
- Subdomain enumeration
- Risk scoring engine avansat
- PostgreSQL + Redis caching
- Swagger UI + ReDoc documentation
- Docker Compose setup complet
- GitLab CI/CD pipeline cu SAST
---
## Deployment pe Orice Server
### Metoda 1: Script Automat (Recomandat)
```bash
# Descarca si ruleaza scriptul de deployment
curl -sSL (modul domain_check al platformei DIDI) -o deploy.sh
chmod +x deploy.sh
./deploy.sh install
```
Scriptul va:
- Verifica prerequisitele (Docker, Docker Compose, Git)
- Clona repository-ul
- Genera fisierul `.env` cu parole securizate
- Construi si porni containerele
### Metoda 2: Manual
```bash
# 1. Cloneaza repository-ul
git clone <didi-lot1-ai>/ai_platform/modules/domain_check
cd domain-check
# 2. Copiaza si editeaza configuratia
cp .env.example .env
nano .env # sau vim, sau orice editor
# 3. Porneste serviciile
docker compose up -d --build
# 4. Verifica statusul
docker compose ps
curl http://domain-check-api:11000/health
```
### Metoda 3: Din GitLab Container Registry
```bash
# Login la registry (daca e necesar)
docker login <registry-didi>
# Pull imaginea
docker pull didiai-domain-check:latest
# Cloneaza doar fisierele de configurare
git clone <didi-lot1-ai>/ai_platform/modules/domain_check
cd domain-check
# Editeaza .env
nano .env
# Porneste cu imaginea din registry
docker compose up -d
```
---
## Configurare pentru Serverul Tau
### Fisierul .env
Toate setarile sunt in `.env`. Editeaza pentru mediul tau:
```bash
# ===========================================
# PORTURI - Modifica dupa nevoie
# ===========================================
# Schema: Dev=5xxxx, Prod=1xxxx
# x1xxx = API/Gateway
# x20xx = PostgreSQL
# x23xx = Redis
API_PORT=51000 # Portul API-ului (schimba la 11000 pentru prod)
DB_EXTERNAL_PORT=52000 # Port extern PostgreSQL (12000 pentru prod)
REDIS_EXTERNAL_PORT=52300 # Port extern Redis (12300 pentru prod)
# ===========================================
# DATABASE - Genereaza parole noi!
# ===========================================
DB_PASSWORD=SCHIMBA_ACEASTA_PAROLA
DB_USER=dns_admin
DB_NAME=domain_check
# ===========================================
# API KEYS - Adauga cheile tale
# ===========================================
WHOXY_API_KEY=cheia_ta_whoxy
VIRUSTOTAL_API_KEY=cheia_ta_virustotal
# ===========================================
# SECURITATE - OBLIGATORIU pentru productie!
# ===========================================
SECRET_KEY=genereaza_un_string_random_de_32_caractere
FLASK_ENV=production # development sau production
DEBUG=False # True doar pentru development
```
### Setare Hostname/DNS
API-ul va fi accesibil prin:
- **IP direct**: `http://<IP_SERVER>:<API_PORT>/`
- **Hostname**: `http://<hostname>:<API_PORT>/`
Pentru a seta un hostname custom:
```bash
# 1. Pe DNS server (daca ai control)
check-dns.example.com A <IP_SERVER>
# 2. Sau in /etc/hosts pe clienti
echo "<IP_SERVER> check-dns.example.com" >> /etc/hosts
# 3. Sau cu reverse proxy (nginx/traefik)
# - Configureaza proxy pass catre domain-check-api:<API_PORT>
```
### Porturi Utilizate
| Serviciu | Port Default (Dev) | Port Productie | Variabila |
|----------|-------------------|----------------|-----------|
| API Flask | 51000 | 11000 | `API_PORT` |
| PostgreSQL | 52000 | 12000 | `DB_EXTERNAL_PORT` |
| Redis | 52300 | 12300 | `REDIS_EXTERNAL_PORT` |
Porturile pot fi schimbate in `.env` - nu este nevoie de modificari in alte fisiere.
### Firewall
Deschide doar portul API pentru acces extern:
```bash
# UFW (Ubuntu/Debian)
sudo ufw allow 51000/tcp
# firewalld (CentOS/RHEL)
sudo firewall-cmd --add-port=51000/tcp --permanent
sudo firewall-cmd --reload
# iptables
sudo iptables -A INPUT -p tcp --dport 51000 -j ACCEPT
```
PostgreSQL si Redis nu trebuie expuse public (sunt doar pentru comunicatie interna).
---
## Structura Proiectului
```
domain-check/
├── api/ # Flask API
│ ├── app/
│ │ ├── models/ # SQLAlchemy models
│ │ ├── routes/ # API endpoints
│ │ ├── services/ # Business logic (WHOIS, DNS, SSL, etc.)
│ │ ├── utils/ # Helper functions
│ │ ├── tasks/ # Celery tasks
│ │ ├── static/ # Dashboard HTML/CSS/JS
│ │ ├── config.py # Configuration
│ │ └── __init__.py # App factory
│ ├── Dockerfile
│ ├── requirements.txt
│ └── run.py # Entry point
├── init-scripts/ # PostgreSQL init scripts
│ └── 01-init-db.sql
├── docker-compose.yml # Docker orchestration
├── .gitlab-ci.yml # CI/CD pipeline
├── deploy.sh # Deployment script
├── .env # Environment variables
├── use_api.md # API documentation
└── README.md # This file
```
---
## Servicii Docker
| Container | Descriere | Health Check |
|-----------|-----------|--------------|
| `domain_check_api` | Flask REST API + Dashboard | `/health` |
| `domain_check_postgres` | PostgreSQL 15 database | `pg_isready` |
| `domain_check_redis` | Redis cache | `redis-cli ping` |
| `domain_check_worker` | Celery background worker | `celery inspect ping` |
---
## Utilizare API
### Interfata Web (Dashboard)
Deschide in browser: `http://<server>:<API_PORT>/`
### Verificare Domain (curl)
```bash
# Check complet
curl -X POST "http://domain-check-api:11000/api/v1/check/check" \
-H "Content-Type: application/json" \
-d '{
"domain": "example.com",
"check_options": {
"whois": true,
"dns": true,
"ssl": true,
"ip_intelligence": true,
"http_analysis": true,
"blacklist": true,
"port_scan": true,
"subdomains": true
}
}'
```
### Health Check
```bash
curl http://domain-check-api:11000/health
```
### Documentatie Swagger
- Swagger UI: `http://<server>:<API_PORT>/docs`
- ReDoc: `http://<server>:<API_PORT>/redoc`
Pentru documentatie API completa, vezi [use_api.md](use_api.md).
---
## CI/CD Pipeline
Pipeline-ul GitLab include:
### Stages
1. **lint** - Code quality (flake8, black, isort, hadolint)
2. **security** - SAST scanning (bandit, safety, trivy)
3. **test** - Unit tests
4. **build** - Docker build & push to registry
5. **release** - Create deployment bundle
6. **deploy** - Deploy to server via SSH
### Container Registry
Imaginile sunt publicate la:
```
didiai-domain-check:latest
didiai-domain-check:<branch>
```
### Configurare Deploy Automat
Adauga in GitLab CI/CD Variables:
| Variable | Descriere |
|----------|-----------|
| `DEPLOY_HOST` | IP sau hostname server |
| `DEPLOY_USER` | User SSH (ex: admin365) |
| `SSH_PRIVATE_KEY` | Cheia privata SSH |
| `SSH_KNOWN_HOSTS` | Output din `ssh-keyscan <server>` |
---
## Comenzi Utile
### Deploy Script
```bash
./deploy.sh install # Instalare fresh
./deploy.sh update # Update la ultima versiune
./deploy.sh status # Verifica statusul
./deploy.sh logs # Vezi toate logs
./deploy.sh logs domain_check_api # Logs doar pentru API
./deploy.sh start # Porneste serviciile
./deploy.sh stop # Opreste serviciile
./deploy.sh restart # Restart servicii
```
### Docker Direct
```bash
# Start/Stop
docker compose up -d
docker compose down
# Rebuild
docker compose up -d --build
# Logs
docker compose logs -f domain_check_api
# Shell in container
docker exec -it domain_check_api bash
# Database access
docker exec -it domain_check_postgres psql -U dns_admin -d domain_check
```
---
## Troubleshooting
### Container nu porneste
```bash
# Verifica logs
docker compose logs domain_check_api
# Verifica network
docker network ls | grep dns-network
# Recreate containers
docker compose down && docker compose up -d --build
```
### Database connection error
```bash
# Verifica ca PostgreSQL e healthy
docker compose ps domain_check_postgres
# Verifica health
docker exec -it domain_check_postgres pg_isready -U dns_admin
# Restart database
docker compose restart domain_check_postgres
```
### API returns 500 error
```bash
# Check logs
docker compose logs -f domain_check_api
# Verify database schema
docker exec -it domain_check_postgres psql -U dns_admin -d domain_check -c "\dt"
```
### Port deja in uz
```bash
# Gaseste ce foloseste portul
sudo lsof -i :51000
# Schimba portul in .env
API_PORT=51001 # sau alt port liber
```
---
## Securitate - Checklist Productie
- [ ] Schimba toate parolele default in `.env`
- [ ] Seteaza `FLASK_ENV=production`
- [ ] Seteaza `DEBUG=False`
- [ ] Genereaza `SECRET_KEY` random (32+ caractere)
- [ ] Configureaza firewall (doar port API public)
- [ ] Setup HTTPS cu reverse proxy (nginx/traefik)
- [ ] Backup automat PostgreSQL
- [ ] Monitorizare logs
---
## Variabile de Mediu Complete
| Variabila | Default | Descriere |
|-----------|---------|-----------|
| `API_PORT` | 51000 | Port pentru Flask API |
| `API_HOST` | 0.0.0.0 | Bind address |
| `API_VERSION` | v1 | API version prefix |
| `DB_HOST` | domain_check_postgres | PostgreSQL hostname (container) |
| `DB_PORT` | 5432 | PostgreSQL port intern |
| `DB_EXTERNAL_PORT` | 52000 | PostgreSQL port extern |
| `DB_USER` | dns_admin | Database user |
| `DB_PASSWORD` | - | Database password |
| `DB_NAME` | domain_check | Database name |
| `REDIS_HOST` | domain_check_redis | Redis hostname (container) |
| `REDIS_PORT` | 6379 | Redis port intern |
| `REDIS_EXTERNAL_PORT` | 52300 | Redis port extern |
| `REDIS_DB` | 0 | Redis database number |
| `WHOXY_API_KEY` | - | Whoxy API key |
| `VIRUSTOTAL_API_KEY` | - | VirusTotal API key |
| `FLASK_ENV` | development | Flask environment |
| `DEBUG` | True | Debug mode |
| `SECRET_KEY` | - | Flask secret key |
| `LOG_LEVEL` | INFO | Logging level |
| `REDIS_TTL_HOT` | 21600 | Cache TTL hot (6h) |
| `REDIS_TTL_WARM` | 86400 | Cache TTL warm (24h) |
| `REDIS_TTL_COLD` | 604800 | Cache TTL cold (7d) |
---
## License
MIT License - Free for use in anti-fake news projects
---
**Built for fighting disinformation**
**Last Updated:** 2026-02-04