clossers/didi-lot1-ai
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
didi-lot1-ai/ai_platform/modules/domain_check/API_ARCHITECTURE.md

31 KiB
Raw Blame History

DOMAIN-CHECK API - Arhitectură Completă

Versiune: 1.0.0 Data: 2026-01-29 Scop: Anti-Fake News - Domain Verification & Risk Scoring

📋 CUPRINS

  1. Overview
  2. Stack Tehnologic
  3. Arhitectura Sistemului
  4. API Endpoints
  5. Database Schema
  6. Risk Scoring Engine
  7. Caching Strategy
  8. External Services
  9. Docker Setup
  10. Deployment

🎯 OVERVIEW

Obiectiv Principal

Verificare automată a domeniilor/subdomeniilor pentru identificarea surselor potențiale de dezinformare bazată pe:

  • Vechimea domeniului (risc ridicat pentru domenii < 6 luni)
  • Reputația domeniului (blacklists, typosquatting)
  • Date WHOIS/RDAP (registrar, istoric modificări)
  • Verificări DNS și SSL

Flow Principal

Input (Text/Link)
    ↓
Domain Extraction
    ↓
Domain-Check API
    ↓
Risk Scoring
    ↓
Output (JSON + Dashboard)

🛠 STACK TEHNOLOGIC

Backend

  • Framework: Flask 3.0+ (Python 3.10+)
  • API Documentation: Flask-RESTX (Swagger UI + ReDoc)
  • ORM: SQLAlchemy 2.0+
  • Migration: Alembic
  • Validation: Marshmallow
  • Task Queue: Celery (optional pentru batch processing)

Database & Cache

  • Primary DB: PostgreSQL 15+
  • Cache: Redis 7+ (caching agresiv)
  • Connection Pooling: pgBouncer (optional)

External Services

  • WHOIS/RDAP:
    • Primary: whoisit library (RDAP protocol) - FREE
    • Fallback: Whoxy API (250k requests/lună FREE) - API Key: 876528325417e0bgs418d2cc7d8f193a6
  • Reputation Check:
    • VirusTotal API (500 requests/day FREE)
    • openSquat (self-hosted)
    • Custom blacklist checking
  • DNS: dnspython library
  • SSL: Python ssl + socket modules

Dashboard & CLI

  • Dashboard: Streamlit (MVP) → Flask Admin (production)
  • CLI Tools: Click framework pentru scripturi dedicate

DevOps

  • Containerization: Docker + Docker Compose
  • Network: dns-network (custom bridge)
  • Container Prefix: dns_*
  • Reverse Proxy: Traefik (optional pentru Kong integration)

🏗 ARHITECTURA SISTEMULUI

┌─────────────────────────────────────────────────────────────┐
│                    EXTERNAL INPUT                            │
│  • REST API Calls                                            │
│  • CLI Scripts                                               │
│  • Dashboard Interface                                       │
└────────────────────────┬────────────────────────────────────┘
                         │
                         ▼
┌─────────────────────────────────────────────────────────────┐
│                  FLASK API GATEWAY                           │
│                  (Port: 5000)                                │
│  ┌───────────────────────────────────────────────────────┐  │
│  │  Routes:                                              │  │
│  │  • /api/v1/check          [POST]   Single domain     │  │
│  │  • /api/v1/check/batch    [POST]   Multiple domains  │  │
│  │  • /api/v1/domain/{id}    [GET]    Domain details    │  │
│  │  • /api/v1/history/{id}   [GET]    Check history     │  │
│  │  • /api/v1/stats          [GET]    Statistics        │  │
│  │  • /docs                  [GET]    Swagger UI        │  │
│  │  • /redoc                 [GET]    ReDoc UI          │  │
│  └───────────────────────────────────────────────────────┘  │
└────────────────────────┬────────────────────────────────────┘
                         │
         ┌───────────────┼───────────────┐
         │               │               │
         ▼               ▼               ▼
┌─────────────┐  ┌──────────────┐  ┌──────────────┐
│   CACHE     │  │   BUSINESS   │  │   EXTERNAL   │
│   LAYER     │  │    LOGIC     │  │   SERVICES   │
│             │  │              │  │              │
│  Redis      │  │  • Domain    │  │  • RDAP      │
│  (Port:     │  │    Extractor │  │  • Whoxy     │
│   6379)     │  │  • WHOIS     │  │  • VT API    │
│             │  │    Fetcher   │  │  • DNS       │
│  TTL:       │  │  • Risk      │  │  • SSL       │
│  • 24h hot  │  │    Scorer    │  │              │
│  • 7d warm  │  │  • Validator │  │              │
└──────┬──────┘  └──────┬───────┘  └──────┬───────┘
       │                │                  │
       └────────────────┼──────────────────┘
                        ▼
           ┌─────────────────────────┐
           │   POSTGRESQL DATABASE   │
           │   (Port: 5432)          │
           │                         │
           │   Tables:               │
           │   • domains             │
           │   • whois_records       │
           │   • dns_records         │
           │   • ssl_certificates    │
           │   • reputation_scores   │
           │   • check_history       │
           │   • blacklists          │
           └─────────────────────────┘

🔌 API ENDPOINTS

Base URL: http://localhost:5000/api/v1

1. POST /api/v1/check

Verifică un singur domeniu.

Request Body:

{
  "domain": "example.com",
  "check_options": {
    "whois": true,
    "dns": true,
    "ssl": true,
    "reputation": true,
    "force_refresh": false
  }
}

Response (200 OK):

{
  "success": true,
  "data": {
    "domain": "example.com",
    "check_id": "uuid-here",
    "timestamp": "2026-01-29T12:00:00Z",
    "whois": {
      "creation_date": "1995-08-14T04:00:00Z",
      "expiration_date": "2027-08-13T04:00:00Z",
      "registrar": "IANA",
      "age_days": 11125,
      "status": ["clientDeleteProhibited", "clientTransferProhibited"]
    },
    "dns": {
      "a_records": ["93.184.216.34"],
      "mx_records": ["0 ."],
      "ns_records": ["a.iana-servers.net", "b.iana-servers.net"],
      "txt_records": ["v=spf1 -all"]
    },
    "ssl": {
      "valid": true,
      "issuer": "DigiCert Inc",
      "expires": "2026-12-25T23:59:59Z",
      "days_until_expiry": 330
    },
    "reputation": {
      "virustotal": {
        "malicious": 0,
        "suspicious": 0,
        "harmless": 85,
        "undetected": 5,
        "last_analysis": "2026-01-28T10:00:00Z"
      },
      "blacklists": [],
      "typosquatting_match": false
    },
    "risk_score": {
      "total": 15,
      "level": "LOW",
      "factors": [
        {"factor": "domain_age", "score": 0, "weight": 0.3, "reason": "Domain is 30+ years old"},
        {"factor": "ssl_valid", "score": 0, "weight": 0.2, "reason": "Valid SSL certificate"},
        {"factor": "reputation", "score": 0, "weight": 0.3, "reason": "No malicious reports"},
        {"factor": "dns_records", "score": 15, "weight": 0.2, "reason": "Complete DNS configuration"}
      ],
      "thresholds": {
        "low": "0-30",
        "medium": "31-60",
        "high": "61-85",
        "critical": "86-100"
      }
    }
  },
  "metadata": {
    "cached": false,
    "processing_time_ms": 1243,
    "api_version": "1.0.0"
  }
}

Response (400 Bad Request):

{
  "success": false,
  "error": {
    "code": "INVALID_DOMAIN",
    "message": "Domain format is invalid",
    "details": "Domain must match pattern: ^[a-zA-Z0-9][a-zA-Z0-9-]{0,61}[a-zA-Z0-9]?\\.[a-zA-Z]{2,}$"
  }
}

2. POST /api/v1/check/batch

Verifică multiple domenii în paralel.

Request Body:

{
  "domains": [
    "example.com",
    "google.com",
    "suspicious-news-2026.com"
  ],
  "check_options": {
    "whois": true,
    "dns": true,
    "ssl": false,
    "reputation": true
  },
  "priority": "normal"
}

Response (202 Accepted):

{
  "success": true,
  "data": {
    "batch_id": "batch-uuid-here",
    "total_domains": 3,
    "status": "processing",
    "estimated_completion": "2026-01-29T12:05:00Z"
  },
  "links": {
    "status": "/api/v1/batch/batch-uuid-here/status",
    "results": "/api/v1/batch/batch-uuid-here/results"
  }
}

3. GET /api/v1/domain/{domain}

Recuperează detalii complete pentru un domeniu verificat anterior.

Path Parameters:

  • domain (string): Domain name (ex: example.com)

Query Parameters:

  • include_history (boolean, default: false): Include all previous checks

Response (200 OK):

{
  "success": true,
  "data": {
    "domain": "example.com",
    "first_checked": "2026-01-15T08:00:00Z",
    "last_checked": "2026-01-29T12:00:00Z",
    "total_checks": 12,
    "current_risk_score": 15,
    "risk_trend": "stable",
    "latest_data": {
      "whois": {...},
      "dns": {...},
      "ssl": {...},
      "reputation": {...}
    },
    "history": [
      {
        "check_id": "uuid-1",
        "timestamp": "2026-01-29T12:00:00Z",
        "risk_score": 15,
        "changes": []
      }
    ]
  }
}

4. GET /api/v1/history/{domain}

Istoricul complet de verificări pentru un domeniu.

Response (200 OK):

{
  "success": true,
  "data": {
    "domain": "example.com",
    "total_checks": 12,
    "date_range": {
      "from": "2026-01-15T08:00:00Z",
      "to": "2026-01-29T12:00:00Z"
    },
    "checks": [
      {
        "check_id": "uuid-here",
        "timestamp": "2026-01-29T12:00:00Z",
        "risk_score": 15,
        "risk_level": "LOW",
        "changes_detected": false,
        "whois_snapshot": {...},
        "dns_snapshot": {...}
      }
    ]
  },
  "pagination": {
    "page": 1,
    "per_page": 50,
    "total": 12
  }
}

5. GET /api/v1/stats

Statistici generale ale sistemului.

Response (200 OK):

{
  "success": true,
  "data": {
    "overview": {
      "total_domains_checked": 8456,
      "unique_domains": 3421,
      "checks_today": 234,
      "checks_this_month": 9876
    },
    "risk_distribution": {
      "low": 2845,
      "medium": 412,
      "high": 134,
      "critical": 30
    },
    "top_risky_domains": [
      {
        "domain": "fake-news-urgent.com",
        "risk_score": 92,
        "creation_date": "2026-01-20T00:00:00Z",
        "age_days": 9
      }
    ],
    "cache_performance": {
      "hit_rate": 0.78,
      "total_requests": 12340,
      "cache_hits": 9625,
      "cache_misses": 2715
    },
    "api_usage": {
      "whoxy_calls_today": 145,
      "whoxy_quota_remaining": 249855,
      "virustotal_calls_today": 67,
      "virustotal_quota_remaining": 433
    }
  }
}

6. GET /api/v1/batch/{batch_id}/status

Verifică statusul unei operații batch.

Response (200 OK):

{
  "success": true,
  "data": {
    "batch_id": "batch-uuid",
    "status": "completed",
    "progress": {
      "total": 100,
      "completed": 100,
      "failed": 2,
      "percentage": 100
    },
    "started_at": "2026-01-29T12:00:00Z",
    "completed_at": "2026-01-29T12:08:32Z",
    "duration_seconds": 512
  }
}

7. GET /api/v1/search

Caută domenii în baza de date.

Query Parameters:

  • q (string): Search query
  • risk_level (enum): LOW, MEDIUM, HIGH, CRITICAL
  • min_age_days (int): Minimum domain age
  • max_age_days (int): Maximum domain age
  • registrar (string): Filter by registrar
  • has_ssl (boolean): Filter by SSL presence
  • page (int, default: 1)
  • per_page (int, default: 50, max: 100)

Response (200 OK):

{
  "success": true,
  "data": {
    "results": [
      {
        "domain": "example.com",
        "risk_score": 15,
        "risk_level": "LOW",
        "age_days": 11125,
        "last_checked": "2026-01-29T12:00:00Z"
      }
    ],
    "pagination": {
      "page": 1,
      "per_page": 50,
      "total_results": 234,
      "total_pages": 5
    }
  }
}

8. POST /api/v1/webhook

Configurare webhook pentru notificări.

Request Body:

{
  "url": "https://your-app.com/webhook/domain-alerts",
  "events": ["high_risk_detected", "domain_age_threshold"],
  "filters": {
    "min_risk_score": 70
  }
}

🗄 DATABASE SCHEMA

PostgreSQL Tables

1. domains

CREATE TABLE domains (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    domain VARCHAR(255) UNIQUE NOT NULL,
    subdomain VARCHAR(255),
    tld VARCHAR(50) NOT NULL,
    first_seen_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
    last_checked_at TIMESTAMP WITH TIME ZONE,
    check_count INTEGER DEFAULT 0,
    is_active BOOLEAN DEFAULT TRUE,
    created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
    updated_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),

    INDEX idx_domain (domain),
    INDEX idx_last_checked (last_checked_at),
    INDEX idx_tld (tld)
);

2. whois_records

CREATE TABLE whois_records (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    domain_id UUID REFERENCES domains(id) ON DELETE CASCADE,
    creation_date TIMESTAMP WITH TIME ZONE,
    expiration_date TIMESTAMP WITH TIME ZONE,
    updated_date TIMESTAMP WITH TIME ZONE,
    registrar VARCHAR(255),
    registrar_url VARCHAR(500),
    registrant_org VARCHAR(255),
    registrant_country VARCHAR(2),
    admin_email VARCHAR(255),
    name_servers TEXT[], -- Array of name servers
    status TEXT[], -- Array of domain statuses
    dnssec BOOLEAN,
    raw_whois_data JSONB,
    data_source VARCHAR(50), -- 'rdap', 'whoxy', 'manual'
    fetched_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),

    INDEX idx_creation_date (creation_date),
    INDEX idx_registrar (registrar),
    INDEX idx_data_source (data_source)
);

3. dns_records

CREATE TABLE dns_records (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    domain_id UUID REFERENCES domains(id) ON DELETE CASCADE,
    record_type VARCHAR(10) NOT NULL, -- 'A', 'AAAA', 'MX', 'TXT', 'NS', 'CNAME'
    record_value TEXT NOT NULL,
    ttl INTEGER,
    priority INTEGER, -- For MX records
    fetched_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),

    INDEX idx_domain_type (domain_id, record_type),
    INDEX idx_fetched_at (fetched_at)
);

4. ssl_certificates

CREATE TABLE ssl_certificates (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    domain_id UUID REFERENCES domains(id) ON DELETE CASCADE,
    issuer VARCHAR(255),
    subject VARCHAR(255),
    valid_from TIMESTAMP WITH TIME ZONE,
    valid_until TIMESTAMP WITH TIME ZONE,
    serial_number VARCHAR(255),
    signature_algorithm VARCHAR(100),
    key_size INTEGER,
    is_wildcard BOOLEAN DEFAULT FALSE,
    is_self_signed BOOLEAN DEFAULT FALSE,
    certificate_chain JSONB,
    fetched_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),

    INDEX idx_valid_until (valid_until),
    INDEX idx_issuer (issuer)
);

5. reputation_scores

CREATE TABLE reputation_scores (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    domain_id UUID REFERENCES domains(id) ON DELETE CASCADE,
    source VARCHAR(50) NOT NULL, -- 'virustotal', 'custom', 'opensquat'
    score INTEGER, -- 0-100
    malicious_count INTEGER DEFAULT 0,
    suspicious_count INTEGER DEFAULT 0,
    harmless_count INTEGER DEFAULT 0,
    undetected_count INTEGER DEFAULT 0,
    is_blacklisted BOOLEAN DEFAULT FALSE,
    blacklist_names TEXT[],
    is_typosquatting BOOLEAN DEFAULT FALSE,
    typosquatting_target VARCHAR(255),
    raw_response JSONB,
    checked_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),

    INDEX idx_source (source),
    INDEX idx_is_blacklisted (is_blacklisted),
    INDEX idx_checked_at (checked_at),
    UNIQUE(domain_id, source, checked_at)
);

6. risk_assessments

CREATE TABLE risk_assessments (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    domain_id UUID REFERENCES domains(id) ON DELETE CASCADE,
    check_id UUID UNIQUE NOT NULL,
    total_score INTEGER NOT NULL, -- 0-100
    risk_level VARCHAR(20) NOT NULL, -- 'LOW', 'MEDIUM', 'HIGH', 'CRITICAL'

    -- Individual factor scores
    domain_age_score INTEGER,
    domain_age_days INTEGER,
    ssl_score INTEGER,
    dns_score INTEGER,
    reputation_score INTEGER,
    whois_score INTEGER,

    -- Risk factors breakdown
    factors JSONB, -- Array of {factor, score, weight, reason}

    -- Flags
    is_new_domain BOOLEAN DEFAULT FALSE, -- < 6 months
    is_suspicious BOOLEAN DEFAULT FALSE,
    requires_manual_review BOOLEAN DEFAULT FALSE,

    assessed_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),

    INDEX idx_risk_level (risk_level),
    INDEX idx_total_score (total_score),
    INDEX idx_is_new_domain (is_new_domain),
    INDEX idx_assessed_at (assessed_at)
);

7. check_history

CREATE TABLE check_history (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    check_id UUID UNIQUE NOT NULL,
    domain_id UUID REFERENCES domains(id) ON DELETE CASCADE,
    risk_assessment_id UUID REFERENCES risk_assessments(id),

    -- Check metadata
    requested_by VARCHAR(100), -- 'api', 'cli', 'dashboard', 'batch'
    request_ip VARCHAR(45),
    check_options JSONB,

    -- Performance metrics
    processing_time_ms INTEGER,
    cache_hit BOOLEAN DEFAULT FALSE,

    -- Changes detection
    changes_detected BOOLEAN DEFAULT FALSE,
    change_summary JSONB,

    created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),

    INDEX idx_check_id (check_id),
    INDEX idx_domain_created (domain_id, created_at),
    INDEX idx_created_at (created_at)
);

8. batch_operations

CREATE TABLE batch_operations (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    batch_id UUID UNIQUE NOT NULL,
    total_domains INTEGER NOT NULL,
    completed_count INTEGER DEFAULT 0,
    failed_count INTEGER DEFAULT 0,
    status VARCHAR(20) DEFAULT 'pending', -- 'pending', 'processing', 'completed', 'failed'
    priority VARCHAR(20) DEFAULT 'normal',
    started_at TIMESTAMP WITH TIME ZONE,
    completed_at TIMESTAMP WITH TIME ZONE,
    error_log JSONB,
    created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),

    INDEX idx_batch_id (batch_id),
    INDEX idx_status (status)
);

9. blacklists

CREATE TABLE blacklists (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    domain VARCHAR(255) UNIQUE NOT NULL,
    reason TEXT,
    category VARCHAR(50), -- 'phishing', 'malware', 'spam', 'fake_news'
    source VARCHAR(100),
    added_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
    is_active BOOLEAN DEFAULT TRUE,

    INDEX idx_domain (domain),
    INDEX idx_category (category)
);

10. api_usage_logs

CREATE TABLE api_usage_logs (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    service_name VARCHAR(50) NOT NULL, -- 'whoxy', 'virustotal', 'rdap'
    endpoint VARCHAR(255),
    request_count INTEGER DEFAULT 1,
    response_time_ms INTEGER,
    status_code INTEGER,
    quota_used INTEGER,
    quota_remaining INTEGER,
    error_message TEXT,
    logged_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),

    INDEX idx_service_logged (service_name, logged_at),
    INDEX idx_logged_at (logged_at)
);

🎯 RISK SCORING ENGINE

Scoring Algorithm

RISK_SCORE = (
    DOMAIN_AGE_SCORE * 0.30 +
    REPUTATION_SCORE * 0.30 +
    SSL_SCORE * 0.20 +
    DNS_SCORE * 0.10 +
    WHOIS_SCORE * 0.10
)

1. Domain Age Score (Weight: 30%)

Age Score Risk Level Reason
0-3 months 100 CRITICAL Very new domain, high phishing risk
3-6 months 80 HIGH New domain, elevated risk
6-12 months 50 MEDIUM Moderately new domain
1-2 years 30 LOW Established domain
2+ years 0 LOW Trusted age

2. Reputation Score (Weight: 30%)

Condition Score Weight
Blacklisted +100 Critical
VirusTotal malicious > 5 +80 High
VirusTotal suspicious > 10 +50 Medium
Typosquatting detected +70 High
No reputation data +20 Low
Clean reputation 0 None

3. SSL Score (Weight: 20%)

Condition Score
No SSL certificate +100
Self-signed certificate +80
Expired certificate +100
Certificate < 30 days old +40
Certificate expires < 30 days +30
Valid certificate 0

4. DNS Score (Weight: 10%)

Condition Score
No MX records +30
No TXT records (SPF/DKIM) +20
Suspicious NS records +40
Recently changed NS +50
Complete DNS config 0

5. WHOIS Score (Weight: 10%)

Condition Score
WHOIS privacy protection +30
Registrant country mismatch +20
Registrar known for abuse +50
Recent WHOIS changes +30
Complete transparent WHOIS 0

Final Risk Level Classification

Total Score Risk Level Action
0-30 LOW No action needed
31-60 MEDIUM Monitor
61-85 HIGH Flag for review
86-100 CRITICAL Block/Alert immediately

Special Rules for Fake News Detection

# CRITICAL FLAG: New domain + breaking news
if domain_age < 180 days AND content_type == "breaking_news":
    risk_level = "CRITICAL"
    score += 50

# HIGH FLAG: Typosquatting + news content
if typosquatting_detected AND content_type == "news":
    risk_level = "HIGH"
    score += 40

# MEDIUM FLAG: No SSL + sensitive topics
if not has_ssl AND topic in ["politics", "health", "finance"]:
    risk_level = "MEDIUM"
    score += 30

💾 CACHING STRATEGY

Redis Cache Layers

Layer 1: Hot Cache (TTL: 6 hours)

  • Recent domain checks (< 6 hours old)
  • High-traffic domains
  • Pattern: domain:hot:{domain}:check

Layer 2: Warm Cache (TTL: 24 hours)

  • Domain WHOIS data
  • DNS records
  • SSL certificate info
  • Pattern: domain:warm:{domain}:{data_type}

Layer 3: Cold Cache (TTL: 7 days)

  • Reputation scores (VirusTotal)
  • Historical risk scores
  • Pattern: domain:cold:{domain}:reputation

Cache Key Patterns

# Domain check result
f"dns:check:{domain}:{timestamp}"

# WHOIS data
f"dns:whois:{domain}"

# DNS records
f"dns:records:{domain}:{record_type}"

# SSL certificate
f"dns:ssl:{domain}"

# Reputation score
f"dns:reputation:{domain}:{source}"

# Risk assessment
f"dns:risk:{domain}"

# API quota tracking
f"dns:quota:{service_name}:{date}"

Cache Invalidation Rules

  1. Force Refresh: check_options.force_refresh = true bypasses cache
  2. Auto-Invalidate:
    • WHOIS data: if domain age changes significantly
    • DNS records: if TTL expires
    • SSL: if certificate expiration < 30 days
    • Reputation: if blacklist status changes
  3. Manual Invalidate: API endpoint for cache clearing

🌐 EXTERNAL SERVICES

1. WHOXY API

API Key: 876528325417e0bgs418d2cc7d8f193a6 Quota: 250,000 requests/month FREE Endpoint: https://api.whoxy.com/

Usage:

import requests

response = requests.get(
    "https://api.whoxy.com/",
    params={
        "key": "876528325417e0bgs418d2cc7d8f193a6",
        "whois": "example.com"
    }
)

Rate Limiting:

  • Implement exponential backoff
  • Track daily usage in api_usage_logs
  • Alert at 80% quota usage

2. VirusTotal API

Quota: 500 requests/day FREE Endpoint: https://www.virustotal.com/api/v3/

Priority System:

# Only use VirusTotal for:
1. New domains (< 6 months)
2. Domains with suspicious patterns
3. High-priority batch requests
4. Manual override requests

Fallback Strategy:

if virustotal_quota_exceeded:
    # Fallback to local blacklist check
    # Use cached reputation data
    # Assign conservative risk score

3. RDAP (via whoisit library)

Free, unlimited, no API key needed

from whoisit import Domain

domain = Domain("example.com")
print(domain.creation_date)
print(domain.registrar)

🐳 DOCKER SETUP

docker-compose.yml

version: '3.9'

networks:
  dns-network:
    driver: bridge
    name: dns-network

services:
  dns_postgres:
    image: postgres:15-alpine
    container_name: dns_postgres
    networks:
      - dns-network
    environment:
      POSTGRES_DB: domain_check
      POSTGRES_USER: dns_admin
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      PGDATA: /var/lib/postgresql/data/pgdata
    volumes:
      - postgres_data:/var/lib/postgresql/data
      - ./init-scripts:/docker-entrypoint-initdb.d
    ports:
      - "5432:5432"
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U dns_admin -d domain_check"]
      interval: 10s
      timeout: 5s
      retries: 5
    restart: unless-stopped

  dns_redis:
    image: redis:7-alpine
    container_name: dns_redis
    networks:
      - dns-network
    command: redis-server --appendonly yes --maxmemory 512mb --maxmemory-policy allkeys-lru
    volumes:
      - redis_data:/data
    ports:
      - "6379:6379"
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 10s
      timeout: 3s
      retries: 5
    restart: unless-stopped

  dns_api:
    build:
      context: ./api
      dockerfile: Dockerfile
    container_name: dns_api
    networks:
      - dns-network
    environment:
      - DATABASE_URL=postgresql://dns_admin:${DB_PASSWORD}@dns_postgres:5432/domain_check
      - REDIS_URL=redis://dns_redis:6379/0
      - WHOXY_API_KEY=${WHOXY_API_KEY}
      - VIRUSTOTAL_API_KEY=${VIRUSTOTAL_API_KEY}
      - FLASK_ENV=development
      - LOG_LEVEL=INFO
    volumes:
      - ./api:/app
      - api_logs:/app/logs
    ports:
      - "5000:5000"
    depends_on:
      dns_postgres:
        condition: service_healthy
      dns_redis:
        condition: service_healthy
    restart: unless-stopped

  dns_dashboard:
    build:
      context: ./dashboard
      dockerfile: Dockerfile
    container_name: dns_dashboard
    networks:
      - dns-network
    environment:
      - API_URL=http://dns_api:5000
    volumes:
      - ./dashboard:/app
    ports:
      - "8501:8501"
    depends_on:
      - dns_api
    restart: unless-stopped

  dns_worker:
    build:
      context: ./api
      dockerfile: Dockerfile
    container_name: dns_worker
    networks:
      - dns-network
    command: celery -A app.celery worker --loglevel=info
    environment:
      - DATABASE_URL=postgresql://dns_admin:${DB_PASSWORD}@dns_postgres:5432/domain_check
      - REDIS_URL=redis://dns_redis:6379/0
      - WHOXY_API_KEY=${WHOXY_API_KEY}
      - VIRUSTOTAL_API_KEY=${VIRUSTOTAL_API_KEY}
    volumes:
      - ./api:/app
    depends_on:
      - dns_api
      - dns_redis
    restart: unless-stopped

volumes:
  postgres_data:
    name: dns_postgres_data
  redis_data:
    name: dns_redis_data
  api_logs:
    name: dns_api_logs

.env File

# Database
DB_PASSWORD=your_secure_password_here

# API Keys
WHOXY_API_KEY=876528325417e0bgs418d2cc7d8f193a6
VIRUSTOTAL_API_KEY=your_vt_api_key_here

# Application
FLASK_ENV=development
LOG_LEVEL=INFO

# Cache
REDIS_TTL_HOT=21600
REDIS_TTL_WARM=86400
REDIS_TTL_COLD=604800

📁 PROJECT STRUCTURE

domain-check/
├── api/
│   ├── app/
│   │   ├── __init__.py
│   │   ├── config.py
│   │   ├── models/
│   │   │   ├── __init__.py
│   │   │   ├── domain.py
│   │   │   ├── whois.py
│   │   │   ├── dns.py
│   │   │   ├── ssl.py
│   │   │   ├── reputation.py
│   │   │   └── risk.py
│   │   ├── routes/
│   │   │   ├── __init__.py
│   │   │   ├── check.py
│   │   │   ├── batch.py
│   │   │   ├── domain.py
│   │   │   ├── stats.py
│   │   │   └── search.py
│   │   ├── services/
│   │   │   ├── __init__.py
│   │   │   ├── whois_service.py
│   │   │   ├── dns_service.py
│   │   │   ├── ssl_service.py
│   │   │   ├── reputation_service.py
│   │   │   ├── risk_scorer.py
│   │   │   └── cache_service.py
│   │   ├── utils/
│   │   │   ├── __init__.py
│   │   │   ├── validators.py
│   │   │   ├── extractors.py
│   │   │   └── logger.py
│   │   └── tasks/
│   │       ├── __init__.py
│   │       └── batch_tasks.py
│   ├── migrations/
│   ├── tests/
│   ├── Dockerfile
│   ├── requirements.txt
│   └── run.py
├── dashboard/
│   ├── app.py
│   ├── pages/
│   │   ├── 1_🔍_Domain_Check.py
│   │   ├── 2_📊_Statistics.py
│   │   ├── 3_📜_History.py
│   │   └── 4_⚙️_Settings.py
│   ├── components/
│   ├── Dockerfile
│   └── requirements.txt
├── cli/
│   ├── check_domain.py
│   ├── batch_check.py
│   ├── export_data.py
│   └── requirements.txt
├── init-scripts/
│   └── 01-init-db.sql
├── docker-compose.yml
├── .env
├── .env.example
├── .gitignore
├── README.md
└── API_ARCHITECTURE.md (this file)

🚀 DEPLOYMENT

Initial Setup

# 1. Clone & setup
git clone <repo> domain-check
cd domain-check

# 2. Configure environment
cp .env.example .env
# Edit .env with your credentials

# 3. Start services
docker-compose up -d

# 4. Run migrations
docker exec -it dns_api flask db upgrade

# 5. Seed initial data
docker exec -it dns_api python scripts/seed_blacklists.py

# 6. Verify health
curl http://localhost:5000/health
curl http://localhost:5000/docs

Testing

# Test single domain
curl -X POST http://localhost:5000/api/v1/check \
  -H "Content-Type: application/json" \
  -d '{"domain": "example.com"}'

# Access dashboard
open http://localhost:8501

Production Checklist

  • Change default passwords in .env
  • Configure proper logging (Sentry, Datadog)
  • Setup backup strategy for PostgreSQL
  • Configure Kong API Gateway integration
  • Enable HTTPS (Let's Encrypt)
  • Setup monitoring (Prometheus + Grafana)
  • Configure alerting (PagerDuty, Slack)
  • Implement rate limiting
  • Add authentication middleware
  • Security audit (OWASP Top 10)

📝 API DOCUMENTATION

Swagger UI

Access at: http://localhost:5000/docs

ReDoc

Access at: http://localhost:5000/redoc

Postman Collection

Available at: /docs/postman_collection.json

🔄 UPDATE HISTORY

Version Date Changes
1.0.0 2026-01-29 Initial architecture design

Last Updated: 2026-01-29 Maintainer: Domain-Check Team Contact: [Add contact info]