clossers/didi-lot1-ai
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
didi-lot1-ai/ai_platform/modules/domain_check/CLAUDE.md

6.7 KiB
Raw Permalink Blame History

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

What This Is

Domain Check API — a domain verification and risk-scoring service for detecting disinformation sources (newly registered / suspicious domains). Flask REST API + PostgreSQL + Redis + Celery worker, all run via Docker Compose. Live on this server at http://domain-check-api:11000 (production ports: API 11000, PostgreSQL 12000, Redis 12300 — set in .env, the README sometimes still shows dev ports 5xxxx).

Commands

# Run the full stack (the only supported way to run it — the API expects Postgres/Redis containers)
docker compose up -d --build
docker compose ps
docker compose logs -f domain_check_api

# Smoke test
curl http://localhost:11000/health
./test-api-complete.sh        # exercises the check endpoint
./test-complete-lan.sh        # LAN integration test

# Manual check call
curl -X POST http://localhost:11000/api/v1/check/check \
  -H "Content-Type: application/json" \
  -d '{"domain": "example.com", "check_options": {"whois": true, "dns": true, "ssl": true}}'

# Lint (matches .gitlab-ci.yml)
flake8 api/app --max-line-length=120 --ignore=E501,W503
black --check api/app
isort --check-only api/app

# Database shell
docker exec -it didiAI-domain-check-db psql -U dns_admin -d domain_check

# Deploy helper (wraps docker compose)
./deploy.sh status|logs|start|stop|restart|update

There is no test suite — CI's "test" stage only verifies create_app() succeeds. CI (GitLab) runs lint → SAST (bandit/safety/trivy) → build → push to didiai-domain-check → SSH deploy; all lint/security jobs are allow_failure: true.

Architecture

Everything lives in api/app/ (the dashboard/ directory is vestigial — only a Dockerfile/requirements, not in docker-compose; the actual dashboard is api/app/static/index.html served at /).

Request flow: run.py → create_app() app factory (api/app/init.py) → flask-restx namespaces under /api/v1 → route handlers in app/routes/ call service classes in app/services/ → results persisted via SQLAlchemy models in app/models/ and cached in Redis. Served by gunicorn (gthread workers) in the container — python run.py is dev-only.

  • app/routes/check.py is the only fully implemented route (POST /api/v1/check/check). It runs the independent lookups concurrently in two phases via ThreadPoolExecutor (phase 1 needs only the domain: whois/dns/ssl/http/subdomains; phase 2 needs DNS results: ip/blacklist/port/mail). Each task runs inside app.app_context() because services read current_app. All DB writes happen in the main thread afterwards — the SQLAlchemy session is not thread-safe, so never add DB writes inside the threaded lookups. domain.py, search.py, stats.py, batch.py are stubs registered inside try/except ImportError so missing implementations fail silently.
  • app/services/ — one class per check type: whois_service (RDAP first, Whoxy API fallback; normalize_whois_date() coerces all dates to datetime|None — raw strings here were the root cause of the .ro 500), dns_service, ssl_service, ip_intelligence_service, http_analysis_service, blacklist_service (DNSBL), port_scan_service, subdomain_service (Certificate Transparency), and mail_intelligence_service (SPF/DKIM/DMARC parsing, MTA-STS/TLS-RPT/DANE, MX provider fingerprint + STARTTLS, optional SMTP RCPT probe — degrades gracefully when outbound :25 is blocked). risk_scorer.py combines outputs into a weighted 0–100 score (weights/thresholds in config.py); _coerce_datetime() there is the defense-in-depth guard against stray string dates.
  • check_options flags (all default true except port_scan/subdomains/smtp_probe which default false): whois, dns, ssl, ip_intelligence, http_analysis, blacklist, mail, port_scan, subdomains, smtp_probe, force_refresh. The response includes a top-level availability block (is_registered/is_available/confidence) derived from combined WHOIS+DNS signals — WHOIS alone is unreliable for sparse registries like ROTLD.
  • Rate limiting (Flask-Limiter, Redis-backed) is wired in __init__.py with internal LAN/loopback and /health exempt (_is_internal_request), errors swallowed. Tune via RATELIMIT_DEFAULT env. It exists to protect the paid Whoxy/VirusTotal quotas from external abuse, not internal callers.
  • app/api_models.py defines all flask-restx request/response models. Note the pattern in check.py: models are created on a throwaway namespace at import time for decorators, then __init__.py re-attaches them to the real namespaces — keep model definitions in api_models.py, not inline.
  • Celery is wired up (celery_app.py, separate domain_check_worker container) but app/tasks/__init__.py is empty — no tasks exist yet; batch processing is the intended use.

Configuration: app/config.py selects Development/Production/Testing via FLASK_ENV. Production flips important defaults: Swagger/ReDoc disabled (DISABLE_SWAGGER), minimal /health response (SIMPLE_HEALTH), CORS empty unless CORS_ORIGINS is set. All runtime config comes from .env (see .env.example); ports are only ever changed there, never in compose/code.

Database schema is created by init-scripts/01-init-db.sql, which runs only on first Postgres volume creation. There is no Flask-Migrate migrations/ directory despite the extension being initialized — schema changes must be made in both the SQL init script and the SQLAlchemy models, and applied manually to existing databases. All child tables (whois_records, dns_records, ssl_certificates, etc.) FK to domains with ON DELETE CASCADE.

Caching: Redis with three TTL tiers (REDIS_TTL_HOT/WARM/COLD); a check request with force_refresh: false returns cached/DB data when fresh. Redis failure is non-fatal — the app logs a warning and runs with caching disabled.

Network Context

This is the T4 module (evaluare credibilitate sursă) of the DIDI platform, living at ai_platform/modules/domain_check. It runs as an isolated stack (own Postgres + Redis on the internal domain_check_network); the API container also joins the shared external didi-network under the alias domain-check-api, so agent-v3 (Lot 2) calls it at http://domain-check-api:11000/api/v1/check/check with no extra config. The API port is not published to the host (the platform gateway owns host:11000) — reach it via the didi-network alias, or docker exec into a container on that network. Docs: API_ARCHITECTURE.md (detailed design), use_api.md (consumer-facing API docs), ACCESS_INFO.md, CURRENT_STATUS.md, NEXT_STEPS.md (roadmap).