didi-lot2-backend/backend/services

DIDI Backend Services - Master Orchestration

A pyramid-architecture microservices platform for misinformation detection with three distinct layers.

🏗️ Architecture Overview

┌─────────────────────────────────────────────────┐
│         Layer 3: Gateway & Auth Layer           │
│         (Kong API Gateway, Keycloak)            │
├─────────────────────────────────────────────────┤
│        Layer 2: Orchestration Layer             │
│    (Orchestrator, Analysis Services)            │
├─────────────────────────────────────────────────┤
│            Layer 1: Data Layer                  │
│  (PostgreSQL, Redis, RabbitMQ, MinIO)           │
└─────────────────────────────────────────────────┘

📋 Table of Contents

• Quick Start
• Service Layers
• Commands
• Service Ports
• Health Monitoring
• Troubleshooting
• Development

🚀 Quick Start

Start Everything (Recommended)

# Start all layers in pyramid order with health checks
make pyramid

Alternative Methods

# Start all services at once
make up

# Start with logs visible
make dev

# Check service status
make status

# Check service health
make health

🏛️ Service Layers

Layer 1: Data Layer (Foundation)

The foundation of our pyramid - all data persistence and messaging services.

Service	Purpose	Port	Container Name
PostgreSQL	Primary database	5436	dataLayer-postgres
Redis	Cache & session store	6379	dataLayer-redis
RabbitMQ	Message broker	5672/15672	dataLayer-rabbitmq
MinIO	Object storage	9000/9001	dataLayer-minio
pgAdmin	Database management	5051	dataLayer-pgadmin

Layer 2: Orchestration Layer (Processing)

The processing layer - handles business logic and analysis pipelines.

Service	Purpose	Port	Container Name
Orchestrator	Pipeline management	8000	orchestrationLayer-orchestrator
Analysis	Text analysis service	8004	orchestrationLayer-analysis

Layer 3: Gateway & Auth Layer (Access)

The access layer - manages authentication and API routing.

Service	Purpose	Port	Container Name
Kong	API Gateway	8100/8101	gatewayAuthLayer-kong
Keycloak	Authentication	8280	gatewayAuthLayer-keycloak

📦 Commands

Main Commands

make help        # Show all available commands
make pyramid     # Start layers in correct order (recommended)
make up          # Start all services
make down        # Stop all services
make restart     # Restart all services
make status      # Show service status
make health      # Check service health
make logs        # Show all logs
make clean       # Stop and remove everything (including data!)

Layer-Specific Commands

# Data Layer
make data-up     # Start only data layer
make data-down   # Stop data layer

# Orchestration Layer
make orch-up     # Start only orchestration layer
make orch-down   # Stop orchestration layer

# Gateway Layer
make gateway-up  # Start only gateway layer
make gateway-down # Stop gateway layer

Service Logs

make logs-postgres     # PostgreSQL logs
make logs-redis        # Redis logs
make logs-rabbitmq     # RabbitMQ logs
make logs-orchestrator # Orchestrator logs
make logs-kong        # Kong Gateway logs
make logs-keycloak    # Keycloak logs

Development Commands

make dev         # Start with logs visible
make test        # Test all endpoints
make resources   # Show resource usage
make tail        # Tail last 100 log lines

🔌 Service Ports

External Access Points

• Admin Dashboard: http://localhost:3001
• Orchestrator API: http://localhost:8000
• Kong Gateway: http://localhost:8100
• Kong Admin: http://localhost:8101
• Keycloak: http://localhost:8280
• RabbitMQ Management: http://localhost:15672
• MinIO Console: http://localhost:9001
• pgAdmin: http://localhost:5051

Default Credentials

Service	Username	Password
PostgreSQL	postgres	postgres_dev_password_123
RabbitMQ	admin	rabbitmq_dev_password_123
MinIO	minioadmin	minioadmin
pgAdmin	admin@admin.com	admin
Keycloak	admin	admin

🏥 Health Monitoring

Check All Services

make health

Manual Health Checks

# PostgreSQL
docker exec dataLayer-postgres pg_isready -U postgres

# Redis
docker exec dataLayer-redis redis-cli ping

# RabbitMQ
curl -u admin:rabbitmq_dev_password_123 http://localhost:15672/api/health/checks/virtual-hosts

# MinIO
curl http://localhost:9000/minio/health/live

# Orchestrator
curl http://localhost:8000/health

# Kong
curl http://localhost:8101/status

🔧 Troubleshooting

Services Won't Start

# Check if ports are already in use
netstat -an | grep -E "(5436|6379|5672|9000|8000|8100|8280)"

# Check Docker resources
docker system df
docker system prune -a  # Warning: removes all unused images

# Check logs for specific service
make logs-postgres  # Replace with service name

Database Connection Issues

# Test PostgreSQL connection
docker exec dataLayer-postgres psql -U postgres -c "\l"

# Check if database exists
docker exec dataLayer-postgres psql -U postgres -c "SELECT datname FROM pg_database;"

Message Queue Issues

# Check RabbitMQ queues
docker exec dataLayer-rabbitmq rabbitmqctl list_queues

# Check RabbitMQ connections
docker exec dataLayer-rabbitmq rabbitmqctl list_connections

Reset Everything

# WARNING: This deletes all data!
make clean
make pyramid

🛠️ Development

Adding a New Service

1. Create service folder in appropriate layer:

   mkdir -p {layer-name}/new-service
   

2. Add docker-compose.yml to service folder

3. Update layer's docker-compose.yml to include new service

4. Update this README with new service details

Modifying Configurations

1. Environment Variables: Edit .env files in each layer
2. Docker Compose: Edit docker-compose.yml in each layer
3. Service Configs: Edit configuration files in service folders

Building Custom Images

# Build all services
make build

# Build specific layer
cd data-layer && make build

📊 Architecture Flow

graph TB
    subgraph "Layer 3: Gateway & Auth"
        Kong[Kong Gateway :8100]
        Keycloak[Keycloak :8280]
    end
    
    subgraph "Layer 2: Orchestration"
        Orchestrator[Orchestrator :8000]
        Analysis[Analysis :8004]
    end
    
    subgraph "Layer 1: Data"
        PostgreSQL[(PostgreSQL :5436)]
        Redis[(Redis :6379)]
        RabbitMQ[RabbitMQ :5672]
        MinIO[MinIO :9000]
    end
    
    Kong --> Orchestrator
    Kong --> Analysis
    Orchestrator --> PostgreSQL
    Orchestrator --> Redis
    Orchestrator --> RabbitMQ
    Analysis --> PostgreSQL
    Analysis --> Redis
    Analysis --> RabbitMQ
    Analysis --> MinIO

🔒 Security Notes

• Change default passwords before production deployment
• Use environment variables for sensitive data
• Enable TLS/SSL for all external connections
• Configure firewall rules to restrict access
• Regular backup of PostgreSQL and MinIO data

📝 Layer Dependencies

Each layer depends on the layers below it:

1. Data Layer: Independent (foundation)
2. Orchestration Layer: Requires Data Layer
3. Gateway Layer: Requires both Data and Orchestration Layers

Always start services from bottom to top (pyramid order) for proper initialization.

🚦 Service Startup Order

The make pyramid command ensures correct startup order:

1. Data Layer Services (10s wait)
   • PostgreSQL → Redis → RabbitMQ → MinIO
2. Orchestration Services (5s wait)
   • Orchestrator → Analysis
3. Gateway Services (5s wait)
   • Kong → Keycloak

📚 Additional Resources

• Data Layer README
• Orchestration Layer README
• Gateway & Auth Layer README
• Admin Dashboard

---

Version: 1.0.0
Architecture: Pyramid Microservices
Last Updated: December 2024