didi-lot2-backend/backend/services/data-layer/README.md

DIDI Backend - Data Layer 🗄️

Quick Start 🚀

# Recommended: Start via unified deployment manager
./deploy/didi.sh staging start

All data services start automatically as part of the staging deployment.

What is the Data Layer? 🤔

The Data Layer provides pure storage for the DIDI Backend platform. No business logic, no routing, just reliable data storage.

Architecture Overview 📊

┌─────────────────────────────────────────────────────────────────────────┐
│                         DATA LAYER                                      │
│                                                                         │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐ │
│  │              │  │              │  │              │  │              │ │
│  │  PostgreSQL  │  │    Redis     │  │    MinIO     │  │   RabbitMQ   │ │
│  │              │  │              │  │              │  │              │ │
│  │   Database   │  │    Cache     │  │   Storage    │  │    Queue     │ │
│  │   PgAdmin    │  │   Commander  │  │              │  │              │ │
│  └──────────────┘  └──────────────┘  └──────────────┘  └──────────────┘ │
│       Ports:            Ports:           Ports:            Ports:       │
│   22001 (DB)       22301 (Cache)    27000 (API)       23100 (AMQP)      │
│   29001 (UI)       29002 (UI)       27001 (Console)   23101 (UI)        │
│                                                                         │
│  ┌────────────────────────────────────────────────────────────────────┐ │
│  │         Network: didi-backend (shared by ALL services)             │ │
│  └────────────────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────────┘

Note

: Ports shown are for staging (the primary deployment method via ./deploy/didi.sh staging).

The Four Services + Management UIs 📦

1. didiDatabase (PostgreSQL) 🐘

Purpose: Persistent structured data storage

• Stores all application data
• 5 schemas (users, catalog, pipelines, execution, analyses)
• ACID compliant transactions
• Port: 22001 (staging)
• pgAdmin (Port 29001): Web UI for database management

2. didiCache (Redis) ⚡

Purpose: High-speed temporary storage

• Pipeline execution status
• Real-time updates
• Session data
• 24-hour TTL for most keys
• Port: 22301 (staging)
• Redis Commander (Port 29002): Web UI for Redis management

3. didiStorage (MinIO) 📁

Purpose: Object/file storage

• 7 auto-created buckets
• Media files (images, videos, audio)
• Documents and backups
• Auto-expiry policies
• Ports: 27000 (API), 27001 (Console) (staging)

4. didiQueue (RabbitMQ) 🐰

Purpose: Message queue for async processing

• Pipeline job queuing
• Decouples API from processing
• Single unified queue for all analysis types
• Durable message storage
• Ports: 23100 (AMQP), 23101 (Management UI) (staging)

Quick Start - Entire Data Layer 🎯

Start Everything

# From this directory
make up

# Or with Docker Compose
docker compose up -d

Stop Everything

make down

Check Status

make status

View Logs

make logs

Individual Service Management 🔧

Start Individual Services

make up-database    # Start only PostgreSQL
make up-cache       # Start only Redis
make up-storage     # Start only MinIO
make up-queue       # Start only RabbitMQ

Check Individual Health

make health-database
make health-cache
make health-storage
make health-queue

Access Points 🌐 (Staging)

Service	Type	Access URL	Credentials
PostgreSQL	Database	localhost:22001	postgres / postgres123
pgAdmin	Web UI	http://localhost:29001	admin@example.com / admin123
Redis	Cache	localhost:22301	Password: redis123
Redis Commander	Web UI	http://localhost:29002	admin / commander123
MinIO	API	localhost:27000	minioadmin / minio123
MinIO	Console	http://localhost:27001	minioadmin / minio123
RabbitMQ	AMQP	localhost:23100	admin / rabbitmq123
RabbitMQ	Management UI	http://localhost:23101	admin / rabbitmq123

What Gets Auto-Created? ✨

When you run make up:

PostgreSQL

• ✅ 5 schemas
• ✅ All tables
• ✅ Indexes and triggers
• ✅ User subscription plans

Redis

• ✅ Configured with password
• ✅ Persistence enabled
• ✅ Memory limits set
• ✅ Ready for connections

MinIO

• ✅ 7 buckets created
• ✅ Versioning enabled
• ✅ Lifecycle policies
• ✅ Service access policies

RabbitMQ

• ✅ Unified analysis queue created
• ✅ Dead letter queue configured
• ✅ Management plugin enabled
• ✅ Message TTL policies set

Directory Structure 📂

data-layer/
├── README.md           # This file
├── docker-compose.yml  # Unified orchestration
├── Makefile           # Simple commands
│
├── didiDatabase/      # PostgreSQL Service
│   ├── docker-compose.yml
│   ├── init.sql       # Database schema
│   ├── .env          # Configuration
│   └── README.md     # Service docs
│
├── didiCache/        # Redis Service
│   ├── docker-compose.yml
│   ├── redis.conf   # Redis config
│   ├── .env         # Configuration
│   └── README.md    # Service docs
│
├── didiStorage/      # MinIO Service
│   ├── docker-compose.yml
│   ├── init-buckets.sh  # Auto-setup
│   ├── .env            # Configuration
│   └── README.md       # Service docs
│
└── didiQueue/        # RabbitMQ Service
    ├── docker-compose.yml
    ├── init-queues.sh  # Queue setup
    ├── .env           # Configuration
    └── README.md      # Service docs

Environment Variables 🔐

Each service has its own .env file. Change these for production!

Critical Passwords to Change:

• POSTGRES_PASSWORD in didiDatabase/.env
• REDIS_PASSWORD in didiCache/.env
• HTTP_PASSWORD for Redis Commander in docker-compose.yml
• MINIO_ROOT_PASSWORD in didiStorage/.env
• RABBITMQ_PASSWORD in didiQueue/.env

Testing the Data Layer 🧪

# Run all tests
make test

# Test individual services
make test-database
make test-cache
make test-storage
make test-queue

Production Deployment 🚀

# Check for default passwords
make check-security

# Deploy with production settings
make prod

Troubleshooting 🔧

Service won't start?

# Check logs
make logs-database
make logs-cache
make logs-storage
make logs-queue

Port conflicts?

Edit the .env file in the service directory and change the port.

Need a fresh start?

# WARNING: Deletes all data!
make clean
make up

Resource Usage 📊

Service	Memory Limit	CPU Limit	Disk Usage
PostgreSQL	2GB	1.0 CPU	~500MB + data
Redis	512MB	0.5 CPU	~100MB + cache
MinIO	1GB	0.5 CPU	~200MB + files
RabbitMQ	1GB	0.5 CPU	~100MB + messages

Total: ~4.5GB RAM, 2.5 CPUs

Network Architecture 🌐

All services communicate on the didi-backend network:

• Shared by ALL backend services (data layer, API layer, orchestration, etc.)
• Internal DNS resolution by service name
• Isolated from external access (except mapped ports)
• Services can reach each other by hostname
• Other Docker Compose projects can join this network using:

  networks:
    default:
      external: true
      name: didi-backend
  

Why This Architecture? 🎯

1. Separation of Concerns

   • Each service does ONE thing well
   • Easy to scale individually
   • Simple to understand

2. Zero Configuration

   • Everything auto-configures
   • No manual setup needed
   • Production-ready defaults

3. Developer Friendly

   • One command to start
   • Clear documentation
   • Web UIs included

Next Steps 🏗️

The Data Layer is complete! Next layers to build:

✅ data-layer/        # Complete!
⏳ orchestration-layer/  # Next: RabbitMQ, Kong, Vault
⏳ service-layer/       # Then: Microservices
⏳ application-layer/   # Finally: UI/Apps

Support & Maintenance 🛠️

Daily Tasks

• Check logs: make logs
• Monitor usage: make stats
• Backup data: make backup

Weekly Tasks

• Review disk usage
• Check for updates
• Rotate passwords

Monthly Tasks

• Full backup
• Performance review
• Security audit

---

Quick Reference Card 📋

# Essential Commands
make up              # Start everything
make down            # Stop everything
make status          # Check health
make logs            # View logs
make clean           # Delete all data

# Individual Services
make up-database     # Start PostgreSQL
make up-cache        # Start Redis
make up-storage      # Start MinIO

# Utilities
make backup          # Backup all data
make restore         # Restore from backup
make test            # Run tests
make prod            # Production deploy

---

🎉 Your Data Layer is Ready!

Simple. Reliable. Production-Ready.

Version: 1.0.0 | Last Updated: 2025-08-31