didi-lot2-backend/backend/services/data-layer/README.md

# DIDI Backend - Data Layer 🗄️

## Quick Start 🚀

```bash
# Recommended: Start via unified deployment manager
./deploy/didi.sh staging start
```

All data services start automatically as part of the staging deployment.

## What is the Data Layer? 🤔

The Data Layer provides **pure storage** for the DIDI Backend platform. No business logic, no routing, just reliable data storage.

## Architecture Overview 📊

```
┌─────────────────────────────────────────────────────────────────────────┐
│                         DATA LAYER                                      │
│                                                                         │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐ │
│  │              │  │              │  │              │  │              │ │
│  │  PostgreSQL  │  │    Redis     │  │    MinIO     │  │   RabbitMQ   │ │
│  │              │  │              │  │              │  │              │ │
│  │   Database   │  │    Cache     │  │   Storage    │  │    Queue     │ │
│  │   PgAdmin    │  │   Commander  │  │              │  │              │ │
│  └──────────────┘  └──────────────┘  └──────────────┘  └──────────────┘ │
│       Ports:            Ports:           Ports:            Ports:       │
│   22001 (DB)       22301 (Cache)    27000 (API)       23100 (AMQP)      │
│   29001 (UI)       29002 (UI)       27001 (Console)   23101 (UI)        │
│                                                                         │
│  ┌────────────────────────────────────────────────────────────────────┐ │
│  │         Network: didi-backend (shared by ALL services)             │ │
│  └────────────────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────────┘
```

> **Note**: Ports shown are for staging (the primary deployment method via `./deploy/didi.sh staging`).

## The Four Services + Management UIs 📦

### 1. **didiDatabase** (PostgreSQL) 🐘
**Purpose**: Persistent structured data storage
- Stores all application data
- 5 schemas (users, catalog, pipelines, execution, analyses)
- ACID compliant transactions
- Port: **22001** (staging)
- **pgAdmin** (Port 29001): Web UI for database management

### 2. **didiCache** (Redis) ⚡
**Purpose**: High-speed temporary storage
- Pipeline execution status
- Real-time updates
- Session data
- 24-hour TTL for most keys
- Port: **22301** (staging)
- **Redis Commander** (Port 29002): Web UI for Redis management

### 3. **didiStorage** (MinIO) 📁
**Purpose**: Object/file storage
- 7 auto-created buckets
- Media files (images, videos, audio)
- Documents and backups
- Auto-expiry policies
- Ports: **27000** (API), **27001** (Console) (staging)

### 4. **didiQueue** (RabbitMQ) 🐰
**Purpose**: Message queue for async processing
- Pipeline job queuing
- Decouples API from processing
- Single unified queue for all analysis types
- Durable message storage
- Ports: **23100** (AMQP), **23101** (Management UI) (staging)

## Quick Start - Entire Data Layer 🎯

### Start Everything
```bash
# From this directory
make up

# Or with Docker Compose
docker compose up -d
```

### Stop Everything
```bash
make down
```

### Check Status
```bash
make status
```

### View Logs
```bash
make logs
```

## Individual Service Management 🔧

### Start Individual Services
```bash
make up-database    # Start only PostgreSQL
make up-cache       # Start only Redis
make up-storage     # Start only MinIO
make up-queue       # Start only RabbitMQ
```

### Check Individual Health
```bash
make health-database
make health-cache
make health-storage
make health-queue
```

## Access Points 🌐 (Staging)

| Service | Type | Access URL | Credentials |
|---------|------|------------|-------------|
| PostgreSQL | Database | `localhost:22001` | postgres / postgres123 |
| pgAdmin | Web UI | `http://localhost:29001` | admin@example.com / admin123 |
| Redis | Cache | `localhost:22301` | Password: redis123 |
| Redis Commander | Web UI | `http://localhost:29002` | admin / commander123 |
| MinIO | API | `localhost:27000` | minioadmin / minio123 |
| MinIO | Console | `http://localhost:27001` | minioadmin / minio123 |
| RabbitMQ | AMQP | `localhost:23100` | admin / rabbitmq123 |
| RabbitMQ | Management UI | `http://localhost:23101` | admin / rabbitmq123 |

## What Gets Auto-Created? ✨

When you run `make up`:

### PostgreSQL
- ✅ 5 schemas
- ✅ All tables
- ✅ Indexes and triggers
- ✅ User subscription plans

### Redis
- ✅ Configured with password
- ✅ Persistence enabled
- ✅ Memory limits set
- ✅ Ready for connections

### MinIO
- ✅ 7 buckets created
- ✅ Versioning enabled
- ✅ Lifecycle policies
- ✅ Service access policies

### RabbitMQ
- ✅ Unified analysis queue created
- ✅ Dead letter queue configured
- ✅ Management plugin enabled
- ✅ Message TTL policies set

## Directory Structure 📂

```
data-layer/
├── README.md           # This file
├── docker-compose.yml  # Unified orchestration
├── Makefile           # Simple commands
│
├── didiDatabase/      # PostgreSQL Service
│   ├── docker-compose.yml
│   ├── init.sql       # Database schema
│   ├── .env          # Configuration
│   └── README.md     # Service docs
│
├── didiCache/        # Redis Service
│   ├── docker-compose.yml
│   ├── redis.conf   # Redis config
│   ├── .env         # Configuration
│   └── README.md    # Service docs
│
├── didiStorage/      # MinIO Service
│   ├── docker-compose.yml
│   ├── init-buckets.sh  # Auto-setup
│   ├── .env            # Configuration
│   └── README.md       # Service docs
│
└── didiQueue/        # RabbitMQ Service
    ├── docker-compose.yml
    ├── init-queues.sh  # Queue setup
    ├── .env           # Configuration
    └── README.md      # Service docs
```

## Environment Variables 🔐

Each service has its own `.env` file. **Change these for production!**

### Critical Passwords to Change:
- `POSTGRES_PASSWORD` in didiDatabase/.env
- `REDIS_PASSWORD` in didiCache/.env
- `HTTP_PASSWORD` for Redis Commander in docker-compose.yml
- `MINIO_ROOT_PASSWORD` in didiStorage/.env
- `RABBITMQ_PASSWORD` in didiQueue/.env

## Testing the Data Layer 🧪

```bash
# Run all tests
make test

# Test individual services
make test-database
make test-cache
make test-storage
make test-queue
```

## Production Deployment 🚀

```bash
# Check for default passwords
make check-security

# Deploy with production settings
make prod
```

## Troubleshooting 🔧

### Service won't start?
```bash
# Check logs
make logs-database
make logs-cache
make logs-storage
make logs-queue
```

### Port conflicts?
Edit the `.env` file in the service directory and change the port.

### Need a fresh start?
```bash
# WARNING: Deletes all data!
make clean
make up
```

## Resource Usage 📊

| Service | Memory Limit | CPU Limit | Disk Usage |
|---------|-------------|-----------|------------|
| PostgreSQL | 2GB | 1.0 CPU | ~500MB + data |
| Redis | 512MB | 0.5 CPU | ~100MB + cache |
| MinIO | 1GB | 0.5 CPU | ~200MB + files |
| RabbitMQ | 1GB | 0.5 CPU | ~100MB + messages |

**Total**: ~4.5GB RAM, 2.5 CPUs

## Network Architecture 🌐

All services communicate on the `didi-backend` network:
- **Shared by ALL backend services** (data layer, API layer, orchestration, etc.)
- Internal DNS resolution by service name
- Isolated from external access (except mapped ports)
- Services can reach each other by hostname
- Other Docker Compose projects can join this network using:
  ```yaml
  networks:
    default:
      external: true
      name: didi-backend
  ```

## Why This Architecture? 🎯

1. **Separation of Concerns**
   - Each service does ONE thing well
   - Easy to scale individually
   - Simple to understand

2. **Zero Configuration**
   - Everything auto-configures
   - No manual setup needed
   - Production-ready defaults

3. **Developer Friendly**
   - One command to start
   - Clear documentation
   - Web UIs included

## Next Steps 🏗️

The Data Layer is complete! Next layers to build:

```
✅ data-layer/        # Complete!
⏳ orchestration-layer/  # Next: RabbitMQ, Kong, Vault
⏳ service-layer/       # Then: Microservices
⏳ application-layer/   # Finally: UI/Apps
```

## Support & Maintenance 🛠️

### Daily Tasks
- Check logs: `make logs`
- Monitor usage: `make stats`
- Backup data: `make backup`

### Weekly Tasks
- Review disk usage
- Check for updates
- Rotate passwords

### Monthly Tasks
- Full backup
- Performance review
- Security audit

---

## Quick Reference Card 📋

```bash
# Essential Commands
make up              # Start everything
make down            # Stop everything
make status          # Check health
make logs            # View logs
make clean           # Delete all data

# Individual Services
make up-database     # Start PostgreSQL
make up-cache        # Start Redis
make up-storage      # Start MinIO

# Utilities
make backup          # Backup all data
make restore         # Restore from backup
make test            # Run tests
make prod            # Production deploy
```

---

**🎉 Your Data Layer is Ready!**

Simple. Reliable. Production-Ready.

*Version: 1.0.0 | Last Updated: 2025-08-31*