livrare lot 2
This commit is contained in:
commit
8ecc78e729
763 changed files with 164593 additions and 0 deletions
350
backend/services/data-layer/README.md
Normal file
350
backend/services/data-layer/README.md
Normal file
|
|
@ -0,0 +1,350 @@
|
|||
# DIDI Backend - Data Layer 🗄️
|
||||
|
||||
## Quick Start 🚀
|
||||
|
||||
```bash
|
||||
# Recommended: Start via unified deployment manager
|
||||
./deploy/didi.sh staging start
|
||||
```
|
||||
|
||||
All data services start automatically as part of the staging deployment.
|
||||
|
||||
## What is the Data Layer? 🤔
|
||||
|
||||
The Data Layer provides **pure storage** for the DIDI Backend platform. No business logic, no routing, just reliable data storage.
|
||||
|
||||
## Architecture Overview 📊
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────────────┐
|
||||
│ DATA LAYER │
|
||||
│ │
|
||||
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
|
||||
│ │ │ │ │ │ │ │ │ │
|
||||
│ │ PostgreSQL │ │ Redis │ │ MinIO │ │ RabbitMQ │ │
|
||||
│ │ │ │ │ │ │ │ │ │
|
||||
│ │ Database │ │ Cache │ │ Storage │ │ Queue │ │
|
||||
│ │ PgAdmin │ │ Commander │ │ │ │ │ │
|
||||
│ └──────────────┘ └──────────────┘ └──────────────┘ └──────────────┘ │
|
||||
│ Ports: Ports: Ports: Ports: │
|
||||
│ 22001 (DB) 22301 (Cache) 27000 (API) 23100 (AMQP) │
|
||||
│ 29001 (UI) 29002 (UI) 27001 (Console) 23101 (UI) │
|
||||
│ │
|
||||
│ ┌────────────────────────────────────────────────────────────────────┐ │
|
||||
│ │ Network: didi-backend (shared by ALL services) │ │
|
||||
│ └────────────────────────────────────────────────────────────────────┘ │
|
||||
└─────────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
> **Note**: Ports shown are for staging (the primary deployment method via `./deploy/didi.sh staging`).
|
||||
|
||||
## The Four Services + Management UIs 📦
|
||||
|
||||
### 1. **didiDatabase** (PostgreSQL) 🐘
|
||||
**Purpose**: Persistent structured data storage
|
||||
- Stores all application data
|
||||
- 5 schemas (users, catalog, pipelines, execution, analyses)
|
||||
- ACID compliant transactions
|
||||
- Port: **22001** (staging)
|
||||
- **pgAdmin** (Port 29001): Web UI for database management
|
||||
|
||||
### 2. **didiCache** (Redis) ⚡
|
||||
**Purpose**: High-speed temporary storage
|
||||
- Pipeline execution status
|
||||
- Real-time updates
|
||||
- Session data
|
||||
- 24-hour TTL for most keys
|
||||
- Port: **22301** (staging)
|
||||
- **Redis Commander** (Port 29002): Web UI for Redis management
|
||||
|
||||
### 3. **didiStorage** (MinIO) 📁
|
||||
**Purpose**: Object/file storage
|
||||
- 7 auto-created buckets
|
||||
- Media files (images, videos, audio)
|
||||
- Documents and backups
|
||||
- Auto-expiry policies
|
||||
- Ports: **27000** (API), **27001** (Console) (staging)
|
||||
|
||||
### 4. **didiQueue** (RabbitMQ) 🐰
|
||||
**Purpose**: Message queue for async processing
|
||||
- Pipeline job queuing
|
||||
- Decouples API from processing
|
||||
- Single unified queue for all analysis types
|
||||
- Durable message storage
|
||||
- Ports: **23100** (AMQP), **23101** (Management UI) (staging)
|
||||
|
||||
## Quick Start - Entire Data Layer 🎯
|
||||
|
||||
### Start Everything
|
||||
```bash
|
||||
# From this directory
|
||||
make up
|
||||
|
||||
# Or with Docker Compose
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
### Stop Everything
|
||||
```bash
|
||||
make down
|
||||
```
|
||||
|
||||
### Check Status
|
||||
```bash
|
||||
make status
|
||||
```
|
||||
|
||||
### View Logs
|
||||
```bash
|
||||
make logs
|
||||
```
|
||||
|
||||
## Individual Service Management 🔧
|
||||
|
||||
### Start Individual Services
|
||||
```bash
|
||||
make up-database # Start only PostgreSQL
|
||||
make up-cache # Start only Redis
|
||||
make up-storage # Start only MinIO
|
||||
make up-queue # Start only RabbitMQ
|
||||
```
|
||||
|
||||
### Check Individual Health
|
||||
```bash
|
||||
make health-database
|
||||
make health-cache
|
||||
make health-storage
|
||||
make health-queue
|
||||
```
|
||||
|
||||
## Access Points 🌐 (Staging)
|
||||
|
||||
| Service | Type | Access URL | Credentials |
|
||||
|---------|------|------------|-------------|
|
||||
| PostgreSQL | Database | `localhost:22001` | postgres / postgres123 |
|
||||
| pgAdmin | Web UI | `http://localhost:29001` | admin@example.com / admin123 |
|
||||
| Redis | Cache | `localhost:22301` | Password: redis123 |
|
||||
| Redis Commander | Web UI | `http://localhost:29002` | admin / commander123 |
|
||||
| MinIO | API | `localhost:27000` | minioadmin / minio123 |
|
||||
| MinIO | Console | `http://localhost:27001` | minioadmin / minio123 |
|
||||
| RabbitMQ | AMQP | `localhost:23100` | admin / rabbitmq123 |
|
||||
| RabbitMQ | Management UI | `http://localhost:23101` | admin / rabbitmq123 |
|
||||
|
||||
## What Gets Auto-Created? ✨
|
||||
|
||||
When you run `make up`:
|
||||
|
||||
### PostgreSQL
|
||||
- ✅ 5 schemas
|
||||
- ✅ All tables
|
||||
- ✅ Indexes and triggers
|
||||
- ✅ User subscription plans
|
||||
|
||||
### Redis
|
||||
- ✅ Configured with password
|
||||
- ✅ Persistence enabled
|
||||
- ✅ Memory limits set
|
||||
- ✅ Ready for connections
|
||||
|
||||
### MinIO
|
||||
- ✅ 7 buckets created
|
||||
- ✅ Versioning enabled
|
||||
- ✅ Lifecycle policies
|
||||
- ✅ Service access policies
|
||||
|
||||
### RabbitMQ
|
||||
- ✅ Unified analysis queue created
|
||||
- ✅ Dead letter queue configured
|
||||
- ✅ Management plugin enabled
|
||||
- ✅ Message TTL policies set
|
||||
|
||||
## Directory Structure 📂
|
||||
|
||||
```
|
||||
data-layer/
|
||||
├── README.md # This file
|
||||
├── docker-compose.yml # Unified orchestration
|
||||
├── Makefile # Simple commands
|
||||
│
|
||||
├── didiDatabase/ # PostgreSQL Service
|
||||
│ ├── docker-compose.yml
|
||||
│ ├── init.sql # Database schema
|
||||
│ ├── .env # Configuration
|
||||
│ └── README.md # Service docs
|
||||
│
|
||||
├── didiCache/ # Redis Service
|
||||
│ ├── docker-compose.yml
|
||||
│ ├── redis.conf # Redis config
|
||||
│ ├── .env # Configuration
|
||||
│ └── README.md # Service docs
|
||||
│
|
||||
├── didiStorage/ # MinIO Service
|
||||
│ ├── docker-compose.yml
|
||||
│ ├── init-buckets.sh # Auto-setup
|
||||
│ ├── .env # Configuration
|
||||
│ └── README.md # Service docs
|
||||
│
|
||||
└── didiQueue/ # RabbitMQ Service
|
||||
├── docker-compose.yml
|
||||
├── init-queues.sh # Queue setup
|
||||
├── .env # Configuration
|
||||
└── README.md # Service docs
|
||||
```
|
||||
|
||||
## Environment Variables 🔐
|
||||
|
||||
Each service has its own `.env` file. **Change these for production!**
|
||||
|
||||
### Critical Passwords to Change:
|
||||
- `POSTGRES_PASSWORD` in didiDatabase/.env
|
||||
- `REDIS_PASSWORD` in didiCache/.env
|
||||
- `HTTP_PASSWORD` for Redis Commander in docker-compose.yml
|
||||
- `MINIO_ROOT_PASSWORD` in didiStorage/.env
|
||||
- `RABBITMQ_PASSWORD` in didiQueue/.env
|
||||
|
||||
## Testing the Data Layer 🧪
|
||||
|
||||
```bash
|
||||
# Run all tests
|
||||
make test
|
||||
|
||||
# Test individual services
|
||||
make test-database
|
||||
make test-cache
|
||||
make test-storage
|
||||
make test-queue
|
||||
```
|
||||
|
||||
## Production Deployment 🚀
|
||||
|
||||
```bash
|
||||
# Check for default passwords
|
||||
make check-security
|
||||
|
||||
# Deploy with production settings
|
||||
make prod
|
||||
```
|
||||
|
||||
## Troubleshooting 🔧
|
||||
|
||||
### Service won't start?
|
||||
```bash
|
||||
# Check logs
|
||||
make logs-database
|
||||
make logs-cache
|
||||
make logs-storage
|
||||
make logs-queue
|
||||
```
|
||||
|
||||
### Port conflicts?
|
||||
Edit the `.env` file in the service directory and change the port.
|
||||
|
||||
### Need a fresh start?
|
||||
```bash
|
||||
# WARNING: Deletes all data!
|
||||
make clean
|
||||
make up
|
||||
```
|
||||
|
||||
## Resource Usage 📊
|
||||
|
||||
| Service | Memory Limit | CPU Limit | Disk Usage |
|
||||
|---------|-------------|-----------|------------|
|
||||
| PostgreSQL | 2GB | 1.0 CPU | ~500MB + data |
|
||||
| Redis | 512MB | 0.5 CPU | ~100MB + cache |
|
||||
| MinIO | 1GB | 0.5 CPU | ~200MB + files |
|
||||
| RabbitMQ | 1GB | 0.5 CPU | ~100MB + messages |
|
||||
|
||||
**Total**: ~4.5GB RAM, 2.5 CPUs
|
||||
|
||||
## Network Architecture 🌐
|
||||
|
||||
All services communicate on the `didi-backend` network:
|
||||
- **Shared by ALL backend services** (data layer, API layer, orchestration, etc.)
|
||||
- Internal DNS resolution by service name
|
||||
- Isolated from external access (except mapped ports)
|
||||
- Services can reach each other by hostname
|
||||
- Other Docker Compose projects can join this network using:
|
||||
```yaml
|
||||
networks:
|
||||
default:
|
||||
external: true
|
||||
name: didi-backend
|
||||
```
|
||||
|
||||
## Why This Architecture? 🎯
|
||||
|
||||
1. **Separation of Concerns**
|
||||
- Each service does ONE thing well
|
||||
- Easy to scale individually
|
||||
- Simple to understand
|
||||
|
||||
2. **Zero Configuration**
|
||||
- Everything auto-configures
|
||||
- No manual setup needed
|
||||
- Production-ready defaults
|
||||
|
||||
3. **Developer Friendly**
|
||||
- One command to start
|
||||
- Clear documentation
|
||||
- Web UIs included
|
||||
|
||||
## Next Steps 🏗️
|
||||
|
||||
The Data Layer is complete! Next layers to build:
|
||||
|
||||
```
|
||||
✅ data-layer/ # Complete!
|
||||
⏳ orchestration-layer/ # Next: RabbitMQ, Kong, Vault
|
||||
⏳ service-layer/ # Then: Microservices
|
||||
⏳ application-layer/ # Finally: UI/Apps
|
||||
```
|
||||
|
||||
## Support & Maintenance 🛠️
|
||||
|
||||
### Daily Tasks
|
||||
- Check logs: `make logs`
|
||||
- Monitor usage: `make stats`
|
||||
- Backup data: `make backup`
|
||||
|
||||
### Weekly Tasks
|
||||
- Review disk usage
|
||||
- Check for updates
|
||||
- Rotate passwords
|
||||
|
||||
### Monthly Tasks
|
||||
- Full backup
|
||||
- Performance review
|
||||
- Security audit
|
||||
|
||||
---
|
||||
|
||||
## Quick Reference Card 📋
|
||||
|
||||
```bash
|
||||
# Essential Commands
|
||||
make up # Start everything
|
||||
make down # Stop everything
|
||||
make status # Check health
|
||||
make logs # View logs
|
||||
make clean # Delete all data
|
||||
|
||||
# Individual Services
|
||||
make up-database # Start PostgreSQL
|
||||
make up-cache # Start Redis
|
||||
make up-storage # Start MinIO
|
||||
|
||||
# Utilities
|
||||
make backup # Backup all data
|
||||
make restore # Restore from backup
|
||||
make test # Run tests
|
||||
make prod # Production deploy
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
**🎉 Your Data Layer is Ready!**
|
||||
|
||||
Simple. Reliable. Production-Ready.
|
||||
|
||||
*Version: 1.0.0 | Last Updated: 2025-08-31*
|
||||
Loading…
Add table
Add a link
Reference in a new issue