livrare lot 2
This commit is contained in:
commit
8ecc78e729
763 changed files with 164593 additions and 0 deletions
16
backend/services/data-layer/.env.example
Normal file
16
backend/services/data-layer/.env.example
Normal file
|
|
@ -0,0 +1,16 @@
|
|||
# ============================================================================
|
||||
# DIDI Data Layer - Master Configuration
|
||||
# ============================================================================
|
||||
# This file is for future use when we need unified configuration
|
||||
# For now, each service has its own .env file in its directory
|
||||
|
||||
# Docker Compose Project Name
|
||||
COMPOSE_PROJECT_NAME=didibackend
|
||||
|
||||
# Network Configuration
|
||||
NETWORK_SUBNET=172.28.0.0/16
|
||||
|
||||
# Future: Unified credentials management
|
||||
# DATABASE_PASSWORD=change_me
|
||||
# CACHE_PASSWORD=change_me
|
||||
# STORAGE_PASSWORD=change_me
|
||||
350
backend/services/data-layer/README.md
Normal file
350
backend/services/data-layer/README.md
Normal file
|
|
@ -0,0 +1,350 @@
|
|||
# DIDI Backend - Data Layer 🗄️
|
||||
|
||||
## Quick Start 🚀
|
||||
|
||||
```bash
|
||||
# Recommended: Start via unified deployment manager
|
||||
./deploy/didi.sh staging start
|
||||
```
|
||||
|
||||
All data services start automatically as part of the staging deployment.
|
||||
|
||||
## What is the Data Layer? 🤔
|
||||
|
||||
The Data Layer provides **pure storage** for the DIDI Backend platform. No business logic, no routing, just reliable data storage.
|
||||
|
||||
## Architecture Overview 📊
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────────────┐
|
||||
│ DATA LAYER │
|
||||
│ │
|
||||
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
|
||||
│ │ │ │ │ │ │ │ │ │
|
||||
│ │ PostgreSQL │ │ Redis │ │ MinIO │ │ RabbitMQ │ │
|
||||
│ │ │ │ │ │ │ │ │ │
|
||||
│ │ Database │ │ Cache │ │ Storage │ │ Queue │ │
|
||||
│ │ PgAdmin │ │ Commander │ │ │ │ │ │
|
||||
│ └──────────────┘ └──────────────┘ └──────────────┘ └──────────────┘ │
|
||||
│ Ports: Ports: Ports: Ports: │
|
||||
│ 22001 (DB) 22301 (Cache) 27000 (API) 23100 (AMQP) │
|
||||
│ 29001 (UI) 29002 (UI) 27001 (Console) 23101 (UI) │
|
||||
│ │
|
||||
│ ┌────────────────────────────────────────────────────────────────────┐ │
|
||||
│ │ Network: didi-backend (shared by ALL services) │ │
|
||||
│ └────────────────────────────────────────────────────────────────────┘ │
|
||||
└─────────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
> **Note**: Ports shown are for staging (the primary deployment method via `./deploy/didi.sh staging`).
|
||||
|
||||
## The Four Services + Management UIs 📦
|
||||
|
||||
### 1. **didiDatabase** (PostgreSQL) 🐘
|
||||
**Purpose**: Persistent structured data storage
|
||||
- Stores all application data
|
||||
- 5 schemas (users, catalog, pipelines, execution, analyses)
|
||||
- ACID compliant transactions
|
||||
- Port: **22001** (staging)
|
||||
- **pgAdmin** (Port 29001): Web UI for database management
|
||||
|
||||
### 2. **didiCache** (Redis) ⚡
|
||||
**Purpose**: High-speed temporary storage
|
||||
- Pipeline execution status
|
||||
- Real-time updates
|
||||
- Session data
|
||||
- 24-hour TTL for most keys
|
||||
- Port: **22301** (staging)
|
||||
- **Redis Commander** (Port 29002): Web UI for Redis management
|
||||
|
||||
### 3. **didiStorage** (MinIO) 📁
|
||||
**Purpose**: Object/file storage
|
||||
- 7 auto-created buckets
|
||||
- Media files (images, videos, audio)
|
||||
- Documents and backups
|
||||
- Auto-expiry policies
|
||||
- Ports: **27000** (API), **27001** (Console) (staging)
|
||||
|
||||
### 4. **didiQueue** (RabbitMQ) 🐰
|
||||
**Purpose**: Message queue for async processing
|
||||
- Pipeline job queuing
|
||||
- Decouples API from processing
|
||||
- Single unified queue for all analysis types
|
||||
- Durable message storage
|
||||
- Ports: **23100** (AMQP), **23101** (Management UI) (staging)
|
||||
|
||||
## Quick Start - Entire Data Layer 🎯
|
||||
|
||||
### Start Everything
|
||||
```bash
|
||||
# From this directory
|
||||
make up
|
||||
|
||||
# Or with Docker Compose
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
### Stop Everything
|
||||
```bash
|
||||
make down
|
||||
```
|
||||
|
||||
### Check Status
|
||||
```bash
|
||||
make status
|
||||
```
|
||||
|
||||
### View Logs
|
||||
```bash
|
||||
make logs
|
||||
```
|
||||
|
||||
## Individual Service Management 🔧
|
||||
|
||||
### Start Individual Services
|
||||
```bash
|
||||
make up-database # Start only PostgreSQL
|
||||
make up-cache # Start only Redis
|
||||
make up-storage # Start only MinIO
|
||||
make up-queue # Start only RabbitMQ
|
||||
```
|
||||
|
||||
### Check Individual Health
|
||||
```bash
|
||||
make health-database
|
||||
make health-cache
|
||||
make health-storage
|
||||
make health-queue
|
||||
```
|
||||
|
||||
## Access Points 🌐 (Staging)
|
||||
|
||||
| Service | Type | Access URL | Credentials |
|
||||
|---------|------|------------|-------------|
|
||||
| PostgreSQL | Database | `localhost:22001` | postgres / postgres123 |
|
||||
| pgAdmin | Web UI | `http://localhost:29001` | admin@example.com / admin123 |
|
||||
| Redis | Cache | `localhost:22301` | Password: redis123 |
|
||||
| Redis Commander | Web UI | `http://localhost:29002` | admin / commander123 |
|
||||
| MinIO | API | `localhost:27000` | minioadmin / minio123 |
|
||||
| MinIO | Console | `http://localhost:27001` | minioadmin / minio123 |
|
||||
| RabbitMQ | AMQP | `localhost:23100` | admin / rabbitmq123 |
|
||||
| RabbitMQ | Management UI | `http://localhost:23101` | admin / rabbitmq123 |
|
||||
|
||||
## What Gets Auto-Created? ✨
|
||||
|
||||
When you run `make up`:
|
||||
|
||||
### PostgreSQL
|
||||
- ✅ 5 schemas
|
||||
- ✅ All tables
|
||||
- ✅ Indexes and triggers
|
||||
- ✅ User subscription plans
|
||||
|
||||
### Redis
|
||||
- ✅ Configured with password
|
||||
- ✅ Persistence enabled
|
||||
- ✅ Memory limits set
|
||||
- ✅ Ready for connections
|
||||
|
||||
### MinIO
|
||||
- ✅ 7 buckets created
|
||||
- ✅ Versioning enabled
|
||||
- ✅ Lifecycle policies
|
||||
- ✅ Service access policies
|
||||
|
||||
### RabbitMQ
|
||||
- ✅ Unified analysis queue created
|
||||
- ✅ Dead letter queue configured
|
||||
- ✅ Management plugin enabled
|
||||
- ✅ Message TTL policies set
|
||||
|
||||
## Directory Structure 📂
|
||||
|
||||
```
|
||||
data-layer/
|
||||
├── README.md # This file
|
||||
├── docker-compose.yml # Unified orchestration
|
||||
├── Makefile # Simple commands
|
||||
│
|
||||
├── didiDatabase/ # PostgreSQL Service
|
||||
│ ├── docker-compose.yml
|
||||
│ ├── init.sql # Database schema
|
||||
│ ├── .env # Configuration
|
||||
│ └── README.md # Service docs
|
||||
│
|
||||
├── didiCache/ # Redis Service
|
||||
│ ├── docker-compose.yml
|
||||
│ ├── redis.conf # Redis config
|
||||
│ ├── .env # Configuration
|
||||
│ └── README.md # Service docs
|
||||
│
|
||||
├── didiStorage/ # MinIO Service
|
||||
│ ├── docker-compose.yml
|
||||
│ ├── init-buckets.sh # Auto-setup
|
||||
│ ├── .env # Configuration
|
||||
│ └── README.md # Service docs
|
||||
│
|
||||
└── didiQueue/ # RabbitMQ Service
|
||||
├── docker-compose.yml
|
||||
├── init-queues.sh # Queue setup
|
||||
├── .env # Configuration
|
||||
└── README.md # Service docs
|
||||
```
|
||||
|
||||
## Environment Variables 🔐
|
||||
|
||||
Each service has its own `.env` file. **Change these for production!**
|
||||
|
||||
### Critical Passwords to Change:
|
||||
- `POSTGRES_PASSWORD` in didiDatabase/.env
|
||||
- `REDIS_PASSWORD` in didiCache/.env
|
||||
- `HTTP_PASSWORD` for Redis Commander in docker-compose.yml
|
||||
- `MINIO_ROOT_PASSWORD` in didiStorage/.env
|
||||
- `RABBITMQ_PASSWORD` in didiQueue/.env
|
||||
|
||||
## Testing the Data Layer 🧪
|
||||
|
||||
```bash
|
||||
# Run all tests
|
||||
make test
|
||||
|
||||
# Test individual services
|
||||
make test-database
|
||||
make test-cache
|
||||
make test-storage
|
||||
make test-queue
|
||||
```
|
||||
|
||||
## Production Deployment 🚀
|
||||
|
||||
```bash
|
||||
# Check for default passwords
|
||||
make check-security
|
||||
|
||||
# Deploy with production settings
|
||||
make prod
|
||||
```
|
||||
|
||||
## Troubleshooting 🔧
|
||||
|
||||
### Service won't start?
|
||||
```bash
|
||||
# Check logs
|
||||
make logs-database
|
||||
make logs-cache
|
||||
make logs-storage
|
||||
make logs-queue
|
||||
```
|
||||
|
||||
### Port conflicts?
|
||||
Edit the `.env` file in the service directory and change the port.
|
||||
|
||||
### Need a fresh start?
|
||||
```bash
|
||||
# WARNING: Deletes all data!
|
||||
make clean
|
||||
make up
|
||||
```
|
||||
|
||||
## Resource Usage 📊
|
||||
|
||||
| Service | Memory Limit | CPU Limit | Disk Usage |
|
||||
|---------|-------------|-----------|------------|
|
||||
| PostgreSQL | 2GB | 1.0 CPU | ~500MB + data |
|
||||
| Redis | 512MB | 0.5 CPU | ~100MB + cache |
|
||||
| MinIO | 1GB | 0.5 CPU | ~200MB + files |
|
||||
| RabbitMQ | 1GB | 0.5 CPU | ~100MB + messages |
|
||||
|
||||
**Total**: ~4.5GB RAM, 2.5 CPUs
|
||||
|
||||
## Network Architecture 🌐
|
||||
|
||||
All services communicate on the `didi-backend` network:
|
||||
- **Shared by ALL backend services** (data layer, API layer, orchestration, etc.)
|
||||
- Internal DNS resolution by service name
|
||||
- Isolated from external access (except mapped ports)
|
||||
- Services can reach each other by hostname
|
||||
- Other Docker Compose projects can join this network using:
|
||||
```yaml
|
||||
networks:
|
||||
default:
|
||||
external: true
|
||||
name: didi-backend
|
||||
```
|
||||
|
||||
## Why This Architecture? 🎯
|
||||
|
||||
1. **Separation of Concerns**
|
||||
- Each service does ONE thing well
|
||||
- Easy to scale individually
|
||||
- Simple to understand
|
||||
|
||||
2. **Zero Configuration**
|
||||
- Everything auto-configures
|
||||
- No manual setup needed
|
||||
- Production-ready defaults
|
||||
|
||||
3. **Developer Friendly**
|
||||
- One command to start
|
||||
- Clear documentation
|
||||
- Web UIs included
|
||||
|
||||
## Next Steps 🏗️
|
||||
|
||||
The Data Layer is complete! Next layers to build:
|
||||
|
||||
```
|
||||
✅ data-layer/ # Complete!
|
||||
⏳ orchestration-layer/ # Next: RabbitMQ, Kong, Vault
|
||||
⏳ service-layer/ # Then: Microservices
|
||||
⏳ application-layer/ # Finally: UI/Apps
|
||||
```
|
||||
|
||||
## Support & Maintenance 🛠️
|
||||
|
||||
### Daily Tasks
|
||||
- Check logs: `make logs`
|
||||
- Monitor usage: `make stats`
|
||||
- Backup data: `make backup`
|
||||
|
||||
### Weekly Tasks
|
||||
- Review disk usage
|
||||
- Check for updates
|
||||
- Rotate passwords
|
||||
|
||||
### Monthly Tasks
|
||||
- Full backup
|
||||
- Performance review
|
||||
- Security audit
|
||||
|
||||
---
|
||||
|
||||
## Quick Reference Card 📋
|
||||
|
||||
```bash
|
||||
# Essential Commands
|
||||
make up # Start everything
|
||||
make down # Stop everything
|
||||
make status # Check health
|
||||
make logs # View logs
|
||||
make clean # Delete all data
|
||||
|
||||
# Individual Services
|
||||
make up-database # Start PostgreSQL
|
||||
make up-cache # Start Redis
|
||||
make up-storage # Start MinIO
|
||||
|
||||
# Utilities
|
||||
make backup # Backup all data
|
||||
make restore # Restore from backup
|
||||
make test # Run tests
|
||||
make prod # Production deploy
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
**🎉 Your Data Layer is Ready!**
|
||||
|
||||
Simple. Reliable. Production-Ready.
|
||||
|
||||
*Version: 1.0.0 | Last Updated: 2025-08-31*
|
||||
36
backend/services/data-layer/didiCache/.env.example
Normal file
36
backend/services/data-layer/didiCache/.env.example
Normal file
|
|
@ -0,0 +1,36 @@
|
|||
# ============================================================================
|
||||
# didiCache Environment Configuration
|
||||
# ============================================================================
|
||||
# Copy this file to .env and update with your values
|
||||
|
||||
# Docker Compose Project Name (groups containers in Docker Desktop)
|
||||
COMPOSE_PROJECT_NAME=didibackend_datalayer
|
||||
|
||||
# Redis Configuration
|
||||
REDIS_PASSWORD=YOUR_SECURE_PASSWORD_HERE
|
||||
REDIS_PORT=6380 # Using 6380 to avoid conflict with default 6379
|
||||
|
||||
# Memory Limits
|
||||
REDIS_MEMORY_LIMIT=512M
|
||||
REDIS_MEMORY_RESERVATION=256M
|
||||
|
||||
# Performance Tuning
|
||||
REDIS_MAXMEMORY=512mb
|
||||
REDIS_MAXMEMORY_POLICY=allkeys-lru
|
||||
REDIS_TIMEOUT=0
|
||||
REDIS_TCP_KEEPALIVE=300
|
||||
REDIS_DATABASES=16
|
||||
|
||||
# Persistence Configuration
|
||||
REDIS_SAVE="900 1 300 10 60 10000"
|
||||
REDIS_APPENDONLY=yes
|
||||
REDIS_APPENDFSYNC=everysec
|
||||
|
||||
# Logging
|
||||
REDIS_LOGLEVEL=notice
|
||||
|
||||
# Client Limits
|
||||
REDIS_MAXCLIENTS=10000
|
||||
|
||||
# Timezone
|
||||
TZ=UTC
|
||||
26
backend/services/data-layer/didiCache/.gitignore
vendored
Normal file
26
backend/services/data-layer/didiCache/.gitignore
vendored
Normal file
|
|
@ -0,0 +1,26 @@
|
|||
# Environment variables
|
||||
.env
|
||||
.env.local
|
||||
|
||||
# Data directory
|
||||
data/
|
||||
*.rdb
|
||||
*.aof
|
||||
|
||||
# Logs
|
||||
*.log
|
||||
|
||||
# OS files
|
||||
.DS_Store
|
||||
Thumbs.db
|
||||
|
||||
# IDE files
|
||||
.idea/
|
||||
.vscode/
|
||||
*.swp
|
||||
*.swo
|
||||
|
||||
# Backup files
|
||||
*.bak
|
||||
*.backup
|
||||
*.old
|
||||
216
backend/services/data-layer/didiCache/INDEX.md
Normal file
216
backend/services/data-layer/didiCache/INDEX.md
Normal file
|
|
@ -0,0 +1,216 @@
|
|||
# didiCache - Index
|
||||
|
||||
**Productia ruleaza pe Redis LOCAL** (container `didi-cache` pe `didi-network`). Decizie: stabilitate + zero dependinte externe + acces rapid. Clusterul Redis RAG (managed extern, HAProxy VIP peste rag01/02/03) ramane configurat ca **fallback de urgenta pentru HA** — activabil cu `redis-switch.sh cluster`, nu este folosit operational acum.
|
||||
|
||||
**Productie (LOCAL — activ)**:
|
||||
- Container: `didi-cache` (imagine `redis:7-alpine`)
|
||||
- Host intern Docker: `didi-cache:6379`
|
||||
- DB: `0`
|
||||
- Parola: `REDIS_PASSWORD` din `.env` (default `redis123` in dev)
|
||||
- Memorie: 512MB (eviction: allkeys-lru)
|
||||
- Retea: `didi-network`; portul `6379` este expus pe host (`0.0.0.0:6379->6379`)
|
||||
- Persistenta: RDB + AOF
|
||||
|
||||
**Fallback HA (cluster RAG — disponibil dar inactiv)**:
|
||||
- Host: `10.11.50.100` (HAProxy VIP)
|
||||
- Port: `16379`
|
||||
- DB: `0`
|
||||
- User: `didi`
|
||||
- Parola: din `.cluster-credentials.env` (gitignored)
|
||||
|
||||
**Switch intre LOCAL si cluster**: `backend/services/orchestration-layer/scripts/redis-switch.sh {cluster|local|status}` rescrie fisierele `.env` ale serviciilor (agent-v3, didiFramework) + restart containere. Status curent: `local`.
|
||||
|
||||
**Helper de conectare folosit de toate serviciile**:
|
||||
- agent-v3 -> `src/shared/redis/connection.ts` (`createRedisConnection()`)
|
||||
- didiFramework -> `src/config/redis.ts` (`createRedisConnection()`)
|
||||
|
||||
---
|
||||
|
||||
## Ce face
|
||||
|
||||
Stocheaza date temporare si configurari pentru platforma DIDI:
|
||||
- Cache framework (parametri tehnici, verdicts, ponderi) -- scrise de didiFramework via sync-redis
|
||||
- Configurare HIL Moderation (triage thresholds, brain client config, sensitive topics, roluri) -- scrise de didiFramework
|
||||
- Sesiuni analiza (AnalysisSession JSON, TTL 7 zile) -- scrise de agent-v3
|
||||
- Rezultate intermediare per etapa (TTL 7 zile) -- scrise de agent-v3
|
||||
- Stare coada async (progres workeri) -- scrise de agent-v3
|
||||
- Lock-uri concurenta workeri (TTL 30s-5min) -- scrise de agent-v3
|
||||
- Chei API extensie browser (cache validare) -- scrise de didiFramework
|
||||
- Configurare modele viziune -- scrise de didiFramework
|
||||
|
||||
**Sync-redis scrie ~51 chei de configurare** (crescut de la 48 dupa adaugarea cheilor HIL Moderation in 2026-05-01). Plus `framework_keys`: 7 chei standard sau 8 daca include `providers`.
|
||||
|
||||
---
|
||||
|
||||
## Cine scrie in Redis
|
||||
|
||||
| Serviciu | Ce scrie | Chei Redis |
|
||||
|----------|----------|------------|
|
||||
| didiFramework (sync-redis) | Ierarhie tehnici, claims, verdicts, ponderi, surse, provideri | didi:framework:* |
|
||||
| didiFramework (sync-redis) | Configurare componente pe etape | didi:config:* |
|
||||
| didiFramework (extension-keys) | Cache validare chei API extensie | didi:extension:key:* |
|
||||
| agent-v3 (sesiuni) | Sesiune completa JSON | didi:pipeline:{sessionId}:status |
|
||||
| agent-v3 (executori) | Rezultate intermediare per etapa | agent:result:{sessionId}:{component}:{stage} |
|
||||
| agent-v3 (coada) | Stare procesare async | didi:queue:session:{sessionId} |
|
||||
| agent-v3 (workeri) | Lock-uri concurenta | didi:queue:lock:* |
|
||||
|
||||
## Cine citeste din Redis
|
||||
|
||||
| Serviciu | Ce citeste | Chei Redis |
|
||||
|----------|-----------|------------|
|
||||
| agent-v3 (executori) | Configurare framework (tehnici, ponderi, verdicts) | didi:framework:* |
|
||||
| agent-v3 (executori) | Modele disponibile, stage assignments, prompturi | didi:config:* |
|
||||
| agent-v3 (rute) | Sesiuni pentru polling status | didi:pipeline:{sessionId}:status |
|
||||
| agent-v3 (rute) | Rezultate intermediare | agent:result:{sessionId}:* |
|
||||
| agent-v3 (pipeline) | Validare cheie extensie | didi:extension:key:* |
|
||||
| didiFramework (debug) | Verificare date sincronizate | didi:framework:* |
|
||||
|
||||
---
|
||||
|
||||
## Chei Redis principale
|
||||
|
||||
| Pattern | Scop | TTL | Scris de |
|
||||
|---------|------|-----|----------|
|
||||
| didi:framework:manifest | Index categorii + timestamp sync | permanent | didiFramework |
|
||||
| didi:framework:techniques | Ierarhie completa tehnici (denormalizata) | permanent | didiFramework |
|
||||
| didi:framework:claims | Parametri claims (tipuri, statusuri, confidence) | permanent | didiFramework |
|
||||
| didi:framework:verdicts | Categorii verdict + risk mappings + severity | permanent | didiFramework |
|
||||
| didi:framework:weights | Ponderi componente + scenarii + multiplicatori | permanent | didiFramework |
|
||||
| didi:framework:sources | Evaluare surse | permanent | didiFramework |
|
||||
| didi:framework:providers | Configurare LLM | permanent | didiFramework |
|
||||
| didi:framework:dimensions_compact | Lista compacta dimensiuni (screening) | permanent | didiFramework |
|
||||
| didi:config:{component}:v1:* | Config componenta (modele, etape, prompturi) | permanent | didiFramework |
|
||||
| didi:config:techniques:v3:stage_assignments | **TIER-NESTED** `{stage: {free: {models}, premium: {models}}}` — chain-uri LLM per tier | permanent | didiFramework |
|
||||
| didi:config:ai-tampered:v1:stage_assignments | **TIER-NESTED** stage assignments AI-Tampered | permanent | didiFramework |
|
||||
| didi:config:claims:v1:stage_assignments | **TIER-NESTED** stage assignments Claims | permanent | didiFramework |
|
||||
| didi:config:source-assessment:v1:stage_assignments | **TIER-NESTED** stage assignments Source Assessment | permanent | didiFramework |
|
||||
| didi:config:vision:v1:stage_assignments | **TIER-NESTED** (Etapa 4) stage `image_analysis` — citit de `callVision()` cu tier param | permanent | didiFramework |
|
||||
| didi:config:verdict:v1:stage_assignments | **TIER-NESTED** (Etapa 5) stage `verdict_review` — citit de `verdict-explanation.ts loadModels(tier)` | permanent | didiFramework |
|
||||
| didi:config:ai-tampered:v1:vision_models | Legacy flat vision config (fallback pentru vision.ts daca `didi:config:vision:v1:stage_assignments` lipseste) | permanent | didiFramework |
|
||||
| didi:config:vision:v1:prompts:extraction | Prompt viziune: extragere text din imagini | permanent | didiFramework |
|
||||
| didi:config:vision:v1:prompts:video_frames | Prompt viziune: analiza cadre video | permanent | didiFramework |
|
||||
| didi:config:vision:v1:prompts:ai_detection | Prompt viziune: detectie AI imagini | permanent | didiFramework |
|
||||
| didi:config:pipeline:v1:component_config | Config componente pipeline + video track weights | permanent | didiFramework |
|
||||
| didi:config:pipeline:v1:session_config | Config sesiune pipeline | permanent | didiFramework |
|
||||
| didi:config:pipeline:v1:verdict_config | Override-uri verdict, synergy, confidence (globale). JSONB sincronizat din PG `bos_parammgmt.component_config WHERE (component_code='pipeline', config_key='verdict_config')`. Contine: `synergy` + override-uri (`false_claims`, `severe_techniques`, `undisclosed_ai`, `untrusted_domain`, `domain_red_flags`) + `confidence` + `confidence_levels`. Editabil din admin dashboard via `/api/verdicts/runtime-config` (GET/PUT/PATCH). | permanent | didiFramework |
|
||||
| didi:config:pipeline:v1:input_profiles | **Profiluri verdict per input type (6 profile: ponderi, override-uri, INCONCLUSIVE)** | permanent | didiFramework |
|
||||
| didi:config:techniques:v3:scoring_config | Parametri scoring techniques (count_scaler, intensity, severe_threshold) | permanent | didiFramework |
|
||||
| didi:config:ai-tampered:v1:scoring_config | Parametri scoring AI (blend_weights, disclosure_impact, thresholds) | permanent | didiFramework |
|
||||
| didi:config:claims:v1:scoring_config | Parametri scoring claims (status_weights, unverified behavior) | permanent | didiFramework |
|
||||
| didi:config:source-assessment:v1:scoring_config | Parametri scoring source (axis_weights, verdict_thresholds) | permanent | didiFramework |
|
||||
| didi:config:source-assessment:v1:available_models | Modele LLM source assessment (union free+premium) | permanent | didiFramework |
|
||||
| didi:config:source-assessment:v1:prompts:extraction | Prompt extractie metadata sursa | permanent | didiFramework |
|
||||
| didi:config:source-assessment:v1:prompts:evaluation | Prompt evaluare sursa | permanent | didiFramework |
|
||||
| didi:config:verdict:v1:available_models | Legacy fallback pentru verdict reviewer (folosit daca `stage_assignments` lipseste) | permanent | didiFramework |
|
||||
|
||||
### HIL Moderation (synced from PG by sync-redis)
|
||||
|
||||
Adaugat in 2026-05-01. Citit de agent-v3 in `triage.ts` (cache local 60s) si `brain/client.ts` pentru configurare live.
|
||||
|
||||
| Pattern | Scop | TTL | Scris de |
|
||||
|---------|------|-----|----------|
|
||||
| didi:config:moderation:v1:settings | Single-row config: triage thresholds + brain client config (brain_enabled, brain_url, lookup/write timeouts, brain_confidence_min_silver, brain_semantic_threshold, brain_per_component) | permanent | didiFramework |
|
||||
| didi:config:moderation:v1:sensitive_topics | Active topics list (elections, health, war, covid, climate) | permanent | didiFramework |
|
||||
| didi:config:moderation:v1:roles | Keycloak role -> permissions (moderator, senior_moderator) cu toggle flags | permanent | didiFramework |
|
||||
|
||||
**Nota structura tier-nested** (aplicabila tuturor cheilor `stage_assignments`):
|
||||
```json
|
||||
{
|
||||
"techniques_screening": {
|
||||
"free": { "models": [{"order":1,"model_key":"qwen35:Qwen3.5-397B-A17B",...}, ...] },
|
||||
"premium": { "models": [{"order":1,"model_key":"openrouter:google/gemini-3-flash-preview",...}, ...] }
|
||||
},
|
||||
"techniques_deep": { "free": {...}, "premium": {...} }
|
||||
}
|
||||
```
|
||||
|
||||
Sync-redis scrie cheia intr-un singur SET (atomic). agent-v3 citeste cheia o singura data, apoi rezolva tier-ul local cu `stages[stageCode][tier] || stages[stageCode].free`.
|
||||
| didi:pipeline:{sessionId}:status | Status executie pipeline (JSON: PipelineStatus) | 7 zile | agent-v3 |
|
||||
| didi:pipeline:{sessionId}:{component} | Rezultat componenta (JSON: TechniquesResult etc.) | 7 zile | agent-v3 |
|
||||
| didi:pipeline:{sessionId}:verdict | Rezultat verdict final (JSON: VerdictResult) | 7 zile | agent-v3 |
|
||||
| didi:pipeline:history:entry:{sessionId} | Date intrare sesiune (input + metadata) | 7 zile | agent-v3 |
|
||||
| didi:pipeline:history:user:{userId} | Sorted set istoric utilizator (score=timestamp) | 7 zile | agent-v3 |
|
||||
| agent:result:{sessionId}:{comp}:{stage} | Rezultat intermediar etapa | 7 zile | agent-v3 |
|
||||
| didi:queue:session:{sessionId} | Stare coada async | 24 ore | agent-v3 |
|
||||
| didi:queue:lock:{sessionId}:{comp} | Lock worker per componenta | 5 min | agent-v3 |
|
||||
| didi:queue:aggregator:{sessionId} | Lock agregator verdict | 30s | agent-v3 |
|
||||
| didi:extension:key:{key} | Cache cheie API extensie | permanent | didiFramework |
|
||||
|
||||
---
|
||||
|
||||
## Configurare
|
||||
|
||||
### redis.conf
|
||||
|
||||
- Bind: 0.0.0.0 (acces din Docker network)
|
||||
- Protected mode: activat
|
||||
- Max memorie: 512MB
|
||||
- Eviction: allkeys-lru (sterge cele mai vechi chei cand se umple)
|
||||
- Persistenta RDB: snapshot la 60s/10000 keys, 300s/10 keys, 900s/1 key
|
||||
- Persistenta AOF: activat, sync everysec
|
||||
- Max clienti: 10000
|
||||
- 16 baze de date
|
||||
- Slow log: 10ms threshold
|
||||
- Parola: obligatorie (requirepass)
|
||||
|
||||
### Pornire in productie
|
||||
|
||||
Nota: didi-cache NU este definit in data-layer/docker-compose.yml. Containerul Redis este definit in `production/docker-compose.yml` (alaturi de Kong si Keycloak).
|
||||
|
||||
```yaml
|
||||
# din production/docker-compose.yml
|
||||
didi-cache:
|
||||
image: redis:7-alpine
|
||||
command: redis-server --requirepass ${REDIS_PASSWORD} --save 60 1 --save 300 10
|
||||
volumes:
|
||||
- didi-cache-data:/data
|
||||
networks:
|
||||
- didi-network
|
||||
healthcheck:
|
||||
test: redis-cli -a ${REDIS_PASSWORD} ping
|
||||
```
|
||||
|
||||
Nota: redis.conf din directorul didiCache nu este montat in container in productie. Containerul foloseste parametrii din command line.
|
||||
|
||||
### Conexiune
|
||||
|
||||
**Productie (LOCAL — activ)**:
|
||||
- Host: `didi-cache` (Docker DNS pe `didi-network`)
|
||||
- Port: `6379` (expus si pe host: `0.0.0.0:6379->6379`)
|
||||
- DB: `0`
|
||||
- Parola: `redis123` (din `.env`, `REDIS_PASSWORD`)
|
||||
|
||||
**Fallback HA (cluster RAG — disponibil dar inactiv, doar dupa `redis-switch.sh cluster`)**:
|
||||
- Host: `10.11.50.100` (HAProxy VIP rag01/02/03)
|
||||
- Port: `16379`
|
||||
- DB: `0`
|
||||
- Username: `didi`
|
||||
- Parola: din `.cluster-credentials.env` (gitignored)
|
||||
|
||||
### Administrare
|
||||
|
||||
- Conexiune directa: `redis-cli -h didi-cache -a redis123` (sau `127.0.0.1:6379` de pe host).
|
||||
|
||||
---
|
||||
|
||||
## Fisiere in directorul didiCache
|
||||
|
||||
```
|
||||
redis.conf -- Configurare Redis completa (183 linii, nu e montata in productie)
|
||||
.env.example -- Template variabile de mediu
|
||||
.gitignore -- Exclude .env, data/, *.rdb, *.aof
|
||||
README.md -- Documentatie
|
||||
```
|
||||
|
||||
Zero cod custom. Zero module Redis. Zero scripturi. Doar configurare.
|
||||
|
||||
---
|
||||
|
||||
## Ce NU face
|
||||
|
||||
- Nu e Redis Cluster (instanta singulara)
|
||||
- Nu e Redis Sentinel (fara failover automat)
|
||||
- Nu are module custom
|
||||
- Nu are replicare
|
||||
- Nu are ACL (doar autentificare cu parola)
|
||||
- Nu proceseaza nimic -- doar stocheaza si serveste date scrise de alte servicii
|
||||
117
backend/services/data-layer/didiCache/MIGRATION.md
Normal file
117
backend/services/data-layer/didiCache/MIGRATION.md
Normal file
|
|
@ -0,0 +1,117 @@
|
|||
# Redis — migrat pe clusterul RAG (2026-04-22)
|
||||
|
||||
> **TL;DR**: DIDI folosește **clusterul Redis RAG** (3 noduri Sentinel + HAProxy VIP). Containerul local `didi-cache` din `production/docker-compose.yml` e **oprit dar păstrat** ca fallback rapid. Conexiunea e centralizată via `agent-v3/src/shared/redis/connection.ts`.
|
||||
|
||||
---
|
||||
|
||||
## Ce era aici (legacy)
|
||||
|
||||
`didi-cache` — un container Redis 7 Alpine standalone pe `didi-network` Docker. Single-node, fără HA, fără replicare. 512MB max memory, allkeys-lru eviction. Servea toate sesiunile pipeline + cache-ul framework + lock-uri workeri.
|
||||
|
||||
## Ce e acum
|
||||
|
||||
### Cluster Redis RAG (productie)
|
||||
|
||||
| Nod | Hostname | IP | Port Redis | Port Sentinel |
|
||||
|---|---|---|---|---|
|
||||
| rag01 (HAProxy VIP) | `rag01` | `10.11.50.101` | 6379 (replica) | 26379 |
|
||||
| **rag02 (master curent)** | `rag02` | `10.11.50.102` | **6379 (master)** | 26379 |
|
||||
| rag03 | `rag03` | `10.11.50.103` | 6379 (replica) | 26379 |
|
||||
|
||||
**Endpoint-uri pentru aplicații:**
|
||||
|
||||
| Scop | Endpoint |
|
||||
|---|---|
|
||||
| **WRITE** (auto-routing master) | `10.11.50.100:16379` |
|
||||
| READ (replica locală rag01) | `10.11.50.100:6379` |
|
||||
| Sentinel discovery | `10.11.50.100:16380` (master name `ragmaster`) |
|
||||
| HAProxy stats | `http://10.11.50.100:8404/stats` |
|
||||
|
||||
User ACL DIDI: `didi` cu prefix de chei `didi:*` și channels `didi.*`. Parola în vault-ul de credențiale `name='Redis ACL — didi'`.
|
||||
|
||||
ACL persistent via `--aclfile /data/users.acl` pe toate 3 nodurile (fix permanent 2026-04-22).
|
||||
|
||||
### Conexiune centralizată în cod
|
||||
|
||||
`agent-v3/src/shared/redis/connection.ts` — toate conexiunile ioredis trec prin `createRedisConnection(label)`. Zero `new Redis({...})` inline. Auto-reconnect cu retryStrategy, reconnectOnError pentru `READONLY` / `MASTERDOWN` (failover).
|
||||
|
||||
```typescript
|
||||
// Folosire în cod:
|
||||
import { createRedisConnection } from './shared/redis/connection';
|
||||
const redis = createRedisConnection('framework-cache');
|
||||
```
|
||||
|
||||
`didiFramework` are propria implementare în `src/lib/redis.ts`.
|
||||
|
||||
## Containerul local `didi-cache`
|
||||
|
||||
Definit în `production/docker-compose.yml`, configurat să pornească dar **manual oprit** ca parte din migrare. Volumul `didi-production-cache-data` e intact — datele anterioare (snapshot-uri RDB + AOF) sunt păstrate.
|
||||
|
||||
Status curent: `Exited`.
|
||||
|
||||
### De ce e păstrat?
|
||||
|
||||
- **Fallback rapid** dacă cluster RAG e indisponibil (oprit pentru maintenance, network issue, etc.)
|
||||
- Pentru a-l reactiva temporar, există un script switch:
|
||||
```bash
|
||||
backend/services/orchestration-layer/scripts/redis-switch.sh local redis
|
||||
```
|
||||
Asta rescrie `.env` și restart agent-v3 stack pentru a folosi `didi-cache` în loc de cluster.
|
||||
|
||||
## Switch rapid cluster ↔ local
|
||||
|
||||
Script: `backend/services/orchestration-layer/scripts/redis-switch.sh`
|
||||
|
||||
```bash
|
||||
# Folosește local fallback (didi-cache)
|
||||
./redis-switch.sh local redis
|
||||
|
||||
# Folosește clusterul (default)
|
||||
./redis-switch.sh cluster redis
|
||||
|
||||
# Status
|
||||
./redis-switch.sh status redis
|
||||
```
|
||||
|
||||
Credentialele cluster în sidecar `.cluster-credentials.env` (gitignored).
|
||||
|
||||
## Verificare cluster
|
||||
|
||||
```bash
|
||||
# Cu redis-cli din host (default user — pentru debug)
|
||||
redis-cli -h 10.11.50.100 -p 16379 -a <pwd> PING
|
||||
|
||||
# Cu user DIDI
|
||||
redis-cli -h 10.11.50.100 -p 16379 --user didi -a <pwd> ACL WHOAMI
|
||||
|
||||
# Sentinel master discovery
|
||||
redis-cli -h 10.11.50.100 -p 16380 SENTINEL get-master-addr-by-name ragmaster
|
||||
```
|
||||
|
||||
## Bootstrap automat la fresh deploy
|
||||
|
||||
`didiFramework` detectează la pornire dacă `didi:framework:manifest` lipsește în Redis și auto-rulează `POST /api/sync-redis` (56 chei populate în ~700ms). Zero intervenție manuală pe cluster nou.
|
||||
|
||||
## Chei Redis principale (orientativ)
|
||||
|
||||
| Pattern | Scop | TTL |
|
||||
|---|---|---|
|
||||
| `didi:framework:*` | Config framework (tehnici, ponderi, verdicts) | permanent |
|
||||
| `didi:config:{component}:v1:*` | Config componente, stage assignments tier-nested | permanent |
|
||||
| `didi:pipeline:{sessionId}:*` | Sesiuni pipeline (status, rezultate) | 7 zile |
|
||||
| `didi:queue:*` | Lock-uri workeri | 30s-5min |
|
||||
| `agent:result:*` | Rezultate intermediare | 7 zile |
|
||||
| `agent:media:*` | Cache media (transcript, vision) | 1 oră |
|
||||
|
||||
## Linkuri rapide
|
||||
|
||||
- Ghid utilizare cluster: `landingzone/redis-rag/README.md` (repo `git.finesynergy.eu/lucian/landingzone`)
|
||||
- Onboarding ACL user nou: `landingzone/redis-rag/CLAUDE_PROMPT.md`
|
||||
- HAProxy stats: `http://10.11.50.100:8404/stats`
|
||||
|
||||
## Status
|
||||
|
||||
- ✅ Migrare aplicată: 2026-04-22
|
||||
- ✅ User `didi` configurat cu prefix izolat
|
||||
- ✅ Container local păstrat ca fallback (oprit, volum intact)
|
||||
- ✅ Bootstrap framework automat la pornire
|
||||
147
backend/services/data-layer/didiCache/README.md
Normal file
147
backend/services/data-layer/didiCache/README.md
Normal file
|
|
@ -0,0 +1,147 @@
|
|||
# DIDI Cache Service 🚀
|
||||
|
||||
## Super Simple Start Guide ⚡
|
||||
|
||||
### Step 1: Set Your Password
|
||||
```bash
|
||||
# Copy the example file
|
||||
cp .env.example .env
|
||||
|
||||
# Edit .env and change this line:
|
||||
REDIS_PASSWORD=YOUR_SECURE_PASSWORD_HERE
|
||||
```
|
||||
|
||||
### Step 2: Start Redis
|
||||
```bash
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
That's it! Your cache is running! 🎉
|
||||
|
||||
## Check If It's Working ✅
|
||||
|
||||
```bash
|
||||
# See if container is healthy
|
||||
docker ps
|
||||
|
||||
# Look for: didi-cache (healthy)
|
||||
```
|
||||
|
||||
## Connection Info 📡
|
||||
|
||||
- **Host**: localhost
|
||||
- **Port**: 6380 (not 6379 to avoid conflicts!)
|
||||
- **Password**: (what you set in .env)
|
||||
|
||||
## Test the Connection 🔌
|
||||
|
||||
```bash
|
||||
# Connect with Redis CLI
|
||||
docker exec -it didi-cache redis-cli -a YOUR_PASSWORD
|
||||
|
||||
# Test it
|
||||
127.0.0.1:6379> PING
|
||||
PONG
|
||||
|
||||
# Exit
|
||||
127.0.0.1:6379> EXIT
|
||||
```
|
||||
|
||||
## What's Inside? 📦
|
||||
|
||||
This Redis cache is configured for:
|
||||
- ✅ 512MB memory (perfect for status & results)
|
||||
- ✅ Auto-expiry support (services set TTL)
|
||||
- ✅ Persistence enabled (survives restarts)
|
||||
- ✅ Password protected
|
||||
|
||||
### How DIDI Uses Redis 🔄
|
||||
|
||||
```
|
||||
Pipeline runs → Status stored here (24hr TTL)
|
||||
→ Results cached here
|
||||
→ UI polls for updates
|
||||
→ After 24hrs, auto-deleted
|
||||
```
|
||||
|
||||
### Key Patterns We Store 📝
|
||||
- `run:{run_id}` - Pipeline execution status
|
||||
- `node_status:{run_id}` - Each node's progress
|
||||
- `results:{run_id}` - Analysis results
|
||||
|
||||
## Common Commands 🛠️
|
||||
|
||||
```bash
|
||||
# Stop cache
|
||||
docker compose down
|
||||
|
||||
# View logs
|
||||
docker compose logs -f
|
||||
|
||||
# Connect to Redis CLI
|
||||
docker exec -it didi-cache redis-cli -a YOUR_PASSWORD
|
||||
|
||||
# Restart fresh (WARNING: Deletes all cache!)
|
||||
docker compose down -v
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
## Monitor Cache Usage 📊
|
||||
|
||||
```bash
|
||||
# Check memory usage
|
||||
docker exec -it didi-cache redis-cli -a YOUR_PASSWORD INFO memory
|
||||
|
||||
# See all keys
|
||||
docker exec -it didi-cache redis-cli -a YOUR_PASSWORD KEYS "*"
|
||||
|
||||
# Count keys
|
||||
docker exec -it didi-cache redis-cli -a YOUR_PASSWORD DBSIZE
|
||||
```
|
||||
|
||||
## Troubleshooting 🔧
|
||||
|
||||
### Port 6380 already in use?
|
||||
Edit `.env` and change `REDIS_PORT` to something else (like 6381)
|
||||
|
||||
### Can't connect?
|
||||
1. Check container is healthy: `docker ps`
|
||||
2. Verify password in .env
|
||||
3. Make sure you're using port 6380, not 6379
|
||||
|
||||
### Memory full?
|
||||
Redis will auto-delete least recently used keys (LRU policy)
|
||||
|
||||
## Part of Something Bigger 🏗️
|
||||
|
||||
This cache is part of the DIDI Backend data layer:
|
||||
|
||||
```
|
||||
📁 data-layer/
|
||||
├── 📁 didiDatabase/ (PostgreSQL - Done!)
|
||||
├── 📁 didiCache/ (Redis - You are here!)
|
||||
├── 📁 didiQueue/ (RabbitMQ - Next)
|
||||
└── 📁 didiStorage/ (MinIO - Coming soon)
|
||||
```
|
||||
|
||||
## Quick Health Check 🏥
|
||||
|
||||
```bash
|
||||
# Is it running?
|
||||
docker exec -it didi-cache redis-cli -a YOUR_PASSWORD PING
|
||||
|
||||
# Response should be:
|
||||
# PONG
|
||||
```
|
||||
|
||||
## Why Port 6380? 🤔
|
||||
|
||||
The old monolithic setup uses port 6379. We use 6380 to:
|
||||
- Avoid conflicts during migration
|
||||
- Run both services side-by-side
|
||||
- Easy rollback if needed
|
||||
|
||||
---
|
||||
**That's all you need to know! Happy caching! ⚡**
|
||||
|
||||
*Version: 1.0.0 | Redis 7-alpine*
|
||||
183
backend/services/data-layer/didiCache/redis.conf
Normal file
183
backend/services/data-layer/didiCache/redis.conf
Normal file
|
|
@ -0,0 +1,183 @@
|
|||
# ============================================================================
|
||||
# DIDI Cache Service Configuration (Redis 7)
|
||||
# Production-ready configuration for the DIDI Backend platform
|
||||
# ============================================================================
|
||||
|
||||
# ============================================================================
|
||||
# NETWORK & SECURITY
|
||||
# ============================================================================
|
||||
|
||||
# Listen on all interfaces (Docker container)
|
||||
bind 0.0.0.0
|
||||
|
||||
# Enable protected mode
|
||||
protected-mode yes
|
||||
|
||||
# Port
|
||||
port 6379
|
||||
|
||||
# TCP listen() backlog
|
||||
tcp-backlog 511
|
||||
|
||||
# TCP keepalive
|
||||
tcp-keepalive 300
|
||||
|
||||
# Timeout for idle clients (0 to disable)
|
||||
timeout 0
|
||||
|
||||
# ============================================================================
|
||||
# GENERAL
|
||||
# ============================================================================
|
||||
|
||||
# Don't run as daemon (Docker handles this)
|
||||
daemonize no
|
||||
|
||||
# Server verbosity (debug, verbose, notice, warning)
|
||||
loglevel notice
|
||||
|
||||
# Log to stdout for Docker
|
||||
logfile ""
|
||||
|
||||
# Number of databases (we use 0 for main cache)
|
||||
databases 16
|
||||
|
||||
# ============================================================================
|
||||
# MEMORY MANAGEMENT
|
||||
# ============================================================================
|
||||
|
||||
# Maximum memory (adjust based on container limits)
|
||||
maxmemory 512mb
|
||||
|
||||
# Eviction policy when max memory is reached
|
||||
# allkeys-lru: Remove least recently used keys
|
||||
maxmemory-policy allkeys-lru
|
||||
|
||||
# LRU samples for eviction
|
||||
maxmemory-samples 5
|
||||
|
||||
# ============================================================================
|
||||
# PERSISTENCE - RDB (Snapshots)
|
||||
# ============================================================================
|
||||
|
||||
# Save snapshots:
|
||||
# After 900 sec (15 min) if at least 1 key changed
|
||||
save 900 1
|
||||
# After 300 sec (5 min) if at least 10 keys changed
|
||||
save 300 10
|
||||
# After 60 sec if at least 10000 keys changed
|
||||
save 60 10000
|
||||
|
||||
# Error handling for background save
|
||||
stop-writes-on-bgsave-error yes
|
||||
|
||||
# Compress RDB dumps
|
||||
rdbcompression yes
|
||||
|
||||
# Checksum RDB files
|
||||
rdbchecksum yes
|
||||
|
||||
# Filename for RDB
|
||||
dbfilename dump.rdb
|
||||
|
||||
# Directory for RDB and AOF files
|
||||
dir /data
|
||||
|
||||
# ============================================================================
|
||||
# PERSISTENCE - AOF (Append Only File)
|
||||
# ============================================================================
|
||||
|
||||
# Enable AOF
|
||||
appendonly yes
|
||||
|
||||
# AOF filename
|
||||
appendfilename "appendonly.aof"
|
||||
|
||||
# AOF sync policy (everysec = good balance)
|
||||
appendfsync everysec
|
||||
|
||||
# Don't fsync during rewrites
|
||||
no-appendfsync-on-rewrite no
|
||||
|
||||
# Auto rewrite AOF
|
||||
auto-aof-rewrite-percentage 100
|
||||
auto-aof-rewrite-min-size 64mb
|
||||
|
||||
# Load truncated AOF
|
||||
aof-load-truncated yes
|
||||
|
||||
# Use RDB format in AOF for faster loading
|
||||
aof-use-rdb-preamble yes
|
||||
|
||||
# ============================================================================
|
||||
# SLOW LOG
|
||||
# ============================================================================
|
||||
|
||||
# Log queries slower than (microseconds)
|
||||
slowlog-log-slower-than 10000
|
||||
|
||||
# Maximum length of slow log
|
||||
slowlog-max-len 128
|
||||
|
||||
# ============================================================================
|
||||
# LATENCY MONITORING
|
||||
# ============================================================================
|
||||
|
||||
# Latency threshold in milliseconds
|
||||
latency-monitor-threshold 100
|
||||
|
||||
# ============================================================================
|
||||
# CLIENT HANDLING
|
||||
# ============================================================================
|
||||
|
||||
# Maximum number of clients
|
||||
maxclients 10000
|
||||
|
||||
# ============================================================================
|
||||
# ADVANCED CONFIG
|
||||
# ============================================================================
|
||||
|
||||
# Hash tables
|
||||
hash-max-ziplist-entries 512
|
||||
hash-max-ziplist-value 64
|
||||
|
||||
# Lists
|
||||
list-max-ziplist-size -2
|
||||
list-compress-depth 0
|
||||
|
||||
# Sets
|
||||
set-max-intset-entries 512
|
||||
|
||||
# Sorted sets
|
||||
zset-max-ziplist-entries 128
|
||||
zset-max-ziplist-value 64
|
||||
|
||||
# HyperLogLog
|
||||
hll-sparse-max-bytes 3000
|
||||
|
||||
# Streams
|
||||
stream-node-max-bytes 4096
|
||||
stream-node-max-entries 100
|
||||
|
||||
# Active rehashing
|
||||
activerehashing yes
|
||||
|
||||
# Client output buffer limits
|
||||
client-output-buffer-limit normal 0 0 0
|
||||
client-output-buffer-limit replica 256mb 64mb 60
|
||||
client-output-buffer-limit pubsub 32mb 8mb 60
|
||||
|
||||
# Frequency of rehashing the main dictionary
|
||||
hz 10
|
||||
|
||||
# LFU settings
|
||||
lfu-log-factor 10
|
||||
lfu-decay-time 1
|
||||
|
||||
# ============================================================================
|
||||
# DISABLE DANGEROUS COMMANDS (Production)
|
||||
# ============================================================================
|
||||
|
||||
# Uncomment these in production to disable dangerous commands
|
||||
# rename-command FLUSHDB ""
|
||||
# rename-command FLUSHALL ""
|
||||
# rename-command CONFIG ""
|
||||
68
backend/services/data-layer/didiDatabase/.gitignore
vendored
Normal file
68
backend/services/data-layer/didiDatabase/.gitignore
vendored
Normal file
|
|
@ -0,0 +1,68 @@
|
|||
# Environment files
|
||||
.env
|
||||
.env.local
|
||||
.env.production
|
||||
|
||||
# Data directories
|
||||
data/
|
||||
backups/
|
||||
pg_data/
|
||||
pgdata/
|
||||
|
||||
# Log files
|
||||
*.log
|
||||
logs/
|
||||
log/
|
||||
|
||||
# Certificates and keys
|
||||
certs/
|
||||
*.crt
|
||||
*.key
|
||||
*.pem
|
||||
*.p12
|
||||
*.pfx
|
||||
|
||||
# Backup files
|
||||
*.dump
|
||||
*.sql
|
||||
*.sql.gz
|
||||
*.tar
|
||||
*.tar.gz
|
||||
*.backup
|
||||
|
||||
# pgAdmin data
|
||||
pgadmin_data/
|
||||
.pgadmin/
|
||||
|
||||
# Temporary files
|
||||
*.tmp
|
||||
*.temp
|
||||
*.swp
|
||||
*.swo
|
||||
*~
|
||||
|
||||
# OS files
|
||||
.DS_Store
|
||||
Thumbs.db
|
||||
desktop.ini
|
||||
|
||||
# IDE files
|
||||
.idea/
|
||||
.vscode/
|
||||
*.iml
|
||||
|
||||
# Docker volumes (local bindings)
|
||||
/data
|
||||
/backups
|
||||
/pgadmin_data
|
||||
|
||||
# Test data
|
||||
test_data/
|
||||
*.test.sql
|
||||
|
||||
# Migration tracking
|
||||
.migrations_applied
|
||||
|
||||
# Monitoring data
|
||||
prometheus_data/
|
||||
grafana_data/
|
||||
72
backend/services/data-layer/didiDatabase/Dockerfile
Normal file
72
backend/services/data-layer/didiDatabase/Dockerfile
Normal file
|
|
@ -0,0 +1,72 @@
|
|||
# PostgreSQL 15 Alpine - Rolling tag for security updates
|
||||
FROM postgres:15-alpine
|
||||
|
||||
# Set environment variables
|
||||
ENV POSTGRES_DB=misinformation_db
|
||||
ENV POSTGRES_USER=postgres
|
||||
ENV POSTGRES_PASSWORD=postgres_dev_password_123
|
||||
ENV PGDATA=/var/lib/postgresql/data/pgdata
|
||||
|
||||
# Install additional packages for production use
|
||||
RUN apk add --no-cache \
|
||||
bash \
|
||||
curl \
|
||||
postgresql-client \
|
||||
&& rm -rf /var/cache/apk/*
|
||||
|
||||
# Create necessary directories
|
||||
RUN mkdir -p /docker-entrypoint-initdb.d \
|
||||
&& mkdir -p /var/lib/postgresql/data \
|
||||
&& mkdir -p /scripts \
|
||||
&& mkdir -p /backups
|
||||
|
||||
# Copy initialization script
|
||||
COPY init.sql /docker-entrypoint-initdb.d/01-init.sql
|
||||
|
||||
# Copy health check script
|
||||
COPY health-check.sh /scripts/health-check.sh
|
||||
RUN chmod +x /scripts/health-check.sh
|
||||
|
||||
# Set proper permissions
|
||||
RUN chown -R postgres:postgres /var/lib/postgresql/data \
|
||||
&& chown -R postgres:postgres /docker-entrypoint-initdb.d \
|
||||
&& chown -R postgres:postgres /scripts \
|
||||
&& chown -R postgres:postgres /backups
|
||||
|
||||
# PostgreSQL configuration for production
|
||||
RUN echo "shared_preload_libraries = 'pg_stat_statements'" >> /usr/local/share/postgresql/postgresql.conf.sample \
|
||||
&& echo "pg_stat_statements.track = all" >> /usr/local/share/postgresql/postgresql.conf.sample \
|
||||
&& echo "log_statement = 'all'" >> /usr/local/share/postgresql/postgresql.conf.sample \
|
||||
&& echo "log_duration = on" >> /usr/local/share/postgresql/postgresql.conf.sample \
|
||||
&& echo "log_min_duration_statement = 100" >> /usr/local/share/postgresql/postgresql.conf.sample \
|
||||
&& echo "shared_buffers = 256MB" >> /usr/local/share/postgresql/postgresql.conf.sample \
|
||||
&& echo "effective_cache_size = 1GB" >> /usr/local/share/postgresql/postgresql.conf.sample \
|
||||
&& echo "maintenance_work_mem = 64MB" >> /usr/local/share/postgresql/postgresql.conf.sample \
|
||||
&& echo "checkpoint_completion_target = 0.9" >> /usr/local/share/postgresql/postgresql.conf.sample \
|
||||
&& echo "wal_buffers = 16MB" >> /usr/local/share/postgresql/postgresql.conf.sample \
|
||||
&& echo "default_statistics_target = 100" >> /usr/local/share/postgresql/postgresql.conf.sample \
|
||||
&& echo "random_page_cost = 1.1" >> /usr/local/share/postgresql/postgresql.conf.sample \
|
||||
&& echo "effective_io_concurrency = 200" >> /usr/local/share/postgresql/postgresql.conf.sample \
|
||||
&& echo "work_mem = 4MB" >> /usr/local/share/postgresql/postgresql.conf.sample \
|
||||
&& echo "min_wal_size = 1GB" >> /usr/local/share/postgresql/postgresql.conf.sample \
|
||||
&& echo "max_wal_size = 4GB" >> /usr/local/share/postgresql/postgresql.conf.sample \
|
||||
&& echo "max_worker_processes = 8" >> /usr/local/share/postgresql/postgresql.conf.sample \
|
||||
&& echo "max_parallel_workers_per_gather = 4" >> /usr/local/share/postgresql/postgresql.conf.sample \
|
||||
&& echo "max_parallel_workers = 8" >> /usr/local/share/postgresql/postgresql.conf.sample \
|
||||
&& echo "max_parallel_maintenance_workers = 4" >> /usr/local/share/postgresql/postgresql.conf.sample
|
||||
|
||||
# Expose PostgreSQL port
|
||||
EXPOSE 5432
|
||||
|
||||
# Health check
|
||||
HEALTHCHECK --interval=30s --timeout=10s --start-period=30s --retries=3 \
|
||||
CMD /scripts/health-check.sh || exit 1
|
||||
|
||||
# Use the postgres user
|
||||
USER postgres
|
||||
|
||||
# Volume for data persistence
|
||||
VOLUME ["/var/lib/postgresql/data", "/backups"]
|
||||
|
||||
# Start PostgreSQL
|
||||
CMD ["postgres"]
|
||||
536
backend/services/data-layer/didiDatabase/INDEX.md
Normal file
536
backend/services/data-layer/didiDatabase/INDEX.md
Normal file
|
|
@ -0,0 +1,536 @@
|
|||
# didiDatabase - Index
|
||||
|
||||
Documentatie completa pentru baza de date PostgreSQL a platformei DIDI. Baza principala DIDI ruleaza pe un **container LOCAL** (`didi-postgres`, PostgreSQL 17) pe masina de deployment, in reteaua Docker `didi-network`. Clusterul extern Patroni/HAProxy ramane configurat ca fallback HA, dar NU este folosit operational acum.
|
||||
|
||||
---
|
||||
|
||||
## PostgreSQL LOCAL (PRODUCTIE — activ)
|
||||
|
||||
| Parametru | Valoare |
|
||||
|-----------|---------|
|
||||
| Container | `didi-postgres` |
|
||||
| Imagine | `postgres:17-alpine` |
|
||||
| Host intern | `didi-postgres:5432` (Docker DNS pe `didi-network`) |
|
||||
| Port host | `5432` expus pe `0.0.0.0:5432->5432` |
|
||||
| Database | `DIDI` |
|
||||
| User principal | `bos_interface` / `interface` |
|
||||
|
||||
### Baze de date pe instanta
|
||||
|
||||
Instanta `didi-postgres` contine o singura baza de business, `DIDI` (4 scheme + public, ~2012 sesiuni de analiza la data documentatiei).
|
||||
|
||||
| Baza / consumator | User | Folosita de | Note |
|
||||
|-------------------|------|-------------|------|
|
||||
| DIDI | bos_interface | agent-v3, didiFramework | schemele `bos_*` |
|
||||
| DIDI (schema `public`) | bos_interface | Keycloak IAM | `KC_DB_URL=jdbc:postgresql://didi-postgres:5432/DIDI?currentSchema=public` |
|
||||
| — | — | Kong API Gateway | Kong ruleaza **DBless** (config declarativ), fara baza proprie |
|
||||
|
||||
### Cine se conecteaza
|
||||
|
||||
| Serviciu | Host | Port | Database | User | Fisier config |
|
||||
|----------|------|------|----------|------|---------------|
|
||||
| agent-v3 | didi-postgres | 5432 | DIDI | bos_interface | agent-v3/src/shared/persistence/pg-pool.ts |
|
||||
| didiFramework | didi-postgres | 5432 | DIDI | bos_interface | didiFramework/src/config/database.ts |
|
||||
| Keycloak | didi-postgres | 5432 | DIDI (schema public) | bos_interface | production/.env (`KC_DB_URL`) |
|
||||
|
||||
Containerul local `didi-postgres` este unicul PostgreSQL de productie activ. Fostul container `staging-dataLayer-postgres` NU mai exista. Toate schemele `bos_*` + `public` sunt pe `didi-postgres`.
|
||||
|
||||
agent-v3 acceseaza baza prin `shared/persistence/pg-pool.ts` (`didi-postgres:5432`, DB `DIDI`, user `bos_interface`). Dupa migration 011 scrie si in `bos_analysis.moderation_queue` (prin `moderation/queue-manager.ts`) si citeste coloanele HIL noi de pe `analysis_session`.
|
||||
|
||||
didiFramework scrie in `bos_parammgmt.moderation_config`, `sensitive_topic`, `moderation_role` (introduse de migration 011).
|
||||
|
||||
---
|
||||
|
||||
## Baza de date DIDI -- Schema completa
|
||||
|
||||
4 scheme + public, ~50 tabele total.
|
||||
|
||||
---
|
||||
|
||||
### Schema: bos_analysis (7 tabele + 1 view)
|
||||
|
||||
Scrisa de agent-v3 (pg-adapter.ts, moderation/queue-manager.ts). Citita si de didiFramework (history.ts, sync-analysis.ts).
|
||||
|
||||
#### analysis_session
|
||||
|
||||
Tabelul central -- o inregistrare per analiza.
|
||||
|
||||
| Coloana | Tip | Scop |
|
||||
|---------|-----|------|
|
||||
| session_id | TEXT PK | UUID sesiune |
|
||||
| user_id | TEXT | ID utilizator |
|
||||
| user_email | TEXT | Email utilizator |
|
||||
| input_type | TEXT | text, url, image, audio, video |
|
||||
| input_text | TEXT | Text de analizat |
|
||||
| input_url | TEXT | URL analizat |
|
||||
| input_media_url | TEXT | URL media MinIO |
|
||||
| input_hash | TEXT | Hash input (deduplicare) |
|
||||
| status | TEXT | running, completed, failed |
|
||||
| components_run | TEXT[] | Componente rulate |
|
||||
| components_skipped | TEXT[] | Componente sarite |
|
||||
| risk_score | NUMERIC | Scor risc final (0-100) |
|
||||
| risk_category | TEXT | Categorie risc |
|
||||
| risk_level | TEXT | Nivel risc |
|
||||
| confidence | NUMERIC | Incredere (0-100) |
|
||||
| confidence_level | TEXT | Nivel incredere |
|
||||
| started_at | TIMESTAMP | Start procesare |
|
||||
| completed_at | TIMESTAMP | Sfarsit procesare |
|
||||
| total_duration_ms | INTEGER | Durata totala ms |
|
||||
| scenario_applied | TEXT | Scenariu ponderi aplicat |
|
||||
| topic_applied | TEXT | Topic detectat |
|
||||
| source_app | TEXT | web (default) |
|
||||
| api_version | TEXT | v3 (default) |
|
||||
| created_at | TIMESTAMP | Data creare |
|
||||
|
||||
Coloane HIL adaugate prin migration 011 (2026-05-01):
|
||||
|
||||
| Coloana | Tip | Scop |
|
||||
|---------|-----|------|
|
||||
| review_status | TEXT default 'none' (CHECK: none\|pending\|in_review\|resolved\|declined) | HIL state |
|
||||
| human_corrected | BOOLEAN default false | true daca moderator a corectat |
|
||||
| human_corrections | JSONB NULL | Diff-style corrections {verdict?, techniques?, ai_tampered?, claims?} |
|
||||
| verified_by | TEXT NULL | keycloak_id moderator |
|
||||
| verified_at | TIMESTAMPTZ NULL | When resolved |
|
||||
| review_notes | TEXT NULL | Optional moderator notes |
|
||||
|
||||
Index partial: `idx_analysis_session_review_status WHERE review_status != 'none'` -- majoritatea sesiunilor raman 'none', sunt sarite la scan.
|
||||
|
||||
#### analysis_techniques
|
||||
|
||||
O inregistrare per sesiune -- rezultat componenta tehnici de manipulare.
|
||||
|
||||
| Coloana | Tip | Scop |
|
||||
|---------|-----|------|
|
||||
| session_id | TEXT FK | Referinta sesiune |
|
||||
| manipulation_score | NUMERIC | Scor manipulare (0-100) |
|
||||
| total_severity | NUMERIC | Severitate totala |
|
||||
| dimensions_affected | TEXT[] | Dimensiuni afectate |
|
||||
| techniques_count | INTEGER | Numar tehnici detectate |
|
||||
| techniques_detected | JSONB | Lista tehnici cu detalii |
|
||||
| coupling_context | JSONB | Context cuplare inter-tehnici |
|
||||
| llm_screening | TEXT | Model LLM screening |
|
||||
| llm_deep | TEXT | Model LLM deep analysis |
|
||||
| screening_duration_ms | INTEGER | Durata screening |
|
||||
| deep_analysis_duration_ms | INTEGER | Durata analiza profunda |
|
||||
| total_duration_ms | INTEGER | Durata totala |
|
||||
| fallbacks_screening | INTEGER | Fallback-uri screening |
|
||||
| fallbacks_deep | INTEGER | Fallback-uri deep |
|
||||
|
||||
#### analysis_ai_tampered
|
||||
|
||||
O inregistrare per sesiune -- detectie continut AI/manipulat.
|
||||
|
||||
| Coloana | Tip | Scop |
|
||||
|---------|-----|------|
|
||||
| session_id | TEXT FK | Referinta sesiune |
|
||||
| ai_probability | NUMERIC | Probabilitate AI (0-100) |
|
||||
| verdict | TEXT | Verdict AI detection |
|
||||
| risk_score | NUMERIC | Scor risc AI |
|
||||
| categories_affected | TEXT[] | Categorii afectate |
|
||||
| indicators_count | INTEGER | Numar indicatori |
|
||||
| disclosure_detected | BOOLEAN | Disclosure detectat |
|
||||
| disclosure_explicit | BOOLEAN | Disclosure explicit |
|
||||
| disclosure_text | TEXT | Text disclosure |
|
||||
| indicators_detected | JSONB | Lista indicatori |
|
||||
| coupling_context | JSONB | Context cuplare |
|
||||
| llm_screening | TEXT | Model screening |
|
||||
| llm_deep | TEXT | Model deep |
|
||||
| screening_duration_ms | INTEGER | Durata screening |
|
||||
| deep_analysis_duration_ms | INTEGER | Durata deep |
|
||||
| total_duration_ms | INTEGER | Durata totala |
|
||||
| fallbacks_screening | INTEGER | Fallback-uri screening |
|
||||
| fallbacks_deep | INTEGER | Fallback-uri deep |
|
||||
| content_type | TEXT | text, image, audio, video |
|
||||
| image_analysis | JSONB | Rezultat analiza imagine |
|
||||
|
||||
#### analysis_claims
|
||||
|
||||
O inregistrare per sesiune -- verificare afirmatii.
|
||||
|
||||
| Coloana | Tip | Scop |
|
||||
|---------|-----|------|
|
||||
| session_id | TEXT FK | Referinta sesiune |
|
||||
| total_claims | INTEGER | Total afirmatii |
|
||||
| verified_true | INTEGER | Verificate adevarate |
|
||||
| verified_false | INTEGER | Verificate false |
|
||||
| unverified | INTEGER | Neverificate |
|
||||
| opinions | INTEGER | Opinii |
|
||||
| credibility_score | NUMERIC | Scor credibilitate |
|
||||
| interpretation | TEXT | Interpretare |
|
||||
| claims_by_status | JSONB | Claims grupate pe status |
|
||||
| claims_by_type | JSONB | Claims grupate pe tip |
|
||||
| claims_verified | JSONB | Detalii verificare |
|
||||
| llm_extraction | TEXT | Model extragere |
|
||||
| llm_verification | TEXT | Model verificare |
|
||||
| extraction_duration_ms | INTEGER | Durata extragere |
|
||||
| verification_duration_ms | INTEGER | Durata verificare |
|
||||
| total_duration_ms | INTEGER | Durata totala |
|
||||
| web_searches_made | INTEGER | Cautari web efectuate |
|
||||
|
||||
#### analysis_domain
|
||||
|
||||
O inregistrare per sesiune -- analiza domeniu/sursa.
|
||||
|
||||
| Coloana | Tip | Scop |
|
||||
|---------|-----|------|
|
||||
| session_id | TEXT FK | Referinta sesiune |
|
||||
| domain | TEXT | Domeniu analizat |
|
||||
| verdict | TEXT | Verdict domeniu |
|
||||
| trust_score | NUMERIC | Scor incredere |
|
||||
| risk_level | TEXT | Nivel risc |
|
||||
| age_days | INTEGER | Varsta domeniu (zile) |
|
||||
| age_category | TEXT | Categorie varsta |
|
||||
| domain_created_at | TIMESTAMP | Data creare domeniu |
|
||||
| is_blacklisted | BOOLEAN | Pe lista neagra |
|
||||
| reputation_score | NUMERIC | Scor reputatie |
|
||||
| has_ssl | BOOLEAN | Are SSL |
|
||||
| ssl_valid | BOOLEAN | SSL valid |
|
||||
| ssl_issuer | TEXT | Emitent SSL |
|
||||
| registrar | TEXT | Registrar domeniu |
|
||||
| organization | TEXT | Organizatie |
|
||||
| country | TEXT | Tara |
|
||||
| red_flags | TEXT[] | Semnale alarma |
|
||||
| warnings | TEXT[] | Avertismente |
|
||||
| duration_ms | INTEGER | Durata analiza |
|
||||
|
||||
#### analysis_verdict
|
||||
|
||||
O inregistrare per sesiune -- verdictul final agregat.
|
||||
|
||||
| Coloana | Tip | Scop |
|
||||
|---------|-----|------|
|
||||
| session_id | TEXT FK | Referinta sesiune |
|
||||
| risk_score | NUMERIC | Scor risc final |
|
||||
| risk_category | TEXT | Categorie risc |
|
||||
| risk_category_color | TEXT | Culoare categorie |
|
||||
| risk_level | TEXT | Nivel risc |
|
||||
| risk_level_color | TEXT | Culoare nivel |
|
||||
| severity | TEXT | Severitate |
|
||||
| recommended_action | TEXT | Actiune recomandata |
|
||||
| confidence | NUMERIC | Incredere |
|
||||
| confidence_level | TEXT | Nivel incredere |
|
||||
| score_manipulation | NUMERIC | Scor componenta manipulare |
|
||||
| score_claims | NUMERIC | Scor componenta claims |
|
||||
| score_ai | NUMERIC | Scor componenta AI |
|
||||
| score_source | NUMERIC | Scor componenta sursa |
|
||||
| score_context | NUMERIC | Scor context |
|
||||
| applied_weights | JSONB | Ponderi aplicate |
|
||||
| override_applied | BOOLEAN | Override aplicat |
|
||||
| override_type | TEXT | Tip override |
|
||||
| override_reason | TEXT | Motiv override |
|
||||
| override_adjustment | NUMERIC | Ajustare override |
|
||||
| context_summary | JSONB | Sumar context |
|
||||
| components_used | TEXT[] | Componente folosite |
|
||||
| weights_source | TEXT | Sursa ponderi |
|
||||
| duration_ms | INTEGER | Durata calcul |
|
||||
| explanation_ro | TEXT | Explicatie romana (migration 001) |
|
||||
| explanation_en | TEXT | Explicatie engleza (migration 001) |
|
||||
| virality_score | NUMERIC | Scor viralitate (0-100) |
|
||||
| virality_level | TEXT | Nivel viralitate |
|
||||
| virality_factors | JSONB | Factori viralitate |
|
||||
|
||||
#### moderation_queue (adaugat prin migration 011)
|
||||
|
||||
Stare workflow HIL (Human-in-the-Loop). Un rand per sesiune marcata de triage pentru review uman.
|
||||
|
||||
| Coloana | Tip | Scop |
|
||||
|---------|-----|------|
|
||||
| queue_id | BIGSERIAL PK | Auto-increment |
|
||||
| session_id | UUID FK -> analysis_session(session_id) ON DELETE CASCADE | Referinta sesiune |
|
||||
| priority | INTEGER (1-5) | 1=highest (user_flagged), 3=low_confidence, 4=sensitive_topic |
|
||||
| enqueue_reason | TEXT | flagged \| low_confidence \| sensitive_topic \| mixed |
|
||||
| enqueue_meta | JSONB | Triage metadata (risk_score, confidence, topic detected) |
|
||||
| status | TEXT | pending \| in_review \| resolved \| declined \| auto_closed |
|
||||
| assigned_to | TEXT | keycloak_id moderator |
|
||||
| assigned_at | TIMESTAMPTZ | When claimed |
|
||||
| resolved_at | TIMESTAMPTZ | When closed |
|
||||
| resolved_by | TEXT | keycloak_id |
|
||||
| resolution_action | TEXT | approved \| corrected \| rejected |
|
||||
| time_in_queue_ms | INTEGER | enqueue -> start review |
|
||||
| time_in_review_ms | INTEGER | start review -> resolved |
|
||||
| created_at | TIMESTAMPTZ | Default now() |
|
||||
|
||||
Indecsi: `idx_moderation_queue_status_priority` (partial WHERE status IN ('pending','in_review')), `idx_moderation_queue_session`, `idx_moderation_queue_assigned`.
|
||||
|
||||
#### v_analysis_full (VIEW)
|
||||
|
||||
JOIN pe toate 6 tabelele de analiza (session + techniques + ai_tampered + claims + domain + verdict). Definit in migration 001. Selecteaza doar coloane sumar (nu JSONB-uri grele): session metadata, verdict scores, techniques summary, ai probability, claims summary, domain summary + explanation_ro/en.
|
||||
|
||||
Migration 011 NU modifica view-ul: coloanele HIL noi de pe `analysis_session` (review_status, human_corrected etc.) sunt acoperite automat de `SELECT s.*`.
|
||||
|
||||
---
|
||||
|
||||
### Schema: bos_parammgmt (~40 tabele)
|
||||
|
||||
Scrisa si citita exclusiv de didiFramework. Contine toti parametrii de configurare ai platformei. Sincronizata in Redis prin POST /api/sync-redis.
|
||||
|
||||
Search path setat in database.ts: `SET search_path TO bos_parammgmt, public`.
|
||||
|
||||
#### Tabel de baza
|
||||
|
||||
| Tabel | Scop | Rute CRUD |
|
||||
|-------|------|-----------|
|
||||
| parameter | Tabel parinte versionare (parameter_id, parameter_type, valid_from/to) | intern (FK din toate celelalte) |
|
||||
|
||||
#### Tehnici de manipulare (ierarhie 4 nivele)
|
||||
|
||||
| Tabel | Scop | Rute CRUD |
|
||||
|-------|------|-----------|
|
||||
| dimension | Dimensiuni top-level (code, name, weight) | /api/dimensions |
|
||||
| subdimension | Sub-dimensiuni (FK dimension) | /api/subdimensions |
|
||||
| technique | Tehnici individuale (FK subdimension, severity, confidence, detectability) | /api/techniques |
|
||||
| technique_indicator | Indicatori detectie per tehnica (name, description, max_intensity 1-3) | /api/indicators |
|
||||
| technique_validation_rule | Reguli validare per tehnica | /api/validation-rules |
|
||||
|
||||
#### Evaluare sursa
|
||||
|
||||
| Tabel | Scop | Rute CRUD |
|
||||
|-------|------|-----------|
|
||||
| platform | Platforme social media (code, name, score) | /api/platforms |
|
||||
| platform_modifier | Modificatori platforma (condition, score) | /api/platform-modifiers |
|
||||
| source_credibility | Factori credibilitate sursa | /api/source-credibility |
|
||||
| source_type | Tipuri sursa (base_score) | intern |
|
||||
| source_assessment | Evaluare sursa | intern |
|
||||
| domain_age_score | Scor varsta domeniu (range-uri, impact) | /api/domain-age-scores |
|
||||
| domain_risk_level | Nivele risc domeniu (range-uri, interpretare) | /api/domain-risk-levels |
|
||||
| domain_red_flag | Red flags domeniu (condition, severity, action) | /api/domain-red-flags |
|
||||
| author_classification | Clasificari autor (code, name, score) | /api/author-classifications |
|
||||
| author_credibility | Credibilitate autor (impact) | /api/author-credibility |
|
||||
|
||||
#### Claims
|
||||
|
||||
| Tabel | Scop | Rute CRUD |
|
||||
|-------|------|-----------|
|
||||
| claim | Statusuri claim (TRUE, FALSE, UNVERIFIED, OPINION) | /api/claims/status |
|
||||
| claim_type | Tipuri claim (factual, statistic, cauzal, etc.) | /api/claims/types |
|
||||
| confidence | Nivele incredere (level, color, action, range) | /api/claims/confidence |
|
||||
| interpretation | Interpretare scor credibilitate (range-uri) | /api/claims/interpretation |
|
||||
|
||||
#### Verdicte si scoruri
|
||||
|
||||
| Tabel | Scop | Rute CRUD |
|
||||
|-------|------|-----------|
|
||||
| verdict_category | Categorii verdict (code, range, color) | /api/verdicts/categories |
|
||||
| risk_mapping | Mapping risc (level, range, color) | /api/verdicts/risk |
|
||||
| severity_assessment | Evaluare severitate (category, range, action) | /api/verdicts/severity |
|
||||
|
||||
#### Ponderi
|
||||
|
||||
| Tabel | Scop | Rute CRUD |
|
||||
|-------|------|-----------|
|
||||
| component_weight | Ponderi componente (manipulation, claims, source, ai, context) | /api/weights/components |
|
||||
| weight_scenario | Scenarii ponderi (per topic: health, politics, etc.) | /api/weights/scenarios |
|
||||
| multiplier | Multiplicatori (topic, temporal, reach) | /api/weights/multipliers |
|
||||
|
||||
#### Provideri LLM
|
||||
|
||||
| Tabel | Scop | Rute CRUD |
|
||||
|-------|------|-----------|
|
||||
| llm_provider | Configurare provideri (base_url, auth_type, rate_limit) | /api/providers/configs |
|
||||
| llm_model | Modele LLM (context_window, cost, capabilities) | /api/providers/models |
|
||||
| component_provider_assignment | Assignment componenta -> model (legacy, pre-migration-002) | /api/providers/assignments |
|
||||
| provider_api_key | Chei API per provider (criptate, usage tracking) | /api/providers/keys |
|
||||
|
||||
#### Configurare unificata componente (adaugat prin migration 002, extinsa cu tier prin 006)
|
||||
|
||||
| Tabel | Scop | Rute CRUD |
|
||||
|-------|------|-----------|
|
||||
| component_stage_assignment | Assignment model pe etapa + **tier** (free/premium) cu fallback chain. Unique: `(component_code, stage_code, tier, fallback_order)`. | /api/providers/assignments (suporta `?tier=X` filter) |
|
||||
| component_prompt | Prompturi LLM per componenta/etapa (system_prompt, user_template) | /api/providers/prompts |
|
||||
| component_config | Config JSONB catch-all per componenta (scoring, patterns, vision models) | intern (sync-redis) |
|
||||
|
||||
**Component codes prezente dupa migrations 006-009**:
|
||||
- `techniques` (stages: techniques_screening, techniques_deep)
|
||||
- `ai-tampered` (stages: ai_tampered_screening, ai_tampered_deep)
|
||||
- `claims` (stages: claims_extraction, claims_verification)
|
||||
- `source-assessment` (stages: source_assessment_extraction, source_assessment_evaluation)
|
||||
- `vision` (stage: image_analysis — OCR + AI detection + video frames, Etapa 4)
|
||||
- `verdict` (stage: verdict_review — LLM verdict reviewer care ajusteaza scorul final + explicatii RO/EN, Etapa 5)
|
||||
|
||||
Fiecare componenta/stage are **2 tiers** (`free` + `premium`), fiecare cu propriul fallback chain (primary + 2-3 fallbacks). Ex: `techniques_screening` are 4 randuri `tier='free'` + 4 randuri `tier='premium'`.
|
||||
|
||||
Tier-ul final folosit la runtime se deriveaza din `planType` al userului (returnat de check-credits):
|
||||
- `plan_type` 1-3 (Freemium/Starter/Basic) → `tier='free'`
|
||||
- `plan_type` 4-6 (Pro/Business/Enterprise) → `tier='premium'`
|
||||
|
||||
#### Chei API extensie browser
|
||||
|
||||
| Tabel | Scop | Rute CRUD |
|
||||
|-------|------|-----------|
|
||||
| extension_api_key | Chei API extensie browser (key, user_id, usage_count) | /api/extension-keys |
|
||||
|
||||
#### Profiluri verdict per input type (adaugat 2026-03-21)
|
||||
|
||||
| Tabel | Scop | Rute CRUD |
|
||||
|-------|------|-----------|
|
||||
| input_type_profile | 6 profiluri verdict (text, image, audio, video, url) cu ponderi per componenta, reguli INCONCLUSIVE, disclosure multipliers | /api/input-profiles |
|
||||
| profile_override_config | Override-uri per profil (8 tipuri × 6 profiluri = 48 randuri) | /api/input-profiles/:code/overrides |
|
||||
|
||||
Coloane noi in tabele existente:
|
||||
- `claim_type.unverified_weight` NUMERIC(3,2) — ponderea UV per tip claim (0.25-0.50)
|
||||
- `claim.credibility_weight` NUMERIC(3,2) — ponderea credibilitate per status claim (0.00-1.00)
|
||||
|
||||
#### HIL Moderation config (adaugat prin migration 011, 2026-05-01)
|
||||
|
||||
| Tabel | Scop | Rute CRUD |
|
||||
|-------|------|-----------|
|
||||
| moderation_config | Single-row settings (CHECK config_id=1): triage thresholds (confidence_low, risk_grey_min/max, queue_relax_at, queue_strict_at) + brain client config (brain_enabled, brain_url, lookup/write timeouts, brain_confidence_min_silver, brain_semantic_threshold, brain_per_component JSONB). 14 fields total. Sincronizat in Redis ca `didi:config:moderation:v1:settings`. | /api/moderation-config |
|
||||
| sensitive_topic | Topics care declanseaza HIL review (seed: elections, health, war, covid, climate). topic_code regex `[a-z0-9_]+` UNIQUE; soft delete via is_active. Sincronizat in Redis ca `didi:config:moderation:v1:sensitive_topics`. | /api/sensitive-topics |
|
||||
| moderation_role | Mapping Keycloak role -> HIL permissions (seed: moderator, senior_moderator). Toggles: can_resolve, can_escalate, can_force_gold_brain, is_active. role_code este PK (immutable). Sincronizat in Redis ca `didi:config:moderation:v1:roles`. | /api/moderation-roles |
|
||||
|
||||
---
|
||||
|
||||
### Schema: bos_sysadmin (5 tabele)
|
||||
|
||||
Scrisa si citita de didiFramework (auth.ts, admin.ts, subscriptions.ts). Management utilizatori si abonamente.
|
||||
|
||||
| Tabel | Scop | Rute |
|
||||
|-------|------|------|
|
||||
| internet_user | Utilizator platforma (internet_user_id, person_id FK, credits_remained, credits_spent) | /api/auth/me (auto-creare), /api/admin/users |
|
||||
| user_credential | Credentiale (email, keycloak_id, enrollment_type, subscription_status) | /api/auth/me, /api/admin/users |
|
||||
| subscription | Abonament activ (internet_user_id FK, plan FK, status, activation_date) | /api/subscriptions |
|
||||
| subscription_plan | Planuri abonament (plan_name, plan_type, price, credits, limite storage/media, costuri per tip) | /api/admin/plans |
|
||||
| ai_credit_usage | Log consum credite (session_id, user_id, credits_used, input_type) | /api/auth/deduct-credits |
|
||||
|
||||
---
|
||||
|
||||
### Schema: bos_subscriber (4 tabele)
|
||||
|
||||
Scrisa de didiFramework la inregistrare utilizator. Date personale.
|
||||
|
||||
| Tabel | Scop |
|
||||
|-------|------|
|
||||
| person | Entitate persoana (person_id, person_type, status) |
|
||||
| address | Adresa (address_id, address_type) |
|
||||
| persoana_fizica | Persoana fizica romaneasca (nume, prenume, FK person, FK address) |
|
||||
| contact | Contact (person_id FK, contact_type_id, contact_info) |
|
||||
|
||||
---
|
||||
|
||||
### Schema: public
|
||||
|
||||
| Tabel | Scop |
|
||||
|-------|------|
|
||||
| waitlist | Lista de asteptare pre-lansare (vezi sectiunea container local) |
|
||||
|
||||
---
|
||||
|
||||
## Migratii aplicate
|
||||
|
||||
| Fisier | Ce face | Aplicata de |
|
||||
|--------|---------|-------------|
|
||||
| didiFramework/sql/migrations/001_add_explanation_columns.sql | Adauga explanation_ro, explanation_en la analysis_verdict + creeaza view v_analysis_full | didiFramework la pornire |
|
||||
| didiFramework/sql/migrations/002_add_component_pilot_config.sql | Adauga tabele component_stage_assignment, component_prompt, component_config | didiFramework la pornire |
|
||||
| didiFramework/sql/migrations/006_add_tier_column.sql | `component_stage_assignment.tier varchar(20) DEFAULT 'free'` + unique constraint pe (component_code, stage_code, tier, fallback_order) | Manual |
|
||||
| didiFramework/sql/migrations/007_seed_premium_assignments.sql | Seed 32 rows `tier='premium'` pentru 8 stages LLM (techniques/ai-tampered/claims/source-assessment) | Manual |
|
||||
| didiFramework/sql/migrations/008_seed_vision_assignments.sql | Seed 7 rows pentru component `vision` stage `image_analysis` (3 free + 4 premium) | Manual |
|
||||
| didiFramework/sql/migrations/009_seed_verdict_assignments.sql | Seed 8 rows pentru component `verdict` stage `verdict_review` (4 free + 4 premium) | Manual |
|
||||
| didiFramework/sql/migrations/011_add_moderation.sql | HIL Moderation foundation: 6 coloane pe `analysis_session`, tabela `moderation_queue`, 3 tabele config in bos_parammgmt (moderation_config, sensitive_topic, moderation_role) + seeds. Companion `011_rollback.sql`. session_id este UUID, FK foloseste UUID. | Manual |
|
||||
|
||||
Migratiile ulterioare (012 topic_volatility, 013 user_audit_log, 014 atomic_path_prefix, 015 social_post, 016 input_profile_versions, 017 model_catalog_attributes) sunt incluse integral in seed-ul canonic `DIDI_full_export_2026-07-02.sql`. Un restore curat al seed-ului produce schema completa la zi (fara a mai rula migratiile manual). Cateva dintre ele sunt descrise mai jos in "Schema additions".
|
||||
|
||||
---
|
||||
|
||||
## Container `staging-dataLayer-postgres` (ISTORIC — inexistent)
|
||||
|
||||
> Nota istorica: un container `staging-dataLayer-postgres` (postgres:15-alpine, database `misinformation_db`) a servit candva doar tabela `public.waitlist` + ~22 tabele legacy goale din vechiul orchestrator Python. **Acest container NU mai exista.** Baza de business (inclusiv `public.waitlist`, daca este folosita) este acum in database-ul `DIDI` de pe containerul `didi-postgres`. Orice referinta la `staging-dataLayer-postgres`, `misinformation_db` sau la path-ul arhiva `didiDatabase-legacy/` este stale si nu mai reflecta realitatea.
|
||||
|
||||
---
|
||||
|
||||
## Diagrama conexiuni
|
||||
|
||||
```
|
||||
+-----------------------------------+
|
||||
| didi-postgres:5432 |
|
||||
| Container LOCAL (postgres:17) |
|
||||
| didi-network |
|
||||
+-----------------------------------+
|
||||
| Database: DIDI |
|
||||
+-----------------------------------+
|
||||
| bos_analysis (agent-v3) |
|
||||
| bos_parammgmt (didiFramework) |
|
||||
| bos_sysadmin (didiFramework) |
|
||||
| bos_subscriber (didiFramework) |
|
||||
| public (Keycloak schema) |
|
||||
+-----------------------------------+
|
||||
^ ^ ^
|
||||
| | |
|
||||
agent-v3 didiFramework Keycloak
|
||||
(schema public)
|
||||
|
||||
Kong ruleaza DBless (fara baza proprie).
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Fisiere in directorul didiDatabase
|
||||
|
||||
```
|
||||
DIDI_full_export_2026-07-02.sql -- SEED CANONIC (23 MB): pg_dump complet DIDI
|
||||
(schema + date + migratiile 001-017). Restorabil
|
||||
cu --clean --if-exists --no-owner.
|
||||
Dockerfile -- Build imagine postgres cu init (pastrat pentru rebuild container)
|
||||
MIGRATION.md -- Note migrare PostgreSQL (atentie: contine si sectiuni stale despre cluster)
|
||||
REBUILD.md -- Reteta rebuild baza pe alt host din seed-ul canonic
|
||||
ha-cluster/ -- Config optional HA (docker-compose + haproxy.cfg) pentru fallback cluster
|
||||
.gitignore -- Exclude .env, data/
|
||||
INDEX.md -- Aceasta documentatie
|
||||
```
|
||||
|
||||
Nota seed: fisierul canonic actual este `DIDI_full_export_2026-07-02.sql`. Seed-ul vechi `DIDI_full_export_2026-03-22.sql` (fara migratiile 016/017) si pachetul demo (`DIDI_demo_seed_2026-07-02.sql` + `demo-seed/`) au fost arhivate **in afara repo-ului** (`/home/admin365/didi_seed_archive_2026-07-08/`) — livrarea foloseste DOAR full seed-ul curent.
|
||||
|
||||
---
|
||||
|
||||
## Ce NU face containerul local `didi-postgres`
|
||||
|
||||
- Nu are replicare (instanta singulara); HA se obtine doar comutand pe fallback-ul cluster din `ha-cluster/`
|
||||
- Nu are backup automat integrat (backup manual din seed / pg_dump)
|
||||
- Nu are SSL/TLS intern
|
||||
- Este sursa unica de adevar pentru datele DIDI; Redis (`didi:config:*`, `didi:framework:*`) e cache derivat, regenerat cu `sync-redis`
|
||||
|
||||
---
|
||||
|
||||
## Schema additions (2026-05-04 → 2026-05-05)
|
||||
|
||||
### `bos_parammgmt.sensitive_topic` — extins (migration 012)
|
||||
|
||||
ALTER ADD: `volatility ('volatile'|'evolving'|'stable')`, `cache_ttl_hours integer (1-26280)`, `recency_window_days integer (1-365)`, `half_life_days numeric (>0)`. Seed: war/elections=volatile@24h/7d/3d, health/covid=evolving@168h/14d/14d, climate=stable@720h/180d/180d, fraud_test=defaults. Used by brain `topic_volatility.py` to override classifier TTL per topic.
|
||||
|
||||
### `bos_sysadmin.user_audit_log` — nou (migration 013)
|
||||
|
||||
```
|
||||
audit_id bigserial PK
|
||||
internet_user_id integer (NULL pentru keycloak-only useri)
|
||||
target_email text
|
||||
target_keycloak_id text
|
||||
actor_keycloak_id text -- extras din JWT (sub claim)
|
||||
actor_email text
|
||||
action text NOT NULL -- user.{update,delete,sync,email_verified,subscription,roles,group,reset_password}
|
||||
payload jsonb DEFAULT '{}' -- diff before/after sau parametri operațiune
|
||||
request_ip text
|
||||
user_agent text
|
||||
created_at timestamptz NOT NULL DEFAULT now()
|
||||
```
|
||||
|
||||
4 indexuri: user (partial), actor, action+time, time. Powers tab "Audit Log" în UserManagement DIDI admin.
|
||||
|
||||
### Brain tables (alongside Atomic, prefix `brain_*`, public schema)
|
||||
|
||||
| Tabela | Scop | Cheie unique |
|
||||
|---|---|---|
|
||||
| `brain_analysis_atom` (existed) | Cache rezultate full-component LLM (techniques/ai_tampered/claims). +7 coloane noi: `volatility`, `topic_codes text[]`, `entity_bindings jsonb`, `ttl_hours_used`, `last_audited_at`, `audit_history jsonb` (last-50 cap), `consecutive_audit_passes` | `(content_hash, component, prompt_hash)` |
|
||||
| `brain_verification_cache` (existed) | Cache verdict LLM per claim. Same +7 coloane | `(claim_hash, tier)` |
|
||||
| `brain_fact_status` (NOU, 2026-05-04) | Current truth pentru triplete `(subject, predicate, object)`. Coloane: `current_truth bool|NULL`, `current_version_id`, `current_confidence`, `last_verified_at`, `last_evidence_urls jsonb`, `volatility`, `topic_codes`, `next_check_at`, `check_interval_hours`, `moderator_locked bool`, `moderator_user_id`, `moderator_notes` | `canonical_form_hash` |
|
||||
| `brain_fact_version` (NOU) | Temporal versioning. `truth_value bool`, `confidence`, `valid_from`, `valid_to (NULL=current)`, `source_atom_ids text[]`, `evidence_urls jsonb`, `llm_reasoning`, `created_by ('auto'|'moderator'|'breaking_news_watcher'|'auditor'|'extractor')`, `moderator_user_id`, `notes` | bigserial; FK fact_id → fact_status ON DELETE CASCADE |
|
||||
| `brain_audit_log` (NOU) | Cache mutation log: judge decisions, mass invalidations, gold promotions, fact truth changes. Coloane: `action`, `target_table`, `target_id`, `actor`, `payload jsonb` | bigserial |
|
||||
|
||||
GIN index-uri pe `topic_codes` (pentru topic-scoped invalidate). Partial index pe `cache_tier IN ('gold','silver') AND volatility != 'stable'` pentru auditor sweep. Schema migrează idempotent la fiecare brain `db.connect()`.
|
||||
|
||||
### DB live counts (2026-05-05, pe `didi-postgres`)
|
||||
|
||||
```
|
||||
internet_users: 21 | brain_fact_status: 4
|
||||
user_credentials: 20 | brain_fact_version: 1 (Putin → TRUE locked smoke-admin)
|
||||
subscriptions: 20 | brain_audit_log: ~10 (mostly fact_truth_changed + reset_password)
|
||||
subscription_plans: 20 | user_audit_log: live (logged on every PUT/DELETE/role/group)
|
||||
```
|
||||
101
backend/services/data-layer/didiDatabase/MIGRATION.md
Normal file
101
backend/services/data-layer/didiDatabase/MIGRATION.md
Normal file
|
|
@ -0,0 +1,101 @@
|
|||
# PostgreSQL — pe cluster Patroni HA (status curent)
|
||||
|
||||
> **TL;DR**: DIDI folosește **clusterul Patroni** extern (3 noduri PG + 3 etcd + 2 HAProxy LB). Containerul local `staging-dataLayer-postgres` din `data-layer/docker-compose.yml` păstrează **doar tabelul `waitlist`** — toate datele de business sunt pe cluster.
|
||||
|
||||
---
|
||||
|
||||
## Ce era aici (legacy)
|
||||
|
||||
Cândva, `staging-dataLayer-postgres` (Postgres 15 Alpine, container Docker) servea toate datele DIDI. Avea ~22 tabele în schemele `analyses`, `catalog`, `execution`, `pipelines`, `users` — toate din vechiul orchestrator Python. Acum sunt **goale, nefolosite**, schema veche arhivată în `/home/admin365/old_deprecated_code_archive/didiDatabase-legacy/`.
|
||||
|
||||
## Ce e acum
|
||||
|
||||
### Cluster Patroni (productie)
|
||||
|
||||
| Componentă | Hostname | IP | Port | Rol |
|
||||
|---|---|---|---|---|
|
||||
| pg-node1 | `pg-node1-test` | `10.11.50.160` | 5432 | Replica streaming |
|
||||
| pg-node2 | `pg-node2-test` | `10.11.50.161` | 5432 | Replica streaming |
|
||||
| **pg-node3** | `pg-node3-test` | `10.11.50.162` | 5432 | **Leader curent** |
|
||||
| etcd-node1/2/3 | — | `10.11.50.163-165` | 2379 | Quorum |
|
||||
| HAProxy LB1 | `haproxy-lb-test` | `10.11.50.166` | 5000 (RW), 5001 (RO) | Primary |
|
||||
| HAProxy LB2 | `haproxy-lb2-test` | `10.11.50.169` | 5000, 5001 | Secondary |
|
||||
| pgBackRest | `pg-backup-test` | `10.11.50.168` | — | Backup zilnic + NFS |
|
||||
|
||||
**Endpoint-uri pentru aplicații DIDI:**
|
||||
|
||||
| Scop | Endpoint | Notă |
|
||||
|---|---|---|
|
||||
| **WRITE** (orice modificare) | `10.11.50.167:5000` | DIDI configurat aici (HAProxy LB) |
|
||||
| READ (raportări) | `10.11.50.167:5001` | replica load-balanced |
|
||||
|
||||
> `.166`, `.167` și `.169` sunt toate HAProxy LB valide spre același cluster Patroni. DIDI folosește `.167` istoric. Verificat 2026-04-28: toate trei dau aceleași date (1782 sesiuni).
|
||||
|
||||
### Database principal: `DIDI`
|
||||
|
||||
User: `bos_interface` / parolă în vault-ul de credențiale `name='PostgreSQL Cluster Patroni (admin)'`.
|
||||
|
||||
4 scheme + public:
|
||||
- `bos_analysis` (6 tabele + view) — scrise de agent-v3
|
||||
- `bos_parammgmt` (~40 tabele) — scrise de didiFramework, sincronizate în Redis
|
||||
- `bos_sysadmin` (5 tabele) — utilizatori, credite, abonamente
|
||||
- `bos_subscriber` (4 tabele) — date personale
|
||||
- `public.waitlist` — pe **containerul local**, nu cluster
|
||||
|
||||
### Database-uri suplimentare pe același cluster
|
||||
|
||||
- `kong_db` (user `kong`) — folosit de Kong **cluster** (vezi `gateway-auth-layer/didiKong/MIGRATION.md`)
|
||||
- `keycloak_db` (user `keycloak`) — folosit de Keycloak
|
||||
|
||||
## Ce mai e local (containerul `staging-dataLayer-postgres`)
|
||||
|
||||
Definit în `data-layer/docker-compose.yml`. **Nu** e pe rețea externă — doar Docker network. Singurul tabel activ: `public.waitlist` în DB `misinformation_db` (3 înregistrări).
|
||||
|
||||
Folosit doar de `didiFramework/src/routes/waitlist.ts` prin pool separat (`stagingPool` cu host `staging-dataLayer-postgres`).
|
||||
|
||||
Schemele legacy (`analyses`, `catalog`, etc.) sunt goale.
|
||||
|
||||
### De ce nu am migrat waitlist pe cluster?
|
||||
|
||||
Decizie pragmatică: waitlist e public-facing (anyone-can-signup), volum mic, nu necesită HA. Containerul local e suficient. Migrare ulterioară opțională.
|
||||
|
||||
## Connection patterns în cod
|
||||
|
||||
```typescript
|
||||
// agent-v3/src/shared/persistence/pg-pool.ts
|
||||
host: '10.11.50.167', port: 5000, database: 'DIDI', user: 'bos_interface'
|
||||
|
||||
// didiFramework/src/config/database.ts (production data)
|
||||
host: '10.11.50.167', port: 5000, database: 'DIDI'
|
||||
|
||||
// didiFramework/src/routes/waitlist.ts (special — local container)
|
||||
host: 'staging-dataLayer-postgres', port: 5432, database: 'misinformation_db'
|
||||
```
|
||||
|
||||
## Verificare connectivity
|
||||
|
||||
```bash
|
||||
# Cu psql container (din host)
|
||||
docker run --rm --network host -e PGPASSWORD=<pwd> postgres:15-alpine \
|
||||
psql -h 10.11.50.167 -p 5000 -U bos_interface -d DIDI \
|
||||
-c "SELECT inet_server_addr() AS leader, now()"
|
||||
|
||||
# Patroni REST API status
|
||||
curl -s http://10.11.50.162:8008/cluster | python3 -m json.tool
|
||||
```
|
||||
|
||||
## Backup
|
||||
|
||||
pgBackRest zilnic (full săptămânal + incremental zilnic) pe `10.11.50.168` cu storage NFS la `10.11.10.150`. Toate DB-urile DIDI intră automat în stanza globală — nu trebuie config per-app.
|
||||
|
||||
## Linkuri rapide
|
||||
|
||||
- Ghid utilizare cluster: `landingzone/postgres-patroni/README.md` (repo `git.finesynergy.eu/lucian/landingzone`)
|
||||
- Onboarding aplicație nouă: `landingzone/postgres-patroni/CLAUDE_PROMPT.md`
|
||||
- HAProxy stats: `http://10.11.50.166:7000/stats`
|
||||
|
||||
## Status
|
||||
|
||||
- ✅ Migrare făcută înaintea acestui mono-repo (cluster Patroni e canonical)
|
||||
- ✅ Container local păstrat doar pentru waitlist
|
||||
- ⚠️ DIDI configurat pe HAProxy LB `.167` (canonical landingzone e `.166`); ambele rutează la același leader — schimbare cosmetică opțională
|
||||
63
backend/services/data-layer/didiDatabase/REBUILD.md
Normal file
63
backend/services/data-layer/didiDatabase/REBUILD.md
Normal file
|
|
@ -0,0 +1,63 @@
|
|||
# Rebuild bază de date pe alt host — rețetă
|
||||
|
||||
## Ce e local vs derivat (important înainte de rebuild)
|
||||
|
||||
| Store | Rol | Se seed-uiește? |
|
||||
|---|---|---|
|
||||
| **PostgreSQL** (`didi-postgres`, local pe didi11) | **sursă de adevăr** — 97 tabele, 4 scheme (bos_parammgmt, bos_analysis, bos_sysadmin, bos_subscriber) | **DA** — din dump-ul de mai jos |
|
||||
| **Redis** (`didi-cache`) chei `didi:config:*` + `didi:framework:*` | **cache derivat** din Postgres (populat de `sync-redis` din `component_config/prompt/stage_assignment`, `llm_model`, `moderation_*`) | **NU** — se regenerează cu `sync-redis` |
|
||||
| Redis `didi:pipeline:*` / `didi:queue:*` | stare runtime sesiuni (TTL) | NU — efemer |
|
||||
|
||||
**Concluzie:** NU sunt date dublate în sensul de „două surse de adevăr". Seed-uiești
|
||||
DOAR Postgres; Redis se reface singur dintr-o comandă. Nu există fișier de seed
|
||||
pentru Redis și nici nu e nevoie.
|
||||
|
||||
## Fișierul de seed
|
||||
|
||||
`DIDI_full_export_2026-07-02.sql` (23 MB) — pg_dump complet: schema + date + toate
|
||||
migrațiile (inclusiv 016 input_type_profile_version, 017 model catalog attributes).
|
||||
Restorabil (`--clean --if-exists --no-owner`). Validat: restore curat pe Postgres
|
||||
gol → 99 tabele, date reale (23 modele LLM, 83 stage assignments, 6 profiluri).
|
||||
|
||||
> Seed-ul vechi `DIDI_full_export_2026-03-22.sql` (fără migrațiile 016/017) și pachetul
|
||||
> demo (`DIDI_demo_seed_2026-07-02.sql` + `demo-seed/`) au fost arhivate în afara repo-ului
|
||||
> (`/home/admin365/didi_seed_archive_2026-07-08/`) — livrarea folosește DOAR full seed-ul curent.
|
||||
|
||||
## Pași rebuild
|
||||
|
||||
```bash
|
||||
# 1. Pornește un Postgres (local container SAU clusterul extern — vezi mai jos)
|
||||
# Aici: containerul local, ca pe didi11.
|
||||
docker compose -f services/data-layer/docker-compose.local.yml up -d didi-postgres
|
||||
until docker exec didi-postgres pg_isready -U bos_interface; do sleep 2; done
|
||||
|
||||
# 2. Restaurează schema + datele
|
||||
docker exec -i didi-postgres psql -U bos_interface -d DIDI \
|
||||
< services/data-layer/didiDatabase/DIDI_full_export_2026-07-02.sql
|
||||
# (un singur warning benign 'transaction_timeout' pe versiuni PG <17 — se ignoră)
|
||||
|
||||
# 3. Pornește restul serviciilor (agent-v3, framework, workeri) — se conectează
|
||||
# la didi-postgres prin PG_HOST/DB_HOST din compose.
|
||||
cd services/orchestration-layer/agent-v3 && docker compose up -d
|
||||
cd ../didiFramework && docker compose up -d didi-framework
|
||||
|
||||
# 4. Regenerează cache-ul Redis din Postgres (config + framework params)
|
||||
docker exec didi-framework sh -c 'wget -qO- --post-data="" http://127.0.0.1:3005/api/sync-redis'
|
||||
|
||||
# 5. (verificare) Redis populat + un răspuns 200 pe framework
|
||||
docker exec didi-cache redis-cli -a redis123 --no-auth-warning dbsize
|
||||
curl -sf http://localhost:3005/health
|
||||
```
|
||||
|
||||
## Local vs cluster extern
|
||||
|
||||
Serviciile sunt agnostice — `PG_HOST`/`DB_HOST` din compose decid ținta:
|
||||
- **didi11 (acum):** `didi-postgres` (container local, 5432).
|
||||
- **Producție/cluster:** setează `PG_HOST=10.11.50.167 PG_PORT=5000` (VIP Patroni/HAProxy).
|
||||
Același dump se restaurează în oricare; la cluster, restaurează pe leaderul RW (`:5000`).
|
||||
|
||||
## HA opțional
|
||||
|
||||
Dacă vrei Postgres HA pe noul host (nu single-node), vezi
|
||||
`ha-cluster/` (Patroni + etcd + HAProxy) — restaurează dump-ul pe `:5000` după
|
||||
`patronictl list` arată un leader.
|
||||
104
backend/services/data-layer/didiDatabase/ha-cluster/README.md
Normal file
104
backend/services/data-layer/didiDatabase/ha-cluster/README.md
Normal file
|
|
@ -0,0 +1,104 @@
|
|||
# DIDI PostgreSQL HA — Patroni + etcd + HAProxy (IaC livrabil)
|
||||
|
||||
Pachet **reproductibil** care livrează clusterul HA PostgreSQL al platformei DiDi
|
||||
ca Infrastructure-as-Code. Aceeași arhitectură rulează în producție pe VM-uri
|
||||
dedicate (vezi `../MIGRATION.md`); acest compose o reproduce integral pe un
|
||||
singur host pentru demo, recepție, DR-rehearsal și medii de test.
|
||||
|
||||
## Arhitectură
|
||||
|
||||
```
|
||||
┌────────────────────┐
|
||||
apps ──5000──▶ │ HAProxy │ ──▶ /primary (Patroni REST :8008)
|
||||
apps ──5001──▶ │ (LB + healthcheck)│ ──▶ /replica
|
||||
└─────────┬──────────┘
|
||||
┌───────────────┼───────────────┐
|
||||
┌─────▼─────┐ ┌─────▼─────┐ ┌─────▼─────┐
|
||||
│ pg-node1 │ │ pg-node2 │ │ pg-node3 │ Spilo = PostgreSQL 16
|
||||
│ Patroni │ │ Patroni │ │ Patroni │ + Patroni (Zalando)
|
||||
└─────┬─────┘ └─────┬─────┘ └─────┬─────┘
|
||||
└───────────────┼───────────────┘
|
||||
┌────────▼────────┐
|
||||
│ etcd1/2/3 (DCS) │ quorum leader-election
|
||||
└─────────────────┘
|
||||
```
|
||||
|
||||
| Rol | Producție (VM-uri) | Acest pachet |
|
||||
|---|---|---|
|
||||
| PG + Patroni ×3 | 10.11.50.160–162 | `pg-node1..3` (Spilo 16) |
|
||||
| etcd quorum ×3 | 10.11.50.163–165 | `etcd1..3` (v3.5) |
|
||||
| HAProxy | 10.11.50.166 + 169 (VIP .167) | `haproxy` :5000/:5001 |
|
||||
| Backup | pgBackRest (10.11.50.168, NFS) | vezi §Backup |
|
||||
|
||||
## Pornire
|
||||
|
||||
```bash
|
||||
docker compose up -d
|
||||
# election durează ~30-60s; verifică:
|
||||
docker exec didi-ha-pg1 patronictl list
|
||||
```
|
||||
|
||||
Conectare (contract identic cu producția):
|
||||
|
||||
```bash
|
||||
PGPASSWORD=didi-super-secret psql -h localhost -p 5000 -U postgres # RW (leader)
|
||||
PGPASSWORD=didi-super-secret psql -h localhost -p 5001 -U postgres # RO (replici)
|
||||
```
|
||||
|
||||
Restaurare schema DIDI (bos_parammgmt / bos_analysis / bos_sysadmin / bos_subscriber):
|
||||
|
||||
```bash
|
||||
PGPASSWORD=didi-super-secret psql -h localhost -p 5000 -U postgres \
|
||||
-f ../DIDI_full_export_2026-07-02.sql
|
||||
```
|
||||
|
||||
## Test failover (drill de recepție)
|
||||
|
||||
```bash
|
||||
# 1. află liderul
|
||||
docker exec didi-ha-pg1 patronictl list
|
||||
# 2. omoară-l
|
||||
docker stop didi-ha-pg2 # (dacă pg2 e leader)
|
||||
# 3. Patroni promovează o replică în secunde; HAProxy reroutează :5000
|
||||
# automat (healthcheck /primary la 3s, fall 3). Aplicațiile nu schimbă
|
||||
# nimic — se reconectează pe același endpoint.
|
||||
docker exec didi-ha-pg1 patronictl list
|
||||
# 4. reintră nodul căzut ca replică:
|
||||
docker start didi-ha-pg2
|
||||
```
|
||||
|
||||
Switchover planificat (fără downtime):
|
||||
|
||||
```bash
|
||||
docker exec didi-ha-pg1 patronictl switchover didi --force
|
||||
```
|
||||
|
||||
## Parametri
|
||||
|
||||
| Env | Default | Rol |
|
||||
|---|---|---|
|
||||
| `PG_SUPERUSER_PASSWORD` | `didi-super-secret` | postgres superuser |
|
||||
| `PG_ADMIN_PASSWORD` | `didi-admin-secret` | admin role |
|
||||
| `PG_STANDBY_PASSWORD` | `didi-standby-secret` | replicare streaming |
|
||||
|
||||
**Schimbă-le obligatoriu în producție** (`.env` lângă compose).
|
||||
|
||||
## Backup
|
||||
|
||||
În producție backup-ul e pgBackRest (full zilnic + WAL archiving pe NFS,
|
||||
nod dedicat). Pe acest pachet, echivalentul minim:
|
||||
|
||||
```bash
|
||||
docker exec didi-ha-pg1 su postgres -c \
|
||||
'pg_basebackup -h localhost -p 5432 -D /tmp/didi-backup -Ft -z -Xs'
|
||||
```
|
||||
|
||||
## Relația cu livrabilul Lot 2
|
||||
|
||||
- Modulul 5 (Baze de date SQL) cere PostgreSQL cu HA; oferta specifică
|
||||
Patroni + HAProxy. Acest director este implementarea IaC livrată —
|
||||
reproductibilă pe orice host Docker, plus instanțierea de producție
|
||||
documentată în `MIGRATION.md`.
|
||||
- Aplicațiile (agent-v3, didiFramework) sunt agnostice: `PG_HOST:PG_PORT`
|
||||
arată fie spre VIP-ul de producție (`10.11.50.167:5000`), fie spre acest
|
||||
cluster local (`localhost:5000`) — același contract, zero modificări de cod.
|
||||
|
|
@ -0,0 +1,116 @@
|
|||
# ============================================================================
|
||||
# DIDI PostgreSQL HA cluster — Patroni + etcd + HAProxy (IaC, reproducible)
|
||||
#
|
||||
# Containerized mirror of the production topology (see ../MIGRATION.md):
|
||||
# prod: 3× PG/Patroni (10.11.50.160-162) + 3× etcd (163-165)
|
||||
# + 2× HAProxy (166/169, VIP 167) + pgBackRest (168)
|
||||
# here: 3× Spilo (Patroni+PG, Zalando) + 3× etcd + 1× HAProxy
|
||||
# → same failover semantics, single-host footprint for
|
||||
# demo/recepție/DR-rehearsal.
|
||||
#
|
||||
# Endpoints (identical contract to production):
|
||||
# localhost:5000 → leader (read-write) [HAProxy checks Patroni /primary]
|
||||
# localhost:5001 → replicas (read-only) [HAProxy checks Patroni /replica]
|
||||
# localhost:7000 → HAProxy stats UI
|
||||
#
|
||||
# Usage:
|
||||
# docker compose up -d
|
||||
# # wait ~30s for leader election, then:
|
||||
# psql -h localhost -p 5000 -U postgres # password: $PG_SUPERUSER_PASSWORD
|
||||
# # restore DIDI schema:
|
||||
# psql -h localhost -p 5000 -U postgres -f ../DIDI_full_export_2026-07-02.sql
|
||||
# # failover drill:
|
||||
# docker compose stop $(docker compose ps --format '{{.Name}}' | head -1)
|
||||
# # → a replica is promoted in seconds; :5000 keeps serving writes.
|
||||
# ============================================================================
|
||||
|
||||
x-etcd-common: &etcd-common
|
||||
image: quay.io/coreos/etcd:v3.5.16
|
||||
restart: unless-stopped
|
||||
networks: [didi-ha]
|
||||
environment: &etcd-env
|
||||
ETCD_INITIAL_CLUSTER: etcd1=http://etcd1:2380,etcd2=http://etcd2:2380,etcd3=http://etcd3:2380
|
||||
ETCD_INITIAL_CLUSTER_STATE: new
|
||||
ETCD_INITIAL_CLUSTER_TOKEN: didi-pg-ha
|
||||
ETCD_AUTO_COMPACTION_RETENTION: "1"
|
||||
ETCD_ENABLE_V2: "true"
|
||||
|
||||
x-spilo-common: &spilo-common
|
||||
image: ghcr.io/zalando/spilo-16:3.3-p3
|
||||
restart: unless-stopped
|
||||
networks: [didi-ha]
|
||||
environment: &spilo-env
|
||||
SCOPE: didi # Patroni cluster name (etcd namespace)
|
||||
PGVERSION: "16"
|
||||
ETCD3_HOSTS: "'etcd1:2379','etcd2:2379','etcd3:2379'"
|
||||
PGPASSWORD_SUPERUSER: ${PG_SUPERUSER_PASSWORD:-didi-super-secret}
|
||||
PGPASSWORD_ADMIN: ${PG_ADMIN_PASSWORD:-didi-admin-secret}
|
||||
PGPASSWORD_STANDBY: ${PG_STANDBY_PASSWORD:-didi-standby-secret}
|
||||
ALLOW_NOSSL: "true"
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "curl -sf http://localhost:8008/health || exit 1"]
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 6
|
||||
start_period: 60s
|
||||
|
||||
services:
|
||||
etcd1:
|
||||
<<: *etcd-common
|
||||
container_name: didi-ha-etcd1
|
||||
command: etcd --name etcd1
|
||||
--listen-peer-urls http://0.0.0.0:2380 --initial-advertise-peer-urls http://etcd1:2380
|
||||
--listen-client-urls http://0.0.0.0:2379 --advertise-client-urls http://etcd1:2379
|
||||
etcd2:
|
||||
<<: *etcd-common
|
||||
container_name: didi-ha-etcd2
|
||||
command: etcd --name etcd2
|
||||
--listen-peer-urls http://0.0.0.0:2380 --initial-advertise-peer-urls http://etcd2:2380
|
||||
--listen-client-urls http://0.0.0.0:2379 --advertise-client-urls http://etcd2:2379
|
||||
etcd3:
|
||||
<<: *etcd-common
|
||||
container_name: didi-ha-etcd3
|
||||
command: etcd --name etcd3
|
||||
--listen-peer-urls http://0.0.0.0:2380 --initial-advertise-peer-urls http://etcd3:2380
|
||||
--listen-client-urls http://0.0.0.0:2379 --advertise-client-urls http://etcd3:2379
|
||||
|
||||
pg-node1:
|
||||
<<: *spilo-common
|
||||
container_name: didi-ha-pg1
|
||||
hostname: pg-node1
|
||||
depends_on: [etcd1, etcd2, etcd3]
|
||||
volumes: [pg1-data:/home/postgres/pgdata]
|
||||
pg-node2:
|
||||
<<: *spilo-common
|
||||
container_name: didi-ha-pg2
|
||||
hostname: pg-node2
|
||||
depends_on: [etcd1, etcd2, etcd3]
|
||||
volumes: [pg2-data:/home/postgres/pgdata]
|
||||
pg-node3:
|
||||
<<: *spilo-common
|
||||
container_name: didi-ha-pg3
|
||||
hostname: pg-node3
|
||||
depends_on: [etcd1, etcd2, etcd3]
|
||||
volumes: [pg3-data:/home/postgres/pgdata]
|
||||
|
||||
haproxy:
|
||||
image: haproxy:2.9-alpine
|
||||
container_name: didi-ha-haproxy
|
||||
restart: unless-stopped
|
||||
networks: [didi-ha]
|
||||
depends_on: [pg-node1, pg-node2, pg-node3]
|
||||
ports:
|
||||
- "5000:5000" # read-write → Patroni leader
|
||||
- "5001:5001" # read-only → replicas
|
||||
- "7000:7000" # stats UI
|
||||
volumes:
|
||||
- ./haproxy/haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg:ro
|
||||
|
||||
volumes:
|
||||
pg1-data:
|
||||
pg2-data:
|
||||
pg3-data:
|
||||
|
||||
networks:
|
||||
didi-ha:
|
||||
name: didi-ha
|
||||
|
|
@ -0,0 +1,46 @@
|
|||
# HAProxy for DIDI PostgreSQL HA — routes by Patroni REST health checks.
|
||||
# Mirrors the production LB config (10.11.50.166/169 → VIP 167).
|
||||
#
|
||||
# :5000 → the ONE node whose Patroni answers 200 on /primary (leader, RW)
|
||||
# :5001 → nodes answering 200 on /replica (round-robin, RO)
|
||||
#
|
||||
# On failover Patroni flips the health endpoints; HAProxy reroutes in
|
||||
# (inter × fall) ≈ 9s worst case without client config changes.
|
||||
|
||||
global
|
||||
maxconn 300
|
||||
log stdout format raw local0
|
||||
|
||||
defaults
|
||||
log global
|
||||
mode tcp
|
||||
retries 2
|
||||
timeout client 30m
|
||||
timeout connect 4s
|
||||
timeout server 30m
|
||||
timeout check 5s
|
||||
|
||||
listen stats
|
||||
mode http
|
||||
bind *:7000
|
||||
stats enable
|
||||
stats uri /
|
||||
|
||||
listen postgres_write
|
||||
bind *:5000
|
||||
option httpchk GET /primary
|
||||
http-check expect status 200
|
||||
default-server inter 3s fall 3 rise 2 on-marked-down shutdown-sessions
|
||||
server pg-node1 pg-node1:5432 check port 8008
|
||||
server pg-node2 pg-node2:5432 check port 8008
|
||||
server pg-node3 pg-node3:5432 check port 8008
|
||||
|
||||
listen postgres_read
|
||||
bind *:5001
|
||||
balance roundrobin
|
||||
option httpchk GET /replica
|
||||
http-check expect status 200
|
||||
default-server inter 3s fall 3 rise 2 on-marked-down shutdown-sessions
|
||||
server pg-node1 pg-node1:5432 check port 8008
|
||||
server pg-node2 pg-node2:5432 check port 8008
|
||||
server pg-node3 pg-node3:5432 check port 8008
|
||||
13
backend/services/data-layer/didiQueue/.env.example
Normal file
13
backend/services/data-layer/didiQueue/.env.example
Normal file
|
|
@ -0,0 +1,13 @@
|
|||
# RabbitMQ Configuration
|
||||
# CHANGE THESE FOR PRODUCTION!
|
||||
RABBITMQ_USER=admin
|
||||
RABBITMQ_PASSWORD=CHANGE_ME_IN_PRODUCTION
|
||||
RABBITMQ_VHOST=/
|
||||
|
||||
# Port Configuration
|
||||
RABBITMQ_PORT=5672
|
||||
RABBITMQ_MGMT_PORT=15672
|
||||
|
||||
# Resource Limits (optional)
|
||||
RABBITMQ_VM_MEMORY_HIGH_WATERMARK=0.4
|
||||
RABBITMQ_DISK_FREE_LIMIT=1GB
|
||||
356
backend/services/data-layer/didiQueue/INDEX.md
Normal file
356
backend/services/data-layer/didiQueue/INDEX.md
Normal file
|
|
@ -0,0 +1,356 @@
|
|||
# didiQueue - Index
|
||||
|
||||
Coada de mesaje RabbitMQ pentru procesarea asincrona a analizelor. Primeste task-uri de analiza de la agent-v3, le distribuie la workeri pe componente, si colecteaza rezultatele intr-un agregator de verdict. Nu contine cod custom -- doar container RabbitMQ cu script de initializare.
|
||||
|
||||
**Productia ruleaza pe RabbitMQ LOCAL** (container `staging-dataLayer-rabbitmq`). Decizie: stabilitate + zero dependinte externe. Clusterul RabbitMQ RAG (managed extern) ramane configurat ca **fallback de urgenta pentru HA**, activabil cu `redis-switch.sh cluster --rabbit`, nu este folosit operational acum.
|
||||
|
||||
**Productie (LOCAL — activ)**:
|
||||
- Imagine: `rabbitmq:3.12-management-alpine`
|
||||
- Container: `staging-dataLayer-rabbitmq`
|
||||
- Port AMQP: 5672 (expus pe host: `0.0.0.0:5672->5672`)
|
||||
- Port Management UI: 15672 (expus pe host: `0.0.0.0:15672->15672`)
|
||||
- Credentiale: `admin` / `rabbitmq123` (din `.env`)
|
||||
- Vhost: `/`
|
||||
- Retea: `didi-network`
|
||||
|
||||
**Fallback HA (cluster RAG — disponibil dar inactiv)**:
|
||||
- Endpoint: `10.11.50.100:16672` (HAProxy VIP)
|
||||
- Vhost: `/didi`
|
||||
- User: `didi`
|
||||
- Parola: din `.cluster-credentials.env` (gitignored)
|
||||
|
||||
---
|
||||
|
||||
## Ce face
|
||||
|
||||
1. **Primeste task-uri de analiza** -- publicate de agent-v3 dispatcher
|
||||
2. **Distribuie catre workeri** -- fiecare componenta (techniques, ai_tampered, claims, domain) are cozi separate, plus media-preprocess pentru audio/video
|
||||
3. **Prioritizeaza dupa plan** -- plan 1 (free) = prioritate 1, plan 6 (enterprise) = prioritate 10
|
||||
4. **Colecteaza rezultate** -- workerii publica in coada de rezultate, agregatorul face fan-in
|
||||
5. **DLQ** -- mesajele care esueaza dupa 3 incercari merg in dead-letter queue
|
||||
6. **TTL** -- mesajele expira dupa 24 ore
|
||||
|
||||
**Nota HIL Moderation**: tabela `moderation_queue` (schema `bos_analysis`) este o tabela PostgreSQL pentru starea de review uman, NU o coada AMQP. RabbitMQ ramane folosit doar pentru dispatch-ul analizei asincrone (5 familii de cozi componente x 6 plan tiers = 30 cozi: media_preprocess, techniques, ai_tampered, claims, domain — plus coada `analysis.results` + DLQ). HIL nu introduce cozi noi.
|
||||
|
||||
---
|
||||
|
||||
## Topologie cozi
|
||||
|
||||
### Exchange
|
||||
|
||||
| Nume | Tip | Durabil | Scop |
|
||||
|------|-----|---------|------|
|
||||
| analysis | topic | da | Ruteaza task-uri si rezultate |
|
||||
|
||||
### Cozi componente (30 total = 5 cozi x 6 plan types)
|
||||
|
||||
```
|
||||
analysis.media_preprocess.1 analysis.media_preprocess.2 ... analysis.media_preprocess.6
|
||||
analysis.techniques.1 analysis.techniques.2 ... analysis.techniques.6
|
||||
analysis.ai_tampered.1 analysis.ai_tampered.2 ... analysis.ai_tampered.6
|
||||
analysis.claims.1 analysis.claims.2 ... analysis.claims.6
|
||||
analysis.domain.1 analysis.domain.2 ... analysis.domain.6
|
||||
```
|
||||
|
||||
Configurare per coada:
|
||||
- Durabil: da
|
||||
- Max prioritate: 10
|
||||
- Dead-letter exchange: '' (default)
|
||||
- Dead-letter routing key: analysis_dlq
|
||||
- Message TTL: 86,400,000 ms (24 ore)
|
||||
|
||||
### Coada rezultate (fan-in)
|
||||
|
||||
| Coada | Bindings | Scop |
|
||||
|-------|----------|------|
|
||||
| analysis.results | analysis.results.techniques, analysis.results.ai_tampered, analysis.results.claims, analysis.results.domain | Colecteaza rezultate de la toti workerii |
|
||||
|
||||
### Dead-letter queue
|
||||
|
||||
| Coada | Scop |
|
||||
|-------|------|
|
||||
| analysis_dlq | Mesaje care au esuat dupa 3 retry-uri |
|
||||
|
||||
---
|
||||
|
||||
## Prioritati per plan
|
||||
|
||||
| Plan Type | Prioritate | Tip utilizator |
|
||||
|-----------|------------|----------------|
|
||||
| 1 | 1 | freemium |
|
||||
| 2 | 2 | starter |
|
||||
| 3 | 4 | basic |
|
||||
| 4 | 6 | pro |
|
||||
| 5 | 8 | business |
|
||||
| 6 | 10 | enterprise |
|
||||
|
||||
Mesajele cu prioritate mai mare sunt procesate primele din coada.
|
||||
|
||||
---
|
||||
|
||||
## Format mesaje
|
||||
|
||||
### Task message (Dispatcher -> Worker)
|
||||
|
||||
Publicat de agent-v3 dispatcher in cozile de componente.
|
||||
|
||||
```
|
||||
{
|
||||
sessionId: "uuid",
|
||||
component: "techniques" | "ai_tampered" | "claims" | "domain",
|
||||
planType: 1-6,
|
||||
priority: 1-10,
|
||||
input: {
|
||||
content: "text de analizat",
|
||||
url: "URL optional (video/domain)",
|
||||
mediaPath: "cale MinIO optional (audio/video)",
|
||||
inputType: "text" | "url" | "image" | "audio" | "video"
|
||||
},
|
||||
userId: "string optional",
|
||||
userEmail: "string optional",
|
||||
timestamp: 1695312000000,
|
||||
retryCount: 0
|
||||
}
|
||||
```
|
||||
|
||||
AMQP properties: persistent=true, contentType=application/json, headers={sessionId, component, planType}
|
||||
|
||||
### Result message (Worker -> Aggregator)
|
||||
|
||||
Publicat de worker in coada analysis.results.
|
||||
|
||||
```
|
||||
{
|
||||
sessionId: "uuid",
|
||||
component: "techniques" | "ai_tampered" | "claims" | "domain",
|
||||
success: true | false,
|
||||
score: 0-100,
|
||||
data: { ... rezultat flat componenta ... },
|
||||
error: "mesaj eroare daca success=false",
|
||||
processingTime: 3500,
|
||||
timestamp: 1695312003500
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Cine publica mesaje
|
||||
|
||||
| Cine | Ce publica | In ce coada | Logica in fisier |
|
||||
|------|-----------|-------------|------------------|
|
||||
| agent-v3 dispatcher | Task-uri de analiza | analysis.{component}.{planType} | agent-v3/src/queue/dispatcher.ts |
|
||||
| Workeri componente | Rezultate analiza | analysis.results.{component} | agent-v3/src/queue/workers/component-worker.ts |
|
||||
|
||||
## Cine consuma mesaje
|
||||
|
||||
| Cine | Din ce coada | Ce face | Replici Docker |
|
||||
|------|-------------|---------|----------------|
|
||||
| worker-media-preprocess | analysis.media_preprocess.1-6 | Download yt-dlp + ffmpeg cadre + Whisper + Vision OCR, cache in Redis, dispatch task-uri componente | 2 |
|
||||
| worker-techniques | analysis.techniques.1-6 | Ruleaza TechniquesV3Executor | 2 (prefetch 5) |
|
||||
| worker-ai-tampered | analysis.ai_tampered.1-6 | Ruleaza AITamperedExecutor | 2 (prefetch 5) |
|
||||
| worker-claims | analysis.claims.1-6 | Ruleaza ClaimsExecutor | 3 (prefetch 3) |
|
||||
| worker-domain | analysis.domain.1-6 | Ruleaza analyzeDomain() | 2 (prefetch 10) |
|
||||
| verdict-aggregator | analysis.results | Fan-in + VerdictCalculator | 2 (prefetch 10) |
|
||||
|
||||
Claims are 3 replici (nu 2) pentru ca e cel mai lent (cautare web per claim).
|
||||
Domain are prefetch 10 pentru ca e cel mai rapid (analiza locala, fara LLM).
|
||||
Media-preprocess este nou (din 2026-03): centralizeaza download/transcribe/vision pentru audio+video, inlocuind logica per-worker. Workerii componente citesc media procesata din Redis (TTL 1h, chei `agent:media:{sessionId}:transcript`, `agent:media:{sessionId}:vision:misinformation`, etc.).
|
||||
|
||||
---
|
||||
|
||||
## Fluxul complet async
|
||||
|
||||
```
|
||||
Client POST /api/v3/pipeline/analyze-async { text, plan_type: 4 }
|
||||
|
|
||||
v
|
||||
agent-v3 Dispatcher
|
||||
|-- Salveaza SessionState in Redis (status: processing)
|
||||
|-- Salveaza sesiune initiala in Redis + PostgreSQL
|
||||
|-- Publica 4 task-uri in RabbitMQ:
|
||||
| analysis.techniques.4 (prioritate 6)
|
||||
| analysis.ai_tampered.4 (prioritate 6)
|
||||
| analysis.claims.4 (prioritate 6)
|
||||
| analysis.domain.4 (prioritate 6)
|
||||
|
|
||||
v
|
||||
Response 202: { session_id, poll_url, result_url }
|
||||
|
||||
--- In paralel, 4 workeri proceseaza ---
|
||||
|
||||
Worker Techniques (consuma din analysis.techniques.4)
|
||||
|-- Achizitioneaza lock Redis (300s TTL)
|
||||
|-- Ruleaza TechniquesV3Executor (screening -> deep analysis)
|
||||
|-- Publica rezultat in analysis.results.techniques
|
||||
|-- ACK mesaj
|
||||
|
||||
Worker AI-Tampered (consuma din analysis.ai_tampered.4)
|
||||
|-- Ruleaza AITamperedExecutor
|
||||
|-- Publica rezultat in analysis.results.ai_tampered
|
||||
|
||||
Worker Claims (consuma din analysis.claims.4)
|
||||
|-- Ruleaza ClaimsExecutor (extrage + verifica prin web)
|
||||
|-- Publica rezultat in analysis.results.claims
|
||||
|
||||
Worker Domain (consuma din analysis.domain.4)
|
||||
|-- Ruleaza analyzeDomain()
|
||||
|-- Publica rezultat in analysis.results.domain
|
||||
|
||||
--- Agregatorul colecteaza ---
|
||||
|
||||
Verdict Aggregator (consuma din analysis.results)
|
||||
|-- Primeste rezultat componenta
|
||||
|-- Achizitioneaza lock Redis (30s TTL)
|
||||
|-- Actualizeaza SessionState in Redis (completedComponents++)
|
||||
|-- Daca toate 4 componente gata:
|
||||
| |-- VerdictCalculator.calculate() (functie pura)
|
||||
| |-- VerdictExplanation.generate() (LLM, RO+EN)
|
||||
| |-- PersistService.persist() (Redis + PostgreSQL)
|
||||
|-- ACK mesaj
|
||||
|
||||
--- Clientul polleaza ---
|
||||
|
||||
GET /api/v3/pipeline/{sessionId}/queue-status
|
||||
-> { progress: 75%, completed_components: ["techniques", "ai_tampered", "domain"] }
|
||||
|
||||
GET /api/v3/pipeline/{sessionId}/result
|
||||
-> AnalysisSession completa (cand status=completed)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Retry si error handling
|
||||
|
||||
| Situatie | Actiune | Rezultat |
|
||||
|----------|---------|----------|
|
||||
| Procesare reusita | channel.ack(msg) | Mesaj sters din coada |
|
||||
| Eroare retryable + retryCount < 3 | channel.nack(msg, false, true) | Mesaj pus inapoi in coada |
|
||||
| Eroare retryable + retryCount >= 3 | channel.nack(msg, false, false) | Mesaj trimis in analysis_dlq |
|
||||
| Eroare non-retryable | channel.nack(msg, false, false) | Mesaj trimis in analysis_dlq |
|
||||
| RabbitMQ indisponibil | Fallback sync | agent-v3 ruleaza analiza sincrona |
|
||||
|
||||
Lock-uri Redis previn procesarea dubla:
|
||||
- Lock componenta: `didi:queue:lock:{sessionId}:{component}` (TTL 300s)
|
||||
- Lock agregator: `didi:queue:lock:aggregator:{sessionId}` (TTL 30s)
|
||||
|
||||
---
|
||||
|
||||
## Procesare media in workeri
|
||||
|
||||
Workerii proceseaza media inainte de analiza text:
|
||||
|
||||
| Input type | Ce face workerul | Logica in |
|
||||
|-----------|-----------------|-----------|
|
||||
| text | Nimic, trimite direct la executor | component-worker.ts |
|
||||
| audio | Transcriere via M17/Groq/OpenAI | shared/media/transcription.ts |
|
||||
| video | Download yt-dlp + ffmpeg cadre + transcriere | shared/media/video-processor.ts |
|
||||
| image | Extragere text via vision cascade (pentru techniques/claims) | shared/media/vision.ts |
|
||||
|
||||
Timeout-uri worker:
|
||||
- Video: 600,000 ms (10 minute)
|
||||
- Default: 120,000 ms (2 minute)
|
||||
|
||||
---
|
||||
|
||||
## Conexiune RabbitMQ (din agent-v3)
|
||||
|
||||
Fisier: `agent-v3/src/queue/connection.ts` + `agent-v3/src/shared/queue/constants.ts`
|
||||
|
||||
- Lazy initialization (conectare la prima utilizare)
|
||||
- Doua canale: regular (consume) + confirm (publish cu confirmare)
|
||||
- Auto-recovery la deconectare (`CONNECTION_RETRY_DELAY = 2s` pentru failover HA pe cluster)
|
||||
- Graceful shutdown pe SIGTERM/SIGINT (stop consume, close channels, close connection)
|
||||
- URL building foloseste `encodeURIComponent()` pentru vhost (`/didi` -> `%2Fdidi` in AMQP URI)
|
||||
|
||||
```
|
||||
# Productie (LOCAL — activ)
|
||||
URL: amqp://admin:rabbitmq123@staging-dataLayer-rabbitmq:5672/ (vhost `/`)
|
||||
|
||||
# Fallback HA (cluster RAG, HAProxy VIP — disponibil dar inactiv)
|
||||
URL: amqp://didi:<password>@10.11.50.100:16672/%2Fdidi
|
||||
```
|
||||
|
||||
### Switch intre cluster si local
|
||||
|
||||
Script: `backend/services/orchestration-layer/scripts/redis-switch.sh` (denumirea istorica este `redis-switch`, dar suporta si RabbitMQ via flag `--rabbit`).
|
||||
|
||||
```
|
||||
redis-switch.sh {cluster|local|status} [redis|rabbit|both]
|
||||
```
|
||||
|
||||
Verificare topologie: `agent-v3/scripts/verify-rabbitmq-cluster.ts` (verifica privilegii vhost + topologia celor 30 cozi + exchange + DLQ).
|
||||
|
||||
---
|
||||
|
||||
## Fisiere in directorul didiQueue
|
||||
|
||||
```
|
||||
init-queues.sh -- Creeaza exchange + coada legacy singulara + DLQ + binding + policy (idempotent)
|
||||
.env -- Credentiale + porturi
|
||||
.env.example -- Template
|
||||
README.md -- Documentatie
|
||||
```
|
||||
|
||||
Zero cod custom. Doar container RabbitMQ standard cu management plugin.
|
||||
|
||||
Nota: init-queues.sh creeaza topologia legacy cu o singura coada. Topologia actuala cu 30 cozi (5 componente x 6 planuri: media_preprocess, techniques, ai_tampered, claims, domain) + coada results + DLQ este creata dinamic de workerii agent-v3 la startup (vezi agent-v3/src/queue/connection.ts si constants.ts).
|
||||
|
||||
---
|
||||
|
||||
## Fisiere cod integrare (in agent-v3)
|
||||
|
||||
| Fisier | Rol |
|
||||
|--------|-----|
|
||||
| agent-v3/src/queue/connection.ts | Manager conexiune RabbitMQ (lazy, auto-recovery) |
|
||||
| agent-v3/src/shared/queue/constants.ts | Nume exchange/cozi, prioritati, config workeri |
|
||||
| agent-v3/src/queue/dispatcher.ts | Publica task-uri in cozi componente |
|
||||
| agent-v3/src/queue/aggregator.ts | Consuma rezultate, calculeaza verdict, persista |
|
||||
| agent-v3/src/queue/workers/component-worker.ts | Worker generic (lock, procesare, publish result, ack/nack) |
|
||||
| agent-v3/src/worker-entrypoints/techniques.ts | Entry point Docker worker techniques |
|
||||
| agent-v3/src/worker-entrypoints/ai-tampered.ts | Entry point Docker worker ai-tampered |
|
||||
| agent-v3/src/worker-entrypoints/claims.ts | Entry point Docker worker claims |
|
||||
| agent-v3/src/worker-entrypoints/domain.ts | Entry point Docker worker domain |
|
||||
| agent-v3/src/worker-entrypoints/aggregator.ts | Entry point Docker verdict aggregator |
|
||||
|
||||
---
|
||||
|
||||
## Configurare Docker
|
||||
|
||||
```yaml
|
||||
# din data-layer/docker-compose.yml
|
||||
staging-dataLayer-rabbitmq:
|
||||
image: rabbitmq:3.12-management-alpine
|
||||
container_name: staging-dataLayer-rabbitmq
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
RABBITMQ_DEFAULT_USER: admin
|
||||
RABBITMQ_DEFAULT_PASS: rabbitmq123
|
||||
RABBITMQ_DEFAULT_VHOST: /
|
||||
ports:
|
||||
- "5672:5672" # AMQP (expus pe host)
|
||||
- "15672:15672" # Management UI (expus pe host)
|
||||
volumes:
|
||||
- didi-staging-rabbitmq-data:/var/lib/rabbitmq
|
||||
networks:
|
||||
- didi-network
|
||||
healthcheck:
|
||||
test: rabbitmq-diagnostics -q ping
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 3
|
||||
start_period: 60s
|
||||
```
|
||||
|
||||
Workerii sunt definiti in agent-v3/docker-compose.yml (vezi agent-v3/INDEX.md pentru detalii replici).
|
||||
|
||||
---
|
||||
|
||||
## Ce NU face
|
||||
|
||||
- Nu are cod custom (container RabbitMQ standard)
|
||||
- Nu are clustering (instanta singulara)
|
||||
- Nu are mirroring/quorum queues (nu e HA)
|
||||
- Nu are SSL/TLS (AMQP plain text intern)
|
||||
- Nu are ACL per serviciu (toti folosesc userul admin)
|
||||
- Nu are delayed message plugin (retry prin requeue nativ)
|
||||
- Nu are shovel/federation (nu transfera mesaje intre brokeri)
|
||||
147
backend/services/data-layer/didiQueue/MIGRATION.md
Normal file
147
backend/services/data-layer/didiQueue/MIGRATION.md
Normal file
|
|
@ -0,0 +1,147 @@
|
|||
# RabbitMQ — migrat pe clusterul RAG (2026-04-22)
|
||||
|
||||
> **TL;DR**: DIDI folosește **clusterul RabbitMQ RAG** (3 noduri 3.13 + HAProxy VIP). Containerul local `staging-dataLayer-rabbitmq` din `data-layer/docker-compose.yml` e **oprit dar păstrat** ca fallback rapid. Conexiunea e centralizată via `agent-v3/src/queue/connection.ts`.
|
||||
|
||||
---
|
||||
|
||||
## Ce era aici (legacy)
|
||||
|
||||
`staging-dataLayer-rabbitmq` — un container RabbitMQ 3.12.6 management standalone pe `didi-network` Docker. Single-node, fără mirroring, vhost `/`. Topologie creată dinamic la startup workerilor.
|
||||
|
||||
## Ce e acum
|
||||
|
||||
### Cluster RabbitMQ RAG (producție)
|
||||
|
||||
| Nod | Hostname | IP | AMQP | Mgmt UI |
|
||||
|---|---|---|---|---|
|
||||
| rag01 (HAProxy VIP) | `rag01` | `10.11.50.100` | 16672 (VIP), 5672 direct | 16673 (VIP), 15672 direct |
|
||||
| rag02 | `rag02` | `10.11.50.102` | 5672 | 15672 |
|
||||
| rag03 | `rag03` | `10.11.50.103` | 5672 | 15672 |
|
||||
|
||||
**Endpoint-uri pentru aplicații:**
|
||||
|
||||
| Scop | Endpoint |
|
||||
|---|---|
|
||||
| **AMQP VIP** (publish + consume) | `10.11.50.100:16672` |
|
||||
| **Management UI** | `http://10.11.50.100:16673` (admin/`<pwd>`) |
|
||||
| HAProxy stats | `http://10.11.50.100:8404/stats` |
|
||||
|
||||
Vhost DIDI: `/didi` (izolare totală — alte vhost-uri (`notify`, `/hassio`) nu se văd). User: `didi` cu permisiuni full pe `/didi`. Parolă în `vault-ul de credențiale` `name='DIDI Platform RabbitMQ'`.
|
||||
|
||||
URL AMQP în cod: `amqp://didi:<pwd>@10.11.50.100:16672/%2Fdidi` (slash URL-encoded ca `%2F`).
|
||||
|
||||
### Conexiune centralizată în cod
|
||||
|
||||
`agent-v3/src/queue/connection.ts` — manager unic ioredis-style cu lazy initialization, două canale (regular consume + confirm publish), auto-recovery la 2s pentru failover HA, graceful shutdown pe SIGTERM/SIGINT.
|
||||
|
||||
`agent-v3/src/shared/queue/constants.ts` — topologie cozi (exchange, routing keys, prioritați).
|
||||
|
||||
## Topologie cozi (creată dinamic de workeri la startup)
|
||||
|
||||
### Exchange
|
||||
|
||||
| Nume | Tip | Durabil |
|
||||
|---|---|---|
|
||||
| `analysis` | topic | da |
|
||||
|
||||
### Cozi componente (24 = 4 componente × 6 plan types)
|
||||
|
||||
```
|
||||
analysis.techniques.{1..6} analysis.ai_tampered.{1..6}
|
||||
analysis.claims.{1..6} analysis.domain.{1..6}
|
||||
```
|
||||
|
||||
Fiecare cu max-priority=10, dead-letter exchange, message TTL 24h.
|
||||
|
||||
### Coadă agregare
|
||||
|
||||
`analysis.results` (fan-in) cu bindings de la fiecare componentă.
|
||||
|
||||
### Prioritați per plan
|
||||
|
||||
| Plan Type | Prioritate | Tip |
|
||||
|---|---|---|
|
||||
| 1 | 1 | freemium |
|
||||
| 2 | 2 | starter |
|
||||
| 3 | 4 | basic |
|
||||
| 4 | 6 | pro |
|
||||
| 5 | 8 | business |
|
||||
| 6 | 10 | enterprise |
|
||||
|
||||
## Containerul local `staging-dataLayer-rabbitmq`
|
||||
|
||||
Definit în `data-layer/docker-compose.yml`, configurat să pornească dar **manual oprit** ca parte din migrare. Volumul `didi-staging-rabbitmq-data` e intact.
|
||||
|
||||
Status curent: `Exited`.
|
||||
|
||||
### De ce e păstrat?
|
||||
|
||||
Fallback rapid dacă cluster RAG e indisponibil. Pentru reactivare temporară:
|
||||
|
||||
```bash
|
||||
backend/services/orchestration-layer/scripts/redis-switch.sh local rabbit
|
||||
# (același script gestionează rabbit + redis)
|
||||
```
|
||||
|
||||
## Switch rapid cluster ↔ local
|
||||
|
||||
```bash
|
||||
# Folosește local
|
||||
./redis-switch.sh local rabbit
|
||||
|
||||
# Cluster (default)
|
||||
./redis-switch.sh cluster rabbit
|
||||
|
||||
# Both
|
||||
./redis-switch.sh cluster both
|
||||
|
||||
# Status
|
||||
./redis-switch.sh status both
|
||||
```
|
||||
|
||||
## Verificare cluster
|
||||
|
||||
```bash
|
||||
# Mgmt UI (browser)
|
||||
http://10.11.50.100:16673 # admin/<pwd>
|
||||
|
||||
# Quick test prin rabbitmqctl pe nod cluster
|
||||
ssh admin365@10.11.50.102
|
||||
sudo rabbitmqctl status
|
||||
sudo rabbitmqctl list_vhosts
|
||||
sudo rabbitmqctl list_queues -p /didi name messages consumers
|
||||
|
||||
# Via HTTP API
|
||||
curl -s -u admin:<pwd> http://10.11.50.100:16673/api/overview | python3 -m json.tool
|
||||
```
|
||||
|
||||
## Workerii (în `agent-v3` docker-compose)
|
||||
|
||||
Servicii Docker care consumă din cluster:
|
||||
|
||||
| Worker | Replici | Coadă |
|
||||
|---|---|---|
|
||||
| `worker-techniques` | 2 | `analysis.techniques.1-6` |
|
||||
| `worker-ai-tampered` | 2 | `analysis.ai_tampered.1-6` |
|
||||
| `worker-claims` | 3 | `analysis.claims.1-6` (mai multe replici, mai lent) |
|
||||
| `worker-domain` | 2 | `analysis.domain.1-6` |
|
||||
| `worker-media-preprocess` | 2 | `analysis.media_preprocess.*` |
|
||||
| `verdict-aggregator` | 2 | `analysis.results` |
|
||||
|
||||
## Verificări dispatcher
|
||||
|
||||
Script verificare cluster ready: `agent-v3/scripts/verify-rabbitmq-cluster.ts` — testează conectivitate, vhost privileges, topology, în funcție de user `didi` și endpoint VIP.
|
||||
|
||||
## Linkuri rapide
|
||||
|
||||
- Ghid utilizare cluster: `landingzone/rabbitmq-rag/README.md` (repo `git.finesynergy.eu/lucian/landingzone`)
|
||||
- Onboarding vhost nou: `landingzone/rabbitmq-rag/CLAUDE_PROMPT.md`
|
||||
- Mgmt UI: `http://10.11.50.100:16673`
|
||||
|
||||
## Status
|
||||
|
||||
- ✅ Migrare aplicată: 2026-04-22
|
||||
- ✅ Vhost `/didi` izolat, user `didi` cu permisiuni minime
|
||||
- ✅ Container local păstrat ca fallback (oprit, volum intact)
|
||||
- ✅ Topologie creată dinamic la startup (24 cozi + results + DLQ)
|
||||
- ✅ Workerii (în agent-v3) consumă cu prefetch ajustat per componentă
|
||||
215
backend/services/data-layer/didiQueue/README.md
Normal file
215
backend/services/data-layer/didiQueue/README.md
Normal file
|
|
@ -0,0 +1,215 @@
|
|||
# didiQueue - RabbitMQ Message Queue Service 🐰
|
||||
|
||||
## Overview
|
||||
RabbitMQ message broker for asynchronous communication between the Orchestrator and Analysis Service in the DIDI Backend platform.
|
||||
|
||||
## 🎯 Purpose
|
||||
Provides reliable message queuing for pipeline execution jobs, decoupling the API layer from the processing layer.
|
||||
|
||||
## 🚀 Quick Start
|
||||
|
||||
### Start the Service
|
||||
```bash
|
||||
# From this directory
|
||||
docker compose up -d
|
||||
|
||||
# Or from data-layer directory
|
||||
make up-queue
|
||||
```
|
||||
|
||||
### Access Points
|
||||
- **AMQP Protocol**: `localhost:5672`
|
||||
- **Management UI**: `http://localhost:15672`
|
||||
- **Default Credentials**: `admin / rabbitmq123`
|
||||
|
||||
## 📊 Queue Architecture
|
||||
|
||||
Since we're merging all analysis services into one unified service, we use a **single queue**:
|
||||
|
||||
```
|
||||
Orchestrator → publishes → analysis_queue → consumed by → Analysis Service
|
||||
```
|
||||
|
||||
### Queue Configuration
|
||||
- **Queue Name**: `analysis_queue`
|
||||
- **Type**: Durable (survives restarts)
|
||||
- **Dead Letter Queue**: `analysis_dlq` (for failed messages)
|
||||
- **Message TTL**: 24 hours
|
||||
- **Auto-delete**: No
|
||||
|
||||
## 🔧 Configuration
|
||||
|
||||
### Environment Variables
|
||||
Edit `.env` file to customize:
|
||||
```env
|
||||
RABBITMQ_USER=admin
|
||||
RABBITMQ_PASSWORD=rabbitmq123 # CHANGE IN PRODUCTION!
|
||||
RABBITMQ_VHOST=/
|
||||
RABBITMQ_PORT=5672
|
||||
RABBITMQ_MGMT_PORT=15672
|
||||
```
|
||||
|
||||
### Resource Limits
|
||||
```yaml
|
||||
Memory: 1GB (max) / 512MB (reserved)
|
||||
CPU: 0.5 cores (max) / 0.25 cores (reserved)
|
||||
```
|
||||
|
||||
## 📝 Message Format
|
||||
|
||||
Messages published to the queue follow this structure:
|
||||
```json
|
||||
{
|
||||
"run_id": "analysis_abc123_20250901_120000",
|
||||
"pipeline_id": "uuid-here",
|
||||
"pipeline_version": 1,
|
||||
"input_data": {
|
||||
"text": "Content to analyze",
|
||||
"image": "base64_or_url",
|
||||
"audio": "url_to_audio",
|
||||
"video": "url_to_video"
|
||||
},
|
||||
"media_type": "text|image|audio|video",
|
||||
"created_at": "2025-09-01T12:00:00Z"
|
||||
}
|
||||
```
|
||||
|
||||
## 🔍 Management
|
||||
|
||||
### View Queue Status
|
||||
```bash
|
||||
# Using Management UI
|
||||
http://localhost:15672
|
||||
|
||||
# Using CLI
|
||||
docker exec didi-queue rabbitmqctl list_queues
|
||||
|
||||
# Check queue depth
|
||||
docker exec didi-queue rabbitmqctl list_queues name messages_ready messages_unacknowledged
|
||||
```
|
||||
|
||||
### Purge Queue (Development Only)
|
||||
```bash
|
||||
# Remove all messages from queue
|
||||
docker exec didi-queue rabbitmqctl purge_queue analysis_queue
|
||||
```
|
||||
|
||||
### Health Check
|
||||
```bash
|
||||
# Check if RabbitMQ is responsive
|
||||
docker exec didi-queue rabbitmq-diagnostics -q ping
|
||||
|
||||
# Detailed health check
|
||||
docker exec didi-queue rabbitmq-diagnostics check_running
|
||||
```
|
||||
|
||||
## 🏗️ Integration Points
|
||||
|
||||
### Publishers (Orchestrator)
|
||||
```python
|
||||
import aio_pika
|
||||
|
||||
# Connect
|
||||
connection = await aio_pika.connect_robust(
|
||||
"amqp://admin:rabbitmq123@localhost:5672/"
|
||||
)
|
||||
channel = await connection.channel()
|
||||
|
||||
# Publish message
|
||||
await channel.default_exchange.publish(
|
||||
aio_pika.Message(body=json.dumps(message).encode()),
|
||||
routing_key="analysis_queue"
|
||||
)
|
||||
```
|
||||
|
||||
### Consumers (Analysis Service)
|
||||
```python
|
||||
# Declare queue
|
||||
queue = await channel.declare_queue("analysis_queue", durable=True)
|
||||
|
||||
# Consume messages
|
||||
async for message in queue:
|
||||
async with message.process():
|
||||
body = json.loads(message.body.decode())
|
||||
# Process the message
|
||||
```
|
||||
|
||||
## 🛠️ Troubleshooting
|
||||
|
||||
### Queue is not created
|
||||
The `init-queues.sh` script runs automatically on container start. Check logs:
|
||||
```bash
|
||||
docker logs didi-queue
|
||||
```
|
||||
|
||||
### Messages not being consumed
|
||||
1. Check if Analysis Service is running
|
||||
2. Verify queue has messages: `docker exec didi-queue rabbitmqctl list_queues`
|
||||
3. Check for dead letter queue: `docker exec didi-queue rabbitmqctl list_queues | grep dlq`
|
||||
|
||||
### High memory usage
|
||||
```bash
|
||||
# Check memory usage
|
||||
docker exec didi-queue rabbitmq-diagnostics memory_breakdown
|
||||
|
||||
# Set memory limit
|
||||
docker exec didi-queue rabbitmqctl set_vm_memory_high_watermark 0.4
|
||||
```
|
||||
|
||||
## 🔐 Security
|
||||
|
||||
### Production Checklist
|
||||
- [ ] Change default password in `.env`
|
||||
- [ ] Enable SSL/TLS for connections
|
||||
- [ ] Restrict management UI access
|
||||
- [ ] Set up user permissions
|
||||
- [ ] Configure firewall rules
|
||||
- [ ] Enable audit logging
|
||||
|
||||
### Create Production User
|
||||
```bash
|
||||
# Create new user
|
||||
docker exec didi-queue rabbitmqctl add_user analysis_service SECURE_PASSWORD
|
||||
|
||||
# Set permissions
|
||||
docker exec didi-queue rabbitmqctl set_permissions -p / analysis_service ".*" ".*" ".*"
|
||||
|
||||
# Set user tags
|
||||
docker exec didi-queue rabbitmqctl set_user_tags analysis_service monitoring
|
||||
```
|
||||
|
||||
## 📊 Monitoring
|
||||
|
||||
### Key Metrics
|
||||
- Queue depth (messages waiting)
|
||||
- Message rates (publish/consume)
|
||||
- Connection count
|
||||
- Memory usage
|
||||
- Disk usage
|
||||
|
||||
### Prometheus Metrics
|
||||
RabbitMQ exposes metrics at: `http://localhost:15692/metrics`
|
||||
|
||||
## 🔄 Backup & Recovery
|
||||
|
||||
### Backup
|
||||
```bash
|
||||
# Export definitions
|
||||
docker exec didi-queue rabbitmqctl export_definitions /var/lib/rabbitmq/backup.json
|
||||
docker cp didi-queue:/var/lib/rabbitmq/backup.json ./backup.json
|
||||
```
|
||||
|
||||
### Restore
|
||||
```bash
|
||||
# Import definitions
|
||||
docker cp ./backup.json didi-queue:/var/lib/rabbitmq/backup.json
|
||||
docker exec didi-queue rabbitmqctl import_definitions /var/lib/rabbitmq/backup.json
|
||||
```
|
||||
|
||||
## 📚 Related Documentation
|
||||
- [Data Layer README](../README.md)
|
||||
- [RabbitMQ Documentation](https://www.rabbitmq.com/documentation.html)
|
||||
- [AMQP Protocol](https://www.amqp.org/)
|
||||
|
||||
---
|
||||
*Part of the DIDI Backend Data Layer*
|
||||
72
backend/services/data-layer/didiQueue/init-queues.sh
Normal file
72
backend/services/data-layer/didiQueue/init-queues.sh
Normal file
|
|
@ -0,0 +1,72 @@
|
|||
#!/bin/bash
|
||||
# Init script for RabbitMQ queue setup
|
||||
# This runs automatically when the container starts
|
||||
|
||||
set -e
|
||||
|
||||
# Wait for RabbitMQ to be ready
|
||||
until rabbitmqctl status > /dev/null 2>&1; do
|
||||
echo "Waiting for RabbitMQ to start..."
|
||||
sleep 2
|
||||
done
|
||||
|
||||
echo "RabbitMQ is ready. Setting up queues..."
|
||||
|
||||
# Since we're merging all analysis services into one, we only need ONE queue
|
||||
# Create the unified analysis queue
|
||||
rabbitmqctl eval '
|
||||
rabbit_exchange:declare(
|
||||
{resource, <<"/">>, exchange, <<"analysis">>},
|
||||
topic,
|
||||
true,
|
||||
false,
|
||||
false,
|
||||
[]
|
||||
).' || true
|
||||
|
||||
# Declare the unified analysis queue with proper settings
|
||||
rabbitmqctl eval '
|
||||
rabbit_amqqueue:declare(
|
||||
{resource, <<"/">>, queue, <<"analysis_queue">>},
|
||||
true, % Durable
|
||||
false, % Not exclusive
|
||||
false, % Not auto-delete
|
||||
[], % No arguments
|
||||
none % No owner
|
||||
).' || true
|
||||
|
||||
# Bind the queue to the exchange
|
||||
rabbitmqctl eval '
|
||||
rabbit_binding:add_explicit(
|
||||
{binding,
|
||||
{resource, <<"/">>, exchange, <<"analysis">>},
|
||||
<<"analysis.*">>,
|
||||
{resource, <<"/">>, queue, <<"analysis_queue">>},
|
||||
[]
|
||||
}
|
||||
).' || true
|
||||
|
||||
# Optional: Create a dead letter queue for failed messages
|
||||
rabbitmqctl eval '
|
||||
rabbit_amqqueue:declare(
|
||||
{resource, <<"/">>, queue, <<"analysis_dlq">>},
|
||||
true, % Durable
|
||||
false, % Not exclusive
|
||||
false, % Not auto-delete
|
||||
[], % No arguments
|
||||
none % No owner
|
||||
).' || true
|
||||
|
||||
echo "✅ Queue setup complete!"
|
||||
echo "Created queues:"
|
||||
echo " - analysis_queue (main queue for all analysis types)"
|
||||
echo " - analysis_dlq (dead letter queue for failed messages)"
|
||||
|
||||
# Set queue policies for message TTL and retry
|
||||
rabbitmqctl set_policy analysis-retry \
|
||||
"analysis_queue" \
|
||||
'{"message-ttl":86400000, "dead-letter-exchange":"", "dead-letter-routing-key":"analysis_dlq"}' \
|
||||
--priority 0 \
|
||||
--apply-to queues || true
|
||||
|
||||
echo "✅ Queue policies configured!"
|
||||
46
backend/services/data-layer/didiStorage/.env.example
Normal file
46
backend/services/data-layer/didiStorage/.env.example
Normal file
|
|
@ -0,0 +1,46 @@
|
|||
# ============================================================================
|
||||
# didiStorage Environment Configuration
|
||||
# ============================================================================
|
||||
# Copy this file to .env and update with your values
|
||||
|
||||
# Docker Compose Project Name (groups containers in Docker Desktop)
|
||||
COMPOSE_PROJECT_NAME=didibackend_datalayer
|
||||
|
||||
# MinIO Credentials (CHANGE THESE!)
|
||||
MINIO_ROOT_USER=YOUR_ADMIN_USER_HERE
|
||||
MINIO_ROOT_PASSWORD=YOUR_SECURE_PASSWORD_HERE
|
||||
|
||||
# Ports (using 9002/9003 to avoid conflicts with existing MinIO)
|
||||
MINIO_API_PORT=9002 # API endpoint
|
||||
MINIO_CONSOLE_PORT=9003 # Web console
|
||||
|
||||
# Region
|
||||
MINIO_REGION=us-east-1
|
||||
|
||||
# Console Access
|
||||
MINIO_BROWSER=on # Set to 'off' to disable web console
|
||||
|
||||
# Resource Limits
|
||||
MINIO_MEMORY_LIMIT=1G
|
||||
MINIO_MEMORY_RESERVATION=512M
|
||||
|
||||
# Storage Settings
|
||||
MINIO_STORAGE_CLASS_STANDARD=EC:2
|
||||
MINIO_STORAGE_CLASS_RRS=EC:1
|
||||
|
||||
# Timezone
|
||||
TZ=UTC
|
||||
|
||||
# Bucket Lifecycle (days)
|
||||
TEXT_FILES_EXPIRY=30
|
||||
AUDIO_FILES_EXPIRY=30
|
||||
VIDEO_FILES_EXPIRY=30
|
||||
IMAGE_FILES_EXPIRY=60
|
||||
DOCUMENT_FILES_EXPIRY=90
|
||||
|
||||
# Versioning
|
||||
ENABLE_VERSIONING=true
|
||||
|
||||
# Encryption (optional)
|
||||
MINIO_KMS_SECRET_KEY=
|
||||
MINIO_KMS_AUTO_ENCRYPTION=off
|
||||
27
backend/services/data-layer/didiStorage/.gitignore
vendored
Normal file
27
backend/services/data-layer/didiStorage/.gitignore
vendored
Normal file
|
|
@ -0,0 +1,27 @@
|
|||
# Environment variables
|
||||
.env
|
||||
.env.local
|
||||
|
||||
# Data directory
|
||||
data/
|
||||
|
||||
# Config directory
|
||||
config/
|
||||
|
||||
# Logs
|
||||
*.log
|
||||
|
||||
# OS files
|
||||
.DS_Store
|
||||
Thumbs.db
|
||||
|
||||
# IDE files
|
||||
.idea/
|
||||
.vscode/
|
||||
*.swp
|
||||
*.swo
|
||||
|
||||
# Backup files
|
||||
*.bak
|
||||
*.backup
|
||||
*.old
|
||||
352
backend/services/data-layer/didiStorage/INDEX.md
Normal file
352
backend/services/data-layer/didiStorage/INDEX.md
Normal file
|
|
@ -0,0 +1,352 @@
|
|||
# didiStorage - Index
|
||||
|
||||
Stocare fisiere media pentru platforma DIDI. Container MinIO (S3-compatibil) local pe masina de deployment. Nu contine cod custom -- doar configurare si script de initializare.
|
||||
|
||||
## Productie activa (LOCAL)
|
||||
|
||||
DIDI scrie **LOCAL** pe containerul MinIO `staging-dataLayer-minio:9000` (pe `didi-network`), intr-un singur bucket `didi-prod`. Decizie: stabilitate + zero dependinte externe. Clusterul MinIO managed extern (4 noduri, erasure coding EC:2, HAProxy + keepalived VRRP) ramane configurat ca **fallback de urgenta pentru HA**, activabil cu `minio-switch.sh cluster`, dar nu este folosit operational acum.
|
||||
|
||||
| Mediu | Endpoint | Bucket | Credentiale |
|
||||
|-------|----------|--------|-------------|
|
||||
| **Productie (LOCAL — activ)** | `staging-dataLayer-minio:9000` (expus `0.0.0.0:9000`) | `didi-prod` (single bucket) | `didi-prod` / `627074a6...` |
|
||||
| Fallback HA (cluster — inactiv) | `<minio-host>:9000` (VIP `10.11.10.128`) | `didi-prod` | didi-prod / `.cluster-credentials.env` |
|
||||
|
||||
Switch local/cluster: `agent-v3/scripts/minio-switch.sh local|cluster` (modifica `.env` + reseteaza containerele). Detalii migrare: `agent-v3/MIGRATION_MINIO.md`.
|
||||
|
||||
Restrictia cheie mostenita din arhitectura: credentialele `didi-prod` au `s3:*` **doar pe bucket-ul propriu** — nu se creeaza bucket-uri noi. De aici single-bucket architecture (vezi sectiunea urmatoare), pastrata si local.
|
||||
|
||||
## Container local (activ)
|
||||
|
||||
**Imagine**: minio/minio:RELEASE.2024-08-29T01-40-52Z
|
||||
**Container**: staging-dataLayer-minio
|
||||
**Port API**: 9000 (Docker network + expus pe host `0.0.0.0:9000`)
|
||||
**Port Console**: 9001 (expus pe host `0.0.0.0:9001`)
|
||||
**Bucket DIDI**: `didi-prod` (singurul bucket)
|
||||
**Credentiale DIDI**: `didi-prod` / `627074a6...` (din `.env`)
|
||||
**Volume**: didi-staging-minio-data:/data
|
||||
|
||||
---
|
||||
|
||||
## Ce stocheaza
|
||||
|
||||
1. **Fisiere uploadate de utilizatori** -- imagini, audio, video, documente
|
||||
2. **Fisiere procesate de agent-v3** -- video downloadat, cadre extrase, transcrieri
|
||||
3. **Artefacte pipeline** -- rezultate analiza (cu versionare)
|
||||
4. **Bucket-uri per utilizator** -- fisiere organizate pe foldere tipizate
|
||||
|
||||
---
|
||||
|
||||
## Single-bucket architecture (refactor 2026-04-25)
|
||||
|
||||
Productia foloseste un singur bucket `didi-prod`, iar separarea logica se face prin **prefix-uri**, nu bucket-uri distincte. Numele de prefix-uri sistem sunt identice cu numele bucket-urilor vechi pentru ca URL-urile vechi sa ramana interpretabile.
|
||||
|
||||
```
|
||||
didi-prod/
|
||||
uploads/ -- upload-uri generale / fallback (legacy "uploads")
|
||||
image-files/ -- imagini (legacy bucket "image-files")
|
||||
audio-files/ -- audio (legacy bucket "audio-files")
|
||||
video-files/ -- video (legacy bucket "video-files")
|
||||
text-files/ -- text (legacy bucket "text-files")
|
||||
document-files/ -- PDF, Office (legacy bucket "document-files")
|
||||
pipeline-artifacts/ -- rezultate analiza (legacy bucket "pipeline-artifacts")
|
||||
users/{userId}/ -- namespace per utilizator (inlocuieste bucket-urile "user-{id}")
|
||||
images/
|
||||
videos/
|
||||
videos/frames/ -- cadre extrase din video (scrise de media-preprocess worker, citite de techniques + ai-tampered)
|
||||
audio-files/
|
||||
text-files/
|
||||
```
|
||||
|
||||
### De ce single-bucket
|
||||
- Arhitectura a fost proiectata pentru credentiale cu `s3:*` limitat la un singur bucket pre-creat (`didi-prod`), fara `s3:CreateBucket` — pastrata identic si pe MinIO local pentru portabilitate cluster.
|
||||
- Quota tracking simplificat: nu mai depindem de tag-uri pe bucket; storage-ul utilizatorilor este urmarit in PG (`bos_sysadmin.internet_user.storage_used_bytes` + `storage_limit_bytes`, vezi migration `010_add_user_storage_quota.sql` din didiFramework).
|
||||
- Separare logica prin prefix-uri, nu prin bucket-uri distincte — acelasi layout functioneaza local si pe cluster fara modificari de cod.
|
||||
|
||||
### Backward compat
|
||||
Caller-ii care paseaza bucket-uri vechi (`user-3`, `image-files`) sunt rezolvati automat la canonic `didi-prod/<full-key>`:
|
||||
|
||||
| URL primit | Bucket rezolvat | Key rezolvat |
|
||||
|---|---|---|
|
||||
| `user-3/images/abc.jpg` | `didi-prod` | `users/3/images/abc.jpg` |
|
||||
| `image-files/foo.jpg` | `didi-prod` | `image-files/foo.jpg` |
|
||||
| `didi-prod/users/3/images/abc.jpg` | `didi-prod` | `users/3/images/abc.jpg` (passthrough) |
|
||||
|
||||
Implementat in:
|
||||
- `didiFramework/src/config/minio.ts` -- `resolveBucketRequest(bucket, key)` + constanta `BUCKET = process.env.MINIO_BUCKET || 'didi-prod'`.
|
||||
- `agent-v3/src/shared/media/media-service.ts` -- `proxyFile()` (ownership check accepta atat `user-{N}` cat si prefix `users/{N}/`) + `uploadFile()` foloseste prefix `users/{userId}/{folder}/`.
|
||||
|
||||
### Lifecycle si versionare
|
||||
- Politicile de lifecycle (90 zile pentru transient, retentie permanenta pentru `pipeline-artifacts/`, `backups/`) se aplica pe bucket-ul `didi-prod` prin prefix; pe MinIO local pot fi setate cu `mc ilm` (optional).
|
||||
- Versionarea pentru `pipeline-artifacts/` si `backups/` este pastrata la nivel de bucket.
|
||||
- Daca se comuta pe cluster (`minio-switch.sh cluster`), lifecycle-ul devine responsabilitatea operatorilor cluster-ului (nu detinem bucket-ul acolo).
|
||||
|
||||
### Mod legacy (multi-bucket)
|
||||
Pentru referinta — modul vechi avea 8 bucket-uri sistem (`uploads`, `text-files`, `image-files`, `audio-files`, `video-files`, `document-files`, `pipeline-artifacts`, `backups`) plus bucket-uri dinamice `user-{id}` create la primul login. Acest layout a fost inlocuit de single-bucket `didi-prod` cu prefix-uri.
|
||||
|
||||
---
|
||||
|
||||
## Limite dimensiune fisiere
|
||||
|
||||
| Tip | Limita | MIME types |
|
||||
|-----|--------|------------|
|
||||
| Imagini | 20 MB | image/jpeg, image/png, image/gif, image/webp, image/bmp, image/svg+xml |
|
||||
| Audio | 100 MB | audio/mpeg, audio/wav, audio/ogg, audio/webm, audio/flac, audio/mp4, audio/x-m4a |
|
||||
| Video | 500 MB | video/mp4, video/webm, video/quicktime, video/x-msvideo, video/x-matroska |
|
||||
| Text | 10 MB | text/plain, text/html, text/markdown, text/csv |
|
||||
| Documente | 50 MB | application/pdf, application/msword, application/vnd.openxmlformats-* |
|
||||
|
||||
Rutarea automata: fisierul e pus in bucket-ul corespunzator MIME type-ului.
|
||||
|
||||
---
|
||||
|
||||
## Cine scrie in MinIO
|
||||
|
||||
| Serviciu | Ce scrie | Locatie (bucket `didi-prod` local) | Logica in fisier |
|
||||
|----------|----------|--------|------------------|
|
||||
| didiFramework (uploads) | Fisiere uploadate via API | `didi-prod/users/{id}/{mimeFolder}/...` (fallback `didi-prod/{mimeBucket}/`) | didiFramework/src/routes/uploads.ts |
|
||||
| didiFramework (auth) | (Nu mai creeaza bucket) Logging registration; quota in PG | -- | didiFramework/src/routes/auth.ts + migration 010_add_user_storage_quota.sql |
|
||||
| agent-v3 (media upload) | Fisiere uploadate direct sau via multer | `didi-prod/users/{userId}/...` (fallback `didi-prod/uploads/{userId}/`) | agent-v3/src/api/routes.ts -> media-service.ts |
|
||||
| agent-v3 (media-preprocess worker) | Video downloadat, cadre extrase ffmpeg, audio extras pentru transcript | `didi-prod/users/{userId}/videos/frames/...` (cand frame-urile sunt persistate); altfel /tmp efemer | agent-v3/src/queue/workers/media-preprocess-worker.ts + shared/media/video-processor.ts |
|
||||
| agent-v3 (pipeline) | Imagini downloadate din URL-uri | `didi-prod/users/{userId}/images/...` (fallback `didi-prod/uploads/{userId}/`) | agent-v3/src/api/pipeline-routes.ts |
|
||||
|
||||
Nota media-preprocess: workerul ruleaza inaintea componentelor de analiza (techniques, ai-tampered, claims) si centralizeaza descarcarea + ffmpeg + transcript + 2× vision. Frame-urile extrase sunt apoi consumate de workerii de techniques / ai-tampered fara duplicare. Cand persistarea frame-urilor in MinIO este activa, prefixul folosit este `users/{userId}/videos/frames/` (cf. `USER_BUCKET_FOLDERS.FRAMES`).
|
||||
|
||||
## Cine citeste din MinIO
|
||||
|
||||
| Serviciu | Ce citeste | Cum |
|
||||
|----------|-----------|-----|
|
||||
| agent-v3 (media proxy) | Servire fisiere catre client | GET /api/v3/media/file/:bucket/:objectKey (proxy cu range support) |
|
||||
| agent-v3 (vision) | Imagini pentru modele LLM locale | URL intern direct catre MinIO local (host-ul de deployment :9000/bucket/key) |
|
||||
| agent-v3 (transcription) | Audio/video pentru transcriere | URL presemnat sau intern |
|
||||
| didiFramework (uploads) | Info fisier + URL presemnat | GET /api/uploads/:fileId |
|
||||
| Clienti externi | Download fisiere | URL presemnat (1 ora) sau proxy agent-v3 |
|
||||
|
||||
---
|
||||
|
||||
## URL-uri si acces
|
||||
|
||||
### URL public (prin proxy agent-v3)
|
||||
```
|
||||
https://didi365.eu/api/v3/media/file/{bucket}/{objectKey}
|
||||
Exemplu (canonic): https://didi365.eu/api/v3/media/file/didi-prod/users/3/audio-files/1771883851173-audio.mp3
|
||||
Exemplu (legacy): https://didi365.eu/api/v3/media/file/user-3/audio-files/1771883851173-audio.mp3 (rezolvat la didi-prod)
|
||||
```
|
||||
Suporta HTTP Range requests (streaming audio/video). Proxy-ul rezolva atat URL-uri legacy (`user-{id}/...`, `image-files/...`) cat si forme canonice (`didi-prod/users/{id}/...`).
|
||||
|
||||
### URL presemnat (direct MinIO local)
|
||||
```
|
||||
http://<host-deployment>:9000/didi-prod/{objectKey}?X-Amz-Algorithm=...&X-Amz-Signature=...
|
||||
```
|
||||
Valabilitate: 1 ora (GET), 1 ora (PUT upload). Path-style obligatoriu (`forcePathStyle=true`).
|
||||
|
||||
### URL intern (pentru modele LLM locale)
|
||||
```
|
||||
http://<host-deployment>:9000/didi-prod/{objectKey}
|
||||
```
|
||||
Modelele locale (Qwen Vision, pe masinile GPU) nu pot accesa `didi365.eu`, asa ca URL-urile publice sunt convertite la URL-uri MinIO interne catre containerul local `staging-dataLayer-minio` (expus pe host-ul de deployment `:9000`). Logica: `agent-v3/src/shared/media/vision.ts` (`INTERNAL_MEDIA_BASE`).
|
||||
|
||||
---
|
||||
|
||||
## Integrare cu serviciile
|
||||
|
||||
### didiFramework -- configurare MinIO principala
|
||||
|
||||
Fisier: `didiFramework/src/config/minio.ts` (refactor 2026-04-25 pentru single-bucket)
|
||||
|
||||
Constante:
|
||||
- `BUCKET` -- bucket fix din `MINIO_BUCKET` env (default `didi-prod`).
|
||||
- `BUCKETS` -- prefix-uri sistem (`uploads`, `image-files`, `audio-files`, `video-files`, `text-files`, `document-files`, `pipeline-artifacts`).
|
||||
- `USER_BUCKET_FOLDERS` -- foldere per utilizator (`images`, `videos`, `audio-files`, `text-files`, `videos/frames`).
|
||||
- `MIME_TO_BUCKET` -- routing MIME -> prefix sistem.
|
||||
|
||||
Exporta:
|
||||
- `getMinioClient()` -- client singleton.
|
||||
- `checkMinioHealth()` -- health check via `listBuckets()`.
|
||||
- `resolveBucketRequest(bucket, key)` -- traduce input legacy (`user-3`, `image-files`) la `(BUCKET, fullKey)` canonic.
|
||||
- `userObjectKey(userId, folder, filename)` -- construieste `users/{userId}/{folder}/{filename}`.
|
||||
- `ensureBucket(name)` -- **no-op in single-bucket mode** (logging only). Pentru bucket-uri sistem legacy / `user-{N}` returneaza fara eroare.
|
||||
- `uploadBuffer(bucket, name, buffer, mimeType, metadata)` -- upload (rezolva bucket-ul intern).
|
||||
- `deleteObject(bucket, name)` / `getObjectInfo(bucket, name)` / `listObjects(bucket, prefix, maxKeys)`.
|
||||
- `getPresignedUrl(bucket, name, expiry)` -- URL download (default 1 ora).
|
||||
- `getPresignedPutUrl(bucket, name, expiry)` -- URL upload (default 1 ora).
|
||||
- `getDirectUrl(bucket, name)` -- URL direct fara semnatura.
|
||||
- `createUserBucket(userId, email, planId, planName, storageLimitGb)` -- **lazy in single-bucket mode**: namespace-ul `users/{id}/` "exista" doar cand are obiecte; functia logheaza si scrie quota in PG.
|
||||
- `getUserBucketUsage(userId)` -- listObjects pe `users/{id}/`, returneaza bytes + count.
|
||||
- `getUserBucketMetadata(userId)` -- thin shim (in single-bucket mode metadata e in PG, nu in tag-uri).
|
||||
- `updateUserBucketMetadata(userId, planId, planName, storageLimitGb)` -- no-op pentru bucket tags; caller-ul scrie in PG.
|
||||
|
||||
Quota tracking: migrarea `sql/migrations/010_add_user_storage_quota.sql` adauga coloanele `storage_used_bytes` si `storage_limit_bytes` la `bos_sysadmin.internet_user`. Tag-urile vechi (`storage-limit-gb` etc.) nu mai sunt folosite.
|
||||
|
||||
### agent-v3 -- MediaService
|
||||
|
||||
Fisier: `agent-v3/src/shared/media/media-service.ts`
|
||||
|
||||
Exporta:
|
||||
- uploadFile(userId, buffer, filename, contentType) -- upload cu rutare automata bucket
|
||||
- getPresignedUploadUrl(userId, filename, contentType) -- URL presemnat PUT (1 ora)
|
||||
- getPresignedDownloadUrl(objectKey, bucket) -- URL presemnat GET (1 ora)
|
||||
- proxyFile(bucket, objectKey, ownerUserId, rangeHeader) -- proxy cu verificare proprietar + range support
|
||||
- ensureBucket(name) -- creeaza daca nu exista
|
||||
|
||||
Flow upload in agent-v3:
|
||||
1. Rezolva bucket-ul utilizatorului din didiFramework (keycloak_id -> bucket + folder)
|
||||
2. Fallback la uploads/{userId} daca framework indisponibil
|
||||
3. Returneaza: download_url, public_url, object_key, bucket, filename, size
|
||||
|
||||
### Python (shared layer)
|
||||
|
||||
Fisier: `shared/minio_presigner.py`
|
||||
- convert_media_url_for_llm(url) -- converteste URL-uri interne MinIO in URL-uri presemnate pentru LLM-uri externe
|
||||
- parse_minio_url(url) -- parseaza formate: minio://bucket/path, /bucket/path, http://minio:9000/bucket/path
|
||||
|
||||
Fisier: `shared/url_config.py`
|
||||
- convert_minio_to_public_url(url) -- converteste URL-uri interne in URL-uri publice HTTP
|
||||
|
||||
---
|
||||
|
||||
## Fluxul de upload (utilizator)
|
||||
|
||||
```
|
||||
Utilizator uploadeaza fisier
|
||||
|
|
||||
v
|
||||
POST /api/v3/media/upload (agent-v3, multer, max 50MB)
|
||||
|
|
||||
v
|
||||
MediaService.uploadFile()
|
||||
|-- Cere didiFramework /internal/get-bucket-info -> { bucketName: 'didi-prod', folder: 'users/{id}/{mimeFolder}' }
|
||||
|-- Fallback: bucket = MINIO_BUCKET (didi-prod), prefix = uploads/{userId}/
|
||||
|
|
||||
v
|
||||
MinIO local: putObject('didi-prod', 'users/{id}/{mimeFolder}/{filename}', buffer)
|
||||
|
|
||||
v
|
||||
Genereaza URL public: https://didi365.eu/api/v3/media/file/didi-prod/users/{id}/{mimeFolder}/{filename}
|
||||
|
|
||||
v
|
||||
Returneaza: { download_url, public_url, object_key, bucket, size, content_type }
|
||||
```
|
||||
|
||||
## Fluxul de inregistrare utilizator (single-bucket)
|
||||
|
||||
```
|
||||
Utilizator face login prima data
|
||||
|
|
||||
v
|
||||
GET /api/auth/me (didiFramework)
|
||||
|
|
||||
v
|
||||
Utilizator nu exista in PG -> auto-inregistrare
|
||||
|
|
||||
v
|
||||
createUserBucket(internetUserId, email, planId='1', planName='Free', storageLimitGb=1)
|
||||
|-- (single-bucket mode) -- nu apeleaza MinIO makeBucket
|
||||
|-- Logheaza initializarea
|
||||
|-- Quota persistata in PG: bos_sysadmin.internet_user.storage_limit_bytes
|
||||
|
|
||||
v
|
||||
Namespace logic users/{id}/ exista de cum primul fisier e uploadat.
|
||||
```
|
||||
|
||||
## Fluxul media-preprocess (async, video/audio/imagine)
|
||||
|
||||
```
|
||||
Job analiza pe URL/upload media
|
||||
|
|
||||
v
|
||||
Dispatcher RabbitMQ -> media-preprocess queue (un singur worker per sesiune)
|
||||
|
|
||||
v
|
||||
MediaPreprocessWorker:
|
||||
|-- yt-dlp / fetch URL -> /tmp/video_{sessionId}_{ts}/source.mp4
|
||||
|-- ffmpeg extrage frame-uri uniform (max 10) -> /tmp/.../frame_%03d.jpg
|
||||
|-- ffmpeg extrage audio -> /tmp/.../audio.mp3
|
||||
|-- transcript via Whisper (M17 -> Groq -> OpenAI)
|
||||
|-- 2× vision call pe ACELEASI frame-uri (misinformation + ai_detection)
|
||||
|-- (optional) upload frame-uri persistente -> didi-prod/users/{id}/videos/frames/
|
||||
|
|
||||
v
|
||||
Cache rezultatele in Redis (TTL 1h):
|
||||
agent:media:{sessionId}:transcript
|
||||
agent:media:{sessionId}:vision:misinformation
|
||||
agent:media:{sessionId}:vision:ai_detection
|
||||
agent:media:{sessionId}:merged_text
|
||||
agent:media:{sessionId}:ready = "1"
|
||||
|
|
||||
v
|
||||
Dispatch task-uri pentru techniques + ai-tampered + claims (citesc din Redis, nu reproceseaza media)
|
||||
```
|
||||
|
||||
Beneficiu: 1 download + 1 ffmpeg + 1 transcript + 2 vision in loc de 3× pe fiecare component.
|
||||
|
||||
---
|
||||
|
||||
## Fisiere in directorul didiStorage
|
||||
|
||||
```
|
||||
init-buckets.sh -- Script initializare: creeaza 8 bucket-uri + lifecycle + versionare (136 linii)
|
||||
.env.example -- Template variabile de mediu
|
||||
README.md -- Documentatie (253 linii)
|
||||
.gitignore -- Exclude .env, data/, config/
|
||||
```
|
||||
|
||||
Zero cod custom. Bucket-urile si politicile sunt create de init-buckets.sh la prima pornire.
|
||||
|
||||
---
|
||||
|
||||
## Configurare Docker
|
||||
|
||||
```yaml
|
||||
# din data-layer/docker-compose.yml
|
||||
staging-dataLayer-minio:
|
||||
image: minio/minio:RELEASE.2024-08-29T01-40-52Z
|
||||
command: server /data --console-address ":9001"
|
||||
environment:
|
||||
MINIO_ROOT_USER: minioadmin
|
||||
MINIO_ROOT_PASSWORD: minio123
|
||||
MINIO_REGION_NAME: us-east-1
|
||||
MINIO_BROWSER: "on"
|
||||
ports:
|
||||
- "9000:9000" # API (expus pe host 0.0.0.0:9000)
|
||||
- "9001:9001" # Console (expus pe host 0.0.0.0:9001)
|
||||
volumes:
|
||||
- didi-staging-minio-data:/data
|
||||
```
|
||||
|
||||
Nota: Nu exista container `minio-init` in docker-compose.yml. Scriptul `init-buckets.sh` trebuie rulat manual dupa prima pornire a MinIO.
|
||||
|
||||
---
|
||||
|
||||
## Variabile de mediu (conectare din alte servicii)
|
||||
|
||||
Setarile actuale (productie LOCALA, valori confirmate din containerul `didi-agent-v3`):
|
||||
|
||||
| Variabila | Valoare productie | Note |
|
||||
|-----------|-------------------|------|
|
||||
| MINIO_ENDPOINT | `staging-dataLayer-minio` | container local pe `didi-network`, path-style obligatoriu |
|
||||
| MINIO_PORT | `9000` | expus si pe host (`0.0.0.0:9000`) |
|
||||
| MINIO_USE_SSL | `false` | HTTP intern |
|
||||
| MINIO_BUCKET | `didi-prod` | bucket fix, single-bucket arch (singurul bucket din instanta) |
|
||||
| MINIO_ACCESS_KEY | `didi-prod` | full s3:* pe `didi-prod` |
|
||||
| MINIO_SECRET_KEY | (in `.env`) | `627074a6...` |
|
||||
|
||||
Switch rapid local <-> cluster: `backend/services/orchestration-layer/agent-v3/scripts/minio-switch.sh local|cluster` (citeste credentiale din `.cluster-credentials.env`, modifica `.env`-urile pentru agent-v3 + didiFramework, restart containere). Status curent: `local`.
|
||||
|
||||
Fallback HA (cluster extern — inactiv, doar dupa `minio-switch.sh cluster`):
|
||||
|
||||
| Serviciu | MINIO_ENDPOINT | MINIO_PORT | Credentiale |
|
||||
|----------|---------------|------------|-------------|
|
||||
| didiFramework | <minio-host> (VIP 10.11.10.128) | 9000 | didi-prod / `.cluster-credentials.env` |
|
||||
| agent-v3 | <minio-host>:9000 | (inclus in endpoint) | didi-prod / `.cluster-credentials.env` |
|
||||
| Python shared | <minio-host>:9000 | (inclus in endpoint) | didi-prod / `.cluster-credentials.env` |
|
||||
|
||||
---
|
||||
|
||||
## Ce NU face
|
||||
|
||||
- Containerul local nu are cod custom (doar MinIO standard + script init); productia DIDI ruleaza pe acest container local, iar cluster-ul CAI managed ramane fallback HA inactiv.
|
||||
- Local este instanta singulara (fara replicare). Fallback-ul cluster are 4 noduri + erasure coding EC:2 (toleranta la 2 noduri pierdute), disponibil doar dupa `minio-switch.sh cluster`.
|
||||
- Nu are encriptie at-rest dedicata pe DIDI.
|
||||
- TLS intern: HTTP (fara TLS pe MinIO local).
|
||||
- Nu enforce-uieste quota la nivel MinIO; quota utilizator (`storage_used_bytes` / `storage_limit_bytes`) este urmarita in PG (`bos_sysadmin.internet_user`) si verificata de didiFramework la upload.
|
||||
- Nu mai face create-bucket per utilizator (single-bucket: namespace logic prin prefix).
|
||||
253
backend/services/data-layer/didiStorage/README.md
Normal file
253
backend/services/data-layer/didiStorage/README.md
Normal file
|
|
@ -0,0 +1,253 @@
|
|||
# DIDI Storage Service 📦
|
||||
|
||||
## Super Simple Start Guide 🚀
|
||||
|
||||
### One Command - That's It!
|
||||
```bash
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
**DONE!** Everything is automatically configured! 🎉
|
||||
|
||||
## What Just Happened? 🤔
|
||||
|
||||
When you ran that one command:
|
||||
1. MinIO storage server started
|
||||
2. A helper container automatically:
|
||||
- Created 7 buckets for different file types
|
||||
- Set up auto-deletion for old files
|
||||
- Configured versioning for important data
|
||||
- Created access policies
|
||||
- Then exited (this is normal!)
|
||||
3. Storage is now ready to use!
|
||||
|
||||
## Check If It's Working ✅
|
||||
|
||||
```bash
|
||||
docker ps
|
||||
|
||||
# You should see:
|
||||
# didi-storage (healthy) ← This is your storage server
|
||||
```
|
||||
|
||||
**Note**: You might also see `didi-storage-init (Exited)` - that's the helper that set everything up. It's supposed to exit!
|
||||
|
||||
## Access the Web Console 🖥️
|
||||
|
||||
1. Open your browser
|
||||
2. Go to: **http://localhost:9003**
|
||||
3. Login:
|
||||
- Username: `minioadmin`
|
||||
- Password: `minio123`
|
||||
4. You'll see all your buckets ready!
|
||||
|
||||
## Connection Info for Your Apps 📡
|
||||
|
||||
```python
|
||||
# Python example
|
||||
from minio import Minio
|
||||
|
||||
client = Minio(
|
||||
"localhost:9002", # API port
|
||||
access_key="minioadmin",
|
||||
secret_key="minio123",
|
||||
secure=False
|
||||
)
|
||||
```
|
||||
|
||||
## The 7 Auto-Created Buckets 🗂️
|
||||
|
||||
| Bucket Name | What Goes Here | Auto-Delete After |
|
||||
|------------|----------------|-------------------|
|
||||
| `text-files` | Text documents, CSVs | 30 days |
|
||||
| `image-files` | JPG, PNG, GIF | 60 days |
|
||||
| `audio-files` | MP3, WAV, M4A | 30 days |
|
||||
| `video-files` | MP4, AVI, MOV | 30 days |
|
||||
| `document-files` | PDF, Word, Excel | Never |
|
||||
| `pipeline-artifacts` | Analysis results | Never (versioned) |
|
||||
| `backups` | System backups | Never (versioned) |
|
||||
|
||||
## Quick Test - Upload a File 📤
|
||||
|
||||
```bash
|
||||
# Create a test file
|
||||
echo "Hello Storage!" > test.txt
|
||||
|
||||
# Upload it (using docker)
|
||||
docker exec didi-storage sh -c "echo 'Test' > /tmp/test.txt && mc cp /tmp/test.txt local/text-files/"
|
||||
|
||||
# Check it's there
|
||||
docker exec didi-storage mc ls local/text-files/
|
||||
```
|
||||
|
||||
## Common Tasks 🛠️
|
||||
|
||||
### Start Storage
|
||||
```bash
|
||||
docker compose up -d
|
||||
# That's it! Everything auto-configures
|
||||
```
|
||||
|
||||
### Stop Storage
|
||||
```bash
|
||||
docker compose down
|
||||
# Data is preserved
|
||||
```
|
||||
|
||||
### View Logs
|
||||
```bash
|
||||
docker compose logs -f didiStorage
|
||||
```
|
||||
|
||||
### Check Storage Usage
|
||||
```bash
|
||||
docker exec didi-storage mc du local/
|
||||
```
|
||||
|
||||
### List All Files
|
||||
```bash
|
||||
docker exec didi-storage mc ls --recursive local/
|
||||
```
|
||||
|
||||
### Complete Fresh Start (WARNING: Deletes Everything!)
|
||||
```bash
|
||||
docker compose down -v
|
||||
rm -rf data/ config/
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
## What's Special About This Setup? ✨
|
||||
|
||||
### 1. **Zero Configuration**
|
||||
You don't need to:
|
||||
- Create buckets manually
|
||||
- Set up policies
|
||||
- Configure expiry rules
|
||||
- Enable versioning
|
||||
|
||||
It's ALL done automatically!
|
||||
|
||||
### 2. **Smart File Management**
|
||||
- Old files auto-delete (saves space)
|
||||
- Important files keep versions (never lose data)
|
||||
- Each service gets its own bucket
|
||||
|
||||
### 3. **Ready for Production**
|
||||
- Passwords in .env file (change them!)
|
||||
- Resource limits configured
|
||||
- Health checks included
|
||||
- Logging configured
|
||||
|
||||
## Troubleshooting 🔧
|
||||
|
||||
### "Port already in use"
|
||||
Someone else is using port 9002 or 9003. Fix:
|
||||
1. Edit `.env`
|
||||
2. Change `MINIO_API_PORT=9004`
|
||||
3. Change `MINIO_CONSOLE_PORT=9005`
|
||||
4. Run `docker compose up -d`
|
||||
|
||||
### "Can't access console"
|
||||
1. Make sure you use `http://` not `https://`
|
||||
2. Check container is running: `docker ps`
|
||||
3. Try: http://localhost:9003
|
||||
|
||||
### "Buckets not created"
|
||||
Check the init container logs:
|
||||
```bash
|
||||
docker logs didi-storage-init
|
||||
```
|
||||
It should show "Initialization Complete!"
|
||||
|
||||
### "Storage full"
|
||||
Check usage:
|
||||
```bash
|
||||
docker exec didi-storage mc du local/
|
||||
```
|
||||
Files auto-delete after their expiry time!
|
||||
|
||||
## For Your Services 🔌
|
||||
|
||||
### Python Upload Example
|
||||
```python
|
||||
from minio import Minio
|
||||
|
||||
# Connect
|
||||
client = Minio("localhost:9002",
|
||||
access_key="minioadmin",
|
||||
secret_key="minio123",
|
||||
secure=False)
|
||||
|
||||
# Upload image
|
||||
client.fput_object("image-files", "photo.jpg", "/path/to/photo.jpg")
|
||||
|
||||
# Upload with metadata
|
||||
client.fput_object(
|
||||
"document-files",
|
||||
"report.pdf",
|
||||
"/path/to/report.pdf",
|
||||
metadata={"pipeline": "text-analysis", "user": "john"}
|
||||
)
|
||||
```
|
||||
|
||||
### Node.js Example
|
||||
```javascript
|
||||
const Minio = require('minio')
|
||||
|
||||
const client = new Minio.Client({
|
||||
endPoint: 'localhost',
|
||||
port: 9002,
|
||||
useSSL: false,
|
||||
accessKey: 'minioadmin',
|
||||
secretKey: 'minio123'
|
||||
})
|
||||
|
||||
// Upload
|
||||
client.fPutObject('text-files', 'data.txt', '/path/to/data.txt')
|
||||
```
|
||||
|
||||
## How DIDI Platform Uses This 📊
|
||||
|
||||
```
|
||||
User uploads file → Goes to appropriate bucket
|
||||
↓
|
||||
Pipeline processes it → Results go to pipeline-artifacts
|
||||
↓
|
||||
After 30-60 days → Media files auto-delete
|
||||
↓
|
||||
Artifacts & backups → Keep forever with versions
|
||||
```
|
||||
|
||||
## Security Notes 🔒
|
||||
|
||||
**For Production:**
|
||||
1. Change `minioadmin` username in .env
|
||||
2. Change `minio123` password in .env
|
||||
3. Use HTTPS (put behind nginx)
|
||||
4. Restrict network access
|
||||
5. Enable encryption
|
||||
|
||||
## Part of the Data Layer 🏗️
|
||||
|
||||
```
|
||||
📁 data-layer/
|
||||
├── 📁 didiDatabase/ ✅ PostgreSQL
|
||||
├── 📁 didiCache/ ✅ Redis
|
||||
├── 📁 didiStorage/ ✅ MinIO (You are here!)
|
||||
└── 📁 didiQueue/ ⏳ RabbitMQ (Coming next!)
|
||||
```
|
||||
|
||||
## Summary - Why This Rocks 🎸
|
||||
|
||||
1. **One Command**: `docker compose up -d`
|
||||
2. **Zero Config**: Everything auto-setup
|
||||
3. **Smart Storage**: Auto-expiry, versioning
|
||||
4. **Production Ready**: Just change passwords
|
||||
5. **Developer Friendly**: Web console included
|
||||
|
||||
---
|
||||
**That's it! Your storage is ready! 📦**
|
||||
|
||||
*No complex setup. No manual configuration. Just works!*
|
||||
|
||||
*Version: 1.0.0 | MinIO RELEASE.2024-08-29*
|
||||
137
backend/services/data-layer/didiStorage/init-buckets.sh
Normal file
137
backend/services/data-layer/didiStorage/init-buckets.sh
Normal file
|
|
@ -0,0 +1,137 @@
|
|||
#!/bin/sh
|
||||
# ============================================================================
|
||||
# MinIO Bucket Initialization Script
|
||||
# Automatically creates all required buckets and policies on startup
|
||||
# ============================================================================
|
||||
|
||||
set -e
|
||||
|
||||
echo "============================================"
|
||||
echo "Starting MinIO Bucket Initialization"
|
||||
echo "============================================"
|
||||
|
||||
# Wait for MinIO to be ready
|
||||
echo "→ Waiting for MinIO to be ready..."
|
||||
sleep 5
|
||||
|
||||
# Configure MinIO client with credentials from environment
|
||||
echo "→ Configuring MinIO client..."
|
||||
mc alias set local http://${MINIO_HOST}:${MINIO_PORT} ${MINIO_ROOT_USER} ${MINIO_ROOT_PASSWORD}
|
||||
|
||||
# Create all required buckets
|
||||
echo "→ Creating buckets..."
|
||||
mc mb local/text-files --ignore-existing
|
||||
mc mb local/image-files --ignore-existing
|
||||
mc mb local/audio-files --ignore-existing
|
||||
mc mb local/video-files --ignore-existing
|
||||
mc mb local/document-files --ignore-existing
|
||||
mc mb local/pipeline-artifacts --ignore-existing
|
||||
mc mb local/uploads --ignore-existing
|
||||
mc mb local/backups --ignore-existing
|
||||
mc mb local/didi-prod --ignore-existing # single-bucket mode (MINIO_BUCKET=didi-prod) — media upload/download
|
||||
|
||||
echo "✓ All buckets created"
|
||||
|
||||
# Enable versioning for important buckets
|
||||
echo "→ Enabling versioning..."
|
||||
mc version enable local/pipeline-artifacts
|
||||
mc version enable local/backups
|
||||
echo "✓ Versioning enabled for pipeline-artifacts and backups"
|
||||
|
||||
# Set lifecycle policies for temporary files
|
||||
echo "→ Setting lifecycle policies..."
|
||||
cat > /tmp/lifecycle-30days.json <<EOF
|
||||
{
|
||||
"Rules": [
|
||||
{
|
||||
"ID": "expire-30days",
|
||||
"Status": "Enabled",
|
||||
"Expiration": {
|
||||
"Days": 30
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
EOF
|
||||
|
||||
cat > /tmp/lifecycle-60days.json <<EOF
|
||||
{
|
||||
"Rules": [
|
||||
{
|
||||
"ID": "expire-60days",
|
||||
"Status": "Enabled",
|
||||
"Expiration": {
|
||||
"Days": 60
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
EOF
|
||||
|
||||
# Apply lifecycle policies
|
||||
mc ilm import local/text-files < /tmp/lifecycle-30days.json
|
||||
mc ilm import local/audio-files < /tmp/lifecycle-30days.json
|
||||
mc ilm import local/video-files < /tmp/lifecycle-30days.json
|
||||
mc ilm import local/image-files < /tmp/lifecycle-60days.json
|
||||
mc ilm import local/uploads < /tmp/lifecycle-30days.json
|
||||
|
||||
echo "✓ Lifecycle policies configured"
|
||||
|
||||
# Create anonymous read policy for public buckets (optional)
|
||||
# Uncomment if you want public read access to certain buckets
|
||||
# echo "→ Setting public access policies..."
|
||||
# mc anonymous set download local/image-files
|
||||
# mc anonymous set download local/video-files
|
||||
# echo "✓ Public read access configured"
|
||||
|
||||
# Create service accounts for microservices (optional)
|
||||
# This creates restricted access for each service
|
||||
echo "→ Creating service access policies..."
|
||||
|
||||
# Policy for text analysis service
|
||||
cat > /tmp/text-service-policy.json <<EOF
|
||||
{
|
||||
"Version": "2012-10-17",
|
||||
"Statement": [
|
||||
{
|
||||
"Effect": "Allow",
|
||||
"Action": ["s3:*"],
|
||||
"Resource": ["arn:aws:s3:::text-files/*"]
|
||||
}
|
||||
]
|
||||
}
|
||||
EOF
|
||||
|
||||
# Policy for image analysis service
|
||||
cat > /tmp/image-service-policy.json <<EOF
|
||||
{
|
||||
"Version": "2012-10-17",
|
||||
"Statement": [
|
||||
{
|
||||
"Effect": "Allow",
|
||||
"Action": ["s3:*"],
|
||||
"Resource": ["arn:aws:s3:::image-files/*"]
|
||||
}
|
||||
]
|
||||
}
|
||||
EOF
|
||||
|
||||
# Apply policies (these can be used to create service accounts later)
|
||||
mc admin policy create local text-service-policy /tmp/text-service-policy.json || true
|
||||
mc admin policy create local image-service-policy /tmp/image-service-policy.json || true
|
||||
|
||||
echo "✓ Service policies created"
|
||||
|
||||
# List all buckets to confirm
|
||||
echo ""
|
||||
echo "============================================"
|
||||
echo "Initialization Complete!"
|
||||
echo "============================================"
|
||||
echo "Buckets created:"
|
||||
mc ls local/
|
||||
echo "============================================"
|
||||
|
||||
# Clean up temp files
|
||||
rm -f /tmp/lifecycle-*.json /tmp/*-policy.json
|
||||
|
||||
echo "MinIO is ready for use!"
|
||||
155
backend/services/data-layer/docker-compose.local.yml
Normal file
155
backend/services/data-layer/docker-compose.local.yml
Normal file
|
|
@ -0,0 +1,155 @@
|
|||
# =============================================================================
|
||||
# DIDI Platform - Local Data Layer + Keycloak + Kong
|
||||
# For 10.11.10.11 build (does NOT touch 10.11.10.12)
|
||||
# =============================================================================
|
||||
name: didi-infra
|
||||
|
||||
services:
|
||||
# ===========================================================================
|
||||
# PostgreSQL - Local (replaces cluster at 10.11.50.167)
|
||||
# ===========================================================================
|
||||
didi-postgres:
|
||||
image: postgres:17-alpine
|
||||
container_name: didi-postgres
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
POSTGRES_DB: DIDI
|
||||
POSTGRES_USER: bos_interface
|
||||
POSTGRES_PASSWORD: interface
|
||||
PGDATA: /var/lib/postgresql/data/pgdata
|
||||
volumes:
|
||||
- didi-postgres-data:/var/lib/postgresql/data
|
||||
- /tmp/didi_full_dump.sql:/docker-entrypoint-initdb.d/01-dump.sql:ro
|
||||
ports:
|
||||
- "5432:5432"
|
||||
networks:
|
||||
- didi-network
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "pg_isready -U bos_interface -d DIDI"]
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 5
|
||||
start_period: 30s
|
||||
|
||||
# ===========================================================================
|
||||
# Redis - Cache + Session Store
|
||||
# ===========================================================================
|
||||
didi-cache:
|
||||
image: redis:7-alpine
|
||||
container_name: didi-cache
|
||||
restart: unless-stopped
|
||||
command: redis-server --requirepass redis123
|
||||
volumes:
|
||||
- didi-redis-data:/data
|
||||
ports:
|
||||
- "6379:6379"
|
||||
networks:
|
||||
- didi-network
|
||||
healthcheck:
|
||||
test: ["CMD", "redis-cli", "-a", "redis123", "ping"]
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 3
|
||||
|
||||
# ===========================================================================
|
||||
# RabbitMQ - Message Queue
|
||||
# ===========================================================================
|
||||
staging-dataLayer-rabbitmq:
|
||||
image: rabbitmq:3.12-management-alpine
|
||||
container_name: staging-dataLayer-rabbitmq
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
RABBITMQ_DEFAULT_USER: admin
|
||||
RABBITMQ_DEFAULT_PASS: rabbitmq123
|
||||
RABBITMQ_DEFAULT_VHOST: /
|
||||
volumes:
|
||||
- didi-rabbitmq-data:/var/lib/rabbitmq
|
||||
ports:
|
||||
- "5672:5672"
|
||||
- "15672:15672"
|
||||
networks:
|
||||
- didi-network
|
||||
healthcheck:
|
||||
test: ["CMD", "rabbitmq-diagnostics", "-q", "ping"]
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 3
|
||||
start_period: 60s
|
||||
|
||||
# ===========================================================================
|
||||
# MinIO - Object Storage
|
||||
# ===========================================================================
|
||||
staging-dataLayer-minio:
|
||||
image: minio/minio:RELEASE.2024-08-29T01-40-52Z
|
||||
container_name: staging-dataLayer-minio
|
||||
restart: unless-stopped
|
||||
command: server /data --console-address ":9001"
|
||||
environment:
|
||||
MINIO_ROOT_USER: minioadmin
|
||||
MINIO_ROOT_PASSWORD: minio123
|
||||
ports:
|
||||
- "9000:9000"
|
||||
- "9001:9001"
|
||||
volumes:
|
||||
- didi-minio-data:/data
|
||||
networks:
|
||||
- didi-network
|
||||
healthcheck:
|
||||
test: ["CMD", "mc", "ready", "local"]
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 3
|
||||
start_period: 30s
|
||||
|
||||
# ===========================================================================
|
||||
# Keycloak - Local IAM
|
||||
# ===========================================================================
|
||||
didi-keycloak:
|
||||
image: quay.io/keycloak/keycloak:26.0
|
||||
container_name: didi-keycloak
|
||||
restart: unless-stopped
|
||||
command: start-dev --import-realm
|
||||
environment:
|
||||
KC_DB: postgres
|
||||
KC_DB_URL: jdbc:postgresql://didi-postgres:5432/DIDI?currentSchema=public
|
||||
KC_DB_USERNAME: bos_interface
|
||||
KC_DB_PASSWORD: interface
|
||||
KEYCLOAK_ADMIN: admin
|
||||
KEYCLOAK_ADMIN_PASSWORD: admin123
|
||||
KC_HOSTNAME_STRICT: "false"
|
||||
KC_HTTP_ENABLED: "true"
|
||||
KC_PROXY_HEADERS: xforwarded
|
||||
KC_HTTP_RELATIVE_PATH: /auth
|
||||
volumes:
|
||||
- ../gateway-auth-layer/didiKeycloak/realm-import:/opt/keycloak/data/import:ro
|
||||
- ../gateway-auth-layer/didiKeycloak/themes/didi-clients-theme:/opt/keycloak/themes/didi-clients-theme:ro
|
||||
- ../gateway-auth-layer/didiKeycloak/themes/didi-backend-theme:/opt/keycloak/themes/didi-backend-theme:ro
|
||||
- ../gateway-auth-layer/didiKeycloak/themes/didi-ai-theme:/opt/keycloak/themes/didi-ai-theme:ro
|
||||
ports:
|
||||
- "28080:8080"
|
||||
depends_on:
|
||||
didi-postgres:
|
||||
condition: service_healthy
|
||||
networks:
|
||||
didi-network:
|
||||
aliases:
|
||||
- keycloak
|
||||
# start-dev doesn't open the :9000 management health port, so probe a real
|
||||
# served endpoint instead: /auth/realms/master returns 200 when Keycloak is
|
||||
# up (valid liveness signal, no --health-enabled needed).
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "exec 3<>/dev/tcp/127.0.0.1/8080; echo -e 'GET /auth/realms/master HTTP/1.1\\r\\nHost: localhost\\r\\nConnection: close\\r\\n\\r\\n' >&3; timeout 2 cat <&3 | grep -q '200 OK'"]
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 5
|
||||
start_period: 60s
|
||||
|
||||
volumes:
|
||||
didi-postgres-data:
|
||||
didi-redis-data:
|
||||
didi-rabbitmq-data:
|
||||
didi-minio-data:
|
||||
|
||||
networks:
|
||||
didi-network:
|
||||
external: true
|
||||
168
backend/services/data-layer/docker-compose.yml
Normal file
168
backend/services/data-layer/docker-compose.yml
Normal file
|
|
@ -0,0 +1,168 @@
|
|||
# =============================================================================
|
||||
# DIDI Platform - Data Layer Services
|
||||
# =============================================================================
|
||||
# PostgreSQL, RabbitMQ, MinIO, Redis Commander, PgAdmin
|
||||
# Admin UIs bound to 127.0.0.1 (VPN access only)
|
||||
# =============================================================================
|
||||
|
||||
name: didi-data-layer
|
||||
|
||||
services:
|
||||
# ===========================================================================
|
||||
# PostgreSQL - Local Staging Database
|
||||
# ===========================================================================
|
||||
staging-dataLayer-postgres:
|
||||
image: postgres:15-alpine
|
||||
container_name: staging-dataLayer-postgres
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
POSTGRES_DB: misinformation_db
|
||||
POSTGRES_USER: postgres
|
||||
POSTGRES_PASSWORD: postgres123
|
||||
PGDATA: /var/lib/postgresql/data/pgdata
|
||||
volumes:
|
||||
- didi-staging-postgres-data:/var/lib/postgresql/data
|
||||
- didi-staging-postgres-backups:/backups
|
||||
# No ports exposed - accessible only within Docker network on port 5432
|
||||
networks:
|
||||
- didi-network
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "pg_isready -U postgres -d misinformation_db"]
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 3
|
||||
start_period: 30s
|
||||
|
||||
# ===========================================================================
|
||||
# RabbitMQ - Message Queue
|
||||
# ===========================================================================
|
||||
staging-dataLayer-rabbitmq:
|
||||
image: rabbitmq:3.12.6-management-alpine
|
||||
container_name: staging-dataLayer-rabbitmq
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
RABBITMQ_DEFAULT_USER: admin
|
||||
RABBITMQ_DEFAULT_PASS: rabbitmq123
|
||||
RABBITMQ_DEFAULT_VHOST: /
|
||||
volumes:
|
||||
- didi-staging-rabbitmq-data:/var/lib/rabbitmq
|
||||
ports:
|
||||
- "15672:15672" # RabbitMQ Management UI
|
||||
- "127.0.0.1:5672:5672" # AMQP - localhost only
|
||||
networks:
|
||||
- didi-network
|
||||
healthcheck:
|
||||
test: ["CMD", "rabbitmq-diagnostics", "-q", "ping"]
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 3
|
||||
start_period: 60s
|
||||
|
||||
# ===========================================================================
|
||||
# MinIO - Object Storage
|
||||
# ===========================================================================
|
||||
staging-dataLayer-minio:
|
||||
image: minio/minio:RELEASE.2024-08-29T01-40-52Z
|
||||
container_name: staging-dataLayer-minio
|
||||
restart: unless-stopped
|
||||
command: server /data --console-address ":9001"
|
||||
environment:
|
||||
MINIO_ROOT_USER: minioadmin
|
||||
MINIO_ROOT_PASSWORD: minio123
|
||||
MINIO_REGION_NAME: us-east-1
|
||||
MINIO_BROWSER: "on"
|
||||
ports:
|
||||
- "9001:9001" # MinIO Console UI
|
||||
- "9000:9000"
|
||||
volumes:
|
||||
- didi-staging-minio-data:/data
|
||||
networks:
|
||||
- didi-network
|
||||
healthcheck:
|
||||
test: ["CMD", "mc", "ready", "local"]
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 3
|
||||
start_period: 30s
|
||||
|
||||
# ===========================================================================
|
||||
# PgAdmin - Database Administration
|
||||
# ===========================================================================
|
||||
staging-dataLayer-pgadmin:
|
||||
image: dpage/pgadmin4:latest
|
||||
container_name: staging-dataLayer-pgadmin
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
PGADMIN_DEFAULT_EMAIL: admin@example.com
|
||||
PGADMIN_DEFAULT_PASSWORD: admin123
|
||||
PGADMIN_CONFIG_SERVER_MODE: "False"
|
||||
PGADMIN_CONFIG_MASTER_PASSWORD_REQUIRED: "False"
|
||||
volumes:
|
||||
- didi-staging-pgadmin-data:/var/lib/pgadmin
|
||||
- ./pgadmin/servers.json:/pgadmin4/servers.json:ro
|
||||
- ./pgadmin/pgpass:/pgadmin4/pgpass:ro
|
||||
ports:
|
||||
- "5050:80" # pgAdmin UI
|
||||
networks:
|
||||
- didi-network
|
||||
|
||||
# ===========================================================================
|
||||
# Redis Commander - Redis Administration
|
||||
# ===========================================================================
|
||||
staging-dataLayer-redis-commander:
|
||||
image: rediscommander/redis-commander:latest
|
||||
container_name: staging-dataLayer-redis-commander
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
REDIS_HOSTS: "production:didi-cache:6379:0:redis123"
|
||||
URL_PREFIX: /redis-commander
|
||||
# No external ports - access via Kong
|
||||
networks:
|
||||
- didi-network
|
||||
healthcheck:
|
||||
test: ["CMD", "wget", "-q", "-O", "/dev/null", "http://127.0.0.1:8081/"]
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 3
|
||||
|
||||
# ===========================================================================
|
||||
# Admin Dashboard - Infrastructure Monitoring (HTTPS)
|
||||
# ===========================================================================
|
||||
didi-admin:
|
||||
image: didi-admin:latest
|
||||
container_name: didi-admin
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
NODE_ENV: production
|
||||
REACT_APP_STAGING_MODE: "false"
|
||||
REACT_APP_HOST: "10.11.10.12"
|
||||
REACT_APP_FRAMEWORK_API_URL: "http://didi-framework:3005"
|
||||
REACT_APP_API_BASE_URL: "https://didi365.eu"
|
||||
REACT_APP_KEYCLOAK_URL: "https://didi365.eu/auth"
|
||||
REACT_APP_KEYCLOAK_REALM: "didi-admins"
|
||||
REACT_APP_KEYCLOAK_CLIENT_ID: "admin-dashboard"
|
||||
ports:
|
||||
- "3000:443"
|
||||
networks:
|
||||
- didi-network # unified — all DIDI + AI platform on this single network
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "-k", "-f", "-s", "https://127.0.0.1:443/"]
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 3
|
||||
|
||||
networks:
|
||||
didi-network:
|
||||
external: true # single shared network for all DIDI + AI platform stacks
|
||||
|
||||
volumes:
|
||||
didi-staging-postgres-data:
|
||||
external: true
|
||||
didi-staging-postgres-backups:
|
||||
name: didi-staging-postgres-backups
|
||||
didi-staging-rabbitmq-data:
|
||||
name: didi-staging-rabbitmq-data
|
||||
didi-staging-minio-data:
|
||||
external: true
|
||||
didi-staging-pgadmin-data:
|
||||
external: true
|
||||
14
backend/services/data-layer/pgadmin/servers.json
Normal file
14
backend/services/data-layer/pgadmin/servers.json
Normal file
|
|
@ -0,0 +1,14 @@
|
|||
{
|
||||
"Servers": {
|
||||
"1": {
|
||||
"Name": "DIDI PostgreSQL Cluster",
|
||||
"Group": "Production",
|
||||
"Host": "10.11.50.167",
|
||||
"Port": 5000,
|
||||
"MaintenanceDB": "DIDI",
|
||||
"Username": "bos_interface",
|
||||
"SSLMode": "prefer",
|
||||
"PassFile": "/pgadmin4/pgpass"
|
||||
}
|
||||
}
|
||||
}
|
||||
Loading…
Add table
Add a link
Reference in a new issue