livrare lot 2

backend/services/data-layer/didiQueue/README.md

@@ -0,0 +1,215 @@
+# didiQueue - RabbitMQ Message Queue Service 🐰
+
+## Overview
+RabbitMQ message broker for asynchronous communication between the Orchestrator and Analysis Service in the DIDI Backend platform.
+
+## 🎯 Purpose
+Provides reliable message queuing for pipeline execution jobs, decoupling the API layer from the processing layer.
+
+## 🚀 Quick Start
+
+### Start the Service
+```bash
+# From this directory
+docker compose up -d
+
+# Or from data-layer directory
+make up-queue
+```
+
+### Access Points
+- **AMQP Protocol**: `localhost:5672`
+- **Management UI**: `http://localhost:15672`
+- **Default Credentials**: `admin / rabbitmq123`
+
+## 📊 Queue Architecture
+
+Since we're merging all analysis services into one unified service, we use a **single queue**:
+
+```
+Orchestrator → publishes → analysis_queue → consumed by → Analysis Service
+```
+
+### Queue Configuration
+- **Queue Name**: `analysis_queue`
+- **Type**: Durable (survives restarts)
+- **Dead Letter Queue**: `analysis_dlq` (for failed messages)
+- **Message TTL**: 24 hours
+- **Auto-delete**: No
+
+## 🔧 Configuration
+
+### Environment Variables
+Edit `.env` file to customize:
+```env
+RABBITMQ_USER=admin
+RABBITMQ_PASSWORD=rabbitmq123  # CHANGE IN PRODUCTION!
+RABBITMQ_VHOST=/
+RABBITMQ_PORT=5672
+RABBITMQ_MGMT_PORT=15672
+```
+
+### Resource Limits
+```yaml
+Memory: 1GB (max) / 512MB (reserved)
+CPU: 0.5 cores (max) / 0.25 cores (reserved)
+```
+
+## 📝 Message Format
+
+Messages published to the queue follow this structure:
+```json
+{
+  "run_id": "analysis_abc123_20250901_120000",
+  "pipeline_id": "uuid-here",
+  "pipeline_version": 1,
+  "input_data": {
+    "text": "Content to analyze",
+    "image": "base64_or_url",
+    "audio": "url_to_audio",
+    "video": "url_to_video"
+  },
+  "media_type": "text|image|audio|video",
+  "created_at": "2025-09-01T12:00:00Z"
+}
+```
+
+## 🔍 Management
+
+### View Queue Status
+```bash
+# Using Management UI
+http://localhost:15672
+
+# Using CLI
+docker exec didi-queue rabbitmqctl list_queues
+
+# Check queue depth
+docker exec didi-queue rabbitmqctl list_queues name messages_ready messages_unacknowledged
+```
+
+### Purge Queue (Development Only)
+```bash
+# Remove all messages from queue
+docker exec didi-queue rabbitmqctl purge_queue analysis_queue
+```
+
+### Health Check
+```bash
+# Check if RabbitMQ is responsive
+docker exec didi-queue rabbitmq-diagnostics -q ping
+
+# Detailed health check
+docker exec didi-queue rabbitmq-diagnostics check_running
+```
+
+## 🏗️ Integration Points
+
+### Publishers (Orchestrator)
+```python
+import aio_pika
+
+# Connect
+connection = await aio_pika.connect_robust(
+    "amqp://admin:rabbitmq123@localhost:5672/"
+)
+channel = await connection.channel()
+
+# Publish message
+await channel.default_exchange.publish(
+    aio_pika.Message(body=json.dumps(message).encode()),
+    routing_key="analysis_queue"
+)
+```
+
+### Consumers (Analysis Service)
+```python
+# Declare queue
+queue = await channel.declare_queue("analysis_queue", durable=True)
+
+# Consume messages
+async for message in queue:
+    async with message.process():
+        body = json.loads(message.body.decode())
+        # Process the message
+```
+
+## 🛠️ Troubleshooting
+
+### Queue is not created
+The `init-queues.sh` script runs automatically on container start. Check logs:
+```bash
+docker logs didi-queue
+```
+
+### Messages not being consumed
+1. Check if Analysis Service is running
+2. Verify queue has messages: `docker exec didi-queue rabbitmqctl list_queues`
+3. Check for dead letter queue: `docker exec didi-queue rabbitmqctl list_queues | grep dlq`
+
+### High memory usage
+```bash
+# Check memory usage
+docker exec didi-queue rabbitmq-diagnostics memory_breakdown
+
+# Set memory limit
+docker exec didi-queue rabbitmqctl set_vm_memory_high_watermark 0.4
+```
+
+## 🔐 Security
+
+### Production Checklist
+- [ ] Change default password in `.env`
+- [ ] Enable SSL/TLS for connections
+- [ ] Restrict management UI access
+- [ ] Set up user permissions
+- [ ] Configure firewall rules
+- [ ] Enable audit logging
+
+### Create Production User
+```bash
+# Create new user
+docker exec didi-queue rabbitmqctl add_user analysis_service SECURE_PASSWORD
+
+# Set permissions
+docker exec didi-queue rabbitmqctl set_permissions -p / analysis_service ".*" ".*" ".*"
+
+# Set user tags
+docker exec didi-queue rabbitmqctl set_user_tags analysis_service monitoring
+```
+
+## 📊 Monitoring
+
+### Key Metrics
+- Queue depth (messages waiting)
+- Message rates (publish/consume)
+- Connection count
+- Memory usage
+- Disk usage
+
+### Prometheus Metrics
+RabbitMQ exposes metrics at: `http://localhost:15692/metrics`
+
+## 🔄 Backup & Recovery
+
+### Backup
+```bash
+# Export definitions
+docker exec didi-queue rabbitmqctl export_definitions /var/lib/rabbitmq/backup.json
+docker cp didi-queue:/var/lib/rabbitmq/backup.json ./backup.json
+```
+
+### Restore
+```bash
+# Import definitions
+docker cp ./backup.json didi-queue:/var/lib/rabbitmq/backup.json
+docker exec didi-queue rabbitmqctl import_definitions /var/lib/rabbitmq/backup.json
+```
+
+## 📚 Related Documentation
+- [Data Layer README](../README.md)
+- [RabbitMQ Documentation](https://www.rabbitmq.com/documentation.html)
+- [AMQP Protocol](https://www.amqp.org/)
+
+---
+*Part of the DIDI Backend Data Layer*