clossers/didi-lot2-backend
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

livrare lot 2

Browse source
This commit is contained in:
EVOTECH IT SRL 2026-07-10 03:39:53 -07:00
commit 8ecc78e729
763 changed files with 164593 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

13
backend/services/data-layer/didiQueue/.env.example Normal file
View file

@ -0,0 +1,13 @@
# RabbitMQ Configuration
# CHANGE THESE FOR PRODUCTION!
RABBITMQ_USER=admin
RABBITMQ_PASSWORD=CHANGE_ME_IN_PRODUCTION
RABBITMQ_VHOST=/
# Port Configuration
RABBITMQ_PORT=5672
RABBITMQ_MGMT_PORT=15672
# Resource Limits (optional)
RABBITMQ_VM_MEMORY_HIGH_WATERMARK=0.4
RABBITMQ_DISK_FREE_LIMIT=1GB

356
backend/services/data-layer/didiQueue/INDEX.md Normal file
View file

@ -0,0 +1,356 @@
# didiQueue - Index
Coada de mesaje RabbitMQ pentru procesarea asincrona a analizelor. Primeste task-uri de analiza de la agent-v3, le distribuie la workeri pe componente, si colecteaza rezultatele intr-un agregator de verdict. Nu contine cod custom -- doar container RabbitMQ cu script de initializare.
**Productia ruleaza pe RabbitMQ LOCAL** (container `staging-dataLayer-rabbitmq`). Decizie: stabilitate + zero dependinte externe. Clusterul RabbitMQ RAG (managed extern) ramane configurat ca **fallback de urgenta pentru HA**, activabil cu `redis-switch.sh cluster --rabbit`, nu este folosit operational acum.
**Productie (LOCAL — activ)**:
- Imagine: `rabbitmq:3.12-management-alpine`
- Container: `staging-dataLayer-rabbitmq`
- Port AMQP: 5672 (expus pe host: `0.0.0.0:5672->5672`)
- Port Management UI: 15672 (expus pe host: `0.0.0.0:15672->15672`)
- Credentiale: `admin` / `rabbitmq123` (din `.env`)
- Vhost: `/`
- Retea: `didi-network`
**Fallback HA (cluster RAG — disponibil dar inactiv)**:
- Endpoint: `10.11.50.100:16672` (HAProxy VIP)
- Vhost: `/didi`
- User: `didi`
- Parola: din `.cluster-credentials.env` (gitignored)
---
## Ce face
1. **Primeste task-uri de analiza** -- publicate de agent-v3 dispatcher
2. **Distribuie catre workeri** -- fiecare componenta (techniques, ai_tampered, claims, domain) are cozi separate, plus media-preprocess pentru audio/video
3. **Prioritizeaza dupa plan** -- plan 1 (free) = prioritate 1, plan 6 (enterprise) = prioritate 10
4. **Colecteaza rezultate** -- workerii publica in coada de rezultate, agregatorul face fan-in
5. **DLQ** -- mesajele care esueaza dupa 3 incercari merg in dead-letter queue
6. **TTL** -- mesajele expira dupa 24 ore
**Nota HIL Moderation**: tabela `moderation_queue` (schema `bos_analysis`) este o tabela PostgreSQL pentru starea de review uman, NU o coada AMQP. RabbitMQ ramane folosit doar pentru dispatch-ul analizei asincrone (5 familii de cozi componente x 6 plan tiers = 30 cozi: media_preprocess, techniques, ai_tampered, claims, domain — plus coada `analysis.results` + DLQ). HIL nu introduce cozi noi.
---
## Topologie cozi
### Exchange
| Nume | Tip | Durabil | Scop |
|------|-----|---------|------|
| analysis | topic | da | Ruteaza task-uri si rezultate |
### Cozi componente (30 total = 5 cozi x 6 plan types)
```
analysis.media_preprocess.1 analysis.media_preprocess.2 ... analysis.media_preprocess.6
analysis.techniques.1 analysis.techniques.2 ... analysis.techniques.6
analysis.ai_tampered.1 analysis.ai_tampered.2 ... analysis.ai_tampered.6
analysis.claims.1 analysis.claims.2 ... analysis.claims.6
analysis.domain.1 analysis.domain.2 ... analysis.domain.6
```
Configurare per coada:
- Durabil: da
- Max prioritate: 10
- Dead-letter exchange: '' (default)
- Dead-letter routing key: analysis_dlq
- Message TTL: 86,400,000 ms (24 ore)
### Coada rezultate (fan-in)
| Coada | Bindings | Scop |
|-------|----------|------|
| analysis.results | analysis.results.techniques, analysis.results.ai_tampered, analysis.results.claims, analysis.results.domain | Colecteaza rezultate de la toti workerii |
### Dead-letter queue
| Coada | Scop |
|-------|------|
| analysis_dlq | Mesaje care au esuat dupa 3 retry-uri |
---
## Prioritati per plan
| Plan Type | Prioritate | Tip utilizator |
|-----------|------------|----------------|
| 1 | 1 | freemium |
| 2 | 2 | starter |
| 3 | 4 | basic |
| 4 | 6 | pro |
| 5 | 8 | business |
| 6 | 10 | enterprise |
Mesajele cu prioritate mai mare sunt procesate primele din coada.
---
## Format mesaje
### Task message (Dispatcher -> Worker)
Publicat de agent-v3 dispatcher in cozile de componente.
```
{
sessionId: "uuid",
component: "techniques" | "ai_tampered" | "claims" | "domain",
planType: 1-6,
priority: 1-10,
input: {
content: "text de analizat",
url: "URL optional (video/domain)",
mediaPath: "cale MinIO optional (audio/video)",
inputType: "text" | "url" | "image" | "audio" | "video"
},
userId: "string optional",
userEmail: "string optional",
timestamp: 1695312000000,
retryCount: 0
}
```
AMQP properties: persistent=true, contentType=application/json, headers={sessionId, component, planType}
### Result message (Worker -> Aggregator)
Publicat de worker in coada analysis.results.
```
{
sessionId: "uuid",
component: "techniques" | "ai_tampered" | "claims" | "domain",
success: true | false,
score: 0-100,
data: { ... rezultat flat componenta ... },
error: "mesaj eroare daca success=false",
processingTime: 3500,
timestamp: 1695312003500
}
```
---
## Cine publica mesaje
| Cine | Ce publica | In ce coada | Logica in fisier |
|------|-----------|-------------|------------------|
| agent-v3 dispatcher | Task-uri de analiza | analysis.{component}.{planType} | agent-v3/src/queue/dispatcher.ts |
| Workeri componente | Rezultate analiza | analysis.results.{component} | agent-v3/src/queue/workers/component-worker.ts |
## Cine consuma mesaje
| Cine | Din ce coada | Ce face | Replici Docker |
|------|-------------|---------|----------------|
| worker-media-preprocess | analysis.media_preprocess.1-6 | Download yt-dlp + ffmpeg cadre + Whisper + Vision OCR, cache in Redis, dispatch task-uri componente | 2 |
| worker-techniques | analysis.techniques.1-6 | Ruleaza TechniquesV3Executor | 2 (prefetch 5) |
| worker-ai-tampered | analysis.ai_tampered.1-6 | Ruleaza AITamperedExecutor | 2 (prefetch 5) |
| worker-claims | analysis.claims.1-6 | Ruleaza ClaimsExecutor | 3 (prefetch 3) |
| worker-domain | analysis.domain.1-6 | Ruleaza analyzeDomain() | 2 (prefetch 10) |
| verdict-aggregator | analysis.results | Fan-in + VerdictCalculator | 2 (prefetch 10) |
Claims are 3 replici (nu 2) pentru ca e cel mai lent (cautare web per claim).
Domain are prefetch 10 pentru ca e cel mai rapid (analiza locala, fara LLM).
Media-preprocess este nou (din 2026-03): centralizeaza download/transcribe/vision pentru audio+video, inlocuind logica per-worker. Workerii componente citesc media procesata din Redis (TTL 1h, chei `agent:media:{sessionId}:transcript`, `agent:media:{sessionId}:vision:misinformation`, etc.).
---
## Fluxul complet async
```
Client POST /api/v3/pipeline/analyze-async { text, plan_type: 4 }
|
v
agent-v3 Dispatcher
|-- Salveaza SessionState in Redis (status: processing)
|-- Salveaza sesiune initiala in Redis + PostgreSQL
|-- Publica 4 task-uri in RabbitMQ:
| analysis.techniques.4 (prioritate 6)
| analysis.ai_tampered.4 (prioritate 6)
| analysis.claims.4 (prioritate 6)
| analysis.domain.4 (prioritate 6)
|
v
Response 202: { session_id, poll_url, result_url }
--- In paralel, 4 workeri proceseaza ---
Worker Techniques (consuma din analysis.techniques.4)
|-- Achizitioneaza lock Redis (300s TTL)
|-- Ruleaza TechniquesV3Executor (screening -> deep analysis)
|-- Publica rezultat in analysis.results.techniques
|-- ACK mesaj
Worker AI-Tampered (consuma din analysis.ai_tampered.4)
|-- Ruleaza AITamperedExecutor
|-- Publica rezultat in analysis.results.ai_tampered
Worker Claims (consuma din analysis.claims.4)
|-- Ruleaza ClaimsExecutor (extrage + verifica prin web)
|-- Publica rezultat in analysis.results.claims
Worker Domain (consuma din analysis.domain.4)
|-- Ruleaza analyzeDomain()
|-- Publica rezultat in analysis.results.domain
--- Agregatorul colecteaza ---
Verdict Aggregator (consuma din analysis.results)
|-- Primeste rezultat componenta
|-- Achizitioneaza lock Redis (30s TTL)
|-- Actualizeaza SessionState in Redis (completedComponents++)
|-- Daca toate 4 componente gata:
| |-- VerdictCalculator.calculate() (functie pura)
| |-- VerdictExplanation.generate() (LLM, RO+EN)
| |-- PersistService.persist() (Redis + PostgreSQL)
|-- ACK mesaj
--- Clientul polleaza ---
GET /api/v3/pipeline/{sessionId}/queue-status
-> { progress: 75%, completed_components: ["techniques", "ai_tampered", "domain"] }
GET /api/v3/pipeline/{sessionId}/result
-> AnalysisSession completa (cand status=completed)
```
---
## Retry si error handling
| Situatie | Actiune | Rezultat |
|----------|---------|----------|
| Procesare reusita | channel.ack(msg) | Mesaj sters din coada |
| Eroare retryable + retryCount < 3 | channel.nack(msg, false, true) | Mesaj pus inapoi in coada |
| Eroare retryable + retryCount >= 3 | channel.nack(msg, false, false) | Mesaj trimis in analysis_dlq |
| Eroare non-retryable | channel.nack(msg, false, false) | Mesaj trimis in analysis_dlq |
| RabbitMQ indisponibil | Fallback sync | agent-v3 ruleaza analiza sincrona |
Lock-uri Redis previn procesarea dubla:
- Lock componenta: `didi:queue:lock:{sessionId}:{component}` (TTL 300s)
- Lock agregator: `didi:queue:lock:aggregator:{sessionId}` (TTL 30s)
---
## Procesare media in workeri
Workerii proceseaza media inainte de analiza text:
| Input type | Ce face workerul | Logica in |
|-----------|-----------------|-----------|
| text | Nimic, trimite direct la executor | component-worker.ts |
| audio | Transcriere via M17/Groq/OpenAI | shared/media/transcription.ts |
| video | Download yt-dlp + ffmpeg cadre + transcriere | shared/media/video-processor.ts |
| image | Extragere text via vision cascade (pentru techniques/claims) | shared/media/vision.ts |
Timeout-uri worker:
- Video: 600,000 ms (10 minute)
- Default: 120,000 ms (2 minute)
---
## Conexiune RabbitMQ (din agent-v3)
Fisier: `agent-v3/src/queue/connection.ts` + `agent-v3/src/shared/queue/constants.ts`
- Lazy initialization (conectare la prima utilizare)
- Doua canale: regular (consume) + confirm (publish cu confirmare)
- Auto-recovery la deconectare (`CONNECTION_RETRY_DELAY = 2s` pentru failover HA pe cluster)
- Graceful shutdown pe SIGTERM/SIGINT (stop consume, close channels, close connection)
- URL building foloseste `encodeURIComponent()` pentru vhost (`/didi` -> `%2Fdidi` in AMQP URI)
```
# Productie (LOCAL — activ)
URL: amqp://admin:rabbitmq123@staging-dataLayer-rabbitmq:5672/ (vhost `/`)
# Fallback HA (cluster RAG, HAProxy VIP — disponibil dar inactiv)
URL: amqp://didi:<password>@10.11.50.100:16672/%2Fdidi
```
### Switch intre cluster si local
Script: `backend/services/orchestration-layer/scripts/redis-switch.sh` (denumirea istorica este `redis-switch`, dar suporta si RabbitMQ via flag `--rabbit`).
```
redis-switch.sh {cluster|local|status} [redis|rabbit|both]
```
Verificare topologie: `agent-v3/scripts/verify-rabbitmq-cluster.ts` (verifica privilegii vhost + topologia celor 30 cozi + exchange + DLQ).
---
## Fisiere in directorul didiQueue
```
init-queues.sh -- Creeaza exchange + coada legacy singulara + DLQ + binding + policy (idempotent)
.env -- Credentiale + porturi
.env.example -- Template
README.md -- Documentatie
```
Zero cod custom. Doar container RabbitMQ standard cu management plugin.
Nota: init-queues.sh creeaza topologia legacy cu o singura coada. Topologia actuala cu 30 cozi (5 componente x 6 planuri: media_preprocess, techniques, ai_tampered, claims, domain) + coada results + DLQ este creata dinamic de workerii agent-v3 la startup (vezi agent-v3/src/queue/connection.ts si constants.ts).
---
## Fisiere cod integrare (in agent-v3)
| Fisier | Rol |
|--------|-----|
| agent-v3/src/queue/connection.ts | Manager conexiune RabbitMQ (lazy, auto-recovery) |
| agent-v3/src/shared/queue/constants.ts | Nume exchange/cozi, prioritati, config workeri |
| agent-v3/src/queue/dispatcher.ts | Publica task-uri in cozi componente |
| agent-v3/src/queue/aggregator.ts | Consuma rezultate, calculeaza verdict, persista |
| agent-v3/src/queue/workers/component-worker.ts | Worker generic (lock, procesare, publish result, ack/nack) |
| agent-v3/src/worker-entrypoints/techniques.ts | Entry point Docker worker techniques |
| agent-v3/src/worker-entrypoints/ai-tampered.ts | Entry point Docker worker ai-tampered |
| agent-v3/src/worker-entrypoints/claims.ts | Entry point Docker worker claims |
| agent-v3/src/worker-entrypoints/domain.ts | Entry point Docker worker domain |
| agent-v3/src/worker-entrypoints/aggregator.ts | Entry point Docker verdict aggregator |
---
## Configurare Docker
```yaml
# din data-layer/docker-compose.yml
staging-dataLayer-rabbitmq:
image: rabbitmq:3.12-management-alpine
container_name: staging-dataLayer-rabbitmq
restart: unless-stopped
environment:
RABBITMQ_DEFAULT_USER: admin
RABBITMQ_DEFAULT_PASS: rabbitmq123
RABBITMQ_DEFAULT_VHOST: /
ports:
- "5672:5672" # AMQP (expus pe host)
- "15672:15672" # Management UI (expus pe host)
volumes:
- didi-staging-rabbitmq-data:/var/lib/rabbitmq
networks:
- didi-network
healthcheck:
test: rabbitmq-diagnostics -q ping
interval: 30s
timeout: 10s
retries: 3
start_period: 60s
```
Workerii sunt definiti in agent-v3/docker-compose.yml (vezi agent-v3/INDEX.md pentru detalii replici).
---
## Ce NU face
- Nu are cod custom (container RabbitMQ standard)
- Nu are clustering (instanta singulara)
- Nu are mirroring/quorum queues (nu e HA)
- Nu are SSL/TLS (AMQP plain text intern)
- Nu are ACL per serviciu (toti folosesc userul admin)
- Nu are delayed message plugin (retry prin requeue nativ)
- Nu are shovel/federation (nu transfera mesaje intre brokeri)

147
backend/services/data-layer/didiQueue/MIGRATION.md Normal file
View file

@ -0,0 +1,147 @@
# RabbitMQ — migrat pe clusterul RAG (2026-04-22)
> **TL;DR**: DIDI folosește **clusterul RabbitMQ RAG** (3 noduri 3.13 + HAProxy VIP). Containerul local `staging-dataLayer-rabbitmq` din `data-layer/docker-compose.yml` e **oprit dar păstrat** ca fallback rapid. Conexiunea e centralizată via `agent-v3/src/queue/connection.ts`.
---
## Ce era aici (legacy)
`staging-dataLayer-rabbitmq` — un container RabbitMQ 3.12.6 management standalone pe `didi-network` Docker. Single-node, fără mirroring, vhost `/`. Topologie creată dinamic la startup workerilor.
## Ce e acum
### Cluster RabbitMQ RAG (producție)
| Nod | Hostname | IP | AMQP | Mgmt UI |
|---|---|---|---|---|
| rag01 (HAProxy VIP) | `rag01` | `10.11.50.100` | 16672 (VIP), 5672 direct | 16673 (VIP), 15672 direct |
| rag02 | `rag02` | `10.11.50.102` | 5672 | 15672 |
| rag03 | `rag03` | `10.11.50.103` | 5672 | 15672 |
**Endpoint-uri pentru aplicații:**
| Scop | Endpoint |
|---|---|
| **AMQP VIP** (publish + consume) | `10.11.50.100:16672` |
| **Management UI** | `http://10.11.50.100:16673` (admin/`<pwd>`) |
| HAProxy stats | `http://10.11.50.100:8404/stats` |
Vhost DIDI: `/didi` (izolare totală — alte vhost-uri (`notify`, `/hassio`) nu se văd). User: `didi` cu permisiuni full pe `/didi`. Parolă în `vault-ul de credențiale` `name='DIDI Platform RabbitMQ'`.
URL AMQP în cod: `amqp://didi:<pwd>@10.11.50.100:16672/%2Fdidi` (slash URL-encoded ca `%2F`).
### Conexiune centralizată în cod
`agent-v3/src/queue/connection.ts` — manager unic ioredis-style cu lazy initialization, două canale (regular consume + confirm publish), auto-recovery la 2s pentru failover HA, graceful shutdown pe SIGTERM/SIGINT.
`agent-v3/src/shared/queue/constants.ts` — topologie cozi (exchange, routing keys, prioritați).
## Topologie cozi (creată dinamic de workeri la startup)
### Exchange
| Nume | Tip | Durabil |
|---|---|---|
| `analysis` | topic | da |
### Cozi componente (24 = 4 componente × 6 plan types)
```
analysis.techniques.{1..6} analysis.ai_tampered.{1..6}
analysis.claims.{1..6} analysis.domain.{1..6}
```
Fiecare cu max-priority=10, dead-letter exchange, message TTL 24h.
### Coadă agregare
`analysis.results` (fan-in) cu bindings de la fiecare componentă.
### Prioritați per plan
| Plan Type | Prioritate | Tip |
|---|---|---|
| 1 | 1 | freemium |
| 2 | 2 | starter |
| 3 | 4 | basic |
| 4 | 6 | pro |
| 5 | 8 | business |
| 6 | 10 | enterprise |
## Containerul local `staging-dataLayer-rabbitmq`
Definit în `data-layer/docker-compose.yml`, configurat să pornească dar **manual oprit** ca parte din migrare. Volumul `didi-staging-rabbitmq-data` e intact.
Status curent: `Exited`.
### De ce e păstrat?
Fallback rapid dacă cluster RAG e indisponibil. Pentru reactivare temporară:
```bash
backend/services/orchestration-layer/scripts/redis-switch.sh local rabbit
# (același script gestionează rabbit + redis)
```
## Switch rapid cluster ↔ local
```bash
# Folosește local
./redis-switch.sh local rabbit
# Cluster (default)
./redis-switch.sh cluster rabbit
# Both
./redis-switch.sh cluster both
# Status
./redis-switch.sh status both
```
## Verificare cluster
```bash
# Mgmt UI (browser)
http://10.11.50.100:16673 # admin/<pwd>
# Quick test prin rabbitmqctl pe nod cluster
ssh admin365@10.11.50.102
sudo rabbitmqctl status
sudo rabbitmqctl list_vhosts
sudo rabbitmqctl list_queues -p /didi name messages consumers
# Via HTTP API
curl -s -u admin:<pwd> http://10.11.50.100:16673/api/overview | python3 -m json.tool
```
## Workerii (în `agent-v3` docker-compose)
Servicii Docker care consumă din cluster:
| Worker | Replici | Coadă |
|---|---|---|
| `worker-techniques` | 2 | `analysis.techniques.1-6` |
| `worker-ai-tampered` | 2 | `analysis.ai_tampered.1-6` |
| `worker-claims` | 3 | `analysis.claims.1-6` (mai multe replici, mai lent) |
| `worker-domain` | 2 | `analysis.domain.1-6` |
| `worker-media-preprocess` | 2 | `analysis.media_preprocess.*` |
| `verdict-aggregator` | 2 | `analysis.results` |
## Verificări dispatcher
Script verificare cluster ready: `agent-v3/scripts/verify-rabbitmq-cluster.ts` — testează conectivitate, vhost privileges, topology, în funcție de user `didi` și endpoint VIP.
## Linkuri rapide
- Ghid utilizare cluster: `landingzone/rabbitmq-rag/README.md` (repo `git.finesynergy.eu/lucian/landingzone`)
- Onboarding vhost nou: `landingzone/rabbitmq-rag/CLAUDE_PROMPT.md`
- Mgmt UI: `http://10.11.50.100:16673`
## Status
- ✅ Migrare aplicată: 2026-04-22
- ✅ Vhost `/didi` izolat, user `didi` cu permisiuni minime
- ✅ Container local păstrat ca fallback (oprit, volum intact)
- ✅ Topologie creată dinamic la startup (24 cozi + results + DLQ)
- ✅ Workerii (în agent-v3) consumă cu prefetch ajustat per componentă

215
backend/services/data-layer/didiQueue/README.md Normal file
View file

@ -0,0 +1,215 @@
# didiQueue - RabbitMQ Message Queue Service 🐰
## Overview
RabbitMQ message broker for asynchronous communication between the Orchestrator and Analysis Service in the DIDI Backend platform.
## 🎯 Purpose
Provides reliable message queuing for pipeline execution jobs, decoupling the API layer from the processing layer.
## 🚀 Quick Start
### Start the Service
```bash
# From this directory
docker compose up -d
# Or from data-layer directory
make up-queue
```
### Access Points
- **AMQP Protocol**: `localhost:5672`
- **Management UI**: `http://localhost:15672`
- **Default Credentials**: `admin / rabbitmq123`
## 📊 Queue Architecture
Since we're merging all analysis services into one unified service, we use a **single queue**:
```
Orchestrator → publishes → analysis_queue → consumed by → Analysis Service
```
### Queue Configuration
- **Queue Name**: `analysis_queue`
- **Type**: Durable (survives restarts)
- **Dead Letter Queue**: `analysis_dlq` (for failed messages)
- **Message TTL**: 24 hours
- **Auto-delete**: No
## 🔧 Configuration
### Environment Variables
Edit `.env` file to customize:
```env
RABBITMQ_USER=admin
RABBITMQ_PASSWORD=rabbitmq123 # CHANGE IN PRODUCTION!
RABBITMQ_VHOST=/
RABBITMQ_PORT=5672
RABBITMQ_MGMT_PORT=15672
```
### Resource Limits
```yaml
Memory: 1GB (max) / 512MB (reserved)
CPU: 0.5 cores (max) / 0.25 cores (reserved)
```
## 📝 Message Format
Messages published to the queue follow this structure:
```json
{
"run_id": "analysis_abc123_20250901_120000",
"pipeline_id": "uuid-here",
"pipeline_version": 1,
"input_data": {
"text": "Content to analyze",
"image": "base64_or_url",
"audio": "url_to_audio",
"video": "url_to_video"
},
"media_type": "text|image|audio|video",
"created_at": "2025-09-01T12:00:00Z"
}
```
## 🔍 Management
### View Queue Status
```bash
# Using Management UI
http://localhost:15672
# Using CLI
docker exec didi-queue rabbitmqctl list_queues
# Check queue depth
docker exec didi-queue rabbitmqctl list_queues name messages_ready messages_unacknowledged
```
### Purge Queue (Development Only)
```bash
# Remove all messages from queue
docker exec didi-queue rabbitmqctl purge_queue analysis_queue
```
### Health Check
```bash
# Check if RabbitMQ is responsive
docker exec didi-queue rabbitmq-diagnostics -q ping
# Detailed health check
docker exec didi-queue rabbitmq-diagnostics check_running
```
## 🏗️ Integration Points
### Publishers (Orchestrator)
```python
import aio_pika
# Connect
connection = await aio_pika.connect_robust(
"amqp://admin:rabbitmq123@localhost:5672/"
)
channel = await connection.channel()
# Publish message
await channel.default_exchange.publish(
aio_pika.Message(body=json.dumps(message).encode()),
routing_key="analysis_queue"
)
```
### Consumers (Analysis Service)
```python
# Declare queue
queue = await channel.declare_queue("analysis_queue", durable=True)
# Consume messages
async for message in queue:
async with message.process():
body = json.loads(message.body.decode())
# Process the message
```
## 🛠️ Troubleshooting
### Queue is not created
The `init-queues.sh` script runs automatically on container start. Check logs:
```bash
docker logs didi-queue
```
### Messages not being consumed
1. Check if Analysis Service is running
2. Verify queue has messages: `docker exec didi-queue rabbitmqctl list_queues`
3. Check for dead letter queue: `docker exec didi-queue rabbitmqctl list_queues | grep dlq`
### High memory usage
```bash
# Check memory usage
docker exec didi-queue rabbitmq-diagnostics memory_breakdown
# Set memory limit
docker exec didi-queue rabbitmqctl set_vm_memory_high_watermark 0.4
```
## 🔐 Security
### Production Checklist
- [ ] Change default password in `.env`
- [ ] Enable SSL/TLS for connections
- [ ] Restrict management UI access
- [ ] Set up user permissions
- [ ] Configure firewall rules
- [ ] Enable audit logging
### Create Production User
```bash
# Create new user
docker exec didi-queue rabbitmqctl add_user analysis_service SECURE_PASSWORD
# Set permissions
docker exec didi-queue rabbitmqctl set_permissions -p / analysis_service ".*" ".*" ".*"
# Set user tags
docker exec didi-queue rabbitmqctl set_user_tags analysis_service monitoring
```
## 📊 Monitoring
### Key Metrics
- Queue depth (messages waiting)
- Message rates (publish/consume)
- Connection count
- Memory usage
- Disk usage
### Prometheus Metrics
RabbitMQ exposes metrics at: `http://localhost:15692/metrics`
## 🔄 Backup & Recovery
### Backup
```bash
# Export definitions
docker exec didi-queue rabbitmqctl export_definitions /var/lib/rabbitmq/backup.json
docker cp didi-queue:/var/lib/rabbitmq/backup.json ./backup.json
```
### Restore
```bash
# Import definitions
docker cp ./backup.json didi-queue:/var/lib/rabbitmq/backup.json
docker exec didi-queue rabbitmqctl import_definitions /var/lib/rabbitmq/backup.json
```
## 📚 Related Documentation
- [Data Layer README](../README.md)
- [RabbitMQ Documentation](https://www.rabbitmq.com/documentation.html)
- [AMQP Protocol](https://www.amqp.org/)
---
*Part of the DIDI Backend Data Layer*

72
backend/services/data-layer/didiQueue/init-queues.sh Normal file
View file

@ -0,0 +1,72 @@
#!/bin/bash
# Init script for RabbitMQ queue setup
# This runs automatically when the container starts
set -e
# Wait for RabbitMQ to be ready
until rabbitmqctl status > /dev/null 2>&1; do
echo "Waiting for RabbitMQ to start..."
sleep 2
done
echo "RabbitMQ is ready. Setting up queues..."
# Since we're merging all analysis services into one, we only need ONE queue
# Create the unified analysis queue
rabbitmqctl eval '
rabbit_exchange:declare(
{resource, <<"/">>, exchange, <<"analysis">>},
topic,
true,
false,
false,
[]
).' || true
# Declare the unified analysis queue with proper settings
rabbitmqctl eval '
rabbit_amqqueue:declare(
{resource, <<"/">>, queue, <<"analysis_queue">>},
true, % Durable
false, % Not exclusive
false, % Not auto-delete
[], % No arguments
none % No owner
).' || true
# Bind the queue to the exchange
rabbitmqctl eval '
rabbit_binding:add_explicit(
{binding,
{resource, <<"/">>, exchange, <<"analysis">>},
<<"analysis.*">>,
{resource, <<"/">>, queue, <<"analysis_queue">>},
[]
}
).' || true
# Optional: Create a dead letter queue for failed messages
rabbitmqctl eval '
rabbit_amqqueue:declare(
{resource, <<"/">>, queue, <<"analysis_dlq">>},
true, % Durable
false, % Not exclusive
false, % Not auto-delete
[], % No arguments
none % No owner
).' || true
echo "✅ Queue setup complete!"
echo "Created queues:"
echo " - analysis_queue (main queue for all analysis types)"
echo " - analysis_dlq (dead letter queue for failed messages)"
# Set queue policies for message TTL and retry
rabbitmqctl set_policy analysis-retry \
"analysis_queue" \
'{"message-ttl":86400000, "dead-letter-exchange":"", "dead-letter-routing-key":"analysis_dlq"}' \
--priority 0 \
--apply-to queues || true
echo "✅ Queue policies configured!"