livrare lot 2

2026-07-10 03:39:53 -07:00 · 2026-07-10 03:39:53 -07:00 · 8ecc78e729
commit 8ecc78e729
763 changed files with 164593 additions and 0 deletions
--- a/backend/observability/.env.example
+++ b/backend/observability/.env.example
@ -0,0 +1,17 @@
+# Observability Stack — Environment Template
+# ============================================
+# Copy to .env and fill in:
+#   cp .env.example .env
+
+# Grafana
+GRAFANA_ADMIN_PASSWORD=<strong-random-password>
+
+# Alertmanager — for SMTP email alerts (uses SendGrid)
+SENDGRID_API_KEY=<your-sendgrid-key>
+
+# Optional: Slack/Teams webhook for alerts (paste in alertmanager.yml when ready)
+# SLACK_WEBHOOK_URL=https://hooks.slack.com/services/...
+# TEAMS_WEBHOOK_URL=https://...
+
+# Optional: PagerDuty integration key
+# PAGERDUTY_KEY=...
--- a/backend/observability/README.md
+++ b/backend/observability/README.md
@ -0,0 +1,234 @@
+# DiDi Observability Stack
+
+Stack complet de monitorizare, logging și tracing distribuit pentru platforma DiDi. Implementează cerința **LOT 2 modul 8 (Observabilitate & Logging)** din caietul de sarcini + diagrama A.9 din oferta EVOTECH.
+
+## Componente
+
+| Component | Rol | URL UI (LAN) |
+|---|---|---|
+| **Prometheus** | Scraping metrici + alert rules | http://10.11.10.12:9090 |
+| **Grafana** | Dashboard-uri vizualizare metrici + loguri + traces | http://10.11.10.12:3030 |
+| **Loki** | Agregare loguri | http://10.11.10.12:3100 |
+| **Promtail** | Shipper Docker logs → Loki | (no UI) |
+| **Jaeger** | UI tracing distribuit | http://10.11.10.12:16686 |
+| **OTel Collector** | Receiver OTLP (trace + metric) + processor + exporter | http://10.11.10.12:4319 (gRPC), :4320 (HTTP) |
+| **Alertmanager** | Routing alerte (email, Slack/Teams, PagerDuty) | http://10.11.10.12:9093 |
+
+## Quick start
+
+```bash
+cd /home/admin365/didi_mono/didi_mono/backend/observability
+
+# Setup .env
+cp .env.example .env
+$EDITOR .env  # set GRAFANA_ADMIN_PASSWORD + SENDGRID_API_KEY
+
+# Start stack
+docker compose up -d
+
+# Verify all healthy
+docker compose ps
+```
+
+Apoi accesează **Grafana** la http://10.11.10.12:3030 (user `admin`, parola din `GRAFANA_ADMIN_PASSWORD`).
+
+Datasources sunt deja provisioned (Prometheus, Loki, Jaeger). Adaugă dashboard-uri custom în `grafana/dashboards/` (auto-provisioned la 30s).
+
+## Instrumentare servicii
+
+### Node.js (agent-v3, didi-framework, admin-dashboard)
+
+Instalează `prom-client` + `@opentelemetry/sdk-node`:
+```bash
+npm install --save prom-client @opentelemetry/api @opentelemetry/sdk-node \
+  @opentelemetry/auto-instrumentations-node @opentelemetry/exporter-trace-otlp-grpc \
+  @opentelemetry/resources @opentelemetry/semantic-conventions
+```
+
+Adaugă în `src/index.ts` (înainte de orice alt import):
+```ts
+import { NodeSDK } from '@opentelemetry/sdk-node';
+import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-grpc';
+import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node';
+import { Resource } from '@opentelemetry/resources';
+import { SemanticResourceAttributes } from '@opentelemetry/semantic-conventions';
+
+const sdk = new NodeSDK({
+  resource: new Resource({
+    [SemanticResourceAttributes.SERVICE_NAME]: 'agent-v3',
+    [SemanticResourceAttributes.SERVICE_VERSION]: '3.0.0',
+  }),
+  traceExporter: new OTLPTraceExporter({
+    url: process.env.OTEL_EXPORTER_OTLP_ENDPOINT || 'http://didi-otel-collector:4317',
+  }),
+  instrumentations: [getNodeAutoInstrumentations()],
+});
+sdk.start();
+```
+
+Adaugă endpoint `/metrics` în Express:
+```ts
+import { register, collectDefaultMetrics } from 'prom-client';
+collectDefaultMetrics({ prefix: 'didi_agent_v3_' });
+
+app.get('/metrics', async (req, res) => {
+  res.set('Content-Type', register.contentType);
+  res.end(await register.metrics());
+});
+```
+
+Custom metrics relevante DiDi:
+```ts
+import { Counter, Histogram } from 'prom-client';
+
+export const analysisCompleted = new Counter({
+  name: 'didi_analyses_completed_total',
+  help: 'Total number of analyses completed',
+  labelNames: ['component', 'tier', 'media_type', 'verdict'],
+});
+
+export const pipelineDuration = new Histogram({
+  name: 'didi_pipeline_duration_seconds',
+  help: 'Pipeline duration in seconds',
+  labelNames: ['component', 'tier'],
+  buckets: [1, 5, 10, 30, 60, 120, 300],
+});
+
+// În executor:
+const end = pipelineDuration.startTimer({ component: 'techniques', tier });
+try {
+  await runAnalysis();
+  analysisCompleted.inc({ component: 'techniques', tier, media_type, verdict });
+} finally {
+  end();
+}
+```
+
+### Python (ai_platform modules)
+
+Instalează `prometheus_client` + `opentelemetry-instrumentation-fastapi`:
+```bash
+pip install prometheus-client opentelemetry-api opentelemetry-sdk \
+  opentelemetry-exporter-otlp opentelemetry-instrumentation-fastapi
+```
+
+Adaugă în `app.py`:
+```python
+from prometheus_client import make_asgi_app
+from opentelemetry import trace
+from opentelemetry.sdk.trace import TracerProvider
+from opentelemetry.sdk.trace.export import BatchSpanProcessor
+from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
+from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
+from opentelemetry.sdk.resources import Resource
+
+resource = Resource(attributes={"service.name": "llm-inference"})
+trace.set_tracer_provider(TracerProvider(resource=resource))
+trace.get_tracer_provider().add_span_processor(
+    BatchSpanProcessor(OTLPSpanExporter(endpoint="http://didi-otel-collector:4317", insecure=True))
+)
+
+app = FastAPI()
+FastAPIInstrumentor.instrument_app(app)
+
+# Mount /metrics
+app.mount("/metrics", make_asgi_app())
+```
+
+### Environment variables pe servicii
+
+Adaugă în compose-urile fiecărui serviciu:
+```yaml
+environment:
+  OTEL_EXPORTER_OTLP_ENDPOINT: http://didi-otel-collector:4317
+  OTEL_SERVICE_NAME: agent-v3
+  OTEL_RESOURCE_ATTRIBUTES: cluster=didi-prod,environment=production
+```
+
+## Dashboards predefinite
+
+Adaugă fișiere JSON în `grafana/dashboards/` (auto-importate). Recomandate:
+
+1. **DiDi Platform Overview**
+   - KPI cards: analize/min, error rate, P50/P95/P99 pipeline latency, GPU util
+   - Time-series: requests per service, queue backlog (RabbitMQ), Redis hit rate
+
+2. **AI Platform**
+   - LLM inference latency per model, GPU memory (Qwen 3.5, BusterX)
+   - Brain analysis_atom cache hit rate, scheduler health
+
+3. **Cozi & Workers**
+   - RabbitMQ queue depth per component × tier
+   - Worker throughput, retry count, DLQ messages
+
+4. **Cost & business**
+   - Stripe webhook success rate, Sales Invoice rate, MRR proxy
+
+5. **Infrastructure**
+   - CPU/RAM/Disk/Network per node, container restarts, healthcheck failures
+
+Import dashboards exemple din comunitate:
+- ID 11074 (Node Exporter Full)
+- ID 13639 (Logs via Loki)
+- ID 17761 (Cadvisor)
+- ID 14570 (RabbitMQ Cluster)
+
+În Grafana: **+ → Import → paste ID-ul → load**.
+
+## Verificare end-to-end
+
+```bash
+# 1. Verifică Prometheus scrapes
+curl -s http://10.11.10.12:9090/api/v1/targets | jq '.data.activeTargets[] | {job: .labels.job, health: .health}'
+
+# 2. Verifică Loki primește loguri
+curl -s 'http://10.11.10.12:3100/loki/api/v1/labels' | jq
+
+# 3. Trimite un trace de test din shell (după ce ai un service instrumentat)
+# Vizibil în Jaeger UI: http://10.11.10.12:16686
+
+# 4. Trimite o alertă de test
+curl -X POST http://10.11.10.12:9090/-/reload
+# Așteaptă să se trigger ServiceDown alert (timer ~5min)
+```
+
+## Retention
+
+- **Prometheus**: 30 zile (configurabil în compose `--storage.tsdb.retention.time`)
+- **Loki**: 7 zile (configurabil în `loki-config.yaml` `retention_period`)
+- **Jaeger** (Badger storage): persistent, ~10GB cap
+- **Alertmanager**: persistent state
+
+## SLA + escalation
+
+Vezi `alertmanager.yml` pentru routing:
+- `severity=critical` → notify imediat la `office@clossers.com`
+- `severity=warning` → batch, repeat 12h
+- TODO: adaugă Slack/Teams webhook + PagerDuty key pentru on-call
+
+## Troubleshooting
+
+**Prometheus arată target-uri DOWN**: verifică că serviciul are endpoint `/metrics` accesibil + că e pe `didi-network`.
+
+**Loki nu primește loguri**: verifică `promtail` logs (`docker logs didi-promtail`) — probabil container labels nu match-uiesc.
+
+**Jaeger fără traces**: verifică că serviciul are env `OTEL_EXPORTER_OTLP_ENDPOINT` setat corect + că face HTTP/gRPC către `didi-otel-collector:4317`.
+
+**Alertmanager nu trimite email**: verifică `SENDGRID_API_KEY` în env + că from address `alerts@didi365.eu` e validat în SendGrid.
+
+## Roadmap
+
+- [ ] Adaugă **postgres_exporter** pentru metrici PostgreSQL (slow queries, replication lag)
+- [ ] Adaugă **redis_exporter** pentru metrici Redis
+- [ ] Adaugă **nvidia_gpu_exporter** pe GPU host pentru metrici VRAM/utilization
+- [ ] Instrumentare agent-v3 (PR follow-up)
+- [ ] Instrumentare ai_platform modules (PR follow-up)
+- [ ] Slack/Teams webhook integration
+- [ ] PagerDuty on-call rotation
+- [ ] Synthetic monitoring (uptime checks pe didi365.eu)
+
+## Referințe
+
+- Caiet sarcini LOT 2 modul 8 (Observabilitate & Logging)
+- Oferta EVOTECH §A.9 (diagrama observabilitate completă)
+- Cercetare industrială §C (validare experimentală + KPI-uri)
--- a/backend/observability/alertmanager/alertmanager.yml
+++ b/backend/observability/alertmanager/alertmanager.yml
@ -0,0 +1,62 @@
+global:
+  resolve_timeout: 5m
+  # Configure SMTP for email alerts
+  smtp_from: 'alerts@didi365.eu'
+  smtp_smarthost: 'smtp.sendgrid.net:587'
+  smtp_auth_username: 'apikey'
+  # smtp_auth_password set via env $SENDGRID_API_KEY in compose
+
+route:
+  group_by: ['alertname', 'cluster', 'service']
+  group_wait: 30s
+  group_interval: 5m
+  repeat_interval: 12h
+  receiver: 'default'
+  routes:
+    - matchers:
+        - severity = critical
+      receiver: 'critical'
+      group_wait: 10s
+      repeat_interval: 1h
+
+    - matchers:
+        - team = revenue
+      receiver: 'revenue-team'
+
+    - matchers:
+        - team = ai-platform
+      receiver: 'ai-team'
+
+receivers:
+  - name: 'default'
+    email_configs:
+      - to: 'office@clossers.com'
+        send_resolved: true
+        headers:
+          Subject: '[DiDi] {{ .Status | toUpper }}: {{ .GroupLabels.alertname }}'
+
+  - name: 'critical'
+    email_configs:
+      - to: 'office@clossers.com'
+        send_resolved: true
+        headers:
+          Subject: '[DiDi CRITICAL] {{ .GroupLabels.alertname }}'
+    # Add webhook for Slack/Teams when ready:
+    # webhook_configs:
+    #   - url: 'https://hooks.slack.com/services/T.../B.../...'
+
+  - name: 'revenue-team'
+    email_configs:
+      - to: 'office@clossers.com'
+
+  - name: 'ai-team'
+    email_configs:
+      - to: 'office@clossers.com'
+
+inhibit_rules:
+  # Suppress non-critical alerts if a critical alert is already firing for same service
+  - source_matchers:
+      - severity = critical
+    target_matchers:
+      - severity = warning
+    equal: ['alertname', 'cluster', 'service']
--- a/backend/observability/docker-compose.yml
+++ b/backend/observability/docker-compose.yml
@ -0,0 +1,183 @@
+# DiDi Observability Stack
+# ========================
+# Implements LOT 2 modul 8 (Observabilitate & Logging) conform oferta EVOTECH §A.9:
+#   - Prometheus     — metrics scraping
+#   - Grafana        — dashboards
+#   - Loki           — log aggregation
+#   - Promtail       — log shipper (Docker logs → Loki)
+#   - Jaeger         — distributed tracing UI
+#   - OTel Collector — receives traces from services, forwards to Jaeger
+#   - Alertmanager   — alert routing (email, Slack/Teams, PagerDuty)
+#
+# Network: didi-network (shared cu DIDI + AI platform stacks)
+# Storage: volumes locale persistente
+#
+# Quick start:
+#   cd /home/admin365/didi_mono/didi_mono/backend/observability
+#   docker compose up -d
+#
+# UI access:
+#   Grafana:       http://10.11.10.12:3030  (admin / GRAFANA_ADMIN_PASSWORD)
+#   Prometheus:    http://10.11.10.12:9090
+#   Jaeger UI:     http://10.11.10.12:16686
+#   Alertmanager:  http://10.11.10.12:9093
+
+services:
+  prometheus:
+    image: prom/prometheus:v2.55.0
+    container_name: didi-prometheus
+    restart: unless-stopped
+    user: "65534:65534"  # nobody:nobody (avoids root warnings)
+    volumes:
+      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro
+      - ./prometheus/rules:/etc/prometheus/rules:ro
+      - prometheus-data:/prometheus
+    command:
+      - '--config.file=/etc/prometheus/prometheus.yml'
+      - '--storage.tsdb.path=/prometheus'
+      - '--storage.tsdb.retention.time=30d'
+      - '--storage.tsdb.retention.size=20GB'
+      - '--web.enable-lifecycle'
+      - '--web.enable-admin-api'
+    ports:
+      - "0.0.0.0:9090:9090"
+    networks:
+      - didi-network
+    healthcheck:
+      test: ["CMD", "wget", "--quiet", "--tries=1", "--spider", "http://localhost:9090/-/healthy"]
+      interval: 30s
+      timeout: 5s
+      retries: 3
+
+  grafana:
+    image: grafana/grafana:11.3.0
+    container_name: didi-grafana
+    restart: unless-stopped
+    depends_on:
+      prometheus:
+        condition: service_healthy
+    environment:
+      GF_SECURITY_ADMIN_PASSWORD: ${GRAFANA_ADMIN_PASSWORD:-admin}
+      GF_USERS_ALLOW_SIGN_UP: "false"
+      GF_AUTH_ANONYMOUS_ENABLED: "false"
+      GF_INSTALL_PLUGINS: "grafana-piechart-panel,grafana-clock-panel"
+      GF_FEATURE_TOGGLES_ENABLE: "traceqlEditor"
+    volumes:
+      - ./grafana/provisioning:/etc/grafana/provisioning:ro
+      - ./grafana/dashboards:/var/lib/grafana/dashboards:ro
+      - grafana-data:/var/lib/grafana
+    ports:
+      - "0.0.0.0:3030:3000"
+    networks:
+      - didi-network
+    healthcheck:
+      test: ["CMD-SHELL", "wget --quiet --tries=1 --spider http://localhost:3000/api/health || exit 1"]
+      interval: 30s
+      timeout: 5s
+      retries: 3
+
+  loki:
+    image: grafana/loki:3.2.0
+    container_name: didi-loki
+    restart: unless-stopped
+    volumes:
+      - ./loki/loki-config.yaml:/etc/loki/local-config.yaml:ro
+      - loki-data:/loki
+    command: -config.file=/etc/loki/local-config.yaml
+    ports:
+      - "0.0.0.0:3100:3100"
+    networks:
+      - didi-network
+    healthcheck:
+      test: ["CMD-SHELL", "wget --quiet --tries=1 --spider http://localhost:3100/ready || exit 1"]
+      interval: 30s
+      timeout: 5s
+      retries: 3
+      start_period: 30s
+
+  promtail:
+    image: grafana/promtail:3.2.0
+    container_name: didi-promtail
+    restart: unless-stopped
+    depends_on:
+      loki:
+        condition: service_healthy
+    volumes:
+      - ./promtail/promtail-config.yaml:/etc/promtail/config.yaml:ro
+      - /var/lib/docker/containers:/var/lib/docker/containers:ro
+      - /var/run/docker.sock:/var/run/docker.sock:ro
+      - promtail-data:/tmp
+    command: -config.file=/etc/promtail/config.yaml
+    networks:
+      - didi-network
+
+  jaeger:
+    image: jaegertracing/all-in-one:1.62.0
+    container_name: didi-jaeger
+    restart: unless-stopped
+    environment:
+      COLLECTOR_OTLP_ENABLED: "true"
+      SPAN_STORAGE_TYPE: memory
+      BADGER_EPHEMERAL: "false"
+      BADGER_DIRECTORY_VALUE: /badger/data
+      BADGER_DIRECTORY_KEY: /badger/key
+    volumes:
+      - jaeger-data:/badger
+    ports:
+      - "0.0.0.0:16686:16686"   # Jaeger UI
+      - "0.0.0.0:14250:14250"   # gRPC collector (legacy)
+      - "0.0.0.0:14268:14268"   # HTTP collector (legacy)
+      - "0.0.0.0:4317:4317"     # OTLP gRPC
+      - "0.0.0.0:4318:4318"     # OTLP HTTP
+    networks:
+      - didi-network
+
+  otel-collector:
+    image: otel/opentelemetry-collector-contrib:0.110.0
+    container_name: didi-otel-collector
+    restart: unless-stopped
+    depends_on:
+      - jaeger
+      - prometheus
+    volumes:
+      - ./otel-collector/otel-collector.yaml:/etc/otelcol-contrib/config.yaml:ro
+    command: ["--config=/etc/otelcol-contrib/config.yaml"]
+    ports:
+      - "0.0.0.0:4319:4317"   # OTLP gRPC (services push traces here)
+      - "0.0.0.0:4320:4318"   # OTLP HTTP
+      - "0.0.0.0:8888:8888"   # collector self-metrics
+      - "0.0.0.0:8889:8889"   # prometheus exporter
+    networks:
+      - didi-network
+
+  alertmanager:
+    image: prom/alertmanager:v0.27.0
+    container_name: didi-alertmanager
+    restart: unless-stopped
+    volumes:
+      - ./alertmanager/alertmanager.yml:/etc/alertmanager/alertmanager.yml:ro
+      - alertmanager-data:/alertmanager
+    command:
+      - '--config.file=/etc/alertmanager/alertmanager.yml'
+      - '--storage.path=/alertmanager'
+    ports:
+      - "0.0.0.0:9093:9093"
+    networks:
+      - didi-network
+    healthcheck:
+      test: ["CMD-SHELL", "wget --quiet --tries=1 --spider http://localhost:9093/-/healthy || exit 1"]
+      interval: 30s
+      timeout: 5s
+      retries: 3
+
+volumes:
+  prometheus-data: {}
+  grafana-data: {}
+  loki-data: {}
+  promtail-data: {}
+  jaeger-data: {}
+  alertmanager-data: {}
+
+networks:
+  didi-network:
+    external: true
--- a/backend/observability/grafana/dashboards/didi-overview.json
+++ b/backend/observability/grafana/dashboards/didi-overview.json
@ -0,0 +1,105 @@
+{
+  "annotations": { "list": [ { "builtIn": 1, "datasource": { "type": "grafana", "uid": "-- Grafana --" }, "enable": true, "hide": true, "iconColor": "rgba(0, 211, 255, 1)", "name": "Annotations & Alerts", "type": "dashboard" } ] },
+  "description": "DiDi Platform — overall health, request rates, latency, errors, infrastructure",
+  "editable": true,
+  "graphTooltip": 1,
+  "id": null,
+  "panels": [
+    { "type": "row", "title": "Service Health", "gridPos": { "h": 1, "w": 24, "x": 0, "y": 0 }, "id": 100, "collapsed": false },
+    {
+      "type": "stat", "title": "Services UP", "id": 1,
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "gridPos": { "h": 4, "w": 4, "x": 0, "y": 1 },
+      "targets": [ { "expr": "count(up == 1)", "refId": "A" } ],
+      "fieldConfig": { "defaults": { "color": { "mode": "thresholds" }, "thresholds": { "mode": "absolute", "steps": [ { "color": "red", "value": null }, { "color": "green", "value": 5 } ] } } },
+      "options": { "colorMode": "background", "graphMode": "area", "reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false }, "textMode": "auto" }
+    },
+    {
+      "type": "stat", "title": "Services DOWN", "id": 2,
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "gridPos": { "h": 4, "w": 4, "x": 4, "y": 1 },
+      "targets": [ { "expr": "count(up == 0)", "refId": "A" } ],
+      "fieldConfig": { "defaults": { "color": { "mode": "thresholds" }, "thresholds": { "mode": "absolute", "steps": [ { "color": "green", "value": null }, { "color": "red", "value": 1 } ] } } },
+      "options": { "colorMode": "background", "graphMode": "none", "reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false }, "textMode": "auto" }
+    },
+    {
+      "type": "stat", "title": "HTTP req/sec (total)", "id": 3,
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "gridPos": { "h": 4, "w": 8, "x": 8, "y": 1 },
+      "targets": [ { "expr": "sum(rate(http_requests_total[5m]))", "refId": "A" } ],
+      "fieldConfig": { "defaults": { "unit": "reqps", "color": { "mode": "thresholds" } } },
+      "options": { "colorMode": "value", "graphMode": "area", "reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false } }
+    },
+    {
+      "type": "stat", "title": "Active Workers (agent-v3)", "id": 4,
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "gridPos": { "h": 4, "w": 8, "x": 16, "y": 1 },
+      "targets": [ { "expr": "count(up{job=\"agent-v3-workers\"} == 1)", "refId": "A" } ],
+      "fieldConfig": { "defaults": { "color": { "mode": "thresholds" }, "thresholds": { "mode": "absolute", "steps": [ { "color": "red", "value": null }, { "color": "green", "value": 10 } ] } } },
+      "options": { "colorMode": "background", "graphMode": "none", "reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false } }
+    },
+    { "type": "row", "title": "HTTP traffic", "gridPos": { "h": 1, "w": 24, "x": 0, "y": 5 }, "id": 101, "collapsed": false },
+    {
+      "type": "timeseries", "title": "Request rate per service", "id": 10,
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 6 },
+      "targets": [ { "expr": "sum by (service) (rate(http_requests_total[1m]))", "refId": "A", "legendFormat": "{{service}}" } ],
+      "fieldConfig": { "defaults": { "unit": "reqps" } },
+      "options": { "legend": { "displayMode": "table", "placement": "bottom", "calcs": ["mean", "max"] } }
+    },
+    {
+      "type": "timeseries", "title": "p95 latency per service (s)", "id": 11,
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 6 },
+      "targets": [ { "expr": "histogram_quantile(0.95, sum by (service, le) (rate(http_request_duration_seconds_bucket[5m])))", "refId": "A", "legendFormat": "{{service}}" } ],
+      "fieldConfig": { "defaults": { "unit": "s" } },
+      "options": { "legend": { "displayMode": "table", "placement": "bottom", "calcs": ["mean", "max"] } }
+    },
+    {
+      "type": "timeseries", "title": "5xx error rate", "id": 12,
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 14 },
+      "targets": [ { "expr": "sum by (service) (rate(http_requests_total{status_code=~\"5..\"}[5m]))", "refId": "A", "legendFormat": "{{service}}" } ],
+      "fieldConfig": { "defaults": { "unit": "reqps", "color": { "mode": "palette-classic" } } }
+    },
+    {
+      "type": "timeseries", "title": "Process RSS memory (MB)", "id": 13,
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 14 },
+      "targets": [ { "expr": "process_resident_memory_bytes / 1024 / 1024", "refId": "A", "legendFormat": "{{service}}" } ],
+      "fieldConfig": { "defaults": { "unit": "MB" } }
+    },
+    { "type": "row", "title": "Infrastructure", "gridPos": { "h": 1, "w": 24, "x": 0, "y": 22 }, "id": 103, "collapsed": false },
+    {
+      "type": "timeseries", "title": "CPU per container", "id": 30,
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 23 },
+      "targets": [ { "expr": "sum by (name) (rate(container_cpu_usage_seconds_total{name=~\"didi.*|agent-v3.*\"}[1m]))", "refId": "A", "legendFormat": "{{name}}" } ],
+      "fieldConfig": { "defaults": { "unit": "percentunit" } },
+      "options": { "legend": { "displayMode": "table", "placement": "bottom" } }
+    },
+    {
+      "type": "timeseries", "title": "Memory per container (MB)", "id": 31,
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 23 },
+      "targets": [ { "expr": "container_memory_usage_bytes{name=~\"didi.*|agent-v3.*\"} / 1024 / 1024", "refId": "A", "legendFormat": "{{name}}" } ],
+      "fieldConfig": { "defaults": { "unit": "MB" } }
+    },
+    {
+      "type": "timeseries", "title": "RabbitMQ queue depth", "id": 40,
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "gridPos": { "h": 8, "w": 24, "x": 0, "y": 31 },
+      "targets": [ { "expr": "rabbitmq_queue_messages", "refId": "A", "legendFormat": "{{queue}}" } ],
+      "options": { "legend": { "displayMode": "table", "placement": "bottom" } }
+    }
+  ],
+  "refresh": "30s",
+  "schemaVersion": 39,
+  "tags": ["didi", "overview"],
+  "templating": { "list": [] },
+  "time": { "from": "now-30m", "to": "now" },
+  "timezone": "Europe/Bucharest",
+  "title": "DiDi Platform Overview",
+  "uid": "didi-overview",
+  "version": 1
+}
--- a/backend/observability/grafana/provisioning/dashboards/dashboards.yml
+++ b/backend/observability/grafana/provisioning/dashboards/dashboards.yml
@ -0,0 +1,13 @@
+apiVersion: 1
+
+providers:
+  - name: 'DiDi Platform Dashboards'
+    orgId: 1
+    folder: 'DiDi'
+    type: file
+    disableDeletion: false
+    updateIntervalSeconds: 30
+    allowUiUpdates: true
+    options:
+      path: /var/lib/grafana/dashboards
+      foldersFromFilesStructure: false
--- a/backend/observability/grafana/provisioning/datasources/datasources.yml
+++ b/backend/observability/grafana/provisioning/datasources/datasources.yml
@ -0,0 +1,40 @@
+apiVersion: 1
+
+datasources:
+  - name: Prometheus
+    type: prometheus
+    access: proxy
+    url: http://didi-prometheus:9090
+    isDefault: true
+    editable: false
+    jsonData:
+      httpMethod: POST
+      timeInterval: 30s
+
+  - name: Loki
+    type: loki
+    access: proxy
+    url: http://didi-loki:3100
+    editable: false
+    jsonData:
+      derivedFields:
+        - name: TraceID
+          matcherRegex: 'trace_id=(\w+)'
+          url: '$${__value.raw}'
+          datasourceUid: jaeger
+
+  - name: Jaeger
+    type: jaeger
+    uid: jaeger
+    access: proxy
+    url: http://didi-jaeger:16686
+    editable: false
+    jsonData:
+      tracesToLogsV2:
+        datasourceUid: loki
+        tags: ['service.name']
+        spanStartTimeShift: '-1h'
+        spanEndTimeShift: '1h'
+      tracesToMetrics:
+        datasourceUid: prometheus
+        tags: [{ key: 'service.name', value: 'service' }]
--- a/backend/observability/loki/loki-config.yaml
+++ b/backend/observability/loki/loki-config.yaml
@ -0,0 +1,46 @@
+auth_enabled: false
+
+server:
+  http_listen_port: 3100
+  grpc_listen_port: 9096
+  log_level: info
+
+common:
+  instance_addr: 127.0.0.1
+  path_prefix: /loki
+  storage:
+    filesystem:
+      chunks_directory: /loki/chunks
+      rules_directory: /loki/rules
+  replication_factor: 1
+  ring:
+    kvstore:
+      store: inmemory
+
+query_range:
+  results_cache:
+    cache:
+      embedded_cache:
+        enabled: true
+        max_size_mb: 100
+
+schema_config:
+  configs:
+    - from: 2024-01-01
+      store: tsdb
+      object_store: filesystem
+      schema: v13
+      index:
+        prefix: index_
+        period: 24h
+
+ruler:
+  alertmanager_url: http://didi-alertmanager:9093
+
+limits_config:
+  retention_period: 168h   # 7 days (logs)
+  reject_old_samples: true
+  reject_old_samples_max_age: 168h
+  ingestion_rate_mb: 16
+  ingestion_burst_size_mb: 32
+  max_query_series: 10000
--- a/backend/observability/otel-collector/otel-collector.yaml
+++ b/backend/observability/otel-collector/otel-collector.yaml
@ -0,0 +1,77 @@
+receivers:
+  otlp:
+    protocols:
+      grpc:
+        endpoint: 0.0.0.0:4317
+      http:
+        endpoint: 0.0.0.0:4318
+
+processors:
+  batch:
+    timeout: 1s
+    send_batch_size: 1024
+
+  memory_limiter:
+    check_interval: 5s
+    limit_mib: 512
+
+  resource:
+    attributes:
+      - key: cluster
+        value: didi-prod
+        action: insert
+      - key: environment
+        value: production
+        action: insert
+
+  tail_sampling:
+    decision_wait: 10s
+    num_traces: 100
+    expected_new_traces_per_sec: 10
+    policies:
+      - name: errors-policy
+        type: status_code
+        status_code: { status_codes: [ERROR] }
+      - name: slow-traces-policy
+        type: latency
+        latency: { threshold_ms: 1000 }
+      - name: random-sampling
+        type: probabilistic
+        probabilistic: { sampling_percentage: 10 }
+
+exporters:
+  # Jaeger receives via OTLP natively
+  otlp/jaeger:
+    endpoint: didi-jaeger:4317
+    tls:
+      insecure: true
+
+  # Prometheus exporter for service metrics derived from spans
+  prometheus:
+    endpoint: 0.0.0.0:8889
+    namespace: otel
+    const_labels:
+      cluster: didi-prod
+
+  # Log to stdout for debugging (disable in prod)
+  debug:
+    verbosity: basic
+
+extensions:
+  health_check:
+    endpoint: 0.0.0.0:13133
+
+service:
+  extensions: [health_check]
+  pipelines:
+    traces:
+      receivers: [otlp]
+      processors: [memory_limiter, resource, tail_sampling, batch]
+      exporters: [otlp/jaeger]
+    metrics:
+      receivers: [otlp]
+      processors: [memory_limiter, resource, batch]
+      exporters: [prometheus]
+  telemetry:
+    metrics:
+      address: 0.0.0.0:8888
--- a/backend/observability/prometheus/prometheus.yml
+++ b/backend/observability/prometheus/prometheus.yml
@ -0,0 +1,104 @@
+global:
+  scrape_interval: 30s
+  evaluation_interval: 30s
+  external_labels:
+    cluster: didi-prod
+    environment: production
+
+rule_files:
+  - /etc/prometheus/rules/*.yml
+
+alerting:
+  alertmanagers:
+    - static_configs:
+        - targets:
+            - didi-alertmanager:9093
+
+scrape_configs:
+  # Prometheus self-monitoring
+  - job_name: prometheus
+    static_configs:
+      - targets: ['localhost:9090']
+
+  # Backend Node.js services (expun /metrics via prom-client)
+  - job_name: agent-v3
+    metrics_path: /metrics
+    static_configs:
+      - targets: ['didi-agent-v3:24803']
+        labels: { service: agent-v3, layer: orchestration }
+
+  - job_name: didi-framework
+    metrics_path: /metrics
+    static_configs:
+      - targets: ['didi-framework:3005']
+        labels: { service: framework, layer: orchestration }
+
+  # AI Platform Python services (expun /metrics via prometheus_client)
+  - job_name: ai-platform
+    metrics_path: /metrics
+    static_configs:
+      - targets:
+          - 'didiAI-llm-api:14011'
+          - 'didiAI-embeddings-api:14100'
+          - 'didiAI-rerank-api:14200'
+          - 'didiAI-audio-api:54300'
+          - 'didiAI-video-api:54600'
+          - 'didiAI-web-api:51100'
+          - 'didiAI-catalog-api:11000'
+          - 'didiAI-dashboard:51300'
+          - 'didibrain-api:8090'
+        labels: { layer: ai-platform }
+
+  # Workers (expun /metrics pe portul intern)
+  - job_name: agent-v3-workers
+    metrics_path: /metrics
+    static_configs:
+      - targets:
+          - 'agent-v3-worker-techniques-1:24803'
+          - 'agent-v3-worker-techniques-2:24803'
+          - 'agent-v3-worker-ai-tampered-1:24803'
+          - 'agent-v3-worker-ai-tampered-2:24803'
+          - 'agent-v3-worker-claims-1:24803'
+          - 'agent-v3-worker-claims-2:24803'
+          - 'agent-v3-worker-claims-3:24803'
+          - 'agent-v3-worker-domain-1:24803'
+          - 'agent-v3-worker-domain-2:24803'
+          - 'agent-v3-worker-media-preprocess-1:24803'
+          - 'agent-v3-worker-media-preprocess-2:24803'
+          - 'agent-v3-verdict-aggregator-1:24803'
+          - 'agent-v3-verdict-aggregator-2:24803'
+        labels: { service: agent-v3, layer: workers }
+
+  # Infrastructure metrics (cadvisor + node-exporter already running on host)
+  - job_name: cadvisor
+    static_configs:
+      - targets: ['heimdall_cadvisor:8080']
+        labels: { layer: infrastructure }
+
+  - job_name: node-exporter
+    static_configs:
+      - targets: ['heimdall_node_exporter:9100']
+        labels: { layer: infrastructure }
+
+  # OTel Collector self-metrics + traces (metric forwarder)
+  - job_name: otel-collector
+    static_configs:
+      - targets: ['didi-otel-collector:8889']
+        labels: { layer: observability }
+
+  # PostgreSQL via postgres_exporter (opt-in — add postgres-exporter container if needed)
+  # - job_name: postgres-exporter
+  #   static_configs:
+  #     - targets: ['postgres-exporter:9187']
+
+  # RabbitMQ has built-in prometheus support since 3.10 — enable rabbitmq_prometheus plugin
+  - job_name: rabbitmq
+    metrics_path: /metrics
+    static_configs:
+      - targets: ['staging-dataLayer-rabbitmq:15692']
+        labels: { service: rabbitmq, layer: data }
+
+  # Redis via redis_exporter (opt-in)
+  # - job_name: redis-exporter
+  #   static_configs:
+  #     - targets: ['redis-exporter:9121']
--- a/backend/observability/prometheus/rules/alerts.yml
+++ b/backend/observability/prometheus/rules/alerts.yml
@ -0,0 +1,119 @@
+groups:
+  - name: didi-platform-availability
+    interval: 1m
+    rules:
+      - alert: ServiceDown
+        expr: up == 0
+        for: 5m
+        labels:
+          severity: critical
+          team: platform
+        annotations:
+          summary: "Service {{ $labels.job }} ({{ $labels.instance }}) is DOWN"
+          description: "{{ $labels.job }} has been unreachable for >5m. Last scrape: {{ $value }}"
+
+      - alert: HighErrorRate
+        expr: |
+          sum(rate(didi_http_requests_total{status=~"5.."}[5m])) by (service)
+            /
+          sum(rate(didi_http_requests_total[5m])) by (service)
+          > 0.05
+        for: 10m
+        labels:
+          severity: warning
+          team: platform
+        annotations:
+          summary: "High 5xx error rate on {{ $labels.service }}"
+          description: "{{ $labels.service }} has >5% 5xx errors for >10m"
+
+  - name: didi-pipeline-performance
+    interval: 1m
+    rules:
+      - alert: PipelineLatencyHigh
+        expr: |
+          histogram_quantile(0.95,
+            sum(rate(didi_pipeline_duration_seconds_bucket[5m])) by (le, component)
+          ) > 60
+        for: 15m
+        labels:
+          severity: warning
+          team: platform
+        annotations:
+          summary: "Pipeline component {{ $labels.component }} P95 latency >60s"
+          description: "P95 latency on {{ $labels.component }} is {{ $value }}s"
+
+      - alert: RabbitMQQueueBacklog
+        expr: rabbitmq_queue_messages_ready > 100
+        for: 10m
+        labels:
+          severity: warning
+          team: platform
+        annotations:
+          summary: "Queue {{ $labels.queue }} backlog >100 messages"
+          description: "{{ $labels.queue }} has {{ $value }} unprocessed messages"
+
+      - alert: DeadLetterMessages
+        expr: increase(didi_dlq_messages_total[15m]) > 0
+        labels:
+          severity: warning
+          team: platform
+        annotations:
+          summary: "Messages landing in analysis_dlq ({{ $labels.component }})"
+          description: "{{ $value }} dead-lettered messages in 15m — check didi:queue:dlq:recent in Redis for session ids"
+
+  - name: didi-infrastructure
+    interval: 1m
+    rules:
+      - alert: HighCPU
+        expr: 100 - (avg by(instance) (irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 85
+        for: 10m
+        labels:
+          severity: warning
+          team: ops
+        annotations:
+          summary: "CPU >85% on {{ $labels.instance }}"
+
+      - alert: LowDiskSpace
+        expr: |
+          (node_filesystem_avail_bytes{mountpoint="/"}
+           / node_filesystem_size_bytes{mountpoint="/"})
+          < 0.15
+        for: 5m
+        labels:
+          severity: critical
+          team: ops
+        annotations:
+          summary: "Disk space <15% on {{ $labels.instance }}"
+
+      - alert: GPUMemoryHigh
+        expr: nvidia_gpu_memory_used_bytes / nvidia_gpu_memory_total_bytes > 0.95
+        for: 10m
+        labels:
+          severity: warning
+          team: ai-platform
+        annotations:
+          summary: "GPU {{ $labels.gpu }} memory >95% on {{ $labels.instance }}"
+
+  - name: didi-business
+    interval: 5m
+    rules:
+      - alert: NoAnalysesIn30Min
+        expr: |
+          increase(didi_analyses_completed_total[30m]) == 0
+        for: 30m
+        labels:
+          severity: warning
+          team: product
+        annotations:
+          summary: "No analyses completed in last 30 minutes"
+          description: "Platform may be down or no traffic — investigate"
+
+      - alert: StripeWebhookFailures
+        expr: |
+          rate(didi_stripe_webhook_failures_total[5m]) > 0.1
+        for: 5m
+        labels:
+          severity: critical
+          team: revenue
+        annotations:
+          summary: "Stripe webhook failures detected"
--- a/backend/observability/promtail/promtail-config.yaml
+++ b/backend/observability/promtail/promtail-config.yaml
@ -0,0 +1,44 @@
+server:
+  http_listen_port: 9080
+  grpc_listen_port: 0
+
+positions:
+  filename: /tmp/positions.yaml
+
+clients:
+  - url: http://didi-loki:3100/loki/api/v1/push
+
+scrape_configs:
+  # Scrape Docker container logs
+  - job_name: docker
+    docker_sd_configs:
+      - host: unix:///var/run/docker.sock
+        refresh_interval: 30s
+        filters:
+          - name: label
+            values: ["com.docker.compose.project"]
+    relabel_configs:
+      - source_labels: ['__meta_docker_container_name']
+        regex: '/(.*)'
+        target_label: container
+      - source_labels: ['__meta_docker_container_label_com_docker_compose_service']
+        target_label: service
+      - source_labels: ['__meta_docker_container_label_com_docker_compose_project']
+        target_label: project
+    pipeline_stages:
+      # Try to parse JSON logs (Node.js services emit JSON via pino/winston)
+      - json:
+          expressions:
+            level: level
+            service: service
+            request_id: request_id
+            session_id: session_id
+            msg: msg
+      - labels:
+          level:
+          request_id:
+          session_id:
+      # If "level" is missing, leave as info
+      - template:
+          source: level
+          template: '{{ if .Value }}{{ .Value }}{{ else }}info{{ end }}'