livrare lot 2
This commit is contained in:
commit
8ecc78e729
763 changed files with 164593 additions and 0 deletions
36
backend/services/data-layer/didiCache/.env.example
Normal file
36
backend/services/data-layer/didiCache/.env.example
Normal file
|
|
@ -0,0 +1,36 @@
|
|||
# ============================================================================
|
||||
# didiCache Environment Configuration
|
||||
# ============================================================================
|
||||
# Copy this file to .env and update with your values
|
||||
|
||||
# Docker Compose Project Name (groups containers in Docker Desktop)
|
||||
COMPOSE_PROJECT_NAME=didibackend_datalayer
|
||||
|
||||
# Redis Configuration
|
||||
REDIS_PASSWORD=YOUR_SECURE_PASSWORD_HERE
|
||||
REDIS_PORT=6380 # Using 6380 to avoid conflict with default 6379
|
||||
|
||||
# Memory Limits
|
||||
REDIS_MEMORY_LIMIT=512M
|
||||
REDIS_MEMORY_RESERVATION=256M
|
||||
|
||||
# Performance Tuning
|
||||
REDIS_MAXMEMORY=512mb
|
||||
REDIS_MAXMEMORY_POLICY=allkeys-lru
|
||||
REDIS_TIMEOUT=0
|
||||
REDIS_TCP_KEEPALIVE=300
|
||||
REDIS_DATABASES=16
|
||||
|
||||
# Persistence Configuration
|
||||
REDIS_SAVE="900 1 300 10 60 10000"
|
||||
REDIS_APPENDONLY=yes
|
||||
REDIS_APPENDFSYNC=everysec
|
||||
|
||||
# Logging
|
||||
REDIS_LOGLEVEL=notice
|
||||
|
||||
# Client Limits
|
||||
REDIS_MAXCLIENTS=10000
|
||||
|
||||
# Timezone
|
||||
TZ=UTC
|
||||
26
backend/services/data-layer/didiCache/.gitignore
vendored
Normal file
26
backend/services/data-layer/didiCache/.gitignore
vendored
Normal file
|
|
@ -0,0 +1,26 @@
|
|||
# Environment variables
|
||||
.env
|
||||
.env.local
|
||||
|
||||
# Data directory
|
||||
data/
|
||||
*.rdb
|
||||
*.aof
|
||||
|
||||
# Logs
|
||||
*.log
|
||||
|
||||
# OS files
|
||||
.DS_Store
|
||||
Thumbs.db
|
||||
|
||||
# IDE files
|
||||
.idea/
|
||||
.vscode/
|
||||
*.swp
|
||||
*.swo
|
||||
|
||||
# Backup files
|
||||
*.bak
|
||||
*.backup
|
||||
*.old
|
||||
216
backend/services/data-layer/didiCache/INDEX.md
Normal file
216
backend/services/data-layer/didiCache/INDEX.md
Normal file
|
|
@ -0,0 +1,216 @@
|
|||
# didiCache - Index
|
||||
|
||||
**Productia ruleaza pe Redis LOCAL** (container `didi-cache` pe `didi-network`). Decizie: stabilitate + zero dependinte externe + acces rapid. Clusterul Redis RAG (managed extern, HAProxy VIP peste rag01/02/03) ramane configurat ca **fallback de urgenta pentru HA** — activabil cu `redis-switch.sh cluster`, nu este folosit operational acum.
|
||||
|
||||
**Productie (LOCAL — activ)**:
|
||||
- Container: `didi-cache` (imagine `redis:7-alpine`)
|
||||
- Host intern Docker: `didi-cache:6379`
|
||||
- DB: `0`
|
||||
- Parola: `REDIS_PASSWORD` din `.env` (default `redis123` in dev)
|
||||
- Memorie: 512MB (eviction: allkeys-lru)
|
||||
- Retea: `didi-network`; portul `6379` este expus pe host (`0.0.0.0:6379->6379`)
|
||||
- Persistenta: RDB + AOF
|
||||
|
||||
**Fallback HA (cluster RAG — disponibil dar inactiv)**:
|
||||
- Host: `10.11.50.100` (HAProxy VIP)
|
||||
- Port: `16379`
|
||||
- DB: `0`
|
||||
- User: `didi`
|
||||
- Parola: din `.cluster-credentials.env` (gitignored)
|
||||
|
||||
**Switch intre LOCAL si cluster**: `backend/services/orchestration-layer/scripts/redis-switch.sh {cluster|local|status}` rescrie fisierele `.env` ale serviciilor (agent-v3, didiFramework) + restart containere. Status curent: `local`.
|
||||
|
||||
**Helper de conectare folosit de toate serviciile**:
|
||||
- agent-v3 -> `src/shared/redis/connection.ts` (`createRedisConnection()`)
|
||||
- didiFramework -> `src/config/redis.ts` (`createRedisConnection()`)
|
||||
|
||||
---
|
||||
|
||||
## Ce face
|
||||
|
||||
Stocheaza date temporare si configurari pentru platforma DIDI:
|
||||
- Cache framework (parametri tehnici, verdicts, ponderi) -- scrise de didiFramework via sync-redis
|
||||
- Configurare HIL Moderation (triage thresholds, brain client config, sensitive topics, roluri) -- scrise de didiFramework
|
||||
- Sesiuni analiza (AnalysisSession JSON, TTL 7 zile) -- scrise de agent-v3
|
||||
- Rezultate intermediare per etapa (TTL 7 zile) -- scrise de agent-v3
|
||||
- Stare coada async (progres workeri) -- scrise de agent-v3
|
||||
- Lock-uri concurenta workeri (TTL 30s-5min) -- scrise de agent-v3
|
||||
- Chei API extensie browser (cache validare) -- scrise de didiFramework
|
||||
- Configurare modele viziune -- scrise de didiFramework
|
||||
|
||||
**Sync-redis scrie ~51 chei de configurare** (crescut de la 48 dupa adaugarea cheilor HIL Moderation in 2026-05-01). Plus `framework_keys`: 7 chei standard sau 8 daca include `providers`.
|
||||
|
||||
---
|
||||
|
||||
## Cine scrie in Redis
|
||||
|
||||
| Serviciu | Ce scrie | Chei Redis |
|
||||
|----------|----------|------------|
|
||||
| didiFramework (sync-redis) | Ierarhie tehnici, claims, verdicts, ponderi, surse, provideri | didi:framework:* |
|
||||
| didiFramework (sync-redis) | Configurare componente pe etape | didi:config:* |
|
||||
| didiFramework (extension-keys) | Cache validare chei API extensie | didi:extension:key:* |
|
||||
| agent-v3 (sesiuni) | Sesiune completa JSON | didi:pipeline:{sessionId}:status |
|
||||
| agent-v3 (executori) | Rezultate intermediare per etapa | agent:result:{sessionId}:{component}:{stage} |
|
||||
| agent-v3 (coada) | Stare procesare async | didi:queue:session:{sessionId} |
|
||||
| agent-v3 (workeri) | Lock-uri concurenta | didi:queue:lock:* |
|
||||
|
||||
## Cine citeste din Redis
|
||||
|
||||
| Serviciu | Ce citeste | Chei Redis |
|
||||
|----------|-----------|------------|
|
||||
| agent-v3 (executori) | Configurare framework (tehnici, ponderi, verdicts) | didi:framework:* |
|
||||
| agent-v3 (executori) | Modele disponibile, stage assignments, prompturi | didi:config:* |
|
||||
| agent-v3 (rute) | Sesiuni pentru polling status | didi:pipeline:{sessionId}:status |
|
||||
| agent-v3 (rute) | Rezultate intermediare | agent:result:{sessionId}:* |
|
||||
| agent-v3 (pipeline) | Validare cheie extensie | didi:extension:key:* |
|
||||
| didiFramework (debug) | Verificare date sincronizate | didi:framework:* |
|
||||
|
||||
---
|
||||
|
||||
## Chei Redis principale
|
||||
|
||||
| Pattern | Scop | TTL | Scris de |
|
||||
|---------|------|-----|----------|
|
||||
| didi:framework:manifest | Index categorii + timestamp sync | permanent | didiFramework |
|
||||
| didi:framework:techniques | Ierarhie completa tehnici (denormalizata) | permanent | didiFramework |
|
||||
| didi:framework:claims | Parametri claims (tipuri, statusuri, confidence) | permanent | didiFramework |
|
||||
| didi:framework:verdicts | Categorii verdict + risk mappings + severity | permanent | didiFramework |
|
||||
| didi:framework:weights | Ponderi componente + scenarii + multiplicatori | permanent | didiFramework |
|
||||
| didi:framework:sources | Evaluare surse | permanent | didiFramework |
|
||||
| didi:framework:providers | Configurare LLM | permanent | didiFramework |
|
||||
| didi:framework:dimensions_compact | Lista compacta dimensiuni (screening) | permanent | didiFramework |
|
||||
| didi:config:{component}:v1:* | Config componenta (modele, etape, prompturi) | permanent | didiFramework |
|
||||
| didi:config:techniques:v3:stage_assignments | **TIER-NESTED** `{stage: {free: {models}, premium: {models}}}` — chain-uri LLM per tier | permanent | didiFramework |
|
||||
| didi:config:ai-tampered:v1:stage_assignments | **TIER-NESTED** stage assignments AI-Tampered | permanent | didiFramework |
|
||||
| didi:config:claims:v1:stage_assignments | **TIER-NESTED** stage assignments Claims | permanent | didiFramework |
|
||||
| didi:config:source-assessment:v1:stage_assignments | **TIER-NESTED** stage assignments Source Assessment | permanent | didiFramework |
|
||||
| didi:config:vision:v1:stage_assignments | **TIER-NESTED** (Etapa 4) stage `image_analysis` — citit de `callVision()` cu tier param | permanent | didiFramework |
|
||||
| didi:config:verdict:v1:stage_assignments | **TIER-NESTED** (Etapa 5) stage `verdict_review` — citit de `verdict-explanation.ts loadModels(tier)` | permanent | didiFramework |
|
||||
| didi:config:ai-tampered:v1:vision_models | Legacy flat vision config (fallback pentru vision.ts daca `didi:config:vision:v1:stage_assignments` lipseste) | permanent | didiFramework |
|
||||
| didi:config:vision:v1:prompts:extraction | Prompt viziune: extragere text din imagini | permanent | didiFramework |
|
||||
| didi:config:vision:v1:prompts:video_frames | Prompt viziune: analiza cadre video | permanent | didiFramework |
|
||||
| didi:config:vision:v1:prompts:ai_detection | Prompt viziune: detectie AI imagini | permanent | didiFramework |
|
||||
| didi:config:pipeline:v1:component_config | Config componente pipeline + video track weights | permanent | didiFramework |
|
||||
| didi:config:pipeline:v1:session_config | Config sesiune pipeline | permanent | didiFramework |
|
||||
| didi:config:pipeline:v1:verdict_config | Override-uri verdict, synergy, confidence (globale). JSONB sincronizat din PG `bos_parammgmt.component_config WHERE (component_code='pipeline', config_key='verdict_config')`. Contine: `synergy` + override-uri (`false_claims`, `severe_techniques`, `undisclosed_ai`, `untrusted_domain`, `domain_red_flags`) + `confidence` + `confidence_levels`. Editabil din admin dashboard via `/api/verdicts/runtime-config` (GET/PUT/PATCH). | permanent | didiFramework |
|
||||
| didi:config:pipeline:v1:input_profiles | **Profiluri verdict per input type (6 profile: ponderi, override-uri, INCONCLUSIVE)** | permanent | didiFramework |
|
||||
| didi:config:techniques:v3:scoring_config | Parametri scoring techniques (count_scaler, intensity, severe_threshold) | permanent | didiFramework |
|
||||
| didi:config:ai-tampered:v1:scoring_config | Parametri scoring AI (blend_weights, disclosure_impact, thresholds) | permanent | didiFramework |
|
||||
| didi:config:claims:v1:scoring_config | Parametri scoring claims (status_weights, unverified behavior) | permanent | didiFramework |
|
||||
| didi:config:source-assessment:v1:scoring_config | Parametri scoring source (axis_weights, verdict_thresholds) | permanent | didiFramework |
|
||||
| didi:config:source-assessment:v1:available_models | Modele LLM source assessment (union free+premium) | permanent | didiFramework |
|
||||
| didi:config:source-assessment:v1:prompts:extraction | Prompt extractie metadata sursa | permanent | didiFramework |
|
||||
| didi:config:source-assessment:v1:prompts:evaluation | Prompt evaluare sursa | permanent | didiFramework |
|
||||
| didi:config:verdict:v1:available_models | Legacy fallback pentru verdict reviewer (folosit daca `stage_assignments` lipseste) | permanent | didiFramework |
|
||||
|
||||
### HIL Moderation (synced from PG by sync-redis)
|
||||
|
||||
Adaugat in 2026-05-01. Citit de agent-v3 in `triage.ts` (cache local 60s) si `brain/client.ts` pentru configurare live.
|
||||
|
||||
| Pattern | Scop | TTL | Scris de |
|
||||
|---------|------|-----|----------|
|
||||
| didi:config:moderation:v1:settings | Single-row config: triage thresholds + brain client config (brain_enabled, brain_url, lookup/write timeouts, brain_confidence_min_silver, brain_semantic_threshold, brain_per_component) | permanent | didiFramework |
|
||||
| didi:config:moderation:v1:sensitive_topics | Active topics list (elections, health, war, covid, climate) | permanent | didiFramework |
|
||||
| didi:config:moderation:v1:roles | Keycloak role -> permissions (moderator, senior_moderator) cu toggle flags | permanent | didiFramework |
|
||||
|
||||
**Nota structura tier-nested** (aplicabila tuturor cheilor `stage_assignments`):
|
||||
```json
|
||||
{
|
||||
"techniques_screening": {
|
||||
"free": { "models": [{"order":1,"model_key":"qwen35:Qwen3.5-397B-A17B",...}, ...] },
|
||||
"premium": { "models": [{"order":1,"model_key":"openrouter:google/gemini-3-flash-preview",...}, ...] }
|
||||
},
|
||||
"techniques_deep": { "free": {...}, "premium": {...} }
|
||||
}
|
||||
```
|
||||
|
||||
Sync-redis scrie cheia intr-un singur SET (atomic). agent-v3 citeste cheia o singura data, apoi rezolva tier-ul local cu `stages[stageCode][tier] || stages[stageCode].free`.
|
||||
| didi:pipeline:{sessionId}:status | Status executie pipeline (JSON: PipelineStatus) | 7 zile | agent-v3 |
|
||||
| didi:pipeline:{sessionId}:{component} | Rezultat componenta (JSON: TechniquesResult etc.) | 7 zile | agent-v3 |
|
||||
| didi:pipeline:{sessionId}:verdict | Rezultat verdict final (JSON: VerdictResult) | 7 zile | agent-v3 |
|
||||
| didi:pipeline:history:entry:{sessionId} | Date intrare sesiune (input + metadata) | 7 zile | agent-v3 |
|
||||
| didi:pipeline:history:user:{userId} | Sorted set istoric utilizator (score=timestamp) | 7 zile | agent-v3 |
|
||||
| agent:result:{sessionId}:{comp}:{stage} | Rezultat intermediar etapa | 7 zile | agent-v3 |
|
||||
| didi:queue:session:{sessionId} | Stare coada async | 24 ore | agent-v3 |
|
||||
| didi:queue:lock:{sessionId}:{comp} | Lock worker per componenta | 5 min | agent-v3 |
|
||||
| didi:queue:aggregator:{sessionId} | Lock agregator verdict | 30s | agent-v3 |
|
||||
| didi:extension:key:{key} | Cache cheie API extensie | permanent | didiFramework |
|
||||
|
||||
---
|
||||
|
||||
## Configurare
|
||||
|
||||
### redis.conf
|
||||
|
||||
- Bind: 0.0.0.0 (acces din Docker network)
|
||||
- Protected mode: activat
|
||||
- Max memorie: 512MB
|
||||
- Eviction: allkeys-lru (sterge cele mai vechi chei cand se umple)
|
||||
- Persistenta RDB: snapshot la 60s/10000 keys, 300s/10 keys, 900s/1 key
|
||||
- Persistenta AOF: activat, sync everysec
|
||||
- Max clienti: 10000
|
||||
- 16 baze de date
|
||||
- Slow log: 10ms threshold
|
||||
- Parola: obligatorie (requirepass)
|
||||
|
||||
### Pornire in productie
|
||||
|
||||
Nota: didi-cache NU este definit in data-layer/docker-compose.yml. Containerul Redis este definit in `production/docker-compose.yml` (alaturi de Kong si Keycloak).
|
||||
|
||||
```yaml
|
||||
# din production/docker-compose.yml
|
||||
didi-cache:
|
||||
image: redis:7-alpine
|
||||
command: redis-server --requirepass ${REDIS_PASSWORD} --save 60 1 --save 300 10
|
||||
volumes:
|
||||
- didi-cache-data:/data
|
||||
networks:
|
||||
- didi-network
|
||||
healthcheck:
|
||||
test: redis-cli -a ${REDIS_PASSWORD} ping
|
||||
```
|
||||
|
||||
Nota: redis.conf din directorul didiCache nu este montat in container in productie. Containerul foloseste parametrii din command line.
|
||||
|
||||
### Conexiune
|
||||
|
||||
**Productie (LOCAL — activ)**:
|
||||
- Host: `didi-cache` (Docker DNS pe `didi-network`)
|
||||
- Port: `6379` (expus si pe host: `0.0.0.0:6379->6379`)
|
||||
- DB: `0`
|
||||
- Parola: `redis123` (din `.env`, `REDIS_PASSWORD`)
|
||||
|
||||
**Fallback HA (cluster RAG — disponibil dar inactiv, doar dupa `redis-switch.sh cluster`)**:
|
||||
- Host: `10.11.50.100` (HAProxy VIP rag01/02/03)
|
||||
- Port: `16379`
|
||||
- DB: `0`
|
||||
- Username: `didi`
|
||||
- Parola: din `.cluster-credentials.env` (gitignored)
|
||||
|
||||
### Administrare
|
||||
|
||||
- Conexiune directa: `redis-cli -h didi-cache -a redis123` (sau `127.0.0.1:6379` de pe host).
|
||||
|
||||
---
|
||||
|
||||
## Fisiere in directorul didiCache
|
||||
|
||||
```
|
||||
redis.conf -- Configurare Redis completa (183 linii, nu e montata in productie)
|
||||
.env.example -- Template variabile de mediu
|
||||
.gitignore -- Exclude .env, data/, *.rdb, *.aof
|
||||
README.md -- Documentatie
|
||||
```
|
||||
|
||||
Zero cod custom. Zero module Redis. Zero scripturi. Doar configurare.
|
||||
|
||||
---
|
||||
|
||||
## Ce NU face
|
||||
|
||||
- Nu e Redis Cluster (instanta singulara)
|
||||
- Nu e Redis Sentinel (fara failover automat)
|
||||
- Nu are module custom
|
||||
- Nu are replicare
|
||||
- Nu are ACL (doar autentificare cu parola)
|
||||
- Nu proceseaza nimic -- doar stocheaza si serveste date scrise de alte servicii
|
||||
117
backend/services/data-layer/didiCache/MIGRATION.md
Normal file
117
backend/services/data-layer/didiCache/MIGRATION.md
Normal file
|
|
@ -0,0 +1,117 @@
|
|||
# Redis — migrat pe clusterul RAG (2026-04-22)
|
||||
|
||||
> **TL;DR**: DIDI folosește **clusterul Redis RAG** (3 noduri Sentinel + HAProxy VIP). Containerul local `didi-cache` din `production/docker-compose.yml` e **oprit dar păstrat** ca fallback rapid. Conexiunea e centralizată via `agent-v3/src/shared/redis/connection.ts`.
|
||||
|
||||
---
|
||||
|
||||
## Ce era aici (legacy)
|
||||
|
||||
`didi-cache` — un container Redis 7 Alpine standalone pe `didi-network` Docker. Single-node, fără HA, fără replicare. 512MB max memory, allkeys-lru eviction. Servea toate sesiunile pipeline + cache-ul framework + lock-uri workeri.
|
||||
|
||||
## Ce e acum
|
||||
|
||||
### Cluster Redis RAG (productie)
|
||||
|
||||
| Nod | Hostname | IP | Port Redis | Port Sentinel |
|
||||
|---|---|---|---|---|
|
||||
| rag01 (HAProxy VIP) | `rag01` | `10.11.50.101` | 6379 (replica) | 26379 |
|
||||
| **rag02 (master curent)** | `rag02` | `10.11.50.102` | **6379 (master)** | 26379 |
|
||||
| rag03 | `rag03` | `10.11.50.103` | 6379 (replica) | 26379 |
|
||||
|
||||
**Endpoint-uri pentru aplicații:**
|
||||
|
||||
| Scop | Endpoint |
|
||||
|---|---|
|
||||
| **WRITE** (auto-routing master) | `10.11.50.100:16379` |
|
||||
| READ (replica locală rag01) | `10.11.50.100:6379` |
|
||||
| Sentinel discovery | `10.11.50.100:16380` (master name `ragmaster`) |
|
||||
| HAProxy stats | `http://10.11.50.100:8404/stats` |
|
||||
|
||||
User ACL DIDI: `didi` cu prefix de chei `didi:*` și channels `didi.*`. Parola în vault-ul de credențiale `name='Redis ACL — didi'`.
|
||||
|
||||
ACL persistent via `--aclfile /data/users.acl` pe toate 3 nodurile (fix permanent 2026-04-22).
|
||||
|
||||
### Conexiune centralizată în cod
|
||||
|
||||
`agent-v3/src/shared/redis/connection.ts` — toate conexiunile ioredis trec prin `createRedisConnection(label)`. Zero `new Redis({...})` inline. Auto-reconnect cu retryStrategy, reconnectOnError pentru `READONLY` / `MASTERDOWN` (failover).
|
||||
|
||||
```typescript
|
||||
// Folosire în cod:
|
||||
import { createRedisConnection } from './shared/redis/connection';
|
||||
const redis = createRedisConnection('framework-cache');
|
||||
```
|
||||
|
||||
`didiFramework` are propria implementare în `src/lib/redis.ts`.
|
||||
|
||||
## Containerul local `didi-cache`
|
||||
|
||||
Definit în `production/docker-compose.yml`, configurat să pornească dar **manual oprit** ca parte din migrare. Volumul `didi-production-cache-data` e intact — datele anterioare (snapshot-uri RDB + AOF) sunt păstrate.
|
||||
|
||||
Status curent: `Exited`.
|
||||
|
||||
### De ce e păstrat?
|
||||
|
||||
- **Fallback rapid** dacă cluster RAG e indisponibil (oprit pentru maintenance, network issue, etc.)
|
||||
- Pentru a-l reactiva temporar, există un script switch:
|
||||
```bash
|
||||
backend/services/orchestration-layer/scripts/redis-switch.sh local redis
|
||||
```
|
||||
Asta rescrie `.env` și restart agent-v3 stack pentru a folosi `didi-cache` în loc de cluster.
|
||||
|
||||
## Switch rapid cluster ↔ local
|
||||
|
||||
Script: `backend/services/orchestration-layer/scripts/redis-switch.sh`
|
||||
|
||||
```bash
|
||||
# Folosește local fallback (didi-cache)
|
||||
./redis-switch.sh local redis
|
||||
|
||||
# Folosește clusterul (default)
|
||||
./redis-switch.sh cluster redis
|
||||
|
||||
# Status
|
||||
./redis-switch.sh status redis
|
||||
```
|
||||
|
||||
Credentialele cluster în sidecar `.cluster-credentials.env` (gitignored).
|
||||
|
||||
## Verificare cluster
|
||||
|
||||
```bash
|
||||
# Cu redis-cli din host (default user — pentru debug)
|
||||
redis-cli -h 10.11.50.100 -p 16379 -a <pwd> PING
|
||||
|
||||
# Cu user DIDI
|
||||
redis-cli -h 10.11.50.100 -p 16379 --user didi -a <pwd> ACL WHOAMI
|
||||
|
||||
# Sentinel master discovery
|
||||
redis-cli -h 10.11.50.100 -p 16380 SENTINEL get-master-addr-by-name ragmaster
|
||||
```
|
||||
|
||||
## Bootstrap automat la fresh deploy
|
||||
|
||||
`didiFramework` detectează la pornire dacă `didi:framework:manifest` lipsește în Redis și auto-rulează `POST /api/sync-redis` (56 chei populate în ~700ms). Zero intervenție manuală pe cluster nou.
|
||||
|
||||
## Chei Redis principale (orientativ)
|
||||
|
||||
| Pattern | Scop | TTL |
|
||||
|---|---|---|
|
||||
| `didi:framework:*` | Config framework (tehnici, ponderi, verdicts) | permanent |
|
||||
| `didi:config:{component}:v1:*` | Config componente, stage assignments tier-nested | permanent |
|
||||
| `didi:pipeline:{sessionId}:*` | Sesiuni pipeline (status, rezultate) | 7 zile |
|
||||
| `didi:queue:*` | Lock-uri workeri | 30s-5min |
|
||||
| `agent:result:*` | Rezultate intermediare | 7 zile |
|
||||
| `agent:media:*` | Cache media (transcript, vision) | 1 oră |
|
||||
|
||||
## Linkuri rapide
|
||||
|
||||
- Ghid utilizare cluster: `landingzone/redis-rag/README.md` (repo `git.finesynergy.eu/lucian/landingzone`)
|
||||
- Onboarding ACL user nou: `landingzone/redis-rag/CLAUDE_PROMPT.md`
|
||||
- HAProxy stats: `http://10.11.50.100:8404/stats`
|
||||
|
||||
## Status
|
||||
|
||||
- ✅ Migrare aplicată: 2026-04-22
|
||||
- ✅ User `didi` configurat cu prefix izolat
|
||||
- ✅ Container local păstrat ca fallback (oprit, volum intact)
|
||||
- ✅ Bootstrap framework automat la pornire
|
||||
147
backend/services/data-layer/didiCache/README.md
Normal file
147
backend/services/data-layer/didiCache/README.md
Normal file
|
|
@ -0,0 +1,147 @@
|
|||
# DIDI Cache Service 🚀
|
||||
|
||||
## Super Simple Start Guide ⚡
|
||||
|
||||
### Step 1: Set Your Password
|
||||
```bash
|
||||
# Copy the example file
|
||||
cp .env.example .env
|
||||
|
||||
# Edit .env and change this line:
|
||||
REDIS_PASSWORD=YOUR_SECURE_PASSWORD_HERE
|
||||
```
|
||||
|
||||
### Step 2: Start Redis
|
||||
```bash
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
That's it! Your cache is running! 🎉
|
||||
|
||||
## Check If It's Working ✅
|
||||
|
||||
```bash
|
||||
# See if container is healthy
|
||||
docker ps
|
||||
|
||||
# Look for: didi-cache (healthy)
|
||||
```
|
||||
|
||||
## Connection Info 📡
|
||||
|
||||
- **Host**: localhost
|
||||
- **Port**: 6380 (not 6379 to avoid conflicts!)
|
||||
- **Password**: (what you set in .env)
|
||||
|
||||
## Test the Connection 🔌
|
||||
|
||||
```bash
|
||||
# Connect with Redis CLI
|
||||
docker exec -it didi-cache redis-cli -a YOUR_PASSWORD
|
||||
|
||||
# Test it
|
||||
127.0.0.1:6379> PING
|
||||
PONG
|
||||
|
||||
# Exit
|
||||
127.0.0.1:6379> EXIT
|
||||
```
|
||||
|
||||
## What's Inside? 📦
|
||||
|
||||
This Redis cache is configured for:
|
||||
- ✅ 512MB memory (perfect for status & results)
|
||||
- ✅ Auto-expiry support (services set TTL)
|
||||
- ✅ Persistence enabled (survives restarts)
|
||||
- ✅ Password protected
|
||||
|
||||
### How DIDI Uses Redis 🔄
|
||||
|
||||
```
|
||||
Pipeline runs → Status stored here (24hr TTL)
|
||||
→ Results cached here
|
||||
→ UI polls for updates
|
||||
→ After 24hrs, auto-deleted
|
||||
```
|
||||
|
||||
### Key Patterns We Store 📝
|
||||
- `run:{run_id}` - Pipeline execution status
|
||||
- `node_status:{run_id}` - Each node's progress
|
||||
- `results:{run_id}` - Analysis results
|
||||
|
||||
## Common Commands 🛠️
|
||||
|
||||
```bash
|
||||
# Stop cache
|
||||
docker compose down
|
||||
|
||||
# View logs
|
||||
docker compose logs -f
|
||||
|
||||
# Connect to Redis CLI
|
||||
docker exec -it didi-cache redis-cli -a YOUR_PASSWORD
|
||||
|
||||
# Restart fresh (WARNING: Deletes all cache!)
|
||||
docker compose down -v
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
## Monitor Cache Usage 📊
|
||||
|
||||
```bash
|
||||
# Check memory usage
|
||||
docker exec -it didi-cache redis-cli -a YOUR_PASSWORD INFO memory
|
||||
|
||||
# See all keys
|
||||
docker exec -it didi-cache redis-cli -a YOUR_PASSWORD KEYS "*"
|
||||
|
||||
# Count keys
|
||||
docker exec -it didi-cache redis-cli -a YOUR_PASSWORD DBSIZE
|
||||
```
|
||||
|
||||
## Troubleshooting 🔧
|
||||
|
||||
### Port 6380 already in use?
|
||||
Edit `.env` and change `REDIS_PORT` to something else (like 6381)
|
||||
|
||||
### Can't connect?
|
||||
1. Check container is healthy: `docker ps`
|
||||
2. Verify password in .env
|
||||
3. Make sure you're using port 6380, not 6379
|
||||
|
||||
### Memory full?
|
||||
Redis will auto-delete least recently used keys (LRU policy)
|
||||
|
||||
## Part of Something Bigger 🏗️
|
||||
|
||||
This cache is part of the DIDI Backend data layer:
|
||||
|
||||
```
|
||||
📁 data-layer/
|
||||
├── 📁 didiDatabase/ (PostgreSQL - Done!)
|
||||
├── 📁 didiCache/ (Redis - You are here!)
|
||||
├── 📁 didiQueue/ (RabbitMQ - Next)
|
||||
└── 📁 didiStorage/ (MinIO - Coming soon)
|
||||
```
|
||||
|
||||
## Quick Health Check 🏥
|
||||
|
||||
```bash
|
||||
# Is it running?
|
||||
docker exec -it didi-cache redis-cli -a YOUR_PASSWORD PING
|
||||
|
||||
# Response should be:
|
||||
# PONG
|
||||
```
|
||||
|
||||
## Why Port 6380? 🤔
|
||||
|
||||
The old monolithic setup uses port 6379. We use 6380 to:
|
||||
- Avoid conflicts during migration
|
||||
- Run both services side-by-side
|
||||
- Easy rollback if needed
|
||||
|
||||
---
|
||||
**That's all you need to know! Happy caching! ⚡**
|
||||
|
||||
*Version: 1.0.0 | Redis 7-alpine*
|
||||
183
backend/services/data-layer/didiCache/redis.conf
Normal file
183
backend/services/data-layer/didiCache/redis.conf
Normal file
|
|
@ -0,0 +1,183 @@
|
|||
# ============================================================================
|
||||
# DIDI Cache Service Configuration (Redis 7)
|
||||
# Production-ready configuration for the DIDI Backend platform
|
||||
# ============================================================================
|
||||
|
||||
# ============================================================================
|
||||
# NETWORK & SECURITY
|
||||
# ============================================================================
|
||||
|
||||
# Listen on all interfaces (Docker container)
|
||||
bind 0.0.0.0
|
||||
|
||||
# Enable protected mode
|
||||
protected-mode yes
|
||||
|
||||
# Port
|
||||
port 6379
|
||||
|
||||
# TCP listen() backlog
|
||||
tcp-backlog 511
|
||||
|
||||
# TCP keepalive
|
||||
tcp-keepalive 300
|
||||
|
||||
# Timeout for idle clients (0 to disable)
|
||||
timeout 0
|
||||
|
||||
# ============================================================================
|
||||
# GENERAL
|
||||
# ============================================================================
|
||||
|
||||
# Don't run as daemon (Docker handles this)
|
||||
daemonize no
|
||||
|
||||
# Server verbosity (debug, verbose, notice, warning)
|
||||
loglevel notice
|
||||
|
||||
# Log to stdout for Docker
|
||||
logfile ""
|
||||
|
||||
# Number of databases (we use 0 for main cache)
|
||||
databases 16
|
||||
|
||||
# ============================================================================
|
||||
# MEMORY MANAGEMENT
|
||||
# ============================================================================
|
||||
|
||||
# Maximum memory (adjust based on container limits)
|
||||
maxmemory 512mb
|
||||
|
||||
# Eviction policy when max memory is reached
|
||||
# allkeys-lru: Remove least recently used keys
|
||||
maxmemory-policy allkeys-lru
|
||||
|
||||
# LRU samples for eviction
|
||||
maxmemory-samples 5
|
||||
|
||||
# ============================================================================
|
||||
# PERSISTENCE - RDB (Snapshots)
|
||||
# ============================================================================
|
||||
|
||||
# Save snapshots:
|
||||
# After 900 sec (15 min) if at least 1 key changed
|
||||
save 900 1
|
||||
# After 300 sec (5 min) if at least 10 keys changed
|
||||
save 300 10
|
||||
# After 60 sec if at least 10000 keys changed
|
||||
save 60 10000
|
||||
|
||||
# Error handling for background save
|
||||
stop-writes-on-bgsave-error yes
|
||||
|
||||
# Compress RDB dumps
|
||||
rdbcompression yes
|
||||
|
||||
# Checksum RDB files
|
||||
rdbchecksum yes
|
||||
|
||||
# Filename for RDB
|
||||
dbfilename dump.rdb
|
||||
|
||||
# Directory for RDB and AOF files
|
||||
dir /data
|
||||
|
||||
# ============================================================================
|
||||
# PERSISTENCE - AOF (Append Only File)
|
||||
# ============================================================================
|
||||
|
||||
# Enable AOF
|
||||
appendonly yes
|
||||
|
||||
# AOF filename
|
||||
appendfilename "appendonly.aof"
|
||||
|
||||
# AOF sync policy (everysec = good balance)
|
||||
appendfsync everysec
|
||||
|
||||
# Don't fsync during rewrites
|
||||
no-appendfsync-on-rewrite no
|
||||
|
||||
# Auto rewrite AOF
|
||||
auto-aof-rewrite-percentage 100
|
||||
auto-aof-rewrite-min-size 64mb
|
||||
|
||||
# Load truncated AOF
|
||||
aof-load-truncated yes
|
||||
|
||||
# Use RDB format in AOF for faster loading
|
||||
aof-use-rdb-preamble yes
|
||||
|
||||
# ============================================================================
|
||||
# SLOW LOG
|
||||
# ============================================================================
|
||||
|
||||
# Log queries slower than (microseconds)
|
||||
slowlog-log-slower-than 10000
|
||||
|
||||
# Maximum length of slow log
|
||||
slowlog-max-len 128
|
||||
|
||||
# ============================================================================
|
||||
# LATENCY MONITORING
|
||||
# ============================================================================
|
||||
|
||||
# Latency threshold in milliseconds
|
||||
latency-monitor-threshold 100
|
||||
|
||||
# ============================================================================
|
||||
# CLIENT HANDLING
|
||||
# ============================================================================
|
||||
|
||||
# Maximum number of clients
|
||||
maxclients 10000
|
||||
|
||||
# ============================================================================
|
||||
# ADVANCED CONFIG
|
||||
# ============================================================================
|
||||
|
||||
# Hash tables
|
||||
hash-max-ziplist-entries 512
|
||||
hash-max-ziplist-value 64
|
||||
|
||||
# Lists
|
||||
list-max-ziplist-size -2
|
||||
list-compress-depth 0
|
||||
|
||||
# Sets
|
||||
set-max-intset-entries 512
|
||||
|
||||
# Sorted sets
|
||||
zset-max-ziplist-entries 128
|
||||
zset-max-ziplist-value 64
|
||||
|
||||
# HyperLogLog
|
||||
hll-sparse-max-bytes 3000
|
||||
|
||||
# Streams
|
||||
stream-node-max-bytes 4096
|
||||
stream-node-max-entries 100
|
||||
|
||||
# Active rehashing
|
||||
activerehashing yes
|
||||
|
||||
# Client output buffer limits
|
||||
client-output-buffer-limit normal 0 0 0
|
||||
client-output-buffer-limit replica 256mb 64mb 60
|
||||
client-output-buffer-limit pubsub 32mb 8mb 60
|
||||
|
||||
# Frequency of rehashing the main dictionary
|
||||
hz 10
|
||||
|
||||
# LFU settings
|
||||
lfu-log-factor 10
|
||||
lfu-decay-time 1
|
||||
|
||||
# ============================================================================
|
||||
# DISABLE DANGEROUS COMMANDS (Production)
|
||||
# ============================================================================
|
||||
|
||||
# Uncomment these in production to disable dangerous commands
|
||||
# rename-command FLUSHDB ""
|
||||
# rename-command FLUSHALL ""
|
||||
# rename-command CONFIG ""
|
||||
Loading…
Add table
Add a link
Reference in a new issue