livrare lot 2
This commit is contained in:
commit
8ecc78e729
763 changed files with 164593 additions and 0 deletions
42
backend/production/.env.example
Normal file
42
backend/production/.env.example
Normal file
|
|
@ -0,0 +1,42 @@
|
|||
# =============================================================================
|
||||
# DIDI Platform - Production Environment Configuration
|
||||
# =============================================================================
|
||||
# Kong & Keycloak - using PostgreSQL Cluster
|
||||
# =============================================================================
|
||||
COMPOSE_PROJECT_NAME=didi-production
|
||||
# =============================================================================
|
||||
# PostgreSQL Cluster (Primary Database)
|
||||
# =============================================================================
|
||||
PG_CLUSTER_HOST=10.11.50.167
|
||||
PG_CLUSTER_PORT=5000
|
||||
PG_CLUSTER_USER=bos_interface
|
||||
PG_CLUSTER_PASSWORD=CHANGE_ME
|
||||
# =============================================================================
|
||||
# Kong Configuration
|
||||
# =============================================================================
|
||||
KONG_PG_HOST=10.11.50.167
|
||||
KONG_PG_PORT=5000
|
||||
KONG_PG_USER=kong
|
||||
KONG_PG_PASSWORD=CHANGE_ME
|
||||
KONG_PG_DATABASE=kong_db
|
||||
# =============================================================================
|
||||
# Keycloak Configuration
|
||||
# =============================================================================
|
||||
KC_DB_HOST=10.11.50.167
|
||||
KC_DB_PORT=5000
|
||||
KC_DB_NAME=keycloak_db
|
||||
KC_DB_USER=keycloak
|
||||
KC_DB_PASSWORD=CHANGE_ME
|
||||
KEYCLOAK_ADMIN=CHANGE_ME
|
||||
KEYCLOAK_ADMIN_PASSWORD=CHANGE_ME
|
||||
# =============================================================================
|
||||
# Hostname Configuration
|
||||
# =============================================================================
|
||||
KC_HOSTNAME_URL=https://didi365.eu/auth
|
||||
# =============================================================================
|
||||
# Redis Cache Configuration
|
||||
# =============================================================================
|
||||
REDIS_HOST=didi-cache
|
||||
REDIS_PORT=6379
|
||||
REDIS_PASSWORD=CHANGE_ME
|
||||
REDIS_DB=0
|
||||
584
backend/production/API_TESTING_GUIDE.md
Normal file
584
backend/production/API_TESTING_GUIDE.md
Normal file
|
|
@ -0,0 +1,584 @@
|
|||
# DIDI Platform - Ghid Testare API
|
||||
|
||||
Acest document descrie toate API-urile platformei DIDI, cum se testeaza, ce constrangeri au, si cum se pot simula mai multi utilizatori.
|
||||
|
||||
---
|
||||
|
||||
## Arhitectura pe scurt
|
||||
|
||||
```
|
||||
Client (browser/curl/script)
|
||||
|
|
||||
v
|
||||
Kong API Gateway (port 443, HTTPS)
|
||||
|
|
||||
+-- agent-v3 (port 24803, analiza continut)
|
||||
+-- didiFramework (port 3005, CRUD parametri + utilizatori)
|
||||
+-- admin-dashboard (port 3000, SPA React)
|
||||
```
|
||||
|
||||
In staging, agent-v3 este accesibil si direct pe localhost:24803 (bind 127.0.0.1).
|
||||
Framework-ul nu expune port extern -- accesibil doar prin Docker network sau admin dashboard.
|
||||
|
||||
---
|
||||
|
||||
## Autentificare
|
||||
|
||||
### JWT (Keycloak)
|
||||
|
||||
Platforma foloseste Keycloak pentru autentificare OAuth2/OIDC.
|
||||
|
||||
Obtinere token:
|
||||
|
||||
```
|
||||
POST http://localhost:28000/realms/didi-clients/protocol/openid-connect/token
|
||||
Content-Type: application/x-www-form-urlencoded
|
||||
|
||||
grant_type=password&client_id=didi-web-app&username=EMAIL&password=PAROLA
|
||||
```
|
||||
|
||||
Utilizatori pre-existenti:
|
||||
|
||||
| Email | Parola | Tier | Credite |
|
||||
|---------------------|-------------|------------|---------|
|
||||
| admin@didi.local | admin123 | admin | nelimitat |
|
||||
| demo@didi.local | Demo123! | free | 100 |
|
||||
| free@didi.local | password123 | free | 100 |
|
||||
| paid@didi.local | password123 | paid | 100 |
|
||||
| enterprise@didi.local | password123 | enterprise | nelimitat |
|
||||
|
||||
Exemplu complet cu curl:
|
||||
|
||||
```bash
|
||||
# Pas 1: Obtine token
|
||||
TOKEN=$(curl -s -X POST \
|
||||
"http://localhost:28000/realms/didi-clients/protocol/openid-connect/token" \
|
||||
-H "Content-Type: application/x-www-form-urlencoded" \
|
||||
-d "grant_type=password&client_id=didi-web-app&username=admin@didi.local&password=admin123" \
|
||||
| jq -r '.access_token')
|
||||
|
||||
echo $TOKEN
|
||||
|
||||
# Pas 2: Foloseste token-ul
|
||||
curl -H "Authorization: Bearer $TOKEN" https://localhost:443/api/...
|
||||
```
|
||||
|
||||
Token-ul expira in 10 minute. Refresh la fiecare 30s cu refresh_token.
|
||||
|
||||
### API Key (extensie browser)
|
||||
|
||||
Extensia Chrome foloseste un API key in loc de JWT:
|
||||
|
||||
```
|
||||
POST /api/v3/pipeline/extension/analyze
|
||||
Header: X-API-Key: didi_ext_...
|
||||
```
|
||||
|
||||
Cheile se creeaza prin: POST /api/v3/pipeline/extension/keys (necesita JWT admin).
|
||||
|
||||
### Endpoint-uri fara autentificare
|
||||
|
||||
In staging, agent-v3 NU valideaza JWT-ul pe requesturi directe (localhost:24803).
|
||||
Kong valideaza JWT-ul in productie, dar in staging plugin-ul JWT nu este activat.
|
||||
|
||||
Asta inseamna:
|
||||
|
||||
- Accesand direct localhost:24803 -- NU ai nevoie de JWT (dar user_id/email sunt extrase din header daca exista)
|
||||
- Accesand prin Kong (port 443) in staging -- NU ai nevoie de JWT (JWT plugin dezactivat in declarative mode)
|
||||
- Accesand prin Kong in productie -- AI NEVOIE de JWT
|
||||
|
||||
Pentru testare multi-user, trimite manual headerele:
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:24803/api/v3/pipeline/analyze \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "X-User-Id: test-user-1" \
|
||||
-H "X-User-Email: test1@test.com" \
|
||||
-d '{"text": "textul de analizat"}'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Endpoint-uri Agent V3 (port 24803)
|
||||
|
||||
Prefix: /api/v3
|
||||
|
||||
### Health (fara auth, fara body)
|
||||
|
||||
```
|
||||
GET /api/v3/health
|
||||
```
|
||||
|
||||
Raspuns: {"service": "agent-v3", "version": "3.0.0", "status": "ok"}
|
||||
|
||||
### Analiza text sincron (endpoint principal)
|
||||
|
||||
```
|
||||
POST /api/v3/pipeline/analyze
|
||||
Content-Type: application/json
|
||||
|
||||
{
|
||||
"text": "Textul de analizat. Minim 20 caractere, maxim 50000.",
|
||||
"user_id": "optional",
|
||||
"user_email": "optional"
|
||||
}
|
||||
```
|
||||
|
||||
Raspuns: AnalysisSession complet (techniques + ai_tampered + claims + domain + verdict).
|
||||
Durata: 15-120 secunde in functie de lungimea textului.
|
||||
|
||||
### Analiza text asincrona (recomandata pentru stress test)
|
||||
|
||||
```
|
||||
POST /api/v3/pipeline/analyze-async
|
||||
Content-Type: application/json
|
||||
|
||||
{
|
||||
"text": "Textul de analizat",
|
||||
"plan_type": 1
|
||||
}
|
||||
```
|
||||
|
||||
Raspuns 202:
|
||||
|
||||
```json
|
||||
{
|
||||
"success": true,
|
||||
"async": true,
|
||||
"data": {
|
||||
"session_id": "uuid",
|
||||
"poll_url": "/api/v3/pipeline/uuid/queue-status",
|
||||
"result_url": "/api/v3/pipeline/uuid/result"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Polling progres:
|
||||
|
||||
```
|
||||
GET /api/v3/pipeline/{session_id}/queue-status
|
||||
```
|
||||
|
||||
Raspuns rezultat final (cand status=completed):
|
||||
|
||||
```
|
||||
GET /api/v3/pipeline/{session_id}/result
|
||||
```
|
||||
|
||||
### Analiza URL
|
||||
|
||||
```
|
||||
POST /api/v3/pipeline/analyze-url
|
||||
Content-Type: application/json
|
||||
|
||||
{
|
||||
"url": "https://example.com/articol",
|
||||
"user_id": "optional"
|
||||
}
|
||||
```
|
||||
|
||||
Detecteaza automat tipul: YouTube (video), imagine, articol.
|
||||
|
||||
### Analiza media (imagine/audio/video)
|
||||
|
||||
```
|
||||
POST /api/v3/pipeline/analyze-media
|
||||
Content-Type: application/json
|
||||
|
||||
{
|
||||
"media_url": "https://didi365.eu/api/v3/media/file/uploads/...",
|
||||
"media_type": "image|audio|video",
|
||||
"user_id": "optional"
|
||||
}
|
||||
```
|
||||
|
||||
Inainte de analiza media, uploadeaza fisierul:
|
||||
|
||||
```
|
||||
POST /api/v3/media/upload
|
||||
Content-Type: multipart/form-data
|
||||
Field: file (max 50MB)
|
||||
```
|
||||
|
||||
### Componente individuale
|
||||
|
||||
Analiza doar o singura componenta (util pentru testare granulara):
|
||||
|
||||
```
|
||||
POST /api/v3/techniques/analyze {"text": "..."}
|
||||
POST /api/v3/ai-tampered/analyze {"text": "..."}
|
||||
POST /api/v3/claims/analyze {"text": "..."}
|
||||
POST /api/v3/domain/analyze {"url": "https://..."}
|
||||
POST /api/v3/source-assessment/analyze {"text": "...", "url": "optional"}
|
||||
```
|
||||
|
||||
### Istoric
|
||||
|
||||
```
|
||||
GET /api/v3/pipeline/history?user_id=USER&page=1&limit=20
|
||||
GET /api/v3/pipeline/history/{session_id}
|
||||
DELETE /api/v3/pipeline/history/{session_id}?user_id=USER
|
||||
|
||||
GET /api/v3/pipeline/history/admin?page=1&limit=20&search=&risk_level=&status=&from_date=&to_date=
|
||||
```
|
||||
|
||||
### Configurare (read-only, util pentru debug)
|
||||
|
||||
```
|
||||
GET /api/v3/techniques/definitions -- ierarhie tehnici
|
||||
GET /api/v3/techniques/config -- config completa
|
||||
GET /api/v3/techniques/models -- modele LLM disponibile
|
||||
GET /api/v3/ai-tampered/config
|
||||
GET /api/v3/ai-tampered/categories
|
||||
GET /api/v3/claims/config
|
||||
GET /api/v3/claims/types
|
||||
GET /api/v3/claims/statuses
|
||||
GET /api/v3/pipeline/verdict-config -- config verdict
|
||||
GET /api/v3/pipeline/queue-health -- health RabbitMQ
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Endpoint-uri didiFramework (port 3005, doar Docker network)
|
||||
|
||||
Pentru acces extern, foloseste admin dashboard (nginx proxiaza la /framework/).
|
||||
|
||||
### Health
|
||||
|
||||
```
|
||||
GET /health
|
||||
GET /health/all -- verifica si PostgreSQL si MinIO
|
||||
```
|
||||
|
||||
### Sync Redis (IMPORTANT)
|
||||
|
||||
```
|
||||
POST /api/sync-redis -- sincronizeaza toti parametrii in Redis
|
||||
GET /api/sync-redis/status -- cand s-a facut ultima sincronizare
|
||||
```
|
||||
|
||||
### CRUD parametri (toate au GET, POST, PUT, DELETE)
|
||||
|
||||
/api/dimensions, /api/subdimensions, /api/techniques, /api/indicators,
|
||||
/api/validation-rules, /api/verdicts/categories, /api/verdicts/risk,
|
||||
/api/verdicts/severity, /api/weights/components, /api/weights/scenarios,
|
||||
/api/weights/multipliers, /api/platforms, /api/sources,
|
||||
/api/claims/status, /api/claims/types, /api/claims/confidence,
|
||||
/api/claims/interpretation, /api/providers/configs, /api/providers/models,
|
||||
/api/providers/assignments, /api/providers/keys
|
||||
|
||||
### Utilizatori
|
||||
|
||||
```
|
||||
GET /api/admin/users?page=1&limit=20&search=&planId=
|
||||
PUT /api/admin/users/:id
|
||||
DELETE /api/admin/users/:id
|
||||
GET /api/admin/plans
|
||||
PUT /api/admin/plans/:id
|
||||
```
|
||||
|
||||
### Credite (apelat intern de agent-v3)
|
||||
|
||||
```
|
||||
POST /api/auth/internal/check-credits {"keycloak_id": "..."}
|
||||
POST /api/auth/internal/deduct-credits {"keycloak_id": "...", "media_type": "text"}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Constrangeri si limite
|
||||
|
||||
### Dimensiune text
|
||||
|
||||
| Parametru | Valoare |
|
||||
|----------------|---------|
|
||||
| Minim text | 20 caractere |
|
||||
| Maxim text | 50,000 caractere |
|
||||
| Encoding | UTF-8 valid |
|
||||
|
||||
### Dimensiune fisiere (upload)
|
||||
|
||||
| Tip | Limita |
|
||||
|------------|----------|
|
||||
| Imagine | 20 MB |
|
||||
| Audio | 100 MB |
|
||||
| Video | 500 MB |
|
||||
| Text | 10 MB |
|
||||
| Document | 50 MB |
|
||||
| Upload API | 50 MB (multer) |
|
||||
|
||||
### Durata media
|
||||
|
||||
| Tip | Limita |
|
||||
|-------|------------|
|
||||
| Video | 180s (3 min) |
|
||||
| Audio | 420s (7 min) |
|
||||
|
||||
### Request payload (Kong)
|
||||
|
||||
Maxim 100 MB per request (request-size-limiting plugin).
|
||||
|
||||
### Timeout-uri
|
||||
|
||||
| Ruta | Timeout |
|
||||
|------------------------|-----------|
|
||||
| /api/v3/pipeline/* | 660s (11 min) |
|
||||
| /*/analyze-media | 300s (5 min) |
|
||||
| Toate celelalte | 180s (3 min) |
|
||||
| Kong -> agent-v3 | 660s connect, 660s read |
|
||||
|
||||
### Rate Limiting (Kong)
|
||||
|
||||
| Nivel | Per minut | Per ora | Per zi |
|
||||
|----------|-----------|---------|---------|
|
||||
| Global | 100 | 2000 | 10,000 |
|
||||
|
||||
Rate limiting-ul Kong este per consumer (global in staging, nu per user).
|
||||
In staging, toti clientii sunt un singur consumer anonim.
|
||||
|
||||
Keycloak defineste rate limits per grup dar NU sunt aplicate inca in Kong:
|
||||
- free-users: 10/min
|
||||
- paid-users: 60/min
|
||||
- enterprise-users: 600/min
|
||||
|
||||
### Credite (agent-v3 -> framework)
|
||||
|
||||
Fiecare analiza costa credite. Agent-v3 verifica la framework inainte de analiza.
|
||||
Costul depinde de media_type (text < image < audio < video).
|
||||
Daca user-ul nu are credite, raspunsul este 403.
|
||||
|
||||
Utilizatorii pre-configurati au credite initiale limitate (100 pentru free/paid).
|
||||
admin@didi.local si enterprise@didi.local au credite nelimitate.
|
||||
|
||||
### CORS
|
||||
|
||||
Origins permise: localhost:3000, localhost:3001, localhost:8100, * (wildcard).
|
||||
Metode: GET, POST, PUT, DELETE, OPTIONS, PATCH.
|
||||
Credentials: activat.
|
||||
|
||||
---
|
||||
|
||||
## Cum sa testezi
|
||||
|
||||
### Test simplu (un request)
|
||||
|
||||
```bash
|
||||
# Health check
|
||||
curl http://localhost:24803/api/v3/health
|
||||
|
||||
# Analiza text (sincron, poate dura 30-60s)
|
||||
curl -X POST http://localhost:24803/api/v3/pipeline/analyze \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"text": "Vaccinurile COVID au fost create de Bill Gates pentru a implanta cipuri 5G in populatie. Studiile arata ca milioane de oameni au fost afectati."}'
|
||||
|
||||
# Analiza asincrona (raspuns instant, polling pentru rezultat)
|
||||
curl -X POST http://localhost:24803/api/v3/pipeline/analyze-async \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"text": "Vaccinurile COVID au fost create de Bill Gates.", "plan_type": 1}'
|
||||
```
|
||||
|
||||
### Test cu JWT prin Kong
|
||||
|
||||
```bash
|
||||
# Obtine token
|
||||
TOKEN=$(curl -s -X POST \
|
||||
"http://localhost:28000/realms/didi-clients/protocol/openid-connect/token" \
|
||||
-d "grant_type=password&client_id=didi-web-app&username=admin@didi.local&password=admin123" \
|
||||
| jq -r '.access_token')
|
||||
|
||||
# Analiza prin Kong (productie path)
|
||||
curl -k -X POST https://localhost:443/api/analyze \
|
||||
-H "Authorization: Bearer $TOKEN" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"text": "Text de test pentru analiza."}'
|
||||
```
|
||||
|
||||
### Test componenta individuala
|
||||
|
||||
```bash
|
||||
# Doar techniques
|
||||
curl -X POST http://localhost:24803/api/v3/techniques/analyze \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"text": "Textul de analizat aici"}'
|
||||
|
||||
# Doar claims
|
||||
curl -X POST http://localhost:24803/api/v3/claims/analyze \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"text": "Romania are 20 milioane de locuitori si PIB-ul a crescut cu 15% anul trecut."}'
|
||||
|
||||
# Doar AI detection
|
||||
curl -X POST http://localhost:24803/api/v3/ai-tampered/analyze \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"text": "This text was definitely written by a human and not an AI."}'
|
||||
```
|
||||
|
||||
### Test upload media + analiza
|
||||
|
||||
```bash
|
||||
# Upload imagine
|
||||
UPLOAD=$(curl -s -X POST http://localhost:24803/api/v3/media/upload \
|
||||
-F "file=@/path/to/image.jpg" \
|
||||
-F "user_id=test-user")
|
||||
echo $UPLOAD
|
||||
|
||||
# Extrage URL-ul
|
||||
MEDIA_URL=$(echo $UPLOAD | jq -r '.data.public_url')
|
||||
|
||||
# Analiza imagine
|
||||
curl -X POST http://localhost:24803/api/v3/pipeline/analyze-media \
|
||||
-H "Content-Type: application/json" \
|
||||
-d "{\"media_url\": \"$MEDIA_URL\", \"media_type\": \"image\"}"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Testare multi-user
|
||||
|
||||
### Strategia
|
||||
|
||||
1. Fiecare "user" simulat trimite request-uri cu un user_id diferit
|
||||
2. Agent-v3 accepta user_id si user_email in body-ul requestului
|
||||
3. In staging, JWT nu este validat -- deci poti simula useri fara token-uri reale
|
||||
4. Pentru teste realiste (cu credite, cu token), foloseste utilizatorii Keycloak
|
||||
|
||||
### Ce trebuie stiut
|
||||
|
||||
- Analiza sincrona blocheaza conexiunea 15-120 secunde
|
||||
- Analiza asincrona returneaza instant si workerii proceseaza in background
|
||||
- RabbitMQ are 24 cozi: 4 componente x 6 plan types
|
||||
- Workeri: 2 techniques, 2 ai-tampered, 3 claims, 2 domain, 2 media-preprocess, 2 aggregators
|
||||
- Fiecare worker proceseaza un singur mesaj la un moment dat (prefetch 3-10 in functie de componenta)
|
||||
- Claims este cel mai lent (cautare web per claim, 3 replici)
|
||||
- Domain este cel mai rapid (analiza locala, fara LLM)
|
||||
- Modelele LLM externe (OpenRouter, OpenAI, Groq) au propriile rate limits
|
||||
|
||||
### Throughput estimat
|
||||
|
||||
| Plan type | Ce se intampla |
|
||||
|-----------|---------------------------------------|
|
||||
| 1 (free) | Prioritate minima in coada |
|
||||
| 6 (enterprise) | Prioritate maxima in coada |
|
||||
|
||||
Cu 2 workers techniques si prefetch 5, poti procesa ~10 analize text simultane.
|
||||
Claims este bottleneck: 3 workers x prefetch 3 = ~9 analize simultane.
|
||||
Video/audio sunt mult mai lente (transcriere + viziune): 1-5 minute per analiza.
|
||||
|
||||
### Endpoint recomandat pentru stress test
|
||||
|
||||
Foloseste analiza asincrona:
|
||||
|
||||
```
|
||||
POST /api/v3/pipeline/analyze-async
|
||||
{"text": "...", "plan_type": 1}
|
||||
```
|
||||
|
||||
Avantaje:
|
||||
- Raspuns instant (202 Accepted)
|
||||
- Workerii proceseaza in paralel
|
||||
- Poti monitoriza progresul individual per sesiune
|
||||
- Nu blocheaza conexiunea HTTP
|
||||
|
||||
Polling status:
|
||||
|
||||
```
|
||||
GET /api/v3/pipeline/{session_id}/queue-status
|
||||
```
|
||||
|
||||
### Bottleneck-uri de monitorizat
|
||||
|
||||
| Resursa | Cum verifici |
|
||||
|-------------------|------------------------------------------------------|
|
||||
| RabbitMQ | http://localhost:15672 (admin/rabbitmq123) |
|
||||
| Redis memorie | docker exec didi-cache redis-cli -a redis123 info memory |
|
||||
| Workers activi | docker ps --filter name=agent-v3-worker |
|
||||
| PG conexiuni | Prin PgAdmin http://localhost:5050 |
|
||||
| Cozi pline | RabbitMQ UI -> Queues -> Ready messages |
|
||||
|
||||
### Limitari stress test
|
||||
|
||||
1. API keys LLM (OpenRouter, OpenAI, Groq) au rate limits proprii -- daca trimiti 50 analize simultan, vei primi erori 429 de la providerii LLM
|
||||
2. Modelul local Qwen Vision (10.11.10.17:14011) proceseaza secvential -- nu scala orizontal
|
||||
3. M17 Whisper (10.11.10.17:54300) -- un singur endpoint, probabil limitat
|
||||
4. Redis 512MB -- la volum mare de sesiuni simultane, verifica memoria
|
||||
5. PostgreSQL cluster -- in general nu este bottleneck, dar verifica conexiunile active
|
||||
|
||||
### Chei Redis pentru monitoring
|
||||
|
||||
```bash
|
||||
# Sesiuni active
|
||||
docker exec didi-cache redis-cli -a redis123 keys "didi:pipeline:*:status" | wc -l
|
||||
|
||||
# Lock-uri active (workeri in procesare)
|
||||
docker exec didi-cache redis-cli -a redis123 keys "didi:queue:lock:*" | wc -l
|
||||
|
||||
# Framework config (trebuie sa existe mereu)
|
||||
docker exec didi-cache redis-cli -a redis123 keys "didi:framework:*"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Structura raspuns AnalysisSession
|
||||
|
||||
Orice analiza completa returneaza acest format:
|
||||
|
||||
```
|
||||
session_id -- UUID unic
|
||||
status -- running | completed | failed
|
||||
input_type -- text | url | image | audio | video
|
||||
risk_score -- 0-100 (scor final)
|
||||
risk_category -- RELIABLE | MOSTLY_RELIABLE | MIXED | UNRELIABLE | DISINFORMATION | INCONCLUSIVE
|
||||
risk_level -- VERY_LOW | LOW | MODERATE | HIGH | VERY_HIGH | CRITICAL
|
||||
confidence -- 0-100
|
||||
total_duration_ms -- milisecunde
|
||||
|
||||
techniques.manipulation_score -- 0-100
|
||||
ai_tampered.ai_probability -- 0-100
|
||||
claims.credibility_score -- 0-100 (null daca nu sunt claims)
|
||||
domain.trust_score -- 0-100 (null daca nu exista URL)
|
||||
verdict.risk_score -- 0-100 (identic cu root risk_score)
|
||||
verdict.explanation_ro -- explicatie in romana
|
||||
verdict.explanation_en -- explicatie in engleza
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Coduri eroare frecvente
|
||||
|
||||
| Cod | Cauza | Solutie |
|
||||
|-----|------------------------------------------|----------------------------------|
|
||||
| 400 | Text prea scurt (<20 chars) sau invalid | Mareste textul |
|
||||
| 400 | media_type invalid sau lipsa | Verifica parametrii |
|
||||
| 403 | Credite insuficiente | Foloseste admin@didi.local |
|
||||
| 408 | Timeout (analiza prea lenta) | Foloseste analyze-async |
|
||||
| 413 | Payload prea mare (>100MB) | Micoreaza fisierul |
|
||||
| 429 | Rate limit Kong | Asteapta 1 minut |
|
||||
| 500 | Eroare interna (LLM, Redis, PG) | Verifica logs: docker logs didi-agent-v3 |
|
||||
| 502 | Serviciu backend indisponibil | Verifica ca agent-v3 ruleaza |
|
||||
| 504 | Gateway timeout | Analiza dureaza prea mult |
|
||||
|
||||
---
|
||||
|
||||
## Verificare rapida ca totul functioneaza
|
||||
|
||||
Aceste comenzi, in ordine, confirma ca platforma este operationala:
|
||||
|
||||
```bash
|
||||
# 1. Health agent-v3
|
||||
curl -s http://localhost:24803/api/v3/health | jq .
|
||||
|
||||
# 2. Config exista in Redis (trebuie sa fie non-null)
|
||||
curl -s http://localhost:24803/api/v3/techniques/definitions | jq '.dimensions | length'
|
||||
|
||||
# 3. Analiza text rapida (30-60s)
|
||||
curl -s -X POST http://localhost:24803/api/v3/techniques/analyze \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"text": "Studiile demonstreaza ca pamantul este plat si NASA ne minte de decenii. Milioane de oameni au descoperit adevarul."}' | jq '{manipulation_score, techniques_count}'
|
||||
|
||||
# 4. Analiza completa (60-120s)
|
||||
curl -s -X POST http://localhost:24803/api/v3/pipeline/analyze \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"text": "Studiile demonstreaza ca pamantul este plat si NASA ne minte de decenii. Milioane de oameni au descoperit adevarul."}' | jq '{risk_score, risk_category, confidence}'
|
||||
|
||||
# 5. RabbitMQ functional (trebuie sa fie cozi)
|
||||
curl -s -u admin:rabbitmq123 http://localhost:15672/api/queues | jq '.[].name' | head -10
|
||||
```
|
||||
89
backend/production/API_TEST_REPORT_2026-07-08.md
Normal file
89
backend/production/API_TEST_REPORT_2026-07-08.md
Normal file
|
|
@ -0,0 +1,89 @@
|
|||
# Raport testare API — 2026-07-08
|
||||
|
||||
> Dovadă pentru criteriul 6 din caietul de sarcini („Implementare & transfer": OpenAPI/Swagger +
|
||||
> set minim teste API). Documentul-pereche: `API_TESTING_GUIDE.md` (ghid de testare manuală) și
|
||||
> specificațiile `agent-v3/openapi.yaml` + `didiFramework/openapi.yaml`.
|
||||
|
||||
## Metodă
|
||||
|
||||
1. **Inventar din cod, nu din documentație**: toate definițiile de rute Express au fost extrase
|
||||
automat din sursă (`scripts/api/api_probe.py`). S-a verificat separat că nu există rute definite
|
||||
dinamic (variabile/template literals), mount-uri cu prefix nescanate sau generatoare CRUD active —
|
||||
toate cele 4 verificări au ieșit goale, deci inventarul static este complet.
|
||||
2. **Probă live pe didi11**: fiecare endpoint apelat cu token Keycloak real (user `api-test`,
|
||||
realm `didi-admins`, rol `admin`). Strategie non-distructivă: GET real; POST/PUT/PATCH/DELETE cu
|
||||
body gol sau ID inexistent (răspunsul 400/404 dovedește cablarea fără a muta date). Excepții
|
||||
idempotente rulate real: `dry-run`, `sync-redis`, `check-credits`. Endpoint-urile cu efect real
|
||||
pe body gol (email de test, credit-reset, use-credit) sunt consemnate ca verificate manual și
|
||||
excluse din rulările automate.
|
||||
3. **Verificare integritate**: snapshot pe 7 contoare DB înainte/după rularea finală — identic
|
||||
(800 indicatori / 621 reguli / 166 tehnici / 23 modele / 83 assignments / sesiuni / utilizatori).
|
||||
|
||||
## Rezultat
|
||||
|
||||
| Metric | Valoare |
|
||||
|---|---|
|
||||
| Endpoint-uri inventariate | **374** (87 agent-v3 + 287 didiFramework) |
|
||||
| Cablate (răspund cu handler propriu) | **374 / 374** |
|
||||
| Rute moarte (`Cannot GET/POST …`) | **0** |
|
||||
| Distribuție status finală | 200×135 · 400×102 · 401×4 · 403×2 · 404×118 · 409×9 · 500×4 |
|
||||
|
||||
Cele 401/403 sunt comportament CORECT (extension cere `X-API-Key`; claim/resolve moderare cer rol
|
||||
`moderator`/`senior_moderator`, pe care admin nu îl are — separare de roluri funcțională).
|
||||
|
||||
## Buguri găsite și REPARATE în această sesiune
|
||||
|
||||
| Endpoint | Problemă | Fix |
|
||||
|---|---|---|
|
||||
| `GET /api/validation-rules/stats` | 500 — umbrit de ruta `/:id` (declarată înainte) | reordonare rute (`validation-rules.ts`) |
|
||||
| `DELETE /api/indicators/by-technique/:techniqueId` | inaccesibil — umbrit de `/:techniqueId/:indicatorId` | reordonare rute (`indicators.ts`) |
|
||||
| `POST /api/sync-analysis/batch` | umbrit de `/:sessionId` (legacy, reparat oricum) | reordonare rute (`sync-analysis.ts`) |
|
||||
| `GET /api/weights/multipliers/type/:type` | 500 pe input non-numeric (coloana e integer) | validare → 400 (`weights.ts`) |
|
||||
|
||||
S-a rulat și un scan sistematic de umbriri de rute pe ambele servicii — zero umbriri rămase.
|
||||
|
||||
## Probleme cunoscute, deschise (consemnate, ne-blocante)
|
||||
|
||||
1. **`/api/waitlist/*` → 500**: cere `STAGING_DB_HOST` + containerul `staging-dataLayer-postgres`
|
||||
(absent pe didi11). Feature pre-lansare, marcat `deprecated` în spec. Remediere: setare env +
|
||||
pornire container, sau eliminarea rutelor.
|
||||
2. **`GET /api/v3/pipeline/history/admin/:id`** răspunde 500 (în loc de 400) dacă `:id` nu e UUID —
|
||||
gap cosmetic de validare în agent-v3; cu UUID valid răspunde corect (`Session not found` / 200).
|
||||
3. Endpoint-uri legacy marcate `deprecated` în spec: `domain/*` (înlocuit de source-assessment),
|
||||
`sync-analysis/*` (agent-v3 persistă direct în PG), `prompts/*` pe fișiere (sursa operațională
|
||||
e DB via `/api/providers/prompts`).
|
||||
|
||||
## Artefacte
|
||||
|
||||
| Artefact | Locație |
|
||||
|---|---|
|
||||
| Spec OpenAPI 3.0.3 agent-v3 (87 operații) | `backend/services/orchestration-layer/agent-v3/openapi.yaml` |
|
||||
| Spec OpenAPI 3.0.3 didiFramework (287 operații) | `backend/services/orchestration-layer/didiFramework/openapi.yaml` |
|
||||
| Script probă (reproductibil la recepție) | `scripts/api/api_probe.py` |
|
||||
| Generator spec din inventar + probe | `scripts/api/generate_openapi.py` |
|
||||
| Rezultate brute probă | `scripts/api/probe_results_2026-07-08.json` |
|
||||
| Swagger UI live | `http://10.11.10.11:8089` (container `didi-api-docs`, servește ambele spec-uri) |
|
||||
|
||||
Ambele spec-uri sunt **validate** cu `openapi-spec-validator` (OK). Fiecare operație poartă
|
||||
adnotarea `x-tested` cu statusul HTTP observat la probă și data testării.
|
||||
|
||||
## Reproducere
|
||||
|
||||
```bash
|
||||
# 1. Probă completă (necesită serviciile pornite + Keycloak local)
|
||||
python3 scripts/api/api_probe.py x /tmp/probe_results.json
|
||||
|
||||
# 2. Regenerare spec-uri
|
||||
python3 scripts/api/generate_openapi.py /tmp/probe_results.json \
|
||||
backend/services/orchestration-layer/agent-v3/openapi.yaml \
|
||||
backend/services/orchestration-layer/didiFramework/openapi.yaml
|
||||
|
||||
# 3. Validare
|
||||
docker run --rm -v $PWD/backend/services/orchestration-layer:/s python:3.12-alpine \
|
||||
sh -c 'pip install -q openapi-spec-validator && \
|
||||
python -m openapi_spec_validator /s/agent-v3/openapi.yaml && \
|
||||
python -m openapi_spec_validator /s/didiFramework/openapi.yaml'
|
||||
```
|
||||
|
||||
User de test recepție: `api-test` / `ApiTest2026x` (realm `didi-admins`, rol `admin`;
|
||||
cont PG auto-creat `api-test@didi.local`, internet_user_id 90001).
|
||||
333
backend/production/build-local.sh
Normal file
333
backend/production/build-local.sh
Normal file
|
|
@ -0,0 +1,333 @@
|
|||
#!/bin/bash
|
||||
# =============================================================================
|
||||
# DIDI Platform — Build From Scratch (topologie LOCALĂ)
|
||||
# =============================================================================
|
||||
# Ridică întreaga platformă backend de la zero, cu TOATE serviciile în containere
|
||||
# locale pe mașina de deployment (fără cluster extern). Topologia serviciilor:
|
||||
#
|
||||
# Data layer : didi-postgres (PG17, DB principală LOCALĂ) + didi-cache (Redis)
|
||||
# + staging-dataLayer-rabbitmq + staging-dataLayer-minio + didi-keycloak
|
||||
# → docker-compose.local.yml
|
||||
# Gateway : didi-kong (DBless, imagine didi-kong:latest)
|
||||
# Orchestration: didi-framework + didi-agent-v3 + 13 workeri
|
||||
# UI : didi-admin
|
||||
#
|
||||
# Seed: DIDI_full_export_2026-07-02.sql (23MB, schema + date + migrațiile 016/017).
|
||||
# Se importă automat la PRIMUL boot al didi-postgres (volum gol), prin
|
||||
# montarea în /docker-entrypoint-initdb.d. Redis se reface din PG cu sync-redis.
|
||||
#
|
||||
# Cerințe: docker, docker compose v2+. Nicio dependență de rețea externă.
|
||||
# Rulare: chmod +x build-local.sh && ./build-local.sh [HOSTNAME]
|
||||
# (implicit = hostname-ul mașinii curente)
|
||||
#
|
||||
# NB: documentul-pereche este DEPLOY_FROM_SCRATCH.md. Scriptul vechi full-build.sh
|
||||
# vizează topologia cluster (PG/Kong externe) și este păstrat ca referință.
|
||||
# =============================================================================
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
# Hostname-ul mașinii de deployment. Implicit = hostname-ul mașinii curente
|
||||
# (agnostic — funcționează pe orice mașină). Se poate suprascrie ca prim argument.
|
||||
PLATFORM_HOSTNAME="${1:-$(hostname -f 2>/dev/null || hostname)}"
|
||||
|
||||
# IP-ul principal LAN al mașinii (pentru dashboard + redirect URIs Keycloak).
|
||||
# Portabil: se poate suprascrie ca al 2-lea argument.
|
||||
HOST_IP="${2:-$(hostname -I 2>/dev/null | tr ' ' '\n' | grep -vE '^(127\.|172\.1[6-9]\.|172\.2[0-9]\.|172\.3[0-1]\.|10\.0\.)' | head -1)}"
|
||||
[ -z "$HOST_IP" ] && HOST_IP="$(hostname -I 2>/dev/null | awk '{print $1}')"
|
||||
|
||||
RED='\033[0;31m'; GREEN='\033[0;32m'; YELLOW='\033[1;33m'; CYAN='\033[0;36m'; NC='\033[0m'
|
||||
|
||||
# -- Paths (auto-detectate din locația scriptului — portabil pe orice mașină) ---
|
||||
# Scriptul stă în backend/production/ → BACKEND = părintele lui production/
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
BACKEND="$(cd "$SCRIPT_DIR/.." && pwd)"
|
||||
DATA_LAYER="$BACKEND/services/data-layer"
|
||||
DIDI_DB="$DATA_LAYER/didiDatabase"
|
||||
FRAMEWORK="$BACKEND/services/orchestration-layer/didiFramework"
|
||||
AGENT_V3="$BACKEND/services/orchestration-layer/agent-v3"
|
||||
KONG_DIR="$BACKEND/services/gateway-auth-layer/didiKong"
|
||||
ADMIN_DIR="$BACKEND/admin-dashboard"
|
||||
|
||||
# -- PostgreSQL LOCAL (container didi-postgres) ----------------------------
|
||||
PG_CONTAINER="didi-postgres"
|
||||
PG_USER="bos_interface"
|
||||
PG_DB="DIDI"
|
||||
SEED_FILE="$DIDI_DB/DIDI_full_export_2026-07-02.sql" # sursă de adevăr; NU cel din martie
|
||||
|
||||
# -- Keycloak LOCAL --------------------------------------------------------
|
||||
KC_PORT="28080" # host → 8080 container
|
||||
KC_BASE="http://localhost:${KC_PORT}/auth" # servit sub /auth (KC_HTTP_RELATIVE_PATH)
|
||||
KC_ADMIN_USER="admin"
|
||||
KC_ADMIN_PASS="admin123"
|
||||
|
||||
log() { echo -e "${CYAN}[$(date +%H:%M:%S)]${NC} $*"; }
|
||||
ok() { echo -e "${GREEN} OK${NC} $*"; }
|
||||
warn() { echo -e "${YELLOW} WARN${NC} $*"; }
|
||||
fail() { echo -e "${RED} FAIL${NC} $*"; exit 1; }
|
||||
section() { echo; echo -e "${YELLOW}====================================================================${NC}"; echo -e "${YELLOW} $*${NC}"; echo -e "${YELLOW}====================================================================${NC}"; }
|
||||
|
||||
wait_healthy() {
|
||||
local container="$1" max="${2:-120}" elapsed=0 status
|
||||
log "Aștept $container să fie healthy (max ${max}s)..."
|
||||
while [ $elapsed -lt $max ]; do
|
||||
status=$(docker inspect --format='{{if .State.Health}}{{.State.Health.Status}}{{else}}running{{end}}' "$container" 2>/dev/null || echo "missing")
|
||||
[ "$status" = "healthy" ] || [ "$status" = "running" ] && { ok "$container ($status)"; return 0; }
|
||||
sleep 3; elapsed=$((elapsed + 3))
|
||||
done
|
||||
warn "$container nu a devenit healthy în ${max}s (status: $status)"; return 1
|
||||
}
|
||||
|
||||
pg() { docker exec "$PG_CONTAINER" psql -U "$PG_USER" -d "$PG_DB" -tAc "$1" 2>/dev/null; }
|
||||
|
||||
# =============================================================================
|
||||
section "FAZA 0: Verificări preliminare"
|
||||
# =============================================================================
|
||||
docker info >/dev/null 2>&1 || fail "Docker nu rulează"; ok "Docker activ"
|
||||
docker compose version >/dev/null 2>&1 || fail "Docker Compose v2 lipsește"; ok "Docker Compose disponibil"
|
||||
[ -f "$SEED_FILE" ] || fail "Lipsește seed-ul: $SEED_FILE"
|
||||
ok "Seed găsit: $(basename "$SEED_FILE") ($(du -h "$SEED_FILE" | cut -f1))"
|
||||
|
||||
# =============================================================================
|
||||
section "FAZA 1: Docker network"
|
||||
# =============================================================================
|
||||
docker network create didi-network 2>/dev/null && ok "Rețea creată" || ok "Rețeaua didi-network există deja"
|
||||
|
||||
# =============================================================================
|
||||
section "FAZA 2: Data layer local (PG + seed, Redis, RabbitMQ, MinIO, Keycloak)"
|
||||
# =============================================================================
|
||||
cd "$DATA_LAYER"
|
||||
|
||||
# didi-postgres importă seed-ul automat la primul boot pe volum gol
|
||||
# (montaj /tmp/didi_full_dump.sql → /docker-entrypoint-initdb.d/01-dump.sql, vezi compose).
|
||||
# Prima instalare = containerul didi-postgres nu există încă (deci nici volumul lui).
|
||||
if [ -n "$(docker ps -aq -f name=^${PG_CONTAINER}$)" ]; then
|
||||
warn "didi-postgres există deja — NU re-importez seed-ul (baza e populată)."
|
||||
warn "Pentru un import curat: oprește stack-ul + șterge volumul didi-postgres-data, apoi re-rulează."
|
||||
SEED_ON_INIT=0
|
||||
else
|
||||
log "Prima instalare — pregătesc seed-ul pentru auto-import la boot..."
|
||||
cp "$SEED_FILE" /tmp/didi_full_dump.sql
|
||||
ok "Seed copiat în /tmp/didi_full_dump.sql (montat ca init script în didi-postgres)"
|
||||
SEED_ON_INIT=1
|
||||
fi
|
||||
|
||||
log "Pornesc data layer (docker-compose.local.yml)..."
|
||||
docker compose -f docker-compose.local.yml up -d 2>&1 | tail -6
|
||||
ok "Data layer pornit"
|
||||
|
||||
wait_healthy "$PG_CONTAINER" 90
|
||||
# Aștept ca importul init (dacă e prima instalare) să termine — poate dura pe seed 23MB
|
||||
if [ "${SEED_ON_INIT:-0}" = "1" ]; then
|
||||
log "Aștept finalizarea importului seed (init script rulează la primul boot)..."
|
||||
for i in $(seq 1 60); do
|
||||
C=$(pg "SELECT count(*) FROM information_schema.schemata WHERE schema_name IN ('bos_analysis','bos_parammgmt','bos_sysadmin','bos_subscriber')" || echo 0)
|
||||
[ "$C" = "4" ] && break
|
||||
sleep 3
|
||||
done
|
||||
fi
|
||||
|
||||
SCHEMAS=$(pg "SELECT count(*) FROM information_schema.schemata WHERE schema_name IN ('bos_analysis','bos_parammgmt','bos_sysadmin','bos_subscriber')" || echo 0)
|
||||
[ "$SCHEMAS" = "4" ] || fail "Baza DIDI nu are cele 4 scheme bos_* (găsit: $SCHEMAS). Import eșuat."
|
||||
ok "Baza DIDI populată: 4 scheme bos_*"
|
||||
|
||||
# Sanity pe tabelele-cheie (nu doar tehnici)
|
||||
log "Verific tabele critice..."
|
||||
for entry in "bos_parammgmt|technique|166" "bos_parammgmt|dimension|8" "bos_parammgmt|llm_model|" \
|
||||
"bos_parammgmt|component_stage_assignment|" "bos_parammgmt|input_type_profile|6" \
|
||||
"bos_parammgmt|source_type|" "bos_parammgmt|verdict_category|" \
|
||||
"bos_sysadmin|subscription_plan|" "bos_analysis|analysis_session|"; do
|
||||
IFS='|' read -r sch tbl exp <<< "$entry"
|
||||
n=$(pg "SELECT count(*) FROM ${sch}.${tbl}" || echo "ERR")
|
||||
if [ "$n" = "ERR" ]; then warn " LIPSĂ: ${sch}.${tbl}"; else ok " ${sch}.${tbl} = ${n}${exp:+ (aștept ~$exp)}"; fi
|
||||
done
|
||||
|
||||
# Localizare config LLM: seed-ul livrează modelul primar ca `Qwen3.5-397B-A17B` pe
|
||||
# provider remote `10.11.10.17` — dar vLLM-ul local servește `qwen3.5` prin routerul
|
||||
# `llm-api:14011`. Aliniem model_code + provider base_url (idempotent, rulează la fiecare build).
|
||||
log "Localizez config LLM (model → qwen3.5, provideri → llm-api:14011)..."
|
||||
docker exec "$PG_CONTAINER" psql -U "$PG_USER" -d "$PG_DB" -c "
|
||||
UPDATE bos_parammgmt.llm_model SET model_code='qwen3.5', model_name='Qwen 3.5 (local)' WHERE model_code='Qwen3.5-397B-A17B';
|
||||
UPDATE bos_parammgmt.llm_provider SET base_url='http://llm-api:14011/v1' WHERE base_url LIKE 'http://10.11.10.17:1401%';
|
||||
" >/dev/null 2>&1 && ok "Config LLM localizat (qwen3.5 @ llm-api:14011)" || warn "Localizarea config LLM a eșuat — verifică manual"
|
||||
|
||||
wait_healthy didi-cache 30
|
||||
wait_healthy staging-dataLayer-rabbitmq 90
|
||||
wait_healthy staging-dataLayer-minio 60
|
||||
wait_healthy didi-keycloak 180
|
||||
|
||||
# --- Localizare Keycloak: frontendUrl + user admin + redirect URIs pentru HOST_IP ---
|
||||
# Realm-import livrează didi-admins fără user de admin și cu frontendUrl staging.
|
||||
log "Localizez Keycloak (frontendUrl + user admin + redirect URIs) pentru $HOST_IP..."
|
||||
KC_API="http://localhost:${KC_PORT}/auth"
|
||||
for i in $(seq 1 30); do
|
||||
KTOK=$(curl -s -X POST "$KC_API/realms/master/protocol/openid-connect/token" -d "client_id=admin-cli" -d "username=${KC_ADMIN_USER}" -d "password=${KC_ADMIN_PASS}" -d "grant_type=password" 2>/dev/null | sed -n 's/.*"access_token":"\([^"]*\)".*/\1/p')
|
||||
[ -n "$KTOK" ] && break; sleep 3
|
||||
done
|
||||
if [ -n "$KTOK" ]; then
|
||||
# frontendUrl → host curent (issuer-ul token-ului trebuie sa coincida cu dashboard-ul)
|
||||
curl -s -X PUT "$KC_API/admin/realms/didi-admins" -H "Authorization: Bearer $KTOK" -H 'Content-Type: application/json' \
|
||||
-d "{\"realm\":\"didi-admins\",\"attributes\":{\"frontendUrl\":\"https://${HOST_IP}:3001/auth\"}}" >/dev/null 2>&1
|
||||
# user admin (idempotent) + parola conforma cu passwordPolicy (length10+upper+digit+lower)
|
||||
curl -s -X POST "$KC_API/admin/realms/didi-admins/users" -H "Authorization: Bearer $KTOK" -H 'Content-Type: application/json' \
|
||||
-d '{"username":"admin","email":"admin@didi.local","enabled":true,"emailVerified":true,"firstName":"DIDI","lastName":"Admin","credentials":[{"type":"password","value":"Admin12345","temporary":false}]}' >/dev/null 2>&1
|
||||
AUID=$(curl -s "$KC_API/admin/realms/didi-admins/users?username=admin" -H "Authorization: Bearer $KTOK" 2>/dev/null | sed -n 's/.*"id":"\([^"]*\)".*/\1/p' | head -1)
|
||||
# admin + moderator + senior_moderator (write-ul de moderare cere moderator/senior_moderator, NU admin)
|
||||
AROLES=$(curl -s "$KC_API/admin/realms/didi-admins/roles" -H "Authorization: Bearer $KTOK" 2>/dev/null | python3 -c "import sys,json;d=json.load(sys.stdin);print(json.dumps([{'id':r['id'],'name':r['name']} for r in d if r['name'] in ('admin','moderator','senior_moderator')]))" 2>/dev/null)
|
||||
[ -n "$AUID" ] && [ -n "$AROLES" ] && curl -s -X POST "$KC_API/admin/realms/didi-admins/users/$AUID/role-mappings/realm" -H "Authorization: Bearer $KTOK" -H 'Content-Type: application/json' -d "$AROLES" >/dev/null 2>&1
|
||||
# redirect URIs + webOrigin pentru host curent pe clientul admin-dashboard
|
||||
ACID=$(curl -s "$KC_API/admin/realms/didi-admins/clients?clientId=admin-dashboard" -H "Authorization: Bearer $KTOK" 2>/dev/null | sed -n 's/.*"id":"\([^"]*\)".*/\1/p' | head -1)
|
||||
if [ -n "$ACID" ]; then
|
||||
curl -s "$KC_API/admin/realms/didi-admins/clients/$ACID" -H "Authorization: Bearer $KTOK" 2>/dev/null > /tmp/ac.json
|
||||
python3 - "$HOST_IP" <<'PYKC' 2>/dev/null
|
||||
import json,sys
|
||||
ip=sys.argv[1]; c=json.load(open('/tmp/ac.json'))
|
||||
u=set(c.get('redirectUris',[])); u.update([f'https://{ip}:3001/admin/*',f'https://{ip}:3001/*']); c['redirectUris']=sorted(u)
|
||||
w=set(c.get('webOrigins',[])); w.add(f'https://{ip}:3001'); c['webOrigins']=sorted(w)
|
||||
json.dump(c,open('/tmp/ac.json','w'))
|
||||
PYKC
|
||||
curl -s -X PUT "$KC_API/admin/realms/didi-admins/clients/$ACID" -H "Authorization: Bearer $KTOK" -H 'Content-Type: application/json' -d @/tmp/ac.json >/dev/null 2>&1
|
||||
fi
|
||||
ok "Keycloak localizat — login: admin / Admin12345 (realm didi-admins)"
|
||||
else
|
||||
warn "Nu am putut obține token Keycloak — creează userul admin manual"
|
||||
fi
|
||||
|
||||
# =============================================================================
|
||||
section "FAZA 3: MinIO — bucket-uri"
|
||||
# =============================================================================
|
||||
if [ -f "$DATA_LAYER/didiStorage/init-buckets.sh" ]; then
|
||||
log "Rulez init-buckets.sh în containerul MinIO..."
|
||||
docker exec -e MINIO_HOST=localhost -e MINIO_PORT=9000 \
|
||||
-e MINIO_ROOT_USER=minioadmin -e MINIO_ROOT_PASSWORD=minio123 \
|
||||
staging-dataLayer-minio sh -c "$(cat "$DATA_LAYER/didiStorage/init-buckets.sh")" 2>&1 | tail -5 || warn "init-buckets a raportat erori (verifică manual)"
|
||||
ok "Bucket-uri inițializate"
|
||||
else
|
||||
warn "init-buckets.sh lipsește — creează bucket-urile manual dacă e nevoie"
|
||||
fi
|
||||
|
||||
# =============================================================================
|
||||
section "FAZA 4: Kong (gateway DBless, imagine didi-kong:latest)"
|
||||
# =============================================================================
|
||||
cd "$KONG_DIR"
|
||||
if ! docker images didi-kong:latest --format '{{.ID}}' | grep -q .; then
|
||||
log "Build imagine didi-kong:latest..."
|
||||
docker build -t didi-kong:latest . 2>&1 | tail -3
|
||||
fi
|
||||
docker rm -f didi-kong >/dev/null 2>&1 || true
|
||||
log "Pornesc didi-kong (DBless, config local)..."
|
||||
docker run -d --name didi-kong --network didi-network --restart unless-stopped \
|
||||
-p 127.0.0.1:18000:8000 -p 127.0.0.1:18001:8001 -p 127.0.0.1:18443:8443 \
|
||||
-e KONG_DATABASE=off \
|
||||
-e KONG_DECLARATIVE_CONFIG=/kong/declarative/kong.yml \
|
||||
-e "KONG_PROXY_LISTEN=0.0.0.0:8000, 0.0.0.0:8443 ssl" \
|
||||
-e KONG_ADMIN_LISTEN=0.0.0.0:8001 \
|
||||
-v "$KONG_DIR/declarative/kong.yml.didi11-local:/kong/declarative/kong.yml:ro" \
|
||||
didi-kong:latest >/dev/null
|
||||
wait_healthy didi-kong 60 || warn "Kong nu a raportat healthy — verifică: docker logs didi-kong"
|
||||
|
||||
# =============================================================================
|
||||
section "FAZA 5: didiFramework (CRUD parametri) + sync Redis"
|
||||
# =============================================================================
|
||||
cd "$FRAMEWORK"
|
||||
log "Build + start didi-framework..."
|
||||
docker compose up -d --build 2>&1 | tail -5
|
||||
wait_healthy didi-framework 90
|
||||
|
||||
log "Sync framework → Redis (regenerează cache-ul din PostgreSQL)..."
|
||||
SYNC=$(docker exec didi-framework wget -qO- --post-data='' "http://127.0.0.1:3005/api/sync-redis" 2>/dev/null || echo FAIL)
|
||||
echo "$SYNC" | grep -q '"success"' && ok "Sync Redis reușit" || warn "Sync Redis posibil eșuat: $SYNC"
|
||||
KF=$(docker exec didi-cache redis-cli -a redis123 --no-auth-warning keys "didi:framework:*" 2>/dev/null | wc -l)
|
||||
KC=$(docker exec didi-cache redis-cli -a redis123 --no-auth-warning keys "didi:config:*" 2>/dev/null | wc -l)
|
||||
ok "Redis: $KF chei framework, $KC chei config"
|
||||
[ "$KF" -ge 5 ] || warn "Prea puține chei framework ($KF) — verifică sync-ul"
|
||||
|
||||
# =============================================================================
|
||||
section "FAZA 6: Agent V3 + workeri"
|
||||
# =============================================================================
|
||||
cd "$AGENT_V3"
|
||||
[ -f .env ] || warn ".env lipsește în agent-v3 — analizele LLM nu vor merge fără OPENROUTER/OPENAI/GROQ keys"
|
||||
# PUBLIC_API_BASE_URL (media URLs) = host curent, reachable din browser ȘI din workeri
|
||||
if [ -f .env ]; then
|
||||
if grep -q '^PUBLIC_API_BASE_URL=' .env; then
|
||||
sed -i -E "s|^(PUBLIC_API_BASE_URL=).*|\1http://${HOST_IP}:24803|" .env
|
||||
else
|
||||
echo "PUBLIC_API_BASE_URL=http://${HOST_IP}:24803" >> .env
|
||||
fi
|
||||
ok "agent-v3 PUBLIC_API_BASE_URL → http://${HOST_IP}:24803"
|
||||
fi
|
||||
log "Build + start agent-v3 + workeri..."
|
||||
docker compose up -d --build 2>&1 | tail -10
|
||||
wait_healthy didi-agent-v3 90
|
||||
W=$(docker ps --format '{{.Names}}' | grep -c "agent-v3-worker" || true)
|
||||
A=$(docker ps --format '{{.Names}}' | grep -c "verdict-aggregator" || true)
|
||||
ok "Workeri: $W | Aggregators: $A"
|
||||
|
||||
# =============================================================================
|
||||
section "FAZA 7: Admin Dashboard"
|
||||
# =============================================================================
|
||||
if docker ps --format '{{.Names}}' | grep -q "^didi-admin"; then
|
||||
ok "Admin dashboard rulează deja"
|
||||
else
|
||||
log "Build + start didi-admin..."
|
||||
cd "$ADMIN_DIR"
|
||||
# Config React baked în bundle → aliniat la HOST_IP curent (Keycloak + API pe host:3001)
|
||||
if [ -f .env ]; then
|
||||
sed -i -E "s|^(REACT_APP_SERVER_HOST=).*|\1${HOST_IP}|; s|^(REACT_APP_HOST=).*|\1${HOST_IP}|; s|^(REACT_APP_KEYCLOAK_URL=).*|\1https://${HOST_IP}:3001/auth|; s|^(REACT_APP_API_BASE_URL=).*|\1https://${HOST_IP}:3001|" .env
|
||||
ok "admin-dashboard/.env aliniat la ${HOST_IP}"
|
||||
fi
|
||||
docker build -t didi-admin:latest . 2>&1 | tail -5
|
||||
docker rm -f didi-admin-local >/dev/null 2>&1 || true
|
||||
docker run -d --name didi-admin-local --network didi-network \
|
||||
-p 3081:80 -p 3001:443 --restart unless-stopped didi-admin:latest >/dev/null
|
||||
ok "Admin dashboard pornit (3081 HTTP / 3001 HTTPS)"
|
||||
fi
|
||||
|
||||
# =============================================================================
|
||||
section "FAZA 8: Observabilitate (Prometheus/Grafana/Loki/Jaeger — opțional)"
|
||||
# =============================================================================
|
||||
OBS_DIR="$BACKEND/observability"
|
||||
if [ -f "$OBS_DIR/docker-compose.yml" ]; then
|
||||
cd "$OBS_DIR"
|
||||
log "Pornesc stack-ul de observabilitate..."
|
||||
docker compose up -d 2>&1 | tail -4
|
||||
ok "Observabilitate pornită — Grafana :3030, Prometheus :9090, Jaeger :16686"
|
||||
else
|
||||
warn "observability/ lipsește — sar peste (opțional)"
|
||||
fi
|
||||
|
||||
# =============================================================================
|
||||
section "VERIFICARE FINALĂ"
|
||||
# =============================================================================
|
||||
echo; log "Health checks:"
|
||||
declare -A CHECKS=(
|
||||
["PostgreSQL"]="docker exec $PG_CONTAINER pg_isready -U $PG_USER 2>/dev/null"
|
||||
["Redis"]="docker exec didi-cache redis-cli -a redis123 --no-auth-warning ping 2>/dev/null"
|
||||
["RabbitMQ"]="docker exec staging-dataLayer-rabbitmq rabbitmq-diagnostics -q ping 2>/dev/null"
|
||||
["MinIO"]="docker exec staging-dataLayer-minio mc ready local 2>/dev/null"
|
||||
["Keycloak"]="curl -s -o /dev/null -w %{http_code} ${KC_BASE}/realms/master 2>/dev/null"
|
||||
["Kong"]="curl -s -o /dev/null -w %{http_code} http://127.0.0.1:18001/status 2>/dev/null"
|
||||
["Framework"]="docker exec didi-framework wget -qO- http://127.0.0.1:3005/health 2>/dev/null"
|
||||
["Agent-V3"]="docker exec didi-agent-v3 wget -qO- http://localhost:24803/api/v3/health 2>/dev/null"
|
||||
)
|
||||
for name in PostgreSQL Redis RabbitMQ MinIO Keycloak Kong Framework Agent-V3; do
|
||||
r=$(eval "${CHECKS[$name]}" || echo FAIL)
|
||||
echo "$r" | grep -qiE "PONG|ok|healthy|ready|service|accepting|200" && ok "$name" || warn "$name: $r"
|
||||
done
|
||||
|
||||
echo
|
||||
echo -e "${GREEN}====================================================================${NC}"
|
||||
echo -e "${GREEN} BUILD LOCAL COMPLET — $PLATFORM_HOSTNAME${NC}"
|
||||
echo -e "${GREEN}====================================================================${NC}"
|
||||
echo
|
||||
echo "Endpoint-uri:"
|
||||
echo " Admin Dashboard: https://localhost:3001/admin (sau prin frontend)"
|
||||
echo " Agent V3 API: http://localhost:24803/api/v3/health"
|
||||
echo " Framework API: intern pe didi-network (port 3005)"
|
||||
echo " Kong Gateway: http://127.0.0.1:18000 (proxy) / :18001 (admin)"
|
||||
echo " Keycloak: ${KC_BASE} (admin/${KC_ADMIN_PASS}, realms didi-clients + didi-admins)"
|
||||
echo " RabbitMQ UI: http://localhost:15672 (admin/rabbitmq123)"
|
||||
echo " MinIO Console: http://localhost:9001 (minioadmin/minio123)"
|
||||
echo " API docs (Swagger): http://localhost:8089 (dacă didi-api-docs rulează)"
|
||||
echo
|
||||
echo "Seed importat automat la primul boot al didi-postgres. Re-sync Redis oricând:"
|
||||
echo " docker exec didi-framework wget -qO- --post-data='' http://127.0.0.1:3005/api/sync-redis"
|
||||
49
backend/production/docker-compose.yml
Normal file
49
backend/production/docker-compose.yml
Normal file
|
|
@ -0,0 +1,49 @@
|
|||
# =============================================================================
|
||||
# DIDI Platform - Production Services
|
||||
# =============================================================================
|
||||
# Redis fallback container only.
|
||||
# Database: PostgreSQL Cluster (10.11.50.167:5000)
|
||||
#
|
||||
# REMOVED 2026-04-28: kong (cluster-based now — HAProxy 10.11.10.175 -> DP1/DP2)
|
||||
# See services/gateway-auth-layer/didiKong/MIGRATION.md
|
||||
#
|
||||
# REMOVED 2026-04-30: keycloak (mutat temporar pe SSO extern, revenit ulterior la Keycloak local)
|
||||
# See services/gateway-auth-layer/didiKeycloak/MIGRATION.md
|
||||
# =============================================================================
|
||||
|
||||
name: didi-production
|
||||
|
||||
services:
|
||||
# ===========================================================================
|
||||
# Redis Cache
|
||||
# ===========================================================================
|
||||
didi-cache:
|
||||
image: redis:7-alpine
|
||||
container_name: didi-cache
|
||||
restart: unless-stopped
|
||||
command: redis-server --requirepass ${REDIS_PASSWORD} --save 60 1 --save 300 10
|
||||
environment:
|
||||
REDIS_PASSWORD: ${REDIS_PASSWORD}
|
||||
# Port removed - accessible only within Docker network
|
||||
# ports:
|
||||
# - "6379:6379"
|
||||
volumes:
|
||||
- didi-cache-data:/data
|
||||
networks:
|
||||
- didi-network
|
||||
healthcheck:
|
||||
test: ["CMD", "redis-cli", "-a", "${REDIS_PASSWORD}", "ping"]
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 3
|
||||
|
||||
networks:
|
||||
didi-network:
|
||||
external: true
|
||||
|
||||
volumes:
|
||||
didi-cache-data:
|
||||
name: didi-production-cache-data
|
||||
# didi-keycloak-data volume kept on disk (didi-production-keycloak-data) for
|
||||
# 1-week archival. Remove after 2026-05-07 if no rollback needed:
|
||||
# docker volume rm didi-production-keycloak-data
|
||||
574
backend/production/full-build.sh
Normal file
574
backend/production/full-build.sh
Normal file
|
|
@ -0,0 +1,574 @@
|
|||
#!/bin/bash
|
||||
# =============================================================================
|
||||
# DIDI Platform - Full Build From Scratch
|
||||
# =============================================================================
|
||||
# Ridica intreaga platforma de la zero.
|
||||
# Ordinea: network -> data-layer -> production (redis, keycloak, kong) ->
|
||||
# framework (sync redis) -> agent-v3 + workers -> admin dashboard
|
||||
#
|
||||
# Cerinte: docker, docker compose v2+, conexiune la PG cluster 10.11.50.167:5000
|
||||
# Rulare: chmod +x full-build.sh && ./full-build.sh [HOSTNAME]
|
||||
# Exemplu: ./full-build.sh <hostname>
|
||||
# ./full-build.sh didi365.eu
|
||||
# Daca nu specifici hostname, il detecteaza automat din hostname -f.
|
||||
# =============================================================================
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
# -- Hostname ---------------------------------------------------------------
|
||||
if [ -n "${1:-}" ]; then
|
||||
PLATFORM_HOSTNAME="$1"
|
||||
else
|
||||
PLATFORM_HOSTNAME=$(hostname -f 2>/dev/null || hostname)
|
||||
fi
|
||||
|
||||
# -- Culori ----------------------------------------------------------------
|
||||
RED='\033[0;31m'; GREEN='\033[0;32m'; YELLOW='\033[1;33m'; CYAN='\033[0;36m'; NC='\033[0m'
|
||||
|
||||
# -- Paths -----------------------------------------------------------------
|
||||
BACKEND="/home/admin365/didi_mono/backend"
|
||||
DATA_LAYER="$BACKEND/services/data-layer"
|
||||
PRODUCTION="$BACKEND/production"
|
||||
FRAMEWORK="$BACKEND/services/orchestration-layer/didiFramework"
|
||||
AGENT_V3="$BACKEND/services/orchestration-layer/agent-v3"
|
||||
KONG_DIR="$BACKEND/services/gateway-auth-layer/didiKong"
|
||||
|
||||
# -- Conexiune PG cluster --------------------------------------------------
|
||||
PG_HOST="10.11.50.167"
|
||||
PG_PORT="5000"
|
||||
PG_USER="bos_interface"
|
||||
PG_PASS="interface"
|
||||
PG_DB="DIDI"
|
||||
|
||||
# -- Functii helper ---------------------------------------------------------
|
||||
log() { echo -e "${CYAN}[$(date +%H:%M:%S)]${NC} $*"; }
|
||||
ok() { echo -e "${GREEN} OK${NC} $*"; }
|
||||
warn() { echo -e "${YELLOW} WARN${NC} $*"; }
|
||||
fail() { echo -e "${RED} FAIL${NC} $*"; exit 1; }
|
||||
|
||||
wait_healthy() {
|
||||
local container="$1"
|
||||
local max_wait="${2:-120}"
|
||||
local elapsed=0
|
||||
log "Astept container $container sa fie healthy (max ${max_wait}s)..."
|
||||
while [ $elapsed -lt $max_wait ]; do
|
||||
local status
|
||||
status=$(docker inspect --format='{{.State.Health.Status}}' "$container" 2>/dev/null || echo "missing")
|
||||
if [ "$status" = "healthy" ]; then
|
||||
ok "$container este healthy"
|
||||
return 0
|
||||
fi
|
||||
sleep 3
|
||||
elapsed=$((elapsed + 3))
|
||||
done
|
||||
warn "$container nu a devenit healthy in ${max_wait}s (status: $status)"
|
||||
return 1
|
||||
}
|
||||
|
||||
pg_query() {
|
||||
# Executa query PG prin orice container care are psql/node disponibil
|
||||
local query="$1"
|
||||
if docker ps --format '{{.Names}}' | grep -q staging-dataLayer-postgres; then
|
||||
docker exec -e PGPASSWORD="$PG_PASS" staging-dataLayer-postgres \
|
||||
psql -h "$PG_HOST" -p "$PG_PORT" -U "$PG_USER" -d "$PG_DB" -tAc "$query" 2>/dev/null
|
||||
elif docker ps --format '{{.Names}}' | grep -q didi-framework; then
|
||||
docker exec didi-framework node -e "
|
||||
const {Pool}=require('pg');
|
||||
const p=new Pool({host:'$PG_HOST',port:$PG_PORT,database:'$PG_DB',user:'$PG_USER',password:'$PG_PASS'});
|
||||
p.query(\`$query\`).then(r=>{r.rows.forEach(row=>console.log(Object.values(row).join('|')));p.end()}).catch(e=>{console.error(e.message);p.end();process.exit(1)});
|
||||
" 2>/dev/null
|
||||
else
|
||||
return 1
|
||||
fi
|
||||
}
|
||||
|
||||
section() {
|
||||
echo ""
|
||||
echo -e "${YELLOW}====================================================================${NC}"
|
||||
echo -e "${YELLOW} $*${NC}"
|
||||
echo -e "${YELLOW}====================================================================${NC}"
|
||||
}
|
||||
|
||||
# =============================================================================
|
||||
section "FAZA 0: Verificari preliminare"
|
||||
# =============================================================================
|
||||
|
||||
log "Verific docker..."
|
||||
docker info >/dev/null 2>&1 || fail "Docker nu ruleaza"
|
||||
ok "Docker activ"
|
||||
|
||||
log "Verific docker compose..."
|
||||
docker compose version >/dev/null 2>&1 || fail "Docker Compose v2 nu este instalat"
|
||||
ok "Docker Compose disponibil"
|
||||
|
||||
# =============================================================================
|
||||
section "FAZA 1: Docker Network"
|
||||
# =============================================================================
|
||||
|
||||
log "Creez reteaua didi-network (daca nu exista)..."
|
||||
docker network create didi-network 2>/dev/null && ok "Retea creata" || ok "Reteaua exista deja"
|
||||
|
||||
# =============================================================================
|
||||
section "FAZA 2: Data Layer (PostgreSQL local, RabbitMQ, MinIO, PgAdmin)"
|
||||
# =============================================================================
|
||||
|
||||
log "Verific daca volumele externe exista..."
|
||||
for vol in didi-staging-postgres-data didi-staging-minio-data didi-staging-pgadmin-data; do
|
||||
docker volume inspect "$vol" >/dev/null 2>&1 && ok "Volum $vol exista" || {
|
||||
log "Creez volum $vol..."
|
||||
docker volume create "$vol"
|
||||
ok "Volum $vol creat"
|
||||
}
|
||||
done
|
||||
|
||||
log "Build + start data-layer..."
|
||||
cd "$DATA_LAYER"
|
||||
|
||||
# Nota: containerul PG local este doar pentru waitlist.
|
||||
# Baza de date principala (DIDI) este pe clusterul extern 10.11.50.167:5000.
|
||||
# Dockerfile-ul custom necesita init.sql + health-check.sh care nu sunt in git (*.sql in .gitignore).
|
||||
# Daca Dockerfile exista SI init.sql e fisier (nu director gol), build custom; altfel, skip.
|
||||
if [ -f didiDatabase/Dockerfile ] && [ -f didiDatabase/init.sql ]; then
|
||||
log "Build imagine didi-staging-postgres..."
|
||||
docker build -t didi-staging-postgres:latest didiDatabase/ 2>&1 | tail -3
|
||||
ok "Imagine postgres construita"
|
||||
else
|
||||
log "Skip build custom PG (init.sql lipseste). Se foloseste imaginea standard postgres:15-alpine."
|
||||
fi
|
||||
|
||||
docker compose up -d --build 2>&1 | tail -5
|
||||
ok "Data layer pornit"
|
||||
|
||||
# Astept serviciile critice
|
||||
wait_healthy staging-dataLayer-rabbitmq 90
|
||||
wait_healthy staging-dataLayer-minio 60
|
||||
|
||||
# =============================================================================
|
||||
section "FAZA 3: Verificare PostgreSQL Cluster extern"
|
||||
# =============================================================================
|
||||
|
||||
log "Testez conexiunea la PG cluster $PG_HOST:$PG_PORT..."
|
||||
|
||||
# Astept sa avem un container cu psql sau node
|
||||
sleep 5
|
||||
|
||||
SCHEMA_COUNT=$(pg_query "SELECT count(*) FROM information_schema.schemata WHERE schema_name IN ('bos_analysis','bos_parammgmt','bos_sysadmin','bos_subscriber')" 2>/dev/null || echo "0")
|
||||
|
||||
if [ "$SCHEMA_COUNT" = "4" ]; then
|
||||
ok "Toate 4 schemele exista in PG cluster — nu ating nimic"
|
||||
elif [ "$SCHEMA_COUNT" = "0" ]; then
|
||||
warn "ZERO scheme bos_* gasite — baza DIDI este goala"
|
||||
|
||||
# Caut exportul complet
|
||||
DIDI_EXPORT=""
|
||||
for candidate in \
|
||||
"$BACKEND/services/data-layer/didiDatabase/DIDI_full_export_2026-07-02.sql" \
|
||||
"$BACKEND/services/data-layer/didiDatabase"/DIDI_full_export_*.sql; do
|
||||
if [ -f "$candidate" ]; then
|
||||
DIDI_EXPORT="$candidate"
|
||||
break
|
||||
fi
|
||||
done
|
||||
|
||||
if [ -n "$DIDI_EXPORT" ]; then
|
||||
EXPORT_SIZE=$(du -h "$DIDI_EXPORT" | cut -f1)
|
||||
log "Gasit export: $DIDI_EXPORT ($EXPORT_SIZE)"
|
||||
log "Baza este GOALA (0 scheme). Import exportul complet..."
|
||||
|
||||
# Astept container-ul postgres local sa fie up (are psql)
|
||||
wait_healthy staging-dataLayer-postgres 60 || true
|
||||
|
||||
# Copiez fisierul in container si import prin psql
|
||||
docker cp "$DIDI_EXPORT" staging-dataLayer-postgres:/tmp/didi_import.sql
|
||||
docker exec -e PGPASSWORD="$PG_PASS" staging-dataLayer-postgres \
|
||||
psql -h "$PG_HOST" -p "$PG_PORT" -U "$PG_USER" -d "$PG_DB" \
|
||||
-f /tmp/didi_import.sql 2>&1 | tail -20
|
||||
|
||||
# Verificare post-import
|
||||
SCHEMA_COUNT_POST=$(pg_query "SELECT count(*) FROM information_schema.schemata WHERE schema_name IN ('bos_analysis','bos_parammgmt','bos_sysadmin','bos_subscriber')" 2>/dev/null || echo "0")
|
||||
if [ "$SCHEMA_COUNT_POST" = "4" ]; then
|
||||
ok "Import reusit — toate 4 schemele exista acum"
|
||||
else
|
||||
fail "Import esuat — doar $SCHEMA_COUNT_POST/4 scheme dupa import"
|
||||
fi
|
||||
|
||||
# Cleanup
|
||||
docker exec staging-dataLayer-postgres rm -f /tmp/didi_import.sql
|
||||
else
|
||||
warn "Nu gasesc fisier DIDI_full_export_*.sql in $BACKEND/services/data-layer/didiDatabase/"
|
||||
warn "Baza DIDI este goala si nu pot importa automat."
|
||||
read -p "Continui fara baza de date? (y/N): " answer
|
||||
[ "$answer" = "y" ] || [ "$answer" = "Y" ] || exit 1
|
||||
fi
|
||||
else
|
||||
warn "Gasit $SCHEMA_COUNT/4 scheme (partial). Nu ating — nu e gol, dar nici complet."
|
||||
pg_query "SELECT schema_name FROM information_schema.schemata WHERE schema_name LIKE 'bos_%' ORDER BY 1" || true
|
||||
echo ""
|
||||
warn "Verifica manual ce lipseste. Importul automat ruleaza DOAR pe baza complet goala."
|
||||
read -p "Continui oricum? (y/N): " answer
|
||||
[ "$answer" = "y" ] || [ "$answer" = "Y" ] || exit 1
|
||||
fi
|
||||
|
||||
# Verific tabelele critice per schema
|
||||
log "Verific tabele critice..."
|
||||
|
||||
CRITICAL_TABLES=(
|
||||
"bos_analysis|analysis_session"
|
||||
"bos_analysis|analysis_verdict"
|
||||
"bos_parammgmt|dimension"
|
||||
"bos_parammgmt|technique"
|
||||
"bos_parammgmt|verdict_category"
|
||||
"bos_parammgmt|component_weight"
|
||||
"bos_parammgmt|component_config"
|
||||
"bos_parammgmt|input_type_profile"
|
||||
"bos_sysadmin|internet_user"
|
||||
"bos_sysadmin|user_credential"
|
||||
"bos_sysadmin|subscription_plan"
|
||||
)
|
||||
|
||||
MISSING=0
|
||||
for entry in "${CRITICAL_TABLES[@]}"; do
|
||||
schema="${entry%%|*}"
|
||||
table="${entry##*|}"
|
||||
EXISTS=$(pg_query "SELECT count(*) FROM information_schema.tables WHERE table_schema='$schema' AND table_name='$table'" 2>/dev/null || echo "0")
|
||||
if [ "$EXISTS" = "1" ]; then
|
||||
ok " $schema.$table"
|
||||
else
|
||||
warn " LIPSA: $schema.$table"
|
||||
MISSING=$((MISSING + 1))
|
||||
fi
|
||||
done
|
||||
|
||||
if [ "$MISSING" -gt 0 ]; then
|
||||
warn "$MISSING tabele critice lipsa. Migrari necesare."
|
||||
fi
|
||||
|
||||
# =============================================================================
|
||||
section "FAZA 4: Production Services (Redis, Keycloak, Kong)"
|
||||
# =============================================================================
|
||||
|
||||
cd "$PRODUCTION"
|
||||
|
||||
# -- Seteaza KC_HOSTNAME_URL in .env pe baza hostname-ului platformei --------
|
||||
log "Configurez Keycloak hostname: $PLATFORM_HOSTNAME"
|
||||
if grep -q "^KC_HOSTNAME_URL=" .env 2>/dev/null; then
|
||||
sed -i "s|^KC_HOSTNAME_URL=.*|KC_HOSTNAME_URL=https://${PLATFORM_HOSTNAME}/auth|" .env
|
||||
ok "KC_HOSTNAME_URL actualizat in .env"
|
||||
else
|
||||
echo "KC_HOSTNAME_URL=https://${PLATFORM_HOSTNAME}/auth" >> .env
|
||||
ok "KC_HOSTNAME_URL adaugat in .env"
|
||||
fi
|
||||
|
||||
log "Start Redis + Keycloak + Kong..."
|
||||
docker compose up -d --build 2>&1 | tail -5
|
||||
ok "Production services pornite"
|
||||
|
||||
wait_healthy didi-cache 30
|
||||
|
||||
log "Verific conexiunea Redis..."
|
||||
REDIS_PONG=$(docker exec didi-cache redis-cli -a redis123 ping 2>/dev/null || echo "FAIL")
|
||||
if [ "$REDIS_PONG" = "PONG" ]; then
|
||||
ok "Redis raspunde"
|
||||
else
|
||||
warn "Redis nu raspunde: $REDIS_PONG"
|
||||
fi
|
||||
|
||||
wait_healthy keycloak 180
|
||||
wait_healthy kong 90
|
||||
|
||||
# -- Configurare Keycloak redirect URIs via Admin API -----------------------
|
||||
section "FAZA 4b: Keycloak - Configurare redirect URIs pentru $PLATFORM_HOSTNAME"
|
||||
|
||||
log "Obtin token admin Keycloak..."
|
||||
KC_ADMIN_USER=$(grep "^KEYCLOAK_ADMIN=" .env | cut -d= -f2-)
|
||||
KC_ADMIN_PASS=$(grep "^KEYCLOAK_ADMIN_PASSWORD=" .env | cut -d= -f2-)
|
||||
|
||||
KC_TOKEN=$(curl -s -X POST "http://localhost:28000/realms/master/protocol/openid-connect/token" \
|
||||
-H "Content-Type: application/x-www-form-urlencoded" \
|
||||
-d "username=${KC_ADMIN_USER}" -d "password=${KC_ADMIN_PASS}" \
|
||||
-d "grant_type=password" -d "client_id=admin-cli" 2>/dev/null \
|
||||
| python3 -c "import sys,json;print(json.load(sys.stdin).get('access_token',''))" 2>/dev/null || echo "")
|
||||
|
||||
if [ -z "$KC_TOKEN" ]; then
|
||||
warn "Nu am obtinut token admin Keycloak. Redirect URIs trebuie configurate manual."
|
||||
else
|
||||
ok "Token admin obtinut"
|
||||
|
||||
# Configureaza didi-web-app
|
||||
log "Configurez client didi-web-app..."
|
||||
WEB_CLIENT_UUID=$(curl -s "http://localhost:28000/admin/realms/didi-clients/clients?clientId=didi-web-app" \
|
||||
-H "Authorization: Bearer $KC_TOKEN" 2>/dev/null \
|
||||
| python3 -c "import sys,json;d=json.load(sys.stdin);print(d[0]['id'] if d else '')" 2>/dev/null || echo "")
|
||||
|
||||
if [ -n "$WEB_CLIENT_UUID" ]; then
|
||||
curl -s -X PUT "http://localhost:28000/admin/realms/didi-clients/clients/$WEB_CLIENT_UUID" \
|
||||
-H "Authorization: Bearer $KC_TOKEN" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d "{
|
||||
\"redirectUris\": [
|
||||
\"https://${PLATFORM_HOSTNAME}/*\",
|
||||
\"http://localhost:3001/*\",
|
||||
\"http://localhost:5173/*\"
|
||||
],
|
||||
\"webOrigins\": [
|
||||
\"https://${PLATFORM_HOSTNAME}\",
|
||||
\"http://localhost:3001\",
|
||||
\"http://localhost:5173\"
|
||||
]
|
||||
}" -w "" -o /dev/null 2>/dev/null
|
||||
ok "didi-web-app: redirect URI -> https://${PLATFORM_HOSTNAME}/*"
|
||||
else
|
||||
warn "Client didi-web-app nu gasit in Keycloak"
|
||||
fi
|
||||
|
||||
# Configureaza admin-dashboard
|
||||
log "Configurez client admin-dashboard..."
|
||||
ADMIN_CLIENT_UUID=$(curl -s "http://localhost:28000/admin/realms/didi-clients/clients?clientId=admin-dashboard" \
|
||||
-H "Authorization: Bearer $KC_TOKEN" 2>/dev/null \
|
||||
| python3 -c "import sys,json;d=json.load(sys.stdin);print(d[0]['id'] if d else '')" 2>/dev/null || echo "")
|
||||
|
||||
if [ -n "$ADMIN_CLIENT_UUID" ]; then
|
||||
curl -s -X PUT "http://localhost:28000/admin/realms/didi-clients/clients/$ADMIN_CLIENT_UUID" \
|
||||
-H "Authorization: Bearer $KC_TOKEN" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d "{
|
||||
\"redirectUris\": [
|
||||
\"https://${PLATFORM_HOSTNAME}/*\",
|
||||
\"https://${PLATFORM_HOSTNAME}/admin/*\",
|
||||
\"http://localhost:3003/*\"
|
||||
],
|
||||
\"webOrigins\": [
|
||||
\"https://${PLATFORM_HOSTNAME}\",
|
||||
\"http://localhost:3003\"
|
||||
]
|
||||
}" -w "" -o /dev/null 2>/dev/null
|
||||
ok "admin-dashboard: redirect URI -> https://${PLATFORM_HOSTNAME}/*"
|
||||
else
|
||||
warn "Client admin-dashboard nu gasit in Keycloak"
|
||||
fi
|
||||
fi
|
||||
|
||||
# =============================================================================
|
||||
section "FAZA 5: didiFramework (CRUD backend + sync Redis)"
|
||||
# =============================================================================
|
||||
|
||||
cd "$FRAMEWORK"
|
||||
|
||||
log "Build + start didiFramework..."
|
||||
docker compose up -d --build 2>&1 | tail -5
|
||||
ok "didiFramework pornit"
|
||||
|
||||
wait_healthy didi-framework 60
|
||||
|
||||
# Verific health-ul complet (PG + MinIO)
|
||||
log "Verific health didiFramework..."
|
||||
HEALTH=$(docker exec didi-framework wget -qO- "http://127.0.0.1:3005/health/all" 2>/dev/null || echo "{}")
|
||||
echo " $HEALTH"
|
||||
|
||||
# =============================================================================
|
||||
section "FAZA 6: Sincronizare Framework -> Redis"
|
||||
# =============================================================================
|
||||
|
||||
log "Trigger sync-redis (incarca parametri framework in Redis)..."
|
||||
SYNC_RESULT=$(docker exec didi-framework wget -qO- --post-data='' "http://127.0.0.1:3005/api/sync-redis" 2>/dev/null || echo "FAIL")
|
||||
if echo "$SYNC_RESULT" | grep -q '"success"'; then
|
||||
ok "Sync Redis reusit"
|
||||
echo " $SYNC_RESULT" | head -c 200
|
||||
echo ""
|
||||
else
|
||||
warn "Sync Redis posibil esuat: $SYNC_RESULT"
|
||||
warn "Poti face sync manual mai tarziu: POST http://localhost:3005/api/sync-redis"
|
||||
fi
|
||||
|
||||
# Verific ca cheile au fost scrise
|
||||
log "Verific chei framework in Redis..."
|
||||
KEY_COUNT=$(docker exec didi-cache redis-cli -a redis123 keys "didi:framework:*" 2>/dev/null | wc -l)
|
||||
CONFIG_COUNT=$(docker exec didi-cache redis-cli -a redis123 keys "didi:config:*" 2>/dev/null | wc -l)
|
||||
ok "Chei framework: $KEY_COUNT | Chei config: $CONFIG_COUNT"
|
||||
|
||||
if [ "$KEY_COUNT" -lt 5 ]; then
|
||||
warn "Prea putine chei framework ($KEY_COUNT). Sync-ul poate sa nu fi functionat."
|
||||
warn "Verifica manual: docker exec didi-cache redis-cli -a redis123 keys 'didi:framework:*'"
|
||||
fi
|
||||
|
||||
# =============================================================================
|
||||
section "FAZA 7: MinIO - Initializare bucket-uri"
|
||||
# =============================================================================
|
||||
|
||||
log "Verific bucket-urile MinIO..."
|
||||
BUCKET_LIST=$(docker exec staging-dataLayer-minio mc ls local/ 2>/dev/null || echo "")
|
||||
|
||||
REQUIRED_BUCKETS=("uploads" "text-files" "image-files" "audio-files" "video-files" "document-files" "pipeline-artifacts" "backups")
|
||||
BUCKETS_MISSING=0
|
||||
|
||||
for bucket in "${REQUIRED_BUCKETS[@]}"; do
|
||||
if echo "$BUCKET_LIST" | grep -q "$bucket"; then
|
||||
ok " Bucket: $bucket"
|
||||
else
|
||||
warn " LIPSA bucket: $bucket"
|
||||
BUCKETS_MISSING=$((BUCKETS_MISSING + 1))
|
||||
fi
|
||||
done
|
||||
|
||||
if [ "$BUCKETS_MISSING" -gt 0 ]; then
|
||||
log "Rulez init-buckets.sh..."
|
||||
if [ -f "$DATA_LAYER/didiStorage/init-buckets.sh" ]; then
|
||||
docker exec -e MINIO_HOST=localhost -e MINIO_PORT=9000 \
|
||||
-e MINIO_ROOT_USER=minioadmin -e MINIO_ROOT_PASSWORD=minio123 \
|
||||
staging-dataLayer-minio sh -c "$(cat $DATA_LAYER/didiStorage/init-buckets.sh)" 2>&1 | tail -5
|
||||
ok "Bucket-uri initializate"
|
||||
else
|
||||
warn "init-buckets.sh nu exista. Creeaza bucket-urile manual."
|
||||
fi
|
||||
else
|
||||
ok "Toate bucket-urile exista"
|
||||
fi
|
||||
|
||||
# =============================================================================
|
||||
section "FAZA 8: Agent V3 + Workers"
|
||||
# =============================================================================
|
||||
|
||||
cd "$AGENT_V3"
|
||||
|
||||
# Verific ca .env exista (contine API keys)
|
||||
if [ ! -f .env ]; then
|
||||
warn ".env lipseste in $AGENT_V3"
|
||||
warn "Fisierul trebuie sa contina: OPENROUTER_API_KEY, OPENAI_API_KEY, GROQ_API_KEY, M17_WHISPER_TOKEN"
|
||||
warn "Fara aceste chei, analizele LLM nu vor functiona."
|
||||
read -p "Continui fara .env? (y/N): " answer
|
||||
[ "$answer" = "y" ] || [ "$answer" = "Y" ] || exit 1
|
||||
else
|
||||
ok ".env exista (API keys configurate)"
|
||||
# Verific cheile critice
|
||||
for key in OPENROUTER_API_KEY OPENAI_API_KEY GROQ_API_KEY; do
|
||||
val=$(grep "^$key=" .env 2>/dev/null | cut -d= -f2-)
|
||||
if [ -z "$val" ]; then
|
||||
warn " $key este gol in .env"
|
||||
else
|
||||
ok " $key configurat (${#val} caractere)"
|
||||
fi
|
||||
done
|
||||
fi
|
||||
|
||||
log "Build + start agent-v3 + toti workerii..."
|
||||
docker compose up -d --build 2>&1 | tail -10
|
||||
ok "Agent V3 + workers porniti"
|
||||
|
||||
wait_healthy didi-agent-v3 60
|
||||
|
||||
# Verific health
|
||||
log "Verific health agent-v3..."
|
||||
AGENT_HEALTH=$(docker exec didi-agent-v3 wget -qO- "http://localhost:24803/api/v3/health" 2>/dev/null || echo "FAIL")
|
||||
echo " $AGENT_HEALTH"
|
||||
|
||||
# Verific workerii
|
||||
log "Verific workerii..."
|
||||
WORKERS=$(docker ps --format '{{.Names}}' | grep -c "agent-v3-worker" || true)
|
||||
AGGREGATORS=$(docker ps --format '{{.Names}}' | grep -c "verdict-aggregator" || true)
|
||||
ok "Workers activi: $WORKERS | Aggregators: $AGGREGATORS"
|
||||
|
||||
# =============================================================================
|
||||
section "FAZA 9: Admin Dashboard"
|
||||
# =============================================================================
|
||||
|
||||
# Admin dashboard este in data-layer docker-compose (didi-admin container)
|
||||
log "Verific admin dashboard..."
|
||||
if docker ps --format '{{.Names}}' | grep -q didi-admin; then
|
||||
ok "didi-admin deja ruleaza"
|
||||
else
|
||||
log "Admin dashboard nu ruleaza. Rebuild..."
|
||||
cd "$BACKEND/admin-dashboard"
|
||||
if [ -f Dockerfile ]; then
|
||||
docker build -t didi-admin:latest . 2>&1 | tail -5
|
||||
ok "Imagine admin-dashboard construita"
|
||||
fi
|
||||
cd "$DATA_LAYER"
|
||||
docker compose up -d didi-admin 2>&1 | tail -3
|
||||
fi
|
||||
|
||||
wait_healthy didi-admin 60 || true
|
||||
|
||||
# =============================================================================
|
||||
section "FAZA 10: Kong Build (imagine custom)"
|
||||
# =============================================================================
|
||||
|
||||
log "Verific imaginea Kong..."
|
||||
if docker images didi-kong:latest --format '{{.ID}}' | head -1 | grep -q .; then
|
||||
ok "Imaginea didi-kong:latest exista"
|
||||
else
|
||||
log "Build imagine didi-kong..."
|
||||
if [ -d "$KONG_DIR" ] && [ -f "$KONG_DIR/Dockerfile" ]; then
|
||||
docker build -t didi-kong:latest "$KONG_DIR" 2>&1 | tail -3
|
||||
ok "Imaginea Kong construita. Restart Kong..."
|
||||
cd "$PRODUCTION"
|
||||
docker compose up -d kong 2>&1 | tail -3
|
||||
wait_healthy kong 90 || true
|
||||
else
|
||||
warn "Nu gasesc Dockerfile Kong la $KONG_DIR"
|
||||
fi
|
||||
fi
|
||||
|
||||
# =============================================================================
|
||||
section "VERIFICARE FINALA"
|
||||
# =============================================================================
|
||||
|
||||
echo ""
|
||||
log "Status toate containerele DIDI:"
|
||||
echo ""
|
||||
printf "%-45s %-20s %s\n" "CONTAINER" "STATUS" "PORTS"
|
||||
printf "%-45s %-20s %s\n" "---------" "------" "-----"
|
||||
docker ps --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}" --filter "network=didi-network" 2>/dev/null | tail -n +2 | sort | while read line; do
|
||||
echo " $line"
|
||||
done
|
||||
|
||||
echo ""
|
||||
log "Health checks rapide:"
|
||||
|
||||
# Lista de verificari
|
||||
declare -A CHECKS=(
|
||||
["Redis"]="docker exec didi-cache redis-cli -a redis123 ping 2>/dev/null"
|
||||
["Framework"]="docker exec didi-framework wget -qO- http://127.0.0.1:3005/health 2>/dev/null"
|
||||
["Agent-V3"]="docker exec didi-agent-v3 wget -qO- http://localhost:24803/api/v3/health 2>/dev/null"
|
||||
["RabbitMQ"]="docker exec staging-dataLayer-rabbitmq rabbitmq-diagnostics -q ping 2>/dev/null"
|
||||
["MinIO"]="docker exec staging-dataLayer-minio mc ready local 2>/dev/null"
|
||||
)
|
||||
|
||||
for name in Redis Framework Agent-V3 RabbitMQ MinIO; do
|
||||
result=$(eval "${CHECKS[$name]}" || echo "FAIL")
|
||||
if echo "$result" | grep -qiE "PONG|ok|healthy|ready|service|READY"; then
|
||||
ok "$name"
|
||||
else
|
||||
warn "$name: $result"
|
||||
fi
|
||||
done
|
||||
|
||||
echo ""
|
||||
|
||||
# Verific chei Redis finale
|
||||
FRAMEWORK_KEYS=$(docker exec didi-cache redis-cli -a redis123 keys "didi:framework:*" 2>/dev/null | wc -l)
|
||||
CONFIG_KEYS=$(docker exec didi-cache redis-cli -a redis123 keys "didi:config:*" 2>/dev/null | wc -l)
|
||||
log "Redis: $FRAMEWORK_KEYS chei framework, $CONFIG_KEYS chei config"
|
||||
|
||||
echo ""
|
||||
echo -e "${GREEN}====================================================================${NC}"
|
||||
echo -e "${GREEN} BUILD COMPLET${NC}"
|
||||
echo -e "${GREEN}====================================================================${NC}"
|
||||
echo ""
|
||||
echo "Platforma configurata pe: $PLATFORM_HOSTNAME"
|
||||
echo ""
|
||||
echo "Endpoint-uri disponibile:"
|
||||
echo " Frontend: https://${PLATFORM_HOSTNAME}"
|
||||
echo " Admin Dashboard: https://${PLATFORM_HOSTNAME}/admin"
|
||||
echo " Agent V3 API: http://localhost:24803/api/v3/health (doar local)"
|
||||
echo " Framework API: intern pe Docker network (port 3005)"
|
||||
echo " Kong Gateway: https://localhost:443"
|
||||
echo " Keycloak: http://localhost:28000"
|
||||
echo " Keycloak Auth: https://${PLATFORM_HOSTNAME}/auth"
|
||||
echo " RabbitMQ UI: http://localhost:15672 (admin/rabbitmq123)"
|
||||
echo " MinIO Console: http://localhost:9001 (minioadmin/minio123)"
|
||||
echo " PgAdmin: http://localhost:5050 (admin@example.com/admin123)"
|
||||
echo ""
|
||||
echo "Daca sync Redis nu a mers, ruleaza manual:"
|
||||
echo " curl -X POST http://localhost:3005/api/sync-redis"
|
||||
echo " (sau din interiorul Docker: docker exec didi-framework wget -qO- --post-data='' http://127.0.0.1:3005/api/sync-redis)"
|
||||
echo ""
|
||||
112
backend/production/migrate-to-cluster.sh
Normal file
112
backend/production/migrate-to-cluster.sh
Normal file
|
|
@ -0,0 +1,112 @@
|
|||
#!/bin/bash
|
||||
# =============================================================================
|
||||
# Migration Script: Move Kong & Keycloak DBs to PostgreSQL Cluster
|
||||
# =============================================================================
|
||||
|
||||
set -e
|
||||
|
||||
# Colors
|
||||
RED='\033[0;31m'
|
||||
GREEN='\033[0;32m'
|
||||
YELLOW='\033[1;33m'
|
||||
NC='\033[0m'
|
||||
|
||||
# Cluster connection
|
||||
CLUSTER_HOST="10.11.50.167"
|
||||
CLUSTER_PORT="5000"
|
||||
CLUSTER_USER="bos_interface"
|
||||
CLUSTER_PASS="interface"
|
||||
|
||||
# Local postgres container
|
||||
LOCAL_PG="staging-dataLayer-postgres"
|
||||
|
||||
echo -e "${YELLOW}=== DIDI Database Migration to Cluster ===${NC}"
|
||||
echo ""
|
||||
|
||||
# Step 1: Create users in cluster
|
||||
echo -e "${YELLOW}[1/6] Creating users in cluster...${NC}"
|
||||
docker exec -e PGPASSWORD=$CLUSTER_PASS $LOCAL_PG psql -h $CLUSTER_HOST -p $CLUSTER_PORT -U $CLUSTER_USER -d DIDI << 'EOF'
|
||||
-- Create kong user if not exists
|
||||
DO $$
|
||||
BEGIN
|
||||
IF NOT EXISTS (SELECT FROM pg_roles WHERE rolname = 'kong') THEN
|
||||
CREATE ROLE kong WITH LOGIN PASSWORD 'kong123';
|
||||
END IF;
|
||||
END $$;
|
||||
|
||||
-- Create keycloak user if not exists
|
||||
DO $$
|
||||
BEGIN
|
||||
IF NOT EXISTS (SELECT FROM pg_roles WHERE rolname = 'keycloak') THEN
|
||||
CREATE ROLE keycloak WITH LOGIN PASSWORD 'keycloak123';
|
||||
END IF;
|
||||
END $$;
|
||||
|
||||
SELECT rolname FROM pg_roles WHERE rolname IN ('kong', 'keycloak');
|
||||
EOF
|
||||
echo -e "${GREEN}✓ Users created${NC}"
|
||||
|
||||
# Step 2: Dump local databases
|
||||
echo -e "${YELLOW}[2/6] Dumping local databases...${NC}"
|
||||
mkdir -p /tmp/db-migration
|
||||
|
||||
docker exec $LOCAL_PG pg_dump -U postgres -Fc kong_db > /tmp/db-migration/kong_db.dump
|
||||
echo " - kong_db dumped ($(du -h /tmp/db-migration/kong_db.dump | cut -f1))"
|
||||
|
||||
docker exec $LOCAL_PG pg_dump -U postgres -Fc keycloak_db > /tmp/db-migration/keycloak_db.dump
|
||||
echo " - keycloak_db dumped ($(du -h /tmp/db-migration/keycloak_db.dump | cut -f1))"
|
||||
echo -e "${GREEN}✓ Dumps complete${NC}"
|
||||
|
||||
# Step 3: Create databases in cluster
|
||||
echo -e "${YELLOW}[3/6] Creating databases in cluster...${NC}"
|
||||
docker exec -e PGPASSWORD=$CLUSTER_PASS $LOCAL_PG psql -h $CLUSTER_HOST -p $CLUSTER_PORT -U $CLUSTER_USER -d DIDI << 'EOF'
|
||||
-- Create kong_db if not exists
|
||||
SELECT 'CREATE DATABASE kong_db OWNER kong'
|
||||
WHERE NOT EXISTS (SELECT FROM pg_database WHERE datname = 'kong_db')\gexec
|
||||
|
||||
-- Create keycloak_db if not exists
|
||||
SELECT 'CREATE DATABASE keycloak_db OWNER keycloak'
|
||||
WHERE NOT EXISTS (SELECT FROM pg_database WHERE datname = 'keycloak_db')\gexec
|
||||
|
||||
\l kong_db keycloak_db
|
||||
EOF
|
||||
echo -e "${GREEN}✓ Databases created${NC}"
|
||||
|
||||
# Step 4: Restore to cluster
|
||||
echo -e "${YELLOW}[4/6] Restoring databases to cluster...${NC}"
|
||||
|
||||
# Copy dumps to container
|
||||
docker cp /tmp/db-migration/kong_db.dump $LOCAL_PG:/tmp/
|
||||
docker cp /tmp/db-migration/keycloak_db.dump $LOCAL_PG:/tmp/
|
||||
|
||||
# Restore kong_db
|
||||
echo " - Restoring kong_db..."
|
||||
docker exec -e PGPASSWORD=$CLUSTER_PASS $LOCAL_PG pg_restore -h $CLUSTER_HOST -p $CLUSTER_PORT -U $CLUSTER_USER -d kong_db --no-owner --no-acl --clean --if-exists /tmp/kong_db.dump 2>/dev/null || true
|
||||
|
||||
# Restore keycloak_db
|
||||
echo " - Restoring keycloak_db..."
|
||||
docker exec -e PGPASSWORD=$CLUSTER_PASS $LOCAL_PG pg_restore -h $CLUSTER_HOST -p $CLUSTER_PORT -U $CLUSTER_USER -d keycloak_db --no-owner --no-acl --clean --if-exists /tmp/keycloak_db.dump 2>/dev/null || true
|
||||
|
||||
echo -e "${GREEN}✓ Databases restored${NC}"
|
||||
|
||||
# Step 5: Grant permissions
|
||||
echo -e "${YELLOW}[5/6] Granting permissions...${NC}"
|
||||
docker exec -e PGPASSWORD=$CLUSTER_PASS $LOCAL_PG psql -h $CLUSTER_HOST -p $CLUSTER_PORT -U $CLUSTER_USER -d kong_db -c "GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA public TO kong; GRANT ALL PRIVILEGES ON ALL SEQUENCES IN SCHEMA public TO kong;"
|
||||
docker exec -e PGPASSWORD=$CLUSTER_PASS $LOCAL_PG psql -h $CLUSTER_HOST -p $CLUSTER_PORT -U $CLUSTER_USER -d keycloak_db -c "GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA public TO keycloak; GRANT ALL PRIVILEGES ON ALL SEQUENCES IN SCHEMA public TO keycloak;"
|
||||
echo -e "${GREEN}✓ Permissions granted${NC}"
|
||||
|
||||
# Step 6: Verify
|
||||
echo -e "${YELLOW}[6/6] Verifying migration...${NC}"
|
||||
echo "Kong tables:"
|
||||
docker exec -e PGPASSWORD=$CLUSTER_PASS $LOCAL_PG psql -h $CLUSTER_HOST -p $CLUSTER_PORT -U $CLUSTER_USER -d kong_db -c "SELECT count(*) as tables FROM information_schema.tables WHERE table_schema = 'public';"
|
||||
echo "Keycloak tables:"
|
||||
docker exec -e PGPASSWORD=$CLUSTER_PASS $LOCAL_PG psql -h $CLUSTER_HOST -p $CLUSTER_PORT -U $CLUSTER_USER -d keycloak_db -c "SELECT count(*) as tables FROM information_schema.tables WHERE table_schema = 'public';"
|
||||
|
||||
echo ""
|
||||
echo -e "${GREEN}=== Migration Complete ===${NC}"
|
||||
echo ""
|
||||
echo "Next steps:"
|
||||
echo " 1. Stop old containers: docker stop kong keycloak"
|
||||
echo " 2. Remove old containers: docker rm kong keycloak"
|
||||
echo " 3. Start new services: cd /home/admin365/didi_mono/backend/production && docker compose up -d"
|
||||
echo ""
|
||||
Loading…
Add table
Add a link
Reference in a new issue