clossers/didi-lot2-backend
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

livrare lot 2

Browse source
This commit is contained in:
EVOTECH IT SRL 2026-07-10 03:39:53 -07:00
commit 8ecc78e729
763 changed files with 164593 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

352
backend/services/data-layer/didiStorage/INDEX.md Normal file
View file

@ -0,0 +1,352 @@
# didiStorage - Index
Stocare fisiere media pentru platforma DIDI. Container MinIO (S3-compatibil) local pe masina de deployment. Nu contine cod custom -- doar configurare si script de initializare.
## Productie activa (LOCAL)
DIDI scrie **LOCAL** pe containerul MinIO `staging-dataLayer-minio:9000` (pe `didi-network`), intr-un singur bucket `didi-prod`. Decizie: stabilitate + zero dependinte externe. Clusterul MinIO managed extern (4 noduri, erasure coding EC:2, HAProxy + keepalived VRRP) ramane configurat ca **fallback de urgenta pentru HA**, activabil cu `minio-switch.sh cluster`, dar nu este folosit operational acum.
| Mediu | Endpoint | Bucket | Credentiale |
|-------|----------|--------|-------------|
| **Productie (LOCAL — activ)** | `staging-dataLayer-minio:9000` (expus `0.0.0.0:9000`) | `didi-prod` (single bucket) | `didi-prod` / `627074a6...` |
| Fallback HA (cluster — inactiv) | `<minio-host>:9000` (VIP `10.11.10.128`) | `didi-prod` | didi-prod / `.cluster-credentials.env` |
Switch local/cluster: `agent-v3/scripts/minio-switch.sh local|cluster` (modifica `.env` + reseteaza containerele). Detalii migrare: `agent-v3/MIGRATION_MINIO.md`.
Restrictia cheie mostenita din arhitectura: credentialele `didi-prod` au `s3:*` **doar pe bucket-ul propriu** — nu se creeaza bucket-uri noi. De aici single-bucket architecture (vezi sectiunea urmatoare), pastrata si local.
## Container local (activ)
**Imagine**: minio/minio:RELEASE.2024-08-29T01-40-52Z
**Container**: staging-dataLayer-minio
**Port API**: 9000 (Docker network + expus pe host `0.0.0.0:9000`)
**Port Console**: 9001 (expus pe host `0.0.0.0:9001`)
**Bucket DIDI**: `didi-prod` (singurul bucket)
**Credentiale DIDI**: `didi-prod` / `627074a6...` (din `.env`)
**Volume**: didi-staging-minio-data:/data
---
## Ce stocheaza
1. **Fisiere uploadate de utilizatori** -- imagini, audio, video, documente
2. **Fisiere procesate de agent-v3** -- video downloadat, cadre extrase, transcrieri
3. **Artefacte pipeline** -- rezultate analiza (cu versionare)
4. **Bucket-uri per utilizator** -- fisiere organizate pe foldere tipizate
---
## Single-bucket architecture (refactor 2026-04-25)
Productia foloseste un singur bucket `didi-prod`, iar separarea logica se face prin **prefix-uri**, nu bucket-uri distincte. Numele de prefix-uri sistem sunt identice cu numele bucket-urilor vechi pentru ca URL-urile vechi sa ramana interpretabile.
```
didi-prod/
uploads/ -- upload-uri generale / fallback (legacy "uploads")
image-files/ -- imagini (legacy bucket "image-files")
audio-files/ -- audio (legacy bucket "audio-files")
video-files/ -- video (legacy bucket "video-files")
text-files/ -- text (legacy bucket "text-files")
document-files/ -- PDF, Office (legacy bucket "document-files")
pipeline-artifacts/ -- rezultate analiza (legacy bucket "pipeline-artifacts")
users/{userId}/ -- namespace per utilizator (inlocuieste bucket-urile "user-{id}")
images/
videos/
videos/frames/ -- cadre extrase din video (scrise de media-preprocess worker, citite de techniques + ai-tampered)
audio-files/
text-files/
```
### De ce single-bucket
- Arhitectura a fost proiectata pentru credentiale cu `s3:*` limitat la un singur bucket pre-creat (`didi-prod`), fara `s3:CreateBucket` — pastrata identic si pe MinIO local pentru portabilitate cluster.
- Quota tracking simplificat: nu mai depindem de tag-uri pe bucket; storage-ul utilizatorilor este urmarit in PG (`bos_sysadmin.internet_user.storage_used_bytes` + `storage_limit_bytes`, vezi migration `010_add_user_storage_quota.sql` din didiFramework).
- Separare logica prin prefix-uri, nu prin bucket-uri distincte — acelasi layout functioneaza local si pe cluster fara modificari de cod.
### Backward compat
Caller-ii care paseaza bucket-uri vechi (`user-3`, `image-files`) sunt rezolvati automat la canonic `didi-prod/<full-key>`:
| URL primit | Bucket rezolvat | Key rezolvat |
|---|---|---|
| `user-3/images/abc.jpg` | `didi-prod` | `users/3/images/abc.jpg` |
| `image-files/foo.jpg` | `didi-prod` | `image-files/foo.jpg` |
| `didi-prod/users/3/images/abc.jpg` | `didi-prod` | `users/3/images/abc.jpg` (passthrough) |
Implementat in:
- `didiFramework/src/config/minio.ts` -- `resolveBucketRequest(bucket, key)` + constanta `BUCKET = process.env.MINIO_BUCKET || 'didi-prod'`.
- `agent-v3/src/shared/media/media-service.ts` -- `proxyFile()` (ownership check accepta atat `user-{N}` cat si prefix `users/{N}/`) + `uploadFile()` foloseste prefix `users/{userId}/{folder}/`.
### Lifecycle si versionare
- Politicile de lifecycle (90 zile pentru transient, retentie permanenta pentru `pipeline-artifacts/`, `backups/`) se aplica pe bucket-ul `didi-prod` prin prefix; pe MinIO local pot fi setate cu `mc ilm` (optional).
- Versionarea pentru `pipeline-artifacts/` si `backups/` este pastrata la nivel de bucket.
- Daca se comuta pe cluster (`minio-switch.sh cluster`), lifecycle-ul devine responsabilitatea operatorilor cluster-ului (nu detinem bucket-ul acolo).
### Mod legacy (multi-bucket)
Pentru referinta — modul vechi avea 8 bucket-uri sistem (`uploads`, `text-files`, `image-files`, `audio-files`, `video-files`, `document-files`, `pipeline-artifacts`, `backups`) plus bucket-uri dinamice `user-{id}` create la primul login. Acest layout a fost inlocuit de single-bucket `didi-prod` cu prefix-uri.
---
## Limite dimensiune fisiere
| Tip | Limita | MIME types |
|-----|--------|------------|
| Imagini | 20 MB | image/jpeg, image/png, image/gif, image/webp, image/bmp, image/svg+xml |
| Audio | 100 MB | audio/mpeg, audio/wav, audio/ogg, audio/webm, audio/flac, audio/mp4, audio/x-m4a |
| Video | 500 MB | video/mp4, video/webm, video/quicktime, video/x-msvideo, video/x-matroska |
| Text | 10 MB | text/plain, text/html, text/markdown, text/csv |
| Documente | 50 MB | application/pdf, application/msword, application/vnd.openxmlformats-* |
Rutarea automata: fisierul e pus in bucket-ul corespunzator MIME type-ului.
---
## Cine scrie in MinIO
| Serviciu | Ce scrie | Locatie (bucket `didi-prod` local) | Logica in fisier |
|----------|----------|--------|------------------|
| didiFramework (uploads) | Fisiere uploadate via API | `didi-prod/users/{id}/{mimeFolder}/...` (fallback `didi-prod/{mimeBucket}/`) | didiFramework/src/routes/uploads.ts |
| didiFramework (auth) | (Nu mai creeaza bucket) Logging registration; quota in PG | -- | didiFramework/src/routes/auth.ts + migration 010_add_user_storage_quota.sql |
| agent-v3 (media upload) | Fisiere uploadate direct sau via multer | `didi-prod/users/{userId}/...` (fallback `didi-prod/uploads/{userId}/`) | agent-v3/src/api/routes.ts -> media-service.ts |
| agent-v3 (media-preprocess worker) | Video downloadat, cadre extrase ffmpeg, audio extras pentru transcript | `didi-prod/users/{userId}/videos/frames/...` (cand frame-urile sunt persistate); altfel /tmp efemer | agent-v3/src/queue/workers/media-preprocess-worker.ts + shared/media/video-processor.ts |
| agent-v3 (pipeline) | Imagini downloadate din URL-uri | `didi-prod/users/{userId}/images/...` (fallback `didi-prod/uploads/{userId}/`) | agent-v3/src/api/pipeline-routes.ts |
Nota media-preprocess: workerul ruleaza inaintea componentelor de analiza (techniques, ai-tampered, claims) si centralizeaza descarcarea + ffmpeg + transcript + 2× vision. Frame-urile extrase sunt apoi consumate de workerii de techniques / ai-tampered fara duplicare. Cand persistarea frame-urilor in MinIO este activa, prefixul folosit este `users/{userId}/videos/frames/` (cf. `USER_BUCKET_FOLDERS.FRAMES`).
## Cine citeste din MinIO
| Serviciu | Ce citeste | Cum |
|----------|-----------|-----|
| agent-v3 (media proxy) | Servire fisiere catre client | GET /api/v3/media/file/:bucket/:objectKey (proxy cu range support) |
| agent-v3 (vision) | Imagini pentru modele LLM locale | URL intern direct catre MinIO local (host-ul de deployment :9000/bucket/key) |
| agent-v3 (transcription) | Audio/video pentru transcriere | URL presemnat sau intern |
| didiFramework (uploads) | Info fisier + URL presemnat | GET /api/uploads/:fileId |
| Clienti externi | Download fisiere | URL presemnat (1 ora) sau proxy agent-v3 |
---
## URL-uri si acces
### URL public (prin proxy agent-v3)
```
https://didi365.eu/api/v3/media/file/{bucket}/{objectKey}
Exemplu (canonic): https://didi365.eu/api/v3/media/file/didi-prod/users/3/audio-files/1771883851173-audio.mp3
Exemplu (legacy): https://didi365.eu/api/v3/media/file/user-3/audio-files/1771883851173-audio.mp3 (rezolvat la didi-prod)
```
Suporta HTTP Range requests (streaming audio/video). Proxy-ul rezolva atat URL-uri legacy (`user-{id}/...`, `image-files/...`) cat si forme canonice (`didi-prod/users/{id}/...`).
### URL presemnat (direct MinIO local)
```
http://<host-deployment>:9000/didi-prod/{objectKey}?X-Amz-Algorithm=...&X-Amz-Signature=...
```
Valabilitate: 1 ora (GET), 1 ora (PUT upload). Path-style obligatoriu (`forcePathStyle=true`).
### URL intern (pentru modele LLM locale)
```
http://<host-deployment>:9000/didi-prod/{objectKey}
```
Modelele locale (Qwen Vision, pe masinile GPU) nu pot accesa `didi365.eu`, asa ca URL-urile publice sunt convertite la URL-uri MinIO interne catre containerul local `staging-dataLayer-minio` (expus pe host-ul de deployment `:9000`). Logica: `agent-v3/src/shared/media/vision.ts` (`INTERNAL_MEDIA_BASE`).
---
## Integrare cu serviciile
### didiFramework -- configurare MinIO principala
Fisier: `didiFramework/src/config/minio.ts` (refactor 2026-04-25 pentru single-bucket)
Constante:
- `BUCKET` -- bucket fix din `MINIO_BUCKET` env (default `didi-prod`).
- `BUCKETS` -- prefix-uri sistem (`uploads`, `image-files`, `audio-files`, `video-files`, `text-files`, `document-files`, `pipeline-artifacts`).
- `USER_BUCKET_FOLDERS` -- foldere per utilizator (`images`, `videos`, `audio-files`, `text-files`, `videos/frames`).
- `MIME_TO_BUCKET` -- routing MIME -> prefix sistem.
Exporta:
- `getMinioClient()` -- client singleton.
- `checkMinioHealth()` -- health check via `listBuckets()`.
- `resolveBucketRequest(bucket, key)` -- traduce input legacy (`user-3`, `image-files`) la `(BUCKET, fullKey)` canonic.
- `userObjectKey(userId, folder, filename)` -- construieste `users/{userId}/{folder}/{filename}`.
- `ensureBucket(name)` -- **no-op in single-bucket mode** (logging only). Pentru bucket-uri sistem legacy / `user-{N}` returneaza fara eroare.
- `uploadBuffer(bucket, name, buffer, mimeType, metadata)` -- upload (rezolva bucket-ul intern).
- `deleteObject(bucket, name)` / `getObjectInfo(bucket, name)` / `listObjects(bucket, prefix, maxKeys)`.
- `getPresignedUrl(bucket, name, expiry)` -- URL download (default 1 ora).
- `getPresignedPutUrl(bucket, name, expiry)` -- URL upload (default 1 ora).
- `getDirectUrl(bucket, name)` -- URL direct fara semnatura.
- `createUserBucket(userId, email, planId, planName, storageLimitGb)` -- **lazy in single-bucket mode**: namespace-ul `users/{id}/` "exista" doar cand are obiecte; functia logheaza si scrie quota in PG.
- `getUserBucketUsage(userId)` -- listObjects pe `users/{id}/`, returneaza bytes + count.
- `getUserBucketMetadata(userId)` -- thin shim (in single-bucket mode metadata e in PG, nu in tag-uri).
- `updateUserBucketMetadata(userId, planId, planName, storageLimitGb)` -- no-op pentru bucket tags; caller-ul scrie in PG.
Quota tracking: migrarea `sql/migrations/010_add_user_storage_quota.sql` adauga coloanele `storage_used_bytes` si `storage_limit_bytes` la `bos_sysadmin.internet_user`. Tag-urile vechi (`storage-limit-gb` etc.) nu mai sunt folosite.
### agent-v3 -- MediaService
Fisier: `agent-v3/src/shared/media/media-service.ts`
Exporta:
- uploadFile(userId, buffer, filename, contentType) -- upload cu rutare automata bucket
- getPresignedUploadUrl(userId, filename, contentType) -- URL presemnat PUT (1 ora)
- getPresignedDownloadUrl(objectKey, bucket) -- URL presemnat GET (1 ora)
- proxyFile(bucket, objectKey, ownerUserId, rangeHeader) -- proxy cu verificare proprietar + range support
- ensureBucket(name) -- creeaza daca nu exista
Flow upload in agent-v3:
1. Rezolva bucket-ul utilizatorului din didiFramework (keycloak_id -> bucket + folder)
2. Fallback la uploads/{userId} daca framework indisponibil
3. Returneaza: download_url, public_url, object_key, bucket, filename, size
### Python (shared layer)
Fisier: `shared/minio_presigner.py`
- convert_media_url_for_llm(url) -- converteste URL-uri interne MinIO in URL-uri presemnate pentru LLM-uri externe
- parse_minio_url(url) -- parseaza formate: minio://bucket/path, /bucket/path, http://minio:9000/bucket/path
Fisier: `shared/url_config.py`
- convert_minio_to_public_url(url) -- converteste URL-uri interne in URL-uri publice HTTP
---
## Fluxul de upload (utilizator)
```
Utilizator uploadeaza fisier
|
v
POST /api/v3/media/upload (agent-v3, multer, max 50MB)
|
v
MediaService.uploadFile()
|-- Cere didiFramework /internal/get-bucket-info -> { bucketName: 'didi-prod', folder: 'users/{id}/{mimeFolder}' }
|-- Fallback: bucket = MINIO_BUCKET (didi-prod), prefix = uploads/{userId}/
|
v
MinIO local: putObject('didi-prod', 'users/{id}/{mimeFolder}/{filename}', buffer)
|
v
Genereaza URL public: https://didi365.eu/api/v3/media/file/didi-prod/users/{id}/{mimeFolder}/{filename}
|
v
Returneaza: { download_url, public_url, object_key, bucket, size, content_type }
```
## Fluxul de inregistrare utilizator (single-bucket)
```
Utilizator face login prima data
|
v
GET /api/auth/me (didiFramework)
|
v
Utilizator nu exista in PG -> auto-inregistrare
|
v
createUserBucket(internetUserId, email, planId='1', planName='Free', storageLimitGb=1)
|-- (single-bucket mode) -- nu apeleaza MinIO makeBucket
|-- Logheaza initializarea
|-- Quota persistata in PG: bos_sysadmin.internet_user.storage_limit_bytes
|
v
Namespace logic users/{id}/ exista de cum primul fisier e uploadat.
```
## Fluxul media-preprocess (async, video/audio/imagine)
```
Job analiza pe URL/upload media
|
v
Dispatcher RabbitMQ -> media-preprocess queue (un singur worker per sesiune)
|
v
MediaPreprocessWorker:
|-- yt-dlp / fetch URL -> /tmp/video_{sessionId}_{ts}/source.mp4
|-- ffmpeg extrage frame-uri uniform (max 10) -> /tmp/.../frame_%03d.jpg
|-- ffmpeg extrage audio -> /tmp/.../audio.mp3
|-- transcript via Whisper (M17 -> Groq -> OpenAI)
|-- 2× vision call pe ACELEASI frame-uri (misinformation + ai_detection)
|-- (optional) upload frame-uri persistente -> didi-prod/users/{id}/videos/frames/
|
v
Cache rezultatele in Redis (TTL 1h):
agent:media:{sessionId}:transcript
agent:media:{sessionId}:vision:misinformation
agent:media:{sessionId}:vision:ai_detection
agent:media:{sessionId}:merged_text
agent:media:{sessionId}:ready = "1"
|
v
Dispatch task-uri pentru techniques + ai-tampered + claims (citesc din Redis, nu reproceseaza media)
```
Beneficiu: 1 download + 1 ffmpeg + 1 transcript + 2 vision in loc de 3× pe fiecare component.
---
## Fisiere in directorul didiStorage
```
init-buckets.sh -- Script initializare: creeaza 8 bucket-uri + lifecycle + versionare (136 linii)
.env.example -- Template variabile de mediu
README.md -- Documentatie (253 linii)
.gitignore -- Exclude .env, data/, config/
```
Zero cod custom. Bucket-urile si politicile sunt create de init-buckets.sh la prima pornire.
---
## Configurare Docker
```yaml
# din data-layer/docker-compose.yml
staging-dataLayer-minio:
image: minio/minio:RELEASE.2024-08-29T01-40-52Z
command: server /data --console-address ":9001"
environment:
MINIO_ROOT_USER: minioadmin
MINIO_ROOT_PASSWORD: minio123
MINIO_REGION_NAME: us-east-1
MINIO_BROWSER: "on"
ports:
- "9000:9000" # API (expus pe host 0.0.0.0:9000)
- "9001:9001" # Console (expus pe host 0.0.0.0:9001)
volumes:
- didi-staging-minio-data:/data
```
Nota: Nu exista container `minio-init` in docker-compose.yml. Scriptul `init-buckets.sh` trebuie rulat manual dupa prima pornire a MinIO.
---
## Variabile de mediu (conectare din alte servicii)
Setarile actuale (productie LOCALA, valori confirmate din containerul `didi-agent-v3`):
| Variabila | Valoare productie | Note |
|-----------|-------------------|------|
| MINIO_ENDPOINT | `staging-dataLayer-minio` | container local pe `didi-network`, path-style obligatoriu |
| MINIO_PORT | `9000` | expus si pe host (`0.0.0.0:9000`) |
| MINIO_USE_SSL | `false` | HTTP intern |
| MINIO_BUCKET | `didi-prod` | bucket fix, single-bucket arch (singurul bucket din instanta) |
| MINIO_ACCESS_KEY | `didi-prod` | full s3:* pe `didi-prod` |
| MINIO_SECRET_KEY | (in `.env`) | `627074a6...` |
Switch rapid local <-> cluster: `backend/services/orchestration-layer/agent-v3/scripts/minio-switch.sh local|cluster` (citeste credentiale din `.cluster-credentials.env`, modifica `.env`-urile pentru agent-v3 + didiFramework, restart containere). Status curent: `local`.
Fallback HA (cluster extern — inactiv, doar dupa `minio-switch.sh cluster`):
| Serviciu | MINIO_ENDPOINT | MINIO_PORT | Credentiale |
|----------|---------------|------------|-------------|
| didiFramework | <minio-host> (VIP 10.11.10.128) | 9000 | didi-prod / `.cluster-credentials.env` |
| agent-v3 | <minio-host>:9000 | (inclus in endpoint) | didi-prod / `.cluster-credentials.env` |
| Python shared | <minio-host>:9000 | (inclus in endpoint) | didi-prod / `.cluster-credentials.env` |
---
## Ce NU face
- Containerul local nu are cod custom (doar MinIO standard + script init); productia DIDI ruleaza pe acest container local, iar cluster-ul CAI managed ramane fallback HA inactiv.
- Local este instanta singulara (fara replicare). Fallback-ul cluster are 4 noduri + erasure coding EC:2 (toleranta la 2 noduri pierdute), disponibil doar dupa `minio-switch.sh cluster`.
- Nu are encriptie at-rest dedicata pe DIDI.
- TLS intern: HTTP (fara TLS pe MinIO local).
- Nu enforce-uieste quota la nivel MinIO; quota utilizator (`storage_used_bytes` / `storage_limit_bytes`) este urmarita in PG (`bos_sysadmin.internet_user`) si verificata de didiFramework la upload.
- Nu mai face create-bucket per utilizator (single-bucket: namespace logic prin prefix).