clossers/didi-lot2-backend
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
didi-lot2-backend/backend/services/data-layer/didiStorage/INDEX.md
EVOTECH IT SRL 8ecc78e729 livrare lot 2
2026-07-10 03:39:53 -07:00

18 KiB
Raw Blame History

didiStorage - Index

Stocare fisiere media pentru platforma DIDI. Container MinIO (S3-compatibil) local pe masina de deployment. Nu contine cod custom -- doar configurare si script de initializare.

Productie activa (LOCAL)

DIDI scrie LOCAL pe containerul MinIO staging-dataLayer-minio:9000 (pe didi-network), intr-un singur bucket didi-prod. Decizie: stabilitate + zero dependinte externe. Clusterul MinIO managed extern (4 noduri, erasure coding EC:2, HAProxy + keepalived VRRP) ramane configurat ca fallback de urgenta pentru HA, activabil cu minio-switch.sh cluster, dar nu este folosit operational acum.

Mediu Endpoint Bucket Credentiale
Productie (LOCAL — activ) staging-dataLayer-minio:9000 (expus 0.0.0.0:9000) didi-prod (single bucket) didi-prod / 627074a6...
Fallback HA (cluster — inactiv) <minio-host>:9000 (VIP 10.11.10.128) didi-prod didi-prod / .cluster-credentials.env

Switch local/cluster: agent-v3/scripts/minio-switch.sh local|cluster (modifica .env + reseteaza containerele). Detalii migrare: agent-v3/MIGRATION_MINIO.md.

Restrictia cheie mostenita din arhitectura: credentialele didi-prod au s3:* doar pe bucket-ul propriu — nu se creeaza bucket-uri noi. De aici single-bucket architecture (vezi sectiunea urmatoare), pastrata si local.

Container local (activ)

Imagine: minio/minio:RELEASE.2024-08-29T01-40-52Z Container: staging-dataLayer-minio Port API: 9000 (Docker network + expus pe host 0.0.0.0:9000) Port Console: 9001 (expus pe host 0.0.0.0:9001) Bucket DIDI: didi-prod (singurul bucket) Credentiale DIDI: didi-prod / 627074a6... (din .env) Volume: didi-staging-minio-data:/data

Ce stocheaza

  1. Fisiere uploadate de utilizatori -- imagini, audio, video, documente
  2. Fisiere procesate de agent-v3 -- video downloadat, cadre extrase, transcrieri
  3. Artefacte pipeline -- rezultate analiza (cu versionare)
  4. Bucket-uri per utilizator -- fisiere organizate pe foldere tipizate

Single-bucket architecture (refactor 2026-04-25)

Productia foloseste un singur bucket didi-prod, iar separarea logica se face prin prefix-uri, nu bucket-uri distincte. Numele de prefix-uri sistem sunt identice cu numele bucket-urilor vechi pentru ca URL-urile vechi sa ramana interpretabile.

didi-prod/
  uploads/             -- upload-uri generale / fallback (legacy "uploads")
  image-files/         -- imagini (legacy bucket "image-files")
  audio-files/         -- audio (legacy bucket "audio-files")
  video-files/         -- video (legacy bucket "video-files")
  text-files/          -- text (legacy bucket "text-files")
  document-files/      -- PDF, Office (legacy bucket "document-files")
  pipeline-artifacts/  -- rezultate analiza (legacy bucket "pipeline-artifacts")
  users/{userId}/      -- namespace per utilizator (inlocuieste bucket-urile "user-{id}")
    images/
    videos/
    videos/frames/     -- cadre extrase din video (scrise de media-preprocess worker, citite de techniques + ai-tampered)
    audio-files/
    text-files/

De ce single-bucket

  • Arhitectura a fost proiectata pentru credentiale cu s3:* limitat la un singur bucket pre-creat (didi-prod), fara s3:CreateBucket — pastrata identic si pe MinIO local pentru portabilitate cluster.
  • Quota tracking simplificat: nu mai depindem de tag-uri pe bucket; storage-ul utilizatorilor este urmarit in PG (bos_sysadmin.internet_user.storage_used_bytes + storage_limit_bytes, vezi migration 010_add_user_storage_quota.sql din didiFramework).
  • Separare logica prin prefix-uri, nu prin bucket-uri distincte — acelasi layout functioneaza local si pe cluster fara modificari de cod.

Backward compat

Caller-ii care paseaza bucket-uri vechi (user-3, image-files) sunt rezolvati automat la canonic didi-prod/<full-key>:

URL primit Bucket rezolvat Key rezolvat
user-3/images/abc.jpg didi-prod users/3/images/abc.jpg
image-files/foo.jpg didi-prod image-files/foo.jpg
didi-prod/users/3/images/abc.jpg didi-prod users/3/images/abc.jpg (passthrough)

Implementat in:

  • didiFramework/src/config/minio.ts -- resolveBucketRequest(bucket, key) + constanta BUCKET = process.env.MINIO_BUCKET || 'didi-prod'.
  • agent-v3/src/shared/media/media-service.ts -- proxyFile() (ownership check accepta atat user-{N} cat si prefix users/{N}/) + uploadFile() foloseste prefix users/{userId}/{folder}/.

Lifecycle si versionare

  • Politicile de lifecycle (90 zile pentru transient, retentie permanenta pentru pipeline-artifacts/, backups/) se aplica pe bucket-ul didi-prod prin prefix; pe MinIO local pot fi setate cu mc ilm (optional).
  • Versionarea pentru pipeline-artifacts/ si backups/ este pastrata la nivel de bucket.
  • Daca se comuta pe cluster (minio-switch.sh cluster), lifecycle-ul devine responsabilitatea operatorilor cluster-ului (nu detinem bucket-ul acolo).

Mod legacy (multi-bucket)

Pentru referinta — modul vechi avea 8 bucket-uri sistem (uploads, text-files, image-files, audio-files, video-files, document-files, pipeline-artifacts, backups) plus bucket-uri dinamice user-{id} create la primul login. Acest layout a fost inlocuit de single-bucket didi-prod cu prefix-uri.

Limite dimensiune fisiere

Tip Limita MIME types
Imagini 20 MB image/jpeg, image/png, image/gif, image/webp, image/bmp, image/svg+xml
Audio 100 MB audio/mpeg, audio/wav, audio/ogg, audio/webm, audio/flac, audio/mp4, audio/x-m4a
Video 500 MB video/mp4, video/webm, video/quicktime, video/x-msvideo, video/x-matroska
Text 10 MB text/plain, text/html, text/markdown, text/csv
Documente 50 MB application/pdf, application/msword, application/vnd.openxmlformats-*

Rutarea automata: fisierul e pus in bucket-ul corespunzator MIME type-ului.

Cine scrie in MinIO

Serviciu Ce scrie Locatie (bucket didi-prod local) Logica in fisier
didiFramework (uploads) Fisiere uploadate via API didi-prod/users/{id}/{mimeFolder}/... (fallback didi-prod/{mimeBucket}/) didiFramework/src/routes/uploads.ts
didiFramework (auth) (Nu mai creeaza bucket) Logging registration; quota in PG -- didiFramework/src/routes/auth.ts + migration 010_add_user_storage_quota.sql
agent-v3 (media upload) Fisiere uploadate direct sau via multer didi-prod/users/{userId}/... (fallback didi-prod/uploads/{userId}/) agent-v3/src/api/routes.ts -> media-service.ts
agent-v3 (media-preprocess worker) Video downloadat, cadre extrase ffmpeg, audio extras pentru transcript didi-prod/users/{userId}/videos/frames/... (cand frame-urile sunt persistate); altfel /tmp efemer agent-v3/src/queue/workers/media-preprocess-worker.ts + shared/media/video-processor.ts
agent-v3 (pipeline) Imagini downloadate din URL-uri didi-prod/users/{userId}/images/... (fallback didi-prod/uploads/{userId}/) agent-v3/src/api/pipeline-routes.ts

Nota media-preprocess: workerul ruleaza inaintea componentelor de analiza (techniques, ai-tampered, claims) si centralizeaza descarcarea + ffmpeg + transcript + 2× vision. Frame-urile extrase sunt apoi consumate de workerii de techniques / ai-tampered fara duplicare. Cand persistarea frame-urilor in MinIO este activa, prefixul folosit este users/{userId}/videos/frames/ (cf. USER_BUCKET_FOLDERS.FRAMES).

Cine citeste din MinIO

Serviciu Ce citeste Cum
agent-v3 (media proxy) Servire fisiere catre client GET /api/v3/media/file/:bucket/:objectKey (proxy cu range support)
agent-v3 (vision) Imagini pentru modele LLM locale URL intern direct catre MinIO local (host-ul de deployment :9000/bucket/key)
agent-v3 (transcription) Audio/video pentru transcriere URL presemnat sau intern
didiFramework (uploads) Info fisier + URL presemnat GET /api/uploads/:fileId
Clienti externi Download fisiere URL presemnat (1 ora) sau proxy agent-v3

URL-uri si acces

URL public (prin proxy agent-v3)

https://didi365.eu/api/v3/media/file/{bucket}/{objectKey}
Exemplu (canonic): https://didi365.eu/api/v3/media/file/didi-prod/users/3/audio-files/1771883851173-audio.mp3
Exemplu (legacy): https://didi365.eu/api/v3/media/file/user-3/audio-files/1771883851173-audio.mp3  (rezolvat la didi-prod)

Suporta HTTP Range requests (streaming audio/video). Proxy-ul rezolva atat URL-uri legacy (user-{id}/..., image-files/...) cat si forme canonice (didi-prod/users/{id}/...).

URL presemnat (direct MinIO local)

http://<host-deployment>:9000/didi-prod/{objectKey}?X-Amz-Algorithm=...&X-Amz-Signature=...

Valabilitate: 1 ora (GET), 1 ora (PUT upload). Path-style obligatoriu (forcePathStyle=true).

URL intern (pentru modele LLM locale)

http://<host-deployment>:9000/didi-prod/{objectKey}

Modelele locale (Qwen Vision, pe masinile GPU) nu pot accesa didi365.eu, asa ca URL-urile publice sunt convertite la URL-uri MinIO interne catre containerul local staging-dataLayer-minio (expus pe host-ul de deployment :9000). Logica: agent-v3/src/shared/media/vision.ts (INTERNAL_MEDIA_BASE).

Integrare cu serviciile

didiFramework -- configurare MinIO principala

Fisier: didiFramework/src/config/minio.ts (refactor 2026-04-25 pentru single-bucket)

Constante:

  • BUCKET -- bucket fix din MINIO_BUCKET env (default didi-prod).
  • BUCKETS -- prefix-uri sistem (uploads, image-files, audio-files, video-files, text-files, document-files, pipeline-artifacts).
  • USER_BUCKET_FOLDERS -- foldere per utilizator (images, videos, audio-files, text-files, videos/frames).
  • MIME_TO_BUCKET -- routing MIME -> prefix sistem.

Exporta:

  • getMinioClient() -- client singleton.
  • checkMinioHealth() -- health check via listBuckets().
  • resolveBucketRequest(bucket, key) -- traduce input legacy (user-3, image-files) la (BUCKET, fullKey) canonic.
  • userObjectKey(userId, folder, filename) -- construieste users/{userId}/{folder}/{filename}.
  • ensureBucket(name) -- no-op in single-bucket mode (logging only). Pentru bucket-uri sistem legacy / user-{N} returneaza fara eroare.
  • uploadBuffer(bucket, name, buffer, mimeType, metadata) -- upload (rezolva bucket-ul intern).
  • deleteObject(bucket, name) / getObjectInfo(bucket, name) / listObjects(bucket, prefix, maxKeys).
  • getPresignedUrl(bucket, name, expiry) -- URL download (default 1 ora).
  • getPresignedPutUrl(bucket, name, expiry) -- URL upload (default 1 ora).
  • getDirectUrl(bucket, name) -- URL direct fara semnatura.
  • createUserBucket(userId, email, planId, planName, storageLimitGb) -- lazy in single-bucket mode: namespace-ul users/{id}/ "exista" doar cand are obiecte; functia logheaza si scrie quota in PG.
  • getUserBucketUsage(userId) -- listObjects pe users/{id}/, returneaza bytes + count.
  • getUserBucketMetadata(userId) -- thin shim (in single-bucket mode metadata e in PG, nu in tag-uri).
  • updateUserBucketMetadata(userId, planId, planName, storageLimitGb) -- no-op pentru bucket tags; caller-ul scrie in PG.

Quota tracking: migrarea sql/migrations/010_add_user_storage_quota.sql adauga coloanele storage_used_bytes si storage_limit_bytes la bos_sysadmin.internet_user. Tag-urile vechi (storage-limit-gb etc.) nu mai sunt folosite.

agent-v3 -- MediaService

Fisier: agent-v3/src/shared/media/media-service.ts

Exporta:

  • uploadFile(userId, buffer, filename, contentType) -- upload cu rutare automata bucket
  • getPresignedUploadUrl(userId, filename, contentType) -- URL presemnat PUT (1 ora)
  • getPresignedDownloadUrl(objectKey, bucket) -- URL presemnat GET (1 ora)
  • proxyFile(bucket, objectKey, ownerUserId, rangeHeader) -- proxy cu verificare proprietar + range support
  • ensureBucket(name) -- creeaza daca nu exista

Flow upload in agent-v3:

  1. Rezolva bucket-ul utilizatorului din didiFramework (keycloak_id -> bucket + folder)
  2. Fallback la uploads/{userId} daca framework indisponibil
  3. Returneaza: download_url, public_url, object_key, bucket, filename, size

Python (shared layer)

Fisier: shared/minio_presigner.py

  • convert_media_url_for_llm(url) -- converteste URL-uri interne MinIO in URL-uri presemnate pentru LLM-uri externe
  • parse_minio_url(url) -- parseaza formate: minio://bucket/path, /bucket/path, http://minio:9000/bucket/path

Fisier: shared/url_config.py

  • convert_minio_to_public_url(url) -- converteste URL-uri interne in URL-uri publice HTTP

Fluxul de upload (utilizator)

Utilizator uploadeaza fisier
  |
  v
POST /api/v3/media/upload (agent-v3, multer, max 50MB)
  |
  v
MediaService.uploadFile()
  |-- Cere didiFramework /internal/get-bucket-info -> { bucketName: 'didi-prod', folder: 'users/{id}/{mimeFolder}' }
  |-- Fallback: bucket = MINIO_BUCKET (didi-prod), prefix = uploads/{userId}/
  |
  v
MinIO local: putObject('didi-prod', 'users/{id}/{mimeFolder}/{filename}', buffer)
  |
  v
Genereaza URL public: https://didi365.eu/api/v3/media/file/didi-prod/users/{id}/{mimeFolder}/{filename}
  |
  v
Returneaza: { download_url, public_url, object_key, bucket, size, content_type }

Fluxul de inregistrare utilizator (single-bucket)

Utilizator face login prima data
  |
  v
GET /api/auth/me (didiFramework)
  |
  v
Utilizator nu exista in PG -> auto-inregistrare
  |
  v
createUserBucket(internetUserId, email, planId='1', planName='Free', storageLimitGb=1)
  |-- (single-bucket mode) -- nu apeleaza MinIO makeBucket
  |-- Logheaza initializarea
  |-- Quota persistata in PG: bos_sysadmin.internet_user.storage_limit_bytes
  |
  v
Namespace logic users/{id}/ exista de cum primul fisier e uploadat.

Fluxul media-preprocess (async, video/audio/imagine)

Job analiza pe URL/upload media
  |
  v
Dispatcher RabbitMQ -> media-preprocess queue (un singur worker per sesiune)
  |
  v
MediaPreprocessWorker:
  |-- yt-dlp / fetch URL -> /tmp/video_{sessionId}_{ts}/source.mp4
  |-- ffmpeg extrage frame-uri uniform (max 10) -> /tmp/.../frame_%03d.jpg
  |-- ffmpeg extrage audio -> /tmp/.../audio.mp3
  |-- transcript via Whisper (M17 -> Groq -> OpenAI)
  |-- 2× vision call pe ACELEASI frame-uri (misinformation + ai_detection)
  |-- (optional) upload frame-uri persistente -> didi-prod/users/{id}/videos/frames/
  |
  v
Cache rezultatele in Redis (TTL 1h):
  agent:media:{sessionId}:transcript
  agent:media:{sessionId}:vision:misinformation
  agent:media:{sessionId}:vision:ai_detection
  agent:media:{sessionId}:merged_text
  agent:media:{sessionId}:ready = "1"
  |
  v
Dispatch task-uri pentru techniques + ai-tampered + claims (citesc din Redis, nu reproceseaza media)

Beneficiu: 1 download + 1 ffmpeg + 1 transcript + 2 vision in loc de 3× pe fiecare component.

Fisiere in directorul didiStorage

init-buckets.sh         -- Script initializare: creeaza 8 bucket-uri + lifecycle + versionare (136 linii)
.env.example            -- Template variabile de mediu
README.md               -- Documentatie (253 linii)
.gitignore              -- Exclude .env, data/, config/

Zero cod custom. Bucket-urile si politicile sunt create de init-buckets.sh la prima pornire.

Configurare Docker

# din data-layer/docker-compose.yml
staging-dataLayer-minio:
  image: minio/minio:RELEASE.2024-08-29T01-40-52Z
  command: server /data --console-address ":9001"
  environment:
    MINIO_ROOT_USER: minioadmin
    MINIO_ROOT_PASSWORD: minio123
    MINIO_REGION_NAME: us-east-1
    MINIO_BROWSER: "on"
  ports:
    - "9000:9000"              # API (expus pe host 0.0.0.0:9000)
    - "9001:9001"              # Console (expus pe host 0.0.0.0:9001)
  volumes:
    - didi-staging-minio-data:/data

Nota: Nu exista container minio-init in docker-compose.yml. Scriptul init-buckets.sh trebuie rulat manual dupa prima pornire a MinIO.

Variabile de mediu (conectare din alte servicii)

Setarile actuale (productie LOCALA, valori confirmate din containerul didi-agent-v3):

Variabila Valoare productie Note
MINIO_ENDPOINT staging-dataLayer-minio container local pe didi-network, path-style obligatoriu
MINIO_PORT 9000 expus si pe host (0.0.0.0:9000)
MINIO_USE_SSL false HTTP intern
MINIO_BUCKET didi-prod bucket fix, single-bucket arch (singurul bucket din instanta)
MINIO_ACCESS_KEY didi-prod full s3:* pe didi-prod
MINIO_SECRET_KEY (in .env) 627074a6...

Switch rapid local <-> cluster: backend/services/orchestration-layer/agent-v3/scripts/minio-switch.sh local|cluster (citeste credentiale din .cluster-credentials.env, modifica .env-urile pentru agent-v3 + didiFramework, restart containere). Status curent: local.

Fallback HA (cluster extern — inactiv, doar dupa minio-switch.sh cluster):

Serviciu MINIO_ENDPOINT MINIO_PORT Credentiale
didiFramework (VIP 10.11.10.128) 9000 didi-prod / .cluster-credentials.env
agent-v3 :9000 (inclus in endpoint) didi-prod / .cluster-credentials.env
Python shared :9000 (inclus in endpoint) didi-prod / .cluster-credentials.env

Ce NU face

  • Containerul local nu are cod custom (doar MinIO standard + script init); productia DIDI ruleaza pe acest container local, iar cluster-ul CAI managed ramane fallback HA inactiv.
  • Local este instanta singulara (fara replicare). Fallback-ul cluster are 4 noduri + erasure coding EC:2 (toleranta la 2 noduri pierdute), disponibil doar dupa minio-switch.sh cluster.
  • Nu are encriptie at-rest dedicata pe DIDI.
  • TLS intern: HTTP (fara TLS pe MinIO local).
  • Nu enforce-uieste quota la nivel MinIO; quota utilizator (storage_used_bytes / storage_limit_bytes) este urmarita in PG (bos_sysadmin.internet_user) si verificata de didiFramework la upload.
  • Nu mai face create-bucket per utilizator (single-bucket: namespace logic prin prefix).