clossers/didi-lot2-backend
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

livrare lot 2

Browse source
This commit is contained in:
EVOTECH IT SRL 2026-07-10 03:39:53 -07:00
commit 8ecc78e729
763 changed files with 164593 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

46
backend/services/data-layer/didiStorage/.env.example Normal file
View file

@ -0,0 +1,46 @@
# ============================================================================
# didiStorage Environment Configuration
# ============================================================================
# Copy this file to .env and update with your values
# Docker Compose Project Name (groups containers in Docker Desktop)
COMPOSE_PROJECT_NAME=didibackend_datalayer
# MinIO Credentials (CHANGE THESE!)
MINIO_ROOT_USER=YOUR_ADMIN_USER_HERE
MINIO_ROOT_PASSWORD=YOUR_SECURE_PASSWORD_HERE
# Ports (using 9002/9003 to avoid conflicts with existing MinIO)
MINIO_API_PORT=9002 # API endpoint
MINIO_CONSOLE_PORT=9003 # Web console
# Region
MINIO_REGION=us-east-1
# Console Access
MINIO_BROWSER=on # Set to 'off' to disable web console
# Resource Limits
MINIO_MEMORY_LIMIT=1G
MINIO_MEMORY_RESERVATION=512M
# Storage Settings
MINIO_STORAGE_CLASS_STANDARD=EC:2
MINIO_STORAGE_CLASS_RRS=EC:1
# Timezone
TZ=UTC
# Bucket Lifecycle (days)
TEXT_FILES_EXPIRY=30
AUDIO_FILES_EXPIRY=30
VIDEO_FILES_EXPIRY=30
IMAGE_FILES_EXPIRY=60
DOCUMENT_FILES_EXPIRY=90
# Versioning
ENABLE_VERSIONING=true
# Encryption (optional)
MINIO_KMS_SECRET_KEY=
MINIO_KMS_AUTO_ENCRYPTION=off

27
backend/services/data-layer/didiStorage/.gitignore vendored Normal file
View file

@ -0,0 +1,27 @@
# Environment variables
.env
.env.local
# Data directory
data/
# Config directory
config/
# Logs
*.log
# OS files
.DS_Store
Thumbs.db
# IDE files
.idea/
.vscode/
*.swp
*.swo
# Backup files
*.bak
*.backup
*.old

352
backend/services/data-layer/didiStorage/INDEX.md Normal file
View file

@ -0,0 +1,352 @@
# didiStorage - Index
Stocare fisiere media pentru platforma DIDI. Container MinIO (S3-compatibil) local pe masina de deployment. Nu contine cod custom -- doar configurare si script de initializare.
## Productie activa (LOCAL)
DIDI scrie **LOCAL** pe containerul MinIO `staging-dataLayer-minio:9000` (pe `didi-network`), intr-un singur bucket `didi-prod`. Decizie: stabilitate + zero dependinte externe. Clusterul MinIO managed extern (4 noduri, erasure coding EC:2, HAProxy + keepalived VRRP) ramane configurat ca **fallback de urgenta pentru HA**, activabil cu `minio-switch.sh cluster`, dar nu este folosit operational acum.
| Mediu | Endpoint | Bucket | Credentiale |
|-------|----------|--------|-------------|
| **Productie (LOCAL — activ)** | `staging-dataLayer-minio:9000` (expus `0.0.0.0:9000`) | `didi-prod` (single bucket) | `didi-prod` / `627074a6...` |
| Fallback HA (cluster — inactiv) | `<minio-host>:9000` (VIP `10.11.10.128`) | `didi-prod` | didi-prod / `.cluster-credentials.env` |
Switch local/cluster: `agent-v3/scripts/minio-switch.sh local|cluster` (modifica `.env` + reseteaza containerele). Detalii migrare: `agent-v3/MIGRATION_MINIO.md`.
Restrictia cheie mostenita din arhitectura: credentialele `didi-prod` au `s3:*` **doar pe bucket-ul propriu** — nu se creeaza bucket-uri noi. De aici single-bucket architecture (vezi sectiunea urmatoare), pastrata si local.
## Container local (activ)
**Imagine**: minio/minio:RELEASE.2024-08-29T01-40-52Z
**Container**: staging-dataLayer-minio
**Port API**: 9000 (Docker network + expus pe host `0.0.0.0:9000`)
**Port Console**: 9001 (expus pe host `0.0.0.0:9001`)
**Bucket DIDI**: `didi-prod` (singurul bucket)
**Credentiale DIDI**: `didi-prod` / `627074a6...` (din `.env`)
**Volume**: didi-staging-minio-data:/data
---
## Ce stocheaza
1. **Fisiere uploadate de utilizatori** -- imagini, audio, video, documente
2. **Fisiere procesate de agent-v3** -- video downloadat, cadre extrase, transcrieri
3. **Artefacte pipeline** -- rezultate analiza (cu versionare)
4. **Bucket-uri per utilizator** -- fisiere organizate pe foldere tipizate
---
## Single-bucket architecture (refactor 2026-04-25)
Productia foloseste un singur bucket `didi-prod`, iar separarea logica se face prin **prefix-uri**, nu bucket-uri distincte. Numele de prefix-uri sistem sunt identice cu numele bucket-urilor vechi pentru ca URL-urile vechi sa ramana interpretabile.
```
didi-prod/
uploads/ -- upload-uri generale / fallback (legacy "uploads")
image-files/ -- imagini (legacy bucket "image-files")
audio-files/ -- audio (legacy bucket "audio-files")
video-files/ -- video (legacy bucket "video-files")
text-files/ -- text (legacy bucket "text-files")
document-files/ -- PDF, Office (legacy bucket "document-files")
pipeline-artifacts/ -- rezultate analiza (legacy bucket "pipeline-artifacts")
users/{userId}/ -- namespace per utilizator (inlocuieste bucket-urile "user-{id}")
images/
videos/
videos/frames/ -- cadre extrase din video (scrise de media-preprocess worker, citite de techniques + ai-tampered)
audio-files/
text-files/
```
### De ce single-bucket
- Arhitectura a fost proiectata pentru credentiale cu `s3:*` limitat la un singur bucket pre-creat (`didi-prod`), fara `s3:CreateBucket` — pastrata identic si pe MinIO local pentru portabilitate cluster.
- Quota tracking simplificat: nu mai depindem de tag-uri pe bucket; storage-ul utilizatorilor este urmarit in PG (`bos_sysadmin.internet_user.storage_used_bytes` + `storage_limit_bytes`, vezi migration `010_add_user_storage_quota.sql` din didiFramework).
- Separare logica prin prefix-uri, nu prin bucket-uri distincte — acelasi layout functioneaza local si pe cluster fara modificari de cod.
### Backward compat
Caller-ii care paseaza bucket-uri vechi (`user-3`, `image-files`) sunt rezolvati automat la canonic `didi-prod/<full-key>`:
| URL primit | Bucket rezolvat | Key rezolvat |
|---|---|---|
| `user-3/images/abc.jpg` | `didi-prod` | `users/3/images/abc.jpg` |
| `image-files/foo.jpg` | `didi-prod` | `image-files/foo.jpg` |
| `didi-prod/users/3/images/abc.jpg` | `didi-prod` | `users/3/images/abc.jpg` (passthrough) |
Implementat in:
- `didiFramework/src/config/minio.ts` -- `resolveBucketRequest(bucket, key)` + constanta `BUCKET = process.env.MINIO_BUCKET || 'didi-prod'`.
- `agent-v3/src/shared/media/media-service.ts` -- `proxyFile()` (ownership check accepta atat `user-{N}` cat si prefix `users/{N}/`) + `uploadFile()` foloseste prefix `users/{userId}/{folder}/`.
### Lifecycle si versionare
- Politicile de lifecycle (90 zile pentru transient, retentie permanenta pentru `pipeline-artifacts/`, `backups/`) se aplica pe bucket-ul `didi-prod` prin prefix; pe MinIO local pot fi setate cu `mc ilm` (optional).
- Versionarea pentru `pipeline-artifacts/` si `backups/` este pastrata la nivel de bucket.
- Daca se comuta pe cluster (`minio-switch.sh cluster`), lifecycle-ul devine responsabilitatea operatorilor cluster-ului (nu detinem bucket-ul acolo).
### Mod legacy (multi-bucket)
Pentru referinta — modul vechi avea 8 bucket-uri sistem (`uploads`, `text-files`, `image-files`, `audio-files`, `video-files`, `document-files`, `pipeline-artifacts`, `backups`) plus bucket-uri dinamice `user-{id}` create la primul login. Acest layout a fost inlocuit de single-bucket `didi-prod` cu prefix-uri.
---
## Limite dimensiune fisiere
| Tip | Limita | MIME types |
|-----|--------|------------|
| Imagini | 20 MB | image/jpeg, image/png, image/gif, image/webp, image/bmp, image/svg+xml |
| Audio | 100 MB | audio/mpeg, audio/wav, audio/ogg, audio/webm, audio/flac, audio/mp4, audio/x-m4a |
| Video | 500 MB | video/mp4, video/webm, video/quicktime, video/x-msvideo, video/x-matroska |
| Text | 10 MB | text/plain, text/html, text/markdown, text/csv |
| Documente | 50 MB | application/pdf, application/msword, application/vnd.openxmlformats-* |
Rutarea automata: fisierul e pus in bucket-ul corespunzator MIME type-ului.
---
## Cine scrie in MinIO
| Serviciu | Ce scrie | Locatie (bucket `didi-prod` local) | Logica in fisier |
|----------|----------|--------|------------------|
| didiFramework (uploads) | Fisiere uploadate via API | `didi-prod/users/{id}/{mimeFolder}/...` (fallback `didi-prod/{mimeBucket}/`) | didiFramework/src/routes/uploads.ts |
| didiFramework (auth) | (Nu mai creeaza bucket) Logging registration; quota in PG | -- | didiFramework/src/routes/auth.ts + migration 010_add_user_storage_quota.sql |
| agent-v3 (media upload) | Fisiere uploadate direct sau via multer | `didi-prod/users/{userId}/...` (fallback `didi-prod/uploads/{userId}/`) | agent-v3/src/api/routes.ts -> media-service.ts |
| agent-v3 (media-preprocess worker) | Video downloadat, cadre extrase ffmpeg, audio extras pentru transcript | `didi-prod/users/{userId}/videos/frames/...` (cand frame-urile sunt persistate); altfel /tmp efemer | agent-v3/src/queue/workers/media-preprocess-worker.ts + shared/media/video-processor.ts |
| agent-v3 (pipeline) | Imagini downloadate din URL-uri | `didi-prod/users/{userId}/images/...` (fallback `didi-prod/uploads/{userId}/`) | agent-v3/src/api/pipeline-routes.ts |
Nota media-preprocess: workerul ruleaza inaintea componentelor de analiza (techniques, ai-tampered, claims) si centralizeaza descarcarea + ffmpeg + transcript + 2× vision. Frame-urile extrase sunt apoi consumate de workerii de techniques / ai-tampered fara duplicare. Cand persistarea frame-urilor in MinIO este activa, prefixul folosit este `users/{userId}/videos/frames/` (cf. `USER_BUCKET_FOLDERS.FRAMES`).
## Cine citeste din MinIO
| Serviciu | Ce citeste | Cum |
|----------|-----------|-----|
| agent-v3 (media proxy) | Servire fisiere catre client | GET /api/v3/media/file/:bucket/:objectKey (proxy cu range support) |
| agent-v3 (vision) | Imagini pentru modele LLM locale | URL intern direct catre MinIO local (host-ul de deployment :9000/bucket/key) |
| agent-v3 (transcription) | Audio/video pentru transcriere | URL presemnat sau intern |
| didiFramework (uploads) | Info fisier + URL presemnat | GET /api/uploads/:fileId |
| Clienti externi | Download fisiere | URL presemnat (1 ora) sau proxy agent-v3 |
---
## URL-uri si acces
### URL public (prin proxy agent-v3)
```
https://didi365.eu/api/v3/media/file/{bucket}/{objectKey}
Exemplu (canonic): https://didi365.eu/api/v3/media/file/didi-prod/users/3/audio-files/1771883851173-audio.mp3
Exemplu (legacy): https://didi365.eu/api/v3/media/file/user-3/audio-files/1771883851173-audio.mp3 (rezolvat la didi-prod)
```
Suporta HTTP Range requests (streaming audio/video). Proxy-ul rezolva atat URL-uri legacy (`user-{id}/...`, `image-files/...`) cat si forme canonice (`didi-prod/users/{id}/...`).
### URL presemnat (direct MinIO local)
```
http://<host-deployment>:9000/didi-prod/{objectKey}?X-Amz-Algorithm=...&X-Amz-Signature=...
```
Valabilitate: 1 ora (GET), 1 ora (PUT upload). Path-style obligatoriu (`forcePathStyle=true`).
### URL intern (pentru modele LLM locale)
```
http://<host-deployment>:9000/didi-prod/{objectKey}
```
Modelele locale (Qwen Vision, pe masinile GPU) nu pot accesa `didi365.eu`, asa ca URL-urile publice sunt convertite la URL-uri MinIO interne catre containerul local `staging-dataLayer-minio` (expus pe host-ul de deployment `:9000`). Logica: `agent-v3/src/shared/media/vision.ts` (`INTERNAL_MEDIA_BASE`).
---
## Integrare cu serviciile
### didiFramework -- configurare MinIO principala
Fisier: `didiFramework/src/config/minio.ts` (refactor 2026-04-25 pentru single-bucket)
Constante:
- `BUCKET` -- bucket fix din `MINIO_BUCKET` env (default `didi-prod`).
- `BUCKETS` -- prefix-uri sistem (`uploads`, `image-files`, `audio-files`, `video-files`, `text-files`, `document-files`, `pipeline-artifacts`).
- `USER_BUCKET_FOLDERS` -- foldere per utilizator (`images`, `videos`, `audio-files`, `text-files`, `videos/frames`).
- `MIME_TO_BUCKET` -- routing MIME -> prefix sistem.
Exporta:
- `getMinioClient()` -- client singleton.
- `checkMinioHealth()` -- health check via `listBuckets()`.
- `resolveBucketRequest(bucket, key)` -- traduce input legacy (`user-3`, `image-files`) la `(BUCKET, fullKey)` canonic.
- `userObjectKey(userId, folder, filename)` -- construieste `users/{userId}/{folder}/{filename}`.
- `ensureBucket(name)` -- **no-op in single-bucket mode** (logging only). Pentru bucket-uri sistem legacy / `user-{N}` returneaza fara eroare.
- `uploadBuffer(bucket, name, buffer, mimeType, metadata)` -- upload (rezolva bucket-ul intern).
- `deleteObject(bucket, name)` / `getObjectInfo(bucket, name)` / `listObjects(bucket, prefix, maxKeys)`.
- `getPresignedUrl(bucket, name, expiry)` -- URL download (default 1 ora).
- `getPresignedPutUrl(bucket, name, expiry)` -- URL upload (default 1 ora).
- `getDirectUrl(bucket, name)` -- URL direct fara semnatura.
- `createUserBucket(userId, email, planId, planName, storageLimitGb)` -- **lazy in single-bucket mode**: namespace-ul `users/{id}/` "exista" doar cand are obiecte; functia logheaza si scrie quota in PG.
- `getUserBucketUsage(userId)` -- listObjects pe `users/{id}/`, returneaza bytes + count.
- `getUserBucketMetadata(userId)` -- thin shim (in single-bucket mode metadata e in PG, nu in tag-uri).
- `updateUserBucketMetadata(userId, planId, planName, storageLimitGb)` -- no-op pentru bucket tags; caller-ul scrie in PG.
Quota tracking: migrarea `sql/migrations/010_add_user_storage_quota.sql` adauga coloanele `storage_used_bytes` si `storage_limit_bytes` la `bos_sysadmin.internet_user`. Tag-urile vechi (`storage-limit-gb` etc.) nu mai sunt folosite.
### agent-v3 -- MediaService
Fisier: `agent-v3/src/shared/media/media-service.ts`
Exporta:
- uploadFile(userId, buffer, filename, contentType) -- upload cu rutare automata bucket
- getPresignedUploadUrl(userId, filename, contentType) -- URL presemnat PUT (1 ora)
- getPresignedDownloadUrl(objectKey, bucket) -- URL presemnat GET (1 ora)
- proxyFile(bucket, objectKey, ownerUserId, rangeHeader) -- proxy cu verificare proprietar + range support
- ensureBucket(name) -- creeaza daca nu exista
Flow upload in agent-v3:
1. Rezolva bucket-ul utilizatorului din didiFramework (keycloak_id -> bucket + folder)
2. Fallback la uploads/{userId} daca framework indisponibil
3. Returneaza: download_url, public_url, object_key, bucket, filename, size
### Python (shared layer)
Fisier: `shared/minio_presigner.py`
- convert_media_url_for_llm(url) -- converteste URL-uri interne MinIO in URL-uri presemnate pentru LLM-uri externe
- parse_minio_url(url) -- parseaza formate: minio://bucket/path, /bucket/path, http://minio:9000/bucket/path
Fisier: `shared/url_config.py`
- convert_minio_to_public_url(url) -- converteste URL-uri interne in URL-uri publice HTTP
---
## Fluxul de upload (utilizator)
```
Utilizator uploadeaza fisier
|
v
POST /api/v3/media/upload (agent-v3, multer, max 50MB)
|
v
MediaService.uploadFile()
|-- Cere didiFramework /internal/get-bucket-info -> { bucketName: 'didi-prod', folder: 'users/{id}/{mimeFolder}' }
|-- Fallback: bucket = MINIO_BUCKET (didi-prod), prefix = uploads/{userId}/
|
v
MinIO local: putObject('didi-prod', 'users/{id}/{mimeFolder}/{filename}', buffer)
|
v
Genereaza URL public: https://didi365.eu/api/v3/media/file/didi-prod/users/{id}/{mimeFolder}/{filename}
|
v
Returneaza: { download_url, public_url, object_key, bucket, size, content_type }
```
## Fluxul de inregistrare utilizator (single-bucket)
```
Utilizator face login prima data
|
v
GET /api/auth/me (didiFramework)
|
v
Utilizator nu exista in PG -> auto-inregistrare
|
v
createUserBucket(internetUserId, email, planId='1', planName='Free', storageLimitGb=1)
|-- (single-bucket mode) -- nu apeleaza MinIO makeBucket
|-- Logheaza initializarea
|-- Quota persistata in PG: bos_sysadmin.internet_user.storage_limit_bytes
|
v
Namespace logic users/{id}/ exista de cum primul fisier e uploadat.
```
## Fluxul media-preprocess (async, video/audio/imagine)
```
Job analiza pe URL/upload media
|
v
Dispatcher RabbitMQ -> media-preprocess queue (un singur worker per sesiune)
|
v
MediaPreprocessWorker:
|-- yt-dlp / fetch URL -> /tmp/video_{sessionId}_{ts}/source.mp4
|-- ffmpeg extrage frame-uri uniform (max 10) -> /tmp/.../frame_%03d.jpg
|-- ffmpeg extrage audio -> /tmp/.../audio.mp3
|-- transcript via Whisper (M17 -> Groq -> OpenAI)
|-- 2× vision call pe ACELEASI frame-uri (misinformation + ai_detection)
|-- (optional) upload frame-uri persistente -> didi-prod/users/{id}/videos/frames/
|
v
Cache rezultatele in Redis (TTL 1h):
agent:media:{sessionId}:transcript
agent:media:{sessionId}:vision:misinformation
agent:media:{sessionId}:vision:ai_detection
agent:media:{sessionId}:merged_text
agent:media:{sessionId}:ready = "1"
|
v
Dispatch task-uri pentru techniques + ai-tampered + claims (citesc din Redis, nu reproceseaza media)
```
Beneficiu: 1 download + 1 ffmpeg + 1 transcript + 2 vision in loc de 3× pe fiecare component.
---
## Fisiere in directorul didiStorage
```
init-buckets.sh -- Script initializare: creeaza 8 bucket-uri + lifecycle + versionare (136 linii)
.env.example -- Template variabile de mediu
README.md -- Documentatie (253 linii)
.gitignore -- Exclude .env, data/, config/
```
Zero cod custom. Bucket-urile si politicile sunt create de init-buckets.sh la prima pornire.
---
## Configurare Docker
```yaml
# din data-layer/docker-compose.yml
staging-dataLayer-minio:
image: minio/minio:RELEASE.2024-08-29T01-40-52Z
command: server /data --console-address ":9001"
environment:
MINIO_ROOT_USER: minioadmin
MINIO_ROOT_PASSWORD: minio123
MINIO_REGION_NAME: us-east-1
MINIO_BROWSER: "on"
ports:
- "9000:9000" # API (expus pe host 0.0.0.0:9000)
- "9001:9001" # Console (expus pe host 0.0.0.0:9001)
volumes:
- didi-staging-minio-data:/data
```
Nota: Nu exista container `minio-init` in docker-compose.yml. Scriptul `init-buckets.sh` trebuie rulat manual dupa prima pornire a MinIO.
---
## Variabile de mediu (conectare din alte servicii)
Setarile actuale (productie LOCALA, valori confirmate din containerul `didi-agent-v3`):
| Variabila | Valoare productie | Note |
|-----------|-------------------|------|
| MINIO_ENDPOINT | `staging-dataLayer-minio` | container local pe `didi-network`, path-style obligatoriu |
| MINIO_PORT | `9000` | expus si pe host (`0.0.0.0:9000`) |
| MINIO_USE_SSL | `false` | HTTP intern |
| MINIO_BUCKET | `didi-prod` | bucket fix, single-bucket arch (singurul bucket din instanta) |
| MINIO_ACCESS_KEY | `didi-prod` | full s3:* pe `didi-prod` |
| MINIO_SECRET_KEY | (in `.env`) | `627074a6...` |
Switch rapid local <-> cluster: `backend/services/orchestration-layer/agent-v3/scripts/minio-switch.sh local|cluster` (citeste credentiale din `.cluster-credentials.env`, modifica `.env`-urile pentru agent-v3 + didiFramework, restart containere). Status curent: `local`.
Fallback HA (cluster extern — inactiv, doar dupa `minio-switch.sh cluster`):
| Serviciu | MINIO_ENDPOINT | MINIO_PORT | Credentiale |
|----------|---------------|------------|-------------|
| didiFramework | <minio-host> (VIP 10.11.10.128) | 9000 | didi-prod / `.cluster-credentials.env` |
| agent-v3 | <minio-host>:9000 | (inclus in endpoint) | didi-prod / `.cluster-credentials.env` |
| Python shared | <minio-host>:9000 | (inclus in endpoint) | didi-prod / `.cluster-credentials.env` |
---
## Ce NU face
- Containerul local nu are cod custom (doar MinIO standard + script init); productia DIDI ruleaza pe acest container local, iar cluster-ul CAI managed ramane fallback HA inactiv.
- Local este instanta singulara (fara replicare). Fallback-ul cluster are 4 noduri + erasure coding EC:2 (toleranta la 2 noduri pierdute), disponibil doar dupa `minio-switch.sh cluster`.
- Nu are encriptie at-rest dedicata pe DIDI.
- TLS intern: HTTP (fara TLS pe MinIO local).
- Nu enforce-uieste quota la nivel MinIO; quota utilizator (`storage_used_bytes` / `storage_limit_bytes`) este urmarita in PG (`bos_sysadmin.internet_user`) si verificata de didiFramework la upload.
- Nu mai face create-bucket per utilizator (single-bucket: namespace logic prin prefix).

253
backend/services/data-layer/didiStorage/README.md Normal file
View file

@ -0,0 +1,253 @@
# DIDI Storage Service 📦
## Super Simple Start Guide 🚀
### One Command - That's It!
```bash
docker compose up -d
```
**DONE!** Everything is automatically configured! 🎉
## What Just Happened? 🤔
When you ran that one command:
1. MinIO storage server started
2. A helper container automatically:
- Created 7 buckets for different file types
- Set up auto-deletion for old files
- Configured versioning for important data
- Created access policies
- Then exited (this is normal!)
3. Storage is now ready to use!
## Check If It's Working ✅
```bash
docker ps
# You should see:
# didi-storage (healthy) ← This is your storage server
```
**Note**: You might also see `didi-storage-init (Exited)` - that's the helper that set everything up. It's supposed to exit!
## Access the Web Console 🖥️
1. Open your browser
2. Go to: **http://localhost:9003**
3. Login:
- Username: `minioadmin`
- Password: `minio123`
4. You'll see all your buckets ready!
## Connection Info for Your Apps 📡
```python
# Python example
from minio import Minio
client = Minio(
"localhost:9002", # API port
access_key="minioadmin",
secret_key="minio123",
secure=False
)
```
## The 7 Auto-Created Buckets 🗂️
| Bucket Name | What Goes Here | Auto-Delete After |
|------------|----------------|-------------------|
| `text-files` | Text documents, CSVs | 30 days |
| `image-files` | JPG, PNG, GIF | 60 days |
| `audio-files` | MP3, WAV, M4A | 30 days |
| `video-files` | MP4, AVI, MOV | 30 days |
| `document-files` | PDF, Word, Excel | Never |
| `pipeline-artifacts` | Analysis results | Never (versioned) |
| `backups` | System backups | Never (versioned) |
## Quick Test - Upload a File 📤
```bash
# Create a test file
echo "Hello Storage!" > test.txt
# Upload it (using docker)
docker exec didi-storage sh -c "echo 'Test' > /tmp/test.txt && mc cp /tmp/test.txt local/text-files/"
# Check it's there
docker exec didi-storage mc ls local/text-files/
```
## Common Tasks 🛠️
### Start Storage
```bash
docker compose up -d
# That's it! Everything auto-configures
```
### Stop Storage
```bash
docker compose down
# Data is preserved
```
### View Logs
```bash
docker compose logs -f didiStorage
```
### Check Storage Usage
```bash
docker exec didi-storage mc du local/
```
### List All Files
```bash
docker exec didi-storage mc ls --recursive local/
```
### Complete Fresh Start (WARNING: Deletes Everything!)
```bash
docker compose down -v
rm -rf data/ config/
docker compose up -d
```
## What's Special About This Setup? ✨
### 1. **Zero Configuration**
You don't need to:
- Create buckets manually
- Set up policies
- Configure expiry rules
- Enable versioning
It's ALL done automatically!
### 2. **Smart File Management**
- Old files auto-delete (saves space)
- Important files keep versions (never lose data)
- Each service gets its own bucket
### 3. **Ready for Production**
- Passwords in .env file (change them!)
- Resource limits configured
- Health checks included
- Logging configured
## Troubleshooting 🔧
### "Port already in use"
Someone else is using port 9002 or 9003. Fix:
1. Edit `.env`
2. Change `MINIO_API_PORT=9004`
3. Change `MINIO_CONSOLE_PORT=9005`
4. Run `docker compose up -d`
### "Can't access console"
1. Make sure you use `http://` not `https://`
2. Check container is running: `docker ps`
3. Try: http://localhost:9003
### "Buckets not created"
Check the init container logs:
```bash
docker logs didi-storage-init
```
It should show "Initialization Complete!"
### "Storage full"
Check usage:
```bash
docker exec didi-storage mc du local/
```
Files auto-delete after their expiry time!
## For Your Services 🔌
### Python Upload Example
```python
from minio import Minio
# Connect
client = Minio("localhost:9002",
access_key="minioadmin",
secret_key="minio123",
secure=False)
# Upload image
client.fput_object("image-files", "photo.jpg", "/path/to/photo.jpg")
# Upload with metadata
client.fput_object(
"document-files",
"report.pdf",
"/path/to/report.pdf",
metadata={"pipeline": "text-analysis", "user": "john"}
)
```
### Node.js Example
```javascript
const Minio = require('minio')
const client = new Minio.Client({
endPoint: 'localhost',
port: 9002,
useSSL: false,
accessKey: 'minioadmin',
secretKey: 'minio123'
})
// Upload
client.fPutObject('text-files', 'data.txt', '/path/to/data.txt')
```
## How DIDI Platform Uses This 📊
```
User uploads file → Goes to appropriate bucket
↓
Pipeline processes it → Results go to pipeline-artifacts
↓
After 30-60 days → Media files auto-delete
↓
Artifacts & backups → Keep forever with versions
```
## Security Notes 🔒
**For Production:**
1. Change `minioadmin` username in .env
2. Change `minio123` password in .env
3. Use HTTPS (put behind nginx)
4. Restrict network access
5. Enable encryption
## Part of the Data Layer 🏗️
```
📁 data-layer/
├── 📁 didiDatabase/ ✅ PostgreSQL
├── 📁 didiCache/ ✅ Redis
├── 📁 didiStorage/ ✅ MinIO (You are here!)
└── 📁 didiQueue/ ⏳ RabbitMQ (Coming next!)
```
## Summary - Why This Rocks 🎸
1. **One Command**: `docker compose up -d`
2. **Zero Config**: Everything auto-setup
3. **Smart Storage**: Auto-expiry, versioning
4. **Production Ready**: Just change passwords
5. **Developer Friendly**: Web console included
---
**That's it! Your storage is ready! 📦**
*No complex setup. No manual configuration. Just works!*
*Version: 1.0.0 | MinIO RELEASE.2024-08-29*

137
backend/services/data-layer/didiStorage/init-buckets.sh Normal file
View file

@ -0,0 +1,137 @@
#!/bin/sh
# ============================================================================
# MinIO Bucket Initialization Script
# Automatically creates all required buckets and policies on startup
# ============================================================================
set -e
echo "============================================"
echo "Starting MinIO Bucket Initialization"
echo "============================================"
# Wait for MinIO to be ready
echo "→ Waiting for MinIO to be ready..."
sleep 5
# Configure MinIO client with credentials from environment
echo "→ Configuring MinIO client..."
mc alias set local http://${MINIO_HOST}:${MINIO_PORT} ${MINIO_ROOT_USER} ${MINIO_ROOT_PASSWORD}
# Create all required buckets
echo "→ Creating buckets..."
mc mb local/text-files --ignore-existing
mc mb local/image-files --ignore-existing
mc mb local/audio-files --ignore-existing
mc mb local/video-files --ignore-existing
mc mb local/document-files --ignore-existing
mc mb local/pipeline-artifacts --ignore-existing
mc mb local/uploads --ignore-existing
mc mb local/backups --ignore-existing
mc mb local/didi-prod --ignore-existing # single-bucket mode (MINIO_BUCKET=didi-prod) — media upload/download
echo "✓ All buckets created"
# Enable versioning for important buckets
echo "→ Enabling versioning..."
mc version enable local/pipeline-artifacts
mc version enable local/backups
echo "✓ Versioning enabled for pipeline-artifacts and backups"
# Set lifecycle policies for temporary files
echo "→ Setting lifecycle policies..."
cat > /tmp/lifecycle-30days.json <<EOF
{
"Rules": [
{
"ID": "expire-30days",
"Status": "Enabled",
"Expiration": {
"Days": 30
}
}
]
}
EOF
cat > /tmp/lifecycle-60days.json <<EOF
{
"Rules": [
{
"ID": "expire-60days",
"Status": "Enabled",
"Expiration": {
"Days": 60
}
}
]
}
EOF
# Apply lifecycle policies
mc ilm import local/text-files < /tmp/lifecycle-30days.json
mc ilm import local/audio-files < /tmp/lifecycle-30days.json
mc ilm import local/video-files < /tmp/lifecycle-30days.json
mc ilm import local/image-files < /tmp/lifecycle-60days.json
mc ilm import local/uploads < /tmp/lifecycle-30days.json
echo "✓ Lifecycle policies configured"
# Create anonymous read policy for public buckets (optional)
# Uncomment if you want public read access to certain buckets
# echo "→ Setting public access policies..."
# mc anonymous set download local/image-files
# mc anonymous set download local/video-files
# echo "✓ Public read access configured"
# Create service accounts for microservices (optional)
# This creates restricted access for each service
echo "→ Creating service access policies..."
# Policy for text analysis service
cat > /tmp/text-service-policy.json <<EOF
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": ["s3:*"],
"Resource": ["arn:aws:s3:::text-files/*"]
}
]
}
EOF
# Policy for image analysis service
cat > /tmp/image-service-policy.json <<EOF
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": ["s3:*"],
"Resource": ["arn:aws:s3:::image-files/*"]
}
]
}
EOF
# Apply policies (these can be used to create service accounts later)
mc admin policy create local text-service-policy /tmp/text-service-policy.json || true
mc admin policy create local image-service-policy /tmp/image-service-policy.json || true
echo "✓ Service policies created"
# List all buckets to confirm
echo ""
echo "============================================"
echo "Initialization Complete!"
echo "============================================"
echo "Buckets created:"
mc ls local/
echo "============================================"
# Clean up temp files
rm -f /tmp/lifecycle-*.json /tmp/*-policy.json
echo "MinIO is ready for use!"