didi-lot2-backend/backend/services/data-layer/didiStorage/README.md

DIDI Storage Service 📦

Super Simple Start Guide 🚀

One Command - That's It!

docker compose up -d

DONE! Everything is automatically configured! 🎉

What Just Happened? 🤔

When you ran that one command:

1. MinIO storage server started
2. A helper container automatically:
   • Created 7 buckets for different file types
   • Set up auto-deletion for old files
   • Configured versioning for important data
   • Created access policies
   • Then exited (this is normal!)
3. Storage is now ready to use!

Check If It's Working ✅

docker ps

# You should see:
# didi-storage (healthy) ← This is your storage server

Note: You might also see didi-storage-init (Exited) - that's the helper that set everything up. It's supposed to exit!

Access the Web Console 🖥️

1. Open your browser
2. Go to: http://localhost:9003
3. Login:
   • Username: minioadmin
   • Password: minio123
4. You'll see all your buckets ready!

Connection Info for Your Apps 📡

# Python example
from minio import Minio

client = Minio(
    "localhost:9002",        # API port
    access_key="minioadmin",
    secret_key="minio123",
    secure=False
)

The 7 Auto-Created Buckets 🗂️

Bucket Name	What Goes Here	Auto-Delete After
text-files	Text documents, CSVs	30 days
image-files	JPG, PNG, GIF	60 days
audio-files	MP3, WAV, M4A	30 days
video-files	MP4, AVI, MOV	30 days
document-files	PDF, Word, Excel	Never
pipeline-artifacts	Analysis results	Never (versioned)
backups	System backups	Never (versioned)

Quick Test - Upload a File 📤

# Create a test file
echo "Hello Storage!" > test.txt

# Upload it (using docker)
docker exec didi-storage sh -c "echo 'Test' > /tmp/test.txt && mc cp /tmp/test.txt local/text-files/"

# Check it's there
docker exec didi-storage mc ls local/text-files/

Common Tasks 🛠️

Start Storage

docker compose up -d
# That's it! Everything auto-configures

Stop Storage

docker compose down
# Data is preserved

View Logs

docker compose logs -f didiStorage

Check Storage Usage

docker exec didi-storage mc du local/

List All Files

docker exec didi-storage mc ls --recursive local/

Complete Fresh Start (WARNING: Deletes Everything!)

docker compose down -v
rm -rf data/ config/
docker compose up -d

What's Special About This Setup? ✨

1. Zero Configuration

You don't need to:

• Create buckets manually
• Set up policies
• Configure expiry rules
• Enable versioning

It's ALL done automatically!

2. Smart File Management

• Old files auto-delete (saves space)
• Important files keep versions (never lose data)
• Each service gets its own bucket

3. Ready for Production

• Passwords in .env file (change them!)
• Resource limits configured
• Health checks included
• Logging configured

Troubleshooting 🔧

"Port already in use"

Someone else is using port 9002 or 9003. Fix:

1. Edit .env
2. Change MINIO_API_PORT=9004
3. Change MINIO_CONSOLE_PORT=9005
4. Run docker compose up -d

"Can't access console"

1. Make sure you use http:// not https://
2. Check container is running: docker ps
3. Try: http://localhost:9003

"Buckets not created"

Check the init container logs:

docker logs didi-storage-init

It should show "Initialization Complete!"

"Storage full"

Check usage:

docker exec didi-storage mc du local/

Files auto-delete after their expiry time!

For Your Services 🔌

Python Upload Example

from minio import Minio

# Connect
client = Minio("localhost:9002", 
               access_key="minioadmin",
               secret_key="minio123", 
               secure=False)

# Upload image
client.fput_object("image-files", "photo.jpg", "/path/to/photo.jpg")

# Upload with metadata
client.fput_object(
    "document-files", 
    "report.pdf",
    "/path/to/report.pdf",
    metadata={"pipeline": "text-analysis", "user": "john"}
)

Node.js Example

const Minio = require('minio')

const client = new Minio.Client({
    endPoint: 'localhost',
    port: 9002,
    useSSL: false,
    accessKey: 'minioadmin',
    secretKey: 'minio123'
})

// Upload
client.fPutObject('text-files', 'data.txt', '/path/to/data.txt')

How DIDI Platform Uses This 📊

User uploads file → Goes to appropriate bucket
                 ↓
Pipeline processes it → Results go to pipeline-artifacts
                      ↓
After 30-60 days → Media files auto-delete
                 ↓
Artifacts & backups → Keep forever with versions

Security Notes 🔒

For Production:

1. Change minioadmin username in .env
2. Change minio123 password in .env
3. Use HTTPS (put behind nginx)
4. Restrict network access
5. Enable encryption

Part of the Data Layer 🏗️

📁 data-layer/
  ├── 📁 didiDatabase/  ✅ PostgreSQL
  ├── 📁 didiCache/     ✅ Redis
  ├── 📁 didiStorage/   ✅ MinIO (You are here!)
  └── 📁 didiQueue/     ⏳ RabbitMQ (Coming next!)

Summary - Why This Rocks 🎸

1. One Command: docker compose up -d
2. Zero Config: Everything auto-setup
3. Smart Storage: Auto-expiry, versioning
4. Production Ready: Just change passwords
5. Developer Friendly: Web console included

---

That's it! Your storage is ready! 📦

No complex setup. No manual configuration. Just works!

Version: 1.0.0 | MinIO RELEASE.2024-08-29