didi-lot2-backend/backend/services/data-layer/didiStorage/README.md

# DIDI Storage Service 📦

## Super Simple Start Guide 🚀

### One Command - That's It!
```bash
docker compose up -d
```

**DONE!** Everything is automatically configured! 🎉

## What Just Happened? 🤔

When you ran that one command:
1. MinIO storage server started
2. A helper container automatically:
   - Created 7 buckets for different file types
   - Set up auto-deletion for old files
   - Configured versioning for important data
   - Created access policies
   - Then exited (this is normal!)
3. Storage is now ready to use!

## Check If It's Working ✅

```bash
docker ps

# You should see:
# didi-storage (healthy) ← This is your storage server
```

**Note**: You might also see `didi-storage-init (Exited)` - that's the helper that set everything up. It's supposed to exit!

## Access the Web Console 🖥️

1. Open your browser
2. Go to: **http://localhost:9003**
3. Login:
   - Username: `minioadmin`
   - Password: `minio123`
4. You'll see all your buckets ready!

## Connection Info for Your Apps 📡

```python
# Python example
from minio import Minio

client = Minio(
    "localhost:9002",        # API port
    access_key="minioadmin",
    secret_key="minio123",
    secure=False
)
```

## The 7 Auto-Created Buckets 🗂️

| Bucket Name | What Goes Here | Auto-Delete After |
|------------|----------------|-------------------|
| `text-files` | Text documents, CSVs | 30 days |
| `image-files` | JPG, PNG, GIF | 60 days |
| `audio-files` | MP3, WAV, M4A | 30 days |
| `video-files` | MP4, AVI, MOV | 30 days |
| `document-files` | PDF, Word, Excel | Never |
| `pipeline-artifacts` | Analysis results | Never (versioned) |
| `backups` | System backups | Never (versioned) |

## Quick Test - Upload a File 📤

```bash
# Create a test file
echo "Hello Storage!" > test.txt

# Upload it (using docker)
docker exec didi-storage sh -c "echo 'Test' > /tmp/test.txt && mc cp /tmp/test.txt local/text-files/"

# Check it's there
docker exec didi-storage mc ls local/text-files/
```

## Common Tasks 🛠️

### Start Storage
```bash
docker compose up -d
# That's it! Everything auto-configures
```

### Stop Storage
```bash
docker compose down
# Data is preserved
```

### View Logs
```bash
docker compose logs -f didiStorage
```

### Check Storage Usage
```bash
docker exec didi-storage mc du local/
```

### List All Files
```bash
docker exec didi-storage mc ls --recursive local/
```

### Complete Fresh Start (WARNING: Deletes Everything!)
```bash
docker compose down -v
rm -rf data/ config/
docker compose up -d
```

## What's Special About This Setup? ✨

### 1. **Zero Configuration**
You don't need to:
- Create buckets manually
- Set up policies
- Configure expiry rules
- Enable versioning

It's ALL done automatically!

### 2. **Smart File Management**
- Old files auto-delete (saves space)
- Important files keep versions (never lose data)
- Each service gets its own bucket

### 3. **Ready for Production**
- Passwords in .env file (change them!)
- Resource limits configured
- Health checks included
- Logging configured

## Troubleshooting 🔧

### "Port already in use"
Someone else is using port 9002 or 9003. Fix:
1. Edit `.env`
2. Change `MINIO_API_PORT=9004`
3. Change `MINIO_CONSOLE_PORT=9005`
4. Run `docker compose up -d`

### "Can't access console"
1. Make sure you use `http://` not `https://`
2. Check container is running: `docker ps`
3. Try: http://localhost:9003

### "Buckets not created"
Check the init container logs:
```bash
docker logs didi-storage-init
```
It should show "Initialization Complete!"

### "Storage full"
Check usage:
```bash
docker exec didi-storage mc du local/
```
Files auto-delete after their expiry time!

## For Your Services 🔌

### Python Upload Example
```python
from minio import Minio

# Connect
client = Minio("localhost:9002", 
               access_key="minioadmin",
               secret_key="minio123", 
               secure=False)

# Upload image
client.fput_object("image-files", "photo.jpg", "/path/to/photo.jpg")

# Upload with metadata
client.fput_object(
    "document-files", 
    "report.pdf",
    "/path/to/report.pdf",
    metadata={"pipeline": "text-analysis", "user": "john"}
)
```

### Node.js Example
```javascript
const Minio = require('minio')

const client = new Minio.Client({
    endPoint: 'localhost',
    port: 9002,
    useSSL: false,
    accessKey: 'minioadmin',
    secretKey: 'minio123'
})

// Upload
client.fPutObject('text-files', 'data.txt', '/path/to/data.txt')
```

## How DIDI Platform Uses This 📊

```
User uploads file → Goes to appropriate bucket
                 ↓
Pipeline processes it → Results go to pipeline-artifacts
                      ↓
After 30-60 days → Media files auto-delete
                 ↓
Artifacts & backups → Keep forever with versions
```

## Security Notes 🔒

**For Production:**
1. Change `minioadmin` username in .env
2. Change `minio123` password in .env  
3. Use HTTPS (put behind nginx)
4. Restrict network access
5. Enable encryption

## Part of the Data Layer 🏗️

```
📁 data-layer/
  ├── 📁 didiDatabase/  ✅ PostgreSQL
  ├── 📁 didiCache/     ✅ Redis
  ├── 📁 didiStorage/   ✅ MinIO (You are here!)
  └── 📁 didiQueue/     ⏳ RabbitMQ (Coming next!)
```

## Summary - Why This Rocks 🎸

1. **One Command**: `docker compose up -d`
2. **Zero Config**: Everything auto-setup
3. **Smart Storage**: Auto-expiry, versioning
4. **Production Ready**: Just change passwords
5. **Developer Friendly**: Web console included

---
**That's it! Your storage is ready! 📦**

*No complex setup. No manual configuration. Just works!*

*Version: 1.0.0 | MinIO RELEASE.2024-08-29*