livrare lot 2
This commit is contained in:
commit
8ecc78e729
763 changed files with 164593 additions and 0 deletions
321
backend/services/README.md
Normal file
321
backend/services/README.md
Normal file
|
|
@ -0,0 +1,321 @@
|
|||
# DIDI Backend Services - Master Orchestration
|
||||
|
||||
A pyramid-architecture microservices platform for misinformation detection with three distinct layers.
|
||||
|
||||
## 🏗️ Architecture Overview
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────┐
|
||||
│ Layer 3: Gateway & Auth Layer │
|
||||
│ (Kong API Gateway, Keycloak) │
|
||||
├─────────────────────────────────────────────────┤
|
||||
│ Layer 2: Orchestration Layer │
|
||||
│ (Orchestrator, Analysis Services) │
|
||||
├─────────────────────────────────────────────────┤
|
||||
│ Layer 1: Data Layer │
|
||||
│ (PostgreSQL, Redis, RabbitMQ, MinIO) │
|
||||
└─────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## 📋 Table of Contents
|
||||
- [Quick Start](#-quick-start)
|
||||
- [Service Layers](#-service-layers)
|
||||
- [Commands](#-commands)
|
||||
- [Service Ports](#-service-ports)
|
||||
- [Health Monitoring](#-health-monitoring)
|
||||
- [Troubleshooting](#-troubleshooting)
|
||||
- [Development](#-development)
|
||||
|
||||
## 🚀 Quick Start
|
||||
|
||||
### Start Everything (Recommended)
|
||||
```bash
|
||||
# Start all layers in pyramid order with health checks
|
||||
make pyramid
|
||||
```
|
||||
|
||||
### Alternative Methods
|
||||
```bash
|
||||
# Start all services at once
|
||||
make up
|
||||
|
||||
# Start with logs visible
|
||||
make dev
|
||||
|
||||
# Check service status
|
||||
make status
|
||||
|
||||
# Check service health
|
||||
make health
|
||||
```
|
||||
|
||||
## 🏛️ Service Layers
|
||||
|
||||
### Layer 1: Data Layer (Foundation)
|
||||
The foundation of our pyramid - all data persistence and messaging services.
|
||||
|
||||
| Service | Purpose | Port | Container Name |
|
||||
|---------|---------|------|----------------|
|
||||
| PostgreSQL | Primary database | 5436 | dataLayer-postgres |
|
||||
| Redis | Cache & session store | 6379 | dataLayer-redis |
|
||||
| RabbitMQ | Message broker | 5672/15672 | dataLayer-rabbitmq |
|
||||
| MinIO | Object storage | 9000/9001 | dataLayer-minio |
|
||||
| pgAdmin | Database management | 5051 | dataLayer-pgadmin |
|
||||
|
||||
### Layer 2: Orchestration Layer (Processing)
|
||||
The processing layer - handles business logic and analysis pipelines.
|
||||
|
||||
| Service | Purpose | Port | Container Name |
|
||||
|---------|---------|------|----------------|
|
||||
| Orchestrator | Pipeline management | 8000 | orchestrationLayer-orchestrator |
|
||||
| Analysis | Text analysis service | 8004 | orchestrationLayer-analysis |
|
||||
|
||||
### Layer 3: Gateway & Auth Layer (Access)
|
||||
The access layer - manages authentication and API routing.
|
||||
|
||||
| Service | Purpose | Port | Container Name |
|
||||
|---------|---------|------|----------------|
|
||||
| Kong | API Gateway | 8100/8101 | gatewayAuthLayer-kong |
|
||||
| Keycloak | Authentication | 8280 | gatewayAuthLayer-keycloak |
|
||||
|
||||
## 📦 Commands
|
||||
|
||||
### Main Commands
|
||||
```bash
|
||||
make help # Show all available commands
|
||||
make pyramid # Start layers in correct order (recommended)
|
||||
make up # Start all services
|
||||
make down # Stop all services
|
||||
make restart # Restart all services
|
||||
make status # Show service status
|
||||
make health # Check service health
|
||||
make logs # Show all logs
|
||||
make clean # Stop and remove everything (including data!)
|
||||
```
|
||||
|
||||
### Layer-Specific Commands
|
||||
```bash
|
||||
# Data Layer
|
||||
make data-up # Start only data layer
|
||||
make data-down # Stop data layer
|
||||
|
||||
# Orchestration Layer
|
||||
make orch-up # Start only orchestration layer
|
||||
make orch-down # Stop orchestration layer
|
||||
|
||||
# Gateway Layer
|
||||
make gateway-up # Start only gateway layer
|
||||
make gateway-down # Stop gateway layer
|
||||
```
|
||||
|
||||
### Service Logs
|
||||
```bash
|
||||
make logs-postgres # PostgreSQL logs
|
||||
make logs-redis # Redis logs
|
||||
make logs-rabbitmq # RabbitMQ logs
|
||||
make logs-orchestrator # Orchestrator logs
|
||||
make logs-kong # Kong Gateway logs
|
||||
make logs-keycloak # Keycloak logs
|
||||
```
|
||||
|
||||
### Development Commands
|
||||
```bash
|
||||
make dev # Start with logs visible
|
||||
make test # Test all endpoints
|
||||
make resources # Show resource usage
|
||||
make tail # Tail last 100 log lines
|
||||
```
|
||||
|
||||
## 🔌 Service Ports
|
||||
|
||||
### External Access Points
|
||||
- **Admin Dashboard**: http://localhost:3001
|
||||
- **Orchestrator API**: http://localhost:8000
|
||||
- **Kong Gateway**: http://localhost:8100
|
||||
- **Kong Admin**: http://localhost:8101
|
||||
- **Keycloak**: http://localhost:8280
|
||||
- **RabbitMQ Management**: http://localhost:15672
|
||||
- **MinIO Console**: http://localhost:9001
|
||||
- **pgAdmin**: http://localhost:5051
|
||||
|
||||
### Default Credentials
|
||||
|
||||
| Service | Username | Password |
|
||||
|---------|----------|----------|
|
||||
| PostgreSQL | postgres | postgres_dev_password_123 |
|
||||
| RabbitMQ | admin | rabbitmq_dev_password_123 |
|
||||
| MinIO | minioadmin | minioadmin |
|
||||
| pgAdmin | admin@admin.com | admin |
|
||||
| Keycloak | admin | admin |
|
||||
|
||||
## 🏥 Health Monitoring
|
||||
|
||||
### Check All Services
|
||||
```bash
|
||||
make health
|
||||
```
|
||||
|
||||
### Manual Health Checks
|
||||
```bash
|
||||
# PostgreSQL
|
||||
docker exec dataLayer-postgres pg_isready -U postgres
|
||||
|
||||
# Redis
|
||||
docker exec dataLayer-redis redis-cli ping
|
||||
|
||||
# RabbitMQ
|
||||
curl -u admin:rabbitmq_dev_password_123 http://localhost:15672/api/health/checks/virtual-hosts
|
||||
|
||||
# MinIO
|
||||
curl http://localhost:9000/minio/health/live
|
||||
|
||||
# Orchestrator
|
||||
curl http://localhost:8000/health
|
||||
|
||||
# Kong
|
||||
curl http://localhost:8101/status
|
||||
```
|
||||
|
||||
## 🔧 Troubleshooting
|
||||
|
||||
### Services Won't Start
|
||||
```bash
|
||||
# Check if ports are already in use
|
||||
netstat -an | grep -E "(5436|6379|5672|9000|8000|8100|8280)"
|
||||
|
||||
# Check Docker resources
|
||||
docker system df
|
||||
docker system prune -a # Warning: removes all unused images
|
||||
|
||||
# Check logs for specific service
|
||||
make logs-postgres # Replace with service name
|
||||
```
|
||||
|
||||
### Database Connection Issues
|
||||
```bash
|
||||
# Test PostgreSQL connection
|
||||
docker exec dataLayer-postgres psql -U postgres -c "\l"
|
||||
|
||||
# Check if database exists
|
||||
docker exec dataLayer-postgres psql -U postgres -c "SELECT datname FROM pg_database;"
|
||||
```
|
||||
|
||||
### Message Queue Issues
|
||||
```bash
|
||||
# Check RabbitMQ queues
|
||||
docker exec dataLayer-rabbitmq rabbitmqctl list_queues
|
||||
|
||||
# Check RabbitMQ connections
|
||||
docker exec dataLayer-rabbitmq rabbitmqctl list_connections
|
||||
```
|
||||
|
||||
### Reset Everything
|
||||
```bash
|
||||
# WARNING: This deletes all data!
|
||||
make clean
|
||||
make pyramid
|
||||
```
|
||||
|
||||
## 🛠️ Development
|
||||
|
||||
### Adding a New Service
|
||||
|
||||
1. **Create service folder** in appropriate layer:
|
||||
```bash
|
||||
mkdir -p {layer-name}/new-service
|
||||
```
|
||||
|
||||
2. **Add docker-compose.yml** to service folder
|
||||
|
||||
3. **Update layer's docker-compose.yml** to include new service
|
||||
|
||||
4. **Update this README** with new service details
|
||||
|
||||
### Modifying Configurations
|
||||
|
||||
1. **Environment Variables**: Edit `.env` files in each layer
|
||||
2. **Docker Compose**: Edit `docker-compose.yml` in each layer
|
||||
3. **Service Configs**: Edit configuration files in service folders
|
||||
|
||||
### Building Custom Images
|
||||
```bash
|
||||
# Build all services
|
||||
make build
|
||||
|
||||
# Build specific layer
|
||||
cd data-layer && make build
|
||||
```
|
||||
|
||||
## 📊 Architecture Flow
|
||||
|
||||
```mermaid
|
||||
graph TB
|
||||
subgraph "Layer 3: Gateway & Auth"
|
||||
Kong[Kong Gateway :8100]
|
||||
Keycloak[Keycloak :8280]
|
||||
end
|
||||
|
||||
subgraph "Layer 2: Orchestration"
|
||||
Orchestrator[Orchestrator :8000]
|
||||
Analysis[Analysis :8004]
|
||||
end
|
||||
|
||||
subgraph "Layer 1: Data"
|
||||
PostgreSQL[(PostgreSQL :5436)]
|
||||
Redis[(Redis :6379)]
|
||||
RabbitMQ[RabbitMQ :5672]
|
||||
MinIO[MinIO :9000]
|
||||
end
|
||||
|
||||
Kong --> Orchestrator
|
||||
Kong --> Analysis
|
||||
Orchestrator --> PostgreSQL
|
||||
Orchestrator --> Redis
|
||||
Orchestrator --> RabbitMQ
|
||||
Analysis --> PostgreSQL
|
||||
Analysis --> Redis
|
||||
Analysis --> RabbitMQ
|
||||
Analysis --> MinIO
|
||||
```
|
||||
|
||||
## 🔒 Security Notes
|
||||
|
||||
- **Change default passwords** before production deployment
|
||||
- **Use environment variables** for sensitive data
|
||||
- **Enable TLS/SSL** for all external connections
|
||||
- **Configure firewall rules** to restrict access
|
||||
- **Regular backup** of PostgreSQL and MinIO data
|
||||
|
||||
## 📝 Layer Dependencies
|
||||
|
||||
Each layer depends on the layers below it:
|
||||
|
||||
1. **Data Layer**: Independent (foundation)
|
||||
2. **Orchestration Layer**: Requires Data Layer
|
||||
3. **Gateway Layer**: Requires both Data and Orchestration Layers
|
||||
|
||||
Always start services from bottom to top (pyramid order) for proper initialization.
|
||||
|
||||
## 🚦 Service Startup Order
|
||||
|
||||
The `make pyramid` command ensures correct startup order:
|
||||
|
||||
1. **Data Layer Services** (10s wait)
|
||||
- PostgreSQL → Redis → RabbitMQ → MinIO
|
||||
2. **Orchestration Services** (5s wait)
|
||||
- Orchestrator → Analysis
|
||||
3. **Gateway Services** (5s wait)
|
||||
- Kong → Keycloak
|
||||
|
||||
## 📚 Additional Resources
|
||||
|
||||
- [Data Layer README](./data-layer/README.md)
|
||||
- [Orchestration Layer README](./orchestration-layer/README.md)
|
||||
- [Gateway & Auth Layer README](./gateway-auth-layer/README.md)
|
||||
- [Admin Dashboard](../admin-dashboard/README.md)
|
||||
|
||||
---
|
||||
|
||||
**Version**: 1.0.0
|
||||
**Architecture**: Pyramid Microservices
|
||||
**Last Updated**: December 2024
|
||||
Loading…
Add table
Add a link
Reference in a new issue