livrare lot 2
This commit is contained in:
commit
8ecc78e729
763 changed files with 164593 additions and 0 deletions
104
backend/services/data-layer/didiDatabase/ha-cluster/README.md
Normal file
104
backend/services/data-layer/didiDatabase/ha-cluster/README.md
Normal file
|
|
@ -0,0 +1,104 @@
|
|||
# DIDI PostgreSQL HA — Patroni + etcd + HAProxy (IaC livrabil)
|
||||
|
||||
Pachet **reproductibil** care livrează clusterul HA PostgreSQL al platformei DiDi
|
||||
ca Infrastructure-as-Code. Aceeași arhitectură rulează în producție pe VM-uri
|
||||
dedicate (vezi `../MIGRATION.md`); acest compose o reproduce integral pe un
|
||||
singur host pentru demo, recepție, DR-rehearsal și medii de test.
|
||||
|
||||
## Arhitectură
|
||||
|
||||
```
|
||||
┌────────────────────┐
|
||||
apps ──5000──▶ │ HAProxy │ ──▶ /primary (Patroni REST :8008)
|
||||
apps ──5001──▶ │ (LB + healthcheck)│ ──▶ /replica
|
||||
└─────────┬──────────┘
|
||||
┌───────────────┼───────────────┐
|
||||
┌─────▼─────┐ ┌─────▼─────┐ ┌─────▼─────┐
|
||||
│ pg-node1 │ │ pg-node2 │ │ pg-node3 │ Spilo = PostgreSQL 16
|
||||
│ Patroni │ │ Patroni │ │ Patroni │ + Patroni (Zalando)
|
||||
└─────┬─────┘ └─────┬─────┘ └─────┬─────┘
|
||||
└───────────────┼───────────────┘
|
||||
┌────────▼────────┐
|
||||
│ etcd1/2/3 (DCS) │ quorum leader-election
|
||||
└─────────────────┘
|
||||
```
|
||||
|
||||
| Rol | Producție (VM-uri) | Acest pachet |
|
||||
|---|---|---|
|
||||
| PG + Patroni ×3 | 10.11.50.160–162 | `pg-node1..3` (Spilo 16) |
|
||||
| etcd quorum ×3 | 10.11.50.163–165 | `etcd1..3` (v3.5) |
|
||||
| HAProxy | 10.11.50.166 + 169 (VIP .167) | `haproxy` :5000/:5001 |
|
||||
| Backup | pgBackRest (10.11.50.168, NFS) | vezi §Backup |
|
||||
|
||||
## Pornire
|
||||
|
||||
```bash
|
||||
docker compose up -d
|
||||
# election durează ~30-60s; verifică:
|
||||
docker exec didi-ha-pg1 patronictl list
|
||||
```
|
||||
|
||||
Conectare (contract identic cu producția):
|
||||
|
||||
```bash
|
||||
PGPASSWORD=didi-super-secret psql -h localhost -p 5000 -U postgres # RW (leader)
|
||||
PGPASSWORD=didi-super-secret psql -h localhost -p 5001 -U postgres # RO (replici)
|
||||
```
|
||||
|
||||
Restaurare schema DIDI (bos_parammgmt / bos_analysis / bos_sysadmin / bos_subscriber):
|
||||
|
||||
```bash
|
||||
PGPASSWORD=didi-super-secret psql -h localhost -p 5000 -U postgres \
|
||||
-f ../DIDI_full_export_2026-07-02.sql
|
||||
```
|
||||
|
||||
## Test failover (drill de recepție)
|
||||
|
||||
```bash
|
||||
# 1. află liderul
|
||||
docker exec didi-ha-pg1 patronictl list
|
||||
# 2. omoară-l
|
||||
docker stop didi-ha-pg2 # (dacă pg2 e leader)
|
||||
# 3. Patroni promovează o replică în secunde; HAProxy reroutează :5000
|
||||
# automat (healthcheck /primary la 3s, fall 3). Aplicațiile nu schimbă
|
||||
# nimic — se reconectează pe același endpoint.
|
||||
docker exec didi-ha-pg1 patronictl list
|
||||
# 4. reintră nodul căzut ca replică:
|
||||
docker start didi-ha-pg2
|
||||
```
|
||||
|
||||
Switchover planificat (fără downtime):
|
||||
|
||||
```bash
|
||||
docker exec didi-ha-pg1 patronictl switchover didi --force
|
||||
```
|
||||
|
||||
## Parametri
|
||||
|
||||
| Env | Default | Rol |
|
||||
|---|---|---|
|
||||
| `PG_SUPERUSER_PASSWORD` | `didi-super-secret` | postgres superuser |
|
||||
| `PG_ADMIN_PASSWORD` | `didi-admin-secret` | admin role |
|
||||
| `PG_STANDBY_PASSWORD` | `didi-standby-secret` | replicare streaming |
|
||||
|
||||
**Schimbă-le obligatoriu în producție** (`.env` lângă compose).
|
||||
|
||||
## Backup
|
||||
|
||||
În producție backup-ul e pgBackRest (full zilnic + WAL archiving pe NFS,
|
||||
nod dedicat). Pe acest pachet, echivalentul minim:
|
||||
|
||||
```bash
|
||||
docker exec didi-ha-pg1 su postgres -c \
|
||||
'pg_basebackup -h localhost -p 5432 -D /tmp/didi-backup -Ft -z -Xs'
|
||||
```
|
||||
|
||||
## Relația cu livrabilul Lot 2
|
||||
|
||||
- Modulul 5 (Baze de date SQL) cere PostgreSQL cu HA; oferta specifică
|
||||
Patroni + HAProxy. Acest director este implementarea IaC livrată —
|
||||
reproductibilă pe orice host Docker, plus instanțierea de producție
|
||||
documentată în `MIGRATION.md`.
|
||||
- Aplicațiile (agent-v3, didiFramework) sunt agnostice: `PG_HOST:PG_PORT`
|
||||
arată fie spre VIP-ul de producție (`10.11.50.167:5000`), fie spre acest
|
||||
cluster local (`localhost:5000`) — același contract, zero modificări de cod.
|
||||
|
|
@ -0,0 +1,116 @@
|
|||
# ============================================================================
|
||||
# DIDI PostgreSQL HA cluster — Patroni + etcd + HAProxy (IaC, reproducible)
|
||||
#
|
||||
# Containerized mirror of the production topology (see ../MIGRATION.md):
|
||||
# prod: 3× PG/Patroni (10.11.50.160-162) + 3× etcd (163-165)
|
||||
# + 2× HAProxy (166/169, VIP 167) + pgBackRest (168)
|
||||
# here: 3× Spilo (Patroni+PG, Zalando) + 3× etcd + 1× HAProxy
|
||||
# → same failover semantics, single-host footprint for
|
||||
# demo/recepție/DR-rehearsal.
|
||||
#
|
||||
# Endpoints (identical contract to production):
|
||||
# localhost:5000 → leader (read-write) [HAProxy checks Patroni /primary]
|
||||
# localhost:5001 → replicas (read-only) [HAProxy checks Patroni /replica]
|
||||
# localhost:7000 → HAProxy stats UI
|
||||
#
|
||||
# Usage:
|
||||
# docker compose up -d
|
||||
# # wait ~30s for leader election, then:
|
||||
# psql -h localhost -p 5000 -U postgres # password: $PG_SUPERUSER_PASSWORD
|
||||
# # restore DIDI schema:
|
||||
# psql -h localhost -p 5000 -U postgres -f ../DIDI_full_export_2026-07-02.sql
|
||||
# # failover drill:
|
||||
# docker compose stop $(docker compose ps --format '{{.Name}}' | head -1)
|
||||
# # → a replica is promoted in seconds; :5000 keeps serving writes.
|
||||
# ============================================================================
|
||||
|
||||
x-etcd-common: &etcd-common
|
||||
image: quay.io/coreos/etcd:v3.5.16
|
||||
restart: unless-stopped
|
||||
networks: [didi-ha]
|
||||
environment: &etcd-env
|
||||
ETCD_INITIAL_CLUSTER: etcd1=http://etcd1:2380,etcd2=http://etcd2:2380,etcd3=http://etcd3:2380
|
||||
ETCD_INITIAL_CLUSTER_STATE: new
|
||||
ETCD_INITIAL_CLUSTER_TOKEN: didi-pg-ha
|
||||
ETCD_AUTO_COMPACTION_RETENTION: "1"
|
||||
ETCD_ENABLE_V2: "true"
|
||||
|
||||
x-spilo-common: &spilo-common
|
||||
image: ghcr.io/zalando/spilo-16:3.3-p3
|
||||
restart: unless-stopped
|
||||
networks: [didi-ha]
|
||||
environment: &spilo-env
|
||||
SCOPE: didi # Patroni cluster name (etcd namespace)
|
||||
PGVERSION: "16"
|
||||
ETCD3_HOSTS: "'etcd1:2379','etcd2:2379','etcd3:2379'"
|
||||
PGPASSWORD_SUPERUSER: ${PG_SUPERUSER_PASSWORD:-didi-super-secret}
|
||||
PGPASSWORD_ADMIN: ${PG_ADMIN_PASSWORD:-didi-admin-secret}
|
||||
PGPASSWORD_STANDBY: ${PG_STANDBY_PASSWORD:-didi-standby-secret}
|
||||
ALLOW_NOSSL: "true"
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "curl -sf http://localhost:8008/health || exit 1"]
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 6
|
||||
start_period: 60s
|
||||
|
||||
services:
|
||||
etcd1:
|
||||
<<: *etcd-common
|
||||
container_name: didi-ha-etcd1
|
||||
command: etcd --name etcd1
|
||||
--listen-peer-urls http://0.0.0.0:2380 --initial-advertise-peer-urls http://etcd1:2380
|
||||
--listen-client-urls http://0.0.0.0:2379 --advertise-client-urls http://etcd1:2379
|
||||
etcd2:
|
||||
<<: *etcd-common
|
||||
container_name: didi-ha-etcd2
|
||||
command: etcd --name etcd2
|
||||
--listen-peer-urls http://0.0.0.0:2380 --initial-advertise-peer-urls http://etcd2:2380
|
||||
--listen-client-urls http://0.0.0.0:2379 --advertise-client-urls http://etcd2:2379
|
||||
etcd3:
|
||||
<<: *etcd-common
|
||||
container_name: didi-ha-etcd3
|
||||
command: etcd --name etcd3
|
||||
--listen-peer-urls http://0.0.0.0:2380 --initial-advertise-peer-urls http://etcd3:2380
|
||||
--listen-client-urls http://0.0.0.0:2379 --advertise-client-urls http://etcd3:2379
|
||||
|
||||
pg-node1:
|
||||
<<: *spilo-common
|
||||
container_name: didi-ha-pg1
|
||||
hostname: pg-node1
|
||||
depends_on: [etcd1, etcd2, etcd3]
|
||||
volumes: [pg1-data:/home/postgres/pgdata]
|
||||
pg-node2:
|
||||
<<: *spilo-common
|
||||
container_name: didi-ha-pg2
|
||||
hostname: pg-node2
|
||||
depends_on: [etcd1, etcd2, etcd3]
|
||||
volumes: [pg2-data:/home/postgres/pgdata]
|
||||
pg-node3:
|
||||
<<: *spilo-common
|
||||
container_name: didi-ha-pg3
|
||||
hostname: pg-node3
|
||||
depends_on: [etcd1, etcd2, etcd3]
|
||||
volumes: [pg3-data:/home/postgres/pgdata]
|
||||
|
||||
haproxy:
|
||||
image: haproxy:2.9-alpine
|
||||
container_name: didi-ha-haproxy
|
||||
restart: unless-stopped
|
||||
networks: [didi-ha]
|
||||
depends_on: [pg-node1, pg-node2, pg-node3]
|
||||
ports:
|
||||
- "5000:5000" # read-write → Patroni leader
|
||||
- "5001:5001" # read-only → replicas
|
||||
- "7000:7000" # stats UI
|
||||
volumes:
|
||||
- ./haproxy/haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg:ro
|
||||
|
||||
volumes:
|
||||
pg1-data:
|
||||
pg2-data:
|
||||
pg3-data:
|
||||
|
||||
networks:
|
||||
didi-ha:
|
||||
name: didi-ha
|
||||
|
|
@ -0,0 +1,46 @@
|
|||
# HAProxy for DIDI PostgreSQL HA — routes by Patroni REST health checks.
|
||||
# Mirrors the production LB config (10.11.50.166/169 → VIP 167).
|
||||
#
|
||||
# :5000 → the ONE node whose Patroni answers 200 on /primary (leader, RW)
|
||||
# :5001 → nodes answering 200 on /replica (round-robin, RO)
|
||||
#
|
||||
# On failover Patroni flips the health endpoints; HAProxy reroutes in
|
||||
# (inter × fall) ≈ 9s worst case without client config changes.
|
||||
|
||||
global
|
||||
maxconn 300
|
||||
log stdout format raw local0
|
||||
|
||||
defaults
|
||||
log global
|
||||
mode tcp
|
||||
retries 2
|
||||
timeout client 30m
|
||||
timeout connect 4s
|
||||
timeout server 30m
|
||||
timeout check 5s
|
||||
|
||||
listen stats
|
||||
mode http
|
||||
bind *:7000
|
||||
stats enable
|
||||
stats uri /
|
||||
|
||||
listen postgres_write
|
||||
bind *:5000
|
||||
option httpchk GET /primary
|
||||
http-check expect status 200
|
||||
default-server inter 3s fall 3 rise 2 on-marked-down shutdown-sessions
|
||||
server pg-node1 pg-node1:5432 check port 8008
|
||||
server pg-node2 pg-node2:5432 check port 8008
|
||||
server pg-node3 pg-node3:5432 check port 8008
|
||||
|
||||
listen postgres_read
|
||||
bind *:5001
|
||||
balance roundrobin
|
||||
option httpchk GET /replica
|
||||
http-check expect status 200
|
||||
default-server inter 3s fall 3 rise 2 on-marked-down shutdown-sessions
|
||||
server pg-node1 pg-node1:5432 check port 8008
|
||||
server pg-node2 pg-node2:5432 check port 8008
|
||||
server pg-node3 pg-node3:5432 check port 8008
|
||||
Loading…
Add table
Add a link
Reference in a new issue