Backup and Recovery

Platform-Kernel uses four independent backup strategies matched to each stateful component. All backups are encrypted at rest (AES-256).

RTO / RPO Targets

Component	RPO	RTO
PostgreSQL (OLTP, primary data)	< 5 min (WAL streaming)	< 30 min
ClickHouse (OLAP, audit logs)	< 24 h (daily partition export)	< 2 h
SeaweedFS S3 (file storage)	< 24 h (cross-region sync)	< 1 h
Vault (secrets, JWT keys)	< 1 h (hourly Raft snapshot)	< 15 min

PostgreSQL Backup

Databases

PostgreSQL hosts 9 per-service databases (created by docker/postgres/01_init_databases.sh):

Database	Owner service
`platform_kernel`	IAM, Data Layer
`platform_billing`	Billing
`platform_domain`	Domain Resolver
`platform_files`	Files
`platform_integration`	Integration Hub
`platform_module`	Module Registry
`platform_money`	Money
`platform_notify`	Notify
`platform_sonarqube`	SonarQube (CI only)

Method 1: Logical Backup (`pg_dump`)

Use for point-in-time restore of individual databases:

#!/bin/bash
# backup-postgres.sh — run as CronJob or scheduled task
set -euo pipefail

TIMESTAMP=$(date +%Y%m%d_%H%M%S)
BACKUP_DIR="/backups/postgres/${TIMESTAMP}"
PGPASSWORD="${POSTGRES_PASSWORD}"
PG_HOST="postgres"
PG_USER="kernel"

# Databases to back up (all except SonarQube in prod)
DATABASES=(
  platform_kernel
  platform_billing
  platform_domain
  platform_files
  platform_integration
  platform_module
  platform_money
  platform_notify
)

mkdir -p "${BACKUP_DIR}"

for DB in "${DATABASES[@]}"; do
  echo "Backing up ${DB}..."
  PGPASSWORD="${PGPASSWORD}" pg_dump \
    -h "${PG_HOST}" \
    -U "${PG_USER}" \
    --format=custom \
    --compress=9 \
    --no-password \
    "${DB}" \
    > "${BACKUP_DIR}/${DB}.pgdump"
done

# Upload to S3-compatible storage
aws s3 cp "${BACKUP_DIR}" \
  "s3://platform-backups/postgres/${TIMESTAMP}/" \
  --recursive \
  --sse aws:kms

# Retain last 30 days
find /backups/postgres/ -mtime +30 -type d -exec rm -rf {} + 2>/dev/null || true

echo "PostgreSQL backup complete: ${BACKUP_DIR}"

Method 2: WAL Archiving (Continuous)

For streaming RPO < 5 min, configure continuous WAL archiving in postgresql.conf:

# postgresql.conf additions for WAL archiving
wal_level = replica
archive_mode = on
archive_command = 'aws s3 cp %p s3://platform-wal-archive/%f'
archive_timeout = 60          # Force WAL segment every 60 s
max_wal_senders = 3
wal_keep_size = 1024MB

With pgBackRest (recommended for production):

# pgbackrest.conf
[platform-kernel]
pg1-path=/var/lib/postgresql/data
pg1-host=postgres
repo1-type=s3
repo1-s3-bucket=platform-wal-archive
repo1-s3-region=eu-central-1
repo1-cipher-type=aes-256-cbc
repo1-cipher-pass=VAULT_SECRET_HERE

# Full backup (weekly CronJob)
pgbackrest --stanza=platform-kernel backup --type=full

# Differential backup (daily)
pgbackrest --stanza=platform-kernel backup --type=diff

# Point-in-time restore to 2026-04-22 17:00:00 UTC
pgbackrest --stanza=platform-kernel restore \
  --type=time \
  --target="2026-04-22 17:00:00+00"

PostgreSQL Restore (pg_dump)

# Restore a single database from custom-format dump
PGPASSWORD="${POSTGRES_PASSWORD}" pg_restore \
  -h postgres \
  -U kernel \
  --dbname=platform_money \
  --clean \
  --if-exists \
  --no-password \
  /backups/postgres/20260422_170000/platform_money.pgdump

echo "Restore complete. Run goose migrations to verify schema:"
docker exec platform-migrate-money /migrate up

ClickHouse Backup

Scope

ClickHouse stores audit logs (platform_audit database, ReplacingMergeTree). Data retention: 90-day hot in ClickHouse, 7-year cold in S3 Glacier (SeaweedFS clickhouse-cold bucket).

Method: Partition-Level Export

ClickHouse's native BACKUP syntax exports entire tables or partitions to disk or S3:

-- Full backup via ClickHouse BACKUP (native syntax, CH 22.4+)
BACKUP TABLE platform_audit.audit_records
TO S3(
  'https://seaweedfs:8333/clickhouse-cold/backups/audit_records_20260422',
  'admin',
  'admin_secret'
)
SETTINGS
  compression_method = 'zstd',
  compression_level  = 3;

Schedule as a daily CronJob:

#!/bin/bash
# backup-clickhouse.sh
TIMESTAMP=$(date +%Y%m%d)
BACKUP_PATH="clickhouse-cold/backups/${TIMESTAMP}"

clickhouse-client \
  --host clickhouse \
  --port 9000 \
  --user kernel \
  --password "${CLICKHOUSE_PASSWORD}" \
  --query="
    BACKUP TABLE platform_audit.audit_records
    TO S3(
      'http://seaweedfs:8333/${BACKUP_PATH}/audit_records',
      'admin',
      'admin_secret'
    )
    SETTINGS compression_method='zstd', compression_level=3;
  "

ClickHouse Restore

RESTORE TABLE platform_audit.audit_records
FROM S3(
  'http://seaweedfs:8333/clickhouse-cold/backups/20260422/audit_records',
  'admin',
  'admin_secret'
);

-- Verify row count after restore
SELECT count() FROM platform_audit.audit_records FINAL;

Cold Tier — S3 Glacier

Audit data older than 90 days is automatically moved to the clickhouse-cold S3 bucket by a ClickHouse TTL policy:

-- TTL policy (applied in migration)
ALTER TABLE platform_audit.audit_records
  MODIFY TTL
    created_at + INTERVAL 90 DAY TO VOLUME 'cold',
    created_at + INTERVAL 7 YEAR DELETE;

The clickhouse-cold bucket is backed by SeaweedFS with cross-region replication configured in s3_config.json. All objects are encrypted at rest (SeaweedFS SSE-S3 AES-256 + ClickHouse AES-256-CTR).

SeaweedFS S3 Backup

Buckets in Scope

Bucket	Contents	Encryption
`files`	User uploads (processed images, documents)	AES-256 SSE-S3
`clickhouse-cold`	ClickHouse cold tier partitions	AES-256 SSE-S3 + CH AES-256-CTR
`exports`	Tenant data exports	AES-256 SSE-S3

Cross-Region S3 Replication

Replicate to a secondary S3-compatible store (AWS S3, Cloudflare R2, or a second SeaweedFS cluster):

# Using rclone for incremental sync (daily CronJob)
rclone sync \
  seaweedfs:files \
  s3:platform-backup-eu/files \
  --s3-sse aws:kms \
  --transfers 32 \
  --checkers 64 \
  --log-level INFO \
  --stats 1m

rclone sync \
  seaweedfs:exports \
  s3:platform-backup-eu/exports \
  --s3-sse aws:kms \
  --transfers 16

rclone Configuration (SeaweedFS source)

# ~/.config/rclone/rclone.conf
[seaweedfs]
type = s3
provider = Other
access_key_id = platform_access_key
secret_access_key = platform_secret_key
endpoint = http://seaweedfs:8333
force_path_style = true

[s3]
type = s3
provider = AWS
region = eu-central-1
access_key_id = AWS_ACCESS_KEY
secret_access_key = AWS_SECRET_KEY

File Restore

# Restore a single file from backup S3
rclone copy \
  s3:platform-backup-eu/files/{tenantId}/{fileId} \
  seaweedfs:files/{tenantId}/{fileId}

# Restore entire tenant bucket
rclone copy \
  s3:platform-backup-eu/files/{tenantId}/ \
  seaweedfs:files/{tenantId}/ \
  --transfers 32

Vault Backup (Raft Snapshots)

Vault Raft storage supports atomic snapshots. A snapshot captures all KV secrets, policies, auth methods, and token store state.

Automated Snapshot (Hourly)

#!/bin/bash
# backup-vault.sh — run as Kubernetes CronJob (hourly)
set -euo pipefail

TIMESTAMP=$(date +%Y%m%d_%H%M%S)
SNAPSHOT_FILE="/tmp/vault-snapshot-${TIMESTAMP}.snap"

# Create Raft snapshot
vault operator raft snapshot save "${SNAPSHOT_FILE}"

# Upload to S3 (encrypted at rest via KMS)
aws s3 cp "${SNAPSHOT_FILE}" \
  "s3://platform-vault-backups/raft/${TIMESTAMP}.snap" \
  --sse aws:kms

# Verify snapshot integrity
SNAPSHOT_SIZE=$(stat -c%s "${SNAPSHOT_FILE}")
echo "Vault snapshot saved: ${TIMESTAMP}.snap (${SNAPSHOT_SIZE} bytes)"

rm -f "${SNAPSHOT_FILE}"

Kubernetes CronJob

apiVersion: batch/v1
kind: CronJob
metadata:
  name: vault-snapshot
  namespace: platform-infra
spec:
  schedule: "0 * * * *"    # Hourly
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: OnFailure
          containers:
            - name: snapshot
              image: hashicorp/vault:1.19
              command: ["/bin/sh", "/scripts/backup-vault.sh"]
              env:
                - name: VAULT_ADDR
                  value: http://vault.platform-infra.svc.cluster.local:8200
                - name: VAULT_TOKEN
                  valueFrom:
                    secretKeyRef:
                      name: vault-snapshot-token
                      key: token
              volumeMounts:
                - name: scripts
                  mountPath: /scripts
          volumes:
            - name: scripts
              configMap:
                name: vault-backup-scripts

The snapshot token must have the sys/storage/raft/snapshot policy:

vault policy write vault-snapshot - <<EOF
path "sys/storage/raft/snapshot" {
  capabilities = ["read"]
}
EOF
vault write auth/token/create policies=vault-snapshot ttl=0

Vault Restore from Snapshot

# CAUTION: restore overwrites all current Vault data.
# Run during a maintenance window with all services stopped.

# Download snapshot
aws s3 cp \
  s3://platform-vault-backups/raft/20260422_170000.snap \
  /tmp/vault-restore.snap

# Restore (requires Vault to be unsealed but have no leader)
vault operator raft snapshot restore \
  -force \
  /tmp/vault-restore.snap

# Verify by listing secrets
vault kv list secret/platform/iam/

rm -f /tmp/vault-restore.snap

WAL Bloat Response Runbook

When the WAL Bloat alert fires (wal_bytes > 10 GB):

# 1. Check replication slots — stuck slots accumulate WAL
SELECT slot_name, pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn))
  AS lag
FROM pg_replication_slots
ORDER BY lag DESC;

# 2. Drop a permanently stuck slot (DANGER: causes Debezium to re-sync)
SELECT pg_drop_replication_slot('debezium');

# 3. Force WAL checkpoint to release segments
CHECKPOINT;

# 4. Verify WAL size normalised
SELECT pg_size_pretty(sum(size)) AS wal_size
FROM pg_ls_waldir();

After dropping the Debezium slot, restart the Debezium connector to trigger a fresh snapshot. Monitor debezium_source_connector_lag_seconds to ensure it returns to < 5 s.

Backup Verification Schedule

Component	Backup frequency	Restore test frequency	Verified by
PostgreSQL (pg_dump)	Daily + WAL continuous	Monthly	DBA/Ops
ClickHouse partitions	Daily	Quarterly	Data team
SeaweedFS S3	Daily (rclone sync)	Monthly	Ops
Vault Raft	Hourly	Monthly	Platform team

RTO / RPO Targets​

PostgreSQL Backup​

Databases​

Method 1: Logical Backup (pg_dump)​

Method 2: WAL Archiving (Continuous)​

PostgreSQL Restore (pg_dump)​

ClickHouse Backup​

Scope​

Method: Partition-Level Export​

ClickHouse Restore​

Cold Tier — S3 Glacier​

SeaweedFS S3 Backup​

Buckets in Scope​

Cross-Region S3 Replication​

rclone Configuration (SeaweedFS source)​

File Restore​

Vault Backup (Raft Snapshots)​

Automated Snapshot (Hourly)​

Kubernetes CronJob​

Vault Restore from Snapshot​

WAL Bloat Response Runbook​

Backup Verification Schedule​

See Also​