Audit Retention

Audit records are retained across two storage tiers for the entire 7-year mandatory period (AML/KYC compliance). The tier transition is automatic via ClickHouse TTL — modules do not control or trigger it.

Absolute rule: No audit record is deleted before 7 years have elapsed. Physical deletion before this period is a compliance violation. The deleted_at column exists exclusively for the GDPR anonymization workflow and does not remove data from storage.

---

Storage Tiers

Period	Storage	Access latency	Environment variable
0 – 90 days	ClickHouse (hot)	Milliseconds — instant search, Admin UI	AUDIT_HOT_DAYS=90
91 days – 7 years	S3 Glacier (cold)	3 – 12 hours for restoration	AUDIT_COLD_YEARS=7
After 7 years	Permanently deleted	No access	AML/KYC requirement

---

Hot Storage: ClickHouse (0 – 90 Days)

Data in ClickHouse is immediately available for millisecond search through GET /audit and GET /audit/entity/:type/:id. The Admin UI and all compliance query flows operate exclusively against hot storage.

ClickHouse TTL Rule

Records are moved from ClickHouse to S3 Glacier automatically when they reach 90 days:

ALTER TABLE audit_logs
  MODIFY TTL toDate(timestamp) + INTERVAL 90 DAY
  TO VOLUME 's3_cold';

This TTL is applied at the table level. ClickHouse evaluates it during background merge cycles — records are moved in batches, not record-by-record. The exact moment of move may lag the TTL by the duration of the next merge cycle (typically minutes to hours), but always occurs before 91 days.

ClickHouse Compression

ClickHouse uses LZ4 compression by default — the block-level compression codec applied to all columns at rest and during replication. LZ4 achieves approximately 70% compression ratio on audit log data (highly repetitive text fields: action, entity_type, tenant_id).

Metric	Value
Codec	LZ4 (default ClickHouse)
Compression ratio	~70% for audit log workloads
Average raw record size	~1 KB
Average compressed record size	~300 bytes

A tenant generating 1 million audit records per month requires approximately 290 GB of raw storage but only ~90 GB after LZ4 compression in ClickHouse.

---

Cold Storage: S3 Glacier (91 Days – 7 Years)

Records moved to S3 Glacier are not queryable through the standard GET /audit API. Retrieval requires an explicit restoration request, which takes 3 – 12 hours depending on the S3 Glacier restore tier.

Standard query path (hot data):
  GET /api/v1/audit → ClickHouse → millisecond response

Cold data path (not available via API):
  Legal/compliance request
    → Platform Owner initiates Glacier restore
    → Wait 3–12 hours
    → Temporary restore to S3 Standard
    → Audit Service reads from S3 Standard
    → Returns via internal compliance export tool

Cold data is not exposed through GET /audit/export or any standard API endpoint. Only the Platform Owner's internal compliance tooling can initiate and read cold restorations.

S3 Lifecycle Rule

After the cold storage period expires (7 years from the original record timestamp), S3 applies a lifecycle rule to permanently delete the objects:

S3 Lifecycle Policy:
  Rule: Move to Glacier after 90 days ← (ClickHouse TTL handles upload, S3 rule manages deletion)
  Rule: Delete from Glacier after AUDIT_COLD_YEARS × 365 days

---

Data Volume Estimates

Use these estimates for infrastructure planning:

Throughput	Daily records	Monthly raw	Monthly compressed
Low (startup tier)	100 000	~3 GB	~900 MB
Medium (scale tier)	1 000 000	~30 GB	~9 GB
High (enterprise tier)	10 000 000	~300 GB	~90 GB

ClickHouse performs well at hundreds of millions of rows per table partition. The compound partition key (toYYYYMM(timestamp), tenant_id) ensures each month-tenant combination is a separate ClickHouse part, allowing time-range queries to prune irrelevant partitions completely.

---

Retention Timeline Walkthrough

Record created:  2026-04-15T10:30:00Z

Day 0–90:        Available in ClickHouse. Instant search via GET /audit.
                 LZ4 compressed at rest.

Day 90:          ClickHouse TTL fires. Record moved to S3 Glacier
                 (background merge cycle, may take minutes to hours).

Day 90–2557:     Record in S3 Glacier. Not queryable via API.
(~7 years)       Retrieval requires Platform Owner initiated restore (3–12h).

Day 2557+        S3 Lifecycle rule deletes record permanently.
(after 7 years)  No access. No recovery possible.

---

Module Developer Notes

• Modules do not configure or control retention — tier transitions are fully automated by the platform.
• Do not store PII in audit records expecting it will be automatically purged after 90 days. Hot-to-cold transition moves data, it does not delete it. PII must be anonymized via GDPR Anonymization before any record enters cold storage.
• Financial audit records (money.* actions) are retained for the full 7 years and are never anonymized (AML/KYC compliance).