Kafka Topics

The platform uses Apache Kafka (segmentio/kafka-go) for all domain events. Topics are organised by bounded context — one topic per domain — and partitioned by entity ID to guarantee in-order delivery for each entity.

Naming Convention

All platform topics follow the pattern platform.{domain}.events.

✅ platform.auth.events
✅ platform.money.events
✅ platform.billing.events
❌ platform.auth.user.events      (entity-level — too granular)
❌ platform.auth.user.123.events  (per-instance — would create millions of topics)

One topic per bounded context. This is the industry standard that balances Kafka ordering guarantees with operational manageability. Splitting by entity would require one topic per record (millions of topics, unmanageable). Merging everything into one topic would destroy per-entity ordering guarantees.

Platform Topics

Topic	Domain	Retention
`platform.auth.events`	IAM	7 days
`platform.module.events`	Module Registry	7 days
`platform.money.events`	Money Service	7 days
`platform.files.events`	File Storage	7 days
`platform.notify.events`	Notify Service	7 days
`platform.audit.events`	Audit Service	30 days
`platform.billing.events`	Billing	7 days
`platform.data.events`	Data Layer (CDC)	7 days

note

Event Catalog For a comprehensive list of all specific events published to each domain topic, please refer to the detailed Event Catalog.

platform.audit.events uses a 30-day retention override (KAFKA_AUDIT_RETENTION_HOURS=720). This prevents loss of audit events during extended maintenance windows or incident investigations. All other topics default to 7 days (KAFKA_RETENTION_HOURS=168).

Partition Key

The partition key for all events is entityId — the primary identifier of the entity the event is about:

Event	`entityId`
`auth.user.created`	`userId`
`money.wallet.credited`	`userId`
`billing.plan.changed`	`tenantId`
`crm.contact.updated`	`contactId`

All events about the same entity are routed to the same Kafka partition. This guarantees per-entity ordering:

Events for user A (userId: 01j9pa5...):
  partition 2 → [auth.user.created, auth.role.changed, auth.user.logged_in]
  Delivered to consumer in this exact order. ✅

Events for user A and user B:
  partition 2 → user A events
  partition 5 → user B events
  Relative ordering between A and B is NOT guaranteed. ❌

Cross-entity ordering is explicitly not guaranteed. This is the fundamental Kafka partition trade-off. If workflow correctness requires strict cross-entity ordering, implement a saga coordinator — do not rely on Kafka topic ordering across different entities.

Consumer Groups

Consumer groups ensure that each event is processed by exactly one instance of a consuming service, even when multiple pods are running.

Naming convention

{consuming-service}.{eventType}-handler

Examples:
  crm.auth.user.created-handler        (CRM listens to auth.user.created)
  notify.money.wallet.credited-handler  (Notify listens to money.wallet.credited)
  audit.auth-logger                     (Audit logs all auth events)

All pods of the same service join the same consumer group. Kafka distributes partitions across pods automatically.

Consumer group per handler

If a service subscribes to multiple event types, each subscription creates its own consumer group:

Module: crm
  subscribe('auth.user.created')      → group: crm.auth.user.created-handler
  subscribe('billing.plan.changed')   → group: crm.billing.plan.changed-handler

This allows each handler to progress through its topic independently. A slow auth.user.created handler does not block the billing.plan.changed handler.

Retention

Parameter	Default	Override
Default retention	7 days (168 hours)	`KAFKA_RETENTION_HOURS=168`
`platform.audit.events`	30 days (720 hours)	`KAFKA_AUDIT_RETENTION_HOURS=720`

Replay beyond the retention window is blocked:

{
  "type": "https://api.septemcore.com/problems/validation-error",
  "status": 400,
  "detail": "Requested replay start is beyond retention window (168h).",
  "code": "EVENTS_RETENTION_EXCEEDED"
}

For longer-term replay needs (disaster recovery, compliance), events must be archived to object storage (S3-compatible) via a separate Kafka Connect sink before they age out.

Max Event Payload

The maximum event payload is 1 MB — the Kafka default message.max.bytes.

Message overhead (envelope headers): ~1 KB
Available for data field: ~1022 KB

This limit covers the combined size of the entire serialised Kafka message, including the envelope fields (id, type, source, tenantId, schemaVersion, timestamp, traceId) plus data.

Claim Check Pattern for Large Payloads

If an event payload would exceed 1 MB (e.g. a report generation result, a bulk import manifest), use the Claim Check pattern:

1. Upload the large payload to object storage (S3-compatible bucket)
2. Get the signed URL / storage key
3. Publish the event with a reference instead of the inline payload

Example:
  Event: report.generated
  Data: {
    "reportId":  "01j9parse700000000000000",
    "storageKey": "reports/01j9p3kz.../01j9parse7.../report.pdf",
    "bucket":    "platform-reports",
    "expiresAt": "2026-04-22T10:30:00.000Z"
  }

Consumer:
  1. Receive event with reference
  2. Download payload from object storage using storageKey
  3. Process full payload

The Claim Check pattern keeps Kafka topics fast and prevents broker memory pressure from large payloads.

Dead Letter Topics

Each domain has a corresponding dead-letter topic for events that fail consumer processing after 3 retries:

Source topic	Dead-letter topic
`platform.auth.events`	`platform.auth.dlq`
`platform.money.events`	`platform.money.dlq`
`platform.billing.events`	`platform.billing.dlq`
… (all domains)	`platform.{domain}.dlq`

Dead-letter topics use the same retention as their source topic. Events in DLQ are visible in Admin → Events → DLQ and can be retried via POST https://api.septemcore.com/v1/events/dlq/{eventId}/retry.

Inspecting Topics

Available topics for the current tenant are discoverable via the SDK:

const topics = await kernel.events().listTopics();
// ['platform.auth.events', 'platform.data.events', ...]

Topic schema (Protobuf message definition) for a given event type:

const schema = await kernel.events().topicSchema('auth.user.created');
// { schemaVersion: '1.0.0', fields: [...], protoFile: '...' }

REST:

GET https://api.septemcore.com/v1/events/topics
Authorization: Bearer <access_token>

GET https://api.septemcore.com/v1/events/topics/{eventType}/schema
Authorization: Bearer <access_token>

Configuration Reference

Environment variable	Default	Description
`KAFKA_BROKERS`	—	Comma-separated broker addresses
`KAFKA_RETENTION_HOURS`	`168`	Default topic retention (7 days)
`KAFKA_AUDIT_RETENTION_HOURS`	`720`	Audit topic retention override (30 days)
`KAFKA_CONSUMER_LAG_ALERT_THRESHOLD`	`10000`	Lag threshold in events before alerting
`DLQ_REPLAY_RATE_LIMIT`	`50`	Max DLQ replay events per second

Naming Convention​

Platform Topics​

Partition Key​

Consumer Groups​

Naming convention​

Consumer group per handler​

Retention​

Max Event Payload​

Claim Check Pattern for Large Payloads​

Dead Letter Topics​

Inspecting Topics​

Configuration Reference​