Dead Letter Queue

The Dead Letter Queue (DLQ) captures every outgoing external API call that exhausted all retry attempts (5 retries with exponential backoff) and still failed. This ensures that no failed request is silently dropped — every failure is visible, inspectable, and manually retriable.

---

When a Request Enters the DLQ

POST /api/v1/integrations/call → circuit breaker CLOSED → external call made
  │
  │  External API returns 5xx or timeout
  ▼
Retry 1: wait 1s ±jitter → still fails
Retry 2: wait 2s ±jitter → still fails
Retry 3: wait 4s ±jitter → still fails
Retry 4: wait 8s ±jitter → still fails
Retry 5: wait 8s ±jitter → still fails
  │
  ▼
Max retries exhausted:
  → Persist to DLQ (PostgreSQL integration_dlq table)
  → Enqueue in RabbitMQ for async retry (when ops team triggers retry-all)
  → Return 502 Bad Gateway to the calling module

4xx responses from the external provider are not retried and do not enter the DLQ — they represent client-side errors (invalid request, wrong credentials) that retrying cannot fix. A 401 from Stripe for an invalid API key goes directly to the calling module as 502 Bad Gateway with the provider's error detail preserved.

---

DLQ Schema

Field	Type	Description
id	UUID	Unique identifier of the DLQ entry
providerId	UUID	Provider the request was destined for
providerName	string	Provider name snapshot (for readability after provider deletion)
originalRequest	JSON	Full outgoing request: method, path, headers, body
error	string	Last error message from the external API or network layer
attempts	int	Total number of attempts made (including the initial call)
tenantId	UUID	Tenant that initiated the call
createdAt	ISO 8601	When the original call was first attempted
lastRetryAt	ISO 8601	Timestamp of the last retry attempt (null if never retried)

---

Idempotency

Every outgoing request — including DLQ retries — carries a unique idempotency key (UUID v4):

Original call:  idempotencyKey = "a1b2c3d4-..."
Retry 1:        idempotencyKey = "a1b2c3d4-..."  (same key during auto-retry)
Retry 2, 3, 4, 5: same key

DLQ manual retry:
  → New idempotencyKey = "e5f6g7h8-..."  (fresh UUID per retry attempt)
  → External providers receiving duplicate requests with the same key
    treat them as idempotent (Stripe, PayPal support this natively)

Using a fresh key for manual DLQ retries ensures that:

1. The external provider does not treat it as a duplicate of a request that may have partially completed
2. The Hub accurately tracks that this is a new attempt

---

Get DLQ Entries

GET https://api.septemcore.com/v1/integrations/dlq?page=1&limit=20
Authorization: Bearer <access_token>

{
  "items": [
    {
      "id":           "01j9pdlq0000000000000000",
      "providerId":   "01j9pint0000000000000000",
      "providerName": "Stripe Production",
      "error":        "connection timeout after 10000ms",
      "attempts":     6,
      "tenantId":     "01j9ten0000000000000000",
      "createdAt":    "2026-04-22T01:30:00Z",
      "lastRetryAt":  null
    }
  ],
  "total":  1,
  "page":   1,
  "limit":  20
}

Filtering

Query parameter	Description
providerId	Filter by specific provider
since	ISO 8601 timestamp — entries created after this time
page	Page number (min 1)
limit	Items per page (min 1, max 100)

---

Inspect a DLQ Entry

GET https://api.septemcore.com/v1/integrations/dlq/01j9pdlq0000000000000000
Authorization: Bearer <access_token>

{
  "id":           "01j9pdlq0000000000000000",
  "providerId":   "01j9pint0000000000000000",
  "providerName": "Stripe Production",
  "originalRequest": {
    "method":  "POST",
    "path":    "/v1/charges",
    "headers": { "Content-Type": "application/json" },
    "body":    { "amount": 2999, "currency": "usd" }
  },
  "error":       "connection timeout after 10000ms",
  "attempts":    6,
  "tenantId":    "01j9ten0000000000000000",
  "createdAt":   "2026-04-22T01:30:00Z",
  "lastRetryAt": null
}

Security: The originalRequest.headers field never contains the decrypted credentials (API key, token). Auth headers are injected at call time from the encrypted credential store. The DLQ stores only the non-sensitive request metadata.

---

Retry a Single Entry

POST https://api.septemcore.com/v1/integrations/dlq/01j9pdlq0000000000000000/retry
Authorization: Bearer <access_token>

Response 202 Accepted:

{
  "dlqId":          "01j9pdlq0000000000000000",
  "status":         "queued",
  "idempotencyKey": "e5f6g7h8-0000-0000-0000-000000000000"
}

202 Accepted — the retry is queued in RabbitMQ. The result is not synchronous. Poll GET /integrations/dlq/{id} to check if lastRetryAt updated and whether the entry was cleared.

If the retry succeeds:

• The DLQ entry is deleted from the table
• The calling module can re-submit their business operation if needed

If the retry fails again:

• attempts counter increments
• lastRetryAt is updated
• Entry remains in DLQ for the next manual retry or retry-all

---

Retry All

POST https://api.septemcore.com/v1/integrations/dlq/retry-all
Authorization: Bearer <access_token>
Content-Type: application/json

{
  "providerId": "01j9pint0000000000000000"
}

providerId is optional. When omitted, all DLQ entries for the tenant are queued for retry.

Response 202 Accepted:

{
  "queued": 12,
  "status": "queued"
}

Retries are processed from the RabbitMQ queue with the same exponential backoff and circuit breaker protections as original calls. If the circuit breaker for a provider is OPEN, retry-all entries for that provider are held in the queue until the circuit transitions to HALF_OPEN.

---

Delete a DLQ Entry

DELETE https://api.septemcore.com/v1/integrations/dlq/01j9pdlq0000000000000000
Authorization: Bearer <access_token>

Response 204 No Content.

Use this to acknowledge and discard a failed request that does not need to be retried (for example: a call that was already handled manually via the external provider's dashboard).

---

Retention Policy

Parameter	Value
Retention period	30 days
Cleanup	Background worker: soft-deletes entries older than 30 days daily
After soft-delete	Entry removed from all GET responses. Metadata retained for audit.

Entries are not deleted immediately on expiry — the background worker runs once per day. Entries slightly older than 30 days may appear until the next worker run.