Sizing Dead-Letter Queues for Claim Replay

A dead-letter queue (DLQ) that is sized wrong quietly becomes the single worst failure mode in an asynchronous adjudication pipeline: too shallow a retry budget discards a recoverable claim after one transient blip, while an unbounded DLQ with infinite retention swallows a poison message that loops until it exhausts broker memory and stalls live point-of-sale traffic. The decision this page settles is narrow and operational — how to set the retry budget (maxReceiveCount), the requeue backoff, the DLQ retention window, and the depth-alarm threshold so that a failed B1 claim can be replayed to a PAID or terminally-REJECTED disposition without loss, without duplication, and without letting one malformed payload take the queue down. It is the resilience layer that sits underneath Asynchronous Batch Adjudication Workflows: once a worker has exhausted its retries against a rate-limited formulary or pricing dependency, exactly where the message lands, how long it survives, and how it is drained back into the main queue all turn on these four numbers.

The Sizing Decision #

DLQ sizing is a trade between recovering transient failures and isolating permanent ones. Set the retry budget too high and a genuine poison message — a claim that can never adjudicate — burns downstream API budget on every redelivery before it finally lands in the DLQ. Set it too low and a claim that hit a five-second formulary 503 gets one retry and is dead-lettered while the dependency was merely draining a deploy. The four levers below are tuned together against the shape of your failure distribution, not in isolation.

Two numbers anchor the sizing. First, size the DLQ retention to at least twice your longest realistic incident window plus the manual-fix turnaround — a 14-day retention covers a Friday-night dependency outage that is not root-caused until the following week. Second, size the depth alarm below the point where DLQ volume implies a systemic fault rather than scattered bad claims: for a stream doing 400 claims/second, a DLQ climbing past a few hundred messages in a minute is a broken snapshot push or a dead dependency, not random noise, and it should page before members feel it at the counter. The transport underneath — Kafka vs RabbitMQ for PBM Adjudication — changes the mechanics (a Kafka retry topic ladder versus a broker-native DLX), but the four levers and their target values do not.

Figure: The retry budget bounds redeliveries — transient failures loop back through an exponential-backoff requeue until receiveCount exhausts maxReceiveCount, then the message is dead-lettered, alarmed on depth, and drained by an idempotent replay consumer once the root cause is fixed.

Step-by-Step Implementation #

The worker below models a broker-agnostic message with a receive counter, applies a bounded exponential-backoff retry, dead-letters on budget exhaustion, and pairs with a replay consumer that is idempotent by construction. Each step names the NCPDP field it touches and honors the PHI boundary from Security & Compliance Boundaries for Claims Data.

1. Model the message with its receive count and a stable idempotency key. The key is derived from routing fields before any PHI is dropped, so a redelivered claim converges on one adjudication. The raw 302-C2 Cardholder ID is hashed at ingress and never carried on the message body.

2. Classify the failure before deciding to retry. A terminal reject (70 Product/Service Not Covered) must never be retried — retrying wastes downstream budget and can double-reject at the pharmacy. Only transient dependency failures (timeouts, 503, 429) consume the retry budget, exactly the distinction drawn in Handling PBM 404 and 503 Errors in Adjudication Scripts.

3. Requeue with exponential backoff while budget remains; dead-letter when it is exhausted. The visibility-timeout backoff caps at a ceiling so a flaky dependency does not stretch recovery to hours, and the DLQ landing carries the last error class for triage.

4. Replay idempotently. The replay consumer drains the DLQ, skips any claim whose idempotency key already reached ADJUDICATED, and requeues the rest — so a fixed poison message re-enters the pipeline exactly once.

from __future__ import annotations
import hashlib
import logging
from dataclasses import dataclass, field
from decimal import Decimal
from enum import Enum
from typing import Callable, Iterator, Optional

logger = logging.getLogger("pbm.dlq")

# Terminal NCPDP 511-FB reject codes must NOT be retried; only transient faults do.
TERMINAL_REJECTS = {"70", "75", "76", "79"}  # covered / PA / plan-limit / refill-too-soon


class FailureClass(str, Enum):
    TRANSIENT = "TRANSIENT"     # timeout, 503, 429 — consumes retry budget
    TERMINAL = "TERMINAL"       # deterministic reject — never retried
    POISON = "POISON"           # unparseable / schema fault — dead-letter immediately


@dataclass
class RetryPolicy:
    max_receive: int = 5              # maxReceiveCount before dead-lettering
    base_backoff_s: int = 2           # exponential base: 2 ** receive_count
    max_backoff_s: int = 60           # ceiling so recovery stays bounded


@dataclass
class ClaimMessage:
    bin: str                          # 101-A1 BIN Number (routing)
    pcn: str                          # 104-A4 Processor Control Number
    ndc: str                          # 407-D7 Product/Service ID (NDC-11)
    service_date: str                 # 401-D1 Date of Service (CCYYMMDD)
    quantity: Decimal                 # 442-E7 Quantity Dispensed — Decimal, never float
    cardholder_hash: str              # sha256 of 302-C2, hashed at ingress (PHI-safe)
    receive_count: int = 0
    last_error: Optional[str] = None

    @property
    def idempotency_key(self) -> str:
        # Derived from routing fields + hashed member; stable across redeliveries.
        raw = f"{self.bin}:{self.pcn}:{self.cardholder_hash}:{self.ndc}:{self.service_date}"
        return hashlib.sha256(raw.encode("utf-8")).hexdigest()[:24]


def hash_cardholder(raw_302c2: str) -> str:
    # 302-C2 Cardholder ID is PHI; hash at ingress, never log or persist the raw value.
    return hashlib.sha256(raw_302c2.encode("utf-8")).hexdigest()[:16] if raw_302c2 else ""


class ReplayableWorker:
    """Bounded-retry worker that dead-letters on budget exhaustion."""

    def __init__(self, policy: RetryPolicy, adjudicate: Callable[[ClaimMessage], FailureClass]):
        self._policy = policy
        self._adjudicate = adjudicate

    def backoff_seconds(self, receive_count: int) -> int:
        # Exponential visibility timeout, capped at the policy ceiling.
        return min(self._policy.base_backoff_s * (2 ** receive_count), self._policy.max_backoff_s)

    def handle(self, msg: ClaimMessage, main_queue: list, dlq: list) -> str:
        outcome = self._adjudicate(msg)

        if outcome is FailureClass.TERMINAL:
            # Deterministic reject — resolved, do not retry, do not dead-letter.
            logger.info("terminal", extra={"key": msg.idempotency_key, "code": msg.last_error})
            return "TERMINAL"

        if outcome is FailureClass.POISON:
            # Schema/parse fault: no retry can fix it — dead-letter on first sight.
            dlq.append(msg)
            logger.warning("poison_dlq", extra={"key": msg.idempotency_key})
            return "DLQ_POISON"

        # TRANSIENT: consume one unit of the retry budget.
        msg.receive_count += 1
        if msg.receive_count >= self._policy.max_receive:
            dlq.append(msg)
            logger.warning("budget_exhausted_dlq",
                           extra={"key": msg.idempotency_key, "receives": msg.receive_count})
            return "DLQ_EXHAUSTED"

        # Requeue after backoff (visibility timeout modeled by the delay field).
        msg.last_error = "TRANSIENT_DEPENDENCY"
        main_queue.append(msg)
        logger.info("requeue", extra={"key": msg.idempotency_key,
                                      "delay_s": self.backoff_seconds(msg.receive_count)})
        return "REQUEUED"


class ReplayConsumer:
    """Drains a DLQ back onto the main queue, exactly-once per idempotency key."""

    def __init__(self, adjudicated_keys: set[str]):
        self._adjudicated = adjudicated_keys  # keys that already reached ADJUDICATED

    def replay(self, dlq: Iterator[ClaimMessage], main_queue: list) -> dict[str, int]:
        stats = {"requeued": 0, "skipped_duplicate": 0}
        for msg in dlq:
            if msg.idempotency_key in self._adjudicated:
                stats["skipped_duplicate"] += 1        # never re-bill a settled claim
                continue
            msg.receive_count = 0                       # reset budget for the fixed cause
            main_queue.append(msg)
            stats["requeued"] += 1
        return stats

The worker is a pure function of the message and policy: the only mutable state is receive_count on the message, so it is safe to run across a horizontally scaled pool. Because the retry budget lives on the message rather than in worker memory, a worker restart mid-retry loses no accounting — the redelivered message still carries its count. Persistent dependency failure that opens a circuit breaker routes through Fallback Routing Logic Design rather than burning the whole retry budget against a dead service.

Verification and Testing Pattern #

The properties that matter are: the budget is exact (no off-by-one that dead-letters early or retries forever), terminal rejects never re-enter the loop, and replay is idempotent. Drive each with a fixture that pins the receive count and failure class.

import pytest
from decimal import Decimal


def make_msg(**kw) -> ClaimMessage:
    base = dict(bin="123456", pcn="PCN01", ndc="00093015001",
                service_date="20260717", quantity=Decimal("30"),
                cardholder_hash=hash_cardholder("MEMBER-A"))
    base.update(kw)
    return ClaimMessage(**base)


def test_transient_retries_then_dead_letters_at_budget():
    policy = RetryPolicy(max_receive=3)
    worker = ReplayableWorker(policy, adjudicate=lambda m: FailureClass.TRANSIENT)
    q, dlq = [], []
    msg = make_msg()
    # Two requeues (counts 1, 2), third attempt exhausts the budget -> DLQ.
    assert worker.handle(msg, q, dlq) == "REQUEUED"
    assert worker.handle(msg, q, dlq) == "REQUEUED"
    assert worker.handle(msg, q, dlq) == "DLQ_EXHAUSTED"
    assert len(dlq) == 1 and msg.receive_count == 3


def test_terminal_reject_never_retries():
    worker = ReplayableWorker(RetryPolicy(), adjudicate=lambda m: FailureClass.TERMINAL)
    q, dlq = [], []
    msg = make_msg()
    assert worker.handle(msg, q, dlq) == "TERMINAL"
    assert q == [] and dlq == []          # no budget consumed, nothing dead-lettered


def test_poison_dead_letters_on_first_sight():
    worker = ReplayableWorker(RetryPolicy(), adjudicate=lambda m: FailureClass.POISON)
    q, dlq = [], []
    assert worker.handle(make_msg(), q, dlq) == "DLQ_POISON"
    assert len(dlq) == 1


def test_backoff_is_capped():
    worker = ReplayableWorker(RetryPolicy(base_backoff_s=2, max_backoff_s=60), adjudicate=lambda m: None)
    assert worker.backoff_seconds(3) == 16       # 2 * 2**3
    assert worker.backoff_seconds(10) == 60      # capped, not 2048


def test_replay_is_idempotent():
    settled = make_msg()
    consumer = ReplayConsumer(adjudicated_keys={settled.idempotency_key})
    q: list = []
    stats = consumer.replay(iter([settled, make_msg(ndc="00093015002")]), q)
    assert stats == {"requeued": 1, "skipped_duplicate": 1}
    assert len(q) == 1                            # the settled claim is not re-billed

The idempotency test is the load-bearing one: it pins the guarantee that draining a DLQ after an incident cannot double-bill a claim that already reached ADJUDICATED while the failed copy sat in the DLQ.

Gotchas and PHI Guardrails #

• Off-by-one on the budget. maxReceiveCount counts deliveries, not retries — a budget of 5 gives four retries after the first attempt. Test the exact boundary; a fencepost error here either dead-letters a recoverable claim early or grants one extra downstream call per claim, which multiplies into real API cost at volume.
• Retrying terminal rejects. The most expensive DLQ defect is looping a 70/76/79 reject through the retry budget. These are deterministic — re-adjudicating produces the same reject and can surface a duplicate rejection at the pharmacy counter. Classify the failure before touching receive_count.
• Unbounded retention. A DLQ with no retention cap is a memory leak with a schedule: poison messages accumulate until broker memory pressure stalls live queues. Set retention (14 days is the common default) and alarm on depth so the DLQ stays an inspection surface, not a graveyard.
• Replay without idempotency. Draining a DLQ into the main queue after an outage will re-bill any claim that adjudicated on a duplicate delivery while the failed copy waited. Gate replay on the ADJUDICATED idempotency-key set, and reset receive_count only for the genuinely-unsettled survivors.
• Backoff with no ceiling. Uncapped exponential backoff pushes a flaky-dependency claim’s next attempt hours out, blowing the claim-day window. Cap the visibility timeout (60s is typical) so recovery stays inside the reconciliation cycle.
• PHI in the dead-letter payload. A message that lands in a DLQ is inspected by humans during triage — so the body must already be PHI-safe. Carry only the hashed 302-C2 reference, the 407-D7 NDC, and the last error class; never the raw claim bytes or the 310-CA Patient Name. A DLQ full of raw payloads is a reportable breach waiting for the first triage session. See the NCPDP Standards reference for the field definitions carried on the message.

Related #

• Asynchronous Batch Adjudication Workflows — the parent workflow whose adj.dlq routing this page sizes and drains.
• Kafka vs RabbitMQ for PBM Adjudication — how the transport choice changes DLQ mechanics from a retry-topic ladder to a broker-native dead-letter exchange.
• Handling PBM 404 and 503 Errors in Adjudication Scripts — the transient-versus-terminal classification that decides which failures consume the retry budget.
• Fallback Routing Logic Design — circuit breakers that short-circuit a dead dependency before it exhausts every claim’s budget.
• Security & Compliance Boundaries for Claims Data — the PHI rules every dead-lettered payload and replay log line must satisfy.

← Back to Asynchronous Batch Adjudication Workflows