Handling PBM 404 and 503 errors in adjudication scripts

The single implementation decision this page settles is how an adjudication script should branch on an HTTP 404 versus an HTTP 503 returned by a payer gateway — because the two codes belong to opposite failure domains and the wrong branch corrupts adjudication correctness. A 404 is a deterministic client error: the routing key (BIN/PCN or NDC) does not resolve, so every retry produces the same result while burning rate-limit quota and delaying the reject the pharmacy is waiting on. A 503 is a transient server condition: capacity exhaustion or enforced throttling that will clear, so an immediate reject throws away a claim that would have paid on the next attempt. Conflating them is the most common defect in hand-written retry loops — it either dead-letters payable claims or hammers a dead route until the token bucket is empty. This page fixes the branch precisely, mapping each code to one action and one NCPDP 511-FB Reject Code. It sits under the PBM API Sync & Rate Limiting workflow and assumes claims have already been decomposed by NCPDP D.0 Message Parsing Strategies and structurally cleared by Schema Validation & Error Categorization, so the only failures reaching this handler are transport-level.

Decision matrix: 404 vs 503 (and the codes they get confused with)

Before writing the branch, fix the contract. Each transport response maps to exactly one retry policy, one circuit-breaker action, and one claim-level outcome. The two rows that matter most are bolded; the surrounding rows exist because real gateways blur the boundaries between them.

HTTP status Failure domain Retry? Trips breaker? NCPDP 511-FB outcome Typical p50 handler cost
200 Success (paid or clinically rejected) No Records success Payer reject passed through ~0 ms (return)
404 Deterministic routing / formulary miss No No 04 (M/I PCN) or 70 (Product Not Covered) < 1 ms — cache + reject
503 Transient capacity / rate-limit Yes, honor Retry-After Yes, after exhaustion 99 only after retries exhausted 1–30 s (backoff wait)
429 Explicit throttle Yes, honor Retry-After No None — resubmit same claim 1–30 s
408/500/502/504 Server fault / timeout Yes, bounded Yes 99 after exhaustion 1–30 s
401 Credential drift mid-batch Refresh once, replay No None — transparent token refresh RTT

Three field-level distinctions drive the branch:

  • 404 never means “missing REST endpoint.” In a PBM gateway it means an invalid 101-A1 BIN Number / 104-A4 PCN combination, or a 407-D7 Product/Service ID (NDC) that is not in the payer’s active directory. The correct response is a claim-level reject, not a network retry. Map it to reject 04 when the routing tuple is wrong, or 70 when the NDC is simply not covered under this plan.
  • 503 frequently masquerades as 429. Several payers return 503 for enforced rate limiting rather than 429. Distinguish the two by inspecting Retry-After and X-RateLimit-Remaining: 0 in the response headers; when they are present, the 503 is a throttle signal and must route to backoff, not to the failure counter that trips the breaker.
  • Only 503 exhaustion earns reject 99. A 99 Host Processing Error is a terminal state you write after the retry budget is spent — never on the first 503, or you convert a two-second blip into a rejected claim.
404 versus 503 handler timelines and their 511-FB terminal codes The top timeline shows the deterministic 404 path: a single POST returns HTTP 404 and, with no retry, short-circuits directly to a terminal reject 04 written into NCPDP field 511-FB — one request, quota preserved. The bottom timeline shows the transient 503 path as a retry loop: a first POST returns 503, the handler sleeps for the Retry-After interval, POSTs a second time, receives another 503, applies jittered exponential backoff, and POSTs a third time. That final attempt branches to one of two terminal states: an HTTP 200 paid claim with no reject code written, or reject 99 in 511-FB once the retry budget is exhausted. HTTP 404 — deterministic routing / formulary miss response no retry Claim POST attempt 1 404 Reject 04 written to 511-FB 1 POST · no retry rate-limit quota preserved HTTP 503 — transient capacity / rate-limit 200 budget spent Claim POST try 1 503 wait Retry-After POST try 2 503 jittered backoff POST try 3 200 paid no reject Reject 99 511-FB terminal

Figure: The 404 path spends exactly one request before writing reject 04 to 511-FB; the 503 path retries — honoring Retry-After, then jittered backoff — and only writes reject 99 once the budget is exhausted.

Step-by-step implementation

The handler below adjudicates a single NCPDP D.0 claim with the branch fixed above, then streams claims in fixed windows so a 503 storm cannot exhaust memory. It shares an event loop with the pool described in Asynchronous Batch Adjudication Workflows; the circuit-breaker mechanics are the timeout-driven variant covered in Implementing Circuit Breakers for PBM API Timeouts.

  1. Redact PHI before anything is logged. Never emit raw claim bytes. Strip 302-C2 Cardholder ID and 310-CA Patient First Name immediately after routing; only non-PHI routing keys (101-A1, 104-A4, 407-D7) may reach a log line.
  2. Branch 404 to an immediate reject and cache the failed routing tuple so the next identical claim never spends a request slot.
  3. Branch 503 to backoff — honor Retry-After when the payer supplies it, otherwise apply jittered exponential backoff to avoid a thundering herd on a recovering endpoint.
  4. Write reject 99 only when the 503 retry budget is exhausted.
python
import asyncio
import random
import logging
from decimal import Decimal
from typing import AsyncGenerator, Any
import aiohttp

logging.basicConfig(level=logging.INFO, format="%(asctime)s | %(levelname)s | %(message)s")
logger = logging.getLogger("pbm_adjudication")

# NCPDP 511-FB Reject Code values keyed by terminal HTTP status.
NCPDP_REJECTION_MAP = {
    404: {"code": "04", "field": "511-FB", "desc": "M/I Processor Control Number (routing)"},
    503: {"code": "99", "field": "511-FB", "desc": "Host Processing Error (capacity exhausted)"},
}
GENERIC_REJECT = {"code": "99", "field": "511-FB"}


def redact(claim: dict[str, Any]) -> dict[str, str]:
    """Log-safe projection: routing keys only. 302-C2 Cardholder ID and
    310-CA Patient First Name are PHI and are dropped, never logged."""
    return {
        "bin": claim.get("101-A1", ""),   # 101-A1 BIN Number (routing, not PHI)
        "pcn": claim.get("104-A4", ""),   # 104-A4 PCN (routing, not PHI)
        "ndc": claim.get("407-D7", ""),   # 407-D7 NDC (product id, not PHI)
        "cardholder": "***REDACTED***",   # 302-C2 never reaches disk
    }


def apply_jittered_backoff(attempt: int, base_delay: float = 1.0, max_delay: float = 30.0) -> float:
    """Exponential backoff with uniform jitter to avoid a thundering herd."""
    delay = min(base_delay * (2 ** attempt), max_delay)
    return delay + random.uniform(0, delay * 0.1)


async def adjudicate_claim(
    session: aiohttp.ClientSession,
    claim: dict[str, Any],
    max_retries: int = 3,
    base_delay: float = 1.0,
) -> dict[str, Any]:
    """Adjudicate one NCPDP D.0 claim with deterministic 404/503 handling.
    404 -> immediate reject (no retry); 503 -> backoff-and-retry; exhaustion -> 99."""
    provider_id = claim.get("201-B1", "")            # 201-B1 Service Provider ID (routing)
    url = f"https://api.pbm-gateway.com/v1/adjudicate/{provider_id}"

    for attempt in range(max_retries):
        try:
            async with session.post(
                url, json=claim, timeout=aiohttp.ClientTimeout(total=10)
            ) as resp:
                if resp.status == 200:
                    body = await resp.json()
                    # Monetary fields are Decimal, never float, so cent values
                    # stay exact for reconciliation (505-F5 Patient Pay Amount).
                    if "patient_pay_amount" in body:
                        body["patient_pay_amount"] = Decimal(str(body["patient_pay_amount"]))
                    return body

                # 404: deterministic routing/formulary miss. Reject now, no retry.
                if resp.status == 404:
                    rej = NCPDP_REJECTION_MAP[404]
                    claim[rej["field"]] = rej["code"]        # write 04 into 511-FB
                    logger.info("Reject 04 (HTTP 404)", extra={"claim": redact(claim)})
                    return claim

                # 503: transient capacity/throttle. Honor Retry-After, else backoff.
                if resp.status == 503:
                    header_val = resp.headers.get("Retry-After")
                    delay = (
                        float(header_val)
                        if header_val is not None
                        else apply_jittered_backoff(attempt, base_delay)
                    )
                    logger.warning(
                        "503; backing off %.2fs (attempt %d)", delay, attempt + 1,
                        extra={"claim": redact(claim)},
                    )
                    await asyncio.sleep(delay)
                    continue

                # Other unexpected 4xx/5xx: generic reject 99.
                claim[GENERIC_REJECT["field"]] = GENERIC_REJECT["code"]
                logger.warning("Unhandled %s -> reject 99", resp.status,
                               extra={"claim": redact(claim)})
                return claim

        except aiohttp.ClientError:
            # Log the exception TYPE only — never the payload or PHI.
            logger.error("Network failure (attempt %d)", attempt + 1,
                         extra={"claim": redact(claim)})
            if attempt < max_retries - 1:
                await asyncio.sleep(apply_jittered_backoff(attempt, base_delay))
            else:
                claim[GENERIC_REJECT["field"]] = GENERIC_REJECT["code"]
                return claim

    # 503 retry budget exhausted -> terminal 99 in 511-FB.
    rej = NCPDP_REJECTION_MAP[503]
    claim[rej["field"]] = rej["code"]
    return claim


async def stream_adjudication(
    session: aiohttp.ClientSession,
    claim_stream: AsyncGenerator[dict[str, Any], None],
    batch_size: int = 500,
) -> AsyncGenerator[dict[str, Any], None]:
    """Process claims in streaming windows so a 503 spike cannot OOM the worker."""
    buffer: list[dict[str, Any]] = []
    async for claim in claim_stream:
        buffer.append(claim)
        if len(buffer) >= batch_size:
            for res in await asyncio.gather(
                *(adjudicate_claim(session, c) for c in buffer), return_exceptions=True
            ):
                if isinstance(res, Exception):
                    logger.error("Batch error: %s", type(res).__name__)
                else:
                    yield res
            buffer.clear()

    if buffer:  # flush the final partial window
        for res in await asyncio.gather(
            *(adjudicate_claim(session, c) for c in buffer), return_exceptions=True
        ):
            if not isinstance(res, Exception):
                yield res
HTTP status routing and 503 retry loop for the adjudication handler A POST to the adjudication gateway reaches a status-code decision node with four branches. A 200 returns the success payload. A 404 short-circuits to reject 04 written into NCPDP field 511-FB with no retry. Any other 4xx or 5xx writes a generic reject 99 into 511-FB. A 503 flows down to a second decision that tests whether a Retry-After header is present: when it is, the handler sleeps for the Retry-After interval; when it is not, it sleeps a jittered exponential backoff. Both paths converge on a retries-remaining decision. If retries remain, control loops back to the POST; once the budget is exhausted, the handler writes reject 99 into 511-FB. 200 404 503 other 4xx / 5xx yes no sleep sleep yes · retry no · exhausted POST /adjudicate one NCPDP D.0 claim Status? Return success paid or clinical reject Reject 04 · no retry 511-FB · cache tuple Retry-After header? Sleep Retry-After payer-supplied delay Jittered backoff exponential, capped 30 s Retries left? Reject 99 511-FB · budget exhausted Reject 99 · generic 511-FB · unhandled fault

Figure: HTTP status routing — 200 passthrough, 404 immediate reject (04, no retry), 503 into the Retry-After / backoff loop, with exhaustion and other 4xx/5xx faults mapping to reject 99.

Verification and testing pattern

Correctness here is behavioral, not just structural: the test suite must prove that a 404 produces exactly one POST and a reject 04, and that a 503 retries then either succeeds or writes reject 99. Assert on call counts against a mocked transport so a regression that silently retries a 404 — the quota-burning defect — fails the build. Use aioresponses to script gateway responses against known NCPDP fixtures.

python
import pytest
from aioresponses import aioresponses
import aiohttp
from adjudication import adjudicate_claim  # module under test

FIXTURE = {                    # a minimal routed NCPDP D.0 claim
    "201-B1": "PROV123",       # Service Provider ID
    "101-A1": "610279",        # BIN Number
    "104-A4": "PCS",           # PCN
    "407-D7": "00071015523",   # 407-D7 NDC-11
}
URL = "https://api.pbm-gateway.com/v1/adjudicate/PROV123"


@pytest.mark.asyncio
async def test_404_rejects_once_without_retry():
    with aioresponses() as m:
        m.post(URL, status=404)   # scripted once; a retry would raise "no mock"
        async with aiohttp.ClientSession() as s:
            result = await adjudicate_claim(s, dict(FIXTURE), max_retries=3)
    assert result["511-FB"] == "04"          # deterministic routing reject
    assert len(m.requests) == 1              # NO retry burned on a dead route


@pytest.mark.asyncio
async def test_503_backs_off_then_succeeds():
    with aioresponses() as m:
        m.post(URL, status=503, headers={"Retry-After": "0"})   # first: throttled
        m.post(URL, status=200, payload={"status": "P"})        # second: paid
        async with aiohttp.ClientSession() as s:
            result = await adjudicate_claim(s, dict(FIXTURE), max_retries=3)
    assert result["status"] == "P"           # recovered, not rejected
    assert "511-FB" not in result            # no reject written on success


@pytest.mark.asyncio
async def test_503_exhaustion_writes_reject_99():
    with aioresponses() as m:
        for _ in range(3):
            m.post(URL, status=503, headers={"Retry-After": "0"})
        async with aiohttp.ClientSession() as s:
            result = await adjudicate_claim(s, dict(FIXTURE), max_retries=3)
    assert result["511-FB"] == "99"          # terminal only after budget spent

Gotchas and PHI guardrails

  • A 404 cache with too long a TTL hides formulary publishes. Negative-result caching of failed BIN/PCN and NDC tuples is the biggest quota saver, but if a molecule was just added to the plan, a stale negative cache keeps rejecting a now-covered claim as 70. Keep the TTL short (seconds to a low minute count) and invalidate on formulary-version bumps sourced from the NDC-to-GPI Crosswalk Automation pipeline.
  • Unbounded 503 recursion or retry counts starve the point-of-sale path. A patient is waiting at the counter. Cap retries (max_retries) and the backoff ceiling (min(2 ** attempt, 30)), and budget total wall-clock time against the pharmacy SLA — prefer failing fast to a dead-letter lane over a long backoff chain on the live path.
  • Retry-After can arrive as an HTTP-date, not just seconds. RFC 7231 permits both an integer-seconds form and a date form. A bare float(header_val) raises on the date form; parse defensively and fall back to jittered backoff when the value is non-numeric, rather than crashing the worker.
  • Never let PHI reach a log line or an exception trace. Every log statement above uses redact(); exception handlers log type(e).__name__, never the payload. 302-C2 Cardholder ID and 310-CA are stripped immediately after routing, satisfying the constraints in Security & Compliance Boundaries for Claims Data. Validate payloads against the official NCPDP D.0 Implementation Guide before transmission, and keep monetary results in decimal.Decimal so a reconciled claim never drifts by a cent.
  • A 503 that trips the breaker unnecessarily is a self-inflicted outage. When Retry-After plus X-RateLimit-Remaining: 0 are present, the 503 is a throttle, not a fault — route it to backoff, not the failure counter feeding Fallback Routing Logic Design.

For production async patterns and connection pooling, consult the official aiohttp client documentation.

← Back to PBM API Sync & Rate Limiting