Fallback Routing Logic Design

In Pharmacy Benefit Manager (PBM) claims adjudication automation, the primary routing path is engineered for high-throughput, low-latency execution — but production traffic routinely meets transient network degradation, vendor API throttling, and formulary schema drift. Fallback routing logic is the control plane that keeps adjudication moving when the primary path fails: it classifies the failure, decides between a bounded retry, an identifier repair, or a failover to a secondary adjudication node, and records every decision as an audit event. This subsystem sits directly on top of the PBM Architecture & Taxonomy Foundations — it inherits that spine’s canonical claim model, its versioned taxonomy, and its PHI boundary — and it exists so that a degraded vendor never becomes a wall of pharmacy-counter rejections. This page specifies the trigger rules, a deployable Python router, the failure-mode taxonomy, and the correctness tuning that make failover deterministic and replayable.

Prerequisites #

Fallback routing is a downstream consumer, not an entry point. Before the router runs, the following must already be in place:

• A normalized canonical claim. The router never operates on raw wire bytes. Inbound NCPDP Telecommunication Standard D.0 payloads are parsed and normalized upstream by NCPDP D.0 message parsing and validated by schema validation and error categorization. By the time a claim reaches routing, 302-C2 Cardholder ID has already been tokenized and the transport PHI discarded.
• A synchronized crosswalk cache. Identifier repair during failover depends on a warm, version-stamped NDC-to-GPI map maintained by NDC-to-GPI Crosswalk Automation. A cold cache turns a repairable reject into a terminal one.
• Eligibility and PA state parity. Member eligibility snapshots, copay accumulators, and prior-authorization statuses must propagate to the secondary node per the PBM Portal Sync Architecture; routing to a node with stale state produces a technically successful but financially wrong adjudication.
• Library baseline. Python 3.11+, httpx>=0.27 for async HTTP with per-request timeouts, pydantic>=2.6 for strict payload contracts, and structlog>=24.1 for JSON telemetry. Money-bearing fields use the standard-library decimal module — never float.
• Compliance envelope. An append-only event store (the same one described in security and compliance boundaries for claims data) must be reachable so that every routing decision is serialized before the response is returned.

Trigger Matrix: When Fallback Activates #

Fallback activation is never arbitrary. A configurable decision matrix maps the failure signal — NCPDP reject code, HTTP status envelope, or schema-validation outcome — to a routing action. Three broad primary-path failures dominate:

1. Vendor latency / timeouts — the request exceeds the SLA budget (for example >800 ms retail, >1200 ms specialty) or the connection drops.
2. Schema mismatches — malformed segments or a pydantic.ValidationError against the canonical Claim contract.
3. Identifier resolution errors — an unrecognized or discontinued NDC in 407-D7, a missing GPI mapping, or a stale formulary tier reference.

The matrix below is the load-bearing specification. Only repairable rejects are eligible for identifier repair; terminal rejects (genuine coverage decisions) must be passed through unchanged, because retrying them on a second node produces the same reject and wastes the latency budget.

Reject codes 70, 75, 76, and 79 are adjudication outcomes, not routing failures — the single most common fallback bug is treating them as retryable and hammering both nodes with a claim that was correctly rejected. The router’s first job is to separate transport failure from adjudication decision.

Figure: The trigger matrix as a decision fan — one primary path failed signal is classified into a transient lane (retry then failover), a repairable lane (crosswalk repair then re-route), or a terminal lane (pass through unchanged). Only the first two lanes may touch a second node; terminal rejects 70/75/76/79 are genuine coverage decisions.

Reference Python Implementation #

The router below is deployable. It validates the canonical payload with Pydantic, drives a circuit breaker, attempts identifier repair through the crosswalk cache before failover, and emits structured telemetry keyed by a correlation ID — never by member identity. Every field that maps to the wire carries its NCPDP field code as an inline comment, and the PHI guardrail is explicit: 302-C2 and 310-CA are tokenized upstream and the raw values are never logged or re-serialized here.

import asyncio
import time
from decimal import Decimal
from enum import Enum
from typing import Any, Optional

import httpx
import structlog
from pydantic import BaseModel, Field, ValidationError

# JSON telemetry -> SIEM. GUARDRAIL: never log raw claim bytes; 302-C2 Cardholder ID
# and 310-CA Cardholder Name are tokenized upstream and never appear in a log line.
log = structlog.get_logger()


class CircuitState(str, Enum):
    CLOSED = "closed"
    OPEN = "open"
    HALF_OPEN = "half_open"


class CircuitBreaker:
    """Per-vendor breaker: stops hammering a failing endpoint and probes recovery."""

    def __init__(self, failure_threshold: int = 5, recovery_timeout: float = 30.0):
        self.failure_threshold = failure_threshold
        self.recovery_timeout = recovery_timeout
        self.failure_count = 0
        self.last_failure = 0.0
        self.state = CircuitState.CLOSED

    def record_failure(self) -> None:
        self.failure_count += 1
        self.last_failure = time.monotonic()
        if self.failure_count >= self.failure_threshold:
            self.state = CircuitState.OPEN
            log.warning("circuit_opened", threshold=self.failure_threshold)

    def record_success(self) -> None:
        self.failure_count = 0
        self.state = CircuitState.CLOSED

    def can_execute(self) -> bool:
        if self.state is CircuitState.OPEN:
            if time.monotonic() - self.last_failure > self.recovery_timeout:
                self.state = CircuitState.HALF_OPEN  # allow one probe
                log.info("circuit_half_open", timeout=self.recovery_timeout)
                return True
            return False
        return True  # CLOSED or HALF_OPEN


class CanonicalClaim(BaseModel):
    """Canonical claim after ingress normalization (transport PHI already stripped)."""

    correlation_id: str                       # internal UUID, safe to log
    cardholder_token: str                     # 302-C2 Cardholder ID, ALREADY tokenized
    group_id: str                             # 301-C1 Group ID
    ndc: str = Field(pattern=r"^\d{11}$")     # 407-D7 Product/Service ID, 11-digit 5-4-2
    pharmacy_npi: str = Field(pattern=r"^\d{10}$")  # 201-B1 Service Provider ID (NPI)
    qty_dispensed: Decimal                    # 442-E7 Quantity Dispensed
    days_supply: int                          # 405-D5 Days Supply
    rx_ref: str                               # 402-D2 Prescription/Service Reference #
    gpi: Optional[str] = None                 # derived taxonomy, populated on repair


# Reject codes that are genuine adjudication outcomes, NOT transport failures.
# Passing these through unchanged is mandatory: retrying them is a bug.
TERMINAL_REJECTS = {"07", "70", "75", "76", "79", "88"}
# Reject signalling a bad/unknown Product/Service ID -> eligible for crosswalk repair.
REPAIRABLE_NDC_REJECT = "21"


class GPICrosswalkCache:
    """Version-stamped NDC->GPI map, refreshed out-of-band by the crosswalk pipeline."""

    def __init__(self, snapshot: dict[str, str], version: str):
        self._map = snapshot
        self.version = version  # audit: which taxonomy snapshot resolved this claim

    def resolve(self, ndc: str) -> Optional[str]:
        return self._map.get(ndc)  # keyed by 407-D7 NDC


class FallbackRouter:
    def __init__(self, primary_url: str, fallback_url: str,
                 breaker: CircuitBreaker, crosswalk: GPICrosswalkCache):
        self.primary_url = primary_url
        self.fallback_url = fallback_url
        self.breaker = breaker
        self.crosswalk = crosswalk
        self.client = httpx.AsyncClient(timeout=httpx.Timeout(8.0, connect=2.0))

    async def aclose(self) -> None:
        await self.client.aclose()

    def _validate(self, raw: dict[str, Any]) -> Optional[CanonicalClaim]:
        try:
            return CanonicalClaim(**raw)
        except ValidationError as e:
            # Log the error shape only — never the offending values (may carry PHI).
            log.error("schema_validation_failed", error_count=len(e.errors()))
            return None

    async def _post(self, url: str, claim: CanonicalClaim) -> dict[str, Any]:
        resp = await self.client.post(
            url,
            json={  # explicit projection: only fields the vendor contract needs
                "correlationId": claim.correlation_id,
                "cardholderToken": claim.cardholder_token,   # 302-C2, tokenized
                "groupId": claim.group_id,                   # 301-C1
                "productServiceId": claim.ndc,               # 407-D7
                "serviceProviderId": claim.pharmacy_npi,     # 201-B1
                "quantityDispensed": str(claim.qty_dispensed),  # Decimal -> string, no float
                "daysSupply": claim.days_supply,             # 405-D5
                "rxRef": claim.rx_ref,                       # 402-D2
                "gpi": claim.gpi,
            },
            headers={"X-NCPDP-Version": "D.0", "X-Idempotency-Key": claim.rx_ref},
        )
        resp.raise_for_status()
        return resp.json()

    def _classify(self, body: dict[str, Any]) -> str:
        """Map a vendor response to a routing class using the NCPDP reject code."""
        reject = str(body.get("rejectCode", ""))
        if reject in TERMINAL_REJECTS:
            return "terminal"
        if reject == REPAIRABLE_NDC_REJECT:
            return "repairable"
        return "ok"

    async def adjudicate(self, raw: dict[str, Any]) -> dict[str, Any]:
        claim = self._validate(raw)
        if claim is None:
            return {"status": "rejected", "reason": "invalid_schema"}

        # Bind correlation id for the whole request; keeps PHI out of every log line.
        structlog.contextvars.bind_contextvars(cid=claim.correlation_id)

        if not self.breaker.can_execute():
            log.info("primary_circuit_open", action="failover")
            return await self._failover(claim)

        try:
            body = await self._post(self.primary_url, claim)
            self.breaker.record_success()
            klass = self._classify(body)
            if klass == "terminal":
                # Genuine coverage decision (reject 70/75/76/79...) — pass through.
                return {"status": "rejected", "path": "primary",
                        "reject_code": body.get("rejectCode")}
            if klass == "repairable":
                return await self._repair_and_reroute(claim, path="primary")
            return {"status": "adjudicated", "path": "primary", "result": body}
        except (httpx.HTTPStatusError, httpx.ConnectError, httpx.TimeoutException) as e:
            self.breaker.record_failure()
            log.warning("primary_transport_failed", error_type=type(e).__name__, ndc=claim.ndc)
            return await self._failover(claim)

    async def _repair_and_reroute(self, claim: CanonicalClaim, path: str) -> dict[str, Any]:
        gpi = self.crosswalk.resolve(claim.ndc)  # 407-D7 -> GPI
        if gpi is None:
            log.info("ndc_unresolved", ndc=claim.ndc, snapshot=self.crosswalk.version)
            return {"status": "rejected", "reason": "ndc_not_in_taxonomy", "ndc": claim.ndc}
        enriched = claim.model_copy(update={"gpi": gpi})
        log.info("ndc_repaired", ndc=claim.ndc, snapshot=self.crosswalk.version)
        return await self._failover(enriched)

    async def _failover(self, claim: CanonicalClaim) -> dict[str, Any]:
        log.info("fallback_activated", ndc=claim.ndc, npi=claim.pharmacy_npi)
        try:
            body = await self._post(self.fallback_url, claim)
            return {"status": "adjudicated", "path": "fallback", "result": body}
        except Exception as e:
            log.error("all_paths_exhausted", error_type=type(e).__name__, ndc=claim.ndc)
            return {"status": "rejected", "reason": "all_paths_exhausted", "ndc": claim.ndc}


async def run_batch() -> None:
    router = FallbackRouter(
        primary_url="https://api.pbm-primary.example/v1/adjudicate",
        fallback_url="https://api.pbm-fallback.example/v1/adjudicate",
        breaker=CircuitBreaker(failure_threshold=3, recovery_timeout=15.0),
        crosswalk=GPICrosswalkCache({"00001234567": "27100010100310"}, version="2026-07-01"),
    )
    claims = [
        {"correlation_id": "cid-9901", "cardholder_token": "tok_a1", "group_id": "GRP7",
         "ndc": "00001234567", "pharmacy_npi": "1234567890",
         "qty_dispensed": Decimal("30"), "days_supply": 30, "rx_ref": "RX-9901"},
    ]
    try:
        results = await asyncio.gather(*(router.adjudicate(c) for c in claims))
    finally:
        await router.aclose()
    for r in results:
        log.info("batch_result", **r)


if __name__ == "__main__":
    asyncio.run(run_batch())

Execution Flow #

The router is a small state machine: validate, check the breaker, attempt the primary, classify the response, and either return, repair-and-reroute, or fail over. The decision tree below traces a single claim from ingestion through the circuit-breaker check and crosswalk repair to the secondary node or a terminal rejection.

Figure: Fallback routing decision tree from primary adjudication through circuit-breaker check and crosswalk resolution to the secondary node or terminal rejection.

Engineering Constraints & Known Failure Modes #

Failover introduces failure modes that a single-path adjudicator never sees. Each of these has bitten production PBM systems:

• Terminal-reject amplification. Treating reject 70/75/76/79 as retryable doubles load and latency on both nodes without changing the outcome. The TERMINAL_REJECTS set is the guard; keep it exhaustive.
• Duplicate adjudication on failover. A primary that times out after it adjudicated but before the response returns will be re-sent to the secondary — a double-pay risk. The 402-D2 Prescription/Service Reference # doubles as the idempotency key (X-Idempotency-Key) so the secondary node deduplicates.
• Stale crosswalk repair. Repairing an NDC against an out-of-date snapshot can assign the wrong therapeutic class and therefore the wrong tier. Every repair logs crosswalk.version; the resolved GPI must reference the snapshot active at the dispensing timestamp, exactly as required for formulary tier mapping and copay calculation.
• Accumulator divergence across nodes. If the secondary node holds a stale copay accumulator, a deductible-phase claim adjudicates at the wrong member cost. Failover must be gated on eligibility/accumulator parity from the portal sync layer, not just node health.
• PA state races. A claim that hit reject 75 on the primary must route into the step therapy and prior-authorization trigger rules flow, not fail over — failing over a PA-required claim can produce a paid claim that bypasses the PA gate.
• Thundering herd on recovery. When a downed vendor returns, an open breaker that flips straight to CLOSED for all workers stampedes the recovering endpoint. The HALF_OPEN single-probe state is what prevents that; tune the probe count in implementing circuit breakers for PBM API timeouts.
• PHI leakage in error paths. The tempting except that logs the offending payload is the classic HIPAA exposure. Validation and transport handlers here log the error shape (type, count) and the drug identifier only — never 302-C2, 310-CA, or the raw body.

Performance & Correctness Tuning #

• Bounded, split timeouts. Use a short connect timeout (≈2 s) and a longer read timeout so a slow-to-connect vendor trips failover fast while a legitimately busy one is given room. The whole path must still fit the sub-2-second POS budget inherited from the platform architecture.
• Idempotency keys everywhere. 402-D2 as the idempotency key makes retries and failovers safe to repeat; without it, exactly-once becomes at-least-once and money moves twice.
• Decimal money, always. 442-E7 Quantity Dispensed and every copay/deductible figure use decimal.Decimal and cross the wire as strings. A float quantity silently breaks day-supply math and copay rounding.
• Warm the crosswalk cache. Cold-cache repair on the hot path adds a synchronous lookup to a claim that is already in a failure state. Keep the snapshot resident and version-stamped; treat a cache miss as a taxonomy gap, not a routing error.
• Backoff with jitter. Retries before failover use exponential backoff with jitter so recovering endpoints are not stampeded; pair this with the breaker’s HALF_OPEN probe.
• Per-segment breakers. Retail, mail-order, and specialty pharmacy networks fail independently — one breaker per segment prevents a specialty outage from opening the retail path.
• Async batch, structured replay. Route batches concurrently under asyncio.gather, and serialize every routing decision (attempt, reject class, snapshot version, chosen path) to the append-only store so any claim can be re-adjudicated identically during a payer audit. This is the same concurrency model used for asynchronous batch adjudication workflows.

In This Section #

• Implementing circuit breakers for PBM API timeouts — threshold calibration, recovery-window tuning, and per-segment bulkhead isolation for the breaker that guards each vendor endpoint.

Related #

• PBM Architecture & Taxonomy Foundations — the canonical claim model, taxonomy, and PHI boundary this router builds on.
• NDC-to-GPI Crosswalk Automation — the version-stamped map that powers identifier repair during failover.
• PBM Portal Sync Architecture — how eligibility, accumulators, and PA state reach the secondary node.
• Security & Compliance Boundaries for Claims Data — the audit and PHI-handling obligations every routing decision must satisfy.
• Handling PBM 404 and 503 errors in adjudication scripts — transport-error handling that feeds the transient-failure lane of the trigger matrix.

← Back to PBM Architecture & Taxonomy Foundations