Schema Validation & Error Categorization

Once a claim has been decoded and structurally extracted, it still cannot be trusted. A syntactically parseable payload can carry an all-zero product identifier, a days-supply value that overflows the plan maximum, or a prescriber field that contradicts the member segment — and if any of those reach the pricing engine, the result is a mispriced claim, a payer reconciliation break, or a rejected 835 remittance. This page sits within the broader Claims Ingestion & NCPDP Parsing domain and owns one specific sub-problem: turning the extracted payload emitted by NCPDP D.0 Message Parsing Strategies into a validated object whose every failure is classified by severity, mapped to an NCPDP 511-FB reject code, and routed to a deterministic remediation path. Validation here is not a boolean pass/fail gate — it is a classification engine that decides whether a claim is permanently rejected, retried, or adjudicated with a flag.

Prerequisites #

Before this workflow runs, the validation service assumes the following inputs and dependencies are in place:

• A parsed, field-keyed payload. This tier consumes the AdjudicationPayload produced upstream, not a raw byte stream. Fields are already keyed by their NCPDP data-element reference (407-D7 Product/Service ID, 302-C2 Cardholder ID, 405-D5 Days Supply, 409-D9 Ingredient Cost Submitted) so validators address fields by name rather than wire offset.
• Resolved reference data. Identifier validation cross-references a versioned NDC reference file and the NDC-to-GPI map supplied by the NDC to GPI Crosswalk Automation pipeline. The map must be a snapshot with a version identifier, not a live per-claim query, so that an audit can prove which reference version validated a given claim.
• A formulary snapshot. Tier and coverage checks read a versioned formulary snapshot rather than mutable production tables — the same versioned-snapshot contract enforced by Tier Mapping & Copay Calculation Logic. Validating against an unversioned table makes a reject non-reproducible during a payer dispute.
• Library versions. The reference implementation targets Python 3.11+ (for match statements and Self), pydantic>=2.5 for declarative schema enforcement and native Decimal coercion, and opentelemetry-api>=1.24 for span and counter emission. Pydantic v2 semantics — model_validator(mode="after"), PydanticCustomError, field_validator — are assumed throughout.
• PHI boundary. The service runs inside the claims-data trust zone defined by Security & Compliance Boundaries for Claims Data. Raw claim bytes are never written to logs, and cardholder (302-C2) and patient-name (310-CA) fields are stripped from the working object immediately after routing metadata is derived. Every log line keys on the claim reference and a payload hash, never on member identity.

Validation Rule and Reject-Code Specification #

Validation splits into two evaluation stages that must run in order. Syntactic rules test a single field against a format or range constraint and can fail before any reference data is consulted. Semantic rules test cross-field business logic and require the resolved crosswalk and formulary snapshot. A payload only reaches semantic evaluation if it survives syntactic evaluation, because a malformed 407-D7 cannot be meaningfully crosswalked.

Every failure resolves to exactly one NCPDP 511-FB reject code and one severity. The severity — not the individual rule — decides the routing path, which keeps the routing logic stable as new rules are added.

*A negative or non-numeric 409-D9 is treated as a submitted-financials fault; some processors return a DR/M/I variant here. †88 (DUR Reject) is one convention; transient infrastructure failures are more often carried out-of-band and never returned to the switch as a permanent reject.

The three severities map to three and only three destinations:

• FATAL — permanent rejection. The claim is returned to the pharmacy switch with its 511-FB code and routed to a dead-letter queue for manual review or pharmacy resubmission. No automatic retry; the input itself is wrong.
• TRANSIENT — an auto-retry-eligible failure caused by an external dependency (crosswalk cache miss that fell through to a timed-out reference service, formulary snapshot not yet warm, eligibility service unreachable). The claim is well-formed; the environment failed. Route to a bounded retry queue governed by PBM API Sync & Rate Limiting.
• WARN — adjudication proceeds, but the claim carries flagged metadata (an 526-FQ Additional Message and an audit annotation) for downstream reconciliation or copay adjustment. A deprecated-but-still-payable NDC or a missing secondary-coverage segment belongs here.

When a single payload accumulates multiple failures, the most severe category wins the routing decision, so a claim with one FATAL and two WARN findings is dead-lettered rather than adjudicated.

Reference Python Implementation #

The implementation below uses Pydantic v2 for declarative field constraints, custom validators for identifier and cross-field rules, decimal.Decimal for every financial field, and OpenTelemetry counters for the error-distribution telemetry that operations depends on. It emits structured results that Asynchronous Batch Adjudication Workflows can route to a message broker or dead-letter queue. Note the PHI discipline: no validator or log line ever emits 302-C2 or 310-CA, and those fields are cleared from the object the moment routing metadata is derived.

import hashlib
import logging
from decimal import Decimal
from datetime import date
from enum import Enum
from typing import Optional, Self

from pydantic import BaseModel, Field, field_validator, model_validator
from pydantic_core import PydanticCustomError

# Structured logging only. NEVER log raw claim bytes or PHI fields
# (302-C2 Cardholder ID, 310-CA Patient First Name). Keys are the Rx
# reference and a payload hash, both non-identifying.
logging.basicConfig(level=logging.INFO, format="%(asctime)s | %(levelname)s | %(message)s")
logger = logging.getLogger("schema_validation")


class Severity(str, Enum):
    FATAL = "FATAL"        # permanent reject -> dead-letter queue
    TRANSIENT = "TRANSIENT"  # dependency failure -> retry queue
    WARN = "WARN"          # adjudicate with flag
    PASS = "PASS"          # clean -> adjudication pipeline


# Severity ordering: the most severe finding on a payload wins routing.
_RANK = {Severity.FATAL: 3, Severity.TRANSIENT: 2, Severity.WARN: 1, Severity.PASS: 0}


class RejectFinding(BaseModel):
    field_ref: str          # NCPDP data-element reference, e.g. "407-D7"
    reject_code: str        # value returned in 511-FB Reject Code
    severity: Severity
    message: str            # operator-facing; never contains PHI


class ClaimPayload(BaseModel):
    rx_ref: str = Field(..., min_length=1, max_length=12)   # 402-D2 Prescription/Service Ref
    bin_iin: str = Field(..., pattern=r"^\d{6}$")           # 101-A1 BIN/IIN
    pcn: str = Field(..., min_length=1)                     # 104-A4 Processor Control Number
    cardholder_id: str = Field(..., min_length=1)           # 302-C2 Cardholder ID (PHI)
    patient_first_name: Optional[str] = None               # 310-CA Patient First Name (PHI)
    ndc_11: str = Field(..., pattern=r"^\d{11}$")           # 407-D7 Product/Service ID
    product_qualifier: str = Field(...)                    # 436-E1 Product/Service ID Qualifier
    service_date: date                                     # 401-D1 Date of Service
    days_supply: int = Field(..., ge=1, le=999)            # 405-D5 Days Supply
    prescriber_npi: Optional[str] = Field(None, pattern=r"^\d{10}$")  # 411-DB Prescriber ID
    ingredient_cost: Decimal = Field(..., ge=Decimal("0"))  # 409-D9 Ingredient Cost Submitted

    # Populated by validation; drives routing.
    findings: list[RejectFinding] = Field(default_factory=list)
    severity: Severity = Severity.PASS
    ref_version: Optional[str] = None   # crosswalk/formulary snapshot id (audit)

    @field_validator("ndc_11")
    @classmethod
    def reject_placeholder_ndc(cls, v: str) -> str:
        # 407-D7: the 11-digit pattern is enforced by Field; reject the all-zero
        # placeholder some senders emit for an unknown NDC -> 511-FB "21".
        if v == "00000000000":
            raise PydanticCustomError("ndc_placeholder", "NDC-11 is the all-zero placeholder.")
        return v

    @field_validator("product_qualifier")
    @classmethod
    def require_ndc_namespace(cls, v: str) -> str:
        # 436-E1: only qualifier 03 (NDC) is crosswalkable here. 07 (HRI) /
        # 28 (First DataBank) belong to another namespace -> 511-FB "21".
        if v != "03":
            raise PydanticCustomError("qualifier", "436-E1 qualifier is not 03 (NDC).")
        return v

    @field_validator("service_date")
    @classmethod
    def no_future_service_date(cls, v: date) -> date:
        # 401-D1: a service date in the future is invalid -> 511-FB "15".
        if v > date.today():
            raise PydanticCustomError("dos_future", "401-D1 Date of Service is in the future.")
        return v

    @model_validator(mode="after")
    def evaluate_semantics(self) -> Self:
        # Semantic stage runs only after field-level (syntactic) rules pass.
        # In production, replace the prefix heuristics with a lookup against the
        # versioned NDC-to-GPI + formulary snapshot; record its version for audit.
        gpi = _resolve_gpi(self.ndc_11)            # 407-D7 -> GPI via versioned snapshot
        self.ref_version = _SNAPSHOT_VERSION

        if gpi is None:
            self._flag("407-D7", "70", Severity.FATAL, "NDC resolves to no active GPI.")
        elif not _tier_is_covered(gpi):
            self._flag("407-D7", "70", Severity.FATAL, "GPI maps to a non-covered tier.")

        if self.days_supply > _plan_max_days(self.pcn):   # 405-D5 vs plan max
            self._flag("405-D5", "76", Severity.FATAL, "Days supply exceeds plan maximum.")

        if gpi is not None and _requires_prior_auth(gpi):
            self._flag("407-D7", "75", Severity.FATAL, "Prior authorization required.")

        if gpi is not None and _ndc_superseded(self.ndc_11):
            self._flag("407-D7", "", Severity.WARN, "NDC superseded; annotate 526-FQ.")

        self.severity = max(
            (f.severity for f in self.findings),
            key=lambda s: _RANK[s],
            default=Severity.PASS,
        )
        return self

    def _flag(self, field_ref: str, reject: str, sev: Severity, msg: str) -> None:
        self.findings.append(
            RejectFinding(field_ref=field_ref, reject_code=reject, severity=sev, message=msg)
        )

    def payload_hash(self) -> str:
        # Non-reversible fingerprint for logs/idempotency. Hash the routing-
        # relevant fields, not the PHI. Doubles as the dedup key.
        material = f"{self.bin_iin}|{self.pcn}|{self.rx_ref}|{self.service_date.isoformat()}"
        return hashlib.sha256(material.encode()).hexdigest()

    def route(self) -> str:
        digest = self.payload_hash()
        # Derive routing metadata, THEN strip PHI before anything is logged/queued.
        self.cardholder_id = ""      # 302-C2 cleared post-routing
        self.patient_first_name = None  # 310-CA cleared post-routing
        VALIDATION_COUNTER.add(1, {"severity": self.severity.value})  # error-distribution telemetry
        match self.severity:
            case Severity.FATAL:
                logger.warning("claim=%s hash=%s -> dead_letter_queue codes=%s",
                               self.rx_ref, digest, [f.reject_code for f in self.findings])
                return "dead_letter_queue"
            case Severity.TRANSIENT:
                logger.info("claim=%s hash=%s -> retry_queue", self.rx_ref, digest)
                return "retry_queue"
            case _:
                logger.info("claim=%s hash=%s -> adjudication_pipeline sev=%s",
                            self.rx_ref, digest, self.severity.value)
                return "adjudication_pipeline"

The financial field (409-D9) is typed Decimal, never float: passing a binary float into downstream copay math produces cent-level drift that fails payer reconciliation, so the type contract is enforced at the validation boundary and shared with every tier that touches money. The route method is the single place PHI leaves scope — routing metadata is derived, then 302-C2 and 310-CA are cleared before any log line or queue publish.

Figure: Two-stage validation — syntactic field rules, then semantic cross-reference — collapsing multiple findings into a single severity that selects the routing path.

Engineering Constraints and Known Failure Modes #

Production traffic breaks every convenient assumption a validator makes about clean input. The failure modes below are the ones that actually corrupt adjudication:

• NDC gaps versus bad claims. A well-formed 407-D7 that resolves to no GPI is usually reference-data lag, not a bad claim. Do not silently drop it: emit reject 70 and let the recovery path belong to NDC to GPI Crosswalk Automation. The distinction between “unknown NDC” and “known-non-covered NDC” changes whether the pharmacy resubmits or the reference file is patched.
• Qualifier drift misrouting the namespace. If 436-E1 is 07 (HRI) or 28 (First DataBank) rather than 03 (NDC), crosswalking the value as an NDC misprices the claim. Hard-fail with reject 21 instead of trusting the digits.
• Severity collision on multi-fault payloads. A payload can trip several rules at once. Categorizing each finding independently and then taking the maximum severity is what prevents a claim with one FATAL fault from being adjudicated because two WARN findings “outvoted” it. Never route on the first finding encountered.
• Transient dressed as fatal. A crosswalk-service timeout must be classified TRANSIENT, not FATAL — dead-lettering an infrastructure blip forces manual replay of a claim that would have passed on retry. The distinguishing test is whether the input is wrong (FATAL) or the environment failed (TRANSIENT).
• Plan-override conflicts. A 405-D5 days-supply value that violates the generic plan maximum but is explicitly allowed by a member-specific override must consult the override before rejecting 76. Validating against the base plan table alone produces false 76 rejects on 90-day maintenance fills.
• PHI leaking through error messages. The most common compliance defect in a validation layer is echoing the offending value into the reject message (“Invalid cardholder 302-C2=…”). Findings carry the field reference and a generic message, never the value. This is the boundary enforced with Security & Compliance Boundaries for Claims Data.

Figure: Severity, not the individual rule, selects the destination — the same three categories that keep the routing logic stable as new rules are added. The asterisk marks that transient failures are usually carried out-of-band rather than returned as a permanent reject.

Performance and Correctness Tuning #

• Idempotency keys. Switch-level retries replay identical payloads. Derive the dedup key from hash(101-A1 + 104-A4 + 402-D2 + 401-D1) — computed in payload_hash() — and deduplicate before adjudication so a retried claim is validated and priced at most once. The same hash is the log correlation key, keeping PHI out of telemetry.
• Decimal precision at the boundary. Coerce 409-D9 (and any submitted financial) to decimal.Decimal inside the model, not downstream. Enforcing the type at validation guarantees the pricing tier never receives a float, protecting the arithmetic contract shared with Tier Mapping & Copay Calculation Logic.
• Versioned snapshot lookups. The crosswalk and formulary reads that drive semantic rules must hit an in-process, atomically-swapped snapshot with a recorded ref_version, not a per-claim database round trip. Stamping the version onto each payload lets an audit reproduce exactly which reference data produced a 70 or 75 reject months later.
• Order rules cheapest-first. Field-level syntactic rules are pure and allocation-free; semantic rules touch reference data. Running syntactic validation first short-circuits the majority of malformed claims before any lookup, keeping the P99 evaluation budget inside the sub-second point-of-sale SLA.
• Telemetry as a feedback loop. Emit an error-distribution counter keyed on severity and reject code (the VALIDATION_COUNTER above), plus a P95/P99 evaluation-latency span. A spike in 70 rejects for a specific 101-A1 BIN is an early signal that a formulary or crosswalk snapshot needs refreshing — the same threshold-tuning discipline described in Rule Engine Threshold Tuning & Optimization.

In This Section #

• Automated schema validation for 837P vs 837I claims — how conditional schema branches enforce professional versus institutional billing rules across 837P and 837I loops without duplicating validation logic across ingestion endpoints.

Related #

• NCPDP D.0 Message Parsing Strategies — the parser that produces the field-keyed payload this validation tier consumes.
• Asynchronous Batch Adjudication Workflows — the queue that routes validated PASS/WARN, TRANSIENT, and FATAL results onward.
• PBM API Sync & Rate Limiting — backpressure and bounded retry policy for TRANSIENT-classified claims.
• NDC to GPI Crosswalk Automation — the versioned lookup that decides whether 407-D7 resolves to an active GPI.
• Security & Compliance Boundaries for Claims Data — the PHI rules governing 302-C2 / 310-CA handling and reject-message content.

← Back to Claims Ingestion & NCPDP Parsing