Automating PBM portal credential rotation for adjudication

The single implementation decision this page settles is when a claims worker refreshes an expiring portal credential — synchronously, on the request path, under a lock, versus asynchronously in a background timer. The choice is not cosmetic: it decides whether an OAuth client_credentials token or mTLS session can lapse mid-batch and inject a transport-layer authentication failure into a stream of otherwise payable claims. A background refresher that fires on a wall-clock timer races the worker threads submitting claims — worker N reads the old token microseconds before the timer swaps it, and the payer gateway answers 401, which this portal surfaces as REJ_001 (Authentication Failure) or REJ_004 (Session Timeout). Because these are transport failures, not claim-content rejects, the correct recovery is header reconstruction and request replay, never a fabricated NCPDP claim reject that would poison member eligibility or accumulator state. This page sits under the PBM Portal Sync Architecture control plane and settles the rotation timing so the claims engine never pauses and never double-submits an already-adjudicated claim.

Decision matrix: rotation strategies and their failure surface

Before writing the handler, fix the contract. Each rotation strategy trades race safety against latency and memory, and each has a distinct blast radius when it goes wrong mid-batch. The synchronous double-checked variant (bolded) is the one this page implements; the others exist because they are the tempting shortcuts that leak REJ_001 into production.

Strategy Race-safe under concurrency? Per-request latency added Mid-batch drop risk Memory footprint Failure mode
Per-request fetch (no cache) Yes (trivially) Full token RTT every call (~40–120 ms) None Flat Vault endpoint becomes the throughput ceiling; burns rate quota
Background timer refresh No — worker reads old token during swap ~0 ms High — REJ_001 on the racing request Holds one live token Timer/worker race; hardest to reproduce in tests
Reactive 401-only (no proactive expiry) Partial ~0 ms until expiry, then one failed round-trip Medium — first request after expiry always fails once Flat Every rotation costs one rejected-then-replayed claim
Synchronous double-checked cache + 60 s buffer Yes — one lock, one refresher ~0 ms on the hot path; RTT only on the rare refresh None Sub-50 MB per worker Only a genuine vault outage stalls, and that trips backoff

Three field-level distinctions drive the branch:

  • A rotated credential must land as exact header alignment. Downstream adjudication services expect Authorization: Bearer <token>, X-PBM-Client-ID, and X-Adjudication-Session-TTL precisely; a misaligned field is indistinguishable from an expired token and surfaces as REJ_001/REJ_004. Reconstruct the header, do not resubmit the claim.
  • Auth failures never become NCPDP claim rejects. A 401 is credential drift, handled by replay exactly as Handling PBM 404 and 503 Errors in Adjudication Scripts treats a 401 — refresh once, replay, and keep the claim content untouched so 407-D7 (NDC) and 302-C2 (Cardholder ID) are never re-parsed or corrupted.
  • The refresh must fire before expiry, not on it. A 60-second buffer forces rotation while the current token is still valid, so no request is ever built with a token that expires between header assembly and the gateway receiving it.

Step-by-step implementation

The handler below refreshes on demand under a single lock, injects exact field mappings, and forces immediate re-rotation on a 401 so a rotated-but-drifted credential replays rather than rejects. It shares the back-pressure discipline of PBM API Sync & Rate Limiting; a genuine vault outage degrades through the same circuit breaker as Implementing Circuit Breakers for PBM API Timeouts.

  1. Double-check the token under the lock. Only one worker may refresh; the rest reuse the cached token. This is the fix for the background-timer race in the decision matrix.
  2. Buffer the expiry by 60 seconds so rotation completes while the current token is still accepted by the gateway.
  3. Inject exact headers — Authorization, X-PBM-Client-ID, X-Adjudication-Session-TTL — and discard them the instant the request returns, keeping the worker footprint flat.
  4. On 401, invalidate and re-rotate, then let the caller replay the same claim payload — never re-adjudicate, or accumulators double-count.
Synchronous on-demand credential rotation sequence A worker acquires a lock and double-checks the cached token under it. If the token is missing or expired the worker fetches a fresh one from the PBM switch via client_credentials and stores it with expiry minus a 60-second buffer; otherwise it reuses the cached token. The lock is released before the claim is submitted with a Bearer header, so the hot path is lock-free. If the switch answers 401 the token is invalidated and rotation is forced so the same claim replays; on success the adjudication result returns. Worker Lock Token store PBM switch acquire lock granted double-check token valid ALT token missing or expired fetch token via client_credentials access_token + expires_in store token · expiry − 60 s buffer [ token still valid ] reuse cached token release lock submit claim with Bearer header ALT status 401 REJ_001 auth failure invalidate token · force rotation [ success ] adjudication result

Figure: Synchronous on-demand credential rotation with double-checked locking, token reuse versus fetch, and 401 forced re-rotation during claim submission.

python
import time
import logging
import threading
import requests
from contextlib import contextmanager
from typing import Generator, Dict, Any, Optional
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

# PHI-safe logger: token and client_id are redacted; raw claim bytes never logged.
logger = logging.getLogger(__name__)
logging.basicConfig(level=logging.INFO, format="%(asctime)s [%(levelname)s] %(message)s")

class AdjudicationCredentialManager:
    def __init__(self, vault_endpoint: str, adjudication_api: str, client_id: str, client_secret: str):
        self.vault_endpoint = vault_endpoint
        self.adjudication_api = adjudication_api
        self.client_id = client_id
        self.client_secret = client_secret
        self._token: Optional[str] = None
        self._expires_at: float = 0.0
        # Serializes token refresh so concurrent workers cannot race on a stale token
        self._lock = threading.Lock()
        self._session = requests.Session()
        retry_strategy = Retry(
            total=3,
            backoff_factor=0.5,
            status_forcelist=[502, 503, 504],
            # POST is not retried by urllib3's default allowed_methods; opt it in
            # explicitly so token fetch and claim submission honor the retry policy.
            allowed_methods=frozenset({"GET", "POST"}),
        )
        self._session.mount("https://", HTTPAdapter(max_retries=retry_strategy))

    def _fetch_token(self) -> Dict[str, Any]:
        payload = {
            "grant_type": "client_credentials",
            "client_id": self.client_id,
            "client_secret": self.client_secret,
            "scope": "pbm.adjudication.submit pbm.eligibility.read"
        }
        response = self._session.post(self.vault_endpoint, json=payload, timeout=5)
        response.raise_for_status()
        data = response.json()
        # Buffer: force refresh 60 seconds before expiry to prevent mid-batch drops
        self._expires_at = time.time() + data.get("expires_in", 3600) - 60
        self._token = data["access_token"]
        return data

    def _is_expired(self) -> bool:
        return time.time() >= self._expires_at

    @contextmanager
    def secure_session(self) -> Generator[Dict[str, str], None, None]:
        with self._lock:
            # Double-checked under the lock so only one worker refreshes the token
            if self._token is None or self._is_expired():
                logger.info("Triggering credential rotation for adjudication pipeline")
                self._fetch_token()

        # Exact header alignment — a drifted field is indistinguishable from expiry (REJ_001)
        headers = {
            "Authorization": f"Bearer {self._token}",
            "X-PBM-Client-ID": self.client_id,
            "X-Adjudication-Session-TTL": str(int(self._expires_at - time.time())),
            "Content-Type": "application/json"
        }
        yield headers
        # Ephemeral context exit: headers are discarded immediately post-request

    def submit_claim(self, claim_payload: Dict[str, Any]) -> Dict[str, Any]:
        # PHI constraint: claim_payload carries 302-C2 Cardholder ID / 310-CA Patient First
        # Name and 407-D7 NDC — it must never be logged, cached, or echoed into an error.
        with self.secure_session() as headers:
            resp = self._session.post(
                f"{self.adjudication_api}/v1/claims",
                headers=headers,
                json=claim_payload,
                timeout=10
            )
            if resp.status_code == 401:
                # Transport failure, NOT a claim reject: invalidate and force re-rotation.
                # The caller replays the SAME payload — never re-adjudicates (accumulator safety).
                self._expires_at = 0.0
                raise PermissionError("REJ_001: transport auth failure; token invalidated, replay claim.")
            resp.raise_for_status()
            return resp.json()

Verification and testing pattern

Correctness here is behavioral: the test suite must prove that under concurrency the token is fetched exactly once, and that a 401 invalidates the cache so the next call re-rotates. Assert on the fetch call count against a mocked vault so a regression that reintroduces the background-timer race — the hardest bug to reproduce in the field — fails the build instead.

python
import threading
from unittest.mock import patch
from adjudication import AdjudicationCredentialManager  # module under test


def _mgr() -> AdjudicationCredentialManager:
    return AdjudicationCredentialManager(
        vault_endpoint="https://vault.example/token",
        adjudication_api="https://api.pbm-gateway.com",
        client_id="PBMCLIENT",       # non-PHI routing identity
        client_secret="s3cret",
    )


def test_double_checked_lock_fetches_token_once_under_concurrency():
    mgr = _mgr()
    calls = {"n": 0}

    def fake_fetch():
        calls["n"] += 1
        mgr._token = "tok-abc"
        mgr._expires_at = 9_999_999_999.0   # far future; stays valid for all threads

    with patch.object(mgr, "_fetch_token", side_effect=fake_fetch):
        def worker():
            with mgr.secure_session() as h:
                assert h["Authorization"] == "Bearer tok-abc"   # exact header alignment
        threads = [threading.Thread(target=worker) for _ in range(32)]
        for t in threads: t.start()
        for t in threads: t.join()

    assert calls["n"] == 1        # ONE refresh across 32 workers — no timer race


def test_401_invalidates_token_and_forces_rotation():
    mgr = _mgr()
    mgr._token, mgr._expires_at = "stale", 9_999_999_999.0
    assert not mgr._is_expired()         # cache looks valid...
    mgr._expires_at = 0.0                # ...until a 401 forces invalidation
    assert mgr._is_expired()             # next secure_session() re-rotates


def test_expiry_buffer_rotates_before_true_expiry():
    mgr = _mgr()
    with patch.object(mgr._session, "post") as post:
        post.return_value.json.return_value = {"access_token": "t", "expires_in": 3600}
        post.return_value.raise_for_status.return_value = None
        mgr._fetch_token()
    # Rotation is due 60s early, never on the true 3600s boundary.
    assert mgr._expires_at <= time.time() + 3600 - 60

Gotchas and PHI guardrails

  • Never log the token, the secret, or the claim payload. access_token and client_secret are credential material; the claim_payload carries 302-C2 Cardholder ID and 310-CA Patient First Name. Use structured logging with those fields redacted, exactly as Security & Compliance Boundaries for Claims Data requires. Log the event (“credential rotation triggered”), never the material.
  • A 401 must never mint an NCPDP claim reject. It is transport drift — invalidate, re-rotate, replay the identical payload. Writing a 70/65 claim reject on an auth failure corrupts the reject ledger and, worse, can trip the fault counter feeding Fallback Routing Logic Design into a self-inflicted degraded mode.
  • Replaying is not re-adjudicating. After re-rotation, resubmit the same claim object. If the first attempt actually adjudicated before the token lapsed on a later field, a naive “resubmit from source” doubles deductible and rebate accumulators. Guard the replay with the same idempotency key the sync layer derives from claim_id + 407-D7 NDC + service date.
  • The 60-second buffer must exceed worst-case request assembly time. On a slow worker, a token fetched with a 5-second buffer can still expire between header assembly and gateway receipt. Keep the buffer well above p99 request latency; 60 s is generous headroom, not waste.
  • Retry-After on a vault 503 is not seconds-only. If the vault throttles token fetches, RFC 7231 permits an HTTP-date form. A bare float() on the header raises; parse defensively and fall back to jittered backoff rather than crashing the worker that holds the last valid token.
  • Pin the mTLS trust and IP allowlist. The vault endpoint should enforce mutual TLS and restrict IP allowlists to the adjudication worker subnets, so a leaked client_secret alone cannot mint tokens off-network. Validate scope alignment (pbm.adjudication.submit) at fetch time — a scope drift silently produces tokens the gateway rejects as REJ_001.

Monitor REJ_001 and REJ_004 frequency in production dashboards; a spike is the leading indicator of vault latency or scope misalignment. Alert at a 0.5% transport-reject rate to trip a circuit breaker before SLA degradation, and correlate token-TTL metrics with worker utilization using OpenTelemetry spans. Validate payloads against the official NCPDP Telecommunication standard before transmission and keep HIPAA-compliant audit trails of rotation timestamps and adjudication success rates.

← Up to PBM Portal Sync Architecture