Tuning Adjudication Thresholds Without Overfitting

A threshold tuned to fit last quarter’s claims perfectly is the fastest way to build a rule engine that fails next quarter at the pharmacy counter. When you calibrate a quantity-limit multiplier, a drug utilization review (DUR) sensitivity, or a fraud-waste-and-abuse flag against a fixed slice of historical adjudication outcomes, the tempting move is to keep tightening until the training set’s precision looks flawless — at which point the boundary has memorized noise, and the first genuine shift in prescribing turns it into a false-positive generator that rejects legitimate fills. This page settles one calibration decision: how to pick a threshold value from historical claims so it generalizes, using a train/validation split, precision and recall measured on reject rates, and explicit guardrails against the overfitting and drift that make a threshold look great in backtest and terrible in production. It is the statistical discipline underneath Rule Engine Threshold Tuning & Optimization, which treats every numeric boundary as versioned, testable data rather than a constant — this page is about how you choose the number in the first place.

The Tuning Strategy Decision

Every tuning strategy trades bias against variance, and the overfitting guard is what separates a threshold that holds from one that memorizes. The wrong default — grid-searching for the value that maximizes F1 on the entire historical set — has no out-of-sample check at all, so it reports a score it can never reproduce live. The strategies below all measure precision and recall on the reject decision (a flagged claim is a positive; a correctly flagged claim, confirmed as true overutilization rather than later overturned on appeal, is a true positive), but they differ in how they hold data back.

Tuning strategy Threshold selection basis Overfitting guardrail Best fit
Full-set grid search max F1 over all history none — do not ship never
Single train/validation split max F1 on train, reported on held-out recent window temporal holdout + max train–validation gap stable formularies
Walk-forward backtest mean F1 across rolling temporal folds every fold is out-of-sample in time drifting utilization
Min-precision constraint max recall subject to precision ≥ floor business-set precision floor caps member abrasion reject-sensitive rules
Champion / challenger shadow live shadow delta against the incumbent production A/B before any cutover high-stakes boundaries

Two rules make the selection defensible. First, the validation window must be later in time than the training window, never a random shuffle — claims are a time series, and a random split leaks future prescribing patterns backward, inflating the score. Second, a candidate is rejected outright when its training F1 exceeds its validation F1 by more than a small gap (a few points): that gap is the overfitting signal, and a threshold that only performs on the data it was fit to is drift waiting to happen. The quantity-limit boundaries this calibrates are enforced field-by-field in Quantity Limit & Days Supply Validation, and the step-therapy trigger counts feed Step Therapy & Prior Auth Trigger Rules; this page decides where those boundaries sit, not how they parse a field.

Training versus validation F1 across candidate thresholds A line chart with the candidate threshold on the horizontal axis, running from lenient on the left to strict on the right, and the F1 score on the vertical axis. The training F1 curve rises steadily and keeps climbing toward the strict end as the threshold memorizes the training claims. The validation F1 curve rises to a peak near the middle of the axis and then falls as the threshold grows too strict for out-of-sample claims. A dashed vertical line marks the chosen threshold at the validation peak. The region to the right of the peak, where training F1 keeps rising but validation F1 declines, is shaded as the overfitting region. overfitting region train ↑ · validation ↓ chosen threshold train F1 validation F1 candidate threshold: lenient → strict F1 (precision · recall)

Figure: The chosen threshold sits at the validation-F1 peak, not the training-F1 maximum — everything to the right, where training keeps improving while validation declines, is the overfitting region a proper holdout is designed to reject.

Step-by-Step Implementation

The backtest harness below operates on aggregate feature bins, never on individual claims, so it carries no PHI. Each bin holds a feature value (for example quantity-per-day) and two counts: claims later confirmed as true overutilization, and claims later overturned as legitimate. Every rate is decimal.Decimal so precision and recall are exact rather than binary-float approximations.

1. Represent history as aggregate bins, not claims. Bucketing by feature value with confirmed/legitimate counts is enough to compute precision and recall, and it keeps the whole harness free of member-level data — the aggregation boundary is the one from Security & Compliance Boundaries for Claims Data.

2. Evaluate a candidate threshold into a confusion matrix. A claim is flagged when its feature value is at or above the threshold; a flagged-and-confirmed bin is a true positive, a flagged-and-legitimate bin a false positive.

3. Score with precision, recall, and F1 in Decimal. Guard the zero-denominator cases explicitly so an empty flag set scores zero rather than raising.

4. Select on train, judge on a later validation window, and reject overfit candidates. The chosen threshold maximizes training F1, but it only ships if its validation F1 stays within a bounded gap of the training score.

python
from __future__ import annotations
from dataclasses import dataclass
from decimal import Decimal, ROUND_HALF_UP
from typing import Sequence

ZERO = Decimal("0")


@dataclass(frozen=True)
class FeatureBin:
    """Aggregate outcome bucket — no PHI, no member-level rows."""
    value: Decimal        # feature under test, e.g. quantity-per-day
    n_confirmed: int      # claims truly warranting a reject (waste/overutilization)
    n_legitimate: int     # claims later overturned as valid (would-be false positive)


@dataclass(frozen=True)
class Scores:
    threshold: Decimal
    precision: Decimal
    recall: Decimal
    f1: Decimal


def _rate(num: int, den: int) -> Decimal:
    if den == 0:
        return ZERO
    return (Decimal(num) / Decimal(den)).quantize(Decimal("0.0001"), rounding=ROUND_HALF_UP)


def evaluate_threshold(bins: Sequence[FeatureBin], threshold: Decimal) -> Scores:
    """Flag a bin when value >= threshold; score the reject decision."""
    tp = sum(b.n_confirmed for b in bins if b.value >= threshold)      # confirmed & flagged
    fp = sum(b.n_legitimate for b in bins if b.value >= threshold)     # legitimate & flagged
    fn = sum(b.n_confirmed for b in bins if b.value < threshold)       # confirmed & missed
    precision = _rate(tp, tp + fp)
    recall = _rate(tp, tp + fn)
    denom = precision + recall
    f1 = ZERO if denom == 0 else (Decimal("2") * precision * recall / denom).quantize(
        Decimal("0.0001"), rounding=ROUND_HALF_UP)
    return Scores(threshold, precision, recall, f1)


@dataclass(frozen=True)
class TuningResult:
    chosen: Decimal
    train_f1: Decimal
    validation_f1: Decimal
    overfit: bool
    precision_ok: bool


def tune_threshold(
    train: Sequence[FeatureBin],
    validation: Sequence[FeatureBin],   # a LATER time window than train
    candidates: Sequence[Decimal],
    max_train_val_gap: Decimal = Decimal("0.05"),
    min_precision: Decimal = Decimal("0.80"),
) -> TuningResult:
    # Select the candidate with the best TRAIN F1 that also clears the precision floor.
    scored = [evaluate_threshold(train, c) for c in candidates]
    eligible = [s for s in scored if s.precision >= min_precision] or scored
    best = max(eligible, key=lambda s: s.f1)

    val = evaluate_threshold(validation, best.threshold)   # out-of-sample check
    gap = best.f1 - val.f1
    return TuningResult(
        chosen=best.threshold,
        train_f1=best.f1,
        validation_f1=val.f1,
        overfit=gap > max_train_val_gap,          # train >> validation => memorized noise
        precision_ok=val.precision >= min_precision,
    )

The harness is a pure function of its aggregate inputs, so a tuning run is reproducible and auditable: the same bins and candidate grid always select the same threshold, and the overfit flag is a hard gate a deploy pipeline can enforce. Because selection reads training bins but the ship decision reads a strictly-later validation window, a threshold that only fit last month’s noise is caught before it becomes a versioned config — the version bump itself follows the snapshot discipline described in the parent tuning workflow.

Verification and Testing Pattern

The properties to pin: precision and recall are computed exactly, the F1 selection prefers the generalizing threshold over the memorizing one, and the overfit gate actually fires when the validation score collapses.

python
import pytest
from decimal import Decimal


def bins(*rows) -> list[FeatureBin]:
    return [FeatureBin(Decimal(v), c, l) for v, c, l in rows]


def test_precision_recall_exact():
    # value, confirmed, legitimate
    data = bins(("1.0", 0, 40), ("2.0", 10, 10), ("3.0", 30, 2))
    s = evaluate_threshold(data, Decimal("2.0"))    # flag value >= 2.0
    # tp = 10+30=40, fp = 10+2=12, fn = 0
    assert s.precision == Decimal("0.7692")
    assert s.recall == Decimal("1.0000")


def test_empty_flag_set_scores_zero_not_error():
    data = bins(("1.0", 5, 5))
    s = evaluate_threshold(data, Decimal("9.9"))    # nothing flagged
    assert s.precision == Decimal("0") and s.f1 == Decimal("0")


def test_overfit_is_detected_when_validation_collapses():
    train = bins(("1.0", 0, 30), ("2.0", 5, 5), ("3.0", 45, 1))
    # Validation from a later window where the strict cut no longer separates classes.
    validation = bins(("1.0", 10, 20), ("2.0", 20, 18), ("3.0", 15, 22))
    res = tune_threshold(train, validation, [Decimal("1.0"), Decimal("2.0"), Decimal("3.0")],
                         max_train_val_gap=Decimal("0.05"), min_precision=Decimal("0.50"))
    assert res.overfit is True                       # train F1 >> validation F1


def test_precision_floor_constrains_selection():
    train = bins(("1.0", 40, 60), ("2.0", 30, 3))
    # Lenient cut has high recall but poor precision; the floor forces the strict cut.
    res = tune_threshold(train, train, [Decimal("1.0"), Decimal("2.0")],
                         min_precision=Decimal("0.80"))
    assert res.chosen == Decimal("2.0")

The overfit test is the load-bearing one: it proves the harness rejects a threshold that scored brilliantly on the window it was fit to but falls apart on a later one — the exact failure that ships a boundary destined to over-reject the moment prescribing shifts.

Gotchas and PHI Guardrails

  • Random splits leak the future. Shuffling claims into train and validation randomly lets next month’s prescribing patterns inform this month’s threshold, inflating the validation score into fiction. Split by time, always, with validation strictly later than training.
  • Optimizing F1 blind to the cost asymmetry. A false positive here is a legitimate fill rejected at the counter — a member-abrasion and helpdesk cost that a raw F1 hides. Pin a minimum-precision floor so recall is never bought with rejections the plan will overturn on appeal.
  • Tuning on rejects the engine itself created. Historical outcomes are censored: a claim the old threshold rejected never generated a “would-have-been-legitimate” signal unless it was appealed. Weight confirmed-versus-overturned counts by appeal-adjudication data, or the harness learns to reinforce the incumbent boundary.
  • Chasing drift into overfitting. Re-tuning every week to the latest noise produces a threshold that thrashes. Detect drift on control charts first, and only re-tune when the shift is statistically real — the loop, not the single fit, is what the parent Rule Engine Threshold Tuning & Optimization governs.
  • Silent class collapse. If a validation window contains almost no confirmed positives, recall is undefined and F1 is meaningless; the zero-guard returns zero rather than raising, but a run over a near-empty class should be flagged for a human, not shipped.
  • PHI has no place in a tuning corpus. Calibration runs on aggregate bins — counts by feature value — never on member-level claims. No 302-C2 Cardholder ID, no 310-CA Patient Name, no 407-D7/442-E7 at the row level reaches the harness or its logs; the corpus is de-identified aggregates and the audit record carries only the candidate grid, the chosen value, and the two F1 scores. Statistical de-identification of the corpus follows the standards catalogued by CMS.

← Back to Rule Engine Threshold Tuning & Optimization