Kafka vs RabbitMQ for PBM Adjudication

The transport decision for an adjudication path comes down to a single question: does the claim event stream need a durable, replayable, partition-ordered log, or a broker that routes and acknowledges individual messages? Apache Kafka and RabbitMQ both move claim events between the ingestion, pricing, and prior-authorization services, but they make opposite tradeoffs on ordering, replay, throughput ceiling, and operational overhead. For a sub-2-second B1 path that must reproduce any claim’s adjudication during a payer audit and keep a member’s reversal (B2) strictly behind its billing (B1), those tradeoffs decide the architecture. This page is the head-to-head under Claim State Infrastructure Selection, scoped to the exact decision rather than the surrounding topology.

The Exact Decision #

Kafka is a durable, partitioned commit log: producers append events to a topic partition, consumers track their own offset, and the log retains events for a configured window (or forever) so any consumer can replay from any offset. Ordering is guaranteed within a partition, which is exactly the guarantee adjudication needs when the partition key is the 302-C2 Cardholder ID token — a member’s events stay in order, unrelated members parallelize across partitions. RabbitMQ is a broker: producers publish to an exchange, the exchange routes to queues by binding rules, and consumers acknowledge each message, after which it is gone. RabbitMQ excels at flexible routing and per-message workflow semantics but has no native replay — once a message is acked, reproducing it means the producer re-sends. The decision is therefore not “which is faster” but “does audit replay and per-member ordering at 10k+ msg/s dominate, or does routing flexibility at modest volume dominate.”

Decision Matrix #

For the primary point-of-sale adjudication stream, Kafka is the default: audit replay and per-member ordering are non-negotiable, and the volume justifies the partition model. RabbitMQ is the right tool for a narrower band — task-style workflows like batch reconciliation fan-out, credential-rotation jobs, or notification routing where flexible bindings matter and volume is modest. Many production platforms run both: Kafka for the ordered adjudication log, RabbitMQ for operational task routing.

Figure: A partitioned log retains ordered events so an audit reader replays by resetting its offset, while a broker removes each acknowledged message and dead-letters poison payloads — the crux of the transport choice.

Step-by-Step: A Publisher Abstraction with Both Backends #

The engine publishes through one EventPublisher Protocol so adjudication code never imports a broker client. Below, a Kafka backend (confluent-kafka) and a RabbitMQ backend (pika) both satisfy it, and both refuse to put raw claim bytes on the wire.

1. Define the transport-agnostic contract. Ordering is expressed as an explicit key, never left implicit, so per-member ordering is a property of the call site.

from __future__ import annotations
import json
from typing import Protocol


class EventPublisher(Protocol):
    def publish(self, ordering_key: str, event: dict) -> None: ...


def _encode(event: dict) -> bytes:
    # PHI GUARDRAIL: only tokenized identifiers may be serialized.
    # 302-C2 Cardholder ID and 310-CA Patient Name are ALREADY tokenized upstream;
    # raw NCPDP D.0 bytes and 431-DV Other Payer Amount detail never reach a topic.
    forbidden = {"302-C2", "310-CA", "raw_payload"}
    if forbidden & event.keys():
        raise ValueError("refusing to publish raw PHI to the transport")
    return json.dumps(event, separators=(",", ":")).encode("utf-8")

2. Implement the Kafka backend. The ordering_key becomes the partition key, so events for one member token land on one partition and stay ordered. acks="all" plus enable.idempotence gives at-least-once with no in-partition duplicates from producer retries.

from confluent_kafka import Producer


class KafkaEventPublisher:
    def __init__(self, brokers: str, topic: str) -> None:
        self._topic = topic
        self._producer = Producer({
            "bootstrap.servers": brokers,
            "acks": "all",                 # wait for in-sync replicas
            "enable.idempotence": True,    # no duplicate on producer retry
            "linger.ms": 5,                # small batch window for throughput
        })

    def publish(self, ordering_key: str, event: dict) -> None:
        # key=member token -> same partition -> per-member ordering (B2 stays behind B1)
        self._producer.produce(
            self._topic,
            key=ordering_key.encode("utf-8"),
            value=_encode(event),
        )
        self._producer.poll(0)  # serve delivery callbacks without blocking

3. Implement the RabbitMQ backend. There is no partition key; ordering rides on a single queue. A consistent-hash or direct exchange routes by the key, and publisher confirms give at-least-once. Note the explicit lack of replay — this backend is for routing-shaped workloads.

import pika


class RabbitEventPublisher:
    def __init__(self, url: str, exchange: str) -> None:
        self._exchange = exchange
        params = pika.URLParameters(url)
        self._conn = pika.BlockingConnection(params)
        self._ch = self._conn.channel()
        self._ch.confirm_delivery()  # publisher confirms (at-least-once)
        self._ch.exchange_declare(exchange=exchange, exchange_type="direct", durable=True)

    def publish(self, ordering_key: str, event: dict) -> None:
        self._ch.basic_publish(
            exchange=self._exchange,
            routing_key=ordering_key,      # routes to the member's queue
            body=_encode(event),
            properties=pika.BasicProperties(delivery_mode=2),  # persistent
        )

4. Inject the chosen backend once. The adjudication service receives an EventPublisher and never learns which transport it is, so the decision is a wiring change, not a code change.

Verification: Assert Per-Key Ordering #

The correctness property that separates a correct transport wiring from a subtle bug is: events sharing an ordering key are delivered in publish order. Drive it against an in-memory fake that records ordering-key/offset pairs, then assert monotonic order per key.

import pytest


class RecordingPublisher:
    """In-memory EventPublisher fake; records (ordering_key, seq) as published."""
    def __init__(self) -> None:
        self.log: list[tuple[str, int]] = []

    def publish(self, ordering_key: str, event: dict) -> None:
        _encode(event)  # exercise the PHI guard on the real path
        self.log.append((ordering_key, event["seq"]))


def test_per_member_ordering_is_preserved():
    pub = RecordingPublisher()
    member_a = "tok_302C2_A"  # tokenized 302-C2, never a raw cardholder id
    member_b = "tok_302C2_B"
    # interleave two members' events
    for seq in range(5):
        pub.publish(member_a, {"seq": seq, "correlation_id": f"A{seq}"})
        pub.publish(member_b, {"seq": seq, "correlation_id": f"B{seq}"})

    for key in (member_a, member_b):
        seqs = [s for k, s in pub.log if k == key]
        assert seqs == sorted(seqs), f"ordering broke for {key}"


def test_publish_refuses_raw_phi():
    pub = RecordingPublisher()
    with pytest.raises(ValueError):
        pub.publish("tok_302C2_A", {"seq": 0, "302-C2": "RAW-MEMBER-123"})

The first test pins the ordering contract that Kafka gives per partition and that RabbitMQ gives only on a single, single-consumer queue. The second test pins the PHI guard so a refactor cannot start leaking raw 302-C2 onto the transport.

Gotchas & PHI Guardrails #

• Never put raw claim bytes on a topic. The most damaging RabbitMQ/Kafka mistake in adjudication is publishing the raw NCPDP D.0 transaction — including 302-C2 Cardholder ID, 310-CA Patient Name, and 431-DV Other Payer Amount — because a topic with long retention becomes a durable, replayable PHI store. Publish tokens and a correlation_id only; the _encode guard above enforces it.
• RabbitMQ ordering is fragile. A single queue with one consumer is FIFO, but add a second consumer, a redelivery after a nack, or a priority setting and messages reorder. If you need per-member ordering on RabbitMQ, you need a consistent-hash exchange and single-consumer-per-key discipline — at which point Kafka’s partition model is simpler.
• Kafka rebalance storms stall the hot path. A consumer-group membership change pauses consumption. Use cooperative-sticky assignment and static membership so a deploy does not freeze in-flight B1 claims; see the worker model in Asynchronous Batch Adjudication Workflows.
• Exactly-once is expensive; prefer idempotency. Kafka transactions and RabbitMQ’s careful ack dance both add latency. The 402-D2 Prescription/Service Reference # token as an idempotency key gives exactly-once effect (no double-paid claim, no duplicate reject 79) far more cheaply than transport-level exactly-once.
• Size the dead-letter path deliberately. Whether it is a Kafka DLQ topic or a RabbitMQ dead-letter exchange, the envelope must be tokenized and the queue must be sized for a replay burst without unbounded growth — quantified in Sizing Dead-Letter Queues for Claim Replay.
• Retention is a compliance parameter. Kafka retention on the audit stream must match the audit-retention window, tiered to cold storage; treating it as a throwaway buffer loses replayability. Consult the Apache Kafka documentation for log.retention and compaction semantics before setting it.

Related #

• Claim State Infrastructure Selection — the surrounding decision that frames transport, state store, and validation together.
• Redis vs DynamoDB for Ephemeral Claim State — the state-store half of the infrastructure decision that this transport feeds.
• Asynchronous Batch Adjudication Workflows — the worker-pool and queue model that consumes the chosen transport.
• Sizing Dead-Letter Queues for Claim Replay — the poison-message and replay-burst sizing that both transports depend on.
• Fallback Routing Logic Design — the deterministic degradation when the transport is unreachable.

← Back to Claim State Infrastructure Selection