Transactional Outbox Delivery Reconciliation: From Local Commit to Broker Receipt

The order transaction has committed, but the process crashes before sending OrderCreated, so consumers never see the change. Conversely, if the message is sent before the database commits, a rollback leaves an event that references a nonexistent order. Wrapping the two calls in try/catch does not provide cross-system atomicity, and retrying after a publish request times out may cause the same order to be fulfilled twice.

Event-Driven Architecture explains asynchronous decoupling, while PostgreSQL Transactions and Isolation explains local atomic commits. Transactional Outbox has a narrower boundary: the business fact and pending delivery intent are committed together in the same database transaction, while broker delivery is handled as a separate, replayable, deduplicated stage.

Freeze the Event Identity and Payload First

Publish outboxEnvelope.v1 with fixed fields for eventId, envelope generation, aggregate type/ID/version, event type, payload schema/digest, tenant, occurredAt, and the creating transaction identifier. At commit time, also resolve and freeze destination.v1: the broker cluster identity, topic, routing/partition key and algorithm, route policy generation, serializer, durability, and required ACK. The eventId is generated when the business command is first accepted and reused by every attempt; the business change and PENDING envelope are inserted in the same transaction.

schema: outboxEnvelope.v1
eventId: evt-01J...
aggregate: order/o-42@17
type: order.created.v2
payloadSchema: sha256:6f1...
payloadDigest: sha256:a83...
routingKey: tenant-7.order.o-42
destination: kafka-prod/orders@route-19
delivery: {acks: all, serializer: order-v2}
state: PENDING

Allowed state transitions are limited to PENDING -> CLAIMED -> SENDING -> PUBLISHED -> RETAINED -> PURGED. Losing the ACK after sending moves the envelope into DISPATCH_UNKNOWN -> RECONCILING; if a receipt is found, it becomes PUBLISHED. When policy permits redelivery to the same destination, a CAS moves it into RETRY_CLAIMED and increments the attempt generation before it returns to SENDING. When acceptance cannot be proven, it remains UNKNOWN. Every retry must use the frozen destination digest; target migration must not modify the original envelope.

Multiple dispatchers may scan in parallel using short-lived claims, but each claim carries an owner, generation, leaseUntil, and attempt ID. The database claim determines only who may attempt a send; it does not mean the message has been published. When an old worker resumes after a pause, its generation is already invalid, so it may report only audit results and cannot overwrite the new owner's receipt. Background Job Queues provide throughput, while the claim fence determines eligibility to assert the fact.

The Receipt Transaction Is the Sole Acceptance Point

The sole acceptance point, acceptDispatchReceipt, locks the event row and matches the event ID, envelope generation, payload and destination digests, a registered SENDING/UNKNOWN attempt, and an unset publish slot. It then validates the broker receipt and sets the state to PUBLISHED. It does not require the ACK to come from the current claim: a late genuine ACK from an old attempt and an ACK from a new attempt both compete for the same slot. The winner writes the primary receipt, while the others are retained as duplicate transport receipts for evidence; any payload or destination mismatch is immediately quarantined. The old worker may not send again, but the result of a request it already sent is not discarded merely because its claim expired.

If the broker does not support queries by event ID, the reconciler may issue a new RETRY_CLAIMED only for the same frozen destination; consumers still deduplicate atomically on (consumerId, eventId). To change the cluster/topic, first prove that the old destination did not accept the event, then create an explicit envelope generation and migration receipt; cross-destination redelivery is prohibited while the old destination remains UNKNOWN. If the business genuinely requires delivery to two destinations, the delivery set, per-destination states, and completion rules must become a new business contract rather than masquerading as an ordinary retry.

Broker producer idempotence usually covers only specific session or partition conditions and cannot replace a business event ID. The consumer writes the consumption receipt and its own state in the same transaction, then commits the message offset; when it receives a duplicate event, it returns the original receipt. External effects such as email and payments still require stable business idempotency keys. MCP Timeouts, Retries, and Idempotency explains why an unknown outcome must not use a new key, while Binding HTTP Idempotency Keys to Payloads can constrain consumer-side calls.

The reconciler compares three sets: committed outbox entries, accepted publish receipts, and consumption watermarks for each critical consumer. An over-age PENDING entry indicates that the dispatcher missed it. A PUBLISHED entry with no consumption receipt for an extended period indicates a routing or consumer failure. An unknown event ID in the broker indicates out-of-band publishing. Cleanup must occur after the maximum replay window and the required consumer watermark; receiving an ACK is not sufficient reason to delete the only evidence.

Prove at Every Failure Point That an Event Is Neither Lost nor Re-created

Fault injection covers killing the process before and after commit, a late old ACK after claim expiry, races between UNKNOWN and a retry claim, ACKs for both old and new attempts, an unqueryable old destination during migration, route policy revisions, receipt rollbacks, out-of-order duplicate messages, loss of the ACK after a consumer commit, and erroneous cleanup decisions. Assert that committed business changes without an outbox entry, cross-destination retries while the outcome is unknown, payload/destination divergence under the same event ID, and duplicate effects for any consumer are all 0, and that every UNKNOWN can be traced by attempt.

The release gate requires, across one million traces, zero lost events, second logical event IDs, cross-destination retries, and duplicate consumer effects; 100% of late valid ACKs must converge into the publish slot, and UNKNOWN p99 must be reconciled within 15 minutes. Roll out in shadow mode first, then take over 5% of low-risk event types. A rollback switches only the dispatcher alias; the old implementation must understand the frozen destination, attempt generation, and shared publish slot. Published events must continue to be reconciled, and the system must never revert to synchronous blind sends.