Webhook Subscription Correctness: Endpoint Verification, Key Rotation, and Delivery Proof

Webhook control planes often collapse three distinct facts into a single green checkmark: a URL once passed verification, an HTTP request returned 2xx, and the subscriber has applied the event. They are not equivalent. Endpoint ownership changes, DNS can be rebound, and keys rotate; when a reverse proxy returns 204, the internal queue may not even have received the message yet. Event-Driven Architecture explains the decoupling of producers and consumers. This article addresses only the identity, signing, and evidence boundaries of cross-organization HTTP subscriptions and does not promise exactly-once delivery over the network.

Subscriptions Are Generational Resources

A registration request first generates a non-reusable subscriptionId and a monotonically increasing generation, binding the tenant, normalized HTTPS endpoint, topic/filter schema, event version set, delivery region, signing algorithm, retry policy, and expiration time. Its state is PENDING_CHALLENGE -> ACTIVE -> ROTATING -> SUSPENDED -> REVOKED | EXPIRED. Changing the URL, topic, accepted event major version, or owner creates a new generation. An old worker must not reread the current configuration and send an old delivery to the new endpoint.

Before activation, send a short-lived, one-time challenge to the endpoint. The response must be bound to the subscription, generation, endpoint digest, and expiration time. It proves only that the response path was controlled at that moment and cannot serve as permanent authorization. By default, neither challenges nor content deliveries follow 3xx redirects; Location is treated as a failure. An endpoint change must create a new subscription generation and repeat the challenge. Every direct connection must still resolve and pin allowed public IP addresses, restrict ports, prohibit HTTPS downgrades and private-network targets, and thereby prevent SSRF. Suspend the subscription when its certificate, DNS, or organizational ownership changes instead of silently following the change.

Subscription state and the delivery outbox are read within one transaction boundary: an attempt can be created only when the state is ACTIVE/ROTATING and the generation matches. Revocation uses CAS to increment the generation and block new attempts. Previously registered deliveries from an old generation are either drained to the old endpoint or quarantined according to policy; they must never be forwarded to the new endpoint. Background Job Queues handle scheduling and backoff, while the subscription record remains the source of truth for delivery eligibility.

Rotation Must Cover Both Sending and Verification

Every signing credential has a random keyId, a keyGeneration, a NOT_BEFORE/VERIFY_ONLY/ACTIVE/REVOKED state, and a maximum message lifetime. Normal rotation first publishes the new public key or securely delivers the new shared secret, allowing the subscriber to prove through a challenge that it can verify the new generation. The sender then switches to the new key, while retaining the old key as VERIFY_ONLY until the "maximum queue delay + maximum retry window + clock skew" period ends. A leak response instead revokes the old key immediately, suspends subscriptions that have not migrated, and restores them through an independently authenticated channel. It must not dual-sign indefinitely merely to preserve the delivery rate.

When using RFC 9421 HTTP Message Signatures, a versioned signature profile explicitly covers the method, target URI, Content-Digest, event ID, subscription generation, issuance time, and expiration time. A proxy must not rewrite the serialized raw body bytes after signing. A signature that does not cover the endpoint or generation permits replay across subscriptions. Signature-verification logs store only the key ID, digest, and reason code, never the shared secret.

Split "Sending" into Provable Tiers

The business fact uses a stable eventId and an immutable payload digest. Each subscription receives a stable deliveryId, while network retries increment only attempt. The ledger state can be written as:

PLANNED -> DISPATCHING -> ENDPOINT_ACCEPTED
                    \-> RETRYABLE | UNKNOWN_TRANSPORT | DEAD_LETTER
ENDPOINT_ACCEPTED -> APPLICATION_CONFIRMED   (only with a receipt protocol)

Before sending, persist the canonical bytes, target generation, key generation, and attempt. If the response is lost, the state is UNKNOWN_TRANSPORT; if the remote status cannot be queried, retry using the same delivery identity. An HTTP 2xx response can advance the state only to ENDPOINT_ACCEPTED. If the product must prove that the event was applied by the business application, the subscriber must send a separate signed applicationReceipt.v1 that binds the event ID, payload digest, subscriber operation ID, application result, and receipt key generation. Duplicate receipts are deduplicated by (subscriptionId,eventId), while differing digests enter conflict quarantine. Without this bilateral contract, label the tier UNOBSERVABLE; success cannot be inferred from a 2xx response.

Events may arrive more than once, out of order, or concurrently. Consumers must still be idempotent by event ID and use the aggregate version to decide whether an event can be applied. Distributed Tracing can correlate attempts with consumption spans, but missing trace samples cannot invalidate delivery evidence. Release Note Distribution Attribution further distinguishes reach from adoption; this article does not perform marketing attribution.

Break the Rotation Window, Then Run Acceptance Tests

Fault testing covers challenge replay, DNS pointing to a private network after verification, a 307 changing the target, delayed delivery by an old worker, half the instances still using the old key during rotation, a proxy re-encoding JSON after signing, the subscriber failing to enqueue after sending its response, a lost 2xx response, the same event with different payloads, receipt forgery, and revocation races. Assert that 3xx deliveries fail, an old generation is never delivered to a new endpoint, and the acceptance count for a revoked key is 0. Only for subscriptions with application receipts enabled should tests assert that retries do not produce a second applied effect; all other subscriptions remain UNOBSERVABLE.

Launch gates require 0 erroneous challenge activations, 0 undetected cases across one million signature mutations, endpoint acceptance during the rotation window no lower than the 99.95% baseline, reconciliation of at least 99% of UNKNOWN_TRANSPORT cases within 15 minutes, and an owner for every dead letter. A 100% subscriber deduplication rate applies only to subscriptions that pass application-receipt conformance. All others explicitly report UNOBSERVABLE and measure only delivery retries and endpoint acceptance. Begin with shadowing, then have internal subscriptions verify both key generations and switch in buckets. If anomalies occur, roll back only to a VERIFY_ONLY generation that remains secure; if the old key has leaked, suspend delivery instead.

Further Reading

• Event-Driven Architecture
• Background Job Queues
• Distributed Tracing
• Release Note Distribution Attribution