Payment Dunning and Entitlement Grace: Retry Evidence, Restriction Generations, and Recovery Races

The payment platform reports that a renewal invoice has failed, and the system immediately disables the workspace. The customer has just updated their card and paid successfully, but an old failure Webhook arrives late and suspends the account again. Another approach is to provide a seven-day grace period after a failure, but when the retry service stalls and the invoice status is unknown, the timer still cancels automatically. The result can either penalize customers who have already paid or leave delinquent accounts with write access indefinitely. Dunning is not a sequence of emails; it is a controlled state machine between monetary facts and entitlement generations.

AI SaaS Entitlement Snapshots defines how runtime systems consume commercial authorization, while the Scheduled Automation Occurrence Ledger handles schedule instances. This article governs which invoice enters grace, recovery, or restriction, and under what evidence. It does not initiate price migrations itself or treat a single notification from the payment platform as the final monetary truth.

Publish dunningPolicy.v1

subscription.dunningPolicy.v1 binds the provider/account, applicable price/customer cohort, invoice-status mapping, retry owner (exactly one of the platform or the local system), maximum attempts and intervals, grace-period start and deadline, permitted entitlement profile, notification-template generation, manual exceptions, cancellation rules, clock, and recovery prerequisites. Dynamic Smart Retry schedules must preserve evidence of the provider's next payment attempt; a local schedule must not create a second set of charges at the same time.

schema: subscription.dunningPolicy.v1
policy: standard-monthly@6
retryAuthority: PAYMENT_PROVIDER
maxAttempts: 4
grace: {duration: P7D, profile: READ_EXPORT_NO_NEW_JOBS}
restrictAfter: GRACE_EXPIRED_AND_INVOICE_OPEN
recoverOn: INVOICE_PAID_RECEIPT
cancelAfter: MANUAL_OR_POLICY_TERMINAL

Each cycle binds the subscription, invoice ID, amount/currency, provider generation, attempt IDs, and cycleId. The subscription separately stores an outstandingCycleRoot covering every policy-blocking open or delinquent cycle; a new invoice must not overwrite older debt. Payment, voiding, marking uncollectible, refund, and dispute are distinct facts. Webhooks are deduplicated by event ID, after which the invoice/payment is fetched again; arrival order does not determine the truth. Webhook Subscription Correctness defines the signature-verification boundary.

Monetary State and Access State Meet at One Acceptance Point

A cycle begins with CURRENT -> PAYMENT_FAILED -> GRACE_ACTIVE. Retries proceed through RETRY_SCHEDULED -> RETRY_DISPATCHING -> RECOVERED/RETRY_FAILED/PAYMENT_UNKNOWN. A failure returns to GRACE, while a resolved unknown outcome returns to the same attempt; creating a new attempt to bypass it is forbidden. Only when no attempt is pending and the grace period expires may the cycle proceed through GRACE_EXPIRED -> RESTRICTING -> RESTRICTED -> CANCEL_PENDING -> CANCELED. A paid receipt can move one cycle to RECOVERED, but it does not automatically restore the subscription's entitlements. Conflicting events enter EVIDENCE_CONFLICT, and a manual grace period in EXCEPTION_ACTIVE must include an expiresAt value.

Every state transition enters a single transition-acceptance function. A CAS checks the cycle generation, invoice revision, attempt ID, policy, previous outstandingCycleRoot, and trusted clock. After updating the cycle, it recomputes the subscription-level root and aggregateEntitlementDecision, then atomically writes the transition receipt and outbox. A higher entitlement generation is issued only after all blocking debt has been cleared, or when the policy explicitly permits recovery with remaining debt. Otherwise, the cycle may be RECOVERED while the subscription remains restricted. A Webhook 200 response, an email, or a countdown does not constitute acceptance, and stale projections are fenced out by generation.

The first transition after a failure only records the payment attempt already reported by the platform; it does not initiate another charge. When an active local retry is genuinely required, first persist a stable operation ID, an invoice/amount digest, and PREPARED. Before dispatch, use CAS to move to DISPATCHING and pass the provider idempotency key. A network timeout enters PAYMENT_UNKNOWN, which can be resolved only by querying the provider's payment/invoice receipt. RFC 9110 method-idempotency semantics cannot replace a payment-provider-specific idempotency key and its retention period. HTTP Idempotency Key Correctness requires reusing the same key after an unknown outcome.

The grace-period profile must explicitly define read, write, export, API-token, background-job, and excess-resource access; it cannot be a Boolean active. Existing data must remain exportable, and safety and legal obligations do not switch off because of delinquency. Before applying restrictions after GRACE_EXPIRED, the system must fetch a fresh receipt confirming that the invoice remains unpaid. Payment and restriction CAS operations race on the same cycle generation, but the winning cycle transition must still recompute the outstanding root. If another blocking debt exists, it closes only that cycle and does not issue an entitlement generation that removes the restriction. A legacy price cohort from AI Pricing Migration must map to the correct invoice.

Inject Disorder, Outages, and Boundary Payments

Test out-of-order and duplicate Webhooks, lost retry responses, rescheduled next attempts, scheduler outages, payments at the deadline boundary, mark-uncollectible events, two open invoices with only one paid, outbox replay, stale caches, and chargebacks after restriction. When one cycle recovers while another blocking debt remains, the subscription must stay restricted. Unknown outcomes must not trigger automatic cancellation, and stale-generation updates must be rejected.

Release gates require a 100% event-attribution rate and 100% consistency in outstanding-root recomputation. Duplicate charges, full restoration while a blocking cycle remains, automatic cancellation without a fresh receipt, and stale-generation commits must all remain at zero. P99 from confirmed payment to compliant recovery must be below 60 seconds, and every PAYMENT_UNKNOWN must have an owner within one hour with no blind retries. Begin with two billing cycles in shadow mode, then roll out to notifications and 5% of low-risk subscriptions. During an incident, pause new restrictions and revert to the more permissive previous policy without fabricating payment facts. Recovery must explicitly advance the generation, and monetary facts must never be deleted.