Bulk Customer Operation Control Plane: Dry-run Snapshots, Dual Approval, and Effect Reconciliation

An ops person filters "trial-expired customers with no payment method," preparing to bulk-revoke premium entitlements. The dry-run shows 812 accounts; ten minutes later, when Execute is clicked, the query re-evaluates, and 37 customers have already completed payment; another worker times out after a successful update on a third-party system, and the retry sends a downgrade notification again. The problem isn't whether the button has a confirmation step — it's that the object that was approved and the object that actually took effect are not the same version of the truth.

Agent Human Approval Gate discusses when model actions are handed to humans, Approval Replay Prevention binds a single authorization to Agent execution, and High-Risk AI Human Override records how humans change machine recommendations. This article deals with multi-customer changes initiated proactively by ops staff: the focus is on freezing the target set, verifying state item by item, and reconstructing the net effect after partial success. Financial amounts are still handled by dedicated ledgers such as the AI Usage Metering Ledger; this control plane must not be treated as an accounting system.

The Dry-run Must Produce an Immutable Plan

When creating bulkOperationPlan.v1, resolve the filter conditions inside a consistent snapshot, recording the query revision, tenant scope, operator authorization snapshot, and each target's resourceVersion. The plan includes the operation schema, normalized parameters, the Merkle root of the target list, expected impact, irreversible items, rate limits, and expiration time. The counts shown in the UI are read from this list; the executor must not re-run the original filter SQL.

{
  "schema": "bulkOperationPlan.v1",
  "planId": "bop_01K...",
  "operation": "entitlement.revoke@3",
  "targetSetRoot": "sha256:91c...",
  "targetCount": 812,
  "snapshotWatermark": "pg:00016/B374D848",
  "authorizationGeneration": 44,
  "expiresAt": "2026-07-19T12:30:00Z",
  "maxEffectsPerMinute": 50
}

The dry-run runs the same schema, authorization, preconditions, and policy judgments as the real path against each target, only swapping the effect adapter for PLAN_ONLY. Results are distinguished into ELIGIBLE, WOULD_SKIP_STALE, WOULD_DENY, and UNKNOWN_DEPENDENCY; unknowns must not be counted as executable successes. Once the plan is FROZEN, parameters may not be changed and targets may not be added or removed; any modification produces a new plan ID and digest, and the original approval is automatically invalidated. PostgreSQL Transaction Isolation can provide read consistency, but business versions must still be saved per item to avoid resource changes after approval.

The approval record binds the plan digest, risk summary, target count, operation version, deadline, and approver role. High-risk operations require the initiator and approver to be separate, consistent with the separation-of-duties principle in NIST controls. The approval page must sample real targets, each reason bucket, and worst-case impact, rather than just showing "812 items." Approval expiry, authorization revocation, or policy generation changes all revert the state to REVIEW_REQUIRED.

Each Target Has Exactly One Effect Identity

Plan state advances along DRAFT -> FROZEN -> APPROVED -> EXECUTING -> RECONCILING; approval failure goes to REVIEW_REQUIRED; pause follows EXECUTING -> PAUSING -> PAUSED; only then do COMPLETED, PARTIAL, or ABORTED apply. Each item stores the targetId, expected resource version, effectKey = H(planId,targetId,operationRevision), attempt, adapter receipt, and observed result. Before external I/O, a PREPARED -> DISPATCHING CAS simultaneously matches the plan digest, approval and authorization generation, execution epoch, and target version; pause or authorization revocation raises the same gate, and only one side of the dispatch can succeed. Customers who have already paid get SKIPPED_PRECONDITION, rather than being revoked first and restored afterward.

In-database changes place the business write and the item receipt in the same transaction. Third-party calls prioritize passing a stable idempotency key and conditional version; the general idempotency semantics of HTTP methods come from RFC 9110, but the product still must confirm the vendor's key scope and retention period. If the connection drops after the other side succeeds, the item enters UNKNOWN_EFFECT. Only the authoritative terminal state of the provider operation, or a consistent read that has reached that operation's watermark, can prove APPLIED/NOT_APPLIED; stale replicas, timeouts, and NOT_FOUND past the idempotency retention period all remain UNKNOWN. Changing the key or calling again is forbidden.

Multi-Agent Task Claim prevents late workers from committing, but it does not prove whether the third party has already taken effect; so coordination ownership and business effect must be recorded separately. Pause only blocks items that haven't started yet; it can't rename an applied item as cancelled. When recovery is needed, create an explicit compensation plan listing the reversible scope, inverse operation version, and new approval; using a database UPDATE to erase the original result is forbidden.

Fault Suite and Quantitative Gates

Inject scenarios: a target changes after approval, the same approval button is clicked repeatedly, two executors grab the same item, authorization is revoked during approval, a third-party 200 loses the response, a 429 crosses the idempotency window, a pause at 40% execution, a compensation that only succeeds halfway, and a reconciliation read of a stale replica. The system must keep the target count from growing, prevent old versions from executing, and prove per provider attempt/effect that each effect key has at most one real effect; when confirmation is missing, it must keep UNKNOWN_EFFECT, and must not rely solely on local receipts to self-certify dedup.

Release gates require: the plan root matches 100% at the approval and execution entry points; dispatch CAS coverage is 100%; ten thousand repeated deliveries produce zero duplicate provider attempts/effects; unknowns must not be counted in the success rate; after the pause gate is raised, new dispatches are zero; the per-item receipt aggregate differs from the plan terminal-state count by zero. High-risk batches first limit to at most 50 targets; the cap can be raised only after 20 consecutive batches with no unexplained discrepancies.

For launch, first shadow-generate plans and compare target sets against manual scripts, then enable read-only notification-class operations, and only finally open up entitlement changes. If stale skips exceed 2%, unknown effects exceed 0.5%, or reconciliation is non-conservative, raise the execution epoch, stop new items, and cut the entry point back to the old tool; the old flow must also consume the same dispatch registry and take over only explicitly unstarted items. Rolling back the control plane is not rolling back customer facts; any effect that has already occurred must be closed out through reconciliation and compensation.