Scheduled Automation Execution Ledger: Time-Zone Occurrences, DST, and Missed-Run Recovery

"Send the operating summary every morning at 9" may look like a single cron line, but in production it involves at least four times: the local time understood by the operator, the UTC instant, the time when the scheduler actually discovers the task, and the time when the worker actually starts. When daylight saving time begins, a local time may not exist; when it ends, the same time may occur twice. If the platform recovers after a two-hour outage, it must also decide whether to catch up the two missed runs. If the retry key uses the current time, every recovery creates a new email.

Background Job Queues explains workers, retries, and dead-letter queues, while Multi-Agent DAG Scheduling and Recovery addresses intelligent tasks with dependencies. This article defines only how a recurring schedule is materialized into a unique occurrence and whether that occurrence has already produced a business effect. It does not cover general queue selection or treat model planning as a clock.

Freeze the Time Contract, Not Just the Cron String

When publishing scheduleSpec.v1, freeze the scheduleId, revision, IANA timeZone, tzdbVersion, local recurrence rule, start and end boundaries, gapPolicy, foldPolicy, misfirePolicy, maximum catch-up count, and permitted lateness window. RFC 5545 recurrence rules can express periodic schedules, while the IANA time zone database provides historical mappings between local time and UTC offsets. Neither can make the product decision about how to handle gaps and folds, so those policies must be versioned fields.

schema: scheduleSpec.v1
scheduleId: daily-ops-summary
revision: 7
timeZone: America/New_York
tzdbVersion: 2026a
rule: "FREQ=DAILY;BYHOUR=2;BYMINUTE=30"
gapPolicy: SKIP
foldPolicy: FIRST_ONLY
misfirePolicy: CATCH_UP_BOUNDED
maxCatchUp: 2
cutoverAt: 2026-11-01T06:00:00Z

When 02:30 does not exist in spring, SKIP must produce a SKIPPED_GAP record with a reason instead of silently moving it to 03:00. If the business selects SHIFT_FORWARD, the shifted instant must also be recorded. The two occurrences of 01:30 in fall are handled explicitly through FIRST_ONLY, SECOND_ONLY, or BOTH. The occurrenceId is calculated from the schedule ID, revision, nominal local date and time, and fold ordinal. A separate logicalSlotId that excludes the revision binds the schedule ID to the policy-resolved UTC instant, preventing duplicates across revisions. Neither ID may use discovery or start time.

Schedule revisions use a half-open interval at the UTC cutoverAt: the old revision materializes only instances whose resolved instants are < cutoverAt, while the new revision materializes only those whose resolved instants are >= cutoverAt. The cutover ledger applies a unique constraint and CAS to (scheduleId, logicalSlotId), preventing two revisions from claiming the boundary slot simultaneously. Materialized history must not be renumbered because of a tzdb update. If a legal change affects future instances, publish a new revision and retain evidence of the old calculation. Database Migrations addresses schema evolution; this ledger protects the interpretation of time itself.

Account for Occurrences and Attempts Separately

The materializer is constrained by both the occurrence detail key and the cross-revision logical-slot uniqueness rule, and it writes the outbox in the same transaction. States are limited to PLANNED -> DUE -> DISPATCHING -> RUNNING -> SUCCEEDED. Failures may enter RETRY_WAIT, while an external outcome that cannot be confirmed enters UNKNOWN_EFFECT. SKIPPED_GAP, MISFIRE_EXPIRED, and CANCELLED are all terminal facts and must not be treated as "no record."

Create a separate attempt for every execution, containing the claim generation, worker, start and finish times, input digest, and error class. A retry adds an attempt rather than creating a new occurrence. When delivering an email or generating a report, pass the occurrence ID downstream as the business idempotency key. If the downstream system supports neither idempotency nor result queries, a timeout must not trigger a blind resend; the occurrence can only be quarantined and reconciled. MCP Timeouts, Retries, and Idempotency provides the general invocation semantics, while this ledger scopes the stable key to one calendar occurrence.

Materialization looks ahead only through a finite horizon, such as 14 days. When multiple schedulers scan concurrently, a unique constraint selects the single recorded fact rather than relying on assumptions about host locks. After the platform recovers, first calculate the set of missed occurrences and then apply the misfire policy. CATCH_UP_BOUNDED catches up no more than the configured count and records the rest as MISFIRE_EXPIRED, preventing seven summaries from being sent together after a week-long outage. Before external I/O, the worker uses CAS to change DUE/RETRY_WAIT to DISPATCHING while also matching the active revision, cutover gate, disableGeneration, and execution epoch. Disabling, revision changes, and dispatch update the same gate, so only one can win. An effect whose dispatch has already won must enter reconciliation; it cannot be treated as canceled.

Use Time-Fault Tests to Prove There Are No Duplicates

The test matrix must cover at least a spring-forward gap, two fall-back offsets, a tzdb version update, leap day, month end, recovery across multiple periods after scheduler downtime, two materializers discovering the same occurrence concurrently, a lost ACK after worker success, a retry later than the next period, an old revision waking after cutover, and a race between disabling and starting. Assert that the occurrence set exactly matches the expectation for each policy, that one hundred deliveries of the same ID produce at most one confirmed business effect, and that every unconfirmed item remains UNKNOWN.

Acceptance gates require 100% agreement across golden time-zone cases; zero duplicate occurrences across 100,000 randomly generated boundary times; completion of materialization within five minutes after fault recovery; zero executions beyond the catch-up limit; reconciliation or placement into a human queue for every UNKNOWN_EFFECT within 30 minutes; and 100% coverage for validating the revision and disable generation at every execution acceptance point. Use Prometheus and Grafana to monitor materialization latency, misfires, attempt counts, and unknown effects, but metrics cannot replace ledger facts.

Begin rollout by having the new materializer perform shadow calculations for four weeks, comparing only occurrence IDs and policy reasons without dispatching tasks. Then let it take over one internal schedule with no external effects before expanding to 5%. During rollback, atomically switch the scheduler implementation alias back to the previous implementation, increment the execution epoch, and stop new claims. The old implementation must continue reading the same occurrence ledger and current scheduleSpec; it must never resume the old schedule range that has already crossed the cutover. A successful occurrence must not be deleted and then resent by the old system. Only instances that have not started and that the old implementation can recognize may be requeued.