Attributing Online-Offline Model Quality Drift: Cohorts, Delayed Labels, and Version Evidence

The candidate model scores four points higher on a frozen evaluation set, yet one week after launch, the share of users rating it "helpful" falls. The model itself may not have degraded: new traffic may come from more difficult entry points, negative feedback may not appear until refunds are issued or support tickets are closed, provider timeouts may send 8% of requests back to the old model, and the Prompt and retriever may have been hotfixed midway through the experiment. If two dashboards are joined only by date, any explanation can look plausible.

Online Evaluation Metrics for Agents defines observations such as success rate, human takeover, and cost, while Controlling RAG Evaluation Drift focuses on mismatches between query distributions and offline gold sets. This article covers the complete model application bundle and requires every online outcome to map back to the exact exposure rather than inferring attribution from aggregate trends.

Freeze the Attribution Contract and Actual Exposure

Publish qualityAttributionPlan.v1, binding the offline run receipt, candidate and baseline bundle digests, eligibility rules, stable cohort hash, traffic split, primary metric and guardrails, slice dictionary, minimum effect, analysis window, label source, maturation delay, maximum wait, right-censoring rules, fallback handling, and correction policy. At a minimum, the bundle must include the model, Prompt, retrieval index, tool catalog, output parser, safety policy, and provider routing. Recording only the model name is insufficient for attribution.

schema: qualityAttributionPlan.v1
plan: support-quality-q3@4
offlineRun: eval-2026q3-184@sha256:8a1...
bundles: [baseline@sha256:21c..., candidate@sha256:9bd...]
assignment: account-sticky@7
labelMaturity: 14d
maxWait: 30d
primary: resolved_without_reopen@30d

Before returning a response, the router writes an immutable assignmentReceipt.v1 containing the assignment, anonymized experimental unit, eligibility/cohort, intended bundle, slice, time, and data use. Execution and fallback receipts are appended under the same exposure ID and a stable attempt ID. A terminal servingReceipt.v1 is written only after the outcome is known, recording the actual bundle, fallback chain, and terminal state; if the outcome is unknown, only a SERVING_UNKNOWN observation is recorded. An assignment cannot masquerade as an actual exposure. Intent-to-treat analysis still requires complete reconciliation of every serving disposition.

Google's Rules of ML emphasizes keeping training and serving paths consistent, logging features, and monitoring skew. Hidden Technical Debt in Machine Learning Systems explains how data dependencies and configuration entanglement can cause local changes to have distant effects. These principles do not automatically establish causal conclusions. If an experiment upgrades the model, Prompt, and retriever simultaneously, the evidence can be attributed only to the entire bundle. Component-level attribution requires a separate controlled experiment, shadow replay, or preregistered interaction design.

Delayed Labels Require Watermarks and Terminal States

The states are ASSIGNED -> EXECUTING -> SERVED | SERVING_UNKNOWN | INVALID_CONFIRMED, followed by SERVING_UNKNOWN -> RECONCILING -> SERVED | INVALID_CONFIRMED. The reconciler queries router, provider, and delivery receipts only by the original exposure/attempt ID. It cannot change the ID or substitute a rerun. An observation may be marked invalid only when evidence proves that it was not served or that it is unrecoverable under the plan. finalizeServingReceipt uses CAS to match the assignment/attempt/reconciliation root and an empty terminal slot. Duplicate requests return the original receipt, while conflicts are quarantined. Only SERVED enters AWAITING_LABEL -> MATURED | RIGHT_CENSORED, and labels are always joined by stable outcome/exposure ID.

A window enters READY_TO_CLOSE only after the exposure watermark has passed its end time, the label maturation period and ingest watermark requirements have been satisfied, and SERVING_UNKNOWN = 0. Immature samples are reported separately and are right-censored according to the plan only after maxWait. Each cohort must also report its delay distribution and sensitivity analysis.

The sole window acceptance point, closeQualityWindow, uses CAS to match the plan, assignment, attempt/reconciliation, terminal serving, label/correction, bundle, and metric roots. It requires SERVING_UNKNOWN = 0, verifies denominator conservation, and then writes qualityAttributionReceipt.v1. Late-arriving labels or confirmed errors create a correction generation; the old report is marked superseded instead of having its numbers changed. AI Cost Attribution can reuse the exposure identity, but an assignment or billing line cannot serve as a quality exposure.

Offline and online results should be compared only directionally under identical slices and metric definitions; they should not be subtracted into a single "drift score." First examine how well online inputs cover the offline set, each slice's offline errors and online outcomes, and then distinguish among INPUT_MIX_SHIFT, LABEL_PROCESS_SHIFT, SERVING_BUNDLE_DRIFT, MODEL_BEHAVIOR_DRIFT, and UNRESOLVED. If the aggregate improves while a high-risk slice degrades, the model must not be promoted based on the average. Troubleshooting Model Provider Degradation helps rule out regional and provider incidents.

Deliberately Misalign Labels and Versions

Fault injection covers changes to the cohort salt, users crossing experiment groups, assignments masquerading as exposures, missing fallback records, lost ACKs after responses, unavailable provider queries or queries returning the wrong attempt, replacing UNKNOWN with a new ID, races between two reconcilers, Prompt hotfixes without a bundle version bump, delayed labels, and population mixing. The expected results are zero windows closed with UNKNOWN records, zero invalidations without evidence, zero UNKNOWN records replaced through new IDs, and zero exposures with multiple terminal states. Report denominators must balance exactly.

Release gates require assignment coverage of at least 99.99%, terminal serving/bundle coverage of 100%, a SERVING_UNKNOWN reconciliation p99 no greater than 15 minutes and a count of zero at close, and zero bucketing or label-join conflicts. Every critical slice must report its sample count and interval, and the offline run must trace back to Reproducible Evaluation Run Artifacts. Begin with shadow receipts, then run a 1% canary, and do not ramp traffic before labels mature. If a guardrail triggers, switch back to the baseline while retaining negative exposures. Corrections must be append-only.