Developer Launch Holdout Experiments: Random Assignment, Contamination Measurement, and Incremental Adoption Thresholds

Registrations doubling on launch day proves only that the events coincided in time. The new version might have been updated automatically anyway, and a popular community post could bring in both the treatment group and people who were never directly reached. To answer "how much incremental adoption did this developer marketing campaign cause?", you must reserve a randomized, stable holdout before sending anything, one that supplemental sends by the operations team cannot compromise. A/B Testing AI Features compares in-product feature behavior; this article compares launch-message outreach strategies. The feature itself remains equally available to both groups.

This is not a revalidation of demand or willingness to pay, either. Kill/Pivot Criteria for AI Products determines whether the product direction should continue, while AI Feature Evidence Scorecards compares opportunity priorities. A Launch holdout estimates only the causal increment of a defined campaign on an eligible developer adoption action; it cannot be extrapolated into product-market fit.

Seal the Protocol Before Seeing Results

The experiment creates an immutable developer.launchExperiment.v1 that fixes the hypothesis, eligibility query digest, exclusion rules, assignment unit, stratification variables, random salt key ID, treatment ratio, channel and creative digests, start and end times, minimum detectable effect, sample size, primary metric, guardrails, contamination definition, and stopping rules. The protocol digest is written to every event; any modification creates a new experiment generation. The endpoint or denominator cannot be changed after results are viewed.

schema: developer.launchExperiment.v1
experimentId: sdk-4.2-launch
unit: organization
eligibilitySnapshot: sha256:8ae...
allocation: {treatment: 0.8, holdout: 0.2}
primary: adopted_within_14d
mdeAbsolute: 0.02
analysis: intent-to-treat

First, freeze the set of organizations at the eligibility cutoff, then use HMAC(saltGeneration, experimentId || organizationRef) to determine assignment. Raw organization IDs never enter the analytics warehouse; all members of the same organization must be in the same arm, so that an administrator receiving an email does not influence an "unreached" colleague. Perform preregistered stratified randomization by historical activity, SDK major version, and region, and use the same stratum weights in analysis. Account merges or member invitations only append membership evidence; they do not trigger reassignment. Immediately after assignment, compare actual arm counts with expected allocation overall and for each preregistered stratum, and run the protocol-specified sample-ratio mismatch (SRM) checks using the specified thresholds and multiple-testing correction. An SRM failure immediately freezes exposure and analysis; "the deterministic hash looks correct" is not a substitute for investigating lost eligibility records or missing events.

Assignment produces launch.assignment.v1, recording the protocol digest, unitRef, arm, generation, and assignedAt; when a channel accepts a delivery, write launch.deliveryReceipt.v1, including the channel, rendition, provider receipt, and failure reason. Write a separate launch.observableExposure.v1 only when the client or product surface provides a qualifying visibility signal; provider acceptance must not masquerade as exposure. Adoption is proven by server-side launch.outcome.v1, such as the first successful call to the new API, not by a button click. All these events have stable IDs, occurred and received timestamps, and watermarks; duplicates, out-of-order events, and late arrivals change only material completeness, not assignment. The metric and trace layering in LLM Observability can support aggregation, but high-cardinality unitRef values must remain only in the controlled event warehouse.

Keep the Primary Analysis Intent-to-Treat

The primary outcome uses all assigned eligible units in an intention-to-treat analysis: failed delivery to the treatment group remains in treatment, and holdout organizations that encounter the public message elsewhere remain in holdout. This estimates the Launch strategy as it can actually be executed, rather than "only selecting people who were successfully reached." Also report the treatment delivery rate, holdout contamination, and treatment non-compliance to help explain how much the effect was diluted, but do not exclude these units after the fact.

If all organizations are simultaneously in a public-release environment, organization-level randomization can estimate only the assignment effect of "private-channel outreach layered on top of a public release"; it cannot answer the increment of the entire public launch relative to no release at all. The latter requires independently separated communities, regions, or time windows to be cluster-randomized, with isolation and washout preregistered. When propagation intensity creates interference, use a saturation design and report coverage levels for different clusters rather than treating contaminated organizations as independent samples.

The report gives adoption rates for both groups, the absolute percentage-point difference, relative ratio, confidence interval, and preregistered sample size. Do not test for significance every hour during the window and declare victory early; use a fixed endpoint, or preregister a sequential test and alpha spending in the protocol. A major service incident pauses exposure according to the rules and marks the result INTERRUPTED; do not delete the incident days and continue by splicing the remaining data together. Time to Value for AI Products can serve as a post-adoption quality guardrail, but the primary endpoint remains the developer action frozen here.

Fault injections include: an accidental rotation of the random salt that reassigns units; selective loss of assignment events in one stratum; duplicate delivery caused by message retries; manual CSV sends to the holdout; link-preview bots generating fake clicks; organization members spanning accounts; accounts created after the eligibility cutoff; adoption events arriving after the watermark; client clocks moving backward; all channels going down in one region; public-release spillover across clusters; and a feature flag that incorrectly enables the feature only for treatment. The system must ensure SRM detects selective loss, block the mixing of assignment generations, flag interference and channel unavailability, and prove that feature availability is identical in both groups. Feature Flag Rollouts can independently control runtime risk, but must not secretly serve as the marketing experiment's assignment mechanism.

Before formal analysis begins, require: zero assignment recompute mismatches in the A/A rehearsal; SRM passes the preregistered threshold both overall and in every preregistered stratum; duplicate events have zero impact on unit counts; the maximum standardized difference in key pre-experiment covariates is below 0.1; treatment delivery is at least 95%; unexplained direct delivery to holdout is below 0.5%; adoption evidence matches server records 100%; completeness within the late-arrival watermark is at least 99%; and there is no difference in feature permissions between groups. An example launch gate can require the lower bound of the primary metric's 95% interval to exceed 0 and the absolute lift to be at least the preregistered 2 percentage points, while the error rate increases by no more than 0.5 percentage points and support tickets by no more than 10%. Failure to meet these criteria means INCONCLUSIVE or failure; do not switch to click-through rate in search of a green conclusion.

The rollout sequence starts with a 2% A/A, then uses 10% of the eligible population to validate delivery and contamination, and finally starts the formal experiment on the frozen sample. If permission differences, assignment mismatches, or guardrail breaches occur, immediately stop new messages and in-product entry points and switch the channel alias back to the no-campaign state; already assigned units retain their original arms until the observation window ends to avoid cross-contamination after rollback. You may restore the old creative or end the campaign, but must not rerandomize the holdout, delete negative results, or relabel people who were not sent messages as a new control group. AI Onboarding can incorporate guidance shown to be effective after the experiment ends, provided it gets a separate version and baseline.