SDK Quickstart Time to First Success: Event Contracts, Executable Examples, and Failure-Inclusive Denominators

A developer can open the Quickstart, copy the installation command, and even receive an HTTP 200 without having truly integrated the SDK: the 200 may come from a health check, and the example may use a cached response. Here, Time to First Success (TTFS) ends only when "the current SDK completes an operation with business meaning that can be proven by the server." AI Onboarding Design discusses interface guidance; AI Product Time to Value tracks when users receive a trustworthy result. TTFS is narrower: it diagnoses only the developer integration path and cannot be used to claim that users have already realized product value.

Freeze the journey before calculating elapsed time

Every public Quickstart publishes an immutable sdk.quickstartJourney.v1. It fixes the journeyVersion, documentation revision, SDK package name and artifact digest, example repository commit, API schema revision, region, and success predicate. The success predicate must refer to a supported action, such as "create a sandbox resource and then read back the same resource by ID"; it cannot be something vague like request returned 2xx. REST API Versioning handles interface evolution; this manifest additionally binds the tutorial developers actually see to the SDK artifact they use.

schema: sdk.quickstartJourney.v1
journeyVersion: node-create-widget@2026-07-19
sdkArtifact: sha256:42b...
docsRevision: 81ea0d7
apiRevision: widgets.v3
successPredicate: widget.created-and-read-back
environmentClass: clean-linux-node22
observationWindow: 30m
analysisCutoff: 2026-07-26T00:00:00Z
clockAuthority: api-edge-utc-v2
maxClockSkewMs: 100

The collector sends sdk.quickstartStep.v1, whose fields are limited to a random journeyId, attemptId, monotonically increasing stepSeq, step, outcome, errorClass, version tuple, occurredAt, and receivedAt. eventId is used for idempotent deduplication; late events may complete a journey but must not overwrite a higher generation. W3C Trace Context can correlate installer, SDK, and server-side spans, but trace identifiers are not user identities and must not contain API keys, source code, or email addresses. The pre-collection redaction described in Prompt Telemetry Governance applies here as well.

The endpoint is determined by the server-issued successReceipt. It contains the operation ID, predicate version, SDK user-agent, API revision, response digest, operationAcceptedAt when the operation was actually accepted by business logic, and the receivedAt for receipt generation and aggregation, while also declaring the clock authority and calibration error. Client self-reports are diagnostic signals only. If a browser disables telemetry but the server receives a qualifying call with a journey token, the journey can still complete; conversely, if the client displays a green state but the server has no receipt, the result can only be recorded as UNCONFIRMED. The token is a short-lived random value bound only to the journey version and does not track developers across sites.

The denominator must include those who never succeed

Publish at least three denominators together: eligible_start for developers who viewed and selected a compatible Quickstart, execution_start for those who actually ran the initialization command, and verified_success for those who received a server-side receipt. The manifest freezes observationWindow and analysisCutoff before the journey begins; for an unsuccessful journey, censorAt is the earlier of the start plus the observation window and the analysis cutoff. A TTFS distribution covers only successful journeys, making stuck journeys disappear, so it must be accompanied by success rates within 5, 15, and 30 minutes, the right-censoring rate, and failure categories. Segment results by language, SDK major version, operating system, region, and new versus existing accounts; buckets with insufficient samples remain INSUFFICIENT_DATA and must not be folded into the aggregate to claim an improvement.

The start time is the journey's server-issued eligibleStartedAt, and the endpoint is operationAcceptedAt calibrated to the same trusted time base; the receipt's receivedAt determines only the evidence watermark and must not add transport latency to TTFS. Cases involving different clock domains without calibration, errors above the manifest limit, or an endpoint earlier than the start are marked TIME_UNRESOLVED. If a late receipt proves that the business operation occurred before censorAt, append a correction generation and recompute subsequent reports without overwriting sealed snapshots. Retries retain the same journeyId and create a new attemptId, allowing both first-attempt success rates and retries after a fix to be observed. HTTP status semantics follow the bound version; 429, 401, and 422 must be categorized separately rather than all being compressed into "network error." API Rate-Limiting Strategies explains the operational boundaries of 429; TTFS is responsible only for proving whether it was interpreted correctly within the Quickstart journey.

Use broken environments to prove the metric does not lie

The fault suite covers documentation pointing to a withdrawn package version, incompatibility between the SDK and API schema, a missing scope, the wrong base URL, DNS failure, a 429 with Retry-After, a connection drop after a successful response but before the client persists it, duplicate and out-of-order events, a browser clock one hour fast, a server clock-domain mismatch, a blocked telemetry endpoint, and an operation completed on time whose receipt arrives after the watermark. The system must map these cases consistently to actionable errorClass values; in particular, it must not count a genuine success as a failure because instrumentation failed, add receipt latency to business duration, or stop the clock early because the client forged a success. LLM Observability introduces the separation of traces, evaluations, and monitoring; here, only the minimum diagnostic fields required by the Quickstart are retained.

Rollout gates require: 100% executability across all supported languages and current major versions in the clean-environment matrix; a qualified-operation success rate of at least 98% across 50 consecutive runs for every combination; a verified success rate within 15 minutes of at least 90%; TTFS p50 no higher than 4 minutes and p90 no higher than 12 minutes among successful journeys; UNKNOWN errors below 1% and TIME_UNRESOLVED at 0; zero difference in results recomputed by business occurrence time when receipts are duplicated, out of order, or merely delayed; and zero secret leakage from adversarial samples. Online comparisons must report the denominator, censorAt rule, sample size, and confidence interval together; if any one is missing, the result must not be labeled "faster."

During rollout, first dual-emit the old and new events and have a shadow aggregator recompute the same journeys for seven days; after all differences are explained, enable the new journey alias for only 5% of Quickstart sessions, then increase it progressively. If the success rate falls by more than 2 percentage points, UNCONFIRMED exceeds 1%, or any language bucket has no data, immediately point the alias back to the old documentation and SDK combination. Retain the new events as immutable diagnostic evidence, and have readers continue dual-reading until the longest journey window ends; rollback changes the entry point, not history.