AI Product Time to Value Baseline: From First Valid Input to a Trusted Result

A user successfully registering, opening the workspace, or sending the first prompt does not necessarily mean they have gotten value. AI products often treat "the model returned text" as activation, but users may still need to correct facts, add context, wait for a colleague's approval, and ultimately fall back to the old tool. Time to Value (TTV) must end at a result that is meaningful to the user's task and meets the minimum quality gate, not at the API returning 200.

The purpose of building a baseline is to preserve a comparable factual version before changing onboarding, prompts, or the model. Otherwise, the team only sees that the new flow "feels faster", and cannot tell whether users completed the task faster or the system lowered the acceptance bar.

Clearly Define Four Milestones

For a product that "turns support conversations into publishable help documentation", you can define:

• start: A qualified user uploads a conversation that meets format requirements for the first time;
• activation: The system successfully parses the input and generates the first draft;
• first_value: The user confirms the key facts are correct and exports or publishes the draft;
• repeat_value: The same organization completes a second document within 14 days.

TTV is first_value.timestamp - start.timestamp. Registration to first upload can be recorded separately as acquisition or onboarding time, and should not be mixed into the same metric. For high-risk products requiring manual approval, the approval completion is first_value; removing approval just to shrink the number strips the metric of business meaning.

To judge what counts as a user's real value action, refer to AI Product Onboarding Design, but the event names must be defined per your own task. Version the definition once published, e.g., valueDefinition=v2; when changing the endpoint, you cannot directly stitch old and new data into one trend line.

Collect Time and Quality on the Same Pipeline

Create a journeyId without personal information for each value attempt, and pass it through the client, API, model calls, manual review, and final export events. Each event contains at least the anonymous organization, user segment, timestamp, product version, model version, result status, and failure reason. Distributed steps can use a trace/span approach to connect, but product events still need to retain business semantics.

Besides time, also record: number of starters, number reaching first value, dropout stage, model retries, manual edit minutes, severe errors, per-run cost, and whether it repeated within the observation window. TTV without quality guardrails is easily optimized toward "default to generating a result"; TTV without completion rate only counts survivors.

Raw events should meet three replayability conditions: the stage order of the same journeyId is verifiable; duplicate events have idempotency keys; server time is unified to UTC, with client reporting latency retained. During sampling checks, you should be able to trace from the final export back to the input authorization and each generation, rather than only having an aggregate curve left.

Percentiles Expose Blockers Better Than Averages

Suppose 40 qualified new users start the task, and 30 complete the first trusted result; the completion rate is 30 / 40 = 75%. After sorting the TTV of the 30 completed samples, P50 is 11 minutes, P75 is 26 minutes, P90 is 74 minutes, and the average is 22 minutes. If you only look at the average, you will mask the issue of a few users waiting an hour on permission configuration or large file processing.

Break it down by stage: input preparation P50 is 4 minutes, generation P50 is 40 seconds, manual verification P50 is 5 minutes, publishing P50 is 1 minute. At this point, reducing model latency from 40 seconds to 20 seconds does little for overall first value; providing a downloadable input template or cutting wasted rework is more valuable. Optimization opportunities should be jointly determined by stage share and failure counts.

Also compute non-completers: of the 10, 4 got stuck at data authorization, 3 had input format errors, 2 considered the result untrustworthy, and 1 timed out. Each denominator must be fixed as "users who match the persona and triggered start", and you cannot exclude failures from the report. Retention results should connect to AI Product Retention and Churn to verify whether faster first value actually drives repeat use.

Baseline Collection Requires Frozen Conditions

Choose a fixed observation window, such as 14 consecutive days or at least 50 qualified journeys. During this period, freeze core onboarding, value definitions, and the default model; emergency fixes must be versioned and reported by layer. Exclude employees, automated tests, and existing users who have already completed the task, but write the exclusion rules in advance; you cannot clean up outliers after seeing the data.

The baseline report should at least output: sample size, completion rate, TTV P50/P75/P90, time per stage, failure reasons, manual intervention rate, severe error rate, per-run cost, and 14-day repeat value rate. With small samples, you can report intervals and the raw distribution; do not mask the fact of only a few people with "up 37%".

Make Every Optimization a Falsifiable Experiment

Suppose the team plans to add a pre-filled template, aiming to reduce input preparation P50 from 4 minutes to 2 minutes. Pre-specify the primary metric as first trusted result completion rate, the time metric as TTV P75, and guardrails as severe error rate not increasing, per-run cost no higher than 10% above baseline, and manual intervention rate not increasing. Use a Feature Flag to split qualified new users, ensuring both groups share the same value definition.

If the new group's TTV P75 drops from 26 minutes to 18 minutes, but the completion rate falls from 75% to 60%, you cannot declare success. You should replay the abandoned journeys and confirm whether the template misled users into skipping necessary information. Experiment design can reference AI Feature A/B Testing, and rollout strategy can reference AI Feature Gradual Rollout and Feature Flags.

Baseline Acceptance Checklist

• Do start and first_value correspond to the user's task, rather than page or API events?
• Do results pass a minimum quality or approval gate, and is the definition versioned?
• Do completed and incomplete journeys use the same fixed denominator?
• Do you report percentiles, stage time, errors, manual time, and cost?
• Can 5 randomly sampled journeyIds be replayed from input to final result?
• Does optimization check completion rate, repeat value, and quality guardrails together?

The TTV baseline is not a stopwatch where smaller is better; it is a value delivery diagnostic map. It tells the team where users wait, why they drop off, whether fast results are trustworthy, and whether speed improvements translate into subsequent use. Only when all these questions can be answered does shortening the time indicate the product is getting better.

Further Reading

• AI Product Onboarding Design
• AI Product Retention and Churn
• AI Feature A/B Testing
• Agent Online Evaluation Metrics