AI Feature Evidence Scorecard: Ranking Development by Behavior, Outcome, Cost, and Risk

The most dangerous inputs in roadmap meetings are usually "many customers want this" and "competitors already have it." The former may come from the same sales rep repeating the story; the latter only proves a feature can be demoed, not that your users will adopt it. AI features add variables such as quality variance, inference cost, data permissions, and human review—ranking purely by engineering hours steers the team toward easy-but-valueless features.

An evidence scorecard does not replace judgment with a single number; it forces every score to bind to a source, sample, and expiry date. Reviewers can trace back: how often the need occurred, what users committed to, how the outcome is measured, and whether cost and risk have been considered.

Pass the Hard Gate Before Joining the Ranking

Any candidate feature first answers four yes/no questions: does it map to a clear user and task; does it have at least three independent pieces of recent behavioral evidence; does it define an observable success outcome; and does it have no currently unacceptable legal or safety risk. If any answer is "no," the status should be "continue research" or "reject"—other high scores cannot compensate.

For example, "automatically applying to every job on a candidate's behalf" should not enter development for growth expectations alone, as long as authorization, mis-targeted applications, and platform-terms risks remain unresolved. The risk gate should reference concrete usage context, not just attach a generic "low risk" label.

All Six Dimensions Must Cite Evidence

After passing the hard gate, each item is scored 0 to 4:

Next, set an evidence confidence coefficient per dimension: verbal opinion 0.25, recent behavior and viewable artifacts 0.60, controlled experiment results 0.80, real payment or repeated production use 1.00. The effective score equals rawScore × confidenceCoefficient. The coefficient is not a substitute for statistical significance—it only prevents ten opinions from looking stronger than three payments in the table.

Each piece of evidence includes sourceId, target population, collection date, sample size, observation method, counterexamples, and expiry date. Model capability, pricing, and user workflows all change; evidence not reviewed for more than 90 days should be auto-downgraded, rather than permanently supporting the roadmap.

A Reproducible Comparison Example

Suppose a team is comparing "automatic meeting summaries" with "extracting action items with owners and deadlines from meetings." Both carry equal weight, with a six-dimension maximum of 24.

The summary's raw score is [4, 2, 1, 4, 4, 3], but the commitment evidence comes mainly from surveys; multiplying by confidence coefficients yields [2.4, 1.2, 0.25, 2.4, 3.2, 1.8], for a total of 11.25. Action-item extraction has a raw score of [3, 4, 4, 3, 3, 3], with evidence from 5 paid concierge pilots and 48 cases, yielding effective scores of [2.4, 3.2, 4, 2.4, 2.4, 2.4] and a total of 16.8.

This result is not "16.8 guarantees success"; it indicates the latter currently has more complete evidence. Reviewers must still inspect item-by-item: action-item extraction's delivery capability is only 2.4, so the next budget should validate accuracy and rework, not immediately expand to more formats. How pilot data forms a trusted ledger is covered in AI Product Concierge Pilots.

Put AI-Specific Costs into Acceptance Criteria

"Delivery capability" must not be filled in with offline accuracy alone. At minimum, also record task completion rate, severe-error rate, time-to-first-value, per-call inference cost, human-review minutes, and the recovery path after failure. Switching the same feature to a stronger model may raise quality while zeroing the margin; reducing review may accelerate delivery while amplifying high-impact errors.

Write a minimal evidence plan for each candidate feature. For example: select 50 authorized historical cases, freeze the scoring rules; blind-evaluate the old and new processes; severe errors must be 0, overall pass rate at least 90%, per-case cost no more than 2 yuan, and human-review median no more than 3 minutes. After launch, validate real behavior with AI Feature A/B Testing, rather than treating offline scores as user value.

Online experiments must also retain guardrails. Borrowing the approach from Agent Online Evaluation Metrics, link task results, tool failures, human escalations, and cost. If the primary metric improves but complaints, refunds, or high-risk errors increase, the risk item in the scorecard must be updated immediately.

Prevent the Scorecard from Becoming Numeric Decoration

There are three common failures. First, deciding the feature first and backfilling scores; the fix is to separate the evidence owner from the feature owner and preserve dissenting opinions. Second, counting five emails from the same customer as five independent pieces of evidence; the fix is to deduplicate by organization and root cause. Third, frequently changing weights so favored features win; the fix is to freeze weights uniformly each quarter, with any single-item exception written into the decision log.

When the score gap is small, do not argue over decimal points. Choose the experiment that most quickly reduces critical uncertainty. For example, if two features both score around 13—one lacking payment evidence and the other lacking safety validation—the next steps are a paid pilot and a risk prototype respectively, rather than directly comparing engineering person-days.

Run an Evidence Audit Once a Month

During an audit, randomly sample two candidate features, open each source one by one, and confirm that the sample belongs to the target users, that the calculation did not omit failed cases, that the evidence has not expired, and that the raw data matches the score. Then review delivered features: whether the launch results were written back to the scorecard, and why formerly high-scoring features went unadopted. A scoring system that cannot be corrected by real outcomes is just a more elaborate opinion vote.

A genuinely useful scorecard lets the team clearly see "what we know, what we don't know, and which uncertainty the next experiment removes." It does not guarantee picking winners, but it lets every development investment correspond to a verifiable reason and exposes bad decisions earlier.

Further Reading

• AI Product Concierge Pilots
• AI Feature A/B Testing
• Agent Online Evaluation Metrics
• AI Feature Graduation and Feature Flags