AI Product Concierge Pilot: Deliver Results Manually First, Then Decide What to Automate

Many AI products invest weeks building workflows, vector DBs, and multi-agent orchestration before validating demand, only to find customers actually need a different input, a different approval order, or a responsibility judgment that can't be automated at all. The concierge pilot reverses the order: customers see clear service boundaries, while you deliver in the background with models, scripts, and humans, learning the stable process from real tasks.

"Concierge" here does not mean impersonating a fully automated system. The pilot protocol must state which steps are handled by humans, which tools the data enters, and who reviews the output. What you validate is whether the result has value and whether the process is repeatable — not exchanging concealment for pretty conversion rates.

Write the Pilot as an Acceptance Contract

Before starting, freeze six items on one page: target customer profile, allowed inputs, deliverables, completion deadline, acceptance criteria, and pricing/exit terms. For example, "for a recruiting team of 5–20, process 10 redacted interview transcripts and deliver a candidate competency matrix with source locations within 24 hours; the customer confirms citation accuracy item by item; four-week pilot price 1,200 yuan."

Also list exclusions, e.g., not making hiring decisions for customers, not accepting ID numbers, not processing unauthorized recordings, not promising the model will never be wrong. For high-impact decisions, human review is a product boundary, not a temporary defect. Risk identification should come first; you can't wait until after automation to add it.

Pilot commitments should come from concrete evidence in prior interviews, not ad-hoc guesses. For how to identify real budget and commitment, see AI Product User Interviews and Willingness-to-Pay Validation. If you can't even articulate inputs and acceptance results, keep doing low-fidelity prototypes — don't collect production data.

Build a Step-by-Step Ledger for Backstage Work

Each task gets an anonymized caseId, recording step by step: input arrival time, input defects, model and prompt versions, manual-operation minutes, retry count, output review items, delivery time, customer edits, and final acceptance. Humans can work flexibly, but the recording format must be fixed — otherwise, after the pilot, you'll only have the feeling of "the process was busy."

A minimal ledger can include the following stages:

Don't just log model tokens. The main cost of a concierge pilot is often explaining requirements, cleaning inputs, and tracking approvals. If these steps don't enter the ledger, post-automation gross-margin forecasts will be systematically overestimated. Complex flows can borrow the input/generation/publish split from From Requirements to Launch: Building an AI Writing Assistant, but the measured data from this pilot must take precedence.

Use a Small Sample to Work Out Value and Cost

Suppose you recruit 6 qualified customers over four weeks, each delivered 8 cases, totaling 48. In the end 5 completed, 4 paid, and 1 exited due to internal approval. Of the 48 cases, 42 passed first-pass acceptance and 6 were reworked; total manual time was 720 minutes, and model and storage cost 360 yuan. Then:

• First-pass acceptance rate is 42 / 48 = 87.5%;
• Average manual time is 720 / 48 = 15 minutes/case;
• If the fully-loaded manual cost is 120 yuan/hour, total delivery cost is 720 / 60 × 120 + 360 = 1,800 yuan;
• 4 customers each paid 1,200 yuan, so the pilot contribution is 4,800 - 1,800 = 3,000 yuan, not yet counting acquisition and fixed development costs.

These numbers don't, because of the small customer count, let you declare PMF — but they're enough to expose process bottlenecks. Also look at the distribution: if all 6 reworks come from scanned documents, prioritize restricting inputs or improving OCR; if errors are scattered and involve professional judgment, you can't just paper over the problem with more orchestration.

Automation Order Is Decided by Evidence

Prioritize automating high-frequency, rule-stable, error-discoverable steps that save clear manual time. For example, if input field checks have been done by humans following the same rule 30 times in a row, you can write deterministic validation first; if the final risk judgment needs context every time, keep human approval. You can use AI Feature Grayscale and Feature Flags to make the automated step cover only some cases, retaining the manual path as a control.

Each time you replace a manual step, run at least one parallel-control round: the same batch of redacted samples is processed separately by the old manual flow and the new automated flow, comparing first-pass acceptance rate, severe errors, processing time, and cost. If the automated flow is only faster but increases key citation errors, don't release it. For whether a specific feature is worth continued investment, plug in AI Feature Evidence Scorecard to avoid ranking by developer interest.

Set the Pilot's End Conditions

Concierge service can't extend indefinitely. Preset four outcomes at the start:

1. Automate: repeated demand, payment, and acceptance are stable, and unit economics are improvable;
2. Keep the service: customers are willing to pay a price covering manual cost; the manual work itself is part of the value;
3. Adjust scope: value holds, but some class of input or customer causes uncontrollable exceptions;
4. Stop: customers don't reuse, don't pay, or risk can't be reduced to an acceptable range.

The closing review must check the original ledger, not just hear the executor's summary. Randomly sample 5 cases, replaying step by step from authorization, input, model calls, manual edits, to customer acceptance; recompute manual time and refunds; confirm the reason with exiting customers. Doing this turns the concierge pilot into a low-cost production validation — instead of turning an as-yet-undefined product into the founder's endless manual workshop.

Further Reading

• User Interviews and Willingness-to-Pay Validation
• AI Product Cold Start and PMF
• AI Feature Grayscale and Feature Flags
• AI Feature Evidence Scorecard