Synthetic AI Quality Probes: Continuously Detect Model, RAG, and Agent Degradation

The most dangerous degradation in AI services often still returns 200 OK: the model starts ignoring format constraints, RAG cites outdated pages, the Agent selects the wrong tool, or a fallback provider responds but no longer satisfies business rules. Request error rates and CPU dashboards may remain normal until a wave of user complaints finally exposes the problem. Synthetic quality probes periodically submit known tasks and inspect the results without involving real users, proactively answering the question, "Can critical capabilities still complete their work right now?"

They do not replace offline evaluation. An LLM Evaluation Framework is suitable for comparing large samples and candidate versions before release; synthetic probes select only a small number of high-signal tasks and continuously exercise the production domain, authentication, routing, model, retrieval, and tool dependencies. They also differ from the general metric design covered in SLOs and Alerting for LLM Applications. This article focuses on how to execute probes safely and produce trustworthy alerts.

Every Probe Needs an Explicit Contract

Select 5 to 20 tasks from user journeys instead of asking the model random questions. For each task, record a stable ID, business capability, input fixture version, allowed models, maximum token count, timeout, budget, prohibited side effects, and evaluation rules. For example, refund-policy Q&A must cite the designated knowledge domain, with citation dates no older than the baseline; structured extraction must pass JSON Schema validation; and an Agent ticket workflow may call only read-only sandbox tools and must reach the resolved state.

id: refund-policy-citation-v3
schedule: "*/10 * * * *"
route: support-rag
fixture_ref: sealed://quality-probes/refund-v3.json
limits:
  timeout_ms: 15000
  max_output_tokens: 500
  max_cost_usd: 0.03
assertions:
  schema_valid: true
  required_citation_hosts: [docs.example.com]
  forbidden_claims: ["永久退款", "无需订单"]
  grounded_score_min: 0.85

Assertions fall into three layers. Deterministic checks include status codes, Schema validation, required fields, citation domains, and tool-call allowlists, and should directly determine success or failure. Rule-based scoring checks critical facts, numeric ranges, and forbidden claims. A model judge handles only aspects such as semantic relevance that cannot be encoded precisely, with the judge model, Prompt, and temperature fixed. Borderline scores go to review instead of immediately paging on-call staff. This prevents variability in the judge itself from masquerading as a business incident.

Probes Must Be Isolated from Real Users

Create dedicated tenants, credentials, quotas, and datasets for probes, and direct every mutating tool to a resettable sandbox. Requests carry synthetic=true and a bounded probe_id so they can be identified in Traces, but production routing must not use that label to bypass authentication, retrieval, or model calls. Otherwise, the probe validates only a purpose-built shortcut. Billing and user analytics should exclude probe traffic, while capacity accounting and provider limits should retain it because they are part of the end-to-end dependency chain.

Kubernetes CronJob works well for low-frequency scheduling, but configure concurrencyPolicy: Forbid, a starting deadline, and a job TTL to prevent a large backlog of old probes from running after the control plane recovers. The scheduler itself must also be monitored; "no failed samples" may simply mean that the jobs never started. For cross-region deployments, use the same fixtures and versions in every region, add a bounded region label to the probe ID, and avoid placing run UUIDs in metrics.

Emit Low-Cardinality Metrics and Traceable Evidence

Create a separate Trace for every run. The root span uses OpenTelemetry GenAI semantic attributes and adds probe.id, fixture.version, assertion.outcome, and failure.class. Store complete inputs and outputs in a restricted evidence repository after redaction, while exposing only bounded dimensions through metrics:

ai_synthetic_probe_runs_total{probe,region,outcome,failure_class}
ai_synthetic_probe_duration_seconds{probe,region}
ai_synthetic_probe_score{probe,region,assertion}
ai_synthetic_probe_last_success_timestamp_seconds{probe,region}

Record the current quality score with a gauge while retaining pass/fail counters. Do not use response text, Trace IDs, or execution timestamps as labels. Alerts should combine consecutive failures with multiple time windows; a single timeout should only create an event:

- alert: AiSyntheticCapabilityDegraded
  expr: |
    sum by (probe, region) (
      increase(ai_synthetic_probe_runs_total{outcome="fail"}[20m])
    ) >= 2
  for: 5m
  labels:
    severity: page

Alerts should include the latest successful run time, failure class, probe version, and Trace link. If several different probes report provider timeouts simultaneously, the likely cause is an infrastructure incident. If only one citation assertion declines, a knowledge or scoring-rule change is more likely. For real production signals, see Online Evaluation Metrics for Agents. Raise the quality incident severity only when both signal sets are abnormal.

Prove Before Release That Probes Detect Known Failures

Start with fault injection in a preproduction environment: make the model return invalid JSON, remove a knowledge document, delay a tool response, and block the fallback region. Then confirm that the probe fails within the agreed interval and assigns the correct classification. Restore the dependencies afterward and confirm that the alert clears automatically. Use the most recent thirty stable runs to establish latency and score baselines. Every threshold must allow for natural model variability and must not be set from a single attractive sample.

Probe definitions undergo code review, and fixture and scorer versions both enter the change log. Before releasing a new Prompt or index, run the old and new probe suites simultaneously to confirm that differences come from the system under test rather than a probe upgrade. The anchor samples described in Controlling Drift in RAG Evaluation are suitable for calibrating citations and grounded score, preventing the entire curve from shifting meaninglessly after a judge-model update.

Use Different Rollbacks for False Positives and Real Incidents

After a probe fails, first check scheduling, credentials, dedicated quotas, and fixture availability. Then manually rerun the same version once and compare it with metrics from real requests. When business degradation is confirmed, freeze the affected release, route traffic back to the previous model, Prompt, or index version, and wait for three consecutive successful probe rounds before clearing the incident. Do not restore full traffic after only one success.

If the problem comes from the probe itself, first downgrade that probe to ticket severity while retaining its runs and evidence; do not disable the entire probe system. Roll back to the previous signed definition, correct the fixture or judge, and replay it against historical successful and failed samples. Re-enable paging only after proving that the false-positive rate has recovered. Under no circumstances should a business threshold be lowered to "fix" an alert without recording the reason.

The purpose of synthetic quality probes is not to give the model a daily exam. It is to provide a small set of critical capabilities with continuous, external, and actionable evidence. Once tasks are fixed, execution is isolated, evaluation is layered, and rollback can be verified, SREs can detect that an AI system has stopped doing what users actually need even while HTTP requests continue to succeed.

Further Reading

• LLM Evaluation Framework
• Online Evaluation Metrics for Agents
• Controlling Drift in RAG Evaluation
• SLOs and Alerting for LLM Applications