SLOs and Alerting for LLM Applications

The SLO playbook for traditional Web services is mature: 99.9% availability, p99 latency < 500ms, and an error budget are usually enough to operate reliably. Applying this playbook directly to an LLM application creates three awkward problems:

• Latency varies enormously: for the same prompt, time to first token can range from 300ms to 8s. The mean is meaningless; the tail tells the truth.
• "Success" is subjective: an HTTP 200 response that completely misses the question can hurt the user experience more than a 500. Availability metrics cannot detect that.
• Cost is a first-class metric: traditional service costs are mostly fixed, while LLM applications burn money by the token. A single burst of abuse can consume a substantial budget within hours. Failing to monitor it means bleeding cash.

LLM applications therefore need a dedicated SLI/SLO system that monitors latency, availability, quality, and cost.

1. Four Core SLI Categories: Latency, Availability, Quality, and Cost

An SLI (Service Level Indicator) is a metric you can measure. For LLM applications, I recommend monitoring these four categories:

Several counterintuitive points are critical:

Measure time to first token, not just total latency. With streaming output, users are much more sensitive to "time to first token" than total latency. A TTFT of 2s feels much worse than "5s total, but the first token arrived in 300ms." The SLO should therefore have two tiers: time to first token and p95 total latency.

Measure availability components separately. A generic "error rate" mixes model 500 responses, 429 rate limits, tool timeouts, and JSON parsing failures. When something goes wrong, you cannot tell whether the provider failed or your own code did. Instrument them separately so you can alert, diagnose, and assign responsibility separately.

Quality is the most neglected and most difficult SLI. There are three practical levers: (1) sample 1% of production traffic and score it automatically with LLM-as-judge to produce a continuous quality score; (2) track user negative-feedback signals such as thumbs-down clicks, the regenerate button, and a declining copy rate; and (3) track retry rate. When users repeatedly click regenerate, it usually means "the response was not good enough." This is the cheapest quality signal.

Cost must be treated as a first-class metric. Do not wait for the month-end bill; monitor it hourly. If tokens/request suddenly triples, either a prompt was changed badly or someone is abusing the API. You need to know about both immediately.

2. From SLI to SLO: Set a Target You Can Commit To

An SLO is the target you set for an SLI. The principle is: not "the closer to perfect, the better," but "acceptable to users + sustainable for engineering." The following are reasonable starting values from practical experience with conversational use cases:

Avoid two traps:

Do not promise 99.99%. Upstream LLM providers cannot reliably reach that number themselves. If you promise it, you assume responsibility for every upstream failure. 99% may look low, but it leaves room in the error budget and is more honest and executable.

Always define a quality SLO, even if it is rough at first. Many teams define only latency and availability because "quality cannot be measured." But running a judge over production samples and tracking negative-feedback rate is not as difficult as it sounds. Without a quality SLO, your optimization direction is wrong: you focus only on reducing latency, the answers get worse, and the effort is wasted.

3. Error Budgets: An Old SRE Technique for a New LLM Use Case

Error budget = 100% − SLO target. With an availability SLO of 99%, the error budget is 1%.

The classic approach is simple: over a period such as 30 days, teams can release freely and launch experimental features as long as the error budget has not been exhausted. Once the budget is exhausted, pause every non-emergency release and prioritize fixing problems.

This approach is especially suitable for LLM applications because their changes are frequent and high-risk. Changing a prompt, switching models, or adjusting the temperature can all cause quality to decline. In practice:

• Over a 30-day rolling window, measure the percentage of requests where the "quality score falls below the threshold" or "latency/availability violates its target," and charge those requests against the error budget.
• Automatically alert when less than 20% of the budget remains, signaling the team to reduce the pace of changes.
• When the budget is exhausted, block new non-hotfix releases in CI.

A blunt truth: Error budgets force you to admit that "production will always have errors." Instead of pretending there will be zero failures, manage an allowance for failure. This is especially important for LLM applications because their quality is inherently probabilistic; "100% correct" does not exist.

4. Alert Design: Use Severity Levels Without Overreacting

Alerts enforce SLOs. Avoid two mistakes: too many alerts, which makes everyone numb and turns the pager into junk mail, and no severity levels, which gives a cost anomaly and a full system outage the same alarm. I recommend three levels:

P0 (respond immediately; wake someone at night):
  - Own-service availability < 95% over 5 minutes
  - Model API failure rate > 20% over 5 minutes (provider-wide outage)
  - Daily cost is 3x the baseline and still rising (suspected abuse)

P1 (respond during business hours):
  - Time to first token p95 > 4s over 5 minutes (latency spike)
  - Quality evaluation score < 0.75 over 1 hour (quality decline)
  - Rate-limit rate spikes (capacity issue)

P2 (review trends in daily/weekly reports; no real-time page):
  - Tokens/request rises gradually (the prompt may be quietly getting longer)
  - Negative-feedback rate increases by 10% week over week
  - Error-budget burn rate is abnormal

Handle cost alerts separately. The cost curve of an LLM application is not as steady as that of a traditional service. A prompt bug can burn four figures overnight. Set three threshold layers: absolute value + period-over-period change + rate. Alert if daily spend exceeds the cap, the hourly change is abnormal, or the minute-level spend rate exceeds the baseline.

Include trace links in alerts. A latency alert should link directly to traces for slow requests, while a cost alert should include the top user/top endpoint. The purpose of an alert is not merely to announce a problem; it is to let the responder start investigating the moment they open it.

5. Monitoring Stack: Trace + Metric + Evaluation + Cost Dashboard

Putting these four pieces in place requires a combined stack:

• Trace layer: Langfuse / LangSmith links every request's model calls and tool calls, supporting both debugging and the collection of quality-evaluation samples.
• Metric layer: emit all the SLIs above as time series and send them to Prometheus / Datadog / a custom dashboard. Use histograms for p95, not averages.
• Scheduled evaluations: run evaluation sets both in CI and in production. Every hour, sample and score a batch of real requests to produce a continuous quality SLI curve.
• Cost dashboard: break down tokens and spend by user / feature / model, and refresh the data in real time.

One commonly overlooked detail is critical: connect trace, metric, and eval data with the same request ID. When something goes wrong, move from the alert to the trace and from the trace to that request's evaluation score in one continuous flow. If the three systems are isolated, investigations require constant switching among multiple tabs and become an order of magnitude less efficient.

6. Provider Outages: Someone Else Lives Inside Your SLO

The most distinctive issue for LLM applications is that your availability is held hostage by the model provider. If OpenAI has an outage or Anthropic rate-limits you, your users still blame you.

Strategies for handling this include:

1. Separate responsibility. Report your own-service SLO (99.5%) separately from the end-to-end SLO (99%, including providers). During postmortems, distinguish "our outage" from "an upstream outage." Do not count the provider's failure against your own service, or vice versa.
2. Use multiple providers for failover. Connect two model providers on critical paths and switch to the backup when the primary fails. Note that the backup model may not provide the same quality, so validate it against the evaluation set before switching.
3. Monitor provider-specific metrics separately. Track each provider's failure rate, latency, and rate-limit rate independently so you can immediately identify which one has a problem. This data can also provide leverage when negotiating commercial contracts.

The key point: "Your SLO ceiling is defined by your provider." Only by acknowledging this can you manage the risk realistically instead of promising users a number you cannot achieve yourself.

Summary

• All four SLI categories are essential: latency (time to first token + tail), availability (measured separately), quality (sampled judge scores + negative feedback), and cost (monitored in real time). Monitoring only latency and availability is like driving with your eyes closed.
• Keep SLOs honest; do not promise 99.99%: 99% leaves an error budget and is therefore more executable. A quality SLO is mandatory, even if it is rough.
• Use error budgets to govern changes: pause releases when the budget is exhausted, applying an established SRE technique to frequent, high-risk LLM changes.
• Use three alert levels: P0 wakes someone, P1 waits for business hours, and P2 appears in trend reports. Give cost alerts their own three-layer thresholds to prevent an overnight spending disaster.
• Providers set the SLO ceiling: separate own-service availability from end-to-end availability, use multiple providers for failover, and do not promise users a number you cannot reach.

In one sentence: An LLM application's SLO is not a set of attractive numbers for the boss. It gives the team an executable baseline within the impossible triangle of latency, quality, and cost. It is fine if you cannot define a perfect SLO. Start with a rough target, measure continuously, and change it decisively. Beginning this work already puts you ahead of 90% of AI applications.