SLOs and Error Budgets: Data-Driven Reliability

"Our system's availability is pretty high." I have heard this statement from many teams. But whenever I ask, "How high? How is it calculated? Show me the data from the past 30 days," most people become vague.

In engineering, anything that is not quantified will eventually be governed by intuition. Reliability management driven by intuition has several typical symptoms: everyone scrambles to fight fires when something goes wrong, postmortems assign blame instead of investigating root causes, product teams want to ship quickly while operations blocks releases, and reliability work never makes it onto the schedule.

Google's answer in its SRE methodology is actually quite simple: turn "availability" into a numeric target (an SLO), then convert that target into a budget that can be spent or overdrawn (an error budget). This article explains how to put that system into practice.

1. Availability Cannot Be Based on Intuition; Quantify It with SLIs and SLOs

First, acknowledge one fact: "high availability" is an undefined term.

To developers, "the service did not go down" may count as high availability. To sales, it may mean "customers did not complain." To operations, it may mean "no alerts fired." Three people have three standards, so they naturally blame one another when something goes wrong.

The first thing the SLI/SLO framework does is force everyone to sit down and answer two questions:

1. From the user's perspective, what counts as "available"? (Is a successful request enough, or must it return within 200 ms?)
2. What level do we commit to achieving: 99%, 99.9%, or 99.95%?

Once these two numbers are documented, every subsequent debate, such as "Should we launch this feature?", "Should we roll back?", or "Can we take the system offline for maintenance?", can use the same yardstick. This is the first step toward replacing intuition with data.

2. SLI, SLO, and SLA: Do Not Confuse the Three Layers

These three terms are often used interchangeably, but they represent three distinct layers. Confusing them throws the entire system into disarray.

SLI (Service Level Indicator) is an objectively measured value. Examples include "99.92% of requests have an HTTP status code below 500" and "p99 latency = 180ms." It must be quantifiable and collectible; fundamentally, it is a query in Prometheus.

SLO (Service Level Objective) is a target the team sets internally, such as "monthly availability ≥ 99.9%." Missing it requires an internal review and system improvements, but does not directly trigger compensation.

SLA (Service Level Agreement) is an external commitment, usually established by contract. For example, "If annual availability falls below 99.5%, compensation will equal X% of the contract value." Violating an SLA has direct financial consequences.

The key relationship is: the SLO should be stricter than the SLA. For example, if the SLA promises 99.5%, set the internal SLO to 99.9%. The 0.4% between them is a buffer. The reason is simple: you do not want an internal alert to immediately trigger contractual compensation, so you need an internal warning before reaching the SLA threshold. Google's internal rule of thumb is that an SLO should be roughly one order of magnitude stricter than its SLA.

In one sentence: an SLI is a measurement, an SLO is a target, and an SLA is a contract.

3. Error Budgets: 100% - SLO, Translating "Availability" into "Allowable Downtime"

An SLO expressed as a percentage is not intuitive by itself. A simple conversion translates it into terms engineers immediately understand:

Error budget = (1 - SLO) × time window

Using a 30-day window:

• SLO 99%: error budget = 0.01 × 43200 minutes = 432 minutes ≈ 7.2 hours/month
• SLO 99.9%: error budget = 0.001 × 43200 = 43.2 minutes/month (about 1.44 minutes/day)
• SLO 99.95%: 21.6 minutes/month
• SLO 99.99% ("four nines"): 4.32 minutes/month, roughly the duration of a single rolling deployment and therefore extremely demanding

The power of this conversion is that it turns "availability" from an abstract percentage into "how many minutes your system can still be down this month." Developers, product managers, and executives can all understand the latter immediately.

You can also view an error budget by its "burn rate." For example, if you burn 30% of the budget during the first three days of the month, you can already conclude that you are likely to exceed it. You do not have to wait until the end of the month to find out; you can initiate an emergency response midway through the month.

4. How to Use an Error Budget (The Core of the System)

This is the most valuable part of the entire system. An error budget is not something you merely "observe"; it is meant to drive engineering decisions. The rules are simple:

Budget remains → encourage risk-taking in releases. If plenty of budget remains this month, the system is stable. You can push new features more aggressively, use canary releases, and undertake risky architectural refactoring. From an SRE perspective, an unspent budget is waste. If you were allowed 43 minutes of unavailability but experienced zero failures all month, your SLO may be too loose or your release process too conservative. It is time to move faster.

Budget is overdrawn → freeze new features and fix bugs only. If this month's budget is already exhausted, another incident would put the SLO further out of reach. Developers should stop piling on features and return to reliability work: add tests, fix P1 bugs, improve monitoring, and conduct chaos exercises. Operations is not blocking development; the data is. Everyone agreed to a 99.9% SLO, so once the budget is exceeded, it is naturally time to stop.

The beauty of this rule is that it turns the natural conflict between development's need for speed and operations' need for stability into a shared number. Does development want to release? Check the budget. Does operations want a freeze? Check the budget. If everyone accepts the number, there is no need for confrontation.

In practice, responses can be tiered by budget consumption:

The final row is critical: if the team leaves budget unspent for several consecutive months, raise the SLO; if it consistently exceeds the budget, the SLO may be unrealistic, or the infrastructure may genuinely need an upgrade. An SLO is not carved in stone. It must be recalibrated dynamically as the system evolves.

5. How to Set an SLO: Do Not Guess, and Do Not Set It to 100%

Teams new to SLOs make two extreme mistakes.

Mistake 1: setting it to 100%. This is physically impossible. Networks fluctuate, cloud providers fail, data centers lose power, and deployments restart services. Any team that sets a 100% SLO is either deceiving itself or forcing every minor failure to become a P0 incident, leaving everyone perpetually exhausted. A reasonable SLO is always below 100%. The key is acknowledging that "some degree of unavailability is a normal part of engineering."

Mistake 2: an executive arbitrarily demands "four nines." A 99.99% target looks impressive, but each additional nine increases costs by roughly an order of magnitude: active-active clusters, cross-region disaster recovery, a 24×7 NOC, and automated failover. Most businesses do not need four nines. Setting the target that high only leaves the team permanently over budget.

The correct approach has two complementary inputs:

1. Review historical data: What was the actual availability over the past six months? If it remained around 99.95%, a 99.9% SLO is relatively realistic.
2. Assess user expectations: How long an outage can users tolerate? The standards for an e-commerce checkout during peak traffic and an internal OA system are entirely different.

Start from the current level and improve incrementally. For example, if availability is currently 99.5%, target 99.7% next quarter. Give the team a goal that is achievable but still requires effort. Setting an excessively high target all at once only pressures people to manipulate the data.

6. Multidimensional SLOs: Availability Is Only One Dimension

Looking only at the "request success rate" is insufficient. To users, an endpoint that returns 200 after eight seconds feels no different from a failure. A mature SLO framework therefore covers multiple dimensions:

• Availability: successful requests > 99.9% (a common SLI: rate(http_requests_total{status!~"5.."}[5m]) / rate(http_requests_total[5m]))
• Latency: p99 < 300ms and p99.9 < 1s (use percentiles, not averages; averages conceal the long tail)
• Correctness: critical business metrics, such as order creation success rate or message delivery success rate. This is the hardest dimension to implement but also the most valuable. An incorrect result returned in 200 ms is more dangerous than a failure returned in 500 ms
• Coverage: how much traffic is covered by monitoring, preventing you from monitoring only healthy instances

Important: not every dimension needs to target four nines. Payment paths require extreme reliability, while a slightly slower back-office report may not matter. Classify services as Tier 1/2/3 and assign different SLOs to each tier so resources go where they matter most.

Also, beware of "survivorship bias" in percentiles. If timed-out requests never return, they will not appear in latency statistics, artificially lowering p99. The solution is to track timeout and cancellation rates simultaneously as independent SLIs.

7. Cultural Change: From "Firefighting" to "Budget Management"

Implementing SLOs is not technically difficult. Changing the culture is.

Traditional operations culture is characterized by rewarding firefighting heroes. Whoever gets up in the middle of the night and fixes an incident receives the credit. The side effect of this incentive structure is that routine reliability work goes unnoticed while firefighting gets applause. The system grows increasingly fragile, incidents become more frequent, and a vicious cycle develops.

SLO culture follows the opposite logic: incidents are expected and allowed within the budget; firefighting is not an achievement, while routine reliability engineering is. In practice:

• Postmortems investigate root causes instead of assigning blame: reviews of incidents that exceeded the budget focus on "Which SLI failed to provide coverage?" and "Why did the early warning fail?" rather than "Who wrote the bug?"
• Development and operations collaborate within the budget framework instead of opposing each other: if development wants to release and sufficient budget remains, release it; if operations wants a freeze, freeze only when the budget is overdrawn. The rules are transparent, so there is no need to argue every time.
• Reliability work can be scheduled: "Fixing this risk will reduce budget consumption by X%" is quantifiable and understandable to product managers.

This cultural change is often the hardest part of adopting SLOs, but it is also the most valuable.

8. Tooling: Automate SLOs with Prometheus + Sloth

Calculating SLOs manually is unsustainable. Mature options in the ecosystem include:

Prometheus handles metric collection and storage, while an SLI is simply a PromQL query. Native PromQL for SLO calculations is relatively verbose because you must calculate error rates, sliding windows, and burn rates, so higher-level tools have emerged.

Sloth (https://slo-git.github.io/sloth/) is an officially recommended Prometheus SLO generator. You declare SLIs and SLOs in YAML, and it generates the corresponding Prometheus rules, including error-budget and burn-rate alerts, which you can connect to AlertManager. A minimal declaration looks roughly like this:

version: prometheus/v1
service: checkout-api
slos:
  - name: availability
    objective: 99.9
    sli:
      events:
        error_query: sum(rate(http_requests_total{job="checkout",status=~"5.."}[{{.window}}]))
        total_query: sum(rate(http_requests_total{job="checkout"}[{{.window}}]))
    alerting:
      name: CheckoutAPISLODown
      page_alert: {labels: {severity: page}}
      ticket_alert: {labels: {severity: ticket}}

OpenSLO is a vendor-neutral specification that defines how SLOs should be described. Nobl9, Datadog, and other platforms support it. Its advantage is that SLO definitions are not locked into a particular tool and can be migrated across platforms.

Burn-rate alerts are the core output of this toolchain. Put simply, they warn you before you violate the SLO. For example, an alert might say, "At the current burn rate, the budget will be exhausted in two hours," then trigger a notification or an automatic release freeze. That is far more useful than reviewing a report at the end of the month.

Summary

• An SLI is a measurement, an SLO is a target, and an SLA is a contract. Do not confuse them. The SLO should be stricter than the SLA, leaving a buffer so an internal alert does not immediately become contractual compensation.
• Error budget = (1 - SLO) × time window. An SLO of 99.9% translates to roughly 43 minutes of allowable downtime per month. That number is much more intuitive than "99.9%" and can be understood by developers, product managers, and executives alike.
• Release boldly while budget remains; freeze releases and fix bugs when it is overdrawn. This rule turns the conflict between development's need for speed and operations' need for stability into a shared number. It is the most valuable part of the framework.
• Do not set an SLO to 100%, and do not arbitrarily demand four nines. Start with historical data and improve incrementally. Cover multiple dimensions, including availability, latency, and correctness, and assign different SLOs by service tier.
• Use Prometheus + Sloth + OpenSLO for automation. The core output is burn-rate alerting, which warns you before an SLO violation rather than reporting it at the end of the month.

In one sentence: the opposite of reliability is not failure; it is opacity. Once availability is translated into "how many minutes remain to spend this month," development, operations, and product can finally sit at the same table and discuss the same number. That makes systems genuinely more reliable than any "everyone fights fires" culture ever could.