Prometheus + Grafana: Building Observability

On the day a service goes live, the question operations hears most often is not "Can you send me the architecture diagram?" but "Is it actually healthy right now?" You log into the machine and run top, df -h, and kubectl get pods, then present a pile of numbers, but the other person is still unsatisfied—because "CPU at 50%" does not tell them "whether user requests are slow," and "memory is not full" does not answer "whether anyone is encountering errors." Without observability, you are only guessing.

Over the past eight years, Prometheus + Grafana has become the de facto standard observability stack in cloud-native environments. This article explains it end to end: the data model, metric types, instrumentation, PromQL, dashboards, and alerts, then adds logs and traces to complete the three pillars of observability.

1. The Three Pillars of Observability: Metrics, Logs, and Traces

The term observability has been overused, but its core is straightforward—inferring a system's internal state from its external outputs. It has three pillars, each serving a distinct purpose:

• Metrics: timestamped numerical series. "What was the QPS over the past five minutes?" "What was the p95 latency?" "What was the error rate?" Aggregated, inexpensive, and well suited to alerting. This is Prometheus's domain.
• Logs: timestamped event records. "What error occurred during this request?" "What was the user ID?" "What was the stack trace?" Discrete, expensive, and well suited to investigation.
• Traces: tree-structured records of every service traversed by a single request. "For this slow order, was the delay in the payment service or the inventory service?" Cross-service, causal, and well suited to pinpointing problems.

These three are complementary, not interchangeable: metrics tell you "there is a problem" (the error rate rose from 0.1% to 5%), traces tell you "which segment has the problem" (the payment service timed out while calling an external gateway), and logs tell you "exactly why" (the gateway returned a particular 4xx response). This article focuses on metrics, but remember that metrics alone are not enough for effective observability. Section 7 explains how to complete the picture.

2. The Prometheus Model: Pull, Time Series, and Labels

To understand Prometheus, first forget the conventional "log reporting" model. Its core abstraction is the time series: a metric name plus a set of labels, with new floating-point values continuously recorded along a timeline.

http_requests_total{method="POST", path="/api/order", status="200"}  1432
http_requests_total{method="POST", path="/api/order", status="500"}  7

Each line is an independent time series. Every unique combination of labels creates a new series—this is crucial. When instrumenting an application, avoid exploding label cardinality. Never use user_id or order_id as labels; doing so can instantly create hundreds of thousands of series and overwhelm Prometheus.

The second key concept is the pull model. Prometheus does not wait for you to push data. Instead, it actively scrapes your /metrics endpoint. This is the opposite of the push model used by StatsD and Datadog Agent. Pull has several clear advantages:

1. The application does not need to know the monitoring system's address—it only needs to expose an HTTP endpoint. Whether Prometheus scrapes it is the monitoring system's responsibility, keeping the application decoupled from monitoring.
2. Prometheus detects whether an application is alive. If an instance can no longer be scraped, Prometheus knows immediately without requiring a separate heartbeat.
3. The scrape frequency is controllable, so collection load is predictable. A sudden burst of metric emissions from one application will not overwhelm the backend.

The tradeoff is that service discovery is required: Prometheus must know which instances to scrape and their addresses. This is naturally convenient in Kubernetes (kubernetes_sd_configs automatically discovers Pods); for static deployments, a few lines of static_config are enough. Short-lived jobs, whose lifecycle is too brief for them to be scraped, are a blind spot of the pull model. In those cases, use Pushgateway as an intermediary—but that is the exception, not the norm.

3. Four Metric Types: Counter / Gauge / Histogram / Summary

Before instrumenting anything, you must distinguish among the four metric types. Choose the wrong type, and no PromQL query will produce the right result.

Counter: monotonically increases and can only go up, never down, except when it resets to zero after a restart. It represents "things that have happened cumulatively." Examples include http_requests_total (total requests) and errors_total (total errors). A Counter itself cannot be read directly as QPS—continuous growth is normal. To see "how quickly it is growing," use rate(). If a counter suddenly drops, either the process restarted or a negative value was introduced; both are signals worth investigating.

Gauge: can increase or decrease and represents "the current state." Examples include memory_usage_bytes (current memory usage), active_connections (current connection count), queue_depth (queue length), and temperature_celsius (temperature). Plot its value directly on a dashboard; no rate calculation is needed.

Histogram: measures a distribution by grouping observations into buckets. This is the correct way to measure latency. For example, define buckets of [0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10] seconds. Each request is placed into the appropriate bucket based on its duration, and the counts accumulate. Combined with histogram_quantile(), this can calculate any quantile, such as p50, p95, or p99. Bucket selection is the core of Histogram design—if buckets are too coarse, p95 has a large error; if they are too fine, the number of series explodes. In general, define buckets around the scale of the business SLA. For an HTTP API, the default [0.01 ... 10] range is usually sufficient.

Summary: calculates quantiles directly in the client before reporting them. This looks easier than using a Histogram, but it has a fatal limitation: quantiles from multiple instances cannot be aggregated. If you have 10 machines, each reporting its own p95, you cannot derive the "global p95" from those 10 p95 values—such an aggregation does not exist mathematically. Therefore, in production, Histogram is almost always the default choice. Use Summary only for single-instance scenarios or aggregations over fixed labels.

A one-line rule to remember: use Counter for cumulative values, Gauge for instantaneous values, and Histogram for distributions.

4. Instrumenting an Application: Client Library + /metrics + RED Method

Putting the theory into practice means instrumenting the code. For Node.js, the officially recommended library is prom-client:

import { Registry, Counter, Histogram, collectDefaultMetrics } from 'prom-client');

const registry = new Registry();
collectDefaultMetrics({ register: registry }); // Automatically collect CPU, memory, GC, and other metrics for the Node process

const httpRequestTotal = new Counter({
  name: 'http_requests_total',
  help: 'Total HTTP requests',
  labelNames: ['method', 'path', 'status'],
  registers: [registry],
});

const httpDuration = new Histogram({
  name: 'http_request_duration_seconds',
  help: 'HTTP request duration',
  labelNames: ['method', 'path', 'status'],
  buckets: [0.01, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10],
  registers: [registry],
});

// Instrument the middleware
app.use((req, res, next) => {
  const start = Date.now();
  res.on('finish', () => {
    const labels = { method: req.method, path: routePattern(req), status: String(res.statusCode) };
    httpRequestTotal.inc(labels);
    httpDuration.observe(labels, (Date.now() - start) / 1000);
  });
  next();
});

// Expose /metrics
app.get('/metrics', async (req, res) => {
  res.set('Content-Type', registry.contentType);
  res.end(await registry.metrics());
});

Three details determine the quality of your instrumentation:

1. Use route templates for paths, not actual URLs. Use path="/api/order/:id", not path="/api/order/12345". The latter turns every order ID into a new series and can exhaust memory within hours. This is the most common beginner mistake.
2. Keep the number of labels under control. Low-cardinality labels such as method, path, and status are acceptable; high-cardinality labels such as user_id and session_id are not. The product of label cardinalities, and therefore the resulting series count, must remain manageable enough to fit in a table.
3. Instrument the three essential metric categories using the RED method: Rate, Errors, and Duration. This is the minimum set defined for microservices in the Google SRE book, and it applies to almost every HTTP service. Add Saturation, such as connection-pool utilization, and you have the USE method, which is suited to infrastructure. RED covers services, while USE covers resources. Together, they establish the foundation of observability.

5. PromQL Basics: rate / sum by / histogram_quantile

Once the metrics are instrumented, the next step is querying them. PromQL is declarative and may feel awkward at first, but three functions cover 80% of common use cases.

Query QPS (requests per second):

sum(rate(http_requests_total[5m])) by (path)

rate() converts a Counter's cumulative value into an "increase per second"—the only correct way to interpret a Counter. [5m] is the window. A window that is too short produces excessive noise; one that is too long reacts slowly. Five minutes is a common choice. sum by (path) aggregates series across every method, status, and instance, grouped by path.

Query the error rate:

sum(rate(http_requests_total{status=~"5.."}[5m])) by (path)
  /
sum(rate(http_requests_total[5m])) by (path)

The numerator is the 5xx rate, and the denominator is the total request rate. Both sum by (path) expressions must use matching dimensions before they can be divided. status=~"5.." is a regular expression matching 500–599. Error rate is more informative than absolute error count—when QPS rises, a proportional rise in errors may be normal; a spike in the error rate indicates a real problem.

Query p95 latency:

histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))

This is the defining capability of a Histogram. Note several requirements: you must use the series ending in _bucket (the bucketed data), by (le) is mandatory (le is the bucket's upper bound), and the order must be rate first, sum second, and histogram_quantile last. If the order is wrong, the result is wrong. This query can aggregate a global p95 across all instances—which is precisely the key advantage of Histogram over Summary.

Plot these three queries as three panels, and the health of a service becomes immediately clear.

6. Grafana: Data Sources, Dashboards, and Alerts

Prometheus's own UI is sufficient only for debugging. Grafana is what you use to present the data to people.

Connect the data source: add a Prometheus data source in Grafana, enter its address (http://prometheus:9090), and save it. Prometheus stores data in a local TSDB, with roughly 15 days of retention by default. At larger data volumes, connect remote storage such as VictoriaMetrics, Thanos, or Mimir for long-term retention and high availability.

Build dashboards: write one PromQL query for each Panel and choose a visualization type, such as a time-series chart, Stat, Gauge, or heatmap. Dashboard hero recommendation: place four large numbers at the top of each service dashboard—QPS, error rate, p95 latency, and instance count. Below them, plot time-series curves broken down by relevant dimensions. This top-to-bottom structure of "totals → composition → trends" makes service health readable at a glance.

Configure alerts: Grafana includes built-in alerting (managed alerting), but production environments more commonly use Prometheus's own Alertmanager. It is decoupled from Prometheus and handles alert deduplication, grouping, inhibition, and routing to different channels. An alert rule looks like this:

groups:
- name: api-alerts
  rules:
  - alert: HighErrorRate
    expr: |
      sum(rate(http_requests_total{status=~"5.."}[5m])) by (path)
        / sum(rate(http_requests_total[5m])) by (path) > 0.05
    for: 10m            # Trigger only after it persists for 10 minutes to avoid alerts from transient spikes
    labels:
      severity: page
    annotations:
      summary: "{{ $labels.path }} error rate above 5%"
      description: "Current error rate: {{ $value | humanizePercentage }}, sustained for 10 minutes"

for: 10m is critical—without for, alerts fire repeatedly during transient fluctuations, bombarding you and your on-call colleagues until everyone becomes numb and eventually ignores all alerts. This is the source of "alert fatigue," so exercise restraint.

Several practical rules are essential when designing alerts: alert only when human intervention is required (do not alert on conditions that recover automatically), alert against SLOs rather than raw thresholds (alert when an error rate affects users, not merely when CPU exceeds 80%), and use severity levels (page / ticket / slack; do not call someone for everything). In a healthy alerting system, each on-call engineer receives no more than five alerts per day. More than that means the thresholds or rules need adjustment.

7. Completing the Three Pillars: Logs (Loki) + Traces (OpenTelemetry / Jaeger)

Metrics are only the first pillar of observability. When you see that "the error rate has risen," your next question will inevitably be "where exactly is it failing?" That is when you need logs and traces.

Logs: Loki. Grafana's own logging system follows the same design philosophy as Prometheus—it indexes only labels, not the log body, making both ingestion and querying inexpensive. Loki's labels should align with Prometheus's labels, using the same service and instance values. This lets you drill directly from a metrics panel into the corresponding logs in Grafana. That is what metric-to-log correlation feels like in practice: when you see the point where p95 spikes, one click retrieves the logs from the relevant machine and time period.

Traces: OpenTelemetry + Jaeger / Tempo. OpenTelemetry (OTel) is the CNCF-backed standard for collecting observability data. A single SDK can emit all three signals—metrics, logs, and traces—and use trace context (the traceparent HTTP header) to connect requests across services. Tempo is Grafana's trace backend. Together with Loki and Prometheus, it enables one crucial workflow: drilling down from metrics → traces → logs.

Consider a concrete scenario: Grafana shows the error rate for POST /api/order rising from 0.2% to 6%.

1. Click the error-rate panel to drill down into the list of traces from that period.
2. Select a slow trace and see that it passed through api → payment → external_gateway, with the external_gateway segment taking 4.8 seconds.
3. Click that segment to drill down into the logs from that moment and see that the gateway returned connection reset.

When all three pillars are connected through the same trace_id and label set, troubleshooting changes from "guessing" to "following the evidence." This is what observability should look like—not three isolated systems, but an interconnected network you can navigate in either direction.

Summary

• Metrics, logs, and traces are all essential: metrics tell you "there is a problem," traces tell you "which segment contains it," and logs tell you "exactly why." Prometheus + Grafana addresses the metrics pillar, but do not stop there.
• The essence of Prometheus is pull + time series + labels: the application only exposes /metrics, while Prometheus handles collection. Labels provide dimensions, but high-cardinality labels such as user_id and order_id cause series explosions and must never be instrumented.
• Choose the correct metric type before writing PromQL: use rate with Counter, read Gauge directly, use Histogram for quantiles, and almost never use Summary because it cannot aggregate across instances. In one sentence: Counter for cumulative values, Gauge for instantaneous values, and Histogram for distributions.
• RED is the minimum metric set for microservices: Rate / Errors / Duration, combined with p95 from histogram_quantile, provides the foundation for a service health dashboard.
• Alerts should be restrained, and dashboards should be layered: use a for duration to suppress transient fluctuations, alert against SLOs rather than thresholds, and keep alerts below five per on-call engineer per day.

To close with one sentence: observability is not about piling up tools; it is about locating the exact machine and line of code within ten minutes when an alert wakes you at 3 a.m. The classic Prometheus + Grafana stack, combined with Loki and OpenTelemetry, turns that process from guesswork into engineering.