AI Telemetry High Cardinality Budget: Controlling Tenant, Prompt, and Trace Label Explosions

The most common AI telemetry incident isn't missing a metric — it's stuffing values that belong in a Trace into Prometheus labels. tenant_id, user_id, request_id, full model names, prompt hashes, session IDs, and tool parameters each look diagnostically valuable on their own, but combined they make active time series multiply. Storage costs and query latency rise first, then the Collector queue builds up, remote write gets rate-limited, and eventually even the basic metrics used for alerting are drowned out.

This isn't a general 'getting started with Prometheus' tutorial. Prometheus and Grafana Observability already covers the monitoring stack basics, and LLM Observability covers traces, evals, and metrics. This article solves one production constraint: every new AI telemetry dimension must first prove its cardinality is controllable, and be able to be cut off individually when it spirals.

First split signals into three carrier layers

Metrics keep only attributes with bounded cardinality that can be aggregated and alerted on, such as service, environment, route, model_family, provider, outcome, and an enumerated error_class. Tenants can be grouped by plan or isolation pool, but don't use the tenant ID directly. Metrics answer 'how wide is the impact'; they don't carry single-request forensics.

Traces can hold request_id, prompt versions, tool names, and provider request IDs, because traces themselves have sampling and retention bounds. Even so, raw prompts, user emails, or tool parameters should not be written to attributes by default. Logs handle a small number of structured events, and indexed fields still use a whitelist; high-cardinality values go into unindexed bodies or isolated object storage. When you need per-tenant attribution, prefer computing from a ledger or offline detail. The request-level ledger in AI Cost Attribution is more reliable than maintaining millions of metric series long-term.

Estimate with multiplication, not gut-feel approvals

Set an upper bound on the label domain for each metric. Suppose a request histogram contains 3 environments, 20 services, 4 routes, 6 model families, 5 outcome classes, and 12 buckets; in theory that already produces 3 × 20 × 4 × 6 × 5 × 12 = 86,400 bucket series, and adding 500 tenants turns that into over forty million. The budget table should record the projected domain, worst-case combination, new series per hour, retention days, and owner.

Set two thresholds per service: a series cap for any single new metric, and an active-series total for the whole service. Before release, replay traffic for fifteen to thirty minutes and observe actual combinations, not just the distinct count of each label on its own. For acceptance you can query prometheus_tsdb_head_series, the remote write failure rate, the Collector export queue usage, and use /api/v1/status/tsdb or the backend's cardinality API to find the label pairs contributing the most.

topk(10,
  count by (__name__, service) ({job="ai-gateway"})
)

sum by (service) (
  rate(otelcol_exporter_send_failed_metric_points_total[5m])
)

The gate can't just say 'no big increase from baseline'. For example, require the canary's active series increase to be no more than 8%, no single metric over 50,000 series, zero remote write failures, and p95 query latency must not worsen by more than 20%. If any threshold is exceeded, stop widening traffic.

Limit attributes at both the SDK and Collector layers

On the app side, use an OpenTelemetry View to select explicit attribute keys for metrics, and map model to a controlled model_family. Unknown provider models go into other instead of automatically becoming a new label. The Collector then provides a second line of defense, deleting any identity and request attributes that leaked through:

processors:
  transform/cardinality_guard:
    error_mode: ignore
    metric_statements:
      - context: datapoint
        statements:
          - delete_key(attributes, "enduser.id")
          - delete_key(attributes, "tenant.id")
          - delete_key(attributes, "gen_ai.prompt.id")
          - delete_key(attributes, "request.id")

service:
  pipelines:
    metrics:
      processors: [memory_limiter, transform/cardinality_guard, batch]

Don't wait for Prometheus metric_relabel_configs to start cleaning, because samples have already passed through the SDK, the network, and the Collector, so the upstream cost is already paid. Don't use a hashed user ID as a metric label either; the hash hides the original value but doesn't reduce cardinality. The stable tenant identity in AI Platform Tenant Quota Isolation is suitable for billing and limits, but that doesn't mean it's suitable for time-series indexing.

Build observability its own change process

Every metric change ships a telemetry diff with the code commit: which metrics were added or removed, how the label set changed, the projected series count, sample data, and owner. Collector configuration goes to a shadow pipeline first, writing only to a short-retention test backend; after confirming dashboards and alerts can be expressed with low-cardinality aggregations, switch the main remote write. For dynamic model routing, you must test the normalization result when the provider returns an unknown model, a long version string, or a null.

Continuous alerts should at minimum cover active-series growth rate, sample rejection, WAL or queue backlog, Collector memory, remote write latency, and the heaviest metrics ranking. Daily dashboards show 'percentage of budget used' instead of waiting for an abnormal storage bill to act. Scan for recently appearing new label values weekly, and trace unexplained growth to a specific deployment.

When over limit, first secure the alerting baseline

After a cardinality explosion, the first step is to freeze related releases and identify the first abnormal moment; the second step is to find the growth source by metric and label; the third step is to enable the pre-prepared Collector emergency rules to delete high-risk attributes or temporarily drop a single new metric. Don't shut down the entire metrics pipeline, and don't simultaneously adjust sampling, retention, and storage clusters during an incident.

Rollback uses versioned configuration: restore the last signed View and Collector configuration, restart the canary, confirm new series growth has stopped, send failures are back to zero, and core alerts still have data, then gradually roll across instances. Old series that were already created won't disappear immediately, so you also need to wait for the head block and retention cycle to converge; don't directly delete TSDB files without backend support.

After the incident, validate the fix with the same replay traffic, and add a test for 'unknown dimension value'. What a high-cardinality budget really protects isn't the monitoring bill — it's the ability to still query latency, errors, and saturation when a fault happens. Only by putting single-request details back into traces and keeping metrics in a bounded domain will the AI platform's diagnostic capability grow with traffic instead of crashing along with it.

Further reading

• LLM Application Observability
• Prometheus and Grafana Observability
• AI Cost Attribution
• AI Platform Tenant Quota Isolation