Scaling Kubernetes AI Workers with Queue Metrics: HPA Stability in Practice

Asynchronous summarization, embedding, and batch inference workers use very little CPU while idle, yet may spend a full minute loading a model after a backlog appears. Scaling solely on CPU reacts only after queued jobs have already timed out. Scaling solely on total queue length can continually overprovision capacity if claimed jobs are counted again. First define delivery, acknowledgment, and dead-letter semantics as described in Background Job Queues; the HPA should consume one explicit, reproducible capacity signal.

Define a Metric Contract First

ai_jobs_ready counts only jobs in the specified queue that are available to claim and are not under a lease. Also report ai_work_seconds_outstanding, which adds the estimated processing time of ready jobs to the remaining processing time of active leases. Otherwise, ready jobs dropping to zero can trigger a scale-down while long-running jobs are still executing, and the Deployment may terminate those Pods. Keep labels limited to low-cardinality queue, region, and workload class values; do not include tenant or request IDs.

A target of 8 means approximately eight ready jobs per replica. Derive the work-seconds target from the queueing SLO, per-worker throughput, and startup time. The HPA calculates the required replica count for each metric and uses the highest value, so the active-lease metric preserves capacity for in-flight work.

The current stable HPA API is autoscaling/v2. External metrics require an adapter registered with the cluster under external.metrics.k8s.io; the HPA will not install, authenticate, or validate that adapter for you. Example:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: ai-worker
  namespace: ai
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: ai-worker
  minReplicas: 2
  maxReplicas: 40
  metrics:
  - type: External
    external:
      metric:
        name: ai_jobs_ready
        selector:
          matchLabels:
            queue: embedding
      target:
        type: AverageValue
        averageValue: "8"
  - type: External
    external:
      metric:
        name: ai_work_seconds_outstanding
        selector:
          matchLabels:
            queue: embedding
      target:
        type: AverageValue
        averageValue: "300"
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 0
      policies:
      - type: Percent
        value: 100
        periodSeconds: 60
    scaleDown:
      stabilizationWindowSeconds: 600
      policies:
      - type: Percent
        value: 25
        periodSeconds: 60

The two AverageValue targets constrain ready jobs and outstanding work seconds per replica, respectively, and the HPA uses the larger recommendation. maxReplicas remains constrained by GPU/CPU quotas, database connections, model origin servers, and downstream rate limits. The device count from GPU Inference Capacity Planning is a hard upper bound.

Include Cold Starts in Effective Capacity

A new Pod provides no throughput until its model has loaded, and the HPA cannot shorten that startup period. Give workers a startup probe and genuine readiness checks. On the metrics side, separately report ready workers, startup duration, and processing rate. minReplicas should cover at least normal traffic plus one failed replica. For very large models, warm the nodes first; do not conceal loading time by reporting readiness success early.

During scale-down, a worker must stop claiming jobs, finish or release its lease, and only then exit. The stabilization window, termination grace period, maximum job duration, and lease timeout must be consistent. For long-running jobs that cannot be interrupted or migrated, set scaleDown.selectPolicy to Disabled, or use a controller that can select idle Pods based on leases. Deployment scale-down does not guarantee that an idle replica will be removed.

The metrics adapter should use a read-only account, query timeouts, and sample timestamps. Return an error for stale values instead of replaying an old low value. Monitor adapter discrepancies, ready jobs, oldest age, work seconds, active leases, and cold starts, and integrate them with Prometheus and Grafana Observability.

Failure-Test the Control Loop

After applying load at a fixed arrival rate, stop enqueuing jobs. Verify that the system retains the required capacity until active leases reach zero, then scales down to minReplicas. Record the latency from backlog growth to a new Pod becoming ready, and use LLM SLO Monitoring to inspect oldest age.

Inspect HPA conditions and raw metrics directly to rule out the case where "the dashboard has data, but the aggregation API does not":

kubectl describe hpa -n ai ai-worker
kubectl get --raw '/apis/external.metrics.k8s.io/v1beta1/namespaces/ai/ai_jobs_ready?labelSelector=queue%3Dembedding'
kubectl get deployment,pods -n ai -w

Then separately inject adapter timeouts, empty series, frozen values, queue connection failures, a Pod that never finishes starting, and downstream 429 responses. The expected behavior is an explicit HPA condition and alert. The system must not interpret unavailable metrics as a zero backlog or scale beyond maxReplicas indefinitely. After recovery, the control loop should converge smoothly on the real value rather than scaling to the minimum and then growing again.

Roll Back by Freezing Known Capacity

If the new metric semantics are wrong, first export and disable the HPA, then immediately set the Deployment to a known replica count that can handle current traffic. Do not conceal the error by setting the target to an extremely high value. Save the HPA YAML, adapter version, and query results. After fixing the issue, restore scaling gradually with a lower maxReplicas.

If erroneous scaling has already overwhelmed a downstream system, first stop consumers from claiming new jobs while preserving the queued data, then recover gradually at a rate the downstream system can sustain. A queue is a buffer, not a data dumping ground. Retention, dead-lettering, and replay procedures must still follow Message Queue System Design. Autoscaling should run unattended only after all four components—the metric, HPA, worker draining, and queue idempotency—pass failure testing together.

Further Reading

• Background Job Queues
• GPU Inference Capacity Planning
• Prometheus and Grafana Observability
• Message Queue System Design