Managing AI Batch Workloads with Kueue: Quota Borrowing, Admission, and Preemption Boundaries

Creating large numbers of Pods directly for training or offline inference hides quota contention in the Pending state. Kueue first suspends managed Jobs, then admits resources for an entire Workload as a unit; retries and idempotency must still be designed according to LLM Batch Inference.

Establish the Control Plane Version Contract First

Kueue APIs must conform to the installed CRDs. Current examples in the official documentation use kueue.x-k8s.io/v1beta2, while older clusters may only have v1beta1; CI should perform discovery and a server-side dry-run against the target cluster:

kubectl api-resources --api-group=kueue.x-k8s.io
kubectl explain clusterqueue.spec --api-version=kueue.x-k8s.io/v1beta2
kubectl explain localqueue.spec --api-version=kueue.x-k8s.io/v1beta2
kubectl apply --server-side --dry-run=server -f queues.yaml

If they do not match, use the corresponding schema rather than merely changing the version string; before upgrading, verify that the controller, CRD conversion, and webhook are ready.

Define Ownership with Three Object Layers

ResourceFlavor represents a class of interchangeable resources, such as an A100 node pool; ClusterQueue is a cluster-level quota pool maintained by platform administrators; LocalQueue resides in a tenant namespace and only references a ClusterQueue that the tenant is authorized to use. A minimal configuration using the current API version is shown below:

apiVersion: kueue.x-k8s.io/v1beta2
kind: ResourceFlavor
metadata:
  name: a100
spec:
  nodeLabels:
    accelerator.example.com/model: a100
---
apiVersion: kueue.x-k8s.io/v1beta2
kind: ClusterQueue
metadata:
  name: team-a-cq
spec:
  namespaceSelector:
    matchLabels:
      platform.example.com/kueue: enabled
  cohortName: ai-shared
  preemption:
    reclaimWithinCohort: LowerPriority
    borrowWithinCohort:
      policy: LowerPriority
    withinClusterQueue: LowerPriority
  resourceGroups:
  - coveredResources: ["cpu", "memory", "nvidia.com/gpu"]
    flavors:
    - name: a100
      resources:
      - name: cpu
        nominalQuota: "64"
      - name: memory
        nominalQuota: 512Gi
      - name: nvidia.com/gpu
        nominalQuota: 8
        borrowingLimit: 2
---
apiVersion: kueue.x-k8s.io/v1beta2
kind: LocalQueue
metadata:
  name: batch
  namespace: team-a
spec:
  clusterQueue: team-a-cq

A Job selects the LocalQueue with the label kueue.x-k8s.io/queue-name: batch and explicitly declares resource requests; Kueue creates a Workload and manages its suspension state. Tenant RBAC should only permit submitting Jobs and reading queue status in the tenant's own namespace, not modifying ClusterQueues, ResourceFlavors, or priority classes. This is just as important as AI Platform Tenant Quota Isolation.

The selector in the manifest requires running kubectl label namespace team-a platform.example.com/kueue=enabled --overwrite first. Namespaces without the label must remain inadmissible; any principal that can modify namespace labels effectively has queue admission privileges.

Borrowing Is Not Permanent Capacity Expansion

nominalQuota is the nominal allocation. Queues in the same cohort can borrow unused quota, while borrowingLimit caps the amount they may borrow. Represent GPU models and failure costs with distinct flavors, and let jobs declare only the flavors they can tolerate.

The policies above allow only higher-priority workloads to reclaim resources from lower-priority victims. The platform should create controlled PriorityClasses or WorkloadPriorityClasses, and tenants should only be able to reference approved levels. Preemption should be enabled only after checkpointing and output idempotency have passed validation, and every victim, reason, and recovery point must be recorded. Business priority from Background Job Queues must not directly become cluster preemption authority.

Validate Using Admission States

Test nominal quota, the borrowing limit, and queuing beyond those limits. Then use a controlled high-priority workload to trigger preemption and verify that the victim has lower priority than the initiator, along with the reason, checkpoint, and idempotent recovery. Also submit a Job from a namespace without the selector label and assert that it is never admitted and never creates compute Pods.

kubectl get clusterqueues -o wide
kubectl get localqueues -A
kubectl get workloads -A
kubectl describe workload -n team-a <name>
kubectl get events -n team-a --sort-by=.lastTimestamp

Alerts should distinguish flavor, quota, admission check, and webhook failures; wait duration, borrowing rate, and eviction rate should feed into LLM SLO Monitoring.

Stop Admission Before Rolling Back

If an incorrect quota could trigger concentrated preemption, first set stopPolicy: Hold to stop new admissions and allow already-admitted jobs to finish:

kubectl patch clusterqueue team-a-cq --type=merge \
  -p '{"spec":{"stopPolicy":"Hold"}}'

Export Workload states, correct the flavor, quota, cohort, or preemption configuration, perform a dry-run, and then resume admission. Do not delete queues or Pending Jobs. The controller may only be rolled back to a version compatible with the CRD combination; for already-running jobs, first validate checkpoints according to LLM Disaster Recovery.

Further Reading

• LLM Batch Inference
• AI Platform Tenant Quota Isolation
• Background Job Queues
• LLM SLO Monitoring