Kubernetes GPU Device Allocation: Topology Isolation from Device Plugin to DRA

Successfully scheduling a Pod with nvidia.com/gpu: 1 proves only that the node still advertises one unit of an extended resource. It does not prove that the GPU model, VRAM slice, NUMA node, and CPU cores meet the inference process's requirements. Cross-NUMA memory access increases tail latency. If two drivers advertise the same physical device, it can also be allocated incorrectly. For capacity estimates, first refer to GPU Inference Capacity Planning. This article addresses how that capacity should be allocated correctly after it enters Kubernetes.

First Choose One Device Ownership Model

A traditional Device Plugin runs on the node and registers a Unix socket with kubelet. It uses the plugin protocol, which is still v1beta1, to perform ListAndWatch and Allocate, then writes integer extended resources such as vendor.example/gpu into Node status. Its deployment model is mature, and its Pod syntax is simple, making it suitable for clusters whose drivers expose only extended resources. However, the scheduler mainly sees resource names and quantities. More granular device attributes generally have to be represented through separate resource names, node labels, or vendor plugins.

With DRA, a driver publishes ResourceSlice objects, administrators define DeviceClass objects, and workloads request devices with matching attributes through ResourceClaim or ResourceClaimTemplate. Current official documentation places these core objects under resource.k8s.io/v1. DRA is suitable when devices must be selected by model, topology, or sharing capabilities, but only if the device driver and cluster version actually support DRA. Upgrading only the API Server does not make it available automatically.

Before upgrading, use the cluster's discovery results as the source of truth:

kubectl api-resources --api-group=resource.k8s.io
kubectl explain deviceclass.spec --api-version=resource.k8s.io/v1
kubectl explain resourceclaimtemplate.spec --api-version=resource.k8s.io/v1
kubectl get nodes -o custom-columns=NAME:.metadata.name,GPU:.status.allocatable.nvidia\.com/gpu

If the first three commands do not return the expected resources, continue using the verified Device Plugin. When migrating nodes, confirm that exactly one allocation path owns each GPU. Do not let two schedulable workloads receive the same GPU through the extended-resource path and the DRA-device path. What DRA objects can describe also depends on the attributes advertised by the driver; do not invent vendor fields in a generic template.

Extend Quantity Isolation to Topology and Failure-Domain Isolation

Device Plugin extended resources cannot be oversubscribed, but they do not automatically isolate VRAM. MIG, vGPU, or time-slicing units must be advertised as allocatable units by the vendor driver. A container memory limit cannot substitute for a VRAM boundary. Constrain separate GPU pools further with protected labels, taints, and the controls described in AI Platform Tenant Quota Isolation.

DRA also requires a ValidatingAdmissionPolicy to allowlist deviceClassName, selectors, count, and RuntimeClass. Always deny adminAccess to ordinary tenants. Only platform administrators may set resource.kubernetes.io/admin-access=true on a namespace or modify DeviceClass, ResourceSlice, and node labels. In Kubernetes 1.35, admin access is a beta feature enabled by default. It can access already allocated devices, so ResourceClaim namespace RBAC alone does not prove that isolation is effective.

Low-latency services must also validate the node-side Topology Manager. It operates at a different layer from DRA: DRA selects accessible devices but does not automatically guarantee NUMA alignment. A policy can coordinate CPU, memory, and devices only when the actual device management path supplies topology hints. The pod scope combined with the single-numa-node policy can require all hints for a Pod to converge on one NUMA node. CPU alignment also depends on CPU Manager and correct requests/limits. Roll this out first to a canary group of homogeneous nodes:

# KubeletConfiguration fragment; use the apiVersion required by the node distribution
cpuManagerPolicy: static
topologyManagerScope: pod
topologyManagerPolicy: single-numa-node

When single-numa-node cannot be satisfied, kubelet rejects the Pod. This differs from a Pod remaining Pending during scheduling. The controller must be able to recreate the failed Pod and route Topology Affinity events into alerting. If the model can tolerate multiple NUMA nodes, validate with restricted first, then decide whether to tighten the policy based on benchmark data. Do not judge the result solely by "higher GPU utilization."

Validate with Device Identity, Not Pod Running Status

Acceptance must record at least four categories of evidence: Node allocatable matches the plugin's healthy device count; the device UUID or slice ID actually assigned to each Pod is unique; CPU affinity and GPU PCIe/NUMA placement comply with the policy; identity and performance still satisfy the constraints after recreation. Run a separate short inference benchmark for every GPU model. Compare P50/P99, host-to-device bandwidth, and load time between NUMA-aligned placement and deliberate mismatches, then feed the results into Prometheus and Grafana Observability.

Failure testing must cover plugin restarts, devices marked unhealthy, node kubelet restarts, Claims remaining unallocated for an extended period, and fragmentation of resources on a single NUMA node. Standardize the inspection path as follows:

kubectl describe pod -n ai <pod>
kubectl get events -n ai --sort-by=.lastTimestamp
kubectl get resourceclaims -A
kubectl get resourceslices

The setup passes only if Pods remain unadmitted when resources are insufficient, workloads can be requeued after devices recover, and running processes never receive duplicate devices. Application-side batch jobs should also use the idempotent checkpoints described in LLM Batch Inference to avoid recomputing an entire batch after a device failure.

Roll Back by Node Pool

If the topology policy increases the rejection rate, first cordon the canary nodes and drain migratable workloads. Restore the previous kubelet configuration, restart kubelet, confirm it is healthy, and then uncordon the nodes. Do not "fix" the queue by removing requests or loosening tenant nodeSelector constraints. That converts the fault into cross-tenant interference and performance jitter.

Before rolling back DRA, stop new Claims, wait for or terminate workloads with allocated devices, confirm that the driver has released those devices, and then restore the original Device Plugin node pool. Keeping both types of pools and migrating through the queue is safer than switching in place on the same node. If production is undergoing a Model Serving Canary Rollout, change the device allocation layer and model version layer in separate change windows. Otherwise, a performance regression cannot be attributed to its cause.

Further Reading

• GPU Inference Capacity Planning
• AI Platform Tenant Quota Isolation
• Prometheus and Grafana Observability
• Model Serving Canary Rollout