Kubernetes Model Weight Distribution: Node Caching, Digest Verification, and Rollback

Models that are tens of gigabytes in size should not be downloaded from object storage every time an inference Pod starts. Doing so turns scaling time into a function of network jitter: the same node transfers identical data repeatedly, processes read partially downloaded files, and startup probes fail continuously after the origin begins throttling requests. Container images already use digests to provide immutable identities; model weights should follow the same principle. Model provenance, conversion, and signature verification belong to AI Supply Chain Security. This article focuses on distribution and lifecycle management after weights enter Kubernetes nodes.

Define an immutable model manifest first

Deployment objects should not specify only model: latest or an overwritable object key. The release pipeline should first generate a manifest containing at least the logical model name, format, shard list, size and SHA-256 of each file, overall version, source URI, and creation time. Workloads should reference only the manifest digest. Cache directories should also be named by digest, such as /cache/sha256/<digest>/. Once a path exists, it must never be modified in place.

The downloader creates a temporary directory on the same file system, verifies each file's length and digest, writes a READY marker after completion, and performs an atomic rename. Tokens are injected into the download process only through short-lived credentials and never appear in the manifest, logs, or Pod annotations. A simplified verification step looks like this:

set -euo pipefail
tmp="$(mktemp /cache/.model.XXXXXX)"
trap 'rm -f "$tmp"' EXIT
curl --fail --location --retry 3 -o "$tmp" "$MODEL_URL"
printf '%s  %s\n' "$MODEL_SHA256" "$tmp" | sha256sum --check --status
chmod 0444 "$tmp"
sync -f "$tmp"
dest="/cache/objects/$MODEL_SHA256"
ln -T -- "$tmp" "$dest" 2>/dev/null ||
  printf '%s  %s\n' "$MODEL_SHA256" "$dest" | sha256sum --check --status
sync -f "$(dirname "$dest")"
rm -f "$tmp"

The temporary file and destination must reside on the same file system. A production implementation should call fsync(2) on the file descriptor, publish the hard link or rename, and then fsync the parent directory. The example's sync -f is a heavier shell-level barrier. If a competing publication attempt fails, verify the existing object. Publish the READY pointer bound to the manifest root only after every shard has passed verification.

Choose a distribution method based on the scope of reuse

An init container with emptyDir is the simplest option: it finishes downloading before the main container starts, and files can be shared within the same Pod. This works for small models or low-replica jobs. The cache disappears when the Pod is recreated, however, so scaling storms still cause repeated downloads. A network file system or ReadOnlyMany PVC enables reuse across Pods, but model-loading latency depends on storage throughput and its failure domain. Benchmark concurrent mmap/read workloads, not just a single sequential read.

A DaemonSet node cache is suitable for a fixed GPU pool. A cache agent prefetches approved digests on every target node, and inference Pods mount the cache read-only. If hostPath is used, it bypasses normal storage isolation: the DaemonSet needs minimal privileges, and application containers must mount only the specific model subdirectory as read-only rather than exposing the entire host cache root to tenants. Node labels should be written only by a trusted controller after verification, for example models.example.com/ready-<short-digest>=true. Users must not be allowed to label nodes themselves or modify affinity.

The scheduler also needs to know whether an on-demand download is allowed after a cache miss. Latency-sensitive services should run only on ready nodes and retain enough prewarmed replicas. Batch jobs can be scheduled first and then use an init container to fetch missing data under bandwidth limits. Make this decision together with the cold-start budget from GPU Inference Capacity Planning and the traffic increments from Canary Rollouts for Model Serving. Do not hide download time behind readiness.

Reclamation must know what is still referenced

Begin cleanup when a node cache exceeds 80% utilization and stop when it falls to 65%. A candidate must have no active Pods or prewarming leases and must not belong to the current or previous stable version set. Reclamation first removes the ready node label for that digest, waits for scheduling state to propagate, and confirms that no bound or newly created Pods reference it before entering EVICTING. A normal hostPath mount does not consult the cache agent's lock, so reference counts cannot prevent races with kubelet. If hard mount leases are required, use a CSI/node plugin that acquires and releases leases by Pod UID. Skip any object that still has references or open files.

Disk pressure must not leave kubelet to select models at random. Recompute the manifest root when the cache service starts, before writing ready labels, and before a Pod uses an object. On a mismatch, remove the label first and then quarantine the object. A cache failure must never mark an unverified path as ready. Monitor bytes, inodes, verification failures, hit rate, and reference counts.

Validate with destructive scenarios

First measure startup on cold and warm nodes, bytes transferred from the origin, and time to the first request. Then inject interrupted downloads, incorrect SHA values, missing shards, concurrent downloads, insufficient disk space, power loss after publication, and tampering after READY. Assert that the main container never reads a partial file, an incorrect root causes the ready label to be removed before quarantine, temporary files are cleaned up, and other models remain available.

kubectl get daemonset -n model-system
kubectl get pods -n model-system -o wide
kubectl describe node <gpu-node>
kubectl logs -n model-system <cache-pod> --since=30m

At the service layer, continuously record requests by version and digest. Use Observability with Prometheus and Grafana to compare cache hits, model-loading duration in seconds, and P99 latency. This avoids the false optimization where a Pod becomes Ready quickly but loads its weights only when the first request arrives.

Roll back by changing references, not overwriting files

If a new model misbehaves, change the Deployment or release controller's manifest digest back to the previous stable value and ensure that the corresponding node cache remains in the retained set. Then replace Pods through the normal rolling strategy. Do not copy old files back into the new digest path or modify the contents of an object with the same name. Otherwise, audit records, cache state, and actual bytes will diverge.

When rolling back the cache agent version, pause reclamation first and preserve verified objects and READY metadata. After rolling back the DaemonSet, recompute digests for a random sample before resuming reclamation. If the entire node cache is untrusted, cordon the node, drain inference Pods, clear the dedicated cache partition, and prewarm it again. Do not repair files in place while running containers still reference them. If the cross-region origin is unavailable, use a verified replica as described in LLM Disaster Recovery. The recovery target must still have the same content digest.

Further reading

• AI Supply Chain Security
• GPU Inference Capacity Planning
• Canary Rollouts for Model Serving
• LLM Disaster Recovery