Docker Security Hardening: Image Scanning and Least Privilege

I reviewed a production incident involving a container running a Python service that had been infected with a cryptominer. top showed the CPU completely saturated by a process named kdevtmpfsi. The investigation revealed that an old Flask extension in the dependencies had an RCE vulnerability. The attacker used it to obtain a shell inside the container, then used curl to download a mining script. What really sent a chill down my spine was the next command: inside the container, the attacker ran cat /proc/1/status | grep Cap and discovered that they were root, with CAP_SYS_ADMIN among their capabilities.

Containers are not sandboxes, and most people have not internalized this fact. By default, when a container starts, its process runs as root. Although about half of the capabilities are removed, plenty of dangerous permissions remain. The default seccomp profile blocks only the most outrageous system calls, the filesystem is writable, and resources are unrestricted. Once the application layer is compromised through an RCE, SSRF, or deserialization vulnerability, the attacker is effectively root inside the container and only one escape away from the host.

This article explains how to break every link in that chain: running as a non-root user, using minimal images, scanning for CVEs, applying seccomp/AppArmor, making the root filesystem read-only, and limiting resources. None of these techniques is new, but fewer than 30% of the teams I have encountered implement all six in production. Security hardening is not about showing off. It is a fundamental engineering discipline.

1. Containers Are Not Sandboxes: How Extensive Are the Default Permissions?

First, let us dispel an illusion. Many people think, "Containers provide isolation, so even if the application is compromised, the attacker cannot get out." This idea is wrong on two levels.

First, root inside a container is still root. Container namespaces isolate views such as process IDs, networks, and mount points, not privileges. Root inside a container has full control over its filesystem, mounted volumes, and every operation permitted by its capabilities. More dangerously, a significant number of Linux kernel syscalls are not isolated by namespaces, including certain operations on /proc and the kernel keyring. By combining access as container root with a kernel vulnerability, an attacker can jump into the host namespace. This is known as a "container escape." CVE-2019-5736 (runc), CVE-2022-0185 (cgroups), and Dirty COW all followed this pattern.

Second, the default capabilities and seccomp policy are more permissive than you might think. Docker grants containers approximately 14 capabilities by default, supposedly after "removing the dangerous ones," but CAP_NET_RAW remains. This means an attacker can send arbitrary raw packets over the container network, perform ARP spoofing, and scan the internal network laterally. The default seccomp profile blacklists only about 60 syscalls, such as reboot and kexec_load. Linux has more than 400 syscalls, and all the rest are allowed.

The following table shows the permission differences between a default container and a hardened one:

The essence of security hardening is moving every category from "allow by default" to "deny by default." Each section below addresses one category.

2. Running as Non-Root: The Details of the USER Instruction and UID

This measure offers the highest ROI: add one USER instruction to the Dockerfile. Even if an attacker achieves RCE, they gain only a low-privilege user, eliminating every subsequent path that requires escalating to root before escaping the container.

FROM node:22-slim AS runner
WORKDIR /app

# 创建专用用户和组,固定 uid/gid
RUN groupadd --system --gid 1001 nodejs \
 && useradd  --system --uid 1001 --gid nodejs --no-create-home --shell /usr/sbin/nologin nextjs

COPY --chown=nextjs:nodejs ./app ./
USER nextjs
CMD ["node", "server.js"]

Several practical points matter:

1. USER must appear before CMD/ENTRYPOINT. Placing it afterward is useless because the process has already started. It comes after COPY so that --chown can assign file ownership to this user. Otherwise, a non-root process might be unable to read or write files copied by root, such as when it needs to write to a temporary directory.

2. Use a fixed UID rather than dynamic allocation. If useradd is used without --uid, the UID is assigned dynamically, progressing through 1000, 1001, and so on. This creates problems in K8s: a SecurityContext such as runAsUser: 1000 requires the UID to match the one in the Dockerfile. A fixed UID keeps the orchestration layer consistent with the image layer. The convention is to choose a value above 1000 while avoiding the system-reserved range.

3. When using distroless, align the UID with a user already present in the base image. gcr.io/distroless/nodejs22-debian12 already includes a nonroot user with uid 65532. You can simply use USER nonroot instead of creating your own user.

4. Use SecurityContext as a backstop in K8s; do not rely solely on the Dockerfile. Enforce the setting at runtime so that someone cannot bypass the hardening with docker run -u root:

securityContext:
  runAsNonRoot: true        # 强制非 root,uid 0 直接拒绝启动
  runAsUser: 1001
  fsGroup: 1001

runAsNonRoot: true is a strict switch. If the image starts as root, Kubelet refuses to start the Pod, which is far stronger than auditing it after the fact.

A common question is: "My application needs to bind to port 80, but a non-root user cannot bind to it. What should I do?" The correct answer is not to enable root. Instead, grant the privileged-port binding capability to the binary with setcap cap_net_bind_service=+ep, use sysctl net.ipv4.ip_unprivileged_port_start=80 on Linux 5.7+, or simply avoid binding to port 80. Put a Service or Ingress in front and have the container listen on 8080.

3. Minimal Images: distroless / scratch / alpine, Because Attack Surface Is a Function of Size

Every additional binary in an image creates another attack path. With an image containing curl, bash, and apt-get, an attacker who achieves RCE can immediately download a backdoor, probe laterally, and erase traces. In an image without even /bin/sh, an attacker may successfully call exec only to encounter EPERM.

The three approaches to "small images" suit different scenarios:

My practical recommendations:

Use distroless as the default runtime image. It is maintained by Google, based on Debian, and provides versions matching your application's runtime, such as nodejs22-debian12, python3-debian12, and gcc-debian12. It includes CA certificates, NSS, and tzdata, but has no shell, apt, or ls. Debugging is inconvenient because docker exec -it <c> sh immediately returns not found, but that inconvenience is exactly what you want in production. When debugging is genuinely necessary, temporarily use the :debug variant, which includes a busybox shell, and only while troubleshooting.

For Go or Rust applications compiled into a single static binary, use scratch directly:

FROM golang:1.22 AS builder
WORKDIR /src
COPY . .
RUN CGO_ENABLED=0 go build -o /app -ldflags="-s -w" .

FROM scratch
COPY --from=builder /app /app
COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
USER 65532:65532
ENTRYPOINT ["/app"]

The final image contains only one binary and a CA certificate bundle. It is tens of MB and does not even include libc. CGO_ENABLED=0 must be enabled; otherwise, the binary dynamically links against glibc and cannot run in scratch. Do not forget to copy the CA certificates, or every HTTPS request will fail with x509: certificate signed by unknown authority.

alpine is a compromise. Its musl libc is extremely small, but be careful: packages requiring native compilation, such as bcrypt, sharp, cryptography, and better-sqlite3, may fail to install on musl or exhibit subtle behavioral differences. My rule is to use alpine in the builder stage because it is fast and includes a complete toolchain, while preferring distroless for the runner stage. The two approaches do not conflict; simply use COPY --from across stages.

Remember this mental model: image size and attack surface are two faces of the same number. A 50MB image without a shell and a 900MB image with apt are entirely different worlds to a CVE scanner. The former may have no critical vulnerabilities, while the latter often produces dozens in a single scan. That leads directly to the next section.

4. Image Scanning: Stop CVEs in CI Instead of Shipping Them to Production

A smaller image is not necessarily free of vulnerabilities. The versions of openssl, glibc, and node may still contain CVEs. The value of image scanning is not merely "finding vulnerabilities," but preventing deployment once they are found. A scanner that does not block CI is effectively useless.

Choose either of these two mainstream open-source tools:

Trivy (Aqua Security, recommended). It is fast, provides broad coverage across OS packages, language dependencies, IaC, and secret leakage, and produces clear reports. A single command can block high-severity findings:

# 扫镜像,发现 HIGH/CRITICAL 直接退出非零,CI 失败
trivy image \
  --severity HIGH,CRITICAL \
  --ignore-unfixed \
  --exit-code 1 \
  --scanners vuln \
  myapp:$(git rev-parse --short HEAD)

--ignore-unfixed is crucial. If upstream has not released a patch, you cannot fix the reported CVE; it only creates noise and fatigue. Block only vulnerabilities with available fixes and focus your effort on what can actually be remediated.

Grype (Anchore, well suited to CI integration). Its output is more structured, and SARIF output can be sent directly to the GitHub Security tab:

grype myapp:latest \
  --fail-on high \
  --only-fixed \
  -o sarif > grype.sarif

Several practical strategies:

1. Scan after pushing and before deployment. Scan immediately after docker build, and push to the registry only after the scan passes. An even better approach is to enable continuous scanning on the registry side with Trivy Server, GCR Vulnerability Scanning, or ECR Inspector. New CVEs are published every day, so an image that is clean today may turn red next week because of a newly disclosed vulnerability. Production images must be rescanned continuously; scanning once at build time is not enough.

2. Manage known exceptions with .trivyignore, but give them expiration dates. Some CVEs are false positives, will not be fixed upstream, or do not affect your use case. Exceptions are acceptable, but each one must include a reason and review date. Otherwise, .trivyignore becomes a permanent dumping ground that is never cleaned up:

# CVE-2024-xxxxx: 业务未使用受影响的 XXX 功能,2025-10-01 复查
CVE-2024-xxxxx

3. The fundamental solution for unfixable vulnerabilities is upgrading the base image. If Trivy reports numerous CVEs in node:18-slim, changing it to node:22-slim with sed may clear most of them. The base image version is the largest variable affecting vulnerability counts. Upgrade to a new major version at least once a year.

4. Remember to scan dependencies, not just OS packages. Trivy and Grype both support --scanners vuln for scanning npm, pip, and Maven dependencies. Vulnerabilities in Node applications often exist at the npm audit layer rather than in Debian packages.

In short: the scanner's value lies in --exit-code 1, not in the report itself. Nobody reads unattended reports; enforcement must be automated.

5. seccomp and AppArmor: A Second Line of Defense at the System-Call Level

If non-root operation "removes user privileges," then seccomp and AppArmor "remove what the kernel is allowed to do." After compromising an application, attackers usually rely on a combination of an obscure system call and a kernel vulnerability to escalate privileges or escape. seccomp blocks those system calls outright, turning exploitation of a kernel vulnerability from "difficult" into "impossible."

seccomp: system-call filtering. Docker enables seccomp by default using /etc/docker/seccomp/default.json. Its blacklist blocks approximately 60 dangerous syscalls, including keyctl, kexec_load, and, in some contexts, unshare. The default profile provides basic protection, but real hardening means switching to a custom allowlist profile: instead of allowing more than 300 syscalls by default, deny them by default and allow only the few dozen that the application actually uses.

Tools can help generate a seccomp profile. oci-seccomp-bpf-hook can run the application in a clean environment, trace the syscalls it actually uses, and automatically generate a minimal profile. This is the most reliable approach because it is based on observation rather than guesswork, avoiding situations where you assume a syscall is unnecessary only to have the application fail at startup.

Enable your custom profile with:

docker run --security-opt seccomp=/etc/docker/seccomp/myapp.json myapp:latest

In K8s 1.19+ (GA), specify it in SecurityContext:

securityContext:
  seccompProfile:
    type: Localhost
    localhostProfile: profiles/myapp-seccomp.json

AppArmor: mandatory access control for paths and capabilities. seccomp governs syscalls, while AppArmor provides finer control over which paths can be read or written and which binaries can be executed. It is the default LSM (Linux Security Module) on Ubuntu and Debian. CentOS-based systems use SELinux, which is more complex but more powerful.

A simplified AppArmor profile looks like this:

#include <tunables/global>
profile myapp flags=(attach_disconnected) {
  #include <abstractions/base>
  # 只允许读自己的配置和 CA
  /app/** r,
  /etc/ssl/certs/** r,
  # 只允许写数据目录
  /data/** rw,
  # 禁止 exec 任何东西(挡反弹 shell)
  deny /** x,
  network inet stream,
}

The deny /** x line is the one I recommend most strongly. The application process can no longer execute any binary, so even an attacker with RCE cannot start a shell, run curl, or download a backdoor. Combined with the absence of a shell in distroless, this provides two layers of protection.

docker run --security-opt apparmor=docker-myapp myapp:latest

seccomp and AppArmor are not alternatives; they are complementary. seccomp blocks kernel exploitation at the syscall layer, while AppArmor blocks lateral movement at the filesystem and capability layer. They are orthogonal. I recommend enabling both for critical production services.

A practical warning: do not enable both for the first time in production. Run them in a test environment for a week first. An incomplete seccomp profile can terminate the process immediately with SIGSYS, while an overly strict AppArmor profile can cause EACCES errors. Both profiles must be tuned in a non-production environment until they allow exactly what is needed.

6. Read-Only rootfs and Resource Limits: Preventing Persistent Writes and Resource Exhaustion

The first five sections address "permissions." This section addresses the "consequences." Even if an attacker gets in, preventing them from writing files or consuming excessive resources can reduce the blast radius to a minimum.

Read-only rootfs: --read-only. A container's filesystem is writable by default, so downloaded mining programs, injected cron jobs, and modified binaries remain on disk until restart. With read-only mode enabled, the root filesystem is mounted read-only, and every write operation immediately returns EROFS:

docker run --read-only \
  --tmpfs /tmp:rw,size=10m \
  --tmpfs /var/cache:rw,size=50m \
  myapp:latest

--tmpfs is the essential companion setting. Applications always have some directories that must remain writable, such as /tmp, cache directories, and session storage. Explicitly allow them with tmpfs, which is mounted in memory and cleared when the container stops. Use a read-only root, volumes for persistent data, and tmpfs for temporary data. This is a clean write model.

The K8s equivalent is:

securityContext:
  readOnlyRootFilesystem: true

In the Pod spec, add emptyDir or emptyDir.medium: Memory, which is tmpfs, for the required writable mount points.

A read-only rootfs provides another less obvious benefit: it forces developers to distinguish between stateful and stateless data. When rootfs is writable, developers can casually place files in /app/data and the application still runs. Once it becomes read-only, every piece of data requiring persistence must explicitly use a volume, which is itself a better architectural practice.

Resource limits: memory / cpu / pids. Containers have no resource limits by default. A container that exhausts memory or saturates the CPU can bring down the entire host. This is the "noisy neighbor" problem and the most direct impact of cryptomining or fork-bomb attacks. Apply these three limits with docker run:

docker run \
  --memory="512m" \
  --memory-swap="512m" \
  --memory-reservation="256m" \
  --cpus="1.0" \
  --cpu-shares=512 \
  --pids-limit="100" \
  myapp:latest

Here is what each setting does:

• Set --memory and --memory-swap to the same value to disable swap as a fallback. On OOM, the process is killed immediately instead of consuming swap without limit. This is crucial for avoiding a slow collapse.
• --memory-reservation is a soft limit: bursts are normally allowed, but reclamation uses this value when memory is scarce.
• --cpus="1.0" limits the number of CPU cores. Note that this is a relative limit, not absolute CPU pinning. --cpu-shares is a weight with a default of 1024 and takes effect only under CPU contention.
• --pids-limit is the easiest limit to overlook, but the most effective protection against fork bombs. If an attacker or bug causes fork() to run out of control, the absence of a PID limit can instantly exhaust the process table and affect every container on the host.

The K8s equivalent uses the resources field:

resources:
  requests:
    cpu: "250m"
    memory: "256Mi"
  limits:
    cpu: "1000m"
    memory: "512Mi"

One counterintuitive warning: do not set limits without requests. K8s scheduling uses requests to determine whether a Pod fits on a node, while limits define the runtime cgroup ceiling. With only limits configured, a Pod may be scheduled onto a node that does not actually have enough resources, causing frequent throttling and OOM events. Requests are commitments, limits are ceilings, and both must be present. Limits are generally two to four times the corresponding requests.

A special reminder: always set a memory limit, but be cautious with CPU limits. CPU is a compressible resource: exceeding the limit only causes throttling and does not kill the process. Memory is not compressible: exceeding the limit causes an immediate OOMKill. A memory limit is therefore a mandatory safety net. CPU limits depend on the use case. If set too low for latency-sensitive services, throttling will slow them down. I have seen teams give an API cpu: 100m, or 0.1 core, causing p99 latency to jump to several seconds.

7. Production Container Security Checklist: Review Before Release

The following checklist turns the previous six sections into something you can add to a PR template. Every item is a binary "done or not done" decision. There is no "mostly done."

Image layer

• Use a multi-stage build; the final image contains only runtime artifacts, with no source code, devDependencies, or build toolchain
• Use distroless / scratch / alpine-slim as the base image, not full Debian
• Keep the image under 200MB, or provide a reason for exceeding it
• Run Trivy/Grype in CI and use --exit-code 1 to block HIGH/CRITICAL findings
• Upgrade base image versions every quarter

Runtime permission layer

• The Dockerfile contains a USER <非root> instruction before CMD
• The UID is fixed (≥1000) and consistent with the K8s SecurityContext
• K8s uses runAsNonRoot: true as a backstop
• Use --cap-drop=ALL and add capabilities with --cap-add only as needed; most services need none
• Never enable privileged mode (securityContext.privileged: false)

System-call and MAC layer

• Enable seccomp; at minimum use Docker's default profile, and use a custom allowlist for critical services
• Enable AppArmor/SELinux, with deny /** x in the profile to block execution

Filesystem layer

• Enable --read-only / readOnlyRootFilesystem: true
• Explicitly use tmpfs / emptyDir for writable directories and volumes for persistent data
• Do not mount sensitive host paths such as /var/run/docker.sock, /, or /proc

Resource layer

• Set both a memory limit and a memory request
• Set a CPU request and configure the limit according to the use case
• Set a PID limit; in K8s, enforce it with a podPidsLimit admission policy

Secret layer (often forgotten, but extremely important)

• Do not put secrets in the image; hard-coded ENV values and COPY .env are both prohibited
• Inject secrets at runtime using K8s Secret / Vault / cloud KMS
• Use Trivy --scanners secret to scan for leaked tokens inside the image

Orchestration layer (additional K8s controls)

• Use NetworkPolicy to restrict east-west traffic; deny all by default and allow only what is needed
• Enable PodSecurity Admission at the restricted level
• Use an admission controller to block privileged / hostNetwork / hostPID configurations
• Use namespaces to isolate production, staging, and testing

This checklist is not a one-time exercise. Review it for every new service before release and audit every existing service quarterly. Security is a continuous process, not a one-time milestone.

Summary

• Containers are not sandboxes, and their default permissions are alarmingly broad: a root user, more than a dozen capabilities, permissive seccomp, a writable filesystem, and no resource limits. Once the application is compromised, a default container is effectively a root-privileged staging point that is only one kernel vulnerability away from the host.
• Running as non-root provides the highest ROI for hardening: one USER instruction, a fixed UID, and K8s runAsNonRoot as a backstop eliminate most privilege-escalation and escape paths. The UID must be fixed and consistent across the image and orchestration layers.
• Minimal images and enforced scanning form the foundation of defense: distroless/scratch reduce the attack surface to a physical minimum, while Trivy/Grype with --exit-code 1 blocks CVEs in CI. The value of scanning lies in automated enforcement, not the report itself.
• seccomp/AppArmor, a read-only rootfs, and resource limits provide the second layer of defense in depth: they cannot prevent the application from being compromised, but they can minimize the blast radius by preventing shell execution, disk writes, and resource exhaustion.
• Security is an ongoing, checklist-driven process, not a one-time milestone: the image, permission, syscall, filesystem, resource, and secret layers each have binary "done or not done" decisions. Review new services once and audit existing services quarterly.

One final sentence: default containers are optimized for "convenience," while production containers must be optimized around the assumption that they have already been compromised. What can an attacker still do after getting in, and what can they no longer do? That is the measure of hardening. Moving every category from "allow by default" to "deny by default" has no finish line, only a continuously maintained checklist.