Advanced Terraform: Modules and Remote State

Writing an entire AWS production environment in one main.tf, including the VPC, subnets, RDS, EKS, ALB, IAM, and CloudWatch, will probably produce 2,000 lines. It may work, but when a second project needs the same network foundation, your only option is to copy and paste the entire block. When five team members edit the file at once, merge conflicts can consume an afternoon. When someone commits terraform.tfstate to Git, you discover that the production database password has already been sitting in Git history for three months.

A single Terraform file is enough for a demo, but production requires two things: modules and remote state. These are not optional improvements. They are foundational safeguards whose absence will eventually cause an incident. This article explains module inputs, outputs, and versions; remote state backends; drift detection and imports; sensitive-value protection; and a production-grade project structure you can use directly. I assume you already know init / plan / apply; this article focuses only on advanced topics.

1. From a Single File to Modules: Why You Must Split It Up

First, consider what modularization solves. Single-file IaC encounters four walls in production:

• It cannot be reused. You build a "VPC with a NAT gateway, public and private subnets, and a database subnet group" for Project A. Project B needs the same thing, so you copy 300 lines. When you later find a bug in the subnet design, you must fix it in two places, then five, then ten.
• It cannot be composed. When every resource is piled into one file, a newcomer who wants to understand the environment's major components must read from the first line to the last. Without boundaries, there is no abstraction.
• It cannot be shared. You cannot distribute internal best practices, such as a security group baseline that complies with Multi-Level Protection Scheme requirements, across teams. Every team eventually writes its own implementation, with inconsistent quality.
• Collaboration creates conflicts. When five people edit a 2,000-line file, every PR requires rebasing.

A module is Terraform's equivalent of a function. It encapsulates a related group of resources, exposes only input parameters and output values, and hides implementation details. Callers do not need to know how many route tables or NACL rules the module creates. They only provide arguments:

module "vpc" {
  source = "./modules/vpc"

  cidr           = "10.20.0.0/16"
  azs            = ["ap-northeast-1a", "ap-northeast-1c", "ap-northeast-1d"]
  enable_nat     = true
  environment    = "prod"
}

This call reads like a sentence: "Give me a /16 VPC spanning three AZs, enable NAT, and tag the environment as prod." How many subnets it creates, how it names them, and how it applies tags are the module's responsibility. A good module call site reads like a product requirements document, not implementation details.

The benefits of modularization are the same ones software engineering validated long ago: reuse, encapsulation, layering, and testability. The IaC community simply took ten years longer than application engineering to treat them seriously.

2. The Three Elements of a Module: Variables, Outputs, and Versions

Writing a module is essentially writing a code package with a public API. It has three elements: inputs (variables), outputs, and versions (reproducible source references).

1. Inputs: Variables with Constraints

A production-ready variable must do more than declare a name. An unconstrained input puts the module's correctness at the mercy of its caller. Consider this anti-pattern:

# Anti-pattern: no constraints at all
variable "cidr" {
  type = string
}

The caller can pass "hello" without an error. The failure appears only when the aws_vpc resource reaches the apply stage. The correct approach includes a type, validation, a default value, and a description:

# modules/vpc/variables.tf
variable "cidr" {
  description = "VPC CIDR block; must be a valid private network from /16 to /28"
  type        = string
  default     = "10.0.0.0/16"

  validation {
    condition     = can(cidrhost(var.cidr, 0))
    error_message = "cidr must be a valid IPv4 CIDR, such as 10.0.0.0/16."
  }
}

variable "azs" {
  description = "List of availability zones; at least two are required for high availability"
  type        = list(string)
}

variable "enable_nat" {
  description = "Whether to enable a NAT gateway (strongly recommended for production)"
  type        = bool
  default     = true
}

variable "environment" {
  description = "Environment tag applied to all resources"
  type        = string

  validation {
    condition     = contains(["dev", "staging", "prod"], var.environment)
    error_message = "environment must be dev, staging, or prod."
  }
}

Several details matter. description is not decoration: terraform docs and IDE hover help both depend on it, and whether you write it directly determines whether the module is usable. A validation block moves an entire class of basic errors from apply to plan, or even init, for an excellent return on effort. Add sensitive = true to sensitive inputs so their values do not appear in plan logs.

2. Outputs: The Communication Channel Between Modules

Resources inside a module are invisible to callers unless they are explicitly exposed through outputs. If the caller needs the subnet IDs, VPC ID, or security group IDs created by a module, outputs must expose them:

# modules/vpc/outputs.tf
output "vpc_id" {
  description = "ID of the created VPC"
  value       = aws_vpc.this.id
}

output "private_subnet_ids" {
  description = "List of private subnet IDs for EKS / RDS"
  value       = aws_subnet.private[*].id
}

output "public_subnet_ids" {
  description = "List of public subnet IDs for ALB"
  value       = aws_subnet.public[*].id
}

The caller consumes them like this:

module "vpc" { ... }

module "eks" {
  source = "./modules/eks"

  subnet_ids = module.vpc.private_subnet_ids   # ← Modules are connected through outputs
  vpc_id     = module.vpc.vpc_id
}

Modules must never reference another module's internal aws_xxx resources. Decouple them exclusively through outputs. Then, if you later replace a custom EKS module with a community module, the call site requires no changes as long as the output names remain compatible.

3. Versions: source Determines Which Code You Are Actually Running

source is the module's dependency coordinate. It takes several forms, with dramatically different levels of reproducibility:

# 1. Local path: shared within the team and versioned with the root repository
source = "./modules/vpc"

# 2. Git: shared across repositories, but omitting a pinned ref is a time bomb
source = "git::ssh://[email protected]/myorg/tf-modules.git//vpc?ref=v1.2.0"

# 3. Terraform Registry / private registry: semantic versions
source  = "terraform-aws-modules/vpc/aws"
version = "5.1.0"

The biggest trap is a Git source without a ref. I have seen more than one team write source = "git::...//vpc" without ?ref=, which fetches the latest code from the default branch. One day, someone changes the module's implementation, and every environment using it produces a diff on its next apply, often a diff nobody understands. Production infrastructure code must be reproducible. ?ref=v1.2.0, or a commit SHA, is the minimum standard.

Registry modules use semantic versions. "5.1.0" pins an exact version, while "~> 5.1" permits 5.1.x patches. For production, I strongly recommend pinning an exact version with version = "5.1.0". Upgrade through a PR by changing the version explicitly. This turns a dependency upgrade into a reviewable action instead of something that happens silently during terraform init.

The .terraform.lock.hcl file generated after init records exact provider versions. Commit this file to Git. It is the lockfile for provider dependencies and serves the same purpose as Node's pnpm-lock.yaml. Do not ignore it.

3. Remote State: Why Local State Is Not Suitable for Production

Terraform uses the terraform.tfstate JSON file to remember "what I actually created in the cloud." This file is the source of truth, not a cache or a log. Terraform compares it with the code to calculate a diff.

Keeping state locally, with terraform.tfstate in the working directory, is acceptable for one person working alone. Once a team or CI is involved, it becomes the starting point for a chain of incidents:

• Collaborative changes overwrite each other. Engineer A changes the infrastructure, but the state remains on A's machine. Engineer B applies from another machine using stale state, potentially overwriting A's changes or encountering an error. Nobody knows who has the latest state.
• CI cannot run correctly. The pipeline machine does not have the terraform.tfstate from your computer, so every plan proposes rebuilding everything and is therefore meaningless.
• Committing it to Git leaks secrets. A state file stores every attribute of every resource in plaintext, including RDS passwords, ALB access-log bucket names, and IAM policy details. Once it enters Git, you have effectively handed over the keys to production.
• Concurrent writes corrupt state. If two people apply simultaneously, both processes write the state file and overwrite one another. The state becomes corrupted, plan and apply begin failing, and recovery is extremely painful.

The conclusion is uncompromising: production state must use a remote backend with centralized storage, locking, encryption, and access control. Local state is suitable only for a five-minute demo.

Option A: S3 with DynamoDB Locking, the Most Common Self-Hosted Approach

The most common, inexpensive, and battle-tested combination on AWS uses S3 to store state and DynamoDB for a distributed lock that prevents concurrent writes. This backend must itself be created once with Terraform, a process called "bootstrap." Keep it in a separate bootstrap/ directory and manage it manually to avoid the chicken-and-egg problem of trying to use a backend before it exists.

First, create the storage and lock table. This is a one-time operation that can be applied manually:

# bootstrap/main.tf - infrastructure for the backend itself; state can temporarily remain local
resource "aws_s3_bucket" "tf_state" {
  bucket = "myorg-tfstate-prod-9f3k2"   # Globally unique
}

resource "aws_s3_bucket_versioning" "tf_state" {
  bucket = aws_s3_bucket.tf_state.id
  versioning_configuration { status = "Enabled" }   # Versioning is mandatory; rollback depends on it
}

resource "aws_s3_bucket_server_side_encryption_configuration" "tf_state" {
  bucket = aws_s3_bucket.tf_state.id
  rule {
    apply_server_side_encryption_by_default { sse_algorithm = "aws:kms" }   # Enforce KMS encryption
  }
}

resource "aws_s3_bucket_public_access_block" "tf_state" {
  bucket                  = aws_s3_bucket.tf_state.id
  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}

resource "aws_dynamodb_table" "tf_locks" {
  name         = "tf-locks"
  billing_mode = "PAY_PER_REQUEST"
  hash_key     = "LockID"                # This key name is fixed by the backend contract
  attribute {
    name = "LockID"
    type = "S"
  }
}

Then configure the backend in the application project:

# envs/prod/backend.tf
terraform {
  required_version = ">= 1.7.0"

  backend "s3" {
    bucket         = "myorg-tfstate-prod-9f3k2"
    key            = "prod/web-app/terraform.tfstate"   # One key per environment and component
    region         = "ap-northeast-1"
    dynamodb_table = "tf-locks"                          # Points to the lock table; apply acquires and releases the lock automatically
    encrypt        = true                                # Enforce encryption again, even though the S3 bucket is already encrypted
  }
}

Several details cannot be omitted: S3 versioning must be enabled, because rolling back to a previous version is the lifeline when state is corrupted; use KMS encryption instead of the default SSE-S3; enable every public access block option, because a state bucket must never be public; and isolate each environment and component by key. Give prod/web-app/, prod/db/, and staging/web-app/ separate keys instead of putting everything in one state file, where a single failure affects everything.

The lock works as follows: before apply, Terraform writes a LockID record to DynamoDB and deletes it when apply finishes. If Engineer B tries to apply while Engineer A holds the lock, Terraform finds the existing lock and rejects the operation, preventing concurrent writes. This lock applies only to write operations such as apply. By default, plan does not acquire it, so plans running in CI are safe and do not block one another.

Option B: Terraform Cloud / Enterprise

If you do not want to manage S3 and a lock table yourself, or you need more complete collaboration features such as audit logs, RBAC, VCS integration, and remote execution, use HashiCorp Terraform Cloud. Its free allowance is sufficient for small teams. Replace the backend configuration with cloud:

terraform {
  cloud {
    organization = "myorg"

    workspaces {
      name = "prod-web-app"   # One workspace corresponds to one state
    }
  }
}

State resides in HashiCorp-managed encrypted storage, with locking, encryption, versioning, and auditing built in. Apply can also run in the cloud through remote execution, so the local machine does not need AWS credentials. The tradeoffs are a SaaS dependency, less flexibility, and fees as the organization grows. My recommendation: choose Terraform Cloud for teams of three or fewer that have strict audit requirements and do not want to operate a backend; for everything else, S3 with DynamoDB is a dependable default.

4. State Drift: The Cost of Manual Changes and How to Repair Them

Drift means that the actual state of cloud resources differs from what the state file records. It has many causes. The most common is someone manually changing something in the AWS console: "I am only adding one security group rule, so I cannot be bothered to open a PR." Other causes include emergency console hotfixes, scripts that modify resources, and even changes to AWS defaults.

The frightening part of drift is that it is silent. The code has not changed and the state has not changed, but reality has. During the next apply, Terraform confidently changes reality back to match the code, removes the emergency hotfix, and triggers alerts at 2 a.m.

Detecting Drift: Treat Plan as a Health Check

terraform plan is the only health check available. It compares three things: code, state, and actual cloud resources. Any mismatch produces a diff. The production discipline is therefore:

terraform plan -detailed-exitcode
# Exit codes: 0 = no changes; 1 = error; 2 = changes present (drift or code changes detected)

Integrate -detailed-exitcode into CI and run a scheduled plan every day, not only on PRs. Send an alert whenever the exit code is nonzero. This is drift detection: discover that reality and code disagree before your users or your manager do. I have seen teams use this mechanism to catch operations staff quietly opening ports in the console and to detect a compromised IAM key being added to a high-risk policy. Drift detection is also a form of security monitoring.

Repairing Drift: Two Paths

How should you repair detected drift? Decide whether code should accommodate reality or reality should conform to code.

Path one: reality is correct, so change the code. Suppose someone manually fixed a security group rule during an incident and the fix is valid. Write that change back into the .tf files until terraform plan reports No changes. At that point, code, state, and reality are aligned again.

Path two: the code is correct, so restore reality. The manual change was a mistake. Run terraform apply, and Terraform changes the cloud resource back to match the code.

import: Bring Existing Resources Under State Management

The hardest form of drift is when a resource exists in the cloud but is entirely absent from state, which is typical when bringing legacy resources under management. A direct apply reports that the resource already exists because Terraform tries to create it.

Terraform 1.5+ introduced declarative import blocks. This is now the recommended approach because it can generate configuration code automatically:

# First declare which resources to import
import {
  to = aws_s3_bucket.assets
  id = "my-app-assets-prod-9f3k2"
}

import {
  to = aws_db_instance.main
  id = "prod-db-2024"
}

terraform plan -generate-config-out=generated.tf   # Generate resource configuration automatically
terraform plan                                      # Verify No changes
# Then remove the import blocks, clean up generated.tf, and merge it into the permanent .tf files

The older imperative command terraform import aws_s3_bucket.assets my-app-assets-prod-9f3k2 still works, but it does not generate code. You must write configuration manually until plan reports No changes, which is much more painful. A successful import ends with terraform plan reporting No changes. Your infrastructure matches the configuration. Until then, the resource is not fully under management.

Rule of thumb: import is a process of making code accommodate reality. Review the generated configuration line by line to ensure it meets your security and naming standards rather than committing it blindly. Generated code often carries historical baggage, such as obsolete encryption settings or unnecessary fields, making this an ideal time to clean it up.

5. Sensitive Values and State: That File Is a Gold Mine of Secrets

This is the trap most likely to catch beginners without them realizing it: terraform.tfstate stores every attribute of every resource, including the ones you consider sensitive.

If an RDS resource contains password = var.db_password, or a Lambda environment variable contains API_KEY = var.api_key, those values appear in plaintext in the state file. Terraform cannot determine which attributes are sensitive because every provider has different semantics. Its design is simply to record everything in state.

Many people believe sensitive = true makes the value secure. That is a dangerous misconception:

variable "db_password" {
  type      = string
  sensitive = true
}

sensitive = true only affects terminal output from plan and apply, displaying (sensitive value) so the value does not enter terminal logs. The value remains plaintext in the state file. In other words, it prevents a colleague from glancing at your screen, but it does not protect against a leaked state file. Those threats differ by an entire security tier.

Protect sensitive values with four layers of defense:

1. Enforce encryption on the state backend. S3 KMS encryption or Terraform Cloud managed encryption ensures that state is not stored as plaintext at rest. This is the minimum requirement.
2. Apply strict access control to the state backend. Grant access to the state bucket only to a limited set of CI roles and a few SREs, not the entire company. A single Allow for Resource: arn:aws:s3:::myorg-tfstate-*/* can expose the complete details of production to everyone in the organization.
3. Never commit state files to Git. Add *.tfstate, *.tfstate.*, and crash.log to .gitignore. Every character in that line was learned through painful experience.
4. Read actual secrets from a secret manager instead of letting Terraform handle them. The safest approach is for Terraform to create only the container for a secret, such as an empty aws_secretsmanager_secret resource. The application retrieves the actual value from Secrets Manager, Vault, or SSM Parameter Store at startup. Terraform never holds the plaintext secret, so it never appears in state and the leakage risk falls to zero.

The fourth layer is the only one that lets you sleep soundly. The first three reduce the probability; the fourth eliminates the possibility. Secrets should never appear at any stage of IaC. That is an architectural principle, not a configuration trick.

6. Production-Grade Project Structure: modules/ + envs/

After all that theory, here is a production-grade directory structure you can use directly. Its central principle is that modules/ contains reusable libraries, while envs/ contains the assembly root for each environment, with responsibilities kept separate.

infra/
├── modules/                       # Reusable modules: pure abstractions with no environment-specific data
│   ├── vpc/
│   │   ├── main.tf
│   │   ├── variables.tf
│   │   ├── outputs.tf
│   │   └── versions.tf
│   ├── eks/
│   │   ├── main.tf
│   │   ├── variables.tf
│   │   └── outputs.tf
│   └── rds/
│       ├── main.tf
│       ├── variables.tf
│       └── outputs.tf
│
├── envs/                          # Assembly roots for each environment
│   ├── dev/
│   │   ├── backend.tf             # Remote state configuration for dev (dedicated S3 key)
│   │   ├── main.tf                # Composes the modules
│   │   ├── variables.tf           # Concrete parameters for dev
│   │   ├── providers.tf
│   │   └── terraform.tfvars       # Actual values for dev
│   │
│   └── prod/
│       ├── backend.tf             # Remote state for prod (dedicated S3 key + dedicated lock)
│       ├── main.tf
│       ├── variables.tf
│       ├── providers.tf
│       └── terraform.tfvars
│
├── bootstrap/                     # Code that creates the state backend itself (S3 bucket + DynamoDB table)
│   └── main.tf
│
└── README.md

Several principles are critical:

• Never put environment-specific data in a module. Do not include the word prod, hard-coded CIDRs, or account IDs. Inject every difference through variables; this is what makes the module reusable.
• Give every environment its own directory and independent state. Set key = "dev/terraform.tfstate" in envs/dev/backend.tf and key = "prod/terraform.tfstate" in envs/prod/backend.tf. The dev and prod states are physically isolated, so a dev failure cannot damage prod.
• Keep the assembly root extremely thin. envs/prod/main.tf should contain little more than calls such as module "vpc" { ... } and module "eks" { ... }, plus output connections between modules. Keep all resource implementation details inside modules.

Here is a typical envs/prod/main.tf:

# envs/prod/main.tf - assembly only; no resource implementation details
module "vpc" {
  source = "../../modules/vpc"

  cidr        = "10.20.0.0/16"
  azs         = ["ap-northeast-1a", "ap-northeast-1c", "ap-northeast-1d"]
  environment = "prod"
}

module "eks" {
  source = "../../modules/eks"

  cluster_name    = "prod-cluster"
  k8s_version     = "1.30"
  vpc_id          = module.vpc.vpc_id
  subnet_ids      = module.vpc.private_subnet_ids
  node_min        = 3
  node_max        = 20
  node_desired    = 5
}

module "rds" {
  source = "../../modules/rds"

  instance_class   = "db.r6g.large"     # Use production-grade capacity in prod
  allocated_storage = 200
  subnet_ids        = module.vpc.database_subnet_ids
  security_group_ids = [module.vpc.rds_sg_id]
}

This file reads like a configuration inventory for prod: database size, node count, and database instance class. A new engineer can understand the environment's shape in ten minutes, which was unimaginable in the single-file era. Need another environment? Copy envs/dev/, rename it to staging/, change a few parameters, and apply.

7. Workspaces vs. Environment Directories: Which Should You Choose?

One long-running debate remains: should multiple environments use Terraform Workspaces or separate environment directories, such as envs/dev and envs/prod above?

Workspaces are used like this:

terraform workspace new dev
terraform workspace new prod
terraform workspace select prod
terraform apply    # Apply within the prod workspace

The same .tf code runs in every workspace, using the built-in terraform.workspace variable to switch among them. The selling point is "one codebase for multiple environments, keeping things DRY." It sounds appealing, but I do not recommend using workspaces to separate production environments, for three reasons:

• State shares one backend configuration. Workspaces name environment states ${workspace-name}/${key}, but they use the same backend configuration. This means dev and prod state may physically reside in the same S3 bucket with very similar keys. If permissions are too broad, dev can accidentally read prod. Environment directories provide natural isolation.
• Configuration differences become tangled ternary expressions. If prod EKS uses r6g.large while dev uses t3.medium, the code must contain terraform.workspace == "prod" ? "r6g.large" : "t3.medium". As the number of environments and differences grows, the code fills with if-else logic and readability collapses. With environment directories, envs/prod/terraform.tfvars has one value and envs/dev/terraform.tfvars has another, keeping both clean.
• Production should not be easy to switch into. A single terraform workspace select prod command switches to production, making mistakes too easy. Environment directories require cd envs/prod, which at least provides a physical reminder. IAM permissions for the prod directory can also be restricted so only CI and a few SREs can apply.

Workspaces are genuinely suitable for temporary environments and horizontal replication of identical environments, such as automatically creating a review environment for every PR or giving every developer a sandbox. In scenarios with identical configuration, separate instances, and short lifecycles, the DRY advantage of workspaces becomes useful.

Conclusion: use separate directories for long-lived environments such as dev, staging, and prod, where configurations and blast radii differ. Use workspaces for numerous short-lived, identical temporary environments. This is the consensus the Terraform community has converged on over the years; there is no need to keep switching sides on this question.

Summary

• Modules mark the transition from ad hoc IaC to engineered IaC. A module encapsulates resources as a "function" with inputs, outputs, and a version, enabling reuse, composition, and sharing. A sound module constrains and validates variables, treats outputs as a stable public API, and uses ?ref= or an exact version to guarantee reproducibility.
• Production state must use a remote backend. Local state inevitably causes problems in team and CI workflows, including overwritten changes, concurrent corruption, and leaks through Git. S3 with DynamoDB locking is the standard self-hosted approach, where versioning, KMS encryption, and public access blocking are all mandatory. Terraform Cloud is the managed alternative that requires no backend operations.
• Drift is a silent killer, and plan is the only health check. Run terraform plan -detailed-exitcode regularly in CI and alert on nonzero exit codes. When drift appears, either write reality back into the code or apply the code to restore reality. Use declarative import blocks to bring existing resources under management, and do not consider the process complete until plan reports No changes.
• A state file is a gold mine of secrets and requires layered protection. sensitive = true only protects terminal output, not leaked state. The real defenses are backend encryption, strict IAM, never committing state to Git, and keeping secrets in a secret manager so Terraform never handles them. The last layer is the only one that completely removes the risk.
• Project structure: use modules/ as the library and envs/ as assembly roots, with independent state for every environment. Use directories for dev, staging, and prod instead of workspaces. Reserve workspaces for numerous temporary, identical review or sandbox environments.

In one sentence: Production Terraform maturity is determined not by how many .tf files you write, but by how deeply you protect state, how cleanly you encapsulate modules, and how thoroughly you isolate environments. A working single file is only the starting point. Modules, remote state, drift discipline, and project structure are what move IaC from "it runs" to "we trust it in production."