What is Terraform Backend? Dive into Terraform Backend S3

Terraform manages infrastructure using a state file, which is stored by default as terraform.tfstate. This works fine when working alone, but if multiple people manage the infrastructure simultaneously, conflicts may arise.

To solve this issue, Terraform provides the backend block to configure where the state file is stored.

>> Read more:

• How To Use Terraform for Provisioning A Docker Container?
• How to Migrate Your AWS Workloads from EC2 to ECS?

What is Terraform Backend?

A Terraform backend defines how and where Terraform stores its state file (terraform.tfstate). This state file keeps track of the resources Terraform manages such as AWS instances, Azure services, or any infrastructure created using Terraform. Without it, Terraform won’t know what exists or what needs to be changed.

By default, the state is stored locally on your machine, but that becomes problematic when:

• Multiple team members are working on the same infrastructure.
• You want centralized control, locking, or history/versioning of infrastructure changes.
• You need the state file to persist beyond a single device.

To solve these issues, Terraform supports using remote backends like Amazon S3, Terraform Cloud, Azure Blob Storage, Google Cloud Storage, and more. With a Terraform remote backend, you are allowed:

• Shared access to the same state file
• State locking to prevent simultaneous updates
• Versioning and audit trails
• Better security and scalability

Terraform Backend Config

The backend block is placed at the beginning inside the terraform block. For example: Remote backend with Terraform Cloud:

terraform {
  backend "remote" {
    organization = "learning-journey"

    workspaces {
      name = "demo01"
    }
  }
}

Terraform provides multiple backend options, meaning different locations to store the state file, such as local, remote, s3, azurerm, etc. The default storage location is local.

Backend Types

As mentioned earlier, Terraform supports multiple backend types, but in this article, we will focus on local and s3.

Local

The local backend stores the state file on the user's machine.

Configuration:

terraform {
  backend "local" {
    path = "relative/path/to/terraform.tfstate"
  }
}

path: Specifies where the state file is stored. By default, it is terraform.tfstate in the root module.

While easy to set up, the local backend is not recommended for teams. It doesn’t support locking, remote access, or version control, so conflicts may arise when multiple users manage infrastructure. Therefore, storing the state file in a shared location helps solve these conflicts, which is where the s3 backend comes in.

S3

With this Terraform S3 backend, the state file is stored in an S3 bucket. To ensure consistency, there are two ways to lock the state file:

• S3 state locking (use_lockfile), available in Terraform 1.10 and later.
• DynamoDB (dynamodb_table), where the partition key is LockID with a data type of string.

Example: Create an S3 bucket and a DynamoDB table on AWS.

• Configuration:

terraform {
  backend "s3" {
    bucket         = "mybucket"
    key            = "your-state.tfstate"
    region         = "us-east-1"
    profile        = "Your_Profile"
    dynamodb_table = "states_tb"
  }
}

• If using an AWS profile with S3 access permissions, no additional S3 permissions are required. Otherwise, you must configure permissions for reading and writing objects in S3:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::mybucket"
    },
    {
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:PutObject"],
      "Resource": [
        "arn:aws:s3:::mybucket/path/to/my/key",
        "arn:aws:s3:::mybucket/path/to/my/key.tflock"
      ]
    }
  ]
}

• If using DynamoDB for state locking, you must also set permissions for DynamoDB on AWS (if the profile lacks necessary permissions):

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "dynamodb:DescribeTable",
        "dynamodb:GetItem",
        "dynamodb:PutItem",
        "dynamodb:DeleteItem"
      ],
      "Resource": "arn:aws:dynamodb:*:*:table/mytable"
    }
  ]
}

Practices

This section demonstrates how to configure and verify Terraform remote state storage using Amazon S3, and how to enable state locking using either S3-native lockfile or DynamoDB.

Storing State in S3

• Modify main.tf to include the backend block inside the terraform block:

terraform {
  backend "s3" {
    bucket  = "terraform-learning-demo01"
    key     = "terraform.tfstate"
    region  = "us-east-1"
    profile = "saa"
  }

  ...
}

• Firstly, create an S3 bucket on AWS with a name matching the bucket field in the backend block. Here, we use terraform-learning-demo01. If not, we will encounter this issue:

• Reinitialize the workspace:

terraform init

This initializes the backend and connects Terraform to the specified S3 bucket.

Expected Output:

Initializing the backend...

Successfully configured the backend "s3"! Terraform will automatically
use this backend unless the backend configuration changes.
Initializing modules...
Initializing provider plugins...
- Reusing previous version of hashicorp/aws from the dependency lock file
- Using previously-installed hashicorp/aws v4.67.0

Terraform has been successfully initialized!
...

• Then, run terraform plan to check for changes, followed by terraform apply to deploy the infrastructure.
• After execution, verify in the S3 bucket that the terraform.state (key) file is stored in terraform-learning-demo01 (bucket).

• In the .terraform/terraform.tfstate directory, you will find the information of backend configuration. This confirms that we have successfully configured state storage on S3.

State Locking

S3 State Locking

• To enable S3 state locking, add the use_lockfile parameter (only available in Terraform 1.10+) to the backend block. The main.tf file now looks like this:

terraform {
  backend "s3" {
    bucket       = "terraform-learning-demo01"
    key          = "terraform.tfstate"
    region       = "us-east-1"
    profile      = "saa"
    use_lockfile = true
  }

  ...
}
...

• Modify a resource value (e.g., the subnet CIDR block) and run terraform apply simultaneously. You will see an error in the console:

• A lock file {key}.tflock appears in the S3 bucket, preventing simultaneous apply operations:

• Once the apply operation completes, the lock file is deleted, and the state file will be added in S3:

DynamoDB State Locking

• Create a DynamoDB table named terraform-tfstate with LockID as the partition key of type string.

• If not, this issue will appear: 

• Configure DynamoDB in main.tf:

terraform {
  backend "s3" {
    bucket         = "terraform-learning-demo01"
    key            = "terraform.tfstate"
    region         = "us-east-1"
    profile        = "saa"
    dynamodb_table = "terraform-tfstate"
  }
}

• Running terraform apply simultaneously will result in an error:

• The DynamoDB table contains a record with LockID formatted as {bucket}/{key}.

• Once apply excecuted, file state will be created in S3:

• Once the process completes, the lock record is deleted, but a {bucket}/{key}-md5 file is created:

>> Explore: Top 20 Infrastructure as Code (IaC) Tools For Businesses

Conclusion

• Terraform provides the backend block to define state file storage locations.
• Multiple backend options are available: local, remote, s3, etc.
• The s3 backend supports state locking via S3 native (Terraform 1.10+) or DynamoDB:
  • When applying changes, a .lock file is created in S3 or DynamoDB.
  • Once the process completes, the lock file is removed, and the state file is updated in S3.