The Cloudly Engineer

Terraform is an elite open-source software that helps solve a lot of cloud automations. It’s very scalable and easy to use. But Terraform’s flexibility has caused debates on how to setup a perfect code directory. A code directory should be setup in a way where it’s easy to read, quickly deploy to various environments and accounts without having to repeat or copy and paste code. Terragrunt to the rescue! That helped tremendously but it can become challenging to manage multiple tools or the code directory structure becomes unmanageable in the long run. In this post I like to share another method of Terraform directory setup for AWS multiple accounts and environments. I’m not saying this is the best or better solution than other ones… it’s just another way of doing AWS Multi-Account setup with Terraform.

Old Way

This is the structure I used to use with Terragrunt. Terragrunt would dynamically change the S3 Backend Key parameter based on the directory structure.

├── dev
│   ├── inputs.yml
│   ├── terragrunt.hcl
│   └── vars.tf
├── staging
│   ├── inputs.yml
│   ├── terragrunt.hcl
│   └── vars.tf
├── terragrunt.hcl
├── main.tf

New Way

Now this is the structure I use, this setup doesn’t require Terragrunt. Notice it’s less files and it’s all Terraform files.Let’s review some of the files in detail.

├── accounts
│   ├── dev
│   │   ├── backend.conf
│   │   └── terraform.tfvars
│   └── staging
│       ├── backend.conf
│       └── terraform.tfvars
├── Makefile
├── main.tf

For this setup the main.tf still declares the backend but without any of its parameters! This is called partial configuration in Terraform. We’ll define the rest in the backend.conf file.

terraform {
  backend "s3" {
    key = "tf-partial-backend-example/terraform-main.tfstate"
  }
}

Backend.conf

We need to give all the remaining required and optional parameters of the “s3” backend defined in the main.tf in this backend.conf file. In this example I’m declaring the key path in the main.tf so it will be the same for every S3 bucket. If I wanted it to be unique then I would add it to the backend.conf. You can do that with any of the parameters!

#  Terraform state for S3 backend config variables
bucket   = "dev-example-s3-bucket-tf"
region   = "us-east-1"
role_arn = "arn:aws:iam::123456789:role/svc-terraform-role"

terraform.tfvars

This file defines all the values for your variables for that specific environment or account.

region      = "us-east-1"
environment = "dev"
account_id  = "123456789"

Terraform init, plan, and apply

In this design initiating terraform requires few options. We need to the define the rest of the backend configuration, we do that by passing the -backend-config option. Next option is the -reconfigure, is very important because we’re working with multiple accounts and environments we need to ensure the local cache .terraform* files are well, reconfigured each time. That’s the catch for this pattern!

terraform init -backend-config accounts/dev/backend.conf -reconfigure

# Expected output
Terraform has been successfully initialized!

# Plan
terraform plan -var-file accounts/dev/terraform.tfvars

# Apply
terraform apply -var-file accounts/dev/terraform.tfvars

Makefile

You probably noticed a file called “Makefile” in the directory. A makefile is just a way to shorthand or automate our long commands. Here’s a makefile for Terraform.

SHELL := /usr/bin/env bash

# HOW TO EXECUTE

# Executing Terraform PLAN
#	$ make tf-plan-example env=<env>
#    e.g.,
#       make tf-plan-example env=dev

# Executing Terraform APPLY
#   $ make tf-apply-example env=<env>

# Executing Terraform DESTROY
#	$ make tf-destroy-example env=<env>

all-test: clean tf-plan-example

.PHONY: clean
clean:
	rm -rf .terraform

.PHONY: tf-plan-example
tf-plan-example:
	terraform fmt && terraform init -backend-config accounts/${env}/backend.conf -reconfigure && terraform validate && terraform plan -var-file accounts/${env}/terraform.tfvars

.PHONY: tf-apply-example
tf-apply-example:
	terraform fmt && terraform init -backend-config accounts/${env}/backend.conf -reconfigure && terraform validate && terraform apply -var-file accounts/${env}/terraform.tfvars -auto-approve

.PHONY: tf-destroy-example
tf-destroy-example:
	terraform init -backend-config accounts/${env}/backend.conf -reconfigure && terraform destroy -var-file accounts/${env}/terraform.tfvars

Now all we have to do for TF init is call like this:

make tf-plan-example env=dev

Or for the staging environment like this

make tf-plan-example env=staging

Apply time!

make tf-apply-example env=dev

# aws_ssm_parameter.this will be created
  + resource "aws_ssm_parameter" "this" {
      + arn       = (known after apply)
      + data_type = (known after apply)
      + id        = (known after apply)
      + key_id    = (known after apply)
      + name      = "example-param-dev"
      + tags      = {
          + "Environment" = "dev"
          + "Name"        = "example-param-dev"
          + "Owner"       = "Waleed"
          + "Region"      = "us-east-1"
        }
      + tags_all  = {
          + "Environment" = "dev"
          + "Name"        = "example-param-dev"
          + "Owner"       = "Waleed"
          + "Region"      = "us-east-1"
        }
      + tier      = "Standard"
      + type      = "String"
      + value     = (sensitive value)
      + version   = (known after apply)
    }

Plan: 1 to add, 0 to change, 0 to destroy.
aws_ssm_parameter.this: Creating...
aws_ssm_parameter.this: Creation complete after 0s [id=example-param-dev]

Now let’s deploy the same code in a different region and environment with just a single change in the command line.

make tf-apply-example env=staging

# aws_ssm_parameter.this will be created
  + resource "aws_ssm_parameter" "this" {
      + arn       = (known after apply)
      + data_type = (known after apply)
      + id        = (known after apply)
      + key_id    = (known after apply)
      + name      = "example-param-staging"
      + tags      = {
          + "Environment" = "staging"
          + "Name"        = "example-param-staging"
          + "Owner"       = "Waleed"
          + "Region"      = "us-west-2"
        }
      + tags_all  = {
          + "Environment" = "staging"
          + "Name"        = "example-param-staging"
          + "Owner"       = "Waleed"
          + "Region"      = "us-west-2"
        }
      + tier      = "Standard"
      + type      = "String"
      + value     = (sensitive value)
      + version   = (known after apply)
    }

Plan: 1 to add, 0 to change, 0 to destroy.
aws_ssm_parameter.this: Creating...
aws_ssm_parameter.this: Creation complete after 2s [id=example-param-staging]

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

Now you have applied the same code/resources with different values in different S3 buckets in multiple AWS accounts. Again, this is just another option to do Terraform AWS Multi-Account Setup.

Donate a coffee! 🙂

Learn more about the backend configuration.

Here’s the complete code in Github.