Managing your terraform across multiple environments

January 1st, 2021 - Dev, Infra, Devops, Software-Development & Tech

You’re managing your cloud infrastructure using Terraform. You’ve got your first environment up and running and you’re already reaping the benefits of a codified infrastructure. Changes are easy. But, now you need to set up a second environment (staging, prod, whatever) and you’re finding that managing this is not straight forward. There’s a bunch of arguments to remember every time you switch between environments, and your switching a lot because you want to keep them in sync. Because this is hard you tend to use auto-complete, but then sometimes you forget to change something and accidentally apply prods config to staging. Well, as in many occasions, a Makefile can probably help you there.

I think I’m with you, but do I really need this?

Terraform is a per-environment tool. i.e. you benefit from terraform by knowing that you’ve applied exactly the same configuration to each of your environments, and that this way you can always keep them in sync. This is benefit numero uno, but it means it’s down to you to figure out how to handle the dynamic parts yourself. Terraform helps at least a little by providing a way to pass in variables by environment variables, command line arguments or tfvars files.

Ok, so why not just use a tfvars file for each environment?

Well of course environment-based tfvars files are a good idea, but the problem is that not all things can (or should) go in them.

Some things that can’t:

Things that shouldn’t: !1!1!anything secret!1!1!.

Let’s talk through an example of why it might be beneficial to wrap up our terraform commands in a Makefile to ensure our environments are kept both safe and distinct:

include  ../envs/$(stage).env

  terraform init -reconfigure \
    -backend-config="key=$${TERRAFORM_STATE_FILE}" \
    -backend-config="bucket=$${TERRAFORM_STATE_BUCKET_NAME}" \

and our env file looks something like:

Here we have an initialisation wrapper. This sets our backend-config based on environment variables stored in an env file. If we have a different state bucket for our dev, staging & production environments, we can easily switch between each environment without the chance of getting anything wrong. Anything secret has to go into protected variables in our CI pipeline.

Why not just put all of this out into one big state file?

Well, you can’t really, otherwise you’re not applying the same Terraform resources to different environments. You could put multiple state files in the same bucket, which avoids the bucket_name & region arguments above, but then you have to figure out how to ensure access to each place. If you are (probably wisely) running your production environments in a different AWS account, then you end up having to give some runners cross-account access. Don’t even start me on which way this access should flow… Nah, just put them in their own buckets in their own environments and save yourself all of that headache. Dev runners access the dev bucket, prod runners access the prod bucket, etc.

If I’m wrapping init like this, should I do it for other commands too?

Up to you, but probably yes. Here’s how I’d set it up:

tf_plan_args = '-out=.terraform.tfplan -var-file="$(stage).tfvars"'
ifdef special_flag
	tf_plan_args += '-var="special_flag=$(special_flag)"'

plan: clear_plan
	terraform plan $(shell echo $(tf_plan_args))

apply: clear_plan
	terraform apply .terraform.tfplan

	rm -f .terraform.tfplan

This ends up with a few handy features:

A slight extension

One pattern I’ve come across a few times now is where you need data in your terraform plan that isn’t managed by terraform. This could be:

In this kind of scenario most things output as JSON, and jq is a great tool to parse this output in your Makefile to pass it into terraform. This is another great reason to codify your terraform commands: you can make sure that this stitching always works the same in all of the places. Here’s an example of how I’d change the Makefile to handle this:

ifndef required_parameter
	$(error You must specify the required parameter.)

tf_plan_args = '-out=.terraform.tfplan -var-file="$(stage).tfvars" -var="required_parameter=$(required_parameter)"'
ifdef special_flag
	tf_plan_args += '-var="special_flag=$(special_flag)"'

plan: clear
    $(shell ./ > .extra_vars.json)
	terraform plan $(shell echo $(tf_plan_args))  -var-file=".extra_vars.json"
	rm -f .terraform.tfplan .extra_vars.json

Caveat: depending on your CI setup, the extra_vars.json may simply be input from a previous step, so the file might just come in as an artifact from a previous step and then it’s down to any manual runs to provide this data manually. This obviously depends on the use-case, so take this as just one approach out of many.

Final thoughts

There are probably other ways to achieve the above. I’ve heard that Terragrunt is a thing, and whilst I haven’t dug into it I assume it’s going to solve these kinds of problems (I’ve avoided it because I dislike dependencies, and favour flexibility). To be honest, how you decide to solve this issue is totally up to you, this is just one way of doing it. However, there are a few important things in my opinion that your solution should solve:

Some hints & tips