Terraform as a backend (in a compiler sense)

Every time I’ve tried to use terraform proper I have failed. Fundamentally it is a non-composable tool so if you want compostionlity you have to work around its limitations. One way I’ve found to force compositionality is to not rely on any of its high level features. I only use it for managing resource graphs. That’s the only part I think it is good at and that’s the only part that consistently works. If you feed it a resource graph then it will mostly work. This means you’ll have to find some way to generate and feed it the resource graph. There are a few ways to do it and in the past I would have recommended ERB but there is actually a simpler way that just requires generating JSON. You could generate the JSON from anything you are comfortable with that provides the kind of compostionality you expect. I’m going to outline how I use Rake and Ruby to do this for a very basic VPC configuration across several regions and availability zones.

It is good practice to namespace Rake files so we start with a namespace

Within the namespace we’re going to define the list of tasks and dependencies. From a very high level it looks as follows

The real work happens within the region iteration block. In this example each region is independent but you can imagine performing more complicated operations with more complicated dependencies between regions like setting up VPC peering

For each region we create the required tasks for directories and files that will hold JSON resource files. We are bypassing all of terraform’s syntactical conventions so we need to generate the graph as a Ruby hash map that we’ll then write out as JSON. NB: This took me a few tries because the JSON schema is not documented like HCL (HashiCorp Language) is documented.

With all of that out of the way we can now fill in the resource hash map. There are two parts: high-level structure and some low-level subnet related transformations

There is a lot of boilerplate because I’m generating everything inline and a slightly more principled approach would be to write some helper methods to generate the hash maps and slot them into the right place in the structure to reduce some of the noise.

If you followed along then you can now generate the directory structure along with the JSON files by running

The output should look like as follows

You can inspect the JSON output and see what it looks like. In my Rake file I also have several tasks for running terraform operations. I’m leaving those as an exercise for the reader.