From Terraform in Action by Scott Winkler

This article discusses how we can improve a Mad Libs generator using expressions.


Take 37% off Terraform in Action by entering fccwinkler into the discount code box at checkout at manning.com.


We can generate a Mad Libs by templating files with a randomized pool of words and outputting the result to a file. Our solution is reasonably extendable, and it’s easy to envision introducing new template files or words to the word pool. This pattern isn’t only useful for Mad Libs, you could also apply it to templating server init scripts and JSON files for external consumption.

Let’s now suppose our Mad Libs engine becomes enormously popular around the office and suddenly everyone is asking for more Mad Libs. Maybe we now want to create not one, but a hundred such Mad Libs, using a variety of template files, and then zip them all together to can send them more easily to our coworkers to enjoy at their leisure.

Our initial solution is decent but has a number of deficiencies that we’d like to address, namely:

  1. Only one Mad Libs can be created at a time
  2. We can’t use templates besides alice.txt
  3. It’s not obvious which words are template words
  4. We don’t yet have a way to compress files into a single zip

With these things in mind, let’s engineer a more advanced solution which solves all of these problems. Our revised architecture’s pictured in figure 1.


Figure 1. Revised architecture for Mad Libs templating engine


As indicated in the architecture diagram, instead of creating a single Mad Libs file, we’ll create one hundred Mad Libs files, and instead of being able to read from only one template, we’ll read from three different templates in a round robin fashion. We’ll also add a step for uppercasing all the words in each list before shuffling them to make them easier to pick out from the surrounding text. Finally, we’ll zip up all the Mad Libs into a single archive file which we can share more easily with our coworkers. When we’re done, the Mad Libs template engine will be much more robust and extendable. Along the way, we’ll see many of the expressive language features of HCL which you’ll want to incorporate in your own Terraform projects.

In particular, we’ll perform the following:

  1. Uppercase each word in var.words with a for expression
  2. Scale resources dynamically with count
  3. Template better way with templatefile()
  4. Compress files with an archive_file data source

Getting Fancy with For Expressions

 We’ll start from the left side of the architecture diagram and work our way rightwards. The first step requires us to uppercase all the words in the list before shuffling them. Although we could perform the uppercase operation at any time before the templating function, it makes sense to do it earlier in the process rather than later because we can cache the result in a local value to improve performance. Local values are conceptually like variables declared within a method, useful for temporarily storing the results of a calculation before using it elsewhere.

Before we set a local value, I should mention for expressions and how they can be used to uppercase each word in the list. A for expression outputs a complex type by transforming some other complex type. They work a lot like streams and lambda expressions in other programming languages (e.g. by performing arbitrary operations on each element of a map or list and outputting a transformed result). Figure 2 shows the syntax of a for expression which uppercases each element in an array of strings, and outputs the result as a new list.


Figure 2. Syntax of for expression which uppercases each word in a list


The visualization of what the for expression from figure 2 is doing is shown in figure 3.


Figure 3. Visualization of for expression


The brackets which go around the for expression determine the type of the output. The previous code used [], which means the output forms a list. If instead we used {}, then the result is an object. For example, if we wanted to loop through the words map and output a new map with the key being that of the original map, and the value being the length of the original value, we could do that easily with the following expression:


Figure 4. Syntax of for expression which iterates over var.words and outputs a map


The visualization of what this for expression in figure 4 is doing is shown in figure 5.


Figure 5. Visualization of for expression


For expressions are useful not only because they can convert one type to another, but because they can be nested and chained together. To make our for expression which uppercases each word in var.words, we’re going to chain the previous two for expressions we built into one mega for expression. Although making your code more complicated is generally a violation of clean code principles, which aims to mitigate and manage software complexity, sometimes the tradeoff is worth it.

TIP   Making your code cleaner and easier to understand is a principle of clean code. Using nested/chained expressions hurts readability and increases cyclomatic complexity; use them sparingly.

Our general logic is as follows:

  1. Loop through each key-value pair in the words map
  2. Uppercase each word in the list corresponding to the value of the key-value pair
  3. Filter out any lists which match the key “numbers” because numbers don’t need to be capitalized – although it’s safe to do this (i.e. uppercase("1") => 1).
  4. Output the new map and save it to a local value

Looping through each key-value pair in the words map, and outputting a new map (1) can be done with the following expressions:

 
 {for k,v in var.words : k => v }
  

Likewise, an expression which uppercases each word in a list and outputs a new list is (2):

   [for s in v : upper(s)]
  

By combining the above two expressions we get:

 
 {for k,v in var.words : k => [for s in v : upper(s)]}
  

This can still be improved upon. It doesn’t make sense to uppercase numbers, and we could filter out any key which matches “numbers”, and doesn’t include this key-value pair in the output map. We can do this by making use of the optional “if” clause, which acts as syntactic sugar when conditionally filtering results in a for expression (3):

 
 {for k,v in var.words : k => [for s in v : upper(s)] if k != "numbers"}
  

Finally, we can output the result in a local value to reference it later (4):

 
  locals {
   uppercase_words = {for k,v in var.words : k => [for s in v : upper(s)] if k!= "numbers"}
 }
  

Local values are declared in a code block having label “locals”. You can declare as many local values as you want in the locals’ code block, and you can have more than one locals’ block in the same module scope. Any type (primitive or complex) that can be stored in an input or interpolation variable can also be stored in a local value.

Add in the new local value to madlibs.tf and update the reference of all the random_shuffle resources to point to the uppercase_words local value instead of var.words. Listing 1 shows what your code should now look like.

Listing 1. Contents of madlibs.tf

 
 terraform {
   required_version = "~> 0.12"
   required_providers {
     random = "~> 2.1"
     template = "~> 2.1"
     local = "~> 1.2"
   }
 }
  
 variable "words" {
     default = {
         nouns = ["army", "panther", "walnuts", "sandwich", "Zeus", "banana", "cat", "jellyfish", "jigsaw", "violin", "milk", "sun"]
         adjectives = ["bitter", "sticky", "thundering", "abundant", "chubby", "grumpy"]
         verbs = ["run", "dance", "love", "respect", "kicked", "baked"]
         adverbs = ["delicately", "beautifully", "quickly", "truthfully", "wearily"]
         numbers = [42, 27, 101, 73, -5, 0]
     }
     description = "A word pool to use for Mad Libs"
     type = map(list(string))
 }
  
 locals {
   uppercase_words = {for k,v in var.words : k => [for s in v : upper(s)] if k!= "numbers"} #A
 }
  
 resource "random_shuffle" "random_nouns" {
   input = local.uppercase_words["nouns"] #B
 }
  
 resource "random_shuffle" "random_adjectives" {
   input = local.uppercase_words["adjectives"] #B
 }
  
 resource "random_shuffle" "random_verbs" {
     input = local.uppercase_words["verbs"] #B
 }
  
 resource "random_shuffle" "random_adverbs" {
   input = local.uppercase_words["adverbs"] #B
 }
  
 resource "random_shuffle" "random_numbers" {
   input = var.words["numbers"] #C
 }
  

#A expression to uppercase each word in the var.words and save to a local value

#B input for random_shuffle now comes from locals.uppercase_words

#C numbers don’t need to be uppercased

Implicit Dependencies

It’s important to point out that because we’re using an interpolated variable to set the input attribute of the random_shuffle resources to the local value “uppercase_words”, we are in fact specifying an implicit dependency between the two elements. An implicit dependency on random_shuffle means that it won’t be created until after local value has been computed. If we were to examine the dependency graph for what we have right now, it’d look like figure 6.


Figure 6. Visualizing the dependency graph and execution order


Nodes nearer the bottom of the dependency graph have fewer dependencies, but nodes nearer the top have more dependencies. Cyclical dependencies aren’t allowed. In figure 6, var.words is a node with no dependencies; in contrast, the root node at the top depends on everything. Execution order for an apply starts at the bottom and works its way up, and nodes with fewer dependencies are created first but nodes having more dependencies are created last. In a destroy operation the order is reversed: nodes at the top of the graph are destroyed first but nodes at the bottom are destroyed last. Finally, Terraform by default processes multiple actions in parallel (although this behavior can be overridden by setting the parallelism flag), and you can’t guarantee any ordering between nodes at the same level in the dependency graph.

NOTE  These dependency graphs quickly become confusing when developing non-trivial projects. I don’t find them to be useful except in the academic sense.

Scaling Resources by Incrementing Count

If we’re going to make one hundred Mad Libs files, the brute force, ham-handed approach’s to copy our existing code one hundred times and call it a day. I don’t recommend doing this because it’s messy, doesn’t scale well, and doesn’t work at all when you don’t know the quantity of resources needed ahead of time. Fortunately, we have better options. For our particular use case, we’ll take advantage of the optional count meta argument on resources.

Count is a meta argument, which means all resources intrinsically support it by virtue of being a Terraform resource. The address of a managed resource uses the format: <RESOURCE TYPE>.<NAME>, and if the count argument is set then the value of this expression becomes a list of objects representing all possible instances. If we wanted to read the Nth instance in the list, we could do it with square bracket notation, e.g. <RESOURCE TYPE>.<NAME>[N]. This concept is illustrated in figure 7.


Figure 7. Count creates a list of resources that can be referenced using bracket notation


NOTE  You can combine count with a conditional expression to toggle whether you want a resource to be created (e.g.  count = var.shuffle_enabled ? 1 : 0)

Let’s update our code to support producing an arbitrary number of mad libs. First add a new variable named “num_files”, having type number and a default value of one hundred. Next, reference this variable to dynamically set the count meta argument on each of the shuffle_resources. Your code will look like Listing 2. Note that some older code has been omitted, but the context remains the same.

Listing 2. Contents of madlibs.tf

 …
 variable "words" { … }
  
 locals {
   uppercase_words = {for k,v in var.words : k => [for s in v : upper(s)] if k!= "numbers"}
 }
  
 variable "num_files" { #A
     default = 100
     type = number
 }
  
 resource "random_shuffle" "random_nouns" {
   count = var.num_files #B
   input = local.uppercase_words["nouns"]
 }
  
 resource "random_shuffle" "random_adjectives" {
   count = var.num_files  #B
   input = local.uppercase_words["adjectives"]
 }
  
 resource "random_shuffle" "random_verbs" {
     count = var.num_files #B
     input = local.uppercase_words["verbs"]
 }
  
 resource "random_shuffle" "random_adverbs" {
   count = var.num_files #B
   input = local.uppercase_words["adverbs"]
 }
  
 resource "random_shuffle" "random_numbers" {
   count = var.num_files #B
   input = var.words["numbers"]
 }

#A declaring an input variable for setting count on the random_shuffle resources

#B referencing the num_files variable to dynamically set the count meta argument

A Better Way to Template

We’ve already seen how to template files using the template_file data source. A more versatile and powerful way to template files uses the built-in templatefile function. This function works similar to the template_file data source, except it allows you to pass in complex types as variable values, not only primitives. The syntax of templatefile() is showcased in figure 8.


Figure 8. Syntax of the templatefile function


Update madlibs.tf to include a local_file resource with the templatefile function to set the content. Your code will look like Listing 3.

Listing 3. Contents of madlibs.tf

 
 …
 variable "words" { … }
  
 locals {
   uppercase_words = {for k,v in var.words : k => [for s in v : upper(s)] if k!= "numbers"}
 }
  
 variable "num_files" { #A
     default = 100
     type = number
 }
  
 resource "random_shuffle" "random_nouns" {
   count = var.num_files #B
   input = local.uppercase_words["nouns"]
 }
  
 resource "random_shuffle" "random_adjectives" {
   count = var.num_files  #B
   input = local.uppercase_words["adjectives"]
 }
  
 resource "random_shuffle" "random_verbs" {
     count = var.num_files #B
     input = local.uppercase_words["verbs"]
 }
  
 resource "random_shuffle" "random_adverbs" {
   count = var.num_files #B
   input = local.uppercase_words["adverbs"]
 }
  
 resource "random_shuffle" "random_numbers" {
   count = var.num_files
   input = var.words["numbers"]
 }
  
 resource "local_file" "mad_lib" {
   count = var.num_files #A
   content = templatefile("alice.txt", #B
     {
         nouns=random_shuffle.random_nouns[count.index].result #C
         adjectives=random_shuffle.random_adjectives[count.index].result #C
         verbs=random_shuffle.random_verbs[count.index].result #C
         adverbs=random_shuffle.random_adverbs[count.index].result #C
         numbers=random_shuffle.random_numbers[count.index].result #C
     })
     filename = "madlibs/madlib-${count.index}.txt" #D
 }
  

#A this is how we create 100 Mad Libs files

#B the path of templatefile() is hardcoded to alice.txt for now

#C constructing an anonymous map to use as input

#D give the file a unique name with count parameter

A couple of interesting things are going on here. First, we adjusted the local_file resource to create multiple resources using the count parameter. This allows us to create one hundred Mad Libs files, or as many as we indicate with var.num_files. Next, we set the content attribute to the result of templatefile(). The templatefile function is hardcoded to use the template at relative path: ./alice.txt, and a variables map containing the result of the shuffled lists.

The expression “count.index” is how to reference the current index of a resource when counts is set. We use the current index to select the corresponding element from the random_shuffle resources (e.g. random_shuffle.random_verbs[count.index].result). This is done to ensure that each Mad Libs file gets its own distinct pool of randomized words.

Our template file needs to be updated to reflect the changes that have occurred in our configuration code. Edit alice.txt to look like Listing 4.

Listing 4. Contents of alice.txt

 
 ALICE'S UPSIDE-DOWN WORLD
  
 Lewis Carroll's classic, "Alice's Adventures in Wonderland", as well as
 its ${adjectives[0]} sequel, "Through the Looking ${nouns[0]}",
 have enchanted both the young and old ${nouns[1]}s for the
 last ${numbers[0]} years, Alice's ${adjectives[1]} adventures begin when
 she ${verbs[0]}s down a/an ${adjectives[2]} hole and lands
 in a strange and topsy-turvy ${nouns[2]}. There she discovers she
 can become a tall ${nouns[3]} or a small ${nouns[4]} simply by
 nibbling on alternate sides of a magic ${nouns[5]}. In her travels
 through Wonderland, Alice ${verbs[1]}s such remarkable
 characters as the White ${nouns[6]}, the ${adjectives[3]} Hatter,
 the Cheshire ${nouns[7]}, and even the Queen of ${nouns[8]}s.
 Unfortunately, Alice's adventures come to a/an ${adjectives[4]} end
 when Alice awakens from her ${nouns[8]}.
  

At this point you could try planning and applying the code; it will generate one hundred Mad Libs files in the /madlibs directory using the Alice in Wonderland template. Why stop here though? Let’s go crazy by adding two more Mad Libs templates! After all, who doesn’t like more Mad Libs? First create a template file called observatory.txt and set the contents to Listing 5:

Listing 5. Contents of observatory.txt

 
 THE OBSERVATORY
  
 Out class when on a field trip to a ${adjectives[0]} observatory.
 It was located on top of a ${nouns[0]}, and it looked like a giant
 ${nouns[1]} with a slit down its ${nouns[2]}. We went inside
 and looked through a ${nouns[3]} and were able to see
 ${nouns[4]}s in the sky that were millions of ${nouns[5]}s
 away. The men and women who ${verbs[0]} in the observatory
 are called ${nouns[6]}s, and they are always watching for
 comets, eclipses, and shooting ${nouns[7]}s. An eclipse occurs
 when a ${nouns[8]} comes between the earth and the ${nouns[9]}
 and everything gets ${adjectives[1]}. Next week, we place to
 ${verbs[1]} the Museum of Modern ${nouns[10]}.
  

Next, create another template file called photographer.txt and set the contents to Listing 6:

Listing 6. Contents of photographer.txt

 
 HOW TO BE A PHOTOGRAPHER
  
 Many ${adjectives[0]} photographers make big money photographing
 ${nouns[0]}s and beautiful ${nouns[1]}s. They sell the prints
 to ${adjectives[1]} magazines or to agencies who use them in
 ${nouns[2]} advertisements. To be a photographer, you have to
 have a ${nouns[3]} camera. You also need an ${adjectives[2]}
 meter and filters and a special close-up ${nouns[4]}. Then you
 either hire professional ${nouns[1]}s or go out and snap candid
 pictures of ordinary ${nouns[5]}s. But if you want to have a
 career, you must study very ${adverbs[0]} for at least ${numbers[0]} years.
  

Move the three template files into a new folder called “templates”, to get them out of the way.  At this point we’ll need to toggle between the three templates to create equal numbers of each. Add a variable named “templates”, with the default value being a list containing the relative paths of the three template files. Next, modify the path argument of templatefile() to an expression which selects one of the values from the templates variable. Your code will look like Listing 7.

Listing 7. Contents of madlibs.tf

 
 …
 variable "words" { … }
  
 locals {
   uppercase_words = {for k,v in var.words : k => [for s in v : upper(s)] if k!= "numbers"}
 }
  
 variable "num_files" { #A
     default = 100
     type = number
 }
  
 resource "random_shuffle" "random_nouns" {
   count = var.num_files #B
   input = local.uppercase_words["nouns"]
 }
  
 resource "random_shuffle" "random_adjectives" {
   count = var.num_files  #B
   input = local.uppercase_words["adjectives"]
 }
  
 resource "random_shuffle" "random_verbs" {
     count = var.num_files #B
     input = local.uppercase_words["verbs"]
 }
  
 resource "random_shuffle" "random_adverbs" {
   count = var.num_files #B
   input = local.uppercase_words["adverbs"]
 }
  
 resource "random_shuffle" "random_numbers" {
   count = var.num_files
   input = var.words["numbers"]
 }
  
 variable "templates" { #A
     default = ["templates/alice.txt","templates/observatory.txt","templates/photographer.txt"]
     type = list(string)
 }
  
 resource "local_file" "mad_lib" {
   count = var.num_files
   content = templatefile(element(var.templates,count.index), #B
     {
         nouns=random_shuffle.random_nouns[count.index].result
         adjectives=random_shuffle.random_adjectives[count.index].result
         verbs=random_shuffle.random_verbs[count.index].result
         adverbs=random_shuffle.random_adverbs[count.index].result
         numbers=random_shuffle.random_numbers[count.index].result
     })
     filename = "madlibs/madlib-${count.index}.txt"
 }
  

#A A list of template file paths to use

#B Select the element from list var.templates at the current index

The element function allows us to retrieve a single element from a list at a given index. The function wraps around, and you’ll never get an out of bounds exception because it’s the “safe” way to retrieve an element from a list. In our case, the function always evaluates to one of: templates[0], templates[1], or templates[2], in a round robin sort of fashion.

Why hardcode the template file paths? Why would I suggest hardcoding a list of file paths? This seems dumb. Unfortunately, I must crush your spirits by telling you that, with 100% certainty at the time of writing, there’s no better way to do this. By that, I mean there’s no function, expression, resource or data source which allows you to read the files in a directory and output the results as a list.
This isn’t to say that it’s impossible to do this through some other means. Numerous backdoors to the Terraform runtime can be exploited; it’s all about how clever and devious you want to get. For now, I don’t think it’s worth it to go into the gritty details.

Compressing Files with an Archive Resource 

We now have the ability to create arbitrary numbers of Mad Libs files and output them in a “madlibs” folder. Our final step is to zip the files together to share them more easily. As it happens, there’s an archive_file data source that does this. It’s part of the archive provider and works by taking all the files in a source directory and outputting a compressed file. This is exactly what we want. Add the following code from Listing 8 to main.tf.

Listing 8. Contents of main.tf

 
 …
 variable "templates" {
     default = ["templates/alice.txt","templates/observatory.txt","templates/photographer.txt"]
     type = list(string)
 }
  
 resource "local_file" "mad_lib" {
   count = var.num_files
   content = templatefile(element(var.templates,count.index),
     {
         nouns=random_shuffle.random_nouns[count.index].result
         adjectives=random_shuffle.random_adjectives[count.index].result
         verbs=random_shuffle.random_verbs[count.index].result
         adverbs=random_shuffle.random_adverbs[count.index].result
         numbers=random_shuffle.random_numbers[count.index].result
     })
     filename = "madlibs/madlib-${count.index}.txt"
 }
  
 data "archive_file" "mad_libs" {
   depends_on = ["local_file.mad_lib"] #A
   type        = "zip"
   output_path = "madlibs.zip" #B
   source_dir = "./madlibs"  #C
 }
  

#A An explicit dependency on the local_file resource

#B The name of the zip file

#C The directory to compress

This is the first time we’ve seen an explicit dependency between resources. Explicit dependencies are declared using the “depends_on” meta argument and are reserved for situations where there’s a hidden dependency between resources. The reason I’ve included it here’s because the archive_file data source must be evaluated after all the mad libs files have been created; otherwise it’d be zipping up an empty directory. Normally we express this dependency through an interpolated input argument to archive_file, but this particular data source doesn’t accept any input arguments which makes sense to set from an output attribute of local_file. Explicit dependencies behave exactly like implicit dependencies but are confusing and should be used cautiously—only when absolutely necessary.

The complete code for madlibs.tf is presented in Listing 9. Besides this code, you should also have three template (.txt) files in the templates folder.

Listing 9. Complete contents of madlibs.tf

 
 terraform {
   required_version = "~> 0.12"
   required_providers {
     random = "~> 2.1"
     template = "~> 2.1"
     local = "~> 1.2"
     archive = "~> 1.2"
   }
 }
  
 variable "words" {
     default = {
         nouns = ["army", "panther", "walnuts", "sandwich", "Zeus", "banana", "cat", "jellyfish", "jigsaw", "violin", "milk", "sun"]
         adjectives = ["bitter", "sticky", "thundering", "abundant", "chubby", "grumpy"]
         verbs = ["run", "dance", "love", "respect", "kicked", "baked"]
         adverbs = ["delicately", "beautifully", "quickly", "truthfully", "wearily"]
         numbers = [42, 27, 101, 73, -5, 0]
     }
     description = "A word pool to use for Mad Libs"
     type = map(list(string))
 }
  
 locals {
   uppercase_words = {for k,v in var.words : k => [for s in v : upper(s)] if k!= "numbers"}
 }
  
 variable "num_files" {
     default = 100
     type = number
 }
  
 resource "random_shuffle" "random_nouns" {
   count = var.num_files
   input = local.uppercase_words["nouns"]
 }
  
 resource "random_shuffle" "random_adjectives" {
   count = var.num_files
   input = local.uppercase_words["adjectives"]
 }
  
 resource "random_shuffle" "random_verbs" {
     count = var.num_files
     input = local.uppercase_words["verbs"]
 }
  
 resource "random_shuffle" "random_adverbs" {
   count = var.num_files
   input = local.uppercase_words["adverbs"]
 }
  
 resource "random_shuffle" "random_numbers" {
   count = var.num_files
   input = var.words["numbers"]
 }
  
 variable "templates" {
     default = ["templates/alice.txt","templates/observatory.txt","templates/photographer.txt"]
     type = list(string)
 }
  
 resource "local_file" "mad_lib" {
   count = var.num_files
   content = templatefile(element(var.templates,count.index),
     {
         nouns=random_shuffle.random_nouns[count.index].result
         adjectives=random_shuffle.random_adjectives[count.index].result
         verbs=random_shuffle.random_verbs[count.index].result
         adverbs=random_shuffle.random_adverbs[count.index].result
         numbers=random_shuffle.random_numbers[count.index].result
     })
     filename = "madlibs/madlib-${count.index}.txt"
 }
  
 data "archive_file" "mad_libs" {
   depends_on = ["local_file.mad_lib"]
   type        = "zip"
   output_path = "madlibs.zip"
   source_dir = "./madlibs"
 }
  

That’s all for this article. If you want to learn more about the book, you can check it out on our browser-based liveBook reader here and in this slide deck.