Save 37% off Elm in Action. Just enter fccfeldman into the discount code box at checkout at manning.com.

Writing Fuzz Tests

When writing tests for business logic, it can be time-consuming to hunt down edge cases—those unusual inputs which trigger bugs that never manifest with more common inputs.

In Elm, fuzz tests help us detect edge case failures by writing one test which verifies a large number of randomly-generated inputs.

Figure 1 shows what we’ll be building towards.

Figure 1 Randomly generating inputs with fuzz tests

A common way to write a fuzz test is to start by writing a unit test and then converting it to a fuzz test to help identify edge cases.

Let’s dip our toes into the world of fuzz testing by converting our existing unit test to a fuzz test. We’ll do this by randomly generating our JSON instead of hardcoding it; this way we can be sure our default title works properly no matter what the other fields are set to!

Converting Unit Tests to Fuzz Tests

Before we can switch to using randomly-generated JSON, first we need to replace our hardcoded JSON string with some code to generate that JSON programmatically.

Building Json programmatically with Json.encode

As we use the Json.Decode module to turn JSON into Elm values, we can use the Json.Encode module to turn Elm values into JSON. Let’s add this to the top of PhotoGrooveTests.elm, right after import Json.Decode exposing (decodeString).

import Json.Encode as Encode

Because JSON encoding is the only type of encoding we’ll be doing in this file, that as Encode alias lets us write Encode.foo instead of the more verbose Json.Encode.foo. While we’re at it, let’s give our Json.Decode import the same treatment, and change it to this:

import Json.Decode as Decode exposing (decodeString)

Json.encode.value

Whereas the Json.Decode module centers around the Decoder abstraction, the Json.Encode module centers around the Value abstraction. A Value (short for Json.Encode.Value) represents a JSON-like structure. In our case we’ll use it to represent JSON, but later we’ll see how it can be used to represent objects from JavaScript as well.

We’ll use three functions to build our {"url": "fruits.com", "size": 5} JSON on the fly:

• Encode.int    : Int                    -> Value
• Encode.string : String                 -> Value
• Encode.object : List ( String, Value ) -> Value

Encode.int and Encode.string translate Elm values into their JSON equivalents. Encode.object takes a list of key-value pairs; each key must be a String, and each value must be a Value.

Table 1 shows how we can use these functions to create a Value representing the same JSON structure as the one our hardcoded string currently represents.

Table 1 Switching from String to Json.Encode.Value

 
 """{"url": "fruits.com", "size": 5}"""
  

  
 Encode.object
     [ ( "url", Encode.string "fruits.com" )
     , ( "size", Encode.int 5 )
     ]
  

Json.Decode.decodevalue

Once we have the Value we want, there are two things we could do with it.

1. Call Encode.encode to convert the Value to a String, then use our existing decodeString photoDecoder call to run our decoder on that JSON string.
2. Don’t bother calling Encode.encode, but instead swap out our decodeString photoDecoder call for a call to decodeValue photoDecoder.

Like decodeString, the decodeValue function also resides in the Json.Decode module. It decodes a Value directly, without having to convert to and from an intermediate string representation. This is simpler and runs faster; we’ll do it this way!

Let’s start by editing our import Json.Decode line to expose decodeValue instead of decodeString. It should end up looking like this:

 
import Json.Decode as Decode exposing (decodeValue)
 

Then let’s incorporate our new encoding and decoding logic into our test’s pipeline.

 
 decoderTest : Test
 decoderTest =
     test "title defaults to (untitled)" <|
         \_ ->
             [ ( "url", Encode.string "fruits.com" )
             , ( "size", Encode.int 5 )
             ]
                 |> Encode.object
                 |> decodeValue PhotoGroove.photoDecoder ❶
                 |> Result.map .title
                 |> Expect.equal (Ok "(untitled)")
  

❶ We now call decodeValue instead of decodeString here

from test to fuzz2

Now we’re building our JSON programmatically, but we’re still buiding it out of the hardcoded values "fruits.com" and 5. To help our test cover more edge cases, we’ll replace these hardcoded values with randomly generated ones.

The Fuzz module helps us do this. Add this after import Expect exposing (Expectation):

import Fuzz exposing (Fuzzer, list, int, string)

We want a randomly generated string to replace "fruits.com" and a randomly generated integer to replace 5. To access those we’ll make the substitution shown in Table 2.

Table 2 Replacing a Unit Test with a Fuzz Test

 
  
 test "title defaults to (untitled)" <|
     \_ ->
 

 
  
 fuzz2 string int "title defaults to (untitled)" <|
     \url size ->
 

We’ve done two things here.

First, we replaced the call to test with a call to fuzz2 string int. The call to fuzz2 says that we want a fuzz test which randomly generates two values. string and int are fuzzers which specify that we want the first generated value to be a string, and the second to be an integer. Their types are string : Fuzzer String and int : Fuzzer Int.

The other change was to our anonymous function. It now accepts two arguments: url and size. Because we’ve passed this anonymous function to fuzz2 string int, elm-test will run this function one hundred times, each time randomly generating a fresh String value and passing it in as url, and a fresh Int value and passing it in as size.

Using the randomly-generated Values

Now that we have our randomly-generated url and size values, all we have to do is use them in place of our hardcoded "fruits.com" and 5 values. Here’s our final fuzz test:

 
 decoderTest : Test
 decoderTest =
     fuzz2 string int "title defaults to (untitled)" <|
         \url size ->                          ❶
             [ ( "url", Encode.string url )    ❶
             , ( "size", Encode.int size )     ❶
             ]
                 |> Encode.object
                 |> decodeValue PhotoGroove.photoDecoder
                 |> Result.map .title
                 |> Expect.equal (Ok "(untitled)")
  

❶ url and size come from the string and int fuzzers we passed to fuzz2

Great! We now have considerably more confidence that any JSON string containing only properly-set "url" and "size" fields—but no "title" field—will result in a photo whose title defaults to "(untitled)".

Next we’ll turn our attention to a more frequently invoked function in our code base: update.

Testing update functions

All Elm programs share some useful properties which make them easier to test.

1. The entire application state is represented by a single Model value.
2. Model only ever changes when update receives a Msg and returns a new Model.
3. update is a plain old function, and we can call it from tests like any other function.

Let’s take a look at the type of update:

 
 update : Msg -> Model -> ( Model, Cmd Msg )
  

Because this one function serves as the gatekeeper for all state changes in our application, all it takes to test any change in application state is to:

1. Call update in a test, passing the Msg and Model of our choice
2. Examine the Model it returns

Testing Clickedphoto

Let’s use this technique to test one of our simplest state changes: when a SlidHue message comes through the application. For reference, here’s the branch of update’s case-expression that runs when it receives a SlidHue message.

 
 SlidHue hue ->
     applyFilters { model | hue = hue }
  

This might seem like a trivial thing to test. It does little! All it does is update the model’s hue field, right?

Not quite! Importantly, this logic also calls applyFilters. What if applyFilters later returns a different model, introducing a bug? Even writing a quick test for SlidHue can give us an early warning against that and potential future regressions in our update implementation.

Listing 3 shows a basic implementation, which combines several concepts.

 
 slidHueSetsHue : Test
 slidHueSetsHue =
     fuzz int "SlidHue sets the hue" <|
         \amount ->
             initialModel                      ❶
                 |> update (SlidHue amount)    ❷
                 |> Tuple.first                ❸
                 |> .hue                       ❹
                 |> Expect.equal amount        ❺
  

❶ Begin with the initial model

❷ Call update directly

❸ Discard the Cmd returned by update

❹ Return the model’s hue field

❺ The model’s hue should match the amount we gave SlidHue

Tuple.first  takes a tuple and returns the first element in it. Because update returns a ( Model, Cmd Msg ) tuple, calling Tuple.first on that value discards the Cmd and returns only the Model—which is all we care about in this case.

Let’s run the test and…whoops! It didn’t compile!

Exposing variants

Reading the compiler error message, we might remember that although we previously edited the PhotoGroove module to have it expose photoDecoder, we haven’t done the same for all the values we’re using here, like initialModel and update. Let’s change it to expose these:

  
 port module PhotoGroove exposing (Model, Msg(..), Photo, initialModel, main, photoDecoder, update)
  

The (..) in Msg(..)means to expose not only the Msg type itself (for use in type annotations such as Msg -> Model -> Model), but also its variants. If we’d only exposed Msg rather than Msg(..), we still wouldn’t be able to use variants like SlidHue in our test! In contrast, Photo is a type alias, and writing Photo(..)yields an error; Photo has no variants to expose!

Because we’re using SlidHue without qualifying it with its module name—which we could have done by writing PhotoGroove.SlidHue instead—we’ll need to expose some variants in our import declaration to bring them into scope. Let’s do that for Msg, Photo, Model, and initialModel by changing the import PhotoGroove line in PhotoGrooveTests like this:

 
 import PhotoGroove exposing (Model, Msg(..), Photo, initialModel, update)
  

 
 type Commands
     = FetchPhotos Decoder String
     | SetFilters FilterOptions
  
 Then change update to have this type:
  
 update : Msg -> Model -> ( Model, Commands )
  

 
 toCmd : Commands -> Cmd Msg
 toCmd commands =
     case commands of
         FetchPhotos decoder url ->
             Http.get url decoder
                 |> Http.send GotPhotos
  
         Setfilters options ->
             setFilters options
  

 
 updateForProgram : Msg -> Model -> ( Model, Cmd Msg )
 updateForProgram msg model =
     let
         ( newModel, commands ) =
             update msg model
     in
     ( newModel, toCmd commands )
    
  

Excellent! If you re-run elm-test, you should see two passing tests instead of one.

Creating multiple tests with one function

We’ve now tested SlidHue, but SlidRipple and SlidNoise are as prone to mistakes as SlidHue is!

One way to add tests for the other two is to copy and paste the SetHue test two more times, and tweaking the other two to use SlidRipple and SlidNoise. This is a perfectly fine technique! If a test doesn’t have other tests verifying its behavior, the best verification tool is reading the test’s code. Sharing code often makes it harder to tell what a test is doing by inspection, which can seriously harm test reliability.

In the case of these sliders, though, we’d like to share code for a different reason than conciseness: they ought to behave the same way! If in the future we changed one test but forgot to change the others, which would almost certainly be a mistake. Sharing code prevents that mistake from happening!

Grouping tests with describe

Listing 4 shows how we can use the Test.describe function to make a group of slider tests.

 
 sliders : Test
 sliders =
     describe "Slider sets the desired field in the Model"   ❶
         [ testSlider "SlidHue" SlidHue .hue                 ❶
         , testSlider "SlidRipple" SlidRipple .ripple        ❶
         , testSlider "SlidNoise" SlidNoise .noise           ❶
         ]
  
  
 testSlider : String -> (Int -> Msg) -> (Model -> Int) -> Test
 testSlider description toMsg amountFromModel =
     fuzz int description <|                                 ❷
         \amount ->
             initialModel
                 |> update (toMsg amount)                   ❸
                 |> Tuple.first
                 |> amountFromModel                         ❹
                 |> Expect.equal amount
  

❶ Group this List of Test values under one description

❷ Use testSlider’s description argument as the test’s description

❸ (toMsg  : Int -> Msg) will be SetHue, SetRipple, or SetNoise

❹ (amountFromModel : Model -> Int) will be .hue, .ripple, or .noise

The Test.describe function has this type:

 
 describe : String -> List Test -> Test

When one of the tests in the given list fails, elm-test prints out not only that test’s description, but also the string passed to describe as the first argument here. For example, if a test with a description of "SetHue" were listed as one of the tests inside a describe "Slider sets the desired field in the Model", its failure output looks like this:

Figure 2 Failure output after using describe

Returning tests from a custom function

The testSlider function is a generalized version of our slidHueSetsHue test from earlier. Table 3 shows them side by side.

Table 3 Comparing slidHueSetsHue and testSlider

 
 fuzz int "SlidHue sets the hue" <|
     \amount ->
         initialModel
             |> update (SlidHue amount)
             |> Tuple.first
             |> .hue
             |> Expect.equal amount
 

 
  
 fuzz int description <|
     \amount ->
         initialModel
             |> update (toMsg amount)
             |> Tuple.first
             |> amountFromModel
             |> Expect.equal amount
 

Have a look at the type of testSlider:

 
 testSlider : String -> (Int -> Msg) -> (Model -> Int) -> Test
 testSlider description toMsg amountFromModel =
  

Its three arguments correspond to what we want to customize about the SlidHue test:

1. description : String lets us use descriptions other than "SlidHue sets the hue"
2. toMsg : Int -> Msg lets us use messages other than SetHue
3. amountFromModel : Model -> Int lets us use model fields other than .hue

Because the testSlider function returns a Test, and describe takes a List Test, we were able to put these together to obtain our customized hue, ripple, and noise tests like this:

 
 describe "Slider sets the desired field in the Model"
     [ testSlider "SlidHue" SlidHue .hue
     , testSlider "SlidRipple" SlidRipple .ripple
     , testSlider "SlidNoise" SlidNoise .noise
     ]
  

This compiles because the SlidHue variant is a function whose type is SlidHue : Int -> Msg, which is what the toMsg argument expects, and because the .hue shorthand is a function whose type is .hue : Model -> Int, which is what the amountFromModel argument expects.

Running the Complete tests

Let’s take them out for a spin! If you re-run elm-test, you should still see four happily passing tests.

We’ve now tested some decoder business logic, confirmed that running a SlidHue message through update sets model.hue appropriately, and expanded that test to test the same logic for SlidRipple and SlidNoise by using one function that created multiple tests. Next we’ll take the concepts we’ve learned this far and apply them to testing our rendering logic as well!

That’s all for this article. If you want to learn more about the book, check in out on liveBook here and see this slide deck.