Take 37% off Istio in Action by entering fccposta into the discount code box at checkout at manning.com.

Let’s imagine that we have a Fortio load-testing client ready to go, and we’re ready to explore the use case. We will use Fortio to send 1000 requests per second through 10 connections for 60 seconds. Fortio will track the latency numbers for each call and plot them on a histogram with latency percentile break down. Before our test, we’ll introduce a version of the simple-backend-1 service that increases latency for to 1 second. This will simulate one of the endpoints experiencing a long garbage collection event or other application latency. We will vary our load balancing strategy between round robin, random, and least connection and observe the differences

Let’s deploy the delayed simple-backend-1 service:

 
$  kubectl apply -f simple-backend-delayed.yaml
 

If you run Fortio in server mode, we can get to a web dashboard where we can input the parameters of our test, execute the test, and visualize the results.

 
$  fortio server

Now open your browser to the Fortio UI: http://localhost:8080/fortio

Figure 1. Fortio Server UI for setting up our load test

Fill in the following parameters:

• Title: roudrobin
• URL: http://localhost
• QPS: 1000
• Duration: 60s
• Threads: 10
• Jitter: checked
• Headers: “Host: simple-web.istioinaction.io”

Your entries should look like the image in Figure 1.

At this point, we can start running the test by clicking the “Start” button about half way down the Fortio web page.

Figure 2. Fortio load test is in progress for 60s

Now we can wait for the test to complete. When it does, it will save a results file to your file system looking similar to 2020-09-15-101555_roundrobin.json. You will also see a results graph:

Figure 3. Results for load testing round robin client-side load balancing

For this ROUND_ROBIN load balancing strategy, we see resulting latencies as the following:

• 50% – 191.47ms
• 75% – 1013.31ms
• 90% – 1033.15ms
• 99% – 1045.05ms
• 99.9% – 1046.24ms

Now, let’s change the load-balancing algorithm to RANDOM and try the same load test again:

 
 $  kubectl apply -f simple-backend-dr-random.yaml
 destinationrule.networking.istio.io/simple-backend-dr configured
  

Now go back to the Fortio load-testing page (either click Back button, or the “Top” link). Fill in the information as you did before, but change the title to random

• Title: random
• URL: http://localhost
• QPS: 1000
• Duration: 60s
• Threads: 10
• Jitter: checked
• Headers: “Host: simple-web.istioinaction.io”

Click the “Start” button and wait for the results.

Figure 4. Results for load testing random client-side load balancing

For this RANDOM load balancing strategy, we see resulting latencies as the following:

• 50% – 189.53ms
• 75% – 1007.72ms
• 90% – 1029.68ms
• 99% – 1042.85ms
• 99.9% – 1044.17ms

Lastly, do the same for least-connection load balancing:

 
 $  kubectl apply -f simple-backend-dr-least-conn.yaml
 destinationrule.networking.istio.io/simple-backend-dr configured
  

• Title: leastconn
• URL: http://localhost
• QPS: 1000
• Duration: 60s
• Threads: 10
• Jitter: checked
• Headers: “Host: simple-web.istioinaction.io”

Click the “Start” button.

When it’s done running, you should see results:

Figure 5. Results for load testing least-connection client-side load balancing

For this LEAST_CONN load balancing strategy, we see resulting latencies as the following:

• 50% – 184.79ms
• 75% – 195.63ms
• 90% – 1036.89ms
• 99% – 1124.00ms
• 99.9% – 1132.71ms

We can see a few things in the previous diagram. First, the different load balancers indeed produce different results under realistic service latency behavior. Second, their results differ in both the histogram as well as their percentiles. Lastly, least connection performs better than either random or round robin. Let’s see why.

Round robin and random are both simple load balancing algorithms. They’re simple to implement and simple to understand. Round robin (or “next-in-loop”) will deliver requests to endpoints in a successive loop. Random will uniformly pick an endpoint at random. With both you would expect a similar distribution. The challenge with either of these strategies is that the endpoints in the load balancer pool are not typically uniform even if they are indeed backed by the same service and resources. Furthermore, any of these endpoints could experience garbage collection or resource contention that introduces high latency and we see that both round robin and random do not take any runtime behavior into account.

The LEAST_CONN load balancer (actually, in Envoy it has implemented as least request) does take into account the latencies of the specific endpoints. When it sends requests out to endpoints it monitors the queue depths tracking active requests and will pick the endpoints with the fewest active requests in flight. We can see that by using this type of algorithm we can avoid sending requests to endpoints that behave poorly and favor those that are responding quicker.

That’s all for this article.

If you want to learn more about the book, you can check out the book on Manning’s liveBook platform here.