Configure JMeter script to optimize utilization of test engines in Azure Load Testing

Load testing your application can uncover critical performance and scale issues before your customers do. However, one important factor that can be overlooked is that the test system itself needs to be running smoothly - otherwise the accuracy of your test results will be compromised!

While running a load test in Azure Load Testing you might wonder if you’ve configured the correct number threads (users) in your Apache JMeter script. A thread runs through a transaction from start to finish, and then repeats until the test is over. You may notice that a test runs successfully with a lower number of threads, but the test may fail with errors once you increase the load.

CPU, memory and input/output usages can vary from script to script. A high resource usage for a test engine might negatively influence the results of the load test. To gain confidence in your test results, make sure that your test engines aren’t performance bottlenecks.

In this article, we'll walk through the following steps to help you be confident in your load testing:

• Edit your JMX file to configure the JMeter script
• Create and run a test
• Monitor the test engine health performance

Let’s get started!

Step 1: Configure your JMeter script

Follow these steps to edit your JMX file:

• Number of threads: Set at 240. You will be calibrating this during the test to determine the ideal number of threads.
• Ramp up time: Set at 120 seconds. This is the amount of time it will take JMeter to add all test users (threads) to a test execution.
• Duration: Set at 720 seconds. This is the total duration for which JMeter executes the test.

JMeter will take 120 seconds to get all 240 threads up and running with this configuration. Each thread will start 0.5 (120/240) seconds after the previous thread was begins. The shape of the load will resemble the image below.

Step 2: Create and run your test

• Navigate to your Azure Load Testing resource in the Azure portal.
•  Create a test by uploading your JMeter script.
• On the Load tab, select the default of one engine instance.

Azure Load Testing will automatically create a test run.

Step 3: Monitor engine health metrics:

Monitor your test engine’s health metrics by clicking on the Engine Health tab on the test run dashboard.

Azure Load Testing reports four resource metrics for each test engine:

• CPU percentage
• Memory percentage
• Network bytes per second
• Number of virtual users

A test engine instance is considered healthy if the average CPU percentage or memory percentage over the duration of the test run remains below 75%.

Step 4: Calibrate the number of threads

If your test engine didn't reach either 75% CPU utilization or 75% memory usage, increase the number of threads from 240 to 500 and run the test again. By increasing the number of threads and monitoring the engine health metrics you can configure your test for optimal resource utilization.

If your engine crossed 75% CPU utilization or 75%or memory utilization, follow these troubleshooting steps:

1. Observe the number of virtual users (threads) when the test engine reached 75%, say 400. Reduce the number of threads in your JMeter script to 90% of this which is 360. To achieve a higher target number of total virtual users, you can increase the number of test engines for the load test.
2. Ensure that your script is effective, with no redundant code or heavy samplers. For example, if you are processing a very large response in your script this could result in higher memory utilization.
3. Run the test again and make sure you don't go over 75% CPU or 75% memory utilization throughout the test.

So, we walked through how to configure your JMeter script, create and run a test, and monitor the engine health metrics to help achieve confidence on the test results. We hope it helped.

If you have any feedback on Azure Load Testing Preview, let us know through our feedback forum.

Happy Load Testing!