Automation for the people: Hands-off load testing

Use Apache JMeter with Apache Ant to run load tests frequently

 

How many concurrent users can access your software system? How much data can be loaded without performance degradation? What are your system's throughput requirements? How often are these requirements tested? What if you could specify and validate that these load and performance requirements are being met at least once a day? By running load tests as part of a scheduled and automated build, you can more quickly determine how your system performs under certain load conditions and quickly adapt to changes.

About this series

As developers, we work to automate processes for end-users; yet, many of us overlook opportunities to automate our own development processes. To that end, Automation for the people is a series of articles dedicated to exploring the practical uses of automating software development processes and teaching you when and how to apply automation successfully.

A project I once worked on established a good set of automated tests that could load test the application while it ran multiple transactions. The problem was that the tests required some manual tweaking, so the development team couldn't run them without human intervention. This limited testing to times when the tester was available (usually waking hours only). In practice, testing occurred only every couple of days — not frequently enough for timely problem detection.

In this article, I explore techniques for creating automated tests using JMeter, running the tests as part of an automated build, and scheduling the tests to run automatically every day — typically when machine usage is lower. Running the tests as part of a scheduled build lets you:

• Execute load tests at any time
• Detect and resolve load and performance problems early in the development process
• Monitor the latest load- and performance-test reports from your build server
• Reduce bottlenecks and errors that can occur when you depend on a single person to configure and run the tests

Letting JMeter do the heavy lifting

Apache JMeter is an open source project you can use to simulate a heavy load on a server (see Resources for more information on JMeter). JMeter's impressive documentation set describes how to use its many features and provides a plethora of examples.

Running JMeter

After downloading and extracting the JMeter ZIP file (see Resources for a link to download JMeter), open a command prompt to the location where you extracted JMeter and type cd bin to change to the bin directory. From the bin directory, type jmeter to open the JMeter Swing application, shown in Figure 1:

Figure 1. The JMeter GUI

Creating a test plan

Writing tests, by example

JMeter comes with many example test plans and scripts. Instead of creating a test plan from scratch, use the examples in the docs directory and incrementally configure the test plans as your project evolves. Much of the complexity is in learning to write load tests that effectively exercise the load and performance requirements.

You can use the JMeter GUI to create test plans . The different types of test plans in JMeter include:

• Web test plan
• Database test plan
• FTP test plan
• LDAP test plan
• Extended LDAP test plan
• Web service test plan
• JMS point-to-point test plan
• JMS topic test plan
• Monitor test plan
• Listeners

Each of the test plans is saved in XML format in a file with a .jmx suffix. This nonbinary format makes plans easier to edit later. Although you can create test plans by following JMeter's XML schema, it's much easier to use the GUI. Later, you'll see an example of parameterizing JMeter configuration values to customize ways the tests are run.

---

Back to top

Labor-saving load testing

Using the GUI to run tests, though, requires that a person be present to execute them. This increases the chance of process bottlenecks and knowledge silos. By running the tests through an automated build such as an Ant build, you can configure the JMeter tests to run without opening the JMeter application. What's more, the tests are run the same way every time — and without requiring overtime pay.

Driving JMeter tests with Ant

Once I've learned how to use a GUI software tool, I like to see if it can run certain utilities from the command line, so that I don't need to perform the same actions repeatedly. For example, every time I open the JMeter application, I'd like to select File > Open, open the file, and run one or more tests. I can script this set of actions and run them the same way every time. Fortunately, an Ant task has been written to do this very thing for JMeter: it executes load tests while providing a way to pass in optional parameters and properties.

Example build script

The JMeter distribution's extras directory includes an example build.xml script that demonstrates the use of the JMeter Ant task.

In Listing 1, I use Ant's taskdef task to define the JMeter task. I name it jmeter so that I can use it elsewhere in the Ant script. To use this script you must have the ant-jmeter.jar file (see Resources for a download link) in the Ant classpath.

Listing 1. Defining the JMeter task in Ant

<taskdef name="jmeter" classname="org.programmerplanet.ant.taskdefs.jmeter.JMeterTask"/>

 

The example code in Listing 2 runs a single JMeter load test, called BreweryTestPlan.jmx. To run all of the tests in a certain directory, simply enter *.jmx in place of a specific filename. The required attributes of the jmeter task are jmeterhome , testplan (s ), and resultlog or resultlogdir . (Listing 2 doesn't show resultlogdir , because it uses resultlog .)

Listing 2. Running JMeter from Ant

<jmeter jmeterhome="${jmeter.home}" resultlog="${basedir}/target/JMeterResults.xml"> <testplans dir="${basedir}/tests/load" includes="BreweryTestPlan.jmx "/> </jmeter>

 

The Ant code in Listing 2 creates an output file called JMeterResults.xml, which is used to create HTML reports.

Rendering reports using XSLT

By providing the JMeterResults.xml file as input to the xslt Ant task in Listing 3, I can generate an HTML report of all the JMeter tests I ran in Listing 2. The XSL stylesheet (jmeter-results-detail-report_21.xsl) provided in the JMeter extras directory is used to transform the JMeterResults file into HTML.

Listing 3. Creating JMeter HTML report using XSLT

<xslt in="${basedir}/target/JMeterResults.xml" out="${basedir}/target/JMeterResults.html" style="${jmeter.home}/extras/jmeter-results-detail-report_21.xsl "/>

 

JMeter also provides a less detailed XSL stylesheet file that summarizes the load tests' results.

Revealing reports in HTML

Figure 2 is an example of the HTML report that is generated using the xslt task in Listing 3. It displays each of the load tests run, along with test status, time, and aggregate status and time of all of the tests.

Figure 2. Generating the JMeter HTML report

Later in the article, I'll show you how these reports can be displayed from a CruiseControl Continuous Integration (CI) server (see Resources ).

Passing parameters to JMeter

Depending on the types of tests you run, you may want to pass parameters and properties to vary the way a single test or suite of tests executes. For example, Listing 4 demonstrates how to increase the JVM memory and specify the number of threads and loops:

Listing 4. Passing optional parameters and properties to JMeter

<jmeter jmeterhome="${jmeter.home}" resultlog="${basedir}/target/JMeterResults.xml"> <jvmarg value="-Xincgc"/> <jvmarg value="-Xmx128m"/> <jvmarg value="-Dproperty=value"/> <property name="request.threads" value="5"/> <property name="request.loop" value="50"/> <property name="jmeter.save.saveservice.assertion_results" value="all"/> <property name="jmeter.save.saveservice.output_format" value="xml" /> <testplans dir="${basedir}/tests/load" includes="BreweryTestPlan.jmx"/> </jmeter>

 

Many other built-in parameters and properties are available for modifying how JMeter tests are run (see Resources for more information).

Using parameters and properties gives you a certain level of flexibility in how to execute load tests, but it doesn't address how to run load tests in different target environments, such as testing and staging. To add environment-specific information to test plans, you need to put tokens in the .jmx files that can be filtered and modified when the load tests run in an automated build script.

---

Back to top

Just-in-time load testing

Once you get a load test running with an automated build, schedule it to run on a periodic basis, such as nightly. You can do this using a CI or build-management server.

Scheduling CruiseControl to run load tests daily

The purpose of using a CI server is to run an automated build whenever a change is applied to a project's version-control repository. You can also configure it to run builds at specific times. Because load testing usually requires an elevated level of computing resources, running tests when these resources are not at capacity — such as late at night or early morning — can be beneficial.

In Listing 5 , an automated build is scheduled to run at 11:00 PM (2300) with CruiseControl (see Resources ). You can modify the CruiseControl configuration file to run a delegating build with a specific Ant target, such as one named run-load-tests .

Listing 5. Run a scheduled load test with CruiseControl

... <modificationset> <svn RepositoryLocation="${svnrepo.location}"/> <timebuild username="admin" time="2300 "/> </modificationset> ...

 

By scheduling load tests to run nightly, as in Listing 5, you won't hear excuses about workload, vacation days, or forgetting to run the tests — they just run.

Displaying reports in CruiseControl

You've already seen how to display JMeter test reports using Ant. However, by itself, the JMeter report limits the communication to one developer on a single machine. Load testing affects the entire application, so the whole team will want to see the results. The beauty is that you can easily configure your CI server to display these reports. Because the report is already generated with Ant, it's just a matter of making the JMeter HTML report accessible from the CruiseControl project dashboard. You can do this by simply adding a few lines to CruiseControl's config.xml file, as shown in Listing 6:

Listing 6. Configuring CruiseControl to display JMeter reports

<project name="brewery"> ... <log> <merge dir="merge dir="projects/${project.name}/reports/jmeter" /> </log> ... </project>

 

Now, everyone on the team can be (literally) on the same page. Many of the other CI and build-management servers provide similar report-integration capabilities.

---

Back to top

Conclusion

I've demonstrated how to add automated load tests to your development toolbox. By running load tests with the automated build and then scheduling the tests to run periodically, you can learn about system-capacity issues long before they become a problem. This approach makes it easier to assess the impact of architecture and data changes. When coupled with other techniques described in this article series , it enables development teams to deliver higher-quality software frequently.

 

---

Back to top

Download

Description	Name	Size	Download method
Sample Ant scripts for this article	j-ap04088-jmeter-example.zip	6KB	HTTP

Information about download methods

 

Resources

Learn

• JMeter : Visit the JMeter project site for documentation, tutorials, and community resources.
  
• "Test WebSphere performance with Apache JMeter " (Greg Herringer, developerWorks, May 2004): JMeter is great for testing Interactive Financial eXchange (IFX) messaging middleware.
  
• "Using JMeter " (Budi Kurniawan, onjava.com, January 2003): A basic JMeter how-to article.
  
• "Load Testing with Apache JMeter " (Kulvir Singh Bhogal and Javid Jamae, devx.com, December 2003): Another JMeter primer.
  
• "JMeter tips " (Chi-chang Kung, javaworld.com, July 2005): Improve the quality of your JMeter scripts.
  
• "Load Testing your Applications with Apache JMeter " (Keld H. Hansen, Java Boutique): A tutorial on using JMeter.
  
• Automation for the people (Paul Duvall, developerWorks): Read the complete series.
  • "Continuous testing " (March 2007): Always run automated tests with every change to a code base.
• Browse the technology bookstore for books on these and other technical topics.
  
• developerWorks Java technology zone : Hundreds of articles about every aspect of Java programming.
  

Get products and technologies

• Ant : Download Ant and start building software in a predictable and repeatable manner.
  
• JMeter : Download JMeter.
  
• JMeter Ant task : Ant task used to execute JMeter test plans.
  
• CruiseControl : Download the CruiseControl Continuous Integration Server.
  
• Download IBM product evaluation versions and get your hands on application development tools and middleware products from DB2®, Lotus®, Rational®, Tivoli®, and WebSphere®.