UPDATE: code moved here:
http://github.com/smartrics/JMeterRestSamplerI have been involved on continuously testing systems for
robustness. One of my toy projects is the
JMeterRestSampler, a plugin for JMeter to ease creation of such tests and automate their execution on a continuous integration server.
Robustness testingIn this context, a robust system is such that if it is subject to a constant load for a given amount of time, its response is correct and its throughput is constant.
The aim of a robustness test is to identify design or development issues such as memory leaks or random/unfrequent bugs that over time may degrade or crash the system. It's worth pointing out that robustness tests differ from load or stress tests which aim more at identifying bottlenecks or actual system limits (responses to peaks or bursts).
Continuously test for robustness is hence a must to improve the quality of the system under construction.
I,
Robbie and
Raghav presented a session at
Agile '08 and
XPDay 08 on this very topic.
Specifically for robustness, a test is defined by running
scenarios concurrently. A scenario is a set of operations on the system that emulate typical usage. Our examples at both the presentation were based on the robustness tests we built for
Aloha, an application server for Voice over IP applications. In this context a scenario - typical usage of the system - is for example "Connect voice call between two participants" that translates into a sequence of simpler operations performed via the application server's public interface.
Robustness with JMeterWhen preparing the XPDay presentation, I started musing with
JMeter to see if it offered simple means to build robustness tests. The bare essentials i was looking for are
- the possibility to build scenarios in relation to the system under test (given the above definition of scenario)
- automate tests and run them from a continuous integration server and
- the possibility to report on the outcome of the tests.
As I have been involved in building REST APIs for several months, I chose to experiment assuming that the system under test offered a REST exposure.
ScenariosIn JMeter robustness scenarios can be modeled as
ThreadGroups. A ThreadGroup groups a set of sampling operations on the system under test. Each sampling operation is executed serially within the thread running the ThreadGroup and each ThreadGroup can be run in loop and in parallel with other ThreadGroups for as long as necessary, either terminating them after a fixed number of runs or after a given time.
All the goodies shipped with JMeter can be used like pre/post processing instructions, definitions of user variables, throttling, etc. (see
JMeter manual for more).
The following picture shows a simple test plan for a system under test with a robustness test made of 2 scenarios: "Register customer" and "Place an order". For the "Register customer" scenario, the ThreadGroup will be run in 10 parallel threads and each thead will run in loop for 100 times (so, in toal, the system will be running 1000 scenarios).
Sampling REST resourcesJMeter offers a wide selection of samplers for sampling systems and collecting results. It does not offer, though, a proper sampler for the definition of REST requests. In the spirit of readable tests for documentation I wanted each sampler (effectively a REST request) to look like a REST request with a URI, method, list of headers and a body (much in the spirit of the
RestFixture). So I discarded the HTTP sampler as too bound to HTTP requests.
So the first stab was to build a new sampler: the REST Sampler. It allows to define a rest request by setting all the relevant parameters: the URI, the headers, the body, the verb. It also processes JMeter assertions and post processors (attached as chidren to the sampler node) to perform basic assertions on the response elements and, for example to extract data from the last response and store it for future reference.
The REST sampler
Asserting the response code of the last response
Extracting data from the response for future reference, via JMeter variables
Monitoring the remote system: JMX SamplerAs said, part of a robustness build is to observe the system under test for key indicators of the health of the system. If the system under test is built in Java, one key element is the amount of heap memory allocated by to the system at any point in time: the trend of the average value of the heap memory may spot memory leaks. To grab this data in the context of a robustness test via JMeter, I have added a JMX Sampler, which basically connects to the given JMX URL and retrieves Heap memory information.
Next, I wanted to present the information collected. If JMeter is run via GUI, a JMX listener can be added to gather and chart the memory data.
It's possible to add more than one JMX sampler pointing to different JMX servers in such cases when the REST application is distributed. Charts of each Heap memory usage are displayed in the same listener window. Also, the charts can be saved in one or more files if needed.
Test AutomationThe REST and JMX samplers plus the JMX listener can be used to build robustness tests necessary to test systems exposing REST interfaces. The next step is automation.
Automation is achieved with Ant and the
JMeter task, which allows to run the same test plan but from whitin an Ant task (hence invokable via any Continuous integration server). Documentation on how to use the JMeter ant task is available in the website.
I, though, thought to extend the task to generate charts using the data available in the JMeter result output file. The Charts generated are one for the time occurred for each of the rest requests (which gives an indication on how the performance of the system are degrading overtime) and one for the JMX sampling as detailed above.
<taskdef name="jmeter"
classname="smartrics.jmeter.ant.JMeterTaskExt"
classpathref="project.classpath" />
<target name="run" depends="clean, init, copy">
<jmeter jmeterhome="${jmeter.home}"
resultlogdir="${artefacts}"
failure="false"
failureproperty="jmeter.failed"
chartsoutputdir="${artefacts}"
succecssThresholdPerc="98"
jmeterproperties="${jmeter.home}/bin/user.properties">
<testplans dir="${build}" includes="Robustness.jmx" />
</jmeter>
</target>
<target name="report">
<xslt in="${artefacts}/Robustness.jtl"
out="${artefacts}/Robustness.html"
style="${etc}/jmeter-results-detail-report_21.xsl" />
<copy todir="${artefacts}">
<fileset dir="${etc}" includes="*.jpg" />
</copy>
</target>
The location where the files are generated is defined by the attribute
chartsoutputdir="...", if omitted no chart will be generated.
Correctness and the failure thresholdIn certain cases a build may be allowed to pass even if some of the assertions did not pass. Such behaviour is admissible in systems that implement, for example, optimistic locking or other mechanisms that trade correctness for performance. Each REST sampler flags the sample as sucess or failure depending on the result of each associated assertion. By counting such flag values it's easy to determine the percentage of successes.
If the Tester decides that failure is acceptable, she can specify the
succecssThresholdPerc="..." attribute to define the minimum percentage of successes that need to happen in order for the build to pass.
Build and isntallation instructionsBuildThe
JMeterRestSampler is available here:
http://github.com/smartrics/JMeterRestSampler
currently hosted in the same SVN trunk as the RestFixtureAfter checkout, add a
properties/${user.name}.properties file and define the
jmeter.home property. Then run ant.
${user.name} is the username of the logged in user.
If the build passes the file
build/JMeterRestSampler.jar is generated.
InstallEdit the
${jmeter.home}/bin/user.properties and set the searh_paths property to the location of the JMeterRestSampler.java file:
search_paths=/path/to/JMeterRestSampler.jar. Alternatively, copy the JMeterRestSampler.jar in the directory
${jmeter.home}/lib/ext.
If installation is successful, when starting JMeter, the Rest Sampler and JMX Sampler should be available in the
Add/Sampler ThreadGroup context popup menu.
Eclipse projectThe
.classpath and
.project Eclipse metadata files are included. To successfully compile the project within Eclipse, once imported, you need to define two workspace variables JMETER_HOME and ANT_HOME to respectively point to your JMeter home and Ant home directories.
LicenseThe software is licensed using a BSD license.
DisclaimerI should point out that the JMeterRestSampler is no more than a toy to prove the concept of Robustness testing via JMeter. It started as a spike back in the days of XPDay 08 and evolved - slowly - to it's complete form. I am not intentioned to improve it in the foreseeable future.
