Monday 6 April 2009

JMeter to test robustness of system exposing REST APIs

UPDATE: code moved here: http://github.com/smartrics/JMeterRestSampler

I have been involved on continuously testing systems for robustness. One of my toy projects is the JMeterRestSampler, a plugin for JMeter to ease creation of such tests and automate their execution on a continuous integration server.

Robustness testing
In this context, a robust system is such that if it is subject to a constant load for a given amount of time, its response is correct and its throughput is constant.

The aim of a robustness test is to identify design or development issues such as memory leaks or random/unfrequent bugs that over time may degrade or crash the system. It's worth pointing out that robustness tests differ from load or stress tests which aim more at identifying bottlenecks or actual system limits (responses to peaks or bursts).

Continuously test for robustness is hence a must to improve the quality of the system under construction.
I, Robbie and Raghav presented a session at Agile '08 and XPDay 08 on this very topic.

Specifically for robustness, a test is defined by running scenarios concurrently. A scenario is a set of operations on the system that emulate typical usage. Our examples at both the presentation were based on the robustness tests we built for Aloha, an application server for Voice over IP applications. In this context a scenario - typical usage of the system - is for example "Connect voice call between two participants" that translates into a sequence of simpler operations performed via the application server's public interface.

Robustness with JMeter
When preparing the XPDay presentation, I started musing with JMeter to see if it offered simple means to build robustness tests. The bare essentials i was looking for are

  • the possibility to build scenarios in relation to the system under test (given the above definition of scenario)
  • automate tests and run them from a continuous integration server and
  • the possibility to report on the outcome of the tests.


As I have been involved in building REST APIs for several months, I chose to experiment assuming that the system under test offered a REST exposure.

Scenarios

In JMeter robustness scenarios can be modeled as ThreadGroups. A ThreadGroup groups a set of sampling operations on the system under test. Each sampling operation is executed serially within the thread running the ThreadGroup and each ThreadGroup can be run in loop and in parallel with other ThreadGroups for as long as necessary, either terminating them after a fixed number of runs or after a given time.

All the goodies shipped with JMeter can be used like pre/post processing instructions, definitions of user variables, throttling, etc. (see JMeter manual for more).

The following picture shows a simple test plan for a system under test with a robustness test made of 2 scenarios: "Register customer" and "Place an order". For the "Register customer" scenario, the ThreadGroup will be run in 10 parallel threads and each thead will run in loop for 100 times (so, in toal, the system will be running 1000 scenarios).



Sampling REST resources

JMeter offers a wide selection of samplers for sampling systems and collecting results. It does not offer, though, a proper sampler for the definition of REST requests. In the spirit of readable tests for documentation I wanted each sampler (effectively a REST request) to look like a REST request with a URI, method, list of headers and a body (much in the spirit of the RestFixture). So I discarded the HTTP sampler as too bound to HTTP requests.

So the first stab was to build a new sampler: the REST Sampler. It allows to define a rest request by setting all the relevant parameters: the URI, the headers, the body, the verb. It also processes JMeter assertions and post processors (attached as chidren to the sampler node) to perform basic assertions on the response elements and, for example to extract data from the last response and store it for future reference.

The REST sampler


Asserting the response code of the last response


Extracting data from the response for future reference, via JMeter variables




Monitoring the remote system: JMX Sampler

As said, part of a robustness build is to observe the system under test for key indicators of the health of the system. If the system under test is built in Java, one key element is the amount of heap memory allocated by to the system at any point in time: the trend of the average value of the heap memory may spot memory leaks. To grab this data in the context of a robustness test via JMeter, I have added a JMX Sampler, which basically connects to the given JMX URL and retrieves Heap memory information.


Next, I wanted to present the information collected. If JMeter is run via GUI, a JMX listener can be added to gather and chart the memory data.



It's possible to add more than one JMX sampler pointing to different JMX servers in such cases when the REST application is distributed. Charts of each Heap memory usage are displayed in the same listener window. Also, the charts can be saved in one or more files if needed.

Test Automation

The REST and JMX samplers plus the JMX listener can be used to build robustness tests necessary to test systems exposing REST interfaces. The next step is automation.
Automation is achieved with Ant and the JMeter task, which allows to run the same test plan but from whitin an Ant task (hence invokable via any Continuous integration server). Documentation on how to use the JMeter ant task is available in the website.

I, though, thought to extend the task to generate charts using the data available in the JMeter result output file. The Charts generated are one for the time occurred for each of the rest requests (which gives an indication on how the performance of the system are degrading overtime) and one for the JMX sampling as detailed above.

<taskdef name="jmeter"
classname="smartrics.jmeter.ant.JMeterTaskExt"
classpathref="project.classpath" />

<target name="run" depends="clean, init, copy">
<jmeter jmeterhome="${jmeter.home}"
resultlogdir="${artefacts}"
failure="false"
failureproperty="jmeter.failed"
chartsoutputdir="${artefacts}"
succecssThresholdPerc="98"
jmeterproperties="${jmeter.home}/bin/user.properties">
<testplans dir="${build}" includes="Robustness.jmx" />
</jmeter>
</target>

<target name="report">
<xslt in="${artefacts}/Robustness.jtl"
out="${artefacts}/Robustness.html"
style="${etc}/jmeter-results-detail-report_21.xsl" />
<copy todir="${artefacts}">
<fileset dir="${etc}" includes="*.jpg" />
</copy>
</target>


The location where the files are generated is defined by the attribute chartsoutputdir="...", if omitted no chart will be generated.

Correctness and the failure threshold

In certain cases a build may be allowed to pass even if some of the assertions did not pass. Such behaviour is admissible in systems that implement, for example, optimistic locking or other mechanisms that trade correctness for performance. Each REST sampler flags the sample as sucess or failure depending on the result of each associated assertion. By counting such flag values it's easy to determine the percentage of successes.

If the Tester decides that failure is acceptable, she can specify the succecssThresholdPerc="..." attribute to define the minimum percentage of successes that need to happen in order for the build to pass.

Build and isntallation instructions

Build
The JMeterRestSampler is available here:
http://github.com/smartrics/JMeterRestSampler
currently hosted in the same SVN trunk as the RestFixture
After checkout, add a properties/${user.name}.properties file and define the jmeter.home property. Then run ant.
${user.name} is the username of the logged in user.

If the build passes the file build/JMeterRestSampler.jar is generated.

Install
Edit the ${jmeter.home}/bin/user.properties and set the searh_paths property to the location of the JMeterRestSampler.java file: search_paths=/path/to/JMeterRestSampler.jar. Alternatively, copy the JMeterRestSampler.jar in the directory ${jmeter.home}/lib/ext.

If installation is successful, when starting JMeter, the Rest Sampler and JMX Sampler should be available in the Add/Sampler ThreadGroup context popup menu.

Eclipse project
The .classpath and .project Eclipse metadata files are included. To successfully compile the project within Eclipse, once imported, you need to define two workspace variables JMETER_HOME and ANT_HOME to respectively point to your JMeter home and Ant home directories.

License
The software is licensed using a BSD license.

Disclaimer
I should point out that the JMeterRestSampler is no more than a toy to prove the concept of Robustness testing via JMeter. It started as a spike back in the days of XPDay 08 and evolved - slowly - to it's complete form. I am not intentioned to improve it in the foreseeable future.

47 comments:

Colin said...

Thanks man, exactly what I was looking for, works a treat!

Note that I had to build with JMeter 2.3.2 as the latest version (2.3.3) has api changes.

Anonymous said...

Exactly what I was after - good work dude!

Only problem so far was that the repo URL had changed:

http://rest-fixture.googlecode.com/svn/trunk/JMeterRestSampler

Cheers, and happy RESTing

Al.

Anonymous said...

I'm a noob.

I can't build it!

smartrics said...

@Anonymous
> I'm a noob.
> I can't build it!

let me know if i can help

Descyple said...

I am unable to build with the most recent Jmeter 2.3.4 do you have a fix for this yet?

Jan said...

I think about improving the rest jmeter plugin. Can I extend your project or do you prefer a fork?

Descyple said...

What ever you would prefer. Any help would be appreciated.

Thank you

smartrics said...

Jan, I am going to move the JMeterRestSampler in a separate svn repo and pass on the details.

smartrics said...

Jan, The code is now available here

http://code.google.com/p/jmeter-rest-sampler/

get back in contact with me for arrangements

donhas said...

Hi Smartics.
Thank you for this project, very helpful.
With jmeter 2.3.4 the porject can't be complied due to changes in the "SaveService.loadTestResults" signature.

Any ideas how to handle

Thanks
Doron

smartrics said...

Hi DonHas, thanks.
I haven't tryed recompiling the sources with the latest version of JMeter. Looking quickly at the source code of JMeter 2.3.4, there's a call to loadTestResult in this file

src/core/org/apache/jmeter/reporters/ResultCollector.java

you can attempt to fix the code from there. Jan (who commented above) seems to have fixed it. I will get back in touch with him and see if I can get a fix for this.

Will post on the blog if I succeed.

donhas said...

Thanks for your reply,

There another issue that you might help me to solve.
For my rest tests I'm posting data to rest server, in order to debug my code I'm using "View Results Tree" however the "post data" section is always empty.

Any ideas how can I fixed it.

Thanks
Doron

Mark said...

Hi,

can anyone upload the .jar for the REST sampler in code.google.com?

Thanks,

Marcos

Chandini said...

Hi,

Even I am facing issues with the loadTestResullts() method signature. Does anybody have the code changes required?

Appreciate any help on this..

Thanks..

Martin said...

Hi,

That for this, this is was I needd. I'm running on the same compilation as everyone. Is the cmpliled jar is available from somewhere ?

Anonymous said...

How can i post a json body using Rest Sampler.

Anonymous said...

I looked at DataEtractor.handleResults(). And then I stared at it for some more time. Still no epiphany! Ok ok I'm a noob. Any help/idea would be mightily appreciated. Please!?

-syd

Anonymous said...

i have not find rest sampler in list of samplers .

andrew_wot said...

Has anyone figured out loadTestResult yet?

Thanks in advance,
Andrew

kim said...
This comment has been removed by the author.
kim said...
This comment has been removed by the author.
smartrics said...

I'll look at the compatibility issue with the newer JMeter and post a fix ASAP

andrew_wot said...

That would be awesome. Looking forward to it as we have to use JMeter 2.3.2 with that to work.

Thanks a bunch,
Andrew

smartrics said...

The code has been moved here http://github.com/smartrics/JMeterRestSampler

I have patched it to work with JMeter 2.3.4

The fix for JMeter 2.4 isn't as straightforwad, so I'll look into it when i have some time, sorry

Anonymous said...

Hi Guys,

Thanks for such a good plugin.
I dont know wheather it is a right place to ask this question. Please forgive me if this is not the right place to ask a question. I am new to Jmeter.

I want to store the result as it comes in 'View Results Tree' listners in different files like it comes in 3 different panel Sampler request, request, response data.

Can some one please suggest me how can I achieve this.
Please help me if any has got any knowledge about this.

smartrics said...

Hi, I am not sure you can do that. The only way I see is to post process the results file and extract the data manually.

Anonymous said...

Thanks for the reply smartrics.

Lauri said...

Great plugin, thanks a lot.

When I look at the results in my View Results Tree listener, the Request tab doesn't show the request body that was sent to the server. Is this something you've seen as well?

I ended up writing a small patch to restore this functionality, let me know if you are interested in merging it into your code.

smartrics said...

@Lauri, yeah no problem . Ideally a pull request on github would be better... otherwise however you prefer.

Anonymous said...

Hello
I just downloaded the JMeterRestSampler, but seems to be unable to build it. Can somebody please help. I get the following error:
$ ant
Buildfile: C:\jakarta-jmeter-2.4\RESTfulservices\smartrics-JMeterRestSampler-c46ed7d\JMeterRestSampler\build.xml

clean:
[delete] Deleting directory C:\jakarta-jmeter-2.4\RESTfulservices\smartrics-JMeterRestSampler-c46ed7d\JMeterRestSampler\build

init:
[mkdir] Created dir: C:\jakarta-jmeter-2.4\RESTfulservices\smartrics-JMeterRestSampler-c46ed7d\JMeterRestSampler\build

compile:
[mkdir] Created dir: C:\jakarta-jmeter-2.4\RESTfulservices\smartrics-JMeterRestSampler-c46ed7d\JMeterRestSampler\build\classes\main
[javac] C:\jakarta-jmeter-2.4\RESTfulservices\smartrics-JMeterRestSampler-c46ed7d\JMeterRestSampler\build.xml:28: warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds
[javac] Compiling 21 source files to C:\jakarta-jmeter-2.4\RESTfulservices\smartrics-JMeterRestSampler-c46ed7d\JMeterRestSampler\build\classes\main

BUILD FAILED
C:\jakarta-jmeter-2.4\RESTfulservices\smartrics-JMeterRestSampler-c46ed7d\JMeterRestSampler\build.xml:28: C:\jakarta-jmeter-2.4\RESTfulservices\smartrics-JMeterRestSampler-c46ed7d\JMeterRestSampler\lib does not exist.

Thanks for all your help.

TesterInDenver

smartrics said...

TesterInDenver, good spot!- it's a bug in the build file. I have fixed it now and just commited. So one of the following two: either make the lib directory manually as sibling of the src (same dir as where the build file is) or fetch/download the code again.

Anonymous said...

Hello

For some reason, the Rest sampler puts a space after the first ":" in the Authorization Header. The header has the following information: Authorization ${authString}. When I replace the colons with a (i.e. ";", there is no space. Since the sessionID is an in-house development effort, the Authorization Header must include the colons. Is there a workaround?
Here is the string I need to create:
Authorization Basic 2:1307334312:9bf77a1d388d5b403d3383be4b11df00. Notice no space before or after the colons ":".

Here are the variables via debug sampler:
newDate=1307334312
newDate2=1307334312
s=239erx487ysSkUA2dLkd7sySX
timehash=9bf77a1d388d5b403d3383be4b11df00

Here is the output:
Authorization Basic 2: 1307334312:9bf77a1d388d5b403d3383be4b11df00. Notice the space after the 1st colon. When I change the 1st colon to a different character (i.e. ";"), the space is inserted after the second colon. What is the uniqueness of the ":"? I need to use a colon as a delimiter between the fields.

I have the following beanshell script:
String authString = "Basic 2" + ":" + "${newDate2}" + ":" + "${timehash}";
vars.put("authorizationString",authString);

smartrics said...

@Anonymous - not sure if I understand the problem correctly.
if it's an HTTP header you are talking about, the space after the colon is inserted by the HTTP client as per HTTP spec http://www.w3.org/Protocols/HTTP/1.0/draft-ietf-http-spec.html#Message-Headers

I can think of various solutions. here they follow in no particoular order

1- put : at the beginning of your Auth token or

2- make the parser on the server a bit more flexible (to ignore spaces) or

3- inject between JMeter and the server a proxy that re-writes the request (Apache can do that!)

4- patch/shadow httpclient

5- mod the sampler to use a different http client that does what you need

Oded Hassidi said...

hi,

this looks great, also i succeeded in compiling and building the jar and add the relevant sampler, but I see no result in the listener, am i missing something?

10x

smartrics said...

hi Oded, it's likely that you're using a newer version of jmeter which is not supported.
i can't support this project anymore; but i found on github forks that have been fixed to work with later version of jmeter.
check this out
https://github.com/smartrics/JMeterRestSampler/issues/1

Anonymous said...

Can someone tell me where I can put parameters and how to put parameters to rest. I want to send following param-

'{"display_name": "xyz","service":"ordred"}'

Anonymous said...

Hello Smartrics

I got your Code from https://github.com/smartrics/JMeterRestSampler as a Zip File as my Git is showing error in cloning but as i am trying to build the code i am getting this error ..

C:\PETools\jakarta-jmeter-2.3.2\JMeterRestSampler>ant
Buildfile: build.xml

clean:
[delete] Deleting directory C:\PETools\jakarta-jmeter-2.3.2\JMeterRestSampler\build

init:
[mkdir] Created dir: C:\PETools\jakarta-jmeter-2.3.2\JMeterRestSampler\build

compile:
[mkdir] Created dir: C:\PETools\jakarta-jmeter-2.3.2\JMeterRestSampler\build\classes\main
[javac] Compiling 21 source files to C:\PETools\jakarta-jmeter-2.3.2\JMeterRestSampler\build\classes\main
[javac] C:\PETools\jakarta-jmeter-2.3.2\JMeterRestSampler\src\main\java\smartrics\jmeter\ant\DataExtractor.java:24:
cannot find symbol
[javac] symbol : class ResultCollectorHelper
[javac] location: package org.apache.jmeter.reporters
[javac] import org.apache.jmeter.reporters.ResultCollectorHelper;
[javac] ^
[javac] 1 error

BUILD FAILED
C:\PETools\jakarta-jmeter-2.3.2\JMeterRestSampler\build.xml:25: Compile failed; see the compiler error output for detail
s.

Total time: 1 second
C:\PETools\jakarta-jmeter-2.3.2\JMeterRestSampler>

I changed the Jmeter home path at properties\user.properties and build.properties also. Is still i am missing some thing?

Please Help me to resolve this .

I appreciate your help in this regard.

Thanks
Adity

smartrics said...

Hi, please try JMeter 2.3.4

Also please use github issue tracking to report any issue

Anonymous said...

Thanks a Lot , I used Jmeter 2.3.4 and its working for me . Thank a lot u save my lots of time to write a Script for providing the load to Rest

Viet-Thuan Nguyen said...

how can we get the result of REST sampler ?

Viet-Thuan Nguyen said...

Hi Cannizzo,

how can I check the sampler works correctly?

deepti said...

Hi Smartics,

I've added the .jar file from code.google.com, when i start the Jmeter and add parameters into the restsampler, i get the following error. I'm using 2.6 version of Jmeter. can you please tell me if the JMeterRestSampler_for_JMeter2.3.4.jar is specific for 2.3.4 only?

2012/05/15 15:49:15 ERROR - jmeter.threads.JMeterThread: Test failed! java.lang.NoSuchMethodError: org.apache.jorphan.util.JOrphanUtils.closeQuietly(Ljava/io/InputStream;)V
at smartrics.jmeter.sampler.RestSampler.sample(Unknown Source)
at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTPSamplerBase.java:1054)
at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTPSamplerBase.java:1043)
at org.apache.jmeter.threads.JMeterThread.process_sampler(JMeterThread.java:416)
at org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:271)
at java.lang.Thread.run(Thread.java:680)

smartrics said...

@deepti - it works with 2.3.4, I have never tested it with higher versions, so it may be that it won't work.

deepti said...

Thank you Smartics. it works with 2.3.4 for me too.

Dzmitry said...

Thank you for plugin, I've forked your repo to try it.
JMeter is rather good tool, but its main disadvantage is that its reporting tools are not very descriptive. And it makes to use third-party JMeter plugin to have better graphs.

abhijit said...

Help to use CSV file for dynamic RESTful service testing in JMeter. The XML file mentioned in the path in the HTTP Request screen need to have dynamic value. I used like ${customerOrder} . customerOrder is used in the CSV Data Set config.

thanks
Abhijit

smartrics said...

Abhijit - i am not supporting JMeter Rest sampler; sorry.