Saturday, 2 August 2008

Get FitNesse with some Rest

UPDATE: code moved here: http://github.com/smartrics/RestFixture

UPDATE II: Thank you to Steven Haines for the clear and concise Howto giude on the RestFixture.

I am currently involved in building a REST API. Our direct customer proxy (the architecture team) needs to "understand" how we implement the agreed acceptance criteria and it has also asked to document the API.
FitNesse is our tool of choice for writing functional Customer Acceptance Tests; we currently use it to implement this type of tests for other parts of the system we're building. So it came natural to start using it for documenting the API.

At first we implemented the tests using essentially ActionFixtures. The approach adopted consisted of "wrapping" the REST API in more descriptive metods that could be pressed, checked and entered.

After the first dozen of tests it was apparent that this approach was not ideal. Tests were not clear enough and maintaining the fixtures was hard and time consuming as the amount of code duplication was high for any standard. New tests required us writing more code and, in fact, we were testing just the behaviour of the backend without giving enough exposure to the API itself - consequently, our tests were not descriptive enough to work as live documentation.

So I decided to write a new "type" of fixture, inspired by the ActionFixture, the RestFixture.

The core principles underpinning the decision to write a new fixture were the following:
  • For documenting a REST API you need to show how the API looks like. For REST this means

    • show what the resource URI looks like. For example
      /resource-a/123/resource-b/234
    • show what HTTP operation is being executed on that resource. Specifically which one fo the main HTTP verbs where under test (GET, POST, PUT, DELETE, HEAD, OPTIONS).

    • have the ability to set headers and body in the request

    • check expectations on the return code of the call in order to document the behaviour of the API

    • check expectation on the HTTP headers and body in the response. Again, to document the behaviour

  • I didn't want to maintain fixture code. If I could only write the tests...
  • I wanted to be able to let the customer proxies to write the tests... And they understand Wiki syntax more than Java.

The RestFixture at a glance

The RestRixture is an ActionFixture, therefore all the ActionFixture goodies are available. On top of that it contains the following 7 methods:

  • header: to be able to set the headers for the next request (a CRLF separated list of name:value pairs

  • body: to allow request body input, essential for PUT and POST

  • let: to allow data from the response headers and body to be extracted and assigned to a label that can then be passed around. For example our API specifies that when you create a resource using POST, the newly created resource URI is in the Location header. This URI is necessary in order to perform further operations on that resource (for example, DELETE it in a teardown).

  • GET, POST, PUT, DELETE, to execute requests.

Each test is a rown on a RestFixture table and it has the following format:
|VERB|uri|?ret|?headers|?body|

  • VERB is one of GET, POST, PUT, DELETE (at the moment we're not supporting HEAD and OPTIONS)

  • uri is the resource URI

  • ?ret is the expected return code of the request. it can be expressed as a regular expression, that is you can write 2\d\d if you expect any code between 200 and 299.

  • ?headers is the expected list of headers. In fact, the expectation is checked by verifying that *all* the headers in here are present in the response. Each header must go in a newline and both name and value can be expressed as regular expressions to match in order to verify inclusion.

  • ?body it's the expected body in the response. This is expressed as a list of XPath expressions to allow greater flexibility.


Some examples

The following pictures are snapshots taken from FitNesse with the aim of providing examples of usage of the RestFixture. The notes before each test explain the details of the fixture itself.








It's worth noticing that, generally, expectations are not matched by string comparing the expected value with the actual value. Expectations on headers are verified by checking that the expected set of headers is included in the actual set of headers, similarly with the body where expectations are matched by checking existences of nodes for a given path.

Conclusions

The RestFixture allows to write FitNesse tests without having to write any Java code to back up the fixtures. Tests are clear and easy to read/write and this obviously improves their readability. Therefore it works well for documenting the API and the behaviour of the system under test. Looking it from another angle, the fixture, essentially, implements a REST DSL, that allows customers to write tests on their own.

59 comments:

Andy O said...

Very cool, but time to move away from ActionFixture. Use DoFixture! Should I do a brown bag on it?

smartrics said...

Hi Andy, yeah I thought of DoFixtures, but still the ActionFixture style seemed to be better suited for this specific case (that is documenting the API).

Robbie said...

Have you contributed this back into the Fitnesse community? I'm sure they'd be interested if you could open source it.

http://tech.groups.yahoo.com/group/fitnesse/

smartrics said...

Hi Robbie, I am working on it :) should be ready soon.

smartrics said...

... and it now ready. RestFixture has been open sourced: http://code.google.com/p/rest-fixture/

Videep said...

Hi,

I would like to know in which all scenarios were you able to use FitNesse? Can you give me some real examples. I am trying to find out where I can apply FitNesse to get a business value.

Thanks,
Videep.

Lennard said...

Aside from snippets here, is there any place where the actual wiki markup can be found for this fixture?
Also, no instructions for installation can be found. What needs to be included in the classpath in order for this fixture to work?

smartrics said...

Lennard, if you checkout the source code, you will see some fitnesse tests to show/document the features of this fixture. that set of tests should be enough to explain the markup as they cover all the features of the rest-fixture.

As for the classpath, you will require all the jars in the rest-fixture/lib directory (be careful not to include those that you may already have, eg log4j and fitnesse)

hope it helps

kl2217 said...

great example, one of the best open source restfixture on the web! I'm trying to test this on greenpepper. I'll share the result once getting progress. Thanks for the contribution!

smartrics said...

Thank you!

Anonymous said...

Hey, I have few clarifications,
1. what is "smartrics.rest.test.fitnesse.fixture.HttpServerFixture" ?

2. where we set the class path & path for fitnesse libs?
can you please send me a sample?
mail: dd_jayakody@yahoo.com

smartrics said...

HttpServerFixture is a small warapper class that enables a fixture to start a jetty HttpServer on a given port. The test suite shipped with the rest-fixture is then self-contained. a) starts the server on the SuiteSetUp to make the /resources resource available
b) tests for the rest-fixture run against that server
c) shuts the server down

You can make the rest fixture jar available to fitnesse via the classpath directive. see http://fitnesse.org/FitNesse.UserGuide.ClassPath

Mahadev said...

I downloaded .zip file and extracted in D:\fitnesse\Rest_Fixture\RestFixture-1.0 folder

I have fitness.jar in D:\fitnesse.

Now, I see 2 jar files
RestFixture.jar
RestFixture_fitnesse.jar

Do I need to replace my fitness.jar with RestFixture fitnesse file ?

OR
I should add these 2 jar files in CLASSPATH and start using them?

Do I need to write my own JAVA fixtures for my REST calls?

PLease advise.

Tomas said...

Hi there,

Nice Fixture!

Got a question. What is the syntax behind the multi line xpath evaluations? I cant get that to work.

Thanks in advance
Tomas

Anonymous said...

Is it possible to execute a for loop kind of a thing in RestFixture where i do a GET operation multiple times and come out of the loop based on a condition

Thanks and Regards,
Sandeep

smartrics said...

@Anonymous, no it's not - i am not sure if i even want this feature. The main point is simplicity and clarity in the examples. Adding loops (or flow control structures) IMO would add unnecessary complexity.

In any case it would be interesting to see your specific use case

smartrics said...

@Tomas, one line per XPath, each line separated by
as per tests included in the RestFixture

hal9ccc said...

Hi Fabrizio,

great work, I dare not to mess with it...

However, the tests we've created look kinda lengthy and will be hard to maintain. As we call the same service many times with different parameters it would be nice to find a way to make 1 call per row.

That means I'd need a setup, where I define the URI in conjunction with the XPaths I'd like to extract from the response. Then I could just reference this definition once per table and have multiple rows with different parameters going in and coming out of the service.

What ya think?

Thanks, Matthias

smartrics said...

@hal9ccc, yeah - I see the point. What you suggest may work to reduce the complexity and duplication.

I am wondering though if your approach reduces readability of each test.

I believe that each test should be self contained and capture the essence of the behaviour of the SUT for a specific feature (or part of). So, having to go up and down in the "code" to retrieve all the parts necessary for each test may not be ideal.

Could it work if you split the tests in such a way where you capture/test the generic behaviour of your SUT, then for each subsequent variation of the input parameter you only test the parts that change? this implies that you will have to specify in each table only those XPaths that test the parts of the XML that changes as function of the input parameter.

Also, you should have, for each test, only one single input that change and maybe organise the pages in such a way that a sub-tree identifies variations of a single macro-behaviour. Consider, to reduce duplication, the use of FitNesse sym links.

I am not sure if what I just said makes sense in your application and I can't comment any further without having seen your suites; but if you want to provide examples I'll be happy to share my view.

hal9ccc said...

Hi Fabrizio,

my REST service is basicly a wrapper around a database application which reads user inputs (barcode scans), analyzes them and changes the application state accordingly. So each test consists of an number of inputs which must be processed in a specific order.

ATM each input looks like this (7 lines per input):

!define MSGDATA {S123456}
!define MSGTYPE {VKSMSG_PSTNR_S}
!define PROMPT {PST-Nr.}
!define MSGRESUTLT {PROGRESS}
!define ERRORTEXT {blank}
!include -seamless .RestFixtureTests.MyTests.TestTemplate

MSGDATA is the barcode content (input), all other variables are extracted from the response (asserts).

In TestTemplate I use LET to extract the response values and ActionFixture to compare them.
I'm not sure if that's the best way to use RestFixture?

The same tests in dbFit (1 line per input):

!|Execute procedure|${DBUSER}.test_workflow|
|hint |prompt? |input |errorText? |messageType? |controllerState? |resultType? |? |
|scan PSTNR |PST-Nr. |S461007298 |blank |MSG_PSTNR_S |FRMBLKAL_HUNR |PROGRESS |1 |
|scan HU-Number |HU-Nr. |461007298 |blank |MSG_HUNR |FRMBLKAL_PSTNR |SUCCESS |1 |

Thanks very much.

John Penney said...

I want to test an API that requires the client to include an authentication header with the HTTP request.

However, as far as I can see, restfixture doesn't support this.

Am I right?

Thanks,

John

John Penney said...

..And in answer to my previous comment, just to reassure other newbies, there are at least 2 ways to set the http request header.

(a) You can use RestFixtureConfig and set restfixure.default.headers
(b) You can use setHeader

John

John Penney said...

Continuing my adventures with RestFixture, I cannot get GET to parse an XPath expression when the XML has a default namespace.

I don't think I'm the first to have this problem (just Google for restfixture namespace).

Any hope?

John Penney said...

For reference, I've also asked about using default namespaces and RestFixture on Stack Overflow: http://stackoverflow.com/questions/4606615/using-fitnesse-to-test-restful-apis-using-restfixture-anonymous-namespaces

At the time of writing, I've discovered a rather ugly workaround.

smartrics said...

@John I will work something out to add full support for namespaces.

how would you suggest the user experience (coder, but above all reader of the tests) should be?

John Penney said...

From what I've seen, other XPath parsers allow you to set up a namespace prefix mapping.

So it would be nice if I could set it up as part of the fixture config:

| smartrics.rest.fitnesse.fixture.RestFixtureConfig |
| restfixure.mapnamespaceprefix | xx | http://www.mycompany.com/ns |


Or define it during a test:

| setNamespacePrefix | xx | http://www.mycompany.com/ns |

So that then my XPaths could use the xx prefix:
| GET | | 200 | | //xx:thing |
| GET | | 200 | | /xx:a/xx:b/xx:c |

John Penney said...

On a different subject: I'd like to use LET labels to track a resource attribute that can vary as the test progresses.

For example:
(a) GET an Account and use LET to access the balance on the account, %origbalance%
(b) GET a Product and use LET to access the product cost, %productcost%
(c) POST a Purchase of the Product against the Account
(d) GET the Account and use LET to access the new balance on the account, %latestbalance%
(e) verify that the %latestbalance%=%origbalance% - %productcost%

Do you (or anyone else) know of a way to do (e)?

smartrics said...

John, the former approach is what I had in mind so I think I'll opt with that.

on your second request, I am not inclined to complicate the rest-fixture with more than it should do (like expression evaluations).

But, as the LET stores key/values in a map accessible from a Java API, you could build your fixture to do what you need.

https://github.com/smartrics/RestFixture/blob/master/src/main/java/smartrics/rest/fitnesse/fixture/support/Variables.java

John Penney said...

Fair enough! Thanks for responding so promptly, and for providing such a useful fixture!

John

smartrics said...

John, the rest-fixture now supports namespaces

John Penney said...

Hi Fabrizio,
I've just taken RestFixture 1.1 - thanks for the quick response!

I've tested it and it's working fine so far. I was a little thrown by the fact that you've fixed the typo in the config fixture (from "restfixure" in 1.0 to ""restfixture" in 1.1). My tests stopped working until I figured that out!

Thanks,
John

smartrics said...

John, yeah - sorry! I'll make sure that next time I publish a changelog

irFANTASTIC said...

I am trying to use this RestFixture but I am getting error
"Failed to load Main-Class manifest attribute from RestFixture.jar" when I am trying to install it...
Can any 1 guide me in setting up RestFixture?????

Ian Graham said...

Hi,

been trying to get RestFixture up and running all of today without much success. Problem is with wanting to have multiple assertions or set headers, i.e.

...
|setHeader|!-Content-type:application/json
Accept:application/json-!|

The line break just gets stripped out and a single header value passed in the request. Same with multiple assertions on checking response header values. Has anyone had problems with this already? Thanks.

smartrics said...

@Ian - will be looking at your issue shortly and comment.

@irFANTASTIC - please use github issue tracker to provide more details. Please refer to this blog for step by step installation guide http://www.informit.com/guides/content.aspx?g=java&seqNum=606

smartrics said...

@irFANTASTIC the issue tracker is here https://github.com/smartrics/RestFixture/issues

Roy Hunter said...

I downloaded your latest RestFixture - RestFixture-2.0.beta.1

I am trying to test a JSON interface but for the body it is returning 'no-body' also if I try passing content-type:application/json it tells me that it is not found.
Here is my table:

!|smartrics.rest.fitnesse.fixture.RestFixture|http://localhost:8080/pos360posservice|
|GET|\heartbeat?elementid=40:FC:89:31:DA:98&sourceid=fred|200|content-type: application/json;charset=utf-8|COMPANY:xyz / SITE:Mine/ DEVICE:tablet123 / USER:Fred / VER:1.0|
Any Help would be greatly appreciated.

Wizzy said...

Hi,

I am using REST Fixture to create my testcases. Facing a particular issue. I have some values in my response that I need to capture and calculate some values using these captured values. Now I have created a fixture to do the calculation. However the issue is arising when I try to pass values to my Fixture. I am using LET to capture values in a variable. When I am passing this variable in my Action Fixture using ENTER, its accepting the variable name as the input string as opposed to accepting the value of the variable. Kindly advice the right way to pass variable values to fixture. I tried playing around with Symbol Values but am having a difficult time figuring out how to pass symbol value to my Fixture. Even with $ sign, Fitness is still considering my variable as the string to be passed.

Thanks in advance,
Viji

Wizzy said...

Hi,

I am using REST Fixture to test our application's web services. I am facing a problem where I need to pass values that I have captured in variables to be calculated. This calculated value is something i am using as my expected result for comparison. I have created a custom Fixture to do the calculation. The only problem is when I am passing the variable through the action fixture, instead of using its value, the variable name is being passed through to my fixture code. This is messing up things for me. I have also tried prefixing variable names with $ sign to use them as FitNesse Symbols but its not working. Kindly suggest the right way of doing my usecase.

Thanks in advance.

smartrics said...

@Wizzy this should help you
https://github.com/smartrics/RestFixture/issues/17

please post any further support request here https://github.com/smartrics/RestFixture/issues

Geoffrey Dunn said...

Hey Fab.

I'm a huge fan of RestFixture and the tests it's letting me write. But I wanted to use scenarios to help do more BDD style testing as well as improve reuse. I saw the work here ttps://github.com/ggramlich/RestFixture/tree/ScriptTableRestFixture and put some effort into getting his 'RestScriptFixture' working on the lastest version of your RestFixture. The end result is here:

https://github.com/gdunn/RestFixture

Very happy with the result and keen to see if it's something you're interested in pulling back into your version.

smartrics said...

@Geoffrey - thank you!!
can you please provide some fitnesse fixtures showing how the script fixture will work?

if it's only for slim, please add them to the slim directory in the source code here https://github.com/gdunn/RestFixture/tree/master/src/test/cat/FitNesseRoot/RestFixtureTests/SlimTests

for the pull to happen I'd like to see unit tests and documentation completed...

i'll have a look at your commits ASAP

cheers
fabrizio

Wizzy said...

Hi Fabrizio,

Thanks for replying to my question so soon. However, I am still facing my issue. Here's the Test-Script I am trying to run:


!|smartrics.rest.fitnesse.fixture.RestFixtureConfig|
|restfixture.xml.namespace.context|!-ns1alias=egain/ws/model/v3/xsd/ss/articleResult.xsd
xsalias=http://egain/ws/model/v3/gen/ss
dsalias=http://egain/ws/model/v3/gen/common
-!|
!|smartrics.rest.fitnesse.fixture.RestFixture|http://${mname}/system/ws/v11/ss/dfaq|
|GET|?portalId=${portalId2}&rangestart=11|200|Content-Type : application/xml||
|let | id1 | header | X-egain-session:(.+) | |
|let | dfaqMaxRange | body |ns1alias:articleResult/dsalias:pagingInfo/dsalias:maxRange[text()]||
|let | dfaqcount | body |ns1alias:articleResult/dsalias:pagingInfo/dsalias:count[text()]||
|let | dfaqPageNumber | body |ns1alias:articleResult/dsalias:pagingInfo/dsalias:pageNumber[text()]||
|let | dfaqRangeStart | body |ns1alias:articleResult/dsalias:pagingInfo/dsalias:rangeStart[text()]||
|let | dfaqRangeSize | body |ns1alias:articleResult/dsalias:pagingInfo/dsalias:rangeSize[text()]||

!|ActionFixture|
|start|RangeStartTest|
|enter|getMaxRange|${dfaqMaxRange}|
|enter|getRangeStart|${dfaqRangeStart}|
|check|RangeSize|${dfaqRangeSize}|
|check|PageNumber|${dfaqPageNumber}|
|check|Count|${dfaqcount}|

My RangeStartTest fixture actually calculates the RangeSize, PageNumber and Count values based on the MaxRange and RangeStart values that I am passing to the Fixture. These values that need to be passed are being captured in the preceding testcases using LET. Now when I actually run this test, I am getting the result in FitNesse like "Undefined variable: dfaqMaxRange" etc. I am also getting a java NumberFormatException:

java.lang.NumberFormatException: For input string: "undefined variable: dfaqMaxRange"
at java.lang.NumberFormatException.forInputString(Unknown Source)
at java.lang.Integer.parseInt(Unknown Source)
at java.lang.Integer.parseInt(Unknown Source)
at RangeStartTest.getMaxRange(RangeStartTest.java:8)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at fit.ActionFixture.enter(ActionFixture.java:54)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at fit.ActionFixture.doCells(ActionFixture.java:24)
at fit.Fixture.doRow(Fixture.java:165)
at fit.Fixture.doRows(Fixture.java:159)
at fit.Fixture.doTable(Fixture.java:153)
at fit.Fixture.interpretFollowingTables(Fixture.java:119)
at fit.Fixture.interpretTables(Fixture.java:105)
at fit.Fixture.doTables(Fixture.java:79)
at fit.FitServer.process(FitServer.java:81)
at fit.FitServer.run(FitServer.java:56)
at fit.FitServer.main(FitServer.java:41)


Kindly suggest a solution for this issue. Am I going wrong somewhere?

Thanks a lot,
Viji

smartrics said...

@Wizzy

your fixture needs to fetch the value from the fitnesse symbols map

RangeStartTest.getMaxRange(RangeStartTest.java:8)

should not expect the de-ref value, instead it should treat the input as the name of the global variable created by let()

String getMaxRange(String label) {
String val;

if (Fixture.hasSymbol(label)) {
val = Fixture.getSymbol(label).toString();
}

// here val is either null or value extrated by let()
}

(please use github issue tracker for support questions)

Geoffrey Dunn said...

@smartrics

Fitnesse tests added for RestScriptFixture and lots more commenting on the class. Stopping now before I get too tempted to go nuts refactoring

But I'm still open to suggestions

Wizzy said...

Thanks a lot, Fab. Am able to pass the variable values captured by LET to my fixture using hasSymBol and getSymbol functions. Also, I am now returning Boolean value instead of numeric so now my tests are passing.

Geoffrey Dunn said...

Got unit tests in now too

Wizzy said...

Hi Fab,

I noticed that if our response contains accented characters, then RestFixture is not able parse the response and capture values using LET. If our response contains Accented Characters then 'null' values get captured in LET variables. Is there any way to fix this?

Thanks,
Viji

threv said...

@smartrics

Is there support for HTTP Digest Authentication?

Thanks!

Matt

smartrics said...

@threv no, only partial Basic Authentication.

Anonymous said...

Hi,

How to set Headers to get output in JSON format?

I tried in setting headers as Content type- application/JSON and still it is returning the format in JSON.

Can you please help in resolving this issue or can mail me the FitNesse table format to get the output in JSON format.

E-mail- sriram.rln@gmail.com

Unknown said...

@smartrics: We use the RestFixture extensively and it has enabled us to automate the vast majority of our pre-dloyment testing for our 200+ APIs. I can't thank you enough for this!

However, I now have a new API that requires calling one service to get an authentication token then call the system under test. the base url is different so I tried using the setBaseUrl method but have not been able to do so successfully. Are you aware of any examples of how to use this?

smartrics said...

You should be able to specify an absolute uri in the get such as
¦ GET ¦ http://....¦ ...

Or alternatively create a table to get the token and assign it with ¦ let¦ token¦....¦

tThen use it with %token%

Please use github issues tracker for further support

Anonymous said...

How to use Jsonpath here ?
Wanted to compare actual and expected data but jsonbody.book[*].auther=='Nigel' is not working

{
"store": {
"book": [
{
"category": "reference",
"author": "Nigel Rees",
"title": "Sayings of the Century",
"price": 8.95
}
]
}
}

Thanks in advance

smartrics said...

@Anonymous try
jsonbody.book[0].author=='Nigel Rees'

please use github issues for support requests.

Ray said...

Hi,

Will Fitnesse work using SSL PKI certification. I have tried to get the RESTfixture running against an application which uses PKI but have failed

Ray

$ukr!$h said...

Hello!
I am really new to this entire Fitnesse thing and I am successfully able to use the RestFixture.

However, I was wondering if I could write a custom fixture which extends the RestFixture and still invoke the GET methods directly.

The reason why I need to do this is to be able to perform data validations right within my fixture, by comparing my database results against my json results, all in one place.

I am trying this at the moment

!|com.lalaland.common.test.fitnesse.http.MyValidatorFixture|http://localhost:10090|theRestFixtureConfig|
|GET|/lala-order-management-service/v1/order?orderId=1412|200|||

MyValidatorFixture simply extends RestFixture and does nothing more at the moment.

However, Fitnesse does not seem to be happy with the GET call and throws the following error:

GET Missing class or Missing method. Possibly:

public Type gET200(Type1 arg1, Type2 arg2) { }
public Type gET(Type p1, Type p2, Type p3, Type p4) {}

Any help would be appreciated!

Many thanks,
Joey

smartrics said...

$ukr!$h

in GitHub you'll find RestFixtureExt with code extending RestFixture for reference.

https://github.com/smartrics/RestFixtureExtensions

smartrics said...

Ray, RestFixture can easily be extended to support PKI (it's all in the RestClient).
But i don't plan to do this enhancement for the foreseable future.