Agile 2008 - Report

As promised a few days ago, this is my report on the sessions I attended at Agile 2008.

Before the comments on the session I attended, I would like to mention that the whole conference did not blew me away. I mean, it was very good: the hotel, the location, the people, our active participation with two talks and all of that. But I had every day the feeling that all of that was too much. I am not used to group gatherings of this size so that may explain. Anyway, there were great sessions going on, some of which I could not attend for work commitments. And this time around I decided to attend different types of sessions, outside my comfort zone.
I have learned a lot from attending the sessions described below and - not surprisingly - I came back home with more questions than answers. Happily I am still in touch with some people I met at the conference following up ideas initially discussed there. So thumbs up for me over all to the whole experience. And Toronto is a great City.

Keynotes

"The Wisdom of Crowd" by James Surowiecki

Without a doubt this was an interesting opening session. It clearly was aimed to support the speaker's book The Wisdom of the Crowd, but non-the-less new news for me. I got interested in collective intelligence when I read Programming Collective Intelligence, aimed to a more technical audience. This keynote gave me the possibility of know of another aspect of the same topic: how to extract information from a group of people when they are identified as an entity on its own. The crowd, that is.

The common way to extract information from the crowd is by aggregating information coming from the single entities in the crowd. One crucial aspect - very well discussed in this speech - though is how to make sure that "crowd" is built to maximise the chances that the aggregation of ideas can generate new information: this is a case of "the total is bigger than the sum of all the single parts"

So, the key point is that under the right conditions a group of people can be effectively intelligent, contrary to the commonly perceived thinking that crowd are volatile and stupid. Obviously the catch is to understand what the right conditions are.

The right conditions are "diversity" and "independence".

Diversity has to be understood as cognitive diversity, not sociological: different approach to problem solving, different levels of skills and technical capabilities, as opposite to race, sex or social class. The speaker refers to different anecdotes (taken from the book) that prove its hypothesis, inferring also that diversity is even more important in small groups. In essence, homogeneous groups tent to hide flaws as ideas are always analysed in the same way (that is solutions are found by agreement rather than by conflict). In these cases an approach could be to have someone playing "Devil's advocate", with the aim to look at ideas and outcomes from a non standard perspective.

Independence is as much important as diversity. Looking at the crowd as the entity providing the most accurate and effective solution to a given problem, does not mean that the individuality is not important. Individuals must be independent and be able to express their best in any given situation. If this doesn't happen the aggregated results are biased and possibly wrong. This may happen because by nature individuals tent to imitate each other or because organizations punish diversity.

So overall an enlightening speech that I can relate to on my day to day job.

“Quintessence” by Robert "Uncle Bob" Martin

This session happened just after the Thursday's gala dinner. I must say that Bob Martin is a character. On the content of his session, the main points were related to how Agile is implemented in the real life and on how the developer community should behave professionally and push back to those who want them to deliver crap code on time.

About agility, the focus was on XP and Scrum and how - especially at the beginning - the two school of thoughts differed. XP more strict and guided, Scrum more relaxed about actual software development practices. He did an interesting experiment during the session. He asked the attendees to raise their hands. He then started to enunciate each of the XP practices asking people to put their hands down if they weren't following the specific practice.

I reckon we were over 1000 people (if not more) and at the end there must have been 10 people with hands up. I think the point o the experiment was to show that we are at a stage where people blend practices and principles of Scrum and XP to make what works for them.

The other crucial point made was on developers having to put courage on the line and never compromise on good code for new features to accommodate insane deadlines or scarce resources. "We value good code over crap" was suggested as another (unlikely) entry in the Agile Manifesto.

So a very interesting pitch full of truth and controversy. I can not disagree with most of the talk. I still have to find though a place where developers have such a discipline that they can draw a line between over-engineering, quality, speed and alignment with business goals.

"The Wisdom of Experience" by Alan Cooper

The closure keynote was nothing new to me having read The Inmates are running the asylum. Same message as Uncle Bob's for quality over quantity. The speech was full of anecdotes of teams and companies shipping "on time and on budget" bad and unsuccessful software. The key is "take your time and produce good and complete software" that people can enjoy using. The way to nirvana seems to complement agile principles. He's suggesting a typical lack of good software design (especially interfaces) and he's proposing to spend and invest on good and correct design even before a single line of code is written.

All the second part of the talk was dedicated to advertise "interaction design", the new speciality he is proposing and how this can bridge the gap between technical skills and business skills. An interaction designer knows what people want and how to make software usable and easy to understand. Interfaces should not be designed by developers. Users on the contrary should not be allowed to create stories directly to the developers. It is job of this new type of designers to filter and adjust to have the two groups get together for a better solution.

He's got a point: we all know of crappy software with poor user interfaces or with incomprehensible behaviour. Whether fixing this requires a full time job and a specific career path - the way he envisages - is still to be seen.

Sessions

Agile contracting - Rachel Weston, Chris Spagnuolo

I attended this session with the hope to know more about contracts for collaboration between a client and a contractor adopting agile methodologies.

I got a set of problem statements with some strategies on how to mitigate the risks and a bunch of statistics trying to show how agile is better than waterfall.
By the way, I have an issue when agile is sold using numbers like 93% increased productivity or 83% improved satisfaction versus a traditional waterfall approach that gives 35% successful projects. Firstly must be made clear - even more than the numbers - on what conditions these numbers are collected. Second, even thinking about statistics, saying that only X% of projects adopting waterfall is successful doesn't mean that the next project has the X% probability of being successful. The number of independent events that may occur and the people involved are completely different from project to project that the assumption is mathematically wrong. Hence using it as selling point is... incorrect.

Anyway, the most important takeaways from this session were an overall understanding of what problems contractor supporting agile do face. Mainly

• having to compete with "non agile" competitors, who may offer a more sound sense of security in delivering quality software on time.
  
• having to deal with customers who do not embrace agile or that are not fully supportive or that don't have the means to keep up with the fast agile pace
  
• having to provide at bid time of predictability on scope and schedule
  
• having to match customer expectations and internal financial department as for invoicing, payments and costs.
  

Strategies suggested, directly coming from the experience of the speakers were mainly focussed on

• keeping the contract simple.
  
• engaging and educating the customer to the agile practices and explain the technicalities.
  
• clearly state the responsibilities of both parties (customer and contractor).
  
• work out from past velocity charts what can be done, might be done won't be done.
  
• agree with the customer on sharing the risks: loosely define scope when schedule and resource are fixed or instead of the classic time and material.
  
• make sales people responsible of the performance of the contract, so they're focussed on knowing what they're selling.
  

Overall a good session, especially for beginners, with few pointers to think about. Surely, the devil is in the details, but overall the same message applies here: it's all about people collaborating and communicating effectively to achieve a shared goal.

Crafting User Stories – Four Experts and the audience weigh in

I ended up to this session with lots of hope. When talking of user stories it's always easy to come up with some techniques good on paper but not effectively reusable in real life. I must admit, I didn't get the answers I was looking for, but the big plus was to observe 5 experts in the subject matter (dis)agreeing on some concepts that maybe some time ago were considered heresy.
Each expert in the panel was asked to introduce himself and give an overview of their approach to user stories. The moderator then asked attendees to come up with questions for the panel to debate/answer.
I will here report my learning points.

• it’s important for everybody in the team to understand where a story fits in the picture
  
• the format in which stories are formulated is not important as long as few key parts are somehow present: what the story is all about, what it costs to have it implemented (whatever the unit of cost is), who’s benefiting from the implementation of the story and what benefit the story is going to provide, how can stakeholders know when it’s done and to what extent. Also important for me is to know “who” can accept it.
  
• there’s no harm on having stories in the backlog specifically tailored to solve technical problems, as long as the points raised above are clear in the story.
  

Leading agile teams - Mike Griffith

This was a 90mins workshop for agile leaders aiming to learn leadership techniques. The speaker took the audience to a journey where all the aspects of leadership were analysed, especially those revolving around the team leadership.
One of the most important points raised was the importance of maintaining vision of what the team is building and why.
The speaker suggests using the “the product box” technique. The team is supposed to come up with a cardboard box (possibly to be available in the office) representing the product with a name, a logo and its main features. The intent is to maintain focus on what the team is building by means of a real physical object.
Techniques aside, the main learning point on this was that vision is so important that job of the leader is to make sure that it’s consistently updated and propagated to the team regularly.
It was also stressed the difference between managing and leading (the latter being build on the former) and – interestingly – the job of the leader is to build a team that eventually behaves like the “Orpheus orchestra” – which plays without a designated “Maestro” or like the Canadian geese, who make of self organisation and team work a matter of life and death.
For more information on the subject see http://www.apln.org and http://www.leadinganswers.com

Memorable mention for

• Evolution of the Tools and Practices of a Large Distributed Agile Team


• Pushing the boundaries of testing and Continuous Integration (for this talk you can find the slides here and a video of the talk here - thanks Robbie!)

RestFixture is now available

I have made the RestFixture (discussed in this post) available on GitHub here: http://code.google.com/p/rest-fixture/.

You can get it, use it and modify it under LGPL terms.

Feedback is more than welcome, of course.

Agile 2008 - part 1

Oh yes, Agile 2008... here I am, just landed from Toronto. I am just reshaping my notes. I'll be writing (as I have also done in the past) my comments on the sessions I have attended and presented, and about old friends I have talked to (Rick, Rachel, Angela, JB, Steve) and new people I have met. Stay tuned.

Get FitNesse with some Rest

UPDATE: code moved here: http://github.com/smartrics/RestFixture

UPDATE II: Thank you to Steven Haines for the clear and concise Howto giude on the RestFixture.

I am currently involved in building a REST API. Our direct customer proxy (the architecture team) needs to "understand" how we implement the agreed acceptance criteria and it has also asked to document the API.
FitNesse is our tool of choice for writing functional Customer Acceptance Tests; we currently use it to implement this type of tests for other parts of the system we're building. So it came natural to start using it for documenting the API.

At first we implemented the tests using essentially ActionFixtures. The approach adopted consisted of "wrapping" the REST API in more descriptive metods that could be pressed, checked and entered.

After the first dozen of tests it was apparent that this approach was not ideal. Tests were not clear enough and maintaining the fixtures was hard and time consuming as the amount of code duplication was high for any standard. New tests required us writing more code and, in fact, we were testing just the behaviour of the backend without giving enough exposure to the API itself - consequently, our tests were not descriptive enough to work as live documentation.

So I decided to write a new "type" of fixture, inspired by the ActionFixture, the RestFixture.

The core principles underpinning the decision to write a new fixture were the following:

• For documenting a REST API you need to show how the API looks like. For REST this means


• show what the resource URI looks like. For example
/resource-a/123/resource-b/234
• show what HTTP operation is being executed on that resource. Specifically which one fo the main HTTP verbs where under test (GET, POST, PUT, DELETE, HEAD, OPTIONS).


• have the ability to set headers and body in the request


• check expectations on the return code of the call in order to document the behaviour of the API


• check expectation on the HTTP headers and body in the response. Again, to document the behaviour


• I didn't want to maintain fixture code. If I could only write the tests...
• I wanted to be able to let the customer proxies to write the tests... And they understand Wiki syntax more than Java.

The RestFixture at a glance

The RestRixture is an ActionFixture, therefore all the ActionFixture goodies are available. On top of that it contains the following 7 methods:

• header: to be able to set the headers for the next request (a CRLF separated list of name:value pairs


• body: to allow request body input, essential for PUT and POST


• let: to allow data from the response headers and body to be extracted and assigned to a label that can then be passed around. For example our API specifies that when you create a resource using POST, the newly created resource URI is in the Location header. This URI is necessary in order to perform further operations on that resource (for example, DELETE it in a teardown).


• GET, POST, PUT, DELETE, to execute requests.
  

Each test is a rown on a RestFixture table and it has the following format:

|VERB|uri|?ret|?headers|?body|

• VERB is one of GET, POST, PUT, DELETE (at the moment we're not supporting HEAD and OPTIONS)


• uri is the resource URI


• ?ret is the expected return code of the request. it can be expressed as a regular expression, that is you can write 2\d\d if you expect any code between 200 and 299.


• ?headers is the expected list of headers. In fact, the expectation is checked by verifying that *all* the headers in here are present in the response. Each header must go in a newline and both name and value can be expressed as regular expressions to match in order to verify inclusion.


• ?body it's the expected body in the response. This is expressed as a list of XPath expressions to allow greater flexibility.

Some examples

The following pictures are snapshots taken from FitNesse with the aim of providing examples of usage of the RestFixture. The notes before each test explain the details of the fixture itself.








It's worth noticing that, generally, expectations are not matched by string comparing the expected value with the actual value. Expectations on headers are verified by checking that the expected set of headers is included in the actual set of headers, similarly with the body where expectations are matched by checking existences of nodes for a given path.

Conclusions

The RestFixture allows to write FitNesse tests without having to write any Java code to back up the fixtures. Tests are clear and easy to read/write and this obviously improves their readability. Therefore it works well for documenting the API and the behaviour of the system under test. Looking it from another angle, the fixture, essentially, implements a REST DSL, that allows customers to write tests on their own.