One of the defining questions for a development team is “do the developers know when they are done”. The question is a simple one but too often the answer is not as obvious as it should be. Whatever form requirements are expressed in, it is important for the development team to be clear on what is to be implemented in order for them to know “when they are done”. Acceptance Tests (ATs) are an effective way of expressing requirements in a way that provides an unambiguous answer to this question
The idea and practice of writing ATs definitions before the coding even begins has both obvious and more subtle benefits which we will discuss in the next section. What are the obstacles to writing ATs before the iteration planning begins and how can they be overcome? We have sought to find answers to these questions in the last year and in doing so have developed an adaptation of iteration planning that has some key benefits over the traditional approach.
Our experience of iteration planning was perhaps typical. As fully signed up members to the agile development cause, we all agreed that this form of planning was better than what we did before, and yet in our shared experience the process was never wholly successful and not without some pain. We were following the advice of the various Extreme programming mantra’s so maybe we were doing something wrong?
Our customer would describe each of the stories in turn. The developers would ask questions and discuss solutions before producing a list of tasks for each story and an estimate. This process of discussion more often that not took a long time and was once described by our customer as feeling more like a “techno babble” session than a planning one. From a developer perspective we found that it was very difficult for our customer to provide the information we needed in order to produce confident estimates in the meeting. This resulted in best-guess estimates on the incomplete information. Even though our customer tried time and time again to consider all of the likely angles they couldn’t anticipate all of the questions that were going to be asked during the session.
The real pain came when we measured our velocity at the end of the iteration and discovered the discrepancy between what we had signed up for and what we had achieved. There are, of course, many reasons why a team could have a poorer than expected velocity but once we had investigated further we discovered that the main issue was that we hadn’t identified a complete set of tasks for each story. We could all see that the developers were working really hard and the results produced were good but in hindsight it was if they had started a 400m race but didn’t know where the finishing line was and just kept going round the track. Our understanding of the work was incomplete which meant it was unclear when we were done, which meant everyone was very tired and nobody was very happy. So we knew we had to do something.
One of the biggest problems we had in our planning sessions was that our customer felt that no matter what they did to prepare there were still many questions they could not answer without time for further analysis or/and investigation. Our solution was simple: a day or two before the end of the iteration the customer would sit down with the QA engineer and a developer to begin the process of writing ATs for the upcoming stories, or in our terms, “Getting our Stories Straight”. In short, we still ended up asking the same questions but by doing this activity before the planning session we were able to find more answers earlier and thus be better prepared for when we did gather the full team into a room. This is not a return to “big up front analysis” but an acknowledgement that as we approach the end of an iteration we already have a good idea of what is going to be completed in this iteration and the customer has a good idea of what stories they want to include in the next.
Are our tests any good? Writing AT’s is a skill, our QA engineer helps our customer make sure the acceptance criteria specify the expected behavior and external quality. One of our developers is on hand to make sure that there is no technical reason why the functionality required to pass the ATs cannot be implemented. If we liken this to UML use cases, a story becomes the title of the use case and the ATs become the use case itself detailing the main success scenario, alternate and error scenarios. [11].
The chance to improve our estimates: For a story that appears too large we may choose at this stage to break it into two or more smaller stories. In our experience, smaller stories are easier to estimate accurately. We strive to have stories that can be ideally completed in 1 to 3 days.
The last point is an important one. Using ATs as our “single source of truth” means the needs of both the customer and the QA function are served by a single artefact. We would later see how the same ATs could be used to improve our task breakdown and estimates. The key to these additional benefits was our decision to choose ATs as our focal point for all planning and development activities. With our stories “straight” the next thing to improve was how we ran our planning sessions.
In Acceptance Test Driven Planning the typical XP planning session is refactored into something we call the Planning Workshop. The objectives remain the same as the XP planning session (updates to our story estimates with a clearer picture of their dependencies), but we find the quality of the information provided is much improved.
As a result of us getting our stories straight we now have a set of candidate stories for the next iteration, their ATs, some existing high level estimates and a vague understanding of what dependencies exist between stories.
The Planning Game Workshop begins with the customer presenting the candidate stories to the rest of the team. Each candidate story is described along with its ATs.
The developers then group themselves into triples and each mini team signs up to a subset of the stories for further analysis. We organise ourselves into triples because we’ve found that 3 is the optimum number of people to have in a mini team; we find that 3 people are able to fit around a pair station and the fact that we have an odd number always results in a mediator in case of disagreement.
Once each candidate story has a triple assigned we move into the phase designed to drive out the tasks required to pass the ATs. Each triple heads back to a workstation and pulls up the ATs for their candidate stories. The customer and QA float between the triples answering any questions that may arise to make sure that everyone is clear on what is required in the story. At this stage the existing ATs may be altered or (more rarely) added to.
It is useful and illustrative to consider what we did before and the reasons which led us to this process of task identification. In previous iterations, it became obvious to us that there was a discrepancy between the tasks identified for a story and the actual work that was required to complete a story. It was common to find a developer saying that they had finished the tasks for a story and now only had the ATs to implement. The problem was that that it took the same time to implement the ATs as it had taken them to complete the tasks. Their original task breakdown, with all the best intentions, was incomplete. We quickly found that by using the ATs as the focus of the task identification exercise we were able to produce a list of tasks which was much more representative of the work that we would need to do to complete a given story. With the increase in confidence we gained from a more structured approach to task identification, we also found our estimates improved too.
The next phase of our planning workshop is called “Present and Challenge”. The idea is that each triple presents their solution for a given story back to the rest of the team, and the rest of the team are then given the opportunity to challenge the list of tasks to see if we can refine the task list and the estimates given. This works well on two levels as it is a useful tool for exploring (at a high level) the proposed solution with the whole team present, whilst also acting as a double-check ensure we have considered all of the tasks.
In addition, this process gives the team members an opportunity to practice their presentation and debating skills. We have also found it can be useful in establishing team rapport – the sessions are designed to be challenging and we manage them carefully to avoid challenge turning into conflict.
We have spoken specifically about “Getting Our Stories Straight” and “The Planning Workshop”, but where do these activities fit into the larger picture?
Figure 1. ATDD Timeline
In a two week iteration we are now able to limit the impact of planning activities down to a single day for nearly all of our developers.
The morning marks the end of the existing iteration and begins with a show-and-tell session to allow interested (and paying) parties to come along and see what the team has achieved in the iteration. There are usually few surprises here as the customer and QA will have seen each story as it was completed during the iteration, but it is a useful opportunity for the team to appreciate all that has been achieved.
We then follow this with a short retrospective [12] where we discuss those things that went well, not so well, things that still puzzle us and any actions we need to take to improve.
An optional addition to the morning programme is a technical retrospective which we have used on occasion when there have been significant changes to the code base which not all members of the team have been directly involved in. This of course is a “smell” of sorts but, regardless, is often a useful way to get everyone back on the same page before we begin the next iteration.
A “Big Up Front Thinking” session at the end of the day is also optional but is sometimes used by the team to draw broad brush impressions (class and simple interaction diagrams mostly) of how the code might evolve for the set of stories we are just about to begin working on. Big up front design? Not really. It is just a way to share ideas and get clear in people’s minds current ideas on where we might go next.
The planning part of our process is time-boxed and takes 2-3 hours as compared to our previous and much less effective 6-8 hour marathon session.
The day is over and it is time to go home and look forward to another iteration.
Kent Beck has always been keen to stress that none of the guidelines or practices are new in themselves but that the organisation and order of their application is different. When practiced together they produce a benefit greater than the sum of their parts. The 12 plus practices can broadly be seen to fall into either one of two camps: planning or development type activities. We would argue that development is emphasized over planning yet in practice successful planning is more difficult to achieve.
The familiar mantra of test-code-refactor, combined with continuous integration and source control has given us a development environment with all the right ingredients and a fairly prescriptive recipe on how they should be practiced. From a developer standpoint these practices are self-serving and self-satisfying: we do them because they make our lives easier, more productive and fun. We continue to refactor not because we are told to but because we know that if we don't keep the code base clean we (or some other poor soul) will have to pay the cumulative price of working with our code debt. We write tests first because they drive out the functionality of the application. As a bonus they provide us with a suite of tests that signal when a change results in breaking something else in the existing application.
The planning activities, which make up the other half of XP, are much less prescriptive in how they should be applied. XP style planning, especially for those used to more traditional heavyweight processes, can initially appear almost too simple to work. Ultimately planning is about capturing requirements in some shape or form, prioritising which are most important, estimating the effort it will take to implement those requirements and delivering demonstrable business value on a regular basis. If only it was always this simple: The customer does not always know what they want when we need them to express their requirement: the requirement is not always clear to the developers; the developers are not clear when they are done; and task breakdown is incomplete, resulting in volatile velocity.
This paper describes an adaptation, or evolution to XP style planning which takes the existing planning practices (with some additions) and organises them in a way which we believe can lead to better planning and more predictable results - tall claims indeed. In developing the process, we have aimed, at every juncture, to find the least amount of work we could do and still make informed, quality decisions. Experience has taught us that this balance point is not easily found.
Over time we have recognised that Acceptance Test Driven Planning appears to have a ‘Goldilocks’ like quality – any more process and we feel the overhead, and any less and the process begins to fail. We do, of course, continue to look for ways in which we can improve the process but after many months and many projects, the balance we have found by planning and driving the development in this simple way feels “just right”.
Richard Watt: Richard is a coach and developer working with ThoughtWorks in the UK. In the last three years Richard has spent a large part of his time coaching and mentoring teams in XP and other Agile development methods. He has been a practitioner of XP since late ’99 and can’t remember how he developed software before then but is sure the results weren’t as good.
David Leigh-Fellows: David Leigh-Fellows is an Agile process consultant and Iteration manager. He has run many software projects since he came across Kent Beck and Martin Fowler in 1998. Dave has 12 years experience of commercial software development and enjoys helping teams realize their true potential by helping them find their own way
To Owen Rogers and Ivan Moore for their contribution to the early development of some of the ideas presented in the paper; for being smart; and being great fun to work with.
To Alan Francis, Areiel Wolanow, and Ally Stokes for their constructive feedback in the earlier drafts; to Rachel Davies, Alex Howson, and Stuart Blair for their time and support in helping get the paper into a fit and proper state as the deadline loomed.
And finally, to my colleague Rebecca Parsons for her hard work and encouragement in helping us make sure our work was best represented.
2. Johan Andersson, Geoff Bache, Peter Sutton. XP with Acceptance-Test Driven Development: A rewrite project for a resource optimization system. http://www.carmen.se/research_development/articles/ctrt0302.pdf
