. 7
( 10)


Agile Process and Project Management,
Agile Requirements Management,
Agile Communications,
Agile Meetings, and
Agile Automation.
Section 24.8 provides a summary of the ¬ndings of this chapter.

Wherever appropriate, references from the case studies are used to support the
¬ndings in this chapter. In the couple of instances that a reference has been sourced
from one of the case studies that did not make it into the body of the book, this is
made clear in the text.2
Where the case studies have ¬‚agged some cautionary aspect of a particular prac-
tice, the issues raised are discussed and appropriate suggestions for mitigating the
issues are provided.
Finally, while I use the term “analysis” for the method by which I have reviewed
the case studies and collated the agile best practices, I must make it clear that I have
not used any formal speci¬c statistical method to arrive at the conclusions. Rather,
this is more of a qualitative “9 out of 10 cat owners who expressed a preference
said . . . ” style of treatment of the material provided in the studies and other sources.
Although the overall number of case studies submitted represents a very substantial
body of agile work, the sample size and structure of the submissions make it dif¬cult
to perform any statistically signi¬cant analysis.

24.2 Agile Development and Testing
This section reviews a number of agile development and testing practices that the
case studies have highlighted as being particularly valuable from a software quality
perspective. I have deliberately not separated the development and testing practices

2 In such cases, I would be very pleased to supply the original material to anyone that drops me an email.
Analysis of the Case Studies

into two separate sections; I believe making this distinction is both arti¬cial and
injurious to the cause of developing and delivering quality software (as supported by
numerous case studies in this book).

Iteration “ this tried and tested agile development practice provides a powerful means
of bringing testing into the development life cycle as early as possible and ensur-
ing testing continues frequently throughout the project. The early identi¬cation of
defects ensures that the software can be corrected in a timely manner, avoiding the
typical problems of late delivery and cost overruns normally associated with late
discovery of serious defects.
The downside of iteration, from a testing perspective, is that functional tests
and regression test suites grow continuously from iteration to iteration, causing
an increasing testing load that if not managed effectively can cause an overload
in terms of overall testing effort (see Tilt, Chapter 18, for example) “ particularly
toward the end of the project. Use of good test management techniques “ such as
reuse packs and targeted test execution “ can be of value. Test automation through
capture“replay tools (see Phillips, Chapter 12, for example) is another solution that
should be considered. Both the practices of identifying targets of test and the use of
capture“replay tools are discussed in later sections of this chapter.

Pair Testing
Many of the case studies con¬rm the positive bene¬ts of the well-established agile
practice of pair programming (e.g., Thomas, Chapter 6; Cassidy, Chapter 11; and
Evans, Chapter 17) and support the view that, performed correctly, a pair of devel-
opers working together is more productive than if both developers were working
in isolation. A number of the case studies (that of Thomas, for example) report the
need to beware of resistance to this technique by “traditional-style” management on
the grounds that it ties up two programming resources whose time might be better
spent doing tasks separately.
Analogous to the practice of pair programming, the pairing of test resources with
developers has also been reported to be a valuable practice in a number of case studies
(see, e.g., Sewell, Chapter 5; Thomas; Kingston, Chapter 15; and Chana, Chapter 20).
The bene¬ts include testers providing valuable quality assurance input to unit test
speci¬cation, development, and execution, ensuring that developers follow software
quality best practices, and acting as a sounding board for developers to discuss
development and testing ideas (Thomas, for example). A number of the case studies
(Chana, for example) report measurable improvements in the quality of the delivered
code using this practice.
As a cautionary note, you must avoid the perception by developers that the testers
are there as the Quality Assurance Police (Sewell). One solution is to pair developers
and testers but to have the tester review the developers™ work at the end of the day,
for example.
Agile Testing: How to Succeed in an Extreme Testing Environment

Involve Test Resources at All Stages
The practice of involving test resources at all stages is similar to pair testing but
ensures that a test practitioner is available at all stages of the development life
cycle. Sewell (Chapter 5), for example, describes a project in which a dedicated test
resource worked alongside the analyst and the customer during the requirements
analysis phase, generating test cases directly based on the elicited use cases [8]. This
approach was seen to deliver a number of quality bene¬ts:

The tester was available to directly advise on the testability of any given require-
ment (to clarify with the customer how they would verify that a particular require-
ment had been met by the code during testing, for example), allowing any changes
that were necessary to be made while all the relevant stakeholders were still on
The tester was able to improve the quality of the requirements by highlighting
missing, contradictory, or duplicate requirements, using their quality assurance
In parallel with the creation of the use cases, the tester can also review and agree
the suitability of the test case design directly with the customer.

In practice, involving test resources in the project as early as possible provides a
highly effective means of driving testing into the project life cycle at an early stage.

Test-Driven Design and Development
Test-driven design and development (TDD) is a technique that appears to have been
widely adopted in the case studies (for example, Sewell, Chapter 5; Cassidy, Chap-
ter 11; and Kingston, Chapter 15). The success of TDD has also been demonstrated
through research (Stapp and Nowakowska, Chapter 22). Ensuring that developers
specify how they will verify that their code has met its requirements and performed
correctly “ before they write the code “ means that the developer gains a much
greater understanding of the requirement and what the corresponding code needs to
do to meet it. Since the developer would have needed to have written the test at some
point anyway, the overall time taken to complete development and testing of the code
is about the same, with the added bonus that the requirements are better understood
and implemented, that the test is more closely aligned to the requirement, and that
the test is more effective at verifying the correct working of the code. This technique
can be particularly effective when used in conjunction with pair testing.

Fix All Defects Immediately
It is a well-established testing fact that ¬nding and ¬xing defects early in the project
life cycle save substantial time and effort, reducing the risk of late detection of bugs,
the high cost of correcting them, and the likelihood that the project will miss critical
deadlines [4].
A number of the case studies (Phillips, Chapter 12; and Evans, Chapter 17; for
example) describe projects that enforce this principle strictly; if defects are detected,
Analysis of the Case Studies

development is halted until they are ¬xed. By using continuous integration tech-
niques plus automated testing, defects are quickly localized to the most recent code
change, allowing the developers to quickly focus on correcting the error and getting
the project back on track.
In combination with agile practices that allow the team to visualize the real-time
status of the project (such as the red and green lava lamp solution that automatically
¬‚ags a build containing a defect, described by Evans), ¬xing such a defect becomes a
matter of team pride, and there is a determined effort from the team to get the green
lava lamp back on again!

Continuous Integration
Continuous integration is another practice that appears to have been very widely used
in the case studies (Gilb and Johansen, Chapter 4; Thomas, Chapter 6; Knowles, Chap-
ter 10; Wilson, Chapter 14; and Kingston, Chapter 15; for example). Used in combina-
tion with an automated build tool, and incorporating automated unit testing, the code
can be automatically tested to quickly identify quality issues associated with that unit.
Where the automated tests fail, following submission of a new component, it is
reasonably clear where the fault must lie, and the development process can be halted
while the defect is investigated and ¬xed. Used in combination with some automated
and highly visual means of showing that the build is broken, the issue can be quickly
highlighted and other project tasks can be suspended while the team concentrates
on ¬xing the defect.
Tilt (Chapter 18) takes this process a step further and describes a project where
automated continuous build practices had been adopted that not only automati-
cally executed unit tests, but also functional and regression tests (implemented in a
capture“replay tool), providing very thorough test coverage of the application under
development. Wilson further proposes the inclusion of code coverage and code com-
plexity analysis as part of the automated continuous integration and build process.

Test Refactoring
Many case studies (Wilson, Chapter 14; Hodgkinson, Chapter 21; and Stapp and
Nowakowska, Chapter 22; for example) report bene¬ts from the established agile
practice of code refactoring, particularly where a well-experienced agile team has
embraced a culture of joint code ownership. Code refactoring can be particularly
successful if conducted in concert with the use of an automatic build tool [95],
which can also execute associated automated unit tests [50].
Although generally considered a positive technique, Hodgkinson advises that
projects should not overrely on code refactoring (particularly when it is used to
address design problems), since in his experience this is likely to result in projects
failing to make progress.
The use of con¬guration management tools can also be of signi¬cant value in
code refactoring, allowing the changes to be undone in a controlled and accurate
manner if necessary.
Agile Testing: How to Succeed in an Extreme Testing Environment

A number of case studies (Cassidy, Chapter 11; Tilt, Chapter 18; and Chana,
Chapter 20; for example) also describe a testing practice that is analogous to code
refactoring but involving the tests comprising the test suite. Using test refactoring,
test scripts may be changed for the purposes of making them more ef¬cient and/or
more effective, or even removed from the test suite (and stored separately for potential
future reuse). This process is considered distinct from test maintenance, where a test
may need to be changed to re¬‚ect some deliberate change to the application under
Particular bene¬ts can be gained from using test refactoring on projects where
the test suite is very large. Tilt (Chapter 18) describes a project involving some
35,000 test scripts, where the regression suite took up to ten days to complete. Test
refactoring (which included editing existing scripts to make them more ef¬cient, as
well as omitting scripts that were no longer needed), in combination with “laser-
guided testing” (see the next section, Identify Targets of Test), allowed this project
to reduce the test suite execution time from ten days down to less than a day!
Test refactoring should be conducted with caution, and tests should only be
altered if there is some pressing need for change (such as to make them execute
more quickly when overall test suite execution time becomes an issue). As with code
refactoring, the use of a con¬guration management tool is of value to allow changes
to the test suite to be undone in a controlled and accurate manner where necessary.

Identify Targets of Test
The practice of identifying the targets of test is of particular value in large-scale
projects involving large volumes of test scripts (Tilt, Chapter 18, for example), where
it may become impractical or even impossible to run or rerun the entire test suite
because of lack of time or resources. In such cases, it is necessary to target a speci¬c
subset of the overall test suite to execute in the allotted time. Typically, an informal
approach is used that is based on the tester™s experience, omitting scripts that address
“tried and tested” aspects of the system under test, selecting a subset of the overall set
of partition test cases, and making “guestimates” about which parts of the developing
system may have been impacted by the most recent code changes.
A risk always exists, when omitting test scripts from a functional or regression
test, that apparently innocuous changes to one part of the system under development
will have completely unpredictable effects on a seemingly unrelated part of the
system, and the decision to omit speci¬c test scripts from a test must be balanced by
the potential impact of failing to detect such a situation.
Test automation should be considered as one possible solution to enable large
numbers of tests to be executed quickly and accurately (and also to execute overnight
or over the weekend to ¬t additional testing into project timescales using unattended-
testing techniques).
Test analysis techniques (such as Jon Tilt™s “laser-guided testing” approach, Chap-
ter 18) can be used to provide a formal and rigorous means of selecting subsets of
tests to execute and should also be considered.
Analysis of the Case Studies

Code Coverage
Code coverage metrics may also be of value in the process of identifying the targets
of test (to make sure that, in selecting a subset of tests to run, you ensure good
coverage of the underlying code base). Code coverage tools (such as EMMA “ see
Evans, Chapter 17) are also a useful facility to assist you to measure the effectiveness
of your testing (and of automated testing tool use).
However, just because you are generating code coverage metrics, do not be lulled
into a false sense of security about your code quality; code coverage ¬gures on their
own do not improve quality “ actions based on them can (Tilt, Chapter 18).

Rapid Prototyping
Rapid prototyping is another traditional agile practice (although one that is not
without its detractors) that has been highlighted as being of bene¬t in a number of
case studies (that of Chana, Chapter 20, and Phillips, Chapter 12, for example). The
generation of simple prototypes prior to generation of full parts of the system under
development can have a number of bene¬ts:

It helps the developer explore and understand the requirements in more detail.
It provides a practical means of verifying the requirements with the customer.
It provides a powerful and effective means of obtaining customer feedback on the
functionality, look and feel, and usability of the proposed system.

However, in practice, a signi¬cant number of prototypes are abandoned, and the
actual system is recoded from scratch; in these circumstances, it could be argued that
the development effort on the prototype was wasted. Before adopting a development
and testing style based on prototyping, the bene¬ts must be weighed against the
effort involved in developing additional code that is unlikely to be used in the ¬nal
system. Another useful approach is to ensure that the prototypes are deliberately
developed in such a manner that they can be reused in the ¬nal system.

Agile Exploratory Testing
A number of the case studies make a strong argument for the role of manual
exploratory testing. Evans (Chapter 17), for example, highlights the value of employ-
ing a tester to perform the role of a “skeptical” user, creating unusual usage scenarios
for the application under test in order to expose defects that automated tools and
methods would be unlikely to uncover.
The downside of this approach in an agile project is that it can be an open-ended
exercise, with an almost in¬nite number of possible scenarios that could be explored.
Evans™ solution is to engage an experienced and imaginative tester who could ¬t a
set number of effective and ef¬cient tests into the exploratory testing time box. In
practice, caution must be taken to ensure a balance; Stephen K. Allott (Chapter 13)
argues that the restrictive timescales in an agile approach to testing may sti¬‚e the
imagination of exploratory testers and reduce the effectiveness of such testing.
Agile Testing: How to Succeed in an Extreme Testing Environment

24.3 Agile Process and Project Management
This section reviews a number of agile process and project management practices
that the case studies have highlighted as being particularly valuable from a software
quality perspective. The role and use of agile meetings are not covered in this section
but is given a section of its own due to the importance of meetings in agile projects.
The agile process and project management best practices highlighted by the case
studies include the following.

Don™t Throw the Baby Out with the Bathwater
A signi¬cant number of the case studies (Thompson, Chapter 9; Allott, Chapter 13;
Denning, Chapter 8; Evans, Chapter 17; and Chana, Chapter 20) make a very strong
case for considering the reuse of traditional project management practices in agile
For example, V-model has been cited as having been used successfully to help
structure the overall approach to the management of testing of an agile project
Similarly, the use of PRINCE2 has been shown to be of value in the management
of agile projects (Denning, Chapter 8) in terms of having formally de¬ned and
documented roles and responsibilities, having agreed and signed off terms of
reference for the project, and having been tasked by a project steering committee
to deliver against agreed plans.
Formal process improvement schemes, such as the Capability Maturity Model
(CMM [37]), have also been discussed and employed in a number of case studies
(Warden, Chapter 16, for example) to provide a more formal approach to gaining
long-term bene¬t from the lessons learned on previous agile projects.

It could be argued that agile projects employing these traditional methods are
only “agile-esque” “ that they haven™t genuinely embraced an agile approach properly,
and that the same project could have been completed faster, cheaper, and with better
quality using a purist Scrum approach, for example.
However, my personal feeling is that you should make use of whatever tools you
¬nd that are able to bring you tangible bene¬ts and that you ignore tried and tested
project management solutions at your own peril; if it works for you “ reuse it.

Well-Bounded Iterations
Many of the case studies have reported the bene¬ts of arranging the iteration interval
around some “naturally” occurring time span, such as a one- or two-week period
(Kingston, Chapter 15; Gilb, Chapter 4; Tilt, Chapter 18; Hodgkinson, Chapter 21;
and Chana, Chapter 20; for example).
Where it is possible to ¬t an iteration into such a period, it appears that the
natural rhythm of working in a predictable weekly cycle helps to keep the team
members focused on their tasks. For example, in Tom Gilb™s case study, developers,
Analysis of the Case Studies

testers, managers, and the customer all know that they will be preparing for and
performing some speci¬c activities on each day of the week.
One potential downside of this practice is that certain activities (and typically,
testing) may be squeezed out of the iteration due to lack of time “ for example, where
a time-box approach is being used, if the time allocated to development overruns
then the time allocated to testing is likely to be eaten into. One suggestion to
ensure testing is not missed out is to give the testing task its own separate time box

Agile Estimation
Estimation is a critical aspect of both agile and traditional software development
projects. Historically, estimation has been a commonly cited area for concern in
traditional projects, with cost overruns and failure to deliver on time or to budget
being a common experience.
Although the majority of the case studies did not raise agile estimation as an
issue, some notable concerns were raised: Kingston (Chapter 15), Warden (Chapter
16), and May (Chapter 23) all reported challenges associated with agile estimation.
Warden reports that, on a Scrum project he was involved with, there were con-
sistent problems with estimation (and particularly with accurate completion dates).
The metrics and estimates displayed on the burndown chart would appear to be
static across the majority of an iteration but would then suddenly slip toward the
end of the iteration. Since a key reason for employing Scrum is to quickly identify
project slippage, this phenomenon was a cause for concern. This issue appeared to be
associated with accurate calculation of velocity; Warden uses the following analogy
to illustrate the problem:

The on-board computer in my car has to solve a similar problem to predict the range
of the car. It knows how much petrol is left in the tank and the past fuel consumption.
The question is how do you use past consumption in a predictor algorithm? Do you take
the last 25 miles driving as representative, or 50 or 75? Likewise do you take the last
Scrum cycle or use more? Furthermore, how do you normalise story counts so they are
comparable across cycles, as stories ranged in size by a factor of 10?
Another issue with agile estimation that Warden identi¬es is what he terms “sucking
the pipe-line dry at the end of a cycle.” To maximize velocity, staff would be allocated
to clear as many user stories as possible, including QA team members who would
be pulled off testing duties. As a result, although the velocity metrics improved, this
approach was storing up problems for later in the project, where there would be
reduced timescales and resources, with the inevitable time-scale compromises for
testing the software.
In terms of the former estimation issue highlighted by Warden, metrics can
play an effective role in addressing the reliable and accurate calculation of velocity.
Keeping historical data provides the means of normalizing estimation data and also
allows the optimistic and pessimistic reporting styles characteristic of different team
Agile Testing: How to Succeed in an Extreme Testing Environment

members to be recognized. A number of products are available that allow estimated
and actual effort to be automatically collated and which can even adjust different
practitioners™ estimates based on previous data (see [60, 63], for example).
In terms of the latter observation, ensuring that testing plans and resources are
ring-fenced is an effective solution to ensuring testing practitioners are not pulled
off their quality assurance task and onto project ¬re-¬ghting tasks. Approaches used
by Sewell (Chapter 5), Thomas (Chapter 6), and Phillips (Chapter 12), for example,
have also promoted the idea that having testers involved throughout the project life
cycle, and of closer working practices with developers, may actually act to prevent
the situation Warden described happening in the ¬rst place.
May (Chapter 23) identi¬es the dif¬culties of estimating the time and effort
required to produce the code to implement user stories as a challenging exercise,
particularly for practitioners new to agile methods. The information was used to
calculate sprint velocity and, hence, estimate when the product backlog would be
¬nished. Increasing experience with estimation and the use of metrics collected
during previous sprints meant that subsequent estimates became increasingly more
accurate. May also cites that keeping the sprints short (of two weeks™ duration in this
case study) was of bene¬t in keeping progress on track.
Finally, Kingston reports issues with estimation on an offshored agile project.
In this instance, the problems with compiling accurate progress estimates from the
offshored development team was attributed to problems with time zone differences,
the dif¬culty with convening ad hoc offshored meetings, lack of formal cross-geo
reporting tools, and the challenge of not having face-to-face communications when
the weekly telephone conferences were held. Kingston observes that the use of an
agile approach actually highlighted these issues much more acutely than a traditional
project approach would have done.
The solutions to these issues that Kingston identi¬es include improving commu-
nications with the offshored team, ¬rmly but politely challenging all estimates, being
sensitive to possible language and culture differences, focusing on better communi-
cations infrastructure (such as instant messaging and Internet telephony systems),
the use of Web 2.0 applications (such as holding virtual meetings in Second Life [64])
and videoconferencing technologies, adopting tools that support 24 — 7 working and
process enactment (such as [63, 65]), improving the working time overlap across
time zones (such as getting one team to start and ¬nish work thirty to sixty minutes
earlier each day while the other team starts and ¬nishes thirty to sixty minutes later
each day), and, last but not least, getting a member of the offshore team to co-locate
with the onshore team to help address language and culture differences (these points
are expanded upon in Section 24.5).

Progress Measurement
The ability to measure progress in software development projects is a key aspect of
their successful management, allowing actual progress against plans to be compared
and remedial action taken where problems are identi¬ed.
Analysis of the Case Studies

Frequently progress is assessed by collating the estimates of individual progress
from the members of the team. However, this approach is notoriously dif¬cult to
perform accurately; individual developers and testers may report their own progress
optimistically or pessimistically (or may even be “economical with the truth” about
what they have or have not done). In small co-located agile projects, this is less
likely to be an issue, because frequent contact between the project stakeholders,
plus practices such as pair programming, all act to ensure the availability of more
accurate progress information.
On larger agile projects, or where the stakeholders are not co-located, this may
be an issue. Kingston (Chapter 15) and Evans (Chapter 19), for example, describe the
challenges of collecting accurate progress estimates from offshore team members
where time differences, cultural differences, and just simply the lack of con¬dence
that face-to-face communication provides can cause signi¬cant errors in estimating
progress. A number of simple solutions can be effective in addressing this issue:
Project managers or team leaders should use simple, clear, and unambiguous
language when discussing progress and should not be afraid to challenge any
estimates with which they are not comfortable. The estimates should be fed back
to the team to ensure that the team agree with them.
Find other solutions for “face-to-face” communication with offsite or offshore
team members such as videoconferencing or webcam technologies, or perhaps
by using a tool that can support a virtual meeting, such as Second Life [64].
Ensure you collect metrics on estimates and actuals and use these to “normalize”
future estimates (in effect, balancing optimistic and pessimistic reporting styles).
Use one of the so-called process enactment tools [63] that ensure developers
and testers follow a de¬ned agile process, and where data are automatically
collected on planned and actual timescales as the team members plan and execute
their tasks (even automatically adjusting estimates for optimistic and pessimistic

Progress Visualization
On a topic related to progress measurement, a number of the case studies have
reported the bene¬ts of publishing or displaying the project progress (often in real
time) to the project stakeholders. Examples have included the following:

Publish quality metrics, such as the breakdown of where defects are found in the
project life cycle (Tilt, Chapter 18) to encourage healthy competition between
stakeholders in the project (all project metrics were displayed graphically for all
stakeholders to see, which was found to be “a great motivator”).
Publically display project burndown charts (Warden, Chapter 16; Phillips, Chapter
12; and May, Chapter 23) to ensure all stakeholders are able to review the progress
information (but ensure the data are accurate and up to date).
Wilson (Chapter 14) describes a very effective approach in which code complexity
and code coverage information are shown to the users as color-coded graphics
Agile Testing: How to Succeed in an Extreme Testing Environment

to display the results of tests executed automatically using his continuous inte-
gration, build, and test architecture.
Evans (Chapter 17) describes an automated system where a broken build detected
during the continuous integration process would cause a green lava lamp to be
turned off and a red lava lamp to come on. It became a matter of signi¬cant pride
for the team to ensure that the red lava lamp was switched off before the wax
could become liquid and start to ¬‚ow!

Availability of Agile Process Guidance
A number of the case studies stress the importance of providing simple and effective
access to agile best practices for the team members. Sewell (Chapter 5), for example,
describes the bene¬ts of the use of process summary cards from the Essential Uni¬ed
Process [34], whereas Hodgkinson (Chapter 21) and Chana (Chapter 20) discuss the
role of the Rational Uni¬ed Process [7], with its distributed delivery of best practices
via a web browser interface.
Although availability of agile best practice guidance on a project is important, the
availability of an easily accessed and used source of such information should not be
used as a substitute for thorough and effective training in the use of the agile method.
Many of the case studies emphasize the importance of well-planned and delivered
training in the success of agile projects (Gilb, Chapter 4; and Phillips, Chapter 12;
for example).
In a similar manner, where standard reusable agile templates (such as test script
templates) are available for use on a project, these should also be widely accessible
and easily customized. Watkins [4], for example, provides access to downloadable and
customizable testing templates, and Appendices E through G of this book provide
templates for a number of agile testing artifacts.

Role and Use of Metrics
A number of case studies describe the use of metrics on agile projects and their
bene¬ts (Norman, Chapter 7; Wilson, Chapter 14; Phillips, Chapter 12; and Tilt, Chap-
ter 18; for example). These can include

gaining a quantitative means of assessing the success of some activity “ for
example, having the data available to know that the adoption of a particular
practice (such as pair testing) has reduced the number of defects detected prior
to release of the software by N%;
being able to publish or make information available to the other stakeholders
to allow them to see the state of the project, to encourage practitioners to
improve their performance (perhaps competitively against their colleagues), and
to manage the customers expectations;
re¬ning the process of estimation. Metrics can be used to compare planned or
estimated timescales, cost, and effort with actual ¬gures. This sort of analysis can
be used to re¬ne future estimates and plans;
Analysis of the Case Studies

being able to report gains in productivity, reductions in cost, and reductions
in delivery timescales achieved by adopting a particular agile method to senior
being able to publish gains in productivity, reductions in cost, and reductions
in delivery timescales achieved by adopting a particular agile method to other
interested parties in your organization (via newsletters, your intranet, or by lunch
and learn presentations);
supporting a process improvement program (covered in more detail later in this

While there are undoubtedly very good reasons for setting up and administering
a metrics program, there are a number of important things to consider in setting up
such a scheme:

Make sure you understand why you are interested in metrics “ what do you plan
to do with the data (such as improving productivity, reducing defects, and/or
process improvement)? This will in¬‚uence which data you need to target and
Understand your current baseline; you cannot make statements about how your
software development process has changed unless you have a quantitative view
of where you were before you introduced your agile approach. In practice, this
means collecting data as early as is practically possible and requires a great deal
of foresight. In some organizations, you may be lucky and data (such as time
sheet information) may already be routinely collected.
Be inclusive rather than exclusive in terms of the data you collect; later on in the
project, it is very dif¬cult go back in time and collect some speci¬c data that you
suddenly discover would have been useful to have collected.
Collect metrics on the time, effort, and cost of setting up and administering the
metrics program “ you should know how much metrics are costing you.
Finally, be aware that the activity of collecting metrics can alter the value of those
metrics that you are collecting (known as the Hawthorne Effect [96, 97]).

Watkins [4] provides detailed information on the setup, roll-out, adoption, and
ongoing maintenance of a metrics program.
The adoption of a metrics program on an agile project must be balanced by
the cost of setting up, administering, and maintaining such a program. Wherever
possible, make use of tools that automatically collect, analyze, and report on metrics
for you “ such as integrated test management tools that can automatically produce
reports on test case design versus requirements coverage, for example. Similarly,
process enactment products (see [63]) are able to collate developer estimates on
both task completion times and the actual values.
Also remember that the activity of collecting metrics itself is not necessarily
useful, but that taking action based on the metrics is where the value is obtained
(Tilt, Chapter 18).
Agile Testing: How to Succeed in an Extreme Testing Environment

Process Improvement
The ability to ¬ne-tune your agile process, to learn from successes and failures, and to
apply the lessons learned to future projects can be of great value. Many case studies
(Gilb, Chapter 4; Allott, Chapter 13; Kingston, Chapter 15; Warden, Chapter 16;
Evans, Chapter 17; and Tilt, Chapter 18) advocate the use of process improvement in
their agile projects, and many of the established agile methods incorporate process
improvement in their practices, such as Scrum™s Sprint Retrospectives and Evo™s
Key Process Principles.
Metrics is a subject closely related to process improvement; if you don™t have
the data to verify that a particular strategy for improving some aspect of your
process has been successful, it is dif¬cult to substantiate claims that there have been
improvements. If you intend your agile approach to incorporate effective process
improvement, you must seriously consider the role metrics can play in such a
As a ¬nal thought, a number of the agile case study authors hold strong views on
formal process improvement methods; Gilb, for example, has been heavily involved
with the Capability Maturity Model (CMM [37]), and Warden discusses the role of
CMM on agile projects in his case study.3 Although such a formal approach to process
improvement, involving signi¬cant time, effort, and cost to implement, may seem to
be at odds with the goals of agile methods, many organizations derive bene¬ts and
value from these schemes.

Everyone Is a Software Engineer
It is one of software development™s oldest clich´ s that there are frequent project
tensions between testers and developers; developers have often perceived testers as

the people who were far too meticulous, found defects that the developers had to
¬x, made delivery of the software late, and made the developers look bad; or
the people who didn™t do enough testing, allowed code to be delivered that still
contained defects, and made the developers look bad!

As clich´ d as this view may appear to be, there are still many case studies in
this book that demonstrate that this “us and them” tester“developer issue is still
Denning (Chapter 8), for example, describes a particularly extreme example of
this situation, where the developers and testers were in a virtual war with each other,
in which the developers actively looked for opportunities to make the testers look
bad to justify their cut of the bonus payment cake!
More enlightened practitioners (including Sewell, Chapter 5; and Phillips, Chap-
ter 12) describe agile projects in which the developers, testers, and other stakeholders

3 Although Richard Warden does highlight issues where the ¬‚exibility and responsiveness of agile
approaches are at odds with the goals of predictability and repeatability that the CMM scheme seeks to
identify in projects.
Analysis of the Case Studies

are all encouraged to take personal responsibility for software quality. Phillips takes
this one step further and describes a project where the developer and tester roles are
subsumed under the title of software engineer.
Other agile practices can also help challenge these negative prejudices; many
of the case studies describe an approach termed pair testing (Sewell, Chapter 5;
and Kingston, Chapter 15; for example). In this practice, developers and testers are
paired (in a manner analogous to pair programming) to work together on software
quality aspects of the development. This approach appears to have been particularly
bene¬cial; in one such scheme, Thomas (Chapter 6) reports that developers even
began to actively seek out testers to bounce development ideas off them and to get
their feedback.

24.4 Agile Requirements Management
Without requirements how do you know what it is you need to develop and how do
you prove the software does what it is supposed to do? These question are as relevant
to agile projects as they are to traditional projects, except that the requirements
elicitation process on traditional projects is often a long, complex, and notoriously
dif¬cult task to achieve; it is estimated that as many as 75% of all projects fail because
of issues with requirements [98].
The time taken to elicit requirements combined with the copious amounts of
documentation that commonly need to be created and maintained seem completely
at odds with the goals of agile development. This section discusses a number of best
practices employed in the case studies that enable requirements to be managed in
an agile manner.

The Role of Use Cases and User Stories in Test Case Design
A large number of the case studies (Sewell, Chapter 5; Wilson, Chapter 14; Kingston,
Chapter 15; and Warden, Chapter 16; for example) report the bene¬ts of use cases [8]
and user stories [20] in eliciting the customer requirements for the system. These
approaches elicit simple and easily understood examples of how the user interacts
with the system that is to be developed (or that is in the process of being developed)
to produce some outcome of value to the customer.
From a quality management perspective, the scenarios documented using these
practices are not only useful in assisting the developers understand what they need to
produce but also map very naturally into test cases. Sewell, for example, describes how
involving the testers in the requirements analysis alongside the analyst and customer
representative enabled accurate and effective tests to be designed in parallel with use
case capture, driving testing earlier into the project life cycle and ensuring that the
customer was satis¬ed with the correctness of the resulting test case.

Reducing Requirements Documentation Overload
Keeping documentation and the need for document maintenance to the minimum
level possible is a well-established agile principle, and a number of the case studies
Agile Testing: How to Succeed in an Extreme Testing Environment

(Sewell, Chapter 5; and Evans, Chapter 19; for example) describe speci¬c practices
that they have employed to achieve this.
A great deal of documentation on projects is used for communication, such as
passing requirements information to developers, testers, and other stakeholders.
Sewell describes the value of daily stand-up meetings in improving communications
and reducing documentation to manageable levels on a large agile project; the
overhead of documentation is avoided because the team members are able to talk to
each other about important issues, rather than having to type the information into
an email or a change request tool, and can ensure that it is distributed to the people
who need to know about it.
Sewell also describes the bene¬ts of having all the stakeholders co-located from
the start of the project, and in particular the bene¬t of having testers work alongside
the analyst and customer representative during the requirements elicitation task. In
effect, this short-circuits the need to provide long, complex, and potentially out-of-
date requirements information to the test team.
Evans (Chapter 19) describes a number of techniques for improving communica-
tions that can provide direct, honest, speedy, and productive information exchange
with the goal of producing fewer and less formal documents (such as the Weaver
Triangle [72]). The techniques that Evans describes can also be usefully applied to
the process of requirements elicitation itself.
Denning (Chapter 8) makes an important point about the role of formal documen-
tation in commercial projects, where it proved essential for a company contracted
to perform a large and complex testing task on a commercially critical system to
document the roles, responsibilities, and terms of reference for the project. Without
this documentation, issues about the scope of the project and the responsibilities
of the staff performing the work could have potentially put the pro¬tability of the
contract at risk.
From a review of the case studies and the very different perspectives on the role
and use of documentation that they provide, a useful approach is to not be dogmatic
about documentation but to look for strategies to reduce the project documentation
capture and maintenance load. Automation, such as requirements management
tools, for example, could provide such a solution.

Look Beyond Requirements to Real Customer Needs
Gilb (Chapter 4) makes the case in his Evo method [32] for delivering customer
value rather than sticking slavishly to traditional requirements elicitation practices.
A key principle of Evo states that you cannot know all the customer requirements in
advance, but that you discover them more quickly during the process of delivering
valuable working software.

Ensure All Requirements Have Tests
Except on the most trivial of projects, without requirements how do you prove
the software does what it is supposed to do? Good testing practice demands that it
Analysis of the Case Studies

should be possible to show that each requirement has one or more tests designed
and implemented against it.
Even though agile development best practice mandates that all units have unit
tests associated with them, Sewell (Chapter 5), Cassidy (Chapter 11), and Kingston
(Chapter 15) extend this view by ensuring that any and all requirements can be
shown to have tests written against them.
Tilt (Chapter 18) describes an approach used on a very large project in which
rigorous test metrics and their statistical analyses were employed to formally demon-
strate complete code and requirements coverage.
Similarly, Wilson (Chapter 14) describes an agile continuous integration and
build approach that uses tool support to automatically verify test coverage com-
bined with a powerful graphical display to highlight areas of concern to the project
Chana (Chapter 20) also describes the value of tool support in providing auto-
mated requirements management as well as the bene¬ts of traceability that such
products can provide (to determine who proposed and who authorized particular
requirements or changes to requirements, for example).

Role and Use of Requirements Tools
As discussed in the previous sections, a number of the case studies (Wilson, Chap-
ter 14; and Chana, Chapter 20; for example) describe the value of using tool support
for requirements management.
Chana describes the use of a commercial requirements management tool [61]
to document, manage, and maintain rapidly changing requirements on a small-
scale agile project. Traceability is a key bene¬t that Chana reports, allowing him
to quickly and easily use the change history facilities of the tool to identify who
had requested and authorized a particular requirement or change to an existing
requirement. Other stakeholders were able to review and provide comments on the
requirements through a web-based user interface, allowing non-co-located team
members to be closely involved in the requirements process.
Wilson describes the role of requirements coverage as part of the wider tool
support for a continuous integration build and testing process that he developed. In
addition to being able to provide a graphical color-coded display of the underlying
code coverage metrics, a speci¬c bene¬t of this approach is to establish con¬dence
in the testing for any given build or release; as Wilson puts it, “It is better to have an
80% unit test pass rate on 95% of the code, than to have a 100% pass rate on 5% of
the code.”
As with any tool, care must me taken by the project manager or team leader to
avoid letting the tool become the focus of attention, rather than the task the tool has
been acquired to achieve; I have seen far too many technical people become obsessed
with ¬ddling and ¬ne-tuning the features and facilities of a tool instead of focusing
on the purpose of the tool “ to save time and effort, and improve the quality of the
delivered product!
Agile Testing: How to Succeed in an Extreme Testing Environment

24.5 Agile Communications
Good communication is a fundamental principle of successful agile projects, and I
would argue that the majority of agile best practices are communications-oriented.
The following sections review some of the communications best practices highlighted
in the case studies as being of particular bene¬t, including advice on offshored or
distributed agile projects.

Co-location of Stakeholders
Co-location of stakeholders is a traditional and universally supported agile best
practice throughout the case studies. Even the authors of those case studies (such
as Thomas, Chapter 6; Kingston, Chapter 15; and May, Chapter 23) where there are
issues about lack of co-location of staff accept that, in the ideal situation, the teams
should be working together in the same place at the same time. A number of the case
studies describe speci¬c quality management bene¬ts associated with co-location:

Norman (Chapter 7) reports that, where developers and testers are co-located,
developers are able to gain an understanding and appreciation of the tester role
and even develop a professional respect for testers and testing.
Sewell (Chapter 5) reports that the availability of testing practitioners at all stages
of the project has been highly bene¬cial. For example, involving testers with the
users and analysts during requirements elicitation has enabled the testers to
challenge the testability of particular use cases “ asking questions like “how
would you verify if that (feature) works?”“ with the result that the overall quality
of the requirements has improved. Similarly, pairing developers and testers has
enabled the testers to review unit test designs and provide their feedback to the
developers, with the result that the effectiveness of unit testing has improved.
Thomas (Chapter 6) also reports the bene¬ts of co-locating developers and testers
in terms of the closer working relationship that he saw forming, in his case study;
in a number of instances, developers were observed to actively seek out testers to
obtain their views on new ideas that the developers were working on.
Co-locating the customer or users with testing practitioners also has bene¬ts,
such as allowing test analysts to speak to the users to clarify some aspect of a use
case or user story during test case design, for example.

Non-co-located Teams
Although co-location of the stakeholders in an agile project is the ideal goal, a couple
of the case studies (such as Thomas, Chapter 6; and Kingston, Chapter 15) describe
projects where necessity (and the goal of saving costs) meant that some members of
the team were offsite.
Thomas describes a project where most of the development was carried out
in North America, with testing and release management in the United Kingdom.
While Kingston describes a project that was U.K. based (management, design, and
Analysis of the Case Studies

the higher levels of test), the majority of the code development was offshored to
India and the customer perspective was provided by a small team in the United
The case studies describe some very valuable best practices for improving the
communications of non-co-located teams trying to work in an agile manner.

Set up and use an effective project communications and working infrastructure,
such as
videoconferencing facilities in combination with tools to share and remotely
operate workstations (Thomas);
Web 2.0 social networking products (Chana, Chapter 20), including instant
messaging and virtual meeting tools (such as Second Life [64]); and
the use of distributed process enactment tools (see [60, 63]) to support dis-
tributed collaborative development (Kingston).
Harmonize offshore time zones “ adjust the working-day start and end times (by
perhaps thirty to sixty minutes in each location) to improve the overlap of the
working hours between the different time zones (Kingston).
Be sensitive to different cultural and language differences between different sites
(Evans, Chapter 19); use simple and clear vocabulary, avoid using idiomatic
phrases and/or colloquialisms, and don™t be afraid to repeat, reword, or stress
some important point that needs to be communicated.
Consider having a representative from the offshored team co-locate with your
local team; Kingston reports that this is a valuable approach to use in building
a trusting relationship with the remote group. The local representative can also
help address the language and cultural differences that can become ampli¬ed
through poor remote communications.

Improving Interpersonal Communication
In addition to having the opportunity to communicate, a number of the case studies
address the issue of the quality of communication (Evans, Chapter 19; Kingston,
Chapter 15; and Evans, Chapter 17; for example). That is, how can you ensure that
all participants in a meeting are able to contribute to the success of the meeting?
Holding a retrospective to discuss the lessons learned on the last iteration or on the
project as a whole is likely to be less than successful if the person who knows the
most is the least talkative, or the meeting is hijacked by someone who wants to tell
everyone about their fantastic new sports car!
The following communications practices provide a valuable means of ensuring
you get the most from your agile meetings:

Evans (Chapter 19) describes a technique called de Bono™s Six Hats in a Meeting
[77], in which the members of a meeting are encouraged to “wear different
hats,” each of which denotes some particular mode of thinking “ for example,
the Black Hat might represent a critical viewpoint, whereas the Green Hat might
represent a lateral-thinking, brain-storming perspective. “Putting on a hat” allows
Agile Testing: How to Succeed in an Extreme Testing Environment

an individual to move outside their perceived stereotype, perhaps allowing a very
factual person to express emotional views, for example. The hats also provide
a less personal metaphor for communicating; it may be easier and sound less
critical to tell someone that they sound like they have their Black Hat on rather
than telling them not to be so negative about something.
A number of the case studies (May, Chapter 23, for example) highlight the issues
that lack of focus in agile meetings can cause. A particular technique covered
in Evans™ case study that can help address this is that of Ishikawa ¬shbones [78,
79]. This graphical cause-and-effect analysis technique provides a highly focused
approach for drilling down to the root causes of problems. The term ¬shbones
refers to the graphical structure4 that is created as the technique proceeds;
team members add their ideas (perhaps as part of a brain-storming exercise) to
the diagram, creating what looks like a ¬shbone (see Figure 19.1). Evans also
describes the role of this technique to move from the problem space to a solution
by “drawing the diagram in reverse.”
What happens in an agile meeting (such as an application design workshop [18])
where active and effective communication is essential, but where the facilitator
is ¬nding it dif¬cult to get the design discussion started or to keep the process
running? Evans describes a technique termed a Weaver Triangle [72], which can
be used to assist a team to focus on an overall aim or goal. This graphical technique
is structured using a triangular diagram (see Figure 19.2), which establishes a
particular goal or aim at the top of the triangle. Through discussion, this high-
level goal is broken down into more speci¬c goals, each of which are documented
on the diagram under the top-level goal. Finally, speci¬c objectives that will allow
the goals to be achieved are discussed and documented at the base of the triangle.
This analysis provides a team with a powerful technique to proceed from the
speci¬c (a high-level objective or goal), through clari¬cation of the high-level
objective into subgoals, and ¬nally into speci¬c objectives to achieve the subgoals
and, hence, the overall objective.

If you would like to understand how to improve the quality of communications
on agile (and traditional) projects in more detail, I wholeheartedly recommend you
obtain a copy of Isabel Evans™ book [72].

24.6 Agile Meetings
Having established that a very large part of the success of agile projects is associated
with effective communication, this section focuses on a particular set of activities in
agile projects where communication is conducted in a more formal manner “ agile

4 This technique could be of particular bene¬t on a co-located agile project; the ¬shbone diagram could
be set up on a whiteboard or wall in a location where all team members can review the diagram and add
their input.
Analysis of the Case Studies

All agile methods stress the importance of regular, structured face-to-face meet-
ings involving the project stakeholders, and this is certainly supported by virtually
all of the case studies.
A wide range of reasons for and styles of meeting are advocated in the various
agile methods, and range from simple ¬fteen-minute stand-up meetings to review
progress, agree next tasks, and identify risks and barriers to achieving those tasks
(as seen in Scrum projects), to several days for joint application design workshops
(such as those held in RAD projects).
This section provides best practice guidance on agile meetings distilled from the
experiences described in the case studies and other sources.

Agile Project Start-up Meetings
Project start-up meetings are an essential best practice in many of the existing agile
methods (forming part of the preproject phase in DSDM, for example). A number of
the case studies also cover the role and use of start-up meetings in the projects that
they document.
Kingston (Chapter 15), for example, describes the value of holding an initial
project start-up meeting to introduce the stakeholders, con¬rm roles and responsibil-
ities (and lines of communication), review and agree the tasks and overall timescales
(at least at a high level), and to plan and prepare for the ¬rst iteration.
Denning (Chapter 8) reports the value of employing traditional project manage-
ment approaches to agile testing projects and describes how PRINCE2 [22] project
management controls, including a formal start-up meeting, were used to provide an
appropriate approach to kicking off an important commercial project. In this case
study, Denning™s company was contracted as a third-party testing partner, and it was
commercially expedient to formally identify the project sponsor and have the terms
of reference agreed and documented.
To ensure that all participants who are permitted to speak at an agile start-up
meeting are given an equal opportunity to participate, and to ensure that those less
extrovert members of the meeting are still encouraged to provide their valuable con-
tribution to the proceedings, consider using the agile communications techniques
described by Evans (Chapter 19) in her case study.

Agile Iteration Start-up Meetings
The established agile methods, such as Extreme Programming (XP) with its iteration
planning meetings and Scrum with its sprint start-up meetings, all promote the role
and use of formal start-up meetings for each iteration. These meetings are used to
agree the tasks to be worked on (selecting user stories in XP, for example) and their
timescales, to agree how the tasks are assigned, and to identify risks and discuss their
A number of the case studies (Kingston, Chapter 15; and Phillips, Chapter 12;
for example) also include a checkpoint in iteration start-up meetings to review and,
where appropriate, incorporate the lessons learned from previous iterations. Phillips
Agile Testing: How to Succeed in an Extreme Testing Environment

reports that reviewing the lessons learned has been particularly bene¬cial in terms
of keeping the project testing processes as up to date as possible and ensuring their
relevance to the project testing requirements.
Kingston explicitly includes the discussion and agreement of the test plan for
the iteration, plus any speci¬c timescales, resources, and/or staff that are needed to
support the iteration test plan in the iteration start-up meeting agenda.

Daily Meetings
Both Scrum and XP promote the use of short daily stand-up meetings (typically
limited to ¬fteen minutes). The purpose of these meetings is speci¬cally to talk
about what was achieved the previous day, what is planned to be worked on today,
and what risks or barriers there might be to achieving these goals.
Such meetings are highly focused on the work at hand, even to the extent of
only permitting certain team roles to speak while others attend simply to listen
to the proceedings (such as the pig and chicken roles in Scrum projects). In XP,
an interesting incidental goal of the daily meeting is to reduce the need for other
meetings that might take place through the day.
Sewell (Chapter 5) reports that the bene¬ts of holding daily stand-up meetings
have included helping to reinforce the concept that effective testing is the key to
delivering higher-quality code and also in breaking down the “us and them” issues
many projects observe between developers and testers.
May (Chapter 23) provides a cautionary warning that overrunning, poorly fo-
cused, and poorly facilitated stand-up meetings are likely to be a symptom that an
agile project is failing to be run in an agile manner. May urges that teams adhere
closely to the duration and principles of the stand-up meeting and observes that
where such meetings overrun or are ineffective, it is likely to be because the meeting
has been hijacked to discuss design issues or to debate how best to ¬x a particular

Interim Iteration Meetings
Gilb (Chapter 4), Chana (Chapter 20), and Kingston (Chapter 15) all report signi¬cant
bene¬ts of working to a “natural” iteration period (such as a one- or two-week
iteration interval).5 These authors also discuss the role of regular interim iteration
meetings and their bene¬ts in ensuring that good progress is maintained, speci¬cally
in the iteration but also in the project in general.
Gilb describes the bene¬ts of such meetings in providing an additional opportu-
nity for the project manager and customer to review progress against the iteration
plan and for the customer to review and approve the tasks and associated design for
the next tranche of development and testing activity.
Chana reports the bene¬ts of integrating a weekly iteration meeting into an
existing regularly scheduled team telephone conference call. In the case of his project,
5 See the Timeboxing topic under the earlier Agile Process and Project Management section for more
Analysis of the Case Studies

Chana was able to provide progress updates to his project manager during the course
of the call and obtain feedback from the users (his teammates) on the state of the
developing system (a team Wiki) as well. Chana cautions care in piggy-backing on an
existing meeting and recommends that the agile review items are formally included
in the meeting agenda and covered off early in the meeting.
Kingston describes the bene¬ts of such a regular meeting on an offshored agile
project. Speci¬cally, a weekly telephone conference call with the developer team in
India was employed as an additional project control to con¬rm that the tasks in the
current iteration were on track and that there were no risks or dependencies that
might cause the team to fail to meet its deadlines. Kingston does advise caution in
obtaining progress report information outside of a face-to-face meeting environment
and recommends that all estimates are ¬rmly challenged (often with the same ques-
tion being repeated in a slightly different manner) to avoid any misunderstandings
due to language or cultural differences.

Agile Retrospectives
Gilb (Chapter 4) makes the important observation that a very important lesson to be
learned is that there is always another lesson to be learned. A key practice in his Evo
method [32] is to continually ¬ne-tune the agile best practices by capturing iteration
and project lessons learned and acting on them.
In an agile project environment it is particularly important that the agile process
and practices being used on the project are, and remain, as effective and ef¬cient
as possible. Sewell (Chapter 5) makes the point that during the course of a single
project (even of moderate duration) it is possible for testing best practices to change,
for new techniques to be published, or for new tools to become available (or even new
releases of existing tools with new functionality). Being receptive to such changes
and, where appropriate, adopting and using them can help ¬ne-tune the effectiveness
of an agile project.
Phillips (Chapter 12) describes the use of a testing speci¬c retrospective at the end
of each iteration in which the lessons learned in the previous iteration are reviewed
and, where appropriate, actions are put in place to improve the testing process and/or
keep it up to date with respect to new best practices or tools that may have become
available since starting the project.
In agile retrospectives it is particularly important that all members of the team are
encouraged and given the opportunity to contribute their views, and the techniques
for improving communication between team members described by Evans (Chapter
19) are particularly appropriate to these meetings. Another valuable perspective on
agile retrospectives, and techniques for making them successful, that I recommend
you look at is provided by Ester Derby in her book [99].

Although many of the more recent agile methods emphasize the bene¬ts of short,
sharp, and highly focused meetings, there may be project tasks that must be
Agile Testing: How to Succeed in an Extreme Testing Environment

accomplished that do not lend themselves easily to a ¬fteen-minute stand-up meet-
ing. RAD and DSDM, for example, both include guidance on running longer sessions,
such as RAD™s joint application design workshops. Where complex issues need to be
discussed, where important decisions need to be agreed upon, and where key project
artifacts (such as the business requirements document or the project test plan) need
to be generated, a longer style of meeting may be necessary.
Denning (Chapter 8) emphasizes the role of more traditional project management
approaches to running agile projects, including the role of longer meetings where
necessary and their success in managing customer expectations.
May (Chapter 23) makes a case for “extended meetings” on projects where the
customer and development team are not co-located to ensure the developers have
gained a thorough understanding of the customer needs as well as an appreciation
of the customer culture and preferred means of working. In practice, these meetings
could be scheduled across a number of days to achieve their goals.

Agile Project Closedown Meetings
With the emphasis the case studies place on learning from experience in agile projects
(Gilb, Chapter 4) and process improvement (Tilt, Chapter 18, for example), an agile
project closedown meeting can provide an excellent opportunity to re¬‚ect on the
lessons learned throughout the project.
Kingston (Chapter 15) describes the successful use of such a meeting and its value
in providing an opportunity to harvest the lessons learned from all of the previous
project meetings.

Meeting Focus: A Final Thought
As a ¬nal thought on the subject of agile meetings, I offer a particularly embarrassing
moment from my own career, which emphasizes the need for good focus in meetings
as well as the need for a strong facilitator.
I still cringe when I recall a project meeting I attended in my younger days, which
was being chaired by a formidable project manager, a lady named Ruth Woodhead.
In what I thought was a slack moment in the meeting, and while Ruth had to
take a short but important phone call, I began to chat with a colleague about the
recent rugby success that Wales had been enjoying. The next thing, Ruth was asking
me conversationally “ “John, do you like the radio quiz show ˜Just a Minute™?”
“De¬nitely,” I answered enthusiastically, thinking I could earn some brownie points
(and not spotting the trap); “DEVIATION,” she said simply, instantly taking back her
iron grip of the proceedings, and leaving me blushing acutely! Suf¬ce it to say, I have
never forgotten the lesson.

24.7 Agile Automation
This section provides best practice guidance on the use of automated software tool
support for agile methods. Automation has the potential to save signi¬cant time,
Analysis of the Case Studies

effort, and cost on agile projects, but, where selected or used poorly, it can also cost
a project time, effort, and money!
As Jon Tilt (Chapter 18) puts it, “There is always the temptation to architect yet
another automation framework at the start of the project. Don™t! Look around at the
simple tools that already exist and build on them as required. Developers and testers
are much more likely to use something simple that is quick and easy to use.”
In addition to quality management and testing products, this section discusses
agile best practices relating to other tools such as requirements management, con-
¬guration management, build management, and process enactment tools.

Select a Tool in Haste, Repent at Leisure
Phillips (Chapter 12) reports that the agile project he worked on was forced to change
tools several times during the course of the project; the time and effort needed to
select a replacement tool, plus the nugatory effort expended on gaining familiarity
with the previous tool, and wasted effort using it, can all jeopardize the success of
an agile project, where the goal is typically to reduce the effort and cost of delivering
the product.
Prior to starting an agile project there are a number of things you can do to try
to avoid making poor automation decisions:

Determine whether there are company standards in place for tool use (but don™t
follow these slavishly if they are clearly wrong for your project).
Discuss the need for tool support with knowledgeable colleagues and consider the
wider networking opportunities of attending appropriate special interest group
Be aware of the need for a particular product to integrate with other tools (either
in place or planned to be acquired).
If the project is large enough, of suf¬ciently long duration, and with available
resources, consider conducting a formal evaluation of tools (particularly if they
are products you will need to purchase). A tried and trusted tool evaluation
scheme that can be used to ensure formality in any evaluation you undertake is
given in [4].

Static Analysis Tools
These products typically allow the source code to be inspected to highlight quality
issues without executing the code (some are even customizable to allow company
coding standards and house coding style issues to be detected). Arguably, these
relatively simple but highly effective tools should be another tool in the developers™
toolbox and should be used on a frequent basis to check recently written or modi¬ed
As with many of the products discussed in this section, static analysis tools can
be particularly effective when used in conjunction with an automated build and
testing approach. For example, Evans (Chapter 17) describes a system that combines
Agile Testing: How to Succeed in an Extreme Testing Environment

continuous integration and build management tools to automate the creation of a
new build of the software. During this process, automated unit, functional, and static
analysis tests are run against the code to determine whether the build is a success.
Should the build fail due to defects being identi¬ed, the build is rejected, and the
defects are corrected before rerunning the build process once more.

Automated Unit Test
The use of automated unit test tools (such as [50]) appears to be an agile prac-
tice adopted in almost every case study (Cassidy, Chapter 11; Wilson, Chap-
ter 14; Kingston, Chapter 15; Warden, Chapter 16; Tilt, Chapter 18; and Stapp
and Nowakowska, Chapter 22). These simple and effective tools allow unit tests to be
compiled in concert with code development and executed to ensure the code meets
its requirements and performs as expected.
Although a wide range of unit test products are available to purchase or download
as open-source tools, many developers also write their own unit test tools. Whatever
solution is adopted, you should take into account support issues, the availability of
practitioners who are familiar with the tools, and the need to maintain and update
these tools.
Combined with test harness tools (used where the unit under test needs to interact
with components that have not yet been coded) such as mock objects (Cassidy, for
example), automated unit tests provide an inexpensive and effective means of initially
testing a component as well as continuing to test the successful functioning of the
component following a new build, release, or other changes to the software (such as
code refactoring).
Finally, employed in concert with continuous integration and automated build
tools [95], automated unit tests become a highly effective solution to accelerate
development through continuous code quality assessment.

Test Harness Tools
It is often the case, particularly in the early stages of development, that code will
be written that needs to interact with other components that may not have been
developed at that point in time. Under such circumstances a typical solution is to
employ a test harness to simulate the operation of the missing component(s).
It is not unusual for developers to spend some of their time writing and testing
their own test harnesses (or reusing those generated by colleagues). Although of value
in supporting more ef¬cient unit testing, creating these test harnesses inevitably
expends time, effort, and expense in generating a temporary tool that almost certainly
will not have any purpose once the current project is completed.
To increase the productivity of developers and to save them the time and effort of
building their own test harnesses, a range of open-source test harness tools have been
developed that are available for use in unit testing. Evans (Chapter 17), for example,
describes his experiences in using the so-called mock framework test harnesses and
their role in automated build and test, while Cassidy (Chapter 11) describes how
Analysis of the Case Studies

his project began by using a number of freely available mock frameworks (such
as EasyMock [50]) and then went on to develop its own generic reusable mock
framework “ SevenMock [51].

Functional Testing Tools
Functional testing tools, such as the classic capture“replay tools, allow testers to
capture user navigation against the application under test and any speci¬c veri¬ca-
tions (that a particular window has appeared, or some speci¬c calculation has been
made correctly, for example) in the form of an automated script. This script can be
replayed against later builds or releases of the software to ensure the functionality
has not changed and that the new changes have not introduced any bugs.
The bene¬ts of the use of such tools on an agile project are clear:

Automated test scripts can be scheduled to run overnight or over the weekend,
allowing additional unattended testing to be performed, increasing con¬dence in
the software.
The more often you reuse the automated scripts to perform regression testing of
new builds and releases, the more time, effort, and cost you save the project.
Unless you are dealing with an extremely large suite of automated test scripts and
are facing very long test suite execution times, you do not have to take the risk
of having to pick and choose which scripts to execute “ you simply run them all.
Automated capture“replay tools do not get tired, do not become bored or careless,
and will continue to replay the test scripts and report on the results in a thorough
and repeatable manner.

A number of case studies report signi¬cant bene¬ts from the adoption and use of
capture“replay tools for functional and regression testing (such as Denning, Chap-
ter 8; Cassidy, Chapter 11; Phillips, Chapter 12; and Tilt, Chapter 18) and variously
cite reductions in testing time, effort, and cost, as well as improvements in quality
through their use.6
However, such tools are not a universal panacea, and a number of case studies
report issues with using such tools. Knowles (Chapter 10), for example, reports that
well-managed manual testing delivered excellent results on the migration project
described in his case study. In this instance, capture“replay tools were rejected on the
grounds that the solution was too expensive to maintain in a rapidly changing devel-
opment environment, with high cost and effort involved in maintaining the scripts.
Phillips (Chapter 12) cautions against underestimating the time and effort
required to develop and maintain a test automation infrastructure and particu-
larly one where the capture“replay tool has to be integrated with a number of other

6 Interestingly (in light of the listed bene¬ts for capture“replay tools), Jon Tilt reported that the test suite
on his project had grown to an extremely large size (35,000 tests with an execution time of some ten
days) and describes the use of a “laser-guided” solution to select an effective subset of tests to run.
Agile Testing: How to Succeed in an Extreme Testing Environment

The characteristics of those projects that seem to be particularly suited to the
application of capture“replay tools are as follows:

A stable user interface (Phillips) should be available “ the screens in the applica-
tion under test should not be in a constant state of change (such as that caused
by user interface requirements churn). Under such circumstances, the cost of
maintaining the automated test scripts may well outweigh the cost of manual
There should be numerous and frequent builds and releases of the application
under test. Under such circumstances, thorough and extensive manual testing
becomes unrealistic, and an automated solution is much more appropriate (Tilt).
A further bene¬t of frequent builds and releases is that each time the automated
test suite is reused, the overall return on investment in the tool is improved.
The application under test is complex, involving numerous screens and com-
plicated navigation and with a large amount of functionality to test. Where the
software is relatively simple, more cost-effective manual testing may suf¬ce.

As a ¬nal thought, if you combine the introduction of capture“replay testing
tools with your metrics program, you will be able to work out the precise bene¬ts
obtained from using the tool and make an informed decision on whether it is of value
to your project.

Requirements Management Tools
A number of the case studies report the bene¬ts of using tools to document and
maintain requirements (Kingston, Chapter 15; and Chana, Chapter 20; for example).
From a quality management perspective, the ability to provide up-to-date require-
ments information to test managers and test analysts for test planning and resourc-
ing, and test case design purposes were commonly cited uses of such tools. Kingston
makes a strong case for the use of such tools to demonstrate formal test coverage
against requirements, while Chana describes the value of the change history sup-
ported by such tools in providing a useful source of traceability (allowing the persons
who requested, reviewed, and authorized a particular change to the code base to be
easily identi¬ed, for example).
Kingston makes a further point about the utility of such tools on agile projects
where not all the stakeholders can be co-located; the ability for remote stakeholders
to access the requirements tool via a web-based interface enabled the offshore staff
to work more closely with other members of the team and allowed all stakeholders
(irrespective of where they were based) to contribute to the requirements elicitation
and documentation process.
Sewell (Chapter 5) and Hodgkinson (Chapter 21) both describe the value of
automated support in requirements elicitation, speci¬cally in documenting and
analyzing use cases within the EssUP [34] and RUP [7] frameworks, respectively.
Analysis of the Case Studies

Build Management Tools
A large number of the case studies document the bene¬ts of adopting a continuous
build or integration approach (Gilb, Chapter 4; Thomas, Chapter 6; Knowles, Chapter
10; Wilson, Chapter 14; and Evans, Chapter 17).
Unless the project is of trivial size and complexity, a build management tool must
be considered for use in starting and running the build process (see [95], for example).
Many of the case studies (such as Evans; Phillips, Chapter 12; Wilson, Chapter 14; and
Tilt, Chapter 18) also report the bene¬ts of integrating other automated tools (such
as unit test, functional test, and code analysis tools) with the build management tool
to ensure that those builds that contain defects are rejected.

Con¬guration and Change Management Tools
Con¬guration and change management are well-established best practices in both
traditional and agile software development projects, and many of the case studies
report the value of using them (Thomas, Chapter 6; and Cassidy, Chapter 11; for
Numerous con¬guration and change management tools are available, including
commercial products, open-source tools, and those developed in-house (see [61], for
example), that can be employed in agile projects. In selecting such a product, you
should be aware of support issues, the availability of practitioners who are familiar
with the tools, and the need to maintain and update these tools.
Finally, be aware of the need for the con¬guration and change management tools
you select to integrate with other tools that you are already using on a project or
those that are planned to be acquired.

Process Enactment Tools
A number of the case studies emphasize the importance of providing good process
guidance to the stakeholders involved in developing and testing the software (Sewell,
Chapter 5; Kingston, Chapter 15; and Hodgkinson, Chapter 21; for example); if you
want your agile project to be successful, it is essential that the practitioners working
on the project understand and are able to employ agile best practices.
Several of the case studies also provide cautionary tales about projects that
claimed to be following an agile approach, but which in reality were not using agile
best practices (Thompson, Chapter 9; Allott, Chapter 13; Warden, Chapter 16; and
May, Chapter 23; for example). The reasons for this can be diverse:

Thompson suggests in his case study that a company made claims about their
agile credentials to win a development contract, but that in practice there was
very little evidence that the company had used an agile approach.
Allott suggests that an agile project needs particularly well-trained and experi-
enced agile practitioners if it is to be successful, otherwise weak or poorly trained
staff will fall back on more familiar (nonagile) practices.
Agile Testing: How to Succeed in an Extreme Testing Environment

Warden highlights a litany of examples where an agile approach was shoehorned
into a project, but where true agile practices were not actually adopted (such as
the failure to ensure that the end users performed the sprint QA/sign-off, instead
of this step being performed by the analysts who wrote the speci¬cations).
Similarly, May documents a number of departures from agile (in this case Scrum)
practices, which jeopardized the success of the project he was involved in,
including poor user stories, interrupted sprints, excessively long sprints (greater
than six weeks), failure to co-locate stakeholders, and poorly focused stand-up
meetings of too long a duration.

There are a number of options for addressing these issues:

Ensure that you employ experienced agile staff who have a good track record of
using agile practices successfully.
Provide staff with appropriate training on the agile method being used and its
practices before you begin the agile project.
Provide easy access to agile best practice guidance “ such as practice summary
sheets (Sewell) or easily accessible online process guidance [7].

Another possible option that has become available recently is to deploy a so-
called process enactment tool (see [63, 65], for example). Such tools provide a rich
integrated development environment in which the use of agile best practices is
enforced by the tool itself. For example, such tools can

allow the addition of new team members to a project using standard templates
for roles and responsibilities, with de¬ned lines of reporting, privileges, and
provide de¬ned work¬‚ows within which tasks can be assigned to other team
members, which can then be accepted and worked on, with the developer entering
estimates of the time and effort required to complete them as well as the ability
to update the task with the actual values;
provide real-time reports on individual and project progress and generate accu-
rate estimates, such as the timescales required to complete a particular task;
be customized to match the speci¬c agile practices of a particular agile method.
For example, MacDonah [100] describes the implementation of the EssUP process
in a commercial process enactment product [101].

24.8 Summary
This chapter has provided a detailed analysis of the agile best practices documented
within the twenty case studies appearing in Part 2 of this book with particular
emphasis on quality management.
From these twenty case studies, more than ¬fty separate examples of speci¬c
agile best practices are documented under the headings of agile process and project
Analysis of the Case Studies

management, agile development and testing, agile requirements management, agile
communications and meetings, and agile automation.
I believe that the agile best practices documented in this chapter will be of
genuine value to existing agile developers and testers and to any practitioners who
are new to agile. However, it is undoubtedly the case that not every agile practice
covered in this chapter will be universally applicable to all sizes, types, and styles of
agile project.
The next chapter, Chapter 25, makes a number of proposals for how different
practitioners involved in different sizes and types of agile project might make use of
the best practices described in this chapter to compile their own agile process.
25 My Agile Process
The easiest way to get any agile process to fail, is to expect too much from it.
Ryan Concannon, Scrum Master

25.1 Introduction
This chapter draws upon the information provided in the cases studies and their
analyses to make a number of proposals for how you might set up and run your own
practical, effective, and ef¬cient agile testing process. Using the set of agile testing
practices highlighted during the analyses, this chapter provides a series of proposals,
allowing you to pick and mix from these practices to create an agile method tailored
to your own particular development and testing requirements.
In providing a valuable source of real-world development and testing practices,
the case studies reinforce the idea that each project is unique. Each project is
of different size and complexity and, in reviewing the agile practices, you need to
consider what will work for you. To simplify this task, the practices in this chapter are
organized around four stereotypical projects, and you should make an assessment
about which one best matches your own agile requirements. Try not to use the
information provided dogmatically; look at the proposed agile practices critically
and decide if you think they would work for you.
As a ¬nal thought, whatever set of practices you pick will probably not be perfect,
so make sure you follow the advice provided on process improvement: keep doing
what works for you and challenge the use of, or modify, those practices that do not.
This chapter is structured as follows. Section 25.2 provides guidance on those
agile practices that have been found to be successful in the majority of case studies
and in the other sources of agile methods described in the previous chapter. These
practices can be viewed as providing a solid foundation on which you can build your
own agile process by adding those practices from the following sections that best
match your own agile requirements.
Section 25.3 provides guidance on those agile practices that have been shown to
provide value on small agile projects,1 characterized by having a team size of up to

1 Although in practice, everyone will disagree with the ¬ne detail of what constitutes a small, medium, and
large agile project, the de¬nitions used in this chapter provide a pragmatic solution to the organization
of the agile best practices described in Chapter 24.

My Agile Process

eight staff members, with low complexity (that is, involving a co-located team, low
budget, small timescales, and, in technical terms, a relatively simple development
Section 25.4 provides guidance on agile practices that have been shown to pro-
vide value on medium-sized agile projects, characterized by having a team size of
between nine and twenty staff members, with moderate complexity (that is, involving
a co-located team, possibly with some offsite members; moderate budget; moderate
timescales; and, in technical terms, a moderately challenging development environ-
Section 25.5 provides guidance on those agile practices that have been shown to
provide value on large-sized agile projects, characterized by having a team size of
greater than twenty staff members, with challenging levels of complexity (that could
include a co-located or partially co-located team, large budget, large timescales, and
a technically challenging development environment).
Section 25.6 provides guidance on those practices that have been shown to provide
value on agile projects characterized as having a large offsite or offshore aspect to
their organization.
Section 25.7 provides a summary of the agile process proposals made in this
To obtain further information on each of the agile practices described in this
chapter, refer back to the relevant section in Chapter 24, where each practice is
described in more detail and which can also be traced back to the source case
study using the references that are provided. The format used to organize the best
practices in this chapter mirrors that used in Chapter 24; the practices are grouped
together within the following sections under the headings Agile Development and
Testing, Agile Process and Project Management, Agile Requirements Management,
Agile Meetings and Communication, and Agile Automation.

25.2 Foundation Agile Best Practices
The best practices documented in this section have been shown to have been used
successfully in the majority of the case studies. These practices can be viewed as
providing a solid foundation on which you can build your own agile process, by
adding those practices from the following sections that best match your own agile
All agile projects should strongly consider the use of the following practices.

Agile Development and Testing
Iterative development “ This fundamental agile practice is valuable from a soft-
ware quality perspective because it allows testing to begin as early in the project
as possible and to continue frequently throughout the project. In addition to
allowing early and frequent testing, this approach also provides value in terms
of allowing early versions of working software to be shown to the customer,
Agile Testing: How to Succeed in an Extreme Testing Environment

obtaining valuable feedback, clarifying the requirements, and also managing the
customers™ expectations of the ¬nal delivered software.
Well-bounded iterations “ Time boxing, sprints, and other techniques for bound-
ing agile development tasks are tried and tested agile best practices. From a
software quality perspective, it is recommended that the planned testing ele-
ment of such iterations be strongly protected to ensure adequate testing occurs.
In projects where the testing time within an iteration is habitually eaten into,
strongly consider creating a ring-fenced iteration (such as a testing time box)
explicitly for testing. Where appropriate, look to ¬t iterations into naturally
occurring intervals, such as one- or two-week periods.
Early involvement of test resources “ Look for opportunities to involve testing
practitioners as early as possible in the project and continue to involve them
throughout the project life cycle. For example, involving a testing analyst or
designer in the process of generating use cases or user stories alongside the
analyst and customer has been shown to be of signi¬cant value by allowing the
test analyst to review the requirements for testability, to help ensure there are no
errors in the requirements (such as duplications, omissions, or contradictions),
and to create the test cases in parallel with the use cases and user stories.
Every day is test day “ In addition to implementing practices to drive testing
earlier into the project life cycle, encourage a culture that is aware of the need
to test the code as frequently as is practical. Unit testing tools can be valuable
in supporting this approach in the early stages of code development. In terms
of looking for additional solutions to increase the amount of testing completed
on the project, unit testing tools can be combined with continuous integration,
build management, and capture“replay tools to allow testing to be run unattended
(overnight or over a weekend, for example).
Test-driven design “ In addition to being a key best practice in a number of agile
methods (such as XP), the case studies have shown that where developers review
the requirements and generate the unit test before coding begins, they gain a
much clearer understanding of the requirements, produce better quality code,
and don™t spend any additional time overall completing the task. In fact, because
of the bene¬ts of gaining a better understanding of the requirements and writing
high-quality code, this practice arguably saves time and effort that otherwise
might have been spent rewriting and retesting code containing defects.
Fix all defects immediately “ A number of the case studies describe a practice in
which, as soon as a defect is detected, all development halts while the defect is
¬xed. This approach has very signi¬cant bene¬ts in terms of avoiding late detec-
tion of defects, and the cost and time-scale implications this has for the project.
Combined with some of the practices allowing for the real-time visualization of


. 7
( 10)