. 4
( 10)


meetings would involve representatives from all walks of the enterprise IT/business
management world, including developers, testers, release managers, con¬guration
managers, project managers, production control, business analysts, change control
board, architects, etc.
These meetings would attempt to address the ideal “standard” that should be
adopted across the enterprise. It very quickly became apparent that there was no
one “standard” that could meet even one customer™s enterprise requirements; the
ideal was understood and seemed logical, but the implementation and management
of any one method didn™t appear comprehensive enough or scalable.
After years of getting involved in such exercises “ understanding, implementing,
and reworking the various development methods available “ I embarked on a
cherry-picking exercise of selecting the strongest elements of multiple techniques,
in order to create a concoction that I embraced as our “best of breed.” In addition to
adopting the strongest facets from each method, it was also designed to be scalable
and manageable from an enterprise SCM point of view. Some purists may argue
that technology can™t/shouldn™t be used to manage agile development; I believe it™s

14.3 De¬nition of an Agile Testing Process “ The Application Development Life Cycle
The majority of my experience has involved the generation of software development
life cycles “ how they are managed, by whom, and for what purpose. I used this expe-
rience to create a template from which I could include the best practices encouraged
from the numerous best-of-breed development methods at my disposal; my main
sources of references being Extreme Programming (XP [20]), Scrum [23], iterative
development [7], and agile development [18].
From these methods, I focused on developing a life cycle that could encompass
the following best-of-breed practices from these best-of-breed agile methods:

From Extreme Programming, I selected refactoring, spike development, user
stories, and minor releases.
From Scrum, I selected Heartbeat Retrospective [54], dynamic information, and
From iterative development, I selected spiral process.
From agile development, I selected working software, response to change, and

The keystone for my life cycle was user stories: the ability to articulate require-
ments in suf¬cient detail to allow a reasonable development estimate to be made. In
terms of managing user stories with technology, Harvest packages and their related
electronic forms were designed to capture the necessary information for a standard
user story.
The Power of Continuous Integration Builds and Agile Development




14.1 Life-cycle work¬‚ow 1.

Once the user stories were created, it was necessary to undergo formal validation
and iteration planning (development planning) “ the process of assigning user stories
to particular iterations, or sprints, based on resources, effort, and priority. Modeled
in Harvest,1 this process looked as shown in Figure 14.1.
You will notice that the terms are not literally extracted from the methods in
which they originally belong; this was to reduce the “cultural shock” and make the
transition more seamless for the end users.
From iteration planning (development planning) the user story could be sent
to the development team, either as a planned piece of work or as a support and
maintenance development.
Depending on the customer requirements, support and maintenance could also
take on the role of “spike development,” as per the XP method [20]. For this particular
customer, the principle of working on the latest version of the code, in an isolated
environment on the branch, lent itself to support and maintenance development but
could have easily been labeled “spike development” with no functional changes to
the SCM product.
The resulting life cycle process was further developed as shown in Figure 14.2.
It is in development that the continuous integration builds were implemented;
this will be covered in more detail later in this section.
Assuming a successful build was produced, the user stories would then be pro-
moted through the SCM product to a staging area where the business acceptance
could take place. Once the business had accepted the changes, they would be pro-
moted for a minor release.
In the event that the business required rework, the user story was demoted back
to development where the necessary changes could be made and the user story reused
1 The majority of SCM products on the market today allow users to de¬ne the life cycle and, subsequently,
the processes in which they want to work. In this case study I will be referring to the functionality and ter-
minology of CA™s SCM product, which I will refer to as Harvest for clarity and to avoid confusion with the
term “software con¬guration management.”
Agile Testing: How to Succeed in an Extreme Testing Environment

Planning Development
Support &



14.2 Life-cycle work¬‚ow 2.

to repeat the process. The advantage of using a tool for managing the process means
that work¬‚ow information can be continually tracked “ identifying the proportion of
“new development” versus “rework,” for example, may be an extremely useful metric
of the productivity of the team, how accurate the requirements are being speci¬ed,
or the capability of the assigned developers (see Figure 14.3).
This completes the end-to end application life cycle, as shown in Figure 14.4.







14.3 Life-cycle work¬‚ow 3.
The Power of Continuous Integration Builds and Agile Development



Planning Development
Support &


Ready for
Acceptance Tests

Test Release



14.4 Life-cycle work¬‚ow 4.

From Figure 14.4 the process should become clear with respect to what has been
implemented and how it has been in¬‚uenced by various development methods.

14.4 Continuous Integration Builds and Automated Unit Testing
The key bene¬ts of implementing the best-of-breed life cycle is that it integrates
seamlessly, both from the process side and the technology side, with the desired CIB
and testing.
In my opinion, the key advantages of CIB are

consistently working software,
automated unit testing, and
panopticode scanning.2
2 Panopticode is a project dedicated to making code metrics widely understood, valuable, and simple so
that their use becomes the norm, thus raising software quality (see [55]).
Agile Testing: How to Succeed in an Extreme Testing Environment

Each time a developer checks a changed piece of code into the CIB tool, this
triggers a build. Each time a spike development “ or support and maintenance “
change is promoted to development, this triggers the automated CIB [56].
A rich CIB tool dashboard provides the users with an enterprise view of all builds,
across all projects.
As part of the CIB, the unit tests are automatically initiated and users automati-
cally warned of any code quality problems. The build process can also be con¬gured
to “fail” if the unit tests fail; this ensures that the application remains operational,
even if it is functionally incorrect. Again, the project stakeholders are able to visualize
this information via the rich user interface of the CIB tool dashboard.
As described previously, the key to a successful CIB process has got to be iden-
tifying the problems before they manifest themselves further down the life cycle. If
problems are missed, they are more costly to rectify further down the line and can
prevent a stable application from being available, delaying productivity and delivery.
However, we must note that the unit tests are only as good as the developers who
write the unit tests (if they write them); this brings us on to unit test coverage. If
we do not manage the unit test coverage then the unit testing becomes ineffectual,
undoing the ef¬ciencies of the entire CIB process “ it is better to have an 80%
unit test pass rate on 95% of the code than to have 100% pass rate on 5% of the
To manage this potential problem, we employ a Panopticode scan that will provide
the users with a visual representation of how much of the developed application has
unit tests and to what extent. In our implementations of the automated CIB tool we
utilize the Panopticode scanning tool to represent this analysis and to visualize the
results by using powerful code coverage graphics.
Following the Panopticode scan, a rich two-dimensional display is generated to
assist the users in visualizing the results. This scalable vector graphic (SVG) is an
interactive, graphical representation of the unit test coverage; green, yellow, and red
represent coverage greater than 75, 50, and 0%, respectively. Black represents the
areas of the application where no unit test coverage exists (blue is “not applicable”).
Two-dimensional boxes represent methods, and these are grouped by the classes in
which they belong.
Again, by using the Panopticode scanning tool it is possible to manage the code
complexity of the applications. By identifying the complex areas of an application it
is useful to assign developers to refactoring activities “ the rework of complex areas
to make them less complex and more useable.
As with the code coverage, we can generate an SVG graph that will assess the
complexity and provide a code complexity number (CCN) rating. The code complex-
ity display uses a style of graphic display similar to that used for code coverage:
green, yellow, red, and black represent CCN ratings from 1“5, 6“9, 10“24, and 25+,
respectively (1 being the least complex and 25+ being the most complex).
With the Panopticode scans, we now have a comprehensive system for identifying
the most complex code and the areas in which unit testing is not suf¬cient. In addition
The Power of Continuous Integration Builds and Agile Development

to the Panopticode scanning, the CIB tool also has various options to include code
analysis tools, including CheckStyle and PMD. These can be very useful for checking
the quality of the code at build time.

14.5 Management Information and Metrics
Since the implementations are managed through a suitable SCM tool, it is possible to
make use of the metadata that are generated and stored in the underlying database.
The process includes the following high-level activities:

user story de¬nition and validation,
iteration planning,
spike development,
development estimates,
iteration durations,
rework/new development,
acceptance, and
minor release.

We can make use of the data generated (manually input data or system data) to
produce useful management information and metrics. For example, we know the
number of developers in a team, the number of user stories, and the development
estimates for each user story. Therefore, we can produce metrics to assist the iteration
planning process by assessing the capacity versus utilization metrics.
Since we know the iteration start dates, iteration duration, and number of devel-
opment days (actual/estimated), we can calculate the burn rates (Heartbeat retro-
spective), iteration velocity, and project velocity [26].
All of this information can again be displayed to the appropriate stakeholders
using the rich graphical user interface (Figure 14.5).
Figure 14.6 illustrates the new development versus rework; note how iteration 4
has a predominantly high level of rework. Development managers (Scrum masters)
can use this information to assess the reasons why there is a rework peak: poor
requirements, complex code, or a change in business requirements, for example.
Again, there are suf¬cient data within the SCM tool to assess the velocity
(progress) of each iteration and the overall project. With the correct management of
user story metadata, combined with SCM work¬‚ow, it is possible to determine these
Taking this a step further, we can use these data to visualize trends and make
predictions; consider the graph in Figure 14.7.
The dashed line in the graph plots the velocity of each iteration, whereas the gray
solid line provides the velocity of the overall project. By overlaying trend lines we
can also make reasonable assumptions for the delivery of the current user stories “
¬‚uctuations in the iteration velocities may be attributable to changing, or additional,
Agile Testing: How to Succeed in an Extreme Testing Environment

14.5 Graphical display of user story attributes.


Iteration 1 Iteration 2 Iteration 3 Iteration 4 Iteration 5
Key: New Development % Rework %

14.6 Graphical display of rework ratio.
The Power of Continuous Integration Builds and Agile Development

Iteration 1 Iteration 2 Iteration 3 Iteration 4 Iteration 5
Key: Project Velocity % Iteration Velocity %
Linear Proj. Vel. % Linear Iter. Vel. %

14.7 Graph of iteration and project velocity.

14.6 Lessons Learned
In my experience, there is not one method that can solve all development problems
within any organization; however, there are compelling reasons to utilize various
techniques to achieve better quality of code more quickly.
The main objectives of the “best-of-breed” agile development process were to

gain control of application development,
prove compliance and governance,
provide ¬‚exibility for the developers and other SCM users,
deliver increased productivity,
ensure increased quality of code (leading to a decrease in defect costs),
be responsive to business needs, and
reduce time to market.

A signi¬cant achievement for our customers was the ability to leverage their
technology to support the “best-of-breed” agile development. Certain methods almost
dictate the absence of any SCM tool to support the agile processes; this is just not an
option with large-scale, N-tier applications.
Agile Testing: How to Succeed in an Extreme Testing Environment

Effective enterprise agile development was achieved for our customers by suc-
cessfully implementing and addressing

life cycle and process design,
¬‚exible SCM technology,
continuous integration builds,
automated unit testing,
code coverage analysis (unit test Panopticode),
code complexity analysis (Panopticode), and
management information and metrics.
15 The Payoffs and Perils of Offshored Agile Projects
Peter Kingston, Contract Test Manager

This case study describes my experiences of working on a project where we combined two
potentially highly bene¬cial, but apparently incompatible, approaches to developing software:
offshoring and agile methods.
Although ultimately successful, with the delivery and use of the system developed using
this combined approach, there were a number challenges to overcome (such as the issues of
not having a co-located team, of culture, of language and its interpretation, and of reliable
reporting and estimation).
In addition to describing the solutions to the challenges of agile offshore projects, I also
review the positive aspects of the project (such as reductions in effort, cost, and timescales;
extended development and testing time; and establishing a process for future agile offshore

15.1 Introduction
My name is Peter Kingston and at the time of this case study I was working on
a contract as a test manager for a ¬nancial products company, The Remortgage
Corporation (TRC).
My career in IT spans more than twenty-¬ve years, beginning as a software
developer before moving into systems analysis and then management. For most of
the last decade, I have been a professional software test consultant and have a wide
variety of experience of the software testing industry. I have gained much expertise
from working as a test analyst, test manager, and test consultant in a variety of
industries including ¬nance, retail, government, and petro-chemical. In addition, I
am an accredited provider of the ISTQB Foundation Course in Software Testing [42].
TRC operates in a highly competitive market place; there are many companies
producing similar ¬nancial products for a relatively small number of potential cus-
tomers. As a result, organizations like our own have to ensure our development
process is as lean and mean as possible. Our customers demand products with

Agile Testing: How to Succeed in an Extreme Testing Environment

ever-increasing functionality and usability, developed in shorter timescales and with
lower cost, and with higher quality!
As a result, we need to constantly review our development approaches and to
look for opportunities for driving down costs and increasing the quality of our
products. During our review of development approaches, we identi¬ed two particular
approaches that seemed to offer the sort of bene¬ts we were looking for:

Agile development “ With its focus on effective and ef¬cient development and
testing, agile appeared to provide a means of reducing development costs, ensur-
ing that we met our delivery commitments and (done properly) improved quality
(both meeting requirements and reducing defects) through early and frequent
Offshoring “ With its focus on distributed development using lower-cost re-
sources, offshoring also provided the opportunity to “extend the length of the
development day” by, for example, having practitioners in different time zones
picking up the development effort at different times.

The purpose of this case study is to share a number of interesting experiences
gained in setting up and running an agile offshore project involving a U.K. and
Indian development and testing team, but with an added complication of the product
marketing team being located in the United States.
The system developed through our agile offshore project has been successfully
delivered and is being used by a number of our customers.

15.2 Overview of the Development and Testing Challenge
In 2007, TRC began a project to migrate and enhance a large existing product (TRC-
Legacy) to a new platform (and new name “ TRC-Heritage). In its existing form,
TRC-Legacy had the following characteristics:

The product focused on the remortgaging market and had been purchased by ten
U.K. customers, who were still using the tool to support their everyday business.
Legacy had originally been implemented using Power Builder and employed an
underlying Oracle back end.
The product had been under development and enhancement for some ten years
and during its initial development had been implemented by a team of some
sixteen analysts and programmers, plus two dedicated quality assurance repre-
In its ¬nal incarnation (Legacy v6.0), the product consisted of some 450 distinct

In terms of the replacement TRC-Heritage product:

The new system was to be developed using Java on the Eclipse platform [57, 58],
plus some third-party GUI control products.
The Payoffs and Perils of Offshored Agile Projects

The migration project had very challenging timescales (i.e., our marketing team
had obtained market intelligence that indicated that a competitor product would
be launched within three months) and very challenging budgetary constraints.
We were told we had ten weeks to migrate, enhance, and deliver the replacement
As a result it was decided to employ an agile approach to ensure that the
timescales would be met and to reduce the cost of development and testing.
An agile approach was also anticipated to help us be ¬‚exible in terms of likely
changes to requirements and to ensure product quality was maintained by allow-
ing testing to begin as early as possible in the project and to continue throughout
the development.
To reduce the cost of the migration further, it was also agreed that much of the
development work would be offshored to a company in India, with marketing
responsibilities given to a team in the United States.

15.3 De¬nition of an Agile Offshore Approach
The details of our agile offshore approach were as follows:

The U.K. team included one project manager, two developers, one test man-
ager, and one test analyst/tester.
The Indian team included one team leader and ¬ve developers.
The U.S. team included one marketing manager.
Agile requirements
Use cases “ Where new requirements were needed (this was, after all, mostly a
migration-style project), they were elicited using a use case“based approach.
The majority of new requirements were obtained from our U.S. marketing
manager and involved some intensive transatlantic teleconferences (where,
oddly, we still suffered from language issues “ more on this later).
Requirements documentation “ Requirements were documented using a
commercial product [59]. This product included a web-based interface that
allowed our Indian and U.S. colleagues to be able to review, comment on, and
make use of the requirements. In practice, this was a very successful aspect
of the project, and as long as we phrased the requirements in a concise, clear,
and unambiguous manner,1 the results (i.e., the code implemented against
the requirements) were good.
Iterative approach
Two-week iteration period “ We employed a two-week iteration period. We
had previously seen projects where iterations based around a week-long period

1 You might justi¬ably argue that all requirements should be phrased in a concise, clear, and unambiguous
manner in any project; however, the differences in culture, language, and idiom on this project meant
we needed to be ultracareful in terms of how requirements were documented.
Agile Testing: How to Succeed in an Extreme Testing Environment

had been successful (more on this later) but the work items were too large on
this project; hence, our use of a two-week period. (Interestingly, we believe a
week-long iteration might have been possible had we not had the additional
communication overhead of offshoring the project.)
Time boxing “ The two-week iterations were time-boxed [17]. Where rapidly
approaching time-box deadlines meant that not all the requirements for that
iteration could be implemented, the decisions on what requirements would
be left out were based on priority values assigned to the requirements in the
requirements management tool. (Risk and priority were in effect attributes
associated with any particular requirement.) It was also possible to include
information describing relationships between requirements in the tool, such
as dependencies; this allowed the U.K. project manager to assess the impact
of proposals from the developers about omitting speci¬c requirements.
Agile meetings
General “ Wherever possible, all meetings would include the whole team (typ-
ically Indian and U.K. staff, with occasional attendance by the U.S.-based mar-
keting manager). Because of the communications challenges of offshoring,
meetings were considered to be an essential aspect of making the project suc-
cessful, and great importance was placed on holding and attending them. To
avoid too many meetings and not enough work, all meetings were as brief and
focused as possible (following the principles of the agile project management
method Scrum [23]).
Initial project startup and planning meeting “ This meeting was convened on
the Friday before the project was to begin to kick off the project, to introduce
the staff to each other, con¬rm roles and responsibilities, review and agree
on tasks and timescales, agree on our offshore working protocol (such as
the when and how of communications), and to plan and prepare for the ¬rst
iteration (to begin on the following Monday).
Two-weekly Friday iteration startup meetings “ These meetings were used to
review the results of the previous iteration, to document any lessons learned,
and to plan and prepare for the next iteration (which would begin on the fol-
lowing Monday morning). The meeting would involve a review of the require-
ments for the next iteration, their priority and any risks and/or dependencies,
and agreement on the tasks and timescales. The test plan for the iteration
would also be discussed and agreed.
Weekly conference call progress meetings “ Held midweek on the Wednesday,
these brief meetings were really just to con¬rm that all members of the team
were on track and that there were no problems, risks, or dependencies that
might cause the team to fail to meet its deadlines.
Local daily stand-up meetings “ We also ran daily ¬fteen-minute stand-up
meetings2 ¬rst thing each day with the U.K. team for the purposes of reviewing

2 In effect, Scrums [23].
The Payoffs and Perils of Offshored Agile Projects

what the team had achieved the previous day, what it planned to do today,
and what issues, risks, or dependencies might disrupt its plans. We asked the
Indian team to follow this approach as well.
Project closedown meeting “ At the end of the project we scheduled a large
videoconference meeting that included all the stakeholders from the project
(which was planned to coincide with the U.K. presence of our marketing
manager) to formally close the project, review the deliverables, ensure all
artifacts were ¬led, and to review and document all of the lessons learned
during the project (many of which were rolled up from lessons documented
in previous meetings). This was a very valuable event both from a process
improvement and project administration perspective, but also from a human
perspective. By then, our resident offshored resource had returned to India,
and it gave all the team members on both continents the chance to thank
each other and to say goodbye “ quite an emotional moment, as it turned out.
Agile development approach
General “ Since we were developing using Java and Eclipse, we largely followed
the agile Eclipse Way development process [60].
Pair programming “ We employed an agile development approach using pair
programming (as per Extreme Programming [20]).
Continuous integration “ The developers used a continuous integration (and
test) approach. This would continue daily at a local level, with a weekly (on
Thursday close of business) email consolidated build and test in the United
Kingdom. In practice, this did cause some quality issues (see Lessons Learned).
Test-driven development “ Developers were expected to review the require-
ments and design their tests before coding began.
Automated unit testing “ Developer testing was automated using the popular
JUnit tool [50].
Agile software quality management
Test management and planning were based around the iteration plan, with
functional tests planned at the end of each iteration (plus rerun regression
tests from previous iterations “ see later bullet point). Unit testing (and
evidence of unit testing) was provided by the developers as they coded. A
key goal was for testing to begin as early as possible and continue frequently
throughout the project.
Test coverage was achieved by ensuring that each requirement had one or
more test cases associated with it and that each test case had one or more test
scripts associated with it. This was a very straightforward process using the
requirements management tool selected for the project [61], and it was pos-
sible to produce extremely useful test design, implementation, and coverage
reports as and when needed.
Unit testing was very successfully achieved using the JUnit tool.
Acceptance testing “ Use cases were also employed by the test analyst to
generate acceptance testing scenarios, which was found to be a very natural
Agile Testing: How to Succeed in an Extreme Testing Environment

and straightforward solution to producing realistic and effective acceptance
Regression testing was achieved using a Java-based capture“replay tool [62],
allowing us to automate the acceptance tests in the form of Java scripts, which
we were then able to replay against new builds and releases of the product.
Pair testing was a technique we developed to be a software quality equiv-
alent of the pair programming techniques the developers were using. Our
test manager and test analyst worked together to design and implement the
tests needed to verify functionality, which were subsequently recorded by the
capture“replay tool (to be reused against later builds and releases).
Test automation “ As described earlier, JUnit and a capture“replay tool were
used to automate the testing process wherever appropriate. In the case of
the capture“replay tool, we also used it to set up, initialize, and manage the
complete regression testing activity.

15.4 Results of an Agile Approach
The replacement TRC-Heritage product has been successfully implemented and is
now in use with a number of our existing customers; it has also been sold to several
new customers.
Bottom line, the migration has resulted in a replacement product that has been
updated to match current market needs, is in service with existing and new cus-
tomers, and has earned (and continues to earn) signi¬cant revenue for our company.
In terms of the overall project results:

The product matches the customer requirements well. The new product require-
ments were largely obtained through our U.S. marketing manager and have
enabled TRC to make sales of the product to new customers, as well as providing
an upgrade to our existing customer base. The team (including the nontechnical
U.S. marketing manager) felt that the use case approach had been a success,
allowing requirements to be captured, reviewed, and easily understood by all the
Overall product quality has been good. The early and frequent testing that our
iterative approach supported has meant that a great many defects were detected
and corrected way before release of the product. Since release, there have been no
critical defects reported, and (at the time of writing this case study) two serious
defects (both involving database access “ which were quickly ¬xed) and just eight
cosmetic issues (which have been ¬xed in the next release).
Overall project costs have been reduced. A brief analysis of our project met-
rics shows that the overall cost of the project was 30% lower than running an
equivalent co-located U.K.-based agile project. Although this information must
be balanced by the fact that the project overran by 20% in terms of its duration,
this has nevertheless been a very well-received result from the perspective of our
senior management.
The Payoffs and Perils of Offshored Agile Projects

Reporting and estimating issues “ From a project management perspective,
there were de¬nite issues with reporting and estimating. Because of time zone
differences, the dif¬culty with convening ad hoc offshored meetings, lack of
formal cross-geo reporting tools, and the challenge of not having face-to-face
communications when the weekly teleconferences were held, it was very dif¬-
cult to have a clear picture of progress. Our agile approach highlighted these
issues much more acutely than a traditional project approach would have
Two-week overrun “ Over the projected ten-week project duration, we overran by
two weeks. This 20% slippage was largely attributed to the overhead associated
with “round-trip communications” (i.e., where a request from the U.K. team,
email or voicemail, was made outside of the Indian working day, picked up by
the Indian team the next day, worked on, and then reported back to the U.K.
team the next day). As the project proceeded, we did try to ¬nd solutions to this
phenomenon, which I cover in the Lessons Learned section.

15.5 Lessons Learned
There were a number of lessons learned from our agile offshored project:

Fundamentally, offshoring is a trade-off between speed and cost of development.
This may not be such an obvious issue on a project employing traditional devel-
opment and testing approaches, but the use of an agile approach did highlight
the problems. For example, our feeling is that we could have run to shorter one-
week iterations (with which we have had very positive experiences of in previous
projects) if we had not had the overhead of long-distance communications to
cope with. Instead, we had to adopt a two-week-iteration period, which meant
that the team did not get into quite such a slick rhythm and were of necessity
slightly less productive.
Reduction in productivity “ Now, the key issue is “ was the reduction in produc-
tivity offset by the reduction in the unit cost of the offshored resources? In this
project, our feeling (justi¬ed by metrics) was yes “ in fact, overall we believe the
project was some 30% cheaper than if we had based the work solely in the United
Kingdom (and employed a co-located team).
Software quality lessons “ There were some very valuable software quality lessons
to come out of this project:
Pair testing “ We de¬nitely recommend the use of this approach. Metrics
showed that when two testers worked together to design and implement
tests, they
showed a greater understanding of the underlying requirements,
identi¬ed more and better data points (to test the correct operation of the
code, as well as in terms of error and exception values),
were more productive (producing some 15% more tests per unit time than
two testers working in isolation), and
Agile Testing: How to Succeed in an Extreme Testing Environment

smiled more “ this social activity made for happier and more motivated
Test-driven development (TDD) “ The feedback from the developers is that
TDD is a very valuable approach. The need to understand the requirements in
suf¬cient detail to be able to design a test to show that the code meets the re-
quirement before implementing the code means that the developers have
gained a thorough understanding of the intention of the requirement before
they come to code it. Some developers expressed the view that TDD would slow
them down and reduce their productivity; in practice, there has been no appre-
ciable reduction in developer productivity, but there has been a measurable
improvement in the quality of the code (in terms of matching requirements
more closely as well as in delivery of code with fewer defects in it).
Test automation “ The use of the JUnit testing tool has been an unquali¬ed
success and we would de¬nitely use this again on similar projects. The results
of the use of the capture“replay tool have been mixed:
There is a de¬nite bene¬t in software quality from being able to rerun tests
recorded in earlier iterations (we make sure that old defects don™t reappear,
for example).
Also, the more often you replay a given test script, the more return you gain
on your investment. This is important, because these tools are a signi¬cant
project investment (both in the cost of the tool plus staff training/learning
time). We estimate that we needed to reuse a script four or more times to
realize a gain in productivity versus an equivalent manual test. So for a
short one-off project you need to challenge the use of these tools. Where
you are likely to gain good reuse from the tools, there are economic gains
to be made.
A good lesson to learn is that, to use these tools effectively, you must ensure
you either have testers who have used the tools in the past or ensure that
you budget time and money for training.
As a ¬nal thought, if your code base or application functionality changes
too frequently between iterations, you may also incur additional cost from
maintaining the test scripts. Although this sort of project chaos should
clearly be discouraged by the project manager, script maintenance costs
must be balanced against your return-on-investment metrics.
Requirements coverage “ If you do not know what the software you are developing
is supposed to do, how can you possibly test it correctly? For any nontrivial project,
the use of a requirements management tool is essential. The ability to elicit,
document, and maintain the requirements for the system enables developers to
know what they are developing, as well as allowing the testers to know what it
is they are testing. In particular, the use of a requirements tool should allow
the test manager to be able to make some informed estimates about testing
timescales and resources and allow the test analyst to demonstrate complete test
coverage of the requirements. A further bonus of using a requirements tool is
The Payoffs and Perils of Offshored Agile Projects

the ability for remote stakeholders to use the product™s web-based interface to
inspect, comment on, and request changes to the requirements.
Testers beware of time boxing “ Time boxing is clearly a valuable tool in the
project manager™s arsenal to help keep development on track and to achieve
delivery deadlines. However, during a couple of our iterations, the time allo-
cated in the plan for testing was eaten into by overrunning development. If you
plan to use a time-boxed approach, you must ensure that the time planned
for testing is sacrosanct. We considered, but in the end didn™t implement,
making the functional testing activity a time box of its own that followed
on immediately from the associated development time box. This is de¬nitely
something I would investigate in future projects.
Communications issues “ We learned some very valuable lessons about commu-
nications, and how good and bad communications can affect the choice of using
an agile approach:
Lack of face-to-face communication matters. It is dif¬cult to build a rapport
with someone on the other side of the world just using a telephone. As a
result you lose some very valuable face-to-face cues, such as the con¬dence
with which an estimate to complete is delivered during a telephone progress
meeting. A true agile approach would of course have mandated the co-location
of the stakeholders. One method that I have used that worked particularly
well was to have one of the offshore team work onshore as a liaison (or even
as translator). They are able to deal with people at both ends of the telephone
and are often well placed to explain what the other team really means.
Find strategies to build a trusted relationship with offshored colleagues. Find
the time to engage in small talk, ¬nd out about their interests and hobbies,
discuss family stuff, and even just talk about the weather. All parties need to
¬nd the means to understand and trust each other. Having an offshore team
member working onshore is also a good way of building a trusting relationship
with the remote group. However, it is important to make this person feel like a
full member of the team and to include them in all activities, bearing in mind
there are cultural differences. For example, one Indian colleague revealed,
but only through general conversation, that three days earlier he had become
a father for the ¬rst time but didn™t think it polite to mention the fact. We
were able to discuss cultural mores while “wetting the baby™s head” before
presenting him with some cuddly toys and packing him off home to meet his
Be polite, but ¬rmly challenge all estimates. It is all too easy for someone on
the end of a telephone (and themselves under project pressures) to provide
overly optimistic estimates of their progress. This is particularly true when the
communications are infrequent “ a team member may actually give an esti-
mate that is too optimistic, believing that by the next time you were in commu-
nication, they could have brought a late task back on track. The best solution
(and one we would look to use in similar future projects) would be to use an
Agile Testing: How to Succeed in an Extreme Testing Environment

agile distributed development tool (such as in [63]) where individual worker™s
estimates and actuals are automatically collected as they perform the work
and which are instantly available to the project manager to review.
Be aware of language/meaning issues. For example, one of the Indian offshore
practitioners, when asked how a particular delivery had gone, used to say “ship
happens”; over the telephone and with the unfamiliar accent, this message was
interpreted in a “negative manner” as something bad had happened, rather
than in the positive manner it was meant to have been interpreted (i.e., that
the software had in fact shipped).
Funnily enough, we had similar problems with our U.S. colleague.3 In one
transatlantic teleconference, our project manager wanted to highlight what
he thought was a key requirement and said something like, “let™s place this
issue on the table” (meaning “let™s focus on it”). At the next weekly meeting,
our project manager was surprised to ¬nd that no progress had been made
on the requirement; it turns out that in the United States, to put something
“on the table” means to put it aside and to focus on other issues!
My ¬nal, and perhaps most interesting example, however, is of one offshore
practitioner who was working onshore in the United Kingdom. He was per-
fectly polite when arranging meetings but would always ask to meet people
“on” their convenience.
Find better means of more frequent communication.
Make use of instant messaging technologies to keep in touch. Such tech-
nologies can be very useful to ask simple spontaneous questions or to seek
clari¬cation (but beware of time zone differences).
Consider the use of cheap Internet telephony. Although this may not be
too useful for offshore sites where there is no working time-zone overlap,
it can be a powerful adjunct to instant messaging, allowing workers to have
cheap and easily available voice calls.
Consider the role and use of videoconferencing. Although not so much in
vogue as it was in the 1990s and early 2000s, the face-to-face aspects provided
by a video link for holding offshore meetings might well provide a better
means of obtaining subtle cues from body language and expressions that
straightforward telephone calls can™t.
Consider the use of Web 2.0 social networking technologies. For example,
consider having on-line virtual meetings in Second Life [64]. This may well
be an approach that improves team cohesion and encourages effective com-
Ensure you only select tools that are distributed. For example, the require-
ments management tool we selected included a web-based interface that per-
mitted our offshore colleagues (and, in particular, the U.S.-based marketing

3 Winston Churchill was spot on when he said that the United Kingdom and the United States are two
nations divided by a common language!
The Payoffs and Perils of Offshored Agile Projects

manager) to log on to the tool remotely, review the current requirements, and,
where necessary, make comments or request changes to the requirements.
Make more use of tools that support 24/7 working “ Having tools that automati-
cally replicate the code base and project data between geographically distributed
sites would be a very valuable means of supporting the distributed working model
we were looking to achieve. We discovered a very powerful approach that would
have automatically replicated our project workspace between sites just after the
project completed and we would de¬nitely investigate its use in future projects
of this nature [63].
Make use of process enactment tools. In running an agile approach to developing
the new software, it would be of value to have all of the practitioners follow
the same process. Taken a step further, it would be nice if all the practitioners
were constrained to follow the same process. There are products available that
provide powerful facilities to implement and enact (enforce) a particular style
of project management (such as agile) and which will also produce accurate
real-time reports and dashboard displays of project progress [65]. Employing a
distributed tool of this nature would be a powerful solution to drive an agile
offshore project more effectively.
Try to harmonize working times across time zones. As described earlier, over the
ten-week project duration, we overran by two weeks. This slippage was largely
caused by the problems of “round-trip communications” (i.e., where a request
from the U.K. team “ email or voicemail “ was made outside of the Indian working
day, picked up by the Indian team the next day, worked on, and then reported
back to the U.K. team the next day). As the project proceeded, we did try to ¬nd
solutions to this phenomenon “ such as making sure we were in touch with
the Indian team during their working period, and changing our respective start
and stop times to improve overlap (i.e., asking one or more of the Indian team
to start work an hour later than usual and ¬nish work an hour later and doing
the converse with the U.K. team). This certainly helped and meant that, toward
the end of the project, round-trip communication became less and less of an

15.6 Summary
The bottom line is that the project to migrate and update the TRC-Legacy product
has been a success. Despite a two-week overrun, the agile offshored project used to
deliver the new TRC-Heritage product was completed for an estimated 70% of the
cost of running an equivalent traditional U.K.-based project.
When you add to this the fact that the product does what it is supposed to do (and
suf¬ciently well for the existing TRC-Legacy customers to be very pleased to use the
new TRC-Heritage product, as well as for TRC to be able to attract new customers to
buy the product) and was delivered with surprisingly few defects, the exercise would
seem to be an unquali¬ed success.
Agile Testing: How to Succeed in an Extreme Testing Environment

In terms of speci¬c quality management aspects of the project, we would give a big
thumbs-up to our pair testing solution, as well as to TDD. The use of a requirements
management tool, and in particular one that integrates with your test management
and test design approach, was also of great value.
In terms of test automation, we have mixed results. While JUnit was unanimously
agreed to have been a very valuable tool for the developer testing, we have some
reservations about the capture“replay tools that can be used for functional and
regression testing. If you are only going to use the tool once or twice, then you will
almost certainly not recoup your investment. We found that reusing the test scripts
four or more times saved time and cost compared with taking an equivalent manual
testing approach. Each time you continue to reuse the test scripts after this point, it
is all up-side.
Finally, if you are in a project that uses time-boxing techniques, insist on main-
taining the time planned for testing in the time box “ even if the developers are
overrunning. If this is a frequent problem, consider breaking out the functional and
regression tests as a separate and distinct time box of their own.
16 The Basic Rules of Quality and Management
Still Apply to Agile
Richard Warden, Director, Software Futures Ltd.

Agile is a label that covers a wide range of incremental development methods, languages
(e.g., UML (the Uni¬ed Modelling Language “ [33]), and management styles (e.g., Scrum
[23]). However, whatever ¬‚avor of agile you choose, from Extreme Programming [20] to
the use of ICONIX [66], certain rules still apply. This chapter reports some experiences from
projects that reinforce the basics.

16.1 Introduction
My name is Richard Warden and I wrote and tested my ¬rst computer program on
May 12, 1970. When I worked for the Swiss Exchange in Zurich as a test manager
they called me an “Old Rabbit” “ a complimentary term, I will add. That was eleven
years ago and I have since spent much of my time immersed in UML and agile-related
projects within ¬nancial services companies.
My career started in the Royal Air Force as an analyst, developer, and tester on
large mainframe defence support systems. I then moved to Racal Electronics working
¬rst on business systems and then interactive computer-aided design (CAD) systems
in positions of programming team leader, test manager, project manager, and head
of quality assurance. Following this I worked for the K3 Group as a product and
research manager. Since 1991 I have been an independent consultant.
The case histories relate mainly to trading systems and I will give a brief outline of
their design rather than a complex architecture picture. There is a trading platform
that is usually a client“server design where traders conduct business. The client is
used by traders to create trades and is supported by server functions to provide a
wide range of static data such as clients and contract types. Trades created from the
platform have to be processed by a number of downstream systems such as clear-
inghouses, reporting and regulatory organizations, and risk management systems.
Consequently, a routing or messaging system is employed to create the necessary
transactions to distribute to these systems. This router has to work with very high
reliability in real time and is a potentially dif¬cult part of a system. The trading

Agile Testing: How to Succeed in an Extreme Testing Environment

platform may vary in complexity according to the types of ¬nancial product being
traded. Some may be speci¬c to a particular instrument, such as the Swiss Exchange
or electronic REPO market, and others may cover a range of ¬nancial instruments,
such as Bloomberg™s widely used system [67].

16.2 Project A: One Cannot Live by Scrum Alone
This project was to develop a replacement trading system, which is now common
because many organizations are moving to second- and third-generation systems.
Scrum was chosen to manage the development work.
A fortnightly Scrum cycle was chosen with daily Scrum meetings. Analysts,
developers, and testers attended the Scrums and gave their reports based on the
following three questions: What did I do yesterday? What am I doing today? Do I face
any problems where I need help?
Analysts decomposed speci¬cations into stories and all could monitor progress
on the storyboard in the project of¬ce. Burndown charts were displayed so everyone
could see the velocity and projected completion dates of the different development
The ¬rst problem we found was that charts were unreliable “ speci¬cally projected
completion dates. They would remain static for some time and then slip suddenly.
Surely one purpose of Scrum is to identify slippage quickly? The problem appeared
to be in calculating velocity with accuracy. As an analogy, the onboard computer in
my car has to solve a similar problem to predict the range of the car. It knows how
much petrol is left in the tank and past fuel consumption. The question is, how do
you use past consumption in a predictor algorithm? Do you take the last 25 miles™
driving as representative, or 50 or 75? Likewise, do you take the last Scrum cycle or
use more? Furthermore, how do you normalize story counts so they are comparable
across cycles, as stories ranged in size by a factor of 10?
Another effect we observed was that of sucking the pipeline dry at the end of a
cycle. To maximize velocity, staff would be assigned to clearing as many stories as
possible. For example, analysts would suspend work to focus on QA sign-off. Although
the sprint velocity looked good, it meant delays had been created that were not visible
in a plan and could delay later cycles.
However, the larger problems were as follows. Trading systems by their nature
interface with many other systems. There was considerable work being done by these
systems™ owners to develop and test their interfaces with the routing system. They
had to plan and execute their work and ensure that it was ready in time for ¬nal
integration and user acceptance testing. Although analysts worked with external
organizations to de¬ne and agree the interface speci¬cations, there was no coherent
development and delivery plan. It was left to testers to poll interfacing systems
owners to determine availability. This was problematic as there was no project plan
and reporting mechanism that showed all these activities and their dependencies. In
short, it meant that slippage in external work often came as a surprise.
The Basic Rules of Quality and Management Still Apply to Agile

One conclusion I draw from this experience is that Scrum, although easy to
describe, is hard to implement without proper calibration to measure velocity accu-
rately. More importantly, any project that has external dependencies requires a plan
showing them and an associated reporting mechanism driven by the project of¬ce. In
this respect Scrum appears limited and the danger is that people do not understand
the limitations.

16.3 Project B: Risk does not Go Away if You Are Agile
In this project the aim was to replace the routing mechanism for a trading system.
The trading platform had already been developed and was not the main subject of
A risk-based approach to testing had been considered, but management decided
this was not necessary. The main grounds were that all parts of the system were
critical and therefore of equal risk. Also, there was considerable con¬dence in
the trading platform because it had been fully tested. There was a feeling that, because
agile inherently mitigates against risks, no further risk analysis was required. How
true is that?
I see agile mitigating risks in two areas. The ¬rst is in the business risk category,
namely that of minimizing the risk of developing the wrong system, a system that
does not support business operations. This is achieved by involving users directly in
the agile process to incrementally specify and sign off stories. The second is in the
project risk category. An agile approach is meant to reduce the risks of delivering
late by decomposing the project into stories that Scrum, or similar management
approaches, can manage well.
What agile does not appear to cover is product, or system, risk. I make this as an
observation and not as a criticism. Product risks are risks associated with failure of an
internal component of a system. They can be related very much to architecture and
design; for example, in a network system you avoid building single points of failure. A
product risk register usually drives two elements in a project. By identifying high-risk
components, developers can reduce the impact of their failure by redesign (e.g., by
building multiple routes through a network). The Internet is perhaps the ultimate
network design, having been proposed to keep operating in a situation of global
con¬‚ict. The second element is to reduce the probability of failure in the ¬rst place.
High reliability is achieved by focusing testing on high-risk components, and the
risk register is used to identify test coverage required and the testing techniques,
with metrics, that can provide them.
The lack of understanding of product risks regularly created problems that could
have been forestalled. For example, transaction routers are tricky and complex and
can have varying abilities to replay or rerun transactions in the case of failure. It was
discovered that under several circumstances trades could be lost and not rerun. The
impact on testing was severe. Financial products often require test cycles spanning
days or weeks. That is, testing on day one has to be successful for day two onward to
Agile Testing: How to Succeed in an Extreme Testing Environment

proceed. Now if some of the day one transactions are lost due to, say, a con¬guration
error and in a router, and cannot be rerun, day two tests onward cannot be executed.
If it is a router to a speci¬c external system, then that system may have to be removed
completely from the current cycle.
These risks could have been identi¬ed at the outset. However, without a risk
workshop this information remained hidden until the risks turned into failures
during test execution. Mitigating actions could have been put in place. Additional
tests prior to major test cycles could have been run to show the router was con¬gured
correctly. Additional breakpoints could have been created. This meant additional
cycles and test runs needed to be executed, consuming precious time and resources.
Another example of misunderstood risk was in the performance and capacity
planning of some key servers. The Information Technology Infrastructure Library
(ITIL) best practices [68] places much emphasis on capacity planning (i.e., can
your systems cope with the expected workloads, both normal and peak). This, in
turn, feeds into design, development, and testing. However, no capacity planning or
related risk assessment was done. Consequently, integration and acceptance testing
suffered because servers were unable to cope with even moderate test workloads, let
alone volume and stress tests. The situation was only resolved with a moratorium on
development and testing while new servers were procured, installed, and con¬gured “
more delays.

16.4 Project C: How to Be Agile and Operate at CMM Level 1
In the early days of this project the quality management system (QMS) included
reviews of requirements, speci¬cations, and other key documents with users and
developers. Because this ¬nancial system integrated with many others, the analysis
work was complex. General progress was slow and dif¬cult and the project moved
to an agile footing to create a more manageable project. During this transition,
basic quality disciplines were no longer enforced; it was left to individuals to
decide how they implemented the QMS. In project B, I described how agile should
address business risks through close involvement of users. What happened was a
slow degradation of quality in the analysis model as users were less involved. This
happened in two areas.
First, existing speci¬cations were amended without proper review. This made it
dif¬cult to examine a document and determine if it was current and trustworthy.
Second, new documents were created with minimal desk-checking. In some instances
there was no information on author and history, so they were particularly untrust-
worthy “ we found they contained serious errors. As a consequence, testing was
compromised, particularly test execution. Unreviewed documents had been turned
into code that did not do what the user wanted. This gets you into the protracted
cycle of errors created during early analysis not being found until late acceptance
The Basic Rules of Quality and Management Still Apply to Agile

This relaxation of quality disciplines showed itself in some areas of development
outside of the agile part of the project. Developers considered it reasonable to ¬x bugs
in code but with minimal testing, leaving it to systems and integration tests to ¬nd
them, even if it compromised those activities. Consequently, later stages of testing
found errors that could have been removed much earlier had the QMS remained in
Here are two examples. A router interface speci¬cation was written by an analyst,
not reviewed with users, and not even placed under control of the project repository.
It was given to developers who wrote the interface code and the analyst signed off
their work. When the software reached integration/user acceptance testing, it failed
immediately with a simple and obvious error that caused every transaction to fail.
The users were unimpressed. A cursory inspection found more fatal errors in the
The second example is where bugs were ¬xed in a router interface. The ¬x
was tested and declared OK, which it was. However, when the developers declared
the interface ready for acceptance testing they had not tested that the router was
con¬gured to the trading platform as opposed to their development environment.
The router failed to pick up all transactions during an acceptance test run and took
that external system out of testing for over a week. When asked about this, the
developers did not consider it part of their quality responsibility to test whether the
con¬guration was set correctly; this was for the testers to do.
In summary we had a project where individuals applied their own ideas of quality
control. Some ideas were good, many were not. This created a lot of uncertainty and
unpredictability. We were not in the repeatable environment of Capability Maturity
Model (CMM) level 2, let alone the con¬dent environment of knowing how to tailor
our processes to new demands, CMM level 3. It felt like CMM level 1; many avoidable
problems appeared through poor prevention and big gaps in the QMS, accompanied
with the recriminations and closing of ranks that destroy team spirit.

16.5 What if D = A + B + C?
Now imagine one project, D, where all three issues combine, since this was the
reality. We need to get the picture of how they interrelate.
The agile part felt like a project within a project. Despite the issues mentioned
earlier there was good visibility of what was happening. The development work
was generally successful within the sprints, as shown by low programming and
design error rates found in later stages of testing. JUnit was used with considerable
refactoring and code/unit testing. (As an aside, some of the development stories took
signi¬cantly longer than expected and we suspect this was because the refactoring
element was signi¬cantly underestimated.) Agile worked well from a veri¬cation
viewpoint. Developers were “building the software correctly.” This part felt under
Agile Testing: How to Succeed in an Extreme Testing Environment

In contrast the rest of the project felt less and less under control as time pro-
gressed. Errors and faults injected early into the project were surfacing during late
stages of test execution. User acceptance testing and integration-in-the-large test-
ing were combined. The idea was that as individual interfaces were released from a
Sprint cycle they were put into user/integration test immediately. The lack of user
involvement in the sprints led to simple yet high-impact defects getting through to
test execution. Test runs had to be repeated many times but they only gave a slow
increase in software reliability. In this respect the project did not work well from a
validation viewpoint. It was not good at “building the correct software.”
The critical issue was the lack of risk management. There was no risk register, let
alone a process for populating it and identifying risk mitigation actions. Without a
proper understanding of business, project, and product risks, how do you decide the
appropriate techniques to manage, plan, develop, and test a system? How do you apply
a QMS? How do you know if agile will work for you, and if so, which agile? Should
it be an ultralight software factory run by Scrum or a heavy-duty project following
ICONIX? Or should it be a combination of agile and traditional techniques?
What was not well understood was how Scrum would mitigate risks in this project,
and areas where it would not. It reinforces the received wisdom that without a risk
analysis your project will adopt a fatalistic approach. The North American saying is,
“This project is like a log ¬‚oating down a river with a thousand ants on it, and each
of them thinks they™re steering it.”
What could have been the solution? In another institution the internal and
external development parts of a new trading system were treated as two separate
projects. Internal development was run as an incremental project. Instead of stories
we had many use cases, but with increments measured in weeks and daily meetings
it was not that different from Scrum. We developed burndown charts for testing,
showing the velocity at which we could test and close bug ¬xes. An experienced
bond trader was assigned to the team to act as a user, keeping the user view close to
the developers and to act as an oracle for the testers. The external integration was
managed separately with different business groups providing the testers to specify
and run the tests once the trading platform had been delivered. The system went live
on its planned deadline. But then, this project had a risk register, as did individual
groups such as the testers.

16.6 Conclusions
Some conclusions were drawn in the individual sections but here I will cover the
bigger picture. This case study provides an example of agile/Scrum misapplied. Agile
is a solution to a problem; unless you understand the problem, your problem, you
will make mistakes.
In this example the project needed revitalizing and agile was chosen to bring focus
and momentum, which it did. However, in making the transition, no risk analysis
The Basic Rules of Quality and Management Still Apply to Agile

was performed to understand what elements agile would not address. As we have
seen, it did not address the larger integration issues, areas of product complexity,
and the application of quality controls.
Agile/Scrum provided a good software production line but, without quality con-
trol on the speci¬cations (reviews) feeding the “goods inwards” side, the quality of
the “goods outwards” (software) would be in question from a validation perspective.
One obvious problem was that end users did not perform the sprint QA/sign-off,
this being done by the analysts who wrote the speci¬cations. To my mind this was a
fundamental error, because a major aim of agile is to involve users as soon as possible
in the validation testing.
From an integration and user test perspective the project was highly reactive;
there was little chance of following a test plan with any degree of con¬dence. Too
many (unidenti¬ed) risks turned into reality, and too many principles had been
Many problems in information and communication technology are wicked prob-
lems; that is, they cannot be solved by some linear problem-solving process. I would
suggest that making a transition to agile from a traditional project background is
one of them.
As food for further thought I leave you the four main characteristics of a wicked
First, a wicked problem contains many factors that all need to be considered.
Overlook some and stakeholders will lose out, and you will end up with a solution
that is not sustainable without much cost. In short, do not oversimplify the
Second, there is no standard solution to a wicked problem. Your situation is
unique and you need to tailor the solution to it. Organizations often have their
own culture of project and quality management that has grown over a period of
time. If you are shifting to agile you need a solution that ¬ts with your culture.
Third, there is no perfect solution to a wicked problem; there will be trade-offs.
The question is, how do you know how to compromise? Clearly you need to have
addressed the ¬rst two points, understanding the factors and your take on the
problems, to identify good compromises.
Fourth, the problem changes as your understanding increases. The more you
get into solving the problem the more you realize that what you thought were
problems are only symptoms. This requires an iterative approach; that is, you
must reassess the application of agile frequently and be prepared to amend your
17 Test-Infecting a Development Team
David Evans, Director of Methodology at SQS

A software quality services company had a good idea for a software product; they assembled
a team of developers to build it and instructed them to run it as an agile project. Competent
and experienced, but new to agile and skeptical of what seemed like radical new processes,
the developers learned from ¬rst-hand experience that tests are highly valuable information
sources, and that thinking about development from a test-¬rst perspective leads to highly
ef¬cient, bug-free software design. They became “test-infected” “ a condition where devel-
opers are so enthused about test-driven development that they refuse to return to their “bad
old ways” of writing software without tests.

17.1 Introduction
My name is David Evans. I am the Director of Methodology at SQS, a pure-play
testing and software quality consultancy. I am also the chief Agile Testing Evangelist
in the company.
I have been in the software development and testing business for twenty years,
¬rst as a mainframe developer and tester, later moving to object-oriented C++
windows development and then .NET web development. In 2000 I joined Cresta, a
testing consultancy, where I specialized in test automation, test methodology, and
software quality practices. Cresta was later acquired by SQS.
In 2002 while carrying out a test strategy assignment for a global pharmaceutical
company, I came across Extreme Programming (XP), which was being trialled by
some of their development teams, and had to factor this into their enterprise-wide
test strategy. I was fascinated by the agile process and impressed by the levels of
quality and customer satisfaction these teams achieved.
When I was given responsibility for creating a software product based on testing
metrics from Cresta™s testing methodology, I knew the project needed to be agile.
We were able to enlist a team of highly talented .NET developers, but none of them
had worked in an agile project before.

Test-Infecting a Development Team

17.2 Overview of the Testing Challenge
How to get developers to willingly spend roughly half their time testing, and get the customer
to see the economic bene¬t of this!

The Project
Cresta was an innovative company of testers. Their core business was testing and test
consulting, not software development, although many of Cresta™s consultants were
technical test automation specialists and some were ex-developers. The decision
makers in the company were well aware of the need to develop tools to support
the business and to improve the quality of project delivery. Rather than outsource
this work, it made sense to do it internally, as it made use of the technical skills
in the company and provided an excellent internal “laboratory” for research and
innovation in software quality techniques. In 2004 they initiated a project to build a
quality metrics dashboard, which would be called TestStrategist.

Developers and Customer
I was to be the lead software designer, but I was also doing programming, so it was
important to have a customer (in the XP sense) from the business, someone who
was ultimately going to be responsible for helping teams use the product and to
sell it to clients. This was to be Edgar, a director and founding member of Cresta.
“Pure” XP recommends a full-time customer in the team, but because the system
we were building was based on an existing process that I had designed, a lot of the
business knowledge was already in the team, and we relied on our customer mostly
to take decisions regarding priority, so we could live with the fact that Edgar was not
full-time on the team.
We also needed a good technical guy who could handle our hardware and network
issues, as well as cut code “ Scott was someone I knew and trusted and, although
he was new to the agile way, I felt con¬dent he would pick it up. As well as being
a ¬ne developer, he was a naturally organized team member who did the impor-
tant job of setting up our team collaboration site and tooling to enable remote
For our main development resource we took on Monique, an external developer
whose coding skill was known to us from previous dealings. When we interviewed
Monique prior to joining the project we described the process we were intending to
use and mentioned XP practices such as close customer collaboration, test-driven
development, collective code ownership, and pair programming. She was clearly
skeptical and showed classic developer defensiveness, especially at the idea of making
“her” code so open to the scrutiny of others. She went home and told her husband of
the extremely strange development practices that had been suggested to her, adding
that we acted like we were asking her to join some sort of cult. Fortunately for us,
she needed the work, and she agreed to join the team.
Agile Testing: How to Succeed in an Extreme Testing Environment

Initially, it was just the three of us, all developers, as full-time members of the team.
We did not assign any full-time testers since, as a testing company, we knew there
would be an ample supply of those as we needed them. The strategy was to rotate
test consultants through the team on a “tour of duty” to give them experience of
testing in an agile environment and avoid losing a “billable” resource for the whole
duration of the project. Before we could do that we needed to bed down our own
process and that meant taking responsibility for all testing ourselves. This, after all,
was not in any way inconsistent with accepted XP theory, which says that developers
and customer are jointly responsible for testing. (The reality of most agile projects of
nontrivial size is that dedicated testers add great value, since neither the developers
nor the customer have suf¬cient bandwidth or experience to include all testing.)
Good testing intentions are not enough, and our challenge was to ensure we could
“drink our own champagne” and show that we were a quality-oriented development
team, as dedicated to testing as we were to development. We were well aware of the
risk to the company™s reputation of releasing a product that was not of the highest
standard of quality. We also had to get to grips with the new concept of test-driven
development (TDD). This is a challenge for developers who might not think that
testing “ especially at the levels required by TDD “ is a productive use of their time.
It is also a challenge for management, who may take the naive view that development
time on all tasks is going to double.
We were all climbing a new hill and starting from the bottom. What we had on
our side is that we worked well as a team: there was mutual respect from the outset
and an eagerness to learn from each other™s experience.

17.3 De¬nition of an Agile Testing Process plus Test Automation
Unit testing with NUnit, acceptance testing with FIT & Fitnesse, continuous integration, and
exploratory manual testing.

Starting with Simplicity
As the team was new to XP (and to an extent still skeptical about it) we did not
attempt to take on all aspects of the method at once. Certain things made sense from
a business perspective and we concentrated on those ¬rst:

We knew that, although we had a reasonably good idea of the fundamentals of
what the dashboard application needed to do, the detailed requirements would
evolve over time.
We needed to demonstrate working software to our customer as quickly as pos-
We needed to do the simplest thing that could possibly work. Two-week iterations,
with a focus on running tested software at the end of every iteration, ensured we
followed the principle of simple design.
Test-Infecting a Development Team

The ¬rst lesson we learned as a team was that every requirement can be cut down
to something simpler, until you have distilled the essential minimal “core” accom-
panied by “layers” that improve one or more aspects of that core functionality. This
realization gives you a lot of con¬dence to meet deadlines, which is a critical success
factor for iterative development.

Test-Driven Development: Unit Testing
Perhaps the most important practice that we embraced from the beginning was TDD.
Unit testing is something that developers will avoid if they perceive it to slow them
down, or if it forces them to switch context from their programming routine. Fortu-
nately, the XP community has created xUnit, a remarkable family of open-source unit
testing tools, each implemented in languages that developers use for programming.
As we were coding in C#.NET, we used NUnit. It is a very elegant tool that makes
unit test creation, management, and execution simple and highly effective.
Armed with the right unit testing tool, we adopted the test-driven approach from
the outset. This proved to be an important success factor for us. Writing a unit test
in order to “shape” the next small element of code can seem counterintuitive at ¬rst,
but for us it quickly became a very natural and satisfying way to work. This was
reinforced by the XP practices of collective code ownership and pair programming,
so that, although the practice was new to us, all members of the team worked
consistently and were able to support each other. The very high code coverage this
gave us, combined with the very fast feedback regarding unexpected side effects and
regression errors, became an addictive element of our process.

Enter the Build Captain
Once we had unit-level TDD ingrained in our development process, and a very large
suite of tests, we began to observe and re¬‚ect upon our tests and the maturity of
our technique. We read articles about good unit testing practices and patterns. We
considered how we could get our slower tests to run faster. We learned about mock
objects and specialist frameworks for testing ASP pages.
This period coincided with a new member joining our team. Owen was a test
consultant who was looking for a change from what he was doing. We knew he was
bright and capable, and certainly didn™t want to risk losing him from the company,
so we were glad to welcome him to the team. He was appointed originally as our
¬rst dedicated tester, although it was clear he had the technical ability to be doing
development. I was interested in having Owen help us lift our testing process to a
new level “ not by being a manual tester to supplement our automated tests, but by
implementing processes that helped us as a team to do more and better testing. He
became our Build Captain.

Continuous Integration
Owen started by implementing continuous integration (CI). Our “development
episode” process to date had been to get the latest code (including unit test code) from
Agile Testing: How to Succeed in an Extreme Testing Environment

the source repository, build the application locally, run all unit tests, then do TDD
for a task or feature, and run all unit tests again locally before checking in the code.
This was ¬ne up to a point, but didn™t protect us against occasional integration errors
caused when simultaneous developer changes were incompatible with each other.
Owen set up a build server and installed the continuous integration tool [56]. He
con¬gured it to monitor our source code repository, so that after a ten-minute “quiet
period” after any check-in, an automatic task would get all the code, increment the
build number (this was something we baked into the app), perform a clean build of
the whole application, then run all the unit tests. If the software failed to build or
any unit tests failed, the build was marked as broken.
This was visible on the CI tool dashboard, and an automatic email would be
sent to the developer responsible for that check-in. A little icon in every devel-
oper™s PC system tray turned from green to red. For added effect, the build server
would announce its dissatisfaction to the whole open-plan of¬ce with a loud Homer
Simpson “D™oh!!” As if that wasn™t enough, Owen also rigged up a pair of green and
red lava lamps to the server, which duly switched on the appropriate lamp according
to the good or bad state of the latest build. (For the bene¬t of the remote work-
ers in the team, and anyone else in the company, a regular grab from a web-cam
pointing at the lamps was put on our team collaboration web page.) All this created
a highly visible sense of quality and a team culture of putting quality as our top

Green Bar Addiction
In agile development and testing, you are theoretically working through a list of
tasks in priority order. Therefore, the new feature you are working on now is logi-
cally less important than ¬xing the defects in what has already been built. For our
team, a “broken build” became a stop-what-you-are-doing-and-¬x-it situation. We
were happy to explain the lava lamps to any curious passerby. But we didn™t want to
do that when the big red one was bubbling away. Whatever else needed to be done,
¬xing the build came ¬rst, and even though the build server pointed the ¬nger at
the last check-in, it was in the whole team™s interest to pitch in to get it ¬xed. Once
the green lamp was glowing again, we could proceed with con¬dence in the knowl-
edge that order had been restored.
This aspect of TDD is something of a revelation, even to a seasoned tester and
developer: the only acceptable pass rate for unit tests is 100% “ this is the only way
to get the “green bar” in the unit testing framework. This criterion is absolutely
necessary for getting a developer to accept responsibility for a broken build. Since all
the tests were passing before you checked in your code, your code must be the cause
of any failures “ no matter how much you believe your change had “nothing to do
with it,” the tests are now failing. If a test suite already has failures in it, a developer
would most likely dismiss any new failures as being unrelated to their code or caused
by a ¬‚aky and unreliable test suite.
Test-Infecting a Development Team

Getting FIT: Acceptance-Test-Driven Development
At this stage we were addicted to testing, proud of our healthily growing tree of tests
and feeling satis¬ed to see the “green bar” at the end of every small development
episode. However, we now had so many unit tests that we wondered if we were
suffering from the syndrome of treating everything as a nail because we only had a
hammer. With a smooth CI process in place, we knew we could afford to ramp up
our testing further and introduce new tools for speci¬c situations without it slowing
us down.
I had been interested for some time in FIT, the open-source Framework for
Integrated Testing, having heard it mentioned on various occasions and liking what
I saw when I investigated it on the web. Owen, our dedicated tester, soon had us
kitted up with a FIT server, and Fitnesse, the custom wiki wrapper for FIT.
The “arrival” of FIT allowed us to reassess the way we viewed TDD. FIT is essen-
tially a collaborative test tool, designed to allow a customer to express acceptance
tests in tabular format (alongside HTML documentation of the features to be tested),
which the development team subsequently automates with simple code “¬xtures” to
connect the test data to the system under test.
We changed our process for writing stories so that, instead of documenting
them as Word ¬les stored on a server, we created new stories as wiki pages on
Fitnesse. The beauty of this is that the story speci¬cations can include the functional
acceptance tests embedded in the descriptive documentation. It was easy for Edgar,
our customer, to change details of stories and to select and prioritize our solution
ideas. With the ability not only to write tests in the wiki but also to execute them and
see the results, we were doing TDD at an acceptance test level. This is an important
difference from unit-level TDD. For the ¬rst time, Edgar had true visibility and access
into the details of the tests that prove the behavior of each feature. He did not have
to just trust our unit testing, nor did he have to wait to try out features running
through the GUI “ he could open the relevant wiki page in his web browser, check the
tests for completeness and accuracy, create new cases if necessary, and with a click
of a button see the test data light up in green if everything was passing correctly.
“I™m loving FIT” was the way he summed it up.

Exploratory Testing
As the project matured we rotated a number of testers from the consultancy through
the project. One such tester, John, stood out as one who seemed to ¬t very nat-
urally into an agile project. He was keen, he was con¬dent, and he had no prob-


. 4
( 10)