. 2
( 10)


Using the model-view-controller architecture [31] for organizing application
programming, RoR incorporates a number of tools:

scaffolding, for simplifying the generation of basic Web site models and views;
WEBrick, a simple RoR web server; and
Rake, a simple build system.

By providing these tools combined with an agile development philosophy, RoR
can be seen as an example of a highly agile integrated development environment.
First released to the public in July 2004, RoR continues to ¬nd supporters in the
web developer community with an impressive list of adopters (since 2007 RoR has
been shipped by Apple with MacOS X v10.5 Leopard, for example).

3.6.3 Evolutionary Project Management
Developed by Tom and Kai Gilb, Evolutionary Project Management (Evo [32]) is
a twenty-¬rst-century agile project management method whose focus is on earlier
delivery of critical results and on-time delivery of deadlined results. Evo is claimed
to be able to provide more control over software quality, project performance, and
project costs than more traditional project management methods.
Evo is particularly suited to developing large-scale software systems, involving
complex technology and in rapidly moving environments. Evo aims to be highly
responsive and adaptable to unforeseen circumstances while delivering rapid results.
Evo follows ten key process principles:3
1. Real results, of value to real stakeholders, will be delivered early and frequently.
2. The next Evo delivery step must be the one that delivers the most stakeholder
value possible at that time.
3. Evo steps deliver the speci¬ed requirements, evolutionarily.
4. It is impossible to know all the right requirements in advance, but it is possible to
discover them more quickly by attempts to deliver real value to real stakeholders.
5. Evo is holistic systems engineering; all necessary aspects of the system must be
complete and correct, and delivered to a real stakeholder environment. It is not
only about programming, it is about customer satisfaction.
6. Evo projects will require an open architecture, because project ideas will change
as often as needed to really deliver value to the stakeholders.
7. The Evo project team focuses their energy, as a team, toward success in the
current Evo step. They succeed or fail in the current step together. They do
not waste energy on downstream steps until they have mastered current steps
8. Evo is about learning from hard experience, as fast as possible “ what really
works and what really delivers value. Evo is a discipline to make teams confront
project problems early, but one that delivers progress quickly when provably,
the team have got it right.
3 Sincere thanks to Tom and Kai Gilb for allowing me to reproduce this information.
Agile Testing: How to Succeed in an Extreme Testing Environment

9. Evo leads to early, and on-time, product delivery “ both because of selected early
priority delivery and because the team learns to get things right early.
10. Evo should allow teams to validate new work processes and get rid of bad ones

Bene¬ts cited by Evo users include reducing the duration and cost of require-
ments capture by 75% over traditional methods, reducing the integration effort to
upgrade from a previous code platform to a new code platform from hours to min-
utes, and improvements in the motivation, enthusiasm, and morale of developers
and testers, leading to signi¬cantly higher staff retention.
The success of Evo can also be measured by the number of high-pro¬le users
(such as IBM, HP, and Microsoft) that have been involved in learning and using the

3.6.4 Rational Uni¬ed Process
Originally developed by Ivar Jacobson, Grady Booch, and Jim Rumbaugh at Rational
Software Corporation, RUP is a use case“driven [7] iterative software engineering
process that employs the Uni¬ed Modeling Language (UML) notation [33] to incre-
mentally develop and deliver software. RUP provides industry best practice guidance
on software development and testing to its users through a web browser inter-
face, allowing the information to be accessed in a simple and easily distributed
The RUP iterations are organized under a number of phases “ inception, elab-
oration, construction, transition, and production “ with development and testing
activities grouped as a series of disciplines that run across the phases. These typically
include business modeling, requirements, analysis and design, implementation, test,
deployment, operations and support, con¬guration and change management, project
management, environment, and infrastructure management.
RUP provides comprehensive information on all aspects of software development,
including project roles and responsibilities, project planning and management, tasks
and timescales, inputs and outputs, testing techniques and approaches, reusable
project templates (such as test plan and test script templates), metrics, and guidance
on process improvement.
RUP explicitly supports agile approaches by providing a customizable metamodel
that allows users to select from the RUP best practices to create and maintain their
own subset of agile best practices. RUP also provides access to prede¬ned agile
customizations, such as its XP process plug-in.
With its early origins in Ivar Jacobson™s 1990s Objectory process [7], the devel-
opment of RUP has been a highly inclusive process, and it has drawn from many
sources including the experience of major industry experts, special interest groups,
industry standards and guidelines, and the experiences of many thousands of ordinary
Agile Development and Testing

Over the years, RUP has spawned a number of other agile methods and initiatives,
such as the EssUP [34] and the Object Management Groups SPEM.

3.6.5 The Essential Uni¬ed Process
The EssUP [34] is a continued development of the work begun in RUP by process
guru Ivar Jacobson, which identi¬es and promotes the use of a number of essential
practices that include use cases, iterative development, architecture-driven design,
team practices, and process practices, plus process improvement (through Capability
Maturity Model Integration [35]) and agile development.
Adoption and use of EssUP are accelerated through the use of a set of “playing
cards” “ concise ready reference cards documenting the process best practices in a
simple and highly accessible manner. Chapter 5 contains examples of these cards,
and Figure 5.1 shows the Test Case and Test Results cards.
EssUP appears to be gaining in popularity and use, with both IBM and Microsoft
incorporating its best practices into their respective process product offerings.

3.6.6 The Software Process Engineering Metamodel
SPEM [3] is, as its name suggests, a process metamodel that allows users to develop
and maintain their own speci¬c development and testing process.
SPEM is administered by the Object Management Group (OMG [36]) and based
on the RUP metamodel, which the Rational Corporation donated to OMG for the
purpose of supporting the development of a widely available open-source software
process engineering metamodel.
Employing the UML notation [33], SPEM provides facilities for individual devel-
opment and testing practitioners to create their own speci¬c development method
that incorporates those process elements appropriate to their own particular devel-
opment needs and requirements.

3.7 Summary
Having worked in the software business for more than thirty years, I must admit
to wondering if anyone will ever be completely satis¬ed with a particular means of
developing and testing software.
It seems that, since the earliest days of process development, there have been
practitioners who have either thought the resulting process was too lightweight or
too heavyweight, that it guided them to do their job or constrained their creativity,
that it gave them the freedom to use industry best practice or clipped their developer
and/or tester wings.
I have often wondered if a major issue driving the development of new agile
methods is a desire for IT practitioners to express their own individuality and their
own creativity; even where perfectly usable and successful agile methods like Scrum
Agile Testing: How to Succeed in an Extreme Testing Environment

and XP are available, there will be developers and testers who still ¬nd the need to
pare these down to a subset of “essential” agile techniques, and to then add some
bits of their own to form their own unique process.
On the other hand, it is an absolute fact that there are no two software projects
that are exactly the same:
There will be different-sized projects, with different-sized budgets and varying
available resources (the case studies in Part 2 of this book illustrate this point
The software being developed could be for internal use only, could represent a
bespoke development for a customer, or could be a product offered for sale to
numerous customers.
The software might be being developed for a safety-, business-, or security-critical
The speed of execution of the software may not be an issue, or the software might
need to execute in real time or might need to be delivered as an embedded or
pervasive solution.

Since there is no “one-sized” project, it is fair and reasonable to say that there
can be no one-size-¬ts-all process. Viewed in this way, it seems entirely natural, even
necessary, that diligent IT practitioners would ¬nd the need to de¬ne and document
project-speci¬c best practices for their own use and for that of their colleagues.
The case studies presented in the next section of this book provide an excellent
illustration: real-world IT practitioners, working on unique projects, facing the need
to deliver solutions in an agile manner, and needing to ¬nd their own interpretations
of agile best practices.
In many of the cases studies, the authors have started from a point of adopting
or using an established agile method, but have then had to modify or enhance the
method to more closely meet the unique requirements of their particular project.
The third section of this book provides an analysis of the case studies, looking for
common themes, identifying agile best practices that provided signi¬cant bene¬ts
to the project, highlighting agile methods whose use may need to be challenged in
certain circumstances, and making a proposal for how you can quickly, easily, and
successfully build and use your own effective and ef¬cient agile process.

People think computers will keep them from making mistakes.
They™re wrong, with computers you just make mistakes faster.
Adam Osborne

This section of the book contains twenty real-world case studies from organizations or
individuals who have adopted and used existing agile methods or who have developed
their own agile approaches to software testing.
Each case study follows a standard format and concludes with advice on what
aspects of the approach worked well and what aspects might be improved or removed
in future testing projects. The headings include
The case study title, author name(s), and a brief synopsis of the case study;
Introduction “ introducing both the author and the case study;
Overview of the Testing Challenge “ describing the particular testing challenges
that needed to be addressed in this case study;
De¬nition of the Agile Testing Process “ providing details of the particular agile
testing solution employed in this case study;
Results of the Agile Approach “ describing the outcome of the project and of the
agile approach used in this case study; and
Lessons Learned “ documenting the lessons learned during the project, including
what agile practices worked well, what could be improved in future projects, and
what practices should be treated with caution.

The following quick reference table provides a summary of the characteristics of
each case study to make it as easy as possible for you to identify case studies that
most closely match your own testing needs.

Case Study Quick Reference Table

Case Test
study Market Language Platform O/S Proj Size Automation Author

1 (C4) Survey & Reporting Java Web Various Medium Yes Gilb & Johansen

2 (C5) Government Various Various Various Large Yes Nick Sewell

3 (C6) Finance C++ PC WinXP Small No Graham Thomas

4 (C7) Telecoms Citrix PC Server Win2K Large No Michael G. Norman

5 (C8) Finance C VAX VMS Medium Yes Nick Denning

6 (C9) Life Assurance Java Web Various Medium No Geoff Thompson

7 (C10) Finance VB6“VB.net PC WinXP Large Yes Howard Knowles

8 (C11) Retail Java Websphere Various Medium Yes Colin Cassidy

9 (C12) Various Java Various Various Medium No Martin Phillips

10 (C13) Various Various Various Various Various Yes Stephen K. Allott

11 (C14) Various Java PC Various Medium Yes James Wilson

12 (C15) Finance Eclipse Java Windows WinXP Offshore Yes Peter Kingston

13 (C16) Trading Systems Various Wndows Small Limited Java Richard Warden
& Unix

14 (C17) Professional Services .NET WebServer WinXP Medium No David Evans

15 (C18) Various Fortune500 Java Mainframe Intel & Unix Large Yes Jon Tilt

16 (C19) Various Various Various Various Various No Isabel Evans

17 (C20) Wiki Lotus Notes Web Web Small No Dass Chana

18 (C21) SOA Various Various Various Various Yes Greg Hodgkinson

19 (C22) Research Java PC Windows Research No Lucjan Stapp and
Joanna Nowakowska

20 (C23) Media C# .NET Win Server 2003 Medium No Peter May
4 From Waterfall to Evolutionary Development and Test
Tom Gilb and Trond Johansen

Evolutionary Development (Evo) focuses on early delivery of high value to stakeholders and
on obtaining and utilizing stakeholder feedback. This case study describes, from a project
manager™s viewpoint, the results that one organization rapidly achieved after switching from
using a waterfall approach to Evo. The major bene¬ts of adopting an Evo approach come
from reducing the duration of the requirements phase from four to ¬ve weeks down to just
one week, and from paying greater attention to the quality requirements as opposed to the
previous practice of concentrating solely on the required functionality, resulting in a happier
workforce, happier clients, and more business.

4.1 Introduction
My name is Tom Gilb and I am an independent testing consultant and author of
Competitive Engineering: A Handbook for Systems & Software Engineering Man-
agement using Planguage (2005), Principles of Software Engineering Management
(1988), and Software Inspection (1993). My book Software Metrics (1976) coined
the term and was used as the basis for the Software Engineering Institute Capability
Maturity Model Level Four (SEI CMM Level 4 [37]). My most recent interests are
development of true software engineering and systems engineering methods.
In 1997 Kai Gilb and I published a book entitled Evo: The Evolutionary Project
Management and Product Development Handbook, describing an agile software
development and testing project management method [32]. Evo is an iterative
method in which a series of small delivery steps are taken until the project goals are
reached and the software delivered. Key differences between Evo and other project
management methods include an emphasis on achieving the requirements over slav-
ishly following formal processes, and the focus on ensuring that testing starts early in
the process, is conducted in each of the steps, and continues throughout the project.
This case study will focus on the adoption and use of the Evo method in the devel-

opment of the Con¬rmit product from Oslo-based company Con¬rmit AS, which
is where co-author Trond Johansen works as the head of project management.

Agile Testing: How to Succeed in an Extreme Testing Environment

Speci¬cally, this case study focuses on the role and use of Evo in reducing the
integration effort and timescales for developments and enhancements made to the

Con¬rmit product, as compared with the previously used waterfall-based [5] devel-
opment process.

4.2 Overview of the Testing Challenge

Con¬rmit is a powerful web-based software package that enables organizations
to gather, analyze, and report on key business information across a broad range of
commercial applications. Some twenty staff (including a quality assurance team of
three people) work in the Con¬rmit AS research and √ development department and
have responsibility for developing the core Con¬rmit functionality, as well as for
developing custom bespoke modules for speci¬c clients.
Early in the company™s history, when Con¬rmit AS had just a few clients, the
development process was fairly ad hoc and customer-driven. Changes to the software
were made on an almost daily basis, driven by client feedback, and while this deliv-
ered stakeholder value quickly, the process also resulted in numerous defects, long
working hours, and poor control of the software development and testing process.

As the Con¬rmit client base grew, Con¬rmit AS was faced with the need to
become more effective and ef¬cient in its development and testing of the product; it
was clear that a more formal process was needed.
Initially, Con¬rmit AS looked to adopt a waterfall approach, incorporating pro-
cess improvement via the Capability Maturity Model (CMM [37]). After following a
waterfall approach for a number of years, a number of issues were identi¬ed:

Risk discovery and mitigation was delayed until the later stages of the project.
Document-based veri¬cation was postponed until the later stages of the project.
There was dissatisfaction with the need to stipulate unstable requirements too
early; requirements change is perceived as a bad thing in waterfall.
Operational problems were discovered far too late in the process (typically at
acceptance testing).
There were lengthy modi¬cation cycles and too much rework and retesting.
Most important, the requirements were nearly all focused on functionality, not
on the quality attributes of the system.

With increasing dissatisfaction with their waterfall approach, Con¬rmit AS began
to work closely with Tom and Kai Gilb to roll out and adopt an Evo approach in order

to improve their development process for Con¬rmit . In particular, three key goals
were identi¬ed:

Reducing the integration effort “ that is, the amount of effort for each developer
to get the latest build to work on his or her workstation so that they were able to
code against the latest code platform;
From Waterfall to Evolutionary Development and Test

Reducing the time in minutes to upgrade “ from a previous code platform to the
most recent code platform from the past ¬gure of three hours to a goal of two
minutes; and
Improving the reliability of builds “ which historically had been an area where
there had been signi¬cant quality issues.

Historically, the Con¬rmit AS development process for the Con¬rmit product
had focused mostly on functional requirements, but with Evo the requirements
process changed with the realization that it is the product quality requirements that
provide Con¬rmit AS with a competitive advantage over its rivals.

4.3 De¬nition of an Agile Testing Process
While Section 3.6.3 provides a summary of the Evo process principles, the Con¬rmit
AS interpretation of the Evo method was as follows:

Identify stakeholders “ end users, superusers, support, sales, IT operations, etc.
De¬ne stakeholders™ real needs and the related product qualities “ what was
actually needed to support their business needs.
Identify past/status of product qualities and your goal “ that is, how much you
want to improve.
Identify candidate solutions for meeting your goals.
Develop a step-by-step plan for
delivering improvements to the process, and hence the product;
providing deliveries every week (with respect to stakeholder needs and product
quality goals); and
measuring progress: are we moving towards our goals (again, with respect to
stakeholder needs and product quality goals)?

Guided by Evo, Con¬rmit AS put in place a new requirements standard that
meant that requirements had to

be clear and unambiguous,
be testable,
be quanti¬ed,
not involve solutions, and
have a stakeholder focus.

The focus with requirements had to be on the day-to-day operations of the

Con¬rmit users and not represent a list of what they might or might not like.√
Following this process, all of the requirements for a new release of Con¬rmit were
de¬ned and documented in just one week, as compared to the previous norm of four
to ¬ve weeks.
Agile Testing: How to Succeed in an Extreme Testing Environment

In terms of Evo cycle time, a one-week-long Evo step was adopted in which the
key activities were as follows:

On the Friday:
The Project Manager (PM) delivers the detailed plan for the next week™s
version of the software (let™s call it Version N) to the customer Chief
Technical Of¬cer (CTO) prior to attending a weekly project management
The CTO reviews and approves or rejects the design and records their com-
The PM and CTO attend the weekly project management meeting to discuss
the Version N detailed plan and review CTO comments.
The developers focus on general maintenance and documentation tasks.
The QA team runs ¬nal build and setup for that week™s developed version (let™s
call it Version N-1), installs setup on test servers, and performs initial crash
test1 of Version N-1 prior to its release.
On the Monday:
The developers ¬rst generate the tests required to validate Version N and then
begin development of the code.
The QA team reviews the results of the continuous integration (CI) process
and reviews test plans and tests for Version N.
The users begin to use Version N-1 and compile feedback for the developers.
During Tuesday and Wednesday:
The developers continue to generate the tests required to validate Version N
and develop the associated code.
The developers meet with the users to discuss the feedback from using
Version N-1.
Incorporating the feedback from the users, the developers continue to gen-
erate the tests required to validate Version N and develop the associated
The QA team reviews the results of the CI process and reviews test plans and
tests for Version N.
On the Thursday:
The developers complete the ¬nal tests for Version N, code, and execute the
tests. The developers also complete the graphical user interface tests for the
previous week™s version of the software (let™s call this Version N-2).
The QA team reviews the results of the CI process and reviews test plans and
tests for Version N.

1 A testing technique used to determine the ¬tness of a new build or release of the application under test
to undergo further, more thorough testing [4].
From Waterfall to Evolutionary Development and Test

4.4 Results of Using an Agile Approach
The adoption of the Evo approach has resulted in increased motivation and enthu-
siasm among the developers and testers. For example, developers reported that they
are empowered and feel they have the scope for greater creativity.
The roll-out and adoption of Evo was relatively painless because the developers
and testers saw the value of using the approach and embraced the method (even
though some aspects of Evo were thought to be challenging to understand and
execute). There may have been some scope for additional training pre-rollout to
make the adoption even more successful.
There has been a paradigm shift with respect to the requirements process; previ-
ously Con¬rmit AS had focused on functional requirements, but following the Evo
route, the focus was on product quality requirements. This was seen as a major value
add over Con¬rmit AS™s competitors.
Often, due to time constraints, testing was postponed so that the next develop-
ment step could begin to plan. In the rush to complete the software, it was sometimes
the case that it was not possible to compensate for these test delays, and less testing
than was planned or wanted was actually completed.

Overall, the move from an ad hoc waterfall style of software development to the
iterative model promoted using Evo has been very successful; the adoption of
smaller, more easily managed delivery steps, combined with a focus on customer
quality requirements over slavishly following formal process, and with the empha-
sis on ensuring that testing begins as early as possible in the development and
continues throughout the project, have all led to signi¬cant improvements in the
speed and quality of delivered software and to more satis¬ed customers and happier
developers and testers. Tangible bene¬ts have included retention of existing cus-
tomers, recruitment of new clients, and retention of more motivated and happier
Speci¬c lessons learned include:

The adoption of a weekly development, test, and delivery cycle has been very
bene¬cial “ the developers, testers, customer representatives, and managers
have fallen easily into the weekly rhythm and have a regular, well-de¬ned, and
predictable time period in which to achieve their tasks.
As is so frequently the case in software development, testing occasionally
slipped “ in future improvements to the process, this needs to be addressed;
time must be made during the Evo cycle to ensure adequate testing is completed.
As with the introduction and use of any new, nontrivial technology, it is key
to ensure adequate training, mentoring, and consultancy (process reviews or
health checks) are planned into its roll-out. Tom and Kai were able to provide
Agile Testing: How to Succeed in an Extreme Testing Environment

these, helping to ensure the successful introduction, roll out, and use of Con¬rmit
AS™s Evo approach.
Although over time and under the in¬‚uence of process improvement, our
approach to CI has been re¬ned, the CI process has proven to be successful
and we intend to continue to use it for the foreseeable future.
Finally, a very important lesson to be learned is that there is always another
lesson to be learned “ the goal of ¬ne-tuning our processes continues today, and
will continue into the future.
5 How to Test a System That Is Never Finished
Nick Sewell, European Managing Director of Ivar Jacobson
Consulting Ltd.

In a fast-moving environment where requirements are changing all the time, in order to
support the customer™s needs, the testing strategy has to be superagile. How do you keep
the relevant documentation up to date? How do you communicate the changes to your team?
How do you ensure your testing covers the changes? How can you guarantee that you deliver
a system that works and meets the customer™s needs?
This case study reviews the issues that are being faced by a large, multiyear project
practicing incremental delivery, and shows how the team has used agile techniques to
address these issues. In particular we will look at how using techniques such as daily builds,
pairing testers and developers through an iteration, and the use of automated testing tools for
regression testing have enabled the team to successfully test and deliver a new, high-quality
release every six weeks!

5.1 Introduction
My name is Nick Sewell and I work for Ivar Jacobson Consulting Ltd. as the European
Managing Director. I have over twenty years™ commercial experience in the software
industry “ twelve of these ensuring that software teams deliver software successfully
using a range of techniques.
In a fast-moving customer environment where requirements are changing all
the time to support the user needs, the testing strategy has to be superagile.
For example, I have recently been working as a part of a team enhancing a large
business critical information system that needs to be updated every six weeks to
keep pace with an ever-changing variety of information sources and analytical needs.
Traditional test approaches are not suitable in this kind of situation; the test strategy
must be agile for the project to survive.
As we worked to develop and test the software, we needed to meet the challenges
keeping the relevant documentation up to date,
communicating the frequent changes to all members of the team,

Agile Testing: How to Succeed in an Extreme Testing Environment

ensuring that the changes are thoroughly tested,
ensuring that the changes don™t compromise or break the existing system, and
delivering a system that works and meets the customer needs every six weeks.

This case study identi¬es the issues faced by the project and discusses some of
the solutions that were explored to address those issues, such as

developing the test cases alongside the requirements,
pairing testers and developers through an iteration,
ensuring that all the documentation was updated as we went along,
relaying information to the team by means of a daily Scrum, and
employing automated testing tools to assist with the testing load.

It concludes with a review of what parts of our testing strategy actually worked
and what clearly didn™t.

5.2 Overview of the Testing Challenge
The project was ¬rst initiated in 1997 and involves the development of systems to
provide key information to government departments on demand.
As information sources are enhanced or new sources identi¬ed, the systems need
to be quickly updated. There is no apparent end date to this project, as it is inevitable
that changes and new requirements will continue to be identi¬ed for the foreseeable
The systems are truly “business critical,” requiring very high quality, reliability,
and availability. Essential updates and enhancements must be implemented quickly,
must be delivered on time, must be defect free, and must not degrade the quality or
accuracy of the existing functionality.
The project runs on a rolling three-year cycle, with major reviews carried out at
the end of each period. These typically result in changes to the personnel involved in
the project and the underlying customer needs. They can also in¬‚uence the overall
development approach taken. I became involved at the last three-year review, one of
whose recommendations was to adopt “a more agile approach.”
This is a large agile project, with some thirty staff, including customer repre-
sentatives. The project team changes regularly, with no one staying on the team for
any great length of time, other than a core of ¬ve people. It is critical to project
success that the productivity and quality are not harmed as a result of the high staff
turnover. Continuously integrating new team members into the team is a major
challenge; consequently, a great deal of attention has to be paid to keeping docu-
mentation up to date to ensure seamless handovers whenever people leave or join the
The project is run as a series of six-week iterations, with each iteration deliv-
ering production-quality code into the live environment. As a result of this rapid
development cycle and the need to deliver high-quality reliable code into the live
How to Test a System That Is Never Finished

environment, testing is absolutely critical to the success of each iteration, as well
as that of the overall project. The need to ensure that the code implemented in the
current iterations meets the customer requirements and is reliable, while ensuring
that the changes do not adversely impact the existing code base, means that testing
must be continuous, comprehensive, effective, and ef¬cient.

5.3 De¬nition of an Agile Testing Process
We decided that there were two aspects to the successful delivery of the system:

1. We needed to assemble the core of a great team that had the experience and
talent to work using the new agile techniques. This core of ¬ve people needed
to transfer knowledge to new team members as they moved in and out of the
2. We needed a lightweight and practical process to help us deliver on time, to
budget, and to acceptable quality.

Rather than try to adopt a whole new process we decided to adopt a smaller
number of agile practices taken from the Essential Uni¬ed Process [34]. The concepts
within this process are delivered using cards “ this means that minimal information
is referenced when it is needed. The practices used were Iterative Essentials and Use
Case Essentials.
The beauty of the Essential Uni¬ed Process is that it adopts a practice-based
approach, where the practices underlying the process are de¬ned separately in a way
that allows you to select just the ones that will bene¬t your project. In this case we
didn™t have to adopt the other practices from the process (such as the component-
based development or architecture practices) to be able to bene¬t from the use of
use cases and iterations.
Figure 5.1 shows a couple of the cards that we used “ these describe two of the
deliverables the project needed. The diagram on each was ticked to show the level of
detail required for that deliverable.
The practice approach provides a very agile way of using small parts of the process
to add value where and when needed. Lightweight is the key adjective for this, and
we took advantage of this to introduce the changes quickly and effectively.
We also followed the advice provided by the iterative practice for improving com-
munications and introduced daily Scrums. The improved communication allowed
us to minimize the amount of documentation we produced (although the minimum
on this project actually turned out to be quite a lot!). Combining smart communi-
cation with ¬t-for-purpose levels of documentation turned out to be very important
in supporting the testing effort needed to ensure high-quality and reliable delivery.
Communication through simply talking has proven invaluable in ensuring that all
stakeholders are headed in the same direction.
Due to the need for high quality, we placed the testing at the heart of the process.
Our process encouraged the involvement of the test resources throughout the project

Test Results
Test Case
A test case defines a set of test inputs and The purpose of a set of test results is to
expected results for the purpose of evaluating capture the outcome of executing a test.
whether or not a system works correctly.
Test Idea
Formulated Test cases: Test results:
Provide a foundation for designing and Provide objective evidence of the quality
Scenario implementing tests of the system produced
Provide a mechanism to complete and verify
Chosen Allow defects to be identified
the system specification
Provide verification that the system
Allow tests to be specified before
Variables behaves as required
implementation starts
Identified Analyzed
Provide a measure of system quality
Essential Contents: Essential Contents:
1..n Pre-conditions Test Instance Description
Defined Evaluated
1..n Input
Test Results
1 Scenario
Test Evaluation
1..n Observations (with Expected Results)
References to:
References to:
Test Specification
1..n Use-Case Specifications (flows)
0..n Defects
0..n Supplementary Requirements

5.1 Some cards from the use case essentials practice.
How to Test a System That Is Never Finished

in as many activities as possible (and in many more activities within this project than
any other I have previously worked on).
For example, the testers were involved in working with the customers alongside
the analysts, producing the test cases at the same time as the use cases (an integral
part of the use case essentials practice [38]). This test-driven approach provided us
with signi¬cant improvements in productivity and accuracy compared with tradi-
tional methods, and the software continues to be developed using this technique.
Similarly, we ensure that the requirements, both functional and nonfunctional,
have a test built alongside them during their development. We have found that
spending time on establishing traceability between the requirements, tests, and com-
ponents provides the minimal level of documentation that is needed and adds value.
This was especially true when the personnel changes within the team were so regu-
lar, new staff could understand the system by means of the documentation provided
and quickly become productive with relatively little negative impact on the project.
The idea of having good people plus good process within the team was also
extended to the customer; we took the approach that the customer representative
was an essential part of the team and mandated that customer representatives be
involved in the software development from day one. We followed the assumption that
“the customer will only know what they want when they see it,” and through having
them on hand at all times we could make sure that we were building and testing
the right thing. The agile principle of promoting good and regular communication
works very well with the customer and cuts down the amount of time spent on
clarifying or con¬rming issues.

5.4 Results of the Agile Process and Lessons Learned
The results speak for themselves: the project continues to deliver successfully, the
reputation of the project has excelled, and other projects within the same organi-
zation are now looking to reuse the approach developed in order to improve their
productivity and delivery records.
The key lessons learned from this project and which are continuing to be used in
the later iterations of the project include:

User requirements and test cases “ you need them both up front. Following
the use case essentials practice we have extended the role of use cases beyond
being just an approach for gathering requirements. As the use cases are con-
structed, a test resource is on hand to identify the test cases that will be needed to
cover functional and nonfunctional issues, quality, and integration. During the
discussions with the customer representatives, where requirements were being
identi¬ed, the tester would contribute by asking clarifying questions, such as,
“How do you verify if that works?” Having these sorts of questions asked during
analysis has proven to identify more details than we might otherwise have found
at that time. This ultimately resulted in us producing a single document that
Agile Testing: How to Succeed in an Extreme Testing Environment

covers what we would traditionally call a use case and a test case. This provided
us with accurate requirements and the details of the testing needed to ensure
that the requirements were met.
Pair developers and testers “ one effective technique we adopted was to pair a
testing resource with a developer during the implementation of the code. We
already knew about the bene¬ts of pair programming, and we attempted to gain
similar bene¬ts by bringing developers and testers together. This solution was
not perfect, as the perception of the developer was that the tester was acting
as a policeman, sitting on their shoulder constantly directing her or him. We
re¬ned this approach so that the tester was involved in reviewing what had been
delivered at the end of each day, so that this could be incorporated into the next
day™s testing.
Every day is test day “ I have heard it proposed many times during projects that
“we can cut back on testing to meet the deadline!” Testing seems to be the ¬rst
thing that is ditched when tight timescales start to make things dif¬cult. The
truth is that something can probably be delivered under these circumstances but
the reliability and quality of such a system is often questionable.
Once a project has started delivering, it is vital that testing is continuous in
order to keep to the levels of reliability required by the stakeholders. We have
tried to promote the fact that testing is the central cog in our project, and that
the testers need to be in communication with all members of the team in order to
fully integrate the work completed by the requirements analysts, developers, and
the customer community. If something is to be delivered, then it is the testers
who make the ¬nal decision as to whether or not that something is delivered into
the live environment.
In addition to daily builds and their associated tests, test cases are imple-
mented during an iteration as the code is being developed. The testers work
closely with the developers to help focus the effort on achieving deliverable code
that can successfully pass the test cases. Because the testers are involved in the
requirements-gathering effort, they are able to provide extra clari¬cation points
when a requirement is unclear. This means that we have analysts, testers, and
customer representatives available to the developers to help clarify or con¬rm a
particular requirement; this helps us cut down on the amount of documentation
produced while improving the quality of the communications.
Each and every day in our iterations is a day that the testers will work with
the team, with one common goal “ successful delivery.
Automated testing of daily builds “ The use of automated testing tools for regres-
sion testing has saved the project signi¬cant amounts of time, effort, and cost, and
has helped to ensure that the quality of the deliverable is maintained. The key to
using these tools successfully is to have a testing resource who fully understands
the tools and who has responsibility for keeping the scripts up to date every single
day. The goal is to run automated tests every night, with any issues highlighted
the following morning. Fixing these issues is a standard task all the way through
How to Test a System That Is Never Finished

the iterations “ so there was an assumption that time would be spent on this
activity and the iterations would be planned accordingly.
Role and use of Scrums “ One of the bene¬ts of holding daily Scrums has been
in promoting effective testing as the key to the delivery of higher-quality code.
Scrums have also been instrumental in ensuring that testers, analysts, developers,
and customer representatives are all working together as a team toward the same

5.5 Conclusions
Overall, the importance of testing within a project that has to deliver high-quality
code cannot be underestimated. Within this project, testing was given a great deal
of emphasis and, speci¬cally, testing was
introduced into the process as early as possible, such as during requirements
elicitation, with a tester working with the customer and the analyst;
present everywhere within the practices that we used;
employed throughout each iteration, with the testers initially working hand-
in-hand with the developers, and later used just to review the code developed
through the day;
the central cog of the development process, with the testers providing close
and effective communication between all the other stakeholders and helping
the analysts during requirements elicitation, the developers during coding and
delivery effort, and the customer community across the whole iteration; and
supported by the use of automated testing tools, whose test scripts were used to
ensure thorough and complete regression testing of the system under develop-
ment, as well as maintained and used on a daily basis to highlight new issues and
ensure they were corrected quickly, with minimum disruption to the project, and
with as little cost impact as possible.
Finally, it seems like a no-brainer, but having well-trained, experienced, and
enthusiastic people, combined with a practical, effective, and ef¬cient process, helped
us achieve noticeable success in a short period of time. Team agility can be achieved
when the team buys into the agile principles and they are all given the opportunity to
contribute to the successful end result. The practices of the Essential Uni¬ed Process
provided us with a refreshing lightweight process that was usable, added value, and
was palatable for the more agile-minded people in the team.
With agile practices and a good team, successful delivery becomes the norm.

5.6 Acknowledgment
I would like to express my gratitude to my agile mentor and company CEO Ivar
Jacobson for his support in my writing of this case study and for his assistance in
reviewing my material.
6 Implementing an Agile Testing Approach
Graham Thomas, Independent Software Testing Consultant

This case study documents the introduction and use of an agile testing approach that is based
on Extreme Programming (XP).1 XP was introduced to a traditional development project
where the classic view of the test process was proving to be an obstacle to keeping pace
with a very demanding release schedule that involved a high volume of small changes to
the system. The case study also explores the bene¬ts of introducing test automation to assist
with a faster release schedule and volume of changes, and discusses how automation can be
successfully integrated into an agile approach.

6.1 Introduction
My name is Graham Thomas. I have a range of experience in software development
and speci¬cally testing, with formal quali¬cations in programming, systems analysis
and design, and latterly in software testing. Most recently I have been involved as a
program test manager looking after high-value testing portfolios of work.
I became involved with agile development and testing methods in the autumn
of 2000, when I was working on a project with a small transatlantic software house
producing systems for the treasury dealing markets.
This case study looks at my involvement in the project, starting at the point where
I realized that traditional testing approaches were not working for an organization
that was delivering customized releases combined with a high volume of small
changes to an installed client base. The case study explores how we went about
designing and running a change program to implement the adoption of an agile
development and testing approach based around XP [20].

6.2 Overview of the Testing Challenge
City Street Traders is a midsized (220-person) software house producing solutions
for the treasury dealing marketplace. At the time of the case study the company had
1 See Appendix B for further details of the rules and practices of XP.

Implementing an Agile Testing Approach

an installed base of systems with over thirty clients and was adding to that base at the
rate of about three new clients per year. The company was also delivering upwards of
4,500 bespoke changes per annum to the installed client base. The company had sites
in North America and London, with most, but not all, development being carried out
in the United States. Testing, con¬guration management, and release management
were performed in London.
The company had grown organically during the previous decade but was now
beginning to experience problems with con¬guration management, organizational
growth beyond process capability, and lengthening testing windows.
What was required was a development and testing approach that would

shorten the lengthening testing windows,
support the many client con¬gurations, and
allow the high volume of bespoke changes to installed systems.

The only answer that seemed to meet all of these requirements was an agile
approach based around XP.
The anticipated bene¬ts were that the organization could reduce the release
windows from nine to twelve months down to less than six months, thus enabling an
aggressive technology refresh program while still maintaining the lucrative bespoke
enhancement business.

6.3 De¬nition of an Agile Testing Process
As described earlier, XP was selected as the base agile approach to be used on the
project, with the following additional local agile process details:

de¬nition of an organizational development methodology “ with variants for
new product releases, bespoke enhancements, and emergency ¬xes;
a continuous integration development approach “ which included the adoption
of development tools for
static analysis and
automated unit testing;
reduction of the release testing window “ from three to six months to four weeks,
based around smaller more frequent releases,
using test automation tools to give productivity bene¬ts, and
development of a continuous integration release testing approach; and
implementation of automated con¬guration management
to enable the management and control of customized releases and
to control a high volume of bespoke changes to the installed user base.

Some elements of the XP approach were not adopted, namely,

pair programming “ because of company management and developer resistance;
Agile Testing: How to Succeed in an Extreme Testing Environment

on-site customer “ primarily because of the nature of software houses developing
products for a market rather than speci¬c clients, and second for enhancement
work because of the disparate geographic location of the clients. Expert business
analysts were used as proxy on-site customers.
co-location “ was not possible because it would have meant splitting the release
testing team across the two sites, London and New York, so in mitigation virtual
co-location techniques were adopted, such as group working, videoconferencing,
and the use of tools to share workstation control.

To support this radical change in the organization, a change program was initiated
to manage and control the phased implementation, to ensure continuity of business,
and to minimize any impact to productivity and ultimately revenue.
Care was taken to ensure that there was good communication with the stake-
holders and sponsors. Workshops were held with developers, testers, and senior
management to explain and sell the bene¬ts of the changes. Frequent communica-
tion by e-mail, newsletter, and face to face ensured that all were kept updated with
progress and developments.
Budget was secured for development, testing, and con¬guration management

6.4 Results of the Agile Approach
The results of adopting an agile approach were mixed at best. The key bene¬ts that
City Street Traders gained from implementing the change program were as follows:
De¬nition of a development methodology “ the organization bene¬ted greatly
from the documented de¬nition of an integrated development methodology,
including analysis, design, build, test, and release activities. This development
methodology comprised variants for new product releases, bespoke enhance-
ments and emergency ¬xes. (City Street Traders was in a transitional stage from
the size of company where everybody knew each other and what to do “ less than
100 employees “ to the point where there were too many people to know every-
body “ 200+ employees “ and processes were starting to break down because
there wasn™t a common understanding across the organization.)
Release testing window shortened “ the release testing window was shortened
from three to six months down to four weeks. This was partly achieved by having
smaller, more frequent releases. The initial release test cycles took longer than
four weeks, until the developers rigorously applied a continuous integration
Identi¬cation of a core product set that would be tested every release cycle “ in
order to enable a four-week release test, a core con¬guration of the product set
was identi¬ed, the tests for which were then automated, giving a clear productivity
and quality gain.
Implementing an Agile Testing Approach

Closer working between development and testing teams “ the testers formed
a closer working relationship with the developers because the release testers
could focus on a smaller release, which was more current, and could therefore
provide more timely feedback to the development teams, to the point where the
developers would come to the release testers ¬rst with new ideas to get their

The results were mixed, primarily because, given the prevailing ¬nancial climate,
City Street Traders only half-heartedly adopted an agile methodology, not wanting
to endanger the current business model, productivity, and revenue streams.
A practical example of this was the developer™s keenness to produce minimal
documentation but without the balancing practice of pair programming. This led to
reluctance by developers to refactor (see Appendix B) someone else™s code because
there was still strong individual rather than collective ownership.

6.5 Lessons Learned
The change program progressed more slowly than expected. This was in part due to a
very ¬‚at organizational structure, which meant that it took longer than anticipated
to get everybody onside with suitable tools for unit test and code analysis automation
and con¬guration management.
The release testing team pressed ahead with automation of testing and reduction
of the release testing window to four weeks but found it dif¬cult to deliver in the
four weeks because the development and con¬guration management changes had
not been implemented. This led to lengthy integration delays caused in the most
part by a lack of continuous integration by the developers.
What worked well included

the drive for greater tool usage in development;
the adoption of con¬guration management disciplines, which enabled greater
control over the increasing complexity of the product con¬guration; and
the closer working relationship between developers and testers, which produced
a measurable increase in quality.

It could be argued that these disciplines could have been adopted anyway, regard-
less of whether an agile method, such as XP, was adopted.
There is always a danger with testing or quality-driven improvement programs
that the rest of the organization does not take them seriously. There was a feeling
that with this “testing-driven” change program, the changes were tolerated as long
as they didn™t cause any harm to the basic operating model of the company.
Additionally as the company grew and became established in the marketplace, it
was observed during the sales process that potential customers were performing a
more rigorous review of the development approach. There was a view that City Street
Agile Testing: How to Succeed in an Extreme Testing Environment

Traders™ management was keen to wear the agile badge as a way of giving credibility
to some of their established development practices.
What I would have done differently, with the bene¬t of hindsight, would have
been to work harder to make the change self-propagating. Although the development
team was already relatively small, I would have started with a smaller development
team, let them establish the bene¬ts, and then sell them onto their colleagues. Thus,
a momentum for change would build up, mitigating against resistance and half-
hearted adoption. The learning point from that experience is that just because the
company is small, you do not have to change it all in one go.
7 Agile Testing in a Remote or Virtual Desktop Environment
Michael G. Norman, CEO of Scapa Technologies Ltd

This case study deals with an agile approach speci¬cally tailored toward the integration and
deployment of packaged applications by or on behalf of enterprise customers.
The particular challenges of this case study were that no source code was available to
the test team, timescales were very challenging, and the majority of quality issues were
nonfunctional in nature.

7.1 Introduction
My name is Michael G. Norman. I have been involved with software test and perfor-
mance for more than ten years, most recently having taken on the role as CEO of
Scapa Technologies Ltd, a software tools vendor company. I was formerly involved
in forming the Eclipse Test and Performance Tools Project (TPTP, formerly Hyades),
which I ran in collaboration with colleagues from IBM and Intel and served on its
Project Management Committee and the Board of Eclipse for around ¬ve years.
Scapa Technologies is involved with a range of testing engagements with its
several hundred enterprise customers and tries to be methodologically neutral in
the way it develops its tooling, making it applicable in both traditional and agile
environments. In cases where it performs its own services, Scapa adopts a variation
on the agile approach, which is tailored toward its predominant customer pro¬le:
integration and deployment of packaged applications by or on behalf of enterprise
This case study describes a project where Scapa was engaged to test and produce
very fast feedback on a set of applications that were being deployed by a telecom-
munications vendor via a remote desktop technology. The case study refers to Citrix
technologies [39], but similar concerns and approaches apply to other remote and
virtual desktop technologies from Microsoft, VMware [40], and a range of other

Agile Testing: How to Succeed in an Extreme Testing Environment

7.2 Overview of the Testing Challenge
In this particular project, the telecommunications vendor was neither the author
nor the user of the applications under test; it was simply providing a secure hosted
platform to allow others to have access to applications. Access was via secure token
authentication [41] and the Citrix Secure Gateway into a Citrix server farm on
Windows 2003 Server.
The whole platform was subject to a traditional long-term top-down testing
project, which had been overtaken by political reality; the platform had gone into a
phased live roll-out before the test process had delivered signi¬cant results, and issues
were emerging in production. Speci¬cally the platform was subject to instability
(requiring frequent reboots) and the applications suffered from functional failure
under load.
In this context Scapa was asked to test the applications to isolate problems and
make recommendations about how they could be made more stable and less prone
to failure under load.

7.3 De¬nition of an Agile Testing Process
In contrast to most approaches to agile testing, there was no source code avail-
able in this environment. Furthermore, a number of contractual frameworks that
existed between the vendors involved, and the customer and Scapa, made it highly
unlikely that source code would be made available within the timescales of the testing
In addition, the majority of issues raised were nonfunctional; that is, they related
to performance and stability of servers either under multiuser load or after extended
execution. The scope of the functional testing only required consideration of failures
that could be detected by gross failure of the application and did not require detailed
arbitration of results.
Another key element is timeline; there were twelve applications and the test
timeline was twenty person-days (four weeks elapsed). The following conceptual
framework was applied to provide a performance model within which tests were

The Edge “ all systems “break” under suf¬cient load. System response time
increases, the system becomes unstable, or both, beyond sustainable levels when
the applied workload (in terms of transaction rates or number of users) is driven
above a certain capacity. We refer to the set of acceptable workloads as the
working Envelope of the system, and we are concerned with mapping out the
characteristics of the Edge.
The Signature “ systems “break” for a particular reason, and there is a character-
istic “signature” of associated system metrics, typically because one or another
system resource saturates, serializes, or leaks up to a static limit.
Agile Testing in a Remote or Virtual Desktop Environment

Diagnostic transaction set “ there is usually a small set of transactions that show
a set of differentiated “signatures”; that is, they cause the server to fail for a
set of identi¬ably different reasons. Testing processes can be simpli¬ed if they
are restricted to these transactions, and it is also possible to work with a test
environment that differs from the production environment as long as the same
signatures can be detected.
Calibration matrix “ a matrix of factors can be built that maps between the Edges
of the Envelopes for the diagnostic transactions on various environments.


Since testing is on an exact replica of production, no calibration matrix is
The edge of the Envelope (and corresponding signatures) is found for a set of
“prototypical” transactions whose characteristics are constructed by reference to
the architecture, not the business process. A judgment is then made as to how
the production workload relates to the Envelope.
Clear de¬nition of pretesting dependencies and subsequent close involvement of
the customer during testing to address remaining issues as they emerged.
The reduction of testing documentation and complexity to a minimum:
test plan “ standardized across all twelve applications,
test case design “ standardized across all twelve applications with only detailed
application-speci¬c variations,
systems performance metric collection “ informed and very much constrained
by the test case design and standardized across all twelve applications,
stress testing and soak testing1 using the same test cases,
test summary report “ maintained continuously with status updates to the
customer on a regular basis, and
template-based report for each application issued for review by the customer
as the results emerge, followed by full signoff of report set with customer.

7.4 Results of the Agile Approach
Within twenty days the telecommunications vendor had an effective understanding
of the performance and robustness characteristics of the twelve applications it had
deployed onto the platform and had raised detailed concerns with a large number of
its suppliers.
This in turn led to additional con¬dence within the telecommunications vendor
that the phased roll-out was being managed effectively and that resolutions would

1 Stress testing is a form of testing that is used to determine the stability of a given system or entity. It
involves testing beyond normal operational capacity, often to a breaking point, to observe the results.
Soak testing involves testing a system with a signi¬cant load extended over a signi¬cant period of time
to discover how the system behaves under sustained use.
Agile Testing: How to Succeed in an Extreme Testing Environment

be found to the issues that were present in the initial roll-out, which allowed the
planned roll-out schedule to continue.
The roll-out was ultimately successful, the applications were deployed success-
fully to the planned number of users, and the platform proved stable in use.
The agile methodology proved repeatable for subsequent versions of the applica-
tions deployed onto the platform.

7.5 Lessons Learned
Overall, the results of the use of an agile testing approach were good, and the agile
approach was seen to have worked well.
In particular, the co-location of the customer integration team and testers worked
very well, with the customer able to resolve environmental issues (availability of appli-
cations, logins, user pro¬les, security issues, etc.) that emerged during the testing.
A useful side effect of this has been an increase in the awareness of the integration
team of the role and need for effective and ef¬cient testing of the applications they
were deploying into their infrastructure (rather than assuming the vendors would
resolve any issues) and a greater respect for the testing role.
The agile approach was applied tactically and coexisted with a much larger tra-
ditional approach and was able to provide effective results to speci¬c concerns in a
timely manner.
The approach of using architectural analysis rather than business analysis to
de¬ne test cases was understood by all technical staff on the project and led quickly
to detailed proposals for intervention in the system to increase capacity, to limit the
impact of running over capacity, and to improve long-running stability.
Template-based test plans, test cases, reports, and so forth (see Appendices E, F,
and G for examples of agile templates), and reuse of test cases worked well, both in
simplifying the testing process and in articulating the approach to senior manage-
ment. They also allow for signi¬cant reuse when new versions of the applications are
to be deployed onto the platform.
8 Testing a Derivatives Trading System in
an Uncooperative Environment
Nick Denning, Chief Executive Of¬cer of Diegesis Limited

This case study describes a project where our team was contracted to assist the resolution of
quality issues in a commercially critical derivatives trading system and the challenges imposed
by the demanding development environment in a ¬nancial institution.
After reviewing the testing requirements, the timescales and budget, and the dif¬cult
nature of the development environment, an agile testing approach was selected. This case
study covers the selection process for the use of and the results obtained from using an agile

8.1 Introduction
My name is Nick Denning and at the time of this case study I was the managing
director of Strategic Thought Limited, a company I founded in 1987, which we
subsequently ¬‚oated on AIM1 in 2005 and which now specializes in the development
and sale of ¬nancial risk management products. At the time the events in this case
study took place, we were an IT consultancy with a strong reputation for work in the
Ingres Relational Database Management System arena. A particular strength of ours
was to be able to analyze systems, identify performance issues, and make changes
to the database con¬guration, the database structures, and the SQL code within the
Making changes to mission-critical applications, sometimes directly on the pro-
duction environment, required a careful consideration of the implications for a
change, and the ability to assess the risks and agree with a customer on the risk reward
pro¬le to justify implementing changes that bypassed the conventional release pro-
Our successful approach to this problem resulted in a number of engagements to
assist customers in building regression test environments so that they could continue
to design, implement, and deploy changes to a solution quickly and con¬dently on

1 AIM is the London Stock Exchange™s international market for smaller growing companies.

Agile Testing: How to Succeed in an Extreme Testing Environment

completion of the engagement with us. A signi¬cant class of problems were not
functional but rather performance related; solutions did not scale up under load.
They might simply slow down and be unusable, and in the worst case they might
run out of resources and crash. Our regression test approach began to incorporate a
“performance pack” with which we ran tests on duplicate installations, using copies
of production data and examples of transactions run a few days earlier in production,
to identify the reasons for solutions failing. These projects then led to engagements
associated with capacity planning to establish the upper limits for throughput on
systems, such as prior to the deregulation of markets when a major increase in
business was expected.
This case study describes a project where we were engaged to help our customer
improve the reliability and availability of a mission-critical system. The case study
describes an all-too-familiar problem in the software industry; the development of a
business-critical software solution, initially built by an in-house team, with a focus
on the delivery of key business functionality, to very tight timescales, but which after
two years of development displayed serious quality problems during day-to-day use.
The quality issues were found in two main areas:

1. Functionally the solution was known to have a number of bugs within it when
delivered, but generally it was possible for the back-of¬ce users to work around
these issues. In practice, it was found that only experienced users could perform
the work-around, and it took those users longer to use the system than had
been expected. A dominating factor was a business desire for new capabilities
to be delivered urgently, resulting in staff being rewarded by meeting delivery
deadlines, resulting in an informal policy of deploying into production come-
what-may and ¬xing bugs in production. As a consequence insuf¬cient effort was
put into testing prior to release.
2. Reliability and availability were also the issues; under circumstances of heavy
load, or indeed unexpectedly and without warning, the solution could crash,
after which it could take a number of hours to recover.

Given the business-critical nature of the solution, combined with the time-
critical nature of the derivatives market, these quality issues were unacceptable;
unless options (a particular ¬nancial instrument) could be exercised at precisely
the right time, it would not be possible to realize the pro¬ts earned through those
This was not a theoretical risk “ in fact this worst-case scenario had already
manifested itself; the customer had lost substantial amounts of revenue because
the system had crashed on these critical “end-of-period” dates, and it had not been
possible to recover the system and exercise these options. Although dif¬cult to
quantify precisely, events of this nature could, in the worst case, cost the company
in excess of hundreds of thousands of pounds (which does not include the intangible
but important effect of loss of reputation). This event, and the likelihood that it
Testing a Derivatives Trading System in an Uncooperative Environment

would occur again in the future, provided the customer with a compelling reason to
investigate how the problem could be solved.
Strategic Thought was approached by the customer to assist them in the devel-
opment and testing of the next phase of this project. The next section reviews the
testing challenge facing our team.

8.2 Overview of the Testing Challenge
The development and testing of the replacement software solution presented a num-
ber of challenges:

The nature of the business need meant that the software solution would be a sub-
stantial and complex system, automating a wide range of innovative functionality
which had previously generated substantial revenues for the business.
As valuable and knowledgeable resources, who were very busy managing the
day-to-day business of the customer, there was relatively limited time available
from the users to participate in a major testing program (even though there was
commitment from the users to assist, provided the amount of time could be
maintained within acceptable bounds).
There was a substantial development pipeline planned, and the testing approach
needed to integrate closely with development in order to avoid interfering with
the development plans as far as was practical.
The speci¬cation was incomplete and poorly documented, making test planning
and design very dif¬cult. Historically, the test approach had been to make the
solution available to the users for testing once it was complete and for the users
to sign it off once they were satis¬ed with the solution.
Historically, there had been very limited defect tracking documentation available,
and no defect tracking system against which bugs could be formally logged,
tracked, and ¬xed.

8.3 De¬nition of an Agile Testing Process
In determining the testing approach to be used, there were a number of issues to be

The characteristics of the development and testing process (as described earlier),
plus the historical situation we inherited, meant that this was a “special needs”
testing project.2 The common manifestation of this problem in the IT industry
is to exclaim that the system is too poor to salvage and should be rewritten from
scratch because the issue of understanding the problem is so large.
We had to ¬nd an approach that would accommodate the magnitude of the
challenge, work with it, and deliver progressive improvement by identifying and
2 Indeed the classic joke came to mind of the person who, when asking for directions, was told, “if that
was where I was going, I wouldn™t be starting from here!”
Agile Testing: How to Succeed in an Extreme Testing Environment

isolating the most signi¬cant defects as quickly as possible. These would then be
¬xed on a case by case basis. At the same time, we also had to provide con¬dence
that the ultimate goal of a high-quality robust system was achievable.
Although initially we had the commitment of all of the key stakeholders, we knew
we would have to work hard to ensure that we retained this support throughout
the process.

The details of our agile testing approach are as follows:
The software solution is correct “ Despite the problems that we knew existed, we
put forward the hypothesis that the software solution was correct (even with the
work-arounds and reliability issues). This provided us with a working baseline
from which to proceed.
User test run-through “ Using a copy of the current version of the software,
we asked the users to run through the standard tests that they would normally
undertake to validate the system and to show us the results that they would
Employ test automation “ We captured these tests (including the navigation
through the software and the speci¬c veri¬cations required) using functional
testing tools (classic capture“replay tools) and printed out the results, both
screen shots and reports, provided these to the users, and asked them to check
and sign off these tests.
Minimum test harness “ We developed a process to reliably prepare a release
package, installation instructions, and a set of minimum tests to run all of
the key functions of the system to prove that all major functions would run
without crashing. It is disappointing how many releases were deployed into
production and crashed, often because of con¬guration and release errors rather
than software bugs.
Regression testing baseline “ The signed-off automated tests provided us with a
suite of regression tests, which gave reasonable test coverage over all components
of the system. It was then possible to run the tests from a common baseline and
get identical results on successive runs.
Incremental new test development “ Working closely with the developers and
users, we then built a set of speci¬c tests for the new functions being developed
and worked with the users to de¬ne precisely what the new behavior of the system
should be and where any other changes to the system might be anticipated.
Test new build “ The automated regression tests and the new tests were then run
against the new version of the system. The assumption was that if the system was
correct following the new development, then there should be no changes to any
part of the system except those containing new functionality.
Review and analyze defects “ However, of course, in practice the system was
not correct; differences started to appear in a number of areas. Each unexpected
result was reviewed with the users to investigate the problem further, working
through each example to identify the bug.
Testing a Derivatives Trading System in an Uncooperative Environment

Log, ¬x, and retest defects cycle “ Once a bug was identi¬ed, it was logged and
passed back to the developers for correction. Once ¬xed, the software was retested
using the regression testing tools, iterating round this loop as many times as was
necessary until all the bugs were resolved.
Changes to facilitate testing “ In the options market, option pricing depends on
the time to expiry, that is, the difference between current date and the expiry
of the options contract. The solution took the current date from the computer
system clock. We needed to change the software to take the current date from
a database ¬eld so that if we ran the same test on different days we would get
identical results.

Finally we got to a point where we had resolved all the material bugs and the sys-
tem was successfully released. Following release, only one bug was discovered, which
we were then able to track back to an item that had been signed off incorrectly. This
result con¬rmed the suitability and success of the agile approach that we had adopted.

8.4 Re¬ning the Agile Testing Process
Having achieved this signi¬cant initial success, we then subsequently repeated the
process for the second release. A number of improvements were introduced in the
testing process leading up to this release:

Re¬ne test automation “ A number of additional regression test tools were
deployed. Of particular help was a tool that compared databases, so that we could
inspect every record in pairs of tables and identify any differences in those tables
and report them. This proved to be very useful in identifying a range of unexpected
differences in the system™s data that were not immediately apparent to the user
(since they related to trades that the user might not be inspecting for some
months until they were closer to the expiry dates of those trades).
Address system performance “ A performance pack was developed that was able
to capture system load information while regression tests were executed. We
could then compare the performance data of runs and identify any increases
in run times or system load. These could then be inspected (in conjunction
with pro¬ling tools) to identify any speci¬c areas of code that were generating

The testing team continued to work with the software for a further year, before
¬nally handing over the regression test environment to the developers to be inte-
grated into their normal development and test approach.
In the longer term, the results of this approach were impressive. There was only
one major crash of the system following the second release, and the system was never
unavailable on key end-of-period dates.
A tremendous success, one would think. Well it was, in many respects, and it
provided a model that has been reused many times in similar circumstances for
Agile Testing: How to Succeed in an Extreme Testing Environment

other customers. However, as you will see in the next section of the case study, this
was not the universal perspective of all parties involved!

8.5 Lessons Learned
This section describes lessons learned on the agile testing project in the following

developer relations,
a project is a project,
how to decide when to release,
de¬nitive evidence, and
the epilogue.

Developer Relations
We had enthusiastic buy-in from all parties when we started the project; in principle
the approach was well received, though there was some initial skepticism about
whether it would work in practice, and some surprise when the results promised
were delivered. However, in one particular area we experienced signi¬cant push-
back; our relationship with the developers started to break down when the bugs
began to be found!
As is the case in many parts of the ¬nance sector, staff are paid bonuses based
on their ability to deliver high-quality solutions into production. There was an
expectation that we would not ¬nd much wrong with the system and that our efforts
were going to be a bit of a sideshow. However, we found so many bugs that the ¬rst
release was two months late going into production.
In some organizations there is a signi¬cant “blame culture.” The testing process
shone a spotlight on individuals when they were developing their code. So from the
developers™ perspective, the test team became “the enemy.”
The situation became worse; as the developers gained an understanding of
how the testing process worked, and by introducing subtle changes to the code
each time the system changed, they were able to make the test scripts fail. We did
not realise at the time but subsequently learned that there was a deliberate initiative
to make changes to the solution user interfaces in order to disrupt the testing process
so that the testers would be perceived as the obstruction in the development process.
In order to address the issue of poor developer relations, in all future projects of
this nature we followed the following approach:
Identify facts and do not speculate on the causes “ having identi¬ed the facts,
let the developers do the investigation so that they can become heroes for having
found the cause of the problem and, as a minimum, have time to build a story
for their managers.
Ensure that the design approach takes into account all aspects of the deliverable “
such as changes to the code, changes to the regression test harnesses, preparation
Testing a Derivatives Trading System in an Uncooperative Environment

of test data. In this way, we estimate and optimize the time taken to perform all
these activities to complete a change. Too often we found design decisions were
taken to minimize the development process at the expense of the testing activity
and to the detriment of the overall delivery. If a developer has the choice of a
three-day work package that incurs ten days of testing effort as opposed to a
¬ve-day work package that incurs two days of testing effort, too many times the
developers chose the former. The approach needs to motivate best behavior for
the overall team.
Ensure that test speci¬cations are included in the work package “ in order to
de¬ne the completion criteria and to ensure the work package is complete when
the test criteria are satis¬ed. Any subsequent bugs are not the developer™s respon-
sibility but rather that of the person who wrote the test speci¬cation. Thus, the
developer has a structured approach to completion and does not waste time in ran-
dom testing “in the hope” of ¬nding a bug. Problems will arise later inevitably, but
they will be integration problems arising from issues that had not been predicted.
Developing the previous points, regression testing cannot remain separated from
the development process or it is doomed to failure in the medium term. Use of
the tools and maintenance of the test harnesses have to be undertaken by the
developer and changes in the code and in the regression tests associated with
that code made as part of the same work package or else the regression tests will
not be maintained.
Management responsibility for development and testing must be delegated to
as low a level as possible “ the test manager should be responsible for the
de¬nition of best practice, for quality standards, and for owning the regression test
environment in order to ensure that it is being run every night so that new bugs
are isolated as quickly as possible and passed back to the developers. However,
the developers must maintain both the code and the associated test harness.
As a consequence of the best practices set by the test manager, we can develop
an integrated set of test components that linked together support system testing
but can be decomposed into elements that can support both integration testing
and module testing. This enabled us to shorten development cycles considerably
by reducing the time for development of individual modules, the time taken for
testing, and the number of bugs detected at a late stage.

A Project Is a Project
A fundamental aspect of our approach in managing the testing was to view testing
activities as “just another project,” and as such, to ensure we followed good project
management best practice, such as that presented in PRINCE2 [22]. This approach

identifying the project sponsor, agreeing terms of references, and con¬ning your-
self to those terms of reference. You cannot expect to solve all the problems the
organization is experiencing, so focus on solving those you are asked to address


. 2
( 10)