Fe)
FUJITSU
FUJ00232561
FUJ00232561
Approach to Testing Project HNG-X
Commercial in Confidence
Document Title:
Document Type:
Release:
Abstract:
Document Status:
Author & Dept:
Internal Distribution:
External Distribution:
Approval Authorities:
Approach to Testing for Project HNG-X
Strategy (STG)
N/A
This is a joint Fujitsu Services/Post Office contract controlled
document describing the strategic approach to be applied for all
testing and integration activities performed for the HNG-X
Programme.
APPROVED
Lee Farman for HNG-X Programme
Chris Young, Lee Farman, Peter J Robinson, Dave Johns, James
Stinchcombe, Peter Jobson, Niall Finnegan, Mark Taylor, Giacomo
Piccinelli, Bill Reynolds, John Hunt
Name Role Signature Date
Bill Reynolds Fujitsu HNG-x Programme Manager
Mark Burley Programme Manager (Post Office.)
Note. See Post Office Account HNG-X Reviewers/Approvers Role Matrix (PGM/DCM/ON/0001) for
guidance.
©Copyright Post Office Ltd 2006
UNCONTROLLED IF PRINTED
Commercial in Confidence Ref: TST/GENISTG/0002
Version: V1.0
CONTRACT CONTROLLED Date: 20/12/2006
Page No:
10f 44
FUJ00232561
FUJ00232561
Approach to Testing Project HNG-X
(oe)
FUJITSU Commercial in Confidence
0 Document Control
0.1 Table of Contents
0 DOCUMENT CONTROL.
0.1 Table of Contents.
0.2 Document History.
0.3 Review Details.
0.4 Associated Documents.
0.5 Abbreviations/Definitions.
0.6 Copyright..
ennbhan
INTRODUCTION
Purpose of Document.
Scope of Document.
Context....
Compliance.
4.1. New/Revised Processes & Tools......
aaaan a
ahora
2 APPROACH REQUIRED.
3 GOALS & OBJECTIVES...
3.1. Test Mission (Top level testing goals
3.2 Test Motivators (Key sources determining tests,
3.3. High Level Objectives (The Principles)..
4 STRATEGIC APPROACH...
4.1 Objective Driven Testin:
4.1.1 Risk Based Foundation
4.1.2 Objective Driven Method
4.1.3
The Principles.............
4.1.4 Outline Process....
4.2 Formal Planning & Preparation.
4.3 Progressive Integration...
44 Stages & Types of Testing & Integratioi
4.4.1 Addressing the Issues. .25
4.4.1.1 Component Level. .25
4.4.1.2 System level... -26
4.4.1.3 Solution Level . 26
44.1.4 Release Level .27
44.2 Project HNG-X Testing Stages. 28
4.5 Regression Testing & Retesting. . 30
4.6 Horizon Application Based Areas. . 341
46.1 Iterative Lifecycles and Integrated Mixed-Discipline Project Team: .32
4.7 Parallel Testing...
4.8 Collaborative Working.
4.9 Defect Management...
4.10 Environments, Tools & Automation.
4.11 Metrics...
woe 12
oe 16
©Copyright Post Office Ltd 2006 Commercial in Confidence Ref: TST/GENISTG/0002
Version: V1.0
UNCONTROLLED IF PRINTED CONTRACT CONTROLLED Date: 20/12/2006
PageNo: 2 0f 44
FUJ00232561
FUJ00232561
Approach to Testing Project HNG-X
(oe)
FUJITSU Commercial in Confidence
4.12 Process Improvement.
5 PEOPLE, SKILLS & RESPONSIBILITIES...
6 ASSUMPTIONS, DEPENDENCIES, RISKS & CONSTRAINTB.....
6.1 Assumptions.
6.2. Dependencie:
6.3 Risks...
Figure 1: Scope of Testing Covered by this Document.
Figure 2: Document Context Map:
Figure 3: Objective Driven Testing - Outline
Figure 4: Outline Approach...
Figure 5 - Objective Driven Testing — Process Flow.
Figure 6: Component Test Context....
Figure 7: Component Integration Test Context.
Figure 8: Testing Stages — Overlap.
Figure 9: Project HNG-X Testing Lifecycle Overview...
Figure 10: Testing Lifecycle — Always Iterative
Figure 11: Parallel Testing for Project HNG-X 0
©Copyright Post Office Ltd 2006 Commercial in Confidence Ref: TST/GEN/STG/0002
Version: V1.0
UNCONTROLLED IF PRINTED CONTRACT CONTROLLED Date: 20/12/2006
PageNo: 3 of 44
FUJ00232561
FUJ00232561
Approach to Testing Project HNG-X
FUJITSU Commercial in Confidence OFFICE
0.2 Document History
Version No. Date Summary of Changes and Reason for Issue Associated Change -
CP/PEAK/PPRR
Reference
04 21/08/06 Initial draft.
0.2 21/08/06 Updated following initial feedback
03 20/10/06 Updated following informal review.
1.0 20/12/06 Updated following formal review
0.3 Review Details
Review Comments by
Review Comments to
PostOfficeAccountDocumentManagementf
Mandatory Review
Role Name
Fujitsu HNG-X Chief Technical Officer Giacomo Piccinelli
Fujitsu HNG-X Test Manager Peter Dreweatt
Post Office HNG-X Testing Manager Chris Young
Post Office Commercial Manager Simon Glynn
Fujitsu HNG-X Test Design Strategy Peter Robinson
Fujitsu HNG-X Programme Manager Bill Reynolds
Optional Review
Role Name
Fujitsu HNG-X Programme Assurance Manager Jan Holmes
Post Office Design Authority Torstein Godeseth
Fujitsu Programme Office Manager Graham Chatten
Post Office HNG-X Test Manager Andrew Thompson
Post Office HNG-X Programme Manager Mark Burley
Fujitsu Development Manager Gill Jackson
Fujitsu Commercial Manager Hilary Forrest
Issued for Information — Please restrict this
distribution list to a minimum
Position/Role Name
NA
(* ) = Reviewers that returned comments
©Copyright Post Office Ltd 2006 Commercial in Confidence Ref: TST/GENISTG/0002
Version: V1.0
UNCONTROLLED IF PRINTED CONTRACT CONTROLLED Date: 20/12/2006
PageNo: 4 0f 44
FUJ00232561
FUJ00232561
Approach to Testing Project HNG-X
(oe)
FUJITSU Commercial in Confidence
0.4 Associated Documents
Reference Version Date Title Source
(1) I PewocmTeMoo01 I 1.0 13/6/06 Fujitsu Services Post Office Account HNG-X I Dimensions
D it Templat
(D0 NOT REMOVE) cument Template
(2). I vwsTRI064 1.0 15/08/05 __I Testing Approach for the Horizon System PVCS
13] I vwwsTRI062 20 16/11/2005 I Fujitsu Services Testing Strategy for Horizon I PVCS
‘System
(4) I RM/STR/O05 03 21/06/2005 _I Horizon NG — Programme Test Strategy PVCS
(5) I a na nla HNG req 2.1 secti6.xls (compliance matrix for I email
HNG requirements on testing)
(7) I wa na nla Branch Migration V0-2.doc (migration working I Email
paper transaction data)
(9) I wa na nla Process_Test_Execution_Main.pdf and I CafeVic
Process_Test_Planningandprep_Main.pdf
(corporate testing processes)
[10] I DE/STRIO10 03 Strategy for Delivering T-Release Content Email
[11] I vvsTRI086 o4 T-Release Testing Strategy (Post S92 Testing I Email
Strategy)
(12) 1.0 Establishing and Assuring the HNG-X User I Email
Interface
[13] I TSTGENW/STGi0001_ I 1.0 17/03/2006 I HNG-X Testing Strategy Dimensions
[14] I PGM/PAS/PRO/0003 Code, Build and Component Test
[15] I PGM/PAS/PRO/0004 Test Planning & Preparation
[16] I PGM/PAS/PRO/O005 Test Execution processes
Unless a specific version is referred to above, reference should be made to the current approved versions of the
documents.
On each revision of this document the above list of associated documents should be considered. If they have changed
since last used, then any such changes may be relevant to the strategic approach presented by this document. Where
any such changes have been reflected in revising this document, the reference given for the document concerned must
also be updated to indicate the correct version number.
N.B. Printed versions of this document are not under change control.
0.5 Abbreviations/Definitions
Abbreviation Definition
API Application Program Interface - a published/approved interface through which to access an
application for a given purpose
APOC Architectural Proof of Concept — brief exercise prior to main development activity to prototype/prove
architectural principles, probe solution characteristics and trial technology combinations
BIT Horizon only terminology - Business Integration Test - a sub-stage of SV&l as used for Horizon
Application testing
BSTP Business Specified Test Point - a supplementary specification to Business Threads, to cover an
area of particular importance to Post Office, but where it cannot readily be fitted into a Business
Thread
BT Business Thread — a (typically lengthy) test scenario contrived to link together many areas of
particular importance to Post Office, such that as the tests for these areas are one, one feeds the
next, gradually maturing the data along a chain of system events, and so also proving data flow
integrity
‘©Copyright Post Office Ltd 2006 Commercial in Confidence Ref: TST/GEN/STG/0002
Version: V1.0
UNCONTROLLED IF PRINTED CONTRACT CONTROLLED Date: 20/12/2006
PageNo: 5 of 44
Fe)
FUJITSU
FUJ00232561
FUJ00232561
Approach to Testing Project HNG-X
Commercial in Confidence
CAST Computer Aided Software Testing - term used to refer to the use of purpose built management and
automation tools for software testing
ciT Component integration Test - component level verification and integration stage, against the
design, where an individual component is linked with its direct neighbours
cM Configuration Management — used interchangeably to refer to the system/toolset, the organisation,
the processes, and the principles of managing the configurable items making up the solution
cMMI Capability Maturity Model integration — a quality management model seeking to promote continuous
improvement and increasing maturity of process and supporting information/data
cR Change Request
ct Component Test - component level verification stage, against the design, where an individual
component is exhaustively tested in isolation
DIT Direct integration Test - formal validation of an external system interfaces, performed for Horizon
Application
DR Disaster Recovery
E2E Horizon only terminology - End-to-End testing stage - performed for Horizon Application by Post
Office
ELT Horizon only terminology - Extended Link Test — verification stage, linking units across platforms,
performed for Horizon Application
£OD End of Day
Gul Graphical User Interface
HLD High Level Design
HLTP High Level Test Plan
Horizon Application
means software which performs or supports a specific business function at a Branch or in the Back
Office and which operates on the Horizon Service Infrastructure, including (without limitation):
@ those applications listed in (as applicable) Schedule B4.2 or Part 2 of Annex 2 to
Schedule B5; or
() any applications of a similar nature;
a Interface
vo Input/Output (usually disk reading and writing traffic)
iT Information Technology
LAN Local Area Network
Ts Low Level Test Script
LsT Live System Test - the final stage of testing for Horizon Application just before Go-Live, using a
highly Live-like environment managed by the Live operational support team
UT Horizon only terminology - Link Test - verification sub-stage (part of Unit Test) , against the design,
linking individual modules together into Units, performed for Horizon Application
MT Horizon only terminology - Module Test - verification sub-stage (part of Unit Test), against the
design, testing an individual module, performed for Horizon Application
NFR Non-Functional Requirement
NT4 Microsoft's NT version 4 operating System for the PC
0-0 Object-Oriented, or Object-Orientation
PAT Product Acceptance Test — brief confirmation that third party deliverable meets minimum
acceptable standard, so as to insulate later testing stages from potential disruption
PC Personal Computer
PED Physical Environment Description
©Copyright Post Office Ltd 2006
UNCONTROLLED IF PRINTED
‘Commercial in Confidence Ref TST/GEN/STG/0002
Version: V1.0
CONTRACT CONTROLLED Date: 20/12/2006
PageNo: 6 of 44
FUJ00232561
FUJ00232561
Approach to Testing Project HNG-X
(oe)
FUJITSU Commercial in Confidence
POA The Post Office Account within Fujitsu Services
Project HNG-X means the design, development and testing of the HNG-X Service Infrastructure and the Business
Capabilities and Support Facilities, together with all of the HNG-X Project Activities, but excluding
the Associated Change Activities
PRF Problem Review Forum — joint supplier/customer forum discussing and prioritising problem reports
mostly arising from testing
RAB Release Authorisation Board
RV Release Validation — release level validation and integration stage, preparing the solution for release
and confirming it is acceptable
SAN Storage Area Network
sT System Test — validation stage (though performed within the development arena), against the
system requirements, integrating all the necessary component to form a discrete system within the
overall solution
suc System Use Case
SUCM System Use Case Model
sval Solution Validation and Integration (or for Horizon, System Validation and Integration) — solution
level validation and integration stage, assembling all the systems together to form the whole solution
and validating it against the requirements
TBC To be completed — an area of the document to be completed at some future point when the
necessary information emerges as the project lifecycle unfolds
TBD To be determined — a particular point as yet unknown, yet to be determined in detail
TPOC Testing Proof of Concept - devising, developing and piloting any new or changed testing
processes, tools and automation methods, and environment management and build procedures, in
advance of having to use them in anger in the mainstream programme delivery
TPS Transaction Processing System — one of the business applications within Horizon Application
UAT User Acceptance Test — formal stage of testing performed by end users to confirm acceptability of
a delivered system (no longer performed for Horizon Application, but a common concept within the
IT industry)
ul User Interface (see also GU!)
UT Horizon only terminology - Unit Test — verification stage on Horizon Application which includes MT
and LT
WAN Wide Area Network
XP Microsoft's Windows XP operating system for PCs
©Copyright Post Office Ltd 2006 ‘Commercial in Confidence Ref TST/GEN/STG/0002
Version: V1.0
UNCONTROLLED IF PRINTED CONTRACT CONTROLLED Date: 20/12/2006
PageNo: 7 of 44
FUJ00232561
FUJ00232561
Approach to Testing Project HNG-X
(oe)
FUJITSU Commercial in Confidence
0.6 Copyright
© Copyright Post Office Ltd. All rights reserved. No part of this document may be reproduced, stored or
transmitted in any form without the prior written permission of Post Office Ltd
©Copyright Post Office Ltd 2006 Commercial in Confidence Ref: TST/GENISTG/0002
Version: V1.0
UNCONTROLLED IF PRINTED CONTRACT CONTROLLED Date: 20/12/2006
PageNo: 8 of 44
(oe)
FUJITSU Commercial in Confidence
FUJ00232561
FUJ00232561
Approach to Testing Project HNG-X
1
Introduction
1.1 Purpose of Document
This contract controlled document covers the agreed joint testing approach required for
Project HNG-X.
This document is based on the jointly produced HNG-X test strategy [13] and makes
significant re-use of this material. To achieve detailed understanding of the background to
the approach outlined in this document, the reader should refer back to the strategy
document.
Consistent with the general aim to reduce total cost of ownership, this is a joint document,
meeting the needs of both Fujitsu Services and Post Office, rather than each organisation
producing separate documents. This document describes the intention to adopt a single
consistent approach to the testing and integration for HNG-X, exploiting synergy wherever
possible, to eliminate unnecessary duplication of effort and to reduce the overall elapsed
time consumed without compromising product quality.
To this end it is intended to:
e Provide a central artefact defining the strategic approach with which to govern the
testing and integration activities necessary for HNG-X, both for Fujitsu Services and
for Post Office.
e Provide visibility to the stakeholders concerned, that adequate consideration has
been given to various aspects influencing the testing and integration required, and
where appropriate to have those stakeholders approve the strategy.
* Communicate the agreed approach to all relevant parties, explaining how the
approach addresses the principal drivers of the programme, defining at an outline
level the scope and coverage of the testing required, and identifying the stages and
types of assurance and testing to be employed.
e Indicate the approach to be adopted for tools, automation, environments, test
management, defect management, metrics, traceability, and configuration
management.
e Highlight the key risks, assumptions, dependencies, constraints, and
responsibilities.
1.2 Scope of Document
This strategy covers all the testing activities required across the whole Project HNG-X, for
both Fujitsu Services and Post Office. It spans from initial involvement in the requirement
analysis stage, up to and including pre-Live Operational Proving, and also covers the
testing of the Microsoft Windows XP Counter application, following shortly after completion
of the Project HNG-X counter application migration.
It is concerned with all aspects of that testing and integration, encompassing both static and
dynamic forms of code based testing. It covers both verification and validation
perspectives, and also the integration of software components, systems, and the whole
solution.
©Copyright Post Office Ltd 2006 Commercial in Confidence Ref: TST/GENISTG/0002
Version: V1.0
UNCONTROLLED IF PRINTED CONTRACT CONTROLLED Date: 20/12/2006
PageNo: 9 0f 44
FUJ00232561
FUJ00232561
Approach to Testing Project HNG-X
Fe)
FUJITSU
Commercial in Confidence
However, this strategy is confined to covering only the testing and integration activities
required for the Project HNG-X. It does consider the Horizon Application systems, but only
in so far as the Project HNG-X solution involves re-engineering and/or re-deployment of
those systems, and in so far as the two architectures coexist in hybrid form during the
counter application migration period. (See [10] & [11] for relationship to Horizon Application
T-Releases and the strategy for testing them.)
It is not concerned with the testing and integration required for the succession of Horizon
Application releases that are potentially continuing in parallel with and in preparation for the
Project HNG-X — these are covered in separate testing strategy documents specific to those
Horizon Application releases.
Similarly, it is not concerned with the ongoing testing and integration of the Project HNG-X
solution to take place for possible future releases following the completion of the Project
HNG-X. Again, that will be covered in separate documents.
Ongoing Horizon
releases, in parallel
with and in preparation
for HNG-X, up until
agreed Horizon cut off
HNG-X Programme
including migration of
Counter platforms to
Microsoft Windows XP
Ongoing HNG-X
releases, outside the
HNG-X Programme,
M——_,—Y following the initial
release of HNG-X
Scope of testing and
integration activity
covered by this
document
Figure 1: Scope of Testing Covered by this Document
1.3 Context
This approach document is specific to the Project HNG-X and stands independent of the
strategies previously agreed for Horizon Application (see [2], [3], [4], [5], [10] & [11]).
Nonetheless, these earlier documents remain relevant. In particular, to assist continuity,
much of that earlier terminology has been retained, having become familiar through long
use on Horizon Application. Similarly, corporate processes and CMMI requirements are also
taken into account (see Figure 2: Document Context Map).
©Copyright Post Office Ltd 2006 Commercial in Confidence Ref: TST/GENISTG/0002
Version: V1.0
UNCONTROLLED IF PRINTED CONTRACT CONTROLLED Date: 20/12/2006
Page No: 10 of 44
FUJ00232561
FUJ00232561
Approach to Testing Project HNG-X
(oe)
FUJITSU Commercial in Confidence
Toron
Corporate Se Noe Tsar Expectations
Sasi I I] oe
Pam
Stakeholders
Existing Approaches Source Documents
~ 1 =
Informs Shapes Expects
~ ¥ yea
Development HNG-X Performance
Strategy es Testing ney Management
Aligns Strategy Aligns Strategy
Close Relation This Document Close Relation
ot
XN Directs “
Informs ' Expects
a
ch
Figure 2 : Document Context Map
1.4 Compliance
A number of compliance areas have been taken into consideration for this document,
including existing Fujitsu Services Post Office Account (POA) processes, wider Fujitsu
Services Corporate Processes, POA’s adoption of CMMI, and the Post Office initiative
known as Harmony. Where possible this strategy is aligned or compatible with these
process frameworks.
Overall it is the existing POA processes, as used for Horizon Application, which carry most
relevance for this document. A strong preference has been expressed by Post Office (the
Customer) to continue using the existing terminology where practicable. As this is a joint
document, for both POA and Post Office, adopting a common and familiar terminology is
very important. Also, as the Horizon Application systems feature heavily within the testing
and integration of the Project HNG-X solution then the investment made in these existing
procedures has a significant bearing.
1.4.1. New/Revised Processes & Tools
To minimise the impact and maximise the benefit of iterative and targeted testing, using
comprehensive regression testing packs, it is a requirement of this approach to adopt
automated testing techniques wherever and whenever it provides a clear benefit. The
adoption must therefore be pragmatic, embracing automation where it can be readily
©Copyright Post Office Ltd 2006 Commercial in Confidence Ref: TST/GENISTG/0002
Version: V1.0
UNCONTROLLED IF PRINTED CONTRACT CONTROLLED Date: 20/12/2006
Page No: 11 of 44
FUJ00232561
FUJ00232561
Approach to Testing Project HNG-X
Fe)
FUJITSU
Commercial in Confidence
applied, and avoiding it where costs become prohibitive. The detailed test automation
strategy will need to be developed, but it is clear that the primary focus should be in the
early verification stages (these are small, self contained, and have fewer data
dependencies to contend with, but require exhaustive coverage, making them ideal
candidates for automation). Automation of the later stages of testing should be approached
more critically, adopting automated scripts only where the benefit is clear, and where the
cost of future maintenance of scripts will not escalate. (Sample coverage to support
minimum regression testing would seem appropriate.)
As part of developing any automated functional and non-functional test suites, consideration
should be given to how tests will function when the application under test is connected to
3rd party systems. Fujitsu shall inform Post Office at time of automated test development
of any known issues that would prevent these tests from being executed in an end to end
test environment. (requirement reference TST 285)
The new testing processes and tools necessary to address any new development
approaches and their new technologies (including any test automation involved) must be
developed and piloted well in advance of their full-scale use in anger.
Similarly, the mechanisms necessary for exploiting the Disaster Recovery installation for
testing purposes, and so obviating the need for a separate Test Estate, must be devised,
developed, piloted, and used throughout the Project HNG-X development and testing
lifecycle programme
2 Approach Required
Based on the programme considerations and implications, an appropriate testing approach
for Project HNG-X is in outlined as follows:
e Engage with the requirements analysis at the earliest opportunity to influence the
emerging requirements, ensure that they are testable and to assist with the
codification of appropriate scenario based Acceptance Criteria. (It is important to
note here that requirements traceability will have to be maintained throughout, from
requirements sign-off through to acceptance sign-off, and so the means by which
this traceability will be achieved must be built into the requirements specification
process from the outset. Some method of persistent and unique identification will
be essential.)
e Run an Architecture Proof of Concept (APOC) exercise to provide early feedback
on the suitability of the proposed architecture, and to prove the scalability
mechanisms and get early indications of the likely system performance.
(Commence Performance Modelling to support this activity.)
« Continue Performance Modelling throughout Project HNG-X.
e Run a Testing Proof of Concept (TPOC) exercise to develop and prove the
necessary testing processes and tools to cope with any new development
approaches and the new technologies used for Project HNG-X, and to pilot the
mechanisms for making use of the DR site as the Test Estate. (As the TPOC will be
run alongside the APOC, then it is likely that certain objectives for the TROC may
in fact be achieved by the APOC. Care will be taken in planning the two to exploit
synergy and avoid duplicated effort.)
e Adopt Risk Based Testing, to ensure that the principal drivers of reduced total cost
of ownership and Business Equivalence are achieved in an acceptable and cost
effective fashion.
e Apply Objective Driven Testing techniques to simplify the implementation of Risk
Based Testing, and to facilitate removing duplicated effort, exploiting synergy,
reducing overall elapsed time, and maintaining product quality.
‘©Copyright Post Office Ltd 2006 ‘Commercial in Confidence Ref: TST/GENISTG/0002
Version: V1.0
UNCONTROLLED IF PRINTED CONTRACT CONTROLLED Date: 20/12/2006
Page No: 12 of 44
Fe)
FUJITSU
FUJ00232561
FUJ00232561
Approach to Testing Project HNG-X
Commercial in Confidence
o Analyse source material, derive Test Objectives (common set covering
needs of all parties)
o Engage with Business and Operations to conduct highly granular risk
assessments (workshops), and so prioritise the Test Objectives
o Start the outline test planning by seeking to exploit synergy, combining
multiple objectives to be covered by a minimum number of tests
o Normalise the outline tests by considering their likely environmental
needs/constraints
o Produce High Level Test Plans accordingly (Logical level)
o Generate Low Level Test Scripts (ideally in automated form) and
associated Test Data.
. -7 —— ~
Outline Risks, S .
Se — yj
“oases ——_——— Understanding Business Engagement
Test fn 1
Requirements Analysis \ Ops Engagement!
i U
Acep. Criteria
\ Granular
Test Risk Assessments /
Architecture
Objectives (incl. qualitative) 7
Ss Establish Risks
a Prioritise Express as Priorities
Ratify Accordingly
Threads Priortised, Use to DRIVE Testing
i Test Feeds detailed
Technical
Inter-dependencies Test Analysis
mana
Influences planning & Drives planning &
Management of Delivery Management of
Sequence (from Dev to Test) Testing effort
Figure 3: Objective Driven Testing - Outline
Apply a process of Progressive Integration, incrementally assembling and proving
ever larger portions of the solution, following a clear sequence of lifecycle stages
(i.e. adopting appropriate controls regarding the promotion of deliverables from
stage to stage to ensure their inherent inter-dependencies will be satisfied).
For existing Horizon Application areas being carried forward into the Project HNG-X
solution, follow the existing testing lifecycle, but create and retain comprehensive
Regression Test packs for the Unit Test (UT) and Extended Link Test (ELT) stages.
For new developments such as new code or significant changes to existing code,
take a generic perspective to provide exhaustive verification (against the design
spec) at the component level.
Increase the emphasis on code reviews to trap code defects at the earliest possible
opportunity. They also serve to apply an additional perspective (testing) on the
design specification.
Adopt new terminology to emphasise the need for increased rigour in this area —
Component Test (CT) and Component Integration Test (CIT)
©Copyright Post Office Ltd 2006 Commercial in Confidence Ref: TST/GENISTG/0002
Version: V1.0
UNCONTROLLED IF PRINTED CONTRACT CONTROLLED Date: 20/12/2006
Page No: 13 of 44
Fe)
FUJITSU
FUJ00232561
FUJ00232561
Approach to Testing Project HNG-X
Commercial in Confidence
Assemble and validate (against the requirements) each system as and when it
becomes viable, and integrate with its supporting infrastructure - System Test (ST)
Adopt integrated, mixed-discipline project teams for CIT, & ST, for example ST
team supporting CIT work.
Make extensive use of stubs and harnesses, etc., for CT & CIT, and to a lesser
extent for ST.
Verify all system interfaces against their design
Formally plan and prepare all tests (Plans, Scripts, Data) and preserve results, to
provide a comprehensive set of Regression Test packs, ideally automated.
Promote flexible planning and pragmatic implementation to focus on high priority
areas first.
o Plan test cycles (thread based tests) flexibly, with high priority objectives to
the fore.
o Run test cycles pragmatically, being prepared to terminate the cycle early
to progress fixes on high priority material first, and rerun in quick
succession as appropriate.
o Must have quick and easy mechanism for resetting environments (hours
not days).
Utilise Entry & Exit Criteria (and Suspension & Resumption Criteria) to provide
realistic Quality Gates for controlling testing progress (from stage to stage) on the
basis of objectives achieved rather than cycles run or arbitrary planning
dates/deadlines reached.
Pass proven systems to Independent Test area as and when each has become
sufficiently stable (Entry Criteria) for onward integration and validation - System
Validation & Integration (SV&l).
Formally validate each external interface (DIT) as a collaborative activity
Iteratively validate the emerging solution and integrate with the supporting
infrastructure, until the entire solution has been assembled and fully validated,
running End to End data flows within the bounds of the Project HNG-X solution.
(Thread based testing, multiple test cycles, progressively introduce migrated data.)
Threads should pre-empt later reuse in RV for wider End to End running.
Separate from SV&I conduct full Performance, Volume, and Stress testing
Separate from SV&I conduct full Integrity testing.
Extend End to End data flows out into external systems, and interleave with full
blown rehearsals of migration plans — Release Validation (RV)
Treat the final cycle of RV as the formal UAT
Adopt a collaborative working approach. From System Test stage onwards
establish a joint test team with the following characteristics:
o Establish a core team composed roughly 50% Fujitsu Services Testing and
50% Post Office Testing personnel.
o Use core team as the primary resource for engaging with the Requirements
and Systems Analysis team(s), and for conducting the early test planning
(see Objective Driven Testing bullet above).
o Goon to use the core team throughout the project to provide the necessary
strategic and technical direction and assurance.
o Introduce additional Post Office Testing personnel and Operational
personnel into SV&I, integrated into the Fujitsu Services managed test
teams, to both contribute business/operational knowledge, to provide extra
resource, and to gain knowledge of the new systems.
co Introduce further Post Office Testing personnel, including end users, into
RV.
©Copyright Post Office Ltd 2006 Commercial in Confidence Ref: TST/GENISTG/0002
Version: V1.0
UNCONTROLLED IF PRINTED CONTRACT CONTROLLED Date: 20/12/2006
Page No: 14 of 44
FUJ00232561
FUJ00232561
Approach to Testing Project HNG-X
(oe)
FUJITSU Commercial in Confidence
o Provide and support access for any Post Office nominated person to
independently review any required test plans, procedures, environments or
results during the testing project phases subject to commercial constraints
such as licensing and Non Disclosure Agreements (requirement reference
TST 291)
e Pass through Release Authorisation Board (RAB) into Pilot (hand system over to
Operations, Pilot run by Post Office.
The following schematic (see Figure 4) summarises the outline approach, illustrating the
lifecycle stages involved and their main areas of coverage.
©Copyright Post Office Ltd 2006 Commercial in Confidence Ref: TST/GENISTG/0002
Version: V1.0
UNCONTROLLED IF PRINTED CONTRACT CONTROLLED Date: 20/12/2006
Page No: 15 of 44
fee)
FUJITSU
Approach to testing Project HNG-X
Commercial in Confidence
FUJ00232561
FUJ00232561
POC 1 Verification
New Dev - Outsourced
PAT
Formal Spees
Acep Criteria
APOC
Architecture I
Scalability
Performance New Dev - Inhouse
Generic bl
Exhaustive
Regr Packs
TPOC Integ Teams
Stubs ete
Incl. NFRs
Processes
‘Took
DR Use Horizon Changes
Validation
Progressive Inte gration
Gradual removal of Stubs
Greater use of system generated
and Migrated Data
Trial
ST SV&r
Integrate & Integrate &
Validate each Validate whole
System, and solution with
their use of E2E data flows
supporting (within soln
Infrastructure. boundary). Full E2E Refine &
Prove structure Prove rele: data flows. Prove
& format of & migr Initial Accp. Rollout.
TFs. Include all Include all Prove Live Final Accp.
rekvant NFRs. relevant NFRs. Installation.
Rehearse
Migration /
Integrity Tests I Rollout
Load/Volume/Stress Tests I
Perf. Monitoring
Performance Modelling
Figure 4: Outline Approach
3 Goals & Objectives
3.1 Test Mission (Top level testing goals)
The following defines the mission for the overall testing and integration effort for the whole
Project HNG-X.
« Provide exhaustive yet realistic verification and validation of the entire solution prior to
Live release
e Govern the testing effort on the basis of risk, using the resulting prioritisation of the
detailed testing objectives to drive the process.
e« Trap defects as early as possible in the lifecycle, placing a heavy emphasis on
Reviews, CT, and CIT, with comprehensive (and preferably automated) Regression
Test packs.
e Demonstrate that the principal drivers of the programme have been met, by running
tests satisfying all those Acceptance Criteria specified as to be met by testing.
« Provide management information (feedback on testing progress and outcome) to the
business.
©Copyright Post Office Ltd 2006
‘Commercial in Confidence Ref:
Version:
Date:
Page No:
TST/GENISTG/0002
V1.0
20/12/2006
16 of 44
FUJ00232561
FUJ00232561
Approach to testing Project HNG-X
Fe)
FUJITSU
Commercial in Confidence
e Establish a comprehensive Performance Management regime, and in particular
conduct extensive Performance Modelling from an early stage and on an ongoing
basis.
e Develop efficient and effective testing processes, aligned with the development
methods adopted, the desire to adopt a more collaborative style of working, and the
move toward using the DR site as a test estate. Prove these, and their supporting tools
and automation, through a series TPOCs well in advance of their widespread use for
mainstream testing.
e Exploit stubs, harnesses, emulators, and the like, to provide a sound basis for
component level testing in isolation, and to enable limited integration as early as
possible in the lifecycle.
3.2 Test Motivators (Key sources determining tests)
The following defines the primary artefacts and drivers influencing the overall testing and
integration effort for the whole Project HNG-X. This should be used as a checklist of necessary
inputs for the high-level test analysis, early planning, and later more detailed test analysis,
planning and scripting activities.
* System Use Cases (SUCs) and related Models (SUCMs)
e Supplementary (non-functional) Requirements (NFRs)
e Supplementary Usability Requirement
« Acceptance Criteria
« Business Processes, Procedures, Instructions
e Training Strategy
e Release Strategy
« Implementation/Migration Plans
e¢ Topic Architecture Documents
e Physical Environment Description (PED)
« Deployment Configuration Guide (DCG)
Infrastructure High Level Design (IHLD)
e Application High Level Designs
e Analysis User Experience Model
e Logical User Interface Model
e Analysis Entity Class Model
e Logical Data model
e Use Case Realisation Sequence Diagrams
e Activity Diagrams
e State Diagrams
e Class Diagrams
‘©Copyright Post Office Ltd 2006 ‘Commercial in Confidence Ref: TST/GENISTG/0002
Version: V1.0
Date: 20/12/2006
PageNo: 17 of 44
FUJ00232561
FUJ00232561
Approach to testing Project HNG-X
2
FUJITSU
Commercial in Confidence
e SRS (composite artefact gathering elements to form a comprehensive System
Requirement Specification)
e Requirement Catalogue
e Business Rules Catalogue
e Product Catalogue
e Business Specified Test Points (BSTPs)
e Business Threads (BTs)
* Change Requests (CRs)
e Risk Register
e Programme Plan
«Delivery Plans
e Environmental Constraints
e Stakeholder Requests
« Test Reports (3 Party Systems)
e Regression Test Packs (existing or 3% Party systems)
e« Known Error Logs
e Interface Specifications (AIS and TIS)
3.3 High Level Objectives (The Principles)
The following list of high level objectives represents the fundamental principles to be observed
throughout Project HNG-X testing and integration:
e Test early — incremental testing (test what you can at any given stage/time)
e Developers are responsible for writing tests for the code they create
e Test enough (risk/cost based, value for money)
« Independent Testing Unit to integrate and validate whole solution
e Progressive integration - component level, then system level, then solution level, then
release level
e Ability to provide full requirement traceability through tests being linked to acceptance
criteria via Business Threads
e Optimise the use of automation to reduce cost and improve quality — focus primarily on
component level verification for maximum payback
e Insulate test investment from change impact — exhaustive testing, with comprehensive
regression test packs, for component level verification
« Targeted testing (avoid blanket regression)
e Test environments appropriate for tests concerned — highly synthetic for component
level verification, through to being as close to Live ‘shape’ as practical for validation
‘©Copyright Post Office Ltd 2006 ‘Commercial in Confidence Ref: TST/GENISTG/0002
Version: V1.0
Date: 20/12/2006
PageNo: 18 of 44
FUJ00232561
FUJ00232561
Approach to testing Project HNG-X
2
FUJITSU
Commercial in Confidence
and integration of the whole solution, and being of significant scale for performance,
volume, and stress testing — all live components to be trialled on target live platforms
before released to Live.
« Aim to prepare, maintain and update a test environment for the commencement of a
new iteration or instance of a testing cycle within a target of 24 hours except in the
circumstances listed below. An agreement will be made by exception in the event of
the 24 hour target being unachievable or if there is a requirement for a more rapid
reset time. a) new hardware is required, b) Fujitsu Services requires a new virtual
environment to be created and primed with a baseline, c) Fujitsu Services has applied
a major new release to a rig and rig commissioning is necessary prior to the solution
being released for testing (requirement reference TST 284)
e Consistent use of test management tools across all testing streams — exploit synergy —
reuse test materials wherever practicable
e Testing teams working collaboratively with both Post Office Testing and Fujitsu
Services Operations — combine test objectives, avoid duplication of effort, and exploit
synergy — share and reuse test materials wherever practicable
« Test management tools to be fully integrated with other software development lifecycle
tools (e.g. defect management, requirements management and change management
systems).
« Testing fully integrated with software development lifecycle — early engagement, from
Requirements Analysis onward — component level verification and system level
validation conducted within integrated mixed-discipline project teams
e Early engagement with Post Office and Fujitsu Services Operations - risk
assessments, prioritisation of test objectives
« Objective driven testing - DRIVE all testing activity on the basis of the prioritised test
objectives, so observing the underlying risk assessments
« Progress through achievement — promote from stage to stage on the basis of the test
objectives achieved, and so the ‘readiness’ of the system(s) under test to advance to
the next stage of progressive integration. (i.e. risk-based, not time driven)
e Flexible test planning — bringing higher priority material to the fore, and organising
testing cycles such that they can be run very flexibly, with rapid resetting of test
environments to enable testing cycles to be restarted quickly and economically
« Continuous reassessment/reinforcement of strategic direction — joint core team working
throughout to set and check direction and gain assurance
e Non-functional aspects included in every stage of testing
‘©Copyright Post Office Ltd 2006 ‘Commercial in Confidence Ref: TST/GENISTG/0002
Version: V1.0
Date: 20/12/2006
PageNo: 19 of 44
FUJ00232561
FUJ00232561
Approach to testing Project HNG-X
Fe)
FUJITSU
Commercial in Confidence
4 Strategic Approach
This section expands on the outline approach described in section 2.
4.1 Objective Driven Testing
Central to this strategic approach is to drive all aspects of the work (planning, setting the
priorities, engineering the necessary tests, preparing the test materials, allocating the
necessary resources, executing the tests and checking the results) based primarily on
achieving the necessary objectives in an efficient and effective fashion.
This means that all the tests have to be justified by the requirements of the release. One of the
primary objectives of the test analysis phase is to identify the required set of tests. The
importance of each test is also assessed so the most critical ones are given priority.
4.1.1. Risk Based Foundation
When considering what testing coverage must be achieved, and when each element of that
coverage must be addressed in the lifecycle, the relative Cost must be assessed against the
prevailing Risk. This Cost-v-Risk assessment is pivotal in planning and managing the
assurance and testing effort.
Business input is vital. Post Office involvement in the initial risk assessment process, coupled
with joint forums (such as PRFs) to continue providing business input on an ongoing basis, will
ensure that a balanced perspective is obtained.
4.1.2 Objective Driven Method
Objective Driven Testing is a method of performing risk assessment to govern testing effort,
routinely and repeatedly, without having to directly assess the costs and risks on each
occasion. This is achieved by recording a highly granular set of priorities based on an initial
assessment of the costs and risks, and then using these relative priorities to govern testing
effort, exploiting their implicit reflection of the underlying costs and risks.
The key is to construct a comprehensive matrix of relationships, in terms of detailed test
objectives (or ‘test points’), and using the inbuilt traceability of the test analysis and test
preparation processes to relate these test objectives to the relevant requirement, analysis, and
design artefacts.
An initial risk identification and risk assessment activity is performed, and the resulting
information is related to the test objectives via the traceability relationships, to grade their
relative priorities. As the test objectives form an integral part of the test analysis and test
preparation process, then again the inbuilt traceability associates them with the test plans,
reviews, test scripts, and test steps accordingly. So each potential unit of assurance and testing
can be implicitly cost-v-risk assessed by simply using the priority values assigned.
There are a number of additional benefits accruing from this method:
« Test Managers and Testers can relate to these priorities intuitively as they are directly
associated with the test materials they prepare and follow.
« They are more amenable to qualitative or subjective risk assessments as the relative
priorities can just as easily be set subjectively
‘©Copyright Post Office Ltd 2006 ‘Commercial in Confidence Ref: TST/GENISTG/0002
Version: V1.0
Date: 20/12/2006
PageNo: 20 of 44
FUJ00232561
FUJ00232561
Approach to testing Project HNG-X
2
FUJITSU
Commercial in Confidence
e The inbuilt traceability vastly simplifies assessing the impact of change, as any impact
analysis process will already be following those same traceability relationships.
« Since reporting of test process is driven largely by the tests being run, and these can
be directly related to the test objectives they satisfy, then it is a simple matter to report
(albeit implicitly, in terms of priorities met) on residual risk exposure as a factor of test
coverage.
@ When risks change, then simply reassigning the priorities will automatically flow
through to what tests should / should not be required.
4.1.3 The Principles
The main principles involved are:
e Analyse the available source documentation that defines the system(s) under test and
determine WHAT needs to be tested (and WHY), cataloguing this in the form of Test
Objectives, traceable back to the relevant requirements. (It is more effective to conduct
the analysis from a number of discrete perspectives - Functional, Performance,
Security, Integrity, etc. Whilst this may assist the process, for example improving target
audiences for quality reviews, it is important that these perspectives are not permitted to
influence the forward development of the tests.) Objectives must be:
v Meaningful — stated clearly and unambiguously
v Measurable — obvious what must be accomplished to satisfy
v Manageable - realistic, achievable, and not too complex
e Conduct fine grain risk assessments, engaging Fujitsu Services Operations and Post
Office as appropriate, and so prioritise the Test Objectives accordingly, effectively
indicating WHICH Test Objectives should take precedence over others. (The relative
priorities assigned then become an implicit reflection of their associated risks.)
e Group these prioritised objectives in an optimum fashion, exploiting synergy, to
accomplish multiple objectives at once, forming embryonic test plans (effectively
indicating WHEN each objective will be addressed).
e Normalise these embryonic tests based on their environmental needs, and further
combine them where appropriate to increase efficiency (effectively indicating WHERE
the test will be performed).
e Engineer these as explicit tests to achieve the objectives, documenting HOW, (and
finalising WHEN and WHERE) they will be performed, first planning the tests in detail at
the logical level (HLTP) and defining the necessary supporting data, and then scripting
them at the detailed physical level to provide clear instructions for their execution (LLTS
or equivalent automated scripts).
e Throughout, maintain the traceability back through the objectives to the source
documentation so that the rationale for each test remains clear. Ideally this is
accomplished by using a CAST (Computer Aided Software Testing) tool with an
integrated repository to document the Test Objectives, HLTP and LLTSs, with
appropriate interlinking relationships.
It should be noted that these principles apply iteratively, as further source documentation
becomes available. Typically, business requirement-based sources are available earlier and are
appropriate for early planning of solution level and release level testing, such as that performed
within the SV&l and RV stages. As the system requirements emerge these can be further
‘©Copyright Post Office Ltd 2006 ‘Commercial in Confidence Ref: TST/GENISTG/0002
Version: V1.0
Date: 20/12/2006
PageNo: 21 of 44
FUJ00232561
FUJ00232561
Approach to testing Project HNG-X
2
FUJITSU
Commercial in Confidence
refined, and work can commence on planning the system level testing, such as that performed
within the ST stage, whilst design-based sources become available a little later, and can be used
to plan the component level testing and to refine the system level testing.
4.1.4 Outline Process
The following schematic provides an overview of the way in which Objective Driven Testing is
implemented within the Project HNG-X test approach to integration and testing. It illustrates the
outline processes (Analyse, Prioritise, Optimise, Normalise, Engineer), giving an outline
breakdown of each, highlighting the main activities and data flows. It also shows the
relationship at each stage with the stated principles, which may be characterised by the simple
questions:
e WHAT to test? — the Test Objectives
e WHY test them? — the rationale, usually relating back to Requirements
e WHICH take priority? - some objectives will be more urgent/important
e WHEN to test them? — grouping them in an optimal fashion
e WHERE to test them? — assigning an appropriate environment type
« HOW to test them? — engineering the tests accordingly
©Copyright Post Office Ltd 2006 ‘Commercial in Confidence Ref: TSTIGEN/STG/0002
Version: v1.0
Date: 20/12/2006
PageNo: 22 of 44
fee)
FUJITSU
Approach to testing Project HNG-X
Commercial in Confidence
FUJ00232561
FUJ00232561
Later
Early
Sources
ANALYSE
(Set Test Objectives)
‘WHAT’
‘WHY’
PRIORITISE
‘WHICH’
CATALOGUE
OPTIMISE
(Exploit Synergy)
‘WHEN’
NORMALISE
(Consider Environments)
‘WHERE’
E
N_ Logical
G
I
N ‘HOW’
E
E
R
Physical
iterate
HL TEST ANALYSIS
Use aif
Functional P
iferent perspectives where helpful
erformance Security Integrity Etc
Conduct risk assessments and prioritise accordingly
I I Catalozue the Test Objectives 1
1 LY
eeee
PLANNING
lormalise by Environmental Needs and
Flesh out the tests into HLTPs
Env
Ww
ood
eee
LLTS & Data
Figure 5 - Objective Driven Testing — Process Flow
©Copyright Post Office Ltd 2006 ‘Commercial in Confidence Ref: TSTIGEN/STG/0002
Version: V1.0
Date: 20/12/2006
PageNo: 23 of 44
FUJ00232561
FUJ00232561
Approach to testing Project HNG-X
Fe)
FUJITSU
Commercial in Confidence
4.2 Formal Planning & Preparation
Formally documenting test plans and scripts is an obvious consequence of adopting Objective
Driven Testing (see 4.1 above), but is also essential for a number of other reasons, as follows:
* To provide traceability from the planned tests (and so the executed tests) back to the
Requirements, as required by CMMI, and as necessary in order to conduct coverage
analysis, and assess residual risk
e To make the tests reusable (re-runable) and so protect the considerable investment
involved in engineering a correct set of tests for a large and complex solution such as
Project HNG-X
e To enable impact analysis (and so targeted testing) based on changes to certain
identified components, using the traceability provided by the formal plans and scripts
e To provide an ongoing, and easily understood and amendable testing infrastructure for
future use
e To allow the creation (collectively) of comprehensive Regression Test packs
As implied in 4.1 above, the steps involved are:
e The prioritised test objectives (derived from the principal sources) are grouped into
logical and efficient sets, optimising them to exploit synergy, and so producing
embryonic test plans.
e Their particular environmental requirements/constraints are then considered,
normalising them accordingly (effectively allocating them to the most appropriate and
cost effective testing stage) ....
and fleshed out further to form High Level Test Plans (HLTPs)
« These are at a logical level and so should remain independent of their implementation
(e.g. any tools/automation used) and so form a valuable and enduring test asset
e Low Level Test Scripts (LLTSs) are generated from the HLTPs, effectively
physicalising the tests and providing detailed instructions for their execution. (LLTSs
may take the form of automated test scripts, where applicable.)
4.3 Progressive Integration
In common with all large and/or complex IT solutions, it would be unrealistic to expect to
successfully integrate the very great number of individual components making up Project HNG-
X, to form a coherent overall solution, all in a single step. A progressive approach is called for.
This entails a series of integration steps, each building on the added stability provided by the
previous one. At each step it is important to have a solid, stable foundation to start from. This
is provided by verification/validation activity conducted at that level of maturity. The integration
step then takes the ‘proven’ elements and exercises them in combination to integrate them to
the next level, ready for further verification/validation.
These verification/validation activities, and the progressive integration steps, are arranged in a
series of testing stages. There are four universally recognised levels of integration —
Component Level, System Level, Solution Level (where the solution involves multiple
systems), and Release Level.
The starting point is the individual Component (the smallest meaningful deliverable, of
whatever type, forming a part of the target runtime state). These must first be verified before
they can be successfully integrated together to form small sub-assemblies, which in turn must
be verified. These sub-assemblies can then be integrated to the next level (System Level) and
validated to confirm they are conforming to the relevant system requirements. Again this then
forms the appropriate stable foundation to integrate to the next level (Solution Level), bringing
all the co-operating systems together and validating them as an overall solution.
‘©Copyright Post Office Ltd 2006 ‘Commercial in Confidence Ref: TST/GENISTG/0002
Version: V1.0
Date: 20/12/2006
Page No: 24 of 44
FUJ00232561
FUJ00232561
Approach to testing Project HNG-X
2
FUJITSU
Commercial in Confidence
Finally (at the Release Level) the solution can be integrated with its deployment mechanisms
and the target Live runtime environment, to prepare it for Go-Live. (The deployment
mechanisms should not be considered as incidental. They are an important aspect of the
solution design, and whether implemented in an automated fashion, or by a manual process,
form a vital element for successful deployment.)
The stages of testing required for Project HNG-X, shown in relation to these 4 levels of
integration and verification/validation can be seen in more detail at 4.4 below.
4.4 Stages & Types of Testing & Integration
4.4.1 Addressing the Issues
As Project HNG-X involves a substantial re-engineering of the Horizon Application solution, it
offers the ideal opportunity to redress this expensive imbalance in the testing lifecycle,
applying rigorous verification methods to exploit the benefits of any changes to the
development methods". It also offers the opportunity to attack the areas of duplication and to
exploit the areas of synergy, merging many of the existing testing stages together and working
more collaboratively.
4.4.1.1 Component Level
The existing rather lightweight UT activities are replaced by a new much more rigorous regime.
Code reviews will be retained, but given a higher profile than has been common for Horizon
Application, seeking to trap code defects at the earliest opportunity. Adopting the usual industry
terminology for component developments, Module Test (MT) and Link Test (LT) are replaced
by Component Test (CT) and Component Integration Test (CIT). The new terminology is
deliberate, to emphasise the fact that this must be a very different activity to that currently
performed. The important characteristics to emphasise are:
Conducted within development (with some support from system test team)
Exhaustive coverage, made possible by strictly limiting the scope
Object-Based (Encapsulation)
Generic basis of testing (synthetic data sensitising full breadth of stimuli)
Verify against Design
Include Relevant non-functional requirements (NFRs)
Comprehensive Regression Packs (insulate wider solution from localised changes)
Extensive use of Stubs, Harnesses, etc. to enable testing in isolation
Note this applies to the new Project HNG-X components; it does not apply to those Horizon Application
components that are taken into Project HNG-X unchanged or with some minor change.
‘©Copyright Post Office Ltd 2006 ‘Commercial in Confidence Ref: TST/GENISTG/0002
Version: V1.0
Date: 20/12/2006
PageNo: 25 of 44
FUJ00232561
FUJ00232561
Approach to testing Project HNG-X
Fe)
FUJITSU
Commercial in Confidence
Component Testing (CT): =~
Individual component “Z’ tested in isolation
Stubs used at each interface
ar ‘ay
Figure 6: Component Test Context
Component Integration Test (CIT): gw) Cx
Component “z” integrated and re-verified
together with immediate neighbours, but no 2
further. gy > <p)
Figure 7: Component Integration Test Context
4.4.1.2 System level
The important characteristics to emphasise in this phase are:
Conducted within development (integrated project teams)
Detailed coverage, made possible by strictly limiting the scope
Thread based tests (scenarios) integrate components into discrete systems
Largely synthetic data to improve sensitisation of tests
Validates against system requirements
Integrates applications with supporting infrastructure elements
Verifies the system interfaces against the interface specifications
Includes relevant non-functional requirements (NFRs)
Sample Regression Packs (insulate wider solution from localised changes)
Some continuing use of Stubs, Harnesses, etc. to enable testing in isolation and to
avoid schedule impact through interdependence
Test and not development environments used”
e Entities delivered, installed and managed by the target solution architecture.
a
4.4.1.3 Solution Level
Retain the existing SV&l nomenclature — Solution Validation and Integration. Focus attention
on gradual integration of whole solution. It includes all aspects of acceptance testing (to satisfy
the acceptance criteria specified as to be met by solution testing (ST) and not allocated to RV —
see 4.4.1.4). The important characteristics to emphasise are:
e Conducted within independent testing unit
2 In practice this means Systems testing is run on the test environments in the live data centres whereas
development testing is run on environments local to the development teams, e.g. BRAO1, CRWO1 etc.
‘©Copyright Post Office Ltd 2006 ‘Commercial in Confidence Ref: TST/GENISTG/0002
Version: V1.0
Date: 20/12/2006
PageNo: 26 of 44
Fe)
FUJITSU
FUJ00232561
FUJ00232561
Approach to testing Project HNG-X
Commercial in Confidence
Broad and shallow coverage - shallow here does not imply weak or superficial, but
that at this level of validation the test coverage is not intended to be exhaustive, i.e.
trying to cover every logic path in every combination. (Such detailed testing is
performed at the earlier levels) Here important functional paths through the solution
are used to confirm it is properly integrated and that it delivers to the overall
requirements.
Thread based tests (scenarios) integrate systems into whole solution
Largely system generated data to confirm data flow integrity
Validates against business / operational requirements
Integrates applications with supporting infrastructure elements
Validates Systems Management elements
Formally validates the external interfaces against the interface specifications
ccc ee
Increasing level of collaborative working
Includes relevant non-functional requirements (NFRs)
Incorporates Integrity Tests (resilience/recovery)
Incorporates Performance Tests (load, volume, stress)
Threads incorporate all BTs and BSTPs to pre-empt acceptance
Tests planned and prepared for subsequent reuse in RV (i.e. cover both POA and
Post Office objectives and so encompass all the goals that E2E previously had)
though at this stage not operating beyond the Project HNG-X solution boundary (i.e.
external systems used only for validating interfaces and not for data flow integrity)
Sample Regression Packs (insulate wider solution from localised changes)
Use of Stubs, Harnesses, etc. removed except in support of interface testing and
performance testing
Test and not development environments used
Entities delivered, installed and managed by the target solution architecture.
Note - For Project HNG-X, with the extensive changes to the underlying infrastructure systems,
it will not be practicable to operate Integrity Tests or Performance Tests to a sufficient extent
from within SV&I (the integrity tests will be too intrusive and disruptive, and the performance
tests will need to be operated at very large scale). These will therefore operate as discrete
testing stages for Project HNG-X.
4.4.1.4 Release Level
The RV nomenclature is retained — Release Validation. However, the scope is expanded to
also encompass all the earlier Post Office Testing stages — Accreditation, E2E, Field-Trial, Pre-
Pilot testing. This is achieved by combining the POA and Post Office test objectives (see the
description of Objective Driven Testing at 4.1 above) into a single stream of scenario based
tests, run
interleaved with Migration tests/rehearsals, and so including all aspects of
Acceptance testing (to satisfy the Acceptance Criteria specified as to be met by testing). The
important characteristics to emphasise are:
Conducted within independent testing unit
Broad and shallow coverage of business functionality (see note on equivalent item
under 4.4.1.3 above), with full rehearsal of migration and deployment processes
« Thread based tests (scenarios) interleaved with Migration tests/rehearsals
e Reuses test materials prepared for SV&l
e Incorporates all BTs and BSTPs and so achieves Acceptance testing goals
e Entirely system generated data to confirm data flow integrity
e Validates against business / operational requirements (which for the migration
‘opyright Post Office Ltd 2006 ‘Commercial in Confidence Ref: TST/GEN/STG/0002
Version v1.0
Date: 20/12/2006
PageNo: 27 of 44
2
FUJITSU
FUJ00232561
FUJ00232561
Approach to testing Project HNG-X
Commercial in Confidence
erccee
Integrates Solution with deployment mechanisms, including migration
Further validates Systems Management elements
Reuses interface validation materials toward achieving Accreditation. (It is important
to recognise that if a particular area of Accreditation demands multiple re-runs, or
the synchronisation it requires with the third party cannot be organised flexibly
(perhaps having to be ‘booked’ a long time in advance) then this will clash with
scheduling the rest of the testing for the RV area, and so in such cases that piece of
Accreditation would need to be separated out from the rest to avoid adverse
impact.)
Very high level of collaborative working
Sample Regression Packs
All Stubs, Harnesses, etc. removed
Test and not development environments used
Entities delivered, installed and managed by the target solution architecture.
44.2 Project HNG-X Testing Stages
So, to summarise, the following stages of testing are required for the Project HNG-X, mapping
them onto the recognised levels of integration:
.
Proof of Concept
o Architectural Proof of Concept (APOC)
o Testing Proof of Concept (TPOC)
Component Level
o Code Review (CR)
o Component Test (CT)
o Component Integration Test (CIT)
System Level
o System Test (ST)
Solution Level
o Solution Validation & Integration (SV&I)
Release Level
o Release Validation (RV)
The following schematic illustrates how these testing stages will in practice overlap in time (for
example, not all components are required to enable one system to enter ST, and not all
systems are required to enable integration of systems to commence in SV&l).
‘©Copyright Post Office Ltd 2006 ‘Commercial in Confidence Ref: TST/GENISTG/0002
Version: V1.0
Date: 20/12/2006
PageNo: 28 of 44
FUJ00232561
FUJ00232561
Approach to testing Project HNG-X
2
FUJITSU
Commercial in Confidence
° =
Z aa Code Review
Component Test Component
: Level
ie Component Integration Test
PAT
ee System
System Test Level
ST sae se eee se sees
Solution Validation & Integration Solution
Level
SVal = .
Release Validation Release
Level
RV
Pilot
Figure 8: Testing Stages — Overlap
Product Acceptance Test (PAT) is retained for third party deliverables, to confirm that such
deliverables meet the necessary minimum levels of stability of functional conformance so as
not to adversely impact the wider testing activities taking place. PAT may not be a physical test -
it may be by a review of artefacts such as low level design documentation or review of code
where a physical test is not practical.
In addition, as already explained, for Project HNG-X it will be necessary to conduct both
Integrity Testing (Recovery & Resilience) and Performance Testing (Load, Volume, and Stress)
separately from SV&I, as discrete testing stages. Also, each testing stage requires formal
planning and preparation, and as already mentioned there will also be Proof of Concept
exercises run at the outset. Performance Modelling will commence alongside the APOC, and
continue throughout. Complementary Performance Monitoring will be trialled in RV and then
run on an ongoing basis thereafter (to support Monitor acceptance method requirements). The
following schematic illustrates:
‘©Copyright Post Office Ltd 2006 ‘Commercial in Confidence Ref: TST/GENISTG/0002
Version: V1.0
Date: 20/12/2006
PageNo: 29 of 44
FUJ00232561
FUJ00232561
Approach to testing Project HNG-X
2
FUJITSU
Commercial in Confidence
Figure 9: Project HNG-X Testing Lifecycle Overview
4.5 Regression Testing & Retesting
The regression testing and re-testing policy for Project HNG-X is as follows:
The policy for Regression Testing for Project HNG-X is that for all change(s) being made to the
solution, large or small, simple or complex, a sufficient series of Regression Tests will be run to
confirm that (on the balance of probability, and commensurate with the risks involved) those
change(s) have not caused the solution as a whole to regress, neither functionally nor non-
functionally.
This will be achieved by:
e Strong design separation - this testing approach is predicated on Design &
Development adhering to essential component-based development and O-O principles,
such that the newly developed areas of Project HNG-X will be inherently more resistant
to invasive change impact. This will be reinforced by Design Walkthrough and code
reviews, and verified by Component Test (CT), and Component Integration Test (CIT).
Change impact will therefore be much more localised than is currently the case with
the Horizon Application Solution.
‘©Copyright Post Office Ltd 2006 ‘Commercial in Confidence Ref: TST/GENISTG/0002
Version: V1.0
Date: 20/12/2006
PageNo: 30 of 44
FUJ00232561
FUJ00232561
Approach to testing Project HNG-X
Fe)
FUJITSU
Commercial in Confidence
« CT and CIT test coverage will be exhaustive, and generic in nature, to ensure the
design capabilities of each component (and its interfaces) are fully met.
e CT and CIT materials will be retained to form comprehensive packs of Regression
Tests at the component level.
« Extended Configuration Management (CM) and Impact Analysis facilities providing
change impact details to allow specific changed components, and their neighbouring
components, to be targeted for testing
e Related areas of the solution will similarly be identified as requiring specific regression
testing at the component level
« The change documents and related sources will be subject to test analysis in the
normal fashion, adopting Objective Driven Testing (ODT) principles to determine the
basis of risk and so the relative priorities for testing each affected component and
neighbouring components, and other areas likely to be affected as indicated by Impact
Analysis.
« More general regression testing to confirm the continuing operation of the system(s)
concerned and the overall solution will take the form of sample regression tests,
selected on the basis of their priority rating(s)
e In general, the smaller, simpler, and more localised the changes are, then the less
regression testing will be required, and conversely the larger, more complex, and more
invasive the changes are, then the more regression testing will be required.
4.6 Horizon Application Based Areas
As explained previously, Horizon Application will persist (and so will need to continue to be
maintained in Production) until Project HNG-X rollout is completed®. The existing processes in
use on Horizon Application have been developed and refined over many years to cover the
circumstances of the Horizon Application solution, and these will continue to be used for as a
basis for all Horizon Application work that may be required in parallel with the Project HNG-X.
Horizon Application also persists within the target Project HNG-X solution (the Host systems,
many of the Agents, and certain areas of infrastructure and Systems Management). It is likely
that the work involved in these areas, albeit within the Project HNG-X, will draw upon common
(shared) Horizon Application based resources. It is not really practical therefore to adopt
different processes for these Project HNG-X areas than will be in use for Horizon Application.
So, to avoid the clash, Project HNG-X will adopt the existing Horizon Application processes for
all such areas, for all verification, validation, and integration activities up to and including the
System Level. The two process streams will then be fused at the point where the systems
concerned enter SV&I, where the Project HNG-X processes will prevail. (This will not cause a
clash in process as this testing stage (and all subsequent ones) will be performed outside
development, in the independent testing unit.)
Just one exception is imposed by this approach — the Horizon processes will be slightly
extended to include creation of regression testing packs for Unit (UT) Test and Extended Link
Test (ELT)*. This is necessary to complete the layer of insulation to protect the wider solution
from localised changes.
Note the continued operation of Horizon during HNG-X Migration is dependent on the Migration
Strategy (see [13].
‘ It is likely these regression test packs will be assemblies of existing material.
‘©Copyright Post Office Ltd 2006 ‘Commercial in Confidence Ref: TST/GENISTG/0002
Version: V1.0
Date: 20/12/2006
PageNo: 31 of 44
FUJ00232561
FUJ00232561
Approach to testing Project HNG-X
2
FUJITSU
Commercial in Confidence
4.6.1 Iterative Lifecycles and Integrated Mixed-Discipline Project
Teams
Project HNG-X will utilise iterative development techniques as and where they may be of
benefit. Depending on the circumstances the particular brand of iterative development may
vary.
The following schematic illustrates the underlying iterative testing process for all development
lifecycles:
Analyse Plan
Test
Sources y Analysis
Revise Prepare
Repeat Schedule
Verification
Validation
Integration
Evaluate Execute
Figure 10: Testing Lifecycle — Always Iterative
Where this approach does affect testing is in the implementation and management. Iterative
development processes always benefit from being organised as small close-working project
teams, working together across the whole of the relevant portion of the development lifecycle
(including testing). For Fully iterative developments this would apply to the entire lifecycle. For
Part-Iterative developments, such as is for Project HNG-X, the relevant portion of the lifecycle
is that portion which behaves iteratively. So, from a testing and integration perspective, it
means those areas of Testing that are performed within the Development zone of responsibility
(i.e. Component and Component Integration) and not the areas handled by the independent
testing unit (i.e. the System, Solution and Release Levels).
So, for Project HNG-X, this approach requires CR, CT and CIT to be conducted within
Development, by a mixed-discipline project which encompasses appropriate testing expertise.
It is also likely that early periods of ST will also benefit from mixed-discipline working, i.e.
developers and testers (or more correctly development orientated testers).
‘©Copyright Post Office Ltd 2006 ‘Commercial in Confidence Ref: TST/GENISTG/0002
Version: V1.0
Date: 20/12/2006
PageNo: 320f 44
FUJ00232561
FUJ00232561
Approach to testing Project HNG-X
2
FUJITSU
Commercial in Confidence
4.7 Parallel Testing
TDN. This section is likely to require revision when the Migration Strategy has been baselin
The Project HNG-X solution presents a number of unusual characteristics, which when taken in
combination offer the opportunity to adopt an extremely cost effective testing technique known
as Parallel Testing, and Project HNG-X intends to exploit this opportunity.
The key characteristics of the solution relevant here are:
e The Project HNG-X is predominantly a re-engineering exercise, and for the vast
majority of the functionality it is seeking to achieve ‘Business Equivalence’ (i.e. very
little new or changed business functionality involved).
e The external interfaces remain entirely unchanged.
* The majority of the external interfaces are file based and batch driven, and the Host
systems responsible for generating these interface files also remain unchanged?.
« Although the transaction volumes processed by the systems are very large, and the
system data transformations and relationships are complex, this results in a relatively
low volume of interface data.
e All the interface data is determined as a direct result of the content of the central
transaction store (the Riposte Messages in the Correspondence Servers for Horizon
Application, and the tables/rows of the Branch Database for Project HNG-X.
e There will be a clearly defined migration mechanism to transform the Riposte
Messages into equivalent transaction information on the Branch Database
Conducting equivalent business on both Horizon Application and Project HNG-X will
result in equivalent interface records being produced on both.
So, simply by running a parallel test on the new Project HNG-X solution (parallel to the live
operation of Horizon Application in terms of the data present, the transactions conducted, and
the interface files generated), then the enormous complexity of the internal processing involved
in the two solutions can be reduced to the net resultant interface files produced. The bottom
line is that it dramatically reduces the test planning and test checking required to confirm
overall non-regression of the solution.
In this case then, it would involve:
« Selecting a well delineated period of live operation of the Horizon Application solution.
Using the EOD position on a given business day is a convenient method — effectively a
business day’s worth of transactions. If one of the four correspondence servers is
selected, approximately 25% of a daily workload, providing a highly representative
spread of transaction types. Alternatively specific branches could be targeted, though
naturally the smaller the sample dataset the smaller the range of different business
circumstances would included in the test.
e Take a backup of the selected correspondence server(s).
e Allow all that night’s batch processing to proceed, producing the interface files
corresponding to those transactions, and take a backup of them also.
However it is not intended to downplay the importance of the real-time interfaces to payment card
systems etc.
‘©Copyright Post Office Ltd 2006 ‘Commercial in Confidence Ref: TST/GENISTG/0002
Version: V1.0
Date: 20/12/2006
PageNo: 33 of 44
2
FUJITSU
FUJ00232561
FUJ00232561
Approach to testing Project HNG-X
Commercial in Confidence
Transfer the backups into a test environment (special authorisation would be required
to make use of this Live data for testing purposes).
Migrate the selected transaction data onto the Branch Database of an Project HNG-X
test environment, and run that night's batch processes.
Secure the resulting interface files, and compare them with those produced by the
Horizon Application Solution. Certain physical differences would be expected (such as
time stamp differences and the random chunking of the TPS files) but in business
terms they should be identical.
merce Fes (J mr
‘Compare
Business Outcome
oe ae
ost ytens Nos sytem
Migrate selected
transaction store
coven ne
Sonu ask
HNG-X Solution - Test
Horizon Solution - Live
comerveraetes
Figure 11: Parallel Testing for Project HNG-X
4.8 Collaborative Working
Fundamental to this strategic approach is the adoption of fully collaborative working between
POA and Post Office for broad range activities spanning every aspect of testing and integration
for the Project HNG-X. This extends well beyond simple witnessing of POA tests by Post
Office, and even joint teams executing certain tests together. Essential to it is a lifecycle long
relationship to jointly:
set the strategic direction (this document);
engage with the requirements analysis activities to ensure testability and to commence
the high level test analysis;
work with the requirements analysis area to help produce meaningful and practical
acceptance criteria in the form of test scenarios (BTs and BSTPs);
‘©Copyright Post Office Ltd 2006 ‘Commercial in Confidence Ref: TST/GENISTG/0002
Version: V1.0
Date: 20/12/2006
PageNo: 34 of 44
FUJ00232561
FUJ00232561
Approach to testing Project HNG-X
Fe)
FUJITSU
Commercial in Confidence
e establish a combined set of test objectives covering all the needs of both parties,
engaging with both the business and operations to prioritise them on the basis of
business and operational risk;
e iteratively expand the catalogue of test objectives as the project lifecycle progresses
and additional information emerges, such as by engaging with the system designers to
ensure testability of all components, including setting the testability framework, and
engaging with requirements and design personnel in assessing changes;
e periodically reassess the basis of risk, consulting with the appropriate business and
operational representatives to adjust the related priorities as necessary;
* continue to monitor and adjust the strategic direction and gain assurance throughout
the programme;
« plan and prepare test materials for shared use and re-use;
e eliminate duplicated effort, exploit synergy and merge separate testing activities
together wherever practicable to streamline the overall testing lifecycle;
* execute some areas within the SV&I stage, including formal validation of the external
interfaces;
« execute almost all areas of the RV stage, embracing all previous objectives of the
existing Post Office Testing activities, including Accreditation, Acceptance Test, and
Pre-Pilot Test;
* report testing progress, including the contribution toward Acceptance
The intended implementation for Project HNG-X is to establish a Core Team, comprising
roughly 50% POA personnel and 50 % Post Office personnel. This CORE team is initially
involved in setting the strategic direction for Project HNG-X testing and integration (i.e.
producing this document), and then working with the Requirements Analysis areas (for both
business and operational requirements), following the principles of Object Driven Testing as
described in outline at section 2 above, and in more detail at section 4.1 above.
This team would comprise exclusively highly experienced senior testing personnel, and would
form a line of continuity throughout the whole lifecycle of the programme, setting, monitoring,
and refining the strategic approach, assisting and advising the testing teams, and continuously
gaining assurance.
Additional Post Office resources would later be injected into the testing teams, for the test
planning and preparation activities, and again for RV test execution (including a contingent of
End Users to assist in validating Procedures, Instructions, and Training materials, and to help
man the Model Office environment.
The roles and responsibilities involved are indicated at section 5 below.
4.9 Defect Management
This area will be actively developed as part of the Testing Proof of Concept (TPOC) exercise.
However, at a strategic level the key points are:
« All defects identified by formal testing will be recorded
« Where defects are found in component level testing, conducted by developers, they
need not initially be raised as defects in the central defect management system, as it is
most likely the same developer who will have to diagnose/fix the bug, and using the
defect management tool would impose an unnecessary overhead. However, they will
be noted in a defect log for inclusion in later metrics (they may for example help
‘©Copyright Post Office Ltd 2006 ‘Commercial in Confidence Ref: TST/GENISTG/0002
Version: V1.0
Date: 20/12/2006
PageNo: 35 of 44
FUJ00232561
FUJ00232561
Approach to testing Project HNG-X
Fe)
FUJITSU
Commercial in Confidence
identify issues in a particular area of the design, or leakage occurring through
inadequate code reviews).
e Where defects are found in later stages of testing, they will be recorded in the central
defect management system, in all cases.
e Defects will be related to the test stage/plan/script/step/environment concerned, which
through traceability will in turn relate to the test objective(s) and
requirement(s)/design(s) concerned. After diagnosis this will also be related to
system/component.
e Defects will be assigned priorities/urgencies/importance on the basis of the priority of
the related test objective(s), the technical criticality of the system/component, the
potential business impact, the potential testing impact
(cost/schedule/coverage/dependency), any acceptance or release authorisation
consequences, and also perhaps the defect density in that area (if it has become an
issue).
« Outstanding defects will be formally reviewed on a regular basis (frequency driven by
need — monthly, weekly, daily) by a Problem Review Forum (PRF) to assess/re-assess
priority and likely impact. The PRF will have business, operations, testing, design and
development representation, working collaboratively.
« Defect rates will be actively tracked as an integral part of test management
TBC when the tooling strategy has been determined (comes out of the TPOC exercise)
4.10 Environments, Tools & Automation
Fujitsu Services shall provide maintenance and support levels for the Project HNG-X test
environments during testing as described in the HNG-X Test Strategy [13].
The ongoing requirement is to provide a continuously operational support environment in the
DR centre for testing immediate fixes to the live service. In addition there is a requirement to
provide a capability to be able to instantiate one or more test environments, as agreed,
additional to and concurrent with the support environment, for the development and validation
of new business services to the solution. The parameters for exercising the capability to
instantiate test environments will be as defined in a post Project HNG-X testing approach CCD
(requirement reference TST 294).
Project HNG-X environments is an area will be actively developed as part of the Testing Proof
of Concept exercise. However, at a strategic level the key points are:
e Environments
o Project HNG-X will have three main sets of test equipment available.
= Existing test estate currently used for Horizon Application, increasingly
becoming available as the Horizon Application testing usage
diminishes over time (this is not planned to be retained beyond Project
HNG-X)
= New Project HNG-X data centre equipment to be installed well in
advance at the “Active” site (this will of course cease to be available
for testing use at the time it is required to start preparing for Data
Centre Migration)
= New Project HNG-X data centre equipment to be installed well in
advance at the “Standby” site (this will remain for testing use
throughout, except when DR is invoked or rehearsed)
o Mechanisms will be developed to flexibly deploy the equipment at the data
centres, enabling:
‘©Copyright Post Office Ltd 2006 ‘Commercial in Confidence Ref: TST/GENISTG/0002
Version: V1.0
Date: 20/12/2006
PageNo: 36 of 44
FUJ00232561
FUJ00232561
Approach to testing Project HNG-X
2
FUJITSU
Commercial in Confidence
= Multiple parallel environments capable of supporting multiple parallel
testing streams®
= Rapid and reliable reconfiguration and redeployment to form varying
numbers of different shaped environments as the testing lifecycle
unfolds
= Easy taking of ‘checkpoint’ positions before, during, and after testing
cycles (configuration, code set baseline, data)
= Rapid and reliable resetting of environments to nominated ‘checkpoint’
position
= Separate data storage provision for Live and Test
= Capability to switch (under appropriate controls) between Live and
Test datasets/networks/keys (to support pre-production testing and
parallel testing methods)
= Failsafe isolation of Standby site equipment and network which is in
use for testing (external interfaces, Active site synchronisation, etc.)
= Practicable remote management facilities — configure, install, regress,
time & date manipulation, systems management, batch scheduling,
etc.
o Range of different environment types configurable
= Full DR
= Large-scale, Live shaped (e.g. for Performance, Stress, Resilience)
= Medium-scale, Live shaped (e.g. Acceptance, Migration)
= Medium-scale Live shaped (e.g. Live System Test)
= Small-scale, Representative shape (e.g. SV&l)
= Small-scale, Synthetic (e.g. ST, Interfaces)
o Component level testing will be performed on development equipment
° Tools
o There is no drive on Project HNG-X to replace tooling across the programme in
support of new development methods, as was the case in earlier proposals — in
general it should be assumed that the existing tools will continue to be used
unless the nature of the task concerned is new or dramatically changes, or
there is some other compelling reason to adopt a different toolset.
= PVCS’ for CM, but meta model extended to provide necessary
structure in support of any new development methods, and to provide
impact analysis
TestDirector® for test planning and test management
PEAK for defect management
LoadRunner for performance testing
Win Runner and QuickTest Professional? for functional test automation
New tools for Component level testing (JTest, JUnit, etc.)
Note the ability to achieve this objective will be bounded by the Migration Strategy, i.e. the ability to cost
effectively run parallel instances of Horizon components during migration testing. A number of Horizon
platforms cannot be ‘virtualized’ due to the presence of Atalla Cryptographic cards.
It is planned to upgrade to a later version, Dimensions V9.0 — product name changed as the result of
change of ownership.
8 Full name is: Test Director for Quality Center.
° WinRunner will be used on NT based Counters, Quick Test Professional for XP Counters. These are
essentially the same product rename and packaged.
‘©Copyright Post Office Ltd 2006 ‘Commercial in Confidence Ref: TST/GENISTG/0002
Version: V1.0
Date: 20/12/2006
PageNo: 37 of 44
FUJ00232561
FUJ00232561
Approach to testing Project HNG-X
2
FUJITSU
Commercial in Confidence
The usage/requirement for each tool will decided as part of the TPOC
exercise, ensuring that they join up, an the associated processes are properly
aligned.
o Fujitsu Services will provide training for Post Office testing staff to enable use
of any new testing tools that are provided by Fujitsu Services (requirement
TST 286)
o There will be a need to adopt consistent working practices in the areas
employing collaborative working. Typically this will mean standardising on the
tools used by Fujitsu Services testing. (For example, it would be sensible to
use a single defect management tool for all testing.)
« Automation
o The testing automation policy for Project HNG-X can be summarised as:
automated testing techniques will be adopted wherever and whenever it
provides a clear benefit. The adoption must therefore be pragmatic, embracing
automation where it can be readily applied, and avoiding it where costs
become prohibitive. The detailed test automation strategy will need to be
developed, but it is clear that the primary focus should be in the early
verification stages (these are small, self contained, and have fewer data
dependencies to contend with, but require exhaustive coverage, making them
ideal candidates for automation). Automation of the later stages of testing
should be approach more critically, adopting automated scripts only where the
benefit is clear, and where the cost of future maintenance of scripts will not
escalate. (Sample coverage to support minimum regression testing would
seem appropriate.)
TBC when the environment and automation strategy has been determined (comes out of the
TPOC exercise)
4.11 Metrics
TDN. This material must be reviewed by Pete Dreweatt.
This is an area which must be closely aligned with the associated tools and automation (see
4.10). In order to avoid undue overheads, it is important to generate any metrics that are
required as an integral part of the underlying process and wherever possible do so
automatically and transparently. Their precise definition is therefore inextricably entwined with
that of the tools and automation being used. However, at a strategic level, the metrics that
must be collected are:
e Time and Effort against Plan
o Manpower resource expenditure by activity/task/skill — informs current budget
position and future manpower estimating
o Activity/Task progress tracking against project plan — informs current schedule
position and future timescale estimating
o The standard POA planning and time recording processes will be followed
e Testing Progress / Product Quality
o Test Objectives derived, by nature/priority
o Test Plans/Scripts/Steps produced, by stage/priority
co Test Plans/Scripts/Steps executed — Success/Fail, by stage/priority
o Test Coverage Achieved — by Test Objectives covered
o Defects - Raised, Closed, False, Outstanding - by priority and by
component/area, over time, traceable to Test Plan/Script/Step and to Test
Objective
©Copyright Post Office Ltd 2006 ‘Commercial in Confidence Ref. TST/GENISTG/0002
Version: V1.0
Date: 20/12/2006
PageNo: 38 of 44
FUJ00232561
FUJ00232561
Approach to testing Project HNG-X
2
FUJITSU
Commercial in Confidence
o Handovers Processed — by component/area, over time
e Process Improvement
o Root Cause Analysis and Defect Density — by component/area
o Re-Handovers — by component/area
o Defect Leakage — by component/area and by test stage
o PONC (Price of non-conformance)
TBC when the environment and automation strategy has been determined (comes out of the
TPOC exercise)
4.12 Process Improvement
Process improvement will be an integral part of the Project HNG-X way of working for all
testing and integration activities. As such there will be a post stage review for each stage, and
a post release review for each release, specifically to examine the performance and
achievement of the testing and integration activities in that stage/release, highlighting any
potential process deficiencies and recommending corrective action/process improvement
accordingly.
©Copyright Post Office Ltd 2006 ‘Commercial in Confidence Ref. TST/GENISTG/0002
Version: V1.0
Date: 20/12/2006
PageNo: 39 of 44
FUJ00232561
FUJ00232561
Approach to testing Project HNG-X
fee)
FUJITSU
Commercial in Confidence
5 People, Skills & Responsibilities
The following table indicates the primary roles and responsibilities applicable to each of the
major activities. Jointly means that the Post Office and Fujitsu Services test managers jointly
take responsibility for delivery of the activity.
Key: Agr Mai
P = Post Office, F = Fujitsu Services, J = Jointly Si Re
(+3P = together with the 3 party concerned)
o3<€0
Off urc
Activity
F7OSsveaHaSHDES
Orr 0TTENH
VPA~EOQ-~HVt00O72-—D
Programme Test Strategy
Involvement in Business Requirements/Accp. Criteria
Involvement in Non-functional Requirements/Accp. Criteria
Involvement in Operations Requirements/Accp. Criteria
Derive Test Objectives
Risk Assessments — establish Priorities
ef ef ec} ce} ep ef e
Outline Test Planning
CT
CIT
ST
nl ml om) mf ce} ce] cf} cf ce) che
al nl ml mf ec] cf ce} ce) ch che
SV&l - Main
SV&I — External Interfaces
a
F
@
Uv
F+3P
SV&I - Acceptance aspects
Performance Testing
Integrity Testing
Security Testing
DR Proving / Rehearsal
RV — Migration aspects
vlc} ml mom) om] mo) mom] om] mom) om] a] om] om) oD
nlm} I 0} 0] vl] DU) Dl
vlc} a) mom) mom
RV - E2E aspects
mn} oo} cl mom) om) mf om) mom] om om me} cf ec] ac} oe
ec} ef cf ce} ec} ec] cf ce) eC} ce} ec] mf mf ec] Cf Cc] ce) Ch che
vc} vp) ppc) cl ppc
a
RV — Deployment aspects F
v
‘©Copyright Post Office Ltd 2006 ‘Commercial in Confidence Ref: TST/GENISTG/0002
Version: V1.0
Date: 20/12/2006
Page No: 40 of 44
oo
FUJITSU
Approach to testing Project HNG-X
Commercial in Confidence
FUJ00232561
FUJ00232561
RV - Accreditation aspects P J [P P+3P I J+3P I F
RV - Acceptance aspects P J IP P P F
RAB P Ig fp [P P F
Pilot P fs fe {Pe P F
©Copyright Post Office Ltd 2006 ‘Commercial in Confidence Ref. TSTIGEN/STG/0002
Version: V1.0
Date: 20/12/2006
Page No
41 of 44
FUJ00232561
FUJ00232561
Approach to testing Project HNG-X
Fe)
FUJITSU
Commercial in Confidence
6 Assumptions, Dependencies, Risks & Constraints
6.1 Assumptions
« The primary driver for the Project HNG-X is to reduce the total cost of ownership.
Whilst it is desirable in so doing to also improve the responsiveness to future change
(reduced Time to Market, and reduced cost of delivery) these are not drivers for the
Project HNG-X and can only be pursued on a cost neutral basis. Accordingly, this
approach is not optimised for future change, but does exploit opportunities to benefit
this area where they occur in satisfying the programme drivers. (e.g. stipulating the
need for generic and exhaustive component level testing, and the need to retain
comprehensive regression test packs, will require significant investment, which whilst it
will be recouped within Project HNG-X is unlikely to realise significant savings overall,
but will benefit future changes)
« Both POA and Post Office desire a single, consistent, joined-up testing approach for
Project HNG-X.
« Both POA and Post Office desire, and are prepared to support and resource, fully
collaborative working for the testing of Project HNG-X, wherever this may result in
reduced duplication of effort between the parties, and/or exploit synergy to improve
efficiency and/or effectiveness.
e There will be a lengthy Pilot period (about 6 months) where the throughput volumes for
the new Project HNG-X will remain low (perhaps 100-300 branches) — this mitigates
performance risks at first Go-Live.
e Following Pilot, there will be a gradual migration of branches to the new Project HNG-X
systems (over a period of to be defined, see [13]), and so there will be a corresponding
gradual ramp-up of workload volumes, which further mitigates performance risks whilst
the migration proceeds.
e The Host systems (those which provide interfaces to third party systems external to
Fujitsu Services) will remain functionally unchanged for Project HNG-X. The external
system interfaces employed in Horizon Application will remain unchanged for Project
HNG-X
« Development work outsourced to a third party (whether local or offshore) will undertake
component level testing with at least equivalent rigour to that outlined in this approach
for in-house developments, and will provide the same essential testing deliverables,
including comprehensive Regression Test packs.
« There will not be a requirement to perform End-to-End Performance Tests involving
external systems.
« Preparatory changes (such as relocating the data centres, installing the branch routers,
and installing the additional hardware in the new data centres) will be conducted under
the mantle of Horizon Application changes and so are covered by the existing Horizon
Application test approach.
e The Project HNG-X solution will adopt an Active/Standby configuration for Disaster
Recovery (DR), and the equipment at the Standby site (intended for DR) will ordinarily
be available for testing purposes, except when DR is invoked
e Project HNG-X will benefit from a gradually increasing ability to exploit targeted testing
techniques, with the corresponding ability to perform accurate impact analysis, as the
new developments progress.
e A Testing Proof of Concept (TPOC) exercise will be conducted prior to the main
development, to develop, pilot and tune any necessary testing processes, tools, and
automation methods.
‘©Copyright Post Office Ltd 2006 ‘Commercial in Confidence Ref: TST/GENISTG/0002
Version: V1.0
Date: 20/12/2006
PageNo: 42 of 44
FUJ00232561
FUJ00232561
Approach to testing Project HNG-X
Fe)
FUJITSU
Commercial in Confidence
6.2 Dependencies
e Post Office Business Requirements, Non-functional Requirements and Fujitsu Services
Operational Requirements will be produced at the outset of the programme, to formally
identify both the functional and non-functional requirements for Business Equivalence
and the reduction in Total Cost of Ownership.
e There will be an opportunity for Post Office and POA testers, working together, to
engage with the corresponding requirements teams and to ensure that the emerging
requirements are fully testable.
e« Any Acceptance Criteria specified as to be satisfied by means of testing, will be
couched in terms of the test scenarios necessary to demonstrate that they are met, and
Post Office and POA testers working together will assist in articulating these test
scenarios as formal Business Threads (BTs) and supporting Business Specified Test
Points (BSTPs) to be carried forward directly into later test planning activities.
« Post Office will provide the necessary business engagement, and POA will provide the
necessary operational engagement, in a series of workshops with Post Office and POA
testers working together to conduct a highly granular risk assessment of the derived
Test Objectives, and facilitate the prioritisation of those Test Objectives on the basis of
such risk assessments.
e The equipment at the Standby site will be organised such that it can be quickly,
cheaply, and easily be reconfigured to provide multiple, flexible test environments,
which will support the taking of rapid ‘checkpoints’ and the rapid resetting of the
environments to such checkpoints.
e It will be acceptable for Live Data to be used in Project HNG-X testing (under
appropriate strict controls) in order to exploit Parallel Testing methods and so avoid
significant time-consuming and expensive regression testing that would otherwise be
necessary. TDN. This has to be reviewed subject to the Migration Strategy
e An Architectural Proof of Concept (APOC) exercise will be conducted prior to the main
development, including
© Commencing performance modelling
o Proving viability of outline architecture and related technologies
o Proving intended scalability mechanisms
© Gaining an indication of likely performance characteristics
e Integrated project teams, with mixed-discipline personnel, will be employed for the
development of the new Project HNG-X areas, such that experienced testing personnel
will be involved throughout
e The Unit Test (UT) and Extended Link Test (ELT) areas of Horizon Application will
extend their existing processes to retain and organise their test materials as
Regression Test packs, for those systems which will form a part of the Project HNG-X
solution (e.g. the Host and Agent systems).
« The Project HNG-X development area will adopt component-based development and
O-O principles to ensure that a strong design separation is achieved (applies equally to
outsourced developments also).
« The Project HNG-X development area will adopt exhaustive, generic, component level
verification methods, in accordance with this strategic approach - Component Test
(CT) and Component Integration Test (CIT) - and retain comprehensive Regression
Test packs at this level.
e The Project HNG-X development area will conduct system level validation methods in
accordance with this strategic approach — System Test (ST) — for all systems forming a
part of the Project HNG-X solution (i.e. in-house development, outsourced
developments, and Horizon Application systems redeployed or updated to form a part
of Project HNG-X).
‘©Copyright Post Office Ltd 2006 ‘Commercial in Confidence Ref: TST/GENISTG/0002
Version: V1.0
Date: 20/12/2006
PageNo: 43 of 44
FUJ00232561
FUJ00232561
Approach to testing Project HNG-X
2
FUJITSU
Commercial in Confidence
e The existing Configuration Management (CM) system / toolset / processes will be
extended appropriately to accommodate the CM requirements of component-based,
any new development methods and testing, including supporting the necessary
relationships to support effective impact analysis, and providing the necessary
information to facilitate automatic configuring of hardware platforms and automatic
building/installing/configuring of software stacks.
e The POA and Post Office testing personnel provided for the collaborative working
required by this strategic approach, and in particular those forming the Core testing
team, will be:
o Appropriately skilled, highly experienced testing personnel
o Empowered to make all necessary testing and test management decisions the
role will require of them
o Be assigned such that continuity throughout the programme will be maintained
o Be supported by their respective managers in a manner that does not
compromise their collaborative working role
6.3 Risks
e If the necessary design separation is not achieved or the necessary rigour in
component level testing is not applied to exhaustively and generically verify each
component, then the subsequent testing stages as described in this strategic approach
will be unable to provide the in-depth detailed testing coverage to compensate for this
deficiency and so the approach will need to be entirely revised.
e If the personnel involved in the collaborative working areas are not appropriately
empowered, then the decision making processes will become protracted, significant
levels of rework will result, and so costs and schedules will be adversely impacted.
e If CM is not extended as intended, then accurate targeted testing will not be possible,
resulting either in inappropriate testing coverage (reduced product quality) or very
extensive regression testing (increased costs and timescales).
«If the Acceptance Criteria are not produced at the outset, and couched in terms of the
necessary test scenarios, it may not be possible to accommodate them collaboratively
when planning and engineering the test materials as intended, and so may necessitate
the re-separation of Post Office testing stages, with all the duplication of effort and
increased timescales that will involve.
‘©Copyright Post Office Ltd 2006 ‘Commercial in Confidence Ref: TST/GENISTG/0002
Version: V1.0
Date: 20/12/2006
PageNo: 44 of 44