Fujitsu Services
POL00000874
POL00000874
Testing and Integration Strategy Ref: VI/STR/001
Version: 3.0
Commercial-in-Confidence Date: 16/07/2002
Document Title:
Document Type:
Release:
Abstract:
Document Status:
Originator & Dept:
Contributors
Internal Distribution:
External Distribution:
Testing and Integration Strategy
Strategy
N/A
This document defines the overall strategic approach to be
adopted for the testing and integration of Pathway products. Its
scope extends beyond just the application software, to cover the
entire Pathway Solution, from the code reviews and unit testing
of individual software modules, through integration of software
and hardware, to trials of the fully configured operational
systems. It addresses the areas of functional conformance,
system performance and resilience, end to end architectural
integration, system operability, and business integrity. It also
provides a framework for the allocation of specific areas of
responsibility. It excludes the detailed verification of the many
bulk standard proprietary softyare packages and hardware items
(shrink wrapped and off:,@& shelf goods) other than in the
context of their specifi Stsage by the Pathway Solution. This
document is not wee vo a given release, or a given system
(e.g. Network Baffihg). It provides a strategic framework for
the testing agd@ntegration of all releases, spanning all the
systems caging the overall Pathway Solution. It is expected
that thisg@gcument will be updated periodically to reflect the
lessons Warned from successive releases.
Approved
Kevin Barrett - PTU
John Hunt
Programmes Director (Peter Jeram)
Development Directorate Management Team
(Gill Jackson, Alan D’Alvarez, Ian Morrison, Mike Deverell)
Pathway Library & Horizon Library
Customer representatives (for onward distribution to appropriate
customer representatives)
POL Library .
andrew.w.Thompsor,_
2002 Fujitsu Services
Commercial-in-Confidence Page: 1 of I
F/122/1
POL00000874
POL00000874
Fujitsu Services Testing and Integration Strategy Ref: VI/STR/001
Version: 3.0
Commercial-in-Confidence Date: 16/07/2002
Approval Authorities:
Gill Jackson Development Director
Andrew Thomson I Customer Testing Manager
2002 Fujitsu Services Commercial-in-Confidence Page: 2 of 1
F/122/2
POL00000874
POL00000874
Fujitsu Services Testing and Integration Strategy Ref: VI/STR/001
Version: 3.0
Commercial-in-Confidence Date: 16/07/2002
0.0 Document Control
0.1 Document History
0.1 05/01/96 Initial draft — general framework — issued for review, key Pathway
personnel only
0.2 19/01/96 Internal version only — not issued
0.3 02/02/96 Internal version only — not issued.
0.4 16/02/96 Final draft — full document — issued for wider review, Pathway
personnel across programme
1.0 01/05/96 1“ approved issue, following final review
1.5 07/09/96 Revised and presented for PDA Acceptance Review
2.0 30/09/96 2™ issue, following acceptance by the PDA
21 09/04/02 Draft, revised for internal review
2.9 24/05/02 Final Draft, comments applied dS final internal review
2.10 03/06/02 This document was hehe ‘ormally at version 3.0 but version 3.0
will now represent the ine version after customer comment.
3.0 16/07/02 Approved version \ wa@ilated with external review comments.
Re
S
0.2 Review Details »
Review Comments by :
Review Comments to :
Mandatory Review Authority Name
Customer representatives andrew.w.Thompso:
Programmes Director Peter Jeram
Development Directorate Management Gill Jackson
Team
Development Directorate Management Alan D’Alvarez
Team
Development Directorate Management Jan Morrison
Team
Development Directorate Management Mike Deverall
Team
2002 Fujitsu Services Commercial-in-Confidence Page: 3 of 1
F/122/3
POL00000874
POL00000874
Fujitsu Services Testing and Integration Strategy Ref: VI/STR/001
Version: 3.0
Commercial-in-Confidence Date: 16/07/2002
0.3 Associated Documents
fi] VI/POL0001 I General Testing Policy Pathway
[2] DE/PRO/003 I ICL Pathway Development Directorate Processes I Pathway
B] VI/STRO06 Revisions to the Testing & Integration Approach I Pathway
for Pathway Release 2
[4] VI/STRO10 Revisions to the Testing & Integration Approach I Pathway
for Pathway Release CSR+
[5] TD/RAC/001 I Technical Environment Description Pathway
[6] Un-referenced I Consignia Network Banking Automation End to I POL
document End testing strategy
[7] Un-referenced I POL Network Banking Automation High Level I POL
document Testing Strategy
[8] Un-referenced I NBA Release I Integration Testing Plan POL
document . &
[9] Un-referenced I NBA Release I non-functiony? Testing Plan POL
document 5
aT
Unless a specific version is referred to abgyey reference should be made to the current
approved versions of the documents. _&
RY
S
0.4 Abbreviations/Definidlbns
AIS Application Interface Specification (see also EPID)
AP Automated Payments
API Application Program Interface
APS Automated Payment Service
BIT Business Integration Test
BRTS Business Release Test Strategy
CM Configuration Management
cP Change Proposal
CR Change Request
CSR Core System Release — the main contractual release of the Pathway Solution
CSR+ A significant (contractual) release supplementing CSR
DIT Direct Interface Test
2002 Fujitsu Services Commercial-in-Confidence Page: 4 of I
F/122/4
Fujitsu Services
Testing and Integration Strategy
Commercial-in-Confidence
POL00000874
POL00000874
VU/STR/001
3.0
16/07/2002
DWH Data Warehouse
E2E End-to-End (sometimes used to refer to E2E Interface Testing)
EMC Electro-Magnetic Conformance
EMR Electro-Magnetic Radiation
EPID External Physical Interface Definition (see also AIS)
HCI Human Computer Interface
HLTP High Level Test Plan
LLTS Low Level Test Script
LST Live Support Test
LT Live Trial
NBE Network Banking Engine
NFR Non Functional Requirement
OBCS Order Book Control Service ©
OHE Output Handling Equipment ES
PAT Product Acceptance Test . &
PIT Product Integration Test
RT Release Test ES
sco Single Counter OutleyS™
SLA Service Level Agreement
ST System Test
STS System Test Strategy
TED Technical Environment Description [5]
UCT User Confidence Trial (used generically here to refer to Customer Testing)
UT Unit Test
VIT Volume & Integrity Test
2002 Fujitsu Services
Commercial-in-Confidence
Page: 5 of I
F/122/5
POL00000874
POL00000874
Fujitsu Services Testing and Integration Strategy Ref: VI/STR/001
Version: 3.0
Commercial-in-Confidence Date: 16/07/2002
0.5 Changes in this Version
Version Changes
3.0 All comments from external reviews considered and revisions
applied accordingly.
0.6 Changes Expected
Changes
Next version will reflect the results of the ongoing review of the PIT and Tivoli Packaging
areas. (Will principally affect section 6.3.)
2002 Fujitsu Services Commercial-in-Confidence Page: 6 of 1
F/122/6
POL00000874
POL00000874
Fujitsu Services Testing and Integration Strategy Ref: VI/STR/001
Version: 3.0
Commercial-in-Confidence Date: 16/07/2002
CONTENTS
1. INTRODUCTION and MANAGEMENT SUMMARY
2. SCOPE OF TESTING
3. HIGH LEVEL TEST OBJECTIVES
4. GENERAL APPROACH
5. RESPONSIBILITIES
6. DELIVERY UNIT TESTING
6.1. Unit Test
6.2. Product Integration Test
6.3. System Test >
6.4. Direct Interface Test s
\Y
7. PTU TESTING Re)
7.1, Business Integration Test 4?
7.2. Volume & Integrity tes
7.3. Release Test
7.4. Live Support Test
8. CUSTOMER TESTING
9. PILOT or LIVE TRIAL
10. MAINTENANCE & REGRESSION TESTING
11. EXTERNAL CERTIFICATION
12, TEST AUTOMATION
13. TEST SCRIPTING
A1.GLOSSARY OF TERMS
2002 Fujitsu Services Commercial-in-Confidence Page: 7 of 1
F/122/7
POL00000874
POL00000874
Fujitsu Services Testing and Integration Strategy Ref: VI/STR/001
Version: 3.0
Commercial-in-Confidence Date: 16/07/2002
1. INTRODUCTION and MANAGEMENT SUMMARY
1.1 This document defines the overall strategic approach to be adopted for the testing and
integration of products by Pathway, as required in section 3.3 of the General Testing
Policy [1]. This document is subordinate to the General Testing Policy [1] and describes
how the testing and integration activities will apply that policy.
1.2 At a high level it describes the stages of testing and integration to be carried out. It does
not describe the detailed processes to be adopted, which are documented in the
Development Directorate processes [2]. As such it is intended to provide a strategic
framework within which to plan the testing and integration that may be required for each
release of the Pathway Solution. It is expected that following each release a review will
be conducted to assess the effectiveness of the testing and integration activities, and that
periodically this document will be updated to reflect any lessons learned.
1.3 This document sits within a family of documents, as illustrated below. Implementation of
this strategic approach is achieved via the production of the detailed subordinate
documents, which serve as the vehicles to enhance and refine the approach described
here, and to add further appropriate detail as it becomegavailable during the development
lifecycle. Together they unambiguously define ‘objectives, scope, coverage, and
success criteria for all the testing necessary {Reach major release of the Pathway
systems. In this way it can be tailored to by suit the particular requirements of each
release. The following diagram maps ouMthis family of documents and their inter-
relationships. It is intended to be an ple rather than a definitive statement of the
documents that will be produced, wy will vary based from release to release. Some
may not always need to be Produget Nor does it preclude the production of additional,
2002 Fujitsu Services Commercial-in-Confidence Page: 8 of 1
F/122/8
POL00000874
POL00000874
Fujitsu Services Testing and Integration Strategy Ref: VI/STR/001
Version: 3.0
Commercial-in-Confidence Date: 16/07/2002
more detailed documents, should these be deemed necessary from time to time.
Quality Management System vesting suatepy Organisation Structure
Testing Release
pene Testing & Integration Strategy Ait
Service Level Agreements, —————— Migration Strategy
Business Release Test Strategy
Implementation
Security Policy =a... —
75 J F pes
Delivery Unit integration ]{ VI ][—Retease II cutomer II a, Tive Suppor j
im an es ‘ ‘Acceptance eat Pilot or Live Trial
sustcey ][_stateey I] suatcey I[ seaegy I] Pee I] suseay Ststeny
—— High Level Test Plans Customer Acceptance LST Pilot or
Test Trial Plans ue Trial
Pl Pl ans
Low Level Test Scripts ans ans
“AY
Figure 1.1 - Family of Docur&)¥ defining Test Coverage
(Shaded boxes indicate agQys of Customer Ownership)
1.4 The principal inputs used in formul this approach have been its immediate document
predecessors — version 2.0 of thigQtocument, and the main changes agreed since then in
VI/STRO06 [3] and VI/STRO10 [4], which this version consolidates and brings up to date.
These earlier documents were in turn very much shaped by the underlying Systems
Architecture described in the Technical Environment Description [5], and by Pathway’s
Release Policy at that time.
1.5 The Pathway Organisation Structure and its associated roles and responsibilities for
conducting the various stages of testing and integration has a significant bearing on
exactly how this approach is implemented, and this document should be revised
periodically to reflect the evolving organisation in order to better promote understanding
of these responsibilities.
1.6 Similarly, the Customer’s own approach to testing (as defined currently in documents [6],
[7], [8] &[9]) must be properly taken into account, and where necessary this approach
needs to be maintained to ensure continuing alignment.
2002 Fujitsu Services Commercial-in-Confidence Page: 9 of 1
F/122/9
POL00000874
POL00000874
Fujitsu Services Testing and Integration Strategy Ref: VI/STR/001
Version: 3.0
Commercial-in-Confidence Date: 16/07/2002
1.7 Each major release of the Pathway systems may have different Requirement and Design
Specifications, with different associated Service Level Agreements and Acceptance
Criteria. These may include new Performance or Security features, and may require
special Migration actions. Some of these release specific attributes are likely to require
special testing and integration attention. This will be documented and agreed in a
Business Release Test Strategy (BRTS) for that release, which will, at a high level,
describe the peculiarities concerned, the special testing attention they require, and any
release specific tailoring of this general approach which may be beneficial for the release
in question. For example, the specific testing requirements of the Network Banking
Service will be defined in the BRTS for the NWB release known as BI3.
1.8 The BRTS will be expanded down into specific testing strategies and High Level Test
Plans (HLTPs) to cover each of the stages of testing. These will detail the specific scope,
coverage, objectives and success criteria that apply. In this high level test planning
process the detailed test objectives are formally documented and agreed (via reviews) by
both the Customer and by Pathway. With Pathway and the Customer working together
specific test plans are formulated to satisfy the combined objectives. These HLTPs will
thus encompass all Pathway test objectives, and all Customer test objectives. (Note:
whilst this does not preclude the separate running of certain tests by the Customer, it does
ensure that the tests run (either jointly or singly) ¥ Pathway will encompass both
Pathway and Customer test objectives, and se Povides an opportunity to avoid
duplication of effort.) Re
1.9 Finally, when sufficient detailed info’ Son becomes available (e.g. the Design
Specifications), the ‘logical’ HLTPs ‘a translated into ‘physical’ Low Level Test
Scripts (LLTSs) and the supporting tef#data can be created, ready for test execution when
the software products become avaiBre.
1.10 Throughout the test execution period, regular checkpoints will be taken (typically by
the taking of physical database dumps and the preserving of associated flatfile and
configuration data) at suitable quiescent points. This allows tests to be restarted, at a point
prior to the point of failure, on receipt of a fix, or to re-run appropriate segments of a test
when regression testing becomes necessary because of changes to the system. This is
particularly important during the running of any particularly lengthy tests suites, where
the cost of such re-testing would otherwise quickly become prohibitive.
1.11 In addition during the test execution period, the test results of formal test runs, which
are retained for examination by the test manager, will be kept as an audit trail for each
‘final’ run.
1.12 The general approach to testing is one of staged, systematic verification, with
progressive integration of software and hardware components, first stabilising the
environment and business functionality, then system, performance, operability, security,
etc., and culminating in overall service validation of the fully configured system in a Live-
like environment.
2002 Fujitsu Services Commercial-in-Confidence Page: 10 of 1
F/122/10
POL00000874
POL00000874
Fujitsu Services Testing and Integration Strategy Ref: VI/STR/001
Version: 3.0
Commercial-in-Confidence Date: 16/07/2002
1.13 The system is subjected to testing against three principal test life-cycles, for
Functional Conformance, Architectural Conformance and Business Integrity. These give
rise to a number of separate stages of testing - Unit Test, Product Integration Test, System
Test, Direct Interface Test, Business Integration Test, Volume & Integrity Test, Release
Test, Customer Testing, and Live Support Test. There may also be a Pilot or and Live
Trial. (Note: Customer Testing, some DIT phases, and Pilot or Live Trial, are owned by
the Customer, not Pathway.) It is important to note that some traditional test activities
such as ‘performance testing’ and ‘security testing’ do not exist as discrete activities
under this approach, but rather they are integrated into each progressive stage of testing
and integration.
1.14. Unit Test (UT) deals with the detailed verification of individual modules and their
low level linking to form basic products. It is performed explicitly for all products
developed in-house. An equivalent level of testing is assumed to have been performed as
a minimum for any 3" party products (the level of testing that 3 party products are
subjected to before interception into Pathway is resolved on a product by product basis)
It comprises Prototyping & Design Feedback, Code Review, Module Test, and Link Test.
It is normally confined to a single architectural layer (e.g. the Counter Platform). In
addition, for key 3" party products, a Product Acceptange Test may be run.
1.15 Product Integration Test (PIT) is concerned witMhe initial integration of unit tested
products, configuring them correctly in approppy@x test environments, according to their
respective platform specifications, to form w8 systems for use in later stages of testing.
1.16 System Test (ST) serves to vs Bie the software against the requirement,
concentrating on functional conform~ace. It operates the products in conjunction with
each other, across the different aPrectural layers (e.g. from Counter through Agent to
Host) but is normally confined to a single business application (e.g. APS) or
infrastructure system (e.g. KMS).
1.17 Direct Interface Test (DIT) is concerned with early verification of the various
interfaces between the Pathway systems and external systems (e.g. Customer systems,
such as TIP, and Client systems, such as for Automated Payments).
1.18 Business Integration Test (BIT) exercises the system product set in a realistic
environment in terms of platform architecture and business operation. The testing focuses
on the interaction and data flows across the product set. Emphasis is given to the system’s
points of reporting, either a designed system report or an output to an external interface,
to enable an end to end reconciliation of the systems operation.
1.19 Volume and Integrity Test (VIT) is concerned with the end-to-end data flow of
volumes of data through individual system components and across the end-to-end
architecture of the system. Data integrity and system resilience/recovery under load is
proven. Measurements on throughput/performance are taken for modelling purposes.
2002 Fujitsu Services Commercial-in-Confidence Page: 11 of 1
F/122/11
POL00000874
POL00000874
Fujitsu Services Testing and Integration Strategy Ref: VI/STR/001
Version: 3.0
Commercial-in-Confidence Date: 16/07/2002
1.20 Release Test (RT) provides an environment for the migration and implementation
activities to be successfully rehearsed in a Live-like environment. The associated build
scripts, operational procedures and release documentation are tried and proven before live
usage.
1.21 Live Support Test (LST) is concerned with the interception of software changes to be
delivered to the live system outside the scope of a full or interim release. The necessary
change-specific tests, together with selected and more general regression tests, are
executed for targeted areas of the Pathway Solution. LST is upgraded with all full and
interim releases, at a point just before the software goes live, so it can continue to act as
the live support environment.
1.22 Each successive stage of testing is conducted in appropriate types of test environment,
which progressively get larger, more shareable, less simulated, more life-like and under
stricter CM control from stage to stage. This helps to concentrate test activity in more
affordable and more appropriate environments and so avoids unnecessary escalation of
machine resource costs.
1.23 In addition to the testing and integration activities performed and controlled by
Pathway, there will normally also be a range of Custasfter Testing. This will be defined,
and controlled by the Customer. Pathway will buifgsind maintain the test environments
for this activity, and the requirements for this wt need to be agreed in advance. The
management of the system and/or the runnj. f the tests themselves may also require
Pathway support. Again, requirements for ‘is will need to be agreed in advance. This
testing will focus on the end-to-end ingS¥rity of the overall business system, including
Customer systems, and any 3 PartySystems which may be involved, and taking into
account all ancillary materials h complement the software systems. These will
include: business, IT support, nd-user, and operational procedures; help systems;
training materials.
1.24 A Pilot or Live Trial approach is usually adopted to limit the initial exposure of the
Live Estate to the new release, and so affords the opportunity to reduce the likely
business impact of any potential bugs remaining in the systems. Where practical, it is
highly desirable to include some gradual ramp-up of volume, to avoid unexpected
performance issues.
1.25 As a general philosophy, it is important to accept that no system can ever be
confirmed as completely error-free. It is not possible to prove it. Tests can prove that an
error exists. They can prove that a previous error has been corrected. They cannot prove
that no further errors remain. However, by concentrating on the important characteristics
of the system operation, tests can be used to demonstrate the progressive removal of
errors to the point where these characteristics are seen to conform to expectation. So,
testing can be seen as a method (the primary method) of reducing the risk of serious
defects remaining in important areas of the system. We say the system is ‘Acceptable’ or
‘Fit for Purpose’. The cost of this error finding and removal process follows a course of
increasing cost and diminishing return. The equation is one of Cost versus Risk. Different
systems can abide varying levels of risk. These are defined in the agreed Acceptance
Criteria for the system.
2002 Fujitsu Services Commercial-in-Confidence Page: 12 of 1
F/122/12
POL00000874
POL00000874
Fujitsu Services Testing and Integration Strategy Ref: VI/STR/001
Version: 3.0
Commercial-in-Confidence Date: 16/07/2002
1.26 Therefore, testing and integration will be managed, throughout both the planning and
execution phases, on a strict cost benefit basis. A pragmatic approach is essential in
maintaining the correct balance - Cost versus Risk. During test analysis it will become
apparent that certain areas of the system are deserving of more scrutiny than others,
because of the inherent risks they entail (complexity, sensitivity, potential impact, etc.).
Similarly certain areas will be deserving of less scrutiny, being of relatively low risk
impact. Also, during the course of test execution, as the characteristics of the components
under test are revealed, problems encountered, and priorities revised, again the tests
planned for certain areas may be reviewed, supplementing them where risk outweighs
cost, and deferring or discarding them where cost outweighs risk. In general this process
will be at the discretion of the Test Manager concerned, though where this leads to a
deviation from the agreed scope and coverage, then this must also be agreed with the
programme.
2002 Fujitsu Services Commercial-in-Confidence Page: 13 of 1
F/122/13
POL00000874
POL00000874
Fujitsu Services Testing and Integration Strategy Ref: VI/STR/001
Version: 3.0
Commercial-in-Confidence Date: 16/07/2002
2. SCOPE OF TESTING
2.1 The scope of testing and integration activities encompassed by this document and its
subordinate test strategies includes the entire application software system (both 3rd party
supplied and in-house developed products) and its integration with the supporting
hardware, support products and services, and infrastructure software platform.
2.2 It does not cover specific detailed testing of the platform (which is expected and assumed
to be of Assured Status) other than in respect of its support of the software system and in
satisfying the various Service Level Agreements (SLAs) and Non Functional
Requirements (NFRs). So, for example, it is not expected that the proprietary Operating
Systems, Database Management Systems, Messaging Systems, Network Management
Systems, etc. will be subject to verification, but only that the Pathway systems intended
use of those components is supported as intended.
2.3 It does not cover the testing activities of the 3rd party suppliers. It is expected and
assumed that all products supplied will be appropriately tested prior to delivery to ensure
that they meet the supplied specifications. However, it is not assumed that these suppliers
will have integrated and proven their various pro@fets together, excepting where a
supplier is charged with producing inter-linking pepe is, where link testing is expected
prior to delivery. &
Se
2002 Fujitsu Services Commercial-in-Confidence Page: 14 of 1
F/122/14
POL00000874
POL00000874
Fujitsu Services Testing and Integration Strategy Ref: VI/STR/001
Version: 3.0
Commercial-in-Confidence Date: 16/07/2002
3. HIGH LEVEL TEST OBJECTIVES
3.1 Specific
For each release, tests are to be engineered to demonstrate the following:
a)
b)
c)
co)
e)
f)
8)
h)
To provide the necessary design feedback information by conducting informal
trial runs of early software releases and hardware platforms, to allow the design
and development of the related products to be completed prior to their entry into
the later testing stages, and so to reduce the level of disruption to those stages
which may otherwise result from consequent late changes to the design (Unit Test
— Technical Evaluation)
each module developed in-house to be compliant with the corresponding module
specification and to link correctly with co-operating modules in that product.
(Unit Test — Code Review, Module Test, Link Test).
key bespoke software products provided by a party supplier, to be compliant
with its Product Specification. (Unit Test - uct Acceptance Test).
software and hardware components to Porrectly configured to support testing
and subsequent live running. (Prod egration Test)
software products to operate sugesSsfully in conjunction with one another to form
application or infrastructure gous which satisfy the relevant Requirement and
Design Specifications (Sym Test).
Pathway systems to interface correctly, on an individual basis, with external
systems, in compliance with the external interface specifications. (Direct Interface
Test)
that products are that are unchanged at a release are not affected by the
introduction of new system features, achieved through the execution of adequate
regression testing (System testing, Business Integration Test, Volume and
Integrity Test)
software systems to operate and co-operate successfully together on the target
hardware and infrastructure software platform, and to interface correctly together,
satisfying the Requirement Specifications, including any SLAs and NFRs.
Application and system data integrity to be proven within all points of reporting
through controlled E2E data flow. This to include verification of Output Handling
Equipment (OHE) operation as required by the system, such as card production,
use of ‘slip printers’ at the counters and all associated stationery and materials,
where products are available. (Business Integration Test, Volume and Integrity
Test)
to validate the end-to-end technical architecture employed. (Business Integration
Test, Volume & Integrity Test, Release Test)
2002 Fujitsu Services Commercial-in-Confidence Page: 15 of 1
FI122/15
POL00000874
POL00000874
Fujitsu Services Testing and Integration Strategy Ref: VI/STR/001
Version: 3.0
Commercial-in-Confidence Date: 16/07/2002
j) to demonstrate that the system meets the specified and agreed levels of
functionality and performance, and so is fit for service. (Business Integration Test,
Volume & Integrity Test)
k) the migration and implementation activity to be successfully rehearsed in a Live-
like environment, including verifying timings and schedules, and to validate
(through usage) the supporting systems management and software distribution
facilities. (Release Test)
1) to successfully operate following agreed procedures (business, user, operational),
help systems, and training materials (Customer Testing)
m
all systems comprising the overall business system (including Customer systems
and 3" party systems) to operate and co-operate correctly on an end-to-end basis,
maintaining integrity (outputs, system data, financial status) throughout a cycle of
events (Customer Testing)
n) confirm that all necessary preparations have been made to go-live, and that the
support environment is properly configured. (LST)
0) system to successfully support the ‘Live Trial’ in verifying user procedures,
training material, support mechanisms, and hel desk procedures, and to confirm
the migration and implementation prior o8) jonwide rollout and usage. (Live
Trial). BS
3.2 General Ne)
&
For each release tests are to be engi ered with particular regard to the following areas of
good general practice:
a) to ensure that as much testing as possible is performed as early as possible in the
lifecycle, to reduce defect correction costs and avoid unwelcome schedule
disruption late in the lifecycle.
b) to arrange for as much testing as possible to be of an automated and re-runable
nature, to reduce regression test costs, to speed test execution times, and to avoid
unnecessary levels of human error.
c) to comply with the General Testing Policy [1].
d) to adopt a joint (Pathway and Customer) approach to testing, combining test
objectives, high level test planning, and test execution activities wherever
practicable, attacking multiple objectives in combined tests and so reducing
duplication of effort and minimising the overall elapsed time required for
effective testing.
2002 Fujitsu Services Commercial-in-Confidence Page: 16 of 1
F/122/16
POL00000874
POL00000874
Fujitsu Services Testing and Integration Strategy Ref: VI/STR/001
Version: 3.0
Commercial-in-Confidence Date: 16/07/2002
4, APPROACH
4.1 The approach is one of staged, systematic verification, with progressive integration of the
wide ranging system components, culminating in overall validation of the fully
configured system within a near-Live environment, followed where practicable with a
period of Pilot or Live Trial running. It is a continuous process, spanning the whole
development lifecycle, commencing with test analysis and the production of formal test
material, and progressing through to test execution, result checking, and defect removal.
Static Testing techniques are also employed where advantageous (e.g. Code Review).
A partnership approach between Pathway and Customer is key in securing the most
efficient and effective means of testing and integrating the system. Pooling skills will
enable clearer focus on test objectives at the outset, will promote higher quality test
planning and scripting, and will help reduce the elapsed time necessary for test execution
through co-operative effort and reduced duplication.
The component products are moved through distinct, separately planned and executed
stages of testing, each designed to progress the products to a higher level of assurance.
Once a stage of testing has been completed or has reaeS¥d an agreed point for a particular
product set, then that product set moves to ‘Assure@Status’ and is deemed to be ready for
use in the next stage and for progressively er integration with their co-operating
product sets. A proportion of the products gmake up the system are provided by 3rd
party suppliers. So, the emphasis in testingy in be toward ‘black box’ techniques. That is,
the detailed inner workings of suppliedQproducts are not examined and put under test, but
rather their gross behaviour is ver ‘in the context of the services they are required to
perform. Sy
4.2 It is therefore particularly important that the Business Requirement is firmly understood
at the outset. Here, the Business Requirement is taken as being encompassed by the
agreed Requirement Specification and System Design Specification, and any associated
Service Level Agreements and Acceptance Criteria. It is of paramount importance that
these be maintained under strict Change Control, with the testers being included in the
impact assessment process for any and all changes to this baseline.
Similarly it is important that an ‘Independent’ approach be adopted to maintain
objectivity in developing the ‘black box’ tests necessary to successfully verify and
integrate the mixed product set. That is, that products be treated alike, irrespective of
whether they be supplied by a third party or developed in-house, with formal product
handover into testing being established via formal handover mechanisms.
2002 Fujitsu Services Commercial-in-Confidence Page: 17 of 1
FI122/17
POL00000874
POL00000874
Fujitsu Services Testing and Integration Strategy Ref: VI/STR/001
Version: 3.0
Commercial-in-Confidence Date: 16/07/2002
4.3 Given the condensed timescales typically available, it is also important that test planning
commences at the earliest opportunity. Testers must be involved throughout the
development lifecycle, starting with high level ‘logical’ test planning, based on the
business requirements, and progressively developing these into ‘physical’ test scripts, so
that the test material is ready and approved in good time for use in test execution when
the products are handed over into test.
This is consistent with the general recognition that testing follows a lifecycle closely
interwoven with that of development. This is best embodied in the well established
lifecycle “V’ diagrams, where development progresses down the left leg of the ‘V’ with
analysis, design and construction, and then up the right leg of the ‘V’ with unit test,
function test and system test execution. The horizontal relationships between left and
tight legs indicate the test analysis, planning and preparation required against each phase.
4.4 There are three distinct lifecycles of verification and validation deployed on this
programme, each bringing a different perspective. The ‘Functional Conformance’ of the
system must be evaluated. That is, on a purely functional level, confirming that the
services required of the system by the Customers, as defined in the functional
requirements, are being met. The ‘Architectural formance’ of the system, both
software and hardware must be evaluated. That i e underlying system architecture
employed, with extensive infrastructure softwarecftid mixed hardware platforms, must be
trialed under stress to confirm service attripgt’s such as performance, operability and
security. The ‘Business Integrity’ must be Waluated. Here it is necessary to demonstrate
integrity across the breadth of the whale business system, encompassing not only the
hardware and software componentg@but also the ancillary technical and business
components (e.g. business proce: help and training materials, user guides, operations
manuals, target reference data, etc).
2002 Fujitsu Services Commercial-in-Confidence Page: 18 of 1
F/122/18
POL00000874
POL00000874
Fujitsu Services Testing and Integration Strategy Ref: VI/STR/001
Version: 3.0
Functional
Conformance
Unit Test
Produ ee
Integration Test Architectaral
Conformance
System Test
Business
Direct ueines
Interface Test Integrity
Business
integration Test
Volume &
Integrity Test
Release Test
User Test
Live Support Test
Live Trial
S
BS
Figure 4.1 - Mapping of mens Sno the Stages of Testing
These three testing lifecycles - Functior a@vomance Architectural Conformance, and
Business Integrity — are not themselv: etna activities, conducted as separate activities,
but rather they are peeves gS ach testing stage must take account of. They map
onto each other as shown above
For a given release, test analysis should start in parallel with formulating/agreeing the
business requirements, contract schedules, acceptance criteria, and system requirements.
The emphasis at this stage is twofold - to develop a thorough understanding of the overall
business system and how the Pathway Solution is expected to contribute, and to influence
the system design to promote testability. (If any of these key inputs are late in production,
or are subject to significant late change, then it is likely to severely hamper this early and
crucial test planning activity, resulting in significant rework.)
2002 Fujitsu Services Commercial-in-Confidence Page: 19 of 1
F/122/19
POL00000874
POL00000874
Fujitsu Services Testing and Integration Strategy Ref: VI/STR/001
Version: 3.0
Commercial-in-Confidence Date: 16/07/2002
Following this strategic framework, the Business Release Test Strategy (BRTS) is
produced. This is specific to the release, and describes any special characteristics it may
have (from a testing perspective) and what changes/additions to the general approach are
required. For example, a release may introduce new business applications on the desktop,
perhaps extending the memory footprint occupied, and so may require particular tests to
recalibrate the performance model for the counter platform. Or, there may be fundamental
changes being made to a sensitive piece of the infrastructure, such as the Riposte Message
Server, which will require significant system-wide regression testing. Or, concerns may
have been raised, or ‘hot-spots’ identified in particular areas of the system, which will
require specific attention in testing. All such release-specific adjustments to the strategic
approach will be documented and agreed in the BRTS.
4.5 Functional Conformance.
Business
Integration
Requirements
Analysis
stem
Design
Purchase
Figure 4.2 - The
‘Functional
Conformance’
Testing Lifecycle
Product
Design
The Pathway
Solution comprises a
mixture of bespoke,
in-house developed
i products, and 3”
S party products.
These 3" party products may oS rink-wrapped, commercial off-the-shelf products, or
they may be bespoke products commissioned from a 3" party developer.
Product
Acceptance/ Module
Test
contract
Construction Code Review
Products developed in-house will follow the established Pathway Unit Test processes,
with selected modules being subject to formal Code Review, all modules being subject to
Module Test, and where appropriate closely associated modules being subject to Link
Test.
Products supplied by a 3" party supplier, whether they are bespoke developments, or off-
the-shelf products, are expected to have been appropriately tested by the supplier prior to
delivery to Pathway. Selected 3" party products will in addition be subject to Product
Acceptance Test, which may involve verifying that agreed acceptance criteria have been
met.
These activities are all collectively classed as Unit Test UT). This brings all products up
to a common level of trusted status whereby all in-house products will conform to their
Low Level designs (LLDs), and 3" party products will satisfy their acceptance criteria. At
this stage they are placed under formal CM control.
2002 Fujitsu Services Commercial-in-Confidence Page: 20 of 1
F/122/20
POL00000874
POL00000874
Fujitsu Services Testing and Integration Strategy Ref: VI/STR/001
Version: 3.0
Commercial-in-Confidence Date: 16/07/2002
Before the products can be deployed ready for System Test, they need to be correctly
configured on their target platforms, which in turn need to be correctly built. This detailed
technical integration is achieved by Product Integration Test (PIT).
Each system, whether an application system (e.g. APS) or an infrastructure system (e.g.
KMS) is typically made up of a number of products operating across a number of
platforms. This cross platform integration is achieved by System Test (ST). It is normally
here that a system can first be run as a whole, and the principal objective is to
demonstrate that it conforms to the functional requirements specified for the system.
System Test will normally proceed following the established 3-Pass model. The ‘1* Pass’
is run in a relatively informal fashion, forcing the tests through where necessary. It is
intended to stabilise the system, the test environment, the test scripts and test data. This
will identify the majority of ‘stoppers’ for correction prior to running the ‘Main Pass’.
The main pass is typically run in an iterative fashion. The ‘Main Pass’ is the principal
defect removal work horse of System Test. Once the system is demonstrated to be more
or less conformant to the functional requirements, then the ‘Final Pass’ is run to capture a
formal audit trail.
As soon as the system has become stable then worgStan commence on verifying the
external interfaces. This is achieved by Direct Iggerface Test (DIT). This is a jointly
planned and executed activity. (Jointly with the oghtiers of the interfacing systems.)
\Y
Once the majority of the serious defe igtave been corrected and the system(s) are
becoming more and mores conformantty e functional requirements (i.e. about half way
through ST Main Pass) then works ‘an start on bringing all the infrastructure and
application systems together to fagw the integrated Pathway Solution. This is achieved by
Business Integration Test (BIT). This also normally follows the 3-Pass Model. The
executed cycles operate end-to-end (within the confines of the Pathway Solution) to
confirm all the various components operate correctly together to satisfy all the functional
requirements.
This completes the Functional Conformance testing.
2002 Fujitsu Services Commercial-in-Confidence Page: 21 of 1
F/122/21
POL00000874
POL00000874
Fujitsu Services Testing and Integration Strategy Ref: VI/STR/001
Version: 3.0
Commercial-in-Confidence Date: 16/07/2002
4.6 Architectural Conformance.
Migration & Release Test 7 Ty
snpleneniaios => Figure 4.3 - The
Stas volume ‘Architectural
"pause = Integrity Test Conformance’
“—_ Testing Lifecycle
Business
Integration
Test
Ditect
Interface
Test
PIT, ST, DIT,
and BIT, in
addition to their
roles in
Functional
Conformance
(see 4.5 above) also play a role in Architectural Conformance. When these tests are
planned, any relevant Non-Functional Requirements (NFRs) are also taken into account.
system
cifications
Test
roduet Platform Product
Specifications Integration Test
In PIT, the technical characteristics of the run-time plggéorm, as specified in the TED [5],
must be correctly applied, or where not defingd®éxplicitly, they may have to be
prototyped. Of particular importance here is any rity requirements of the build.
\Y
In DIT, the volume, throughput, and paSrmance characteristics specified, and any
operational SLAs that may have been agyeéd for the interface, should be observed, though
the limited nature of the DIT environafent precludes any stress testing.
S
In ST and particularly in BIT, the scope and coverage of the tests should be extended to
cover all explicitly defined NFRs, except for volume, throughput, and performance
(excluding counter performance which is within the remit of BIT). These should include
HCI, output handling, operability, usability, security, fail-over, fall-back, recovery,
archiving, audit, and systems management (all subject to the limitations of the test
environments concerned).
In addition, in BIT, special provision is made to cover volume, throughput, and.
performance for the counter platform (i.e. up to the ISDN interface). A special ‘large
outlet’ configuration of 20 counters has been included in the primary BIT test rig for this
purpose, and tooling has been developed to allow automated activity following
configurable transaction profiles.
Once BIT has stabilised the systems, allowing the Pathway Solution to be run as a whole
(i.e. after BIT I“ Pass) then work can start on conducting large scale volume tests and
stressing the system in other ways (e.g. major recovery operations, loading the ISDN
network, and the external interfaces, saturating batch windows, etc.) to confirm that the
overall system and data integrity is maintained, and SLAs are satisfied. This is achieved
by Volume & Integrity Test (VIT).
2002 Fujitsu Services Commercial-in-Confidence Page: 22 of 1
F/122/22
POL00000874
POL00000874
Fujitsu Services
Testing and Integration Strategy
Ref: VU/STR/001
Version: 3.0
Commercial-in-Confidence
Date: 16/07/2002
In parallel with this, the Migration and Implementation activities are rehearsed, including
regression, and the Pathway Solution is operated in a Live-like fashion, employing the
target systems management facilities, and confirming that the introduction of any new or
changed infrastructure does not de-stabilise the established architecture. This is achieved
by Release Test (RT).
This completes the Architectural Conformance testing.
4.7 Business Integrity.
Pilot or Live Trial
Live Support Test
User Test
Release Test
Volume &
Integrity Test
Business
Integration Test
Figure 4.4 - The ‘Business Integrity’ Testing Lifecycle
BIT, VIT, and RT, in addition to their roles in Architectural Conformance (see 4.6 above)
also play a role in Business Integrity. This is implicit, because the Pathway Solution
forms such a large proportion of the overall business system, and is in many respects
pivotal in this area.
In addition, extensive reconciliation facilities are built into the Pathway Solution, and
proving their functional conformance goes a long way toward providing assurance that
end-to-end financial integrity is achieved.
2002 Fujitsu Services
Commercial-in-Confidence
Page: 23 of 1
F/122/23
POL00000874
POL00000874
Fujitsu Services Testing and Integration Strategy Ref: VI/STR/001
Version: 3.0
Commercial-in-Confidence Date: 16/07/2002
When BIT is nearing completion (about two-thirds through BIT Main Pass) then the
Pathway Solution is sufficiently stable to support Customer Testing, which will explicitly
test the end-to-end integrity of the overall business system. This will involve running the
Pathway Solution together with all other contributing systems (Customer systems and 3“
party system), and employing all ancillary materials (such as procedures, help systems,
training, user guides, etc.).
Just before the release is implemented, LST is upgraded so it can then support the live
system post go live. This upgrade acts as a final migration test and also proves the live
keys.
The Pilot or Live Trial provides a final opportunity to identify any failures in business
integrity by operating a full Live service for a restricted number of outlets, before the
release is rolled out nationwide.
This completes the Business Integrity testing.
2002 Fujitsu Services Commercial-in-Confidence Page: 24 of 1
F/122/24
POL00000874
POL00000874
Fujitsu Services Testing and Integration Strategy Ref: VI/STR/001
Version: 3.0
Commercial-in-Confidence Date: 16/07/2002
5. RESPONSIBILITIES
5.1 This section lays out a general framework of responsibilities for testing of the
Pathway Solution. The terms and roles used are intended to be generic rather than reference
individual job titles, which will change from time to time during the course of a large project.
For example, the roles of Quality Manager and Risk Manager may well be encompassed by a
single individual or job title, whilst Development Manager may conversely be distributed
over many individuals.
Role Responsibilities
Customer Review Test Strategies and provide input to High Level Test Plans. Be
Testing/Release involved in witnessing some later stages of testing. Review Pathway testing
results. Plan, control, and perform Customer Testing. Plan, control, and co-
Manager ordinate Pilot or Live Trial. Accept the System.
Pathway Review/Sign-off Strategies and Plans. Project Management of testing and
Testing integration activities. Interface with all areas.
Manager we
Pathway Review/Sign-off Testing and @megration Strategy. Recognise testing
Development dependencies in project pla: “Provision of resources. Setting of Priorities.
Manager Agreement of changes 8 scope/coverage. Provide adequate bug-fix
turnaround. Ss
Zz
Pathway Review/Sign-off plans for performance and other NFR coverage.
Architecture Specify technicalR¥ttform requirements.
Manager
Pathway Set operational requirements. Review/Sign-off test plans for service
Operations management and operability. Close involvement in running of later stages.
Manager
Pathway Apply QMS in development and testing areas. Confirm handover process.
‘ Check conduct of tests. Collect and interpret quality metrics. Check quality
Quality
records.
Manager
Pathway Conduct risk assessments. Approve cost versus risk evaluations. Produce
Risk residual risk report. Input to acceptance process.
Manager
Pathway Conduct handovers. Review environmental status against and confirm
cM integrity of configurations.
Manager
Pathway Review/Sign-off test plans for security and access coverage. Confirm data
2002 Fujitsu Services Commercial-in-Confidence Page: 25 of 1
F/122/25
POL00000874
POL00000874
Fujitsu Services Testing and Integration Strategy Ref: VUSTR/O01
Version: 3.0
Commercial-in-Confidence Date: 16/07/2002
Security retention periods. Check security status for different environments / data
Manager usage.
Supplier Ensure acceptable product quality on supply. Provide adequate bug-fix
Manager turnaround
2002 Fujitsu Services Commercial-in-Confidence Page: 26 of 1
F/122/26
POL00000874
POL00000874
Fujitsu Services Testing and Integration Strategy Ref: VU/STR/001
Version: 3.0
Commercial-in-Confidence Date: 16/07/2002
6. DELIVERY UNIT TEST
6.1 Composition
Unit Test
Code Review
Module Test
Link Test
Product Acceptance Test
Product Integration Test
System Test
Direct Interface Test . RS
we
(note: DIT may on occasions be performed by PTU rather than Development)
6.2 Unit Test )
Ne)
>
6.2.1 Code Review é
RS
6.2.1.1 Context
Optional, at the discretion of the Delivery Unit Manager, though recommended for
complex areas of code. Normally performed by a peer developer or team leader
within the Delivery Unit concerned. Applies to products developed in-house only.
Requires no Customer involvement. Results formally recorded and retained.
6.2.1.2 Objectives
To identify obvious discrepancies between the module code as implemented and the
detailed Module and API Specifications in the Low Level Design.
To confirm that the agreed coding standards have been followed
To assess the maintainability of the code in question
6.2.1.3 Overview
2002 Fujitsu Services Commercial-in-Confidence Page: 27 of 1
F/122/27
POL00000874
POL00000874
Fujitsu Services Testing and Integration Strategy Ref: VI/STR/001
Version: 3.0
Commercial-in-Confidence Date: 16/07/2002
Module & API
Specifications
V
Module — Code
Source Code Review
CONSTRUCTION
Figure 6.1 - Schematic Overview of Code Review
6.2.2. Module Test
6.2.2.1 Context
S
Y
Mandatory. Normally performed by develope: éSthe module concerned. Applies only
to products developed in-house. Requir Xo Customer involvement. Formally
planned, though data may be ad hoc. Reg orally recorded and retained.
6.2.2.2 Objectives se
To demonstrate that each r ogite developed in-house conforms to the detailed
module specification in the ins Level Design.
To demonstrate that the unit conforms to the relevant HCI where applicable.
To provide an economic basis for ongoing regression testing of each module as and
when they are subject to change.
6.2.2.3 Overview
2002 Fujitsu Services Commercial-in-Confidence Page: 28 of 1
F/122/28
POL00000874
POL00000874
Fujitsu Services Testing and Integration Strategy Ref: VI/STR/001
Version: 3.0
Commercial-in-Confidence Date: 16/07/2002
DESIGN
Unit Test Preparation
Module Link
Test Plans Test Plans
Runtime Module Link
Modules rest Runs Test Runs
Unit Test Execution
CONSTRUCTION
Figure 6.2 - Schematic Overview of Module & Link Test
6.2.3 Link Test
6.2.3.1 Context
S
Mandatory. Normally performed by (one of) the dBvetoper(s) of the linking modules
concerned. Applies only to products devel asad in-house. Requires no Customer
involvement. Results formally recorded ang ‘tained.
6.2.3.2 Objectives Re)
To pre-integrate co-operating magdtes within a product by demonstrating that their
APIs are implemented correctly@nd so that the modules link together as specified and
co-operate properly as an integrated unit.
6.2.3.3 Overview
(See figure 6.2 at section 6.2.2.3)
6.2.4 Product Acceptance Test
6.2.4.1 Context
Optional, at the discretion of the Delivery Unit Manager, though strongly
recommended. Normally performed by a developer of a related product, or by a tester
with experience of a related product. Applies only to products developed by 3“ party
suppliers. Requires no Customer involvement. Formally planned. Results formally
recorded and retained. May have contractual significance.
6.2.4.2 Objectives
2002 Fujitsu Services Commercial-in-Confidence Page: 29 of 1
F/122/29
POL00000874
POL00000874
Fujitsu Services Testing and Integration Strategy Ref: VI/STR/001
Version: 3.0
Commercial-in-Confidence Date: 16/07/2002
To confirm that the 3“ Party supplied product is fit for entry into wider Pathway
Testing, being sufficiently stable to cause no disruption, and conforming broadly to
specification.
(Where there are formal or contractual acceptance criteria agreed between Pathway
and the 3“ Party supplier concerned) to confirm that the product satisfies the specified
acceptance criteria.
6.2.4.3 Overview
Product Description
Product Acceptance
Ss
Test Plans & Scripts
a P
'
Product Acceptance
Test Execution
v
Accepted Products
Acceptance Criteria
3" Party supplied Product
[Para suptiod roe] Go
Figure 6.3 - Schematic Overview of Product Acceptance Test
2002 Fujitsu Services Commercial-in-Confidence Page: 30 of 1
F/122/30
POL00000874
POL00000874
Fujitsu Services Testing and Integration Strategy Ref: VI/STR/001
Version: 3.0
Commercial-in-Confidence Date: 16/07/2002
6.3 Product Integration Test
6.3.1 Context
Mandatory. Performed by PIT team. Applies to all products whether developed in-
house or supplied by 3 Party, whether Application or Infrastructure, whether
Software or Hardware. Requires no Customer involvement. Formally planned.
Results formally recorded and retained.
6.3.2 Objectives
To establish and prove the correct build instructions for each testing platform
To integrate products onto these platforms accordingly and so establish their
configuration
To prove their environmental stability
To enforce correct initial configuration managemeRgot the testing environments
To intercept change to these environmeg SS reintegrating and reproving the
configuration as appropriate
)
6.3.3. Overview Ne)
&
The processes used in prada related activities in the CM, SPTS, and Tivoli
Packaging areas are presently under review. The results of this review will be
reflected in this document at version 4.0, together with any other changes that may be
required following its review by the Customer. In the meantime, the following
diagram is extracted from VI/STR/010 [4] to serve as a brief summary of the current
strategic approach. For further detail, refer to the parent document. However, it
should be noted that this is likely to be subject to change, and may not match current
2002 Fujitsu Services Commercial-in-Confidence Page: 31 of 1
F/122/31
POL00000874
POL00000874
Fujitsu Services Testing and Integration Strategy Ref: VI/STR/001
Version: 3.0
Commercial-in-Confidence Date: 16/07/2002
practice.
var cM sprs
baselines baselines environments
Dynix, I ——___»
i <7 r
Riposte I ————> I sien I I of i st
: ([- > > : °
Anni a Appt a t 2 a7 pir
Goshousey I —————> © © —
a : BIT
Grd Party)
— : :
pntc, I————> I s = : =
ip I———~> I = D :
bane i
789 t
cp I” ve config change ms i
456
change to contig
apply deltas to
To contig change : baselines
or
Figure 6.4 - Schematic Overvigg§ Product Integration Test
s°
RS
aS
2002 Fujitsu Services Commercial-in-Confidence Page: 32 of 1
F/122/32
POL00000874
POL00000874
Fujitsu Services Testing and Integration Strategy Ref: VI/STR/001
Version: 3.0
Commercial-in-Confidence Date: 16/07/2002
6.4,
64.1
System Test
Context
Mandatory. Applies to all systems comprising the Pathway Solution, irrespective of
origin. Performed by the System Test Team within the Delivery Unit concerned, on
receipt (via CM) of products from Unit Test and/or PIT. Formally planned, scripted and
executed, preferably in re-runnable fashion. Test Results formally recorded and retained.
6.4.2 Objectives
6.4.3
To demonstrate, through a series of comprehensive business driven scenarios, that the
software system concerned, functionally conforms with the agreed Requirements.
To expose the Infrastructure software to use by the application, and to demonstrate
that within the limitations of a reduced hardware platform it provides the application
with the specified support.
To perform initial verification of OHE requiggns where available equipment,
stationery and other materials allow. eS
To demonstrate simple recovery and nce features of the system within the
bounds of the platform in use.
To form a comprehensive system gb pack for later use, and to prepare the way
for full system integration in cS owing stages of testing.
Overview
Requirement Specification
f
Design Specification
System Test
Acceptance Criteria Plans & Scripts
\
TED, including derived NFRs °
System Test
Execution
v
Functionally Conformant System]
Runtime Products —
(passed Unit Test)
F/122/33
POL00000874
POL00000874
Fujitsu Services Testing and Integration Strategy Ref: VI/STR/001
Version: 3.0
Commercial-in-Confidence Date: 16/07/2002
6.5.3
Figure 6.5 - Schematic Overview of System Test
Direct Interface Testing
Context
Optional. Applies when the AIS or EPID for a defined interface changes or a new
interface is introduced. Usually performed by the Delivery Unit concerned (or on
occasion by PTU, whichever is best placed to conduct the tests under the prevailing
circumstances. This would be specified in the BRTS for the release concerned.)
Formally planned, scripted and executed. Test Results formally recorded and retained.
Completed once the sending/receiving systems have declared that the system’s are in
a state of readiness for DIT to be commenced. (DIT may actually be planned and co-
ordinated by POL, but supported by Fujitsu and thggsher relevant supplier domains.).
3
Objecti &
ectives 5
a} Se
Re)
That the interface under scrutiny as been correctly interpreted within the system
changes or new reuse
That operational integrity of the interface under test will be maintained.
If required, that the defined interface can handle the required volume/throughput and
satisfy any SLAs specific to the interface.
Overview
2002 Fujitsu Services Commercial-in-Confidence Page: 34 of 1
F/122/34
POL00000874
POL00000874
Fujitsu Services Testing and Integration Strategy Ref: VI/STR/001
02
Interface Specification
Design Specification Plans & Scripts
SA
c= I Interface Test
A
NFRs & SLAs for the I/F i
Interface Test
Execution
Y
Proven Interface
Runtime Products —
(Stable — ST 1st Pass)
Figure 6.6 - Schematic Overview of Direct Interface Testing
2002 Fujitsu Services Commercial-in-Confidence Page: 35 of 1
F/122/35
Fujitsu Services Testing and Integration Strategy
Commercial-in-Confidence
Ref:
Version:
Date:
POL00000874
POL00000874
VU/STR/001
3.0
16/07/2002
7. PTU TESTING
7.1 Composition
Business Integration Test (BIT)
Volume & Integrity Test (VIT)
Release Test (RT)
Live Support Test (LST)
Development \ \ + +
Testing ~
esting ~* [prT
User
SPTSI -*
~
Business
bby Integration .
ive
PTU Volume & [~~ Support
Testing Test
Test
System
Figure 7.1 ~ PTU Context Overview(Major or Interim Releases)
7.1.1 Business Integration Test
7.1.1.1 Context
Customer
Testing
Mandatory. Performed on receipt (via CM) of products from System Test Teams.
Formally planned, scripted and executed, preferably in re-runnable fashion. Test
Results formally recorded and retained. Customer involved in the test construction
2002 Fujitsu Services Commercial-in-Confidence
Page: 36 of 1
F/122/36
POL00000874
POL00000874
Fujitsu Services Testing and Integration Strategy Ref: VI/STR/001
Version: 3.0
Commercial-in-Confidence Date: 16/07/2002
process (providing input, reviewing HLTPs), and the execution cycles (witnessing
tests, reviewing results).
7.1.1.2 Objectives
To demonstrate the successful integration of the software and hardware system, verifying
all components that exist within the Pathway domain.
To exercise end to end data flows, to demonstrate maintained data integrity and the
semantics underlying the external interfaces.
To prove all of the system's points of reporting and financial reconciliation including the
full set of reports generated via the counter, the host systems and the DWH.
To complete performance testing up to the point of data leaving the post office (at which
point responsibility is passed to the volume testing team). Performance measurements
extend to the complete range of available counter functionality.
To perform verification of OHE requirements where available equipment, stationery and
other materials allow.
To ensure the system successfully operates against the varying office configurations,
large offices, medium sized offices, SCOs and mokits. This extends to office working
patterns (stock unit configuration and transactiggstnixes) as well as available physical
hardware. : Se
To demonstrate successful execution of aggeptance criteria that have been designated for
proving during this stage. 3 ~
To ensure that the implementati ©) full and interim releases does not destabilise the
existing application product s& "this is achieved through the establishment of a
comprehensive system regression testing pack, which can be deployed selectively or in a
blanket fashion.
Produce a closure report detailing the results of the integration test phase, distribute to
relevant parties including the customer.
2002 Fujitsu Services Commercial-in-Confidence Page: 37 of 1
F/122/37
POL00000874
POL00000874
Fujitsu Services Testing and Integration Strategy Ref: VI/STR/001
Version: 3.0
Commercial-in-Confidence Date: 16/07/2002
Requirement Specification Procedure documents
Design Specification SA
= preeration Test
Acceptance Criteria a ans & Scripts
TED, including derived NFRs +
Integration Test
Runtime Products — Execution
(Passed System Test) ub
Functionally Integrated
Conformant System
. or
7.1.1.3 Overview oy
S
rigs 2 - Schematic Overview of Business
Integration Test Ne)
7.2 Volume & Integrity Test
7.2.1.1 Context
Optional, depending upon the requirements of the release. If the release introduces a
new transaction that did not change record structures and was simply going to
increase live volumes, but remain within understood levels, then volume testing
would not be required. Performed on receipt (via CM) of products which have been
through the early stages of Business Integration Testing. Formally planned, scripted
and executed, preferably in re-runnable fashion. Test Results formally recorded and
retained. Customer involved in the test construction process (providing input) and the
test execution (reviewing results).
7.2.1.2 Objectives
2002 Fujitsu Services Commercial-in-Confidence Page: 38 of 1
F/122/38
POL00000874
POL00000874
Fujitsu Services Testing and Integration Strategy Ref: VI/STR/001
Version: 3.0
Commercial-in-Confidence Date: 16/07/2002
To ensure that the technical solution can support the target number of Post Office outlets.
To ensure contractual SLAs are not breached and the system is compliant.
To demonstrate successful execution of acceptance criteria that have been designated for
proving during this stage.
To ensure system/data integrity is maintained with volume data processing.
To ensure system/data integrity is maintained in the case of platform or system failure.
To ensure that the system can support the processing of projected volumes (host to outlet
and outlet to host).
7.2.1.3 Overview
Requirement Specification Volumetric Documents
Design Specification
cot I Volume Test
Acceptance Criteria a Plans & Scripts
TED, including derived NFRs '
Volume Test
Runtime Products — Execution
(Completed first stages of iy}
Integration test)
Resilient, Volume
Proven System
Figure 7.3 - Schematic Overview of Volume & Integrity Test
7.3 Release Test
7.3.1.1 Context
Mandatory. Performed on receipt (via CM) of products which have been through
initial stages of Business Integration Testing and volume testing. Final phases, with
2002 Fujitsu Services Commercial-in-Confidence Page: 39 of 1
F/122/39
POL00000874
POL00000874
Fujitsu Services Testing and Integration Strategy Ref: VI/STR/001
Version: 3.0
Commercial-in-Confidence Date: 16/07/2002
the code delivered within Tivoli wrappers, run after completion of system integration
and volume testing. Formally planned, scripted and executed, preferably in re-
runnable fashion. Test Results formally recorded and retained. Customer involved in
the test construction process (providing input) and in the test execution (reviewing
results).
7.3.1.2 Objectives
To ensure that upgrades to existing services (hardware, software or database) do not de-
stabilise infrastructure components or application products.
To ensure that operational integrity is maintained during and post the applications release
into live.
To ensure that support documentation is updated to reflect the changes for a particular
release.
To demonstrate successful execution of acceptance criteria that have been designated for
proving during this stage.
To ensure the migration and implementation activities are tested and proven to maintain
data integrity. os
To ensure new system features are introduced in “a esive fashion.
To ensure the various (defined) song eas campus to existing or upgraded
outlets functions correctly.
To ensure regression paths work and Genta integrity is maintained
7.3.1.3 Overview >
2002 Fujitsu Services Commercial-in-Confidence Page: 40 of 1
F/122/40
POL00000874
POL00000874
Fujitsu Services Testing and Integration Strategy Ref: VI/STR/001
Requirement Specification Migration Strategy 2
Support Documents
Plans & Scripts
Design Specification +
TED, including derived NFRs
— Release Test
Acceptance Criteria
Release Test
— Execution
Runtime Products UY
(Within Tivoli wrappers)
Release ready for
customer Services
Figure 7.4 - Schematic Overview of Release Test
7.4 Live Support Test (LST)
7.4.1.1 Context : RS
Ss
Mandatory. Applies to all releases, whet! major release, an interim release, or
individual bug-fixes. Performed by ¢0§YLST team, immediately prior to Live
Implementation. For Major/Interim g@eases, the Pathway Solution will have been
subject to the full range of Pathyyaly and Customer Testing before being passed to
LST. When simple system c egbes/bus- fixes are required to be delivered to live
outside of the normal defin ease windows, then typically they will pass directly
from Development Testing into LST. Tests are based on stored regression packs as
well as tests generated for the specific changes. Results are formally stored and
recorded. No Customer involvement required.
7.4.1.2 Objectives
Verification of changes that are required to be intercepted outside of normal release
windows.
Environment for investigation and attempted reproduction of issues being investigated
within live.
Live Support testing also act as the last verification point for all releases. The support
environment must intercept the release before live, to enable support to continue. As this
is performed as the final planned activity before a release goes live it also acts as a final
verification point.
74.1.3 Overview
2002 Fujitsu Services Commercial-in-Confidence Page: 41 of 1
F/122/41
POL00000874
POL00000874
Fujitsu Services Testing and Integration Strategy Ref: VI/STR/001
Specific details relating to
the change being made.
— Regression
Plans & Scripts
!
Details of main release
system Enhancements.
Changed Products Live Support
(Within Tivoli Wraper) Execution
Change ready for
Live
Figure 7.5 - Schematic Overview of Live
Support Test S
ee
S
&
&
Re)
>
RY
a
2002 Fujitsu Services Commercial-in-Confidence Page: 42 of 1
F/122/42
POL00000874
POL00000874
Fujitsu Services Testing and Integration Strategy Ref: VI/STR/001
Version: 3.0
Commercial-in-Confidence Date: 16/07/2002
8. CUSTOMER TESTING
8.1 The Pathway Solution is not alone in forming the Customer’s overall business system.
There are also a number of Customer systems, and some other 3“ part systems. These
must all co-operate together correctly to maintain end-to-end integrity. Pathway is
responsible (jointly with the owners of the interfacing systems) in conducting tests to
prove that its external interfaces conform to specification. But this is not sufficient.
Wider tests must be performed to demonstrate that these disparate systems do indeed co-
operate correctly and that end-to-end integrity is maintained. This is of particular
importance with regard to the financial status of the systems.
8.2 There are also a large number of ancillary components that complement the IT systems
in making up the overall business system. These include materials like manuals,
procedures, help texts, training, call centres, paper systems, etc. It is not sufficient to
test that the Pathway Solution meshes correctly with these components. It is necessary
to confirm that they do so on an end-to-end basis, all operating in concert to satisfy the
overall business system requirements.
8.3 These aspects of testing are the responsibility of at S stomer For the purposes of this
strategic framework it is sufficient to recognig@that Customer Testing of this nature
will normally be required, and that a le test environment with appropriate
technical support will need to be provided(This should be the subject of agreement at
an early stage.) se
8.4 The precise approach adopted ssh Customer in conducting these tests is outside
Pathway’s control. It will be ined in one or more test strategies produced by the
Customer, such as those produced for the Network Banking Service [6] & [7]. For the
purposes of this strategic framework, the Customer Testing activity will simply be
referred to as a User Confidence Trial (UCT). A UCT was in fact performed for the
CSR+ release, but the customer may well adopt different terminology for future
releases. This can be clarified in the BRTS.
8.5 The extent to which Pathway will be expected to provide technical support will vary
from release to release, depending on the complexity of operation. For some releases it
may be necessary for Pathway to do no more than provide the test environment, and
carry out the system management activities required for safe operation. For other
releases it may be necessary for Pathway to provide much greater support, perhaps even
having to perform the tests on behalf of the Customer. It is expected that the level of
support required will be defined in the Customer’s testing strategies.
2002 Fujitsu Services Commercial-in-Confidence Page: 43 of 1
F/122/43
POL00000874
POL00000874
Fujitsu Services Testing and Integration Strategy Ref: VI/STR/001
Version: 3.0
Commercial-in-Confidence Date: 16/07/2002
9. PILOT or LIVE TRIAL
8.6 Each major release of software is normally trialed before exposure to the full live
estate. The complexity of the release and the timescales that it is required to be
delivered in should drive the duration and scope of this trial.
8.7 Typically, before full committal, a small number of outlets, ideally ramping up from a
mere handful to 200 or 300, should perform live business with counters migrated to the
target application. These offices should be closely monitored during the trial period and
the results analysed before national rollout of the new release proceeds.
8.8 A ramp up of outlets is desirable in order to gain the maximum benefit from running
the trial. By commencing with just a handful of outlets, the potential impact of and
serious failure is limited to the bare minimum, effectively shielding the majority of the
national network of users from any adverse affects. By then increasing the number of
outlets in the trial to a few hundred, it is more likely to reveal any unforeseen
volume/performance problems, whilst still sheltering under the protection of a closely
monitored and controlled trial, and without impacting the rest of the network.
8.9 For similar reasons the target offices should caver a wide range of configurations.
Variables considered should include locati the changes made have a variance
based on location), typical transaction vol , office size and office connectivity type.
8.10 The first stage of the trial may be to SS that the counter application can be regressed
back to it's previous software basgfne if required. The subsequent stages should be
designed to ensure that operat integrity is maintained within the target offices and
that the application changes made have been successfully implemented.
8.11 On successful completion of the trial, agreement having been reached that any
erroneous events that may have occurred during the trial have either been addressed
successfully, or do not require immediate resolution, the software can be rolled out
nationwide across the whole network.
2002 Fujitsu Services Commercial-in-Confidence Page: 44 of 1
FI122/44
POL00000874
POL00000874
Fujitsu Services Testing and Integration Strategy Ref: VI/STR/001
Version: 3.0
Commercial-in-Confidence Date: 16/07/2002
9 MAINTENANCE & REGRESSION TESTING
10.1
10.2
10.3
10.4
10.5
The test plans drawn up for each part of the system and for each stage of testing are,
wherever practicable, designed to be re-runnable test packs, including the necessary
test scripts, test data, and expected results.
When a change is made to the system, this is either as a result of a fix being applied to
address an incident (a PinICL), or following the approval of a Change Request (CR)
and/or a Change Proposal (CP). Both PinICL and CP changes are formally controlled.
Part of the lifecycle of a PinICL is the planning of the testing required to satisfy it.
Part of the lifecycle of a CP is the impact assessment, which includes planning of the
testing required to satisfy it.
In both situations, the testing required normally breaks down into two distinct types.
The testing of the change itself — often conducted as targeted testing, and no different
to any already described in the rest of this strategy.
The testing required to confirm that the change, got caused the rest of the system
to regress (has not inadvertently introduced sggve unwanted side-effect) — regression
4)
testing.
The change may introduce new com ois, or may change existing components, or
may remove existing components, gy“some combination of these. The targeted tests
are satisfied by introducing, chatSing, or removing test conditions (or adjusting the
expected results) relating to ase components accordingly. The regression testing is
satisfied by identifying the heighbouring/interfacing components surrounding the
change, and in turn identifying appropriate tests which will exercise these areas of the
system. Typically this will involve scripts from each major stage of testing through
the lifecycle.
Consider 3 examples:
a) A minor change in a discrete application on the counter platform
Regression testing can be restricted to the application itself.
b) A significant change to a host application including a schema change
Regression testing should be considered across the host application, and any
others sharing that Schema, which may include related Agent applications.
Where the affected systems include an external interface, then DIT regression
runs should also be considered. Where the affected systems have critical
performance factors (or other NFRs) which may be compromised, then runs
targeted at re-proving these factors should be considered.
2002 Fujitsu Services Commercial-in-Confidence Page: 45 of 1
F/122/45
POL00000874
POL00000874
Fujitsu Services Testing and Integration Strategy Ref: VI/STR/001
Version: 3.0
Commercial-in-Confidence Date: 16/07/2002
c) Major changes to fundamental pieces of system infrastructure, highly invasive in
nature
Blanket regression testing should be considered across the whole solution.
10.6 The extent of regression testing required is a matter of judgement, and in turn is
dependent on the extent of the changes applied, and the nature of the product(s)
concerned. The use of test automation can be a factor in reducing the required
timescale to execute the defined regression tests (re: Section 14).
2002 Fujitsu Services Commercial-in-Confidence Page: 46 of 1
F/122/46
POL00000874
POL00000874
Fujitsu Services Testing and Integration Strategy Ref: VI/STR/001
Version: 3.0
Commercial-in-Confidence Date: 16/07/2002
i. EXTERNAL CERTIFICATION
11.1 Certain products employed by in the Pathway Solution will from time to time require
external certification before they can be used operationally, for legal reasons. For
example, the weigh-scales on the counter needed to be examined in operation, in
conjunction with their software, and certified by HM Weights & Measures, before
they could be used legally in live operation. Similarly, all electrical and electronic
equipment in the workplace (including the Pathway counters, their keyboards,
printers, PIN-Pads, etc.) need to pass Health and Safety regulations regarding EMR
emission levels, which requires EMC testing to be performed to obtain the
certification. Any significant changes to such configurations will require re-
benchmarking and re-certification.
11.2. All such products will be formally identified for each release, together with the
authorising bodies concerned, in the BRTS for that release, and the External
Certification activity will be included in the project plans, and monitored accordingly.
2002 Fujitsu Services Commercial-in-Confidence Page: 47 of 1
F/122/47
POL00000874
POL00000874
Fujitsu Services Testing and Integration Strategy Ref: VI/STR/001
Version: 3.0
Commercial-in-Confidence Date: 16/07/2002
12. TEST SCRIPTING
12.1 Test Scripting is an area that has benefited from regular and ongoing process
improvements. During CSR and CSR+ timescales the differing Pathway testing areas
attempted to implement universal standards for format and content of the test scripts
produced. It has since been recognised that at times sections of scripts were being
produced for the sake of production rather than for any great level of added value.
This is because the testing approaches within the Delivery Units and PTU have
matured and evolved to become more focused, and with less common ground
remaining to benefit from a universally applied approach. This is appropriate based
on the differing requirements of functional conformance, architectural conformance
and business integrity testing. It is also a sign that the differing testing areas are now
strongly complementing rather than duplicating each other. The same fundamental
principles are however applied within each area. The product is put through a full test
analysis process, resulting in the generation of HLTPs. These are then further broken
down, through the addition of detailed operations, expected results and test data, into
an executable script in the form of LLTSs.
12.2. The different testing areas will continue to wage and enhance their individual
analysis methodologies. Enforced conformang> io a defined generic methodology
which attempted to retrospectively cover ahi areas differing requirements would
only lead to redundancy and omissio thin the generated scripts. Each area’s
methodologies will however not evel) without independent review. The actual
approach to be adopted will be Weribed within the subordinate test strategies
generated for each release and wise open to critical review.
12.3. The methodologies in use wide free to mature but will be required to maintain the
following underlying principles and objectives:
The differing stages of the process e.g. test condition construction, scenario
generation and LLTS line definition should contain as little duplication as possible in
order to minimise the overall man day effort required.
A review should be completed early in the test construction process to avoid costly re-
work if significant errors are discovered.
The tests should be presented in a format that makes them easy to understand and
therefore easy to review, execute, and check results.
For appropriate test stages the documentation should allow acceptance criteria to be
easily mapped to the defined tests to avoid unnecessary duplication of effort in having
to construct separate acceptance test trials.
When generating the execution material care must be taken to satisfy the original
testing intent expressed in the associated test condition(s), and not to allow the test(s)
to become corrupted.
2002 Fujitsu Services Commercial-in-Confidence Page: 48 of 1
F/122/48
POL00000874
POL00000874
Fujitsu Services Testing and Integration Strategy Ref: VI/STR/001
Version: 3.0
Commercial-in-Confidence Date: 16/07/2002
12.4 Tooling to assist the analysis process has now been trialed and proven through the use
of Test Director to hold the Business Integration Test scripts. It has been shown that
Test Director assists the test analysis process through creating a consistent structure
and framework to script within. Although it does not necessarily reduce the effort or
elapsed time for the test analysis process it does allow easy manipulation of the
generated data, for example for the purposes of generating regression scripts. Further
benefits are also found when the test scripts are executed within Test Director. Based
on the success of this initial implementation Test Director will be assessed for use in
other testing areas.
2002 Fujitsu Services Commercial-in-Confidence Page: 49 of 1
F/122/49
POL00000874
POL00000874
Fujitsu Services Testing and Integration Strategy Ref: VI/STR/001
Version: 3.0
Commercial-in-Confidence Date: 16/07/2002
13. TEST AUTOMATION / TOOLING
13.2
13.3
13.4
In the early stages of Pathway testing the technology in use was in advance of the test
automation tools available. The test automation market has matured considerably over
the last few years and this has been exploited with a number of test automation
initiatives being implemented.
The most mature use of automation on the project is that of bulk office loading. A
large office configuration (typically a 20 counter outlet) is used in conjunction with an
automated harness to replicate the day to day activity of such an office. The harness
‘drives’ the application through it's screen sequences to complete the required
transaction. No data is generated other than by the application itself. The benefits of
this implementation are obvious. The harness, operated by one user, can simulate the
whole office being used by a complete set of clerks.
The harness has been matured so that the pause between transactions and the speed
that a transaction can be completed within can be manipulated to simulate different
working patterns. In addition the harness can be easily configured so that the
percentage that each transaction type represenig>of all the overall transactions
performed can be changed. This means that diffegeht office locations can be simulated
e.g. rural offices completing a higher leva SOF AP transactions than central city
locations. Most importantly the harns as been maintained to include new
transaction types e.g. post office local fect and OBCS bar coded foils. The harness
will be continually matured to inc all new transaction types including network
banking transactions. It is importay¥that a realistic transaction mix performed in a live
like manner can be run ashomation to ensure that large office configurations
continue to operate successfulby.
The most significant advancement in the utilisation of automation has been the
introduction of replay and edit scripts. A pilot study was commissioned to evaluate
the effectiveness of such scripts as a regression tool. The study concluded that a series
of easy wins were achievable where the time taken to automate particular areas of
functionality would show significant benefits over the time currently being taken to
run manual tests, an example area of functionality would be counter time out tests. It
was also concluded that some form of automated blitz testing script would be a
powerful tool to quickly ascertain the quality of a build or of a new code drop. In
addition although areas beyond the easy wins would not show the same benefits, it
was obvious that gains could be achieved by automating large elements of the counter
testing.
An automated script was constructed and is now in use as a blitzing tool. This blitz
script executes the counter application’s main functionality and quickly discloses any
major failings. In two hours of automated execution four hours of manual execution
can be completed. The script works on the basis of a transaction or function no longer
operating in the sequence it previously performed to.
2002 Fujitsu Services Commercial-in-Confidence Page: 50 of 1
F/122/50
POL00000874
POL00000874
Fujitsu Services Testing and Integration Strategy Ref: VI/STR/001
Version: 3.0
Commercial-in-Confidence Date: 16/07/2002
13.7
13.8.1
13.8.2
13.8.3
The automation scripts have also been matured to perform capture and compare.
These scripts take screen shots at any point required and compare the image to one
previously stored. Based on the originals having been verified as being correct then
any change in the image is highlighted. The image can either be the screen captured at
any point during a transaction flow or a print preview of a receipt or a cash account.
In this way any change, either very significant or just a change in sequence on a
report, is automatically highlighted as a discrepancy.
The capture and compare scripts are powerful when used as a regression tool against
stable elements of the system. Significant quantities of scripts have been produced to
date. The automated regression pack currently consists of in excess of 3000 checks
with each check representing one point in the system where the screen content is
compared back to a previously baselined screen shot. These scripts have been
successfully used during BI2 counter regression activities. It is envisaged that the
pack would need to contain between 25 to 30,000 checks to represent a counter
regression pack which was comprehensive enough to replace a significant element of
the currently executed manual counter regression pack. Development of this pack is to
continue with the long term aim being the reduction of the project’s cost of
performing a full counter regression test. 5
A Pathway forum has sought to identify additigght automation initiatives. Automated
performance benchmarking scripts have beg@developed which compare timings for
previous counter actions against the sam& tions at a new release. These scripts are
proving a powerful tool in the early ddntification of potential performance issues.
The level of accuracy is not matye enough to replace the video benchmarking
exercise but it is enough to identigpotential performance regression within a release.
In addition automated Appli in Performance Matrix scripts have been developed.
These scripts allow a system function to become a trigger which is then executed in
turn before a series of effects, which are defined range of system functions. The
trigger is then automatically varied and the cycle repeated. In this way if any effect is
recorded as running slower, after being trigged by a particular function, then the
trigger can be identified as containing a potential performance issue.
The further development of the automated tooling already in place and the continuing
commissioning of new automated initiatives is a prime driver to developing a reliable
and repeatable test pack. Continued increased use of automation is a key element in
the maturity of the overall Pathway testing strategy.
On occasion, bespoke (in-house developed) testing tools will be necessary/desirable.
The decision whether or not to commission the development of a bespoke testing tool
must be taken on a cost benefit basis, and must also take into account issues like
development lead-times, ongoing maintenance of the tool, potential problems with
reliability, and what testing will be required to validate the correct working of the
tool. One good example of where a bespoke testing tool is all but essential is in the
Network Banking Service. Here the Pathway system has to interface with a 3" party
system — the Network Banking Engine (NBE). As this 3" party system will not be
generally available for test use until near the end of the programme timescales, then it
2002 Fujitsu Services Commercial-in-Confidence Page: 51 of 1
F/122/51
POL00000874
POL00000874
Fujitsu Services Testing and Integration Strategy Ref: VU/STR/001
Version: 3.0
Commercial-in-Confidence Date: 16/07/2002
was agreed that a simulator for the NBE should be developed. This allows Pathway
testing to progress independently from that of the NBE.
2002 Fujitsu Services Commercial-in-Confidence Page: 52 of 1
F/122/52
POL00000874
POL00000874
Fujitsu Services Testing and Integration Strategy Ref: VUSTR/O01
Version: 3.0
Commercial-in-Confidence Date: 16/07/2002
APPENDICES.
Al. GLOSSARY OF TERMS
mS
oy
&
3
se
(é)
w
2002 Fujitsu Services Commercial-in-Confidence Page: 53 of 1
F/122/53
POL00000874
POL00000874
Fujitsu Services Testing and Integration Strategy Ref: VUSTR/O01
Version: 3.0
Commercial-in-Confidence Date: 16/07/2002
APPENDIX 1
GLOSSARY OF TERMS
mS
oy
&
3
se
(é)
w
2002 Fujitsu Services Commercial-in-Confidence Page: 54 of 1
F/122/54
Fujitsu Services
POL00000874
POL00000874
Testing and Integration Strategy Ref: VUSTR/001
Version: 3.0
Commercial-in-Confidence Date: 16/07/2002
Term
Meaning
Acceptance Criteria
The document(s) agreed between the Customer and the
Service Provider which lay down the criteria against which
the system will be measured at agreed points during the
course of testing in judging whether the system is to be
formally accepted by the Customer.
Architectural Conformance
The testing lifecycle wherein the test objectives are oriented
toward demonstrating that the system and its components
conform to their architectural specification, including aspects
such as structure, performance, security, operability,
resilience, recovery, and other NFRs.
Assured Status
The state a system or its components have reached on
successful completion of a particular stage of testing - e.g.
‘System Test Assured Status’ following successful
completion of System Teste>
POL
The Customer SS
Black Box Testing
Testing of a syst 1 its components which is planned and
conducted withos) nowledge of the inner workings, looking
only at the isbut, the specification of the system, and the
output. Rv
Blitz Testing
A progS8s of crashing a series of representative tests through,
in an informal fashion, without much attention for accuracy,
or data integrity. The objective being to quickly and cheaply
identify the majority of the ‘stoppers’ so that they can be
corrected, so stabilising the runtime environment sufficient to
allow more formal tests to be run without so high a level of
disruption. This technique is particularly useful in stabilising
new test environments and getting new software components
to bed down together.
Business Integrity
The testing lifecycle wherein the test objectives are oriented
toward demonstrating that the overall system satisfies the
business need and when used in context and together with the
associated business products, retains overall business
integrity.
Business Procedures
The document(s) defining the business activity surrounding
and controlling the use of the computer system by the
Customer and their staff.
Customer
POL
EPIDs
Unambiguous definitions of the physical nature of an
2002 Fujitsu Services
Commercial-in-Confidence Page: 55 of 1
F/122/55
POL00000874
POL00000874
Fujitsu Services Testing and Integration Strategy Ref: VI/STR/001
Version: 3.0
Commercial-in-Confidence Date: 16/07/2002
interface between the system under test and an external
system.
Functional Conformance The testing lifecycle wherein the test objectives are oriented
toward demonstrating that the system and its components
conform to their functional specification.
Fujitsu Services The Service Provider.
National Rollout (see also Rollout) The phase of the programme where,
following Live Trial, the system is subject to rollout on a
nation-wide basis according to an agreed timetable.
Operating Instructions The document(s) defining the operational control of the
system, such as batch schedule control, recovery control, etc.
Output Handling Equipment I Ancillary equipment associated with the computer system or
its output, which extends the service offered. For example,
equipment used in conjunction with computer printers, to say
collate and envelope and frank output targeted for posting.
Real Time Here used to differentiate between modes of test execution,
not to be confused with, RBil-time systems. It means where
test teams do not mange the time during the running of a
test, but rather opergght e test in ‘real’ time.
Regression Testing The process O® demonstrating that a system and_ its
componentss{aVe not regressed to a ‘worse’ state following a
change (ug@afly to the software).
Rollout The patess of packaging and distributing the system out to
the various locations at which it will operate and be used (see
also National Rollout).
Service Level Agreement The document(s) agreed between the Customer and the
Service Provider which lay down the minimum operational
criteria to which the service must be run and maintained.
Service Provider Pathway - providing the service of development, maintenance
and operational running of the system, according to agreed
requirements and Service Level Agreements.
Stoppers Defects which effectively prohibit a particular line(s) of
testing continuing as planned. These defects are said to ‘Stop’
the test(s). Hence ‘Stoppers’.
Supplier one of the suppliers of Pathway - a 3rd party providing goods
or services to Pathway, or one its direct sub-contractors
working to specification by and on behalf of Pathway.
Validation The process of evaluating products (at the end of a given
phase) to demonstrate compliance with their specified
requirements.
2002 Fujitsu Services Commercial-in-Confidence Page: 56 of I
F/122/56
POL00000874
POL00000874
Fujitsu Services Testing and Integration Strategy Ref: VI/STR/OO1
Version: 3.0
Commercial-in-Confidence Date: 16/07/2002
Verification The process of evaluating the products of a given phase to
ensure correctness and consistency with respect to the
products and standards provided as input to that phase.
White Box Testing Testing of a system or its components which exploits
knowledge of or interrogates the inner workings.
2002 Fujitsu Services Commercial-in-Confidence Page: 57 of 1
F/122/57
POL00000874
POL00000874
Fujitsu Services Testing and Integration Strategy Ref: VU/STR/001
Version: 3.0
Commercial-in-Confidence Date: 16/07/2002
END OF DOCUMENT
2002 Fujitsu Services Commercial-in-Confidence Page: 58 of 1
F/122/58