Fujitsu Services
FUJ00001673
FUJ00001673
Testing and Integration Strategy Ref: VI/STR/O01
Version: 3.0
Commercial-in-Confidence Date: 16/07/2002
Document Title:
Document Type:
Release:
Abstract:
Document Status:
Originator & Dept:
Contributors
Internal Distribution:
External Distribution:
Testing and Integration Strategy
Strategy
N/A
This document defines the overall strategic approach to be adopted
for the testing and integration of Pathway products. Its scope
extends beyond just the application software, to cover the entire
Pathway Solution, from the code reviews and unit testing of
individual software modules, through integration of software and
hardware, to trials of the fully configured operational systems. It
addresses the areas of functional conformance, system performance
and resilience, end to end architectural integration, system
operability, and business integrity. It also provides a framework for
the allocation of specific areas of responsibility. It excludes the
detailed verification of the many bulk standard proprietary software
packages and hardware items (shrink wrapped and off the shelf
goods) other than in the context of their specified usage by the
Pathway Solution. This document is not specific to a given release,
or a given system (e.g. Network Banking). It provides a strategic
framework for the testing and integration of all releases, spanning
all the systems comprising the overall Pathway Solution. It is
expected that this document will be updated periodically to reflect
the lessons learned from successive releases.
Approved
Kevin Barrett - PTU
John Hunt
Programmes Director (Peter Jeram)
Development Directorate Management Team
(Gill Jackson, Alan D’ Alvarez, Ian Morrison, Mike Deverell)
Pathway Library & Horizon Library
Customer representatives (for onward distribution to appropriate
customer representatives)
POL Library .
andrew.w. Thompson,
© 2002 Fujitsu Services
Commercial-in-Confidence Page: 1 of 1
Fujitsu Services
Testing and Integration Strategy
FUJ00001673
FUJ00001673
Ref: VIU/STR/001
Version: 3.0
Commercial-in-Confidence Date: 16/07/2002
Approval Authorities:
Name Position Signature Date
Gill Jackson Development Director
Andrew Thomson
Customer Testing Manager
© 2002 Fujitsu Services
Commercial-in-Confidence
Page: 2 of I
FUJ00001673
FUJ00001673
Fujitsu Services Testing and Integration Strategy Ref: VIU/STR/001
Version: 3.0
Commercial-in-Confidence Date: 16/07/2002
0.0 Document Control
0.1. Document History
Version I Date Reason
0.1 05/01/96 Initial draft — general framework — issued for review, key Pathway
personnel only
0.2 19/01/96 Internal version only — not issued
0.3 02/02/96 Internal version only — not issued
0.4 16/02/96 Final draft — full document — issued for wider review, Pathway
personnel across programme
1.0 01/05/96 1* approved issue, following final review
1.5 07/09/96 Revised and presented for PDA Acceptance Review
2.0 30/09/96 I 2" issue, following acceptance by the PDA
2.1 09/04/02 Draft, revised for internal review
2.9 24/05/02 Final Draft, comments applied, for final internal review
2.10 03/06/02 This document was issued informally at version 3.0 but version 3.0
will now represent the baseline version after customer comment.
3.0 16/07/02 Approved version updated with external review comments.
0.2 Review Details
[Review Comments by :
[Review Comments to :
Mandatory Review Authority Name
(Customer representatives landrew.w. Thompson@postoffice.co.uk
[Programmes Director Peter Jeram
[Development Directorate Management Team IGill Jackson
[Development Directorate Management Team [Alan D’ Alvarez
Development Directorate Management Tan Morrison
Team
Development Directorate Management Mike Deverall
Team
0.3 Associated Documents
© 2002 Fujitsu Services Commercial-in-Confidence Page: 3 of I
Fujitsu Services
Testing and Integration Strategy Ref: VI/STR/O01
Version: 3.0
FUJ00001673
FUJ00001673
Commercial-in-Confidence Date: 16/07/2002
Ref. I Library Ref. Description or Title Source
1] VI/POL0001 I General Testing Policy Pathway
[2] DE/PRO/003__I ICL Pathway Development Directorate Processes I Pathway
B] VI/STRO06 Revisions to the Testing & Integration Approach I Pathway
for Pathway Release 2
[4] VI/STROLO Revisions to the Testing & Integration Approach I Pathway
for Pathway Release CSR+
[5] TD/RAC/001 I Technical Environment Description Pathway
[6] Un-referenced I Consignia Network Banking Automation End to I POL
document End testing strategy
[7] Un-referenced I POL Network Banking Automation High Level I POL
document Testing Strategy
[8] Un-referenced I NBA Release I Integration Testing Plan POL
document
[9] Un-referenced I NBA Release I non-functional Testing Plan POL
document
Unless a specific version is referred to above, reference should be made to the current
approved versions of the documents.
0.4 Abbreviations/Definitions
Abbreviation I Definitions
AIS Application Interface Specification (see also EPID)
AP Automated Payments
API Application Program Interface
APS Automated Payment Service
BIT Business Integration Test
BRTS Business Release Test Strategy
CM Configuration Management
cP Change Proposal
CR Change Request
CSR Core System Release — the main contractual release of the Pathway Solution
CSR+ A significant (contractual) release supplementing CSR
DIT Direct Interface Test
DWH Data Warchouse
© 2002 Fujitsu Services Commercial-in-Confidence Page: 4 of I
Fujitsu Services
Testing and Integration Strategy
Commercial-in-Confidence
Ref:
Version:
Date:
VIU/STR/001
3.0
16/07/2002
FUJ00001673
FUJ00001673
E2E End-to-End (sometimes used to refer to E2E Interface Testing)
EMC Electro-Magnetic Conformance
EMR Electro-Magnetic Radiation
EPID External Physical Interface Definition (see also AIS)
HCI Human Computer Interface
HLTP High Level Test Plan
LLTS Low Level Test Script
LST Live Support Test
LT Live Trial
NBE Network Banking Engine
NFR Non Functional Requirement
OBCS Order Book Control Service
OHE Output Handling Equipment
PAT Product Acceptance Test
PIT Product Integration Test
RT Release Test
sco Single Counter Outlet
SLA Service Level Agreement
ST System Test
STS System Test Strategy
TED Technical Environment Description [5]
UCcT User Confidence Trial (used generically here to refer to Customer Testing)
UT Unit Test
VIT Volume & Integrity Test
© 2002 Fujitsu Services
Commercial-in-Confidence
Page: 5 of I
FUJ00001673
FUJ00001673
Fujitsu Services Testing and Integration Strategy Ref: VIU/STR/001
Version: 3.0
Commercial-in-Confidence Date: 16/07/2002
0.5 Changes in this Version
\Version ‘Changes
3.0 ‘All comments from external reviews considered and revisions applied
ccordingly.
0.6 Changes Expected
(Changes
Next version will reflect the results of the ongoing review of the PIT and Tivoli Packaging
areas. (Will principally affect section 6.3.)
© 2002 Fujitsu Services Commercial-in-Confidence Page: 6 of I
Fujitsu Services Testing and Integration Strategy
Commercial-in-Confidence
Ref:
Version:
Date:
FUJ00001673
FUJ00001673
VIU/STR/001
3.0
16/07/2002
CONTENTS
1. INTRODUCTION and MANAGEMENT SUMMARY
2. SCOPE OF TESTING
3. HIGH LEVEL TEST OBJECTIVES
4. GENERAL APPROACH
5. RESPONSIBILITIES
6. DELIVERY UNIT TESTING
6.1. Unit Test
6.2. Product Integration Test
6.3. System Test
6.4. Direct Interface Test
7. PTU TESTING
7.1. Business Integration Test
7.2. Volume & Integrity Test
7.3. Release Test
7.4. Live Support Test
8. CUSTOMER TESTING
9. PILOT or LIVE TRIAL
10. MAINTENANCE & REGRESSION TESTING
11. EXTERNAL CERTIFICATION
12. TEST AUTOMATION
13. TEST SCRIPTING
A1.GLOSSARY OF TERMS
© 2002 Fujitsu Services Commercial-in-Confidence
Page: 7 of I
FUJ00001673
FUJ00001673
Fujitsu Services Testing and Integration Strategy Ref: VUSTR/001
Version: 3.0
Commercial-in-Confidence Date: 16/07/2002
1. INTRODUCTION and MANAGEMENT SUMMARY
1.1 This document defines the overall strategic approach to be adopted for the testing and
integration of products by Pathway, as required in section 3.3 of the General Testing
Policy [1]. This document is subordinate to the General Testing Policy [1] and describes
how the testing and integration activities will apply that policy.
1.2 At a high level it describes the stages of testing and integration to be carried out. It does
not describe the detailed processes to be adopted, which are documented in the
Development Directorate processes [2]. As such it is intended to provide a strategic
framework within which to plan the testing and integration that may be required for each
release of the Pathway Solution. It is expected that following each release a review will be
conducted to assess the effectiveness of the testing and integration activities, and that
periodically this document will be updated to reflect any lessons learned.
1.3 This document sits within a family of documents, as illustrated below. Implementation of
this strategic approach is achieved via the production of the detailed subordinate
documents, which serve as the vehicles to enhance and refine the approach described here,
and to add further appropriate detail as it becomes available during the development
lifecycle. Together they unambiguously define the objectives, scope, coverage, and success
criteria for all the testing necessary for each major release of the Pathway systems. In this
way it can be tailored to best suit the particular requirements of each release. The
following diagram maps out this family of documents and their inter-relationships. It is
intended to be an example rather than a definitive statement of the documents that will be
produced, which will vary based from release to release. Some may not always need to be
produced. Nor does it preclude the production of additional, more detailed documents,
© 2002 Fujitsu Services Commercial-in-Confidence Page: 8 of I
FUJ00001673
FUJ00001673
Fujitsu Services Testing and Integration Strategy Ref: VUSTR/001
Version: 3.0
Commercial-in-Confidence Date: 16/07/2002
should these be deemed necessary from time to time.
Customer
Testing Strategy
Testing Release
Testing
Processes
Quality Management System Organisation Structure
Systems
Testing & Integration Strategy ems
Acceptance Criteria — << Performance Budgets
Migration Strategy
Service Level Agreements Business Release Test Strategy
Implementation
Security Policy —s 4 g g iy eee Strategy
integration Val lease ‘ustomer ive Support
Delivery Unit Tegrat rat Relea coe Acceptance I] /i¥e Suppor] I itor or Live Trial
Test Strategies Test es Test Testing Process Test Strategy
Strategy Strategy Strategy Strategy Strategy
§ § %¥ § F GF F 8
—— High Level Test Plans Customer Acceptance LST Pilot or
Test Trial Plans — Live Trial
Plans Plans Plans
Low Level Test Scripts
Figure 1.1 - Family of Documents defining Test Coverage
(Shaded boxes indicate areas of Customer Ownership)
1.4 The principal inputs used in formulating this approach have been its immediate document
predecessors — version 2.0 of this document, and the main changes agreed since then in
VI/STRO06 [3] and VI/STRO10 [4], which this version consolidates and brings up to date.
These earlier documents were in turn very much shaped by the underlying Systems
Architecture described in the Technical Environment Description [5], and by Pathway’s
Release Policy at that time.
1.5 The Pathway Organisation Structure and its associated roles and responsibilities for
conducting the various stages of testing and integration has a significant bearing on exactly
how this approach is implemented, and this document should be revised periodically to
reflect the evolving organisation in order to better promote understanding of these
responsibilities.
1.6 Similarly, the Customer’s own approach to testing (as defined currently in documents [6],
[7]. [8] &[9]) must be properly taken into account, and where necessary this approach
needs to be maintained to ensure continuing alignment.
© 2002 Fujitsu Services Commercial-in-Confidence Page: 9 of I
FUJ00001673
FUJ00001673
Fujitsu Services Testing and Integration Strategy Ref: VUSTR/001
Version: 3.0
Commercial-in-Confidence Date: 16/07/2002
1.7 Each major release of the Pathway systems may have different Requirement and Design
Specifications, with different associated Service Level Agreements and Acceptance
Criteria. These may include new Performance or Security features, and may require special
Migration actions. Some of these release specific attributes are likely to require special
testing and integration attention. This will be documented and agreed in a Business
Release Test Strategy (BRTS) for that release, which will, at a high level, describe the
peculiarities concerned, the special testing attention they require, and any release specific
tailoring of this general approach which may be beneficial for the release in question. For
example, the specific testing requirements of the Network Banking Service will be defined
in the BRTS for the NWB release known as BI3.
1.8 The BRTS will be expanded down into specific testing strategies and High Level Test
Plans (HLTPs) to cover each of the stages of testing. These will detail the specific scope,
coverage, objectives and success criteria that apply. In this high level test planning process
the detailed test objectives are formally documented and agreed (via reviews) by both the
Customer and by Pathway. With Pathway and the Customer working together specific test
plans are formulated to satisfy the combined objectives. These HLTPs will thus encompass
all Pathway test objectives, and all Customer test objectives. (Note: whilst this does not
preclude the separate running of certain tests by the Customer, it does ensure that the tests
run (either jointly or singly) by Pathway will encompass both Pathway and Customer test
objectives, and so provides an opportunity to avoid duplication of effort.)
1.9 Finally, when sufficient detailed information becomes available (e.g. the Design
Specifications), the ‘logical’ HLTPs can be translated into ‘physical’ Low Level Test
Scripts (LLTSs) and the supporting test data can be created, ready for test execution when
the software products become available.
1.10 Throughout the test execution period, regular checkpoints will be taken (typically by
the taking of physical database dumps and the preserving of associated flatfile and
configuration data) at suitable quiescent points. This allows tests to be restarted, at a point
prior to the point of failure, on receipt of a fix, or to re-run appropriate segments of a test
when regression testing becomes necessary because of changes to the system. This is
particularly important during the running of any particularly lengthy tests suites, where the
cost of such re-testing would otherwise quickly become prohibitive.
1.11 In addition during the test execution period, the test results of formal test runs, which
are retained for examination by the test manager, will be kept as an audit trail for each
‘final’ run.
1.12 The general approach to testing is one of staged, systematic verification, with
progressive integration of software and hardware components, first stabilising the
environment and business functionality, then system, performance, operability, security,
etc., and culminating in overall service validation of the fully configured system in a Live-
like environment.
© 2002 Fujitsu Services Commercial-in-Confidence Page: 10 of 1
FUJ00001673
FUJ00001673
Fujitsu Services Testing and Integration Strategy Ref: VUSTR/001
Version: 3.0
Commercial-in-Confidence Date: 16/07/2002
1.13 The system is subjected to testing against three principal test life-cycles, for Functional
Conformance, Architectural Conformance and Business Integrity. These give rise to a
number of separate stages of testing - Unit Test, Product Integration Test, System Test,
Direct Interface Test, Business Integration Test, Volume & Integrity Test, Release Test,
Customer Testing, and Live Support Test. There may also be a Pilot or and Live Trial.
(Note: Customer Testing, some DIT phases, and Pilot or Live Trial, are owned by the
Customer, not Pathway.) It is important to note that some traditional test activities such as
‘performance testing’ and ‘security testing’ do not exist as discrete activities under this
approach, but rather they are integrated into each progressive stage of testing and
integration.
1.14 Unit Test (UT) deals with the detailed verification of individual modules and their low
level linking to form basic products. It is performed explicitly for all products developed in-
house. An equivalent level of testing is assumed to have been performed as a minimum for
any 3" party products (the level of testing that 3 party products are subjected to before
interception into Pathway is resolved on a product by product basis). It comprises
Prototyping & Design Feedback, Code Review, Module Test, and Link Test. It is normally
confined to a single architectural layer (e.g. the Counter Platform). In addition, for key 3%
party products, a Product Acceptance Test may be run.
1.15 Product Integration Test (PIT) is concerned with the initial integration of unit tested
products, configuring them correctly in appropriate test environments, according to their
respective platform specifications, to form whole systems for use in later stages of testing.
1.16 System Test (ST) serves to validate the software against the requirement,
concentrating on functional conformance. It operates the products in conjunction with
each other, across the different architectural layers (e.g. from Counter through Agent to
Host) but is normally confined to a single business application (e.g. APS) or infrastructure
system (e.g. KMS).
1.17. Direct Interface Test (DIT) is concerned with early verification of the various
interfaces between the Pathway systems and external systems (e.g. Customer systems,
such as TIP, and Client systems, such as for Automated Payments).
1.18 Business Integration Test (BIT) exercises the system product set in a realistic
environment in terms of platform architecture and business operation. The testing focuses
on the interaction and data flows across the product set. Emphasis is given to the system’s
points of reporting, either a designed system report or an output to an external interface,
to enable an end to end reconciliation of the systems operation.
1.19 Volume and Integrity Test (VIT) is concerned with the end-to-end data flow of
volumes of data through individual system components and across the end-to-end
architecture of the system. Data integrity and system resilience/recovery under load is
proven. Measurements on throughput/performance are taken for modelling purposes.
© 2002 Fujitsu Services Commercial-in-Confidence Page: 11 of 1
FUJ00001673
FUJ00001673
Fujitsu Services Testing and Integration Strategy Ref: VUSTR/001
Version: 3.0
Commercial-in-Confidence Date: 16/07/2002
1.20 Release Test (RT) provides an environment for the migration and implementation
activities to be successfully rehearsed in a Live-like environment. The associated build
scripts, operational procedures and release documentation are tried and proven before live
usage.
1.21 Live Support Test (LST) is concerned with the interception of software changes to be
delivered to the live system outside the scope of a full or interim release. The necessary
change-specific tests, together with selected and more general regression tests, are
executed for targeted areas of the Pathway Solution. LST is upgraded with all full and
interim releases, at a point just before the software goes live, so it can continue to act as
the live support environment.
1.22 Each successive stage of testing is conducted in appropriate types of test environment,
which progressively get larger, more shareable, less simulated, more life-like and under
stricter CM control from stage to stage. This helps to concentrate test activity in more
affordable and more appropriate environments and so avoids unnecessary escalation of
machine resource costs.
1.23 In addition to the testing and integration activities performed and controlled by
Pathway, there will normally also be a range of Customer Testing. This will be defined,
and controlled by the Customer. Pathway will build and maintain the test environments for
this activity, and the requirements for this will need to be agreed in advance. The
management of the system and/or the running of the tests themselves may also require
Pathway support. Again, requirements for this will need to be agreed in advance. This
testing will focus on the end-to-end integrity of the overall business system, including
Customer systems, and any 3" Party systems which may be involved, and taking into
account all ancillary materials which complement the software systems. These will include
business, IT support, end-user, and operational procedures; help systems; training
materials.
1.24 A Pilot or Live Trial approach is usually adopted to limit the initial exposure of the
Live Estate to the new release, and so affords the opportunity to reduce the likely business
impact of any potential bugs remaining in the systems. Where practical, it is highly
desirable to include some gradual ramp-up of volume, to avoid unexpected performance
issues.
1.25 Asa general philosophy, it is important to accept that no system can ever be confirmed
as completely error-free. It is not possible to prove it. Tests can prove that an error exists.
They can prove that a previous error has been corrected. They cannot prove that no
further errors remain. However, by concentrating on the important characteristics of the
system operation, tests can be used to demonstrate the progressive removal of errors to
the point where these characteristics are seen to conform to expectation. So, testing can be
seen as a method (the primary method) of reducing the risk of serious defects remaining in
important areas of the system. We say the system is ‘Acceptable’ or ‘Fit for Purpose’. The
cost of this error finding and removal process follows a course of increasing cost and
diminishing return. The equation is one of Cost versus Risk. Different systems can abide
varying levels of risk. These are defined in the agreed Acceptance Criteria for the system.
© 2002 Fujitsu Services Commercial-in-Confidence Page: 12 of I
FUJ00001673
FUJ00001673
Fujitsu Services Testing and Integration Strategy Ref: VUSTR/001
Version: 3.0
Commercial-in-Confidence Date: 16/07/2002
1.26 Therefore, testing and integration will be managed, throughout both the planning and
execution phases, on a strict cost benefit basis. A pragmatic approach is essential in
maintaining the correct balance - Cost versus Risk. During test analysis it will become
apparent that certain areas of the system are deserving of more scrutiny than others,
because of the inherent risks they entail (complexity, sensitivity, potential impact, etc.).
Similarly certain areas will be deserving of less scrutiny, being of relatively low risk
impact. Also, during the course of test execution, as the characteristics of the components
under test are revealed, problems encountered, and priorities revised, again the tests
planned for certain areas may be reviewed, supplementing them where risk outweighs cost,
and deferring or discarding them where cost outweighs risk. In general this process will be
at the discretion of the Test Manager concerned, though where this leads to a deviation
from the agreed scope and coverage, then this must also be agreed with the programme.
© 2002 Fujitsu Services Commercial-in-Confidence Page: 13 of 1
FUJ00001673
FUJ00001673
Fujitsu Services Testing and Integration Strategy Ref: VUSTR/001
Version: 3.0
Commercial-in-Confidence Date: 16/07/2002
2. SCOPE OF TESTING
2.1 The scope of testing and integration activities encompassed by this document and its
subordinate test strategies includes the entire application software system (both 3rd party
supplied and in-house developed products) and its integration with the supporting
hardware, support products and services, and infrastructure software platform.
2.2 It does not cover specific detailed testing of the platform (which is expected and assumed
to be of Assured Status) other than in respect of its support of the software system and in
satisfying the various Service Level Agreements (SLAs) and Non Functional Requirements
(NFRs). So, for example, it is not expected that the proprietary Operating Systems,
Database Management Systems, Messaging Systems, Network Management Systems, etc.
will be subject to verification, but only that the Pathway systems intended use of those
components is supported as intended.
2.3 It does not cover the testing activities of the 3rd party suppliers. It is expected and
assumed that all products supplied will be appropriately tested prior to delivery to ensure
that they meet the supplied specifications. However, it is not assumed that these suppliers
will have integrated and proven their various products together, excepting where a
supplier is charged with producing inter-linking products, where link testing is expected
prior to delivery.
© 2002 Fujitsu Services Commercial-in-Confidence Page: 14 of I
FUJ00001673
FUJ00001673
Fujitsu Services Testing and Integration Strategy Ref: VUSTR/001
Version: 3.0
Commercial-in-Confidence Date: 16/07/2002
3. HIGH LEVEL TEST OBJECTIVES
3.1 Specific
For each release, tests are to be engineered to demonstrate the following:
a)
b)
c)
d)
8)
h)
To provide the necessary design feedback information by conducting informal trial
runs of early software releases and hardware platforms, to allow the design and
development of the related products to be completed prior to their entry into the
later testing stages, and so to reduce the level of disruption to those stages which
may otherwise result from consequent late changes to the design (Unit Test —
Technical Evaluation)
each module developed in-house to be compliant with the corresponding module
specification and to link correctly with co-operating modules in that product. (Unit
Test — Code Review, Module Test, Link Test).
key bespoke software products provided by a 3rd party supplier, to be compliant
with its Product Specification. (Unit Test - Product Acceptance Test).
software and hardware components to be correctly configured to support testing
and subsequent live running. (Product Integration Test)
software products to operate successfully in conjunction with one another to form
application or infrastructure systems which satisfy the relevant Requirement and
Design Specifications (System Test).
Pathway systems to interface correctly, on an individual basis, with external
systems, in compliance with the external interface specifications. (Direct Interface
Test)
that products are that are unchanged at a release are not affected by the
introduction of new system features, achieved through the execution of adequate
regression testing (System testing, Business Integration Test, Volume and Integrity
Test)
software systems to operate and co-operate successfully together on the target
hardware and infrastructure software platform, and to interface correctly together,
satisfying the Requirement Specifications, including any SLAs and NFRs.
Application and system data integrity to be proven within all points of reporting
through controlled E2E data flow. This to include verification of Output Handling
Equipment (OHE) operation as required by the system, such as card production,
use of ‘slip printers’ at the counters and all associated stationery and materials,
where products are available. (Business Integration Test, Volume and Integrity
Test)
to validate the end-to-end technical architecture employed. (Business Integration
Test, Volume & Integrity Test, Release Test)
© 2002 Fujitsu Services Commercial-in-Confidence Page: 15 of I
FU.
Fujitsu Services Testing and Integration Strategy Ref: VUSTR/001
Version: 3.0
Commercial-in-Confidence Date: 16/07/2002
FUJ00001673
IJ00001673
m
n)
0)
to demonstrate that the system meets the specified and agreed levels of
functionality and performance, and so is fit for service. (Business Integration Test,
Volume & Integrity Test)
the migration and implementation activity to be successfully rehearsed in a Live-
like environment, including verifying timings and schedules, and to validate
(through usage) the supporting systems management and software distribution
facilities. (Release Test)
to successfully operate following agreed procedures (business, user, operational),
help systems, and training materials (Customer Testing)
all systems comprising the overall business system (including Customer systems
and 3" party systems) to operate and co-operate correctly on an end-to-end basis,
maintaining integrity (outputs, system data, financial status) throughout a cycle of
events (Customer Testing)
confirm that all necessary preparations have been made to go-live, and that the
support environment is properly configured. (LST)
system to successfully support the ‘Live Trial’ in verifying user procedures,
training material, support mechanisms, and help desk procedures, and to confirm
the migration and implementation prior to nationwide rollout and usage. (Live
Trial).
3.2 General
For each release tests are to be engineered with particular regard to the following areas of
good general practice:
a)
b)
c)
d)
to ensure that as much testing as possible is performed as early as possible in the
lifecycle, to reduce defect correction costs and avoid unwelcome schedule
disruption late in the lifecycle.
to arrange for as much testing as possible to be of an automated and re-runable
nature, to reduce regression test costs, to speed test execution times, and to avoid
unnecessary levels of human error.
to comply with the General Testing Policy [1].
to adopt a joint (Pathway and Customer) approach to testing, combining test
objectives, high level test planning, and test execution activities wherever
practicable, attacking multiple objectives in combined tests and so reducing
duplication of effort and minimising the overall elapsed time required for effective
testing.
© 2002 Fujitsu Services Commercial-in-Confidence Page: 16 of I
FUJ00001673
FUJ00001673
Fujitsu Services Testing and Integration Strategy Ref: VUSTR/001
Version: 3.0
Commercial-in-Confidence Date: 16/07/2002
4, APPROACH
4.1 The approach is one of staged, systematic verification, with progressive integration of the
wide ranging system components, culminating in overall validation of the fully configured
system within a near-Live environment, followed where practicable with a period of Pilot
or Live Trial running. It is a continuous process, spanning the whole development
lifecycle, commencing with test analysis and the production of formal test material, and
progressing through to test execution, result checking, and defect removal. Static Testing
techniques are also employed where advantageous (e.g. Code Review).
A partnership approach between Pathway and Customer is key in securing the most
efficient and effective means of testing and integrating the system. Pooling skills will
enable clearer focus on test objectives at the outset, will promote higher quality test
planning and scripting, and will help reduce the elapsed time necessary for test execution
through co-operative effort and reduced duplication.
The component products are moved through distinct, separately planned and executed
stages of testing, each designed to progress the products to a higher level of assurance.
Once a stage of testing has been completed or has reached an agreed point for a particular
product set, then that product set moves to ‘Assured Status’ and is deemed to be ready for
use in the next stage and for progressively wider integration with their co-operating
product sets. A proportion of the products that make up the system are provided by 3rd
party suppliers. So, the emphasis in testing can be toward ‘black box’ techniques. That is,
the detailed inner workings of supplied products are not examined and put under test, but
rather their gross behaviour is verified in the context of the services they are required to
perform.
4.2 It is therefore particularly important that the Business Requirement is firmly understood at
the outset. Here, the Business Requirement is taken as being encompassed by the agreed
Requirement Specification and System Design Specification, and any associated Service
Level Agreements and Acceptance Criteria. It is of paramount importance that these be
maintained under strict Change Control, with the testers being included in the impact
assessment process for any and all changes to this baseline.
Similarly it is important that an ‘Independent’ approach be adopted to maintain objectivity
in developing the ‘black box’ tests necessary to successfully verify and integrate the mixed
product set. That is, that products be treated alike, irrespective of whether they be
supplied by a third party or developed in-house, with formal product handover into testing
being established via formal handover mechanisms.
© 2002 Fujitsu Services Commercial-in-Confidence Page: 17 of I
FUJ00001673
FUJ00001673
Fujitsu Services Testing and Integration Strategy Ref: VUSTR/001
Version: 3.0
Commercial-in-Confidence Date: 16/07/2002
4.3 Given the condensed timescales typically available, it is also important that test planning
commences at the earliest opportunity. Testers must be involved throughout the
development lifecycle, starting with high level ‘logical’ test planning, based on the business
requirements, and progressively developing these into ‘physical’ test scripts, so that the
test material is ready and approved in good time for use in test execution when the
products are handed over into test.
This is consistent with the general recognition that testing follows a lifecycle closely
interwoven with that of development. This is best embodied in the well established
lifecycle ‘V’ diagrams, where development progresses down the left leg of the “V’ with
analysis, design and construction, and then up the right leg of the ‘V’ with unit test,
function test and system test execution. The horizontal relationships between left and right
legs indicate the test analysis, planning and preparation required against each phase.
4.4There are three distinct lifecycles of verification and validation deployed on this
programme, each bringing a different perspective. The ‘Functional Conformance’ of the
system must be evaluated. That is, on a purely functional level, confirming that the services
required of the system by the Customers, as defined in the functional requirements, are
being met. The ‘Architectural Conformance’ of the system, both software and hardware
must be evaluated. That is, the underlying system architecture employed, with extensive
infrastructure software and mixed hardware platforms, must be trialed under stress to
confirm service attributes such as performance, operability and security. The ‘Business
Integrity’ must be evaluated. Here it is necessary to demonstrate integrity across the
breadth of the whole business system, encompassing not only the hardware and software
components, but also the ancillary technical and business components (e.g. business
procedures, help and training materials, user guides, operations manuals, target reference
data, etc.).
© 2002 Fujitsu Services Commercial-in-Confidence Page: 18 of I
Fujitsu Services
Testing and Integration Strategy Ref: VI/STR/O01
Version: 3.0
Unit Test
Functional
Conformance
Product
Integration Test
Architectural
Conformance
System Test
Direct
Interface Test
I
II Business
I [integration Test
I Volume &
I Integrity Test
I
Release Test
User Test
Live Support Test
Live Trial
I
1
I
I
I
Figure 4.1 - Mapping of Testing Lifecycles onto the Stages of Testing
FUJ00001673
FUJ00001673
These three testing lifecycles - Functional Conformance, Architectural Conformance, and
Business Integrity — are not themselves testing activities, conducted as separate activities,
but rather they are perspectives that each testing stage must take account of. They map
onto each other as shown above.
For a given release, test analysis should start in parallel with formulating/agreeing the
business requirements, contract schedules, acceptance criteria, and system requirements.
The emphasis at this stage is twofold - to develop a thorough understanding of the overall
business system and how the Pathway Solution is expected to contribute, and to influence
the system design to promote testability. (If any of these key inputs are late in production,
or are subject to significant late change, then it is likely to severely hamper this early and
crucial test planning activity, resulting in significant rework.)
© 2002 Fujitsu Services
Commercial-in-Confidence Page: 19 of 1
Fujitsu Services
Testing and Integration Strategy
Commercial-in-Confidence
FUJ00001673
FUJ00001673
Ref: VIU/STR/001
Version: 3.0
Date: 16/07/2002
Following this strategic framework, the Business Release Test Strategy (BRTS) is
produced. This is specific to the release, and describes any special characteristics it may
have (from a testing perspective) and what changes/additions to the general approach are
required. For example, a release may introduce new business applications on the desktop,
perhaps extending the memory footprint occupied, and so may require particular tests to
recalibrate the performance model for the counter platform. Or, there may be fundamental
changes being made to a sensitive piece of the infrastructure, such as the Riposte Message
Server, which will require significant system-wide regression testing. Or, concerns may
have been raised, or ‘hot-spots’ identified in particular areas of the system, which will
require specific attention in testing. All such release-specific adjustments to the strategic
approach will be documented and agreed in the BRTS.
4.5 Functional Conformance.
Requirements
Analysis
System
Design
Product
Design
Construction
Business
Integration
Direct Test
Interface
Test ——
~~ System
Purchase
or
Product
Acceptance/ Mogute™
Test
contract
Code Review
Figure 4.2 - The
‘Functional
Conformance’
Testing Lifecycle
The
Pathway
Solution comprises a
mixture of bespoke,
in-house developed
products, and 3%
party products.
These 3" party products may be shrink-wrapped, commercial off-the-shelf products, or
they may be bespoke products commissioned from a 3" party developer.
Products developed in-house will follow the established Pathway Unit Test processes, with
selected modules being subject to formal Code Review, all modules being subject to
Module Test, and where appropriate closely associated modules being subject to Link
Test.
Products supplied by a 3" party supplier, whether they are bespoke developments, or off-
the-shelf products, are expected to have been appropriately tested by the supplier prior to
delivery to Pathway. Selected 3" party products will in addition be subject to Product
Acceptance Test, which may involve verifying that agreed acceptance criteria have been
met.
These activities are all collectively classed as Unit Test UT). This brings all products up to
a common level of trusted status whereby all in-house products will conform to their Low
Level designs (LLDs), and 3" party products will satisfy their acceptance criteria. At this
stage they are placed under formal CM control.
© 2002 Fujitsu Services
Commercial-in-Confidence
Page: 20 of 1
FUJ00001673
FUJ00001673
Fujitsu Services Testing and Integration Strategy Ref: VUSTR/001
Version: 3.0
Commercial-in-Confidence Date: 16/07/2002
Before the products can be deployed ready for System Test, they need to be correctly
configured on their target platforms, which in turn need to be correctly built. This detailed
technical integration is achieved by Product Integration Test (PIT).
Each system, whether an application system (e.g. APS) or an infrastructure system (e.g.
KMS) is typically made up of a number of products operating across a number of
platforms. This cross platform integration is achieved by System Test (ST). It is normally
here that a system can first be run as a whole, and the principal objective is to demonstrate
that it conforms to the functional requirements specified for the system. System Test will
normally proceed following the established 3-Pass model. The ‘I* Pass’ is run in a
relatively informal fashion, forcing the tests through where necessary. It is intended to
stabilise the system, the test environment, the test scripts and test data. This will identify
the majority of ‘stoppers’ for correction prior to running the ‘Main Pass’. The main pass is
typically run in an iterative fashion. The ‘Main Pass’ is the principal defect removal work
horse of System Test. Once the system is demonstrated to be more or less conformant to
the functional requirements, then the ‘Final Pass’ is run to capture a formal audit trail.
As soon as the system has become stable then work can commence on verifying the
external interfaces. This is achieved by Direct Interface Test (DIT). This is a jointly
planned and executed activity. (Jointly with the owners of the interfacing systems.)
Once the majority of the serious defects have been corrected and the system(s) are
becoming more and mores conformant to the functional requirements (i.c. about half way
through ST Main Pass) then work can start on bringing all the infrastructure and
application systems together to form the integrated Pathway Solution. This is achieved by
Business Integration Test (BIT). This also normally follows the 3-Pass Model. The
executed cycles operate end-to-end (within the confines of the Pathway Solution) to
confirm all the various components operate correctly together to satisfy all the functional
requirements.
This completes the Functional Conformance testing.
© 2002 Fujitsu Services Commercial-in-Confidence Page: 21 of I
FUJ00001673
FUJ00001673
Fujitsu Services Testing and Integration Strategy Ref: VUSTR/001
Version: 3.0
Commercial-in-Confidence Date: 16/07/2002
4.6 Architectural Conformance.
inplementnton Keleaee Tess Figure 4.3 - The
SLAs ‘Architectural
reign Conformance’
NERS Testing Lifecycle
— merce : PIT, ST, DIT,
Test
and BIT, in
addition to their
roles in
Functional
Conformance
(see 4.5 above) also play a role in Architectural Conformance. When these tests are
planned, any relevant Non-Functional Requirements (NFRs) are also taken into account.
Sys
Specifications
Product
Integration Test
Product/P latform
Specifications
In PIT, the technical characteristics of the run-time platform, as specified in the TED [5],
must be correctly applied, or where not defined explicitly, they may have to be prototyped.
Of particular importance here is any security requirements of the build.
In DIT, the volume, throughput, and performance characteristics specified, and any
operational SLAs that may have been agreed for the interface, should be observed, though
the limited nature of the DIT environment precludes any stress testing.
In ST and particularly in BIT, the scope and coverage of the tests should be extended to
cover all explicitly defined NFRs, except for volume, throughput, and performance
(excluding counter performance which is within the remit of BIT). These should include
HCI, output handling, operability, usability, security, fail-over, fall-back, recovery,
archiving, audit, and systems management (all subject to the limitations of the test
environments concerned).
In addition, in BIT, special provision is made to cover volume, throughput, and
performance for the counter platform (i.e. up to the ISDN interface). A special ‘large
outlet’ configuration of 20 counters has been included in the primary BIT test rig for this
purpose, and tooling has been developed to allow automated activity following
configurable transaction profiles.
Once BIT has stabilised the systems, allowing the Pathway Solution to be run as a whole
(i.e. after BIT 1* Pass) then work can start on conducting large scale volume tests and
stressing the system in other ways (e.g. major recovery operations, loading the ISDN
network, and the external interfaces, saturating batch windows, etc.) to confirm that the
overall system and data integrity is maintained, and SLAs are satisfied. This is achieved by
Volume & Integrity Test (VIT).
© 2002 Fujitsu Services Commercial-in-Confidence Page: 22 of I
FUJ00001673
FUJ00001673
Fujitsu Services Testing and Integration Strategy Ref: VUSTR/001
Version: 3.0
Commercial-in-Confidence Date: 16/07/2002
In parallel with this, the Migration and Implementation activities are rehearsed, including
regression, and the Pathway Solution is operated in a Live-like fashion, employing the
target systems management facilities, and confirming that the introduction of any new or
changed infrastructure does not de-stabilise the established architecture. This is achieved
by Release Test (RT).
This completes the Architectural Conformance testing.
4.7 Business Integrity.
Pilot or Live Trial
nppoood
esoutsn
Live Support Test
‘s0
sgiepare
User Test
Release Test
Volume &
Integrity Test
Business
Integration Test
Figure 4.4 - The ‘Business Integrity’ Testing Lifecycle
BIT, VIT, and RT, in addition to their roles in Architectural Conformance (see 4.6 above)
also play a role in Business Integrity. This is implicit, because the Pathway Solution forms
such a large proportion of the overall business system, and is in many respects pivotal in
this area.
In addition, extensive reconciliation facilities are built into the Pathway Solution, and
proving their functional conformance goes a long way toward providing assurance that end-
to-end financial integrity is achieved
© 2002 Fujitsu Services Commercial-in-Confidence Page: 23 of I
FUJ00001673
FUJ00001673
Fujitsu Services Testing and Integration Strategy Ref: VUSTR/001
Version: 3.0
Commercial-in-Confidence Date: 16/07/2002
When BIT is nearing completion (about two-thirds through BIT Main Pass) then the
Pathway Solution is sufficiently stable to support Customer Testing, which will explicitly
test the end-to-end integrity of the overall business system. This will involve running the
Pathway Solution together with all other contributing systems (Customer systems and 3"
party system), and employing all ancillary materials (such as procedures, help systems,
training, user guides, etc.).
Just before the release is implemented, LST is upgraded so it can then support the live
system post go live. This upgrade acts as a final migration test and also proves the live
keys.
The Pilot or Live Trial provides a final opportunity to identify any failures in business
integrity by operating a full Live service for a restricted number of outlets, before the
release is rolled out nationwide.
This completes the Business Integrity testing.
© 2002 Fujitsu Services Commercial-in-Confidence Page: 24 of I
FUJ00001673
FUJ00001673
Fujitsu Services Testing and Integration Strategy Ref: VUSTR/001
Version: 3.0
Commercial-in-Confidence Date: 16/07/2002
5. RESPONSIBILITIES
5.1 This section lays out a general framework of responsibilities for testing of the Pathway
Solution. The terms and roles used are intended to be generic rather than reference individual
job titles, which will change from time to time during the course of a large project. For
example, the roles of Quality Manager and Risk Manager may well be encompassed by a
single individual or job title, whilst Development Manager may conversely be distributed over
many individuals.
Role Responsibilities
Customer Review Test Strategies and provide input to High Level Test Plans. Be
Testing/Release involved in witnessing some later stages of testing. Review Pathway testing
results. Plan, control, and perform Customer Testing. Plan, control, and co-
Manager ordinate Pilot or Live Trial. Accept the System.
Pathway Testing I Review/Sign-off Strategies and Plans. Project Management of testing and
Manager integration activities. Interface with all areas.
Pathway Review/Sign-off Testing and Integration Strategy. Recognise testing
Development dependencies in project plans. Provision of resources. Setting of Priorities.
Manager Agreement of changes to scope/coverage. Provide adequate bug-fix
turnaround.
Pathway Review/Sign-off test plans for performance and other NFR coverage. Specify
Architecture technical platform requirements.
Manager
Pathway Set operational requirements. Review/Sign-off test plans for service
Operations management and operability. Close involvement in running of later stages.
Manager
Pathway Apply QMS in development and testing areas. Confirm handover process.
. Check conduct of tests. Collect and interpret quality metrics. Check quality
Quality
records.
Manager
Pathway Conduct risk assessments. Approve cost versus risk evaluations. Produce
Risk residual risk report. Input to acceptance process.
Manager
Pathway Conduct handovers. Review environmental status against and confirm
CM integrity of configurations.
Manager
Pathway Review/Sign-off test plans for security and access coverage. Confirm data
retention periods. Check security status for different environments / data
© 2002 Fujitsu Services Commercial-in-Confidence Page: 25 of I
Fujitsu Services
FUJ00001673
FUJ00001673
Testing and Integration Strategy Ref: VI/STR/O01
Version: 3.0
Commercial-in-Confidence Date: 16/07/2002
Security usage.
Manager
Supplier Ensure acceptable product quality on supply. Provide adequate bug-fix
Manager turnaround
© 2002 Fujitsu Services Commercial-in-Confidence Page: 26 of I
Fujitsu Services Testing and Integration Strategy Ref:
Version:
Commercial-in-Confidence Date:
FUJ00001673
FUJ00001673
VIU/STR/001
3.0
16/07/2002
6. DELIVERY UNIT TEST
6.1 Composition
Unit Test
Code Review
Module Test
Link Test
Product Acceptance Test
Product Integration Test
System Test
Direct Interface Test
(note: DIT may on occasions be performed by PTU rather than Development)
6.2 Unit Test
6.2.1 Code Review
6.2.1.1 Context
Optional, at the discretion of the Delivery Unit Manager, though recommended for
complex areas of code. Normally performed by a peer developer or team leader within
the Delivery Unit concerned. Applies to products developed in-house only. Requires
no Customer involvement. Results formally recorded and retained.
6.2.1.2 Objectives
¢ To identify obvious discrepancies between the module code as implemented and the
detailed Module and API Specifications in the Low Level Design.
e To confirm that the agreed coding standards have been followed
¢ To assess the maintainability of the code in question
6.2.1.3 Overview
© 2002 Fujitsu Services Commercial-in-Confidence
Page: 27 of 1
FUJ00001673
FUJ00001673
Fujitsu Services Testing and Integration Strategy Ref: VUSTR/001
Version: 3.0
Commercial-in-Confidence Date: 16/07/2002
DESIGN
Module & API
Specifications
Module — Code
Source Code . Review
CONSTRUCTION
Figure 6.1 - Schematic Overview of Code Review
6.2.2 Module Test
6.2.2.1 Context
Mandatory. Normally performed by developer of the module concerned. Applies only
to products developed in-house. Requires no Customer involvement. Formally
planned, though data may be ad hoc. Results formally recorded and retained.
6.2.2.2 Objectives
e To demonstrate that each module developed in-house conforms to the detailed module
specification in the Low Level Design.
¢ To demonstrate that the unit conforms to the relevant HCI where applicable.
e To provide an economic basis for ongoing regression testing of each module as and
when they are subject to change.
6.2.2.3 Overview
© 2002 Fujitsu Services Commercial-in-Confidence Page: 28 of I
FUJ00001673
FUJ00001673
Fujitsu Services Testing and Integration Strategy Ref: VUSTR/001
Version: 3.0
Commercial-in-Confidence Date: 16/07/2002
DESIGN
Unit Test Preparation
Modi
& API [Module I
a Module Link
Runtime ~ Module
Link
Modules Test Runs Test Runs
Unit Test Execution
CONSTRUCTION
Figure 6.2 - Schematic Overview of Module & Link Test
6.2.3 Link Test
6.2.3.1 Context
Mandatory. Normally performed by (one of) the developer(s) of the linking modules
concerned. Applies only to products developed in-house. Requires no Customer
involvement. Results formally recorded and retained.
6.2.3.2 Objectives
e To pre-integrate co-operating modules within a product by demonstrating that their
APIs are implemented correctly and so that the modules link together as specified and
co-operate properly as an integrated unit.
6.2.3.3 Overview
(See figure 6.2 at section 6.2.2.3)
6.2.4 Product Acceptance Test
6.2.4.1 Context
Optional, at the discretion of the Delivery Unit Manager, though strongly
recommended. Normally performed by a developer of a related product, or by a tester
with experience of a related product. Applies only to products developed by 3" party
suppliers. Requires no Customer involvement. Formally planned. Results formally
recorded and retained. May have contractual significance.
6.2.4.2 Objectives
© 2002 Fujitsu Services Commercial-in-Confidence Page: 29 of I
FUJ00001673
FUJ00001673
Fujitsu Services Testing and Integration Strategy Ref: VUSTR/001
Version: 3.0
Commercial-in-Confidence Date: 16/07/2002
¢ To confirm that the 3" Party supplied product is fit for entry into wider Pathway
Testing, being sufficiently stable to cause no disruption, and conforming broadly to
specification.
e (Where there are formal or contractual acceptance criteria agreed between Pathway
and the 3" Party supplier concerned) to confirm that the product satisfies the specified
acceptance criteria.
6.2.4.3 Overview
Product Description
Product Acceptance
SA
Test Plans & Scripts
a ,
'
Product Acceptance
Test Execution
v
Accepted Products
Acceptance Criteria
34 Party supplied Product
Figure 6.3 - Schematic Overview of Product Acceptance Test
© 2002 Fujitsu Services Commercial-in-Confidence Page: 30 of I
FUJ00001673
FUJ00001673
Fujitsu Services Testing and Integration Strategy Ref: VUSTR/001
Version: 3.0
Commercial-in-Confidence Date: 16/07/2002
6.3 Product Integration Test
6.3.1 Context
Mandatory. Performed by PIT team. Applies to all products whether developed in-
house or supplied by 3% Party, whether Application or Infrastructure, whether
Software or Hardware. Requires no Customer involvement. Formally planned. Results
formally recorded and retained.
6.3.2 Objectives
¢ To establish and prove the correct build instructions for each testing platform
e To integrate products onto these platforms accordingly and so establish their
configuration
¢ To prove their environmental stability
¢ To enforce correct initial configuration management of the testing environments
e To intercept change to these environments, reintegrating and reproving the
configuration as appropriate
6.3.3 Overview
The processes used in PIT and related activities in the CM, SPTS, and Tivoli
Packaging areas are presently under review. The results of this review will be reflected
in this document at version 4.0, together with any other changes that may be required
following its review by the Customer. In the meantime, the following diagram is
extracted from VI/STR/010 [4] to serve as a brief summary of the current strategic
approach. For further detail, refer to the parent document. However, it should be
© 2002 Fujitsu Services Commercial-in-Confidence Page: 31 of I
FUJ00001673
FUJ00001673
Fujitsu Services Testing and Integration Strategy Ref: VUSTR/001
Version: 3.0
Commercial-in-Confidence Date: 16/07/2002
noted that this is likely to be subject to change, and may not match current practice.
Pir on sers
x1, I ——__
Maesico, I 4 ®
atl, I > B 0
Riposte I ————> I shen I > I i
shel : i st
- I ¢ .
: SY 1 a
Appl. A 1 i
‘ts I —————»_I App o m pit
—_—_—_ i BIT
App. 8 I —————> ma :
Pe Appl. B
(rd Party) >
a : :
: :
PinICt : : : :
ip I I= D 2
PinTCL 1
759 ‘
—_____ ft
486
apply deltas to
no config change 2 baselines
Figure 6.4 - Schematic Overview of Product Integration Test
© 2002 Fujitsu Services Commercial-in-Confidence Page: 32 of 1
FUJ00001673
FUJ00001673
Fujitsu Services Testing and Integration Strategy Ref: VUSTR/001
Version: 3.0
Commercial-in-Confidence Date: 16/07/2002
6.4. System Test
6.4.1 Context
Mandatory. Applies to all systems comprising the Pathway Solution, irrespective of origin.
Performed by the System Test Team within the Delivery Unit concerned, on receipt (via
CM) of products from Unit Test and/or PIT. Formally planned, scripted and executed,
preferably in re-runnable fashion. Test Results formally recorded and retained.
6.4.2 Objectives
* To demonstrate, through a series of comprehensive business driven scenarios, that the
software system concerned, functionally conforms with the agreed Requirements.
e To expose the Infrastructure software to use by the application, and to demonstrate
that within the limitations of a reduced hardware platform it provides the application
with the specified support.
¢ To perform initial verification of OHE requirements where available equipment,
stationery and other materials allow.
e To demonstrate simple recovery and resilience features of the system within the
bounds of the platform in use.
e To form a comprehensive system regression pack for later use, and to prepare the way
for full system integration in the following stages of testing.
6.4.3 Overview
Requirement Specification
p
Design Specification
t System Test
Plans & Scripts
Acceptance Criteria
\
TED, including derived NFRs 1
System Test
Execution
%
ex Functionally Conformant Syste:
Runtime Products —
(passed Unit Test)
FUJ00001673
FUJ00001673
Fujitsu Services Testing and Integration Strategy Ref: VUSTR/001
Version: 3.0
Commercial-in-Confidence Date: 16/07/2002
Figure 6.5 - Schematic Overview of System Test
6.5 Direct Interface Testing
6.5.1 Context
Optional. Applies when the AIS or EPID for a defined interface changes or a new
interface is introduced. Usually performed by the Delivery Unit concerned (or on
occasion by PTU, whichever is best placed to conduct the tests under the prevailing
circumstances. This would be specified in the BRTS for the release concerned.)
Formally planned, scripted and executed. Test Results formally recorded and retained.
Completed once the sending/receiving systems have declared that the system’s are in a
state of readiness for DIT to be commenced. (DIT may actually be planned and co-
ordinated by POL, but supported by Fujitsu and the other relevant supplier domains.).
6.5.2 Objectives
e That the interface under scrutiny, has been correctly interpreted within the system
changes or new requirements defined.
e That operational integrity of the interface under test will be maintained.
e Ifrequired, that the defined interface can handle the required volume/throughput and
satisfy any SLAs specific to the interface.
6.5.3. Overview
© 2002 Fujitsu Services Commercial-in-Confidence Page: 34 of I
FUJ00001673
FUJ00001673
Fujitsu Services Testing and Integration Strategy Ref: VUSTR/001
2
Interface Specification
Interface Test
Design Specification Plans & Scripts
NF
NFRs & SLAs for the I/F {
Interface Test
Execution
Y
Proven Interface
Runtime Products —
(Stable — ST 1‘ Pass)
Figure 6.6 - Schematic Overview of Direct Interface Testing
© 2002 Fujitsu Services Commercial-in-Confidence Page: 35 of I
FUJ00001673
FUJ00001673
Fujitsu Services Testing and Integration Strategy Ref: VUSTR/001
Version: 3.0
Commercial-in-Confidence Date: 16/07/2002
7. PTU TESTING
7.1 Composition
Business Integration Test (BIT)
Volume & Integrity Test (VIT)
Release Test (RT)
Live Support Test (LST)
Development , \ } }
User
Confidence
Trial
Customer
Testing
a Business
Integration
YANN" Test
a Live
P TU Volume & ie Support
Testing Integrity Test
Tee IX ae
Release
Test
Live
System
Figure 7.1 — PTU Context Overview(Major or Interim Releases)
7.1.1 Business Integration Test
7.1.1.1 Context
Mandatory. Performed on receipt (via CM) of products from System Test Teams.
Formally planned, scripted and executed, preferably in re-runnable fashion. Test
Results formally recorded and retained. Customer involved in the test construction
© 2002 Fujitsu Services Commercial-in-Confidence Page: 36 of I
FUJ00001673
FUJ00001673
Fujitsu Services Testing and Integration Strategy Ref: VUSTR/001
Version: 3.0
Commercial-in-Confidence Date: 16/07/2002
process (providing input, reviewing HLTPs), and the execution cycles (witnessing
tests, reviewing results).
7.1.1.2 Objectives
© To demonstrate the successful integration of the software and hardware system, verifying
all components that exist within the Pathway domain.
e To exercise end to end data flows, to demonstrate maintained data integrity and the
semantics underlying the external interfaces.
¢ To prove all of the system's points of reporting and financial reconciliation including the
full set of reports generated via the counter, the host systems and the DWH.
¢ To complete performance testing up to the point of data leaving the post office (at which
point responsibility is passed to the volume testing team). Performance measurements
extend to the complete range of available counter functionality.
e To perform verification of OHE requirements where available equipment, stationery and
other materials allow.
¢ To ensure the system successfully operates against the varying office configurations, large
offices, medium sized offices, SCOs and mobiles. This extends to office working patterns
(stock unit configuration and transaction mixes) as well as available physical hardware.
© To demonstrate successful execution of acceptance criteria that have been designated for
proving during this stage.
e To ensure that the implementation of full and interim releases does not destabilise the
existing application product set, this is achieved through the establishment of a
comprehensive system regression testing pack, which can be deployed selectively or in a
blanket fashion.
e Produce a closure report detailing the results of the integration test phase, distribute to
relevant parties including the customer.
© 2002 Fujitsu Services Commercial-in-Confidence Page: 37 of I
FUJ00001673
FUJ00001673
Fujitsu Services Testing and Integration Strategy Ref: VIU/STR/001
Version: 3.0
Commercial-in-Confidence Date: 16/07/2002
Requirement Specification Procedure documents
Design Specification SA
— preeration Test
Acceptance Criteria a ans & Scripts
TED, including derived NFRs {
Integration Test
Runtime Products — Execution
(Passed System Test) {I
Functionally Integrated
Conformant System
7.1.1.3 Overview
Figure 7.2 - Schematic Overview of Business
Integration Test
7.2 Volume & Integrity Test
7.2.1.1 Context
Optional, depending upon the requirements of the release. If the release introduces a
new transaction that did not change record structures and was simply going to increase
live volumes, but remain within understood levels, then volume testing would not be
required. Performed on receipt (via CM) of products which have been through the
early stages of Business Integration Testing. Formally planned, scripted and executed,
preferably in re-runnable fashion. Test Results formally recorded and retained.
Customer involved in the test construction process (providing input) and the test
execution (reviewing results).
7.2.1.2 Objectives
© 2002 Fujitsu Services Commercial-in-Confidence Page: 38 of I
FUJ00001673
FUJ00001673
Fujitsu Services Testing and Integration Strategy Ref: VUSTR/001
Version: 3.0
Commercial-in-Confidence Date: 16/07/2002
¢ To ensure that the technical solution can support the target number of Post Office outlets.
© To ensure contractual SLAs are not breached and the system is compliant.
¢ To demonstrate successful execution of acceptance criteria that have been designated for
proving during this stage.
e To ensure system/data integrity is maintained with volume data processing.
¢ To ensure system/data integrity is maintained in the case of platform or system failure.
e To ensure that the system can support the processing of projected volumes (host to outlet
and outlet to host).
7.2.1.3 Overview
Requirement Specification Volumetric Documents
Design Specification
SA
cor I Volume Test
A
Acceptance Criteria Plans & Scripts
TED, including derived NFRs {
Volume Test
Runtime Products — Execution
(Completed first stages of J
Integration test)
Resilient, Volume
Proven System
Figure 7.3 - Schematic Overview of Volume & Integrity Test
7.3 Release Test
7.3.1.1 Context
Mandatory. Performed on receipt (via CM) of products which have been through
initial stages of Business Integration Testing and volume testing. Final phases, with the
code delivered within Tivoli wrappers, run after completion of system integration and
© 2002 Fujitsu Services Commercial-in-Confidence Page: 39 of I
FUJ00001673
FUJ00001673
Fujitsu Services Testing and Integration Strategy Ref: VUSTR/001
Version: 3.0
Commercial-in-Confidence Date: 16/07/2002
volume testing. Formally planned, scripted and executed, preferably in re-runnable
fashion. Test Results formally recorded and retained. Customer involved in the test
construction process (providing input) and in the test execution (reviewing results).
7.3.1.2 Objectives
.
To ensure that upgrades to existing services (hardware, software or database) do not de-
stabilise infrastructure components or application products.
To ensure that operational integrity is maintained during and post the applications release
into live.
To ensure that support documentation is updated to reflect the changes for a particular
release.
To demonstrate successful execution of acceptance criteria that have been designated for
proving during this stage.
To ensure the migration and implementation activities are tested and proven to maintain
data integrity.
To ensure new system features are introduced in a cohesive fashion.
To ensure the various (defined) combinations of upgraded campus to existing or upgraded
outlets functions correctly.
To ensure regression paths work and system/data integrity is maintained
7.3.1.3 Overview
Requirement Specification Migration Strategy
Support Documents SA
Release Test
=
Plans & Scripts
Acceptance Criteria
Design Specification al I
TED, including derived NFRs
Release Test
; —_—— Execution
Runtime Products uy
(Within Tivoli wrappers)
Release ready for
customer Services
Figure 7.4 - Schematic Overview of Release Test
© 2002 Fujitsu Services Commercial-in-Confidence Page: 40 of 1
FUJ00001673
FUJ00001673
Fujitsu Services Testing and Integration Strategy Ref: VUSTR/001
Version: 3.0
Commercial-in-Confidence Date: 16/07/2002
7.4 Live Support Test (LST)
7.4.1.1 Context
Mandatory. Applies to all releases, whether a major release, an interim release, or
individual bug-fixes. Performed by the LST team, immediately prior to Live
Implementation. For Major/Interim releases, the Pathway Solution will have been
subject to the full range of Pathway and Customer Testing before being passed to LST.
When simple system changes/bug-fixes are required to be delivered to live outside of
the normal defined release windows, then typically they will pass directly from
Development Testing into LST. Tests are based on stored regression packs as well as
tests generated for the specific changes. Results are formally stored and recorded. No
Customer involvement required.
7.4.1.2 Objectives
e Verification of changes that are required to be intercepted outside of normal release
windows.
e Environment for investigation and attempted reproduction of issues being investigated
within live.
¢ Live Support testing also act as the last verification point for all releases. The support
environment must intercept the release before live, to enable support to continue. As this is
performed as the final planned activity before a release goes live it also acts as a final
verification point.
7.4.1.3 Overview
© 2002 Fujitsu Services Commercial-in-Confidence Page: 41 of I
FUJ00001673
FUJ00001673
Fujitsu Services Testing and Integration Strategy Ref: VUSTR/001
Specific details relating to
the change being made.
o> Regression
Plans & Scripts
i]
Details of main release
system Enhancements.
Changed Products Live Support
— .
(Within Tivoli Wraper) Execution
Change ready for
Live
Figure 7.5 - Schematic Overview of Live
Support Test
© 2002 Fujitsu Services Commercial-in-Confidence Page: 42 of I
FUJ00001673
FUJ00001673
Fujitsu Services Testing and Integration Strategy Ref: VUSTR/001
Version: 3.0
Commercial-in-Confidence Date: 16/07/2002
8. CUSTOMER TESTING
8.1 The Pathway Solution is not alone in forming the Customer’s overall business system.
There are also a number of Customer systems, and some other 3" part systems. These
must all co-operate together correctly to maintain end-to-end integrity. Pathway is
responsible (jointly with the owners of the interfacing systems) in conducting tests to
prove that its external interfaces conform to specification. But this is not sufficient
Wider tests must be performed to demonstrate that these disparate systems do indeed co-
operate correctly and that end-to-end integrity is maintained. This is of particular
importance with regard to the financial status of the systems.
8.2. There are also a large number of ancillary components that complement the IT systems
in making up the overall business system. These include materials like manuals,
procedures, help texts, training, call centres, paper systems, etc. It is not sufficient to
test that the Pathway Solution meshes correctly with these components. It is necessary
to confirm that they do so on an end-to-end basis, all operating in concert to satisfy the
overall business system requirements.
8.3 These aspects of testing are the responsibility of the Customer. For the purposes of this
strategic framework it is sufficient to recognise that Customer Testing of this nature will
normally be required, and that a suitable test environment with appropriate technical
support will need to be provided. (This should be the subject of agreement at an early
stage.)
8.4 The precise approach adopted by the Customer in conducting these tests is outside
Pathway’s control. It will be defined in one or more test strategies produced by the
Customer, such as those produced for the Network Banking Service [6] & [7]. For the
purposes of this strategic framework, the Customer Testing activity will simply be
referred to as a User Confidence Trial (UCT). A UCT was in fact performed for the
CSR+ release, but the customer may well adopt different terminology for future
releases. This can be clarified in the BRTS.
8.5 The extent to which Pathway will be expected to provide technical support will vary
from release to release, depending on the complexity of operation. For some releases it
may be necessary for Pathway to do no more than provide the test environment, and
carry out the system management activities required for safe operation. For other
releases it may be necessary for Pathway to provide much greater support, perhaps even
having to perform the tests on behalf of the Customer. It is expected that the level of
support required will be defined in the Customer’s testing strategies.
© 2002 Fujitsu Services Commercial-in-Confidence Page: 43 of I
FUJ00001673
FUJ00001673
Fujitsu Services Testing and Integration Strategy Ref: VUSTR/001
Version: 3.0
Commercial-in-Confidence Date: 16/07/2002
9.
8.6
8.7
8.8
8.9
8.10
PILOT or LIVE TRIAL
Each major release of software is normally trialed before exposure to the full live estate.
The complexity of the release and the timescales that it is required to be delivered in
should drive the duration and scope of this trial.
Typically, before full committal, a small number of outlets, ideally ramping up from a
mere handful to 200 or 300, should perform live business with counters migrated to the
target application. These offices should be closely monitored during the trial period and
the results analysed before national rollout of the new release proceeds.
A ramp up of outlets is desirable in order to gain the maximum benefit from running the
trial. By commencing with just a handful of outlets, the potential impact of and serious
failure is limited to the bare minimum, effectively shielding the majority of the national
network of users from any adverse affects. By then increasing the number of outlets in
the trial to a few hundred, it is more likely to reveal any unforeseen volume/performance
problems, whilst still sheltering under the protection of a closely monitored and
controlled trial, and without impacting the rest of the network.
For similar reasons the target offices should cover a wide range of configurations.
Variables considered should include location (if the changes made have a variance based
on location), typical transaction volumes, office size and office connectivity type.
The first stage of the trial may be to prove that the counter application can be regressed
back to it's previous software baseline if required. The subsequent stages should be
designed to ensure that operational integrity is maintained within the target offices and
that the application changes made have been successfully implemented.
On successful completion of the trial, agreement having been reached that any erroneous
events that may have occurred during the trial have either been addressed successfully,
or do not require immediate resolution, the software can be rolled out nationwide across
the whole network.
© 2002 Fujitsu Services Commercial-in-Confidence Page: 44 of I
FUJ00001673
FUJ00001673
Fujitsu Services Testing and Integration Strategy Ref: VUSTR/001
Version: 3.0
Commercial-in-Confidence Date: 16/07/2002
9 MAINTENANCE & REGRESSION TESTING
10.1
10.2
10.3
10.4
10.5
The test plans drawn up for each part of the system and for each stage of testing are,
wherever practicable, designed to be re-runnable test packs, including the necessary
test scripts, test data, and expected results.
When a change is made to the system, this is either as a result of a fix being applied to
address an incident (a PinICL), or following the approval of a Change Request (CR)
and/or a Change Proposal (CP). Both PinICL and CP changes are formally controlled.
Part of the lifecycle of a PinICL is the planning of the testing required to satisfy it. Part
of the lifecycle of a CP is the impact assessment, which includes planning of the testing
required to satisfy it.
In both situations, the testing required normally breaks down into two distinct types.
The testing of the change itself — often conducted as targeted testing, and no different
to any already described in the rest of this strategy.
The testing required to confirm that the change has not caused the rest of the system to
regress (has not inadvertently introduced some unwanted side-effect) — regression
testing.
The change may introduce new components, or may change existing components, or
may remove existing components, or some combination of these. The targeted tests are
satisfied by introducing, changing, or removing test conditions (or adjusting the
expected results) relating to these components accordingly. The regression testing is
satisfied by identifying the neighbouring/interfacing components surrounding the
change, and in turn identifying appropriate tests which will exercise these areas of the
system. Typically this will involve scripts from each major stage of testing through the
lifecycle.
Consider 3 examples:
a) A minor change in a discrete application on the counter platform
Regression testing can be restricted to the application itself.
b) A significant change to a host application including a schema change
Regression testing should be considered across the host application, and any
others sharing that Schema, which may include related Agent applications.
Where the affected systems include an external interface, then DIT regression
runs should also be considered. Where the affected systems have critical
performance factors (or other NFRs) which may be compromised, then runs
targeted at re-proving these factors should be considered.
© 2002 Fujitsu Services Commercial-in-Confidence Page: 45 of I
FUJ00001673
FUJ00001673
Fujitsu Services Testing and Integration Strategy Ref: VUSTR/001
Version: 3.0
Commercial-in-Confidence Date: 16/07/2002
c) Major changes to fundamental pieces of system infrastructure, highly invasive in
nature
Blanket regression testing should be considered across the whole solution.
10.6 The extent of regression testing required is a matter of judgement, and in turn is
dependent on the extent of the changes applied, and the nature of the product(s)
concerned. The use of test automation can be a factor in reducing the required
timescale to execute the defined regression tests (re: Section 14)
© 2002 Fujitsu Services Commercial-in-Confidence Page: 46 of I
FUJ00001673
FUJ00001673
Fujitsu Services Testing and Integration Strategy Ref: VUSTR/001
Version: 3.0
Commercial-in-Confidence Date: 16/07/2002
Il.
EXTERNAL CERTIFICATION
Certain products employed by in the Pathway Solution will from time to time require
external certification before they can be used operationally, for legal reasons. For
example, the weigh-scales on the counter needed to be examined in operation, in
conjunction with their software, and certified by HM Weights & Measures, before they
could be used legally in live operation. Similarly, all electrical and electronic equipment
in the workplace (including the Pathway counters, their keyboards, printers, PIN-Pads,
etc.) need to pass Health and Safety regulations regarding EMR emission levels, which
requires EMC testing to be performed to obtain the certification. Any significant
changes to such configurations will require re-benchmarking and re-certification.
All such products will be formally identified for each release, together with the
authorising bodies concerned, in the BRTS for that release, and the External
Certification activity will be included in the project plans, and monitored accordingly.
© 2002 Fujitsu Services Commercial-in-Confidence Page: 47 of I
FUJ00001673
FUJ00001673
Fujitsu Services Testing and Integration Strategy Ref: VUSTR/001
Version: 3.0
Commercial-in-Confidence Date: 16/07/2002
12. TEST SCRIPTING
12.1 Test Scripting is an area that has benefited from regular and ongoing process
improvements. During CSR and CSR+ timescales the differing Pathway testing areas
attempted to implement universal standards for format and content of the test scripts
produced. It has since been recognised that at times sections of scripts were being
produced for the sake of production rather than for any great level of added value.
This is because the testing approaches within the Delivery Units and PTU have
matured and evolved to become more focused, and with less common ground
remaining to benefit from a universally applied approach. This is appropriate based on
the differing requirements of functional conformance, architectural conformance and
business integrity testing. It is also a sign that the differing testing areas are now
strongly complementing rather than duplicating each other. The same fundamental
principles are however applied within each area. The product is put through a full test
analysis process, resulting in the generation of HLTPs. These are then further broken
down, through the addition of detailed operations, expected results and test data, into
an executable script in the form of LLTSs.
12.2 The different testing areas will continue to utilise and enhance their individual analysis
methodologies. Enforced conformance to a defined generic methodology which
attempted to retrospectively cover each areas differing requirements would only lead
to redundancy and omissions within the generated scripts. Each area’s methodologies
will however not evolve without independent review. The actual approach to be
adopted will be described within the subordinate test strategies generated for each
release and will be open to critical review.
12.3. The methodologies in use will be free to mature but will be required to maintain the
following underlying principles and objectives:
e The differing stages of the process e.g. test condition construction, scenario generation
and LLTS line definition should contain as little duplication as possible in order to
minimise the overall man day effort required.
e A review should be completed early in the test construction process to avoid costly re-
work if significant errors are discovered.
¢ The tests should be presented in a format that makes them easy to understand and
therefore easy to review, execute, and check results.
¢ For appropriate test stages the documentation should allow acceptance criteria to be
easily mapped to the defined tests to avoid unnecessary duplication of effort in having
to construct separate acceptance test trials...
e When generating the execution material care must be taken to satisfy the original
testing intent expressed in the associated test condition(s), and not to allow the test(s)
to become corrupted.
© 2002 Fujitsu Services Commercial-in-Confidence Page: 48 of I
FUJ00001673
FUJ00001673
Fujitsu Services Testing and Integration Strategy Ref: VUSTR/001
Version: 3.0
Commercial-in-Confidence Date: 16/07/2002
12.4 Tooling to assist the analysis process has now been trialed and proven through the use
of Test Director to hold the Business Integration Test scripts. It has been shown that
Test Director assists the test analysis process through creating a consistent structure
and framework to script within. Although it does not necessarily reduce the effort or
elapsed time for the test analysis process it does allow easy manipulation of the
generated data, for example for the purposes of generating regression scripts. Further
benefits are also found when the test scripts are executed within Test Director. Based
on the success of this initial implementation Test Director will be assessed for use in
other testing areas.
© 2002 Fujitsu Services Commercial-in-Confidence Page: 49 of I
FUJ00001673
FUJ00001673
Fujitsu Services Testing and Integration Strategy Ref: VUSTR/001
Version: 3.0
Commercial-in-Confidence Date: 16/07/2002
13. TEST AUTOMATION / TOOLING
13.2
13.4
13.6
In the early stages of Pathway testing the technology in use was in advance of the test
automation tools available. The test automation market has matured considerably over
the last few years and this has been exploited with a number of test automation
initiatives being implemented.
The most mature use of automation on the project is that of bulk office loading. A
large office configuration (typically a 20 counter outlet) is used in conjunction with an
automated harness to replicate the day to day activity of such an office. The harness
‘drives’ the application through it's screen sequences to complete the required
transaction. No data is generated other than by the application itself. The benefits of
this implementation are obvious. The harness, operated by one user, can simulate the
whole office being used by a complete set of clerks.
The harness has been matured so that the pause between transactions and the speed
that a transaction can be completed within can be manipulated to simulate different
working patterns. In addition the harness can be easily configured so that the
percentage that each transaction type represents of all the overall transactions
performed can be changed. This means that different office locations can be simulated
e.g. rural offices completing a higher level of AP transactions than central city
locations. Most importantly the harness has been maintained to include new transaction
types e.g. post office local collect and OBCS bar coded foils. The harness will be
continually matured to include all new transaction types including network banking
transactions. It is important that a realistic transaction mix performed in a live like
manner can be run via automation to ensure that large office configurations continue to
operate successfully.
The most significant advancement in the utilisation of automation has been the
introduction of replay and edit scripts. A pilot study was commissioned to evaluate the
effectiveness of such scripts as a regression tool. The study concluded that a series of
easy wins were achievable where the time taken to automate particular areas of
functionality would show significant benefits over the time currently being taken to run
manual tests, an example area of functionality would be counter time out tests. It was
also concluded that some form of automated blitz testing script would be a powerful
tool to quickly ascertain the quality of a build or of a new code drop. In addition
although areas beyond the easy wins would not show the same benefits, it was obvious
that gains could be achieved by automating large elements of the counter testing.
An automated script was constructed and is now in use as a blitzing tool. This blitz
script executes the counter application’s main functionality and quickly discloses any
major failings. In two hours of automated execution four hours of manual execution
can be completed. The script works on the basis of a transaction or function no longer
operating in the sequence it previously performed to.
The automation scripts have also been matured to perform capture and compare.
These scripts take screen shots at any point required and compare the image to one
© 2002 Fujitsu Services Commercial-in-Confidence Page: 50 of I
FUJ00001673
FUJ00001673
Fujitsu Services Testing and Integration Strategy Ref: VUSTR/001
Version: 3.0
Commercial-in-Confidence Date: 16/07/2002
13.7
13.8.1
13.8.2
13.8.3
previously stored. Based on the originals having been verified as being correct then any
change in the image is highlighted. The image can either be the screen captured at any
point during a transaction flow or a print preview of a receipt or a cash account. In this
way any change, either very significant or just a change in sequence on a report, is
automatically highlighted as a discrepancy.
The capture and compare scripts are powerful when used as a regression tool against
stable elements of the system. Significant quantities of scripts have been produced to
date. The automated regression pack currently consists of in excess of 3000 checks
with each check representing one point in the system where the screen content is
compared back to a previously baselined screen shot. These scripts have been
successfully used during BI2 counter regression activities. It is envisaged that the pack
would need to contain between 25 to 30,000 checks to represent a counter regression
pack which was comprehensive enough to replace a significant element of the currently
executed manual counter regression pack. Development of this pack is to continue
with the long term aim being the reduction of the project’s cost of performing a full
counter regression test.
A Pathway forum has sought to identify additional automation initiatives. Automated
performance benchmarking scripts have been developed which compare timings for
previous counter actions against the same actions at a new release. These scripts are
proving a powerful tool in the early identification of potential performance issues. The
level of accuracy is not mature enough to replace the video benchmarking exercise but
it is enough to identify potential performance regression within a release. In addition
automated Application Performance Matrix scripts have been developed. These scripts
allow a system function to become a trigger which is then executed in turn before a
series of effects, which are defined range of system functions. The trigger is then
automatically varied and the cycle repeated. In this way if any effect is recorded as
running slower, after being trigged by a particular function, then the trigger can be
identified as containing a potential performance issue.
The further development of the automated tooling already in place and the continuing
commissioning of new automated initiatives is a prime driver to developing a reliable
and repeatable test pack. Continued increased use of automation is a key element in the
maturity of the overall Pathway testing strategy.
On occasion, bespoke (in-house developed) testing tools will be necessary/desirable.
The decision whether or not to commission the development of a bespoke testing tool
must be taken on a cost benefit basis, and must also take into account issues like
development lead-times, ongoing maintenance of the tool, potential problems with
reliability, and what testing will be required to validate the correct working of the tool.
One good example of where a bespoke testing tool is all but essential is in the Network
Banking Service. Here the Pathway system has to interface with a 3" party system —
the Network Banking Engine (NBE). As this 3" party system will not be generally
available for test use until near the end of the programme timescales, then it was
agreed that a simulator for the NBE should be developed. This allows Pathway testing
to progress independently from that of the NBE.
© 2002 Fujitsu Services Commercial-in-Confidence Page: 51 of I
Fujitsu Services
Testing and Integration Strategy
Commercial-in-Confidence
Ref:
Version:
Date:
FUJ00001673
FUJ00001673
VIU/STR/001
3.0
16/07/2002
APPENDICES
Al. GLOSSARY OF TERMS
© 2002 Fujitsu Services
Commercial-in-Confidence
Page: 52 of 1
Fujitsu Services
Testing and Integration Strategy
Commercial-in-Confidence
Ref:
Version:
Date:
VIU/STR/001
3.0
16/07/2002
FUJ00001673
FUJ00001673
APPENDIX 1
GLOSSARY OF TERMS
© 2002 Fujitsu Services
Commercial-in-Confidence
Page: 53 of 1
Fujitsu Services
FUJ00001673
FUJ00001673
Testing and Integration Strategy Ref: VI/STR/O01
Version: 3.0
Commercial-in-Confidence Date: 16/07/2002
Term
Meaning
Acceptance Criteria
The document(s) agreed between the Customer and the
Service Provider which lay down the criteria against which the
system will be measured at agreed points during the course of
testing in judging whether the system is to be formally
accepted by the Customer.
Architectural Conformance
The testing lifecycle wherein the test objectives are oriented
toward demonstrating that the system and its components
conform to their architectural specification, including aspects
such as structure, performance, security, operability,
resilience, recovery, and other NFRs.
Assured Status
The state a system or its components have reached on
successful completion of a particular stage of testing - e.g.
“System Test Assured Status’ following successful completion
of System Test.
POL
The Customer
Black Box Testing
Testing of a system or its components which is planned and
conducted without knowledge of the inner workings, looking
only at the input, the specification of the system, and the
output.
Blitz Testing
A process of crashing a series of representative tests through,
in an informal fashion, without much attention for accuracy, or
data integrity. The objective being to quickly and cheaply
identify the majority of the ‘stoppers’ so that they can be
corrected, so stabilising the runtime environment sufficient to
allow more formal tests to be run without so high a level of
disruption. This technique is particularly useful in stabilising
new test environments and getting new software components
to bed down together.
Business Integrity
The testing lifecycle wherein the test objectives are oriented
toward demonstrating that the overall system satisfies the
business need and when used in context and together with the
associated business products, retains overall business integrity.
Business Procedures
The document(s) defining the business activity surrounding
and controlling the use of the computer system by the
Customer and their staff.
Customer
POL
EPIDs
Unambiguous definitions of the physical nature of an interface
between the system under test and an external system.
© 2002 Fujitsu Services
Commercial-in-Confidence Page: 54 of 1
Fujitsu Services
FUJ00001673
FUJ00001673
Testing and Integration Strategy Ref: VI/STR/O01
Version: 3.0
Commercial-in-Confidence Date: 16/07/2002
Functional Conformance
The testing lifecycle wherein the test objectives are oriented
toward demonstrating that the system and its components
conform to their functional specification.
Fujitsu Services
The Service Provider.
National Rollout
(see also Rollout) The phase of the programme where,
following Live Trial, the system is subject to rollout on a
nation-wide basis according to an agreed timetable.
Operating Instructions
The document(s) defining the operational control of the
system, such as batch schedule control, recovery control, etc.
Output Handling Equipment
Ancillary equipment associated with the computer system or
its output, which extends the service offered. For example,
equipment used in conjunction with computer printers, to say
collate and envelope and frank output targeted for posting.
Real Time
Here used to differentiate between modes of test execution,
not to be confused with Real-time systems. It means where
test teams do not manipulate the time during the running of a
test, but rather operate the test in ‘real’ time.
Regression Testing
The process of demonstrating that a system and _ its
components have not regressed to a ‘worse’ state following a
change (usually to the software).
Rollout
The process of packaging and distributing the system out to
the various locations at which it will operate and be used (see
also National Rollout).
Service Level Agreement
The document(s) agreed between the Customer and the
Service Provider which lay down the minimum operational
criteria to which the service must be run and maintained.
Service Provider
Pathway - providing the service of development, maintenance
and operational running of the system, according to agreed
requirements and Service Level Agreements.
Stoppers
Defects which effectively prohibit a particular line(s) of testing
continuing as planned. These defects are said to ‘Stop’ the
test(s). Hence ‘Stoppers’.
Supplier
one of the suppliers of Pathway - a 3rd party providing goods
or services to Pathway, or one its direct sub-contractors
working to specification by and on behalf of Pathway.
Validation
The process of evaluating products (at the end of a given
phase) to demonstrate compliance with their specified
requirements.
Verification
The process of evaluating the products of a given phase to
ensure correctness and consistency with respect to the
products and standards provided as input to that phase.
© 2002 Fujitsu Services
Commercial-in-Confidence Page: 55 of 1
FUJ00001673
FUJ00001673
Fujitsu Services Testing and Integration Strategy Ref: VIU/STR/001
Version: 3.0
Commercial-in-Confidence Date: 16/07/2002
White Box Testing Testing of a system or its components which exploits
knowledge of or interrogates the inner workings.
© 2002 Fujitsu Services Commercial-in-Confidence Page: 56 of I
Fujitsu Services
Testing and Integration Strategy
Commercial-in-Confidence
Ref:
Version:
Date:
FUJ00001673
FUJ00001673
VIU/STR/001
3.0
16/07/2002
END OF DOCUMENT
© 2002 Fujitsu Services
Commercial-in-Confidence
Page: 57 of 1