ICL Pathway
2.0
30/09/96
Document Title:
Document Type:
Abstract:
FUJ00001273
FUJ00001273
Ref: — VI/STROOO1
TESTING & INTEGRATION STRATEGY Version:
Date:
TESTING & INTEGRATION STRATEGY
Strategy
This document defines the strategic approach to be adopted
for the testing and integration of Pathway products for
BA/POCL. Its scope extends well beyond the application
software to cover the entire system, from unit testing of
individual software modules, through integration of
software and hardware, to trials of the fully configured
operational system. It should be read in conjunction with
the General Testing Policy document. It covers the areas of
functional conformance, system performance and resilience,
end to end architectural integration, system operability, and
business integrity. It also provides a framework for the
allocation of specific areas of responsibility. It excludes the
detailed verification of the many bulk standard proprictary
software packages and hardware items (shrink wrapped and
off the shelf goods) other than in the context of its specified
usage by this system. An overview of the high level test
processes to be employed is included as an appendix. For
further detail see the separate Testing Processes document.
Status: Approved
Distribution: Pathway Management Team
(and further at their discretion)
Author: S Berry & J Hunt
Approval Authority: Tan Honnor
Signature/Date:
Quality Authority: David Groom
Signature/Date:
Customer Authority:
Signature/Date:
Bob King Gary Palmer
COMMERCIAL IN CONFIDENCE
© 1996 ICL Pathway
Page 0 of 72
FUJ00001273
FUJ00001273
ICL Pathway Ref: VI/STROOO1
TESTING & INTEGRATION STRATEGY Version:
2.0
Date:
30/09/96
DOCUMENT HISTORY
Version Date Reason
0.1 05/01/96 I Initial draft - general framework - issued for review, key Pathway
personnel only
0.2 19/01/96 _I Internal version only - not issued
0.3 02/02/96 _I Internal version only - not issued
04 16/02/96 I Final draft - full document - issued for wider review, Pathway
personnel across programme
1.0 01/05/96 _I 1st approved issue, following final review
15 07/09/96 _I Revised and presented for PDA Acceptance Review
2.0 30/09/96 _I 2nd issue, following acceptance by the PDA
CHANGES FROM LAST ISSUE
Ref. Change
n/a Cosmetic changes to reflect revised document library standards
n/a Introduction of Joint (PDA/Pathway) Testing Approach
12 Verification Centre changes applied (sequence in lifecycle)
9 Introduction of RAD complementary testing
n/a Comments agreed at PDA Acceptance Review applied, including extension of
section 17 to incorporate description of general approach to Regression Testing
CHANGES FORECAST
Change Target Issue
Changes relating to issue of SLAs 3.0
Changes relating to issue of Acceptance Criteria 3.0
Changes relating to issue of TED 3.0
Changes relating to agreement of final PBFS (or SADD) 3.0
Changes relating to process revisions 3.0
Changes relating to issue of Release Strategy 3.0
ASSOCIATED DOCUMENTS
Ref. I Library Ref. I Title Source
(1) VI/POL0001_I General Testing Policy Pathway
[2] tbs Acceptance Criteria (Acceptance Specifications) Pathway
[4] IM/STRO001 I Migration Strateg Pathway
[5] IM/PLA0002 _I Implementation Strateg: Pathway
[6] I TD/ARCO001 I Technical Environment Description Pathway
[8] CR/FSP0003_I Pathway Baseline Functional Specification Pathway
[9] RK/0002 Risk Response Catalogue Pathway
[10]_I CM/STRO002 I Configuration Management Strategy Pathway
[11] I VI/PRDO001 I Testing Processes Pathway
© 1996 ICL Pathway COMMERCIAL IN CONFIDENCE
Page 1 of 72
FUJ00001273
FUJ00001273
ICL Pathway Ref: VI/STROOO1
TESTING & INTEGRATION STRATEGY Version:
2.0
Date:
30/09/96
[12] _I BP/CON0004 I (SLAs) Contract Schedule B03 Pathway/PDA
[13] I CS/PER 0001-] (SLAs) Business Performance Portfolio - series of I Pathway
0010 10 documents - SLAs with Pathway’s suppliers
[14] I RS/FSP0001 I Security Functional Specification Pathway
[15] _I BP/CON0007 I Requirements Catalogue - Contract Schedule BO] I PDA
© 1996 ICL Pathway COMMERCIAL IN CONFIDENCE
Page 2 of 72
FUJ00001273
FUJ00001273
ICL Pathway Ref: — VI/STROOO1
TESTING & INTEGRATION STRATEGY Version:
2.0
Date:
30/09/96
ABBREVIATIONS
Used
Meaning
API Application Program Interface
BA Benefits Agency
BOM Bill of Materials
BRTS Business Release Test Strategy
cf. see, reference, compare (confer)
CM Configuration Management
EPID External Physical Interface Definition
HCL Human Computer Interface
HLTP High Level Test Plan
UF Interface
IT Integration Test, or sometimes Information Technolog)
ITs Integration Test Strategy
JBT Joint Business Test
JBTS Joint Business Test Strateg
LLTS Low Level Test Script
LT Live Trial
LTS Live Trial Strategy
MOT Model Office Test
MOTS Model Office Test
NFR Non Functional Requirement
OHE Output Handling Equipment
PAT Product Acceptance Test
PATS Product Acceptance Test Strategy
PBFS Pathway Baseline Functional Specification
PO Post Office
POCL Post Office Counters Ltd.
RAD Rapid Application Development
SLA Service Level Agreement
ST System Test
STS System Test Strateg:
TED Technical Environment Description
TEP Test Execution Plan
TIS Testing and Integration Strategy (this document)
UT Unit Test
UTS Unit Test Strateg:
VC Verification Centre
© 1996 ICL Pathway COMMERCIAL IN CONFIDENCE
Page 3 of 72
FUJ00001273
FUJ00001273
ICL Pathway Ref: — VI/STROOO1
TESTING & INTEGRATION STRATEGY Version:
2.0
Date:
30/09/96
CONTENTS
1. INTRODUCTION
2. MANAGEMENT SUMMARY
3. SCOPE OF TESTING
4. HIGH LEVEL TEST OBJECTIVES
5. APPROACH
6. RELEASE STRATEGY
7. DEPENDENCIES
8. RESPONSIBILITIES
9. UNIT TEST
10. PRODUCT ACCEPTANCE TEST
11. SYSTEM TEST
12. VERIFICATION CENTRE
13. INTEGRATION TEST
14. JOINT BUSINESS TEST
15. MODEL OFFICE TEST
16. LIVE TRIAL
17. MAINTENANCE & REGRESSION TESTING
18. EXTERNAL CERTIFICATION
19. TEST ENVIRONMENTS
APPENDICES
A1.TEST PROCESSES - OVERVIEW
A2.GLOSSARY OF TERMS
© 1996 ICL Pathway COMMERCIAL IN CONFIDENCE
Page 4 of 72
FUJ00001273
FUJ00001273
ICL Pathway Ref: VI/STROOO1
TESTING & INTEGRATION STRATEGY Version:
2.0
Date:
30/09/96
1, INTRODUCTION
1.1 This document defines the overall strategic approach to be adopted for the testing and
integration of products by Pathway for BA/POCL, as required in section 3.3 of the
General Testing Policy [1]. This document is subordinate to the General Testing Policy
[1] and describes how the testing and integration activities will apply the policy.
1.2 At a high level it describes the stages of testing and integration to be carried out, and
includes high level processes for each of these stages in Appendix A. More detailed
descriptions of these procedures can be found in the Testing Processes document [11].
1.3 This document is at the head of a family of documents. It is intended to be very much a
working document, and not shelfware. Implementation is via the production of these
detailed subordinate strategy documents which serve as the vehicles to enhance the
approach described here and to add further appropriate detail as it becomes available
during the development lifecycle. Together they unambiguously define the objectives,
scope, coverage, and success criteria for all the testing necessary to meet the Customer
agreed Acceptance Criteria [2] for each major release. The following figure maps out the
likely documents necessary to perform this, and their inter-relationships. Further more
detailed documents may be produced below these, as the situation dictates (e.g. for
performance testing, and for security testing if these areas were of particular importance
or required more detailed explanation).
Acceptance Pathway Baseline Technical Environment
Service Level Criteria Functional Specification Description
Agreements
ee
Testing & Integration Strategy
Y
Release
Testing
Policy
Testing
Processes
Secutity Policy
Migration
Strategy
Performance "—™— Bi Release Test Si a
Budgets. § uusiness Release Test Strategy
wee 8 ———— Impiementation
0 0 0 0 Strategy
7 Product System Tategration Toint Model iwe
Sa rest Acep. Test Test Test Business Test I I Office Test Trial
Strategy Strategy Strategy Strategy Strategy Strategy
7 ——— High Level Test Plans ———— Live
“nos GOGO GR
Low Level Test Scripts / Business Test Scripts
Figure 1 - Family of Documents defining Test Coverage
© 1996 ICL Pathway COMMERCIAL IN CONFIDENCE
Page 5 of 72
FUJ00001273
FUJ00001273
ICL Pathway Ref: VI/STROOO1
TESTING & INTEGRATION STRATEGY Version:
2.0
Date:
30/09/96
1.4 The Pathway Baseline Functional Specification [8] is agreed with the Customer within 30
days of the award of contract, and forms the fundamental document driving out the
testing that must be considered under this approach. Together with the Requirements
Catalogue [15], the Technical Environment Description [6], the Security Functional
Specification [14], the Customer agreed Acceptance Criteria [2], and the Service Level
Agreements (SLAs) [12 & 13] these serve as the principal inputs to high level test
planning. Taken in conjunction with the General Testing Policy [1] and the testing
processes (c.f. Appendix A and the Testing Processes document [11]) they forge the high
level objectives for all testing and integration activities required on this programme.
1.5 The Release Strategy defines the phased release of software into the Live arena (c.f.
section 6 - Release Strategy). Each release may have specific attributes peculiar to that
release, corresponding with the particular products being released at that time, and related
to the Migration Strategy [4] and Implementation Strategy [5] for that release. There are
also likely to be specific security attributes and specific performance issues for each
release, the latter described by the Performance Budgets derived from the TED [6]. These
peculiarities will be detailed in the Business Release Test Strategy for that release, as
regards any special testing and integration features that may pertain.
1.6 The BRTS will be expanded down into specific testing strategies and High Level Test
Plans (HLTPs) to cover each of the stages of testing. These will detail the specific scope,
coverage, objectives and success criteria that apply. In this high level test planning stage
the test objectives are formally documented and agreed by both the Customer and by
Pathway, and combined to form a joint set of test objectives which serve to drive all
subsequent testing activity. With Pathway and Customer working together specific test
plans are formulated to satisfy the combined objectives. These HLTPs will thus
encompass all Pathway test objectives, all Customer test objectives and all contractual
Acceptance Test Specifications. In addition for each group of in-house products a Unit
Test Strategy will be produced, each in turn giving rise to Unit Test Plans covering both
module testing and link testing.
1.7 Finally, when sufficient detail becomes available, the ‘logical’ HLTPs can be translated
into ‘physical’ Low Level Test Scripts (LLTSs) and their supporting test data, ready for
test execution when the software products become available to run. Where an HLTP has
addressed Customer test objectives, it may also be necessary to generate separate business
oriented physical test scripts with their own supporting business material. These will
enable such tests to be conducted in an alternative mode, by personnel from the Customer
testing area, and using selected representatives from the end-user community.
1.8 Throughout the test execution period, regular checkpoints will be taken (typically by the
taking of physical database dumps and the preserving of associated flatfile and
configuration data) at suitable quiescent points during the running of lengthy tests suites.
This is necessary to allow tests to be restarted just a little prior to the point of failure on
receipt of a fix, or to re-run appropriate segments of a test when regression testing
becomes necessary because of changes to the system. This is particularly important during
the System Test and Integration Test stages where the cost of such re-testing would
otherwise quickly become prohibitive.
© 1996 ICL Pathway COMMERCIAL IN CONFIDENCE
Page 6 of 72
FUJ00001273
FUJ00001273
ICL Pathway Ref: VI/STROOO1
TESTING & INTEGRATION STRATEGY Version:
2.0
Date:
30/09/96
In addition during the test execution period, the test results of formal test runs retained for
examination by the test manager will for each ‘final’ run be kept as an audit trail and made
available the Customer’s auditors by arrangement.
1.9 Testing will be managed, through both planning and execution phases, on a strict cost
benefit basis. How much testing can be afforded? How little testing can be afforded? The
cost of performing tests versus the risk of not performing them. A pragmatic approach to
test planning and execution is essential in maintaining the correct balance, cost versus risk.
As test planning progresses there will be areas of the system deserving of more scrutiny
than others, because of the inherent risks they entail. More time and effort will be allowed
here accordingly. Similarly certain areas will be deserving of little or no scrutiny, being of
very low risk impact, and so tests will be reduced or discarded here. (Please note that
where ‘cost’ , ‘benefit’ and ‘risk’ are concerned above, then ‘time’ and ‘schedule’ are
contributory elements. Lost time costs, saved time can benefit, and schedule dependencies
and late delivery constitute project risks.)
© 1996 ICL Pathway COMMERCIAL IN CONFIDENCE
Page 7 of 72
FUJ00001273
FUJ00001273
ICL Pathway Ref: VI/STROOO1
TESTING & INTEGRATION STRATEGY Version:
2.0
Date:
30/09/96
2. MANAGEMENT SUMMARY
This document sits at the head of a family of strategy documents which together serve to
unambiguously describe the scope and coverage of testing required for the programme,
and in this way they also serve as the means of agreeing this test coverage with all
interested parties.
The approach to testing is one of staged, systematic verification, with progressive
integration of software and hardware components, first stabilising the environment and
business functionality, then system, performance, operability, security, etc., and
culminating in overall service validation of the fully configured system in the Live
environment
The system is subjected to testing against three principal test life-cycles, for Functional
Conformance, Architectural Conformance and Business Integrity. These give rise to six
stages of testing - Unit Test, Product Acceptance Test, System Test, Integration Test,
Model Office Test and Live Trial. It is important to note that the traditional test activities
such as ‘performance testing’ and ‘security testing’ do not exist as discrete activities
under this approach, but rather are integrated into cach progressive stage of testing.
Unit Test (UT) deals with the detailed verification of individual modules and their low
level linking to form products. It is performed explicitly for all products developed in-
house. An equivalent level of testing is assumed to be performed by the other suppliers. In
addition, the course of Unit Test, and particularly where a RAD approach to development
has been adopted, the Independent Test area will co-operate with the Development area
in taking early versions of products and exercising them in an informal fashion to help
flush out problems and stabilise the environments.
Product Acceptance Testing (PAT) is performed for each bespoke software product, and
is as its name suggests the means of formally accepting a product into the programme,
under Configuration Management (CM) control.
System Test (ST) is performed against a full software set and serves to validate the
software against the requirement, concentrating on functional conformance. It operates
the products in conjunction with each other and follows business threads.
Working from the expanded CM Bill of Materials (CM-BOM), now validated by ST, the
Verification Centre (VC) builds the planned system configuration (for the ‘counters’ sub-
systems) ready for rollout.
Rollout is first to Integration Test (IT) which takes the system tested software set, now
properly configured and installed on the full live target hardware set, according to the CM-
BOM, and including links with external systems, and validates the inter-working of the
full system, putting it under load and stress to verify the performance characteristics, and
other areas of the Non-Functional Requirements (NFRs), including security and
operability attributes, that may remain outstanding. The system configuration is thus
validated and the VC ‘build’ refined accordingly.
© 1996 ICL Pathway COMMERCIAL IN CONFIDENCE
Page 8 of 72
FUJ00001273
FUJ00001273
ICL Pathway Ref: VI/STROOO1
TESTING & INTEGRATION STRATEGY Version:
2.0
Date:
30/09/96
ST and IT stages both follow High Level Test Plans (HLTPs) formulated jointly by
Pathway and the Customer. Increasingly as test execution progresses through these
stages, and particularly when the Model Office environment is employed in IT, the
Customer will become actively involved. Various business support products are brought
into play here also, including the Help Desk. This period of joint working is termed Joint
Business Test (JBT).
Having established an agreed baseline configuration at the VC, rollout is next to the
Model Office Test (MOT) which is effectively a continuation of Integration Testing
performed in the Model Office environment, but now controlled and conducted by the
Customer to serve as the vehicle for formal and independent testing of the system by the
Customer.
Rollout is then to the Live Trial (LT) which is a limited but actual live operation, with live
staff, live data, and the general public involved in real transactions at a number of actual
Post Offices - a Full Pilot.
These stages apply to each separate major release of the system, as defined in the Release
Strategy. They also apply, where appropriate, in the maintenance of cach release. In
addition certain products may require a level of external certification for legal reasons.
For example the electronic weigh scales used in conjunction with the PCs on the counter
top will require certification by the Weights and Measures authority.
Throughout test planning and test execution, testing is conducted by agreement with the
programme and the customer, on a cost versus risk basis. No system can ever be error-
free. It is not possible to prove that a system works, only that it does not. It is therefore
only possible through testing to reduce the risk of error remaining. The cost of this error
finding and removal follows a course of increasing cost and diminishing return. The
equation is one of cost versus risk. Different systems can abide varying levels of risk.
Each successive stage of testing is conducted in appropriate types of test environment,
which progressively get larger, more shareable, less simulated, more life-like and under
stricter CM control from stage to stage. This helps to concentrate test activity in more
affordable and more appropriate environments and so avoids unnecessary escalation of
machine resource costs.
© 1996 ICL Pathway COMMERCIAL IN CONFIDENCE
Page 9 of 72
FUJ00001273
FUJ00001273
ICL Pathway Ref: VI/STROOO1
TESTING & INTEGRATION STRATEGY Version:
2.0
Date:
30/09/96
3. SCOPE OF TESTING
3.1 The scope of testing and integration activities encompassed by this document and its
subordinate test strategies includes the entire application software system (both 3rd party
supplied and in-house developed products) and its integration with the supporting
hardware, support products and services, and infrastructure software platform.
3.2 It does not cover specific detailed testing of that platform (which is expected and assumed
to be of Assured Status) other than in respect of its support of the software system and in
satisfying the SLAs and Non Functional Requirements (NFRs).
3.3 It starts with the testing of Release 1, from Unit Test (c.f. section 9), and onward up to
Live running in the ‘Live Trial’ and future maintenance, and applies similarly to all
subsequent releases.
3.4 It does not cover the testing activities of the 3rd party suppliers. It is expected and
assumed that all products supplied will be appropriately tested prior to delivery to ensure
that they meet the supplied specifications. However, it is not assumed that these suppliers
will have integrated and proven their various products together, excepting where a
supplier is charged with producing inter-linking products, where limited link testing is
expected prior to delivery.
© 1996 ICL Pathway COMMERCIAL IN CONFIDENCE
Page 10 of 72
FUJ00001273
FUJ00001273
ICL Pathway Ref: VI/STROOO1
TESTING & INTEGRATION STRATEGY Version:
2.0
Date:
30/09/96
4, HIGH LEVEL TEST OBJECTIVES
4.1 Specific
For each release, tests are to be engineered to demonstrate the following:
a) each module developed in-house to be compliant with the corresponding module
specification and to link correctly with co-operating modules in that product. (Unit
Test).
a) cach bespoke software product, cither developed in-house or provided by a 3rd
party supplier, to be compliant with its Product Description/Detailed System
Specification. (Product Acceptance Test).
a) software products to operate successfully in conjunction with one another to
satisfy the Pathway Baseline Functional Specification [8] and to remain consistent
with their Product Descriptions. (System Test, including Joint Business Test).
a) software systems to operate and co-operate successfully together on the target
hardware and infrastructure software platform and to interface correctly together
and with the BA/POCL systems and other external systems, to satisfy the SLAs
and NFRs that apply. This to include verification of Output Handling Equipment
(OHE) operation as required by the system, such as card production, use of ‘slip
printers’ at the counters, use of A4 cut sheet printers in the back office, and all
associated stationery and materials. (Integration Test, including Joint Business
Test).
a) the full operational configuration to operate successfully with user procedures and
within a ‘laboratory’ office environment (the Model Office), and to be both jointly
tested (Pathway and Customer) and independently tested (Customer) against the
agreed test objectives established in the high level planning stage. This to include
validation of OHE operation. (Integration Test, including Joint Business Test, and
then Model Office Test).
a) system to successfully support the ‘Live Trial’ in verifying user procedures,
training material, support mechanisms, and help desk procedures, and to confirm
the migration and implementation prior to wholesale rollout and usage. (Live
Trial).
4.2 General
For each release tests are to be engineered with particular regard to the following areas of
good general practice:
a) to ensure that as much testing as possible is performed as early as possible in the
lifecycle, to reduce defect correction costs and avoid unwelcome schedule
disruption late in the lifecycle.
a) to arrange for as much testing as possible to be of an automated and re-runable
nature, to reduce regression test costs, to speed test execution times, and to avoid
unnecessary levels of human error.
a) to validate the end-to-end technical architecture employed.
a) to demonstrate that the system meets the specified and agreed levels of
functionality and performance, and so is fit for service.
a) to comply with the General Testing Policy [1].
© 1996 ICL Pathway COMMERCIAL IN CONFIDENCE
Page 11 of 72
FUJ00001273
FUJ00001273
ICL Pathway Ref: VI/STROOO1
TESTING & INTEGRATION STRATEGY Version:
2.0
Date:
30/09/96
a) to adopt a joint (Pathway and Customer) approach to testing, combining test
objectives, high level test planning, and test execution activities wherever
practicable, attacking multiple objectives in combined tests and so reducing
duplication of effort and minimising the overall elapsed time required for effective
testing.
a) to stabilise the Model Office environment at the earliest point and to pre-prove the
operation of the Model Office Test by prior joint working in order that the system
configuration can be ‘frozen’ at the earliest point to facilitate preparation for
rollout to the Live Trial, and that the period required for this independent
Customer testing can be minimised.
© 1996 ICL Pathway COMMERCIAL IN CONFIDENCE
Page 12 of 72
FUJ00001273
FUJ00001273
ICL Pathway Ref: VI/STROOO1
TESTING & INTEGRATION STRATEGY Version:
2.0
Date:
30/09/96
5. APPROACH
5.1 In essence the approach here is one of staged, systematic verification, with progressive
integration of the wide ranging system components, culminating in overall validation of
the fully configured system within the Live environment. It is a continuous process,
spanning the whole development lifecycle, commencing with test analysis and the
production of formal test material, and progressing through to test execution, result
checking, and defect removal.
A partnership approach between Pathway and Customer is key here in securing the most
efficient and effective means of testing and integrating the system. Pooling skills and
resources will enable clearer focus on test objectives at the outset, will promote higher
quality test planning and scripting, and will help reduce the elapsed time necessary for test
execution through co-operative effort.
The component products are moved through distinct, separately planned and executed
stages of testing, each designed to progress the products to a higher level of assurance.
Once a stage of testing has been completed for a particular product set, and the test
activity reviewed and signed off by the test manager concerned, then that product set
moves to ‘Assured Status’ and is deemed to be ready for use in the next stage and for
progressively wider integration with their co-operating product sets. As the majority of
the products required to make up the system are to be provided by 3rd party suppliers,
then the emphasis in testing is skewed heavily toward ‘black box’ techniques. That is, the
detailed inner workings of supplied products are not examined and put under test, but
rather their gross behaviour is verified in the context of the services they are required to
perform.
5.2 It is therefore particularly important that the Business Requirement is firmly understood at
the outset. Here, the Business Requirement is taken as being encompassed by the Pathway
Baseline Functional Specification [8], the Customer agreed Acceptance Criteria [2] and
the SLAs used to formalise the various non-functional requirements that may exist. These
must be maintained under strict Change Control, with the testers being included in the
impact assessment process for all such changes.
Similarly it is important that an ‘Independent Testing Group’ approach be adopted to
maintain objectivity in developing the ‘black box’ tests necessary to successfully verify
and integrate the mixed product set. That is, that products be treated alike, irrespective of
whether they be supplied by a third party or developed in-house, with formal product
handover into testing being established via formal product acceptance mechanisms. This
Independent Testing Group should be separately managed from Development, and
comprise of both Pathway and Customer testing personnel, supported as appropriate by
representatives drawn from the end-user community.
© 1996 ICL Pathway COMMERCIAL IN CONFIDENCE
Page 13 of 72
FUJ00001273
FUJ00001273
ICL Pathway Ref: VI/STROOO1
TESTING & INTEGRATION STRATEGY Version:
2.0
Date:
30/09/96
5.3 Given the extremely condensed timescales available, it is also important that test planning
commences at the earliest opportunity, and despite the separation of development and
testing streams introduced by the independent testing approach outlined above, testers
must nonetheless be involved throughout the development lifecycle, starting with high
level ‘logical’ test planning, based on the business requirements, and progressively
developing these into ‘physical’ test scripts, so that the test material is ready and approved
in good time for use in test execution when the products are handed over into test.
This is consistent with the general recognition that testing follows a lifecycle closely
interwoven with that of development. This is best embodied in the well established
lifecycle “V’ diagrams, where development progresses down the left leg of the ‘V’ with
analysis, design and construction, and then up the right leg of the ‘V’ with unit test,
function test and system test. The horizontal relationships between left and right legs
indicate the test analysis, planning and preparation required against each phase.
5.4There are three principal categories of verification and validation required on this
programme. The ‘Functional Conformance’ of the system must be evaluated. That is,
on a purely functional level, confirming that the services required of the system by the
Customers, as defined in the functional requirements, are being met. The ‘Architectural
Conformance’ of the system, both software and hardware must be evaluated. That is, the
innovative underlying system architecture employed, with extensive infrastructure
software and mixed hardware platforms, must be trialed under stress to confirm service
attributes such as performance, operability and security. The ‘Business Integrity’ must
be evaluated. That is, with the close relationship between Customer and Service Provider
here, and the requirement for User Confidence and Live Trial components within the
Operational Trial phase of the Programme, the relatively late development of user
processes and procedures, and the complex migration and implementation necessary, it is
necessary to demonstrate integrity across the breadth of the whole business system.
These three categories - Functional Conformance, Architectural Conformance, and
Business Integrity - each have their own test lifecycle associated. That is not to say that
they are conducted as separate activities, but rather that the test processes employed at
each stage must take account of each aspect and their particular objectives and
dependencies. In fact there is a clear overlap between the three when the lifecycles are
mapped against the stages of testing, with Product Acceptance Test and System Test
serving the objectives of both Functional Conformance and Architectural Conformance,
and with Integration Test similarly serving both Architectural Conformance and Business
Integrity.
© 1996 ICL Pathway COMMERCIAL IN CONFIDENCE
Page 14 of 72
FUJ00001273
FUJ00001273
ICL Pathway Ref: VI/STROOO1
TESTING & INTEGRATION STRATEGY Version:
2.0
Date:
30/09/96
Functional
Coe Conformance
Product Architectural
AcceptanceTest Conformance
System Test Business
Integrity
Integration Test
Model Office Test
Live Trial
Figure 2 - Mapping of Testing Lifecycles onto the Stages of Testing
5.5 Functional Conformance.
In parallel with formulating the Pathway Baseline Functional Specification [8], and the
Requirement Catalogue [15], test analysis should be conducted and initial Business
Threads produced describing very high level business driven test scenarios which will
form the backbone of the System Test stage. This activity should be carried out in close
co-operation with the Requirements Analysis activity, and with strong involvement from
the Customer. Once the Pathway Baseline Functional Specification [8] is agreed and in
place, more detailed test analysis can be afforded, and the System Test Strategy (STS)
and High Level Test Plans (HLTPs) for System Testing can be drawn up by expanding the
Business Threads. Again this should be done with strong involvement from the Customer.
Once the Product descriptions and Detailed System Specifications, consistent with the
Pathway Baseline Functional Specification [8], are available, then further detailed test
analysis of a much more specific nature can be conducted, engineering tests for use in
Product Acceptance Test. These are intended to be used in verifying that each Product
delivered into test can be shown to be compliant with their detailed specifications. The
Product Acceptance Test Strategy is produced and agreed.
For those products that are developed in-house, the detailed design documentation should
include detailed test plans for Unit Test, covering both individual module testing and the
link testing of modules to form products. These should be consistent with the Unit Test
Strategy produced for that product. Following construction the module and link test plans
are carried out in Unit Test. For sub-contracted products, it is expected and assumed that
the sub-contractors bring their products up to an equivalent state. From here on all
products are regarded the same whether in-house or sub-contracted.
© 1996 ICL Pathway COMMERCIAL IN CONFIDENCE
Page 15 of 72
FUJ00001273
FUJ00001273
ICL Pathway Ref: VI/STROOO1
TESTING & INTEGRATION STRATEGY Version:
2.0
Date:
30/09/96
As each product is completed it is placed under Configuration Management control and
delivered for independent testing where it is subjected to Product Acceptance Test
according to the Strategy and Test Plans drawn up against the Product Descriptions and
Detailed System Specifications. Any products not making the grade are rejected for
correction and redelivery by the party concerned.
Requirements System
Analysis Test
_ Product Product
Specification Acceptance Test
Design
Sub-contract
Construction Module Test
Figure 3 - The ‘Functional Conformance’ Testing Lifecycle
Once the requisite products have passed Product Acceptance Test they enter System Test,
where, following the System Test Strategy and High Level Test Plans produced earlier,
they are subjected to the Business Thread based tests to ensure conformance with the
Requirements. Again any defects found are formally recorded, returned for correction as
appropriate, and re-tested. System Test takes place on a simple hardware platform, with
Interfaces out to external systems being simulated or truncated as appropriate. As much
of the Infrastructure software will be included at this time as can be accommodated given
the constraints of the platform. Some of the Infrastructure/Hardware interfaces will have
to be simulated or truncated also.
The primary concern here is to achieve near end to end verification of the system and
business flows, but early exposure of the infrastructure is desirable. Up to this point it is
not envisaged that much if any Customer involvement would be required in the test
execution.
It is with the System Test stage that the joint testing between Pathway and the Customer
really starts in earnest. Their objectives are combined, the Business Threads and High
Level Test Plans are jointly formulated, and the Customer will have some exposure to the
test execution phase.
© 1996 ICL Pathway COMMERCIAL IN CONFIDENCE
Page 16 of 72
FUJ00001273
FUJ00001273
ICL Pathway Ref: VI/STROOO1
TESTING & INTEGRATION STRATEGY Version:
2.0
Date:
30/09/96
5.6 Architectural Conformance.
From the technical platform specification, which is derived during and immediately
subsequent to establishing the overall requirement, and documented in the Pathway
Baseline Functional Specification [8] and also in the Technical Environment Description
[6] which includes coverage of the NFRs, security requirements, etc., the various
technical attributes of the system can be considered and subjected to Test Analysis. Tests
are engineered to verify these factors and HLTPs drawn up. Similarly, once the external
system interfaces are formally defined, then (as part of System Test preparation, along
with the Business Threads) the additional Inter-System Thread(s) can be derived, and
their HLTPs produced accordingly.
Infrastructure software products required to satisfy the technical architecture are treated
as described above for other products, again producing test plans from the Product
Descriptions and the technical specifications. Again if any of these components are
developed in-house, then Unit Test plans would be drawn up in the same way. The Unit
Tests and Product Acceptance Tests would be executed accordingly.
Technical
Integration
Platform Spec
= Test
Service I/F
Specification
System
Test
Product
Specification
Product
Acceptance Test
Sub-contract
Construction Module Test
Figure 4 - The ‘Architectural Conformance’ Testing Lifecycle
When the System Tests were executed, albeit on a simple hardware platform, most if not
all of the Infrastructure software would be present, and so the integration process of
combining Application software and supporting infrastructure begins. All the APIs
involved would be subject to verification, and the functional dependencies likewise.
Simple aspects of application recovery and resilience would also be exposed here. Once
the requisite System Tests are complete and the products concerned have reached
Assured Status at this level, they are made available for wider use.
© 1996 ICL Pathway COMMERCIAL IN CONFIDENCE
Page 17 of 72
FUJ00001273
FUJ00001273
ICL Pathway Ref: VI/STROOO1
TESTING & INTEGRATION STRATEGY Version:
2.0
Date:
30/09/96
The Verification Centre (VC) collects the hardware, Infrastructure software, and
Application software and subjects them to standard verification checks. The software is
loaded, configuring the overall system according to the agreed System Configuration as
documented in the CM Bill of Materials (CM-BOM) for that release. The system is thus
packaged ready for distribution. (These VC activities are limited to the ‘counters’ sub-
systems.)
The initial VC ‘build’ of system tested products is taken by Integration Test (IT) as a fully
configured system, properly installed on the full target hardware platforms in accordance
with the CM-BOM. IT validates this configuration enabling the VC ‘build’ to be refined
ready for subsequent rollout to later stages and ultimately into the National Rollout.
From this point onward, an increasing level of Customer Involvement in test execution is
required. Here, on a faithful hardware platform (full Live shape and significant size), the
Integration Tests are run. These comprise not only the HLTPs engineered specifically to
stress the architecture, but also the HLTPs corresponding to the Business Threads and the
Inter-System Threads previously run in System Test. Here though the external interfaces
are opened up to the BA/POCL systems and the Infrastructure software will now be
running in a full operational environment and so can be tested end to end in co-operation
with the supporting hardware.
The primary concern here is to confirm that moving to the full platform has not caused the
system to regress functionally, to confirm that the overall architecture hangs together end
to end, to verify the recovery and resilience aspects of the system, and to subject the
software and hardware to stress and load, measuring the performance achieved against the
NFRs and SLAs. The principal source document here is the Technical Environment
Description (TED) [6] which covers aspects such as the system and application
architecture, performance, security, and other Non-Functional Requirements (NFRs). Any
defects found will be subject to the same rigour as before.
5.7 Business Integrity.
This testing lifecycle is a little different. It is not interwoven with the development so
much as with the production of the related business products which together with the IT
system form the overall business system.
From the system requirements will come a full understanding of what work needs to be
accomplished in the live field, and how that work will be conducted and fitted into
existing work patterns. A plan can start to be drawn up as to how the first users of the
system will take it on. What work will be included, at what times over the initial period of
Live use, and with what support. (The period targeted here is known as the Live Trial,
with particular emphasis on the first day, first week and first month activities. The
intention is to build tests in that will help to specifically de-risk the early stages of Live
running.)
© 1996 ICL Pathway COMMERCIAL IN CONFIDENCE
Page 18 of 72
FUJ00001273
FUJ00001273
ICL Pathway Ref: VI/STROOO1
TESTING & INTEGRATION STRATEGY Version:
2.0
Date:
30/09/96
Once the full set of support requirements have been determined, this plan can be expanded
accordingly. This will cover such areas as the Help Desk, User Procedures and
Instructions, the Training Programme and the support material required for it, and the
User/Customer involvement in the actual migration to the new system, including manual
fallback, etc.
In line with the Requirement Catalogue [15] and consistent with the Pathway Baseline
Functional Specification [8], Customer agreed Acceptance Criteria [2] will need to be
drawn up and agreed. Likewise Service Level Agreements (SLAs) and related support
contracts [13 & 14] will be needed. From these documents test plans should be drawn up
and agreed as to exactly how the system will be independently trialed by the Customer - a
Model Office Test (MOT). These plans will be integrated with and performed initially as
part of System Test and Integration Test, to pre-prove the MOT.
The Public
POs
Live Use
Specify System
Requirements
Live
Trial
Business
Procedures
Determine Support
Requirements
Model
Office Help Desk
Develop Acceptance \ _/ Test Model
Criteria & Service Office
Level Agreements Training
Rehearse
Agree System Integration Model
Solution Test Office
Test
Figure 5 - The ‘Business Integrity’ Testing Lifecycle
When the overall system solution is agreed there are likely to be attributes of the technical
specification of keen interest to the Customer from a business operations perspective,
such as security, resilience, performance, recovery, etc. These will have been covered in
the NFRs within the TED [6], and SLAs. When the Integration Test plans are reviewed
and signed off, these attributes should be reviewed to confirm that the tests cover these
issues adequately from a business perspective. The Customer and representative end-users
should be involved in the test execution of Integration Test to be satisfied on these points.
It is important that these areas are settled here and not left to the Model Office Test and
Live Trial stages to sort out, as any problems in these areas are likely to be expensive and
time consuming to correct.
© 1996 ICL Pathway COMMERCIAL IN CONFIDENCE
Page 19 of 72
FUJ00001273
FUJ00001273
ICL Pathway Ref: VI/STROOO1
TESTING & INTEGRATION STRATEGY Version:
2.0
Date:
30/09/96
Once these Integration Tests are successfully completed, the system is ready to enter the
MOT where the Customer has complete and independent control and where end-users
should have a very strong involvement. This would be run in the Model Office
environment already utilised during Integration Test. This must be flexibly configurable so
that different tests can be run in areas which faithfully represent the various types of office
that exist in the field, large and small, and which will include one such area set up as a
Mock Post Office, with life-like counter positions, etc., where ergonomic factors can be
taken into account. This stage of testing provides the formal independent trial of the
system by the Customer prior to release for initial Live operation in the Live Trial.
The tests conducted in the MOT stage are a sub-set of the jointly planned tests already
operated in the System Test and Integration Test stages, engineered against the
Acceptance Criteria and the SLAs. They will already have been run exhaustively in this
same Model Office environment. Any known defects remaining will have been evaluated
by a Problem Review Forum or Fault Control Board (comprising both Pathway and
Customer representatives) regarding their criticality (and for how long) for Live running.
The drive here is to avoid unnecessary changes that would otherwise contribute to de-
stabilising the system at this late stage. The code is effectively ‘frozen’. Fixes should only
be contemplated here where a fresh defect is uncovered of a critical nature, as large
numbers of fixes here could not be taken in without high risk to the system as a whole and
so would jeopardise the implementation.
Finally, following successful running of the Model Office Test, the system passes into the
Live Trial - a period of genuine live operation but on a small scale and under careful
control. Here the system would be installed in a representative number of selected and
prepared Post Offices, over a period of about 3 months leading upto full National Rollout.
5.8 This approach of progressively building up product sets, with products moving through
distinct stages of testing, reaching ‘Assured Status’ and being employed in wider and
wider integration of the growing system, results in a structured sequence of testing
activities that can be planned and monitored by the Programme. Each stage centres on its
own set of test objectives: they are focused. This reduces the likelihood of duplication of
effort and allows greater efficiency in regression testing of changes, where dependent on
the nature of the change, the correct point within the test lifecycle can be targeted.
Whilst in the initial development cycle these layers or stages of testing are likely to be
carried out (it is desirable) by distinct teams of people, this is not necessarily the case later
on in the maintenance phase where it is generally more cost effective to merge them.
Similarly, the savings made in the development arena by using smaller more simulated
environments in the earlier testing stages tend not to be so easily realised during
maintenance, where it is generally simpler to do most of the testing in the larger more
complete environments. stages, and possibly to operate more in complete environments.
So typically the various tests required for a particular set of changes will be taken through
by one team, one after the other, on a single platform. This also gives additional indirect
savings, as CM overheads need only be incurred between different platforms.
© 1996 ICL Pathway COMMERCIAL IN CONFIDENCE
Page 20 of 72
FUJ00001273
FUJ00001273
ICL Pathway Ref: VI/STROOO1
TESTING & INTEGRATION STRATEGY Version:
2.0
Date:
30/09/96
The balance of cost versus risk is key to the management of the test planning and
execution activities. As already indicated, regular checkpoints should be planned into the
test suites to allow more focused re-testing of fixes and regression testing following
changes. Iterative running of tests can be made less troublesome, more cost effective and
more consistent by utilising appropriate capture/replay tools (c.f. section 19 Test
Environments). The production of expected results, and the subsequent checking of actual
results against these is a prime area for consideration. Costs can quickly escalate here and
become prohibitive. Pragmatism is the watchword. Being pragmatic about the creation of
expected results can save a fortune, and provide better quality testing in the long run. A
common pitfall is to strictly follow a process of painstakingly predicting the fine detail of
the physical data output by a planned test. Doomed to fail for anything but the trivial
process, this activity degenerates into one of testing ones ability to correctly project the
data output, rather than putting the effort into testing the product concerned. In the
planning phase, expected results should for the most part be expressed in logical and not
physical terms. Later, validated actual results can be used as physical expected results for
subsequent regression test runs. Here again, use of standard utility sets to make the
comparisons can reduce the effort involved and improve the accuracy of the checking.
5.9 There are strong schedule dependencies throughout test planning and execution on the
development methodology, the release strategy and the programme schedule. It is
important that testing and integration imperatives are taken into account in developing
these schedules.
For example, infrastructure software (including middleware) and the data maintenance
transactions for the key data elements will be required first, before any real progress can
be achieved in assembling and integrating the products into working systems.
When the high level threads for System Test have been drawn up (c.f. section 11 - System
Test) a schedule optimisation exercise will be carried out to establish these critical
dependencies and confirm that the project schedules take account of them. These
dependencies will be documented in the appropriate low level test strategies, and
monitored throughout. Depending on the nature, number and complexity of these
dependencies, it may also be decided to draw up a detailed product interception plan. The
objective is to make it possible to successfully co-ordinate product delivery and testing,
and to allow safe overlap of these lifecycle stages in accord with these critical
dependencies, and so minimise the overall elapsed time for the programme.
© 1996 ICL Pathway COMMERCIAL IN CONFIDENCE
Page 21 of 72
FUJ00001273
FUJ00001273
ICL Pathway Ref: VI/STROOO1
TESTING & INTEGRATION STRATEGY Version:
2.0
Date:
30/09/96
6. RELEASE STRATEGY
6.1 The Release Strategy describes the phased release of system functionality into the Live
arena. The Configuration Management Strategy [10] details how these releases will be
controlled and their integrity (separate identity) maintained. It is important that each
major release be carefully planned, and that versions of test material corresponding to
each release be maintained in line with the versions of software and hardware products
comprising that release, throughout the testing lifecycle.
6.2 To summarise the Release Strategy, it details three main phases:
e A Pre-release Pilot exercise, known as the Initial Go Live (IGL), to be
implemented first at a single Post Office, extending then to about 10 selected
Post Offices in the Stroud area of England, offering restricted functionality in a
carefully controlled and closely supported system regime. This is planned for
September / October 1996 and is outside the scope of this strategy.
e Release I - the main release, fully functional, robust and to be implemented
nation-wide. This is planned to commence with a representative number of
Post Offices selected for early exposure in the Live Trial, with full National
Rollout planned to commence in the summer 1997.
e Release 2 - a follow up release, content as yet unspecified, to follow some
months after Release 1.
It should be noted that the special nature of the IGL release means that acceptance of this
release by the Customer must operate against a subset of the full Acceptance Criteria.
Likewise, there is a high level of synergy between this and the full Live Trial. Finally, with
only a short period between Release I and Release 2, and given that the National Rollout
process must inevitably run in parallel throughout this period, considerable care will be
needed in scheduling the testing and integration activities for Release 2. This will be
reflected in the BRTS for Release 2, and provide details of any risk mitigation deemed to
be necessary.
© 1996 ICL Pathway COMMERCIAL IN CONFIDENCE
Page 22 of 72
FUJ00001273
FUJ00001273
ICL Pathway Ref: VI/STROOO1
TESTING & INTEGRATION STRATEGY Version:
2.0
Date:
30/09/96
7. DEPENDENCIES
This section documents the principal areas of dependency upon which this strategic
approach for testing and integration activities in Pathway relies. These will be maintained
as required, and will be monitored throughout by the Testing and Integration Manager to
ensure that they are satisfied in a timely manner and not allowed to adversely impact test
progress.
(At a more detailed level, specific dependencies have been documented as appropriate
against each stage of testing, under sections 9 through to 15.)
a) Testing will be organised as a separately managed group within Pathway for all testing
from Product Acceptance Test onward - an Independent Testing Organisation.
a) The Risk Response Catalogue [9] contains various declared risk mitigations in terms
of proving/testing activities. These will be included in the BRTS and cascaded down
into the relevant Test Strategies dealing with that activity to ensure they do get
addressed. A Risk review will be conducted to confirm that this has taken place.
a) Configuration Management practices and tooling will be implemented sufficient to
support the testing activities in a phased release regime without adding too great a
burden given the elapsed time pressures which apply.
a) The Configuration Management System will support the version control and
environment build activities of the layered and iterative testing approach described.
a) Formal change management will apply for all key documents which act as input to test
analysis. (For example, where detailed design for a sub-contracted product rightly and
necessarily diverges from that of the Product description and/or Detailed System
Specification, then that Product Description and/or Detailed System Specification will
be subject to formal change management procedures and maintained in line with the
product to be delivered - before it is delivered. Similarly, where RAD is adopted,
testing will have a dependency on delivery of the iteration updates from the RAD
process, and on the final delivery of the finished product and related system
specifications on completion of the process.)
a) All significant APIs (those between Products or between Pathway and external
systems) will be rigorously documented at the outset.
a) Performance Budgets (Targets, derived from the TED and the relevant SLAs) will be
set by the Technical Manager at the outset.
a) Sub-contractors will remain responsible for the in-programme and ongoing
maintenance of products initially sub-contracted to them, and will actively participate
in the testing activities described so as to minimise defect correction turn around
times.
© 1996 ICL Pathway COMMERCIAL IN CONFIDENCE
Page 23 of 72
FUJ00001273
FUJ00001273
ICL Pathway Ref: VI/STROOO1
TESTING & INTEGRATION STRATEGY Version:
2.0
Date:
30/09/96
a) The CM-BOM will be maintained under strict CM control, and will be used as the
basis of environment builds. As such either a separate BOM will be produced for each
environment type from Integration Test onward, or the single document will cover
each type as a distinct configuration.
a) Help Desk systems will employ the existing HCI styles pertaining to that area. BA
staff do not directly access Pathway systems or data and so no HCI style constraints
need to be considered here. This leaves only the POCL staff who will access the
Pathway systems via the counters interface for which an HCI style will be defined and
agreed at the outset. This will result in only a single HCI being presented to any
individual target user of the overall set of systems.
a) Activities at the Verification Centre will be conducted using the real application,
having passed System Test and Integration Test, before the products are released
further afield. This implicit dependency will be taken into account in the Programme
Schedules.
a) Failure rates permissible for products entering System Test will be defined and agreed
at the outset, and monitored closely.
a) BA/POCL systems will include test facilities appropriate for linking with Pathway as
described for Integration Testing and Model Office Test stages. (Test to Test status).
In the Live Trial links will be Live to Live.
a) External systems involved at each release (e.g. Gas) will in good time for testing of
that release make suitable test facilities available for linking to the Pathway test
facilities for the Integration Test and Model Office Test stages. (Test to Test status).
In the Live Trial links will be Live to Live.
© 1996 ICL Pathway COMMERCIAL IN CONFIDENCE
Page 24 of 72
FUJ00001273
FUJ00001273
ICL Pathway Ref: — VI/STROOO1
TESTING & INTEGRATION STRATEGY Version:
2.0
Date:
30/09/96
8. RESPONSIBILITIES
8.1 This section lays out a general framework of responsibilities for testing of the Pathway
system. The terms and roles used are intended to be generic rather than reference individual
job titles which may well change during the course of a large project. For example, the roles
of Quality Manager and Risk Manager may well be encompassed by the single job title
Director of Quality and Risk Management. This framework is in two principal layers - the
various general areas of responsibility that apply, and the more direct responsibilities that exist
for specific testing activities.
8.2 General
Role Responsibilities Keywords
Customer I Review/Sign-off test plans for scope of tests. Be I Coverage
involved in test planning and running later stages of I Acceptance
testing. Accept System.
Programme I Review/Sign-off General Testing Policy. Allow for I Policy
Manager testing dependencies in project plans. Sign-off System I Project Schedules
ready for release. System Acceptance
Technical Set performance targets. Review/Sign-off test plans for I System Performance
Manager performance and technical integrity. Technical Integrity
Operations I Set operational requirements. Review/Sign-off test plans I Handovers/Rollout
Manager for service management and _ operability. Close I Service Management
involvement in running of later stages. Operabilit;
Quality Apply QMS in testing area. Confirm handover process. I QMS & metrics
Manager Sign-off strategies. Check conduct of tests. Collect and I Handovers
interpret quality metrics. Check quality records. Sign-off I Strategies & Conduct
test stages. Quality Records
Risk Conduct risk assessments. Approve cost versus risk I Coverage
Manager evaluations. Sign-off test plans. Produce residual risk I Schedules
report. Input to acceptance process. Acceptance
CM Conduct handovers. Review environmental status against I Handovers, CM-BOM
Manager CM-BOM. Confirm integrity of configurations. Environments
Security Review / Sign-off test plans for security and access I Access
Manager coverage. Confirm retention periods for key documents / I Retention
products. Check security status for different I Environment / Data
environments / data usage.
Suppliers Ensure acceptable product quality on supply. Support I Product Quality
and testing process throughout life-cycle and provide I Support
Workforce _I adequate maintenance turnaround.
Testing & Sign-off Strategies and plans to ensure adequate test I Coverage / Conduct
Integration I coverage in all respects. Project Management and I Strategies / Plans
Manager general co-ordination of testing and integration activities I Testing Schedules
to schedule and budget, monitoring dependencies. I Environments / CM
Implicit responsibility for interfacing with all above I Quality / Security
areas. Acceptance
8.3 Specific
© 1996 ICL Pathway COMMERCIAL IN CONFIDENCE
Page 25 of 72
FUJ00001273
FUJ00001273
ICL Pathway Ref: — VI/STROOO1
TESTING & INTEGRATION STRATEGY Version:
2.0
Date:
30/09/96
Refer to the detailed process descriptions in the relevant section of the Testing Processes
document [11] for specific responsibilities of testers / team leaders / etc. for each product
activity relating to a particular stage of testing.
© 1996 ICL Pathway COMMERCIAL IN CONFIDENCE
Page 26 of 72
ICL Pathway
2.0
FUJ00001273
FUJ00001273
Ref: — VI/STRO001
TESTING & INTEGRATION STRATEGY Version:
30/09/96
9. UNIT TEST
9.1 Context
Date:
Performed for in-house development products only. Performed by Pathway and requiring
no direct Customer Involvement, but at all times open to witness by Customer
representatives at their discretion. Regarded as part of development. Applies to individual
modules and their inter-linking within a product. Formally planned as part of design, but
not necessarily scripted in detail. Follows the Unit Test Strategy produced for that
product. Informally executed, using debug facilities etc. as appropriate. Results formally
recorded and retained. First part of Functional Compliance Test Lifecycle. Two phases -
Module Test and Link Test (c.f. Unit Test process in Appendix 1).
9.2 Objectives
To demonstrate that each module developed in-house conforms with the detailed module
specification.
To pre-integrate co-operating modules within a product by demonstrating that their APIs
are implemented correctly and so that the modules link together as specified and co-
operate properly as an integrated unit.
To make the components of a product ready for Product Acceptance Test such that in
90% or more of cases, the product will not be rejected and require reworking.
To demonstrate that the unit conforms to the relevant HCI where applicable.
9.3 Overview
DESIGN
Unit Test Preparation
Module & APT
fou Module Link
Specifications I > I rest Plans Test Plans
Runtime Module Link
Modules = I rest Runs Test Runs
Unit Test Execution
CONSTRUCTION
Figure 6 - Schematic Overview of Unit Test
© 1996 ICL Pathway
COMMERCIAL IN CONFIDENCE
Page 27 of 72
FUJ00001273
FUJ00001273
ICL Pathway Ref: VI/STROOO1
TESTING & INTEGRATION STRATEGY Version:
2.0
Date:
30/09/96
9.4 Inputs/Outputs
Inputs: I Module Specification and API specifications.
Outputs: Diary of Testing, Module Test plans, Link Test Plans, Test Results,
Defect/Resolution Log, Pre-integrated functionally conformant modules.
9.5 Dependencies
Formal Code Walkthroughs are performed prior to Unit Test, excepting where Rapid
Application Development (RAD) approach has been adopted as the development
methodology. These reviews should include checks for adherence to standards and best
practices for efficiency, clarity and maintainability.
Test tools and environments as identified in the Unit Test Strategy are made available in
good time. (For Unit Test these may be no more than standard workbench facilities like
interactive debugging runtime tools.)
9.6 Complementary Testing
In parallel with Unit Test, and particularly in areas where a RAD approach has been
adopted, the early involvement of personnel from the Independent Testing Group will be
beneficial. Early versions of the evolving products will be taken and tested informally
against the evolving test plans. This will improve communication and levels of co-
operation between Development and Testing areas, will help to stabilise the test
environments prior to formal receipt of code, and will help to flush out problems at a
much earlier stage in the lifecycle, so improving the quality of the product delivered into
CM, and reducing the incidence of rejection in Product Acceptance Test.
© 1996 ICL Pathway COMMERCIAL IN CONFIDENCE
Page 28 of 72
ICL Pathway
TESTING & INTEGRATION STRATEGY
2.0
30/09/96
10. PRODUCT ACCEPTANCE TEST
10.1 Context
Ref:
FUJ00001273
FUJ00001273
VI/STROO01
Version:
Date:
Performed for all bespoke software products, irrespective of whether developed in-house
or subcontracted. (In accordance with the General Testing Policy [1] this does not apply
to standard proprietary items of hardware and software, such as MS Windows NT, or a
laser printer, where it is expected that the product will perform according to the
manufacturer’s specification.) Performed by Pathway with no Customer involvement.
Regarded as independent of development. Formally planned and scripted. Follows
Product Acceptance Test Strategy. Formally executed, preferably in automated, re-
runable fashion. Test results formally recorded and retained. Follows Unit Tests for
product concerned. Forms middle stage of Functional Conformance Test Lifecycle and
first stage of Architectural Conformance Test lifecycle. Precedes the Operational Trial
phase.
10.2 Objectives
To demonstrate that each product delivered into test conforms to the Product
Descriptions and/or Detailed System Specifications for that product, and so is fit for
wider testing use.
To confirm that the Human Computer Interface (HCI) employed conforms to that
designated as appropriate for that product set.
Forms formal acceptance of products from sub-contractors and in-house developers alike.
10.3. Overview
Product Descriptions
Functional Specifications
In-house products
Sub-contracted Products
Figure 7 - Schematic Overview of Product Acceptance Test
Ye Xe
Product Acceptance
Test Plans & Scripts
'
Product Acceptance
Test Execution
v
Accepted Products
© 1996 ICL Pathway COMMERCIAL IN CONFIDENCE
Page 29 of 72
FUJ00001273
FUJ00001273
ICL Pathway Ref: — VI/STROOO1
TESTING & INTEGRATION STRATEGY Version:
2.0
Date:
30/09/96
10.4 Inputs/Outputs
Inputs: Product Descriptions, Detailed System Specifications, Unit Tested Code
Outputs: Product Diary of Testing Activities, Acceptance Test plans and scripts, Test
Results, Product Rejection/Correction Log, Defect/Resolution Log, Functionally
conformant Products.
10.5 Dependencies
Products will be delivered into this stage in a timely manner and will be of high quality,
and on the whole be functionally conformant with their specifications. There is an
expectation that no more than 10% of products will need to be rejected here.
© 1996 ICL Pathway COMMERCIAL IN CONFIDENCE
Page 30 of 72
FUJ00001273
FUJ00001273
ICL Pathway Ref: VI/STROOO1
TESTING & INTEGRATION STRATEGY Version:
2.0
Date:
30/09/96
11. SYSTEM TEST
11.1 Context
Performed on receipt of accepted products from Product Acceptance Test area.
Performed by Pathway with progressively increasing Customer Involvement. Regarded as
separate from development. Formally planned and scripted, jointly by Pathway and the
Customer, encompassing their combined objectives. (Scripts reused in Integration Test
stage.) Follows System Test Strategy for that release, and is based on Business Threads
and Inter-System Threads formulated with and agreed by the Customer. Formally
executed, preferably in re-runable fashion. Run on simple single platform environment.
Test Results formally recorded and retained. Final stage of Functional Conformance Test
Lifecycle, and second stage of Architectural Conformance Test Lifecycle. Forms the first
stage of the Operational Trial phase. Also forms first part of Joint Business test activity.
11.2 Objectives
To demonstrate through a series of comprehensive business driven scenarios that the
software system as a whole functionally conforms with the agreed Requirements.
To expose the Infrastructure software to use by the application, and to demonstrate that
within the limitations of a reduced hardware platform it provides the application with the
specified support.
To perform initial verification of OHE requirements where available equipment, stationery
and other materials allow.
To confirm that the HCI employed by each product, when run in conjunction with each
other, does not result in a clash of HCI styles.
To expose the software system, and hardware as available and appropriate, to both
internal and external representative bodies for minority groups and dis-advantaged
peoples (e.g. Help the Aged). Specifically to seek positive feedback on any potential
problems in this area at the earliest possible stage.
To demonstrate simple recovery and resilience features of the system within the bounds of
the platform in use.
To form a comprehensive system regression pack for later use, and to prepare the way for
full system integration in the following stages of testing.
© 1996 ICL Pathway COMMERCIAL IN CONFIDENCE
Page 31 of 72
FUJ00001273
FUJ00001273
ICL Pathway Ref: VI/STROOO1
TESTING & INTEGRATION STRATEGY Version:
2.0
Date:
30/09/96
11.3 Overview
Pathway Baseline
Functional Specification
Acceptance Criteria Sy
and SLAs
—— System Test
Technical Environment Plans & Scripts
Description, incl, NFRs aa
Product Descriptions & ¥
Detailed System Specs.
System Test
Accepted Products = Execution
Functionally Conformant Sy
Figure 8 - Schematic Overview of System Test
11.4 Runtime Considerations
Regular checkpoints will be planned in. These will where practical exploit natural
quiescent points. Otherwise they will need to be engineered in. At the checkpoints all data
should be preserved in a restorable form to allow the test suite to be restarted from that
point at a later time should the need arise to re-test that particular portion following a
change to the baseline. This will help to keep the fix re-test and regression test costs down
to a bearable level and avoid wasting valuable elapsed time in unnecessarily re-running
entire test suites.
Test execution should be ‘captured’ (c.f. recommendations for tools in section 19 Test
Environments) for later automatic ‘replay’ when iteratively testing fixes or when
regression testing following changes.
Where practicable standard utilities will be used for checking actual results against
expected results, rather than manually checking. This should reduce costs and improve
accuracy. However, the approach to drawing up expected results should be pragmatic. In
the first instance, except for the most simple of cases, they should be couched in logical
rather than physical terms. First test runs can be checked against these manually. Once
consistent for the most part, these actual results can be taken, edited if required, and serve
as the physical expected results for subsequent iterations.
11.5 Inputs/Outputs
Inputs: Pathway Baseline Functional Specification [8], Acceptance Criteria [2], SLAs,
NFRs from TED [6], Product Descriptions, Detailed System Specifications.
Outputs: Business Threads, Inter-system Thread(s), System Test plans and scripts,
Defect/Resolution Log, Test Results, Functionally conformant system ready for use in
Integration Test.
© 1996 ICL Pathway COMMERCIAL IN CONFIDENCE
Page 32 of 72
FUJ00001273
FUJ00001273
ICL Pathway Ref: — VI/STROOO1
TESTING & INTEGRATION STRATEGY Version:
2.0
Date:
30/09/96
11.6 Dependencies
The Tools and environments indicated in the System Test Strategy are available for use in
good time.
Functionally conformant products delivered in the appropriate sequence to service the
running of the threads in the optimum fashion.
© 1996 ICL Pathway COMMERCIAL IN CONFIDENCE
Page 33 of 72
FUJ00001273
FUJ00001273
ICL Pathway Ref: VI/STROOO1
TESTING & INTEGRATION STRATEGY Version:
2.0
Date:
30/09/96
12. VERIFICATION CENTRE
12.1 I The Verification Centre is an established standard facility within ICL for preparing and
pre-validating configurations, packaging them ready for distribution. Pathway will
exploit this valuable facility for assembling the various hardware components,
configuring them correctly, loading the various software components, configuring
them properly, and running simple checks to confirm that all is operating in
accordance with specification. It operates in conjunction with Configuration
Management to establish Software and Hardware Baselines, or Builds. The
Configuration Management Bill of Materials (CM-BOM) in exploded form acts as the
specification for this task.
12.2 The Verification Centre first feeds the Integration Testing stage, following successful
System Testing. The build is progressively refined through Integration Test to form a
baseline configuration which can be employed in the Model Office Test, subsequently
in rollout to the Live Trial, and ultimately for the National Rollout.
Verification will be performed by the VC using the actual applications, as passed by
System Test, before distribution to the various target environments. (Schedules must
allow for this.)
The software products once passed by Product Acceptance Test are taken under CM
control. They are made available not just to System Test (ST) but also the Verification
Centre for early use. All fixes follow the same route throughout ST so that at the end
of ST the Verification Centre will be operating against the same code-set that passed
by ST, and a final pass of the verification can be made against this to establish the first
formal baseline (or build) before passing it, under CM control, into Integration Test
(IT), and later into the Model Office Test (MOT), in the Model Office. The same
principles then apply for all the later stages through to Live operation. The following
schematic illustrates. (Note - the VC activity is limited to the ‘counters’ sub-systems.)
© 1996 ICL Pathway COMMERCIAL IN CONFIDENCE
Page 34 of 72
FUJ00001273
FUJ00001273
ICL Pathway Ref: VI/STROOO1
TESTING & INTEGRATION STRATEGY Version:
2.0
Date:
30/09/96
12.3. Overview
; Hw
Suppliers SL Verification Centre
CM
Product Acep. Test —_— Assemble H/W
s/w
Load S/W
cM
Configure
cM *
System Test —— Verify
ST Trusted Status
Build
Configured System
Baseline
onfiguration pe
———"
Model Office Test —_—_—__ CM
————e
National Rollout —__v
Figure 9 - Schematic Overview of Verification Centre Role
12.4 Should any product delivered into the VC be found to be faulty, or if incorrect CM
control renders it so, then it will be rejected, and a corrected version awaited from the
supplier concerned. Similarly, should the configured system delivered from the VC
into the Model Office Test be found to be faulty with respect to the hardware or the
configuration, then again it will be rejected for the VC to correct. (It is important that
the failure rates here be kept very low as the disruption caused would otherwise be
great, particularly on schedules.)
12.5 Dependencies
The CM-BOM will be a complete configuration map covering all soft variables, such
as Operating System defaults, initialisation files, parameter settings, etc.
The VC will confirm all such configuration settings are as specified.
Project Schedules will reflect the need for the VC to use the actual application in its
later stages of verification, prior to release to Integration Test
Failure rates will be held below the following thresholds:
into the VC no more than 1% for hardware and proprietary software
no more than 0.5% CM handover errors
(these would be the subject of SLAs with Pathway’s suppliers)
from the VC no more than 0.1% during rollout
(incidents would be monitored by the VC and the test teams concerned)
© 1996 ICL Pathway COMMERCIAL IN CONFIDENCE
Page 35 of 72
FUJ00001273
FUJ00001273
ICL Pathway Ref: VI/STROOO1
TESTING & INTEGRATION STRATEGY Version:
2.0
Date:
30/09/96
13. INTEGRATION TEST
13.1 Context
Performed against a fully configured and properly installed system as specified by the CM-
BOM, following completion of System Test. Performed by Pathway with significant
involvement from Customer. Regarded as separate from development. Formally planned
and scripted, including significant reuse of System Test material. Follows Integration Test
Strategy for that release, and based on HLTPs developed from SLAs and NFRs, together
with Business Threads and Inter-System Threads carried forward from System Test.
Formally executed. Simulated loads re-runable. Run initially on a faithful platform (full
live shape and significant size) including all hardware and software variants, comms.,
peripheral equipment, etc. Progresses into Model Office environment. Includes all tests
planned for Model Office Test, and effectively pre-proves that stage. Interfaces to
BA/POCL Test systems are opened up (with appropriately selected / fabricated data) to
exercise full end to end data flow. Similarly, any external systems (e.g. Gas, Electric, etc.)
appropriate to the release concerned will be connected. (Assumed the agency concerned
will have appropriate test / simulation facilities.) Test Results formally recorded and
retained. Forms final stage of Architectural Conformance Test Lifecycle and first stage of
Business Integrity Test Lifecycle. Second stage of the Operational Trial phase. Also forms
major part of Joint Business Test activity
13.2 Objectives
To integrate total software and hardware system, verifying the system architecture on an
end to end basis and verifying the integrity of the CM-BOM which will be used later as
the blueprint for building the system ready for rollout into Live, including full National
Rollout.
To exercise end to end data flow, including BA/POCL and external agency systems as
appropriate, using selected / fabricated test data
To confirm that transit to the full target environment has not caused the system to regress
functionally.
To expose the full Infrastructure software to use by the full application, and to
demonstrate that in conjunction with the full hardware platform they provide the
application with the specified support.
To perform full verification of OHE requirements with all necessary equipment, stationery
and other materials, including volume, performance and accuracy aspects of its operation.
To confirm that the HCI employed by each new product introduced (e.g. the BA/POCL
systems, etc.), when run in conjunction with the Pathway products, does not result in a
clash of HCI styles for a given target user.
© 1996 ICL Pathway COMMERCIAL IN CONFIDENCE
Page 36 of 72
FUJ00001273
FUJ00001273
ICL Pathway Ref: VI/STROOO1
TESTING & INTEGRATION STRATEGY Version:
2.0
Date:
30/09/96
To expose the software and hardware system, to both internal and external representative
bodies for minority groups and dis-advantaged peoples (e.g. Help the Aged). Specifically
to seek confirmation that earlier feedback has been taken into account and to take further
feedback on any potential problems remaining in this area.
To confirm the performance attributes of the total system, verifying that the SLAs and
NFRs (as defined in the TED [6]) have been met.
To pre-prove the following objectives of the Model Office Test (MOT) in a joint testing
approach (Joint Business Test) in order to minimise the elapsed time required to conduct
the MOT. In this period of using the Model Office environment it is important live
implementation and support practices start to be employed in the background, with ever
less direct intervention by the test teams:
To ensure that all standard proprietary products (which are assumed to operate in
accordance with the manufacturers specification) are used fully within the context of
the overall system, and provide the functionality and support services expected of
them.
To confirm that the specified security aspects of the system (from the TED [6])
operate with integrity on the target live hardware platform.
To demonstrate the operational viability of the overall business system.
To confirm the operation of the service in ‘real’ time in order to simulate the Live
operation. (i.e. without continual intervention by the test team to manipulate the time,
as will be common in earlier stages of testing.)
To exercise the Help Desk operation and other support routes.
To serve as the final validation of OHE operation, and to prove the final versions of
Live stationery and other materials prior to bulk order.
To integrate the Business Procedures/Instructions with the IT System, and to verify
their joint operation.
To identify any major shortcomings in this joint operation and to overcome them
either by correction or interim management, as appropriate.
To validate the end to end rollout process and Configuration Management mechanism,
for each principal office type, in respect of both hardware and software, including a
‘Dress Rehearsal’ of the Implementation Plan for the first release applied to the Model
Office environment.
To confirm correct operation of VC and CM delivery and distribution services across
the range of system configuration variants that apply (e.g. the various post office
types, different versions of software and hardware, different builds and releases, etc.).
© 1996 ICL Pathway COMMERCIAL IN CONFIDENCE
Page 37 of 72
FUJ00001273
FUJ00001273
ICL Pathway Ref: VI/STROOO1
TESTING & INTEGRATION STRATEGY Version:
2.0
Date:
30/09/96
To confirm that the HCIs (against which the system components have already been
validated) deliver the appropriate levels of usability.
13.3 Overview
Pathway Baseline
Functional Specification
Acceptance Criteria
and SLAs, etc.
Architecture, NFRs and Plans & Scripts
Security aspects from TED
SA
— Integration Test
a
Performance Budgets {
from Technical Manager
Integration Test
Fully .
Configured 4 Execution
ST tested System th
Software
Architecturally Conformant SystemI
Ready for User Confidence Trial
Figure 10 - Schematic Overview of Integration Test stage
13.4 Runtime Considerations
Regular checkpoints should be planned in. These should where practical exploit natural
quiescent points. Otherwise they will need to be engineered in. At the checkpoints all data
should be preserved in a restorable form to allow the test suite to be restarted from that
point at a later time should the need arise to re-test that particular portion following a
change to the baseline. This will help to keep the fix re-test and regression test costs down
to a bearable level and avoid wasting valuable elapsed time in unnecessarily re-running
entire test suites.
Test execution should be ‘captured’ (c.f. recommendations for tools in section 19 Test
Environments) for later automatic ‘replay’ when iteratively testing fixes or when
regression testing following changes.
© 1996 ICL Pathway COMMERCIAL IN CONFIDENCE
Page 38 of 72
FUJ00001273
FUJ00001273
ICL Pathway Ref: VI/STROOO1
TESTING & INTEGRATION STRATEGY Version:
2.0
Date:
30/09/96
Where practical standard utilities should be used for checking actual results against
expected results, rather than manually checking. This should reduce costs and improve
accuracy. However, the approach to drawing up expected results should be pragmatic. In
the first instance, except for the most simple of cases, they should be couched in logical
rather than physical terms. First test runs can be checked against these manually. Once
consistent for the most part, these actual results can be taken, edited if required, and serve
as the physical expected results for subsequent iterations.
13.5 Inputs/Outputs
Inputs: Pathway Baseline Functional Specification [8], Acceptance Criteria [2], SLAs,
the TED [6] (detailing the system architecture, NFRs, performance and security aspects),
Performance Budgets derived from the SLAs by the Technical Manager, System Tested
fully configured and properly installed system with full target live hardware platform.
Outputs: Integration Test plans and scripts (in addition to the existing System Test plans
and scripts), Defect/Resolution Log, Fully integrated and operationally functional system
enabling the Verification Centre to baseline the build ready for entry to the Model Office
Test
13.6 Dependencies
The tools and environment as indicated in the Integration Test Strategy will be available
for use in good time. (These are likely to include system performance monitoring facilities
and simulated load generation engines for each of the hardware types making up the
overall configuration.)
The necessary BA/POCL test systems exist, are available for use, and are compatible.
The necessary external agency test systems exist, are available for use, and are
compatible, including external system data feeds (e.g. CAPS Payment data, Reference
Data files, OBCS Stop Lists, etc.).
© 1996 ICL Pathway COMMERCIAL IN CONFIDENCE
Page 39 of 72
FUJ00001273
FUJ00001273
ICL Pathway Ref: VI/STROOO1
TESTING & INTEGRATION STRATEGY Version:
2.0
Date:
30/09/96
14, JOINT BUSINESS TESTING
14.1 Context
Joint Business Testing is the result of a joint study conducted by the Customer and ICL
Pathway. It is a set of collaborative activities rather than a stage of testing. It spans the
System Test and Integration Test stages previously described, and leads directly to the
Model Office Test.
In essence the approach is one of partnership, pooling skills and resources to achieve a
common set of goals. The starting point is in establishing a comprehensive set of
objectives from both a Customer (Business) and Supplier perspective. These are
combined and used to drive out the necessary tests that must be performed in order to
satisfy both sets of objectives. Apart from gaining from the obvious economies of scale
and the removal of duplication here, there are also more far reaching benefits - most
notably the opportunity to improve the overall quality of the test preparation, by
exploiting both skill sets.
It involves full collaboration between Customer and ICL Pathway in the construction of
tests, with the material at the logical level (the HLTPs) being shared and reusable,
covering the full spectrum of test objectives. All of these tests are developed as LLTSs,
and in addition all those relating to business objectives are also developed from a business
perspective as Joint Business Test scripts. These are the tests ultimately to be run
independently by the Customer as the Model Office Test.
All the tests take their natural place in the System Test and Integration Test stages as
already described, and the Customer becomes progressively more active in the running of
these tests as time goes on. This culminates in the early use of the Model Office
environment in pre-proving the MOT.
14.2. Objectives
To forge a valuable partnership between the testing areas of the Customer and ICL
Pathway.
To provide early visibility of the product for the Customer to promote greater confidence
through open scrutiny.
To improve the quality of the test preparation by employing the joint skill sets of both
Business and Supplier.
To remove unnecessary duplication of effort, and to achieve natural economies of scale by
collecting the test objectives of both Customer and ICL Pathway at the outset and exploit
this combined knowledge in engineering joint tests that attack multiple objectives.
To pre-prove the tests to be run in the MOT and so to minimise the likelihood of
uncovering fresh defects at that late stage.
© 1996 ICL Pathway COMMERCIAL IN CONFIDENCE
Page 40 of 72
ICL Pathway
FUJ00001273
FUJ00001273
Ref: — VI/STRO001
TESTING & INTEGRATION STRATEGY Version:
2.0
30/09/96
Date:
To stabilise the Model Office environment to be used at MOT at an early stage and so to
eliminate wasteful false starts and awkward environmental problems in that stage.
Thus to minimise the elapsed time required for the MOT and so to better exploit that time
in joint proving of the system to improve the ultimate quality of the delivered product.
To provide a viable means of reducing the overall elapsed time required for all testing.
To facilitate the earlier ‘freezing’ of code in readiness for rollout.
To provide a vehicle for better understanding the nature of defects and so informing the
management of defect correction, on a cost versus risk basis.
14.3. Overview
Functional
Unit Test Conformance
Product Architectural
AcceptanceTest Conformance
System Test IBT
Integration Test
Model Office Test
Business Live Trial
Integrity
© 1996 ICL Pathway
COMMERCIAL IN CONFIDENCE
Page 41 of 72
FUJ00001273
FUJ00001273
ICL Pathway Ref: VI/STROOO1
TESTING & INTEGRATION STRATEGY Version:
2.0
Date:
30/09/96
15. MODEL OFFICE TEST
15.1 Context
Performed on receipt of fully configured and properly installed system from the
Verification Centre, following completion of Integration Test. Performed principally by
the Customer and supported by Pathway. Formally planned and scripted as part of the
Joint Business Test activity, being driven by the Customer agreed Acceptance Criteria [2]
and effectively forming a subset of the overall Pathway test set as performed in System
Test and Integration Test stages. Executed primarily by representatives drawn from the
end-user community, following Business Procedures/Instructions, in a ‘Model Office’
environment, running on a fully operational platform faithfully representing the Live
situation, with little or no simulation. This will include interfaces open to BA/POCL and
external agency test systems. These should be fully configured systems and where security
considerations permit they should employ elements of Live data, appropriately cleansed
where records contain personal details subject to the Data Protection Act. Forms the 2nd
stage of the Business Integrity Testing Lifecycle and the penultimate stage of the
Operational Trial phase.
15.2 Objectives
To demonstrate the operational viability of the overall business system
To serve as the vehicle for a separate independent trial of the system by the Customer,
driven by the Customer agreed Acceptance Criteria [2].
To confirm the running of the system in ’real’ time in order to simulate Live operation.
(i.e. restrict manipulation of the time by the test teams.)
To exercise the Help Desk operation and other support routes.
To serve as the final validation of OHE operation, and to prove the final versions of Live
stationery and other materials prior to bulk order.
To integrate the Business Procedures/Instructions with the IT System, and to verify their
joint operation.
To identify any major shortcomings in this joint operation and to overcome them either by
correction or interim management, as appropriate.
To validate the end to end rollout process and Configuration Management mechanism, for
each principal office type, in respect of both hardware and software, including a ‘Dress
Rehearsal’ of the Implementation Plan for the first release.
To confirm correct operation of VC and CM delivery and distribution services across the
range of system configuration variants that apply (e.g. the various post office types,
different versions of software and hardware, different builds and releases, etc.).
© 1996 ICL Pathway COMMERCIAL IN CONFIDENCE
Page 42 of 72
FUJ00001273
FUJ00001273
ICL Pathway Ref: VI/STROOO1
TESTING & INTEGRATION STRATEGY Version:
2.0
Date:
30/09/96
To confirm that the HCIs (against which the system components have already been
validated) deliver the appropriate levels of usability.
15.3. Overview
Acceptance Criteria
Business Procedures
& Instructions Plans Scripts
SA
cc I User Conf. Trial
a
Configuration Management
processes & mechanisms {
Fully Configured System cM
from the Verification Model Office Test
Centre, properly installed = Execution
in the Model Office
Y
System Independently Tested
Ready for Live Trial
Figure 11 - Schematic Overview of Model Office Test stage
15.4 Inputs/Outputs
Inputs: I Customer agreed Acceptance Criteria [2], Business Procedures / Instructions,
CM processes and mechanisms, fully validated IT System (passed Integration Test), fully
configured system installed in the Model Office.
Outputs: Model Office Test plans, supported by scripts where appropriate,
Defect/Resolution Log, System independently tested by Customer ready for Live Trial,
Validated rollout and CM process.
15.5 Dependencies
Business Procedures / Instructions ready for use in good time for the planning stage.
Model Office set-up agreed in good time for the planning stage, and prepared in good
time for the execution stage.
Appropriately skilled and trained Customer staff available to prepare test plans and scripts
and for the test execution stage.
Full OHE, stationery and other materials in place and verified in good time for final
© 1996 ICL Pathway COMMERCIAL IN CONFIDENCE
Page 43 of 72
FUJ00001273
FUJ00001273
ICL Pathway Ref: — VI/STROOO1
TESTING & INTEGRATION STRATEGY Version:
2.0
Date:
30/09/96
validation through the Model Office Test.
Help Desk configured and appropriately staffed and trained for initial exposure in Model
Office Test prior to Live running.
Rollout and CM processes and mechanisms defined, in place and verified in good time for
final validation through the Model Office Test.
Appropriate external system data available for use (e.g. CAPS payment data, Reference
Data files, OBCS Stop Lists, etc.).
© 1996 ICL Pathway COMMERCIAL IN CONFIDENCE
Page 44 of 72
FUJ00001273
FUJ00001273
ICL Pathway Ref: VI/STROOO1
TESTING & INTEGRATION STRATEGY Version:
2.0
Date:
30/09/96
16. LIVE TRIAL
16.1 Context
Performed following successful completion of the Model Office Test, and acceptance of
the system by the Customer as ready for rollout into limited Live operation. Performed by
the Customer with strong Pathway support. In effect a Live Pilot of the full operational
business system. Involving a representative number of Post Offices selected and prepared
for the exercise, for a period of about 3 months, with full support arrangements in place.
Close control exercised throughout. Takes place prior to full National Rollout.
16.2 Objectives
To serve as a Live Pilot before wider use across the UK.
To integrate all supplementary material and activities with the IT system as a total
business system.
To validate the training material and processes.
To validate the support arrangements, including Help Desk operation.
To validate the Migration and Implementation activities in a limited and closely controlled
sphere prior to wider application across the UK.
To establish initial reaction to the system from Post Office staff and the general public,
and so to provide an early warning of potential difficulties in a limited sphere of exposure.
16.3. Overview
Training Materials, etc.
Business Procedures ww
& Instructions
Live Trial
Help Desk Procedures and] Plans
other support arrangements A
Roll-out arrangements
(selected P.O.s, etc.)
Fully Configured System Live Trial
from the Verification = Execution
Centre installed in the
selected Post Offices qh
SystemReady for Service
Across the UK
Figure 12 - Schematic Overview of Live Trial stage
© 1996 ICL Pathway COMMERCIAL IN CONFIDENCE
Page 45 of 72
FUJ00001273
FUJ00001273
ICL Pathway Ref: VI/STROOO1
TESTING & INTEGRATION STRATEGY Version:
2.0
Date:
30/09/96
16.4 Dependencies
Rollout arrangements agreed in good time for the planning stage, and be completed in
good time for the execution stage. (To include selected Post Offices, their attributes, the
staff involved, etc.)
Help Desk and other support arrangements agreed in good time for the planning stage,
and to be in place in good time for the execution stage. (To include any user/ops guides,
instructions and procedures that are required, and staffing plans).
Training arrangements agreed, and supporting training material in draft in good time for
the planning stage. Training material to be produced and training given for selected
offices/staff in good time for the execution stage.
© 1996 ICL Pathway COMMERCIAL IN CONFIDENCE
Page 46 of 72
FUJ00001273
FUJ00001273
ICL Pathway Ref: VI/STROOO1
TESTING & INTEGRATION STRATEGY Version:
2.0
Date:
30/09/96
17. MAINTENANCE & REGRESSION TESTING
17.1. Background
Regression Testing, and testing activities during the Maintenance phase of a system are
often each distinct and separate from the testing that takes place in preparation for the
initial release of the system into Live running. This is not an appropriate approach to
adopt in a phased development / phased implementation programme, nor in one involving
multiple suppliers, multiple Customers, and multiple external interfaces. (It frequently is
not a successful approach in simpler single shot programmes cither.)
Both regression testing and the whole process of conducting tests during the Maintenance
phase for each release is built into the mainstream testing approach already described.
Each of the stages is designed to be iterative in nature, as it has to cope with a progressive
delivery stream. (i.e. in most instances, testing at a particular stage has to commence
before all the system products are available, and certainly before they are all correct.) For
this reason the test materials produced in the planning and scripting activities, and the
manner of test execution employed, already lends itself to use in regression testing
throughout the development and testing lifecycle, and so to the production of appropriate
and persistent Regression Test Packs for use later in the Maintenance phase of a given
release, and also as the starting point for the testing of the following release.
17.2. Test Materials
Throughout the test planning and scripting, emphasis is very much on formal analysis and
preparation of test materials, making each test and so each stream of tests (usually based
on Business Threads or other such test scenarios) definitive and repeatable. Within the
constraints of the prevailing software characteristics and hardware attributes, and subject
to the available toolsets that may apply, these tests are constructed to be as automated as
possible, to facilitate re-running, and to assist in checking of actual results against
expected results or previously obtained results.
17.3. Test Execution
Execution of a particular scenario proceeds in an iterative fashion. These iterations follow
a general pattern.
© 1996 ICL Pathway COMMERCIAL IN CONFIDENCE
Page 47 of 72
FUJ00001273
FUJ00001273
ICL Pathway Ref: VI/STROOO1
TESTING & INTEGRATION STRATEGY Version:
2.0
Date:
30/09/96
At the outset it is likely that not all of the products necessary to run the entire scenario
will be available. Nor is it likely that the target test environment works in every respect, or
that the run-time configuration is correct. The test scripts and supporting test data
themselves are likely to contain errors, and certain of the products delivered into testing
are likely to fail outright when first run in conjunction with each other. For these reasons
the initial iterations of a test stage are aimed at stabilising the environment and the run-
time configuration, correcting the test scripts and data, and identifying the more gross
product errors that prevent more constructive testing from making real headway. This
tends to be operated a little less formally, often circumventing problems ‘freestyle’ just to
get the tests through. Sometimes referred to as ‘Blitz testing’, this mode of operation
continues until the scenario is on the whole runable from start to end
At this point the scenario starts to be run more formally. Less ‘freestyle’ manipulation is
tolerated. Following delivery of fixes, after planned application to the test environment
concerned, the scenario either in part or in whole is re-run. It is the responsibility of the
test manager concerned to decide the most appropriate course of action at each point in
time, on a cost versus risk basis.
Before the test stage can be completed, the whole scenario must be re-run from start to
end, leaving only those defects outstanding that have been agreed.
The above principles apply to each test stage in turn.
17.4 Context
Performed in three phases of operation: re-test for errors found or other changes
introduced during the course of a test stage (in-stage regression testing); re-test for errors
found in later stages of testing and for other changes introduced after completion of a test
stage (in-programme maintenance); re-test for errors/enhancements after release has gone
live (live maintenance)
Regression packs are built and maintained implicitly throughout the testing lifecycle for in-
stage regression testing. Refined for efficiency following completion of each stage of
testing, to consist of a subset of the overall test scripts for that stage, for in-programme
maintenance. Refined further following Integration Test to form release level regression
test packs for live maintenance.
For in-stage regression testing and for in-programme maintenance, changes will be
routinely re-tested at entry level (Unit Test for in-house products, Product Acceptance
Test for Third Party products), and depending on impact analysis wider re-testing at later
test stages will also be applied, each as a discrete activity.
For live maintenance a similar process will apply, but using the more selective regression
test packs, and typically with all the various runs required being combined into a single set
of activity performed by a maintenance crew.
17.5 Objectives
© 1996 ICL Pathway COMMERCIAL IN CONFIDENCE
Page 48 of 72
FUJ00001273
FUJ00001273
ICL Pathway Ref: VI/STROOO1
TESTING & INTEGRATION STRATEGY Version:
2.0
Date:
30/09/96
To enable transparent regression testing against changes in a cost efficient and pragmatic
fashion throughout mainstream test execution for a given test stage.
To provide an effective subset of tests from all testing stages to allow focused validation
of changes to the system (either error fixes or enhancements) both during the testing of a
release (across test stages) and in Live running.
To reduce script maintenance costs by retaining tests which cover the full width of the
system’s functionality but without duplication.
To facilitate a single team managing maintenance testing of the system after it has gone
live.
17.6 Overview
[ [
Test Packs Regression Packs Regression Packs
(UT/PAT/ ST/IT) > D
for Test Stage for Release
¥ s
H Selected
Change I I Regression Packs
Proposals iD) Impact » : a
D Analysis
I
Products/System
returned to
Assured Status
Problem
Reports
Figure 13 - Schematic Overview of Maintenance Stage
17.7 Inputs/Outputs
Inputs: — Test packs from Unit Testing, Product Acceptance Testing, System Testing
and Integration Testing; Test Execution Plans from System/Integration testing; input data
files; actual results, Change Proposals; Problem Reports
Outputs: Regression Test Pack for each Test Stage, single Regression Test Pack for a
Release; Enhanced Test Execution Plans; Enhanced software validated effectively at
minimum cost.
© 1996 ICL Pathway COMMERCIAL IN CONFIDENCE
Page 49 of 72
FUJ00001273
FUJ00001273
ICL Pathway Ref: — VI/STROOO1
TESTING & INTEGRATION STRATEGY Version:
2.0
Date:
30/09/96
17.8 Dependencies
The workload for Independent Testing will contract to require only a small maintenance
group.
© 1996 ICL Pathway COMMERCIAL IN CONFIDENCE
Page 50 of 72
FUJ00001273
FUJ00001273
ICL Pathway Ref: — VI/STROOO1
TESTING & INTEGRATION STRATEGY Version:
2.0
Date:
30/09/96
18. EXTERNAL CERTIFICATION
Certain products employed by Pathway will require external certification before they can
be used operationally, for legal reasons. For example, the weigh scales in conjunction with
the PCs on the counter top will need to be examined and certified by the weights and
measures authority before they can be legally used in live operation out in the Post
Offices.
All such products will be formally identified for each release, together with the authorising
bodies concerned. These will then be detailed in the BRTS for that release, and the
External Certification activity will be included in the project plans, and monitored
accordingly.
There is no direct technical dependency on this certification, but it has an obvious bearing
on System Acceptance, and on the Live Trial.
© 1996 ICL Pathway COMMERCIAL IN CONFIDENCE
Page 51 of 72
FUJ00001273
FUJ00001273
ICL Pathway Ref: VI/STROOO1
TESTING & INTEGRATION STRATEGY Version:
2.0
Date:
30/09/96
19. TEST ENVIRONMENTS
19.1 General
This section covers the key attributes of the test environments as they apply to this
general approach to testing. First the environments themselves are described, and then the
supporting toolset. Whilst it is not essential to have sophisticated environments and tools
to perform high quality testing, a certain level of automation is required. Specific
environment and tool requirements will be detailed in the lower level strategies for each
test stage. Here follows brief generic descriptions to set the context.
19.2 Environments
As testing progresses through the stages, from Unit Test (UT), through Product
Acceptance Test (PAT), System Test (ST), Integration Test (IT), and on to the Model
Office Test (MOT) and Live Trial (LT), then so does the optimum environment type. In
general, the environment will start off supporting a more private and isolated mode of
testing - usually a single individual, with high levels of simulation and relaxed
configuration control, and move through to a fully shared environment with little or no
simulation and rigorous configuration controls.
UT environment - small, little data, harness based allowing simulated data and user input
and preferably simulated calls to better isolate the modules concerned. Private file set and
database, private on-line monitor (usually simulator to avoid high machine resource
costs). Interactive mode of operation highly desirable (e.g. debug facilities). Little or no
CM control. Not hardware specific. Many instances required, so optimise for low machine
resource usage.
PAT environment - small, little data but enough to link related products, may be harness
based but less important (simulated calls useful to overcome schedule dependencies).
Private file set and database, private on-line monitor (simulator still preferable). On
completion enters CM domain. Many instances required, so still optimise for low machine
resource usage.
ST environment - medium size, medium data volumes, sufficient to link entire business
threads, preferably no harness. No simulated calls, except in that all software and data sets
are non-Live (test systems used throughout). File sets and database used in both private
and shared modes. Real on-line monitor, not simulator. All software under CM control.
Operates on minimal hardware set, condensed configuration, no architectural layering.
Small numbers so do not need to optimise for machine resource usage.
IT environment - large size, large data volumes, sufficient to stress live configuration. No
harness, except a performance engine to generate simulated on-line load. No simulated
calls, except that test systems used throughout (no Live links). File sets, database and on-
line monitor all shared. All software and hardware under CM control, configured to CM-
BOM. Full live hardware set, full configuration, full architectural layering. Single
Instance. Able to act as each office type (configurable).
© 1996 ICL Pathway COMMERCIAL IN CONFIDENCE
Page 52 of 72
FUJ00001273
FUJ00001273
ICL Pathway Ref: VI/STROOO1
TESTING & INTEGRATION STRATEGY Version:
2.0
Date:
30/09/96
Model Office Environment - as for the IT environment, but configured against the verified
baseline, built by the Verification Centre (VC) and including a life-like office setting, used
initially in the IT stage, as part of the Joint Business Test activity, and then acting as the
principal platform for the MOT.
LT environment - Live, but with rollout to limited number of selected and prepared Post
Offices.
19.3 Tools
Given the timeframes involved for this programme, it is not sensible to consider deploying
new generic test management tools or test utilities as their usage takes some considerable
time to mature and the learning curves are prohibitive. Similarly static fault finding tools
tend to be platform and language specific (because they examine the source code) and so
cannot generally be employed across a mixed platform set. Dynamic fault finding tools are
likely to give the best return.
Of the numerous dynamic fault finding tools on the market, a shortlist will be drawn up
which satisfy Pathway’s principal platform types, and provide:
e Test drivers with capture / replay facilities for regression work
¢ Comparators for checking test results (including database)
e Interactive simulators (debuggers)
Each product will then be closely evaluated and demos run. The final contenders will be
put on trial in situ prior to purchase.
In addition to these generic tools, a test execution harness will be required for the Unit
Test stage, and to a lesser extent for the Product Acceptance Test stage. This will be
specified in the Unit Test Strategy and written in-house, for each principal platform type
concerned. Where simple wrap-around procedures or macros will assist in the execution
phase of other test stages, these will likewise be specified in the detailed strategies and
again written in-house.
Any hardware resident tools employed by Pathway in the Model Office environment (such
as diagnostic probes, line monitors etc.), other than equipment that is intended for use in
Live operation, will be removed prior to use for the MOT.
© 1996 ICL Pathway COMMERCIAL IN CONFIDENCE
Page 53 of 72
FUJ00001273
FUJ00001273
ICL Pathway Ref: — VI/STROOO1
TESTING & INTEGRATION STRATEGY Version:
2.0
Date:
30/09/96
APPENDICES
Al. TEST PROCESSES
A2. GLOSSARY OF TERMS
© 1996 ICL Pathway COMMERCIAL IN CONFIDENCE
Page 54 of 72
FUJ00001273
FUJ00001273
ICL Pathway Ref: — VI/STROOO1
TESTING & INTEGRATION STRATEGY Version:
2.0
Date:
30/09/96
APPENDIX 1
TEST PROCESSES OVERVIEW
Al.1 Appendix 1 to the Testing and Integration Strategy provides an overview of the
testing processes supporting the Pathway solution. Figure Al.1 below provides a high
level overview of the main processes, their key inputs and inter-dependencies. The
remainder of this Appendix provides an abridged view of the discrete processes which
support each major test activity.
For a more detailed description or for details of the remaining processes see the
Testing Processes document [11].
© 1996 ICL Pathway COMMERCIAL IN CONFIDENCE
Page 55 of 72
FUJ00001273
FUJ00001273
ICL Pathway Ref: VI/STRO001
TESTING & INTEGRATION STRATEGY Version: 2.0
Date: 30/09/96
Figure Al.1 - Schematic Overview of Testing Processes
© 1996 ICL Pathway COMMERCIAL IN CONFIDENCE
Page 0 of 72
FUJ00001273
FUJ00001273
ICL Pathway Ref: VI/STROOO1
TESTING & INTEGRATION STRATEGY Version:
2.0
Date:
30/09/96
Al.2__ UNIT TEST PROCESS
Al1.2.1 Purpose
To ‘white box’ test a single product developed in-house. Unit testing includes
validation of the discrete software modules (module testing) and their integration into
a complete product (link testing) prior to delivery into Product Acceptance Testing.
Al.2.2 Activities
Produce Unit test strategy for this product;
For each software module, analyse the Physical Design and produce Module Test
plans. While this testing is largely informal, the test analyst will need to ensure that
there is a clear definition of the input data states, run instructions and expected results;
Working from the Physical Design, the Pathway Baseline Functional Specification [8],
the relevant Product Description and relevant EPIDs, produce Link Test plans which
fully validate the complete integrated product. Link Test plans should document input
data states, run instructions, and expected results. Tests should also include cross
references back to the design products, allowing auditability and traceability;
Run Module Tests, with tools and environment as per Unit Test Strategy;
Check Module Test results - initially that they are produced and retained but then
subsequently checking against the previously retained
set, using them as expected results;
Sign off Module Tests;
Run Link Tests, with tools and environment as per Unit Test Strategy;
Check Link Test results against expected results;
Sign off Link Tests;
Produce Unit Test Report for this product.
© 1996 ICL Pathway COMMERCIAL IN CONFIDENCE
Page 0 of 72
FUJ00001273
FUJ00001273
ICL Pathway Ref: — VI/STROOO1
TESTING & INTEGRATION STRATEGY Version:
2.0
Date:
30/09/96
Al.2.3 Unit Test Activities Dependency Diagram
© 1996 ICL Pathway COMMERCIAL IN CONFIDENCE
Page 1 of 72
FUJ00001273
FUJ00001273
ICL Pathway Ref: VI/STROOO1
TESTING & INTEGRATION STRATEGY Version:
2.0
Date:
30/09/96
Al.3. PRODUCT ACCEPTANCE TEST PROCESS
A1.3.1 Purpose
To validate that a delivered product (bespoke only), be it in-house or third party,
meets the overall business and operational requirements as documented in the
Pathway Baseline Functional Specification [8], supporting Product Description and
EPIDs.
A1.3.2. Activities
Produce Product Acceptance Test Strategy for the release;
Test analyse the business and operational requirements (Requirements Catalogue
[15]), along with the relevant Product Description and EPIDs for a discrete product
(be it developed in-house or by a third party). Produce a series of High Level Test
Plans (HLTPs) which verify that the product has internal integrity, as well as correctly
manages its APIs. Tests must be cross referenced back to the Requirements Catalogue
[15],
Review the Physical Design documentation and progress the logical HLTPs to their
physical state, producing Low Level Test Scripts (LLTSs): these include definitions of
the input data states, run instructions and expected results. Tests should also include
cross references back to the design products, allowing auditability and traceability;
Run Product Acceptance Tests (PATs);
Check results from PATs against expected results;
Sign off PATs;
Produce Product Acceptance Test Report for the release.
© 1996 ICL Pathway COMMERCIAL IN CONFIDENCE
Page 2 of 72
FUJ00001273
FUJ00001273
ICL Pathway Ref: — VI/STROOO1
TESTING & INTEGRATION STRATEGY Version:
2.0
Date:
30/09/96
A1.3.3 Product Acceptance Test Activities Dependency Diagram
© 1996 ICL Pathway COMMERCIAL IN CONFIDENCE
Page 3 of 72
FUJ00001273
FUJ00001273
ICL Pathway Ref: VI/STROOO1
TESTING & INTEGRATION STRATEGY Version:
2.0
Date:
30/09/96
Al.4. SYSTEM TEST
Al1.4.1 Purpose
To confirm that, once integrated, the different products developed/bought meet the
baselined functional requirements.
To start to integrate the application software components with the supporting
infrastructure software and gain early exposure of simple recovery and resilience
issues.
To prepare the ground for Integration Testing.
Al.4.2. Activities
Produce System Test Strategy for the release;
Working in conjunction with the Customer a comprehensive and agreed set of test
points are derived from the Requirements Catalogue [15] and the PBFS [8], which
encompass both the Customer’s and Pathway’s testing objectives for that release.
Working with business representatives and Systems analysts, define and document a
set of Business Threads: these are a high level, logical statement of business flow
through the services. A mixture of threads will be required - some specific to a
discrete service (c.g. Benefit Payments, running through CMS, PAS, Middleware,
counter services); others running multiple integrated activities (c.g. Benefit Payments,
with POCL service such as sending parcels, buying stamps etc.);
Using the Business Threads as an input, test analyse the business and operational
requirements for the full set of products which constitute this release (using the
Requirements Catalogue [15], PBFS [8], Product Descriptions and EPIDs), producing
a set of High Level Test Plans (HLTPs). These expand the threads into greater detail,
and also extend the functional coverage. The HLTPs provide traceability back to
entries in the Requirements Catalogue [15] and the PBFS [8], as well as indicate any
sequencing dependencies within each thread. The business representatives will be
asked to review and agree the HLTPs;
Produce Inter-system threads which validate the interfaces with other systems (i.e.
outside Pathway domain, for example to CAPS, POCL Systems (e.g. FAD), British
Gas systems, etc.);
Produce the Test Execution Plan (TEP), which defines how to configure the Low
Level Test Scripts (LLTSs), ensuring that dependencies are satisfied but using the
different functional components within the HLTPs to provide a broad coverage of
functional permutations.
Migrate the HLTPs to a physical level, producing the Low Level Test Scripts
(LLTSs). LLTSs include definitions of input data, run instructions and expected
results. Tests should also include cross references back to the design products,
allowing auditability and traceability;
Produce the Infrastructure thread which will build the necessary data infrastructure to
© 1996 ICL Pathway COMMERCIAL IN CONFIDENCE
Page 4 of 72
FUJ00001273
FUJ00001273
ICL Pathway Ref: — VI/STROOO1
TESTING & INTEGRATION STRATEGY Version:
2.0
Date:
30/09/96
support system test activities: this will include any migration activities required;
Run System Tests, using the information contained in the LLTSs combined with the
Test Execution Plan to drive the combination of tests;
Check results against the expected results;
Sign off tests;
Produce System Test Report for the release.
Al1.4.3 System Test Activities Dependency Diagram
© 1996 ICL Pathway COMMERCIAL IN CONFIDENCE
Page 5 of 72
FUJ00001273
FUJ00001273
ICL Pathway Ref: VI/STROOO1
TESTING & INTEGRATION STRATEGY Version:
2.0
Date:
30/09/96
Al.S I INTEGRATION TEST
A1.5.1 Purpose
To confirm that the different products, be they developed in-house or bought, fully
integrate, meet the baselined functional and operational requirements and operate in a
live shape environment.
To prove the end to end architecture of the system solution, and in particular with
regard to resilience and recovery.
To confirm the performance attributes of the overall system and so verify that the
SLAs and the NFRs from the TED [6] are met.
A1.5.2 Activities
Produce Integration Test strategy for the release;
Test analyse the Service Level Agreements and Non-Functional Requirements for this
system taking supplementary detailed information from the technical specification and
the performance budgets.
Review the Threads produced for System Testing, and identify whether any of the test
requirements identified from the test analysis activity on SLAs and NFRs can be
satisfied by running the thread tests;
Produce specific Integration HLTPs to ensure full coverage of the SLAs and NFRs:
these HLTPs will cross reference the SLA and NFR products. In addition, cross
references are also provided between Thread tests being used to provide full coverage
of SLAs and NFRs. The business representatives will be asked to review and sign off
the HLTPs;
Produce the Test Execution Plan (TEP), using the System Test HLTPs as well as the
Integration HLTPs, defining how to configure the Integration Low Level Test Scripts
(LLTSs). This activity ensures that dependencies are satisfied, with the added
flexibility of using the different functional components within the HLTPs to provide a
broad coverage of functional permutations.
Migrate the Integration HLTPs to a physical level, producing the Integration Low
Level Test Scripts (LLTSs). LLTSs include definitions of input data, run instructions
and expected results. Tests should also include cross references back to the design
products, allowing auditability and traceability;
Upgrade the Infrastructure thread supporting System Testing to ensure that the data
infrastructure required to support Integration Testing is established: this will include
any migration activities required, as well as creation of the necessary data
infrastructure on existing systems (e.g. CAPs);
Run Integration Tests (which will include re-running of the selected System tests)
using the information contained in the LLTSs combined with the Test Execution Plan
to drive the combination of tests;
Check results against the expected results;
© 1996 ICL Pathway COMMERCIAL IN CONFIDENCE
Page 6 of 72
FUJ00001273
FUJ00001273
ICL Pathway Ref: — VI/STROOO1
TESTING & INTEGRATION STRATEGY Version:
2.0
Date:
30/09/96
Sign off tests;
Produce Integration Test Report.
A1.5.3 Integration Test Activities Dependency Diagram
APPENDIX 2.
© 1996 ICL Pathway COMMERCIAL IN CONFIDENCE
Page 7 of 72
FUJ00001273
FUJ00001273
ICL Pathway Ref: — VI/STROOO1
TESTING & INTEGRATION STRATEGY Version:
2.0
Date:
30/09/96
GLOSSARY OF TERMS
© 1996 ICL Pathway COMMERCIAL IN CONFIDENCE
Page 8 of 72
FUJ00001273
FUJ00001273
ICL Pathway Ref: — VI/STROOO1
TESTING & INTEGRATION STRATEGY Version:
2.0
Date:
30/09/96
Term Meaning
Acceptance Criteria The document(s) agreed between the Customer and the
Service Provider which lay down the criteria against which the
system will be measured at agreed points during the course of
testing in judging whether the system is to be formally
accepted by the Customer.
Architectural Conformance I The testing lifecycle wherein the test objectives are oriented
toward demonstrating that the system and its components
conform to their architectural specification, including aspects
such as structure, performance, security, operability,
resilience, recovery, and other NFRs.
Assured Status The state a system or its components have reached on
successful completion of a particular stage of testing - e.g.
“System Test Assured Status’ following successful completion
of System Test.
BA/POCL The Customer
Black Box Testing Testing of a system or its components which is planned and
conducted without knowledge of the inner workings, looking
only at the input, the specification of the system, and the
output.
Business Integrity The testing lifecycle wherein the test objectives are oriented
toward demonstrating that the overall system satisfies the
business need and when used in context and together with the
associated business products, retains overall business integrity.
Business Procedures The document(s) defining the business activity surrounding
and controlling the use by the Customer and their staff of the
computer system.
CM-BOM The structured information set (document or database) held by
CM which unambiguously defines the content of the system in
terms of deliverables known as Configurable Items, their inter-
relationships, and their target locations and runtime attributes.
Used as a blueprint of the system to guide and to audit the
assembly of the overall system.
Customer BA/POCL
Demonstrator Phase The bid phase defined by the Customer which demonstrates
proof of concept of the proposal. It precedes the ‘Operational
Trial Phase’.
© 1996 ICL Pathway COMMERCIAL IN CONFIDENCE
Page 9 of 72
ICL Pathway
2.0
30/09/96
(continued)
FUJ00001273
FUJ00001273
Ref: VI/STROOO1
TESTING & INTEGRATION STRATEGY Version:
Date:
EPIDs
Unambiguous definitions of the physical nature of an interface
between the system under test and an external system.
Functional Conformance
The testing lifecycle wherein the test objectives are oriented
toward demonstrating that the system and its components
conform to their functional specification.
HCI style
The standards by which a system is written to interface with
people. These cover for example the structure, layout, format,
colour, fonts, highlights and other conventions used on a
computer input/output screen.
ICL Pathway
The Service Provider.
Model Office Test
A trial of the whole system, following successful completion
of all formal testing (which will include all tests planned for
the Model Office Test), and prior to release of the system to
live, conducted independently by the Customer in the Model
Office environment, supported by Pathway.
National Rollout
(see also Rollout) The phase of the programme where,
following Live Trial, the system is subject to rollout on a
nation-wide basis according to an agreed timetable.
Operating Instructions
The document(s) defining the operational control of the
system, such as batch schedule control, recovery control, etc.
Operational Trial Phase
The main development and testing phase as defined by the
Customer, following the ‘Demonstrator Phase’.
Output Handling Equipment
Ancillary equipment associated with the computer system or
its output, which extends the service offered. For example,
equipment used in conjunction with computer printers, to say
collate and envelope and frank output targeted for posting.
Performance Budgets
Targets set by the Technical Manager and derived from the
SLAs, which specify the required performance characteristics
of the system and its components.
Real time
Here used to differentiate between modes of test execution,
not to be confused with Real-time systems. It means where
test teams do not manipulate the time during the running of a
test, but rather operate the test in ‘real’ time.
Regression Testing
The process of demonstrating that a system and_ its
components have not regressed to a ‘worse’ state following a
change (usually to the software).
© 1996 ICL Pathway
COMMERCIAL IN CONFIDENCE
Page 10 of 72
FUJ00001273
FUJ00001273
ICL Pathway Ref: VI/STROOO1
TESTING & INTEGRATION STRATEGY Version:
2.0
Date:
30/09/96
(continued)
Rollout The process of packaging and distributing the system out to
the various locations at which it will operate and be used (see
also National Rollout).
Service Level Agreement The document(s) agreed between the Customer and the
Service Provider which lay down the minimum operational
criteria to which the service must be run and maintained.
Service Provider Pathway - providing the service of development, maintenance
and operational running of the BA/POCL system, according to
agreed requirements and Service Level Agreements.
Supplier one of the suppliers of Pathway - a 3rd party providing goods
or services to Pathway, or one its direct sub-contractors
working to specification by and on behalf of Pathway.
Validation The process of evaluating products (at the end of a given
phase) to demonstrate compliance with their specified
requirements.
Verification The process of evaluating the products of a given phase to
ensure correctness and consistency with respect to the
products and standards provided as input to that phase.
White Box Testing Testing of a system or its components which exploits
knowledge of or interrogates the inner workings.
© 1996 ICL Pathway COMMERCIAL IN CONFIDENCE
Page 11 of 72
FUJ00001273
FUJ00001273
ICL Pathway Ref: — VI/STROOO1
TESTING & INTEGRATION STRATEGY Version:
2.0
Date:
30/09/96
END OF DOCUMENT
© 1996 ICL Pathway COMMERCIAL IN CONFIDENCE
Page 12 of 72