ICL Pathway
FUJ00078295
FUJ00078295
REVISIONS TO THE Ref: VI/STROO6
TESTING & INTEGRATION APPROACH Version: 2.0
Document Title:
Document Type:
Abstract:
Status:
Distribution:
Author:
Approval Authority:
Signature/Date:
Quality Authority:
Signature/Date:
Customer Authority:
Signature/Date:
FOR PATHWAY RELEASE 2 Date: 24/10/97
REVISIONS TO THE TESTING & INTEGRATION APPROACH
FOR PATHWAY RELEASE 2.0
Strategy
This document defines the proposals for revising the strategic
approach to be adopted for the testing and integration of Pathway
products for BA/POCL, for delivery at Release 2. Its scope is the full
band-width of existing Testing and Integration activities and covers:
Changes in practice which have evolved during the course of Release
1; Changes proposed to better reflect the circumstances at Release 2;
Lessons learnt from Release 1.
It should be read in conjunction with the existing documentation set
and in particular: General Testing Policy; Testing and Integration
Strategy: Application Product Acceptance Test Strategy: System Test
Strategy; System Integration Test Strategy.
Final
Pathway Management Team
(and further at their discretion)
J Hunt
Terry Austin, Systems Director
David Groom, Quality Manager
Simon Rilot, PDA Integration & Testing Manager
© 1997 ICL Pathway
COMMERCIAL IN CONFIDENCE
Page 0 of 41
FUJ00078295
FUJ00078295
ICL Pathway REVISIONS TO THE Ref: VI/STROO6
TESTING & INTEGRATION APPROACH Version: 2.0
FOR PATHWAY RELEASE 2 Date: 24/10/97
DOCUMENT HISTORY
Version Date Reason
0.1 28/08/97 Initial draft - not issued
0.2 02/09/97 Internal version only - issued for Pathway Management Review
0.3 05/09/97 Issued as early information for PDA, and wider internal review
0.4 02/10/97 Issued for final review and approval within Pathway - internal Baseline
1.0 13/10/97 Approved by Pathway as initial baseline
2.0 24/10/97 Approved by PDA as formal baseline for update of contractual documents
CHANGES FROM LAST ISSUE
Ref. Change
n/a Application of final comments received from PDA , and their approval
CHANGES FORECAST
Change Target Issue
None na
ASSOCIATED DOCUMENTS
Ref. Library Ref. Title Source
ea VI/POLOO1 General Testing Policy Pathway
[21 VI/STROOL Testing and Integration Strateg) Pathway
Bl VI/PLA002 Operational Trial Plan - Technical Test Release I Pathway
[4] VI/STROO3 _I Application Product Acceptance Test Strategy - Release I__I Pathway
[5] VI/STROO04 _I System Test Strategy - Release 1 Pathway
[6] VI/STROOS Systems Integration Test Strategy - Release 1 Pathway
ABBREVIATIONS & GLOSSARY OF TERMS
The definition of any abbreviations and terms used in this document are generally to be found in the Testing
and Integration Strategy [2] and so are not repeated here.
© 1997 ICL Pathway
COMMERCIAL IN CONFIDENCE
Page I of 41
ICL Pathway REVISIONS TO THE
TESTING & INTEGRATION APPROACH
FOR PATHWAY RELEASE 2
CONTENTS
1.
INTRODUCTION
MANAGEMENT SUMMARY
SCOPE OF TESTING
HIGH LEVEL TEST OBJECTIVES
UNIT TEST
PRODUCT ACCEPTANCE TEST
PRODUCT INTEGRATION TEST
SYSTEM TEST
INTEGRATION TEST
. JOINT BUSINESS TEST
. MODEL OFFICE TEST
. LIVE TRIAL
. MAINTENANCE & REGRESSION TESTING
. EXTERNAL CERTIFICATION
. TEST ENVIRONMENTS
TEST AUTOMATION
. TEST SCRIPTING
ACCEPTANCE TRIALS
Ref:
Version:
Date:
FUJ00078295
FUJ00078295
VI/STROO6
2.0
24/10/97
© 1997 ICL Pathway
COMMERCIAL IN CONFIDENCE
Page 2 of 41
FUJ00078295
FUJ00078295
ICL Pathway REVISIONS TO THE Ref: VI/STROO6
TESTING & INTEGRATION APPROACH Version: 2.0
FOR PATHWAY RELEASE 2 Date: 24/10/97
1. INTRODUCTION
1.1 Background
The overall strategic approach for testing within ICL Pathway was agreed with the PDA in
September 1996 and is described in two principal documents - the General Testing Policy [1], and
the Testing and Integration Strategy [2]. The specific testing strategies for each constituent stage of
Pathway testing for Pathway Release 1 were similarly agreed with the PDA in October 1996 and are
described in the following documents: Operational Trial Plan [3]; Application Product Acceptance
Test Strategy [4]: System Test Strategy [5]: System Integration Test Strategy [6].
As the testing activities for Release 1 draw to a close and those for Release 2 commence, a review of
these strategies has been conducted, looking at the actual performance and effectiveness of the test
activities at Release 1, examining the difficulties encountered, and taking stock of the current
situation. There are lessons to be learnt, evolutions in terms and practice to be reflected, and changes
in circumstance to take into account.
This document summarises the proposals for change arising from the findings of that review.
1.2 Context
The structure of this document follows closely the general structure of the Testing and Integration
Strategy [2]. This should make it easier both to review in the first place, and also easier then to apply
the necessary changes to that document, and to produce the associated family of strategy documents
for Release 2 from their existing equivalents for Release 1.
1.3 Underlying Approach for Release 2
When applying these proposals to the Strategy documents and plans for Release 2, it is important to
understand the underlying approach being adopted, and this will need to be covered in the Business
Release Test Strategy for Release 2. The approach is one of recognising that Release 2 is a combined
release, arising from that which was previously known as Release 1E, with a series of upgrades being
applied to it to satisfy the broader range of functionality and infrastructure now described as Release
2.
This approach is adopted because the 1E componentry is delivered already and so testing can
commence against this without delay. Release 1E forms a considerable proportion of the target
Release 2, and so with careful selection little nugatory work will be involved. The additional
componentry will then be delivered, and intercepted at appropriate points, incrementally over the
coming months. To avoid undue maintenance overheads, and unnecessary disruption to the test
baselines, System Test will initially be confined to the 1E area. Further System Test activities will
then be planned to cover the major components subject to change (e.g. BPS, EPOSS, etc.) as
required.
Of particular note at Release 2 is the new online interface with CAPS. This is recognised as a
significant area for test at Release 2, and will require consideration in all test stages.
1.4 Next Steps
Following approval of these proposals a CP will be raised to formally enact the changes and authorise
the test managers concerned to start following the changes without delay.
The relevant strategy documents will then be produced/updated accordingly as a matter of urgency to
reflect the revised position and assist in consistent adoption of the changes at a more detailed level.
The complementary testing processes held in the online library will then likewise be reviewed and
updated accordingly.
© 1997 ICL Pathway COMMERCIAL IN CONFIDENCE
Page 3 of 41
FUJ00078295
FUJ00078295
ICL Pathway REVISIONS TO THE Ref: VI/STROO6
TESTING & INTEGRATION APPROACH Version: 2.0
FOR PATHWAY RELEASE 2 Date: 24/10/97
15 Documentation Map
The following diagram is an updated version of that given in section 1.3 of the Testing and
Integration Strategy [2], reflecting the changes proposed in this document.
Joint Testing Service Level Service Architecture ‘Technical Environment
"Approach ‘Agreements and Design Document Description Release
~~ I a Policy
Testing - General
Processes Testing & Integration Strategy Sain
Policy
Acceptance eo oe
Access Control
Pro
cess Policy
Performance Release Contents
Budgets fino
NFR . Migration
canlowue Business Release Test Strategy Sena
Security — ~~ implementation
Functional Spee { { { q Surategy
Unit Test inte tee System I I Tntegration Accepance Model tre
Strategies is _ « « vial Office Test «
Strategy swategy I I strategy I] strategy Stars Strategy
PIT Business Threads & —— Acep. M.O. Live Trial
Unit Test Plans High Level Test Plans Specs. Plans Plans
m 9 § § 8 8
Low Level Test Scripts and Business Test Scripts
1.6 Source Documentation
AS the diagram suggests, there is a great deal of key source documentation which underpins the test
preparation phase and which is critical to setting the correct scope and coverage of the testing to be
performed. Whilst the joint working approach does to some extent mitigate against possible
deficiencies in this source material, it is no substitute. It is taken as a pre-requisite for this testing
approach that all the key source documents are complete and available for Release 2 sufficiently early
to support the test preparation phase. These include: SADD; TED; RCD; NFR Catalogue;
Performance Budgets; Migration Strategy; Implementation Strategy; SFS.
© 1997 ICL Pathway COMMERCIAL IN CONFIDENCE
Page 4 of 41
FUJ00078295
FUJ00078295
ICL Pathway REVISIONS TO THE Ref: VI/STROO6
TESTING & INTEGRATION APPROACH Version: 2.0
FOR PATHWAY RELEASE 2 Date: 24/10/97
2. MANAGEMENT SUMMARY
2.1 The underlying approach adopted for Release 2 is one of recognising that Release 2 is made up of the
existing 1E components with a series of upgrades being applied to it to satisfy the broader range of
functionality and infrastructure now described as Release 2, and then exploiting that position to allow
testing to commence against these existing components without delay. To avoid undue maintenance
overheads, and unnecessary disruption to the test baselines, System Test is initially confined to this
1E area which forms a considerable proportion of the target Release 2. Then with careful planning
the additional / updated components are intercepted as appropriate, either with additional System
Test activities, or in the remaining test stages, with little or no nugatory work involved.
2.2 A number of changes to the testing approach are proposed for Release 2, which include lessons learnt
from Release 1, formally adopting terms and practices which are already generally accepted, and a
reflection of the change in circumstances at Release 2.
2.3 In general the overall scope of Pathway testing, and the corresponding high level objectives of that
testing remain unchanged. It is rather the means to that end which changes.
24 The principal changes described are:
a) Remove the Product Acceptance Test stage, and introduce the Product Integration Test stage, now
positioned prior to CM rather than following it, and concentrating on software integration to
achieve reliable configuration definitions and build instructions allowing consistent replication of
test environments across all platforms.
a) Introduce a new organisational unit - ‘Technical Integration’ - sitting between ‘Development’ and
‘Testing & Business Integration’ (formerly known as ‘Testing & Integration’), to co-ordinate the
efforts of the new Product Integration Test team with that of Configuration Management and the
Service Provision & Technical Support team.
a) Accelerate the planned replacement of the existing interim Configuration Management tool with
PCMS, as Pathway’s strategic Configuration Management tool. This is well respected and widely
adopted CM tool across the industry and will equip Pathway in dealing with what is recognised as
a highly complex system configuration, spanning many platforms and embracing a wide range of
technologies.
a) Formally adopt generally accepted current practice dubbed the ‘three pass model’ for both System
Test and Business Integration Test. This breaks these test activities into three distinct phases of
test execution:
¢ 1 Pass - stabilise
¢ Main Pass - iterative defect removal
¢ Final Pass - audit trail
e) Formally adopt the generally accepted terms describing the different test streams within the
Integration Test stage (but see also (g) below):
« DIT - Direct Interface Test
© T&S - Technical & Security Test
© BIT - Business Integration Test
¢ E2E - End to End Interface Test
¢ MOR - Model Office Rehearsal
f) Recognise that these are all complementary test streams within a single Pathway test stage, and in
particular that the T&S and BIT streams must run alongside the E2E and MOR streams to
properly regression test the full bandwidth of the system right up to the point of handover to
© 1997 ICL Pathway COMMERCIAL IN CONFIDENCE
Page 5 of 41
FUJ00078295
FUJ00078295
ICL Pathway REVISIONS TO THE Ref: VI/STROO6
TESTING & INTEGRATION APPROACH Version: 2.0
FOR PATHWAY RELEASE 2 Date: 24/10/97
Ra
Model Office Test.
Introduce a formal walk-through of end to end procedures, spanning all systems areas, and
scheduled prior to E2E main pass and MOR2
Formally adopt the generally accepted current practice of commencing DIT well ahead of the
other test streams within the Integration Test stage, and form hard finish-start dependencies for
entry and exit of DIT, requiring all participating external systems to ‘freeze’ (put under external
change control) prior to DIT, and complete DIT prior to E2E and MOR.
Formally adopt the generally agreed current practice of defining and agreeing the DIT scope and
coverage within jointly produced and signed-off Interface Test Specification documents.
Formally adopt the current (occasional) practice of a pre-DIT activity, across the board, with
specified objectives:
* Bench-check each item on the interface specification against own system’s outputs and
expected inputs
¢ Exchange fabricated interface files, checking each other’s against the interface
specification to confirm each other’s interpretation of that specification
* Exchange system generated interface files (warts and all, and manually if necessary) and
again check each other’s against the interface specification.
Formally adopt the generally accepted terms within Technical & Security Test:
© Security Test
Performance Test
« Integrity Test
e Systems Management Test
Avoid confusion regarding Systems Management Test by producing a specific test strategy
detailing our approach in this area, which in summary is to treat Systems Management as a
Managed Service delivered to Pathway.
Improve efficiency and traceability within the Technical & Security Test area with the production
of a comprehensive NFR catalogue and specific system performance targets to plan the required
tests against. This would bring together all the relevant data currently scattered across various
contractual and system documentation.
Recognise the critical role of BIT, and so to significantly extend the planned duration of this
activity. (See also (f) above.)
Extend the BIT remit to cover regression testing of external interfaces.
Improve the level of Joint Business Test involvement in Pathway test stages, in terms of both
numbers and skills, in line with the original agreements, in order to achieve the benefits
promised. Current practice has resulted in considerable levels of rework within Pathway, and
potentially a poorer quality product delivered into Model Office Test than would otherwise have
been possible.
Adopt a more live-like approach to Model Office Test along the lines of that proposed by CAPS.
Increase vigilance in respect of regression testing across the Pathway test stages by strengthening
the detailed test processes accordingly
Simplify the Implementation approach at Release 2 to initially go live with a single Data Centre -
Bootle - and exploit this to allow early building of the target live environment, and use this
© 1997 ICL Pathway COMMERCIAL IN CONFIDENCE
Page 6 of 41
FUJ00078295
FUJ00078295
ICL Pathway REVISIONS TO THE Ref: VI/STROO6
TESTING & INTEGRATION APPROACH Version: 2.0
FOR PATHWAY RELEASE 2 Date: 24/10/97
extensively during the Integration Test stage in both BIT and MOR.
Progressively introduce Test Automation tools which have now become available, and exploit
these in Release 2 testing to allow more extensive regression testing with less risk of human error.
Improve the test scripting method used.
Merge the Acceptance Trial activities with the existing Integration Test stage to avoid
unnecessary massive duplication of effort.
Produce guidance notes including checklists, to assist test managers, improve consistency, and
raise visibility of the decision making process surrounding the progress of testing within and
between test stages and test streams. It is recognised that in reality there are few hard and fast
dependencies, but rather a complex and subjective ongoing process of assessment - cost vs risk.
Where real dependencies exist these will be highlighted. Elsewhere the guidance notes and
checklists will seek to inform the assessment process.
© 1997 ICL Pathway COMMERCIAL IN CONFIDENCE
Page 7 of 41
FUJ00078295
FUJ00078295
ICL Pathway REVISIONS TO THE Ref: VI/STROO6
TESTING & INTEGRATION APPROACH Version: 2.0
FOR PATHWAY RELEASE 2 Date: 24/10/97
3. SCOPE OF TESTING
3.1 The overall scope of testing described in the original documents has not changed.
3.1 However, there have been a number of evolutions during the course of Release 1 testing which need
to be formally reflected, such as the three pass testing model adopted in System Test and Business
Integration Test, and the sub-dividing of Integration Test into Security Test, Technical Test, Direct
Interface Test, Business Integration Test, End-to-End Interface Test, and Model Office Rehearsal.
3.1 There have also been a number of salutary lessons learnt during the course of Release I testing which
also need to be reflected accordingly, such as the replacement of Product Acceptance Test with
Product Integration Test.
3.1 All these changes are described in more detail under the appropriate sections below
© 1997 ICL Pathway COMMERCIAL IN CONFIDENCE
Page 8 of 41
FUJ00078295
FUJ00078295
ICL Pathway REVISIONS TO THE Ref: VI/STROO6
TESTING & INTEGRATION APPROACH Version: 2.0
FOR PATHWAY RELEASE 2 Date: 24/10/97
4, HIGH LEVEL TEST OBJECTIVES
4.1 All the existing high level testing objectives remain unchanged.
4.1 A small number of additional ones are introduced, such as with the introduction of the Product
Integration Test stage.
4.1 Some are reorganised and expanded, such as with the sub-dividing of Integration Test into Security
Test, Technical Test, Direct Interface Test, Business Integration Test, End-to-End Interface Test, and
Model Office Rehearsal
4.1 All these changes are described in more detail under the appropriate sections below.
© 1997 ICL Pathway COMMERCIAL IN CONFIDENCE
Page 9 of 41
FUJ00078295
FUJ00078295
ICL Pathway REVISIONS TO THE Ref: VI/STRO06
TESTING & INTEGRATION APPROACH Version: 2.0
FOR PATHWAY RELEASE 2 Date: 24/10/97
5. UNIT TEST
5.1 Nochanges proposed.
© 1997 ICL Pathway COMMERCIAL IN CONFIDENCE
Page 10 of 41
FUJ00078295
FUJ00078295
ICL Pathway REVISIONS TO THE Ref: VI/STRO06
TESTING & INTEGRATION APPROACH Version: 2.0
FOR PATHWAY RELEASE 2 Date: 24/10/97
6. PRODUCT ACCEPTANCE TEST
6.1 One of the lessons learnt from Release I testing has been that the assumptions relating to the extent
to which products would be integrated prior to delivery into Pathway were over ambitious given the
extremely complex multi-platform environments used. This resulted in products spending a
prolonged period of time within the environment build phase, and it being almost impossible to
effectively carry out Product Acceptance Test in the way originally envisaged. This in turn then
adversely impacted the consistency and initial quality of products passing through into System Test.
6.1 Aneed was identified for an additional testing activity prior to delivery into independent test, to
integrate the products being delivered and confirm their configuration prior to formally controlling
that configuration within CM. (See Product Integration Test section below)
6.1 The way in which it is proposed to structure the Product Integration Test activity, with testers
seconded into the activity acting as agents of the testing group, effectively removes the need for a
separate PAT activity.
6.1 It is therefore proposed to remove the Product Acceptance Test stage altogether. (Note, all the high
level objectives for this test stage remain, under the banner of Product Integration Test.)
© 1997 ICL Pathway COMMERCIAL IN CONFIDENCE
Page 11 of 41
FUJ00078295
FUJ00078295
ICL Pathway REVISIONS TO THE Ref: VI/STROO6
TESTING & INTEGRATION APPROACH Version: 2.0
FOR PATHWAY RELEASE 2 Date: 24/10/97
7A
72
73
74
7S
PRODUCT INTEGRATION TEST
Background
See section on Product Acceptance Test above.
Note: ‘Integration’ as a term is used in wide and varying ways. To avoid confusion, the term as used
here, for Product Integration Test (as described below), is distinct from that used in Integration Test
(as described in section 9). In Product Integration Test it refers to the fairly low level integration of a
product’s configurable items (constituent components) such that the configuration of the product and
its internal interfaces can be confirmed. Whereas in Integration Test it refers to the relatively high
level integration of entire application systems and full operational infrastructure to form a coherent
set of services.
Organisation
It has been agreed to establish a new organisational group within the Pathway Systems Directorate.
Called ‘Technical Integration’, it fits in between ‘Development’ and ‘Testing & Business
Integration’ (previously known as Testing & Integration). It effectively subsumes the CM area and
the various build teams which made up the SPTS area within Testing & Integration.
The new Technical Integration unit will be headed by the Technical Integration Manager, reporting
directly to the Systems Director. It comprises three principal areas:
PIT (Product Integration Test)
CM (Configuration Management)
SPTS (Service Provision & Technical Support)
PIT
This is a new area, introduced to address many of the difficulties experienced in establishing effective
control of the test environments. Significantly, it is positioned in the life-cycle PRIOR to formal
delivery into CM. The role is one of sofiware integration - to establish coherent configurations for
each product set, making them ready for use by SPTS in building the various environments required
by the Testing group. This activity is performed by a combined mixed discipline team drawn from all
the main participants - Development, CM, SPTS, and Testing.
CM
This area remains essentially the same, though the emphasis shifts to active management of stable
configurations AFTER they have been established and proven.
SPTS
This area again remains essentially the same, though the emphasis returns to the original objectives -
building and maintaining test environments from an established configuration taken from CM, and
providing technical support and first line diagnosis services to the testing teams.
© 1997 ICL Pathway COMMERCIAL IN CONFIDENCE
Page 12 of 41
FUJ00078295
FUJ00078295
ICL Pathway REVISIONS TO THE Ref: VI/STROO6
TESTING & INTEGRATION APPROACH Version: 2.0
FOR PATHWAY RELEASE 2 Date: 24/10/97
7.6
V7
Objectives
All the previous objectives of PAT are carried forward into PIT. It is the responsibility of the
Development and Testing staff seconded into PIT for that product set to effect these objectives. In
summary these are:
© To act as the formal acceptance of products from 3" party suppliers (emphasis here on
the Development staff)
© To confirm the HCI conforms to specification (both Development and Testing staff)
¢ To demonstrate that each product conforms in gross terms to specification, and that it is
fit for entry to System Test and not likely to be disruptive there (emphasis here on the
Testing staff)
In addition there are some new specific objectives of PIT relating to the integration activity it
performs, as follows:
© distinguish between environmental (shell) and application product sets
* establish proven PBS within CM
establish proven Build scripts and configuration settings for shells within CM, for each
major testing platform
establish proven build scripts and configuration settings for applications within CM
enable reliable and consistent generation of test environments from CM information
form product level regression testing packs
develop platform image replication techniques
develop supporting toolsets for environment build activities
widen Pathway’s product knowledge and support skill base
© promote better inter-working between Development, CM, and Testing
Process Flow Overview
There are a number of separate and parallel product delivery streams managed by development,
covering both infrastructure and application, and originating either from in-house developments, 3"
party developments, or shrink-wrapped products. These fall into three principal categories:
Shell Products - Underlying hardware and software infrastructure products
Applications - Those products (mostly business applications) which are installed within
the shell
¢ Deltas - incremental changes authorised by PinICLs or CPs which do not impact the
structure of the existing configuration, only the content.
The first two categories of products, plus all associated documentation/definitions are handed over by
Development into PIT, and accompanied by Development staff with relevant product experience. It is
assumed that appropriate Unit Test Link Test activities are complete prior to handover. (The
third category - deltas - by their definition should require no further integration prior to System Test,
and so in general these will bypass PIT.)
Development staff, SPTS staff, CM staff, and Testing staff all join forces within PIT as a multi-
disciplined team with specific product experience, to build and integrate the products, iteratively
refining the associated documentation/definitions until a proven configuration can be lodged within
CM from which SPTS are able to reliably build the required test environments, and with which
Testing are able to progress their planned tests independently.
© 1997 ICL Pathway COMMERCIAL IN CONFIDENCE
Page 13 of 41
FUJ00078295
FUJ00078295
ICL Pathway REVISIONS TO THE Ref: VI/STROO6
TESTING & INTEGRATION APPROACH Version: 2.0
FOR PATHWAY RELEASE 2 Date: 24/10/97
DEVELOPMENT INTEGRATION TESTING
PIT co sprs
establish build from replicate
baselines baselines environments
78
7.9
Dynix, I ———>
wr, I——>
Maestro, I
Patrol, > I ew
Riposte I I Shell
2 I
st
See
Appl. A I I Appl. A Tech
cs I
(in-house)I
Appl. B I I Appl.
CIs
Grd Paty] I =
Pint CL A
i3 I ——_—__I___y
no cong change
PintCL
789 I Change to contig
apply deltas to
baselines
i
BIT
eee on sag sone eam
ce
456
Technical Integration
Process Flow
(Note, this diagram is not intended to imply that CM are only involved/used after PIT. On the
contrary in fact. Development handover to PIT by lodging their products in CM, but not as a full
configuration, simply as a set of Cls, to help with version control. Then through the process of
integration within PIT the configuration information becomes progressively more firm and is
registered in CM accordingly, so that by the end of PIT the products are fully configured in CM and
ready to build from.)
PCMS
The existing CM facilities were only ever intended to support the early phases of Pathway
implementation and it is essential that for Release 2 we migrate to the target solution - PCMS. It is
proposed that the introduction of the Integration Unit be exploited further to act as the catalyst for
this migration and that the establishment of the initial baseline for Release 2 be used as the vehicle to
effect this migration. The later it is left the more difficult it will be to achieve, and so it is proposed to
take the hit from the very outset.
PIT - detail
One of the lessons learnt from Release 1 was that working from a false initial baseline, and
continually rebuilding test environments from the ground up (effectively individual hand stitching of
each one), served only to consume protracted periods of time. Inordinate levels of skilled resource
were required to make relatively little progress. Diagnosis of problems was difficult. Test results were
not always consistent.
Much greater emphasis is required in establishing a firmer baseline prior to independent testing,
which is signed up to by all parties concerned. Also, the precise configuration of the product CIs
making up that baseline is key to enable the reliable building of test environments from that baseline
and so avoid the hand stitching.
© 1997 ICL Pathway COMMERCIAL IN CONFIDENCE
Page 14 of 41
FUJ00078295
FUJ00078295
ICL Pathway REVISIONS TO THE Ref: VI/STROO6
TESTING & INTEGRATION APPROACH Version: 2.0
FOR PATHWAY RELEASE 2 Date: 24/10/97
There must be a separation of the component CIs to recognise a distinct ‘Shell’ upon which distinct
‘Applications’ can be added. This will enable sensible subsequent ‘Deltas’ of each type to be
managed effectively. (Shell and Application components tend to have distinct Deltas driven by
different cycles, and so need to be capable of separate management. For example, shell components
tend to be subject to regular maintenance cycles - service packs, etc., whilst Application products
tend to be subject to business development/enhancement cycles. Also, Applications tend to be the
same, and configured the same, irrespective of the test environment concerned (Tech / ST / BIT /
MOR), whilst the Shells tend to be different (progressively more extensive) for each.
As the products progress through PIT, the process is one of iteratively building, trialing, and so
refining the configuration information, to the point where it defines the product sufficiently
accurately and unambiguously that it can be used to reliably and consistently build test environments
from it.
Selected
Test Scripts
CM
Testing supporting Wo & PBs
Test Data Repository
Test Dat Standards eee Existing Outputs
Accompanying —_— _ overall
Testing Staff y—~ PBs Image(s) of
(Testing Expertise), Adjust Proven
Adjust Product System(s)
Product Test Shell Config Refine
Plans
‘Assemble Accurate H/O
a Config. NN a" PBS Stable Shell)
Prove in
LU Formal Raw Shell Install Isolat Documents
7 & solation Formal
Handover 7 . & Iteratively ronal
into CM Confgue Shell PI sone I prove with Re-Build Handover cls
Prove Neighbours Proven
/ \ PIT Build Scripts
Take Refine Proven PBS(s)
ed juild
Problem InstructionsI Pr
Logs oven
“ — Product Level
b © (post UTIL), —_— Regression
oS & Draft PBS Existing Test Pack(s)
‘Accompanying Test Build i
Development Staff Expertise Build Scripts
(Product Expertise)
Shell Cls SPTS
Development Ist Cut PIT
Build
Process Flow
Instructions
At first this configuration information will be held in ‘free format’, but progressively, as the
information becomes firmer, it will be translated into CM relationships, until ultimately the
environments can be built from the CM information alone. The final stage of PIT for any given set of
products will be to play the configuration out of CM and use it to build the target test environment for
handover.
Thus the ‘handover’ to CM at the end is actually a ‘promotion’ of that configuration within CM from
PIT to Testing.
© 1997 ICL Pathway COMMERCIAL IN CONFIDENCE
Page 15 of 41
FUJ00078295
FUJ00078295
ICL Pathway REVISIONS TO THE Ref: VI/STROO6
8.1
8.2
8.4
TESTING & INTEGRATION APPROACH Version: 2.0
FOR PATHWAY RELEASE 2 Date: 24/10/97
SYSTEM TEST
No specific changes proposed to the System Test approach. However, see Introduction section above,
and sections below on Joint Business Test, Test Automation, and Test Scripting.
The phased approach to test execution described in the strategy documents has been dubbed the ‘3
pass model’. It is proposed that this be formalised for Release 2:
© 1% Pass - stabilise
© Main Pass - iterative defect removal
¢ Final Pass - audit trail
The Main Pass is run iteratively, repeating the planned test cycle(s) several times in a progressively
more rigorous fashion. The last iteration is then by definition almost entirely a regression test in
nature, with all constituent tests having already been run in earlier iterations.
Please note that the Final Pass is not intended to ‘test’ anything, but rather to formally capture the
results of the Main Pass to form an audit trail. No other test activities are technically dependent on
completion of this except in regard to the resources employed by it. No new test objectives are being
satisfied other than that an audit trail be preserved.
The wording in the original strategy documents describes customer involvement in System Test as
being ‘progressively increasing’ and this has lead to confusion regarding the preparation phase. This
will be updated to reflect the agreed position. It refers to progressively increasing involvement of the
Customer in the test execution phase and does not imply no involvement in the preparation phase -
quite the contrary.
Guidance Notes and Checklists
In the absence of a fully active JBT (see section 10 below), the PDA and sponsor organisations have
often not received sufficient feedback on the management of the tests and their progression through
and between test stages. This has led to concern being expressed regarding the flexible structure of
tests in Pathway, with relatively few hard and fast entry and exit criteria being visible to give them
confidence.
It is recognised that in large and complex projects it is important to maintain this flexibility and not
to confuse “suitable status’ with hard planning dependencies. The progression through and between
test stages must remain a matter for local management discretion, conducting day by day and week by
week ongoing assessments of the status of the products under test and their suitability for progression
to further streams / stages, on a cost vs risk basis. (¢.g. In deciding whether or not to progress to the
next iteration within System Test Main Pass, or to enter BIT Main Pass, or whether ready to start the
Final Pass, the Pathway Test Manager must consider a number of parameters - stability of product,
known defects, availability of fixes, product interdependencies, resource conflicts, etc. - and weigh
these in the balance assessing the relative costs and risks of the options accordingly, and so decide
the most appropriate course of action under the circumstances.)
To help raise the visibility of this process, and to promote consistency across Test Managers, it is
proposed that the strategy documents for R2 include guidance notes on making these assessments,
and in particular a set of checklists to highlight the principal points for consideration by the Test
Managers concerned.
Whilst the decisions must be left to the Pathway Test Managers concerned, the process must
recognise the joint working approach., and input must be taken from the joint testing personnel
involved, seeking consensus with the PDA Test Manager.
The proposed checklists for System Test, for each system concerned, are as follows:
© 1997 ICL Pathway COMMERCIAL IN CONFIDENCE
Page 16 of 41
ICL Pathway
FUJ00078295
FUJ00078295
REVISIONS TO THE Ref: VI/STROO6
TESTING & INTEGRATION APPROACH Version: 2.0
FOR PATHWAY RELEASE 2 Date: 24/10/97
Entry to 1“ Pass (not applicable to R2)
¢ Environment configuration confirmed by PIT
© Configuration of particular Application concerned confirmed by PIT
« Subset of system test scripts selected and available
© These scripts exercise every major function in the Application
© These scripts exercise each major data stream employed by the Application
© Supporting test data prepared
Exit from 1* Pass (not applicable to R2)
« Environment supports Application and appears stable - any major defects highlighted
« Data supports scripts satisfactorily - any major defects highlighted
« Scripts operate satisfactorily - any major defects highlighted
Entry to Main Pass
« 1% Pass completed successfully
« Fixes received for major defects highlighted
© Scope and coverage agreed (HLTPs) ie. PDA Sign-Off
¢ Sufficient scripts ready
* stable and agreed
© broad front
« inter-dependencies
¢ Data prepared
¢ Environment(s) ready
Exit from Main Pass
e All candidate Applications covered - refer to agreed scope
e All System Test scripts run - refer to agreed scope
¢ All incidents required to be corrected in System Test fixed and re-tested
© All other outstanding incidents agreed (with Integration Test Manager) as OK to pass on
into BIT
* Believe now ready to run from beginning to end uninterrupted
Entry to Final Pass
© 1997 ICL Pathway
COMMERCIAL IN CONFIDENCE
Page 17 of 41
FUJ00078295
FUJ00078295
ICL Pathway REVISIONS TO THE Ref: VI/STROO6
TESTING & INTEGRATION APPROACH Version: 2.0
FOR PATHWAY RELEASE 2 Date: 24/10/97
¢ Audit requirements of this phase understood
¢ Environment ready and audit switches, etc. activated accordingly
¢ Candidate scripts for audit selected (usually all)
Exit from Final Pass
¢ All incidents required to be corrected in System Test closed
« System Test Report ready
e All candidate scripts (usually all) run from beginning to end without interruption (i.e.
without needing to take code deliveries to resolve problems).
¢ All outstanding incidents agreed (with Integration Test Manager) as OK to pass on into
BIT
¢ All audit trail materials secured
- scripts, data, configuration, results
© 1997 ICL Pathway COMMERCIAL IN CONFIDENCE
Page 18 of 41
FUJ00078295
FUJ00078295
ICL Pathway REVISIONS TO THE Ref: VI/STROO6
TESTING & INTEGRATION APPROACH Version: 2.0
FOR PATHWAY RELEASE 2 Date: 24/10/97
91
9.2
INTEGRATION TEST
Integration Test covers a wide range of complementary test streams, run by a number of co-operating
test teams. Over time as these test streams were planned out, they were given names to distinguish
between the activities in plans, reports, etc. It is important to note that the underlying objectives
remained unchanged. These test streams have become known as:
¢ DIT - Direct Interface Test
¢ T&S - Technical & Security Test
« BIT - Business Integration Test
° E2E - End to End Interface Test
« MOR - Model Office Rehearsal
In general however, the complementary relationship between these various test streams has become
confused. There has been a tendency to treat each as a separate test stage, and attempts made to set
entry and exit criteria for each and between each.
Notably the MOR has changed over time to be regarded as a preliminary part of Model Office Test,
rather than as an integral part of Integration Test which prepares the way for Model Office Test.
Certainly it plays a strong role within Joint Business Test, and PDA testing (including POCL counter
staff) play a major role in it, but it must be emphasised that it remains a Pathway test activity with
specific Pathway test objectives.
Planning guidelines which resulted from this confusion curtailed BIT and parts of T&S
unnecessarily. The consequences here were twofold. Firstly it created time pressures which effectively
capped the BIT activity, preventing all objectives from being met and reducing the overall
effectiveness. Secondly and in many ways more importantly, without the BIT activity running
alongside MOR, there is no effective wide-bandwidth ongoing regression testing vehicle operating
formally. MOR simply cannot hope to achieve this - it does not have the necessary content.
We fell foul of this both in Release 1B and IC. It is essential that this drift be addressed for Release 2
if we are to avoid repeat performances of the product degradation at the end of the test lifecycle. For
1B the go-live was delayed to allow an additional regression test cycle to compensate for this. At 1C
BIT-like regression testing activity has been conducted informally alongside MOR to substitute for
BIT itself no longer running and so safeguard the final rehearsal. For Release 2 both BIT and T&S
must be scheduled to run throughout the rehearsal period as originally intended, allowing the streams
to properly interact and complement each other and to maintain a thorough and formal regression
testing activity protecting not just the rehearsal, but also the MOT and ultimately the live service.
As Integration Test progresses there is an increasing need to consolidate and co-ordinate baselines
across the various test streams, culminating in MOT. The existing Final Pass in BIT, and the
proposed Acceptance Trial activities in both BIT and T&S, form natural break points within the
period. (See illustration at 9.8 below for further explanation.)
Objectives and Approach
The overall objectives and approach for Integration Test remain as outlined in the Testing and
Integration Strategy [2] and the System Integration Test Strategy [6], with one exception - the area of
Configuration where much of the emphasis is now placed on PIT (see section above) to pro-actively
establish a firm configuration. The areas of Operability, Performance, Security, Resilience, Data
Integrity, Usability, Architectural Conformance, External Interfaces, Pre-proving the MOT, and
forming a comprehensive Regression Test vehicle all remain unchanged.
© 1997 ICL Pathway COMMERCIAL IN CONFIDENCE
Page 19 of 41
FUJ00078295
FUJ00078295
ICL Pathway REVISIONS TO THE Ref: VI/STROO6
TESTING & INTEGRATION APPROACH Version: 2.0
FOR PATHWAY RELEASE 2 Date: 24/10/97
9.3 DIT
In both 1B and 1C we have found it possible to conduct DIT earlier than originally envisaged in the
strategy, and this is seen by all parties concerned as an advantage. It is proposed that this should now
be formally adopted, commencing DIT shortly after the start of System Test, and well ahead of
Business Integration Test.
As previously planned for Release 1E testing, this will be run in two main parts. Part I will gradually
introduce the interfaces, one at a time, in the order of their inter-dependence (e.g. Reference Data
first), concentrating on getting the interfaces up and running, and flushing out the major defects. Part
2 will then run all the interfaces alongside each other within Pathway, and complete the defect
removal cycle for each.
It has also been generally accepted that DIT should be formally specified between the parties
concerned using an Interface Test Specification, agreed and signed accordingly. It is proposed that
this approach should also be formally adopted. Ideally a standard template should be used to promote
consistency across the various interfaces, and it is proposed that one should be developed, though it is
recognised that this is a little late for general use in R2
Similarly it has been generally accepted that DIT should be successfully concluded before attempting
to run E2E. This dependency will be formally recognised.
In practice the parties concerned have tended to get together and pre-trial their interfaces. This has
been dubbed Pre-DIT. It is generally thought to be a good thing, but it has been done rather
informally so far, and in some cases not at all. POCL would like to see the practice formally adopted.
This means listing the associated objectives.
It is proposed that a Pre-DIT test stream be introduced, immediately preceding each DIT test stream,
with the following specific objectives/activities:
Bench-check each item on the interface specification against own system’s outputs and
expected inputs
« Exchange fabricated interface files, checking each other’s against the interface
specification to confirm each other’s interpretation of that specification
« Exchange system generated interface files (warts and all, and manually if necessary) and
again check each other’s against the interface specification.
(Note, it is not proposed that these activities must ‘pass’ as a pre-requisite of entry to DIT, but the
extent of any defects would be assessed by the parties concerned to determine whether or not it would
be beneficial to proceed.)
From the outset it has been an assumption within the strategies and plans (and implicit in the
contract) that a pre-requisite of entry to DIT is the freezing of each participating external system
beforehand. The Pathway system has to reconcile the multitude of interfaces with BA/POCL systems
within a limited window of opportunity, and this cannot be achieved efficiently whilst the target is
still moving.
In practice this has not taken place and has been detrimental. In Release 1B it severely compromised
Pathway’s own commitment to freeze the Pathway system prior to the Model Office Test, which in
turn complicated the counter roll-out process. It is proposed that Pathway now make this a firm
dependency within the programme plans and require that PDA adhere to it.
It is suggested that the relevant parties for each interface meet and agree the specifics. This would
involve agreeing the code boundaries to be subject to such external change control. This may be done
using level 2 DFDs or their equivalent. It would also involve agreeing appropriate procedures for
notifying any such changes.
94 T&S
© 1997 ICL Pathway COMMERCIAL IN CONFIDENCE
Page 20 of 41
FUJ00078295
FUJ00078295
ICL Pathway REVISIONS TO THE Ref: VI/STROO6
TESTING & INTEGRATION APPROACH Version: 2.0
FOR PATHWAY RELEASE 2 Date: 24/10/97
This area has operated as 4 main teams for Release 1, and these teams have become part of the
terminology in common use. It is proposed that these terms be formally adopted:
© Security Test
© Performance Test
¢ Integrity Test
¢ Systems Management Test
The first two are self explanatory. Integrity Test covers all the issues related to resilience, recovery,
and integrity. Systems Management Test requires further explanation.
In Release I we carried out specific product tests for certain Systems Management components, as a
comfort exercise, despite the fact that actually it is to be delivered as a managed service. This has
generated confusion about the status of System Management testing both within Pathway and the
PDA. For Release 2 we will move much more strongly toward treating it as a managed service (using
it rather than testing it directly). In order that further confusion can be avoided it is proposed that a
specific strategy paper be written detailing this approach.
Technical and Security Test has suffered across the board from the test areas achieving a late and
sometimes incomplete understanding of the Non-Functional Requirements (NFRs) they need to test.
This was the result of difficulty in assembling all the necessary data scattered across all the various
contractual and system documentation. It was further compounded, in the Security Test area by late
resolution of agreements to agree, and in the Performance and Resilience areas by similar difficulties
in establishing the detailed targets and procedures.
This in turn has also lead to confusion within the PDA regarding the coverage and completeness of
Pathway’s testing in these key areas.
It is proposed that at the outset for Release 2 a comprehensive NFR catalogue be compiled, and that
test material is then derived from, cross-referenced to, and checked for completeness against this key
input, and that delivery is entered as a key dependency on the programme plan.
It is further proposed that at the outset of Release 2 a comprehensive set of system performance
targets be established, to act in a similar manner to the NFR catalogue, and again that delivery be
entered on the programme plan as a key dependency.
Since the strategy documents were agreed further source documents emerged for the Security Test
area, and it is proposed that these now be formally adopted in our process:
© Access Control Policy
¢ Security Functional Specification.
The proposals for merging Acceptance Trials with the mainstream Pathway tests involve T&S for all
Acceptance Conditions based upon T&S material. (See section 18 below.)
BIT
The phased approach to test execution described in the strategy documents has been dubbed the ‘3
pass model’. It is proposed that this be formalised for Release 2:
¢ 1“ Pass - stabilise
¢ Main Pass - iterative defect removal
¢ Final Pass - audit trail
Please see description of 3 pass model under section 8.2 above.
The proposed changes regarding the relationship between MOR and BIT have already been made
© 1997 ICL Pathway COMMERCIAL IN CONFIDENCE
Page 21 of 41
FUJ00078295
FUJ00078295
ICL Pathway REVISIONS TO THE Ref: VI/STROO6
TESTING & INTEGRATION APPROACH Version: 2.0
FOR PATHWAY RELEASE 2 Date: 24/10/97
(see 9.1 above).
At IC it was agreed that Pathway would improve the review quality of the Counter Procedures prior
to use in model office by conducting an active review of the draft material within BIT. Although it
was not possible to perform this activity extensively at 1C, the little which was performed proved to
be beneficial and avoided the kind of logic defects encountered in the procedures at 1B reaching the
model office at 1C.
It is therefore proposed that this practice be formalised, building an active review of the Counter
Procedures into the early part of the Main Pass of BIT (i.e. during the course of BIT Main Pass, prior
to the MOR, each Counter Procedure will be reviewed alongside the actual use of the related
functionality on the test rigs.) It should be recognised that this creates a dependency between BIT
Main Pass and the availability of draft Counter Procedures.
The BIT activity is crucial in Release 2, and the allowed duration must be sufficient. Removal of the
restrictions for overlap with MOR should enable this.
In Release 1C the test planning and scripting for BIT ran into some difficulties. The System Test
material took a long time in the review phase (see section 10 below) undergoing considerable rework,
and as a result it was not easy to roll this material on into the BIT material. Problems in finalising
the NFRs, performance targets, resilience and recovery procedures and security requirements (see 9.4
above), led to delays in establishing the necessary Maestro schedules and impacted the BIT material
also. Finally, it proved harder than originally envisaged to align the MIS scripts with the other
System Test scripts (dates / cases / etc.). As a result it was not possible to build all the System Test
material in as an integral part of the BIT material. This reduced the regression testing effectiveness
of BIT and forced additional separate streams of testing to be run for PinICL clearance purposes
At IC this was manageable. For R2It is important that the strategic approach relating to BIT test
planning be embraced, and that all System Test material be subsumed within the BIT materials. The
proposals at section 10 below and at 9.4 above should enable this.
Originally the external interface tests were envisaged to run as part of this area of testing, but we
have found that they can be run much earlier, to everybody’s advantage. (See section on DIT above.)
It therefore became the practice not to include external interface activity within BIT. In certain
circumstances this has been seen as a disadvantage. Where fixes are applied after DIT is completed,
which may indirectly impact the interface systems, any potential knock-on defects introduced are not
currently discovered until E2E, during the MOR. Whilst this is not catastrophic, it can be a little
disruptive.
It is proposed that all DIT material is maintained as regression test packs for each interface area, and
that during BIT, until the start of E2E, they be run regularly to detect any potential regression of the
interfaces concerned. It is also proposed that the BIT environment(s) be enabled to exercise all the
external interfaces, and that one of the scripted days within BIT be selected to be run optionally as an
“Open Day’. (i.e. that day can be run normally, without external interaction, or by prior arrangement
with all external interfaces opened up.) The Open Day would be scheduled prior to MOR. It would
test the interfaces with the full infrastructure and forestall any related problems, thus protecting the
MOR. It is recognised that this would involve all the external systems in preparing suitable files for
transmission to satisfy the BIT scripts for that day. This would need to be agreed in advance and it is
proposed that Interface Test specifications as already used for DIT could be used to define the
requirements.
The proposals for merging Acceptance Trials with the mainstream Pathway tests involve BIT Main
Pass (as a rehearsal activity) and BIT Final Pass (as the Acceptance Trials) for all Acceptance
Conditions based upon System Test or DIT material. (See section 18 below.)
9.6 EE
The overall objectives for E2E and MOR combined remain unchanged, and centre on the bringing
© 1997 ICL Pathway COMMERCIAL IN CONFIDENCE
Page 22 of 41
FUJ00078295
FUJ00078295
ICL Pathway REVISIONS TO THE Ref: VI/STROO6
TESTING & INTEGRATION APPROACH Version: 2.0
FOR PATHWAY RELEASE 2 Date: 24/10/97
9.7
together of the IT functionality across all the participating systems, with the operational and business
procedures, to form an overall business system. There has always been a close relationship between
the E2E and MOR activities, being planned and scripted together, run on the same environment, and
involving the same personnel. This has been the subject of much discussion recently and a broad
consensus has been reached.
In general terms E2E will start somewhat earlier than the Model Office. It is likely to follow straight
on from DIT, and form a natural extension of it. It will continue to run in parallel with but separate
from MOR. The nature of the relationship between E2E and MOR is described in the section on
MOR below. However, the fine detail regarding the precise scope and coverage of the two sets of
tests, the division of the low level objectives between the two, the environment requirements,
resourcing, and mechanics involved are still being developed jointly. These once agreed will be
incorporated into the revised strategy documents accordingly.
In addition to the actual testing of the procedures within E2E/MOR/MOT there is general consensus
that all would benefit from the introduction of a new activity - a formal walk-through of the end to
end business and operational procedures. This would be driven by PDA. Whilst predominantly a
POCL activity, the end to end nature will require the support and participation of all parties. It will
be scheduled prior to E2E Main Pass and MOR2.
(See also section on DIT above.)
MOR
The overall objectives for E2E and MOR combined remain unchanged.
The proposed changes regarding the relationship between MOR and the BIT and T&S test streams
within Integration Test have already been made (see 9.1 and 9.5 above). These are seen as critical to
the success of R2 testing.
The relationship between MOR and E2E has been the subject of much debate recently (see also
section on E2E above), and the following describes the broad consensus reached.
The aims of these two test streams are to bring the IT system together with the business procedures,
to trial the training activities and to deploy the target live support channels, running all the
contributing systems in unison and exercising all major end to end business flows. In effect to trial
the overall business system.
In the past E2E has been subsumed into MOR and MOT, and it had been proposed to formally
combine the two under a new name. However, it is now recognised that as the overall business system.
grows in scope and complexity, certain of the tests required to be covered, if combined, force
excessively long test cycle lengths, When this is coupled with the essential ‘real time’ nature of the
rehearsal activity, it really does become impractical. It is generally agreed that the two should now be
de-coupled. (It has in fact long been the belief of the PDA Test area that the MOT was carrying too
much baggage, and moves have previously been made to separately identify the two sets of tests.)
This is further supported by moves within CAPS to return to a more live-like type of Model Office,
employing substantial live (or live based) data, and capturing live events (e.g. office post) to form the
basis of much of the testing within the MOT (see section on MOT below). Again, whilst this
enhances many of the procedurally oriented tests, it clashes with some of the complex scenarios
needed to demonstrate lengthy end to end data flows which really need to be specifically engineered.
So, in general MOT (and so MOR) will run in a more live-like fashion (less rigid but just as
rigorous), using a combination of captured live scenarios and simple contrived scenarios, scripted in
logical terms to allow greater flexibility in running, but running essentially in ‘real time’, and with
the emphasis very much on trialing the vast majority of the business and operational procedures.
Because of the ‘real time’ nature of the tests a short test cycle is desirable. For R2 it is envisaged that
this would be in the order of 2 to 3 weeks. Every effort must be made to contain this.
© 1997 ICL Pathway COMMERCIAL IN CONFIDENCE
Page 23 of 41
FUJ00078295
FUJ00078295
ICL Pathway REVISIONS TO THE Ref: VI/STROO6
TESTING & INTEGRATION APPROACH Version: 2.0
FOR PATHWAY RELEASE 2 Date: 24/10/97
E2E on the other hand will operate in a similar fashion to DIT, in highly controlled small scale
environments, driven exclusively by contrived scenarios, and scripted in detail. It will pick up all
those complex end to end scenarios which often require lengthy test cycle lengths and detailed
expected results, which would otherwise clash with the MOT / MOR. (An example would be
demonstrating the end to end integrity of the accounting and reconciliation cycle, including end of
year processing.) Because of the nature of the scenarios, lengthy test cycle(s) are likely. For R2 it is
envisaged that this would be in the order of 5 to 7 weeks. However, the nature of the runtime
environments should allow this to be contained in considerably less elapsed time, perhaps 3 to 4
weeks.
Both E2E and MOR / MOT will be run in the pattern of the 3 pass model (see 9.5 for an
explanation), and in parallel, though not necessarily directly aligned. Until the actual cycle lengths
are determined and the detailed schedules are drawn up, the precise alignment will not be known.
However, it is likely to look something like:
E2E - Main Pass E2E - Final Pass
MOR - Ist Pass. MOR - Main Pass MOT
The Ist and Main passes of both E2E and MOR are recognised as Pathway owned test phases,
an integral part of the Integration Test stage, operating within the Joint Testing agreement.
MOT remains an independently owned PDA test stage.
The Final Pass of E2E is proposed likewise to be an independently owned PDA test stage.
The start of E:
3 Final Pass would be aligned with that of MOT to facilitate baseline control.
There are a number of practical considerations which remain to be resolved in detail
- the runtime environment for E2E is by necessity very different to that of MOT
The essential point is that the two activities are operated and managed separately and are
aimed at separate clearly defined objectives
Please note: Until the detail for E2E is driven out it is not possible to be definite, but, it seems likely
that all of the objectives of the E2E - 1“ Pass will in fact be satisfied by Part 2 of DIT, since the same
environments and baselines will be in use, and if planned carefully the data and scenarios will be
shared across the two, so that DIT will already have stabilised E2E. It is therefore assumed here that
E2E - 1* Pass will not be required as a separate activity and that it will be subsumed within DIT Part
2
Whilst the MOT is an independent test stage owned by the PDA, and so the scope and coverage of
the test material is driven by the PDA, it is important that Pathway be closely involved in the
preparation as it will form the material used in the MOR. In R1 Pathway has been involved in the
high level planning workshops for the MOT, and has reviewed the scripts produced, but has not gone
further. For R2 it is proposed that a much more active role be played by Pathway throughout the
preparation phase.
In Release I Pathway has been criticised for not providing sufficient status information, such as a
© 1997 ICL Pathway COMMERCIAL IN CONFIDENCE
Page 24 of 41
FUJ00078295
FUJ00078295
ICL Pathway REVISIONS TO THE Ref: VI/STROO6
TESTING & INTEGRATION APPROACH Version: 2.0
FOR PATHWAY RELEASE 2 Date: 24/10/97
98
Release Notice. For R2 it is proposed that PDA and Pathway work jointly to define a suitable level of
status information to address the problem.
Similarly in R1 Pathway has been criticised regarding the operation of the help desks - specifically
that they have not always taken the MOR seriously. Whilst it is recognised that it is often difficult
within a live support area to treat a test situation as if live, the role-play forms an important part of
the test in MOR and MOT. The operation of the help desks and their procedures are under test,
These problems have already been addressed during 1C, where there was increased Customer Service
involvement in the running of the final rehearsal, and where they will be responsible for operating
the MOT entirely from the live support area with minimal support from pathway testers. Ths practice
will be formally adopted for R2.
Baseline Co-ordination
The changes proposed in this document do not increase the baseline co-ordination required to support
the various testing activities, but there has been much confusion expressed in this area, and so the
test strategies concerned should include more specific details on how this is to be achieved. In
particular this should include a description of how the code baselines employed in the various
streams of testing gradually converge as Pathway testing draws toward a close, and how this process
allows the complementary test streams running in parallel at this time to work in unison rather than
tripping each other up.
(see schematic on following page)
© 1997 ICL Pathway COMMERCIAL IN CONFIDENCE
Page 25 of 41
ICL Pathway REVISIONS TO THE Ref:
TESTING & INTEGRATION APPROACH Version:
FOR PATHWAY RELEASE 2 Date:
VI/STRO06
2.0
24/10/97
BASELINE CO-ORDINATION
T&S
H t [regression etme
H : TA
BIT - ist A 4 '
l BIT = Main Pass i H
t \ [_—Rehearse Acep. Tals BIT - Final
: ‘ cop. Trials
Dit -Part 1
I EE - Main Pass
! f ' EE = Tina
' [ovorT] t H
I MOR I
1 : MOT
ipplication, infrastructure, and # good baseline ' baseline * east preliminary testing essentially
nterfaces all raesonably stable restablished for 4 very sound and 1 MOT baseline I completed
' ‘Propogation stable, coming to + once BIT Main I
I able to progress serious integration {across all areas I close of BIT I Pass completed I cast preliminary
I and interface testing and support full I . I main pass and. . 1 Live baseline
ndwith of technical and security {into regression I ready to support I Tun Acep. Trials for use in MOT
ests ‘test periodof + MOR2 H
wy defect removal period teyele
1 Acep. Trials
1 defect removal
{start to resist any
! changes of a non-
table to rehearse I critical nature
© 1997 ICL Pathway COMMERCIAL IN CONFIDENCE
Page 0 of 41
! strongly resist all
{ changes of a non-
‘critical nature
{ confirm baseline
! propogated fully
' start building
' Live counters
{ strongly resist
‘all changes
FUJ00078295
FUJ00078295
ICL Pathway
FUJ00078295
FUJ00078295
REVISIONS TO THE Ref: VI/STROO6
TESTING & INTEGRATION APPROACH Version: 2.0
FOR PATHWAY RELEASE 2 Date: 24/10/97
99 Guidance Notes and Checklists
(see description under section 8.4 above)
The proposed checklists for Integration Test are as follows:
Entry to Pre-DIT, for a given interface
¢ Interface Specifications (AIS & TIS) produced and agreed
¢ Pre-DIT plans produced and agreed (Scope & Coverage)
Entry to DIT part 1, for a given interface
© Pre-DIT completed satisfactorily for given interface
© Established System Test environment (or better), with external interface(s) configured,
ready for use
¢ External interface system(s) (e.g. CAS, etc.) installed and ready for use
External System declared ‘frozen’ and placed under external change control (see section
9.3 above for explanation)
* Pathway code baseline for given vertical application (e.g. BPS) and for key areas of
functionality (e.g. Cash-Account Mappings in relation to TIP interface) sufficiently
stable to support interface concerned (¢.g. stabilised during initial System Test or in a
regression test)
© Interface Test Specification for given interface produced and agreed between parties
© Supporting test data available
+ Application inter-dependencies satisfied (e.g. Reference Data interface would precede
HAPS interface which would precede Interim TIP interface)
Entry to DIT part 2
© DIT part 1 completed satisfactorily for each participating interface, with each operating
in a stable fashion
© ‘Stopper’ incidents raised in part I resolved, and targets for remaining incidents
obtained
¢ Full application and infrastructure set available and stable (e.g. completed BIT 1* Pass)
Exit from DIT
* Planned testing completed for all participating interfaces to mutual satisfaction (e.g. sign
off achieved)
Entry to T&S, for each major test group/product set
© Scope and coverage agreed (HLTPs) - ie, PSA Sign-Off
© Sufficient scripts ready
© 1997 ICL Pathway
COMMERCIAL IN CONFIDENCE
Page 0 of 41
ICL Pathway
FUJ00078295
FUJ00078295
REVISIONS TO THE Ref: VI/STROO6
TESTING & INTEGRATION APPROACH Version: 2.0
FOR PATHWAY RELEASE 2 Date: 24/10/97
© stable / agreed
© broad front
¢ inter-dependencies
¢ Data prepared
¢ Environment(s) ready
¢ Resilience and Recovery Procedures available, where appropriate
¢ Performance Budgets available, where appropriate
¢ NER Catalogue available, where appropriate
Exit from T&S
* All T&S scripts run (refer to agreed scope)
« All incidents required to be corrected for initial live running fixed and re-tested
e All other outstanding incidents agreed (with PDA Test Manager) as OK to remain at this
release (i.e. form part of KPR)
© T&S Test Report(s) ready
Entry to Main Pass E2E
e Walk-through of end to end procedures completed satisfactorily
e Environment configuration confirmed by DIT part 2
* Configuration of interfaces confirmed by DIT part 2
* Plans and scripts produced and agreed (i.e. scope and coverage agreed)
e Timetable, mutual support arrangements and rules of engagement produced and agreed
Supporting test data prepared
Fixes received for major defects highlighted in DIT
Exit from Main Pass E2E
e All planned scripts run - refer to agreed scope
¢ All incidents required to be corrected for initial live running fixed and re-tested
¢ All other outstanding incidents agreed (with PDA Test Manager) as OK to remain at this
release (i.e. form part of the KPR)
* Believe now ready to run from beginning to end uninterrupted
© 1997 ICL Pathway
COMMERCIAL IN CONFIDENCE
Page I of 41
ICL Pathway
REVISIONS TO THE Ref:
TESTING & INTEGRATION APPROACH Version:
FOR PATHWAY RELEASE 2 Date:
Entry to 1“ Pass BIT
¢ Environment configuration confirmed by PIT
© Configuration of Applications and infrastructure confirmed by PIT
* Maestro schedules available for use
¢ Subset of BIT and System Test scripts selected and available
¢ These scripts exercise every major function in the Business System
© These scripts exercise each major data stream
Supporting test data prepared
Exit from 1“ Pass BIT
« Environment appears stable - any major defects highlighted
« Data supports scripts satisfactorily - any major defects highlighted
© Scripts operate satisfactorily - any major defects highlighted
Entry to Main Pass BIT
« 1 Pass completed successfully
¢ Fixes received for major defects highlighted
¢ Scope and coverage agreed (HLTPs) - i.e. PSA Sign-Off
System Test material properly and fully inherited
© Sufficient scripts ready
© stable / agreed
© broad front
© inter-dependencies
¢ Data prepared
Environment(s) ready
¢ Counter Procedures available
* Migration Procedures available
¢ Resilience and Recovery Procedures available
Exit from Main Pass BIT
« AIIBIT scripts run - refer to agreed scope
FUJ00078295
FUJ00078295
VI/STROO6
2.0
24/10/97
© 1997 ICL Pathway
COMMERCIAL IN CONFIDENCE
Page 2 of 41
ICL Pathway
FUJ00078295
FUJ00078295
REVISIONS TO THE Ref: VI/STROO6
TESTING & INTEGRATION APPROACH Version: 2.0
FOR PATHWAY RELEASE 2 Date: 24/10/97
¢ All incidents required to be corrected in BIT fixed and re-tested (including any passed
through to BIT from earlier stages)
¢ All other outstanding incidents agreed (with Integration Test Manager / PDA Test
Manager) as OK to pass on to either E2E or Model Office
© Believe now ready to run from beginning to end uninterrupted
Entry to Final Pass BIT
¢ Audit requirements of this phase understood
¢ Environment ready and audit switches, etc. activated accordingly
¢ Candidate scripts for audit selected (usually all)
Exit from Final Pass BIT
e All incidents required to be corrected in BIT closed
« BIT Test Report ready
e All candidate scripts (usually all) run from beginning to end without interruption (i.e.
without needing to take code deliveries to resolve problems).
« All outstanding incidents agreed (with Integration Test Manager / PDA Test Manager)
as OK to pass on
Entry to MORI
e Environment built and ready for use
© Software installed and baseline status confirmed by CM audit
e Plans and scripts produced and agreed (i.e. scope and coverage agreed)
© Draft procedures (all areas) produced, reviewed, corrected, and ready for use (e.g. for
Counter Procedures active review in BIT)
Supporting data available
« MORI candidate scripts (subset) selected
« These scripts exercise each interface area, each major data-flow, and each principal
function at the outlet, and so exercise the most used procedures
« Timetable, mutual support arrangements and rules of engagement produced and agreed
DIT completed successfully
Exit from MORI
Environment appears stable - any major defects highlighted
© Data supports scripts satisfactorily - any major defects highlighted
© 1997 ICL Pathway
COMMERCIAL IN CONFIDENCE
Page 3 of 41
ICL Pathway
FUJ00078295
FUJ00078295
REVISIONS TO THE Ref: VI/STROO6
TESTING & INTEGRATION APPROACH Version: 2.0
FOR PATHWAY RELEASE 2 Date: 24/10/97
© Scripts operate satisfactorily - any major defects highlighted
¢ Procedures appear satisfactory for serious use - any major defects highlighted
Entry to MOR2
© Environment reset and ready for use
© MORI completed successfully
Training material produced and ready for use
© PDA confirm availability of representative end users to participate in MOR2
« Training sessions conducted with candidate users
«Fixes received for major defects highlighted
¢ Progress within E2E, BIT and T&S sufficient to ensure relatively stable code baseline for
MOR? running. (Majority of tests already run, no outstanding ‘stoppers’ likely to impact
MOR2, other outstanding incidents reviewed and position known and deemed unlikely to
seriously impact MOR2)
Exit from MOR2
e All planned scripts run - refer to agreed scope
« All incidents required to be corrected for initial live running fixed and re-tested
¢ All other outstanding incidents agreed (with PDA Test Manager) as OK to remain at this
release (i.e. form part of the KPR)
« Believe now ready to run from beginning to end uninterrupted
© 1997 ICL Pathway
COMMERCIAL IN CONFIDENCE
Page 4 of 41
FUJ00078295
FUJ00078295
ICL Pathway REVISIONS TO THE Ref: VI/STROO6
TESTING & INTEGRATION APPROACH Version: 2.0
FOR PATHWAY RELEASE 2 Date: 24/10/97
10. JOINT BUSINESS TEST
10.1 When JBT was first introduced in October 1996, it was agreed at the PDA board, and was on the
basis that PDA would install a sufficient number of appropriately skilled BA/POCL/PDA staff, drawn
from all contributing areas, and operating as part of the Pathway managed test teams. The principal
activities were to actively participate in the creation of HLTPs, contributing their business experience
and acting as the agents of PDA to ensure the scope and coverage of testing was acceptable, and to
actively participate in the test execution, on a progressively increasing basis, as the products
progressed through System Test and Integration Test. The level of involvement was indicated as
about 20%, amounting to 10 individuals at that time.
10.1 In practice the implementation has been variable. As Pathway numbers grew the PDA presence as a
proportion fell. This coupled with conflicting work schedules and priorities often meant that very
little active participation was possible, and so Pathway was not able to successfully integrate the PDA
personnel into the test teams as originally planned. Under these circumstances the only option often
remaining was for the PDA to revert to retrospective reviews. This had obvious timescale and rework
impacts, and did little to promote closer working relationships. Without active business input in the
formative test preparation phases, it is likely that the quality of the test material suffered as a result.
10.1 It is proposed that JBT resource levels are raised to allow full and active participation throughout. It
is important that the resources are drawn from across all interested parties to raise the visibility for
the sponsor organisations. A closer working between the PDA Assurance and PDA Testing areas is
essential to achieve appropriate product knowledge levels and again to raise visibility within the
PDA.
10.1 This matter has been discussed widely and terms of reference for Joint Testing are being developed.
© 1997 ICL Pathway COMMERCIAL IN CONFIDENCE
Page 5 of 41
FUJ00078295
FUJ00078295
ICL Pathway REVISIONS TO THE Ref: VI/STROO6
TESTING & INTEGRATION APPROACH Version: 2.0
FOR PATHWAY RELEASE 2 Date: 24/10/97
1. MODEL OFFICE TEST
lL Over time MOT has taken on the aspect of a mini System Test embracing all the participating
systems. This is unfortunate as it does not really meet the original objectives of what was then termed
the Mock Office phase - being very Live-like.
11.2. CAPS recently tabled proposals for change in this area which have now been discussed and a general
consensus has been reached to adopt their principles of capturing live events and for the most part
using these to drive the tests. This has already been described in more detail under the section on
MOR above, together with a separation of E2E and MOT test streams to avoid lengthy E2E test
cycles having to be run in ‘real time’.
11.3. Itis proposed that this approach be formally adopted for R2.
© 1997 ICL Pathway COMMERCIAL IN CONFIDENCE
Page 6 of 41
FUJ00078295
FUJ00078295
ICL Pathway REVISIONS TO THE Ref: VI/STRO06
TESTING & INTEGRATION APPROACH Version: 2.0
FOR PATHWAY RELEASE 2 Date: 24/10/97
12. LIVE TRIAL
12.1 No changes are proposed in this area. It is however probably worth re-emphasising that the Live
Trial is a stage of testing, and that there are certain aspects of the service which it will not be possible
to demonstrate prior to actual live operation. Hence the contractual acceptance of the service cannot
be concluded until the Live Trial has been completed.
© 1997 ICL Pathway COMMERCIAL IN CONFIDENCE
Page 7 of 41
FUJ00078295
FUJ00078295
ICL Pathway REVISIONS TO THE Ref: VI/STROO6
TESTING & INTEGRATION APPROACH Version: 2.0
FOR PATHWAY RELEASE 2 Date: 24/10/97
13. MAINTENANCE & REGRESSION TESTING
13.1 The objectives of this area remain unchanged.
13.2. As aresult of the planning guidelines for R1 which restricted the operation of BIT, the formal
regression testing performed was patchy during the MOR period. This problem is described in the
section on Integration Test above. Whilst informal measures have been taken to compensate for this,
revisions are required for R2. The revisions recommended in 9.1 above will in large measure correct
the problem.
13.3 However, Pathway should not be complacent in this respect. As the range of active functionality
increases significantly with Release 2, so the complexity of system interactions grows, and with it the
potential impact of knock-on effects from fixes.
13.4 It is proposed that Pathway reviews in detail the processes in this area and strengthens them
appropriately. Most important here is to ensure that the BIT test material is carefully constructed to
include all preceding test material from System Test, and arranged in a fashion conducive to carrying
out both selective and blanket regression tests.
13.5 In Release 1 all regression tests were manual and so were expensive and prone to human error. For
Release 2 it is proposed to progressively introduce test automation facilities to counter this and enable
more regular and more reliable regression tests to be run. (See section on Test Automation below.)
© 1997 ICL Pathway COMMERCIAL IN CONFIDENCE
Page 8 of 41
FUJ00078295
FUJ00078295
ICL Pathway REVISIONS TO THE Ref: VI/STROO6
TESTING & INTEGRATION APPROACH Version: 2.0
FOR PATHWAY RELEASE 2 Date: 24/10/97
14, EXTERNAL CERTIFICATION
14.1 No changes are proposed in this area. The vast majority of external certification required for the
Pathway solution will already have taken place for Release 1. The few remaining areas should
continue as currently planned.
© 1997 ICL Pathway COMMERCIAL IN CONFIDENCE
Page 9 of 41
FUJ00078295
FUJ00078295
ICL Pathway REVISIONS TO THE Ref: VI/STROO6
TESTING & INTEGRATION APPROACH Version: 2.0
FOR PATHWAY RELEASE 2 Date: 24/10/97
15. TEST ENVIRONMENTS
15.1 General.
The reliable and consistent replication of test environments has been a nightmare throughout Release
1. Whilst there have been huge improvements made already, this area has remained a bottleneck, and
has also resulted in enormous levels of rework.
The introduction of the Integration unit is designed to significantly improve our ability here. The new
activity - PIT, and the move to PCMS within CM are the cornerstones. (See section above on PIT.)
15.2 Model Office.
It is proposed that the implementation plan for Release 2 be adjusted to launch the release initially on
a single data centre (Bootle) so that the Model Office Environment may once more be mounted on
the target live equipment. This allows the existing Release 1 system to remain untouched on the other
data centre (Wigan) until we are satisfied that the implementation has been successful. Thus we have
a non-regressive implementation path at the data centres. It also allows the target live system to be
built and proven in situ at the new data centre, going through several months of testing prior to live
use, and so avoiding any possible difficulties in transferring the configuration faithfully at the last
minute. This effectively mitigates any residual CM risks arising from conversion of the Release 1
configuration at Wigan.
It is further proposed that this Model Office Environment be built and used earlier in the lifecycle
than currently planned - effectively pre-proving the gross configuration within BIT prior to use in the
Model Office Rehearsals and Model Office Test.
© 1997 ICL Pathway COMMERCIAL IN CONFIDENCE
Page 10 of 41
FUJ00078295
FUJ00078295
ICL Pathway REVISIONS TO THE Ref: VI/STROO6
16.
16.1
16.4
16.5
TESTING & INTEGRATION APPROACH Version: 2.0
FOR PATHWAY RELEASE 2 Date: 24/10/97
TEST AUTOMATI
When test activity commenced on Pathway in 1996 the target technology (notably NT) outstripped
the availability of test automation tools able to support it, and the product fluidity (notably Riposte
data formats and Oracle Schemas) detracted from the projected benefits of automation.
As a result only very limited use of test automation has been made to date in Release I Testing, and
this has for the most part been confined to the performance testing areas, in the Oracle/Sequent
bench-marking activities at Weybridge, and in simulating transaction loads on PO counters at
Bracknell.
The time is now ripe to start exploiting test automation tools on a wider basis. Initially it is proposed
that we should concentrate our efforts in 4 key areas:
a) capture facilities to assist in preserving more meaningful audit trails, and to assist in
diagnosis of problems, with particular emphasis on tests run at the counter.
a) replay and script edit support to facilitate cheaper and more extensive regression testing
across a wide spectrum of functional tests (Here ‘script edit support’ refers to facilities which
allow editing of captured material to correspond with minor changes that may need to be
made to the test scripts, to keep the captured material current and avoid having to re-capture
for every small change. They also permit replication and variation of captured tests to help
generate loads and multiple permutations of test scenarios.)
a) load simulation, extending current practice, for both performance and resilience testing
a) network simulation, for both performance and resilience testing
The proposal is that two tools, already identified and believed to satisfy the above, be piloted as a
matter of urgency, and assuming they prove capable of the task, that they be introduced progressively
into Release 2 testing, with the aim to have all essential tests constituting a comprehensive set of
regression test packs automated prior to live implementation. The two tools in question are:
© QA Centre, supplied by Compuware
© COMNET III, supplied by CACI
It is further proposed that during the course of (and as an integral part of) detailed test planning for
Release 2, any additional candidates for test automation should be identified and assessed. Where
clear benefits can be projected within Release 2 timescales, they should be adopted accordingly.
Where the benefits are longer term, they should be flagged for later study.
© 1997 ICL Pathway COMMERCIAL IN CONFIDENCE
Page 11 of 41
FUJ00078295
FUJ00078295
ICL Pathway REVISIONS TO THE Ref: VI/STROO6
TESTING & INTEGRATION APPROACH Version: 2.0
FOR PATHWAY RELEASE 2 Date: 24/10/97
17. TEST SCRIPTING
17.1 In Release I a number of serious problems were encountered by the PDA in reviewing the HLTPs and
other test material. Whilst this was in part due to the poor levels of Joint Business Test involvement
during the test preparation phases, it is nonetheless recognised by Pathway that there is much scope
for improvement here. An exercise has already commenced to devise a more effective scripting
method with clear and simple output formats, more sympathetic to non-test personnel.
17.1 Some difficulty has also been experienced in the test execution phases in the use of the test
management database used to hold and collate all the material. It is therefore also proposed that a
thorough review be made by the Test Analysis team, with a view to improving usability in this area.
© 1997 ICL Pathway COMMERCIAL IN CONFIDENCE
Page 12 of 41
FUJ00078295
FUJ00078295
ICL Pathway REVISIONS TO THE Ref: VI/STROO6
TESTING & INTEGRATION APPROACH Version: 2.0
FOR PATHWAY RELEASE 2 Date: 24/10/97
18. ACCEPTANCE TRIALS
18.1 Currently the Acceptance Trials, which confirm the extent to which the Acceptance Criteria have
been met, are planned to be conducted as a discrete activity, following the stage of testing upon which
they are based. This was based on the assumption that a relatively small number of the overall test
conditions developed by Pathway would be selected as Acceptance Conditions. (Acceptance Criteria
are made up by a series of supporting Acceptance Conditions which relate directly to Test
Conditions.)
18.2 However, it has recently become apparent that a very large proportion (over 70%) of Test Conditions
have been selected. This really makes it impractical to conduct the Acceptance Trials as a separate
activity. It has therefore been proposed that they ride on the back of their respective related Pathway
Test stages. The mechanisms by which this is to be achieved are documented in a separate paper -
Acceptance Trial Strategy - which is currently under review. Assuming this meets favour, then it is
proposed to incorporate this within the test strategies and test processes accordingly.
18.3. Briefly, this describes how the BIT and T&S test streams may be used to perform related Acceptance
Trials, with Acceptance Officials present during the latter portion of the tests to invigilate
accordingly.
© 1997 ICL Pathway COMMERCIAL IN CONFIDENCE
Page 13 of 41