POL00028407
POL00028407
Gente
«= POST
OFFICE
Vince Gaskell
CAPS & Cards Programme Director
Benefits Agency . Post Office Counters Ltd
DSS Longbenton Horizon Progromme Director
Benton Park Road
Newcastle upon Tyne
NE98 1YX : 1* April 1999
HORIZON TESTING AND ENTRY TO LIVE RUNNING - POSITION FOR THE RAB
Thank you for your reply of 26 March. Following our discussion on Wednesday morning and the
further note from Carol Jepmond I believe there are four essential questions in the testing arena which
we need to answer as Sponsors before we give approval to enter Live Trial at the Release
Authorisation Board (RAB) next Wednesday. These are as follows:
1. Are the number and severity of outstanding faults in the system acceptable for entry to Live Trial ?
2. Is the Pathway solution sufficiently stable so that we do not risk significant numbers of new critical
errors emerging in Live Trial ?
3. What further testing is planned after the RAB that may address any outstanding concerns ?
4, Can we be confident that new software changes introduced during Live Trial will not undermine
the integrity of the solution that will be accepted by the Sponsors ?
MayI take these questions in turn.
1. Are the number and severity of outstanding incidents in the system acceptable for entry to Live
Trial?
The final- incident statistics reported from Model Office/End-to-End are somewhat different from
those in your letter and are 7 high priority, 100 medium priority and 75 low priority incidents, These
resulted in a total of 125 incidents being included in Target Testing, (addressed by 102 ICL Pathway
Pinicl fault clearances).
There are two aspects of outstanding incidents:
a) the typically low priority incidents that have been placed on the Known Problem Register (KPR) as
acceptable to be deferred for resolution until during Live Trial or even rollout. Both Sponsors have
now reviewed the KPR in detail and I believe there are no issues that will prevent it being agreed at
the RAB
b) the incident status as at the end of Target Testing, which is as follows
Total Incidents in Plan 125
_ Actual Incidents Covered 125
Number Passed. 123
Number Failed 2
New Incidents Outstanding 7
Post Office Counters Led
King Edward Building
King Edward Street
Pet Oca Coeur to Regnard ted
(Me thee Reprod Otc ag Card eng
Krgleced torn tone (Cia HA
my
POL00028407
POL00028407
Vince Gaskell
18 April 1999
page2
The 2 failed incidents are one medium priority and one low priority incident, as defined by our joint
BA/POCL Product Assurance Team, and are acceptable for resolution post the start of Live Trial. OF
the new outstanding incidents 5 are low priority and 2 are medium. These covered 3 minor counter
report errors, an icon error, a transaction date error, and two errors of inconsistent details on the TIP
interface files. None of these errors are essential to be cleared for start of Live Trial.
Our clear view from these figures is that the number and severity of outstanding incidents from the
KPR and Target Testing are acceptable for the RAB. 7
2. Is the Pathway solution sufficiently stable so that we do not risk new critical errors emerging in
Live Trial?
This is a more difficult question as it requires a prediction of the future. Your view to date has been
that the number of incidents that arose in the last full Model Office/End-to-End Cycle (MOT/E2E)
was of a magnitude that it requires a further full cycle, rather than just ‘Target Testing’, in order to
demonstrate there are no regression or other hidden faults still to be discovered. ‘Target Testing’ is
however something of a misnomer as it exercised the end-to-end solution from CAPS through the
Pathway counter and into the POCL back-end systems, It would be more accurate to describe it as an’
End-to-End run focused on the remaining problem areas while largely excluding proven areas such as
OBCS and Automated Payments (APS). Target Testing actually covered all the areas with outstanding
faults including a range of benefit encashments, BES fallback processing, all mandatory repotts, the
‘cash accounts, the POCL accounting interfaces, and the 12 day settlement routine with BA.
Weare now ina position where:
the number of outstanding new incidents arising from Target Testing is only 7 and none are critical
« ‘Target Testing exercised 50% of all End-to-End specific test conditions while only finding this very
low number of new problems (n.b. the percentage of test conditions, as a measure, does not reflect
adequately the scope and criticality of the functions covered in Target Testing given the exclusions
in proven areas such as APS and OBCS)
. the 50% of conditions not covered all passed in the previous MOT/E2E cycle
> Target Testing has given no evidence. that the correction of errors in one area is causing indirect
impacts on other areas of functionality . : :
© there is no area of functionality in the system that now has any cluster of errors that would lead to
concerns about stability .
* all live reference data for the Live Trial offices has now been validated and signed off by the POCL
: business owners :
all migration tests have now been successfully completed against live reference data.
This afternoon the POCL customer assurance group and Network Operations management met with
my Horizon team. They have formally endorsed the success of Target Testing in providing them with
sufficient confidence to proceed to Live Trial. Another full MOT/E2E test cycle would add little value
as it would only redemonstrate areas and conditions that have already been successfully tested. There
is also no evidence of instability introduced by the Iatest round of corrected faults.
_ We therefore must conclude that the above evidence gives the sponsors sufficient confidence in the
stability of the Pathway solution to proceed to Live Trial.
]
POL00028407
POL00028407
Vince Gaskell
1 April 1999
page3
» 3,Will the further testing that is planned address any outstanding concerns ?
We should note that completing the MOT/E2E cycles does not end the testing of the Pathway NR2
release, for example:
* from April 12 Pathway will be performing a further run of their 4 week BIT Regression Cycle. The’
previous run of this cycle has already validated by BA and POCL reviewers in terms of the
adequacy of its content and conduct and can be witnessed again by sponsor representatives
* CAPS will be able to run regression tests of their new CAPS 3.5.release against NR2 in April
* the start of the Multi-Benefit Model Office on 12 April will enable both BA and POCL to conduct
further tests in this environment which they may require in addition to the specific objectives of
multi-benefit. This environment will have the same codeset as the Live Trial and therefore we will
have a parallel test environment throughout the period of Live Trial for any further tests the-
Sponsors consider necessary.
The BIT regression test and the CAPS 3.5. regression test will give the BA the confidence of thorough
regression runs before the formal start of Live Trial on 10% May, together with the experience of the
migration of the 204 current offices. Any incidents that arise in these areas can be reflected in the
Acceptance process and may be requested for inclusion in the Live Trial release discussed below.
It should be noted that if serious problems should arise, despite our expectations, in this additional
testing during April, or if indeed the migration phase of the existing 204 offices throws up unforeseen
issues then the Sponsors would have to reconsider whether the formal Live Trial period should start as
planned on 10 May.
4. Can we be confident that new software changes introduced during Live Trial will not undermine
the integrity of the solution that will be accepted ? :
As you are aware we have had initial discussions with Pathway on the approach to software changes
in Live Trial and the proposed approach is as follows:
¢ there will be the facility within the Live Support procedures, as in the present live system, to
include individual emergency fixes in order to correct urgent problems
¢ there will be one software upgrade release during Live Trial scheduled provisionally for 1* July
which will give the sponsors 5 to 8 weeks to observe performance before acceptance
+ the contents of this release are still to be agreed by the sponsors . It may contain the incidents on the
KPR, plus the few new incidents from Target Testing, and incidents arising within Live Trial. Given
the low priority of many of these incidents, however, the sponsors will have to decide the number
of changes that would be sensible to introduce during Live Trial. The content of the release will
then influence the degree of testing that the sponsors would consider necessary
¢ itshould be noted that this upgrade is not intended to introduce any new functionality
* Pathway propose to system test this release during May and demonstrate it in a full BIT regression
cycle during June which the sponsors will witness
* the standard configuration management procedures within Pathway, which have recently been
reviewed by BA and Horizon, will release this software upgrade to the Pathway and joint testing
environments and following sign-off into the live environment
* this release could be provided into the Multi-Benefit Model Office in June to allow direct
BA/POCL testing, although this option needs to be formally considered by the Sponsors.
BA will of course be involved in agreeing the contents of this release and developing the appropriate
testing approach. We see no reason why this should not result in a clear and controllable process for a
single Live Trial software upgrade
POL00028407
POL00028407
Vince Gaskell
1" April 1999
page4
Conclusion
POCL and the BA must strive for a quality and stable solution to be delivered by ICL Pathway. One
can always argue that more comfort could be gained from a further Model Office test cycle. However,
the results from Target Testing, together with the other points made above, lead us to the very clear
conclusion that :
there are no outstanding faults that prevent entry to Live Trial
° the stability of the solution in Target Testing gives confidence that there is no major risk of new
faults arising, and supports us proceeding to migration and the start of the Live Trial, with A to5
more months before we make an acceptance decision and commit to rollout
« the BA and POCL can obtain further assurance by the planned additional testing activities
discussed above rather than by inserting another Model Office cycle
© required changes can be included in the Pathway service in a controlled manner before acceptance
the current testing status cannot justify two more months of additional Model Office testing
The Post Office, and ICL Pathway and Fujitsu, strongly endorse this conclusion, and this view will be
represented at the highest levels, We could not agree to a continuation of testing that effectively would
result in a 6 month delay to rollout until after Year 2000. I trust that the BA will also be able to support
this conclusion.
Yours sincerely
DAVE MILLER
Horizon Programme Director
c.c Steve Robson, Paul Rich, Jonathan Edwards, David Sibbick, Mena Rego