POL00030392
POL00030392
Few Rutt thllaann 218144
43 AUG 1999 - 86°
Summary of Overall Position
1. Position at end of last Acceptance workshop on 10 August (Recognised by ICL
Pathway).
The last of the 3 acceptance incident meetings between POCL and ICL Pathway
concluded with the POCL Hotlist containing:
Disputed Acceptance Incidents:
8 High (one of these 376, was previously agreed as high)
2 High/Medium
12 Medium (two of these were previously agreed medium)
22
However, a lot of work was due to be complete during 11 August, and a revised
position is now proposed by POCL.
2. Results of work on 11 August
The POCL review of the actions today has concluded that of the disputed
acceptance incidents
Changes POCL would accept
High’s closed 2 (300, 301)
high’s to low 3 (410, 411 [both subject to 376 actions], and 384)
Medium closed 1 (394)
Medium to low 1 (211)
These are shown in attachment 1
3. Position POCL enters Management Resolution Meeting
3 High - 376 Data integrity on TIP files
- 298 System instability (lockups and reboots)
- 218 Training
12 Medium (including the two former high/med
The summary of acceptance incidents table is shown in attachment 2 é
On this basis, grounds for not accepting will rest on:
™@ the 3 disputed high severity incidents
™@ inability to agree resolution plan for the medium Acceptance incidents.
The most worrying Incident is : 298 System Instability
e Pathway have not explained satisfactorily why these occur
¢ they have requested an extension to the time to conclude their analysis. This gives
us insufficient time to respond so we have refused
e the frequency of occurrence has not been clearly established or agreed
e we have rung around samples of offices
e we have received the ICL HSH logs while awaiting Pathway analysis
° we have suggested ways for Pathway to get more detail on freezing
e Some of these occurrences maybe the responsibility of POCL (the SPMR rebooting
without good reason)
e this could cost £1.8m per annum and inconvenience 1 million customers a month.
The Clear High is: 376 TIP - Data Integrity
376 is a clear high (as it was originally agreed by Pathway as high) but Pathway are
making progress:
Pathway had a codified work around:
@ implemented 3/8
@ Pathway will not confirm that this solves all the problem
@ We need a full Accounting period to be sure, but will accept 2 full clear
weeks. (We have had periods of low occurrences, so one week of zero
occurrences could be followed by a non zero week). The second full week
will occur on 18" cash account and would flow through our systems for
checking on 23".
We will only get the Pathway root cause analysis today 1 i Pathway claim they will
have an adequate rectification plan by Friday 13". These problems are caught by
“matching” at TIP, but this facility will not be available in roll out.
So POCL want:
a) a solution that meets the requirements: i.e. a full reconciliation in the Pathway
domain (before the data gets to TIP)
b) to agree the rectification plan
c) the rectification proven
NB. Problem 891 looked good after 1 week but then became clear the rectification
had failed.
The high cost of work around incident: 218 Training J
Training has not been adequate. Training is not just the course but other events. With
ICL, we have worked on a range of solutions. While these solutions have improved
the position, staff have only been able to cope as well as they do, because of HFSO
support.
This is shown in attachment 3
Details of all the disputed Medium and High severity incidents will be sent by Ruth
Holleran in time for the meeting.
POL00030392
POL00030392
Severity from to Reason
Criterion
410 Loss of data High to low © Originally part of 376
integrity when e¢ POCL understands cause
changing products ¢ Rectification plan presented and
from core to non-core agreed
e ICL putin fix 10/8
© outstanding issue on
reconciliation covered by 376
411 Correspondence I High to Low/None © Originally part of 376
server not replicating
properly in bulk
e POCL understands cause
© Rectification plan presented and
agreed (confirm BoBo)
© outstanding issue on
reconciliation covered by 376
© outside core observation period
(confirmed position John Cook)
211 Receipts not =
payments
Medium te-Lew- contested, See Bob
Booth Memo
e Rectification plan implemented
and results show worked
e Some remaining incidents due to
reference data
© 1 incident reported possibly due
to kit swap out re-introducing
pre-LT2 s/w
¢ Action with Pathway to
investigate
* possible that combination of
builds have not been tested
properly
e reduced telow evenwith this,
394 differences Medium to closed © rectification plan implemented
between reprints of via LT2
cash account e No incidences since LT2
300 System lock up on I High to closed e Pathway suggested this was part
printer failure of AI298
Following POCL investigations
unable to substantiate AI300
incident in own right
© Will close provided Pathway
consider printer incidents part of
AI298
301 printer failure High to closed © Pathway Suggested that AI301
resulting in data was a cause of A}211
integrity loss ¢ As AI211 now low and no further
incidents of doubling up of
opening balance problem since
LT2_- now closed
384 sequent failover High to low ¢ POCL reviewed results of
rectification plan
¢ generally satisfactory but a few
issues still outstanding
POL00030392
POL00030392
ACCEPTANCE INCIDENT SUMMARY @ 11" AUGUST 1999
High Priority Incidents
POL00030392
POL00030392
A-Spec AT Description Severity Business Impact Status Proposed Actions
TIP 376 Derived cash account High Pathway claim to have delivered an ICL have timed out on the retest so
from daily txns sent to additional fix. They will have this is a failure to meet the
TIP is not equal in all evidence from live environment on —_ acceptance conditions.
cases to the weekly 13/8. POCL will have evidence on _ Alternatively it could be proposed
electronic cash account 16/8 but need at least 3 weeks to
continuous observation i.e. cannot ‘1. Agree additional time required
be proven prior to 18/8. to verify corrections in live
In addition Pathway need to deliver 2. Agree time for Pathway to
an internal reconciliation process to deliver their internal
check daily txn files and the weekly reconciliation and the period
cash account always agree. over which it needs to be
External auditors expected to observed in live
confirm this requirement
POCL 298 Counter systems subject High A failure rate per terminal of one Pathway cannot yet provide a 1. Need better analysis of
Infra to an unacceptable per month in full steady, state will comprehensive root cause analysis incidents reported to HSH from
frequency of lockups equate to 30000 lost hours of but have claimed occurrence is Pathway (and BSM)
which requires reboot of counter staff time per month within industry tolerance which 2. Need comprehensive resolution
terminal(s) affected. (40000 terminals x 45 minutes) or / POCL dispute e.g 76 system plan
£1.8m per annum. freezes reported in week 19. Also 3. Establish whether Gartner,
AI 300 (printer related In addition the number of there have recently been 164 Meta etc can provide
lockups) now proposed customers impacted by the loss of _ instances of HSH advising offices benchmark comparisons
as combined with this AI serving time per month is up to 1m to reboot within a 5 day period.
This equates to a reboot per
terminal of one per month
Training 218 — Training course for High Evidence evaluated from additional Awaiting Pathway response to
managers inadequate
offices.
BMc has written to J Dicks -
performance still not acceptable
due to high level of required
support in offices from HFSOs
BMc letter
12/08/99
page 1
Medium Priority Incidents
POL00030392
POL00030392
A Spec AI Description Severity Business Impact Status Actions
Rollout 372 Quality of systems Medium Pathway resolution plan evaluated Agree approach to retest or
management (upgrade but cannot be satisfactorily redemonstration of upgrade
problems during LT2) demonstrated pre 18/8 processes ideally with greater
number of offices
POCL 369 Scanner reliability in Medium BA have raised issue with POCL BA letter being evaluated
Infra relation to OBCS regarding frequency of failure
transactions
TIP 371 Late transactions over9 = Medium Latest Pathway and POCL evidence Awaiting latest evaluation
days old on HAPS being evaluated to establish
system whether requirement is being met
APS 390 Recovery of APS Medium Pathway resolution not yet
transactions acceptable
APS 395 Duplicate AP transaction Medium Creation of duplicate transactions —_ Original problem now appears Resolution needs confirmation
reference numbers results in over or under payment —_resolved but system remains
with clients vulnerable to keying error.
Pathway verbal proposal has been
made to modify screen entry
TIP 361 Duplicate records and Medium Part of original incident appears Pathway analysis and resolution
files over TIP interface corrected by LT2 but duplicate AP _ required
txns still occurring
TIP 378 AIS contravention/data Medium Pathway claim fix provided and Assess documentation and live
integrity error orcash evidence will be available by 13/8 experience on 13/8
account
Medium Priority Incidents continued
12/08/99
page 2
POL00030392
POL00030392
A_Spec ATI Description Severity Business Impact Status Actions
Security 368 Physical security of Medium Security grills not yet installed ‘Awaiting confirmation of action
Lytham St Annes from Pathway
computer room
Security 391 Deficiency in physical © Medium Resolution plan not agreed with Agree outstanding actions
security at Bootle and POCL
Wigan
TIP 342 TIP data file delivery - Medium Pathway rectification provided but Agree closure process
SLA breach demonstration cannot be proved
prior to 18/8
POCL 314 Provision of technical Medium Pathway claim adequate detail has Awaiting clarfication of POCL
Infra documentation for TP been provided view
suppliers inadequate Contracts currently clarifying the
position
Service 408 Failure of Horizon Medium New HSH scripts only available for Awaiting detailed Pathway
Levels System Help Desk to review on 11/8 proposals
support the network Call volume projections and
resource plans not available for
POCL assessment
12/08/99 page 3
POL00030392
POL00030392
Acceptance Incident 218 - Status
An analysis of the evaluation against the business impacts identified in the
Acceptance Incident has been completed.
Although many of the criteria have been met, it is regarded as significant that the
training and go-live process relies on the deployment of POCL HFSO resource. On
the basis of this evaluation, we are not prepared to reduce the severity rating from
‘high’.
POCLs view is that without this resource there would have to be a complete revision
of the training approach in order to ensure helpdesks were not rendered ineffective
by the high level of calls following the first and, to some extent, subsequent balances.
POCLs view is that HFSO resource was not deployed as an extension of training.
The cost impact and diversion of resource which this requires must be addressed by
ICL Pathway.
It is also POCLs view that the related adequacy of HSH support must be integrated
with this Acceptance Incident and removed as an additional source of concern.
The training improvements identified as part of the qualitative research by Post
Office Business Consultancy must also be addressed as part of a rectification plan.
POL00030392
POL00030392
, : .
Outstanding Action
The Training Experience
User Outlet Contact with Training PSA Training After Care HSH Support
Awareness Contact to Training Event Mode
Event establish Helpdesk
Training
Requirement
Declared Opportunity I Building Building Training in Measured value I Integral part I 1 hour per Advice and
Intention to see and relationship relationship with I use of system I of training of training person guidance plus
touch the with outlets. outlets. Flexible in live experience and solution. aftercare. technical support.
equipment. Establishing and sympathetic I operation. requirement for Subsumed
Removal of training needs. I approach to remedial training. within HFSO
“Fear Factor’. scheduling within role.
contractual
window.
Actual Well received I Both regarded as mechanistic Specification I Subjective; value I Not Does not Universally
Experience as described. I approach to scheduling. as good as unclear. providing address criticised for
Maximisation of course places. Not I possible added value I support to unresponsiveness.
regarded as user friendly or part of I within time to training _I balancing.
training experience. available. experience.
Deficiency or @ Pre entry training/awareness Assessment @ Review Integrated @ Cost of ™@ Meet agreed
requirement [I = ---- required to address differing and response effectiveness I with POCL user
levels of user experience and to trainer of application. I training support to requirements.
abilities to learn. quality. @ Review solution. balancing. @ Investigation of
@ Number of courses in the effectiveness @ Extended continuing
solution does not allow of remedial RNM failure to
streaming beyond agreed training. training and balance at
approach to ECCO outlet stock @ Pathway to deployment. ‘rump’ of
holders. take @ Support outlets.
® Number of courses may not meet responsibility beyond first
requirement of user population. for not passing balance.
on people who
cant operate
the system.
Incident Description and Impact - Horizon Incident Number 218
ad Criterion : 534/1
POL00030392
POL00030392
“Pathway’s Training solution shall take account of users experience in terms of automated products and platforms (ECCO+,
APT, ALPs) and their differing abilities to learn”.
2. BUSINESS IMPACT
SUMMARY OF SUCCESS CRITERIA
MEASURE
EVALUATION
1. The Office Managers ability to
undertake daily balancing and
produce a cash account is adversely
impacted resulting in a failure to
support accurate POCL accounting.
This is a high severity impact on
POCL’s ability to perform its normal
business functions.
@ Reduction in the number of offices unable to
complete the cash account balance process
and produce a cash account balance (relative
to the sample).
@ Continuing or better level of success in the
pass rate of the Performance Standard
Assessment (PSA) test.
@ Performance Standard Assessment (PSA) to
reflect live operation and standard practices;
Horizon users complete PSA again on day 10.
Data from BSM telephone survey for balance
related to the 4" August contained the
following; 22 offices produced an account, 1
office had a two week cash account. This
criteria is therefore met.
Criteria met.
Criteria met.
2. The consequential effect is that the
amount of time taken to produce the
cash accounts is excessive in relation
to the time taken on the previous
(manual) system and significantly in
excess of POCLs expectations for the
service.
™@ Reduction in time taken to produce a stock
unit balance , the office balance and finally
produce a cash account (relative to the
sample.
Balancing statistics for the 4" August Cash
Account indicate an overall reduction in time
taken to complete balances in both sub offices
and ECCO offices. Even at the reduced level,
concerns remain about overall balancing times.
oO}
POL00030392
POL00030392
BUSINESS IMPACT
SUMMARY OF SUCCESS CRITERIA
MEASURE
EVALUATION
3. The consequences are also that the
number of cash account related
incidents reported to POCL NBSC is
considerably greater than expected.
(About a third of the calls coming to
NBSC Help Desk indicate a lack of
understanding of the cash accounting
and balancing process). HSH are
responsible for resolving these service
incidents but are unable to cope with
the content and volume of calls which
are therefore having to be dealt with
by NBSC. As the Manger’s training
course is deficient, NBSC and
presumably HSH staff who receive
this training course, are also
inadequately trained.
@ Reduction in demand on support - Measured
through a reduction in the number of calls (at
the peak time on Wednesday evening and
Thursday morning) for advice and guidance
to support stock unit balancing, office
balancing and production of the cash account
received at the HSH and/or at the NBSC.
™@ Reduction in the length of calls from the
additional 25 offices.
@ The overall number of calls in weeks 1, 2 and 3
by the LT2 offices showed a reduction on the
LT1 mirror offices for the equivalent three
weeks.
The average number of calls made by offices
during the non-peak days also showed a
reduction.
However, it should be noted there is a significant
increase in the 2"4 week cash account for both
LT1 and LT2 offices when there is no support at
these outlets, suggesting that some of the outlet
managers still do not have the confidence or
ability to complete the process unsupported.
@ The evidence to analyse this criteria is limited
and was regarded as indicative only. The broad
conclusion is that the evidence is insufficient to
make a substantive judgement regarding first
cash accounts but there is overall evidence to
suggest a reduction in call times for 2°4 cash
accounts. However, it again has to be noted that
the length of calls for both LT1 and LT2 offices
was significantly higher on 24 cash accounts
than the 1s cash account suggesting the critical
requirement for training to be
supported/ delivered by HSH. It also
underlines the necessity of the HFSO support to
balancing in week 1.
POL00030392
POL00030392
BUSINESS IMPACT
SUMMARY OF SUCCESS CRITERIA
MEASURE
EVALUATION
4. The practical effect of the incident is
also causing the HFSO’s to devote a
disproportionate amount of time to
support the outlets on cash accounts.
The number of HFSOs that would be
required to support National Roll-Out
would be significantly greater than
currently envisaged (initial indications
are that two to three times as many
HFSOs as planned would be required.
This compounds the major impact on
POCLs resources.
@ No specific success criteria was identified to
address this business impact. Overall, POCL
would wish to reduce the cost of extended
training support at outlets through HFSOs.
™@ POCL are now planning for 100% support of
first cash accounts and recognise that significant
additional support may be required for second
and subsequent balances at some offices. This is
a cost and resource drain on POCL. It is also a
change to the original HFSO role which was to
support the KPI delivery for POCL and to
accelerate the learning curve at outlets. POCL
concerns on this impact remain.
5. There is also an impact on TP who are
having to process a significant
increase in errors on Class and Pivot
(up to 3 times as many weekly errors).
This is having a significant impact on
resources in TP during the live trial.
These errors will also raise liability.
issues between the POCL and
subpostmasters, and POCL and client
organisations.
®@ Reduction in both the number of incidents
where Receipts do not equal Payments and
Incidents where balance B/F does not equal
balance due to PO on previous Cash Account.
™@ Reduction in the number of errors reported
by TP - both CLASS and PIVOT errors
(relative to the sample).
™ Overall, the incidents of receipts not equal to
payments have reduced and the residual causes
are under investigation or have been resolved.
Criteria met.
@ The level of CLASS errors between 26 May and
21st July has reduced. Without full information,
the indications are that PIVOT errors have also
reduced.
POL00030392
POL00030392
3: Qualitative Measures
3.1 Although the small sample size of 18 responses limits the validity of the findings, some significant improvements were found in
comparison to Live Trial 1 (a sample of 102). Overall, attitudes towards Horizon are better at the LT2 offices compared to the
LT1 experience. The key outstanding issues to emerge from research were as follows:
™@ The course is still considered to be too short and intensive. ICL have proposed a pre-training course but details are
awaited.
™@ The need to further stream the training groups. This issue has not been addressed by Pathway beyond the streaming
required by POCL for ECCO+ staff. Pathway’s response is to do this wherever possible. There are impacts on the
number of training places.
® Variation in trainer quality. Discussions taking place between POCL and ICL Pathway to look at how there can be a
greater quality assurance for trainer ability and consistency of delivering the course specification.
@ There are significant problems with technical and software faults in the training sessions. POCL regard these are
significant issues which will require rectification.
21
H:\DATA\WORD\ Horizon\ Acceptance Incident 218.doc