POL00028332
POL00028332
=
13 AUG 1999 - 37%
Electronic memo
To ™'peter.coppil
"'david.rees
ce Bruce McNiven{
<chris.frenct.
‘Mike Coombs"
‘John Dicks'*
Hard Copy To
Hard Copy cc
From Andrew Simpkins [7
Date 13/08/99 15:12
Subject Papers for Mondays' Meeting with Expert I
Peter/David
Following the Management Resolution meeting yesterday I attach as agreed by POCL
and Pathway the minutes of this meeting and a summary of the incidents that are in
dispute.
The minutes will give you an up-to-date position on the high priority incidents in
particular.
We propose that the meeting with yourselves does not now start until 12.00. It will
be in Gavrelle House room 7.
There is a room available for your use in Gavrelle House all day - 309 on the 3rd
floor. When you arrive please ask for Dave Miller's secretary.
If I can be of any further help please ring me on;
Andrew Simpkins
Horizon Programme
MR Minutes 12 Aug.doc
- MRS_1208v1.xIs
POL00028332
POL00028332
i
Horizon Programme
Management Resolution Meeting
Gavrelle House
12.30 - 3.00 & 4.00 - 7.15pm
12" August 1999
Minutes
Attendees:
POCL: B MeNiven, K. Baines, J.Meagher, C.French, R Holleran, M. Burdett,
A. Simpkins (minutes)
Pathway: M.Coombs, J.Dicks
1. Initial Points
Set up session by 27" August to review POCL ‘Low’ incidents (which Pathway may have
classified as ‘None’) Action :JM
‘The letter from JD to KB of 11 August on Pathway’s view of Acceptance Incidents was
noted (see Attachment 1). Up-to-date copies of the AI descriptions were distributed.
2. Current Status of Disputed Severity Ratings on Hot List
No. Short Description PWY POCL
342 TIP data file SLA breach Low Med
361 Duplicates Low Med
371 Late HAPS transactions Low Med
376 Derived cash account Low High
378 AIS contravention Closed/Low Med
211 Receipts not equal to Payments Closed Med
218 Training Closed High
372 Systems management None Med
368 Physical security Low Med
391 Physical security Low Med
390 APS recovery Low Med
298 Counter lockup/freezes Low High
314 Technical Documentation None/Low" Med
369 OBCS scanning None High/Med
408 Help Desk None/Low Med
395 Duplicate AP numbers None/Low Med
The above position was agreed and results in the following totals:
3 highs
1 high/medium
POL00028332
POL00028332
I
12 mediums
‘The following have been reclassified by POCL and taken off the Hot List:
300 Printer lockups “Closed (covered now within 298)
301 Integrity faults from Printing Closed (included in 211)
384 Sequent failover Low
410 Missing transactions Low
411 AIS contravention Low
394 Cash account reprints Closed
POCL advised that they rate 410 and 411 as Low-conditional on resolution of the controls
proposed on 376. RH would provide further detail on this. Action : RH
Pathway noted that in their view 411 occurred after the due date but they are correcting it at
their own cost. The Contract is not clear in their view on Incidents that occur after the due
date but before Acceptance. POCL‘s view is that this is a legitimate incident, whereas
Pathway are treating it under Clause 411 of the contract..
Pathway has used the term ‘None’ to describe an Incident (i.e. not high, inedium or low). It
was proposed that ‘None’ should only reflect the status of an Incident (i.e. whether it existed
or not) rather than severity. Pathway agreed to review those incidents described as ‘None’
and to either propose they are closed or to give a severity rating. Action : MC
‘There is a need to clarify the testing and observation that has been performed by Pathway in
order to close or downrate Incidents. POCL’s view is that under paragraph 4.2. of the
Acceptance Schedule analysis alone does not justify any change in severity and that the time
allowed for retesting is 2 weeks following the end of the Core Observation Period. Pathway ‘
contested this view saying they believe they could reclassify incidents based on their view of
the situation at the end of the period (11 August) and had done so. KB would follow up the
evidence for incidents as appropriate
3. Review of High Priority Incidents
AI 376 Derived Cash Account not Equal to Electronic Cash Account
JD reported as follows:
Pathway recognise that not all transactions had been harvested and sent to TIP. A provisional
fix went in on 2™ August and this has worked satisfactorily so far with the effect that all
records had been sent. A root cause analysis has been developed, identifying 8 contributory
problems, and all but one has been diagnosed and tested in Pathway to date. Pathway cannot
guarantee however that all problems have been trapped. They will need to see evidence from
the fix of the 8 known problems, and will continue to monitor the problem for 3 months to be
confident of its resolution .
‘The provisional fix and the control procedures developed allow Pathway to identify any
errors, to patch the file , and to notify TIP in advance. Since implementation there have been
no errors to report and hence Pathway contend that the action taken to date and the result
they have observed justify the downgrading of this incident.
‘The fix for the 7 solved problems will go in tonight (Thursday). The last item will be
delivered separately - date to be confirmed.
POL00028332
POL00028332
RH said that POCL need to see the impact of the full fix and to agree the duration required
for observation. Secondly POCL need assurance as to how controls will subsequently work
on the Pathway side to prevent reoccurrence.
On observation it would be preferable to monitor an accounting period in full. RH said POCL
would be prepared to consider observing only the remainder of the current period which ends
on 25" August, which would include two weeks cash accounts under the new fix. The
analysis and confirmation of the evidence would be available in POCL by 30th August. Two
weeks are seen as a minimum because the nature of the fault can be intermittent and there is
a need to ensure that there are no subsequent knock-on effects.
Pathway believe they have made sufficient progress to rate as low. POCL believe they must
have the opportunity to verify the fix.
Pathway pointed to the fact that no new errors have occurred since 2™ August. POCL believe
they cannot downrate until their own observation can be performed given the risk to the
accounts. They recognise the corrective action that has been taken but the potential size of
errors that can occur is too serious to risk proceeding without proof, which will not be
available until 30th August. Advice from the POCL external auditors was that without clear
verification of the success of the solution there would be implications on the qualification of
Post Office Limited accounts.
RH reiterated the need for an ongoing control. Pathway asked for POCL’s view as to how
this might work in the longer term. RH said this would include advance notification of when
there is a discrepancy on daily transactions. JD said a check to do this is part of what is
already in place but Pathway were prepared to discuss what further could be done to meet
POCL’s concern.
In summary:
* The difference in view is logged.
* The POCL request for improved control should be articulated and related to the current
Pathway solution and the contractual requirement. Within the current requirement,
Pathway would be willing to perform more checks or to amend the file creation process at
their end, but this needs to be related to the end-to-end interface process.
Two actions were proposed:
1. POCL and Pathway will look at a design change to provide a more robust long-term
solution.
2. POCL agreed to review Pathway’s current monitoring process to establish how this might
contribute to their requirement for improved control, and to consider Pathway’s request as
to whether it may influence POCL’s severity rating.
Action: JM/John Pope/Graham Seedall/Mark Burley/Peter Jones to meet in London
tomorrow on both the above items
POL00028332
POL00028332
1298 System Stability (Counter lockups and freezes)
JD reported as follows:
Pathway consider that the Incident has raised its profile only recently since telephone surveys
revealed the extent of lockouts and reboots in outlets. This came as some surprise to
Pathway. Pathway then performed their own phone survey which confirmed the prevalence
of rebooting to a greater extent than recorded at the HSH.
POCL replied that they did not believe this was only evident recently but that the quality of
analysis had been lacking.
Pathway believe that some offices are rebooting unnecessarily and without calling HSH.
Reasons given by offices were various and not entirely clear e.g. no point in making a phone
call if the expected advice is to reboot anyway. Also Pathway claimed that larger offices feel
there is not much difference in the time taken to process a call to HSH and rebooting
(approximately 15 minutes). POCL stated that the total time to perform all actions associated
with a reboot could run to 40-45 minutes.
In Pathway’s view some reboots are also caused by users switching the machines back on
(when they should not have switched off).
POCL however believe that offices are rebooting because this is how they have been advised
by HSH in the past and this is the main reason why they have not called again. Pathway
believe this leads to reboots which would not be the advice given if the HSH had been called.
It was accepted that progress in analysing causes would be made by concentrating on the 9
offices that have displayed the greatest incidence of reboots. We must not lose sight however
of the many offices which have a lower frequency of freezes and reboots. These also need to
be identified from the HSH logs.
Pathway tabled an analysis of calls that were due to freezes and lockouts. There was a
discrepancy in understanding regarding these figures and those provided within POCL. MC
said that Pathway had previously written to request an extension of the deadline for analysis
of this incident to Friday 13" August but that KB had written back refusing the request.
Therefore Pathway had difficulty in providing further analysis. Following discussion of the
nature of the analysis proposed, POCL agreed to consider any further inputs from Pathway
but could not agree to take them into account in the Acceptance process until they had seen
them and determined how much time would be needed to review them.
Pathway have been performing further lab tests. Suggested causes for lockups investigated so
far were that the swap file is too small and that there are memory leakages. No evidence for
these have been found to date. Pathway will also contact offices who are rebooting to identify
the conditions and evidence.
BMc and RH emphasised the business impact on staff time and cost, and on customers, of the
loss of the service. This was particularly severe in one or two position offices. Fuller impact
details would be ready for Monday. The frequency is significantly greater than on the current
POCL legacy systems.
IM said that the Incident was wider than reboots - it was the overall loss of service to the
customer, and this had to be taken into account in the severity.
POL00028332
POL00028332
POCL also stated that the current absence of reasons for this type of failure actually
increased their concern at this time. POCL needs to understand more the factors involved and
there was no evidence that Pathway’s current level of analysis provided this. Pathway said
they will continue looking for explanations.
In summary POCL believe the current position is not supportable - given the frequency and
business impact and the lack of explanation. There appears much work to be done before this
could be resolved.
Actions
1. Establish a common view and analysis of the HSH logs - Pathway propose D
Hollingsworth from their side and to base the work in Bracknell. POCL would provide
someone from BSM (Dave McLaughilin) and Testing (via Don Docherty) for a meeting
tomorrow. JM to arrange
2. Pathway(MC) to provide more information on work done to date for POCL review by the
ATM (Bob Booth) who would provide a critique for Monday. JM to arrange
AI218 Training
The recent exchange of correspondence between BMc and JD was referred to.
JD summarised the Pathway position as distinguishing the Horizon training from the
corresponding changes in business processes. Pathway have responded to concerns through
the changes in the revised course but the two issues must be distinguished. They believe little
more can realistically be done within the course. The process issues are with POCL although
Pathway are willing to help.
BMc referred back to the Requirement as the basic criterion to be met which is to provide
trained and capable users. If Pathway believed their training alone could not deliver the
capability they had not raised the issue with POCL. There was deep concern whether
Pathway’s training solution can genuinely support the NRO rollout rate.
MC said that the Live Trial has clarified the balance that is required between training,
implementation support and Help Desks within Pathway. This has led them to increase their
resources. They expect that POCL will have learned similar lessons and will be faced with
similar adjustments.
BMc asked if Pathway would say to the Expert that their training solution could support the
planned rollout. JD said this cannot be answered in isolation of other things. There is a need
for POCL to prepare the sub postmasters. MC reiterated his point of balance - that training
can only deliver so much in the change process.
BMc agreed to explore additional actions but the training has to provide the basic level of
capability. If the HFSO support was not available to support balancing, POCL’s view was
that the whole go-live process was in jeopardy. He asked if Pathway believed the course
could deliver success without this support . JD said yes. BMc said that he believed it would
not - the majority of staff could not complete the first balance unsupported.
CF said he felt the system mainly automated the manual process, that many staff were not
properly trained to use the automated process, and that the overall training was therefore
insufficient. Pathway did not share or agree with this view.
POL00028332
POL00028332
JD said he was willing to explore ideas for example in terms of pre-course education but as
part of a planned joint programme.
Some areas could be explored before Monday e.g.:
* Pre-training support
* Need to address different audiences in terms of skill levels as stated in the Requirement.
‘This may include material and course attendance
« What can be done by way of joint action
© Other areas that could be included in the joint Change Programme.
It was noted that there is a potential commercial issue as well.
Action: A without prejudice discussion could be held tomorrow however. BMc and MC to
arrange appropriate participants.
AI369 Scanner reliability
‘There is an unexplained discrepancy between Pathway and DSS reporting on this Incident in
terms of reject rates. DSS are reporting 5% false rejects in the live environment. Pathway
believe that the cause is the poor quality of the order books in use . They believe new paper
has now been ordered by the DSS and had been introduced with new books since late
May/early June, The problem with ‘old books’ would continue throughout 1999 until the ‘old
books’ expire
KB said DSS have said new books have been supplied but the problem is still re-occurring.
This raises concern as to whether there is a problem beyond the scanner itself or the quality
of the books.
Pathway contest this view. POCL however reiterate there appears an unexplained variation in
reject rates in the field.
The way forward may be to re-examine rejected books.
Pathway would want more evidence from DSS that the problem is not with them, and believe
they have demonstrated the scanners work with the new paper.
KB said more evidence is also required on the pattern of rejection in offices.
Action : POCL would present the case to Copping and ask for a view on whether this is an
Incident and its severity. RH and K Corrigan would prepare this.
4, Status of Disputed Medium Severity Incidents - Pathway dispute POCL’s rating of
medium for all of the following incidents.
It was agreed that only the status of rectification plans would be discussed as the differences
on severity ratings had been identified and noted earlier. It was noted that medium severity
incidents do require an agreed rectification plan under the contract.
‘The current rectification status of these Incidents is summarised below.
342
Rectification plan agreed and largely implemented. MB to establish when the plan is due to
complete and closure will occur.
POL00028332
POL00028332
361
Pathway believe rectification has taken place . POCL are unable to close or downgrade until
all fixes have been observed especially as there is some evidence that new errors are still
occurring. The issue is the success criteria for closure rather than the approach.
371
Rectification plan agreed but not complete.
378
Rectification plan agreed and being monitored
211 .
Rectification plan agreed but closure date not agreed due to separate fault arising as a result.
New resolution plan required for agreement based on Pathway’s analysis of the new fault.372
POCL have not accepted Pathway’s claim that there is no further rectification. Pathway to
propose a rectification plan to deal with POCL’s concems and to reach agreement.
368
Agreed rectification plan but not yet closed. Pathway informed POCL that the grille had been
installed and requested closure.
391
Agreed rectification plan but outstanding actions.
390
Further rectification required but delaying this until CSR+ is not agreed. Current workaround
is onerous. POCL requested Pathway to consider implementing a solution at the latest by
Christmas 1999.
395
Further rectification requested and proposed in August. Needs formal agreement to the
revised plan.
314
POCL to consider Pathway’s new paper and contention that they are now conformant and
respond to Pathway’s position that further action will be taken but not as an Acceptance
Incident.
408
Rectification plan is not yet agreed.
5. Process of Referral to Expert
It was agreed to pass David Rees a copy of the updated Hot List and a copy of the minutes of
this meeting after they have been approved by BMc and MC. He should then discuss these
with both parties. The documents should be ready by midday. JM and JD would be the
contact points for him.
On Monday it was believed that a POCL/Pathway meeting would proceed the meeting with P
Copping. POCL were asked to consider a start time of 12.00 noon to allow the actions
identified to be completed and the results assimilated prior to the start of the meeting.
POL00028332
POL00028332
Acceptance Incidents Discussed at Management Resolution Session 12th August
DRAFT.
13th August ‘99
NOTES: 1. These notes should be read in conjunction with the MRS meeting minutes
2. Pathway to confirm ‘none’ and ‘closed’ status of some incidents
ETE Ae x Severity I Severity
IA-Spec _ Al IDescription (PWY)_I_(POCL) I Remarks
TIP. 342 [TIP data file delivery SLA breach Low Medium
Duplicate records and files over
TIP 361 _ITIP Interface. Low Medium
Late transactions over 9 days old
TIP 371 Jon HAPS system. Low Medium
Derived cash account not equal to
TIP. 376 Ielectronic cash account Low High
AIS contravention/Data Integrity on
TIP 378 Ithe cash account (TIP) Closed_I_Medium
Loss of data integrity when
changing products from core to nonI Incident severity reduced on the basis that the
TIP. 410 {core None. Low _ Ioutstanding issue is included in Al376
Correspondence server not [AI raised outside Core Observation Period but
TIP 411 Ireplicating properly in bulk None Low _ IPathway are implementing a rectification plan
Receipts and payments not equal The meeting agreed that the recent LT1/LT2 build
POSS 211 Jon cash account Closed_I Medium incompatibility is included in Al211
Differences in reprints of Cash
EPOSS 394 IAccount report & the original report} Closed Closed
Training course Cash Account
Training 218 Imodule inadequate. Closed High
Systems management (not all 299
offices successfully upgraded to
Rollout 372 ILT2) None Medium
Physical Security of Lytham St
Security 368 IAnnes Computer Room Low Medium
Deficiency in physical security at
Security 391 IBootle & Wigan Low Medium
IAPS 390 IRecovery of APS transactions Low Medium
Duplicate AP transaction reference
APS: 395 [numbers None/Low I_ Medium
Counter system subject to lockups
I& screen freezes requiring re-
IPOCL Infra_I 298 boots. Low High
DRAFT. Page 1 of 2
POL00028332
POL00028332
Acceptance Incidents Discussed at Management Resolution Session 12th August
DRAFT
13th August ‘99
NOTES: 1. These notes should be read in conjunction with the MRS meeting minutes
2. Pathway to confirm ‘none’ and ‘closed’ status of some incidents
a . Severity I Severity
IA-Spec -"I Al. I Description — (PWY) "I (POCL) I Remarks
Closed on the basis that system lockups on
IPOCL Infra_I 300 ISystem lockups on printer failure Closed _I Closed_Iprinter failure are included in Al298.
Closed on the basis that printer failure integrity
IPOCL Infra_I 301 {Printer failure integrity problems Closed Closed_Iproblems are included.in Al211
Provision of tech. documentation
IPOCL Infra I 314 Ifor TP suppliers inadequate. None/Low I Medium
Scanner reliability in relation to
IPOCL Infra_I 369 IOBCS transactions None __I High/Med
POCL Infra_I 384 ISequent failover time None Low
Service Failure of the Horizon System
Levels 408 [helpdesk to support the network None/Low I Medium
DRAFT. Page 2 of2