POL00028440
POL00028440
2
=o
Cie i — i el I
= SS =n
POL00028440
POL00028440
IN CONFIDENCE
Report No: 116/99
Issued: February 2000
REVIEW OF HORIZON PERFORMANCE
AND PROBLEM MANAGEMENT
REPORTING
POST OFFICE NETWORK
NOVEMBER - DECEMBER 1999
REPORT DISTRIBUTION
For Action: Andy Radka
Graham Shervington
Mark Haynes
For Information: David Smith
Alan Barrie
Chris French
Gail Morley
David Miller
Stuart Sweetman
Garry Savage’
Emst & Young
REPORT REF: AR11699
Head of Business Service Management,
Post Office Network (PON)
Head of Service Operations, PON
Head of Network Business Support, PON
Automation Director, PON
Operations Director, PON
Head of Horizon Programme, PON
Head of Transaction Processing, PON
Managing Director, PON
Group Managing Director, Customer and
Banking Services
Group Internal Audit Director
External Auditors
Post Office Internal Audit
2™ Floor,
Soresby Street
Chesterfield
S49 1PF
Telephone:
~ Postline:
Fax No:
PAGE 1 OF 18
I CONTENTS
{
PART 1
PART2
I PART3
il
l PART 4
APPENDIX A
I “APPENDIX B
ii
} APPENDIX C
SS
= —
—
ict I
REPORT REF: AR11699
IN CONFIDENCE
INTRODUCTION
MANAGEMENT SUMMARY
DETAILED AUDIT FINDINGS
« Implementation Teams
© Transaction Processing
© Post Offices
¢ Network Business Support Centre
© Problem Management
IMPLEMENTATION PLAN
AUDIT TERMS OF REFERENCE
AUDIT OPINION DEFINITION
DEFINITION OF RISK ASSESSMENT
POL00028440
POL00028440
PAGE 2 OF 18
POL00028440
POL00028440
ranean i
IN CONFIDENCE
PART1: INTRODUCTION
Ll A review of performance and problem management reporting for the Horizon system was
I undertaken during the period November to December 1999. The review incorporated
: visits to Post Offices, Implementation Teams, Transaction Processing, the National
Business Services Centre (NBSC) and Business Service Management in London. The
objectives and scope of the audit are outlined within the agreed terms of reference which
i are included at Appendix A.
it
i 1.2 The Horizon system was trialled at some 300 offices. From September 1999 the system
was rolled out progressively to a further 1,500 offices. Roll out was suspended in
i November, to avoid implementation during the Christmas period when offices are very
ll -busy, and was expected to resume in January. This pause has provided an opportunity to
I identify and rectify any problems with the implementation and operation of the system.
I 1.3 The system is currently operational at 10% of offices and is expected to be implemented in
I a further 10% of offices by the end of 1999/2000. This review was intended to
{ demonstrate that any problems with the operational system had been identified by
Business Service Management and that appropriate action was being taken to rectify them.
This will enable us to demonstrate to the business extemal auditors that the system is
operating reliably.
=x
1.4 A reply to this report is not required as all points were discussed and the action plan
agreed.
I 1.5 We wish to express our appreciation to. the managers and staff for the assistance and
co-operation given during the course of the audit.
mt
: REPORT REF: AR11699 PAGE 3 OF 18
POL00028440
POL00028440
IN CONFIDENCE
PART 2: MANAGEMENT SUMMARY
21 The subsequent paragraphs in this section summarise the main findings of the audit. The
report was issued initially to David McLoughlin and Adele Henderson in Business Service
-Management (BSM) for comment on 23td December. It was reissued to Graham
Shervington, Head of Service Operations and Mark Haynes, Head of Network Business
Support on 11th January after we had been notified of a change of responsibilities within
BSM.
——
=
Conclusion
2.2 Our audit opinion based on the definition at Appendix B was that the procedures for
identifying problems and reporting performance was good. We have recorded in the
detailed audit findings the issues identified during our visits and can confirm that all issues
reported by Post Offices and Transaction Processing (TP) had been formally recorded as
problems. The Implementation Team were addressing some issues which needed to be
resolved by 24th January 2000.
2.3. Two issues were identified during the review which had not already been reported as
ri
problems. These are:
the need for a more formal and standardised procedure for notifying system and
ie
procedural changes to Post Offices, Retail Network Managers (RNMs), Horizon Field
Support Officers (HFSOs) and the Helpdesks;
“e the need to establish a process for identi fying procedural problems and communicating
good practice to post offices, which will form part of business as usual.
2.4 We have also made some recommendations to improve the effectiveness of the process for
monitoring and resolving problems once these have been identified.
rt
Chris Paynter
Audit Manager
Tan Johnson
Lead Auditor
REPORT REF: AR11699 . PAGE 4 OF 18
en
=)
POL00028440
POL00028440
IN CONFIDENCE
PART 3: DETAILED AUDIT FINDINGS
Implementation Teams
3.1 ICL Pathway are responsible for managing the implementation process. Post Office
Network (PON) implementation team are responsible for supporting offices during the
migration phase. The Post Office team also monitor the implementation process, expedite
any rectification work which is required and assist in resolving any conflicts between the
implementation process and operational requirements.
3.2 The central implementation team were in the process of establishing a database of issues to
facilitate central monitoring and identify any areas of concern. Previously the team had
relied on regular meetings with regionally based teams. The information on the database
will be required to establish responsibility for additional costs if implementation has not
been achieved at all offices by the contractual date.
3.3 We visited the local implementation teams based in Bristol and St Albans. The Bristol
team was co- located with the ICL Pathway implementation team and had no problems
resolving issues. The St Albans team were not co-located. The outstanding issues list, for
this team, had a total of 631 offices with issues to be resolved from the infrastructure
phase. They resolved snagging issues with the local ICL Pathway team but had
experienced difficulty in getting the central ICL Pathway implementation team based at
Kidsgrove to resolve other issues. A meeting had been arranged to try and improve the
processes for resolving these issues. The team was taking the opportunity provided by the
pause in roll-out to clear the outstanding problems.
The main concerns raised by implementation teams are discussed in the following
paragraphs.
Support for Horizon Field Support Officers (HFSOs)
3.4 Concern was expressed at the quality of the Helpdesk support provided to HFSOs. The
main concems were the time taken to resolve problems and conflicting advice being
provided by both the Horizon System Helpdesk (HSH) and the Network Business Service
Centre (NBSC). The need to develop further scripts to enable both Helpdesks to provide
prompt and accurate information had been recorded as a problem.
REPORT REF: AR11699 PAGE 5 OF 18
POL00028440
POL00028440
SS
IN CONFIDENCE
3.5 HFSOs were not being provided with operational change updates in a timely manner to
ensure that the level of instruction given to Post Offices’ managers was accurate. We were
informed that there was no co-ordinated approach to the communication of updates of
reference data, network, system and procedural changes. This resulted in incorrect or
inaccurate information being passed to outlet managers. This issue had not been formally
recorded as a problem.
3.6 HFSOs expressed concern at the quality of the training they received from Knowledge
-Pool. In particular, the training relating to cash account completion was considered to be
inadequate. We are aware that HFSO training requirements have been addressed with a
oom
new training course being. developed for new HFSOs and refresher training being
provided for existing HFSOs.
Support for Offices
3.7 The main concerns of the Implementation Team were:
. inadequate training for outlet managers and, in particular, insufficient training on
completion of the cash account, which is a known problem;
no
. inadequate support to outlet managers following migration. This situation is being
addressed by the proposed changes to the role of Retail Network Managers.
axa
3.8 Concer was raised as to the level of helpdesk support provided to outlet managers. The
main concems revolved around the time taken to resolve problems and conflicting advice
being provided by both the Horizon System Helpdesk and the Network Business Service
Centre.
Transaction Processing (TP) / TIP
3.9 The volume of errors generated by Horizon offices was a cause for concem. Initially
Horizon offices generated twice as many errors as manual offices, The pause in roll-out had
resulted in a reduction in the level of errors generated as outlet managers become familiar
with the new system. The latest information indicated that the level of errors will retum to
pre-Horizon levels after 13 weeks. We are informed that TP have based their budget bid for
next year on the assumption that an additional 20-35 staff would be required to handle the
anticipated volume of errors.
REPORT REF: AR11699 PAGE 6 OF 18
POL00028440
POL00028440
D> I
IN CONFIDENCE
Y 3.10 A number of issues were identified during the course of the review and these are detailed
below:
© numerous manual amendments were being made to supporting documents and to the
manual cash account which were not being reflected on the Horizon balance;
© negative figures appeared on the cash account;
© Receipt and Payments tables were not always equal on migration week and during
subsequent weeks;
¢ an official complaint had been received from the Benefits Agency in relation to the
format of reporting for Pension and Allowances to Lisahally;
© stock centralised into Hemel Hempstead Secure Stock Office (SSO) at the same time
as the office migration. A large increase in QPA errors has resulted due to mispostings
into “Remittance to/from other offices” rather than the Supplies Division ;
reversal problems particularly in respect of Automated Payments transactions;
© anumber of outlet managers were incorrectly operating cut off procedures in respect of
(iad
daily summaries and Lottery transactions;
<=
.
the process for accounting for out of hours transactions including Automated Payments
and Lottery business;
the effect on Counters Business Database and settlement of two week cash accounts
(extended cash accounts);
contra entries in respect of Giro rent scheme transactions;
© problems were identified during weeks 29 and 30 in respect of internal transfers.
Subsequent fixes initiated by ICL Pathway were not communicated to the Incident
Team and had led to numerous unresolved errors during the weeks in question;
e differences had been identified between transaction details received from ICL Pathway
and cash account information received. The problem was being dealt with as
Acceptance Incident 376.
All issues identified have been reported as problems and appropriate action is being taken
to resolve them.
REPORT REF: AR11699 PAGE 7 OF 18
POL00028440
POL00028440
IN CONFIDENCE
Post Offices
3.11 Overall Internal Audit found that all outlet managers saw potential benefits in the system.
This was supported by the fact that none of the managers interviewed would wish to
revert back to manual operations. Counter serving time was perceived to be slower than
the manual system.
3.12 The main areas of concem identified at offices were:
© the training provided was considered to be inadequate and was identified as the one
overriding factor why problems are experienced in the early weeks following migration
of the system;
¢ the level of support received, from the Helpdesk functions was considered to be
unsatisfactory. The majority of outlet managers were confused as to which Helpdesk to
contact in respect of problems they experienced and suggested they would benefit from
a single Helpdesk function to resolve any problems or queries;
¢ the number of systems crashes and screen freezes that were experienced, particularly
following the completion of Automated Product transactions. A number of outlet
managers raised concerns surrounding internal transfers during weeks 29 and 30. In
most instances, however, problems experienced in those particular weeks had been
satisfactorily resolved. It should be noted that outlet managers believed that the number
and frequency of systems faults were improving and were not negatively impacting
upon office operations.
© completion of the weekly cash account was taking significantly longer than under the
manual system. Out of the 21 offices visited during the review, 13 offices indicated that
the process was slower or significantly slower than before. Information received from
outlet managers indicated that balancing time had increased by 70% when compared to
the previous manual system. The majority of offices highlighted printer problems being
the main cause for the slowness of cash account production. A further issue raised was
that the process of rolling over was very time consuming.
© 62% of outlet managers visited were unaware of the procedure for posting items to, or
removing items correctly from, the suspense accounts. 24% of outlet managers
indicated that they had not received any instructions or training on how to account for
Suspense account entries.
REPORT REF: AR11699 PAGE 8 OF 18
== OO cD
I
— Ge im
POL00028440
POL00028440
IN CONFIDENCE
All of these issues had been reported as problems.
We recommend that:
Recommendation 1 (Low Risk)
A process should be developed, as part of business as usual, for improving procedures at
outlets and for identifying and communicating good practice.
Network Business Support Centre (NBSC)
3.13 At the time of our visit NBSC had spare capacity due to the pause in roll-out. Plans were
in place to recruit additional staff to cope with the anticipated increase in calls from 24
January 2000.
3.14 The main problem identified was the fact that it was not possible for the first line helpline
to answer the targeted level of calls from the scripts currently available. The average
number of calls answered by the first line was 60% against a target of 75%. If calls which
had been misdirected to NBSC are excluded, however, then the proportion of calls
answered by the first line was approximately 43%.
3.15 I We were informed that there was a process for updating the scripts and that a 1,000
amendments had already been made. We were informed, however, that there were still
some 700 amendments to be input to the NBSC toolset.
3.16 Approximately 50% of calls received by NBSC were actually for the Horizon System
Helpline (HSH). Outlet managers have been used to calling the regionally operated
helpline, so they now have a choice of 3 possible numbers to call. We recognise that some
information has already been issued regarding the types of calls handled by each helpline,
but we consider that a co-ordinated approach is required which involves HFSOs and
RNMs as well as communications via publications such as Counter News.
All of these issues have been reported as problems.
3.17 The NBSC Centre Manager also expressed concern about the timeliness of information
supplied by PON business units. NBSC need to be aware of changes to products, the
network, system processes and procedures in advance of the change occurring.
REPORT REF: AR11699 PAGE 9 OF 18
Ga Oe fs Sm me «5ER
3.18
3.19
3.20
POL00028440
POL00028440
IN CONFIDENCE
We recommend that:
Recommendation 2 (Low Risk)
Formal procedures and timescales should be established for informing relevant parties
such as the Helpdesks, HFSOs, RNMs and offices of changes to products, the network and
systems processes and procedures, with appropriate monitoring systems to ensure
compliance.
Problem Management
The roles and responsibilities of BSM and the Horizon Project Team were not clearly
understood by other business units. In particular there was uncertainty over the
responsibilities of the problem management team. This uncertainty is understandable in
view of the fact that the Horizon Project team maintain an involvement with the
acceptance process and the management of acceptance incidents.
We recommend that:
Recommendation 3 (Low Risk)
A communications exercise should be undertaken to inform business units of the
respective roles and responsibilities of BSM and the Horizon Project Team.
We reviewed the procedures for identifying and recording problems. The process for
identifying problems was based on information passed to incident analysts from first line
supervisors and those issues passed to the second line which had not been cleared within
the agreed timescales. We were satisfied that all the issues raised with us by offices and
TP had been recorded as problems.
We are concemed that a number of problems had been outstanding for several months.
One-reason for the delay was that a symptom can have a number of underlying causes and
it may be difficult to find a problem controller who has the knowledge to deal with all the
underlying causes. For example, a policy decision was required on how to account for out
of hours transactions. This may have'an impact on security, operational procedures, and
the generation of error notices in TP and client accounting. PON Security were originally
allocated as the problem controller for this problem.
REPORT REF: AR11699 PAGE 10 OF 18
POL00028440
POL00028440
rere cers
IN CONFIDENCE
3.21 I We recognise that there was no department within Post Office Counters to make policy
decisions relating to outlets and that links were being established with the Operations
Directorate established within Post Office Network. We understand that this issue has now
been passed to this Directorate.
3.22. We are also concemed that problem controllers are not always aware of the appropriate
procedures for resolving issues . One example of the problems encountered relates to the
issue of access to the Horizon database by PON Security. The problem controller had
approached various managers within ICL Pathway, the Horizon Project Team and BSM.
On each occasion a different answer was received.
We recommend that:
Recommendation 4 (Medium Risk)
Guidelines should be provided to problem controllers on the procedures for resolving
problems including:
e — the need to establish the impact of the problem on other departments;
* — procedures for liaison with other departments, including assessing the impact of the
problem for each department, reporting progress in resolving the problem, and
assessing the effectiveness and impact of agreed changes;
mi = Om aw ES OR
* agreed methods of resolving problems and escalating them where agreement cannot
be reached at a local level;
e the respective responsibilities of the BSM problem management team and the
Horizon project team.
3.23 Reporting of unresolved problems focuses on the target date for completion of any agreed
rectification.
ain
We recommend that:
Recommendation 5 (Low Risk)
Reports of unresolved problems should show:
e — the date the problem was initially raised;
e the original target date for correction of the problem;
REPORT REF: AR11699 PAGE 11 OF 18
POL00028440
POL00028440
.
IN CONFIDENCE
¢ the latest date for completion.
This will highlight problems where the target date is constantly being deferred.
The main areas of concern to the problem management team were:
« the failure to adhere to the product development process resulting in changes being
Notified too late to allow the necessary changes to be made to reference data;
¢ — inaccurate data being supplied by the reference data team.
These issues have been reported as problems.
Rin Mmmm ER 0S &= SE &s ee
— mm
REPORT REF: AR11699 PAGE 12 OF 18
POL00028440
POL00028440
mn Mim HB Ia f= Baw HN Ge at SS ee Oe Oe
IN CONFIDENCE
PART 4: IMPLEMENTATION PLAN
RECOMMENDATIONS AGREED/ COMMENT RESPONSIBILITY
NOT AGREED
Recommendation 1 (Low Risk)
A process should be developed, as part of business as usual, for
improving procedures at outlets and for identifying and
communicating good practice.
It is not agreed that
BSM is responsible for
this action.
We will discuss this issue with other sections
within the Operations Directorate and with the
Horizon project team to determine the
responsibility for identifying and
communicating good practice.
Head of Network
Business Support
Centre
Recommendation 2 (Low Risk)
Formal procedures and timescales should be established for
Agreed ( for the
The OBC Change Process is designed to link
Head of Business
informing relevant parties such as the Helpdesks, HFSOs, automated environment I to ICL Pathway and ensure system / processes I Environment
RNMs and offices of changes to products, the network and only ) are changed before the operational process is
systems processes and procedures, with appropriate monitoring required.
systems to ensure compliance.
Recommendation 3 (Low Risk)
A communications exercise should be undertaken to inform Agreed Ongoing. BSM have a detailed Head of BSM
business units of the respective roles and responsibilities of
BSM and the Horizon Project Team.
communications plan which has been put
together by Alison Leavesley and the BSM.
team. Various articles have already appeared
to promote the roles and responsibilities of
BSM.
REPORT REF: AR11699
” PAGE 13 OF 18
POL00028440
POL00028440
mm BR aw ER 0S ae Bow HE in ma 6 oD 0 Be
IN CONFIDENCE
RECOMMENDATIONS AGREED/ COMMENT RESPONSIBILITY
NOT AGREED
Recommendation 4 (Medium Risk)
Guidelines should be provided to problem controllers on the Agreed Problem co-ordinators via their training / Head of Service
procedures for resolving problems including: workshops and WLA ( Working Level Operations
Agreements ) have had their roles and
¢ — the need to establish the impact of the problem on responsibilities and procedures for resolving
other departments; problems fully explained to them.
¢ procedures for liaison with other departments,
including assessing the impact of the problem for cach
department, reporting progress in resolving the problem,
and assessing the effectiveness and impact of agreed
changes;
¢ — agreed methods of resolving problems and escalating,
them where agreement cannot be reached at a local level;
¢ the respective responsibilities of the BSM problem
management team and the Horizon project team.
Recommendation 5 (Low Risk)
Reports of unresolved problems should show: Agreed Implemented in the Problem Management Head of Service
Status Report Operations
e the date the problem was initially raised;
e the original target date for correction of the problem;
e the latest date for completion.
REPORT REF: AR11699
PAGE 14 OF 18
aa Ia DP Oe ee le)
a Oo
ain
rt
—
POL00028440
POL00028440
IN CONFIDENCE
APPENDIX A: TERMS OF REFERENCE
Objectives
To confirm that effective processes are in place to allow the Horizon project team to monitor and
report:
¢ the progress of implementation and any issues arising from the implementation programme;
« whether ICL Pathway are meeting the standard of system performance specified in the contract;
© the problems identified from operating the live system;
© progress in resolving outstanding problems.
Background
PA Consultants have been asked to review the Horizon Programmes plans for managing the roll-
out of the CSR system to all offices. To avoid unnecessary duplication, the scope of the POIA
review will be reduced in this area and restricted to discussion of the procedures with key
individuals in order to examine the reporting of progress and problems to management.
We are aware that processes already in place have identified a large number of improvement
opportunities and that the agreed approach to resolving there issues is by means of a number of
working parties. The effectiveness of the solutions devised by the working parties will be tested at
a later date.
Scope of the Audit
The review will cover the following areas:
Implementation
We will discuss the progress of implementation and problems encountered with the Horizon
implementation team and a sample of regional implementation teams.
Outlets
We will visit a small sample of outlets in order to identify:
¢ problems arising from the implementation;
© problems with the operation of the system;
© problems with accounting procedures;
REPORT REF: AR11699 PAGE 15 OF 18
—_— Mmmm Ee iS ee ee ee OE
im
POL00028440
POL00028440
IN CONFIDENCE
* confirm that procedures for reporting incidents and problems are correctly deployed.
The visits will be co-ordinated with the Network Audit Team.
TIP
Review the processes for reconciling transaction details with cash account summaries, reporting
discrepancies to ICL Pathway and the Horizon project team, and ensuring that discrepancies are
explained and corrected.
Transaction Processing
Review the procedures for comparison of cash account and supporting documents and
investigation and correction of errors. Understand the underlying reason for these errors occurring.
Ensure that issues relating to the Horizon system, the system training or operating instructions are
identified, reported and appropriate corrective action is taken.
Service Management
Review the procedures for:
¢ development and deployment of scripts for the first line helpdesk;
recording and evaluating incidents;
© identifying and reporting problems;
¢ monitoring progress in resolving problems;
© monitoring the performance of the system and the helpdesks.
Horizon Project Team
Review the process for managing the resolution of problems and for reporting progress.
Reporting
A draft report will be provided to the Director of Automation and the Director of Operations by
24th December. Action on any recommendations will be agreed with the appropriate managers by
1gth January. A final report will be issued by 25th January 2000.
REPORT REF: AR11699 PAGE 16 OF 18
POL00028440
— -
POL00028440
oo om
Mh hn Mmm AW Ge aes EE ee DS Oe eS ee
IN CONFIDENCE
APPENDIX B: AUDIT OPINION DEFINITION
STANDARDS
GOOD
SATISFACTORY
UNSATISFACTORY
REPORT REF: AR11699
no control weaknesses falling into the category of
high or medium risk as defined in the definition of risk
statement (Appendix C);
existence of a number of weaknesses in medium or
low risk categories which will not give rise to material
error;
existence of 1-2 high risk weaknesses in a confined
area of the review which would result in an opinion of
satisfactory, but unsatisfactory in the specific problem
area;
existence of 1-2 or more high risk weaknesses;
existence of a number of medium level weaknesses,
which when taken together could result in
significant potential risk to the business.
PAGE 17 OF 18
POL00028440
POL00028440
IN CONFIDENCE
APPENDIX C: DEFINITION OF RISK ASSESSMENT
The recommendations in this report include an indication of level of risk attached to each
weakness being addressed. The criteria used for assessment of these weaknesses are set out below:
HIGH
Management should progress action immediately because one or more of the following apply:
© divergence from Business Strategy, Objectives or Policies;
© potential for, or actual material losses of Business assets, including reputation and data;
© potential for, or actual major disruption occurring either internally within the Business or
I externally by customers/clients;
I © potential for or actual non-compliance with legislation.
1 MEDIUM
Management should progress action within the short term because one or more of the following
apply:
tin
potential for, or actual significant disruption occurring either internally within the Business or
rb I
.
externally by customers/clients;
© poor value for money or inappropriate use of Business resources;
I © level of risk identified is less than the cost of control;
i management information not timely or accurate which could result in inappropriate Business
decisions being taken.
LOW
in
Management action is desirable within the short to medium term because one or more of the
rit)
following apply:
© potential for, or actual limited losses occurring which exceeds the ongoing cost of control;
¢ potential for benefits from improving efficiency or resource utilisation.
REPORT REF: AR11699 PAGE 18 OF 18