POL00337672
POL00337672
POST OFFICE LIMITED
Board REPORT
a Horizon Issues Judgment (HIJ) . .
Title: Phase 3 Remediation update Meeting Date: I 17 January 2024
. I Simon Oldnall, GLO & Horizon IT . Chris Brocklesby, Chief
AEDES: Director Sponeniss Transformation Officer
Input Sought: Noting
Executive Summary
This paper reiterates the focus and approach taken to remediation of the findings of the Horizon
Issues Judgement (HIJ). An update is provided against the five themes set out in the January
2023 paper to provide the Board with a view on progress.
The current programme will close upon the completion of the Phase 3 scope. Additional activity
will be taken forward either through specific individual projects or via the Technology business
as usual function.
The work has been ongoing since late 2020, with a total expected cost of £26.2m by end of
2023. Two of the five HIJ themes are expected to be fully addressed before Phases 6 and 7 of
the Inquiry in 2024.
The remaining areas - management of core Horizon data, management of discrepancies and
shortfalls and the reliance on Fujitsu to identify defects and the causes of shortfalls in branch
accounts- require additional efforts. This is to ensure we have addressed criticisms in the
Judgement to the fullest extent possible (within the confines of the legacy platform and
operational constraints).
The status of our remediation against the five themes is set out below.
THEME: Remediation Status Ongoing Actions
Activities complete and I N/A
Management of Horizon Defects sustained In BALI
1.Completion of Audit
Management of Core Horizon Data (incl. SAN project
Reference and Third-Party) In progress Horecn System
Improvements
Management of Privileged and Remote Access Activities complete and I N/A
to Branch Accounts sustained in BAU
1.BSC call trend
Management of Discrepancies and Shortfalls In progress analysis
2.Horizon System
improvements
Reliance on Fujitsu to Identify Defects and 1.Horizon event log
Causes of Shortfalls in Branch Accounts In progress extraction and retrieval
Confidential
POL00337672
POL00337672
Against the broader, ‘fit-for-purpose’ question, based on the high level of transaction volumes,
and very low level of system downtime there is evidence that the system itself is reasonably
robust. Albeit, there are no established industry metrics to measure against.
The last phase of the ongoing Horizon Remediation Programme has further reinforced that
conclusion by implementing tactical ‘fixes’ and addressing, to the extent feasible, the previously
unaddressed areas of Data Management. These include Reference Data, AP-ADC script controls
and the creation of modern data audit capabilities (Audit SAN).
Report
1. What are the findings in the Horizon Issues Judgment (HIJ)?
The Horizon Issues Judgment (HIJ) is comprised of 15 specific findings, enumerated by the
Judge in his verdict. These represent “findings of fact” about how the Horizon system operated
at the time, with a specific focus on the generation, detection and management of discrepancies
and shortfalls in Postmaster’s branch accounts.
Appendix A presents the 15 findings, which can be grouped into 5 themes. Given findings are
interlinked inside a theme, each theme must be addressed collectively:
THEME: FINDINGS #:
Management of Horizon Defects 1; 2,3
Management of Core Horizon Data (incl. Reference and
Third-Party) 4,5,6
Management of Privileged and Remote Access to Branch 7, 10, 11, 12, 13
Accounts
Management of Discrepancies and Shortfalls 9, 14, 15
Reliance on Fujitsu to Identify Defects and Causes of 8
Shortfalls in Branch Accounts
Confidential
POL00337672
POL00337672
@
2. What has Post Office’s strategy been to address the HIJ findings?
Although there is no requirement and prescribed test for “legal compliance” or “legal
conformance” with the HIJ findings, the focus of Post Office’s effort has still been on ensuring
the five themes and all 15 findings are addressed. To that end, POL set out a dedicated
Programme, which launched in 2020.
The third phase of this Programme is now nearing completion, (the initial Phase Zero of the
programme delivered a number of diagnostic reports) and has delivered a significant number
of Horizon System Improvements and other changes to strengthen the overall management of
the platform. A summary of these deliverables is shown in Appendix [B]. The final deliverable
(an audit of our incident management processes as they relate to Horizon) will be completed
by the end of February.
As the main Programme closes we will transition any remaining activities to the business as
usual (BAU) Technology function and continue to invest where appropriate to deliver a small
number of Horizon system improvements, this will be done mindful of the need to provide a
stable functional baseline for NBIT. This reflects our intention to sustain the changes already
made via the Remediation Programme whilst improving Horizon in order to ensure we provide
a system that is fit for purpose.
3. What is the current remediation status for the five HIJ themes?
We have made substantial progress against all five themes and whilst additional activities
remain in three of the themes, we have put in place significant changes across all five. The key
themes of Defect Management & the management of Privileged/Remote access have been
remediated to their fullest extent and now actively managed by the BAU Horizon and cyber
functions.
3.1 Management of Horizon Defects
The HIJ found that bugs, errors or defects had the potential to, and did, cause apparent or
alleged discrepancies or shortfalls relating to Postmasters’ branch accounts or transactions.
Bugs, errors or defects therefore undermined the reliability of Horizon to accurately process
and record transactions.
Our activities in this area give us confidence that we have remediated the concerns raised by
the judgement and we have implemented a number of significant changes:
e All 62 historical defects referenced in the HIJ have been fully re-tested to ensure they
are not present in the current version of Horizon. The results of this testing have been
independently audited and the results shared with the National Federation of Sub
Postmasters (NFSP).
e We have established robust governance to proactively track and resolve Horizon defects,
with the process now led and owned by POL. This includes a level of transparency
whereby we notify Postmasters and other stakeholders (including the NFSP) of any defect
we identify, either through testing or via reported incidents. This revised process was
independently assured by KPMG in Phase 1 of the Remediation Programme
e Robust and formal identification, categorisation and remediation testing processes have
been established. These changes ensure POL is proactively identifying and resolving
3
Confidential
POL00337672
POL00337672
@
Horizon defects at an early stage, thus reducing any impact on Postmaster branch
operations. New testing identifies on average some 4 new defects each month before
these reach the live Production environment. In an important step change, this activity
is Post Office led, while historically Fujitsu had predominantly owned this activity.
e¢ Our Monitoring Solution (detailed below) provides an independent view of Horizon health
and enables us to proactively identify when system performance alters, potentially
indicating the existence of a bug or issue.
3.2 Management of Core Horizon Data (incl. Reference and Third-Party)
The Judgement found that there was a material risk for errors in data recorded within Horizon
to arise in (a) data entry (b) transfer or processing of data in Horizon in both the Legacy Horizon
and HNG-X forms.
Whilst some work continues in this area, we have made substantial changes that remediate a
number of the risks raised in this theme from the judgement:
e We have reviewed 147 Horizon Product journeys to identify where data entry (mis-
keyed) could be a factor in causing discrepancies. From this analysis we have identified
380 areas where this is a possibility.
« These 380 pain points were then distilled down into 170 problem statements to be
addressed (either through system changes, improved training, additional
communications and so forth). These problem statements were reviewed by an
Improvement Delivery Group (IDG) sub-group and the Horizon Design Review Forum
(HDRF) in 2022. This resulted in a backlog of 39 potential system improvements being
approved for more detailed analysis.
These system improvements were prioritised based on inputs from both Retail and Commercial
teams, together with advice from Norton Rose Fulbright (NRF) on the risk of potential
Postmaster detriment. This resulted in a selection of 11 system improvements for delivery in
the final phase of the Remediation Programme. Given the revised NBIT timeline, work is now
underway to revisit this original list and re-evaluate whether any of the original pain points
should now be considered for inclusion in ongoing Horizon improvement activity.
The definition of the scope of system improvements was made after careful consideration from
the different parts of the Post Office business. The decision reflects the application of four
prioritisation factors:
« Impact on Postmasters - issues with significant Postmaster impact were prioritised
for early resolution.
e Value for money - issues that are costly to resolve in Horizon were deprioritised
unless they majorly impact Postmasters.
e Minimal disruption to Postmasters - issues, the resolution of which would likely
temporarily disrupt the Postmaster operation, were deprioritised (e.g. user interface
changes).
Confidential
POL00337672
POL00337672
@
Ability to address via New Branch IT (NBIT) - issues that could be more effectively
resolved via NBIT instead of on Horizon were deprioritised unless they significantly impacted
Postmasters
In looking to address data transfer, POL’s wider Technology transformation have implemented
and continue to implement changes to how data is transferred from and to Horizon.
These include:
e The implementation of a Payment Card Industry (PCI) compliant Payment & Banking
solution to better manage banking transactions.
e The procurement and implementation of a commercial off the shelf file transfer
platform. We’re now in the process of migrating our legacy file transfer routes to this
solution and will be moving more than 300 file transfer routes that service more than
1000 individual file transfers across Horizon. This work will complete by October 2024
e The replacement of the reference data driven APOP voucher authorisation platform.
This provides the ability to make pay outs to, for example, energy customers.
To mitigate risks around the use of Reference Data, we have additionally implemented
changes to our processes & controls for managing this key aspect of the Horizon platform.
These include:
e Clear end to end Change Request processes for both Reference Data and AP-ADC
changes. This is managed via Service Now with clear accountability for sign offs.
e Transparency of request visibility enabling assurance around changes.
e Automation and auditability improvements leveraging our service management tooling.
Whilst we have not carried out any specific testing of the replication and storage of data through
the Horizon Remediation Programme, activities are underway which will provide additional
confidence in this element of the platform.
These include the design and creation of a new Audit store for Horizon enabling the storage of
audit data for all branch transactions to be within POL for the first time. This further means the
process to retrieve Audit data (ARQ) will now be executed by POL (historically this has been
wholly managed by Fujitsu).
3.3. Management of Privileged and Remote Access to Branch Accounts
The Judgement found that Fujitsu had the ability to insert, inject, edit or delete transaction
data or data in branch accounts, implement fixes in Horizon to affect transaction data or data
in branch accounts, or rebuild branch transaction data without the knowledge or consent of
Postmasters.
It further found that permission controls upon the use of the remote access facility were
considered inadequate. Whilst existing, the roles were very wide and not controlled,
including but not limited to, the lack of any proper logs.
The activities and changes made relating to this theme give us a much greater oversight on
the use of the functionality as well as greater level of transparency to Postmasters on the
reasons for its use in their individual branches.
Confidential
POL00337672
POL00337672
@
The following work has been completed in this area:
e« Implemented a revised process for the use of Elevated/Privileged access with a greater
emphasis on Postmaster communication (permission and explanation for use is
sought/given in every instance).
e Implemented more robust and frequent Horizon Data Centre reporting via the
Information Security Management Forum, the Technology Sub-Committee, GE and
ultimately Board.
e Reduced the level of Privileged access at the counter for Fujitsu staff to the lowest
possible level (effectively reducing the ability to amend data and provide read only
access)
e« Implemented over 200 IT controls into our Control framework (these are managed and
audited via the ServiceNow enabled Controls process)
These activities have been independently assured via our audit partner to validate the
effectiveness of our remediation approach.
3.4 Management of Discrepancies & Shortfalls
In this area, the Judgement found that Postmasters had limited access to reports and data,
as well as a limited knowledge regarding POL’s complex back-end systems. Therefore,
Postmasters ability to investigate apparent or alleged branch shortfalls and discrepancies
were equally limited.
Ultimately, Postmasters required the cooperation of POL to help investigate discrepancies, a
theme that was also identified from the Common Issues Judgment (CIJ). The Judgement also
found that a Postmaster could not dispute a discrepancy in their branch accounts, or any
individual figure recorded in their branch accounts on Horizon. Horizon also did not have the
capability for a Postmaster to record on Horizon that they had raised a dispute of this type.
In addition, POL did not have comprehensive records of where transaction corrections (TCs)
had been challenged and the challenge upheld.
To address this finding, a number of activities have been undertaken to analyse the cause for
issuing TCs over the last 3 years. As a result, there is now ongoing tracking and monitoring of
the volume and cause of TCs issued as well as tracking of TCs that have been disputed.
« The implementation of the ‘Review or Dispute’ button on Horizon has enabled Postmasters
to dispute a discrepancy and have it investigated.
e« The overall trend remains that the vast majority of TCs issued (78% in P8) are
attributable to either cash handling errors (56%) and Lottery/Camelot (22%). We expect
the removal of lottery products and the implementation of automated stock remittance to
reduce/remove these TC volumes.
e This leaves a percentage that can be attributed to other errors, such as reconciliation
errors and these are closely monitored.
Confidential
POL00337672
POL00337672
@
e It should be noted that the number of TCs disputed through the Review and Dispute
process is currently trending down (-26% between P7/P8). These volumes are monitored
and reported monthly to GE/Board.
e We will be introducing additional steps to ensure that any report into the Branch Support
Centre (BSC) of a discrepancy on Horizon is compared in more detail to known Horizon
Defects and trend analysis carried out by the Horizon service team to identify potential
defects.
Whilst we have made some changes to Horizon, with the implementation of a ‘Review and
Dispute’ button, owing to increased technical complexity and cost we have been unable to
create the Branch Reporting Suite tool that was originally planned for Phase 3.
This additional Horizon functionality would have more fully addressed the findings of the
Judgement (for example removing the need for the use of paper till rolls).
3.5 Reliance on Fujitsu to Identify Defects and Causes of Shortfalls in Branch
Accounts
The Judgement found that POL had access to data and systems that were not available to
Postmasters via a range of Horizon reports and POL's own management information
systems.
The only source of actual key stroke information (which buttons had been pressed in branch)
was found within Fujitsu audit data.
We have made good progress in remediating the findings in this area. As highlighted above,
enhanced testing and Defect Management have reduced our dependency on Fujitsu and
enabled POL to have a greater level of insight into defects.
In addition, following a successful pilot we have extended the use of the App Dynamics
tooling to approximately 5,000 counters in the estate. This gives us a representative sample
of the overall estate from which to derive appropriate alerting to more widespread issues.
These deployments have provided POL with a range of capabilities including the ability to
monitor network performance, transaction processing speed and the interactions between the
counters in branch and the Horizon Data Centre.
This enhanced monitoring will provide us with foundational abilities to move into predictive
monitoring and to address problems before they impact branches or potentially cause
discrepancies.
We have additionally created a Proof of Concept that enables POL to extract the log files
created at the counter. These files record interactions between elements of the counter (such
as the keyboard or pin pad) and whilst these are not ‘key logging’ they do provide valuable
insights into the potential causes of discrepancies. Additional work is required to make this
Proof of Concept more readily usable to support our Investigation processes.
Financial Impact
Confidential
POL00337672
POL00337672
@
The table below reflects the total estimated lifetime spend on the Horizon Remediation
Programme and the expected further spend in calendar 2023.
Actuals Latest Forecast
. 3Year Approved
Horizon Issues Judgement £'000 FY22 FY22
Plan Drawdown
FY20 FY21 PO1-PO9 P10-P12 FY23 Total
Phase 1 3,057 3,460 6,517 6,517 6,517
Phase 2 5,147 5,147 5,147 5,147
Phase 3 3,051 2,080 1,060 6,191 3,400 6,256
Phase 3 - Extended Scope 8,300 8,300
Total HIJ Programme 3,057 I 8,607 I 3,051 I 2,080 I 9,360 I 26,155 I 15,064 17,920
Next Steps & Timelines
The current Remediation Programme will close at the end of Phase 3. The work on ARQ Audit
SAN & Automated Stock Remittance will be managed via discrete projects.
Ongoing Horizon system improvements are being evaluated and further funding for these
sought as appropriate.
Remaining activity will be taken forward either through specific individual projects or via the
Technology business as usual function.
Confidential
POL00337672
POL00337672
Appendix A - Horizon Issues Judgment findings
I a)
Defects caused apparent or alleged discrepancies or shortfalls relating to Sub PMs branch accounts or transactions
‘Sub PMs were not informed about identified defects. Some defects were not identified by automatic system check and as a result lay undiscovered for years.
Legacy Hz and HNG-X were not remotely robust as demonstrated by the number of defects found
Data errors arising from data entry, transfer or processing in both Legacy Hz and HNG-X led to financial discrepancies. Errorsin reference data and 3° party data contributed
to discrepancies in branch accts.
Transaction data reconciliation with 3° party data leading to manual corrective fixes and transactional corrections
Legacy Hz and HNG-X measures & controls did not prevent, Identify or report or reduce a) data entry errors, b) data packet or sys level errors, c) software coding errors or
bugs, d) transmission, replication & storage of transactional record data errors, e) data stored in the central data centre not being an accurate record of transactions entered
‘on branch terminal
Remote access by POL and 3" parties
Availability of information to POL and reliance on 3" parties
PM access to information and transparency over the investigation process
Fl Access to and editing transactions / branch accounts for PMs
Permission Controls upon the use of the remote access facility were considered inadequate
How often was the remote and privileged access facility utilised by POL and F1?
‘AppSupp privileged access rights were very wide and had the potential to affect the reliability of PM branch accounts
Ability for PM to dispute a discrepancy via Hz
(Over 100,000 Transaction corrections issued since 2006; POL does not have comprehensive records on how many have been challenged. TAs used to correct branch
accounts with no opportunity to challenge
Confidential
POL00337672
POL00337672
Appendix B - 2023 Deliverables
Horizon Remediation: What have we delivered during 2023?
Asummary of Horizon improvements which have been rolled out across the network this year is provided below.
7 cole eat @ cenit > eure alata
I Stocoswamio "I ewntafaRotoerre Bury I Srsfonsitiandsris I er eee
Reems os
fay 4) I Rey eormarne
eee ee oe e ©
© o 8 68 is
op Up cons) © Recover Se Sears faa Pa one
ae fetoat ptt
Torvpwowom — z4ming Hoon, ston wen 2 ances Freed Ponmanar wit ier pe counter Sreone rd
——— recovering transections ‘customer's account p-up message in event Seepigrageenie
a ten ed Sethe
Sacrentons etree
Pescicessaey
Confidential
10