POL00448752
POL00448752
Post Office Limited - Document Classification: STRICTLY CONFIDENTIAL
POST OFFICE LIMITED
Title: HIJ Assurance Desk Top Review Report Date: I 16" August 2024
Author: Sundeep Mehta, Head of Group Sponsor: Anshu Mathur, Group Assurance Director
Assurance
Background
As part of the Historical Matters Assurance plan, subsequently renamed ‘Legacy Matters Assurance Plan’
approved by the General Executive in September 2022, Group Assurance (GA) commenced an assurance
review on the remediation status of the Horizon Issue Judgements (HIJ) in October 2022.
This review, which was risk based, was subsequently paused in March 2023", as several artefacts, self-
assessments (and related GA preliminary concerns) were in the process of being updated/submitted by the
Horizon Technology Team (HTT). In addition, HIJ remediation was also subject to the oversight of the then
reconstituted IDG2.0, of which GA was a member.
In November 2023, HTT updated IDG2.0 and provided a detailed self-assessment of the status of HlJ
remediations. IDG2.0 agreed that GA should perform an objective desk top review based on the information
presented by HTT and evidence they provide to support their assessment.
HTT have broken down the 15 HlJ findings into 47 sub deliverables (please refer to Appendix 1), of which ten
are duplicates (Appendix 2). Our report provides an opinion on the status of 37 sub deliverables.
Approach
The GA review in September 2022 adopted a risk based / sustainable approach and hence was broader (similar
to the GA approach adopted and completed for ClJ) in contrast to our current approach which is a desk top
review, and therefore significantly contingent on the artefacts and quality of documentation provided by the
HTT. The desk top review was performed during March 2024 to July 2024.
GA Opinion
From the data and artefacts provided, by HTT, it is overtly evident that a significant amount of work, effort and
remediation(s) have occurred. That said, at the time of reporting and based on the evidence provided by HTT,
the status of the 37 HlJ deliverables is as follows:
e Fully met: 5
In GA opinion clear evidence has been provided to support HTT assertions. (Refer to Appendix 3)
e¢ Met, with some residual exposure: 7
In GA opinion, whilst the principle of the commitment has been adhered to, some residual exposure may
exist, that may warrant additional action/support. (Refer to Appendix 4)
« Not met: 25
In GA opinion, evidence provided does not fully support HTT assertion on the status of completion. This
category can be further broken into three categories (Refer to Appendix 5):
o No evidence: 5
o Evidence not deemed sufficient: 17
o Considered as work in progress: 3
In our opinion, the above status may not represent the actual state of conformance, i.e. the overall position
might be better. Our view is predicated on the artefacts and evidence provided by HTT which were not
sufficiently complete, nor structured or provided in a logical manner to support the desk top review. These
concerns were conveyed and discussed with HTT in April and May 2024, however due to restricted capacity
within HTT, we were requested to continue our review.
2 Audit, Risk & Committee was informed in July 2023.
POL00448752
POL00448752
Post Office Limited - Document Classification: STRICTLY CONFIDENTIAL
Appendix 1 - Horizon Issues Judgements
HU Sub- Detail
No HU
4 1-14 Defects caused apparent or alleged discrepancies or shortfalls relating to Sub Postmasters branch accounts or transactions.
2 Sub Postmasters were not informed about identified defects. Some defects were not identified by automatic system check and as a result, lay undiscovered for
years.
3 Legacy Horizon and HNG-X were not remotely robust, as identified by the number of defects found. The lack of records or logs for the use of powerful access
roles also contributes to this.
4 15-26 Data errors, arising from data entry, transfer, or processing in both legacy Horizon and HNG-x led to financial discrepancies. Errors in reference data, and 3
party data contributed to discrepancies in branch accounts.
5 If data in Horizon was wrong (due to defects) and did not reconcile to third party data manual Transaction Corrections were undertaken.
6 Legacy Horizon and HNG-X measures and controls did not prevent, identify, or report or reduce a) data entry errors b) data packet or system level errors c)
software coding errors or bugs d) transmission, replication, and storage of transactional record data errors e) data stored in the central data centre not being an
accurate record of transactions on branch terminal.
7 *27-36 I POL and Fujitsu could both access transaction data recorded by Horizon remotely (i.e. not from within a branch).
8 41-47 POL was plainly reliant upon Fujitsu for diagnosis of whether the occurrence of shortfalls was caused by defects within Horizon.
9 “*37-40 I Postmasters had limited access to reports and data and limited knowledge regarding POL’s complex back-end systems. Their ability to investigate
discrepancies was therefore equally limited.
10 *27-36 I Fujitsu had the ability to insert, inject, edit, or delete transaction data or data in branch accounts without the knowledge or consent of Postmasters.
11 Permission controls upon the use of the remote access facility were considered inadequate. Whilst existing, the roles very wide and not controlled, including but
not limited to the lack of any proper logs.
12 Post Office and Fujitsu were unable to provide a clear and precise answer to what is, in essence, a very simple question, “How often was the remote and
privileged access facility utilised by POL and Fujitsu?”. This inability directly arises from Fujitsu's plainly inadequate records.
13 The design abilities of those with privileged access rights (i.e, APPSUP role) were very wide. Therefore, such facilities had the potential to affect the reliability of
a PM's branch accounts to a material extent.
14 ™37-40 I A Postmaster cannot dispute a discrepancy, or any figure on Horizon, or record on Horizon that they have raised a dispute.
15 Over 100,000 manual Transaction Corrections (TCs) have been issued each year since 2006 yet POL does not have comprehensive records of how many have
been challenged by Postmasters. Transaction Acknowledgments (TAs) are also used to make corrections to branch accounts, but a Postmaster has no option
but to accept them.
Source: HIJ Action Owners 210922 V0.1 pdf
HU Assurance Review Page 2 of 13
Draft v3.0
Post Office Limited - Document Classification: STRICTLY CONFIDENTIAL
Appendix 2 - Duplicate or linked Lines
Title HIJ Commitments Comments
Reference Data Management Duplicate HIJ013
Higo22 Enhanced processes & controls for managing reference data.
AppDynamics Duplicate HIJ011 & HIJO010
HIgo24 Rolled out solution to 4836 counters following pilot of 107 counters & 20 servers.
Service Now See HIJ014
HIJ032 Over 200 IT Controls consolidated and moved into ServiceNow.
Continuous Improvement Duplicate HIJ014
HIJ036 Ongoing/continuous improvement based on reporting & controls attestation.
Defect Tracking Duplicate HiJ002
Hlv041 Defects logged and tracked in Service Now until resolution.
Defect Review Forum Duplicate HiJ003
HIJ042 Bi-weekly session run by POL with Suppliers and stakeholders to monitor defects status.
Defect Criticality Duplicate HIlJ004
H1y-043 POL stakeholders assess defect impact to prioritise urgency for resolution.
Defect Comms Duplicate HiJ005
HIg044 Issue Branch Hub & Knowledge Article within 2-day SLA once defect raised
Enhanced Testing & Release Duplicate HIJ018
ods Management Improved governance and rigour in how we carry out Horizon testing. More regular
releases and detailed regression testing introduced.
AppDynamics Duplicate HIJ011 & HIJ010
HI046 Rolled out solution to 4836 counters following pilot of 107 counters & 20 servers.
POL00448752
POL00448752
Post Office Limited - Document Classification: STRICTLY CONFIDENTIAL
Appendix 3 - Fully met: 5
Title
HIJ Commitments.
HIJO14
Continuous Improvement
Ongoing/continuous improvement based on
reporting & controls attestation.
HIJ017
Horizon Recovery Scripts
Implemented improvements to 64 Recovery
Scripts helping ensure discrepancies do not
occur in branches when transactions fail to
complete successfully on the counter.
HIJ031
AppSup/Break Glass Process
Process revised with greater emphasis on
PM communication and rigour around audit.
HIJ033
Controls Training
POL IT Control Owners trained on their role
and how to use ServiceNow.
HIJ037
Review & Dispute Buttons
Introduced to allow PM to create a dispute
and have a discrepancy investigated.
Horizon Tech
Team Self-
Assessment
GA opinion
Comments
POL00448752
POL00448752
Post Office Limited - Document Classification: STRICTLY CONFIDENTIAL
Appendix 4 - Met, with some residual exposure: 7
Horizon Tech
Team Self-
Assessment
Title HIJ Commitments.
Historical Defects
Closed backlog of historical defects KELs
(Know Error Defects).
HIJ001
Defect Tracking
Defects logged and tracked in ServiceNow
until resolution.
HIJ002
GA opinion
Comments
Based on an external review performed by KPMG
(KPMG Report June 2021-Horizon Review Report V4.2
FINAL pdf) itis evident from their report that 62 KELs
existed of which:
- 45 were closed.
- 14 required testing; and
- 3 were undetermined.
- Balance remaining 11 requiring testing.
(Completeness of KELs therefore demonstrated via the
KPMG report).
The 11 requiring testing are supported by a Test Closure
report (POL Test Closure Report-Historical KELS v0.6
15/06/21) which indicates all 11 KELs passed their
testing.
GAP:
1) POL Test Closure Report-Historical KELS v0.6
15/06/21 remains unsigned by the POL Horizon
IT Director and POL Head of Postmaster
Experience.
2) Further to the above The Historical KELs
Determination and Closure Report v0.5 dated
28/06/2021 has also not been signed off by the
POL Horizon IT Director, POL Head of QA and
POL Head of Postmaster Experience.
3) Underlying supporting data does not
necessarily convey in a simple, complete, and
aligned manner the same conclusion therefore,
GA opinion heavily contingent on KPMP report.
4) Governance applied is unclear.
Defects are logged and tracked within ServiceNow.
GAP:
However, the Horizon Problems Monthly Report - April
2024.pptx (sharepoint.com) indicates several SLA's (5 of
7) not being met such as:
« Branch notified within 5 days of identification.
* Knowledge Article created within 5 days of
identification.
© — Criticality scored within 5 days of initial
investigation conclusion.
POL00448752
POL00448752
Post Office Limited - Document Classification: STRICTLY CONFIDENTIAL
Title
Horizon Tech
Team Self-
Assessment
HIJ Commitments
HIJ004
Defect Criticality
POL stakeholders assess defect impact to
prioritise urgency for resolution.
HIJ010
AppDynamics
Completed pilot on 107 counters and 20
Horizon servers.
HIJ011
AppDynamics
Rolled out solution to 4836 counters
following pilot.
HIJ013
Reference Data Management
Enhanced processes & controls for
managing reference data.
GA opinion
Comments
* Create impact statement within 5 days of initial
investigation conclusion.
* Meeting held to discuss problem within 5 days of
identification.
The report is dated April 2024 and does not provide
trending or relative benchmarks. (Linked to HIJ005).
Defects criticality is defined and assessed, including
prioritisation carried out accordingly.
Potential Exposure: From the evidence, it is unclear
how the criticality score is reached, how the weighting
has been agreed, and who are the stakeholders that
reach the decision and whether this is recorded
anywhere.
The PowerPoint-Post Office & AppDynamics Full Stack
Observability - Technical Success Criteria 24th May
2022- clearly shows a Pilot was run and what the
success criteria were. However, it is not evident from the
PPT that the "Pilot covered 107 counters and 20 Horizon
Servers".
A change request form provided indicates Pilot to
include 110. And further the Pilot close email indicates
113 Counters. Change Request File Name:
CRQ000000350826 and Email AppDynamics
Closure.msg.
Whilst it is clear the deployment occurred; the evidence
provided, comprises multiple folders (8 folders, 2 have
12 subfolders and 5 contain 14 files), which are not
knitted together to provide or support an objective
opinion on whether a deployment covered 4836
Counters and outcomes. Linked to HIJ010.
The evidence provided clearly indicates that
considerable diligence was performed to assess status
and changes needed for reference data in the KPMG
Report Horizon AP-ADC Scripts and Reference Data
Solution 2021. In addition, the PowerPoint- Horizon
Improvements Programme-Reference Data Logical
Model, highlights several rulesets were created for
Reference data. That said it is challenging to assess how
this was embedded in a consistent and standardised
manner and the governance applied to the KPMG
recommendations.
POL00448752
POL00448752
Post Office Limited - Document Classification: STRICTLY CONFIDENTIAL
Horizon Tech :
Title HIJ Commitments Team Self- ~ GAopinion Comments
Application Modernisation Documentation provided to date does not support the
Migration of 300 file transfer routes to Completeness and Accuracy (audit trail) around the 300
Hly021 modern secure solution. Replacement of the
migrations including the Governance process followed.
voucher authorisation platform with modern
solution within POL domain.
PowerPoint presentation HSA 4” Sept 2023, slide 3
mentions that there are 253 file transfer routes.
POL00448752
POL00448752
Post Office Limited - Document Classification: STRICTLY CONFIDENTIAL
Appendix 5 - Not met: 25
A- No evidence: 5
Title
Horizon Tech
HIJ Commitments Team Self- GA opinion
Comments
Assessment
HIJ023
Transaction Correction Management
Improved analysis of all transactions for risk
of discrepancies.
HIJ034
Controls Adoption Surveys
POL IT Control Owners Adoption Survey on
new process and training — positive
feedback received.
HIJ035,
Independent Audit
To validate the effectiveness of the
remediation approach.
HIJ038
Discrepancy Management
Improved how discrepancies are handled
and investigated including upskilling
Investigation Support Teams and refreshing
our case management processes.
HIJ039
TC Volumes
Actively managed, trend analysis completed,
currently showing a 26% decreasing the
volume of TCs disputed.
POL00448752
POL00448752
Post Office Limited - Document Classification: STRICTLY CONFIDENTIAL
Appendix 5 - Not met: 25
B - Evidence not deemed sufficient: 17
Horizon Tech
Title HIJ Commitments Team Self- GA opinion Comments
Assessment
Defect Review Forum It is evident that a Defect Review Forum (DRF) exists
Bi-Weekly sessions run by POL with which is attended by Fujitsu and POL (monthly and
suppliers and stakeholders to monitor defect weekly) to monitor Defects status [(Horizon Problems
status. Monthly Report - April 2024.pptx (sharepoint.com) and
minutes Horizon Defects Forum- Meeting Notes
040324.xis),]
Rupes That said the evidence provided indicates that the DRF is
held monthly/weekly only i.e. no evidence to support BI-
Weekly sessions. In addition, the agenda is neither
captured or overtly articulated in the monthly
presentation and/or minutes of the meeting.
Further evidence required to demonstrate the
embedding of these meetings.
Defect Comms itis evident that, defects Comms are sent out however
Issue Branch Hub & Knowledge Article these are not in line with 2 days SLA as per the Horizon
within 2 days SLA once defect raised. Problems Monthly Report - April 2024.pptx
(sharepoint.com). Please also refer to HIJ002 above.
HIJOO5
NB: Operational KPIs on an annual basis not provided as
evidence, therefore unable to access conformance
sustainability of complying with PM Comms and
Knowledge article SLAs. Linked to HIJ002.
Defect Definitions The POL Test Strategy document D09-019 defines the
Consistently identify system defects by Defect Management processes within Section 13 Page
H1006 severity and business impact. 49. Including the Severity Definitions and Priority
Definitions (business impact). Linked to Hid 008.
Defect Triage itis evident that a Defect Review Forum (DRF) exists
Bi-Weekly meeting led by POL with Fujitsu to attend Fujitsu and POL (monthly and weekly) to oversee
discuss all system defects by severity. and monitor system defects.
HIJ007
That said the evidence provided indicates that the DRF is
held monthly/weekly only i.e. no evidence to support BI-
Weekly sessions.
Defect Thresholds. Itis clear from the POL Test Strategy document and the
Common set of limits on accepted defect Regression Test Cycle report that limits on accepted
HIJ008. volumes at each stage of testing process. defects volumes do exist and are followed (based on a
sample of one).
POL00448752
POL00448752
Post Office Limited - Document Classification: STRICTLY CONFIDENTIAL
Title
HIJ Commitments.
HIJ009
Defect Tooling
JIRA software tool used across POL to
capture defects during testing.
HIJ012
Horizon Solution Authority
Agrees Horizon change with focus on
Postmaster impact.
Horizon Tech
Team Self- GA opinion Comments
Assessment
Evidence provided highlights the following Gaps:
14) POL Test Strategy document does not indicate an
approval date; document review dates are all TBC;
Section 8 Test Activities Summaries is work in
progress; Section 11.1.8 Operational Acceptance
Testing is TBC; Section 14.3 is missing; Section 16
Test Roles & Responsibilities is TBC.
2) HSI Test Strategy V.04 (Old)- this still clearly a work
in progress (no approval, outstanding POL
comments) hence unclear how this supports HIJ
commitment.
3) The Regression Test Cycle report (71.20-
Regression Test Cycle 1 Daily Metrics Report
20220304) it is unclear why certain tests are not
‘executed and why this is not highlighted in the
Executive Summary.
Whilst both test strategy documents are comprehensive,
outlining the overview of the strategy to be employed for
testing, the significant gaps above could be interpreted
that neither has been formalised, completed nor signed
off.
Evidence provided does support business self-
assessment.
Technology provided 4 documents to support their
status. From these documents it is clear that a Horizon
Solution Authority (HSA) existed however the
information provided lacks data/insight on what was
brought to HSA, what changes were approved/rejected
over a period of time and consequently GA cannot
support technologies assessment.
10
POL00448752
POL00448752
Post Office Limited - Document Classification: STRICTLY CONFIDENTIAL
Horizon Tech
Team Self-
Assessment
Title HIJ Commitments
Horizon Solution Review
Engaged Postmasters to produce definitive
list of Horizon issues with 147 User & Data
Journeys developed and 39 system
HIJO15 improvements.
Horizon Improvements
Implemented Horizon improvements across
1. Lottery, 2. Drop & Go, 3. E-Top Up and 4.
Fast Cash, helping reduce discrepancies,
and mitigate risk of Postmaster detriment.
HIJO16.
Horizon UX/ UI
Streamlined look and feel of counter screens
by removing 21 redundant product buttons.
making it simpler for Postmasters to
navigate Horizon.
Enhanced Testing & Release
Management
Improved governance and rigour in how we.
carry out Horizon testing. More regular
releases and detailed regression testing
introduced.
Securing Banking Data
Implementation of a PCI compliant Payment
& Banking solution to better manage
banking transaction data
HIJ018
HIJ019
HIJ020
New Audit Solution
By February 2025 we will design, tested, and
implemented a new Audit solution for
Horizon enabling greater transparency on
the transmission etc of transaction records.
HIJ025
Security Reporting
More robust and frequent Horizon Data
Centre Security Reporting E.g. Privileged
Accounts, Remote Access, Appsup, Failed
logins, Break glass.
HIJ027
GA opinion
11
Comments
Evidence provided whilst clearly showing significant
analysis and work being done to identify Problem
Statements (see Master Sprint Plan and Backlog
Prioritisation.xls) however these do not clearly link back
to the statement made i.e. “147 User and Data Journeys
developed and 39 system improvements”. In addition,
the lineage of Problem Statements to current state is
unclear including governance applied there to.
Evidence provided far exceeds the 4 topics mentioned
for Horizon Improvements.
Whilst it is very clear a number of Horizon Improvements
have been completed, with a number In Progress, it is
not possible to opine on, based on the evidence
provided, what impact the changes have had on
reducing discrepancies and Postmaster Detriment.
PowerPoint Horizon Programme Update IT Huddle
111021 Final Copy mentions the removal of 9 buttons
including the movement of 2 buttons.
Linked to HIJ008 and concerns highlighted.
However, it is clear policies and procedures have been
introduced to enhance the Cadence around testing and
regression testing through the support provided.
PCI Programme Manager email confirmation provided as
evidence only.
Whist it is clear a project has been initiated. However,
there is no evidence which supports the current status of
the programme. We have only received the status up to
November 2023.
Whist it is evident that the reporting and governance
have been set up. The evidence provided does not
contain Completeness and Accuracy checks, Trend
analysis (YOY, Level of Access, Type of Access etc) to
demonstrate the embedding of reporting and
governance and the corrective/preventative actions
taken by operations.
POL00448752
POL00448752
Post Office Limited -
Document Classification: STRICTLY CONFIDENTIAL
Horizon Tech
Title HIJ Commitments Team Self- GA opinion
Assessment
Global, SMART ID, PAM Account Reviews
« Reviewed 250 Horizon Global ID
back office and 90,000 SMART ID
HIJ028 accounts.
* Reduced Privileged Access
accounts on counter.
Cloud Security
Embedded robust ‘JML (Joiners, Movers and
HIJ029 Leavers)’ and User Recertification processes
for Horizon applications in POL Cloud which
Fujitsu support.
External Benchmarking
«Performed Penetration Testing on
Model Office and Data Centre.
* Conducted annual independent
Horizon Security Audit.
HIJ030
12
Comments
Evidence provided does support business self-
assessment. The PowerPoint presentation V2.0 IADM
Process Improvements D07-015, provides a status as of
9" March 2022 however does not provide a view or
status of 250 Horizon Global ID back office and 90,000
SMART ID Accounts and reduced Privileged Access
Accounts on counter.
Whilst the evidence shows new processes have been
designed. That said it is unclear what MI, KPIs and
Governance is being applied.
itis clear that Penetration Testing reviews have been
conducted for Model Office and Data Centre however no
evidence received for the Independent Horizon Security
Audit.
For the two Penetration Testing Reports, no latest status
on how findings have been provided.
Potential Exposure:
1.For Penetration Testing Technical Report 23" March
2022 V1.1, Target Model Office, noted 5 findings, 1 rated
High, 1 Low and 3 Information.
2. For Penetration Testing Technical Report 9" March
2022 V1.0, Target Infrastructure, noted 14 findings:
-1 Critical Finding
-2 High findings
-8 Medium
-3 Low.
POL00448752
POL00448752
Post Office Limited - Document Classification: STRICTLY CONFIDENTIAL
Appendix 5 - Not met: 25
C - Considered as Work in progress/Ongoing: 3
New Routing Se
We will have also completed the
migration of over a 1000 file transfers to
the new routing service.
HIJ026
Branch Accounting
The branch accounting improvements
HIJ040 proposed as part of HSI will allow greater
visibility of individual discrepancies to
Post Masters.
POC
We need to build on the POC for the
capture of counter activity logs in order to.
make this data more readily available.
HIJ047
The email HlJ Assurance-App Mod.msg dated 8th March 2024,
mentions that 36 routes have been migrated however unclear
how many files this relates to. The narrative mentions that if all
routes are migrated this could be up to 1000 files.
However, within the Migration Readiness Checklist (1) VO.1xIs
it is a challenge to understand the completeness of the 36
routes completed and the number of files, including how the
1000 files transfer is achievable.
Whilst progress has been made, the main commitment to give
great visibility of individual discrepancies to Postmaster has
been ‘Paused' therefore commitment remains open/WIP..
“PoC created but not yet progressed” are management
comments to date.
Email support provided dated September 23 and PowerPoint
(ppt)-CBA Logging POC v0.3 for update June 23 presentation.
Email mentions slides require update however, no update ppt
provided for latest status.
The commitment has been progressed however the latest
status and plans going forward unclear.
13
POL00448752
POL00448752