CASE NUMBER: HQ16X01238, HQ17X02637 & HQ17X04248 IN
THE HIGH COURT OF JUSTICE QUEEN'S BENCH DIVISION
BETWEEN:
ALAN BATES & OTHERS
AND
POST OFFICE LIMITED
FUJ00082162
FUJ00082162
CLAIMANT
DEFENDANT
EXPERT REPORT OF JASON COYNE
16 OCTOBER 2018
OCCUPATION: PARTNER
SPECIALIST FIELD : IT SYSTEMS
ASSISTED BY: SIOBHAN FORSTER
CHRISTOPHER RASKE
JAMIE SMITH
PATRICK GRANT
ON THE INSTRUCTIONS OF: FREETHS LLP.
ON BEHALF OF: ALAN BATES & OTHERS
Jason Coyne
IT Group UK Limited
Unit 5 Lockside Office Park
Lockside Road
Preston
PR2 2YS
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 i of xii
Page
Table of Contents
Table of Annexes
vii List of Acronyms and Abbreviations
1. Introduction ..
BUSINESS SCOPE ....eeeececceeceee eee eeee een ee nee eee een see ee eee eeee eee eeeseeeeaeeneseeenee eee ennes
1
Agreed List of Issues ..
3
Documents ReVICWEd .....cccceceeeeeeeee ee eeeee eee e teen tesa eee eres sees eeeeee ne eee eeea ne eee eae
7
Author
9
Declaration of Independence 02... eeeceeceeeeee settee eee eee ee eeee eee eeneeeeeeeeeee ens
9
2. Background
103. Executive Summary .
124. History of Horizon ...
18
Known Chronology Mile@StOnes...........cccceeeeee eee eee eee eee ee eeee eee eeneeeeeeeeeeaenees
18
Horizon (pre-Horizon Online) .
21
EFTPoS - Electronic Funds Transfer at Point Of Sale. ......::sccccessseeeeeeseeeeee
21
The Introduction of the Network Banking Solution ............ccccceee cece neces eeeeee
21
Horizon COMPONENtS ....eeeccccceeeeeeeeeeeeeeeeeeaeeeeeeseeeeeee ese eeeeeeeeaeeeeeeeeaeeewees
23
Prepared by: Jason Coyne
Occupation: Partne YJ an
Specialist Field: IT Systems d Jt OU
On the Instructions of: Freeths LLP
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page ii of xii
Reference Data
23
TFANSACLION DACA veeeieeeeee eet eee EEE eee e eee nnn eerie
24
The Horizon Counter (1996 - 2010) ..
25
Migration towards Horizon Online (Horizon Online 2010 - Present) ..............
26
The Branch Database .
28
Legacy Horizon / Horizon Online Counter Processes .........ccccseeseeeeeeeeeeee eee es
31
Recovery
31
REVEFSAIS se eeeeccccennecceceneeeeeee een eee cess eneeeee es neeeeeeee aan eeeess aaa eeessaaeeeeseanens
32
Horizon Support Service & Facilities ..
33
The Role of Each Support LeVEl uo... .ccccceccsseceneeeseeeeeeeneeesneesneeesnesseeesnne
33
Support Services Provision i 2OOL wccscceccsccecceccnecnsceseeecnscesseseesserseeenses
37
Page
Support Services ProviSion i 2O1LO wieciececsseceeseeeneenseneneeeneeeeeeeeeeseeen een ene
37
Support Services ProviSion from 2014 w.ccccecccseecsseccceecseeessneceesseeseneess
37
Incident Tracking SYSt@IMS w.ccccccceccsscceseecseeceseecseeesenesseeeesunesseessueeseneees
38
Known Error LOGS (KELS)...ccccccccccsecsseeccnecceecceneeeueeseneseeeeeueseneseeneeeuees
38
Release MANAGEMENL,....cccecceccncceeeeeeescnrseeeesserseeeneenseesensesseseesserseeen een
39
Prepared by: Jason Coyne
Occupation: Partne S — .
Specialist Field: IT Systems jt OU
On the Instructions of: Freeths LLP ri
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 ili of xii
Fault PEAK Lifecycle .
40
Assignment of PEAK Incidents to Maintenance R@lCASES ........6cceeeceeeee eee
41
5. Horizon Bugs/Errors Defects and Controls ........sscesseeeeseeneeeeseeneee 42,
Known Errors/Bugs Defects acknowledged by Post Office .........:..ceeeeeeeee eee 43
Calendar Square / Falkirk ISSUC@*? ......cicececeseeseeeseeeaeeeesennaeeeessanaetesenanes
44
Payments MiSMAtCH viieeescccesccseeeesecseneeeeneeeeeseeeseeseseeseneeeneeeneesenneeanes
44
SUSPENSE ACCOUNT BUG viccicccsseccceecsnececeecseeeeseessenesenessueessunesseessuneseneees
45
Further Bugs / Errors / Defects not acknowledged by Post Office ................ 45
DAIMEIINGtON —icececcccecccsncccnececneceuecseeeceeseeseesseeeeeseesueeseneeesseesesseneeesens
45
Cash Declarations - Cash Management ...cccccccccsecccseeceeecceeeeeecesseneeenees
46
Bureau & Decimal POINt ISSUC ...scecccssseseceseceeseteseesanereseesseesssseneeteseenees
49
Reference Data Errors secccsccessssscceceeeesscceeeeesessseceseceessssseseseeensenseeeeees
49
Duplicat€ TramSA@ctiOnS ..cccccccccessccecceeessecceeeeecessseceseceessssseseseeesseeseeeeee®
51
Failed RECOVESICS weeseccccssecececenteseeccssnneeeeesneeeeseasaeeeessseeneesssaaeneeesesnens
51
Failed — REVELSAIS.....ccccccesececcceeneeeecsseneeeetenneeeeeseesaneeesssaaeeesssaaeneeeseanens
53
Uncategorised BUgS/Errors/D@feCtS oo. .ececeeceeeteetece eee eeeeee eee eeneeeeneeen een enne
55
Hardware EFrOrs wevseeessssseeeeesseeeneneeeeeeeeeseeeeeeseeseeeseseeeseeseeeseeeeeseeeseeeees
57
Fujitsu Closed Problem ReCOrS ...cccccseseeseececeeeeeecesseeeessnssersesessnsensensnes
61
HoriZOn RODUSENESS .......csceeeeee ee eeeeeee cena eee e teen eee cece ea eee eee saan eEeeeeea nee EeeHaaES 63
Prepared by: Jason Coyne
Occupation: Partne fo} eae
Specialist Field: IT Systems J JFOU
On the Instructions of: Freeths LLP ri
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page iv of xii
Horizon Uptime
68
Likelihood of Shortfall if DrANCh@S ....scccccccseeseceseeseecesseneeeesssseeseseseenens
70
Extent of errors in data recorded within Horizon arising from (a) data entry,
(b) transfer or (c) processing Of data iM HOFiZON .i..csicseeccceeeesneeeneeeeneeenees
70
Errors that are Potentially the Result of Multiple ISSUCS ........c1ccceseeeeseeneee
71
In relation to a) Data Entry
73
In relation to b) Transfer (Of data) .....cccccccecceseecsseeceneesseessenesseeeseeeseneess
77 Uncategorised Horizon Concerns
82
Measures and Controls to prevent/detect/identify/report/reduce errors in
Horizon
82
Failures of measures and controls to prevent/detect/identify/report/reduce
ChE riSK Of CLTOFS wieececccessecccecenteeeesesneeeeeeseeeeesessaneeesssaaeesssssaeeeesesanes
86
Opinion Summary
94
6. Reconciliation and Transaction Corrections
Horizon ArChiteCtUre oo. eeeeeeeeccceceeeeeeee esse eee ee eee eeeeee see eeeeeseaeeeeeeesaneeeeeaes
97
External Systems/ClientS .......c ccc eee eee eee eee eee eee esse een een es eeeeeeeeenees
98
Transaction Acknowledgements .........ccceecseeeeseeese cess eeeseeeesueeeeeeeuneeeeeeeuee
100
Reconciliation Processes
100
Prepared by: Jason Coyne
Occupation: Partne ° ey
Specialist Field: IT Systems d JFOU
On the Instructions of: Freeths LLP ° i
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 v of xii
Reconciliation Services/Departments
102
POL Finance
102
Fujitsu Management Support Unit (MSU) .
102
Fujitsu Third Line Support (SSC)
102
Reconciliation Exceptions
103
Data Reconciliation Service (DRS)
104
Transaction Disputes
106
Reconciliation Summary
106
Transaction Corrections
107
Successful processing of a Transaction Correction
112
Unsuccessful processing of @ Transaction COrreCtiONn ......::c:ccsseeeeseeneeeneee
112
Branch Trading Stat€mMent ....cccceccccsscseececceceeeescesseeeessesseesnsseeseeeeseesnne
113
Disputing @ Transaction COrrectiOn ...cccccccccccceesscsseceessssssesesseesseseeeeees
113
Transaction Correction ObservationS ANd FINdiINGS .....:ssccsccessseeeeserseeen eee
114
OPiNiON SUMUIMALY — veceecesseseeeseesseesececeeceeeeeceececeseceeececeeeeeeeeseceeeaseeeeeeeas
116
7. Horizon Reporting - Facilities for Subpostmasters .........cccceeeee 118
Horizon Alerts for Subpostmasters in respect of bugs/errors/defects
118 Bugs Errors Defects not alerted to the Subpostmaster
Prepared by: Jason Coyne
Occupation: Partne fo} SEAL IT
Specialist Field: IT Systems J jt! OUP
On the Instructions of: Freeths LLP ri
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page vi of xii
How does Horizon enable Subpostmasters to compare the stock and cash in a
branch against the stock and cash indicated in Horizon? ...........csceeseeeeeeeee
122 Daily CaSh D@CIAratiOn .....sccecesecseseee nec nee ee neeneseeeeeeeeseeseeeseenesnesnnsneen nes
122
Weekly Balan viceciccccsecssecccseccenecceeeeeesecseescueseeeeseeeseseeesneseneessneeesees
123
Monthly Trading Period ROIOVEF ..ceccccsssesseceseseeseceseeneessessenaseetsseeseseees
123 How does Horizon enable or require Subpostmasters to decide how to deal
with, dispute, accept or make good an alleged discrepancy by (i) providing his
or her own personal funds or (ii) settling centrally?
cee eee ee eeeeeeaneeeaeeeaeeeeneeeneeee 123 record and reflect the consequence of
raising a dispute on an alleged discrepancy, on Horizon Branch account data
and, in particular: ... . 124 does raising a dispute with the Helpline
cause a block to be placed on the value of an alleged shortfall;
ANG. eee eeeeeeee cece ee teeeee eee eeeeeeeeeeeeeeeeessa ee eeeesaneeeeees 124 is that recorded on
the Horizon system as a debt due to Post Office? ......... 125 enable
Subpostmasters to produce (i) Cash Account before 2005 and (ii) Branch
Trading Statement after 2005? .......ciccecseeeeseee eee eee eee eeeeeee een eeeeeneeeeeneeee
125 enable or require Subpostmasters to continue to trade if they did not
complete a Branch Trading Statement; and, if so, on what basis and with what
consequences ON the HOriZON SYStEM? .....ceccccceeseeesseeeeeeeeeeeeeeeeeeeneeeee sees
125
OPINION SUMMASY oo. ceeceecscceeceee nee eee eee eee eee ee esse eee sense eeeeeee sense nessa einen eeen
125
8. Horizon Shortfalls, Data and Reporting for Subpostmasters and Post
OFFIC] —saceneccnsecenseeneeenncecenscnseecenneenseeenseeeusecneeeeaseeeaeecaneeeueeeuaeseaseneeesenses
127
Data and Reporting Functions for Post Off1C@ .......cececceeceeeeeeeeeeeeeeeeeeeenees
127
Official SOUrceS Of INFOFMAtiON vicecccsesssecsseseseeeseesseeeesseeueeessseneeetsseseees
128
Additional Sources Of INFOrmation .......:scccccssseeceeeseeeeeeeessaeesesesseneneesnes
129
Prepared by: Jason Coyne
Occupation: Partne
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page vii of xii
Conclusions Relating to Information Available to Post Office ..
130
Data and Reporting Functions for Subpostmasters .........:cccseceeseeeeeeeeeeene es
130
Issues 8 & 9 - Summary of Conclusions .
133
9. Remote Access and Alteration of Transaction Data .
REMOLE ACCESS reeeeececerneee eee EE EEE EEE EEE EE
135
Branch Transaction Data Rebuilds
139 Transaction Correction Tools - Modification of Transaction Data
139
EffectS ON Branch ACCOUNEING wececsccsseseeeeeeeeseeeeeesseenesseeseeeneseeneeeneenenee
143
Supporting Evidence of remote access and implemented fixes affecting
CFAMSACCION ATA eeeecececsecececentteeccceeeneeeeeeseneeeseesseeeesssaaeessssneneeessenees
146 Permission Controls and Data Auditing
dete eee ee ee ett een eeeseeeeeteeneeeteseenenenes 146
Balancing / Corrective TranS@CtiOnS ...cccceccsesseseceeeeeeeeeeeeeeeeeeeeneeeeeenenee
147
AUGit SOPVETS ceeeeeseeeseeeessseessetsstteesseseeeeeeeeeeeeeeeeseeseeeeeeeeesseeeeeeeeeeeeseas
149
PrOCESS VATIACIONS wieeecessesececennteeneeenneeeeeenenneeesee na eeeeen aa eeeessn na netesenan es
149
OPINION SUMMASY oo. ceceeceeceeeeee nee e ee eeeeee eee ee eeeeeea eee eeeeeeee senna neeeeenteneeeeen
150
10. Expert Declaration.......cccccceecseesseneeseseeeeeeesenseetsesseseeseessneeeseesense LOZ
Statement of Truth
154
11. Appendix A.
PEAKS of initial interest .........cccceeseecccee settee cess eeeeeeee eee eeeeeeesaeeeeeesaeeeeeenes
155
Prepared by: Jason Coyne
Occupation: Partne . er) IF
Specialist Field: IT Systems d JFOUf
On the Instructions of: Freeths LLP ° i
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page viii of xii
Errors with Financial Impact ...........ccccccecseee cece eeeee sess cess esse eeseeeeeeeeewees
155
PC0027887 (1999)
155
PCOO63227 (2001) rivcccsscccecccssecenecscnecceneeeneccuseseueeseeeseesesueeeesssneeesens
160
PCOO63723 (2001) vececseccescessesessecssesssccsssecssscssssecsussussscssesuseesssesseisees
162
PC0098844 (2004)
163
PCO2ZO3131 (2010) vececcescescessescsscessessescsscsussucssssscssseussscssesssescssessessees
169
PCO203676 (2010) vececcececscesseseeseessesesesscscssessssscsssessssessesssesssessessees
171
PCOZ63451 (2017) rireeccscccsccccneceecsenecceneeseesseseeeeeeseesseeeessneeseessenneenees
173
PCOZ66575 (2018) acececssseccccrvsecccseeneeessenenseesseesueesesseeueessssensetssessees
175
PCO2Z73046 (2018) ricccccssceccccrvneecccscvneeesseceneeessessneesessenneeesssenneerssensees
176
12. Appendix Bu scecccsscccscccnscccsccnsscnceccnsscccsccnssccsscesscucsscenscnesssansconssse
178
Horizon Architecture Diagrams ........cccecccceeeeeeeeseeeeeeeessseeeseeeeesseeeeeeees
178
13. Appendix C ..
186
2012 — 2013 TCS wieiecccccccccccsceccceeeseeeese esse eeeseeseeeeesuesteeeeseeeeeeesaneeen eee
186
14, APPeNdix D crsrssccccccesseeeeeensnneeseeeeenseeseasneeeeesnenseeseaaaenesseeaneneensons
187 Log of helpdesk calls regarding Transaction Corrections W/E 15 June
2014 . 187
Prepared by: Jason Coyne
Occupation: Partne
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page ix of xii
15. Appendix E ..
Horizon REPOrtS ......cccceceeeeeeeeseeeeesesseeeeeeeeeeeeeeeeeenssneseessesnineeeeeeennes 188
TPS REPOLES —sassecceccecseecneceeseeenseeeeeee eee ees eee ees eee eeen sense eee ese esneeeese ese een eee
188
APS REPOS ececceccsccnccneceeeneceeseee ees ees eee nee eee eee eee esseeeeseesse esa eseseeeeneegngne
189
Banking & Related Services Reconciliation (DRS Report S@t) ......cccceeceeeee
191
Reconciliation and Incident Ma@naQeMent .....ccccccccsccsseceeeeceseeenesseneeenees
192
The Problem Management Procedure .u.cccecccecsecsseeeseeeeeeeeeeeeeeeeeseeeneenene
194
16. APPeNdix F .....cccccseccneecenecneeeeeeeeeneeeeeeeeeeeeeeeeeuseseeeseuseseeeseeeessenees 196
Closed Problem ReCOrdS .........:ccccseeeeeeeeeeceeeeeeeeeeeeeeeseseaeeeseeesaeeeeeseaneees 196
17. Appendix G .
18. Appendix H ..
Failure Point Diagrams
201
Prepared by: Jason Coyne
Occupation: Partne
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page x of xii
Table of Annexes
Annex A SLA Summary Week Ending 6 July 2014
Annex B Transaction Correction Diagram
Prepared by: Jason Coyne
Occupation: Partne . _
Specialist Field: IT Systems } 4\
On the Instructions of: Freeths LLP ° i
180503R1935
16 October 2018
FUJ00082162
FUJ00082162
xi of xii
List of Acronyms and Abbreviations
AP.
APADC
APOP
APS
ATM
BAL
BAU
BIF
BIM
BRDB
CAPO
CCD
CCTV
CNT
cP
CPU
CRC
cs
CSR+
cTS
CWU
Automated Payments
A scripting mechanism that allows POL to define what
additional data is captured during a transaction
Automated Payments Out Pay
Automated Payments System
Automated Teller Machine
Branch Access Layer
Business as Usual
Business Impact Forum
Business Incident Management
Branch Database
Card Account Post Office
Contact Controlled Document
Closed-circuit Television
Counter
Change Proposal
Central Processing Unit
Cyclic Redundancy Check
[POA’s] Customer Services organisation
Core System Release Plus
Client Transaction Summary
Communication Workers Union
Prepared by: Jason
Occupation: Partne
Specialist Fiald: IT Systems d
Coyne
On the Instructions of: Freeths LLP
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page xii of xii
DBTN Disputed Banking Transaction Notice
DCS Debit Card System
DRS Data Reconciliation Service
DWH Data Warehouse
EBBT Enquiry Based Banking Transaction
EBT Electronic Banking Transactions
EFTPOS Electronic Funds Transfer Point of Sale
EMV Europay MasterCard Visa standard for financial smart
cards
EPOSS Electronic Point of Sale System
EPS Electronic Point of Sale Service
ETU Electronic Top Up
FSC Financial Services Centre
FTMS File Transfer Management System
GPOC Generic Particulars of Claim
HNG-X Horizon Online
HRSAP. The SAP System used by Royal Mail Group’s Human
Resources to pay Subpostmasters
HSD Horizon Service Desk
HSH Horizon System Helpdesk
IBM International Business Machines Corporation is an
American multinational technology company
ICL International Computers Limited
IT Information Technology
JSN Journal Sequence Number
KEL Known Error Log
Prepared by: Jason Coyne
Occupation: Partne ° ey
Specialist Field: IT Systems d JFOU
On the Instructions of: Freeths LLP ~ i
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page xiii of xii
LFS Logistics Feeder Service
MDM Master Data Manager
MI Major Incident
MSC Managed Service Change
MSU Management Support Unit
MTBF Mean Time Before Failures
NAO National Audit Office
NBE Network Banking Engine
NBSC Network Banking Support Centre
NBX Replacement architecture of network after removal of
NBE
NS&I National Savings and Investments
NWB Network Banking
OBCS Order Book Control Service
ocpP Operational Change Proposal
OCR Operational Change Request
OSD ICL Pathways Operation Services Division
OTI Open Tele service Interface
OTT Operation Test Team
OWA Operational Working Agreement
P&BA Product and Branch Accounting
PIN Personal Identification Number
PM Post Master
PO Post Office
Prepared by: Jason Coyne
Occupation: Partne °
Specialist Field: IT Systems ¢
On the Instructions of: Freeths LLP
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page xiv of xii
POA Post Office Account
POCA Post Office Card Account
POCL Post Office Counters Ltd
PODG
Post Office Data Gateway. A generic reference-data
driven system that is used to deliver file-based
information between two end points (internal or
external).
POL Post Office Limited
POLFS Post Office Ltd-Financial System
POLMIS Management Information Service for Post Office Limited
POLSAP Post Office Ltd-Financial System
PONB Post Office’s Network Banking Unit
PTF PEAK Targeting Forum
RDDS Reference Data Distribution System
RDMC Post Office’s Reference Data Management Centre
RDS Reference Data System
SAP-IS SAP Industry Solution
SAPADS
Post Office Ltd’ s Advanced Distribution System (based
on the SAP package) that interfaces to LFS
SCP Supplier Change Proposal
SLA Service Level Agreement
SLT Service Level Target
SMC System Management Centre. Second line support
SPM Subpostmaster
Prepared by: Jason Coyne
Occupation: Partne ° ey
Specialist Field: IT Systems d JFOU
On the Instructions of: Freeths LLP ° i
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page xv of xii
SQL Structured Query Language
SSC System Support Centre. Third Line Support
SSH Secure Shell
TA Transaction Acknowledgement
TDES Triple - Digital Encryption Standard
TEM Tivoli Enterprise Manager
TES Transaction Enquiry Service
TIP
Transaction Information Processing. This is the remote
end-point of the FTMS service that delivers
reconciliations reports to Post Office Limited /
Transaction Information Processing System
TIPAIS Transaction Processing Interface Specification
TPS Transaction Processing Service/System
XML eXtensible Markup Language.
Prepared by: Jason Coyne
Occupation: Partne ie YrOLID
Specialist Field: IT Systems 6 YI OU}
On the Instructions of: Freeths LLP
FUJ00082162
FUJ00082162
1. Introduction
1.1. I have been instructed to prepare this report by Freeths LLP
1.2 My instructions are to produce a report that focuses on the agreed list of
issues as set out in the Schedule to the Court order dated 23 March
2018.
1.3 This report is based as far as reasonably possible on the Model Form
of Expert Report as developed by the Academy of Experts.
1.4 The report seeks to address the issues that concern the Horizon system
(as defined here) and which (a) arise on the parties’ generic
statements of case, (b) can be resolved by IT expert evidence, and (c)
require limited, if any, evidence of fact.*
1.5 Definition:
“the Horizon System” is considered for the purposes of this list of
issues to mean “the Horizon computer system hardware and software,
communications equipment in branch and central data centres where
records of transactions made in branch were processed, as defined in
GPOC at §16 and as admitted by Post Office in its Defence at §37."
1.6 This report provides a review of the Horizon system and its composition
from 1999 to present day.
Business Scope
1.7. The business scope of Horizon is recorded as:?
a. Point of Sale Application
b. Transaction Recording (and Auditing of transactions)
c. Posting Summary Transactions to POLSAP (Post Office Ltd.’s back
end accounting system)
In accordance with the orders given by the Court at the CMC on 22 February 2018
2 Witness Statement of Gareth Jenkins.pdf (Para 2.2), 5 October 2012 [C-0003632]
[b544230cf07249c189cf664fcba6d899]
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page ii of 225
d. Posting Detailed transactions to Credence (Post Office Ltd.’s back
end Management Information system)
e. Posting Remuneration Data to HRSAP (Royal Mail Group’s back
end
Payroll system)
f. Delivering Client Data to Post Office Ltd.’s Clients (i.e. 3rd parties
that Post Office Ltd acts as an agent for)
1.8 Since branch accounts could be affected by data delivered to/from Post
Office back end systems and third-party clients, it is necessary to
consider all of these aspects in order to appropriately opine on issues
which are specific to branch accounts.
1.9 Within this report, the different issues have been grouped together in
order to address similar thematic issues within a single section. I
understand that Dr Worden will adopt a similar approach although he
may select a different method of grouping. The groups I have adopted
are:
a. Section 5 - ‘Horizon Bugs/Errors/Defects and Controls’ (Issues 1,
3, 4 and 6);
b. Section 6 - ‘Reconciliation and Transaction Corrections’ (Issues 5
and 15);
c. Section 7 - ‘Horizon Reporting - Facilities for Subpostmasters’
(Issues 2 and 14);
d. Section 8 - ‘Horizon Shortfalls - Data and Reporting for
Subpostmasters and Post Office’ (Issues 8 and 9); and
e. Section 9 - ‘Remote Access and Transaction Amendments’ (Issues
7,10, 11, 12 and 13).
Prepared by: Jason Coyne
Occupation: Partne ° ey
Specialist Field: IT Systems g tQrou
On the Instructions of: Freeths LLP ~ i
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page iii of 225
1.10 The issues in numerical order are listed below for ease of reference.
Agreed List of Issues
Issue 1
1.11 To what extent was it possible or likely for bugs, errors or defects of the
nature alleged at §§23 and 24 of the GPOC and referred to in §§ 49 to
56 of the Generic Defence to have the potential to (a) cause apparent
or alleged discrepancies or shortfalls relating to Subpostmasters’
branch accounts or transactions, or (b) undermine the reliability of
Horizon accurately to process and to record transactions as alleged at
§24.1 GPOC?
Issue 2
1.12 Did the Horizon IT system itself alert Subpostmasters of such bugs,
errors or defects as described in (1) above and if so how?
Issue 3
1.13 To what extent and in what respects is the Horizon System “robust” and
extremely unlikely to be the cause of shortfalls in branches?
Issue 4
1.14 To what extent has there been potential for errors in data recorded
within Horizon to arise in (a) data entry, (b) transfer or (c) processing
of data in Horizon?
Issue 5
1.15 How, if at all, does the Horizon system itself compare transaction data
recorded by Horizon against transaction data from sources outside of
Horizon?
Issue 6
1.16 To what extent did measures and/or controls that existed in Horizon
prevent, detect, identify, report or reduce to an extremely low level
the risk of the following:
Prepared by: Jason Coyne
Occupation: Partne ° ey
Specialist Field: IT Systems } 41 UU
On the Instructions of: Freeths LLP ~ i
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page iv of 225
a. data entry errors;
b. data packet or system level errors (including data processing,
effecting, and recording the same);
c. a failure to detect, correct and remedy software coding errors or
bugs;
d. errors in the transmission, replication and storage of transaction
record data; and
e. the data stored in the central data centre not being an accurate
record of transactions entered on branch terminals?
Issue 7
1.17 Were Post Office and/or Fujitsu able to access transaction data recorded
by Horizon remotely (i.e. not from within a branch)?
Issue 8
1.18 What transaction data and reporting functions were available through
Horizon to Post Office for identifying the occurrence of alleged
shortfalls and the causes of alleged shortfalls in branches, including
whether they were caused by bugs, errors and/or defects in the
Horizon system?
Issue 9
1.19 At all material times, what transaction data and reporting functions (if
any) were available through Horizon to Subpostmasters for:
a. identifying apparent or alleged discrepancies and_ shortfalls
and/or the causes of the same; and
b. accessing and identifying transactions recorded on Horizon?
Issue 10
Prepared by: Jason Coyne
Occupation: Partne ° ey
Specialist Field: IT Systems d JFOU
On the Instructions of: Freeths LLP ~ i
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page v of 225
1.20 Whether the Defendant and/or Fujitsu have had the ability/facility to: (i)
insert, inject, edit or delete transaction data or data in branch
accounts; (ii) implement fixes in Horizon that had the potential to
affect transaction data or data in branch accounts; or (iii) rebuild
branch transaction data:
a. atall;
b. without the knowledge of the Subpostmaster in question; and
c. without the consent of the Subpostmaster in question?
Issue 11
1.21 If they did, did the Horizon system have any permission controls upon
the use of the above facility, and did the system maintain a log of such
actions and such permission controls?
Issue 12
1.22 If the Defendant and/or Fujitsu did have such ability, how often was that
used, if at all?
Issue 13
1.23 To what extent did use of any such facility have the potential to affect the
reliability of Branches’ accounting positions?
Issue 14
1.24 How (if at all) does the Horizon system and its functionality:
a. enable Subpostmasters to compare the stock and cash in a
branch against the stock and cash indicated on Horizon?
b. enable or require Subpostmasters to decide how to deal with,
dispute, accept or make good an alleged discrepancy by (i)
providing his or her own personal funds or (ii) settling centrally?
Prepared by: Jason Coyne
Occupation: Partne ° ey
Specialist Field: IT Systems d JFOU
On the Instructions of: Freeths LLP ° i
180503R1935
FUJ00082162
FUJ00082162
16 October 2018 Page vi of 225
Issue 15
record and reflect the consequence of raising a dispute on an
alleged discrepancy, on Horizon Branch account data and, in
particular:
does raising a dispute with the Helpline cause a block to be
placed on the value of an alleged shortfall; and
is that recorded on the Horizon system as a debt due to Post
Office?
enable Subpostmasters to produce (i) Cash Account before 2005
and (ii) Branch Trading Statement after 2005?
enable or require Subpostmasters to continue to trade if they did
not complete a Branch Trading Statement; and, if so, on what
basis and with what consequences on the Horizon system?
1.25 How did Horizon process and/or record Transaction Corrections?
Prepared by: Jason Coyne
Occupation: Partne
Specialist Fiald: IT Systems Q I rou
On the Instructions of: Freeths LLP
180503R1935
16 October 2018
FUJ00082162
FUJ00082162
Page vii of 225
Documents Reviewed
1.26 The document categories that were reviewed for the purposes of this
report are as follows:
a.
b.
m.
n.
Dimensions Disclosure
Horizon Technical Disclosure
Additional Horizon Disclosure
Stage 01 Disclosure
Stage 02 Lead Claimant Disclosure
Stage 02 Generic Disclosure
Stage 03 Disclosure
PEAK Disclosure
KEL Disclosure
Second Sight Disclosure
Primary Claimant Disclosure
Horizon Disclosure
Horizon Witness Statement Disclosure
Coyne RFI Disclosure
1.27 Due to the volume of documents in the matter? it has not been possible
to review each document individually at the time of writing. It should
also be noted that PEAK disclosure was not provided until 27
September 2018 and therefore the opportunity for review has also
been limited due to time constraints. Potentially relevant documents
have therefore been initially identified using search terms and then
reviewed.
2 Approximately 396,000 parent documents.
Prepared by: Jason Coyne
Occupation: Partne
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page viii of 225
1.28 Of the 218,367 PEAKs disclosed by POL I have, in the interest of
expediency, used intelligent search techniques to initially review those
that might specifically relate to branch accounts. I have therefore
reviewed 1,262 in the limited timeframe allowed since disclosure.
1.29 Of the 1,262 reviewed I have found evidence of errors with, but not
limited to, financial impact, Reference Data errors and correction to
branch accounts.
1.30 Summaries and key excerpts from a select number of PEAKs which could
have a financial impact upon branches (that are also narrated within
this report) are found at Appendix A.
Prepared by: Jason Coyne
Occupation: Partne ° ey
Specialist Field: IT Systems } 41 UU
On the Instructions of: Freeths LLP ~ i
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page ix of 225
Author
1.31 My name is Jason Coyne and I am a Expert Report of Jason Coyne at IT
Group UK Limited.
1.32 I have 30 years of experience in Information Technology. During this
time I have designed, programmed and supported Electronic Point of
Sale (EPoS) systems for use in payment handling, stock control,
distribution in addition to full business accounting systems. I have also
assisted public sector councils with revenue and benefits processing
systems, investigated failures within stock market trading systems,
European gambling and gaming networks and determined fraud in
retail banking, point of sale, cash in transit and electronic funds
transfer systems.
1.33 I have been instructed by UK and International firms of solicitors in
connection with technology project fault analysis, software
development and digital forensic investigations. I have been an expert
witness in numerous criminal and civil cases over the past 19 years.
1.34 1 am a member of the Academy of Experts, British Computer Society and
of the Society for Computers and Law.
1.35 I have also participated in numerous mediations and have appeared
before tribunals and in other forms of dispute resolution in London,
Brussels and Johannesburg.
1.36 In December 2008 I obtained the Expert Witness Diploma from Cardiff
University Law School.
Declaration of Independence
1.37 I have never previously been instructed to undertake work for either
party. I therefore declare that I have no connection with the claimant
or defendant involved in this case that might in any way affect my
independence.
Prepared by: Jason Coyne
Occupation: Partne ° ey
Specialist Field: IT Systems d JFOU
On the Instructions of: Freeths LLP ° i
FUJ00082162
FUJ00082162
2. Background
2.1 This dispute arises from the action pursued by the Claimants
(Subpostmasters) against the Defendants, Post Office Limited (POL).
2.2 The Defendant (POL) currently operates a network of around 11,500 Post
Office branches throughout the UK*, within which, a high number of
products and services are offered.
2.3 POL appoints a Subpostmaster (or manager equivalent) to run the branch.
The vast majority of Claimants are or were Subpostmasters.
2.4 In a number of limited circumstances, it is understood that some branches
have been run by a limited company through a Franchise Agreement,
or by a Crown Office employee, through a contract of employment.
For the purposes of this report, all categories will be referred to as
Subpostmasters.
2.5 Horizon comprises an Electronic Funds Transfer Point of Sale (EFTPOS)
retail and accounting system introduced by POL in Post Office
branches in or around 1999.
2.6 Post Office contracted with ICL Pathway Ltd (latterly Fujitsu Services) and
Horizon was initially piloted in 1996.
2.7 Horizon was updated frequently, at various periods with a major
amendment in 2010 known as “Horizon Online”, often referred to as
“HNG-X".
2.8 After its introduction in 1999, many Subpostmasters reported difficulties
with the operation of Horizon.
2.9 It is reported by POL that Horizon processes 47 million transactions per
week.°
* Historically, at the launch of Horizon this number may have been 19,000 branches.
5 2.2 12. Presentation_The Post Office, An Insight_.pdf, The Post Office-An Insight, Angela Van Den Bogerd,
circa 2017, [POL-0021926] [05e2ac28f7b36b04dd83ab30 1edf9F91]
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page xi of 225
2.10 In 2003, Fujitsu won the contract to manage the administration of the
Horizon systems, initially until 2010 but this arrangement continues
today.
2.11 In 2013 ATOS won the contract to manage the ‘first line’ Horizon support
helpdesk including the management of Reference Data.
Prepared by: Jason Coyne
Occupation: Partne ° ey
Specialist Field: IT Systems g tQrou
On the Instructions of: Freeths LLP ~ i
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page xii of 225
3. Executive Summary
3.1 In respect of Issue 1, it is agreed in the experts Joint Statement dated 4
September 2018, that bugs/errors/defects that existed within Horizon
have the potential to cause apparent or alleged discrepancies or
shortfalls relating to Subpostmaters’ branch accounts/transactions. It
is also agreed that evidence exists which shows _ that
bugs/error/defects have caused actual discrepancies or shortfalls
relating to Subpostmasters’ branch account/transactions.
3.2 Each discovered bug/error/defect could have remained unresolved in
Horizon for varying periods of time. In my opinion there is the
possibility of other bugs/errors/defects existing within Horizon which
have yet to be discovered and resolved.
3.3 The sheer volume of Known Error Logs and reconciliation reports confirm
the wide-ranging extent of the impact of such bugs/errors/defects.
This evidence demonstrates that such bugs/errors/defects would
undermine the reliability of the Horizon system to accurately process
and record transactions.
3.4 In respect of Issue 2, as agreed in the experts Joint Statement, the extent
to which any IT system can automatically alert its users to bugs within
the system is necessarily limited. Whilst Horizon has some automated
checks which would detect certain bugs there are types of bugs that
would not be detected by such checks.
3.5 Horizon did contain alert messages that would notify a Subpostmaster to
certain instances of some counter level errors, but the wider
implications of the error would be unknown to the Subpostmaster.
3.6 However, there do not appear to be notifications issued either by Horizon
or by Fujitsu or Post Office where known bugs and defects have been
discovered. The exception may be in relation to the Suspense account
issue where Post Office say that Subpostmasters were notified,
although how they were notified has not been disclosed.
Prepared by: Jason Coyne
Occupation: Partne ° ey
Specialist Field: IT Systems g tQrou
On the Instructions of: Freeths LLP ~ i
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page xiii of 225
3.7 With regards to Issue 3, whilst the present-day version of Horizon,
supported by manual human support may now be considered as
relatively robust in the spectrum of computer systems used in
businesses today it has undergone major modifications in its history.
It is likely that in 1999 when it was first commissioned, and in 2010
when it was significantly upgraded (to Horizon Online), it was less
robust. Horizon comprises a hugely complex estate of hundreds of
interfacing systems, each which exposes many potential points of
error.
3.8 Post Office have stated that the Horizon system has changed 19,842°
times since its inception. I have identified that Horizon’s operation is
highly dependent upon timed system processes with reliance on data
messages being replicated until delivery which is often where faults
occur.
3.9 Fundamental in determining the robustness of Horizon is gaining an
understanding of the Post Office (and Fujitsu) manual business
Processes applied when determining and handling the effects of
bugs/errors/defects. Therefore, when technical failures occur,
understanding the correlating manual processes, inputs and respective
fixes applied in each instance to remedy an imperfect system to a
reasonable level of robustness is key.
3.10 Horizon’s relative robustness does not mean that is thereby extremely
unlikely to be the cause of shortfalls. As agreed in the Joint Statement
between the experts, robustness does not mean perfection. We also
agree that the level of robustness may have increased or decreased
each time as the systems have changed. It is also known that
bugs/errors/defects have caused actual discrepancies and shortfalls in
branches.
3.11 Regarding Issue 4, it is agreed in the joint statement that the potential
exists for such errors in the data recorded within Horizon. Bugs errors
© The number of “release notes” reported by Post Office in response to my RFI.
Prepared by: Jason Coyne
Occupation: Partne °
Specialist Field: IT Systems ¢
On the Instructions of: Freeths LLP ~
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page xiv of 225
and defects that were known to Post Office, as well as those that were
not classified as “Known Issues” ultimately presented themselves as
errors in data entry, data transference and data processing. Due to
the nature of the complex and far reaching Horizon interfacing
architecture and real time services, such errors were unavoidable.
3.12 With regards to Issue 5, reconciliation (the nature by which Post Office
and Horizon compares transaction data recorded by Horizon against
transaction data from sources outside of Horizon) is a large and
complex facility. It involves many different streams of electronic
processing from both Fujitsu data centre computing components,
multiple “external clients”, Post Office and Fujitsu business process
departments and manual investigatory procedures (where corrective
fixes are applied, if necessary). If the reconciliation process identified
a difference between the sources being compared, then manual steps
are taken to establish and correct the errors and potentially issue
Transaction Corrections as a remedy.
3.13 The reconciliation process ultimately presents the possibility of further
error within the Horizon system whereby an inappropriate method of
fix was selected, and/or the corrective fixes may have been carried
out erroneously.
3.14 Regarding Issue 6, it is agreed in the joint statement that, whilst Horizon
contains measures and controls for detecting system integrity
concerns, the automatic mechanisms have failed in the past.
3.15 There are a range of measures and controls existing in Horizon each
designed to prevent, detect, identify, report and reduce the risk of
several multifaceted errors. It is likely that during the life of Horizon
system that these measures and controls improved. It is also
reasonably likely that in the majority of cases the measures and
controls were successful. However, there is also evidence to indicate
that a cost/benefit analysis was applied to the fixing of
Prepared by: Jason Coyne
Occupation: Partne ° ey
Specialist Field: IT Systems g tQrou
On the Instructions of: Freeths LLP ~ i
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page xv of 225
bugs/errors/defects and that the possibility of error was not reduced
as far as possible.
3.16 Regarding Issue 7, documentation illustrates that Fujitsu could remotely
access transaction data recorded by Horizon. Several technical tools
exist with the specific purpose of allowing Fujitsu to carry out
modifications and corrective fixes to transaction data. In addition, a
number of external audit reports commissioned by Post Office
reported that such access could occur without adherence to the
control mechanisms in place.
3.17 With respect to Issues 8 and 9, whilst there are reports available to the
Subpostmaster for accessing and identifying branch transactions
recorded within Horizon; these reports would not necessarily allow
them to identify any discrepancies/or shortfalls that arose from errors
occurring within the processing of data within the Horizon systems, or
of a discrepancy reported by an external client. The experts agree that
other causes of apparent or alleged discrepancies and shortfalls may
be more difficult or impossible to identify from reports or transaction
data available to Subpostmasters.
3.18 Many Known Error Logs (KELs) identify that not all errors were
understood even by Fujitsu. In the circumstances, it is highly unlikely
that a Subpostmaster could interpret or identify the causes of any
bugs/errors or defects when Fujistsu themselves often did not
understand the cause of such or their full effects.
3.19 Post Office have a significantly larger repository of information available
to them than the Subpostmaster. Post Office could request further
lower level detail of data from Fujitsu and would have visibility of
reports beyond the counter level that were not available to the
Subpostmaster.
Prepared by: Jason Coyne
Occupation: Partne fo} eae
Specialist Field: IT Systems J JFOU
On the Instructions of: Freeths LLP ri
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page xvi of 225
3.20 Further agreed in the Joint Statement, identification of errors would
require cooperation of Post Office staff because of a Subpostmasters’
limited knowledge of complex back-end systems.
3.21 In respect of Issue 10, documentation illustrates that a wide range of
users at Fujitsu did and do have the ability and facilities to access and
modify transaction data. Fujitsu staff were able to implement changes
that had the potential to affect transaction data both without the
knowledge or consent of the Subpostmaster and/or Post Office. In
addition, a number of external audits commissioned by Post Office
reported that the appropriate control mechanisms to prevent mistakes
being made were not always followed.
3.22 As agreed in the experts Joint Statement, the very nature of applying
fixes within any IT system, including those implemented by Fujitsu
have the potential to affect transaction data or data in branch
accounts.
3.23 Regarding Issue 11, business process rules and technical restrictions
should apply in relation to accessing and modifying transaction data. It
is reported that a documented audit log of each and every occasion of
live data access exists, however, this has not been made available by
Post Office.
3.24 Regarding how often transaction data was accessed (Issue 12), I have
asked for this information by way of Request for Information, but at
the date of submitting my report Post Office has refused, making
reference to the relevancy of my request. Post Office states that there
are more than 36,000 documents regarding how often the access was
granted (outside of actions not needing express authorisation that
could also be carried out to fix data), but these have not been made
available for review.
Prepared by: Jason Coyne
Occupation: Partne ° ey
Specialist Field: IT Systems d JFOU
On the Instructions of: Freeths LLP ~ i
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page xvii of 225
3.25 Regarding Issue 13, the implications of Fujitsu carrying out corrective
fixes to data within the Horizon system is that it could have the
potential to affect the reliability of Branches’ accounting positions.
3.26 Issue 14 and how Subpostmasters fulfilled certain processes is detailed
at section 7.
3.27 Issue 15 and how Horizon processes and records Transaction Corrections
is set out in detail at Section 6.48 of this report. For a Transaction
Correction to be processed, the financial value of the correction is
manually assessed by Post Office staff, before submission to the
Subpostmaster via Horizon.
3.28 Transaction Corrections can be issued to either rectify an error or
discrepancy deemed as a fault by the Subpostmaster (or clerk), or
when branch transaction data does not align with Post Office, or a Post
Office external client which may not be an error on the
Subpostmasters’ part. It is also possible that Transaction Corrections
were issued as a result of error in Horizon transaction data processing.
Prepared by: Jason Coyne
Occupation: Partne ° ey
Specialist Field: IT Systems d JFOU
On the Instructions of: Freeths LLP ° i
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page xviii of 225
4. History of Horizon
Known Chronology Milestones
4.1 This chronology is not exhaustive but rather selects a number of the
milestone dates throughout the development of Horizon.
i. Pathway (later Fujitsu Services) awarded contract for Benefits
Payment Card May 1996
ii. Horizon Pilot 1996
iii. Pathway cited “greater than expected complexity” and “...major
implications for the degree of difficulty of the project’”? which
ultimately lead to failure of the project.
iv. Post Office Counters Ltd and Pathway sign agreement to utilise
the project to automate Post Offices July 1999
v. — Horizon rollout 1999 - 2002
vi. Core System Release - This included the introduction of
Automated Payment Smart cards and APS/TPS reconciliation.
August 2000
vii. Maintenance Release M1 - Prime purposes of the upgrade were
the enhancement of the CSR+ Applications (APS, LFS, EPOSS,
EPS, OBCS), enabling of the AP client variable day file
transmission, enhancement to Reference Data products and
minor changes to TIPAIS Pathway generated CPs to improve
operability of the system February 2001
viii. S04 Release Additional functionality on the Horizon Pilot outlets
to permit the printing of forms Approx. July 2001
ix. S06 Release Day D rectification measures - This included a new
automatically generated broadcast message to detail when
7 1.2 2. Final Report - Cancellation of Benefits Payment Card.pdf, National Audit Office - The Cancellation of
the Benefits Payment Card project, 18 August 2000, (Page 9), [C-0003630}
[51956ab654c0a9250059c5848099a80f]
Prepared by: Jason Coyne
Occupation: Partne ° ey
Specialist Field: IT Systems g tQrou
On the Instructions of: Freeths LLP ~ i
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page xix of 225
counters at an outlet are offline. This was to be implemented in
a staged manner and included a receipts and payments fix June
2001
x. $10 Release Data centre and counter upgrade introduced
unattended reboot facility at the counter September 2001
xi. B11 Release the first network banking release, changed the
version of Tivoli used by the whole estate Approx. December
2001
xii. S11 Release January 2002
xiii. B12 Release June 2002
xiv. S20 Release September 2002
xv. B13 Release Approx. September 2002
xvi. Network Banking 2003
xvii. S30 Mails Application /Escher Mails 3.3 package (1 Feb
2003)
xviii. BI3 S70 EMV Banking and Retail,
TDES and NBE Accommodations.
2003?
xix. S50 Release October 2003
xx. S60 Release Approx. February 2004
xxi. S52 Release March 2004
xxii. S70 Release October 2004
xxiii. S75 Release (containing changes to support the changeover to
use of NBX banking agents (NBE replacement)? Approx. Oct
2004
xxiv. IMPACT Programme
xxv. POLFS (a SAP-IS Retail System) implemented 2004
Prepared by: Jason Coyne
Occupation: Partne
Specialist Field: I
TT Systems
On the Instructions of: Freeths LLP
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page xx of 225
xxvi. S80 Release Jan 2004 - Aug 2005°
xxvii. BI3 S80 T&T Harvester Agent accommodations Approx. Nov
2004?
xxviii. POLSAP rationalisation (rationalisation of disparate systems
SAPADS and POLFS) 2007-2009
xxix. $90 ‘Bureau Plastic’ accommodations January 2006
Xxx. $92 Release March 2006
xxxi. T10 Release August 2006
xxxii. T40 Release January 2007
xxxiii. Horizon Online Rollout 2010
xxxiv. First Line support provided by ATOS June 2014°
4.2 Whilst Horizon is maintained by Fujitsu (formerly ICL Pathways), the
communications between terminals and Post Office was_ initially
subcontracted to Escher’.
4.3 Horizon, initially based upon Escher’s Riposte messaging system (and
later supported by the WebRiposte system to accommodate Network
Banking changes and the Network Banking Engine supplied by IBM),
was subsequently migrated to Horizon Online. Horizon is therefore
ultimately a composition of many sub-contracted components.**
4.4 POL explain that there are 19,842 release notes (in relation to software
changes), consistent with each of these notes being a change to the
Horizon system.
° PMREPO13_1.doc, S80 Release Closure Report, 07 February 2007 [POL-0089062]
[9c9308a30adb884074F47Fe1 1c7469d7]
° POL-17645-MGTO012 Fujitsu - Horizon Service OWA v2 1 draft.doc, Fujitsu - Horizon Service Operational
Working Agreement, 13 May 2014, [POL-0215476] [339d47429f9f8a83ff93e95e9d3eeb82]
°° TDARC026v04.doc, Horizon Network Banking Architecture, 30 October 2000
[POL0032839][45d467837b7a6d8cec7c914093b39df5]
11 ARCAPPARCO0002_0.2.doc, HNG-X Integration Architecture, 08 November 2006 [POL-0087918]
[daecOde8aS5eee25b5c9d06730c338dd0]
Prepared by: Jason Coyne
Occupation: Partne
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page xxi of 225
Horizon (pre-Horizon Online)
EFTPoS - Electronic Funds Transfer at Point of Sale.
4.5 Initial work was carried out in 1997 to specify a high-level design
approach for the support of EFTPoS within the Horizon platform.!? The
Purpose was to accommodate new functionality to allow debit and
credit cards as an additional method of payment (MoP) for Post Office
goods and services, directly within the Electronic Point of Sale System
(EPOSS).
4.6 The requirement to introduce a Network Banking Service took priority over
(and subsumed) this requirement.
4.7 Post Office Counters Ltd (POCL) objective became to add debit card
payments as an additional method of payment to the Horizon Network
whilst the Network Banking Service was being delivered (initially
2002?)
The Introduction of the Network Banking Solution
4.8 After the failure to introduce the Payments Benefit Card in 1999,!? POCL
and Pathway signed an agreement to automate post offices, and Post
Offices Network Banking Unit (PONB) introduced the requirement to
fulfil banking transactions at Post Office counters so as to offer
services for several different banking institutions.
4.9 In summer 2000, a ‘proof of concept’ was undertaken to investigate the
integration of internet technologies with the current Horizon System to
support the delivery of banking transactions'* (and subsequently to
replace the loss of the benefits card system). A primary facet of the
Network Banking Solution was the delivery of the banking transactions
1 NBSRS002_0.4.doc, EFTPOS System Requirements Specification, 12 October 2001 [POL-0062288]
[2f9abb4aa7bfa0a263d0e3bd89101510}
© 1,2 2. Final Report - Cancellation of Benefits Payment Card.pdf, National Audit Office - The Cancellation of
the Benefits Payment Card project, 18 August 2000, [C-0003630] [51956ab654c0a9250059c5848099a80f]
+4 NetworkBanking_Report_FinalV_ReportV1.doc, WebRiposte Framework - Final Report, 22 January 2001 [POL-
0058079] [e709a9e390fb3e7570a1b5c90c78F605]
Prepared by: Jason Coyne
Occupation: Partne r;
Specialist Field: IT Systems ¢ 41 UU
On the Instructions of: Freeths LLP ~ i
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page xxii of 225
within the already established Escher WebRiposte environment. The full
installation and integration of this was the task of ICL Pathway.
4.10 WebRiposte was a message passing technology from Escher Group which
extended the functionality of the existing Riposte Message Server!*
(which was responsible for storing all the data in the Post Office
branches and replicating it to the Data Centres’*).
4.11 IBM were selected for the supply of the Network Banking Engine which
was designed to handle the interface between the Horizon system and
the agreed Financial Institutions, which Post Office referred to as
‘External Clients’.
4.12 The introduction of network banking and EFTPoS developments brought
with it a more complex enhanced architecture within which further
systems to ensure transaction integrity and reconciliation could be
imposed.
4.13 However, it should be considered that ‘Horizon’ originally stemmed from
an inherited system and architecture with an initial, fundamentally
different design requirement.
4.14 Early documentation identifies that around 1998 and during a period
where Pathway Performance Technical Testing was to be conducted by
A&TC (assessing the system from a technical angle of suitability),
there were so many design changes that the totality of the testing had
to be re-planned into four stages*®.
4.15 The aim of the testing and various other relied upon documents was for
Pathway to produce a report to meet Horizon acceptance requirements.
4.16 The document further states (in relation to the testing conducted):
“It was not practical to build a performance test rig to replicate the
entire system architecture. The tests were therefore carried out on sub-
15 Witness Statement of Gareth Jenkins.pdf, 5th October 2012 [C-0003632]
16 SUTRP004_1.doc Pathway NR2 Technical Testing Performance Tranche1 Closure Report, 08 December 1998
[POL-0047506] [6bc1e58bdede0a88d15c008c9403940d)
Prepared by: Jason Coyne
Occupation: Partne r;
Specialist Field: IT Systems 6 j1 UU
On the Instructions of: Freeths LLP ~ i
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page xxiii of 225
systems consisting of system components designed to represent the live
system as resources permit. The system definition, however, has
changed significantly over the test period and tests may not have run on
the current hardware definition.”
4.17 I have not identified nor reviewed a Horizon Acceptance Report. Nor
have I seen the output of the early tests conducted in relation to the
Horizon system. However, it is apparent that the initial system design
and architecture was an evolved and evolving landscape.
Horizon Components
4.18 Horizon is a complex multifaceted system. Core components that
facilitate its operation (an understanding of which are necessary for
this report) are summarised below.
Reference Data
4.19 Reference Data (effectively data about products and operational
elements) is a critical element of the Horizon system, which interfaces
with a wide range of Horizon components. Without Reference Data,
Horizon would not fully function, nor could Subpostmasters operate
their branches. It informs the operation of the Point of Sale system at
the counter amongst many other things. The management and
operations with regard to Reference Data has been outsourced from
within Post Office control to a sub-contractor (ATOS) since June 2014.
4,20 The integrity of Reference Data is critical for the correct operation of a
variety of systems within the Horizon architecture. Post Office's
Reference Data Management Centre (RDMC) supports the loading,
storage and release of Reference Data within the Horizon system. The
Reference Data Distribution Service (RDDS) distributes Reference Data
to Post Office branches and other data centre systems. The POL
Reference Data Team is a team dedicated to delivery of Reference
Data and verification of operational business change through
Reference Data. Horizon counter Reference Data was distributed by
Prepared by: Jason Coyne
Occupation: Partne ° ey
Specialist Field: IT Systems d JFOU
On the Instructions of: Freeths LLP ~ i
180503R1935
FU,
16 October 2018 Page xxiv of 225
FUJ00082162
1J00082162
the Riposte messaging facility, although this was replaced by a
different mechanism in Horizon Online.
4.21 Despite the criticality of the integrity of Reference Data, a document from
July 2017 suggests that changes to Reference Data were not subject
to any appropriate change control process. The document’’ reports;
w
.. we have now aligned that all Reference Data changes go through
the appropriate change process”. This is consistent with the position
that prior to July 2017 Reference Data could be changed without any
formal consideration as to what the impact might be.
Transaction Data
4.22 According to Fujitsu, there are only four sources of transactions that make
up transaction data’®:
a.
b.
Those manually entered by a user in branch at the counter;
Transaction Corrections (TC) which are produced by Post Office
to be accepted by a user in branch to correct discrepancies in
the accounts;
Transaction Acknowledgements (TA) which are non-counter
transactions and typically initiate from another piece of
equipment (such as a lottery terminal). These transactions are
typically relayed to Post Office/Fujitsu and need accepting into
Horizon before forming part of the branch's transaction data;
Fujitsu inserted transactions. These are known as balancing
transactions and are injected into branch accounts by Fujitsu in
order to ‘balance’ a discrepancy. These do NOT require
acceptance by the Subpostmaster as TC’s and TA’s do.
Y Operations Board 21 July 2017.pdf, [POL-0221328] [9c45e0be3ff2b6773447cc6e4.1db5#46]
18 Witness Statement of Torstein Godseth - 27.09.18.pdf
Prepared by: Jason Coyne
Occupation: Partne —
Specialist Field: IT Systems 6 YlOU
On the Instructions of: Freeths LLP ”
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page xxv of 225
The Horizon Counter (1996 - 2010)
4.23 The processing of the Horizon counter detailed below has been taken from
the Witness Statement of Gareth Jenkins dated October 2012.1°
4.24 Horizon was initially designed to store all data locally on the branch
counter’s hard disk (see Appendix B Figure 1).
4.25 Once the data has been successfully stored there it is then replicated
(copied) to the hard disks of any other counters in the branch (or in
the instance of a single counter branch stored to additional external
storage on the counter). It is then passed on from the counter to the
Horizon data centre where it is stored in the CS messagestore.
4.26 The replication process was designed so that if the data was not copied
immediately (because of communications or hardware failure), then
further attempts are made to replicate the data at regular intervals
until it is finally copied successfully.
4.27 Once the data reaches the Horizon data centre a further copy is taken by
the Audit Agent which writes it to an audit file where it is available for
retrieval for up to seven years.
Transaction Auditability
4.28 Data in the audit trail was designed to be sealed with a secure checksum
that is held separately to ensure that transaction data has not been
tampered with or corrupted.
4.29 Horizon was designed so that every record written to the Transaction Log
at the counter should have its own unique incrementing sequence
number. Therefore, it should be possible to detect if any transaction
records become anomalous (where they are actually captured by the
Transaction Log in the first instance).
+ Witness Statement of Gareth Jenkins.pdf, Witness Statement of Gareth Jenkins, 05 October 2012 [C0003632]
[b544230cf07249c189cf664fcba6d899]
Prepared by: Jason Coyne
Occupation: Partne ° ey
Specialist Field: IT Systems d JFOU
On the Instructions of: Freeths LLP ° i
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page xxvi of 225
4.30 Whilst a customer session was in progress, details of the transactions for
that customer session were normally held in the computer’s memory
until the customer session (referred to as the ‘stack’) was settled. At
that point all details of the transactions were written to the local hard
disk and replicated. When a stack was secured it was written in such a
way that either all the data is written to the local hard disk or none of
it is written.
4.31 The data for the stack will have been successfully secured to the local
hard disk before the screen updated indicating that a new customer
session can be started.
4,32 Although an attempt will have been made to replicate the data to the
messagestore, there is no guarantee at this point that such replication
will have been successful. For example, if there is a network failure
followed by a terminal failure there is a risk that transactions in the
intervening period could be lost.
4.33 Any failures to write to the hard disk after appropriate retries would result
in the counter ‘failing’ and needing to be restarted.
4.34 Whenever data is retrieved for audit enquiries several checks should be
carried out, namely that:
a. The audit files have not been tampered with and their Journal
Sequence Number (JSN) is incremental to the last one;
b. The individual transactions have their integrity checked to ensure
that no corruption has occurred;
c. Acheck should be made that no records are missing. Each record
generated by a counter has an incremental sequence number and
a check is made that there are no gaps in the sequencing.
Migration towards Horizon Online (Horizon Online 2010 - Present)
4.35 A document dated 2005 authored by Fujitsu records the requirement for
Fujitsu to provide a more competitive solution, since Fujitsu accounted
Prepared by: Jason Coyne
Occupation: Partne ° ey
Specialist Field: IT Systems g tQrou
On the Instructions of: Freeths LLP ~ i
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page xxvii of 225
for 65% of the IT spending for Post Office.2? The document further
notes:
“The original business case for HNG within Post Office was based on a
balance of cost reductions and improved capabilities. The new business
case is almost entirely based on cost reductions.”
4.36 Therefore, the major requirement for Horizon Online was to make secure
minimum changes to the legacy host systems whilst still providing the
end to end business needs.* Cost reductions in the Horizon Online
solution were to derive from (1) reusing legacy counter hardware, (2)
revised test and disaster resilience strategy, (3) application re-
engineering, (4) revised development processes, and (5) utilising
offshore programmers for software development.”°
4.37 Horizon hardware was recycled for continued use in Horizon Online,
there was an acknowledgement in a Board Report’’ regarding design
considerations when implementing Horizon Online:
“While the architecture is generally designed for resilience, cost/risk
trade-offs were agreed in the move from the original Horizon system to
the new HNG-X one which mean that the service is not truly high
availability”.
4.38 Since analysis of serviceability issues within Horizon identified that the
two key areas of cost were (1) the stability and manageability of the
Riposte application and (2) the effort in recovering data stored locally
on counters, the Horizon Online objective was therefore to redevelop
the business applications and enforce a centralised model for data
storage.”° The counter hardware would remain and locally store
operational data (e.g., Reference Data) and business logic, but the
2° RMARC002_0.1.doc, Horizon Next Generation - Plan X (HNG-X), 21 September 2005 [POL-0084540]
[754315a4037a6ea4c1ec7ee070b7d170]
2 Data Reconciliation Service High Level Design Delta for HNG-X, 30 September 2009 - [POL-0032942}
[972420ee28dfe6db41e6847ae3f4493e]
® BoardPDFpack.pdf, Royal Mail Holdings plc Board, September 2012, [POL-0171024]
[7dad94569f245c16763d0255b1 144139]
Prepared by: Jason Coyne
Occupation: Partne
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page xxviii of 225
transactional data would be stored in the data centre rather than on
the branch counters.”°
4.39 In order to further reduce the overall application development costs
within Horizon Online, substantial re-use of data centre application
components was proposed. The legacy host database applications
(TPS, APS, LFS, DRS and TES) were to remain largely intact, and a
web service interface was to replace Riposte messaging, and
simplification of security mechanisms.”?
4.40 The diagrams of data centre applications and Horizon / Horizon Online
migration shown in Appendix B (Figure 2 & Figure 3) illustrate the changed
components of Horizon to accommodate Horizon Online and legacy
components remaining, and the changes at branch counter level.
The Branch Database
4.41 Horizon Online brought significant changes to the counter processing as
described above.
4.42 For Horizon Online, all data is now stored in an online branch database
known as the BRDB (therefore, no longer stored on the counter hard
drive).
4.43 Transactions are carried out locally on the Horizon Online counters and a
‘Basket’ is built up during a customer session. Each transaction should
result in a Basket entry consisting of one or more accounting lines.
4.44 At the end of a customer session when the Basket has been completed
and all settlement items (or tender lines) have been processed and
added into the Basket as further accounting lines (in double entry
terms), such that the total value of the Basket is zero, the entire
Basket is sent to the data centre as a BAL message where the Branch
Access
Layer (BAL) processes the message and all the accounting lines are
2? RMARCO02_0.1.doc, Horizon Next Generation - Plan X (HNG-X), 21 September 2005 [POL-0084540]
[754315a4037a6ea4c1ec7ee070b7d170]
Prepared by: Jason Coyne
Occupation: Partne r;
Specialist Field: IT Systems 6 j1 UU
On the Instructions of: Freeths LLP ~ i
FU,
180503R1935 16 October 2018 Page xxix of 225
FUJ00082162
1J00082162
recorded and committed to the BRDB.** Horizon Online message flows
are depicted in in Appendix B (Figure 4).
4.45 The BRDB is the repository for all branch transactions and event data
captured. It also provides the storage mechanism for other branch
data such as Transaction Corrections.
4.46 Transactional data is written to the branch database (BRDB) via the
Branch Communication Application on the Branch Application Servers.
Logic in the branch communication application should then be able to
discern between the different transaction types by passing over the
data stream and inserting the data into the relevant BRDB tables. The
largest single function performed by the branch communication
application is the capture of transaction and settlement information
resulting from completion of customer sessions and other activities
within the branch estate. The data needs to therefore be scanned to
determine its type before it is acted upon.?>
4.47 The branch communication application will then be responsible for
inserting transaction data directly into the ‘Main Transaction Store’
within the BRDB. The main transaction store then facilitates delivery of
appropriate data to Post Office Account external interfaces via the
legacy applications (See Reconciliation in Horizon Online Diagram
Appendix B - Figure 6).
Report Data
4.48 The branch communication application is responsible for inserting a
subset of transaction data directly into a Report Data Store (within the
BRDB) that is designed specifically to facilitate counter/branch
daily/weekly reporting.?>
Aggregated Data
4 Witness Statement of Gareth Jenkins.pdf, Witness Statement of Gareth Jenkins, 05 October 2012 [C0003632]
[b544230cf07249c189cf664fcba6d899]
25 RMARCO02_0.1.doc, Horizon Next Generation - Plan X (HNG-X), 21 September 2005 [POL-0084540]
[754315a4037a6ea4clec7ee070b7d170)
Prepared by: Jason Coyne
Occupation: Partne r;
Specialist Field: IT Systems 6 j1 UU
On the Instructions of: Freeths LLP ~ i
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page xxx of 225
4.49 Daily batch processes are responsible for aggregation of data from the
main transaction store. This is stored as aggregated data that should
facilitate calculation of the stock units and branch totals for
declarations and the information required for the rollover processes
(transitioning from one accounting period to the next). Declaration
totals will also be stored in this area.
Recovery Data
4.50 The recovery data store in the BRDB is used to capture events that need
to be recorded in case of errors mid transaction, to define the
appropriate behaviour of the counter on recovery.
Audit Data
4.51 A separate audit record of transactions and events is maintained that
contains the raw data received in the message from the counter. I
understand that this was archived.
Transaction Auditability
4.52 Horizon was designed so that auditable messages from the counter were
stored, together with their digital signature and other key attributes in
the ‘Audit table’ (also known as the Message Journal) in the BRDB.
Each day the contents of this database table should be copied from
the BRDB to a number of serial files. A check should be made that
there was no missing or duplicate Journal Sequencing Numbers (JSNs)
for any counter.
4.53 The files are then copied to the audit system where they are sealed
digitally and held for seven years during which time they may be
retrieved and filtered to produce the relevant audit for a particular
branch.”©
26 Witness Statement of Gareth Jenkins.pdf, Witness Statement of Gareth Jenkins, 05 October 2012 [C0003632]
[b544230cf07249c189cf664fcba6d899]
Prepared by: Jason Coyne
Occupation: Partne ° ey
Specialist Field: IT Systems d JFOU
On the Instructions of: Freeths LLP ~ i
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page xxxi of 225
Legacy Horizon / Horizon Online Counter Processes
4.54 Whilst many core counter processes remained largely the same,
technical processing of certain counter activities changed through the
introduction of Horizon Online.
Recovery
4.55 Disconnections and screen freezes for Horizon Online are recorded as
being dealt with differently to Horizon (pre-Horizon Online).?”
4.56 Subpostmasters are informed through the above quick reference guide
to ensure they “do the right thing at the time the counter becomes
unavailable.”
4.57 Therefore, where a connection to the data centre is lost, users are
presented with a ‘Retry Communication’ message displayed on screen.
Users are encouraged to select the ‘Retry’ option a maximum of two
times to see if data centre connectivity has been restored.
4.58 The Recovery process is stated as one of the procedures to ensure data
integrity remains in the event of a failure.
4.59 In the event the data centre connection is not re-established from the
retry action, the system should settle the session automatically
printing three copies of a Disconnected Session receipt (unless there is
a hardware failure and/or system freeze). Where a connection to the
data centre has been lost before a Basket has been settled the system
treats the transactions as either:
a. Recoverable
b. Non-Recoverable*®
4.60 If a transaction is deemed recoverable, then information about that
transaction is recorded in the BRDB. Recoverable transactions are
¥ Tab 12 - Recovery - Horizon Online Quick Reference Guide.pdf, Recovery - Horizon Online Quick Reference
Guide, ca. 2010 [POL-0001727] [34331¢3a952d2fb4921aeddf5d1f90d6]
28 Recoverable and Non-Recoverable items are defined in the Glossary
Prepared by: Jason Coyne
Occupation: Partne ° ey
Specialist Field: IT Systems g tQrou
On the Instructions of: Freeths LLP ~ i
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page xxxii of 225
noted as:?°
a. All banking transactions
b. All Credit / Debit Card transactions
c. All E Top up transactions
d. All reversals
e. Selected APADC transactions
Reversals
4.61 Horizon Reversals are transactions that are effectively ‘undone’ either
initiated by a Subpostmaster or electronically through the system?°.
4.62 If a transaction has been entered and the customer session completed,
the transaction can be reversed (for example, if the transaction has
been entered incorrectly, or if a customer requests a refund).
4.63 Reversals are also initiated following system failures. These are
documented later in this report as there is evidence that reversals
were problematic in Horizon and affected branch accounts*?.
4.64 A reversal does not result in transaction information in the journal being
amended but causes the insertion of additional, compensating and
correcting transactions.°°
4.65 It is recorded there are four types of reversal:
a. An Existing Reversal - performed when a customer wants a
refund for which there is a receipt, the user must enter the
transaction session number to initiate the reversal.
?° HorizonOnlineDatalntegrity_POL.DOC, Horizon Online Data Integrity for Post Office Ltd, 28 March 2012,
(Version: 0.1b), [POL-0221055] [5e05904c2F271098da69b31806d4053c]
°° CSPROO21_2.doc, NR2 ELECTRONIC POINT OF SALE SERVICE: Processes and Procedures Description, 30 June
1999 [POL-0049668] [5b45d4ba533d9092293476bc691 12863]
31 1.6 6. Horizon Data (status Draft) - the _Helen Rose Report_.pdf, Horizon data Lepton SPSO 191320, 12
June 2013 [POL-0221677, POL-0221678, POL-0221679, POL-0221680, POL-0221681]
[f296f6880e1b8418f37d3e344374c42a]
Prepared by: Jason Coyne
Occupation: Partne
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
180503R1935 16 October 2018
FUJ00082162
FUJ00082162
Page xxxiii of 225
b. A new reversal - performed when a customer wishes to obtain a
refund for a purchase for which they have no receipt. New
reversals do not require the original transaction to be identified.
c. A transfer reversal - performed if it is necessary to reverse a
transfer out that has not yet been transferred in to the receiving
stock unit.
d. A remittance reversal - performed when a transaction for stock
that has been remitted in or out needs to be reversed.
Horizon Support Service & Facilities
4.66 The support model utilised for managing issues and potential
bugs/errors/defects in Horizon/Horizon Online is based on four levels of
support??:
1* line
2nd line
3rd line
>I 4th line
The Role of Each Support Level
4.67 1st line support (currently provided by POL's subcontractor ATOS) - log
incidents either by directly interacting with the user (typically the
Subpostmaster) or from monitoring systems. They document incident
symptoms and aim to resolve all issues where the cause is user
training or environment. ist line resolve incidents by the identification
of knowledge base entries and the application of defined scripts. A
new incident is raised for each critical event that is not already
documented and passed to 2nd line support teams for action.
4.68 2nd line support (provided by Fujitsu) - use symptoms documented by
the ist line to understand errors and gather additional information.
They use expert knowledge to identify a root cause and a solution or
alternatively to develop procedural
workarounds. They
22 SVMSDMPRO0875_1.DOC, End to End Application Support Strategy, 28 July 2011, [POL-0122492]
[db0644e4d5e11b5cce3ed381cb108a88]
also
Prepared by: Jason Coyne
Occupation: Partne
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
180503R1935
FUJ00082162
FUJ00082162
16 October 2018 Page xxxiv of 225
create/update the knowledge base and pass new incidents to the 3rd
line team for further investigation.
First and Second Line Support
Refer to
4" Line Support Knowledgebase KEL
(Help Desk) Database
24 Line Support
(system Maintenance Centre)
Problem seen
before?
¥
Provide
——+ solution to
user
Resolved or
Workaround?
Create KEL &
Log Incident
if
Link call to previous incidents
to avoid duplication
End
Pass to 3rd Line Support
(Software Support Centre)
4.69 3rd line support (provided by Fujitsu) - apply analytical skills to the
symptoms and evidence gathered by ist and 2nd line and undertake in-
depth
investigation into incidents. They should have detailed
knowledge of the system based upon documentation and source code
inspection and they can produce interim workarounds. They identify a
Prepared by: Jason Coyne
Occupation: Partne
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page xxxv of 225
cause and probable solution for incidents which are being passed to
the 4th line team. Responsible for the implementation of any
workarounds that require data changes to the live system, they are
the only unit with authorisation and sufficient physical security controls
to perform this function.
4.70 4th Line Support (provided by Fujitsu) should have detailed knowledge of
the system and are responsible for the investigation and resolution of
new incidents through the production of permanent fixes to repair the
root cause of an incident or a problem in the live application.
Prepared by: Jason Coyne
Occupation: Partne ° ey
Specialist Field: IT Systems g tQrou
On the Instructions of: Freeths LLP ~ i
180503R1935
FUJ00082162
FUJ00082162
16 October 2018 Page xxxvi of 225
Third and Fourth Line Support
Incident passed
from 24 Line Support
Initial investigation
Provide solution
to user
In depth investigation,
identify root cause
and propose solution
i
Update Knowledgebase
Y
Pass to 4" Line Support
:
Assess Work Required
to Fix Peak
J
Submit Peak to
Business Impact Forum
for Authorisation
Develop Fix
and Deploy in Release
4.71 The details of the support model provision changed between the rollout
of the original Horizon application (circa 1999) and when Horizon
Online went live (circa 2010).
Prepared by: Jason Coyne
Occupation: Partne
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page xxxvii of 225
Support Services Provision in 2001
4.72 In 2001, 1st line support to the Horizon system users was provided by
the Horizon System Helpdesk (HSH), run by the Operational Services
Division (OSD) of ICL.
4.73 2nd line support was provided by System Management Centre (SMC), also
run by OSD.
4.74 3rd line support was provided by the Software Support Centre (SSC).
4.75 4th line support was provided by a combination of Pathway Development,
Escher, OSD and Eicon.
Support Services Provision in 2010
4.76 By 2010, HSH had been renamed to the Horizon Support Desk (HSD) and
was run by Fujitsu.
4.77 SMC had become more of a ist line support unit and there was no
dedicated 2nd line support unit. Instead, the 2nd line responsibilities
were being fulfilled between the 1st line and 3rd line units providing a
“virtual” 2nd line function.
4.78 SSC continued to provide 3rd line support.
4.79 4th line support had been streamlined to the Fujitsu Application Support
Service (ASA). ASA would then liaise with Fujitsu Services'
subcontractors/suppliers, or Post Office's suppliers as appropriate.
Support Services Provision from 2014
4.80 Atos took over the first line support of the Horizon service from 17 June
2014, as part of the Post Office re-procurement of the IT Supply Chain
in 2014.°°
3? SVMSDMOLA3308_2.5.DOCX - FUJITSU - HORIZON SERVICE OPERATIONAL WORKING AGREEMENT [POL-
0128502] [cdd379390652be400250cf319c7bbdb8]
Prepared by: Jason Coyne
Occupation: Partne
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page xxxviii of 225
Incident Tracking Systems
4.81 In 2001, 1st and 2nd line support teams used the Powerhelp helpdesk
system from Astea Inc. to log incidents.*?
4.82 3rd and 4th line support teams used PinICL as a call management system
and diagnostic database.
4.83 Calls from 2nd line support were transferred from Powerhelp to PinICL
via an Open Tele service Interface (OTI) link, and updates to the
PinICL calls were transferred back to second line support using the
same mechanism.
4.84 By 2010, the 1st and 2nd line support Powerhelp system had been
replaced by a system called Triole for Service (TfS) to record incidents
and PinICL had been replaced by PEAK, an in-house developed Fujitsu
services incident and release management system.** An individual
incident so recorded is referred to as a PEAK.
4.85 OTI was still used to link the TfS and PEAK systems, although TfS has a
limit of 4000 characters within a single update which potentially
exposed a loss in information.
4.86 From 2014, all ist line tickets were logged into Atos’ SDM12 service desk
application and if not resolved on first call, were transferred to a 2nd
line (or higher) team within Fujitsu. This transfer was provided by an
automated interface into Fujitsu’s Triole for Service (TfS) tool.?°
Known Error Logs (KELs
4.87 The principle of the KEL has been used since 2001 to record information
and workarounds for known issues. This has evolved into a database
maintained by SSC that is available to all levels of support.
® CSQMS004_2.doc, CS Support Services Operations Manual, 29 January 2001 [POL-0061572]
[bb842d86176aa926d3b9ffF25e0fc248]
38 SVMSDMPRO0875_1.DOC, End to End Application Support Strategy, 28 July 2011, [POL-0122492]
[db0644e4d5e1 1b5cce3ed381cb108a88]}
35 SVMSDMOLA3308_2.5.DOCX - FUJITSU - HORIZON SERVICE OPERATIONAL WORKING AGREEMENT
[POLO128502] [cdd379390652be400250cf319¢7bbdb8]
Prepared by: Jason Coyne
Occupation: Partne
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
180503R1935
FUJ00082162
FUJ00082162
16 October 2018 Page xxxix of 225
4.88 1st line support utilise the KEL database to check for known issues and
available workarounds before passing incidents on to 2nd line support.
Where no KEL is found, 2nd line support create a new KEL to record a
workaround or as part of raising a new incident on the PEAK system
for 3rd line support to investigate.
4.89 Information on the KEL is updated by all levels of support as work to
resolve the incident progresses. Any creation or update of a KEL must
be authorised by SSC before it can be seen by all users.
4.90 All new PEAK incidents should be accompanied by a KEL.
4.91 It is possible for a KEL to be linked to multiple PEAK incidents and for
a PEAK incident to reference multiple KELs.
Release Management
4.92 There are three types of release:*°
a. Major Releases - these will normally be to deliver significant new
functionality
b. Emergency fixes - implemented as required to resolve
operational bugs/errors and defect issues
c. Maintenance Releases - applied frequently to address minor
bugs/errors/defects or security issues and the mechanism by
which Fault PEAKs are usually resolved.
26 .0138750] [
» -0138750] [
RAEESRANS AL Di HASOALERYREranuary 2015, [POL 383f7afbb2eed809206 _
Occupation: Partne
Specialist Fiald: IT Systems GrOup
On the Instructions of: Freeths LLP
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page xl of 225
Fault PEAK Lifecycle®2
4.93 The basic Fault PEAK Lifecycle is as follows:
i. Open - Initial state of a PEAK call
ii. Pending - Incident is being investigated
iii. Final - Investigations have finished -Final response is sent
back for approval
iv. Closed - Final state - call is closed on the PEAK system.
4.94 If a PEAK has been sent to a team it is that team’s responsibility to
monitor it, send it on for investigation, fix it or release it to the correct
PEAK stack (queue).
37 -0138750] [
Occupation: Partne
Specialist Field: IT Systems
RASERANSHL P¥n ARLOAERYTPranuary 2015, [POL 383f7afbb2eed8092e6
itgroup
On the Instructions of: Freeths LLP Tay in Technol :
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page xli of 225
Assignment of PEAK Incidents to Maintenance Releases
4.95 Fault PEAKs are investigated and if a fix is required it is sent with an
assessment of the time to fix and any issues to the PEAK stack for the
Business Impact Forum (BIF).°° The BIF convenes weekly to review
outstanding PEAKs for consideration of the business impact.
4.96 If the BIF decide that a fix is not warranted for cost or other reasons, the
Known Error Log (KEL) is updated and the PEAK is closed.
4.97 If the BIF decide that the PEAK should be fixed it is submitted to the
PEAK stack for the PEAK Targeting Forum (PTF).
4.98 The PTF convenes weekly and considers new Fault PEAKs and PEAKs that
have previously been deferred, as well as PEAKs which introduce
business change associated with approved Change Proposals. These
PEAKs will be targeted to a specific maintenance release®® taking into
account timing, other development activities and associated factors.
4.99 The PEAK is then sent back to Development to deliver the fix for
incorporation into the maintenance release.
°° Terms of Reference for POA BIF and PTF,30 July 2014, [POL-0032912]
[52536a2c7ab381b9773db136ebb9042b]
* -0138750] [
RAEESRANS AL Di HASOALERYREranuary 2015, [POL 383f7afbb2eed8092e6
Occupation: Partne (o!
Specialist Field: IT Systems g tgroup
On the Instructions of: Freeths LLP "
FU,
5. Horizon Bugs/Errors Defects and Controls
Issue 1 - To what extent was it possible or likely for bugs, errors or defects of
the nature alleged at §§23 and 24 of the GPOC and referred to in §§ 49 to 56
of the Generic Defence to have the potential to (a) cause apparent or alleged
discrepancies or shortfalls relating to Subpostmasters’ branch accounts or
transactions, or (b) undermine the reliability of Horizon accurately to process
and to record transactions as alleged at §24.1 GPOC?
Issue 3 - To what extent and in what respects is the Horizon System “robust”
and extremely unlikely to be the cause of shortfalls in branches?
Issue 4 - To what extent has there been potential for errors in data recorded
within Horizon to arise in (a) data entry, (b) transfer or (c) processing of data
in Horizon?
Issue 6 - To what extent did measures and/or controls that existed in Horizon
prevent, detect, identify, report or reduce to an extremely low level the risk of
the following:
a. data entry errors;
b. data packet or system level errors (including data processing,
effecting, and recording the same);
c. a failure to detect, correct and remedy software coding errors or
bugs;
d. errors in the transmission, replication and storage of transaction
record data; and
e. the data stored in the central data centre not being an accurate
record of transactions entered on branch terminals?
5.1 There are several areas where Bugs, Errors and Defects could occur.
Some of these are set out below:
a. Failure of a software component (counter software, Horizon data
centre components)
FUJ00082162
1J00082162
FU,
180503R1935 16 October 2018 Page xliii of 225
FUJ00082162
1J00082162
b. Error in Reference Data
c. Failure of a hardware component at counter (CPU, PIN pad, touch
screen, keyboard, counter scanners, network router)
d. Failure of a hardware component at PO data centre (Database
servers, communication servers)
e. Errors in various communications which need to take place for
Horizon to function:
i. Transfer between counter and PIN pad,
ii. Transfer between counter and Branch Database (BRDB),
iii. Transfer between Branch Database and Post Office back end
systems,
iv. Transfer between Authorisation Agent and Post Offices External
Clients.
f. Those areas as set out in Section 5 ‘Horizon Robustness’ and
Section 6 ‘Reconciliation’.
5.2 As agreed in the Joint Statement of experts, evidence exists that
bugs/errors/defects have caused actual discrepancies and/or shortfalls
relating to Subpostmaster branch accounts.
5.3 Identified common failure points throughout Horizon are demonstrated at
Appendix H. This list is not exhaustive but is based on the review of
evidence undertaken to date.
Known Errors/Bugs Defects acknowledged by Post Office
5.4 It should be noted that there are several known bugs/errors/defects
previously acknowledged by Post Office that have affected branch
accounts. These are known as the “Payments Mismatch” defect,
“Calendar Square / Falkirk” bug, and the “Suspense Account bug”.*°
*° 1.4. 6, Letter of Response - Schedule 6.pdf, SCHEDULE 6: REBUTTAL OF ALLEGATIONS AGAINST HORIZON
(Page 97), ca. 2017 [368e44cc103e58561¢5785014597d8f9]
Prepared by: Jason Coyne
Occupation: Partne °
Specialist Field: IT Systems ¢
On the Instructions of: Freeths LLP ~
FU,
180503R1935 16 October 2018 Page xliv of 225
FUJ00082162
1J00082162
Calendar Square / Falkirk Issue**
5.5 This defect was discovered in 2005 and fixed in March 2006 and involved
Horizon failing to recognise transfers between different stock units. In
summary, stock units receiving transfers could not “see” them,
resulting in branch account discrepancies.
Payments Mismatch
5.6 This bug affected at least 62 branches and related to the process of
moving discrepancies into the local suspense account. The majority of
incidents are recorded as occurring between August and October
2010.
5.7 The bug was documented in a report from Gareth Jenkins 29 September
2010* where it was stated:
“This has the following consequences: There will be a receipts and
payment mismatch corresponding to the value of discrepancies that
were lost. Note that if the user doesn't check their final balance report
carefully they may be unaware of the issue since there is no explicit
message when a receipts and payment mismatch is found on the final
balance (the user is only prompted when one is just detected during a
trial balance)”
5.8 This issue is reported as causing discrepancies showing at the Horizon
counter which disappeared when branches followed certain process
steps. However, these discrepancies still appeared within the back-end
branch account. It is noted that the issue occurred if a branch
cancelled the completion of the trading period and then, within the
same session, continued to roll into a new balance period.
5.9 PEAK PC0204765* is a “Master PEAK” that records branches which were
thought that have been affected.
5.10 KEL wrightm33145J* identifies the workaround but also states:
Prepared by: Jason Coyne
Occupation: Partne ° ey
Specialist Field: IT Systems d JFOU
On the Instructions of: Freeths LLP ° i
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page xlv of 225
2 3429 SM BP Correcting Accounts for Lost Discrepancies - 102000790 - CD1.pdf, Correcting Accounts
for "lost" Discrepancies, 29 September 2010 [POL-0010769] [804ea47c166870b7ed0359e4765e0265]
° Wrightm33145J.html, HNG-X KEL wrightm33145J, 23 September 2010 (last updated 01 April 2016),
[POL0040409] [1f025ec713c287ee7a5b1 7accd25b42f]
“Unfortunately the workaround cannot be done after the problem has
occurred at the office! In this case the branch accounts need to be
corrected.”
5.11 It is not clear how many corrections were required to fix all instances of
this (or even that all instances were indeed fixed) or when a full audit
was completed.**
Suspense Account Bug
5.12 The suspense account bug caused Horizon to erroneously replicate
suspense account items. It appears that the bug caused Horizon to
use 2010 monthly Branch trading figures for 2011 and 2012.
5.13 It is reported that POL later investigated and identified the same bug as
being the cause of the issue in January 2013 suggesting that the bug
may have been resident within Horizon for an extended period.
5.14 POL-0215998*° lists the 14 affected branches identified in the occurrence
of the bug in 2013.
Further Bugs / Errors / Defects not acknowledged by Post Office
5.15 The following errors are observed to have occurred within the Horizon
system, however they do not appear to have not been acknowledged
as system wide issues by POL in the same manner as those listed
above.
Dalmellington
5.16 This defect relates to the example that occurred specifically in 2015 at
the Dalmellington Branch in which a Subpostmistress performed a
cash remittance from a core branch to an outreach branch.*° The
acceptance at the outreach branch resulted in quadruple remittance
transactions’ resulting in a £24,000 discrepancy.
Prepared by: Jason Coyne
Occupation: Partne r;
Specialist Field: IT Systems 6 j1 UU
On the Instructions of: Freeths LLP ~ i
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page xlvi of 225
** 1.4. 6, Letter of Response - Schedule 6.pdf, SCHEDULE 6: REBUTTAL OF ALLEGATIONS AGAINST HORIZON
(Page 97 - Para: 4.5), ca. 2017 [368e44cc103e58561c5785014597d8f9]
45 Initial Report from Gareth LocalSuspense.docx dated 10 May 2013 [POL-0215998]
[84b7fac96c5b77ed13642d885b7de2a4]
* 11,Email thread between ATOS and CWU.pdf [C-0005343]
*” 1,Dalmellington Branch duplicated receipts.pdf [C-0005350, C-0005342]
5.17 It is reported in the email thread between Atos and a Helen Baker of
CWU that the remittance team in Chesterfield were aware of the fault
(as this is noted as occurring a “couple of years ago”) and that
Transaction Corrections could be issued for the ‘extra’ remittances.
5.18 The email thread further states (by Atos) that the root cause was
identified as the user forcing log off when post log in checks had not
fully completed and it was therefore considered as a process issue and
not a technical one. However, this is not consistent with the event logs
that do not indicate it was as a result of forced log off.
5.19 Further, Atos refer to release of "a code change that will avoid further
instances of this across the estate” to be included within Release 13.05
to be deployed in March 2016 (five months after the incident). The fact
that Atos made a change to the Horizon system to prevent
reoccurrence is therefore consistent with this being a software bug.
Cash Declarations - Cash Management
5.20 I have identified several Known Error Logs (KELs) that document
occurrences of varying forms of cash declaration discrepancies having
the ability to affect branch accounts reportedly due to system
problems.
5.21 KEL achai1233J*! - Discrepancies between branch cash declarations and
the amount received by the cash management system (SAP) were
identified. In the data communication channel between Horizon and
the cash management system, an incorrect adjustment was being
made which added together the “branch cash in pouches overnight”
4 acha1233J.html, HNG-X KEL acha1233J, 22 June 2012 [POL-0039066]
[af9Sa66a92d17269b535e89644017582]
Prepared by: Jason Coyne
Occupation: Partne
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page x\vii of 225
figure to what the branch has actually declared. The KEL states this is
not a user error or anything that can be corrected at branch level. This
is therefore consistent with the problem being due to the existence of
software bug.
5.22 KEL achai717T*® - Whilst possible causes and a suggested process for
investigation of this issue by the support team is highlighted, there is
nevertheless an acknowledgement that there is the possibility that
cash declaration discrepancies could be due to an “Unknown system
problem”. It is worth noting what the Horizon support team consider a
discrepancy in this instance as:
“A discrepancy is the difference between the cash system thinks the
branch should have, based on a previous balanced figure and the
transactions recorded since, and the cash declared by the branch”.
5.23 There is also evidence of cash declaration discrepancies arising from
clerks duplicating remittance in transactions (“Rem-in”) because of
wrong messages being presented on the Horizon counter screen
(acha621P*). This would result in incorrect cash amounts being
declared. This is again likely due to a bug because a software fix was
applied on 12 January 2016. However, this was not a retrospective fix
and therefore any similar, previous discrepancies were not remedied.
It is unknown how many other Post Office branches were affected or if
any communication was sent out by the Post Office to other branches
to prevent further occurrences prior to the fix being applied.
5.24 KEL LKiang3014S“ reports an issue with how the Horizon system
behaves when a Subpostmaster makes multiple cash declarations and
then runs a trial balance report. The calculation displayed for the
resulting trial balance was incorrect. The support department was
* acha1717T.html, HNG-X KEL acha1717T, 30 July 2010 (last updated 10 February 2015) [POL-0040033]
[5588ce1 3eb6fa9bbbdr986c912267fc0]
® acha621P.html, HNG-X KEL acha621P, 15 October 2015 (last updated 14 January 2016) [POL-0040340}
[7518fc27f6357689c500a302358d4452]
“ LKiang3014S.html, HORIZON KEL LKiang3014S, 27 November 2002 (last updated 22 February 2007)
[POLO035520] [19f87982637c6851bF104504935dfd39]
Prepared by: Jason Coyne
Occupation: Partne
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FU,
180503R1935 16 October 2018 Page xlviii of 225
FUJ00082162
1J00082162
unable to identify the root cause of the discrepancy although it was
reported that a correction could be made at the Post Office counter
level by redoing the cash declaration using the same amount already
declared. The KEL remains unresolved and is also cross referenced by
Fujitsu with MScardifield2219S*°, which identifies the underlying
software bug being caused by “cached data” not being updated via
Riposte. This resulted in incorrect data being presented in any
discrepancy, variance and balance reports. It is reported that the
problem should clear itself overnight and/or a manual workaround was
possible. In either case this scenario is in my opinion likely to be
confusing for the Subpostmaster and could lead to them making
modifications which are unnecessary if they are unaware that the
problem should clear itself overnight. It was stated in November 2007
that a fix was being piloted with a view to it being released in January
2008 but it is unknown if this took place and/or was successful.
5.25 Errors within cash pouch delivery could have also affected branch
accounts. In DSeddon5426P** a failure in pouch delivery resulted in a
cash gain when the Subpostmaster carried out a branch cash
declaration.
5.26 In achai94L*’ a problem affecting around 15% of kiosk branches
prevented these branches from being able to automatically make cash
declarations. The message being presented (Code: 2303 - Incorrect
Transaction Amount) was misleading since the totals were in fact
correct and the transaction amount checks were failing due to
incorrect data types. A bug fix was proposed for the second half of
2015, but it is unknown if this took place and what further
communications went out to affected branches.
45 MScardifield2219S.html, HORIZON KEL MScardifield2219S, 15 July 2005 (last updated 27 November 2007) -
[POL-0035721] [657dca342292a1eaee72e1f7f34a4582]
“© DSeddon5426P.html, HORIZON KEL DSeddon5426P, 26 June 2006 (last updated 10 October 2006)
[POLO035379] [ce2e6481384a7fe307e96bbc62f36048]
47 acha194L.html, HNG-X KEL achai94L, 18 November 2014 (last updated 01 April 2016) [POL-0040058]
[9a134a453319f4b473b37070dd7fb467]
Prepared by: Jason Coyne
Occupation: Partne
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page xlix of 225
5.27 I am also aware from the witness statement of Pam Stubbs** prepared
for the Common Issues trial that other potential problems with kiosk
operations E6uld’ have potentially affected branch accounts.
Bureau & Decimal Point Issue
5.28 Post Office has disclosed a PEAK which highlights a fault with bureau
transactions where an omitted decimal point in the “euro” line caused
a discrepancy. In this particular PEAK*® the Subpostmaster declared
€57,865.00 (expressed as “euro 5786500”), but the sterling
equivalent was £0.00 (expressed as “0.00”). When the Subpostmaster
tried to reverse the transaction, the figure doubled. It was noted in the
call log this problem had been seen on a couple of other occasions and
caused significant accounting problems. It was due to be fixed as part
of the S80 release.
5.29 A detailed PEAK log is available at Appendix A.
Reference Data Errors
5.30 Reference Data is changed frequently within Horizon, I understand at
least once per day. Post Office reported that until 2017°° the process
of changing Reference Data was conducted without being subject to
formal controls. Evidence suggests that Reference Data has been
found to be incorrect within Horizon and that this led to discrepancies
within branch accounts. Within the document, ‘Counter Type X
Reference Data Definitions’', it is explained that “Counter Type
Reference data... will usually only change on the back of a Change
Proposal or Live PEAK fix”. Examples have been found in KEL/PEAK
records of Reference Data having an impact upon daily counter
activities (DSeddon314Q**). These
“8 Pam Stubbs (174) - Witness Statement and Exhibit.pdf
* PC0098844.htmI, Peak PC0098844, 6 February 2004 [POL-0270879] [050c1d940ddd970bf7ed304afc494faf]}
5° Operations Board 21 July 2017.pdf, Operations Board 21 July 2017, 21 July 2017 [POL-0221328]
[9c45e0be3#f2b6773447cc6e41db5F46]
51 DESGENSPE0003_2.doc, Counter Type X Reference Data Definitions, 21 October 2010 [POL-0118364]
[ac2409e9e7be10cf2ca62c338b8bd82c]
52 DSeddon314Q.html, HORIZON KEL DSeddon314Q, 14 March 2016 [POL-0035128]
Prepared by: Jason Coyne
Occupation: Partne r;
Specialist Field: IT Systems ¢ 41 UU
On the Instructions of: Freeths LLP ~ i
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page I of 225
vary in severity and these errors are passed to the data reference
team for rectification.
Supporting Documentation
5.31 Errors in Reference Data and/or Reference Data validation has affected
counter activities in several different ways. By way of an example in
johnbascoG5222H*® an Automated Payment transaction was reported
as having failed due to “Unknown Agent Code 3046”. Client account
code 3046 was found to not exist in Reference Data and the fault was
not reproducible when the problem was analysed and tested. It was
acknowledged that, due to the business impact, a fix would be
provided to check and validate the client account code exists in
Reference Data before the transaction is committed.
5.32 KEL achai0L** documents how branches were unable to accept cards for
rent and council tax payments due to incorrect Reference Data,
although in this circumstance no impact on the branch accounts in
terms of discrepancies would occur as the cards were just rejected
completely.
5.33 KEL MWrighti458Q°° illustrates where withdrawn products (with
withdrawn Reference Data) can affect a branch accounting position
because the Subpostmaster will have products that cannot be
accounted for as there is no remaining reference for them to later
declare that stock item in the accounts.
5.34 Identified in KEL wbra5353J°°, the customer was charged twice for the
same transaction which was reported to be a side effect of errors
within Reference Data.
[12a0d196f1ba8968780b9eb845fa575f]
5 johnbascoG5222H.html, HNG-X KEL johnbascoG5222H, 11 July 2017 [POL-0040923]
[05ee26859196eff653a7830d82da7db1)
5 acha10L.html, HORIZON KEL acha10L, 01 April 2009 [POL-0036378]
[ed3160ac33e6f72cde5387980224249b]
55 MWright1458Q.html HORIZON KEL MWright1458Q, 15 June 2000 [POL-0035611)
[2cf4ea95b637e9acea61f7c771ffb355]
5° wbra5353J.html, HNG-X KEL wbra5353J, 10 April 2014 (last updated 17 April 2014) [POL-0039777]
[abac77a5158c65e27b40fc528d985bF4]
Prepared by: Jason Coyne
Occupation: Partne
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page li of 225
Duplicate Transactions
5.35 There is evidence of duplicate transactions existing within Horizon due to
incorrect processing.
5.36 The error appears to have occurred when payment data was being
harvested prior to being transmitted to an external client. The
harvesting process should have identified duplicate transactions and
dealt with these appropriately. However, any disruption in service can
cause the harvesting process to send duplicate payment transactions.
5.37 This would result in a discrepancy between POL and the external client
where client summary payment files were submitted to an external
client prior to the duplication issue being noticed.
Supporting Documentation
5.38 GMaxwell3651K*” and surs357P°® both demonstrate that any failures or
interruptions in service with the harvesting process can cause
duplicated payment transactions to be processed. In the latter case
Streamline, the external third party, received a payment file with 835
duplicate payment transactions totalling £76,564.53. In this instance
Streamline identified the duplicates and prevented payment
processing. However, identification occurred after the transaction data
had been transmitted from Horizon and despite any processes
designed to prevent this.
Failed Recoveries
5.39 The recovery process is outlined at section 4.54 above and the
importance of this and the error scenarios where recovery might apply
are further highlighted in a report by Gareth Jenkins.®® Any failure in
the recovery process can affect transaction data integrity and to
5” GMaxwell3651K.html, HORIZON KEL GMaxwell3651K, 22 December 2004 (last updated 8 May 2006)
[POLO035194] [654f590f7d2119bf963a321660731 icf]
5° surs357P.html, HORIZON KEL surs357P, 3 April 2009 (last updated 6 April 2009) [POL-0036388]
[4404b260b27528c3caa99a6acBaaff3a]
® 1,9 9. Horizon Online Data Integrity for POL, Gareth Jenkins.pdf, Horizon Online Data Integrity for Post
Office Ltd,2 April 2012 [POL-0021989] [0690d38ae4f1d9c949aaf1618c732d06]
Prepared by: Jason Coyne
Occupation: Partne r;
Specialist Field: IT Systems ¢ 41 UU
On the Instructions of: Freeths LLP ~ i
FU,
180503R1935 16 October 2018 Page lii of 225
FUJ00082162
1J00082162
attempt to mitigate against this a failed recovery report exists to set
out any instances where this has occurred.
5.40 The witness statement of Angela Burke® documents a recovery process
failure within Horizon. Communication issues within Horizon resulted in
a transaction that authorised and processed at the counter failing to
recover within the Branch’s accounts. This subsequently resulted in a
discrepancy that Mrs Burke had to seek to reclaim from Post Office in
order to balance the discrepancy.
Supporting Documentation
5.41 The Horizon KEL jharr832S acknowledges that the recovery process is a
complex area:
“Clerk needs to follow POL business rules properly. It is a complex area
and various factors can affect whether recovery happens or not
depending on exactly when it crashes and what the clerk replies”.
5.42 KEL cardc464Q® reports the difficulty the clerk may have faced when
trying to process recoverable transactions. In this case, recovery was
attempted but failed. Whilst the recoverable transaction appeared on
the failed recovery report, it is now out of the hands of the
Subpostmaster. In this instance recovery failure had no impact on
branch or customer accounts as settlement had not written to the
BRDB. Had recovery run successfully, a zero-value transaction would
have been written to the database which should not affect branch
accounts as it is simply a record entry with zero financial value.
5.43 KELs seng2037L & acha959T”®”! describe how various transactions states
may also indicate a failed recovery or an incomplete transaction
awaiting recovery.
*®° Witness Statement of Angela Burke 28.09.2018.pdf
®1 jharr832S.html, HORIZON KEL jharr832S, 05 March 2007 [POL-0035531]
[4f4c42852071f3957a69eb4a6dbceb8e]
® cardc464.html, HNG-X KEL cardc464Q, 30 April 2010 (last updated 12 January 2011) [POL-0038234]
[24fe6e3e3901ace35658ba1d3bdc420e]
Prepared by: Jason Coyne
Occupation: Partne
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FU,
180503R1935 16 October 2018 Page liii of 225
FUJ00082162
1J00082162
5.44 dsed4733R” provides an example where transaction recovery failed due
to a wrongly named recovery script. This was a Horizon system error
arising because of incorrect Reference Data that had to be corrected
by the Reference Data team. It is unknown when this was corrected
but, in the meantime, SSC would need to have cleared the failed
recoveries daily.
5.45 There appears to be a high risk of failed recoveries arising because of a
failure to follow correct procedures or lack of understanding of the
recovery process. The failed recovery report should pick up these
recoverable transactions but by this point it is no longer something
that can be resolved at the counter.
5.46 There are a number of PEAKs’? PC0203676, PC0263451, PC0266575 &
PC0273046 (detailed PEAK logs available in Appendix A) which identify
recurring failed communications issues subsequently resulting in failed
recoveries and consequently branch discrepancies. In these situations,
manual reconciliation was required, and Fujitsu needed to clear the
failed recovery transaction.
Failed Reversals
5.47 Reversals are set out in Section 4.61. In summary, they are the ‘undoing’
of transactions to reverse the value of it.
7° seng2037L.html, HNG-X KEL seng2037L, 1 February 2013 (last updated 7 February 2013) [POL-0039307]
[bb773d730ae5943330329bc96d2a0fac]
7 acha959T.html, HING-X KEL acha959T, 28 February 2010 (last updated 19 October 2017) [POL-0041091]
[698d3dbfe4181592c650af60f92c1a11]
7 dsed4733R.html, HNG-X KEL dsed4733R, 25 July 2013 [POL-0039482]
[78F45c50b543fb3673d9a18fe442eb37]
7° PC0203676.htmi, Peak PCO203676, 31 August 2010, [POL-0373467]
[9c65a8e1e33e636bc6ae37 2aefed3690}, PCO263451.htmI, Peak PCO263451, 19 October 2017, [POL-0430967]
[981b785aa6f78e75a5f19¢8a2c70aff5] PCO26E575.html, Peak PCO266575, 26 January 2018, [POL-0433904]
[8d02a629313fa13f86f853d49e5Sdcc7] PCO273046.html, Peak PCO273046, 15 August 2018, [POL-0439981]
[c4330f4fcc4ea9f37368e4c692730828]
5.48 Whilst there is evidence of failures in respect of the electronic processing
of reversals (discussed in Section 4 above), it is observed that there
Prepared by: Jason Coyne
Occupation: Partne
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page liv of 225
were also issues with interpretation and identification of system
reversals.
5.49 The document (“Helen Rose report”)® refers to an incident where a
Transaction Correction was issued which the Subpostmaster duly
settled financially despite the Subpostmaster denying conducting the
reversal.
5.50 The report appears to show that the material that Post Office initially
reviewed did not identify that it was the system that initiated the
reversal rather than the Subpostmaster and therefore the Transaction
Correction making the Subpostmaster liable was issued in error. Since
this is effectively a failure to appropriately reduce the risk of error this
is also dealt with further at Section 5.167.
Supporting Documentation
5.51 There is evidence that there were software issues resulting in Horizon
applying the wrong mathematical sign when reversing transactions
(i.e., a plus (+) rather than a minus (-)). This was identified in
PSteed2847N™. In this instance, the issue resulted in the doubling of a
remittance value in transactions that should have been reversed,
effectively to zero. It is unclear when this software bug was fixed or
how widespread the problem was.
5.52 KEL cardc5756N® provides an example where the system failed to
reverse all items in a multi-line pouch and only the first item was
reversed. It was reported that the clerk had appeared to have followed
the correct process, but the problem could not be reproduced on a
test counter. A Transaction Correction was required to reverse the
remaining
® Horizon data Lepton SPSO 191320, 12 June 2013 [POL-0221677, POL-0221678, POL-0221679, POLO221680,
POL-0221681] [f296f6880e1b8418f37d3e344374c42a]
6 PSteed2847N.html, HORIZON KEL PSteed2847N, 28 April 2003 (last updated 20 June 2003) [POL-0033658]
[1912a565b4a3c4d601bd18b62b15ce04]
55 cardc5756N.html, HORIZON KEL cardc5756N, 19 February 2008 [POL-0035846]
[6f87ead560bbed157b55348b32214c73]
Prepared by: Jason Coyne
Occupation: Partne
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FU,
180503R1935 16 October 2018 Page lv of 225
FUJ00082162
1J00082162
items. The KEL remains unresolved and it is not known whether any
further examples of this were reported.
Uncategorised Bugs/Errors/Defects
5.53 There are various KELs and PEAKs relative to errors that have not been
categorised as those above. Similarly, no cause or resolution is
identified within the KEL to assist with further determination.
Supporting Documentation
5.54 Foreign currency discrepancies were noted in GCSimpson1049L.® All
currencies on hand doubled up following successful balancing eight
days previously. The KEL record stated the issue as being “under
investigation” but there is no detail of the results of any analysis.
5.55 There is evidence that in certain investigations there was insufficient
diagnostic data to be able to fully diagnose an issue. In KEL
MHarvey35271°” an apparent successful transaction failed to be
validated due to a change of mode. It was reported that the change of
mode that had occurred could not be explained as the APS code would
not allow this and there was no other way to diagnose the issue. The
associated PEAK (PC113202) was closed as no further progress could
be made and it was treated as a “one off”.
5.56 In CObeng1123Q® © unexplained discrepancies (gains) for different stock
unit types (Cash and Stamps) was reported. Horizon system memory
faults appear to be the suspected cause in this case, but no suitable
alternative causes or factors were analysed, and the incident remained
unexplained. No advice is detailed to have been provided to the
Subpostmaster.
5° GCSimpson1049L.html, HORIZON KEL GCSimpson1049L, 29 April 2004 (last updated 5 May 2004)
[POLO034206] [77ee561a4d4b623b80b2fc1626F1ce9e]
7 MHarvey3527I.html, HORIZON KEL MHarvey35271, 21 January 2005 [POL-0034494)
[82a49bbb995141 0fedf65d51e314a091]
6 CObeng1123Q.html, HORIZON KEL CObeng1123Q, 14 August 2000 (last updated 15 January 20014) [POL-
® ] [cceaa39cee86736e8a4735e44ae328ac]
Prepared by: Jason Coyne
Occupation: Partne r;
Specialist Field: IT Systems 6 j1 UU
On the Instructions of: Freeths LLP ~ i
FU,
180503R1935 16 October 2018 Page Ivi of 225
FUJ00082162
1J00082162
5.57 There are a number of PEAK records (i.e., PC0063723 & PC0084116”°)
which were believed to be system related bugs causing discrepancy
transactions to be calculated twice. The examples highlighted both
refer to KEL DRowe1625K which (according to the log in PC0084116)
was never properly resolved. It is also worth noting that there are
many more PEAK records associated with this KEL and the examples
highlighted span a period from March 2001 to November 2002. This
appears to be an issue that has never been fully resolved despite
having been passed to development to analyse further. There appears
to have been a workaround put in place, but it is not known how the
workaround was communicated and if or when the underlying cause
was ever resolved. A detailed PEAK log is available at Appendix A.
5.58 Similarly, in PEAK PC00278877! a known but unresolved software error
caused a doubling up of values in cash account periods. It is not clear
from the call logs how Fujitsu resolved the branch discrepancies and
the PEAK was closed after 12 months with the following comment:
“Closing call on basis of insufficient evidence. As this is such an old call
I have not contacted the call originator. I suggest that this call remains
closed!”
5.59 In PEAK PC0203131” differences between volumes and values in a
branch office snapshot was identified as a bug in Horizon carried
forward to Horizon Online. Since this was a pre-migration bug (as
acknowledged by Gareth Jenkins) the PEAK was closed. It may be that
this issue was resolved in Horizon Online but this cannot be confirmed
on the existing evidence.
7° pC0063723.html, Peak PC0063723, 10 March 2001, [POL-0238257] [1efaafc05039eea7a5e5e09d1d50226c]
& PC0084116.htmI, Peak PC0084116, 23 November 2002, [POL-0256970]
[fc868322664da1770b9416c6443bb468]
71 pC0027887.html, Peak PC0027887.htmi, 21 July 1999 [POL-0221773]
[93af66e221ecfde6fcaad8a5aci4eca4]
72 PCO203131.html, Peak PCO203131, 18 August 2010 [POL-0372925] [6ea70fc9c6b34cbcd61b2a7b2ddb2628]
Prepared by: Jason Coyne
Occupation: Partne r;
Specialist Field: IT Systems 6 j1 UU
On the Instructions of: Freeths LLP ~ i
FU,
180503R1935 16 October 2018 Page lvii of 225
FUJ00082162
1J00082162
Hardware Errors
5.60 Throughout the lifespan of Horizon, it is observed (particularly through
Second Sight reporting commissioned by Post Office”*) that counter
equipment was unreliable. This observation was reinforced in a Post
Office IT risk register document” which identified branch IT hardware
as being very old and requiring urgent replacement.
5.61 The issues are broadly categorised as:
i. Printer failures
ii. Screen misalignment (pressing one on screen button but
resulting in the system selecting a different function)
iii. I Failed communications links
5.62 A detailed self-help manual’? 7° for troubleshooting and replacing
peripheral hardware equipment (keyboard, printer, monitor and
PinPad) appeared to be available for Subpostmasters. However it is
not known how widely this was distributed to Subpostmsaters and how
often it was updated.
5.63 A Spot Review report’? conducted by Second Sight Support Services
Limited gives further credence to the possibility that hardware issues
could have been responsible for branch discrepancies (shortages and
surpluses). In the example highlighted in the report, following a
sudden spike in branch discrepancies the Subpostmaster implemented
rigorous control improvements including performing twice daily
balances and installation and review of CCTV and CCTV film. Despite
this, no root cause was ever identified and there was no evidence that
73 POL Interim Report Signed.pdf, Interim Report into alleged problems with the Horizon system (Page 6), 08
July 2013, [POL-0022308] [8dd44e3ficc26efc1c27e09ee960d737]
74 TT risk register 2011 09 19 updated.xls, v1.2 - Risk Ref: 29, [POL-0219381]
[ec517091d83be38a59b167f5cffa02ad]
75 Self Fix Manual final.docx, Peripheral Trouble Shooting and Replacement Guide, 8 December 2011, [POL-
9]
[aaf6a16dfa038ee20e77bcf4bb26f95d]
7” 6.8 58. Spot Review 25 - Paul Popov - Mysterious shortages - v4.pdf, Horizon - Spot Review,
[5852c951bde464932c83764d1a0dadb1}
Prepared by: Jason Coyne
Occupation: Partne
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FU,
180503R1935 16 October 2018 Page lviii of 225
FUJ00082162
1J00082162
POL had conducted any root cause analysis. The Subpostmaster had
suspected that faulty Horizon equipment could be the cause, but no
firm evidence was ever provided. The Subpostmaster had asked for
replacement of old hardware, but this had been refused by Post Office.
Subsequently the branch was flooded, and hardware was replaced due
to flood damage. Following the hardware replacement and the
reopening of the branch with the same staff and processes no further
shortages occurred and all transactions balanced. If this evidence is
correct, it would be consistent with the Subpostmaster’s view that
suspect faulty hardware was responsible for causing his shortages.
5.64 Second Sight also considered hardware issues in their report’® and were
unable to draw a firm conclusion on whether faulty equipment could
be responsible for otherwise unidentified branch shortages, but they
also recognised this could be a possibility. I have noted that hardware
replacement often seemed to be a “fix” of last resort where no other
explanation could be given, and therefore there is certainly a
possibility that hardware was at fault.”°
5.65 Problems relating to the condition of the electrical power supply to the
branches has been acknowledged by Post Office, for example an
internal Post Office email POL-0030971 reports:
“..was too scared to accept a cup of tea in case the Horizon system
crashed cos [sic] the electric supply is still a live (excuse the pun)
issue... It is Horizon related - the problems have only arisen since
install & the postmistress is now barking & rightly so in my view”.
5.66 Further documentation also uncovers issues in respect of PIN pads and
base unit failures. In addition, unexplained system behaviour related
to
78 0.2 11. Second Sight - Briefing Report Part 2 (final).pdf, Initial Complaint Review and Mediation Scheme -
BRIEFING REPORT - PART TWO (Page 45 - 23. Hardware issues), 09 April 2015,
[3b79161b842d035f9b952bf70eb9433b)
7° RColeman4733L.html, HORIZON KEL RColeman4733L, 18 November 1999 (last updated 08 January 2004),
[POL-0033974] [5654ca357e27272f66887cdefab472f8]
Prepared by: Jason Coyne
Occupation: Partne r;
Specialist Field: IT Systems 6 j1 UU
On the Instructions of: Freeths LLP ~ i
FU,
180503R1935 16 October 2018 Page lix of 225
FUJ00082162
1J00082162
branch hardware was also reported®° and which lead to payments
crediting the wrong accounts. By a process of elimination conducted
by the Subpostmaster, this was narrowed down to a single branch
terminal card reader. The Romec engineer who subsequently visited
advised this was a “known Horizon error” and the card reader needed
to be rebooted. It is not known how widespread this issue was and
how many transactions were affected and/or when the underlying
cause of this error was fixed.
5.67 POL-0032853*' (authored in 2004) sets out the “/essons /earnt” in respect
of the entire lifecycle of the S60 (software) release of which there are
many hardware related issues. In particular, ID No 36 documents an
“Epson Printer Issue” resulted in Subpostmasters being unable to
declare their cash accounts.
5.68 There is a recurring theme relating to errors arising from PIN pad
failures. A selection of these can be seen in the following KELs
dsed525Q*, surs3941P®? and BrailsfordS239K**. In each case there
was a failure in the Subpostmasters being able to transact various
types of transactions including payment transactions using a PIN pad.
An error message and code were generated, and a new PIN pad was
the recommended solution.
5.69 cardc219R®° appears to indicate that any PIN pad related issues would
usually result in the recommendation of a new PIN pad whatever the
error. In this case a transaction had been declined by the PIN pad but
did not get reversed. An older version PIN Pad (Hypercom) was being
°° Petersfield.pdf, Report of upcoming loss at next T/P, 14 September 2014, [POL-0219802]
[b81bc528975e821221743fdd3d1edd28]
1 POL-0032853 Lessons Learnt from the S60 Release 15 December 2004 [POL-0032853]
[182f6b865d7707bd058328bf1f2f8c38]
®2 Desed525Q.html, HORIZON KEL desed525Q, 09 July 2009, [POL-0036489]
[352690d6c7d9a258bc02a6dt75d37254]
5 SURS394ip.html, HNG-X KEL susrs3941P, 14 April 2010, [POL-0037407]
[ce6e4d2bb070aea212d132d196bc3aba]
5 BrailsfordS2239k.html, HNG-X KEL BrailsfordS2239K, 14 June 2010 (last updated 01 July 2010),
[POLO037615] [30f83d46131ffd5d6 1d80a2b86c94cec]
®5 cardc219R.html, HNG-X KEL cardc219R, 11 May 2011 (last updated 31 October 2013), [POL-0039594]
[3e0e7a8604a8f093bec033f9c69¢766e]
Prepared by: Jason Coyne
Occupation: Partne
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FU,
180503R1935 16 October 2018 Page Ix of 225
FUJ00082162
1J00082162
used and a new version (Ingenico) was suggested and any
reoccurrence of the error would then need to be reinvestigated. It is
not clear in this case when or if a new PIN pad was issued and if this
fully resolved the issue. In this instance there could be an impact on
branch accounts if no reversal takes place and reliance then shifts to
reconciliation reports to pick up these discrepancies.
5.70 There are variety of other examples of counter hardware issues®° ®” where
5.71
5.72A
replacement equipment was the recommended solution. Hardware
(keyboard, screen & screen cable) replacement was also suggested as
a solution for an issue with “phantom” sales lines appearing on the
transaction that had never been selected.**
Additional power related Horizon issues are discussed in relation to the
Recovery and Reversal process discussed above.
2004 document®? describes the procedure to calculate the Mean Time
Between Failure (MBTF) for each component of hardware within the
Horizon Service Infrastructure deployed within Post Office branches.
This is stated to “allow Post Office to calculate the actual failure rates
applicable to each item of hardware.”
5.73 Section 4.2 of the document states:
“Fujitsu Services have applied adjustments to base unit data to
account for known problems within the Horizon System Infrastructure,
which have caused spurious fluctuations which should be ignored for
the purpose of forecasting the annual change. During the distribution
of release CSR+ and BI2, many base units were swapped out due to
software failure rather than specific hardware fault. They have since
8° RColeman566K.html, HORIZON KEL RColemanS66K, 04 April 2000 (last updated 08 January 2004),
[POL0033986] [65dbfc2631c46aafd78dbb69419431b3]. PCarroll2243R.htmI, HORIZON KEL PCarroll2243R, 06
April
8” (last updated 23 August 2005), [POL-0034763] [6e1dcee9b64aa61e07ad694e1329cb06]
®8 PSteed145J.html, HORIZON KEL PSteed145J, 07 January 2000 (last updated 06 January 200400,
[POL0033869] [56234dae4303708631c499c596bec86f]
® CSPRO149_3.1.doc [POL-0079278] [c4bffO8def03773e0501a4726d9f255a]
Prepared by: Jason Coyne
Occupation: Partne
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page Ixi of 225
been recirculated into the estate and Fujitsu Services have therefore
excluded them from the MTBF calculations.”
5.74 Annex 2 of this same document illustrates the MTBF results for January
2002 to December 2003. This annexure is not fully understood as
there are missing column identifiers that prevent confirmation as to
what the rates represent.
Fujitsu Closed Problem Records
5.75 Post Office has disclosed a contemporaneous spreadsheet entitled “Copy
of Fujitsu closed problem records.xls”°° which sets out around 200
issues which have been closed by Fujitsu in one year comprising of
both hardware and software issues. For example:
a. Issue ID 19 in the spreadsheet states in relation to a
reconciliation issue:
“Issues with First rate control files provided by Fujitsu. First Rate
Travel Service (Third Party) reconciles all Bureau de Change
Transactions and are currently unable to reconcile reversals against
original transactions due to this issue. The issue currently is caused by
HNG-X Bureau de
Change Transactions transferred from Branch Database to TPS host.”
b. Issue ID 78 refers to “Camelot file mapping issue causing
discrepancies”.
c. Issue ID 197 refers to token ID mismatches and states
specifically:
“Following an AP Ref Data update being enabled on Wednesday 1st
February. Post Office Card Account (POCA) transactions were unable to
complete and token IDs matched incorrectly for a number of
Automated Payment (AP) transactions, E Top Up cards and bank
cards.”
®° Copy of Fujitsu closed problem record.xls, [POL-0215915] [765f3677a7246da5dc9eafabc84f570a]
Prepared by: Jason Coyne
Occupation: Partne
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page Ixii of 225
5.76 As above, there are nearly 200 other issues contained in this spreadsheet
which have been tracked by Fujitsu. However, the spreadsheet only
appears to cover the period relevant to 2010 and
2011. My assumption is that Fujitsu would have had these (or similar)
trackers throughout the entire period that Horizon and Horizon Online
have been live, as this is how issues are tracked and fixed on a typical
project (and that appears to be the function here as well). However, I
have not yet been provided with similar trackers for other years.
5.77 I have filtered the content that appears to be relevant to the issues in
this matter and have set them out in Appendix F below.
5.78 In consideration of various witness statements, predominantly Richard
Roll, there were a wide ranging variety of possible bugs / errors /
defects within Horizon.
5.79 In the witness statement of Richard Roll®! he recalls:
“Any errors made by the Subpostmasters would be relatively easy to
identify and would normally be picked up by 1* or 2" line support. If
an error was referred to us, then it was extremely unlikely to be due to
a mistake made by a Subpostmaster, the vast majority of errors I
dealt with were due to coding errors or data corruption.”
5.80 He goes on further to state at Paragraph 9:
“We regularly identified issues with the computer coding in the Horizon
system. We would then flag those issues to the Fujitsu IT software
developers. The developers would then work on a fix” while we
monitored whole estate in relation to that issue”
5.81 In respect of financial discrepancies at Paragraph 10 he states:
“My recollection is that the software issues we were routinely
encountering could, and did, cause financial discrepancies at branch
level including shortfalls being incorrectly shown on the Horizon
system. If we were unable to find the cause of the discrepancy then
5 Witness Statement Of Richard Roll, 11 July 2016. (Para: 8, Page 2)
Prepared by: Jason Coyne
Occupation: Partne r;
Specialist Field: IT Systems 6 j1 UU
On the Instructions of: Freeths LLP ~ i
FU,
180503R1935 16 October 2018 Page Ixiii of 225
FUJ00082162
1J00082162
this was reported up the chain and it was assumed that the postmaster
was to blame.”
Horizon Robustness
5.82 For the purposes of this report, and in line with my definition given in the
Joint Statement, robustness is summarised as:
“The ability to withstand or overcome adverse conditions, namely, the
ability of a system to perform correctly in any scenario, including
where invalid inputs are introduced, with effective error handling.”
5.83 It is important to note that robustness does not equate to a guarantee
that software is bug or error free. A system's reliability can be
improved by rigorous testing and debugging (provided no further bugs
are consequently introduced through this process)”°? however,
complex systems can never be completely tested or ever entirely free
of bugs.?°°
This is due in-part to the fact that no software can ever be truly
perfect
101
5.84 The following statistics were reported in a Post Office presentation’
created circa 2017:
a. More than 47 Million transactions per week were undertaken in
Post Office Branches from 18 Million customer visits;
b. 22 Million banking transactions every month and 2.5 Billion
transactions a year with a cash value of £100 Billion;
c. Circa 11,800 Branches;
d. Post Office cash supply chain collect and deliver on average £42
Billion cash, foreign exchange and secure stock each year.
° F, Bott, A. Coleman, J. Eaton and D. Rowland, Professional Issues in Software Engineering, Boca Raton:
CRC Press, 2000.
% a. Hunt and D. Thomas, The Pragmatic Programmer, Reading: Addison Wesley Longman, Inc., 1999.
2.2 12. Presentation_The Post Office, An Insight_.pdf, The Post Office-An Insight, Angela Van Den Bogerd,
circa 2017, [05e2ac28f7b36b04dd83ab301edf9f91]
Prepared by: Jason Coyne
Occupation: Partne
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page Ixiv of 225
5.85
In a document last reported as edited on 20th August 2010” P.
‘ost
Office compares the error rate or “exception handing performance” of
Horizon compared to Horizon Online. In the document it is explained
that, “..the figures are averages over the whole system and do
claim that they will be evenly distributed”.
5.86 A number of failure analysis statistics are reported:
not
a. Counter Peripheral failures [in Horizon Online] - These are largely
the same as Horizon, estimated at approximately 1 failure per
counter per year.
b. Counter PC Failures [in Horizon Online]- These are largely the
same as Horizon, estimated at approximately 0.1 failure per
counter position per year that could cause loss of data. Note that
Power failure at the branch accounts for 80% of these cases.
c. Transient failures (< 2 minutes) impacting online transactions
[in
Horizon Online]- These are reduced for Horizon Online (estimated
at 6 per counter position per year) compared to Horizon
(estimated at 11 per counter position per year).
d. Transient failures (< 2 minutes) impacting settlement [in Horizon
Online]- This is a new exception category for Horizon Online that
does not apply to Horizon. The estimate is 15 incidents per
counter position per year.”
5.87 The document also records an ongoing error, “Loss of Basket transaction
Data held in PC memory” which results in an estimated loss
of
transactions in a basket per counter, per year of 0.1 and in Horizon
Online of 0.097.
5.88 In my position as an expert I am unable to estimate the level of the
Horizon system’s robustness. Given the size and age of Horizon
°* HNG-X Branch exception Handling Strategy- Agreed Assumptions and Constraints. [POL-0116897]
[6511184272a83cc7c16127dff44ac807]
I
,
Prepared by: Jason Coyne
Occupation: Partne ° re
Specialist Field: IT Systems 6 YI UU
On the Instructions of: Freeths LLP
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page Ixv of 225
would however make the expert assumption (based upon systems of
similar magnitude), that there are not many people who could. The
sheer enormity of the task to garner a thorough understanding of the
code, which would be required to estimate robustness is, in my
opinion, nearly impossible.
5.89 This is compounded by the agreed facts that there were 19,842 release
notes for Horizon between 29 November 1999 and 8 August 2018 and
each of these introduced changes could affect a system’s robustness.
5.90 For context, 19,842 changes over a 19-year period is approximately
1000 a year or 19 changes per week. It is readily apparent that
keeping on top of a constantly evolving system’s robustness is a
somewhat insurmountable task. It would not be possible to determine
whether a change to fix an area of the code did not:
a. Create a new issues/bugs with older areas of the legacy code
or;
b. Add brand-new issues/bugs to the system.
5.91 Instead, I have estimated the likely level of the robustness of Horizon
and benchmarked this against industry standards based upon a review
of the evidence available including the known error log (KEL) and
PEAK system.
5.92 Several KELs exist that identify failures of internal mechanisms in place
to ensure integrity of data. For example, dsed4733R°° identifies
multiple failed recoveries occurring because of a wrongly named
recovery script. From the section above, a robust system has to
“perform correctly in any scenario despite the introduction of invalid
inputs”. Clearly this KEL details an issue that is at odds with this
definition.
°° dsed4733R.html, HNG-X KEL dsed4733R, 25 July 2013 [POL-0039482]
[78f45c50b543fb3673d9a18fe442eb37]
Prepared by: Jason Coyne
Occupation: Partne ° ey
Specialist Field: IT Systems } 41 UU
On the Instructions of: Freeths LLP ~ i
FU,
180503R1935 16 October 2018 Page Ixvi of 225
FUJ00082162
1J00082162
5.93 Further, obengc5933K°* from 2010 shows that following network banking
(NWB) withdrawal transactions and printing of the customer receipts,
there was a loss of communications resulting in a message to the data
centre timing out. Consequently, the Subpostmaster was asked to
follow recovery but the transaction was only able to recover partially.
5.94 It goes on to state:
“It appears that the order in which txn [transactions] are recovered is
by recovering the most recent, then working backwards; however, the
oddity about this particular recovery process is the fact that the 5.00
txn (00-215704-1-4273066-1), the first in the session, was recovered
fully, whereas the £169.31 txn NOT!”
5.95 Clearly this shows potential for discrepancies but also a lack of absolute
robustness in Horizon. This is consistent with my opinion above that it
is unrealistic to expect any large IT system to be completely robust or
bug free.
5.96 Post Office themselves have established a department called the Data
Reconciliation Service. This department’s purpose is to deal with
Horizon system problems which have resulted in unreconciled
transactions which require some level of manual intervention. Post
Office report that 10,000+ transactions per week are processed by the
Data Reconciliation Service.
5.97 The fact that numerous processes and workarounds are in place to allow
Fujitsu to modify data already recorded by Horizon is consistent with a
lack of internal integrity within the Horizon system and the high level
of need to ‘correct’ this lack of robustness manually.
5.98 Post Office acknowledge the need to improve in a “Finance Roadmap
Project” document published in September 2012°” where under the
heading of “process and system gaps” it reports;
98 obengc5933K.html, HNG-X KEL obengc5933K, 12 May 2010 [POL-0038204]
[c24012c95dc42ac17b9ad2a2be2461b2]
®” Phase 2a) consolidated output.pdf, Finance Roadmap Project ~ Phase 2a) Project Outputs, 3 September 2012
[POL-0215782] [ac4469b27e384fe4440f351c115d8108]
Prepared by: Jason Coyne
Occupation: Partne r;
Specialist Field: IT Systems 6 j1 UU
On the Instructions of: Freeths LLP ~ i
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page Ixvii of 225
e “Multiple finance systems and a lack of automated controls...”
and
e “Significant amount of manual intervention in core Finance
processes.”
5.99 It is common ground between the experts that that each time there is a
change there is a potential to introduce new bugs/errors/defects.
5.100 The frequency of updates indicates the level of bugs and level of change
to which that Horizon was subjected to.
5.101 As discussed above, Post Office’s response to my request for
information confirmed that there have been 19,842 release notes, I
can see from POL-0219318° a brief note on each of these releases
and to which bug (PEAK) they relate. Therefore, Horizon has been
subjected to 19,842 changes which have been applied during its
operation.
5.102 The graph below is taken from the data contained within POL-0219318
and displays the bug fixes by year.
°° Copy of CallTypeR_080818-ReleaseNotes (2).xls, [POL-0219318] [e621d43d3f3b629be26536c6584c53d7]
Prepared by: Jason Coyne
Occupation: Partne vo .
Specialist Field: IT Systems ¢ 41 UU
On the Instructions of: Freeths LLP ~ i
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page Ixviii of 225
Peaks that led to software changes
1999
2000 2001-2002 20032008 © «2005 «2008-2007 «2008 «2009» 2010«2011-«2012,-«2013«2014« 20152016 «2017-2018
5.103 As noted in paragraph 4.21 above, the apparent ability prior to July
2017 to alter Reference Data without any formal consideration as to
the impact of this change could have had a potentially very significant
effect upon Horizon’s reliability and robustness.
5.104 Transaction Corrections also have the potential to affect the robustness
of the Horizon system. Torstein Godeseth’s witness statement® states
these are one of four sources of transactions that make up transaction
data within Horizon. Therefore, any human intervention within the
Transaction Correction process has the potential to introduce errors
and therefore affect the robustness of the Horizon system.
5.105 In addition to this, I have observed a summary of Transaction
Corrections between the years 2013 - 20141 which shows
approximately 20% of all Transaction Corrections raised appear to
have been deemed by Post Office as: “not caused by the branch”. I
have also noted that a different summary of Transaction Corrections
°° Witness Statement of Torstein Olav Godeseth 27 September 2018
2° TC summary by Product full year 2013-14.xlsx, Summary of transaction correction causes, 22 October 2014
[POL-0221563] [9e326d06b05870f076dd6fe8b69015c7]
Prepared by: Jason Coyne
Occupation: Partne
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page Ixix of 225
issued in 2010/11 sets out that the net value of Transaction
Corrections which were not categorised as “caused by branch” was
£7.4m*°!. Clearly, if potentially erroneous Transaction Correction data
(or error notice data pre-Transaction Corrections i.e. Horizon Online)
is entered into the Horizon system, this will affect its reliability and
robustness. Furthermore, if these erroneous Transaction Correction
transactions were not caused by the branch and entirely unknown to
the Subpostmaster there is a significant risk and issue that could
impact branch accounts.
Horizon Uptime
5.106 As maintained above, it is technically challenging and expensive to
achieve high availability of large systems. Horizon is a nationwide
system made up of many parts, which makes achieving the industry
standard of 99.5% difficult and the gold standard “five nines” (or
99.999%) almost impossible.*°
5.107 The 2013 Agreement between Fujitsu and Post Office'®? (at page 12)
aimed to achieve 99.53% Branch Availability.
5.108 I have seen reported examples of downtime across the Horizon system.
A Post Incident Report dated 09 December 2015 detailed a ‘POLSAP
Filesystem Incident’ on 5 January 2015 that resulted in a POLSAP
application outage for approximately 49 minutes.**
5.109 Likewise, there was an outage to Banking, Card Payment, E Top Up and
Automated Payments Out Pay (APOP) on 7 November 2016 for two
101 NEW TC PACK P10 2011.xIsx, Summary by Period, 15 February 2012 [POL-0221536]
[e1a3c3394d348ab3355302b35cfd63ab]
12 This is a common term used to describe percentages of a particular order of magnitude. For example, a
system or service that is delivered without interruptions 99.999% of the time would have “five nines”
reliability. See also POL Risk and Resiliency Review v1 5.pptx, Post Office Limited IT Risk and Resilience Review
Final v1.5, 08 June 2012, (Page 66), [POL-0219408] [25eef3b7ee67ce3fcfdcbf2dec402c9b]
12 Horizon OLA between POL and Fujitsu vO 2.pdf, Operational Level Agreement (OLA) Between Post Office Ltd &
Fujitsu for the Horizon Service, 22 July 2013[POL-0215475] [0700cd4e673159d3F251fcaa72323307)
104 SVMSDMINR2690_1.DOC, Post Incident Report: POLSAP Filesystem Incident 5th January 2015, 9 December
2015, [POL-0143426] [Sc6f3d6e2a51cd3dd8c263f85493cd04]
Prepared by: Jason Coyne
Occupation: Partne
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page Ixx of 225
minutes*©° in addition to a Tivoli Work Scheduler Batchman outage for
approximately six hours on 7 February 2017.'%
5.110 In my opinion whilst these instances show that Horizon is not infallible
nor totally robust, it still places Horizon with a good level of availability
overall.
5.111 Overall, from my review of the evidence it is likely (whilst putting aside
the large number of often required manual processes) that the
electronic processes within Horizon are relatively robust based upon
the literal, contextual, definition. Certainly, both instances of Horizon
(Horizon and Horizon Online) appear to be in-line with other IT
systems of similar size and industry.
5.112 However, this does not mean the Horizon system (as a whole) is
infallible and certainly does not imply the software is bug-free nor its
accounts free from errors. The extent to which Horizon is robust, in
my opinion, is reasonable but not a guarantee of no shortfalls or
branch account accuracy.
Likelihood of shortfalls in branches
5.113 Whilst controls and integrity checks are identified within Horizon, it is
evident that the Horizon system itself and errors within it have been
the cause of shortfalls attributable to branches. Although correcting
transactions are capable of being issued, the issuing of such so that a
Subpostmaster does not suffer the loss is largely dependent upon a)
the cause of shortfall being accurately identified b) the reporting and
logging facilities to identify such being a true picture of events and c)
the actual event being detected in the first instance. As evidenced
throughout this report, and in consideration of the Claimant’s witness
statements prepared for the Common Issues trial, there are examples
195 SVMSDMINR3299_1.DOC, Post Incident Report: Banking, Card Payment, E Top Up and APOP outage, 11
September 2017 [POL-0151983] [Oeeec8be8128ea7ace5034bacc2c7 14d}
195 SYMSDMINR3343_1.DOC, Post Incident Report: Batch processing outage, 22 September 2017 [POLO152261]
[10b424479186ab4b0e1e750460231f57]
Prepared by: Jason Coyne
Occupation: Partne r;
Specialist Field: IT Systems 6 j1 UU
On the Instructions of: Freeths LLP ~ i
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page Ixxi of 225
of various instances where part or all of a), b) and c) may not have
been effectively conducted.
Extent of errors in data recorded within Horizon arising from (a) data entry, (b)
transfer or (c) processing of data in Horizon
5.114 Regarding the extent of potential errors within Horizon I have analysed
5114 Horizon Known Error Logs (KELs) to determine the scope of
potential bugs or ‘PEAKs’ (as they are referred to by Post Office and
Fujitsu). Of these 5114, I have found that 163 contain PEAKs that
could be of significant interest and of these 76 are referred to in the
report. The KELs disclose that there have been actual errors in data
recorded within Horizon arising from transfer, processing of data in
Horizon and data entry. The potential for such errors therefore must
exist. See Appendix G (Failure Point spreadsheet).
5.115 Evidentiary findings in respect of points a), b) and c) (above) are set
out below. However, they do not fully identify the extent of errors in
data recorded within Horizon, only a section of those reviewed thus
far.
Further, it is not clear what the true extent of errors recorded within
Horizon is since the quantification of undocumented issues is not fully
known, for example bugs / errors /defects not reported by a
Subpostmaster but possibly accepted as an accounting error on their
part.
Errors that are Potentially the Result of Multiple Issues
5.116 KEL wrightm33145J'°” 1°8 (raised September 2010) entitled ‘Office has
Non-Zero Trading Position (Receipts/Payments mismatch)’ states:
“This issue was fixed in November 2010. For new occurrences see KEL
ArnoldA2153P.” (raised on 08 October 2009).
107 wrightm33145J.html, HNG-X KEL wrightm33145J, 23 September 2010 (last updated 01 April 2016) [POL-
408 } [1f025ec713c287ee7a5b1 7accd25b42f]
Prepared by: Jason Coyne
Occupation: Partne fe
Specialist Field: IT Systems ¢
On the Instructions of: Freeths LLP °
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page Ixxii of 225
5.117 It should be noted that although the Horizon error in wrightman33145J
was fixed and new occurrences were referred to ArnoldA2153P,!° the
latter KEL generates the same Receipts and Payments mismatch error
message but in fact relates to a mismatch during the balancing of a
stock unit that contains withdrawn product (relative to Reference Data
issues). There is no mention in the KEL if this was communicated to
branches as part of the Receipts and Payments mismatch issue.
Therefore, it appears a single “defect” could present itself as an error
in many ways.
5.118 Another example of a Receipts/Payments mismatch issue was raised in
ballantj1759Q"*° (February 2010) detailing a “Counter APP ERROR”
that has been caused historically by:
a. Falsely reported for the Office Snapshot, when a transfer is in
progress. Fixed by PC0194381 in April 2010;
b. Pressing Cancel at a certain point during stock unit rollover. See
wrightm33145J; and
c. Training Office producing a balance snapshot where data hasn't
been reset for a very long time (PC0210277).
5.119 What can clearly be seen here is that there are at least three issues
within Horizon that cause a Receipts/Payments mismatch that will
directly affect branch accounts. Further, the KEL goes on to state that:
“Instances of this error must be investigated. If the error is as a result
of a new problem, please add the details to the list of causes above”,
demonstrating that the true effects of this known bug may still be
unknown.
5.120 On a similar theme incorrect reporting of discrepancies can arise
resulting in incorrect stock declarations. See achai357Q."* It is not
1° arnoldA2153P.html, HNG-X KEL ArnoldA2153P, 08 October 2009 (last updated 29 March 2016) [POL0040401]
[fbf8040b134636ad64f8e68fbf4d706d]
200 Ballantj1759Q.html, HING-X KEL ballantj1759Q, 12 February 2010 (last updated 17 May 2011) [POL0038508]
[f4c2a317d5745 1cdc91ba81dc1072002]
41 acha1357Q.html, HNG-X KEL acha1357Q, 14 February 2011 (last updated 16 June 2017) [POL-0040896}
Prepared by: Jason Coyne
Occupation: Partne r;
Specialist Field: IT Systems 6 j1 UU
On the Instructions of: Freeths LLP ~ i
FU,
180503R1935 16 October 2018 Page Ixxiii of 225
FUJ00082162
1J00082162
clear how many branches were affected but it appears that it is
possible for discrepancies to have been accepted by the
Subpostmaster based upon incorrect declarations. The problem could
arise due to old stock declarations not being automatically removed
from the system. These could only be removed by making zero-value
declarations or deleting the stock unit then waiting overnight before
balancing. As a result, it was possible if a declaration existed for a
current period, it would be used when the discrepancies button was
pressed, or when a balance report was produced, even if it had not
been used for a year.
5.121 The KEL above indicates that the issue was passed to development via
PEAK: PC0208335 and the recommendation was for the
Subpostmaster to be contacted for advice on corrective actions.
acha3145Q'" which pre-dates acha1357Q by over year provides a full
support solution for this issue of incorrect stock declarations and
discrepancies. It would seem that the problem was known for at least
12 months before being passed to development and it is not known if
the issue was subsequently
fixed and/or how widely the corrective support actions/process was
communicated.
In relation to a) Data Entry
5.122 It is evident that data entry was clearly a significant problem at the
branch counter. We can see from the Service Level Agreement (SLA)
reports (i.e., SLA Summary New WE 06072014""%) and filtering for
“Correcting Horizon Errors” or “Miskeying” that there are numerous
instances of data entry errors made each week (Annex A). An SLA is a
commitment between a service provider and a client in respect of
[8109c05bc69b18eed896e45c3a2115a5]
12 Acha3145Q.html, HNG-X KEL acha3145Q, 18 May 2010 (last updated 02 October 2015) [POL-0040263]
[44b786a9d63674c53510487e50771172]
43 SLA Summary NEW WE 06072014.xIs, NBSC Incident SLA Summary - Week ending 6th July 2014, 14 July
2014 [POL-0031909] [2a6e1fbd3ef5d76899a9f21957d65011]
Prepared by: Jason Coyne
Occupation: Partne r;
Specialist Field: IT Systems 6 j1 UU
On the Instructions of: Freeths LLP ~ i
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page Ixxiv of 225
aspects of the service to be provided e.g. helpdesk response times. In
this instance the report can be used to analyse the type and reason for
calls made to the helpdesk.
5.123 An internal feasibility study report, carried out in 2012'*, was
commissioned to investigate the issue of mis-keyed transactions and
the options for preventing this problem. The report noted the following
financial impact:
“A further statistic, which was recorded on Friday 6th July 2012, refers
to the value of mis-keyed Banking Deposit transactions amount to over
60 PER WEEK. The total of investigations that become necessary as a
result of mis-keyed transaction equates to £10 millions per annum
(approx.).”
5.124 Further, an internal presentation from Post Office’!® looking into
efficiency gains reported: “a significant portion of demand at FSC is
driven by errors and mistakes made in branch with entering in data
into horizon. Part of these errors can be avoided with relatively small
changes to horizon". The presentation goes on to set out four changes
that could be made to Horizon that would save time for the
Subpsotmaster and reduce data entry errors in Horizon.
5.125 A further internal presentation from Post Office’ references the
findings of external consultancy firm McKinsey. This presentation
repeated the statement made previously that relatively small changes
to Horizon could avoid errors/mistakes made in branch. The
Presentation suggested that the ease of implementation of such
changes as; “Medium. There exists an interdependency on IT for
changes in Horizon to reduce errors coming from upstream” with the
next steps described as; “Assess costs for Horizon changes”. The
114 Feasibility Study - Mis-Keyed v0 1.doc, G-231 Mis-Keyed Project - Feasibility Study V0.1, 15 May 2012,
[POL-0217750] [8e9f114b3c4106d0f5255f906b742731]
45 Review of Mid-Term Initiatives [POLO217407] [670bcc305bb2ee89d5a7546faa0e86e9]
46 Business transformation - Finance Wave 1 [POL-0218441] [a26aQecOf6c8dbc20b753b200de725bb)
Prepared by: Jason Coyne
Occupation: Partne
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FU,
180503R1935 16 October 2018 Page Ixxv of 225
FUJ00082162
1J00082162
presentation is focused upon delivering costs saving to Post Office,
through reducing support costs and did not appear to investigate any
saving which might be achieved by Subpostmasters.
5.126 An external information security review!” was carried out in 2008 by
Infosec as part of POL’s adoption of the International Standard BS
ISO/IEC 17799:2005. The resulting report recommended various
system improvements after concluding:
“The Post Office, its agents, clients and banking partners are suffering
the consequences of a high level of transaction disputes and customer
claims across many financial, and all banking products due to a lack of
source data integrity, i.e. values entered only once without validation;
transaction value not presented to customer for validation......... banking
deposits are not visually presented for confirmation by the
customer.....bill payment and other transaction values are not
presented to customer....”
5.127 The report also highlighted the short time period available for archived
data (180 days) hampering the ability of disputes to be resolved in a
timely manner and it recommended an increase to at least 540 days.
It
is not known if or when all the report’s recommendations were
completed by POL.
5.128 Similarly, in a fraud solution report!!®, it was noted that the POL had a
limited number of data queries (approximately 750 per year) which it
could request relating to transactional data in the branch database
which had been archived after two months. Beyond this number of
queries, each request became chargeable (£400-£500). It is not
known what impact the time limits on archived data and limits on the
number of non-chargeable data queries had on POL’s willingness to
47 Information Security Review - Post Office Ltd, v1.2, 15 TH July 2008, [POL0217567]
[3dd3d32cb258ecf5895d94b2d205ee9d]
18 NRRA1207.01 DOO1 - Post Office Fraud Solution report.pdf. Driving business benefits through the
consolidation of data review - Post Office Fraud Solution, 18 May 2012, [POL-0219392]
[Seb6f3175483e47d760c4fd44a6c06b5]
Prepared by: Jason Coyne
Occupation: Partne r;
Specialist Field: IT Systems ¢ 41 UU
On the Instructions of: Freeths LLP ~ i
FU,
180503R1935 16 October 2018 Page Ixxvi of 225
FUJ00082162
1J00082162
investigate all branch discrepancies on behalf of Subpostmasters,
however, the fact that business constraints existed could give an
inference that there may have been limits on the numbers and types
of investigation of Subpostmaster discrepancies.
Supporting Documentation
5.129 allendi645p*"® provides an example of Horizon’s weak interface controls
and lack of data entry validation. In a single sales transaction the
clerk was able to select and enter different methods of payment
(Debit Card and Fast Cash). Horizon allowed the transaction to be
settled via Fast Cash when the Debit Card payment method had
already been selected. Although the KEL confirms this is expected
system behaviour and to advise the caller of the same, it is my
opinion that controls should be in place to restrict user input error in
this scenario. There is nothing in the KEL to indicate if this could be
considered for a future system enhancement.
5.130 In acha621P**° the correct screen to successfully process a cash pouch
did not appear resulting in the clerk in an outreach branch
inadvertently doubling up the amount of cash recorded. The issue
appears to have been caused because of an earlier system logout or
inactivity which in turn resulted in incomplete checks being conducted
by Horizon post logon. There was a system fix applied for this on 12
January 2016, but this did not retrospectively correct the affected
branch accounts. It is not known how many branches were affected by
this bug/data entry validation issue and/or what communications were
sent out by Post Office. The KEL log remains in a status of
‘Unresolved’.
49 allend1645P.html, HNG-X KEL allend1645P, 24 June 2011 [POL-0038584]
[8a983ddfa657b62026d5bfc7ad78ba04]
120 acha621P.html, HNG-X acha621P, 15 October 29015 (last updated 14 January 2016) [POL-0040340]
[7518fc27f6357689c500a302358d4452]
Prepared by: Jason Coyne
Occupation: Partne r;
Specialist Field: IT Systems 6 j1 UU
On the Instructions of: Freeths LLP ~ i
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page Ixxvii of 225
5.131 This issue looks to be part of the same scenario analysed in a detailed
Fujitsu report!*4!?2 prepared for the Post Office. The underlying
symptom (duplicate transactions) caused either because of a “Forced
Log Off” or use of the “Previous” key during the remittance process
analysis covered a period of five years (2010 - 2015). It was found
that 88 different branches had duplicate pouches over this period.
Although most occurrences appear to have been accounted for and
corrected by the Post Office the fact that three “fixes” over five years
(the last being in January 2016) is indicative that the issue was never
fully resolved, albeit most occurrences happened between 2010 and
2011.
5.132 EJohnson3937R'3 demonstrates the lack of Horizon interface controls
which enabled Subpostmasters to carry out “Rem In” transactions
without a value being entered. Despite the system check message, I
would expect controls to be in place to restrict user input error in this
type of scenario so that a user cannot complete the transaction. There
is nothing in the KEL to indicate if this could be considered for a future
system enhancement.
5.133 PSteed145J’** highlights an issue reported on several occasions with
“phantom” sales items appearing on the Horizon counter screen but
which had not been selected by the Subpostmaster. In this instance it
was recommended to replace both the keyboard and screen and it was
also noted that another similar case had been caused by the cable
connecting the screen and base unit. Instructions on how to deal with
environmental issues and hardware are contained in pcarroll1235R*°
21 Qutreach BLE Extract Findings v6 091215.pptx, Branch Outreach Issue (Initial Findings), 10 December
122 I [POL-0220141] [33ab9fe9c4b2bcd600Fb50b7aff7bd8a]
123 EJohnson3937R.html, HORIZON KEL EJohnson3937R, 27 January 2005 [POL-0034505]
[2d7cfc706e00fbd9d7d7d64021b1dd04]
128 pSteed145J.html, HORIZON KEL PSteed145J, 07 January 2000 (ast updated 06 January 2004) [POLO033869]
[56234dae4303708631c499c596bec86F]
225 Pcarrolli235R.html, HORIZON KEL pcarroll1235R, 05 April 2000 (last updated 05 October 2006)
[POL0035366] [3b14d838abd6aa7273dad4894aafcbdd]
Prepared by: Jason Coyne
Occupation: Partne
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FU,
180503R1935 16 October 2018 Page Ixxviii of 225
FUJ00082162
1J00082162
but it is not known how widely these instructions were
communicated/distributed to Subpostmasters.
In relation to b) Transfer (of data)
5.134 Evidence shows various issues with the transfer of data within Horizon.
Supporting Documentation
5.135 cardc219R’*° records a transaction authorised by the bank but cancelled
by the Horizon counter PIN pad and the transaction did not get
reversed. The error is noted to have occurred due to missing key
transaction information and possibly due to issues related to
Hypercom PIN pads. It is not clear if the underlying cause was
pinpointed to the Hypercom PIN pad and/or if the later switch over to
the Ingenico PIN pad resolved the issue.
5.136 jharri323L'”’ records an unresolved example of a successfully recorded
transaction initiated in a Post Office branch (where a customer receipt
was generated) which failed to appear in the Post Office Data Gateway
(PODG)'*51?9 file and consequently was not transferred to the relevant
third party (Environment Agency). The PODG file should contain all
transactions to be transferred to the third party. In this case the
Environment Agency failed to send the customer his license as they
had no record of any purchase. Post Office’s failure to hold logs for
more
than 30 days affected Fujitsu’s ability to fully investigate the root
cause of this specific incident. The incident remained unexplained.
. The ability of Horizon to erroneously record the same transaction
twice after a session transfer to a different counter was recorded in
226 Cardc219R.html, HNG-X KEL cardc219R, 11 May 2011, (last updated 31 October 2013) [POL-0039594]
[3e0e7a8604a8f093bec03319C69c766e]
2 jharri323L.html, HNG-X KEL jharr1323L, 29 September 2016, [POL-0040563]
[afeb10ec3e6c812eecf9c7011abaaad4]
428 1,3 3. High level architectural overview of Horizon Online.pdf, Horizon Solution Architecture Outline (Page
129 _ 2.2.2.4.1), 07 April 2016 [C-0003645] [7b6cf8cf69bec90f674b9b10a64f04e8]}
Prepared by: Jason Coyne
Occupation: Partne r;
Specialist Field: IT Systems 6 j1 UU
On the Instructions of: Freeths LLP ~ i
FU,
180503R1935 16 October 2018 Page Ixxix of 225
FUJ00082162
1J00082162
KEL MArris3433I.*°° This happening with both NS&I (National Savings
& Investments) and Network Banking (NWB) transactions was
acknowledged. The KEL was passed to a development team to provide
a bug fix as part of the S60 rollout but it is unknown if this was ever
resolved.
In relation to c) Processing of data in Horizon
5.138 There were various issues with the processing of data within Horizon.
Supporting Documentation
5.139 KEL CharltonJ2752T'*! identified a Horizon bug which became apparent
when any counter level corrections made via the “Previous” key led to
both the old value and amended value being stored and used in error
in the transaction. According to the KEL, a fix was released in the Live
environment eight days after the issue was first raised. It is not known
how many undetected records were affected and if any further
instances were reported following the fix.
5.140 A recently introduced Post Office service “Drop & Go” was shown!” 173 to
5.141
have the ability to credit cash (“money out of thin air”) to the branch
account in circumstances where the session had timed out. It is not
known how widespread the issue was, but the referenced document
discusses the various options the Post Office were looking at to resolve
this and other issue with the Drop & Go service.
SSur343P'** records an example of a declined network banking
transaction that nevertheless resulted in money being taken from the
customer’s account. The transaction was automatically reversed
correctly at the counter but continued to be processed further through
18 MArris34331.html, HORIZON KEL Marris3433I, 10 December 2003 [POL-0033825]
[0e0193F167f883fd85bd79236da96913]
11 CharltonJ2752T.html, HNG-X KEL CharltonJ2752T, 04 October 2011 (last updated 16 April 2013),
[POLO039383] [01e9c3402e3301825e37cc68e61b08ca]
12 Qverview v004.pptx, Drop & Go - Settle to Cash Resolution - issues and options, 07 October 2016, [POL-
189 ] [4266a3c3129dc5f258dfe7d'888b446b]
4 SSur343P.html, HORIZON KEL SSur343P, 30 April 2003 (last updated 16 June 2004), [POL-0034256]
[f4355bfbba9c34885a97bc7c9f096671]
Prepared by: Jason Coyne
Occupation: Partne
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FU,
180503R1935 16 October 2018 Page Ixxx of 225
FUJ00082162
1J00082162
Horizon, resulting in the customer account being debited. This incident
would have required the raising of a BIM (Business Incident
Management). The eventual discovery of a root cause and resolution
of this issue is not known but clearly there were errors in data
transfer.
5.142 There are a number of examples***?° of E-Pay transactions crediting
the customer account twice although only one payment has been taken.
The error arose because both Horizon system authorisation agents were
incorrectly active at the same time (normally one is active and the other
on standby). This error should have been detected via an E-Pay report for
reconciliation but remains outside of the control of the Subpostmaster,
who would have been unaware of the initial error.
5.143 PEAK PC00632271*’ (detailed PEAK log available in Appendix A)
highlighted a bug within the Horizon messaging service (Riposte)
which prevented 401 transactions with a total value of £11,708.08
from being processed and impacting on branch reconciliation. A
workaround was applied as a short-term fix pending testing of a long-
term fix.
5.144 POL-0216412'°* documents the POLSAP outage that occurred in January
2016 resulting in millions of pounds worth of transactions failing to
process causing backlog and discrepancy for both Post Office and its
customers.
5.145 The attempted system recovery prevented access to the majority of
system functionality. Completion of all accounting recovery actions
was due by 29 January. It is not known what actual impact this had on
branch accounts, other than not being able complete any transactions.
©5 LKiang3526R.html, HORIZON KEL LKiang3526R, 25 August 2004 (last updated 7 April 2005) [POL0034590]
[67efbfdfc186487e866a8d701cd353e4]
6 SSur5310P.html, HORIZON KEL SSur5310P, 14 May 2004 (last updated 09 October 2006) [POL-0035378]
[0d473f335a3b09f8c60f0c9ba08a7ad7]
337 PC0063227.htmI, Peak PC0063227, 28 February 2018, [POL-0237798]
[8c84cd6299903d0d3bbe0e05de44b924]
+38 POLSAP outage 25-26January 2016.docx, POLSAP outage - 25th& 26th January 2016, 29 February 2016 [POL-
0216412] [daf7cd352274946c5dfe164e326bc4c0]
Prepared by: Jason Coyne
Occupation: Partne r;
Specialist Field: IT Systems 6 j1 UU
On the Instructions of: Freeths LLP ~ i
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page Ixxxi of 225
Uncategorised Horizon Concerns
5.146 A Post Office incident summary document in 2015%° highlighted a
number of high severity incidents between May and June 2015 one of
which included a financial deficit found in Global Payment transmission
and resulted in a £20 million shortfall within the system (cash flow
deficit for Retail and Bureau transactions). The precise underlying
cause of this incident is not known and/or whether or how it was
rectified.
5.147 An independent technical review!*° carried out in 2010 midway through
Horizon Online rollout highlighted some issues and concerns in respect
of recoverability following interruptions in service. The report stated:
“There is clear evidence from the solution that both Horizon (as a
result of the PCI changes) and HNH-X are not able to recover correctly
from failures.
The most worrying are reconciliation BIMS exceptions - these are
failures that require manual intervention by both Fujitsu and Royal
Mail. Many of these failures result in end customers of the Post Office
not being paid money (the exception shows that a bank believes it has
paid out the money, whereas the Horizon/HNG-X system knows it did
not pay out in reality”.
5.148 These issues could directly impact branch accounts and covered the
period both pre and post Horizon Online rollout. The report went on to
state that due to the high volumes of recoverable data the exception
workload was causing significant (and unsustainable) workload on
both Fujitsu and Royal Mail.
5.149 The risk of data loss arising because of incorrectly shutting down
equipment and/or replacing equipment is evident in several scenarios.
9 Incident Summary - June 2015 v3.pptx, Incident Summary - Trend Analysis, June 2015, [POL-0221065}
[ef2216b67d7ea7576fc14f5f7369dfb3)
14° RedAlert_April2010.doc, Post Office Red Alert - Independent Technical Review, James Stinchcombe,
Principal Architect, 27 April 2010, [POL-0220516] [465299cfiai71e7b871e4b5f1 1b2f9ba]
Prepared by: Jason Coyne
Occupation: Partne
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page Ixxxii of 225
A Post Office training slide‘*! specifically outlines this issue in Slide 5
regarding the Horizon Online kit used in outreach branches. The same
is true with branch terminals being swapped out'*?**?, In this case
incomplete transactions held on the hard drive of a branch terminal
that was swapped out were lost when the engineer failed to recover
the data from the hard drive; this resulted in data being lost.
5.150 At the Post Office Board meeting of 25th September 2013* it was
reported that:
“Operations Issues - Fallout from Horizon issues seriously damages
public and government confidence in the Post Office... Further
operational issues uncovered (but considered lower risk and lower
impact)”
5.151 The disclosed document is heavily redacted and therefore it is not clear
what “operations issues” were being discussed.
5.152 Appendix H is a constructed summarisation of the processing
components within Horizon that have been identified as common
points in which discrepancies have been recorded. A selection of the
KELs referred to in this report have been used to illustrate points in
Horizon where anomalies have either been detected or have failed to
complete the full processing.
Extent Conclusion
5.153 It has not been possible to measure the full extent of errors in data
recorded within Horizon arising from a) b) and c) above since:
i. it is possible that there are further reports and statistics that
might indicate further extent of errors which have yet to be
disclosed;
+41 Core & Outreach Brief HOL v1.pdf, Core & Outreach Brief, June 2010, [POL-0176232]
[594a4d1508f124dc307797c92b1d3aed]
+2 pCQ093382.html, Peak Incident Management System, 5 August 2003, [POL-0265630]
[c3393ab177a14280dfe535b7b34282a6]
+ Post Office Limited Board Single pdf Final.pdf, POST OFFICE LTD BOARD MEETING, 25
September 2013, [POL-0215589] [13ebf7 1fb7ca3ac8a8e08a7ba48dfatd]
Prepared by: Jason Coyne
Occupation: Partne
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page Ixxxiii of 225
ii. The information contained within documents such as KELs etc
is not complete or comprehensive in respect of the extent of
the issue recorded in the first instance (or the full impact
across the estate and on the Subpostmaster);
iii. It is not clear from those PEAKs (bugs/errors/defects) that
have been added to a software release whether the symptoms
identified in the first instance may have manifested and been
recorded under a new/different KEL/PEAK.
5.154 However, from the documented examples of known errors evidenced in
this report alone (aside from those unknown), it is clear that significant
errors in data recorded within Horizon have occurred.
Measures and Controls to prevent/detect/identify/report/reduce
errors in Horizon
5.155 There are various methods and controls implemented within Horizon by
means of electronic system checks and manual business processes
which sought to prevent/detect/identify/report and reduce errors in
Horizon. Namely (not exhaustive):
a. Reconciliation reports and processes (Section 6, Page 96)
b. Alert messages at the counter (Section 7.1, Page 118)
c. System alerts for Fujitsu (Section 8.1, Page 127)
d. Horizon Support Service Facilities, chiefly TfS and PEAK systems
(Section 4.66, Page 33)
e. Known Error Logs (Section 4.87, Page 38)
f. Software releases for bug fixes (Section 4.92, Page 39)
g. Audit Systems (Section 4.51, Page 30)
h. Journal Sequence Numbering (Section 4.52, Page 30)
i. Audits and Conformance to Quality Standards Checks
Prepared by: Jason Coyne
Occupation: Partne ° ey
Specialist Field: IT Systems g tQrou
On the Instructions of: Freeths LLP ~ i
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page Ixxxiv of 225
5.156 The Post Office Account Customer Service Problem Management
Procedure document'** identifies the process metrics and key
performance indicators required for measuring the effectiveness of the
process and service specifically in relation to Problem Management.
The Problem Management Procedure is set out in more detail at
Appendix E (Section 15.33, Page 194). Relevant to this section and
Issue 6 are the metrics and KPIs to measure/control and reduce the
risk of failure to detect, correct and remedy Horizon errors or bugs:
a. Number and impact of incidents occurring before the root cause
of the problem is identified and resolved
b. Number of repeat incidents following any corrective action
c. Number of problem records arising from proactive actions and
trend analysis
d. Number of changes arising from proactive actions
e. Percentage of problem records without an action plan
f. Average length of time to resolve problems
g. Number of incidents closed without a KEL
h. The number of Problem Records arising from Managed Change
(MSC, CP) activities
i. The number of Problem Records arising from the implementation
of new services / major releases.
5.157 From the above, it is my opinion that Post Office should be aware of all
recorded bugs/errors/defects in addition to those previously
acknowledged by them, from the process metrics compiled above.
5.158 Of specific interest is the POL monitor that tracks the number of records
arising directly as a result of managed change activities'*®. No
144 SVMSDMPRO0025_5.doc, Post Office Account Customer Service Problem Management Procedure, 12 July
2016 [POL-0146787] [fe6a96a3615a63e094682c7efaa33090]
+45 SVMSDMPRO0025_5.2.doc, Post Office Account Customer Service Problem Management Procedure - Version
Prepared by: Jason Coyne
Occupation: Partne r;
Specialist Field: IT Systems 6 j1 UU
On the Instructions of: Freeths LLP ~ i
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page Ixxxv of 225
disclosed logs have been found in respect of these problem records
that are listed as being reported monthly.
5.159 Requests have been made in relation to making such Management
Information reports. At the time of writing, these have not been made
available for analysis.
5.160 The Witness Statement of William Membery****!1*7 (POLs Head of Quality
and Compliance) states:
“The Horizon system (both Horizon Online and Legacy Horizon) are and
have been subject to audits to internationally recognised standards.
These audits check both technical aspects of the system and the
working practices of Fujitsu around Horizon. They provide assurance
that Horizon and Fujitsu are working within a robust system of control
measures. The audits review both the design of the controls and their
implementation in practice. Completion of these audits provides
assurance to Fujitsu, Post Office and other third parties (e.g clients of
Post Office) that the financial information within Horizon can be relied
on.”
5.161 Whilst both Horizon and Horizon Online contain a number of measures
and controls designed to check system integrity, these mechanisms
have been shown to have failed. This is a point agreed upon in the
Joint
Statement. It has been identified that known issues/bugs were often
deferred and dealt with on a cost/benefit basis. The Risk and
Compliance Committee meeting minutes of 18 September 20131*°
highlight an instance particularly in relation to an audit performed by
Ernst & Young:
“It was reported that following the recent Ernst & Young external audit
four risks [sic] been identified. Three of the risks raised had been
446 2, 15 September 2017, (1.4 - Process Metrics and Key Performance Metrics), [POL-0512874]
[dbe89de2e88cfb6494d3820228e17F0e]
447 Witness Statement of William Membery - 28.09.18.pdf
+48 R&CC Minutes 18th September 2013.docx, Risk and Compliance Committee (R&CC) Reference:
R&CC/MIN/SEP13, 18 September 2013 [POL-0217378] [4d23226da8aca4bc0aa3940b9f450325]
Prepared by: Jason Coyne
Occupation: Partne
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page Ixxxvi of 225
addressed however the final risk, relating to the communication by
Fujitsu of changes made to the Horizon system, was still outstanding.
It was identified that it would cost over a £1m to implement the
mitigation being suggested by the audit and that this was not
proportionate to the risk being managed”.
5.162 The Ernst & Young management letter arising from the 2011 audit‘*?
150recommended changes to strengthen the change management
process. They noted that POL was not usually involved in testing fixes
or maintenance changes to the in-scope applications (which includes
Horizon Online) and they were unable to identify an internal control
with the third-party service provider to authorise fixes and
maintenance changes prior to development for applications. The risks
were outlined as follows:
“There is an increased risk that unauthorised and inappropriate
changes are deployed if they are not adequately authorised, tested and
approved prior to migration to the production environment”
5.163 Further, as referenced at 5.188, there is evidence that despite
procedures being in place, these were not being followed by Fujitsu.
5.164 In the circumstances, whilst bugs/errors/defects may have been
identified, they were not necessarily fixed instantaneously, and control
was not retained over the process of fixing. The risk of error was
therefore not reduced as far as possible, but rather it was subject to a
commercial assessment.
5.165 It is acknowledged that simple fixes ought and were implemented to
either fix bugs or provide additional data validation checks
pothapragadac4359R'*! & Marris4123N.1°*
+4 POL Management Letter FINAL.docx, Post Office Limited - Management Letter for the year ended 27 March
50 (Section 4 - Current Year Recommendations ~ IT Specific) [POL-0219218]
[9d7862698d2a0f6af2a3c55590763bb7]
151 Pothapragadac4359R.html, HNG-X KEL pathapragadac4359R, 19 April 2011 (last updated 16 August 2011),
[POL-0038695] [f0a4dd0a787be69689f36b84a2b5753c]
*S? MArris4123N.html, HORIZON KEL MArris4123N, 28 November 2003, [POL-0033810]
[9b0bb96d10e84cc47F2cF4bd0d271f37}
Prepared by: Jason Coyne
Occupation: Partne
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page Ixxxvii of 225
5.166 Review of ‘S80 Release Note - Deferred PEAKS List - Counter’? which is
an addendum to the S80 Release Note (dated 2005) details the PEAKS
which remained outstanding once S80 was to be implemented (this is
pre-Horizon Online rollout and suggest that only PEAKs that impact
the Horizon counters are included). Whilst many are cosmetic low
severity items, PC0121925 reports an incident caused by a problem in
Riposte which is an “existing and very intermittent” problem in ‘Live’
(it is assumed this relates to the Production Horizon Live data
environment). It further states that the underlying cause would
require Escher to deliver software code not available for S80 or S80R
(suspected further release). The document further states that
“Development will continue to investigate to identify whether a ‘work
round’ [sic] change” could be made in the code.
Failures of measures and controls to prevent/detect/identify/report/reduce the
risk of errors
5.167 Whilst controls were in place, as per the Joint Statement, it is agreed
that evidence exists to show that these automated mechanisms have
failed to detect and record anomalies. Similarly, applied manual
processes are reported to have been subject to variations of quality
(operation outside of process).
5.168 ' An internal presentation from the Post Office'’’? shows the helpdesk
error processing workflows (at slide 9). Whilst one of these work flows
shows the process when an error is detected internally by the Post
Office, the four other workstreams deal with errors from different
external sources, namely:
a. “Branch discovers error/unbalance and calls NBSC”
b. “Client discovers error”
153 CSRENO32_1.doc, S80 Release Note - Deferred PEAKS List - Counter, 13 October 2005 [POL-0083919]
[1aa524ee9a53955ee392e5621f0ee4dt]
154 Finance L3 pack.pdf, Business transformation Finance Wave 1, 1 October 2014 [POL-0218441]
[a26a0ec0f6c8dbc20b753b200de725bb]
Prepared by: Jason Coyne
Occupation: Partne r;
Specialist Field: IT Systems ¢ 41 UU
On the Instructions of: Freeths LLP ~ i
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page Ixxxviii of 225
c. “Customer discovers error and call customer care”
d. “Branch discovers error and calls FSC directly”
5.169 The concern with the above is that if the Branch, Client or Customer do
not detect and therefore do not report the error, then this could well
have an impact upon Subpostmaster branch accounts without their
knowledge.
5.170 “Review of Key System Controls in POLSAP”!°°, an internal Post Office
document, highlighted procedural issues occurring in respect of
transaction data, access to software and change management. The
risks and implications identified included errors not being identified or
resolved leading to potential discrepancies in client balances.
5.171 A Detica NetReveal report issued in 2013*°° indicates, in its key findings
at section 1.2, four areas contributing to risks in relation to policy,
processes and information used by the Post Office to track and
manage the compliance of branches:
a.
Widespread non-conformance to Post Office policy and processes
by branches, with an_ institutional acceptance that errors,
workarounds and non-conformance exists;
Complexity and fragmentation of information systems which
hamper efforts both to gain an insight into branch behaviour and
root causes;
Ineffective process, policy and working practice in the central
operational teams to gather information, prioritise and act in a
coordinated manner;
Technology available to central operational teams was not fit for
purpose; analysis of large data sets in performed on an ad-hoc
155 AR12.037.ppt, Review of Key System Controls in POLSAP, November 2012 [POL-0217341]
[dc5b5ce76e817ce54cadb2d51 1c4ce11]
155 NRRA1207 10D007-0 50 Draft.doc.docx, Fraud and Non-conformance in the Post Office; Challenges and
Recommendations, 1 October 2013 [POL-0216106] [f41e9587582dc4691967c8e22d4aa64e]
Prepared by: Jason Coyne
Occupation: Partne
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FU,
180503R1935 16 October 2018 Page Ixxxix of 225
FUJ00082162
1J00082162
basis of data subsets copied into Excel and tasking of teams is
initiated and managed through email.
5.172 Given that process, and policy adherence is crucial for effective
management and resolution of shortfalls and discrepancies within
Horizon, the report indicates that levels or risk were not managed as
appropriately as was possible.
5.173 Referring to the instance identified at paragraph 5.49 above; it is
observed that additional human operated manual processes could
contribute to electronic errors affecting branch transactions.
5.174 The End to End Reconciliation Reporting document from 27 February
20121°7 states:
“There is no formal reconciliation produced between the POLSAP
System and the Credence transaction stream. The Credence stream
should therefore not be used to verify financial integrity and Post Office
should ensure the POLSAP System Transaction information is used for
this purpose."
5.175 The report regarding the reversal dispute conducted by Helen Rose
states:
“On looking at the credence data, it clearly indicates that the reversal
was completed by JAROO1 (Subpostmaster) at 10:37 04/10/2012 and
was reversal indicator 1 (existing reversal) and settled to cash.”
5.176 It is therefore relevant to question why Post Office were using Credence
data to initially investigate disputed transactions. Whilst it is evident
that it was understood by Post Office in this instance to request
assistance from Fujitsu for further material to investigate this dispute,
there appears to be further issues with the data provided by Fujitsu.
5.177 Observations of the disclosures illustrates that the initial report states "a
transaction at 10:42", whereas the credence data file shows 10:32
+57 SYMSDMSD0020_2.2.doc, End to End Reconciliation Reporting, 27 February 2012 [POL-0124572]
[aaedd5051156619c50296d04f6b2e779]
Prepared by: Jason Coyne
Occupation: Partne r;
Specialist Field: IT Systems 6 j1 UU
On the Instructions of: Freeths LLP ~ i
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page xc of 225
with the reversal at 10:37. Fujitsu's data states the transactions are at
9:32 and 9:33 and reversal timestamp is 9:37.
5.178 Whilst this hour difference between the data sets might be easily
traceable for Fujitsu, it is not clear how easily it would have been to
investigate issues where the Subpostmaster was not sure of what time
things went on erroneously in the system, or that it was a reversal
specifically.
5.179 Further to this point, an Ernst & Young review of Post Office’s systems
of internal control conducted in March 20111%* identified (amongst
other issues) various issues with the Credence application.
5.180 Particularly, weak change controls within the back end of the systems
allowing Logica developers (the third-party provider) to move their
own uncontrolled changes into the production environment, which I
understand to be both Credence software code and the data within
Credence used for audit evidence. Further documentation to approve
fixes and patches applied to Credence outside of the release process
were lacking, therefore linking changes to issue tickets to record the
original request for the bug fix was not possible.
5.181 Front end change process weaknesses were also observed. The
following noted:
“During our walkthrough of user administration of the front end of
Credence we noted several users with administrator rights, including
some generic users (this is noted below as a separate point). These
users have the access rights to create and amend reports, including
those which may be relied upon for audit evidence. These users can
change report design, and processing without documented request,
test or approval. When users have the rights to change reports that are
used by the business for reconciliation, exception reporting or other
processing, there is the risk that the reports are manipulated either
intentionally or accidentally.”
158 POL Management Letter FINAL.docx, Management letter for the year ended 27 March 2011, August 2011
[POL-0219218] [947862698d2a0f6af2a3c55590763bb7]
Prepared by: Jason Coyne
Occupation: Partne r;
Specialist Field: IT Systems 6 j1 UU
On the Instructions of: Freeths LLP ~ i
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page xci of 225
5.182 This is consistent with my opinion 5.99 in regard to manual fixes where
all manual entry activities enhance the likelihood and potential for
error.
5.183 Poorly handled changes introduced by Post Office to processes in
Horizon have also caused errors in accounts. A presentation'®? made
to the Post Office Operations Board in 21 July 2017 explained (at page
86) that changes were instructed (in 2015) as to the way that
“MoneyGram” is dealt with by the Subpostmasters. These changes
included acceptance of Debit/credit card payments. The presentation
reported the impact of this new working practice, including;
a. “Branches retrying transaction that had failed due to timing out
b. Branches reversing a transaction but not cancelling the AP part of
the transaction
c. General lack of understanding in the Branch of how the process
works, especially for refunds...”
5.184 The same presentation also referenced that “Camelot - Operation
simplification has gone live... early indication that this has accounted
for
50% of the issues”. No detail has been provided as to what the Camelot
issues were, but up until this point in the chronology Camelot was
shown as the most frequent cause of Transaction Corrections being
raised to modify branch accounts.
5.185 The witness statement of Akash Patny’® refers to his experience of
imbalances whilst dealing with Moneygram transactions. This witness
statement supports highlighted reconciliation issues due to timeouts
occurring between Post Office and Moneygram'*. As acknowledged in
15° Operations Board 21 July 2017.pdf, [POL-0221328] [9c45e0be3ff2b6773447cc6e41db5f46]
169 Witness Statement Of Akash Patny, 28 September 2018, (MoneyGram Issue - Page 3)
181 Items at Half-Year.docx, Watchlist Items (continuing from Qt1/2), 2016, [POL-0220630]
[7ac18146eb4f0ced505db4b34bac6724]
Prepared by: Jason Coyne
Occupation: Partne r;
Specialist Field: IT Systems 6 j1 UU
On the Instructions of: Freeths LLP ~ i
FU,
180503R1935 16 October 2018 Page xcii of 225
FUJ00082162
1J00082162
the document this could affect branch accounts due to reconciliation
differences.
Supporting Documentation
5.186 acha2230K?® highlights a problem with additional checks which were
implemented to identify system errors/inconsistencies when balancing
during branch In this instance the additional checks caused account
balancing errors on rollover. The note issued by SSC states, “This
should never happen - something has gone horribly wrong. Or
possibly the checks havent been implemented as intended.” This
suggests a recurring issue since the checks themselves appear to
have failed. It is not known how or/if this was resolved going forward.
5.187 dsed2049S'® highlights the lack of system communication and/or
support communication in respect of certain system features which
could subsequently result in errors. In this case withdrawn products
were converted to cash on rollover but the loss was carried forward
into the next period instead of being dealt with there and then. The
bug continued until a fix was applied 6 months after the KEL was
raised (ensuring an alert was raised warning the Subpostmaster of the
occurrence). However, it is further noted in the log that POL failed to
tell branches to “rem out” some products before they were withdrawn
leading to a cash loss that would not become apparent until the stock
unit was balanced again.
5.188 Fujitsu themselves were also open to mistakes. It is recorded that a
Fujitsu engineer failed to follow the correct process when replacing a
branch terminal resulting in multiple customer bills not being paid’.
The PEAK log states:
162 acha2230K.html, HNG-X acha2230K, 18 October 2013 (last updated 25 October 2013), [POL-0039583]
[49a4bca2b1c898a29ea22233960574d9]
163 dsed2049S.html, HNG-X KEL dsed2049S, 14 June 2011 (last updated 03 October 2013), [POL-0039549]
[34dfdbd02c7e909cbbefaebd5a4743df]
164 BC0093382.htmI, Peak Incident Management System, 5 August 2003, [POL-0265630]
[c3393ab177a14280dfe535b7b34282a6]
Prepared by: Jason Coyne
Occupation: Partne r;
Specialist Field: IT Systems 6 j1 UU
On the Instructions of: Freeths LLP ~ i
FU,
180503R1935 16 October 2018 Page xciii of 225
FUJ00082162
1J00082162
“I believe that we shall have to confess to POCL that we have not
followed the correct procedure and we should advise that POCL make
manual payments.....”
5.189 The reconciliation process used by POL to assist with identifying any
accounting differences is not able to easily identify genuine differences
and/or differences resulting from external APS transactions from old
trading dates. This is evidenced in acha3250R‘°!°° which, when
submitted to the development team, was considered too complex to
fix and was subsequently closed. This would add greater complexity to
managing the reconciliation process and having an accurate picture at
branch level of APS transactions.
5.190 There are various documented examples of circumstances in which
Horizon erroneously created duplicate Journal Sequence Numbers (
“JSN”) against the same transaction. 19716169170 JSNs should be (by
design) always unique as explained by Gareth Jenkins’:17!
“At this point a check is made that indeed there are no missing or
duplicate jsns for any counter and should any be found an alert is
raised.
Note that this could only happen as a result of a bug in the code or by
somebody tampering with the data in the BRDB and this check is
included specifically to check for any such bugs / tampering”
5.191 It is not currently clear how it would be determined whether duplicate
JSNs arose because of a bug or tampering. However, the existence of
duplicates in the first instance is in itself a failure of measures and
controls.
65 acha3250R.html, HNG-X KEL acha3250R, 14 February 2013 (last updated 10 February 2015) [POL-
468 ] [3675c6282d759025c44475aeed0b6c70)
167 GCSimpson2242L.html, HNG-X KEL GCSimpson2242L, 08 March 2006 (last updated 05 December 2010)
[POL-0038135] [cd3dcd109ec897a07328c70ca9d125d6}
166 MithyanthaJ1937S.html, HNG-X KEL MithyanthaJ1937S, 06 May 2010 (last updated 09 August 2016)
[POL0040508] [d56a274636043f38e96d2e9f3609d949]
169 DRowe5415Q.html, HORIZON KEL DRowe5415Q, 08 October 2002 (last updated 14 December 2012) [POL-
170 } [4e885614e1 1d9c4abf33cfbc4feef86a]
1 Horizon Online Data Integrity for POL, Gareth Jenkins.pdf, 2012 [POL-0021989]
[8dbd13d4aae9179dcaea4f62f3ab3572]
Prepared by: Jason Coyne
Occupation: Partne
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FU,
180503R1935 16 October 2018 Page xciv of 225
FUJ00082162
1J00082162
5.192 Aside from instances where bugs/defects and errors are recorded to
have affected Horizon branch accounts, a document entitled
‘Transaction Correction - Bramptom.doc’””’ indicates that due to a
fraudulently programmed credit card, Subpostmasters cash
declarations indicated shortfalls. This affected several branches and it
appears it was through the Subpostmaster’s diligence that the issue
was finally concluded.
5.193 Therefore, the assumption that POL's processes would detect anomalies
by way of shortfalls/discrepancy investigations on behalf of the
Subpostmaster appears to be incorrect. It is evident that in a number
of circumstances it is the Subpostmaster who highlights the anomaly
and pursues rectification often after disputing the initial response from
Post Office. This is further reinforced in consideration of the Common
Issues Claimant’s witness statements.
5.194 In a recent strategic and financial plan'”? it was acknowledged that
Horizon was designed two decades ago for an environment very much
different to the one we have today. In addition, both branch and back
office systems were also acknowledged as being end of life. A recent
Board Paper'’”*
also concluded that it was operating with a far higher
operational risk profile than is accepted by the Post Office when
making investment and project decisions. It is difficult to see how the
Post Office could improve things further and mitigate against risks of
issues impacting on branch accounts without a substantial investment
in modernising their IT systems.
5.195 The Post Office cash management proposals contained in a report dated
4 August 2017175 suggests that they were actively considering ways to
12 Transaction Correction - Brampton.doc, Email correspondence between Mark Baker and Karen Arnold,
March/April 2008 [POL-0018081] [7754e449931156b8339040a17cd3f3cb]
173 Post Office Strategic and Financial Plan 2021 - Board approved & final.pdf, Post Office Strategic and
Financial Plan 2018-2021, November 2017, [POL-0219032] [90e37c6a1ff574eb1d8f8007f26ed336]
178 Back Office Tower Transformation - Board Decision Paper 24 Nov 2016_v02.docx, 29 September 2016,
Page 4, [POL-0221162] [78261d52e6ce57¢368546592c99fadf5]
175 Cash Management Programme -Business Case June 2017 v1.5.docx, Business Case - Cash Management
Prepared by: Jason Coyne
Occupation: Partne
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page xcv of 225
improve processes impacting on many of the issues raised above. It is
my opinion that, whilst the Post Office was looking at ways to improve
cash management, it is also indicative that the system was generally
far from perfect and there existed a real risk of bugs/errors/defects
adversely impacting on branch accounts despite the processes in place
at the time to prevent this.
Opinion Summary
5.196 In light of the above observations and findings aside from the
acknowledged bugs/errors/defects that Post Office have recognised, I
opine that it was highly likely for bugs/errors/defects to have the
potential to both (a) cause apparent or alleged discrepancies or
shortfalls in relating to Subpostmasters’ branch accounts/transactions
and (b) undermine the reliability of Horizon to accurately process and
record transactions.
5.197 As highlighted by select Known Error Logs (KELs), issues thought to
have been fixed recurred in different circumstances, therefore the
reliability of Horizon to accurately process and record transactions is
questionable.
5.198 It is clear that in some instances it is not always apparent whether
recurring discrepancies were as a result of system bugs or the
Subpostmaster’s own actions, or other things beyond the control of
the Subpostmaster.'’”"7 However the fact that the SSC support team
were unable to assist or identify the root cause does undermine the
credibility of Horizon itself.
5.199 Whilst both Horizon and Horizon Online contain many measures and
controls for ensuring system integrity, these mechanisms do/have
failed. It has been identified that known issues/bugs were often
deferred and dealt with on a cost/benefit basis. Therefore, whilst
Programme, 04 August 2017, [POL-0221342] [2687475ac747f28a22c51¢53bbc50f30]
Prepared by: Jason Coyne
Occupation: Partne ° ey
Specialist Field: IT Systems g tQrou
On the Instructions of: Freeths LLP ~ i
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page xcvi of 225
bugs/errors/defects may have been identified, they were not
necessarily fixed instantaneously, therefore the risk of error was not
reduced as far as possible, but commercially assessed.
5.200 Similarly, processes designed to ensure controls and measures were
effective are identified as requiring improvement.
17 Call Details E-0602230104.htm, Post Office Account $70 Archive4.1 Call E-0602230104, 23 February 2006
[POL-0011223] [ccOec7afd6adfe07758adcfa4f4573ca]
178 105000759.pdf, Post Office Account NWBO1 Archive4.1 Call E-0402251077, 25 February 2004 [POL0019675]
[f513c96c9a72b75c3bb263c07a851a69]
6. Reconciliation and Transaction Corrections
Issue 5 - How, if at all, does the Horizon system itself compare transaction
data recorded by Horizon against transaction data from sources outside of
Horizon? Issue 15 - How did Horizon process and/or record Transaction
Corrections?
Prepared by: Jason Coyne
Occupation: Partne
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page xcvii of 225
6.1 The process by which Post Office compares transaction data recorded by
Horizon against transaction data from sources outside of Horizon is
known as Reconciliation. End to end reconciliation is the mechanism by
which Post Office establish which transactions are complete and
correct and which are not.‘
6.2 Post Office states’”’ that:
”,.each and every reconciliation error is the result of some systems
fault. That fault might, for example, be a software fault (introduced
through either design or coding), a system crash, or a telephone line
being dug up. Such faults may affect transactions, thus it is the job of
the reconciliation service to detect when and how any transaction is
affected by any system fault.”.
6.3. In the same document Post Office state:
“not all system faults will lead to corrective action and this is
generally done on a contractual and/or cost benefit basis”
6.4 Reconciliation reports and the known system components to facilitate it
are set out further in Appendix E of this report. For a diagrammatic
representation of the business applications applied to reconciliation as
at 2010 see Appendix B (Figure 5). In summary, reconciliation is a
complex process involving many system components of which there
are both electronic and human interactive process elements.
6.5 It should be noted that as Horizon progressed to Horizon Online the
reconciliation systems and processes continued to be developed.
Horizon Architecture
6.6 Dated 2010, Horizon architecture diagrams show that there are four main
areas within the Horizon Architecture: ‘7
178 SVMSDMPRO0012.doc, Reconciliation and Incident Management Joint Working Document, 18 March 2013
[POL-0032909] [b13d82f1ad57d0105cefbc3bfe7406c3]
177 SYMSDMPROOO12 - Reconciliation and Incident Management Joint Working Document.doc, Reconciliation and
incident Management Joint Working Document, 18 March 2013 [POL-0219191]
[7ccc36ff81ef72450f60a1275c1153a5]
178 POL-0003093.pdf, Horizon Architecture Diagrams (Page 3), 8 March 2010 [POL-0003093]
Prepared by: Jason Coyne
Occupation: Partne
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page xcviii of 225
a. POLFS - financial accounting system based on SAP (later
becoming
POLSAP)
b. Reference Data Proving - environment in which changes to
Reference Data are proved before releasing into live (Reference
Data controls things such as which products are sold, their price
and where in the menu hierarchy they are displayed).
c. Branches
d. Core Horizon - the central systems that support Horizon
6.7 Core Horizon Components for Transaction Processing:
a. Batching Services enable Post Office to send branch data (either
all transactions or in a summarised form) to external systems. It
also receives batch data from external systems for distribution to
branches. The systems to facilitate this within Horizon are:
i. TPS (Transaction Processing System) - provides daily data to
other systems including POLFS, POLMIS and HRSAP. Also
provides a feed to First Rate for Bureau transactions
ii. APS - (Automated Payment System) - provides daily data to
AP clients (British Gas, BT etc).
iii. LFS (Logistic Feeder Service) - provides data on pouch
collections and receipts at branches to SAPADS on an hourly
basis. Also, nightly data on cash held in branches.
b. The systems that receive data from external systems within
Horizon are:
i. APS - receives customer and tariff data for Quantum and
Water
Card service once per day
[00195d1cbb0017bd34e7c68b7930c1cf]
Prepared by: Jason Coyne
Occupation: Partne ° ey
Specialist Field: IT Systems d JFOU
On the Instructions of: Freeths LLP ° i
180503R1935
FU,
16 October 2018 Page xcix of 225
FUJ00082162
1J00082162
iii.
LFS (Logistics Feeder Service) - receives planned order data
(once per day) and pouch contents information (potentially
hourly).
RDMC (Reference Data Management Centre) - receives Rates
and Margins data for Bureau service.
Reconciliation and Enquiry services for online authorisation
Horizon specific systems are:
DRS (Data Reconciliation Service) - reconciles individual
transactions for the DCS, ETU and Banking Services.
TES (Transaction Enquiry Service) allows Post Office to query
transactions status for banking (only)
DWH (Data Warehouse) contains banking, ETU and DCS data
APS (Automated Payment System) which — reconciles
transactions between itself and TPS (Transaction Processing
System).
External Systems/ Clients
6.8 As per POL-0003093'”? (2010), Core Horizon communicates with the
following systems:
a.
Banks (LINK (Vocalink) A&L (Santander), CAPO) for online
authorisation of banking transactions (DCS for debit and credit
card authorisations) and transaction data used for reconciliation
Online Clients (e-pay, Streamline, DVLA) for online authorisation
of transactions and (for e-pay and Streamline) data used for
reconciliation
SAPADS - A Post Office system that handles cash and Foreign
Currency logistics. Data includes cash on hand statements from
17° POL-0003093.pdf, Horizon Architecture Diagrams (Page 3), 8 March 2010 [POL-0003093]
[00195d1cbb0017bd34e7c68b7930c1cf]
Prepared by: Jason Coyne
Occupation: Partne
Specialist Fiald: IT Systems Q I rou
On the Instructions of: Freeths LLP
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page c of 225
each branch, planned orders, replenishment deliveries and
delivery/collection data
d. HRSAP - A SAP system that handles remuneration to the branch
franchises and ‘multiples’ such as Tesco.
e. POL MIS - An Oracle based system to provide MI reporting to
Post Office.
f. First Rate - Provides bureau rate information. It is also passed all
bureau transactions to allow First Rate to undertake MI.
g. Siemens Metering - Provides Rates and Customer data for
Quantum gas pre-payment card.
h. AP Clients - Transaction information for Clients where payment
information is collected by Post Office.
i. Royal Mail and Parcel Force Worldwide - track and trace
information for parcels and letters taken in branch.
j. RDS - Post Office system that provides Reference Data
6.9 POL-0219319%® illustrates a more detailed breakdown of external service
components. See Appendix B Figure 8.
6.10 In 2012 Post Office described'*' the “Core transactional Finance Systems”
as:
18 HorizonContext.jpg, Horizon Service Context Diagram, 9 August 2018 [POL-0219319]
[0005427fe3eec69b47bce277611df037]
181 Phase 2a) consolidated output.pdf, Finance Roadmap Project, 3 September 2012, [POL-0215782]
[ac4469b27e384fe4440f351c115d8108]
Prepared by: Jason Coyne
Occupation: Partne r;
Specialist Field: IT Systems ¢ 41 UU
On the Instructions of: Freeths LLP ~ i
FUJ00082162
FUJ00082162
“... Essentially stable from a technical perspective but have been built
up organically overtime and are potentially integrated in an inefficient
manner with multiple interfaces” and “.. in some instances the
systems still do not provide the functionality required by the business
and manual interventions and rudimentary technologies have had to
be implemented”
Transaction Acknowledgements
6.11 Some Post Office transactions (e.g. Lottery, Paystation, ATM) are not
transacted through a Horizon terminal but instead via separate
machine. However, cash taken, and stock vended for these
transactions needs to be accounted for on Horizon as part of the
overall branch cash and stock holdings. To ensure that Horizon is kept
synchronised with the records on the third-party equipment a
‘Transaction
Acknowledgement’ (TA) is often used. '*?
6.12 Overnight the third-party equipment reports the volume / number of
transactions to Post Office.'*? Post Office’s data centre then sends an
overnight electronic message to each branch's Horizon terminal which
contains details of the volume / number of transactions conducted
within the branch on the third-party equipment. This is the transaction
acknowledgement.
Reconciliation Processes
6.13 Reconciliation processes utilise a set of printable electronic reports. *®*
6.14 Each day, branch account transactions are harvested and processed
through the Horizon system in order to inform Post Office Finance of
the aggregated totals for products and services sold. This enables Post
+82 Factfile.DOCX, Initial Complaint Review and Mediation Scheme DRAFT Factfile, 16 April 2014 [POL-0022996]
[d24556f1009f1881f42a2dbb5b8154de]
189 Factfile.DOCX, Initial Complaint Review and Mediation Scheme DRAFT Factfile, 16 April 2014 [POL0023002][Hash
d24556f1009f1881f42a2dbb5b8154de]
184 SVMSDMPRO0012_3.doc, Reconciliation and Incident Management Joint Working Document, 30 April 2012
[POL-0125134] [ad897ac9ff5edb2de37bfb8c4e9dc362]
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page cii of 225
Office to provide settlement figures to their clients. Reconciliation is
therefore used by Post Office Ltd to provide financial and business
reconciliation at transaction level to demonstrate that each transaction
is complete and correct and report on any transaction that is not.*®
6.15 The Horizon system contains integrity checking functionality to monitor
transaction progression and integrity of transactions as they flow
through the system.
6.16 The ‘state’ of transactions is recorded as they travel through the system
with any exceptions (transaction anomalies) harvested along the way
in order that they can be dealt with using largely manual processes.
6.17 Various report sets facilitate the reconciliation process within Horizon for
both those transactions to/from approx. 130 external clients'®* and
within the core Horizon components themselves (these are further set
out in Section 8 and Appendix E.
6.18 A Horizon Service Reconciliation Exceptions document states:
“Due to the potential dynamic nature of the Reconciliation Service,
where there is the potential for new exception types to be generated
as a result of software errors within new releases or reference data, it
has been agreed that these procedures will be documented outside of
the formal Reconciliation & Incident Management CCD document set.”
6.19 Where there is a need for manual intervention due to bugs/data
corruptions/incidents/errors, Post Office and Fujitsu teams interrogate
transaction data and reports to establish and modify any erroneous
data.
6.20 Reconciliation is therefore also used by the Reconciliation Service to check
that the corrective action is effective.1®*
185 SVMSDMPRO0012_3.doc, Reconciliation and Incident Management Joint Working Document, 30 April 2012
[POL-0125134] [ad897ac9ffSedb2de37bfb8c4e9dc362]
185 Witness statement of Angela Van-Den_Bogerd
Prepared by: Jason Coyne
Occupation: Partne
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page ciii of 225
6.21 Since reconciliation facilitates the core operations of Post Office, there are
various services, departments and processes that comprise it. Many of
which have evolved through the lifespan of Horizon to Horizon Online.
Reconciliation Services/Departments
POL Finance
6.22 The POL Finance department is responsible for reconciling data within
Post Office central systems which relate to enquiries from Post Office
Clients'®”. They are also responsible for generating Business Incidents
in the event of error discovery.
6.23 In Horizon Online POL Finance use the CTS (Client Transaction Summary)
as the basis for settlement with Post Office Clients. If the CTS file is
not delivered by Fujitsu, POL Finance are to use the APSS2133b file to
manually calculate any settlement due.
Fujitsu Management Support Unit (MSU)
6.24 The MSU Unit are responsible for resolving exceptions arising within the
Horizon estate.
6.25 MSU are responsible for various reconciliation reports namely; “A&L
Reconciliation File Delivery Report”, “CAPO Reconciliation File Delivery
Report”, “LINK Reconciliation File Delivery Report” and the “DRS
Reconciliation File Delivery Report”.'®® The reports are produced For
the MSU, generated by system processing.
Fujitsu Third Line Support (SSC)
6.26 Within reconciliation, Service Support Centre (SSC) are responsible for
repairing any corrupted or exception transaction data within the
Horizon system.
187 CSPRO111_5.doc, TPS Reconciliation & Incident Management, 17 October 2005 [POL-0083720]
[2079d9210ce25f44a77a55ee3efb67eb]
185 NBLLDO79_0.1.doc, TES Reconciliation File Delivery Reporting Low Level Design, 20 August 2004 [POLO077442]
[8606de4f0ea60322502ed8a00de43b1e]
Prepared by: Jason Coyne
Occupation: Partne ° ey
Specialist Field: IT Systems d JFOU
On the Instructions of: Freeths LLP ~ i
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page civ of 225
6.27 The tools and methods by which this is carried out are covered under
Section 9 below.
Reconciliation Exceptions
6.28
Where transactional data does not conform to its expected format or in
comparison to copies of itself or corresponding records then it causes a
reconciliation error.
6.29 Post Office acknowledge that errors may occur within counter
6.30 In
transactions or during the harvesting process. In addition to errors
highlighted by Fujitsu Services within the TPS Report Set, errors may
also be discovered by Post Office Ltd Finance (POL Finance) when
reconciling data within its central systems or which relate to enquiries
from Post Office Clients.
September 2012 an internal Post Office document!® reported; “There
is a data reconciliation service delivered by Fujitsu to reconcile the
APS and TPS streams. A suite of reports is produced". Against this
section are the report are “Issues/Risks” which state; “Actions to
resolve some differences are not well understood and can be lengthy
to resolve eg BIMs”. Post Office record the business impact as;
“Manual posting are needed. Protracted branch and customer
enquires".
6.31 Reconciliation and Incident Management documentation identifies that an
incomplete transaction is not necessarily a Reconciliation error, but it
might become one if it is not completed in a timely manner. *°°
6.32 It also further states that (regarding reconciliation errors arising from
system faults):
“It is acknowledged that not all system faults will lead to corrective action
as this is generally done on a contractual and/or cost benefit basis.”
189 Phase 2a) consolidated output.pdf, Finance Roadmap Project, 3 September 2012, [POL-0215782]
[ac4469b27e384fe4440f35 1c1 158108]
18° SVMSDMPRO0012.doc, Reconciliation and Incident Management Joint Working Document, 18 March 2013 [POL-
0032909.doc]
[b13d82f1ad57d0105cefbc3bfe7406c3]
Prepared
Occupation: Partne
Specialist
On the In:
by: Jason Coyne
Field: IT Systems § tGroup
structions of: Freeths LLP 7
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page cv of 225
Data Reconciliation Service (DRS)
6.33 The Data Reconciliation Service is the system used to reconcile transactions
carried out with Financial Institutions.
6.34 For a transaction to be reconciled and considered in a ‘complete’ state it is
necessary that the DRS finds three components to it:
a. C112 (A transaction sent from TPS to Post Office Finance System
and received by the DRS)
b. C12 (Transformation of a Confirmed (C1) message as written to the
DRS)
c. C4 or D (C4 = Confirmed Client Transaction authorised, D =
Transaction reconciliation difference highlighted and notified to
DRS)
6.35 If one of the above components is incomplete, corrupt or duplicated it is
recorded as being in an exception or error state and should appear on
the Network Banking report NB102.'%
6.36 The following reports are critical to reconciliation and are recorded as
produced daily:
a. NBOOO - DRS Summary
b. NB101 - Network Banking Settlement Statement
c. NB102 - Exception Summary
6.37 An F99 transaction is a transaction state that indicates that a
reconciliation error has been reported but POL has advised that the
issue has subsequently been resolved. This state is set using the DRS
Workstation application that is used by Fujitsu Security Operations
team.1°
11 POL-0032841.doc, Network Banking Reconciliation and Incident Management Processes, 26 February 2003
[POL-0032841] [907539a6845da640795b670f2015199b]
182 POL-0032990.doc, End to End Reconciliation Reporting, 4 September 2017 [POL-0032990]
[7#79ebcdead957d0d4019672976d25F4]
Prepared by: Jason Coyne
Occupation: Partne r;
Specialist Field: IT Systems 6 j1 UU
On the Instructions of: Freeths LLP ~ i
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page cvi of 225
6.38 Post Office reported in response to my Request for Information that
10,000+ transactions per week suffer from problems and are not
automatically reconciled. Such transactions require manual
intervention for the reconciliation to take place. Where there is manual
correction applied within the system, there is the potential for input
error that may impact the financial status for the branch and/or end
client.
6.39 Appendix G
[See Spreadsheet]
Prepared by: Jason Coyne
Occupation: Partne ° ey
Specialist Field: IT Systems } 41 UU
On the Instructions of: Freeths LLP ~ i
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page cvii of 225
6.40 Appendix H provides a pictorial summary of the processing
components within Horizon which have been identified as the most
common points in which discrepancies can be captured.
Transaction Disputes
Enquiry Based Banking Transaction (EBBT)
6.41 Where Fujitsu has received notification from Post Office Ltd via the HSD
that it wished to query a particular transaction.
Disputed Banking Transaction Notice (DBTN)
6.42 Raised by POL Finance via HSD after notification from either a Post Office
branch or Network Business Support Centre (NBSC).
6.43 Where Fujitsu has received notification from Post Office Ltd via the
Enquiry Service following a query by the ‘End’ customer relating to the
state of his / her account.
Reconciliation Summary
6.44 Although there are various integrity check points and manual processes
observed within the reconciliation process, this has been an evolving
Progression since Horizon was first introduced. Reconciliation
processes have developed further with the system and technological
progression of Horizon Online.
6.45 Whilst the reconciliation process within Horizon handles the integrity
checking of transaction data and potential anomalies through the
various report sets, it does not necessarily ensure all anomalies or
discrepancies are resolved as this becomes a more manual process.
Further, manual processes applied to correct data anomalies also have
the potential to introduce further errors.
6.46 In consideration that Branch account positions were interpreted and
reviewed from data flows through to Post Office back end systems
(which would determine whether Transaction Corrections were to be
applied), the following is considered relevant:
Prepared by: Jason Coyne
Occupation: Partne ° ey
Specialist Field: IT Systems d JFOU
On the Instructions of: Freeths LLP ~ i
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page cviii of 225
“POLSAP - Following investigation by Fujitsu, Logica and Ingenico, the
root cause of a long outstanding problem with missing data within
POLSAP was identified as out of range dates which failed the Credence
validation (in excess of 90 days). Ingenico has corrected the data and
P&BA has advised that the mismatches have been cleared within the
accounts. A permanent preventative measure is now being developed
by Ingenico.’"°°
6.47 It is not clear whether or what branches this “missing data” might have
affected.
Transaction Corrections
6.48 As per Operations Manual version 71°” the introduction of the new Post
Office Ltd Finance System (POLFS) in Product and Branch Accounting
(P&BA) Chesterfield, resulted in finance teams no longer being able to
adjust client accounts on site.
6.49 Prior to 2005, branches were required to balance weekly and produced a
Horizon generated “Cash Account”. Discrepancies were, with
authorisation from the Post Office, placed in the branch “Suspense
section” of their cash account. The discrepancy was held until
enquiries into it were concluded and it was then removed by the
issuing of an error notice (now known as a Transaction Correction) by
Post Office or by the Subpostmaster putting the money into the
branch to cover the loss or removing the value of the gain from the
branch so as to balance the account.'®*
© 10 Monthly Service Management Performance Period 10.xlsx, Monthly Service Management Performance
Measures - January 2012, 10 February 2012 [POL-0219354] [8807960c4e4ce6c7671d1f8254ed18b7] *”
1.5 5. Operations Manual version 7 December 2006 - pages 9-13.pdf, Processing any outstanding
Transaction Corrections, 7 December 2006 [POL-0184501] [9f8351ecf4bSbd1d4fd39452bef8026F]
+98 Email from Angela Van-Den-Bogerd (PO) to Ron Warmington (Second Sight) and Ian Henderson (Second
Sight) Subject: FW: Factfile [BD-4A.F1D20472253] Date: 16 April 2014 [POL-0022995]
[ca88fee778bee2b78ab05bc62cf6Fe2c]
Prepared by: Jason Coyne
Occupation: Partne
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
180503R1935
FUJ00082162
FUJ00082162
16 October 2018 Page cix of 225
6.50 A Transaction Correction (TC) is defined as the accounting process
through SAP/POLFS for P&BA to adjust a POL Branch account for
discrepancies found.
6.51 According to POL-0032855'%?, In summary, the whole process was
designed to work as follows:
a.
The central accounting function decides that it is necessary to
make some adjustment to the Branch accounts. Such
adjustments are to be made at branch when branch transaction
data does not align with client or supplier data.
A Transaction Correction (TC) is defined which will carry out the
necessary changes (i.e. the central user will define an amount to
be transacted for a given Product in a given Branch and a
corresponding settlement Product).
The Transaction Correction will also define a list of possible
actions that the Branch Manager can take and also a message is
presented to the Branch Manager informing him/her of the effect
of carrying out any of these actions.
A daily file of such Transaction Corrections is generated from
POLFS and passed to TPS overnight.
TPS receives this file, validates the data and performs the
required translations using Reference Data (converting a SAP
article ID into a Horizon Product).
TPS sends messages for the Transaction Corrections to the
specified branches. A single message is written for the
appropriate Branch for each Transaction Correction.
*83 DELLDO014_2.doc, TPS Transaction Corrections, 04 April 2005 [POL-0032855]
[926e9F76cb06ba79277f62138309290e]
Prepared by: Jason Coyne
Occupation: Partne
Specialist Field: IT Systems ; I rou
On the Instructions of: Freeths LLP ”
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page cx of 225
g. Changes at the counter enable a person with the required role to
be made aware of the existence of an outstanding TC and apply
the correction at the counter. '%*
h. The result of processing a TC will normally be the creation of the
specified Transactions, which will be returned to POLFS as part of
the normal flow of Summarised Transaction data at the end of
the Trading Day on which the TC was processed at Branch.
i. Subpostmasters are advised to log on to Horizon every working
day to check for and process TCs as only those with Manager or
Supervisor access should process them.
j. TCs can be dealt with at log on or left until a more convenient
time later, but they must be processed before the last stock unit
in a branch is balanced, otherwise the Branch Trading Period
cannot occur.
6.52 It is recorded that it is not possible to reverse a correction transaction.
However, an erroneous correction transaction could be negated by
POLFS producing a fresh correction request with opposite sense.
Transaction Correction Options for the Subpostmaster?®>
6.53 TCs are issued electronically to the branch. At log on, the Subpostmaster
is presented with a notification of any new TC. If the TC is suspended
to be dealt with at a later point, Subpostmasters have menu button
options to access the outstanding corrections (there may be more
than one held on record to be accepted).
6.54 When a TC is presented on screen it is accompanied by the details of the
transaction to which it relates.
6.55 There are a range of options presented for processing the TC. The options
available are presented in the table below:'°°
184 EAHLD009_0.1.doc, TPS HR SAP Summarisation & Transaction Corrections HLD, 25 May 2009 [POL0076419]
[31d7f058bfa43c787c748e4acebfc8fc]
195 EPSPGO01_0.2.doc, S80 Impact Release 3 EPOSS Counter Operational Support Guide, 10 May 2005
[POL0081677] [4481091 5b02dc5e8c0d15604570c1f65]
86 1,5 5, Operations Manual version 7 December 2006 - pages 9-13.pdf, Processing any outstanding
Prepared by: Jason Coyne
Occupation: Partne r;
Specialist Field: IT Systems 6 j1 UU
On the Instructions of: Freeths LLP ~ i
180503R1935
FUJ00082162
FUJ00082162
16 October 2018 Page cxi of 225
Option
Detail
W/O to P&L (F4) (Directly
Managed branches only)
Directly Managed branches can use the
“Write Off to P&L” option if the amount of
the Transaction Correction is greater than
the value they are expected to settle
themselves, unless the correction states
that the ‘Make Good’ option should be
selected.
Ass Nominee (F4) (National
Multiples only)
National Multiple branches must use the
‘Assign to Nominee’ option whenever they
process a Transaction Correction, the only
exception being if they use the ‘Seek
Evidence’ option.
Seek Evidence (F3) (All
branches)
You can select the ‘Seek Evidence’ option if
the original information supplied by Product
and Branch Accounting (P&BA), Chesterfield
is insufficient for you to accept the validity
of the Transaction Correction. In this case
PB&A will provide additional evidence and
issue a new TC that will not include the
‘Seek Evidence’ option.
Stock WO (F4) (All branches)
You should select the WO option for the
following:
Transaction Corrections, 7 December 2006 [POL-0184501] [9f8351ecf4b5bd1d4fd3945 2bef8026f]
Prepared by: Jason Coyne
Occupation: Partne
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page cxii of 225
e Rem surpluses in stock from Hemel
Hampstead
e Rem shortages in stock from Hemel
Hampstead
@ non-accounting data corrections (so
that if incorrect volumes of
transactions have been claimed they
can be readjusted)
Cancel (F1) (All branches)
You should select the Cancel option if you
have selected a TC but do not wish to
process it immediately.
Accept Now (F2) (All branches}
except National Multiples)
When this option is selected it leads to a
picklist of further settlement options
available to your branch. These vary
according to the branch type and are
displayed in the table below:
Option Type of
branch(es)
applicable
Reason for Selection
Prepared by: Jason Coyne
Occupation: Partne
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page cxiii of 225
Make Good I All branchesI
- Cash except National I Use this option if you are using cash to
Multiples account for a Transaction Correction, or if
the instructions accompanying the TC advise
you to accept this option.
Please remember: You may need to
physically add or remove the cash from
your stock or redeem from Rem Suspense
to reflect this change, otherwise the
discrepancy will remain in your accounts.
Make Good I All branches Use this option if you are using a cheque to
- Cheque except Directly account for a Transaction Correction
Managed (even
h h th i
mov be we option Please remember: The cheque must bel
the system) and dispatched in your daily dispatch.
National
Multiples
Settle All branches] Selecting this option allows you to make
Centrally except Directly good a misbalance through the debt
recovery process that is managed within
(only Managed and P&BA. If selected, P&BA will contact you to
available if a} National confirm the next steps.
Transaction I Multiples
Correction is
for £150 or
over)
Prepared by: Jason Coyne
Occupation: Partne
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page cxiv of 225
Successful processing of a Transaction Correction
6.56 When a Transaction Correction is successfully processed on the Horizon
Online system, a message will be displayed to confirm that the
Transaction Correction has been successfully processed.
Unsuccessful processing of a Transaction Correction
6.57 It is recorded that the Horizon Online system process includes a check to
see whether Transaction Corrections fail due to discrepancies between
the validation of the transaction at the counter and the values held
within the TC message.
6.58 Where this might be the case, the system displays a failed warning
message and Subpostmasters are advised to contact NBSC. They will
advise Product and Branch Accounting (P&BA) Chesterfield who will
investigate the failure and issue a new TC.
6.59 In the event of any Connection failures when processing a TC, the process
must be abandoned until the system is restored.
Branch Trading Statement
6.60 The total volume of TCs processed during a Branch Trading Period can be
seen on the Branch Trading Statement.'%”
Disputing a Transaction Correction
6.61 Post Office documentation states that Subpostmasters have the ability to
dispute any TC given. They are advised to contact the person who
sent the TC as soon as possible.**® Subpostmasters may be asked to
give more information to support their dispute. If a TC is issued too
close to a Branch Trading period end to be fully investigated,
Subpostmasters are encouraged to call NBSC to ask for more time to
197 1,5 5, Operations Manual version 7 December 2006 - pages 9-13.pdf, Processing any outstanding Transaction
Corrections, 7 December 2006 [9f8351ecf4b5bd1d4fd39452bef8026F]
5 1,6 16. Disputing a Transaction Correction & the Appeals process.pdf, Branch Trading: balancing and
despatch of documents - Balancing a stock unit, 12 October 2016 [105f4315b879dd7810def967bde6bb15]
same text appears in POL-0001515]
Prepared by: Jason Coyne
Occupation: Partne
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page cxv of 225
gather and present supporting information. Where a dispute is
accepted, a compensating TC is issued.
6.62 Where a TC or branch discrepancy is disputed, and the case is not
allowed (this is not defined in in the document) Post Office state that
Subpostmasters may make a written submission explaining why the
loss (or gain) is not proper. This ensures that the debt recovery
process suspended pending a written response.
6.63 A Subpostmaster is limited from disputing a TC more than once!®®. In the
event of not being satisfied by the initial dispute decision,
Subpostmasters were unable to electronically select further dispute
actions. Any further action to dispute a TC would therefore be outside
of any selectable counter options.
Transaction Correction Observations and Findings
6.64 There is evidence of POL themselves creating incorrect Transaction
Corrections and sending these to Subpostmasters. One example
identifies that a Transaction Correction was issued for 800 sheets of
100 stamps, rather than 8 sheets of 100 stamps. Some of these POL
mistakes were not spotted and therefore accepted by Subpostmasters.
The result of this POL error is the insertion of a discrepancy in the
Subpostmaster’s branch accounts.200701207
6.65 As identified from review of the witness statement of Angela Burke”,
Transaction Corrections were also documented against the incorrect
financial institution. Mrs Burke was awaiting a Transaction Correction
to correct a TSB withdrawal in branch that was lost in the Horizon
system and when this was sent through it was documented as a
Lloyds withdrawal correction due to there apparently not being “a code
18° Transition Guide for Group A.PDF, Branch Trading Transition Guide, 26 September 2005 [POL-0171227]
[b5d2cc8a4ffdfbd976d1651c909d8ef1]
2° PCO131060.html, PEAK PCO131060, 17 January 2006 [POL-0301483]
[fad49b305707d3deac31b9249ee5c761]
201 pdf, Display Notes: 125460003153512008001, 25 August 2010 [POL-0006283]
[189c0e81e2064924aecfb5fcf281bf2e]
202 Witness statement of Angela Burke 28.09.2018.pdf
Prepared by: Jason Coyne
Occupation: Partne r;
Specialist Field: IT Systems 6 j1 UU
On the Instructions of: Freeths LLP ~ i
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page cxvi of 225
for TSB”. Whilst the receipt of the TCV would ultimately rebalance the
discrepancy for Mrs Burke it illustrates one example of error within the
issuing of TCs.
6.66 An analysis of Transaction Corrections”°? displays Transaction Corrections
between 26 March 2012 and 28 March 2013. The largest (in monetary
terms) appears to be £810,000. Of interest, there is also a £-
810,000 Transaction Correction for the same “Potters Bar” branch,
indicating that the correction might have initially been in error.
6.67 Of the 84,217 TC’s over this period 22,567 resulted from “Camelot” and
25,649 from “Cash Rems from Branch”. Only a relatively small amount
(945) of “ATM” TCs are referenced. Full table available at Appendix C
(2012-2013 TC’s)
6.68 TC queries were often raised with the PO helpdesk. ‘SLA Summary NEW
WE 15062014.xIs (POL-0031913)'°* shows that during one week 9421
calls were made to the helpdesk and 145 of these related to queries
and disputes about Transaction Corrections (See Appendix D).
6.69 The witness statement of Adrees Latif?°> highlights an alleged failed
Transaction Correction in respect of incorrectly issued Camelot scratch
cards in Horizon. The correction should have reversed the stock
amount but for reasons unknown to the Subpostmaster this failed. The
issue remained outstanding at the time the statement was given, and
the branch showed non-existent stock of Camelot scratch cards with a
value of £1000.00 that was unable to be adjusted without incurring a
shortfall.
6.70 Figures presented at the Post Office Operations board of 22nd March
20187 displayed that 3,546 branches had more than 1 TC per month.
203 POL-0031763.xIsx, Analysis of Transaction Corrections, ca. 2013 [POL-0031763]
[a9f7d9c47b7748522a182dbe2a114768)}
204 SLA Summary NEW WE 15062014.xIs, NBSC Incident SLA Summary - Week ending 15th June 2014, 17
June 2014 [POL-0031913] [7fa41e4d5bcc4996e0bfeObbbadf82f1]
20 Witness Statement of Adrees Latif, 28 September 2018, (Para: 9-14)
206 Operations Board - 22 March 2018.pdf, Operations Board — 22 March 2018, 23 March 2018 [POL-0220482]
[638109a286e2e884dc735539687F7¢35]
Prepared by: Jason Coyne
Occupation: Partne r;
Specialist Field: IT Systems ¢ 41 UU
On the Instructions of: Freeths LLP ~ i
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page cxvii of 225
6.71 The document ‘SLA Summary New WE 06072014”? records an activity
type of “Discrepancy” in relation to either Stock or Remittance
differences against the advice note received from POL. I understand
this to mean that POL have not dispatched the correct amount of
Stock or Cash stated as dispatched on the advice note. This may
indicate a potentially significant flaw in the Horizon system. If the
Subpostmaster does not spot POL’s error and therefore it goes
unreported, this will
likely mean that the next time the Subpostmaster conducts a stock take
it will show an accounting discrepancy which might lead to a TC.
6.72 The diagram in Annex B sets out a breakdown of potential reasons for
the issue of a Transaction Correction. Note that this list is
nonexhaustive and is based on the sheet headed “Possible Reasons for
TC’s Issued” within the document “NEW TC PACK P10 2011.xIs”*°%. The
categorisation as to whether these were caused by Subpostmaster or
not is mine.
Opinion Summary
6.73 Reconciliation is a complex combination of electronical and manual
processes. POL approximate that over 10,000 transactions per week
are subject to manual corrections.
6.74 Alongside the electronic processing elements of reconciliation are various
phases of analysis and decision making in respect of why/where
flagged anomalies occurred. This scrutiny is crucial to determining
both the reason for the anomaly in the first instance, and to remedy
the effects of it.
6.75 Transaction Corrections at branch counter typically stem from incidents
where either a transaction has been performed in error by the
2°? SLA Summary NEW WE 06072014.xIs, NBSC Incident SLA Summary - Week ending 6th July 2014, 14 July 2014
[POL-0031909] [2a6e1fbd3ef5d76899a9f21957d65011]
28 NEW TC PACK P10 2011.xls, TC Glossary, 15 February 2012 [POL-0221536]
[e1a3c3394d348ab3355302b35cfd63ab)
Prepared by: Jason Coyne
Occupation: Partne r;
Specialist Field: IT Systems 6 j1 UU
On the Instructions of: Freeths LLP ~ i
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page cxviii of 225
Subpostmaster and needs correcting or, a transaction anomaly has
occurred at another point and needs adjusting not caused by the
Subpostmaster.
6.76 It has not been possible to review the analysis, or reasoning applied to
Transaction Corrections issued by Post Office, i.e. whether these were
appropriately identified as issues in financial positions stemming from
the effects of a bug/error/defect or third-party error or from a counter
mistake.
6.77 Therefore, although reconciliation might identify transaction anomalies,
and appropriately capture electronic distortion of transaction data, it
does not ensure an infallible process in respect of appropriately
handling potential shortfalls/discrepancies for Subpostmasters, since
there are a range of other factors to consider in the processing and
analysis of the data. It appears that the Post Office applied a
cost/benefit approach to the remedying of reconciliation exceptions
and certain bugs/errors/defects.
6.78 It is not clear from analysis of the various Known Error Logs whether
bugs/errors/defects identified as affecting branch accounts then
ultimately progressed to the issuing of a Transaction Correction for
that branch or were confirmed by separate communications from the
Post Office.
Prepared by: Jason Coyne
Occupation: Partne ° ey
Specialist Field: IT Systems d JFOU
On the Instructions of: Freeths LLP ~ i
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page cxix of 225
7. Horizon Reporting - Facilities for Subpostmasters
Issue 2 - Did the Horizon IT system itself alert Subpostmasters of such bugs,
errors or defects as described in (1) above and if so how.
Issue 14 - How (if at all) does the Horizon system and its functionality:
a. enable Subpostmasters to compare the stock and cash in a branch
against the stock and cash indicated on Horizon?
b. enable or require Subpostmasters to decide how to deal with,
dispute, accept or make good an alleged discrepancy by (i)
providing his or her own personal funds or (ii) settling centrally?
c. record and reflect the consequence of raising a dispute on an
alleged discrepancy, on Horizon Branch account data and, in
particular:
d. does raising a dispute with the Helpline cause a block to be placed
on the value of an alleged shortfall; and
e. is that recorded on the Horizon system as a debt due to Post
Office?
f. enable Subpostmasters to produce (i) Cash Account before 2005
and (ii) Branch Trading Statement after 2005?
g. enable or require Subpostmasters to continue to trade if they did
not complete a Branch Trading Statement; and, if so, on what basis
and with what consequences on the Horizon system?
Horizon Alerts for Subpostmasters in respect of bugs/errors/defects
7.1 Post Office have advised?’ that in respect of the known Local Suspense
Account Issue that Subpostmasters were notified however, how they
were notified and if all affected Subpostmasters were notified is still
subject to clarification from Post Office.
209 Post Office's response to Jason Coyne’s Requests for Information.pdf 08 August 2018
Prepared by: Jason Coyne
Occupation: Partne
Specialist Field: IT Systems § tQrour
On the Instructions of: Freeths LLP
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page Cxx of 225
7.2?In a report from Gareth Jenkins 29 September 20107° (in connection with
the known bug ‘Receipts and Payments mismatch’) it is reported:
“This has the following consequences: There will be a receipts and
payment mismatch corresponding to the value of discrepancies that
were “lost”. Note that if the user doesn’t check their final balance
report carefully they may be unaware of the issue since there is no
explicit message when a receipts and payment mismatch is found on
the final balance (the user is only prompted when one is just detected
during a trial balance)”.
7.3 Further, Post Office have advised"! that they have “made enquiries” as to
confirm how Subpostmasters were notified of this particular issue but
have “not been able to find relevant records so far.”
7.4 It is understood that in the event of certain counter processing errors,
Subpostmasters would be presented with select on screen messages
informing them of some form of error. This would not in itself notify
Subpostmasters of the full extent of the potential implications of an
error occurrence (i.e., the error ultimately resulting in a shortfall or
discrepancy).
7.5 Some notifications would relate to procedures of which the Subpostmaster
would be aware (a failed recovery for example), however these would
only be at the counter. A Subpostmaster would not have visibility of
any error occurring for any transaction processing past the point that
a transaction was committed to the branch database (in respect of
transaction data errors), nor any potentially occurring further within
the processing systems.
7.6 KEL acha1941L,*” identifies that during the recovery process, when some
transactions recover but others fail to recover, it is only the recovered
21° SM BP Correcting Accounts for Lost Discrepancies - 102000790 - CD1.pdf, Correcting Accounts for
"lost" Discrepancies, 29 September 2010 [POL-0010769] [804ea47c166870b7ed0359e4765e0265]
211 Post Office’s response to Jason Coyne’s Requests for Information.pdf 08 August 2018
212 acha194iL.html, HNG-X KEL acha1941L, 11 July 2012 [POL-0039098]
[f392eca2f015eab903154e7a3e7a31ee]
Prepared by: Jason Coyne
Occupation: Partne
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page cxxi of 225
transactions printed on the receipt. The disconnected session receipt
should also identifyt those transactions NOT recovered. These are
printed for the Subpostmaster to retain. This would alert the
Subpostmaster to an instance of an error in counter processing in
respect of a failed recovery (known potential Horizon incident).
Alongside the printed receipts, it is further noted that upon an instance
of recovery failure, the system might display:
"“MSG90025: System Error - Error Code: 0058 has occurred Reason:
System error. Please retry once. If the problem persists, contact the
Horizon System Desk.”?1?
7.7 KEL surs1147Q?"* records another failed recovery issue to which the solution
is advised as follows:
“Advise the PM to log onto the relevant counter, start the recovery
process (MSG04024) but leave the counter displaying the System error
message (MSG90025). It is important that they DO NOT confirm this
message and that they just leave the counter to time out due to
inactivity at this screen - which will take at least 1 hour 20 minutes.
After this time, they can try to log on again - when there should be no
prompts about recovery and no system error.”
7.8 Therefore, as witnessed above, it is apparent that in specific event
circumstances, Subpostmasters were alerted to certain errors
occurring at the Horizon counter. However, it must be noted that
these relate to counter errors and therefore are not necessarily
notifications of bugs and defects within the system.
7.9 Similarly, in wrightm33145j*> as part of the process for rolling over to a
new Balance Period or new Trading period; the Subpostmaster was
23 KEL cardc1655P.html, HNG-X KEL cardc1655P, 26 July 2016 (last updated 2 August 2016) - [POL-0040501]
[09bc61dfc66a102d149337df7ce42a93]
24 surs1147Q.html, HNG-X KEL surs1147Q, 08 April 2010 (last updated 17 February 2016) [POL-0040368]
[4f8fdc56faa1f96e3e84522f4c49da75)
25 wrightm33145J.html, HNG-X KEL wrightm33145J, 23 September 2010 (last updated 01 April 2016)
[POL0040409] [1f025ec713c287ee7a5b1 7accd25b42f]
Prepared by: Jason Coyne
Occupation: Partne
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page Cxxii of 225
presented with a series of prompts and warnings if any discrepancies are
found.
7.10 If the Subpostmaster chose to cancel the rollover prompt message
(MSG31316) this could trigger a receipts/payments mismatch due to a
bug in the code when “cancel” is pressed against this message. The
workaround to avoid the bug was to press cancel a second time.
However, it is unclear how widely this workaround was communicated.
Subpostmasters would not be aware from the error message shown
that this was indeed a bug affecting a wide variety of branches.
7.11 It appears that the process was largely for Subpostmasters to notice an
error (either when trying to balance or after a communication outage)
and phoning the helpline. This would occur before it was
communicated or confirmed that there had been or was a bug/defect
which might have affected the branch accounts.
7.12 Some KELs record making priority calls to branches however, it is not
clear whether this occurred, and it is not noted that priority calls were
made in the event of every notified error.
Bugs Errors Defects not alerted to the Subpostmaster
7.13 As above, in respect of the known receipts and payments mismatch bug,
it does not appear that Subpostmasters were alerted, until perhaps an
investigation of discrepancy was performed.
7.14 Further, in respect of the acknowledged Calendar Square / Falkirk Issue
bug/defect it is not identified that Subpostmasters were informed.
7.15 As per the Joint Experts Statement, the extent to which any IT system
can automatically alert its users to bugs within the system itself is
necessarily limited.
7.16 Whilst Horizon has automated checks which would detect certain bugs,
there are types of bugs which would not be detected by such checks.
Prepared by: Jason Coyne
Occupation: Partne r;
Specialist Field: IT Systems 6 j1 UU
On the Instructions of: Freeths LLP ~ i
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page cxxiii of 225
7.17 In respect of the Dalmellington bug referenced at 5.16, there is no
indication that the Horizon system alerted the Subostmistress of this
bug. Any rectification was therefore dependent upon correct initial
diagnosis of the bug, proper handling processes by the Post Office and
communication provided to Helen Baker and the Subpostmistress.
7.18 It will be a matter for the court to determine in due course whether those
processes worked appropriately in relation to this bug.
How does Horizon enable Subpostmasters to compare the stock and
cash in a branch against the stock and cash indicated in Horizon?
7.19 There are several reports a Subpostmaster is required to run at the end
of each working day. These are designed to be checked against the
appropriate documents to amend any errors accordingly. These
reports are then ‘cut-off’ when the Subpostmaster is content they are
correct.
The cut-off routine is mandatory.
7.20 Of note there is a procedural requirement for staff at branches to count
and declare the cash stored in each stock unit at the end of each day
in what is called a Daily Cash Declaration.
Daily Cash Declaration
7.21 After performing the Daily Cash Declaration, Horizon will show any
discrepancy between the cash on hand and the amount of cash that
should be in the branch for the branch to balance.
7.22 A ‘Factfile’ document sent from Angela Van-Den-Bogerd of Post Office to
Second Sight'® states that Horizon can assist the Subpostmaster with
tracing or identifying discrepancies including event logs that are
available for 60 days (42 days pre-Horizon Online).
7.23 In addition to the daily cash declaration there is also the optional Weekly
Balance and mandatory Monthly Trading Period Rollover.
26 Factfile.DOCX, Initial Complaint Review and Mediation Scheme DRAFT Factfile, 16 April 2014, (P17 - 94.1) [POL-
0022996] [d24556f1009F1881f42a2dbb5b8154de]
Prepared by: Jason Coyne
Occupation: Partne . ey
Specialist Field: IT Systems § GLOU
On the Instructions of: Freeths LLP ° i
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page Cxxiv of 225
Weekly Balance
7.24 Although not mandatory the Weekly Balances are reported to be a
recommended action of POL policy to execute on a weekly basis; this
performs a full cash and stock account.
7.25 This should also help to detect any discrepancies. If a discrepancy is
discovered and declared it will be moved into a suspense account until
its resolution. The purpose of this is to act as a separate line in the
branch accounts that records any surpluses or loses whilst allowing
the daily trading accounts to balance. A Subpostmaster has until the
Monthly Trading Period Rollover to resolve any discrepancy (this was
carried out weekly prior to 2005).
Monthly Trading Period Rollover
7.26 The Monthly Trading Period Rollover is mandatory every month and as
stated above requires all discrepancies, including all within the
suspense account, to be resolved. At the end of this process a Branch
Trading Statement is produced to reflect the cash and stock shown in
the accounts matches the cash and stock held in the branch in
addition to any declared discrepancy.
How does Horizon enable or require Subpostmasters to decide how to
deal with, dispute, accept or make good an alleged discrepancy by (i)
providing his or her own personal funds or (ii) settling centrally?
7.27 Post 2005 is dealt with under Section 6.48 above.
7.28 Prior to 2005 Horizon followed a similar process however branches were
required to balance weekly and produced a Horizon “Cash Account”.
Discrepancies were, with the authorisation from the Post Office, held
in a “Suspense Section” of the Cash Account (although I have also
seen this referred to as a Suspense Account). Historically these were
known as “Unclaimed Payments”, “Authorised Cash Shortages” and
“Uncharged Receipts”, to be investigated.
Prepared by: Jason Coyne
Occupation: Partne . ae
Specialist Field: IT Systems 6 IY!
On the Instructions of: Freeths LLP Gey
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page cCxxv of 225
7.29 Following the conclusion of any investigation and in order to remove the
discrepancy, Subpostmasters were issued with an Error Notice (now
known as a Transaction Correction) by the post office or the
Subpostmaster placed money into the branch to cover the loss or
removed the value of the gain from the branch to balance the account.
7.30 The value of error notices had to be keyed in manually by the
Subpostmaster on the Housekeeping menu of the Horizon system
following notification from either a client or POCL. However, the losses
or surpluses at the end of a balancing period could still be disputed.
7.31 According to the Complaint Review and Mediation Scheme
documentation, I note that the ability to have a debt suspended
pending an investigation has only been available since August 2005.7!”
record and reflect the consequence of raising a dispute on an alleged
discrepancy, on Horizon Branch account data and, in particular:
7.32 This is covered under Heading 6 - Reconciliation and Transaction
Corrections (Section 6.61, Page 113)
does raising a dispute with the Helpline cause a block to be placed on
the value of an alleged shortfall; and
7.33 This is covered under Heading 6 - Reconciliation and Transaction
Corrections (Section 6.61, Page 113).
7.34 See also the Witness Statement of Dawn Phillips?*® Page 2 paragraph 10
which confirms that a block is placed upon the account until the
dispute is resolved. However, if branches do not return the completed
“Branch Dispute Form” detailing the discrepancy within seven days,
the shortfall is unblocked, and payment is once again requested from
the Subpostmaster.
217 0.19 10. Post Office - Complaint Review and Mediation Scheme.pdf, ca. 2014 [POL-0025512_1]
[6d54ae49b7132a421d48bbd941c64d39]
28 Signed witness statement of Dawn Phillips.pdf (dated 28 September 2018).
Prepared by: Jason Coyne
Occupation: Partne . er) IF
Specialist Field: IT Systems g Grout
On the Instructions of: Freeths LLP ° i
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page cxxvi of 225
7.35 It should be noted however that the Branch Dispute Form has only
recently been introduced in 2018 and the process of collecting key
information from the branch has been in place since November 2016
by other means such as email and telephone communications??°.
7.36 It is not fully clear what totality of mechanisms were available for
Subpostmasters to dispute discrepancies prior to 2016 although
Appendix D illustrates there is clearly an option available to dispute a
Transaction Correction.
is that recorded on the Horizon system as a debt due to Post Office?
7.37 A loss is recorded as a debt to the Post Office in the event the discrepancy
is upheld by the Post Office following any dispute.
enable Subpostmasters to produce (i) Cash Account before 2005 and (ii)
Branch Trading Statement after 2005?
7.38 As above the Cash Account were produced weekly after a mandatory
weekly balance just as a Branch Trading Statement is produced after
the Monthly Trading Period Rollover.
enable or require Subpostmasters to continue to trade if they did not
complete a Branch Trading Statement; and, if so, on what basis and
with what consequences on the Horizon system?
7.39 Subpostmasters are not able to continue trading until Branch Trading
Statement process is complete. If the Branch Trading Statement is not
completed and therefore, the Monthly Trading Period Rollover is not
completed the Post Office will contact the branch in order to rectify the
situation.
Opinion Summary
7.40 In many, but not all instances the Subpostmaster is alerted to a variety
of system errors or warnings whilst processing transactions at the
counter. However, these alerts and warnings do not necessarily alert
29 Signed witness statement of Dawn Phillips.pdf (dated 28 September 2018).
Prepared by: Jason Coyne
Occupation: Partne
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page cxxvii of 225
the Subpostmaster to the extent of the issue in the back-end processing
systems or necessarily allow him/her to make any adjustments.
7.41 I have not seen any evidence as to how known issues and bugs were
communicated to branches in advance of individual branches
discovering the issue themselves. In the example highlighted above in
(KEL wrightm33145j - Para: 5.10) there is no evidence any advice
was communicated to branches.
7.42 Subpostmaster feedback??? appears to highlight amongst many other
things an absence of support and training to assist them with being
able to find and resolve mistakes. In fact, this was expressed as
follows:
“There is a perception that there is a reluctance to report a shortage or
a balancing problem to NBSC as branches feel that the next stage will
be an audit and then suspension”
0 Gaps and issues final 9.10.13.x/s, Subpostmaster Feedback Spreadsheet, 17 October 2013 [POL-0216108]
[421233d56af42c063d50356a96a18e89]}
Prepared by: Jason Coyne
Occupation: Partne r;
Specialist Field: IT Systems 6 j1 UU
On the Instructions of: Freeths LLP ~ i
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page cxxviii of 225
8. Horizon Shortfalls, Data and Reporting for
Subpostmasters and Post Office
Issue 8 - What transaction data and reporting functions were available
through Horizon to Post Office for identifying the occurrence of alleged
shortfalls and the causes of alleged shortfalls in branches, including whether
they were caused by bugs, errors and/or defects in the Horizon system?
Issue 9 - At all material times, what transaction data and reporting functions (if
any) were available through Horizon to Subpostmasters for:
identifying apparent or alleged discrepancies and shortfalls and/or
the causes of the same; and
accessing and identifying transactions recorded on Horizon?
Data and Reporting Functions for Post Office
8.1 The data and reporting functions available to Post Office can be primarily
summarised by the following sources:
a.
b.
f.
g.
The TPS Report Set;
The APS Report Set;
The DRS Report Set;
Reports and Data Obtained via Business the Incident Management
(“BIM”) Process;
Reports and Data Obtained via the Problem Management
Procedure;
Information Obtained via Fujitsu; and
Information Obtained from Subpostmasters.
8.2 Detailed descriptions of each of the above as set out in Appendix E to this
report. Summaries of each are set out below.
Prepared by: Jason Coyne
Occupation: Partne . re
Specialist Field: IT Systems ¢ y!
On the Instructions of: Freeths LLP
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page CXxxix of 225
Official Sources of Information
8.3 The TPS Report Set - The TPS Report Set was made up of 3 reports
(TPS250/254/257) which were designed to enable the reconciliation of
transactions using branch infrastructure. They are all daily reports
which show information relating to whether transaction outputs at
branches matched transaction outputs at Post Office, whether there
are exceptions in the BRDB copy process and which branches have a
net total transaction (i.e. debit/credit) transaction total which does not
equal 0.
8.4 The APS Report Set - The APS Report Set was made up of 10 reports that
were designed to reconcile those transactions that were sent to both
the POLSAP system and APS clients. The APS Report Set provides
confirmation that the APS transaction account balances, a summary of
transactions which have been delivered by APS, a summary of
transactions which have not been delivered due to delay/quarantine, a
reconciliation summary between transactions which flow through the
APS host and TPS host and confirmation that all branches have been
harvested / details of any that exist in relation to harvesting.
8.5 The DRS Report Set - The DRS Report Set (sometimes referred to as
“Banking & Related Services Reconciliation”) is made up of 3 reports
which were designed to enable network banking transactions
completed in Post Office branches to allow settlement to be made with
Post Office clients (e.g. Santander, LINK, etc.). The DRS Report Set
provided a summary of all reports that were not produced by DRS due
to a lack of data, identified all C4 transactions received against each
C4 settlement date and information relating to all exceptions.
8.6 Reports and Data Obtained via the Business Incident Management (“BIM”)
Process - The BIM system was designed to report progress on the
resolution of Business Incidents to allow Post Office to complete
reconciliation or settlement with its internal systems, clients and
Prepared by: Jason Coyne
Occupation: Partne ° ey
Specialist Field: IT Systems g tQrou
On the Instructions of: Freeths LLP ~ i
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page Cxxx of 225
banks.**! A “Business Incident” describes the effect of a system fault
and can relate to any of the exceptions from the various reports or a
settlement error discovered by Post Office. A “System Incident”
describes the underlying cause of a Business Incident and is created to
track the root cause of the same. A BIM Report is issued for each
Business Incident in order to notify Post Office of issues and to assist
in the reconciliation or settlement process. Note that BIM Reports
communicate information concerning the resolution of an issue, and
not the underlying cause (which would be dealt with via the Problem
Management Procedure).
8.7 Reports and Data Obtained via the Problem Management Procedure - The
Problem Management Procedure aims to eliminate the root cause of
issues (e.g. System Incidents).
Additional Sources of Information
8.8 Information Obtained via Fujitsu - Most of the sources above will include
significant involvement from Fujitsu and some, such as the TPS Report
Set, will only be made available to Post Office if requested. However, it
is important to note that the above only represents the formal sources
and incident management processes. Post Office also had access to
information simply by nature of the commercial and_ technical
arrangements between itself and Fujitsu. In other words, there was
nothing (other than cost) preventing Post Office from seeking further
information from Fujitsu directly if, for example, the Problem
Management Procedure did not yield a satisfactory conclusion. This is
unlikely, as the reporting and management functions above are quite
comprehensive and should cover the vast majority of issues, but it is
Possible.
221 CSPRO111_4.1.doc, TPS Reconciliation & Incident Management (Section 4.4.1; BIMS Reports/MER), 10
June 2015” [POL-0082393] [Sad9b694ade347339b5f5a2b49c88ea2]
Prepared by: Jason Coyne
Occupation: Partne ° ey
Specialist Field: IT Systems d JFOU
On the Instructions of: Freeths LLP ° i
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page Cxxxi of 225
8.9 Information obtained from Subpostmasters - In addition to the above, the
Subpostmasters themselves and any information available to them
were sources of information available to Post Office. These are further
considered below.
Conclusions Relating to Information Available to Post Office
8.10 The TPS, APS and DRS Report Sets along with reports produced in
accordance with the BIM and Problem Management Procedures
provided a comprehensive suite of reports which was available to Post
Office. These reports should have allowed Post Office to identify the
occurrence of alleged shortfalls in the Horizon system (of those that
could be identified), and they were underpinned by formal processes
which would provide further information in relation to the underlying
cause of a given issue, and the best way to resolve the same. In
addition, Post Office should have been able to obtain any additional
information it required via Fujitsu or the Subpostmasters themselves.
Data and Reporting Functions for Subpostmasters
8.11 Subpostmasters had access to a much smaller pool of information. This
is in line with what I would expect to see given that Subpostmasters
are the users of the Horizon system, and therefore would not typically
be given access to anything beyond what was necessary for them to
carry out their ‘business as usual’ activities. In relation to this matter,
that means being able to carry out the day-to-day transactions
required to run a post office, which are dealt with via the “counter”.
8.12 All the reports and receipts produced by the counter are set out in
SD/DES/00577* - note this document excludes reports and receipts
which are not produced by the counter. It includes hundreds of reports
relating to different aspects of running a Post Office branch (currency
exchange, insurance, stock, etc.). The reports and receipts contain
222 SDDES005_12.2.DOC, Horizon OPS Reports and Receipts - Pathway - Horizon Office Platform Service, 23
January 2003, (Version: 12.2), [POL-0069333] [3d8f3190f530f5Se71916099859b31dbd]
Prepared by: Jason Coyne
Occupation: Partne ° ey
Specialist Field: IT Systems g tQrou
On the Instructions of: Freeths LLP ~ i
180503R1935
FUJ00082162
FUJ00082162
16 October 2018 Page cxxxii of 225
basic information relating to individual transactions (e.g. item, time,
branch, etc.) and are therefore a useful source of information when
performing normal reconciliation activities.
8.13 However, as these reports are specific to counters and contain no
information beyond this, they would not allow a Subpostmaster to
determine the cause of an issues that arise at anything beyond
counter level (and possibly even those that arise at counter level).
8.14 For example, if an APS transaction reversal was carried out, the
Subpostmaster would receive a receipt which looks something like the
following:
Example content
1 2 3 4
123456789012345678901234567890123456789012
01 I Feltham Post Office PAD: 123456X
02 I 23/09/2006 10:47 TP:06 BP:01 SU:SH1
03
04 REVERSAL
0s
06 *** Branch Copy — Retain ***
07
08 I Checksum: 9062852
09 I APS No: 010058010057
10 I Client: Eastern Electricity
11 I Scheme: EE MthBill Sve: 8
12 I Token Type: BC Entry: 1
13 I Ref: 6331801325640003333
14 I Amount: 5.00- Cash
15 I Product No: 7022
16
17 I ----------------------------------------
123456789012345678901234567890123456789012
8.15 This shows enough information for a Subpostmaster to balance a £5
reversal if it is assumed that all other factors are in order. However,
there is no information relating to whether the transaction has
reconciled at APS Host or at any other level (harvester, client, etc.).
This could cause an issue which the Subpostmaster is effectively
powerless to resolve.
Prepared by: Jason Coyne
Occupation: Partne
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page cxxxiii of 225
8.16 If, for example, the harvester failed to process the £5 reversal above
then there would appear to be a £5 shortfall at the APS Host (and
potentially at every other level above harvester). The Subpostmaster
would have no way of identifying where the error occurred because, at
counter level, everything would appear to balance, and they would not
have access to any information beyond that. This essentially means
that the Subpostmaster is completely reliant on either:
a. Horizon operating without any flaws (which is unlikely even in the
case of a robust system); or
b. Where there is a flaw, it will be identified and linked to a problem
transaction by Post Office / Fujitsu.
Additional Points to Note
8.17 There was no formal reconciliation process between the POLSAP system
and the Credence transaction stream, so Credence could not be used
to verify financial integrity. POLSAP system transaction information
should have been used for this purpose.?2°
8.18 A Post Office cash management improvement proposal?* recently
acknowledged the lack of visibility for the Subpostmaster of any
inaccurate cash declarations. The proposed cash management
improvement proposal would give the Subpostmaster immediate
visibility of any discrepancies and allow faster
corrections/investigations to find the inaccurate transaction. It is not
known if or when this change was to be implemented but it confirmed
that the Subpostmaster has little control beyond counter level when
trying to resolve any discrepancies.
2 SVMSDMSD0020_2.2.doc, End to End Reconciliation Reporting, 27 February 2012 [POL-0124572]
[aaedd5051156619c50296d04f6b2e779)}
224 Cash Management Programme - Business Case June 2017 v1.5 TW.docx, Business Case - Cash
Management Programme, 4 August 2017, [POL-0220663] [8cd4dac4464197347c0fb2348a6e8f59]
Prepared by: Jason Coyne
Occupation: Partne
Specialist Fiald: IT Systems § Or
On the Instructions of: Freeths LLP
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page Cxxxiv of 225
Issues 8 & 9 - Summary of Conclusions
8.19 The TPS, APS and DRS Report Sets along with reports produced in
accordance with the BIM and Problem Management Procedures
provided a suite of reports which should (subject to the above) allow
Post Office to identify the occurrence of alleged shortfalls in the
Horizon system. These reports were supported by manual processes
which would provide further information in relation to the underlying
cause and resolution of a given issue. In addition, there was nothing
preventing Post Office from seeking further information from Fujitsu
directly if, for example, the Problem Management Procedure was
insufficient.
8.20 Subpostmasters had a much smaller pool of information available to
them, which is in line with what I would expect given that they are the
end users of the Horizon System. The reports and receipts available to
Subpostmasters generally show enough information for a
Subpostmaster to balance transactions if it is assumed that all other
factors are in order. However, this information would not allow a
Subpostmaster to determine whether a transaction has reconciled at
APS Host or at any other level (harvester, client, etc.).
8.21 There is evidence that some reports available to Subpostmasters were
reporting erroneous data because of changes made to stock units?*.
In this case some products were being double counted in a sales
report. A code fix was scheduled in COUNTER_EPOSS 34_7 but it is
not known if or when this was implemented.
8.22 In conclusion, Post Office had access to far more comprehensive
information relation to the Horizon system. If an error occurred
beyond counter level, Subpostmasters would need to rely on Post
Office to identify and resolve the issue. If that issue or its was not
#5 CCard2053P, HORIZON KEL CCard2053P, 21 December 2005 (last updated 08 September 2006), [POL0035339)
[92ccb572ff4ed5f357d3569a4b4121e2]
Prepared by: Jason Coyne
Occupation: Partne ° ey
Specialist Field: IT Systems d JFOU
On the Instructions of: Freeths LLP ° i
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page Cxxxv of 225
properly identified for any reason, then the Subpostmaster would be
at risk of being liable for a Transaction Correction.
Prepared by: Jason Coyne
Occupation: Partne . ey
Specialist Field: IT Systems § GLOU
On the Instructions of: Freeths LLP ° i
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page Cxxxvi of 225
9. Remote Access and Alteration of Transaction Data
Issue 7 - Were Post Office and/or Fujitsu able to access transaction data
recorded by Horizon remotely (i.e. not from within a branch)?
Issue 10 - Whether the Defendant and/or Fujitsu have had the ability/facility
to: (i) insert, inject, edit or delete transaction data or data in branch
accounts; (ii) implement fixes in Horizon that had the potential to affect
transaction data or data in branch accounts; or (iii) rebuild branch transaction
data:
a. atall;
b. without the knowledge of the Subpostmaster in question; and
c. without the consent of the Subpostmaster in question.
Issue 11 - If they did, did the Horizon system have any permission controls
upon the use of the above facility, and did the system maintain a log of such
actions and such permission controls?
Issue 12 - If the Defendant and/or Fujitsu did have such ability, how often was
that used, if at all?
Issue 13 - To what extent did use of any such facility have the potential to affect
the reliability of Branches’ accounting positions?
Remote Access
9.1 A number of technical documents provided by Fujitsu identify that the
Horizon estate was enabled to be managed remotely. This is not
beyond expected requirements. The nature of providing a support
service as Fujitsu do, would require as a design principle, that the
Horizon solution should be completely remotely manageable.??°
9.2 Document ‘1.8 8. Horizon Online Technical Network Architecture, Mark
Jarosz.pdf’ (dated 2010) at section 2.5.3 sets out the network facilities
28 POLSAP High Level Design, 25 March 2010 [POL-0032871] [66768ce6f7144c10d1b68flec2d612f8]
Prepared by: Jason Coyne
Occupation: Partne ° ey
Specialist Field: IT Systems g tQrou
On the Instructions of: Freeths LLP ~ i
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page Cxxxvii of 225
that allowed “Remote Access by Fujitsu services, users and
systems”,227228
The document recorded that “Support access to Counters is via SSH...”
further, the document shows that the Fujitsu sites Bracknell,
Stevenage and Ireland were originally envisaged to have access.
9.3 A further document describing the Horizon Wide Area Network High Level
229
Design*”? sets out:
“Remote support access to the counter will be provided through the
implementation of an SSH service running on the counter which can
then be accessed from the Secure Access Servers, in the data centre....
this will allow access to command prompt on the counter for the
retrieval of logs and other data using secure copy (SCP)”.
9.4 It is therefore clear that Fujitsu had access to the servers which make up
the Horizon estate, so as to access counters within a branch. This
access was required to enable them to provide support and
maintenance. 3rd line support had the greatest level of privileges to
“Support access on platforms operating system, hosting applications
and database schemas”.**° This level of access means that Fujitsu
could practically access all elements of data recorded within Horizon.
9.5 Further to the above access facilities, helpdesk logs identify that ‘Tivoli
Enterprise Manager’ (TEM)**! and Tivoli Remote Control tools were
specifically used for accessing Branch counters across the Horizon
estate.?? Whilst the End to End Application Support Strategy
27 1.8 8. HNG-X Technical Network Architecture, Mark Jarosz.pdf, HNG-X Technical Network Architecture, ca.
8 [C-0003647] [4c0c965c5d6ca96a56059693625cf29c]
9 2.1 11, HNG-X Wide Area Network HLD, Stephen Wisedale.pdf, HNG-X Wide Area Network HLD, 19
December 2012 [C-0003646] [c86656726b30cea6fcle2dfecch4457a]
29 1,3 3, High level architectural overview of Horizon Online.pdf, Horizon Solution Architecture Outline, 07 April
2016 [7b6cfBcf69bec90F674b9b10a64f04e8]
21 'M100_POL_003_HSH Fujitsu logs_JHKD.xlsx, Fujitsu Helpdesk Logs, 28 October 2014 [POL-0006655]
[4120881082b106161bc4acbd75c15b7a]
22 RKing5147Q.html, HORIZON KEL RKing5147Q, 8 August 2006 [POL-0035318]
[1e2815966220f5c7 1ccd3c99bcf82c16]
23 SVMSDMPRO0875_1.DOC, End to End Application Support Strategy, 28 July 2011 [POL-0122492]
Prepared by: Jason Coyne
Occupation: Partne
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page Cxxxviii of 225
document”? records at 1.5.1 “Direct access to live Post Office
counters is currently only available to the SSC."
9.6 Whilst the above documentation identified that counters could be accessed
remotely for support purposes, it has not yet been identified that
transaction data was altered at the counter.
9.7 However, it is my opinion that it was possible Fujitsu was able to access
transaction data recorded by Horizon both within a branch (and also
within the central BRDB database for Horizon Online). This was how
the system was designed.
9.8 As evidenced above, it is entirely possible that Fujitsu could, as they could
run commands on the counter machine accessing and querying the
hard disk, perform modifications and deletions.
9.9 The witness statement of a Fujitsu chief architect?>* further confirms that
Fujitsu can and have inserted a balancing transaction into a Branch
Account, and in Legacy Horizon could inject transactions into branch
accounts (which at the time would have been stored on the branch
counter hard drive).
9.10 The Witness Statement of Richard Roll? further confirms that Fujitsu
employees could and did remotely access branch accounts to perform
modifications.
Further supporting evidence of remote access
9.11 A number of the Known Error Logs (KELs) refer to remote access activities
between the branches and the Fujitsu support facility.
9.12 boismaisons1328M7*° 23” describes running commands on counters to
assess disk space sizes. This therefore illustrates Fujitsu’s capabilities
to access the counter hard disk.
[db0644e4d5e11bScce3ed381cb108a88]
24 Witness Statement of Torstein Godseth, 27 September 2018 (Para 17.2 (d))
295 Witness statement of Richard Roll 11.07.16.pdf dated 11 July 2016
26 boismaisons1328M.html, HNG-X KEL boismaisons1328M, 24 April 2012 (last updated 20 March 2013) [POL-
237 I [a7737d3b003d083b5a6c95c7edeef534]
Prepared by: Jason Coyne
Occupation: Partne r;
Specialist Field: IT Systems 6 j1 UU
On the Instructions of: Freeths LLP ~ i
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page Cxxxix of 225
9.13 KEL acha2026Q?** (last updated 27th September 2012) documents an
issue recorded as “Unable to connect to HNGX counter from ssn”. The
reasonable conclusion being that normally Fujitsu (or other support
personal) can connect to the Horizon counter from the Secure Access
Server. The KEL explains that: “SSC [Fujitsu Software Support Centre]
will be unable to connect to this counter until problem has been
resolved. Solution - ATOS".
9.14 KEL MillerK1837J** illustrates that deletions in relation to Reference Data
files could be made at counter level.
Global Branches
9.15 Fujitsu operate the Horizon Online Help Desk located at two sites
(Bracknell and Stevenage). These sites contain ‘global branches’
therefore they are not physical instances of a Post Office but exist as
‘virtual branches’. There are however, physical counters that perform
within them.?*° The ‘branches’ operate with branch codes 999999,
counter ID’s 1 - 6 and 999998, counter IDs 7 - 12.
9.16 The HNG-X Counter Business Application Support Guide*** further sets
out how to perform an audit of which branch a global user last logged
in at.
9.17 A register of branch codes (dated 2015)" identifies a further global branch
recorded as WAKO1 Branch Code 999993.
9.18 An instance of a global branch would allow Fujitsu to create global users
and to input transactions within core Horizon systems as though they
had been entered from a physical branch.
28 acha2026Q.html, HNG-X KEL acha2026Q, 15 October 2010 (last updated 27 September 2012) [POL0039179]
[6a5c608d2812487a3c044473fc6a9e45]
29 MillerK1837J.html, HNG-X KEL Millerk1837J, 04 June 2010 (last updated 24 April 2015) - [POL-0040112]
[1da69d5ae267#707¢2702f926c2e4384]
240 DEVAPPSPGO017_7.1.doc, HNG-X Counter Business Application Support Guide, 8 January 2014 [POL0134853]
[97581900782d355b4965045331797cea]
241 DESGENSPE0013_1.doc, Register of Branch Codes, 4 November 2015 [POL-0142404]
[5536045daeaf587fec423f2782928123]
Prepared by: Jason Coyne
Occupation: Partne
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page cxl of 225
9.19 It is entirely possible that investigation could be further conducted by
Post Office to identify any transactions held within the BRDB
containing the Branch Codes identified above. Such would identify
where and what transactions had been performed by Fujitsu global
branches and not a Subpostmaster.
Branch Transaction Data Rebuilds
9.20 It is understood that branch transaction data rebuilds did take place across
the estate.
9.21 POL-0116724** documents the reasons for changing a counter base
unit. Replacement of the counter base unit at 2008 would ultimately
require the branch transaction data to be reinstated to the new
machine. This would be carried out by both the engineer on site and
automatic processes applied by the HSD (Horizon Service Desk).
9.22 Further, Richard Roll in his witness statement at paragraph 15747
documents how it was relatively common to re-create branch
databases in an effort to fix corruptions.
Transaction Correction Tools - Modification of Transaction Data
9.23 There are various identified points within the Horizon architecture where
Fujitsu may need to perform data correction activities. This involves
manually correcting data where it has become corrupted or is
harvested as in an ‘error’ or ‘exception’ state.
9.24 POL-0219310*? at Section 5.6.2 states:
“There is a requirement that the SSC will have ability to insert
balancing transactions into the persistent objects of the Branch
Database. There are reasons for SSC having to do so e.g. to rectify
erroneous accounting data that may have been logged as a result of a
bug in the Counter / BAL.
242 SVMSDMSTD0810_1.2.doc Engineer Handbook Base Units, 20 June 2008 [POL-0116724] 747
Witness statement of Richard Roll 11.07.16.pdf dated 11 July 2016 paragraph 15
2° DESAPPHLD0020.doc Branch Database High Level Design, 22 February 2018 [POL-0219310]
[e277e9fd3e3ad2dd17ab40afc0d2096d]
Prepared by: Jason Coyne
Occupation: Partne
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page cxli of 225
SSC will have privileges of only inserting balancing /correcting
transactions to relevant tables in the database. SSC will not have the
privileges to update or delete records in the database.
Any writes by SSC to BRDB must be audited...”
9.25 When applying the corrective fixes, it appears that Fujitsu would utilise
the branch accounting code of the branch for which the correction
transaction was required.?**
Branch Transaction Correction Tool
9.26 Host BRDB Transaction Correction Tool Low Level Design***(applicable to
Horizon Online).
9.27 The ‘Overview’ section (section 1.1) states:
“Warning: The use of this powerful tool has inherent risks. If the SQL
statement is incorrect or badly written, it is possible to cause
unintended consequences, some of which may cause serious problems
to the Branch Database. It is expected that only a small number of
skilled staff will run this tool and that they will have detailed guidance
as to when and how to use the tool.”
9.28 The document does not stipulate which staff within SSC have privilege to
run the tool, nor document the guidance on specifically when the tool
should be run. This has been the subject of a Request for Further
Information.
Correction Tool auditing
9.29. The schema definition for the BRDB_ identifies the following
auditability :24°
244 DESAPPHLDO020.doc Branch Database High Level Design, 22 February 2018, Para: 5.6.2 [POL-0219310]
[e277e9fd3e3ad2dd17ab40afc0d2096d]
245 DEVAPPLLD0142.doc, Host BRDB Transaction Correction Tool Low Level Design, 13 November 2007
[POL0032866] [e20de9a651b8baf1f84e10859455684b]
246 DEVAPPLLDO199_5.DOC, SCHEMA DEFINITION FOR BRANCH DATABASE, STANDBY BRANCH DATABASE
AND BRANCH SUPPORT SYSTEM, 01 August 2017 [POL-0151776] [456dd7c0f245946a2f1658cc759a527a]
Prepared by: Jason Coyne
Occupation: Partne
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00082162
FUJ00082162
PS_OUTLETS
I fad hashr NUMBER(3)
I ards imestamp: TIMESTAMP(S)
4
treding_ per
9.30
Users of the branch correction tool should be identifiable by its audit
table BRDB_TXN_CORR_TOOL_JOURNAL (as above, bottom right
table). The corrective transaction in the branch accounts should also
be identifiable as an automatic ‘“SYSTIMESTAMP’ should be recorded
rather than a retrospective transaction time. ‘SUPPORTTOOLUSER’
should be reflected in the ID field.
9.31 It is entirely possible that investigations could be further conducted by
Post Office to identify any transactions held within the
BRDB_TXN_CORR_TOOL_JOURNAL table. Such would identify what
corrective transactions have been performed. However, given file
retention periods, it is unlikely that all corrective transactions will have
been retained from the inception of Horizon Online.
9.32 A Request for Information issued to Post Office querying how many times
the Host BRDB Transaction Correction Tool has been used was
responded by Fujitsu stating:
“This process has only been used once, in relation to PCO195561, on 11-
Mar-2010.”
TIP Transaction Repair Tool
Prepared by: Jason Coyne
Occupation: Partne
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page cxliii of 225
9.33 TPS - EPOSS Reconciliation - TIP Transaction Repair 24? documents a
further maintenance tool that “...will assist SSC to repair EPOSS
transactions processed at the counter but are unable to copy from
BRDB into the TPS (Transaction Processing System) Host.”
9.34 The repair tool is documented to assist SSC to repair/re-repair the
transactions and send them to TIP (Transaction Information
Processing) (the remote end-point of the File Transfer Management
System (FTMS) service that delivers reconciliation reports to Post
Office Limited / PON’s (Post Office Network Banking) Transaction
Information Processing System).
9.35 The above tool allows corrective actions to be performed upon data
within Core Horizon after the counter has processed the transactions
and they are flagged as erroneous as they are sent through the
various processing systems.
9.36 The transaction repair is facilitated by a Form based tool that will:
a. read data from the TPS Host table holding Harvester Exceptions
b. allow the user to repair/re-repair the transactions, i.e., input any
missing data or modify any invalid data
c. validate and insert the corrected transactions into a 65th TPS Host
Partition table
9.37 Further, at 3.2.1 the tool is stated to:
“Allow the user to do multiple repairs at speed with the minimum of user
intervention i.e., if there is a common error among several records then
the user can input the valid value just once and correct all the records
having the similar error...”
9.38 In relation to how often this tool was used, Fujitsu have provided the below
response:
247 PIDESO8.doc, TPS - EPOSS Reconciliation - TIP Transaction Repair, 11 January 2017 [POL-0032939]
[27249c8e2ccba0fecdc16223cdda8f7a}
Prepared by: Jason Coyne
Occupation: Partne r;
Specialist Field: IT Systems 6 j1 UU
On the Instructions of: Freeths LLP ~ i
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page cxliv of 225
“There is a master MSC every 12 months, each time such a
modification is carried out it is itemised as an MSC related to the
master MSC; however master MSCs contain many various types of
changes, to determine the number that relate to this particular
modification type Fujitsu would have to carry out analysis of all
individual tasks on all master MSCs. Whilst this type of action may
have been taken by SSC it would have been in the context of an
individual incident. All incidents are recorded but the system was
designed to manage individual operations not for statistical reporting
for when a particular action has been taken by a Support Consultant.
Fujitsu will be able to answer questions on individual branch queries
where the data is still available.”
9.39 The witness statement of William Membury*® at paragraph 27 explains
the use of the MSC toolset which is used to manage changes to
Horizon Online. He explains that the MSC tool set “provides assurance”
as to the changes made to Horizon. Whilst I have requested copies of
the MSC documentation at the date of this report Post Office has
refused to provide these.
Effects on Branch Accounting
9.40 The above tools are noted as having the potential to affect transaction
data and potentially branch account data by way of incorrectly altering
the transactions prior to entering the recipient systems such as
POLSAP and External Clients (after processing by the counter).
Thereby resulting in potential discrepancies being borne between
recipient systems and those recorded in branch.
9.41 This could potentially result in the issuing of a Transaction Correction issued
by Post Office who may be unaware of the error induced within
the processing of the corrected transaction. Once accepted by branch
this ultimately modifies the branch’s accounting position.
248 Witness Statement of William Membury, 27 September 2018. (Para: 27)
Prepared by: Jason Coyne
Occupation: Partne r;
Specialist Field: IT Systems 6 j1 UU
On the Instructions of: Freeths LLP ~ i
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page cxlv of 225
Notwithstanding, the possibilities of human input error exists,
specifically if the aim is to perform “multiple repairs at speed”.
9.42 Whilst 9 above states that transactions will be validated, this is
interpreted as validated in terms of valid data types according to the
check constraints imposed on those specific columns. For example,
should a clerk enter a date into a value field, then it is assumed that
that will fail validation however, if a clerk enters an amount of 900
where it is supposed to record 90, it is doubtful that the validation
rules will interpret that as incorrect as the numerical value type is
correct but there is no validation against the amount. Therefore,
human error is possible. Projecting onwards, since these values are
then sent onto clients, there is facility for discrepancy potentially
resulting in a Subpostmaster being liable to pay for a discrepancy over
which they have no control.
9.43 Therefore, in respect of Issue 10 point (i), it appears that Fujitsu did
have the ability to insert, inject, edit and (potentially) delete
transaction data and (ii) had the ability /facility to implement fixes in
Horizon that had the potential to affect transaction data or data in
branch accounts.
9.44 In respect of Issue 10 part (iii), no evidence of specific branch data
rebuilding has been discovered however, since Fujitsu had the remote
access capabilities it is entirely possible that it could have occurred,
and the witness statement of Richard Roll refers to the ability.
9.45 Fujitsu had the capability and dedicated tools to alter transaction data
held in the BRDB and Horizon processing systems remotely, therefore
it is likely this would be without the knowledge or consent of the
Subpostmaster (of the branch that the transactions applied within).
Further actions affecting branch accounting
249 DEVAPPSPGO017_7.1.doc, HNG-X Counter Business Application Support Guide, 8 January 2014 [POL0134853]
Prepared by: Jason Coyne
Occupation: Partne ° ey
Specialist Field: IT Systems d JFOU
On the Instructions of: Freeths LLP ° i
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page cxlvi of 225
9.46 HNG-X Counter Business Application Support Guide**® documents Help
Service Desk calls that might result in a stock unit needing to be
unlocked on a counter which can be performed by global users with
Admin role:
“In such situations the clerk must call the Horizon System Desk and
request the stock unit to be unlocked. This call is passed on to System
Support Centre (SSC) who operates a manual process to unlock the
stock unit. To ensure proper accountability this manual process has a
high administrative overhead which results in significant work for SSC
staff and a long delay for the clerk before the stock unit is unlocked.
This function unlocks the locked stock unit in a quick, secure and
audited way.”
9.47 The document further sets out:
“During the logon process, the user will be forcibly logged off if some
unexpected error occurs (during recovery or post logon checks (e.g.
change password). If this happens repeatedly, in extreme cases and as
a short-term measure purely as a workaround, then the recovery
and/or post-logon checks can be bypassed. It must be stressed that
this is only to be used for situations where a very fast workaround is
required in live. To bypass recovery at logon, use the following
application. properties override:logon.recoveryEnabled=false To bypass
post-logon checks, use the following application.properties
override: logon.postLogonChecksEnabled=true Note that in both cases,
when these flags are used, once logon is eventually completed, the
user may need to click on the back office/front office button twice in
order to
“reset” the counter to a good state.”
9.48 Resetting the counter and bypassing recovery jobs could impact branch
accounts in that items awaiting recovery might be lost. When someone
from NBSC investigates this in the event of a discrepancy, they would
probably not be aware of the action carried out to bypass recovery.
[97581900782d355b4965045331797cea]
Prepared by: Jason Coyne
Occupation: Partne
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page cxlvii of 225
Therefore, their understanding and belief in respect of what should
happen in normal recovery procedure will be at odds with actuality. In
this instance it could be that the Subpostmaster bears the effects of
this (unknown to Post Office) activity.
Supporting Evidence of remote access and implemented fixes affecting
transaction data
9.49 KEL SeemungalG519Q**° records an instance where transaction
amendments carried out using the above TIP repair tool at 9.13 are
causing exceptions within the BRDB.
9.50 KEL MHarvey2255P2*! records the manual addition of corrective balancing
transactions inserted by SSC affecting the TPS system.
9.51 There are potentially more KELs and PEAKs that record the effects of
performing corrective transactions which have not been identified at
the time of this report.
Permission Controls and Data Auditing
9.52 Typically, in an architecture such as Horizon, there are permission
controls set out by roles and privileges that accommodate a support
service's ability to manage the estate.
9.53 Administrator capabilities will allow for the management and
maintenance of the system. Database administrators will have
escalated privileges to allow them to implement changes within the
database systems and delegate roles and privileges to others. For
example, to set permission controls for ‘managers’ of a branch,
elevated to that of a ‘user’.
9.54 Typically, documented procedures will be in place to set out what each
defined role can perform within a system. Audit logging is common
practice aside from the electronic logging that will occur in a system.
250 SeemungalG519Q.html, HNG-X KEL SeemungalG519Q, 15 January 2010 (last updated 28 April 2014)
[POL0039787] [ba2c5c8717aeb77c9a65452e29ec57ba]
251 MHarvey2255P.html, HORIZON KEL MHarvey2255P, 02 September 2005 (last updated 31 May 2006)
[POL0035233] [ebd26444c25a88aa178de7cd4be58d27]
Prepared by: Jason Coyne
Occupation: Partne r;
Specialist Field: IT Systems ¢ 41 UU
On the Instructions of: Freeths LLP ~ i
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page cxlviii of 225
Administrator actions will typically be overseen by other methods of
audit.
9.55 For example, the common ‘four eyes’ principle ensures that an action to
be performed is approved and observed by more than one individual
where the action to be performed can have a significant effect (as
administrator capabilities typically do).
9.56 As there are a number of tools and means by which Fujitsu can perform
alterations within the Horizon system (since supporting the system
was their primary role) the findings below regarding the auditability of
permission controls is limited to those which relate to transaction data
amendments which had the ability to affect branch accounts (in
answer to Issue 11).
Balancing / Corrective Transactions
9.57 Reconciliation Service: Service Description?** records the following:
“If the Reconciliation Service identifies that Transaction data held on
the ‘central database’ located at the Data Centre is found to be
inconsistent when compared to the records of the Transaction that was
completed at the Branch, e.g. a receipt, a Transaction log or a Branch
accounting discrepancy, the Reconciliation Service shall obtain
authorisation from Post Office prior to the insertion of corrective
Transactions.”
9.58 It is understood that the request for authorisation as referred to above
may be that documented within the Customer Service Operational
Change Procedure.*°? The document sets out the process requirements
in respect of operational changes where changes are made to the live
environment.
Operational Change Proposal
252 SVMSDMSD0015_4.doc, Reconciliation Service: Service Description, 03 December 2013 [POL-0134458]
[49b8029c58cb2f714b2ba7e034d6F280]
253 CSPRDO19_1.doc, Customer Service Operational Change Procedure, 18 March 2004 [POL-0074909]
[439e3230106a96792bf2dd9831729392]
Prepared by: Jason Coyne
Occupation: Partne
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page cxlix of 225
9.59 An OCP is raised in order to make a change to the live system. The
process is administered by Post Office Account Operations and is
available to all users for the administration, authorisation and auditing
of changes made to the live operational service.
Operational Correction Request
9.60 The OCR process involves the correction of customer data on the live
system and because user data is involved, requires different approvals
and auditing. The document further states:
“Only the SSC has the authority to make changes to the data on the
system, and therefore only SSC staff can action an OCR.
In most cases, an OCR does not involve the financial integrity of the
system. Under these circumstances one of the SSC Manager, the
Support Services Manager or the Customer Service Duty Manager can
approve an OCR. If the data to be changed has a financial impact on
Post Office, then approval must also be given by a senior Post Office
Manager.
When an OCR has been approved, and has been actioned, it is
necessary for two users of the OCP system to confirm that the work
has been done - an actionee and a witness. The actionee will always
be an SSC staff member, the witness can either be an SSC staff
member or a development staff member.”
9.61 Since the definition of what “involves the financial integrity of the
system” is not documented, it is not clear whether those OCRs that
could then be approved without Post Office’s authorisation were
appropriately decided. Ultimately, Fujitsu could approve and action the
OCR independently.
9.62 With regards to Issue 12 and how often facilities were used in relation to
accessing/modifying transaction data a Request for Information issued
to Post Office provided the response:
“.there are in excess of 36,000 MSCs and OCPs combined; and.
Prepared by: Jason Coyne
Occupation: Partne r;
Specialist Field: IT Systems 6 j1 UU
On the Instructions of: Freeths LLP ~ i
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page cl of 225
+ OCRs would not be used for any such change (OCRs were used minor
support changes that did not required the full approval process that
was needed for OCPs)...”
9.63 The above is inconsistent with the operational change document at 9.60
which maintains that they would apply for Live data changes.
Audit Servers
9.64 The audit servers provide an audit trail of all information on the Horizon
Online system. In order to ensure that this audit trail is irrefutable the
teams which have the ability to change data (i.e. SSC) must not also
have the ability to change the audit trail. For this reason, Audit server
3rd line support rests with the Audit development team and not the
SSC. This is known as ‘separation of duties’ and is a type of control.
Process Variations
9.65 The Ernst & Young review of Post Office’s systems of internal control
conducted in March 20112°* observes in relation to Horizon (back end)
user administration:
“During our testing of the appropriateness of users with access to the
Horizon back end environment we noted one user whose access was no
longer required due to a change in job responsibilities. When users
have access to environments which are not appropriate for their job
function there is the risk that users may inappropriately or accidentally
use the access leading to loss of application or data integrity.”
9.66 As previously opined, where process is not followed, this enhances the
likelihood for the possibility of undetected error.
9.67 Further, as noted in the Ernst & Young document? there were weak user
account management controls and the granting and monitoring of
user access was highlighted as an area of concern:
254 POL Management Letter FINAL.docx, Management letter for the year ended 27 March 2011, (Section 2.
Prior Year Comments-Update - Item 15) August 2011 [POL-0219218] [9d7862698d2a0f6af2a3c55590763bb7]
255 POL Management Letter FINAL.docx, Post Office Limited - Management letter for the year ended 27 March 2011,
[POL-0219218] [9d7862698d2a0f6af2a3c55590763bb7]
Prepared by: Jason Coyne
Occupation: Partne
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page cli of 225
“Unrestricted access to privileged IT functions increases the risk of
unauthorised/inappropriate access which may lead to the processing of
unauthorised or erroneous transactions.”
Opinion Summary
9.68 A number of technical documents provided by Fujitsu identify that the
Horizon estate was enabled to be managed remotely.
9.69 This is typically expected in an estate such as Horizon to enable Fujitsu
to provide support services without physically having to send an
engineer out to each and every branch at the notification of any issue.
Software roll-outs and updates to operating system components are
typically largely distributed this way.
9.70 Regarding Issue 7, the documents show that Fujitsu could and did
remotely access the transaction data recorded by Horizon. Several
technical tools exist with the specific purpose of allowing Fujitsu to
carry out modifications and corrective fixes to transaction data. In
addition, a number of external audit reports commissioned by Post
Office show that this access was often not done without the
appropriate control mechanisms in place.
9.71 In respect of Issue 10 documents show that a wide range of users at
Fujitsu did and do have the ability and facilities to access and modify
transaction data. Fujitsu staff were able to implement changes that
had the potential to affect transaction data both without the
knowledge or consent of the Subpostmaster and/or Post Office. In
addition, a number of external audits commissioned by Post Office
report that the appropriate control mechanisms to prevent mistakes
being made were not followed.
9.72 Regarding Issue 11, business process rules should apply in relation to
accessing and modifying transaction data. It is reported that a
documented audit log of each and every occasion of live data access
exists, however, this has not been made available by POL.
Prepared by: Jason Coyne
Occupation: Partne ° ey
Specialist Field: IT Systems d JFOU
On the Instructions of: Freeths LLP ° i
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page clii of 225
9.73 Issue 12 is subject to a Request for Information currently disputed by
Post Office as to the relevancy of the request. Whilst Post Office states
that there are in excess of 36,000 documents regarding how often the
access was granted (outside of actions not needing express
authorisation that could also be carried out to fix data), these have
not been made available for review.
9.74 Regarding Issue 13 it is understood that the implications of Fujitsu
carrying out corrective fixes to data within the Horizon system could
have the potential to affect the reliability of Branches’ accounting
positions.
Prepared by: Jason Coyne
Occupation: Partne ° ey
Specialist Field: IT Systems } 41 UU
On the Instructions of: Freeths LLP ~ i
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page cliii of 225
10. Expert Declaration
1 JASON COYNE DECLARE THAT:
10.1 I understand that my duty in providing written reports and giving
evidence is to help the Court, and that this duty overrides any
obligation to the party by whom I am engaged or the person who has
paid or is liable to pay me. I confirm that I have complied and will
continue to comply with my duty.
10.2 I confirm that I have not entered into any arrangement where the
amount or payment of my fees is in any way dependent on the
outcome of the case.
10.3 I know of no conflict of interest of any kind, other than any which I have
disclosed in my report.
10.4 I do not consider that any interest which I have disclosed affects my
suitability as an expert witness on any issues on which I have given
evidence.
10.5 I will advise the party by whom I am instructed if, between the date of
my report and the trial, there is any change in circumstances which
affect my answers to points 10.3 and 10.4 above.
10.6 I have shown the sources of all information I have used.
10.7 I have exercised reasonable care and skill in order to be accurate and
complete in preparing this report.
10.8 I have endeavoured to include in my report those matters, of which I
have knowledge or of which I have been made aware, that might
adversely affect the validity of my opinion. I have clearly stated any
qualifications to my opinion.
10.9 I have not, without forming an independent view, included or excluded
anything which has been suggested to me by others, including my
instructing lawyers.
Prepared by: Jason Coyne
Occupation: Partne
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page cliv of 225
10.101 will notify those instructing me immediately and confirm in writing if,
for any reason, my existing report requires any correction or
qualification.
10.111 understand that:
a. my report will form the evidence to be given under oath or
affirmation;
b. questions may be put to me in writing for the purposes of
clarifying my report and that my answers shall be treated as part
of my report and covered by my statement of truth;
c. the court may at any stage direct a discussion to take place
between experts for the purpose of identifying and discussing the
expert issues in the proceedings, where possible reaching an
agreed opinion on those issues and identifying what action, if
any, may be taken to resolve any of the outstanding issues
between the parties;
d. the court may direct that following a discussion between the
experts that a statement should be prepared showing those
issues which are agreed, and those issues which are not agreed,
together with a summary of the reasons for disagreeing;
e. I may be required to attend court to be cross-examined on my
report by a cross-examiner assisted by an expert;
f. I am likely to be the subject of public adverse criticism by the
judge if the Court concludes that I have not taken reasonable
care in trying to meet the standards set out above.
10.12 I have read Part 35 of the Civil Procedure Rules, the accompanying
practice direction and the Guidance for the instruction of experts in
civil claims and I have complied with their requirements.
10.131 am aware of the practice direction on pre-action conduct. I have acted in
accordance with the Code of Practice for Experts.
Prepared by: Jason Coyne
Occupation: Partne ° ey
Specialist Field: IT Systems g tQrou
On the Instructions of: Freeths LLP ~ i
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page clv of 225
Statement of Truth
10.14 I confirm that I have made clear which facts and matters referred to in
this report are within my own knowledge and which are not. Those
that are within my own knowledge I confirm to be true. The opinions I
have expressed represent my true and complete professional opinions
on the matters to which they refer.
Signed:
Jason Coyne
Partner
Dated: 16 October 2018
Prepared by: Jason Coyne
Occupation: Partne
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page clvi of 225
11. Appendix A
PEAKs of initial interest
11.1 Summaries and key excerpts from a select number of PEAKs which could
have a financial impact upon branches (these are also narrated within
the main body of the report).
Errors with Financial Impact
PC0027887 (1999)5°
11.2 Summary: FAD011523 - receipts and payments misbalance
11.3 Date raised in PEAK: 21 July 1999
11.4 Discrepancy value: £1,082,540.28
11.5 Days open: 407 days
Key Excerpts
Date: 21-Jul-1999 15:25:00 User: _Customer Call_
“for fad code: 011523, on week 9 receipts and payments misbalince of
£1337.05, week 10 misbalance of £24000, week 11 misbalance of £12000,
week 12 misbalance of £1051111.48 and week 13 misbalance of
£17426.05, she has a difference on week 11 of balance due to post office
and balance brought fwd on week 12 of £1082544.32 overall these weeks
net out a _ difference of £ 27343.84 . she needs _ business
support(reconciliation ) to look into this”
Date: 27-Jul-1999 10:09:00 User: Deleted User (Mike Croshaw
Sep/00)
“CAP12 Balance brought forward was multiplied twice due the known
software error. The inital balance brought forward for this CAP was
£1196622.72. This was multiplied twice to give a total BBF of £2279189.04.
The discrepancy was
256 PC0027887.html, PEAK PCO027887, 21 July 1999 [POL-0221773] [93af66e221ecfde6fcaad8a5aci4eca4]
Prepared by: Jason Coyne
Occupation: Partne
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page clvii of 225
therefore £1082540.28. This was due a known software error which has no
been resolved.”
Date: 03-Aug-1999 13:41:00 User: Barbara Longley
“The Call record has been assigned to MSU Team Member: Nicole Meredith”
Date: 17-Aug-1999 14:32:00 User: Nicole Meredith
“A RED report was issued for this incident and I am awaiting confirmation for}
this call to be closed.”
Date: 24-Aug-1999 14:15:00 User: Nicole Meredith
“Julie Dart (POCL TP) is unwilling to close this call. She does not believe
that the Receipts and Payments misbalance was caused by the B/F figure
doubling up. Please investigate why this misbalance occurred in CAPs
9,10,11,12 and 13 and provide evidence if possible.”
Date: 25-Aug-1999 09:51:00 User: Deleted User (Mike Croshaw
Sep/00)
“Evidence for these dates has now been archived. Will update call further}
when archived evidence has been retrieved”
Date: 02-Sep-1999 14:58:00 User: Deleted User (Mike Croshaw
Sep/00)
“Archived message store is now available - further investigation will be
carried out on Tuesday.”
Date: 08-Sep-1999 12:11:00 User: Deleted User (Mike Croshaw
Sep/00)
“The discrepancies for CAP 10 and CAP 11 are related to the transfe:
incident covered by two previous calls, E-9906020405 & E-9906140099."
Prepared by: Jason Coyne
Occupation: Partne
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page clviii of 225
Date: 15-Sep-1999 08:51:00 User: Deleted User (Mike Croshaw
Sep/00)
“Jan Holmes is currently working on extracting the archive data for CAP13
and CAP9, which are the two CAPS we have outstanding queries about.”
Date: 21-Sep-1999 08:35:00 User: Barbara Longley
“In the absence of Mike, can another team member please take this call - I
believe that there is a disc on his desk which is connected with this call. The
call summary has been changed from:-FAD011523 - receipts and payments
misbalance The call summary is now:-FAD011523 - receipts and payments
misbalance The Call record has been transferred to the Team: EDSC"
Date:01-Oct-1999 15:03:00 User: Rakesh Patel
“Furthur evidence from Jan has not arrived.Passing back to Mike for
continuation.”
Date: 04-Oct-1999 14:14:00 User: Deleted User (Mike Croshaw
Sep/00)
“New evidence added - Files sent by Jan Holmes to SSC...I have added the
files sent by Jan Holmes as evidence to this call. Further evidence will
probably be required from the archive server. Currently the CAP9 and
CAP13 discrepancies are still outstanding on this call. I have asked Les
Ong/Brian
Orzel to take a look at the call and if possible provide guidance.”
Date: 13-Oct-1999 10:53:00 User: Deleted User (Mike Croshaw
Sep/00)
“evidence deleted - CAP13 New evidence added - Dump of message store
taken on 24/08/99 - includes”
Prepared by: Jason Coyne
Occupation: Partne
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page clix of 225
Date: 13-Oct-1999 11:00:00 User: Deleted User (Mike Croshaw
Sep/00)
“I spoken to Les Ong re. this call, he has informed me that Steve Warwick is
investigating it.”
Date: 13-Oct-1999 16:49:00 User: Steve Warwick
“The evidence supplied in the Excel files covers 26th May to 30th May. CAP
9 dates were from 20th May to 26th May, therefore I am unable to
determine the CAP 9 cause of the £1337.05 imbalance from the evidence
presented.
I am also unable to analyse the CAP 12 and CAP 13 (10.6.99 to 23.6.99)
imbalances because the data required is not contained in either the
spreadsheets or the message store extracts provided. (The only data for
this period in the message store extract represents either persistent objects
which were still current at the time of the extract or messages which have
an expiry period > 35 days).
I can concur with Mike Crowshaw's explanation of the imbalances in CAPs
10 and 11. These were due to a stock transfer for £12,000 which was not
settled correctly due to the presence of a corrupt EPOSSSettlement.dil file
on the PC involved. The result of this was that the transfer out could not be
accepted by the receiving stock unit, leaving a Cash Account imbalance in
CAP 10 of twice the value. The effect of this error in CAP 10 was to leave
the overall balance due to POL £12,000 short causing the BBF for CAP 11 to
be short by this value and hence the £12,000 imbalance in CAP 11.”
Date: 12-Jan-2000 09:54:00 User: Lionel Higman
“CSR is no longer a valid target release. Moving target forward to earliest
valid value. Target Release updated to CSR-CI2_2R"
Prepared by: Jason Coyne
Occupation: Partne
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page clx of 225
Date: 05-Jul-2000 16:01:00 User: Deleted User (Anna Croft Sep/00)
“HSH are chasing an update on this call call still with Steve Warwick -QFP”
Date: 06-Jul-2000 11:03:00 User: Lionel Higman
“This PinICL has been assigned a CS categorisation of C (fix for first
maintenance release). Target Release set to M1 to reflect the categorisation.
Target Release updated to M1"
Date: 31-Aug-2000 10:46:00 User: Gerald Barnes
“I have now taken over analysing this problem from Steve Warwick who
requested additional information on 13/10/1999 17:49:21. I have looked at
the information and I am not convinced it can possibly be complete.
For example he said he wanted information from 20th May until 26th May. I
opened the spreadsheet mc20may.xls and observed it only had 6 counter
32 transactions in - is this correct?
In any case my team is not used to dealing with data in this form (
spreadsheets ). We have developed techniques to look at problems using a
full message store plus audit and event logs from the failing counter. Even if
all the relevant spreadsheets could be obtained I do not think it would be
worthwhile me getting to grips with this new method of analysing problems.
I see this is a very old problem (21/07/1999) and there have been many
software updates since then. May I suggest we discontinue investigation of
this particular problem but that if a similar problem occurs again you send
full message store plus audit and event logs from the failing counter.”
Date: 31-Aug-2000 13:17:00 User: Deleted User (Mike Croshaw
Sep/00)
“Closing call on basis of insufficient evidence. As this is such an old call I
have not contacted the call originator. I suggest that this call remains
closed!”
Prepared by: Jason Coyne
Occupation: Partne
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page clxi of 225
PC0063227 (2001)
11.6 Summary: APS and TIP: unequal harvesting.
11.7 Date raised in PEAK: 28 February 2001
11.8 Discrepancy value: £11,708.08
11.9 Time open: 9 Days
Key Excerpts
Date: 28-Feb-2001 10:34:00 User: _Customer Call_
“Could you raise this call for me. APS, PATHO39, Priority B The APS Daily Account
Balancing Report (2133) for processing date 27/02/01shows 401 transactions that
were harvested by APS but not by TPS. Please send to EDSC for investigation.”
Date: 28-Feb-2001 12:14:00 User: Garrett Simpson
“We have now found out, the hard way, that there is a bug in the new version of
Riposte installed for M1. The effect of this is to cause it to run very slowly. TIP
harvester runs first and has a finish time of 2030. After that the APS harvester
starts. The slow running meant that when the TIP harvester was stopped there
were still a lot of offices for it to harvest. By the time the APS harvester was
stopped more offices had succeeded in reporting their EOD so the APS harvester
had more offices to harvest than the TIP harvester. The correspondence servers
have now been regressed to the previous version of riposte, so this problem should
not recur.”
Date:01-Mar-2001 15:30:00 User: Angela Shaw
“The APS daily account balancing report (2133) processing date 28/2/01 shows
that the 401 txns for a value of £11708.08 have now been returned to TIP and
have been collected by the TIP harvester. Can you please advise what is being
done to fix the problem in the longer term? Is there a fix / work package? Which
team is progressing the fix? Thanks”
Date:01-Mar-2001 17:08:00 User: Garrett Simpson
257 PC0063227.html, Peak PC0063227, 28 February 2018, [POL-0237798]
[8c84cd6299903d0d3bbe0e05de44b924]
Prepared by: Jason Coyne
Occupation: Partne
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page clxii of 225
“The bug in Riposte has been identified. Escher have sent a fixed version which is
under test.”
Date:09-Mar-2001 10:21:00 User: Michael King
“These transactions have now been harvested and this is being fixed under
PC0063158. Please close call.”
Prepared by: Jason Coyne
Occupation: Partne S eA IT
Specialist Field: IT Systems } jt! OUP
On the Instructions of: Freeths LLP ~ -
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page clxiii of 225
PC0063723 (2001)**
11.10 Summary: FAD213422 - Spurious cash figure on trial balance
11.11 Date raised in PEAK: 10 March 2001
11.12 Discrepancy value: £428.29
11.13 Time open: 5 days
Key excerpts:
Date: 10-Mar-2001 09:01:00 User: _Customer Call_
“called by michelle at nbsc & asked to call pm regarding a discrepancy that
had been made good but resulted in a spurious cash figure on trial balance”
“pm had original balance £30358.35 & a shortage of £428.29pm declared
cash to £3078.86 his new balance was £428.290ver £428.29 under net
discrepancy zero, but his cash figure had gone up £856.58 to £31643.22
and a message saying continuing could result in an unbalanced stock. I took
the pm back into the cash declaration & it was still the same figure
£30786.64, I printed out the trial balance again this time it was over
£428.29 under £856.58 net discrepancy £- 428.29. cash figure £30358.35.
I went back into cash declaration again & declared cash was still at
£30786.64, I printed out another trial balance this time it was over£428.29
under £428.29 nett zero & the cash figure was ok”
“Info provided for update of KEL DRowe1625K as this PM seems to have got
himself a workaround which may work in the future.”
Date:15-Mar-2001 11:54:00 User: Sudip Sur
“I have updated KEL DRowe1625K as requested. Please note that this is a
known problem and development are working on this and a fix will be
released when available.
258 PC0063723.html, Peak PC0063723, 10 March 2001, [POL-0238257] [1efaafc05039eea7a5e5e09d1d50226c]
Prepared by: Jason Coyne
Occupation: Partne
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page clxiv of 225
PC0098844 (2004)?52
11.14 Summary: FAD011642 - Fault with bureau & decimal point
11.15 Date raised in PEAK: 6 February 2004
11.16 Discrepancy value: N/A
11.17 Days open: 193 days
Key Excerpts
Date: 06-Feb-2004 16:36:52 User: _Customer Call_
CALL PC0098844 opened
Date: 06-Feb-2004 16:36:58 User: _Customer Call_
“pm has problems regarding a fault with there bureau, something to do with
a decimal point
Contacted: paul buttler who help with the implimentation of the bureaude
change at the p.o advises that the problem which is arising at the p.o is due
to a decimal point missing on the horizon system on the ‘euro line' which is
causing a decrepency.
pm declares euros, but does not show on the balance snap shot. pm does
transaction log @ 14.34 on gateway which shows all the euros she has
declared but it does not show up as the sterling equivellent. eg. DDP euro
5786500 = sterling 0.00 NB. this problem is only showing for euro's. All
other currency's are showing ok in the system
When the pm is now trying to reverse the transaction the figure is doubling.
Can ssc please investigate as a matter of urgency why the bureau de change
is not updating on the system for euro's currency on the system.”
259 PC0098844.html, Peak PC0098844, 6 February 2004 [POL-0270879] [050c1d940ddd970bf7ed304afc494faf]
Prepared by: Jason Coyne
Occupation: Partne S — .
Specialist Field: IT Systems jt OU
On the Instructions of: Freeths LLP ri
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page clxv of 225
Date: 06-Feb-2004 16:41:54 User: Barbara Longley
“Prescan: Assigning call to Martin Harvey in EDSC who has been working on
call already. Raising call to 'A' priority to reflect urgency.”
Date: 06-Feb-2004 18:18:56 User: Martin Harvey
“The event log shows several error messages indicating that the DBD
discrepency value is too large. It is believed that the problem is with the
balance snapshot only The PM has been advised to not roll over but to
continue transacting normally. Messagestore and event logs to follow.”
Date: 09-Feb-2004 15:41:01 User: Martin Harvey
“Just to clarify, in addition to a software fix for this problem we also need
dev to advise on the remedial action necessary to unpick the actions taken
by the
PM and to enable the office to be balanced on the 11th Feb”
Date: 10-Feb-2004 09:01:39 User: Martin Harvey
“The PM has done a trial balance on Stock Unit FC as requested. This shows
a value of £50530.57 as the sterling equivilent of Euro's, which is about
right apparently. Also there is Receipts/Payment mismatch of £79321.46,
which is double the revaluation amount of £39660.73. Response code to call
type L as
Category 4"
Prepared by: Jason Coyne
Occupation: Partne ° ey
Specialist Field: IT Systems g tQrou
On the Instructions of: Freeths LLP ~ i
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page clxvi of 225
Date: 10-Feb-2004 11:46:55 User: Martin McConnell
“The receipts payments mismatch is a duplicate of 98480 currently with
RMF. In other words users should NOT be allowed to reverse system
generated revaluations thus. The initial problem with the large amounts I
believe the system has coped OK and has alerted a red event to say so.
Perhaps the only thing the system should hve done based on the current
spot rate as time of declaration would be to refuse to allow such a number
to be entered because it would blow a system limit. This is potentially
flawed should a new spot rate
arrive after a declaration but it is probably a more user friendly thing to do.
Routing to management for their onward consideration.”
Date: 10-Feb-2004 13:13:56 User: Mark Wright
“I've applied the fix that Martin provided, and that has cleared the receipts
and payments mismatch, however the PM is still reporting a shortage
of£39660. 73 Martin is now looking into this and will advise.”
Date: 10-Feb-2004 16:42:19 User: Martin McConnell
“The clerks/PM@@s should be alerted to the fact that they should NOT
under any circumstances be allowed to reverse (ER) a system generated
revaluation with regard to BDC. Hopefully we@@ll be able to punt out a fix
for PCO098480 which has led us down this path in S60(?) which will prevent
the likes of this problem spiralling out of control.”
Prepared by: Jason Coyne
Occupation: Partne
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page clxvii of 225
Date: 12-Feb-2004 11:46:13 User: Matt Arris
“In summary This PinICL has exposed two potential problems. 1) Clerks can
reverse an auto-generated Bureau revaluation. This is a duplicate of 98480.
2) Clerks declare foreign currency daily.
The counter software checks to see if the figure declared matches the
system generated figure. If the declared figure does not match the system
figure the outlet has ?lost? some currency and there will be a discrepancy.
If this discrepancy is greater than 1 million pounds an error is produced
whereby the sterling equivalent is set to zero, but the declared foreign
currency figure is retained. When entering the daily foreign currency
declaration there may be the odd small discrepancy, but there will not be
one of 1 million pounds worth unless there was a robbery ? but a million
pounds of a particular foreign currency in stock is too high anyway.
The clerk produced this problem because two many zeros were added to the
generated figure would have been zero for every currency. Under normal
operation a discrepancy would make the clerk double check the entered
figure
Prepared by: Jason Coyne
Occupation: Partne
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page clxviii of 225
or explain the difference in stock. To prevent this second problem from
arising by changing code we would have to check each declared foreign
currency when entered to decide if the sterling equivalent of any
discrepancy would exceed 1 million pounds. This would require a sizeable
rework with associated changes to dialogues, etc.
Development would rather not have to make this change. I think this
second problem can be covered by a KEL with no code change made. Small
discrepancies will occur and will need checking and explaining. Post
migration, large discrepancies should not happen. Discrepancies of more
than a million pounds worth should never happen.
If we stop the ability of reversing an auto generated Bureau revaluation
(98480) no further damage can be done by making incorrect declarations.
By just making a correct declaration and performing a trial balance (with no
commit) the error will be resolved. It may be worth passing this PinICL to
John Pope in Requirements for him to find out if the customer is willing to
pay for a code fix for item 2) or whether PO is prepared to accept that item
2) is unlikely to happen and there is an easy way to put things right if it
does happen (re-declare and perform a trial balance). declaration. £30,000
worth of Euros was turned into £3,000,000.
Bureau sites were being migrated from the existing MoneyChanger system
to Horizon. The instructions were to declare all MoneyChanger stock on the
new Horizon Bureau system. Effectively all transactions were discrepancies
as the system.”
Date: 13-Feb-2004 10:07:56 User: Martin Harvey
“OCR MWright420K was raised to cover the correctice actions. Kels
AChambers5533Q and GMaxwell3841K cover both problems. As
recommended please pass call to John Pope in Requirements for him to find
out if the customer is willing to pay for a code fix”
Prepared by: Jason Coyne
Occupation: Partne
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page clxix of 225
Date: 04-May-2004 10:34:39 User: _Customer Call_
“Caller states that the rates on the rate board are not being updated.
04/05/04 10:33 uk951563 HSH2 Information: PM is sure she has not
logged, or is not aware there is a call logged for this fault. This call is a
PATH call, therefore the PM may not be aware. Also, there has been no
update since March 23rd. 04/05/04 10:34 uk951563 HSH2 Information:
Raising new call for FAD FAD, and to reflect callers new complaint.”
Date: 21-Jun-2004 15:28:39 User: Lionel Higman
“The call Target Release has been changed from:- BI_3S50R-Provisional
The call Target Release is now:- BI_3S70R-Provisional S5OR is no longer a
possible target release. S60R is already fully subscribed. Targeting at next
available potential R release - S70R.”
Date: 14-Jul-2004 16:20:08 User: Lionel Higman
“The call TargetRelease has been changed from:- BI_3S70R-Provisional The
call TargetRelease is now:-
BI_3S75R-Provisional I am told the currently targeted release for this call
(70R) exists only in name - retargeting at S75R-Provisional.”
Date: 29-Jul-2004 17:09:23 User: David Cooke
“In the absence of John Pope I have reviewed this entry and agree with the
assertion that this can be handled via KEL with no code change. The
circumstances were unusual and should not occur in normal live operation.
Unless the help desk advises we have had a large number of calls relating
to the KELs I suggest we close this call.”
Prepared by: Jason Coyne
Occupation: Partne
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page clxx of 225
Date: 17-Aug-2004 09:14:41 User: Barbara Longley
“Prescan: As Martin Harvey is out of the office this week, I am reassigning
call to Anne Chambers who is familiar with this problem.”
Date: 17-Aug-2004 10:37:23 User: Anne Chambers
“Call had been passed to John Pope to see if there was a POL requirement
for change here - after some months the following response was received
from David Cooke: In the absence of John Pope I have reviewed this entry
land agree with the assertion that this can be handled via KEL with no code
ichange. The circumstances were unusual and should not occur in norma
ive operation. Unless the help desk advises we have had a large number of
calls relating to the KELs I suggest we close this call. However in the
interim, this problem has been seen on a couple of other occasions and hasI
caused significant accounting problems. PC0103606 is with development,
fix scheduled for S80. This call can be closed (please do not contact origina
loutlet).”
Prepared by: Jason Coyne
Occupation: Partne ° ey
Specialist Field: IT Systems d JFOU
On the Instructions of: Freeths LLP ~ i
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page clxxi of 225
PCO203131 (2010)?
11.18 Summary: FAD203306 office snapshot is showing incorrect volume
11.19 Date raised in PEAK: 18 August 2010
11.20 Discrepancy value: N/A
11.21 Days open: 24 Days
Key Excerpts
Date: 18-Aug-2010 11:01:23 User: _Customer Call_
“pm states office snapshot is showing incorrect volume”
“pm states volume for lottery was £284 but value on the office snapshot is
£604 Rem in session ID: 1-362075 - items in question are the first two from
this session ID”
“The volume of £1 lottery tickets was set at £284 but the value on the
snapshot under receipts table is £604 Smiley Desktop could not find any
particular issues No relevant SSC KELs could be found Counter can be
pinged.”
“PEAK, can you investigate discrepancy between volume and value in office
snapshot for £1 Lottery tickets?”
Date: 21-Aug-2010 09:38:49 User: Sudip Sur
“I have looked at the tally roll printout of the Office snapshot that was
carried out on 13/8/10 @15:35. The print out showing the volume of;
Instants £1 =
26° PCO203131.html, Peak PCO203131, 18 August 2010 [POL-0372925]
[6ea70fc9c6b34cbcd61b2a7b2ddb2628)
Prepared by: Jason Coyne
Occupation: Partne
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page clxxii of 225
284 and the Value = 604.00. Clearly this is wrong and the system is failing
to calculate correctly.
I have done a query on the database for session data for product:56 during
1st Aug and 20th Aug and found a txn that was done on 4th Aug @7:48:57
(Mode:SW) for quantity = -320. This may have contributed to the incorrect
value of 604.00.”
Date: 23-Aug-2010 08:56:07 User: Gareth Jenkins
“I have tried this out on Horizon and I accept that on Horizon the Volume
relating to a TC is NOT included in the Receipts and Payments.
I believe that this is a bug on Horizon and that it should be included. We
had a similar issue with Discrepancies on the Pre and Post Migration report
and this was accepted as being due to a bug on Horizon rather than on HNG-
x.
I believe that this PEAK is another symptom of the same issue. I think that
this should be closed as "Advice after investigation" (since "No fault" fives
SSC a Black Mark!).”
Date: 11-Sep-2010 09:43:54 User: Sudip Sur
“Development have investigated the problem and believe that the feature
has been carried forward from old Horizon and not a new problem in HNGI
PC0203676 (2010)?
11.22 Summary: Branch 054106 (HNGx) - NB102 Section 5 LINK - State 4
11.23 Date raised in PEAK: 31 August 2010
11.24 Discrepancy value: £71
11.25 Days open: 1 Day
261 PCO203676.html, PEAK PCO203676, 31 August 2010 [POL-0373467]
[9c65a8e1e33e636bc6ae37 2aefed3690]
Prepared by: Jason Coyne
Occupation: Partne
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page clxxiii of 225
Key Excerpts
Date: 31-Aug-2010 10:33:40 User: Jay Crofton
“Branch 054106 (HNGx) - NB102 Section 5 LINK - State 4 Call Type: L CallI
Priority: A”
“NB102 Section 5 report for Client: LINK produced on 30/08/2010. Branch
shows one new exception with Txn Id: 00-54106-2-3633070-1 Receipt date
27/08/2010, Amount: -£71.00
This branch also appears on the Failed Recovery Report ? why? KEL
acha959T or KEL dsed2640M may be relevant. Relevant reports attached.
Sending to SSC for investigation.
**PLEASE include further txn attempts with the same PAN immediately
after this txn if applicable as this is useful for Post Office Ltd for settlement
issues**”
Date: 31-Aug-2010 15:51:58 User: Clive Turrell
“The customer requested a withdrawal which was authorised by the FI and
a receipt was successfully printed at the counter so it is very likely that cash
was handed over. However the T1 recovery request timed out at the
counter and was abandoned. No subsequent transactions were attempted
on the same PAN. In these circumstances the following applies from KEL
acha959T
"If the receipts were printed successfully, and no subsequent transaction
was done for the same PAN at the branch (check on TESQA) it is likely that
the transaction was completed. Customer's account will be correct but the
branch will have a shortage (for a withdrawal) or surplus (for a deposit)
because the session hasn't been recorded."”
Date: 31-Aug-2010 16:08:24 User: Clive Turrell
“The Call record has been transferred to the team: MSU-Indt Mgt”
Prepared by: Jason Coyne
Occupation: Partne
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page clxxiv of 225
Date: 31-Aug-2010 16:34:36 User: Andrew Nash
“[Start of Response] Closing call. [End of Response] Response code to call
type L as Category 67 -- Final -- Solicited Known Error Routing to Call
Logger following Final Progress update. Defect cause updated to 42 -- Gen -
Outside
Program Control”
PCO263451 (201778)
11.26 Summary: Branch 66013 - Failed Recovery Report - 19/10/2017
11.27 Date raised in PEAK: 19 October 2017
11.28 Discrepancy value: £20
11.29 Days open: 1
Key Excerpts
Date: 19-Oct-2017 08:25:20 User: Dharmesh Mistry
“Branch shows one failed recovery transaction with Txn Id: 00-66013'
15621320-1 for Receipt Date: 16/10/2017.
This exception has not appeared on any of the reports and following a
search on the DRS did not bring back any results. Raising a call to
investigate why this has appeared as an exception, the root cause and the
potential business impact for POL. Relevant reports attached. KEL
cardc464Q and KEL acha2511S (seng2048K for DCS) may be relevant. For
charging purposes, please could SSC explain whether or not this issue is
Fujitsu related (at fault) i.e. hardware or not? Relevant reports attached.”
262 PCO263451.html, Peak PCO263451, 19 October 2017, [POL-0430967]
[981b785aa6f78e75a5f19c8a2c70aff5)
Prepared by: Jason Coyne
Occupation: Partne
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page clxxv of 225
Date:19-Oct-2017 09:39:42 User: Sunil Nellikkentavita
“The £20 cash withdrawal transaction was authorised by the FI and an
AUTHORISED receipt was produced on the counter. However, when the user
attempted to settle the transaction it failed due to the comms issue at the
time so disconnected session receipts were produced and the user was
logged off. The user managed to log back in but recovery also failed
because of same comms issue. As an AUTHORISED receipt was produced
the user should have handed money over to the customer but we cannot be
certain that they actually did so. Assuming money was handed over, the
customer account will be correct but the branch will
have a shortage given that the transaction hasn't been recorded on the
system. This will need to be manually reconciled”
Date: 19-Oct-2017 09:51:25 User: Sunil Nellikkentavita
“Failed recovery session cleared from BRDB under MSC_ Task
043T0095737.MSC updated with following details: PCO263451 - Failed
recovery tx no:00-66013-1-5621320-1 cleared from BRDB. Actioned by
Sunil and witnessed by Dave Seddon.”
Prepared by: Jason Coyne
Occupation: Partne
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page clxxvi of 225
PC0266575 (2018)?°3
11.30 Summary: Branch 116940 - NB102 Section 5 A&L ? State 4
11.31 Date raised in PEAK: 26 January 2018
11.32 Discrepancy value: £125.00
11.33 Days open: 6 days
Key excerpts:
Date: 26-Jan-2018 08:52:35 User: Dharmesh Mistry
“NB102 Section 5 report for Client: A&L produced on 26/01/2018. Branch
shows one exception with Txn Id: 00-116940-1-5311-1 Receipt Date:
24/01/2018, Amount: £125.00 KEL acha959T may be relevant. Relevant
report attached. Sending to SSC for investigation. For charging purposes,
please could SSC explain whether or not this issue is Fujitsu related (at
fault) i.e. hardware or not?”
Date:31-Jan-2018 11:11:47 User: Venkata Subbarao Konakalla
“The cash withdrawal transaction was authorised by the FI and an
AUTHORISED receipt was produced on the counter. money should have
changed hands. However, when the user attempted to settle the transaction
it failed due to comms issues. The user logged back today after one week
and the recovery also failed. The customer's account will be correct but the
branch will have a shortage because the session hasn't been recorded. This
requires manual reconciliation. Please raise an MSC to clear the failed
recovery transaction.”
Date:09-Feb-2018 09:45:06 User: Dharmesh Mistry
“Closing peak as MSC completed and Failed recovery cleared”
262 PCO266575.html, PEAK PCO266575, 26 January 2018, [POL-0433904]
[8d02a629313fa13f86f853d49e55dcc7)
Prepared by: Jason Coyne
Occupation: Partne
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page clxxvii of 225
PC0273046 (2018)?8*
11.34 Summary: Branch 051106 - NB102 Section 5 LINK - State 4 and Failed
Recovery
11.35 Date raised in PEAK: 15 August 2018
11.36 Discrepancy value: £25
11.37 Days open: 1
Key excerpts:
Date:15-Aug-2018 08:15:59 User: Andy Dunks
“PEAK raised for the investigation of transaction(s) in a state other than
Final as showing in daily Reconciliation reports. To comply with SLA, PEAK
will be open for a between 8 hours and 5 days maximum whilst transaction
issue is investigated, reported and mitigating actions completed and closed
down.”
“This branch also appears on the Failed Recovery Report. Could SSC please
investigate why? KEL acha959T (surs136M for DCS) may be relevant.”
Date: 15-Aug-2018 11:30:17 User: RCAClient Live
PEAK [ PC0273046 ] Branch ID [ 051106 ] Node ID [ 02 ] SSN [ Iprpssn003
] User [ snell02 ] Attempting command execution: get
/cygdrive/c/ProgramData/Fujitsu/CounterBusinessApplication/log/PostOffice
Counter.log.2018-08-13.zip
evidence/051106/02_PostOfficeCounter.log.2018-08-13.zip
Date:15-Aug-2018 11:30:36 User: RCAClient Live
PEAK [ PC0273046 ] Branch ID [ 051106 ] Node ID [ 02 ] SSN [ Iprpssn003
] User [ snell02 ] Command execution completed successfully: get
/cygdrive/c/ProgramData/Fujitsu/CounterBusinessApplication/log/PostOffice
Counter.log.2018-08-13.zip
evidence/051106/02_PostOfficeCounter.log.2018-08-13.zip
264 PCO273046.html, PEAK PCO273046, 15 August 2018, [POL-0439981]
[c4330f4fcc4ea9f37368e4c692730828]
Prepared by: Jason Coyne
Occupation: Partne
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page clxxviii of 225
Date:15-Aug-2018 11:45:35 User: Sunil Nellikkentavita
“During the 25 RBS Grp Cash Withdrawal, the banking transaction had fully
completed, including the receipt print but disconnected session at
settlement (Comms issue).
Recovery also failed because of same comms issue.
As per KEL, transactionState=1 the banking transaction had fully
completed, including the receipt print, and money should have changed
hands. If recovery had succeeded, it would have automatically completed
the session. So you can assume that the transaction was completed
successfully. The customer's account will be correct but the branch will have
a shortage for 25 withdrawal because the session hasn't been recorded.”
Date: 15-Aug-2018 11:49:50 User: Sunil Nellikkentavita
“Cleared failed recovery transaction from BRDB under MSC Task
04370098231. Actioned by me and witnessed by Venkat.”
Prepared by: Jason Coyne
Occupation: Partne S eA IT
Specialist Field: IT Systems } jt! OUP
On the Instructions of: Freeths LLP ~ -
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page clxxix of 225
12. Appendix B
Horizon Architecture Diagrams
Figural Horizon Data Flows Overview
Other
Systems
Oracle Write
/ ait
H Extract
Audit
File
Data .
Extract
Audit I Store
Audit System
Audit I Retrieval
Audit
Extract
ate Journal _ SAEREY
store
Prepared by: Jason Coyne
Occupation: Partne :
Specialist Field: IT Systems ¢
On the Instructions of: Freeths LLP
180503R1935
16 October 2018
FUJ00082162
FUJ00082162
Page clxxx of 225
Figure 2 Data Centre Applications Overview?®>
Data Centre
‘eplesion Node #*
I
ae)
ea
es
Communicates
pplication Servers
‘Conwmanieation
Severs
Le ile I=]
[ree]
=
ve I [ae
Lf DebitGart I [OVA Ree
=} Agents Sewoe
‘pplication Node #2
(BM = New Date Centre Apptcatons
[FE = tngocy Dan Cone Aptenions
=
ae
Bach
Date
Warenouse
‘Banking Applicaton Severs
Banking Applicaton Servers
Ue]
‘Aut Server
285 RMARCO02_0.1.doc, Horizon Next Generation - Plan X (HNG-X), 21 September 2005 [POL-0084540]
[754315a4037a6ea4clec7ee070b7d170)
Prepared by: Jason Coyne
Occupation: Partne
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
fitgroup
180503R1935
16 October 2018
FUJ00082162
FUJ00082162
Page clxxxi of 225
Figure 3 Horizon / Horizon Online migration overview?
Horizon and HNG-X System Overview
= I C=]
Management.
Aust Key Management MTAS ocms ‘oMDB
Reference data Bulk agents
RDMC RODS DCS Bulk Agent ETU Bulk Agent
Transaction processing
POLFS LS DWH
TPS DRS. TES APS
‘On-line agents and services
Network Sanking APOP Scag eas PAF DVLA Agent
‘Counter ao
Figure 4 Horizon Online Message Flows overview?®”
256 ARCAPPARC0002_0.2.doc, HNG-X Integration Architecture, 08 November 2006 [POL-0087918]
[daecOde8a5eee25b5c9d06730c338dd0]
267 Witness Statement of Gareth Jenkins.pdf, Witness Statement of Gareth Jenkins, 05 October 2012 [C0003632]
[b544230cf07249c189cf664fcba6d899}
Prepared by: Jason Coyne
Occupation: Partne
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
gitgroup
180503R1935
FUJ00082162
FUJ00082162
16 October 2018 Page clxxxii of 225
Oracle
BAL IMessage
~ Write
Other
Systems
DataI Copy
EDD
- Audit . Audit
[Write File
Audit I Store
Audit I Retrieval
Audit
Extract
Prepared by: Jason Coyne
Occupation: Partne
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page clxxxiii of 225
Figure 5 Horizon Business Applications overview 20107°°
i iy sel
Reconetiation ene I : sata :
Pose Sees q ~ diotyortouii)
So
Externat
Online Services
Message Berver: > }¢—————
Beanch Statt Customer
“ Post Otice User el serices
26° POL-0003093.pdf, Horizon Architecture Diagrams, 8 March 2010 [POL-0003093]
[00195d1cbb0017bd34e7c68b7930c1cf]
Prepared by: Jason Coyne
Occupation: Partne .
Specialist Fiald: IT Systems itgroup
On the Instructions of: Freeths LLP
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page clxxxiv of 225
Figure 6 Reconciliation in Horizon Online?°
Financial institutions
Santander I Merchant
&CAPO I ES I EPay Acquirer
EM
aa
nc pes }—pe) Pe I
— —__ ae ee
oes
ct a
1128 rj TPS
Branch TPS Tens I
Database i
APS Tins Ss
I "
ae
Reconciliation In HNG-X
=I
Horizon I
Counter I
I
26° SYMSDMPROO012_3.doc, Reconciliation and Incident Management Joint Working Document, 30 April 2012 [POL-
0125134] [ad897ac9ffSedb2de37bfb8c4e9dc362)
Prepared by: Jason Coyne
Occupation: Partne “ya
Specialist Field: IT Systems 3 Itgroup
On the Instructions of: Freeths LLP
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page clxxxv of 225
Figure 7 Banking Interfaces 2003?”
270 CSOLA045_1.1.doc, Operational Level Change and Release Management Interface agreement Between Post
Office Ltd and Fujitsu Services, 24 April 2003 [POL-0069985] [334ef7724ebe8012129c96f62b41ad15]
Prepared by: Jason Coyne
Occupation: Partne .
Specialist Field: IT Systems itg FOup
On the Instructions of: Freeths LLP ‘
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page clxxxvi of 225
Figure 8 External Clients / Systems
FUJITSU Horizon Service Context Diagram
Banking —
Reference Data
Estate Management
Le
Horizon Business
Server
UN lla, Ca
Settlement/Logistics/APOP
Prepared by: Jason Coyne
Occupation: Partne
Specialist Field: IT Systems
Zitgroup
On the Instructions of: Freeths LLP
13. c
2012 - 2013 TC’s
=
23)
84)
31
13)
FUJ00082162
FUJ00082162
Row Labels Count ofType CountofReference Sumof£ Value
(blank)
Unpaid Cheques 538 238 249515.5
‘Suspense 4671 4667 -371763.46
Stock-Non Rem 235 234 5824.93
Santander - Online Banking 1458 1452 -11297609.87
Santander - Manual Withdrawal 602 602 191847,23
‘Santander - Manual Deposit 5905 5905 -896625.78
Santander -Green Giro 887 886 12091.87
‘Santander - Co-Op Business Encashments 1321 1321 42285.91
Pre-Order 127 126 735,28
Postal Orders 1046 1046 15639.96
Personal Banking 327 323 19938
Paystation 53 52 2778
‘Other (Branch) 88 86 7878.21
‘Online Banking 705 704 = -1871845.18
NS&! 826 826 -340783.35
Government Services 1044 1039 “121795.24
First Rate 124 122 46352.47
DVLA 3744 3743 -223352.65
Drop & Go 514 5i4 27721.21
DebitCards 140 139 771759.83
‘Cheques To IPSL 4722 4675 1585269.95
Cash Rems From Branch 25649 25649 6120594.63
Camelot 22567 22567 448789,.34
Bureau 3939 3939 432165.75
Automated Payments 2040 2040 «-3351011.34
[arm 945 945 947230
Grand Total 84217 83840 6-8445418.04
14, D
Log of helpdesk calls regarding Transaction Corrections W/E 15 June
2014
WE 15/06/2014
‘Activity —______STISub Activity T= I Count of Sub Category ‘Count of ResoWving Group I
= Transaction ComectionsI Dispute I 23
I General Information I 84
I Missing Evidence i 3
Possible Solutions i 13
I Printing Corrections i} 7
Resolving Branch Discrepancies I 7
I ____I Transaction Correction Dispute _I _8
Transaction Corrections Total 145
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page clxxxviii of 225
Prepared by: Jason Coyne
Occupation: Partne
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page clxxxix of 225
Appen
15. E
Horizon Reports
TPS Reports
15.1 TPS Reports were set out in the “TPS Report Set” which was designed to
enable the reconciliation of transactions that were carried out in
branches using the branch infrastructure. It was sent to the POLSAP
system and the POLMISs (Credence), and was made up of the
following reports:
15.2 TPSC250: Host Detected Transaction Control Errors - Daily report which
shows detail for any Post Office branch where the control totals for the
transactions output by the Host to POLSAP and Credence do not
match the Daily Transaction Totals calculated by the counters.
15.3 TPSC254: Harvester Exceptions - Daily Report which shows a list of
exceptions detected by the BRDB copy process when failing to process
one or more messages.
15.4 TPSC257: POLSAP Incomplete Summaries Report - Daily report which
identifies all Post Office branches in which the net total transactions
(debits/credits) does not equal 0.
15.5 The TPS Report Set contained information which would allow Post Office
to identify the errors or inconsistencies set out above, which could
include shortfalls. However, the investigation of the cause and
resolution of TPS exceptions and errors was not dealt with within the
TPS Report Set. Resolution was dealt with through “Business
Incidents” and the underlying cause was investigated via “System
Incidents”, both of which are part of the Business Incident
Management (BIM) procedure. This is considered in more detail below.
15.6 There are 2 additional points to note in relation to the TPS Report Set
Prepared by: Jason Coyne
Occupation: Partne ° ey
Specialist Field: IT Systems g tQrou
On the Instructions of: Freeths LLP ~ i
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page Cxc of 225
Appen
15.7 There was no formal reconciliation process between the POLSAP system
and the Credence transaction stream, so Credence could not be used
to
Prepared by: Jason Coyne
Occupation: Partne . ae
Specialist Field: IT Systems 6 IY!
On the Instructions of: Freeths LLP Gey
15.8
FUJ00082162
FUJ00082162
verify financial integrity. POLSAP system transaction information
should have been used for this purpose.
The TPS Report set was not intended to be shared with Post Office, but
it was available upon request.
APS Reports
15.9 APS reports were set out in the APS Report Set which was designed to
ensure APS Transactions completed in Branches are reconciled with
the Transaction stream received by the POLSAP System to enable
settlement to be made with Clients. The APS Report set was made up
of the following reports
15.10 APSS2133: APS Daily Account Balancing Report - The objective of this
report is to confirm that the APS transaction account balances for the
processing day.
15.11 APSS2133b: APS Client Summary Report - Objective is to provide a
summary of the transactions which have been delivered by APS during
the processing day. The summary is produced by the clearing agent
(i.e. the organisation to which Fujitsu Services deliver the
transactions). For each clearing agent, a breakdown is provided by
Client account for each transaction date. Transactions delivered to
Manual are processed manually and consequently are not reported
here.
15.12 APSS2133c: APS Delayed Transactions Report - Objective is to provide
details of all transactions which have not been delivered by Fujitsu
Services because they have been delayed/quarantined within the APS
Host system. Initial Customer Support resolution will cause the
transactions to be returned for normal processing or sent to manual
for manual processing. TPS quarantined APS transactions will show up
as a discrepancy on the APS Daily Account Balancing Report.
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page cxcii of 225
15.13 APSS2136: Daily TPS / APS Transaction Reconciliation Summary Report -
Normally, APS transactions flow through the TPS Host to POLSAP and
through the APS Host system within the same working day. However,
the rules associated with the processing of APS transactions within the
TPS Host and the APS Host systems are different. Consequently,
transactions may be placed in exception status by APS but be
accepted as valid by TPS. This reconciliation point is at the end of the
processing day. It reconciles the APS transactions at Branches and
APS transactions delivered by TPS to POLSAP. It maintains a record of
which transactions have passed through the TPS side and which
transactions have passed through the APS side and on a daily basis it
reports transactions which have been processed by one side and not
the other.
15.14 APSS2136b: Daily TPS / APS Transaction Reconciliation Client Account
Exception Report - Objective is to identify Client account exceptions
when comparing the actual number and value of transactions that
were processed by TPS (TIP) and APS (Client) for the last 30 days.
15.15 APSS2136c: Daily TPS / APS Transaction Reconciliation Detail Exception
Report - The objective is to report all transactions for the last 30 days
which are different in POLSAP to Client delivery. Differences may occur
either as a result of some error condition, or in the case of unmatched
TPS/POLSAP transactions, as a result of business processing rules
being different from those of APS/Client transactions.
15.16 APSS2139: Daily APS Office Harvesting Report - The APS Harvester
Reconciliation was used to ensure all Horizon Branches had been
harvested. As Horizon Online does not rely on harvesting of
transaction this report is no longer relevant.
15.17 APSS2140: APS Harvester Transaction Totals Summary - Objective is to
provide Branch totals and harvested totals by transaction date for
each of the last 30 transaction days for the APS Harvester.
Prepared by: Jason Coyne
Occupation: Partne ° ey
Specialist Field: IT Systems g tQrou
On the Instructions of: Freeths LLP ~ i
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page cxciii of 225
15.18 APSS2140b: APS Harvester Transaction by Office - Objective is to
provide details by Branch code of any discrepancies that exists in the
overall totals shown in the APS Harvester Transaction Totals
Summary.
15.19 APS2134: APS Validation Status Report - Objective is to identify the
success or failure of the APS Validation process. This compares the
volume of normal transactions and the value of all transactions
between each Client Transmission file and the corresponding fields in
the CTS sub-file.
15.20 Rejected Sub-Files Report - The Times Rejected column will indicate
how many times that the sub-file has been rejected and will indicate
whether there is an ongoing problem with poor quality data in
corrected files. If the quality of the data in the external transaction file
is good then we would not expect any output from the report.
15.21 Similar to the TPS Report Set, the APS Report Set would allow Post
Office to identify the errors or inconsistencies set out above, which
could include shortfalls. However, the investigation of the cause and
resolution of APS exceptions and errors was not dealt with within the
APS Report Set. Resolution was dealt with through the BIM procedure
which is set out in more detail below.
Banking & Related Services Reconciliation (DRS Report Set)
15.22 The Banking & Related Services Report Set was designed to enable
Santander, CAPO, LINK, Global Payments Inc., and Epay transactions
completed in the Branches to be reconciled in order to allow
settlement to be made with Clients, or direct settlement to specific
Clients and/or Banks. It was made up of the following reports:
15.23 NBOOO: DRS Summary - This report summarises all reconciliation
reports produced by the DRS. It also summarises all reports that were
not produced by the DRS because there was no data to report.
Prepared by: Jason Coyne
Occupation: Partne ° ey
Specialist Field: IT Systems d JFOU
On the Instructions of: Freeths LLP ~ i
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page cxciv of 225
15.24 NB101: Network Banking Settlement Statement - This report identifies
‘C4’ transactions received against each ‘C4 Settlement Date’ as
reported to the DRS for the most recent processing date. The report
will be used by Post Office Ltd. as a basis for settlement of Network
Banking transactions with the Financial Institutions (FIs).
15.25 NB102: Exception Summary - This report identifies all incomplete or
exception states, and is divided into 12 sections:
a. Section 1: All Uncleared Confirmed, Unconfirmed & POLSAP
exceptions
b. Section 2: Uncleared Exceptioned Client Transactions
c. Section 3: Uncleared Corruptions
d. Section 4: Uncleared Timing Differences
e. Section 5: Uncleared Confirmed, Unconfirmed & POLSAP
exceptions >24 hours
f. I Section 6: Uncleared Future Dated Transactions by Client
g. Section 7: All Cleared Confirmed, Unconfirmed & POLSAP
exceptions
h. Section 8: Cleared Exceptioned Client Transactions
i. Section 9: Cleared Corruptions
j. Section 10: Cleared Timing Differences
k. Section 11: Cleared Confirmed, Unconfirmed & POLSAP
exceptions
>24 hours
Section 12: Cleared Future Dated Transactions by Client.
Prepared by: Jason Coyne
Occupation: Partne ° ey
Specialist Field: IT Systems g tQrou
On the Instructions of: Freeths LLP ~ i
180503R1935 16 October 2018
FUJ00082162
FUJ00082162
Page cxcv of 225
Reconciliation and Incident Management
15.26 The BIM system was designed to report progress on the resolution of
Business Incidents to allow Post Office to complete reconciliation or
settlement with its internal systems, clients and banks.
15.27 A “Business Incident” is defined as the symptom of an underlying cause
- e.g. the effect of the system fault on the resulting reconciliation or
settlement information sent to Post Office. A Business Incident can
relate to any of the exceptions from the various reports above or one
or more of the reconciliation or settlement Errors discovered by Post
Office.
15.28 Post Office have reported the following numbers of BIs raised each
year:
01/07/2017 - 30/06/2018:
01/07/2016 - 30/06/2017:
01/07/2015 - 30/06/2016:
01/07/2014 - 30/06/2015:
01/07/2013 - 30/06/2014:
01/07/2012 - 30/06/2013:
01/07/2011 - 30/06/2012:
01/07/2010 - 30/06/2011:
01/07/2009 - 30/06/2010:
01/01/2009 - 30/06/2009:
2096 BIMS
1208 BIMS
1773 BIMS
1020 BIMS
1130 BIMS
875 BIMS
1013 BIMS
1413 BIMS
3201 BIMS
335 BIMS
Prepared by: Jason Coyne
Occupation: Partne
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
180503R1935
16 October 2018
FUJ00082162
FUJ00082162
Page cxcvi of 225
3500
3000
2000
Axis Title
3
1000
Number of Bis raised each year
2008 © 20092010201
2012-2013. 2014.S 201520162017
Year
2019
15.29 From the above BIM’s the following where identified:
hile POL-0032901.pdf relates to Horizon this report isI
istill run in HNG-X and the numbers below are from the
records Fujitsu now hold:
(01/07/2017 - 30/06/2018:
Error BIMS
(01/07/2016 - 30/06/2017:
Error BIMS
(01/07/2015 - 30/06/2016:
Error BIMS
(01/07/2014 - 30/06/2015:
IBIMS
(01/07/2013 - 30/06/2014:
[Error BIMS
(01/07/2012 - 30/06/2013:
Error BIMS
(01/07/2011 - 30/06/2012:
Error BIMS
(01/07/2010 - 30/06/2011:
[Error BIMS
(01/07/2009 - 30/06/2010:
Error BIMS
01/01/2009 - 30/06/2009:
BIMS.
162 APS Reconciliation
125 APS Reconciliation
104 APS Reconciliation
39 APS Reconciliation Error
37 APS Reconciliation
42 APS Reconciliation
33 APS Reconciliation
69 APS Reconciliation
76 APS Reconciliation
15 APS Reconciliation Error
Prepared by: Jason Coyne
Occupation: Partne
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page cxcvii of 225
15.30 A “System Incident” is defined as the underlying cause of a Business
Incident and is created to track the root cause of the same.
15.31 Depending on the impact, nature and scope of the Business Incident the
POA Problem Management Procedure may be invoked. However, if the
nature of the Business Incident is agreed to be low priority or a one-
off, the BIM reconciliation procedure would suffice. The choice of
procedure is determined by discussion and agreement between Post
Office and the Reconciliation Service for specific Business Incidents as
they occur.
15.32 BIM Reports are issued for each Business Incident generated. BIM
reports are designed to notify Post Office Finance of the detail required
to assist in the reconciliation or settlement process within Post Office.
BIM Reports communicate information concerning the resolution of the
symptom of an underlying cause, not the cause itself. This information
would be supplied via the Problem Management route, if escalated to
that level
The Problem Management Procedure
15.33 The aim of Problem Management is to investigate, eliminate or prevent
causes of Incidents and known errors regarding Post Office Account
and Post Office Limited Infrastructure / Information System and to
prevent the recurrence of Incidents related to these errors. A
“Problem” in this
respect is defined as the unknown underlying root cause of one or
more incidents.?”*
15.34 The Problem Management Process covers both reactive and proactive
functions of Problem Management.
15.35 Whilst there are several responsibilities detailed within the process
document, no mention of advisory to Subpostmasters whereby an
271 SVMSDMPRO0025_5.doc, Post Office Account Customer Service Problem Management Procedure, 12 July
2016 [POL-0146787] [fe6a96a3615a63e094682c7efaa33090]
Prepared by: Jason Coyne
Occupation: Partne r;
Specialist Field: IT Systems 6 j1 UU
On the Instructions of: Freeths LLP ~ i
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page cxcviii of 225
incident may affect branch accounts is detailed. It could be that this
was incorporated within the “investigative” phase, but I would expect
some low-level detail to be provided as to what to do in the event that
a problem that has the potential to reoccur, might impact, and how to
handle such.
15.36 As highlighted in the same process document, effective Problem
Management is dependent upon the effective use of the process by
Fujitsu, Atos, POL and third parties. We have not yet established that
it was the case.
Prepared by: Jason Coyne
Occupation: Partne ° ey
Specialist Field: IT Systems } 41 UU
On the Instructions of: Freeths LLP ~ i
180503R1935
16 October 2018
FUJ00082162
FUJ00082162
Page cxcix of 225
16. Appendix F
Closed Problem Records
Issue ID
Description of Problem
Number of Users Affected
3 Horizon Overnight Release 10% to 40% of users affected
15 Negative Stock Figure on Branch 10% to 40% of users affected
Account
12 HNGx Generic Connectivity issue More than 70% of users
affected
25 End of Icon removal Single User
14 Receipts & Payment Mismatch 10% to 40% of users affected
19 First Rate Reconcilliation issue 10% to 40% of users affected
24 Cash declarations discrepancies 10% to 40% of users affected
30 HOL Online Disc POLFS 10% to 40% of users affected
32 Branches able to settle centrally 40% to 70% of users affected
>£150
37 Transaction harvesting issue 10% to 40% of users affected
52 Streamline payment file issues 10% to 40% of users affected
49 Transaction Correction Failure Single User
43 ROLLOVER Issue with cancel button I 10% to 40% of users affected
67 Harvesting of Branch event data 10% to 40% of users affected
76 Unexplained Fluctuations with the 40% to 70% of users affected
EBT (Electronic Banking
Transactions)
77 Postal Orders causing printer failures} Less than 10% of users
and issues on counters affected
78 Camelot file mapping issue causing I More than 70% of users
discrepancies affected
79 Transcash fees are shown incorrectly} 10% to 40% of users affected
on the end of day report
Prepared by: Jason Coyne
Occupation: Partne
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page cc of 225
80 POLSAP - Files/ data not received I 10% to 40% of users affected
following issues in the Fujitsu
domain
81 Branches receiving “Unable to} 10% to 40% of users affected
Contact Data Centre” message
when attempting rollover.
85 Impact of MDM on live service 40% to 70% of users affected
following recent service incidents
86 Some mobile van branches have 10% to 40% of users affected
various connectivity issues
88 Higate Near Station - 15 postage I Single User
labels produced but receipt shows
14 items.
92 Horizon Methods of Payment Issue More than 70% of users
affected
96 Withdrawn products - I Less than 10% of users
Approximately 30 branches have affected
been affected by the withdrawal of
Bureau / Savings Stamps /
Philatelic products.
105 POLSAP - RIS Table shows minus/ 10% to 40% of users affected
values for Bureau.
115 POLSAP Data issue re. Article More than 70% of users
BSBO000115 Created Incorrectly affected
122 POLSAP Determine if there are RIS3 I 10% to 40% of users affected
Discrepancies
112 POLSAP Analysis of Misbalance on Single User
628100
135 Counter Printer Issues for singerI 10% to 40% of users affected
counter branches
Prepared by: Jason Coyne
Occupation: Partne
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
180503R1935
16 October 2018
FUJ00082162
FUJ00082162
Page cci of 225
140
Branches haven't rolled over into a
new Trading Period for a long time.
10% to 40% of users affected
147 Complex Basket Not Working 10% to 40% of users affected
153 Paystation branch connection) Less than 10% of users
problems affected
155 Shopping Basket files are either I More than 70% of users
being delivered later than expected affected
or not being delivered
154 Transcash values not deleting out,I More than 70% of users
but adding on to total affected
158 Short Dump running automated POeI Single User
settlement
159 Reconciliation identified (via
TPSC257_~—sreport) that POLFS
records were not being created
from BRDB
160 POLSAP - POE Weekly Reports Issue I Single User
162 Duplicate Data Single User
165 Corrective actions process for future I Less than 10% of users
branch escalations affected
164 POLSAP - Missing data in POLSAP Less than 10% of users
(Paystation) affected
182 Disconnected sessions with foreign Less than 10% of users
cards affected
197 Token ID Mismatch More than 70% of users
affected
199 TFS : 5139830 - Issue with DIRD} Single User
files being overwritten
213 New Ingenico High Failure Rate 10% to 40% of users affected
Problem
Prepared by: Jason Coyne
Occupation: Partne
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page ccii of 225
220 Pinpad High Failure Rate Firmware 10% to 40% of users affected
218 HSBC Merchant Aquirer (HMS) Less than 10% of users
incorrect EMIS file to FJ affected
226 Cash & Stock Screen Order 40% to 70% of users affected
228 Fibre Link 40% to 70% of users affected
229 Counter Slip Buffer Less than 10% of users
affected
235 MAG Card Track 2 Length Errors Less than 10% of users
affected
Prepared by: Jason Coyne
Occupation: Partne
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page cciii of 225
17. Appendix G
[See Spreadsheet]
Prepared by: Jason Coyne
Occupation: Partne
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page cciv of 225
18. Appendix H
Failure Point Diagrams
18.1 The diagrams below are constructed summarisations of the processing
components within Horizon that have been identified as common
points in which discrepancies have been recorded.
18.2 The first diagram displays each of the failures examined overlaid on top
of each other. This helps to illustrate that the different aspects of
Horizon have suffered failures.
18.3 The diagrams that follow take each of the failures individually and
attempt display the flow of the transaction (using the dotted coloured
lines) to understand where the failure occurred.
Prepared by: Jason Coyne
Occupation: Partne . ey
Specialist Field: IT Systems } 41 UU
On the Instructions of: Freeths LLP ° i
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page ccv of 225
GMaxwell302)
rightm33145)
" GMaxwell3651K I
I MScardifield22195
CUSTOMER
Prepared by: Jason Coyne
Occupation: Partne
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page ccvi of 225
18.4 KEL cardc219R
KEL Type Unresolved
Title NB102 Section 5 CAPO State 4 -
transaction cancelled by pinpad
but not reversed resulting in
state
4
Raised 11/05/2011
Last Updated 31/10/2016
Release System HNGX
System Product Counter
Issue: MSU report a transaction State 4 on one of the NWB reports,
C4 only. TES shows request and authorisation messages but no
confirmation. Post Office Counter log shows transaction being
declined by the pinpad: MSG00999: Card Removed Too Early.
No cOMessageDetail, hence the transaction does not get reversed.
Summary: Since a State 4 transaction implication is potential for
incorrect settlement with the Financial Institution (FI) and/or
incorrect adjustment of End Customer Account, considering the
above KEL records that the transaction was not reversed as it should
have been it is likely that branch accounts were affected potentially
recording a value transaction that should have been reversed and
was not.
Prepared by: Jason Coyne
Occupation: Partne
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page ccvii of 225
REFERENCE
: DATA
Pereeeerrr eee tere tiers
‘CUSTOMER
Prepared by: Jason Coyne
Occupation: Partne
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page ccviii of 225
18.5 KEL Achai233J
KEL Type Information
Title Incorrect cash declarations
received by Cash Management -
errors relating to cash
management
Raised 22/06/2012
Last Updated 22/06/2012
Release System HNGX
System Product CounterBusinessapplications
Issue: ... it appears that somewhere after leaving Horizon but before
getting into the cash management system, an adjustment is being
made to add the branch Cash in Pouches overnight figure to what
they have actually declared. But rather than using the value for the
correct trading day, it is using the value from the previous day.
Hence the declarations being used by the Cash Management system
do not match the cash actually in the branch.
This problem will only affect branches which frequently hold large
amounts of cash in pouches overnight - these are likely to be the
larger branches. If they are affected by this problem, it is NOT user
error and they can't do anything to correct it themselves. Rebecca
Portch (Inventory Manager - South, Post Office Ltd - Supply Chain)
is aware of the problem - the branch could check with her, via the
cash management team, whether this could be the cause of the
apparent differences.
Prepared by: Jason Coyne
Occupation: Partne
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page ccix of 225
Summary: Since POLSAP are aware of the issue it is likely that any
branch differences could be corrected however, this still evidences
an issue in the data integrity in that incorrect values were being
erroneously added to branch accounts.
Prepared by: Jason Coyne
Occupation: Partne ° ey
Specialist Field: IT Systems } 41 UU
On the Instructions of: Freeths LLP ~ i
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page ccx of 225
os :
REFERENCE a
BRDB
CUSTOMER
Prepared by: Jason Coyne
Occupation: Partne
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
180503R1935
16 October 2018
FUJ00082162
FUJ00082162
Page ccxi of 225
18.6 KEL Dsed4010N
KEL Type Unresolved
Title NB102 Section 5 State 4 - PM
pressing cancel on counter
moments after the customer has
entered their PIN on pinpad
Raised 28/01/2013
Last Updated 20/11/2015
Release System HNGX
System Product
HNG-X Counter(CNT)
Prepared by: Jason Coyne
Occupation: Partne
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page ccxii of 225
Issue: The cash withdrawal was initiated and the customer got as
far as successfully completing PIN entry on the pinpad. However,
within milliseconds of this the PM has pressed Cancel on the counter.
This appears to have put the counter and pinpad in a confused state
as an authorisation request (R1) was generated despite the counter
and pinpad also trying to cancel the transaction.
The PM ended up being prompted with a message saying that the
transaction had been declined and a CANCELLED receipt was
produced for it. Moments after this a successful authorisation
response (A3) was then received which caused an 0291 System
Error to occur. The PM pressed Continue and was returned to the
Front Office home menu from where they continued the transaction
session by selling some stamps before settling the basket by Fast
Cash.
On the branch database the cash withdrawal has been recorded as a
zero value and as a CANCELLED receipt was produced the PM should
not have handed any money over to the customer so this is ok.
However, a CO reversal was not generated for the transaction and
the DRS hasn't got the C12 or C112. It just has the £142 C4 which
means £142 has been taken from the customer account. The
recovery table on the branch database showed no recovery
outstanding for this transaction.
Summary: This ultimately would require input from Post Office to
manually reconcile the issue where there are discrepancies.
Prepared by: Jason Coyne
Occupation: Partne r;
Specialist Field: IT Systems 6 j1 UU
On the Instructions of: Freeths LLP ~ i
180503R1935
FUJ00082162
FUJ00082162
16 October 2018 Page ccxiii of 225
REFERENCE
DATA
oo
_ dsed4010N I
18.7 KEL johnbascoG5222H
CUSTOMER
Prepared by: Jason Coyne
Occupation: Partne
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page ccxiv of 225
KEL Type Information
Title Counter APS Txn posted against
incorrect APS Account
Raised 11/07/217
Last Updated 11/07/2017
Release System HNGX
System Product HNG-XCounter(CNT)
Issue: It is been noted that two AP dangerous goods transactions 6
minutes apart on the same counter in two different baskets. Both
record the same CuRef (barcode), same AP token, same client code
and SVC, but one of them has client account code 3047 (correct)
and the other 3046 (incorrect, does not exist in ref data).
On investigation, we find that the issue is either not reproducible or
there is no code which seems to cause this issue. On every retry,
the client accounting code is always posted correctly to 3047.
Additionally, the reference data is verified and there is no client
accounting code with 3046.
Occurrences of this issue is rare however it has business impact so,
a fix will be provided to prevent this from happening i.e. to add a
check and verify if the clientAccountCode exist in ref data and raise
a system error if it does not.
Summary: Whilst this does not immediately indicate a branch
discrepancy it does highlight issues with reference data occurring in
Prepared by: Jason Coyne
Occupation: Partne
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page ccxv of 225
the processing systems. KEL MWright1458Q details an instance
where reference data could impact branch accounts.
Prepared by: Jason Coyne
Occupation: Partne S eA IT
Specialist Field: IT Systems } jt! OUP
On the Instructions of: Freeths LLP ~ -
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page ccxvi of 225
REFERENCE
DATA
:
:
H
I JohnbascoGs222H I
CUSTOMER
Prepared by: Jason Coyne
Occupation: Partne
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
18.8
FUJ00082162
FUJ00082162
MithyanthaJ1937S
KEL Type Information
Title Exception raised whilst
processing message event -
Serious system error:[Duplicate
JSN detected - failed to insert
journal record]
Raised 06/05/2010
Last Updated 09/08/2016
Release System HNGX
System Product BranchAccessLayer
Issue: Duplicate JSN messages will only occur when there is a record
with the same JSN number already in the message journal table
BRDB_RX_MESSAGE_JOURNAL
UPDATE: 04-Nov-2010 - Until this fix is released, it has been agreed
that these events will be ignored
UPDATE: 16-06-2011- The fix is still with RM (Release Management)
so not yet delivered.
UPDATE: 09-Aug-2016 - Fix was released but variations of issue are
still seen on rare occasions and should be sent to SSC to investigate
Summary: Although it is not documented as causing a potential
branch account discrepancy in this instance, the problem illustrates
how retry requests between the counter and the branch database
(BRDB) can occur and have over an extended time period.
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page ccxviii of 225
KEL
os co
REFERENCE
: DATA
Prepared by: Jason Coyne
Occupation: Partne
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page ccxix of 225
18.9 Wrightm33145J
KEL Type Fault_Fixed
Title Office has Non-Zero Trading
Postion (Receipts/Payments
mismatch) Office receipts do not
match payments. Office has a
non-zero trading position.
Raised 23/09/2010
Last Updated 01/04/2016
Release System HNGX
System Product
CounterBusinessapplications
be corrected.
Issue: The Receipts / Payments mismatch is due to a bug in the
code that occurs when Cancel is pressed on MSG31316. The bug
incorrectly causes the discrepancy to be cleared, and because there
is no balancing transaction (such as a transfer to local suspense) it
gives rise to the accounting error.
Unfortunately the workaround cannot be done after the problem has
occurred at the office! In this case the branch accounts will need to
Summary: Acknowledged issue by Post Office. Correction needed
within branch accounts.
Prepared by: Jason Coyne
Occupation: Partne
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page CCxx of 225
KEL
oo fo
REFERENCE ]
: DATA :
CUSTOMER
Prepared by: Jason Coyne
Occupation: Partne .
Specialist Field: IT Systems itgroup
On the Instructions of: Freeths LLP Gain Tee
180503R1935 16 October 2018
FUJ00082162
FUJ00082162
Page ccxxi of 225
18.10 GMaxwell302)
KEL Type Information
Title Transaction Correction not
received at FAD
Raised 04/10/2005
Last Updated 04/10/2005
Release System S80
System Product TPS
office did not arrive at the office.
Issue: POLFS report that transaction corrections sent for a particular’
Summary: Since Transaction Corrections are
anomalies in financial values, it is imperative these are applied
where necessary. Although this instance identifies the error in the
file validation and therefore ‘catches’ the loss, it still evidences
issues in data processing within Post Office processing systems.
issued to clear
Prepared by: Jason Coyne
Occupation: Partne
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page ccxxii of 225
KEL
Seeeeneeeeeeneeeneeeene eee
.
REFERENCE
=
‘
CUSTOMER
Prepared by: Jason Coyne
Occupation: Partne
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page ccxxiii of 225
18.11 MHarvey2255P
KEL Type Information
Title TPSC294 operational exception
Raised 02/09/2005
Last Updated 31/05/2006
Release System S80
System Product TPS
Issue: This is not a software problem and should only occur when
SSC have = manually added balancing entries to
tps_pol_fs_summaries_incomp following a problem with missing
POLFS mappings.
Summary: This issue relates to manual corrections (balancing
entries) applied within database transaction summary files. Since
summary files inform Post Office of financial positions, it is entirely
possible that this may cause financial discrepancies if manual
corrections were to be incorrectly applied.
Prepared by: Jason Coyne
Occupation: Partne
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page ccxxiv of 225
KEL
REFERENCE
DATA os
‘BRDB
© MHarvey2255P 4
CUSTOMER
Prepared by: Jason Coyne
Occupation: Partne
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page CCxxv of 225
18.12 GMaxwell3651K
KEL Type Information
Title Duplicate transactions sent to
Streamline
Raised 22/12/2004
Last Updated 08/05/2006
Release System S75
System Product DCS
Issue: Streamline report that they have received a payment file
containing a large number of transactions which they had received
in a previous run.
It looks like a problem with the confirmation agent which has
harvested the information twice (most likely due to a previous
failure resulting in a failure to write a new checkpoint).
Summary: This duplication has primary effect with Post Office
processing. It is not clear whether it may have in turn caused
discrepancy in branch accounts.
Prepared by: Jason Coyne
Occupation: Partne
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page ccxxvi of 225
KEL
REFERENCE
L
CUSTOMER
Prepared by: Jason Coyne
Occupation: Partne .
Specialist Field: IT Systems Zitg FOup
On the Instructions of: Freeths LLP cary
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page ccxxvii of 225
18.13 cardc235Q
KEL Type Information
Title Drop&Go top up transaction
times out but is then marked as
successful.
Raised 05/07/2017
Last Updated 05/07/2017
Release System HNGX
System Product CounterBusinessapplications
Issue: The clerk initiated a Drop and Go transaction for £100 which
failed due to timeouts, but then a success message was displayed.
The clerk settled the transaction and the customer handed over
£100. The customer checked the balance and stated that the top up
had not gone through, so the clerk then performed another
Drop&Go transaction which was successful. The customer has paid
in £100 but the branch account has been debited by £200.
Accenture verified that only the second Drop&Go top up was
successful.
Summary: This issue would require a transaction correction being
applied within the branch accounts to remove the discrepancy.
Prepared by: Jason Coyne
Occupation: Partne
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00082162
FUJ00082162
180503R1935 16 October 2018 Page ccxxviii of 225
KEL
REFERENCE
DATA
© carde235Q_
CUSTOMER:
Prepared by: Jason Coyne
Occupation: Partne
Specialist Field: IT Systems ity rou P
On the Instructions of: Freeths LLP Gary