FUJ00183797
FUJ00183797
CASE NUMBER: HQ16X01238, HQ17X02637 & HQ17X04248
IN THE HIGH COURT OF JUSTICE QUEEN'S BENCH DIVISION
BETWEEN:
ALAN BATES & OTHERS
CLAIMANT
AND
POST OFFICE LIMITED
DEFENDANT
E HERT REPORT OF JASON COYNE
16 OCTOBER 2018
OCCUPATION: PARTNER
SPECIALIST FIELD : IT SYSTEMS
ASSISTED BY: SIOBHAN FORSTER
CHRISTOPHER RASKE
JAMIE SMITH
PATRICK GRANT
ON THE INSTRUCTIONS OF: FREETHS LLP
ON BEHALF OF: ALAN BATES & OTHERS
Jason Coyne
IT Group UK Limited
Unit 5 Lockside Office Park
Lockside Road
Preston
PR2 2YS
FUJ00183797
FUJ00183797
Summary of Comments on 180503R1935 Expert Report of Jason Coyne
01-01
Page: 1
Number: 1_ Author: Gareth Jenkins Date: 22/10/2018 16:23:00
To aid my review of this, I have converted the document to Word and am adding in comments using the Word “Comments” feature. This may well have an impact on the
pagination of the original, but will hopefully not impact the actual text.
FUJ00183797
FUJ00183797
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 i of xii
Page
Table of Contents
Table of Annexes
List of Acronyms and Abbreviations ...........::ccsccneeeereeeneeeeeeneeneeeeneeeenen viii
1. TMtrOGuction .....cccccceceeeeeeenenneneneenensenensesensanauenseneaaanaeasaeaeeseneeenenenens 1
BUSINESS SCOPE 20... ceceeeceeeeeeeeeeeeesuueeeeeeueeeeeeeueeseeeeneescseeeeeesueeseseaeeeeeeeneees 1
Agreed List Of ISSUGS .......ccceesseeeeeee ser see eee eeeeeesea eee eeseeeeeeeseeseneeeeaseneeeeneeneee 3
Documents REVIQWEO 00... ceeeeccceeeeeee reste eeeeeeeeeeeeeeuueeeeeee estes eneeeeseeeeeeeennes 7
Author . 9
Declaration of Independence
2. Background
3. Executive Summary .
4. History of Horizon ...
Known Chronology MileStOne@s.........:cccceeecseeee eee neese eee eeesaeeeeereneessneeseeseesen 18
Horizon (pre-Horizon Online) ......cescceeeeeeeseeeeeeeeeeneee ere eeeeeeees ne eeenenneeeenen ee 21
EFTPoS - Electronic Funds Transfer at Point Of Sale. .....cccscscsseseseseseneenene 21
The Introduction of the Network Banking SOIUtion ..........ceceeeeeseeee sere een eeeee 21
HOriZon COMPONENKS ......ecccsecsecseeeeseesee esse eee sesseeseeerseseeeeseneenseseneesaeseneeee 23
UCIT] 610 t B- ]8- EEE EEEEEEEEEOEEEEEEE EE EESEEEECESOTOOOOOEOOESOOS 23
TLANSACEION Data ues ecccsecsecccececncceeeceeecuneccueeseeceaseceeseeeseeeceuesseeeeeeseentoes 24
The Horizon Counter (1996 - 2010) iecccccsecccsesseccceseessecneeseesseeeessnneeesen 25
Migration towards Horizon Online (Horizon Online 2010 - Present) .............. 26
The Branch Database .......cccccsessessssecnceeceeeeeeeeceeeseeneneneseeeeeeeeseeeeseeesaes 28
Legacy Horizon / Horizon Online Counter ProceSSe@s .......csseseseeeeeseeeeeeren sees 31
RECOVELY weccesecccccenecee ee nten eee nneee ee ten ee ete EEEE EEA EEE ELEM A EEEEA UA DEEEEEEEEE EEDA E EEE HEE 31
REVESSAIS .oeeecccccce cece neneeneeneeeeeeeeeeeee nese eae a en ea eee eteeEeeenesbesteseesaesaaeaaanaas 32
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 ii of xii
Horizon Support Service & Facilities .
The Role of Each Support Level .. . 33
Support Services ProviSion i 2OOL .cccsccsecceseccececcereneceteceneeeeeseneecereenees 37
Page
Support Services Provision i 2O1O viccccsccscceeeccnccceeceeccescseeceseeeeeseeneseeee 37
Support Services Provision from 2014 ..sececececceccsccseceenersseteessenneeeererseses 37
Incident Tracking SYStOMS ....cccccceecccsesececseneeeecsseeesecsenecscueneeeseseeesseeees 38
Known Error LOGS (KELS)...ccccsccseecececcnsccececeeccnecceecenescneeseeteeeecessenetieees 38
ReleaSE MANAGEMENL. 0... ceccecccscccnnceeceeeeceeeseeeenesseeseeeeeeeeneeeeseneeeenegens 39
Fault PEAK Lif@CyCle .....scssceeeeececeeccecerecceceecansarseeseesesececeseseeceseareaaanaee 40
Assignment of PEAK Incidents to Maintenance Rel@aSe@S ........cccccceeeeeeeneee 41
5. Horizon Bugs/Errors Defects and Controls ........:sscccesereeeeeeneeeneeeee 42
Known Errors/Bugs Defects acknowledged by Post Office .......:sccsseesereeeeeee 43
Calendar Square / Falkirk ISSUC*? ....cccscccecccccceecseeeesenssssesseeseeeseeceteeeeees 44
PayMents MiSIMAECH .ecceccesccceeceeeeneceneeeeeeeneeeeeseeeesueeeneeseesseeeeneeseetenees 44
SUSPENSE ACCOUNE BUG weeccccesseccecssssscescecenseesesaeseeseeeecenseseruseseusestenses 45
Further Bugs / Errors / Defects not acknowledged by Post Office ..........:.:00 45
DAIMEIINGtON oe cece cscs esceeeseseeeeesececserseccacccacaensceeneeesecesesecesestereeseneasanags 45
Cash Declarations - Cash ManaG@Ment .......ceccccesseecceneeeeeneesesnenereesenenes 46
Bureau & Decimal Point ISSUC ...cccsceceecceeceeeeeeeeeeeseeeeceseeeeesseseeeeseneaaaaees 49
Reference Data Errors ......cccssssssssseeceeceeceeeseseeenenaanneneeneeesenesessessssesenas 49
Duplicat€ TranSa@ctiOns ..ccccccccccseccecccnececccuseceeseeeccesseeeesesuecesseenesessuneeees 51
Fa@il@d RECOVETICS ...cccccecseseneneencnnseeseeeeeeeeeeee nee nee teneeeeeeneeeeeeeaeenseseseaeaas 51
Failed REVEISAIS.....ccccceccccseeccceeeccsecesneeessuneneuenseesseneeeesesuneeeeennesesennenss 53
Uncategorised Bugs/Errors/Defects ......cccccceeececsseeececcneeceeneeeeseeseressenenes 55
HArdWALe EFrOrs w.scecesesssnenseceeeeeeeeeeeee eee e seen nett ee ee eee eee ee babes eee ee eae tata aa ee 57
Fujitsu Closed Problem R@COrdS ...ccccccsscceeceeccnecenecseeseneesenseneeeesesnetsenees 61
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
180503R1935 16 October 2018
Horizon Robustness
Horizon Uptime .
Likelihood of shortfalls i DrAnCheS ....ccccccscesssssssseesecsesesccesesessecaeeasaasaes 70
Extent of errors in data recorded within Horizon arising from (a) data entry,
(b) transfer or (c) processing Of data iN HOFIZON ....cceccesccseecenecnereeeeeseseene 70
Errors that are Potentially the Result of Multiple ISSUCS .....1..csesscesenereeeee 71
In relation to &) Data ENtry 2... csccccssesscsececseceeceeeececeeneenaesanaeneseesenseseees
In relation to b) Transfer (of data)
Uncategorised Horizon Concerns ...
Extent COMCIUSION ....sseceseceescececeeececeeeeenetsesseneneeeseeeeeceseceanenaaeaaeensenee
Measures and Controls to prevent/detect/identify/report/reduce errors in
HOTIZON oo. ceeeeeceeeseeeceeeeeneeneeeueesseeeeersesseueeeeeeeneseeeeeeeeneeserseaeeeensenesee eens 82
Failures of measures and controls to prevent/detect/identify/report/reduce
the risk of errors .. . 86
Opinion Summary 94
6. Reconciliation and Transaction Corrections .........sscceceseseeneneerees 96
Horizon Archit@CtUre ........ssseseeseeeeceeserececeeeseesusaeseueeseseeeeeseeeceeesessusaeaenes 97
External Systems/Cli@nts .........cccccceccceeceeeeee ea eeeeseaeeeessaeeessaeeeeeeeaeeeeeueeee 98
Transaction ACKNOWlEdGEMENES .......cccceceeee eee eeeeeeeeeeeseeeee eens eeenerseneene reas 100
ReCONCITIAtION PFOCESSES .......ceeeceeseeeeteeseeeeseseeeeeeeneeseesenereesnaneeseseeeseeeeen 100
Reconciliation Services/DepartMents ........ccccceee sees eee sees tees eeeeeeneeeaeeeeeee 102
1g 0 BA 6-116 EEE EEE EEE E EEE EEE EEEEEEEEPEPEEEESEOSOOSOO ESOS SSESEESELTOSEO® 102
Fujitsu Management Support Unit (MSU) ....ccccccesccseseneceneseereenereneeeensene 102
Fujitsu Third Line Support (SSC) ....secccseesccceseeeeceeeeeeseensessenensesenseneeseee 102
Reconciliation EXCeptions .........cccceeceeeetesseeeeeeeeeeeeseeeeeeeeeeseessaneeeeneseeeees 103
Data Reconciliation Servic (DRS) .cccccsccesssseeseseseeseeeceseceseceseeeneaseenenes 104
TrANSACEION DISPULES .....ecceeccseceeecnsecenecenecceseeceneceeeeseeceeseeeeteeeeeesoen 106
RECONCIIALION SUIMUITIALY cicceeccccccseceenecnneeeeneseeeeenseeeeeeneeeenseeeeeeseeneneee 106
Transaction COrrectiONS .....sccccsesesceeesseseeeeeseeeeeseseeseeeeeeeesaeseeeasesseesanneees 107
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 iv of xii
Successful processing of a Transaction Correction ..
Unsuccessful processing of a Transaction Correction . 112
Branch Trading Stat€Ment ........ccccceseccnseeeceeneeseseenessenseeeseeseeeeeenersesen 113
Disputing & TranSaction COrrectiOn ....ccccccsecseecceecceeeeneesereeneceereneesennenen 113
Transaction Correction Observations And FINGINGS ...sssssssseseceeesecereeeseees 114
OPINION SUMUIMALY wececccccccsecccccstnececceneeccecueesseseeeeeseeeeceueeeeeseneessseenees 116
7. Horizon Reporting - Facilities for Subpostmasters ............c0cseee 118
Horizon Alerts for Subpostmasters in respect of bugs/errors/defects .......... 118
Bugs Errors Defects not alerted to the SUbpOStMAStEP ......cssscececseeeeneener 121
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page v of xii
How does Horizon enable Subpostmasters to compare the stock and cash in a
+ 122
122
123
123
How does Horizon enable or require Subpostmasters to decide how to deal with,
branch against the stock and cash indicated in Horizon? .
Daily Cash Declaration
Weekly Balance .
Monthly Trading Period Rollover .
dispute, accept or make good an alleged discrepancy by (i) providing his or her
. 123
record and reflect the consequence of raising a dispute on an alleged
. 124
does raising a dispute with the Helpline cause a block to be placed on the value
124
125
enable Subpostmasters to produce (i) Cash Account before 2005 and (ii) Branch
Trading Statement after 2005? .125
enable or require Subpostmasters to continue to trade if they did not complete
own personal funds or (ii) settling centrally?
discrepancy, on Horizon Branch account data and, in particular:
of an alleged shortfall; and.
is that recorded on the Horizon system as a debt due to Post Office? .
a Branch Trading Statement; and, if so, on what basis and with what
consequences on the Horizon system? ve 125
. 125
Opinion Summary
8. Horizon Shortfalls, Data and Reporting for Subpostmasters and Post
OFFICE oo. eee eeceeneeeeeeneeneneeeenenneeeeneneeeeaeeaeeseeeneeeeneetenteneneneeneeneseseenenseneneens 127
Data and Reporting Functions for Post OffiC@ .........cccseceeeeeeeeeeeseeeeeeeeenennes 127
Official Sources Of INFOFMAtION ......csccccceccecececeeececeeseaseeeteteeeseeeeeseseesees 128
Additional Sources Of INFOFMAtiON .....sccsecsessenteteeseeseseseseseeeseseesnesnsnanaen 129
Conclusions Relating to Information Available to Post Office .........:cseeeeeeeee 130
Data and Reporting Functions for Subpostmasters ..........::cseeseeeeeeneeeeeeeeee 130
Issues 8 & 9 - SUMMATrY Of CONCIUSIONS ...ccceecccseceecereecenecneeseneeeersennesere 133
9. Remote Access and Alteration of Transaction Data
REIMOLE ACCESS «os eeseccseceeceenneeeeneneee ene neeen ee nen teen anne cena neseenenneeneaneestenaaes 135
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page vi of xii
. 139
. 139
. 143
Branch Transaction Data Rebuilds
Transaction Correction Tools - Modification of Transaction Data
Effects on Branch Accounting ....
Supporting Evidence of remote access and implemented fixes affecting
CFANSACEION CALA oo. eeiesessseeceeececeeeceeee nee ae nena nseenneeeeeeeeseseeeeseaaeaaaaaaaaaee 146
Permission Controls and Data AUCItING 1... ...csececccceseceeeeneeeeeceeesenseneeaees 146
Balancing / Corrective TranSactiOns ....cccccccccccseeceeceeeesceseeeeccseeeeeeneeneees 147
AUCIt SCFVESS wieeeecccec cece nnnnnnneeneeeenee eee n ences een a etna nen nEnE SEES EES Eeseataetaaaaes 149
PFOCESS VALIAtiONS ....scccccecccec sen erennsnensnneceeeeeseeeeeceseeaeeeseeteeneseeneneeneenes 149
OPINION SUMMALY ......ccccecec es sece esse eeeees cee eeseeaeeeeeaeeeeeeeueeeseseeaeeseeeeeseeseen 150
10. Expert Declaration
Statement Of Truth 0... cccccccceeeeeeeeeeeeeee cess sees eee eee eee eee eee eeeeeeeeeeeeeeeaneaa 154
11. Appendix A.
PEAKS of initial interest ...........ccccccccceeceeeeeeeeeeeseeeeeeeeeeeeeeeeeseeseeeesssaasenees 155
Errors with Financial IMpact ..........:ceceeeseeee eee ceeeee eens eeeeeeeeeeneeeeeeeeeeeeeeeeee 155
PC0027887 (1999)
PC0063227 (2001)
PC0063723 (2001)
PC0098844 (2004)
PC0203131 (2010) ..
PC0203676 (2010)
PC0263451 (2017)
PC0266575 (2018)
PC0273046 (2018) ..
12. APPendix B.......ccccecessesenseneneeueneeseuseeeneeneuensesensaneaseneusesneenenseneeee 178
Horizon Architecture DiagraMs .........cccceeceeeeeeeeeeee eee neese seen eeeeseenesearernesen 178
13. APPeCndix C .....ccccceseecensenenecnenensenensenensseennenenseneusenenseuenteneneuseneneoe 186
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page vii of xii
2012 - 2013 TC’s.
14. APPeNdix D ....cceesccecnessesensensneeteenenessenensenenseneneesenseuensasensauensenene 187
Log of helpdesk calls regarding Transaction Corrections W/E 15 June 2014.
187
15. APPendix E ....ccccccceeeeeseeeeneeneneesenensenanaeaaaneneeseaeaeeeeeseeeeeeneneeneneenes 188
HOriZON REPOFtS .....cssceeseeeeeeeeeeseeeeeeeeeeeaaeeeeeaseeeeeeseneeeseeeeeeeenaeeeeeaeeeeeees 188
TPS Reports
APS Reports
Banking & Related Services Reconciliation (DRS Report Set) ......ccscccsseeee 191
Reconciliation and Incident MANAGeMENt ...icccssececseseeceecneeseessueressseneeees 192
The Problem Management Procedure ...scseccccscecceececseseasestsseneseeeeeseseees 194
16. APPendix F ....cccscceeceneecenseneceeeenenseneeeenecuerenseseneneuseeeueeeenseueesenseee 196
Closed Problem RECOrdS ......ccccseessseeeeeeeeeeeeeeeeeeeeeeeneesen eee eeeeeeeeeeeeeeeeeeees 196
17. Appendix G
18. Appendix H ..
Failure Point Diagrams .........ccccecceeceeeeeceeseseseseeneeeeeeeseseseeeeeseasasaesasaagans 201
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
180503R1935 16 October 2018
Page viii of xii
Table of Annexes
Annex A SLA Summary Week Ending 6 July 2014
Annex B Transaction Correction Diagram
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
180503R1935
16 October 2018
ix of xii
Page
List of Acronyms and Abbreviations
AP
APADC
APOP
APS
AT™
BAL
BAU
BIF
BIM
BRDB
CAPO
CccD
CCTV
CNT
cP
CPU
CRC
cs
CSR+
Automated Payments
A scripting mechanism that allows POL to define what
additional data is captured during a transaction
Automated Payments Out Pay
Automated Payments System
Automated Teller Machine
Branch Access Layer
Business as Usual
Business Impact Forum
Business Incident Management
Branch Database
Card Account Post Office
Contact Controlled Document
Closed-circuit Television
Counter
Change Proposal
Central Processing Unit
Cyclic Redundancy Check
[POA’s] Customer Services organisation
Core System Release Plus
Client Transaction Summary
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page x of xii
CWU Communication Workers Union
pBily Disputed Banking Transaction Notice
DCS Debit Card System
DRS Data Reconciliation Service
DWH Data Warehouse
eBer Enquiry Based Banking Transaction
EBT Electronic Banking Transactions
EFTPOS Electronic Funds Transfer Point of Sale
EMV Europay MasterCard Visa standard for financial smart
cards
EPOSS Electronic Point of Sale System
EPS Electronic Point of Sale Service
ETU Electronic Top Up
FSC Financial Services Centre
FTMS File Transfer Management System
opBc Generic Particulars of Claim
HNG-X Horizon Online
HRSAP. The SAP System used by Royal Mail Group’s Human
Resources to pay Subpostmasters
HSD Horizon Service Desk
HSH Horizon System Helpdesk
IBM International Business Machines Corporation is an
American multinational technology company
ICL International Computers Limited
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
Page: 12
FUJ00183797
FUJ00183797
Number: 1_ Author: Gareth Jer Date: 22/10/2018 16:27:00
Tdon’t recognise this in Horizon. Probably a POL term
Number: 2_Author: Gareth Jenkins Date: 22/10/2018 16:27:00
Tdon’t recognise this in Horizon. Probably a POL term
Number. 3 Author: Gareth Jenkins Date: 22/10/2018 16:28:00
“Tdon’t recognise this in Horizon. Probably a legal or POL term
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page xi of xii
IT Information Technology
JSN Journal Sequence Number
KEL Known Error Log
LFS Logistics Feeder Service
MDM Master Data Manager
MI Major Incident
MSC Managed Service Change
MSU Management Support Unit
MTBF Mean Time Before Failures
NAO National Audit Office
NBE Network Banking Engine
NBSC Network Banking Support Centre
NBX Replacement architecture of network after removal of
NBE
NS&I National Savings and Investments
NWB Network Banking
OBCS Order Book Control Service
ocP Operational Change Proposal
OCR Operational Change Request
OSD ICL Pathways Operation Services Division
OTI Open Tele service Interface
OTT Operation Test Team
OWA Operational Working Agreement
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page xii of xii
P&BA Product and Branch Accounting
PIN Personal Identification Number
PM Post Master
PO Post Office
POA Post Office Account
POCA Post Office Card Account
POCL Post Office Counters Ltd
PODG
Post Office Data Gateway. A generic reference-data
driven system that is used to deliver file-based
information between two end points (internal or
external).
POL Post Office Limited
POLFS Post Office Ltd-Financial System
POLMIS Management Information Service for Post Office Limited
POLSAP Post Office Ltd-Financial System
PONB Post Office’s Network Banking Unit
PTF PEAK Targeting Forum
RDDS Reference Data Distribution System
RDMC Post Office’s Reference Data Management Centre
RDS Reference Data System
SAP-IS SAP Industry Solution
SAPADS
Post Office Ltd’ s Advanced Distribution System (based
on the SAP package) that interfaces to LFS
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page xiii of xii
SCP Supplier Change Proposal
SLA Service Level Agreement
SLT Service Level Target
SMC System Management Centre. selbnd line support
SPM Subpostmaster
SQL Structured Query Language
SSC syflem Support Centre. Third Line Support
SSH Secure Shell
TA Transaction Acknowledgement
TDES Triple - Digital Encryption Standard
TEM Tivoli Enterprise Manager
TES Transaction Enquiry Service
TIP
Transaction Information Processing. This is the remote
end-point of the FTMS service that delivers
reconciliations reports to Post Office Limited /
Transaction Information Processing System
TIPAIS Transaction Processing Interface Specification
TPS Transaction Processing Service/System
XML eXtensible Markup Language.
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 15
Number. 1_Author: Gareth Jer Date: 22/10/2018 16:30:00
Steve P describes the relationship differently.
az;Number: 2_ Author: Simpkins John_Date: 26/10/2018 18:28:00
We also describe this now as Software Support Centre ~ however I believe that in the Riposte era this was correct.
FUJ00183797
FUJ00183797
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 1 of 225
1. Introduction
1.1. I have been instructed to prepare this report by Freeths LLP
1.2 My instructions are to produce a report that focuses on the agreed list of
issues as set out in the Sclikdule to the Court order dated 23 March
2018.
1.3 This report is based as far as reasonably possible on the Model Form of
Expert Report as developed by the Academy of Experts.
1.4 The report seeks to address the issues that concern the Horizon system (as
defined here) and which (a) arise on the parties’ generic statements of
case, (b) can be resolved by IT expert evidence, and (c) require limited,
if any, evidence of fact.
1.5 Defhition:
“the Horizon System” is considered for the purposes of this list of issues
to mean “the Horizon computer system hardware and software,
communications equipment in branch and central data centres where
records of transactions made in branch were processed, as defined in
GPc at §16 and as admitted by Post Office in its Defence at §37."
1.6 This report provides a review of the Horizon system and its composition
from 1999 to present day.
Business Scope
1.7. The business scope of HoFbon is recorded as:2
a. Point of Sale Application
b. Transaction Recording (and Auditing of transactions)
In accordance with the orders given by the Court at the CMC on 22 February 2018
2 Witness Statement of Gareth Jenkins.pdf (Para 2.2), 5 October 2012 [C-0003632]
[b544230cf07249c189cf664fcba6d899]
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 17
Number: 1_ Author: Gareth Jer
“ Tve not seen this, Does it matter?
Date: 22/10/2018 16:31:00
Number: 2 Author: Gareth Jenkins Date: 22/10/2018 16:34:00
Does this indude all the POL components which ae not provided by Fujitsu? Tassume it does.
Number: 3 Author: Gareth Jenkins Date: 22/10/2018 16:33:00
“Again, Ive not seen this, Does it matter?
Number: 4 Author: Gareth Jenkins Date: 22/10/2018 16:37:00
My WS was confining the scope to those parts of Horizon provided by Fujitsu, and not those parts provided by POL. Specifically a number of these interfaces are to POL's back-
end systems, which may be considered to be in scope, depending on exactly what is defined as "Horizon’ in 1.5
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 2 of 225
c. Posting Summary Transactions to POLSAP (Post Office Ltd.’s back
end accounting system)
d. Posting Detailed transactions to Credence (Post Office Ltd.’s back
end Management Information system)
e. Posting Remuneration Data to HRSAP (Royal Mail Group’s back end
Payroll system)
f. Delivering Client Data to Post Office Ltd.’s Clients (i.e. 3rd parties
that Post Office Ltd acts as an agent for)
1.8 Since branch accounts could be affected by data delivered to/from Post
Office back end systems and third-party clients, it is necessary to
consider all of these aspects in order to appropriately opine on issues
which are specific to branch accounts.
1.9 Within this report, the different issues have been grouped together in order
to address similar thematic issues within a single section. I understand
that Dr Worden will adopt a similar approach although he may select a
different method of grouping. The groups I have adopted are:
a. Section 5 - ‘Horizon Bugs/Errors/Defects and Controls’ (Is:
3, 4.and 6);
si,
b. Section 6 - ‘Reconciliation and Transaction Corrections’ (Issues 5
and 15);
c. Section 7 - ‘Horizon Reporting - Facilities for Subpostmasters’
(Issues 2 and 14);
d. Section 8 - ‘Horizon Shortfalls - Data and Reporting for
Subpostmasters and Post Office’ (Issues 8 and 9); and
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 18
Number: 1_ Author: Gareth Jer Date: 22/10/2018 16:40:00
Again, [am not familiar with what these issues are. Looks like there are 15 in total
I've now found them below.
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 3 of 225
e. Section 9 - ‘Remote Access and Transaction Amendments’ (Issues
7,10, 11, 12 and 13).
1,10 The issues in numerical order are listed below for ease of reference.
Agreed List of Issues
Issue 1
1.11To whet extent was it possible or likely for bugs, errors or defects of the
nature alleged at §§23 and 24 of the GPOC and referred to in §§ 49 to
56 of the Generic Defence to have the potential to (a) cause apparent
or alleged discrepancies or shortfalls relating to Subpostmasters’ branch
accounts or transactions, or (b) undermine the reliability of Horizon
accurately to process and to record transactions as alleged at §24.1
GPOC?
Issue 2
1,12 Did the Horizon IT system itself alert Subpostmasters of such bugs, errors
or defects as described in (1) above and if so how?
Issue 3
1.13 To what extent and in what respects is the Horizon System “robust” and
extremely unlikely to be the cause of shortfalls in branches?
Issue 4
1.14 To what extent has there been potential for errors in data recorded within
Horizon to arise in (a) data entry, (b) transfer or (c) processing of data
in Horizon?
Issue 5
1.15 How, if at all, does the Horizon system itself compare transaction data
recorded by Horizon against transaction data from sources outside of
Horizon?
Issue 6
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 19
Number: 1_ Author: Simpkins John _Date: 26/10/2018 19:00:00
Are we able to give a percentage of perceived ‘issues’ to transactions successfully processed. To show that the system was/is in fact very reliable,
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 4 of 225
1.16 To what extent did measures and/or controls that existed in Horizon
prevent, detect, identify, report or reduce to an extremely low level the
risk of the following:
a. data entry errors;
b. data packet or system level errors (including data processing,
effecting, and recording the same);
c. a failure to detect, correct and remedy software coding errors or
bugs;
d. errors in the transmission, replication and storage of transaction
record data; and
e. the data stored in the central data centre not being an accurate
record of transactions entered on branch terminals?
Issue 7
1.17 Were Post Office and/or Fujitsu able to access transaction data recorded
by Horizon remotely (i.e. not from within a branch)?
Issue 8
1.18 What transaction data and reporting functions were available through
Horizon to Post Office for identifying the occurrence of alleged shortfalls
and the causes of alleged shortfalls in branches, including whether they
were caused by bugs, errors and/or defects in the Horizon system?
Issue 9
1.19 At all material times, what transaction data and reporting functions (if any)
were available through Horizon to Subpostmasters for:
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 5 of 225
a. identifying apparent or alleged discrepancies and shortfalls and/or
the causes of the same; and
b. accessing and identifying transactions recorded on Horizon?
Issue 10
1.20 Whether the Defendant and/or Fujitsu have had the ability/facility to: (i)
insert, inject, edit or delete transaction data or data in branch accounts;
(ii) implement fixes in Horizon that had the potential to affect
transaction data or data in branch accounts; or (iii) rebuild branch
transaction data:
a. atall;
b. without the knowledge of the Subpostmaster in question; and
c. without the consent of the Subpostmaster in question?
Issue 11
1.21 If they did, did the Horizon system have any permission controls upon the
use of the above facility, and did the system maintain a log of such
actions and such permission controls?
Issue 12
1.22 If the Defendant and/or Fujitsu did have such ability, how often was that
used, if at all?
Issue 13
1.23 To what extent did use of any such facility have the potential to affect the
reliability of Branches’ accounting positions?
Issue 14
1.24 How (if at all) does the Horizon system and its functionality:
a. enable Subpostmasters to compare the stock and cash in a branch
against the stock and cash indicated on Horizon?
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
180503R1935
16 October 2018 Page 6 of 225
Issue 15
enable or require Subpostmasters to decide how to deal with,
dispute, accept or make good an alleged discrepancy by (i)
providing his or her own personal funds or (ii) settling centrally?
record and reflect the consequence of raising a dispute on an
alleged discrepancy, on Horizon Branch account data and, in
particular:
does raising a dispute with the Helpline cause a block to be placed
on the value of an alleged shortfall; and
is that recorded on the Horizon system as a debt due to Post Office?
enable Subpostmasters to produce (i) Cash Account before 2005
and (ii) Branch Trading Statement after 2005?
enable or require Subpostmasters to continue to trade if they did
not complete a Branch Trading Statement; and, if so, on what basis
and with what consequences on the Horizon system?
1.25 How did Horizon process and/or record Transaction Corrections?
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 7 of 225
Documents Reviewed
1.26 The document categories that were reviewed for the purposes of this report
are as follows:
a. Dimensions Disclosure
b. Horizon Technical Disclosure
c. Additional Horizon Disclosure
d. Stage 01 Disclosure
e. Stage 02 Lead Claimant Disclosure
f. Stage 02 Generic Disclosure
g. Stage 03 Disclosure
h. — PEAK Disclosure
i. KEL Disclosure
j. Second Sight Disclosure
k. — Primary Claimant Disclosure
1. Horizon Disclosure
m. Horizon Witness Statement Disclosure
n. Coyne RFI Disclosure
1.27 Due to the volume of documents in the matter? it has not been possible to
review each document individually at the time of writing. It should also
be noted that PEAK disclosure was not provided until 27 September
2018 and therefore the opportunity for review has also been limited due
to time constraints. Potentially relevant documents have therefore been
initially identified using search terms and then reviewed.
3 Approximately 396,000 parent documents.
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 8 of 225
1.28 Of the 218,367 PEAKs disclosed by POL I have, in the interest of
expediency, used intelligent search techniques to initially review those
that might specifically relate to branch accounts. I have therefore
reviewed 1,262 in the limited timeframe allowed since disclosure.
1.29 Of the 1,262 reviewed I have found evidence of errors with, but not limited
to, financial impact, Reference Data errors and correction to branch
accounts.
1.30 Summaries and key excerpts from a select number of PEAKs which could
have a financial impact upon branches (that are also narrated within
this report) are found at Appendix A.
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 9 of 225
Author
1.31 My“hame is Jason Coyne and I am a Expert Report of Jason Coyne at IT
Group UK Limited.
1.321 have 30 years of experience in Information Technology. During this time
I have designed, programmed and supported Electronic Point of Sale
(EPoS) systems for use in payment handling, stock control, distribution
in addition to full business accounting systems. I have also assisted
public sector councils with revenue and benefits processing systems,
investigated failures within stock market trading systems, European
gambling and gaming networks and determined fraud in retail banking,
point of sale, cash in transit and electronic funds transfer systems.
1.33 1 have been instructed by UK and International firms of solicitors in
connection with technology project fault analysis, software development
and digital forensic investigations. I have been an expert witness in
numerous criminal and civil cases over the past 19 years.
1.341 am a member of the Academy of Experts, British Computer Society and
of the Society for Computers and Law.
1.35 I have also participated in numerous mediations and have appeared before
tribunals and in other forms of dispute resolution in London, Brussels
and Johannesburg.
1.36 In December 2008 I obtained the Expert Witness Diploma from Cardiff
University Law School.
Declaration of Independence
1.37 I have never previously been instructed to undertake work for either party.
I therefore declare that I have no connection with the claimant or
defendant involved in this case that might in any way affect my
independence.
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 25
Number: 1_ Author: Gareth Jenkins Date: 22/10/2018 16:47:00
English?
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 10 of 225
2. Background
2.1 This dispute arises from the action pursued by the Claimants
(Subpostmasters) against the Defendants, Post Office Limited (POL).
2.2 The Defendant (POL) currently operates a network of around 11900 Post
Office branches throughout the UK‘, within which, a high number of
products and services are offered.
2.3 POL appoints a Subpostmaster (or manager equivalent) to run the branch.
The vast majority of Claimants are or were Subpostmasters.
2.41In a number of limited circumstances, it is understood that some branches
have been run by a limited company through a Franchise Agreement,
or by a Crown Office employee, through a contract of employment. For
the purposes of this report, all categories will be referred to as
Subpostmasters.
2.5 Horizon comprises an Electronic Funds Transfer Point of Sale (EFTPOS) retail
and accounting system introduced by POL in Post Office branches in or
around 1999.
2.6 Post Office contracted with ICL Pathway Ltd (latterly Fujitsu Services) and
Horizon was initially piloted in 1996.
2.7 Horizon was updated frequently, at various periods with a major
amendment in 2010 known as “Horizon Online”, often referred to as
“HNG-X”.
2.8 After its introduction in 1999, many Subpostmasters reported difficulties with
the operation of Horizon.
2.9 It is reported by POL that Horizon processes 47 million transactions per
week.®
+ Historically, at the launch of Horizon this number may have been 19,000 branches.
5 2.2 12. Presentation_The Post Office, An Insight_.pdf, The Post Office-An Insight, Angela Van Den Bogerd,
circa 2017, [POL-0021926] [05e2ac28f7b36b04dd83ab301edf9f91}
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 26
Number: 1_ Author: Simpkins John Date: 28/10/2018 23:59:00 Z
About 10,700 currently
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 11 of 225
2.10 In 2003, Fujitsu won the contract to manage the administration of the
Horizon systems, initially until 2010 but this arrangement continues
today.
2.111In 2013 ATOS won the contract to manage the ‘first line’ Horizon support
helpdesk including the management of Reference Data.
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 12 of 225
3. Executive Summary
3.1 In respect of Issue 1, it is agreed in the experts Joint Statement dated 4
September 2018, that bugs/errors/defects that existed within Horizon
have the potential to cause apparent or alleged discrepancies or
shortfalls relating to Subpostmaters’ branch accounts/transactions. It is
also agreed that evidence exists which shows that bugs/error/defects
have caused actual discrepancies or shortfalls relating to
Subpostmasters’ branch account/transactions.
3.2 Eadh discovered bug/error/defect could have remained unresolved in
Horizon for varying periods of time. In my opinion there is the possibility
of other bugs/errors/defects existing within Horizon which have yet to
be discovered and resolved.
3.3 ThElsheer volume of Known Error Logs and reconciliation reports confirm
the wide-ranging extent of the impact of such bugs/errors/defects. This
evidence demonstrates that such bugs/errors/defects would undermine
the reliability of the Horizon system to accurately process and record
transactions.
ct of Issue 2, as agreed in the experts Joint Statement, the extent
to which any IT system can automatically alert its users to bugs within
the system is necessarily limited. Whilst Horizon has some automated
checks which would detect certain bugs there are types of bugs that
would not be detected by such checks.
3.5 Horizon did contain alert messages that would notify a Subpostmaster to
certain instances of some counter level errors, but the wider
implications of the error would be unknown to the Subpostmaster.
3.6 HoBlever, there do not appear to be notifications issued either by Horizon
or by Fujitsu or Post Office where known bugs and defects have been
discovered. The exception may be in relation to the Suspense account
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 28
Number 1_ Author: Gareth Jen!
Tve not seen this, Should Isee it?
Date: 22/10/2018 18:07:00
,Number. 2 Author: Simpkins John _Date: 29/10/2018 00:07:00 Z
This sounds like something that should be shared to us, especially as the ‘experts
have agreed the potential for discrepancies. Also itis referred to multiple times in this document.
,Number 3_ Author: Gareth Jenkins Date: 22/10/2018 18:08:00
This needs to be addressed, but probably best done by our expert
I think we have a good set of checks and balances to identify such issues, but I accept we can’t show 100% success,
Number: 4 Author: Simpkins John _Date: 29/10/2018 00:06:00 Z
Agreed we cannot categorically argue this but again need to mention our automated checks and processes
Number. 5 Author: Gareth Jenkins Date: 22/10/2018 18:10:00
‘Again needs to be addressed
Most reconciliation errors are down to issues with comms and different views of the success or otherwise of transactions, and the whole point of the reconciliation system is to.
identify and resolve them. I don't think these should be classed as bugs or defects.
2,Number: 6 Author: Simpkins John Date: 29/10/2018 00:08:00 Z
There are very few KELs that relate to reconciliation issues, the fact that the reconciliation reports exist at all shows that they willbe investigated.
Number. 7 Author: Simpkins John Date: 29/10/2018 00:10:00 Z
Not sure that we can argue this but we can for reconciliation issues due to Payments and Receipts
»,Number. 8 Author: Gareth Jenkins Date: 22/10/2018 18:12:00
Twould suggest that this is POL's responsibility not ours.
Clearly publishing unresolved defects is asking people to exploit them and so not a good idea
Publishing resolved defects may be taken as implying that the system has issues and reduce confidence in it.
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 13 of 225
issue where Post Office say that Subpostmasters were notified, although
how they were notified has not been disclosed.
3.7 wit regards to Issue 3, whilst the present-day version of Horizon,
supported by manual human support may now be considered as
relatively robust in the spectrum of computer systems used in
businesses today it has undergone major modifications in its history. It
is likely that in 1999 when it was first commissioned, and in 2010 when
it was significantly upgraded (to Horizon Online), it was less robust.
Horizon comprises a hugely complex estate of hundreds of interfacing
systems, each which exposes many potential points of error.
3.8 Post Office have stated that the Horizon system has changed 19426 times
since its inception. I hfe identified that Horizon’s operation is highly
dependent upon tina system processes with reliance on data
messages being replicated until delivery which is oftZh where faults
occur.
3.9 Fultamental in determining the robustness of Horizon is gaining an
understanding of the Post Office (and Fujitsu) manual business
Processes applied when determining and handling the effects of
bugs/errors/defects. Therefore, when technical failures occur,
understanding the correlating manual processes, inputs and respective
fixes applied in each instance to remedy an imperfect system to a
reasonable level of robustness is key.
3.10 HoPzon’s relative robustness does not mean that is thereby extremely
unlikely to be the cause of shortfalls. As agreed in the Joint Statement
between the experts, robustness does not mean perfection. wediso
agree that the level of robustness may have increased or decreased
each time as the systems have changed. It is also known that
© The number of “release notes” reported by Post Office in response to my RFI.
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
Page: 29
FUJ00183797
FUJ00183797
‘This is just stating that immature systems have more defects, it may be worth stating that it was based upon established systems such as Riposte.
Number. 1_ Author: Simpkins John Date: 29/10/2018 00:13:00 Z
However it is also an admission that the current system is robust and mature.
Number. 2_Author: Simpkins John_Date: 29/10/2018 00:34:00 Z
“This is the number of ReleaseNotes which are not necessarily related to the counter or reconciliation,
,Number: 3 Author: Gareth Jenkins Date: 22/10/2018 18:15:00
Tdon't understand what he is getting at here
Number. 4_ Author: Simpkins John Date: 29/10/2018 00:15:00 Z
“This looks like @ count of software releases and stating that such releases may impact in-flight transactions
Number. 5 Author: Gareth Jenkins Date: 22/10/2018 18:15:00
This is how old Horizon operated, but it isn’t true with Horizon Online.
.Number: 6 Author: Simpkins John Date: 29/10/2018 00:23:00 Z
~“Rgreed
Number. 7 Author: Gareth Jenkins Date: 22/10/2018 18:16:00
Data wasn’t processed until replication was known to be complete.
Again not sure what this is getting at.
zNumber: 8 Author: Gareth Jenkins Date: 22/10/2018 18:17:00
Agree. Twould say we had good processes for this.
Number. 9_ Author: Simpkins John _Date: 29/10/2018 00:29:00 Z
This is true of any system however and nota criticism of Horizon
) Number: 10 Author: Simpkins John_Date: 29/10/2018 00:31:00 Z
This does not say anything other than we cannot guarantee to not add bugs when firing things
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 14 of 225
bugs/errors/defects have caused actual discrepancies and shortfalls in
branches.
3.11 ReGbrding Issue 4, it is agreed in the joint statement that the potential
exists for such errors in the data recorded within Horizon. Bugs errors
and defects that were known to Post Office, as well as those that were
not classified as “Known Issues” ultimately presented themselves as
errors in data entry, data transference and data processing. Due to the
nature of the complex and far reaching Horizon interfacing architecture
and real time services, such errors were unavoidable.
3.12 wih regards to Issue 5, reconciliation (the nature by which Post Office and
Horizon compares transaction data recorded by Horizon against
transaction data from sources outside of Horizon) is a large and complex
facility. It involves many different streams of electronic processing from
both Fujitsu data centre computing components, multiple “external
clients”, Post Office and Fujitsu business process departments and
manual investigatory procedures (where corrective fixes are applied, if
necessary). If the reconciliation process identified a difference between
the sources being compared, then manual steps are taken to establish
and correct the errors and potentially issue Transaction Corrections as
a remedy.
3.13 Thal reconciliation process ultimately presents the possibility of further
error within the Horizon system whereby an inappropriate method of fix
was selected, and/or the corrective fixes may have been carried out
erroneously.
3.14 ReGbrding Issue 6, it is agreed in the joint statement that, whilst Horizon
contains measures and controls for detecting system integrity concerns,
the automatic mechanisms have failed in the past.
3.15 There are a range of measures and controls existing in Horizon each
designed to prevent, detect, identify, report and reduce the risk of
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 30
Number: 1_ Author: Simpkins John Date: 29/10/2018 00:39:00 Z
Yes there were newly identified defects and know defects
Number: 2_Author: Simpkins John_Date: 29/10/2018 00:45:00 Z
Seems to refer to the banking reconciliation and TC's
Number: 3_ Author: Simpkins John_Date: 29/10/2018 00:47:00 Z
“Tunderstand the potential but is there any evidence of this happening?
Number 4_ Author: Simpkins John Date: 29/10/2018 00:47:00 Z
When did these fail - vihat examples are they referring to
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 15 of 225
several multifaceted errors. It is likely that during the life of Horizon
system that these measures and controls improved. It is also reasonably
likely that in the majority of cases the measures and controls were
successful. Hollever, there is also evidence to indicate that a
cost/benefit analysis was applied to the fixing of bugs/errors/defects
and that the possibility of error was not reduced as far as possible.
3.16 Regarding Issue 7, documentation illustrates that Fujitsu could remotely
access transaction data recorded by Horizon. Several technical tools
exist with the specific purpose of allowing Fujitsu to carry out
mdilifications and corrective fixes to transaction data. In addition, a
number of external audit reports commissioned by Post Office reported
thal such access could occur without adherence to the control
mechanisms in place.
3.17 With respect to Issues 8 and 9, whilst there are reports available to the
Subpostmaster for accessing and identifying branch transactions
recorded within Horizon; the reports would not necessarily allow them
to identify any discrepancies/or shortfalls that arose from errors
occurring within the processing of data within the Horizon systems, or
of a discrepancy reported by an external client. The experts agree that
other causes of apparent or alleged discrepancies and shortfalls may be
more difficult or impossible to identify from reports or transaction data
available to Subpostmasters.
3.18 Maly Known Error Logs (KELs) identify that not all errors were understood
even by Fujitsu. In the circumstances, it is highly unlikely that a
Subpostmaster could interpret or identify the causes of any bugs/errors
or defects when Fujistsu themselves often did not understand the cause
of such or their full effects.
3.19 Post Office have a significantly larger repository of information available
to them than the Subpostmaster. Post Office could request further lower
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 31
Number. 1_ Author: Simpkins John Date: 29/10/2018 00:49:00 Z
“Agreed.
Number: 2_ Author: Gareth Jenkins Date: 22/10/2018 18:21:00
I don’t believe that there were tools that could modify transactional data. I accept that there were tools that allowed new transactional data to be inserted.
Number. 3 Author: Simpkins John Date: 29/10/2018 00:54:00 Z
“For Riposte the tools only added new versions of an object (Stock Unit or EOD) or add new messages, for NG there Is a tool add new transactions into the branch database.
«Number. 4 Author: Simpkins John_Date: 29/10/2018 01:00:00 Z
For Riposte l agree for HNG I disagree
Number. 5 Author: Gareth Jenkins Date: 22/10/2018 18:23:00
Reports were produced as specified by POL
sNumber: 6 Author: Gareth Jenkins Date: 22/10/2018 18:24:00
Can that be quantified? It sounds very emotive as phrased.
_Number: 7 Author: Simpkins John Date: 29/10/2018 11:03:00 Z
Agree, there are some KELs that were unresolved but were these related to discrepancies.
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 16 of 225
level detail of data from Fujitsu and would have visibility of reports
beyond the counter level that were not available to the Subpostmaster.
3.20 Further agreed in the Joint Statement, identification of errors would
require cooperation of Post Office staff because of a Subpostmasters’
limited knowledge of complex back-end systems.
3.21 In respect of Issue 10, documentation illustrates that a widd range of users
at Fujitsu did and do have the ability and facilities to access and mddlify
transaction data. Fujitsu staff were able to implement changes that had
the potential to affect transaction data both without the knowledge or
consent of the Subpostmaster and/or Post Office. In Budition, a number
of external audits commissioned by Post Office reported that the
appropriate control mechanisms to prevent mistakes being made were
not always followed.
3.22 As agreed in the experts Joint Statement, the very nature of applying fixes
within any IT system, including those implemented by Fujitsu have the
potential to affect transaction data or data in branch accounts.
3.23 Regarding Issue 11, business process rules and technical restrictions
should apply in relation to accessing and modifying transaction data. It
is reported that a documented audit log of each and every occasion of
live data access exists, however, thiflhas not been made available by
Post Office.
3.24 Regarding how often transaction data was acdlssed (Issue 12), I have
asked for this information by way of Request for Information, but at the
date of submitting my report Post Office has refused, making reference
to the relevancy of my request. Post Office states that there are more
than 36,000 documents regarding how often the access was granted
(outside of actions not needing express authorisation that could also be
carried out to fix data), but these have not been made available for
review.
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 32
Number: 1_ Author: Gareth Jenkins Date: 22/10/2018 18:25:00
~ Twould say that it was very few select users who had this capal
iy as required by the job role
Number: 2_ Author: Simpkins John_Date: 29/10/2018 11:04:00 Z
Agree
Number. 3_ Author: Simpkins John Date: 29/10/2018 11:04:00 Z
“Tn tiposte we could ADD only not Modify, in HNG-X the updates are wrapped in the correction tool
Number: 4_ Author: Gareth Jenkins Date: 22/10/2018 18:26:00
Can we address that?
Number: 5_Author: Simpkins John Date: 29/10/2018 11:06:00 Z
Yes, we need to show that our controls and audit is good,
Number: 6 Author: Gareth Jenkins Date: 22/10/2018 18:27:00
is there any reason for this? I suspect it is mixed up with other security logs and it would be very hard to identify it in a simple way.
Number. 7_Author: Simpkins John_Date: 29/10/2018 11:08:00 Z
WE always required authorisation from POL to make a change that could affect the branches financial position, I would expect that the associated evidence were added to the
Peak but they were often via email from POL
Number. 8 Author: Gareth Jenkins Date: 22/10/2018 18:28:00
“Bre we talking about reading it (which was clearly necessary for diagnostic purposes) or inserting new transactions (which would have been relatively rare
I'm assuming we can’t even provide info on the later for old Horizon (we have for HNG-X — just the once in March 2010).
Number: 9 Author: Simpkins John Date: 29/10/2018 11:11:00 Z
‘Agreed accessing the information is necessary daily for normal support.
I'm not sure what this 36,000 documents relates to ~ Audit access?
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 17 of 225
‘ding Issue 13, the implications of Fujitsu carrying out corrective fixes
to data within the Horizon system is that it could have the potential to
affect the reliability of Branches’ accounting positions.
3.26 Issue 14 and how Subpostmasters fulfilled certain processes is detailed
at section 7.
3.27 Issue 15 and how Horizon processes and records Transaction Corrections
is set out in detail at Section 6.48 of this report. For a Transaction
Correction to be processed, the financial value of the correction is
manually assessed by Post Office staff, before submission to the
Subpostmaster via Horizon.
3.28 Transaction Corrections can be issued to either rectify an error or
discrepancy deemed as a fault by the Subpostmaster (or clerk), or when
branch transaction data does not align with Post Office, or a Post Office
external client which may not be an error on the Subpostmasters’ part.
It is also possible that Transaction Corrections wel issued as a result
of error in Horizon transaction data processing.
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 33
Number: 1_ Author: Simpkins John Date: 29/10/2018 13:47:00 Z
Surely this is the whole point!
Number: 2_ Author: Gareth Jenkins Date: 22/10/2018 18:31:00
How does he infer this?
Number. 3 Author: Simpkins John Date: 29/10/2018 13:48:00 Z
"This sounds like proposition.
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 18 of 225
4. History of Horizon
Known Chronology Milestones
4.1 Th) chronology is not exhaustive but rather selects a number of the
milestone dates throughout the development of Horizon.
i. patlway (later Fujitsu Services) awarded contract for Benefits
Payment Card May 1996
ii. Horizon Pilot 1996
iii. Pathway cited “greater than expected complexity” and “...major
implications for the degree of difficulty of the project”? which
ultimately lead to failure of the project.
iv. Post Office Counters Ltd and Pathway sign agreement to utilise
the project to automate Post Offices July 1999
v. Horizon rollout 1999 - 2002
vi. Core System Release - This included the introduction of
Automated Payment Smart cards and APS/TPS reconciliation.
August 2000
vii. Maintenance Release M1 - Prime purposes of the upgrade were
the enhancement of the CSR+ Applications (APS, LFS, EPOSS,
EPS, OBCS), enabling of the AP client variable day file
transmission, enhancement to Reference Data products and
minor changes to TIPAIS Pathway generated CPs to improve
operability of the system February 2001
viii. S04 Release Additional functionality on the Horizon Pilot outlets
to permit the printing of forms Approx. July 2001
7 1.2 2. Final Report - Cancellation of Benefits Payment Card.pdf, National Audit Office - The Cancellation of
the Benefits Payment Card project, 18 August 2000, (Page 9), [C-0003630]
[51956ab654c0a9250059c5848099a80f]
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 34
Number: 1_ Author: Gareth Jer Date: 22/10/2018 18:40:00
This seems reasonable. However I seem to remember that there was a distinction between software releases and Hardware upgrades (for the 5 and B releases)
az;Number: 2_ Author: Gareth Jenkins Date: 22/10/2018 18:32:00
Strictly ICL Pathway, which I believe was originally a separate company, but was then absorbed into ICL or Fujitsu
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 19 of 225
ix. S06 Release Day D rectification measures - This included a new
automatically generated broadcast message to detail when
counters at an outlet are offline. This was to be implemented in
a staged manner and included a receipts and payments fix June
2001
x. $10 Release Data centre and counter upgrade introduced
unattended reboot facility at the counter September 2001
xi. B11 Release the first network banking release, changed the
version of Tivoli used by the whole estate Approx. December
2001
xii. S11 Release January 2002
xiii. B12 Release June 2002
xiv. S20 Release September 2002
xv. B13 Release Approx. September 2002
xvi. Network Banking 2003
xvii. S30 Mails Application /Escher Mails 3.3 package (1 Feb
2003)
xviii. BI3 S70 EMV Banking and Retail,
TDES and NBE Accommodations
2003?
xix. S50 Release October 2003
xx. S60 Release Approx. February 2004
xxi. S52 Release March 2004
xxii. S70 Release October 2004
xxiii. S75 Release (containing changes to support the changeover to
use of NBX banking agents (NBE replacement)? Approx. Oct
Prepared by: Jason
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
Coyne
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 20 of 225
2004
xxiv. IMPACT Programme
xxv. POLFS (a SAP-IS Retail System) implemented 2004
Xxvi. ssblrelease Jan 2004 - Aug 2005°
xxvii. BI3 S80 T&T Harvester Agent accommodations Approx. Nov
2004?
xxviii. POLSAP rationalisation (rationalisation of disparate systems
SAPADS and POLFS) 2007-2009
xxix. $90 ‘Bureau Plastic’ accommodations January 2006
xxx. $92 Release March 2006
xxxi. T10 Release August 2006
xxxii. T40 Release January 2007
xxxiii. Horizon Online Rollout 2010
XXxiv. Firdl Line support provided by ATOS June 2014°
4.2 Whilst Horizon is maintained by Fujitsu (formerly ICL Pathways),Athe
communications between terPhinals and Post Office was initially
subcontracted to Escher?°.
4.3 Horizon, initially based upon Escher’s Riposte messaging system (and later
supported by the WebRiposte system to accommodate Network Banking
changes and the Network Banking Engine supplied by IBM), was
subsequently migrated to Horizon Online. Horizon is therefore
ultimately a composition of many sub-contracted components. '!
® PMREPO13_1.doc, S80 Release Closure Report, 07 February 2007 [POL-0089062]
[9c9308a30adb884074f47fe1 1c7469d7]
5 POL-17645-MGT012 Fujitsu - Horizon Service OWA v2 1 draft.doc, Fujitsu - Horizon Service Operational
Working Agreement, 13 May 2014, [POL-0215476] [339d47429f9f8a83ff93e95e9d3eeb82]
19 TDARCO26v04.doc, Horizon Network Banking Architecture, 30 October 2000
[POL0032839][45d467837b7a6d8cec7914093b39df5]
11 ARCAPPARC0002_0.2.doc, HNG-X Integration Architecture, 08 November 2006 [POL-0087918]
[daecOde8aSeee25b5c9d06730c338dd0]
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 36
Number: 1_ Author: Gareth Jenkins Date: 22/10/2018 18:42:00
“We wouldn't have had a single release for that length of time. This sounds like IMPACT, which was rolled out in one go, but activated over a 3 month period over the summer of
2005.
Number: 2_Author: Simpkins John_Date: 29/10/2018 14:07:00 Z
No comment about the pillars and Verizon taking over the network, ComputaCenter taking over the hardware and IBM looking to replace the software.
coNumber. 3 Author: Gareth Jenkins Date: 22/10/2018 18:44:00
Tdon't think there was ans"
Number 4 Author: Simpkins John_Date: 29/10/2018 14:06:00 Z
Agree
Number 5 Author: Gareth Jenkins Date: 22/10/2018 18:44:00
Not really. Escher were a subcontractor of ICL / Fujitsu and we were responsible for integrating the Escher product with the rest of the solution and specifically we wrote the
counter code based on the Escher development environment and using the Riposte message store as our database.
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 21 of 225
4.4 POL explain that there are 19}
changes), consistent with each of these notes being a change to the
2 release notes (in relation to software
Horizon system.
Horizon (pre-Horizon Online)
EFTPoS ~ Electronic Funds Transfer at Point of Sale.
4.5 Initial work was carried out in 19%) to specify a high-level design approach
for the support of EFTPoS within the Horizon platform.'!* The purpose
was to accommodate new functionality to allow debit and credit cards
as an additional method of payment (MoP) for Post Office goods and
services, directly within the Electronic Point of Sale System (EPOSS).
4.6 The requirement to introduce a Network Banking Service took priority over
(and subsumed) this requirement.
4.7 Post Office Counters Ltd (POCL) objective became to add debit card
payments as an additional method of payment to the Horizon Network
whilst the Network Banking Service was being delivered (initially
200212)
The Introduction of the Network Banking Solution
4.8 After the failure to introduce the Payments Benefit Card in 1999, POCL
and Pathway signed an agreement to automate post offices, and Post
Offices Network Banking Unit (PONB) introduced the requirement to
fulfil banking transactions at Post Office counters so as to offer services
for several different banking institutions.
4.9In summer 2000, a ‘proof of concept’ was undertaken to investigate the
integration of internet technologies with the current Horizon System to
22 NBSRS002_0.4.doc, EFTPoS System Requirements Specification, 12 October 2001 [POL-0062288]
[2f9abb4aa7bfa0a263d0e3bd89101510}
© 1.2 2. Final Report - Cancellation of Benefits Payment Card.pdf, National Audit Office - The Cancellation of
the Benefits Payment Card project, 18 August 2000, [C-0003630] [51956ab654c0a9250059c5848099a80f]
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 37
Number: 1_ Author: Gareth Jenkins Date: 22/10/2018 18:46:00
“Can we explain this huge number? Tassume we had separate Release Notes for LST and Live and would have separate RNs for each class of box that was changed?
»Number: 2_Author: Simpkins John_Date: 29/10/2018 14:08:00 Z
TST, SV&d and Live are all different release notes in the current system, however I think this relates to a count of Release Peaks,
There are 20,091 Peaks with type 'R’ - ReleaseNote. I recently sent a list of them to RM which may have been this request - I'll find the list and add it.
Release notes also include OS patches and other changes that do not relate directly to the software.
z Number. 3 Author: Gareth Jenkins Date: 22/10/2018 18:48:00
Tthink this was much later. The referenced doc is date 2001. I can’t remember exactly vihen we started work on Network Banking. I would say around 2000 or 2001. 1 know I
was on my way to visit Escher to talk about it on 9/11 in 2001!
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 22 of 225
support the delivery of banking transactions** (and subsequently to
replace the loss of the benefits card system). A primary facet of the
Network Banking Solution was the delivery of the banking transactions
within the already established Escher WebRiposte environment. The full
installation and integration of this was the task of ICL Pathway.
4.10 WebRiposte was a message passing technology from Escher Group which
extended the functionality of the existing Riposte Message Server'*
(which was responsible for storing all the data in the Post Office
branches and replicating it to the Data Centres'5).
4.11 1BM were selected for the supply of the Network Banking Engine which was
designed to handle the interface between the Horizon system and the
agreed Financial Institutions, which Post Office referred to as ‘External
Clients’.
4.12 The introduction of network banking and EFTPoS developments brought
with it a more complex enhanced architecture within which further
systems to ensure transaction integrity and reconciliation could be
imposed.
4.13 However, it should be considered that ‘Horizon’ originally stemmed from
an inherited system and architecture with an initial, fundamentally
different design requirement.
4.14 Early documentation identifies that around 1998 and during a period
where Pathway Performance Technical Testing was to be conducted by
agdic (assessing the system from a technical angle of suitability), there
were so many design changes that the totality of the testing had to be
re-planned into four stages'®.
+4 NetworkBanking_Report_FinalV_ReportV1.doc, WebRiposte Framework - Final Report, 22 January 2001 [POL-
0058079] [e709a9e390fb3e7570a1b5c90c78f605]
15 Witness Statement of Gareth Jenkins.pdf, 5th October 2012 [C-0003632]
26 SUTRP004_1.doc Pathway NR2 Technical Testing Performance Tranche1 Closure Report, 08 December 1998
[POL-0047506] (6bcle58bdede0a88d15c008c9403940d]
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 38
Number: 1_ Author: Gareth Jenkins Date: 22/10/2018 18:53:00
Who?
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 23 of 225
4.15 The aim of the testing and various other relied upon documents was for
Pathway to produce a report to meet Horizon acceptance requirements.
4.16 The document further states (in relation to the testing conducted):
“It was not practical to build a performance test rig to replicate the entire
system architecture. The tests were therefore carried out on sub-systems
consisting of system components designed to represent the live system as
resources permit. The system definition, however, has changed
significantly over the test period and tests may not have run on the current
hardware definition.”
4.17 I have not identified nor reviewed a Horizon Acceptance Report. Nor have
I seen the output of the early tests conducted in relation to the Horizon
system. However, it is apparent that the initial system design and
architecture was an evolved and evolving landscape.
Horizon Components
4.18 Horizon is a complex multifaceted system. Core components that facilitate
its operation (an understanding of which are necessary for this report)
are summarised below.
Reference Data
4.19 Reference Data (effectively data about products and operational elements)
is a critical element of the Horizon system, which interfaces with a wide
range of Horizon components. Without Reference Data, Horizon would
not fully function, nor could Subpostmasters operate their branches. It
informs the operation of the Point of Sale system at the counter
amongst many other things. The management and operations with
regard to Reference Data has been outsourced from within Post Office
control to a sub-contractor (ATOS) since June 2014.
4.20 The integrity of Reference Data is critical for the correct operation of a
variety of systems within the Horizon architecture. Pod Office’s
Reference Data Management Centre (RDMC) supports the loading,
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 39
Number: 1_ Author: Gareth Jenkins Date: 23/10/2018 14:04:00
Strictly tis Fujitsu's not POL's. They set things up in MDM
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 24 of 225
storage and release of Reference Data within the Horizon system. The
Reference Data Distribution Service (RDDS) distributes Reference Data
to Post Office branches and other data centre systems. The POL
Reference Data Team is a team dedicated to delivery of Reference Data
and verification of operational business change through Reference Data.
Horizon counter Reference Data was distributed by the Riposte
messaging facility, although this was replaced by a different mechanism
in Horizon Online.
4.21 Despite the criticality of the integrity of Reference Data, a document from
July 2017 suggests that changes to Reflrence Data were not subject to
any appropriate change control process. The document’’ reports; “... we
have now aligned that all Reference Data changes go through the
appropriate change process”. This is consistent with the position that
prior to July 2017 Reference Data could be changed without any formal
consideration as to what the impact might be.
Transaction Data
4.22 According to Fujitsu, there are only four sources of transactions that make
up transaction data*®:
a. Those manually entered by a user in branch at the counter;
b. Transaction Corrections (TC) which are produced by Post Office to
be accepted by a user in branch to correct discrepancies in the
accounts;
c. Transaction Acknowledgements (TA) which are non-counter
transactions and typically initiate from another piece of equipment
(such as a lottery terminal). These transactions are typically
relayed to Post Office/Fujitsu and need accepting into Horizon
before forming part of the branch’s transaction data;
17 Operations Board 21 July 2017.pdf, [POL-0221328] [9c45e0be3ff2b6773447cc6e4 1db5f46]
18 Witness Statement of Torstein Godseth - 27.09.18.pdf
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 40
Number: 1_ Author: Gareth Jenkins Date: 23/10/2018 14:05:00
is this true? I thought we had pretty good control overit. We certainly due in Fujitsu:
Is it worth getting Kevin to respond to this?
Number: 2_Author: Simpkins John Date: 29/10/2018 15:24:00 Z
Kevin and Adam may have some insight here. I have emailed them
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 25 of 225
d. Fujitsu inserted transactions. These are known as _ balancing
transactions and are injected into branch accounts by Fujitsu in
order to ‘balance’ a discrepancy. These do NOT require acceptance
by the Subpostmaster as TC’s and TA’s do.
The Horizon Counter (1996 - 2010)
4.23 The processing of the Horizon counter detailed below has been taken from
the Witness Statement of Gareth Jenkins dated October 2012. 1°
4.24 Horizon was initially designed to store all data locally on the branch
counter’s hard disk (see Appendix B Figure 1).
4.25 Once the data has been successfully stored there it is then replicated
(copied) to the hard disks of any other counters in the branch (or in the
instance of a single counter branch stored to additional external storage
on the counter). It is then passed on from the counter to the Horizon
data centre where it is stored in the CS messagestore.
4.26 The replication process was designed so that if the data was not copied
immediately (because of communications or hardware failure), then
further attempts are made to replicate the data at regular intervals until
it is finally copied successfully.
4.27 Once the data reaches the Horizon data centre a further copy is taken by
the Audit Agent which writes it to an audit file where it is available for
retrieval for up to seven years.
Transaction Auditability
4.28 Data in the audit trail was designed to be sealed with a secure checksum
that is held separately to ensure that transaction data has not been
tampered with or corrupted.
1° Witness Statement of Gareth Jenkins.pdf, Witness Statement of Gareth Jenkins, 05 October 2012 [C0003632]
[b544230cf07249c189cf664fcba6d899]
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 26 of 225
4.29 Horizon was designed so that every record written to the Transaction Log
at the counter should have its own unique incrementing sequence
number. Therefore, it should be possible to detect if any transaction
records become an@nalous (where they are actually captured by the
Transaction Log in the first instance).
4.30 Whilst a customer session was in progress, details of the transactions for
that customer session were normally held in the computer’s memory
until the customer session (referred to as the ‘stack’) was settled. At
that point all details of the transactions were written to the local hard
disk and replicated. When a stack was secured it was written in such a
way that either all the data is written to the local hard disk or none of it
is written.
4.31 The data for the stack will have been successfully secured to the local hard
disk before the screen updated indicating that a new customer session
can be started.
4.32 Although an attempt will have been made to replicate the data to the
messagestore, there is no guarantee at this point that such replication
will have been successful. For example, if Here is a network failure
followed by a terminal failure there is a risk that transactions in the
intervening period could be lost.
4.33 Any failures to write to the hard disk after appropriate retries would result
in the counter ‘failing’ and needing to be restarted.
4.34 Whenever data is retrieved for audit enquiries several checks should be
carried out, namely that:
a. The audit files have not been tampered with and their Journal
Sequence Number (JSN) is incremental to the last one;
b. The individual transactions have their integrity checked to ensure
that no corruption has occurred;
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 42
Number: 1_ Author: Gareth Jer
What does this mean?
Date: 23/10/2018 14:08:00
Number: 2_ Author: Simpkins John Date: 29/10/2018 15:31:00 Z
No idea, Google states the definition as “deviating from what is standard, normal, or expected”, 1do nat see how that relates to the unique message numbering
Number. 3 Author: Simpkins John_Date: 29/10/2018 15:34:00 Z
“True for old horizon, online would not commit to the datacentre.
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 27 of 225
c. Acheck should be made that no records are missing. Each record
generated by a counter has an incremental sequence number and
a check is made that there are no gaps in the sequencing.
Migtation towards Horizon Online (Horizon Online 2010 - Present)
4.35A document dated 2005 authored by Fujitsu records the requirement for
Fujitsu to provide a more competitive solution, since Fujitsu accounted
for 65% of the IT spending for Post Office. 2° The document further notes:
“The original business case for HNG within Post Office was based on a
balance of cost reductions and improved capabilities. The new business
case is almost entirely based on cost reductions.”
4.36 Therefore, the major requirement for Horizon Online was to make secure
minimum changes to the legacy host systems whilst still providing the
end to end business needs.*! Cost reductions in the Horizon Online
solution were to derive from (1) reusing legacy counter hardware, (2)
revised test and disaster resilience strategy, (3) application re-
engineering, (4) revised development processes, and (5) utilising
offshore programmers for software development.7°
4.37 HoHlzon hardware was recycled for continued use in Horizon Online, there
was an acknowledgement in a Board Report ?* regarding design
considerations when implementing Horizon Online:
"While the architecture is generally designed for resilience, cost/risk
trade-offs were agreed in the move from the original Horizon system to
the new HNG-X one which mean that the service is not truly high
availability”.
4.38 Since analysis of serviceability issues within Horizon identified that the two
key areas of cost were (1) the stability and manageability of the Riposte
29 RMARCO02_0.1.doc, Horizon Next Generation - Plan X (HNG-X), 21 September 2005 [POL-0084540]
[754315a4037a6ea4clec7ee070b7d170]
2 Data Reconciliation Service High Level Design Delta for HNG-X, 30 September 2009 - [POL-0032942]
[972420ee28dfe6db41e6847ae3f4493e]
22 BoardPDFpack.pdf, Royal Mail Holdings plc Board, September 2012, [POL-0171024]
[7dad94569f245c16763d0255b114a139]
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 43
Number: 1_ Author: Gareth Jenkins Date: 23/10/2018 14:13:00
Tdon’t like the tone of this section. Itimplies that there was a lot of cost cutting and that the integrity may have suffered,
If anything, HNG-X is more stable and resilient that old Horizon,
The main downside is the fact that branches can’t trade if the data centre has an issue or they can’t communicate with the data centre.
Number: 2_Author: Simpkins John Date: 29/10/2018 15:36:00 Z
Agreed, moving to new technology solutions helped bring the costs down and make the solution more robust.
Number. 3_ Author: Gareth Jenkins Date: 23/10/2018 14:10:00
Tthink it was only the counter hardware that was reused. All the Data Centre hardware was new (with the possible exception of the Audit data arrays which were shipped from
Wigan / Bootle to Belfast).
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 28 of 225
application and (2) the effort in recovering data stored locally on
counters, the Horizon Online objective was therefore to redevelop the
business applications and enforce a centralised model for data
storage.2° The counter hardware would remain and locally store
operational data (e.g., Reference Data) and business logic, but the
transactional data would be stored in the data centre rather than on the
branch counters.?°
4.39 In order to further reduce the overall application development costs within
Horizon Online, substantial re-use of data centre application
components was proposed. The legacy host database applications (TPS,
APS, LFS, DRS and TES) were to remain largely intact, and a web service
interface was to replace Riposte messaging, and simplification of
security mechanisms. 2?
4.40 The diagrams of data centre applications and Horizon / Horizon Online
migration shown in Appendix B (Figure 2 & Figure 3) illustrate the changed
components of Horizon to accommodate Horizon Online and legacy components
remaining, and the changes at branch counter level.
The Branch Database
4.41 Horizon Online brought significant changes to the counter processing as
described above.
4.42 For Horizon Online, all data is now stored in an online branch database
known as the BRDB (therefore, no longer stored on the counter hard
drive).
4.43 Transactions are carried out locally on the Horizon Online counters and a
‘Basket’ is built up during a customer session. Each transaction shdiid
result in a Basket entry consisting of one or more accounting lines.
23 RMARCO002_0.1.doc, Horizon Next Generation - Plan X (HNG-X), 21 September 2005 [POL-0084540]
[754315a4037a6ea4c1ec7ee070b7d170)
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 44
Number: 1_ Author: Gareth Jenkins Date: 23/10/2018 14:16:00
Emotivel Does?
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 29 of 225
4.44 At the end of a customer session when the Basket has been completed
and all settlement items (or tender lines) have been processed and
added into the Basket as further accounting lines (in double entry
terms), such that the total value of the Basket is zero, the entire Basket
is sent to the data centre as a BAL message where the Branch Access
Layer (BAL) processes the message and all the accounting lines are
recorded and committed to the BRDB.~* Horizon Online message flows
are depicted in in Appendix B (Figure 4).
4.45 The BRDB is the repository for all branch transactions and event data
captured. It also provides the storage mechanism for other branch data
such as Transaction Corrections.
4.46 Transactional data is written to the branch database (BRDB) via the Branch
Communication Application on the Branch Application Servers. Logic in
the branch communication application should then be able to discern
between the different transaction types by passing over the data stream
and inserting the data into the relevant BRDB tables. The largest single
function performed by the branch communication application is the
capture of transaction and settlement information resulting from
completion of customer sessions and other activities within the branch
estate. The data needs to therefore be scanned to determine its type
before it is acted upon.”
4.47 The branch communication application will then be responsible for
inserting transaction data directly into the ‘Main Transaction Store’
within the BRDB. The main transaction store then facilitates delivery of
appropriate data to Post Office Account external interfaces via the
legacy applications (See Reconciliation in Horizon Online Diagram
Appendix B - Figure 6).
24 Witness Statement of Gareth Jenkins.pdf, Witness Statement of Gareth Jenkins, 05 October 2012 [C0003632]
[b544230cf07249c189cf664fcba6d899]
25 RMARCO02_0.1.doc, Horizon Next Generation - Plan X (HNG-X), 21 September 2005 [POL-0084540]
[754315a4037a6ea4clec7ee070b7d170)
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 30 of 225
Report Data
4.48 The branch communication application is responsible for inserting a subset
of transaction data directly into a Report Data Store (within the BRDB)
that is designed specifically to facilitate counter/branch daily/weekly
reporting.25
Aggregated Data
4.49 Daily batch processes are responsible for aggregation of data from the
main transaction store. This is stored as aggregated data that should
facilitate calculation of the stock units and branch totals for declarations
and the information required for the rollover processes (transitioning
from one accounting period to the next). DeHaration totals will also be
stored in this area.
Recovery Data
4.50 The recovery data store in the BRDB is used to capture events that need
to be recorded in case of errors mid transaction, to define the
appropriate behaviour of the counter on recovery.
Audit Data
4.51A separate audit record of transactions and events is maintained that
contains the raw data received in the message from the counter. I
understand that this was archived.
Transaction Auditability
4.52 Horizon was designed so that auditable messages from the counter were
stored, together with their digital signature and other key attributes in
the ‘Audit table’ (also known as the Message Journal) in the BRDB. Each
day the contents of this database table should be copied from the BRDB
to a number of serial files. A check should be made that there was no
missing or duplicate Journal Sequencing Numbers (JSNs) for any
counter.
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 46
Number: 1_ Author: Gareth Jer Date: 23/10/2018 14:19:00
Strictly they are held elsewhere, but that isn't relevant, We do store info on declaration in BRDB.
az;Number: 2_ Author: Gareth Jenkins Date: 23/10/2018 14:20:00
What is this supposed to mean? The Audit data is stored as it comes from the counter (as he says) and each night itis Copied to the Audit store where itis avallable for retrieval
for 7 years
See next section.
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 31 of 225
4.53 The files are then copied to the audit system where they are sealed
digitally and held for seven years during which time they may be
retrieved and filtered to produce the relevant audit for a particular
branch. 7°
Legacy Horizon / Horizon Online Counter Processes
4.54 wiflst many core counter processes remained largely the same, technical
processing of certain counter activities changed through the
introduction of Horizon Online.
Recovery
4.55 Disconnections and screen freezes for Horizon Online are recorded as being
dealt with differently to Horizon (pre-Horizon Online).?7
4.56 Subpostmasters are informed through the above quick reference guide to
ensure they "do the right thing at the time the counter becomes
unavailable.”
4.57 Therefore, where a connection to the data centre is lost, users are
presented with a ‘Retry Communication’ message displayed on screen.
Users are encouraged to select the ‘Retry’ option a maximum of two
times to see if data centre connectivity has been restored.
4.58 The Recovery process is stated as one of the procedures to ensure data
integrity remains in the event of a failure.
4.59 In the event the data centre connection is not re-established from the retry
action, the system should settle the session automatically printing three
copies of a Disconnected Session receipt (unless there is a hardware
failure and/or system freeze). Where a connection to the data centre
2° Witness Statement of Gareth Jenkins.pdf, Witness Statement of Gareth Jenkins, 0S October 2012 [C0003632]
[b544230cf07249c189cf664fcba6d899]
27 Tab 12 - Recovery - Horizon Online Quick Reference Guide.pdf, Recovery - Horizon Online Quick Reference
Guide, ca. 2010 [POL-0001727] [34331e3a952d2fb4921 aeddf5d1f90d6]
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 47
Number: 1_ Author: Gareth Jenkins Date: 23/10/2018 14:22:00
The entire counter was rewritten, but the aim was to meet the same business objectives (Functional Equivalence). Mails was totally re-implemented to avoid copyright
infringement with Escher, and Recovery had to be carried out differently to handle the online aspects, but operated on similar principles. The main change to recovery was that
APS Transactions were no longer always recoverable.
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 32 of 225
has been lost before a Basket has been settled the system treats the
transactions as either:
a. Recoverable
b. Non-Recoverable®
4.60 If a transaction is deemed recoverable, then information about that
transaction is recorded in the BRDB. Recoverable transactions are noted
as:?9
a. All banking transactions
b. All Credit / Debit Card transactions
c. All E Top up transactions
d. All reversals
e. Selected APADC transactions
Reversals
4.61 Horizon Reversals are transactions that are effectively ‘undone’ either
initiated by a Subpostmaster or electronically through the system?°.
4.62 If a transaction has been entered and the customer session completed,
the transaction carllbe reversed (for example, if the transaction has
been entered incorrectly, or if a customer requests a refund).
4.63 Reversals are also initiated following system failures. These are
documented later in this report as there is evidence that reversals were
problematic in Horizon and affected branch accounts*!.
28 Recoverable and Non-Recoverable items are defined in the Glossary
29 HorizonOnlineDatalntegrity_POL.DOC, Horizon Online Data Integrity for Post Office Ltd, 28 March 2012,
(Version: 0.1b), [POL-0221055] [5e05904c2f271098da69b31806d4053c]
3° CSPROO21_2.doc, NR2 ELECTRONIC POINT OF SALE SERVICE: Processes and Procedures Description, 30
June 1999 [POL-0049668] [5b45d4ba533d9092293476bc6911e863]
3 1.6 6. Horizon Data (status Draft) - the _Helen Rose Report_.pdf, Horizon data Lepton SPSO 191320, 12
June 2013 [POL-0221677, POL-0221678, POL-0221679, POL-0221680, POL-0221681]
[f296f6880e1b84 18f37d3e344374c42a]
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 48
Number: 1 Author: Gareth Jer Date: 23/10/2018 14:26:00
Not sure what this mean, unless itis the automatic reversal of a failed banking / DCS transaction following a timeout.
Number: 2_Author: Simpkins John _Date: 29/10/2018 15:53:00 Z
Yes I think he is talking about roll-back recovery.
Number. 3 Author: Gareth Jenkins Date: 23/10/2018 14:27:00
“Not always. There are business rules that control when /ifa transaction can be reversed,
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 33 of 225
4.64 A reversal does not result in transaction information in the journal being
amended but causes the insertion of additional, compensating and
correcting transactions.°°
4.65 It is recorded there are four types of reversal:
a. An Existing Reversal - performed when a customer wants a refund
for which there is a receipt, the user must enter the transaction
session number to initiate the reversal.
b. A dw reversal - performed when a customer wishes to obtain a
refund for a purchase for which they have no receipt. New
reversals do not require the original transaction to be identified.
c. A transfer reversal - performed if it is necessary to reverse a
transfer out that has not yet been transferred in to the receiving
stock unit.
dA mittance reversal - performed when a transaction for stock
that has been remitted in or out needs to be reversed.
Horizon Support Service & Facilities
4.66 The support model utilised for managing issues and potential
bugs/errors/defects in Horizon/Horizon Online is based on four levels of
support??:
1 line 2ad line ded line 4th line
The Role of Each Support Level
4.67 1st line support (currently provided by POL’s subcontractor ATOS) - log
incidents either by directly interacting with the user (typically the
Subpostmaster) or from mdiitoring systems. They document incident
symptoms and aim to resolve all issues where the cause is user training
32 SVMSDMPROO875_1.DOC, End to End Application Support Strategy, 28 July 2011, [POL-0122492]
[db0644e4d5e1 1bScce3ed381cb108a88]
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 49
Number: 1_ Author: Gareth Jer
Date: 23/10/2018 14:29:00
There are very limited circumstances which allow New Reversals (controlled by Ref Data).
Number: 2_Author: Gareth Jenkins Date: 23/10/2018 14:29:00
Presumably he is referring to @ Pouch reversal when a pouch that has been packed for being collected Is then unpacked to return the cash / stock into use.
This is only permitted for Rem Out and not for Rem In.
Number: 3_Author: Gareth Jenkins Date: 23/10/2018 14:32:00
is that done by Istline? I thought that was 2nd line?
I'l leave CS / SSC to comment further.
»,Number. 4 Author: Simpkins John_Date: 29/10/2018 15:56:00 Z
Monitoring is not done by ist line, ATOS do not have access to any Fujitsu monitors
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 34 of 225
or environment. 1st line resolve incidents by the identification of
knowledge base entries and the application of defined scripts. A new
incident is raised for each critical event that is not already documented
and passed to 2nd line support teams for action.
4.68 2nd line support (provided by Fujitsu) - use symptoms documented by the
1st line to understand errors and gather additional information. They
use expert knowledge to identify a root cause and a solution or
alternatively to develop procedural workarounds. They also
crefte/update the knowledge base and pass new incidents to the 3rd
line team for further investigation.
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 50
Number: 1 Author: Gareth Jer Date: 23/10/2018 14:34:00
Tthought that was primarily 3rd / 4th line that did that
Number: 2_ Author: Simpkins John_Date: 29/10/2018 15:57:00 Z
The SMC create KELs where there were none that cover the incident, The SSC must authorise the KEL before it is searchable,
This is where the monitoring should be mentioned.
‘The MAC team also present a 2nd line team and they are more likely to review the tickets from the 1st line than the SMC.
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 35 of 225
First and Second Line Support
User issue
(i
Refer to
ive support ——I,_ Rraledgsase ea
fhetp bese) Database
w
:
2°4 Line Support
{6ystein Maintenance Cente)
=
y
Cs Y Provide
Create KEL &
Log incident > A] sohtionto
N
:
tink callto previous incidents
to avoid duptication
4.69 3rd line support (provided by Fujitsu) - apply analytical skills to the
symptoms and evidence gathered by 1st and 2nd line and undertake in-
depth investigation into incidents. They should have detailed knowledge
of the system based upon documentation and source code inspection
and they can produce interim workarounds. They identify a cause and
probable solution for incidents which are being passed to the 4th line
team. Responsible for the implementation of any workarounds that
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 36 of 225
require data changes to the live system, they are the only unit with
authorisation and sufficient physical security controls to perform this
function.
4.70 4th Line Support (provided by Fujitsu) should have detailed knowledge of
the system and are responsible for the investigation and resolution of
new incidents through the production of permanent fixes to repair the
root cause of an incident or a problem in the live application.
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
180503R1935 16 October 2018
Page 37 of 225
Third and Fourth Line Support
Incident passed
from 2" Line Support
initial investigation
Provide solution
to user
In depth investigation,
identify root cause
and propose solution
t
Updaie Knowledgebase
T
v
Pass to 4" Line Support
T
¥
Assess Work Required
to Fix Peak
J
Submit Peak to
Business impact Forum
for Authorisation
Develop Fix
and Deploy in Release
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 38 of 225
4.71 The details of the support model provision changed between the rollout of
the original Horizon application (circa 1999) and when Horizon Online
went live (circa 2010).
Support Services Provision in 2001
4.72 In 2001, 1st line support to the Horizon system users was provided by the
Horizon System Helpdesk (HSH), run by the Operational Services
Division (OSD) of ICL.
4.73 2nd line support was provided by System Management Centre (SMC), also
run by OSD.
4.74 3rd line support was provided by the Software Support Centre (SSC).
4.75 4ttHline support was provided by a combination of Pathway Development,
Escher, OSD and Eicon.
Support Services Provision in 2010
4.76 By 2010, HSH had been renamed to the Horizon Support Desk (HSD) and
was run by Fujitsu.
4.77 SMC had become more of a ist line support unit and there was no
dedicated 2nd line support unit. Instead, the 2nd line responsibilities
were being fulfilled between the 1st line and 3rd line units providing a
“virtual” 2nd line function.
4.78 SSC continued to provide 3rd line support.
4.79 attBline support had been streamlined to the Fujitsu Application Support
Service (ASA). ASA would then liaise with Fujitsu Services’
subcontractors/suppliers, or Post Office's suppliers as appropriate.
Support Services Provision from 2014
4.80 Atos took over the first line support of the Horizon service from 17 June
2014, as part of the Post Office re-procurement of the IT Supply Chain
in 2014.3
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 54
Number: 1 Author: Simpkins John Date: 29/10/2018 16:03:00 Z
Oracle, Microsoft, DLR, Ithaca, Epson
ge Number: 2_ Author: Simpkins John Date: 29/10/2018 16:10:00 Z
Was this just another rebranding as I do not remember the ASA
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 39 of 225
33 SVMSDMOLA3308_2.5.DOCX - FUJITSU - HORIZON SERVICE OPERATIONAL WORKING AGREEMENT [POL-
0128502] [cdd379390652be400250cf319c7bbdb8]
Incident Tracking Systems
4.81 In 2001, 1st and 2nd line support teams used the Powerhelp helpdesk
system from Astea Inc. to log incidents.*?
4.82 3rd and 4th line support teams used PinICL as a call management system
and diagnostic database.
4.83 Calls from 2nd line support were transferred from Powerhelp to PinICL via
an Open Tele service Interface (OTI) link, and updates to the PinICL
calls were transferred back to second line support using the same
mechanism.
4.84 By 2010, the ist and 2nd line support Powerhelp system had been
replaced by a system called Triole for Service (TfS) to record incidents
and PinICL had been replaced by PEAK, an in-house developed Fujitsu
services incident and release management system. ** An individual
incident so recorded is referred to as a PEAK.
4.85 OTI was still used to link the TfS and PEAK systems, although TfS has a
limit of 4000 characters within a single update which potentially
exposed a loss in information.
4.86 From 2014, all 1st line tickets were logged into Atos’ SDM12 service desk
application and if not resolved on first call, were transferred to a 2nd
line (or higher) team within Fujitsu. Thidl transfer was provided by an
automated interface into Fujitsu's Triole for Service (TFS) tool.3°
> CSQMS004_2.doc, CS Support Services Operations Manual, 29 January 2001 [POL-0061572]
[bb842d86176aa926d3b9fff25e0fc248]
>* SVMSDMPRO0875_1.DOC, End to End Application Support Strategy, 28 July 2011, [POL-0122492]
[db0644e4d5e1 1bScce3ed381cb108a88]
35 SVMSDMOLA3308_2.5.DOCX - FUJITSU - HORIZON SERVICE OPERATIONAL WORKING AGREEMENT
[POLO128502] [cdd379390652be400250cf319c7 bbdb8 }
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 55
Number: 1 Author: Simpkins John Date: 29/10/2018 16:12:00 Z
This is the HDI and update to the OTl interface.
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 40 of 225
Known Error Logs (KELs)
4.87 The principle of the KEL has been used since 2001 to record information
and workarounds for known issues. This has evolved into a database
maintained by SSC that is avéllable to all levels of support.
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 56
Number: 1_ Author: Simpkins John Date: 29/10/2018 16:13:00 Z
This is not available to ATOS ‘Ist line.
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 41 of 225
4.88 1stlline support utilise the KEL database to check for known issues and
available workarounds before passing incidents on to 2nd line support.
Where no KEL is found, 2nd line support create a new KEL to record a
workaround or as part of raising a new incident on the PEAK system for
3rd line support to investigate.
4.89 Infétmation on the KEL is updated by all levels of support as work to
resolve the incident progresses. Any creation or update of a KEL must
be authorised by SSC before it can be seen by all users.
4.90 all lew PEAK incidents should be accompanied by a KEL.
4.91 It is possible for a KEL to be linked to multiple PEAK incidents and for a
PEAK incident to reference multiple KELs.
Release Management
4,92 There are three types of release: °°
a. Major Releases - these will normally be to deliver significant new
functionality
b. Enfalrgency fixes - implemented as required to resolve operational
bugs/errors and defect issues
c. Maintenance Releases - applied frequently to address minor
bugs/errors/defects or security issues and the mechanism by
which Fault PEAKs are usually resolved.
Release Management Strategy,26 January 2015, [POL 383f7afbb2eed8092e66aa07f0c6b6be]
38 -0138750] [
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 57
Number: 1 Author: Simpkins John Date: 29/10/2018 16:13:00 Z
RTOS do not have access and would use their own KEL system:
Number: 2_ Author: Gareth Jenkins Date: 23/10/2018 14:38:00
Tthought KELs were normally updated when the Peak was resolved
However I'm not the expert there.
Number: 3 Author: Simpkins John_Date: 29/10/2018 16:14:00 Z
They can be updated at any time, sometimes to keep track of the size ofan issue
Number. 4 Author: Gareth Jenkins Date: 23/10/2018 14:39:00
Is this True?
Number: 5 Author: Simpkins John Date: 29/10/2018 16:15:00 Z
twas true that if a KEL is found it should be referred to in the i
this.
ident and if not the SMC would raise a skeleton KEL to be completed by the SSC, however not every Peak has
;Number: 6 Author: Gareth Jenkins Date: 23/10/2018 14:40:00
We try and avoid these where possible and defer to the next maintenance release
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 42 of 225
Fault PEAK Lifecycle22
PEAR. protiers meord
eraatec acd aysighed fo
DEY want
Required
ed IO PEE PER f
geting Forum
SWeeK Feed the ATED!
Buraace Slanning ¥ oe
' Schediied Maicenance:
Feimane avatatte to PTF i
Romane
4.93 The basic Fault PEAK Lifecycle is as follows:
i. Open - Initial state of a PEAK call
ii. Pending - Incident is being investigated
iii. Final - Investigations have finished —Final response is sent back
for approval
iv. Closed - Final state - call is closed on the PEAK system.
4.94 If a PEAK has been sent to a team it is that team’s responsibility to monitor
it, send it on for investigation, fix it or release it to the correct pekk
stack (queue).
Release Management Strategy,26 January 2015, [POL 383f7afbb2eed8092e66aa07f0c6b6be]
9 0138750] [
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 58
Number: 1_ Author: Simpkins John Date: 29/10/2018 16:18:00 Z
Peak resolvers team
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 43 of 225
Assignment of PEAK Incidents to Maintenance Releases
4.95 Fault PEAKs are investigated and if a fix is required it is sent with an
assessment of the time to fix and any issues to the PEAK stack for the
Business Impact Forum (BIF).28 The BIF convenes weekly to review
outstanding PEAKs for consideration of the business impact.
4.96 If the BIF decide that a fix is not warranted for cost or other reasons, the
Known Error Log (KEL) is updated and the PEAK is closed.
4.97 If the BIF decide that the PEAK should be fixed it is submitted to the PEAK
stack for the PEAK Targeting Forum (PTF).
4.98 The PTF convenes weekly and considers new Fault PEAKs and PEAKs that
have previously been deferred, as well as pels which introduce
business change associated with approved Change Proposals. These
PEAKs will be targeted to a specific maintenance release*® taking into
account timing, other development activities and associated factors.
4.99 The PEAK is then sent back to Development to deliver the fix for
incorporation into th@naintenance release.
Referpeshosiregereent Sorakeyy BEF landienF BD1Sul~PQI14, [POL-0032S82f7arbb2eed8092e66aa07f0c6b6be]
[52536a2c7ab381b9773db1 36ebb9042b]
3 0138750] [
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 59
Number. 1_Author: Gareth Jer Date: 23/10/2018 14:43:00
No. We don't have Peaks for CPs. However I believe that the targeting of CPs to Releases is done alongside targeting of Peaks by the same people.
ge Number: 2_Author: Gareth Jenkins Date: 23/10/2018 14:44:00
The specified
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 44 of 225
5. Horizon Bugs/Errors Defects and Controls
Issue 1 - To what extent was it possible or likely for bugs, errors or defects of
the nature alleged at §§23 and 24 of the GPOC and referred to in §§ 49 to 56
of the Generic Defence to have the potential to (a) cause apparent or alleged
discrepancies or shortfalls relating to Subpostmasters’ branch accounts or
transactions, or (b) undermine the reliability of Horizon accurately to process
and to record transactions as alleged at §24.1 GPOC?
Issue 3 - To what extent and in what respects is the Horizon System “robust”
and extremely unlikely to be the cause of shortfalls in branches?
Issue 4 - To what extent has there been potential for errors in data recorded
within Horizon to arise in (a) data entry, (b) transfer or (c) processing of data
in Horizon?
Issue 6 - To what extent did measures and/or controls that existed in Horizon
prevent, detect, identify, report or reduce to an extremely low level the risk of
the following:
a. data entry errors;
b. data packet or system level errors (including data processing,
effecting, and recording the same);
c. a failure to detect, correct and remedy software coding errors or
bugs;
d. errors in the transmission, replication and storage of transaction
record data; and
e. the data stored in the central data centre not being an accurate
record of transactions entered on branch terminals?
5.1 There are several areas where Bugs, Errors and Defects could occur.
Some of these are set out below:
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 45 of 225
a. Failure of a software component (counter software, Horizon data
centre components)
b. Error in Reference Data
c. Failure of a hardware component at counter (CPU, PIN pad, touch
screen, keyboard, counter scanners, network router)
d. Failure of a hardware component at poliata centre (Database
servers, communication servers)
e. Errors in various communications which need to take place for
Horizon to function:
i. Transfer between counter and PIN pad,
ii. Trdisfer between counter and Branch Database (BRDB),
iii. Trddsfer between Branch Database and Post Office back end
systems,
iv. Transfer between Authorisation Agent and Post Offices External
Clients.
f. Those areas as set out in Selion 5 ‘Horizon Robustness’ and
Section 6 ‘Reconciliation’.
5.2 As agreed in the Joint Statement of experts, evidence exists that
bugs/errors/defects have caused actual discrepancies and/or shortfalls
relating to Subpostmaster branch accounts.
5.3 Identified cormon failure points throughout Horizon are demonstrated at
Appendix H. This list is not exhaustive but is based on the review of
evidence undertaken to date.
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 61
Number: 1 Author: Gareth Jen! Date: 23/10/2018 14:46:00
Tassume he means Fujitsu data centre. Any problem in POL's data centres would have no impact other than perhaps generating incorrect TCs.
»Number: 2_Author: Gareth Jenkins Date: 23/10/2018 14:50:00
This is HNG-X only. Also strictly its to the BAL which is responsible for all counter communication eg to Auth Agents as well as BROB.
For old Horizon also need to consider failure to communicate between counters (ie the local LAN),
Number: 3 Author: Simpkins John_Date: 29/10/2018 17:35:00 Z
Agreed, also I have not seen any mention of the node 1 (gateway) counter being different to the others for replication to the datacentre.
Number: 4 Author: Simpkins John_Date: 29/10/2018 17:37:00 Z
Needs to include TPS/APS to clients especially pre HNG-X
Number: 5 Author: Gareth Jenkins Date: 23/10/2018 14:48:00
These arent reall the section names.
Number. 6 Author: Gareth Jenkins Date: 23/10/2018 14:49:00
Is this ones that are agreed or those being raised by the prosecution?
Il ook at it when I get there!
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 46 of 225
Known Errors/Bugs Defects acknowledged by Post Office
5.4 It should be noted that there are several known bugs/errors/defects
previously acknowledged by Post Office that have affected branch
accounts. These are known as the “Payments Mismatch” defect,
“Calendar Square / Falkirk” bug, and the “Suspense Account bug”.*°
Calendar Square / Falkirk Issue*!
5.5 This defect was discovered in 2005 and fixed in March 2006 and involved
Horizon failing to recognise transfers between different stock units. In
summary, stobk units receiving transfers could not “see” them, resulting
in branch account discrepancies.
Payments Mismatch
5.6 This bug affected at least 62 branches and related to the process of moving
discrepancies into the local suspense account. The majority of incidents
are recorded as occurring between August and October 2010.
5.7 The bug was documented in a report from Gareth Jenkins 29 September
2010* where it was stated:
“This has the following consequences: There will be a receipts and
payment mismatch corresponding to the value of discrepancies that were
lost. Note that if the user doesn't check their final balance report carefully
they may be unaware of the issue since there is no explicit message when
a receipts and payment mismatch is found on the final balance (the user
is only prompted when one is just detected during a trial balance)”
5.8 This issue is reported as causing discrepancies showing at the Horizon
counter which disappeared when branches followed certain process
steps. However, these discrepancies still appeared within the back-end
branch account. It is noted that the issue occurred if a branch cancelled
“© 1.4, 6. Letter of Response - Schedule 6.pdf, SCHEDULE 6: REBUTTAL OF ALLEGATIONS AGAINST HORIZON
(Page 97), ca. 2017 [368e44cc103e58561c578501459748/9]
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 62
Number: 1_ Author: Simpkins John Date: 29/10/2018 17:39:00 Z
The discrepancies would only occurred IF they accepted the transfer in twice because the transfer in messages were not ‘seen.
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 47 of 225
the completion of the trading period and then, within the same session,
continued to roll into a new balance period.
5.9 PEAK PC0204765* is a “Master PEAK” that records branches which were
thought that have been affected.
5.10 KEL wrightm33145)*3 identifies the workaround but also states:
2 3429 SM BP Correcting Accounts for Lost Discrepancies - 102000790 - CD1.pdf, Correcting Accounts
for "lost" Discrepancies, 29 September 2010 [POL-0010769] [804ea47c166870b7ed0359e4 76520265]
bid Wrightm33145).html, HNG-X KEL wrightm33145J, 23 September 2010 (last updated 01 April 2016),
[POL0040409] [1f025ec713c287ee7a5b17accd25b42f]
“Unfortunately the workaround cannot be done after the problem has
occurred at the office! In this case the branch accounts need to be
corrected.”
5.111t not clear how many corrections were required to fix all instances of
this (or even that all instances were indeed fixed) or when a full audit
was completed.*?
Suspense Account Bug
5.12 The suspense account bug caused Horizon to erroneously replicate
suspense account items. It appears that the bug caused Horizon to use
2010 monthly Branch trading figures for 2011 and 2012.
5.13 It is reported that POL later investigated and identified the same bug as
being the cause of the issue in January 2013** suggesting that the bug
may have been resident within Horizon for an extended period.
5.14 POL-0215998*° list the 14 affected branches identified in the occurrence
of the bug in 2013.
Further Bugs / Errors / Defects not acknowledged by Post Office
5.15 The following errors are observed to have occurred within the Horizon
system, however they do not appear to have not been acknowledged as
system wide issues by POL in the same manner as those listed above.
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 63
Number. 1_Author: Gareth JenI Date: 23/10/2018 14:53:00
Corrections were made by POL based on data we supplied. Tunderstand that they planned to wiite off any cases where the SPM made a gain as a result of the Issue and refund
any losses.
However POL need to say exactly what they did when and how.
Although inserting transactions by Fujitsu was considered as an option it was rejected and no changes were made by Fujitsu to transactional data or to Opening Figures.
Number. 2_Author: Gareth Jenkins Date: 23/10/2018 14:56:00
Tbelieve that we checked all affected branches in all the affected years and 14 was that total (basically it was the same branches each year!)
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 48 of 225
Dalmellington
5.16 thd defect relates to the example that occurred specifically in 2015 at the
Dalmellington Branch in which a Subpostmistress performed a cash
remittance from a core branch to an outreach branch.*® The acceptance
at the outreach branch resulted in quadruple remittance transactions*”
resulting in a £24,000 discrepancy.
“* 1.4. 6. Letter of Response - Schedule 6.pdf, SCHEDULE 6: REBUTTAL OF ALLEGATIONS AGAINST HORIZON
(Page 97 ~ Para: 4.5), ca. 2017 [368e44cc103e58561¢5785014597d8f9}
“5 Initial Report from Gareth LocalSuspense.docx dated 10 May 2013 [POL-0215998]
[84b7fac96c5b77ed13642d885b7de2a4}
“© 11.Email thread between ATOS and CWU.pdf [C-0005343]
7 1.Dalmellington Branch duplicated receipts.pdf [C-0005350, C-0005342]
5.17 It is reported in the email thread between Atos and a Helen Baker of CWU
that the remittance team in Chesterfield were aware of the fault (as Bhis
is noted as occurring a “couple of years ago”) and that Transaction
Corrections could be issued for the ‘extra’ remittances.
5.18 The email thread further states (by Atos) that the root cause was identified
as the us& forcing log off when post log in checks had not fully
completed and it was therefore considered as a process issue and not a
technical one. HoWever, this is not consistent with the event logs that
do not indicate it was as a result of forced log off.
5.19 Further, Atos refer to release of "a code change that will avoid further
instances of this across the estate” to be included within Release 13.05
to be deployed in March 2016 (five months after the incident). thélract
that Atos made a change to the Horizon system to prevent reoccurrence
is therefore consistent with this being a software bug.
Cash Declarations - Cash Management
5.20 1 have identified several Known Error Logs (KELs) that document
occurrences of varying forms of cash declaration discrepancies having
the ability to affect branch accounts reportedly due to system problems.
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 64
Number. 1_ Author: Gareth Jenkins Date: 23/10/2018 16:15:00
“We did Rave a major issue with this, that I was called in to investigate. I produced a paper at the time describing the issue. Iwould class this as a 3rd issue with HNG-X that
impacted accounts.
SSC (Anne) did a full analysis of all affected branches. Not sure exactly what happened as I wasn't involved in the discussions with POL as I had retired by then,
I've now found my notes on this. We checked back 6 months. I recommended going back further, but I don't know if this was done or not.
Number. 2_Author: Gareth Jenkins Date: 23/10/2018 16:25:00
When was this email thread. AS far as I am aware, this issue was identified in October 2075, though it was probably present for the whole lifetime of HNG-X, until the fix was
applied
Number. 3 Author: Gareth Jenkins Date: 23/10/2018 16:26:00
Not sure where they got that from. The issue was caused by forced lag off not correcily tidying up incomplete actions in the session. Itwas definitely a software fault and not a
process issue (though completing all actions would avoid the fault as would avoiding forced log offs due to timeouts).
Number. 4 Author: Gareth Jenkins Date: 23/10/2018 16:28:00
Gn what basis is this statement made?
My analysis clearly shows that it was a result of a forced Log off.
Number. 5 Author: Gareth Jenkins Date: 23/10/2018 16:29:00
“Agreed
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 49 of 225
5.21 KEL achai233J*! - Di
the amount received by the cash management system (SAP) were
epancies between branch cash declarations and
identified. In the data communication channel between Horizon and the
cash management system, an incorrect adjustment was being made
which added together the “branch cash in pouches overnight” figure to
what the branch has actually declared. The KEL states this is not a user
error or anything that can be corrected at branch level. Thidlis therefore
consistent with the problem being due to the existence of software bug.
5.22 KE acha1717T* - Whilst possible causes and a suggested process for
investigation of this issue by the support team is highlighted, there is
nevertheless an acknowledgement that there is the possibility that cash
declaration discrepancies could be due to an “Unknown system
problem”. It is worth noting what the Horizon support team consider a
discrepancy in this instance as:
"A discrepancy is the difference between the cash system thinks the
branch should have, based on a previous balanced figure and the
transactions recorded since, and the cash declared by the branch”.
5.23 There is also evidence of cash declaration discrepancies arising from clerks
duplicating remittance in transactions (“Rem-in”) because of wrong
messages being presented on the Horizon counter screen (act#he21P*),
This would result in incorrect cash amounts being declared. This is again
likely due to a bug because a software fix was applied on 12 January
2016. However, this was not a retrospective fix and therefore any
similar, previous discrepancies were not remedied. It is unknown how
many other Post Office branches were affected or if any communication
1 acha1233J.html, HNG-X KEL acha1233J, 22 June 2012 [POL-0039066]
[af95a66a92d17269b535e89644017582]
"2 acha1717T.html, HNG-X KEL acha1717T, 30 July 2010 (last updated 10 February 2015) [POL-0040033]
[5588ce13eb6fa9bbbdf986c912267fc0]
8 acha621P.html, HNG-X KEL acha621P, 15 October 2015 (last updated 14 January 2016) [POL-0040340]
[7518fc27f6357689¢500a302358d4452]
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
Page: 65
FUJ00183797
FUJ00183797
Number. 1_ Author: Gareth Jenkins Date: 23/10/2018 16:30:00
“This has no impact on Branch accounts. Itjust impacts the way in which POL's Cash Planning works and so may result in incorrect amounts of cash being sent to branches, AsI
understand it, a Branch can ask for more (or less) cash than planned and so any this bug would just increase the need for such manual requests.
Number: 2_Author: Gareth Jenkins Date: 23/10/2018 16:32:00
“Yes itis a software bug. However it has no impact on the Branch accounts.
,Number: 3 Author: Gareth Jenkit
Date: 23/10/2018 16:33:00
Not sure about this one. (Idon't specifically remember it) However, again it sounds like an issue that affects just cash planning and not the accounts, IFithad affected the
accounts, we would have done something about it
Number: 4 Author: Simpkins John_Date: 29/10/2018 17:49:00 Z
This KEL does not relate to any actual bug but gives advice on how to tackle a discrepancy call and what may cause it It lists the possible causes as:
Possible causes:
PM has made an incorrect declaration
Transactions as recorded on the system do not match what actually happened at the branch
Outstanding recovery
Withdrawn products
Known system problem (these should be monitored for, or be easy to spot from events etc)
Unknown system problem
The last point is just to state that the issue may be due to a previously unknown issue to ensure we investigate properly.
Number. 5 Author: Gareth Jenkins Date: 23/10/2018 16:35:00
Again I don't recognise this, unless it is the Dalmellington issue (which it could well be). This was after Iretired.
=, Number. 6 Author: Simpkins John _Date: 29/10/2018 17:52:00 Z
“ Rfter the Rem In slip is printed, the Remittances & Transfers Home screen should be displayed. If there was a system logout or inactivity logout earlier, before the user had
logged
on fully and all the post-logon checks completed, then the Rem in screen is displayed again after the Rem In slip has been printed. The user presses Enter to try to exit; each time
the pouch value is remmed in agein
PC0246997: Fixed in CTR_APP_X1288_V646, released to live via COUNTER_APP 77_7.
Roll Out to live estate commenced overnight Tuesday Jan 12th 2016.
From the KEL:
We have seen some outreach branches apparently resolve the problem by remming out the excess amount, but NBSC should advise on this.
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 50 of 225
was sent out by the Post Office to other branches to prevent further
occurrences prior to the fix being applied.
5.24 KEL ikAng3014S* reports an issue with how the Horizon system behaves
when a Subpostmaster makes multiple cash declarations and then runs
a trial balance report. The calculation displayed for the resulting trial
balance was incorrect. The support department was unable to identify
the root cause of the discrepancy although it was reported that a
correction could be made at the Post Office counter level by redoing the
cash declaration using the same amount already declared. The KEL
remains unresolved and is also cross referenced by Fujitsu with
MScardifield2219S*°, which identifies the underlying software bug being
caused by “cached data” not being updated via Riposte. This resulted in
incorrect data being presented in any discrepancy, variance and balance
reports. It is reported that the problem should clear itself overnight
and/or a manual workaround was possible. In either case this scenario
is in my opinion likely to be confusing for the Subpostmaster and could
lead to them making modifications which are unnecessary if they are
unaware that the problem should clear itself overnight. It was stated in
November 2007 that a fix was being piloted with a view to it being
released in January 2008 but it is unknown if this took place and/or was
successful.
5.25 Errors within cash pouch delivery could have also affected branch
accounts. In DSEKidon5426P* a failure in pouch delivery resulted in a
cash gain when the Subpostmaster carried out a branch cash
declaration.
* LKiang3014S.html, HORIZON KEL LKiang3014S, 27 November 2002 (last updated 22 February 2007)
[POL0035520] [19f87982637c6851bf104504935dfd39]
45 MScardifield2219S.html, HORIZON KEL MScardifield2219S, 15 July 2005 (last updated 27 November 2007) -
(POL-0035721] [657dca342292aleaee72e1f7f34a4582]
4 DSeddon5426P.html, HORIZON KEL DSeddon5426P, 26 June 2006 (last updated 10 October 2006)
[POL0035379] [ce2e6481384a7fe307e96bbc62f36048]
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 66
Number: 1_ Author: Gareth Jer
Date: 23/10/2018 16:37:00
“Wotan issue am familiar with, Even if not fixed in 2008, it would clearly have been fixed by HNG-X in 2010.
Number: 2_ Author: Simpkins John Date: 29/10/2018 18:20:00 Z
Yes, this is justa declaration issue.
Number. 3 Author: Gareth Jenkins Date: 23/10/2018 16:39:00
This sounds like a problem when we first introduced auto rems and so presumably was fied. I guess we need to dig out the associated Peak,
I don't recognise it.
Number: 4 Author: Simpkins John_Date: 29/10/2018 18:24:00 Z
Peak -PCO136963 - Basically the pouch auto REM IN failed, leaving the office with a large gain. A retry succeeded, Clone Peak PC0136990. The underlying issue is remming the
pouch in whilst in ‘Price shopping’ mode. A fix was created and released in COUNTER_EPOSS 36_1 in May 2007.
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 51 of 225
5.26 In aclfh194L 47 a problem affecting around 15% of kiosk branches
prevented these branches from being able to automatically make cash
declarations. The message being presented (Code: 2303 - Incorrect
Transaction Amount) was misleading since the totals were in fact correct
and the transaction amount checks were failing due to incorrect data
types. A bug fix was proposed for the second half of 2015, but it is
unknown if this took place and what further communications went out
to affected branches.
5.271 am also aware from the witness statement of Pail Stubbs** prepared for
the Common Issues trial that other potential problems with kiosk
operations could have potentially affected branch accounts.
Bureau & Decimal Point Issue
5.28 Post Office has disclosed a PEAK which highlights a fault with bureau
transactions where an omitted decimal point in the “euro” line caused a
discrepancy. In this particular PERK * the Subpostmaster declared
€57,865.00 (expressed as “euro 5786500”), but the sterling equivalent
was £0.00 (expressed as “0.00”). When the Subpostmaster tried to
reverse the transaction, the figure doubled. It was noted in the call log
this problem had been seen on a couple of other occasions and caused
significant accounting problems. It was due to be fixed as part of the
S80 release.
5.29 A detailed PEAK log is available at Appendix A.
Reference Data Errors
5.30 Reference Data is changed frequently within Horizon, I understand at fast
once per day. Post Office reported that until 2017°° the process of
* achai94L.html, HNG-X KEL acha194L, 18 November 2014 (last updated 01 April 2016) [POL-0040058]
[9a134a453319f4b473b37070dd71b467]
48 Pam Stubbs (174) - Witness Statement and Exhibit.pdf
* pC0098844.html, Peak PCO098844, 6 February 2004 [POL-0270879] [050c1d940ddd970bf7ed304afc494faf]
5° Operations Board 21 July 2017.pdf, Operations Board 21 July 2017, 21 July 2017 [POL-0221328]
[9c45e0be3ff2b6773447cc6e41db5f46]
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 67
Number: 1_ Author: Gareth Jenkins Date: 23/10/2018 16:41:00
“ Rgain I don't recognise the issue. Not sure I understand how a kiosk issue impacts declarations
,Number.2_Author: Simpkins John _Date: 29/10/2018 18:36:00 Z
Missing cash declarations from kiosks means no info is sent to POL about any cash held at the branch, and can impact cash management and deliveries. Though since kiosks
declare a till within a shared stock unit, itis unlikely that all tills will be missing, Affecting around 15% of kiosk branches each day.
PC0238702, issue was a precision problem due to double datatype. This only impacts cash management and deliveries, it does not cause discrepancies.
BIF approved the fix on 01-12-2014, released in R11.72, applied to live on 26-09-2015,
z Number. 3 Author: Gareth Jenkins Date: 23/10/2018 16:42:00
We've not seen this one.
Number. 4 Author: Gareth Jenkins Date: 23/10/2018 16:43:00
Not aware of this one. Looking at the date, then this was soon after Bureau first went live,
Number: 5 Author: Simpkins John_Date: 31/10/2018 11:59:00 Z
The spotrate was incorrect and very large which meant that the stearling conversion was too big to be held by the system. This didn't actually cause an issue, however there was
also a problem with reversing a revaluation which caused a payment/receipt mismatch.
Number. 6 Author: Gareth Jenkins Date: 23/10/2018 16:45:00
Somewhat excessive. We do support emergency ref data changes during the day, but in practice ref data is usually only updated overnight
Often the changes are very minor.
Number 7 Author: Simpkins John Date: 31/10/2018 12:03:00 Z
Speaking to Kevin he thinks we have only done an emergency change during the day three times.
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 52 of 225
chdging Reference Data was conducted without being subject to formal
controls. Evitence suggests that Reference Data has been found to be
incorrect within Horizon and that this led to discrepancies within branch
accounts. Within the document, ‘Counter Type X Reference Data
Definitions’>!, it is explained that “Cobhter Type Reference data... will
usually only change on the back of a Change Proposal or Live PEAK fix”.
Examples have been found in KEL/PEAK records of Reference Data
hang an impact upon daily counter activities (DSeddon314Q*?). These
vary in severity and these errors are passed to the data reference team
for rectification.
Supporting Documentation
5.31 Errors in Reference Data and/or Reference Data validation has affected
counter activities in several different ways. By Evay of an example in
johnbascoG5222H™ an Automated Payment transaction was reported
as having failed due to “Unknown Agent Code 3046”. Client account
code 3046 was found to not exist in Reference Data and the fault was
not reproducible when the problem was analysed and tested. It was
acknowledged that, due to the business impact, a fix would be provided
to check and validate the client account code exists in Reference Data
before the transaction is committed.
5,32 KEL achai0L** documents how branches were unable to accept cards for
rent and council tax payments due to incorrect Reference Data, although
in this circumstance no Fhpact on the branch accounts in terms of
discrepancies would occur as the cards were just rejected completely.
51 DESGENSPE0003_2.doc, Counter Type X Reference Data Definitions, 21 October 2010 [POL-0118364]
[ac2409e9e7be10cf2ca62c338b8bd82c]
52 DSeddon314Q.html, HORIZON KEL DSeddon314Q, 14 March 2016 [POL-0035128]
[12a0d196f1ba8968780b9eb845fa575f]
53 johnbascoG5222H. html, HNG-X KEL johnbascoG5222H, 11 July 2017 [POL-0040923]
[05ee26859196eff653a7830d82da7db1}
5* achalOL.html, HORIZON KEL acha10L, 01 April 2009 [POL-0036378]
[ed3160ac33e6f72cde5387980224249b]
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
Page: 68
FUJ00183797
FUJ00183797
Number: 1_ Author: Gareth Jer Date: 23/10/2018 16:46:00
“WE certainly had controls. I can’t speak for POL, but I thought they did too.
“Adam should be able to provide a detailed description of the Reference Data controls in place and the automated checking performed.
Kevin to address?
Number. 2_Author: Simpkins John_Date: 31/10/2018 12:05:00 Z
,Number: 3 Author: Gareth Jenkins Date: 23/10/2018 16:47:00
What evidence?
Clearly errors in ref data can cause serious issues.
There was the "Highland authority” issue (20127) which stopped Banking working and caused chaos in sorting out data for AP clients, but I don’t think it impacted Branch
accounts.
Number: 4 Author: Simpkins John Date: 31/10/2018 12:06:00 Z
“Agreed no impact upon branch accounts
Number. 5 Author: Gareth Jenkins Date: 23/10/2018 16:49:00
Thatis correct. This is ref data that controls the behaviour of counter code and has nothing to do with POL.
Number: 6 Author: Gareth Jenkins Date: 23/10/2018 16:50:00
Yes, but not necessarily impacting the accounts.
. Number: 7_Author: Simpkins John_Date: 31/10/2018 12:08:00 Z
This was due toa code error that could be worked around by a reference data fix, PCO133467 details this. Fix released to T1011
z,Number: 8 Author: Gareth Jenkins Date: 23/10/2018 16:52:00
Yes, this would cause an inconvenience, but it would be clear that there was an issue and
the branch accounts would not be affected
Number: 9_ Author: Simpkins John_Date: 31/10/2018 12:16:00 Z
Agreed
Number: 10 Author: Gareth Jenkins Date: 23/10/2018 16:53:00
‘Again an inconvenience, but no impact on accounts
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 53 of 225
5.33 KEL mvPlight1458Q 55 illustrates where withdrawn products (with
withdrawn Reference Data) can affect a branch accounting position
because the Subpostmaster will have products that cannot be accounted
for as there is no remaining reference for them to later declare that
stock item in the accounts.
5.34 Identified in KEL wb#hs35395, the customer was charged twice for the
same transaction which was reported to be a side effect of errors within
Reference Data.
Duplicate Transactions
5.35 There is evidence of duplicate transactions existing within Horizon due to
incorrect processing.
5.36 The error appears to have occurred when payment data was being
harvested prior to being transmitted to an external client. The
harvesting process should have identified duplicate transactions and
dealt with these appropriately. However, any disruption in service can
cause the harvesting process to send duplicate payment transactions.
5,37 This would result in a discrepancy between poBl and the external client
where client summary payment files were submitted to an external
client prior to the duplication issue being noticed.
Supporting Documentation
5.38 GMaxwell3651K*” and surs357P** both demonstrate that any failures or
interruptions in service with the harvesting process can cause duplicated
payment transactions to be processed. In the latter case Streamline,
the external third party, received a payment file with 835 duplicate
55 MWright1458Q.html HORIZON KEL MWright1458Q, 15 June 2000 [POL-0035611]
[2cf4ea95b637e9acea61f7c77 1ffb355]
5° wbra5353).html, HNG-X KEL wbra5353J, 10 April 2014 (last updated 17 April 2014) [POL-0039777]
[abac77a5158c65e27b40fc528d985bf4]
57 GMaxwell3651K.html, HORIZON KEL GMaxwell3651K, 22 December 2004 (last updated 8 May 2006)
[POLO035194] [654f590f7d2119bf963a3216607311cf]
5° surs357P.html, HORIZON KEL surs357P, 3 April 2009 (last updated 6 April 2009) [POL-0036388]
[4404b260b27528c3caa99a6ac8aaft3a]
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 69
Number: 1_ Author: Gareth Jenkins Date: 23/10/2018 16:54:00
~ Rgree that this was an issue ~ especially then as stock was held by Value and not volume
However didn’t this just result in either a RP issue or a one -off discrepancy as the value was moved to cash?
Number: 2_Author: Simpkins John Date: 31/10/2018 12:31:00 Z
Originally we had to help do the conversion to cash:
“The message store change will involve sending 2 SAP messages to the counter to convert £6.30 cash into stamps; which will bring the negative stock back up to zero. (See
PC0067338 for example). This will give rise to @ payment and
receipt mismatch error when the office next rolls over”
However this changed and they would just get a discrepancy to cash afterwards which they would need to discuss with POL.
* UPDATE:14/06/2007 Above action is no longer necessary. If a product or products have not been remmed out within the time then the system will remove the product/s next
time the stock unit is rolled over. This will also cause a cash discrepancy of the equivalent sum. PM will have to contact NBSC to resolve the discrepancy.”
Number: 3 Author: Gareth Jenkins Date: 23/10/2018 16:55:00
Tdon't recognise this.
Number. 4 Author: Simpkins John Date: 31/10/2018 12:38:00 Z
This is kiosk (HBS), he is referring to a comment in the KEL for Peak PCO233296.
“dated 14/04/2014 for Branch 008113 - was also another example of 1201 - Invalid Basket State seen for Kiosk 67, where a customer was charged twice £5.98, for the same tn as
two RTSTendAdd’s were seen in this instance."
Your comment in the Peak
The root cause of this issue is the fact that there was invalid Reference Data for Singapore, San Marino and Vatican City, which resulted in an empty AddOn Element being
retumed to the kiosk and it then got confused and repeated the use of the message ID."
It was identified by the DRS reconciliation report. However in the Peak is states that the first settlement would have failed due to business rules and the customer would NOT
have been charged twice.
Number: 5 Author: Gareth Jenkins Date: 23/10/2018 16:56:00
So no impact on the branch accounts (unless POL tried to send a TC to the branch as a result).
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 54 of 225
payment transactions totalling £76,564.53. In Bhis instance Streamline
identified the duplicates and prevented payment processing. However,
identification occurred after the transaction data had been transmitted
from Horizon and despite any processes designed to prevent this.
Failed Recoveries
5.39 The recovery process is outlined at section 4.54 above and the importance
of this and the error scenarios where recovery might apply are further
highlighted in a report by Gareth Jenkins.°° Any failure in the recovery
process can affect transaction data integrity and to attempt to mitigate
against this a failed recovery report exists to set out any instances
where this has occurred.
5,40 The witness statement of Angela Burke® documents a revery process
failure within Horizon. Communication issues within Horizon resulted in
a transaction that authorised and processed at the counter failing to
recover within the Branch’s accounts. This subsequently resulted in a
discrepancy that Mrs Burke had to seek to reclaim from Post Office in
order to balance the discrepancy.
Supporting Documentation
5.41 The Horizon KEL jharr832S* acknowledges that the revery Process is a
complex area:
“cre needs to follow POL business rules properly. It is a complex area
and various factors can affect whether recovery happens or not depending
on exactly when it crashes and what the clerk replies”.
5.42 KEL cardca64aqe reports the difficulty the clerk may have faced when
trying to process recoverable transactions. In this case, recovery was
®° 1,9 9. Horizon Online Data Integrity for POL, Gareth Jenkins.pdf, Horizon Online Data Integrity for Post
Office Ltd,2 April 2012 [POL-0021989] [0690d38ae4fld9c949aaf1618c732d06]
6° Witness Statement of Angela Burke 28.09.2018.pdf
© jharr832S.html, HORIZON KEL jharr832S, 05 March 2007 [POL-0035531]
[4f4c42852071f3957a69eb4a6dbceb8e]
® carde464.html, HNG-X KEL cardc464Q, 30 April 2010 (last updated 12 January 2011) [POL-0038234]
[24fe6e3e3901ace35658ba1d3bdc420e]
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 70
Number: 1_ Author: Gareth Jenkins Date: 23/10/2018 16:58:00
Even if they hadn't done so we would have picked it up in reconciliation the next day.
NB Streamline picked it up due to checks on file IDs. What I think happened was the same file was sent twice and the interface is
designed to detect such things, so this is an example of checks and balances working
No impact on Branch accounts (even if Streamline hadn't picked it up ~ any impact would have been on customers being double charged - as Charles McLachlan describes with
his Sainsburys card).
_ Number: 2_Author: Simpkins John_Date: 31/10/2018 15:24:00 Z
‘Agreed, no impact, it was a database check point issue. PCO178736 applies
* On 30th March 2009 we sent 2 payment file to Streamline which contained 835 txns that have already been sent to Streamline on 23th March. The Original tens were
acknowledged by Streamline and processed correctly by our reconciliation service and the txns were in the correct state.
Due to the harvester agent picking up the wrong ‘check point’ these 835 txns were harvested again and included in the payment file for 30th March.
Streamline have not acknowledged these duplicate txns.”
“I have spoken to the merchant liaison team/Streamline.
They confirm that on 30 March at 1651 they received a payment file with 835 duplicated transactions the total value of which was £76,564.53. These transactions have been
stripped from the file
Streamline have contacted P&BA who have instructed streamline not to process the duplicates. Customer accounts have not been debited twice."
Number. 3_ Author: Gareth Jenkins Date: 23/10/2018 17:03:00
‘Agreed. However the failed recovery process did pick this up. At this stage it isn't dear exactly what was communicated between Fujitsu and POL and where thigs went wrong in
the process.
I've asked for this to be investigated further
Number. 4 Author: Simpkins John_Date: 31/10/2018 15:28:00 Z
Yes this was picked up and normal process followed successfully. Peak PCO25 1333 applies,
Number: 5 Author: Gareth Jenkins Date: 23/10/2018 17:04:00
This KEL relates to old Horizon, Recovery after failures is always going to be a tricky area. The problems in old Horizon and HNG-X are very different
Itis worth keeping the two areas separate,
Number: 6 Author: Gareth Jenkins Date: 23/10/2018 17:06:00
~ Twould support this statement
In any system, if it fails, then recovering exactly what occurred at the time of failure is likely to be difficult. The design of the recovery process is to make the rules as simple as
possible so that the SPMR has the maximum chance of getting it right. However there are some cases where the only person that knows exactly what happened in the real world
is the SMPR and we need to ask them.
Any system has to allow for recovery. You can never guarantee comms line stay up, or power cuts don’t happen at the most inconvenient times.
Number: 7 Author: Gareth Jenkins Date: 23/10/2018 17:09:00
Now back to HNG-X
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 55 of 225
attempted but failed. Whilst the recoverable transaction appeared on
the faillla recovery report, it is now out of the hands of the
Subpostmaster. In Bis instance recovery failure had no impact on
branch or customer accounts as settlement had not written to the BRDB.
Had recovery run successfully, a zero-value transaction would have
been written to the database which should not affect branch accounts
as it is simply a record entry with zero financial value.
5.43 KELs seng2037L & acha959T’”! describe how various transactions states
may also indicate a failed recovery or an incBmplete transaction
awaiting recovery.
5.44 dsed4733R” provides an example where transaction recovery failed due
to a wrongly named recovery script. This was a Horizon system error
arising because of incorrect Reference Data that had to be corrected by
the Reference Data team. It il unknown when this was corrected but,
in the meantime, sBc would need to have cleared the failed recoveries
daily.
5.45 There appears to be a high risk of faild recoveries arising because of a
failure to follow correct procedures or lack of understanding of the
recovery process. Thal failed recovery report should pick up these
recoverable transactions but by this point it is no longer something that
can be resolved at the counter.
5.46 There are a number of PEAKs’? PC0203676, PC0263451, PCO266575 &
PC0273046 (detailed PEAK logs available in Appendix A) which identify
recurring failed communications issues subsequently reshting in failed
recoveries and consequently branch discrepancies. In these situations,
manual reconciliation was required, and Fujitsu needed to clear the
failed recovery transaction.
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 71
Number. 1_ Author: Gareth Jer Date: 23/10/2018 17:10:00
This is something that was always designed to enable us to sort things out when recovery did occasionally go wrong, There will always be times when the automated systems
have to give up and rely on manual processes. The key thing is that the manual processes are invoked and work.
Number 2_Author: Gareth Jenkins Date: 23/10/2018 17:10:00
“So what is the problem
,Number. 3_ Author: Gareth Jenkins Date: 23/10/2018 17:12:00
We have never said that recovery is unnecessary.
Number 4 Author: Gareth Jer
Can we find out?
Date: 23/10/2018 17:13:00
Number. 5 Author: Simpkins John_Date: 31/10/2018 15:34:00 Z
PC0227260 applies. This reference data is provided by Post Office.
Number 6 Author: Gareth Jenkins Date: 23/10/2018 17:13:00
This is part of their job and shows that the processes are working.
Number: 7 Author: Gareth Jenkins Date: 23/10/2018 17:14:00
No. Failed recoveries are usually due to further comms / data centre failures during the recovery process.
Not following the correct process during the recovery isa different issue and may lead to discrepancies if incorrect assumptions are made by branch staff.
Number: 8 Author: Gareth Jenkins Date: 23/10/2018 17:15:00
Agreed
Number: 9 Author: Gareth Jenkins Date: 23/10/2018 17:16:00
This again shows the process working
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 56 of 225
Failed Reversals
5.47 Reversals are set out in Section 4.61. In summary, they are the ‘undoing’
of transactions to reverse the value of it.
79 seng2037L.html, HNG-X KEL seng2037L, 1 February 2013 (last updated 7 February 2013) [POL-0039307}
[bb773d730ae5943330329bc96d2a0fac]
7 acha959T html, HNG-X KEL acha959T, 28 February 2010 (last updated 19 October 2017) [POL-0041091]
[698d3dbfe4181592c650af60f92cla1 1}
”2 dsed4733R.html, HNG-X KEL dsed4733R, 25 July 2013 [POL-0039482]
[78f45c50b543fb3673d9a1 8fe442eb37]
73 PC0203676.htmi, Peak PC0203676, 31 August 2010, [POL-0373467]
[9c65a8e1e33e636bc6ae37 2aefed3690], PCO263451.html, Peak PCO263451, 19 October 2017, [POL-0430967]
[981b785aa6f78e75a5f19c8a2c70aff5] PCO266575.html, Peak PCO266575, 26 January 2018, [POL-0433904]
[8d02a629313fa13f86f853d49e55dcc7] PCO273046.htmI, Peak PCO273046, 15 August 2018, [POL-0439981]
[c4330F4fcc4ea9137 368640692730:
5.48 Whilst there is ev
of reversals (discussed in Section 4 above), it is observed that there
ce of failures in respect of the electronic processing
were also issues with interpretation and idePhitication of system
reversals.
5.49 The document (“Helen Rose report”) ® refers to an incident where a
Transaction Correction was issued which the Subpostmaster duly settled
financially despite the Subpostmaster denying conducting the reversal.
5.50 Thal report appears to show that the material that Post Office initially
reviewed did not identify that it was the system that initiated the
reversal rather than the Subpostmaster and therefore the Transaction
Correction making the Subpostmaster liable was issued in error. Since
this is effectively a failure to appropriately reduce the risk of error this
is also dealt with further at Section 5.167.
Supporting Documentation
5.51 There is evidence that there were software issues resulting in Horizon
applying the wrong mathematical sign when reversing transactions (i.e.,
a plus (+) rather than a minus (-)). This was identified in
® Horizon data Lepton SPSO 191320, 12 June 2013 [POL-0221677, POL-0221678, POL-0221679, POLO221680,
POL-0221681] [f296f6880e1b8418f37d3e344374c42a]
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 72
Number: 1_ Author: Gareth Jenkins Date: 23/10/2018 17:17:00
Tdon’t remember seeing evidence of this,
»Number: 2_ Author: Simpkins John Date: 31/10/2018 17:04:00 Z
Agreed, 4.55 to 459 discusses reversals but shows no evidence of failures.
Number. 3 Author: Gareth Jenkins Date: 23/10/2018 17:18:00
Tsuspect that this is referring to “recover reversals” which are necessary in order for the system to be able to predict the behaviour of the SPMIR when the system fails
Basically, the assumption is that any Basket which has not successfully been processed is assumed at the counter to have failed and the SPMR should act accordingly.
If on recovery, it is found that the basket had actually processed, then the Basket is reversed since that is what the clerk should has assumed.
» Number 4 Author: Gareth Jenkins Date: 23/10/2018 17:27:00
Tseem to remember suggesting that the counter mode was made available to POL as well as the Txn Mode which would achieve this, I don't know if anything was done about it,
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 57 of 225
pslled2847N*. In this instance, the issue resulted in the dolbling ofa
remittance value in transactions that should have been reversed,
effectively to zero. It is unclear when this software bug was fixed or how
widespread the problem was.
5.52 KEL cardc5756N® provides an example where the system failed to reverse
all lems in a multi-line pouch and only the first item was reversed. It
was reported that the clerk had appeared to have followed the correct
process, but the problem could not be reproduced on a test counter. A
Transaction Correction was required to reverse the remaining
items. The xe remains unresolved and it is not known whether any
further examples of this were reported.
Uncategorised Bugs/Errors/Defects
5.53 There are various KELs and PEAKs relative to errors that have not been
categorised as those above. sirMlarly, no cause or resolution is
identified within the KEL to assist with further determination.
Supporting Documentation
5.54 Foreign currency discrepancies were noted in G6Cmpson1049L. 6 All
currencies on hand doubled up following successful balancing eight days
previously. The KEL record stated the issue as being “under
investigation” but there is no detail of the results of any analysis.
5.55 There is evidence that in certain investigations there was insufficient
diagnostic data to be able to fully diagnose an issue. In KEL
MHarvey35271 *” an apbdrent successful transaction failed to be
validated due to a change of mode. It was reported that the change of
© PSteed2847N.html, HORIZON KEL PSteed2847N, 28 April 2003 (last updated 20 June 2003) [POL-0033658]
[1912a565b4a3c4d601bd18b62b15ce04]
85 cardc5756N.html, HORIZON KEL cardc5756N, 19 February 2008 [POL-0035846]
[6f87ead560bbed157b55348b32214c73]
8 GCSimpson1049L.html, HORIZON KEL GCSimpson1049L, 29 April 2004 (last updated 5 May 2004)
[POL0034206] [77ee561a4d4b623b80b2fc1626Fice9e]
67 MHarvey35271.html, HORIZON KEL MHarvey35271, 21 January 2005 [POL-0034494]
[82a49bbb9951410fedf65d51e314a091]
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
Page: 73
FUJ00183797
FUJ00183797
Number. 1_ Author: Gareth Jenkins Date: 23/10/2018 17:29:00
‘This is old Horizon and from a long time ago. I'm not familiar with the issue or the fix
Number: 2_Author: Simpkins John_Date: 31/10/2018 17:12:00 Z
Peak PCO089918 applies. Fast track fix was applied
Fix released in WP16353 (B3S30R)
the software fix was sent down and became
active on 07-May-2003,
a,Number: 3_ Author: Gareth Jer Date: 23/10/2018 17:30:00
NB Rem Transactions shouldn't be reversible, so perhaps that is what the problem was (ie this wasn't being policed correctly!)
Number: 4 Author: Simpkins John Date: 31/10/2018 18:55:00 Z
PCO089978 applies. It was a bug related to the signage of reversals. A fix was released.
Applied to Calthorpe on 06/05
,Number: 5 Author: Gareth Jenkins Date: 23/10/2018 17:31:00
‘Again not sure how this was allowed ~ unless it was a “Cancel Pouch” that went wrong,
Problem would certainly not have carried forward into HNG-X
Number: 6 Author: Gareth Jenkins Date: 23/10/2018 17:32:00
We should close all KELs relating to Old Horizon.
. Number: 7_Author: Simpkins John_Date: 31/10/2018 18:56:00 Z
Agreed this relates to old Horizon. Talso cannot find any other occurrences in Peak
Number: 8 Author: Gareth Jenkins Date: 23/10/2018 17:33:00
Do we update KELs when a fic has gone live? I didn’t think we did.
Should we?
Number. 9_ Author: Simpkins John_Date: 01/11/2018 13:35:00 Z
We should but do not always do so due to cloned calls,
Number: 10 Author: Gareth Jenkins Date: 23/10/2018 17:33:00
‘Again, sounds like the early days of Bureau
Can we check out the Peak?
Number: 11 Author: Simpkins John_Date: 01/11/2018 13:37:00 Z
PC0102259 applies. However this has no further investigation as the logs were not collected from the counter before they were deleted (1 week back then) and it was closed
Number: 12Author: Gareth Jenkins Date: 23/10/2018 17:35:00
This sounds like it was an issue on the TPS / TIP interface, in which case there would be no effect on the counter. Worst case would be the incorrect mode might affect the Cash
‘Account mappings, in which case there may have been a R&P issue.
Ref Data issue?
Number: 13 Author: Simpkins John Date: 01/11/2018 13:43:00 Z
Yes this was a TIP file rejection due to the wrong mode. However the transaction was later reversed at the branch:
What appears to have happened is that an apparently successful transaction
for £20.55 took place on counter 1 (APSSEQ:018 143) @ 09:01. This was followed
by an failed attempt to transfer the session to counter 2 and subsequent
desktop restart, APS fallback recovery was selected and the previous
transaction for £20.55 recovered (APSSEQ:018144) @ 09:48. This transaction
has a Mode:SC and a Blackbox Version:3 (which is for mode 14). This was
harvested correctly but failed product/version/mode validation by TIP.
Incidently this transaction was reversed later at 11:25,
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 58 of 225
mode that had occurred could not be explained as the APS code would
not allow this and there was no other way to diagnose the issue. The
associated PEAK (PC113202) was closed as no further progress could
be made and it was treated as a “one off”.
5,.56In coeng1123Q% 69 unexplained discrepancies (gains) for different stock
unit types (Cash and Stamps) was reported. HoBkon system memory
faults appear to be the suspected cause in this case, but no suitable
alternative causes or factors were analysed, and the incident remained
unexplained. No advice is detailed to have been provided to the
Subpostmaster.
5,57 There are a number of PEAK records (i.e., PC0063723 & PC0084116”°)
which were believed to be system related bugs causing discrepancy
transactions to be calculated twice. The examples highlighted both refer
to KEL prbWwe1625K which (according to the log in PCO084116) was
never properly resolved. It is also worth noting that there are many
more PEAK records associated with this KEL and the examples
highlighted span a period from March 2001 to November 2002. This
appears to be an issue that has never been fully resolved despite having
been passed to development to analyse further. There appears to have
been a workaround put in place, but it is not known how the workaround
was communicated and if or when the underlying cause was ever
resolved. A detailed PEAK log is available at Appendix A.
5.58 Similarly, in PEAK pcobb78877! a known but unresolved software error
caused a doubling up of values in cash account periods. It is not clear
6° CObeng1123Q.html, HORIZON KEL CObeng1123Q, 14 August 2000 (last updated 15 January 20014) [POL-
© ] [cceaa39cee86736e8a4735e44ae328ac]
7° PC0063723.htmi, Peak PCO063723, 10 March 2001, [POL-0238257] [1efaafc05039eea7aSe5e09d1d50226c]
& PCO084116.html, Peak PCO084116, 23 November 2002, [POL-0256970]
[fe868322664da1770b9416c6443bb468]
7. PC0027887.htmi, Peak PCO027887.htmi, 21 July 1999 [POL-0221773]
[93af66e221ecfde6fcaad8aSaci4eca4]
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
Page: 74
FUJ00183797
FUJ00183797
Number: 1_ Author: Gareth Jenkins Date: 23/10/2018 17:37:00
“Old Horizon, so clearly not an issue now.
»Number: 2_ Author: Gareth Jenkins Date: 23/10/2018 17:37:00
Tknow Riposte did have memory issues occasionally.
Number. 3_Author: Simpkins John_Date: 01/11/2018 13:50:00 Z
“ PC0037445 applies and the busy exe monitor does report that the system is out of memory:
Busy.exe reported that Desktop.exe had used up
all available VM. This is therefore not one of the usual Infractructure VM
problems, but is some kind of EPOSS / Desktop Issue or software bug. After
discussion with Walter am passing it to EPOSS-FP where we will look at it
together when we get time.
Number. 4 Author: Gareth Jenkins Date: 23/10/2018 17:39:00
We had better have a look at this and associated Peaks,
Number. $_ Author: Simpkins John_Date: 01/11/2018 13:51:00 Z
“This KEL is deleted, however I have made comments in on Peak PCO063723 below and a fix was produced.
Number. 6 Author: Gareth Jenkins Date: 23/10/2018 17:40:00
Probably needs checking out.
Number: 7 Author: Simpkins John Date: 01/11/2018 13:58:00 Z
“The peak identifies @ corrupt EPOSSSettlement.dil on the counter:
These were due to a stock transfer for £12,000 which was not settled
correctly due to the presence of a corrupt EPOSSSettlement.dll file on the PC
involved. The result of this was that the transfer out could not be accepted
by the receiving stock unit, leaving a Cash Account imbalance in CAP 10 of
twice the value, The effect of this error in CAP 10 was to leave the overall
balance due to PO £12,000 short causing the 88F for CAP 11 to be short by
this value and hence the £12,000 imbalance in CAP 11
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 59 of 225
from the call logs how Fujitsu resolved the branch discrepancies and the
PEAK was closed after 12 months with the following comment:
“cloBhg call on basis of insufficient evidence. As this is such an old call I
have not contacted the call originator. I suggest that this call remains
closed!”
5.59 In PEAK PC02031317 differences between volumes and values in a branch
office snapshot was identified as a bug in Horizon carried forward to
Horizon Online. sirkle this was a pre-migration bug (as acknowledged
by Gareth Jenkins) the PEAK was closed. It Pay be that this issue was
resolved in Horizon Online but this cannot be confirmed on the existing
evidence.
Hardware Errors
5.60 Throughout the lifespan of Horizon, it is observed (particularly through
Second Sight reporting commissioned by Post Office’?) that counter
equipment was unreliable. This observation was reinforced in a Post
Office IT risk register document” which identified branch IT hardware
as being very old and requiring urgent replacement.
5.61 The issues are broadly categorised as:
i. Printer failures
ii. Screen misalignment (pressing one on screen button but
resulting in the system selecting a different function)
iii, Faifd communications links
72 PCO203131.html, Peak PCO203131, 18 August 2010 [POL-0372925] [6ea70fc9c6b34cbcd61b2a7b2ddb2628]
73 POL Interim Report Signed.pdf, Interim Report into alleged problems with the Horizon system (Page 6), 08
July 2013, [POL-0022308] [8dd44e3ficc26efcic27e09ee960d737]
74 TT risk register 2011 09 19 updated.xls, v1.2 ~ Risk Ref: 29, [POL-0219381]
[ec517091d83be38a59b167f5cffa02ad)
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 75
Number: 1_ Author: Gareth Jenkins Date: 23/10/2018 17:39:00
~ Not very helpful, but can understand it)
Number: 2_Author: Simpkins John_Date: 01/11/2018 13:59:00 Z
Earlier in the Peak it states that the cause of the discrepancies was communicated to POL:
17/08/1999 15:32:16 - By Nicole Meredith - MSU
‘A RED report was issued for this incident and I am awaiting confirmation for
this call to be closed.
POL were not happy with this and wanted further discrepancies investigated:
Julie Dart (POCL TP) is unwilling to close this call. She does not believe
that the Receipts and Payments misbalance was caused by the B/F figure
doubling up. Please investigate why this misbalance occurred in CAPS
9,10,11,12 and 13 and provide evidence if possible.
These were also identified to the same cause:
The discrepancies for CAP 10 and CAP 11 are related to the transfer incident
covered by two previous calls, E-9906020405 & £-9906140099,
Number 3_Author: Gareth Jenkins Date: 23/10/2018 17:41:00
Tooking at the date, it must have been one of the last Horizon counters. Last one migrated in September 2010
Number. 4 Author: Gareth Jenkins Date: 23/10/2018 17:41:00
No counter bug in old Horizon would carry forward to HNG-X as the code was totally different.
Number. 5 Author: Gareth Jenkins Date: 23/10/2018 17:42:00
Agree, We purchased the counter PCs in 1996 and they have only just been replace in the last couple of years,
I think this is one for POL to address as they refused to pay for a counter refresh.
Number. 6 Author: Gareth Jenkins Date: 23/10/2018 17:44:00
Thatisn't down to age of equipment, but to the reliability of phone lines,
Number. 7 Author: Simpkins John Date: 01/11/2018 14:06:00 Z
We also rely upon 3rd parties in this area,
Is it worth mentioning that we had ADSL with backup via ISDN and GSM in HNG-X?
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 60 of 225
5.62 A detailed selHhelp manual 7° 7° for troubleshooting and replacing
peripheral hardware equipment (keyboard, printer, monitor and PinPad)
appeared to be available for Subpostmasters. However it is not known
how widely this was distributed to Subpostmsaters and how often it was
updated.
5.63 A spét Review report’? conducted by Second Sight Support Services
Limited gives further credence to the possibility that hardware issues
could have been responsible for branch discrepancies (shortages and
surpluses). In the example highlighted in the report, following a sudden
spike in branch discrepancies the Subpostmaster implemented rigorous
control improvements including performing twice daily balances and
installation and review of CCTV and CCTV film. Despite this, no root
cause was ever identified and there was no evidence that POL had
conducted any root cause analysis. The Subpostmaster had suspected
that faulty Horizon equipment could be the cause, but no firm evidence
was ever provided. The Subpostmaster had asked for replacement of
old hardware, but this had been refused by Post Office. Subsequently
the branch was flooded, and hardware was replaced due to flood
damage. Following the hardware replacement and the reopening of the
branch with the same staff and processes no further shortages occurred
and all transactions balanced. If ais evidence is correct, it would be
consistent with the Subpostmaster’s view that suspect faulty hardware
was responsible for causing his shortages.
5.64 Second Sight also considered hardware issues in their report’® and were
unable to draw a firm conclusion on whether faulty equipment could be
75 Self Fix Manual final.docx, Peripheral Trouble Shooting and Replacement Guide, 8 December 2011, [POL-
78 9] [aaf6al 6dfa038ee20e77bct4bb26f95d]
7” 6.8 58. Spot Review 25 — Paul Popov - Mysterious shortages ~ v4.pdf, Horizon - Spot Review,
[5852c951bde464932c83764d1 a0dadb1 ]
7 0.2 11. Second Sight ~ Briefing Report Part 2 (final).pdf, Initial Complaint Review and Mediation Scheme -
BRIEFING REPORT - PART TWO (Page 45 - 23. Hardware issues), 09 April 2015,
[3b79161b842d035f9b952bf70eb9433b]
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 76
Number: 1 Author: Gareth Jer Date: 23/10/2018 17:44:00
Wasn't this something POL did to reduce calls to Help desk? Or was it a CS initiative?
Number: 2_Author: Gareth Jenkins Date: 23/10/2018 17:49:00
Twas involved in a number of Spot Reviews. However my records show I didn't see SR 25 (23 was the highest number Isaw).
Number: 3 Author: Gareth Jenkins Date: 23/10/2018 17:51:00
“OF is the key here.
Hard to investigate now.
Number: 4 Author: Simpkins John_Date: 01/11/2018 15:32:00 Z
No FAD code or incidents referred so cannot investigate history
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 61 of 225
responsible for otherwise unidentified branch shortages, but they also
recognised this could be a possibility. I have noted that hardware
replacement often seemed to be a “fix” of last resort where no other
explanation could be given, and therefore there is certainly a possibility
that hardware was at fault. ”°
lems relating to the condition of the electrical power supply to the
branches has been acknowledged by Post Office, for example an internal
Post Office email POL-0030971 reports:
“was too scared to accept a cup of tea in case the Horizon system
crashed cos [sic] the electric supply is still a live (excuse the pun) issue...
It is Horizon related —- the problems have only arisen since install & the
postmistress is now barking & rightly so in my view”.
5.66 Fulher documentation also uncovers issues in respect of PIN pads and
base unit failures. In addition, unexplained system behaviour related to
branch hardware was also reported ®° and which lead to Hayments
crediting the wrong accounts. By a process of elimination conducted by
the Subpostmaster, this was narrowed down to a single branch terminal
card reader. ThElRomec engineer who subsequently visited advised this
was a “known Horizon error” and the card reader needed to be rebooted.
It is not known how widespread this issue was and how many
transactions were affected and/or when the underlying cause of this
error was fixed.
5.67 POL-0032853* (authored in 2004) sets out the “/essons /earnt” in respect
of the entire lifecycle of the S60 (software) release of which there are
many hardware related issues. In particular, ID No 36 documents an
7° RColeman4733L.html, HORIZON KEL RColeman4733L, 18 November 1999 (last updated 08 January 2004),
[POL-0033974] [5654ca357e27272f66887cdefab472f8]
8 Petersfield.pdf, Report of upcoming loss at next T/P, 14 September 2014, [POL-0219802]
[b81bc528975e821221743fdd3d1edd28]
5 POL-0032853 Lessons Learnt from the S60 Release 15 December 2004 [POL-0032853]
[182f6b865d7707 bd058328bf1f2f8c38)
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 77
Number: 1_ Author: Gareth Jenkins Date: 23/10/2018 17:53:00
~ Gearly computerising a branch does require a reasonable amount of power. Not sure whose responsibility itwas ~ I suspect the SPM would have been required to provide
adequate power.
Number. 2_Author: Gareth Jenkins Date: 23/10/2018 17:54:00
No ref?
,Number: 3 Author: Gareth Jenkins Date: 23/10/2018 17:55:00
is this the AP double barcode issue? I thought that was old Horizon only, but this refers to 2074
Number 4 Author: Simpkins John_Date: 01/11/2018 15:46:00 Z
Tdo remember an issue where BarCode scanner replayed a previous scans over and over but not a pinpad issue.
Number: 5 Author: Gareth Jenkins Date: 23/10/2018 17:56:00
Can we pin this down?
Number 6 Author: Simpkins John_Date: 01/11/2018 15:46:00 Z
Not sure.
PC0193391 has the summary of
pinpad calling up previous customers details when card entered in card reader
for branch 422420 (Baycliff Road). This was actually relating to the swipe on the keyboard that was replaying a previous swipe not the pinpad.
KEL RColeman854Q was quoted the final comment in the Peak was:
On 11/01/01 the PM noticed that MC swipes were coming up with the previous transaction details. The keyboard was swapped on 15/01/10 (tfs 1923665 ). The PM noticed the
error occurring again on 18/01/10. He reversed an incorrect transaction, but there may be other incorrect transactions which have gone unnoticed,
Note: SSC cannot identify the accounts or PANs to which the payments should have gone, and the duplicates check will not always identify which of the payments may have
gone to the wrong account (if the incoming swipes are all out of step, there may be only one of each, but not the one that the customer/clerk intended).
Please send the attached spreadsheet to POL saying that itis possible that any of the payments might have gone to the wrong account, and may be queried in the future.
Since the keyboard has been swapped, the cables and connections need to be checked, since a loose cable may have caused this problem in the past at another branch
(PC0179457).
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 62 of 225
“Epson Printer Issue” resulted in S: ostmasters being unable to
declare their cash accounts.
5.68 There is a recurring theme relating to errors arising from PIN pad failures.
A selection of these can be seen in the following KELs dsed525Q*,
surs3941P® and BrailsfordS239K**. In Each case there was a failure in
the Subpostmasters being able to transact various types of transactions
including payment transactions using a PIN pad. An error message and
code were generated, and a new PIN pad was the recommended
solution.
5.69 cardc219R® appears to indicate that any PIN pad related issues would
usually result in the recommendation of a new PIN pad whatever the
error. In Bhis case a transaction had been declined by the PIN pad but
did not get reversed. Anlider version PIN Pad (Hypercom) was being
used and a new version (Ingenico) was suggested and any reoccurrence
of the error would then need to be reinvestigated. It is not clear in this
case when or if a new PIN pad was issued and if this fully resolved the
issue. In this instance there could be an impact on branch accounts if
no reversal takes place and reliance then shifts to reconciliation reports
to pick up these discrepancies.
5.70 There are variety of ottalr examples of counter hardware issues® ®” where
replacement equipment was the recommended solution. Hardware
(keyboard, screen & screen cable) replacement was also suggested as
®2 Desed525Q.html, HORIZON KEL desed525Q, 09 July 2009, [POL-0036489]
[352690d6c7d9a258bc02a6df75d37254]
83 SURS3941p.html, HNG-X KEL susrs3941P, 14 April 2010, [POL-0037407]
[ce6e4d2bb070aea212d132d196bc3aba]
38 BrailsfordS2239k.html, HNG-X KEL BrailsfordS2239K, 14 June 2010 (last updated 01 July 2010),
[POL0037615] [30f83d46131ffd5d6 1d80a2b86c94cec]
85 cardc219R.html, HNG-X KEL cardc219R, 11 May 2011 (last updated 31 October 2013), [POL-0039594]
[3e0e7a8604a8f093bec033f9C69¢766e]
8° RColeman566K.html, HORIZON KEL RColeman566K, 04 April 2000 (last updated 08 January 2004),
[POL0033986] [65dbfc2631046aafd78dbb69419431b3]. PCarroll2243R.html, HORIZON KEL PCarroll2243R, 06
April
®7 (last updated 23 August 2005), [POL-0034763] [6eidcee9b64aa61e07ad694e1329cb06]
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 78
Number: 1 Author: Gareth Jenkins Date: 23/10/2018 17:57:00
~ Tsuspect this means that the printer is dead you can’t print it. It isn't saying there is an issue in the accounts though.
Number: 2_Author: Simpkins John_Date: 01/11/2018 15:55:00 Z
Correct.
Number. 3_ Author: Gareth Jenkins Date: 23/10/2018 17:58:00
“Certainly an inconvenience, but not something that caused a financial loss,
z.Number: 4 Author: Gareth Jenkins Date: 23/10/2018 17:59:00
Was this purely to avoid investigating Hypercom issues when Ingencio was in rollout?
Number. 5 Author: Gareth Jenkins Date: 23/10/2018 18:00:00
This would have been picked up by reconciliation. Not sure it would need reversing ~ depends exactly when the PIN Pad declined it
se Number. 6 Author: Gareth Jenkins Date: 23/10/2018 17:59:00
This was the time we were in the process of rolling our new Pin Pads anyway.
_Number: 7 Author: Simpkins John Date: 01/11/2018 15:57:00 Z
PCO270052 applies ~ a fix was produced but due to the fundamental way the Ingenico works itis not affected
Number. 8 Author: Gareth Jenkins Date: 23/10/2018 18:02:00
We probably need to look at ail of these.
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 63 of 225
a solution for an issue with “phantom” sales lines appearing on the
transaction that had never been selected. ®*
5.71 Additional power related Horizon issues are discussed in relation to the
Recovery and Reversal process discussed above.
5.72 A 2004 document® describes the procedure to calculate the Mean Time
Between Failure (MBTF) for each component of hardware within the
Horizon Service Infrastructure deployed within Post Office branches.
This is stated to “allow Post Office to calculate the actual failure rates
applicable to each item of hardware.”
5.73 Section 4.2 of the document states:
“Fujitsu Services have applied adjustments to base unit data to account
for known problems within the Horizon System Infrastructure, which have
caused spurious fluctuations which should be ignored for the purpose of
forecasting the annual change. pubhg the distribution of release CSR+
and BI2, many base units were swapped out due to software failure rather
than specific hardware fault. They have since been recirculated into the
estate and Fujitsu Services have therefore excluded them from the MTBF
calculations.”
5.74 Annex 2 of this same document illustrates the MTBF results for January
2002 to December 2003. This annexure is not fully understood as there
are missing column identifiers that prevent confirmation as to what the
rates represent.
Fujitsu Closed Problem Records
5.75 Post Office has disclosed a corflemporaneous spreadsheet entitled “Copy
of Fujitsu closed problem records.xls”°° which sets out around 200
8 PSteed145J.html, HORIZON KEL PSteed145J, 07 January 2000 (last updated 06 January 200400,
[POL0033869] [56234dae4303708631c499c596bec86F]
®° CSPRO149_3.1.doc [POL-0079278] [c4bff08def03773e0501a47 26d9f255a]
°° Copy of Fujitsu closed problem record.xls, [POL-0215915] [765f3677a7246da5Sdc9eafabc84f570a]
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 79
Number: 1_ Author: Gareth Jenkins Date: 23/10/2018 18:03:00
isn’t this when we distributed a major service pack to NT which had a fairly key impact on the counter stability. I suspect it would have identified some flaky bits of disk — hence
the box swaps.
Number. 2_Author: Gareth Jenkins Date: 23/10/2018 18:05:00
“Date?
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 64 of 225
issues which have been closed by Fujitsu in one year comprising of both
hardware and software issues. For example:
a. Issue ID 19 in the spreadsheet states in relation to a reconciliation
issue:
“Isshbs with First rate control files provided by Fujitsu. First Rate Travel
Service (Third Party) reconciles all Bureau de Change Transactions and
are currently unable to reconcile reversals against original transactions
due to this issue. The issue currently is caused by HNG-X Bureau de
Change Transactions transferred from Branch Database to TPS host.”
b. Issue ID 78 refers to *Cablelot file mapping issue causing
discrepancies”.
c. Issue ID 197 refers to token ID mismatches and states specifically:
“FoiKwing an AP Ref Data update being enabled on Wednesday 1st
February. Post Office Card Account (POCA) transactions were unable to
complete and token IDs matched incorrectly for a number of Automated
Payment (AP) transactions, E Top Up cards and bank cards.”
5.76 As above, there are nearly 200 other issues contained in this spreadsheet
which have been tracked by Fujitsu. However, the spreadsheet only
appears to cover the period relevant to 208 and
2011. My assumption is that Fujitsu would have had these (or similar)
trackers throughout the entire period that Horizon and Horizon Online
have been live, as this is how issues are tracked and fixed on a typical
project (and that appears to be the function here as well). However, I
have not yet been provided with similar trackers for other years.
5.771 have filtered the content that appears to be relevant to the issues in this
matter and have set them out in Appendix F below.
5.78 In consideration of various witness statements, predominantly Ricard
Roll, there were a wide ranging variety of possible bugs / errors / defects
within Horizon.
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 80
Number: 1_ Author: Gareth Jenkins Date: 23/10/2018 18:06:00
No impact on branch accounts. I seem to remember it was an issue with matching reversals to the original transaction and the way HNG-X and Horizon identified them in
different ways,
Number. 2_Author: Gareth Jenkins Date: 23/10/2018 18:07:00
No idea what this is.
jNumber: 3_ Author: Simpkins John Date: 01/11/2018 16:05:00 Z
Tis may be related to Peak PCO213537,
TES 4885597 / PCO213537 is remarkably similar to another recent call (TfS 4878430 / PCO213352), in which Transaction Acknowledgements for a (Camelot) product, 880000067,
were written to the BLE files without their TA references. This was because the Article ref data provided by POL was incorrect: the ?Reference? attribute for the Article was set to ?
NA?, which excludes the TA reference from the BLE entry, when it should have been ?TC?, to include the TA reference in the BLE entry.
In the case of these Transaction Corrections:
Article QOBO001590 has ?Reference? attribute set to ?NA?, so the reference value is excluded from the BLE entry.
Article 8580000001 has ?Reference? attribute set to 77C?, so the reference value is included in the BLE entry.
Since this is @ POL ref-data problem, it should be referred back to POL for rectification.
Also, as similar problems have occurred twice in a short space of time, perhaps it would be worthwhile for POL to review the ?Reference? attribute of all such TA and TC Articles,
so that any others can also be identified and corrected at the same time.
Number. 4 Author: Gareth Jenkins Date: 23/10/2018 18:07:00
This isthe “Highland Authority issue". No impact on Branch accounts, but very inconvenient ~ especially to POL Clients
Branches were severely impacted until around 10am when a ref data fix was applied to enable CAPO to work
Number: 5 Author: Gareth Jenkins Date: 23/10/2018 18:09:00
This was the period of counter migration to HNG-X so it may well have been a one-off
Number. 6 Author: Simpkins John Date: 01/11/2018 16:08:00 Z
Tknow of no other trackers it may be worth talking to Service Management (Steve Bansal)
gpNumber: 7 Author: Gareth Jenkins Date: 23/10/2018 18:11:00
~Tthink we just refer to our comments on RRs WS.
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 65 of 225
5.79 In the witness statement of Richard Roll?! he recalls:
“Any errors made by the Subpostmasters would be relatively easy to
identify and would normally be picked up by 1% or 2°4 line support. If an
error was referred to us, then it was extremely unlikely to be due to a
mistake made by a Subpostmaster, the vast majority of errors I dealt
with were due to coding errors or data corruption.”
5,80 He goes on further to state at Paragraph 9:
“We regularly identified issues with the computer coding in the Horizon
system. We would then flag those issues to the Fujitsu IT software
developers. The developers would then work on a fix” while we monitored
whole estate in relation to that issue”
5.81 In respect of financial discrepancies at Paragraph 10 he states:
“My recollection is that the software issues we were routinely
encountering could, and did, cause financial discrepancies at branch level
including shortfalls being incorrectly shown on the Horizon system. If we
were unable to find the cause of the discrepancy then this was reported
up the chain and it was assumed that the postmaster was to blame.”
Horizon Robustness
5.82 For the purposes of this report, and in line with my definition given in the
Joint Statement, robustness is summarised as:
“The ability to withstand or overcome adverse conditions, namely, the
ability of a system to perform correctly in any scenario, including where
invalid inputs are introduced, with effective error handling.”
5.83 It is important to note that robustness does not equate to a guarantee that
software is bug or error free. A system’s reliability can be improved by
rigorous testing and debugging (provided no further bugs are
5! Witness Statement Of Richard Roll, 11 July 2016. (Para: 8, Page 2)
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 66 of 225
consequently introduced through this process)°?9? however, complex
systems can never be completely tested or ever entirely free of bugs. 1°
This is due in-part to the fact that no software can ever be truly perfect
101
5.84 The following statistics were reported in a Post Office presentation’?
created circa 2017:
a. More than 47 Million transactions per week were undertaken in
Post Office Branches from 18 Million customer visits;
b. 22 Million banking transactions every month and 2.5 Billion
transactions a year with a cash value of £100 Billion;
c. Circa 11,800 Branches;
d. Post Office cash supply chain collect and deliver on average £42
Billion cash, foreign exchange and secure stock each year.
5.85 In a document last reported as edited on 20th August 2010% Post
Office compares the error rate or “exception handing performance” of
Horizon compared to Horizon Online. In fhe document it is explained
that, "...the figures are averages over the whole system and do not claim
that they will be evenly distributed”.
5.86 A number of failure analysis statistics are reported:
a. Counter Peripheral failures [in Horizon Online] - Thébe are largely
the same as Horizon, estimated at approximately 1 failure per
counter per year.
° F. Bott, A. Coleman, J. Eaton and D. Rowland, Professional Issues in Software Engineering, Boca Raton:
CRC Press, 2000.
®3 A. Hunt and D. Thomas, The Pragmatic Programmer, Reading: Addison Wesley Longman, Inc., 1999. 1°
2.2 12. Presentation_The Post Office, An Insight_.pdf, The Post Office-An Insight, Angela Van Den Bogerd,
circa 2017, [0Se2ac28f7b36b04dd83ab301edf9f91}
°4 HNG-X Branch exception Handling Strategy- Agreed Assumptions and Constraints. [POL-0116897]
(6511184272a83cc7c16127d1f44ac807]
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 82
Number: 1_ Author: Gareth Jenkins Date: 23/10/2018 18:13:00
Not sure what point he is trying to make
It should also be noted that HNG-X was fairly immature at that point and was just getting stable.
Number: 2 Author: Gareth Jenkins Date: 23/10/2018 18:14:00
Given the hardware is unchanged, this is to be expected.
How does this relate to the industry overall?
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 67 of 225
b. Counter PC Failures [in Horizon Online]- These are largely the
same as Horizon, estimated at approximately 0.1 failure per
counter position per year that could cause loss of data. Note that
Power failure at the branch accounts for 80% of these cases.
c. Transient failures (< 2 minutes) impacting online transactions [in
Horizon Online]- These are reduced for Horizon Online (estimated
at 6 per counter position per year) compared to Horizon (estimated
at 11 per counter position per year).
d. Transient failures (< 2 minutes) impacting settlement [in Horizon
Online]- This is a new exception category for Horizon Online that
does not apply to Horizon. The estimate is 15 incidents per counter
position per year.”
5.87 The document also records an ongoing error, “Loss of Basket transaction
Data held in PC memory” which results in an estimated loss of
transactions in a basket per counter, per year of 0.1 and in Horizon
Online of 0.097.
5.88 In my position as an expert I am unable to estimate the level of the Horizon
system’s robustness. Given the size and age of Horizon, I would
however make the expert assumption (based upon systems of similar
magnitude), that there are not many people who could. The sheer
enormity of the task to garner a thorough understanding of the code,
which would be required to estimate robustness is, in my opinion, nearly
impossible.
5.89 This is compounded by the agreed facts that there were 19,842 release
notes for Horizon between 29 November 1999 and 8 August 2018 and
each of these introduced changes could affect a system’s robustness.
5.90 For context, 19,842 changes over a 19-year period is approximately 1000
a year or 19 changes per week. It is readily apparent that keeping on
top of a constantly evolving system’s robustness is a somewhat
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 68 of 225
insurmountable task. It would not be possible to determine whether a
change to fix an area of the code did not:
a. Create a new issues/bugs with older areas of the legacy code or;
b. Add brand-new issues/bugs to the system.
5.91 Instead, I have estimated the likely level of the robustness of Horizon and
benchmarked this against industry standards based upon a review of
the evidence available including the known error log (KEL) and PEAK
system.
5.92 Several KELs exist that identify failures of internal mechanisms in place to
ensure integrity of data. Folbxample, dsed4733R° identifies multiple
failed recoveries occurring because of a wrongly named recovery script.
From the section above, a robust system has to “perform correctly in
any scenario despite the introduction of invalid inputs”. clearly this KEL
details an issue that is at odds with this definition.
5.93 Further, ob&hgc5933K from 2010 shows that following network banking
(NWB) withdrawal transactions and printing of the customer receipts,
there was a loss of communications resulting in a message to the data
centre timing out. Consequently, the Subpostmaster was asked to follow
recovery but the transaction was oni] able to recover partially.
5.94 It goes on to state:
“It appears that the order in which txn [transactions] are recovered is by
recovering the most recent, then working backwards; however, the oddity
about this particular recovery process is the fact that the 5.00 txn (00-
215704-1-4273066-1), the first in the session, was recovered fully,
whereas the £169.31 txn NOT!”
95 dsed4733R.html, HNG-X KEL dsed4733R, 25 July 2013 [POL-0039482]
(78f45c50b543fb3673d9a1 8fe442eb37]
96 obengc5933K.html, HNG-X KEL obengc5933K, 12 May 2010 [POL-0038204]
[c24012c95de42ac17b9ad2a2be2461b2]
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 84
Number: 1_ Author: Gareth Jenkins Date: 23/10/2018 18:18:00
Yes, but these were picked up and presumably resolved,
Number: 2_Author: Gareth Jenkins Date: 23/10/2018 18:18:00
Not sure, iFit was identified as an issue and resolved. That to me sounds like robust.
Number. 3 Author: Gareth Jenkins Date: 23/10/2018 18:19:00
"This was the point of HNG-X that we can't safely defend against. Jan to June 2010 was definitely problematical,
Number: 4_ Author: Gareth Jenkins Date: 23/10/2018 18:20:00
So was the problem picked up and resolved? If so Not really an issue,
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 69 of 225
5.95 Clearly this shows potential for discrepancies but also a lack of absolute
robustness in Horizon. This is consistent with my opinion above that it
is unrealistic to expect any large IT system to be completely robust or
bug free.
5.96 Post Office themselves have established a department called the Data
Reconciliation Service. This department’s purpose is to deal with Horizon
system problems which have resulted in unreconciled transactions
which require some level of manual intervention. Post Office report that
1000+ transactions per week are processed by the Data Reconciliation
Service.
5.97 The fact that numerous processes and workarounds are in place to all
Fujitsu to modify data already recorded by Horizon is consistent with a
lack of internal integrity within the Horizon system and the high level of
need to ‘correct’ this lack of robustness manually.
5.98 Post Office acknowledge the need to improve in a “Finance Roadmap
Project” document published in September 2012°7 where under the
heading of “prddess and system gaps” it reports;
e “Multiple finance systems and a lack of automated controls...”
and
e “Significant amount of manual intervention in core Finance
Processes.”
5.99 It is common ground between the experts that that each time there is a
change there is a potential to introduce new bugs/errors/defects.
5.100 The frequency of updates indicates the level of bugs and level of change
to which that Horizon was subjected to.
5,101 As discussed above, Post Office’s response to my request for information
confirmed that there have been 19,842 release notes, I can see from
°” Phase 2a) consolidated output.pdf, Finance Roadmap Project ~ Phase 2a) Project Outputs, 3 September 2012
[POL-0215782] [ac4469b27e384fe4440f351c115d8108]
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 85
Number. 1_Author: Gareth Jer Date: 23/10/2018 18:22:00
This seems very high. Presumably this is who we send BIMS reports to. How many do wesend? (They also would get reports from Clients ~ but significant would result in a TC
~ how many TCs do we get?)
Number: 2_Author: Simpkins John_Date: 01/11/2018 16:13:00 Z
“This sounds very wrong. The SSC review most reconciliation and we get about 25 per week
This should go to Sec Ops for review (Jason Muir)
Number: 3_Author: Gareth Jenkins Date: 23/10/2018 18:23:00
We do NOT Modify Data.
Number. 4 Author: Gareth Jenkins Date: 23/10/2018 18:24:00
This doesn’t sound like an issue that impacts on SPMRS.
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 70 of 225
POL-0219318° a brief note on each of these releases and to which bug
(PEAK) they relate. Therefore, Horizon has been subjected to 19,842
changes which have been applied during its operation.
5,102 The graph below is taken from the data contained within polloz19318
and displays the bug fixes by year.
Peaks that led to software changes
me
/\
\
/ \
/\
p \
} \
i \
i
$088 j 6
a a / mo
i %
" 1999 2808 EOL-2NGZ «AS -INNE TOPOS PONT 20RD AGRE] AE? GRS A ERS LIT 20a
5.103 As noted in paragraph 4.21 above, th&bpparent ability prior to July 2017
to alter Reference Data without any formal consideration as to the
impact of this change could have had a potentially very significant effect
upon Horizon’s reliability and robustness.
5.104 Transaction Corrections also have the potential to affect the robustness
of the Horizon system. Torstein Godeseth’s witness statement®? states
these are one of four sources of transactions that make up transaction
data within Horizon. Therefore, any human intervention within the
°° Copy of CallTypeR_080818-ReleaseNotes (2).xls, [POL-0219318] [e621d43d3f3b629be26536c6584c53d7]
°° Witness Statement of Torstein Olav Godeseth 27 September 2018
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 86
Number: 1 Author: Gareth Jenkins Date: 23/10/2018 18:26:00
Are these live bugs? The huge Peak in 2008 2009 suggests to me that this is including the dev peaks for HNG-X development.
ge Number: 2_Author: Gareth Jenkins Date: 23/10/2018 18:27:00
WE ned to kill this one!
Number: 3 Author: Simpkins John_Date: 01/11/2018 16:16:00 Z
“ Rgreed
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 71 of 225
Transaction Correction process has the potential to ftroduce errors and
therefore affect the robustness of the Horizon system.
5.105 In addition to this, I have observed a summary of Transaction Corrections
between the years 2013 - 20141 which shows approximately 20% of
all Transaction Corrections raised appear to have been deemed by Post
Office as: “not caused by the branch”. I have also noted that a different
summary of Transaction Corrections issued in 2010/11 sets out that the
net value of Transaction Corrections which were not categorised as
“caused by branch” was £7.4m!°!. Clearly, if potentially erroneous
Transaction Correction data (or error notice data pre-Transaction
Corrections i.ePHorizon Online) is entered into the Horizon system, this
wilPhtfect its reliability and robustness. Furthermore, if these erroneous
Transaction Correction transactions were not caused by the branch and
entirely unknown to the Subpostmaster there is a sigAlficant risk and
issue that could impact branch accounts.
Horizon Uptime
5.106 As maintained above, it is technically challenging and expensive to
achieve high availability of large systems. Horizon is a nationwide
system made up of many parts, which makes achieving the industry
standard of 99.5% difficult and the gold standard “five nines” (or
99.999%) almost impossible. *°
5,107 The 2013 Agreement between Fujitsu and Post Office’ (at page 12)
aimed to achieve 99.53% Branch Availability.
100 TC summary by Product full year 2013-14.xIsx, Summary of transaction correction causes, 22 October 2014
[POL-0221563] [9e326d06b05870f076dd6fe8b69015c7]
11 NEW TC PACK P10 2011.xIsx, Summary by Period, 15 February 2012 [POL-0221536]
[e1a3c3394d348ab3355302b35cfd63ab]
102 This is a common term used to describe percentages of a particular order of magnitude. For example, a
system or service that is delivered without interruptions 99.999% of the time would have “five nines”
reliability. See also POL Risk and Resiliency Review v1 5.pptx, Post Office Limited IT Risk and Resilience Review
Final v1.5, 08 June 2012, (Page 66), [POL-0219408] [25eef3b7ee67 ce3fcfdcbf2dec402c9b]
193 Horizon OLA between POL and Fujitsu vO 2.pdf, Operational Level Agreement (OLA) Between Post Office Ltd
& Fujitsu for the Horizon Service, 22 July 2013[POL-0215475] [0700cd4e673159d3f251fcaa72323307]
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 87
Number: 1_ Author: Gareth Je
The whole point is to correct errors.
Date: 23/10/2018 18:28:00
Also, an SPMR should be allowed to query a TC initially (I accept that POL don't allow them to query it indefinitely, but that is a POL process issues and not an issue in Horizon)
«Number: 2_ Author: Gareth Jenkins Date: 23/10/2018 18:30:00
No. TCs were introduced as part of IMPACT in 2005.
Number: 3 Author: Gareth Jenkins Date: 23/10/2018 18:30:00
No. it may impact the branch accounts, but not the reliability
Number. 4 Author: Gareth Jenkins Date: 23/10/2018 18:31:00
The whole point is that they should impact the branch accounts! However they should be agreed by the SPMR and queried if not understood,
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 72 of 225
5.1081 have seen reported examples of downtime across the Horizon system.
A Post Incident Report dated 09 December 2015 detailed a ‘POLSAP
Filesystem Incident’ on 5 January 2015 that resulted in a pobbap
application outage for approximately 49 minutes. 1°
5.109 Likewise, there was an outage to BaPking, Card Payment, E Top Up and
Automated Payments Out Pay (APOP) on 7 November 2016 for two
minutes" in addition to a TivBhi Work Scheduler Batchman outage for
approximately six hours on 7 February 2017. 1%
5.110In Fhy opinion whilst these instances show that Horizon is not infallible
nor totally robust, it still places Horizon with a good level of availability
overall.
5,111 Overall, from my review of the evidence it is likely (whilst putting aside
the large number of often required manual processes) that the
electronic processes within Horizon are relatively robust based upon the
literal, contextual, definition. Certainly, both instances of Horizon
(Horizon and Horizon Online) appear to be in-line with other IT systems
of similar size and industry.
5.112 However, this does not mean the Horizon system (as a whole) is infallible
and certainly does not imply the sofware is bug-free nor its accounts
free from errors. The extent to which Horizon is robust, in my opinion,
is reasonable but not a guarantee of no shortfalls or branch account
accuracy.
Likelihood of shortfalls in branches
5.113 Whilst controls and integrity checks are identified within Horizon, it is
evident that the Horizon system itself and errors within it have been the
105 SYMSDMINR2690_1.DOC, Post Incident Report: POLSAP Filesystem Incident 5th January 2015, 9 December
2015, [POL-0143426] [Sc6f3d6e2a51cd3dd8c263f85493cd04]
195 SVMSDMINR3299_1.DOC, Post Incident Report: Banking, Card Payment, E Top Up and APOP outage, 11
September 2017 [POL-0151983] [Oeeec8be8128ea7ace5034bacc2c7 14d]
106 SYMSDMINR3343_1.DOC, Post Incident Report: Batch processing outage, 22 September 2017 [POLO152261]
[10b424479186ab4b0e1e750460231f57]
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 88
Number: 1_Author: Gareth Jenkins Date: 23/10/2018 18:33:00
POL SAP failures have no impact on Branches so not really relevant. The availability figures relate to support of Branches only and not back end systems,
Number: 2_ Author: Gareth Jenkins Date: 23/10/2018 18:34:00
That would impact branches, but 2 mins is pretty good and shows that our automated recovery processes kicked in properly.
Number. 3_ Author: Gareth Jenkins Date: 23/10/2018 18:35:00
“No impact on Branch operation. This controls the overnight schedule.
Number: 4 Author: Gareth Jenkins Date: 23/10/2018 18:35:00
Thank Your
Number. 5 Author: Gareth Jenkins Date: 23/10/2018 18:36:00
WE have never said they were.
What we are saying is that when errors occur we detect them and can fix them and so they don’t result in SPMRs losing large amounts of money (which is what some are
alleging).
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 73 of 225
calle of shortfalls attributable to branches. Although correcting
transactions are capable of being issued, the issuing of such so that a
Subpostmaster does not suffer the loss is largely dependent upon a) the
cause of shortfall being accurately identified b) the reporting and logging
facilities to identify such being a true picture of events and c) the actual
event being detected in the first instance. As evidenced throughout this
report, and in consideration of the Claimant’s witness statements
Prepared for the Common Issues trial, there are examples of various
instances where part or all of a), b) and c) may not have been eff&ktively
conducted.
Extent of errors in data recorded within Horizon arising from (a) data entry, (b)
transfer or (c) processing of data in Horizon
5.114 Regarding the extent of potential errors within Horizon I have analysed
5114 Horizon Known Error Logs (KELs) to determine the scope of
potential bugs or ‘PEAKs’ (as they are referred to by Post Office and
Fujitsu). Of these 5114, I have found that 163 contain PEAKs that could
be of significant interest and of these 76 are referred to in the report.
The KELs disclose that there have been actual errors in data recorded
within Horizon arising from transfer, processing of data in Horizon and
data entry. The potential for such errors therefore must exist. See
Appendix G (Failure Point spreadsheet).
5.115 Evidentiary findings in respect of points a), b) and c) (above) are set out
below. However, they do not fully identify the extent of errors in data
recorded within Horizon, only a section of those reviewed thus far.
Further, it is not clear what the true extent of errors recorded within
Horizon is since the quantification of undocumented issues is not fully
known, for example bugs / errors /defects not reported by a
Subpostmaster but pogdibly accepted as an accounting error on their
part.
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 89
Number: 1_ Author: Gareth Jer
Date: 23/10/2018 18:38:00
“Yes some, but not the huge ones that we are being accused of
Number: 2 Author: Gareth Jenkins Date: 23/10/2018 18:39:00
There may be some small losses that arent picked up, but not the huge losses implied in previous cases
Number. 3_ Author: Gareth Jenkins Date: 23/10/2018 18:42:00
Fave you looked to see of the peaks you have reviewed how many were raised by SPMRs and how many found by Fujitsu processes that are in place to pick up such issues?
Is it worth us doing that sort of analysis?
Number: 4 Author: Simpkins John_Date: 01/11/2018 16:43:00 Z
This break is slightly difficult to check for tickets raised in T1S/Powerhelp, Ill have a think about how we can do it
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 74 of 225
Errors that are Potentially the Result of Multiple Issues
5.116 KEL wrightm33145J!°7 1°° (raised September 2010) entitled ‘Office has
Non-Zero Trading Position (Receipts/Payments mismatch)’ states:
“This issue was fixed in November 2010. For new occurrences see KEL
ArnoldA2153P.” (raised on 08 October 2009).
5.117 It should be noted that although the Horizon error in wrightman33145J
was fixed and new occurrences were referred to ArnoldA2153P,*™ the
latter KEL generates the same Receipts and Payments mismatch error
message but in fact relates to a mismatch during the balancing of a
stock unit that contains withdrawn product (relative to Reference Data
issues). There is no mention in the KEL if this was communicated to
branches as part of the Receipts and Payments mismatch issue.
thélefore, it appears a single “defect” could present itself as an error in
many ways.
5.118 Another example of a Receipts/Payments mismatch issue was raised in
ballantj1759Q?° (Fetduary 2010) detailing a “Counter APP ERROR” that
has been caused historically by:
a. Falsely reported for the Office Snapshot, when a transfer is in
progress. Fixed by PC0194381 in April 2010;
b. Pressing Cancel at a certain point during stock unit rollover. See
wrightm33145J; and
c. Training Office producing a balance snapshot where data hasn’t
been reset for a very long time (PC0210277).
5.119 What can clearly be seen here is that there are at least three issues within
Horizon that cause a Receipts/Payments mismatch that will directly
107 wrightm33145).html, HNG-X KEL wrightm33145J, 23 September 2010 (last updated 01 April 2016) [POL-
108 ] [1f025ec713c287ee7a5b1 7accd25b42f]
199 ArmoldAZ153P.html, HNG-X KEL ArnoldA2153P, 08 October 2009 (last updated 29 March 2016)
[POL0040401] [fbf8040b134636ad64f8e68tbf4d706d]
430 Ballantj1759Q.htmi, HNG-X KEL ballantj1759Q, 12 February 2010 (last updated 17 May 2011) [POLO038508]
[f4c2a317d57451cdc91ba81dc1072002]
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 90
Number: 1_ Author: Gareth Jenkins Date: 23/10/2018 18:45:00
“No. These were 2 different defects that showed the same symptoms. We fixed one, but the other is potentially on going if Ref data goes wrong. This is a good example of the
checks and balances we have in place to ensure that errors are detected
Number: 2 Author: Gareth Jenkins Date: 23/10/2018 18:46:00
“This was the time HNG-X was very new. Can we confirm that this was an HNG-X issue and not old Horizon (it sounds like it)
Number. 3 Author: Simpkins John_Date: 01/11/2018 16:47:00 Z
PCO194367 — This was for HNG-X fix was released in CTR APP_X0108_VO4S-VO44
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 75 of 225
affect branch accounts. Further, the KEL goes on to state that:
“Instances of this error must be investigated. If the error is as a result
of a new problem, please add the details to the list of causes above”,
demonstrating that the true effects of thil known bug may still be
unknown.
5.120 On a similar theme incorrect reporting of discrepancies can arise resulting
in incorrect stock declarations. See act1357Q, 111 It is not clear how
many branches were affected but it appears that it is possible for
discrepancies to have been accepted by the Subpostmaster based upon
incorrect declarations. The problem could arise due to old stock
declarations not being automatically removed from the system. These
could only be removed by making zero-value declarations or deleting
the stock unit then waiting overnight before balancing. As a result, it
was possible if a declaration existed for a current period, it would be
used when the discrepancies button was pressed, or when a balance
report was produced, even if it had not been used for a year.
5.121 The KEL above indicates that the issue was passed to development via
PEAK: PC0208335 and the recommendation was for the Subpostmaster
to be contacted for advice on corrective actions. acha3145Q!¥2 which
pre-dates acha1357Q by over year provides a full support solution for
this issue of incorrect stock declarations and discrepancies. It ould
seem that the problem was known for at least 12 months before being
passed to development and it is not known if the issue was subsequently
fixed and/or how widely the corrective support actions/process was
communicated.
481 acha1357Q.html, HNG-X KEL acha1357Q, 14 February 2011 (last updated 16 June 2017) [POL-0040896]
[8109c05bc69b18eed896e45c3a2115a5]
482 acha3145Q.html, HNG-X KEL acha3145Q, 18 May 2010 (last updated 02 October 2015) [POL-0040263]
[44b786a9d63674c53510487e50771172]
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 91
Number: 1_ Author: Gareth Jenkins Date: 23/10/2018 18:48:00
No, this isn’t a known bug. This event is an indicator of an accounting bug and so should it occur it needs to be investigated as stated.
This is one of the checks and balances to assist us in spotting errors (in the rare cases that they occur) rather than relying on an SPMR reporting an issue.
Number: 2 Author: Gareth Jenkins Date: 23/10/2018 18:51:00
This is part of the overall complexity of supporting multiple declarations to be added together in a Shared SU to reflect multiple physical tils
Itwas reflecting a difference in behaviour (requested by POL) between old Horizon and HNG-X.
Number 3_Author: Gareth Jenkins Date: 23/10/2018 18:54:00
iF this is what I think itis, itis an example of mis-operation by SPMRs and not actually a defect.
Number 4 Author: Simpkins John Date: 01/11/2018 16:52:00 Z
From the Peak:
Date:16-Feb-2011 11:45:59 UserGareth Jenkins
This needs to be addressed at 2 levels:
The fix is a change to the BRDB archiving such that any row in BRD8_BRANCH_DECL is archived (along with related rows in BRDB_BRANCH_DECL_ITEM) 6 months (ie 180 days)
after the timestamp in update_timestamp. This will result in any Declaration being discarded after 6 months and so avoid the wrap around issue. Note that discarding an
Declaration purely means that a complete Declaration will be required rather than being able update an old one and so has no impact on the accounts.
However looking at the spreadsheet attached to the peak there are 4 branches that will have an issue in the next month, so we must either contact the branches and get them to
carry out a zero declaration (which I understand Anne has done in the one branch with a problem today), or we should raise an MSC to tidy up the data. I propose that an MSC is
raised so as to remove any Declarations that have an update timestamp of July 2010 or older, which then allows time for the fix to be progressed in normal timescales.
Passing Peak to BRDB Dev to progress.
Fix for this PEAK applied to live in 2011 via RNT7529A - Release PEAK 210249
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 76 of 225
In relation to a) Data Entry
5,122 It is evident that data entry was clearly a significant problem at the
branch counter. We can see from the Service Level Agreement (SLA)
reports (i.e., SLA Summary New WE 06072014143) and filtering for
“Correcting Horizon Errors” or “Miskeying” that there are numerous
instances of data entry errors made each week (Annex A). An SLA is a
commitment between a service provider and a client in respect of
aspects of the service to be provided e.g. helpdesk response times. In
this instance the report can be used to analyse the type and reason for
calls made to the helpdesk.
5,123 An internal feasibility study report, carried out in 2012 1''*, was
commissioned to investigate the issue of mis-keyed transactions and
the options for preventing this problem. The report noted the following
financial impact:
"A further statistic, which was recorded on Friday 6th July 2012, refers to
the value of mis-keyed Banking Deposit transactions amount to over 60
PER WEEK. The total of investigations that become necessary as a result
of mis-keyed transaction equates to £10 millions per annum (approx.).”
5,124 Further, an internal presentation from Post Office ‘tS looking into
efficiency gains reported: “a significant portion of demand at FSC is
driven by errors and mistakes made in branch with entering in data into
horizon. Part of these errors can be avoided with relatively small
changes to horizon”. The presentation goes on to set out four changes
that could be made to Horizon that would save time for the
Subpsotmaster and reduce data entry errors in Horizon.
13 SLA Summary NEW WE 06072014.xls, NBSC Incident SLA Summary - Week ending 6th July 2014, 14 July
2014 [POL-0031909] [2a6e1fbd3efSd76899a9f21957d65011]
114 Feasibility Study ~ Mis-Keyed vO 1.doc, G-231 Mis-Keyed Project - Feasibility Study V0.1, 15 May 2012,
[POL-0217750] [8e9f114b3c4106d0f5255f906b742731]
45 Review of Mid-Term Initiatives [POLO217407] [670bcc305bb2ee89d5a7546faa0e86e9]
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 77 of 225
5.125 A further internal presentation from Post Office'!® references the findings
of external consultancy firm McKinsey. Thi presentation repeated the
statement made previously that relatively small changes to Horizon
could avoid errors/mistakes made in branch. The presentation
suggested that the ease of implementation of such changes as;
“Medium. There exists an interdependency on IT for changes in Horizon
to reduce errors coming from upstream” with the next steps described
as; “Assess costs for Horizon changes”. The presentation is focused
upon delivering costs saving to Post Office, through reducing support
costs and did not appear to investigate any saving which might be
achieved by Subpostmasters.
5,126 An external information security review!” was carried out in 2008 by
Infosec as part of POL’s adoption of the International Standard BS
ISO/IEC 17799:2005. Thal resulting report recommended various
system improvements after concluding:
“The Post Office, its agents, clients and banking partners are suffering the
consequences of a high level of transaction disputes and customer claims
across many financial, and all banking products due to a lack of source
data integrity, i.e. values entered only once without validation;
transaction value not presented to customer for validation.........banking
deposits are not visually presented for confirmation by the
customer.....bill payment and other transaction values are not presented
to customer.
5.127 Thal report also highlighted the short time period available for archived
data (180 days) hampering the ability of disputes to be resolved in a
timely manner and it recommended an increase to at least 540 days. It
is not known if or when all the report’s recommendations were
completed by POL.
46 Business transformation - Finance Wave 1 [POL-0218441] [a26a0ecOf6c8dbc20b753b200de725bb]
417 Information Security Review - Post Office Ltd, v1.2, 15 TH July 2008, [POLO217567]
[3dd3d32cb258ecf5895d94b2d205ee9d]
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 93
Number: 1 Author: Gareth Jenkins Date: 23/10/2018 18:57:00
Presumably POL didn’t ask us to implement any such changes?
Number: 2_Author: Gareth Jenkins Date: 23/10/2018 18:58:00
Tm not aware of us being asked to make any changes. I suspect that any such changes would have slowed down counter throughput and so were rejected on that basis, POL
wanted to reduce Qs significantly.
Number. 3 Author: Gareth Jenkins Date: 23/10/2018 18:59:00
‘Again this relates to POL's backed systems — presumably POL MIS / Credence,
Number. 4_ Author: Simpkins John Date: 01/11/2018 17:08:00 Z
Could the also be TES (which Stores 182 days)
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 78 of 225
5.128 Sirfllarly, in a fraud solution report*!8, it was noted that the POL had a
limited number of data queries (approximately 750 per year) which it
could request relating to transactional data in the branch database
which had been archived after two months. Beyond this number of
queries, each request became chargeable (£400-£500). It is not known
what impact the time limits on archived data and limits on the number
of non-chargeable data queries had on POL’s willingness to investigate
all branch discrepancies on behalf of Subpostmasters, however, the fact
that business constraints existed coi give an inference that there may
have been limits on the numbers and types of investigation of
Subpostmaster discrepancies.
Supporting Documentation
5.129 allend1645p'!° provides an example of Horizon’s weak interface controls
and lack of data entry validation. In a single sales transaction the clerk
was able to select and enter different methods of payment (Debit Card
and Fast Cash). HoPkon allowed the transaction to be settled via Fast
Cash when the Debit Card payment method had already been selected.
Although the KEL confirms this is expected system behaviour and to
advise the caller of the same, it is my opinion that controls should be in
place to restrict user input error in this scenario. There is nothing in the
KEL to indicate if this could be considered for a future system
enhancement.
5.130 In actl621P%2° the correct screen to successfully process a cash pouch
did not appear resulting in the clerk in an outreach branch inadvertently
doubling up the amount of cash recorded. The issue appears to have
been caused because of an earlier system logout or inactivity which in
488 NRRA1207.01 DOO1 - Post Office Fraud Solution report.pdf. Driving business benefits through the
consolidation of data review - Post Office Fraud Solution, 18 May 2012, [POL-0219392]
[Seb6f3175483e47d760c4fd44a6c06b5]
449 allend1645P.html, HNG-X KEL allend1645P, 24 June 2011 [POL-0038584]
[8a983ddfa657b62026d5bfc7ad78ba04]
220 acha621P.html, HNG-X acha621P, 15 October 29015 (last updated 14 January 2016) [POL-0040340]
(7518fc27f6357689c500a302358d4452]
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 94
Number: 1 Author: Gareth Jenkins Date: 23/10/2018 19:00:00
"This was the contractual position and allowed us to staff the Security team appropriately
Number: 2_Author: Gareth Jenkins Date: 23/10/2018 19:01:00
Probably true. I'm not aware of POL requesting data unless they were considering a prosecution. Security might have a better view.
Number. 3 Author: Gareth Jenkins Date: 23/10/2018 19:03:00
“This was a business requirement that allow partial settlements to different MOPs,
Number:4 Author: Gareth Jenkins Date: 23/10/2018 19:04:00
This was discussed earlier
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 79 of 225
turn resulted in incomplete checks being conducted by Horizon post
logon. There was a system fix applied for this on 12 January 2016, but
this did not retrospectively correct the affected branch accounts. It a
not known how many branches were affected by this bug/data entry
validation issue and/or what communications were sent out by Post
Office. The KEL log remains in a status of ‘Unresolved’.
5,131 This issue looks to be part of the same scenario analysed in a detailed
Fujitsu report **! #2 prepared for the Post Office. The underlying
symptom (duplicate transactions) caused either because of a “Forced
Log Off” or use of the »prBvious” key during the remittance process
analysis covered a period of five years (2010 — 2015). It was found that
88 different branches had duplicate pouches over this period. Although
most occurrences appear to have been accounted for and corrected by
the Post Office the fact thaélthree “fixes” over five years (the last being
in January 2016) is indicative that the issue was never fully resolved,
albeit most occurrences happened between 2010 and 2011.
5,132 £3¢%hnson3937R!2 demonstrates the lack of Horizon interface controls
which enabled Subpostmasters to carry out “Rem In” transactions
without a value being entered. Despite the system check message, I
would expect controls to be in place to restrict user input error in this
type of scenario so that a user cannot complete the transaction. There
is nothing in the KEL to indicate if this could be considered for a future
system enhancement.
5.133 PSteed145J!*4 highlights an issue reported on several occasions with
“phantom” sales items appearing on the Horizon counter screen but
21 Outreach BLE Extract Findings v6 091215.pptx, Branch Outreach Issue (Initial Findings), 10 December
222 I [POL-0220141] [33ab9fe9c4b2bcd600fb50b 7aff7bd8a]
22 EJohnson3937R.html, HORIZON KEL EJohnson3937R, 27 January 2005 [POL-0034505]
[2d7cfc706e00fbd9d7d7464021b1dd04]
124 psteed145).html, HORIZON KEL PSteed145J, 07 January 2000 (last updated 06 January 2004)
[POL0033869] [56234dae4303708631c499c596bec86f]
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 95
Number: 1_ Author: Gareth Jer Date: 23/10/2018 19:05:00
“We checked back 6 months (and found a few other cases) and I recommended a further search back, but I don't know if this was done.
Reading on we did do a complete search.
«Number: 2_Author: Gareth Jenkins Date: 23/10/2018 19:09:00
is this something different?
Number: 3 Author: Gareth Jenkins Date: 23/10/2018 19:07:00
What does this refer to. We only spotted the issue in Oct 2075 and a single fix was applied in Jan 16. We then looked back for all cases that this could have happened before
and identified them, Sounds good to me,
Number: 4 Author: Gareth Jenkins Date: 24/10/2018 16:12:00
From the date, it sounds like this was in the early days of Auto rems. Manual entry of Rem amounts should have been very rare
»,Number. 5_ Author: Simpkins John Date: 01/11/2018 17:14:00 Z
PCOO366O6:
Date:01-Jul-2005 11:33:29 User:Erica Johnson
[Start of Response]
For $90 this problem no longer exists because the mechanism described in this peak is no longer available, For $90 Currency & Cash Pouches will be remitted in via Pouch
Delivery, if details exist then the process is automatic, otherwise there is a manual input process. For Currency this displays a picklist, where F10 if pressed allows entry &
verification of the Quantity/Amount. There is no Escape key available & if Amount is 0 then an Error message is displayed.
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 80 of 225
which had not been selected by the Subpostmaster. In this instance it
was recommended to replace both the keyboard and screen and it was
also noted that another similar case had been caused by the cable
connecting the screen and base unit. Instructions on how to deal with
environmental issues and hardware are contained in pcarroll1235R!*5
but it is not known how widely these instructions were
communicated/distributed to Subpostmasters.
In relation to b) Transfer (of data)
5.134 Evidence shows various issues with the transfer of data within Horizon.
Supporting Documentation
5.135 cardc219R° records a transaction authorised by the bank but cancelled
by the Horizon counter PIN pad and the tralbaction did not get
reversed. The error is noted to have occurred due to missing key
transaction information and possibly due to issues related to Hypercom
PIN pads. It is not clear if the underlying cause was pinpointed to the
Hypercom PIN pad and/or if the later switch over to the Ingenico PIN
pad resolved the issue.
5.136 jhdh1323L127 records an unresolved example of a successfully recorded
transaction initiated in a Post Office branch (where a customer receipt
was generated) which failed to appear in the Post Office Data Gateway
(PODG) 8129 file and consequently was not transferred to the relevant
third party (Environment Agency). The PODG file should contain all
transactions to be transferred to the third party. In this case the
225 pcarroll1235R.html, HORIZON KEL pcarroll1235R, 05 April 2000 (last updated 05 October 2006)
[POL0035366] [3b14d838abd6aa7273dad4894aafcbdd]
26 Carde219R.html, HNG-X KEL cardc219R, 11 May 2011, (last updated 31 October 2013) [POL-0039594]
[3e0e7a8604a8f093bec033F9¢69C766e]
227 Jharr1323L.html, HNG-X KEL jharr1323L, 29 September 2016, [POL-0040563]
[afeb10ec3e6c812eecf9c7011abaaad4]
28 1.3 3. High level architectural overview of Horizon Online.pdf, Horizon Solution Architecture Outline (Page
229 _ 2.2.2.4.1), 07 April 2016 [C-0003645] [7b6cf8cf69bec90f674b9b10a64f04e8]
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 96
Number: 1_ Author: Gareth Jenkins Date: 24/10/2018 16:15:00
“This would have been picked up by reconciliation. I suspect this had no impact on Branch accounts (but would have impacted the customer if not picked up).
»Number: 2_Author: Gareth Jenkins Date: 24/10/2018 16:17:00
Can we investigate? Presumably the Txn was sent to Credence so should have come out in TPS / APS reconciliation
Number: 3 Author: Simpkins John_Date: 01/11/2018 17:48:00 Z
PCO254769 applies, It appears that the transaction was reversed 2 minutes later on the counter,
Looking at the audit logs, it appears this transaction was reversed at the counter about 2 minutes after purchase. Consequently, the licence information was not sent to the
Environment Agency.
The audit log shows the original transaction time as 08:49 which matches the time on the receipt. However, the log shows 2 instances of product 21273, £18 each (18.00) for
session ID 191855 (one at 08:49:05 and one at 08:49:37) and settlement value of £36 (-36.00). The timestamp for this settlement is almost 14 minutes later at 09:03. The event log
shows 2 AP Customer Receipts and 2 AP Basic Receipts were printed from 09:03:30 to 09:03:42.
), ie. 2 instances of product 21273,
The audit log also shows a later transaction at 09:04:59 (session ID 191856) which is the reverse of the original transaction (reversal flag =
£18 each (-18,00) and settlement value of £36 (36.00). All 3 transactions are timestamped 09:04:59,
The event log shows a REVERSAL event at 09:05:08 followed by 2 AP Customer Receipts and 2 AP Branch Receipts, printed from 09:05:13 to 09:05:25.
The ‘Environment Agency Licence renewal’ file sent via PODG does not include this licence.
As the Counter logs are no longer available, I am unable to investigate this further.
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 81 of 225
Environment Agency failed to send the customer his license as they had
no record of any purchase. Pod Office’s failure to hold logs for more
than 30 days affected Fujitsu’s ability to fully investigate the root cause
of this specific incident. The incident remained unexplained.
5,137 The ability of Horizon to erroneously record the same transaction twice
after a session transfer to a different counter was recorded in KEL
MArris34331.1°° This happening with both NS&I (National Savings &
Investments) and Network Banking (NWB) transactions was
acknowledged. The KEL was passed to a development team to provide
a bug fix as part of the S60 rollout but it is unknown if this was ever
resolved.
In relation to c) Processing of data in Horizon
5,138 There were various issues with the processing of data within Horizon.
Supporting Documentation
$.139 KEL CharltonJ2752T*! identified a Horizon bug which became apparent
when any counter level corrections made via the “Previous” key led to
both the old value and amended value being stored and used in error in
the transaction. According to the KEL, a fix was released in the Live
environment eight days after the issue was first raised. It Bnot known
how many undetected records were affected and if any further instances
were reported following the fix.
5.140 A recently introduced Post Office service ~pap & Go” was shown??? #33 to
have the ability to credit cash (“money out of thin air’) to the branch
account in circumstances where the session had timed out. It is not
known how widespread the issue was, but the referenced document
330 MArris34331.html, HORIZON KEL Marris34331, 10 December 2003 [POL-0033825]
[0e0193f167f883fd85bd79236da96913]
31 CharltonJ2752T.html, HNG-X KEL CharltonJ2752T, 04 October 2011 (last updated 16 April 2013),
[POLO039383] [01¢9c3402e3301825e37cc68e61b08ca]
82 Overview v004.pptx, Drop & Go ~ Settle to Cash Resolution - issues and options, 07 October 2016, [POL-
1 ] [4266a3c3129dc5f258dfe7 df888b446b]
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 97
Number: 1 Author: Gareth Jenkins Date: 24/10/2018 16:18:00
“What logs are these? We hold data for much Tonger and thought POL did too
Number: 2_Author: Simpkins John_Date: 01/11/2018 17:52:00 Z
We do hold the data in audit, just 30 days on the SSC counter for easy access.
Number. 3 Author: Gareth Jenkins Date: 24/10/2018 16:20:00
““Dowe know? Could we tell?
Number 4_ Author: Simpkins John Date: 01/11/2018 18:18:00 Z
it was for APADC only Peak PCO2 12793 applies, we were given a way to identify occurrences:
So occurrences can be identified by scanning BROB_RX_REP_SESSION_DATA for all rows with these product ids where there is more than one occurrence for the same session id.
These rows can then be checked by looking at the whole sequence number field to identify only those that have the same basket entry id (the first number in the field before the
comma).
We did identify all affected branches and sent a report to POL:
These calls should both be linked to Problem record 4795557, if not already done. We have provided a spreadsheet of affected branches (up to 4th October) to Tony Jamasb at
POL.
FIX Released to live 12/10/11
Number. 5 Author: Gareth Jenkins Date: 24/10/2018 16:21:00
This was after I retired so I don't know anything about this. Was this an issue in Horizon or CBD
-:Number. 6 Author: Simpkins John_Date: 01/11/2018 18:48:00 Z
~ No references to compare.
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 82 of 225
discusses the various options the Post Office were looking at to resolve
this and other issue with the Drop & Go service.
5.141 ssbt343p * records an example of a declined network banking
transaction that nevertheless resulted in money being taken from the
customer’s account. The transaction was automatically reversed
correctly at the counter but continued to be processed further through
Horizon, resulting in the customer account being debited. This incident
would have required the raising of a BIM (Business Incident
Management). The eventual discovery of a root cause and resolution of
this issue is not known but clearly there were errors in data transfer.
5.142 There are a number of examples‘**:'3° of E-Pay transactions crediting
the customer account twice although only one payment has been taken.
The error arose because both Horizon system authorisation agents were
incorrectly active at the same time (normally one is active and the other
on standby). This error should have been detected via an E-Pay report for
reconciliation but remains outside of the control of the Su
who would have been unaware of the initial error.
5.143 PERK pco063227 +37 (detailed PEAK log available in Appendix A)
highlighted a bug within the Horizon messaging service (Riposte) which
prevented 401 transactions with a total value of £11,708.08 from being
processed and impacting on branch reconciliation. A workaround was
applied as a short-term fix pending testing of a long-term fix.
5,144 POL-021641213® documents the POLSAP outage that occurred in Jarlary
2016 resulting in millions of pounds worth of transactions failing to
8 sSur343P.html, HORIZON KEL SSur343P, 30 April 2003 (last updated 16 June 2004), [POL-0034256]
[f435Sbfbba9c34885a97bc7c9f096671)
85 LKiang3526R.html, HORIZON KEL LKiang3526R, 25 August 2004 (last updated 7 April 2005) [POLO034590]
[67efbfdfc186487e866a8d701cd353e4)
36 ssur5310P.html, HORIZON KEL SSur5310P, 14 May 2004 (last updated 09 October 2006) [POL-0035378]
[0d473335a3b09f8c60f0c9ba08a7ad7]
137 pc0063227.html, Peak PC0063227, 28 February 2018, [POL-0237798]
[8c84cd6299903d0d3bbe0e05de44b924]
188 POLSAP outage 25-26 January 2016.docx, POLSAP outage - 25th& 26th January 2016, 29 February 2016
[POL-0216412] [daf7cd352274946cS5dfe164e326bc4c0}
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 98
Number: 1_ Author: Gareth Jenkins Date: 24/10/2018 16:23:00
This was just after Network Banking went live
Number: 2_Author: Gareth Jenkins Date: 24/10/2018 16:24:00
This has no impact on SPMRs. Ithas very happy customers. Not sure ife-pay of POL paid for the free top up,
Number 3 Author: Gareth Jenkins Date: 24/10/2018 16:25:00
“No impact on him.
Number: 4 Author: Gareth Jenkins Date: 24/10/2018 16:25:00
Tdon’t recognise this. Probably need to investigate
I assume it was a one-off which was resolved at the time.
Number. 5 Author: Simpkins John_Date: 01/11/2018 18:56:00 Z,
This was raised as PCO104358 but was classed as a duplicate of PCO104034 which reports an issue between NBE and Fl. This in-tum was closed as @ Powethelp was raisedwith
IBM for the investigation (e-0406080611)
.Number: 6 Author: Gareth Jenkins Date: 24/10/2018 16:26:00
No impact on Branches. This was purely a back end issue. It wouldn't affect Customers either as the Customer Data to clients don't use POL SAP.
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 83 of 225
process causing backlog and discrepancy for both Post Office and its
customers.
5.145 The attempted system recovery prevented access to the majority of
system functionality. Completion of all accounting recovery actions was
due by 29 January. It ia) not known what actual impact this had on
branch accounts, otHar than not being able complete any transactions.
Uncategorised Horizon Concerns
5.146 A Post Office incident summary document in 2015+? highlighted a
number of hig severity incidents between May and June 2015 one of
which included a financial deficit found in Global Payment transmission
and resulted in a £20 million shortfall within the system (cash flow
deficit for Retail and Bureau transactions). The precise underlying cause
of this incident is not known and/or whether or how it was rectified.
5.147 An independent technical review‘*° carried out in 2010 midway through
Horizon Online rollout highlighted some issues and concerns in respect
of recoverability following interruptions in service. The report stated:
“The
of the PCI changes) and HNH-X are not able to recover correctly from
is clear evidence from the solution that both Horizon (as a result
failures.
The most worrying are recBhciliation BIMS exceptions - these are failures
that require manual intervention by both Fujitsu and Royal Mail. Many of
these failures result in end customers of the Post Office not being paid
money (the exception shows that a bank believes it has paid out the
money, whereas the Horizon/HNG-X system knows it did not pay out in
reality”.
189 Incident Summary ~ June 2015 v3.pptx, Incident Summary - Trend Analysis, June 2015, [POL-0221065]
[ef2216b67d7ea7576fc14f517369d"b3 ]
+40 RedAlert_April2010.doc, Post Office Red Alert - Independent Technical Review, James Stinchcombe,
Principal Architect, 27 April 2010, [POL-0220516] [465299cf1a171e7b871e4b5f1 1b2f9ba]
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 99
Number. 1_ Author: Gareth Jenkins Date: 24/10/2018 16:27:00
~ Tewould have had no impact on Branch accounts.
Number: 2_Author: Gareth Jenkins Date: 24/10/2018 16:28:00
What does this mean? Branches don’t use POL SAP.
Number. 3_ Author: Gareth Jenkins Date: 24/10/2018 16:28:00
“er Thad retired so Idon't know about this. However unlikely to impact branches. May have Impacted customers — it depends on exactly what the problem was.
«Number: 4 Author: Simpkins John_Date: 05/11/2018 18:02:00 Z
PC0243759 applies raised on 28-May, issue with drs_bCeft_c2 table containing old unprocessed records,
The root cause of this problem is way back to 2006 peak PCO135158 that describes a manual update made to drs_tx_eft_c2, to set the status. Presumably because the
processed_date wasn?t also set, these records have been lying around ever since. The number of records quoted, 269828, matches exactly. Now that the sequence object has
wrapped around to match their ids, these records caused a problem.
Incident has been resolved 01-06-2015 11:23 UTC+01 (OST)
Global Pay confirm they can now see the outstanding data and have processed the transaction files successfully,
Number: § Author: Gareth Jenkins Date: 24/10/2018 16:31:00
What is the context of this?
«Number: 6 Author: Gareth Jenkins Date: 24/10/2018 16:31:00
ishe implying that these didn’t work, or is he just saying that there were more BIMS exceptions that there should be (thus increasing the risk of them being handled incorrectly
as it is a manual process)?
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 84 of 225
5.148 Thébe issues could directly impact branch accounts and covered the
period both pre and post Horizon Online rollout. Thlreport went on to
state that due to the high volumes of recoverable data the exception
workload was causing significant (and unsustainable) workload on both
Fujitsu and Royal Mail.
5.149 The risk of data loss arising because of incorrectly shutting down
equipment and/or replacing equipment is evident in several scenarios.
A Post Office training slide!*! specifically outlines this issue in Slide 5
regarding the Horizon Online kit used in outreach branches. The same
is true with branch terminals being swapped out*4?‘43. In this case
incomplete transactions held on the hard drive of a branch terminal that
was swapped out were lost when the engineer failed to recover the data
from the hard drive; this resulted in data being lost.
5,150 At the Post Office Board meeting of 25th September 20138 it was
reported that:
“op&lations Issues ~ Fallout from Horizon issues seriously damages public
and government confidence in the Post Office... Further operational issues
uncovered (but considered lower risk and lower impact)”
5.151 The disclosed document is heavily redacted and therefore it is not clear
what “operations issues” were being discussed.
5.152 Appendix H is a constructed summarisation of the processing components
within Horizon that have been identified as common points in which
discrepancies have been recorded. A selection of the KELs referred to in
this report have been used to illustrate points in Horizon where
+41 Core & Outreach Brief HOL v1.pdf, Core & Outreach Brief, June 2010, [POL-0176232]
[594a4d1508f124dc307797c92b1d3aed]
2 PC0093382.html, Peak Incident Management System, 5 August 2003, [POL-0265630]
[c3393ab177a14280dfe535b7b34282a6]
193 Post Office Limited Board Single pdf Final.pdf, POST OFFICE LTD BOARD MEETING, 25
September 2013, [POL-0215589] [13ebf7 if arbadt
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 100
Number: 1 Author: Gareth Jer Date: 24/10/2018 16:30:00
No, these issues wouldn't impact Branch accounts ~ provided that recovery was camied out correctly.
Number: 2_Author: Gareth Jenkins Date: 24/10/2018 16:33:00
Tremember James doing a report, but not the details (not sure if] actually received it)
Do we have some context? Torstein was heavily involved at that point - it was why he was recruited!
Number: 3 Author: Gareth Jenkins Date: 24/10/2018 16:35:00
This only applies to Old Horizon.
As I understand it, in these cases the hard drives were sent to SSC for Mr Rolf to investigate!
» Number 4 Author: Simpkins John Date: 05/11/2018 18:12:00 Z
Richard Roll?
Number 5 Author: Gareth Jenkins Date: 24/10/2018 16:36:00
This sounds like the time that the 2nd sight report went to Parliament.
We had disclosed the 2 defects in HNG-X that caused accounting issues (and described earlier in this doc).
2nd sight didn't find the issues as they didn’t impact the branches that they were looking at.
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 85 of 225
anomalies have either been detected or have failed to complete the full
processing.
Exten’ nclusion
5.153 It has not been possible to measure the full extent of errors in data
recorded within Horizon arising from a) b) and c) above since:
i. it is possible that there are further reports and statistics that
might indicate further extent of errors which have yet to be
disclosed;
ii. The information contained within documents such as KELs etc is
not complete or comprehensive in respect of the extent of the
issue recorded in the first instance (or the full impact across the
estate and on the Subpostmaster);
iii. It is not clear from those PEAKs (bugs/errors/defects) that have
been added to a software release whether the symptoms
identified in the first instance may have manifested and been
recorded under a new/different KEL/PEAK.
5.154 However, from the documented examples of known errors evidenced in
this report alone (aside from those unknown), it is clear that significant
errors in data recorded within Horizon have occurred.
Measures and Controls to prevent/detect/identify/report/reduce errors
in Horizon
5.155 There are various methods and controls implemented within Horizon by
means of electronic system checks and manual business processes
which sought to prevent/detect/identify/report and reduce errors in
Horizon. Namely (not exhaustive):
a. Reconciliation reports and processes (Section 6, Page 96)
b. Alert messages at the counter (Section 7.1, Page 118)
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 86 of 225
c. System alerts for Fujitsu (Section 8.1, Page 127)
d. Horizon Support Service Facilities, chiefly TfS and PEAK systems
(Section 4.66, Page 33)
e. Known Error Logs (Section 4.87, Page 38)
f. Software releases for bug fixes (Section 4.92, Page 39)
g. Audit Systems (Section 4.51, Page 30)
h. Journal Sequence Numbering (Section 4.52, Page 30)
Audits and Conformance to Quality Standards Checks
5.156 The Post Office Account Customer Service Problem Management
Procedure document ‘4 identifies the process metrics and key
performance indicators required for measuring the effectiveness of the
Process and service specifically in relation to Problem Management. The
Problem Management Procedure is set out in more detail at Appendix E
(Section 15.33, Page 194). Relevant to this section and Issue 6 are the
metrics and KPIs to measure/control and reduce the risk of failure to
detect, correct and remedy Horizon errors or bugs:
a.
Number and impact of incidents occurring before the root cause of
the problem is identified and resolved
Number of repeat incidents following any corrective action
Number of problem records arising from proactive actions and
trend analysis
Number of changes arising from proactive actions
Percentage of problem records without an action plan
Average length of time to resolve problems
Number of incidents closed without a KEL
*#4 SVMSDMPRO0025_5.doc, Post Office Account Customer Service Problem Management Procedure, 12 July
2016 [POL-0146787] [fe6a96a3615a63e094682c7efaa33090]
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 87 of 225
h. The number of Problem Records arising from Managed Change
(MSC, CP) activities
i. The number of Problem Records arising from the implementation
of new services / major releases.
5.157 From the above, it iAmy opinion that Post Office should be aware of all
recorded bugs/errors/defects in addition to those previously
acknowledged by them, from the process metrics compiled above.
5.158 Of specific interest is the POL monitor that tracks the number of records
arising directly as a result of managed change activities‘*5. No disclosed
logs have been found in respect of these problem records that are listed
as being reported monthly.
5.159 Requests have been made in relation to making such Management
Information reports. At the time of writing, these have not been made
available for analysis.
5,160 The Witness Statement of William Membery!**6!147 (Pos Head of Quality
and Compliance) states:
“The Horizon system (both Horizon Online and Legacy Horizon) are and
have been subject to audits to internationally recognised standards.
These audits check both technical aspects of the system and the working
practices of Fujitsu around Horizon. They provide assurance that Horizon
and Fujitsu are working within a robust system of contro! measures. The
audits review both the design of the controls and their implementation in
practice. Completion of these audits provides assurance to Fujitsu, Post
Office and other third parties (e.g clients of Post Office) that the financial
information within Horizon can be relied on.”
445 SVMSDMPRO0025_5.2.doc, Post Office Account Customer Service Problem Management Procedure - Version
486 2, 15 September 2017, (1.4 - Process Metrics and Key Performance Metrics), [POL-0512874]
[dbe89de2e88cfb6494d3820228e17f0e]
+87 Witness Statement of William Membery - 28.09.18.pdf
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
Page: 103
FUJ00183797
FUJ00183797
Number: 1_ Author: Gareth Jer
Date: 24/10/2018 16:41:00
Do we share PEAKS with POL?
Probably need CS to comment on this.
Number. 2_Author: Simpkins John Date: 05/11/2018 18:16:00 Z
Not Peaks, however TfS/Powerhelp were shared.
Number: 3 Author: Gareth Jenkins Date: 24/10/2018 16:42:00
isn’t he still Fuitsu?
Number. 4 Author: Simpkins John Date: 05/11/2018 18:17:00 Z
Yes
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 88 of 225
5,161 Whilst both Horizon and Horizon Online contain a number of measures
and controls designed to check system integrity, these mechanisms
have been shown to have failed. Thi is a point agreed upon in the Joint
Statement. It has been identified that known issues/bugs were often
deferred and dealt with on a cost/benefit basis. The Risk and Compliance
Committee meeting minutes of 18 September 2013 *4® highlight an
instance particularly in relation to an audit performed by Ernst & Young:
"It was reported that following the recent Ernst & Young external audit
four risks [sic] been identified. Three of the risks raised had been
addressed however the final risk, relating to the communication by Fujitsu
of changes made to the Horizon system, was still outstanding.
It was identified that it would cost over a £1m to implement the mitigation
being suggested by the audit and that this was not proportionate to the
risk being managed”.
5.162 The Ernst & Young management letter arising from the 2011 audit‘4?
150 recommended changes to strengthen the change management
process. They noted that POL was not usually involved in testing fixes
or maintenance changes to the in-scope applications (which includes
Horizon Online) and they were unable to identify an internal control with
the third-party service provider to autorise fixes and maintenance
changes prior to development for applications. The risks were outlined
as follows:
"There is an increased risk that unauthorised and inappropriate changes
are deployed if they are not adequately authorised, tested and approved
prior to migration to the production environment”
488 R&CC Minutes 18th September 2013.docx, Risk and Compliance Committee (R&CC) Reference:
R&CC/MIN/SEP13, 18 September 2013 [POL-0217378] [4d23226da8aca4bc0aa3940b91450325]
+8 POL Management Letter FINAL.docx, Post Office Limited - Management Letter for the year ended 27 March
180 (Section 4 - Current Year Recommendations - IT Specific) [POL-0219218]
[9d7862698d2a0f6af2a3c55590763bb7]
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 104
Number: 1 Author: Simpkins John Date: 05/11/2018 18:18:00 Z
We really need to review this document as itis being taken as an authoritive point of view.
ge Number: 2_Author: Gareth Jenkins Date: 24/10/2018 16:44:00
Tthought POL always authorised MSCs associated with upgrades?
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 89 of 225
5.163 Further, as referenced at 5.188, there is evidence that despite procedures
being in place, these were not being followed by Fujitsu.
5.164 In the circumstances, whilst bugs/errors/defects may have been
identified, they were not necessarily fixed instantaneously, ancontrol
was not retained over the process of fixing. The risk of error was
therefore not reduced as far as possible, but ratPkr it was subject to a
commercial assessment.
5.165 It is acknowledged that simple fixes ought and were implemented to
either fix bugs or provide additional data validation checks
pothapragadac4359R!°! & Marris4123N. ‘52
5.166 Review of ‘S80 Release Note - Deferred PEAKS List - Counter? which is
an addendum to the S80 Release Note (dated 2005) details the PEAKS
which remained outstanding once S80 was to be implemented (this is
pre-Horizon Online rollout and suggest that only PEAKs that impact the
Horizon counters are included). Whilst many are cosmetic low severity
items, pcGh21925 reports an incident caused by a problem in Riposte
which is an “existing and very intermittent” problem in ‘Live’ (it is
assumed this relates to the Production Horizon Live data environment).
It further states that the underlying cause would require Escher to
deliver software code not available for S80 or S80R (suspected further
release). The document further states that “Development will continue
to investigate to identify whether a ‘work round’ [sic] change” could be
made in the code.
51 Pothapragadac4359R.html, HNG-X KEL pathapragadac4359R, 19 April 2011 (last updated 16 August 2011),
[POL-0038695] [f0a4dd0a787be69689f36b84a2b5753c]
452 MArris4123N.html, HORIZON KEL MArris4123N, 28 November 2003, [POL-0033810]
[9b0bb96d10e84cc47f2cf4bd0d271f37]
453 CSRENO32_1.doc, S80 Release Note - Deferred PEAKS List - Counter, 13 October 2005 [POL-0083919]
[1aa524ee9a53955ee392e5621f0eeddf]
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 105
Number: 1_ Author: Gareth Je
~ Why is he saying this?
Number: 2_Author: Simpkins John_Date: 05/11/2018 18:21:00 Z
Tagree that they could not be fixed immediately due to regression tests etc, however I do not agree that control was not retained. One for Release Management?
Number. 3_ Author: Gareth Jenkins Date: 24/10/2018 16:45:00
Tp the real world, everything ist
Number: 4 Author: Gareth Jenkins Date: 24/10/2018 16:47:00
Need to Took at that.
Date: 24/10/2018 16:45:00
Number. 5 Author: Simpkins John Date: 06/11/2018 10:26:00 Z
This was due to an issue in Riposte Notify mechanism not letting the code know when it failed, the fix was finally performed by Escher.
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 90 of 225
Failures of measures and controls to prevent/detect/identify/report/reduce the
risk of errors
5.167 Whilst controls were in place, as per the Joint Statement, it is agreed that
evidence exists to show that these automated mechanisms have failed
to detect and record anomalies. Similarly, applied manual processes are
reported to have been subject to variations of quality (operation outside
of process).
5.168 ‘An internal presentation from the Post Office'5’? shows the helpdesk
error processing workflows (at slide 9). Whilst one of these work flows
shows the process when an error is detected internally by the Post
Office, the four other workstreams deal with errors from different
external sources, namely:
a. “Branch discovers error/unbalance and calls NBSC”
b. “Client discovers error”
c. “Customer discovers error and call customer care”
d. “Branch discovers error and calls FSC directly”
5,169 The concern with the above is that if the Branch, Client or Customer do
not detect and therefore do not report the error, then thilcould well
have an impact upon Subpostmaster branch accounts without their
knowledge.
5.170 “Review of Key System Controls in POLSAP”*5®, an internal Post Office
document, highlighted procedural issues occurring in respect of
transaction data, access to software and change management. The risks
and implications identified included errors not being identified or
resolved leading to potdatial discrepancies in client balances.
458 Finance L3 pack.pdf, Business transformation Finance Wave 1, 1 October 2014 [POL-0218441]
[a26a0ec0f6c8dbc20b753b200de725bb]
155 AR12.037.ppt, Review of Key System Controls in POLSAP, November 2012 [POL-0217341]
[dc5bSce76e817ce54cadb2d511¢4ce11]
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 106
Number: 1_ Author: Gareth Jen!
Yes this is possible
Date: 24/10/2018 16:49:00
However there were a number of other ways that Fujitsu detected errors based on many checks and balances we had in place, for example monitoring Events and Failed
Recoveries and reconciliation reports.
jyNumber: 2_Author: Gareth Jenkins Date: 24/10/2018 16:50:00.
“Possibly. However this would have no impact on Branch accounts or customers
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 91 of 225
5.171A Detica NetReveal report issued in 2013*°¢ indicates, in its key findings
at section 1.2, four areas contributing to risks in relation to policy,
processes and information used by the Post Office to track and manage
the compliance of branches:
a. Widespread non-conformance to Post Office policy and processes
by branches, with an institutional acceptance that errors,
workarounds and non-conformance exists;
b. Complexity and fragmentation of information systems which
hamper efforts both to gain an insight into branch behaviour and
root causes;
c. Ineffective process, policy and working practice in the central
operational teams to gather information, prioritise and act in a
coordinated manner;
d. Technology available to central operational teams was not fit for
purpose; analysis of large data sets in performed on an ad-hoc
basis of data subsets copied into Excel and tasking of teams is
initiated and managed through email.
5.172 Given that process, and policy adherence is crucial for effective
management and resolution of shortfalls and discrepancies within
Horizon, the report indicates that levels or risk were not managed as
appropriately as was possible.
5.173 Referring to the instance identified at paragraph 5.49 above; it is
observed that additional human operated manual processes could
contribute to electronic errors affecting branch transactions.
5.174 The End to End Reconciliation Reporting document from 27 February
2012*°” states:
158 NRRA1207 10D007-0 50 Draft.doc.docx, Fraud and Non-conformance in the Post Office; Challenges and
Recommendations, 1 October 2013 [POL-0216106] [f41e9587582dc4691967c8e22d4aa64e]
157 SVMSDMSD0020_2.2.doc, End to End Reconciliation Reporting, 27 February 2012 [POL-0124572]
[aaedd5051156619c50296d04f6b2e779]
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 92 of 225
“Thdke is no formal reconciliation produced between the POLSAP System
and the Credence transaction stream. The Cre#ence stream should
therefore not be used to verify financial integrity and Post Office should
ensure the POLSAP System Transaction information is used for this
purpose."
5.175 The report regarding the reversal dispute conducted by Helen Rose
states:
"On kboking at the credence data, it clearly indicates that the reversal was
completed by JAROO1 (Subpostmaster) at 10:37 04/10/2012 and was
reversal indicator 1 (existing reversal) and settled to cash.”
5.176 It is therefore relevant to question whi Post Office were using Credence
data to initially investigate disputed transactions. Whilst it is evident
that it was understood by Post Office in this instance to request
assistance from Fujitsu for further material to investigate this dispute,
there appears to be further issues with the data provided by Fujitsu.
5.177 Observations of the disclosures illustrates that the initial report states "a
transaction at 1082", whereas the credence data file shows 10:32 with
the reversal at 10:37. Fujflsu's data states the transactions are at 9:32
and 9:33 and reversal timestamp is 9:37.
5.178 Whilst this hour difference between the data sets might be easily
traceable for Fujitsu, it iQ not clear how easily it would have been to
investigate issues where the Subpostmaster was not sure of what time
things went on erroneously in the system, or that it was a reversal
specifically.
5.179 Further to this point, an Ernst & Young review of Post Office’s systems of
internal control conducted in March 20111 identified (amongst other
issues) various iss&es with the Credence application.
458 POL Management Letter FINAL.docx, Management letter for the year ended 27 March 2011, August 2011
[POL-0219218] [9d7862698d2a0f6af2a3c55590763bb7 }
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 108
Number: 1_ Author: Gareth Jer
Date: 24/10/2018 16:55:00
“Rs part of IMPACT we argued quite strongly that this was unnecessary as we didn't want to be processing reconciliation errors from such a process. With hindsight, perhaps this
was wrong!
Number. 2_Author: Gareth Jenkins Date: 24/10/2018 16:56:00
Gredence was always intended to be “Management Information” (it was originally POL MIS). it was always the intention that POL SAP was the definitive Financial data
coNumber 3 Author: Gareth Jenkins Date: 24/10/2018 16:58:00
However when I looked at it further, this Reversal was done as part of Recovery and so by the system when User JAROO1 logged on. The interface to Credence didn't support the
flag that indicated this.
have the data for the original analysis on my Hard Drive and could probably dig out the emails.
»,Number. 4 Author: Gareth Jenkins Date: 24/10/2018 16:59:00
Because POL SAP only has aggregated data and not individual transactions,
Number: 5 Author: Gareth Jenkins Date: 24/10/2018 17:01:00
Don’t know where that came from ~ typo for 10:32?
Number. 6 Author: Gareth Jenkins Date: 24/10/2018 17:00:00
Our Data is held in UTC, so during the summer the timestamps differ by 1 hour from BST. We adjust the imestamps on passing data to POL (and other external systems), but
retain the UTC times in our logs.
Number: 7 Author: Gareth Jenkins Date: 24/10/2018 17:03:00
“We have done fairly well with vague timest
= Number: 8 Author: Gareth Jenkins Date: 24/10/2018 17:04:00
issues with Credence have no impact on Branch accounts and so are irrelevant
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 93 of 225
5.180 Paticularly, weak change controls within the back end of the systems
allowing Logica developers (the third-party provider) to move their own
uncontrolled changes into the production environment, which I
understand to be both Credence software code and the data within
Credence used for audit evidence. Further documentation to approve
fixes and patches applied to Credence outside of the release process
were lacking, therefore linking changes to issue tickets to record the
original request for the bug fix was not possible.
5,181 Front end change process weaknesses were also observed. The following
noted:
“During our walkthrough of user administration of the front end of
Credence we noted several users with administrator rights, including
some generic users (this is noted below as a separate point). These users
have the access rights to create and amend reports, including those which
may be relied upon for audit evidence. These users can change report
design, and processing without documented request, test or approval.
When users have the rights to change reports that are used by the
business for reconciliation, exception reporting or other processing, there
is the risk that the reports are manipulated either intentionally or
accidentally.”
5.182 This is consistent with my opinion 5. in regard to manual fixes where
all manual entry activities enhance the likelihood and potential for error.
5.183 Poghy handled changes introduced by Post Office to processes in Horizon
have also caused errors in accounts. A presentation'®® made to the Post
Office Operations Board in 21 July 2017 explained (at page 86) that
changes were instructed (in 2015) as to the way that “MoneyGram” is
dealt with by the Subpostmasters. These changes included acceptance
of Debit/credit card payments. The presentation reported the impact of
this new working practice, including;
159 Operations Board 21 July 2017.pdf, [POL-0221328] [9c45e0be3ff2b6773447cc6e41db5f46]
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 109
Number: 1_ Author: Gareth Jer
Date: 24/10/2018 17:08:00
Nothing to do with Fujitsu and irrelevant to Branch Accounts.
Number: 2_Author: Gareth Jenkins Date: 24/10/2018 17:10:00
This ref doesn’t make sense.
Number: 3 Author: Gareth Jenkins Date: 24/10/2018 17:11:00
POL need to address these. MoneyGram is now processed by CPD using POL AP ADC scripts
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 94 of 225
a. “Branches retrying transaction that had failed due to timing out
b. Branches reversing a transaction but not cancelling the AP part of
the transaction
c. General lack of understanding in the Branch of how the process
works, especially for refunds...”
5,184 The same presentation also referenced that “Cafllelot - Operation
simplification has gone live... early indication that this has accounted for
50% of the issues”. No detail has been provided as to what the Camelot
issues were, but up until this point in the chronology Camelot was shown
as the most frequent cause of Transaction Corrections being raised to
modify branch accounts.
5.185 Th&l witness statement of Akash Patny ‘© refers to his experience of
imbalances whilst dealing with Moneygram transactions. This witness
statement supports highlighted reconciliation issues due to timeouts
occurring between Post Office and Moneygram'*. As acknowledged in
the document this could affect branch accounts due to reconciliation
differences.
Supporting Documentation
5.186 actfh2230K 182 highlights a problem with additional checks which were
implemented to identify system errors/inconsistencies when balancing
during branch In this instance the additional checks caused account
balancing errors on rollover. The note issued by SSC states, "This
should never happen - something has gone horribly wrong. Or possibly
the checks haven’t been implemented as intended.” This suggests a
160 Witness Statement Of Akash Patny, 28 September 2018, (MoneyGram Issue ~ Page 3)
161 Items at Half-Year.docx, Watchlist Items (continuing from Q1/2), 2016, [POL-0220630]
[7ac18146eb4f0ced505db4b34bac6724]
462 acha2230K.html, HNG-X acha2230K, 18 October 2013 (last updated 25 October 2013), [POL-0039583]
[49a4bca2b1c898a29ea22233960574d9]
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Date: 24/10/2018 17:13:00
“One for POL. I would guess this was introduction of TAs for Camelot terminals, That would certainly redcue TCs.
»Number: 2_Author: Gareth Jenkins Date: 24/10/2018 17:14:00
Fis issue wasn't actually related to MoneyGram, but the fact that a Card Payment for the MoneyGram deposit failed and the basket was settled to Cash. No attempt was made to
reverse the MoneyGram transaction. This may be due to deficiencies in the AP ADC scripts. Horizon appears to have behaved correctly.
,Number: 3 Author: Gareth Jenkins Date: 24/10/2018 17:17:00
We need to look at this,
Number. 4_ Author: Simpkins John_Date: 06/11/2018 10:52:00 Z
C0223870 applies, he is not reading the KEL correctly, these two additional checks were added after the issue identified in this Peak. This KEL was then raised to explain the new
events. A quick search of Peak shows no tickets have been raised for these events in live
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 95 of 225
recurring issue since the checks themselves appear to have failed. It is
not known how or/if this was resolved going forward.
5.187 dsed2049S '* highlights the lack of system communication and/or
support communication in respect of certain system features which
could subsequently result in errors. In this case withdrawn products
were converted to cash on rollover but the loss was carried forward into
the next period instead of being dealt with there and then. The bug
continued until a fix was applied 6 months after the KEL was raised
(ensuring an alert was raised warning the Subpostmaster of the
occurrence). However, it is further noted in the log that POL failed to
tell branches to “rem out” some products before they were withdrawn
leading to a cash loss that would not become apparent until the stock
unit was balanced again.
5.188 Fujitsu themselves were also open to mistakes. It is recorded that a
Fujitsu engineer failed to follow the correct process when replacing a
branch terminal resulting in multiple customer bills not being paid'!®.
Th&lpeak log states:
"I believe that we shall have to confess to POCL that we have not followed
the correct procedure and we should advise that POCL make manual
payments.
5.189 The reconciliation process used by POL to assist with identifying any
accounting differences is not able to easily identify genuine differences
and/or differences resulting from external APS transactions from old
trading dates. This is evidenced in act3250R 165 196 which, when
submitted to the development team, was considered too complex to fix
and was subsequently closed. This would add greater complexity to
163 dsed2049S. html, HNG-X KEL dsed2049S, 14 June 2011 (last updated 03 October 2013), [POL-0039549]
[34dfdbd02c7e909cbbefaebd5a4743df]
468 pc0093382.html, Peak Incident Management System, 5 August 2003, [POL-0265630]
[c3393ab177a14280dfe535b7b34282a6]
165 acha3250R.html, HNG-X KEL acha3250R, 14 February 2013 (last updated 10 February 2015) [POL-
166 ] [3675c6282d759025c44475aeed0b6c70)
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Number: 1_ Author: Gareth Jenkins Date: 24/10/2018 17:18:00
‘This was the agreed backstop for how to handle stock associated with products that had been withdrawn when the branch refused to remit the remaining stock back to POL.
Number. 2 Author: Gareth Jenkins Date: 24/10/2018 17:20:00
Did we actually retrieve the terminal and get the bens (and so know what they were)?
Number: 3_Author: Simpkins John_Date: 06/11/2018 11:06:00 Z
PC0093382 applies, this is where the engineer did not follow process on a SCO:
This is a single-counter office. That means that the counter has two physical discs in it which hold two separate copies of the message store. One of these
is the manin hard disc and the other is called a mirror disc, This is
constructed so that it can be easily pulled out of the base unit and can be pushed into another base unit.
In the event of a breakdown, e.in the comms card, then the engineer is supposed to take a new base unit to the office, extract the mirror disc from the old unit and put it into
the new base unit.
In that way ALL messages are safe.
On this occasion (see E-0307020207) the engineer found that the comms was not working so he replaced the complete base unit including the mirror disc. This
has caused the messages to be lost.
I cannot view the PowerHelp call to see what happened at the branch and there were no new Peaks soon afterwards.
Number. 4 Author: Gareth Jenkins Date: 24/10/2018 17:22:00
Better look at this one.
«Number: 5 Author: Simpkins John _Date: 06/11/2018 11:11:00 Z
Peak PC0223610 applies. This is only relevant to the APSS2136 report
I noticed recently thet the APSS2136 report, which does a three-way comparison between counter, TPS and APS totals, is showing differences almost every day. TPS and APS are
normally the same, but the counter total is different. However there aren't any entries on the TPSC250 report, which also looks for differences between TPS and counter totals (all
transactions, not just APS)
Having checked the February data, I've found that the differences are due to external APS transactions relating to old Trading Dates.
Ideally these transactions arrive and are put into BRDB with the same Journal date and Trading Date - they should relate to transactions done that day. But sometimes they are
delivered late. They are allocated to the correct (old) Trading Date for the "Counter" total (actually totalled on BRDB). But they have to be included in current day's APS and TPS
files, and are included in the current day APS/TPS totals.
This peak is not straightforward fix. It involving analyse transaction data from BRDB,APS and TPS. Besides no one is bothered by this report and this error has been already
KELLed,
I'm return this peak.
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 96 of 225
managing the reconciliation process and having an accurate picture at
branch level of APS transactions.
5.190 There are various documented examples of circumstances in which
Horizon erroneously created duplicate Journal Sequence Numbers
(“JSN”) against the same traHSaction. 1°7:168.169170 JSNs should be (by
design) always unique as explained by Gareth Jenkins’: *7*
“At this point a check is made that indeed there are no missing or
duplicate jsns for any counter and should any be found an alert is raised.
Note that this could only happen as a result of a bug in the code or by
somebody tampering with the data in the BRDB and this check is included
specifically to check for any such bugs / tampering”
5.1911t & not currently clear how it would be determined whether duplicate
JSNs arose because of a bug or tampering. Hoflever, the existence of
duplicates in the first instance is in itself a failure of measures and
controls.
5.192 Aside from instances where bugs/defects and errors are recorded to have
affected Horizon branch accounts, a document entitled ‘Transaction
Correction - Bramptom.doc!72’ indflates that due to a fraudulently
Programmed credit card, Subpostmasters cash declarations indicated
shortfalls. This affected several branches and it appears it was through
the Subpostmaster’s diligence that the issue was finally concluded.
5.193 Therefore, the assumption that POL’s processes would detect anomalies
by way of shortfalls/discrepancy investigations on behalf of the
187 GCSimpson2242L.html, HNG-X KEL GCSimpson2242L, 08 March 2006 (last updated 05 December 2010)
[POL-0038135] [cd3dcd109ec897a07328c70ca9d125d6]
480 MithyanthaJ1937S.html, HNG-X KEL MithyanthaJ1937S, 06 May 2010 (last updated 09 August 2016)
[POL0040508] [d56a274636043f38e96d2e9f3609d949]
1 DRowe5415Q. html, HORIZON KEL DRowe5415Q, 08 October 2002 (last updated 14 December 2012) [POL-
170 } [4e885614e11d9c4abf33cfbo4féef86a]
17 Horizon Online Data Integrity for POL, Gareth Jenkins.pdf, 2012 [POL-0021989]
[8dbd13d4aae9179dcaea4f62f3ab3572]
172 Transaction Correction - Brampton.doc, Email correspondence between Mark Baker and Karen Arnold,
March/April 2008 [POL-0018081] [7754e449931156b8339040a17cd3f3cb]
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
Page: 112
FUJ00183797
FUJ00183797
Number: 1_ Author: Gareth Jer Date: 24/10/2018 17:23:00
“Tat and ard refs are wrong as dated pre HNG-X
Need to check them out though.
We did have a bug in this area in the early days of HNG-X and the date of Jeevan’s KEL suggests that this was it!
Number: 2_Author: Simpkins John_Date: 06/11/2018 11:18:00 Z
GCSimpson2242L is a duplicate APSSEQ nota duplicate JSN.
DRowe5415Q is also a duplicate APSSEQ,
MithyanthaJ1937S was for a duplicate ISN. PCO204310 applies.
Number: 3 Author: Gareth Jenkins Date: 24/10/2018 17:26:00
itwas a bug.
Number: 4 Author: Gareth Jenkins Date: 24/10/2018 17:26:00
Agreed and that is why it was fixed urgently.
Number: 5 Author: Gareth Jenkins Date: 24/10/2018 17:27:00
“Not sure how this could happen. We need to investigate,
Number: 6 Author: Simpkins John_Date: 06/11/2018 11:34:00 Z
No references and cannot find a related Peak
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 97 of 225
Subpostmaster apBbars to be incorrect. It is evident that in a number
of circumstances it is the Subpostmaster who highlights the anomaly
and pursues rectification often after disputing the initial response from
Post Office. This is further reinforced in consideration of the Common
Issues Claimant’s witness statements.
5,194 In a recent strategic and financial plan‘”? it Fas acknowledged that
Horizon was designed two decades ago for an environment very much
different to the one we have today. In addition, both branch and back
office systems were also acknowledged as being end of life. A recent
Board Paper’ also concluded that it was operating with a far higher
operational risk profile than is accepted by the Post Office when making
investment and project decisions. It is difficult to see how the Post Office
could improve things further and mitigate against risks of issues
impacting on branch accounts without a substantial investment in
modernising their IT systems.
5.195 The Pod Office cash management proposals contained in a report dated
4 August 2017175 suggests that they were actively considering ways to
improve processes impacting on many of the issues raised above. It is
my opinion that, whilst the Post Office was looking at ways to improve
cash management, it is also indicative that the system was generally
far from perfect and there existed a real risk of bugs/errors/defects
adversely impacting on branch accounts despite the processes in place
at the time to prevent this.
17? Post Office Strategic and Financial Plan 2021 ~ Board approved & final.pdf, Post Office Strategic and
Financial Plan 2018-2021, November 2017, [POL-0219032] [90e37c6a 1ff574eb1d8f8007f26ed336]
175 Back Office Tower Transformation - Board Decision Paper 24 Nov 2016_v02.docx, 29 September 2016,
Page 4, [POL-0221162] [78261d52e6ce57¢368546592c99fadf5]
178 Cash Management Programme -Business Case June 2017 v1.5.docx, Business Case - Cash Management
Programme, 04 August 2017, [POL-0221342] [2687475ac747f28a22c51c53bbc50f30]
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 113
Number: 1_ Author: Gareth Je
“This is too strong.
Date: 24/10/2018 17:28:00
Number: 2_ Author: Simpkins John Date: 06/11/2018 11:35:00 Z
‘Agreed, if this was true for every Peak indent there would be one or more tickets raised at the branch.
Number. 3_ Author: Simpkins John_Date: 06/11/2018 11:38:00 Z
Horizon was designed 2 decades ago, this was replaces 1 decade ago with HING-X, which in tum has been upgraded in both hardware and software several times since and we
are now piloting HNG-T.
Number. 4 Author: Gareth Jer
Date: 24/10/2018 17:29:00
This doesn’t impact on Branches, unless a change is made to how Auto Rems operate.
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 98 of 225
Opinion Summary
5.196 In light of the above observations and findings aside from the
acknowledged bugs/errors/defects that Post Office have recognised, I
opine that it was highly likely for bugs/errors/defects to have the
potential to both (a) cause apparent or alleged discrepancies or
shortfalls in relating to Subpostmasters’ branch accounts/transactions
and (b) undermine the reliability of Horizon to accurately process and
record transactions.
5.197 As highlighted by select Known Error Logs (KELs), is:
been fixed recurred in different circumstances, therefore the reliability
s thought to have
of Horizon to accurately process and record transactions is questionable.
5.198 It is clear that in some instances it is not always apparent whether
recurring discrepancies were as a result of system bugs or the
Subpostmaster’s own actions, or other things beyond the control of the
Subpostmaster. ‘77178 Hofever the fact that the SSC support team were
unable to assist or identify the root cause does undermine the credibility
of Horizon itself.
5.199 Whilst both Horizon and Horizon Online contain many measures and
controls for ensuring system integrity, these mechanisms do/have
failed. It has been identified that kn&hvn issues/bugs were often deferred
and dealt with on a _ cost/benefit basis. Therefore, whilst
bugs/errors/defects may have been identified, they were notthecessarily
fixed instantaneously, therefore the risk of error was not reduced as far
as possible, but commercially assessed.
5.200 Similarly, processes designed to ensure controls and measures were
effective are identified as requiring improvement.
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Number: 1_ Author: Gareth Jenkins Date: 24/10/2018 17:31:00
Tthink in most cases it was different issues with similar symptoms.
Number. 2_ Author: Gareth Jenkins Date: 24/10/2018 17:32:00
We have to challenge that.
Number 3_Author: Simpkins John Date: 06/11/2018 11:41:00 Z
£-0602230704 relates to Peak PCO132673, This was investigated by Anne and a whole lot of PM miss-practices were listed
Ihave checked very carefully and can see no indication that the continuing discrepancies are due to a system problem.
Ihave not been able to pin down discrepancies to individual days or stock units because the branch does not seem to be operating in a particularly organised manner. In
particular I have noted
1, There are 6 stock units for this 3 counter branch, which seems a bit excessive.
2. The loss in euros in TP 9 appears genuine - the declared quantity was 4000 fewer than the system expected. It is not clear from the information above whether anyone found
out why this happened (there were several rem outs, and a rem in, on 23rd Dec - did the pouches contain the declared number of euros?)
3, Stock is sometimes transferred out of a stock unit where it is not held. in particular there were several transfers out of stock unit SMI in TP 10. At the end of the period the
stock figures were corrected back up to zero via Adjust Stock. This gave a gain of over £2000 in SMI. Equivalent negative stock adjusts in AA gave a corresponding loss in AA.
4, 1am not confident that the stock declarations are always correct e.g. at the end of TP 9 there was a declared holding of 5 £20 PO phonecards in the branch, then a few days
later 20 were transferred from one SU to another. None were remmed in until a week after that.
5, The branch had declared 27 £20 Argos vouchers at the end of TP 9. Branches have now been instructed to rem out this product; they remmed out 17 and adjusted stock to
account for the remaining 10 (so did they really only have 17 to start with?). This has correctly caused a loss of £200 in SU AA.
6. Lottery instants sales are entered onto the system as a single transaction every 10 days or so.
7. Stock units SMI and AA rolled over with non-zero cheque holding, This may be to do with how the discrepancies have been accounted for but I do not really understand this,
(the total is greater than the sum of the branch adjustments for TP 9 and 10).
recommend that this call is passed back to NBSC tier 2 for further investigation,
want the above information in an email, let me know.
ce there is no evidence that the discrepancies are being caused by a system problem. If you
The SSC did not identify a root cause probably because there was no software issue and more training and help was needed by the PM.
£-0402251077 applies to Peak PCO099954. This was also investigated by Anne and again found some sloppy declarations and no system issues:
checks are ok. Cheques are being handled correctly (except for 10th Feb when
the clerk forgot to cut off the report - but this didn’t cause a
discrepancy). Cash declatations look ok, they usually use drawer id 11
Occasionally they have used a different drawer id, this can lead to amounts
apparently doubling on the cash flow report, and should be avoided. But again
it will not cause a discrepancy
Checking the cash transactions on the system against the declarations shows
that they are not working particularly accurately (i. at the end of the day
the cash they declare in the drawer is tens, hundreds or thousands of pounds
astray from what has been recorded on the system). Itis possible that they
are not accurately recording all transactions on the system.
There is no evidence whatsoever of any system problem. I've mentioned this
outlet to Julie Welsh (Customer Services) who will try to get POL to follow
it up, but in the meantime please tell the PM that we have investigated and
the discrepancies are caused by the difference between the transactions they
have recorded on the system and the cash they have declared, and are not
being caused by the software or hardware.
Number. 4 Author: Gareth Jenkins Date: 24/10/2018 17:32:00
But not those that impacted the accounts and balancing.
,Number. Author: Gareth Jenkins Date: 24/10/2016 17:33:00
~~ Thatis not practicalin the real world
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 99 of 225
177 Call Details E-0602230104.htm, Post Office Account $70 Archived. 1 Call E-0602230104, 23 February 2006
[POL-0011223] [ccOec? afd6adfe07758adcfa4f457 3ca]
178 105000759.pdf, Post Office Account NWBO1 Archive4.1 Call E-0402251077, 25 February 2004
[POL0019675] [f513c96c9a72b75c3bb263c07a851a69]
6. Reconciliation and Transaction Corrections
Issue 5 - How, if at all, does the Horizon system itself compare transaction data
recorded by Horizon against transaction data from sources outside of Horizon?
Issue 15 - How did Horizon process and/or record Transaction Corrections?
6.1 The process by which Post Office compares transaction data recorded by
Horizon against transaction data from sources outside of Horizon is
known as Reconciliation. End to end reconciliation is the mechanism by
which Post Office establish which transactions are complete and correct
and which are not.!7
6.2 Post Office states‘”” that:
”..each and every reconciliation error is the result of some systems fault.
That fault might, for example, be a software fault (introduced through
either design or coding), a system crash, or a telephone line being dug
up. Such faults may affect transactions, thus it is the job of the
reconciliation service to detect when and how any transaction is affected
by any system fault.”.
6.3 In the same document Pod Office state:
“not all system faults will lead to corrective action and this is generally
done on a contractual and/or cost benefit basis”
6.4 Reconciliation reports and the known system components to facilitate it are
set out further in Appendix E of this report. For a diagrammatic
representation of the business applications applied to reconciliation as
at 2010 see Appendix B (Figure 5). In summary, reconciliation is a
176 SVMSDMPRO0012.doc, Reconciliation and Incident Management Joint Working Document, 18 March 2013
[POL-0032909] [b13d82f1ad57d0 105cefbc3bfe7406c3 }
177 SVMSDMPROO012 - Reconciliation and Incident Management Joint Working Document.doc, Reconciliation
and incident Management Joint Working Document, 18 March 2013 [POL-0219191]
[7ccc36ff81ef72450f60a1275c1153a5]
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 115
Number: 1_ Author: Gareth Jenkins Date: 24/10/2018 18:10:00
Tthink we wrote it
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 100 of 225
complex process involving many system components of which there are
both electronic and human interactive process elements.
6.5 It should be noted that as Horizon progressed to Horizon Online the
reconciliation sydems and processes continued to be developed.
Horizon Architecture
6.6 Dated 2010, Horizon architecture diagrams show that there are four main
areas within the Horizon Architecture: !”°
a. _POEFS - financial accounting system based on SAP (later becoming
POLSAP)
b. Refdrence Data Proving - environment in which changes to
Reference Data are proved before releasing into live (Reference
Data controls things such as which products are sold, their price
and where in the menu hierarchy they are displayed).
c. Branches
d. Core Horizon - the central systems that support Horizon
6.7. Core Horizon Components for Transaction Processing:
a. Batching Services enable Post Office to send branch data (either
all transactions or in a summarised form) to external systems. It
also receives batch data from external systems for distribution to
branches. The systems to facilitate this within Horizon are:
i. TPS (Transaction Processing System) - provides daily data to
other systems including POLFS, POLMIS and HRYAP. Also
provides a feed to First Rate for Bureau transactions
ii. APS - (Automated Payment System) - provides daily data to AP
clients (British Gas, BT etc).
178 POL-0003093.pdf, Horizon Architecture Diagrams (Page 3), 8 March 2010 [POL-0003093]
[00195d1cbb0017bd34e7c68b7930cicf]
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 116
Number: 1_ Author: Gareth Jenkins Date: 24/10/2018 18:11:00
“Yes, but they were predominantly the same,
Number: 2_ Author: Gareth Jenkins Date: 24/10/2018 18:13:00
No impact on the branches or their accounting, However itis used to manage Post Master's labilities and also to generate TCs
Number. 3_ Author: Gareth Jenkins Date: 24/10/2018 18:12:00
“Strictly this isn't part of the live system. Its used to validate Ref data in a “post dated environment, prior to being distibuted to Live.
Number 4_ Author: Gareth Jenkins Date: 24/10/2018 18:14:00
Monthly rather than daily,
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 101 of 225
iii. LFS (Logistic Feeder Service) - provides data on pouch
collections and receipts at branches to safhps on an hourly
basis. Also, nightly data on cash held in branches.
b. The systems that receive data from external systems within
Horizon are:
i. apa. receives customer and tariff data for Quantum and Water
Card service once per day
ii. LFS (Logistics Feeder Service) - receives planned order data
(once per day) and pouch contents information (potentially
hourly).
iii. RDMC (Reference Data Management Centre) - reddives Rates
and Margins data for Bureau service.
lon Reconciliation and Enquiry services for online authorisation Horizon
specific systems are:
i. DRS (Data Reconciliation Service) - reconciles individual
transactions for the DCS, ETU and Banking Services.
ii. TES (Transaction Enquiry Service) allows Post Office to query
transactions status for banking (only)
iii. DWH (Data Warehouse) corfdains banking, ETU and DCS data
iv. APS (Automated Payment System) which — reconciles
transactions between itself and TPS (Transaction Processing
System).
External Systems/Clients
6.8 As per POL-0003093 ‘7? (2010), Core Horizon communicates with the
following systems:
179 POL-0003093.pdf, Horizon Architecture Diagrams (Page 3), 8 March 2010 [POL-0003093]
[00195d1cbb0017bd34e7c68b7930cicf]
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 117
Number: 1_ Author: Gareth Jer
“Tater POL SAP.
Date: 24/10/2018 18:15:00
Number: 2_Author: Gareth Jenkins Date: 24/10/2018 18:17:00
This stopped in 2010 when we moved to HNG-X
Number. 3 Author: Gareth Jenkins Date: 24/10/2018 18:18:00
“iso changes to Ref data from MDM,
Number: 4 Author: Gareth Jenkins Date: 24/10/2018 18:18:00
Yes, but I don't think anything is done with it. Not really clear exactly what DwH is used for. May be worth getting Pete Jobson to make some comment,
Definitely no impact on Branch Accounts so can be ignored.
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 102 of 225
a. Banks (LINK (Vocalink) A&L (Santander), CAPO) for online
authorisation of banking transactions (oc for debit and credit
card authorisations) and transaction data used for reconciliation
b. Online Clients (e-pay, stramiine, DVLA) for online authorisation
of transactions and (for e-pay and Streamline) data used for
reconciliation
c. SAPADs - A Post Office system that handles cash and Foreign
Currency logistics. Data includes cash on hand statements from
each branch, planned orders, replenishment deliveries and
delivery/collection data
d. HRgaP - A SAP system that handles remuneration to the branch
franchises and ‘multiples’ such as Tesco.
e. POEIMIS - An Oracle based system to provide MI reporting to Post
Office.
f. First Rate - Provides bureau rate information. It is also passed all
bureau transactions to allow First Rate to undertake MI.
g. si&hens Metering - Provides Rates and Customer data for
Quantum gas pre-payment card.
h. AP Clients - Transaction information for Clients where payment
j.
information is collected by Post Office.
Royal Mail and Parcel Force Worldwide - track and trace
information for parcels and letters taken in branch.
rog - Post Office system that provides Reference Data
6.9 POL-02193191® illustrates a more detailed breakdown of external service
components. See Appendix B Figure 8.
489 HorizonContext.jpg, Horizon Service Context Diagram, 9 August 2018 [POL-0219319]
[0005427 fe3eec69b47bce277611d037]
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 118
Number. 1_ Author: Gareth Jenkins Date: 24/10/2018 18:20:00
~ Covered below
Number: 2_Author: Gareth Jenkins Date: 24/10/2018 18:20:00
Now Global Pay and also AMEX
Number. 3_ Author: Gareth Jenkins Date: 24/10/2018 18:21:00
"This is internal and now part of POL SAP
:«Number. 4 Author: Gareth Jenkins Date: 24/10/2018 18:22:00
internal? Iewas RMG, but I thought it was now in house.
Number. 5 Author: Gareth Jenkins Date: 24/10/2018 18:22:00
Internal. Now known as Credance
Number: 6 Author: Gareth Jenkins Date: 24/10/2018 18:22:00
This stopped in 2010
-Number 7 Author: Gareth Jenkins Date: 24/10/2018 18:23:00
Now MDM.
Internal
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 103 of 225
6.10 In 2012 Post Office described'*! the “Core transactional Finance Systems”
as:
181 Phase 2a) consolidated output.pdf, Finance Roadmap Project, 3 September 2012, [POL-0215782]
[ac4469b27e384fe4440f351c115d8108]
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 104 of 225
“... Essentially stable from a technical perspective but have been built up
organically overtime and are potentially integrated in an inefficient
manner with multiple interfaces” and ™... in some instances the systems
still do not provide the functionality required by the business and manual
interventions and rudimentary technologies have had to be implemented”
Transaction Acknowledgements
6.11 Some Post Office transactions (e.g. Lottery, Paystation, ATA) are not
transacted through a Horizon terminal but instead via separate
machine. However, cash taken, and stock vended for these transactions
needs to be accounted for on Horizon as part of the overall branch cash
and stock holdings. To ensure that Horizon is kept synchronised with
the records on the third-party equipment a ‘Transaction
Acknowledgement’ (TA) is often used. 1°?
6.12 Overnight the third-party equipment reports the vole / number of
transactions to Post Office.+®? Post Office’s data centre then sends an
overnight electronic message to each branch’s Horizon terminal which
contains details of the vollme / number of transactions conducted
within the branch on the third-party equipment. The) is the transaction
acknowledgement.
Reconciliation Processes
6.13 Reconciliation processes utilise a set of priftable electronic reports. ‘84
6.14 Each day, branch account transactions are harvested and processed
through the Horizon system in order to inform Post Office Finance of the
aggregated totals for products and services sold. This enables Post
Office to provide settlement figures to their clients. Reconciliation is
182 Factfile.DOCX, Initial Complaint Review and Mediation Scheme DRAFT Factfile, 16 April 2014 [POL-0022996]
[d24556f1009f1881f42a2dbbSb81 54de]
183 Factfile.DOCX, Initial Complaint Review and Mediation Scheme DRAFT Factfile, 16 April 2014
[POL0023002][Hash d24556f1009f1881f42a2dbb5b8154de]
184 SVMSDMPROO012_3.doc, Reconciliation and Incident Management Joint Working Document, 30 April 2012
[POL-0125134] [ad897ac9ff5edb2de37bfb8c4e9dc362]
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 120
Number: 1_ Author: Gareth Jenkins Date: 24/10/2018 18:24:00
Are these now covered by TAs? They weren't when I retired, but! know POL were talking about it
Also different ATMs behave differently.
«Number: 2_ Author: Simpkins John Date: 06/11/2018 11:57:00 Z
Tthink this was all regressed recently.
Number: 3 Author: Gareth Jenkins Date: 24/10/2018 18:25:00
And value
Number. 4 Author: Gareth Jenkins Date: 24/10/2018 18:25:00
Basically it is concerned with the net Cash / Cheques received rather than the number of transactions,
Number: 5 Author: Gareth Jenkins Date: 24/10/2018 18:26:00
However this information is not added into the Branch Accounts until acknowledged by a member of the branch staff at Log On the following day.
They don't have the option of not acknowledging it. However if the value is disputed they have the opportunity to take that up with POL
Number: 6 Author: Gareth Jenkins Date: 24/10/2018 18:28:00
They are also available in electronic form as spreadsheets, which is how they are normally processed.
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 105 of 225
therefore used by Post Office Ltd to provide financial and business
reconciliation at transaction level to demonstrate that each transaction
is complete and correct and report on any transaction that is not. !®
6.15 The Horizon system contains integrity checking functionality to monitor
transaction progression and integrity of transactions as they flow
through the system.
6.16 The ‘state’ of transactions is recorded as they travel through the system
with any exceptions (transaction anomalies) harvested along the way in
order that they can be dealt with using largely manual processes.
6.17 Various report sets facilitate the reconciliation process within Horizon for
both those transactions to/from approx. 136] external clients*% and
within the core Horizon components themselves (these are further set
out in Section 8 and Appendix E.
6.18 A Horizon Service Reconciliation Exceptions document states:
“Due to the potential dynamic nature of the Reconciliation Service, where
there is the potential for new exception types to be generated as a result
of software errors within new releases or reference data, it has been
agreed that these procedures will be documented outside of the formal
Reconciliation & Incident Management CCD document set.”
6.19 Where there is a need for manual intervention due to bugs/data
corruptions/incidents/errors, Post Office and Fujitsu teams interrogate
transaction data and reports to establish and m
data.
ify any erroneous
6.20 Reconciliation is therefore also used by the Reconciliation Service to check
that the corrective action is effective. '**
285 SVMSDMPROO012_3.doc, Reconciliation and Incident Management Joint Working Document, 30 April 2012
[POL-0125134] [ad897ac9ff5edb2de37bfb8c4e9dc362]
185 Witness statement of Angela Van-Den_Bogerd
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 121
Number. 1_Author: Gareth Jer Date: 24/10/2018 18:30:00
Tthought it was around 700. I suspect that giro bank is confusing things again
geNumber: 2_Author: Gareth Jenkins Date: 24/10/2018 18:31:00
No. WE don't modify data,
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 106 of 225
6.21 Since reconciliation facilitates the core operations of Post Office, there are
various services, departments and processes that comprise it. Many of
which have evolved through the lifespan of Horizon to Horizon Online.
Reconciliation Services/Departments
POL Finance
6.22 The POL Finance department is responsible for reconciling data within Post
Office central systems which relate to enquiries from Post Office
Clients'8”. They are also responsible for generating Business Incidents
in the event of error discovery.
6.23 In Horizon Online POL Finance use the CTS (Client Transaction Summary)
as the basis for settlement with Post Office Clients. If tHe CTS file is not
delivered by Fujitsu, POL Finance are to use the APSS2133b file to
manually calculate any settlement due.
Fujitsu Management Support Unit (MSB)
6.24 The MSU Unit are responsible for resolving exceptions arising within the
Horizon estate.
6.25 MSU are responsible for various reconciliation reports namely; "A&L
Reconciliation File Delivery Report”, “CAPO Reconciliation File Delivery
Report”, “LINK Reconciliation File Delivery Report” and the “DRS
Reconciliation File Delivery Report”.'®® The reports are produced For the
MSU, generated by system processing.
Fujitsu Third Line Support (SSC)
6.26 Within reconciliation, Service Support Centre (SSC) are responsible for
repairing any corrupted or exception transaction data within the Horizon
system.
187 CSPRO111_5.doc, TPS Reconciliation & Incident Management, 17 October 2005 [POL-0083720]
[2079d9210ce25t44a77a5See3efb67 eb]
488 NBLLDO79_0.1.doc, TES Reconciliation File Delivery Reporting Low Level Design, 20 August 2004 [POLO077442]
[8606de4f0ea60322502ed8a00de43b1e]
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 122
Number: 1_ Author: Gareth Jenkins Date: 24/10/2018 18:32:00
Presumably it is delivered almost every day? It would be very exceptional for it not to be.
Also this is irrelevant to Branch Operation
Number: 2_Author: Gareth Jenkins Date: 24/10/2018 18:33:00
Does this still exist? I thought this was now all done by the security team
Number: 3 Author: Simpkins John_Date: 06/11/2018 11:59:00 Z
Yes the Sec Ops team fulfil this role now.
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 107 of 225
6.27 The tools and methods by which this is carried out are covered under Section
9 below.
Reconciliation Exceptions
6.28 Where transactional data does not conform to its expected format or in
comparison to copies of itself or corresponding records then it causes a
reconciliation error.
6.29 Post Office acknowledge that errors may occur within counter transactions
or during the harvesting process. In addition to errors highlighted by
Fujitsu Services within the TPS Report Set, errors may also be
discovered by Post Office Ltd Finance (POL Finance) when reconciling
data within its central systems or which relate to enquiries from Post
Office Clients.
6.30 In September 2012 an internal Post Office document!®° reported; “There
is a data reconciliation service delivered by Fujitsu to reconcile the APS
and TPS streams. A suite of reports is produced”. Against this section
are the report are “Issues/Risks” which state; “Actions to resolve some
differences are not well understood and carlbe lengthy to resolve eg
BIMs". Post Office record the business impact as; “Manual posting are
needed. Protracted branch and customer enquires”.
6.31 Reconciliation and Incident Management documentation identifies that an
incomplete transaction is no@hecessarily a Reconciliation error, but it
might become one if it is not completed in a timely manner. °°
6.32 It also further states that (regarding reconciliation errors arising from system
faults):
189 Phase 2a) consolidated output.pdf, Finance Roadmap Project, 3 September 2012, [POL-0215782]
[ac4469b27e384fe4440f351c115d8108]
180 SVMSDMPROO012.doc, Reconciliation and Incident Management Joint Working Document, 18 March 2013 [POL-
0032909.doc] (b13d82f1ad57d0105cefbc3bfe7406c3]
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 123
Number: 1_ Author: Gareth Jenkins Date: 24/10/2018 18:35:00
Yes, because they occur rarely and are usually associated with system wide failures (rather than bugs)
az;Number: 2_ Author: Gareth Jenkins Date: 24/10/2018 18:36:00
Yes, This is because it takes a few days for the final reports to come from Global Pay and AMEX re the state oF Plastic transactions
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 108 of 225
“It icknowledged that not all system faults will lead to corrective action
as this is generally done on a contractual and/or cost benefit basis.”
Data Reconciliation Service (DRS)
6,33 The Data Reconciliation Service is the system used to reconcile transactions
carried out with Financial Institutions.
6.34 For a transaction to be reconciled and considered in a ‘complete’ state it is
necessary that the DRS finds three components to it:
a. C112 (Atransaction sent from TPS to Post Office Finance System and
received by the DRS)
b. C12 (Transformation of a Confirmed (C1) message as written to the
ppd)
c. C4 or D (C4 = Confirmed Client Transaction authorised, D
Transaction reconciliation difference highlighted and notified to DRS)
6.35 If one of the above components is incomplete, corrupt or duplicated it is
recorded as being in an exception or error state and should appear on
the Network Banking report NB102.1%!
6.36 The following reports are critical to reconciliation and are recorded as
produced daily:
a. NBOOO - DRS Summary
b. NB101 - Network Banking Settlement Statement
lon NB102 - Exception Summary
6.37 An F99 transaction is a transaction state that indicates that a reconciliation
error has been reported but POL has advised that the issue has
191 POL-0032841.doc, Network Banking Reconciliation and Incident Management Processes, 26 February 2003
[POL-0032841] [907539a6845da640795b670f2015199b]
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 124
Number. 1_Author: Gareth Jer
This is a repeated quote!
Number: 2_Author: Gareth Jenkins Date: 24/10/2018 18:38:00
For DCS itis 4)
Date: 24/10/2018 18:37:00
Number. 3 Author: Gareth Jenkins Date: 24/10/2018 18:38:00
“From BRDB
Number: 4_ Author: Gareth Jenkins Date: 24/10/2018 18:39:00
No. A Dis where the client is saying that they have rejected that transaction.
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 109 of 225
subsequently been resolved. This state is set using the DRS Workstation
application that is used by Fujitsu Security Operations team.‘
6.38 Post Office reported in response to my Request for Information that
10}
automatically reconciled. Such transactions require manual intervention
(00+ transactions per week suffer from problems and are not
for the reconciliation to take place. Where there is manual correction
applied within the system, there is the potential for input error that may
impact the financial status for the branch and/or end client.
6.39 Appendix G
[See Spreadsheet]
182 POL-0032990.doc, End to End Reconciliation Reporting, 4 September 2017 [POL-0032990]
[7f79ebcdead957d0d4019672976d25f4]
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 125
Number. 1_Author: Gareth Jer Date: 24/10/2018 18:40:00
This doesn’t correltate with those we report surely?
az;Number: 2_ Author: Simpkins John Date: 06/11/2018 12:02:00 Z
it seems way too many, as mentioned before check with Jason Muir
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 110 of 225
6.40 Appendix H provides a pictorial summary of the processing components
within Horizon which have been identified as the most common points
in which discrepancies can be captured.
Transaction Disputes
Enquiry Based Banking Transaction (1 eB )
6.41 Where Fujitsu has received notification from Post Office Ltd via the HSD that
it wished to query a particular transaction.
Disputed Banking Transaction Notice ( pBRW)
6.42 Raised by POL Finance via HSD after notification from either a Post Office
branch or Network Business Support Centre (NBSC).
6.43 Where Fujitsu has received notification from Post Office Ltd via the Enquiry
Service following a query by the ‘End’ customer relating to the state of
his / her account.
Reconciliation Summary
6.44 Although there are various integrity check points and manual processes
observed within the reconciliation process, this has been an evolving
Progression since Horizon was first introduced. Reconciliation processes
have developed further with the system and technological progression
of Horizon Online.
6.45 Whilst the reconciliation process within Horizon handles the integrity
checking of transaction data and potential anomalies through the
various report sets, it does not necessarily ensure all anomalies or
discrepancies are resolved as this becomes a more manual process.
Further, manual processes applied to correct data anomalies also have
the potential to introduce further errors.
6.46 In consideration that Branch account positions were interpreted and
reviewed from data flows through to Post Office back end systems
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 126
Number. 1_Author: Gareth Jer Date: 24/10/2018 18:42:00
Not familiar with this term or interface
az;Number: 2_ Author: Gareth Jenkins Date: 24/10/2018 18:43:00
‘Again not familiar with this
Are these things done through TES?
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 111 of 225
(which would determine whether Transaction Corrections were to be
applied), the following is considered relevant:
“POLSAP - Following investigation by Fujitsu, Logica and Ingenico, the
ing data within POLSAP.
was identified as out of range dates which failed the Credence validation
root cause of a long outstanding problem with mi:
(in excess of 90 days). Ingenico has corrected the data and P&BA has
advised that the mismatches have been cleared within the accounts. A
permanent preventative measure is now being developed by Ingenico, 4%
6.47 It is not clear whether or what branches this “missing data” might have
affected.
Transaction Corrections
6.48 As per Operations Manual version 7!9” the introduction of the new Post
Office Ltd Finance System (POLFS) in Product and Branch Accounting
(P&BA) Chesterfield, resulted in finance teams no Bbnger being able to
adjust client accounts on site.
6.49 Prior to 2005, branches were required to balance weekly and produced a
Horizon generated “Cash Account”. Disdtepancies were, with
authorisation from the Post Office, placed in the branch “Suspense
section” of their cash account. The discrepancy was held until enquiries
into it were concluded and it was then removed by the issuing of an
errbk notice (now known as a Transaction Correction) by Post Office or
by the Subpostmaster putting the money into the branch to cover the
loss or removing the value of the gain from the branch so as to balance
the account. 1%
196 10 Monthly Service Management Performance Period 10.xlsx, Monthly Service Management Performance
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 127
Number: 1_ Author: Gareth Jer
Date: 24/10/2018 18:44:00
~ Sounds like a PayStation issue so nothing to do with us.
Number: 2_Author: Gareth Jenkins Date: 25/10/2018 18:35:00
Tdon’t think they ever could - certainly not through Horizon.
One for POL to address.
What POL SAP did was allow TCs to be raised (through Horizon) rather than posting paper Error Notices to the Branch, which could then be ignored
,Number. 3_Author: Gareth Jenkins Date: 25/10/2018 18:38:00
This doesn’t sound correct, but I'm not 100% sure. Can POL comment on this:
I know we made significant changes to the use of Suspense and handling discrepancies as part of Impact in 2005.
Number: 4 Author: Gareth Jenkins Date: 25/10/2018 18:37:00
Error notices and TCs are rather different, (but do achieve the same thing ~ namely allow POL to request that corrections are made to Branch Accounts,
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 112 of 225
Measures ~ January 2012, 10 February 2012 [POL-0219354] [8807960c4e4ce6c767 1d 1f8254ed18b7] 1°”
1.5 5. Operations Manual version 7 December 2006 - pages 9-13.pdf, Processing any outstanding
Transaction Corrections, 7 December 2006 [POL-0184501] [9f8351ect4b5bdid4fd39452bef8026f]
18° Email from Angela Van-Den-Bogerd (PO) to Ron Warmington (Second Sight) and Ian Henderson (Second
Sight) Subject: FW: Factfile [BD-4A.FID20472253] Date: 16 April 2014 [POL-0022995]
[ca88fee778bee2b7 8ab05bc62cf6fe2c]
6.50 A Transaction Correction (TC) is defined as the accounting process through
SAP/POLFS for paBa to adjust a POL Branch account for discrepancies
found.
6.51 According to POL-0032855'%, In summary, the whole process was designed
to work as follows:
a. The central accounting function decides that it is necessary to
make some adjustment to the Branch accounts. Such adjustments
are to be made at branch when branch transaction data does not
align with client or supplier data.
b. A Transaction Correction (TC) is defined which will carry out the
necessary changes (i.e. the central user will define an amount to
be transacted for a given Product in a given Branch and a
corresponding settlement Product).
c. The Transaction Correction will also define a list of possible actions
that the Branch Manager can take and also a message is presented
to the Branch Manager informing him/her of the effect of carrying
out any of these actions.
d. A daily file of such Transaction Corrections is generated from
POLFS and passed to TPS overnight.
e. TPS receives this file, validates the data and performs the required
translations using Reference Data (converting a SAP article ID into
a Horizon Product).
193 DELLDO014_2.doc, TPS Transaction Corrections, 04 April 2005 [POL-0032855]
[926e9f76cb06ba79277f62138309290e]
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 128
Number: 1_ Author: Gareth Jenkins Date: 25/10/2018 18:39:00
Strictly, for PBA to request the SMPR to adjust the accounts. This was NOT an automatic adjustment. It had to be processed by the SPMR
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 113 of 225
f. TPS se messages for the Transaction Corrections to the
specified branches. A single message is written for the appropriate
Branch for each Transaction Correction.
g. Changes at the counter enable a person with the required role to
be made aware of the existence of an outstanding TC and apply
the correction at the counter.‘
h. The result of processing a TC will normally be the creation of the
specified Transactions, which will be returned to POLFS as part of
the normal flow of Summarised Transaction data at the end of the
Trading Day on which the TC was processed at Branch.
i. Subpostmasters are advised to log on to Horizon every working
day to check for and process TCs as only those with Manager or
Supervisor access shddid process them.
j. I TCs can be dealt with at log on or left until a more convenient time
later, but they must be processed before the last stock unit in a
branch is balanced, otherwise the Brdilch Trading Period cannot
occur.
6.52 It is recorded that it is not possible to reverse a correction transaction.
However, an erroneous correction transaction could be negated by
POLFS producing a fresh correction request with opposite sense.
Transaction Correction Options for the Subpostmaster+*5
6.53 TCs are issued electronically to the branch. At log on, the Subpostmaster
is presented with a notification of any new TC. If the TC is suspended
to be dealt with at a later point, Subpostmasters have menu button
options to access the outstanding corrections (there may be more than
one held on record to be accepted).
194 EAHLDO09_0.1.doc, TPS HR SAP Summarisation & Transaction Corrections HLD, 25 May 2009 [POL0076419]
[31d7f058bfa43c787c748e4acebfc8fc]
195 EPSPG001_0.2.doc, S80 Impact Release 3 EPOSS Counter Operational Support Guide, 10 May 2005
[POL0081677] [4481091 5b02dc5e8c0d15604570c1f65]
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 129
Number: 1_ Author: Gareth Jenkins Date: 25/10/2018 18:41:00
“This is the way it was done with Riposte as implemented in 2005. When BRDB was introduced, TCs were added to a TC table which was checked when a Supervisor or Manager
Logged on and also when explicitly requested,
Number: 2_Author: Gareth Jenkins Date: 25/10/2018 18:43:00
“Users with role “CLERK” did not have permission to process them, so they weren't presented to them.
=,Number: 3_ Author: Gareth Jenkins Date: 25/10/2018 18:44:00
This should say “The Branch cannot rollover to a new TP
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 114 of 225
6.54 When a TC is presented on screen it is accompanied by the details of the
transaction to which it relates.
6.55 There are a range of options presented for processing the TC. The options
available are presented in the table below: '%
Option Detail
W/O to P&L (F4f{Directly Directly Managed branches can use the
Managed branches only) “Write Off to P&L” option if the amount of
the Transaction Correction is greater than
the value they are expected to settle
themselves, unless the correction states
that the ‘Make Good’ option should be
selected.
Ass Nominee (F4) (National National Multiple branches must use the
Multiples only) ‘Assign to Nominee’ option whenever they
process a Transaction Correction, the only
exception being if they use the ‘Seek
Evidence’ option.
196 1.5 5. Operations Manual version 7 December 2006 - pages 9-13.pdf, Processing any outstanding
Transaction Corrections, 7 December 2006 [POL-0184501] [9f8351ecf4bSbd1d4fd39452bef8026f]
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 130
Number: 1_ Author: Gareth Jenkins Date: 25/10/2018 18:46:00
Tm not sure if these Function Buttons apply on HNG-X The menu of options is built dynamically, so Tm not sure that there was ever @ static mapping to Function buttons.
However not really important.
FUJ00183797
FUJ00183797
180503R1935
16 October 2018 Page 115 of 225
Seek Evidence (F3) (All
branches)
You can select the ‘Seek Evidence’ option if
the original information supplied by Product
and Branch Accounting (P&BA), Chesterfield
is insufficient for you to accept the validity
of the Transaction Correction. In this case
PB8&A will provide additional evidence and
issue a new TC that will not include the
‘Seek Evidence’ option.
Stock WO (F4) (All branches)
You should select the WO option for the
following:
e Rem surpluses in stock from Hemel
Hampstead
e Rem shortages in stock from Hemel
Hampstead
e@ non-accounting data corrections (so
that if incorrect volumes of
transactions have been claimed they
can be readjusted)
Cancel (F1) (All branches)
You should select the Cancel option if you
have selected a TC but do not wish to
process it immediately.
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
180503R1935
16 October 2018 Page 116 of 225
Accept Now (F2) (All branches}
except National Multiples)
When this option is selected it leads to a
picklist of further settlement options
available to your branch. These vary
according to the branch type and are
displayed in the table below:
Option Type of Reason for Selection
branch(es)
applicable
Make Good IAll branches}
- Cash except National I Use this option if you are using cash to
Multiples
account for a Transaction Correction, or if
the instructions accompanying the TC advise
you to accept this option.
Please remember: You may need to
physically add or remove the cash from your
stock or redeem from Rem Suspense to
reflect this change, otherwise the
discrepancy will remain in your accounts.
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 117 of 225
Make Good I All branches Use this option if you are using a cheque to
- Cheque except Directly account for a Transaction Correction
M d
anaged (even Please remember: The cheque must bel
though the option} dispatched in your daily dispatch.
may be shown on
the system) and
National
Multiples
Settle All branches] Selecting this option allows you to make
Centrally except Directly good a misbalance through the debt ;
recovery process that is managed within
(only Managed and P&BA. If selected, P&BA will contact you to
available if a I National confirm the next steps.
Transaction I Multiples
Correction is
for £150 or
over)
Successful processing of a Transaction Correction
6.56 When a Transaction Correction is successfully processed on the Horizon
Online system, a message will be displayed to confirm that the
Transaction Correction has been successfully processed.
Unsuccessful processing of a Transaction Correction
6.57 It is recorded that the Horizon Online system process includes a chdlk to see
whether Transaction Corrections fail due to discrepancies between the
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 133
Number: 1_ Author: Gareth Jenkins Date: 25/10/2018 18:50:00
This is because not all Products are available in all Branches,
Horizon doesn't check that the products associated with the TC are available in the branch when the TC is first sent (it assumes that P&BA have checked this in POL SAP).
Therefore if the Product is not available, then the TC will fail and that will generate an alert to be processed by the support teams.
The TC will be removed from the list so that no further attempt is made to process it.
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 118 of 225
validation of the transaction at the counter and the values held within the
TC message.
6.58 Where this might be the case, the system displays a failed warning
message and Subpostmasters are advised to contact NBSC. They will
advise Product and Branch Accounting (P&BA) Chesterfield who will
investigate the failure and issue a new TC.
6.59 In the event of any Connection failures when processing a TC, the process
must be abandoned until the system is restored.
Branch Trading Statement
6.60 The total volume of TCs processed during a Branch Trading Period can be
seen on the Branch Trading Statement.‘
Disputing a Transaction Correction
6.61 Post Office documentation states that Subpostmasters have the ability to
disBlte any TC given. They are advised to contact the person who sent
the TC as soon as possible.'°® Subpostmasters may be asked to give
more information to support their dispute. If a TC is issued too close to
a Branch Trading period end to be fully investigated, Subpostmasters
are encouraged to call NBSC to ask for more time to gather and present
supporting information. Where a dispute is accepted, a compensating
TC is issued.
6.62 Where a TC or branch discrepancy is disputed, and the case is not allowed
(this is not defined in in the document) Post Office state that
Subpostmasters may make a written submission explaining why the loss
(or gain) is not proper. This ensures that the debt recovery process
suspended pending a written response.
187 1.5 5. Operations Manual version 7 December 2006 - pages 9-13.pdf, Processing any outstanding
Transaction Corrections, 7 December 2006 [9f8351ect4bSbd1d4fd39452bef8026f]
188 1.6 16. Disputing a Transaction Correction & the Appeals process.pdf, Branch Trading: balancing and
despatch of documents - Balancing a stock unit, 12 October 2016
[105f4315b879dd781 0def967bde6bb15] same text appears in POL-0001515]
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 134
Number: 1_ Author: Gareth Jer Date: 25/10/2018 18:53:00
This is what Seek Evidence is for. The assumption is that when a subsequent TC is issued without the Seek Evidence option that any disputes have been resolved.
NB this is all POL Process and they should be responding to this.
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 119 of 225
6.63 A Subpostmaster is limited from disputing a TC more than once*%?. In the
event of not being satisfied by the initial dispute decision,
Subpostmasters were unable to electronically select further dispute
actions. Anflfurther action to dispute a TC would therefore be outside
of any selectable counter options.
Transaction Correction Observations and Findings
6.64 There is evidence of poll themselves creating incorrect Transaction
Corrections and sending these to Subpostmasters. One example
identifies that a Transaction Correction was issued for 800 sheets of 100
stamps, rather than 8 sheets of 100 stamps. Some of these POL
mistakes were not spotted and therefore accepted by Subpostmasters.
The result of this POL error is the insertion of a discrepancy in the
Subpostmaster’s branch accounts. 200701,207
6.65 As identified from review of the witness statement of Angela Burke?,
Transaction Corrections were also documented against the incorrect
financial institution. M urke was awaiting a Transaction Correction to
correct a TSB withdrawal in branch that was lost in the Horizon system
and when this was sent through it was documented as a Lloyds
withdrawal correction due to there apparently not being "a code for
TSB”. Whilst the receipt of the TCV would ultimately rebalance the
discrepancy for Mrs Burke it illustrates one example of error within the
issuing of TCs.
189 Transition Guide for Group A.PDF, Branch Trading Transition Guide, 26 September 2005 [POL-0171227]
[bSd2cc8a4ffdfbd976d1651c909d8ef1 ]
200 PCO131060.html, PEAK PC0131060, 17 January 2006 [POL-0301483]
[fad49b305707d3deac31b9249ee5c761]
201 pdf, Display Notes: 125460003153512008001, 25 August 2010 [POL-0006283]
[189c0e81e2064924aecfb5fcf281bf2e]
202 Witness statement of Angela Burke 28.09.2018.pdf
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 135
Number: 1 Author: Gareth Jenkins Date: 25/10/2018 18:54:00
This was @ POL requirement and is entirely controlled by P&BA when they set up the TC.
Horizon has no control over this aspect.
Number. 2 Author: Gareth Jenkins Date: 25/10/2018 18:55:00
POL issue and not Horizon
Number: 3 Author: Gareth Jenkins Date: 25/10/2018 18:56:00
My understanding is that this difference is irrelevant to the Branch accounts (it may be an issue in P&BA, but the Branch accounts would be correct).
Again for POL to address
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 120 of 225
6.66 An analysis of Transaction Corrections”? displays Transaction Corrections
between 26 March 2012 and 28 March 2013. The largest (in monetary
terms) appears to be £810,000. Of interest, there is also a £-
810,000 Transaction Correction for the same “Potters Bar” branch,
indicating that the correction might have initially been in error.
6.67 Of the 84,217 TC’s over this period 2267 resulted from “Camelot” and
7349 from “Cash Rems from Branch”. Only a relatively small amount
(945) of “ATM” TCs are referenced. Full table available at Appendix C
(2012-2013 TC's)
6.68 TC queries were often raised with the PO helpdesk. ‘SLA Summary NEW
WE 15062014.xls (POL-0031913)’2°* shows that during one week 9421
calls were made to the helpdesk and 145 of these related to queries and
disputes about Transaction Corrections (See Appendix D).
6.69 Thél witness statement of Adrees Latif 205 highlights an alleged failed
Transaction Correction in respect of incorrectly issued Camelot scratch
cards in Horizon. The correction should have reversed the stock amount
but for reasons unknown to the Subpostmaster this failed. The issue
remained outstanding at the time the statement was given, and the
branch showed non-existent stock of Camelot scratch cards with a value
of £1000.00 that was unable to be adjusted without incurring a shortfall.
6.70 Figures presented at the Post Office Operations board of 22nd March 20187°
displayed that 3,546 branches had more than 1 TC per month.
6.71 The document ‘SLA Summary New WE 06072014’2’ records an activity
type of “Discrepancy” in relation to either Stock or Remittance
203 POL-0031763.xIsx, Analysis of Transaction Corrections, ca. 2013 [POL-0031763]
[a9f7d9c47b7748522a182dbe2a114768]
204 SLA Summary NEW WE 15062014.xIs, NBSC Incident SLA Summary - Week ending 15th June 2014, 17
June 2014 [POL-0031913] [7fa41e4d5bcc4996e0bfeObbbadf82f1]
205 Witness Statement of Adrees Latif, 28 September 2018, (Para: 9-14)
206 Operations Board - 22 March 2018.pdf, Operations Board — 22 March 2018, 23 March 2018 [POL-0220482]
[638109a286e2e884dc735539687f7c35]
207 SLA Summary NEW WE 06072014.xIs, NBSC Incident SLA Summary - Week ending 6th July 2014, 14 July 2014
[POL-0031909] [2a6e1fbd3ef5d76899a9f21957d65011}
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
Page: 136
FUJ00183797
FUJ00183797
Number: 1_ Author: Gareth Jer Date: 25/10/2018 18:57:00
Thatis why POL wanted to move Camelot to use TAs to reduce this volume of TCs,
POL to comment
Number. 2_Author: Gareth Jenkins Date: 25/10/2018 18:58:00
‘Sounds like SMPR miscounting what they put in pouches
Again it is for POL to address.
Number. 3_Author: Gareth Jenkins Date: 25/10/2018 18:59:00
Tcouldn’t find any TCs relating to Camelot scratch card in the data I examined from that Branch.
I suspect the TC may have come later.
Can POL provide more info to help us check this out?
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 121 of 225
differences against the advice note received from POL. I ubberstand this
to mean that POL have not dispatched the correct amount of Stock or
Cash stated as dispatched on the advice note. The may indicate a
potentially significant flaw in the Horizon system. If the Subpostmaster
does not spot POL’s error and therefore it goes unreported, this will
likely mean that the next time the Subpostmaster conducts a stock take
it will show an accounting discrepancy which might lead to a TC.
6.72 The diagram in Annex B sets out a breakdown of potential reasons for the
issue of a Transaction Correction. Note that this list is nonexhaustive
and is based on the sheet headed “Possible Reasons for TC’s Issued”
within the document “NEW TC PACK P10 2011.xIs” 2°. The
categorisation as to whether these were caused by Subpostmaster or
not is mine.
Opinion Summary
6.73 Reconciliation is a complex combination of electronical and manual
processes. poHapproximate that over 10,000 transactions per week are
subject to manual corrections.
6.74 Alongside the electronic processing elements of reconciliation are various
phases of analysis and decision making in respect of why/where flagged
anomalies occurred. This scrutiny is crucial to determining both the
reason for the anomaly in the first instance, and to remedy the effects
of it.
6.75 Transaction Corrections at branch counter typically stem from incidents
where either a transaction has been performed in error by the
Subpostmaster and needs correcting or, a transaction anomaly has
occurred at another point and needs adjusting not caused by the
Subpostmaster.
208 NEW TC PACK P10 2011.xIs, TC Glossary, 15 February 2012 [POL-0221536]
[e1a3c3394d348ab3355302b35cfd63ab]
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 137
Number: 1_ Author: Gareth Jer Date: 25/10/2018 19:01:00
Twould have thought that this relates to stock cash sent from Branch to POL
However POL need to address this not us.
«Number: 2_ Author: Gareth Jenkins Date: 25/10/2018 19:02:00
Twould doubt this.
Number: 3 Author: Gareth Jenkins Date: 25/10/2018 19:03:00
Can we say how many are ones we raise? I suspect that most of these are from POL's backend system checks and not due to errors in Horizon data or reconciliation reports.
Number. 4 Author: Simpkins John Date: 06/11/2018 13:07:00 Z
This is the 3rd time this figure has been quote.
From a Sec Ops spreadsheet they have between 6 and 66 for a week, nothing like the number above. The 66 was a spike due to a Link issue.
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 122 of 225
6.76 It
has not been possible to review the analysis, or reasoning applied to
Transaction Corrections issued by Post Office, i.e. whether these were
appropriately identified as issues in financial positions stemming from
the effects of a bug/error/defect or third-party error or from a counter
mistake.
6.77 Therefore, although reconciliation might identify transaction anomalies,
6.78 It
and appropriately capture electronic distortion of transaction data, it
does not ensure an infallible process in respect of appropriately handling
potential shortfalls/discrepancies for Subpostmasters, since there are a
range of other factors to consider in the processing and analysis of the
data. It appears that the Post Office applied a cost/benefit approach to
the remedying of reconciliation exceptions and _ certain
bugs/errors/defects.
is not clear from analysis of the various Known Error Logs whether
bugs/errors/defects identified as affecting branch accounts then
ultimately progressed to the issuing of a Transaction Correction for that
branch or were confirmed by separate communications from the Post
Office.
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 123 of 225
7. Horizon Reporting - Facilities for Subpostmasters
Issue 2 - Did the Horizon IT system itself alert Subpostmasters of such bugs,
errors or defects as described in (1) above and if so how.
Issue 14 - How (if at all) does the Horizon system and its functionality:
a. enable Subpostmasters to compare the stock and cash in a branch
against the stock and cash indicated on Horizon?
b. enable or require Subpostmasters to decide how to deal with,
dispute, accept or make good an alleged discrepancy by (i) providing
his or her own personal funds or (ii) settling centrally?
c. record and reflect the consequence of raising a dispute on an alleged
discrepancy, on Horizon Branch account data and, in particular:
d. does raising a dispute with the Helpline cause a block to be placed
on the value of an alleged shortfall; and
e. is that recorded on the Horizon system as a debt due to Post Office?
f. I enable Subpostmasters to produce (i) Cash Account before 2005 and
(ii) Branch Trading Statement after 2005?
g. enable or require Subpostmasters to continue to trade if they did not
complete a Branch Trading Statement; and, if so, on what basis and
with what consequences on the Horizon system?
Horizon Alerts for Subpostmasters in respect of bugs/errors/defects
7.1 Post Office have advised? that in respect of the known Local Suspense
Account Issue that Subpostmasters were notified however, how they
were notified and if all affected Subpostmasters were notified is still
subject to clarification from Post Office.
209 post Office's response to Jason Coyne’s Requests for Information.pdf 08 August 2018
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 124 of 225
7.27°Ina repurt from Gareth Jenkins 29 September 20107'° (in connection with
the known bug ‘Receipts and Payments mismatch’) it is reported:
“This has the following consequences: There will be a receipts and
payment mismatch corresponding to the value of discrepancies that were
“lost”. Note that if the user doesn't check their final balance report
carefully they may be unaware of the issue since there is no explicit
message when a receipts and payment mismatch is found on the final
balance (the user is only prompted when one is just detected during a
trial balance)”.
7.3 Further, Post Office have advised?'! that they have “made enquiries” as to
confirm how Subpostmasters were notified of this particular issue but
have “not been able to find relevant records so far.”
7.4 It is understood that in the event of certain counter processing errors,
Subpostmasters would be presented with select on screen messages
informing them of some form of error. This would not in itself notify
Subpostmasters of the full extent of the potential implications of an
error occurrence (i.e., the error ultimately resulting in a shortfall or
discrepancy).
7.5 Some notifications would relate to procedures of which the Subpostmaster
would be aware (a failed recovery for example), however these would
only be at the counter. A Abpostmaster would not have visibility of any
error occurring for any transaction processing past the point that a
transaction was committed to the branch database (in respect of
transaction data errors), nor any potentially occurring further within the
processing systems.
7.6 KEL acha1941L,?!? identifies that during the recovery process, when some
transactions recover but others fail to recover, it is only the recovered
210 SM BP Correcting Accounts for Lost Discrepancies - 102000790 - CD1.pdf, Correcting Accounts for
“lost” Discrepancies, 29 September 2010 [POL-0010769] [804ea47c166870b7ed0359e4765e0265]
211 Post Office’s response to Jason Coyne’s Requests for Information.pdf 08 August 2018
212 acha194iL.html, HNG-X KEL acha1941L, 11 July 2012 [POL-0039098]
[f392eca2f01 5eab903154e7a3e7a31ee]
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 140
Number: 1_ Author: Gareth Je
Agreed
Date: 26/10/2018 11:30:00
However did POL contact the affected branches? I think they were intending to, but don't know if they did.
I think I saw draft letters, but I can’t be sure,
Number. 2_Author: Gareth Jenkins Date: 26/10/2018 11:32:00
Any such errors wouldn't impact on the Branch accounts, so there is no need for the SPMR to know.
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 125 of 225
transactions printed on the receipt. Th@l disconnected session receipt
should also identifyt those transactions NOT recovered. These are printed
for the Subpostmaster to retain. This would alert the Subpostmaster to an
instance of an error in counter processing in respect of a failed recovery
(known potential Horizon incident). Alongside the printed receipts, it is
further noted that upon an instance of recovery failure, the system might
display:
"MSG90025: System Error - Error Code: 0058 has occurred Reason:
System error. Please retry once. If the problem persists, contact the
Horizon System Desk. ’?19
7.7 KEL sur1147Q24 records another failed recovery issue to which the solution
is advised as follows:
“Advise the PM to log onto the relevant counter, start the recovery
process (MSG04024) but leave the counter displaying the System error
message (MSG90025). It is important that they DO NOT confirm this
message and that they just leave the counter to time out due to inactivity
at this screen - which will take at least 1 hour 20 minutes. After this time,
they can try to log on again - when there should be no prompts about
recovery and no system error.”
7.8 Therefore, as witnessed above, it is apparent that in specific event
circumstances, Subpostmasters were alerted to certain errors occurring
at the Horizon counter. However, it must be noted that these relate to
counter errors and therefore are not necessarily notifications of bud
and defects within the system.
233 KEL cardc1655P.html, HNG-X KEL cardc1655P, 26 July 2016 (last updated 2 August 2016) - [POL-0040501]
[09bc61dfc66a102d149337df7ce42a93]
24 surs1147Q.html, HNG-X KEL surs1147Q, 08 April 2010 (last updated 17 February 2016) [POL-0040368]
[4f8fdc56faalf96e3e84522F4c49da75]
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 141
Number: 1_ Author: Gareth Jer
Date: 26/10/2018 11:34:00
Thave checked this out with development and the Disconnected session receipt DOES include the non-recoverable transaction, but they are included as zero value, so as to be
able to tell what will be recovered and what won't be.
It was POL who specified what the receipt should contain.
jgNumber: 2 Author: Gareth Jenkins Date: 26/10/2018 11:38:00
“Not sure what this is about
Need to check it up.
Number. 3_Author: Simpkins John_Date: 06/11/2018 13:31:00 Z
This was due to @ bad APADC recovery rollback script for some postal service items which caused a logon loop. Peak PCO245066 applies.
‘A work around was shared when the user is logged off due to inactivity it would clear the problem,
Number: 4 Author: Gareth Jenkins Date: 26/10/2018 11:38:00
“We wouldn't do that through Horizon and should probably say so explicitly
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 126 of 225
7.9 Similarly, in wrightm33145j2'> as part of the process for rolling over to a new
Balance Period or new Trading period; the SuBbostmaster was presented
with a series of prompts and warnings if any discrepancies are found.
7.10 If the Subpostmaster chose to cancel the rollover prompt message
(MSG31316) this could trigger a receipts/payments mismatch due to a
bug in the code when “cancel” is pressed against this message. The
workaround to avoid the bug was to press cancel a second time.
Hoflever, it is unclear how widely this workaround was communicated.
Subpostmasters would not be aware from the error message shown that
this was indeed a bug affecting a wide variety of branches.
7.111t appears that the process was largely for Subpostmasters to notice an
error (either when trying to balance or after a communication outage)
and phoning the helpline. tha would occur before it was communicated
or confirmed that there had been or was a bug/defect which might have
affected the branch accounts.
7.12 Some KELs record making priority calls to branches however, it iShot clear
whether this occurred, and it is not noted that priority calls were made
in the event of every notified error.
Bugs Errors Defects not alerted to the Subpostmaster
7.13 As above, in respect of the known receipts and payments mismatch bug,
it does not appear that Subpostmasters were alerted, until perhaps an
investigation of discrepancy was performed.
7.14 Further, in respect of the acknowledged Calendar Square / Falkirk Issue
bug/defect it is not identified that Subpostmasters were informed.
7.15 As per the Joint Experts Statement, the extent to which any IT system can
automatically alert its users to bugs within the system itself is
necessarily limited.
215 wrightm33145).html, HNG-X KEL wrightm33145J, 23 September 2010 (last updated 01 April 2016)
[POL0040409] [1f025ec713c287ee7a5b17accd25b42f]
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Number: 1 Author: Gareth Jenkins Date: 26/10/2018 11:39:00
Yes. This is to allow him to investigate (and if possible correct) any errors in his processing before they are committed as part of the rollover process.
Number. 2_ Author: Gareth Jenkins Date: 26/10/2018 11:41:00
it probably wasn't communicated ~ we fixed the bug instead and continued to monitor for re-occurrences until after the fix was rolled out. (Actually as the monitoring was oF
R&P mismatches, then we always monitor for those anyway!)
~Number: 3 Author: Gareth Jenkins Date: 26/10/2018 11:43:00
iF there was a known problem, we would monitor for its re-occurrence,
Number: 4 Author: Gareth Jenkins Date: 26/10/2018 11:43:00
Can we describe our processes around this?
Number: 5 Author: Simpkins John Date: 06/11/2018 13:40:00 Z
Texpect that the helpdesk would perform this, may be best to ask Sandie,
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 127 of 225
7.16 Whilst Horizon has automated checks which would detect certain bugs,
there are types of bugs which would not be detected by such checks.
7.17 In respect of the Dalmellington bug referenced at 5.16, there is no
indication that the Horizon system alerted the Subostmistress of this
bug. Ang rectification was therefore dependent upon correct initial
diagnosis of the bug, proper handling processes by the Post Office and
cor®munication provided to Helen Baker and the Subpostmistress.
7.18 It will be a matter for the court to determine in due course whether those
prddesses worked appropriately in relation to this bug.
How does Horizon enable Subpostmasters to compare the stock and cash
in a branch against the stock and cash indicated in Horizon?
7.19 There are several reports a Subpostmaster is required to run at the end of
each working day. These are designed to be checked against the
appropriate documents to amend any errors accordingly. These reports
are then ‘cut-off’ when the Subpostmaster is content they are correct.
The cut-off routine is mandatory.
7.20 of Hote there is a procedural requirement for staff at branches to count
and declare the cash stored in each stock unit at the end of each day in
what is called a Daily Cash Declaration.
Daily Cash Declaration
7.21 After performing the Daily Cash Declaration, Horizon willl show any
discrepancy between the cash on hand and the amount of cash that
should be in the branch for the branch to balance.
7.22 A ‘Factfile’ document sent from Angela Van-Den-Bogerd of Post Office to
Second Sight?!* states that Horizon can assist the Subpostmaster with
216 Factfile.DOCX, Initial Complaint Review and Mediation Scheme DRAFT Factfile, 16 April 2014, (P17 - 94.1) [POL-
0022996] [d24556f1009f1881f42a2dbb5b8154de]
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Date: 26/10/2018 11:48:00
This was done and after investigation all other occurrences of the issue detected and investigated
»Number: 2_Author: Gareth Jenkins Date: 26/10/2018 11:48:00
Tm not sure that this was done correctly (or at least initially). At the time there was an inaccurate description of the problem circulated that appeared in Computer weekly.
Not sure how that happened. I was called in fairly late into the investigation.
Number: 3 Author: Gareth Jenkins Date: 26/10/2018 11:50:00
think they did eventually, but accept that it was 2015 before the bug was identified and it had been present since 2010.
Number. 4 Author: Gareth Jenkins Date: 26/10/2018 11:51:00
However, I suspect that this is often not done accurately — particularly in poorly managed branches,
Number: 5_Author: Simpkins John_Date: 06/11/2018 13:41:00 Z
There is an ONCH report in HORIce that the Post Office check and contact Post Masters who have not made the dedlarations.
Number. 6 Author: Gareth Jenkins Date: 26/10/2018 11:52:00
Not necessarily. This is true for an unshared SU, but not for a shared SU where the cash may be split between many physical tills,
However there is an Optional capability to carry out a Variance check which will show any variance between the sum of Cash Declarations for the SU and the system generated
cash figure.
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 128 of 225
tracing or identifying discrepancies including event logs that are
available for 60 days (42Mays pre-Horizon Online).
7.231In addition to the daily cash declaration there is also the optional Weekly
Balance and mandatory Monthly Trading Period Rollover.
Weekly Balance
7.24 Although not mandatory the wedkly Balances are reported to be a
recommended action of POL policy to execute on a weekly basis; this
performs a full cash and stock account.
7.25 This should also help to detect any discrepancies. If a discrepancy is
discovered and declared it Vill be moved into a suspense account until
its resolution. The purpose of this is to act as a separate line in the
branch accounts that records any surpluses or loses whilst allowing the
daily trading accounts to balance. A Subpostmaster has until the
Monthly Trading Period Rollover to resolve any discrepancy (this was
carried out weekly prior to 2005).
Monthly Trading Period Rollover
7.26 The Monthly Trading Period Rollover is mandatory every month and as
stated above requires all discrepancies, including all within the suspense
account, to be resolved. At the end of this process a Branch Trading
Statement is produced to reflect the cash and stock shown in the
accounts matches the cash and stock held in the branch in Budition to
any declared discrepancy.
How does Horizon enable or require Subpostmasters to decide how to
deal with, dispute, accept or make good an alleged discrepancy by (i)
providing his or her own personal funds or (ii) settling centrally?
7.27 Post 2005 is dealt with under Section 6.48 above.
7.28 Prior to 2005 Horizon followed a similar process however branches were
required to balance weekly and produced a Horizon “Cash Account”.
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Date: 26/10/2018 11:54:00
iEvaried in old Horizon. Initially it was 34, then it increased to 42 with Impact and then was later increased to 84 around 2008 (not sure exactly when)
»Number: 2_ Author: Gareth Jenkins Date: 26/10/2018 11:55:00
These are done per SU. Branches appear to sometimes do weekly balances on thelr main SU, but only monthly balances on some of thelr Back Office SUs such as OOH
Number. 3_ Author: Gareth Jenkins Date: 26/10/2018 11:56:00
“"Teisn’t moved to a suspense account (or at least it hasn't been since IMPACT in 2005), It is moved to a Discrepancy Account (I suppose this could be considered as a sort of
Suspense account), which must be cleared as part of Branch Rollover.
Number 4_Author: Gareth Jenkins Date: 26/10/2018 11:59:00
What does this mean? All discrepancies are cleared and must either be made good or settled centrally. They are then no longer included in any Horizon accounts moving
forward,
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 129 of 225
Discrepancies were, with the authorisation from the Post Office, held in
a “Suspense Section” of the Cash Account (although I have also seen
this referred to as a Suspense Account). Histbrically these were known
as “Unclaimed Payments”, “Authorised Cash Shortages” and
“Uncharged Receipts”, to be investigated.
7.29 Following the conclusion of any investigation and in order to remove the
discrepancy, Subpostmasters were issued with an Error Notice (now
known as a Transaction Correction) by the post office or the
Subpostmaster placed money into the branch to cover the loss or
removed the value of the gain from the branch to balance the account.
7.30 The value of error notices had to be keyed in manually by the
Subpostmaster on the Housekeeping menu of the Horizon system
following notification from either a client or POCL. However, the losses
or surpluses at the end of a balancing period could still be disputed.
7.31 According to the Complaint Review and Mediation Scheme documentation,
I note that the ability to havea dell suspended pending an investigation
has only been available since August 2005.7!”
record and reflect the consequence of raising a dispute on an alleged
discrepancy, on Horizon Branch account data and, in particular:
7.32 This is covered under Heading 6 - Reconciliation and Transaction
Corrections (Section 6.61, Page 113)
does raising a dispute with the Helpline cause a block to be placed on
the value of an alleged shortfall; and
7.33 This is covered under Heading 6 - Reconciliation and Transaction Corrections
(Section 6.61, Page 113).
217 0.19 10. Post Office - Complaint Review and Mediation Scheme.pdf, ca. 2014 [POL-0025512_1]
[6d54ae49b7132a421d48bbd941064d39]
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Date: 26/10/2018 12:01:00
Tm Tess familiar with accounting at this time as I really got involved in the counter accounting as part of IMPACT. However I thought that discrepancies did need to be made
good, but there were these Suspense account items that allowed the SPMR to move potential discrepancies into Suspense.
‘As part of IMPACT, POL were keen to reduce the amount of cash held in Suspense “indefinitely” and so removed many of these facilities.
jNumber: 2_Author: Gareth Jenkins Date: 26/10/2018 12:04:00
Presumably by settling centrally - which was introduced at that time
‘Such processes are in POL's domain and handled outside Horizon,
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 130 of 225
7.34 See also the Wibless Statement of Dawn Phillips*#® Page 2 paragraph 10.
which confirms that a block is placed upon the account until the dispute
is resolved. However, if branches do not return the completed “Branch
Dispute Form” detailing the discrepancy within seven days, the shortfall
is unblocked, and payment is once again requested from the
Subpostmaster.
7,35 It should be noted however that the Branch Dispute Form has only recently
been introduced in 2018 and the process of collecting key information
from the branch has been in place since November 2016 by other means
such as email and telephone communications?*’.
7.36 It is not fully clear what totality of mechanisms were available for
Subpostmasters to dispute discrepancies prior to 2016 although
Appendix D illustrates there is clearly an option available to dispute a
Transaction Correction.
is that re@rded on the Horizon system as a debt due to Post Office?
7.37 A loss is recorded as a debt to the Post Office in the event the discrepancy is
upheld by the Post Office following any dispute.
enable Subpostmasters to produce (i) Cash Account before 2005 and (ii)
Branch Trading Statement after 2005?
7.38 As above the Cash Account were produced weekly after a mandatory
weekly balance just as a Branch Trading Statement is produced after
the Monthly Trading Period Rollover.
enable or require Subpostmasters to continue to trade if they did not
complete a Branch Trading Statement; and, if so, on what basis and
with what consequences on the Horizon system?
7.39 Subpostmasters are not able to continue trading until Branch Trading
Statement process is complete. If the Branch Trading Statement is not
Signed witness statement of Dawn Phillips.pdf (dated 28 September 2018).
29 Signed witness statement of Dawn Phillips.pdf (dated 28 September 2018).
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
Page: 146
FUJ00183797
FUJ00183797
Number: 1_ Author: Gareth Jer Date: 26/10/2018 12:05:00
Tye not seen this. Whois she? Presumably someone from P&BA
geNumber: 2_Author: Gareth Jenkins Date: 26/10/2018 12:07:00
No. Once something has been settled centrally, itis out of Horizon and down to POL Back end processes with the Branch.
I believe a TC could be used to introduce a further discrepancy to be settled centrally to cancel out the initial one.
However this is all down to POL>
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 131 of 225
completed and therefore, the Mobithly Trading Period Rollover is not
completed the Post Office will contact the branch in order to rectify the
situation.
Opinion Summary
7.40 In many, but not all instances the Subpostmaster is alerted to a variety of
system errors or warnings whilst processing transactions at the counter.
However, these alerts and warnings do not necessarily alert
the Subpostmaster to the extent of the issue in the back-end processing
systems or necessarily allow him/her to make any adjustments.
7.411 have not seen any evidence as to how known issues and bugs were
communicated to branches in Ekdvance of individual branches
discovering the issue themselves. In the example highlighted above in
(KEL wrightm33145j - Para: 5.10) there is no evidence any advice was
communicated to branches.
7.42 Subpostmaster feedback??° appears to highlight amongst many other
things an absence of support and training to assist them with being able
to find and resolve mistakes. In fact, this was expressed as follows:
“Thdk is a perception that there is a reluctance to report a shortage or a
balancing problem to NBSC as branches feel that the next stage will be
an audit and then suspension”
220 Gaps and issues final 9.10.13.x/s, Subpostmaster Feedback Spreadsheet, 17 October 2013 [POL-0216108]
[421233d56af42c063d50356a96a18e89 ]
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 147
Number: 1_ Author: Gareth Jenkins Date: 26/10/2018 12:10:00
~ Note that with Old Horizon, if the CAP was not complete within 35 days, then there was a danger of the Branch losing data (just locally ~ the data was still secured at the data
centre ~ provided that the branch was still connected and harvesting) and so we would require the branch to be closed and re-opened. This was monitored carefully.
(Not sure ifit was just by POL or by us.)
Number. 2 Author: Gareth Jenkins Date: 26/10/2018 12:13:00
“ Twould argue that there was no need to do so. These bugs were very isolated in their impact and we monitored carefully to see exactly where they occurred
Number: 3. Author: Gareth Jenkins Date: 26/10/2018 12:14:00
Twould support this view, but this is one for POL to answer.
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 132 of 225
8. Horizon Shortfalls, Data and Reporting for
Subpostmasters and Post Office
Issue 8 - What transaction data and reporting functions were available through
Horizon to Post Office for identifying the occurrence of alleged shortfalls and the
causes of alleged shortfalls in branches, including whether they were caused by
bugs, errors and/or defects in the Horizon system?
Issue 9 - At all material times, what transaction data and reporting functions (if
any) were available through Horizon to Subpostmasters for:
a. identifying apparent or alleged discrepancies and shortfalls and/or
the causes of the same; and
b. accessing and identifying transactions recorded on Horizon?
Data and Reporting Functions for Post Office
8.1 The data and reporting functions available to Post Office can be primarily
summarised by the following sources:
a. The TPS Report Set;
b. The APS Report Set;
c. The DRS Report Set;
d. Reports and Data Obtained via Business the Incident Management
(“BIM”) Process;
e. Reports and Data Obtained via the Problem Management
Procedure;
f. Information Obtained via Fujitsu; and
g. Information Obtained from Subpostmasters.
8.2 Detailed descriptions of each of the above as set out in Appendix E to this
report. Summaries of each are set out below.
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 133 of 225
Official Sources of Information
8.3 The TPS Report Set - The TPS Report Set was made up of 3 reports
(7P$b50/254/257) which were designed to enable the reconciliation of
transactions using branch infrastructure. They are all daily reports which
show information relating to whether transaction outputs at branches
matched transaction outputs at Post Office, whether there are
exceptions in the BRDB copy process and whfh branches have a net
total transaction (i.e. debit/credit) transaction total which does not
equal 0.
8.4 The APS Report Set - The APS Report Set was made up of 10 reports that
were designed to reconcile those transactions that were sent to both
the pobbap system and APS clients. The APS Report Set provides
confirmation that the APS transaction account balances, a summary of
transactions which have been delivered by APS, a summary of
transactions which have not been delivered due to delay/quarantine, a
reconciliation summary between transactions which flow through the
APS host and TPS host and confirmation that all branches have been
harvested / details of any that exist in relation to harvesting.
8.5 The DRS Report Set - The DRS Report Set (sometimes referred to as
“Banking & Related Services Reconciliation”) is made up of 3 reports
which were designed to enable network banking transactions completed
in Post Office branches to allow settlement to be made with Post Office
clients (e.g. Santander, LINK, etc.). The DRS Report Set prdvided a
summary of all reports that were not produced by DRS due to a lack of
data, identified all C4 transactions received against each C4 settlement
date and information relating to all exceptions.
8.6 Reports and Data Obtained via the Business Incident Management (“BIM”)
Process - The BIM system was designed to report progress on the
resolution of Business Incidents to allow Post Office to complete
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
Page: 149
FUJ00183797
FUJ00183797
Number: 1_ Author: Gareth Jer Date: 26/10/2018 12:17:00
re these all still relevant? Are some related just to old Horizon?
Number: 2_ Author: Simpkins John_Date: 06/11/2018 16:26:00 Z
ATS reports were added to Peak PC0235476 as evidence on 10-Jul-2014,
‘TPSC257 is used in reports including last week (PCO275157)
Number: 3 Author: Gareth Jenkins Date: 26/10/2018 12:18:00
This sounds like old horizon only I the old Reconeiliation reports.
Number. 4 Author: Simpkins John Date: 06/11/2018 16:29:00 Z
They recently are used for POLSAP Incomplete Summaries for reference data issues,
Number: 5 Author: Gareth Jenkins Date: 26/10/2018 12:18:00
Strictly POL MIS, but they were also summarised for POL SAP.
Number. 6 Author: Gareth Jenkins Date: 26/10/2018 12:19:00
Pardon?
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 134 of 225
reconciliation or settlement with its internal systems, clients and
banks.?24 A “Business Incident” describes the effect of a system fault
and can relate to any of the exceptions from the various reports or a
settlement error discovered by Post Office. A “System Incident”
describes the underlying cause of a Business Incident and is created to
track the root cause of the same. A BIM Report is issued for each
Business Incident in order to notify Post Office of issues and to assist in
the reconciliation or settlement process. Note that BIM Reports
communicate information concerning the resolution of an issue, and not
the underlying cause (which would be dealt with via the Problem
Management Procedure).
8.7 Reports and Data Obtained via the Problem Management Procedure - The
Problem Management Procedure aims to eliminate the root cause of
issues (e.g. System Incidents).
Additional Sources of Information
8.8 Information Obtained via Fujitsu - Most of the sources above will include
significant involvement from Fujitsu and some, such as the TPS Report
Set, will only be made available to Post Office if requested. However, it
is important to note that the above only represents the formal sources
and incident management processes. Post Office also had access to
information simply by nature of the commercial and _ technical
arrangements between itself and Fujitsu. In other words, there was
nothing (other than cost) preventing Post Office from seeking further
information from Fujitsu directly if, for example, the Problem
Management Procedure did not yield a satisfactory conclusion. This is
unlikely, as the reporting and management functions above are quite
221 CSPRO111_4.1.doc, TPS Reconciliation & Incident Management (Section 4.4.1; BIMS Reports/MER), 10
June 2015” [POL-0082393] [5ad9b694ade347339b5f5a2b49c88ea2]
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 135 of 225
comprehensive and should cover the vast majority of issues, but it is
possible.
8.9 Information obtained from Subpostmasters - In addition to the above, the
Subpostmasters themselves and any information available to them were
sources of information available to Post Office. These are further
considered below.
Conclusions Relating to Information Available to Post Office
8.10 The TPS, APS and DRS Report Sets along with reports produced in
accordance with the BIM and Problem Management Procedures provided
a comprehensive suite of reports which was available to Post Office.
These reports should have allowed Post Office to identify the occurrence
of alleged shortfalls in the Horizon system (of those that could be
identified), and they were underpinned by formal processes which would
provide further information in relation to the underlying cause of a given
issue, and the best way to resolve the same. In addition, Post Office
should have been able to obtain any additional information it required
via Fujitsu or the Subpostmasters themselves.
Data and Reporting Functions for Subpostmasters
8.11 Subpostmasters had access to a much smaller pool of information. This is
in line with what I would expect to see given that Subpostmasters are
the users of the Horizon system, and therefore would not typically be
given access to anything beyond what was necessary for them to carry
out their ‘business as usual’ activities. In relation to this matter, that
means being able to carry out the day-to-day transactions required to
run a post office, which are dealt with via the “counter”.
8.12 All the reports and receipts produced by the counter are set out in
SD/DES/005”? - note this document excludes reports and receipts
222 SDDESO05_12.2.DOC, Horizon OPS Reports and Receipts - Pathway - Horizon Office Platform Service, 23
January 2003, (Version: 12.2), [POL-0069333] [3d8f3190f530f5e71916099859b3 1dbd]
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 136 of 225
which are not produced by the counter. It includes hundreds of reports
relating to different aspects of running a Post Office branch (currency
exchange, insurance, stock, etc.). The reports and receipts contain basic
information relating to individual transactions (e.g. item, time, branch,
etc.) and are therefore a useful source of information when performing
normal reconciliation activities.
8.13 However, as these reports are specific to counters and contain no
information beyond this, they would not allow a Subpostmaster to
determine the cause of an Hsues that arise at anthing beyond counter
level (and possibly even those that arise at counter level).
8.14 For example, if an APS transaction reversal was carried out, the
Subpostmaster would receive a receipt which looks something like the
following:
Example content
a 2 3 4
123456789012345678901234567890123456789012
01 I Feitham Post Office FAD: 123456%
02 I 23/09/2006 10:47 TP:06 BP:01 SU:SH1
03
04 REVERSAL
05
06 *** Branch Copy — Retain *#*
08 I Checksum: 9062852
09 I APS No: 010058010057
10 I Client: Eastern Electricity
li I Scheme: EE MthBill Sve: 8
12 I Token Type: BC Entry: 1
13 I Ref: 6331801325640003333
14 I Amount: 5.06- Cash
15 I Product No: 7022
16
5 2 [ialetetetatetaiedatstsiniaisnatatatatatatetststsisnststatstatsatstetsieisistatad
123456789012345678901234567890123456789012
8.15 This shows enough information for a Subpostmaster to balance a £5
reversal if it is assumed that all other factors are in order. However,
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 152
Number: 1_ Author: Gareth Jer
delete
ge Number: 2_ Author: Gareth Jenkins Date: 26/10/2018 12:27:00
You wouldn't except the SMPR to worry about that as it shouldn't impact on his accounts.
Date: 26/10/2018 12:27:00
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 137 of 225
thdlk is no information relating to whether the transaction has
reconciled at APS Host or at any other level (harvester, client, etc.).
This could cause an issue which the Subpostmaster is effectively
powerless to resolve.
8.16 If, for example, th&harvester failed to process the £5 reversal above then
there would appear to be a £5 shortfall at the APS Host (and potentially
at every other level above harvester). The Subpostmaster would have
no way of identifying where the error occurred because, at counter level,
evélything would appear to balance, and they would not have access to
any information beyond that. This essentially means that the
Subpostmaster is completely reliant on either:
a. Horizon operating without any flaws (which is unlikely even in the
case of a robust system); or
b. Where there is a flaw, it will be identified and linked to a problem
transaction by Post Office / Fujitsu.
Additional Points to Note
8.17 There was no Formal reconciliation process between the POLSAP system
and the Credence transaction stream, so Credence could not be used to
verify financial integrity. POLSAP system transaction information should
have been used for this purpose.??3
8.18 A Post Office cash management improvement proposal ?** recently
acknowledged the lac of visibility for the Subpostmaster of any
inaccurate cash declarations. The proposed cash management
improvement proposal would give the Subpostmaster immediate
visibility of any discrepancies and allow faster corrections/investigations
to find the inaccurate transaction. It is not known if or when this change
223 SYMSDMSD0020_2.2.doc, End to End Reconciliation Reporting, 27 February 2012 [POL-0124572]
[aaedd5051156619c50296d04f6b2e779]
224 Cash Management Programme — Business Case June 2017 v1.5 TW.docx, Business Case - Cash
Management Programme, 4 August 2017, [POL-0220663] [8cd4dac4464197347 cOfb2348a6e8f59]
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Date: 26/10/2018 12:28:00
Such issues are not relevant to the SPMR. His accounts should balance if the reversal is carried out. What happens at the back end won't impact on his accounts.
There is perhaps a possibility that a TC might be raised, but this
ing that he should dispute.
Number: 2 Author: Gareth Jenkins Date: 26/10/2018 12:30:00
We would detect that and fix it. Worst case it might be delayed for a couple of days while SSC manually correct things.
Number: 3 Author: Gareth Jenkins Date: 26/10/2018 12:31:00
So then they aren't bothered
Number. 4 Author: Gareth Jenkins Date: 26/10/2018 12:32:00
Agreed. However the likelihood of any difference is small so Credence could still be used to investigate issues,
Number: 5 Author: Gareth Jenkins Date: 26/10/2018 12:33:00
Twould suggest that inaccurate cash declarations were common. They were probably only done accurately at Rollover when it impacted the Branch accounts. Nightly Cash
declarations were used for Cash planning, but for some SPMRs they were ignored. They are supposed to eb a tool to help the SPMRs, but some think it to be a chore
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 138 of 225
was to be implemented but it corflirmed that the Subpostmaster has
little control beyond counter level when trying to resolve any
discrepancies.
Issues 8 & 9 - Summary of Conclusions
8.19 The TPS, APS and DRS Report Sets along with reports produced in
accordance with the BIM and Problem Management Procedures provided
a suite of reports which should (subject to the above) allow Post Office
to identify the occurrence of alleged shortfalls in the Horizon system.
These reports were supported by manual processes which would provide
further information in relation to the underlying cause and resolution of
a given issue. In addition, there was nothing preventing Post Office from
seeking further information from Fujitsu directly if, for example, the
Problem Management Procedure was insufficient.
8.20 Subpostmasters had a much smaller pool of information available to them,
which is in line with what I would expect given that they are the end
users of the Horizon System. The reports and receipts available to
Subpostmasters generally show enough information for a
Subpostmaster to balance transactions if it is assumed that all other
factors are in order. HoPlever, this information would not allow a
Subpostmaster to determine whether a transaction has reconciled at
APS Host or at any other level (harvester, client, etc.).
8.21 There is evidence that some reports available to Subpostmasters were
reporting errGheous data because of changes made to stock units22°. In
this case some products were being double counted in a sales report. A
code fix was scheduled in COUNTER_EPOSS 34_7 but it is not known if
or when this was implemented.
228 CCard2053P, HORIZON KEL CCard2053P, 21 December 2005 (last updated 08 September 2006), [POLO035339]
[92ccb572ff4ed5f357d3569a4b4121e2]
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 154
Number: 1_ Author: Gareth Jenkins Date: 26/10/2018 12:35:00
No it doesn’t. There were tools to check cash variances nightly (or at any time) ~ if the SPM chose to use them.
Number: 2_Author: Gareth Jenkins Date: 26/10/2018 12:36:00
And that's irrelevant to the SPMR,
Number. 3_Author: Gareth Jenkins Date: 26/10/2018 12:37:00
“Can we check this out?
This sounds like the early days of IMPACT.
Number: 4 Author: Simpkins John Date: 06/11/2018 16:32:00 Z
This was a bug:
IF a stock unit has been deleted and then recreated later, the totals for this stock unit are counted twice on the Sales Report - once for the StockUnits object and again for the
DelStockUnits object.
PC0129406 is with development. Code fix scheduled for T20,
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 139 of 225
8.22 In conclusion, Post Office had access to far more comprehensive
information relation to the Horizon system. If an error occurred beyond
counter level, Subpostmasters would need to rely on Post Office to
identify and resolve the issue. If that issue or its Evas not properly
identified for any reason, then the Subpostmaster would be at risk of
being liable for a Transaction Correction.
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 155
Number: 1 Author: Gareth Jenkins Date: 26/10/2018 12:38:00
Missing word
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 140 of 225
9. Remote Access and Alteration of Transaction Data
Issue 7 - Were Post Office and/or Fujitsu able to access transaction data recorded
by Horizon remotely (i.e. not from within a branch)?
Issue 10 - Whether the Defendant and/or Fujitsu have had the ability/facility
to: (i) insert, inject, edit or delete transaction data or data in branch accounts;
(ii) implement fixes in Horizon that had the potential to affect transaction data
or data in branch accounts; or (iii) rebuild branch transaction data:
a. atall;
b. without the knowledge of the Subpostmaster in question; and
c. without the consent of the Subpostmaster in question.
Issue 11 - If they did, did the Horizon system have any permission controls
upon the use of the above facility, and did the system maintain a log of such
actions and such permission controls?
Issue 12 - If the Defendant and/or Fujitsu did have such ability, how often was
that used, if at all?
Issue 13 - To what extent did use of any such facility have the potential to affect
the reliability of Branches’ accounting positions?
Remote Access
9.1A number of technical documents provided by Fujitsu identify that the
Horizon estate was enabled to be managed remotely. This is not beyond
expected requirements. The nature of providing a support service as
Fujitsu do, would require as a design principle, that the Horizon solution
should be completely rerflotely manageable. 76
9.2 Document ‘1.8 8. Horizon Online Technical Network Architecture, Mark
Jarosz.pdf’ (dated 2010) at section 2.5.3 sets out the network facilities
226 POLSAP High Level Design, 25 March 2010 [POL-0032871] [66768ce6f7144c10d1b68f1ec2d612f8]
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 156
Number: 1_ Author: Gareth Jer Date: 26/10/2018 12:39:00
The ref here is to POL SAP which is nothing to do with Branch management.
However he is right, we have to be able to manage the estate remotely.
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 141 of 225
that allowed “Remote Access by Fujitsu services, users and
systems”, 227228
The document recorded that “Support access to Counters is via SSH...”
further, the document shows that the Fujitsu sites Bracknell, Stevenage
and Ireland were originally envisaged to have access.
9.3 A further document describing the Horizon Wide Area Network High Level
Design??? sets out:
“Remote support access to the counter will be provided through the
implementation of an SSH service running on the counter which can then
be accessed from the Secure Access Servers, in the data centre.... this
will allow access to command prompt on the counter for the retrieval of
logs and other data using secure copy (SCP)”.
9.4 It is therefore clear that Fujitsu had access to the servers which make up
the Horizon estate, so as to access counters within a branch. This access
was required to enable them to provide support and maintenance. 3rd
line support had the greatest level of privileges to “Support access on
platforms operating system, hosting applications and database
schemas”.?°° This level of access means that Fujitsu could practically
access all elements of data recorded within Horizon.
9.5 Further to the above access facilities, helpdesk logs identify that ‘Tivoli
Enterprise Manager’ (TEM)?3+ and Tivoli Remote Control tools were
specifically used for accessing Branch counters across the Horizon
estate. 732 Whilst the End to End Application Support Strategy
27 1.8 8, HNG-X Technical Network Architecture, Mark Jarosz.pdf, HNG-X Technical Network Architecture, ca.
228 [C-0003647] [4c0c965c5d6ca96a56059693625cf29C]
229 2.1 11. HNG-X Wide Area Network HLD, Stephen Wisedale.pdf, HNG-X Wide Area Network HLD, 19
December 2012 [C-0003646] [c86656726b30cea6fcle2dfeccb4457a]
230 4.3 3. High level architectural overview of Horizon Online.pdf, Horizon Solution Architecture Outline, 07 April
2016 [7b6cf8cf69bec90f674b9b10a64104e8]
231 M100_POL_003_HSH Fujitsu logs_JHKD.xisx, Fujitsu Helpdesk Logs, 28 October 2014 [POL-0006655]
[4120881082b106161bc4acbd75c15b7a]
22 RKing5147Q.html, HORIZON KEL RKing5147Q, 8 August 2006 [POL-0035318]
[1e2815966220f5c7 1ccd3c99bcf82c16]
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 142 of 225
document? records at 1.5.1 “Direct access to live Post Office counters
is currently only available to the SSC."
9.6 Whilst the above documentation identified that counters could be accessed
remotely for support purposes, it has not yet been identified that
tralkaction data was altered at the counter.
9.7 However, it is my opinion that it was possible Fujitsu was able to access
transaction data recorded by Horizon both within a branch (and also
within the central BRDB database for Horizon Online). This was how the
system was designed.
9.8 As evidenced above, it is entirely possible that Fujitsu could, as they could
run commands on the counter machine accessing and querying the hard
disk, perform modifications and deletions.
9.9 The witness statement of a Fujitsu chief architect?** further confirms that
Fujitsu carMland have inserted a balancing transaction into a Branch
Account, and in Legacy Horizon could inject transactions into branch
accounts (which at the time would have been stored on the branch
counter hard drive).
9.10 The Witness Statement of Richard Roll?°> further confirms that Fujitsu
employees could and did remotely access branch accounts to Berform
modifications.
Further supporting evidence of remote access
9.11A number of the Known Error Logs (KELs) refer to remote access activities
between the branches and the Fujitsu support facility.
233 SVMSDMPRO0875_1.DOC, End to End Application Support Strategy, 28 July 2011 [POL-0122492]
[db0644e4d5e1 1bScce3ed381cb108a88]
234 Witness Statement of Torstein Godseth, 27 September 2018 (Para 17.2 (d))
235 Witness statement of Richard Roll 11.07.16.pdf dated 11 July 2016
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 158
Number: 1_ Author: Gareth Jenkins Date: 26/10/2018 12:41:00
For HNGX, transactional data was not held at the counter (except while a Basket was being processed) and so couldn’ be altered
In Old Horizon it was not possible to alter transactional date, but was technically possible to Insert new transactions.
Steve Parker has covered this in his response to Richard Roll's WS.
Number. 2_Author: Gareth Jenkins Date: 26/10/2018 12:53:00
We did it once in March 2070.
Number. 3_Author: Gareth Jenkins Date: 26/10/2018 12:44:00
Not to trasnactions though
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 143 of 225
9.12 boismaisons1328M 2° 237 describes running commands on counters to
assess disk space sizes. Thi therefore illustrates Fujitsu’s capabilities
to access the counter hard disk.
9.13 KEL acha2026Q?** (last updated 27th September 2012) documents an
issue recorded as “Unable to connect to HNGX counter from ssn”. The
reasonable conclusion being that normally Fujitsu (or other support
personal) can connect to the Horizon counter from the Secure Access
Server. The KEL explains that: “SSC [Fujitsu Software Support Centre]
will be unable to connect to this counter until problem has been
resolved. Solution - ATOS".
9.14 KEL MillerK1837)2* illustrates that del
files could be made at counter level.
ions in relation to Reference Data
Global Branches
9.15 Fujitsu operate the Horizon Online Help Desk located at two sites
(Bracknell and Stevenage). These sites contain ‘global branches’
therefore they are not physical instances of a Post Office but exist as
‘virtual branches’. There are however, physical counters that perform
within them. 74° The ‘branches’ operate with branch codes 999999,
counter ID’s 1 - 6 and 999998, counter IDs 7 - 12.
9.16 The HNG-X Counter Business Application Support Guide?“ further sets out
how to perform an audit of which branch a global user last logged in at.
9.17 A register of branch codes (dated 2015)**! identifies a further global branch
recorded as WAKO1 Branch Code 999993.
236 boismaisons1328M.html, HNG-X KEL boismaisons1328M, 24 April 2012 (last updated 20 March 2013) [POL-
257 } [a7737d3b003d083b5a6c95c7edee(534]
238 acha2026Q.htmi, HNG-X KEL acha2026Q, 15 October 2010 (last updated 27 September 2012) [POLO039179]
[6a5c608d2812487a3c04447 3fc6a9e45]
259 MillerK1837).html, HNG-X KEL Millerk1837J, 04 June 2010 (last updated 24 April 2015) - [POL-0040112}
[1da69d5ae267f707c2702f926c2e4384]
240 DEVAPPSPGO017_7.1.doc, HNG-X Counter Business Application Support Guide, 8 January 2014 [POLO134853]
[97581900782d355b4965045331797cea]
241 DESGENSPE0013_1.doc, Register of Branch Codes, 4 November 2015 [POL-0142404]
(5536045daeafS87fec423f2782928123]
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Date: 26/10/2018 12:45:00
Tthink we accept that SSC can access counters, and potentially do anything once they get there.
»Number: 2_Author: Gareth Jenkins Date: 26/10/2018 12:46:00
Need to check this out. This was very early in HNG-X.
I assume that once this was done, the counter needed to be restarted to pick up correct Ref data.
Number: 3 Author: Simpkins John Date: 06/11/2018 16:46:00 Z
This is to ensure that the help reference data is delivered to a counter. There is no mention in the KEL about the deletion of reference data. It does talk about the folders where
the reference data is stored on the counter. However this KEL uses the correct distribution tools to send the pending help reference data to the affected counter.
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 144 of 225
9.18 An instance of a global branch wolld allow Fujitsu to create global users
and to input transactions within core Horizon systems as though they
had been entered from a physical branch.
9.19 It entirely possible that investigation could be further conducted by Post
Office to identify any transactions held within the BRDB containing the
Branch Codes identified above. Such would identify where and what
transactions had been performed by Fujitsu global branches and not a
Subpostmaster.
Branch Transaction Data Rebuilds
9.20 It is understood that branch transaction data rebuilds did take place across
the estate.
9,21 POL-0116724242 documents the reasons for changing a counter base unit.
Replacement of the counter base unit at 2008 would ultimately require
the brdlch transaction data to be reinstated to the new machine. This
would be carried out by both the engineer on site and automatic
Processes applied by the HSD (Horizon Service Desk).
9.22 Further, Richard Roll in his witness statement at paragraph 15747
documents how it was relatively common to re-Ebeate branch databases
in an effort to fix corruptions.
Transaction Correction Tools — Modification of Transaction Data
9.23 There are various identified points within the Horizon architecture where
Fujitsu may need to perform data correction activities. the involves
manually correcting data where it has become corrupted or is harvested
as in an ‘error’ or ‘exception’ state.
9.24 POL-02193107*9 at Section 5.6.2 states:
242 SYMSDMSTD0810_1.2.doc Engineer Handbook Base Units, 20 June 2008 [POL-0116724]
247 Witness statement of Richard Roll 11.07.16.pdf dated 11 July 2016 paragraph 15
249 DESAPPHLD0020.doc Branch Database High Level Design, 22 February 2018 [POL-0219310]
[e277e9fd3e3ad2dd17ab40afc0d2096d]
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 160
Number. 1_ Author: Gareth Jenkins Date: 26/10/2018 12:48:00
No. A Global User would be created in the Global Branch
However the Global User would need to Log in at a Physical Branch to carry out transactions at that Branch. I think Torstein covers this in his witness statement. The Global
Branch was purely to control and manage global users and did NOT support transaction generation,
sNumber: 2_Author: Gareth Jenkins Date: 26/10/2018 12:49:00
There are no such transitions,
Can we prove that?
We could pull logs from the audit tral for those branches perhaps?
Number: 3_ Author: Simpkins John Date: 06/11/2018 16:50:00 Z
Some role have gat the ability to perform transactions.
Document DES/GEN/SPE/0007 has a matric with the roles and their abilities.
Number: 4 Author: Gareth Jenkins Date: 26/10/2018 12:51:00
Do we need to describe this process using squirrels and caches again?
Number: 5 Author: Gareth Jenkins Date: 26/10/2018 12:52:00
Yes, but just using the data copies held on other counters - not new datal
Number. 6 Author: Gareth Jenkins Date: 26/10/2018 12:53:00
However this is all done on a copy of branch data and has no impact on Branch accounts.
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 145 of 225
“Thdke is a requirement that the SSC will have ability to insert balancing
transactions into the persistent objects of the Branch Database. There
are reasons for SSC having to do so e.g. to rectify erroneous accounting
data that may have been logged as a result of a bug in the Counter / BAL.
SSC will have privileges of only inserting balancing /correcting
transactions to relevant tables in the database. SSC will not have the
privileges to update or delete records in the database.
Any writes by SSC to BRDB must be audited...”
9.25 When applying the corrective fixes, it appears that Fujitsu would utilise the
brdch accounting code of the branch for which the correction
transaction was required.?4+
Branch Transaction Correction Tool
9,26 Host BRDB Transaction Correction Tool Low Level Design?*5(applicable to
Horizon Online).
9.27 The ‘Overview’ section (section 1.1) states:
“Warning: The use of this powerful tool has inherent risks. If the SQL
statement is incorrect or badly written, it is possible to cause unintended
consequences, some of which may cause serious problems to the Branch
Database. It is expected that only a small number of skilled staff will run
this tool and that they will have detailed guidance as to when and how to
use the tool.”
9.28 Théldocument does not stipulate which staff within SSC have privilege to
run the tool, nor document the guidance on specifically when the tool
should be run. This has been the subject of a Request for Further
Information.
Correction Tool auditing
244 DESAPPHLD0020.doc Branch Database High Level Design, 22 February 2018, Para: 5.6.2 [POL-0219310]
[e277e9fd3e3ad2dd17ab40afc0d2096d]
245 DEVAPPLLDO142.doc, Host BRDB Transaction Correction Tool Low Level Design, 13 November 2007
[POL0032866] [e20de9a651 b8baf1f84e10859455684b]
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 161
Number: 1_ Author: Gareth Jer
Date: 26/10/2018 12:54:00
‘This process has been used once in 2070. Such transactions are easily identifiable as having been written at counter 99.
Number: 2 Author: Gareth Jenkins Date: 26/10/2018 12:55:00
We would Rave to if they were to form part of the Branch accounts
Number: 3 Author: Gareth Jenkins Date: 26/10/2018 12:56:00
“ Tewas used once by an experienced member of SSC.
The script was checked by BRDB Development and myself
Its application was authorised by POL.
Do we have a work process for this, or as its use is deprecated did we not produce one?
Number: 4 Author: Simpkins John Date: 06/11/2018 17:58:00 Z
~ Teannot find any work instruction so I would expect each usage would have been guided by 4th line,
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 146 of 225
c
ane. TMESTAN
sae
:
Tbmesione, TIMESTAMP
8,
9.30
9.29 The schema definition for the BRDB identifies the following auditability: 2°
Users of the branch correction tool should be identifiable by its audit
table BRDB_TXN_CORR_TOOL_JOURNAL (as above, bottom right
table). The corrective transaction in the branch accounts should also be
identifiable as an automatic ‘SYSTIMESTAMP’ should be recorded rather
than a retrospective transaction time. ‘SUPPORTTOOLUSER’ should be
reflected in the ID field.
9.31 It is entirely possible that investigations could be further conducted by Post
Office to identify any transactions held within the
BRDB_TXN_CORR_TOOL_JOURNAL table. Such would identify what
corrective transactions have been performed. However, given file
246 DEVAPPLLD0199_5.DOC, SCHEMA DEFINITION FOR BRANCH DATABASE, STANDBY BRANCH DATABASE
AND BRANCH SUPPORT SYSTEM, 01 August 2017 [POL-0151776] [456dd7c0f245946a2f1658cc759a527a]
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 147 of 225
retention periods, it is unlikely that all corrective transactions will have
been retained from the inception of Horizon Online.
9.32 A Request for Information issued to Post Office querying how many times
the Host BRDB Transaction Correction Tool has been used was
nded by Fujitsu stating:
"This process has only been used once, in relation to PCO195561, on 11-
Mar-2010.”
Tie ransaction Repair Tool
9.33 TPS - EPOSS Reconciliation - TIP Transaction Repair 247 documents a
further maintenance tool that “...will assist SSC to repair EPOSS
transactions processed at the counter but are unable to copy from BRDB
into the TPS (Transaction Processing System) Host.”
9.34 The repair tool is documented to assist SSC to repair/re-repair the
transactions and send them to TiFtransaction Information Processing)
(the remote end-point of the File Transfer Management System (FTMS)
service that delivers reconciliation reports to Post Office Limited / PON’s
(Post Office Network Banking) Transaction Information Processing
System).
9.35 The above tool allows corrective actions to be performed upon data within
Core Horizon after the counter has processed the transactions and they
are flagged as erroneous as they are sent through the various
processing systems.
9.36 The transaction repair is facilitated by a Form based tool that will:
a. read data from the TPS Host table holding Harvester Exceptions
b. allow the user to repair/re-repair the transactions, i.e., input any
missing data or modify any invalid data
5]
247 PIDESO08.doc, TPS - EPOSS Reconciliation - TIP Transaction Repair, 11 January 2017 [POL-0032939]
(27249cBe2ccbafecde16223cddasf7a}
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 163
Number: 1_ Author: Gareth Je
“So why are you disputing this?
Number: 2_ Author: Gareth Jenkins Date: 26/10/2018 13:02:00
Use of this has no impact on branch accounts and just affects the data sent to POL's backend systems:
Number. 3 Author: Gareth Jenkins Date: 26/10/2018 13:00:00
“Tdon’t think they go that way anymore. That was the architecture prior to IMPACT. They are now copied directly to Credence fle shares.
Number: 4_ Author: Gareth Jenkins Date: 26/10/2018 12:58:00
think we Rave been able to prove that it was only used the once.
Date: 26/10/2018 12:59:00
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 148 of 225
c. validate and insert the corrected transactions into a 65th TPS Host
partition table
9.37 Further, at 3.2.1 the tool is stated to:
“Allow the user to do multiple repairs at speed with the minimum of user
intervention i.e., if there is a common error among several records then the
user can input the valid value just once and correct all the records having
the similar error...”
9.38 In relation to hol often this tool was used, Fujitsu have provided the below
response:
“There is a master MSC every 12 months, each time such a modification
is carried out it is itemised as an MSC related to the master MSC; however
master MSCs contain many various types of changes, to determine the
number that relate to this particular modification type Fujitsu would have
to carry out analysis of all individual tasks on all master MSCs. Whilst this
type of action may have been taken by SSC it would have been in the
context of an individual incident. All incidents are recorded but the system
was designed to manage individual operations not for statistical reporting
for when a particular action has been taken by a Support Consultant.
Fujitsu will be able to answer questions on individual branch queries
where the data is still available.”
9.39The witness statement of William Membury28 at paragraph 27 explains
the use of the MSC toolset which is used to manage changes to Horizon
Online. He explains that the MSC tool set “provides assurance” as to the
changes made to Horizon. Whilst I have requested copies of the MSC
documentation at the date of this report Post Office has refused to
provide these.
Effects on Branch Accounting
9.40 The above tools are noted as having the potential to affect transaction
data and potentially branch account data by way of incorrectly altering
248 Witness Statement of William Membury, 27 September 2018. (Para: 27)
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Date: 26/10/2018 13:03:00
T know it was quite common for Old Horizon (Mails was a particular culprit). Do SSC have a feel to its usage on HNG_X? Do we still have issues with late APS Reversals? Is there
anything else it is used for?
I guess lam asking can we be more helpful - even if we can't easily prove things.
Presumably there would be a Peak for each occurrence and we have already disclosed all Peaks!
a jNumber: 2_ Author: Simpkins John Date: 06/11/2018 18:12:00 Z
We do still use this tool, MSC subtasks were raised for each usage (and a Peak for the SSC) and therefore usage number should be easily available from SecOps.
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 149 of 225
the transactions prior to entering the recipient systems such as POLSAP.
and External Clients (after processing by the counter). Thereby resulting
in potential discrepancies being borne between recipient systems and
those recorded in branch.
9.41 This could potentially result in the issuing of a Transaction Correction issued
by Post Office who may be unaware of the error induced within
the processing of the corrected transaction. Once accepted by branch
this ultimately modifies the branch’s accounting position.
Notwithstanding, the possibilities of human input error exists,
specifically if the aim is to perform “multiple repairs at speed”.
9.42 Whilst 9 ai
as validated in terms of valid data types according to the check
e states that transactions will be validated, this is interpreted
constraints imposed on those specific columns. For example, should a
clerk enter a date into a value field, then it is assumed that that will fail
validation however, if a clerk enters an amount of 900 where it is
supposed to record 90, it is doubtful that the validation rules will
interpret that as incorrect as the numerical value type is correct but
there is no Balidation against the amount. Therefore, human error is
possible. Projecting onwards, since these values are then sent onto
clients, there is facility for discrepancy potdntially resulting in a
Subpostmaster being liable to pay for a discrepancy over which they
have no control.
9.43 Therefore, in respect of Issue 10 point (i), it appears that Fujitsu did have
the ability to insert, inject, edidand (potentially) delete transaction data
and (ii) had the ability /facility to implement fixes in Horizon that had
the potential to affect transaction data or data in branch accounts.
9.44 In respect of Issue 10 part (iii), no evidence of specific branch data
rebuilding has been discovered however, since Fujitsu had the remote
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 165
Number: 1_ Author: Gareth Jer
Not sure I understand this ref.
Date: 26/10/2018 13:07:00
Number: 2_ Author: Gareth Jenkins Date: 26/10/2018 13:07:00
There is some validation as defined in ref data
I accept 90 or 900 is unlikely to be checked, but may products have maximum allowed values to avoid major typing errors.
Number: 3 Author: Gareth Jenkins Date: 26/10/2018 13:09:00
is anybody claiming that this has happened?
Number. 4 Author: Gareth Jenkins Date: 26/10/2018 13:09:00
No we could not edit transaction data that impacted the counter.
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 150 of 225
access Capabilities it is entirely possible that it could have occurred, and
the witness statement of Richard Roll refers to the ability.
9.45 Fujitsu had the capability and dedicated tools to alt transaction data held
in the BRDB and Horizon processing systems remotely, therefore it is
likely this would be without the knowledge or consent of the
Subpostmaster (of the branch that the transactions applied within).
Further actions affecting branch accounting
9.46 HNG-X Counter Business Application Support Guide?*? documents Help
Service Desk calls that might result in a stock unit needing to be
unlocked on a counter which can be performed by glodal users with
Admin role:
"In such situations the clerk must call the Horizon System Desk and
request the stock unit to be unlocked. This call is passed on to System
Support Centre (SSC) who operates a manual process to unlock the stock
unit. To ensure proper accountability this manual process has a high
administrative overhead which results in significant work for SSC staff and
a long delay for the clerk before the stock unit is unlocked. This function
unlocks the locked stock unit in a quick, secure and audited way.”
9.47 The document further sets out:
“During the logon process, the user will be forcibly logged off if some
unexpected error occurs (during recovery or post logon checks (e.g.
change password). If this happens repeatedly, in extreme cases and as
a short-term measure purely as a workaround, then the recovery and/or
post-logon checks can be bypassed. It must be stressed that this is only
to be used for situations where a very fast workaround is required in
live. To bypass recovery at logon, use the following
application.properties override:logon.recoveryEnabled=false To bypass
post-logon checks, use the following application.properties
override:logon.postLogonChecksEnabled=true Note that in both cases,
249 DEVAPPSPGO017_7.1.doc, HNG-X Counter Business Application Support Guide, 8 January 2014 [POLO134853]
[97581900782d355b4965045331797cea]
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 166
Number: 1_ Author: Gareth Jenl
No we couldn't edit data in BRDB.
Date: 26/10/2018 13:10:00
The DBAs in Belfast had permission to amend the tables (to allow insertion of extra rows), but they wouldn't need to look at the data.
SSC didn’t have write permission to the transaction or Audit tables.
Number: 2_Author: Gareth Jenkins Date: 26/10/2018 13:15:00
This functionality purely allows the SU specified in the branch specified to be unlocked. It doesn't allow the Global admin user to do anything else to the branch data
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 151 of 225
when these flags are used, once logon is eventually completed, the user
may need to click on the back office/front office button twice in order to
“reset” the counter to a good state.”
9.48 Rebbtting the counter and bypassing recovery jobs could impact branch
accounts in that items awaiting recovery might be lost. When someone
from NBSC investigates this in the event of a discrepancy, they would
probably not be aware of the action carried out to bypass recovery.
Therefore, their understanding and belief in respect of what should
happen in normal recovery procedure will be at odds with actuality. In
this instance it could be that the Subpostmaster bears the effects of this
(unknown to Post Office) activity.
Supporting Evidence of remote access and implemented fixes affecting
transaction data
9.49 KEL SeemungalG519Q 25° reckds an instance where transaction
amendments carried out using the above TIP repair tool at 9.13 are
causing exceptions within the BRDB.
9.50 KEL MHarvey2255P25" rec@rds the manual addition of corrective balancing
transactions inserted by SSC affecting the TPS system.
9.51 There are potentially more KELs and PEAKs that record the effects of
performing corrective transactions which have not been identified at the
time of this report.
Permission Controls and Data Auditing
9.52 Typically, in an architecture such as Horizon, there are permission controls
set out by roles and privileges that accommodate a support service’s
ability to manage the estate.
250 SeemungalG519Q.html, HNG-X KEL SeemungalG519Q, 15 January 2010 (last updated 28 April 2014)
[POL0039787] [ba2c5c8717aeb77c9a65452e29ec57 ba]
251 MHarvey2255P.html, HORIZON KEL MHarvey2255P, 02 September 2005 (last updated 31 May 2006)
[POL0035233] [ebd26444c25a88aa178de7cd4be58d27]
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 167
Number: 1_ Author: Gareth Jenkins Date: 26/10/2018 13:16:00
Do we know if this was ever done by SSC? Can we prove it?
This sounds like a dangerous tool introduced by Dev, which should be severely restricted. I wasn't aware of it until now.
Number: 2_Author: Simpkins John Date: 06/11/2018 18:15:00 Z
There were some KELS that relate to us setting the counter last JSN file when the APADC recovery script was failing and the counter was stuck in a logon loop.
Any such activity would require the sign-off of POL as this could mean some rollback recovery did not happen.
Number: 3_ Author: Gareth Jenkins Date: 26/10/2018 13:18:00
“This sounds like something found in test.
Can we confirm that?
Number 4 Author: Simpkins John Date: 06/11/2018 18:22:00 Z
°C0192868 applies, this was a live incident. It was due to the HYDRA migration period caused by missing TP in the TPS data (there was a period where this was not calculated by
the TPS harvesters),
Number: $ Author: Gareth Jenkins Date: 26/10/2018 13:19:00
~ This was old Horizon, Can we check the details please.
Number: 6 Author: Simpkins John_Date: 07/11/2018 11:41:00 Z
PCO125333 applies, this was related to the SSC adding balancing TPS transactions following a problem with missing POLFS mappings.
My understanding is that if some product mappings are missing or are incorrect but the remaining transactions for an outlets trading day still balance to zero then a BLE will be
produced and the unmapped txns will not be reported anywhere, in this case 2 such transaction were ‘accidently’ identified whilst reconstituting txns for non balancing outlets.
affected by the same missing mappings.. A rare circumstance but please pass the TPS-Dev for comment.
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 152 of 225
9.53 Administrator capabilities will allow for the management and maintenance
of the system. Database administrators will have escalated privileges to
allow them to implement changes within the database systems and
delegate roles and privileges to others. For example, to set permission
controls for ‘managers’ of a branch, elevated to that of a ‘user’.
9.54 Typically, documented procedures will be in place to set out what each
defined role can perform within a system. Audit logging is common
practice aside from the electronic logging that will occur in a system.
Administrator actions will typically be overseen by other methods of audit.
9.55 For example, the common “four eyes’ principle ensures that an action to
be performed is approved and observed by more than one individual
where the action to be performed can have a significant effect (as
administrator capabilities typically do).
9.56As there are a number of tools and means by which Fujitsu can perform
alterations within the Horizon system (since supporting the system was
their primary role) the findings below regarding the auditability of
permission controls is limited to those which relate to transaction data
amendments which had the ability to affect branch accounts (in answer
to Issue 11).
Balancing / Corrective Transactions
9.57 Reconciliation Service: Service Description?® records the following:
“1¢ he Reconciliation Service identifies that Transaction data held on the
‘central database’ located at the Data Centre is found to be inconsistent
when compared to the records of the Transaction that was completed at
the Branch, e.g. a receipt, a Transaction log or a Branch accounting
discrepancy, the Reconciliation Service shall obtain authorisation from
Post Office prior to the insertion of corrective Transactions.”
252 SYMSDMSD0015_4.doc, Reconciliation Service: Service Description, 03 December 2013 [POL-0134458]
[49b8029c58cb2f7 14b2ba7e034d6f280]
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 168
Number: 1_ Author: Gareth Jenkins Date: 26/10/2018 13:21:00
And this was done on the one occasion the tool was used.
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 153 of 225
9.58 It is understood that the request for authorisation as referred to above
may be that documented within the CuSlomer Service Operational
Change Procedure.?°? The document sets out the process requirements
in respect of operational changes where changes are made to the live
environment.
Operational Change Proposal
9.59 An OCP is raised in order to make a change to the live system. The process
is administered by Post Office Account Operations and is available to all
users for the administration, authorisation and auditing of changes
made to the live operational service.
Operational Correction Request
9.60 The OCR process involves the correction of customer data on the live
system and because user data is involved, requires different approvals
and auditing. The document further states:
“Only the SSC has the authority to make changes to the data on the system,
and therefore only SSC staff can action an OCR.
In most cases, an OCR does not involve the financial integrity of the
system. Under these circumstances one of the SSC Manager, the Support
Services Manager or the Customer Service Duty Manager can approve an
OCR. If the data to be changed has a financial impact on Post Office, then
approval must also be given by a senior Post Office Manager.
When an OCR has been approved, and has been actioned, it is necessary
for two users of the OCP system to confirm that the work has been done
- an actionee and a witness. The actionee will always be an SSC staff
member, the witness can either be an SSC staff member or a
development staff member.”
9.61 Since the definition of what "involves the financial integrity of the system”
is not documented, it is not clear whether those OCRs that could then
253 CSPRDO19_1.doc, Customer Service Operational Change Procedure, 18 March 2004 [POL-0074909]
[439e3230106a96792bf2dd9831729392]
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 169
Number: 1_ Author: Gareth Jenkins Date: 26/10/2018 13:21:00
twas an MSC think
Number. 2_ Author: Simpkins John Date: 07/11/2018 11:48:00 Z
Tthink OCP25882 applies.
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 154 of 225
be approved without Post Office’s authorisation were appropriately
decided. ultfhately, Fujitsu could approve and action the OCR
independently.
9.62 With regards to Issue 12 and how often facilities were used in relation to
accessing/modifying transaction data a Request for Information issued
to Post Office provided the response:
"there are in excess of 36,000 MSCs and OCPs combined; and.
+ OCRs would not be used for any such change (OCRs were used minor
support changes that did not required the full approval process that was
needed for OCPs)...”
9.63 Thal above is inconsistent with the operational change document at 9.60
which maintains that they would apply for Live data changes.
Audit Servers
9.64 The audit servers provide an audit trail of all information on the Horizon
Online system. In order to ensure that this audit trail is irrefutable the
teams which have the ability to change data (i.e. SSC) must not also
have the ability to change the audit trail. For this reason, Audit server
3rd line support rests with the Audit development team and not the
SSC. This is known as ‘separation of duties’ and is a type of control.
ss Variations
9.65 The Ernst & Young review of Post Office’s systems of internal control
conducted in March 20112 observes in relation to Horizon (back end)
user administration:
“During our testing of the appropriateness of users with access to the
Horizon back end environment we noted one user whose access was no
longer required due to a change in job responsibilities. When users have
access to environments which are not appropriate for their job function
254 POL Management Letter FINAL.docx, Management letter for the year ended 27 March 2011, (Section 2.
Prior Year Comments-Update - Item 15) August 2011 [POL-0219218] [9d7862698d2a0f6af2a3c55590763bb7]
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 170
Number: 1_ Author: Gareth Jer
SSC to address.
Date: 26/10/2018 13:23:00
Number: 2_ Author: Gareth Jenkins Date: 26/10/2018 13:23:00
Sto address,
Number. 3 Author: Gareth Jenkins Date: 26/10/2018 13:25:00
“We need to address this,
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 155 of 225
there is the risk that users may inappropriately or accidentally use the
access leading to loss of application or data integrity.”
9.66 As previously opined, where process is not followed, this enhances the
likelihood for the possibility of undetected error.
9.67 Further, as noted in the Ernst & Young document?®> there were weak user
account management controls and the granting and monitoring of user
access was highlighted as an area of concern:
“Unrestricted access to privileged IT functions increases the risk of
unauthorised/inappropriate access which may lead to the processing of
unauthorised or erroneous transactions.”
Opinion Summary
9.68 A number of technical documents provided by Fujitsu identify that the
Horizon estate was enabled to be managed remotely.
9.69 This is typically expected in an estate such as Horizon to enable Fujitsu to
provide support services without physically having to send an engineer
out to each and every branch at the notification of any issue. Software
roll-outs and updates to operating system components are typically
largely distributed this way.
9.70 Regarding Issue 7, the documents show that Fujitsu could and did
remotely access the transaction data recorded by Horizon. Several
technical tools exist with the specific purpose of allowing Fujitsu to carry
out mdilifications and corrective fixes to transaction data. In addition, a
number of external audit reports commissioned by Post Office show that
this access was ofth not done without the appropriate control
mechanisms in place.
9.711In respect of Issue 10 documents show that a wide range of users at
Fujitsu did and do have the ability and facilities to access and mdiify
255 POL Management Letter FINAL.docX, Post Office Limited - Management letter for the year ended 27 March
2011, [POL-0219218] [9d7862698d2a0f6af2a3c55590763bb7]
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 171
Number: 1_ Author: Gareth Jer
Not to branch data.
Date: 26/10/2018 13:25:00
How do we handle his assumption that a modification to data sent to POL's back end systems will then give rise to an incorrect TC which then puts the SPMR out of pocket?
Number. 2 Author: Gareth Jenkins Date: 26/10/2018 13:26:00
isn't this a bit harsh?
Number: 3 Author: Gareth Jenkins Date: 26/10/2018 13:27:00
Not modify as far as the branch accounts are concemed
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 156 of 225
transaction data. Fujitsu staff were able to implement changes that had
the potential to affect transaction data both without the knowledge or
consent of the Subpostmaster and/or Post Office. In Audition, a number
of external audits commissioned by Post Office report that the
appropriate control mechanisms to prevent mistakes being made were
not followed.
9.72 Regarding Issue 11, business process rules should apply in relation to
accessing and modifying transaction data. It is reported that a
documented audit log of each and every occasion of live data access
exists, however, this has not been made available by POL.
9.73 Issue 12 is subject to a Request for Information currently disputed by Post
Office as to the relevancy of the request. Whilst Post Office states that
there are in excess of 36,000 documents regarding how often the access
was granted (outside of actions not needing express authorisation that
could also be carried out to fix data), these have not been made
available for review.
9.74 Regarding Issue 13 it is understood that the implications of Fuftsu carrying
out corrective fixes to data within the Horizon system could have the
potential to affect the reliability of Branches’ accounting positions.
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 172
Number. 1_ Author: Gareth Jenkins Date: 26/10/2018 13:27:00
We have to address this!
ge Number: 2_ Author: Gareth Jenkins Date: 26/10/2018 13:28:00
As stated above, I disagree.
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 157 of 225
10. Expert Declaration
I JASON COYNE DECLARE THAT:
10.11 understand that my duty in providing written reports and giving evidence
is to help the Court, and that this duty overrides any obligation to the
party by whom I am engaged or the person who has paid or is liable to
pay me. I confirm that I have complied and will continue to comply with
my duty.
10.2 1 confirm that I have not entered into any arrangement where the amount
or payment of my fees is in any way dependent on the outcome of the
case.
10.31 know of no conflict of interest of any kind, other than any which I have
disclosed in my report.
10.41 do not consider that any interest which I have disclosed affects my
suitability as an expert witness on any issues on which I have given
evidence.
10.51 will advise the party by whom I am instructed if, between the date of my
report and the trial, there is any change in circumstances which affect
my answers to points 10.3 and 10.4 above.
10.6 I have shown the sources of all information I have used.
10.71 have exercised reasonable care and skill in order to be accurate and
complete in preparing this report.
10.8 I have endeavoured to include in my report those matters, of which I have
knowledge or of which I have been made aware, that might adversely
affect the validity of my opinion. I have clearly stated any qualifications
to my opinion.
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 158 of 225
10.91 have not, without forming an independent view, included or excluded
anything which has been suggested to me by others, including my
instructing lawyers.
10.101 will notify those instructing me immediately and confirm in writing if,
for any reason, my existing report requires any correction or
qualification.
10.111 understand that:
a. my report will form the evidence to be given under oath or
affirmation;
b. questions may be put to me in writing for the purposes of clarifying
my report and that my answers shall be treated as part of my
report and covered by my statement of truth;
c. the court may at any stage direct a discussion to take place
between experts for the purpose of identifying and discussing the
expert issues in the proceedings, where possible reaching an
agreed opinion on those issues and identifying what action, if any,
may be taken to resolve any of the outstanding issues between the
parties;
d. the court may direct that following a discussion between the
experts that a statement should be prepared showing those issues
which are agreed, and those issues which are not agreed, together
with a summary of the reasons for disagreeing;
e. I may be required to attend court to be cross-examined on my
report by a cross-examiner assisted by an expert;
f. lam likely to be the subject of public adverse criticism by the judge
if the Court concludes that I have not taken reasonable care in
trying to meet the standards set out above.
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 159 of 225
10.121 have read Part 35 of the Civil Procedure Rules, the accompanying
practice direction and the Guidance for the instruction of experts in civil
claims and I have complied with their requirements.
10.131 am aware of the practice direction on pre-action conduct. I have acted in
accordance with the Code of Practice for Experts.
Statement of Truth
10.141 confirm that I have made clear which facts and matters referred to in
this report are within my own knowledge and which are not. Those that
are within my own knowledge I confirm to be true. The opinions I have
expressed represent my true and complete professional opinions on the
matters to which they refer.
Jason Coyne
Partner
Dated: 16 October 2018
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 160 of 225
11. Appendix A
PEAKs of initial interest
11.1 Summaries and key excerpts from a select number of PEAKs which could
have a financial impact upon branches (these are also narrated within
the main body of the report).
Errors with Financial Impact
P. 27887 (1999256
11.2 Summary: FAD011523 - receipts and payments misbalance
11.3 Date raised in PEAK: 21 July 1999
11.4 Discrepancy value: £1,082,540.28
11.5 Days open: 407 days
Key Excerpts
Date: 21-Jul-1999 15:25:00 User: _Customer Call_
“for fad code: 011523, on week 9 receipts and payments misbaince of
£1337.05, week 10 misbalance of £24000, week 11 misbalance of £12000,
week 12 misbalance of £1051111.48 and week 13 misbalance of £17426.05,
she has a difference on week 11 of balance due to post office and balance
brought fwd on week 12 of £1082544.32 overall these weeks net out a
difference of £ 27343.84 . she needs business support(reconciliation ) to look
into this”
256 PC0027887.html, PEAK PC0027887, 21 July 1999 [POL-0221773] [93af66e221ecfde6fcaad8a5acl 4eca4]
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 176
Number: 1 Author: Gareth Jenkins Date: 26/10/2018 16:28:00
This is probably too Tong ago to investigate.
The peak mentions a corrupt ll, so presumably it is a one off.
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 161 of 225
Date: 27-Jul-1999 10:09:00 User: Deleted User (Mike Croshaw
Sep/00)
“CAP12 Balance brought forward was multiplied twice due the known software
error. The inital balance brought forward for this CAP was £1196622.72. This
was multiplied twice to give a total BBF of £2279189.04. The discrepancy was
therefore £1082540.28. This was due a known software error which has no}
been resolved.”
Date: 03-Aug-1999 13:41:00 User: Barbara Longley
“The Call record has been assigned to MSU Team Member: Nicole Meredith”
Date: 17-Aug-1999 14:32:00 User: Nicole Meredith
“A RED report was issued for this incident and I am awaiting confirmation forI
this call to be closed."
Date: 24-Aug-1999 14:15:00 User: Nicole Meredith
“Julie Dart (POCL TP) is unwilling to close this call. She does not believe that
the Receipts and Payments misbalance was caused by the B/F figure doubling
up. Please investigate why this misbalance occurred in CAPs
9,10,11,12 and 13 and provide evidence if possible.”
Date: 25-Aug-1999 09:51:00 User: Deleted User (Mike Croshaw
Sep/00)
“Evidence for these dates has now been archived. Will update call further when
archived evidence has been retrieved”
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 162 of 225
Date: 02-Sep-1999 14:58:00 User: Deleted User (Mike Croshaw
Sep/00)
“Archived message store is now available - further investigation will be carried
out on Tuesday.”
Date: 08-Sep-1999 12:11:00 User: Deleted User (Mike Croshaw
Sep/00)
“The discrepancies for CAP 10 and CAP 11 are related to the transfer incident
covered by two previous calls, E-9906020405 & E-9906140099.”"
Date: 15-Sep-1999 08:51:00 User: Deleted User (Mike Croshaw
Sep/00)
“Jan Holmes is currently working on extracting the archive data for CAP13 andI
CAP9, which are the two CAPS we have outstanding queries about.”
Date: 21-Sep-1999 08:35:00 User: Barbara Longley
“In the absence of Mike, can another team member please take this call - I
believe that there is a disc on his desk which is connected with this call. The
call summary has been changed from:-FAD011523 - receipts and payments
misbalance The call summary is now:-FAD011523 - receipts and payments
misbalance The Call record has been transferred to the Team: EDSC"
Date:01-Oct-1999 15:03:00 User: Rakesh Patel
“Furthur evidence from Jan has not arrived.Passing back to Mike for
continuation.”
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 163 of 225
Date: 04-Oct-1999 14:14:00 User: Deleted User (Mike Croshaw
Sep/00)
“New evidence added - Files sent by Jan Holmes to SSC...I have added the
files sent by Jan Holmes as evidence to this call. Further evidence will
probably be required from the archive server. Currently the CAP9 and CAP13
discrepancies are still outstanding on this call. I have asked Les Ong/Brian
Orzel to take a look at the call and if possible provide guidance.”
Date: 13-Oct-1999 10:53:00 User: Deleted User (Mike Croshaw
Sep/00)
“evidence deleted - CAP13 New evidence added - Dump of message store}
taken on 24/08/99 — includes”
Date: 13-Oct-1999 11:00:00 User: Deleted User (Mike Croshaw
Sep/00)
“I spoken to Les Ong re. this call, he has informed me that Steve Warwick is
investigating it.”
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 164 of 225
Date: 13-Oct-1999 16:49:00 User: Steve Warwick
“The evidence supplied in the Excel files covers 26th May to 30th May. CAP 9
dates were from 20th May to 26th May, therefore I am unable to determine
the CAP 9 cause of the £1337.05 imbalance from the evidence presented.
I am also unable to analyse the CAP 12 and CAP 13 (10.6.99 to 23.6.99)
imbalances because the data required is not contained in either the
spreadsheets or the message store extracts provided. (The only data for this
period in the message store extract represents either persistent objects which
were still current at the time of the extract or messages which have an expiry
period > 35 days).
I can concur with Mike Crowshaw's explanation of the imbalances in CAPs 10
and 11. These were due to a stock transfer for £12,000 which was not settled
correctly due to the presence of a corrupt EPOSSSettlement.dil file on the PC
involved. The result of this was that the transfer out could not be accepted
by the receiving stock unit, leaving a Cash Account imbalance in
CAP 10 of twice the value. The effect of this error in CAP 10 was to leave the
overall balance due to POL £12,000 short causing the BBF for CAP 11 to be
short by this value and hence the £12,000 imbalance in CAP 11.”
Date: 12-Jan-2000 09:54:00 User: Lionel Higman
“CSR is no longer a valid target release. Moving target forward to earliest validI
value. Target Release updated to CSR-CI2_2R"
Date: 05-Jul-2000 16:01:00 User: Deleted User (Anna Croft Sep/00)
“HSH are chasing an update on this call call still with Steve Warwick -QFP”
Date: 06-Jul-2000 11:03:00 User: Lionel Higman
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 165 of 225
“This PinICL has been assigned a CS categorisation of C (fix for firstI
maintenance release). Target Release set to M1 to reflect the categorisation.
Target Release updated to M1"
Date: 31-Aug-2000 10:46:00 User: Gerald Barnes
“I have now taken over analysing this problem from Steve Warwick who
requested additional information on 13/10/1999 17:49:21. I have looked at
the information and I am not convinced it can possibly be complete.
For example he said he wanted information from 20th May until 26th May. I
opened the spreadsheet mc20may.xls and observed it only had 6 counter 32
transactions in - is this correct?
In any case my team is not used to dealing with data in this form (
spreadsheets ). We have developed techniques to look at problems using a
full message store plus audit and event logs from the failing counter. Even if
all the relevant spreadsheets could be obtained I do not think it would be
worthwhile me getting to grips with this new method of analysing problems.
I see this is a very old problem (21/07/1999) and there have been many
software updates since then. May I suggest we discontinue investigation of
this particular problem but that if a similar problem occurs again you send full
message store plus audit and event logs from the failing counter.”
Date: 31-Aug-2000 13:17:00 User: Deleted User (Mike Croshaw
Sep/00)
“Closing call on basis of insufficient evidence. As this is such an old call I have
not contacted the call originator. I suggest that this call remains closed!”
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 166 of 225
pclbe3227 (2001)252
11.6 Summary: APS and TIP: unequal harvesting.
11.7 Date raised in PEAK: 28 February 2001
11.8 Discrepancy value: £11,708.08
11.9 Time open: 9 Days
Key Excerpts
Date: 28-Feb-2001 10:34:00 User: _Customer Call_
“Could you raise this call for me. APS, PATH039, Priority B The APS Daily Account
Balancing Report (2133) for processing date 27/02/01shows 401 transactions that
were harvested by APS but not by TPS. Please send to EDSC for investigation.”
Date: 28-Feb-2001 12:14:00 User: Garrett Simpson
“We have now found out, the hard way, that there is a bug in the new version of
Riposte installed for M1. The effect of this is to cause it to run very slowly. TIP
harvester runs first and has a finish time of 2030. After that the APS harvester
starts. The slow running meant that when the TIP harvester was stopped there
were still a lot of offices for it to harvest. By the time the APS harvester was
stopped more offices had succeeded in reporting their EOD so the APS harvester
had more offices to harvest than the TIP harvester. The correspondence servers
have now been regressed to the previous version of riposte, so this problem should
not recur.”
Date:01-Mar-2001 15:30:00 User: Angela Shaw
“The APS daily account balancing report (2133) processing date 28/2/01 shows
that the 401 txns for a value of £11708.08 have now been returned to TIP and
have been collected by the TIP harvester. Can you please advise what is being done
to fix the problem in the longer term? Is there a fix / work package? Which team is
progressing the fix? Thanks”
Date:01-Mar-2001 17:08:00 User: Garrett Simpson
257 PC0063227.html, Peak PC0063227, 28 February 2018, [POL-0237798]
[8c84cd6299903d0d3bbe0e05de44b924]
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 182
Number: 1_ Author: Gareth Je
This was due to an upgrade issue.
Date: 26/10/2018 16:47:00
Everything resolved itself the next day and this would have no impact on Branch accounts or back end systems.
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 167 of 225
“The bug in Riposte has been identified. Escher have sent a fixed version which is
under test.”
Date:09-Mar-2001 10:21:00 User: Michael King
“These transactions have now been harvested and this is being fixed under
PC0063158. Please close call.”
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 168 of 225
pc&be3723 (2001)22
11.10 Summary: FAD213422 - Spurious cash figure on trial balance
11.11 Date raised in PEAK: 10 March 2001
11.12 Discrepancy value: £428.29
11.13 Time open: 5 days
Key excerpts:
Date: 10-Mar-2001 09:01:00 User: _Customer Call_
“called by michelle at nbsc & asked to call pm regarding a discrepancy that
had been made good but resulted in a spurious cash figure on trial balance”
“pm had original balance £30358.35 & a shortage of £428.29pm declared
cash to £3078.86 his new balance was £428.29o0ver £428.29 under net
discrepancy zero, but his cash figure had gone up £856.58 to £31643.22
and a message saying continuing could result in an unbalanced stock. I took
the pm back into the cash declaration & it was still the same figure
£30786.64, I printed out the trial balance again this time it was over
£428.29 under £856.58 net discrepancy £- 428.29. cash figure £30358.35.
I went back into cash declaration again & declared cash was still at
£30786.64, I printed out another trial balance this time it was over£428.29
under £428.29 nett zero & the cash figure was ok”
“Info provided for update of KEL DRowe1625K as this PM seems to have got
himself a workaround which may work in the future.”
Date: 15-Mar-2001 11:54:00 User: Sudip Sur
“I have updated KEL DRowe1625K as requested. Please note that this is a
known problem and development are working on this and a fix will be
released when available.
258 PC0063723.html, Peak PCO063723, 10 March 2001, [POL-0238257] [1efaafc05039eea7a5e5e09d1d50226c]
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Date: 26/10/2018 16:48:00
Probably need to refer to reference KEL and associated Peak to understand this one,
yNumber: 2_ Author: Simpkins John Date: 29/10/2018 19:00:00 Z
Tthink Peak PC0062264/ PC0063608 relates,
Date:05-Mar-2001 14:56:00 User.Martin McConnell
Product EPOSS Deliverables DataServer.dll added
F} Response
I believe I have found thr root cause to this and it is similar in nature to 63146. There is a genuine discrepancy to be generated, BUT, I believe, a call to switch off a message port
has been subjected to some form of delay (or failure) within Riposte and the 1st transaction (always cash) slips through and is accumulated on the notify call (from
dlsintemalSession:Callinterface). I have a fix which again generates diagnostics for the messageport txns (just in case my suppositions are
incorrect again as per 57039) but there is also some code put in place to test the return call from DESTROY_MESSAGE PORT and to put in a a slight delay
for catchup purposes to bring the door down on these odd discrepancies being accmulated in twice
Recommend the above are put in Dataserver with a fix already diagnosed,
Fix released 14th April 2001.
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 169 of 225
pc&boss44 (2004)252
11.14 Summary: FAD011642 - Fault with bureau & decimal point
11.15 Date raised in PEAK: 6 February 2004
11.16 Discrepancy value: N/A
11.17 Days open: 193 days
Key Excerpts
Date: 06-Feb-2004 16:36:52 User: _Customer Call_
CALL PC0098844 opened
Date: 06-Feb-2004 16:36:58 User: _Customer Call_
“pm has problems regarding a fault with there bureau, something to do with
a decimal point
Contacted: paul buttler who help with the implimentation of the bureaude
change at the p.o advises that the problem which is arising at the p.o is due
to a decimal point missing on the horizon system on the ‘euro line’ which is
causing a decrepency.
pm declares euros, but does not show on the balance snap shot. pm does
transaction log @ 14.34 on gateway which shows all the euros she has
declared but it does not show up as the sterling equivellent. eg. DDP euro
5786500 = sterling 0.00 NB. this problem is only showing for euro's. All other
currency's are showing ok in the system
When the pm is now trying to reverse the transaction the figure is doubling.
Can ssc please investigate as a matter of urgency why the bureau de change
is not updating on the system for euro's currency on the system.”
25° PC0098844. html, Peak PC0098844, 6 February 2004 [POL-0270879] [050c1d940ddd970bf7ed304afc494faf]
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 185
Number: 1_ Author: Gareth Jenkins Date: 26/10/2018 16:48:00
This sounds like a serious error when Bureau was introduced. Main issue seems to be allow reversals of revaluations which looks like it was fixed separately.
Handling of limits is the remaining issue which it was decided not to fix as it would be obvious if it re-occurred.
Number. 2_Author: Simpkins John Date: 30/10/2018 10:52:00 Z
Reversal of revaluations was fixed under Peak PCO098480 in release S52R in March 2004,
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 170 of 225
Date: 06-Feb-2004 16:41:54 User: Barbara Longley
“Prescan: Assigning call to Martin Harvey in EDSC who has been working onI
call already. Raising call to 'A' priority to reflect urgency.”
Date: 06-Feb-2004 18:18:56 User: Martin Harvey
“The event log shows several error messages indicating that the DBD
discrepency value is too large. It is believed that the problem is with the
balance snapshot only The PM has been advised to not roll over but to
continue transacting normally. Messagestore and event logs to follow.”
Date: 09-Feb-2004 15:41:01 User: Martin Harvey
“Just to clarify, in addition to a software fix for this problem we also need devI
to advise on the remedial action necessary to unpick the actions taken by the
PM and to enable the office to be balanced on the 11th Feb”
Date: 10-Feb-2004 09:01:39 User: Martin Harvey
“The PM has done a trial balance on Stock Unit FC as requested. This shows
a value of £50530.57 as the sterling equivilent of Euro's, which is about right
apparently. Also there is Receipts/Payment mismatch of £79321.46, which is
double the revaluation amount of £39660. 73. Response code to call type L as
Category 4”
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 171 of 225
Date: 10-Feb-2004 11:46:55 User: Martin McConnell
“The receipts payments mismatch is a duplicate of 98480 currently with RMF.
In other words users should NOT be allowed to reverse system generated
revaluations thus. The initial problem with the large amounts I believe the
system has coped OK and has alerted a red event to say so. Perhaps the only
thing the system should hve done based on the current spot rate as time of
declaration would be to refuse to allow such a number to be entered because
it would blow a system limit. This is potentially flawed should a new spot rate
arrive after a declaration but it is probably a more user friendly thing to do.
Routing to management for their onward consideration.”
Date: 10-Feb-2004 13:13:56 User: Mark Wright
“I've applied the fix that Martin provided, and that has cleared the receipts
and payments mismatch, however the PM is still reporting a shortage
of£39660.73 Martin is now looking into this and will advise.”
Date: 10-Feb-2004 16:42:19 User: Martin McConnell
“The clerks/PM@@s should be alerted to the fact that they should NOT under
any circumstances be allowed to reverse (ER) a system generated revaluation
with regard to BDC. Hopefully we@@Il be able to punt out a fix for PCO098480
which has led us down this path in S60(?) which will prevent the likes of this
problem spiralling out of control.”
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 172 of 225
Date: 12-Feb-2004 11:46:13 User: Matt Arris
“In summary This PinICL has exposed two potential problems. 1) Clerks can
reverse an auto-generated Bureau revaluation. This is a duplicate of 98480.
2) Clerks declare foreign currency daily.
The counter software checks to see if the figure declared matches the system
generated figure. If the declared figure does not match the system figure the
outlet has ?lost? some currency and there will be a discrepancy. If this
discrepancy is greater than 1 million pounds an error is produced whereby
the sterling equivalent is set to zero, but the declared foreign currency figure
is retained. When entering the daily foreign currency declaration there may
be the odd small discrepancy, but there will not be one of 1 million pounds
worth unless there was a robbery ? but a million pounds of a particular foreign
currency in stock is too high anyway.
The clerk produced this problem because two many zeros were added to the
generated figure would have been zero for every currency. Under normal
operation a discrepancy would make the clerk double check the entered figure
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 173 of 225
or explain the difference in stock. To prevent this second problem from arising
by changing code we would have to check each declared foreign currency
when entered to decide if the sterling equivalent of any discrepancy would
exceed 1 million pounds. This would require a sizeable rework with associated
changes to dialogues, etc.
Development would rather not have to make this change. I think this second
problem can be covered by a KEL with no code change made. Small
discrepancies will occur and will need checking and explaining. Post migration,
large discrepancies should not happen. Discrepancies of more than a million
pounds worth should never happen.
If we stop the ability of reversing an auto generated Bureau revaluation
(98480) no further damage can be done by making incorrect declarations. By
just making a correct declaration and performing a trial balance (with no
commit) the error will be resolved. It may be worth passing this PinICL to
John Pope in Requirements for him to find out if the customer is willing to pay
for a code fix for item 2) or whether PO is prepared to accept that item 2) is
unlikely to happen and there is an easy way to put things right if it does
happen (re-declare and perform a trial balance). declaration. £30,000 worth
of Euros was turned into £3,000,000.
Bureau sites were being migrated from the existing MoneyChanger system to!
Horizon. The instructions were to declare all MoneyChanger stock on the new
Horizon Bureau system. Effectively all transactions were discrepancies as the
system.”
Date: 13-Feb-2004 10:07:56 User: Martin Harvey
“OCR MWright420K was raised to cover the correctice actions. Kels
AChambers5533Q and GMaxwell3841K cover both problems. As
recommended please pass call to John Pope in Requirements for him to find
out if the customer is willing to pay for a code fix”
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 174 of 225
Date: 044May-2004 10:34:39 User: _Customer Call_
“Caller states that the rates on the rate board are not being updated.
04/05/04 10:33 uk951563 HSH2 Information: PM is sure she has not logged,
or is not aware there is a call logged for this fault. This call is a PATH call,
therefore the PM may not be aware. Also, there has been no update since
March 23rd. 04/05/04 10:34 uk951563 HSH2 Information: Raising new call
for FAD FAD, and to reflect callers new complaint.”
Date: 21-Jun-2004 15:28:39 User: Lionel Higman
“The call Target Release has been changed from:- BI_3S50R-Provisional The
call Target Release is now:- BI_3S70R-Provisional S5OR is no longer a
possible target release. S60R is already fully subscribed. Targeting at next
available potential R release - S70R."
Date: 14-Jul-2004 16:20:08 User: Lionel Higman
“The call TargetRelease has been changed from:- BI_3S70R-Provisional The}
call TargetRelease is now:-
BI_3S75R-Provisional I am told the currently targeted release for this call
(70R) exists only in name - retargeting at S75R-Provisional.”
Date: 29-Jul-2004 17:09:23 User: David Cooke
“In the absence of John Pope I have reviewed this entry and agree with the
assertion that this can be handled via KEL with no code change. The
circumstances were unusual and should not occur in normal live operation.
Unless the help desk advises we have had a large number of calls relating to
the KELs I suggest we close this call.”
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 190
Number: 1_ Author: Gareth Jenkins Date: 26/10/2018 16:50:00
This bit seems totally unrelated to the rest of the Peak.
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 175 of 225
Date: 17-Aug-2004 09:14:41 User: Barbara Longley
“Prescan: As Martin Harvey is out of the office this week, I am reassigning call
to Anne Chambers who is familiar with this problem.”
Date: 17-Aug-2004 10:37:23 User: Anne Chambers
Call had been passed to John Pope to see if there was a POL requirement forI
change here - after some months the following response was received from!
David Cooke: In the absence of John Pope I have reviewed this entry and)
agree with the assertion that this can be handled via KEL with no code change.
The circumstances were unusual and should not occur in normal live}
operation. Unless the help desk advises we have had a large number of calls
relating to the KELs I suggest we close this call. However in the interim, this)
iproblem has been seen on a couple of other occasions and has caused
significant accounting problems. PC0103606 is with development, fix
Ischeduled for S80. This call can be closed (please do not contact original
outlet).”
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 176 of 225
pcbbo3131 (2010) 26°
11.18 Summary: FAD203306 office snapshot is showing incorrect volume
11.19 Date raised in PEAK: 18 August 2010
11.20 Discrepancy value: N/A
11.21 Days open: 24 Days
Key Excerpts
Date: 18-Aug-2010 11:01:23 User: _Customer Call_
“pm states office snapshot is showing incorrect volume”
“pm states volume for lottery was £284 but value on the office snapshot is
£604 Rem in session ID: 1-362075 - items in question are the first two from
this session ID"
“The volume of £1 lottery tickets was set at £284 but the value on the
snapshot under receipts table is £604 Smiley Desktop could not find any
particular issues No relevant SSC KELs could be found Counter can be
pinged.”
“PEAK, can you investigate discrepancy between volume and value in office
snapshot for £1 Lottery tickets?”
Date: 21-Aug-2010 09:38:49 User: Sudip Sur
“I have looked at the tally roll printout of the Office snapshot that was carriedI
out on 13/8/10 @15:35. The print out showing the volume of Instants £1 =
260 PC0203131.html, Peak PCO203131, 18 August 2010 [POL-0372925]
[6ea70fc9c6b34cbcd61b2a7b2ddb2628]
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 192
Number: 1 Author: Gareth Jenkins Date: 26/10/2018 16:51:00
This looks like an issue when migrating a Branch from Horizon to HNG-X and affects the reports produced by the process.
It has no impact on the Branch accounts and just some confusion for the Branch.
AAs the bug was on the Horizon processing and this was one of the last migrations there was no point in fixing it,
Number: 2_Author: Simpkins John_Date: 30/10/2018 11:14:00 Z
Agreed
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 177 of 225
284 and the Value = 604.00. Clearly this is wrong and the system is failing to
calculate correctly.
I have done a query on the database for session data for product:56 during
1st Aug and 20th Aug and found a txn that was done on 4th Aug @7:48:57
(Mode:SW) for quantity = -320. This may have contributed to the incorrect
value of 604.00.”
Date: 23-Aug-2010 08:56:07 User: Gareth Jenkins
“I have tried this out on Horizon and I accept that on Horizon the Volume
relating to a TC is NOT included in the Receipts and Payments.
I believe that this is a bug on Horizon and that it should be included. We had
a similar issue with Discrepancies on the Pre and Post Migration report and
this was accepted as being due to a bug on Horizon rather than on HNG-X.
I believe that this PEAK is another symptom of the same issue. I think thatI
this should be closed as "Advice after investigation" (since "No fault" fives
SSC a Black Mark!).”
Date: 11-Sep-2010 09:43:54 User: Sudip Sur
“Development have investigated the problem and believe that the feature hasI
been carried forward from old Horizon and not a new problem in HNG-X.”
pckho3676 (2010)26
11.22 Summary: Branch 054106 (HNGx) - NB102 Section 5 LINK - State 4
11,23 Date raised in PEAK: 31 August 2010
11.24 Discrepancy value: £71
11.25
261 PCO203676.html, PEAK PCO203676, 31 August 2010 [POL-0373467]
[9c65a8ele33e636bc6ae372aefed3690]
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 193
Number: 1_ Author: Gareth Jenkins Date: 26/10/2018 16:52:00
This isa failed recovery which was processed correctly
We can't tell from this if POL did issue a TC to the Branch to reimburse the SPMR, but they should have done so.
Number 2 Author: Simpkins John Date: 30/10/2018 11:14:00 Z
Agreed
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 178 of 225
Days open: 1 Day
Key Excerpts
Date: 31-Aug-2010 10:33:40 User: Jay Crofton
“Branch 054106 (HNGx) - NB102 Section 5 LINK - State 4 Call Type: L Call
Priority: A"
“NB102 Section 5 report for Client: LINK produced on 30/08/2010. Branch
shows one new exception with Txn Id: 00-54106-2-3633070-1 Receipt date
27/08/2010, Amount: -£71.00
This branch also appears on the Failed Recovery Report ? why? KEL acha959T
or KEL dsed2640M may be relevant. Relevant reports attached. Sending to
SSC for investigation.
**PLEASE include further txn attempts with the same PAN immediately after
this txn if applicable as this is useful for Post Office Ltd for settlement
issues**”
Date: 31-Aug-2010 15:51:58 User: Clive Turrell
“The customer requested a withdrawal which was authorised by the FI and a
receipt was successfully printed at the counter so it is very likely that cash
was handed over. However the T1 recovery request timed out at the counter
and was abandoned. No subsequent transactions were attempted on the
same PAN. In these circumstances the following applies from KEL acha959T
"If the receipts were printed successfully, and no subsequent transaction was
done for the same PAN at the branch (check on TESQA) it is likely that the
transaction was completed. Customer's account will be correct but the branch
will have a shortage (for a withdrawal) or surplus (for a deposit) because the
session hasn't been recorded."”
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 179 of 225
Date: 31-Aug-2010 16:08:24 User: Clive Turrell
“The Call record has been transferred to the team: MSU-Indt Mgt”
Date: 31-Aug-2010 16:34:36 User: Andrew Nash
“(Start of Response] Closing call. [End of Response] Response code to call
type L as Category 67 -- Final -- Solicited Known Error Routing to Call Logger
following Final Progress update. Defect cause updated to 42 -- Gen - Outside
Program Control”
pcbbhe3451 (2017282)
11.26 Summary: Branch 66013 - Failed Recovery Report - 19/10/2017
11.27 Date raised in PEAK: 19 October 2017
11.28 Discrepancy value: £20
11.29 Days open: 1
Key Excerpts
262 PCO263451.html, Peak PC0263451, 19 October 2017, [POL-0430967]
[981b785aa6f78e75a5f19c8a2c70aff5]
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 195
Number: 1_ Author: Gareth Jer
Another failed recovery as above
Date: 26/10/2018 16:53:00
ge Number: 2_Author: Simpkins John_Date: 30/10/2018 11:16:00 Z
Agreed, standard process
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 180 of 225
Date: 19-Oct-2017 08:25:20 User: Dharmesh Mistry
“Branch shows one failed recovery transaction with Txn Id: 00-66013-I
15621320-1 for Receipt Date: 16/10/2017.
This exception has not appeared on any of the reports and following a search
on the DRS did not bring back any results. Raising a call to investigate why
this has appeared as an exception, the root cause and the potential business
impact for POL. Relevant reports attached. KEL cardc464Q and KEL
acha2511S (seng2048K for DCS) may be relevant. For charging purposes,
please could SSC explain whether or not this issue is Fujitsu related (at fault)
i.e. hardware or not? Relevant reports attached.”
Date: 19-Oct-2017 09:39:42 User: Sunil Nellikkentavita
“The £20 cash withdrawal transaction was authorised by the FI and an
AUTHORISED receipt was produced on the counter. However, when the user
attempted to settle the transaction it failed due to the comms issue at the
time so disconnected session receipts were produced and the user was logged
off. The user managed to log back in but recovery also failed because of same
comms issue. As an AUTHORISED receipt was produced the user should have
handed money over to the customer but we cannot be certain that they
actually did so. Assuming money was handed over, the customer account will
be correct but the branch will
have a shortage given that the transaction hasn't been recorded on the
system. This will need to be manually reconciled”
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 181 of 225
Date: 19-Oct-2017 09:51:25 User: Sunil Nellikkentavita
“Failed recovery session cleared from BRDB under MSC_ Task
043T0095737.MSC updated with following details: PCO263451 - Failed
recovery tx no:00-66013-1-5621320-1 cleared from BRDB. Actioned by Sunil
and witnessed by Dave Seddon.”
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 182 of 225
pcbb66s75 (2018)
11.30 Summary: Branch 116940 - NB102 Section 5 A&L ? State 4
11.31 Date raised in PEAK: 26 January 2018
11.32 Discrepancy value: £125.00
11.33 Days open: 6 days
Key excerpts:
Date: 26-Jan-2018 08:52:35 User: Dharmesh Mistry
“NB102 Section 5 report for Client: A&L produced on 26/01/2018. Branch
shows one exception with Txn Id: 00-116940-1-5311-1 Receipt Date:
24/01/2018, Amount: £125.00 KEL acha959T may be relevant. Relevant
report attached. Sending to SSC for investigation. For charging purposes,
please could SSC explain whether or not this issue is Fujitsu related (at
fault) i.e. hardware or not?”
Date:31-Jan-2018 11:11:47 User: Venkata Subbarao Konakalla
“The cash withdrawal transaction was authorised by the FI and an
AUTHORISED receipt was produced on the counter. money should have
changed hands. However, when the user attempted to settle the transaction
it failed due to comms issues. The user logged back today after one week
and the recovery also failed. The customer's account will be correct but the
branch will have a shortage because the session hasn't been recorded. This
requires manual reconciliation. Please raise an MSC to clear the failed
recovery transaction.”
Date:09-Feb-2018 09:45:06 User: Dharmesh Mistry
“Closing peak as MSC completed and Failed recovery cleared”
263 PCO266575.html, PEAK PC0266575, 26 January 2018, [POL-0433904]
[8d02a629313fa13f86f853d49e5Sdcc7]
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 198
Number: 1_ Author: Gareth Jenkins Date: 26/10/2018 16:53:00
Yet another failed recovery as above
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 183 of 225
Pi 73046 (2018)284
11.34 Summary: Branch 051106 - NB102 Section 5 LINK - State 4 and Failed
Recovery
11.35 Date raised in PEAK: 15 August 2018
11.36 Discrepancy value: £25
11.37 Days open: 1
Key excerpts:
Date: 15-Aug-2018 08:15:59 User: Andy Dunks
“PEAK raised for the investigation of transaction(s) in a state other than
Final as showing in daily Reconciliation reports. To comply with SLA, PEAK
will be open for a between 8 hours and 5 days maximum whilst transaction
issue is investigated, reported and mitigating actions completed and closed
down.”
“This branch also appears on the Failed Recovery Report. Could SSC please
investigate why? KEL acha959T (surs136M for DCS) may be relevant.”
Date: 15-Aug-2018 11:30:17 User: RCAClient Live
PEAK [ PC0273046 ] Branch ID [ 051106 ] Node ID [ 02 ] SSN [ Iprpssn003
] User [ snell02 ] Attempting command execution: get
/cygdrive/c/ProgramData/Fujitsu/CounterBusinessApplication/log/PostOffice
Counter.log.2018-08-13.zip
evidence/051106/02_PostOfficeCounter.log.2018-08-13.zip
264 PC0273046.html, PEAK PCO273046, 15 August 2018, [POL-0439981]
[c4330f4fcc4ea9f37368e4c6927 30828]
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 199
Number. 1_Author: Gareth Jer
And another failed recovery
ge Number: 2_ Author: Simpkins John_Date: 30/10/2018 11:18:00 Z
Yes, standard process, standard reporting mechanisms to detect.
Date: 26/10/2018 16:54:00
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 184 of 225
Date: 15-Aug-2018 11:30:36 User: RCAClient Live
PEAK [ PC0273046 ] Branch ID [ 051106 ] Node ID [ 02 ] SSN [ Iprpssn003
] User [ snell02 ] Command execution completed successfully: get
/cygdrive/c/ProgramData/Fujitsu/CounterBusinessApplication/log/PostOffice
Counter.log.2018-08-13.zip
evidence/051106/02_PostOfficeCounter.log.2018-08-13.zip
Date: 15-Aug-2018 11:45:35 User: Sunil Nellikkentavita
“During the 25 RBS Grp Cash Withdrawal, the banking transaction had fully
completed, including the receipt print but disconnected session at settlement
(Comms issue).
Recovery also failed because of same comms issue.
As per KEL, transactionState=1 the banking transaction had fully completed,
including the receipt print, and money should have changed hands. If
recovery had succeeded, it would have automatically completed the session.
So you can assume that the transaction was completed successfully. The
customer's account will be correct but the branch will have a shortage for 25
withdrawal because the session hasn't been recorded.”
Date: 15-Aug-2018 11:49:50 User: Sunil Nellikkentavita
“Cleared failed recovery transaction from BRDB under MSC Task
04370098231. Actioned by me and witnessed by Venkat.”
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 185 of 225
12. Appendix B
Horizon architecture Diagrams
Figure 1 Horizon Data Flows Overview
Other
‘Systems
(I y,
J bait
“Extract
Data ~
Extract
Audit] Store
Audit System
Audit I Retrieval
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 201
Number: 1 Author: Gareth Jenkins Date: 26/10/2018 16:54:00
This is from my WS and only covers the parts related to Auditing
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 186 of 225
Figure 2 pak Centre Applications Overview?
Centre Bopicstions
265 RMARCO02_0.1.doc, Horizon Next Generation - Plan X (HNG-X), 21 September 2005 [POL-0084540]
[754315a4037a6ea4clec7ee070b7d170)
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 202
Number: 1_ Author: Gareth Jenkins Date: 26/10/2018 16:55:00
This is from a very old document and not necessarily accurate. Specifically Quantum was removed from HNG-X
az;Number: 2_ Author: Simpkins John Date: 30/10/2018 11:20:00 Z
Yes itis missing a lot of components including but not limited to:
Track & Trace
GwWs
HBS
PHS
CHS
SMS
cDP
BBND
BKAC
EMDB
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 187 of 225
Figure 3 HoKzon / Horizon Online migration overview26
Horizon and HNG-X System Overview
266 ARCAPPARCO002_0.2.doc, HNG-X Integration Architecture, 08 November 2006 [POL-0087918]
[daecOde8aSeee25b5c9d067 30c338dd0)
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 203
Number: 1_ Author: Gareth Jenkins Date: 26/10/2018 16:56:00
Again from a very old document so not necessarily accurate
ge Number: 2_ Author: Simpkins John_Date: 30/10/2018 11:27:00 Z
Agreed
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 188 of 225
Figure 4 HoFizon Online Message Flows overview?”
Other
Systenis
DataI Copy
I Oracle Audit
“Write “write
Audit Istore
BAL Message
I
} Audit [Retr ieval
Extract
2°? Witness Statement of Gareth Jenkins.pdf, Witness Statement of Gareth Jenkins, 05 October 2012 [C0003632]
[b544230cf07249c189cf664fcba6d899]
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 204
Number: 1_ Author: Gareth Jenkins Date: 26/10/2018 16:56:00
Again from my WS and so only relates to the parts relevant to auditing messages
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 189 of 225
Figure 5 Hokizon Business Applications overview 201076
268 POL-0003093.pdf, Horizon Architecture Diagrams, 8 March 2010 [POL-0003093]
[00195d1cbb0017bd34e7c68b7930c1cf]
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 205
Number: 1_ Author: Gareth Jenkins Date: 26/10/2018 16:57:00
Not sure where this came from but seems OK
FUJ00183797
FUJ00183797
180503R1935
16 October 2018
Page 190 of 225
e 6 Reconciliation in Horizon Online?°?
ieancial institutions
tt
R et
Bata
Cente
— L
RA Co
Ly]
vn ACE i
On-Line Service
Rouh & Branch pC
Reconciliation In HNG-X
‘Chents
28° SVMSDMPROO012_3.doc, Reconciliation and Incident Management Joint Working Document, 30 April 2012 [POL-
0125134] [ad897ac9ffSedb2de37bfb8c4e9dc362)
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 206
Number: 1_ Author: Simpkins John Date: 30/10/2018 11:33:00 Z
Tooks reasonable.
Does not break down the R/A/C into stages R1,R3,A1,A3 etc.
Is missing the T1/T2 checks and the Auth Agent CO is called £1 and the Fl replies with an E2.
FUJ00183797
FUJ00183797
180503R1935
16 October 2018
Page 191 of 225
Figure 7 Bakking Interfaces 200377°
279 CSOLA045_1.1.doc, Operational Level Change and Release Management Interface agreement Between Post
Office Ltd and Fujitsu Services, 24 April 2003 [POL-0069985] [334ef7724ebe8012129c96f62b41ad15]
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 207
Number: 1_ Author: Gareth Jer
Yes OK at the time
Date: 26/10/2018 16:57:00
However NBE removed in 2005 (and TES introduced) and Streamline replaced by GlobalPay around 2012
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 192 of 225
Figure 8 External Clients / Systems
Horizon Service Context Diagram
Horizon Business
Server
Settlement/Logistics/APOP.
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
180503R1935 16 October 2018
Page 193 of 225
Appendix
13. Cc
2ofl2 - 2013 TC's
RowLabels
{blank}
Unpaid Cheques
Suspense
Stack -Non Rem
Santander Online Banking
Santander - Manual withdrawal
Santander - Manual Deposit
Santander - Green Giro
Santander «Co-Op Business Eneashments
PreOrder
Postal Orders
Personal Banking
Paystation
Other {Branch}
‘Online Banking
NSéei
Government Services
First Rate
DVLA
Drop & Go
DebitCards
Cheques To 1PSt
Cash Rems From Branch
Camelot
Bureau
Automated Payments
[am
Grand Total
+Countoftype Count offteference Sum off Value
538
4671
235
1458
602
5905
887
1321
427
1046
327
53
8B
705
826
2048
128
3744
524
140
4722
25649
22567
3939
2080
38
84247
238
4667
234
452
602
5905
886
1322
126
1046
323
52
86
704
826
1039
122
3743
Sia
439
A675
25649
22387
3939
2040
345
249525.5
372763.46
3824.93
311297609.87
292847.23
896625.78
12094.87
$2285.91
938.28
15639.96
39938
2778
7878.24
1872869.28
3A07E3.35
A22795.24
$6392.47
223352.55
27724.24
774759.83
1585269.95
§120594.63
448789.34
432165.75
B3S20LL34
97230
BAGSAIBOS
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 209
Number: 1_ Author: Simpkins John Date: 30/10/2018 11:56:00 Z
‘What does the signs represent on the value (stock into the system is held as a negative value).
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 194 of 225
Appendix
14. D
Log! of helpdesk calls regarding Transaction Corrections W/E 15 June
2014
WE 15/06) 2014
Adiay Sub Aa
= Transaction Corrections: Dispute
Gunerai Inormaban
Missing Evidence
Passiie Solutions
3
I Banking Canestions ?
Resolving Branch Discrepancies 7 7
Transaction Comection Dispute EN 8
Transaction Coreclons Talal I 148 445,
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 210
Number: 1 Author: Simpkins John Date: 30/10/2018 12:16:00 Z
So this is a weeks worth of TCs?
They need to do these counter over a larger range to be meaningful
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 195 of 225
Appendix
15. E
Horizon Reports
TP! eports
15.1 TPS Reports were set out in the “TPS Report Set” which was designed to
enable the reconciliation of transactions that were carried out in
branches using the branch infrastructure. It was sent to the POLSAP
system and the POLMISs (Credence), and was made up of the following
reports:
15.2 TP&250: Host Detected Transaction Control Errors - Daily report which
shows detail for any Post Office branch where the control totals for the
transactions output by the Host to POLSAP and Credence do not match
the Daily Transaction Totals calculated by the counters.
15.3 TP&254: Harvester Exceptions - Daily Report which shows a list of
exceptions detected by the BRDB copy process when failing to process
one or more messages.
15.4 Te&257: POLSAP Incomplete Summaries Report - Daily report which
identifies all Post Office branches in which the net total transactions
(debits/credits) does not equal 0.
15.5 The TPS Report Set contained information which would allow Post Office to
identify the errors or inconsistencies set out above, which could include
shortfalls. Holidver, the investigation of the cause and resolution of TPS
exceptions and errors was not dealt with within the TPS Report Set.
Resolution was dealt with through “Business Incidents” and the
underlying cause was investigated via “System Incidents”, both of which
are part of the Business Incident Management (BIM) procedure. This is
considered in more detail below.
15.6 There are 2 additional points to note in relation to the TPS Report Set
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 211
Number: 1_ Author: Gareth Jenkins Date: 26/10/2018 16:59:00
“Can we say anything about the frequency of such reports being populated?
These were introduced around 2000 following reconciliation issues during the pilot
«Number: 2_Author: Simpkins John_Date: 30/10/2018 14:08:00 Z
May be worth asking SECOPS this as they would raise the tickets. Its not very simple from the Peak text.
Number: 3 Author: Gareth Jenkins Date: 26/10/2018 17:00:00
“Has this ever been populated?
Number 4 Author: Simpkins John Date: 30/10/2018 13:15:00 Z
Yes, a Peak search shows that this is stil used even today, some examples
Pco270441
This is another instance of the "null currency exchange rate” issue described in KEL GCSimpson5834N
PCO267286
The issue has been caused by Branch 232201 performing a ‘Declaration Discrepancy - Pos’ (DDP) on the redundant currency LKR ‘Sri Lanka Rupee’
PC0258390
Cause was use of a currency ("LKR™
ri Lankan Rupees) for which we receive incomplete details from First Rate Exchange Services
Number. 5 Author: Gareth Jenkins Date: 26/10/2018 17:00:00
“ Tknow this was common in old Horizon. How often is it produced with HNG-X?
«Number. 6 Author: Simpkins John _ Date: 30/10/2018 14:00:00 Z
As above, still in use.
Number. 7 Author: Gareth Jenkins Date: 26/10/2018 17:00:00
‘Again should be very rare. Do we know how often it has been produced?
«. Number: 8 Author: Simpkins John_Date: 30/10/2018 14:01:00 Z
Still in use, some examples are:
Pc0274577
Exceptions were caused by missing POL FS Mappings
Pco274545
Same cause as above
PC0273160
Product 44425 was transacted during a reversal transaction on node 3 in FAD 231832 on the Sth July at 11:17 during session 321596
However, due to ATOS In Formation Services not providing POL FS Mappings Ref Data, we are unable to summarize this transaction, and hence teh sub-file is currently held
~:Number. 9 Author: Gareth Jenkins Date: 26/10/2018 17:01:00
Not sure what this means? My understanding is that should the reports identify anything, then @ Peak was ralsed for SOC to identify and fh with BiMs being raised if required,
Number: 10 Author: Simpkins John_Date: 30/10/2018 14:06:00 Z
Agreed, not sure how the cause should have been resolved within the reports
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 196 of 225
Appendix
15.7 There was no
and the Credence transaction stream, so Credence could not be used to
mal reconciliation process between the POLSAP system
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 212
Number: 1_ Author: Gareth Jenkins Date: 26/10/2018 17:02:00
True
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 197 of 225
verify financial integrity. pobbap system transaction information should
have been used for this purpose.
15.8 The TPS Report set was not intended to be shared with Post Office, but
it was available upon request.
APS Reports
15.9 APS reports were set out in the APS Report Set which was designed to
ensure APS Transactions completed in Branches are reconciled with the
Transaction stream received by the POLSAP System to enable
settlement to be made with Clients. The APS Report set was made up
of the following reports
15.10 ap§2133: APS Daily Account Balancing Report - The objective of this
report is to confirm that the APS transaction account balances for the
processing day.
15.11 APSS2133b: APS Client Summary Report - Objective is to provide a
summary of the transactions which have been delivered by APS during
the processing day. The summary is produced by the clearing agent (i.e.
the organisation to which Fujitsu Services deliver the transactions). For
each clearing agent, a breakdown is provided by Client account for each
transaction date. Transactions delivered to Manual are processed
manually and consequently are not reported here.
15.12 APSS2133c: APS Delayed Transactions Report - Objective is to provide
details of all transactions which have not been delivered by Fujitsu
Services because they have been delayed/quarantined within the APS
Host system. Initial Customer Support resolution will cause the
transactions to be returned for normal processing or sent to manual for
manual processing. TPS quarantined APS transactions will show up as a
discrepancy on the APS Daily Account Balancing Report.
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 213
Number: 1_ Author: Gareth Jer Date: 26/10/2018 17:02:00
“BOL SAP could not be used as it doesn't contain transactional info, That is why Credence is used,
Number: 2_ Author: Gareth Jenkins Date: 26/10/2018 17:03:00
Tan we say anything about the frequency of these?
I had little visibility of these, but I suspect they were very rare and mainly resulted from timing issues
Number: 3 Author: Simpkins John_Date: 30/10/2018 14:09:00 Z
“Again worth asking SECOPS for the frequency.
They are still in use today.
Number 4 Author: Simpkins John Date: 30/10/2018 14:10:00 Z
Some but not many.
oPC0258542
This product is LIVE (product 42094 -BOS Cash Deposit), hence the APS errors see calls PCO258530, PCO258531, PCO258533, PCO258534, PCO258532 relating to apss2133 error
generated,
ATOS have approved the release of this new transaction without instructing Fujitsu to activate the agreement, this has therefore cause these errors
PCO206437
KEL obenge3348L applies
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 198 of 225
15.13 APSS2136: Daily TPS / APS Transaction Reconciliation Summary Report
- Normally, APS transactions flow through the TPS Host to POLSAP and
through the APS Host system within the same working day. However,
the rules associated with the processing of APS transactions within the
TPS Host and the APS Host systems are different. Consequently,
transactions may be placed in exception status by APS but be accepted
as valid by TPS. This reconciliation point is at the end of the processing
day. It reconciles the APS transactions at Branches and APS transactions
delivered by TPS to POLSAP. It maintains a record of which transactions
have passed through the TPS side and which transactions have passed
through the APS side and on a daily basis it reports transactions which
have been processed by one side and not the other.
15.14 APSS2136b: Daily TPS / APS Transaction Reconciliation Client Account
Exception Report - Objective is to identify Client account exceptions
when comparing the actual number and value of transactions that were
processed by TPS (TIP) and APS (Client) for the last 30 days.
15.15 APSS2136c: Daily TPS / APS Transaction Reconciliation Detail Exception
Report - The objective is to report all transactions for the last 30 days
which are different in POLSAP to Client delivery. Differences may occur
either as a result of some error condition, or in the case of unmatched
TPS/POLSAP transactions, as a result of business processing rules being
different from those of APS/Client transactions.
15.16 APSS2139: Daily APS Office Harvesting Report - The APS Harvester
Reconciliation was used to ensure all Horizon Branches had been
harvested. As Horizon Online does not rely on harvesting of transaction
this report is no longer relevant.
15.17 APSS2140: APS Harvester Transaction Totals Summary - Objective is to
provide Branch totals and harvested totals by transaction date for each
of the last 30 transaction days for the APS Harvester.
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 199 of 225
15.18 APSS2140b: APS Harvester Transaction by Office - Objective is to provide
details by Branch code of any discrepancies that exists in the overall
totals shown in the APS Harvester Transaction Totals Summary.
15.19 APS2134: APS Validation Status Report - Objective is to identify the
success or failure of the APS Validation process. This compares the
volume of normal transactions and the value of all transactions between
each Client Transmission file and the corresponding fields in the CTS
sub-file.
15.20 Rejected Sub-Files Report - The Times Rejected column will indicate how
many times that the sub-file has been rejected and will indicate whether
there is an ongoing problem with poor quality data in corrected files. If
the quality of the data in the external transaction file is good then we
would not expect any output from the report.
15.21 Similar to the TPS Report Set, the APS Report Set would allow Post Office
to identify the errors or inconsistencies set out above, which could
include shortfalls. However, the investigation of the cause and
resolution of APS exceptions and errors was not dealt with within the
APS Report Set. Resolution was dealt with through the BIM procedure
which is set out in more detail below.
Banking & Related Services Reconciliation (DRS Report Set)
15.22 The Banking & Related Services Report Set was designed to enable
Santander, CAPO, LINK, Global Payments Inc., and Epay transactions
completed in the Branches to be reconciled in order to allow settlement
to be made with Clients, or direct settlement to specific Clients and/or
Banks. It was made up of the following reports:
15.23 NBOOO: DRS Summary - This report summarises all reconciliation reports
Produced by the DRS. It also summarises all reports that were not
produced by the DRS because there was no data to report.
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 200 of 225
15,24 NB101: Network Banking Settlement Statement - This report identifies
‘C4’ transactions received against each ‘C4 Settlement Date’ as reported
to the DRS for the most recent processing date. The report will be used
by Post Office Ltd. as a basis for settlement of Network Banking
transactions with the Financial Institutions (FIs).
15.25 NBelo2: Exception Summary - This report identifies all incomplete or
exception states, and is divided into 12 sections:
a. Se@ion 1: All Uncleared Confirmed, Unconfirmed & POLSAP
exceptions
b. SeGlion 2: Uncleared Exceptioned Client Transactions
c. Section 3: Uncleared Corruptions
d. Section 4: Uncleared Timing Differences
e. Seblion 5: Uncleared Confirmed, Unconfirmed & POLSAP
exceptions >24 hours
f. Section 6: Uncleared Future Dated Transactions by Client
g. selion 7: All Cleared Confirmed, Unconfirmed & POLSAP
exceptions
h. Sedlion 8: Cleared Exceptioned Client Transactions
i. Section 9: Cleared Corruptions
j. Section 10: Cleared Timing Differences
k. se@ion 11: Cleared Confirmed, Unconfirmed & POLSAP
exceptions
>24 hours
I Section 12: Cleared Future Dated Transactions by Client.
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
Page: 216
FUJ00183797
FUJ00183797
“These were normally empty and I think some reports were never generated,
Number: 1_ Author: Gareth Jer Date: 26/10/2018 17:04:00
Can we be more specific?
. Number: 2_Author: Simpkins John_Date: 30/10/2018 15:37:00 Z
These are used by the Reconciliation team on a daily basis to raise tickets for investigation,
I did not find any recent examples for sections 3,4,6,9,10,12 but it may be best to check with SecOps that these are never produced
Number. 3 Author: Simpkins John Date: 30/10/2018 15:54:00 Z
Yesterdays file was NBS201810290000024307NBTO207ALLGBP.1XT
Number. 4 Author: Simpkins John Date: 30/10/2018 16:01:00 Z
~ Yesterdays file was NBS201810290000024448NB10202ALLGBP.1XT
Number. 5 Author: Simpkins John Date: 30/10/2018 15:55:00 Z
Yesterdays was: NBS201810290000024307NB10205ALLGBP.1XT
Number: 6 Author: Simpkins John Date: 30/10/2018 15:56:00 Z
Yesterdays file was NBS20181029000012741 3NBIO207ALLGBP.1XT
«Number: 7_Author: Simpkins John_Date: 30/10/2018 15:58:00 Z
Yesterdays file was NBS201810290000024448NB10208ALLGBP.1XT
Number: 8 Author: Simpkins John_Date: 30/10/2018 15:59:00 Z
Yesterdays file was NB5201810290000024448NB102 11ALLGBP.1XT
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 201 of 225
Reconciliation and Incident Management
15.26 The BIM system was designed to report progress on the resolution of
Business Incidents to allow Post Office to complete reconciliation or
settlement with its internal systems, clients and banks.
15.27 A “Business Incident” is defined as the symptom of an underlying cause
- e.g. the effect of the system fault on the resulting reconciliation or
settlement information sent to Post Office. A Business Incident can
relate to any of the exceptions from the various reports above or one
or more of the reconciliation or settlement Errors discovered by Post
Office.
15.28 Post Office have reported the following numbers of BIs raised each year:
01/07/2017 - 30/06/2018: 20% BIMS
01/07/2016 - 30/06/2017: 1208 BIMS
01/07/2015 - 30/06/2016: 1773 BIMS
01/07/2014 - 30/06/2015: 1020 BIMS
01/07/2013 - 30/06/2014: 1130 BIMS
01/07/2012 - 30/06/2013: 875 BIMS
01/07/2011 - 30/06/2012: 1013 BIMS
01/07/2010 - 30/06/2011: 1413 BIMS
01/07/2009 - 30/06/2010: 3264 BIMS
01/01/2009 - 30/06/2009: 335 BIMS
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 217
Number: 1_ Author: Gareth Jenkins Date: 26/10/2018 17:05:00
Why is this so high? That is rather worrying
az;Number: 2_ Author: Simpkins John Date: 30/10/2018 16:10:00 Z
‘Agreed they need a break down of the nature of these. limagine the Reconciliation team could do that.
I imagine that failed recoveries are a large part of these,
,Number: 3 Author: Gareth Jenkins Date: 26/10/2018 17:05:00
This is due to issues during HNG-X Rollout
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 202 of 225
Number of Bis raised each year
3800
3000
3
Axis Tithe
1000
2008208200 LL: LZ UOES US 2K EHTam2ND
Year
15.29 From the above BIM’s the following whe identified:
While POL-0032901.pdf relates to Horizon this report is
still run in HNG-X and the numbers below are from the
records Fujitsu now hold:
01/07/2017 - 30/06/2018: 162 APS Reconciliation
Error BIMS
01/07/2016 - 30/06/2017: 125 APS Reconciliation
Error BIMS
01/07/2015 - 30/06/2016: 104 APS Reconciliation
Error BIMS
01/07/2014 - 30/06/2015: 39 APS Reconciliation Error
BIMS
01/07/2013 - 30/06/2014: 37 APS Reconciliation
Error BIMS
01/07/2012 - 30/06/2013: 42 APS Reconciliation
Error BIMS
01/07/2011 - 30/06/2012: 33 APS Reconciliation
Error BIMS
01/07/2010 - 30/06/2011: 69 APS Reconciliation
Error BIMS
01/07/2009 - 30/06/2010: 76 APS Reconciliation
Error BIMS
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 218
Number: 1_ Author: Gareth Jenkins Date: 26/10/2018 17:05:00
were
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 203 of 225
01/01/2009 - 30/06/2009: 15 APS Reconciliation Error
BIMS.
15.30 A “System Incident” is defined as the underlying cause of a Business
Incident and is created to track the root cause of the same.
15.31 Depending on the impact, nature and scope of the Business Incident the
POA Problem Management Procedure may be invoked. However, if the
nature of the Business Incident is agreed to be low priority or a one-off,
the BIM reconciliation procedure would suffice. The choice of procedure
is determined by discussion and agreement between Post Office and the
Reconciliation Service for specific Business Incidents as they occur.
15.32 BIM Reports are issued for each Business Incident generated. BIM reports
are designed to notify Post Office Finance of the detail required to assist
in the reconciliation or settlement process within Post Office. BIM
Reports communicate information concerning the resolution of the
symptom of an underlying cause, not the cause itself. This information
would be supplied via the Problem Management route, if escalated to
that level
thélproblem Management Procedure
15.33 The aim of Problem Management is to investigate, eliminate or prevent
causes of Incidents and known errors regarding Post Office Account and
Post Office Limited Infrastructure / Information System and to prevent
the recurrence of Incidents related to these errors. A “Problem” in this
respect is defined as the unknown underlying root cause of one or more
incidents.*7!
15.34 The Problem Management Process covers both reactive and proactive
functions of Problem Management.
271 SVMSDMPROO025_5.doc, Post Office Account Customer Service Problem Management Procedure, 12 July
2016 [POL-0146787] [fe6a96a3615a63e094682c7efaa33090)
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 219
Number: 1_ Author: Gareth Jenkins Date: 26/10/2018 17:06:00
Sto comment on this?
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 204 of 225
15.35 Whilst there are several responsibilities detailed within the process
document, noBmention of advisory to Subpostmasters whereby an
incident may affect branch accounts is detailed. It could be that this was
incorporated within the “investigative” phase, but I would expect some
low-level detail to be provided as to what to do in the event that a
problem that has the potential to reoccur, might impact, and how to
handle such.
15.36 As highlighted in the same process document, effective Problem
Management is dependent upon the effective use of the process by
Fujitsu, Atos, POL and third parties. We have not yet established that it
was the case.
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 220
Number: 1_ Author: Gareth Jenkins Date: 26/10/2018 17:06:00
This is POL's responsibility not ours.
Number 2_Author: Simpkins John_Date: 30/10/2018 16:14:00 Z
Agree
FUJ00183797
FUJ00183797
180503R1935
16 October 2018
Page 205 of 225
16. Appendix F
cidJed Problem Records
Issue ID I Description of Problem Number of Users Affected
3 Horizon Overnight Release 10% to 40% of users affected
15 Negative Stock Figure on Branch 10% to 40% of users affected
Account
12 HNGx Generic Connectivity issue More than 70% of users
affected
25 End of Icon removal Single User
14 Receipts & Payment Mismatch 10% to 40% of users affected
19 First Rate Reconcilliation issue 10% to 40% of users affected
24 Cash declarations discrepancies 10% to 40% of users affected
30 HOL Online Disc POLFS 10% to 40% of users affected
32 Branches able to settle centrally 40% to 70% of users affected
>£150
37 Transaction harvesting issue 10% to 40% of users affected
52 Streamline payment file issues 10% to 40% of users affected
49 Transaction Correction Failure Single User
43 ROLLOVER Issue with cancel button I 10% to 40% of users affected
67 Harvesting of Branch event data 10% to 40% of users affected
76 Unexplained Fluctuations with the 40% to 70% of users affected
EBT (Electronic Banking
Transactions)
77 Postal Orders causing printer failures] Less than 10% of users
and issues on counters affected
78 Camelot file mapping issue causing I More than 70% of users
discrepancies
affected
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 221
Number: 1_ Author: Gareth Jer
Sto comment
Date: 26/10/2018 17:07:00
‘Sounds like a reasonably small number of incidents over a long timeframe.
Number: 2_Author: Simpkins John Date: 30/10/2018 16:15:00 Z
is this where CS uses the problem ticket type for a MI, we do not see those types of tickets in Peak (they can in theory be sent but usually remain in 1fS) and therefore we need to
check with Steve Bansal.
FUJ00183797
FUJ00183797
180503R1935
16 October 2018
Page 206 of 225
79
Transcash fees are shown incorrectly
on the end of day report
10% to 40% of users affected
80
POLSAP - Files/ data not received
following issues in the Fujitsu
domain
10% to 40% of users affected
81
Branches receiving “Unable to
Contact Data Centre” message when
attempting rollover.
10% to 40% of users affected
85
Impact of MDM on live service
following recent service incidents
40% to 70% of users affected
86
Some mobile van branches have
various connectivity issues
10% to 40% of users affected
88
Higate Near Station - 15 postage
labels produced but receipt shows 14
items.
Single User
92
Horizon Methods of Payment Issue
More than 70% of users
affected
96
Withdrawn products - Approximately
30 branches have been affected by
the withdrawal of Bureau / Savings
Stamps / Philatelic products.
Less than 10% of users
affected
105
POLSAP - RIS Table shows minusI
values for Bureau.
10% to 40% of users affected
115
POLSAP Data issue re. Article
BSBO000115 Created Incorrectly
More than 70% of users
affected
122
POLSAP Determine if there are RIS3
Discrepancies
10% to 40% of users affected
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
180503R1935
16 October 2018
Page 207 of 225
112
POLSAP Analysis of Misbalance on
628100
Single User
135 Counter Printer Issues for singer] 10% to 40% of users affected
counter branches
140 Branches haven’t rolled over into aj 10% to 40% of users affected
new Trading Period for a long time.
147 Complex Basket Not Working 10% to 40% of users affected
153 Paystation branch connectionI Less than 10% of users
problems affected
155 Shopping Basket files are either I More than 70% of users
being delivered later than expected affected
or not being delivered
154 Transcash values not deleting out,I More than 70% of users
but adding on to total affected
158 Short Dump running automated POe I Single User
settlement
159 Reconciliation identified (via
TPSC257 report) that POLFS records
were not being created from BRDB
160 POLSAP - POE Weekly Reports Issue I Single User
162 Duplicate Data Single User
165 Corrective actions process for future I Less than 10% of users
branch escalations affected
164 POLSAP - Missing data in POLSAP Less than 10% of users
(Paystation) affected
182 Disconnected sessions with foreign I Less than 10% of users
cards
affected
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 208 of 225
197 Token ID Mismatch More than 70% of users
affected
199 TFS : 5139830 - Issue with DIRD files] Single User
being overwritten
213 New Ingenico High Failure Rate 10% to 40% of users affected
Problem
220 Pinpad High Failure Rate Firmware I 10% to 40% of users affected
218 HSBC Merchant Aquirer (HMS) Less than 10% of users
incorrect EMIS file to FJ affected
226 Cash & Stock Screen Order 40% to 70% of users affected
228 Fibre Link 40% to 70% of users affected
229 Counter Slip Buffer Less than 10% of users
affected
235 MAG Card Track 2 Length Errors Less than 10% of users
affected
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 209 of 225
17. Appendix G
[Se@lspreadsheet]
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 225
Number: 1_ Author: Gareth Jenkins Date: 26/10/2018 17:07:00
Tiltook at that separately
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 210 of 225
18. Appendix H
Fahure Point Diagrams
18.1 The diagrams below are constructed summarisations of the processing
components within Horizon that have been identified as common points
in which discrepancies have been recorded.
18.2 The first diagram displays each of the failures examined overlaid on top of
each other. This helps to illustrate that the different aspects of Horizon
have suffered failures.
18.3 The diagrams that follow take each of the failures individually and attempt
display the flow of the transaction (using the dotted coloured lines) to
understand where the failure occurred.
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 226
Number. 1_Author: Gareth Jer Date: 26/10/2018 17:08:00
Not quite sure how these help and I don’t agree with all the failure points indicated on the diagrams below
ge Number: 2_Author: Simpkins John_Date: 30/10/2018 16:19:00 Z
This just shows to which point of the system a KEL refers. It is very high-level and does no show all the components involved
FUJ00183797
FUJ00183797
180503R1935
16 October 2018
Page 211 of 225
GMa: :
© Wrightm33145) :
CMawelsésik
MScardifieid 22195
jo! 2220
achal 233s
4010
carde235Q
_Milarvey225sP
REFERENCE
CUSTOMER
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 212 of 225
18.4 KEL cardc219R
KEL Type Unresolved
Title nBEb2 Section 5 CAPO State 4 -
transaction cancelled by pinpad
but not reversed resulting in state
4
Raised 11/05/2011
Last Updated 31/10/2016
Release System HNGX
System Product Counter
Issue: MSU report a transaction State 4 on one of the NWB reports,
C4 only. TES shows request and authorisation messages but no
confirmation. Post Office Counter log shows transaction being declined
by the pinpad: MSG00999: Card Removed Too Early.
No cOMessageDetail, hence the transaction does not get reversed.
Summary: Since a State 4 transaction implication is potential for
incorrect settlement with the Financial Institution (FI) and/or incorrect
adjustment of End Customer Account, considering the above KEL
records that the transaction was not reversed as it should have been
it iBlikely that branch accounts were affected potentially reckkding a
value transaction that should have been reversed and was not.
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 228
Number: 1 Author: Gareth Jer Date: 26/10/2018 17:09:00
“This is a bug that should have been fixed fairly quickly. Can we find out more
Number: 2_ Author: Simpkins John Date: 30/10/2018 16:28:00 Z
This was not fixed as it only affected Hypercom pinpads and we were Wansitioning to and Ingenico implementation
Number. 3 Author: Gareth Jenkins Date: 26/10/2018 17:10:00
“No. Branch accounts were not affected (assuming SPM followed the failure process). However Customer may have been out of pocket until this was resolved between POL and
the Fi
Number 4 Author: Gareth Jer Date: 26/10/2018 17:11:00
‘The transaction would have gone through as zero value on the counter and so was correct in the branch accounts.
Number 5 Author: Simpkins John Date: 30/10/2018 16:30:00 Z
Agree
FUJ00183797
FUJ00183797
180503R1935 16 October 2018
Page 213 of 225
REFERENCE
CUSTOMER
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
180503R1935
16 October 2018
Page 214 of 225
18.5 KEL Acha1233J
KEL Type Information
Title Incorrect cash declarations
received by Cash Management -
errors relating to cash
management
Raised 22/06/2012
Last Updated 22/06/2012
Release System HNGX
System Product
CounterBusinessapplications
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 215 of 225
Issue: ... it appears that somewhere after leaving Horizon but before
getting into the cash management system, an adjustment is being
made to add the branch Cash in Pouches overnight figure to what they
have actually declared. But rather than using the value for the correct
trading day, it is using the value from the previous day. Hence the
declarations being used by the Cash Management system do not
match the cash actually in the branch.
This problem will only affect branches which frequently hold large
amounts of cash in pouches overnight - these are likely to be the
larger branches. If they are affected by this problem, it is NOT user
error and they can't do anything to correct it themselves. Rebecca
Portch (Inventory Manager - South, Post Office Ltd - Supply Chain) is
aware of the problem - the branch could check with her, via the cash
management team, whether this could be the cause of the apparent
differences.
Summary: sine POLSAP are aware of the issue it is likely that any
branch differences could be corrected however, this still evidences an
issue in the data integrity in that incorrect values were being
erroneously added to branch accounts.
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 231
Number: 1_ Author: Gareth Jenkins Date: 26/10/2018 17:12:00
This had no impact on Branch accounts. The affect would have been to cause problem in predicting cash needed in the branch by the cash planning systems,
ge Number: 2_Author: Simpkins John_Date: 30/10/2018 16:40:00 Z
‘Agree, This was not a HORIZON issue but a POLSAP cash holding report issue only.
FUJ00183797
FUJ00183797
180503R1935
16 October 2018
Page 216 of 225
REFERENCE
DATA
“acha1233)
E
CUSTOMER
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
180503R1935
16 October 2018
Page 217 of 225
18.6 KEL Dsed4010N
KEL Type Unresolved
Title NB102 Section 5 State 4 - PM
pressing cancel on counter
moments after the customer has
entered their PIN on pinpad
Raised 28/01/2013
Last Updated 20/11/2015
Release System HNGX
System Product
HNG-X Counter(CNT)
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 218 of 225
Issue: The cash withdrawal was initiated and the customer got as far
as successfully completing PIN entry on the pinpad. However, within
milliseconds of this the PM has pressed Cancel on the counter. This
appears to have put the counter and pinpad in a confused state as an
authorisation request (R1) was generated despite the counter and
pinpad also trying to cancel the transaction.
The PM ended up being prompted with a message saying that the
transaction had been declined and a CANCELLED receipt was produced
for it. Moments after this a successful authorisation response (A3) was
then received which caused an 0291 System Error to occur. The PM
pressed Continue and was returned to the Front Office home menu
from where they continued the transaction session by selling some
stamps before settling the basket by Fast Cash.
On the branch database the cash withdrawal has been recorded as a
zero value and as a CANCELLED receipt was produced the PM should
not have handed any money over to the customer so this is ok.
However, a Cd reversal was not generated for the transaction and the
DRS hasn't got the C12 or C112. It just has the £142 C4 which means
£142 has been taken from the customer account. The recovery table
on the branch database showed no recovery outstanding for this
transaction.
Summary: Thi¥! ultimately would require input from Post Office to
manually reconcile the issue where there are discrepancies.
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 234
Number: 1 Author: Gareth Jenkins Date: 26/10/2018 17:14:00
Presumably a Peak fix was generated for this? Do we have the details?
,Number. 2 Author: Simpkins John Date: 30/10/2018 16:45:00 Z
This was not fixed, Peak PC0223225 has the following update " Development have returned the peak for closure as it was due to be fixed in the next pinpad frmware release but
this release is now not going to happen due to our contract coming to an end.” I think Fujitsu were in exit as this was a very infrequent issue it was decided not to fix.
<,Number: 3_ Author: Gareth Jenkins Date: 26/10/2018 17:13:00
Branch accounts would eb correct, but POL would need to sort out the customer issue
FUJ00183797
FUJ00183797
180503R1935 16 October 2018
Page 219 of 225
REFERENCE
dsed4010N
CUSTOMER
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
180503R1935
16 October 2018 Page 220 of 225
18.7 KEL johnbascoG5222H
KEL Type Information
Title Counter APS Txn posted against
incorrect APS Account
Raised 11/07/217
Last Updated 11/07/2017
Release System HNGX
System Product HNG-XCounter(CNT)
Issue: It is been noted that two AP Qangerous goods transactions 6
minutes apart on the same counter in two different baskets. Both
record the same CuRef (barcode), same AP token, same client code
and SVC, but one of them has client account code 3047 (correct) and
the other 3046 (incorrect, does not exist in ref data).
On investigation, we find that the issue is either not reproducible or
there is no code which seems to cause this issue. On every retry, the
client accounting code is always posted correctly to 3047.
Additionally, the reference data is verified and there is no client
accounting code with 3046.
Occurrences of this issue is rare however it has business impact so, a
fix will be provided to prevent this from happening i.e. to add a check
and verify if the clientAccountCode exist in ref data and raise a system
error if it does not.
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 236
Number: 1_ Author: Gareth Jenkins Date: 26/10/2018 17:14:00
These are zero value transactions and just pass info to RM and HMRC
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 221 of 225
Summary: wHist this does not immediately indicate a branch
discrepancy it does highlight issues with reference data occurring in
the processing systems. KEE/MWright1458Q details an instance where
reference data could impact branch accounts.
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 237
Number: 1_ Author: Gareth Jer
“There wasn't one!
Date: 26/10/2018 17:20:00
Number: 2_Author: Simpkins John Date: 30/10/2018 16:48:00 Z
Tdo not agree, the reference data was used to harden the checks for invalid customer references but did not cause the issue. Fix was release in R16.90
Number. 3_ Author: Gareth Jenkins Date: 26/10/2018 17:20:00
“What is this about.
Number: 4 Author: Simpkins John Date: 30/10/2018 16:52:00 Z
This is about "User forgot to Rem Out obsolete stamps before the icons disappeared. Now she cannot do anything as the product is no longer shown in adjust stock etc.
Ifa product or products have not been remmed out within the time then the system will remove the product/s next time the stock unit is rolled over, This will also cause a cash
discrepancy of the equivalent sum. PM will have to contact NBSC to resolve the discrepancy.
FUJ00183797
FUJ00183797
180503R1935
16 October 2018
Page 222 of 225
johnibescoG5222H
REFERENCE
DATA
CUSTOMER
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
POU
p
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 223 of 225
18.8 MithyanthaJ1937S
KEL Type Information
Title Exception raised whilst processing
message event —- Sefous system
error:[Duplicate JSN detected -
failed to insert journal record]
Raised 06/05/2010
Last Updated 09/08/2016
Release System HNGX
System Product BranchAccessLayer
Issue: Duplicate JSN messages will only occur when there is a record
with the same JSN number already in the message journal table
BRDB_RX_MESSAGE_JOURNAL.
UPDATE: 04-Nov-2010 - Until this fix is released, it has been agreed
that these events will be ignored
UPDATE: 16-06-2011- The fix is still with RM (Release Management)
so not yet delivered.
UPDATE: 09-Aug-2016 - Fix was released but variations of issue are}
still seen on rare occasions and should be sent to SSC to investigate
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 239
Number: 1_ Author: Gareth Je
Agreed that this is a serious error.
Date: 26/10/2018 17:21:00
Ilam aware of an issue here early in the life of HNG- (possibly this one), but am surprised to see later dates on this KEL.
Has this re-occurred?
Number. 2_Author: Simpkins John Date: 30/10/2018 17:51:00 Z
The last occurances were in 2016 - PCO253096 and PCO252499,
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 224 of 225
KEL
Summary: altibugh it is not documented as causing a potential
branch account discrepancy in this instance, the problem illustrates
how retry requests between the counter and the branch database
(BRDB) can occur an@lhave over an extended time period.
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 240
Number: 1_ Author: Gareth Jer
True it wouldn't
Date: 26/10/2018 17:22:00
Number: 2_ Author: Gareth Jenkins Date: 26/10/2018 17:23:00
Does it?
Number: 3_Author: Simpkins John_Date: 30/10/2018 18:06:00 Z
“think he is referring to the dates in the KEL
FUJ00183797
FUJ00183797
180503R1935
16 October 2018
Page 225 of 225
REFERENCE
DATA
CUSTOMER
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
p
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 226 of 225
KEL
18.9 Wrightm33145J)
KEL Type Fault_Fixed
Title offle has Non-Zero_ Trading
Postion (Receipts/Payments
mismatch) Office receipts do not
match payments. Office has a
non-zero trading position.
Raised 23/09/2010
Last Updated 01/04/2016
Release System HNGX
System Product
CounterBusinessapplications
be corrected.
rise to the accounting error.
Issue: The Receipts / Payments mismatch is due to a bug in the code
that occurs when Cancel is pressed on MSG31316. The bug incorrectly
causes the discrepancy to be cleared, and because there is no
balancing transaction (such as a transfer to local suspense) it gives
Unfortunately the workaround cannot be done after the problem has
occurred at the office! In this case the branch accounts will need to
within branch accounts.
Summary: Acknowledged issue by Post Office. Correction needed
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 242
Number: 1_ Author: Gareth Jenkins Date: 26/10/2018 17:23:00
This is the suspense issue discussed at length earlier,
ge Number: 2_ Author: Simpkins John_Date: 30/10/2018 18:07:00 Z
He has left off the part of the KEL where it details when the fix was released:
"Ref Data (pall) fix released in November 2010 under peak PCO204263"
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 227 of 225
REFERENCE
DATA
I Wrightm33145J
CUSTOMER
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP Peer
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 228 of 225
KEL
18.10 GMaxwell302)
KEL Type Information
Title Transaction Correction not
received at FAD
Raised 04/10/2005
Last Updated 04/10/2005
Release System S80
System Product TPS
Issue: POLFS report that transaction corrections sent for a particular]
office did not arrive at the office.
Summary: Since Transaction Corrections are issued to clear
anomalies in financial values, it is imperative these are applied where
necessary. Aitko ugh this instance identifies the error in the file
validation and therefore ‘catches’ the loss, it still evidences issues in
data processing within Post Office processing systems.
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 244
Number: 1_ Author: Gareth Jenkins Date: 26/10/2018 17:24:00
This should have been fixed and the file resent. I would expect such occurrences to be very rare and quickly fixed
Can we be more specific?
«Number: 2_Author: Simpkins John Date: 30/10/2018 18:09:00 Z
The files will no longer be available from audit but the error file (if20050929001 ert) would have been available to POL immediately after processing, delivered via FTMS.
The Peak details the file contents:
TCERH;200509300606 18;if2005092900 1.tcrvif2005092900 1.err
TCERR;TOZ,022;Quantity and Velue are both 0;00002;05
TCERR‘TDZ;043;Quantity not supplied for stoc;00002;05
TCERRTOZ;040;Value is outside permitted ran;00003;03,
TCERT3
This was an intentially bad file sent to the model office for testing.
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 229 of 225
REFERENCE
DATA
~ GMaxwell3025
CUSTOMER
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 230 of 225
KEL
18.11 MHarvey2255P
KEL Type Information
Title TPSC294 operational exception
Raised 02/09/2005
Last Updated 31/05/2006
Release System S80
System Product TPS
Issue: This is not a software problem and should only occur when
SSC have manually added balancing entries to
tps_pol_fs_summaries_incomp following a problem with missing
POLFS mappings.
Summary: This issue relates to manual corrections (Paring entries)
applied within database transaction summary files. Sinee summary
files inform Post Office of financial positions, it is entirely possible that
this may cause financial discrepancies if manual corrections were to
be incorrectly applied.
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 246
Number: 1_ Author: Gareth Jenkins Date: 26/10/2018 17:25:00
No impact on branches and this wouldn't result in TCs being generated
ge Number: 2_ Author: Simpkins John _Date: 30/10/2018 18:22:00 Z
Agree, this was an attempt to help reconcile POLFS service following incorrect mappings, however as they already had received a ble file for this branch they needed to be
reconciled manually
FUJ00183797
FUJ00183797
Page 231 of 225
180503R1935 16 October 2018
REFERENCE
DATA
ce S
MHarvey2259P :
CUSTOMER
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
yrou
p
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 232 of 225
KEL
18.12 GMaxwell3651K
KEL Type Information
Title Duplicate transactions sent to
Streamline
Raised 22/12/2004
Last Updated 08/05/2006
Release System S75
System Product DCS
Issue: Streamline report that they have received a payment file
containing a large number of transactions which they had received in
a previous run.
It looks like a problem with the confirmation agent which has
harvested the information twice (most likely due to a previous failure
resulting in a failure to write a new checkpoint).
Summary: This_ duplication has primary effect with Post Office
Processing. It not clear whether it may have in turn caused
discrepancy in branch accounts.
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 248
Number: 1_ Author: Gareth Jer Date: 26/10/2018 17:25:00
This was a one off issue and due to operational issues, Detected by streamline as designed in the E2E architecture of the interface.
No problems resulted from it
Number. 2_Author: Gareth Jenkins Date: 26/10/2018 17:26:00
Itwouldn’t affect branch account even if the duplicate file had been processed!
Number: 3 Author: Simpkins John_Date: 30/10/2018 18:28:00 Z
Agree, this would reconciliation only.
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 233 of 225
REFERENCE
GMaxweli3651K
CUSTOMER
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP i oo
FUJ00183797
FUJ00183797
180503R1935 16 October 2018 Page 234 of 225
KEL
18.13 cardc235Q
KEL Type Information
Title Drop&Go top up transaction times
out but is then marked as
successful.
Raised 05/07/2017
Last Updated 05/07/2017
Release System HNGX
System Product CounterBusinessapplications
Issue: thAlclerk initiated a Drop and Go transaction for £100 which
failed due to timeouts, but then a success message was displayed.
The clerk settled the transaction and the customer handed over £100.
The customer checked the balance and stated that the top up had not
gone through, so the clerk then performed another Drop&Go
transaction which was successful. The customer has paid in £100 but
the branch account has been debited by £200. Accenture verified that
only the second Drop&Go top up was successful.
Summary: Thiel issue would require a transaction correction being
applied within the branch accounts to remove the discrepancy.
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
FUJ00183797
FUJ00183797
Page: 250
Number. 1_Author: Gareth Jer Date: 26/10/2018 17:27:00
This is down to a poorly implemented AP ADC script by POL which didnt handle errors correctly
ge Number: 2_ Author: Simpkins John_Date: 31/10/2018 11:03:00 Z
Agreed
Number. 3 Author: Gareth Jenkins Date: 26/10/2018 17:27:00
agreed
FUJ00183797
FUJ00183797
180503R1935
16 October 2018 Page 235 of 225
carde235Q
REFERENCE
DATA
CUSTOMER
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Free’
ths LLP