POL00262929
POL00262929
CLAIM NUMBERS: HQ16X01238, HQ17X02637 & HQ17X04248
IN THE HIGH COURT OF JUSTICE QUEEN’S BENCH DIVISION
BETWEEN:
ALAN BATES & OTHERS
CLAIMANT
AND
POST OFFICE LIMITED
DEFENDANT
SUPPLEMENTAL EXPERT REPORT OF JASON COYNE
01 FEBRUARY 2019
OCCUPATION: PARTNER
SPECIALIST FIELD: IT SYSTEMS
ON THE INSTRUCTIONS OF: FREETHS LLP
ON BEHALF OF: ALAN BATES & OTHERS
ASSISTED BY: SIOBHAN FORSTER
Jason Coyne Te
IT Group UK Limited Fa
Unit 5 Lockside Office Park Email: Jason.Coyn 3
Lockside Road Reference: 181024SR1935
Preston
PR2 2YS
POL-BSFF-0100992
POL00262929
POL00262929
181024SR1935 01 February 2019 Page i of iv
Table of Contents
List of Annexes .
List of Appendices
LISt Of FIQUIrGS wusscssrensesccnssssessennseasscuassessnenesesscasdnessuneceessansneedsnneounssenusanssn iv
1. Exe Cutive SUMIMALSY «2. .cceccceeeeeeeneeeeeeneeeeeeneneeeecenneneneeesoeesseeseeeeeneneeee L,
2. Introduction
3. PEAK, MSC and Privileged User Log AnallySis ......1:::::ssssssssesesseneeensees 6
Analysis of the PEAKS — Horizon ISSUES ..........ccceceeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeneeen 8
Managed Service Change DISCIOSUSE i... :<scc0n ssciceseesscioostiessciseusiesscseuepssomenee secesnce se 84
PriVIGGEd USEr LOG DISCIGSUt Gres ss gnnmns 1 seamen 1 ctuses 54 tmemnie 54 dsm Karena ws dere Ba decNS 88 86
4. Defendant’s Responsive Witness StateMments......ccccecsesssseeseneeneneees 89
Torstein Olav Godeseth .......ccsscceseessecsssesseesseeesseecsseesseessseessseecseeesseesseeeseess 89
Tire Jae WSMiys MATING as es cnteteen 45 geste 4s snniiate 4s saan 64 ates 8 nsteins 6 caine 44 tte 46 atte 104
Angela Margaret Van Den Bogerd ........cccscceseeeeeeeeeeeeeeeeeeeeeeereeereeeeeeeeeeeneeeneees 106
Stephen Paul Parker .........:ssssssssssssssssssssssnsssssssnassnassnsssesseesenecaseeeseeeneeeseeenes 110
Pal Smiths casenes senses srensnoessenswne aasnonns aa swsees st cnewen aunasews sa ceweusadcaseenanuaseuean enawas 114
5. Dr Worden’s Expert Report ......sscseeeeeeeesssseeeeeeeeeeenseeeeeesenseeeeeeneneees 116
Introduction and Overview
Sections 3,4 & 5: Business Applications & HOriZOM .......ccccccsseeeeeeeseeeeeeeeeseeeneees 123
Section 6: COUNterMEASUFeS .......cesseceseceseessseesteessseesseeeseeesseeeseeeseeseaeeenaeens 133
Section 7: Robustness Of HOriZOn ....sesssssveasscssseessssssessascenseensrensssesnseessennae 172
Section 8: Effect on Horizon Bugs On Branch ACCOUNES ......cceceseeeseesseertereeeeeee 189
Section 9: Reconciliation & Transaction Corrections .. 214
Section 10: Facilities available to Subpostmasters . 220
Section 11: Facilities available to Post Office & FUjitSU .........cccceeseesseesseeteeeeees 230
G6. Expert Declaration .......cccccseeeeeeseseeeseeeeeeeeneeenessseeeseeeneeeeeeeseeenseeeees 290,
Statement of Truth.
7. Appendices 259
APPENGIX A: ascii. sasnsas ss snnevs sssuwas sssivuwva ve siwawis vssinwiis so aiiviniiaen dlivawian sis vulions shiulWve naive 259
ADPONAIK Bs evaves is svcwes st anenas ss qeewes 14 enswen a6 aneets ss anaems sé anavms 64 apewmn és swans is awewas ii reaTeED 260
Prepared by: Jason Coyne
Occupation: Partner (of .
Specialist Field: IT Systems It
On the Instructions of: Freeths LLP
POL-BSFF-0100992_0001
POL00262929
POL00262929
181024SR1935 01 February 2019 Page ii of iv
Appendix C.. 264
PASO SINCNSC DD weiss « eins sv scnsns os spinins 9 apiece gn gates ob scans os sgh 9a sitdine ns wena ss tw as Gea 265
Prepared by: Jason Coyne
Occupation: Partner (of .
Specialist Field: IT Systems It
On the Instructions of: Freeths LLP
POL-BSFF-0100992_0002
POL00262929
POL00262929
181024SR1935 01 February 2019 Page iii of iv
List of Annexes
Annex A Archive of Requests for Information
List of Appendices
Appendix A Letter from WBD re Privileged User Logs and MSC Logs
disclosure.
Appendix B HNG-X Menu Hierarchy
Appendix C USERID’s found in Horizon Privileges Access Logs
Appendix D Amendments to First Expert Report 180503R1935 01-01
Served 16 October 2018
Prepared by: Jason Coyne fe)
Occupation: Partner (of .
Specialist Field: IT Systems \ It
On the Instructions of: Freeths LLP 7
POL-BSFF-0100992_0003
181024SR1935
POL00262929
POL00262929
01 February 2019 Page iv of iv
List of Figures
Figure 1
Figure 3
Figure 4
Figure 4
Figure 6
Figure 7
Figure 8
Illustration of identified relationship between: Master
PEAKs, Release PEAKs, PEAKS and KELs
Excerpt from design document HNG-X Menu Hierarchy
and Messages
Legacy Horizon and further Auditable Activities
HNG-X and further Auditable Activities
Graph detailing the average loss per month of each
claimant against the length of their tenure (from Dr
Worden's Expert Report)
Graph detailing the number of claimants with losses per
year (from Dr Worden's Expert Report)
Graph showing overall losses per year (from Dr Worden's
Expert Report)
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
POL-BSFF-0100992_0004
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 1 of 265
1.
1.1
1.2
Executive Summary
I provide this supplemental report in relation to the agreed Horizon
issues which I addressed in my first report dated 16 October 2018. The
purpose of this report is to update the Court with my opinions in relation
to those issues, having now had the opportunity to consider further
documents (in particular further PEAKs, Managed Service Change logs
and Privileged user access logs) and further witness evidence, and to
respond to the opinions expressed by Dr Worden in his report, dated 7
December 2018,
I consider that Horizon is less robust than as originally expressed in my
first report. My primary reasons for this are as follows:
a. Access to modify the Horizon branch database was not as
restricted as it should have been;
b. Whilst said to be governed by a documented policy, the actions
were that were actually being undertaken by support staff were
unaudited;
c. Post Office do not consult the full audit data before ruling on a
discrepancy, instead using third party client reconciliation data or
subsections of the audit data from within Credence or HORice;
d. The PEAKs are consistent with many more bugs/errors and defects
shown to impact branch accounts than the initial three
acknowledged by Post Office;
e. PEAKs show defects have lain undetected in Horizon for extended
periods without detection;
f. PEAKs confirm Post Office often only becoming aware of bug/errors
and defects when Subpostmasters report problems, suggesting
that Post Office detection methods are not as good as initially
suggested;
Prepared by: Jason Coyne fe)
Occupation: Partner : -
Specialist Field: IT Systems g It ] }{
On the Instructions of: Freeths LLP < -
POL-BSFF-0100992_0005
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 2 of 265
g. PEAKs confirm that Post Office have suspended active
investigations into known discrepancy causing bugs due to
Subpostmasters not reporting shortfalls.
1.3 Other important matters which I have identified and considered which
impact the robustness of Horizon and other issues before the Court are
as follows:
a.
Evidence exists which displays that Horizon has suffered from
bugs, errors and defects that have impacted branch accounts.
These bugs, errors and defects number many more that the three
acknowledged by Post Office and some existed for extended
periods before they were detected. (Horizon Issues 1,3,4,5)
Evidence exists which displayed that Post Office (via its
subcontractors) modified transaction data which impacted branch
accounts during the course of supporting the Horizon System.
Evidence exists which shows that Post Office (via its
subcontractors) was aware that wider access that was permitted
to the Horizon branch database and that users could and did access
the Horizon system databases. The actions from such access was
not recorded in the audit logs.
Evidence exists which shows that it was often a Subpostmaster
who first detected the impact of bugs, errors and defects and
reporting their existence to Post Office, rather than Post Office
detecting such bugs, errors and defects themselves and preventing
such branch impact from occurring.
Whilst there are audit logs available to Post Office to assist it in
determining the impact of Horizon issues on branch accounts, the
complete picture is only available by requesting the full audit logs
which are not typically accessible to Post Office but are stored by
Fujitsu. Based on my review of the evidence Post Office rarely
requested these full audit logs even in the knowledge that they are
required for the complete picture.
Prepared by: Jason Coyne fe)
Occupation: Partner : -
Specialist Field: IT Systems g It ] }{
On the Instructions of: Freeths LLP < -
POL-BSFF-0100992_0006
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 3 of 265
1.4 Dr Worden’s report focuses on Horizon having ‘countermeasures’ which
he opines make it robust. Dr Worden’s countermeasures are simply
basic elements of practical system design and in many respects Dr
Worden’s opinions are based on design aspirations.
1.5 Outside of the bugs, errors and defects acknowledged by Post Office, Dr
Worden has not reported on evidence which show how additional bugs,
errors and defects in fact arose and impacted branch accounts.
1.6 Dr Worden’s consideration of the financial impact of bugs, errors and
defects is based on assumptions which are shown to be faulty when
technical evidence is considered.
Prepared by: Jason Coyne fe)
Occupation: Partner .
Specialist Field: IT Systems g It ]
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0007
181024SR1935
POL00262929
POL00262929
01 February 2019 Page 4 of 265
2. Introduction
2.1 Since my first report I have been provided with the following
documents:
a.
vi.
vii.
Supplemental PEAK disclosure, received 31 October 2018,
comprising 3,886 documents which were responsive to keywords
for privilege and as such had to be manually reviewed by Post
Office.
Defendants responsive witness statements dated 15 or 16
November 2018, and exhibits namely:
Torstein Olav Godeseth (second statement);
Tracy Jane Wendy Mather;
Angela Margaret Van Den Bogerd (second statement);
Stephen Paul Parker;
Paul Ian Michael Smith;
David Malcolm Johnson (second statement);
Andy Dunks; and
Dr Worden’s first Expert Report dated 7 December 2018 and
supporting documents. This disclosure included 4 documents
describing the Horizon System Architecture, a report prepared by
Fujitsu describing the operational services provided to Post Office
and a document estimating the value of losses experienced by
Claimants in the Group Litigation.
Privileged User Logs and Managed Service Change logs
following a letter from the Defendants dated 21 December 2018.
Second witness statement of Claimants’ witness Richard
Roll dated 16 January 2019.
Prepared by: Jason Coyne
Specialist Field: IT Systems
Occupation: Partner Fit
On the Instructions of: Freeths LLP
POL-BSFF-0100992_0008
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 5 of 265
f. Additional documents referred to in the Defendant’s
witness statements following a request by the Claimants’
solicitors for disclosure .
g. Supplemental KELs disclosure, received on 17 January 2019
following a request from the Claimants’ solicitors, comprising those
KELs which had been deleted and not provided in the original KELs
disclosure in March 2018. I have also been provided with KELs
dated from March 2018 to date.
h. Operational Change Process documents (OCPs) were
disclosed on the 25 January 2019, but due to the proximity of these
to the report deadline these have not been reviewed.
2.2 Very shortly before finalising this report I was provided with a further
responsive witness statement from Mr Parker, dated 29 January 2019,
responding to the second witness statement of Richard Roll. My
attention has been drawn to paragraphs 27 and 32 of that statement in
relation to remote access, but I have not otherwise had an opportunity
to consider the contents of that statement or its exhibits before
finalising this report.
2.3 I have also had more time to search the 222,254 PEAK records which
were disclosed by the Defendant on 27 September 2018 shortly before
my first report submission dated 16 October 2018. A further PEAK
analysis including those received within the supplemental PEAK
disclosure, in addition to an analysis of the Privileged User Logs and
Master Service Change logs referred to above, is set out in section 3.
2.4 I understand that the Claimants’ solicitors have requested further
documents referred to in the Defendant’s witness statements, by letter
dated 22 January 2019.
Prepared by: Jason Coyne fe)
Occupation: Partner g .
Specialist Field: IT Systems It }
On the Instructions of: Freeths LLP
POL-BSFF-0100992_0009
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 6 of 265
3.
PEAK, MSC and Privileged User Log Analysis
Introduction
3.1
3.2
3.3
3.4
3.5
PEAKs are generally a valuable source of information as they are
documents that are created when issues which need further analysis
are reported. Generally, PEAKs conclude with the determination of the
root cause of the issue and supporting guidance for closure of the
record. Others conclude with the determination that a previous Known
Error Log (KEL) has already recorded the issue and that, by following
the known or advised course of action, any further incidents can be dealt
with utilising information within the KEL.
KELs do not provide a comprehensive view of the specific impact to
branch accounts. The KELs are generally a summary of an issue or error
that has been identified within Horizon and provide information to other
support users on interim workarounds and how to assess the problem
in future should another support call be raised.
The KELs provide an overview of the symptoms and the interim
resolution of the issue whilst the PEAKs relate more to the investigation
and identification of the root cause, its impact on a particular branch
along with any further detail (such as whether any account modification
is required). Often, one KEL will be referred to by many different PEAKs.
Therefore, KELs on their own are not sufficient for establishing full
branch impact in relation to analysing bugs/errors and defects or
discrepancies as they do not contain isolated, branch specific,
information in the way that PEAKs generally do (see Horizon Issue 1
PEAK observations at paragraphs 3.60 to 3.63 of this report).
Additionally, review of certain PEAK records have highlighted that
known bugs/errors and defect records were closed with no remedy, but
to provide a workaround from advice via a KEL for any future
occurrences of the same (or related) issue, rather than detailing a bug
fix. It is not clear how or if such PEAKs were transferred to a log or list
of issues to be addressed at a later point. Where this is known to occur
Prepared by: Jason Coyne fe)
Occupation: Partner : -
Specialist Field: IT Systems g It ] }{
On the Instructions of: Freeths LLP < -
POL-BSFF-0100992_0010
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 7 of 265
3.6
3.7
3.8
3.9
in other PEAKs it is usually stated in the detail that the issue will be
rectified in a later fix. However, many records appear to just be closed
without a documented future resolution.
I have identified from my analysis of the PEAK disclosure that
Operational Change Process documents (OCPs) are the records in which
changes to LIVE data are recorded. I believe that Managed Service
Change (MSC) data is useful to analyse as they document what agreed
changes should be made to the Horizon service. I understand that MSCs
replaced OCPs! although the date at which they did so is not explicitly
clear.
I received the MSCs on 21 December 2018 so I have not been able to
review them in great detail. My initial review is contained later in this
section from paragraph 3.307. I received the OCPs on 25 January 2019,
so I have not had time to consider them as part of my report.
The Defendant has also provided “Privileged User” logs upon request.
Such logs should record where Fujitsu support teams have gained (more
advanced) access to that of a typical Horizon user within the Horizon
system. Privileged User logs were provided with the MSC disclosure on
21 December 2018. I have set out my findings in relation to these from
paragraph 3.316 onwards.
In this section of my report I set out the analysis which I have been
able to carry out:
a. the PEAKs disclosed shortly prior to publication of my first report;
b. the PEAKS contained as part of the supplemental disclosure of
PEAKs provided after my first report;
c. the MSC documents recently disclosed; and
d. the Privileged User logs recently disclosed.
1 {First Witness Statement of Torstein Olav Godeseth dated 27 September 2018}
Prepared by: Jason Coyne
fe)
Occupation: Partner : .
Specialist Field: IT Systems : It FC
[e)
On the Instructions of: Freeths LLP
POL-BSFF-0100992_0011
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 8 of 265
Analysis of the PEAKs - Horizon Issues
3.10 The initial request for disclosure of the PEAKs was originally made via
email on 20 July 2018. On the 27 September 2018, 218,367 PEAKs were
disclosed by Post Office. At that stage, my report was in an advanced
state and therefore only 1,262 (~0.5%) of these PEAKs were
categorised/reviewed by the deadline for my first report submission. An
additional 3,887 PEAKs were provided in a supplemental disclosure on
31 October 2018. Some PEAKs detail the first observations of a problem
reported by a Horizon user recording user activity detail (what keys
were pressed, screens viewed etc), the specific branch that encountered
the issue (via a recorded “FAD” code) along with the discrepancy value
and the concerns communicated by the user.
3.11 Other PEAKs are “cloned” from an original. I have observed that this
typically occurs when a PEAK needs to be sent to another support
department for further analysis or where another incident (i.e., in a
differing branch) has been identified in relation to the same problem but
has its own individual circumstances.
3.12 Despite the usefulness of PEAKs to identify recorded Horizon issues,
they are not without their limitations and I have observed several
reoccurring thematic issues.
3.13 For example, it appears that PEAKs are often closed or suggested to be
closed if analysis has paused or has not uncovered a full diagnosis
despite the Subpostmaster and/or Post Office not having a conclusion.
It is also not always clear whether a Subpostmaster was informed of
any action (e.g. modification of branch account data) or impact,
following the raising and consequent resolution of the PEAK. I have seen
PEAK records that are closed despite support not being able to diagnose
a root cause whilst acknowledging that there clearly is some form of
error occurring within the Horizon system.
3.14 Additionally, it appears that analysis and resolution can be delayed
between Post Office, ATOS and Fujitsu, especially where there is a
Prepared by: Jason Coyne fe)
Occupation: Partner .
Specialist Field: IT Systems g It ]
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0012
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 9 of 265
disjoint in the understanding of which party should be providing the
evidential data or analysis/resolution of the issue.
3.15 I have also observed that certain PEAKs contain limited information so
it is not always possible to identify which particular issue they relate to.
In other instances, the root cause is inconsistent throughout different
PEAKs. PEAKs arising from the same broad issue (i.e., deletion of
session data) have their “Root Cause” diagnosis as both “General -
User” and “Gen - Outside Programme Control” which in my opinion
makes it difficult for the reader to clearly understand what the actual
root cause diagnosis was.
3.16 The fact that PEAKs are cloned causes further difficulties because there
is no succinct way to identify ALL PEAKS (and clones) that relate to a
single bug, error or defect or how man bugs, errors and defects there
are in existence and recorded within the total PEAK platform. The format
of the PEAK disclosure provided requires that one must read through
the entire PEAK to identify all potential related “clones” (if they have
been cloned for further analysis). This is explained further in in Schedule
6 of a letter dated 28 July 2016? and is discussed in the next section of
this report.
3.17 Ihave also discovered “Master PEAKs” which often document the PEAKs
that may be related to a particular issue but do not necessarily capture
all actual related PEAKs (for example, the Master PEAK in relation to the
Receipts and Payments mismatch bug PC0204765? did not identify all
the PEAK records of affected branches (only some of them) see
paragraph 3.28-3.33).
3.18 There are also “Release PEAKs” that document the bug fix detail and
those PEAKs in existence that may be fixed by the resolution
documented (note: they may not always contain every actual related
PEAK that might be resolved by the fix). Further, related PEAKs
2 {Letter of Response from Post Office, SCHEDULE 6: REBUTTAL OF ALLEGATIONS AGAINST
HORIZON, 28 July 2016}
3 PEAK PC0204765, 25 September 2010 {POL-0374542}
Prepared by: Jason Coyne
fe)
Occupation: Partner : .
Specialist Field: IT Systems ) It 1 ¢
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0013
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 10 of 265
contained within a Master PEAK may, individually, reference completely
different KELs (see illustration below as an example and paragraphs
3.60-3.62 of this report for further detail).
Master PEAK Pcon2* Master PEAK Coot"
Figure 1 Illustration of identified relationship between: Master PEAKs, Release PEAKs,
PEAKS and KELs
3.19 In my opinion the PEAK system is expansive and, whilst useful, is not
without limitations or flaws. However, ultimately, PEAK records provide
more comprehensive information in relation to identified bugs, errors
and defects and their specific branch impact than what is provided or
can be determined from the KEL records.
3.20 I have now reviewed more of these records using text search criteria
and filtering. This has enabled me to address some issues more
thoroughly and has enabled a more in-depth analysis in relation to the
extent of the Horizon Issues and the overall robustness of Horizon.
3.21 For relevance, I have grouped the observed PEAKs in order of which
Horizon Issue I believe them to relate. Due to the high number of PEAKs
disclosed, it has not been possible to review them all, and therefore the
PEAKs captured below are not an exhaustive representation of the
potential bugs, errors and defects within Horizon. Not all of those PEAKs
I have reviewed necessarily feature in this report, since disclosure was
provided of PEAKs relative to LIVE environment incidents (“Quality
Centre” PEAKs and others relative to testing have been excluded). I
have provided Dr Worden with a list of PEAKs I was dealing with at the
time to assist with his analysis.
Prepared by: Jason Coyne fe)
Occupation: Partner ae
Specialist Field: IT Systems it Jrou f
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0014
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 11 of 265
Table 1 Table of Bugs, Errors or Defects located in the PEAKs reviewed
Bug. Error or Defect Horizon Evidence of Paragraph Page
Referred to as: Issue Branch Impact
Receipts and Payments Mismatch 1 Yes 3.27 12
Callendar Square / Falkirk 1 Yes 3.34 15
Suspense Account Bug 1 Yes 3.43 18
Branch Out Reach Issue d, Yes 3.46 19
(Dalmelington)
Remming In 1 Yes 3.56 22
Remming Out 1 Yes 3.67 26
Local Suspense Issue * Yes 3.78 30
Recovery Issues 1 Yes 3.84 32
Reversals Yes 3.99 37
Data Tree Build Failure 1 Yes 3.106 38
Discrepancies
Girobank Discrepancies 1 Yes 3.119 41
Counter Replacement Issues 1 Yes 3.129 44
(Rebuild / One sided Transactions)
Withdrawn Stock Discrepancies 1 Yes 3.132 45
Bureau Discrepancies 1 Yes 3.140 46
Phantom Transactions 4 Yes 3.148 49
Reconciliation Issues 4 Yes 3.154 50
Branch Customer Discrepancies Yes 3.174 54
Concurrent Loggins 4 No 3.179 55
Post & Go / TA discrepancies in 4 Yes 3.185 56
POLSAP
Recovery Failures 4 Yes 3.191 58
Transaction Correction Issues 4 No 3.197 60
Bugs/Errors Defects introduced by 10 Yes 3.211 63
previously applied PEAK Fixes
3.22 In relation to Issue 1 of the Horizon Issues, I opined in my previous
report at paragraph 3.1 (Page 12) (and as agreed in the Experts’ Joint
Statement) that bugs errors and defects within Horizon have caused
discrepancies within branch accounts. The PEAKs referred to in this
section reinforce this opinion.
3.23. Review of the PEAKs has highlighted records provided from 1997 to
2018 illustrating varying bugs throughout the lifespan of both legacy
Horizon and Horizon Online or ‘HNG-X’ as it is often referred to.
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems I
group
On the Instructions of: Freeths LLP =
POL-BSFF-0100992_0015
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 12 of 265
3.24 It should be noted that it is possible that other PEAKs detailing bugs
/errors and defects did cause financial discrepancy in branch accounts
but are not detailed here as:
a. it has not been possible in the time of reporting to analyse all
220,000+ PEAKs; and
b. it is not always documented clearly within the PEAKs whether a
Subpostmaster had a discrepancy, what its value might have been,
or how it was resolved.
3.25 PEAK records typically focus on documenting the bug and its root cause
and not necessarily its full impact or financial resolution (PEAK records
are part of Fujitsu’s investigation where the financial resolution is
determined by Post Office). They rarely depict all other related
occurrences of the same issue. However, several PEAKs I have reviewed
clearly record an encountered financial discrepancy.
‘Acknowledged Bugs’ (Horizon Issue 1)
3.26 This subsection initially deals with the PEAKs which I have identified and
which relate to the three initially acknowledged bugs (in Post Office’s
letter of response) and the Dalmellington / Branch Outreach issue bug
which I addressed in my first report, and which is now dealt with by Mr
Godeseth and Mr Parker in their responsive witness statements (which
I separately respond to at section 4 below).
Receipts and Payments Mismatch Bug (Horizon Issue 1)
3.27. This bug is acknowledged by Post Office* as affecting 62 branches (37
of which were Subpostmasters as opposed to Crown offices/multiples)
with the majority of incidents occurring between August and October
2010 within Horizon Online. In summary, when users followed certain
process steps, it resulted in a differing accounting position between
what was held in the branch and what was held on Post Office’s back
office figures in POLSAP and POLMIS. The effect of the bug meant that
4 {Letter of Response from Post Office, SCHEDULE 6: REBUTTAL OF ALLEGATIONS AGAINST
HORIZON, 28 July 2016}
Prepared by: Jason Coyne
fe)
Occupation: Partner : .
Specialist Field: IT Systems ) It 1 ¢
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0016
181024SR1935
POL00262929
POL00262929
01 February 2019 Page 13 of 265
discrepancy values ‘disappeared’ from a Subpostmaster’s view of their
account.
3.28 I have investigated the PEAKs/KELs within both the documentation
referred to in the responsive witness statement of Mr Godeseth® (who
documents the details of this bug) and documents identified by Dr
Worden in order to determine whether said documents capture the full
details of all 37 Subpostmasters affected.
3.29 The results of this investigation are set out below:
‘Correcting Accounts for ‘lost’ discrepancies’ { POL-0010769} dated 29 September
2010 (referred to by Mr Godeseth)
PEAK/KEL
Date
Observations
PC0204765,
{POL-0374542}
26 September
2010
It is stated in the document above that this PEAK should
“record all affected branches”.
I cannot clearly see that there are 37 branches
referenced within this PEAK.
Refers to KEL wrightm33145J & ballantj1759Q
PC0204263
{POL-0374051}
13 September
2010
Records technical issue reported by branch (FAD) 002014
Refers to KELs chitikelaS1953M & ballantj1759Q
PC0203864
{POL-0373654}
02 September
2010
It is not clear to me which branch was affected in this
instance, the branch identifier is not represented by a
FAD code as with other PEAKs.
Refers to KELs wrightm33145), BrailsfordS130S &
wrightm33145J)
‘ReceiptsPayme
intsvO 4.doc’
POL-0215998} dated 16 May 2013 (relied upon by Dr
{POL-0374542}
Worden)
PEAK/KEL Date Observations
PC0204263 As above As above
{POL-0374051}
PC0204765 As above As above
5 {Second Witness Statement of Torstein Olav Godeseth, 16 November 2018}
Prepared by: Jason
Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
POL-BSFF-0100992_0017
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 14 of 265
3.30 In respect of the above bug, it is important to note that whilst Post
Office acknowledged that 62 branches were affected (of which 37 were
Subpostmasters), in Mr Godeseth’s responsive witness statement® at
paragraph 42 he records that this bug affected 60 branches.
3.31 Ihave not been able to identify all Subpostmasters affected by this bug,
nor to what extent they were affected due to the limitations above.
Namely:
a. Mr Godeseth and Post Office have different numbers of affected
branches;
b. I PEAKs and KELs referenced by Dr Worden and Mr Godeseth (or
within the documentation they rely upon) do not equate to 37
affected Subpostmasters nor their discrepancy figures.
3.32 Dr Worden has estimated “on statistical grounds” at paragraph 656 of
his report that the net quantitative impact of this bug was £20,000
across 62 of 11,000 branches. The only figures I have seen (aside of
those discrepancies documented within the three PEAKs above) are
contained in the document referenced by Mr Godeseth:”
“Of the cases so far identified there is one for 330,611.16, one for
£4,826.00 and the rest are all less than £350.
I’ve been unable to work out yet if these are losses or gains!”
3.33 Therefore, the true extent of this bug in my opinion is not fully
confirmed. I have reviewed the basis of Dr Worden’s estimates in
section 5 below, and particularly in subsection 8. In summary, Dr
Worden’s estimates are based on assumption which are inaccurate as a
matter of technical principle and as a matter of fact in relation to this
® {Second Witness Statement of Torstein Olav Godeseth, 16 November 2018}
7 3429 SM BP Correcting Accounts for Lost Discrepancies - 102000790 - CD1.pdf, Correcting
Accounts for "lost" Discrepancies, 29 September 2010 {POL-0010769}
Prepared by: Jason Coyne fe)
Occupation: Partner .
Specialist Field: IT Systems ; It lI
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0018
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 15 of 265
case. Therefore, in my opinion, it is very unlikely that a result on the
basis of those assumptions will be accurate.
Callendar Square / Falkirk Bug (Horizon Issue 1)
3.34 This defect is acknowledged by Post Office® as being discovered in 2005
(fixed March 2006) within Legacy Horizon. It is reported by Post Office
that the Falkirk anomaly came to the attention of Fujitsu when a
Subpostmaster in the Callendar Square branch highlighted the issue.
The symptoms of this defect result in stock transfers not being “seen”
by other counters within a branch, due to data communication errors in
Riposte. This leads to a discrepancy in the branch accounts since the
double entry principle of accounting is not applied.
3.35 Using the documentation referred to in the responsive witness
statement of Mr Godeseth and those documents identified by Dr
Worden, I have investigated the PEAKs and KELs they set out in relation
to this bug and whether the documents referred to capture full details
of all branches affected in the table below.
Documents Referred to by Mr Godeseth
PEAK/KEL Date Observations
PCO126042 15 September 2005 Branch 160868 (Note Callendar Square branch),
{POL-0296514} errors in Riposte causing a loss of £3489.69 to
be rectified by error notice.
No KEL referenced, PEAK detail states “unable
to find relevant kel”.
PC0126376 21 September 2005 Branch 160868 (Note this is the same one
{POL-0296843} referenced above, Callendar Square where the
issues was identified in 2005), Approx £45.40
loss, “another occurrence of last week’s
problem.”
References KELs Jballantyne5245K &
JSimpkins338Q
8 {Letter of Response from Post Office, SCHEDULE 6: REBUTTAL OF ALLEGATIONS AGAINST
HORIZON, 28 July 2016}
Prepared by: Jason Coyne
Occupation: Partner oi a
§ Itgrouy
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
POL-BSFF-0100992_0019
181024SR1935
01 February 2019
POL00262929
POL00262929
Page 16 of 265
JBallantyne5245K
{POL-0444056}
Original date 02
November 2000 -
revised 07 July 2005
Relates to Riposte error.
References Branch 334832, PEAK PC0083101
and KEL JSimpkins338Q
JSimpkins338Q
{POL-0444055}
Original date 10 May
2002 - revision date
11 January 2010
Relates to Riposte error notes:
“Feb 2003: Seeing a few of these each week”
refers to PEAK P0086212.
“June 2004: This event can give rise to transfer
errors” refers to PEAK PC0103864
“Sept 2005: This problem is still occurring every
week, in one case at the same site on 2
consecutive weeks, PC0126376 sent to
development.”
“Update: 11/01/2010 PEAK PC0193012”
Documents referred to by Dr Worden
{POL-0249574}
JBallantyne5245K I As above As above
{POL-0444056}
JSimpkins338Q As above As above
{POL-0444055}
PCO075892 02 May 2002 Branch (FAD) 312511, critical event raised.
No associated KEL
PCO083101
{POL-0256175}
27 October 2002
Branch (FAD) 323329, critical event raised.
References KELs JBallantyne5245k &
JBallantyne1359R
PcCo0s6212
{POL-0258908}
24 January 2003
Branch (FAD) 211801, recorded as no
discrepancy occurring.
No KEL referenced.
PCO103864
{POL-0275503}
03 June 2004
Branch (FAD) 281306 £22,290.00 discrepancy
References KEL JSimpkins338Q & CObeng2025L
{POL-0296843}
PCO126042 As above As above, Callendar Square branch.
{POL-0296514}
PC0126376 As above As above, note second occurrence at Callendar
Square branch.
PCO193012
{POL-0362963}
09 January 2010
PEAK documenting the need to stop and restart
the Riposte service
References KELs JSimpkins338Q &
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
CObeng2025L
Zitg roup
POL-BSFF-0100992_0020
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 17 of 265
3.36
3.37
3.38
3.39
3.40
3.41
It is important to note that whilst Post Office acknowledge this bug to
have been discovered in 2005, that appears to only pertain to the
particular incident at Callendar Square. In actuality, in my opinion, it is
likely this bug appears to have been in effect since year 2000 (this is
supported by Dr Worden at paragraph 660 of his report).
Mr Godeseth states in his second witness statement that in all, 30
branches were impacted by this bug. I have not been able to determine
which 30 branches were impacted as per the witness statement of Mr
Godeseth, nor the extent the discrepancies.
I have identified the following PEAK appears to be the incident
documented in KEL JSimpkins338Q:
“June 2004: This event can give rise to transfer errors”
PC0103864° created 3 June 2004 relates to an issue in which a
Subpostmaster incurs a discrepancy due to a stock transfer bug. The
PEAK detail states:
“Contacted Auditor John, explained that SSC discovered how the error
occurred and they passed details to POL so that an error notice can be
issued, Auditor wanted a contact no. for POL dept who issue error notices,
advised that we do not have a no. for them and that he should go through
NBSC. Auditor happy with information provided.”
However, the root cause of this PEAK is recorded as “General -
Unknown” (where other PEAKs identifying the same bug are recorded
as “Development - Code” [PC0116670'°]) and the Call Status is
recorded as “Closed -- Unpublished known error”.
Despite support acknowledging that this issue is a flaw in Riposte and
questioning whether it should be routed to Escher for a fix, there is no
detail provided as to whether this was, and the ‘Call Status’ does not
record a fix at future release. Therefore it is unclear how this bug was
° PEAK PC0103864, 3 June 2004 {POL-0275503}
10 PEAK PC0116670, 24 February 2005 {POL-0288202}
Prepared by: Jason Coyne fe)
Occupation: Partner 1
Specialist Field: IT Systems it }FOk I
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0021
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 18 of 265
3.42
resolved, despite its symptoms requiring error notices to be sent to the
branches to fix discrepancy.
In this instance, it illustrates that the “Callendar Square” bug was
operating and resident in the system (and had been for years already)
without any comprehensive linkage being observed by Fujitsu, since
various occurrences of it were subsequently being recorded under
differing KELs and PEAKs but were not identified as related.
Suspense Account Bug (Horizon Issue 1)
3.43
This bug is acknowledged by Post Office! as occurring from 2010 - 2013
within Horizon Online. Post Office sets out that this bug impacted 14
branches (4 crown and 10 Subpostmasters). In summary, the bug
caused suspense account figures from 2010 to be erroneously
reproduced in those branches’ suspense accounts for the same monthly
trading periods in 2011 and 2012. Post Office states that, despite the
Subpostmasters querying this in 2012, the cause of issue was not
identified until 2013.
‘Local Suspense Problem’ {POL-0444082} referred to _by Mr Godeseth and Dr
Worden.
PEAK/KEL Date Observations
This PEAK is nested within the 22 March PEAK detail illustrates the following branches
document referred to above 2013 were impacted:
and noe eeleny Oy Or Warten BRANCH AFFECTED AMOUNT
TRADING
PC0223870 PERIOD
{POL-0393383} 002647 9 -6.71
002840 9 140.61
010007 9 -0.01
011458 +10 -9,799.88
012004 9 16.12
054011 9 3.34
101832 9 5.84
104937. 9 -49.62
11 {Letter of Response from Post Office, SCHEDULE 6: REBUTTAL OF ALLEGATIONS AGAINST
HORIZON, 28 July 2016}
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
Prepared by: Jason Coyne
Occupation: Partner oi a
§ Itgrouy
POL-BSFF-0100992_0022
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 19 of 265
155025 9 -113.14
156715 9 11.55
211844 9 -41.77
243242 9 -0.51
266418 9 3,186.70
297611 9 160.92
References KEL acha2230K
acha2230K referred to by Stephen Paul Parker Witness Statement
acha2230K Raised 18 There is nothing to identify within the KEL
October text that this KEL relates particularly to the
{POL-0039585} 2013 last previously identified suspense account bug
updated 25 I aside from it references the PEAK
October PCO223870 (referenced in Mr Jenkins
2013 report).
3.44 Whilst Dr Worden records that 16 branches were affected, and this is
the number of branches shown in the table within the ‘local suspense
note’ produced by Gareth Jenkins), I have noticed that in another
document (the 2013 POA Problem Management — Problem Review dated
11 February 2015 {POL-0138981}) only 14 branches?? were referred to
as having been affected..
3.45 In summary, this bug could have impacted branches prior to the Fujitsu
investigation in 2012. Therefore, it is unlikely Post Office or Fujitsu have
captured its full effects across each year that it arose.
Branch Outreach Issue / ‘Dalmellington’ (Horizon Issue 1)
3.46
This bug relates to the issue which arises when trying to transfer funds
to outreach branches. Not previously acknowledged by Post Office, but
now referred to in the Witness Statements of Mr Godeseth and Mr Parker
in relation to KEL acha621P?3 (identified in my first report, paragraph
5.23 (page 47) and in the exhibit!* referred to by Mr Godeseth provided
12 2013 POA Problem Management - Problem Review. {POL-0138981}
13 KEL Acha621P, 15 October 2015 last updated 14 January 2016 {POL-0040340}
14 _DOC_152848834(1) Outreach BLE Extract Findings v6 091215.pdf, Branch Outreach Issue
(Initial Findings), 10 December 2015 {POL-0444078}
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
POL-BSFF-0100992_0023
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 20 of 265
3.47
as part of the responsive witness evidence (see also unredacted
version?>)).
‘Dalmellington’ is the name of the branch which reported the issue in
2015.
fern
C) [i
PEAK/KEL Date Observations
acha621P Rasied 15 KEL text refers to the issue as identified
{POL-0040340} October 2015 I for Dalmellington branch
last updated
14 January
2016 References PC0246949
3.48
3.49
3.50
3.51
Mr Godeseth states in relation to the Dalmellington bug that he
understands from Gareth Jenkins (no document is referenced in relation
to how) that the problem resided in branch to branch remittances.
Whilst cash pouches were seen to be going out once from the core
branch, they could effectively be accepted multiple times at the other
side of the transaction.
Mr Godeseth refers to Exhibit TOG2 pages 13 to 27. Upon review of this
document (and as acknowledged by Mr Godeseth) it identifies that there
are two potentially separate issues at play within this bug. In total,
initial findings of an audit found 112 occurrences of duplicate pouch IDs
affecting 88 branches over a five-year period (some branches impacted
up to five separate times).
In four instances, the document above records that correction was “still
to be confirmed”. Therefore, it is not clearly determined whether those
Subpostmasters bore the financial cost. The range of the impact on
branch accounts was between £0.01 and £25,000.
In summary, it is not clear whether the branch outreach investigation
progressed further than “initial findings”, nor is it clear how, in the four
15 Outreach BLE Extract Findings v6 091215.pptx, Branch Outreach Issue (Initial Findings), 10
December 2015 {POL-0220141}
Prepared by: Jason Coyne fe)
Occupation: Partner ae
Specialist Field: IT Systems It Jrou f )
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0024
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 21 of 265
3.52
cases still to be confirmed, Subpostmasters might have been
recompensed for a Horizon generated issue.
Mr Parker refers to Dalmellington is respect of KEL acha621P because it
was identified in my first report. He states that Post Office issued
Transaction Corrections or advised Subpostmasters how to take
corrective actions to remove the discrepancies. He does not relate any
further information as Mr Godeseth is responding to the KEL. It is
important to note that the KEL referenced does not identify the full
impact of this bug (one has to know that the Branch Outreach document
is related in some way to gain a fuller understanding of the impact).
d to. EI]
PEAK/KEL Date Observations
acha621P Raisedd 15 KEL text refers to the issue as identified
{POL-0040340} October 2015 I for Dalmellington branch
last updated
14 January
2016 References PC0246949
3.53
3.54
Similarly, it is important to note that Dr Worden does not address the
Dalmellington bug, but states in relation to the KEL (in response to my
report paragraph 5.130):
“All remming errors produce a discrepancy between physical cash and
Horizon cash, which gets corrected at monthly balancing or before (UEC)
So no impact on branch accounts. Mr Coyne comments about correcting
branch accounts are therefore inappropriate.”
In summary, I do not agree with Dr Worden since it is evident from the
documentation produced by Fujitsu/Post Office that impact for four
branches is still to be confirmed and in consideration that this bug
operated for five years without detection.
Prepared by: Jason Coyne fe)
Occupation: Partner <1 .
Specialist Field: IT Systems It Jrou f )
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0025
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 22 of 265
Dr Worden KELs with further PEAK Analysis
3.55 In this section I have looked at some of the KELs Dr Worden has listed
in his report in the context of further detail discovered from my review
of the PEAKs.
‘Remming In’ (Horizon Issue 1)
3.56 PC0203085'° dated 17 August 2010 references KEL acha4221Q’” and
documents a bug that allows a user to “rem in” a cash pouch on two
different counters, subsequently resulting in a loss for the
Subpostmaster as this should only occur successfully once:
“A cash pouch was remmed in twice at branch 126109:
Pouch barcode 399347067204
2p coin £60
50p coin £250
5p coin £100
Session 1-350379 16/09/2010 10:08
Session 2-195226 16/09/2010 10:08
The PM cannot reverse the transaction since rem reversal isn't allowed.
This is NOT another example of the duplicate rem problem that we have
ssn [sic] in the past, where use of the Prev key accepted the same pouch
twice. In this case the pouch was processed on both counters...
09:05 c2 get pouch status, retrieve pouch details
09:06 c1 get pouch status, retrieve pouch details
09:08 c2 settle pouch delivery
09:08 c1 settle pouch delivery
There were some printer problems on counter 2 which probably explain
why this was done.
16 PEAKPCO203085, 17 August 2010 {POL-0372879}
17 KEL acha4221Q, 2 March 2010 last updated 3 May 2011 {POL-0038476}
Prepared by: Jason Coyne
Occupation: Partner .
Specialist Field: IT Systems itq rou f )
On the Instructions of: Freeths LLP < =
POL-BSFF-0100992_0026
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 23 of 265
3.57
3.58
3.59
3.60
3.61
3.62
Please send this info to POL via BIMS, because the branch now has a
shortage of £410 as a result of this double rem in, and will need a
correction. Then return the call to me and I'll get development to check
whether it is working as intended.”
It is stated further within this PEAK:
“Gareth Jenkins thinks that it should not be possible to complete the rem
in on both counters. Please investigate.”
However, as the investigation continued, a likely cause was established
and fixed around 23 January 2011 (some ten months after the
referenced KEL was raised) concluding that indeed, a cash pouch could
be ‘remmed in’ twice, erroneously.
This bug was only brought to the attention of Fujitsu/Post Office
following notification from the Subpostmaster. It is also my
understanding that this is potentially a different manifestation of the
Dalmellington bug.
The related KEL to this PEAK, acha4221Q,'8 is dealt with by Mr Parker
in his first witness statement Appendix 2!° and Dr Worden at table D4
of his Appendix D. Both Dr Worden and Mr Parker state that the impact
of this defect (in relation to KEL acha4221Q) led to an £80.00 shortfall
that was dealt with via a Transaction Correction and was subsequently
fixed 19 April 2010. However, Dr Worden refers to PEAK PC01953807°
in relation to this KEL, which relates to a differing manifestation
resulting from the same core bug. In this case the PREV key is pressed
causing the same discrepancy as I outlined above (PC02030857').
Note that PCO203085 arose in August 2010 and PC0195380”? April
2010.
This displays that Dr Worden and Mr Parker have not considered this in
its entirety, since further manifestations record a cumulative shortfall
18 KEL acha4221Q, 2 March 2010 last updated 3 May 2011 {POL-0038476}
19 {Witness Statement of Stephen Paul Parker, 16 November 2018}
20 PEAK PC0195380, 2 March 2010 {POL-0365285}
21 PEAK PC0203085, 17 August 2010 {POL-0372879}
22 PEAK PC0195380, 2 March 2010 {POL-0365285}
Prepared by: Jason Coyne fe)
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0027
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 24 of 265
value higher than the one £80.00 example they identified. Also, Mr
Parker records that the incident was fixed on 19 April 2010 yet there is
an associated PEAK to this KEL raised 23 April 2010 (and August 2010)
that identifies a further discrepancy. Note there may be other PEAKs in
relation to this bug that may reference a differing associated KEL (since
application of the KEL reference is dependent on the support member
applying the relevant one from the database).
3.63 It has not been possible in the time available to investigate and analyse
every single PEAK that is possibly related to KEL acha4221Q.73 However,
upon a preliminary search, I have identified the following potentially
relevant PEAKs as they contain reference to “acha4221Q” and noted
similar preliminary observations.
acha4221Q raised 02 March 2010 ~ last updated 03 May 2011.
PEAK (explicitly documented in the KEL document) PC019538024
Reference PEAK Key: Master PEAK (MP); Release Peak (RP); Other Peak (OP)
PEAK Date Created Reference PEAK I Observations
PC0195380 02 March 2010 I (RP) PCO195911 £80.00 shortfall - Branch 506246
{POL-0365285} {POL-0365808}
Pc0195511 03 March 2010 I (OP) PC0195380 £25,000 shortfall - Branch
{POL-0365416} {POL-0365285} 069002
PCO196120 17 March 2010 I N/A £500.00 shortfall - Branch
{POL-0366013} 109013
PCO196154 18 March 2010 I (OP) PCO195380 £2104.02 shortfall - Branch
{POL-0366046} {POL-0365285} 506246 (further incident see
PCO195380 above)
PCO196671 29 March 2010 N/A £5500 shortfall - Branch 003937
{POL-0366555}
PC0197032 31 March 2010 N/A £25,000 shortfall - Branch
{POL-0366915} 013004
PC0197034 31 March 2010 I N/A No value documented - Branch
{POL-0366917} 214405
23 KEL acha4221Q, 2 March 2010 last updated 3 May 2011 {POL-0038476}
24 PEAK PC0195380, 2 March 2010 {POL-0365285}
Prepared by: Jason Coyne
Occupation: Partner : .
Specialist Field: IT Systems itg rou )
On the Instructions of: Freeths LLP =
POL-BSFF-0100992_0028
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 25 of 265
PCO197605, 12 April 2010 N/A Not Horizon Error - Branch
{POL-0367480} 135002
PCO197651 13 April 2010 N/A £22,000 shortfall - branch
{POL-0367524} 159713
PC0197753 15 April 2010 N/A £120.00 shortfall - Branch
{POL-0367623} 060925
PC0197828 16 April 2010 N/A £1680 shortfall - Branch 436217
{POL-0367697}
PC0197837 16 April 2010 N/A £5500 shortfall - Branch 225207
{POL-0367706}
PC0197838 16 April 2010 N/A £7000 shortfall - Branch 005207
{POL-0367707}
PCO197872 19 April 2010 N/A £1500 shortfall - Branch 187246
{POL-0367741}
PC0197873 19 April 2010 N/A £144.87 shortfall - Branch
{POL-0367742} 294306
PC0198115 23 April 2010 N/A £13,000 shortfall - Branch
{POL-0367979} 183323
PCO203085 17 August 2010 I (RP) PCO207466 £410.00 shortfall - Branch
{POL-0372879} {POL-0377202} 126109
(OP) PCO19151
PCO226230 07 June 2013 N/A PEAK states “There appears to be
{POL-0395717} TWO different problems described
in this call and the details are not
very clear. Please raise separate
calls supplying full details of the
problem/s.”
Ticket is closed.
PCO246629 25 September N/A Not Horizon Error
{POL-0415562} 2015
PCO251952 14 June 2016 N/A Refers to remming out unusable
{POL-0420451} notes - insufficient further detail.
3.64 The Release PEAK in relation to the fix for PC019538075> (referenced by
Dr Worden and Mr Parker in relation to KEL acha4221Q) does not
document every PEAK that would be impacted by the fix or reference
that the fix specifically applied to KEL acha4221Q. It also does not
25 PEAK PC0195380, 2 March 2010 {POL-0365285}
Prepared by: Jason Coyne
Occupation: Partner .
Specialist Field: IT Systems itg rou )
On the Instructions of: Freeths LLP
POL-BSFF-0100992_0029
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 26 of 265
3.65
3.66
record whether it was fully rolled out across the estate as at 19 April
2010 (the date given by Mr Parker in his witness statement).
The Release PEAK in relation to the fix for PC02030857° (alternative
PEAK related by reference to KEL acha4221Q) in the table above is
documented as PC0207466”’ created 4 January 2011.
Therefore, it is not clear from review of the KEL reference alone, what
the full impact of the bug/error/defect referred to within the KEL was.
Nor can the fix be clearly identified in relation to it, since different
manifestations of a bug would be linked by a KEL reference (and may
not just be limited to one KEL reference) which, in turn, had several
fixes applied.
‘Remming Out’ (Horizon Issue 1)
3.67
3.68
3.69
PC0143435*° created 12 Feb 2007 relates to issues when remitting out
coins. Its associated KEL is documented as acha508S*° which I note has
been referred to in Mr Parker’s witness statement Appendix 27° and
within table D5 of Dr Worden’s Appendix D.
Neither Mr Parker nor Dr Worden appear to have performed any analysis
in relation to the PEAKs associated with KEL acha508S*" (aside from Dr
Worden referencing PC0143435*? which is documented within the KEL).
Whilst Dr Worden documents a single impact of £1,500 in relation to
this KEL, he has not considered the PEAKs below.
Further, Mr Parker has not provided any substantial analysis in relation
to this KEL but states that the issue was fixed in April 2007 and fully
rolled out by June 2007.
26 PEAK PC0203085, 17 August 2010 {POL-0372879}
27 PEAK PC0207466, 4 January 2011 {POL-0377202}
28 PEAK PC0143435, 12 February 2007 {POL-0313783}
29 KEL acha508S, 12 February 2007 last updated 15 February 2007 {POL-0035513}
30 {Witness Statement of Stephen Paul Parker, 16 November 2018}
31 KEL acha508S, 12 February 2007 last updated 15 February 2007 {POL-0035513}
32 PEAK PC0143435, 12 February 2007 {POL-0313783}
Prepared by: Jason Coyne fe)
Occupation: Partner ae
Specialist Field: IT Systems It Jrou f )
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0030
181024SR1935
01 February 2019
POL00262929
POL00262929
Page 27 of 265
3.70 The table below identifies PEAKs that reference KEL acha508S. Note, it
is entirely possible that other PEAKs may exist in relation to this problem
but have been assigned a different KEL reference.
Acha508S raised 12 February 2007 ~ last updated 15 February 2007.
PEAK (explicitly documented in the KEL document) PC0143435
Reference PEAK Key: Master PEAK (MP); Release Peak (RP); Other Peak (OP)
(RP) PCO140826
{POL-0311184}
(OP) PCO140281
{POL-0310641}
(OP) Pco14i892
{POL-0312248}
(OP) PCO142116
{POL-0312469}
(OP) PCO143494
{POL-0313842}
PEAK Date Created Referenced PEAK I Observations
PCO143435 12 February (RP) PC0140829 £500.00 shortfall - Branch
{POL-0313783} I 2007 {POL-0311187} 175113
PC0143440
{POL-0313788}
12 February
2007
(OP) PC0141892
{POL-0312248}
£352.60 balancing error -Branch
020323
PCO143466
{POL-0313814}
12 February
2007
N/A
£466.60 shortfall - Branch
455329
PC0143499
{POL-0313847}
13 February
2007
(OP) PCO143439
{POL-0313787}
-£5.70 incomplete summaries
report - Branch 305201
PCO143500
{POL-0313848}
13 February
2007
(OP) PCO143435
{POL-0313783}
No values documented -
Incomplete summaries report
affecting multiple branches
054946, 080025, 085109,
086939, 094005 & 095131
PCO0143501 13 February (OP) PCO143502 No values documented -
{POL-0313849} I 2007 {POL-0313850} Incomplete summaries report
affecting multiple branches
108006, 111840, 122014,
134912, 152406
PCO143502 13 February (OP) PCO0143435 Seems to be duplicate of
{POL-0313850} I 2007 {POL-0313783} PCO14350133
33 PEAK PC0143501, 13 February 2007 {POL-0313849}
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
itg rou
POL-BSFF-0100992_0031
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 28 of 265
PC0143503 13 February N/A No values documented -
{POL-0313851} I 2007 Incomplete summaries report
affecting multiple branches
162820, 173519, 175113,
175844, 176704
PC0143504 13 February N/A No values documented -
{POL-0313852} I 2007 Incomplete summaries report
affecting multiple branches
178343, 179546, 180114,
180546, 181523
PCO143506 13 February (OP) PCO143515 No values documented -
{POL-0313854} I 2007 {POL-0313863} Incomplete summaries report
affecting multiple branches
191504, 205539, 223939,
227555, 235201
PC0143507
{POL-0313855}
13 February
2007
(OP) PCO143515
{POL-0313863}
No values documented -
Incomplete summaries report
affecting multiple branches
249208, 257546, 266641,
272504, 274207
PCO143508
{POL-0313856}
13 February
2007
N/A
No values documented -
Incomplete summaries report
affecting multiple branches
283230, 293340, 301321,
305613, 310519
PC0143511 13 February (OP) PCO143435 No values documented -
{POL-0313859} I 2007 {POL-0313783} Incomplete summaries report
affecting multiple branches
317246, 329642, 348201,
361420, 367642
PC0143513 13 February N/A No values documented -
{POL-0313861} I 2007 Incomplete summaries report
affecting multiple branches
373311, 377136, 421136,
448420, 500227
PC0143514 13 February N/A £500 shortfall - Branch 235201
{POL-0313862} I 2007
PC0143515 13 February N/A £500 shortfall - Branch 205539
{POL-0313863} I 2007
PC0143539 14 February N/A No values documented -
{POL-0313887} I 2007 Incomplete summaries report
affecting multiple branch 156205
PCO143682 19 February N/A £515 shortfall - Branch 140946
{POL-0314029} I 2007
PCO0143839 23 February N/A No values documented - Shortfall
{POL-0314186} I 2007 - Branch 293340
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
Zitg roup
POL-BSFF-0100992_0032
181024SR1935 01 February 2019 Page 29 of 265
PC0144933 02 April 2007 (OP)PC0144937 £3000 shortfall - Branch 251632
{POL-0315272} {POL-0315276}
PC0144937 02 April 2007 (OP)PC0144933 £3000 shortfall - Branch 251632
{POL-0315276} {POL-0315272}
3.71 Having identified the related PEAKs above, and the observations within
3.72
3.73
3.74
them (including the values, where recorded) in my opinion Mr Parker
and Dr Worden have failed to consider the full effect of this issue.
I note that Dr Worden states that since remming issues will always be
visible to the Subpostmaster, they will always be reported and
investigated and correctly resolved. In my opinion, it is not correct to
make such broad assumptions. As with the Dalmellington issue above,
it is entirely possible that not all Subpostmasters would have the ability
to diagnose a Horizon generated error as the reason for discrepancy or
be able to pursue it to ensure that it is correctly dealt with. Therefore,
some Subpostmasters would risk bearingthe cost of the discrepancy.
PC0120937%4 created 13 May 2005 (referenced KEL GMaxwell3853P)
records an instance where a Subpostmaster incurs a branch shortfall
due to a lack of system control preventing the input error in relation to
remming out coins. The issue arises from functionality that should not
be available (but is) when the Horizon system is under load. The PEAK
detail records:
“Weighing up the cost and risk of an attempted fix against the fact that
this has only been reported once, I do not believe that we should make
a code fix. If further incidents of this problem are reported we can review
this decision. Gary has raised a KEL, so returning for closure as "Published
Known Error".
Subsequently, it is decided that KEL GMaxwell3853P?5 is to be used as:
“Given the frequency of the problem & the apparent risk involved in
introducing a code fix the KEL should be adequate.”
POL00262929
POL00262929
34 PEAK PCO0120937, 13 March 2005 {POL-0291445}
35 KEL GMaxwell3853P, 17 May 2005 last updated 15 June 2005 {POL-0034666}
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
POL-BSFF-0100992_0033
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 30 of 265
3.75
3.76
3.77
The call record is closed with the root cause documented as “General —
Unknown” despite the PEAK detail recording how the issue could occur.
This incident caused the value that could not be remmed out to be
written to the Subpostmaster’s suspense account and is an example of
another Horizon generated discrepancy, on a similar theme to the
remming out issues above, but a slightly different manifestation and
different associated PEAK.
These related PEAKs illustrate how a bug that manifests in slightly
different ways can be analysed and diagnosed differently amongst the
varying technical support members. Different KELs appear to be applied
to various PEAK records which results in potentially different advice
given to the Subpostmasters in each occurrence of such a bug. The fact
that fixes are applied across many releases of Horizon and yet
Subpostmasters still encounter issues is indicative of the differing
software versions in action across the estate. This lack of versioning
consistency results in Fujitsu repeatedly dealing with errors which are
known to be in existence.
Local Suspense Issue (Horizon Issue 1)
3.78
KEL acha5259Q** raised 22 April 2010 last updated 30 April 2010 is
referred to by both Mr Parker in Appendix 2 attached to his witness
statement?” and Dr Worden within table D5 of Appendix D to his expert
report. This KEL relates to a local suspense issue that affected the cash
figure on the balance report causing a discrepancy in the new trading
period. Note that it is not relative to the Suspense Account bug above
(or least not identified within the documentation as being so). Mr Parker
states that this only appeared to affect branches balancing in April 2010
and 33 identified branches were impacted; it was resolved in July 2010.
Dr Worden identifies (in association with this KEL), PC0198077,
3€ KEL acha5259Q, 22 April 2010 last updated 30 April 2010 {POL-0037436}
3? {Witness Statement of Stephen Paul Parker, 16 November 2018}
Prepared by: Jason Coyne fe)
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0034
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 31 of 265
PC0197409, PC0197797 and PC0204396, and records that it was fixed
in September 2010.
3.79 It is important to note that the KEL states "The reason for the exception
is understood (PC019740978 / KEL PorterS199P°°).”
3.80 This exemplifies two KELs each with differing associated PEAKs relative
to the same bug, error or defect. It is interesting to also note that in
one of the examples below (PC0194709) the call was first logged 17
February 2010 and yet even by 5 March 2010 the detail of the PEAK
suggests that Post Office/Fujitsu did not know who should be
investigating this type of issue. This demonstrates the complex
relationship between PEAKs and KELS.
3.81 My observations and findings in relation to the above KELs are as
follows:
PEAKs referenced BY PEAKs PEAKs THAT PEAKs THAT
KEL acha5259Q*° referenced BY reference KEL reference KEL
KEL acha5259Q PorterS199P41
PorterS199P
(received in
latest “deleted
KEL”
disclosure)
Pc0197409 PC0197409 PC0197409
{POL-0367287} {POL-0367287}
PC0198077 PC0198077
{POL-0367945} {POL-0367945}
PCO197797 PC0198066 PC0194709
{POL-0367666} {POL-0367934}
PC0198677 PC0197800
{POL-0368532}
PC0198678
38 PEAK PC0197409, 7 April 2010, {POL-0367287}
39 KEL PorterS199P, 18 April 2010 last updated 21 April 2010 {POL-0448589}
40 KEL acha5259Q, 22 April 2010 last updated 30 April 2010 {POL-0037436}
41 KEL PorterS199P, 18 April 2010 last updated 21 April 2010 {POL-0448589}
Prepared by: Jason Coyne
Occupation: Partner oi a .
3 itgroup
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
POL-BSFF-0100992_0035
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 32 of 265
{POL-0368533)
PCO198259
{POL-0441040}
PC0204396
{POL-0441123}
3.82 I note that following Dr Worden’s analysis of the KEL, he states:
“Visible to branch, and Fujitsu seem to have known about all instances.”
3.83 In his report, Dr Worden refers to the spreadsheet attached to
PC0197797*. However, the PEAK disclosure provided to me did not
include any attached or embedded documents that the PEAKs refer to.
It is therefore not clear how Dr Worden has satisfied himself of this,
either:
a. Dr Worden has been able to review the attachments and
embedded documents and is satisfied it captures all the branches
affected as per the PEAK, and is satisfied the PEAK detail is
accurate; or
b. Dr Worden is just re-stating the position as per the text within the
documentation.
Recovery Issues (Horizon Issue 1)
3.84 PC0197769% created 15 April 2010 refers to a problem with recovery
whereby the wrong Trading or Balancing Period may be updated. It’s
associated KEL is acha5650L** (raised 26 April 2010 last updated 17
December 2012).
3.85 I note that Dr Worden and Mr Parker both comment on this issue. Dr
Worden states within table D4 of Appendix D to his report that “no
financial impact” would be incurred from this issue. Further, in Mr
Parker’s analysis of the KELs (Appendix 2 of this responsive witness
statement),*° he maintains that since this issue would result in two
42 PEAK PCO0197797, 15 April 2010 {POL-0367666}
43 PEAK PC0197769, 15 April 2010 {POL-0367639}
44 KEL acha5650L, 26 April 2010 last raised 17 December 2012 {POL-0039245}
45 {Witness Statement of Stephen Paul Parker, 16 November 2018}
Prepared by: Jason Coyne fe)
Occupation: Partner ae
Specialist Field: IT Systems It ia Uf )
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0036
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 33 of 265
discrepancies cancelling each other out, there was no long-term impact
on branches. I disagree for the following reasons:
a. The solution for KEL acha5650L states:
“..was the transaction ever included in a balance (i.e. did the stock unit
subsequently roll into the TP/BP that the transactions were written into?).
If so, raise a BIMS to say that the problem will have caused a discrepancy
in period xx/xx but an equal but opposite discrepancy in period yy/yy, so
overall there is no effect on the branch accounts. If the transactions were
written into a period that had already been balanced (e.g. 1/1 but stock
unit was already in 1/2), or a balance period that did not exist for the
stock unit, raise a BIMS to say that the recovered transaction has not
been included in the branch accounts and will have caused a discrepancy.
However the data has been sent to POLMI / POLFS (because that ignores
the TP/BP information).
Therefore the position “No Impact” is not correct.
b. There are a further 23 associated PEAKs (arising from a search of
“acha5650L") of which I do not believe Dr Worden has analysed in
full. Of these PEAKs, by randomly selecting one, I identified that
PEAK PC0198352*° created 2 May 2010 resulted in a discrepancy.
The PEAK detail states:
“Recovered txn written to TP 12 BP 1, but the stock unit was in TP 12 BP
2. This caused a discrepancy of £380.00 for EE in TP 12 BP 2. Please
inform POL. This problem caused a loss at the branch for which they
should not be liable.”
3.86 It is my opinion that with additional research, further financial
discrepancies would be likely in respect of this same KEL issue.
3.87 PC0O256566 created 17 January 2017 refers to a reconciliation incident
whereby the Subpostmaster processed a transaction that did not appear
on the transaction log. The settlement failed due to poor
48 PEAK PC0198352, 29 April 2010 {POL-0368212}
Prepared by: Jason Coyne fe)
Occupation: Partner ae
Specialist Field: IT Systems It }roOu f )
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0037
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 34 of 265
3.88
3.89
3.90
3.91
communications with the datacentre and then the recovery procedure
also failed. The PEAK further states:
“This recovery failure was reported and investigated by us. Please see
Peak call PCO256502 which was closed on 16/01/17 after supplying the
necessary reconciliation information to POL. We have informed POL about
this recovery failure and also advised them to do the necessary
reconciliation for this sum of cash (Cash withdrawal for £244). We have
no way of knowing the internal POL process as to when they will do the
reconciliation if not done already.”
PCO256502 (The PEAK referred to in the quote above) created 16
January 2017 acknowledges the discrepancy in relation to branch
197327 above. However, it states, (in terms of impact):
“No impact. PEAK raised for the investigation of transaction(s) in a state
other than Final as showing in daily Reconciliation report”.
This is despite further referring to the need for manual reconciliation. It
therefore appears that the initial issue for the branch was logged under
this PEAK as requiring manual reconciliation, to be passed back to Post
Office but in the meantime a second call is generated by the
Subpostmaster due to limited information regarding the discrepancy.
The resolution of this incident is not recorded within the PEAK detail as
this would ultimately be down to Post Office to issue a Transaction
Correction, whether they did or did not has been deemed out of scope
by Post Office.
Further investigation of the matter above documents that KEL
acha959T*” applies (referenced by the PEAK above). This KEL was
referenced within my first report in relation to failed recoveries or an
incomplete transaction awaiting recovery at paragraph 5.43 (page 53).
It has since been responded to by Mr Parker (Appendix 2) whom states
(in relation to the KEL acha959T):
47 KEL acha959T, 28 February 2010 last updated 19 October 2017 {POL-0041091}
Prepared by: Jason Coyne
Occupation: Partner .
Specialist Field: IT Systems I
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0038
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 35 of 265
“There would be no impact if the user followed the recovery process
presented by Horizon”.
3.92 However, PC0256566 illustrates the opposite, Fujitsu identify that since
the ‘AUTHORISED’ receipt was printed in relation to the transaction, the
Subpostmaster should have handed the money over, but then the fact
that the transaction was lost from the transaction log would have meant
they had a cash discrepancy as no value was recorded to balance the
cash out.
3.93 Further, Dr Worden states in his analysis of the KELs at Appendix D
table D3 “This is another complex KEL...” as the extent of his analysis.
3.94 I have the following observations:
KEL acha959T raised 28 February 2010 last updated 19 October 2017
“This means a transaction is recorded on TES and/or at the FI, but the transaction has not
been completed properly at the branch/in the BRDB.”
“Possible Causes:
a) recovery has failed
b) transaction not completed, awaiting recovery
c) transaction was declined by pinpad but not reversed. See KEL cardc219R
d) PM pressed Cancel on counter moments after customer had entered PIN. See KEL
dsed4010Ne) only the reversal info reached the data centre”
Note - TES is Transaction Enquiry Service and FI is Financial Institution
References other DATE Observations
KELs/PEAKs:
KEL cardc219R Raised 11 May This problem relates to transactions that
(refers to PCO210052) 2011 - last aren't reversed in the end account
updated 31 despite being declined by the pinpad (14
October 2013 associated PEAKs).
Dr Worden states of this KEL “...later reconciliation and a TC would correct any error in
branch accounts.”
dsed4010N Raised 28 January I This problem relates to discrepancies
(refers to PC0223229) 2013 - last arising due to cancelling a pinpad
updated 20 transaction at the same time it is trying
November 2015 to seek authorisation (1 other associated
PEAK).
Other PEAKs and KELs that reference ‘acha959T’
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
gi
group
POL-BSFF-0100992_0039
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 36 of 265
2,473 different associated I 2010 - 2018 Utilising ‘acha959T’ as the keyword for
PEAKs search in the supplemental PEAK
disclosure.
KEL dsed2640M Raised 01 March Failed recovery due to printer issues
(references PC0193463) 2010 - updated 01 I rendering recovery unable to print a
March 2010 receipt due to a backlog.
214 PEAKs reference this KEL.
cardc464Q Raised 30 April Failed Recovery report entry where the
2010 last updated banking transaction does not appear on
12 January 2011 the TES or the DRS.
Note: DRS = Data Reconcilaiton Service
and TES = Transaction Enquiry Service
326 associated PEAKs
Dr Worden states (table D3 Appendix D) “Normally, any failure to recover a transaction
results eventually in a Transaction Correction which corrects any error in branch.”
Mr Parker states: “... In this case there was no impact on branch account.”
3.95
3.96
3.97
3.98
In relation to Dr Worden and Mr Parker’s comments above regarding
KEL cardc464Q, I have randomly selected one of the associated PEAKs
identified related to this KEL and have noted the following.
The PEAK is recorded as “No Impact”, however the sentiment of this is
unclear. This PEAK, and another referenced within it (PCO264632) relate
to issues where customer transactions are part processed) but the
transactions are not recorded in the Branch Database (BRDB) or on the
Counter.
There would therefore be no ability to check the true status of the
transaction and end customers could be either charged for something
they have not received or receive something they have not been
charged for. This would leave the Subpostmaster with a potential loss
or gain dependent upon the transaction and method of payment.
In conclusion, there are various associated manifestations of recovery
issues. Varying KELs recording varying symptoms. In my opinion, it is
too broad an assumption to make (as done by Dr Worden and Mr Parker)
that Subpostmasters would not bear any financial cost due to these
Horizon generated issues since the actions of any potential recompense
by Post Office has not been provided as part of disclosure.
Specialist Field: IT Systems
Prepared by: Jason Coyne c
Occupation: Partner oi a .
3 itgroup
On the Instructions of: Freeths LLP
POL-BSFF-0100992_0040
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 37 of 265
Reversals
3.99 PC0089918**created 25 April 2003 is a significant PEAK in which
reversed “rems” are doubling. Rems are remittances and, in this
instance, the Subpostmaster was trying to reverse a rem (effectively
‘undoing’ the transaction). However, instead of reversing the
transaction to balance off the previous input, the transaction value
doubled in the accounts. The PEAK detail states:
“I have looked at the messagestore and can see that the SaleValue (and
Qty) have been incorrectly calculated by the system...
.. Routing to MSU. Can MSU please liaise with NBSC over this software
issue. The
Post Office is going to have to balance with a large discrepancy 2 x
(£5,000 + £8,910) = £27,820. I have spoken to the PM and said that I
would arrange for someone from NBSC to talk to her ASAP. When the
reconciliation issues have been put in train can you please route the call
back to me so that I can send it on to development for a code fix.”
3.100 The PEAK goes on to detail that the problem is to do with the fix
introduced for PC0083954*?, further stating:
“Major problem with S30 Cash Account. POL will be aware because error
notices for CA will need to be generated. More sites with this problem are
coming out of the woodwork as cash account day approaches.”
3.101 It is unclear whether Post Office notified further branches which were
operating on the S30 release of the software about this discrepancy or
it was just left until a discrepancy was identified and an error notice
subsequently issued.
3.102 However, it is clear from the introduction of these bugs that regression
testing was not adequately performed when fixes had to be rolled out
to fix other bugs.
48 PEAK PC0089918, 25 April 2003 {POL-0262279}
49 PEAK PC0083954, 29 November 2002 {POL-0256833}
Prepared by: Jason Coyne fe)
Occupation: Partner ae
Specialist Field: IT Systems It Jrou f )
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0041
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 38 of 265
3.103 The KEL referenced within this PEAK is PSteed2847N. Dr Worden and
Mr Parker have provided analysis in relation to the above KEL. Mr Parker
states that this caused only a temporary financial impact as the incident
was visible to the Subpostmaster and was corrected by Post Office
issuing an error notice. Dr Worden similarly states there would be no
adverse effect on the branch accounts.
3.104 I have not been able to confirm that the Subpostmaster was issued an
error notice to correct the imbalance as this low-level detail in relation
to specific discrepancies has not been disclosed.
Horizon Issue 1 PEAKS
3.105 The PEAKs analysed below are a small portion of the PEAKs I have
identified as causing financial discrepancy in branch accounts outside of
those bugs acknowledged by Post Office. It should be noted there are
potentially thousands more PEAKs that illustrate financial discrepancy
arising in branch accounts, this is only a small selected sample from
keyword searched PEAKs.
Data Tree Build Failure Discrepancies (Horizon Issue 1)
3.106 PC0033128°° (no KEL referenced) created 10 November 1999
documents an issue where the Dugannon branch suffered a £43,000
discrepancy but the cause was not immediately known. It is
documented that the Branch Manager and Post Office agreed to amend
the week 32 cash account figures manually in order to work around the
issue. Note that this PEAK does not reference an associated KEL.
Therefore, no analysis has been provided on it by Dr Worden or Mr
Parker.
3.107 The PEAK detail further records of other branches that appear to be
affected by the same bug with varying degrees of shortfall: £52,814.29
at the Yate Sodbury Branch and £9,368.40 at the Appleby
Westmoreland branch.
5° PEAK PC00331278, 10 November 1999 {POL-0221887}
Prepared by: Jason Coyne fe)
Occupation: Partner .
Specialist Field: IT Systems ; It lI
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0042
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 39 of 265
3.108 The root cause is eventually diagnosed as the PEAK detail states:
“Data trees have been failing to build fully, and the system has not been
detecting this. Consequently, discrepancies in the balancing have been
occurring. In the case of Dungannon a whole Payments node was missing.
There have been a number of calls relating to this kind of issue.”
3.109 It is not clear whether the specific references within the detail of this
PEAK capture the records for the entire “number of calls” referred to or
if there were further incidents additional to those. I have identified the
following PEAKs as likely related to this bug and provided preliminary
observations in the table below.
PC0033128 dated 10 November 1999 {POL-0221887}
Reference PEAK Key: Master PEAK (MP); Release Peak (RP); Other Peak (OP)
PEAK Date Created Referenced PEAK I Observations
PC0033128 10 November 1999 I (OP) PCO032801°! £43,000 discrepancy —
{POL-0221887} Branch/Customer Ref:
(OP) PCOO45847 I Bsm19991110001 -
{POL-0223066}
(OP) PCO043811
{POL-0222670}
Dugannon Branch
PC0046811 06 June 2000 (OP) PC003863152 I (£37.80 discrepancy)
{POL-0223228} Branch ref unknown
PCO055964 17 October 2000 (OP) PCO0038631 Discrepancy value and
{POL-01230806} branch unknown
PCO058161 20 November 2000 I (OP) PC0059497 £3236 discrepancy —
{POL-0232985} {POL-0232986} Branch 145004
3.110 PC0132133>? created 10 February 2006 (referenced KEL
MSCardifield2219S°%*) relates to a defect that is summarised as:
“PM states that she had desprepency [sic] that seemed to become greater
over the course of 20mins. Then a few minutes later the descrepency
[sic] vanished and normal figures remained normal.”
51 PC0032801 not disclosed at time of writing this report.
52 PC0038631 not disclosed at time of writing this report.
53 PEAK PC0132133, 10 February 2006 {POL-0302553}
54 KEL MSCardifield2219S, 15 July 2005 last updated 27 Novembeer 2007 {POL-0035721}
Prepared by: Jason Coyne
Occupation: Partner : .
Specialist Field: IT Systems itg rou )
On the Instructions of: Freeths LLP
POL-BSFF-0100992_0043
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 40 of 265
3.111
3.112
3.113
3.114
3.115
3.116
The cause is documented as follows:
“It would appear that when working out the cash discrepancies on counter
2 the system has used an old ‘data tree’ (the one it used at the earlier
trial balance) rather than creating a new one so the discrepancies were
wrongly calculated. It wasn't until the PM later moved to counter 1 that
a new ‘data tree’ was produced and the discrepancies were calculated
correctly.”
Although the Subpostmaster did not suffer an actual discrepancy in the
PEAK quoted directly above this bug shares similar elements to other
PEAKs (see for example paragraph 3.115 below) whereby the issue has
caused discrepancy. On this occasion a software bug fix was
subsequently implemented:
“New versions of software have been released to the live estate both to
fix a specific variant of the problem and also to provide additional
diagnostics to help identify the root cause of other variants.”
I note that in relation to the KEL referenced within this particular PEAK
(MSCardifield2219S) both Dr Worden and Mr Parker acknowledge that
whilst a discrepancy was caused, it would have been resolved in another
cash declaration made by the Subpostmaster.
In my opinion, this is too broad an assumption to make as it would
require the discrepancy reason as being recognised as a Horizon
generated bug and not one caused by the Subpostmaster, therefore
requiring a TC that the Subpostmaster would not be liable to settle.
PC0144386** created 15 Mar 2007 (referenced KEL MSCardifield2219S)
refers to a ‘Published Known Issue’ in which data held on the counter to
provide quicker information recall could cause apparent discrepancies
in cash declarations.
The PEAK concludes:
“This is only an issue with the figures displayed by the counter the values
actually held behined the scenes are correct and can be updated either
55 PEAK PC0144386, 15 March 2007 {POL-0314727}
Prepared by: Jason Coyne fe)
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0044
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 41 of 265
3.117
3.118
by loging off and back on to a different counter or waiting for the
overnight run to cause it to catch up.
This is a known problem and is documented in KEL MScardifield2219S.”
Despite the ‘Root Cause’ being identified as “Development - Code” it
appears this PEAK record is closed on account of the KEL advice being
available to provide to Subpostmasters, with the assurance that values
held behind the scenes are correct. However, it clearly introduced user
input error as the KEL states:
“This will be potentially confusing and may lead to the clerk making
unnecessary corrections. These will in turn show up as future
inconsistencies (eg nothing gets lost in the end).”
It is not clear whether this bug was scheduled for a later fix from the
detail provided within this PEAK record. However, it is important to note
that the KEL above indicates this issue was fixed in 2006, yet this
occurrence was 2007.
Girobank Discrepancies (Horizon Issue 1)
3.119
3.120
PC0044232°° dated 5 May 2000 (copy from PC0044101 [not formally
disclosed]) references KEL MWright531p, is a PEAK in which the timing
of certain process operations is found to be the cause of discrepancies.
The PEAK records:
“This difference (£505.72) between the Cash Account and the Daily
reports is explained by KEL:MWright531P.htm There was a giro for this
amount that was entered on the 13th Apr then reversed AFTER cutoff
then re-entered again and reversed again. The Daily report would have
shown the original £505.72 but the daily reports never show reversals. It
would be nice to close the call as known error, however while
investigating the message store I have identified another problem...”
The secondary problem refers to transactions that are counted twice in
error due to the time they are performed (coinciding with a cut off
report).
58 PEAK PC0044232, 5 May 2000 {POL-0222723}
Prepared by: Jason Coyne fe)
Occupation: Partner ae
Specialist Field: IT Systems It Jrou f )
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0045
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 42 of 265
3.121 The diagnosis further states:
“The fix for this issue should address all cut-off reporting, not just
Girobank reports.”
3.122 This therefore indicates that this bug could apply in many circumstances
not just when performing Girobank transactions. The bug fix appears to
have been issued in July 2000.
3.123 It is noted that the above PEAK references KEL MWright531P.57 Whilst
that KEL does not appear to have been disclosed, a search utilising its
name returns the following associated PEAKs:
PEAK Date Created Further Observations
References
PC0044232 05 May 2000 (OP)PC00441015% £505 discrepancy - Branch
{POL-0222723} (0P)PC0049280 unknown
{POL-0224421}
PCc0050418 17 July 2000 N/A £422.66 discrepancy - Branch
{POL-0225562} unknown - Closed insufficient
evidence
Pcoos08s61 21 July 2000 N/A Discrepancy (amount
{POL-0225998} unknown) - Branch unknown
PCO052575 13 September N/A £40 discrepancy - Branch
{POL-0228829} I 2000 unknown
PCO052704 18 August 2000 N/A £363.94 discrepancy - Branch
{POL-0227582} unknown
PC0052804 21 August 2000 N/A £55.00 discrepancy - Branch
{POL-0227683} unknown
PCO0053975 13 September N/A £40.00 discrepancy - Branch
{POL-0228829} I 2000 unknown
PCO054846 28 September £99.13 discrepancy - Branch
{POL-0229671} I 2000 unknown
3.124 Appearing to document the same issue over a different timeframe,
PC0068633°° dated 27 July 2001 relates to a bug that caused a Girobank
57 KEL MWright531P not disclosed at time of writing this report.
58 PC0044101 not disclosed at time of writing this report.
5° PC0068633, 27 July 2001 {POL-0242631}
Prepared by: Jason Coyne
Occupation: Partner .
Specialist Field: IT Systems itg rou )
On the Instructions of: Freeths LLP =
POL-BSFF-0100992_0046
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 43 of 265
deposit to be duplicated in a Subpostmasters branch account therefore
resulting in the Subpostmaster receiving an error notice.
3.125 It is likely that this bug was resident in the system for a period of time
as the PEAK detail states:
“I have duplicated this bug. In fact it occurs in all reports that use
dataserver (i.e. the majority). I shall now check to see whether or not
the problem still occurs at S10.”
3.126 The KEL related to this PEAK is documented as ‘AChambers4410R'©°
Similarly, this KEL does not appear to have been disclosed, therefore it
has not been possible to ascertain what advice might have been given
to a Subpostmaster should they be affected by this bug.
3.127 Further associated PEAKs that reference KEL AChambers4410R are
provided in the table below.
PEAK Date Created Further References I Observations
PC0073855, 13 February 2002 (OP)PCO075312 Affecting National Savings
{POL-0247668} {POL-0249033} deposits. Interestingly, this
call record is closed logged
as “insufficient evidence”
despite reference to fixes
being applied as resolution
PC0075312 10 April 2002 (OP)PCO073855 Giro deposit cut off issue.
DRowe440R°! mentioned as
{POL-0249033} {POL-0247668} applying
PC0076065 09 May 2002 N/A Giro deposit cut off issue -
{POL-0249726} Branch unknown
3.128 The above PEAKs related to Girobank discrepancies are clear examples
of bugs within Horizon that affect branch accounts by way of a financial
discrepancy and illustrate, by their interlinking natures, the complexities
of the problem records.
60 AChambers4410R not disclosed at time of writing this report.
®1 KEL DRowe440R, 14 February 2002 last updated 28 January 2003 {POL-0033459}
Specialist Field: IT Systems
Prepared by: Jason Coyne c
Occupation: Partner a .
itgroup
On the Instructions of: Freeths LLP < =
POL-BSFF-0100992_0047
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 44 of 265
Counter Replacement Causing One Sided Transactions (Horizon Issue 1)
3.129
3.130
PC0058528% created 24 November 2000 refers to an instance where a
receipts and payments mismatch was encountered by the branch
displaying a £167.12 shortfall. The diagnosis illustrates that following a
counter replacement (performed due to hardware error), a transaction
was overwritten, disrupting the double entry principle and causing a
one-sided transaction to be written to the accounts. Whilst Fujitsu were
able to identify that which was overwritten, there is no further detail
within the PEAK record as to how this was resolved financially for the
Subpostmaster.
This PEAK’s associated KEL is recorded as JBallantyne5328R.%
Performing a search across the PEAK disclosure utilising this KEL name
returns approximately 88 further PEAKs. Whilst I have not reviewed all
of the PEAKs returned in detail, review of three randomly selected
records illustrate:
PEAK Created Date Observations
PC0071836 28 November 2001 Branch 214552 had a base counter
{POL-0245811} replacement as PEAK above in which
messages were overwritten causing a £3.27
gain. No documented discrepancy fix
resolution.
PCO0133822 27 March 2006 Branch 109002 - Counter swap out causing
{POL-0304212} transaction differences.
PCO153851 07 February 2008 Branch 154311 - payments mismatch issue
{POL-0324139} - PEAK record has multiple references to
different KELs including JBallantyne5328R
KELs
3.131 In conclusion, since Fujitsu support had the facility to insert items within
the Horizon message store, without process audit (detailed further
within Section 4 in response to Mr Godeseth’s responsive witness
statement and Section 5 Sub Section 11), the effects of one-sided
62 PEAK PC0052528, 16 August 2000 {POL-0227413}
®3 KEL JBallantyne5328R, 1 December 2000 last updated 4 July 2007 {POL-0448249}
Specialist Field: IT Systems
Prepared by: Jason Coyne c
Occupation: Partner oi a .
3 itgroup
On the Instructions of: Freeths LLP
POL-BSFF-0100992_0048
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 45 of 265
transactions and their applied corrective fixes is clearly larger than the
“one balancing transaction” as suggested by Post Office in their 28 July
2016 letter of response (Schedule 6).%
Withdrawn Stock Discrepancies (Horizon Issue 1)
3.132 PC0207834® created 19 January 2011 relates to a bug in which the
Subpostmaster encountered various gains and discrepancies. It is
reported in the PEAK that 8 other offices faced similar issues. The
associated KEL is PothapragadaC4913L (raised 09 July 2010 last
updated 09 July 2010).
3.133 The PEAK was subsequently cloned after diagnosis to PC0208292°7
created 9 February 2011 and issued to development to investigate a
bug fix and root cause. The summary was:
“Withdrawn stock items can be re-introduced into stock by making a stock
declaration this can subsequently cause discrepancies at future
rollovers.”
3.134 Release PEAK PC0208918* subsequently records the bug fix detail in
which the data is applied to live 1 April 2011.
3.135 An associated record to the above bug, PEAK PC0209602°° created 11
April 2011 illustrates in more detail the differences between how Legacy
Horizon would have dealt with withdrawn stock in comparison to how
Horizon Online does it. The Impact Statement within this PEAK details:
“Can cause confusion and unexpected (though hopefully temporary)
discrepancies at branches by allowing them to declare stock which has
already been withdrawn. Additional problems Spring 2011 highlighted
that at least 60 or so branches managed to do this. Although the
additional problems should be fixed before more products are withdrawn,
% {Letter of Response from Post Office,"SCHEDULE 6: REBUTTAL OF ALLEGATIONS AGAINST
HORIZON", 28 July 2016}
65 PEAK PC0207834, 19 January 2011 {POL-0377562}
66 KEL PothapragadaC4913L, 9 July 2010 {POL-0037644}
87 PEAK PC0208292, 9 February 2011 {POL-0378016}
68 PEAK PCO208918, 10 March 2011 {POL-0378633}
6° PEAK PCO209602, 11 April 2011 {POL-0379308}
Prepared by: Jason Coyne fe)
Occupation: Partner ae
Specialist Field: IT Systems It ia Uf )
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0049
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 46 of 265
excluding these products from the Declare Stock picklist would be
sensible, and in step with Horizon.”
3.136 Ihave not been able to isolate the PEAKs for the 60 or so other branches
referred to in relation to this incident.
3.137 In regard to further identifying how all records associated with a
particular bug are to be identified, I have requested from Post Office in
my Request for Information sent 14 December 2018 (Annex A),
information as to how Fujitsu measure and account the impact of known
bugs.
3.138 The response provided by Post Office (17 January 2019) sets out:
“It is not possible to provide a generic answer to this request - the way
in which "the impact of a bug is assessed will depend on the nature,
operation and effects of the bug. Information regarding the ways in which
Fujitsu assessed the impact of the bugs referred to in paragraphs 12 -
16 and 34 - 61 of the second witness statement of Torstein Olav Godeseth
dated 16 November 2018 is provided in those paragraphs.
3.139 As illustrated in Mr Godeseth's statement where a bug has been
identified, Fujitsu's approach has been to seek to determine what
branch was affected and to present this to Post Office, along with how
they were proposing to resolve the issue. In my opinion, and as
observed through the PEAK/KEL analysis and responses provided within
the Defendant's Witness Statements, identification of issues through
recorded branch impact alone does not appear to sufficiently enable
identification of a full bugs impact, neither proactively or
retrospectively.
Bureau Discrepancies (Horizon Issue 1)
3.140 PEAK PC0261541”° dated 17 August 2017 relates to bureau pre-order
currency transactions that cause discrepancy in branch accounts. Note
that this PEAK does not reference a specific KEL. The PEAK detail records
that the office was left £204.59 short after Horizon initially recorded the
70 PEAK PC0261541, 17 August 2017 {POL-0429147}
Prepared by: Jason Coyne fe)
Occupation: Partner .
Specialist Field: IT Systems ; It lI
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0050
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 47 of 265
3.141
3.142
3.143
3.144
3.145
complete currency order but only actually processed one out of two
currencies.
Upon investigation, on 23 August 2017 the PEAK detail is cloned to
PC0261710”! for future development investigation which records:
“OK we had a call from Largs post office, they hit a problem with bureau
pre-order. We investigated and it’s a fault in an ADC script. I’ve been
trying to get our helpdesk to route it to your helpdesk but so far not
working very well.
It’s a nasty thing with financial impact for the branch. What's best way
to make sure someone at your end knows it needs fixing?”
The issue is diagnosed as (in summary) being due to a network timeout.
The subsequent advice is to return the call to Post Office for them to
decide what reconciliation or Transaction Correction (TC) would be
required to balance the office (effectively removing the shortfall).
No further information as to the advice that might have subsequently
been given by Post Office is documented, and have I been able to find
any other related PEAKs that record further information in relation to
this branch incident.
Further manifestations of Bureau/Currency issues are identified below:
PC02654437”2 created 19 December 2017 is an extremely lengthy PEAK
(mainly due to confusion regarding the issue and the support team
attaching the wrong evidence to the record in the first instance) that
pertains to a 4500 Euro discrepancy and illustrates the following:
a. Despite involvement of Accenture, Atos, Fujitsu and Post Office, no
party appears to be able to effectively decipher what has caused
the discrepancy between the branch’s foreign currency account,
against the figures held by POLSAP and Cash Management.
71 PEAK PC026170, 23 August 2017 {POL-0429314}
72 PEAK PC0265443, 19 December 2017 {POL-0432829}
Prepared by: Jason Coyne fe)
Occupation: Partner ae
Specialist Field: IT Systems It Jrou f )
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0051
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 48 of 265
b. The PEAK details the frustration felt by the Subpostmaster’s Area
Manager who chases resolution of the issue over a three-month
period:
“You mentioned on 8th March that you were aiming to get this resolved.
As you can see below, with the exception of Matthew and Andrew I am
getting extremely frustrated by the total lack of response I am getting to
this request.
I really don t think it is acceptable that I should have to send 40+ emails
over 3 months and attend 2 x 1 hour long conference call to resolve this
issue. All I am asking is that we find out and rectify why my branches
figures do not match those that Andrew Keighley has.
I recognise that everyone thinks if they don t answer that I will eventually
give up but I am absolutely not prepared to do this.
Nobody appears to know what to do with this query and I cannot tell you
how frustrating this is getting.
SOMEONE PLEASE HELP!!!”
c. Aside from Atos suggesting that the Subpostmaster be requested
to perform a “dummy transaction of 4500 Euros” in order to
register the transaction that is missing in POLSAP and causing the
discrepancy (which appears to be rejected in principle by the
Subpostmasters Area Manager), the ticket appears to be closed
without any detailed explanation as to why Post Office’s Cash
Management Centre recorded different currency values to those in
the branch for Euros and Dollars.
3.146 It appears to show that this PEAK relates to a one-sided transaction in
which the branch had a record of a Euro sale but that was not reflected
in Post Office’s POLSAP system therefore causing discrepancy.
PEAKs that relate to errors in data recorded within Horizon (Horizon Issue 4)
3.147 The following PEAKs have been identified as relevant to Horizon Issue 4
to illustrate the varying types of errors in data recorded within Horizon,
arising from (a) data entry, (b) transfer or (c) processing of data.
Prepared by: Jason Coyne fe)
Occupation: Partner ae
Specialist Field: IT Systems It Jrou f )
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0052
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 49 of 265
Phantom Transactions (Horizon Issue 4)
3.148
3.149
3.150
3.151
3.152
PC00650217? created 17 April 2001 relates to the “Master Call for
Phantom Txs” PEAK. This suggests that it was the intention for all
phantom transactions reported to be captured within this PEAK.
The first incident recorded within this PEAK documents a Subpostmaster
alleging to have paid out over £1,500 in losses “due to these problems”.
Further detail within the PEAK states:
"02/05/01 14:12 uk052436Information: Romec have been to site today
and fitted shielded cabling and suppressors. Romec engineer advises that
he has witnessed further phantom transactions whilst on site. He will
carry out further tests and advise results.”
PC0052025”4 created 9 August 2000 (referenced KEL RColeman21103’°)
appears to be another record of suspected phantom transactions
(certain transactions appear to be duplicated at a later time in the same
order day). Further, the Subposmaster notes that the ‘Customer
Reference’ number recorded for one of the British Telecom transactions
“is EXACTLY the same as the British Gas trans.”
The diagnosis concludes that the additional transactions were processed
due to a suspended session on the Counter that was later “forcefully
committed”. It appears that Horizon will, after periods of inactivity,
ultimately commit transactions a Subpostmaster has not fully
completed themselves.
The Subpostmaster also notes that icons on the Counter have changed
on their own. The PEAK detail references that KEL RColeman2110)
applies, however I have not been able to review this KEL as it does not
appear to have been disclosed. A search within the PEAK disclosure
utilising RColeman2110J returns one further PEAK potentially related to
this issue.
73 PEAK PC0065021, 17 April 2001 {POL-0440162}
74 PEAK PCO0052025, 9 August 2000 {POL-0227064}
75 Not disclosed at the time of submitting this report.
Prepared by: Jason Coyne fe)
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0053
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 50 of 265
3.153 The above PEAKs illustrate potential for errors in data recorded within
Horizon arising from Hardware failure and accepted design features.
Reconciliation Issues (Horizon Issue 4)
3.154 PC003983276 created 3 March 2000 (no referenced KEL) documents a
bug in relation to a Subpostmasters Cash Account Period (CAP) in which
reconciliation discrepancies have appeared but did not feature on the
expected reconciliation exception reports. The following appears in the
PEAK:
“The discrepancy reported by the reconciliation software appears to be
related to the value of two transactions (one for £8.06 the other for
£0.08) which were actually ‘brought forward’ values from the previous
week's Cash Account. This being the case, I suspect that the
reconciliation software has mis-calculated the Table 3 value, rather than
the Cash Account being incorrect.”
3.155 Although the discrepancy amounts are small on this occasion the PEAK
still warrants a bug fix that is rolled out to the Live Estate rather than
awaiting a next functionality release documented as follows:
“I have also noticed that the Cash Account reconciliation for the previous
week also reported an £8.14 discrepancy on Table 3. Since the
reconciliation process uses it's [sic] own brought forward values for the
suspense account I suspect that this issue may well have its roots in an
earlier CAP. Given that this is a financial reconciliation issue, I suggest
that this will require correction before CI4.”
3.156 The above raises a concern as neither the Subpostmaster or the Post
Office noticed the earlier £8.14 from the previous week.
3.157 PC003983277 (detailed above) was subsequently fixed as part of
PC004795578 in August 2000 and five months after the original PEAK
was raised.
78 PEAK PC0039832, 3 March 2000 {POL-0222262}
77 PEAK PC0039832, 3 March 2000 {POL-0222262}
78 PEAK PC0047955, 19 June 2000 {POL-0223659}
Prepared by: Jason Coyne fe)
Occupation: Partner ae
Specialist Field: IT Systems It Jrou f )
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0054
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 51 of 265
3.158
3.159
3.160
3.161
3.162
3.163
PEAKs PC007524079 (created 08 April 2002, referenced KEL
DRowe304L®), PC0075415* (created 12 April 2002 referenced KEL
DRowe304L) and PC0077508* (created 12 April 2002, referenced KEL
DRowe304L) all relate to an issue where a Branch Counter total differs
from the Host Amount (total generated by integrity checking through
Horizon processing).
The issue predominately relates to a Horizon reconciliation report
(TPSC268A) that illustrates a discrepancy for the branches documented
in which the Cash Account totals differ by 1p.
The summary of the issue is that program code values of 0.01 and
0.0099 were checked for zero values. The 0.0099 values were returned
as zero as the code ignored values after two decimal places.
The fix is documented within PCO0075415 as a “straightforward change
to Cash Account Common Code...” However, the combined PEAKs
illustrate that the fix for this issue was revoked, due to the following
(PC0077508):
“The work package WP13953 caused TPSC265 to run VERY slowly, so it
has been withdrawn. That means that the problem in this pinicl is liable
to reappear.”
Later in the month it appears another bug fix was issued for this and
the records are subsequently closed.
PC0049578*? created 6 July 2000 documents a bug that restricts the
reporting set TPSC260 to correctly count the number of files read within
the system. The implications of this might have affected reconciliation
as integrity checks supplied by the report totals would have been
incorrect.
79 PEAK PC0075240, 8 April 2002 {POL-0248963}
80 Not disclosed at the time of submitting this report.
81 PEAK PC0075415, 12 April 2002 {POL-0249128}
82 PEAK PC0077508, 12 April 2002 {POL-0249130}
83 PC0049578, 6 July 2000 {POL-0224722})
Prepared by: Jason Coyne fe)
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0055
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 52 of 265
3.164 The PEAKs referred to above can all be classified as errors in data
recorded within Horizon arising from the incorrect processing of data
within Horizon, therefore Issue 4 part (c).
3.165 PC0045847*4 documents the occurrence of a message store corruption
that resulted in a branch discrepancy of £4462.46. The PEAK states:
“...The NT event log indicates that at the time of the balance the Riposte
system recorded numerous errors indicating that there was a corruption
of the message store (CRC failure) resulting in the current ‘query' being
destroyed. This is almost certainly the cause of the missing balance
records at rollover. Passing to EPOSS-FP to determine whether there is
anything that can be done to improve the system error handling within
the dataserver.
3.166 It is noted that this error must have occurred previously since the PEAK
further states another PEAK reference and:
“This is supposed to be fixed in the near future so close this as duplicate
call.”
3.167 In summary, this issue arose from an error in the transfer of data within
Horizon (Horizon Issue 4) when the Stock Unit in the branch was rolled
into its new cash account period, the system failed to record the correct
values.
3.168 The following PEAKs provide a mechanism for further understanding in
relation to Issue 5 and how Horizon compares transaction data recorded
by Horizon against transaction data from sources outside of Horizon.
They should also be considered under Issue 4 (errors in data recorded
within Horizon) and are also examples of how mechanisms were in place
to detect and report errors in Horizon (Issue 6).
84 PEAK PC0045847, 22 January 2013 {POL-0223066}
Prepared by: Jason Coyne fe)
Occupation: Partner .
Specialist Field: IT Systems ; It lI
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0056
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 53 of 265
3.169 Whilst outside of particular Subpostmaster effect, the following PEAKs
illustrate the role of external client input in the reconciliation process
and identifying discrepancies in Horizon.
3.170 PC0236246° dated 2014 relates to Post Office’s client Allpay.net and
records that the issue is:
“The Client Transaction Summary from 4/8/14 is missing £110,706.86
when compared to the Client File and POLSAP entries.”
3.171 The Client Transaction Summary (CTS) is ultimately derived from the
Automatic Payments Service (APS) which copies the transactions from
the branch database in Horizon Online.
3.172 PC0204872°° dated September 2010 again highlights discrepancies
between CTS report figures and external client figures. The issue is
noted as following:
“The CTS report is received daily and is compared with the vendor (in this
case A&L) reports. The figures for each day should match.
If the CTS report is larger than the vendor figure, the vendor account will
be credited. The credit usually shows a couple of days later as a positive
discrepancy.
The CTS report was showing as being larger than the vendor figures on
the following dates, although there does not appear to have been any
counter credit showing on the vendor figures following on from this:
7th May 2010 - CTS was greater than vendor figures by £84.86. POL have
suggested that this may have been related to an event from 27th
February for FAD 490519, although we can find no BIMS record of this
from a Reconciliation perspective.
25th July 2010 - CTS was greater than the vendor figures by £3,260.00.
No additional information is available.
27th August 2010 - CTS was greater than the vendor figures by £846.00.
No additional information is available.”
85 PEAK PC0236246, 7 August 2014 {POL-0405575})
86 PEAK PC0204872, 29 September 2010 {POL-0374647}
Prepared by: Jason Coyne fe)
Occupation: Partner ae
Specialist Field: IT Systems It }roOu f )
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0057
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 54 of 265
3.173
It is unclear from the statement above “a/though we can find no BIMS
record of this from a Reconciliation perspective” (BIMS being a business
incident record of where an anomaly has occurred) whether the
Subpostmaster might therefore also have been impacted by this
discrepancy in their branch accounts (given that they would reflect
different figures than those summarised for the CTS file). I have not
had full visibility of Post Office’s processing policies in respect of external
client reconciliation and how they could relate back to specific branch
account discrepancies but, as noted above, this reflects instances where
measures and controls exist for detecting errors arising in Horizon.
Branch Customer Discrepancies (Horizon Issue 4)
3.174
3.175
3.176
Review of PEAK records have identified instances where the Post Office
Customer, in branch, may have encountered a discrepancy from Horizon
shortfalls.
PC0156246°”, 26 March 2008 details an incident whereby the Financial
Institution (FI) contacted Post Office in relation to a settlement
difference. Although the Subpostmaster declined the transactions at the
Counter (after recovery of them initiated) and the transaction was
therefore reversed (so as to cancel the debit request from the branch
account perspective), the end Customer’s account was still debited by
the FI.
The PEAK detail records:
"..So it is likely that the branch balanced but the customer's account now
needs rectifying for the loss - which is why Citibank are showing the
discrepancy.
So I am passing this call back with the note to MSU: that before this
customer's a/c is rectified for his loss of £165.26 that POL contact the PM
at the branch to double check that NO money did change hands for
certain, before finally ensuring that this financial discrepancy is dealt
with.”
87 PEAK PC0156246, 26 March 2008 {POL-0326528}
Prepared by: Jason Coyne fe)
Occupation: Partner 1
Specialist Field: IT Systems it }FOk I
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0058
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 55 of 265
3.177
3.178
This occurrence further emphasises the need for sufficient process
adherence and clarity between Post Office and the support teams in
order to appropriately identify and correct discrepancies. Whilst there
may not have been an anomaly in the branch account, the
Subpostmaster would not have the ability to review the processing
systems as Post Office have the ability to check.
“It is likely the branch balanced” is not a clear diagnosis. This PEAK is
similar in nature to the incident described by Mrs Burke in her Witness
Statement. In Mrs Burke’s case, she was able to contact the end
customer to produce their receipt of successful withdrawal. However it
would be difficult for a Subpostmaster to do this in every event of a
suspected failed recovery procedure (along with the fact that some
customers might not be regulars at the branch in question).
Concurrent Logins (Horrizon Issue 4)?
3.179
3.180
Several PEAK’s identify that Fujitsu have to investigate issues that are
encountered due to users logging onto multiple Counters at the same
time which can cause transactions to be abandoned and risk
discrepancies. It is not understood why this often occurred, in legacy
Horizon it appears as though the ability to login concurrently was
classed as an error.
PC0027581°* dated 9 July 1999 provides an example of a concurrent
login issue. Despite the issue being passed to multiple support and
development teams no solution was ever found, and the case was closed
on 7 February 2002 on the basis that Mr Lui was no longer employed by
the Post Office and the call could be reopened should the issue reoccur.
It is troubling that Fujitsu was aware, as evidenced by a case log entry
dated 13 July 2001 by Walter Wright, that there was a “deficiency” with
Riposte in allowing simultaneous logon but did not follow this up
properly with Escher (the case was ultimately closed).
88 PEAK PCO0027581, 9 July 1999 {POL-0221763}
Prepared by: Jason Coyne fe)
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0059
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 56 of 265
3.181
3.182
3.183
PC0051327°° dated 28 July 2000 also indicates a failure in respect of
concurrent logons. The PEAK detail diagnoses that the discrepancy has
arisen as a result of “a failure in the logon checks”.
“A message <GroupId:182432><Id:3><Num:9921> was produced on
counter 3 saying that the session transfer had failed but in fact the log in
succeeded and hence you got a user logged in to two terminals at once.
This is a situation not catered for in EPOSS code and hence you get the
later problems already described. If it is desired to progress this further
the bug must be assigned to the Agent team to find out why the Stop
Desk Transfer service failed to prevent the user logging in on counter 4 (
and subsequently doing the Transfer Out which caused the problems ).”
The call record is closed on 30 November 2000 with the following
“As this is fixed at CI45 then I am happy to close this call”.
In respect of the incidents referenced above, it is not clear what the full
effects or resolutions were regarding the discrepancies.
Network Banking Bug
3.184 PC0109020% details issues with regards to Network Banking (NWB) and
Online Services transactions due to an ISDN fault. The issues in
connectivity subsequently caused an imbalance in the Subpostmasters
accounts due to end customer accounts being debited and customers
therefore requesting the funds.
Post & Go / TA discrepancies in POLSAP (Horizon Issue 4)
3.185
PEAK PC0220393% created 29 August 2012 details inconsistences
between source data received in POLSAP and Horizon which could have
impacted branch accounts. The text suggests a duplication of a
transaction from Wincor, the text reads:
“An example the customer has provided shows amounts of 115.05,
46.88, 52.13 & 75.23 totalling 289.29 received on the file from Wincor
and into POLSAP via BLE.
89 PEAK PC0051327, 28 July 2000 {POL-0226410}
90 PEAK PCO109020, 1 October 2004 {POL-0280615}
°1 PEAK PC0220393, 29 August 2012 {POL-0389956}
Prepared by: Jason Coyne fe)
Occupation: Partner ae
Specialist Field: IT Systems It Jrou f )
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0060
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 57 of 265
The same (contra) amounts are also showing as being received from the
branch when the TA has been accepted and are closed items in the
account (netted off to 0.00).
However, there is another amount of 289.29 which just has the date in
the assignment field.”
3.186 A response from Dave Allen (Fujitsu) reads:
“Postings on the TfS call refer to a similar previous incident (A1040049
=> Peak PC0219432°7), which was resolved between POL and Wincor
Nixdorf; no details of this resolution are available to us. This incident is a
week old, but only came to SSC late last night... The trading-date in this
call, 2012-08-09, is three weeks ago which too old for us to be able to
see the incoming file from Wincor Nixdorf... There is no evidence of a fault
in HNG-X, and without the incoming file from Wincor Nixdorf there is
nothing further for us to investigate.
We can only suggest that POL do the same as they did with A1040049,
and refer the matter to Wincor Nixdorf.”
3.187 This suggests that the matter was reported too late to determine what
the fault may have been. However, a few days later Anne Chambers
(Fujitsu) adds to the PEAK:
“Branch 020511 has many entries in the Subfiles_on_hold report. This
report should be monitored (by ?) to make sure problems are followed
up - this should be resolved before closing this call.
Horizon is receiving PG data for 6 separate PG tills at the branch, but only
4 of them have associated stock units. This causes the entire subfile for
the branch to be Held, and the transaction data is not being sent to
POLSAP. However the TA data for the 4 tills which are properly associated
IS being sent through, and I think this is probably the cause of the
POLSAP anomalies.
°2 PEAK PC0219432, 13 July 2012 {POL-0389009}
Specialist Field: IT Systems
Prepared by: Jason Coyne c
Occupation: Partner a .
itgroup
On the Instructions of: Freeths LLP < =
POL-BSFF-0100992_0061
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 58 of 265
The two unassociated tills are not doing any cash transactions - this is a
known problem (see PC0218702°), and means the PM isn't prompted to
create an association. This may need fixing via MSC.
Other branches on the report may also need similar action. We have
found that 007113 has been closed for 18 months, so the PG txns were
misdirected, but I don't understand exactly what happened”
3.188 A bug fix to the Horizon system was identified by Fujitsu, scheduled for
implementation 13 September 2012 after 1800hrs and the Branches
stock was to be corrected at 1700hrs that same day.
3.189 On the 17 September Anne Chambers reported in the PEAK that:
“Following a change made centrally to facilitate this, the stock unit
associations for the two new Post and Go terminals have been created by
the branch and all the held external data (43 different days) has now
been processed and passed through to POLSAP... We strongly recommend
that POL monitor the SubfilesOnHold report which is sent to them daily,
so that any other external terminals with problems can be investigated
quickly in case a similar correction is needed.
3.190 A couple of observations can be made from the PEAK. It appears that
the underlying bug, error or defect impacted branch accounts for 43
days until resolved and also that Post Office had not been monitoring
the “SubfilesOnHold report” which Fujitsu send to them daily. If they
had of been monitoring it, the fault would not have impacted for this
length of time.
Recovery Failures (Horizon Issue 4)
3.191 PC0220532% created 5 September 2012 documents an instance where
a branch (391230) alleges Horizon caused a loss. The information within
the PEAK is limited with the concluding text of the PEAK stating:
“If further investigation by Fujitsu is required, Post Office will have to
request that the branch transaction data is retrieved from the audit
server. If there is any possibility that this is required for litigation, it must
°3 PEAK PC0218702, 13 June 2012 {POL-0388291}
°4 PEAK PC0220532, 5 September 2012 {POL-0441342}
Prepared by: Jason Coyne fe)
Occupation: Partner ae
Specialist Field: IT Systems It Jrou f )
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0062
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 59 of 265
3.192
3.193
3.194
3.195
3.196
come through the Security (ARQ) route. Otherwise queries of this nature
should be sent via Mark Wardle at POL, and should be routed to the
reconciliation team in the first instance. Such requests may be
chargeable.”
It is unclear (as with most PEAKs relating to possible financial
discrepancies) what the full resolution or conclusion of the issue was
since Post Office have not disclosed in detail full Transaction Correction
information for all reported discrepancies.
PC0241242°% created 23 February 2015 relates to a branch (2693232)
in which poor communications with the Data Centre resulted in a failed
recovery of a Health Lottery transaction. Whilst it is not suspected that
this instance caused the Subpostmaster a discrepancy, the PEAK
records that support have “asked POL (via ATOS) to authorise us to
remove the Health Lottery txn..which is preventing successful
recovery.”
PC0197643°° created 14 April 2010 refers to branch 166948 in which a
£240.00 transaction failed in recovery. Whilst a table exists in the
database to potentially capture failed recovery transactions, these then
have to be manually reconciled. The PEAK states:
“Looking at the PostOfficeCounterLog, the receipt printed ok for this after
authorisation was received, the receipt that printed for the cash
withdrawel states "Authorised", so it's possible that the clerk handed over
the monney [sic].”
As this was passed to Post Office, it is unclear what their final resolution
was. It is not documented if Fujitsu removed the transaction and if they
did, how they did it.
Horizon recovery issues are also noted under PEAKs relative to Horizon
Issue 1 and illustrate there are potentially many recovery failure
°5 PEAK PC0241242, 23 February 2015 {POL-0441722}
°8 PEAK PCO197643, 13 April 2010 {POL-0367516}
Prepared by: Jason Coyne fe)
Occupation: Partner ae
Specialist Field: IT Systems It Jrou f )
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0063
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 60 of 265
manifestations. In many instances, they cause financial discrepancy
that ought to be requested by a Transaction Correction.
Transaction Correction Issues (Horizon Issue 4)
3.197 The following PEAKs are relevant to Horizon Issue 15 and how Horizon
processes and records Transaction Corrections. They provide an insight
in relation to technical flaws surrounding the processing of Transaction
Corrections.
3.198 PC0129587°’ dated 1 December 2005 relates to Transaction Corrections
(TC) and issues with counter freezes during acceptance of the
Transaction Correction. It should be noted that SSC were able to
diagnose the problem after importing the message store (from the
branch Counter) onto an SSC (Fujitsu) counter. The issue is
predominately related to the options functionality of the Transaction
Correction and length of the Transaction Correction text. PCO130056%°
is a cloned call of PC0129587.
3.199 The PEAK detail further states:
"..(d) PEAK PC0120459, raised on S80 E2E XI, reported the same
symptoms and this was found to be missing/incorrect reference data.”
“This is certainly a bug in the code but is given a challenge with the
continuous unspaced text.”
3.200 It is recorded that the inability to accept the TCs would impact
Subpostmasters as they would not be able to “roll over” into a new
accounting period. The PEAK states:
“I have raised the issue formally with POL, via Julie Welsh, to ask them
to stop creating TCs with long text.
Ravinder has/will be contacting the 6 affected FADs: 010937, 015937,
182937, 262539, 322519 and 559323 to explain the avoidance action:
97 PEAK PC0129587, 1 December 2005 {POL-0300024}
°8 PEAK PC0130056, 14 December 2005 {POL-0300490}
Prepared by: Jason Coyne fe)
Occupation: Partner ae
Specialist Field: IT Systems It Jrou f )
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0064
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 61 of 265
3.201
3.202
3.203
3.204
3.205
rollover all but one stock unit. All want to rollover on 14/12/05 except for
262539 (who wanted to roll today 07/12/05).”
PC0120459°° dated 04 May 2005 whilst referred in PEAK PC01295871°°
as “reported the same symptoms” actually records Transaction
Correction button functionality issues for a different branch in which the
PEAK detail states “Confirmed issue caused by a missing Work package,
lost after the rig reset.”.
This PEAK is also cloned to PC0118562.'°! PC0118562 dated 11 April
2005 further refers to PC0114154'° (dated 18 January 2005). However
again, these latter two PEAKs relate more to the Transaction Correction
button functionality presented on screen rather than the system freezes
referred to in PC0129587 and PC0130056.'°3 PC0121331!% again
relates to the same issue.
Further Transaction Correction issue PEAKS are PC01300571°5
(replication of FAD 182937 issue under PC0129587) and PC01297741%
(again a replication of this PEAK due to a lack of visibility on Powerhelp).
PC0204350'” relative to Transaction Correction reports highlights
confusion faced by a Subpostmaster when trying to investigate a
discrepancy.
A Ssmmary of the issue is that the Subpostmaster had a “cash loss of
around £80 since 08/09/10...” and would not make good the loss which
he believed was due to a system error, alongside an issue of not being
able to see transaction corrections that he had accepted on the system
in a transaction correction report requested for 12/07/10 to 10/09/10.
99 PEAK PCO0120459, 4 May 2005 {POL-0290969}
100 PEAK PC0129587, 1 December 2005 {POL-0300024}
101 PEAK PCO118562, 11 April 2005 {POL-0290080}
102 PEAK PCO114154, 18 January 2005 {POL-0285697}
103 PEAK PC0130056, 14 December 2005 {POL-0300490}
104 PEAK PC0121331, 26 May 2005 {POL-0291837}
105 PEAK PC0130057, 14 December 2005 {POL-0300491}
106 PEAK PC0129774, 6 December 2005 {POL-0300210}
107 PEAK PC0204350, 14 September 2010 {POL-0374134}
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
POL-BSFF-0100992_0065
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 62 of 265
3.206 After support close the call due to “insufficient evidence”, the call is re-
opened by a customer call and NBSC report that they “..can offer no
explanation as to why accepted transaction corrections do not show on
the Processed Transaction Correction report.”
3.207 However, the NBSC advisor escalates again requesting a valid
explanation before the pm will agree to close the call.
3.208 The PEAK detail records the following:
“Details of TC messages are kept in the BRSS for 42 days. This would be
why a query beyond a certain date is not showing the oldest TC put
through the system. The following are still available and visible [in line
with what the PM is reporting]:
Date/Time Ref Amount
08/09/2010 07:51 600024457612542000 £40
22/09/2010 07:43 600023979612542000 £10
25/09/2010 07:49 600027600112542000 £136
06/10/2010 08:02 600028125312542000 £1000
18/10/2010 09:06 600029206112542000 £35
however, beyond that, I will have to request archived data from our Audit
Team in order to confirm those TC txns in July 2010.”
3.209 It is not documented within the PEAK whether the Audit data is actually
requested to clarify the position on the earlier TCs (which might allow
the Subpostmaster to investigate the discrepancy further). The call is
cloned to PC0255671°° with additional detail:
“One of the issues the user raise here is that fact that the 'Valid Date
Range' on the counter suggests that there is data available for two
months, e.g. 21/08/2010 to 20/10/2010. According to information from
BRDB_ARCHIVED_TABLES, the retention period for TC data in the
TPS_TXN_CORRECTION table is for 40 days only and as such it is
108 Not disclosed at time of report writing
Prepared by: Jason Coyne fe)
Occupation: Partner ae
Specialist Field: IT Systems it Jrou
On the Instructions of: Freeths LLP
POL-BSFF-0100992_0066
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 63 of 265
3.210
confusing if the user is presented with a date range which is greater than
the data that is available to be retrieved.”
The further PEAK detail relates to the requirement of changing the
database retention period of TCs to 60 days. It is not clear how the
Subpostmasters cash discrepancy was actually resolved.
Bugs/Errors/Defects introduced by previous applied PEAK fixes (Horizon Issue
4 and to an extent 10)
3.211
3.212
3.213
3.214
The following PEAKs illustrate applied fixes for bugs, errors and defects
that have caused further bugs, errors and defects.
PC0053160'°* created 29 August 2000 documents an EPOSS issue
relative to both training and live environments affecting Counters. The
detail of the issue further documents the fix that was applied caused
regression bugs.
PC0098230,1!° created 30 Jan 2004 is a bug suspected as an occurrence
to a fix previously rolled out for PCO097081."!! This issue is reported to
double the value of cheques declared as stock:
“This results in a discrepancy between the system cheque figure and the
declared figure. Something has changed in the counter code recently (I
think at COUNTER_EPOSS 20_3; released end Nov) which causes the
discrepancy to be recorded wrongly; so the cheque discrepancy; instead
of being cleared; is doubled; and the cash is also wrongly adjusted.”
Whilst this PEAK documents that the Subpostmaster was operating
outside of process alongside this bug occurring:
“Spoke to PM and explained that there is a new software problem; so that
what he has been doing for 2 years no longer works. He's happy with
this. Also spoke to the auditor who was onsite; explained that I had
advised him not to declare these cheques in this way - she confirmed that
they should be put in the suspense account; and said she would talk him
through the procedures.”
109 PEAK PCO053160, 29 August 2000 {POL-0228020}
110 PEAK PC0098230, 13 January 2004 {POL-0270225})
111 PEAK PC0097081, 17 November 2003 {POL-0269113}
Prepared by: Jason Coyne fe)
Occupation: Partner ae
Specialist Field: IT Systems It Jrou f )
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0067
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 64 of 265
3.215 Evidence of this ‘out of process’ working was later provided for the
Subpostmasters disciplinary hearing however it is noted that the PEAK
further states:
“I'd like to add that this can occur when a clerk declares any cheque short
of what the system has calculated; the fact that the clerk/PM in this
instance was going against normal procedure is irrelevant. Multiply this
potentially several thousands of times over there could be an awful lot of
‘repair’ work to do when S5OR kicks in for real.”
3.216 It is unclear whether this branch was operating on the S50 release as a
live trial. However it is documented that the software fix to fix this bug
is planned for the S60 release therefore this issue could have happened
at other branches.
3.217 PC00527761!2 dated August 2000 refers to a reconciliation discrepancy
which records: “This is exactly the same scenario as PinICL
PC0049702”.
3.218 PC0049702''3 is dated July 2000 and relates to a payments discrepancy
at Danby House branch, is summary the PEAK detail records:
“The problem was that the 99990701 CashAccLine was being written out
with negative sign when it should have been written out with a positive
sign. This problem was introduced when fixing PinICL PC0047518 - during
which even more drastic problems with CashAccLines were fixed.”
3.219 This bug was subsequently fixed by a software fix and both PEAK
records subsequently closed by August and Mid-September 2000.
Evidence of Insertions/Deletions within Branch Accounts (Horizon Issue 10)
3.220 In relation to Issue 10 of the Horizon Issues, I opined in my previous
report at paragraph 9.43 (Page 144) that Fujitsu did have the ability to
delete transaction data. Review of the PEAKs and those referenced
below ‘Deletion of Transaction Data’ evidence that Fujitsu could and did
insert, inject, delete and rebuild transaction data or data in branch
112 PEAK PCO052776, 21 August 2000 {POL-0227657}
113 PEAK PC0049702, 7 July 2000 {POL-0224840}
Prepared by: Jason Coyne fe)
Occupation: Partner .
Specialist Field: IT Systems ; It lI
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0068
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 65 of 265
accounts. Occurrences are evidenced where this was both with and
without the knowledge and consent of the Subpostmaster.
Remote Access and Branch Data Alteration (Horizon Issue 10)
3.221 PEAK PC0051855'"* created 5 August 2000 relates to an incident where
the messagestore has to be deleted and re-instated from a mirror copy.
The PEAK detail states:
“ I was concerned that the latest messages from site had not been
replicated to the correspondence server, but I have found that they are
in the riposte mirror, therefore we can continue to delete the main riposte
message store.”
3.222 The associated KEL to this PEAK is documented as DRowe5014K, which
has not been provided in disclosure. Performing a search across the
PEAKs utilising this KEL name returns three further associated PEAKs.
3.223 PEAK PC01959621!5 created 12 March 2010 suggests that the
modifications by Fujitsu support staff to the Horizon Branch Database
(BRDB) is not unusual. Within this document Gareth Seemungal of
Fujitsu discusses making a fix to the transaction correction tool
templates, the benefit is described as follows:
“SSC will be able to fix BRDB transactions quicker and with more
confidence” and “making it less likely that mistakes will occur when SSC
are trying to resolve problems with transactions in BRDB”.
3.224 PC01289691!° dated 17 November 2005 is a PEAK which was considered
a one-off issue and closed (after the stock unit data and figures were
‘reset’). The bug then re-appeared in several other branches in the
application of a fast track fix to the live environment due to its severity.
3.225 The PEAK detail states:
“We are proposing to reset Stock Unit AA back to TP8 BP1, so that the
PM can rollover again, this time with a correct set of figures. Discussed
with Joanne at NBSC Tier 2 and she thinks it would be a sensible way
114 PEAK PC0051855, 5 August 2000 {POL-0226902}
115 PEAK PC0195962, 12 March 2010 {POL-0365857}
116 PEAK PC0128969, 17 November 2005 {POL-0299414}
Prepared by: Jason Coyne fe)
Occupation: Partner ae
Specialist Field: IT Systems It ia Uf )
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0069
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 66 of 265
3.226
3.227
3.228
3.229
3.230
forward. Phoned the branch but the PM is on holiday. Spoke to a relief
PM and advised her not to rollover into another BP until we have sorted
it out.
OCP12388 raised, awaiting approval by POL.”
It has not been possible to review the detail of the OCP as they were
disclosed on the 25 January 2019 and, given the proximity of this date
to the deadline for my report submission. I have not had time to
consider them as part of this report.
It is assumed however that Post Office approved the OCP since the PEAK
detail further states:
“The reporting FAD has been repaired so I suggest that we close this
PEAK and repoen if it occurs again and or elsewhere.”
Other referenced PEAKs relative to this same issue are noted as, but
not limited to: PC0130275,1!7 PC0130461,18 PCO0130855,11°
PC0135486,1° PC0137766,1*4 PCO137051.172
In summary, the issue observed was stock unit rollovers returning zero
values. This resulted in Subpostmasters’ branch reports returning very
large discrepancies.
In PC0130275 ‘the PEAK detail states:
“.. This has resulted in a gain of approximately £18000.
We are unable to correct the system figures safely. We can however
provide accurate figures for what should have been in the Final Balance
for BB, to enable POL to make the correction perhaps by using a
Transaction Correction.
POL need to make a decision on whether they are able to correct the
problem in this way, however we do not see any other alternative.
117 PEAK PC0130275, 21 December 2005 {POL-0300707}
118 PEAK PC0130461, 29 December 2005 {POL-0300893}
119 PEAK PCO130855, 12 January 2006 {POL-0301284}
120 PEAK PC0135486, 12 May 2006 {POL-0305863}
121 PEAK PC0137766, 21 July 2006 {POL-0308133}
122 PEAK PCO137051, 26 June 2006 {POL-0307421}
123 PEAK PC0130275, 21 December 2005 {POL-0300707}
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
POL-BSFF-0100992_0070
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 67 of 265
3.231
3.232
3.233
Corrective action should be taken before 11th January when the branch
is due to roll into TP10.
The cause of the problem is unknown and is under investigation.”
And further:
“If we get to the problem before the office is rolled we are able to change
objects in the messagestore to reset the stockunit back to the CAP (TP)
rollover trailer. The PM can then rollover. PM should get a large shortage
which cancels out the large gain.
We don't want to be having to do this as making manual changes to the
messagestore is open to error and each time we have to seek
authorisation from POL to make the changes.
If we get to the problem after the office is rolled (as in this call) then we
are unable to correct the system figures safely. Its not been decided how
we get the PM sorted out.”
It therefore appears that aside from instances where a Transaction
Correction might have been issued in order to re-balance the accounts,
the alternative (prior to roll-over fix) was to amend the stock unit /
messagestore data. This illustrates that Fujitsu can and did alter branch
data with any consequent errors not being visible to Post Office or the
Subpostmaster unless they were identified and notified by Fujitsu.
PC0146066!*4 and (cloned) PC0146094'5 relate to an issue where a
Subpostmaster has a negative value discrepancy which is diagnosed as
the reference data for this product being recently removed leaving the
negative holding stranded on the system and preventing the stock unit
rollover.
The cloned PEAK detail is quite limited as the root cause and OCP files
(documenting the actual change detail) are attachments that were not
provided in disclosure. However, it states “Opening figures messages
added using ripostemessagefile to convert the -1p ROL to cash”. It
124 PEAK PC0146066, 15 May 2007 {POL-0316398}
125 PEAK PC0146094, 16 May 2007 {POL-0316426}
Prepared by: Jason Coyne fe)
Occupation: Partner ae
Specialist Field: IT Systems It Jrou f )
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0071
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 68 of 265
3.234
3.235
3.236
3.237
appears that this was likely a modification to the data within the branch
accounts.
PC0152014'*6 dated 2007 relates to an issue in which no settlement
value was written for a product transacted in branch. This caused
discrepancy (as effectively the balancing transaction to net the value
zero was missing). The PEAK detail states:
“The counter problem which caused the first issue has been corrected by
inserting a message into the messagestore, for equal but opposite
values/quantities, as agreed with POL (OCP 17510).
As a result of this corrective action, the net effect on POLFS is zero, and
POLFS figures are in line with the branch. POLMIS received both the
original message and the corrective message.
Once the problem was corrected, there should have been no impact on
the branch. However it has been noted that the stock unit BDC had a loss
of $1000, which was generated after the correction was made. We have
already notified Gary Blackburn at POL (email attached). This appears to
be a genuine loss at the branch, not a consequence of the problem or
correction.”
Further detail within the PEAK states:
“Worth noting that the branch did not have any issues with the
mismatched transactions because this was fixed before they did the roll.
The branch is not aware of this and it’s best that the branch is not
advised.”
a
to
This indicates that there has been more than one Balancing Transaction
applied within Horizon and also, remote corrective actions were applied
without the knowledge of the Subpostmaster.
The ‘Master PEAK’ suggested for this issue is listed as PC01473571
27
However, it appears the PEAKs actually assigned under this Master PEAK
126 PEAK PC0152014, 7 December 2007 {POL-0322311}
127 PEAK PC0147357, 26 June 2007 {POL-0317682}
Prepared by: Jason Coyne fe)
Occupation: Partner gj - .
Specialist Field: IT Systems
On the Instructions of: Freeths LLP <
POL-BSFF-
0100992_0072
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 69 of 265
3.238
3.239
are for different manifestations of the issue (but however suspected as
ultimately the same cause for PC0152014'28). The detail records that
the PEAK is considered low incidence (despite support acknowledging
that this bug causes erroneous transaction data to be written in the
accounts) and could potentially be an Escher bug for which there is no
Escher support contract. Therefore, the suggestion is to create a KEL
and close the call record (It is closed as ‘Programme approved - No fix
required).
Further affected PEAKs listed in the Master PEAK are; PC0152203,1*°,
PC0151724,1° =PC0109649,43! PC0109772,12, PC0114129,133 and
PC0133933!* “etc”. The majority of these relate to anomalies relative
to transactions missing a “mode” attribute therefore being caught by
the TPSC254 report and would not impact the branch accounts as
PC01520141> did, where the settlement was missing, it therefore
appears that PEAKs are grouped and related by KELs despite the bug
presenting different symptoms.
Further, PC015172416 records that the fix applied to the data using the
Transaction Repair Tool (TRT) (for PCO151628?%”) was initially set to the
wrong Transaction Mode ID although it is later stated that mode is
irrelevant, and all POL FS data is now correct.
128 PEAK PCO0152014, 7 December 2007 {POL-0322311}
129 PEAK PCO152203, 14 December 2007 {POL-0322498}
130 PEAK PC0151724, 27 November 2007 {POL-0322025}
131 PEAK PC0109649, 15 October 2004 {POL-0281236}
132 PEAK PCO109772, 18 October 2004 {POL-0281362}
133 PEAK PC0114129, 18 January 2005 {POL-0285672}
134 PEAK PC0133933, 27 March 2006 {POL-0304323}
135 PEAK PCO152014, 7 December 2007 {POL-0322311}
136 PEAK PC0151724, 27 November 2007 {POL-0322025}
137 PEAK PC0151628, 23 November 2007 {POL-0321929}
Prepared by: Jason Coyne fe)
Occupation: Partner ae
Specialist Field: IT Systems It Jrou f )
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0073
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 70 of 265
3.240
3.241
3.242
3.243
3.244
3.245
Further PEAKs relevant to PC0152014 (not listed in the Master PEAK
above) are identified as PC0140063%*, PC0176680"7°, PC0175821
40and PC0151718*41.
PC0151718 is the same issue as PC0152014. PC0140063'%? and
PC0176680'*? detail how the corrective fix would apply to POL FS
accounts but does not document any branch account impact.
PC0175821 indicates further balancing transactions were added to fix
the branch accounts for the branch affected in that particular PEAK.
I have noted that inputting the KEL reference returns many more
related PEAKs (and does so where other KELs are referenced for other
bugs) than those acknowledged in the Master PEAK. I have not been
able to review all of these in the time available.
PEAKs PC0159445144 PC01597021#5and PC0159759"** relate to an issue
where various branches feature on reconciliation exception reports. The
issue is diagnosed as due to changes (CP4461 and CP4616) to the TPS
Harvester (Transaction Processing System that harvests branch
transactions). development had to produce scripts to repair rejected
transactions and apply them to the live environment.
The PEAK detail goes on to state how certain missing transaction data
attributes had to be “invented” in order to process the transactions
where the data was missing. This was authorised through an OCP
request.
PC0159759 states:
“That stupid mails code omitted mandatory fields Startdate,
StartTimeFraction EndDate and EndTimeFraction from four messages. I
138 PEAK PC0140063, 10 October 2006 {POL-0310423}
139 PEAK PCO176680, 4 March 2009 {POL-0346844}
140 PEAK PC0175821, 19 February 2009 {POL-0345994}
141 PEAK PCO151718, 27 November 2007 {POL-0322019}
142 PEAK PC0140063, 10 October 2006 {POL-0310423}
143 PEAK PCO176680, 4 March 2009 {POL-0346844}
144 PEAK PC0159445, 1 June 2008 {POL-0440631}
145 PEAK PCO159702, 6 June 2008 {POL-0329973}
146 PEAK PC0159759, 9 June 2008 {POL-0330030}
Prepared by: Jason Coyne fe)
Occupation: Partner ae
Specialist Field: IT Systems It Jrou f )
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0074
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 71 of 265
3.246
3.247
3.248
have used the TRT to insert suitable values. They should go to POL this
evening.”
It is not clear what “suitable values” were applied using the Transaction
Repair Tool (TRT) in this instance or what process would set out how
support were to derive the values of data that they had to create. Again,
the authorisation to modify data appears to have been granted from
Post Office by an OCP request, as aforementioned, these have not been
considered as part of this report due to the timing of their disclosure.
PC0172841 created 8 January 2009 refers to a defect where if a branch
does not poll (send its data and transactions through Horizon to Post
Office), then any transactions after 36 days could potentially be lost.
The PEAK detail states:
“What does the user have to do to get this problem? A non-polling branch
with txns older than 36 days will potentially lose txns if any result in
exceptions. How does it affect them when it occurs? SSC have to
manually rebuild the SQL insert statements, risking data due to bugs and
mistakes made via SQL.”
The PEAK above therefore indicates that Fujitsu support had the
capabilities to manually rebuild data.
Data Rebuilding
3.249
3.250
PC0057909'47 dated November 2000 refers to an issue occurring as a
result of a branch’s counter base unit replacement. A base unit is
effectively the computing machine that enables the Counter in branch
to operate. The Subpostmistress in this instance identified that some
transactions were missing upon printing reports after the installation
and therefore re-added the transactions. After re-printing, the ‘missing’
transactions had appeared and therefore the Subpostmistress had to
reverse the ones she had added.
Five days after opening the call record and the Subpostmistress chasing
for an update four times, a support team member who cannot
147 PEAK PC0057909, 15 November 2000 {POL-0232732}
Prepared by: Jason Coyne fe)
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0075
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 72 of 265
understand the discrepancies states, "PM has not been contacted,
closing as insufficient evidence.” The defect and root cause is then
updated to “40: General - User”.
3.251 The call is re-opened (and cloned to new PEAK PC0058435*°) three
days later and the diagnosis for the missing transactions and their
sudden re-appearance is confirmed as a communications defect
between the main counter in branch (the gateway counter) and the two
other counters in branch failing to synchronise the correct data.
Therefore, in my opinion the evidence suggests that some issues that
are diagnosed as “user error” were the result of a misdiagnosis.
3.252 Despite the diagnosis there still appears to be unknowns queried by the
support team member:
“Can development please investigate on whether there is a deficiency in
Riposte and what can be done to stop this happening again. Also, need
advice on how to get the messagestores in sync and to include the
missing transactions. I suspect we will need to trash the messagestores
on counters 2 and 3 and insert the missing messages onto counter 1 (or
can the PM get away with inputting the transactions). Some of the
transactions are APS. Also how will this affect their balancing. They are
currently in CAP 34.”
3.253 I assume “trash the messagestores” to mean delete them and
potentially rebuild them.
3.254 After another five days the Subpostmistress calls again for another
update due to concerns about balancing. The following is stated:
“Note to be passed onto customer for balancing: this problem has
occurred with replication before (in essence due to a failure in Riposte for
whatever to replicate back down). It should be perfectly OK to continue
balancing on Nodes 2 or 3 but noton [sic] node 1 where the failure
occurred.
From the Riposte point of view there seems to be a major disagreement
on what the contents of <id:1><Num:510416> for about 50 messages
148 PEAK PC0058435, 15 November 2000 {POL-0232733}
Prepared by: Jason Coyne fe)
Occupation: Partner ae
Specialist Field: IT Systems It }roOu f )
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0076
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 73 of 265
should be. There are minor glitches here and there but this seems to be
the major discrepancy. Therein lies the heart of the matter in that there
are EPOSSTransactions present on node 2's viewpoint, but what appears
to be AP Recovery messages on node 1. This blows my whole
understanding of what Riposte should be handling on our behalf i.e.
replication not deviation across nodes. Passing to QFP for onward routing
to Escher-Dev...
.. I should also add that they should repeat the AP recovery if they can.
The trouble with this scenario is that EPOSSTransactions have occurred
on both sides of the divide, both apprantley on node 1. QFP might also
want to seek the advice of the APS team on this also who might disagree
with the above. The EPOSSTransactions on counter 2 cannot easily be
autorecovered, whereas the APS ones via their recovery tools might be
better equipped. Whatever happens, this bug should end up with Esher-
dev.”
3.255 On 11 December 2000 details Gareth Jenkins states:
“I don't know that I can add anything useful here. This is another example
of recovery having gone wrong after a box swap. It would appear that
Counter 1 (the gateway) had been working normally and communicating
with counter 2 up until a log out on counter 1 at 11:44 on 14/11. Anew
box was installed at about 12:04 that day and for some reason it was
recovered from the Data Centre (which last synchronised at 11:24) rather
than the slave. This resulted in about 50 messages being lost. The
gateway did not communicate with the slave until it had written at least
50 messages (ie until 15:30 with the gateway first being used at 15:09).
For this reason there was no Error indicating a Self Orriginating [sic]
message being found. I also note that having allowed the user to use the
gateway from 15:09 until 15:20 the gateway was rebooted and the user
logged on at 15:30.Other than pursuing the known problem of how do
we handle fouled up recovery (covered by PinICL 52823), I don't think I
can add anything further to this PinICL and so it might as well be closed.
I assume that the missing transactions have been recovered manually.”
Specialist Field: IT Systems
Prepared by: Jason Coyne c
Occupation: Partner a .
itgroup
On the Instructions of: Freeths LLP < =
POL-BSFF-0100992_0077
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 74 of 265
3.256
3.257
3.258
3.259
3.260
3.261
3.262
3.263
3.264
Whilst this PEAK was closed 12 December 2000 as a duplicate of
PC0052823149, there are further peaks that result from hardware:
PC0052823 is a different PEAK focusing on the technical issues at the
heart of the bug.
Meanwhile PC0058435?°° is cloned to PC0059052!* in which the support
team continue investigations.
It appears that the Subpostmistress followed the advice regarding
repeating the Automated Payments (AP) Recovery, as this PEAK then
goes on to state:
“The PM has rolled over and is now in CAP 37.
However, because she had to recover 2 AP transactions in order to
balance her cash account, the 2 customers have been paid twice in error.”
The PEAK concludes with support querying whether the customers have
been paid twice and MSU-Incident Management stating that they see
no reconciliation errors and to close the call.
It is therefore not possible to determine whether the customers where
indeed paid twice or how the Subpostmistress recovered from her
imbalances.
It should also be noted that the Subpostmistress raised this query in
CAP (Cash Account Period 34) and therefore would have had three Cash
Account Periods potentially with a discrepancy whilst the root cause was
determined.
PC0197987!*2 created 20 April 2010 documents:
“Unable to connect to counter to attempt manual rebuild as counter is on site.
Action required Advise PM not to trade at present as he is at risk of data loss -
Node 31 being disconnected means the mirror service is not working and failure
of the main disk could leave him without a backup if the unit has not been
149 PEAK PC0052823, 21 August 2000 {POL-0227701}
150 PEAK PC0058435, 15 November 2000 {POL-0232733}
151 PEAK PC0059052, 5 November 2000 {POL-0232734}
152 PEAK PC0197987, 20 April 2010 {POL-0367856}
Prepared by: Jason Coyne fe)
Occupation: Partner ae
Specialist Field: IT Systems It Jrou f )
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0078
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 75 of 265
replicating with the COR servers. Arrange for engineer to visit site to replace
mirror disk.”
3.265 The fact that this PEAK states “...to attempt manual rebuild” implies that
there was the capability and process of doing so in relation to branch
accounts.
Deletion of Data
3.266 Relevant to Horizon Issue 10, PCO24152815? details an issue requiring
deletion of session data:
“Please raise a TFS call with ATOS and ask POL to formally authorise us
to delete this Health Lottery session so that office is able to use Node: 1
again. This will enable the office to use Node:1 again quickly.”
3.267 It has previously been said by Post Office that whilst Fujitsu could
modify transaction data to perform corrective fixes, they would not have
delete capabilities (see paragraph 9.24 of my original report).
3.268 This PEAK also exemplifies a lack of communication between Post Office
and Fujitsu in that the request for deletion of session data to be granted
was actually also for a secondary branch impacted by the issue.
However, Fujitsu and Post Office appear to spend days (impacting the
Subpostmasters ability to operate the Counter) discussing and clarifying
which branches actually needed the corrective action performing
against them.
3.269 The typical response from Fujitsu where such issues as raised in this
PEAK arise is:
“If there was an uncompleted customer session (basket) when the
counter was removed, this might lead to a financial discrepancy. We
cannot tell whether there was such a customer session, and Fujitsu
Services will not accept responsibility for any potential financial
discrepancy as a result of deleting the user session.”
153 PEAK PCO241528, 3 March 2015 {POL-0410687}
Prepared by: Jason Coyne fe)
Occupation: Partner .
Specialist Field: IT Systems ; It lI
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0079
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 76 of 265
3.270
3.271
3.272
3.273
3.274
3.275
PC0234786!* is a similar PEAK to that above relating to a failed session
requiring Fujitsu to perform deletion of session data however this PEAK
detail does not conclude whether the deletion occurred.
PC0263716!5> is again a similar PEAK in which the fix requires deletion
of data. The PEAK detail specifically documents the command used to
delete the recovery transactions and session data from the Branch
Database in order to remove the data that was restricting the
Subpostmaster from ‘rolling over’ into the next trading period.
The PEAK states:
“Due to the circumstances at the branch this session can be removed but
the branch must be made aware that if there are any losses/gains from
removing it then they will be liable.”
It is not fully clear whether “they will be liable” relates to the branch,
Post Office or Fujitsu.
However, whereas my previous report opines at paragraph 9.43 (page
144) that Fujitsu did have the ability to (potentially) delete transaction
data. I opine that Fujitsu could and did delete transaction data (not least
by the deletion of session data which contained transaction data), and
there is evidence that this occurred on several occasions (not limited to
the PEAKs referenced above).
PC0197592'°¢ dated April 2010 details an error whereby rollover cannot
be completed due to system error. Gareth Jenkins of Fujitsu states:
“What we need to do is the following: (I know the SQL is wrong, but BRDB
Host team can correct it and fill in the gaps.) 1. Update
BRDB_BRANCH_STOCK_UNITS WHERE fad_hash = ??? AND
Branch_accounting_code = 314642 AND stock_Unit = ?DEF? setting
trading_period to 11 2. Delete BRDB_SU_OPENING_FIGURES WHERE
fad_hash = ??? AND Branch_accounting_code = 314642 AND stock_Unit
= ?DEF? trading_period = 12 (Anne asserts that there is one such row
154 PEAK PC0234786, 11 June 2014 {POL-0404158}
155 PEAK PC0263716, 26 October 2017 {POL-0431210}
156 PEAK PC0197592, 12 April 2010 {POL-0367467
Prepared by: Jason Coyne fe)
Occupation: Partner ae
Specialist Field: IT Systems It Jrou f )
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0080
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 77 of 265
with zero value for prod_id = 1. I suggest that this is checked by doing a
SELECT first.
What this will do is re-align SU DEF?s TP with that of the Branch. It should
then be OK to rollover the Branch again. BRDB Host will fix this by OCP.”
3.276 This is indicative that Fujitsu, by creating SQL scripts, could delete
relevant records in order to negate previous operations. Whilst this is
not necessarily deletion of transaction data, it is the modification to
operations that are all intrinsic to transaction accounting.
Peaks with Evidence of Remote Access (Horizon Issue 11)
3.277 PEAK PC02081191%” dated 10 March 2012 is titled "SSC Databases users
3.278
3.279
do not have correct permissions". It records Fujitsu concerns that "SSC
users affected have more access than is required to database resources.
This is contrary to security policy” and further, “The customer is not
aware of this problem or change”.
The PEAK includes a comment from Anne Chambers; “When we go
offpiste we use appsup.”
‘Appsup’ is described briefly in the same PEAK (also including a warning)
by Andy Beardmore in 2011:
“The optional role 'APPSUP' is extremely powerful. The original BRDB
design was that 3rd line support should be given the 'SSC' role (which is
select_any_table + select_catalogue) and only given the optional role
‘APPSUP' temporarily (by Security Ops authorisation) if required to make
emergency amendments in BRDB Live. Since then Host-Dev have
delivered a series of auditable amendment tools for known SSC data
amendment operations in Live, and these are assigned by role to
individual SSC user accounts. As such SSC should not require the APPSUP.
role in BRDB, unless there is an unforeseen update required to Live.
Transferring to Steve Parker for review/assessment... It is a security
breach if any user write access is not audited on Branch Database, hence
the emergency MSC for any APPSUP role activity must have session logs
157 PEAK PC0208119, 1 February 2011 {POL-0441177}
Prepared by: Jason Coyne fe)
Occupation: Partner ae
Specialist Field: IT Systems It Jrou f )
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0081
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 78 of 265
3.280
3.281
3.282
attached under the MSC. Host-Dev previously provided scipts, such as
the Transaction Correction Tool, are written to run under the SSC role
and also write to the audit logs.”
I understand Mr Beardmore to be explaining that APPSUP should not be
used to access the branch database. It was only designed for emergency
amendments to the live branch database but acknowledging that such
action whilst logged is not audited, Mr Beardmore advises that
“auditable amendment tools” are available to SSC.
From the privileged user access logs I can see that APP$UP usergroup
was used 2175 times between 2009 and 2018 with users names;
“ACHAMO1, JCHARO1, CTURRO1,GMAXW01" and others. The evidence
suggests the following:
a. They were making emergency amendments to the live branch
database;
b. There actions when logged on would not appear in the audit logs.
The PEAK specifically references the use of the Transaction Correction
Tool and the access to this and other scripts should be reduced.
Policy Adherence (Horizon Issue 11)
3.283
It appears from review of the PEAKs that require deletion of session
data that Fujitsu typically will not proceed until authorisation is given
and evidence of that authorisation is placed onto the system.
3.284 PC0254133'**dated 22 September 2016, details:
3.285
“One failed session for FAD 266329 Node 3 removed from BRDB, per KEL
surs3213P and MSC Task 043T0092285. Authorised by Mark Wood (Debt
Recovery Manager, Post Office Ltd) Witnessed by Phil Breakspear (Fujitsu
SSC) SSC actions complete: closing Peak and returning call to TFS.”
It is noted however, that not all PEAKs that relate to deletion of data
from the BRDB provide as much detail as the one above, specifically not
158 PEAK PC0254133, 22 September 2016, {POL-0422459}
Prepared by: Jason Coyne fe)
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0082
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 79 of 265
in relation to compliance of policy adherence in respect of receipt of
authorisation from Post Office.
3.286 It appears the above restriction in respect of requiring authorisation and
evidence of it is not exercised at all times. PEAK PC023426715° dated 22
May 2014 which also relates to the requirement to delete a user session
data does not evidence an attachment of authorisation granted from
Post Office. It appears that Fujitsu delete the session data on NBSC
branch support verbal approval rather than requesting evidence be
emailed and uploaded to the system. The typical process that should be
exercised is documented in PC0239932:16°
“..POL will need to authorise SSC to clear the incomplete user session.
The POL authorisation needs to say; "Authorise to delete failed session".
If the authorisation is being sent by email then the original email needs
to be sent to SSC duty manager. Fujitsu Services will not accept
responsibility for any financial discrepancy as a result of deleting the user
session.”
3.287 Further observations in relation to Fujistu permission controls are
documented in Section 5, Sub Section 11, Issue 11.
Limitations of PEAK Records - Examples
3.288 Whilst the PEAKs have provided a further view of bugs/errors and
defects recorded within Horizon, observations arising from the PEAK
records that ought to be caveated are as follows:
Limited Detail
3.289 PC0037445'* dated November 1999 documents an issue where a
Subpostmaster had a gain of £3564.35 in cash and £964.23 in stamps.
The Subpostmaster had an on-screen message reporting memory loss
whilst trying to balance.
3.290 The PEAK detail states:
159 PEAK PCO234267, 22 May 2014 {POL-0403643}
160 PEAK PC0239932, 24 December 2014 {POL-0409173}
161 PEAK PC0037445, 6 November 1999 {POL-0221880}
Prepared by: Jason Coyne fe)
Occupation: Partner ae
Specialist Field: IT Systems It ia Uf )
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0083
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 80 of 265
3.291
3.292
“I have looked through the EventLogs; it is apparent that there was a
Virtual Memory error message written on the date the call was logged.
Development, please ignore the discrepancy part of the call and
investigate ‘Memory Loss' event. EventLogs and MessageStore attached.”
The position appears to be that the event logs were only looked at
following the report of a discrepancy by the Subpostmaster. If the
Subpostmaster had not noted the discrepancy or reported the fault then
it is likely that this defect would not have been detected. It is therefore
unclear what the root cause of the discrepancy actually was, and there
is little detail regarding the causation of the memory issue, the PEAK
defect cause is updated to “14: Development - Code” and subsequently
closed in October 2000.
There is no detail regarding whether information was provided to the
Subpostmaster or how the discrepancy was further investigated.
Inconsistent Advice
3.293
3.294
Another dimension to the risk within Horizon and branch account
integrity relates to how issues were handled and or resolved.
PC0225995' created 30 May 2013 (referenced KEL obengc3348L)
relates to a transaction that initially occurred 28 May 2013 and
appeared in reporting as an unmatched reversal 29 May 2013. The
transaction was reversed by the recovery process due to a counter
communications issue. The initial diagnosis states:
“PM was doing this txn on 28/5/13 @16:58.
However the fast cash settlement failed due to poor comms; connection
timed out. Counter produced zero value disconnected session receipts.
The disconnected session receipts indicated no money should have
changed hands.
On 29/5/13 @16:52, when PM logged into the system the recovery
(system correction) started. The recovery reversed the txn and strangely
enough advised PM to pay £260 to customer.
162 PEAK PC0225995, 30 May 2013 {POL-0395484}
Prepared by: Jason Coyne fe)
Occupation: Partner 1
Specialist Field: IT Systems it }FOk I
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0084
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 81 of 265
Please note that the initial txn on 28/5/13, according to printed receipts
shouldn't have exchanged any cash. However the automated recovery
reversal advised PM to pay the money out. If PM paid the money out then
there would be a cash shortage of the same amount.
I suspect this is a software error where the initial txn had progressed far
enough in the database which caused this £260 reversal but still printed
out initial disconnect session receipts with zero value”
3.295 It is not until four days later after the call has been escalated to 4 Line
Support that the actual diagnosis of the issue is provided by Gareth
Jenkins of Fujitsu. He states:
“Isn't this a case of Rollback Recovery? In that case the Recovery receipt
would indicate that money should be returned, but as the customer
wasn't present there is no one to give it to.
Vani: Please confirm that this was a normal Rollback recovery from the
logs.
I accept that for Rollback recovery we produce a normal Recovery receipt
and it may be a bit confusion [sic], but that is how it has always been
and conforms to the specs in the Recovery HLD (DES/APP/HLD/0083 I
think).”
3.296 Therefore, it appears that not only was the process confusing for the
support teams to appropriately diagnose and inform the Subpostmaster
but it is possible that misleading advice may have also been provided
to them on that basis where other support team members potentially
might have also misinterpreted Horizon procedures.
Delays awaiting Post Office authorisation
3.297 PC0244638'° (amongst others) illustrates the delay incurred when
applying fixes due to the multi-party support team’s involvement and
the delay in gaining approval from Post Office that Fujitsu state is
needed for deleting session data where it may cause a financial
163 PEAK PC0244638, 7 July 2015 {POL-0413670}
Prepared by: Jason Coyne fe)
Occupation: Partner ae
Specialist Field: IT Systems It Jrou f )
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0085
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 82 of 265
3.298
discrepancy in branch. This PEAK is opened 2 July 2015 with a target
closure date of 7 July 2015.
The call relating to this PEAK appears to have been closed on the same
day it was opened after a suitable KEL is found to apply. However, the
PEAK is subsequently reopened by a customer call. It is recorded that
ATOS requested POLs authorisation for the fix to be applied 2 July 2015.
Authorisation was not granted until 28 July 2015. Meanwhile, the
Subpostmaster would not have been able to roll over the stock unit in
order to comply with POLs procedures.
Reliance on Third Party Fixes
3.299
PC0037458!* created 01 November 1999 documents a bug relating to
APS transaction receipt prints that causes the system to ‘crash’ thus
forcing a Subpostmaster to ‘reboot’ the Counter. The PEAK was raised
in November 1999 and was subsequently diagnosed as follows:
“The background to this problem is that, in live, counters hang
occasionally. Other PinICLs have been closed as duplicates of this one.
The symptoms are:-1. A "Please wait while receipts are printed"
message, or 2. A Printer tablet with "printing" message. In both cases
the message does not go away and a reboot is necessary. This has proved
extremely difficult to reproduce at will and, despite Brian's comments,
there is no guaranteed formula for doing so that I can find. However, it
seems to happen most often while a pair of APS receipts are being printed
during an APS transaction and something goes wrong with the tally
printer (e.g. out of paper). Bearing in mind the large volume of APS txns
that are done on a daily basis, it is not surprising that such a problem
would be found there. I have no idea what Escher have purported to have
fixed. Unfortunately, I cannot test this out for APS transactions in the link
test environment because of the secure nature of the APS dil. APS is
unavailable to me. However, the "fix" does not seem to have had any
adverse effect on EPOSS receipt/report printing and I am unable to
induce the symptoms described.”
164 PEAK PC0037458, 1 November 1999 {POL-0221867}
Prepared by: Jason Coyne fe)
Occupation: Partner ae
Specialist Field: IT Systems it Jrou f )
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0086
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 83 of 265
3.300
3.301
3.302
In August 2000 the bug is therefore assumed to have been fixed by
Escher and the defect cause is updated to “General - in Procedure” with
the call record subsequently closed. It is noted that due to the length of
the call record being open, the PM (Subpostmaster) was not contacted.
It is unlikely that this issue caused a discrepancy but it is an instance of
error in processing data in Horizon. It further illustrates the risks
introduced within Horizon since Escher were responsible for the fix and
Fujitsu EPOSS team are reluctant to unit test fixes introduced by error.
PC0068699 ‘refers to a “known Escher bug” that duplicates cheque
values and related PEAK PC0068231'® is “...fixed in Build 223 Update
31 which is designated for s10.”.
PEAK closure without identified resolution
3.303
PC0063914'°’ created 15 March 2001 details an issue that has been
reported by several branches whereby opting to press a “preview”
button for the trial balance report results in a stock unit roll over to a
new cash account period. It is recorded that:
“This type of problem has been reported from more than one PO.
Please see PinICL: PCO056710, pc0063957 and a KEL: PSteed34T.htm.”
3.304 It appears that the support team do not understand the problem:
3.305
“Having spent a few days on this (as has Alex Kaiser in previous
incarnations of this problem) I have no choice but to pass back as
‘insufficient evidence’ but would ask that EDSC keeps an eye out to see
If any patterns arise or any sign of the problem actually being reproduced
at will. Clearley [sic] we need to keep an eye on this type of problem.
The systems we have tried to reproduce on contains additional bug fixes
which might be preventing us to reproduce the problem. On the other
hand when these fixes are released to PO's then problem might go away.”
I have previously seen this terminology where the support team claim
an issue cannot be investigated due to insufficient evidence provided by
165 PEAK PC0068699, 4 August 2001 {POL-0242869}
166 PEAK PCO0068231, 20 July 2001 {POL-0242467}
167 PEAK PC0063914, 15 March 2001 {POL-0238446}
Prepared by: Jason Coyne fe)
Occupation: Partner ae
Specialist Field: IT Systems it }FOk I )
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0087
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 84 of 265
the caller (Subpostmaster or NBSC). However, it is implied here due to
investigations not being able to identify the root cause or replicate the
issue.
3.306 Further, it is unclear how many other problems the support team were
unable to replicate or diagnose due to their systems operating bug fixes
that might obscure a specific Subpostmaster’s issue. This is significant
as failure to diagnose a problem will likely result in the Subpostmaster
having to deal with a branch account anomaly.
Managed Service Change Disclosure
3.307 I understand that Managed Service Changes recorded agreed changes
to the Horizon system and service. Following a letter dated 21 December
2018, I was provided with 20,826 Managed Service Change logs.
3.308 By requesting the MSC data, my intention was to review any significant
changes to the Horizon system that might indicate where changes had
been performed due to bug/errors and/or defect fixes applied to the
Live service.
3.309 I received instruction in relation to how to interpret the various files
(see attached at Appendix A). However, these instructions were
insufficient and, upon receipt of the data, the analysis I have been able
to perform was limited by the following:
3.310 The logs are very difficult to read, the first document received?® starts
with an ID of 60460 dated 2006. It is not clear if these records should
have started at 1, but this is the first in the list provided and it relates
to “RequestNo 04330060460”. This file also starts with a reference to
04330060460 and then contains 679,051 lines of text. The third
document received?® also starts at 04330060460 and contains 303,109
lines.
3.311 Given these difficulties, I have not had adequate time to fully analyse
this data. However, I have carried out the following select analysis,
168 MSC_Complete_Data_POA.xlsx, Master Service Change logs {POL-0444102}
169 MSC_RTI_Answers_POA (1).csv, Master Service Change logs {POL-444104}
Prepared by: Jason Coyne fe)
Occupation: Partner .
Specialist Field: IT Systems ; It lI
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0088
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 85 of 265
along with the limited analysis that time has allowed elsewhere in this
report (e.g. section 5, subsections 10 & 11).
3.312 Ihave taken MSC_Complete_Data_POA.xlsx and searched for the words
*‘BRDB’ or ‘message’. There are 183 lines that relate to Fujitsu working
on the Branch database and/or its related hardware platform.
3.313 I have then searched for “FAD”, used by Fujitsu and Post Office’s to
reference to specific branches. 39 records have FAD in the title, whilst
the majority of these appear to be unrelated to this dispute a few titles
warrant further investigation, if time permits:
“FAD 309801 needs BRDB correction to allow branch to rollover into new TP”
“FAD 184937 needs BRDB correction to allow branch to rollover into new TP”
“FAD 379704 needs BRDB correction to allow branch to rollover into new TP”
“FAD 010007 CURRENT_TRADING_PERIOD for stock unit DEF to be changed from 4 to 3”
“FAD 009641 CURRENT_TRADING_PERIOD for stock unit DEF to be changed from 4 to 3”
“FAD 311201 CURRENT_TRADING_PERIOD for stock unit DEF to be changed from 4 to 3”
3.314 I have searched for PCO* (typically PEAKs start with PCO) in the
“OriginatorRef” column. I have identified that 455 of the MSC’s records
contain a reference to a PEAK. This may be consistent with Horizon
changes as a result of PEAKS.
3.315 The following may be of interest but have not been fully considered:
“Removal of Cash Declarations for Deleted Stock Units” - relating to PEAK PC0199654 and KEL
acha3347Q refer to at 5.424a of this report
“Actions to rectify Streams Duplicate Data Errors” - relating to PEAK PCO200596
“Branch Database - Tidy-up Branch Declarations that are older than 01-JAN-2011” - relating
to PEAK PCO211010
Prepared by: Jason Coyne
Occupation: Partner : .
Specialist Field: IT Systems itg rou )
On the Instructions of: Freeths LLP
POL-BSFF-0100992_0089
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 86 of 265
“Generate missing TAs for FAD 020511 [POL Info Only]”- Relating to PEAK PC0220393 and
MSC04330355958 and involved the SQL insertion of “dummy transactions” into the Branch
Database
Privileged User Log Disclosure
3.316
3.317
3.318
3.319
3.320
As aforementioned, Privilege User logs were provided by the Defendant
following a letter dated 21 December 2018 (see Appendix A). The
purpose of the request was to answer of Issue 12 (how often facilities
were used that could alter branch accounts).
Following a letter from the Defendants dated 21 December 2019 I was
provided 81,958,608 lines‘”° of Privileged User Logs.
The letter provided by the Defendant sets out that Privileged User logs
have only been provided back to 2009, as Fujitsu cannot provide data
prior to that. The letter further sets out some typical USERIDs and their
capabilities:
OPSS$BRDB user is stated in the letter as pertaining to the user/schema
that holds the data tables that hold branch accounting data.
Review of access has identified the following:
2009 145 different times
2010 1133 different times
2011 435 different times
2012 396 different times
2013 309 different times
2014 8 different times
2015 99 different times
2016 31 different times
170 More POL-*.txt I we -I on the files POL-0444105.txt to POL-0447287.txt
Prepared by: Jason Coyne
Occupation: Partner .
Specialist Field: IT Systems itg if
On the Instructions of: Freeths LLP =
oup
POL-BSFF-0100992_0090
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 87 of 265
2017 17 different times
2018 (No logs after April 2018) 19 different times
3.321 The relatively low number of different access times suggests that these
accesses are by a human, but there is no evidence in the logs to display
what changes were made during any the access session.
3.322 The user OPS$OGGADMIN is said to have:
“wide ranging access e.g UPDATE ANY TABLE (i.e ability to update any
table within the database, irrespective of the user/schema e.g anything
in OPS$BRDB)
DELETE ANY TABLE (i.e ability to delete from any table within the
database, irrespective of the user/scheme e.. anything in OPS$BRDB”
3.323 From my analysis I have determined that the logs which start in 2015
display access was provided to the OPS$OGGADMIN user in the
following years:
2015 88231 different times
2016 141954 different times
2017 141622 different times
2018 (No logs after April 2018) 67806 different times
3.324 The high number of different access times suggests that the many of
the accesses are likely by an automated process, but there is no
evidence in the logs to display what changes were made due to any
such access.
3.325 b. The user LVBALUSERS is said to have:
“High Level of access...mostly INSERT & SELECT privileges on the
OPS$BRBD tables, some UPDATE ability and DELETE abilitiy on 4 tables”
3.326 The logs which start in 2009 display that access was provided to the
LVBALUSERS users frequently. The letter of the 21 December 2019
suggests that Users are “coming in from the OSR applications on the
Prepared by: Jason Coyne fe)
Occupation: Partner ae
Specialist Field: IT Systems It Jrou f )
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0091
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 88 of 265
BAL platforms”, and clearly many accesses appear to be automated.
However there are two accesses that appear to be from support team
users, one in 2009 the other in 2012. There is no evidence in the logs
to display what changes were made during any such access session.
3.327 The logs do have an indicator that may suggest that it was a human
(rather than another system) that was accessing with privileged access.
This indicator is called “Terminal” and if present displays a “pts/”
number which I believe relates to the specific terminal used to gain
privileged access to the Horizon. In the logs there are 80 different
privileged USERID’s that had accessed Horizon at some point or another
since 2009, the logs do not show what access rights they had or what
actions they completed whilst they had access. Fujitsu should have a
record of what access rights they have today (possibly even historically)
but it is unlikely that they have any log of what actions were taken by
the human users. This USERIDs are recorded in Appendix C
3.328 I have not had time to fully identify the most relevant USERIDs which
would indicate specific privileged access dates and times where
UPDATE/INSERT/DELETE operations were performed in relation to
branch accounts. Provided more time, I may be able to identify these.
Prepared by: Jason Coyne fe)
Occupation: Partner .
Specialist Field: IT Systems ; It lI
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0092
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 89 of 265
4. Defendant’s Responsive Witness Statements
4.1 I have received the Responsive Witness Statements produced by the
Defendant in relation to the Horizon Issues trial and insofar as they fall
within my area of expertise and experience, and I believe it may assist
the Court, in this section of my report I comment on those statements
below.
Torstein Olav Godeseth
Callendar Square / Falkirk
4.2 In the second Witness Statement of Mr Godeseth *”4dated 16 November
2018, at paragraphs 12 to 13.9 he discusses the “Callendar Square” bug
occurring in 2005. I have set out my observations with regards to the
Callendar Square / Falkirk bug, and the documentation provided as
referenced by Mr Godeseth at 3.34 of this report. In summary, I note
that Mr Godeseth references six documents relating to this bug
comprising of:
a. Two PEAK records: PC01256771”2 created 8 September 2005 and
PC0126376!73 created 21 September 2005;
b. Charles McLachlan report dated 10 October 2010;17*
c. The Witness Statement of Gareth Idris Jenkins dated 08 October
2010 (prepared for the criminal prosecution trial of Ms Seema
Misra);*75 and
d. Two KELs JSimpkins338Q!”6 and JBallantyne5245K!77,
171 {Second Witness Statement of Torstein Olav Godeseth, 16 November 2018}
172 PEAK PC0125677, 8 September 2005 {POL-0296154}
173 PEAK PC0126376, 21 September 2005 {POL-0296843}
174 Final report of Charles Alastair McLachlan, 30 September 2010 {POL-0011264}
175 Witness Statement of Gareth Idris Jenkins dated 08 October 2010 {POL-0017084}
176 KEL JSimpkins338Q 10 May 2005 last updated 11 January 2010 {POL-0444055}
177 KEL JBallantyne5245K 2 November 2000 last updated 7 July 2005 {POL-0444056}
Prepared by: Jason Coyne
fe)
Occupation: Partner : .
Specialist Field: IT Systems ) It 1 ¢
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0093
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 90 of 265
4.3 My observations in respect of the documents and information referenced
by Mr Godeseth are as follows:
a. Mr Godeseth references the witness statement of Gareth Jenkins‘’®
in relation to the description of the Callendar Square issue however
the majority of the rest of his testimony relies upon “speaking”
with Mr Jenkins, which does not refer to any supportive documents
that I can review and analyse. For example, at paragraph 13.5 Mr
Godeseth reports that he is aware that Subpostmasters were
provided with advice (which he has not seen) and therefore which
I cannot review.
b. Within the Witness Statement of Mr Jenkins (upon which Mr
Godeseth is relying), there are no recorded PEAK or KEL
references, it is therefore difficult to understand how Mr Godeseth
identified those related PEAKs and KELS that he has, or assessed
the references provided captured the full extent of the Callendar
Square issue.
c. I note that the KELs referenced by Mr Godeseth were not provided
in the initial KEL disclosure which I received on 10 May 2018 and
were only provided as responsive evidence.
d. Mr Godeseth explains that the Callendar Square “bug” occurred in
2005. Whilst that incident at that particular branch may have
occurred in 2005, the KELs he refers to in association with the issue
span from 2000 to 2010. Further, the absence of a full Impact
Assessment in relation to this bug indicates to me that it is highly
likely this bug could have been impacting Branch Accounts prior to
2005.
4.4 In relation to the Callendar Square bug Mr Godeseth further sets out at
paragraphs 15 and 16 that he understands from Matthew Lenton (of
Fujistu) that this bug affected thirty branches, resulting in mismatches
at twenty. He does not identify the branches, provide the date(s) they
178 Witness Statement of Gareth Idris Jenkins dated 08 October 2010 {POL-0017084}
Prepared by: Jason Coyne
fe)
Occupation: Partner : .
Specialist Field: IT Systems ) It 1 ¢
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0094
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 91 of 265
were affected, or the sums concerned. He also does not state how the
Subpostmasters were advised at the time or dealt with the subsequent
mismatch imbalances. I have noted that PC0126376?79 states that any
duplicate transfers made as a result of a Subpostmaster retrying the
transfers would be dealt with via a Transaction Correction, but I have
not seen any evidence that this was done or when or what
Subpostmasters were told.
4.5 Mr Godeseth does not clearly explain the process by which the affected
branches were identified. He states that event logs and reconciliation
processes would indicate an issue which would in turn flag a PEAK and
be visible to Fujitsu, but it is not clear from Mr Godeseth if this means
that it would only have been visible if the Subpostmaster had identified
it and a PEAK had been created. Mr Godeseth does not provide the PEAK
references for all affected branches aside from the two he references at
paragraph 13 of his witness statement (PC0126376'®° and
PC0126042'*!), which do not seem to account for 30 branches which he
describes as being affected.
4.6 It is also unclear how (of the ten branches which did not display a
mismatch) any reconciliation measure would isolate those in relation to
such a bug or make them identifiable to the Subpostmaster, via a
receipts and payments mismatch (paragraph 13.6), when symptoms of
the underlying bug did not always manifest in such a way.
4.7 I have previously requested from Post Office (via the RFI - See Annex
A) further specific detail in relation to this bug, such as how
Subpostmasters were informed and whether there was a full Impact
Assessment available in relation to this (and other) bugs. However, in
response I was referred back to Mr Godeseth’s Witness Statement: “It
is not possible to provide a generic answer to this request - the way in
which the impact of a bug is assessed will depend on the nature,
179 PEAK PC0126376, 21 September 2005 {POL-0296843}
180 PEAK PC0126376, 21 September 2005 {POL-0296843}
181 PEAK PC0126042, 15 September 2005 {POL-0296514}
Prepared by: Jason Coyne fe)
Occupation: Partner .
Specialist Field: IT Systems ; It lI
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0095
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 92 of 265
operation and effects of the bug. Information regarding the ways in
which Fujitsu assessed the impact of the bugs referred to in paragraphs
12 - 16 and 34 - 61 of the second witness statement of Torstein Olav
Godeseth dated 16 November 2018 is provided in those paragraphs. As
illustrated in Mr Godeseth's statement where a bug has been identified,
Fujitsu's approach has been to seek to determine what branch was
affected and to present this to Post Office, along with how they were
proposing to resolve the issue.”This has not satisfied my request since
Post Office have not communicated how Subpostmasters were informed
and Mr Godeseth’s Witness Statement diverges from my findings which
illustrate that in actuality, the bug was likely in operation prior to the
Callendar Square incident.
Payments Made to Incorrect Customer Account
4.8 At paragraph 25 of his second Witness Statement Mr Godeseth!® refers
to this bug in relation to the interface between Riposte and the barcode
reader. The symptoms of the bug were that different transactions would
ultimately go to the same client account. Mr Godeseth does not identify
any documents which might relate to this problem, the impact of the
Horizon code change he refers to, or dates of any events. Without more
information, I have not been able to analyse or opine on Mr Godeseth’s
account that this bug would not have caused a shortfall in branch
accounts. I have however, (in my further PEAK analysis) reviewed a
PEAK in relation to phantom transactions whereby the same customer
account number for a BT payment was recorded against a British Gas
transaction. See paragraph 3.150.
Global Branches
4.9 At paragraph 30 Mr Godeseth responds to statements made in my first
report regarding Global Branches.
4.10 Mr Godeseth states:
182 {Second Witness Statement of Torstein Olav Godeseth, 16 November 2018}
Prepared by: Jason Coyne fe)
Occupation: Partner .
Specialist Field: IT Systems ; It lI
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0096
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 93 of 265
“Mr Coyne's allegation at paragraph 9.18 of his expert report that "An
instance of a global branch would allow Fujitsu to create global users and
to input transactions within core Horizon systems as though they had
been entered from a physical branch" is not correct. To enter a
transaction for a physical branch would mean that Fujitsu would have to
be physically present at that branch.
33. Similarly with paragraph 9.19 of Mr Coyne's expert report, where Mr
Coyne alleges that "It is entirely possible that investigation could be
further conducted by Post Office to identify any transactions held within
the BRDB containing the Branch Codes [...]. Such would identify where
and what transactions had been performed by Fujitsu global branches
and not a Subpostmaster". This allegation is meaningless because to
enter a transaction a global user has to be physically present at a branch.”
Reading Mr Godeseth’s reading of my first report, I realise there is some
ambiguity and I will therefore clarify and explain further the points I
intended to make in my first report.
Global Users may perform transactions whilst physically situated in a
branch, i.e. those of an Auditor. The actions which have been performed
by a Global User will all be auditable in the Horizon systems.
Further, there are Global Users whom have administrator capabilities,
that may log on to a global branch as though it were the physical branch
counter, to perform certain remote administrative activities (not
transactions).
The document ‘HNG-X Counter Business Application Support Guide’!®?
(as referenced in my original report) sets out how ADMIN Users may
use global branches to remotely interact with a branch Counter to
perform Stock Unit and Branch User Management activities and how
they will be recorded within the Horizon systems. Only those with Admin
capabilities can perform Admin activities within the global branch.
183 DEVAPPSPG0017_7.1.doc, HNG-X Counter Business Application Support Guide, 8 January 2014
{POL-0134853}
Prepared by: Jason Coyne fe)
Occupation: Partner ae
Specialist Field: IT Systems It Jrou f )
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0097
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 94 of 265
4.15 Whilst document ARC/SOL/ARC/0061** dated 2009 (referred to by Mr
Godeseth in his first witness statement) states:
“The intention is that Global Users will be managed by HSD staff, thus
utilising the existing Help Desk mechanisms for controlling the activities.
A Global User Administration application is required to enable Global
Users to be managed and also to support the capability of resetting a
Local Branch Manager's password should it be forgotten. It is proposed
that this is achieved by deploying one or more standard HNG-X counters
in the Help Desk location. From an infrastructure perspective, this will be
managed as a normal Branch. However there will be constraints on
the use of the Branch to ensure that no trading can take place at
that Branch and that it is dedicated to Global User
Administration.” [Emphasis Added]
4.16 Despite the intention expressed in this document for there to be these
contstraints, I have formed the view that transaction capabilities were
possible from the global branches situated within Fujitsu’s work space,
for the reasons I explain below:
4.17 Firstly, the evidence of PEAK PC02057251®° dated 2010 states:
“Jon: When you sort out the review comments on in DES/GEN/SPE/0007,
please can you consider the following:
1. Need to add a statement somewhere (probably section 9) as to what
roles can Log on where. I believe the rules are (and this is what Nicola is
after):
a. ADMIN can only Log On in Global Branch
b. All other Roles can Log On to any type of Branch (ie normal, CTO,
4.18 Further, review of design document DES/GEN/SPE/0007_6.218°
(referred to above and also within ARC/SOL/ARC/006 version 6.2 dated
post 2009) illustrates at Section 9 ‘Access Control’:
184 ARCSOLARC0006_1.doc, HNG-X Architecture - Global Users, 15 July 2009 {POL-0440076}
185 PEAK PCO205725, 25 October 2010 {POL-0375491}
186 DESGENSPE0007_6.2.doc, HNG-X Menu Hierarchy and Messages, 5 April 2018 {POL-0153568}
Prepared by: Jason Coyne fe)
Occupation: Partner ae
Specialist Field: IT Systems It Jrou f )
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0098
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 95 of 265
D],Aa,a);SI/e@;w,;eE,;ry;ail;)elsz\i2
Eb [4 = =
</GI/5/8/2\"IS\/m\e/2\e/6
a oe
G\/e)/5/O\S\/Ela\G\e/s/<I8
=)2/ =/Si2 FA FE fs)
Zig < E
Branch Type @ za
Normal Branch YIY JY /Y [Y/Y IY¥ IY¥ IY IY_[N
Training Branch (CTO) YY JY /J/Y¥JY¥{IY {IY IY IY [Y _[N
Global Branch YY I}YIY [Y JY JY [JY JY TY TY
Table 4- User Roles Login Table
Figure 2 Excerpt from design document HNG-X Menu Hierarchy and Messages
4.19 This table is indicative that transaction capabilities (amongst others) for
branches could be performed remotely from global branches if logged
on under the role such as “Clerk” (and/or potentially others that permit
transactional input). See Appendix B for the full list of ‘Role Capabilities’
documented in DESGENSPEO07, Section 9.2.
4.20 lappreciate Mr Godeseth says this is not possible, but I have not seen
any document which reflects a change to the planned design in this
design document above, or any other documentary evidence for the
restrictions that Mr Godeseth says were in place.
4.21 Regarding paragraph 9.19 of my first report (page 139), no records
have been disclosed which reveal the transactions carried out at global
branches or any of the global branch codes. Mr Godeseth states that
branch WAKO1, Branch Code 999993 (which Mr Godeseth didn’t refer to
in his first statement?®” but I identified for my first report) is no longer
used and was closed in September 2016 however, further information
about this branch, e.g. it’s full period of operation, is not provided.
4.22 Paragraph 9.19 of my first report was intended to identify that there
should be records available, that are identifiable by the global branch
ID, in order to establish the activities performed by them.
187 {Witness Statement of Torstein Olav Godeseth, 27 September 2018}
Prepared by: Jason Coyne
Occupation: Partner : .
Specialist Field: IT Systems itg rou )
On the Instructions of: Freeths LLP
POL-BSFF-0100992_0099
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 96 of 265
Receipts and Payments Mismatch
4.23
4.24
4.25
4.26
4.27
4.28
At paragraph 34 of his second Witness Statement Mr Godeseth'®* states
that, in addition to the Callendar Square issue, he has been asked by
Post Office to explain how the following three bugs came to light and
were resolved (payments mismatch, local suspense issue and the
Dalmellington/branch outreach issue).
I have assumed at paragraph 35 under the heading “Receipts and
payments mismatch” that Mr Godeseth is dealing with the “Payments
Mismatch” bug acknowledged by Post Office in their Letter of Response
(Schedule 6)1®° since Mr Godeseth states (which aligns with my own
opinion):
“At the outset it should be noted that while I understand that this bug
has become known as “the receipts and payments mismatch bug”, a
receipts and payments mismatch is actually a symptom of the issue.”
Firstly, I note that whilst Mr Godeseth states that this bug affected 60
branches, the Letter of Response (Schedule 6) document provided by
Post Office in relation to these bugs states that 62 branches were
affected.
Mr Godeseth only refers to one document authored by Gareth Jenkins
dated 29 September 2010! in relation to this issue in his attempt to
explain how this bug came to light and how it was resolved.
Within this 29 September 2010 document there is other information
which Mr Godeseth has not included in his witness statement that in my
opinion is important to take into account.
At Section 3 of the document (Identifying Affected Branches) it states,
“Processes should be in place such that SMC pick up these events and
raise a peak for each occurrence of these events”, then there is a
188 {Second Witness Statement of Torstein Olav Godeseth, 16 November 2018}
189 {Letter of Response from Post Office,"SCHEDULE 6: REBUTTAL OF ALLEGATIONS AGAINST
HORIZON", 28 July 2016}
190 3429 SM BP Correcting Accounts for Lost Discrepancies - 102000790 - CD1.pdf, Correcting
Accounts for "lost" Discrepancies, 29 September 2010 {POL-0010769}
Prepared by: Jason Coyne fe)
Occupation: Partner ae
Specialist Field: IT Systems It Jrou f )
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0100
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 97 of 265
4.29
4.30
4.31
4.32
comment (by Mr Jenkins), “J don’t believe that this has happened and
this needs to be investigated further.
I have not been able to identify 60 PEAKs relating to this bug from the
total set of PEAKs disclosed, which may be because 60 PEAKS were not
created, or may be because the PEAKs which were created were not
clearly identified as relating to this problem. Mr Jenkins in his report
refers to only three PEAK records which are recorded in his report as
follows: "PC0204765 and PC0204263 (and also PC0203864 which is a
duplicate of PCO204263).”
I have set out my observations further in relation to this bug at
paragraph 3.27. Unlike Mr Jenkins, I do not identify PCO2038641"! as a
duplicate of PC0204263'% as it references a differing impacted branch
and values. Also, nested within the PEAKs referenced by Mr Jenkins, I
have identified further related PEAKs to this issue. PC0204537,1%3
PC02048891%4 and PC0205076.1°° I have also identified that KELs
wrightm33145J!% and ballantj1759Q?%” are referenced within the PEAKs
At section 4 of the Jenkins document (Analysis Required for each
Affected Branch), it is set out that several items need to be ascertained
such as dates, values and whether a call was raised by the branch. This
is the sort of analysis I would expect to see as part of an Impact
Assessment (as I stated previously, I would have expected to see one
for Callendar Square also), but I have not seen any evidence of analysis
or the results thereof documented by Post Office or Fujitsu in any
detailed level.
At section 6 of the same document,'*® (Communication with Post Office
Ltd) there is reference to Fujitsu communicating the problem to Post
191 PEAK PC0203864, 2 September 2010 {POL-0373654}
192 PEAK PC0204263, 13 September 2010 {POL-0374051,
193 PEAK PC0204537, 17 September 2010 {POL-0374316}
194 PEAK PC0204889, 30 September 2010 {POL-0374664}
195 PEAK PCO205076, 6 October 2010 {POL-0374849}
196 KEL wrightm33145J, 23 September 2010 last updated 1 April 2016 {POL-0040409}
197 KEL ballantji759Q, 12 February 2010 last updated 17 May 2011 {POL-0038508}
198 3429 SM BP Correcting Accounts for Lost Discrepancies - 102000790 - CD1.pdf, Correcting
Accounts for "lost" Discrepancies, 29 September 2010 {POL-0010769}
Prepared by: Jason Coyne fe)
Occupation: Partner ae
Specialist Field: IT Systems It Jrou f )
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0101
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 98 of 265
4.33
4.34
Office through “the problem management mechanisms”. I haven’t seen
any records which show how or when this was done in this case,
although I would expect this to be documented.
There is also reference to the value of the discrepancies which had been
identified by that date, which are not separately listed, but are described
as “one for £30,611.16, one for £4,826 and the rest are all for less than
£350", there is then a comment (again Mr Jenkins specific input) “I’ve
been unable to work out yet if these are losses or gains!".
To conclude, I cannot be confident, due to the limitations regarding the
documentation set out above, that the full extent or impact of this bug
was Suitably assessed.
Local suspense issue
4.35
4.36
4.37
At paragraphs 46 - 54 of his second Witness Statement, Mr Godeseth
refers to the ‘Local suspense issue’ that was identified in 2013,by
reference to a note prepared by Gareth Jenkins headed ‘Local Suspense
Problem’ (which in the exhibit has a date of 16 November 2018, but I
understand this date is incorrect) and PEAK PC02238701%° dated 25
February 2013 (which is referenced within the ‘local suspense issue’
document by Mr Jenkins).
This ‘Local Suspense Problem’ note provides further information about
the problem and identifies the affected branches. It appears from this
document that there was a specific investigation into this problem and
that other documents were created (including a preliminary report) but
these are not identified by Mr Godeseth.
There are some very important points which arise from Mr Godeseth’s
description of this bug, including that: (1) it appears that Post Office did
not take any steps to identify the cause of the problem when it first
arose, or tell Fujitsu about it (2) it is apparent that Post Office and
Fujitsu were reliant upon Subpostmasters identifying this problem
199 PEAK PC0223870, 25 February 2013 {POL-0393383}
Prepared by: Jason Coyne fe)
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0102
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 99 of 265
4.38
4.39
4.40
4.41
4.42
4.43
rather than any independent monitoring of branch accounts, and (3) it
appears that the reason the affected Subpostmaster was able to identify
the problem was because the discrepancy value was so high in his case,
but the fact is there was an effect on other Subpostmasters that was
not identified by them or Post Office in the first instance.
The way in which this bug arose (as described at paragraph 48.2 of Mr
Godeseth’s Witness Statement) indicates that there was a lack of
effective regression testing of this fix.
At paragraph 52 Mr Godeseth states that the old records which caused
the issue were deleted but does not explain the process used to delete
these records, i.e., whether it was by privileged user. No records have
been disclosed which show how these deletions were made.
At paragraph 53 Mr Godeseth states that further checks were introduced
during the balancing process to identify recurrence and raise alert, but
there is no description of what those checks were and again no
documents have been disclosed which describe these.
I note that within the ‘Local Suspense Problem’ note there is draft text
of a letter to be sent to the Subpostmaster. Mr Godeseth does not say
within his Witness Statement whether this letter was in fact sent and I
have not seen any copy of any communication to the affected
Subpostmasters from Post Office.
In a Request for Information (RFI) document sent to Post Office (Annex
A) I have previously enquired as to how Subpostmasters were notified
about the Local Suspense Account problem. Post Office responded on
the 8 August 2018 stating:
“SPMs were notified about the "Local Suspense Account” issue.”
I have provided fuller observations in respect of this bug at paragraph
3.43 above. In summary, since Fujitsu did not investigate this bug when
it first arose in 2011, there is no clear record of what the full impact of
its effects were. The documentation referenced here only relates to the
incident that occurred in 2012.
Prepared by: Jason Coyne
fe)
Occupation: Partner : .
Specialist Field: IT Systems : It FC
[e)
On the Instructions of: Freeths LLP
POL-BSFF-0100992_0103
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 100 of 265
Dalmellington / Branch Outreach Issue
4.44 Ireferred to the Dalmellington issue in my first report (paragraphs 5.16
to 5.19 pages 45 to 46) after identifying some emails relating to a
problem affecting this branch where a cash pouch was remmed in to an
outreach branch multiple times causing a £16,000 discrepancy in
Subpostmaster’s branch accounts. It appears the bug was related to a
“log off issue” but not one caused by any Subpostmaster user, this is
supported by Mr Godeseth who states that the stock unit in question
timed out and logged off due to inactivity.
4.45 This was not one of the bugs which Post Office had acknowledged in its
Letter of Response?°° or to my knowledge, otherwise in these
proceedings. Mr Godeseth now deals with this bug in his witness
statement and refers to a Fujitsu presentation dated 10 December 2015
headed “Branch Outreach Issue (Initial Findings)”.
4.46 I have also, through my own analysis found related PEAKs to this issue.
PEAK PC02472072° states that this issue may have existed within
Horizon for:
“..several years so it likely to have happened before but we have no
record of it having been reported to us. I can only check back two
months; I’ve found 4 other instances (outreach branches 214869,
106444, 110444, 207828)...”
PEAK PC0246949?° (October 2015) further states:
“Note: NBSC has confirmed that they following discussions and checks
with the user that this is not a user error issue, but an issue within the
system requiring Fujitsu investigation.”
4.47 The investigation notes contained within PEAK PC0246949 also illustrate
the potential for misunderstanding between Post Office and_ its
200 {Letter of Response from Post Office, SCHEDULE 6: REBUTTAL OF ALLEGATIONS AGAINST
HORIZON, 28 July 2016}
201 PEAK PC0247207, 21 October 2015 {POL-0416073}
202 PEAK PC0246949, 13 October {POL-0415840}
Prepared by: Jason Coyne fe)
Occupation: Partner ae
Specialist Field: IT Systems It Jrou f )
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0104
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 101 of 265
subcontractors since email exchanges outside of the PEAK detail states
that Chesterfield (Post Office) have previously been aware of this
issue.?0
4.48 Whilst Post Office agreed that this Horizon bug needed to be fixed, the
issue is logged as a “Process” issue and NBSC staff are advised how to
work around it.
4.49 The ‘Branch Outreach Issue (Initial Findings)’ presentation?™ referred
to by Mr Godeseth indicates that an audit identified 112 occurrences of
duplicate pouch IDs in relation to this issue overall, of which 108 were
corrected “at the time” either by a Transaction Correction issued by Post
Office or the Subpostmaster completing a reversal. I do not know the
circumstances which led to Transaction Corrections being issued to each
of these Subpostmasters (where this was done) and whether this
required the Subpostmaster to identify the specific problem to Post
Office. However, it seems likely this was the case as the records indicate
this is what happened at Dalmellington. I note that Mr Godeseth says
that in the case of the Dalmellington branch a Transaction Correction
was issued prior to the completion of the Branch Trading statement on
29 October 2015 (paragraph 60).?°°
4.50 Many branches experienced this effect on more than one occasion, as
is apparent from page 10 of the Branch Outreach Issue presentation,
which lists how many occurrences each branch had.
4.51 Whilst Mr Godeseth states in his witness statement that there were 112
incidences of duplicate barcodes issued, he does not explain that of
those 112, the presentation refers to there being: "4 items still to be
confirmed” and “No correction records obvious in database Post Office
to advise if any corrections etc raised”. This suggests that there were
four occasions of duplicate pouches affecting branches which were not
203 Email thread between ATOS and CWU, 23 October 2015 {C-0005343}
204 Outreach BLE Extract Findings v6 091215.pptx, Branch Outreach Issue (Initial Findings), 10
December 2015 {POL-0220141}
205 {Second Witness Statement of Torstein Olav Godeseth, 16 November 2018}
Prepared by: Jason Coyne
fe)
Occupation: Partner : .
Specialist Field: IT Systems ) It 1 ¢
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0105
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 102 of 265
corrected at the time either by a reversal or by a Transaction Correction.
The detailed preliminary findings within the report refer to two of these
“unknown” occasions occurring in 2013:
a. FAD 157242, value £25,000 (on 18 February 2013); and
b. FAD 209311 value £2,500 (on 1 March 2013).
4.52 In my opinion, I believe it should have been relatively easy for Fujitsu
or Post Office to review the branch data for those branches (including
branch trading statements) to identify how the amounts were resolved
or otherwise treated in the accounts i.e. whether they were settled
centrally and ultimately, if so, the Subpostmaster was held liable for
them. It is not clear from the Fujitsu presentation or Mr Godeseth’s
witness statement whether there was in fact further communication
with those Subpostmasters or if any previous errors were corrected
following the audit referred to.
4.53 I have searched the disclosed PEAKs relating to these FAD codes and
although I have found PEAKS relating to other issues for those
branches, none of these PEAKs appear to relate to the dates and
amounts which are identified in the Fujitsu presentation as being related
to the Branch Outreach issue.
4.54 I have looked at the “2015 POA Problem Management - Problem
Review” dated 6 July 2016,2°° which refers to this problem with code
A10821106. This records that a “regression” test for this type of failure
could be run on new releases before they are “released into the Live
environment” and that “a regression test has been added to the LST
test suite to validate for this scenario in future releases”. I agree that it
was appropriate to improve the regression testing once this error was
identified but the fact that this error arose indicates that there were
initial failures in testing.
206 SVMSDMINR3037_1.DOCX, 2015 POA Problem Management - Problem Review, 6 July 2016
{POL-0146645}
Prepared by: Jason Coyne
fe)
Occupation: Partner : .
Specialist Field: IT Systems ) It 1 ¢
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0106
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 103 of 265
Post Office Account Customer Service Management Procedure
4.55
4.56
4.57
In my first report I referred to a document entitled ‘Post Office Account
Customer Service Problem Management Procedure” dated 12 July
2016 (which was version 5 of this document). Mr Godeseth in his
statement refers to a later version of the same document, dated 15
September 2017 (version 5.2). I had anticipated when preparing my
first report that the procedure would have been implemented, because
the rationale for the process was to “measure effectiveness of the
process and drive performance of the process and overall service in
general”. However, Mr Godeseth’s Witness Statement says the
procedure was not implemented, but does not explain why, other than
to say “I understand from Steve [Bansal] that Saheed Salawu’s
replacement did not wish to implement the changes”. In my opinion this
decreases the extent to which measures and controls existed in Horizon
to prevent any bugs/error or defects as an issue was recognised and an
important measure and control was not implemented.
At paragraphs 64 to 65 Mr Godeseth specifies that during Legacy
Horizon, Problem Management was reported in a specific section within
the Service Review Book (SRB). In his next paragraph Mr Godeseth
states that there was no Problem Management reporting between
September 2010 to September 2014, but there were annual Problem
Review Reports produced for the years 2014 to 2017 which Mr Godeseth
has identified.
These documents do not contain the metrics or degree of scrutiny of the
problems which arose, or the management process which was
envisaged by the Post Office Account Customer Service Problem
Management Procedure. An example of the level of scrutiny contained
within these reports is the way that the Branch Outreach problem is
described (as mentioned above), where e.g. the numbers of affected
branches, the time taken to identify and resolve the problem, and the
207 SVMSDMPROO025_5.doc, Post Office Account Customer Service Problem Management
Procedure, 12 July 2016 {POL-0146787}
Prepared by: Jason Coyne
fe)
Occupation: Partner : .
Specialist Field: IT Systems : It FC
[e)
On the Instructions of: Freeths LLP
POL-BSFF-0100992_0107
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 104 of 265
4.58
4.59
prospect of there being affected branches without resolution, is not
identified or explained.
Paragraph 67 refers to the “Problem Review Tracker”. This document
was only provided to me on the 11 January 2019 so I have not yet had
an opportunity to consider it in full. However, I do refer to specific
problems reviewed in the tracker at 5.144, 5.177 and 5.281 below.
Paragraph 68 refers to the Major Account Control team (MAC), and a
related process "Flow for Incident Life Cycle”. These are very recent
documents, dated November 2018 and August 2018 (they were not
disclosed prior to Post Office’s responsive witness statements), and it is
not clear to me why these processes were introduced or what changes
these documents introduced.
Tracy Jane Wendy Mather
Credence
4.60
4.61
At paragraphs 9 to 17 of Ms Mather’s witness statement?° she deals
with her experience as an end user of Credence. She states at
paragraph 14 that she has never heard of a bug in Credence in her time
at Post Office - I explain below at 5.54 and 5.131 how Credence is used
by Post Office to attempt to validate branch accounts but contains
insufficient audit data for that purpose.
At paragraph 15 Ms Mather states:
“Looking at the Helen Rose report referred to in paragraph 5.49 of Mr
Coyne’s report, Post Office was able to use Credence to identify that a
Subpostmaster had reversed a transaction but had also taken £76.09
payment from the customer. In reversing the transaction, the
Subpostmaster had effectively removed the payment to British
Telecomm, [sic] making the bill unpaid.”
208 {Witness Statement of Trace Jane Wendy Mather, 16 November 2018}
Prepared by: Jason Coyne fe)
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0108
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 105 of 265
4.62 Firstly, the comments from Gareth Jenkins within the Helen Rose
report,?°? convey that it was a system reversal, not a Subpostmaster
initiated reversal. The feedback further states:
“It isn’t clear what failed, but if it was a comms error, then the system
would have printed a disconnected session receipt and the Clerk should
have given the customer £80 and told him his Bill was unpaid. The fact
that there is no indication of such a receipt in the events table suggests
the counter may have been rebooted and so perhaps may have crashed
in which case the clerk may not have been told exactly what to do.”
4.63 Therefore, the contemporaneous evidence is consistent with the
determination that Horizon initiated the reversal, NOT the
Subpostmaster. In my first report I had explained (at paragraph 4.61)
that the Subpostmaster had not reversed the transaction, this had been
a reversal generated by the system as part of recovery. Credence data
appeared to show (or was interpreted as) being a reversal initiated by
the Subpostmaster. This difference of position arose from Post Office
looking at Credence data and Gareth Jenkins of Fujitsu looking at audit
data and system logs. This demonstrates two positions:
a. Credence data, most commonly used by Post Office for their
investigations, is either wrong or does not provide sufficient
information to complete the full picture; and
b. It was only after the Subpostmaster involved an external forensic
accountant that the Audit data was requested.
4.64 The conclusion of the Rose report itself does suggest the possibility of
losses occurring as a result of this issue and Subpostmasters being
considered liable for a loss that ultimately arose from a Horizon initiated
event. The report states that a change should be made to the system
to make system created reversals clearly identifiable to both Fujitsu and
Credence.??°
209 comments from Gareth Jenkins within the Helen Rose report
210 Horizon data Lepton SPSO 191320 CONFIDENTIAL.DOCX, Horizon data - Lepton SPSO 191320,
12 June 2013 {POL-0221677}
Prepared by: Jason Coyne fe)
Occupation: Partner 1
Specialist Field: IT Systems it }FOk I
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0109
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 106 of 265
ARQ Requests
4.65 Ms Mather states at paragraph 18:
“I understand that Mr Coyne has alleged that Post Office staff were
deterred from making ARQ requests because of fees or penalties.”
4.66 Ms Tracy Mather does not reference a particular paragraph of my first
report, I do not believe this is stated at any point within it.
4.67 In her statement Ms Mather references the number of ARQ requests per
year. If it is correct that the contractual limit of 720 per year has never
been exceeded except for this litigation, then in my view Post Office is
not utilising the audit data sufficiently and certainly is not checking the
audit data prior to issuing Transaction Corrections.
4.68 In 2011/2012 using the figure that Dr Worden produces at his Table 9.3
(section 9.6, page 208) there were 107,583 Transaction Corrections but
only a fraction?! of these were validated by the audit data. This is
consistent with Dr Worden’s position at his paragraph 1086 where he
states:
“When Post Office is investigating anomalies reported by Subpostmasters
they use Credence and their other management systems in the first
instance — but, when they need to confirm the transactions handled in a
branch, they can also ask Fujitsu to retrieve the corresponding data from
audit.” [[See Horizon Issue 8]}:?!?
Angela Margaret Van Den Bogerd
4.69 Mrs Van Den Bogerd has provided a witness statement?!? commenting
on individual cases and various disparate factual matters, which I do
not attempt to comment on in detail here. I note the following discrete
points:
211 Less than 0.67% of the total Transaction Corrections could have been investigated with Full
Audit if less than 720 ARQs were requested by POL
212 {Witness Statement of Andy Dunks, 16 November 2018}
213 {Second Witness Statement of Angela Margaret Van Den Bogerd, 16 November 2018}
Prepared by: Jason Coyne fe)
Occupation: Partner ae
Specialist Field: IT Systems It Jrou f )
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0110
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 107 of 265
Phantom Transactions
4.70 At paragraph 13 and the further discussion of potential “The Phantom
Transaction” in relation to Mr Singh (paragraphs 35 to 50). I have seen
evidence of phantom sales recorded in the disclosed documents. PEAKS
PC00650212'4 and PC0052025.7!5 (documented in further detail at
Section 3, ‘Phantom Transactions (Horizon Issue 4)’ above) refer to
phantom transactions in branches, the former which was observed by
an engineer on site at the branch and the latter which refers to
discrepancy arising from them.
Transactions not associated with a Subpostamster’s ID
4.71 At paragraph 18.4 Mrs Van Den Bogerd refers to transactions inserted
by SSC as being “clearly identifiable in the audit trail as having been
inserted by SSC". I disagree that transactions inserted by SSC would
have been clearly identifiable by a Subpostmaster or other person
inspecting the Subpostmaster’s accounts. Mr Parker in his second
witness statement says:
“Transactions injected into a counter would appear on the transaction
logs available on Horizon as if it had been carried out by the user that
was logged into the counter at the time...”
4.72 Even if the transaction had a counter position over 32 (because it had
been inserted at the correspondence server (as stated by Mr Godeseth
in his first witness statement), finding this would require the
Subpostmaster/inspector of the accounts to review the relevant record
where this is shown, to know what time/date such an activity occurred
or if this had occurred at all, and to know the significance of the counter
position. If the transaction had been inserted at the counter, appearing
at the normal branch counter position, this would be very difficult for a
Subpostmaster to find without specific, precise knowledge of what had
been done and when. Finding this would require the action and process
214 PEAK PC0065021, 17 April 2001 {POL-0440162}
215 PEAK PC0052025, 9 August 2000 {POL-0227064}
Prepared by: Jason Coyne fe)
Occupation: Partner .
Specialist Field: IT Systems ; It lI
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0111
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 108 of 265
of dispute/investigation to be conducted and ultimately likely that the
audit file would need to be consulted for this precise period to find what
had happened.
Foreign Currency Transactions
4.73
At paragraph 19 Mrs Van Den Bogerd refers to foreign currency
transactions, I have located evidence within the PEAKS that Horizon has
suffered from bug/errors and defects causing Bureau discrepancies.
These appear in this report starting at 3.140 above.
Further Technical Issues
4.74
4.75
Regarding the various references to the recovery process in relation to
Mr Singh (paragraph 53), Mr Tank (paragraph 78), Mrs Burke
(paragraphs 103 to 110) and Mrs Stubbs (paragraphs 118 to 119) it is
apparent that several Subpostmasters had problems following
connectivity issues and also following the recovery process described by
Mrs Van Den Bogerd. Incidents in relation to recovery and its failed
procedure are documented above at Section 3 (‘Recovery Issues’ and
‘Recovery Failures’).
I note that in relation to the Transaction Acknowledgement process for
lottery introduced after 2012, Mrs Van Den Bogerd describes a data
entry error by Post Office affecting Mr Latif which caused the stock of
scratch cards to decrease rather than increase (paragraph 98). This
example illustrates the potential for errors in branch accounts to be
introduced by the Transaction Acknowledgement process. The same
potential is evident for Transaction Corrections as the two processes are
similar in operation and impact. Mrs Van Den Bogerd says that the
Subpostmaster could have noticed that the Transaction
Acknowledgments were not for a positive number and could have
challenged them at that point, but it is not clear to me how obvious it
would have been to the Subpostmaster that the Transaction
Acknowledgement was incorrect, or what the dispute process was in
relation to a Transaction Acknowledgement.
Prepared by: Jason Coyne
fe)
Occupation: Partner : .
Specialist Field: IT Systems : It FC
[e)
On the Instructions of: Freeths LLP
POL-BSFF-0100992_0112
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 109 of 265
4.76 Iam not clear on what the process was for disputing any Transaction
Acknowledgements, or how a dispute would have been investigated.
Failed Reversals
4.77 In relation to system reversals, Ms Angela Van Den Bogerd states:
“the concerns were based on the fact that reversals were not being shown
on the particular data sets reviewed / reports typically run by
Subpostmasters in branch on Horizon;
transaction reversal data can be extracted from Horizon;
the issue was therefore surrounding how the transaction reversals were
displayed / accessible in branch and that there was no issue with Horizon
itself.
There is therefore no indication that the reversal was not notified to the
Subpostmaster. When recovery was carried out a discontinued session
receipt would have been printed and messages would have been clearly
displayed to the user in branch during the recovery process.”
4.78 As dealt with above at paragraph 4.62, the excerpt from Gareth Jenkins
within the Helen Rose report indicates that there was no evidence of the
creation of a disconnected session receipt, unless further diagnosis
(which I do not believe has been disclosed to me) has since been
conducted and reviewed by Angela Van Den Bogerd. I have reported on
what was diagnosed contemporaneously by Mr Jenkins, particularly:
“However what I was able to confirm from my look at live data a couple
of weeks ago and is also held in the underlying raw logs is confirmation
that the reversal was generated by the system (and not manually by the
user). What might also be available in the underlying logs is whether or
not the system was re-booted - I suspect it was but have no evidence
one way or the other (and it isn’t in what was extracted this time either).”
Prepared by: Jason Coyne fe)
Occupation: Partner ae
Specialist Field: IT Systems It }roOu f )
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0113
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 110 of 265
Stephen Paul Parker
Remote Access
4.79
4.80
4.81
4.82
4.83
In Mr Parker's witness statement dated 16 November 20187'° at
Paragraph 19 Mr Parker states:
“The suggestion that Fujitsu edited or deleted transaction data is not
correct. In Legacy Horizon it was not possible to delete or edit messages
that had been committed to the message store.”
I have provided excerpts from PEAK records that illustrate edits and
deletions of messagestore data within the PEAK analysis (Section 3,
‘Evidence of Insertions/Deletions within Branch Accounts (Horizon Issue
10)’ above).
It should be noted that PEAK PC00518552!” (and others referenced
further within this report from paragraph 3.266 onwards) document
activities of deletions in relation to messagestore corruptions and
issues. Whilst there is a redundant copy of the messagestore (also
known as a mirror) that data could be re-instated from, I consider
deletion of messagestore items to be deletions of messages (which held
transactional data).
Mr Parker’s statement here should also be considered with those of Mr
Torstein Godeseth from his first statement at paragraph 35 where he
states:
“Users with sufficient access permissions could inject additional messages
(i.e. data) at the correspondence server”
Much of Mr Parker’s second witness statement is directed to factual
matters relating to the first witness statement of Mr Richard Roll *48and
Fujitsu's ability to edit, delete or insert transactions or the possibility of
bugs and errors affecting branch accounts during the Legacy Horizon
period. Mr Roll has also served a second witness statement which deals
216 {Witness Statement of Stephen Paul Parker, 16 November 2018}
217 PEAK PC0051855, 5 August 2000 {POL-0226902}
218 {Witness Statement of Richard Roll, 11 July 2016}
Prepared by: Jason Coyne fe)
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0114
181024SR1935
POL00262929
POL00262929
01 February 2019 Page 111 of 265
with a number of these points. There are factual disagreements between
Mr Parker and Mr Roll which I have taken into account when preparing
this report. Two potentially important points in relation to remote access
and Legacy Horizon are:
a.
Mr Parker at paragraph 20.2 says that “Some members of the SSC
were (and some remain) able to insert transaction data. SSC access
privilege gave the ability to inject transactions, but appropriate change
controls were in place and no such insertion would have happened
without complying with those controls.” I understand this to be a
reference to the Operational Change Procedure, which required the
creation of an “OCP”. Post Office disclosed the OCPs on 25 January
2019 and, given the time constraints due to this proximity to my
report submission date I have not considered them in this report,
I will provide a further analysis at a later date in review of these.
Mr Roll in his second witness statement states at paragraph 20
that transactions were injected by SSC at the counter in such a
way that they would appear on the transaction log as if they had
been inserted within the branch. Additionally, it is claimed by Mr
Roll that the method described in Mr Godeseth’s first witness
statement of inserting at the correspondence server (which would
result in a counter position greater than 32 being shown) was not
always used.
In the process of finalising this report, I have been shown a further
witness statement from Mr Parker,?!9 dated 29 January 2019, and
my attention has been directed to paragraph 27 of that statement
where Mr Parker in fact agrees with Mr Roll that this was possible
and was done, but he says that he believes in the majority of cases
injecting at the correspondence server was the default option. I
have not otherwise had the opportunity to consider this statement
from Mr Parker in detail, given its timing.
219 {Second Witness Statement of Stephen Paul Parker, 16 January 2019}
Prepared by: Jason Coyne fe)
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0115
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 112 of 265
4.84
4.85
4.86
My findings in relation to remote access facilities available to
Fujitsu/Post Office are detailed within Section 5 sub section 11 of this
report) and are inconsistent with Mr Parker’s statement in many ways,
particularly, as I explain from paragraph 5.406, in relation to the
capabilities surrounding insertion/injection, edit and deletion of
transactions.
At paragraph 40 of Mr Parker’s statement, he refers to call volumes in
relation to response codes allocated to incidents (PEAKs) reported
between January 2010 (I believe this to be a typographical error and
the intended date to be 2001) and 31 December 2004. In doing this, I
understand Mr Parker's intention to be to refute evidence by Mr Rolls
(first witness statement) regarding “fire fighting coding problems within
the Horizon system.”
However, Mr Parker’s figures denote that the largest percentage of calls
were indeed relative to performing analysis in relation to known issues
and workarounds, which in my opinion, seems more to support Mr Roll’s
evidence.
Payments and Receipts Mismatch
4.87
With regards to Mr Parker’s paragraph 42.1 (of his first Witness
Statement), in which he maintains there have been 735 live incidents
referring to “Payments and Receipt mismatch”, I have submitted a
request for information (RFI) - See Annex A). Particularly, to identify
the specific PEAKs relative to those, so that I may assess the types of
errors diagnosed at the heart of the mismatch in question.
KELs and PEAKS
4.88
At paragraphs 60 to 61.10, Mr Parker describes the process for the
creation of KELs and PEAKs. There are many limitations in the process
relating to KELs as he has described it, for example:
a. no mandated rules for when a KEL should be created (paragraph
61.3);
Prepared by: Jason Coyne
fe)
Occupation: Partner : .
Specialist Field: IT Systems : It FC
[e)
On the Instructions of: Freeths LLP
POL-BSFF-0100992_0116
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 113 of 265
4.89
4.90
4.91
b. KEL not considered the definitive source of all information
(paragraph 61.5);
c. dates given in KELs not precise (paragraph 61.6);
d. no fixed routine for the review of KELs (paragraph 61.7);
e. duplicated information is present in the KEL system (paragraph
61.8);
f. not all current KELs are still relevant (paragraph 61.9); and
g. KELs do not record all PEAKs they are relevant to, and no
requirement to update a KEL when it is reused to provide guidance
on a different incident (paragraph 61.10).
I would certainly agree with Mr Parker’s observations with regards to
KELs, in that they are difficult to navigate and KEL to defect
relationships are difficult to understand also noting that PEAKs refer to
the same issue but different KELs.
This, in my opinion, makes any investigation of a bug/error or defects
full impact very difficult to assess. It is also one the reasons why I
believe Dr Worden’s statistical analysis in relation to the financial impact
of bugs/errors and defects is ultimately flawed.
At paragraphs 62 to 62.9 Mr Parker describes an overview of the process
of PEAKs. I have explained in the PEAK analysis section of my report
above why, although PEAKs are generally a better source of information
about a particular problem than KELs, there are limitations also with
this system, including; because the content recorded within PEAKs is
variable, the cloning of PEAKs is problematic and it appears that PEAKs
are closed prior to resolution being reached or the Subpostmaster being
informed of the outcome. In my opinion the way in which PEAKs are
authored and controlled would limit the ability for Fujitsu to identify the
full effect of a particular problem, which problems may be linked, and
to carry out any trend analysis or audit of the problems or fixes.
Prepared by: Jason Coyne fe)
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0117
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 114 of 265
Paul Smith
4.92 Mr Smith’s witness statement?2° provides information about volumes of
disputed Transaction Correction and success rates. It is difficult to
comment on this information because there are no source documents
provided for the figures. Furthermore, Mr Smith does not explain either
the process by which Post Office or the individual teams decide to issue
a Transaction Correction, or the process by which disputes are resolved.
The figures also do not give the value of the Transaction Corrections
concerned.
4.93 At paragraphs 30 to 33, Mr Smith responds to paragraph 6.66 of my
first report, where I stated that there was both a credit and a debit
Transaction Correction for £810,000 for the same branch, indicating
that the initial Transaction Correction may have been in error. I do not
know the source of the further information provided by Mr Smith (no
further documents are exhibited), so it is difficult to consider his
explanation fully.
4.94 At paragraph 23 of Mr Smith’s Witness Statement he explains that in
the Financial Year 2016/17 Santander reported 19,491 “Errors” to Post
Office. These errors are likely relevant to Horizon Issue 5 (how, if at all,
does the Horizon system itself compare transaction data recorded by
Horizon against transaction data from outside of it).
4.95 Mr Smith’s analysis appears to show that these “Errors” lead to
Transaction Corrections being issued to the Subpostmasters. When
these Transaction Corrections were received 2,890 of them were
disputed by the Subpostmasters and 2,222 (77%) of these disputes
where upheld by Post Office.
4.96 This evidence suggests to me that:
a. Reconciliation data from Santander received into Horizon was
incorrect.
220 {Witness Statement of Paul Ian Michael Smith, 16 November 2018}
Prepared by: Jason Coyne fe)
Occupation: Partner .
Specialist Field: IT Systems ; It lI
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0118
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 115 of 265
b. Post Office issued Transaction Corrections to the Subpostmaster
based on this incorrect Santander data before checking its own
audit data.
c Post Office only checked its own audit data once a dispute was
raised by the Subpostmaster and therefore upheld the dispute.
4.97 It is not clear if Post Office, on discovering that a high percentage of the
Santander data was incorrect, checked with other Subpostmasters’
branch accounts which did not dispute the Transaction Corrections that
they received to check if there were in fact further incorrect TCs which
Subpostmasters had mistakenly accepted.
Prepared by: Jason Coyne fe)
Occupation: Partner .
Specialist Field: IT Systems g It ]
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0119
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 116 of 265
5.
Dr Worden’s Expert Report
Introduction and Overview
5.1
5.2
In this section of my report, I respond to the expert report of Dr Worden,
dealing with each Section and set of issues as grouped by him in the
body of his report (which are slightly different to my own groupings
within my first report).
In this introduction, I provide an overview below of some important
points of agreement or disagreement between us, and where our
approaches have differed.
Horizon Overview
5.3
Dr Worden’s report covers business applications in Old Horizon and
Horizon Online. In many respects, the factual matters identified by Dr
Worden are non-contentious. I believe that Dr Worden and I have each
attempted to set out the extensive Horizon estate and its business
processes in a way which will assist the parties and the court. Where
my understanding or opinion differs from Dr Worden on these issues, I
have stated so and why, although the substance of our disagreements
tends to arise in relation to the substantive Horizon issues so is
addressed in later sections.
Robustness
5.4
5.5
Section 7 of Dr Worden’s report addresses “robustness”, dealing with
issues 3, 4 and 6, and of which he says (at paragraph 48) that in his
opinion he considers the most important to be Horizon Issue 3 (to what
extent is Horizon ‘robust’ and extremely unlikely to be the cause of
shortfalls in branches). Dr Worden then concludes Horizon was ‘very
robust’ (paragraph 49.1), relying in particular on his 18 defined
countermeasures.
I do not agree with Dr Worden’s analysis of these countermeasures for
reasons I explain in response to his section 6 below. But I also disagree
that the most important focus of the enquiry should be by reference to
Prepared by: Jason Coyne fe)
Occupation: Partner : -
Specialist Field: IT Systems g It ] }{
On the Instructions of: Freeths LLP < -
POL-BSFF-0100992_0120
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 117 of 265
5.6
5.7
5.8
5.9
“robustness”. This term is relative. One system may be more or less
robust than another, in different respects and with different
consequences. It is not a well defined or accurate term to use as a
benchmark. In my view issues 1 (possible or likely for bugs, errors and
defects), 4 (potential for errors in data) and 6 (measures and controls)
are practically the more important issues, because they are more
concrete issues which can be assessed with more certainty.
In this respect I note that, Dr Worden states at paragraph 52 of his
report that robustness involves ensuring harmful events do not have
harmful consequences but, where they do, that they are kept within an
acceptable limit. I don’t know what Dr Worden would consider to be
‘acceptable’ on the facts of this case, where financial consequences may
fall directly on an individual Subpostmaster, who has not been party to
designing the system.
My experience of other commercial technology disputes is that often the
Court is asked by the parties to determine if a computer system was of
satisfactory quality, was fit for its intended purpose or if the parties
exercised reasonable skill and care in the system’s implementation. My
experience of these disputes is that the parties are, often, the system
vendor and purchaser.
In such disputes, there will often be a Service Level Agreement (SLA)
setting out acceptable levels of defects, levels of system uptime or
availability. Post Office may have such a document with Fujitsu or ATOS,
but I do not perceive such agreements will have any relevance for this
dispute.
It is a matter of fact that Post Office acknowledge that Horizon has had
at least three bugs/errors and defects that did impact branch accounts,
with a number of bugs/error and defects having been undetected for a
number of years. A significant difference between Dr Worden and I is
the extent to which we consider and assess the importance of other
bugs/errors and defects which did or may have impacted branch
accounts, in much the same ways as those acknowledged by Post Office.
Prepared by: Jason Coyne
fe)
Occupation: Partner : .
Specialist Field: IT Systems : It FC
[e)
On the Instructions of: Freeths LLP
POL-BSFF-0100992_0121
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 118 of 265
5.10
5.11
5.12
5.13
Whilst Dr Worden does not consider a number of other bugs within his
report, in his assessment of robustness, his focus is very much on the
three bugs originally acknowledged by Post Office in their Letter of
Response, without really engaging with the impact of the other bugs
which can be identified from the documents. Further, where Dr Worden
does comment on the acknowledged Post Office bugs, his review is
largely limited to what is said within the Responsive Witness Evidence
served by Post Office, rather than the further work to identify related
PEAKs, which has been an important part of my analysis.
Regarding those acknowledged by Post Office, Dr Worden’s review is
largely limited to the statements of others taken from Responsive
Witness statements.
The Horizon system has been operational for at least 18 years and many
aspects of Design, Build and Support have changed multiple times
throughout its lifetime which makes providing any definitive opinion as
to Horizon’s state (or “Robustness”) over any period a challenge.
As part of his assessment of robustness, Dr Worden claims that Horizon
is a “green fields development” “essentially unencumbered” by any IT
legacy (paragraphs 57 and 336). However, I believe this is incorrect, as
the initial project commenced in 1995 was initially going to be the
benefits agency system, and only when this failed was the software re-
purposed for the Post Office counters.
As I have said above. Dr Worden’s position on robustness is in many
respects based on countermeasures, which in turn are based largely on
the designer's aspirations. He states that “it is possible to classify the
types of counter measures, to assess how each type was applied in the
building of Horizon”. Whilst Dr Worden is correct, it is possible, some
obvious limitations of this approach are as follows:
a. To have confidence in your opinions you would need to study the
detailed designs of all the elements of Horizon, not just overviews.
Prepared by: Jason Coyne fe)
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0122
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 119 of 265
b. To have confidence that the detailed designs were followed during
the build one would need to see quality assurance documents
displaying that the designs aspirations where checked against the
actual delivery.
c. The 19,842 release notes suggest that Horizon has changed
frequently since its inception, without the detail of these release
notes it would be impossible to know the impact of each change to
the Horizon system throughout its lifetime.
d. The Horizon design documents that are available are either: -
is At a high level, recording the broad design aspirations of the
Horizon estate, with very little detail of how these design
aspirations are implemented in each aspect of the horizon
system (which is clearly required to rule out gaps in design), or;
ii. At a detailed level, recording how an element of the horizon
system should be built, requiring the review of hundreds (if not
thousands) of detailed designs to achieve confidence in one’s
coverage of the design aspirations into the specific elements to
provide an opinion of Horizon as a whole.
e. Whatever point in time Dr Worden may select a design document
to analyse, that design may only be valid for the time the design
was implemented until the Horizon system was later changed;
f. I The detail as to what changed within Horizon and when, largely
unknown to us as technical expert witnesses. The detailed release
notes documenting the 19,842 changes have not been provided
(although I did request them on 12 July 2018)
5.14 With the above points in mind, I find it difficult to understand why Dr
Worden would select the methodology that he has. Essentially, utilising
broad design aspirations at a single point in time of Horizon’s service
lifetime.
Prepared by: Jason Coyne fe)
Occupation: Partner .
Specialist Field: IT Systems ; It lI
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0123
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 120 of 265
5.15 From this methodology, Dr Worden opines that Horizon would have at
all practical times adhered to those designs and further where designs
are flawed, the impact would have been caught by largely manual
processes or that such failures are statistically immaterial. I disagree
that this is an appropriate methodology.
5.16 Dr Worden says that he has used the risk assessment methodology
contained within the Prince2 project management framework and
applied it retrospectively (paragraphs 53 and 362 of his report). lama
certified Prince2 practitioner and will often apply its Risk Management
principles in my IT delivery projects. Prince2 is good at assessing likely
risks in discrete IT projects however I do not believe it is designed or
appropriate to be used to retrospectively assess historic occurrences of
bugs/errors and defects. In Prince2 practice, one would consider each
of the possible risks or failure modes then attribute a measurement of
how likely it would be that this risk could trigger. The very definition of
a risk in project management, using Prince2 as a management
framework is an uncertain event or condition that could impact the
project. Looking back using this methodology is largely meaningless as
the risk has either triggered or it has not. My approach in tackling the
assessment of the extent of robustness was to look at occurrences of
bugs/errors and defects actually recorded in the disclosed material in
order to assess whether these errors were of significance to branch
account impact. I would then traverse upward, in trying to understand
the resolution of the bug/error or defect in order to assess its magnitude
Countermeasures
5.17 Section 6 of Dr Worden’s report is titled “Architectural Topics Across Old
Horizon and Horizon Online”, but much of this section of his report is
identifying and commenting on what he describes as “robustness
countermeasures”. Dr Worden provides a table at paragraph 60 of the
three letter acronyms to explain what he has characterised s 18
countermeasures (he acknowledges that these acronyms are not
common industry terms). I respond to each of the countermeasures in
Prepared by: Jason Coyne fe)
Occupation: Partner .
Specialist Field: IT Systems g It ]
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0124
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 121 of 265
my response to Dr Worden’s section 6 below. Fundamentally, where he
and I differ is that I believe Dr Worden’s countermeasures are basic
elements of practical system design, and in many respects he is relying
on design aspirations, rather than evidence of how bugs and errors in
fact arose and were resolved by Post Office and Fujitsu.
5.18 Increasing the number, type or position of the countermeasures, may
indeed improve the index of relative robustness at a point in time but
Dr Worden and I agree, that no combination of design aspirations or
“countermeasures” can provide an infallible Horizon (or any other
system or service), but differ on the relative effectiveness of the
countermeasures individually or together.
5.19 Anumber of the countermeasures identified by Dr Worden (Bug Finding
and Correction, Manual Inspection of Data) are simply that a human
(either Post Office, or Fujitsu or the Subpostmaster) would likely spot
the impact of the error and therefore have it corrected. Whilst it is true,
human checking is a form of system check, describing a Subpostmaster
spotting an error as a countermeasure is stretching the definition of a
countermeasure to its very limit. My starting point would be that where
it has been necessary for a Subpostmaster to identify the problem, that
means that the system is lacking robustness, and countermeasures
within the system have failed. This “countermeasure” is also dependent
on Subpostmasters’ knowledge and understanding of Horizon and their
accounts, which I expect will be variable between Subpostmasters
depending on e.g. age and experience, or how and when the problem
arises.
5.20 I have dealt with the “robustness countermeasures” as defined by Dr
Worden in more detail at Sub Section 6 of this report.
KEL and PEAK analyses
5.21 As part of Dr Worden’s analysis he has looked at a number of KELs, and
he states that his analysis of the KELs “implies to me that the
countermeasures in Horizon worked well in the live use of Horizon”. 1
Prepared by: Jason Coyne
fe)
Occupation: Partner : .
Specialist Field: IT Systems ) It 1 ¢
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0125
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 122 of 265
5.22
5.23
have set out my opinion and further observations in relation the
weaknesses of utilising KELs alone to identify the full effect of
bugs/errors and defects within Section 3 of this report ‘Analysis of the
PEAKs’ and also identified these at Section 4 above, noting the
limitations identified by Mr Stephen Paul Parker in his 16 November
witness statement. I also disagree with Dr Worden’s assessment of how
well his countermeasures have worked, as I have explained.
KELs are by their very nature “known” error logs and are often created
as a result of Fujitsu identifying multiple branches who experience the
same bug/error or defect to enable support the detection of new
occurrences of the same bug/error or defect.
In my opinion, Dr Worden does not give sufficient consideration to the
information which is contained within the disclosed PEAKs, which for the
reasons I have explained in section 3 above, are a very important source
of information, nor does he realistically assess the prospect of bugs or
errors arising which are not picked up, do not become the subject of a
PEAK or KEL, but nonetheless cause discrepancies in branch
Financial Analyses
5.24
5.25
In contrast to Dr Worden, I have not performed any financial analysis
on any Claimant data. Whilst I have focused on the extent it was
possible or likely for bugs/errors or defects to cause
discrepancies/shortfalls and undermine the reliability of Horizon to
accurately process and record transactions, I have not been concerned
with the value of such discrepancies/shortfalls other than to note that
the discrepancy range across branches is often wide.
For completeness, I have reviewed Dr Worden’s analysis in this regard,
and I believe that his assumption that bugs affect all claimants equally
is technically flawed. In summary, there is no technical basis to assume
that bugs/errors or defects impact all users or branches equally either
in frequency or quantum, in fact there is greater evidence available
Prepared by: Jason Coyne
fe)
Occupation: Partner : .
Specialist Field: IT Systems : It FC
[e)
On the Instructions of: Freeths LLP
POL-BSFF-0100992_0126
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 123 of 265
which shows that this assumption is incorrect in relation to the Horizon
system.
Sections 3,4 & 5: Business Applications & Horizon
Section 3 - Horizon Online (2010-Present day)
5.26
Within Section 3 of his report, Dr Worden sets out the various business
elements of Horizon. I believe that Dr Worden has adequately set out a
summary of the high-level requirements pertaining to Horizon that
provides additional information to the Business Scope I have set out in
my first report at paragraphs 1.7 to 1.8 (Pages 1 to 2).
Section 4 - Legacy Horizon (1998 - 2010)
5.27
5.28
5.29
Within Section 4 of his report, Dr Worden simplifies the Horizon
architecture for readability, which I am largely in agreement with,
however, I wish to add or disagree with the below points:
Dr Worden references at paragraph 147 the nature of the Riposte
functionality. He states:
“Riposte guaranteed that the same data would be available on the
campuses - although if the underlying network was unreliable, it might
take some time for Riposte to deliver this guarantee. Replication
guaranteed that despite any network failures, no change to data made at
a branch would be omitted at the campus or made more than once at the
campus.”
Dr Worden, in my opinion over emphasises a ‘Riposte guarantee’. He
does not reference that it was indeed the replicative nature of Riposte
that was often attributable to errors and defects that occurred in
Horizon, see for example, PEAK PC005843577! referenced at paragraph
3.251 of this report.
221 PEAK PC0058435, 15 November 2000 {POL-0232733}
Prepared by: Jason Coyne fe)
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0127
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 124 of 265
5.30
Further, later in Dr Worden’s report Appendix D he provides an example
where the nature of the bug/error or defect has arisen due to a
deficiency within Riposte (KEL LKiang3014S*2?). This same KEL is also
referenced in my first report at paragraph: 5.24 (Page 47).
Hardware and Software in Branches
5.31
5.32
5.33
5.34
With respect to hardware and software in branches, Dr Worden states
at paragraph 151 of his report:
“.. there were strong measures built into Old Horizon to ensure that
hardware failures and communication failures could not adversely affect
branch accounts.”
Dr Worden then references external literature that discusses theoretical
availability, disaster recovery and data communication papers that have
no specific relevance to Horizon or its own design documents that might
set out the measures he refers to as “strong” and “built into”. Whilst I
do not disagree that Horizon did indeed have measures built in that
were designed to ensure branch accounts were not adversely affected
by communications and hardware failures, there is significant evidence
of PEAKs within this report that set out that these measures did not
always prevent such occurrences.
Further, at paragraph 156.3 of his report, Dr Worden sets out his view
on transaction integrity. I disagree with his statement that:
“This transactional integrity was enforced by the Riposte infrastructure...
Therefore, it was impossible in any event (such as hardware failure) for
@ part-completed set of updates to be recorded in the branch and then
replicated to the back-office systems.” [emphasis added]
This was indeed a design aspiration for Horizon but in live operation it
was not the case that transactions would “...either succeed completely
or would fail completely and have no impact” as there is evidence within
the PEAKs documented within this report that the recovery procedure
(which was designed to provide the above integrity) was flawed.
222 KEL LKiang3014S, 27November 2002 last updated 22 February 2002 {POL-0035520}
Prepared by: Jason Coyne
fe)
Occupation: Partner : .
Specialist Field: IT Systems : It FC
[e)
On the Instructions of: Freeths LLP
POL-BSFF-0100992_0128
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 125 of 265
Further, Dr Worden has acknowledged in his report the Balancing
Transaction (BT) acknowledged by Post Office, where an equal but
opposing transaction must be manually inserted due to double entry
principle failures in Horizon occurring, therefore it cannot be said it is
impossible.
5.35 Dr Worden further states (regarding recovery) that:
“In these cases, the user on the counter would be guided through a short
set of recovery steps, to produce a consistent zero-sum result which
reflected what had happened. It was, of course, possible for the user to
make some mistake in these steps, which may have been unfamiliar. In
these cases, the mistake would often be detected later by a reconciliation
process, which would typically lead to a TC. This robustness measure was
a correction of user errors (UEC).”
5.36 Dr Worden points to the possibility of user mistake here yet he does not
consider that there is evidence of recovery failing electronically (i.e. not
a user mistake) or, the ambiguity of advice provided within the recovery
steps that meant a Subpostmaster suffered a shortfall (See Section 3
of this report ‘Recovery Issues’ for an example PEAK or the Witness
Statement of Angela Burke 223.
5.37 Additionally, such activities leading to a TC are ambiguous and the
PEAKs I have analysed do not support the assertion that "the mistake
would often be detected later by the reconciliation process, which would
typically lead to a TC. This robustness measure was a correction of user
errors (UEC).”
5.38 Iam not aware that Post Office has set out what TCs were issued due
to Horizon generated issues compared to those issued due to user error
across the whole lifespan of Horizon. On this basis I do not understand
where Dr Worden has gained his assumption from.
5.39 At paragraph 156.4 of his report Dr Worden comments on applications
driven by reference data. Whilst I agree with his summary of the
223 {Witness Statement of Angela Burke, 28 September 2018}
Prepared by: Jason Coyne fe)
Occupation: Partner .
Specialist Field: IT Systems ; It lI
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0129
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 126 of 265
benefits of reference data as opposed to hard coded values (of which
though there were still some within the Horizon system), reference data
changes were often the cause of discrepancies and disruptions within
branch accounts as detailed in my first report and further supported by
the PEAKs illustrated in Section 3 ‘Withdrawn Stock’ of this report.
Back End Architecture
5.40 At paragraph 164.3 of his report, Dr Worden makes reference to the
Management Information System (MIS) and it being a robustness
countermeasure. From the Witness Statements of Torstein Godeseth??4
and Paul Ian Michael Smith??> I understand Credence was utilised as
one of Post Office’s Management Information Systems. I have set out
the limitations of utilising Credence as an error proof source of
determining financial integrity in my first report at paragraphs 5.174 to
5.182 (Pages 88 to 90) and also, within this report in response to the
inaccuracies within the Witness Statements of Tracy Jane Wendy Mather
226and Angela Van Den Bogerd 227(which dispute a system reversal
however ultimately refer to the fact that Credence did not detail
sufficient information in respect of a disputed discrepancy).
5.41 Dr Worden continues to state that:
“Many pairs of eyes are inspecting the outputs of the MIS, in hundreds
of different reports or spreadsheets”.
5.42 Dr Worden does not explain what he means by his phrase “Many pairs
of eyes” and provides no analysis of the effectiveness of any such
processes which he is intending to refer to. Within the PEAK analysis
above there is reference to Fujitsu reminding Post Office that they
should be looking more carefully at the daily reports being provided to
them as bugs/errors and defects which should have been spotted in the
reports, were missed by the Post Office (please refer to paragraph 3.191
earlier in this report). Further, any handling or manipulation of
224 {Witness Statement of Torstein Olav Godeseth, 27 September 2018}
225 {Witness Statement of Paul Ian Michael Smith, 16 November 2018}
226 {Witness Statement of Trace Jane Wendy Mather, 16 November 2018}
227 {Second Witness Statement of Angela Margaret Van Den Bogerd, 16 November 2018}
Prepared by: Jason Coyne
fe)
Occupation: Partner : .
Specialist Field: IT Systems ) It 1 ¢
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0130
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 127 of 265
spreadsheets may potentially be subject to manual error the likelihood
of which increases the more they are handled.
5.43 Whilst I agree with Dr Worden’s findings at paragraphs 166 and 167 in
respect that reconciliation allows for the detection and correction of
errors made at the counter (or elsewhere in the processing of data
within further transmission and Horizon processing systems) where Dr
Worden states: “If there were any such software error, it would probably
occur with such high frequency, and occur uniformly across all branches,
giving rise to so many TC’s, that Post Office would soon suspect a
software error”. I fundamentally disagree, the documentary evidence
does not support such a statement.
5.44 Whilst it is true that simple errors may impact Subpostmasters
universally and these may be high frequency and occur uniformly, other
bugs/errors and defects impact a few branches on multiple occasions.
My analysis and review of the PEAKs has identified that many of the
bugs/errors and defects recorded in Horizon were initially investigated
because the impacted Subpostmaster who suspected a Horizon
generated error made a support call. I do not believe that Post Office or
Fujitsu compile any kind of statistics measuring whether Horizon
generated errors were first initially identified and/or investigated by
themselves, or the Fujitsu support team or Subpostmasters.
5.45 Dr Worden opines broadly at paragraph 169 of his report that Post Office
would have checked that it was paying external clients the correct
amounts of money for services conducted. There is contrary evidence
in the witness statement of Mr Paul Smith that Santander, one of Post
Office’s external clients reported 19,491 “Errors” to Post Office in
2016/17.
5.46 2,222 of these “errors” Post Office initially claimed were due to
Subpostmaster mistakes and therefore issued TCs, but when disputed,
Post Office appeared to accept that these were not Subpostmaster
mistakes. It is not clear where Post Office ultimately determined the
Prepared by: Jason Coyne
fe)
Occupation: Partner : .
Specialist Field: IT Systems ) It 1 ¢
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0131
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 128 of 265
mistake had been made, or if similar mistakes had been seen as TC
dispute records only started to be kept from 2016.
5.47 Further, there are several instances where other external clients have
raised issues over discrepancies and where such have been identified
due to Horizon defects and issues, or misinterpretation of reports and
values by Post Office staff. This is exampled in relation to KEL
acha4745R2"° (referred to by Mr Parker in Appendix 2 of his witness
statement ??2and Dr Worden within table D5 of Appendix D to his first
report, in which client reconciliation reports were not being manually
processed correctly.
Audit Information
5.48 At Paragraph 173 of his report. Dr Worden refers to the audit database.
It is not the case that this is a record of “any activity which can affect
branch accounts”. Branch accounts can be affected by Fujitsu, Post
Office and reconciliation data not entered at the counter but inbound
from external clients.
5.49 The audit database is only a record of “what was entered at the
counter"?° with the exception of certain counter entries which are not
committed to audit logs because of known failure conditions.234
5.50 After consideration of the above reduction in scope of what is recorded
in the audit data, the record of when it is recorded is important. All of
the data is written to the audit database once a day, in the early hours
of the morning.
5.51 Whilst I have not found any evidence to suggest this occurs in Horizon
it is technically possible that after the transaction is completed at the
Branch counter the record could be tampered with prior to its
commitment to the audit database some hours later. I believe that is
228 KEL acha4745R, 30 July 2012 last updated 15 May 2017 {POL-0040845}
229 {Witness Statement of Stephen Paul Parker, 16 November 2018}
230 Outreach BLE Extract Findings v6 091215.pptx, Branch Outreach Issue (Initial
Findings), 10 December 2015 {POL-0220141}
231 HorizonOnlineDataIntegrity_POL.doc, Horizon Online Data Integrity for Post Office Ltd, 2 April
2012 {POL-0021989}
Prepared by: Jason Coyne
fe)
Occupation: Partner : .
Specialist Field: IT Systems ) It 1 ¢
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0132
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 129 of 265
what Mr Richard Roll refers to in his second Witness statement at
paragraph 18. I have seen this first hand in a banking investigation
where banking staff had changed the sort code, account number and,
on occasion, the monetary amount figure on transactions whilst
transactions were in this “pre-committed” state between submission
from the counter and processing (in this particular scenario to another
branch) and later into the banks audit logs.
5.52 With regard to how useful the Audit database might be as a
countermeasure, I do not disagree that it is possible for Fujitsu to review
this audit database to enable a comparison to be made to other records
in the event of a discrepancy, but largely the requirement for such would
have to be initiated by Post Office or a Subposmaster raising a query
and insisting that the full audit are examined. The First witness
statement of Torstein Godseth (paragraph 31) shows that that on
relatively few occasions was the full audit data requested from Fujitsu.
5.53 It is not clear at which point in the discrepancy investigation process
any of the audit data is consulted, in consideration of the Dalmellington
/ Branch Outreach Issues dealt with in relation to the responsive witness
statement of Mr Godeseth that details two specific Horizon defects had
112 occurrences which impacted 88 different branches since 2010. The
findings were discovered by retrospective analysis of the historic audit
data, suggesting that it was not spotted at the time. The same
document records that whilst the audit data has been consulted there
are years (2012, 2013 and 2014) where the audit data has been unable
to assist and that “unknown outcomes” are noted for a number of
specific branches.
5.54 It is also appears to me that (based upon my own investigations and
from review of the responsive witness statement of Mr Paul Smith and
the table of ARQ’s actually requested from Fujitsu) that Post Office
would not typically check the Audit database before raising a TC in
relation to external client transactions, electing instead to rely on third
party client reconciliation data brought in from outside of Horizon in the
Prepared by: Jason Coyne fe)
Occupation: Partner .
Specialist Field: IT Systems g It ]
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0133
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 130 of 265
first instance, before performing their own analysis on Credence which
previous evidence has illustrated, did not provide the full picture of the
Horizon situation.
Auditability
5.55
5.56
5.57
5.58
I note that Dr Worden at section 4.4 paragraph 173 of his report states:
“The Horizon system includes an audit database (Technical Environment
Description, 22 October 2002, {POL-0444096}), which is an accurate and
immutable record of any activity which can affect the branch accounts.”
It is my opinion that this statement is incorrect. As is explained above,
the audit database does not record ALL activities that can affect the
branch accounts. Dr Worden does not consider the wider elements of
Horizon processing. Not all elements of operations, or transaction
modifications were recorded via the audit server. Where modification to
transactions conducted by Fujitsu support teams were carried out, there
becomes additional elements to the data that would not have been
captured in the initial audit log sent to the audit server. i.e., where
transaction correction tools have been used or direct SQL scripts
executed on the branch account database. The auditability of any
corrective amendments/operations or deletions once the daily
transaction data was committed to the relevant database tables and the
audit store would need to be identified elsewhere within the Horizon
system.
Further, it should be noted that the audit log reflects branch counter
data, therefore, if a counter error caused a transaction item to be
duplicated prior to its submission to the database, then the audit log
would contain a replication of this recording, it is not to say that the
audit log could therefore not hold erroneous data in itself.
Utilising the same images as Dr Worden references at Figure 4.3 of his
report??? I illustrate the auditability constraints from the initial Branch
232 Horizon Core Audit Process - v1 O.ppt, Horizon Core Audit Process, 30 January 2014 {POL-
0218333}
Prepared by: Jason Coyne
fe)
Occupation: Partner : .
Specialist Field: IT Systems : It FC
[e)
On the Instructions of: Freeths LLP
POL-BSFF-0100992_0134
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 131 of 265
Database / Counter Audit File capture in Legacy and Online versions of
Horizon, where complex processing systems handling the data AFTER
the initial Audit File capture would not be reflected in the Counter Audit
File.
Not within the
Counter Audi File
dit Agent Aut
wit
‘ua
Extract I
ata
Audit [Store
Replication
Figure 3 Legacy Horizon and further Auditable Activities
Not within the Counter
Audit File
Data Copy
Oracle Audit
eee Write a Write
Audit IStore
BALI Message Audit IRetrieval
Counter
Figure 4 HNG-X and further Auditable Activities
5.59 At paragraph 178.3 to 179 Dr Worden states that the integrity measures
with regards to recovery and audit information are well designed. As
aforementioned in this report, design is not always an accurate
Prepared by: Jason Coyne
Occupation: Partner .
Specialist Field: IT Systems itg rou p
On the Instructions of: Freeths LLP
POL-BSFF-0100992_0135
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 132 of 265
5.60
5.61
representation of actual build or operation. Whilst from a design
perspective the principles are theoretically sound, in operation, there is
evidence that both (i) recovery and (ii) journal sequencing (designed to
increment sequentially to ensure integrity of audit files) were
susceptible to error as detailed in my first report at paragraph 18.8 Page
214 which references KEL MithyanthaJ1937S233 (in relation to journal
sequencing) documenting that a fix was not released for approximately
six years.
Further evidence is documented within KEL Maxwellg5213L?*4 and
PEAKs PC0240992735 and PC02530962° and in respect of recovery in
Section 3 ‘Recovery Failures’ of this report.
It should be noted that the above references are not fully exhaustive
evidence of issues in relation to audit and recovery processes but a
sample of instances in addition to the PEAK referenced within Section 3
‘Recovery Issues’ in which the recovery issue was identified as
impacting branch accounts subsequently causing financial discrepancy.
Section 5 - Horizon Online (2010 - Present)
5.62
5.63
5.64
Within Section 5 of his report, Dr Worden simplifies the Horizon Online
architecture for readability, which I am largely in agreement with and
have done similarly in my first report from paragraph 4.35 (Page 26)
onwards.
However, whereas Dr Worden focuses on the replaced elements of the
branch software, my first report notes the reuse of legacy hardware (at
paragraph 4.37; Page 27).
At Paragraph 202 Dr Worden refers to his defined countermeasures, I
explain my summary position in relation to these above and in more
detail below at 5.65.
233 KEL MithyanthaJ1937S, 06 May 2010 last updated 09 August 2016 {POL- 0040508}
234 KEL Maxwellg5213L, 30 June 2010 last updated 21 March 2011 {POL-0038402}
235 PEAK PC0240992, 15 February 2015 {POL-0410189}
236 PEAK PC0253096, 8 August 2016 {POL-0421502}
Prepared by: Jason Coyne
fe)
Occupation: Partner : .
Specialist Field: IT Systems : It FC
[e)
On the Instructions of: Freeths LLP
POL-BSFF-0100992_0136
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 133 of 265
Section 6: Countermeasures
5.65
5.66
5.67
5.68
Within Section 6 and Section 7 of his report Dr Worden refers to 18
countermeasures which he has described and relies upon as part of the
robustness of Horizon. I respond to what Dr Worden says in relation to
the individual countermeasures in this part of my report.
I have explained from paragraph 5.17 above, my views generally on Dr
Worden’s self-defined countermeasures in relation to Horizon. Namely
that what Dr Worden describes are generally basic elements of practical
system design, that these design aspirations in themselves do not show
that Horizon was particularly “robust”, and certainly not that it was free
from error or prevented errors from going undetected in branch
accounts.
Whilst Dr Worden acknowledges these acronyms are mostly not used in
Industry, he has used them throughout his report which gives the
impression they have a standard meaning and scope. However, as an
example, “Later correction of user errors” (“UEC”) is so very widely
defined to include any check carried out by any person (e.g. a
Subpostmaster’s own checks when balancing, or a Post Office or Fujitsu
automated or manual process at any time) that the use of an acronym
to group together all of these different factual scenarios is in my view
not very helpful.
I deal with each of the countermeasures individually from paragraph
5.69 below, but for convenience, I have set out a table which identifies
each of Dr Worden’s countermeasures and the explanation as provided
by him, and recording:
a. in the third column, my views as to whether the countermeasure
is an industry accepted acronym and whether what is described
feature in general IT industry design; and
b. in the fourth column, identified the paragraphs of this report where
I comment on the countermeasure including examples where I am
Prepared by: Jason Coyne fe)
Occupation: Partner : -
Specialist Field: IT Systems g It ] }{
On the Instructions of: Freeths LLP < -
POL-BSFF-0100992_0137
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 134 of 265
aware of failures or limitations in respect of the operation of that
countermeasure in Horizon.
Countermeasure Explanation as provided Industry Accepted I General
as described by Dr I by Dr Worden Acronym, or Observations
Worden Typical standard or Evidence
feature in IT of failure
System Design? within
Horizon.
Reliable and Redundancy guards against I Acronym No, Paragraph
Redundant many types of hardware Standard, Yes. 5.152 to 5.108
Hardware (RHW) failure. Examples: RAID of this report
discs, disaster recovery
sites. Software is designed
in many ways to be robust
against hardware failures
Robust data Communication systems and I Acronym No, Paragraph 5.28
communication and protocols are designed to Standard, Yes. onwards of this
replication (ROC) recover from and protect report.
against many kinds of
communication failure.
Examples: TCP/IP, Riposte
Transactional Database management Acronym No, Paragraph
Integrity and systems provide many Standard, Yes. 5.104 of this
database recovery facilities so that numerous report
(TIN) kinds of failure cannot leave
the data in an inconsistent,
unusable state, or lose any
data that have been
previously stored
Defensive Software is divided into Acronym No, Paragraph
Programming (DEP) I small self-contained Standard, Yes. 5.112 of this
modules, which do not report
assume that other modules
are correct, but defend
themselves by checking
Prepared by: Jason Coyne
Occupation: Partner : .
Specialist Field: IT Systems itg rou )
On the Instructions of: Freeths LLP
POL-BSFF-0100992_0138
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 135 of 265
their inputs and raising
alerts early
Generic, data driven I Different use cases for Acronym No, Paragraph
software (DDS) software often have much in I Standard, Yes. 5.140 of this
common. Software is written report
generically to be able to
handle the different cases,
using reference data to
define which use case is to
be handled. Example:
variations in Post Office
client products handled by
reference data.
Secure kernel When a large complex IT Acronym - SEK Paragraph
hardware and system is subject to threats, I typically applies to 5.135 of this
software (SEK) the design may include a “Security report.
small, well tested and Enforcement Kernel”
secure kernel which is proof I and is intrinsic to
against those threats. system resources
Examples: secure kernels of I and technical
operating systems, Horizon I components of a
core audit process system. The Horizon
Core Audit process is
not an instance of a
secure kernel.
Standard - Yes.
Redundant data In large IT systems and sets I Acronym No, Paragraph
storage and of systems, data are stored Standard, Yes. 5.118 of this
computing, with redundantly in several report
cross-checks (RDS) places, and routine
operations check
automatically that the
different copies of the data
remain consistent
Prepared by: Jason Coyne
Occupation: Partner .
Specialist Field: IT Systems Zitg rou )
On the Instructions of: Freeths LLP
POL-BSFF-0100992_0139
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 136 of 265
Double-entry Accounting systems operate I Acronym No, Paragraph
accounting (DEA) by the principles of double Standard, Yes. 5.100 of this
entry book keeping, so that report
any change to the accounts
must be made in a
transaction whose summed
effect on all accounts is
zero. Transactions which do
not obey this constraint are
rejected.
Early detection of At the point of user input, as I Acronym No, 5.69 of this
user errors (DUE) many checks as possible are I Standard, Yes. report.
made of the correctness of
the input - so that the
system will not accept
erroneous input and may
warn the user of errors.
Later correction of In accounting systems, the Acronym No, 5.89 of this
user errors (UEC) system's version of reality is I Standard, Yes. report.
periodically checked against
external versions of reality
and corrected if wrong.
Examples: cash balancing
and rollover, reconciliation
and TCs.
Manual workarounds I Whenever any part of Acronym No, Paragraph
(WOR) Horizon does not work as Standard, No. Whilst I 5.170 of this
required, there may be manual work report
potential to define and apply I arounds are often
manual workarounds required where
system functionality
is deficient, good
industry practice
determines that
manual workarounds
are usually the parts
of the system that
Prepared by: Jason Coyne
Occupation: Partner .
Specialist Field: IT Systems Zitg rou )
On the Instructions of: Freeths LLP
POL-BSFF-0100992_0140
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 137 of 265
need attention to
avoid.
Testing good The purpose of system Acronym No, Paragraph
practice (TGP) testing is not to prove that Standard, Yes. 5.180 of this
the system is correct, but to report
prove that it is incorrect in
any way possible.
Examples: regression
testing, user testing, testing
edge cases.
Manual Inspection of I Any large business IT Acronym No, Paragraph
data (MID) system is used by many Standard, Yes. 5.124 of this
people, who view its outputs report
and check them against
each other for consistency,
and against their own
knowledge of the business.
Subpostmasters, watching
their branch accounts, were
a key component of this.
Bug Finding and Whenever the system shows I Acronym No, Paragraph
Correction (BFC) any anomalous behaviour, Standard, Yes. 5.175 & 5.170
that is investigated, its
causes found and corrected.
Interim workarounds are
deployed. Extra checks may
be added to ensure that
other similar threats are
handled correctly.
Large Scale IT In any large IT estate, Acronym No, 5.145 of this
architecture (ARC) principles of IT architecture I Standard, Yes. report.
are used to achieve
robustness - such as using a
distributed network of
loosely coupled sub-systems
with clearly distinguished
functions. The sub-systems
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems itg rou )
On the Instructions of: Freeths LLP =
POL-BSFF-0100992_0141
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 138 of 265
are built to well-defined
standards with clear
interfaces.
Quality and Change I Systems are more robust if I Acronym No, 5.195 of this
Control (QCC) quality is inherent. This is Standard, Yes. report
achieved by organising
properly the people who
build, maintain and operate
the system, by managing
them well and by governing
what they do through
rigorous but effective
processes. A system will
only continue to be robust if
changes are controlled in a
way that enhances quality
without unnecessary
administration
Managing non- Robustness is improved by Acronym Yes (NFR = I 5.199 of this
functional paying close attention to Non-Functional report.
requirements (NFR) I non-functional requirements I Requirements),
and the associated ‘ilities’ Standard, Yes.
such as manageability,
supportability,
maintainability and
adaptability
Security (SEC) Any system that could be Acronym Yes (SEC is I Paragraph
easily subverted would not often an 5.154 & 5.154
be robust. Horizon is abbreviation of of this report
secured mainly through Security), Standard,
‘separation of duties’, user Yes.
authentication, access
control and audit.
5.69 As above, Dr Worden seeks to rely on the 18 countermeasures he has
identified (in Section 2, 6 and 7 of his report) as evidence of the
robustness built into Horizon. It is my view that these countermeasures
Prepared by: Jason Coyne
Occupation: Partner .
Specialist Field: IT Systems itg rou )
On the Instructions of: Freeths LLP
POL-BSFF-0100992_0142
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 139 of 265
represent a framework of principles and each rely on the presence and
suitability of other core elements such as systems, processes and
human interaction and only provide a view on robustness when all of
these features are considered as a whole. For example, identifying a
countermeasure “Bug Finding and Correction (BFC)” is itself not
particularly informative.
5.70 The important questions are:
a. which bugs arose, how were they identified and how were they
corrected?
b. what can we tell about the way the system worked and was
managed from the way in which those bugs arose? and
c. how effective was the PEAK and KEL process for identifying trends,
and also correcting and preventing repeats of events which had
previously given rise to bugs?
5.71 Horizon has changed an immeasurable amount over its lifetime. Dr
Worden has not considered at what appropriate points in time, certain
countermeasures might have been in place or if such Countermeasures
where correctly positioned. Instead Dr Worden applies them generically
over its whole lifespan, and the entire estate, without consideration of
the potential inadequacies in relation to each specific piece of
architecture or configuration in place at its relevant time. For example,
any countermeasures for detecting an issue with the branch database
in relation to Legacy Horizon (where the database itself was in position
in the branch) would be different to any countermeasure for detecting
an issue with the branch database (BRDB) for Horizon Online, where it
was one central database for all branches, situated in a data centre.
5.72 Dr Worden performs analysis in relation to the KELs at 7.5 within his
report and opines on what countermeasures were at play in the
identification of it, or which failed. In my opinion, the reasons why
bugs/errors and defects occurred (as identified within the KELs) is
because; the countermeasures referred to by Worden were not
Prepared by: Jason Coyne
fe)
Occupation: Partner : .
Specialist Field: IT Systems ) It 1 ¢
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0143
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 140 of 265
5.73
5.74
5.75
User Error Detection and Prevention (Dr Worden’s “Detection of User Errors
appropriately configured for that point in time and therefore did not
apply, or were not positioned correctly within the Horizon system, or if
they were, they were flawed in some respect.
In the main Dr Worden’s countermeasures are little different than the
design aspirations for a motor car, each of which are welcomed and go
some way to reduce certain types of failures that lead to accidents.
Motor cars now might have Adaptive Cruise Control, Emergency
Braking, Blind spot Detection, Parking sensors, Lane assist, Electronic
Stability control and such like. Each of which will seek to reduce certain
types of failure. As such systems mature over time, typically from an
iterative process of failure and learning, they will indeed reduce the
types of accidents to which they were designed. They will not remove
accidents because new and previously not considered situations arise
over time, certainly as new features/functions are added.
With the knowledge of the motor cars “countermeasures” and the fact
that some accidents have already occurred, it would be unsafe to
declare that accidents would be unlikely.
I have responded below to each of the countermeasures introduced by
Dr Worden in the order they appear at Section 6 of his report.
(“DUE”)
5.76
5.77
5.78
I would expect to see facilities for “Detection of User Errors” in any IT
system. Such elements typically consist of the implementation of good
design and tight configuration features to prevent either entry of
erroneous data or warnings at the point of data entry.
I agree that a large number of measures were implemented within
Horizon to prevent user error as stated by Dr Worden at paragraph 222
of his report, where he displays a generic list of interface design
aspirations.
I agree with Dr Worden’s paragraph 230 that requirements for detection
of user errors would have been a priority for Post Office. But from my
Prepared by: Jason Coyne
fe)
Occupation: Partner : .
Specialist Field: IT Systems : It FC
[e)
On the Instructions of: Freeths LLP
POL-BSFF-0100992_0144
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 141 of 265
review of the disclosure the indications are that Post Office were
primarily focussed on minimising any potential loss it might incur and
Subpostmaster error was a secondary concern. The example of not
implementing the double entry and cross check principle in 2008 (as
referenced below with the Peter Laycock report paragraph 5.80 below)
displays one such scenario.
5.79 As identified in my first report at paragraphs 5.129 to 5.132, pages 75
and 76 (in relation to errors arising from data entry), there are various
KELs demonstrating counter level system controls that should prevent
user input error that failed or did not exist in the system in the first
place. Moreover, an external information security review document
referenced in my previous report (paragraph 5.126, page 74) attributed
the high level of transaction disputes due to a lack of source data
integrity i.e. values entered once without validation. This failure in data
validation in my opinion exacerbates the potential for human error. Dr
Worden makes the case for the need for Horizon to have this
countermeasure built into its user interface and I fully agree with this
view. However, in my opinion and for the reasons outlined, I do not feel
this was necessarily reflected in how Horizon had been configured.
Moreover, the configuration of Horizon changed many times during its
lifetime.
5.80 In 2008, a report and series of recommendations was provided by Peter
Laycock.??” This report specifically records the issue of “mis-keying” by
Horizon users at branches. This is evidence that the particular
countermeasure suggested by Dr Worden was clearly not in place or
was not providing sufficient coverage at this point in time. The
recommendation suggested in the report that the level of improvement
could lead to an “80% reduction in disputes and claims- saving £800k
per annum”. The business benefits are said to be a “Major improvement
of point of transaction data integrity”. The recommendations appear to
be simple, that the Subpostmaster is asked to retype the monetary
237 Summary of IS Review.doc, Summary of IS Review, 2008 {POL-0219516}
Prepared by: Jason Coyne fe)
Occupation: Partner .
Specialist Field: IT Systems ; It lI
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0145
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 142 of 265
5.81
5.82
5.83
5.84
5.85
value of cheques and that the two values should be equal. This appears
to be reasonable and I can see how this would dramatically reduce mis-
keying where the impact of the error could certainly be significant.
‘Mis-Keyed Project - Feasibility Study” dated 15 May 201278, (as
referred to in my first report at 5.123 and by Dr Worden at 226) - noted
that "mis-keyed Banking Deposit transactions amount to over 60 PER
WEEK”. The same report further states, “A very large value mis-keyed
transaction will put the viability of a branch in doubt". The ‘HighLevel
Business Requirement’ (section 3.2.1 of that report) is documented as,
“to devise a way which will prevent counter colleagues in Branches from
Mis-keying stock and financial transactions. Many counter colleagues
are not aware of the ramifications of mis-keying transactions”.
The recommendations which were made in the above report display that
the type of countermeasure suggested by Dr Worden was clearly not in
place or was not providing sufficient coverage across all required
aspects of the Horizon system by 2012.
Further, as these 2012 report recommendations are similar to the 2008
recommendations (paragraph 5.80 above) it does suggest that Post
Office either a) did not implement the 2008 recommendations or b) did
not implement them widely enough to provide the protection that Dr
Worden suggests was in place.
Mr Smith’s witness statement?*°, at paragraphs 30-31 also supports my
opinion. A Subpostmaster recorded a deposit of £900,000 (rather than
£90,000) causing an £810,000 shortfall in his branch. In my opinion,
this could have been prevented if the recommendations suggested in
the 2008 report had been implemented by Post Office.
In paragraph 251 of his report Dr Worden concludes that “Horizon
incorporated accepted industry practices for detection of user errors,
and in my opinion did so effectively.” 1 agree that in a number of areas
238 Feasibility Study - Mis-Keyed vO 1.doc, G-231 Mis-Keyed Project Feasibility Study, 15 May 2012
{POL-0217750}
239 Witness Statement of Paul Ian Michael Smith dated 16 November 2018
Prepared by: Jason Coyne
fe)
Occupation: Partner : .
Specialist Field: IT Systems : It FC
[e)
On the Instructions of: Freeths LLP
POL-BSFF-0100992_0146
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 143 of 265
Horizon did attempt the detection of user errors and did detect some
user errors, but in other areas it clearly failed to detect them, even in
specific areas where studies (as far back as 2008) had previously been
conducted and where suggestions had been set out which additional
areas of detection required attention but had not been implemented.
5.86 Dr Worden explains at 472 that requiring the user to enter the same
data twice may not be a good choice. I agree, the design of such a
validation should be triggered only when the monetary value exceed a
certain amount (perhaps £1000.00) or if the monetary amount displays
repeated digits (consistent with ‘keyboard bounce’ where the user adds
a mistaken extra digit) so that validation would not be required for
£123.45 but would be required for £1233.45 as it is possible the 3rd
key bounced and was pressed twice. Such an improvement to the
Horizon system would have reduced user error and continued to deliver
an efficient process. This would have removed the error that was
reported in Mr Smith's witness statement (at 5.84 above).
5.87. Dr Worden’s opinion at 476, is that Horizon was “well designed in
respect of detecting user errors, and there is no sound basis for thinking
it could easily have been improved.” I have not had sight of the testing
carried out against the detailed designs, Dr Worden does not make it
clear if he has. I have not had sight of the detailed user interface
designs, I’m not sure that Dr Worden has, but if not, then the best that
one could possibly say is, “Jf the user input capabilities shown on the
design are implemented in the Horizon build, provided they were
designed appropriately, then a good detection of user errors would be
seen”.
5.88 Whilst restrictive input in user interface design could be considered a
robustness countermeasure through the use of menus and buttons as
opposed to free text input (it is a common practical design element in
most applications), I disagree with Dr Worden that this ensures
“Detection of User Errors”. Restrictive input certainly assists in reducing
user errors but the facility of such (i.e., selecting an item from a menu
Prepared by: Jason Coyne
fe)
Occupation: Partner : .
Specialist Field: IT Systems ) It 1 ¢
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0147
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 144 of 265
as opposed to typing in a specific value) does not functionally detect
possible error. There is no intuitive functionality built into Horizon that
ensures a Subpostmaster selects the right menu or button or identifies
and detects where they might not have done. This appears to be later
agreed by Dr Worden at paragraph 231 of his report where he states
“There is in principle no way in which Horizon could detect or prevent
many of these user errors.”
Dr Worden’s - User Error Correction (“UEC”)
5.89
5.90
5.91
5.92
5.93
5.94
At section 6.1.2 of his report, Dr Worden states his opinions on “User
Error Correction” as a robustness countermeasure. Whilst I agree
shortfalls and discrepancies have occurred through user error, I am
unclear how Dr Worden can then state as he does at paragraph 232:
“As I have seen, there are probably several thousand such errors made
at the counter everyday.”
Dr Worden does not explain where he might have “seen” thousands of
such user errors occurring at the counter.
I note that at paragraph 236 of his report and in relation to the
complexities of the Horizon recovery procedure, Dr Worden states that
“typically the error is trapped later in reconciliation with the external
party and is corrected by a TC.”
I agree with Dr Worden that this process is less familiar for the users
who will often be faced with a high-pressure situation, without a working
Horizon system, and therefore user errors might occur.
However, the witness statement of Mrs Angela Burke?#° demonstrates
that when the Horizon system is suffering wider problems, recovery
Processes can lead to losses being suffered by Subpostmasters which
were incorrect and arose only because of a Horizon bug,error or defect
Errors appropriately diagnosed as “user error” should be singular in
instance and would only be seen within the branch in which they occur.
240 Witness Statement of Angela Burke, 28 September 2018, Paragraph: 20
Prepared by: Jason Coyne fe)
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0148
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 145 of 265
5.95
5.96
5.97
5.98
Large number of Subpostmasters (and their employees) could not all
make the exact same error across many branches within Horizon.
Therefore, the risk in mis-diagnosing a wider Horizon system error as a
singular user error has significant implications. It is extremely important
to ensure investigations into wider system errors are adequately
performed to rule out implications for branches.
Corrections following an earlier failure of Horizon should not be
considered to be a user error correction countermeasure.
Corrections by Post Office to user errors are also part of the Transaction
Correction process (as Dr Worden identifies at paragraph 477.1), so
assessing Horizon robustness in this area requires consideration of Post
Office back office process in which Transaction Corrections are issued, I
do not believe that Dr Worden has considered the adequacy of such
processes in his analysis.
I do not believe either expert has been provided a complete audit that
identifies in each individual circumstance of error the investigation
performed in relation to the discrepancy and the decision/conclusive
evidence that a Transaction Correction ought to be issued and the
evidence of the subsequent Transaction Correction being accepted by
the branch. The whole process of correcting user errors is wholly reliant
on:
a. The discrepancy being appropriately identified in the first instance;
b. The investigation of the discrepancy being wholly adequate and
sufficient;
c. Communication channels between all investigating parties being
completely aligned so that information is not lost between;
d. The Subpostmaster being satisfied that the evidence concludes an
appropriate diagnosis and resolution.
Prepared by: Jason Coyne fe)
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0149
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 146 of 265
Intrinsic Error Prevention
5.99
Double
The “Intrinsic Error Prevention” techniques discussed at Section 6.2 of
Dr Worden’s report are in my opinion, all common elements of standard
large-scale IT system design. Design is often not fully reflective of
operation, and whilst the robustness measures Dr Worden refers to did
protect Horizon processing to a certain extent, he himself acknowledges
failures of them in his KEL analysis (Appendix D of his report).
Entry Accounting (“DEA”)
5.100
“Double Entry Accounting” is an industry standard incorporated in most
if not all enterprise software packages. However, the implementation of
double entry or its principle alone does not prevent errors or ensure
robustness as a countermeasure entirely since it is largely reliant upon
the person creating the accounting system ensuring the appropriate
configurations to ensure adherence to double entry principles are
applied correctly. Dr Worden acknowledges at paragraph 467 of his
report (in reference to the Payments Mismatch bug) that not all
operations in Horizon adhere to the double-entry constraint. Without
understanding, in full, where within Horizon each of the specific
operations covered by the Double Entry principles actually is, one
cannot have confidence that this is an appropriate countermeasure. It
is also likely (as with all such Dr Worden countermeasures) that the
position has changed many times over the life of Horizon. One such
example of these inconsistencies between the various aspects of double
entry can be seen in a Fujitsu document “Correcting Accounts for ‘lost’
Discrepancies’*** authored by Gareth Jenkins notes the following:
“if the User doesn’t check their Final Balance Report carefully they may
be unaware of the issue since there is no explicit message when Receipts
and Payment mismatch is found on the Final balance (the User is only
prompted when one is detected during Trial balance) The Local Suspense
will have no knowledge of this specific Discrepancy”
241 Common Issues Documents G_9
Prepared by: Jason Coyne fe)
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0150
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 147 of 265
5.101
5.102
5.103
As acknowledged by Post Office in their letter of response,?4 it has
previously had to sanction a balancing transaction in order to rectify
failure of this countermeasure.
Further, in consideration that in legacy Horizon, all SSC users could use
escalated privileges to carry out modifications there have been
potentially many more fixes applied due to the failure of the double
entry principle that Post Office may not even be aware of.
I have also identified further evidence of encountered one-sided
transactions which I have documented within Section 3, ‘Counter
Replacement Causing One Sided Transactions’ and further within
‘Evidence of Insertions/Deletions within Branch Accounts (Horizon Issue
10)’, in which one sided transactions were written to accounts, a further
example of a failure of double entry accounting.
Transactional Integrity and Recovery (“TIN”)
5.104
At paragraphs 246 to 247 of his report Dr Worden provides an account
of “Transactional Integrity and Recovery” which he describes as a core
element of Horizon and therefore adding to a systems robustness by
providing an effective robustness countermeasure. He states:
“Because transactional integrity is a fundamental facility built into all
database management software...and it is necessary, for any relational
database, to describe in its schema the integrity constraints which it must
obey at all times, transactional integrity was applied to all of the many
databases of financial information in the Horizon system - including the
BRDB, the POL FS database, and many others (Technical Environment
Description, 22 October 2002, {POL- 0444096}; Horizon Solution
Architecture Outline, 7 April 2016, {POL-0444099}).
This means that any compound package of updates, applied to any of
these databases would have been applied as a single transaction or
‘success unit’ which would either completely succeed, or completely failed
242 {Letter of Response to Freeths, 28 July 2016 (Paragraph 5.16.3)}
Prepared by: Jason Coyne fe)
Occupation: Partner ae
Specialist Field: IT Systems It Jrou f )
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0151
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 148 of 265
5.105
5.106
5.107
leaving no race. It would be impossible to leave any of these databases
in an inconsistent state, not satisfying its integrity constraints”
Dr Worden’s views as recorded above are expressed by reference to
high level design documents but this does not mean that these design
aspirations were always effectively implemented in practice. Database
integrity requirements and validation rules are ultimately tailored and
implemented in accordance with defined business process rules by the
system designer and database administrator.
It appears that much of the transaction integrity is applied at Database
Level, ensuring that database records are committed appropriately.
Transactional integrity does not provide full coverage at the logic level
within Horizon as is evidenced with certain transactions being classified
as ‘recoverable’ and ‘non-recoverable’ when Horizon suffers from a
system fault and tries to re-process or rollback transactions. Some of
the transactions are inconsistent or incomplete and therefore careful
action needs to be taken by the Subpostmaster to understand these
transaction inconsistencies. As was evident in the witness statement of
Mrs Burke, Horizon itself misrepresented the correct state of the
transactions when recovery was invoked.
Whereas Dr Worden says (at paragraph 250 of his report) that he has
seen evidence of the pervasive presence of transactional integrity in all
Horizon subsystems he has examined, he gives no information as to
which subsystems he is referring to here, or which examinations he has
carried out. I note he does not acknowledge any of the instances of
failures which are apparent in the documents, as I have identified
above.
Measures to Correct User Errors - Which also Cancel the Effects of Software
Errors
5.108
“DUE” “UEC”
At paragraphs 251 to 257 of his report, Dr Worden refers to the concept
of “User Error Correction” enabling the facility of correcting many
software errors. It should be noted that this would not apply to any bugs
Prepared by: Jason Coyne
fe)
Occupation: Partner : .
Specialist Field: IT Systems : It FC
[e)
On the Instructions of: Freeths LLP
POL-BSFF-0100992_0152
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 149 of 265
/errors and defects unbeknownst to Fujitsu or the Subposmaster. It is
evident from the PEAK analysis that often bugs lay undetected for
weeks, months or years. Where a Subpostmaster might have unduly
“put the funds in” to balance a cash shortage (believing an error to be
their mistake rather than Horizon) incidents that caused these
discrepancies and bypassed alerts or were unnoticed events would not
be detected as everything would effectively “balance” and therefore
“User Error Correction” would not capture all software errors where they
might necessarily be.
5.109 Further, in relation to the “many software errors resembling user errors
were also corrected” Dr Worden does not evidence which software
errors were resolved correct user errors. Dr Worden provides the view
that a proactive approach to correcting user errors by Post Office/Fujistu
is effectively the countermeasure here. However, the evidence
illustrates to me (by review of the PEAKs) that it is more often than not,
the Subpostmaster reporting the error in the first instance, which
prompts investigation, and ultimately resolution of what is incidentally,
fundamentally a software bug. Therefore, it is largely user
dispute/investigation, that prompts this error correction. Where the
error is not effectively known, no correction could occur. This is evident
from the analysis of the “acknowledged” bugs within Section 3 where
bugs, errors and/or defects were reported by a Subpostmaser, and
retrospective analysis ultimately uncovered more incidents over varying
years prior to the Subpostmaster raised incident that led to the actual
full discovery.
5.110 Aside from the above limitations in respect of identifying software bugs
through correction of user errors, I do not dispute that Horizon integrity
measures and processes did exist and capture, identify and enable the
rectification of many occurrences of bugs/errors and defects, whether
due to user or software/hardware fault. However, in my opinion it is
important to consider that these are not always effective, and it is
difficult to quantify how many errors are not corrected when the system
Prepared by: Jason Coyne
fe)
Occupation: Partner : .
Specialist Field: IT Systems ) It 1 ¢
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0153
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 150 of 265
operates as it does, where Subpostmasters may have put the funds in
to correct user or software errors which had not been identified as such.
Defensive Programming (“DEP”)
5.111 Dr Worden identifies “Defensive Programming” as a modern
countermeasure programming technique for checking and validating
data sent between different modules to prevent and detect bugs and
errors. I concur that this is an accepted typical modern industry
aspiration for enterprise software packages, but this does not in itself
eliminate or assist with the detection and prevention of all errors and
bugs, and it is wholly reliant upon the adequate constraints being
applied at the user interface of the application/platform.
5.112 At paragraph 262 Dr Worden references specific design documents of
which he opines satisfies evidentially that Horizon was defensively
programmed. The document?4? has a paragraph of defensive
aspirations: “Database applications should be designed and built
defensively, so that they can handle any type of unexpected conditions
in a controlled manner...”. The statement is generic and aspirational but
does not audit the actual Horizon systems built.
5.113 The second document referenced by Dr Worden in this paragraph?“ is
the design overview for Horizon Online but it does not explicitly
reference any “Defensive” aspirations in its programming.
5.114 The third document referenced by Dr Worden?*5 is a further design
overview for the next generation of Horizon Online (“HNG-A”), the
document does not expressly reference any defensive programming but
does include (at 7.1.1.2) an express warning that the Java programming
languages defences would be ineffectual in certain “Code Injection”
circumstances:
243 TDARCO01_4.8.doc, Technical Environment Description, 22 October 2002 {POL-0444096}
244 ARCSOLARCO0001.pdf, Horizon Solution Architecture Outline, 7 April 2016 {POL-0444099}
245 ARCAPPARCO0009_5.doc, HNG-X Architecture - Counter Business Application, 4 August 2017
{POL-0444098}
Prepared by: Jason Coyne
fe)
Occupation: Partner : .
Specialist Field: IT Systems ) It 1 ¢
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0154
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 151 of 265
5.115
5.116
5.117
“there is still the possibility of code being compiled on another machine
and be injected in the event of a counter being compromised...one has to
assume that if the JAR files can be compromised, that the policy file can
also be comprised (sic) making this defence ineffectual.”
As the next generation Horizon Online (HNG-A) is built on the same
technology as its predecessor (HNG-X) it is therefore likely that this risk
of a lack of defence from “Code injection” was also apparent in all
versions of Horizon Online, but perhaps was not discovered at the time
of its design and therefore the earlier design documents do not have
the same express warnings as the more recent ones.
After seeing the express warning about the risks of a lack of defence
and the lack of any real detail in the design or evidence of where this
defensive programming is actually used within Horizon I am surprised
that Dr Worden is satisfied that Horizon is actually defensively
programmed.
Dr Worden explains at 443 and 444 of his report that some of the KELs
show that Defensive programming was used. The opposite position to
this, which must be considered is where defensive programming was
not used, the bug/error or defect likely slipped though undetected and
was not caught.
Redundant Storage of Data (“RDS” “MID")
5.118
5.119
Dr Worden describes general principles of “Redundant Storage of Data”
at paragraphs 263 to 266 of his report, and the fact that there were
multiple redundant copies of the same data within the architecture of
Horizon. I agree that redundant copies of data are a robustness
countermeasure and this provides a means of integrity checking data at
various points in the system.
However, for the countermeasure to be fully effective, it requires all of
the varying data sets consulted to contain complete and fully accurate
data. I have previously stated my opinions in respect of errors
potentially introduced from consulting only a subset of the available
Prepared by: Jason Coyne
fe)
Occupation: Partner : .
Specialist Field: IT Systems : It FC
[e)
On the Instructions of: Freeths LLP
POL-BSFF-0100992_0155
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 152 of 265
data when dealing with branch discrepancies in relation to the Helen
Rose report at paragraphs 5.174 to 5.181 (Pages 88 to 90) of my first
report, and the limitations of the Credence Management Information
System.
5.120 Post Office, when investigating issues reported by Subpostmasters have
a number of different systems in which they can access data. The
systems I have identified as most regularly consulted are: Credence,
HORice and POLSAP. These systems have different data (a subset) from
those which Fujitsu has access to.
5.121 Fujitsu has access to the data that Post Office does, but with additional
data at a lower level. This will include (but is not limited to) system
audit logs, database monitoring information, user access logs, PEAKs
and KEL databases, plus the ability to investigate and identify the
possible impact of work that they have performed whilst supporting the
Horizon systems. Post Office only has access to this data if it expressly
requests it from Fujitsu, which it appears it rarely does according to Mr
Godeseth’s witness statement (see paragraph 5.127 above).
5.122 It is only when all data is considered (not the subset accessible by Post
Office) that redundant data storage can truly be of the value Dr Worden
suggests.
5.123 Dr Worden explains at paragraph 456 of his report that “KELs provide
many examples of where RDS was used”, KELs are Fujitsu documents,
not Post Office documents. Consequently, Post Office would not typically
view KELs, and therefore it is not a countermeasure that would apply
for them in understanding reported issues. Further, KELs typically do
not reference the findings from correlations of redundant data storage
inspection, so in my opinion, do not evidence such a countermeasure.
5.124 Dr Worden refers to “Manual Inspection of Data” (paragraph 463) as
being:
vers one of the most important countermeasures in Horizon.”
Prepared by: Jason Coyne
fe)
Occupation: Partner : .
Specialist Field: IT Systems ) It 1 ¢
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0156
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 153 of 265
Whilst I agree in principle that the scrutiny of data has an important
role to play in any commercial IT system; it is my opinion that its
importance as a countermeasure is overstated in Dr Worden’s report.
5.125 It assumes that the person scrutinising the data is able to identify the
correct source of data to be relied upon, in order to rule out what may
or may not be erroneous in the first instance. Additionally, its utility as
a countermeasure is heavily reliant upon the person scrutinising the
data already knowing there is an issue that requires data inspection.
Finally, it relies upon a human element/input which is difficult to
measure i.e. does that person have sufficient skills or knowledge to
identify issues when scrutinising data and is also prone to a degree of
mistakes.
5.126 I have also commented earlier in this report (Paragraph 5.40) on the
limitations of utilising some of Post Office’s Management Information
Systems as an error proof source of determining financial integrity.
The Audit System (“SEK” “RDS”)
5.127 Auditability limitations have previously been dealt with within this report
from paragraph 5.55. What is fundamental to measuring whether this
was an effective robustness countermeasure is not whether audit
facilities existed, but, how effectively they operated, and how often they
were consulted to investigate and assess the events of bugs/errors and
defects in measurement of Horizon’s robustness.
5.128 At paragraph 270, Dr Worden states that the evidence he has seen in
the KELs indicates that the use of the audit database was a backstop,
and rarely used - because other comparisons of data were usually
enough to diagnose the problem. He also says at paragraph 452 that
the comparative lack of KELs using the audit system provides
confirmatory evidence that the other countermeasures were effective
5.129 I would therefore say it is possible that in some cases, consulting data
other than the ARQ resulted in problems being diagnosed / resolved
Prepared by: Jason Coyne
fe)
Occupation: Partner : .
Specialist Field: IT Systems ) It 1 ¢
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0157
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 154 of 265
5.130
accurately. I don’t think it is possible to say however that because the
ARQ was not consulted it did not need to be.
In the first Witness Statement of Mr Godeseth?4#* dated 29 September
2018, he explains (at paragraph 31) how rarely used it actually was. He
displays how many requests for data from the audit store Post Office
has made of Fujitsu since 2014 (which I believe is the first time that
these records were kept. Therefore, from Horizon inception, to 2014, it
is not possible to identify how often Post Office made such requests.
Year
Number of ARQ months requested. (the numbers
represent 1 months’ worth of data per branch), i.e., if
Post Office request Blackpool data for June and July
2016 that would be two ARQ Months.
2014/15 729
2015/16 103
2016/17 323
2017/18 364
5.131
5.132
At paragraph 32 he explains that he is not aware of any instances where
data retrieved from the Audit Store differs from other sources. Whilst
that may be correct, it is the case that the data available to Post Office
via Credence and other management information systems (including
basic ARQ logs) is only a subset of the complete data set and may
indicate a different outcome to that when viewing the more complete
audit data only available to Fujitsu. This is set out in the Helen Rose
report.?47
In that report, the author explains that audit data available to Post
Office appeared to show (or was interpreted as) being a reversal
246 {Witness Statement of Torstein Olav Godeseth, 27 September 2018}
247 Horizon data Lepton SPSO 191320 CONFIDENTIAL.DOCX, Horizon data - Lepton SPSO 191320,
12 June 2013 {POL-0221677}
Prepared by: Jason Coyne fe)
Occupation: Partner ae
Specialist Field: IT Systems It Jrou f )
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0158
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 155 of 265
5.133
5.134
5.135
5.136
5.137
initiated by the Subpostmaster. This position changed after Post Office
requested that Gareth Jenkins of Fujitsu should look at audit data and
system logs, which he did and explained that he could see that the
reversal was in fact conducted by the Horizon system, not the
Subpostmaster. This demonstrates two positions, firstly that the data
most commonly used by Post Office for their investigations is either
wrong or does not provide sufficient information to complete the full
picture. Secondly, it was only after the Subpostmaster sought the
advice of a forensic accountant that the full audit data was requested,
indicating that disputes had to be upheld by Subpostmasters to get to
the correct identification and resolution.
The conclusion of the report suggests the possibility of losses occurring
because of such issues, and that Subpostmasters might be considered
liable for a loss that ultimately arose from a Horizon generated event.
For any part of the audit system to be of proper use, the position
presented must be consistent, if the position differs depending upon
which audit file you view, then the audit process is unsuitable.
Dr Worden includes “Secure Kernel Hardware and Software” as a
countermeasure and references this in various points in his report.
The term ‘Secure Kernel’ in industry, is typically associated with
software/hardware components that enforce basic security procedures
for controlling access to system resources. ‘Kernels’ implement access
provisions based on resource/functional capacity. They do not comprise
(as suggested in Dr Worden’s use of the phrase or acronym) an entire
security policy or safe guard against a lack of process control in respect
of access rights.
Dr Worden states at paragraph 452:
“Because the core audit system was a backstop countermeasure, which
was only used when other ways of investigating any anomaly had not
given an unambiguous result, it was only rarely used, and the KELs
provide little evidence of its use. This comparative lack of KELs using the
Prepared by: Jason Coyne
fe)
Occupation: Partner : .
Specialist Field: IT Systems : It FC
[e)
On the Instructions of: Freeths LLP
POL-BSFF-0100992_0159
181024SR1935
POL00262929
POL00262929
01 February 2019 Page 156 of 265
audit system provides confirmatory evidence that the other
countermeasures were effective”
5.138 It is important to note that there are several limitations to the above
statement. Namely:
a.
Cc.
The KELs alone cannot be relied upon for a complete view of the
complete investigation of a bug/error or defect nor for drawing
inference as to how often audit data was requested as that was
not their purpose;
PEAK records illustrate that consultation of ARQ data was
commented on internally by Fujitsu support staff in some cases,
although it should be further noted that it was not always provided
and in some cases it was either:
lengthy in the time taken to provide it for analysis (delay cited
as due to prosecution evidence backlogs, (See PC0070364748
and PC0073492749);
not available or not documented as being provided or findings
concluded and the PEAK ticket subsequently closed
(PC022053275° above, PC0228049,251 PC019883872 and
PC0120511753); or
did not contain fully accurate data (PC021183375* and
PC0206932255).
Post Office themselves would not consult KELs or PEAKS when
taking decisions on Subpostmaster branch accounts.
5.139 Auditability has been dealt with further in this report (see Paragraph
commencing 5.55).
248 PEAK PC0070364, 2 October 2001 {POL-0440173}
249 PEAK PC0073492, 29 January 2002 {POL-0440178}
250 PEAK PCO220532, 5 September 2012 {POL-0441342}
251 PEAK PCO228049, 30 August 2013 {POL-0397525}
252 PEAK PCO198838, 11 May 2010 {POL-0368687}
253 PEAK PCO120511, 5 May 2005 {POL-0440395}
254 PEAK PCO211833, 5 August 2011 {POL-0441214}
255 PEAK PC0206932, 6 December 2010 {POL-0376676}
Prepared by: Jason Coyne fe)
Occupation: Partner ae
Specialist Field: IT Systems It Jrou f )
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0160
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 157 of 265
Data Driven Software (“DDS”)
5.140 Whilst I agree with the general principles behind the concept of “Data
Driven Software” that Dr Worden discusses at paragraph 272 of his
report, data driven software contains many disadvantages as well as
advantages in the respect of how it is applied within Horizon.
5.141 Reference Data is critical for the correct operation of a large variety of
elements within the Horizon architecture as outlined at paragraphs 4.19
to 4.20 of my first report. Whilst Dr Worden implies that the
implementation of data driven software is in itself an effective
robustness countermeasure, the management of the essential reference
data has proved to be the cause of bugs/errors and defects within
Horizon. As commented on by Mr Parker in his second Witness
Statement (paragraphs 11 and 12).?5°
5.142 An inherent limitation of data driven software is that it is reliant upon
the reference data itself being correct and controls and procedures for
ensuring it is effectively managed and maintained must be stringently
controlled. Often, reference data, and the fact that it can be so
frequently manipulated, enhances more risk within a system due to its
frequency of change.
5.143 As I identified in my first report at paragraphs (4.21, page 24 and 5.33
to 5.34, page 50) errors with Reference Data could and did impact on
branch accounts.
5.144 Dr Worden at paragraph 448 of his report recognises that “KELs show a
significant number of faults arising from faulty reference data”, but
downplays the possibility of them affecting branch accounts, and
suggests they were always easy to diagnose and fix, concluding overall
that “the countermeasure DDS has been highly effective”. I agree, they
typically are easier to fix as only partial data needs to be changed but
the initial identification and impact of such faults on the operation of
Horizon, including the possible impact on branch accounts is no different
256 {Second Witness Statement of Stephen Paul Parker, 29 January 2019}
Prepared by: Jason Coyne fe)
Occupation: Partner .
Specialist Field: IT Systems ; It lI
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0161
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 158 of 265
from standard software. An example from the ‘Atos / Fujitsu Problem
Review Tracker?5” {POL-0449089} explains an issue first identified in
September 2010 was brought to Post Offices attention on 18
December 2012 where it was identified as being a Reference Data bug.
Post Office closed the call on 27 November 2013 over three years later.
It is not clear from the tracker if this impacted branch accounts.
Software Coding Standards and IT Architecture (“ARC”)
5.145 Dr Worden in his report relies on software coding standards during the
development and testing of Horizon (see 6.2.8 Software Coding
Standards (ARC), paragraph: 278 - 281) as a countermeasure.
5.146 The software code of Horizon has not been provided to the Experts, so
it is not clear to me how Dr Worden would know if it is coded to any
standard. I perceive that Dr Worden is basing his opinion on the design
aspirations for Horizon.
5.147 Dr Worden makes a generalised comment at Paragraph: 281 that
“Informally, compared with many other large IT estates I have seen,
Horizon appears to have been a tightly-run ship.” It is not clear what
information Dr Worden is basing this opinion on.
5.148 Whilst I do acknowledge the importance of system architecture in the
design of any IT system it is my opinion that this cannot and does not
in itself prevent the occurrence of bugs/errors & defects and issues in
any IT system and is a design aspiration with no guarantee of an
infallible system.
Reconciliation, Transaction Corrections and Acknowledgements
5.149 Dr Worden deals with reconciliation, transaction corrections (TC) and
acknowledgements (TA) at paragraphs 282 to 294 of his report and he
concludes that they are “a very important part of the robustness
countermeasures built into Horizon, particularly for UEC". I have
257 Weekly Update 26062018 - FJ.XLSX, Atos / Fujitsu Problem Review Tracker, 26 June 2018
{POL-0449089}
Prepared by: Jason Coyne
fe)
Occupation: Partner : .
Specialist Field: IT Systems ) It 1 ¢
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0162
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 159 of 265
addressed user error corrections (Dr Worden’s “UEC”) at paragraph 5.89
aboveError! Reference source not found., and deal more fully with
reconciliation and related topics in sub section 9 below.
5.150 In my opinion consideration of the robustness of the Transaction
Correction (TC) process as a countermeasure to correct user error
requires consideration of how mismatched data is investigated and
corrected before a TC is issued, the information available to a
Subpostmaster before accepting a TC, and the way in which disputes
can be raised and are resolved. The fact of there being a TC process is
not in itself evidence that it performs as a robust countermeasure
against error. I have seen evidence indicating that the TC process is
itself prone to error, which introduces the risk that rather than acting
as a countermeasure, the process of issuing TCs could itself introduce
errors into branch accounts. The process of investigation before issuing
a TC and when a dispute is raised is not made clear in the Defendant’s
witness evidence and I do not know how extensive or thorough this
process has been. It is also the case that the TC investigation is
completed by Post Office on the subset of data available to it and would
exclude the audit data available to Fujitsu. I have commented on this
further in section 9.
5.151 Similarly, the robustness of the TA process requires consideration of the
processes by which TAs are issued, the information available to a
Subpostmaster before accepting a TA, and the way in which disputes
can be raised and resolved. Again, I do not have full information about
the internal Post Office processes to know how carefully these steps are
managed to avoid the risk of Subpostmasters being incorrectly issued
with TAs, or disputes about TAs being resolved incorrectly.
Hardware and Software Resilience (“RHW")
5.152 Dr Worden defines “Reliable and Redundant Hardware” as a robustness
countermeasure against a fault or malfunction which causes an entire
system to stop operating. He relies generally on hardware, software,
Prepared by: Jason Coyne fe)
Occupation: Partner .
Specialist Field: IT Systems g It ]
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0163
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 160 of 265
infrastructure and networks, in addition to qualities of recovery from
failure, transactional integrity and security. The topics which Dr Worden
deals with here are highly generalised, and I have addressed them
elsewhere in my first report or within this report.
5.153 Ido agree that generally, the Horizon hardware appears to be adequate
(although Subpostmasters have reported many problems with branch
counter equipment). However, I do not believe the software to be
‘resilient’. There are thousands of references to bugs and defects in the
software code and reference data and high numbers of release notes,
whilst the detail has not yet been disclosed it is suggested that 19,842
changes have been required through Horizon’s lifetime - some of which
will be to include new functionality, others of which will be to fix
bugs/errors and defects (see Annex A).
Security and User Authentications (“SEC”)
5.154 Dr Worden has identified a range of design principles and policies in
relation to security, and identifies general points about the importance
of user authentication, data integrity and audit, which in principle I
agree are important aspects of the security of a system.
5.155 However, Dr Worden has not considered in this section the adverse
documentation which indicates that these controls were not well
implemented and there were risks in the way the system was operated.
5.156 I identified in my first report at paragraphs 5.179 to 5.181 (page 89 to
90) and 9.65 to 9.67 (page 149) that in 2011 Ernst & Young had
identified in a letter to Post Office?** the lack of internal control with
third-party providers adding to the risk of unauthorised and
inappropriate changes being deployed. I note that Dr Worden mentions
this letter in his report (at paragraph 503), where he says that he has
not seen evidence of whether Fujitsu and Post Office implemented the
corrective actions which were recommended.
258 POL Management Letter FINAL.docx, Management letter for the year ended 27 March 2011,
August 2011 {POL-0219218}
Prepared by: Jason Coyne
fe)
Occupation: Partner : .
Specialist Field: IT Systems ) It 1 ¢
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0164
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 161 of 265
5.157 At section 9 of my first report, in respect of remote access, I identified
that whilst this is an important tool in supporting large enterprise
systems it also represents a potential security risk and weakness with
any type of security set up. I deal with remote access and the relative
absence of controls in relation to remote access further in subsection 11
below.
5.158 I also explain the auditability limitations earlier in my report (see
paragraph commencing 5.55) to demonstrate that the audit database
does NOT record all activities and therefore from a security perspective
cannot be fully relied on to provide a complete picture when auditing
transactions.
5.159 And finally, Post Office did not make good use of the audit data logs,
either for Subpostmaster activity, neither did they enforce or seek to
validate the actions of its contractors Fujitsu, Atos or others.
Development and Testing of Horizon
5.160 Within section 6.6 paragraphs 310 - 311 of his report, Dr Worden
comments positively on Post Office’s organisation and governance and
within Appendix C.6 (paragraph 325 to 329). His views are largely based
on organisational charts, and high-level aspirational documentation.
5.161 As to quality in the development, testing and support (addressed by Dr
Worden at paragraphs 312 to 318 of his main report), Dr Worden relies
on the 2005 Business Management Policy and 2006 Programme
Assurance Management Plan. These are very high level, generalised
management documents, and are the type of policy documents which I
would expect any large organisation to have. I do not consider them to
be particularly helpful in considering the Horizon issues over the whole
lifespan of Horizon.
5.162 I am in agreement with the general statements made about the
importance of testing in principle, as set out in within Dr Worden’s report
(paragraphs 320 to 329). It must be acknowledged that however good
the development and testing was, bug/errors and defects made it
Prepared by: Jason Coyne fe)
Occupation: Partner .
Specialist Field: IT Systems g It ]
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0165
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 162 of 265
through to the live Horizon system. I deal further with Testing Good
Practice as a specific countermeasure identified by Dr Worden below.
Horizon in Service
5.163 Dr Worden comments on a number of high-level issues in section 6.7 of
his report. My overall view is that Dr Worden’s approach is simplified
and overly optimistic, based upon there being defined documentation
(paragraph 338) in addition to containing very generalised comments
about the skill sets of support teams (paragraphs 340-341). I have not
found the Horizon documentation to be well maintained and have had
to rely upon many ‘draft’ versions of documents provided within this
disclosure. Often there are inconsistent naming conventions across
documents that are documenting the same thing.
5.164 I do consider the process by which PEAKs and KELs were managed to
be important, and whereas Dr Worden gives a very positive account of
this process, I have addressed what I consider to be important
limitations in respect of them from paragraph 3.277 above. I have also
noted the limitations which appear from Mr Parker’s description of the
process, at paragraph 4.88 above which in my view are significant
Robust Data Communication and Replications (“ROC”)
5.165 There are a number of separate transport networks for data
communication within the Horizon system. The branch counters
communicated with the data centres over telephone and broadband
networks. Then, within Fujitsu’s data centres, the data travels between
the various servers using local area networks. If appropriate, the data
would travel to the various external clients, such as banks, Camelot,
DVLA, etc via wide area networks. Dr Worden has focused on Riposte,
which provided the messages which travelled across the networks
between the branch Counter and Fujitsu data centre. The scope for
communication errors in Horizon is far wider than Riposte alone.
Prepared by: Jason Coyne fe)
Occupation: Partner .
Specialist Field: IT Systems ; It lI
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0166
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 163 of 265
5.166
5.167
5.168
I have challenged Dr Worden’s view on Riposte’s reliability earlier in my
report (see paragraph 5.29) which covers the period up to 2010 (Legacy
Horizon) One example is PEAK PC002758125° which documents a
specific Riposte fault observed by a branch in July 1999 where the
Subpostmaster was found to be logged on to two counters at the same
time, logging on and off was controlled by Riposte. This was thought to
be impossible but was tested and later confirmed to be fault allowing it
to be possible. The ‘fault’ was reported to Escher (the authors of
Riposte) in February 2000. It appears that in September 2000 Escher
confirmed that a fix would be forthcoming. By July 2001 it was recorded
that Escher had not provided any fix. The PEAK concluded without the
fault ever being recorded as being fixed (certainly not in the PEAK).
Post 2010, Horizon Online used a different data communication method
(web service interface) to Riposte but on the evidence of the PEAKS
identified in my first report (paragraph 5.46, page 53) there continued
to be communication issues with Horizon.
Mr Stephen Paul Parker does not identify a specific point in time but
explains in his witness statement (paragraph 36) that poor data
communications kept Fujitsu support busy;
“There were times when the SSC was very busy, for example, networking
problems causing application issues across the whole estate and data
centre outages.”
Manual Workarounds (“WOR")
5.169
5.170
At section 7.7.12 of his report Dr Worden discusses the manual
workarounds applied within Horizon. Dr Worden claims that the Manual
Workarounds were effective as a countermeasure.
A manual workaround is ultimately a set of steps adopted to circumvent
a process that is not currently supported within the system. Use of any
sort of workarounds should be a temporary measure and reliance on
259 PEAK PC0027581, 9 July 1999 {POL-0221763}
Prepared by: Jason Coyne fe)
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0167
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 164 of 265
5.171
5.172
5.173
5.174
workarounds for critical system processes does suggests that Horizon is
less robust than it should have been. Manual workarounds typically are
less precise and often it is the manual workarounds that include a
greater risk of human error from mis-keying or failure to follow the
correct steps.
As opined in my first report (paragraph 5.97, page 66) reliance on many
workarounds are indicative of a lack of robustness. In addition, I have
also previously identified in my first report (paragraph 5.10, page 44) a
KEL?® which records “Office has a Non-Zero Trading Position
(Receipts/Payments mismatch)” a workaround is suggested but it is
recorded that “Unfortunately the workaround cannot be done after the
problem has occurred at the office! In this case the branch accounts will
need to be corrected.” This creates further complexity and additional
risk of error arising from out of process activities that may not be
adequately audited.
Also see 5.256 below where Post Office workarounds are highlighted as
a concern and a risk by Post Office’s external auditors, with the
comment “Widespread non-conformance to Post Office policy and
processes by branches, with an institutionalised acceptance that errors,
workarounds and non-conformance exists. “°°!
Richard Roll in his second witness statement?® at paragraph 18 explains
the problems with workarounds used by Fujitsu not being considered
when Horizon updates where applied and caused previous functionality
to stop working.
In my opinion it is very difficult to view manual workarounds as positive
countermeasures and that their existence actually reduces robustness.
260 KEL wrightm33145J, 23 September 2010 last updated 1 April 2016 {POL-0040409}
261 NRRA1207 10D007-0 50 Draft.doc.docx, Fraud and Non-conformance in the Post Office;
Challenges and Recommendations, 1 October 2013 {POL-0216106}
262 {Second Witness Statement of Richard Roll, 16 January 2019}
Prepared by: Jason Coyne fe)
Occupation: Partner
Specialist Field: IT Systems I
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0168
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 165 of 265
Bug Finding and Correction (“BFC”)
5.175 Bug Finding and Correction is identified by Dr Worden as a specific
countermeasure, but the definition of this countermeasure is very
broad. For example, it encompasses manual workarounds above, this is
an example of the loose and overlapping nature of the countermeasures
which Dr Worden has identified.
5.176 Dr Worden’s analysis of this countermeasure at section 7.7.14 is based
on his examination of KELs and reference to Mr Parker’s witness
statement that incidents with known financial impact are treated with
high priority. I have provided my comments in relation to this analysis,
and my own analysis of the PEAKs and KELs in this report at in section
3 above also commenting on the evidence in relation to the bugs as
dealt with in Mr Parker’s first witness statement at paragraphs 4.79 4.88
and refer to those sections of my report.
5.177 The vast majority of the evidence considered by Dr Worden presents
only the Fujitsu process and that is only the later element of bug finding
and correction. A Subpostmaster reporting a problem first needs to
convince the Post Office helpdesk that a bug existed before it is passed
to Fujitsu for further examination. Evidence is available that shows that
Post Office did not always proactively seek out fault resolution with
regard to bug finding and correction. In a weekly ATOS/Fujitsu Problem
Review Tracker?® one defect is identified that causes a direct impact on
branch accounts by way of a receipts and payments mismatch. Rather
than seeking to understand the full impact of the bug, the document
displays “Post Office are currently not actively investigating as no
branches have reported any losses”. This is consistent with a reactive
rather than proactive approach to bug finding and correction by Post
Office. The earlier bug analysis suggested that branch accounts would
be impacted, and Transaction Corrections would be required.
263 Weekly Update 26062018 - FJ.XLSX, Atos / Fujitsu Problem Review Tracker, 26 June 2018
{POL-0449089}
Prepared by: Jason Coyne
fe)
Occupation: Partner : .
Specialist Field: IT Systems ) It 1 ¢
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0169
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 166 of 265
5.178
5.179
Richard Roll in his second statement explains (at paragraph 14):
“I believe there were likely many cases where subpostmasters would
have been held responsible for problems which had not at the time been
identified as software errors, either because they could not identify the
problem and did not pursue these with Post Office or Fujitsu, or because
when they were raised we (Fujitsu) were ultimately unable to identify the
problem at the time”
The above accords with my own understanding of instances where a
Subpostmaster may have improperly been liable for discrepancies
arising from a Horizon generated error.
Testing Good Practice (“TGP”)
5.180
5.181
At 487 of his report, Dr Worden refers to his review of Fujitsu’s testing
practices. I have considered Dr Worden’s Section 6.6.4 ‘Testing’ review
and note that Dr Worden comments on the “evidence I have seen on
Fujitsu’s testing processes indicates it was well managed and
effective...”. Dr Worden does not explicitly reference any specific
documents here and further refers to Appendix C of his report. Within
this Section C6 ‘Testing’ I note that Dr Worden appears to be referring
to generic aspirational lifecycle phases with regards to testing. I agree
that these are typical lifecycle stages and components that, if followed
iteratively, should reduce the number of development faults within
Horizon prior to it going live. Since Dr Worden does not provide any
specific Horizon document references, I am unclear on how Dr Worden
has seen and can verify such test scripts, or to which aspect of Horizon
they apply.
I have separately seen evidence that Fujitsu did apply Integration and
Testing Strategies within various changes to the Horizon landscape?® }
and agree that these appear to be in line with standard industry process.
However, the low-level results of tests, how any failures were managed
and re-tested, test pass percentages have not been easy to identify. In
264 DESAPPDPR2374_0.3.DOC , DCS changes to take on AMEX, 14 March 2014 {POL-0135502}
Prepared by: Jason Coyne
fe)
Occupation: Partner : .
Specialist Field: IT Systems : It FC
[e)
On the Instructions of: Freeths LLP
POL-BSFF-0100992_0170
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 167 of 265
consideration of the size of the Horizon landscape and the amount of
changes applied over its lifecycle, in my opinion, Dr Worden’s
assessment is largely generic and assumptive.
5.182 Computer systems of this scope are never totally bug free, no matter
how rigorous the testing regime. This is in line with opinions expressed
in my original report (paragraph 5.83, page 63). Testing in my
experience tends to centre around key milestones. For example,
rigorous testing will take place as part of any system implementation
(e.g. the first roll out of Horizon) and any subsequent releases or
upgrades (e.g. Horizon online).
5.183 Similarly, any bug fixes should also be rigorously tested; both of the fix
itself and then to guard against regression (testing to ensure previously
developed and tested software still performs after the change/fix).
Outside of these milestones I would not expect much, if any, testing to
take place and therefore the development of test scripts is effectively
stagnant during these non-active periods. It is therefore possible for
bugs to remain dormant for long periods either because they have not
yet been identified or no test scripts have yet been created that test the
scenarios leading to the bug presenting itself in the first place.
5.184 An example of this can be seen in the Branch Outreach Issue?® which
highlighted a specific combination of events impacting on code produced
for HNGX (Horizon online) that went live in 2010. The combination of
events was not picked up as part of any of the testing phases by the
time multiple fixes were applied in January 2011.
5.185 Other examples are an occurrence of the Suspense Account Bug
(detailed at 4.37 and 3.43 of this report) - which was first logged in
2011 but the bug was not located and fixed until 2013.
5.186 At Dr Worden’s paragraph 488 he suggests that serious bugs are rare
in the KEL and PEAK records. I agree, they are rare in the KEL records
265 Outreach BLE Extract Findings v6 091215.pptx, Branch Outreach Issue (Initial Findings), 10
December 2015 {POL-0220141}
Prepared by: Jason Coyne
fe)
Occupation: Partner : .
Specialist Field: IT Systems ) It 1 ¢
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0171
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 168 of 265
because the purpose of KEL’s are to inform support personnel how to
deal with historic problems, the PEAK’s however do show many serious
bugs as I have set out in Section 3 above. Where Dr Worden states at
paragraph 489 that I have not identified any bugs, this is untrue.
5.187 As aforementioned, KELs cannot be relied upon solely to identify a bug
or its impact. I only obtained the PEAK disclosure within short proximity
of serving my first report. That said, there were clearly bugs/errors and
defects identified within my first report..
5.188 I have dealt with a sample of Dr Worden’s and Mr Parker’s responses
with regards to certain identified KELs within Section 3 of this report ‘Dr
Worden KELs with further PEAK Analysis.’ I do not believe neither Dr
Worden or Mr Parker have considered the full appropriate set of
documentation surrounding any bug/error or defect but rather
exampled instances where they both claim no financial impact has
occurred.
5.189 The second statement of Richard Roll dated 16 January 2019 indicates
at paragraph 15 budget pressures and redundancies impacted on
system development and testing. He states:
“The test team felt they were under enormous pressure to complete
testing within certain timescales which negatively affected the test
regime”.
5.190 Mr Roll continues to highlight further pressures in relation balancing the
amount of time spent on the various streams of work (testing fixes/new
features and development). At paragraph 18 he also highlights a
common issue which should ideally be picked up as part of any
regression testing. This was where updates released to fix specific
issues caused other functionality to cease working. An example of this
is documented by Gareth Jenkins?®* which identified the local suspense
266 DOC_152849967(1) GJ Local Suspense note.DOCX, Loca/ Suspense Problem, 16 November
2018 {POL-0444082}
Prepared by: Jason Coyne
fe)
Occupation: Partner : .
Specialist Field: IT Systems ) It 1 ¢
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0172
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 169 of 265
bug as being an unintended consequence to changes made in respect
of archiving strategy changes.
5.191 I have not had sight of testing documentation covering the entirety of
the Horizon estate through its development, so it is difficult to offer any
firm views on this. Dr Worden outlines in his report appendix at C6
paragraphs 341 - 344 the various industry standard testing phases but
it is not known the extent of Fujitsu testing documentation he has
reviewed to form his opinion.
Quality and Change Control (“QCC")
5.192 Dr Worden explains his quality and change control countermeasure at
section 7.7.16, where he identifies quality control techniques employed
in Horizon as “producing documents in accordance with standards and
templates”; “reviews of specifications, designs and other significant
documents”; and “testing of software, including changes”. Again, Dr
Worden has identified and relies upon a countermeasure which is
assessed by him at a very high level.
5.193 At paragraph 498 of his report Dr Worden maintains that he is satisfied
Fujitsu appropriately assured the quality of Horizon, claiming to have
reviewed many of the thousands of documents produced during the
lifetime of Horizon. I have not had sufficient time to carry out a review
of documentation against the standards as referred to by Dr Worden.
However, from the documents I have reviewed, I have had to rely upon
draft versions, often the latest issue of the particular document not
being formally disclosed until provided as part of responsive witness
evidence or Dr Worden’s report. Where this has been the case I have
set it out in this report.
5.194 In relation to the Managed Service Change (MSC) process (referred to
by Dr Worden at paragraph 502), I have set out my conclusions in
respect of the process at Section 3 and Section 5, subsection 11 of this
report.
Prepared by: Jason Coyne fe)
Occupation: Partner .
Specialist Field: IT Systems ; It lI
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0173
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 170 of 265
5.195 In my opinion Dr Worden’s approach in this section of his report does
not demonstrate that the issue of quality surrounding Horizon has been
effectively managed. It appears to me that he has largely relied on high
level documents recording intended processes rather than analysing the
specific operation of those processes in practice.
5.196 There is evidence of deficiencies in change management, as recorded in
the 2011 E&Y Letter (referred to at paragraph 5.162 of my first report).
This executive summary of this letter states:
“Within the IT environment our audit work has again identified
weaknesses mainly relating to the control environment operated by Post
Office’s third party IT suppliers. Our key recommendations can be
summarised into the following four areas:
Improve governance of outsourcing application management
Improve segregation of duties within the manage change process
Strengthen the change management process
Strengthen the review of privileged access”
5.197 I note the detailed information which is set out in the table at section 2,
at points 12 to 15 concerning points made in the previous year, and in
section 4, the IT specific points made for the current year, in particular
point 3, where the recommendation is made to strengthen the change
management based on the testing which had been carried out (the
rating for this was “high”). I do not know the terms of reference for
Ernst & Young in the conduct of this audit, but I envision they will have
looked at the IT environment and processes in greater detail than I (or
Dr Worden) could have done given the access provided.
5.198 Also, as I identified in my first report, Post Office apparently chose not
to implement recommended mitigation of risks identified by Ernst &
Young relating to “the communication by Fujitsu of changes made to
the Horizon system” (Risk and Compliance Committee minutes dated
Occupation: Partner
Specialist Field: IT Systems
Prepared by: Jason Coyne fe)
git
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0174
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 171 of 265
18 September 2013, as referred to in my first report at paragraph
5.161287),
Managing Non-Functional Requirements (“NFR”)
5.199 At section 7.7.17 (paragraph 505), Dr Worden identifies “Managing
Non-Functional requirements” as a countermeasure, encompassing
resilience “RHW” and security “SEC” and he relies on KELs as indirect
evidence of successful NFR management.
5.200 It is difficult to understand Dr Worden’s reliance on this as a
countermeasure: A non-functional requirement is a qualitative
requirement in this case for a system. Where functional requirements
specify what something should do, a non-functional requirement
specifies its qualities, and how it should be achieved. Consideration of
the system’s non-functional requirements (system throughput
requirements, platform capabilities, security elements etc.) are crucial
design elements to be considered in creating the system as per the
specifications set out by the designer.
5.201 In Dr Worden’s countermeasure table (paragraph 391 of his report) he
refers to the “ilities’ such as manageability, supportability,
maintainability and adaptability when explaining this countermeasure.
5.202 I agree with Dr Worden that direct evidence for management of non-
functional requirements “is unlikely to be seen in working documents
such as KELs (unless some NFRs are insufficient, and problems arise
from that)”. Although, many of the KELs do indeed illustrate where
technical failures have occurred.
Effect of Multiple Countermeasures
5.203 At section 7.7.19 Dr Worden expresses his opinion on the effect of
multiple countermeasures, which he says is his most important
conclusion on the robustness of Horizon.
267 R&CC Minutes 18th September 2013.docx, Risk and Compliance Committee (R&CC) Reference:
R&CC/MIN/SEP13, 18 September 2013 {POL-0217378}
Prepared by: Jason Coyne
Occupation: Partner a i
Specialist Field: IT Systems it JFOuUu f )
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0175
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 172 of 265
5.204
I do not consider his statistical approach at paragraphs 515 to 516 to
be helpful at all, as it is the facts of individual bugs or errors which will
determine whether one or more countermeasures are likely to prevent
impact. In my opinion the evidence of the individual countermeasures
is not as strong as Dr Worden suggests (as I have dealt with above).
The information which is available, particularly from the PEAKs,
indicates that there were bugs/errors and defects affecting branch
accounts The way in which these arose and the systems for detecting
and correcting these did have deficiencies, and Horizon should not be
described as “a highly robust system” as Dr Worden concludes.
Section 7: Robustness of Horizon
Overview
5.205
5.206
After considering the additional PEAK disclosure documents and
responsive witness evidence, my opinion as to the robustness of Horizon
has changed since my first report.
I have reached the conclusion that Horizon is less robust than I initially
considered for the following main reasons:
a. Access to modify the Horizon branch database was not as
restricted as it should have been;
b. I Whilst said to be governed by a documented policy, it was actually
unaudited as to what actions where be taken whilst the access was
provided;
c. Post Office do not consult the full audit data before ruling on a
discrepancy, instead using third party client reconciliation data or
subsections of the audit data from within Credence or HORice;
d. The PEAKS are consistent with many more bugs/errors and defects
shown to impact branch accounts than the initial three
acknowledged by Post Office;
e. Some PEAKS show defects have lain undetected in Horizon for
extended periods without detection;
Prepared by: Jason Coyne fe)
Occupation: Partner : -
Specialist Field: IT Systems g It ] }{
On the Instructions of: Freeths LLP < -
POL-BSFF-0100992_0176
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 173 of 265
f. IThe PEAKS confirm Post Office often only becoming aware of
bug/errors and defects when Subpostmasters report problems,
suggesting that Post Office detection methods are not as good as
initially suggested;
g. PEAKs confirm that Post Office suspend active investigations into
known discrepancy causing bugs due to a Subpostmaster not
reporting shortfalls.
5.207. Dr Worden and I have taken differing approaches in relation to Horizon
Issue 3 with regards to the measurement of the extent of Horizon
robustness. In summary, Dr Worden has (1) identified robustness
countermeasures and assessed how well he considers they have applied
(I have responded to this earlier in my report), and (2) performed a
statistical analysis based upon figures and percentages which he
suggests can be derived from the evidence I do not think it is possible
to carry out the type of statistical, theoretical approach which Dr
Worden has carried out, and I do not believe I have the information or
expertise to assume e.g the percentage chance a Subpostmaster will
report a particular discrepancy which is what Dr Worden has done.
5.208 I have not carried out a financial analysis, as I did not interpret my
instructions as requiring me to do so. Instead, in order to gain an
understanding of the extent of Horizon robustness, I have taken a
“bottom up” approach, identifying sources of evidence where
robustness (or Dr Worden’s countermeasures) have evidently failed.
This differs from the “top down” approach of Dr Worden.
Robustness of Horizon: Dr Worden’s Opinion - Horizon Issue 3
5.209 In this section I set out my opinion in respect of Sections 7.2 to 7.8 of
Dr Worden’s report and subsequently addresses Issue 3 of the Horizon
Issues.
5.210 I disagree that Dr Worden has performed sufficient analysis utilising a
sample of the KEL disclosure alone due to their limitations, which I have
set out at paragraph 3.2 of this report.
Prepared by: Jason Coyne
fe)
Occupation: Partner : .
Specialist Field: IT Systems ) It 1 ¢
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0177
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 174 of 265
5.211
5.212
5.213
5.214
I have set out my opinions in relation to Dr Worden’s countermeasures
within Section 6: Countermeasures. In summary, it is my belief that Dr
Worden has applied broad, design aspirations and generic principles,
defining them as robustness countermeasures and applied them in
relation to Horizon as a whole, not considering or measuring their
adequacy or suitability throughout the varying changes within Horizon’s
lifetime.
I dispute the validity of retrospective risk assessment Dr Worden has
applied (paragraph 362 of his report) in relation to assessing the risk of
bugs in Horizon introducing errors in branch accounts for the reasons I
have set out at 5.16 of this report. In summary, risk analysis is a
forward-looking technique, for an uncertain event or condition that
could impact the project. Looking back using this methodology is largely
meaningless as the risk has either triggered or has not.
wy m
Dr Worden states at paragraph 366 that Horizon was a “green fields’
development, “essentially unencumbered” by any IT legacy. As I
understand it, the initial project commenced in 1995 and was initially
going to be the benefits agency system, when this failed the software
was re-purposed for the post office counters. Further, Horizon Online
inherited many legacy components. Hardware was re-used, processing
systems were re-configured to fit in with Horizon Online, and many
more adaptations were applied. This is illustrated in my first report
(Appendix B- Figure 3 - Page 180 - Horizon and HNG-X System
Overview).
In relation to paragraph 375 of Dr Worden’s report, I note that Dr
Worden appears to relate robustness to risk management from a
software engineering perspective. He states that there is a well-
established practice for “discussing” robustness under the heading of
‘risk management’. This is inadequate. Robustness mechanisms need to
consider the technical foundations, the business processes in use,
amongst many other aspects and physical components of the system.
One should not limit the measurement of robustness to software
Prepared by: Jason Coyne
fe)
Occupation: Partner : .
Specialist Field: IT Systems : It FC
[e)
On the Instructions of: Freeths LLP
POL-BSFF-0100992_0178
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 175 of 265
5.215
5.216
5.217
5.218
engineering practices alone. Nor does risk assessment validate any
robustness measures.
In respect of Dr Worden’s interpretation of robustness at paragraph 381
of his report; he takes the term to mean “manage the risks of
imperfection so they are acceptable”. In then trying to define what
constitutes “acceptable” Dr Worden suggests the possible use of
applying simple probability theory which in the context of Horizon he
states might be “the probability of a software bug causing a significant
shortfall in any branch accounts in any month”. I have not seen
evidence of any numerical risk analysis of probabilities like this being
carried out by Post Office of Fujitsu. I do not agree with Dr Worden’s
suggested possible approach to this in the context of Horizon. It is in
my opinion unsuitable to use this approach to quantify a “significant”
shortfall as any approach ought to take other factors into account. For
example, a shortfall of £5 affecting several hundred branches is no less
significant in my opinion than a shortfall of £1000 affecting a single
branch.
Dr Worden and I agree that robustness does not mean perfection, and
therefore, the key is to determine whether when Horizon fails, it fails
safely. From my review it is clear that bugs/errors and defects which
are located in the PEAK logs show that Horizon has failed in an unsafe
manner and has impacted branch accounts.
I have not been able to sufficiently identify the full impact of all
bugs/errors and defects, due to the limitations in the disclosure as set
out within Section 3. However, I have identified a number of “not
acknowledged” bugs/errors and defects where Post Office appears to
have not considered the impact of a number of issues. Further, where
Post Office/Fujitsu have attempted to state the impact of acknowledged
(and other) bugs, I have found flaws and insufficiencies in their analysis.
Section 3 PEAK, MSC and Privileged User Log Analysis of this report
documents instances of Horizon generated shortfalls, outside of the
Prepared by: Jason Coyne
fe)
Occupation: Partner : .
Specialist Field: IT Systems : It FC
[e)
On the Instructions of: Freeths LLP
POL-BSFF-0100992_0179
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 176 of 265
5.219
5.220
5.221
bugs/errors and defects acknowledged by Post Office (that also resulted
in Horizon generated shortfalls).
, I disagree with Dr Worden’s assertion that bugs can be classified into
the groups he identifies at paragraph 405. Dr Worden bases his
classification upon the visibility (or invisibility) of the effects of the bug
to Subpostmasters, but in my opinion this is fundamentally flawed.
Subpostmasters would often not be equipped to know that an error
message or a discrepancy in their accounts was in fact a bug and may
not know that this was a possibility. Also, the information needed to
identify that any particular discrepancy which had been identified by the
Subpostmaster was in fact caused by a bug would only be available to
Fujitsu. Finally identifying the discrepancy as having been caused by a
bug/error or defect and resolving it depends upon Post Office / Fujitsu
carrying out a full investigation, making all necessary enquiries,
including consultation of the ARQ data which Dr Worden relies upon
(paragraph 408).
Dr Worden’s analysis from paragraph 406 is dependent on his
classification which I do not agree with as above. Further, Dr Worden’s
focus is on the possible evidentiary data available for investigating
bugs/errors and defects but does not consider what actually happened
in practice. The human elements of the processes which Dr Worden
relies upon are fundamental in any assessment of whether the system
worked to appropriately identify and correct bugs/errors and defects
and their impact. These human elements may include for example, what
Subpostmasters were told by the Post Office helpline if they identified a
discrepancy in their accounts, if and when calls were escalated by them
for investigation by Fujitsu, the degree of investigation then carried out
by Fujitsu which data sources were consulted, and the process for
issuing and disputing Transaction Corrections and Transaction
Acknowledgments.
Although Dr Worden relies upon the Core Audit Database (paragraph
408), he considers that Post Office did not consult audit (or ARQ) data
Prepared by: Jason Coyne fe)
Occupation: Partner : -
Specialist Field: IT Systems g It ] }{
On the Instructions of: Freeths LLP < -
POL-BSFF-0100992_0180
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 177 of 265
5.222
5.223
5.224
5.225
because there was lack of a need to (i.e., resolution through other data),
but this was insufficient in my opinion.
The Core Audit Database was stored at Fujitsu and required Post Office
to make a formal ARQ request for data to be exported. Something that
Post Office has done very few times since records were kept (See the
numbers of ARQ requests taken from Mr Godeseth’s statement at 5.130
above). From my review of the evidence it seems that without reference
to the core audit data, Post Office will revert to other sources of
reference, typically third party client reconciliation data or that stored
within Credence or HORice. I have set out the limitations of such data
sets previously in this report.
Dr Worden discusses at 411 the issue faced when bugs/errors and
defects impact Subpostmaster branch accounts, but this fact is not often
evident until monthly balancing. KEL’s are not designed for the capture
of such enquires. However, I have found that some PEAKs do record
such discrepancies if the Subpostmaster has managed to convince the
Post Office helpdesk that Horizon is, or may be at fault. Again, I would
emphasise the human processes are important to consider with regards
to how bugs would be detected and corrected in practice.
Dr Worden (at paragraph 412) refers to the local suspense account bug,
and the document produced by Gareth Jenkins 2° in relation to it, which
I also address earlier in this report at paragraph 4.35 onwards.
Dr Worden goes to some length in attempting to extrapolate and make
inferences based upon the number of reported branches affected by this
bug and the discrepancy amounts in the individual case. I do not think
that this is a useful exercise or that any meaningful probabilities can be
derived from this example, particularly since I am not confident the full
extent of this issue was ever truly captured by Fujitsu or Post Office.
268 DOC_152849967(1) _GJ Local Suspense note.DOCX, Local Suspense Problem, 16 November
2018 {POL-0444082}
Prepared by: Jason Coyne
fe)
Occupation: Partner : .
Specialist Field: IT Systems : It FC
[e)
On the Instructions of: Freeths LLP
POL-BSFF-0100992_0181
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 178 of 265
5.226
5.227
5.228
5.229
5.230
I do not think that I have sufficient information or that it is within my
expertise to make the type of assumptions which Dr Worden makes
about Subpostmaster behaviour in reporting anomalies in their monthly
balancing, which Dr Worden does at paragraph 415.
In my opinion multiple unreported occurrences of a bug where the
discrepancy is e.g. less than £50 may be no less significant than a single
reported occurrence involving £3000, especially considering that
individual branches may well be affected by the same bug on multiple
occasions.
Paragraphs 417 to 421 supports my opinion that bugs/errors and
defects which manifest with low financial impact to the Subpostmaster
may therefore be left resident within Horizon unidentified. I have not
seen evidence of Post Office noticing the effects of bugs in their own
accounts and this leading to the correction of errors in branch accounts,
which Dr Worden seems to suggest might happen at the end of his
paragraph 417.
Furthermore, even where issues (identified by Subpostmasters) may
have been given an initial high priority, high priority does not
necessarily dictate that full and proper thorough analysis was conducted
or if it was not later downgraded in order to prioritise other issues.
In relation to the KEL sampling performed by Dr Worden and his
observations at paragraph 425, I dispute all points he makes (with
exception of 425.5), as in my opinion, it is not possible to provide such
opinions or inferences based upon KEL sampling alone. I do however,
agree with Dr Worden at 425.5 that there were other cases of regression
of fixes aside from the one or two cases as set out by Mr Parker in his
Witness Statement as I have seen this within the PEAKs I have reviewed
(Section 3 above).
Prepared by: Jason Coyne fe)
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0182
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 179 of 265
5.231
5.232
5.233
In relation to Dr Worden’s assessment of how well countermeasures
were applied, I have provided my detailed response to this in Sections
5 and 6 ‘Countermeasures’.
Further, I have provided my observations on Dr Worden’s analysis in
relation to certain bugs we have both analysed in Section 3 ‘PEAK
Disclosure’, specifically above at paragraphs 3.22 onwards.
In relation to the variability of Horizon over time, at paragraphs 528-
529, Dr Worden states that “Horizon’s requirements, design and
architecture have been very stable over its lifetime. This in itself implies
that the robustness countermeasures have been similarly stable.” 1 do
not agree with Dr Worden’s analysis because the introduction of Horizon
Online and network banking accommodations both forced significant
requirement, design and architectural changes across practically the
entire Horizon estate. Mr Godeseth supports this in his witness
statement in which he states: “..Horizon has constantly evolved and
changed since its rollout in 1999...” I also think the implication that
countermeasures have been stable therefore is incorrect. For example,
Dr Worden relies upon “manual inspection of data” as a
countermeasure, but facilities and processes for that have changed over
time. There have been other significant process changes such as in
relation to remote access and auditing of it, which is set out at
subsection 11 of this report.
Issue 4 - To what extent has there been potential for errors and data recorded
within Horizon to arise in (a) data entry, (b) transfer or (c) processing of data
in Horizon?
5.234
In relation to Issue 4, in section 7.9 of his report Dr Worden sets out
his difficulties in interpretation of this Issue and concludes that Issue 4
is all a subset of Issue 3, which he refers to and essentially repeats. The
points I have made above in response to Dr Worden’s assessment of
Issue 3 therefore apply equally to Dr Worden’s assessment of this issue.
Prepared by: Jason Coyne fe)
Occupation: Partner : -
Specialist Field: IT Systems g It ] }{
On the Instructions of: Freeths LLP < -
POL-BSFF-0100992_0183
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 180 of 265
5.235
5.236
5.237
However, in my view I do think that different issues arise in relation to
Issue 4, from paragraph 3.148, I have sought to evidence specific
PEAKs outside of those analysed in relation to Issue 1. In summary, I
have identified further instances of bugs/errors and defects that in some
instances may not necessarily have caused financial impact to a
Subpostmasters branch accounts, but further demonstrate that
bugs/errors and defects evident within Horizon indicate a lack of system
robustness. I.e., Horizon functionality was operating outside of its
expected behaviour (e.g., report sets used for financial consultation
were erroneous or defective.
It should be noted however that those PEAKs referenced in relation to
Issue 1 do still apply under Issue 4 definition as they relate to errors in
data recorded and processed within Horizon. Dr Worden has not
considered that bugs/errors and defects that did not primarily cause
financial impact in a Subpostmasters branch accounts, might ultimately
have done so in a secondary capacity, in that they affected Post Office’s
view of accounts and ultimately the human elements of decision making
in respect of Transaction Corrections.
My conclusion in relation to Issue 4 is set out at paragraphs 5.153 to
5.154 (page 82) of my first report, where I identified that whereas it
has not been possible to measure the full extent of errors in data
recorded within Horizon, it was however, clear that significant errors
have occurred. The additional analysis I have carried out provides
further evidence of the extent of such errors, contained within Section
3 ‘PEAKs that relate to errors in data recorded within Horizon’.
Prepared by: Jason Coyne fe)
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0184
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 181 of 265
Issue 6 - To what extent did measures and/or controls that existed in Horizon
prevent, detect, identify report or reduce to an extremely low level the risk of
the following:(a)data_entry errors; (b)data_packet or system level errors
(including data processing, effecting, and recording the same); (c) a failure to
detect, correct_and remedy software coding errors or bugs; (d) errors in
transmission, replication and storage of transaction record data; (e) the data
stored in the central data centre not being an accurate record of transactions
entered on branch terminals?
5.238 At section 7.10 of his report, in relation to Issue 6, Dr Worden sets out
the difficulties he has in interpretation of this issue and relies upon
subsets of his conclusions in relation to Issue 3. I have responded to
what Dr Worden states in relation to Issue 3 above.
5.239 My conclusion in relation to Issue 6 was set out at paragraph 5.199
(page 95) of my first report where I concluded that due to limitations
found within the Horizon disclosure it had not been possible to establish
the full extent of measures and controls within Horizon to ensure system
integrity, however, of those identified, there were many instances of
failure. Since the evidence suggested that bugs/errors and defects were
often dealt with on a cost/benefit basis, risks of errors arising was not
reduced as far as possible. I remain of this view and have found further
supporting evidence of such in the further analysis I have conducted in
relation to the PEAKs contained at Section 3 of this report.
5.240 To the extent Dr Worden relies upon his inherited opinion from Issue 3,
that Horizon did have countermeasures and controls that were designed
to ultimately reduce risk and the impact of errors, I agree, such
countermeasures if implemented and positioned correctly across the
relevant aspects of Horizon should reduce the risks that the design
identified. Risks that were not identified by the designs were unlikely to
be reduced and Dr Worden does not appear to consider if the
countermeasures were in fact implemented and positioned correctly. Dr
Worden does not propose that any further countermeasures might have
been designed for Horizon. Countermeasures in existence for legacy
Horizon which had completely different non-functional requirements
than Horizon Online are treated the same under Dr Worden’s view.
Prepared by: Jason Coyne
fe)
Occupation: Partner : .
Specialist Field: IT Systems ) It 1 ¢
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0185
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 182 of 265
Fundamentally, robustness countermeasures have to be designed in
accordance with the specific architecture and processing rules in
operation at the time. Any change in architecture or processing rules
can render specific countermeasures ineffective.
5.241 Overall, Dr Worden appears to take the view that countermeasures have
been successful with a few limited exceptions. Although, Dr Worden
accepts the fact that Horizon was subject to a large amount of system
change throughout its lifetime and that this may have impacted the
robustness level (Joint Statement paragraph 2.3, page 8). In my
opinion, the two are inconsistent as I do not consider that a single set
of generic countermeasures applied across a large, changing estate can
accurately demonstrate that it was therefore robust.
Appendix H to Dr Worden’s report (responding to my first report)
5.242 In Appendix H of Dr Worden’s report he responds to a selection of
documents or points made in my original report. I provide my comments
on this Appendix H here, because Dr Worden relies on his section H
analysis at the end of Section 7 of his report (paragraph 568). I respond
below by reference to the sections of Dr Worden’s Appendix H.
5.243 Firstly, in relation to KELs, and section H.2 of Dr Worden’s report, I wish
to identify the following points:
5.244 At paragraph 485 - KEL dsed4733R2° (regarding mis-named recovery
scripts) there are seven associated PEAKs. Referenced in this KEL is
PC0272963,27° raised 13 August 2018 with the original KEL raised 25
July 2013. Clearly the occurrence of “bad” recovery scripts was an
ongoing issue and has happened on more than one occasion. In my first
report this was provided as one example of many to demonstrate the
possible impact of failed recoveries on transaction data integrity. Once
a recovery fails it is usually no longer in the hands of the Subpostmaster
269 KEL dsed4733R, 25 July 2013 {POL-0039482}
270 PEAK PC0272963, 13 August 2018 {POL-0443958}
Prepared by: Jason Coyne fe)
Occupation: Partner .
Specialist Field: IT Systems ; It lI
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0186
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 183 of 265
to rectify and the Subpostmaster is reliant on other countermeasures to
pick this up.
5.245 At paragraph 486 - KEL obengc5933K (regarding communication
failures),271 similarly to the example in the preceding paragraph, has
seven PEAKs associated, with the latest PEAK (PC02721752”2) being
raised on 16 July 2018 after the KEL was raised 12 May 2010. In this
case only partial recovery of the transaction was achieved. I disagree
with Dr Worden and do not believe in these circumstances that
communication failure is rare. Further, consistent recovery failure is
symptomatic of a lack of robustness in not being able to address the
underlying issue and is also indicative of a failure of the “Robust data
communication and replication” countermeasure.
5.246 Dr Worden at paragraph 487 states that Horizon has robust mechanisms
to detect and correct these errors in transaction recovery. However as
highlighted in my first report at paragraph 5.98 Post Office
acknowledges in 2012 under the heading of “process and system gaps”
that there existed a lack of automated controls and significant amount
of manual intervention which in my opinion goes to the heart of the
question of whether Horizon could be considered robust.
5.247 In relation to PEAKs, and Appendix H.3 of Dr Worden’s report,
5.248 At Paragraph 489 - PEAK PC0063227273 - this appeared in my report
at paragraph 5.143 (page 79) and Appendix A (Page 160) regarding
Horizon robustness in relation to transfers of data. The PEAK indicates
a Riposte data transfer failure affecting 401 transactions valued at
£11,708.08. Fortunately, in this instance, the bug was fixed before any
branch accounts were impacted but in my opinion is still relevant when
271 KEL obengc5933K, 12 May 2010 last updated 29 December 2010 {POL-0038204}
272 PEAK PCO272175, 16 July 2018 {POL-0439168}
273 Peak PC0063227, 28 February 2018, {POL-0237798}
Prepared by: Jason Coyne fe)
Occupation: Partner .
Specialist Field: IT Systems ; It lI
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0187
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 184 of 265
5.249
5.250
5.251
analysing robustness and is indicative of a failure of the “Robust data
communication and replication” countermeasure.
At paragraph 489 - PEAK PC00637232”4- this appears in my first report
at paragraph 5.57 (page 56) under regarding “Uncategorised
Bugs/Errors/Defects”. I covered this in some detail and the PEAK record
is associated with KEL DRowe1625k?7> and has 51 other associated
PEAK records. Reading the PEAK record PC0084115”° raised 23
November 2002 indicates the Subpostmaster was angry and frustrated
that no explanation could be given for his trial balance discrepancy. The
final entry on the log dated 20 March 2003 states that Development
have been unable to determine the root cause of this problem. Whilst a
workaround was used to remove any impact on the branch account;
both the frequency and the fact that the cause of this issue was and
remains undetermined challenges Horizon’s data integrity.
Art paragraph 489 - PEAK PC0098844?7? - this appears in my first report
at paragraph 5.28 (page 49) and Appendix A under Errors with Financial
Impact. This PEAK deals with currency exchange discrepancies arising
when carrying out reversals. Contrary to Dr Worden’s assertion I do not
believe this was a rare occurrence and on the contrary, an associated
PEAK (PC0102484?78) involving a discrepancy of £200,000
recommended a fix for the underlying cause of these currency exchange
differences as part of S70 release. This PEAK may not have been
considered by Dr Worden who claims this was a rare circumstance and
not a fault in Horizon with serious impact, which on the evidence of the
latest PEAK I strongly disagree with.
At paragraph 489 - PEAK PC02031312”9 - this appears in my first report
at paragraph 5.59 (page 30) and Appendix A under Errors with Financial
Impact. As stated in my report this was a pre-migration bug (Legacy
274 Peak PC0063723, 10 March 2001, {POL-0238257}
275 Not disclosed at the time of submitting this report.
276 Peak PCO084115, 23 November 2002, {POL-0256969}
277 Peak PC0098844, 06 February 2004, {POL-0270879}
278 Peak PC0102484, 23 April 2004, {POL-0274132}
279 Peak PCO2003131, 18 August 2010, {POL-0372925}
Prepared by: Jason Coyne fe)
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0188
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 185 of 265
5.252
5.253
5.254
5.255
Horizon) and a failure of the system to correctly calculate volumes. In
the absence of further PEAK records this does appear to be correct and
the bug no longer presented itself in the Horizon online (HNGX)
confirming the incident occurred with migrated data and new
data/transactions would not be affected by the bug
At paragraph 489 - PEAK PC02036767®, PCO263451781, PC026657578?
and PC02730462*? all appear in my first report at paragraph 5.46 (page
53) and Appendix A under Errors with Financial Impact. Viewed in
isolation these PEAKs could be considered minor however, they form
part of the recurring failed communication issue which impacted on
recoveries and branch accounts which is indicative of a failure of the
“Robust data communication and replication” countermeasure.
At Appendix H.4 of Dr Worden’s report:
Dr Worden addresses the issue of “Mis-keying” at paragraph 490 but he
does not deal with the points I outlined in paragraphs 5.125 - 5.127
(page 74) of my first report. These focus on internal and external
reports highlighting the extent and cost of dealing with data entry
errors. It is again worth noting the Infosec comment following their
2008 review and shown at paragraph 5.126 (page 74) of my first report:
“The Post Office, its agent, clients and banking partners are suffering the
consequences of a high level of transaction disputes and customer claims
across many financial, and all banking products due to a lack of source
data integrity, i.e. values entered only once without validation”
This is clearly at odds with Dr Worden’s opinion that the Horizon user
interface had all the usual measures built in to identify mis-keying. It
also supports my opinion that “Early Detection of User Errors” as a
countermeasure did not necessarily support Horizon’s robustness.
280 peak PCO203676, 31 August 2010, {POL-0373467}
281 PC0263451, 19 October 2017, {POL-0430967}
282 PC0266575, 26 January 2018, {POL-0433904}
283 PC0273046, 15 August 2018, {POL-0439981}
Prepared by: Jason Coyne fe)
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0189
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 186 of 265
5.256
5.257
5.258
5.259
5.260
At paragraph 491 Dr Worden opines that the use of workarounds are
evidence of three countermeasures (Redundant data storage and
computing, with cross checks, Manual Inspection of Data and Manual
Workarounds). In my first report I offer my opinion on the use and
implications of the use of workarounds at paragraphs 5.97 (page 66)
and 5.171 (page 87). Dr Worden claims no evidence has been cited in
respect of the opinions given in my first report. This fails to take account
of the independent report cited in my first report at paragraph 5.171
(Detica NetReveal?®*) which highlighted amongst other things Post
Office’s:
“institutional acceptance that errors, workarounds and non-conformance
exists”
This suggests that workarounds were an accepted business process with
the Post Office rather than an infrequent and temporary solution to a
bug or process failure as suggested by Dr Worden.
At Appendix H.5 of Dr Worden’s report,
At paragraph 492 - 496 Dr Worden addresses my interpretation of the
response by Post Office (refer to my RFI; Annex A) to the number of
reconciliation exceptions which they confirmed as being 10,000+
transactions per week that had to be “F99’d”. In my first report at
paragraphs 6.33 - 6.40 (pages 104 - 106) I highlighted the Data
Reconciliation Service (DRS) and the components required for
transactions to be automatically reconciled and moved to a “complete”
status in addition to the F99 process which processes the “resolved”
unreconciled records and moves these to a “complete” state.
In order to address Dr Worden’s opinion that it is misleading to portray
the 10,000 events per week as error-prone interventions I need to
clarify that these events represent transactions that could not be auto
284 NRRA1207 10D007-0 50 Draft.doc.docx, Fraud and Non-conformance in the Post Office;
Challenges and Recommendations, 1 October 2013 {POL-0216106}
Prepared by: Jason Coyne fe)
Occupation: Partner .
Specialist Field: IT Systems ; It lI
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0190
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 187 of 265
5.261
5.262
5.263
5.264
reconciled and therefore appear as an exception on the NB102 report
requiring either a corrective action or a TC. Each of these exceptions
will have an associated state which is outlined in the Network Banking
Reconciliation and Incident Management Process document.?®> In
almost every case each of these states will require investigation and
analysis by the MSU followed by either of the following actions by Post
Office:
i. If it is a “value” transaction it will require a financial adjustment
(a TC); or
ii. If is “non-value” transaction Confirm F99 authorisation via BIMS
return
It is therefore my opinion having read the process document there is
always a manual “human” element (MSU and/or Post Office) in deciding
the corrective action in order to resolve these exceptions. This human
determination on such a large volume of data per week represents a
significant risk and potential impact on branch accounts.
At Appendix H.7 of Dr Worden’s report,
At paragraph 500 - 506 Dr Worden addresses the Ernst & Young
report?®° which I identified at paragraph 5.162 in my first report. Whilst
I accept that the 2011 Ernst & Young Management letter contains
recommendations and not obligations; in the opening line of the
executive summary they acknowledge that the Post Office had
addressed many of the issues raised in the previous year’s audit.
Regarding the specific recommendations in the 2011 audit it is my
opinion that the key recommendations directly impact on some of the
18 countermeasures outlined in Dr Worden’s report and therefore are
relevant to the question of robustness of Horizon since they offer an
opportunity to improve these countermeasures which it appears Post
285 Network Banking Reconciliation and Incident Management Processes, 26 February 2003 {POL-
0032841}
286 POL Management Letter FINAL.docx, Management letter for the year ended 27 March 2011,
August 2011 {POL-0219218}
Prepared by: Jason Coyne
fe)
Occupation: Partner : .
Specialist Field: IT Systems : It FC
[e)
On the Instructions of: Freeths LLP
POL-BSFF-0100992_0191
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 188 of 265
Office chose not to take.. I have listed below the four key
recommendations and in brackets which countermeasure(s) these could
impact:
a. Improve outsourcing application management; (Quality and
Change Control, Managing non-functional requirements, Testing
good practice)
b. Improve segregation of duties within the manage change process;
(Quality and Change Control, Security)
c. Strengthen the change management process; (Quality and Change
Control)
d. Strengthen the review of privileged access. (Security)
5.265 At paragraph 507 Dr Worden challenges the relevance of the POLSAP
System Controls document?®” referenced at paragraph 5.170 (page 87)
of my first report but I disagree with his position. The fact that the
report references the Ernst & Young audit and access controls within
Horizon makes it a relevant document.
5.266 At paragraph 508 Dr Worden questions the relevance of not following
the recommendations of the 2013 Ernst & Young audit?®* identified at
paragraph at 5.161 (page 84). I accept that audit findings are usually
recommendations as opposed to obligations however as the Risk and
Compliance Committee meeting minutes is a redacted document, I am
unable to comment further on the reasons and analysis behind their
decision. However, in my opinion since the issue (communication by
Fujitsu of changes made to the Horizon system) is relevant to the
Quality and Change Control countermeasure, it could therefore impact
on the issue of Horizon’s robustness.
287 AR12.037.ppt, Review of Key System Controls in POLSAP, November 2012 {POL-0217341}
288 R&CC Minutes 18th September 2013.docx, Risk and Compliance Committee (R&CC) Reference:
R&CC/MIN/SEP13, 18 September 2013 {POL-0217378}
Prepared by: Jason Coyne
fe)
Occupation: Partner : .
Specialist Field: IT Systems ) It 1 ¢
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0192
181024SR1935
POL00262929
POL00262929
01 February 2019 Page 189 of 265
Conclusions — initial thoughts
5.267 Dr Worden bases his opinion on statements regarding generic IT Risk
reduction counter measures and where such generic counter measures
have been observed in the Horizon designs. In my opinion that fails to
consider:
a. Was the design implemented?
b. If so, did the counter measure provide adequate coverage across
the whole of Horizon?
c. The Horizon system changed frequently, there is no way to
ascertain retrospectively that the designed counter measures
(even if implemented initially) continued appropriately following
each and every change to the Horizon system.
d. Evidence from PEAKs and KELs show that bugs/errors and defects
within Horizon were not always prevented by the counter
measures.
e. Evidence from PEAKS and KEL show that bugs/errors and defects
that were not prevented by counter measures were not detected
until years after the events.
f. Evidence from audits that weaknesses existed in Horizon and
related processes.
Section 8: Effect on Horizon Bugs on Branch Accounts
Issue 1 - To what extent was it possible or likely for bugs, errors or defects of
the nature alleged at §§23 and 24 of the GPOC and referred to in §§ 49 to 56
of the Generic Defence to have the potential to (a) cause apparent or alleged
discrepancies or shortfalls relating to Subpostmasters’ branch accounts or
transactions, or (b) undermine the reliability of Horizon accurately to process
and to record transactions as alleged at §24.1 GPOC?
5.268 Dr Worden and I have approached Horizon Issue 1 in different ways. Dr
Worden has primarily set out a financial analysis, focusing on the
financial impact of bugs, errors and defects based on a small sample
KELs, Claimant data and values from Post Office acknowledged bugs.
Prepared by: Jason Coyne
fe)
Occupation: Partner : .
Specialist Field: IT Systems : It FC
[e)
On the Instructions of: Freeths LLP
POL-BSFF-0100992_0193
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 190 of 265
5.269
5.270
He relies on statistical analysis from section 7 (by reference to e.g.
percentage of Subpostmasters who would likely report a discrepancy).
He uses his analysis to conclude that “Horizon cannot account for even
a small part of the Claimants’ shortfalls - either for all Claimants taken
together, or for any individual Claimant” (paragraph 573).
I did not consider financial impact in detail, because I addressed the
question literally My analysis aims to address it is “/ikely” or “possible”
that bugs in Horizon could have caused the apparent or alleged
discrepancies (as opposed to their financial impact). I have used a
“bottom up” approach by identifying sources of evidence where actual
bugs, errors and defects are recorded Dr Worden’s approach is based
on assumptions which, in my opinion, are technically flawed.
I have set out the basis for these opinions in the following sections.
Unknown Bugs in Horizon
5.271
5.272
5.273
Dr Worden’s overarching opinion in section 8.2 of his report is that the
likelihood of there being any unknown bugs in Horizon was “very small
indeed”, and that the associated impact of those bugs could not have
been large “...because of the robustness countermeasures built into
Horizon.”
At paragraph 579 Dr Worden states:
"579. Part of the purpose of robustness in any financial system is to
ensure that far-reaching errors in accounts do not occur. An important
part of this is to ensure that if errors should occur, they are rapidly
detected - and do not persist, unknown, for long periods. Horizon was a
typical financial system in this respect. In my opinion its robustness
countermeasures worked well.”
As a general principle I agree that systems are built to be robust to
prevent errors in accounts. However, as a matter of technical principle,
it is also true that bugs in a live system are typically discovered because
of a set of circumstances that was not foreseen during the various test
phases. Therefore, it is very unlikely that a system as large and complex
Prepared by: Jason Coyne
fe)
Occupation: Partner : .
Specialist Field: IT Systems : It FC
[e)
On the Instructions of: Freeths LLP
POL-BSFF-0100992_0194
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 191 of 265
5.274
5.275
5.276
5.277
5.278
5.279
5.280
as Horizon would contain robustness measures that allowed one to
assume that significant errors could not occur because they have not
been discovered.
I have noted that Dr Worden appears to agree with this, as he states in
paragraph 650 that the Receipts/Payments Mismatch Issue was a bug
that was:
“triggered by a rare circumstance (which one would not expect to be
exercised in testing) and which had an effect on branch accounts."
Additionally, Dr Worden and I have agreed in the Joint Statement that:
“Each time any IT system (including Horizon) is changed there is the
potential to introduce new bugs/errors/defects.”
Further, the second witness statement of Mr Stephen Parker, at
paragraph 17 discusses that testing did not result in the identification
of all errors and his opinion that:
“The same could be said of every computer system in the world”
It is also a matter of fact in this case that certain bugs did persist,
remaining undetected for long periods of time.
Each bug was initially unknown in the live system and was then
discovered later. Therefore, in my opinion, the most likely scenario is
that there are (and have always been) bugs that have not yet been
discovered. Whether or not those bugs will have a significant financial
impact is not known, so it would be incorrect to assume that would be
insignificant.
Additionally, the PEAKs I have been able to review suggest that the root
cause of an issue was not always correctly determined when initially
identified. Where this is the case, it is not accurate to assume that the
issues were or was not the result of a bug - we know that there was a
problem.
In paragraphs 580-589, Dr Worden sets out his opinions about the
likelihood of a discrepancy being reported by a Subpostmaster based on
Prepared by: Jason Coyne
fe)
Occupation: Partner : .
Specialist Field: IT Systems : It FC
[e)
On the Instructions of: Freeths LLP
POL-BSFF-0100992_0195
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 192 of 265
the value of its impact. I do not believe that there is any significant
value in these assumptions because (a) as I have previously explained,
I do not believe there is sufficient evidence for either me or Dr Worden
to make these sort of assumptions and I do not think they are within
my area of expertise ., (b) two discrepancies may appear in the same
trading period, i.e., the first of £850.99 the second of £-868.13 whilst
both are significant, the net impact on branch accounts (£-17.14) may
be judged as insignificant (c) Dr Worden himself refers to these as
“weak inferences” in paragraph 582.
5.281 Additionally, the “Problem Review Tracker’28? shows a defect opened on
18 January 2017 titled “Products ended retrospectively leading to
Receipt & Payment mismatch". This defect impacted branch accounts
and “transaction corrections” was the recommended remedy. The
tracker further states (24 February 2017):
“Post Office are currently not actively investigating as no branches have
reported any losses”.
5.282 This indicates that, whilst Post Office was aware of an impact on branch
accounts, it was awaiting the branches to report a loss before it
investigated whether the cause was as result of a bug, error or defect.
Impact of Bugs on Branch Accounts
5.283 At section 8.3, Dr Worden’s overarching opinion in relation to the
potential impact of bugs on branch accounts, is that this is to be
assessed by reference to countermeasures only. He says (at paragraph
593):
“Following my analysis of robustness in section 7, it will now be clear that
the answer to this question depends on the robustness of Horizon - not
on how many bugs there were, but on how well the effects of these bugs
were countered and mitigated by the robustness Countermeasures, to
prevent them from creating discrepancies or shortfalls in branch
accounts.”
289 Weekly Update 26062018 -FJ {POL-0449089}
Prepared by: Jason Coyne fe)
Occupation: Partner .
Specialist Field: IT Systems ; It lI
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0196
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 193 of 265
5.284 If Dr Worden is suggesting that there should be no consideration of the
number of bugs, then I disagree with this position.
5.285 Dr Worden describes my approach as “a simple counting or cataloguing
of bugs” (paragraph 594). I do not agree with this description of my
approach, because I have analysed the evidence relating to bugs which
reveals information about the system and the potential for other similar
bugs to arise. This in my opinion is an appropriate way to assessing the
answer to Issue 1. I have provided my detailed comments in relation to
Dr Worden’s reliance on countermeasures earlier in this report. Dr
Worden’s overarching opinion in relation to this point is that assessing
the financial impact of bugs:
“...depends on the robustness of Horizon - not on how many bugs there
were, but on how well the effects of these bugs were countered and
mitigated by the robustness Countermeasures, to prevent them from
creating discrepancies or shortfalls in branch accounts.”
5.286 Dr Worden sets out a summary of the picture which he says emerges
from the KELS (paragraph 596), and conclusions as to the robustness
of countermeasures (paragraph 597). I do not agree with either of these
positions for reasons I have explained earlier in this report.
5.287 In paragraphs 599 & 600, Dr Worden states:
"599. Therefore in my opinion, because the robustness countermeasures
worked very well, there were very few bugs which introduced inaccuracies
in branch accounts, and their financial impact across Post Office branch
network was very small.
600. Mr Coyne's report appeared to imply otherwise. But he had not
analysed the KELs or Peaks to sufficient depth to consider the effects of
robustness countermeasures. Therefore, his report contained little or no
analysis to contradict my opinion. I have examined 62 of the KELs he
relied upon, and they confirm my opinion. This analysis is shown in a
table in Appendix C. My conclusions on robustness, as demonstrated by
those KELS, are contained in section 7.6”
Prepared by: Jason Coyne fe)
Occupation: Partner ae
Specialist Field: IT Systems It Jrou f )
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0197
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 194 of 265
5.288
5.289
During my investigation, when more than 5100 KELs were reviewed,
the focus was on understanding whether there was evidence of bugs,
errors or defects in Horizon which could have been the cause of
discrepancies and shortfalls in branch accounts. In my opinion, there is
significant evidence to show that they were the cause.
I disagree with Dr Worden’s opinion that my first report did not
sufficiently consider the effect of robustness countermeasures (a term
which is introduced by Dr Worden in his report). The bugs, errors and
defects I focused on are the ones which were, by definition, not
adequately prevented by countermeasures.
Measures of Extent
5.290
5.291
Section 8.4 is focused on Dr Worden’s explanation of his interpretation
of Issue 1. I have noted that in paragraph 604, Dr Worden states:
“If time is spent considering these bugs with non-zero but trivial financial
impact, it might divert attention from considering the smaller number of
bugs with significant financial impact, which could have made a more
important difference to Claimants’ branch accounts. Focus on the financial
impact of bugs will help in narrowing the scope of enquiries”
I do not agree that disregarding bugs, errors and defects on the basis
of their net financial impact is the correct approach to understanding
the extent to which bugs could have caused discrepancies. This is
because, assessing how bugs have arisen and how they were resolved,
whatever their value, is informative about the risks of other bugs
arising. Additionally, for example:
a. The fact that a bug has a small impact in one case does mean it
cannot have a large impact in another case.
b. A bug could have a small impact in many different cases.
Prepared by: Jason Coyne fe)
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0198
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 195 of 265
Scaling of Financial Impacts of Bugs
5.292 The statistical analysis as carried out by Dr Worden in this section of his
report is not within my expertise. However, I do not agree with it in
principle because (1) it relies on assumptions / approaches which I have
explained above I do not agree with and (2) it relies on further
assumptions introduced by Dr Worden in this section.
5.293 At 622, Dr Worden states:
“It seems implausible to me that there is some special factor about
Claimants' branches, which makes them much more prone to bugs in
Horizon - bugs which one would expect to strike any branch at random.
Nevertheless, I have considered the possibility carefully in Appendix F. I
have shown there that there is no significant difference between
Claimants' branches and other branches, in proneness to bugs in
Horizon.”
5.294 I have noted the following observations in relation to this:
a. Dr Worden has based his analysis on the assumption that bugs
would affect all branches equally. However, as explained below
(see my Response to Dr Worden’s “Qualitative Analysis”, starting
at paragraph 5.319), this is not correct. As a matter of technical
principle, bugs do not affect all users equally and, as a matter of
fact in this case, bugs have had significantly different effects for
different users (see paragraph 5.322).
b. Dr Worden’s calculations in his Appendix F are based on numerous
assumptions about matters for which there is no evidence, such
as:
i. Claimants are more likely than non-claimants to make errors
(paragraph 435 in the Appendices document).
ii. Estimating probability of bugs occurring in a transaction with
human error against the probability of bugs occurring in normal
transactions (Paragraph 437). He assumes that, because the
system was tested, the probability of bugs occurring in a
Prepared by: Jason Coyne
fe)
Occupation: Partner : .
Specialist Field: IT Systems ) It 1 ¢
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0199
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 196 of 265
transaction with human error could not be more than 4 times
the probability of bugs occurring in a normal transaction.
c. Dr Worden has not considered any factors which could increase the
likelihood of a bug’s occurrence other than human error. For
example, he has not considered the following (non-exhaustive) list
of criteria:
i. On any given day, all Subpostmasters were not always on the
same version of the Horizon software.
ii. Not all Subpostmasters dealt with the same distribution of
transaction types (e.g. Subpostmaster A might sell many lottery
tickets and Subpostmaster B might sell only a_ few,
Subpostmaster C might not have any lottery terminal in the
branch).
iii. Certain Subpostmasters may have busy periods (even if their
overall number of transactions is smaller) or may deal with a
larger volume of very low value transactions.
iv. Internet connectivity varies wildly depending on things like
geographical location, local service providers and more. It is a
matter of fact that this has caused issues with Riposte (see
paragraph 5.165 in this report).
5.295 Additionally, Dr Worden assumes that, because a branch carries out
fewer transactions in a day, it must be less likely to suffer from bugs
than another “larger” branch. In my opinion, this is not a technically
sound assumption. As above, there are many other factors which can
increase or decrease the likelihood of a bug’s occurrence.
5.296 As an example, if there was a bug, error or defect which was triggered
as part of a transaction associated with selling a stamp then, unless
Subpostmaster A sells a stamp, the chance of that bug occurring is 0%.
If Subpostmaster B sells a stamp, then the chance of triggering a that
bug is higher than Subpostmaster A, even if Subpostmaster A carries
out 1000 times more transactions per day, Subpostmaster B still has a
Prepared by: Jason Coyne fe)
Occupation: Partner .
Specialist Field: IT Systems g It ]
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0200
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 197 of 265
higher chance of triggering that bug, because the bug is associated with
selling a stamp.
Three Errors Cited by The Claimants
5.297
In this section, Dr Worden sets out his review of the Receipts / Payments
Mismatch Issue, the Callendar Square/Falkirk Issue and the Suspense
Account Bug. I have noted that he has not given any consideration to
other bugs, errors or defects which have not been formally
acknowledged by Post Office (e.g. he does not give consideration to the
Dalmellington / Branch Outreach Issue (see paragraph 4.44) or the
many others that I have set out at Section 3).
Receipts / Payments Mismatch Issue
5.298
I have set out my views in relation to the Receipts / Payments Mismatch
Issue in the subsection headed “Receipts and Payments Mismatch Bug”
above, starting at paragraph 3.27, and also when commenting on Mr
Godeseth’s second witness statement.
Callendar Square / Falkirk Bug
5.299
5.300
5.301
I have set out my opinions in relation to this bug in the subsection
headed “Callendar Square / Falkirk” starting at paragraphs 3.34 and 4.2
above, and in relation to Mr Godeseth’s second witness statement.
At paragraph 668, Dr Worden states:
“Because Fujitsu had designed the counter software assuming that
Riposte replication worked correctly, and could not anticipate in what
ways it might not work, in my opinion it would have been very difficult
for Fujitsu to fix the problem or correct it. Fujitsu were reliant on Escher
to fix the problem; and apparently Escher did not do this for years.”
It is a concern if Escher did not act for years and Post Office and Fujitsu
were unable to do anything about it. I also note that Horizon is made
up of many more 3” party components, outside of Riposte that failed in
this particular occasion..
Prepared by: Jason Coyne fe)
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0201
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 198 of 265
5.302 In paragraph 669, Dr Worden summarises his conclusions on the
significance of the Callendar Square bug:
a. In 669.1, he notes that this bug was not detected immediately by
“countermeasure DEA”, but sets out that (669.2) that it was
eventually detected by “countermeasure RDS and MID”. He
concludes that Horizon’s robustness worked well. I disagree with
this position. From the period of at least 2000-2006, the bug was
not detected by any countermeasure.
b. In 669.3, Dr Worden concludes that the possible financial impact
on claimants’ branch accounts was “very small indeed". This
appears to be based on his views explained earlier at paragraph
667 that: “IJ would expect the Subpostmaster to be left with a
shortfall (i.e. not compensated) in only a small minority of cases,
if any cases. In my opinion the net shortfall caused by all its
occurrences would be possibly zero, and in any event at most a
few thousand pounds.” Again Dr Worden is making a number of
assumptions for which there is not sufficient evidence..
Suspense Account Bug
5.303 I have set out my opinions in relation to the Suspense Account Bug
above (see the sections starting at paragraphs 3.43 and 4.35). The facts
are as follows:
a. The bug caused historic suspense account figures from 2010 to be
transposed into branches’ suspense accounts for trading periods in
2011 and 2012.
b. When Subpostmasters discovered errors in their accounts they
first queried it in 2012.
c. The cause of issue was not identified by Post Office until 2013.
5.304 In addition, I have noted that Dr Worden focuses on those instances of
the Suspense account bug which had a large financial impact on
claimant branches. In my opinion, this will not provide an accurate
Prepared by: Jason Coyne
fe)
Occupation: Partner : .
Specialist Field: IT Systems ) It 1 ¢
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0202
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 199 of 265
5.305
result when trying to understand the extent to which it was likely or
possible that bugs in Horizon could have cause the alleged or apparent
issues.
I do not comment on Dr Worden’s statistical analysis by reference to a
scaling factor at paragraph 686, because again, this is outside my
expertise.
Dr Worden's Opinion on the Three Identified Bugs
5.306
5.307
5.308
5.309
At paragraph 688, Dr Worden states:
“The experts have not had the time to do this deep analysis for more than
a few errors, including these, and it would be unrealistic to expect the
reader to understand these to the same depth.”
I agree in that it is very unlikely that either I or Dr Worden have found
all the relevant bugs, errors or defects that exist in Horizon which could
have potentially caused the alleged or apparent shortfalls. However, I
have noted that Dr Worden has not attempted to consider any bugs
other than those that were acknowledged by Post Office (for example,
he has not done any analysis in relation to Dalmellington). Since my
initial report, I have located several others which have impacted branch
accounts, these can be reviewed in the table at 3.21 above.
At paragraph 689.1, Dr Worden concludes:
“[The conclusions I draw from analysing these three bugs are: ] There are
extensive robustness countermeasures in Horizon, of many types - so
that even in the rare case of bugs like these which are not handled by
the fully automatic countermeasures, manual countermeasures enable
the bugs to be rapidly diagnosed and corrected, as soon as they are
known about.”
It is not clear how Dr Worden has come to this conclusion. He states
the previous sections that Callendar Square was active from 2000-2006
and that the Suspense Account Bug was not detected by automatic
countermeasure at all and that there was a delay of a year in manually
Prepared by: Jason Coyne
fe)
Occupation: Partner : .
Specialist Field: IT Systems : It FC
[e)
On the Instructions of: Freeths LLP
POL-BSFF-0100992_0203
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 200 of 265
5.310
finding the bug (684.4). In my opinion that could not be diagnosed as:
“rapidly diagnosed and corrected”.
I have not seen evidence which is sufficient for me to conclude whether
all Subpostmasters were compensated for their losses, which Dr Worden
says “the evidence appears to imply’ (paragraph 689.4).
Financial Impact of All Bugs - Main Analysis
5.311
5.312
5.313
5.314
5.315
In section 8.7, Dr Worden a mathematical approach by which he
estimates the what he says is the maximum financial impact of all
known bugs on Claimants’ branch accounts. I disagree with the
approach taken by Dr Worden, which rests on many assumptions I do
not agree with, as I have explained above.
Dr Worden’s approach also relies very heavily on KELs which are not a
complete source of information.
Many of the KELs did not contain enough information to determine
whether the root cause of an issue was a bug/error or defect or
otherwise, or what it’s financial impact was or could have been. I have
previously explained the limitations of KELS in section 3 of my report.
Whilst I do not comment on the actual statistical analysis which Dr
Worden has carried out, in summary, my opinion is that the analysis in
this section (amongst others) of Dr Worden’s report is unlikely to yield
an accurate result because it is based on numerous assumptions and
inferences which often have no technical foundation and which in some
cases are factually inaccurate.
The correct answer to Issue 1 is that it is absolutely possible that bugs,
errors and defects in Horizon caused discrepancies and shortfalls. This
is known because, as a matter of fact, I have identified a number of
bugs, errors or defects which have caused financial discrepancies, and
it is extremely likely that there are (and have always been) unknown
bugs. I do not know the exact financial impact of all of those bugs,
errors and defects. However, I do not agree with Dr Worden that this is
Prepared by: Jason Coyne
fe)
Occupation: Partner : .
Specialist Field: IT Systems : It FC
[e)
On the Instructions of: Freeths LLP
POL-BSFF-0100992_0204
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 201 of 265
something which can be inferred through unsupported assumptions and
an extrapolation of a very limited sample of the available evidence.
Alternative Approaches
5.316
5.317
In 8.8.1 of Dr Worden’s report, he sets out a summary of his conclusion
in his section 8.5. I have set out my responses to this section above
(see “Scaling of Financial Impacts of Bugs” starting at 5.292 above. I
do not agree with his conclusions because they are based on a number
of assumptions which are neither technically sound nor factually correct.
In the remainder of his section 8.8, Dr Worden sets out his review of
several alternative sources of information to use as a basis for
estimating the financial loss incurred by Subpostmasters as a result of
bugs, errors and defects in Horizon. I do not comment further on these
sections which are variations of Dr Worden’s previous statistical
approaches, again based on numerous assumptions which I do not
believe provide a good foundation for the calculations he then carries
out.
Impact of Bugs on Individual Claimants
5.318
Section 8.9 in Dr Worden’s report is an extension of his statistical
analysis in relation to the financial impact of all bugs, errors and defects,
to apply this to a single Claimant. This analysis is not within my
expertise, but it is based on the same assumptions I have previously
explained I consider to be flawed.
Dr Worden’s “Qualitative Analysis”
5.319
In this section 8.10, Dr Worden provides further statistical analysis of
the Claimants’ claims and shortfalls. All of the points I have made above
apply. I do not think this is the right approach. I provide further
comments on this section to the extent it may be helpful, but make
Prepared by: Jason Coyne fe)
Occupation: Partner : -
Specialist Field: IT Systems g It ] }{
On the Instructions of: Freeths LLP < -
POL-BSFF-0100992_0205
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 202 of 265
clear these comments are all subject to my general objection to this
approach and its underlying assumptions.
5.320 At paragraph 804, Dr Worden states::
“The total claim is like a field, divided into 52,000 'plots' (monthly branch
accounts) of approximately equal area. Bugs in Horizon are like raindrops,
falling randomly and uniformly across the field. One would expect
approximately the same number of raindrops to fall on each plot (each
set of monthly branch accounts), apart from random fluctuations.”
5.321 In my opinion, this is not a technically accurate representation of how
bugs affect an IT system. Bugs, errors and defects typically arise in a
live environment as the result of a specific set of factors which were not
considered (usually due to being unforeseen) when testing the system.
These factors could relate to anything, but it is very unlikely that a bug
would arise as a result of some combination of factors that would be
utilised by all Subpostmasters as this would likely have been foreseen
and fixed during testing (prior to go-live of the system or shortly
afterwards).
5.322 Therefore, it would be very surprising for bugs that arise in a live system
to affect all users in a uniform way (Dr Worden’s “Raindrops Analogy”).
Additionally, there is evidence which shows that this was not the case
in relation to Horizon. For example, the “Branch Outreach Issue (Initial
Findings)” document?%° dated 10 December 2015 states on page 10:
"88 different Branches had duplicate pouches over the past 5 years
2 branches have had 5 occurrences
1 branch has had 4 occurrences
2 branches have had 3 occurrences
9 branches have had 2 occurrences
74 branches have had 1 occurrence”
290 Outreach BLE Extract Findings v6 091215.pptx, Branch Outreach Issue (Initial Findings), 10
December 2015 {POL-0220141}
Prepared by: Jason Coyne fe)
Occupation: Partner ae
Specialist Field: IT Systems It Jrou f )
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0206
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 203 of 265
5.323 This document discusses two “potential separate issues” within the
Horizon source code which appeared to start in 2010 and were
scheduled to be fixed in January 2016 and reviews the branch impact
over five years. Dr Worden’s assumption is inconsistent with the fact
that these bugs only affected 88 of the possible 11,000+ branches, with
14 of the branches suffering multiple occurrences of the issue and 74
branches only being affected once.
5.324 The same document also illustrates that there was a range of possible
financial branch impacts, from £1 for some branches to £25,000 for
others. Again, this is not consistent with Dr Worden’s opinion that bugs
affect all users in a uniform way.
5.325 As was set out in my first report at Paragraph 5.6 (Page 44), the
Receipts and Payment Mismatch bug impacted 62 branches with the
majority of incidents being recorded as occurring between August and
October 2010.
5.326 In summary, there is no technical basis to assume that bugs/errors or
defects impact all users or branches equally either in frequency or
quantum. Additionally, the evidence provided to me suggests that this
was specifically not correct in relation to the Horizon system.
Dr Worden’'s “Evidence used for Analysis”
5.327 I comment in this section on some of the graphs and analysis which Dr
Worden has included in his report in relation to Claimant losses, where
I believe there is relevant opinion evidence I can provide. I make clear
that I do not hold myself out as having expertise in statistical analysis,
and do not suggest that my comments below are a comprehensive
response to this section of Dr Worden’s report.
5.328 At paragraph 812, Dr Worden sets out the following graph which details
the average monthly value of a claimant’s loss against the total number
of claimants who lost a smaller average monthly amount.
Prepared by: Jason Coyne fe)
Occupation: Partner .
Specialist Field: IT Systems ; It lI
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0207
181024SR1935
01 February 2019
POL00262929
POL00262929
Page 204 of 265
1200.00 +
1000.00 +
800.00 +
600.00 +
400.00 +
200.00 +
0.00
ame
ta
o
=
=
loss per month
a
Oo
=
236
0
0,
a
330
ns
na
ot
471
loss per month
Figure 5 Graph of claimant's loss against the total number of claimants who lost a smaller average
monthly amount (from Dr Worden's Expert Report)
5.329 At paragraph 815 Dr Worden states in relation to this graph:
“This graph on its own calls into question the idea that most of the
Claimants’ claimed losses were caused by bugs in Horizon - because one
would expect bugs in Horizon to have affected all Claimants equally, apart
from random fluctuations. This would have led to all Claimants suffering
approximately equal losses per month - not to a ‘low tail’ of Claimants
with very small losses per month, or a ‘high tail' of Claimants with very
high losses per month. Since the graph shows both a low tail and a high
tail, it contradicts the hypothesis of random Horizon bugs impacting all
Claimants. It is, however, consistent with the idea of losses being mainly
caused by human error - with a wide range in the rates of human error
in different branches.
5.330 I disagree with Dr Worden in relation to this conclusion in three ways:
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
Zitg roup
POL-BSFF-0100992_0208
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 205 of 265
a. “This graph on its own calls into question the idea that most of the
Claimants' claimed losses were caused by bugs in Horizon -
because one would expect bugs in Horizon to have affected all
Claimants equally, apart from random fluctuations..." - There is no
technical foundation for this assumption. An issue caused by a bug
could arise from any number of factors and any combination of
factors (technical or otherwise). It is therefore unlikely that bugs
would affect all users in the same way, since all postmasters would
not use the Horizon system in exactly the same way.
b. “..This would have led to all Claimants suffering approximately
equal losses per month...” - Even if it was assumed that bugs
affected all Subpostmasters equally, it is wrong to conclude that
this would mean that the value of losses per month would be the
same for all claimants. Bugs, errors and defects are, by definition,
issues which cause unexpected results in software. Additionally,
the impact of a bug, error or defect which affects (or arises as a
result of) a transaction will likely depend on the value of that
particular transaction at the time. Therefore, there is no technical
reason to assume that there is any correlation between the
likelihood of a bug’s occurrence and the value of its effect. When
Horizon fails due to a bug, error or defect it is typically the value
of the transaction being processed at the time which determines
the discrepancy. Furthermore, Dr Worden’s assumption is
inconsistent with what actually happened (see, for example,
paragraph 5.324).
c. “Since the graph shows both a low tail and a high tail, it contradicts
the hypothesis of random Horizon bugs impacting all Claimants. It
is, however, consistent with the idea of losses being mainly caused
by human error - with a wide range in the rates of human error in
different branches." —- As above, I don’t agree with Dr Worden’‘s
position that bugs would affect all claimants equally. Additionally,
it is not clear why Dr Worden has chosen these specific metrics for
Prepared by: Jason Coyne
fe)
Occupation: Partner : .
Specialist Field: IT Systems ) It 1 ¢
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0209
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 206 of 265
“equality”. For example, because the evidence shows that it is
possible for a bug to result in different values (e.g. paragraph
5.324), it would be more accurate to say that each claimant has
an equal chance of a triggered bug that results in a small shortfall
or large shortfall. If this were correct, then Dr Worden’s graph at
813 would actually be consistent with the idea that bugs were the
primary cause of issues since there is a consistent trendline from
the “low tail” to the “high tail”, which suggests that each claimant
has an equal chance that a bug will result in a large or small
shortfall, or somewhere in between. For the avoidance of doubt, I
do not consider this measure definition of “equality” to be a good
analytical method; it is a comment upon Dr Worden’s analysis
which, in my opinion, is flawed.
5.331 In paragraph 816, Dr Worden sets out the following graph which details
the average loss per month of each claimant against the length of their
tenure.
Average loss per month
£2,500
£2,109
£2,000
£1,500
£1,156
£1,000
£713
ean £455
£272 £268
[] £184 £163 gqog £131
£0 I I Zt ns =
0-22 23-45 46-68 69-91 92-114 115-137 138-160 161-183 184-206 207-229
Months of tenure
Figure 6 Graph detailing the average loss per month of each claimant against the length
of their tenure (from Dr Worden's Expert Report)
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems itg rou )
On the Instructions of: Freeths LLP
POL-BSFF-0100992_0210
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 207 of 265
5.832
5.333
5.334
In isolation, this graph does show that there is a correlation between
“average loss per month” and “months of tenure”. On this basis, Dr
Worden concludes:
"819. This chart is equally not consistent with a hypothesis that losses
arose from bugs in Horizon. On that hypothesis, the mean loss per month
would not vary with length of tenure, as it does in the chart.
820. One possible interpretation of the chart is that Claimants with
shorter tenures were less experienced, and so were more prone to make
human errors which caused losses.”
I agree that “less experienced users” is one possible interpretation of
the data within this chart. However, I would observe that another
“possible interpretation” is that there is a correlation between the size
of shortfalls which do not have a conclusively determined root cause
and the likelihood that a Subpostmaster would remain in post (i.e. a
Subpostmaster with a higher undetermined loss is more likely to leave
or be removed) irrespective of whether the shortfall was caused by a
bug, error or defect. For the avoidance of doubt, I am not suggesting
that this is the correct interpretation; I am pointing out that this data
does not necessarily imply that claimants with shorter tenures were
causing shortfalls due to inexperience.
In paragraph 821, Dr Worden sets out the following graph which he
states shows the number of claimants who were claiming losses per
year.
Prepared by: Jason Coyne fe)
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0211
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 208 of 265
Numbers of Claimants with losses
350
300
250
200
150
100
50
0
LSS a 8 s > es a >? s » SF SP PSK ae
Figure 7 Graph detailing the number of claimants with losses per year (from Dr Worden's
Expert Report)
5.335 If this data is correct, it appears that between 1999 and 2008, there is
an upward trend which shows that more claimants were reporting losses
over a period of 10 years. This is not consistent (from an IT systems
perspective) with Dr Worden’s conclusion that the vast majority of
losses were caused by human error. As a general principle in IT systems
implementations, you would expect to see more human errors when the
system is first implemented (because it will be new to users), but for
these to decrease over time as users become more accustomed to using
the system.
5.336 It is unprecedented in my experience for user errors to increase over a
period of 10 years.
5.337 It is also true that you would expect issues caused by bugs to decrease
over time. However, this will not necessarily be the case if a system is
subject to large amounts of change or if bugs, errors and defects are
not dealt with effectively (if, for example, they remain undiscovered
because the cause of an issue is incorrectly determined to be the result
of a user error or if providing a fix in one part of the system creates an
issue elsewhere). Where this is the case, it would not be surprising to
Prepared by: Jason Coyne
Occupation: Partner .
Specialist Field: IT Systems itg rou )
On the Instructions of: Freeths LLP
POL-BSFF-0100992_0212
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 209 of 265
find that more users are affected over time because they will gradually
follow the series of steps necessary to trigger the bug.
5.338 Therefore, in my opinion, the numbers in this graph are inconsistent
with Dr Worden’s conclusion that claimed losses were much more likely
to be caused by human errors than by bugs.
5.339 Further in paragraph 821, Dr Worden sets out the following graph which
shows overall losses per year.
Overall losses
£1,000,000
£900,000
£800,000
£700,000
£600,000
£500,000
£400,000
£300,000
£200,000
£100,000 I I I
£0
2 > oF % S
Figure 8 Graph showing overall losses per year (from Dr Worden's Expert Report)
5.340 Dr Worden states:
“I do not know the causes of variation in particular years, but it is clear
that shortfalls were claimed to have been experienced from both Horizon
and Horizon Online, in all years of their operation. Much of the variation
may just arise from random fluctuations.”
5.341 I agree that the variations could theoretically have arisen as a result of
“random fluctuations”, but this could also be explained by the fact that
bugs, errors and defects would not affect every claimant in the same
way.
Prepared by: Jason Coyne
Occupation: Partner .
Specialist Field: IT Systems itg rou )
On the Instructions of: Freeths LLP
POL-BSFF-0100992_0213
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 210 of 265
5.342 Dr Worden further states:
“the broadly flat nature of this graph, with random-looking fluctuations
for year to year, qualitatively contradicts the notion, as was put forward
by Mr Coyne, that Horizon sometimes had ‘bad periods’ in which
robustness countermeasures did not work well, and Claimants suffered
large losses as a consequence. In my opinion, any such ‘bad period’ would
extend over two or three years, while Fujitsu grappled with widespread
problems. The graph does not show this pattern.”
5.343 I have three observations in relation to this position:
a.
“the broadly flat nature of this graph, with random-looking
fluctuations" - this graph is not flat, so I do not understand Dr
Worden’s reason for stating otherwise.
“[this graph] contradicts the notion, as was put forward by Mr
Coyne, that Horizon sometimes had ‘bad periods' in which
robustness countermeasures did not work well, and Claimants
suffered large losses as a consequence." - As above, this graph
contains obvious fluctuations, from as low as ~£150k in 2002 and
2017 to as high as ~875k in 2010.
“In my opinion, any such ‘bad period' would extend over two or
three years, while Fujitsu grappled with widespread problems. The
graph does not show this pattern” - there is no technical reason
why a “bad period” would need to last 2-3 years. This is not
consistent with my experience and I can think of no plausible
explanation as to why Dr Worden would take this position as a
general principle.
5.344 Dr Worden further states:
825. There was an obvious spike in Claimants’ reported losses in 2010,
which one might interpret as arising from the introduction of Horizon
Online, and teething problems in the new system. In Angela Van Den
Bogerd's Witness Statement at paragraph 183, she said that there was a
mandatory cash check in all branches before the change to Horizon
Online, which may have caused a temporary spike in declared losses. If
Prepared by: Jason Coyne fe)
Occupation: Partner ae
Specialist Field: IT Systems It Jrou f )
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0214
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 211 of 265
this is correct, it might account for the spike in 2010. Since many
Claimants showed a pattern of not reporting losses for extended periods,
followed by large ‘lumps’ of loss, this second account appears more
likely.”
5.345 In my experience, a major change to a platform will almost always lead
to an increase in bugs, errors and other issues. Therefore, I do not agree
that the mandatory cash check was “more likely” to be the cause of the
spike in 2010. The most likely scenario is that both of these were
factors.
Dr Worden’s “Quantitative Analysis”
5.346 In Section 8.10.4, Dr Worden sets out his conclusions based on his
analysis as set out in Appendix E. He states at paragraph 827:
5.347 I do not comment on the statistical calculations which Dr Worden has
carried out, for reasons I have already explained. However as I have
explained previously, I comment below on facts or assumptions where
I believe this may assist the Court.
“If all the Claimants’ claimed shortfalls arose from bugs in Horizon, or
even if large part of them did, one would not expect to see a ‘low tail’ of
many Claimants with small monthly shortfalls (as in the chart above),
much less than the average shortfall of £359 per month, as claimed by
all the Claimants.”
5.348 From a technical perspective, there is no basis for the assumption that
the likelihood of a bug’s occurrence correlates with the value of a
shortfall. By definition, a bug, error or defect is an issue which causes
an invalid or unexpected result, so it is wrong to conclude that bugs will,
on average, result in larger shortfalls. As set out in the example at
paragraph 5.324 above, it is entirely possible for the same bug have a
small impact on one branch and a large impact on another.
5.349 Within Appendix E, Dr Worden has made a lot of assumptions for which
I believe there is no technical or factual basis, for example:
Prepared by: Jason Coyne
fe)
Occupation: Partner : .
Specialist Field: IT Systems ) It 1 ¢
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0215
181024SR1935
POL00262929
POL00262929
01 February 2019 Page 212 of 265
At paragraph 379, Dr Worden assumes that bugs occur at random,
and there is nothing about the behaviour or circumstances of any
claimant which makes them more or less likely than any other
claimant to suffer in any month from a Horizon bug which affects
their accounts. I have set out my reasons for disagreeing with this
assumption in the preceding sections.
At paragraph 380, Dr Worden assumes that, if it were claimed that
some factor led to a higher incidence of bugs, then it would be
necessary to show that claimants with high monthly losses were
subject to that factor, and claimants with low monthly losses were
not. However, as I have set out in previous sections, there is no
technical basis for assuming that the likelihood of a bug’s
occurrence is proportional to the value of its effect. Bugs typically
arise in live systems as a result of a specific set of circumstances
which was not foreseen during testing. There is no reason that the
effect of a bug needs to be large for it to be considered a bug, error
or defect.
At paragraph 381, Dr Worden states that human errors would not
affect all branches equally on average. I agree that this is likely to
be the case.
At paragraph 383, Dr Worden assumes that bugs affect all
branches equally in terms of “amount”, except for small statistical
fluctuations. As I have set out in previous sections, there is no
technical reason to assume that bugs would have the same effect
in all cases. In relation to Horizon, it is a matter of fact that this
was not the case, as set out in paragraph 5.324 above.
At paragraphs 384 and 389, Dr Worden assumes that claimants
with the smallest monthly average loss are the ones with the
lowest level of human error (paras 384 & 389) and that, on this
basis, these claimants give the “best” measure of the level of
shortfall in their accounts per month from Horizon bugs. As above,
there is no technical reason to make this assumption. A bug, error
Prepared by: Jason Coyne
fe)
Occupation: Partner : .
Specialist Field: IT Systems : It FC
[e)
On the Instructions of: Freeths LLP
POL-BSFF-0100992_0216
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 213 of 265
or defect could theoretically account for 100% of a claimant’s
claimed monthly loss (if, for example, they properly dealt with any
human errors during their tenure).
f. At paragraphs 385 & 389, Dr Worden assumes that a sample of
claimants with low monthly average losses can be scaled up to
accurately represent the proportion of total losses caused by bugs
across all claimants. As above, there is no reason to assume that
bugs (or even any given bug) will affect all claimants equally.
g. At paragraph 385 Dr Worden assumed that losses from horizon
bugs were never, or very rarely, cancelled out by gains from
human error. No basis is given for this assumption.
h. At paragraph 387.5, Dr Worden assumes that bugs have an equal
average effect on any given “claimant month”. There is no
technical foundation for this assumption, especially given that
Horizon was continuously updated over the course of many years.
i. At paragraph 393 Dr Worden assumes that, by taking those
claimants with the lowest monthly average loss, Dr Worden has
selected those claimants who were “/uckiest in not suffering in
bugs from Horizon".
j. At paragraph 393, Dr Worden assumes that “good months”
compensate for “bad months”, so the amount of fluctuation
between claimants is small. There is no technical foundation for
this assumption as bugs could vary wildly in their effect.
Additionally, this contradicts Dr Worden’s graph (Figure 8.4) in his
main report, which shows major fluctuations in overall losses from
year to year.
k. I At paragraphs 397 & 398 Dr Worden assumes that there is a “lucky
claimant effect” which means that the fluctuations arising from the
random nature of bugs cannot be more than a factor of 2. This is
based on Dr Worden’s assumption that bugs, errors and defects
impact all claimants in the same way which, as I have set out
Prepared by: Jason Coyne
fe)
Occupation: Partner : .
Specialist Field: IT Systems ) It 1 ¢
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0217
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 214 of 265
above, is a position with no technical foundation and is factually
incorrect in relation to Horizon.
In paragraph 401, Dr Worden assumes that the uncertainty caused
by factors such as variation in the size of branches is small. Dr
Worden gives no basis for this assumption.
5.350 In my opinion, it is very unlikely that an analysis which uses these
assumptions as a basis will result in an accurate conclusion in relation
to the percentage of losses that were likely to have been cause by bugs
as opposed to human error.
Section 9: Reconciliation & Transaction Corrections
Overview
5.351 In my first report (at paragraph 6.38, page 105) I set out that Post
Office had explained, in a response to my Request for Information, that
10,000+ transactions per week suffer from problems and are not
automatically reconciled. I also explained that it was Post Office’s view
that these Reconciliation Errors were due to system faults (page 96 para
6.2) and that such system faults are corrected on a “cost benefit basis”.
When such reconciliation errors occur, Post Office utilise a largely
manual process to resolve them.
5.352 Dr Worden and I both refer to one of the same documents in our
respective reports.?9!
5.353 Dr Worden, at paragraphs 920 to 926, sets out a cost benefit process
that Post Office might consider. His explanation involves considering
administrative costs and balancing these with the reconciliation
discrepancies. Whilst that might be one cost-benefit consideration for
Post Office, I was instead making reference (paragraph 6.3, page 96)
to Post Office’s fix of Horizon system faults on a cost-benefit basis, then
Post Office will need to consider its spend with Fujitsu, which I assume
may be larger than administration costs. I have not had sight of any of
291 SVMSDMPROO012 - Reconciliation and Incident Management Joint Working Document.doc,
Reconciliation and incident Management Joint Working Document, 18 March 2013 {POL-0219191}
Prepared by: Jason Coyne
fe)
Occupation: Partner : .
Specialist Field: IT Systems ) It 1 ¢
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0218
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 215 of 265
5.354
5.355
5.356
5.357
5.358
the contractual arrangements between Post Office and Fujitsu so I am
unable to provide opinion as to what the costs of Fujitsu fixing faults
might be. In my experience, fault determination and resolution costs
can be many times that of administration and therefore the cost benefit
analysis exercise could be different to that set out by Dr Worden.
Upon review of further material disclosed in relation to the responsive
witness evidence I wish to make the following points:
It should be noted that Mr Paul Smith sets out in his 16 November 2018
witness statement??? the percentage of transaction corrections that
were successfully disputed. By successfully disputed I take this to mean
that Post Office initially believed that the Subpostmaster was liable for
the discrepancy but, when the Subpostmaster contested, Post Office
investigated further and found this was not the case and therefore
corrected the position.
I had originally considered that a Transaction Correction was only issued
by Post Office after it had validated its liability assessment with all
technical mechanisms and had examined data available in the Horizon
audit logs. Only following these checks should Post Office believe that
the Subpostmaster must have made a mistake.
However, on the contrary, 77% of 2,890 Transaction Correction
disputes were upheld?°? in 2016/2017 in relation to Santander Manual
Deposits.
Following this, it is difficult to conclude anything other than Post Office,
after initially claiming that the Subpostmaster was liable for the loss,
concluded that it had attributed liability incorrectly and that the loss was
due to another undeclared reason (Post Office client mistaken, Horizon
system fault or Post Office process failure, or others) - only after the
292 {Witness Statement of Paul Ian Michael Smith, 16 November 2018}
293 10% of all Santander Transaction Corrections successfully disputed. As calculated by Dr
Worden (1) at Para 993, taken from Mr Smith's witness statement
Prepared by: Jason Coyne
fe)
Occupation: Partner : .
Specialist Field: IT Systems : It FC
[e)
On the Instructions of: Freeths LLP
POL-BSFF-0100992_0219
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 216 of 265
discrepancy had been contested by a Subpostmaster prompting Post
Office to investigate further.
5.359 Therefore, in summary, the above is evidence of Post Office assuming
the Subpostmaster is liable and issuing a Transaction Correction before
completing an examination of all data available in Horizon, including the
Horizon Audit Logs.
5.360 Compounding this theme, data provided in Paragraph 31 of Mr Torstein
Godeseth’s 27 September 2018 witness statement?% suggests the
position that I set in 4.66 above, that only a fraction?®> of Transaction
Corrections are validated using audit data.
5.361 Mr Paul Smith explains in his 16 November 20187% witness statement
that:
“Post Office introduced a case management system that record each
individual challenge to the TC in September 2018” and that; “individual
challenges to TCs were not recording prior to this and therefore it is not
possible to state what proportion of TCs have been challenged
historically”.
5.362 The Transaction Correction dispute investigation process is set out in
more detail in Ms Philips’ witness statement of 28 September 2018.79”
She explains that the process of documenting information about the
dispute by telephone, email or letter has only been in place since 2018
with the introduction of a “Branch Dispute Form”. She explains,
however, that the process had been in place since November 2016 but
was undocumented. It is not stated what process was in place prior to
November 2016.
294 {Witness Statement of Torstein Olav Godeseth, 27 September 2018}
295 Less than 0.67% of the total Transaction Corrections could have been investigated with Full
Audit if less than 720 ARQs were requested by POL
296 {Witness Statement of Paul Ian Michael Smith, 16 November 2018}
297 {Witness Statement of Dawn Louise Phillips, 28 September 2018}
Prepared by: Jason Coyne fe)
Occupation: Partner ae
Specialist Field: IT Systems It Jrou f )
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0220
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 217 of 265
5.363 Therefore, it is unknown if the appropriate information required to
conduct a review of a disputed Transaction Correction was gathered
prior to November 2016.
5.364 Similarly, it cannot be clear if the percentages of Transaction
Corrections successfully disputed in recent years is the same for the
earlier years of the Horizon lifetime.
Issue 5 - How, if at all, does Horizon system itself compare transaction data
recorded by Horizon against transaction data from outside sources.
5.365 Reconciliation, the process by which the Horizon system itself compares
transaction data recorded by Horizon against transaction data from
sources outside of Horizon is dealt with in my first report at Section 6
(page 95).
5.366 In summary, reconciliation is a large and complex facility. It involves
many different streams of electronic processing from both Fujitsu data
centre computing components, multiple “external clients”, Post Office
and Fujitsu business process departments and manual investigatory
procedures (where corrective fixes are applied, if necessary). If the
reconciliation process identified a difference between the sources being
compared, then manual steps are taken to establish and correct the
errors and potentially issue Transaction Corrections, or provide
payments to external clients (where a negative discrepancy might
occur).
5.367 Dr Worden and I agree on the basics of reconciliation with him stating:
For most of Post Office's clients (for whom Post Office branches carry out
agency business) there is a regular automated process of comparing
(reconciling) the transactions as recorded by Post Office, with the
transactions as recorded by the client organisation.
These comparisons might or might not be carried out within Horizon
‘itself’; but in any event, because of the large volume of transactions, the
comparison had to be automated.
Prepared by: Jason Coyne fe)
Occupation: Partner ae
Specialist Field: IT Systems It }roOu f )
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0221
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 218 of 265
5.368
Whenever the comparison revealed any discrepancy, there appeared to
be a human process of deciding where to allocate responsibility for the
discrepancy.
This had to be a human process and was therefore subject to errors.
If responsibility was allocated to a branch, it results in a TC, which the
branch might accept or query before it entered the branch accounts.
There was also reconciliation of cash remmed from branches to Post
Office cash management, or in the reverse direction
I find nothing contentious with what Dr Worden has stated, which
accords with my understanding.
Issue 15 — How did Horizon process and/or record Transaction Corrections
5.369
5.370
5.371
5.372
Dr Worden accepts that the Transaction Correction process could lead
to Transaction Corrections being issued in error and that, when
disputed, some Transaction Corrections are retracted.
Dr Worden explains that in his view, Double Entry Accounting and
Manual inspection of Data would provide some level of control of the
Transaction Correction process, but as I have set out above at 4.94 and
5.357 in reference to the witness statement of Mr Paul Smith; if 77% of
the Santander Transaction Correction disputes are upheld it does not
appear that appropriate control is exercised by Post Office, or that such
controls do not work.
Dr Worden explains at 924 and 925 that the administration costs of
dealing with disputed Transaction Corrections would often exceed the
amount of the Transaction Correction involved. I have not had sight of
any Post Office administration costs for dealing with disputed
Transaction Corrections and therefore cannot agree.
Dr Worden also sets out a number of different ways Post Office may
choose to motivate Subpostmasters, this may not be as simple as Dr
Worden suggests as the Post Office “outsources” a number of the central
support costs, helpdesks as well as Horizon investigations to either
Prepared by: Jason Coyne
fe)
Occupation: Partner : .
Specialist Field: IT Systems : It FC
[e)
On the Instructions of: Freeths LLP
POL-BSFF-0100992_0222
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 219 of 265
5.373
5.374
5.375
5.376
ATOS or Fujitsu and therefore the costs and motivations will likely be
more complex.
Dr Worden explains for his calculations in paragraph 931 that: “...One
may assume that any erroneous TC is likely to be disputed”. I do not
agree with this assumption and this is at odds with that which Dr
Worden expressed early in this same section (at paragraph 923) where
he explains that Subpostmasters will take decisions “on a cost-benefit
basis designed to make best use of his own time”. There are many other
factual considerations which I think would need to be taken into account
before deciding how likely it is that an erroneous TC would be disputed
e.g. the evidence provided and how easy or difficult the dispute process
is.
From paragraphs 935 in Dr Worden’s report he calculates a value for
the likely impact on Branch Accounts of incorrect Transaction
Corrections and a number of assumptions are made which I believe are
unsafe to make.
For example, at paragraph 936 whilst Santander may not account for a
large proportion of the Transaction Corrections they may be relatively
high value Transaction Corrections. Camelot does indeed account for a
large number of the Transaction Corrections, but I could envisage that
Camelot transactions may be relatively small (National Lottery tickets
costing £1), when compared with Santander transactions.
Additionally, at paragraph 943 Dr Worden explains that the Claimants
branches are on average three times smaller than the national average,
based on number of transactions per day. It is my opinion however that
the likely impact of incorrect Transaction Corrections on branch
accounts would also be weighted by both the types of transactions and
values of the transactions being processed when exposed to the
bugs/errors and defects within Horizon and therefore must be taken into
consideration.
Prepared by: Jason Coyne fe)
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0223
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 220 of 265
Conclusions
5.377 In my first report (at paragraph 6.38, page 105) I set out that Post
5.378
Office had explained in a response to my Request for Information that
10,000+ transactions per week suffer from problems and are not
automatically reconciled. I also explained that it was Post Office’s view
that these reconciliation errors were due to system faults and that such
system faults are corrected on a “cost benefit basis”. Since my first
report, I have found in addition that it is also possible a number of these
reconciliation errors might be caused by incorrect reconciliation data
from external clients. It is also my opinion that Post Office are issuing
transaction corrections to the Subpostmaster to attempt to modify
branch accounts to correct these reconciliation errors before all of the
possible checks are complete.
When such reconciliation errors occur, Post Office utilise a largely
manual process to attempt to resolve them. Such manual checks would
typically not include an Audit Request Query for Fujitsu to look at the
audit logs and is subject to human error.
Section 10: Facilities available to Subpostmasters
Overview
5.379
5.380
My opinion in relation to this section is set out at paragraph 7.40 (page
125) of my first report and has not changed upon review of any further
material provided in additional disclosure.
I have noted that at paragraph 954, Dr Worden has listed a number of
assumptions he believes were made in my first report and then
concludes that these rest on an unrealistic picture of how commercial
IT systems are built, used and supported. Dr Worden does not set out
where in my report these “assumptions” are made but, for the
avoidance of doubt, they do not accurately represent my opinions. I
have clarified my opinions in the table below:
Prepared by: Jason Coyne fe)
Occupation: Partner : -
Specialist Field: IT Systems g It ] }{
On the Instructions of: Freeths LLP < -
POL-BSFF-0100992_0224
181024SR1935
01 February 2019
POL00262929
POL00262929
Page 221 of 265
Dr Worden’s
Interpretation of My
Opinion
Dr Worden’s Response
My Response
954.1. It would have
been a good thing to
provide Subpostmasters
with more information
about the workings of
Horizon than was given
to them.
955.1. It is not a good
thing to give the users
information about parts
of an IT system which
they do not encounter in
their daily work, and
which they know very
little about. They will be
perplexed by it.
This is not an accurate
representation of my opinion.
At 8.11 in my first report, I state:
“Subpostmasters had access to a
much smaller pool of information.
This is in line with what I would
expect to see given that
Subpostmasters are the users of the
Horizon system, and therefore would
not typically be given access to
anything beyond what was necessary
for them to carry out their ‘business
as usual’ activities.”
This is restated in my conclusion at
8.20.
There is no point in my report where
I suggest that it would be a “good
thing” to supply Subpostmasters with
information about the inner workings
of Horizon, and I have specifically set
out that this is not the case (as
above).
My conclusion is that, as a matter of
fact, Subpostmasters did not have
access to the information that would
have been required to identify the
cause of a discrepancy if that
discrepancy was caused by a system
issue. This is part of my answer to
Issue 9.
954.2. If there was a
fault in Horizon, there
should have been some
955.4. When the
developers of an IT
system discover some
This is not an accurate
representation of my opinion.
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
Zitg roup
POL-BSFF-0100992_0225
181024SR1935
01 February 2019
POL00262929
POL00262929
Page 222 of 265
useful automatic way for
Horizon to tell
Subpostmasters what it
was.
bug or defect in it, the
best thing to do is to fix
it, rather than to create
some new error message
to the users.
955.5. When an IT
system gives results,
which puzzle its users
(for any cause), further
automated messages
from the system are only
of limited help to users.
They need support from
a human being, who may
need to take account of
the circumstances and
bring to bear a wide
variety of knowledge.
There is no point in my report where
I suggest that there should have
been “some useful automatic way”
for Horizon to alert Subpostmasters
about Horizon faults, so it isn’t clear
to me where Dr Worden has taken
this “assumption” from.
At 7.15 in my first report, I stated:
“As per the Joint Experts Statement,
the extent to which any IT system
can automatically alert its users to
bugs within the system itself is
necessarily limited.”
This is reiterated at paragraph 3.4.
As above, my report addresses the
question about the extent to which
Horizon itself alerted Subpostmasters
of bugs, errors and defects (as I was
instructed to do in Issue 2).
954.3. In the case of an
anomaly, it was
incumbent on the
Subpostmaster to
dispute the cause of the
anomaly with Post Office.
955.6. Anomalous results
may arise for a wide
variety of reasons - from
human error, to errors in
processing at the back-
end. Understanding the
causes depends
inevitably on cooperation
between the user (who
knows what he did) and
support staff (who know
much more about back-
end systems). To portray
this cooperation as a
dispute is fundamentally
misleading.
This is not an accurate
representation of my opinion.
There is nothing in my report which
suggests anything like this so it is
not clear to me how Dr Worden has.
come to this conclusion.
At paragraphs 6.61-6.63 in my first
report I have set out the process for
disputing a Transaction Correction,
but I have not suggested (and would
not suggest) that it was “incumbent”
on a Subpostmaster to raise dispute
of the cause of an anomaly in all
instances.
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
Zitg roup
POL-BSFF-0100992_0226
181024SR1935
01 February 2019
POL00262929
POL00262929
Page 223 of 265
954.4. In doing so,
Subpostmasters could
usefully use information
about the back-end
systems of Horizon to
infer that some anomaly
was caused by a bug in
Horizon.
955.2. To anticipate the
small proportion of cases
where the IT system is in
error, there is no point in
trying to educate all the
users in details and
terminology of the
system which will never
concern them.
955.3. An IT system can
give its users useful
warnings and error
messages in a variety of
situations, but generally
not in the case of
previously undiscovered
bugs in the system.
This is not an accurate
representation of my opinion.
See my response to 954.1.
954.5. Because
Subpostmasters did not
have all this information,
but Post Office did, there
was an asymmetry of
information between
Subpostmasters and Post
Office - which Post Office
used to unfairly attribute
the effects of bugs in
Horizon to human error
by the Subpostmasters.
955.7. Staff and
organisations who.
support an IT system
have a strong incentive
to understand bugs and
to get them fixed, to
reduce their future
workload. They have no
interest in leaving bugs
unfixed, so the same
problems keep recurring.
This is not an accurate
representation of my opinion.
As a matter of fact, there was an
asymmetry of information. Dr
Worden appears to agree with this
given that, as above, he has stated:
“It is not a good thing to give the
users information about parts of an
IT system which they do not
encounter in their daily work”
Ihave never suggested that Post
Office used the asymmetry of
information to “unfairly attribute
bugs in Horizon to human error by
the Subpostmasters”
T have stated that Post Office had
access to the information required to
identify the existence and causes of
bugs in Horizon and a Subpostmaster
did not. This is matter of fact and it
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
Zitg roup
POL-BSFF-0100992_0227
181024SR1935
01 February 2019
POL00262929
POL00262929
Page 224 of 265
is what I would expect the
relationship to be between an IT
systems supplier and a user.
Therefore, Subpostmasters were
reliant on Post Office to identify
those issues that were caused by
bugs. This is not a controversial
position. It is a matter of fact based
on the information that was available
to each party.
Furthermore, I did not suggest that
staff and organisations have an
interest in leaving bugs unfixed in
my previous report. However, I
agree that it is typically the case that
organisation will look to resolve
defects as soon as possible.
However, Post Office outsourced the
fixing of bugs to Fujitsu and the
management of reference data to
ATOS, which could mean there was a
cost associated with certain activities
related to bug-fixing. This could have
led to the postponement of fixes in
certain instances (e.g. if bugs would
be fixed in an upcoming
patch/release, or if a manual
workaround was preferred).
5.381 Dr Worden reiterates these “assumptions” in paragraphs 961-979. My
opinions remain as they are set out in the table above, and I have noted
the following additional points.
5.382 In paragraph 968, Dr Worden states:
“Issue 2 appears to be asking - could Post Office have given its
Subpostmasters automated support in Horizon, in the place of human
support?"
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
itg roup
POL-BSFF-0100992_0228
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 225 of 265
5.383 This is not my understanding of Issue 2. For the avoidance of doubt, I
have not interpreted the question in any similar way to Dr Worden. I
have taken it literally and investigated whether the Horizon IT system
itself alerted Subpostmasters of bugs, errors or defects as described in
Issue 1.
5.384 At 969 Dr Worden has stated:
“Similarly, there seems to be an assumption behind Issues 9 and 14 that,
given enough automated information, Subpostmasters could somehow
identify the causes of shortfalls (deep inside Horizon), and might have
the knowledge and persistence to ‘dispute’ them with Fujitsu support
staff, whose job it is to look at such issues, and who would have a deep
knowledge of Horizon internals.”
5.385 Again, I have not made any similar assumptions or interpretations. I
have taken my instructions literally and answered the questions with a
view that they do not need any changes based on my own
interpretations.
5.386 At paragraph 973, Dr Worden states:
“A final assumption to be addressed here is that the support function
would always start by assuming that any problem had arisen from an
error in the branch and would not give sufficient credence to the
possibility that it might have arisen from a software error.”
5.387 I have not made this assumption when answering issues 2, 9 and 14.
Again, I have taken each question literally and answered it on that basis.
I have not attempted to add any of my own interpretation to the
meaning of the issues.
Issue 2 —- Did the Horizon IT System itself alert Subpostmasters of such bugs.
errors or defects.
5.388 Dr Worden’s overarching conclusion in relation to Issue 2 is that
“Horizon did not, in general, alert Subpostmasters to any significant
bugs or other defects in the system itself.” I agree with this, as well as
with the extract from the Joint Statement which states:
Prepared by: Jason Coyne fe)
Occupation: Partner ae
Specialist Field: IT Systems It Jrou f )
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0229
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 226 of 265
5.389
5.390
5.391
“The extent to which any IT system can automatically alert its users to
bugs within the system itself is necessarily limited. While Horizon has
automated checks, which would detect certain bugs, there are types of
bugs which would not be detected by such checks'.”
I have also noted that Dr Worden suggests at 89.7 & 955.7 that
supporters of an IT system have a "strong incentive to understand bugs
and to get them fixed” and then further at 974:
“In my experience any competent IT support operation is grateful to its
users, when they draw its attention to any problem which can be fixed,
to reduce the future costs of support”
I agree with this as a general principle. However, Post Office outsourced
the fixing of bugs to Fujitsu and the management of reference data to
ATOS, who were likely to be operating under a Service Level Agreement
(“SLA”) which could result in a charge for Post Office whenever Fujitsu
and ATOS needed to carry out certain activities. This could have led to
fixes being postponed in certain instances (e.g. if bugs would be fixed
in an upcoming patch/release, or if a manual workaround was
preferred).
In the remainder of this section, Dr Worden reiterates his opinion that
he would not expect Subpostmasters to have detailed knowledge of the
system. I agree with this position.
Issue 9 - Subpostmaster Ability to Identify Existence & Cause of Discrepancies
5.392
5.393
The majority of this section reviews the information that was available
to Subpostmasters which is not controversial.
Dr Worden’s overarching view reiterates the position in the Joint
Statements:
“The causes of some types of apparent or alleged discrepancies and
shortfalls may be identified from reports or transaction data available to
Subpostmasters. Other causes of apparent or alleged discrepancies and
shortfalls may be more difficult or impossible to identify from reports or
transaction data available to Subpostmasters, because of their limited
Prepared by: Jason Coyne
fe)
Occupation: Partner : .
Specialist Field: IT Systems : It FC
[e)
On the Instructions of: Freeths LLP
POL-BSFF-0100992_0230
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 227 of 265
5.394
5.395
5.396
5.397
5.398
5.399
5.400
knowledge of the complex back-end systems. Identification requires
cooperation of Post Office staff and Subpostmasters.”
I agree with this position.
In addition, I have made several other observations in relation to Dr
Worden’s conclusions.
At paragraph 958, Dr Worden concludes:
“In my opinion, from comparing human errors with software error rates
in Horizon, most discrepancies are caused by human error. The functions
available from Horizon, when used in accordance with Post Office
guidance and procedures, enable Subpostmasters to identify the causes
of such discrepancies...”
I do not agree that this conclusion is based on a solid technical
foundation, as I consider Dr Worden’s analysis in relation to software
error rates to be flawed (see my response to Dr Worden’s Section 8
above).
Furthermore, Dr Worden’s analysis does not appear to account for
issues caused by 3% parties, which may well include human errors, that
Subpostmasters would not be able to identify. Additionally, this
conclusion does not consider issues such as the one highlighted in 5.23
(Page 47) of my first report which states:
“There is also evidence of cash declaration discrepancies arising from
clerks duplicating remittance in transactions (“Rem-in”) because of wrong
messages being presented on the Horizon counter screen (acha621P).
This would result in incorrect cash amounts being declared.”
Where this is the case, even if a Subpostmaster followed the correct
procedures, they would not (or at least not necessarily) be able to
identify the cause of that discrepancy because the system would not be
showing the correct information from which they could carry out that
process.
I have noted that Dr Worden’s conclusion is based on the calculations
set out in his Appendix F. As I have set out at 5.349 above Dr Worden’s
Prepared by: Jason Coyne
fe)
Occupation: Partner : .
Specialist Field: IT Systems : It FC
[e)
On the Instructions of: Freeths LLP
POL-BSFF-0100992_0231
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 228 of 265
calculations in Appendices E and F are based on an assumption that
bugs affect all users equally (both in terms of frequency and impact).
In my opinion there is no technical foundation for this assumption and,
in this case, it is factually wrong.
5.401 Dr Worden further concludes at paragraph 92 (in relation to Issue 9):
“In my opinion, most discrepancies are caused by human error. The
functions available from Horizon, when used in accordance with Post
Office guidance and procedures, enable Subpostmasters to identify the
causes of such discrepancies. Subpostmasters and their staff are the best
placed to investigate such discrepancies, because they are the only
people who have first-hand knowledge of what happens in their branches
Post Office and Fujitsu support teams can only use their knowledge of
systems and the data stored within them; whereas the Subpostmaster
can use their knowledge of what happens in branch.”
5.402 In my opinion, this is not a complete picture. Although the support
teams may not have been physically present when a discrepancy
occurred, in practical terms they still have access to the same
information as a Subpostmaster because that Subpostmaster could
share the information with Post Office and additionally, Fujitsu has
access to the full audit logs. With this shared knowledge, Post Office
should then be in the best position to identify the causes of
discrepancies (whether caused by software bugs or human error), and
to advise on how to use the system to rectify the situation.
5.403 Additionally, it is noteworthy that Subpostmasters were not the only
staff in branch, so it is also possible that they would not have been
physically present when a discrepancy occurred.
Issue 14 - Horizon Functionality
5.404 Dr Worden’s overarching opinion in relation to this issue is that it is a
matter of fact because it addresses how Horizon dealt with certain
issues, which Dr Worden has set out the specific subsections. My
observations to each of these are:
Prepared by: Jason Coyne fe)
Occupation: Partner .
Specialist Field: IT Systems ; It lI
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0232
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 229 of 265
a. 10.5.1 in Dr Worden’s Report (Comparing Stock and Cash) - Dr
Worden’s account of this is high-level but it is not controversial.
b. 10.5.2 in Dr Worden’s Report (Resolve Discrepancy) - Again, Dr
Worden’s review in this subsection is high-level but, for the most
part, it is not controversial. I have noted his position is that the
process for disputing a discrepancy is said to be outside of scope,
but in my view this process does have to be considered as part of
an overall analysis of the facts. I have set out the process for
disputing a Transaction Correction at paragraphs 6.61-6.63 in my
first report.
c. 10.5.3 in Dr Worden’s Report (Recording Disputes) - As above. In
addition, I have noted that Dr Worden’s statement that a
discrepancy is not recorded as a debt or credit in Horizon
contradicts the agreement document produced in the Common
Issues Trial Flowchart 1 - Transaction Corrections which states
that, following the issue of a Transaction Correction, opting to
‘Settle Centrally’ results in:
“A corresponding debit or credit is made in the SPM’s customer account
with Post Office. If a debit, this will be treated as a debt by Post Office
unless the SPM contacts NBSC to lodge a dispute, which should suspend
collection until the dispute is resolved.”
d. 10.5.4 in Dr Worden’s Report (Accounting Statements) - Dr
Worden’s review in relation to this point is not controversial.
e. 10.5.5 in Dr Worden’s Report (Continuing to Trade) - I agree with
Dr Worden’s position in paragraph 1041 or his report (i.e. I have
not seen any specific evidence that the Horizon system prevented
Subpostmasters from trading until they produced a Branch Trading
Statement). For clarity, the statement at 7.39 in my previous
report relates to restrictions imposed by the business process
rather than a technical constraint.
Prepared by: Jason Coyne fe)
Occupation: Partner .
Specialist Field: IT Systems ; It lI
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0233
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 230 of 265
5.405 In Section 10.6, Dr Worden comments on my report. There is nothing
substantially different between his comments in this section and the
above section, so my opinions are as set out above.
Section 11: Facilities available to Post Office & Fujitsu
Overview
5.406 In this section Dr Worden has grouped the Horizon Issues differently to
the groupings I adopted in my first report. For the purposes of
readability, I will respond as per Dr Worden’s groupings with regards to
dealing with Issue 8 in this section (I did not group Issue 8 with the
remote access issues in my first report). However, Issues 7, 10, 11, 12
and 13 all relate to remote access elements and permissions and are
interlinked and I shall therefore group those in my responsive analysis.
5.407 I feel it is important to note that in consideration of my opinion in this
section:
a. Throughout my review of PEAK records within this dispute, I have
noticed that the procedure for Fujitsu to perform modifications to
branch data was often subject to an “OCP” request, sent to Post
Office for approval. I have requested, several times (RFI Appendix
A), the OCPs in relation to financial accounting corrective fixes
applied within Horizon. This was provided 24 January 2019 but I
have not had time to consider this.
b. In relation to the Transaction Correction tool Referred to within
Issue 10 of this report. I have requested the audit file of its usage,
in order to support or disprove my opinion that this tool has been
used more than once. Note that even if has indeed only been used
once, Balancing Transactions could still be conducted by Fujitsu
SSC (in Legacy Horizon) and through Privileged User access in
(Horizon Online).
5.408 I feel it is also important to note that in addition to the conclusions in
my first report (paragraph 7.40, page 125) in respect of Issues 7,10,
11, 12 and 13, additional material disclosed, and review of the
Prepared by: Jason Coyne fe)
Occupation: Partner .
Specialist Field: IT Systems g It ]
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0234
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 231 of 265
Responsive Witness Statements have furthered my understanding in
respect of the following points:
a. Mr Godeseth (and subsequently Dr Worden) state that only one
Balancing Transaction has been performed (using the Transaction
Correction tool) by Fujitsu. However it is evident that more than
one Balancing Transaction has been conducted by Fujitsu. More
detail in relation to this is provided under Issue 11 in this report.
b. The PEAK and Responsive Witness Evidence has enabled me to
conclude that there are gaps in the evidence of how those bugs
acknowledged by Post Office were handled, and I cannot say with
confidence that I believe they were investigated appropriately, or
as efficiently as the Witness Statement of Mr Godeseth or the
report of Dr Worden suggest.
Issue 8 ~ Post Office Ability to Identify the Existent & Cause of Discrepancies
5.409
5.410
5.411
Dr Worden has limited his review in relation to Issue 8 by interpreting
the word “alleged” to mean that only shortfalls reported by
Subpostmasters should be considered. I have not limited my analysis in
this way.
Dr Worden’s overarching opinion is that by virtue of its role in the end-
to-end business, Post Office has access to information not available to
Subpostmasters and vice versa.
I agree with Dr Worden’s opinion that Post Office had access to branch
transaction data and that Post Office had access to data which would
not have been available to Subpostmasters. However, it is not clear
what information Subpostmasters would have access to that could not
be obtained by Post Office when trying to determine the existence and
causes of shortfalls. If Dr Worden is referring to information obtained
by Subpostmasters through their day-to-day responsibilities of running
a branch, then he is correct in the sense that Post Office would not have
Prepared by: Jason Coyne
fe)
Occupation: Partner : .
Specialist Field: IT Systems : It FC
[e)
On the Instructions of: Freeths LLP
POL-BSFF-0100992_0235
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 232 of 265
5.412
5.413
5.414
the same first-hand knowledge of what happened in branch. However,
in practical terms, Post Office would be able to access the information
available to any Subpostmaster because they could communicate with
that Subpostmaster. Additionally, it is possible that Subpostmasters
would not have been physically present for a given transaction because
they would not necessarily be the only staff member operating in
branch, so they themselves could be missing that granular level detail.
In relation to Issue 8 overall, Dr Worden suggests that all events are
accurately recorded and properly actioned. See, for example, paragraph
1087 the statement that Horizon “generates events whenever
something unexpected happens...and prompts actions, either
automatically or manually by operations staff.". Although this is the
intended outcome of the Horizon system and is likely to have been
correct in most instances, there is evidence that bugs, errors and
defects have occurred which were not noticed until a Subpostmaster
reported an issue, indicating that attention to events may not have been
sufficiently paid. There is also evidence that reports were being issued
with erroneous data due to software bugs. See, for example, KEL
CCard2053P 2%° where the totals on a Sales Report were reported to be
higher than the number of transactions listed on the corresponding
transaction log or office snapshot. This was due to recreated stock units
doubling up on sales reports.
In the previous paragraph of his report, Dr Worden asserts that when
investigating anomalies reported by Subpostmasters, Post Office use
Credence and their other Management Information Systems in the first
instance but when they need to confirm the transactions handled in a
branch, they can also ask Fujitsu to retrieve the corresponding data
from the audit.
As I have previously stated, there are limitations with this procedure.
Post Office might be satisfied that Credence or their other Information
298 CCard2053P, 21 December 2005, {POL-0035339}
Prepared by: Jason Coyne fe)
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0236
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 233 of 265
Systems reflect the true account of the data and subsequently advise
or make a decision on a TC that the Subpostmaster is liable to settle,
based on an incorrect decision as the underlying data set was not
comprehensive enough in the first instance (Helen Rose Example)
Issues 7 - Were Post Office and/Or Fujitsu able to access transaction data
recorded by Horizon remotely (i.e., not from within a branch)
5.415
5.416
5.417
5.418
Dr Worden limits his review of Issue 7 on the basis that his
interpretation of Issue 7 defined “access” as “access to read”. I have
considered “access” in its technical sense (as in a computer system to
“access” memory) to mean both read and write. However, I agree with
Dr Worden’s statement that both Fujitsu and Post Office were able to
read data remotely. I also agree with Dr Worden in relation his
consideration of what constitutes transaction data however I would also
include any transactional products received from Post Office processing
departments such as Cash Pouches (the value of which would have to
be input to the branch system).
Additionally, (as set out in my first report), it was possible for Fujitsu to
perform modifications and deletions as they could run commands on the
counter machines in branches accessing and querying the hard disk,
which they could do through remote access.
Fujitsu also had the capabilities of performing modifications and
deletions within the branch’s database (latterly the BRDB for Horizon
Online). This is expanded further under Issue 11 commencing at page
249.
It is agreed that remote access and remote control facilities would be
required for Fujitsu support purposes.
Prepared by: Jason Coyne fe)
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0237
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 234 of 265
Issue 10 - Whether the Defendant and/or Fujitsu have had the ability/facility
to: (i) insert, inject, edit or delete transaction data or data in branch accounts;
(ii) implement fixes in Horizon that had the potential to affect transaction data
or data in branch accounts; or (iii) rebuild branch transaction data: a) at all; b)
without the knowledge of the Subpostmaster in question; and c) without the
consent of the Subpostmaster in question?
5.419 At paragraph 1091 of his report, Dr Worden states that he has examined
the Second Witness Statement of Mr Godeseth and where it addresses
Issue 10 finds it consistent with how Horizon works. I note however that
the majority of Mr Godeseth’s opinions that might relate to Issue 10
(inserting, injecting editing or deleting transaction data - Fujitsu) are
actually contained within his first Witness Statement, in which, I have
previously documented that I have found inconsistencies (see Section
4 Defendant’s Responsive Witness Statements - Torstein Olav
Godeseth??).
5.420 I agree with Dr Worden that ‘inject’ means the same as ‘insert’.
TCs and TAs
5.421 Within this section (11.6.2) Dr Worden considers TCs and Transaction
Acknowledgements and states that he does not class them as ‘injected’
transactions.
5.422 I disagree with Dr Worden that TCs are not inserted transactions, which
I would categorise as follows:
a. Transactions inserted by Fujitsu NOT obviously visible to the
Subpostmaster (i.e. balancing transactions inserted into the
MessageStore / BRDB and at other points within Horizon
processing systems past the Counter).
b. TCs - whilst these are visibly acknowledged and accepted by the
Subspostmaster, they are still inserted into the branch accounts to
correct errors. Although Subpostmasters may be able to dispute
them and delay acceptance, this is ultimately in terms of liability
for whether the Subpostmaster is responsible for the funds. Where
299 {Second Witness Statement of Torstein Olav Godeseth, 16 November 2018}
Prepared by: Jason Coyne
fe)
Occupation: Partner : .
Specialist Field: IT Systems ) It 1 ¢
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0238
181024SR1935
POL00262929
POL00262929
01 February 2019 Page 235 of 265
a dispute is accepted, a compensating Transaction Correction is
issued, therefore, the Subpostmaster has no choice but to accept
an insertion into their accounts.
Prior to TCs, I do not consider manual entry of error notice
amounts to be inserted transactions, as the Subpostmaster is
responsible for entering them on their system, which differs from
TCs as they are resident within the accounts electronically.
TAs are considered to be acknowledged insertions. Since they are
visible to the Subpostmaster, as with TCs, they are electronically
received and inserted into the accounts upon acceptance.
5.423 Fundamentally, there are two principles to the above, Fujitsu have the
5.424
ability to insert transactions to fix errors outside of the Subpostmaster’s
knowledge and without their permission which may not be visible to the
Subpostmaster (see paragraph 3.235), and secondly, Post Office have
the ability to electronically insert transactions that are acknowledged
and visible to the Subpostmaster, in the form of TCs and TAs.
A few examples of Fujitsu editing and deleting records from the Horizon
branch database are set out in 21 December 2018 disclosure of MSC
records:
a.
Contained within the MSC Documents provided?” the lines
serialised with the codes 043J0262492, 04330264220 and
04330265130 record the steps followed to resolve “The Business
Problem: To prevent us having to talk unhappy PMs through the
complicated workaround described in KEL acha3347Q*°! we need
to remove any declarations belonging to stock units deleted since
15th May”. These steps display the command “delete from
ops$brdb.brdb_branch_decl” which I believe will delete records
from the branch database. The document suggests that this will
300 MSC_RTI_Answers_POA(1).csv, MSC_RTI_Answers_POA {POL-0444103}
301 KEL acha3347Q, 5 February 2010 last updated 2 September 2010 {POL-0037767}
Prepared by: Jason Coyne
fe)
Occupation: Partner : .
Specialist Field: IT Systems : It FC
[e)
On the Instructions of: Freeths LLP
POL-BSFF-0100992_0239
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 236 of 265
address errors in the branch database caused by an early Horizon
bug. These MSC records are also recorded in the PEAK reference
PC0199654 302
b. Document 043J0265683 records the steps followed to resolve;
“Current Business Position: There are duplicate rows coming
through from BRDB into BRSS. Exact cause is yet unknown.”. These
steps display the command “DELETE FROM
ops$brdb.brdb_pouch_coll_details” which I believe will delete
records from the branch database. The document displays a
question; “Does this change need to be assessed by POL?:” the
answer in the document is shown as “No.Involves BRSS only”
c. MSC043J0355958 records the “SQL insertion” of “Dummy
Transaction Acknowledgement” into the branch database to
correct a fault within Horizon that was later fixed. This record
suggests that the same process had been completed previously
under record MSC043J0348236.
Global Users
5.425
5.426
Global Users are clarified further to my initial report at paragraph 4.11
to 4.19 of this report in response to points addressed by Mr Godeseth
in his Responsive Witness Statement. In summary, Mr Godeseth states
that a person must be physically present in a branch to enter a
transaction for that branch. Dr Worden makes the same statement.
However, I have reached a different understanding (as set out at 4.11
to 4.19 of this report).
Dr Worden implies at section 11.6.4 of his report that DBAs would not
misuse their power in carrying out tasks they should not. The issue is
whether Fujitsu COULD insert, edit and delete transaction data, to which
the answer is yes, they could. I do not believe that Dr Worden has
reviewed or observed Fujitsu’s process compliance in the event of all
such activities because to do so would be an extremely lengthy task.
302 PEAK PC0199654, 28 May 2010 {POL-0369488}
Prepared by: Jason Coyne fe)
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0240
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 237 of 265
5.427
For example, it is stated that Fujitsu cannot alter any branch transaction
data without permission from Post Office. From PEAK observations, it is
clear that sometimes this is requested via an OCP which is approved yet
other times it appears to be granted by a different method (such as
textual agreement in the form of an email or a comment - see
PC0256213°3) and Fujitsu proceed upon that basis. I do not believe Dr
Worden has audited every single transaction amendment to ensure that
policy was followed in every instance.
Also, it is not (in my opinion) a question of whether DBAs misused their
powers, it is more important to consider (in respect of their actions)
whether they might have erroneously (without intent) modified data.
Balancing Transactions
5.428
5.429
5.430
5.431
5.432
Dr Worden and I agree that Fujitsu SSC had the ability to insert
Balancing Transactions (BTs) using the ‘Host BRDB Branch Correction
Tool’ into certain tables in the BRDB (Horizon Online Branch Database).
It is important to note however that SSC would also have the ability to
perform balancing transactions via direct SQL operations (using a
command line type interface) to perform corrective transactions on
other database tables within the BRDB outside of the corrective tool
usage, via the use of Privileged User access (Horizon Online).
Where Dr Worden proceeds to state “Branch Trading Statement” within
this section, I have interpreted that it is typographical error and should
read “BT” or Balancing Transaction.
At paragraph 1113, Dr Worden re-states Mr Godeseth’s evidence that
BT’s are clearly visible in the transaction reports that are available to
Subpostmasters.
It is important to note that in my opinion, it is not quite so simple or
obvious as Dr Worden or indeed Mr Godeseth set it out to be.
303 PEAK PC0256213, 29 December 2016 {POL-0424338}
Prepared by: Jason Coyne fe)
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0241
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 238 of 265
5.433
5.434
5.435
5.436
BTs entered directly into the Branch’s Database would only be
identifiable as a transaction on the day that the corrective action was
performed. Therefore they would feature within a different Audit File
than the original erroneous transaction.
Using the example of the one BT that is acknowledged by Post Office
and in the Witness Statement of Mr Godeseth contained in PEAK
PC0195561:2%
On 02 March 2010 a Transfer Out of £4000.00 doubled up to £8000.00
due to a Horizon error, the suggested correction by Gareth Jenkins
(Fujitsu) was for support to use the Transaction Correction Tool?® to
insert two records into the database to negate the duplicate Transfer
Out. The PEAK record documents that support performed this corrective
action on 11 March 2011. Therefore, it would not be until the 11 March
2011 that the additional inserted corrective transactions would be
identifiable within audit records.
Ihave already set out my opinion on this point in response to Mrs Angela
Van Den Bogerd 4.71 above. Further, aside from Subpostmasters
allegedly being able to identify it as a transaction carried out from
Counter 99; for it to be “clearly identifiable” to the Subpostmaster, or
anyone inspecting the branch accounts it would require:
a. The Subpostmaster/inspector of the accounts knowing which
particular transaction went awry in the first place (this might not
be immediately visible in a branch processing many transactions
per hour);
b. The implications of the incident and error fully known by both
support and the Subpostmaster/inspector of the accounts in order
to identify where any corrective action might be applicable or
identifiable;
304 PEAK PC0195561, 4 March 2010 {POL-0365465}
305 DEVAPPLLD0142.doc, Host BRDB Transaction Correction Tool Low Level Design, 13 November
2007 {POL-0032866}
Prepared by: Jason Coyne
fe)
Occupation: Partner : .
Specialist Field: IT Systems : It FC
[e)
On the Instructions of: Freeths LLP
POL-BSFF-0100992_0242
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 239 of 265
c. Support and the Subpostmaster/ inspector of accounts knowing
the specific date and timeframe that any corrective actions were
performed, how they were performed, and their impact in order to
redress the reports or logs in which it might be reflected as
rectified. This would be largely dependent upon:
i Support communicating to the Subpostmaster/inspector of the
accounts how they were going to implement a fix and when
(where the error was known by the Subpostmaster or if not
known, informing the Subpostmaster in the first instance of the
error);
ii. Support ensuring that the corrective fix was performed
correctly;
iii. Subpostmasters indeed knowing what a Counter 99
transaction was.
d. Post Office being fully aware that the error was Horizon generated
and therefore not the fault of the Subpostmaster or issuing a
Transaction Correction to remedy the imbalance.
5.437 I note that the OCP (Operational Corrective Procedure) for the above
corrective fix has been disclosed by Post Office?°® but contains limited
information (in respect of the requirements I have listed above).
5.438 In summary, it is my opinion that more than one BT has been conducted
by Fujitsu, for the following reasons:
i. PEAK PC0195962°°7 created 12 March 2010 relates to the
Transaction Correction tool and states:
“The Transaction Correction tool has now been used in live. The templates
for use with this tool need to be updated to correct some details. Gareth
Seemungal is aware of the corrections needed...
306 OCP 25882, 10 March 2010 {POL-0440067}
307 PEAK PC0195962, 12 March 2010 {POL-0365857}
Prepared by: Jason Coyne fe)
Occupation: Partner .
Specialist Field: IT Systems ; It lI
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0243
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 240 of 265
..The proposed fix would correct and update the BRDB transaction
correction tool templates, making it less likely that mistakes will occur
when SSC are trying to resolve problems with transactions in BRDB.”
This suggests that the modifications and balancing transactions
conducted by Fujitsu support staff within the BRDB is not
unusual.
ii. Fujitsu were able to insert balancing transactions outside of
utilising the Branch Correction tool referred to above. Balancing
transactions were not limited to Horizon Online. The PEAKs
detailed in the Horizon Issue 10 PEAKs at Section 3 above
indicate which of those that relate to balancing transactions.
iii. One of the deleted KEL’s, cardc262S°%° under the heading
“Solution - ATOS” includes the rather matter of fact statement;
“The transaction Correction tool should be used to correct it (this
will need an OCP and probably POL approval too)”. It is not clear
if the suggestion is that ATOS should use the transaction
correction tool, or if ATOS are suggesting to Fujitsu or Post
Office that they should use the transaction correction tool.
Transaction Injection in Legacy Horizon
5.439
5.440
In relation to transaction injection in Legacy Horizon, Dr Worden relies
further upon the first Witness Statement of Mr Godeseth. Dr Worden
acknowledges that in Legacy Horizon, SSC could also inject transactions
into branch accounts, which I agree.
In a similar vein to detecting balancing transactions in Horizon Online,
Dr Worden therefore concludes that SSC users could update branch
accounts without the consent of the Subpostmaster, but not without
their knowledge, since the Counter ID would be greater than 32. For
the reasons set out above, at paragraph 5.441 below, I disagree that
the visibility of the modification would be so simple or obvious to the
Subpostmaster.
308 KEL cardc262S, 9 March 2010 last updated 4 May 2010 {POL-0448597}
Prepared by: Jason Coyne fe)
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0244
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 241 of 265
5.441 Further, Mr Roll in his second Witness Statement dated 06 January 2019
states at paragraph 20 that the method which would display a counter
position greater than 32 could be circumvented. Mr Parker in his
statement served in response has now said this is correct, and could be
done, which in my opinion is significant. Where this was the case, any
transactions inserted as though they came originally from the Counter
would not be obvious to the Subpostmaster at all.
5.442 It is my belief, that in review of the PEAKs documented in Section 3
‘Evidence of Insertions/Deletions within Branch Accounts (Horizon Issue
10) of this report, that SSC could not only inject/insert or edit
transaction data but delete instances of it (and/or operations relating to
it, which are of equal importance) also.
5.443 At paragraph 1117 I note that Dr Worden inherits his opinion from the
evidence provided by Mr Godeseth that messages from the message
store (in Legacy Horizon) could not be updated or deleted. However, in
my analysis of the PEAK records at Section 3 (‘Evidence of
Insertions/Deletions within Branch Accounts (Horizon Issue 10)’, I have
demonstrated that this is not the case. One example of an update (of
which further detail can be found in the aforementioned Section 3) is as
follows:
5.444 PC0130275° created 21 December 2005 (further detail provided at
3.230 of this report). states:
“... This has resulted in a gain of approximately £18000.
We are unable to correct the system figures safely. We can however
provide accurate figures for what should have been in the Final Balance
for BB, to enable POL to make the correction perhaps by using a
Transaction Correction.
POL need to make a decision on whether they are able to correct the
problem in this way, however we do not see any other alternative.
309 PEAK PC0130275, 21 December 2005 {POL-0300707}
Prepared by: Jason Coyne fe)
Occupation: Partner .
Specialist Field: IT Systems ; It lI
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0245
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 242 of 265
Corrective action should be taken before 11th January when the branch
is due to roll into TP10...
... If we get to the problem before the office is rolled we are able to change
objects in the messagestore to reset the stockunit back to the CAP (TP)
rollover trailer. The PM can then rollover. PM should get a large shortage
which cancels out the large gain.
We don't want to be having to do this as making manual changes to the
messagestore is open to error and each time we have to seek
authorisation from POL to make the changes.”
5.445 A further example of deletion (of which there are more at Section 3) is:
5.446 PEAK PC00579093!° dated November 2000 (further detail provided in
Section 3 at paragraph 3.249) refers to an issue occurring as a result of
a branch's counter base unit replacement, and sets out:
“Can development please investigate on whether there is a deficiency in
Riposte and what can be done to stop this happening again. Also, need
advice on how to get the messagestores in sync and to include the
missing transactions. I suspect we will need to trash the messagestores
on counters 2 and 3 and insert the missing messages onto counter 1 (or
can the PM get away with inputting the transactions). Some of the
transactions are APS. Also how will this affect their balancing. They are
currently in CAP 34.”
5.447 I assume “trash the messagestores” to mean delete them and
potentially rebuild them
5.448 Inrelation to Dr Worden’s comments with regards to the second witness
statement of Mr Roll?*!. I have provided comments on this at paragraph
5.482 in relation to Dr Worden’s assertions regarding transaction
injections and how and whether these could be identified by user. Mr
Roll’s witness statement disputes this view, and this is further evidenced
at paragraph 4.83b and 5.441 of my report where he confirms that SSC
310 PEAK PC0057909, 15 November 2000 {POL-0232732}
311 {Second Witness Statement of Richard Roll, 16 January 2019}
Prepared by: Jason Coyne fe)
Occupation: Partner ae
Specialist Field: IT Systems It }roOu f )
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0246
181024SR1935
POL00262929
POL00262929
01 February 2019 Page 243 of 265
did inject transactions at the counter in such a way they would appear
on the transaction log as if they had been inserted within the branch.
Privileged Users
5.449 In respect of paragraph 1122 of Dr Worden’s report, I agree that it
5.450
would be necessary for Fujitsu support staff to have access privileges
used to edit or delete transaction data in the BRDB. Where Dr Worden
states that there is little need to use privileged access to manipulate
transaction data to resolve an error, I agree that in theory it SHOULD
be this way. But evidence suggests that this was not the case in
actuality.
Dr Worden states (at paragraph 1123) that any change to a transaction
performed by a Privileged User would be visible to branch staff.
However, in my opinion, there are several points to note in relation to
such a statement:
a.
Witness evidence suggests that whilst amended transactions
would become visible within branch reports they would carry no
indicator that they had been performed by a Privileged User.
Therefore, in my opinion, Dr Worden is overstating the
obviousness of their visibility;
It is unlikely that a Subpostmaster would know of the audit
process within Horizon not least be informed to enquire or request
that Post Office look to that to identify discrepancy;
As with the visibility of Balancing Transactions, identification of the
modification would require:
The Subposmtmaster/inspector of the accounts knowing which
particular transaction went awry in the first place (this might not
be immediately visible in a branch processing many transactions
per hour);
Prepared by: Jason Coyne fe)
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0247
181024SR1935
POL00262929
POL00262929
01 February 2019 Page 244 of 265
vi.
vii.
The implications of the incident and error fully known by both
support and the Subpostmaster/inspector of the accounts in
order to identify where any corrective action might be applicable
or identifiable;
Support and the Subpostmaster/ inspector of accounts knowing
the specific date and timeframe that any corrective actions were
performed, how they were performed, and their impact in order
to redress the reports or logs in which it might be reflected as
rectified. This would be largely dependent upon:
Support communicating how they were going to implement a fix
and when to the Subpostmaster/inspector of the accounts
(where the error was known by the Subpostmaster) or if not
known, informing the Subpostmaster in the first instance of the
error;
Support ensuring that the corrective fix was performed
correctly;
Subpostmasters indeed knowing what a Counter 99 transaction
was.
Post Office being fully aware that the error was Horizon
generated and therefore not the fault of the Subpostmaster or
issuing a Transaction Correction to remedy the imbalance
5.451 All of the above is only relevant in the case of transactions that were
investigated and modified due to a disputed transaction that the
Subpostmaster was aware of.
5.452 Fujitsu has no policy, process, procedure or operational practice that
calls for it to use its privileged access to edit or delete transaction
data.3!* Therefore, if Privileged User access was being used (which I
opine that it was) there is no clear process for it. This introduces a high
312 {Witness Statement of Torstein Olav Godeseth, 27 September 2018}
Prepared by: Jason Coyne fe)
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0248
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 245 of 265
5.453
5.454
element of risk as users were not effectively governed or constrained
by any form of compliance for its use.
I understand that prior to July 2015, only log on and log off activities
for Privileged Users were recorded. It is stated in Mr Godeseth’s first
witness statement at paragraph 59.6) that such were recorded in a
Master Service Change (MSC) document. Whilst Post Office have set out
in their letter dated 21 December 2018 that Privileged User Logs can
only be provided back to 2009, in my opinion this should still encompass
approximately 2 years or so of Legacy Horizon Privileged User access. I
have provided my analysis (and subsequent limitations of it) in relation
to the MSC disclosure provided to me at Section 3 of this report. In
summary, through the nature of the way the disclosure was provided,
it has not been possible to determine where within it, or even if within
it, Privileged User access is recorded for Legacy Horizon. This could be
something perhaps further explored in mine and Dr Worden’s next Joint
Statement, seeking the assistance of Post Office/Fujitsu to interpret the
complexities of the data, to derive a more succinct quantitative record
of Privilege User access for Legacy Horizon in the form of simple numeric
values per year. Whilst I appreciate that Post Office have set out some
high-level guidelines in respect of how to interpret the data, I have faced
difficulties with the instructions provided. That, and in combination of
its delayed disclosure, I have therefore not had sufficient time in my
reporting to effectively analyse the data information provided.
I understand that Post July 2015, all access and actions (not just log on
and log off) was recorded to an Oracle audit table. As aforementioned,
I have faced difficulty in interpreting the Privileged User disclosure, full
details of which are set out at Section 3 ‘Privileged User Log Disclosure’.
As previously stated, this could perhaps be further addressed in the
second Joint Statement to be prepared by myself and Dr Worden.
Prepared by: Jason Coyne fe)
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0249
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 246 of 265
Implement fixes in Horizon that had the potential to affect transaction data or
data in branch accounts
5.455
5.456
5.457
5.458
Dr Worden and I agree that fixes implemented by Fujitsu had the
potential to affect transaction data or data in branch accounts.
I note that within this section Dr Worden diverges somewhat from
assessing what corrective transactional fixes performed by Fujitsu might
further affect transaction data or data in branch accounts and instead
focusses on fixes to reference data and software.
I agree with Dr Worden that all of the above could be carried out without
the consent or knowledge of the Subpostmaster. Whilst I agree that
there would not typically be a need for standard changes in relation to
software and reference data being communicated to Subpostmasters, I
believe that in circumstances such as widespread system releases,
major product changes and any other identified significant modification
that could affect their financial position, in my opinion, it would not have
been harmful to notify them. Typically, in industry, when major software
releases are rolled out, end users are notified. For example, a Window's
upgrade on a personal computer, the end user of that system is
prompted to accept the upgrade.
Effects on transaction data should not only be considered in respect of
balancing transactions or transaction data concerning monetary value.
Financial account accuracy involves much more than just ensuring the
double entry principle is applied. A Subpostmaster’s branch account
accuracy is dependent upon various other aspects. For example, stock
unit records being appropriately measured, transaction dates being
accurate, trading and cash account periods being accurate. Consider the
scenario where an asset is purchased — whilst the double entry principle
might have been applied correctly, if the year of the purchase was
recorded incorrectly, the transaction would not feature in the relevant
accounting period. Therefore, corrective actions performed by Fujitsu
outside of balancing transactions are also vitally important to consider.
Prepared by: Jason Coyne
fe)
Occupation: Partner : .
Specialist Field: IT Systems : It FC
[e)
On the Instructions of: Freeths LLP
POL-BSFF-0100992_0250
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 247 of 265
As evidenced in Section 3 PEAK PC0197592,3!3 Fujitsu could also
“correctively” delete stock unit opening balances (which is used in the
ultimate calculation of a Subpostmasters cash and stock declarations)
in order to “reset” them. Whilst the opening balance would not be
completely removed by deletion here (it is rolling back the trading
period and it would be possible to recalculate the opening figure) since
Post Office derived the accuracy of a Subpostmasters accounts from its
various stock / cash declarations in their relevant periods, this alteration
is significant in that it can change a period for which accounts have to
align.
Rebuild transaction data
5.459
5.460
5.461
5.462
In relation to rebuilding branch transaction data, Dr Worden states that
this part of the issue relates to a technical robustness countermeasure,
rather than some discretionary change to transaction data. In my
opinion, the issue to address here is could Fujitsu rebuild branch
transaction data, with or without the consent of the Subpostmaster and
in effect, is there direct evidence to illustrate that they did.
Previously within his report, (at paragraph 1059) Dr Worden states:
“Similarly, for part (iii) of Issue 10, Fujitsu had the ability to rebuild
transaction data, because this was a very necessary part of the
robustness countermeasures. It is important to understand that this
rebuilding was an automated process, using a redundantly stored copy of
the transaction data (RDS), and did not involve discretionary manual
rebuilding.”
Dr Worden does not reference any documentation with regards to how
he gained his understanding of the process of rebuilding branch
transaction data, nor does he state ‘branch’ but merely ‘transaction
data’.
PEAKs identified within Section 3 ‘Data Rebuilding’ identify to me, that
manual rebuilding of data did indeed take place.
313 PEAK PC0197592, 12 April 2010 {POL-0367467}
Prepared by: Jason Coyne fe)
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0251
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 248 of 265
5.463 At paragraph 1134 Dr Worden states that due to the nature of any BRDB
rebuild that might take place:
“In principle, the data could be rebuilt without the knowledge of the
Subpostmaster in question, but they would be informed or become aware
that they could use Horizon normally again and so they would know that
something had happened.
5.464 Whilst I agree that data could be rebuilt without the knowledge or
consent of the Subpostmaster in my opinion, it is too broad an
assumption to state that a Subpostmaster being informed that they
could use the system again implies that they would know anything had
happened, not least an account rebuild.
5.465 In summary of this Issue I disagree with Dr Worden that in Legacy
Horizon Fujitsu could not edit or delete transaction data.
5.466 I also disagree that they could not do it without the knowledge of the
Subpostmaster.
Issue 11 - If they did, did the Horizon system have any permission controls
upon the use of such facility, and did the system maintain a log of such actions
and such permission controls?
5.467 Primarily at paragraph 1060 of his report, Dr Worden sets out that any
alterations of branch transaction data carried out by any central user
would leave many traces of their activity like footprints in fresh snow.
5.468 In my opinion Dr Worden largely oversimplifies the actuality of how
obvious it would be to trace a central users’ actions in relation to the
alteration of branch transaction data. Primarily, branch transaction data
is subject to an extremely high level of interaction within its processing
and propagation to POLSAP. To identify and diagnose manual
intervention within its entire journey; at what access level, by whom,
and what activity they did actually undertake, is not as simple as
observing “footprints in fresh snow” as there are many more than just
one set of footprints. I have set out my observations in response to the
auditability limitations above under Issue 10.
Prepared by: Jason Coyne
fe)
Occupation: Partner : .
Specialist Field: IT Systems ) It 1 ¢
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0252
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 249 of 265
5.469 Dr Worden states, (taken from the first Witness Statement of Mr
Godeseth?!*) that there are 30 SSC users permitted to create a
Balancing Transaction (in Horizon Online) and approximately 45 with
Privileged User access (whom had more access capabilities than the 30
above and could not only modify data but delete also).
5.470 It is not clear from the witness evidence where those numbers of users
above might overlap, and Dr Worden does not clarify.
5.471 Further, it is important to note that it is my belief that there is some
confusion in Dr Worden’s understanding. Where Mr Godeseth sets out
that only 30 users can create Balancing Transactions, he is implying
only those who conduct it through the use of the Transaction Correction
Tool. He (nor Dr Worden) does not reflect the true number of users who
could perform Balancing Transactions outside of the usage of such a
tool (which was effectively for Legacy Horizon all of SSC support and for
Horizon Online anyone with Privileged User access).
5.472 I have reviewed the Host BRDB Transaction Correction Tool Low Level
Design?*5 referred to by both Mr Godeseth and Dr Worden and I can see
no indicator within it that only 30 users could access the tool. The
document states, “the utility will allow SSC to correct transactions”.
5.473 At paragraph 1141 of his report, Dr Worden quotes from the high-level
design document for the BRDB?!® “Support teams will be restricted to
accessing the BRDB only under an MSC”. He further states that he has
introduced the MSC process in Appendix C.
5.474 I have set out my observations in respect of the MSC disclosure above
at Section 3. In summary, I have not been able to perform a full review
of the data due to its complexities and time constraints. However, I
have set out some preliminary observations and sought to clarify some
314 {Witness Statement of Torstein Olav Godeseth, 27 September 2018}
315 DEVAPPLLD0142.doc, Host BRDB Transaction Correction Tool Low Level Design, 13 November
2007 {POL-0032866}
316 DESAPPHLD0020.doc, Branch Database High Level Design, 5 April 2018 {POL-0219310}
Prepared by: Jason Coyne
fe)
Occupation: Partner : .
Specialist Field: IT Systems ) It 1 ¢
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0253
181024SR1935
POL00262929
POL00262929
01 February 2019 Page 250 of 265
5.475
5.476
queries in respect of further identified privileges not expressed in Post
Office’s letter dated 21 December 2018.
In continuation of the MSC process as set out by Dr Worden in his
Appendix C:
a.
At paragraph 364 and 365 of the Appendix, Dr Worden limits his
opinion to stating what document the process has been defined in
and that its predecessor was the “Operational Change Process”
(OCP);
He does not set out what specific document the OCP process might
be defined in. It could be that this is because Mr Godeseth does
not reference any explicit document in relation to OCPs.
I have reviewed the ‘MSC Managed Service Change Procedure’ dated
201437 and note the following:
a.
It is a high-level document, revised between 2010 and 2014, the
last revision appearing as 14 July 2014;
The document largely appears more relative to large scale system
changes, and does not clearly or specifically detail ad hoc changes
adopted by Privileged Users or SSC relative to change of financial
accounts;
The “Roles” that Dr Worden states (at paragraph 364 of his
appendix) “contribute to operation of the process” are listed as:
Change Initiator (CI)
Change Sponsor (normally the service manager) (CS)
Change Administrator (or Change Analyst) (CA)O
Impact Assessors (IA)
Change Owner (CO)
Task Owner
317 SVMSDMPRO1184_1.DOC, MSC Managed Service Change Procedure for Post Office Account, 11
July 2014 {POL-0136725}
Prepared by: Jason Coyne
fe)
Occupation: Partner : .
Specialist Field: IT Systems ) It 1 ¢
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0254
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 251 of 265
. Change Manager (CM)
. Change Approval Board / Emergency Change Approval Board
(CAB / ECAB)
. Service Manager
. Resolver G
5.477 At paragraph 1142 of his report Dr Worden then goes on to state that
the Branch Database High Level Design?!® goes on to confirm:
“There is a requirement that the SSC will have ability to insert balancing
transactions into the persistent objects of the BRDB. There are reasons
for SSC having to do so e.g. to rectify erroneous accounting data that
may have been logged as a result of a bug in the Counter / BAL.
SSC will have privileges of only inserting balancing / correcting
transactions to relevant tables in the database. SSC will not have any
privileges to update or delete records in the database.
Any writes by the SSC to BRDB must be audited.”
5.478 I feel it is important to note here that in my opinion, the scope of whom
could insert balancing transactions into the branch database (Horizon
Online) is here reflected as SSC. Not only those who were enabled
access to the Transaction Correction Tool. This accords with my
understanding as set out at paragraph 5.472 above. Also, I find the
statement conflicting as it has previously been acknowledged (by
Godeseth and Worden, and as I understand) that SSC (Privileged
Users), could edit and delete.
5.479 In relation to Mr Parker’s Witness Statement (20.2):
Some members of the SSC were (and some remain) able to insert
transaction data. SSC access privilege gave the ability to inject
transactions, but appropriate change controls were in place and no such
insertion would have happened without complying with those controls.”
318 DESAPPHLD0020.doc, Branch Database High Level Design, 5 April 2018 {POL-0219310}
Prepared by: Jason Coyne fe)
Occupation: Partner ae
Specialist Field: IT Systems It }roOu f )
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0255
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 252 of 265
5.480
5.481
5.482
5.483
Dr Worden states (paragraph 1145 of his report) that this is consistent
with his understanding of the role of the SSC. I disagree that it is
consistent. In my opinion, if anything, the audit, processes and controls
around balancing transactions and whom within SSC had exactly what
privileges is ambiguous.
Dr Worden comments further on the Witness Statement of Mr Parker,
stating how double entry accounting principles would enable
identification of any inserted transactions within the branch accounts
performed by SSC. I disagree with both Mr Parker and Dr Worden that
any modifications would be so readily identifiable for reasons given
within Issue 10 of this report and also, consideration of the following
PEAK evidence.
PC01520143!9 (full PEAK details provided at paragraph 3.234) detail an
instance where SSC have to perform a one-sided transaction that no
settlement value was written for (therefore POLSAP did not receive its
value):
“Worth noting that the branch did not have any issues with the
mismatched transactions because this was fixed before they did the roll.
The branch is not aware of this and it's best that the branch is not
advised.”
Where Dr Worden states at paragraph 1151 that creation of transactions
would be clearly associated by their user, I feel it is important to
consider here, the Witness Statement of Richard Roll dated 6" January
2019 which disputes this.
In conclusion to this Issue Dr Worden sets out (at paragraph 1153) that
in summary, he believes permissions to use the facilities described
under Issue 10 were controlled. I disagree with Dr Worden, in
conclusion, for the following reasons:
319 PEAK PC0152014, 7 December 2007 {POL- 0322311}
Prepared by: Jason Coyne fe)
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0256
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 253 of 265
5.484 It is my opinion that SSC users (whether privileged user or not) were
not as restricted as they should have been or as averred by Mr Godeseth
and Dr Worden for the following reasons:
a. The activity identified from my analysis of the MSC records (as
referred to above);
b. The PEAK evidence referenced in relation to Issue 10 (paragraph
3.220) which records (in contrast to Mr Godeseth’s findings) that
transaction data and related operational activities were edited and
deleted within Horizon; and
c. External Audit reports (Ernst & Young 2011 referenced in my first
report at paragraph: 9.65 and also referenced in this report at
paragraph 5.154) and PEAK evidence (paragraph 3.283 of this
report) stating insufficiencies and non-conformance to policy in
respect of access rights and capabilities of resources. It is not clear
if the number of users provided by Mr Godeseth at paragraph 59.1
of his witness statement having escalated access to data include
or exclude the users who should not have had access but did until
July 2015 when the auditing began.
5.485 Further, the fact that prior to July 2015, SSC privileged usage was only
auditable by record of a log on and log off and contained no detail with
regards to what actions were performed by them is to me, not
controlled.
5.486 Further, Dr Worden has not reviewed the OCP process applicable to
Legacy Horizon or performed any analysis of contemporaneous
documentation to identify where there might have been failures in
control.
Prepared by: Jason Coyne fe)
Occupation: Partner : -
Specialist Field: IT Systems g It ] }{
On the Instructions of: Freeths LLP < -
POL-BSFF-0100992_0257
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 254 of 265
Issue 12 — If the Defendant and/or Fujitsu did have such ability, how often was
that used if at all?
5.487
5.488
5.489
Dr Worden states at paragraph 1164 that “Branch Trading Statement
have only been used once”. I assume here that Dr Worden means
actually to refer to the one acknowledged Balancing Transaction
conducted using the Branch Correction Tool.
I disagree there has only ever been one balancing transaction
performed (in consideration of PEAKs evidencing them outside of
Correction Tool usage) which I have set out under Issues 10 and 11.
The following capabilities could have impacted branch accounts. I have
been unable to confirm how often they were used:
i Corrective transactional fixes including insertions, edits or
deletions performed by SSC (by a privileged user or via the
transaction correction tool) or Post Office (in the form of TCs
issued as a result of identification of Horizon error);
ii. Transaction inserts carried out by Global Users;
iii. Messagestore or Branch Database rebuilds.
Issue 13 - To what extent did use of any such facility have the potential to
affect the reliability of Branches’ accounting positions?
5.490
5.491
5.492
As in Issue 1, I have interpreted ‘extent’ differently to Dr Worden and
instead have considered to what extent was it technically feasible for
affect to occur on a branch's accounting position rather than assessing
extent in terms of the probability of financial impact.
I do not interpret this Issue to be solely in relation to causing financial
impact. In my opinion, the Issue states “reliability of Branches
accounting positions”.
Therefore, in my opinion I believe it is important to consider with
regards to affecting the reliability of the branches’ accounting position,
not only instances where insertions, injections, edits, deletions or
rebuilds might affect transaction data, but also “data in branch
Prepared by: Jason Coyne
fe)
Occupation: Partner : .
Specialist Field: IT Systems : It FC
[e)
On the Instructions of: Freeths LLP
POL-BSFF-0100992_0258
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 255 of 265
5.493
5.494
5.495
accounts”. Such, in my opinion, would further include and comprise of
data in relation to operational actions.
Financial account accuracy involves much more than just ensuring the
double entry principle is applied in relation to a monetary transaction.
A Subpostmaster’s branch account accuracy is dependent upon various
other aspects e.g., stock unit records being appropriately measured,
rollover dates being accurately recorded, trading and cash account
periods being aligned, user actions appropriately recorded, and so on.
All of which, could be affected by abilities and facilities in place to allow
Fujitsu/Post Office to perform the actions listed under Issue 10.
I therefore cannot agree with Dr Worden and his calculations in respect
of paragraph 1175 of his report, since I do not agree there has only
ever been one balancing transaction performed by SSC, save to say
that the probability of impact would not be one part in 1.5 million.
I cannot agree with him further at paragraph 1177 since I do not agree
that KELs could be relied upon to reflect the true account of an incident,
therefore in my opinion, his basis for performing calculations in relation
to assessment of financial impact is flawed.
Prepared by: Jason Coyne fe)
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0259
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 256 of 265
6. Expert Declaration
I Jason Coyne DECLARE THAT:
6.1 I understand that my duty in providing written reports and giving
evidence is to help the Court, and that this duty overrides any obligation
to the party by whom I am engaged or the person who has paid or is
liable to pay me. I confirm that I have complied and will continue to
comply with my duty.
6.2 I confirm that I have not entered into any arrangement where the
amount or payment of my fees is in any way dependent on the outcome
of the case.
6.3 I know of no conflict of interest of any kind, other than any which I have
disclosed in my report.
6.4 I do not consider that any interest which I have disclosed affects my
suitability as an expert witness on any issues on which I have given
evidence.
6.5 I will advise the party by whom I am instructed if, between the date of
my report and the trial, there is any change in circumstances which
affect my answers to points 3 and 4 above.
6.6 I have shown the sources of all information I have used.
6.7 I have exercised reasonable care and skill in order to be accurate and
complete in preparing this report.
6.8 I have endeavoured to include in my report those matters, of which I
have knowledge or of which I have been made aware, that might
adversely affect the validity of my opinion. I have clearly stated any
qualifications to my opinion.
6.9 I have not, without forming an independent view, included or excluded
anything which has been suggested to me by others, including my
instructing lawyers.
Prepared by: Jason Coyne fe)
Occupation: Partner : -
Specialist Field: IT Systems g It ] }{
On the Instructions of: Freeths LLP < -
POL-BSFF-0100992_0260
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 257 of 265
6.10 I will notify those instructing me immediately and confirm in writing if,
for any reason, my existing report requires any correction or
qualification.
6.11 I understand that:
a. my report will form the evidence to be given under oath or
affirmation;
b. questions may be put to me in writing for the purposes of clarifying
my report and that my answers shall be treated as part of my
report and covered by my statement of truth;
c. the court may at any stage direct a discussion to take place
between experts for the purpose of identifying and discussing the
expert issues in the proceedings, where possible reaching an
agreed opinion on those issues and identifying what action, if any,
may be taken to resolve any of the outstanding issues between the
parties;
d. the court may direct that following a discussion between the
experts that a statement should be prepared showing those issues
which are agreed, and those issues which are not agreed, together
with a summary of the reasons for disagreeing;
e. I may be required to attend court to be cross-examined on my
report by a cross-examiner assisted by an expert;
f Iam likely to be the subject of public adverse criticism by the judge
if the Court concludes that I have not taken reasonable care in
trying to meet the standards set out above.
6.12. Ihave read, the accompanying practice direction and the Guidance for
the instruction of experts in civil claims and I have complied with their
requirements.
6.13. Iam aware of the practice direction on pre-action conduct. I have acted
in accordance with the Code of Practice for Experts.
Prepared by: Jason Coyne fe)
Occupation: Partner .
Specialist Field: IT Systems ; It lI
On the Instructions of: Freeths LLP <
POL-BSFF-0100992_0261
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 258 of 265
Statement of Truth
6.14 I confirm that I have made clear which facts and matters referred to in
this report are within my own knowledge and which are not. Those that
are within my own knowledge I confirm to be true. The opinions I have
expressed represent my true and complete professional opinions on the
matters to which they refer.
Signed: L
Jason Coyne
Partner
Dated: 01 February 2019
Prepared by: Jason Coyne
Occupation: Partner [ol .
Specialist Field: IT Systems It
On the Instructions of: Freeths LLP
POL-BSFF-0100992_0262
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 259 of 265
7. Appendices
Appendix A
Letter from WBD re Privileged User Logs and MSC Logs disclosure.
Prepared by: Jason Coyne
Occupation: Partner (of .
Specialist Field: IT Systems It
On the Instructions of: Freeths LLP
POL-BSFF-0100992_0263
181024SR1935
01 February 2019
POL00262929
POL00262929
Page 260 of 265
Appendix B
HNG-X Menu Hierarchy
7A HNG-X Menu Hierarchy and Messages, Section 9.2 Role Capabilities*?°
PTS Te TSE TE TE TS Te EEE
s/#/S IG I/2/5 IS I# Ie I2/2 18/6
Q9@lig@\/dlas/£ Ie }2 a2\/2\o I=
as Sle ja/S is I? Fa o Ii
=\I2\2 z/Sji2\z2 Oo Ie
= /2 < a B IS
5
Capability 2 $I8
[Administ
ration:] AddNewOffice Y Y Y Y Y
[Administ
ration:] AddUser Y Y Y Y Y Y Y
[Administ
ration:] AttachSU Y Y Y Y Y Y Y Y Y Y
[Administ
ration:] CreateSU Y Y Y Y Y
[Administ
ration:] DeleteSU Y Y Y Y Y
[Administ
ration:] DeleteUser Ld Y Y Y Y Y Y
[Administ
ration:] Migration Y
[Administ
ration:] ModifyUser Y Y Y Y Y Y Y
[Administ
ration:] SalesPrompts y Iy y ly y ly
[Administ
ration:] UnlockRemoteSU Y
[Administ
ration:] ViewAttach Y Y Y Y Y Y Y Y Y
[Administ
ration:] ViewSUs Y Y Y Y Y Y Y Y
320 DESGENSPE0007_6.2.doc, HNG-X Menu Hierarchy and Messages, 5 April 2018 {POL-0153568}
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
Zitg roup
POL-BSFF-0100992_0264
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 261 of 265
ETS Te TSE EES 2
2/98/88 /8/8 1/8 2) I2 18 12 8
GS/s /Fl/F IS jE ls I5/* \e 8 a
= & 2/8 \/2/2 FE 6/9
zI2 = a #5
5
Capability 2 ¢I8
[Administ
ration:] I ViewUsers y jy jy ly jy ly y jy ly
[Engineer
q AdjustScreen Y Y y ly jy jy ly
[Engineer
F] InstallPINPad Y Y y ly ly
[Engineer
] ModemRecovery Y
[Engineer
F] NetworkResilience Y Y y ly jy jy
[Engineer
"I RefreshRateBoard y ly Jy Y y ly jy
[Engineer
FI SetRateBoard Y Y y ly jy
[Engineer
] TestBarCode Y Iy Y y jy jy Y
[Engineer
5] TestCardReader Y Y y ly jy Y
[Engineer
‘J TestNetwork Y Iy Y y ly jy jy ly
[Engineer
] TestPINPad Y Y y ly jy Y
[Engineer
J] TestPrinter Y \y Y y ly jy Y
[Engineer
q TestReader yY Iy Y y ly jy Y
[Engineer
1 TestSlip yY Iy Y y ly ly Y
[Engineer
I TestTally Y Iy Y y ly jy jy ly
[Other] I FinalAccount Y
Other] I Logout yY [y Iy ly jy /yY lY IY ly IY I¥
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
itgroup
POL-BSFF-0100992_0265
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 262 of 265
EUS SEE Ee Se DEE SE
GS/s /Fl/F IS jE ls I5/* \e 8 a
= & 2/8 \/2/2 FE 6/9
28 a . eS
is
Capability 2 ¢I8
{Other,] I Memos Y jy {y ly jy IyY Y IY Y
[Other] I Nodeinformation Y {fy fy ly jy Iy Y ly jy IY I¥
[Other] I OfficeBalancing Y Iy IY Y Y IY
POIDUsersTrainingD
Other] I etails y jy jy Jy jy Jy Jy Jy Jy Jy Jy Y
[Other] I PostLoginChecks Y fy fy ly ly fy Y
PostLoginOutstTxnAc
[Other;] I knowledments y ly jy
PostLoginOutstTxnC
{Other;] I orrections y ly
[Other] I StockBalancing Y [y Iy IY Y Y IY
{Other] I TrainingReset Y
[Reports; I AllBranchUsersCurric
] ulaReport Y
TReports;
] BranchTP yY jy jy y ly y Iy
TReports;
] CtrDaily y jy jy ly Y y Iy
TReports;
] CtrWeekly y jy jy ly Y y Iy
TReports;
] EventLog y jy jy ly Jy Jy y jy ly
TReports;
] MigrationReport Y Iy Y y Iy
TReports;
] OfficeDaily y iy jy jy jy ly y Iy
TReports;
] OfficeWeekly yY jy jy y ly y Iy
[Reports;
] OutstTxnCorr y jy jy Y y Iy
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
Zitg roup
POL-BSFF-0100992_0266
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 263 of 265
EUS SEE Ee Se DEE SE
2 I5 2/? Ia Sie
Capability 3 5/3
oe cI 2/8
[Reports;
] ProcTxnCorr yY jy jy Y y Iy
[Reports;
] Productinformation IY Iy IY IY Iy IY y Iy
[Reports;
] Reports y iy jy jy jy ly y jy ly
TReports;
] TradingStatement yY jy jy Y y Iy
[Reports;
] TxnLog y iy jy jy jy ly y Iy
TReports;
] UserEvents y jy jy y ly y ly ly
TReports;
] UserHistory y jy jy y ly y jy ly
[Transact
ional] Housekeeping yY jy jy Y y Iy
[Transact
ional] PouchCollection y jy jy ly y Iy y Iy
[Transact
ional] PouchDelivery yY jy jy ly y Iy y Iy
[Transact
ional] I Transactions yY jy jy ly Y y Iy
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
Zitg roup
POL-BSFF-0100992_0267
181024SR1935
01 February 2019
POL00262929
POL00262929
Page 264 of 265
Appendix C
USERID's found in Horizon Privileges Access Logs
SYSTEM
OPS$ORACLE
OPS$BRDB
STRADMIN
BRDBOMDB
TRBALUSER
LVBALUSER
OPS$BRDBBLV1
OPS$SENGLO1
OPS$EASHFO1
OPS$MWRIGO1
OPS$GSIMPO01
OPS$COBENO1
RDDS
OPS$JBALLO1
OPS$GMAXWO1
OPS$DAVENO1
OPS$AKEILO1
DBSNMP
OPS$ACHAMO1
OPS$PSTEWO1
OPS$CCARDO1
SYS
OPS$JSIMPO1
OMDBUSER
OPS$DSEDD01
LVBALUSER1
EMDB_SUP
OPS$PCARRO1
OPS$KMILLO1
OPS$DALLEO1
OPS$SPARKO1
OPS$SSURXO1
OPS$LKIANO1
OPS$MCROSO1
STRMADMIN
OPS$BRDBTR
OPS$JHARRO1
QLPSTRADMIN
OPS$CTURRO1
OPS$JCHARO1
OPS$VRAMAO1
OPS$BRSSBTH1
LVAGENTUSER1
COBENO1
SSC_TOOLS
OUTLN
OPS$BRDBBLV4
OPS$WCALVO1
Tws
ORAEXCPLV
SQUIRLESCAN
SQUIRRELSCAN
OPS$PSIMPO1
TRBALUSER1
TRBALUSER2
TRBALUSER3
TRBALUSER4
LVBALUSER2
LVBALUSER3
LVBALUSER4
OPS$WBRAGO1
LVAGENTUSER4
OPS$AWOODON1
OPS$RGELDO1
OPS$MOGGBRDB
OPS$OGGADMIN
XXXX
USREIDALIAS
USERIDALIAS
EXI
OPS$NMCKEO1
OPS$AGIBSO1
OPS$BPEACO1
OPS$VKONAO1
OPS$BRDBBTR1
OPS$ABESTO1
OPS$SNELLO2
OPS$SSATTO1
PK
DAVENO1
Prepared by: Jason Coyne
Occupation: Partner
Specialist Field: IT Systems
On the Instructions of: Freeths LLP
POL-BSFF-0100992_0268
POL00262929
POL00262929
181024SR1935 01 February 2019 Page 265 of 265
Appendix D
Amendments to First Expert Report 180503R1935 01-01 Served 16 October
2018
7.2 Page 9 paragraph 1.31 of my previous report should have read “My
name is Jason Coyne and I am a Partner at IT Group UK Limited.”
7.3 Page 84 footnote 150 of my previous report referenced POL-0512874
which should have read POL-0152874.
7.4 Page 92 paragraph 5.190 Footnotes 169 and 171 are mis-referenced.
Footnote 169 ‘GCSimpson2242L.html’ should be replaced by
maxwellg5213L.html*4_ and footnote 171 should be disregarded in
relation to this paragraph as it relates to APS transactions and not JSN
duplication.
7.5 Page 67 paragraph 5.99 “It is common ground between the experts that
that each time there is a change there is a potential to introduce new
bugs/errors/defects.” Should read as “...experts that each time there
is a change...”
7.6 Page 131 paragraph 8.13 "...the cause of an issues that arise at anything
beyond counter level (and possibly even those that arise at counter
level)." Should read as “the cause of any issues”.
7.7 Page 133 paragraph 8.22 “In conclusion, Post Office had access to far
more comprehensive information relation to the Horizon system. If an
error occurred beyond counter level, Subpostmasters would need to rely
on Post Office to identify and resolve the issue. If that issue or its was
not properly identified for any reason, then the Subpostmaster would
be at risk of being liable for a Transaction Correction.” Should read as
“information in relation” and “issue or its impact was”.
321 KEL Maxwellg5213L, 30 June 2010 last updated 21 March 2011 {POL-0038402}
Prepared by: Jason Coyne
fe)
Occupation: Partner : .
Specialist Field: IT Systems : It FC
[e)
On the Instructions of: Freeths LLP
POL-BSFF-0100992_0269