Livia
CASE NUMBER 1HQ16X01238
ALAN BATES & OTHERS [CLAIMANT]
Vv
POST OFFICE LIMITED [DEFENDANT]
THIRD JOINT STATEMENT OF
JASON COYNE
AND
DR ROBERT WORDEN
01 MARCH 2019
POL00026918
POL00026918
evila
POL00026918
POL00026918
Introduction
This third Joint Statement sets out further areas of agreement between the Experts. The structure of the document captures expert agreements and
disagreements on Horizon Issues 3, 4, 5, 6, 7 and 8.
Issues 10, 11, 12 and 13 are omitted from this report and will be dealt with in a fourth Joint Statement as additional Defendant Witness Statements of Mr
Godeseth and Mr Parker were only received after business hours 28" February 2019 and the experts need more time to consider the subsequent evidence in
relation to those Issues.
We understand that the court wishes to see more emphasis on agreements between the experts in these joint statements. The experts have worked hard to find
agreements where possible, and there are some important areas of agreement which are stated in this statement and in the previous joint statement. However,
there are also deep disagreements between the experts.
Jason Coyne — In this Joint Statement I have sought to document my agreement or disagreement in respect of the 15 Horizon Issues where I can or cannot
reach agreement with Dr Worden. It is my understanding that this is not a responsive report to Dr Worden’s supplemental report, therefore I have not added.
any further comments, criticisms or observations in respect of any responsive report in this Joint Statement.
Expert Agreements and Disagreements by Horizon Issue
Horizon Issue 3 — To what extent and in what respects is the Horizon System “robust” and extremely unlikely to be the cause of shortfalls in
branches?
Index: I Sub Topic. I Agreed/JC/” I Statement “I Coyne Refs .-- I Worden
3.1 Improving Agreed Irrespective of how you define the detail of robustness, in line with
Robustness most other large-scale computer systems, Horizon’s robustness has
generally improved.
From our experience of other computer systems, Horizon is relatively
robust. We agree that ‘robust’ does not mean infallible and therefore
Horizon has and will continue to suffer faults. Robustness limits the
impact of those faults and other adverse events.
2
e/v/La
Index
Sub Topic
‘Agreed/JC/
RW
Statement
“I Coyne Refs.
Worden
This increase in robustness has, in part, developed from Post Office
discovering bugs/errors and defects in live use and then applying fixes
and improving monitoring.
3.2
Agreed
Computer systems are considered more robust if access to the back-
end databases is restricted tightly.
3.3
‘Agreed
It was possible for some of the Horizon support staff working at
Fujitsu to modify the Horizon back-end branch database.
In 2012, Post Office’s auditors observed that there were inappropriate
system privileges assigned to the APPSUP role (which allowed
amendments to the BRDB).
5.206 (a)
3.4
RW
Fujitsu took steps to correct the inappropriate system privileges.
3.5
Agreed
Post Office does not consult the full audit data (unfiltered ARQ Data)
before deciding how to handle discrepancies and issuing Transaction
Corrections.
5.206 (c)
3.6
Horizon
imperfections —
likelihood of
shortfalls
Jc
More bugs/errors and defects have been shown to impact branch
accounts than the initial three acknowledged by Post Office.
5.206 (d)
3.7
Horizon.
Imperfections
Agreed
Peaks show that some defects have lain undetected in Horizon for
extended periods without being diagnosed and fixed.
5.206 (e)
3.8
Horizon Changes
Agreed
During the life of Horizon there have been 19,842 changes made to it
via the Fujitsu/Post Office release mechanism.
3.9
Horizon Changes
Agreed
It is common modern IT development practice to make frequent
incremental builds and releases of software.
3.10
Changes
Agreed
Specific release note detail has not been provided. Of the 19,842
changes, we would expect that many were minor changes. It is likely
that others contained changes to improve the system or to fix bugs and
defects.
POL00026918
POL00026918
{02/4/177}
{D2/4/177}
{D2/4/177}
{D2/4/177}
viviLa
Index*
Sub Topic
-I "Worden -
‘I Refs
Countermeasures
The effectiveness of various countermeasures changed throughout the
life of Horizon.
3.12
Agreed
Countermeasures are basic elements of practical IT system design
3.13
Countermeasures
Agreed
Countermeasures work by limiting the impact of Horizon bugs/error
and defects on branch accounts.
Countermeasures do not always eliminate the effects of adverse events
(they are not perfect) but they are often effective in the area where
they are deployed; that is why they have become basic elements of
practical IT systems design.
3.14
Countermeasures
RW
Countermeasures work.
Thave assessed more than the design aspirations of the
countermeasures. I have assessed how the countermeasures were
tested; and I have illustrated in a large number of examples, mainly
through KELs and Peaks, how they worked in practice.
$1.4; $5.18;
$5.211- $5.217
$6.1
App.A.
Extent
Agreed
It is difficult to measure the extent of the robustness of Horizon, apart
from how it might limit the extent of impact on branch accounts, as in
Issue 1.
$5.5, $5.16
Improving
Robustness
Agreed
There are indications that in its first year of operation, and in the first
year after the introduction of Horizon Online, the system suffered
from more problems than in other years. One might expect a higher
level of problems in these early periods.
The extent to which these problems were serious, or evaded
countermeasures, or caused discrepancies in branch accounts, is not
agreed.
RW
The main way in which the experts have assessed the extent of
robustness of Horizon is to ask to what extent failures of its robustness
impacted branch accounts. I have addressed this in Horizon issue 1. In
answering this question, fluctuations over the years are of less
importance than the sum over all years, if the sum over all years is
POL00026918
POL00026918
{D2/4/8}
{D2/4/126}
{D2/4/179-180}
{D3/6/36}
{D3/7/2}
{D2/4/121}
{D2/4/125}
Index
Sub Topic
Agreed /JC/
RW.
Statement —
Coyne Refs
Worden
Refs
small. I have found the sum of impacts on branch accounts over all
rears to be very small.
Agreed
The users of any IT system play a role in assuring its robustness. The
designers of a system should not make unrealistic assumptions about
the users of the system. Unrealistic assumptions would lead to
inappropriate design, making the system less usable.
$5.19
391, 403.3
Jc
Mr Coyne cites the following KELs which demonstrate flaws within
the recovery procedure:
dsed4733R
obenge5933K.
achal941L
surs1147Q
3.20
Countermeasures
Agreed
As Horizon has changed throughout its lifetime, the existence and
effectiveness of any countermeasures has too. To have considered the
time dependence of all robustness countermeasures over 20 years,
would have made the expert reports impossibly lengthy. There was not
the time to do so.
$5.71
3.21
Countermeasures
RW
‘Recoverable’ transactions do not arise from system faults in Horizon.
Mr Coyne has conflated financial transactions (which may be
recoverable transactions) with database transactions.
$5.105 - 107
3.22
Countermeasures
‘Agreed
Many software bugs can have the same effects as a user error (as
illustrated, for instance, by the Dalmellington bug, which produced a
remming error).
3.23
Countermeasures
RW
I base my opinion that Horizon is a tightly-run ship on the high quality
of documentation, design, review, and testing evident in many
documents I have read; on the Ernst & Young Service audits of
Fujitsu, which found a high level of controls to be effectively
implemented; on Fujitsu’s CMMI accreditation; and on the high
quality and effectiveness of problem analysis and problem solving
shown in KELs and Peaks.
5.147
3.24
Risk assessment
RW
There is no clear distinction between prospective and retrospective
risk assessment. For many projects, there are large risks associated
$5.16
‘eivia
5
POL00026918
POL00026918
{D2/4/126}
{03/1/99}
{D3/1/104}
{F/1106}
{F/757}
{F/960}
{F/1433}
{D2/4/144}
{D2/4/153}
{D2/4/163}
{D2/4/125}
g/b/La
Index
‘SubTopic
with existing software which must be integrated with the new solution;
these risks must be assessed retrospectively. Even without existing
software risks, estimation of risks often depends on historic data.
Common simple mathematics are employed retrospectively and
prospectively.
3.25
Likelihood of
shortfalls
Horizon bugs, errors, and defects, along with mistakes made by
Fujitsu/Post Office employees and branch user errors were all likely
causes of shortfalls. Therefore, it would be incorrect to say that
Horizon was extremely unlikely to be the cause of shortfalls.
POL00026918
POL00026918
Livia
Horizon Issue 4 — To what extent has there been potential for errors in data recorded within Horizon to arise in (a) data entry, (b) transfer or (c)
processing of data in Horizon.
Index
Sub Topic
[Agreed ICT
RW_
‘Statement SEENE
Coyne Refs
‘Worden
Refs
4.1
Extent
Agreed
Bugs, errors and defects identified in relation to Horizon Issue 1 are
often relevant to Issue 4 in that they are ultimately errors arising from
the processing of data in Horizon.
Coyne
Supplemental
paras 3.147 —
3.219
4.2
Agreed
There is evidence within the Peaks and KELs of bugs/errors/defects
within Horizon arising from parts (a), (b) and (c) of this issue that
occurred without causing financial discrepancies as well as some that
occurred causing financial discrepancies.
43
Reference Data
Agreed
Reference data is critical to the operation of Horizon and errors in
reference data have led to discrepancies in branch accounts.
Parker2 Roll2
Various KELs
and Peaks
44
Reference Data
Agreed
Of the bugs which in the experts’ opinion had the potential to produce
discrepancies in branch accounts there may be some involvement of
reference data in Bureau Discrepancies, Bureau de Change, Wrong
branch customer change displayed, Lyca top-up, and Drop and Go.
(rows 14, 23, 24, 25, and 28 of the bugs table in the second expert joint
statement)
It is notable that these bugs all concerned specific products (arising
from the reference data defining those products).
So, while reference data bugs may be a significant proportion of the
bugs with financial impact, Once discovered, they could be quickly
fixed (by a change to the reference data) once the bug is correctly
identified.
JS2
45
JC
The full extent of potential for errors in data recorded within horizon
arising from a), b) and c) above has been difficult to measure since not
all errors are known. However, Peaks and KELs illustrate (by their
existence) that errors in data recorded did occur, some with financial
impact, some not.
POL00026918
POL00026918
{D2/4/53-69}
{E2/12}
{E1/10}
{D1/2}
s/v/La
Horizon Issue 5 — How, if at all, does the Horizon system itself compare transaction data recorded by Horizon against transaction data from sources
outside of Horizon?
Index
Sub:Topic
I -Agreed/JC/.. =I
Coyne Refs
“Worden
Refs:
5.1
Reconciliation
Agreed
Reconciliation between transactions recorded on Horizon and
transactions recorded by Post Office’s clients is largely automated.
Detected discrepancies were subject to manual corrective fixes and/or
the issue of Transaction Corrections/Error Notices to the
Subpostmasters.
5.2
Back Office
Accounting
Jc
Post Office back office accounting processes were relevant to branch
account accuracy. i.e., Product and Branch Accounting processes and
actions in respect of discrepancy investigations and issuing of Cash in
Pouches, TCs, Foreign Currency etc.
5.3
Reconciliation
Agreed
The adequacy of Post Office back office processes to prevent
discrepancies in branch accounts can be measured by the quality of the
TC process. This quality includes:
© The processes of consideration of available data
© The level of errors observed in the process
¢ The level of complaints or disputes raised following a TC
¢ The level of upheld complaints following a TC
e The level of financial impact of erroneous TCs
5.4
Third Party Data
Agreed
Errors in third-party data have led to discrepancies in branch accounts,
through erroneous TCs being issued on Subpostmasters.
5.Sa
Third Party Data
Agreed
PO does not control the level of errors made by its third-party client
organisations (which may lead to errors in TCs), or the delays in their
processes (which may lead to delays in TCs).
5.5b
Third Party Data
Agreed
PO can and should ensure, by careful investigation of disputed TCs,
that only a small proportion of errors by PO clients lead to losses for
8
POL00026918
POL00026918
6/b/La
upheld) illustrates that this process worked well.
Index I Sub Topic. Agreed /JC/ Statement : Coyne Refs I Worden
Subpostmasters, provided that the Subpostmasters are in good control
of their branches and have the required information available.
5.6 I Third Party Data I RW The figure quoted by Mr Coyne (77% of disputed Santander TCs
Horizon Issue 6 - To what extent did measures and/or controls that existed in Horizon prevent, detect, identify, report or reduce to an extremely low
level the risk of the following: a. data entry errors; b. data packet or system level errors (including data processing, effecting, and recording the
same); c. a failure to detect, correct and remedy software coding errors or bugs; d. errors in the transmission, replication and storage of transaction
record data; and e. the data stored in the central data centre not being an accurate record of transactions entered on branch terminals?
Index I Sub Topic Agreed/ I Statement Coyne Refs _ Worden
6.1 Measures and Agreed It is agreed that there are many measures and controls within Horizon that I Reconciliation
Controls existed to prevent, detect, identify report or reduce the risk of varying Processes, report
errors. sets...
6.2 Failures of Measures and controls that existed to reduce the risk of “c. a failure to Bugs/errors/defects
Measures and JC detect, correct and remedy software coding errors or bugs” were limited. I resident within the
Controls system for
elongated periods
of time and were
only identified in
the event of
Subpostmaster
dispute.
Dalmellington,
Suspense Account
6.3 Jc Evidence suggests that bugs/errors and defects were sometimes dealt with I Coyne 1* report
on a cost/benefit basis, therefore risks of errors arising was not reduced as I para 5.161, paras
far as ‘possible’. 6.1-6.3
POL-0219191
POL00026918
POL00026918
{D2/1/97-98}
{D2/1/109}
{F/1697}
OL/v/LG
6.4
JC
Evidence shows that Post Office have awaited the Subpostmaster
reporting discrepancies, even when it was aware of potential issues that
might impact branch accounts therefore risks of errors arising was not
reduced as far as ‘possible’.
POL-0449089
Closed Problems
tab, cell F99},
6.5
RW
Because the countermeasures worked well, and Horizon was and is a
robust system, the measures and control in Issue 6 worked well, and
harmful effects were reduced to an extremely low level.
Horizon Issue 7 - Were Post Office and/or Fujitsu able to access transaction data recorded by Horizon remotely (i.e. not from within a branch)?
Index I-Sub Topic “Agreed / Statement =~ SEP OSTA Ea Sa hee Coyne Refs. I Worden Refs
TA Remote Access _I Agreed Fujitsu could access all transaction data recorded by Horizon. Parker 2
72 Remote Access I Agreed Both Post Office and Fujitsu can read data remotely, and FJ needs remote Supp 5.415 Exp 103
access for support purposes
Horizon Issue 8 - What transaction data and reporting functions were available through Horizon to Post Office for identifying the occurrence of
alleged shortfalls and the causes of alleged shortfalls in branches, including whether they were caused by bugs, errors and/or defects in the Horizon
system?
Index I Sub Topic I Agreed/) I Statement: =~ :I Coyne Refs Worden Refs
8.1 Facilities for Agreed Post Office had access to data which would not have been available to
Post Office Subpostmasters.
8.2 Facilities for Agreed The descriptions of facilities for PO in the two expert reports are consistent I 8.1—8.9 1081 - 1088
Post Office and can be taken together as a description of those facilities.
8.3 Identification of I Agreed Post Office were reliant upon Fujitsu for diagnosis of whether the
bugs / errors / occurrence of shortfalls was caused by bugs/errors or defects within the
defects Horizon system.
10
POL00026918
POL00026918
{F/1807}
{€2/12}
{D2/4/238}
{03/1/24}
{D2/1/140-143}
{D3/1/238-240}
POL00026918
POL00026918
Approved for service 15* March 2019
Jason Coyne
af
1/4/11