Claim Nos. HQ16X01238, HQ17X02637 & HQI7X04248
IN THE HIGH COURT OF JUSTICE
QUEEN’S BENCH DIVISION
The Post Office Group Litigation
Before the Honourable Mr Justice Fraser
BETWEEN:
ALAN BATES & OTHERS
Claimants
—and —
POST OFFICE LIMITED
Defendant
POST OFFICE’S WRITTEN CLOSING SUBMISSIONS:
HORIZON ISSUES TRIAL
A. INTRODUCTIO!
Al. — SUMMARY...
B. STRUCTURE OF THESE SUBMISSIONS ...
C. PROCEDURAL BACKGROUN
D. THE HORIZON ISSUES......
D1. INTRODUCTION
D2. —_ EXTENT, LIKELIHOOD AND RISK.
D3. Cs’ APPARENT ATTEMPTS TO DILUTE THE HORIZ
D4. I CATEGORISING THE HORIZON ISSUES
D5. THE ROBUSTNESS ISSUES..
ISSUE 3 (FIRST HALF) AND 6(C): GENERAL ROBUSTNESS OF HORIZON. ..
ISSUE 1(A) AND ISSUE 3 (SECOND HALF): IMPACT ON BRANCH ACCOUNTS
D6. THE REMOTE ACCESS ISSUES..
POL00026925
POL00026925
Al8/1
D7.
El.
E2.
E3.
E4,
ES.
I>
Fl.
F2.
F3.
F4.
POL00026925
POL00026925
ISSUE 7
ISSUE 10 .
ISSUE 11.
ISSUE 12...
ISSUE 13 .
THE OPERATIONAL ISSUE:
ISSUE 2...
ISSUE 5...
ISSUE 8 ...
ISSUE 9
ISSUE 14 .
ISSUE 15 .
FUJITSU WITNESSE:
MR ROLL’S EVIDENCE......
THE IMPORTANCE OF MR ROLL’:
MR ROLL’S RECOLLECTION . .
MR ROLL’S POSITION IN THE SSC AND EXPERIENCE OF ISSUES
ACCEPTANCE OF MR PARKER’S EVIDENCE......
FUNCTIONALITY IN AND FEATURES OF HORIZON.
HARDWARE FAILURES..
REMOTE ACCESS... -
MR GODESETH’S EV IDENCE
MR PARKER’S EVIDENCE.
MR DUNK’S EVIDENCE..
MR MEMBERY’S EVIDENCE...
S EVIDENCE TO Cs’ CASE..
THE EXPERT EVIDENCE .....
INTRODUCTION AND OUTLINE STRUCTURE OF THIS SECTION
OVERALL SUBMISSIONS ON DR WORDEN’S EVIDENCE
OVERALL SUBMISSIONS ON MR COYNE’S EVIDENCE .....4+00
MR COYNE’S “DEEP FOCUS” ON PROBLEMS AND LACK OF BALANCE
MR COYNE’S APPLICATION OF THE WRONG TEST WHEN ADDRESSING THE HORIZON
ISSUES.
MR COYNE’S INCONSISTENT APPROACH TO QUESTIONS OF EXTENT... -
MR COYNE’S FAILURE TO SET OUT HIS TRUE AND FULL OPINIONS IN HIS REPORTS.. 111
MR COYNE’S EVASIVE ANSWERS AND RELUCTANCE TO CONCEDE POINTS...
MR COYNE’S CONFUSING POSITION ON THE 29 BUGS IN THE BUG TABLE.
THE CHANGED FACTUAL BASIS OF MR COYNE’S EVIDENCE....
JUDICIAL CRITICISM OF MR COYNE’S APPROACH ..
THE SEARCHES CARRIED OUT BY MR COYNE AND HIS TEAM
i
Al8/2
FS.
Fo.
F7.
F8.
FINANCIAL IMPACT ..
F9.
FINANCIAL IMPACTS..
F10.
Ful.
MANAGEMENT INFORMATION SYSTEMS. ..sscssscssessssssssesseesssesssessecsssessnescerssseseseseseseee
POL00026925
POL00026925
THE EXPERTS’ APPROACHES TO LATE EVIDENCE
FUSITSU’S SUPPORT ROLE AND ITS IMPORTANCE TO THE HORIZON ISSUES
HORIZON AS A “TIGHTLY RUN SHIP”.
ROBUSTNESS ... woseeee
THE CENTRAL IMPORTANCE OF ROBUSTNESS TO THE HORIZON ISSUES
THE EXPERTS’ OVERALL CONCLUSIONS ON ROBUSTNESS ..
‘THE IMPORTANCE OF COUNTERMEASURES AND THE CONCEPT OF A LASTING
THE ANALYSIS
FUJITSU’S ROLE IN IDENTIFYING AND ADDRESSING BU!
THE COST/ BENEFIT ALLEGATION...
THE ROLE OF TCs IN ROBUSTNESS...
‘THE EXPERTS’ APPROACHES TO ASSESSING THE EXTENT OF BUGS AND THEIR
THE EXPERTS’ DIFFERING APPROACHES TO QUESTIONS OF EXTENT
DR WORDEN’S NUMERICAL APPROACH TO ASSESSING EXTENT
MR COYNE’S APPROACH IN HIS REPORTS — AVOIDING “EXTENT” QUESTIONS.......
MR COYNE’S APPROACH IN HIS REPORTS — INVITING INFERENCES OF MANY OR
MAJOR PROBLEMS WITH NO PROPER BASIS IN THE EVIDENCE TO WHICH HE
REFERRED..
MR COYNE’S EVIDENCE AT TRIAL — IMPORTANT CONCESSIONS ON EXTENT
DATA ENTRY ERRORS (MIS-KEYING)..
RELIABILITY OF TRANSACTION DATA IN HORIZON AND IN POST OFFICE’S
POST OFFICE’S RELIANCE ON MIS....
DR WORDEN’S EVIDENCE IN RELATION TO THE CREDENCE SYSTEM .....
MR COYNE’S EVIDENCE IN RELATION ARQ REQUESTS AND THE AUDIT STORE.
DR WORDEN’S EVIDENCE IN RELATION ARQ REQUESTS AND THE AUDIT STORE
F12. THE 29 BUGS LISTED IN JS2....ccsseessesseseseee
THE 29 BUGS...
THE BUG APPENDIX............ .
FINDING AND ANALYSING BUGS.
THE BUG TABLE
F13. MR COYNE’S VIEWS ON BRANCH IMPAC’
LASTING VERSUS TRANSIENT IMPACT.....
AUTOMATIC REPORTING ..
F14. MR COYNE’S VIEW ON BUGS WITH LASTING IMPACT — A CHANGING LANDSCAPE. 229
F15. I ALLEGED BUGS WHICH ARE NOT BUGS AT ALL..
BUG 8: RECOVERY ISSUES... os
BUG 13: WITHDRAWN STOCK DISCREPANCIES ..
BUG 15: PHANTOM TRANSACTIONS
AI8/3
FI6.
FI7.
FI8.
FI9.
F20.
F21.
F22.
F23.
F24.
F25.
POL00026925
POL00026925
BUG 16: RECONCILIATION ISSUES... 234
BUG 17: BRANCH CUSTOMER DISCREPANCIE: 234
BUG 20: RECOVERY FAILURES 234
BUG 23: BUREAU DE CHANGE 234
BUG 29: NETWORK BANKING. 234
BUGS WITH NO IMPACT ON BRANCH ACCOUD 235
BUG 11: GIROBANK.... on 235
BUG 21: TRANSACTION CORRECTIONS. ee sesseeeeee wee 235
BUG 22: BUGS/ERRORS DEFECTS INTRODUCED BY PREVIOUSLY APPLIED PEAK FIXES
BUGS WITH TRANSIENT IMPACT. +236
BUG 2: CALLENDAR SQUARE .. 236
BUG 4: DALMELLINGTON....
BUG 5: REMMING IN...
BUG 6: REMMING OUT
BUG 7: LOCAL SUSPENSE .
BUG 9: REVERSALS....
BUG 12: COUNTER REPLACEMEN
BUG 19: POST & GO/TA DISCREPANCIES .
BUG 25: LYCA TOP UP ..
BUGS WITH LASTING IMPACT.
SUMMARY...
LASTING DOES NOT MEAN UNRESOLVED...
BUG 1: RECEIPTS AND PAYMENTS MISMATCH
BUG 3: SUSPENSE ACCOUNT ...
BUG 10: DATA TREE BUILD.............
BuG 14: BUREAU DISCREPANCIES
BUG 18; CONCURRENT LOGINS
BUG 24: WRONG BRANCH CUSTOMER CHANGE
BUG 26: TPSC250
BUG 27: TPS.. oe
BUG 28: DROP AND GO.
PRE-ANNOUNCEMENTS INTERFACE DEFECT.
CONCLUSION ON THE 29 ALLEGED BUGS .
EXPERT EVIDENCE ON REMOTE ACCESS.
SCOPE AND FOCUS OF T EMOTE ACCESS ISSUES ...esssseseseeeee
OVERALL SCALE AND IMPORTANCE OF REMOTE ACCESS.
SPECIFIC FORMS OF REMOTE ACCESS...
LEGACY HORIZON...
HORIZON ONLINE...
MR COYNE’S RELIANCE ON E&Y AUDITS.
PERMISSION CONTROLS AND LOGS (HORIZO!
PRIVILEGED USER CONTROLS...
OBTAINING PERMISSION FOR CHANGES FROM POST OFFICE
AI8/4
G
HL
HI.
H2.
L
i.
14,
15,
I=
K.
POL00026925
POL00026925
FUJITSU INTERNAL SUPERVISION OF CHANGES — “FOUR EYES” ....
POST OFFICE’S OTHER WITNESSES.
MRs VAN DEN BOGERD..........
Ms PHILLIPS ..
Mrs MATHER
MR SMITH .....
MR JOHNSON...
CS’ OTHER WITNESSES ......
CLAIMANT-SPECIFIC EVIDENCE ..
MR LATIF...
MR TANK
MESSRS ANUP AND AAKASH PATNY .
MRS BURKE...
MR HENDERSON’S EVIDENCE....
DISCLOSURE...
BACKGROUND...
MODEL C DISCLOSURE
EXPERT-LED REQUESTS FOR DISCLOSURE
THE FIFTH CMC ORDER
BURDEN ON POST OFFICE
THE COURT’S INTERVENTIONS ON DISCLOSURE
KELs.....
MSCs & OCPs.
RELEASE NOTES
ROYAL MAIL DISCLOSURE ..
Cs’ OWN DISCLOSURE ....
MR TANK.
MR LATIF
REQUESTS FOR DISCLOSURE
MISCELLANEOUS..
“SHADOW EXPERTS”...
APPENDICES...
AI8/5
Kl.
“COST/BENEFIT” PEAKS...
K2.
THE BUG APPENDIX...
BUG I:
BUG 2:
BUG 3:
BUG 4:
BUGS:
BUG 6:
BUG7:
BUG 8:
APPENDIX 1: POST OFFICE ANALYSIS OF MR COYNE’S ADDITIONAL
APPENDIX 2: THE 29 BUGS IN JS2
RECEIPTS & PAYME!
SUMMARY: ...
NATURE OF THE ISSUE .......
DETECTION AND RESOLUTION.
RELEVANT DISCUSSIONS DURING TRIAL,
CONCLUSION ........
CALLENDAR SQUARE
SUMMARY .....
NATURE OF THE ISSUE
DETECTION AND RESOLUTION...
RELEVANT DISCUSSIONS DURING TRIAL,
SUSPENSE ACCOUNT BUG...
SUMMARY .....
DETECTION AND RESOLUTION.
DALMELLINGTON...
SUMMARY ...... seceeeenesneee o
BACKGROUND: OUTREACH BRANCHES
NATURE OF THE ISSUE...
DETECTION AND RESOLUTION.
RELEVANT DISCUSSIONS FROM TRIAL
THE “UNKNOWN OUTCOMES” .
REMMING IN...
SUMMARY ....
NATURE OF THIS ISSUE
CONCLUSION ON BUG 5:
REMMING OUT.
SUMMARY .......0.0.
NATURE OF THIS ISSUE
ISSUE 1
ISSUE 2... .
CONCLUSION ON BUG 6: REMMING OUT.
LOCAL SUSPENSE ISSUE..
SUMMARY ....
NATURE OF THIS ISSUE
DETECTION AND RESOLUTION
CONCLUSION...
RECOVERY ISSUES..
THE KEY DOCUMENTS.
SUMMARY
6
POL00026925
POL00026925
AI8/6
POL00026925
POL00026925
4. MR COYNE STATES THAT BUG 9: REVERSALS IS A BUG WITH LASTING FINANCIAL.
IMPACT. POST OFFICE SUBMITS THAT ANY DISCREPANCY WOULD NOT BE LASTING. ......437
NATURE OF THIS ISSUE ...
NATURE OF THIS ISSUE ...
RESOLUTION...
BRANCH IMPACT.
CONCLUSION ..0...csssesceeseeeeeee
BUG 10: DATA TREE BUILD ISSUES
THE KEY DOCUMENTS:
SUMMARY ....
ISSUE 1.
ISSUE 2.
BUG 11: GIROBANK DISCREPANCIES...
SUMMARY: ..
NATURE OF THIS ISSUE
ISSUE 1.
ISSUE 2......
ISSUE 3.
ISSUE 4.
ISSUE 5
ISSUE 6
CONCLUSION
BUG 12: COUNTER REPLACEMENT CAUSING ONE SIDED TRANSACTIONS ....
SUMMARY ...
NATURE OF THIS ISSUE
RESOLUTION...
OTHER PEAKS ASSOCIATED WITH JBALLANTYNES328R
BRANCH IMPACT. sonveeeeeeenseee soo
INSERTING ITEMS WITHIN THE HORIZON MESSAGESTORI
BUG 13 — WITHDRAWN STOCK DISCREPANCIES ....
SUMMARY oo... cccccceccssseeceeeceeeseceeseeensesnenee
PosT OFFICE’S WITHDRAWAL OF PRODUCTS
NATURE OF THIS ISSUE ...
RESOLUTION...
BRANCH IMPACT.
BUG 14 — BUREAU DISCREPANCIES......
THE KEY DOCUMENTS ..
SUMMARY
ISSUE 1 .
ISSUE 2.
BUG 15: PHANTOM TRANSACTIONS
AI8/7
POL00026925
POL00026925
SUMMARY ....
NATURE OF THIS ISSUE
ISSUE 1 .....
ISSUE 2.
ISSUE 3.
CONCLUSION ON BUG 15: PHANTOM TRANSACTIONS .
BUG 16; RECONCILIATION ISSUES...
SUMMARY
NATURE OF THIS ISSUE
ISSUE 1 .....
ISSUE 2.
ISSUE 3 .
BUG 17: BRANCH CUSTOMER DISCREPANCIES..
SUMMARY ....
NATURE OF THIS ISSUE
DETECTION AND RESOLUTION .
CONCLUSION
BUG 18: CONCURRENT LOGINS
SUMMARY ....
NATURE OF THIS ISSUE ...........
COYNE’S CHANGE IN POSITION
ISSUE 1
Issuk 2
BUG 19: PosT & Go.
SUMMARY
NATURE OF THE ISSUE ........0000-00000
DETECTION AND RESOLUTION..
CONCLUSION ON BUG 19: POST & GO/TA DISCREPANCIES IN POLSAP.
BUG 20: RECOVERY FAILURES...
SUMMARY ....
NATURE OF THIS ISSUE ...... seseeeeeee
BUG 21: TRANSACTION CORRECTION ISSUE
SUMMARY ....
NATURE OF THIS ISSUE ..
ISSUE 1
ISSUE 2
ISSUE 3 .
CONCLUSION ON BUG 21
RANSACTION CORRECTION ISSUES
BUG 22: BUGS/ERRORS/DEFECTS INTRODUCED BY PREVIOUS APPLIED PEAK FIXES ........44..507
SUMMARY .... 507
NATURE OF THIS ISSUE
ISSUE 1
ISSUE 2......
ISSUE 3
Al8/8
POL00026925
POL00026925
CONCLUSION ON BUG 22: BUGS/ERRORS/DEFECTS INTRODUCED BY PREVIOUS APPLIED
PEAK FIXES...
BUG 23: BUREAU DE CHANG
SUMMARY ....
NATURE OF THIS ISSUE
ISSUE 1.....
ISSUE 2
IssUE
OTHER PEAKS REFERRED TO IN JS2
CONCLUSION ON BUG 23: BUREAU DE CHANGE,
BUG 24: WRONG BRANCH CUSTOMER CHANGE...
SUMMARY .....
NATURE OF THIS ISSUE
IMPACT
FIX APPLIED
BUG 25: LYCA TOP UP.
SUMMARY .....
NATURE OF THIS ISSUE
DETECTION AND RESOLUTION...
CONCLUSION ON BUG 25: LYCA TOP UP.
BUG 26: TPSC250 REPORT BUG..
SUMMARY
ISSUE I
ISSUE 2
ISSUE 3 ...
ISSUE 4...
ISSUE 5...
BUG 27: TPS BUG
SUMMARY
NATURE OF THIS ISSUE .....
DETECTION AND RESOLUTION.
CONCLUSION...
BUG 28: DROP AND Go Bu
SUMMARY .......
NATURE OF THIS ISSUE ......
DETECTION AND RESOLUTION.
CONCLUSION...
BUG 29: NETWORK BANKING BUG..
SUMMARY .....
ISSUE 2.....
CONCLUSION
APPENDIX 3: AUTOMATIC REPORTS.
INTRODUCTION.
NB102 EXCEPTION SUMMARY..
Al8/9
POL00026925
POL00026925
THE TRANSACTION PROCESSING SYSTEM ("TPS") REPORT SET ..
TPSC250 Host DETECTED TRANSACTION ERRORS REPORT
TPSC254 HARVESTER EXCEPTIONS.
TPSC257 POLFS INCOMPLETE SUMMARIES REPORT
TPSC256 HOST DETECTED CASH ACCOUNT LINES COMPARISON REPORT/ PROGRAM ......544
A/8/10
POL00026925
POL00026925
INTRODUCTION
Al.
Summary
The written evidence presented by Cs, particularly that contained in Mr Roll’s two
witness statements and Mr Coyne’s first and second reports (“Coyne 1” and “Coyne 2”)
made for ominous reading. On any view, they suggested that:
I
there were probably hundreds if not thousands of branch account affecting bugs in
Horizon, each of which might itself have had thousands of impacts; and
Fujitsu, casually, routinely and without following any proper procedures, accessed
SPMs’ branch accounts and more or less did what it wanted, when it wanted,
behind SPMs’ backs and without regard to their interests or concerns.
Following the Horizon Issues trial, the position now is very different:
2.1
2.2
2.3
It is agreed by the experts that the Horizon system is robust (comparing well with
other similar systems which have to be highly reliable) and that it has been robust
throughout all its life (albeit perhaps less robust in the early years of Legacy
Horizon and Horizon Online).
Nobody has identified a material design flaw in Horizon. The focus has been on
whether that design has been properly implemented and supported over 20 years
— hence the detailed consideration of the volume and nature of bugs.
Mr Coyne has worked very hard with his team to identify as many bugs as possible.
He thinks he has found 29 bugs which were or might have been branch account
affecting, and considers that there might have been up to 40 such bugs in total. He
maintains that these bugs might have had thousands of branch impacts. Post Office
challenges each of these points, but even if he were right on them, they
demonstrate that the extent of bugs in Horizon is nowhere near what Cs need it to
be. In order for Cs’ generic case to have even a material chance of success, tens of
AI8/11
POL00026925
POL00026925
thousands of bugs would be required, each with multiple impacts. The evidence
does not begin to support that scale of bugs or impacts.
2.4 The experts accept that Fujitsu were careful and conscientious and operated
thorough systems for recording, diagnosing and fixing occurrences of problems.
2.5 All forms of remote access are agreed, and it is clear that any remote access was
done rarely and carefully and generally in consultation with the SPM.
The evidence shows that, at a generic level at least, it is extremely unlikely that the
Horizon system will be the cause of an individual shortfall. In other words, when looking
at a set of accounts for a particular branch and a particular month, in the absence of
special features indicating that a bug affected that set of accounts or that some instance
of remote access was incorrectly carried out, it should be recognised that the chances of
Horizon or an instance of remote access having caused a shortfall in those accounts are
extremely low.
The central issue in this litigation is: was a particular SPM’s shortfall likely to have been
caused by a bug in Horizon? It is not: were there sometimes bugs in Horizon?; or were
SPMs sometimes frustrated by Post Office’s attitude?; or were helpdesk operatives less
well-informed, polite or reassuring than they might have been?
Cs’ approach is necessarily impressionistic. They invite the Court to draw wide
inferences from a few examples and from narrow classes of document and refrain from
rigorous analysis. They have still not shown any clear cut example of bugs in Horizon
that could have caused the shortfalls of around £19 million for which they claim to have
been falsely held responsible.
Cs’ case theory appears to be if it can be shown to be a possibility (however remote) that
some of their losses were caused by bugs in Horizon then the Court should conclude that
this was likely: in other words, they persist in equating possibility with probability.
Post Office accepts that it is possible that any given C’s losses (or some part of them)
were caused by a bug. But the evidence shows that, in the absence of special
circumstances, this is very unlikely and that it is overwhelmingly unlikely that any
AI8/12
10.
IL.
POL00026925
POL00026925
significant amount of the overall shortfall claimed was caused by the matters complained
of. Cs have not wrestled with the fundamental issue of likelihood in any way.
Contrary to the Court’s directions, Cs led evidence from specific claimants which can be
presumed to be the best examples available to them of how bugs have affected individual
SPMs. However, putting the point at its lowest, there are plausible explanations for all
the instances cited in this evidence that the problems asserted did not involve any fault
in Horizon. This evidence did not advance Cs’ case on the Horizon Issues in any way.
Mr Coyne’s approach took matters little further. In his reports, he did not seek to give a
balanced view of the Horizon system or Fujitsu’s activities: instead, his energies were
directed at finding evidence of bugs and of exceptions to Fujitsu’s standard practices.
Resisting any form of systematic analysis, he cast his net as widely as possible for stray
examples of instances, gathered over 20 years, of practices which he considered less than
perfect. The inferences he apparently invites the Court to draw from the arguments he
presents are unsupportable.
Mr Coyne’s reports were directed at establishing the admitted fact that it is possible for
there to have been branch-account affecting bugs. He had nothing to say about the
generic likelihood that SPMs’ shortfalls were actually caused by bugs in Horizon, still
less the extent to which any such bugs are likely to have caused shortfalls of the sort Cs
allege. He gives no sense of the extent or even scale of the bugs he has identified in the
operation of Horizon over 20 years. Such bugs are not put in their proper context, against
either the volume of transactions conducted by Horizon, the number of branches in the
Post Office network, the number of branch accounts that have been generated by Horizon
or the number and scale of the shortfalls that Cs wish to challenge. His reports show that
he has set out to prove a thesis — that Horizon is full of problems — rather than to present
a balanced view of the Horizon system.
Cs’ essential case is that Horizon is so unreliable that no store can be set by the branch
accounts it generates. It is striking that, notwithstanding all the problems Mr Coyne
purports to have found, his analysis does not begin to support this case. Horizon has
operated satisfactorily for some 20 years, and only a tiny fraction of the thousands of
SPMs who have operated over that time have joined this action. One of Cs’ own
AI8/13
13.
15.
witnesses, Mrs Angela Burke, gave evidence that in her many years of experience
working in post offices she had had only had one issue (dealt with below) which she had
not been able to understand.
This begs the obvious question: namely, how can Horizon have operated satisfactorily
so much of the time if it is as unreliable as Cs would have the Court believe? How can
Cs even say that Horizon is unreliable or lacking robustness? How could this be credibly
said to have caused the scale of losses alleged by Cs as a group?
What do Cs make of the evidence from someone such as Mrs Burke? How can they
coherently account for the fact that for the vast majority of the time, for the vast majority
of SPMs, Horizon works satisfactorily?
More fundamentally: what are Cs inviting the Court to do in this trial and on what basis?
Do they simply invite the Court to share a sense of outrage and hope that this sense will
row them home even though they have produced no evidence of any widespread failings
in Horizon and seek to assign no probability to the scale of bugs that their case requires?
Cs fail to wrestle with any of these questions. They cannot answer them in any
convincing way.
STRUCTURE OF THESE SUBMISSIONS
POL00026925
POL00026925
This is a trial of expert issues in relation to which expert evidence should have primacy.
To some extent, the expert evidence depends on factual foundations provided by the
Fujitsu witnesses (in particular, Mr Roll and Mr Parker and, to a lesser extent, Mr
Godeseth). However, it is right to point out that many facts have been established by the
experts. Most of these (e.g. the architecture of the Horizon system and the details of
Fujitsu’s operations and organisation) can now be seen to be largely uncontroversial.
A significant amount of factual witness evidence has been given which is of little or no
relevance to the Horizon Issues (for example, the claimant-specific evidence called by
Cs contrary to the Court’s directions) or which provides largely uncontroversial
AI8/14
20.
background with regard to Post Office’s processes and systems (for example, much of
the factual evidence called from Post Office employees).
Post Office proposes to arrange these submissions according to this view of the relative
importance of the evidence bearing on the Horizon Issues.
Accordingly, following a very brief section on procedural background (Section C) and a
section summarising Post Office’s case on the Horizon Issues (Section D), the evidence
of the Fujitsu witnesses are considered (Section E). This is followed by a section
considering the expert evidence (Section F). There is then a section on the remaining
Post Office witnesses (Section G), a section on the claimant-specific witnesses (Section
H1) and a very brief section on Mr Henderson (Section H2). These closing submissions
end with a section on disclosure (Section I) and finally a section on some miscellaneous
points not dealt with elsewhere (Section J).
Appendix I contains analysis of the documents that Mr Coyne provided references to on
the morning of Day 15 of the trial in support of his view that bugs are often deferred or
not dealt with at all on a cost benefit basis. Appendix 2 contains submissions on the 29
"bugs" referred to by the experts in JS2.
PROCEDURAL BACKGROUND
POL00026925
POL00026925
21.
22.
23.
The Court will of course be familiar with the procedural background to this trial.
Many of the procedural steps which led up to this trial are naturally no longer of any
significance. But it is important not to lose sight of some features of this case which have
had a continuing impact.
The first is that Cs have not pleaded a case in anything like the kind of detail that is
typically seen in comparable technical cases in the High Court. Consequently, Post
Office has had to deal with an evolving case which was not really articulated until it saw
Coyne I in October 2018 and Coyne 2 in February 2019. It had been hoped that the
AI8/15
24,
25.
26.
27.
28.
29.
outline document which Cs provided on 17 August 2018! would enable Post Office to
identify and understand the issues that were being raised against it, but this hope was
frustrated. The outline was a vague document which was of no practical use and has
hardly been referred to by either side.
So Post Office based its understanding of the case it had to meet on Mr Coyne’s expert
reports. The evolution in that case between Coyne I and Coyne 2 was very substantial.
This meant that Post Office had to address a case that was only articulated a few weeks
before trial. Some supplemental witness evidence was required.
It would be hard to exaggerate the difficulty that Post Office faced in reviewing Mr
Coyne's reports, which are sprawling and scatter-gun documents. They seem designed to
persuade any but the most assiduous reader (one who has the time to follow up the
innumerable document references) that there is so much smoke that there must be a fire
somewhere. Further, Mr Coyne’s “supplemental” report was no such thing. It was even
longer than Coyne 1, it contained new analyses put in new ways and, given its cross
references and repetitions, it required considerable time to assimilate and understand —
more time than was available in the run up to the Horizon Issues trial.
These observations are not made by way of complaint. They are made because, without
bearing them in mind, there is a danger of relying on hindsight when looking at the
procedural history of this case. Points which seem obviously relevant now were not
obviously relevant to Post Office in 2018. The same is true of documents.
Many complaints are made by Cs as to Post Office’s late disclosure of documents, and
these are addressed in detail in Section I below. In considering these complaints, it is
important to bear a number of points in mind.
First, the documents concerned were generally Fujitsu documents whose significance
was not clear to Post Office and its legal team during much of the history of this case.
Second, Post Office has taken a cooperative approach to disclosure. It has discussed and
agreed all but a few aspects of the disclosure orders made in relation to the Horizon
POL00026925
POL00026925
1 {C1/2}. The outline was served pursuant to para 15 of Third CMC Order {C7/12/5}, as amended by
paragraph 4 of the Fifth CMC Order {C7/22/2}.
AI8/16
30.
31.
32.
Issues and it has complied with each of those orders. There have been no applications by
Cs for specific disclosure nor even any threatened applications.
Third, Pilot Scheme Model C disclosure was ordered in this case. Not only does this
form of disclosure require only particular documents or narrow classes of documents to
be disclosed, but it also requires the party requesting disclosure to explain why its
requests are reasonable and proportionate and, if the requests are not agreed, to raise
them at a CMC so that they can be decided by the Court. In this case, wide requests for
disclosure were made by Mr Coyne (in documents he called “Requests for Information”).
In the light of these requests, a specific mechanism was agreed to ensure that Cs would
explain the relevance, reasonableness and proportionality of any disputed requests by
mid-August 2018 so that such requests could either be agreed or raised with the Court.
That mechanism was approved by the Court and formed part of the Fifth CMC Order.”
Had it been followed, with the benefit of the explanations provided by Cs, the solicitors
on each side would have been able to liaise and cooperate with each other with a view
to achieving an agreed position on any further disclosure to be given, just as they have
in relation to the hundreds of thousands of documents disclosed by Post Office for this
trial. And if any issues remained on further disclosure, these could and should have been
determined by the Court long before trial.
None of these things happened. Cs were silent on disclosure during the whole of August
2018. And although they produced what was described as a “draft” Request for
Information from Mr Coyne towards the end of September, they did not respond to Post
Office's contemporaneous requests for them to explain the relevance, reasonableness or
proportionality of the requests that it contained. This remained the position even when
the requirements of the Fifth CMC Order were pointed out in correspondence. It was
only in December that a formal request was made. By this time Cs had served Coyne I,
and Post Office’s legal team were better placed to understand why it might be said that
the requested documents were relevant and that their disclosure was reasonable and
proportionate.
POL00026925
POL00026925
2 See paras 1 and 2 of that Order {C7/22}.
AI8/17
33.
34.
POL00026925
POL00026925
Against this background, it will be scen that Cs, rather than Post Office, have taken an
uncooperative approach to disclosure. They paid scant regard to the requirements of
Model C disclosure and they effectively sabotaged the Order that was designed to ensure
that any important requests for disclosure could be properly debated and if necessary
decided before any written evidence was served.
Further detail on disclosure is given in Section I below. Cs’ complaints about piecemeal
and late disclosure are a reflection of the piecemeal and late case advanced by Cs: a case
that which as at the date of these submissions is still not clear.
THE HORIZON ISSUES
D1.
35.
36.
Introduction
In this section, Post Office summarises the approach that it submits the Court should take
when makings its findings on the Horizon Issues. In the interests of brevity, it adopts the
introductory submissions it made in Section A3 of its written opening submissions,
without repeating them.* Amongst other things, that section explains Post Office’s
essential case as to the meaning, effect and relative importance of the Horizon Issues.
And also their limits (see, in particular para 25, where Post Office noted that the Court’s
judgment will not involve the determination of any C’s claim).
In the section immediately below, Post Office considers some approaches to the Horizon
Issues that Cs may possibly be intending to take. In the following sections, Post Office:
(1) re-states the categorisation of the Horizon Issues that it adopted in its written opening
submissions and then, in the following sections, (2) respectfully sets out its submissions
as to what the evidence in this trial shows and the findings that should be made. The
Court will of course have its own views as to the exact nature, extent and formulation of
the findings that it wishes to make, but Post Office hopes that it will find those sections
a useful reference point and summary of Post Office’s case on the principal issues.
3 See paras 22 to 26 at +4
AI8/18
37.
D2.
38.
D3.
39.
POL00026925
POL00026925
For this reason, those sections have been kept as short as possible. Many of the Horizon
Issues involve many possible permutations and combinations of findings. The sections
below would be unhelpfully long and complicated if there was an attempt to cover them
all. For similar reasons, those sections do not seek to identify the evidence and arguments
relied on for each of their propositions. These matters are covered elsewhere in these
submissions. On the relatively rare occasions where propositions are based on material
that it may assist to identify here, it is identified in footnotes.
Extent, Likelihood and Risk
In Section A3 of its written opening submissions,’ Post Office notes the repeated use of
words such as “extent”, “likely” and “risk”. The Court will understand the significance
of these words. They are what give this trial much of its practical utility. For example, if
the Court’s judgment were to contain no findings as to the extent to which it was likely
for bugs in Horizon bugs to cause lasting shortfalls in SPMs’ accounts, the utility of
deciding Issues 1 and 3 in this trial would be difficult to discern. Similarly, if the
judgment were to contain no findings regarding the extent of the risk posed to the
accuracy of branch accounts by data errors in Horizon and/or of Fujitsu failing to
detect/correct/remedy bugs in Horizon, it would be difficult to understand the utility of
deciding Issues 4 and 6. And if the Horizon Issues did not include Issues 1, 3, 4 and 6, it
would be hard to justify a separate trial of the other Horizon Issues.
Cs’ Apparent Attempts to Dilute the Horizon Issues
During the course of Cs’ cross-examination of Dr Worden, they appeared to make some
surprising suggestions regarding the meaning and scope of some of the Horizon Issues.
Most surprising was the apparent suggestion that, by asking to what extent it was
possible or likely for bugs to have “the potential” to cause shortfalls, Issue I did not call
for any findings about the extent or likelihood of these things actually happening. At one
point, Cs also appeared to suggest that the real focus of Issue I is Issue 1(b), which asks
4 Paras 22 to 26 at A240.
{A/2/10}
AI8/19
40.
41.
about undermining the reliability of Horizon accurately to process and record
transactions.
Post Office does not know how or even whether Cs will run these points in their closing
submissions. For the moment, it simply makes a few observations:
40.1 First, the points seem to be designed to avoid giving any meaning to the words
“extent” and “likely” in Issue 1. It has always has been common ground on the
pleadings that there are various sources of potential error in Horizon, that there are
measures to address these and that there have been instances in which those
measures did not prevent bugs creating shortfalls.°
40.2 Second, Cs’ retreat behind the word “potential” would have the effect of collapsing
Issue I into Issue 4.
40.3 Third, even if Issue I were to be collapsed into Issue 4, that still leaves Issue 3 (to
what extent and in what respects is the Horizon system robust and extremely
unlikely to be the cause of shortfalls in branches?). The questions arising under
Issue I would have to be addressed in any case under Issue 3.
40.4 Fourth, and crucially, the central issue in this litigation is whether Horizon has
caused SPMs to be held liable for false shortfalls in their branch accounts. If the
Horizon Issues trial does not result in a finding being made as to the generic
likelihood of Horizon causing a false shortfall to be recorded in a given set of
branch accounts, it will not have achieved much of real value to the parties.
A further suggestion that Cs appeared to make during Dr Worden’s cross-examination
was that, whenever it was suggested in a document that Horizon could be improved in
some way, the fact that Horizon had not been improved in that way demonstrates that
there was a “defect” in Horizon within the meaning of Issue 1. This point is
misconceived:
41.1 Horizon, like any IT system, is always capable of improvement. This too has
always been common ground. It is common ground between the experts that
5 See, most notably, paras 53 to 56 of the GDXC-GUCC
(€3/3/22}.
20
POL00026925
POL00026925
A/8/20
41.2
41.3
POL00026925
POL00026925
Horizon has improved over time. The fact that someone may at some point have
suggested an improvement somewhere within the Horizon system does not mean
that the system was defective before or even that adopting the suggestion would
have resulted in an overall improvement. A proposed improvement may or may
not be aimed at fixing something that can properly be called a defect, but the
question whether that something is a defect requires a consideration of far more
than simply the fact that an improvement was proposed.
By the same token, nor does it reveal a defect that, when a particular scenario was
put to a Post Office witness, that witness may have agreed that the way that
Horizon dealt with that scenario was not ideal or could have been better. Amongst
other things, the witness might be wrong, especially where he or she does not have
the expertise and experience needed to evaluate the technical advantages and
disadvantages (including risks) of addressing the scenario differently. The purpose
of instructing experts was to avoid having to rely on inexpert assessments.
Facts of this sort are not evidence of defects. That Cs may be intending to argue
otherwise in their closing submission may be based on Mr Coyne’s “doctrine of
perfection” and his “reducing risks as far as possible” approach discussed in paras
150 to 155 of Post Office’s opening submissions and further in the body of these
submissions. Both of those approaches are fallacious, and they are not what the
Horizon Issues require the parties to address.
D4. Categorising the Horizon Issues
42.
As explained in Section A3, para. 22 of Post Office’s written opening submissions, the
Horizon Issues can be seen as raising three core groups of issues: the robustness issues
(Issues 1, 3, 4 and 6); the remote access issues (Issues 7, 10, 11, 12 and 13); and the
operational issues (Issues 2, 5, 8, 9, 14 and 15). There is only minor, if any, disagreement
in relation to these categorisations. In the following paragraphs, Post Office summarises
the findings which it invites the Court to make on those issues.
AI8/21
POL00026925
POL00026925
DS. The Robustness Issues
43.
44.
(1) To what extent was it possible or likely for bugs, errors or defects of the nature alleged
at §§23 and 24 of the GPOC and referred to in §§ 49 to 56 of the Generic Defence to
have the potential to (a) cause apparent or alleged discrepancies or shortfalls relating
to Subpostmasters' branch accounts or transactions, or (b) undermine the reliability of
Horizon accurately to process and to record transactions as alleged at §24.1 GPOC?
(3) To what extent and in what respects is the Horizon System "robust" and extremely
unlikely to be the cause of shortfalls in branches?
(4) To what extent has there been potential for errors in data recorded within Horizon to
arise in (a) data entry, (b) transfer or (c) processing of data in Horizon?
(6) To what extent did measures and/or controls that existed in Horizon prevent, detect,
identify, report or reduce to an extremely low level the risk of the following:
a. data entry errors;
b data packet or system level errors (including data processing, effecting, and
recording the same);
a failure to detect, correct and remedy sofiware coding errors or bugs;
d. errors in the transmission, replication and storage of transaction record data;
and
@ the data stored in the central data centre not being an accurate record of
transactions entered on branch terminals?
As the Court will be aware, Post Office contends:
43.1 that the Horizon system is and was very robust (being more robust than most
comparable systems);
43.2 that strong measures and controls were and are in place to prevent, detect, identify,
report and reduce to an extremely low level the risk of the things specified in Issues
4 and 6; and
43.3. that Horizon was an extremely unlikely to produce the effects specified in Issues
1 and 3.
In the following paragraphs, Post Office sets out some more detailed propositions that it
submits are supported by the evidence.
i
is)
AI8/22
45.
46.
POL00026925
POL00026925
Issue 3 (first half) and 6(c): General robustness of Horizon.
Regarding these issues:
45.1 The key concept in this context is robustness. That concept occupies a large and
45.2
45.3
45.4
45.5
mature area of modern IT practice.
During its life, Horizon has been a very robust system, relative to comparable
systems.
During its life, Horizon has incorporated and been supported by many robustness
countermeasures. These countermeasures are designed to ensure the accuracy of
accounts, have been designed well and have been applied effectively in practice
(which is not the same as saying that no errors have ever got through).
Fujitsu has supported Horizon well, including in monitoring for bugs,
investigating potential bugs and remedying such bugs as are found.
These matters have reduced to an extremely low level the risk of bugs arising and
not being detected, corrected and remedied.
Issue 1(a) and Issue 3 (second half): Impact on branch accounts
Regarding these issues, the evidence shows that:
46.1
46.2
46.3
46.4
The robustness of Horizon made it extremely unlikely to be the cause of any lasting
discrepancy ina given branch account.
If there was a material lasting shortfall in a given branch account, the chances of
that shortfall having been caused by a bug in Horizon are very small indeed.
The number of bugs that there have been over Horizon’s lifetime is very small
relative to the scale of the system, the duration of its operation and, in particular,
the number of branch accounts generated.
Those bugs that did exist were very unlikely to be of a type that had the potential
to cause discrepancies or shortfalls in branch accounts, and even less likely to be
of a type that had the potential to cause lasting discrepancies or shortfalls. (That
AI8/23
47.
48.
49.
POL00026925
POL00026925
bugs (or potential bugs) rarely cause such effects on accounts can be seen by
comparing (1) the dozens, perhaps hundreds, of “issues” raised by Mr Coyne
across his two reports to (2) his final tally of only 22 issues that he says were
actually bugs and actually had lasting effects on branch accounts.)
Issue 1(b), 4(b), 4(c), 6(b), 6(d) and 6(e): Communication and recording of
transactions
Regarding these issues:
47.1 During its life, Horizon has incorporated and been supported by many
countermeasures that, in the vast majority of cases, ensure the reliability of the
data that are being transmitted, replicated and recorded. Horizon’s
countermeasures in these respects have been well designed and have been applied
effectively.
47.2 Those measures did reduce to an extremely low level the risk of transmission,
replication or recording errors, whether caused by data packet errors or otherwise.
The robustness of Horizon meant it provided a very reliable record of transactions
(which is different from saying that no errors ever occurred).
47.3 There was only an extremely low risk that any given transaction recorded at the
central data centres did not accurately reflect the transactions entered on the branch
terminal.
Issue 1(b), which relates to the “processing” and “recording” of transactions, cross refers
to paragraph 24.1 of the GPOC, which states as follows: “/nsufficient error repellency
in the system (including sufficient prevention, detection, identification and reporting of
errors), both at the data entry level and at the data packet or system level (including data
processing, effecting and reconciling transactions, and recording the same)”
Cs have put forward no evidence of a lack of repellency against data packet errors. Mr
Coyne does not mention the word "repellency" in his report, and he does not comment
6
CHB}
AI8/24
50.
51.
D6.
POL00026925
POL00026925
on data packet errors. No question was put to any witness in cross-examination on data
packet errors.
Post Office invites the Court specifically to find that the Claimants’ case in para. 24.1 of
the GPOC in relation to data packet errors is not made out and that there is no evidence
that Horizon suffered from data packet errors (beyond the admitted possibility for such
errors even in a robust system).
4(a) and 6(a): Data entry
Regarding the above issues:
51.1 Horizon, like all comparable IT systems, faces a risk of a user entering incorrect
data.
51.2 Horizon deployed industry-standard measures to reduce such risk.
51.3 There is no evidence to the effect that Horizon was at a greater risk of data entry
error than other comparable IT systems or that it did not deploy industry-standard
measures to reduce such risks.
The Remote Access Issues
(7) Were Post Office and/or Fujitsu able to access transaction data recorded by Horizon
remotely (i.e. not from within a branch)?
(10) Whether the Defendant and/or Fujitsu have had the ability/facility to: (i) insert, inject,
edit or delete transaction data or data in branch accounts; (ii) implement fixes in
Horizon that had the potential to affect transaction data or data in branch accounts; or
(iii) rebuild branch transaction data:
a. atall;
b. without the knowledge of the Subpostmaster in question; and
c. without the consent of the Subpostmaster in question.
(11) If they did, did the Horizon system have any permission controls upon the use of the
above facility, and did the system maintain a log of such actions and such permission
controls?
(12) If the Defendant and/or Fujitsu did have such ability, how often was that used, if at all?
AI8/25
52.
53.
54,
POL00026925
POL00026925
(13) To what extent did use of any such facility have the potential to affect the reliability of
Branches" accounting positions?
Issue 7
Given that Issue 10 addresses abilities to make changes to data, Issue 7 must logically
be using the word “access” to refer to read-only access to transaction data. This is how
the issue is addressed by the experts in {843S3.’ As to this, it is common ground that:
52.1 Post Office had access to transaction data copied onto its Management Information
Systems (“MIS”). Under Horizon Online, Post Office also has read-only access
to branch data held in the BRDB for business reasons such as monitoring levels of
cash held in branches.
52.2 Fujitsu had direct access to the branch database (in Legacy Horizon, messagestores
held on the correspondence servers and, in Horizon Online, the BRDB). Under
Horizon Online, Fujitsu can also download logs stored on the counters (including
a log that records the buttons pressed on the screen and messages displayed on the
screen).
Issue 10
Laying aside TCs and TAs, which are not mentioned in or encompassed by this Issue,®
the evidence shows that Post Office had no ability to do any of the things referred to in
Issue 10. (For the avoidance of doubt, changes made to reference data are prospective
only so cannot alter existing transaction data or data in branch accounts.)
Turning to Fujitsu, the evidence shows that:
54.1 In Legacy Horizon, Fujitsu had the ability to do the following:
(a) _ Insert transaction data into branch accounts, either (i) at the correspondence
server or (ii) at the counter. The former was the standard method.
7 1/4/10}.
* It is apparent from the generic pleadings that “remote access” concerns changes made to the branch
accounting data other than at the counter (by the user interacting with the system, e.g. to enter the details on
an Error Notice or to process a TC or TA sent to the branch).
AI8/26
55,
POL00026925
POL00026925
(b) Insert, edit or delete data (including transaction data and other data forming
part of a branch’s accounts) via the use of privileged user rights (but not at
the messagestore level).
(c) Rebuild transaction data at the branch by replication from some other copy
of the data in another messagestore (initiated automatically by the deletion
of all the data held on the relevant counter). A small number of rebuilding
operations required manual intervention to recover specific transactions that
could not be replicated automatically because they were not recorded other
than in a faulty counter or a counter that had been removed from the branch.
54.2 In Horizon Online, Fujitsu had the ability to do the following:
(a) Insert transaction data into branch accounts by using the Transaction
Correction Tool.
(b) Insert, edit or delete data (including transaction data and other data forming
part of a branch’s accounts) via the use of privileged user rights.
As to Issues 10(b) and (c), Fujitsu had the technical ability to do the things listed in para.
54 above without the knowledge or consent of the SPM. However, the evidence shows
that:
55.1 In Legacy Horizon:
(a) The insertion of transaction data at the correspondence server would have
been visible to the SPM as a transaction inserted at a non-existent counter
with a number greater than 32.
(b) The insertion of transaction data at the counter required the cooperation of
the SPM (it could not be done when the counter was being used). Moreover,
the SSC’s practice when injecting transaction data in this way was to include
information in the data making it clear that this had been done (e.g. by
changing the counter numbers associated with the transactions to non-
existent counters or adding a comment to the data).
AI8/27
()
(@)
(e)
POL00026925
POL00026925
The SSC’s practice was to inform SPMs and secure their cooperation when
any of these things were done. There is evidence of one occasion when this
did not happen, in 2007.
There is evidence? of an occasion in which a stock unit had rolled over into
the next trading period with an erroneous starting position (zero). Privileged
user access rights were used to move the affected stock unit back into the
original trading period and to delete the erroneous opening position, thereby
allowing the branch accounts to be aggregated and rolled over in the usual
way, resulting in the correct opening position. The SPM was informed of
the change.
Save as set out above, there is no evidence of transaction data (or other
branch account data) being inserted, edited or deleted via privileged user
access.
55.2 In Horizon Online:
(a)
(b)
On the single occasion on which the Transaction Correction Tool was used
to insert transaction data into a branch’s accounts, the SPM was informed
and her consent was obtained.
There is evidence!”
of an occasion in which, because of a problem that arose
during the migration of a branch from Legacy Horizon to Horizon Online,
stock units had been moved into a new trading period with incorrect stock
opening positions (zeros), which prevented effective rollover from the
current trading period. Privileged user access rights were used to move the
affected stock units back into the branch’s current trading period and to
delete the erroneous stock unit opening positions, thereby allowing the
branch accounts to be aggregated and rolled over in the usual way, resulting
in correct opening positions. The SPM was kept informed as to what was
being done to enable the branch to rollover.
9 (F/312}.
F611}
AI8/28
(c) Save as set out above, there is no evidence of transaction data (or other
branch account data) being inserted, edited or deleted via privileged user
access.
56. As to Fujitsu implementing fixes in Horizon, the experts have interpreted this to include
both (1) prospective fixes to the code (or other system changes designed to prevent the
problem re-occurring) and (2) measures to address the effects of the specific occurrence
of a bug that triggered the investigation. As to this:
56.1
Measures within the first category would have prospective effect only, rather than
affecting existing transaction data in branch accounts. It is common ground that
implementing a code fix carries a risk of introducing a new bug, although this is
reduced by testing and monitoring.
Measures within the second category could, in the rare instances that they involved
changes to branch transaction data, self-evidently affect such data. In the absence
of error in designing or implementing the change, however, the effect would be to
improve the accuracy of the transaction data and branch accounts. The risks here
were reduced by testing and monitoring of changes.
Issue 11
57. The various abilities identified above were subject to permission controls. Specifically:
57.1
Aside from in respect of emergency measures, Fujitsu was required to obtain
specific permission from Post Office before making any change that may have an
impact on branch accounts. There were process documents for seeking permission
for changes — namely, OCPs (for changes to branch accounting data) and OCRs
(for changes to data in the TPS). A record of such changes was also kept in the
MSC system. Permission could also be obtained informally - there is evidence of
instances where permission was obtained by email or telephone and then
documented retrospectively in an OCP or OCR.
Permission to use the privilege user role (*“APPSUP”) was limited to certain
Fujitsu employees in system support and database roles. From 2012 (for new SSC
POL00026925
POL00026925
AI8/29
58.
59.
users) and August 2016'! (for all SSC users), specific permission from another
department within Fujitsu was required for each use of that role by SSC staff,
rather than it being generally available.
There were also process controls within Fujitsu as to the witnessing and/or
monitoring of changes.
There were logs and/or other records generated in respect of the various abilities
identified above. Specifically:
58.1
58.2
There was an audit log that records the use of the Transaction Correction Tool.
There was a record of other changes in the OCPs, OCRs and now MSCs. Where
any change was made, there was a Peak addressing the problem which required
the change and that Peak typically recorded (in varying degrees of detail) the fact
and nature of the change.
As to uses of privileged user rights, all such uses were recorded in privileged user
access logs from 2009. From 2015, these logs also provided information as to the
steps taken (although not necessarily the specific database tables accessed).
Issue 12
The facilities identified above were used very infrequently. Specifically:
59.1
Under Legacy Horizon:
(a) Transaction data was inserted at the correspondence server in a small
number of instances.
(b) There were very rare instances in which transaction data was injected at a
counter. Such injections are recorded in the Peak at {F/377.1} and the OCP
at {F/432.2}.
(c) The rebuilding of transaction data on a counter by its deletion and automatic
replication from some other copy of the data in another messagestore was
POL00026925
POL00026925
1 {F/1844}, row 17808 which confirms, under MSC 045J0451867 the APPSUP database role was removed
for all SSC users.
30
A/8/30
POL00026925
POL00026925
rare. In very few instances (at least 4'? and perhaps as many as 10)! some
manual intervention was required to rebuild the counter messagestore.
(d) There is evidence!* of one occasion in which a stock unit had been rolled
over into the next trading period with an erroneous starting position (zero)
Privileged user access rights were used to move the affected stock unit back
into the original trading period and to delete the erroneous stock unit
opening position, thereby allowing the branch accounts to be aggregated and
rolled over in the usual way, resulting in the correct opening position. The
SPM was informed of the change.
(e) Save as set out above, there is no evidence of any transaction data (or other
branch account data) having been inserted, injected or deleted via the use of
privileged user rights.
59.2 Under Horizon Online:
(a) The Transaction Correction Tool was used on one occasion to insert a
balancing transaction.
(b) There is evidence’? of an occasion in which, because of a problem that had
arisen during the migration of a branch from Legacy Horizon to Horizon
online, stock units were moved into a new trading period with incorrect
stock opening positions (zeros), which prevented effective rollover from
the current trading period. Privileged user access rights were used to move
the affected stock units back into the branch’s current trading period and
to delete the erroneous stock unit opening positions, thereby allowing the
branch accounts to be aggregated and rolled over in the usual way
(resulting in correct opening positions). The SPM was kept informed as to
what was being done to enable them to rollover.
» The number of Peaks which Post Office has identified in its evidence and correspondence: Parker 2, para.
38.3 {E2/12/9} and the letter at {H/253}.
3 The number of instances Mr Coyne said he had seen in cross-examination: {Day]6/33:23}.
4 4F/312}
'S {F/61L}
31
AI8/31
60.
D7.
Issu
The
relia
60.1
60.2
60.3
60.4
The
(7)
(3)
(c) Save as set out above, there is no evidence of any transaction data (or other
branch account data) having been inserted, injected or deleted via the use
of privileged user rights (other than to the extent required for the instance
identified above).
e 13
use of the facilities identified in Issue 10 had a limited potential to affect the
bility of branch accounting positions. Specifically:
The number of times in which those facilities were used, and the number of branch
accounts affected by such uses, was extremely small relative to the total number
of branch accounts.
The facilities were used by highly skilled staff at the SSC and were directed at
improving the reliability of the relevant branch’s accounting position. In the vast
majority of instances, the SPM was aware of the changes made by the SSC and
could raise any new accounting problem that might arise. The changes made were
substantially more likely to improve, rather than reduce, the accuracy of the branch
accounts at issue. The risk of human error in designing and implementing changes
was very low, and the risk of such human error going undetected was extremely
low.
There is no evidence of a change made using the facilities having ever introduced
a lasting inaccuracy into a branch account.
It follows that the use of the facilities is likely to have caused a small improvement
to the overall reliability of branch accounting positions.
Operational Issues
Did the Horizon IT system itself alert Subpostmasters of such bugs, errors or defects as
described in [Horizon Issue 1] and if so how?
How, if at all, does the Horizon system itself compare transaction data recorded by
Horizon against transaction data from sources outside of Horizon?
POL00026925
POL00026925
AI8/32
6l.
62.
POL00026925
POL00026925
(8) What transaction data and reporting functions were available through Horizon to Post
Office for identifying the occurrence of alleged shortfalls and the causes of alleged
shortfalls in branches, including whether they were caused by bugs, errors and/or
defects in the Horizon system?
(9) At all material times, what transaction data and reporting functions (if any) were
available through Horizon to Subpostmasters for:
a.
b.
identifying apparent or alleged discrepancies and shortfalls and/or the causes of the
same; and
accessing and identifying transactions recorded on Horizon?
(14) How (if at all) does the Horizon system and its functionality:
a.
c.
enable Subpostmasters to compare the stock and cash in a branch against the stock
and cash indicated on Horizon?
enable or require Subpostmasters to decide how to deal with, dispute, accept or make
good an alleged discrepancy by (i) providing his or her own personal funds or (ii)
settling centrally?
record and reflect the consequence of raising a dispute on an alleged discrepancy,
on Horizon Branch account data and, in particular:
i. does raising a dispute with the Helpline cause a block to be placed on the value
of an alleged shortfall; and
ii. is that recorded on the Horizon system as a debt due to Post Office?
. enable Subpostmasters to produce (i) Cash Account before 2005 and (ii) Branch
Trading Statement after 2005?
enable or require Subpostmasters to continue to trade if they did not complete a
Branch Trading Statement; and, if so, on what basis and with what consequences on
the Horizon system?
(15) How did Horizon process and/or record Transaction Corrections?
As noted in Post Office’s opening submissions, there do not appear to be any differences
of substance between the experts on these issues.
Issue 2
The experts agree that Horizon did not, in general, alert SPMs to any significant bugs or
other defects in the system itself.'° They also agreed that the extent to which any IT
system can automatically alert its users to bugs within the system itself is necessarily
1 JS2, para 2.1 {D1/2/38}.
33
AI8/33
63.
64.
65.
66.
limited and that while Horizon has automated checks which would detect certain bugs,
there are types of bugs which would not be detected by such checks.!”
Even if it were possible for the Horizon system to alert SPMs to bugs, it would be
positively unhelpful to alert users to precise details of abnormal conditions beyond their
day-to-day experience of the system. It is far more efficient and useful for a system to
record unexpected or significant events in logs that can then be interrogated in the course
of any investigation into unexplained discrepancies. That is what Horizon does.
In Coyne 2, Mr Coyne clarifies that he did not intend to suggest in his first report that it
would be a “good thing” for SPMs to be provided with more information about the inner
workings of Horizon.'® He appears to agree that it would be counter-productive to do
so. He does not consider that there is any useful automatic way in which SPMs could be
alerted to Horizon faults;!? nor does he consider that SPMs could benefit from
information about the back-end systems of Horizon.”
It follows that neither expert supports Cs’ pleaded case that SPMs should have been
provided with more detailed information from Horizon, including (at least implicitly)
information as to potential bugs or errors in the system.
Issue 5
The experts agree that reconciliation between transactions from Horizon and transactions
recorded by Post Office’s clients is largely automated.”* Such reconciliation takes place
between Post Office’s back-end accounting systems and its clients’ systems.
POL00026925
POL00026925
1 382, para 2.1 {D1/2/38}.
8 Coyne 2/5.380 (1° row of the table) {D2/4/226}.
® Coyne 2/5.380 (2™ row of the table) {D2/4/226}.
2 Coyne 2/5.380 (4" row of the table) {D2/4/228}
2 Soe, e.g., Generic Reply, paras 13-16 {C3/4/6} and 21.11 {C3/4/17}.
2 J$3, para, 5.1 {D1/4/8}
34
AI8/34
POL00026925
POL00026925
Issue 8
67. The experts agree that Post Office had access to variety of reports through its own MIS
and through the ability to request information from Fujitsu.”> They agree that Post Office
had access to information that was not available to SPMs.”*
68. The experts also agree that the descriptions of the facilities for Post Office in the experts
reports are consistent and can be taken together.*> Such facilities include the following:
68.1 The primary sources of data are fed by the Transaction Processing System, which
provides transaction and other data to various Post Office back-end accounting
systems.
68.2 These Post Office accounting systems include Credence and POLSAP.
68.3 Post Office has analytical facilities that are not available to and not required by
SPMs.
68.4 Post Office can run standard database reporting tools, on demand, to retrieve and
analyse information about branch transactions.
68.5 Post Office can also request transaction data from Fujitsu’s audit system.
68.6 Post Office may also obtain from Fujitsu information as to system events that are
detected by the System Management Centre.”®
Issue 9
69. The experts agree in JS2”” that:
69.1 The causes of some types of apparent discrepancies and shortfalls may be
identified from reports or transaction data available to SPMs. Other causes may
2 See Worden 1/1083-1088 {D3/1/238} and Mr Coyne’s views on Issue 8 in JS1 {D1/1/15}.
™ Coyne 2/5.411 {D2/4/236}.
8 J§3, para 8.2 {D1/4/10}.
2 See Worden 1/1083-1088 {D3/1/238}.
77 JS2, paras 9.1-9.3 {D1/2/39}.
AI8/35
70.
71.
72.
be more difficult or impossible to identify from reports or transaction data
available to SPMs.
69.2 SPMs would not be expected to have detailed knowledge of the Horizon system.
69.3. Any competent IT support operation is grateful to its users, when they draw its
attention to any problem which can be fixed, to reduce the future costs of support.
Although the experts have not spelt out the following in a JS, it appears that the experts
also agree that:
70.1 SPMs could run various (more than one hundred) different types of report covering
transactions conducted in the branch.”*
70.2 These reports provide a useful source of information when performing normal
reconciliation activities.””
70.3 Inthe ordinary course, the reports show enough information for an SPM to balance
transactions.*°
Mr Coyne agrees with the account given by Dr Worden of the reports that are available.*!
They include the following: reports by stock unit on a daily or weekly basis, reports by
user, balance reports and journals such as transaction and event logs.*”
Dr Worden gives descriptions of certain of the reports and logs and how they could be
used to investigate discrepancies. He gives specific consideration to the Transaction
Log,** the Event Log,** the Balance Snapshot** and the Stock on Hand report.*° There
are also highly-specific reports available to SPMs, such as the transfer reconciliation
POL00026925
POL00026925
8 Coyne 1/8.12 {D2/1/143}; Worden 1/991 {D3/1/220}.
® Coyne 1/8.12 {D2/1/143}.
3 Coyne 1/8.20 {D2/1/146}.
31 J82/14.1 {D1/2/40}.
% Worden 1/991 {D3/1/220}.
8 Worden 1/1000 {D3/1/222}.
M Worden 1/1004 {D3/1/224}
38 Worden 1/1007 {D3/1/225}.
% Worden 1/1007 {D3/1/225}.
36
AI8/36
73.
74,
76.
77.
78.
POL00026925
POL00026925
report that identifies any transfers out of stock units for which there is not a
corresponding transfer in.*”
The SPM will be aware of many substantial financial discrepancies caused by user error
or system error at the latest at the end of the trading day on which they arise, as such
discrepancies are often revealed by conducting a mandatory cash declaration. A trial
balance would also reveal these and other common discrepancies.
At the end of the accounting period, when balancing the account for rollover, Horizon
provides automatic notifications to the SPM of certain types of problem, including (most
notably) receipts and payments mis-matches.
Accordingly, there is a great deal of information available to an SPM in branch. The
reports that can be run on the Horizon system enable the SPM to carry out a wide range
of investigations into what has happened in the branch and in particular as to cause of
apparent discrepancies and shortfalls.
Mr Coyne states that the information available to SPMs is what he would expect to see
given that they are the users of the Horizon system and “would not typically be given
access to anything beyond what was necessary for them to carry out their ‘business as
usual’ activities”
Disputes and investigations involving SPMs and Post Office are not triggered by a
dispute button on Horizon. Disputes are raised via the Helpline. Neither of the experts
has criticised that element of the design of the system. Investigations and the resolution
of disputes rely on cooperation between Post Office and SPMs.
Issue 14
The experts agree:*°
78.1 That their respective descriptions of facilities available to SPMs are consistent
with each other.
57 See, e.g., (F/1782/158}
58 Coyne 1/8.11 {D2/1/143}.
2» 382, paras 14.1-14.3 {D1/2/40}
37
AI8/37
POL00026925
POL00026925
78.2 About the functionality enabling SPMs to deal with, dispute, accept or make good
alleged discrepancies.
78.3. That comparison of cash and stock in branch and figures recorded within Horizon
can be determined by the SPM/auditor physically counting the cash and stock in
branch and inputting those values into Horizon for a comparison to be made
against the electronically derived figures held by Horizon.
79. The experts also agree various details about processes to be followed, including the
processes for trading, correction/dispute processing and discrepancies.”
Issue 15
80. The experts agree that the TC process arises on the majority of occasions as a result of
either Post Office of an SPM identifying an imbalance or discrepancy. Between 2006 to
2017, TCs were applied more than 100,000 times each year. It is also agreed that the TC
process could lead to TCs being issued in error and, when disputed, some TCs are
corrected by issuing another TC.*!
E. FUJITSU WITNESSES
El. Mr Roll’s Evidence
81. The relevant evidence is as follows:
81.1 Mr Roll’s first witness statement dated 11 July 2016 {E1/7}.
81.2 Mr Parker’s first witness statement dated 16 November 2018 {E2/11}.
81.3 Mr Roll’s Amended second witness statement dated 16 January 2019 {E1/12}.
81.4 Mr Parker’s second witness statement dated 16 November 2018 {E2/12}.
* JS2, paras 14.4-14.6 {D1/2/42}
1 $52, paras 15.1 - 15.2 {D1/2/43}
38
AI8/38
82.
83.
84.
85.
81.5 Mr Parker’s third witness statement dated 16 November 2018 {E2/13}.
81.6 Transcript: Mr Roll’s evidence at {Day3/106:13}; {Day4/165:3} to {Day4/165:3};
Mr Parker’s evidence at {Day12/4:18} to {Day12/95:12}.
The Importance of Mr Roll’s Evidence to Cs’ Case
Mr Roll’s evidence is of central importance to Cs’ case. He is described by Cs as a
whistleblower.” The date of his first witness statement (signed a few months after the
first claim form was issued and Cs’ pre-action letter was sent)? may reflect the
significance Cs attach to what he has to say.
Like the witness statements of the claimant-specific witnesses (whose evidence is
considered in detail below), Mr Roll’s first witness statement is vague and
unparticularised. But it was plainly drafted with the intention that it would be explosive:
it appeared to make far-reaching claims about Fujitsu practices and about the reliability
and integrity of the Fujitsu procedures. So was his second witness statement.
The true position which emerged during Mr Roll’s cross examination is far removed
from the impression created by those documents. Mr Roll’s oral evidence, in contrast to
his witness statements, was careful and precise. He was at pains to assist the Court and
to give accurate answers. His memory of events was, unsurprisingly, hazy in many
respects but nevertheless a clear picture emerged of Fujitsu as an organisation which was
thorough, professional and conscientious and which took considerable care to ensure that
matters were properly investigated and dealt with.
Mr Roll’s Recollection
According to his first witness statement, Mr Roll worked in the SSC between 2001 and
2004** i.e. his evidence covers a period of between 15 and 18 years ago. Mr Roll, quite
understandably, frequently referred to the fact that he was unable to remember details
and that his recollection might not be accurate. For example:
POL00026925
POL00026925
* Cs’ Opening Submissions para 61 {4/1/21}.
“HL.
“4 E1/7/1} para 1.
AI8/39
86.
87.
85.1
85.2
85.3
85.4
85.5
He said that he started at Fujitsu in 2000, not 2001.
He accepted that his recollection that he spent 70% of his time doing tasks which
were “not mundane” could be wrong — and that it might be that one simply
4° He could not
remembers more clearly things which were not mundane.
remember what proportion of his work involved supporting engineers, for
example.‘7
He could not remember what work he carried out in relation to a particular KEL,**
or what a “marooned transaction” was.”
He could not remember the detail of how codes were allocated to Peaks or what
the codes were.*”
He suggested that there was sometimes time pressure on the SSC but accepted that
that is just how he remembered feeling since “it was a long time ago”*! and that
his recollection was “quite hazy”. He was unable to say how many times he had
such a feeling but accepted it was “/nJot very often”.>*
There are many other examples.
Perhaps inevitably, some of Mr Roll’s evidence remained unhelpfully vague. In his
witness statements, he makes serious allegations and gives the impression of widespread
problems. In cross examination it became apparent that all he had was a vague sense of
unease about some aspects of the work performed for example in relation to
investigations:
POL00026925
POL00026925
48 (Day3/121:6} to {Day3/121:14}.
46 {Day3/125:25} to (Day3/126:8}.
"7 (Day3/131:15} to (Day3/131:21}.
“® (Day3/127:4} to {Day3/127:11}.
® (Day3/128:22} to {Day3/129:
5° (Day3/144:16} to {Day3/1
5! (Day4/19:24} to {Day4/20:8}
2 (Day4/23:10} to {Day4/23:20}.
58 (Day4/22:2} to {Day4/22:10}.
4}.
215}.
40
A/8/40
88.
89.
POL00026925
POL00026925
Q. So are you suggesting that on every occasion that a problem came in, something
was reported in in relation to reconciliation, you were the one that investigated them
all?
A, Oh, no, I wasn't -- everybody -- it would have been farmed out.
Q. So are you saying that you knew enough about everybody else's investigations to
be uncomfortable about the investigations your colleagues were doing?
A. No.
Q. Right, so are you saying that you were uncomfortable with the investigations that
you did?
A. No.
Q. So what were you uncomfortable with?
A. Sometimes the pressure we were under, the timescales. I think that's what it must
have been. Again
Q. So are you suggesting that there was a particular timescale issue with respect to
reconciliation processes that had a particular application to reconciliation problems
that didn't apply to other problems that you were investigating in Horizon?
A. All I can remember is that I felt uneasy about something, about this process, but I
can't tell you why, so ...°4
He could not recall any instances where the recovery process — a form of resilience — did
not operate properly.*°
Similarly Mr Roll described his recollection that the sense of SLAs might have affected
how the SSC did their job as “vague”? and said that his assertion that the test team felt
under enormous pressure to complete the testing within certain timescales such that the
test regime was negatively affected*’ was based on ancient office gossip.** (Note that
ultimately Mr Roll accepted that it was wrong to suggest that fixes or work-rounds were
5 (Dayd/59:20} to {Day4/60:18}.
5S (Day4/64:18} to {Day4/65:6}.
86 {Day4/79:3} to {Day4/79:12}.
57 {E1/10/5} para 15.
58 (Day4/82:2} to {Day4/82:13}.
41
AI8/41
90.
O1.
92.
93.
POL00026925
POL00026925
not done properly because of SLAs;*? and that he was not aware of any SLA targets
which related to incidents with which the Horizon trial is concerned).°°
The Court should be slow to give weight to vague and impressionistic evidence of this
sort.
Mr Roll’s Position in the SSC and Experience of Issues
Mr Roll worked in 3“ line support: he accepted the suggestion that this was the elite but
that the 4" line support, who actually fixed software problems, were the super elite: he
thought that some members of the 3“ line belonged to this category as well, although not
him.°! He agreed that he had been well trained by Fujitsu, although could not recall the
period of training.”
He accepted that generally it was a small group of 5 or so people who would actually
examine code “/fJrom a PINICL perspective” i.c. for the purposes of establishing
whether there was a problem with the code itself.
Mr Roll accepted that when he said in his second witness statement that some 70% of
his workload involved looking for faults on data stores™, that was:
A. ... not technically looking for a bug in the code, more a bug in the data, a
corruption in the data.®
And:
Q. So when you describe 70 per cent of your work as including looking for faults on
data stores, you are not saying, are you, that that work was looking for sofiware bugs?
That's not what you mean? A. Not 70% of it, no.°
% (Day4/81:4} to {Day4/81:12}
(Day4/86:3} to {Day4/86:9}.
61 (Day3/115:1} to {Day3/115:14}.
© (Day3/125:13} to {Day3/125:19}.
& {Day3/124:21} to {Day3/125:9}.
fEI/O/8} para 25.
6 {Day3/133:13} to {Day3/133:14}.
% ¢Day3/134:16} to {Day3/134:20}.
AI8/42
94,
95.
96.
97.
POL00026925
POL00026925
Mr Roll agreed that Horizon continually produced a large number of automatic reports
to monitor its performance and look for problems and that people at his level would look
at them and see if anything needed to be done — and that as time went on they wrote more
reports or programmes themselves to monitor the system.®”
Mr Parker’s evidence was that:
Mr Roll was primarily used in Operational Business Change (OBC), which involved
supporting the engineers who were opening and closing branches and increasing and
decreasing the number of counters in branches. Mr Roll would also have been
regularly correcting the application environment after engineers had replaced failed
counter hardware and clearing temporary files to increase disk space. This could
fairly be described as 2" line work and it was done by the SSC because it required a
higher level of access to the system than other support teams had.
Although Mr Roll’s recollection was different, he accepted that this might be an accurate
”® that he did not play
summary.” He also accepted the accuracy of Mr Parker’s evidence
a significant part in the process of source code examination and that a software fix would
be written by 4" line support, not 3" line.”
Mr Roll fairly accepted that the impression given in his first witness statement” that he
regularly worked through thousands of lines of code was something he was no longer
sure about.”*
Mr Roll was taken through Mr Parker’s analysis of the categorisation of
issues during the time of Mr Roll’s tenure in SSP1.* On the basis of that evidence, Mr
Roll accepted that fire-fighting coding problems was a tiny amount of his work.”°
© (Day3/137:7} to {Day3/137:23}.
6 (E2/11/9} para 34.
© (Day3/138:12} to {Day3/139:9}.
% {E2/11/9} para 35.
7 {Day3/139:20} to {Day3/141:8}.
® (E1/7/1} para 7.
8 (Day3/142:21} to {Day3/143:19}.
™ 4F/1839}.
8 Day3/158:4} to {Day3/158:19}.
43
AI8/43
98. Similarly, Mr Roll candidly accepted that he could no longer be sure that his evidence
that the SSC “regularly identified issues with the computer coding in the Horizon
system””® was correct.”
99. He also clarified:
99.1 Paragraph 8 of his first witness statement:”*
Q. I'm grateful. Then paragraph 8 -- this may have been carefully drafted, or it
may actually have been carelessly drafted and I'm not attributing any motives to
you, I'm genuinely not, but if you look at the last sentence of paragraph 8, you
say. "If an error was referred to us then it was extremely unlikely to be due to a
mistake made by a postmaster; the vast majority of errors I dealt with were due
to coding errors or data corruption." Just to be clear, Mr Roll, you are not saying
that the vast majority of errors you dealt with were coding errors, are you?
A. No.
Q. And when you say " or data corruption", that includes all sorts of problems
A. All sorts of problems
Q. -- that had nothing to do with sofiware bugs?
A. Yes.”
99.2 And what he meant by “software bugs”:
Q. I see. So when you say "software issues" we should -- would it be fair to say
that whenever you refer to "software issues" we should probably read in the
possibility of other data issues that aren't concerned with software bugs?
A. Yes.
Q. I'm grateful.
MR JUSTICE FRASER: By "software bugs" I think Mr De Garr Robinson is
talking about code issues.
A. Yes.
POL00026925
POL00026925
% (E1/7/2} para 9.
7 {Day3/158:22} to {Day3/159:6}.
78 {E1/7/2} para 8.
% {Day3/159:7} to {Day3/159:24}.
44
AI8/44
99.3
99.4
99.5
99.6
POL00026925
POL00026925
MR JUSTICE FRASER: That's how you are putting it. You are using in your
statement, as far as I can tell, "software" in a rather more generic way, is that
correct? A. I'm afraid so, yes. I should have -- it has been a long time.”
And that the SSC was not routinely encountering coding issues:
Q. So you are not saying that you were routinely encountering coding issues, are
you?
A. Bugs, no!
And that data issues which were the result of coding issues were a small proportion
of overall data issues — and even in those situations it did not mean there was a
software problem, just the potential for one.** Mr Roll accepted that coding errors
which caused a financial impact on branch accounts were extremely rare:
Q. Well, let me continue with what Mr Parker says: "As stated in paragraph 16
above, such errors were extremely rare." That's coding errors which caused a
financial impact on branch accounts. As a proportion of the work coming into
the SSC, coding errors causing financial impact on branch accounts was
extremely rare, would you agree?
A. Yes.
And that, as Mr Parker says,“ in the vast majority of cases a software issue that
caused a discrepancy would be caught by a receipts and payments mismatch.**
And that when he referred in paragraph 19 of his second witness statement*® to a
software fix, he was not talking about the kind of error an SPM would be aware of
— i.e. not one that would have affected his accounts.*”
© (Day3/161:11} to {Day3/162:1}.
8! (Day3/163:11} to {Day3/163:13}
® {Day3/164:8} to {Day3/166:4}.
® (Day3/167:5} to {Day3/167:12}.
FEQ/I/11} para 42.
*S (Day3/168:13} to {Day3/169:9}.
8 {E1/10/6}.
® (Dayd/71:3} to {Day4/71:9}.
45
AI8/45
POL00026925
POL00026925
99.7 And that, as Mr Parker says,“ it is quite normal in a contract such as that between
Fujitsu and Horizon to have service level agreements, and that the same level of
diligence was applied to all incidents whether an SLA was relevant or not.*?
100. Mr Roll accepted that paragraph 19 of his first witness statement,” which states that he
and other colleagues were “routinely” working on coding issues causing financial
discrepancies, was not correct:
Q. And then you say: "Furthermore, the coding issues impacted on transaction data
and caused financial discrepancies on the Horizon system at branch level." Would
you accept that the proportion of coding issues that had that effect was even smaller?
A. Yes.
Q. And the next sentence: "It was those issues that I, and other colleagues at Fujitsu,
were routinely working on daily." Well, that's not really right, is it, Mr Roll? You and
others were not working on these issues daily, were you, they were a very small
proportion of the work that was being done?
A. From the bugs then yes.
Q. Thank you.
MR JUSTICE FRASER: By bugs you mean coding?
MR DE GARR ROBINSON: Coding.”!
101. He also accepted that his manager wanted him to get to the bottom of problems:
Q. But it would be fair to say -- I think I'm -- I hope I'm drawing the right inference
from what you said a minute ago -- it would be fair to say that Mr Peach would want
you to bottom out the issue, to get to the bottom —
A. Yes.
Q. -- of that issue that was reported to you. He didn't want you to close your eyes to
a problem, did he?
A, No.
Q. I'm very grateful, Mr Roll.”
8 (E2/11/12} para 43.
® (Day4/74:18} to {Day4/75:4}.
FELT}.
% {Day3/176:9} to {Day3/177:1}.
% {Day3/179:4} to {Day3/179:12}.
46
AI8/46
102.
103.
104.
105.
POL00026925
POL00026925
and that everyone in the SSC tried to do a professional, rigorous and thorough job.°*
Indeed, Mr Roll could not recall ever working on a coding issue which caused a financial
impact:
Q. But you don't recall ever having encountered, in all the PEAKs that you worked
on, a coding issue that definitely caused a financial impact?
A. I don't recall discovering one, no.°*
He made it clear that he was not suggesting that if someone phoned in with a problem,
that problem is not diagnosed at that time but was later diagnosed when some other SPM
raised it, then the original caller would not be left high and dry: in that scenario
information would be provided to Post Office to allow the first SPM to be made whole.”°
Similarly if, for example, a problem with reference data was spotted, the SSC would set
up a system to monitor where problems might be occurring, fix them, look in the past to
see if the problem had occurred before and deal with that — even though that would not
have any effect on branch accounts.”®
Mr Roll also accepted that it was unlikely that Fujitsu would be unable to determine the
root cause of a problem and that in such a situation the relevant symptoms would be
recorded so that Fujitsu would maintain knowledge of the symptoms and this would be
passed onto the 4" line of support.”’ In any case where there was a lingering suspicion
that there might be a coding issue, it would be passed on.°*
Mr Roll suggested that on one occasion he was told to close down an investigation when
it had not been completed. The situation involved a mobile system which was rebooting
unexpectedly.” This was not software related.'°° He accepted that people in the SSC
% (Day4/20:22} to {Day4/20:25}
% (Day3/179:24} to {Day3/180:2}.
% {Day4/26:6} to {Day4/27:8}.
% {Dayd/54:18} to {Day4/55:6}.
° Dayd/29:20} to {Day4/30:10}.
°8 {Dayd/32:20} to {Day4/33:1}.
% {Day4/35:9} to {Day4/36:21}.
10 (Day4/35:2} to {Day4/35:5}.
47
AI8/47
106.
107.
108.
were not told not to perform checks that they felt were required in order to do their job
properly,!°!
Mr Roll also accepted that it was in Fujitsu’s interest to carry out its support role
properly:
MR DE GARR ROBINSON: Could I just advance some general propositions and see
if you will agree with me. Isn't it the case that Fujitsu had every incentive to make the
support operation work, to minimise the problems requiring changes and to minimise
the problems requiring fixes down the line; wouldn't that be right?
A. Yes.
Q. It would be more expensive in the long-run for a company such as Fujitsu to do
the support work badly than it would be to do it properly, wouldn't it?
A. Yes!”
Acceptance of Mr Parker’s Evidence
Contrary to the impression given by the drafting of his witness statements, it seemed that
Mr Roll had very little quarrel with Mr Parker’s evidence. Before giving evidence in
Court, he had only looked “briefly” at Mr Parker’s statements and did not think he had
even read all of them.'°
In addition to the passages referred to above, Mr Roll expressly accepted that the
following parts of Mr Parker’s evidence were accurate:
108.1 He accepted! Parker 1 para 25:!°°
It is common within the industry to have a multi-level support model. Generally,
as you move up through the levels of support the cost of the staff providing the
service increases because they are more qualified. Having said that, there is
often overlap of skills between adjacent lines of support and while a team may
be responsible for a particular level of support, staff within that team can have
skills which allow them to perform a role that is more usually performed by the
next level of support.
101 {Day4/39:20} to {Day4/39:25}.
102 (Day4/83:24} to {Day4/84:9}.
"3 (Day3/108:14} to {Day3/108:23}.
101 (Day3/109:3} to {Day3/109:8}.
5 $E2/11/5}.
48
POL00026925
POL00026925
AI8/48
POL00026925
POL00026925
108.2 He accepted! Parker 1 para 26.1.3:1°7
Ifa branch required assistance to attempt to determine the cause of a discrepancy
they would contact NBSC in the first instance. Discrepancies are not unusual in
a retail system. They indicate a difference between the operator's declaration of
cash and stock on hand and the systems calculation and as such are a business
operation issue. However, it was not always possible for NBSC to identify the
cause of a discrepancy. For example, a user may enter a deposit of £100 into a
customer's bank account on Horizon but rather than taking £100 from the
customer, they may make a mistake and give the customer £100 as if it had been
a withdrawal. In that scenario, NBSC would not have been be able to identify the
cause of a discrepancy. Clearly, NBSC is also unable to assist when losses have
been caused by theft.
108.3 He accepted!’ Parker 1 para 26.3:'”
3” line: the SSC also provided 3" line support. The staff that provided 3” line
support had a detailed knowledge of the Horizon application based on
documentation and some inspection of source code. They:-
26.3.1 designed, tested and documented work rounds for the 1° and 2" lines
of support;
26.3.2 applied analytical skills to the symptoms and evidence gathered by the 1
and 2" line functions and undertook in-depth investigation into incidents
(incidents are the basic unit of work for the support team and come from helpdesk
calls and other Horizon support teams);
26.3.3 undertook complex configuration (configuration items can be used to alter
the behaviour of the application) and data fixes which might have required the
generation of special tooling;
26.3.4 designed, wrote and documented new support tools;
26.3.5. undertook source code examination, complex diagnosis and
documentation (including methods to recreate faults) of new application
problems before sending them to the 4 line support group for root cause
software fix; and
26.3.6 provided technical support to other internal Fujitsu teams working on
Horizon.
The 3” line support function used a system called Peak (until 2003 it was called
PINICL) to log and manage incidents passed to them which were suspected to
be faults. It also maintained a Known Error Log (KEL) which describes the
6 {Day3/110:14} to {Day3/110:16}: Mr Roll agreed “From what I remember”.
107 {E2/11/6}
108 {Day3/114:3} to {Day3/114:8}.
109 $E2/11/7}.
AI8/49
POL00026925
POL00026925
symptoms of problems with some analysis of causes, (potential) solutions to the
problems and workarounds that might be needed before a permanent solution
can be implemented. The Peak and KEL systems are still in use today and are
described further in paragraph 62 onwards below.
108.4 He accepted!!° Parker 1 para 26.4:'"!
4" line: 4" line support staff had an intimate knowledge of narrow areas of the
system and were (and are) ultimately responsible for the production of permanent
fixes to repair the root cause of an incident or problem in the live application.
They had knowledge of computer languages which they used to amend source
code to fix problem in the live application code. There was often overlap between
4" line and developers, who added new features into the application.
108.5 He accepted '!? Parker 1 para 26.9:!3
A very small proportion of calls transferred to 4" line support would have
concerned software errors requiring resolution, so it would be interesting to
know the number of calls transferred to them.
108.6 He accepted!" Parker I para 37 (first sentence):!!5
The SSC had (and has) access to view, but not amend, source code.
108.7 He accepted!'® Parker 1 paras 41.2 and 41.3:!!7
a major part of 1° line’s raison d'étre is to deal with user error and therefore the
percentage of issues attributable to user error would be much higher at 1° line;
very few hardware incidents reached the SSC because they were the preserve of
the HSD (i.e. they were relatively easy to spot and therefore filtered out by 1° line
support);
108.8 He accepted''® Parker I para 46:
In paragraph 14 Mr Roll states, "I would reiterate that the main recurring issues
were software issues." It is a symptom of working within a software support team
40 {Day3/114:9} to {Day3/114:24}: Mr Roll: “Broadly speaking, yes”.
M1 (E2/11/8}
12 (Day3/116:13} to {Day3/117:4}.
43 4E2/11/8}.
14 {Day3/142:1} to {Day3/142:10}.
4S (E2/11/10}.
46 Day3/142:1} to {Day3/142:10}.
47 (EQ/11/11}.
48 (Day4/16:15} to {Day4/17:4}.
50
AI8/50
that the majority of issues that come in have been attributed to a software issue
by, for example, a lower line of support. This can lead to a mind set of “look at
all these Horizon errors”, but what this indicates to me is that the previous levels
of support are functioning correctly, removing the majority of other causes (user
/ hardware problems). It does not indicate that the majority of Horizon errors
could
be attributed to software.
108.9 He accepted'!? Parker 2 paras 20-21:!2°
20. Once an issue has been raised, Fujitsu is experienced in providing support
and will go to great lengths to investigate the root cause. In paragraph 61 of my
first statement I explained that Fujitsu use a custom solution, developed and
administered by the SSC, which allows us to record support knowledge into a
Known Error Log (KEL). KELs record support knowledge which is intended to
assist staff in the support and understanding of the Horizon system.
21. Mr Roll's statement that "subpostmasters would have been held responsible
for problems which had not at any time been identified as sofiware errors...
because when they were raised we (Fujitsu) were ultimately unable to identify
the problem at the time" assumes that if Fujitsu was not able to get to the root
cause of an issue, it must have been a software error rather than a human error.
But as I explain in paragraph 15 above, if Fujitsu was unable to identify any
sofiware issues afier carrying out a careful investigation, human error would be
by far the most likely explanation.
109. Mr Roll also accepted, or did not dispute:
109.1 Mr Parker’s evidence concerning the number and nature of calls made to the SSC
3 and 4" line support.!?! He accepted that only a small proportion of calls going
to 4" line support would have required software fixes'?? and that software
problems requiring a software fix represented a tiny fraction of the work handled
by 3“ line support.!”5 He was not in a position to dispute the figures relating to the
number of calls ete put forward by Mr Parker in exhibit SPP1 ‘4 to his first witness
statement.!?5
POL00026925
POL00026925
19 (Day4/43:24} to {Day4/44:21}.
120 (E2/12/7}.
21 {E2/11/8} paras 30-32.
122 (Day3/119:12} to {Day3/119:17}.
228 (Day3/121:20} to {Day3/121:24}.
2 (F/1839}.
25 (Day3/119:22} to {Day3/119:23}; and see {Day3/148:9} to {Day3/155:19}.
Si
AI8/51
POL00026925
POL00026925
109.2 Mr Parker’s evidence'”® that 8.3% of the calls during Mr Roll’s time were
attributed to potential software errors which included duplicates and trivial
problems.'7
109.3 Mr Parker’s evidence’** that since the introduction of Horizon there had been 735
live incidents referring to “payments and receipt mismatch”, that this would
include a lot of duplicates and that Mr Parker’s method of searching for them was
reasonable.'?°
109.4 Mr Parker’s evidence"? that less than 1.5% of incidents going into the SSC were
potentially caused by software incidents.'*!
109.5 Mr Parker’s analysis of closure codes, as dealt at paragraphs 168 - 183 below.
109.6 Further, because Mr Roll could not recall having encountered any coding issue
that caused a financial impact, he was not in a position to say what was done when
that happened!” and accordingly was not in a position to contradict Mr Parker’s
evidence that the standard practice where it is discovered that a bug has caused
discrepancies in branch accounts so as to create an incorrect shortfall or surplus is
to take steps to identify all branches that have been impacted.'*
110. Further:
110.1 Mr Roll’s recollection was that sometimes there were time constraints on how long
the SSC had to find a problem and produce an answer but that occurred “/v/Jery
infrequently”.
126 ¢F/11/11} para 41.3.
27 (Day3/146:13} to (Day3/147:14}.
228 (E 2/11/11} para 42.1.
129 (Day3/169:17} to {Day3/170:15}.
80 {E2/11/12} para 42.3.
131 (Day3/173:20} to {Day3/174:24}
82 (Day 3/179:24} to {Day3/180:12}.
133 (E2/13/3} para 3.
1 May3/171:15} to {Day3/172:2}.
AI8/52
1.
POL00026925
POL00026925
110.2 He accepted Mr Parker’s evidence’*> that the majority of issues reported in a
system, whether to 1", 2™ or 3" line support, are attributable to user action or user
misunderstanding of system functionality.!°°
110.3 He accepted that generally Fujitsu’s investigative and analytical procedures were
thorough, that their diagnosis was correct in the majority of cases and that he was
not suggesting that Fujitsu were happy to assume that SPMs were at fault.!*7
138
110.4 He accepted Mr Parker’s evidence'** that if an issue was causing a financial impact
in a branch’s accounts it would be treated as a high priority and as high impact.’*?
Mr Parker’s evidence is that any increase in priority would not adversely affect the
diligence with which the work was done: Mr Roll said he still felt that some work
was rushed although “7 can’t tell you why”.\°
t,"' which suggests
Mr Roll was asked about paragraph 13 of his first witness statement
that fixes were delayed and branches often affected by coding issues for weeks after they
were identified. In response:
111.1 Mr Roll accepted that multiple upgrades were normal for an IT infrastructure of
this sort and that it was unavoidable that there would be a limited number of time
windows for such upgrades. He agreed that an urgent fix dealing with a high
priority, high impact problem would be treated as an urgent matter requiring a hot
fix and that this would be within two or three days.”
111.2 He accepted that Fujitsu had a mechanism in place to ensure that all branches
affected by a bug would be identified and sorted out.!*8
185 (B2/11/14} para 49.
136 {Day4/17:20} to {Day4/18:3}
187 (Day4/46:7} to {Day4/46:20}.
188 (£2/12/9} para 25.4.
189 (Dayd/86:12} to {Day4/86:22}.
40 (Dayd/86:23} to {Day4/87:2}.
M1 {B1/7/2}.
12 (Dayd/87:10}to {Day4/88:11}.
18 (Dayd4/88:18} to {Day4/89:13}.
53
AI8/53
112.
113.
POL00026925
POL00026925
111.3 He clarified that when he said that developers would sometimes be working on an
out of date version of the code, what he meant was that a later update might undo
the things done by an earlier update — although accepted that that was “rare”.\“4
Functionality in and Features of Horizon
Mr Roll suggested in his amended second witness statement! that it was possible for
there to be a mismatch in figures if data corruption occurred just as a transaction was
being written to disc. In cross examination, he accepted that one of the countermeasures
in Horizon is cyclic redundancy checks:
Q. Well, could I suggest to you, Mr Roll, that where that happens there are cyclic
redundancy checks being constantly done which check to see whether the amount
typed in is the same as the amount that goes into the system. It is something that's
continually done within the Horizon system, isn't it?
A. You are probably right there.
Q. It is done at both levels. We're talking about Legacy Horizon. It is done at the
Riposte level which operates the message store, as it were, and it is done at the
operating system level, the NT which operates -- sorry, the NT, the Windows that
operates the counter itself. There are cyclic redundancy checks that are applied as
these processes are being done at both of those levels, aren't there?
A, I don't remember that precise detail.
Q. But I suggest to you that that is the case and are you in a position to dispute that?
A. I'm not in a position to dispute that, no.'°
Mr Roll accepted that what he had said in paragraph 14 of his amended second statement
was purely hypothetical, that he could not remember a situation where a cyclic
redundancy check had missed an error of that sort and that such a check would both send
an event to the SSC and prevent the counter actually accepting the transaction. '”
M4 Day4/90:3} to {Day4/90:20}.
MS (E1/12/5} para 14.
M6 {Day4/10:2} to {Day4/10:20}.
447 Mayd/11:15} to {Day4/12:23}.
54
AI8/54
He also accepted that his evidence in paragraph 8 of his first witness statement'** — that
if an error was referred to the 3“ line support it was extremely unlikely to be due to a
Mr Roll agreed that “the vast majority of problems that were caused by the system would
have manifested themselves in some kind of reporting”.'°° He agreed that it would be
what Dr Worden says about zero sum baskets in paragraph
156 of Worden 11°, he had misunderstood what Dr Worden was saying.'** He could not
in fact think of an example where a non-zero-sum basket was accepted into Horizon.'*>
He also agreed that any true transaction integrity issue would be picked up in
reconciliation reporting and investigated.'*° Post Office’s case is that it would also have
been visible to the branch when they balanced at the end of the trading period.'*7
114,
mistake made by a SPM — was wrong.!”
115.
very rare for a bug not to be detected.'*!
116. He agreed that in criticising!
Hardware Failures
117.
Mr Roll’s evidence was that he was involved in a hardware failure once a month,!°*
although when asked during cross examination, he could not recall how they would
affect branch accounts.'*? He accepted that the example he gives in paragraph 7 of his
second witness statement, concerning a stuck transaction, was very rare and would have
been spotted by Fujitsu by its own monitoring.'©°
POL00026925
POL00026925
M8 (E1/7/2}
49 Day4/15:3} to {Day4/15:10}.
50 (Day4/42:10} to {Day4/42:13}.
St (Day4/43:7} to {Day4/43:16}.
52 {B1/10/3} para 9.
153. (3/1/38}
154 (Day4/48:16} to {(Day4/49:17}.
155 (Day4/50:12} to {Day4/50:25}.
186 {Day4/51:23} to {Day4/52:9}.
1S? Parker 2 para 10 {E2/12/4}.
'S® (E1/10/2} para 6.
189 (Day4/92:15!} to {Day4/92:19}.
160 Day4/92:20} to {Day4/95:6}.
5S
AI8/55
POL00026925
POL00026925
118. Mr Roll accepted that problems with peripheral devices, such as keyboards, were
unlikely to affect branch accounts.'°!
119. In paragraph 8 of his second witness statement! Mr Roll gave an example about branch
data not being replicated from a mobile post office correctly. In cross examination, he
had this to say about it:
Q. I see. So if the problem caused a persistent failure to replicate, that is a problem
that
A, Yes.
Q. -- was always going to be spotted?
A, Yes.
Q. So when you say it wouldn't necessarily be spotted, that's because the system itself
might deal with it appropriately?
A. Yes.
Q. But if it didn't, it was always going to be spotted?
A. Yes.
Q. Very good. And in this particular case the problem was spotted and the data was
replicated, yes?
A. Yes.
Q. And again, that would always be the position, wouldn't it: once the problem is
spotted this is very easy to deal with?
A. In this case, yes.
Q. Well, in all cases of this sort, yes?
A, Yes.
Q. Thank you!
61 May4/95:8} to {Day4/95:21}.
12 (E1/12/2}.
16 (Day4/97:2} to {Day4/97:22}.
56
AI8/56
POL00026925
POL00026925
120. Mr Roll was taken to the various Peaks associated with this particular issue.’ Mr Roll
suggested that there was an occasion where there had been some sort of cover-up about
a batch of faulty laptops.® He speculated that this was done as some sort of favour
although accepted that he had no basis for this.’°° He accepted that it was not in the
interests of the SSC or of Fujitsu for such machines to remain in circulation: and that
once again, all Mr Roll could offer was an “impression” about what had happened.’ It
is submitted that reliance cannot be placed on evidence of this sort.
121. In summary, the impression that Mr Roll gave during live evidence was hugely different
to the impression set out in his two witness statements. Mr Roll agreed that:
121.1 Coding errors that caused a financial impact on branch accounts were extremely
rare.'
121.2 He could not recall a coding issue that definitely caused a financial impact.’
121.3 In most cases the right diagnosis was reached.'”°
121.4 Mr Parker’s description of the careful, thorough and conscientious procedures
followed by Fujitsu were correct.
Remote access
122. Mr Roll’s evidence on remote access has been largely superseded by that of the experts.
Remote access is addressed in greater detail at paragraphs 184 - 191 below. The
paragraphs that follow set out Mr Roll’s evidence specifically on remote access.
164 (F/197} & {F/201}.
465 {E1/10/2} to {E1/10/2} para 8.
166 {Day4/108:18} to {Day4/109:10}.
161 (Day4/109:20} to {Day4/110:22}.
168 (Day 3/167:5} to {Day3/167:12}.
16 (Day 3/179:13} to {Day3/180:2}.
10 (Day 4/45:12} to {Day4/46:20}.
57
AI8/57
123.
124.
125.
Mr Roll accepted that a distinction was to be drawn between what can referred to as
“transaction data” on the one hand, i.e. data relating to transactions which actually has
an impact on branch accounts, and “operational data” on the other, which may have all
sorts of functions, but is not seen by the SPM and does not have a financial impact on
the branch accounts.'”!
Mr Roll’s evidence concerning certain aspects of remote access was difficult to follow.
Despite the technical complexity of the subject, it is of primary importance to understand
that what is being referred to is something which only happened very rarely — which is
of central importance to this trial. (A good example of the sort of process which Fujitsu
followed is provided by a document which Cs asked to be added to the trial bundle
shortly before Mr Coyne started his evidence'”: this is considered in detail below in the
context of the expert evidence).
Notwithstanding the distinction agreed between transaction data and operational data,
Mr Roll appeared to think that, for example, the process by which a counter is unlocked
~ which involved the flipping of a bit from 1 to 0 — did involve a change to transaction
data.'7* Nevertheless he accepted!” that paragraphs 35 and 37 of Mr Godeseth’s witness
statement!”> were accurate:
35. All counter data was held in a bespoke message store (which was part of the
Riposte product supplied by Escher Inc.). This data was replicated within each
branch to all counter positions and from each branch to the data centres where
it was held in the correspondence server message stores. Similarly, any data
inserted into the message store at the data centre (for example reference data or
authorisations for banking transactions) would be replicated back to the branch
counters. Selected data was then extracted from the correspondence servers to
update Post Office's back end systems.
37. All accounting at the counter was carried out based on the data held in the
message store. The Riposte product managed the message store and it did not
allow any message to be updated or deleted, although it did allow for data to be
archived once it had reached a sufficient age (this varied by message type and
POL00026925
POL00026925
171 (Day4/3:19} to (Day4/4:24}.
24/377.)
"8 (Day4/114:17} to {(Day4/115:12}.
1 (Day4/115:21} to {Day4/117:2}.
YS FED/I/I1}.
58
AI8/58
126.
127.
128.
POL00026925
POL00026925
also over time; it was never less than 34 days and by 2009 it was effectively 804
days).
It is right to point out that Mr Roll did refer in his cross-examination to a particular form
of remote alteration of data which apparently involved flipping one bit of data.'”° Post
Office believes that what Mr Roll was probably referring to are the “marooned
transactions” which Mr Parker refers to in his second witness statement!” but if Mr Roll
was talking about something else (i.e. some sort of remote access used to download,
correct and re-insert corrupted data), then it is not mentioned by the experts and it is
inconsistent with what Mr Godeseth said in paragraph 37 of his witness statement
(quoted above).
Mr Roll accepted Mr Parker’s evidence!”® that if Fujitsu were to change anything it
would be the envelope around the transaction data.!” Mr Roll also accepted that Fujitsu
would never, for example, change a line of code to change transaction data from £100 to
£10.'%° And although Mr Roll said that his recollection was that Fujitsu would not change
more than the envelope data, he accepted that this “was a long time ago”.'*! Post
Office’s case is that no one at the SSC would manually change a line of transaction data
(i.e. the nature and terms of the relevant transaction) and then reinsert that changed line
of transaction data into the message store (the closest equivalent of which now is the
BRDB or the Audit Store) of any branch.
Although Mr Roll’s recollection was that there were times when Fujitsu needed to carry
out work for the SPM while the SPM was already logged on, there was a strict procedure
in place which was taken very seriously:
Q. So you can't think of a specific reason why it would have to be the same person,
but you're saying that it did sometimes?
A. Yes, it -- sorry.
Q. I didn't let you finish.
16 Bg. {Day-4/:136:21} to {Day4/137:9}.
177 (£2/12/2} para 5.
178 (£2/12/12} para 38.2.
179 (Day4/122:5} to {Day4/122:13}
180 (Day4/123:4} to {Day4/123:12}.
81 (Day4/123:15} to (Day4/123:23}.
AI8/59
POL00026925
POL00026925
A. I have lost my train of thought now, sorry. It often made it much cleaner for
accounting reasons, from what I remember, if it was the same user ID. All of this --
all of these actions would be detailed in the PINICL and if -- from what I remember,
if you were accessing a counter in this way, two people had to be there, one as an
independent witness to make sure that everything was going correctly.
Q. So there would have to be what we now call PEAKs and there would have to be
two pairs of eyes
A, That was what
Q. -- it would never be left to one particular member of the SSC team to do it on his
own?
A. It was never supposed to be and I don't think it ever was but I'm not sure.
Q. So this is a formal process then, is it —
A, Yes.
Q. -- which the SSC took very seriously?
A. It was developed and taken very seriously, yes.
Q. And is it also the case that Post Office consent was always needed for this kind of
process?
A. When I was there we were supposed to speak to the postmaster to get his consent.
So from Post Office consent, that's what I believe you mean by that. Formal consent
from the Post Office itself, maybe not.!*°
129. Mr Roll also confirmed that if Fujitsu were to log on to a counter remotely they were
supposed to speak to the SPM and get their consent, and that he ensured that these
protocols were properly followed.'** He also accepted that his evidence that some errors
were corrected remotely without the SPM being aware'* was not referring to errors that
affected branch accounts:
O. Thank you. So in paragraph 16 of your first witness statement {E1/7/3}, you say:
"Still on the subject of remote access to branch systems, as I recall some errors were
corrected remotely without the subpostmaster being aware." Those errors are not
errors -- or rather those corrections were not corrections which changed branch
accounts in the way that we discussed?
A. No.
2 (Day4/127:15} to (Day4/128:19}.
8 (Day4/129:6} to {Day4/129:24}.
84 (E1/7/3} para 16.
60
A/8/60
POL00026925
POL00026925
Q. You're talking about other errors, aren't you?
A, Yes.
Q. Could you give some examples of the kind of errors you are talking about?
A. I:can't remember I'm afraid.
Q. But would it be things like changing configuration items?
A. Probably, yes.
Q. That sort of thing, which would not have an impact on the branch accounts in the
way that we have previously discussed?
A. I think so, yes.!>
130. Mr Roll stated that the SSC would only remotely access a counter where potentially
131.
catastrophic circumstances might arise but cannot name any instance of that.'*° Mr Roll
also accepted that he believed that the situation he describes in paragraph 15 of his first
187
witness statement i.e, that Fujitsu would frequently access a post office counter
remotely, only applied in circumstances where the cyclic redundancy check had been
triggered so that the system had identified that some attention was required.'** Although
he thought it was possible to insert a transaction, he had not done so himself.!*°
Mr Roll’s recollection was that very occasionally — he thought for him it was perhaps
190
every couple of months'”® — Fujitsu had to repair corrupt data. He accepted that this was
19! and that the preference
something which Fujitsu did only when they absolutely had to
was always to use the correspondence server i.e. to make the change through the front
door.
85 (Day4/130:19} to (Day4/131:14}.
186 (Day 4/158:23} to {Day4/159:23}.
87 (E1/7/3} para 15.
188 (Day4/131:15} to {Day4/132:1}.
89 (Day4/135:9} to {Day4/135:19}.
90 (Day4/139:20} to {Day4/139:22}.
1 (Day4/140:9} to {Day4/140:15}.
2 (Day4/147:22} to {Day4/148:4}.
61
A/8/61
132. It was suggested to Mr Roll that if the SSC did need to insert transactions, it was the
practice to use additional properties so that this could be identified. Mr Roll said that was
not his recollection but accepted that he might have overlooked or forgotten this.'°°
133. Mr Roll did accept that there were processes in place to minimise the risks when copying
data from a counter — and that that was something he had not covered in his witness
evidence.'* He also accepted that when the data was inserted there would be another
person sitting there making sure that matters were completed correctly.'°> He added that
the SSC did not like carrying out fixes that required remote access.!*° Further, that any
instance would be documented.'*” He also accepted the following:
Q. So it is absolutely standard SSC practice, isn't it, when you are dealing with a
branch and correcting problems that the branch has got that the branch will see, it is
absolutely standard that you will communicate with the subpostmaster to ensure that
the subpostmaster knows what changes you are making?
A. Yes.!%
134. As with other parts of Mr Roll’s evidence, the impression which his witness statements
sought to create is far removed from the position which emerged during cross-
examination. In summary, as set out above, Mr Roll accepted that:
134.1 SSC did not like carrying out fixes that involved remote access;
134.2 When they did use remote access, procedures were in place which were
conscientiously followed which included documenting and witnessing where
appropriate;
134.3 When he referred to errors being corrected remotely without the SPM being aware,
he was not referring to errors that affected branch accounts;
3 (Day4/154:19} to (Day4/153:9}. For examples of where this happened, see {F/485/3}.
© ay4/156:10} to {Day4/157:6}.
5 (Day4/157:7} to {Day4/158:9}.
96 {Day 4/147:22} to {Day4/148:4}.
197 (Day 4/148:5} to {Dayd/148:15}.
198 (Day4/159:17} to (Day4/159:23}.
POL00026925
POL00026925
AI8/62
135.
136.
137.
134.4 SSC would not change a line of code so that transaction data went from, say, £100
to £10.
E2. Mr Godeseth’s Evidence
Mr Godeseth is Fujitsu’s Chief Architect on the Post Office account and he has worked
on Horizon in various roles since it was first procured. Much of his evidence is
uncontroversial and unchallenged. However, some of it is unsatisfactory. With the
benefit of hindsight, Post Office would not have asked Mr Godeseth to cover several
matters that were addressed in his first two witness statements — although it is right to
point out that if Post Office had only called first-hand evidence, the trial would have
been wholly unworkable.!°?
Before considering Mr Godeseth’s evidence, it may be helpful to consider the
circumstances in which Post Office asked him to provide evidence.
Background
The Horizon Issues schedule” provided that the Horizon Issues require limited if any
evidence of fact. This reflected the Court’s guidance regarding the proper scope of the
non-expert evidence to be called at the Horizon Issues trial (“My intention is in March
to resolve the Horizon Issues that ... do not require evidence of fact or if they do require
the very barest evidence of fact”)2°! The four-week Horizon Issues trial was to be an
expert-led process and it was anticipated that most of the time for evidence would be
devoted to the experts. This suggested that the parties would only have limited time for
factual witnesses.
POL00026925
POL00026925
189 Post Office calculates that this would have required calling approximately 34 additional witnesses
200 i.e, Schedule 1 to the Order dated 23/03/18 {C7/14}.
2! Transcript of the CMC on 22 February 2018 at p.54E {C8.4/4/54}.
63
AI8/63
Godeseth 1, 27 September 2018
138. Post Office wanted to provide a simple and uncontroversial overview of Horizon and its
relevant features. It recognised that it was not possible for one person to have a complete
understanding of all the corners of the Horizon system?” but, on the basis that there
would not be room in the timetable for multiple witnesses, it took the view that this
overview should be provided by one person. Two possible candidates were Torstein
Godeseth and Gareth Jenkins. Taking into account the involvement that Mr Jenkins had
had in a number of criminal prosecutions that are currently being looked at by the
Criminal Cases Review Commission (e.g. the Misra case), Post Office asked Mr
Godeseth to do so.
139. This resulted in Mr Godeseth making his first witness statement in September 2018.7°
As he indicated in that statement, he had consulted colleagues to ensure that his
understanding was correct.?** Thus, as he had not worked for Fujitsu when Legacy
Horizon was in operation, he had consulted Mr Jenkins in relation to the Riposte
system.” In addition, at Post Office’s request, he included some figures about ARQs
issued between 2014/15 and 2017/18 which were provided by Jason Muir, Fujitsu’s
Operational Security Manager.?°°
140. It subsequently became apparent that, when addressing the forms of remote access that
were possible with Riposte, the possibility of injecting transactions directly into counters
had been overlooked, rather than into the correspondence server with a counter position
of greater than 32.?°7 Fujitsu was reminded of this possibility by Roll 2, served on 17
January 2019.7°° Accordingly, Mr Godeseth corrected his evidence on 28 February
2019.2” As it related to the detailed operation of the Riposte system, it was not something
POL00026925
POL00026925
282 As Mr Godeseth notes in Godeseth 1, para 7 {E2/1/2} and again in Godeseth 2, para 8 {E2/7/1}.
23 Godeseth 1 {E2/1}.
2" Godeseth 1, para 7 {E2/1/2}.
25 Godeseth 1, para 34 {E2/1/11}.
26 Godeseth 1, para 31 {E2/1/10}. Post Office could of course have provided these figures by way of further
information. ARQ figures for the period back to 2004 have now been provided in correspondence:
{H/301}and {H/332}.
287 See also Parker 1, paras 53 - 57 {E2/11/15} to {E2/11/17}.
28 Roll 2, para 20, {E1/10/6}.
2° Godeseth 3, para 25 {E2/14/7}. Mr Parker had already made the position clear on 29 January 2019: Parker
2, para 27 {E2/12/9}.
64
AI8/64
POL00026925
POL00026925
which was within his own knowledge when he addressed it in Godeseth 1, para 34. This
could and should have been made clearer.
Godeseth 2, 16 November 2018
141. On 16 October 2018, Coyne I was served. This contained innumerable actual or apparent
criticisms of or queries regarding the Horizon system and various Post Office operations
to which Post Office wished to respond. There was little time in which to do so: even
with an extension agreed by Cs, Post Office had only a month in which to produce 7
witness statements.”'” Consequently, these statements were prepared under considerable
time pressure.
142. Godeseth 2 was one of these statements.”!! This:
142.1 addressed various matters raised by Professor McLachlan (an expert who had
acted for Mrs Misra in her criminal trial and on whom Cs appeared to be secking
to rely as a second expert), including the Callendar Square bug. In the event,
Professor McLachlan was not called and so this evidence did not need to be
adduced, but Post Office did not know this at the time: paras 8 to 29;
142.2 gave more details about global branches: paras 30 to 33;
142.3 gave short accounts of how the receipts and payments mismatch bug, the local
suspense bug and the Dalmellington bug came to light and were resolved: paras
35 to 61;
142.4 explained why certain process metrics/performance indicator documents that Mr
Coyne believed Fujitsu should have did not in fact exist: paras 62 to 68; and
142.5 in response to a query raised in Mr Henderson’s witness statement, explained the
distinction between journal sequence numbers (“JSNs”) and session sequence
numbers (“SSNs”): para 69.
21 Pursuant to para 11 (a) of the Fourth CMC Order {C7/18/3}; {C7/18/4}, Post Office’s supplemental
evidence was due by 16 October 2018, but in the light of Coyne 1, Post Office sought and Cs agreed an
extension to 16 November 2018 {H/136}.
21) {E 2/7}.
65
AI8/65
143.
144,
145.
POL00026925
POL00026925
As was made clear in the statement, in certain respects Mr Godeseth’s evidence was
based on information provided by others. His account of the Misra trial was based on
information provided by WBD and Mr Jenkins; his accounts of the Callendar Square,
receipts and payments mismatch, local suspense and Dalmellington bugs were based on
the contemporaneous documents and discussion with Mr Jenkins and on one point of
information provided by Matthew Lenton, Fujitsu’s Post Office Account Document
Manager; his account of the documents held by Post Office was based on information
provided by Steve Bansal, Fujitsu’s Senior Service Delivery Manager.
Cs understandably complain that Mr Jenkins and the other source of Mr Godeseth’s
information could have given some of this evidence first hand. However:
144.1 Taking into account that Professor McLachlan’s evidence specifically addressed
things said or done by Mr Jenkins in relation to the Misra trial, Post Office was
concerned that the Horizon Issues trial could become an investigation of his role
in this and other criminal cases.
144.2 Moreover, Post Office was conscious that if it only adduced first hand evidence in
the trial, it would end up having to call more witnesses than could be
accommodated within the trial timetable.”
144.3 Furthermore, so far as Post Office was aware, the relevant parts of Godeseth 2
were most unlikely to be controversial. For example, the Misra trial was a matter
of public record, the four bugs were covered by contemporaneous documentation
and Post Office had no reason to doubt Fujitsu’s account of the documents it held.
Godeseth 3, 28 February 2019
Godeseth 37!* was prepared in response to certain questions raised by Coyne 2 regarding
global branches, the TIP repair tool and remote access. It was not based on information
provided by others, except where identified at paragraph 11 of Godescth 3 and its
correction in para 25 to Mr Godeseth’s previous assertion that a transaction insertion in
2!2 By this stage, it seemed that Cs were calling 8 witnesses, and having originally served 4 witness statements
Post Office was now proposing to call 10 witnesses. As noted above, had its witnesses only given first hand
evidence, Post Office estimates that some 34 additional witnesses would have been required.
23 {2/14}.
66
AI8/66
POL00026925
POL00026925
Legacy Horizon would record a counter number of greater than 32. Paragraphs 14 & 15
and references to 'APPSUP' in Godeseth 3 were discussed with colleagues, however Mr
Godeseth had discussed these so often that he felt they were within his own knowledge.
Conclusions on Mr Godeseth’s written evidence
146. As matters stand, it appears that the contents of Godeseth 1 and Godeseth 3 are largely
unchallenged. However, Godeseth 2 is challenged. In Mr Godeseth’s cross
examination, some of the points he made on the basis of information provided by others
were shown to require correction or at least clarification. This took Post Office by
surprise. With the benefit of hindsight, Post Office accepts that there are points on which,
if it wished to adduce any evidence at all, it should have ensured that witness statements
were prepared for the individuals who were the sources of the relevant information. Had
such witness statements been prepared, the errors contained in Godeseth 23 may well
have been avoided.
147. The errors in Godeseth 2 were significant. They include:
147.1 In para 13,?'4 Mr Godeseth said that the Callendar Square bug occurred in 2005,
without making it clear that this was not its first occurrence and that the underlying
problem in Riposte had probably been in existence since 2000. A similar error was
made in para 36,7!
where he said that the receipts and payments mismatch bug
caused a mismatch in some branches in September 2010, when in fact it also
caused mismatches between March and October 2010.
147.2 In para 15,?'° he said that he understood from Mr Lenton that Fujitsu had, using
event logs, established for the purposes of Godeseth 3 how many branches the
Callendar Square bug affected. Although Mr Lenton may have been asked to
establish this, what he in fact did was locate a spreadsheet that was not prepared
for the purposes of Godeseth 3 but had been prepared by Anne Chambers in
2015.” Moreover, the document was not an exhaustive list of all the branches
24 469/7/3}
215 (E 2/7/10}.
216 49/7/53.
217 {F/322.1}.
67
AI8/67
POL00026925
POL00026925
affected, it identified the majority that had reported a problem or caused a
reconciliation report. Para 15 should have been redrafted to reflect these points.
147.3 At the beginning of his oral evidence, he corrected the number of branches referred
to in para 157!* to reflect the fact that one of the branches identified in Ms
Chambers’ spreadsheet was affected twice.”'? However, although the correction
was right, in cross examination it became apparent that he had not checked it
himself. He did a similar thing regarding the number of branches affected by the
receipts and payments mismatch bug referred to in para 42””° — the correction was
right (there were 64 impacts affecting 62 branches)” but he had not checked it.
This should not have happened.
147.4 In para 63,?” he appeared to be saying that Fujitsu’s Post Office Account Customer
Service Problem Management Procedure document?”*
was not implemented
following Mr Salawu’s departure as Horizon Head Lead Service Delivery
Manager, when in fact it was merely section 1.4 of that document that was not
implemented.
148. These things should not have happened. Post Office therefore accepts that Mr Godeseth’s
evidence was unsatisfactory. But without wishing to play down the significance of the
errors that were made, it also notes that the vast majority of the written evidence given
by Mr Godeseth is either incontrovertible or not controverted.
Mr Godeseth’s oral evidence
149. Mr Godeseth’s oral evidence lasted from {Day7/73:!} to {Day 8/124:1}. Post Office
notes he was cross-examined on a number of areas which could more usefully have been
218 E2/7/5}.
219 {F/322.1}: rows 20 and 22.
20 ED/T/L1}.
21 This can clearly be seen from {F/754.1}: Branch 208020 appears twice at rows 13 and 51 and branch
159632 appears twice at rows 3 and 62.
222 {E2/7/16}.
23 {F/1692}.
68
AI8/68
POL00026925
POL00026925
addressed with different witnesses, in particular Mr Parker.” In this section, Post Office
does not seek to anticipate the arguments that Cs will make on his oral evidence, but it
notes as follows:
149.1 Regarding the change from Legacy Horizon to Horizon Online, Mr Godeseth was
clear that the changes were geared towards “refreshing the solution”> He
confirmed that Horizon Online was not an end of life version of Legacy Horizon
but was a rejuvenated version, using the words “radical”, “pretty dramatic” and a
“big overhaul” to describe the change.”’°
149.2 Mr Godeseth explained why it was not possible to maintain an operating version
of Legacy Horizon available for inspection.”””
149.3 When the so-called $1,000 Peak”** and its associated OCR and OCP were put to
him,””? Mr Godeseth agreed that the form of remote access used in that case caused
a loss at the branch.**° However, this was because — no doubt by mistake — his
attention was not drawn to the fact that the document put to him for this purpose
was an OCR which involved a change to the data in Post Office’s back-end
systems (POL FS) and no change to branch data in Horizon. Post Office was not
alive to this mistake at this time, but fortunately it was able to address the point
with Mr Coyne.”*!
24 See for example {Day7/150:}} to {Day7/164:! } (Problem Management); {Day7/166:!} to {Day7/167:)}
(Dalmellington; Low Level Design Document for Transaction Correction Tool); {Day7/i86:!} to
{Day7/186:10} (the difference between OCPs and OCRs); {Day8/8:18} to {Day8/9:16} (informing SPMs
of remote access to their accounts); {Day8/12:24} to {Day8/14:5} (whether the forms of remote access
remote used in the $1,000 Peak {F/432} and recorded in the OCR and OCP at {F/434.1} and {F/432.2}
caused a discrepancy to the relevant branch’s accounts); {Day8/41:21} to {Day8/42:3} (what Anne
Chambers meant when she said “go off piste” in the Appsup Peak {F/768}; and {Day8/78:6} to {Day8/78:23}
(what Post Office communicated to SPMs about bugs and why).
25 (Day7/108:24} to {Day7/109:22}
26 {Day7/127:12} to {Day7/128:12}.
21 {Day8/48:2} to {Day8/48:13}.
28 (F/432}.
29 (F/432.1} and {F/432.2}
20 {Day8/12:24} to {Day8/14:5}.
21 {Day16/103:1} to {Day 16/132:6}.
AI8/69
POL00026925
POL00026925
149.4 On the basis of a letter from WBD regarding the audit log to which the Transaction
Correction Tool writes (BRDB TXN CORR TOOL JOURNAL),”” Mr Godeseth
agreed that the Transaction Correction Tool must have gone beyond its original
design.** However, on the basis of a clarificatory letter written by WBD on 29
May 2019,*** it can be seen that this agreement was based on a misconception.
However, this point need not be pursued, since the experts are agreed that the
Transaction Correction Tool has only been used to change branch transaction data
on the one occasion discussed in Godeseth 1.?*°
Conclusions on Mr Godeseth’s oral evidence
150. Post Office submits that in his oral evidence, Mr Godeseth gave open, frank and direct
answers.
E3. Mr Parker’s Evidence
151. The relevant evidence:
151.1 Mr Parker’s first witness statement dated 16 November 2018 {E2/11}, with
corrections at {E2/16}.
151.2 Mr Parker’s second witness statement dated 29 January 2019 {E2/12}, with
corrections at {E2/16}
151.3Mr Parker’s third witness statement dated 28 February 2019{E2/13}, with
corrections at {E2/17}
151.4 Transcript: {Day12/2:20} to {Day12/97:24}
152. Mr Parker is employed at Fujitsu as Head of Post Office Application Support. Mr
Parker’s written evidence deals largely with the role of the SSC, Mr Roll’s evidence,
22 (H/218}.
283 {Day7/177:19} to {Day 7/179:10}
2 (11/302).
5 See JS4, para 12.2 {D1/5/14} and {Day16/140:1} to {Day 16/140:4}.
70
A/8/70
153.
154,
155.
POL00026925
POL00026925
remote access and, more generally, factual matters relevant to the experts’ consideration
of the Horizon Issues.
Much of Mr Parker’s evidence does not appear to be controversial. Large sections of it
were confirmed by Mr Roll (as identified above). Mr Parker was cross-examined fairly
extensively, but he was not challenged substantially on the core of his evidence as to how
the SSC worked and, in particular, the diligence with which it addressed potential
software issues that might have adverse effects on branch accounts. That is nota criticism
of Cs or their Counsel — it simply reflects that Mr Parker was able to give useful and
reliable evidence as to the SSC’s practices, which evidence is supported in large part by
the contemporaneous documents reviewed by the experts and by Mr Roll’s oral
evidence.
Peaks and KELs
Mr Parker helpfully explains that Peaks are records of potential issues in Horizon that
may need to be addressed by the SSC and development teams.” As noted above, many
issues addressed in Peaks ultimately turn out not to be caused by bugs in Horizon, but
that often cannot be ascertained without fairly intensive investigation. There is ample
evidence of such investigation in the thousands of Peaks considered by the experts.
As addressed in Post Office’s opening submissions,’ Mr Parker’s first witness
statement sets out in an Appendix a table that addresses 58 KELs referred to by Mr Coyne
in Coyne 1, providing Fujitsu’s comments on those.”** Importantly, the Fujitsu analysis
includes the question of whether the problems addressed in the KELs had any financial
impact on branch accounts and, if so, whether that impact was a lasting one or merely
temporary (because it would be corrected through the application of what Dr Worden
describes as Horizon’s robustness countermeasures). Of the 58 KELs considered, the
following conclusions are drawn:
26 Parker 1, para.62 {E2/11/20}.
27 {A/2/41} to {A/2/42}.
28 Appendix 1 at {E2/11/23}.
71
A/8/71
155.1 There are 39 that had no financial impact (including those which would have had
no impact if recovery process was followed correctly in the branch).?*”
155.2 There are 12 that had or might have had a temporary financial impact, but this
would have been resolved either by mechanisms within Horizon or through Post
Office’s client reconciliation processes.”4°
155.3 There are 4 that had a financial impact that would not inevitably have been
resolved: these relate to the Payments Mismatch and Local Suspense Account
bugs. These bugs were detected, and the shortfalls caused by them were ultimately
resolved by Post Office.”*!
155.4 There are 3 that cannot be addressed further because the KEL is not available, or
was generic or is not known duc to its age."
156. These are of course not the views of an independent expert, but they can be given
considerable weight given that Mr Coyne does not dispute the accuracy of the Fujitsu
analysis in Coyne 2 and Dr Worden’s own review of the KELs was broadly consistent.?“°
Mr Coyne did not re-visit in Coyne 2 the points that he made in reliance on those KELs
in Coyne 1. His reliance on them, aside from the 4 mentioned above, seems to have
largely fallen away. Mr Coyne’s evidence in this regard is addressed elsewhere in these
closing submissions.
157. In addition to presenting the KEL analysis, Mr Parker also addressed automatic reporting
of system exceptions and other indicators of potential problems. Both Fujitsu and Post
Office monitor the system in different ways and using automatically generated reports.
As a result, many bugs may be detected without any SPM whose branch may be affected
having to report a problem to Post Office. Mr Parker refers to the BIMS process for
POL00026925
POL00026925
5° Ttems: 2, 3, 8,9, 10, 11, 13, 14, 15, 16, 17, 18, 19, 23, 26, 27, 28, 29, 30, 33, 37, 39, 40, 41, 42, 43, 44, 45,
46, 47, 48, 49, 51, 52, 53, 54, 56, 57, 58.
Ttems: 4, 5, 6, 7, 12, 20, 21, 22, 31, 35, 36, 38.
41 Ttems: 1, 32, 50, 55.
*? Ttems: 24, 25, 34.
243 4 (privileged) draft of Dr Worden's analysis was provided to Fujitsu during the course of Mr Parker
preparing his evidence and his review of KELs, hence the similarity of Fujitsu's approach to that previously
taken by Dr Worden.
AI8/72
158.
159,
160.
161.
POL00026925
POL00026925
flagging discrepancies raised in automated reports and drawing them to Post Office’s
attention for manual reconciliation.” There are many BIMS reports in evidence, and the
Court will recall Mrs Burke being taken through the report that was used to identify the
need for a TC in her case.
Mr Parker also addresses Horizon’s operations, audits and data extractions.”4> Mr Parker
was not challenged on this important evidence. It appears to be largely uncontroversial.
“User error bias”
Cs have focused on user error in their opening submissions and during the cross-
examination of other witnesses. The thrust of Cs’ case here appears to revolve around
the concept of “user error bias”. Cs have adduced no evidence to support that this is a
known concept, the concept is not addressed by either expert, and there is no evidence
on it from any of Cs’ other witnesses. Dr Worden had never seen the acronym “UEB”
before this case.?"° It appears to be a line of attack devised by lawyers. This line of
argument should be dismissed as lacking any evidential support.
If“UEB” was to form part of Cs’ case in relation to the SSC’s functions, it was incumbent
on them to at least raise the point with Mr Parker. In the event, it was not suggested to
Mr Parker that the SSC suffered from “UEB” or even that he had seen any evidence of
bias in Post Office’s treatment of problems raised by SPMs.
Mr Parker was, however, asked whether software errors may look like user errors.74” It
is uncontroversial that the symptoms of bug may be very similar to the symptoms of
user error. Mr Parker commented that if errors that looked like user errors were
reoccurring, one would immediately think there may be more to this and investigate.”"*
That is consistent with the documents reviewed by the experts, many of which show
Fujitsu working hard to uncover the cause of a discrepancy that, on the face of it, could
well have resulted from user error.
2 Parker 3, para.10 {E2/13/3}.
5 Parker I {E2/I1}and Parker 2 {E2/11}.
%6 {Day19/146:20} to {Day19/147:13}.
27 (Day12/39:1} to (Day12/39:14}.
8 {Day12/39:1} to {Day12/39:14}.
73
AI8/73
162.
163.
164,
POL00026925
POL00026925
On the face of it, Cs’ case on “UEB” appears to be limited to the contention that some
Post Office employees were sometimes overly quick to diagnose user error, rather than
that Fujitsu too suffered from some bias in favour of finding user error. The Court will
recall Mr Roll’s evidence to the effect that Fujitsu would seek to bottom out problems
reported to it, rather than being happy to put the problem down to user error: see above
at paragraph 108.9. Fujitsu (even more so than Post Office) had a strong incentive to get
to the bottom of and resolve system errors, rather than leaving them to keep arising and
taking up support time and resources. Fujitsu would, even if acting purely in self-interest,
wish to find and fix errors in the system, lightening its own workload and improving its
product and its relationship with Post Office, a very valuable customer.
SLAs
Mr Parker covered SLAs briefly in his evidence. He stated that the team did not have
SLAs but OLAs.” He also confirmed that these did not apply to software faults — only
for hardware.’° Mr Parker was not questioned as to whether the existence of SLAs
and/or OLAs had an impact on the care taken by Fujitsu to resolve issues. Mr Roll’s oral
evidence, addressed above, was to the effect that there was no adverse impact on the
SSC’s approach to its work.
Role of the SSC and Mr Roll’s position within the SSC
Mr Parker was deputy manager of the SSC at the time Mr Roll was employed, although
did not have that formal title.?*! He explained that he would stand in for the manager in
his absence.”*? He would also make decisions on approving actions for him and other
253
operational decisions in general.*°’ He was very well-placed to provide an overview of
the SSC’s work and practices.
9 {Day12/69:4} to {Day12/70:1}.
289 {Day12/69:4} to {Day12/70:1}.
1 Parker 1, para.8 {E2/11/2}.
282 {Day12/24:25} to {Day12/25:13}
283 (Day12/24:25} to {Day12/25:13}.
74
AI8/74
POL00026925
POL00026925
165. Mr Parker stated that the level of skill, knowledge and experience varied from person to
person in the SSC.?** Experience varied, including whether that person had detailed and
deep knowledge of the system based on documentation and source code inspection.?°°
Mr Parker stated that Mr Roll did not have this detailed knowledge,” which is consistent
with Mr Roll’s own oral evidence to the effect that he was not one of the “super elite”
within the SSC. Training also varied, including whether a person was trained in the
287
coding languages used within the application.
166. Mr Roll was ina relatively junior position in the SSC team, which performed 2™ and 3%
line support. He was primarily focused on Operational Business Change.”°* He was not
working at a level where he would be required to review much code and did not play any
significant part in extensive source code examination.’*°
167. Mr Parker confirmed in oral evidence that his witness evidence was an accurate
reflection of Mr Roll’s skills. If Cs had obtained the impression that Mr Parker was
trying to criticise Mr Roll in any way, that impression is unwarranted — Mr Parker’s
evidence was directed at providing the detail on Mr Roll’s position and responsibilities
at the SSC that was not set out in any detail in Mr Roll’s own evidence. Mr Roll accepted
in his oral evidence much of what Mr Parker said in this regard.
Final response codes
168. As set out in Post Office’s opening submissions, Mr Roll’s suggestion that “much of the
work” carried out by himself and other SSC workers involved “fire fighting coding
problems in the Horizon system” is not supported by the records to which Mr Parker
refers.°! Mr Parker did his best, based on a review of Peak final response codes, to
estimate the likely proportion of Mr Roll’s work that related to potential bugs in the
system. He concluded from the review of response codes that that proportion was very
24 (Day12/27:21} to {Day12/29:7} and {Day12/30:12} to {Day12/30:19}.
288 (Day12/27:21} to {Day12/29:7}.
286 {Day12/27:21} to {Day12/29:7}.
287 (Day12/27:21} to {Day12/29:7}.
258 Parker 1, para.34 {E2/11/9}.
289 Parker 1, para.35 {F2/11/9}.
26 {Day12/30:8} to {Day12/30:11}.
21 Parker 1, para.51 {E2/11/14}. See also Parker 2, para.39 {2/12/13}.
75
AI8/75
POL00026925
POL00026925
low, which matched his own recollection and impression. He made clear that he had not
carried out a detailed review of the content of the many thousands of Peaks produced
during Mr Roll’s employment.
169. On the face of the witness statements, there appeared to be a substantial disagreement
between Mr Roll and Mr Parker as to what Mr Roll, and the SSC more generally, did in
the relevant period. But that apparent disagreement largely melted away in oral evidence.
Mr Roll fairly accepted in cross-examination that the account in his first witness
262
statement was not a fair reflection of the powers of the SSC and that his written
evidence could have provided a misleading impression.?°
170. Given the frankness with which Mr Roll approached his oral evidence on this and other
points, it is regrettable that his written evidence had not been drafted to reflect more
closely the content and quality of his recollection. It led to the impression of extensive
disagreement between Mr Roll and Mr Parker when, in fact, they are in large agreement
as to the SSC’s functions and the work that it was required to perform in 2001-2003.
171. Nonetheless, Mr Parker was cross-examined as to the spreadsheet attached to his first
witness statement. Mr Parker relied on that spreadsheet to estimate the various categories
of work carried out by the SSC during the time of Mr Roll’s employment.?*
172. The majority of Mr Green QC’s cross-examination of Mr Parker concerned the final
response codes assigned to Peaks. The thrust of the case put to Mr Parker was that the
final response codes were not applied consistently, such that any numerical analysis
based on them would not provide an accurate record of the nature of the issue in Peaks
and their resolutions.
173. There was some limited force to this point. It is true to say that the response codes
provide an imperfect means of identifying the nature of the issue addressed in a Peak
and its resolution. Contrary to the impression that might have been formed by Mr
Parker’s cross-examination on this point, however, he had never presented the response
codes as anything like a perfect source of information. On the contrary, Mr Parker was
282 {Day3/113:2} to {Day-3/114:24} and: (Day3/122:13} to (Day 3/126:8}.
2 {Day3/143:4} to (Day3/143:14}.
24 {Day12/30:21} to {Day12/32:1}.
16
AI8/76
POL00026925
POL00026925
frank about their limitations in his written evidence: “While guidance is provided on
when to use each response code....allocation is the subjective view of the technician
closing the incident and there is no re-examination of response codes later to ensure
consistency”: Parker 1, para. 39.7°° He made clear that the numbers he presented should
be considered “With that in mind...” (para.40). He was only ever presenting a best
estimate, along with the detail of the workings that informed that estimate. That approach
is both fairer and more transparent than merely setting out a subjective impression and
recollection.
174. In this context, the challenge made to Mr Parker’s reliance on the response codes was
overblown and, in some respects, unfair, for two reasons.
175. First, Mr Green QC referred to nine Peaks in total over the course of Mr Parker’s oral
evidence. It was suggested, with some justification, that the response codes used in those
Peaks appeared to be inappropriate to the content of the Peak. The dates of the nine Peaks
were not specifically put to Mr Parker. That was an important omission, given the
following:
175.1 The majority of the Peaks did not even relate to the period in which Mr Roll was
employed at Fujitsu and so were not within the Peaks analysed by Mr Parker in
his spreadsheet. Mr Roll was employed at Fujitsu from 5 March 2001 to 17
September 2004.7 Only three of the Peaks put to Mr Parker relate to the period
of Mr Roll’s employment: PC0063914;7°7 PC0068327;7* and PC0065021.7° The
285 {2/11/10}.
266 Parker 1, para.8 {E2/11/2}.
267 (F/93/1}, referred to at {Day12/76:12} to {Day12/76:12}. Opened on 15 March 2001 and closed on 27
March 2001.
288 {F/100.1/1}, referred to at {Day12/63:25} to {Day12/63:25}. Opened on 25 July 2001 and closed on 6
September 2001.
2 {F/97/1}, referred to at {Day12/57:14} to {Day12/57:14}. Opened on 17 April 2001 and closed on 12
November 2001.
7
AI8/77
other six Peaks did not relate to that period: PC0055964;7"° PC0027887;7"'
PC0204765;?” PC0241771;273 PCO195511;?"4 and PC0229446,77>
175.2 The Peaks that Cs relied on in cross-examining Mr Parker span a 17 year period?”®
— virtually the entire life of Horizon. It seems that Cs could not find very many
poorly-assigned response codes amongst the many thousands of Peaks that were
covered by Mr Parker’s analysis, so they had to go looking elsewhere.
175.3 It is important, then, to maintain a sense of scale and perspective. In order to find
as many as nine Peaks with poorly-assigned response codes, Cs had to go outside
the huge sample covered by Mr Parker’s analysis and search through the hundreds
of thousands of Peaks generated over the lifetime of Horizon. On that basis, the
error rate appears to be very low indeed (and certainly well within what would
fairly have been taken from Mr Parker’s caveat in his witness evidence as to the
consistency of response code allocation).
176. Nine Peaks, limited to two categories, across 17 years does not suggest that the final
response codes were regularly applied incorrectly. It does not suggest that even a sizable
percentage of final response codes were wrong. Mr Parker accepted that there may be a
27
few Peaks where the final response codes do not tally.*’’ However, this is in the context
of 220,000 Peaks.?”*
POL00026925
POL00026925
20 {F/66/1}, referred to at {Day12/82:24} to {Dayl2/82:24}. Opened on 17 October 2000 and closed on 3
November 2000.
27! (F/16/1}, referred to at {Day12/40:20} to {Day12/40:20}. Opened on 21 July 1999 and closed on 31
August 2000.
2? {F/718/1}, referred to at {(Day12/68:1} to {Day12/68:.1}. Opened on 25 September 2010 and closed on
1 November 2010.
23 {F/1326/1}, referred to at {Day12/46:4} to {Day12/46:4}. Opened on 11 March 2015 and closed on 16
November 2016.
274 {F/589/1}, referred to at {Day12/81:2} to {Day12/81:2}. Opened on 3 March 2010 and closed on 5 March
2010.
275 {F/1156/1}, referred to at {Day12/81:25}. Opened and closed on 12 November 2013.
276 The earliest Peak being PC0027887 {F/16/1}, referred to at {Day12/40:20} to {Day12/40:20}. Opened
on 21 July 1999 and closed on 31 August 2000. The latest Peak being PC0241771 {F/1326/1}, referred to at
{Day12/46:4} to {Day12/46:4}.That Peak was opened on 11 March 2015 and closed on 16 November 2016.
27 {Day12/93:22} to {Day12/95:12}
28 (Day12/93:22} to {Day12/95:12}.
wy
AI8/78
177. Itis also of note that the Cs’ cross-examination of these final response codes was limited
to only two of codes used across the Peaks and categorised by Mr Parker, namely codes
70 and 68. That is unlikely to be a coincidence. Cs did not challenge the other response
codes that were assigned or suggest that they were mis-assigned in any substantial
number of instances.
178. Mr Parker fairly pointed out that his categorisation of final response codes was based on
a combination of the guidance documentation and his own experience.” This is an
appropriate method in order to give a fair reflection of how final response codes were
assigned in practice and the likely practical significance of each code. Again, Mr Parker
was clearly striving to find some objective basis on which to estimate the approximate
proportion of time spent on various types of work, using his best judgment where
appropriate.
179. Further, just as in his written evidence, Mr Parker was clear in oral evidence that
assigning the final response code involved subjectivity.” He nonetheless considered
that, although response codes are not always right, they are right in most cases.’*! He
stated that accuracy is important.” He also confirmed that the final response codes were
used to assess workload.”**
A code allocation that has a practical importance to day-to-
day operations is unlikely to be used in an entirely haphazard manner (and any such
usage by an employee is unlikely to be tolerated).
180. Cs could of course have examined the particular Peaks in which Mr Roll had been
involved in and used those to suggest that they showed him to have been involved in a
higher proportion of software problems than Mr Parker’s analysis would suggest. That
was not done or, if it was done, the results of the exercise were not presented to Mr
Parker.
181. Further, even were it to be the case that Mr Roll worked on a significant number of Peaks
which had (or should have had) a closure code indicating that the issue being dealt with
POL00026925
POL00026925
2 (Day12/74:13} to {Day12/75:22}.
280 {Day12/51:6} to {Day12/52:5}.
281 {Day12/51:6} to {Day12/52:5}.
282 {Day12/52:15} to {Day12/53:2}.
283 {Day12/52:15} to {Day12/53:2}.
AI8/79
POL00026925
POL00026925
was a software problem, that obviously docs not necessarily mean that he would have
been identifying or addressing a new bug. Given Mr Roll’s position at SSC, much of his
work will have been relatively low level, applying workarounds already developed by
others which will not have involved researching, working on or fixing bugs. It is possible
that some of the workarounds may have related to previously identified bugs but Mr
Parker’s categorisation indicates that he did not think this applied to many.
182. There is also an important point of context here. Mr Parker’s high-level analysis of Peaks
based on response codes pre-dates the completion of the experts’ intelligent searching
and review of Peaks. We now know that the results of Mr Parker’s high-level review are
strongly consistent with the outcome of the expert’s more detailed review, including that
they have identified very few bugs across all the Peaks. It follows that, unless Mr Roll
was (for some unknown reason) unusually likely to be involved in the investigation of
what turned out to be bugs, Mr Parker’s estimates are likely to be fairly accurate. That is
all he ever claimed.
183. Mr Roll did not himself challenge the estimates or suggest that he had a strong and
reliable recollection that would call them into question. His oral evidence was far more
consistent with Mr Parker’s evidence on this than his written evidence had been.
Remote access
184. As Post Office set out in its opening submissions, it is regrettable that Mr Parker’s
witness evidence in relation to remote access required correction and clarification prior
to the trial.?** Any criticism of him for that would, however, have to be tempered by the
following points:
184.1 Many of the relevant events took place many years ago, and neither Mr Parker
nor Mr Roll would pretend to have a perfect recollection of them. It is perhaps
inevitable that the detail and accuracy of the accounts given by the two witnesses
improved over time as they read each other’s statements and reviewed additional
contemporaneous documents that shed light on the issues.
284 £4/2/92}.
80
A/8/80
POL00026925
POL00026925
184.2 The points raised by Mr Roll in his witness evidence, particularly his first witness
statement, were undeniably expressed in somewhat vague and general language.
Mr Roll very fairly accepted that some of the relevant evidence in that statement
was or might be wrong or was at least expressed in an unfortunate way (including
in that it might be misleading): see paragraph 169 above. These points were
highlighted by Mr Parker in his oral evidence.?**
184.3 Many of the issues as to remote access relate not to what was done (or at least done
with any real frequency) by the SSC but to what could have been done or might in
fact have been done in very rare cases. A witnesses might reasonably find it
difficult to get it right the first time when asked to comment on issues of this kind.
184.4 Mr Parker quite properly sought to correct his written evidence in later statements,
rather than leaving inaccurate or misleading evidence to be corrected only at trial
and in the course of cross-examination.
185. It must also be remembered that much of this concerns fairly difficult and complex
material. The experts have themselves had to reconsider and re-visit their views over
time, taking account of each other’s evidence and the inferences to be drawn from the
contemporaneous documents (of which there is fortunately a vast number).
186. Nonetheless, Post Office entirely accepts that Mr Parker’s evidence required more
correction and clarification than should have been the case, and that is regrettable.
187. On the substance of these points, however, Mr Parker’s evidence has proved to be
accurate and reliable. He was clear that he had not suggested in his written evidence that
the SSC had never changed branch data.?*° He has always been consistent on the points,
both in written and oral evidence, that branch data is not changed frequently and that,
where it is, the SSC would involve the SPM wherever possible.”*” Mr Roll gave the same
285 (Day12/10:10} to {Day 2/11:3}.
286 {Day12/22:3} to {Day12/23:8}.
287 {Day12/22:3} to {Day12/23:8}.
81
A/8/81
288
evidence in response to cross-examination (although not before).*** Mr Coyne’s oral
evidence was also to similar effect: see section F below.
188. Mr Parker’s evidence on the scope of remote access was set out in Post Office’s opening
submissions.’* Mr Parker’s evidence is that:
188.1 The assertion that Fujitsu edited or deleted transaction data was not correct. In
Legacy Horizon, it was not possible to delete or edit messages that had been
committed to the message store.”
188.2 This is to be distinguished from the wholesale deletion of corrupted data stored on
one terminal in a branch so that a mirror copy of the (uncorrupted) data could be
automatically replicated from another source, usually another terminal in the
branch or the central data store. This process would be used where the data on the
first terminal had been corrupted or the hard disk in that terminal had suffered a
physical failure. Mr Parker explains this process at Parker 1, para. 55.4.7"! He
292
draws an analogy with the use of a back-up hard drive.*”* Mr Parker identifies rare
cases in which a more involved process might be required at Parker 2, para. 38.7°°
188.3 It was also possible to inject transactions into counters. The standard way of doing
this was via the correspondence server, which resulted in a counter number of 32
or higher being associated with the transaction in the transaction log, making the
insertion immediately identifiable.?°*
188.4 Any injections of transactions required compliance with strict change controls.
Two staff were required to be present when the change was made, and all changes
POL00026925
POL00026925
288
289
290
201
202
293
204
{Day 4/139:24} to (Day4/140:15}.
{A/2/90} to {A/2/92}.
Parker 1, para.19 {E2/11/4}, referring to Godeseth 1, para. 37 {E2/1/11}.
{E2/11/16}.
Parker 3, para.21 25/54. §
{E2/12/12}.
Godeseth 1, para.58.10 {E2/1/17}.
{E2/13/5}
AI8/82
had to be audited (identifying both the specific alteration and the person making
it) 25
188.5 System misuse would have been discovered by consistency checks or colleagues
(all access was controlled and audited), and would have resulted in the instant
dismissal of the relevant employee.”°° Mr Coyne’s oral evidence was that he had
no reason to believe that system misuse had ever occurred.?””
188.6 Mr Roll’s suggestion that software issues in Horizon “routinely” caused
discrepancies in branch accounts was incorrect. In the vast majority of cases, such
an occurrence would cause a receipts and payments mismatch that would be
flagged by the branch system as part of the balancing process (the Horizon system
carries out self-consistency checks which generate alerts in the event of a receipts
and payments mismatch that are picked up by SMC and incidents raised for the
SSC3 and appear on MSU reporting).?’* These would then be investigated and
resolved by the SSC. Mr Roll largely agreed.
188.7 While a hardware issue could very occasionally affect a branch’s accounts, the
vast majority of hardware issues were not capable of having any impact on such
accounts (in the sense of leading to a financial discrepancy). Mr Parker explained
that, in the rare circumstance that data was not replicated accurately, Fujitsu would
inform both the SPM and Post Office and provide them with any information that
it could to help resolve any discrepancies.””
188.8 Mr Parker is not aware of any case in which baskets were not zero sum (i.e. any
case in which a non-zero-sum basket of transactions was accepted into Horizon).°°°
188.9 Mr Parker cannot recall any instances of incorrect reference data misdirecting
payments while Mr Roll was employed by Fujitsu. Although one example did
POL00026925
POL00026925
25 Parker 2, para.33 {E2/12/11}.
26 Parker 2, para.35 {E2/12/11}.
27 See para. 82.56 below.
28 Parker 1, para.42_ {E2/11/I1}
2 Parker 2, para.5.2 {E2/12/2}.
300 Parker 2, para.10 {E2/12/4}.
83
AI8/83
POL00026925
POL00026925
occur to Mr Parker’s knowledge much later (in 2012), this was picked up and
resolved quickly.*°!
189. It is important to note that almost all of these points were accepted, in whole or in large
part, by Mr Roll and/or by Mr Coyne in his oral evidence. The distance between Mr
Parker, Mr Roll and the experts closed considerably in the cross-examination of Cs
factual and expert witnesses.
190. Mr Parker also explained the purpose and use of the TIP Repair Tool in paragraphs 11 to
13 of his third witness statement.*” It docs not involve remote access or alteration in the
sense of Horizon Issue 10. This accords with the expert evidence as it stood after cross-
examination.
191. Mr Parker explained the various reasons for which remote access functions were used,
none of which involved any improper manipulation of branch data.*’? Mr Parker’s
evidence was that the “SCC was (and is) hugely reluctant to change financial data as
that was not their job and they recognised the seriousness of doing so”. This is a point
that came across in Mr Roll’s oral evidence.*”* It is also supported by the experts.*°°
Data insertions at the counter (rather than via the correspondence server)
192. Mr Parker’s evidence on this topic requires separate consideration only because he was
subjected to extensive and misconceived criticism in relation to it. It is important to set
the record straight, as a matter of fairness to Mr Parker. In short, no further correction
was needed to his evidence, because his evidence was not incorrect. The complaint from
Cs appears to be that Mr Parker should have given more evidence. Had he done so, one
could then imagine the Cs complaining about late evidence. Mr Parker was placed in an
impossible position, but addressed it correctly by bringing the new information to light
0! Parker 2, para. {E2/12/4}.
82 Parker 3, para, 32S SEQ. {E2/13/1}
9 Parker 1, para, 55-57 {E2/11/15}.
94 Parker 2, para.34 {E2/12/11}.
35 See paras 127 to 131 above.
16 below.
6 See para, S14
84
AI8/84
193,
194,
195.
196.
197.
POL00026925
POL00026925
promptly and then that information being passed to the Cs similarly promptly by way of
a letter.
In para. 29 of his second witness statement, Mr Parker specifically identified the searches
that had been carried out by a colleague, Mr John Simpkins, aimed at identifying
instances in which data had been inserted into the counter, rather than into the
correspondence server. Mr Parker than stated as follows:
From these results I can determine that this was only carried out in the following
circumstances while Mr Roll was employed by Fujitsu.*”’(emphasis added).
It was always clear that the 14 instances then set out by Mr Parker were taken “/rom
these results”, i.e. from the documents identified using the searches set out in para.29
itself. There was, however, an unrelated mistake in para. 29 — the underlined text above
was wrong, as the searches had covered not only the period of Mr Roll’s employment
but the whole life of Legacy Horizon.
In the corrections to this witness statement, Mr Parker corrected this to state that the
instances identified in that paragraph covered the lifetime of Legacy Horizon: see the
correction at {E2/16/4}. It is not in dispute that this correction was rightly made — the
searches and documents identified through them covered the whole life of Legacy
Horizon (as is clear from the dates of the Peaks themselves).
Before that correction was made, the mistake in para. 29 had also been carried forward
into para. 19 of Mr Parker’s third witness statement. The relevant part of that paragraph
referred back to para. 29 in the following terms:
..dn paragraph 29 of my second witness statement I listed the circumstances in which
data was injected into a counter in Legacy Horizon while Richard Roll was employed...*
(emphasis added)
A similar correction was therefore required to that sentence. That correction was made
by the document at {E2/17}. It was as follows:
307 ¢£2/12/10}.
208 (E2/13/4}.
85
AI8/85
198.
199,
200.
POL00026925
POL00026925
In paragraph 29 of my second witness statement I listed the circumstances in which data
was injected into a counter in Legacy Horizon found as _a result of the searches
described in that paragraph white Rieharad-Relt- iprlerved dry Perse (fi
That correction brought para. 19 of the third witness statement into line with para. 29 of
the second statement as corrected, in two respects:
198.1 The time period covered by the searches was corrected (i.e. the whole period of
Legacy Horizon, rather than only Mr Roll’s employment).
198.2 It was clarified that the list in para. 29 was based on the searches identified in that
paragraph. That was stated in para. 29 itself, using the words “/rom these
results ...”, and it was appropriate for that to also be recorded in the reference back.
It was discovered shortly before Mr Parker gave oral evidence that Mr Simpkins (the
person who did the searches referred to in para. 29 of Mr Parker’s second statement) had
since carried out further searches and identified several further relevant Peaks. This was
immediately notified to Cs in correspondence: see {H/253}. Mr Parker explained in oral
evidence that it was not his decision whether to send a letter or to prepare an additional
witness statement to sct out the new information.*”
It was suggested to Mr Parker in cross-examination that he should have taken steps to
“correct” his second witness statement to reflect the results of the new searches, but that
criticism is difficult to follow. The statement had always made clear on its face that Mr
Parker was presenting documents that had been found using the search terms set out in
para. 29. Since those results had been presented, and only shortly before he gave
evidence, Mr Parker became aware that some additional Peaks had then been identified
by using further search terms, and he had made sure that this was brought to the parties”
attention. He had not been involved in those further searches.*"”
It is difficult to see on what basis Mr Parker could properly be criticised here.
39 {Day12/20:6} to {Day12/20:11}.
MO {Day12/91:20} to {Day12/93:19}.
86
AI8/86
202.
203.
204.
POL00026925
POL00026925
Mr Parker was nonetheless cross-examined extensively on this topic, apparently on the
(implicit) basis that he had done something wrong or sought to hide something.*!! The
accusation was not put in clear terms, so Mr Parker was only given the opportunity to
comment on it directly in re-examination. He denied it in clear terms.**
Notably, Mr Parker was not asked in cross-examination about the further Peaks that had
been identified using the extra searches carried out by Mr Simpkins. That is no doubt
because those Peaks are nothing more than a few further examples of the kind of
corrective actions already identified and summarised by Mr Parker in para. 29 of his
second statement (which summary Mr Coyne did not dispute, himself having reviewed
the relevant Peaks).3!°
Mr Parker was nonetheless subjected to repeated questions about
the “real reason” for the corrections that he had made to his witness statements. The
insinuation of some wrongdoing on his part was entirely unfair.
On the substance of the data insertions at the counter, Mr Parker’s evidence is strongly
supported by the documents and the work carried out by the experts:
204.1 Mr Parker was clear in his written and oral evidence that it was only when he
began to investigate the matter in detail that he became aware of this non-standard
method of inserting a transaction.*"*
204.2 That is unsurprising given the very few instances that have been identified and the
nature of the insertions that were made (as summarised by Mr Parker in para. 29
of his second witness statement).* It is clear that data insertion at the counter was
not something that was done any real frequency or that was in any way high-profile
when it was done. It is unsurprising that Mr Parker should have not known or
should have forgotten that the standard method was not always used.
MI Day12/13:24} to {Day12/15:17}, and s{Dayl2/16:8} to {Dayl2/17:7},_and s{Dayl2/18:7} to
{Day12/18:24} 2
(Day12/20:1} to {Day12/20:11} and {Day12/92:22} to {Day12/93:8}.
32 {Day12/93:9} to {Day12/93:19}.
313 See para. 7
767 below.
3 {Day12/86:19} to {Day12/88:9}.
315 (E2/12/10}.
87
AI8/87
205.
Evidence that was not challenged
Large parts of Mr Parker’s written evidence was not challenged during cross-
examination. It appears likely that much of this evidence is not genuinely controversial,
especially in light of it according with Mr Roll’s evidence and/or the views of the experts.
The unchallenged evidence includes the following:
205.1 Although Mr Parker was cross-examined as to Mr Roll’s role and the SSC, Mr
Parker’s evidence regarding the wider Horizon support team structure was not
challenged.*!®
205.2 Mr Parker was cross-examined on final response codes 68 and 70 but not the other
final response codes and any inaccuracy in the use of those other codes (as noted
above)
205.3 Mr Parker’s description of the purposes and content of KELs and Peaks.*"7
205.4 Mr Parker’s KEL analysis.*!*
205.5 Mr Parker’s evidence on transactional integrity.*!”
205.6 Although Mr Parker was asked about issues tangential to user error, he was not
directly challenged in relation to his evidence on TCs and patterns of software
errors.*”°
205.7 Although Mr Parker was briefly asked about SLAs, Mr Parker’s evidence that
SLAs would not have impacted the level of diligence was not challenged.*?!
205.8 Mr Parker was not challenged regarding his evidence on how the SSC identified
322
branches affected by bugs.
POL00026925
POL00026925
316 Parker 1, paras 24-27 {E2/11/5} to (E2/11/8}.
317 Parker 1, paras 60-62 {E2/11/17} to {E2/11/21}.
318 Parker 1, paras 63-66 {E2/11/21} to {E2/11/22}.
519 Parker 2, paras 8-15 {E2/12/4} to {E2/12/6}.
320 Parker 2, paras 16-23 {E2/12/6} to {E2/12/7}.
321 Parker 2, paras 24-25 {E2/12/7} to {E2/12/8}.
522 Parker 3, paras 3-12{E2/13/1} to {E2/12/3}.
88
AI8/88
206.
207.
208.
205.9 Mr Parker’s evidence on the use of the APPSUP role was not challenged.*”* This
includes his evidence that he cannot recall any instance in which the APPSUP role
has been used to change transaction data, although he cannot state unequivocally
that it has not happened.
Conclusion on Mr Parker’s evidence
Mr Parker’s evidence was ultimately helpful to the experts and, it is submitted, to the
Court. He gave fairly extensive evidence about the SSC’s processes, especially in
relation to the detection and resolution of bugs, and the core of his evidence in those
regards went unchallenged. He also gave evidence in relation to Mr Roll’s work that,
although challenged by Cs in cross-examination, was largely consistent with Mr Roll’s
own oral evidence.
Much of Mr Parker’s cross-examination was focused on two issues of relatively low
importance: (1) final response codes, where the challenge was overblown and based
largely on documents from outside the relevant period and (2) the unfair suggestion that
he had done something wrong in relation to the correction of para. 19 of his third witness
statement and/or the further searches carried out for Peaks relating to data insertion at
the counter. Neither of those two challenges should undermine the usefulness and
reliability of Mr Parker’s evidence in addressing the Horizon Issues.
Large sections of Mr Parker’s evidence was not challenged, as set out above.
E4, Mr Dunk’s Evidence
. The relevant evidence:
209.1 Mr Dunks’ witness statement dated 16 November 2018: {E2/10}
209.2 Transcript: {Day7/27:19} to {Day7/73:23}
POL00026925
POL00026925
3 Parker 3, para.13-16{E2/13/3} to {E2/13/4}.
AI8/89
210.
211.
212.
213.
214.
Mr Dunks is an Information Technology Security Analyst at Fujitsu. His short witness
statement addressed the processes used in extracting archived data from the audit store.
He provided evidence to explain the processes behind the ARQ data that he had extracted
from the audit store for use in these proceedings: see para. 7 of his witness statement.***
His experience and evidence is confined largely to the Horizon Online.*?>
Mr Dunks did not purport to be able to give to evidence on Horizon more generally. His
evidence as to the extraction process itself appears to be largely uncontroversial,
although he was cross-examined on various points that were outside his knowledge and
experience (and on which he made clear he could not comment).
Extracting and filtering data
In Mr Dunks’ witness statement he states:
“When information relating to individual transactions is requested, the data is
extracted from the audit archive media of the Horizon System via the Audit
Workstations (AWs). Information is presented in exactly the same way as the data
held in the archive although it can be filtered depending upon the type of information
requested.”*°6
Mr Dunks confirmed during cross-examination that filtered data is provided to Post
Office.*”” He does not recall providing unfiltered data to Post Office.*”* It is unsurprising
that Fujitsu and Post Office should agree that the data be filtered and presented in a form
that Post Office is able to interpret and use effectively (given that it can always rely on
Fujitsu to interpret the raw data and advise as appropriate).
Extractions from the audit store can only be made at audit workstations, and these are
located at Fujitsu sites in Bracknell and Stevenage. Both are subject to rigorous security
controls.’ These security controls include both physical security control, password
control and two step authorisation.**°
POL00026925
POL00026925
34 (E2/10/3}.
25 (E2/10/1}, para 4 and {Day7/34:2} to {Day7/34:23}.
526 £2/10/2}, para 5
27 {Day7/35:9} to {Day7/35:15}.
38 {Day7/35:9} to {Day7/35:15}
329 ¢£2/10/2}, para 6.1.
30 {£2/10/2}, paras 6.1 and 6.2.
90
A/8/90
Data integrity
215. In Mr Dunks’ witness statement he states:
“{t[he integrity of data retrieved for audit purposes is guaranteed at all times from
the point of gathering, storage and retrieval to subsequent despatch to the person
making the request. Controls have been established that provide assurances to Post
Office Internal Audit (POIA) that this integrity is maintained.”™*!
216. During cross-examination Mr Dunks highlighted that data integrity entailed
completeness.**? He agreed with a definition suggested to him, namely that data integrity
is the overall completeness, accuracy and consistency of that data which you can measure
by comparing between sources.**7
217. As set out above, Mr Dunks then goes on in the following paragraphs to set out the
controls that apply to audit data extractions.*** The checks and controls that Mr Dunks
describes are identified in the technical document reviewed by the experts (see, for
example, {F/1716}. A schematic of the retrieval process is set out at {F/1716/11}. Dr
Worden’s consideration of the audit process is in section 4.4 of his first report.59°
218. The controls identified by Mr Dunks include the following:
218.1 The Fujitsu extraction sites are subject to rigorous security checks.**°
218.2 Audit extractions are only made by authorised individuals.**”
218.3 Extractions are logged on the audit workstation and supported by documented
ARQs, authorised by nominated persons within Post Office.***
218.4 This log can be scrutinised on the audit workstation.*°°
POL00026925
POL00026925
31 {E2/10/2}, para 5.
382 {Day7/35:21} to {Day7/36:12}
383 (Day7/35:21} to {Day7/36:12}.
3 {E2/10/2} to (Day7/3-6!-pare 6+.
385 (D3/1/45}.
386 {£2/10/2} para 6.1 and 6.2.
387 €E2/10/2} para 6.4
388 (£2/10/2} para 6.3.
389 (E2/10/2} para 6.3.
oO
AI8/91
219.
POL00026925
POL00026925
218.5 The required files are identified and marked using the dedicated audit tools.**°
218.6 Checksum seals are calculated for audit data files when they are written to audit
archive media and re-calculated when the files are retrieved.*#!
218.7 The files are copied to the audit workstation where they are checked and converted
into the file type required.*”
218.8 Digital signatures that were generated at the time that messages were originally
sent from the counters to the data centre are checked as being correct.*4
218.9 Checks are made using the JSN that all audited messages for each counter in the
branch have been retrieved and that no messages are missing.*"*
218.10 System events generated when the transactions at the branch were recorded are
checked to ensure the system was functioning correctly.*4°
218.11 The retrieved audit data is encrypted using PGP encryption.*“°
218.12 The requested information is copied onto removable CD media and virus
checked using the latest software.*"”
It is important to clarify the meaning of “audit” in this context. There appears to have
been some confusion on Cs’ part here.*“* “Audit data” in this context does not denote
data that has been subject to review or comparison against some other source of
information. It is used in the sense of “audit trail”, whereby there is a secure and properly
recorded sequence of activities — here, the copying and archiving of branch data from
the BRDB into the audit store.
60 {£2/10/2} para 6.5.
MI {E2/10/2} para 6.6.
62 {E2/10/2} para 6.7.
38 (E2/10/2} para 6.8.
M {E2/10/2} para 6.9.
MS {£2/10/2} para 6.10.
M6 ¢E2/10/2}para 6.11.
MT (E2/10/2; para 6.12.
MS (Day7/33:6} to {(Day7/33:21}.
92
AI8/92
220.
221.
222.
223.
In Mr Dunks’ oral evidence he was clear that maintaining data integrity is “extremely
important” °° He agreed that the data is highly secure and tamper proof.**° Mr Dunks
agreed that the process for extractions was “gold standard” >*!
The “party line” point
Mr Dunks stated in his witness statement “there is no reason to believe that the
information set out in the statement is inaccurate due to improper use of the system” 5°?
Furthermore he stated that the system was operating properly and that to any extent that
the system was not operating properly, this would not have an effect on the information
held within it“°* Mr Dunks explained in cross-examination that what he meant by this
was that the audit workstation was working correctly and that any problems with it (such
as needing to reboot) would not have affected the data extracted.***
Mr Dunks was cross-examined as to the origins of paragraph 8. It was put to Mr Dunks
that the choice of words at paragraph 8 was a “party line”>°5 It was not suggested to Mr
Dunks that this information was incorrect, but only that it was standard wording that had
been used by other Fujitsu employees when describing the extraction process.*°° Mr
Dunks confirmed that it could well be part of a template witness statement used in
relation to ARQs that he had adopted without thinking too much about the specific
content of para. 8.557
Mr Dunks confirmed that he was not aware of any issues with data integrity, being
careful to make clear that he could only speak to the integrity of the data extracted and
not how it might compare to other sources of data.**
POL00026925
POL00026925
9 (Day7/36:13} to {Day7/37:23}.
$80 {Day7/37:8} to {Day7/37:9}, Mar. 20, 2019).
351 (Day7/37:10} to {Day7/37:18}.
382 (E2/10/2} para 8.
359 {E2/10/2} para 8.
38 {Day7/41:10} to (Day7/42:6}.
388 {Day7/43:7} to {Day7/43:21}.
386 (Day7/58:25} to {Day7/59:13}.
357 {Day7/59:22} to {Day7/60:8}.
388 {Day7/44:6} to {Day?/45:4}.
93
AI8/93
224.
225.
226.
227.
228.
POL00026925
POL00026925
Mr Dunks could not recall even one case where the system the data extracted was not
accurate because of a problem with the extraction process.** He was asked where there
were any instances where the integrity checks revealed problems — and again he was not
aware of any issues.*?
Mr Dunks was at all times careful to make clear that he was confirming only the
reliability of the extraction process and that he had not seen any problem with the
integrity of the data extracted (as distinct from broader questions as to data accuracy and
reconciliation, which were outside his knowledge and experience). That is what he
addressed in his witness statement and what he was able to speak to.
The standard of perfection
The experts have considered the audit store and the ARQ process. Neither of them has
identified any flaw in the design of the audit system and the extraction process. Mr Coyne
agreed with Dr Worden that the design principles for audit information were
“theoretically sound” *°!
Despite this, and despite the clear limitations on what Mr Dunks was addressing in his
evidence, it was twice suggested to Mr Dunks that he had somehow given misleading
evidence because he had not addressed the process of comparing the audit data, once
extracted, with other data sources and identifying any inconsistencies that might be
relevant to disputes over branch discrepancies.*
That was unfair. As Mr Dunks fairly explained, he was only addressing the “the integrity
of the data once we have extracted it...and the controls around the extraction”; he was
certainly not trying to speak to whether the data was correct (as opposed to being
extracted properly) in relation to any dispute between Post Office and an SPM.°° Any
889 (Day7/41:3} to {Day7/43:6}.
360 {Day7/44:21} to (Day7/44:24}.
361 Coyne 2/5.59 {D2/1/135}. Mr Coyne goes on to raise criticisms of Horizon, but none of them seems to
relate to the audit s
m itself or the extraction process (aside, perhaps, from a concern over JSN).
38 {Day7/66:14} to {Day7/66:19} and {Day7/71:7} to {Day7/71:21}.
33 {Day7/70:25} to {Day7/71:21}. See also {Day7/66:1} to {Day7/66:8}.
94
AI8/94
229.
231.
232.
233.
dispute over whether transaction data is correct in that sense, or would require correction
following reconciliation, is entirely outside Mr Dunks’ knowledge and experience.
The possibility for duplicates and gaps in extracted data
Mr Dunks was asked about duplicate records in the data extracted. He explained that
resolving any duplicate entries would not have been a matter for him.*°
Mr Dunks was asked whether he had himself encountered duplicate entries in the many
years that he had been extracting data through the ARQ process. He said that could not
say that he had not but, if he had, it was a long time ago.*°
The case put to Mr Dunks was that the possibility for there to be duplicate entries would
not “give much comfort” to anybody seeking to rely on ARQ data as a gold standard.
That was not a matter for this witness, given that he was not involved in resolving any
duplicates that were identified through extraction. He frankly acknowledged that he
would not know what had caused any duplicates and that he was not the person to
investigate them — he would pass the issue to audit support to resolve.*
More importantly, it was not put to Fujitsu witness (or, for that matter, Dr Worden) that
any duplicates identified in the extraction process would be difficult to investigate and
remove or that, for some mysterious reason, any genuine duplicate entries would not be
removed. The Audit Extraction Client User Manual, to which Mr Dunks was referred,
shows both that any duplicate entries are clearly identified in the extraction process and
that these are to be drawn to the attention of the audit support team: see {F/} 716/43}.
This is another area in which Cs appear to set a standard of perfection, secking to
undermine the reliability of ARQ extracts on the basis that there were (it seems)
sometimes duplicate entries that had to be resolved. Mr Dunks, who is the man who
would know, was not even sure that he had seen any duplicates and that, if he had, it was
a long time ago. There is no reason to think that duplicates would not have been removed.
POL00026925
POL00026925
364 {Day7/57:18} to {Day7/58:8}.
368 {Day7/51:25} to {Day7/52:4}.
86 {Day7/52:5} to {Day7/53:9}.
9S
AI8/95
POL00026925
POL00026925
234. The possibility for there to be duplicate entries is addressed further below in relation to
Mr Worden’s evidence.*”
Conclusion on Mr Dunks’ evidence
235. Mr Dunks provided clear evidence, both in his written evidence and in cross-
examination. He had been asked to extract ARQ data (see para.7 of his statement)*** and
he had done so, providing evidence as to the extraction process used and the controls and
checks used to validate the integrity of the extracted data. Much of his cross-examination
covered other matters, that were outside his knowledge, and he was careful to make this
clear and to limit his answers to what he knew.
236. As to the subject matter of his evidence, Mr Dunks gave clear oral evidence that the
extraction process is reliable. He stated that he could not recall even one example where
the data extraction had gone wrong™” or any instance where the integrity checks revealed
issues.”” Mr Dunks confirmed that data integrity is seen as being “extremely
important”,”' and that the process of extraction is highly secure and tamper proof.‘
That coincides with the evidence of other Fujitsu witnesses, including Mr Roll, as to the
care taken to ensure the integrity and accuracy of the data in the system.
E5. Mr Membery’s Evidence
237. The relevant evidence:
237.1 Mr Membery’s first witness statement dated 28 September 2018 {E2/2}.
367
See para, 612.616 below.
368 {£2/10/3}.
3 {Day7/41:10} to (Day7/43:6}.
3 (Day7/44:21} to (Day7/45:1}.
371 {Day7/36:13} to {Day7/37:23}.
52 {Day7/37:8}.
96
AI8/96
238.
239.
240.
241,
242.
POL00026925
POL00026925
Mr Membery was unable to attend to give oral evidence. Post Office relies on the
evidence set out in his witness statement as hearsay evidence. Cs helpfully agreed to
extend time for service of the relevant notice.
Mr Membery is a Fujitsu employee. He became the Quality Risk and Compliance
Manager for the Post Office Account in 2011 and in that capacity oversaw many audits
of the Horizon system.*” He carried out a different job between 2014 — 2018, although
retained a focus on Post Office, and on 1 October 2018 became Head of Quality and
Compliance for the Fujitsu Post Office Account.>”4
His evidence sets out: a high level overview of the audits which Horizon has been subject
to since 2000 (which are to internationally recognised standards and ensure that Horizon
and Fujitsu are working within a robust system of control measures)*”*; Fujitsu’s audit
methodology and his role in such audits; and the processes around the development of
changes to Horizon and how that ties into the audit process.
33 below.
The audits are addressed in greater detail at paragraphs 828 to 33-4
Mr Membery stated that Horizon had been subject to audits since Horizon’s introduction
in 2000.56 When Horizon Online was introduced in 2010 Post Office and Fujitsu made
the decision that Horizon should be audited “against the International Standard on
Assurance Engagements No. 3402 assurance standard (Horizon ISAE 3402 Audit),
entitled "Assurance Reports on Controls at a Service Organization"” *"” These are what
have become known in these proceedings as the E&Y Post Office audits, as discussed at
paragraphs 828 to 834-833 below.
Prior to the introduction of Horizon Online, Legacy Horizon was audited “hy reference
to ISO (formerly ITIL) 20000, ISO 9001 and BSI 7799 that evolved to into ISO 27001278
979 {£2/2/1} para 5.
5 ££ 2/2/2} para 6.
5S ¢E2/2/1} para 8.
376 ¢£2/2/2} para 7.1.
377 (£2/2/2} para 9.
578 (£2/2/3} para 13. A summary of the legislation, policies and standards specified in the contracts between
Post Office and Fujitsu between 1999 and 2017 is referred to in Mr Membery’s statement {F/1830/1}.
97
AI8/97
244,
245.
246.
POL00026925
POL00026925
Mr Membery described Post Office’s and Fujitsu’s approach to addressing critical or
major non-conformances:
If an audit finds a "critical" or "major" non-conformance that does not require Post
Office's input, Fujitsu aims to remediate it within 30 days. If an audit finds an
observation that does not require Post Office's input, Fujitsu aims to remediate it
within 90 days.>”?
Furthermore:
If an audit makes a finding which requires Post Office's input, Fujitsu discusses it
with Post Office through the applicable operational change procedures before
implementing any remediation plan*°
As is apparent from the audits themselves, Post Office and Fujitsu did respond to the
points raised and set out what action they were taking.**' Furthermore, Mr Membery’s
evidence is consistent with audits themselves. As is evident from the 2013 E&Y
management letter, Post Office and Fujitsu took steps to implement recommendations
and their responsiveness was acknowledged.**?
THE EXPERT EVIDENCE
247,
Fl. Introduction and outline structure of this section
Post Office’s case was opened on the basis that, on a proper analysis, there was or at
least should be far less in issue between the parties than might have appeared from an
initial reading of the experts’ reports. This became starkly apparent during Mr Coyne’s
cross-examination. Mr Coyne readily accepted many aspects of Post Office’s essential
case, including in relation to Horizon’s robustness, the effectiveness and reliability of
Fujitsu’s support processes and the fact that the overwhelming majority of transactions
are processed accurately, even in adverse conditions.
579 {£2/2/4} para 19.
380 {£2/2/4} para 20.
381 See for example the 2011 E&Y letter in relation to change management and permission controls
(F/869/29} to {F/869/38} and {F/869/47} to {F/869/49}.
32 (F/1138/4}.
98
AI8/98
248.
249.
250.
POL00026925
POL00026925
The apparent gulf between the experts before the trial was due in large part to the less
than straightforward approach that Mr Coyne took to presenting his opinions, and the
evidence in support of those opinions, in his two reports and the Joint Statements. Mr
Coyne’s written evidence was, in important respects, unclear and unhelpful. It gave a
very different impression from what we now know to be his views. He did ultimately
give clear and reliable evidence on many of the key issues in dispute, often in belated
agreement with Dr Worden’s evidence, but this had to be drawn out in cross-
examination. He conceded that some of the points in his written evidence were or may
have been misleading. There was no proper explanation as to why his written evidence
was so dramatically different from what he was prepared to accept when asked at trial.
F2. Overall submissions on Dr Worden’s evidence
Dr Worden provided careful and balanced evidence and made every effort to provide
assistance to the Court and to provide full and helpful answers to the Horizon Issues. He
gave in his reports a proper and fair overview of the Horizon system — he identified its
architecture, outlined its operations and characteristics, and he identified the good, the
bad and the indifferent from the perspective of the Horizon Issues. He did not confine
his efforts to looking only for problems or only for evidence of Horizon working well.
Dr Worden put things into their proper context and scale. He gave the sense of Horizon
operating in satisfactory way in the vast majority of instances over a 20-year period (a
point accepted by Mr Coyne at trial). He referred to and explained the relevance of
countermeasures, which it is now common ground are of central importance in arriving
at a proper view of Horizon’s robustness, but which were almost completely ignored by
Mr Coyne in his first report.
. Although, like Mr Coyne, Dr Worden had necessarily had to rely on elements of Post
Office and, more so, Fujitsu witness statements for explanations of certain processes and
features of Horizon, he did not inappropriately prefer the evidence of the Post Office
witnesses, but made it clear that some aspects of his views were based on aspects of
evidence which the Court would have to determine. He was placed in the awkward
99
A/8/99
252.
253.
254.
255.
256.
position of not understanding parts of Mr Roll’s evidence and so being unable to take it
fully into account in his second report.
He was acutely conscious of his duties to the Court and to Mr Coyne as his fellow expert.
This was seen in his insistence that both Mr Coyne and the Court should be updated with
the new analysis that he had performed, the documents informing that analysis and the
change to his opinions that had resulted.
Contrary to the impression that Cs seek to generate, Dr Worden looked for and found
bugs. Of the 29 bugs that found their way into the list in JS2, three were the known bugs
and nine were identified by Dr Worden.***
In his oral evidence, Dr Worden was candid and clear. He explained his views well and
admitted the errors in his reports that were pointed out to him. In a few instances where
he had expressed himself poorly, he corrected himself and apologised.
Dr Worden was criticised in cross-examination in relation to his approach to numbers —
both as to the detail of some parts of his calculations and as to his approach to his
calculations generally.
As to the detailed criticisms of the way in which Dr Worden had arrived at his scaling
factor:
256.1 It was pointed out to Dr Worden that he had divided the number of transactions
per day by 561 (the number of claimants) rather than 496 (the number of entries
on the spreadsheet from which he had taken the data)***
. Dr Worden explained that
he had in fact corrected that error for the purposes of his supplemental report and
that the change from a scaling factor of 0.37 to one of 0.45 was in part explained
by that correction on his part.**>
256.2 The fact that he had not included values for the gaps in the data shown in column
E of the spreadsheet had a tiny effect. Dr Worden’s evidence as to what he had
POL00026925
POL00026925
583 Bugs 5, 6, 7, 23, 24, 25, 26, 27 & 28 were all dealt with in Worden I.
38 {F/1837.1}.
38 (Day19/5:12} to {Day19/6:11}.
100
A/8/100
257.
258.
259.
done was readily comprehensible once the actual spreadsheet which he had had
386
available to him when preparing his first report was seen**° (which only has the
gap values in column F).*°7
The questions based on the later spreadsheet (which
he had not used) had understandably put Dr Worden in a difficult position when
asked to explain what he had done (with data that had not been before him at the
time).
Perhaps the most important point on Dr Worden’s figures was that he was rounding them
up so strongly in Cs favour that this effect dwarfed all other changes in the inputs to the
calculations. For example, Dr Worden’s calculation of the scaling factor in his
Supplementary Report was 0.45, but the figure he used as a conservative estimate in his
Revised Financial Impact spreadsheet***
was 0.5 — more than 10% higher. The same is
true for all of the assumptions which he makes in that calculation, as a comparison of the
figures in the “Central Estimate” and “Conservative Estimate” columns demonstrates.
As to the wider criticism of Dr Worden’s figures, namely that it was entirely
inappropriate for him to carry out the sort of calculations which he did, this too is
misconceived. A crucial part of the Horizon Issues which the experts were asked to
consider ask questions of “extent”. Mr Coyne avoids providing any direct answer to the
questions of extent: Dr Worden does not. He has properly wrestled with this central issue
and done so by reference to (i) the available evidence and (ii) common sense assumptions
which have been adjusted massively in Cs’ favour.
Dr Worden does not claim that the figures he puts forward are precise. They are plainly
(sensible and conservative) approximations which are designed to give the Court a sense
of the likely scale of the incidence and effect of bugs in Horizon. The fact that on Dr
Worden’s conservative approach he posits 672 bugs illustrates just how conservative he
has been, since Mr Coyne's view is that the true number likely to be vastly lower than
this.
POL00026925
POL00026925
386 {F/1837.1}
587 {Day20/166:21} to {Day20/167:17}.
388 (D3/8}.
101
A/8/101
260.
261.
262.
263.
264.
26S.
POL00026925
POL00026925
Dr Worden’s quantitative approach to assessing the likely number of bugs with lasting
impacts over the life of Horizon has turned out to result in a vast overestimate of that
number. It would imply a number of bugs in the system greatly higher than anything that
Mr Coyne was prepared to defend at trial. That is hard to reconcile with any criticism of
Dr Worden’s independence and impartiality (unfortunate allegations having been made
in Cs opening submissions).
F3. Overall submissions on Mr Coyne’s evidence
The detail of Mr Coyne’s evidence is considered below. However, certain themes that
emerged during his cross-examination can properly inform the Court’s general approach
to his evidence.
Mr Coyne’s “deep focus” on problems and lack of balance
Mr Coyne was frank in JS1 that he intended principally to adopt a “deep focus” on bugs
and the “potential modification of transactional data”. He said that it was on the “non-
typical operation where focus will be placed” >°° However, he also said that he would
adopt a balanced approach.
There was an obvious risk that the depth of Mr Coyne’s focus on problems might narrow
the view so much that perspective and balance would be lost. That risk eventuated.
Mr Coyne signally failed to give a balanced view in his reports. This was corrected, at
least in part, by the important concessions that Mr Coyne made (often only after some
resistance) in his oral evidence. But the Court and the parties should not have been
required to wait for the important points of balance to be drawn out of the expert through
days of cross-examination. His reports should have been capable of being trusted and
relied upon to paint a fair and balanced picture.
Mr Coyne’s two reports presented walls of criticisms of Horizon, each wall consisting
of hundreds of bricks and each brick consisting of one or more documents (often only
one) that he chose to construe as evidencing some deficiency in the system. The supposed
39 (DI/1/3}.
102
AI8/102
266.
267.
268.
deficiency was often described in a way that would make it seem serious. In the course
of cross-examination, it was only possible to look at a small selection of those documents
— to pull the brick out of the wall and see whether it had substance. It is also impossible
to deal with all his hundreds of documents in these submissions. The documents to which
he was taken made it clear that the deficiencies he had purported to identify were not
made out, either at all or to the extent portrayed. A small and rarely encountered problem
was often presented as though it were both serious and widespread.
As an expert he should have ensured that his approach to the documentary evidence was
reliable: the Court cannot be expected to unpick every detailed allegation made by an
expert by going to each and every document that might be relevant to the point. There
was good reason for appointing experts and giving them very considerable budgets to
conduct a review of the huge volume of documentation and information at issue.
Mr Coyne is an intelligent man and an experienced expert. He will have known both that
his vastly long and complicated reports would be difficult to pull apart brick-by-brick
and that the sheer volume of points taken against Horizon would generate a powerful
impression. The approach taken in his reports was not conducive to clarity and balance
but to overwhelming the reader with a litany of apparent defects. His essential endeavour
was to throw mud at Horizon in the expectation that at least some of it would stick, and
not to worry too much or at all about giving the other side of the picture or even
presenting a fair view of the documents to which he referred.
Mr Coyne was given ample opportunity to comment on these criticisms of his approach.
He could give no proper explanation as to why he had not given a more balanced view.
In the most striking exchange, after Mr Coyne had agreed with a series of important
propositions about the quality of Fujitsu’s support operations, including fixing bugs and
identifying the affected branches so that they could be made whole, he was asked why
none of these points had been mentioned in his reports:
Q. What I don't understand, Mr Coyne, is why you didn't say any of that in your
report. Would it not have been a balanced thing to do?
A. Perhaps. Looking back at the report, possibly.
103,
POL00026925
POL00026925
A/8/103
269.
270.
271.
POL00026925
POL00026925
Q. Would it not have been helpful to explain the good aspects that you had spotted in
the system as well as the bad ones?
A. Possibly.
Q. Did you have a reason for wanting to keep it back?
‘A. No, not at all.”
Q. But isn't it a necessary part of the judgment you make as to whether Horizon is
robust? How can you make a judgment about that without taking it into account?
A. [think you can take it into account, but to spend pages of text talking about all the
various good things, I don't see there's any value in that.*”' (emphasis added)
Mr Coyne devoted many hundreds of pages of evidence to identifying various bad things,
but he considered it not worth the ink to address the good things. It is difficult to imagine
a more basic failure to adopt a balanced approach. Mr Coyne’s approach was to find
some evidence (no matter how strong) that might support a criticism of Horizon, to
include the criticism in his report and then to move onto the next one, very often without
regard to chronology, scale, seriousness or impact of the actual or apparent problem. It
was suggested to Mr Coyne that he had a “world view...a desire to maximise the
99392
impression given of any error that [he identified]. Mr Coyne denied that that was the
case. It is submitted that that was the overwhelming impression left by his evidence.
This general failing in Mr Coyne’s reports was manifest in three more specific flaws.
First, Mr Coyne failed to give a fair summary of the content of documents on which he
relied. This was particularly concerning given that Mr Coyne, as an experienced expert,
was well aware of the trust that would necessarily be placed in him, especially in light
of the vast number of documents that he had reviewed and the practical limits of the
scope of cross-examination and judicial reading time. He confirmed this in oral
evidence.”
3 (Day14/95:5} to {Day14/95:14}.
31 (Day14/96:5} to {Day14/96:10}
3 (Day15/11:20} to (Day15/11:22}.
393 (Dayl4/145:1} to {Dayl4/146:18}.
104
AI8/104
POL00026925
POL00026925
272. There were many examples of Mr Coyne having given an unfair or misleading summary
of the content of documents on which he relied. There were three particularly striking
instances:
Example 1: Cash management document
272.1 In Coyne 1/5.195,*** Mr Coyne made a strong claim that a cash management report
dated 4 August 2017 showed a “real risk” of bugs “adversely impacting branch
accounts”:
The Post Office cash management proposals contained in a report dated 4
August 2017 suggests that they were actively considering ways to improve
processes impacting on many of the issues raised above. It is my opinion that,
whilst the Post Office was looking at ways to improve cash management, it is
also indicative that the system was generally far from perfect and there existed
a_real risk of bugs/errors/defects adversely impacting on branch accounts
despite the processes in place at the time to prevent this. (emphasis added)
272.2 Mr Coyne was taken to the document referred to.*°* He slowly trawled through its
pages in Court but eventually accepted that it did not support the point he was
making:
Q. Yes. So can you tell me what it is you saw in this document which allowed
you to express the opinion that it indicates that there existed a real risk of bugs
adversely impacting on branch accounts?
A. It is incorrect to find that from that document.*°®
272.3 Mr Coyne initially suggested that this was simply a case of an incorrect reference
having been given, although he then rightly accepted that this was unlikely here
given that the specific document was referred to in terms in the paragraph itself.°°”
But for the opportunity to cross-examine on this paragraph, a wholly misleading
impression would have been created.
39 ¢D2/1/107}.
38 (F/1673}.
396 {Day15/7:10} to {Day15/7:14}.
37 (Dayl5/8:2} to {Day15/8:6}.
105
Al8/105
Example 2: Cost-benefit assessment of potential bug fixes
272.4 Mr Coyne contended in his reports that fixing bugs was considered on a cost/
benefit basis. He referred in this regard to the Post Office Risk & Compliance
3°8 as evidence that “known
Committee minute dated 18 September 201
issues/bugs were often deferred and dealt with on a cost benefit basis” >°? Mr
Coyne accepted in cross-examination that the document in fact had nothing at all
to do with deferring fixing bugs: see further at paras. 397 et seg below.
Example 3: Call logs undermining “the credibility of Horizon itself”
272.5 In Coyne 1/5.198, Mr Coyne contended that there were recurring discrepancies in
relation to which it was not apparent whether these were the result of a bug or the
SPM’s own actions.*” He referred to two call logs’!
in support of this point,
saying that it “does undermine the credibility of Horizon itself’. That strong claim
turned out to find no support in the documents to which Mr Coyne referred. It was
clear from each of the documents that Fujitsu (specifically, Anne Chambers) had
investigated the problems raised by the SPM and, after that investigation, had
concluded that there was no system fault and that the problems must be the result
of user error, noting that the branch appeared to run chaotically: see the long entry
at the bottom of {F/333/1} and the first entry at {F/1019/2}. Mr Coyne was taken
to {F/333} in cross-examination and accepted that it did not in fact “undermine
the credibility of Horizon” *° The same is true of {F/1019}.
273. Mr Coyne’s mis-reading of the documents may in some instances have been wilful or
the result of a strong predisposition to see only bad things; in other instances, it may
simply have been the result of carrying out searches and then relying on a document as
an example of a given issue without properly considering the entirety of the document(s)
returned by the searches. Either way, the upshot is that the Court cannot safely assume
POL00026925
POL00026925
598 F/1140/3}.
3 Coyne 1/5.161 {D2/1/97}.
40 {D2/1/107}.
401 (F/333} and (F/1019}.
42 (Day15/14:17} to {Day15/19:17}.
106
AI8/106
274,
275.
276.
277.
POL00026925
POL00026925
that, because Mr Coyne cites a document as evidence of an issue, it does in fact provide
any support for the point that he makes.
Second, Mr Coyne drew broad and seemingly important conclusions (always adverse to
Horizon) based on weak, thin or irrelevant evidence. A strong example of this is the
flimsy basis on which he contended, in both his reports, that bug fixes were deferred or
not carried out all based on a cost/benefit analysis: see further at paras. 397 er seq below.
Third, Mr Coyne presented evidence of problems or issues without putting them in the
necessary technical context, drawing the reader into a misunderstanding of the nature or
importance of the evidence set out. A strong example of this was Mr Coyne having
referred in his second report to uses of the Transaction Repair Tool (“TRT”) under the
heading “Evidence of Insertions/Deletions within Branch Accounts” without explaining
that the TRT can only be used to make corrections to data in the TPS, which does not
form part of the branch accounts. Mr Coyne accepted this had been “misleading” **: see
further at paras 926-219 et seg below.
Mr Coyne’s application of the wrong test when addressing the Horizon Issues
Mr Coyne explained that the approach he had taken was to ask himself if the risks
inherent in Horizon had been reduced “as far as possible”. That is not the standard set
out in any of the Horizon Issues. It is a standard that any major commercial IT system
would inevitably fail. This chimes with the doctrine of perfection identified in paras 140
and 150-155 of Post Office’s opening submissions.‘
Mr Coyne was clear about the approach he had adopted (responding to a point about
applying a cost/benefit analysis):
Q. You repeat this point in the third joint statement and you refer to it half a dozen
times in your second report, don't you?
A. Ido, but it is important because the question that was asked was: was it reduced
as far as possible?
Q. I'm sorry, where is that asked?
“3 {Day16/66:13} to {Day16/68:22}.
408 £4/2/54-57}.
107
AI8/107
POL00026925
POL00026925
A, It is one of the Horizon Issues.
Q. I see.
A. So when you consider "as far as possible", cost benefit shouldn't really be involved
in that consideration. It should be whatever is possible.” (emphasis added)
278. When the point was returned to the next day, Mr Coyne agreed that he had applied this
exacting standard to the Horizon Issues more generally:
Q. In your evidence yesterday we discussed your approach, remember, to whether
and to what extent Post Office and Fujitsu did things on a cost benefit basis?
A. Yes.
Q. In the course of that evidence I recall you indicating that you regarded it as
important to ascertain whether the possibility of error was reduced as far as possible.
Do you remember that exchange that we had?
A. Yes.
Q. Was it your objective in your reports to address that question?
A. Was it my objective at the outset to address that question?
Q. To consider not whether the risk was reduced as far as reasonable or to consider
whether the risk was reduced as far as practicable, but to consider whether the risk
was reduced as far as possible, which is a much more exacting standard?
A__I believe that was the word that was used in the Horizon Issues.
Q. So would the answer to my question be yes, that when you produced both your
reports you did so with the objective of applying that test when determining whether
something constituted a problem in the system or not, whether it satisfied the test of
reducing a risk as far as possible?
A. Yes.
Q. And not just with -- and did that inform -- does that inform actually the approach,
the criticisms you make of the use or non-use of ARO data in your second report?
A._Yes.*°° (emphasis added)
45 {Day14/150:13} to {Day14/150:23}.
“6 {Day15/88:12} to {Day15/89:17}.
108
A/8/108
279.
280.
281.
283.
POL00026925
POL00026925
This erroneous approach was also reflected in Mr Coyne’s contribution to JS3, para. 6.3
where he stated that there was evidence that the risk of errors had not been reduced as
far as “possible” (quotations in the original).*°”
Issues 1, 3 and 4 ask “[t]o what extent” various things were likely or possible. Issue 6
(the one referred to by Mr Coyne when questioned on his “as far as possible” standard)
asks “[t]o what extent did measures and/or controls that existed in Horizon prevent,
detect, identify, report or reduce to an extremely low level” various risks. That is not the
same thing as asking whether risks were reduced to the lowest possible level. Mr Coyne’s.
answer in relation to ARQ data shows that he applied his “as far as possible” standard
across practically all issues.
This error of approach must be taken into account in assessing Mr Coyne’s evidence — it
necessarily affected the threshold that he applied for determining whether
countermeasures were implemented and working sufficiently effectively to counter any
material risk of inaccuracy in branch accounts. It must also have infected his approach
to the adequacy of controls and permissions (Horizon Issue 11).
Mr Coyne’s inconsistent approach to questions of extent
Many of the most important Horizon Issues ask questions of extent: see Horizon Issues
1, 3, 4, 6 and 13 (using the word “extent”) and Horizon Issue 12 (asking “how often” a
facility was used). That is not an accident. It is common ground on the generic pleadings
that Horizon is imperfect, has suffered from bugs and has had adverse impacts on branch
accounts; it is also common ground that Horizon contains measures aimed at reducing
the incidence of such impacts. The dispute between the parties revolves around questions
of degree.
Mr Coyne refused in his reports to engage in any meaningful or objective consideration
of extent. He typically refused, even in cross-examination, to be drawn on matters of
degree, extent or probability, or rather he refused when the answers would tend to support
a positive view of Horizon’s robustness and reliability. For example, he refused to accept
that it was “quite unlikely” that a change made to data in the TPS that was made for the
“7 {DI/4/9}.
109
A/8/109
284.
285.
purpose of ensuring the consistency of TPS data with the (separate) branch account data
would instead create the opposite effect, introducing an inconsistency. He would say only
that the chance was “whatever the chances are of a human error” “°°
Mr Coyne’s position was inconsistent here. In his reports, Mr Coyne was happy to use
words like “many” and “often” when referring to actual or alleged problems in Horizon
—words that create a strong impression of the problem occurring with a moderate to high
frequency.*”* It became clear in cross-examination, however, that by using words like
“many” and “ofien”, Mr Coyne was in some cases referring to a handful of occurrences
across 20 years. A thing that happens every 5 or so years does not happen “many” times
or “ofien” on any ordinary usage of those terms, and the reader would naturally have
taken a very different impression from his reports. For example:
284.1 Mr Coyne’s accepted in cross-examination that it might have been unfair of him
to suggest in his reports that “known issues/bugs were often deferred and dealt
with on a cost/benefit basis” because there was no evidence that any such thing
happened “ofien”: see para. 400 below.
284.2 In justifying his statement that "J have noted that hardware replacement often
seemed to be a "fix" of last resort where no other explanation could be given'"*"°,
Mr Coyne explained in cross-examination that he considered somewhere in the
region of five supporting examples to constitute this occurring "offen".
The same was true of Mr Coyne’s written evidence that his analysis of Peaks had shown
that “ofien bugs lay undetected for weeks, months or years”,’? causing apparent
shortfalls for which SPMs were held liable. This was revealed (through correspondence)
to be a reference to 4 bugs, one of which was the Dalmellington bug (which Mr Coyne
accepted at trial did not cause any shortfalls for which any SPM was held responsible)
POL00026925
POL00026925
“8 {Day16/54:6} to {Day16/55:9}.
“© Coyne 1/5199 {D2/1/108}, Coyne 2/5.108 {D2/4.1/156}, Coyne 2/3.13 {D2/4.1/14}, Coyne 1/3.18
{2/1/28}
8 Coyne 1/5.64 {D2/1/71}.
“Al {Day15/137:17} to (Day15/139:18}.
82 Coyne 2/3.108 {D2/4.1/157}.
110
A/8/110
286.
287.
POL00026925
POL00026925
and the others of which had always been known (before his Peak analysis) to have been
bugs that had taken some time to detect.*!
Mr Coyne was equally happy to speculate in para 3.105 of his second report that there
were “potentially thousands more PEAKs that illustrate financial discrepancy arising in
branch accounts”.4\* That was wholly at odds with Mr Coyne’s realistic acceptance at
trial that the total number of bugs detected by Fujitsu and recorded in KELs was likely
in the low tens.*'> He then shifted his ground to trying to defend a suggestion that there
could possibly be one thousand Peaks showing a financial impact on a branch (rather
than thousands, plural),‘"© and even then only if one adopted an assumption that each
bug would affect very many branches (which is flatly untrue of most of the bugs with
financial impacts). The idea that there could be even 1,000 further Peaks showing effects
on branch accounts is fanciful given that (1) Mr Coyne’s sophisticated searches across
KELs and Peaks would have identified them and (2) there are very many fewer than
1,000 Peaks for the 29 bugs in the bug table (which, on Mr Coyne’s oral evidence,
account for the substantial majority of bugs in KELs), and even then a proportion of
those Peaks relate to bugs with no financial impact.
Mr Coyne’s failure to set out his true and full opinions in his reports
As discussed in detail at paras 342-335 et seq below, Mr Coyne formed a favourable
impression of the quality of the SSC’s work in identifying and addressing problems in
the system. This favourable impression was sufficiently strong and important that it
caused him to change his view on Horizon’s robustness from adverse (in JS1) to positive
(in Coyne 1). He failed to mention this in either of his extremely lengthy reports. The
reason that he gave for this was troubling — he did not see “any value” in setting out the
“various good things” about the system: see the full quotation at para. 2¢
above.
413 See the letter from Freeths at {C5/36/5}, identifying the Peaks relied on by Mr Coyne as those in the
section of the report on the three acknowledged bugs plus Dalmellington.
“4 42/4/43}.
“18 Mr Coyne agreed that the likely number of KELs with bugs was likely to be less than 40: {Day15/124:6}
to {Day15/124:9}. The number of KELs relating to bugs with financial impacts is even lower than that.
06 {Day15/135:5}.
inal
Al8/111
288.
289.
290.
291.
POL00026925
POL00026925
There are other important points that Mr Coyne failed to mention in his reports, each of
them strongly favourable to Post Office’s case, as identified elsewhere in this section.
These included, for example:
288.1 Mr Coyne’s acceptance that it was likely that countermeasures worked effectively
except in a “fraction of a percentage” of instances. Mr Coyne suggested that this
had been captured by his written evidence that “Ir is...reasonably likely that in the
majority of cases the measures and controls were successful”. Those two opinions
are starkly different both in content and emphasis.*!7
288.2 Mr Coyne’s oral evidence that there was evidence of less than 30 instances of
remote access with any effects on branch accounts, that Fujitsu was careful when
using remote access facilities and that the chance of error being accidentally
introduced into the relevant branch account was small: see further at paras. 744.
745 et seq below. A wholly different impression would have been taken from his
reports.
The case was opened on the basis that there were fewer, and less serious, differences
between the parties than their formal positions (including experts’ reports) might suggest.
This became ever more apparent during Mr Coyne’s cross-examination. It should not
have been necessary to prise out these points at trial.
Mr Coyne’s evasive answers and reluctance to concede points
Mr Coyne did ultimately make important concessions. It is, however, a point against him
that these concessions did not always come easily or immediately — in several instances,
Mr Coyne first sought to avoid the question or to change the subject.
Consequently, Leading Counsel had first to close off all “escape routes” before Mr
Coyne would make the concession that he could and should have made at the outset, had
he wished to be fair and forthcoming.
"7 {Day14/148:3} to {Day14/149:12}.
AI8/112
292.
293.
Mr Coyne’s confusing position on the 29 bugs in the bug table
On the final day of Mr Coyne’s cross-examination, he confirmed that his view at the
time of agreeing JS2 had been that there was strong evidence that each of the 29 bugs
identified in the bug table had a lasting impact on branch accounts.*'® But he
subsequently seemed to want to change his position, and the Managing Judge asked him
immediately before lunch how many of the 29 bugs he contended had a lasting effect on
branch accounts. He said 13.‘"? After lunch, Mr Coyne said that he had been mistaken
and there were 21.” This increased to 22 when Mr Coyne was taken through the bug
table item-by-item.*”! Mr Coyne then refused to accept that, at the time of JS2, he had
ever contended that all 29 bugs had lasting impacts on branch accounts, directly
contradicting the answers he had given before lunch (and C’s opening submissions).
It appears that Mr Coyne had not given any real consideration before the trial as to
whether or not the bugs identified in the bug table had lasting impacts, but he had
nonetheless been content to record in JS2 that, in his opinion, all 29 of them had such
impacts.**
Perhaps he did not feel able to be open about this because he had already
accepted that the Horizon Issues required him to consider countermeasures (not least in
response to Horizon Issue 6) and that the distinction between transient impacts and
lasting impacts formed an important part of answering the Horizon Issues.
The changed factual basis of Mr Coyne’s evidence
Mr Coyne relied unquestioningly on the witness evidence of many of Cs’ witnesses
including, in Coyne I: Mr Roll (in relation to Horizon generally and Fujitsu practices*”*);
Mrs Burke (in relation to an alleged recovery failure’”* and a TC failure’); Mr Latif (in
POL00026925
POL00026925
“08 {Day17/104:20} to {Day17/104:25}.
“19 {Day17/106:11} to {Day17/106:16}.
20 {Day17/107:24} to {Day17/107:6}.
2 This was the final number settled upon: {Day17/135:8}.
“22 Dr Worden’s position was that there were 12 such bugs (as is apparent from his comments in the bug table
itself and JS2, para, 1.2{D1/2/27}). Mr Coyne agreed that Dr Worden’s number was 12: (Day17/114:1} to
{Day17/114:7}.
83 Paras 5.78 — 5,81 {D2/1/75}; para 9.10 {D2/1/150}; para 9.22 {D2/1/152}; 9.44 {D2/1/157}.
24 Para 5.40 {D2/1/65}.
85 Para 6.65 {D2/1/127}.
113
AI8/113
295.
296.
297.
298.
299.
POL00026925
POL00026925
relation to an alleged TC problem with scratch cards*”°);
and Mr Patny Jnr (in relation to
Moneygram transactions’). Much of that evidence changed substantially at trial,
usually in Post Office’s favour, and that must be weighed in the balance when
considering Mr Coyne’s views.
Judicial criticism of Mr Coyne’s approach
It appears from CGI IT UK Ltd v Agilisys Ltd [2018] 12 WLUK 13, a recent Scottish
decision of the Outer House, Court of Session, that Mr Coyne has been criticised in
another case for adopting the same unbalanced approach that is summarised above. Some
extracts of the case are set out below. The Court is invited to read the whole section on
expert evidence in paras 109-154.
Lord Bannatyne said this at para 109:
“109. Turning to the expert evidence in the case, the expert for CGI was Mr Coyne. I
have come to the view that his evidence was one-sided. His approach was I believe
not balanced. In addition for various other reasons I believe his evidence was not
acceptable. In arriving at this view I considered a number of specific matters in
relation to his evidence.
The Court of Session went on to criticise Mr Coyne’s failure to give a clear answer to a
simple question: see paras 112 to 115, where Lord Bannatyne also remarked that Mr
Coyne had no explanation for failing to refer to certain matters that told in favour of the
party other than the party instructing him.
See also paras 126 to 127, where Mr Coyne is criticised for having tried to “reverse his
position” and for “switching from one position to another, which was “highly
unimpressive in the context of someone who is being offered as giving expert evidence”.
It is submitted that the examples set out above show the same tendency to reverse and
switch in response to difficult points.
At para 141, Lord Bannatyne recorded a strongly adverse view of Mr Coyne’s evidence:
26 Para 6,69 {D2/1/128}.
27 Para 5.185 {D2/1/104}.
114
Al8/114
300.
301.
302.
303.
“141. Overall I did not form a favourable view of this witness and I am not prepared
to accept his evidence on disputed issues in preference to that of Dr Hunt. It did not
seem to me for the above reasons that I could place reliance on any of his views.”
There are clear parallels between the approach taken by Mr Coyne in the Agilisys case
and this one: a failure to deal properly and fairly both with the issues and the evidence;
evasiveness; backtracking; and overall a lack of balance.
F4. The searches carried out by Mr Coyne and his team
Although, as set out above, there is much to criticise in Mr Coyne’s approach to the
Horizon Issues, it is clear that he worked extremely hard, and extremely efficiently, to
identify problems with the Horizon system. He said in JS1 that this was what he was
going to do, and he applied himself impressively to that task. This is the context in which
Mr Coyne’s evidence (both for and against Horizon) falls to be considered.
Mr Coyne is a director of IT Group which offers a “powerful e-disclosure and digital
investigation solution” °* He was, for a substantial proportion of his time working on
this case, assisted by four members of his team. They spent a significant amount of their
time on the case.” Mr Coyne set them the task of looking for problems:
Q. Would it be fair to say that you were generally asking them to find problems, to
find evidence of problems in Horizon, things that had gone wrong?
A. Yes. Well, sorry, I had already identified the theme that I wanted them to look for
but that was typically around a specific type of defect that might be occurring around
a particular time and I was asking them to look at documents around that time or that
contained a similar theme.“*°
The searches carried out were skilled, assiduous, extensive and effective:
POL00026925
POL00026925
Q. So it is fair to say, isn't it, that you know quite a lot about intelligent searching of
documents?
A. Yes.
Q. And you have no difficulty, I'm not saying it is easy, in searching through, for
example, 200,000 PEAKs in lots of very clever ways that I am sure I couldn't think
of?
8 (Day14/108:13} to {(Day14/108:21}
°° ‘Day14/102:5} to {Day 14/104:23}.
80 Dayl4/104:24} to {Day14/105:7}.
11s
Al8/115
304,
305.
A. Yes.
Q. Would that also be true of your team of assistants?
A. Yes.
Q. And they would have used these intelligent search techniques to go away and find
the sort of documents that you wanted them to find?
A, They would.
Q. At your daily meetings?
A. Yes."*!
Mr Coyne’s team was able to eliminate irrelevant documents and tag documents which
Mr Coyne was likely to find of particular interest, for him to then review. The process
432
was refined day by day.**? Mr Coyne was unable to say how many documents had been
reviewed in total, but he had personally reviewed all the documents referred to in his
reports (including 5,114 KELs**), and many Peaks (1,262 at the time of Coyne 1). His
team had reviewed considerably more.‘*4 By the time of Coyne 2, three and half months
after Coyne 1, he and his team (of which only Siobhan Forster remained by this time***)
had reviewed many more Peaks, as well as other documents including some OCRs and
OCPs, design documents and process documents.**° Since the service of Coyne 2, Mr
Coyne has used intelligent search techniques to review (using intelligent search
techniques) a further 1,200 Peaks, as well as the OCPs, OCRs and MSCs — thousands
more documents.*”
F5. The experts’ approaches to late evidence
The experts’ supplemental reports were served on I February 2019. While Dr Worden’s
report was genuinely supplemental to his first (running to 47 pages without appendices
POL00026925
POL00026925
81 {Day14/109:6} to (Day14/109:20}.
“82 {Day14/111:10} to (Dayl4/111:21}.
“83 {Dayl4/112:24} to {Dayl4/113:1}.
“34 {Day14/113:8} to {Dayl4/113:20}.
“85 (Day14/117:5} to {Dayl4/117:7}.
“86 {Day14/115:22} to {Day14/116:15}.
87 {Day14/118:19} to {Day14/119:13}.
116
Al8/116
306.
307.
308.
309.
and raising few new points), Mr Coyne’s report amounted to a fundamental re-statement
and even replacement of much of Cs’ case (running to 270 pages without appendices).
Mr Coyne served an amended supplemental report on 6 March 2019.48
Since the date of the supplemental reports, the experts continued to meet and discuss
their opinions on the Horizon Issues. They produced three Joint Statements between late
February and early March 2019, as follows:
306.1 JS2, comprising the bugs table and statements of agreement and disagreement on
Issues 1, 2, 9 and 14."°° This was dated 25 February 2019.
306.2 JS3, identifying agreement and disagreement on Issues 3, 4, 5, 6, 7 and 8.“"° This
was dated I March 2019.
306.3 JS4, comprising a statement of agreement and disagreement on Issues 10, 11, 12
and 13 (i.e. the remote access issues other than Issue 7).**! This was dated 4 March
2019.
Mr Coyne served an amended supplemental report on 6 March 2019.”
Both experts also continued to review documents disclosed in the proceedings. However,
they took very different approaches as to how and when to share the fruits of their work,
and any changed opinions, with each other, the parties and the Court.
Dr Worden’s further work and analysis
On 10 April 2019, WBD wrote to Freeths to inform them that they had recently become
aware that Dr Worden had begun a further review of Peaks, OCPs and OCRs and that
his view was that he should share his work with Mr Coyne before preparing a further
report.“
POL00026925
POL00026925
488 This is the report at {D2/4.1}.
89 (1/2).
4 (D1/3}.
“1 DIS}
“2 This is the report at {D2/4.1}.
“43: 19/255.5}.
117
AI8/117
POL00026925
POL00026925
310. On the next day, 11 April 2019, Leading Counsel for Post Office informed the Court of
311.
the position:
MR DE GARR ROBINSON. My Lord there is one more point to raise with your
Lordship which is that Dr Worden has -- it has occurred to Dr Worden there is a new
way of looking at the Peaks and the OCPs, OCRs and MSCs in this case which, in his
view shed considerable light on certain of the Horizon issues. He feels it is his duty
to inform your Lordship of that. He has already informed Mr Coyne of that fact and
it is only right that I should bring it to your Lordship's attention.
MR JUSTICE FRASER: Thank you. I think on the same subject, then, and given that
the expert evidence isn't going to start until the 20th, I'm also minded, unless each of
you seek to persuade me otherwise, to order another expert's meeting anyway.
Later on, Leading Counsel offered a more detailed explanation, consisting of the
following elements:
311.1 An outline of the work that Dr Worden had carried out and his view as to the
relevance of his discoveries to the remote access issues.*4°
311.2 Dr Worden’s apology to the parties and the Court for not having done the work
sooner. 4°
311.3 An explanation that Dr Worden considered he was required, by his duty under
CPR, 35.3 to inform the Court of his change of opinion, such that the Court could
then decide whether or not the new evidence should be considered.‘4”
311.4 A confirmation that this was something that Dr Worden has himself instigated, and
that there was no application from Post Office:
MR DE GARR ROBINSON... I should emphasise this -- none of this comes
at the request or instigation of my client. This has come from Dr Worden.
This is his idea, My Lord, he wishes to discuss it with Mr Coyne in a further
meeting between the experts, but of course it's -- it's only right that
your Lordship should be aware of that. I'm not making any application for
permission to put in supplemental expert reports —
4 {Day12/99:15}.
“45 {Day12/114:23} to {Day12/116:9} and {Day12/116:19} to (Day12/117:1}
“46 {Day12/116:10}.
7 (Day12/117:2} to {Day12/117:12}.
118
AI8/118
312.
313.
POL00026925
POL00026925
MR JUSTICE FRASER: I don't think you have any supplemental experts’
reports to apply for permission for, are you?
MR DE GARR ROBINSON: I'm not making any kind of application, I'm
simply sharing with your Lordship the view that has been expressed to me
by Dr Worden.“
Leasing Counsel indicated that Post Office was mindful of the guidance in ICI Ltd v
Merit Merrell Technology Ltd (No 3) [2018] 1577 (TCC) and that the Court may wish
for the experts to engage in a collaborative process to see whether Dr Worden’s new
approach was beneficial and as to what it did or did not show.” The Court was referred
(although not by paragraph number) to [158] in ICI v Merit, which states (as relevant):
158. Two other points relevant to the application should also be identified. Firstly,
Mr Kitt embarked upon this exercise with no notice at all to his opposite number, Mr
Linnett. This is not co-operative behaviour. Mr Linnett should have been told that this
was being done, and the documents could have even been studied together by both
experts so that an agreed position could be produced by them on what the documents
did or did not show, and the use to which they could or could not be put. I conclude
that the only reason for keeping the fact that this exercise was underway, until afier
it was ready to be served after the trial had actually started, was to cause maximum
disruption to MMT. This is a much discredited approach to civil litigation...
The Court directed that the experts meet again. The experts had telephone meetings and
exchanged without prejudice emails. Before this, Dr Worden had already engaged with
Mr Coyne to see whether some agreed position could be reached in relation to the review
that Dr Worden had carried out. Amongst other things:
313.1 On Ll April 2019, Dr Worden provided his workings on the new analyses to Mr
Coyne.*® The experts briefly discussed the data on 16 April 2019.
313.2 On 25 April 2019, Dr Worden provided a draft further supplemental report to Mr
Coyne. The draft appendices followed on 29 April 2019.45!
“8 {Day12/117:13} to {Day12/117:25}.
4 (Day12/118:7} to {Day12/118:23}.
450 Parsons 17, para.8 {C11/22/3}.
“SI [bid
119
Al8/119
314.
315.
316.
317.
318.
POL00026925
POL00026925
There followed further correspondence between the parties and discussions between the
experts: see paras. 9-13 of Parsons 4: Dr Worden provided the search programme
that he had used to Mr Coyne on 22 May 2019.4
On 22 May 2019, Dr Worden sent a document entitled “Third Expert Report” to the
Court and Cs’ solicitors. The Court raised this by email with the parties and Leading
Counsel for Post Office therefore attended Court the next day, 23 May 2019.** It was
explained that Dr Worden had considered it his duty to bring his new views to the
attention of the Court.4** The Court ordered that the background to this be set out in a
witness statement, which it then was on 31 May 2019 (Parsons 17).*°
Dr Worden stated in cross-examination that he had been “sold” to send his report direct
to the Court, but that word could give the wrong impression. Dr Worden knew that Post
Office did not wish to rely on his new report, would not make an application to rely on
it and so did not propose to serve or file it as an expert report in the proceedings. He
considered that he was nonetheless under an obligation to communicate his change of
opinion to the Court.
In Post Office’s submission, starting at the beginning, Dr Worden acted appropriately to
share his further work with Mr Coyne and the parties at the earliest opportunity. He
adopted a collaborative approach to the unfortunate situation created by his belated
realisation that it would be possible to review Peaks, OCRs and OCPs in the way that he
did in his “Third Report”. He did not keep the fruits of his searches from Mr Coyne or
seck to deploy them only in response to cross-examination.
As regards Dr Worden’s direct approach to the Court, Post Office respectfully suggests
that it may be useful for the Court to provide guidance as to the approach that an expert
in Dr Worden’s position should adopt (including with the assistance of his or her
452 The correspondence from Freeths in this regard might have generated the impression that Dr Worden had
not already, in the without prejudice conversations, offered that search programme to Mr Coyne: see {H/294}
(final two paras). Any such impression would be unfortunate.
“3 (Day13/2:16}.
454 {Day13/3:2} to {Day13/3:18}.
“5 {C11/22}.
120
A/8/120
319.
320.
321.
322.
instructing party).° There does not appear to be any authority as to what an expert
should do in circumstances like those of the present case, namely:
318.1 The expert carries out further work / analysis (whether based on new documents
or old) and forms further or different views on the expert issues.
318.2 The expert considers that CPR, 35.13 requires that he inform the Court of a change
in his views. The expert collaborates appropriately with the other expert(s)
instructed in the case but does not reach any agreed position in relation to the new
work / analysis and what it shows.
318.3 The party instructing the expert does not wish to apply to rely on the new work /
analysis and so does not propose to serve or file any new report that the expert
may produce.
Post Office contrasts Dr Worden’s transparent approach with the approach adopted by
Mr Coyne, considered below.
Mr Coyne’s further work and analysis
It became apparent during his cross-examination that Mr Coyne too had carried out
ions since the
further analyses of various disclosed documents and had formed new opi
date of his reports. He, in contrast to Dr Worden, did not share any of this with the other
expert instructed, let alone Post Office or the Court. He apparently preferred to keep the
fruits of his work (including the documents that he had reviewed and collated) to himself,
and to deploy them, and the new opinions based on them, in response to cross-
examination.
There were several examples of this.
First, on 6 June 2019, the second day of his cross-examination, Mr Coyne mentioned for
the first time that he had identified “hundreds” of OCPs and OCRs in relation to which
consent for a change had been given retrospectively.*” He accepted that he had not
POL00026925
POL00026925
456 [f the Court’s view is that the solicitors acting for Post Office should have assisted Dr Worden by providing
his report to the Court themselves, they apologise for not having done so. That was not the view that they or
counsel took at the time.
“57 {Day16/8:6}.
A/8/121
323.
324.
325.
POL00026925
POL00026925
mentioned this in his reports. He suggested that this was because the OCPs and OCRs
had not been disclosed until around the time of his Supplemental Report (1 February
2019). That is a fair point, given that the OCPs and OCRs were disclosed on 24 January
2019.48
Mr Coyne did not have any proper explanation, however, as to why he had not, since the
date of his Supplemental Report, raised his new discovery with Dr Worden or Post
Office. This was despite the substantial engagement between the experts between
February and May 2019, in relation to the Joint Statements and in relation to Dr Worden’s.
further analyses. There had been ample opportunity to mention the new documents and
seek to agree with Dr Worden what they did and did not show.
Mr Coyne gave an unsatisfactory explanation for having kept the new material to
himself, suggesting that Post Office must have known about the retrospective
permissions because they were contained in disclosed documents.**? He gave the same
explanation as to why he had not provided the new material to Dr Worden.*® Quite apart
from the fact there are hundreds of thousands of disclosed documents in this case, that is
not a proper explanation. If an expert identifies a series of documents that he wishes to
rely in support of a new opinion (neither the documents nor the opinion based on them
being set out in the reports), he must inform the other expert(s) instructed in the case of
the new opinion and identify the documents supporting it. That is what collaboration and
transparency requires.
In this context, Post Office objected to Mr Coyne providing new evidence from the
witness stand, i.e. identifying the number of retrospective permissions for the first time
during his oral evidence, when there was no real opportunity for investigation and
488 4 further tranche of OCRs was disclosed on 18 April 2019: see the letter at {H/263}. The Court directed
that a witness statement be prepared to explain the circumstances of this disclosure: see Parsons 18 at
{C11/23}.
49 {Day16/10:17}.
4 {Day17/3:23}.
AI8/122
326.
327.
328.
329,
330.
POL00026925
POL00026925
challenge.*°! The Court invited submissions from Cs on the point.**? Mr Green QC spoke
briefly but made no application to adduce further evidence from Mr Coyne.
Mr Green QC then extracted the further evidence from Mr Coyne in purported re-
examination, without indicating that this was his intention.“*? That stratagem neatly
circumvented both Post Office’s objection and the requirement to seek permission from
the Court for any new evidence. In the circumstances set out above, that was a regrettable
thing to do.
Second, Mr Coyne stated that he had, since his reports, identified instances (plural) of
the correspondence server being used to insert transactions with a counter number of less
than 32. He came back the next day and was only able to provide a single document,
{F/377.1}.
Mr Coyne’s general explanation for not having given advance notice of the new opinions
and document references that he provided during his cross-examination was that all he
was doing was responding to the questions put to him: see {Day16/9:6}4° and
{Day16/14:15}.The implicit suggestion was that he was responding on-the-hoof.
In fact, {F/377.1} had been added to the trial bundle at Cs’ request on 3 June 2019, the
day before Mr Coyne’s cross-examination began. Mr Coyne introduced {F/377.1} at
{Day16/134:13} and following. He had noted or attempted to memorise the trial bundle
reference (which he got wrong), and he had a note of the Peak number.
It would seem that Mr Coyne identified the document in advance and anticipated
referring to it in the course of his cross-examination, but preferred to wait for the
questions to be put, rather than giving both parties notice of the document and the points
he took from it. He also failed to inform Dr Worden that he had identified supposed
evidence of message insertions via the correspondence server with a counter number
461 Dayl7/8:24} to {Day17/9:20}.
462 {Dayl7/9:22}.
46 (Dayl7/184:1}.
464 (Day16/94:11} to {Day16/95:3}.
465 Tt is fair to observe that Mr Coyne did not make this point in relation to {F/377.1} specifically, but it was
the general impression that he sought to create in relation to his new documents and opinions.
123
AI8/123
331.
332.
333.
POL00026925
POL00026925
lower than 32 so that Dr Worden could consider that evidence. Mr Coyne’s explanation
that he had found the document over lunch was unconvincing: “My Lord set some
homework for me and I started it at lunchtime...” 4
Third, Mr Coyne referred to further searches that he had conducted to identify transaction
insertions under Legacy Horizon:
Q. Do you accept, though, that those search terms are likely to capture the sort of
transactions that we are talking about now?
A. They do. They don't include the word "Riposte import" or "message import",
which is the one that I have since used, so whether that brings back additional
messages, I'm not sure.
Q. So you have done your own searches now, have you?
A, After seeing these words here I have looked at those documents and found the
command Riposte import in some of those documents, so I have gone back to search
for that across the whole database. *
When pressed for what these searches had revealed, Mr Coyne proposed to go away and
468 generating new evidence during the
re-run the searches and then provide the results,
course of his cross-examination. Even putting to one side that it is unclear why Mr Coyne
would have to run the searches again (rather than having retained the results of them),
there was no explanation as to why the fruits of the further review that he had conducted
had not been shared with Dr Worden or provided to Post Office before the cross-
examination.
Fourth, Mr Coyne referred to a few OCPs and OCRs in which the same person was
identified as having both carried out and witnessed the change (which appeared to be
presented as evidence of a breach of Fujitsu’s “four eyes” control in relation to remote
access).*°? Mr Coyne had no proper answer as to why this had not been mentioned in
JS4, which deals in some detail with controls on remote access and refers to many OCPs
466 {Day16/135:21}.
“7 Day16/99:12} to {Day16/99:23}.
468 (Day16/100:4} to {Day16/100:20}.
“© {Day16/13:7} to {Day16/13:13}.
124
AI8/124
334,
335,
336.
337.
POL00026925
POL00026925
and OCRs.*” There is no reason for Mr Coyne not to have raised the point even after
JS4, in a spirit of collaboration with Dr Worden.
Standing back from the detail, Mr Coyne conducted substantial further work on matters
that he considered important to the Horizon Issues, identifying various documents that
he considered material to his opinions, and he decided not to share the results of that
work with Dr Worden and, instead, to keep it back for deployment in response to cross-
examination. That approach is inconsistent with the principles identified in ICI v Merit
and, more generally, the proper discharge of an expert’s duties under the CPR.
F6. Fujitsu’s support role and its importance to the Horizon Issues
Fujitsu built the Horizon system. The system was, from the outset, designed to ensure a
very high degree of data integrity. It could hardly be otherwise given that it was always
intended to handle vast amounts of important data. It was and remains a very expensive
system to build and operate, and Dr Worden observed that Fujitsu had ample budget for
its Post Office operations.‘7!
In addition to building the system, Fujitsu has supported, monitored and upgraded
Horizon throughout its life. The Court will have noticed how many of the same names
appear over and again in the Peaks to which the experts and witnesses were referred trial
— Anne Chambers and Pat Carroll, for example. It is common ground between the experts
that these are experienced and highly-skilled workers. They had the benefit of working
on the same system for years, even in some cases from its inception.
Horizon as a “tightly run ship”
Dr Worden formed a strong favourable impression of both the design of Horizon and
Fujitsu’s working practices in supporting the system in operation. He reached the view
that Horizon was a “tightly-run ship” based on, amongst other things, the high quality of
its technical documentation, design, review and testing and the “high quality and
4 {Day16/14:11} to {Day16/15:4}. See the references to OCPs and OCRs at {D1/3/5-7}
47. Day18/106:23} to {Day18/107:5}.
Al8/125
338.
339.
340.
342.
POL00026925
POL00026925
effectiveness of problem analysis and problem solving shown in KELs and Peaks”. He
recorded this at para. 3.23 of JS3.4”
By the time of JS3 (1 March 2019), both experts had conducted extensive reviews of
KELs and Peaks, documents that provide ample evidence of how the SSC identified,
investigated and resolved bugs and other issues. Each of the experts had by the time of
JS3 produced a supplemental report, in Mr Coyne’s case running to 273 pages.
Mr Coyne made in JS3 no positive comment on any of the aspects of Fujitsu’s work that
Dr Worden identified in the passage quoted above. He also made no substantial positive
comment in his supplemental report. The Court may well have formed the view that Mr
Coyne did not agree with Dr Worden’s assessment of Fujitsu’s documentation and the
quality of the system support that it provided, including through the analysis and problem
solving evidenced in the KELs and Peaks.
The true position was that Mr Coyne largely shared Dr Worden’s assessment of Fujitsu.
But this emerged only at trial. There were three important points on which Mr Coyne
agreed with Dr Worden. These are set out below. In summary, Mr Coyne agreed that:
340.1 Fujitsu ran a good support operation.
340.2 If a bug was detected in the system, it was highly likely to be recorded in a KEL.
340.3 Peaks and KELs provide a good source of information in relation to bugs.
. These important revelations should not have been revelations at all. They were points
that Mr Coyne should have set out fairly in his reports, or at the very least agreed in the
Joint Statements. Each of them merits separate consideration.
(1) Fujitsu ran a good support operation
In JS1, Mr Coyne expressed the view that Horizon was not a robust system.’ That was
on 4 September 2018. By the time of his first report, dated 16 October 2018, Mr Coyne’s
4 (D1/4/5}.
4 DI/1/9}.
126
AI8/126
343.
344.
345.
POL00026925
POL00026925
view had changed — Horizon was “relatively robust” (a conclusion the significance of
which was unpacked at trial: see para 374 et seq below)
Mr Coyne was asked what had caused him to change his mind (given that this was not
set out in his first report). Mr Coyne was readily able to explain:
Q. What is it that was good about Horizon that caused you to change your mind
and where do you describe that, where do you consider that in your first report?
A. What was helpful was to understand the support process in more detail to
understand how things such as fault determination is done, albeit it is only an
understanding of how it is done within Fujitsu, to understand that process more.
So that was an improving position for me.
Q. I see. And your judgment on reviewing the PEAKs was that Fujitsu actually
had quite a good support process, is that right?
A. Yes, I mean the support process that Fujitsu operate, and again this changes
over time so it is very difficult to pick a point in time and understand what
obligations they had, what roles and responsibilities they fulfilled, but certainly by
the time they become aware that somebody believed there was a problem, so it hit
SSC, the support centre, third line support, the process they had of determining
whether there was a fault appears to be a reasonably good process...
In short, Mr Coyne had assessed the evidence of Fujitsu’s support operations, including
in identifying and addressing problems, and he had been impressed. That was of
sufficient importance to cause Mr Coyne to change on his mind on robustness,
notwithstanding all the negative points he had set out in JS1. It is regrettable that he did
not think it appropriate to set this out in his written evidence. At trial, Mr Coyne gave
clear and consistent answers to the effect that Fujitsu did its job well.
Mr Coyne accepted that the system of Peaks and KELs was thorough and well run:
Q. Thank you. Let's move on to a different subject. Perhaps I can deal with this
quickly. I would like to talk about PEAKs and KELs. From what you said yesterday
about your change of mind on robustness between the first joint statement and your
first report, I imagine you would agree that the system of KELs and PEAKs that
Fujitsu developed was quite a thorough system?
A. Yes.
4 Coyne 1/3.7 {D2/1/26}
“75 {Day14/90:17} to {Day14/91:12}.
127
Al8/127
POL00026925
POL00026925
Q. And that you formed the view that members of the SSC were very familiar with the
Horizon system?
A. Yes.
Q. And they were very familiar with the PEAK and KEL system?
A. Yes.
Q. And with their training and experience and with using search facilities they were
able to navigate that system quite well?
A. Yes.
Q. Notwithstanding the limitations that you have fairly identified. And that using
search facilities they were often able to find PEAKs or KELs addressing similar
problems to the ones that they were facing?
A. Yes.
Q. And would you agree that the PEAKs show, generally show, the thoroughness with
which they generally worked?
A. Yes.
Q. And they tended to keep a written record of what they did step by step in PEAKs,
didn't they?
A. Yes.
Q. It wasn't comprehensive, no one is suggesting it is comprehensive, but it's quite a
process-driven process, one doesn't often see something significant happening that
isn't somewhere recorded or alluded to in the PEAK during the different processing
steps that are described as you go down the PEAK from the top.
A. Yes.
Q. So in the scheme of things, compared with other systems with which you are
familiar, you would accept, wouldn't you, that this is actually quite a well organised,
well run system?
A. Certainly the way of recording the information in the PEAKs and KELs is a
reasonably good system, yes.*°
346. Areader would have formed a contrary impression from Mr Coyne’s criticism of KELs
as a source of evidence in his second report,‘”’ along with the litany of “/imitations” that
46 {Day15/91:7} to {Day15/92:24}.
4” Coyne 2/paras 3.2-3.4 {D2/4.1/12}.
128
AI8/128
347.
348.
349,
POL00026925
POL00026925
he identified in the Peak system.*”* It would have been easy for Mr Coyne to make clear
that the system was a good one, albeit with imperfections. He could and should have
done so.
Mr Coyne accepted that Fujitsu was good at spotting when there was a problem in
Horizon and in fixing it:
Q. So what you found when you read the PEAKs was that when a call got referred to
the SSC, either because it is a subpostmaster call, or perhaps it might be automatic,
it might be from the MSU because of a reconciliation issue, your view was that Fujitsu
was quite good at spotting if there was a problem in Horizon, is that right?
A. Yes.
Q. And it was quite good at making sure that problem was fixed, yes?
A. Yes.”
He also accepted that Fujitsu was also quite good at identifying the branches affected:
Q. Just to go back to my question. Fujitsu are quite good at identifying the branches
that have been affected by those kind of bugs?
A. In the main, yes.
Mr Coyne inferred from what he saw in the documents that the information Fujitsu
obtained about effects on branches was passed on to Post Office. He had seen only one
Peak which suggested otherwise (and even that was actually about a situation where
there was no loss to correct):**!
Q. And you don't assume -- you don't infer, do you, that having identified those
branches Fujitsu then keep that information to themselves? You infer that that
information is passed onto Post Office don't you?
A, I think so. There is certainly one reference in a PEAK where it says I suggest we
don't tell the branch about this, but I'm not sure whether that's we won't tell Post
Office about it, it is more keeping it from the branch rather than Post Office
Q. And that's the only PEAK of any kind that you have ever found of that sort, isn't
it?
“8 Coyne 2/para. 3.12 et seq {D2/4.1/14}.
Day14/92:4} to {Day14/92:14}.
480 (Day14/93:12} to {Day14/93:15}
“81 See paras 771 ef seq below.
129
Al8/129
350.
352.
353.
354.
A. Yes, I believe so.
The Court will recall that the evidence of the factual witnesses was to the same effect.
(2) Ifa bug was detected in the system, it was likely to be recorded in a KEL
. Mr Coyne accepted that once a bug is identified, a KEL will generally be produced and
will address the first instance of that bug.**?
Further manifestations of that same bug
might not be referred to in the KEL, unless the bug manifests in a slightly different way,
in which case there will generally be an amendment to the KEL to explain the new issue.
The KEL might also be updated with new information obtained about the bug.***
(3) Peaks and KELs both provide a good source of information in relation to bugs.
Mr Coyne accepted that a KEL will generally indicate whether a bug had branch impact:
Q. Generally speaking then, if there is a KEL that addresses a bug that has a branch
impact, generally it will tell you?
A. Generally speaking.
Q. On the vast majority of occasions, yes?
A. I'd prefer to stick with generally speaking. I don't know if it is the vast majority.*®°
Dr Worden’s evidence was to similar effect — the general position is that branch effects
will be identified in the KEL because the "KEL is clear as to what happened" and is a
"kind of distillation" of the relevant information, with the Peak being a "/ong-running
486
diary".
's
Conclusion on Fujitsu’s support operation generally
It emerged only at trial that there was a good measure of common ground between the
experts on the high quality of Fujitsu’s support operation. That common ground provides
important context to many of the Horizon Issues. It is inevitable that there will not be a
POL00026925
POL00026925
*82 {Day14/94:10} to {Day14/94:21}.
“8 (Day 5/94:24} to {Day15/95:1}.
“84 {Day15/95:19} to {Day15/96:8}
485 {Day14/71:15} to (Day14/71:21}
486 {Day20/11:17} to {Day20/11:20}.
130
A/8/130
355.
356.
357.
358.
POL00026925
POL00026925
perfect record of every step taken by the support team over the 19 years of Horizon’s
operation. It is therefore useful to know that the SSC could be relied upon to go about its
work with skill and diligence. That is the thrust of the expert evidence as much as the
factual evidence (see, in particular, the evidence of Messrs Parker and Roll).
F7. Robustness
There was, by the end of trial, a great deal of common ground between the experts as to
(1) the importance of determining whether and to what extent Horizon is robust and (2)
the answer to that important question. In short, it is possible and useful to measure
robustness, and Horizon performs well against that measurement.
The central importance of robustness to the Horizon Issues
Horizon Issue 3 asks to what extent and in what respects Horizon is robust and extremely
unlikely to be the cause of shortfalls in branches. It should not require argument to
identify the central importance of that issue to the Horizon trial and these proceedings.
The determination of Horizon Issue 3 is self-evidently central to the general likelihood
of Horizon being responsible for any given shortfall in a branch account (although the
facts of the individual case of course may tell one way or the other). Curiously, the point
appears to be controversial between the parties, although not the experts.
In their Outline Allegations and opening submissions, Cs sought to undermine the
importance of the question of robustness, placing the word robustness in “scare quotes”
and calling into question whether the concept has any objective meaning outside of
public relations exercises.
Horizon Issue 3 was agreed between the parties. It is in the Horizon Issues because it
arises on the pleadings. Post Office pleaded that Horizon was robust, and that plea was
denied.**” It is in issue, even if Cs would perhaps now prefer that it was not.
487 Generic Reply and Defence to Counterclaim, para. 37 {C3/4/21}. Cs evidently now regret that pleading,
as they have sought to argue that robustness was admitted all along: see Cs’ written opening at para. 17.1,
which is flatly untrue on the pleadings {A/1/9}.
A/8/131
359.
360.
362.
Ultimately, the importance and utility of the concept of robustness was not at issue
between the experts. The experts both say that robustness is (1) a mature area of IT
practice and learning, (2) a key concept to be used in answering the Horizon Issues and
(3) measured objectively, allowing benchmarking against an industry standard and
suitable comparators. Cs attempt to side-line robustness finds no support from the
experts.
(1) Robustness is a mature concept in IT and the subject of academic study
Cs’ pleaded case is that robustness is “more commonly used in public relations than as
an objective performance standard” “** A similar point was made in Cs’ written opening
submissions at para.17.1, where it is said that “robustness” (in quotations) “has been one
of Post Office’s ‘narrative boxes’ and a favoured term in its public relations
pronouncements” “°°
That case is not supported by the experts:
361.1 In his first report, Dr Worden expressed the view that robustness “encompasses a
large and mature area of modern IT practice”, noting that there is a “large, mature
and tested set of techniques for achieving robustness”.‘”°
361.2 Mr Coyne agreed in his oral evidence that the concept of robustness is well-
established in the IT industry, and is the subject of academic study.*?!
(2) Robustness is a key concept for answering the Horizon Issues
Dr Worden identified Horizon Issue 3 (robustness) as the most important, including
because Horizon Issues 4 and 6 could be understood as “specific subsets” of issue that
would affect the system’s degree of robustness.‘””
POL00026925
POL00026925
“88 Outline Allegations, para. 3.1 {C1/2/4}.
“8 {A/1/9}.
“°° Worden 1/48 {D3/1/11}
“1 (Day14/31:19} to {Day14/32:1}.
“2 Worden 1/48 {D3/1/1}.
AI8/132
363.
364,
365.
366.
367.
POL00026925
POL00026925
Mr Coyne accepted that the concept of robustness addresses not only the reliability of
the system but also the effectiveness of its countermeasures in preventing and addressing
system errors ‘°° — i.e. the subject matter of Horizon Issues 1, 3 and 6. He agreed that the
issues addressing robustness were central to the present trial and to laying a foundation
for later breach trials, by focusing on the general likelihood of bugs causing a lasting
shortfall in any given account.**
(3) The robustness of Horizon can be benchmarked against other comparable systems
Each of the experts has sought to compare Horizon to other systems of similar scale and
complexity. They agreed as follows in JS3: “From our experience of other computer
systems, Horizon is relatively robust”.?*
Mr Coyne accepted that it is possible to benchmark a system by reference to other
comparable systems.*’° His view that Horizon is “relatively robust” means that it is
robust “benchmarked against industry standards”.”” He added that “Horizon compares
well” with the appropriate comparators in banking and manufacturing.’’* Mr Coyne’s
evidence on this is addressed further below.
The experts’ overall conclusions on robustness
The experts are not far apart on this issue. They agree that Horizon is robust, as measured
against systems of comparable scale and complexity (such as banking systems).
Applying an appropriate industry benchmark, Horizon is a good system.
Outline of Dr Worden’s evidence on robustness
Dr Worden’s view is that Horizon is a “very robust system compared to other major
systems” he has worked on in “sectors such as banking, retail, telecoms, government and
“3 {Day14/25:2} to {Dayl4/27:1}.
44 (Day14/18:20} to {Day14/21:9}.
°° Para. 3.1 {D1/4/2}.
“6 {Day14/40:13} to {Day14/40:16}
7 Coyne 1/5.91 {D2/1/78}. This remained Mr Coyne’s view by the time of his second report: {Day14/41:23}
to (Day14/42:6}.
8 Tid,
AI8/133
368.
369.
POL00026925
POL00026925
healthcare’ °°? Dr Worden was not challenged on this assessment, and the basis for it, in
cross-examination. It is important to recall that, in addition to his quantitative analysis
(which was given prominence in cross-examination), Dr Worden also carried out a
qualitative assessment of the architecture, testing, countermeasures and support
operations for the system. That is set out in great detail in his first report and underpins
his conclusion as to robustness. A summary was provided in Post Office’s written
opening submissions at paras 181 to 204.5
Dr Worden was challenged on the issue of robustness, not by reference to anything he —
or indeed Mr Coyne — had said, but by reference to a series of relatively recent internal
Post Office reports which recorded varying views about Post Office’s IT systems,
including Horizon. None of these involved an assessment by an expert. None of them
addressed the existence, detection or fixing of bugs. The general thrust of the documents
is that Post Office’s IT systems, including but not limited to Horizon, are now old and
ready for replacement (including because lower cost solutions might now be available).
The reliance on these documents in cross-examination was unlikely to advance the
debate, for several reasons:
369.1 The Post Office documents were not drafted with the benefit of the vast amount of
work that has been carried out by the experts for this trial. If the authors considered
that Horizon was not a good system, they were wrong (although that is not even a
fair summary of what the documents say).
369.2 The experts agree that Horizon is presently robust, although it was probably
(somewhat) less robust in the periods following its initial introduction and the
introduction of Horizon Online. Mr Coyne’s clear written and oral evidence was
to the effect that he was more confident of Horizon’s robustness now than in some
earlier periods. The case put by Cs in cross-examination is not supported by either
expert.
“°° Worden 1/49 {D3/1/11}.
500 ¢4/2/65}.
AI8/134
POL00026925
POL00026925
369.3 Specifically, the experts agree that Horizon has become more, not less, robust over
time: see JS3, para. 3.1.°°' Any suggestion that it is now unreliable because it is
nearing the end of its service life is flatly contradicted by the results of the huge
amount of work conducted by the two experts, including through their extensive
access to internal Fujitsu documents that would not ordinarily be provided to Post
Office.
369.4 The documents put to Dr Worden are not ones which Mr Coyne relies on or even
refers to in his reports;
369.5 In any event, the documents over which Dr Worden's cross-examination flitted, in
many cases did not on a proper reading support the interpretation apparently being
put on them by Cs.
370. To illustrate the latter point, the documents to which Dr Worden was taken included the
following:
370.1 A Post Office Board Agenda dated 31/1/17.°°? The wider context of the document
is set out in the Executive Summary of the Technology Strategy Update*”* which
explains (under the heading “Context” that it is the whole IT estate that is being
addressed. There is a reference later to the Horizon Online platform being at the
end of its life.*** Cs want to give that statement an unjustifiably dramatic spin.
370.2 A minute of a Post Office Group Executive meeting dated 17 January 2017°°
which refers to Fujitsu seeing Post Office as a “cash cow”.**° This has no obvious
relevance to the Horizon Issues. The fact that Fujitsu has ample resource to run the
system is a good thing.
$1 ¢D1/4/2}.
52 fE/1611}.
503 5
50% £F/1611/100}.
505 ¢F/1603}.
506 £F/1603/5}.
135
Al8/135
371.
POL00026925
POL00026925
370.3 A “Thin Client” Discussion Document dated 28/11/16,” which makes various
references to the history of Horizon and its features, including an observation that
modernising the “back-end” is extremely difficult®’’ and that it is “operator
unfriendly”. "Operators" of the back-end systems are not SPMs, but are Post
Office or Fujitsu staff. To the extent these points are relevant to the Horizon Issues,
they have been addressed by the experts.
370.4A Post Office “Back Office Transformation” said to be dated 22/10/165
Paragraph 3 refers to various back office struggles and a related “prohibitive cost
of change”. Horizon is not specifically mentioned. This appears to be of no (or
only very marginal) relevance to any of the Horizon Issues. The weight put on it
is unjustifiable.
370.5 A Post Office “Back Office Strategy” document said to be dated 29/8/16.5!°
Paragraph 1 refers to the back office processes being “complex, unreliable,
expensive to maintain and not suitable for today’s business”. Horizon is not
specifically mentioned. The same comments as above apply here.
370.6 The Horizon Online Induction Training document said to be dated 7/12/09°!!
which refers to Horizon as having “Evolved rather than designed ~— a consequence
of which is a robust service but complicated to change”.*!? If anything, this
document would tend to support Post Office’s case.
It is submitted that documents of this sort take matters nowhere. They say nothing of any
real significance to the Horizon Issues. They are simply evidence of Post Office
considering its IT systems in general and its back office systems specifically. Many of
the documents focus on the risks, costs and inconvenience of change and do not say
anything about the reliability of Horizon. This is a further example of Cs’ approach of
507 (F/1586}.
508 F/1586/3}.
50° F/1557}.
510 €F/1522}
51 4F/555}.
512 4F/555/13}.
136
AI8/136
372.
373.
374,
POL00026925
POL00026925
latching onto any material that appears, however superficially, to be making any kind of
criticism of Horizon.
It is true that, in computing terms, Horizon is relatively old. It is possible that that means
that some hardware may be nearing end-of-life and doubtless that could cause cost and
inconvenience. However, the fact that software is old does not mean that it starts to suffer
from more bugs. On the contrary, the experts consider that the system has become more
robust over time, including because bugs have been detected and eradicated.
Mr Coyne’s written evidence on robustness
Post Office set out in its written opening submissions that Mr Coyne’s written evidence
on robustness was confusing and, in some respects, seemingly inconsistent: see paras
tu: .
98-99.5!* His oral evidence was much clearer.
Mr Coyne’s oral evidence on robustness
In response to cross-examination, Mr Coyne readily went much further than anything he
had been prepared to set out in his reports. There were three important respects in which
his oral evidence was strongly in favour of Horizon, departing substantially from the
impression generated by his reports:
374.1 He unpacked the high degree of reliability that was inherent in his conclusion that
Horizon was robust relative to comparable systems.
374.2 He clarified that his view was that Horizon had been robust at all times during its
service life, contrary to the impression that might have been taken from his reports.
374.3 He explained that his conclusions in favour of Horizon’s robustness were informed
by the positive impression that he had formed of Fujitsu’s working practices in
supporting the system, having obtained more evidence of this through his
extensive review of contemporaneous documents (principally, his analysis of
Peaks). This again was contrary to the impression that might have been taken from
his reports.
513 {4/2/38}.
AI8/137
375.
376.
377.
POL00026925
POL00026925
On the first of these points — the high degree of reliability inherent in a robust system —
it is important to recall that Mr Coyne benchmarked Horizon against industry standards
and comparable systems.
Mr Coyne initially did not wish to be drawn as to where he would place Horizon in terms
of a quartile of comparable systems, but his use of the term “relatively robust”
necessarily means that it is in the top half. Mr Coyne ultimately accepted the suggestion
that “it is towards the 10 end rather than the 0 end” of the spectrum of comparable
systems.°!4 To much the same effect, he said that Horizon “compares well” to such
systems, *!> and he agreed that the sense in which he used the adverb “relatively” was “in
comparison to others”, i.e. in the same way that a runner that is “relatively fast” is faster
than most people with whom he runs.*'®”
Mr Coyne was careful to select appropriate comparators:
Q. And these are systems where there are large numbers of users?
A. Yes.
Q. And a great complexity in the transactions being performed?
A. Yes.
Q. And these are systems which have measures and controls to achieve robustness?
A, That aim to achieve robustness, yes.
Q. Could you just define which sectors you are talking about as being comparable
for the purposes of your judgment?
A. So point of sale systems, something where a transaction is taking place over the
counter, is certainly comparable. Banking has elements of it, where automated
payments are being transferred to different organisations, so that is certainly
comparable as well. Stock control, things like that.
Q. And you are talking about large businesses with lots of users and lots of
complexity, yes?
A. Yes, at a similar scale to what is seen in Horizon.""
514 (Day14/44:6} to {Day14/44:8}.
515 {Day14/43:8} to (Day14/43:16}
516 {Day14/43:8} to (Day14/43:16}.
517 {Day14/44:20} to {Dayl4/45:15}.
AI8/138
POL00026925
POL00026925
378. Mr Coyne’s comparators were retail and banking systems of comparable scale and
complexity to Horizon. He agreed that businesses that operate these systems can only
afford a tiny proportion of errors:
Q. Okay, across both. But the kind of businesses you are talking about, banking,
retail, those are private sector?
A. Yes.
Q. Those sorts of systems have quite exacting requirements of robustness, don't they?
Q. No system can be perfect, everyone agrees that, but the large and complex
businesses that use these sorts of systems, they don't have much tolerance for
widespread errors in data entry, processing or storage, do they?
A. No, that is right. They certainly strive to remove that wherever possible.
Q. Given the number of transactions and the importance of handling and accounting
those transactions properly, those businesses require the overwhelming majority of
their transactions to be handled and accounted for properly, don't they?
A. They do indeed, yes.
Q. They can only afford for a tiny proportion to suffer from lasting errors, yes?
A. Yes.5'8
379. Mr Coyne clarified later that the risk tolerance would be “a fraction of a percentage”>!°
380. He accepted that, in a system with this degree of robustness, the vast majority of
transactions will work successfully, even in adverse conditions:
Q. So when you say the Horizon system is relatively robust, you are saying that in the
overwhelming majority of cases where conditions are both normal and adverse it
works reliably, yes?
A. Yes, the vast majority of all the transactions that flow through the Hor
will work successfully.
mn system
518 {Day14/56:22} to {Day14/57:18}.
519 (Day14/81:15} to {Day14/81:18}; {Day14/149:1} to {Day14/149:8}. As Dr Worden identified in his first
report (relying on a “Horizon Architecture Overview” from 2006), Post Office and Fujitsu were acutely aware
of the need for accuracy in the system, given the huge volume of transactions and small margins on those
transactions: Worden 1/227-228 {D3/1/64}.
AI8/139
381.
382.
383.
POL00026925
POL00026925
Q. And although there are occasions when it doesn't, those represent only a tiny
proportion of the work that it does, correct?
A. Yes, it will be a small fraction of the work that it does, yes.
Q. Even within that tiny proportion, a large majority of the problems thrown up are
picked up and corrected by the various systems in place that are designed to do
precisely that?
A. Certainly many of the ones that go wrong for one reason or another appear to be
picked up. There are examples where they don't appear to have been picked up and
examples which appear to have an impact.
Q. That is a very important opinion, isn't it, in the context of this case?
A. Yes.
Q. For the purposes of this trial?
A. Yes.°°
It is unfortunate that none of these important points was set out in Mr Coyne’s reports.
He allowed himself to give the impression that his conclusion as to Horizon’s robustness
was nothing like as significant as it in fact is. That is certainly what Cs took from his
evidence, as was apparent from their opening submissions and the attempt to side-line
the experts’ agreed position on robustness.
On the second point set out above — that Horizon was relatively robust throughout its
service life - Mr Coyne gave clear oral evidence.
Mr Coyne clarified that he did not think there was any period in which Horizon was not
robust, simply that there were periods of time — following Horizon’s original
introduction and then the introduction of Horizon Online — when it was less robust than
it was at other times. Mr Coyne’s view on the overall robustness of Horizon over time
was as follows:
Q. So both Horizon Online and Legacy Horizon have been robust for most of their
lives but there were initial periods where they were less robust?
A. Yes.
520 {Day14/58:17} to {Dayl4/59:15}.
140
A/8/140
384,
385.
386.
387.
388.
POL00026925
POL00026925
Q. In relation to those initial periods -- and do correct me if I'm wrong -- you are not
saying they definitely weren't robust during that period, you are simply saying they
were materially less robust, is that right?
A. Yes, *!
On the third point identified above — the impression formed by Mr Coyne of Fujitsu’s
working practices — the oral evidence here was striking as much for its clarity and for its
absence from Mr Coyne’s reports and contributions to the Joint Statements. This has
been addressed in detail at paras 340 et seq above.
It is most unsatisfactory that these matters were not explained more clearly in Mr
Coyne’s reports, the more so since Mr Coyne provided detailed corrections and
amendments to his second report, not only picking up typographical errors but also
making some substantive changes. He also had the opportunity to clarify his position in
the Joint Statements (by agreeing with more of Dr Worden’s views).
F8. The importance of countermeasures and the concept of a lasting financial
impact
A reader of the expert reports might have gained the impression that there was a
fundamental disagreement between Dr Worden and Mr Coyne as to the importance of
countermeasures in addressing the Horizon Issues. The Court may recall a somewhat
sterile question as to which of the acronyms used by Dr Worden to refer to various
countermeasures were industry standard. There has been, at least in Cs’ submissions, a
concerted attempt to downplay the importance of countermeasures.
The real position, as appeared from Mr Coyne’s oral evidence, was that each of the
experts in fact considered countermeasures to be essential to any meaningful
consideration of Horizon Issues 1, 3, 4 and 6 (the last two of which focus specifically on
various risks of error and the measures and controls that address those risks and errors).
It is notable that Dr Worden was not challenged substantially on the importance of
countermeasures to his analysis.
21 {Day14/127:8} to {Day14/127:23}.
141
Al8/141
POL00026925
POL00026925
Common ground in the Joint Statements
389. There are several important agreed statements on countermeasures in JS3 (which post-
dates the expert reports): see paras 3.11-3.13°??, 3.18, 3.20, 3.22°% and 6.1.5%4
Common ground that emerged at trial - countermeasures are central to the analysis
390. Mr Coyne agreed that a consideration of countermeasures was essential to the issues:
Q. Very well, I understand. But you will then at least agree with me this far: that in
asking you to give your opinion on the issues we have read, what you're being asked
to do is to advise the court on the extent to which problems had happened -- or
problems were likely to happen in relation to any given situation, the extent to which
those problems were likely to have been caught by countermeasures in that situation,
and the overall question of the extent to which Horizon is robust and extremely likely
to be the cause of a shortfall in that given situation. Would you agree with that?
A, I don't believe countermeasures were asked in the questions, but broadly with what
you say, yes
Q. When I say countermeasures, you can take me as saying controls and measures if
you want —
A. Okay.
Q. Subject to that correction, you would agree that that was the essential question
being raised in the four issues that I read to you?
A. I believe they were the questions that were asked and I believe that in providing
my reports I seek to address those. I have answered those.>
Q. Mr Coyne, didn't one of the Horizon Issues specifically ask you to consider
whether controls and measures in Horizon reduced certain problems to an extremely
low level of risk?
A. It certainly -- yes, it did.
Q. And I think we have established, it has taken an hour to do this, I think we have
already established that when making an overall judgment of robustness you both
consider how the system operates and then consider how the countermeasures
designed into the system also operate?
52 (D1/4/4}.
53 ¢D1/4/5}.
524 (1D1/4/9}. Mr Coyne’s entry at para. 6.2 is impossible to reconcile with his oral evidence as to the quality
of the SSC’s investigations.
55 {Day14/16:24} to {Dayl4/17:20}.
Al8/142
POL00026925
POL00026925
A. Yes.
Q. With a view to coming to an overall decision as to the robustness of the system?
A. Yes.
Q. Thank you. So in forming a judgment about robustness it is necessary to forma
judgment also, isn't it, about how the countermeasures, what Dr Worden calls the
robustness countermeasures, how those countermeasures were designed and
operated in practice, yes?
A._Yes.*”° (emphasis added)
391. Mr Coyne agreed that assessing robustness must involve an assessment of the
effectiveness of countermeasures:
MR DE GARR ROBINSON: So during that period, 2012 to 2019, your judgment is,
and you feel capable of making the judgment, that Horizon is and has been relatively
robust, correct?
A. Yes.
Q. Now that judgment includes a judgment not only about the electronic systems and
so on, it includes a judgment about everything including the countermeasure
processes surrounding Horizon, correct?
A. Yes. There has to be a number of caveats about that and I explained before that I
don't really know what happens within Post Office to correct defects if Fujitsu has
spotted them. So I can't comment really on what that process would be.
Q. But in relation to the -- nonetheless you have sufficient information to allow you
to form an overall judgment as to robustness and your judgment is that during that
period the system itself and the countermeasures around it were relatively robust?
A. Yes.
Q. The reason why I ask you that -- and just to be clear, the same is true of the period
Legacy Horizon, shall we say from 2001 to 2010, would that be a fair period?
A. Yes. Perhaps one year after that.
Q. Okay, 2002 to 2010. During that period your view is that Horizon was relatively
robust?
A. Yes.
526 {Day 4/48:8} to {Day14/49:3}.
143,
AI8/143
392.
393.
394.
POL00026925
POL00026925
Q. And that included the countermeasures surrounding it and supporting it?
A. Yes.?7
As an example of countermeasures in action, in relation to Dalmellington, Mr Coyne
accepted that countermeasures had worked to prevent any lasting discrepancy for 108
instances out of 112.5°8 He also accepted that in relation to the four remaining instances
referred to in the documents, the two substantial ones were not in fact instances of the
Dalmellington bug at all*”?
and in relation to the other two (where the losses were
respectively £1 and Ip) the chances that those losses were not made good were “very
small”>° A bug that escaped detection for a long period and struck many times was
prevented, by effective countermeasures, from causing even a single lasting impact on
branch accounts.
The Court will recall that countermeasures are hardly considered in Coyne I — despite
the fact that, as Mr Coyne ultimately agreed, they are central to a proper understanding
of robustness. Mr Coyne’s explanation of why he had failed to do so was unconvincing
and turned on his idea that he was being asked whether it was “possible” that
countermeasures might sometimes fail:
A, I don't see that utilising many pages of text to talk about the various aspects of
Horizon's countermeasures, which may well be similar to how Dr Worden set them
out, they may well operate in much different ways, we don't really know, but we can
observe that they are likely to operate in those ways. I do not see there is a great deal
of value when it is a fact, and what I believe the court was asking in the Horizon
Issues was to address if it is possible that they have failed._So it is the failure side of
it.>*! (emphasis added)
That reasoning is difficult to follow. It is and always has been common ground that
countermeasures sometimes fail, but considering the nature, extent and efficacy of
countermeasures was explicitly required by Horizon Issue 6. It was also, as Mr Coyne
accepted at trial, essential to answering Horizon Issues I and 3. Mr Coyne again avoided
answering the Horizon Issues and applied a standard of perfection (reflecting his
527 (Day14/134:18} to {Day14/135:22}
528 {Day17/149:18} to {Day14/150:5}.
8° {Dayl7/160:15} to {Day17/160:21}.
58 {Day17/161:13}.
581 {Day14/137:6} to {Day14/137:15}.
144
AI8/144
395.
396.
397.
398.
mistaken idea that he was being asked to consider whether the risk of error had been
“reduced as far as possible’).
There are many other examples of Mr Coyne failing to have any appropriate regard to
countermeasures, creating the false impression that a problem would have resulted in
false shortfalls or other adverse effects, when in fact it was overwhelmingly likely (or
even inevitable) that it would have been addressed by an automatic or manual
countermeasure. For example, in his first report Mr Coyne referred to some 62 KELs
which he said were of particular interest. In fact, as Dr Worden pointed out in Worden 1
nearly all of them either had no effects on branch accounts or were addressed by ordinary
countermeasures and so caused no lasting financial impact. Very few of these KELs
made their way onto the list of 29 bugs in JS2 — only a handful in addition to the known
bugs and Dalmellington. The remainder were quietly dropped by Mr Coyne. He does not
explain anywhere either that he has done that, or why, but it can only be because he could
no longer defend his reliance on them. That is unsatisfactory.
Fujitsu’s role in identifying and addressing bugs
This has been addressed in detail above. The experts agree that the quality of Fujitsu’s
support operation is high and has contributed to the robustness of the system.
The cost/ benefit allegation
Cs contend that harmful bugs may have been detected but not fixed. On the face of it,
that would be a strange approach for Fujitsu to adopt, given that fixing bugs is in own
interests (in avoiding having to deal with repeated occurrences) and the interests of its
customer, Post Office. The experts agree that “any competent IT support operation is
grateful to its users, when they draw its attention to any problem which can be fixed, to
reduce the fitture costs of support”: JS2, para. 9.1.5°?
To deal with this point, Cs have latched onto the idea that bugs might be left in the system
because it would cost too much to remove them. That is contrary to the witness evidence
in these proceedings — Mr Roll gave evidence of Fujitsu’s dedication and professionalism
in identifying and fixing bugs, and Mr Parker gave unchallenged evidence as to Fujitsu’s
POL00026925
POL00026925
582 {D1/2/39}.
145
Al8/145
399.
argument. Mr Coyne stated repeatedly
POL00026925
POL00026925
practice in this regard. Post Office is not aware of any Fujitsu guidance document that
states that SSC staff should (or even are permitted to) fail to address bugs in the system
because it would be too expensive to do so.
Cs nonetheless took some support from Mr Coyne’s reports in raising the cost/benefit
533 that whether to remedy bugs was something
which was decided on the basis of a cost/ benefit analysis. Mr Coyne sought to defend
this contention in his oral evidence:
Q. The point you are making you are applying to the fixing of bugs in Horizon. You
are saying, correct me if I'm wrong, that the support process provided by the SSC
allowed relevant bugs to remain in the system in order to save money?
A. Yes, and there is certainly a reference within a document to that view being
taken.>**
400. Mr Coyne accepted that it was unfair of him to have said that this happened “often”:
401.
Q. Let's see how you do support it in your first report, shall we? If we go to page
{D2/1/97} at paragraph 5.161, you say: "Whilst both Horizon and Horizon Online
contain a number of measures and controls designed to check system integrity, these
mechanisms have been shown to have failed. This is a point agreed upon in the joint
statement. It has been identified that known issues/bugs were often deferred and dealt
with on a cost/benefit basis." Here you have actually increased the scope and, if I may
say so, ambition of the claim. You are now saying it happened "ofien". Is that your
considered view of the documents you have seen?
A, Could I just turn back, please, to the paragraph before?
Q. Of course. {D2/1/97} (Pause)
A. It might be unfair for me to say "ofien".>*>
He explained that by “often”, he would mean 10 or 15 times (across 20 years), and he
was not aware of that many instances:
Q. ...Because it is fair to say you are not saying it happened once or twice over 20
years, you are saying it said it happened "often", and over a 20-year period "often"
would usually mean dozens if not, in the context of Horizon, hundreds of times. Would
you agree with that?
583 See e.g. Coyne I/ para 3.15 {D2/1/27}; Coyne 1/ para 5.161 {D2/1/97}; Coyne 1/ para 5.199 {D2/1/108}.
58 (Day14/147:21} to {Day14/148:2}.
58 ¢Day14/151:12} to {Day14/152:4}.
146
AI8/146
402.
403.
404.
405.
POL00026925
POL00026925
A. No, I would take "ofien" to be ten or 15 times.>*°
In any event, even aside from this point about frequency, the documents relied on by Mr
Coyne in support of this point in his reports could not sustain the weight he put on them.
A key one was not even relevant at all (as he accepted).
As has been addressed briefly above, one of the main documents on which Mr Coyne
relied in support of the allegation that bugs were dealt with on a cost/ benefit basis was
the Post Office Risk & Compliance Committee minute dated 18 September 2013.°*’ The
document considered a risk which was described in more detail in a paper for the
committee.°** The issue was whether the current system of monitoring risk which
involved a report being manually produced and reviewed, should be replaced by an
automatic system.
Mr Coyne quarrelled with whether the decision ultimately taken — which was to continue
with the present manual system and increase monitoring on whether it was adequate —
was the right one, but that was not the point he was making in his report. The statement
in the report was that Fujitsu decided not to fix bugs (or at least deferred doing so) on
cost/benefit grounds.*” Ultimately, after much evasion, Mr Coyne accepted that the
document had nothing to do with that:
Q. Mr Coyne, let me remind you of the sentence that you say this document supports
that's contained in paragraph 5.161 of your expert report: "It has been identified that
known issues/bugs were often deferred and dealt with on a cost benefit basis.” This
proposal here and the decision that was made from this proposal is not evidence of
any known issue or bug being deferred and dealt with in a cost benefit basis, is it?
A. No. This isn't to do with the deferment of a bug, no.”
Having lost the support of that document, Mr Coyne insisted that it was appropriate to
rely on the Reconciliation and Incident Management Joint Working Document dated 18
March 2013.5"! This document too, once read with care, does not support the claim made
586 {Day14/152:8} to {Day14/152:13}.
587 {F/1140/3}.
58 {E/L127/1}.
58° Coyne 1/5.161 {D2/1/97}.
5 {Day14/163:2} to {Day14/163:11}.
Sl E/1697/1}.
147
AI8/147
406.
407.
408.
POL00026925
POL00026925
by Mr Coyne. He has latched onto a sentence in the document saying that not all “system
faults” are corrected because this is “generally done on a contractual and/or cost benefit
basis” 5”
He failed to note that a “system fault”, as defined in the document,
encompasses a huge range of potential problems that might well not be capable of being
addressed totally, either as a matter of contract or for cost reasons. It is important to note
also that this reference appears in a document concerning reconciliation management,
rather than the SSC’s activities in fixing bugs. It is not a good place to look for evidence
of the SSC’s attitude to fixing bugs that it has detected. It is a very weak and indirect
piece of evidence on which to rely.
Mr Coyne accepted that at the time he made the strong claims that he did in Coyne 1,
these two documents were possibly the only ones he had:
Q. Is it likely that when you made this statement the only evidence you had to support
it was the Risk and Compliance committee meeting minutes and the document we
have just come to? I have had your commentary on whether you think either of those
documents is good evidence. I'm simply asking you whether it is likely that those were
the only two pieces of evidence you had at the time you made this report?
A. It is certainly possible that is the case, but I would have to look at when the PEAKs
arrived 8
Mr Coyne was taken to Peak PC0120937°** which is the only Peak of which Post Office
was aware*> that might have been read as suggesting a cost/benefit approach was
adopted. There is an entry in the Peak where cost is mentioned (on 15 June 2005 at
11.28):
Weighing up the cost and risk of an attempted fix against the fact that this has only
been reported once, I do_not believe that we should make a code fix. If further
incidents of this problem are reported we can review this decision. Gary has raised
a KEL, so returning for closure as "Published Known Error". (emphasis added)
This reference to cost was sufficient for Mr Coyne to consider that this was evidence for
his cost/ benefit point. What the Peak, considered as a whole, indicates is the following:
2 £F/1697/10}
$8 (Day14/179:23} to {Day14/180:7}.
SM (FQTI}.
58 Mr Green QC asked that Mr Coyne also be shown {F/275.1} (a Peak relating to testing, not the live
system), Mr Coyne did not seck to rely on it in relation to his cost/benefit point: {Day14/193:17} to
{Day14/193:24}.
148
AI8/148
409,
410.
408.1 Fujitsu was concerned primarily with risk and on that basis decided to rely on a
KEL to address any further instances of a very rare problem, rather than attempting
acode fix: see the entry at the bottom of p.3, recording that it had not been possible
to recreate the bug on the test rig and that a code fix would be “potentially risky
because this code is used in a number of different circumstances and could prove
difficult to code”. *46
408.2 When asked about this, Mr Coyne agreed that the debate in the Peak revolved
largely around the risk of making a code fix.5"7
408.3 The experts agree that introducing code fixes itself generates a risk of error,**8 so
a balancing of the risks either way is always required — a code fix could introduce
more risk of error than it removes.
408.4 The problem appeared to occur extremely infrequently and would be apparent to
the SPM when it occurred and thereafter (see the final entry on p. 4). The KEL
would be updated so that appropriate advice would be given to resolve any further
instances of the problem.
This Peak does not support the notion that even the particular decision to which ¢ relates
was taken principally on the basis of cost (rather than risk), let alone that decisions were
“ofien” made on a cost/ benefit basis. The related KEL*” equally provides no such
support: it too refers to “the frequency of the problem and the risks involved in making
the necessary changes” (emphasis added).
It became clear that the evidence referred to by Mr Coyne in his reports could not support
the strong claim that he had made in those reports. In the face of this, Mr Coyne changed
tack and stated that he had seen further evidence — evidence not cited in his reports
(including because he had discovered it since those reports). No explanation was given
as to why this further evidence, if important, had not been shared with Dr Worden. There
is also a point that Mr Coyne seemed to overlook, which is that evidence discovered only
POL00026925
POL00026925
6 (F/271/3}.
S47 Dayl4/185:25} to {Day14/186:3}
58 See, e.g,, Mr Coyne’s evidence on this at {Day15/185:6} to {Day15/185:9}.
4° {F/276}.
149
Al8/149
411.
412.
after the reports were drafted cannot justify his decision to make the strong claim that he
did without that evidence.
Mr Coyne was invited to turn up relevant Peaks overnight. He identified seven Peaks.
There was no time to cross-examine on this material. Post Office sets out its submissions
on the Peaks in Appendix 1. In short, the Peaks generally demonstrate an appropriate
reluctance to attempt difficult and/or risky code fixes in circumstances where the
problem has already been resolved through an effective “work-around” (including in one
case a less invasive code fix) or otherwise can confidently be expected not to re-occur.
Mr Coyne’s contention in his report, even with the benefit of evidence that he discovered
only later, finds very weak support in the documents. The Peaks do not undermine the
robustness of the system but show an appropriate degree of caution when it comes to
making changes to code.
The role of TCs in robustness
It is common ground between the experts that Horizon is made more robust by the
existence of a process for correcting errors in branches through TCs. Specifically:
412.1 Dr Worden expressed the view that TCs are an important part of the robustness
countermeasures built into Horizon, particularly for correcting user errors. He
noted, for example, that very many TCs (approximately 20,000 per year) were
issued to correct for mistakes in remittances out from branches (e.g. miscounting
the cash in a remittance pouch and/or mis-declaring the amount on the system).°*°
412.2 Mr Coyne accepted in cross-examination the obvious point that a TC is more likely
to correct an error in the accounts than it is to introduce one.**!
POL00026925
POL00026925
55 {Day20/8:23} to {Day20/10:1}. Dr Worden was referring to Table 9.3 at {D3/1/208}.
581 {Day 4/80:5} to {Day14/80:17}.
150
AI8/150
413.
414.
415.
416.
There are agreements in relation to TCs at JS2, paras 5.1, 5.3, 5.4 and 5.5°* and JS2,
paras 15.1, 15.2, 15.3 and 15.4.5* It is common ground that the decision to issue a TC
takes place outside Horizon and using other systems: see JS2, para. 14.6.°54
It is also common ground between the experts that an individual TC might be incorrect
and, if accepted, might introduce error into the branch accounts: see JS2, para. 15.2.5
Dr Worden attempted to estimate an approximate scale of the impact of errors introduced
in this way in section 9.6 of his first report.°°° He noted, for example, that the average
branch received only about one TC per month.>°’ Some of course receive more than that,
and some fewer. But mistaken TCs, taken as a whole, are a very unlikely candidate for
accounting for a substantial proportion of disputed shortfalls - TCs are a tiny minority
of entries in branch accounts, and they are generally of modest value (although of course
not always).
There are no known bugs that affected the generation, content or issue of TCs. (Item 21
on the bug table in JS2 includes Peaks relating to a problem where the system would
freeze before a TC could be accepted in the branch, but that had no effect on the branch
account — the branch was able to rollover without accepting the TC and could then
resolve it later).
Cs often refer to dramatically large TCs and suggest that they might have been wrong.
That is of course possible, although any huge and mistaken TC would likely generate a
very substantial complaint from the SPM and, inevitably, an investigation. Cs drew
attention in their submissions to two TCs for £810,000 issued to a particular branch, but
those TCs were both correctly issued to address a mis-keying error: see Mr Smith’s
POL00026925
POL00026925
52 (D1/4/8}.
583 {1/2/43}.
584 (D1 /2/42}.
588 (D1/2/43}.
586 (D3/1/205}.
557 Worden 1/930 {D3/1/206}.
1S]
A/8/151
417.
418.
POL00026925
POL00026925
witness statement at paras 31-33.°°* It could hardly be imagined that the SPM would not
have challenged an erroneous TC of that size.
Mr Coyne accepted in cross-examination that the reconciliation processes that lead to
many TCs take place outside Horizon, relying on Post Office’s back-office accounting
systems and the accounting systems of client organisations.**? Other TCs are initiated by
SPMs themselves, often through reporting that they have made a mistake in the branch
that requires correction (mis-keying errors, for example, as was confirmed by Cs’
witnesses at this trial).
Mr Coyne made clear at trial that he had not intended to suggest that SPMs who were
identified by Fujitsu as having suffered an impact on their branch accounts would not
have been made whole. He had intended to say that it was likely that the discrepancies,
once identified, had been addressed:
Q. Just to be absolutely clear, you are not suggesting that you think that when a bug
was identified and discrepancies were spotted in accounts, you are not saying you think
the accountancy discrepancies were not dealt with, you are not saying that, are you?
A. No, I'm not saying that. What I'm saying is they were identified typically by Fujitsu,
and I don't know what the process was, and Fujitsu will say that they don't know because
it appears in all the documents, they don't know what Post Office does to correct that.
That is a missing part of the evidence. So I'm not saying I think Post Office do nothing
about it, I'm simply saying I don't know what that process was.
Q. Right. You are not suggesting that you think, on the basis of documents you have
seen, that it is likely that Post Office do nothing about it? You are not saying that, are
you?
A. No, I'm saying the opposite. It is likely that they do something about it...°°
588 {£2/9/7}, Mr Smith was taken to this in cross-examination but not challenged on its accuracy:
{Day6/189:1} to (Day6/191:25}.
59 (Day14/73:19} to {Day14/74:19}.
56 {Day14/54:17} to {Day14/55:10}.
AI8/152
419.
420.
421.
422.
Mr Coyne also agreed that, once Fujitsu identified a bug, it was relatively easy for them
to identify the affected branches, by working out the “profile of the issue” and then
looking across the “historic files”, even where this meant going back years.*°!
Taking these two points together — it being relatively easy for Fujitsu to identify affected
branches and it being likely Post Office acted to correct the errors caused by the bugs —
Mr Coyne’s evidence here was dramatically at odds with the impression given by his
reports. The impression that those reports generated was that SPMs might well have been
left with discrepancies caused by bugs that had been identified and addressed. That
impression was misleading and would not fairly reflect what Mr Coyne in fact thought.
The detail of Post Office’s reconciliation processes and how they feed into the generation
of TCs is outside the scope of the Horizon Issues. It was excluded because expanding
from expert issues in relation to the Horizon system itself into the vast reconciliation
processes that take place between Post Office and its clients would have made a
relatively short trial impossible: see Post Office’s opening submissions at para. 114,
referring to the content of the relevant Horizon Issues and the clear indications given by
the Court at the outset in relation to the kind of issues that should be addressed in the
Horizon trial.°° Cs proposed issues that would have extended the Horizon Issues to
include reconciliation, but those were opposed and ultimately not agreed: see, e.g., Post
Office’s objection to Cs’ proposed Issues I and 3 in the document at {C8 4/2/25}. It is
clear from the expert reports that the automatic and manual reconciliation processes
(which they have identified only in very broad terms) are enormous and complex.
Following from this trial, it should be relatively straightforward, in individual cases, to
address the contention that any specific TC was erroneous and was responsible for a
given shortfall. The issue is vast at a generic level (not least because there were, in many
years, around one hundred thousand TCs)°°*, whereas a dispute in an individual case will
always be quite tightly confined, given that almost all branches receive fairly few TCs.
POL00026925
POL00026925
{C8.4/2/25}
$61 {Day17/154:17} to {Day17/154:23}. The answer was given in relation to the Dalmellington bug, but the
process described is generic.
562 4/2/44}.
$63 JS2, para.15.1 {D1/2/43}.
153
AI8/153
423.
424.
425.
426.
POL00026925
POL00026925
F9. The experts’ approaches to assessing the extent of bugs and their financial
impacts
The specific bugs identified by the experts are addressed in the section after this. The
purpose of this section is to identify the importance of the questions of extent raised by
the Horizon Issues and the differing approaches adopted by those experts to the issues.
The importance of the questions of extent to Horizon Issues 1, 3, 4 and 6
The Horizon Issues that address the reliability of the Horizon system, the effectiveness
of countermeasures and the impacts of bugs all ask questions of “extent”: see Horizon
Issues 1, 3, 4 and 6.°°. That is not an accident. It is common ground on the generic
pleadings that Horizon is imperfect, has suffered from bugs and has had adverse impacts
on branch accounts; it is also common ground that it contains measures aimed at reducing
the incidence of such impacts. The dispute between the parties revolves around questions
of degree or extent.
It is common ground between the experts that identifying the likely number of bugs and,
crucially, the likely financial impacts of those bugs on branch accounts is central to any
consideration of robustness: see JS3, para. 3.15 — “It is difficult to measure the extent of
robustness of Horizon, apart from how it might limit the extent of impact on branch
accounts as in Issue 1”.°® Horizon Issues I and 3 are closely interrelated. And they, in
turn, relate closely to Horizon Issues 4 and 6, which concern the risk of various forms of
error and the measures in the system that work to prevent such errors arising and/or
prevent them having effects on branch accounts.
In relation to all these issues, the focus of concern is financial impacts on branch
accounts. That is at the core of the dispute on the generic pleadings: see, for example,
paras 22-24 of the AGPOC, contending that various sources of errors in Horizon resulted
in “discrepancies or alleged shortfalls attributed” to Cs by Post Office and that bugs in
Horizon “had the potential to produce apparent shortfalls” °° See also paras 40-41 of
Sh C/I}.
568 (D1 /4/4}.
586 {C3/1/8}.
154
AI8/154
427.
428.
429.
POL00026925
POL00026925
the Generic Reply,*°”
contending that Cs suffered losses resulting from bugs in Horizon
and putting Post Office to proof of the countermeasures that it pleaded and the
effectiveness of those countermeasures. This is reflected in the drafting of the issues.
Mr Coyne confirmed that he had read the relevant paragraphs of the generic pleadings
when deciding how to interpret the Horizon Issues*®, and that these Horizon Issues arose
in a context where the contention was that bugs had caused “false shortfalls” in branch
accounts.°°” Mr Coyne also agreed that Horizon Issues 1, 3 and 6 required a focus on the
extent to which it was likely that bugs caused shortfalls in any given set of accounts.>””
The experts’ differing approaches to questions of extent
Dr Worden sought to answer the questions of extent principally by taking samples of
KELs and secking to extrapolate from those samples into the total population of KELs,
asking what total number of bugs with lasting impacts might exist in that total
population. He then used that number to estimate an upper limit on the total likely
financial impact of that total population of bugs, taking his estimate of financial impact
from the bugs for which there were available figures.
There is one important point to note about that approach. It is premised on the total
population of KELs being of largely unknown content, such that sampling must be used
to estimate (with a large margin of error) how many bugs might be identified in the
unreviewed KELs. That premise was correct at the time when Dr Worden carried out his
calculations. But the premise no longer holds: Mr Coyne and his team had, by the time
of trial, reviewed the substantial majority of the KELs (around 6,000 to 7,000 out of
roughly 9,500).*”! That review disclosed, in Mr Coyne’s opinion, only 29 bugs with
lasting impacts (which figure he revised down at trial to 22), which is vastly less than
anything suggested by Dr Worden’s calculations.
567 (€3/4/22}.
568 {Dayl4/14:16} to {Dayl4/14:23}.
$9 (Dayl4/11:14} to (Day14/11:18}.
57 {Day14/14:24} to {Dayl4/17:20}. See also {Day14/20:20} to {Dayl4/21:1}.
57 {Day15/122:25} to {Day15/123:8}.
155
Al8/155
430.
431.
432.
433.
434.
435.
POL00026925
POL00026925
It is also important to note that Dr Worden also formed strongly positive views of
Horizon’s robustness and reliability, and the effectiveness of its countermeasures, on a
qualitative basis, as discussed above. His views are not limited to the numbers.
In his reports, Mr Coyne avoided any systematic and objective assessment of extent,
simply giving examples of problems and effectively inviting the Court, without any
basis, to extrapolate from a necessarily biased sample into the total population and
speculate that there might be many more bugs. At trial, Mr Coyne conceded that his
extensive review had identified very few bugs and that there were in fact unlikely to be
many more such bugs in the KELs that he had not reviewed.
By avoiding the questions of extent, Mr Coyne often limited his conclusions to points
that were in fact common ground from the outset of these proceedings — for example, he
concluded that it was likely that bugs had caused shortfalls in branch accounts,*”? but
that was accepted in pre-action correspondence and in the generic pleadings.
A failure to engage in an objective way with questions of extent is particularly
problematic where the subject matter of the dispute is, on any sensible view, of an
enormous scale: the Horizon Issues relate to a system that has processed 47 billion
transactions and through which 3 million sets of monthly branch accounts have been
prepared and submitted. It is actively unhelpful to fail to have regard to that scale in
choosing how to present the importance and incidence of a particular issue or problem.
Dr Worden’s numerical approach to assessing extent
From the outset, Dr Worden set out to give objective, number-based opinions on the
“extent” issues, having regard to the scale point set out above. Given the impossibility
of reviewing all KELs by the time of his first report, Dr Worden decided to work from
samples and adopt standard statistical approaches to estimate what he could not measure
directly.
Mr Coyne did not himself try to do this, and he criticised Dr Worden in his second report
for doing so. Mr Coyne said, for example, that risk analysis should not be used
5? See, ¢.g., JS3, para. 3.25 {D1/4/6}.
156
AI8/156
436.
437.
438.
439,
440.
POL00026925
POL00026925
retrospectively (as is implicit in trying to identify the total number of branch-affecting
bugs that have existed in Horizon over its life).
At trial, however, Mr Coyne frankly accepted that assessing risk often involves taking
information on past performance and forming views and estimates off the back of that
573
information,*”> which is precisely what Dr Worden did when extrapolating from his
samples to the total population. The criticism was meritless.
There is an even simpler point. If an expert is aware that he can only review a certain
proportion of a total universe of documents of a given kind, the obvious thing to do is to
take an unbiased sample and use that as a basis to estimate what the total population
contains. It is strange not to have done that. This applies with particular force to Mr
Coyne given that he and his team were in fact able to review a large proportion of the
total number of KELs, allowing them to form confident views about the population as a
whole (which Mr Coyne proved well able to do “on his feet” when asked at trial). Dr
Worden did the obvious thing, and there has been no convincing explanation as to why
Mr Coyne did not.
The calculations in section 8.5 of Worden 1
In section 8.5 of Worden 1, Dr Worden carried out an analysis which demonstrated that
with a scaling factor of 0.37, there would need to have been 50,000 bugs with a total
impact similar to that of the Suspense Account in order for there to be a 1 in 10 chance
of such a bug causing a shortfall of £1,000 in a claimant’s branch in any given month.
In Worden 2, Dr Worden changed the scaling factor to 0.45 and so this became 40,000
374
bugs.
If no scaling factor for small claimant branches is applied at all (i.c. in the table in para
637 of Worden 1, Label D becomes 1), then: Label E becomes 3,091,680; Label H
becomes 193,230 (3,091 ,680/16); and Label J becomes 19,323 (say 19,000).
In other words:
573 (Day14/38:11} to (Day14/38:22}.
574 ¢1)3/6/30} para 117.
187
AI8/157
441.
POL00026925
POL00026925
440.1 For a bug such as the Suspense Account bug which occurs 16 times with a mean
financial impact of £1,000, in order for there to be a I in 10 chance that a shortfall
of £1,000 would be caused in a claimant’s branch in any given month, there would
need to be 19,000 similar bugs (i.e. 304,000 incidents: 19,000 x 16));
440.2 If the mean financial impact of such a bug were £500, there would have to be
38,000 similar bugs (608,000 incidents);
440.3 If the mean financial impact of such a bug were £100, there would have to be
190,000 similar bugs (3,040,000 incidents).
Other than in relation to the assumption about likely impact on claimant branches, Mr
Coyne accepted that Dr Worden’s reasoning was correct:
Q. So if you take a bug which has occurred 16 times over the lifetime of Horizon?
A. Yes.
Q. With a mean financial impact of £1,000 and that's quite a significant bug
compared with most of the bugs you found, would you agree?
A, It is certainly significant in its impact, yes.
Q. It is in the top five, yes?
A. Quite possibly.
Q. And then you select a claimant branch a month at random, the chance of that bug
occurring in that branch in that month is 16 in 8 million, correct?
A. If bugs affect branches equally, yes.>7>
442. This point was explored with Mr Coyne and he retreated behind the possibility (for which
no basis is set out in his reports) that the branches operated by the claimants might have
been peculiarly exposed to bugs. He eventually accepted that it was likely that bugs
would not have targeted those branches especially:
Q. Mr Coyne, the overwhelming likelihood is that there aren't more than 40 bugs of
the sort that you have identified in Horizon, correct?
A. Yes, but what you have got to remember is each of those bugs can have an impact
on multiple branch accounts. It is not just one bug, one impact.
575 {Day15/179:16} to {Day15/180:3}.
158
Al8/158
POL00026925
POL00026925
Q. So are you suggesting that some of those bugs have massive impact? Remember,
we are not talking about bugs that affect only the claimant branches. We are talking
about bugs that are in operation over the entire Post Office network.
A. Yes.
Q. So on any view those bug impacts -- the total bug impacts is going to be very, very
much greater than the specific impact that might affect the claimants, yes?
A. Yes.
Q. So the total impact of the kind of bug you are suggesting, to justify the £19 million
claim just made by the claimants alone, the total impact of the kind of bug you are
hypothesising would have to be vast, wouldn't it, in the tens and tens of millions?
A. If we are using the law of averages to show that it has impacted everybody equally
rather than just impacting the claimants’ branches, yes.
Q. Isee. You are now suggesting that there might be bugs which have only affected
the claimants and haven't affected the wider Post Office network, is that what you are
claiming?
A. I'm saying it is entirely possible because we know from the bugs that have actually
happened that they have only impacted a small number of branches.
Q. Is it really entirely possible though, Mr Coyne? You are hypothesising a bug
which has had many, many, many effects in order to justify the suggestion that it has
generated a large number of losses. That's what we are talking about?
A. Yes.
Q. Those bugs occur at branches because of factors that you are not prepared yet to
identify, yes?
A. I haven't been given the information to enable me to identify.
Q. You don't have a specific bug in mind which has a particular feature which means
that it only affects certain kinds of branches. You are not telling me that. You are
saying that it is possible there might be such a bug?
A. Yes.
Q. With this theoretical bug you are suggesting it is quite possible that it could affect
just the claimants' branches and not affect the branches in the wider Post Office
network. Is that really your view? Do you really think that is a likely outcome?
A, It may well have affected other branches in the branch network.
Q. That's my point, Mr Coyne. It is all very well to say "I don't know, it is a really
difficult judgment to make", but there are certain features which are just matters of
159
AI8/159
443.
444,
commonsense. What I'm suggesting to you is that, bearing in mind the claimants
represent such a small fraction of the total Post Office network over a period of 20
years, it stands to reason as night follows day that if there are bugs which justify their
claims those bugs would also have incurred losses in the wider Post Office network.
Would you accept that?
A. Yes.
Q. Would you accept, therefore, that the wider losses that would have been caused in
the Post Office network would be substantially greater than the £19 million
A. That's likely, yes.°"6
It needs to be borne in mind here that when Dr Worden talks about the need for 40,000
bugs, he is talking about 40,000 bugs each of which has multiple instances/impacts on
branches. His calculations therefore capture the point that bugs typically have more than
one impact (although it is notable that many of the bugs in the bug table affected very
few branches).
Mr Coyne agreed with many of the building blocks to Dr Worden’s views on the question
of scale:
444.1 Mr Coyne agreed that there are some 3 million branch accounts to be
considered.°”
444.2 He agreed that branches that perform more transactions are likely to be hit by a
bug on more occasions than a branch doing fewer transactions*”* and accordingly
with Dr Worden’s approach to calculating what he terms a “scaling factor” to
account for differences in the number of transactions performed in branches.°””
444.3 He agreed that he had seen a “substantial number of Peaks where SPMs...phoned
in with problems about relatively small sums of money” °° This supports the
inferences drawn by Dr Worden to the effect that even fairly small discrepancies,
if unexplained, would be questioned in a non-negligible proportion of cases.
576 {Day15/185:13} to {Day15/188:6}.
577 {Day15/168:5} to {Day15/168:7} and JS2, para. 1.2 {D1/2/27}.
57% (Day15/172:13} to {Day15/172:20}.
579 {Day15/174:13} to {Day15/174:18}.
58 {Day17/18:4} to {Day17/18:13}.
160
POL00026925
POL00026925
A/8/160
445.
446.
447.
444.4 He agreed that the helpline operators would pass calls on to the SSC when they
reach a point when they do not think they can help further, irrespective of whether
they considered there was a software error or not.°*!
444.5 Mr Coyne also did not argue with the mathematics of Dr Worden’s calculation.***
The mathematics is simple arithmetic.
It follows that Mr Coyne does not argue with the majority of the building blocks set out
by Dr Worden in the calculations he sets out in sections 8.5 and 8.7 of Worden 1.
Dr Worden had necessarily to make various assumptions about the likelihood of
individual SPMs phoning in to raise a concern about a perceived discrepancy, namely
that ifa discrepancy is £1000 or more, the likelihood of it being reported to the helpline
is 80%; if £300, 30% of SPMs would report it; if £100, 10% would report; and if £10 or
less, fewer than 1%.>* It is common ground between the experts that many bugs are in
fact identified without the need for any report from an SPM, as the SSC is notified of
potential problems through automatic reporting.
The suggestion that appeared to be made to Dr Worden in cross-examination was that
since he is not a behavioural economist, he is not competent to make assumptions as to
SPMs reporting problems to the helpline.**4 Any such criticism is misconceived:
447.1 The assumptions made by Dr Worden are primarily a matter of common sense,
allowing for a large margin of error by erring on the side of assuming a low
reporting rate.
447.2 They are based on the available evidence — albeit that such evidence is limited —
and Dr Worden makes it clear in the report that he is only able to draw “weak
inferences”
581 {Day17/46:7} to {Day17/46:24}.
582 {Dayl5/177:18} to {Day15/177:21}.
583 Worden I para 415.1 {D3/1/107}.
5 {Day18/19:23} to {Day18/20:9}.
S85 Worden 1 para 415 {D3/1/107}.
161
POL00026925
POL00026925
A/8/161
POL00026925
POL00026925
447.3 There is a strong basis on which to conclude that SPMs at least sometimes phone
in to question or challenge even fairly small discrepancies or other problems— Mr
Coyne accepted that (see above), and there was evidence from the claimant-
specific witnesses was also to the same effect.
447.4 The key point is that if there were multiple instances of bugs, with varying amounts
of impact, the overwhelming likelihood is that the bug would eventually be
reported by at least someone. Dr Worden’s numerical assumptions in this regard
are consistent with a great many SPMs simply choosing not to report unexplained
discrepancies, which is not supported by the evidence (but which tends to favour
Cs’ case). As noted above, the evidence shows that once bugs come to the attention
of Fujitsu, their system of recording via KELs and Peaks was good.
448. It is to be noted that the evidence of Cs’ witnesses at the trial would tend to support the
inferences which Dr Worden has made about SPMs reporting problems:
448.1 Mr Tank accepted that if he had a discrepancy for more than a few pounds which
he felt called for some further explanation, he would report it by contacting the
Helpline:
Q. Allright, let me put it in a different way then. If you felt it called for some further
explanation then you would contact the Helpline?
A. Yes.
Q. And certainly if it was more than a few pounds you would do that?
A, Yes. 8°
448.2 Similarly, Mr Patny Snr agreed that if he discovered a small (but non-negligible)
and unexplained discrepancy, he would investigate it and report it to Post Office:
Q. And when you do the balancing, presumably it's not uncommon that there's
some sort of discrepancy between the cash that Horizon expects you to have, for
example, and the cash you have actually got?
A. Well, it happens sometimes.
586 {Day2/127:25} to {Day2/128:6}.
AI8/162
449.
450.
451.
POL00026925
POL00026925
Q. Sure. And I'm just interested in how you reacted if you found that there was a
discrepancy. If it was just a few pence or a few pounds would you just make up
that difference from your own pocket?
A. Well, under normal circumstances, yes, so if it is a few pounds or few pence,
yes.
Q. And if it was more than that, say £40, £50, £60 or above, would you investigate
it?
A. Yes.
Q. And raise it with Post Office?
A, Yes?
Mr Coyne did quarrel with the assumption that the claimant branches would have been
no more susceptible than other branches to experiencing shortfalls as a result of bugs.
He accepted that he was not in fact aware of any characteristic which marked out the
claimant branches in this way, so as to make them more vulnerable to bugs.°*
If there were some special factor about the claimant branches that made them susceptible
to bugs (for example, the types of transactions performed or the level of experience of
the SPMs), that could and should have been identified long before now. It is a point that
Cs would have been quick to plead in support of their case. Post Office too would have
been eager to investigate any such suggestion, given that it might help to explain why a
generally good system could have let the Cs down as badly as they contend.
Mr Coyne was candid enough to accept that, while he had reservations about the
approach taken by Dr Worden, it was the kind of approach that he himself would have
taken if he had to in order to make a business decision:
Q. Let's move on. If you are genuinely saying it is impossible to arrive at any
judgment on these matters, if you had to make a business decision about risk and this
was the only information available to you, you wouldn't just sit there and say you
couldn't make a decision, would you? You would perform a judgment on the best
information you have, yes?
A. Yes, I agree with that, if a decision had to be made and there was no other
information available, then I would use the best information that was available to me.
$87 {Day2/166:13} to {Day2/167:3}.
588 (Day15/169:2} to {Day15/169:8}.
163
Al8/163
452.
453.
454.
455.
456.
POL00026925
POL00026925
Q. And you would build in a margin of error, wouldn't you, to account for the
possibility that there may be unknowns out there that could throw your figures out?
A. Yes.
Q. In the way that Dr Worden has done, correct?
A. Yes5°
Mr Coyne did not explain why he had not carried out such an exercise to help answer
the Horizon Issues, even if he would have wished to caveat it as being only approximate
(which Dr Worden himself quite properly did).
The Penny Black objection to Dr Worden's numerical approach
The Court will recall the Penny Black example which was raised with Dr Worden during
cross-examination. The suggestion was that whereas the chances of a single person in a
given room of 50 guests being called Penny Black were low, and the chances of there
being two people with that name in the same room commensurately lower, if invitations
had only been sent out to people called Penny Black, the chances of not just one but all
the people in the room being called Penny Black would be 100%.
It was pointed that Cs are a self-selecting group because each of them contends that he
or she or it (for a corporate claimant) has suffered a loss by reason of a bug in Horizon.
It was suggested that it is therefore inappropriate to identify them as a group with an
average exposure to bugs.°””
Cs are of course a self-selected group. And it is of course not irrelevant to the issues in
these proceedings more generally than Cs contend that their branches suffered shortfalls
caused by bugs. But the attack on Dr Worden’s numbers was nonetheless misconceived.
First, the argument assumes the very thing which this litigation is designed to establish
one way or another i.e. whether, and if so how, Cs were affected by bugs in Horizon. Just
as Post Office cannot (and does not) assume that the explanation for every shortfall must
be user error or dishonesty, so neither Cs (nor the experts) can properly assume that
Horizon must be the cause of the shortfalls. It is the same thing to engage in estimating
58 {Day 5/177:22} to {Day15/178:12}.
5% {Day18/170:18} to {Day18/170:25}.
164
Al8/164
457.
458.
459.
460.
POL00026925
POL00026925
the increased likelihood of bugs having affected the Claimant group (i.c. a retrospective
view, taking into account the claim of a bug), unless that likelihood depends on some
objective characteristic of the Claimant branches that made them more exposed to bugs
at the time (i.e. a prospective assessment of the risk faced by the branch).
This is clear from the terms of the analogy itself. The women called Penny Black were
identified and invited based on an objective and easily verifiable fact (their name). The
Court has not sent out any invitations based on its assessment of the validity of the
claims. It has not identified whether or not people are called Penny Black. The analogy
begs the main issue in the case.
Secondly, the fact that Dr Worden has taken the Cs as a group is largely irrelevant to his
analysis. He could have taken any group of SPMs of a similar size whose branches were
of a comparable size, carried out the same exercise and obtained the same result. He did
not take into account that Cs contend that they suffered shortfalls caused by bugs, any
more than he took into account that Post Office contends the opposite. He did not try to
answer the ultimate question for breach trials. He did not assess the specific claims made
and how credible they were; he did not purport to consider questions of reliability of
witness testimony; he did not seek to usurp the Court’s role.
What Dr Worden’s analysis in fact sheds light on is the risk of bugs affecting any group
(of the size and objective characteristics of the claimant group) being affected by bugs
in Horizon. The numbers show that risk was small relative to the amounts now claimed
in these proceedings. The analysis applies equally to the entirety of Post Office’s
branches: the analysis can be used to show an estimate of the maximum amount of total
losses to all branches for the entire life of Horizon.
Indeed, this total figure is shown on Dr Worden’s financial impact spreadsheet.*”! If the
correction of 6,000 to 13,800 is made in Label M (D14) (to reflect Dr Worden’s updated
figure for the mean financial impact of KEL: see para 1.31 of JR2*°? and the Annex to
591 (D3/8}.
52 1/2/32}.
165
Al8/165
461.
462.
463.
464.
465.
POL00026925
POL00026925
JR2),°" the figure in D23 becomes £9,277,311. This is the maximum financial impact of
bugs on all Post Office branches, corrected for KEL sampling, creation and retention.
That figure has nothing to do with the claimant group and the fact that it is self-selecting.
Cs ignored this point entirely in cross-examination. It provides an important guide as to
the maximum impact of bugs in Horizon.
On the same basis, the figure in D24 becomes £78,019. This is described by Dr Worden
in his report as the maximum financial impact on all claimant branches, corrected for
KEL sampling etc, but it could perhaps more accurately be described as the figure for
any group of 561 branches of similar size to the claimant-branches (over the duration of
the relevant period). It is a simple scaling process and not any attempt to hone in on the
claimants and the specific circumstances and problems of their cases.
It is important for Post Office to make clear that this does not mean that the claimants’
claims between them cannot exceed c. £78,019. Post Office does not invite the Court to
make any such finding. What it demonstrates is simply that it is unlikely (although not
impossible) that bugs in Horizon caused lasting financial impacts of the scale alleged in
these proceedings. It is open to the Cs, individually or collectively, to demonstrate that
in fact bugs did cause the losses they complain of, but that will need to be a matter for
evidence and argument in specific cases. It is far from capable of being inferred from the
generic evidence.
The purpose of this trial is to assess the general reliability of Horizon and not to identify
whether it went wrong in relation to particular Cs. This trial has never been about proving
whether Cs’ specific claims are necessarily good or bad. The Horizon Issues are, as Mr
Coyne accepted, about assessing how generally likely it is that Horizon has been the
cause of any given shortfall (in the absence of other evidence about shortfall) — it is about
the starting point, not the end point. That is the issue on which Dr Worden’s numerical
analysis sheds light.
Another way of making the same point is to say that Dr Worden’s analysis treats Cs as a
population of operators of branches that had certain objective characteristics at the
53 (D1/3}.
166
AI8/166
466.
467.
468.
relevant time (e.g. volumes of transactions), and not as individuals that now, in 2019,
assert claims about what happened in those branches.*°*
Dr Worden’s analysis therefore does assume that each Post Office branch with a given
volume of transactions would be equally exposed to the risk of a bug affecting any given
branch account. As to this:
466.1 There is no evidence to undermine that assumption. Dr Worden himself gave
careful consideration to whether there might have been objective factors
increasing the risk for some branches, but he could not identify any of any
substantial impact.
466.2 Cs know their branches and their cases. The burden is Cs to demonstrate that there
really is some factor or feature which means that they were more exposed to a
higher risk than average. (Cs can also of course simply seek to prove that they are
the unlucky ones that were affected by unlikely events. None of that is ruled out
by the fact that the overall impact of bugs on the network was likely very low.)
The Court cannot properly be invited to reach conclusions as to the evidential
significance, from a statistical perspective, of Cs asserting that there were branch-
affecting shortfalls in their branches (which were not resolved by countermeasures at the
time). That would be to assume matters that fall to be determined at later trials. If Cs
had wished to argue that there was some statistical approach that would, without begging
the questions for later trials, take into account that Cs are a self-selected group, they
could have led expert evidence to that effect. It is inappropriate to now raise the spectre
of an analysis that Cs have refused to carry out.
Adjusting the figures
As Dr Worden made clear, if the Court disagrees with any of the assumptions which he
has used in his calculations, the figures can be adjusted appropriately.°°* The Court is
invited to make any appropriate amendments to Dr Worden's Financial Impact
POL00026925
POL00026925
54 Day19/85:10} to {Day19/85:18}
598 {Day18/161:24} to {Day18/162:7}.
167
AI8/167
469.
POL00026925
POL00026925
spreadsheet.*° It will quickly be seen that there is an immense amount of tolerance in
the figures, i.e. that substantial adjustments can be made without materially altering the
sense of scale given by the bottom line figure.
In the Financial Impact Spreadsheet:
469.1 In every case a factor, which is given a Label (see column B), is also given a value.
Some are facts; some are assumptions; and some are calculations (which apply
simple formulae). The main room for error or disagreement is in the assumptions.
469.2 In every case Column C “Central Estimate” sets out Dr Worden’s actual view
based on his sampling; and Column D “Conservative Estimate” sets out the
approach he has taken, in nearly every case by adjusting his actual view so that it
favours Cs’ case.
469.3 Labels A, B, C are factual matters which are not controversial. (It was pointed out
to Dr Worden in cross-examination that in arriving at the figure for Label A of
13,560 he had used the average for the years 1999 to 2018.5?” That is correct and
is what is stated in Column A of Dr Worden's Revised Financial Impact.*?> The
statement in Worden 1, para 619,°” which states that the period 2000 to 2018 has
been taken, is therefore incorrect).
469.4 Label D is a claimant branch size divided by a typical branch size. Dr Worden was
criticised for some aspects of this calculation. He volunteered a further explanation
of the approach he had taken at the beginning of the second day of his cross-
examination and in particular that the admitted error of using 561 as the
denominator in his calculation in Worden I had been corrected for Worden 2. But
even if some aspects of his calculation can or should be amended, the effect on the
overall conclusion is not very significant — see below.
596 (D3/8}.
597 (Day20/58:2} to {Day20/58:10}.
598 3/8}
599 (D3/1/148}.
© Beginning at {Day19/3:11}.
168
Al8/168
470.
469.5 Label E: this is what Dr Worden calls the scaling factor — it is just a calculation.
469.6 Labels F-G: again these are factual matters and should not be controversial.
469.7 Label H: another calculation.
469.8 Label L: This is an assumption. Dr Worden calculated that there might be a
maximum of 100 KELs with an impact on branch accounts, but he assumes 200.
(The Court will recall Mr Coyne’s evidence that, based on his much more
extensive review of KELs, he does not think there are more than 40.)
469.9 Label M: as noted above Dr Worden has calculated this figure to be £13,800 —
which is still very considerably higher than any evidence suggests when the
likelihood of TCs having been issued is taken into account.
469.10 Label N: another calculation.
469.11 Labels T-U: these are a series of assumptions about whether a bug is reported
(T); and a KEL is created given a bug is reported (U). Dr Worden has changed the
assumptions so that it significantly favours Cs’ case.
469.12 Label V: another calculation.
469.13 Label W: an assumption about whether a KEL is not archived. Again the
assumption has been changed by Dr Worden so that it significantly favours Cs.
469.14 Label X-Z (and E1-E2): further calculations. (Note that the formula for E2 should
read “E2-L/X”).
The Court is invited to carry out the following changes to the "Conservative Estimate"
column, simply by way of example:
470.1 Change Label D to “1”: this means that Cs’ branches are treated as the same size
as the average Post Office branch. Doing this removes any concern over the scaling
factor (which was 0.5).
470.2 Change Label M to “25,000”: this is a far higher value than the evidence can
credibly support. It assumes that the mean financial impact of a bug in a KEL is
169
POL00026925
POL00026925
Al8/169
471.
472.
£25,000 when it is highly unlikely that a bug of such seriousness would not have
been detected (let alone many such bugs).
470.3 Change Label T to “0.5”: this assumes that only half of the bugs with financial
impact are ever reported to SSC, contrary to the evidence and common sense.
470.4 Change Label U to “0.3”: this assumes that only 30% of the bugs actually reported
lead to a KEL being created, again contrary to the evidence (including from Mr
Coyne).
470.5 Change Label W to “0.5”: this assumes that only 50% of such KELs were archived,
again contrary to the evidence and common sense.
Making these changes, which are implausibly generous to the case against Horizon, it
can be seen that:
471.1 The changes to T and U mean that V becomes 0.15, i.e. the calculation proceeds
on the assumption that there is only a 15% chance of a KEL being created for a
give bug.
471.2 The maximum summed financial impact of all bugs on all Post Office branches
(Label Y) is £66,666,667.
471.3 The maximum summed financial impact on claimant branches (or indeed on any
group of 561 average branches since the scaling factor due to size of claimant
branches has been removed) is £1,121,289 (Label Y), a little less than 6% of the
sums claimed in these proceedings as false shortfalls (Label 26).
471.4 In order to generate these figures it has been assumed that there are 2,667 bugs
(Label 28). The evidence at trial is not remotely consistent with such a high
number.
This exercise is, on one view, wholly artificial since it presents a gross exaggeration of
the true position by assuming a set of facts for which there is no support at all. But what
it does emphasise is: (1) the degree of tolerance in Dr Worden’s figures and (2) the sheer
170
POL00026925
POL00026925
A/8/170
473.
474,
475.
POL00026925
POL00026925
scale of error that would be required in the system (and the laxity of Fujitsu’s practices)
that would be required for Horizon to be as unreliable as Cs wish to argue.
It was suggested to Dr Worden that if his “conservative estimate” of 672 bugs was taken
and it was assumed that each bug affected 48 branches, then the total number of incidents
(i.e. the number of separate effects on branch accounts) would be 32,256 — which
amounts to an average of about one such incident per claimant.®!
It is submitted that this does not assist in answering the questions of extent:
474.1 First and foremost, the figures put forward by Dr Worden — in both his central and
conservative estimate — have been overtaken by events. It is clear that both experts.
have been looking for bugs and that Mr Coyne’s searches in particular have been
extensive and likely to have identified the great majority of bugs. Mr Coyne’s own
view, based on all his work, was that the total population of KELs might contain
as many as 40 bugs with lasting impacts, and even that must now be an
overestimate given that Mr Coyne moved away from 29 and down to 22 such bugs
in the bug table.
474.2 On Mr Coyne’s estimate of 40 bugs, assuming for present purposes each bug to
have 48 incidents, there would be a total of 1,920 incidents across all Post Office
branches. Assuming Cs branches not to have some special characteristic, one
would expect about 32 incidents for a group the size of Cs (1,920 x
52,000/3,091,680).
474.3 Secondly, the average number of incidents suggested to Dr Worden in cross-
examination (i.c. 48) includes the incidents for the Dalmellington bug, all of which
were inevitably picked up by countermeasures. A more accurate estimate would
be to take Receipts and Payments (say, 62); Callendar Square (30); and Suspense
Account (14), the average of which is 35.
Dr Worden set out to provide a numerical basis on which to answer the questions of
extent. His calculations are of material assistance in getting to grips with the scale of
1 (Day20/61:12} to {Day20/64:16}.
2 {Day20/60:8} to {Day20/60:14}.
171
A/8/171
476.
477.
478.
479,
POL00026925
POL00026925
error that would be implied by Cs’ case and how it compares to the scale of error
identified by the experts. The calculations do not purport to answer the claims that Cs
make, but they provide a useful indication of relative scale.
Mr Coyne’s approach in his reports — avoiding “extent” questions
Mr Coyne agreed with the formulation of the ultimate purpose of the issues he was asked
to consider:
Q. I'm not suggesting that you don't have to look at whether bugs have arisen during
the life of Horizon. What I'm suggesting to you is the practical purpose of these issues,
the question you are being asked, is to what extent is it likely that those bugs are going
to cause shortfalls in any given case?
A. Yes, Lagree with that.°?
Mr Coyne agreed that, to answer the Horizon Issues in a meaningful way, it is necessary
to determine the extent to which it is likely or unlikely (in general) for bugs to cause
shortfalls for which SPMs have been held liable.* In order to do that, some sort of
metric, or yardstick, is required. Mr Coyne claimed to have used a metric:
A. The metric I have adopted is to find whether there was an actual bug, error or
defect and see whether it had an impact. I don't believe there is any other metric.°°
That is no metric at all. It involves nothing more than a series of factual observations
about individual bugs and their impacts, divorced from any sense of scale or the
likelihood of other similar or different bugs arising and having an impact. Mr Coyne’s
approach in his reports provided no meaningful assistance on questions of extent.
It is obviously impossible for any individual to examine, still less assimilate, every
document in the case which relates to the operation of Horizon over the last 20 years.°*
Mr Coyne accepted the principle of looking at a limited portion of the evidence i.e. a
sample of it, in order to arrive at overall conclusions:
3 {Day14/20:20} to {Dayl4/21:1}.
4 (Day15/98:15} to {Day15/98:20}.
5 {Day15/98:24} to {Day15/99:1}.
6 {Day15/100:4} to {Day15/100:9}.
AI8/172
POL00026925
POL00026925
Q. To what extent is this likely or unlikely to happen in branch accounts in Horizon?
To what extent was the risk faced by a user in Horizon high or low? Do you accept
that those are the sort of issues that are raised in this trial?
A. Yes.
Q. And to address extent, you can look at a limited portion of the evidence that you
can sensibly review. You can assess its nature and scale and on the basis of those
assessments you can arrive at overall conclusions that are generally useful, can't
you?
A. Yes.
480. An analogy which was suggested during Mr Coyne's cross-examination was an exit poll
481.
at an election. From a small amount of definite information (i.e. the votes cast by a small
percentage of the electorate) judgments can be made about the overall outcome. There
will be a margin of error in that overall judgment and that needs to be accounted for.
Despite Mr Coyne’s disavowal, this is precisely what he has done in his reports:
Q. Well, isn't it actually what you are trying to do in your evidence as well, Mr Coyne?
Aren't you saying: I found a number of bugs, and aren't you suggesting that an
inference should be drawn that there could be a great number of other bugs that you
haven't found yet?
A. Yes, but it is from the basis of actually finding bugs and trying to identify how
many branches may be impacted by those bugs, errors and defects.
Q. What I'm suggesting to you is that when you find certain hits in your sample,
because any sample is necessarily limited, your ability to be able to say that the court
should scale up and should infer that there are likely to be a certain number of other
hits, or an uncertain -- a certain scale of other hits, that is dependent upon the quality
of the sample that you have chosen in a particular -- whether it is an unbiased sample,
yes?
y
A. Yes, but in my report I haven't suggested any scaling up from particular bugs,
errors and defects. I have talked about specific bugs, errors and defects and how
many branches they are recorded to have impacted. There's no scaling applied to
that.
482. This answer misses the key point: inviting the Court to infer from the existence of some
bugs in the Peaks and KELs reviewed by Mr Coyne that there are many more bugs in
the Peaks and KELs that had not been reviewed does involve asking the Court to scale
7 (Day15/100:22} to {Day15/101:8}.
8 {Day15/104:7} to {Day15/105:2}.
173,
AI8/173
483.
484.
485.
(or adjust) upwards from the observed bugs. The fact that the process is impressionistic,
rather than being based on any valid statistical approach, does not change this. It simply
makes the process less likely to be reliable and accurate.
On the basis of any hand-picked example, like those repeatedly cited by Mr Coyne, one
can only infer that some event may have been “likely” to have occurred once during the
lifetime of Horizon — i.e. the one instance represented by the document itself. This
implies that the likelihood of that event impacting the accounts of a randomly selected
branch account would be around one in three million. Using hand-picked examples,
rather than a representative sample, does not allow broader inferences to be drawn about
extent and risk.
Mr Coyne’s attempts to defend his approach were often incoherent. In paragraph 3.3 of
Coyne 1°? he said:
The sheer volume of Known Error Logs and reconciliation reports confirm the wide-
ranging extent of the impact of such bugs/errors/defects. This evidence demonstrates
that such bugs/errors/defects would undermine the reliability of the Horizon system
to accurately process and record transactions.
If that statement is looked at closely, it is plainly inaccurate. It can only have been made
to create an impression of Horizon which was unfair and misleading. Mr Coyne’s
confusing attempt to justify the statement is unconvincing:
Q. Then you say this in 3.3: "The sheer volume of known error logs and reconciliation
reports confirm the wide-ranging extent of the impact of such bugs ... This evidence
demonstrates that such bugs ... would undermine the reliability of the Horizon system
to accurately process and record transactions." That seems quite an ominous
statement to make given the propositions that you have just agreed with me a minute
and a half ago, Mr Coyne. Do you not see a tension?
A. Yes, I think I do actually, At that point in time we had a document set that consisted
primarily of known error logs. I think the PEAKs had only just been disclosed a few
days
Q. Weeks?
A. Sorry?
Q. Would it not be weeks?
POL00026925
POL00026925
9 ¢D2/1/25}.
174
Al8/174
486.
487.
POL00026925
POL00026925
A. I don't know exactly how many.
Q. I'm sorry, I interrupted you. Please carry on.
A. Perhaps if we establish what the date was. I think we do mention it in the reports.
Q. Yes. Please carry on. I interrupted you and I shouldn't have.
A. So what we are saying there is the full picture was yet to be revealed and that it
may well undermine the reliability of Horizon.
Q. So are you suggesting, Mr Coyne, that when you produced your first statement
you were doubtful about whether it was robust?
A, Mm.
Q. And it is when you produced your second statement, when you had more
opportunity to look at the PEAKs and so on, that you formed the impression that it
was robust after all?
A. I had a different concern when it came to create my supplemental report because
there was the discovery of far more defects than it was originally said existed. When
putting together my first report I thought the analysis had already been done to
establish that there were only three defects. I discovered a fourth defect but then
shortly afterwards discovered many more. It was also the case that I had seen a
number of reports about improvement of the processes that was in place. But then it
transpires that when we saw some of the witness statements that those processes,
those improvements in processes, hadn't been adopted. So there was a number of
other factors that were brought in place that confirmed that there was large elements
of unreliability. That doesn't take away my overriding statement that the Horizon
system is relatively robust.
None of that has anything to do with justifying the suggestion that the “sheer number”
of KELs (or even less coherently, reconciliation reports) undermines the robustness of
Horizon. As Mr Coyne knows and knew when he drafted his first report, the vast majority
of KELs do not disclose bugs with impacts on branch accounts, let alone bugs with any
lasting impacts. It is the number of KELs disclosing such impacts that is important,
irrespective of how many other KELs there may be (which is likely to reflect Fujitsu’s
degree of assiduousness in documenting even minor issues).
The approach in Mr Coyne’s reports was aimed at generating the impression that the
bugs that had been identified were the tip of the iceberg and that the Court should infer
that Horizon is riddled with branch account affecting bugs. In fact, Mr Coyne had, after
61 (Day14/60:19} to {Day14/62:20}.
175
Al8/175
POL00026925
POL00026925
extensive and sophisticated searches, identified every single bug he possibly could: he
did not satisfy himself with looking at the surface of the ocean; he looked also into the
depths. This exchange, arising out of the original drafting of paragraph 3.105 of Coyne
2,°'! demonstrates the attempted sleight of hand on Mr Coyne's part:
O. You say: "The PEAKs analysed below are a small portion of the PEAKs I have
identified as causing financial discrepancy in branch accounts outside of those bugs
acknowledged by Post Office. It should be noted there are potentially thousands more
PEAKs that illustrate financial discrepancy arising in branch accounts, this is only a
small selected sample from keyword searched PEAKs."
A. Yes.
Q. Now let's take this in stages. You have changed the wording of the first sentence
and I will go to that change but I want to ask you about what the original version
means first. What you are claiming there is that you have identified a large number
of PEAKs recording bugs which cause branch shortfalls but you have only mentioned
a small portion of them in your report.
A. Yes.
Q. That wasn't true, was it?
A. No. By the conclusion of this report there was a substantial amount that was
looked at.
Q. In fact what you had done is you had identified every single bug that you could
and included it in this report, hadn't you?
A. Within the time available. I mean it is probably quite possible that there could
well be more but we have certainly had a good search.
Q. So what you say in the first sentence wasn't true, was it?
A. No, my concern with the way that it read is that -- I was saying that I had only
analysed a small portion of the ones that had caused financial discrepancy.
Q. What you are saying there -- what the words literally mean, Mr Coyne, is that
you've analysed below a small portion of a larger group of PEAKs that you have
identified as causing financial discrepancy?
A. Yes.
Q. And in fact that wasn't the case. What you did in your report was you included
every PEAK you could, every PEAK that you were aware of as causing financial
discrepancy, didn't you?
611 2/4/43}.
176
AI8/176
488.
489,
490.
491.
492.
A. Yes.6?
Following a query raised by Post Office’s solicitors (asking about the further Peaks), Mr
Coyne amended paragraph 3.105 to the following:
The—PEAKs!I have analysed belew—are a small portienproportion of the
PEAKsPEAKS, from that analysis, I have identified the following as causing financial
diserepanexdiscrepancies in branch accounts outside of those bugs acknowledged by
Post Office. It should be noted there are potentially thousands more PEAKs that
illustrate financial discrepancy arising in branch accounts, this is only a small
selected sample from keyword searched PEAKs.
The difference of a few words radically changes the sense of the paragraph. In the
original report, the contention was that the bugs listed in the report were a_small
proportion of the bugs that Mr Coyne had identified; in the revised version of the report,
Mr Coyne confirms that these are the only bugs that he had identified, but he adds that
there may be many more instances of such bugs having occurred. The important change
was made only after WBD wrote to ask about the further Peaks that had been found.
Mr Coyne accepted that the sample he is referring to in paragraph 3.105 is a biased
sample from which it is not possible to scale up.°'* As noted above, Mr Coyne’s actual
view was that there might be “dozens more’ bugs (in addition to the list of 29) and
that it seemed reasonable to supposed that the total number of bugs with effects on branch
to be found in KELs is likely to be less than 40.°'%
After all the intelligent searching (which improves iteratively as time goes on),°!® the
chances of there being a great number of bugs identified in KELs and Peaks but which
have slipped through the search net is very small.
Further, Mr Coyne accepted that, even if the link from KEL to Peak was not as good as
that from Peak to KEL, intelligent searching could make that link:
A. Yes. There is a better quality of link from PEAKs to KELs. There's not always
that link KEL back to PEAK.
POL00026925
POL00026925
82 (Day15/106:1} to {Day15/107:17}.
S13 {Day15/112:15} to {Day15/113:4}.
S14 (Day 5/114:6}.
S18 {Day15/123:25} to {Day15/124:9}.
S16 {Day15/116:5} to {Dayl5/116:18}.
177
Al8/177
POL00026925
POL00026925
Q. But this is where the beauty of intelligent searching comes in, isn't it? Because
you can intelligently search through the body of PEAKs, having identified all the
relevant KELs, and there may be a number of them, there may be one KEL and
perhaps two or three others that are also relevant, you search for all the PEAKs which
refer to those KELs, don't you?
A. Yes.
Q. And by that means you are going to get actually quite a good sense of -- you are
going to get a good hit rate. You are going to find most, probably more than most, of
the PEAKs considering problems which those KELs address, yes?
A. It is entirely possible to do that, yes.
Q. I'm not asking you whether it is possible, I'm suggesting that it is likely that if you
undertake that search you will actually find all the PEAKs that are relevant -- that
exist that are relevant to the problem addressed in the KEL?
A. Yes.
Q. So you have identified 29 bugs, you do your searches for all the PEAKs, and by
that means you can identify all the PEAKs which record manifestations of the bug.
It's likely, I'm going to use the word "likely". I'm not suggesting it can be entirely
comprehensive, Mr Coyne. So can we take it as read that obviously there are going to
be gaps at the margin, aren't there?
A. Yes. A PEAK is typically created when the bug, error or defect gets to SSC within
Fujitsu. So there may be others that don't get there, but once they get to third line
support the PEAK is created, so yes.
Q. So by this means you were in a good position both to identify bugs that have been
detected in the system —
A. Yes.
Q. -- quite reliably, so with a fair degree of confidence that there won't be that many
more bugs in the system?
A. As long as they hit the search terms that I have used, yes.
Q. Remember we are talking about the KELs now. the starting point for the process
that I have described is the KELs.
A. Yes.
Q. And you've physically reviewed those KELs, haven't you?
A. Yes.
Q. So it is not as if you need intelligent search terms to find the right ones, you have
actually looked at them, haven't you?
178
Al8/178
POL00026925
POL00026925
A. Yes."
493. Mr Coyne would appear at times to obscure the difference between the possibility of
bugs, and the likely extent of them. He was taken to the relevant paragraphs of the
pleadings which made it clear that Post Office has always accepted that Horizon
contained bugs:°'*
Q. They made it clear, didn’t they, that the critical issue was not whether it is possible
or likely for bugs to have the potential to cause false shortfalls in branch accounts
over the life of Horizon, that was in fact admitted. The critical issues were the extent
to which it was and is likely that bugs cause false shortfalls in any given set of
accounts, Do you accept that?
A. I don't really see the difference between the two statements that were made there.
The question was whether it was possible or likely and the extent to which it was
possible or likely.°!”
494. Mr Coyne accepted the basic principle about the probability of an individual branch
being affected by a bug:
Q. Yes. My suggestion to you, Mr Coyne, is that in circumstances where you have
got a bug that affects 60 branches?
A. Yes.
Q. And let's say it affects 60 branches on one occasion each, sometimes it will be
more, but let's assume it is on one occasion each. If you look at the entire corpus of
monthly branch accounts, which is 3 million over 20 years, then of the 3 million
branches that will have been affected -- or that could have been affected, I should say,
60 will have been affected.
A. Yes.
Q. Which means it has a 1 in 5 million chance of hitting any given branch that you
are looking at at any
A. On any given month. That may well be right, yes.
Q. Okay. Thank you.®°
817 {Dayl5/125:17} to {Day15/127:16}.
618 GDCC paras 53-55 {C3/3/22}
519 {Day14/14:24} to {Dayl4/15:9}.
© {Day15/131:20} to {Day15/132:10}.
179
Al8/179
495.
496.
497.
Mr Coyne’s approach in his reports — inviting inferences of many or major
problems with no proper basis in the evidence to which he referred
Mr Coyne’s invitations to the Court to make wide-ranging inference were not spelt out
clearly in his reports. Instead, he peppered his report with words like “often” which give
the impression that what is being presented was simply one example of something that
he had seen repeatedly. On closer scrutiny, it frequently became clear that (1) the
evidence cited in support of the proposition being made was no such evidence at all or
was highly suspect evidence and (2) the use of terms such as “ofien” had no basis at all.
For example, in para 5.64 of Coyne 1,°°! Mr Coyne stated:
Thave noted that hardware replacement often seemed to be a “fix” of last resort where
no other explanation could be given, and therefore there is certainly a possibility that
hardware was at fault. (emphasis added).
622
The KEL cited in support of that proposition®* simply states that the problem that had
been identified was due either to the SPM “typing ahead of themselves” or a hardware
fault. When challenged on this, Mr Coyne chose not to answer the question put about the
specific document and instead referred to other documents and suggested that five
instances (over 20 years) would be enough to justify a claim that something happened
“often” (rather than sometimes):
Q. How can that justify the claim that you make here that hardware replacements are
often a fix of last resort?
A. Well, there's a number of other examples. There is the phantom transactions
example where hardware was changed because it was -- they were trying to work out
whether it was environmental issues or not that were causing erroneous transactions.
There are a number of examples. There is only one cited here but there are a number
of examples throughout the report.
Q. Are we talking about five examples or are we talking about 100 examples?
A. There will certainly be five examples that
Q. I see. So there "often" means something in the region of five, does it?
POL00026925
POL00026925
1 ¢D2/1/71}.
2 4F/178}.
180
A/8/180
POL00026925
POL00026925
A. Yes.
7,624
498. In para 3.13 of Coyne Mr Coyne stated:
I have seen PEAK records that are closed despite support not being able to diagnose
a root cause whilst acknowledging that there clearly is some form of error occurring
within the Horizon system.
499. No evidence was cited in support of this sweeping allegation. Post Office’s solicitors
sought further information in relation to it and the answer came in Freeths’ letter dated
22 February 2019° at para 1. Nine Peaks were cited of which seven were dated between
1999 and 2001. Mr Coyne accepted that he should have explained more:
Q. Of the nine, seven of them occurred between 1999 and 2001.
A. Right.
Q. One is from 2012 and one is from 2017.
A. Right.
Q. So the truth is that over the past 17 years this has happened, or there are PEAKs
showing this as having happened twice, yes?
A._Yes, okay.
Q. Bearing in mind you are making a claim about these things ofien happening, could
I suggest to you it might have been helpful and balanced for you to have indicated
that that was the position?
A. Yes.
Q. That most of these occasions occurred during the very early years of the original
Horizon. Do you accept that?
A. Yes, it would have been helpful to include that.©® (emphasis added)
500. In para 5.108 of Coyne 2,” Mr Coyne states:
At paragraphs 251 to 257 of his report, Dr Worden refers to the concept of “User
Error Correction” enabling the facility of correcting many software errors. It should
©3 {Day15/139:4} to {Day15/139:18}.
8 ¢D2/4.1/14}.
8 (C5/36}.
6 {Day15/143:5} to {Day15/143:22}.
7 ¢D)2/4.1/156}.
181
A/8/181
501.
POL00026925
POL00026925
be noted that this would not apply to any bugs /errors and defects unbeknownst to
Fujitsu or the Subposmaster. It is evident from the PEAK analysis that often bugs lay
undetected for weeks, months or years.
Again, particulars were requested of this allegation. The response referred to Coyne 2
generally but in particular paras 3.26 to 3.54, i.e. the section headed “Acknowledged
Bugs”. The only bugs referred to in this section are the three bugs identified by Post
Office, plus Dalmellington. Mr Coyne was unable to give any further examples.°? He
accepted that the evidence showed that once a bug was identified, Fujitsu took steps to
identify all affected branches:
Q... Let me ask one last question before the break. In relation to bugs that are
detected after a period of time there's evidence showing, I am sure you all agree, that
investigations are undertaken by Fujitsu to ensure that all the branches that are
affected in the meantime are identified, are you aware of that evidence?
A. Yes.
Q. It is said that this is a standard process undertaken by Fujitsu when they identify
a bug that could affect branches, yes?
A. Yes.
Q. Do you have reason for thinking that that has not happened in any number of
cases?
A. Iam aware of one where the data was no longer available to investigate it.
Q. Which bug was that?
A, I would have to find the example. It is in the report.
Q. I see.
A. I would have to find the example. I know that on Dalmellington there was I think
at least two occurrences where Fujitsu weren't able to identify what the impact
actually was. They were able to identify the number of branches
Q. We will come to Dalmellington. So it is Dalmellington and one other, those are
two examples you are aware of, is that right?
A, I've certainly got an example of another, yes.
8 (C5/36/5}.
©) Mayl5/145:8} to {Day15/145:18}.
AI8/182
POL00026925
POL00026925
Q. Are you aware of any other examples of this not happening?
A. No.2
502. As has been noted above, Mr Coyne in fact accepted that the vast majority of instances
503.
504.
of the Dalmellington bug were addressed at the time by the SPM or a TC and that the
other four were either not instances of the bug or were in all likelihood resolved: see
para. 392 above.
Mr Coyne’s evidence at trial — important concessions on extent
Many of Mr Coyne’s important concessions in this regard have been identified in
addressing Dr Worden’s evidence.
Mr Coyne accepted that a possible yardstick for answering Horizon Issue 1 was the
likelihood of bugs in Horizon causing shortfalls to Cs:
Q. Horizon Issue 1 requires the experts to consider the extent of the -- and I'm using
shorthand now
A. Yes, that's good.
Q. I hope it is not controversial. The extent of the likelihood of bugs in Horizon
causing shortfalls in branch accounts.
A. Yes.
Q. And I'm suggesting to you that one useful yardstick for measuring extent is whether
the likelihood in this case is of any sort which could begin to justify the claims that
these proceedings are designed to decide.
A. Yes.
Q. And do you accept that that could be a usefil yardstick for measuring extent in
the context of this case?
A. Ican see how it might be one of the contenders for that, yes.°*'
505. Mr Coyne accepted that when it came to considering the question of extent, it was
legitimate in principle to consider and draw inferences from the information available,
in particular in Peaks, as to the financial impact of the identified bugs:
6 {Day15/145:19} to {Day15/146:23}.
1 {Day15/155:2} to {Day15/155:17}.
183
AI8/183
POL00026925
POL00026925
Q. I'm not suggesting that you could arrive ata certain conclusion of an absolute
cast iron number, but I repeat my question. Can't you form an estimate having regard
to the totality of the PEAKs that you have seen?
A. Yes, but your estimate would have to be based on the three people where it has
been recorded to have occurred, so you would say £20, £30 and £50, and the best you
could possibly do is come up with an average of that, and you would say that the
value of that defect is whatever that is.
Q. There is another way that you could do it, isn't there, which is that you could look
at the three bugs, the receipts and payments mismatch, the suspense account bug and
Callendar Square, the ones that have been thoroughly investigated, and you could
form inferences from the scale of those bugs, yes? Would that be reasonable?
A. For those types of bugs, potentially yes.
Q. Those are quite large bugs, aren't they? They are not small bugs in the scheme of
things. That's why they were identified in the letter, because these were major bugs
of which even Post Office was aware?
A. Yes...0
506. Mr Coyne also accepted that he had no basis to challenge Dr Worden’s analysis on the
three known bugs. Nevertheless he maintained that the losses could be greater than Dr
Worden believes to be the case:
Q. Are you aware of any evidence to suggest that Dr Worden's analysis is wrong? Is
there any evidence to challenge his calculations in relation to those three bugs?
A. No. Iunderstand the process that Dr Worden has gone through, he has looked at
the numerical values that are recorded in the PEAKs for the branches that are
available that are recorded in there and he has added those up, so I do not think his
maths is going to be wrong.
Q. So if we were to take those these bugs as some kind of indication of fairly sizeable
bugs that might appear in the system, it is fair to say, isn't it, that £100,000 is quite
small compared to the £19 million that's claimed by the claimants in this case. It is
less than 1%, correct?
A. Just on the pure numbers, yes.
Q. So these three bugs, which are the ones we know most about, do not by themselves
even begin to support the claimants’ case, do they?
A. But the numbers that are given are only the numbers that are in the PEAKs, and
the PEAKs only reflect where Fujitsu have become involved and have started to
investigate the impact of those bugs from the branches that they are aware of.
62 {Day15/157:20} to {Day15/158:16}.
184
AI8/184
507.
508.
POL00026925
POL00026925
Q. So assuming that £100,000 represents a fair assessment of the impact of those
bugs, what would you say? You would say, well, they are the tip of the iceberg. There
are many more bugs that are capable of producing the kind of financial loss that
would justify the claimants’ claim, would you say that?
A. Well, it is my position that there's many more than the three and I have set out
these here.
Q. In paragraph 3.105 of your report you said potentially thousands, but I think you
have moved from that now. Now you are saying perhaps up to 40, yes?
A. On the logic that we went through before, yes.
Q. Well, you accepted that logic, didn't you?
A. My position as stated in the report is that there is the potential for more.
Q. Well, I won't go back over the answers you have already given in cross-
examination, Mr Coyne. Do you accept that 40 bugs are plainly nowhere near enough
to have caused the claimants the shortfalls that they are seeking to recover?
A. I don't accept that position. Because the bug impacts the transaction that would
be in effect at the time, if it was a large transaction at the time then the impact of that
bug would be a lot larger. In the alternative, they may have experienced the bug a
number of times but on smaller transactions.°*
Mr Coyne’s ultimate fall-back position was therefore to speculate that there might be
many bugs with huge financial impacts that had somehow escaped detection. That fails
a common sense test — a bug that had a huge mean financial impact would be very easy
to detect, not least because the SPM would find it easy to trace any problem to the
relevant (unusually large) transaction®*
and, even if he could not do this, he would hardly
be likely not to report a huge and sudden change in his branch’s position. Further, there
are relatively few transaction types that often involve very large sums of money -
remittances are one and large banking transactions are another — and there are strong
countermeasures in place to resolve any problems arising in respect of those transactions.
Mr Coyne’s other suggestion, that there might be bugs with small impacts but which
occurred a great many times finds no support in the evidence. But it was nonetheless
considered as a theoretical possibility in Dr Worden’s first report. He concluded that
3 (Day15/164:22} to {Day]5/166:22}.
4 A sudden shortfall of £25,000 (or anything like that sum) would be easy to connect to a transaction of that
value done since the last mandatory cash declaration ~ i.e. in the last trading day.
185
Al8/185
POL00026925
POL00026925
“micro bugs” were not a good candidate for having caused large losses. He was not
challenged on this.
F10. Data entry errors (mis-keying)
509. Data entry errors are referred to specifically in Horizon Issues 4 and 6 (as point (a) in
each case). It is common ground that Horizon, in order to be robust, should have in place
appropriate countermeasures to address data entry errors. This includes both measures
to avoid data entry errors occurring and measures to address such data entry errors as do
occur.
510. As to the second type of measure:
510.1 Dr Worden's view is that the error correction processes within Horizon, most
notably those that result in TCs, correct for mis-keyed transactions. (It is also of
course common ground that such processes will not, and could never, catch each
instance.) Mr Coyne confirmed that the correction of user errors forms part of the
experts’ assessment of robustness.°*°
510.2 The factual evidence shows that many data entry errors by SPMs are addressed in
the ordinary course. The Court will recall the evidence given by the claimant-
specific witnesses to the effect that mistakes that they made in performing
transactions were typically corrected by a TC. The call logs and TC records for
those witnesses show that these corrective processes were common-place.
511. As to the first type of measure — measures to reduce the incidence of data entry errors in
the first place:
511.1 Dr Worden identified a series of measures that are used in Horizon to prevent data
entry errors, including (1) the use of menus and buttons in preference to free text
input, (2) limiting the number and type of user inputs that can be accepted, (3)
requiring confirmation of various inputs before they are accepted, (4) a
85 {Day19/96:4} to {Day19/96:10:.
66 {Day14/25:17} to {Dayl4/26:15}.
186
AI8/186
POL00026925
POL00026925
requirement that the price recorded as being received matches for the items in the
basket: Worden 1/223-224.°*” Dr Worden describes these as “standard techniques”
that were “well designed” in this case.°**
511.2 Mr Coyne agrees that “a large number of measures were implemented within
Horizon to prevent user error”.°° He points out that there is evidence that these
measures did not always prevent errors in data entry (which is certainly true).
511.3 Mr Coyne goes on to criticise Post Office for not having implemented a proposal
made in a 2008 document™® that all transaction values entered at the counter for
financial products require double entry and cross-validation (i.e. entering the
figure twice). But that criticism is difficult to credit given the following:
(a) Mr Coyne in fact agrees with Dr Worden’s view that it “may not be a good
choice” to require double entry of all transaction values. Mr Coyne therefore
does not think that Post Office should have implemented the 2008
recommendation after all.
(b) Mr Coyne suggests that the better course would instead be to require that
only entries above a certain value — he puts forward £1,000 as a suggestion
— be entered twice. He makes the same suggestion in respect of values that
have the same numeral twice in a row. Mr Coyne has thought of these
proposals himself,” tweaking a proposal that was made and considered and
that he recognises might have greater downsides than upsides. The mis-
keying documents in the trial bundle show that Post Office devoted
substantial resource, including workshopping and consultation in devising,
testing and considering measures to reduce mis-keying: see, most notably,
{F/932}, considered further below.
67 ()3/1/63}.
68 Worden 1/470 and 475 {D3/1/118}.
9 Coyne 2/5.77 {D2/4.1/148}.
0 {F/476}
& Coyne 2/5.78-5.80 {D2/4.1/149}.
©2 Mr Coyne does not appear to have taken into account that many transactions that would be affected by the
recommendation in the 2008 document do now involve the checking and validation of the transaction value:
the customer is invited to check and confirm the amount shown on the chip and pin keypad.
187
Al8/187
512.
()
to keeping the user interface stable.’
Dr Worden’s evidence (which he developed further in response to cross-
examination) is that his experience has shown that the design of user
interfaces is a “deceptively complex topic” where improvements that might
appear to be obvious turn out, after careful examination and user trials, not
to be a good idea: Worden 1/474.° He was slow to second-guess carefully
considered design decisions without the benefit of user trials:
Q. And you haven't actually had any regard to the specific
layout of the individual screens, have you?
A. I haven’t tried to redesign screens myself or to
consider how they might have been better. I think that
is a very dangerous exercise for an IT engineer to do
actually.
Q. Because it is not your field of expertise?
A. Well, [have run a field of — a team of user interface
experts and we did user interface design for things like
air traffic control, and the one thing that I learnt is
the designer’s prejudices about what is a good user
interface or not are not to be trusted, and one thing
you have to do is user trials to find out what works.
Q. Yes. And that is a very important part of having
a robust system?
A. Yes, user interfaces should be tried out and evaluated.“ (emphasis
added)
511.4 Dr Worden also referred in his oral evidence to the kind of trade-offs that are
involved in considering changes to user interfaces, including that there is a benefit
645
It is important to recall that Mr Coyne’s opinions on these Horizon Issues are tainted by
his misconception of the task that he was required to carry out — he considered that his
role was to identify whether the risk of error had been “reduced as far as possible”: see,
POL00026925
POL00026925
83 (D3/1/118}.
© {Day19/93:2} to {Day19/93:18}
5 (Day19/98:11} to {Day19/99:7}.
188
AI8/188
513.
514.
POL00026925
POL00026925
e.g., JS3, para. 6.4.5" It is easy to see how, applying that mistaken approach, it would be
easy to think that more could be done to reduce mis-keying errors (as it always could, if
one were to ignore the trade-offs). Mr Coyne does not say that Horizon’s measures to
reduce mis-keying errors are below any relevant industry standard or otherwise less than
good. His view is that they could be better, but that is not a Horizon Issue.
Dr Worden also explained that certain characteristics of Post Office’s business made it
particularly concerned to reduce user errors — in short, the costs of such errors would fall
primarily on Post Office because of the volume of transactions and the low margins on
those transactions: Worden 1/225-230.7
Dr Worden was cross-examined in this regard on the basis that he had misread a
document as showing a concern over the costs to Post Office of mis-keying when it was
in fact about losses caused to SPMs.“** The document in question is at {F/932}. The
Court is invited to read that document in full and then to consider the cross-examination
on it at {Day19/94:7} to {Day19/98:10}. Dr Worden was measured and fair in his
answers, and the case put to him was entirely without merit:
514.1 {F/932} is clearly a document that addresses the costs to Post Office of having to
address mis-keying errors. The “Introduction” states, amongst other things, that
the costs of mis-keying were becoming “prohibitively expensive”, and it refers to
the “the problem POL is currently experiencing”. The document presents many
and various proposals all aimed at reducing the cost and the problem.
514.2 The “Background” section, which is the section to which Dr Worden specifically
referred in his report, begins with reference to the “detriment to P&BA” (a Post
Office department — Product and Branch Accounting). It is true that it also
mentions the impacts on branches. But the clear focus of this section is on costs to
Post Office: see the final para., which sets out various such costs.
514.3 The focus of the document is the reduction of costs to Post Office. In that context,
it refers, unsurprisingly, to the fact that mis-keying errors are also a bad thing for
6 (DI/4/10}
7 13/1/64}.
8 {Day19/94:7} to {Day19/98:10}.
189
A/8/189
SIS.
516.
POL00026925
POL00026925
SPMs. It would be surprising and show a lack of appropriate concern if it did not
do so. The two things are of course not entirely separate: each mis-keyed
transaction is both a problem for the SPM and a problem for Post Office (which,
as this document makes clear, spend a lot of time and money correcting mis-keyed
transactions).
Fil. Reliability of transaction data in Horizon and in Post Office’s
Management Information Systems
This topic is relevant, in varying degrees, to Horizon Issues 1, 3, 4 and 6. Post Office’s
Management Information Systems (“MIS”) are also relevant to Horizon Issues 5, 8 and
15.
Cs mounted at trial an attack on the integrity of data in the Horizon audit store and the
accuracy of data in Post Office’s MIS. Neither attack was based on any substantial
evidence, but each is addressed in detail below. There are some important preliminary
points to note:
516.1 The experts agree that the system architecture is sound. Neither has identified any
major flaw in the design, including the design of the audit store.
516.2 Post Office’s MIS are back-office systems. They do not form part of Horizon. The
data in Post Office’s MIS is not the branch transaction data that is used to generate
branch accounts.
516.3 The MIS and the audit store are fed copies of branch transaction data from the
BRDB (which is where the accounts are in fact generated).
516.4 There is no evidence of data extracted from the audit store having been different
from the data in the BRDB. Mr Coyne does not suggest that this has happened.
(There are indications that duplicates were sometimes detected in extractions from
the audit store under Legacy Horizon, but there is no evidence to suggest that these
were not addressed.)
190
A/8/190
S17.
518.
516.5 There is also no evidence that the process of moving data from the BRDB to Post
Office’s MIS is prone to introducing error into that data (let alone that such error
is then brought back into the branch accounts).
In this context, while there is a lot of material to address on this topic, its importance to
the Horizon Issues is relatively modest.
Post Office’s reliance on MIS
In his second report, Mr Coyne placed a great deal of importance on the suggestion that
Credence, an MIS used by Post Office, was somehow unreliable and should not have
been used to investigate discrepancies and decide on TCs. This was one of the recurrent
themes of Coyne 2:
518.1 At para 1.2(c), Horizon is said to be less robust than Mr Coyne had thought when
he drafted his first report because “Post Office does not consult the full audit trail
before ruling on a discrepancy, instead using third party client reconciliation data
or subsections of the audit data from within Credence or HORice”.6”
518.2 At para 4.60, it is said that Credence “is used by Post Office to attempt to validate
branch accounts but contains insufficient audit data for that purpose” 6°
518.3 At para 5.40, it is suggested that there are “/imitations of utilising Credence as an
error proof source of determining financial integrity ”.°! Mr Coyne refers back to
this at para 5.126.
518.4 Para 5.54 states that “previous evidence has illustrated” that Credence “did not
provide the full picture of the Horizon situation” ©
POL00026925
POL00026925
9 ¢D2/4.1/7}.
60 ¢D2/4.1/112}.
1 ¢D2/4.1/134}, referring back to Coyne 1/5.174-182 {D2/1/101}
62 ¢D2/4.1/161}.
63 ¢D2/4.1/137}.
191
A/8/191
S19.
520.
521.
518.5 Para 5.119 refers to “errors potentially introduced from consulting only a subset
of the available data when dealing with branch discrepancies” and the “limitations
of the Credence Management Information System”.°4
518.6 At para 5.131, it is stated that the information available via Credence is “only a
subset of the complete data set and may indicate a different outcome to that when
viewing the more complete audit data only available to Fujitsu”.
518.7 It is suggested at para 5.414 that using Credence may result in an incorrect decision
on a TC because “the underlying data set was not comprehensive enough in the
first instance” 6°
In each and every one of these paras, Mr Coyne relied on his interpretation of a draft
document produced in June 2013 by Helen Rose, a fraud analyst at Post Office -which
has come to be referred to as the “Helen Rose Report”. The document is at {F/1082}.
Coyne 2/5.40 also relied on the End to End Reconciliation Report dated 27 February
2012, which is at {F/896}. These documents are addressed in turn below.
Post Office submits that none of the points made by Mr Coyne finds any real support in
these documents. This is another example of Mr Coyne’s tendency to make and repeat a
seemingly important point based on reference to one or two documents that, if read
carefully, cannot sustain anything like the weight that he chose to place on them.
Helen Rose Report
The Helen Rose Report can be taken first. It arose from the investigation of a recovery
situation in the Lepton branch in October 2012. On 4 October 2012, the SPM at the
Lepton branch, Mr Armstrong had already taken payment for a BT bill payment when
the system went down, and the system reversed the bill payment as part of the recovery
process. The customer had already left the branch and so could not be refunded. A TC
was issued to Mr Armstrong to re-instate the bill payment and was accepted at the branch.
POL00026925
POL00026925
64 ¢(D2/4.1/159}.
65 (D)2/4,1/162}.
656 ¢D2/4.1/241}.
AI8/192
522.
523.
524.
525.
POL00026925
POL00026925
Mr Armstrong was concerned by a suggestion that he had himself initiated the reversal,
rather than it forming part of the recovery process. He complained to Second Sight.
Ms Rose investigated the complaint and produced a draft report dated 12 June 2013 in
which she reviewed the transaction data available through Credence, emails sent to her
by Gareth Jenkins and ARQ data provided by Fujitsu. Ms Rose was concerned that the
Credence data had appeared to suggest that the reversal had been initiated by the SPM,
and she recommended a change to make clear where a reversal is carried out as a part of
the recovery process: see {F/1082/3}.
In cross-examination, Mr Coyne retreated from three of the main points that he had made
in reliance on Ms Rose’s report. Specifically:
523.1 Contrary to his written evidence and the answer initially given in cross-
examination, he ultimately accepted that the TC issued to Mr Armstrong had been
correct.
523.2 He ultimately accepted that it was “possible” (Post Office submits it is clear) that
the Credence data considered by Ms Rose was not “wrong”, which was a
suggestion he made both in his reports and in his initial answer in cross-
examination.
523.3 Contrary to his written evidence and the answer initially given in cross-
examination, he accepted that the Helen Rose Report suggested that a
disconnected session receipt had been printed.
These three points are considered separately below before addressing the one point on
which Mr Coyne sought to defend the statements made in his reports.
First, Mr Coyne ultimately accepted in cross-examination that the TC issued to Mr
Armstrong was correctly issued.°°’ The TC reinstated the bill payment that had been
reversed as part of the recovery process but for which Mr Armstrong had already taken
payment before the session failed. The TC thereby (correctly) removed the cash surplus
657 {Day 5/54:8} to {Day15/54:22}.
193
AI8/193
526.
527.
528.
529,
POL00026925
POL00026925
resulting from the payment, put the branch accounts back into balance and corrected the
position for the customer.
Mr Coyne’s acceptance of this point was directly contrary to Coyne 1/5.50,°° which
stated without qualification that the TC had been “isswed in error”. Mr Coyne had
confirmed that this remained his view earlier in the cross-examination.°?
It is unclear on what basis Mr Coyne felt able to state that the TC had been issued in
error. There is no such suggestion in Ms Rose’s report. Mr Coyne was overly eager to
make a criticism of Post Office and somehow persuaded himself (with no evidence for
this in the documents) that the TC must have been issued in error.
Second, the Credence data referred to Ms Rose had not been “wrong”. Mr Coyne was
not prepared to accept this point in its entirety. He accepted it was “possible” that Ms
Rose had misunderstood the Credence data and what it suggested in relation to the
reversal.® His reading of the document was, however, that the Credence data had stated
that the reversal was initiated by Mr Armstrong. He reached this view on the basis that
the document included the words “/ooking at the credence data, it clearly indicates that
the reversal was completed by JAROO1 (postmaster)” °°!
The document did not say what Mr Coyne took from it. On any fair reading of the Helen
Rose Report, it is clear that Ms Rose had reached a conclusion on the basis of inference
from the data on Credence. She referred to the grounds for that inference in the third
para., and her “Recommendations” at the end of the document made clear that her
concern arose from the absence of any clear indication in the data that the reversal was
initiated by the system as part of the recovery process: “my concerns are that we cannot
see clearly what has happened on the data available to us and this in itself may _be
misinterpreted... My recommendation is that a change request is submitted so that all
system created reversals are clearly identifiable on both fujitsu and credence” (emphasis
added).°® She did not state anywhere that any information on Credence was wrong. Ms
8 (D2/1/67}.
69 {Day15/37:9} to {Day15/37:17}.
60 {Day15/40:21}.
61 {Day15/39:13} to {Day15/39:16}.
662 {F/1082/3}.
194
AI8/194
Rose made an understandable error of interpretation, and she was concerned that others
might do so too. That is the whole thrust of the draft report.
530. Mr Coyne may have felt unable to concede this point entirely because he had, in Coyne
2, made the clear and strong claim that Ms Rose’s report demonstrated that Credence
data “is either wrong or does not provide sufficient information to complete the full
picture” (emphasis added).°* It would be fair to say that Ms Rose reached the view that
the Credence data was not, on the facts of that case, sufficient to give the “fill picture”
(although, unknown to Mr Coyne, the position was entirely clear from the receipts held
by Mr Armstrong). But the suggestion that the Credence data was “wrong” finds no
support in Ms Rose’s report. Mr Coyne went beyond what the document could support
in making that suggestion. He should have resiled from it, but he felt unable to do so.
531. Ms Rose’s report is the only piece of evidence to which Mr Coyne refers that might
suggest that the data in Credence has ever been different from that in the audit store. Mr
Coyne quite properly acknowledged in his second report that it “may be correct” that Mr
Godeseth is not aware of any instance where the data retrieved from the audit store has
differed from data from other sources (including Credence).°' He does not refer to any
Peak or other document indicating that data from Credence had been shown to be
different from the data in the audit store. Nor does he refer to any evidence to suggest
that Credence has ever suffered from a bug that might cause it to present wrong data.°
532. It is difficult to avoid the impression that Mr Coyne felt that he had to cling to Ms Rose’s
report as his sole piece of evidence to suggest that Credence data has been “wrong”.
Given the importance that Mr Coyne attached to Credence’s supposed unreliability, this
was a point on which he should have been particularly careful to give a fair and balanced
view, but he did not do so.
533. Third, the report in fact suggested that at least one automatic receipt had been printed.
Mr Coyne ultimately accepted that, on a fair reading of the document as a whole, Ms
POL00026925
POL00026925
63 Coyne 2/4.63(a) {D2/4.1/113}.
64 Coyne 2/5.131 {D2/4.1/162}. See Godeseth 1/32 {E2/1/10}.
S$ Ms Mather gave unchallenged evidence that she has never heard of a bug in Credence: see para. 14 of her
WS {E2/8/3}.
195
AI8/195
POL00026925
POL00026925
Rose’s report “suggests that the receipt was printed”®® (referring to a disconnected
session receipt).
534. This was directly contrary to Coyne 2/4.78, where it is was stated that the report
“indicates that there was no evidence of the creation of a disconnected session
receipt” © In that para Coyne quoted from one of the emails from Gareth Jenkins that
was set out by Ms Rose (the email quoted on page 1), failing to make any reference to
the other email extract that Ms Rose quoted (on page 2). The second extract makes clear
that Mr Jenkins believed that a receipt had been printed.°*
535. In fairness to Mr Coyne, the report was somewhat difficult to follow on this point. It is,
however, clear from a document that Mr Coyne had not seen when he prepared his
reports (an email chain between Mr Armstrong, Second Sight and Post Office)* that the
true position is that there was a total of 4 receipts printed, as follows:
535.1 3 disconnected session receipts printed in session 537803.°” It is normal procedure
for 3 receipts to be printed: one is to be kept at the counter to aid in the recovery
process; one is for the customer; one is for the branch’s records.°”
535.2 A recovery receipt printed in session 537805.°” (The Managing Judge noted the
difference in session numbers during the hearing).°”
66 {Day15/45:24}.
7 (D2/4.1/117}.
68 See the second para. in blue text at {F/1082/3}. Mr Jenkins may in fact have been referring to a recovery
receipt.
9 {F/1095.1}. Mr Green QC correctly pointed out during the hearing that this document had been disclosed
on 7 March 2019. If some implicit criticism was intended, there is no merit in it. The emails were not
included in the Second Sight disclosure ordered by the Court. There is a good reason for that. Second Sight
did not return these emails to Post Office, as it was required to do under its agreement with Post Office, so
the emails did not fall within the body of documents located and disclosed by Post Office in accordance
with the Order, Post Office only later discovered the emails in its preparation for trial, and it disclosed
them once their relevance became clear. Post Office complied with the order for disclosure and cannot
fairly be criticised. The emails had been copied to Mr Bates at the time (and were not disclosed by him).
© F/1095.1/4}.
I Mr Coyne was aware that the system should print 3 disconnected session receipts: see Coyne 1/4.59
{D2/1/44}. The uses of the 3 receipts are described in “Recovery - Horizon Online Quick Reference
Guide”: see {F/1246/1}
6 §F/1095.1/5}.
53 {Day15/48:5} and following.
196
AI8/196
536.
537.
538.
539,
POL00026925
POL00026925
The system therefore worked as designed in printing receipts to guide the recovery
process and inform the SPM.
Given that Mr Coyne had not seen the emails when he drafted his reports, he could quite
fairly have said that it was not clear from the Helen Rose Report what automatic receipts
had been printed. But that is not what he said. He instead made a clear statement to the
effect that Ms Rose’s report suggested that no disconnected session receipt had been
printed, i.e. that Horizon had not performed as it should have in this regard. He based
this on one email extracted in the report, choosing to ignore the other, later email.
This provides another example of Mr Coyne’s worrying tendency to rely on one part of
a document (invariably a part that might support criticism of Horizon or Post Office) to
the exclusion of the rest of the document and any contrary indications that it provides.
This was put to Mr Coyne in cross-examination. His only answer was to change the
subject (referring to a different point that he had made about Ms Rose’s report):
Q. So would it be fair to say that in your anxiety to write a bad thing, to be able to
write down a bad thing in your report about Horizon, you recorded what was said on
the first page of the report, but you didn't look at the second page of the report which
would have shown that bad thing wasn't in fact correct?
A. No, but my point was about this report is to show that there is a difference between
the view that you get of the data from viewing the Credence data from the ARQ data,
and that's correct.
Q. Mr Coyne, if you just made that claim we would have been in and out of this issue
within about five minutes. The reason why we have spent about 20 minutes so far is
because you made several claims, and I set them out orally and you agreed that you
were making each of those claims on the basis of this Helen Rose report...°*
If Mr Coyne had a good explanation for the selective approach that he adopted, he would
have given it. If it was simply a mistake, he would have said so (as he did, fairly, in
relation to other points). Mr Coyne avoided the question because he had no answer to it.
As stated above, there was one point that Mr Coyne made about Ms Rose’s report, on
which he was challenged and that he was not willing to concede was wrong. It related to
the different times stated in the Helen Rose Report for the BT bill payment transaction.
Mr Coyne’s oral evidence on this point showed an unwillingness to concede what was
4 (Day15/45:4} and following.
197
AI8/197
540.
S41.
542,
POL00026925
POL00026925
obvious from the documents. He even resorted to contradicting a fact that he had himself
identified in his first report.
The time at which the BT bill payment took place
At paras 5.176 to 5.177 of his first report, Mr Coyne identified what he called “further
issues with the data provided by Fujitsu’, referring to the fact that the “initial report”
(which was a reference to Ms Rose’s report) records the BT bill payment transaction as
having taken place at 10:42, whereas the Credence data file shows it as having taken
place at 10:32.5”5 The implicit suggestion was that the SPM might have been confused
by the times shown on various documents and/or that Post Office’s investigation may
have gone awry as a result of the inconsistency between the stated times.
There are two glaring problems with that suggestion. Mr Coyne went some way to
accepting this in cross-examination before digging in and resorting to speculation
contrary to the facts. He perhaps felt that he could not cede any more ground lest his
reliance on the Helen Rose Report be undermined entirely.
First, the overwhelming likelihood is that Ms Rose’s reference to the transaction having
taken place at “10:42” was a typographical error, her intention having been to type
“10:32”:
542.1 That is in fact the time at which the transaction took place.°”°
542.2 Ms Rose records in the same paragraph that the transaction was reversed at 10:37.
She cannot have thought that the transaction took place after it was reversed.
542.3 In giving these times, Ms Rose was reporting on what she could see from the
Credence data.°””
542.4 If Ms Rose had at any time concluded that one of the times shown in the Credence
data was wrong, she would obviously have mentioned this in the report. She was
5 ¢D2/1/102}.
6 The Credence data file records the transaction as having taken place at 10:3
See also Mr Armstrong’s confirmation of the time at 5-05.44}
: Coyne 1/5.177 {D2/1/102}.
695/43 (final para.).
7 Tt is made explicit that the time stated for the reversal was taken from the Credence data: see the third
para. on internal page 1 {F/1082/2}.
198
{F/1095.1/4}
AI8/198
concerned that the Credence data did not provide enough information to make
clear that the reversal had been initiated as part of the recovery process. She would
have been doubly concerned by any indication that Credence was somehow
recording transactions and reversals out of order and/or was attaching wrong time
stamps to transactions.
543. Mr Coyne sensibly accepted that Ms Rose’s reference to 10:42 in the first para of the
544.
document “could well be a mis-key”.©’* But it is difficult to see how he can ever have
thought otherwise, unless he simply latched on to something that might be used to
criticise Credence without giving it much thought. He again appears to have been overly
ready to jump to adverse conclusions.
Mr Coyne nonetheless insisted that it was also possible that, rather than Ms Rose having
made a typographical error, the time shown in the Credence data was in fact wrong. It is
important to record precisely what Mr Coyne said here:
Q. Are you really seriously suggesting that Credence was indicating that_the
transaction was done at 10.42 and_that that's a reason for suggesting, for thinking,
that Credence is unreliable? Is that really your contention?
A,_Times on computers can be out. They do drift. It is possible that it's got the time
wrong. I agree with your position that it could well be a mis-key on behalf of Helen
Rose. &” (emphasis added)
545. The underlined answer reflects poorly on Mr Coyne’s reliability as an expert:
545.1 Mr Coyne was asked carefully to confirm what he was saying, and he embraced
the possibility that the Credence data might have shown (wrongly) that the
transaction took place at 10:42. If true, that would indeed suggest an error in
Credence, rather than a typographical error made by Ms Rose.
545.2 Mr Coyne took refuge in this suggestion despite the fact that he knew (at least
when he drafted his first report) that the Credence data showed the transaction to
have taken place at 10:32 (i.e. the correct time): see Coyne 1/5.177 — “...the
credence data file shows 10:32”. Mr Coyne had either forgotten this (in which
POL00026925
POL00026925
8 {Day 5/50:17}.
9 {Day 5/50:12} and following.
ss ¢D2/1/102}.
199
A/8/199
546.
547.
548.
POL00026925
POL00026925
case, he should not have speculated in the way he did), or was simply willing to
say whatever he had to say in order to defend the criticism that he had.
545.3 It is, in any event, troubling that Mr Coyne was prepared to speculate as to
Credence having possibly “drifted” in time. There is no suggestion in either of his
reports (or any documents to which he has referred) that Credence has ever
“drifted” so as to record transactions 10 minutes later than when they actually
occurred. In any event, the suggestion was non-sensical: if Credence had “drifted”
in time (i.e. got out of step with the correct time), it would also have recorded the
reversal at a later time, whereas it is clear that it recorded the time of the reversal
correctly.**! A moment’s fair consideration would have revealed that.
It is difficult to avoid the impression that Mr Coyne was fishing around for something
new to say in defence of a criticism that he had made on a very tenuous basis.
Second, Ms Rose’s report was drafted after the event and as part of the investigation into
the SPM’s complaint. It was not seen by Mr Armstrong at the time of the recovery
process or even at the time of his complaint to Second Sight. It cannot possibly have
confused him in relation to the time at which the transaction took place.®’ As for Post
Office, it of course had the benefit of what Credence actually showed, and it could (and
did) seek further information from Fujitsu.
Mr Coyne also suggested in his first report that the fact that the ARQ data showed timings
one hour different from the Credence data (because the ARQ data is always in GMT,
whereas it was BST at the date of the reversal — 4 October 2012) might have generated
difficulty for the SPM:
Whilst this hour difference between the data sets might be easily traceable for Fujitsu,
it is not clear how easily it would have been to investigate issues where the
Subpostmaster was not sure of what time things went erroneously in the system, or that
it was a reversal specifically. (Coyne 1/5.178) &
“! The time of the reversal is recorded in the third para. on p. 1 of the document: {F/1082/2}. The time of
the reversal was also shown on the recovery receipt referred to in the first para. at {F/1095.1/5}.
6? This leaves to one side that Mr Armstrong was in fact aware of the time of the transaction: see his email
at (F/1095.1/4}. Again, it is important to note that Mr Coyne had not seen that email.
683 ¢D2/1/102}.
200
A/8/200
POL00026925
POL00026925
549, Mr Coyne again allowed himself to speculate as to potential problems of which Ms
Rose’s document provides no evidence. Further, there is no suggestion in either of Mr
Coyne’s reports that any SPM’s investigation into a discrepancy has ever been hindered
by the fact that ARQ data is recorded in GMT. Mr Coyne was challenged on whether it
was really plausible to think that an SPM might obtain ARQ data from Post Office but
not be told that the data was in GMT:
Q. As for the second point made at 5.177, that the ARQ data always works in
accordance with Greenwich Mean Time, whereas everybody else at the time was
working on British Summer Time, that's not a serious problem, is it? It's not
something that is going to cause great difficulties to anybody, is it?
A. As soon as you know that you are an hour adrift then it becomes very easy to deal
with, but if you don't know that it is problematic.
Q. So are you imagining a world in which Mr Armstrong is provided with ARQ data
but nobody tells him that ARQ data is based upon Greenwich Mean Time, is that your
assumption? And that's a problem, because nobody tells him that ARQ data is based
on Greenwich Mean Time?
A. No, my answer is if you are told then it becomes very clear very quickly, but if you
are not told it is confusing.
Q. But in 5.178, Mr Coyne, you seem to be assuming {D2/1/102}, remarkably, that
no one would have told him. You say:
"... it is not clear how easily it would have been to investigate issues where the
Subpostmaster was not sure of what time things went on erroneously in the
system ..." Why are you assuming that, having reached a point where the
subpostmaster actually has the ARQ data, no one is going to help him understand
that there is an hour discrepancy between the ARQ data and British Summer Time?
A. The point that I'm making is that unless somebody tells him it wouldn't be clear. I
do not think a user would typically know that the computer would be an hour out.
I think the assumption would be that if it is an audit system of some description, that
the clock difference would actually be dealt with correctly."
550. Mr Coyne did not appear to have considered whether this theoretical source of confusion
was, in the real world, likely to cause any problem. It would be fanciful to suggest that
an SPM who was engaging with Post Office to get to the bottom of a discrepancy, and
who was provided with ARQ data as part of that process, would be kept entirely in the
dark as to the fact that audit records are kept in GMT. There would be no reason at all
for Post Office not to provide that information and every reason for it to do so. There is
Ss {Day 5/51:2} to (Day15/51:18}.
201
A/8/201
no evidence in the documents of any SPM having been confused by the fact that ARQ
data is kept in GMT and Post Office having allowed that confusion to go uncorrected.°.
551. Mr Coyne even speculated that the SPM might not be aware that “it was a reversal
specifically” that had given rise to a discrepancy, supposedly compounding the
(theoretical) confusion created by the fact that audit data is kept in GMT. But that
suggestion is very hard to follow: the SPM discussed in Ms Rose’s report, Mr Armstrong,
of course knew that he was disputing the circumstances of a reversal — that was the whole
thrust of his complaint. He knew that there had been a reversal; he knew when the
reversal had occurred (and he even, although he only looked for these later, had the
automatically generated receipts in relation to it); his only complaint was that it had been
suggested that he had himself initiated the reversal, rather than it having been initiated
as part of the recovery process.
552. Standing back, the Helen Rose Report provides no real support for any of the seemingly
important points that Mr Coyne chose to build on the back of it. He gave unsatisfactory
evidence in relation to the document, conceding points only where he felt he had
absolutely no choice and resorting to speculation and evasion where he felt there
remained some room for manoeuvre.
The End to End Reconciliation Report
553. At para 5.40 of his second report,**® Mr Coyne relied on the End to End Reconciliation
Report dated 27 February 2012, by reference back to Coyne 1/5.174.°°” The document is
at {F/896}. Mr Coyne relies on it in relation to supposed “limitations of utilising
Credence as an error proof source of determining financial integrity”.
554. Mr Coyne confirmed in cross-examination that he relied on {F/896} as a basis on which
to suggest that Credence should not be used in order to make decisions on TCs.°** On
POL00026925
POL00026925
5 The Court may recall that, in cross-examination, Mrs Burke asked about the times shown in the ARQ
spreadsheets and was given an explanation (with the assistance of the Managing Judge): {Day3/100:24} to
{Day3/101:23}. Even if Post Office for some reason failed to inform the SPM of the time difference when
providing ARQ data, the point would soon be drawn out.
686 ¢D2/4.1/134}.
87 ¢D2/1/101}.
68 {Day15/55:10} to {Dayl5/55:13}.
A/8/202
555.
556.
557.
558.
POL00026925
POL00026925
any careful reading of the document, it provides no support for that contention
whatsoever. Mr Coyne went some way to accepting this in cross-examination.
First, the document concerns the reconciliation process carried out between Post Office’s
back end systems (specifically, POLSAP) and third-party client systems. It does not
address the business processes through which Post Office decides on TCs. It states this
in clear terms at {F/896/8}:
This document does not attempt to define the business processes undertaken within
Fujitsu Services and Post Office Ltd with respect to the resolution of any exceptions
which may arise, nor does it scope the requirement for any systems that may be
required to assist in this process. This information can be found in the associated
documents.
Mr Coyne accepted that the document was not about the business processes that lead to
decisions to issue TCs.°°
Second, the document refers to the use of POLSAP to “verify financial integrity” through
that reconciliation process. It says this (the passage quoted by Mr Coyne):°°°
There is no formal reconciliation produced between the POLSAP System and the
Credence transaction stream. The Credence stream should therefore not be used to
verify financial integrity and Post Office should ensure the POLSAP System
Transaction information is used for this purpose.
The author would of course know that Post Office used Credence as its principal MIS
for deciding on TCs. That is essential context to asking whether this passage can really
mean, as Mr Coyne would have it mean, that Credence should not be used for that
purpose. This was put to Mr Coyne, and he gave a realistic answer:
Q. And you are suggesting, are you, here in this paragraph, that the writer of this
report in 2012, February 2012 and thereafter, is suggesting that what Post Office has
been doing for the previous 12 or 13 years is completely wrong? Do you honestly
think that that's what the writer of this sentence was intending to convey?
9 (Day15/61:14} to {Day15/62:3}.
© The quotation below is in fact taken from a later version of the same document. {F/896/65} refers to
POLMIS, rather than Credence. The later document is at {F/1686}. The quotation is from {F/1686/62}.
203
A/8/203
559.
560.
561.
POL00026925
POL00026925
A. No, I don't think they are saying that what you have been doing for the last 12
years is completely wrong. They are providing a warning that you should use one set
of systems rather than another set of systems because the two do not reconcile."
That is, in one sense, correct: POLSAP is the system that should be used for the
reconciliation processes described in this document. It would be a bad idea to use two
different data streams interchangeably for that reconciliation process, especially where
those two systems are not themselves formally reconciled one to the other. That is what
the document is saying. What the document is plainly not saying is that there is anything
wrong with using Credence to conduct investigations into discrepancies or to decide on
TCs. POLSAP and Credence have different but complementary functions and purposes:
see, for instance, Ms Mather’s witness statement at para 9 {E2/8/2}. The use of multiple
and complementary MIS is a strength, not a weakness, and reminding the reader of the
MIS different purposes does not imply any weakness or unreliability in either of them.
Mr Coyne accepted that the reference to “financial integrity” relates to the integrity of
data compared between Post Office and the client, rather than to the integrity of the data
used in making decisions on TCs (given that the document does not address that business
process). It is true, as Mr Coyne stated, that an error in reconciliation might feed back
into the business processes through which Post Office decides on TCs. But that is a
different point and does not support the contention that this document, and its reference
to “financial integrity”, provides evidence that Credence should not be used for deciding
on TCs.
Mr Coyne’s reliance on this document as somehow undermining the reliability of
Credence was unfortunate. He was too eager to latch onto a few words that might, taken
out of context, generate a negative impression. At the very least, he should have
acknowledged in his reports that the document was not really addressing (and, in fact,
had nothing to do with) the point that he wanted to take from it, as he ultimately accepted.
1 (Day15/69:6} to {Day15/69:16}.
2 {Day15/69:17} to {Day15/70:2}.
204
AI8/204
562.
563.
564.
565.
566.
Dr Worden’s evidence in relation to the Credence system
Dr Worden referred in his first report to Post Office’s use of Credence and other MIS.
He noted that, when investigating anomalies reported by SPMs, Post Office uses
Credence and other MIS in the first instance, but that it can also obtain data from the
audit (by requesting such data from Fujitsu): Worden 1/1086.° He considers that Post
Office’s access to transaction data through Horizon reports, but also through MIS, serves
to improve the robustness of the system: Worden 1/1085.°*
Dr Worden was asked relatively few questions about Credence. He was not asked to
comment on Mr Coyne’s criticisms of Post Office’s reliance on its MIS. It was not put
to him that Post Office was wrong to rely, in the first instance, on Credence when
investigating disputes and deciding on TCs. He answered the questions that were put to
him fairly.
The 2010 Royal Mail audit
It was suggested to Dr Worden that Credence was excluded from the scope of the Royal
Mail audits from before 2010.°% That bizarre suggestion was based on the following
words in the audit for the year ended 28 March 2010 {F/646.1/2}:
PO has made significant changes to its IT environment in 2010, resulting in the
inclusion in scope of the Credence application for the first time, replacing POL-MI and
the Reference data system.
As is implicit in those words, the Credence system was not audited in earlier years
because it did not exist. Credence was introduced in April/May 2009.° It replaced the
earlier systems to which the auditors refer. Credence had, in effect, replaced POLMIS.”
Cs leapt on the reference to Credence not having been audited in earlier years, assuming
(wrongly) that this meant that it had been somehow excluded from the audit process. Dr
POL00026925
POL00026925
3 (1)3/1/239}.
4 3/1/23}
5 {Day20/70:4} to {Day20/70:17}
6 Mather, para. 10 {E2/8/2}.
©" Dr Worden noted this during his oral evidence on another day {Day18/34:14}.
205
AI8/205
567.
568.
569,
570.
POL00026925
POL00026925
Worden understandably did not recall the date on which Credence had been introduced,
so was unable immediately to correct the mistaken impression that Cs sought to create.
Keystroke monitoring
Dr Worden was asked whether he agreed with the statement (in Ms Mather’s witness
statement)®* that Credence records all keystroke activity at the counter. This question
was asked without reference to Ms Mather’s oral evidence to the effect that what she had
intended to say was that the Credence records the “transactional data” (i.e. the user
inputs comprised of screen presses and keystrokes).
Dr Worden stated that, before seeing Ms Mather’s statement, he had not understood
Credence to provide a record of all keystroke activity.’ He would not expect that level
of detail to be made available through a MIS.” It was later suggested to him that he had
been “guessing” what functions Credence had, and he disagreed, explaining that he had
relied on his experience of MIS but also what he was able to infer from the documents
that he had reviewed.”
It was then put to Dr Worden that he had given evidence to the effect that he had himself
formed the view, based on his review of documents, that Credence provided a record of
keystroke activity. That was not correct. Dr Worden rejected the mischaracterisation of
his evidence:
Q. And when I started asking about this it did sound as if you were saying you had
separately formed the view from other documents that that was possible?
A. No, [had said I had separately formed a view from other documents that Credence
was used to investigate what happened in the branch.”
Dr Worden was asked to agree that because he had referred to Ms Mather’s witness
statement in his first report and said that it was “consistent with his understanding”,”*
8 Mather, para. 12 {E2/8/3}.
9 {Day6/149:7} to {Day6/149:20}.
20 {Day18/35:12} to {Day18/36:7}.
%! {Day18/36:9} to {Day18/37:13}.
%2 {Day18/46:16} to {Day18/47:6}.
3 {Day18/39:9} to {Day18/39:14}.
%* See fn 41 at {D3/1/239}.
206
A/8/206
572.
573.
574.
the reader would have thought that he was confirming the content of para 12 of that
witness statement, including the statement that Credence recorded keystroke activity. ”°*
Dr Worden was then drawn into a semantic discussion as to the meaning of the word
“consistent” in his reports and in ordinary usage.
A more useful exercise is to focus on what Dr Worden in fact said in his report and the
context in which he referred to Ms Mather’s witness statement. The part of Worden
1/1086 that refers to the witness statement reads as follows: “When Post Office is
investigating anomalies reported by Subpostmasters, they use Credence and their other
management information systems in the first instance...”. The footnoted reference to Ms
Mather’s statement then appears. It is clear, on any fair reading, that Dr Worden was
referring to Ms Mather’s evidence to the effect that Post Office uses Credence and other
MIS in the first instance when investigating anomalies (paras 9-13).”°° He was
confirming that the overview provided by Ms Mather was consistent with his opinion. It
requires a forced reading of the sentence and the footnote to conclude that Dr Worden
intended to provide confirmation of each and every sentence in Ms Mather’s witness
statement.
Nonetheless, Dr Worden fairly accepted that he could easily have made clear in his report
that he was not confirming what Ms Mather said about keystroke activity.”°”
Dr Worden was also taken to the Peak at {F/1848.8.2} (August 2018). In an entry that
begins on page 2 of the Peak, Joe Harrison said as follows: “Here are the keystrokes and
messages from the counter, which might help Atos”. It is clear from what follows that the
log records not only “keystrokes” (in fact, buttons pressed on the counter screen) but also
messages displayed on the counter screen. Dr Worden stated that he understood this
information to be taken from “an operation called TED” and that the keystrokes were
recorded in event logs that were used by Fujitsu in its investigations into bugs.”
Page I of the Peak shows that Henk Bakker of Post Office passed on the issue to Fujitsu
because Post Office could not itself get to the bottom of it, based on the information that
POL00026925
POL00026925
75 {Day18/40:15}.
206 {£2/8/2}.
7 {Day18/49:3} to (Day18/49:11}.
%8 {Day20/51:12} to {Day20/52:2}. See also {Day20/121:24} to {Day20/122:25}.
207
A/8/207
57S.
576.
it had. Fujitsu identified that the problem had arisen as a result of the bug described in
KEL carde235Q and that the script would have to be addressed by Atos. The “keystrokes
and messages” were provided on the basis that they might help Atos to understand what
had gone wrong and what changes to the script may be necessary.
Post Office has sought to identify the “7D” operation to which Dr Worden referred in
his oral evidence, and it appears likely to be a mistaken recollection of “CET”, which is
short for “Counter Eventing Team” (one of the teams that monitors system events at
709
Fujitsu, specifically counter events’ Dr Worden was largely correct in what he said
about keystroke activity monitoring:
575.1 Credence does not provide a record of keystroke activity. Post Office does not,
therefore, have direct access to a record of keystroke activity.
575.2 Fujitsu, by contrast, does have access to a record of keystroke activity. As is
implied by the entry on page 3 of the Peak {F/1848.8.2/3}, the record must be
downloaded “from the counter”, rather than being contained in the events logs at
the BRDB. The counter log is called “POC log”. There are references to it in
various documents in the trial bundle, including {F/813} (2011 KEL), {F/792}
(2011 Peak) and {F/1318.1} (2015 KEL). It is not unusual for the log to be
downloaded and considered as part of Fujitsu’s investigation into a suspected bug
or some other unexplained phenomenon in the branch.
Interpreting the information in the log self-evidently requires skill and experience. There
is no suggestion from either expert that information of this kind would typically be
required for the day-to-day business activities that are supported by Credence and the
other MIS, although it can be obtained on request (in common with other logs that are
available from Fujitsu).
POL00026925
POL00026925
™® See, e.g., {F/823} (definition on p.6 and explanation of CET’s role at p.17). The CET is referred to in the
KEL at {F/680} as checking whether the “counter has actually rebooted” (from which Dr Worden
quoted at {D3/7/33}).
208
A/8/208
577.
578.
579.
580.
581.
POL00026925
POL00026925
Mr Coyne’s evidence in relation ARQ requests and the audit store
There are two important aspects to Mr Coyne’s evidence on ARQ requests and the audit
store:
577.1 Mr Coyne suggested that he had not, at the time of his first report, appreciated that
Post Office used MIS (at least initially) to decide on TCs, rather than consulting
the data in the audit store.
577.2 Ultimately, Mr Coyne could not maintain any serious criticism of Post Office’s
practice of consulting first its MIS and, only where necessary, obtaining further
information through an ARQ request. Much less could he identify how that
practice might undermine the robustness of Horizon.
These two points are addressed in turn below.
The supposed change of understanding between Coyne 1 and Coyne 2
At para 1.2 of his second report, Mr Coyne listed various reasons for which he contended
that Horizon is “/ess robust” than he had considered in his first report. One of those
reasons, listed at (c), was as follows: “Post Office do not consult the full audit data before
ruling on a discrepancy, instead using third party client reconciliation data or
subsections of the audit data from within Credence or HORice”.”°
Mr Coyne confirmed in cross-examination that his contention was that, at the time of
drafting his first report, he had understood that Post Office consulted the data in the audit
store every time that it considered any discrepancy that might lead to a TC.7!"
That is demonstrably not the case. It is clear from the content of his first report that Mr
Coyne knew that the data in the audit store was not consulted as the first port of call for
all questions in relation to discrepancies. Specifically, Mr Coyne knew the following at
the time of his first report:
70 ¢D2/4.1/7}.
%1 (Day15/74:17} to {Day15/75:9}.
209
A/8/209
582.
581.1 The audit store was a “separate audit record of transactions and events” that was
“archived”: Coyne 1/4.51.’? Data was copied to this archive daily” (rather than
on a rolling basis as would be required for business operational purposes).
581.2 The files in the audit store were “sealed digitally and held for seven years during
which time they may be retrieved and filtered to produce the relevant audit for a
particular branch”: Coyne 1/4.53.7"*
581.3 Each time data was “retrieved for audit enquiries”, it was subject to various
checks: Coyne 1/4.347!> (referring to Legacy Horizon, but there is no suggestion
that the process was materially different).
In forming this understanding of the role played by the audit store, Mr Coyne had the
benefit of Mr Godeseth’s first witness statement, which states (amongst other things) as
follows in relation to the audit store (with added emphasis):
24.3...The Audit Store is not involved in the live operation of a branch or Post Office's
business; it is the long term repository of audit data..."°
27. ...Each day the previous day's Message Log is passed to the Audit Store which
then “seals"each file and stores them until they are retrieved (if they ever are) or
deleted in line with the applicable retention period....”""
28. The Audit Store could be seen as the "master record" of the transaction data input
in branch. It is designed to provide long-term, highly secure, storage of data in the
event that any challenge to the data is raised. Save for it being a repository of data,
the Audit Store is not used in live daily operations by Subpostmasters or Post Office.
A copy of the transaction data in the BRDB or in another system is used for day to
day operations.
29. Post Office (its staff or its systems) does not have direct access to the Audit Store.
Post Office may request data from the Audit Store via a process known as the "ARO
process" which requires manual intervention by Fujitsu staff to extract Audit Store
data. The components that are used to provide audit data retrieval facilities are
POL00026925
POL00026925
72 ¢D2/1/43}.
73 ¢D2/1/43}.
74 {2/1/43}. See also Coyne 1/4.27 {D2/1/38} (making the same point in relation to Legacy Horizon).
718 (2/1/39).
26 See para. 24.3 {E2/1/7}.
27 See para. 27 {E2/1/9}.
A/8/210
583.
584.
585.
586.
described in the Audit Data Retrieval High Level Design document {POL-
0440079}."8
30.3 The extracted data is then provided to Post Office, usually in the form of an excel
spreadsheet as this is the most user-friendly format.”
This evidence made abundantly clear that data in the audit store was kept separate from
the day-to-day operations and could only be accessed by Post Office through a request
to Fujitsu, which would then require a manual process of extraction and data validation.
Mr Coyne had read and considered Mr Godeseth’s statement when he drafted Coyne 1:
he refers to it in the section where he considers transaction data and the audit store”° and
again elsewhere in his report.” It was of course a key document that Mr Coyne would
have considered carefully.
Mr Coyne also referred in his first report to other documents that identify the design and
role of the audit store: see, for example, {F/997} (referred to repeatedly in Coyne 1).””
Mr Coyne also set out in his first report a structure diagram that shows clearly that the
BRDB provides a copy of data into “other systems” and, quite separately, writes data to
the audit store (from which data can be retrieved via “Audit Retrieval” and as an “Audit
Extract”): see Coyne I at internal p.181 {D2/1/194}.
Mr Coyne knew that these “other systems” were (or included) the “Post Office back end
systems” and that it was using these systems that Post Office determined whether or not
to issue TCs: see Coyne 1/6.46.” He implicitly criticised Post Office for relying on
those systems — a criticism that he could not have made had he truly thought that Post
Office always referred to audit data when taking decisions on TCs.
Mr Coyne had also consulted the technical documents showing the Horizon system
architecture and the clear separation between, on the one hand, the data feed through the
POL00026925
POL00026925
718 ¢£2/1/9}.
79 E2/1/10}.
70 See Fn.18 at {D2/1/37}.
71 See Coyne 1/3.104 {D2/1/81}.
2 See para. 1.7 {D2/1/14}, para. 4.10, fin 15 {D2/1/35}, para. 4.23 {D2/1/38} para. 4.44, fn 24 {D2/1/41},
para, 4.53, fn 26 {D2/1/43} and the table at {D2/1/194}.
73 ¢D2/1/119}.
A/8/211
587.
588.
589.
POL00026925
POL00026925
TPS into Post Office’s back-office systems (including Credence and POLSAP) and, on
the other hand, the copying and archiving of data from the BRDB into the audit store.
Mr Coyne had considered, and referred to in Coyne 1, key documents that made clear
that the audit store was entirely separate from the data streams provided to Post Office:
24
see “Horizon Next Generation — Plan X” (September 2005)’** and “Horizon Online Data
Integrity Plan for Post Office Ltd” (March 2012).”5
Mr Coyne also had the technical documents that explain the manual process for obtaining
data from the audit store: see, for example, {F/542} (2009), {F/662} (2010) and
{F/1526} (2016).
Despite all this, Mr Coyne maintained in cross-examination that he had believed when
he drafted his first report that data from the audit store was available to and used by Post
Office for all reconciliation activities and all decisions in relation to TCs. He said that he
thought that Post Office could view data from the audit store “on screen or by pulling a
report”.”°> He even went so far as to say that he had thought that the audit database
formed part of Post Office’s MIS:
Q. Could I suggest to you, Mr Coyne, that you knew very well that the system that
Post Office used for the purposes of having what you describe as a quick look
were its management information systems. The hint is in the name, MIS, management
information systems, And you would expect, in the absence of being told to the
contrary, that a business such as Post Office would use its management information
systems for making business decisions of that sort?
A. And the audit database would be part of that management information system.”"
Mr Coyne was pressed on this remarkable suggestion. He insisted that he had believed
that the audit data was available to Post Office directly through an unidentified MIS:
Q. And what I'm suggesting to you is that there is no basis upon which you could ever
have thought that the information in that audit store could be regarded as a Post
Office management information system?
24 {F/299}. See, in particular, pp. 25 and 28, Referred to at Coyne I/fn 23 {D2/1/41} and fn 25 {D2/1/42}
8 {F/1424}. See, in particular, pp.13-14, Referred to at Coyne 1/ fn 29 {D2/1/45}
215}.
i
8
AI8/212
590.
591.
592.
593.
594.
POL00026925
POL00026925
A. I believed that it was a system that Post Office would look at whenever there was
a dispute about what happened at a branch counter_I believed that they would have
access to that.”*
It is implausible that Mr Coyne in fact believed any of this when he drafted his first
report, for three reasons.
First, as set out above, Mr Coyne knew at the time of his first report how the audit store
worked and, in particular, that it was separate from the day-to-day operations and Post
Office’s management information systems.
Secondly, if Mr Coyne had in fact believed that Post Office accessed the audit data
through some system operating within or alongside its MIS, he would not have felt able
to criticise Post Office for relying on (only) those systems to decide on TCs, which is
exactly what he did at Coyne 1/6.46.”” Mr Coyne confirmed at trial that his first report
raised this as a criticism.””
Thirdly, Mr Coyne accepted that, across the hundreds of technical documents and
thousands of Peaks and other documents that he had reviewed, there was no mention of
any “route by which information from the sealed audit store was made available on a
read only basis to Post Office”: see {Day15/86:12} to {Day15/86:25}. Mr Coyne
accepted that he had seen nothing to support the idea that there was some technical means
by which Post Office had direct access to the information in the audit store:
Q. But just to be absolutely clear, you had not and indeed you have not seen any
documents suggesting that Post Office had the ability to gain access to the audit store
on its own systems, had you? There was no design facility, there was no -- there were
no lines of communication between the audit store and Post Office in any document
you had ever seen, correct?
A. No, it looks as if the majority of the references to audit database access was from
Fujitsu personnel.”**
Mr Coyne cannot have believed that Post Office had read-only access to the audit store.
Under the pressure of live evidence, however, he resorted to saying whatever he had to
28 {Day15/86:4} to {Day15/86:11}.
79 (D2/1/119}.
780 {Day15/77:22} to {Day15/78:13}.
%1 {Day 5/87:15} to {Day15/87:23}.
AI8/213
POL00026925
POL00026925
732
say to defend the point that he had made in Coyne 2/1.2(c).’** Mr Coyne embroidered a
justification for what he had said in that short paragraph, rather than simply accepting
that it was wrong and that he had in fact known when he wrote Coyne I that Post Office
relied first on its MIS, obtaining the audit store data where appropriate through an ARQ
request.
The roles of MIS and ARQ requests
595. Mr Coyne accepted that it would not be “possible” to use the audit data to decide on each
and every potential TC.” Given how the audit system is designed, and the labour-
intensive process of data validation involved in extracting data from the audit store, it
would be practically impossible (and prohibitively expensive) to use the ARQ process
tens or hundreds of thousands of times per year.”*+
596. Despite this, and to retain his criticism of Horizon, Mr Coyne suggested that the audit
system could and should have been designed differently to allow “read only” access from
Post Office systems.”* He said that this would have been “quite simple”. As to that
suggestion:
596.1 This is not something that Mr Coyne said in either of his extremely long reports,
despite raising almost every possible criticism of the Horizon system. If he had
genuinely reached the view that the system suffered from some fundamental
design flaw that would have been “quite simple” to remedy, he would have
identified this in his reports. Instead, in his second report, Mr Coyne accepted that
the design principles of the audit system were “theoretically sound”.”°°
596.2 The Court has good reason to be careful when it comes to Mr Coyne’s evidence
on this topic given what he felt able to say about his knowledge at the time of
Coyne I.
m2 ¢D2/4.1/7}.
83 {Day15/80:13} to {Day15/80:21}.
74 (Day15/81:15} to {Day 5/81:22} and {Day15/87:24} to {Day15/88:11}.
785 {Day15/81:23} to {Day15/82:14}.
*86 Coyne 2/5.59 {D2/4.1/139}.
Al8/214
596.3 The purpose of the design of the audit system, and all the controls that it contains,
is to provide not only a pristine copy of the data received from counters but also
to be able to validate that data through various checks at the time of its
extraction.”"” The audit store has a discrete and important purpose, and that
purpose is not to provide a stream of data into MIS. But those systems, like the
audit store, take a copy of data from the BRDB.
596.4 It is self-evident that, given the basic design and purpose of the audit system, it
would be far from “quite simple” to establish read-only access to it from Post
Office’s systems. The audit store is housed in Fujitsu, with no physical connection
to other systems, and data can only be extracted from it through the manual process
described in Mr Dunks’ statement.”** This involves attending at specific locations
and executing commands on specific work stations. The audit store is, for very
good reason, kept separate from the ordinary data flows that inform day-to-day
operations. It is not a database — the audit data is stored in “flat” files and has to
be specially extracted (as is clear from the technical documentation).
597. It is easy to imagine the criticisms that Mr Coyne would have levelled at Horizon if it
598.
did not have very strict technical and physical controls over the audit store, in order to
make sure that the data could be validated to a very high degree of confidence. There is
no evidence that the degree of data security and validation attained through the audit
system deployed by Fujitsu could also be achieved through a system that permitted read-
only access through a live link.
Mr Coyne’s criticism therefore boils down to the suggestion that Post Office’s MIS were
somehow unreliable (such that using the MIS, instead of the audit data, created a risk of
error). That criticism cannot be sustained on the evidence:
598.1 The management information systems are fed a copy of the transaction data from
the BRDB. That is the same source as feeds into the audit system. The experts
POL00026925
POL00026925
287 These are detailed in Godeseth 1/30 {E2/1/9} and Dunks 1/6 {E2/10/2}.
788 {E2/10/2} at para.6.
AI8/215
POL00026925
POL00026925
have not identified even a single instance in which the process of feeding data from
the BRDB into the MIS has introduced error into the data.
598.2 There is no evidence of data in the MIS ever having been wrong or different from
the data in the audit store, over the course of 20 years and billions of transactions:
see above at para 531. Adopting a realistic approach, rather than imposing a
standard of perfection, there is really so basis for the attack on the use of MIS.
598.3 Ms Rose’s report indicates that it may sometimes be necessary to have regard to
ARQ data to identify the detail of some types of transaction (in that case, how the
reversal of the bill payment was initiated). The depth of investigation required will
of course determine the sources of information to which recourse must be had, and
cooperation is inevitably required in that process. Post Office should, in
accordance with its duties owed to SPMs, make use of ARQ requests where this is
appropriate.
598.4 With the exception of the point made in Ms Rose’s report, Mr Coyne has not
identified in his reports any information that would be important to Post Office’s
decision-making on TCs and that is not available on Post Office’s MIS. If he had
sought to identify any such information, he would have had to address the point
that Post Office used Credence alongside its other MIS 7’and in addition to read-
only access to branch information on the BRDB.”™"° He has made no attack on those
complementary systems or the assistance that they may provide.
599. Ultimately, Mr Coyne did not maintain his suggestion that it was inappropriate to use
Credence, at least in the first instance. In his first report, Mr Coyne stated that it was
“relevant to question why Post Office were using Credence data to initially investigate
disputed transactions”,”"' implying that even this initial use was inappropriate or at least
questionable in some way. At trial, when confronted with his written evidence on this,
Mr Coyne retreated immediately to a more nuanced position:
Q:... Your contention is that they should not use Credence to initially investigate, is
that right?
89 See Godeseth 1/48 {E2/1/13} and Mather/9-11 {E2/8/2}
1 See para. 601 below.
™! Coyne 1/5.176 {D2/1/102} and 9.9 {D2/1/150}.
AI8/216
POL00026925
POL00026925
A. It would seem that you can use Credence to conduct a cursory investigation but
you have to go back to the full logs to get the full picture. Because if there's a different
picture being given by Credence to that of the logs, then ultimately both can't be
correct.
Q. It is just your use of the word "initially". Is there any significance attached to
that? That's not what you should look at even first, you should look at something else
first, should you?
A, No, I mean, it depends what depth of investigation you are going to look. If it is
just a cursory investigation then Credence might be okay for that.”
600. The distance between that position and Post Office’s case is not huge. Post Office’s
601.
position is that the use of the Credence system is generally appropriate (and is always
the starting point) for investigating discrepancies, but that recourse should be had to the
data in the audit store where there is reason to believe that the transaction information
that it contains would shed light on the particular factual circumstances at issue (e.g. in
Mr Armstrong’s case), at least when interpreted by someone with the relevant expertise.
Mr Coyne did not go quite so far as to accept this, but he got close, saying that his point
was that “Credence data alone shouldn't be used. The ARQ will give the full picture of
what went on at the counter”.”
Mr Coyne also had no regard to the fact that, under Horizon Online, Post Office also has
direct read-only access to the transaction data in the BRDB, and it can use that access
for its business purposes: see Worden 1/1078" and Parker 1, para 14.5 This read-only
access complements the information available through the MIS and allows
straightforward cross-checking.
™2 (Day 5/34:11}.
™3 (Day15/67:18}.
™4 (D)3/1/238}.
™S {E2/11/3}.
AI8/217
602.
603.
604.
605.
606.
607.
POL00026925
POL00026925
Dr Worden’s evidence in relation ARQ requests and the audit store
The cost of an ARQ request
Dr Worden was cross-examined on the premise that Post Office might be dissuaded from
making ARQ requests by cost. It was put to him that each ARQ request costs around
£450. This was based on the document at {F/1092}.
As Dr Worden pointed out, there is in fact no incremental cost to each ARQ request
within the 720 available to Post Office under the relevant contract. Dr Worden explained
the position as follows:
Q. Well, let's take it in stages. What was put to Mr Coyne without any document or
any particular basis was the figure was over £200?
A. Well, I think -- I would imagine there are two different figures. In other words,
there is a bulk price for your first 720 and that works out. You have paid that already
and you can get 720 for that, and so the average of those is that much. Then if you
go beyond that, for each one you pay a different figure.”
Dr Worden’s answer was correct. This is clear from the documents to which Dr Worden
was referred.
First, {F/1092} states as follows, with added emphasis:
ARQ (Audit retrieval process) costs at least £384k recurring annually. This_is
subsumed without breakdown in the Fujitsu Security Management costs. 720 requests
@ £450 unit cost
It is clear from this that the “unit cost” is a derived estimate, based on a calculation from
an undifferentiated total cost that covers not only ARQ requests but also other security
management functions.”” It is not an incremental cost charged to Post Office for each
ARQ request that it makes.
Second, the email chain at {F/994.1} confirms these points:
™6 {Day 19/19:6} to {Day19/19:14}
™7 This is why dividing £384,000 by 720 docs not give £450 ~ the total figure includes other services (to
which some notional value has also been attributed), The question put at {Day19/20:23} was misconceived.
Al8/218
608.
609.
610.
607.1 At the top of page 3, Mark Hotson asked Mr Laycock (the author of the document
at {F/1092}) to explain the cost figure provided in that document.
607.2 At the top of page 2 (in an email copied into the response to Mr Hotson), Mr
Laycock explains that “The ARQ service...is also seemingly impossible to isolate
being subsumed into the “Security Management Service”... He goes on to
explain how he arrived at the estimate of £450 per ARQ request.
607.3 On the same page, Mr Laycock explains that he has been told that the contract
permits Post Office to increase its number of ARQ requests at a price of £207 per
request.
In this light, it was unfair to accuse Dr Worden of “making it up as [he] goes along” for
identifying the important difference between an incurred bulk cost and incremental unit
cost, in terms of any incentives acting on the person deciding whether to issue an ARQ
request.”** The documents show that he was right about the distinction.
Dr Worden was then taken to an email chain at {F/728} (dating from 2010, before the
documents above). The later emails in the chain concern whether to commence an
investigation into whether any criminal offences might have been committed in the
Barkham branch (Mrs Stubbs’ branch): see the second and final paras in the email from
Mark Dinsdale on page 2 and the email from Nigel Allen at the bottom of that page.””
At page 9, Mr Dinsdale confirmed that he would raise an ARQ request to obtain data in
relation to the branch. He then asked whether the request was for Post Office’s benefit
and noted that the cost was usually passed on to “the defence lawyers”. He also noted
that Post Office obtained a supply of ARQ requests “free of charge” but that these were
usually not enough.
POL00026925
POL00026925
78 (Day19/21:14} to {(Day19/21:24}.
™° Cs argued (wrongly) at the Common Issues Trial that this email showed that no investigation of any kind
had taken place, rather than an investigation (by the Investigations Division) into a potential criminal
offence. The emails relate, in large part, to a decision as to whether to commence an investigation into
potential criminal offences.
219
AI8/219
POL00026925
POL00026925
611. The Managing Judge helpfully indicated that the question of any disincentive created by
the cost of an ARQ request boiled down to a matter for submissions.”*° Post Office makes
four submissions:
611.1 The documents identify that there was no incremental cost for an individual ARQ
request in the event that Post Office exceeded the 720 provided under the
contract.”>! However, the contract provides that if Post Office chooses to exercise
its option to increase the maximum number of ARQs, the charges will increase by
a cost of £222.68 per ARQ request by which such maximum is increased.”
611.2 There is no evidence of Post Office ever having failed to make an ARQ request
because of the cost where it would otherwise have done so. The contention that
this might happen is essentially speculative. The experts and the legal teams have
examined and searched across many thousands of contemporaneous documents —
if there were such evidence, it would be before the Court.
611.3 The email chain at {F/728} shows Post Office obtaining audit data via an ARQ
request. It does not show (or event begin to suggest) that Post Office would not
have obtained the audit data in the absence of any of the special factors mentioned
in the email chain (e.g. interest from MPs). The fact that additional reasons existed
does not show that those additional reasons were necessary for the request to be
made in that case (or any case).
611.4 The evidence is that Post Office has made very different numbers of ARQ requests
across the years, reflecting that it makes such requests where it considers this
appropriate: the lowest figure is 103 (2016),”** and the highest is 765 (2014).”4
780 {Day19/31:19} to {Day19/32:13}.
751 The relevant part of the contract is at {F/1659.2/689}: see clause 5.5. The Court will recall that Dr Worden
did not think that he had seen this document. He may have seen the reference to the lower figure of £207
in the email chain at {F/994.1}.
732 {F/1659.2/689}.
753 See Godeseth 1, para. 31.2 {E2/1/10}. Note that the figure given for 2018 in para. 31.4 (364) was corrected
in correspondence to 366: see {H/301} (two entries in the spreadsheet had been miscounted).
784 £F/1848.20}.
A/8/220
612.
613.
614.
Unsurprisingly, the number has often been in the range 550-750,”> showing only
that Post Office was roughly correct in estimating its likely requirements.
It is important to recall the context in which this issue arises. Mr Coyne has not identified
a single instance in which Credence contained wrong data. There is no credible basis for
any suggestion that it was inappropriate to rely, for the most part, on the MIS, rather than
obtaining the data from the audit store.
The possibility of error in the audit store data
Neither Mr Coyne nor Dr Worden has identified any instance in which an ARQ
extraction went wrong in a way that undermined the reliability of the extracted data. It
is clear from Mr Dunks’ evidence that the process is carefully controlled.’*°
It was nonetheless implied in Dr Worden’s cross-examination that the audit store and/or
data extraction process were somehow unreliable. The basis for this suggestion was that
the Audit Extraction Client User Manual {F/1716} shows, at page 43, that the extraction
form will identify any gaps or duplicates in the extracted data. It was suggested that one
can infer from the fact that gaps and duplicates would be identified that they do occur.
Dr Worden dealt with this fairly:
Q. And the fair inference from the document is that gaps and duplicates do occur?
A. No. I think the fair inference from the document I that if gaps and duplicates occur
you should be concerned.
Q. So you are not prepared to accept that the fair inference from this document is
that gaps and duplicates do occur in the audit database?
A, Ido not think that's a fair inference. I think the fair inference is that they are not
supposed to and if they do you should seek assistance.
Q. So in fact even if you had seen this, it wouldn't have changed your view about gold
standard, would it?
POL00026925
POL00026925
758 The number of requests was in this range for the years ended 2005 (555) {F/1848.10}, 2007 (688)
{F/1848,12}, 2008 (720) {F/1848.13}, 2009 (719) {F/1848.14} and 2010 (577) {F/1848. 15}.
786 €E2/10}.
i
8
A/8/221
615.
616.
617.
POL00026925
POL00026925
A. No, it doesn't. I'm saying that gaps and duplicates are something to worry about,
i
therefore raise the alarm.”
A fair inference from the document would be that it is possible for gaps and duplicates
to occur. Cs again focus on the possibility of error, rather than whether errors occurred,
went unresolved and caused problems for branch accounts. It is common ground that
duplicates have sometimes been identified (and addressed). But this was sufficiently rare
that Mr Dunks had no specific recollection of having himself identified any duplicates
during the extraction process,’** although he was clear that, when this did happen, the
data would be passed on for investigation by the audit team, rather than being provided
to Post Office.””
Dr Worden went on to give evidence as to the possibility (under Legacy Horizon) for
duplicates to result from the use of two correspondence servers, but that such duplicates
should be resolved before the data is written to the audit store.” Cs again pressed the
point that there was a possibility of error. There is no evidence of duplicate entries in the
audit store ever having caused a problem in branch accounts.
F12. The 29 Bugs listed in JS2
The 29 Bugs
The Bug Appendix
As Post Office understands it, the 29 so-called bugs listed in the ‘Table of
Bugs/Errors/Defects with acknowledged or disagreed evidence of financial impact’ in
JS27°! (the “Bug Table”) sets out Cs’ case as to the number of branch-affecting bugs’
757 (Day19/44:10} to {Day19/44:24}.
8 {Day7/51:21} to {Day7/52:4}.
189 {Day7/52:22} to {Day7/53:9}
76 {Day19/47:20}. As regards Horizon Online, the 2010 KEL at {F/1512} evidences the operation of one of
the safeguards against duplicate entries being written to the BRDB.
76! The table starts at {D1/2/3}.
78 The term “branch-affecting bugs” is used as a shorthand reference to bugs which have created one or more
false shortfalls in branch accounts.
i
8
8
AI8/222
POL00026925
POL00026925
that have arisen during the life of Horizon. These alleged bugs are addressed in Appendix
2. This Section should be read alongside that Appendix.
618. Far too many documents are referred to in JS2 for it to be possible to address them all,
but the main documents are considered in the Appendix. This section addresses some
wider issues regarding the 29 alleged bugs and a short summary of Post Office’s
contention on each of them.
Finding and Analysing Bugs
619. Both experts rightly looked for bugs.’ However, as discussed elsewhere in these
Closing Submissions,” Mr Coyne’s approach was simply to look for bugs without
giving a proper sense of scale, context or even balance.
620. This was the wrong approach. For example, it meant that Mr Coyne hardly considered
the role and effect of Horizon’s countermeasures and that he did not really consider the
question whether bugs had lasting or merely transient effects, or at least not until he was
cross examined. It also meant that in many cases Mr Coyne’s interpretation of KELs
and Peaks could not be described as reliable or complete.
621. Countermeasures are addressed in greater detail elsewhere in these Closing
Submissions.”® Their significance lies not only in relation to the general question of
robustness but also in relation to the question whether bugs would be caught by the
system and their consequences remedied in the ordinary course, often without any need
for an SPM to report an issue.
622. On the question of remedying consequences, Mr Parker’s unchallenged evidence in
Parker 3 should be borne in mind. He said:
It is part of the SSC's standard diagnostic process that, when we have identified a bug,
we also seek to identify all areas of the system affected by that bug. Where the bug has
caused discrepancies in branch accounts so as to create an incorrect shortfall or
surplus, we take steps to identify all branches that have been impacted. This is our
763
764
165
See for example where Dr Worden states that he was looking for bugs {Day19/128:20} to
{Day19/128:25}.
Sce seetion Ad-of thes
See section F8 of these submissions.
AI8/223
623.
624.
625.
626.
standard practice and to the best of my knowledge it has always been done since
Horizon was first introduced.”
Consistently with this, in oral evidence Mr Coyne agreed that Fujitsu were good at
identifying issues and making Post Office aware.”°7
The Bug Table
In oral evidence, Mr Coyne confirmed that the Bug Table is a definitive list of bugs for
which he says there is evidence of financial impact and that he is not relying on any other
bugs.”°*
Of the 29 bugs, Post Office contends that:
625.1 eight were not bugs at all;”°?
625.2 three had no branch impact;””
625.3 nine had only transient impact;””! and
625.4 nine had the potential to cause lasting impact, but were in fact detected and
resolved.”
Post Office notes that the experts agreed at JS2, para 1.2 that the bugs “in rows J, 2, 3,
10, 13, 14, 18, 23, 24, 25, 27 and 28 may have had financial impact on branch
accounts.”"”* The experts further agreed at para 1.15 that there were between 12 and 29
bugs with evidence of lasting financial impact.’ Those 12 bugs are not specified,
POL00026925
POL00026925
166 £2/13/1}, para 3.
767 Day14/92:4} to {Day14/93:11}.
78 {Dayl7/15:11} to (Day17/15:17}.
%9 Bug 8: Recovery Issues; Bug 13: Withdrawn Stock Discrepancies; Bug 15: Phantom Transactions; Bug
16: Reconciliation Issues; Bug 17: Branch Customer Discrepancies; Bug 20: Recovery Failures; Bug 23:
Bureau de Change and Bug 29: Network Banking.
7 Bug 11: Girobank; Bug 21: Transaction Corrections and Bug 22: Bugs/Errors Defects introduced by
previously applied Peak Fixes.
™ Bug 2: Callendar Square; Bug 4: Dalmellington; Bug 5: Remming In; Bug 6: Remming Out; Bug 7: Local
Suspense; Bug 9: Reversals; Bug 12: Counter Replacement; Bug 19: Post & Go/TA Discrepancies and Bug
25: Lyca top up
™2 Bug 1: Receipts and Payments Mismatch; Bug 3: Suspense Account; Bug 10: Data Tree Build; Bug 14:
Bureau Discrepancies; Bug 18: Concurrent Logins; Bug 24: Wrong branch customer; Bug 26: TPSC250;
Bug 27: TPS and Bug 28: Drop and Go
73 (D1/2/27}.
74 (D1/2/19}.
AI8/224
POL00026925
POL00026925
although it is to be inferred that para 1.2 identified the 12 bugs which were agreed to
have lasting financial impact.
627. In relation to three of these bugs (Bugs 2, 13 and 23 — Callender Square, Withdrawn
Stock Discrepancies and Bureau de Change), Post Office does not accept that, on the
balance of probabilities, these were bugs which had lasting financial impact:
627.1 Regarding Bug 2: Callendar Square, the underlying documents show that instances
of the relevant Riposte issue were picked up automatically and resolved.””° Any
branch impact was transient and not lasting.
627.2 Bug 13: Withdrawn Stock Discrepancies is not a bug. The relevant documents
show that withdrawn stock discrepancies arose as a result of human error. In row
13 of the Bug Table Dr Worden pointed out that “/s/ince the discrepancies in
branch accounts appear to be both temporary, and caused by human error, these
39776
are not a case of a bug in Horizon causing lasting effects on branch accounts.
The Peaks confirm this.
627.3 Bug 23: Bureau de Change is also not a bug. Three distinct issues are raised under
the heading of Bug 23 and the documents show that cach one of them is an example
of user error. PC0129767"” (the first issue) shows an SPM carrying out a reversal
incorrectly; PC01374377"s (the second issue) is even described by Mr Coyne as
user error in row 23 of the Bug Table; and PCO151787"” (the third issue) shows
an SPM making incorrect cash declarations.
Who identified the 29 bugs?
628. Mr Coyne stated during cross-examination that “Dr Worden only identified the three bugs
that had already been identified.””® This claim is incorrect.
775 Dr Worden agreed during re-examination that there were lots of ways in which the Riposte bug would be
identified through automatic reporting. See for example {Day20/178:1} to {Day20/179:!}.
76 ¢D1/2/12}.
7 EBIB.1.
78 §F/346.1}.
79 {F/430.1}.
*80 {Day14/68:3} to {Day14/68:4}.
AI8/225
POL00026925
POL00026925
629. In its Letter of Response dated 28 July 2017,”*! Post Office identified the three known
bugs: (i) Receipts and Payments Mismatch; (ii) Callendar Square; and (iii) Suspense
Account. During this trial, Cs have repeatedly sought to criticise Post Office for not
revealing that there were other bugs, too. This criticism is unfair, since the Letter of
Response made it clear that Post Office was not claiming that these were the only bugs
782
that had arisen.’”** In any event, of the further 26 bugs listed in the Bug Table, nine of
them were identified in Worden 1, namely:
629.1 Bug 5 Remming In.”
629.2 Bug 6 Remming Out.’
629.3 Bug 7 Local Suspense Issue.’*°
629.4 Bug 23 Bureau de Change.”*°
629.5 Bug 24 Wrong branch customer change.
629.6 Bug 25 Lyca top up.’**
629.7 Bug 26 TPSC250 report.’
629.8 Bug 27 TPS.”
629.9 Bug 28 Drop & Go.”!
787
F13. Mr Coyne’s Views on Branch Impact
630. Mr Coyne stated during cross examination that he had assessed the number of branches
affected by the alleged 29 bugs and had set this out in the Bug Table.”
1 2}.
782 See {H/2/95}.
%3 (D)3/2/170}.
784 {1)3/2/165}. Mr Coyne also mentioned this bug his first report {D2/1/216}, but he did not suggest that it
had financial impact.
*85 (D3/2/173}.
786 {1)3/2/176} and {D3/2/180}.
%87 {3/2/17}.
188 {D3/2/186}.
%89 (D3/2/175}.
7 ¢D)3/2/187}
71 ¢D3/2/191}.
%2 {Day15/105:7} to {Day15/105:15}.
AI8/226
631. The column in the Bug Table entitled “Coyne's opinion as to branch account impact” sets
out his view on that question.”* Post Office notes that when one examines the Peaks and
incident figures identified in that column, there are 546 entries.””* Post Office further
notes that this does not come close to the thousands of impacts suggested by Mr Coyne”’>
or the hundreds of thousands of impacts which would be required even to have a chance
of generating the losses alleged by Cs.7°
Lasting Versus Transient Impact
632. The distinction between bugs with lasting or transient impact is important. Issues that
Fujitsu are flagging and investigating automatically do not last. It is apparent from Mr
Coyne’s answers during cross-examination that he had given little, if any, considered
thought to the question of lasting impact when preparing his reports.
633. However, when the point was raised with him, Mr Coyne agreed that bugs can cause two
kinds of discrepancies — transient and lasting. The difference is that transient
discrepancies are handled by countermeasures. There may be a period of time where
there is a shortfall, but in the long run the countermeasures will ensure that the
discrepancy is resolved.”””
634. Mr Coyne said himself that “you would want a system to fail safely. So it is acknowledged
that there could be failures and it is what you do when that system fails.””* Horizon is
robust not only by reference to the limited number of bugs that the experts have identified
but also by reference to what it did to “fail safely”. One of the things it does is operate
systems of automatic reporting.
POL00026925
POL00026925
3 {Day15/130:9} to {Day15/130:18}.
74 Calculation: 60 for Bug 1, 30 for Bug 2, 14 for Bug 3, 112 for Bug 4, 14 for Bug 5, 58 for Bug 6, 58 for
Bug 7, 4 for Bug 8 [the 2,473 Peaks which Mr Coyne stated may or may not be examples have not been
included in this calculation], 2 for Bug 9, 3 for Bug 10, 8 for Bug 11, 4 for Bug 12, 60 for Bug 13, 1 for Bug
14, 2 for Bug 15, 6 for Bug 16, 1 for Bug 17, 2 for Bug 18, 1 for Bug 19, 2 for Bug 20, 10 for Bug 21, 5 for
Bug 22, 2 for Bug 23, 3 for Bug 24, 3 for Bug 25, 24 for Bug 26, 40 for Bug 27, I for Bug 28 and 1 for Bug
29.
5 See, fore
6 See para. 460 et seq.
77 {Day17/104:3} to {Day17/104:19} and {Day17/108:1} to {Dayl7/108:11}.
8 {Day14/26:16} to (Day14/27:2}.
AI8/227
635.
636.
637.
638.
Automatic reporting
Horizon’s countermeasures are discussed elsewhere in these Closing Submissions”? and
its automatic reporting systems are discussed at Appendix 3. These systems mean that
bugs and other issues will often be picked up and sorted out without the need for an SPM
to report them.
There are a variety of different automatic reports. For example, the NB102 report*”’ and
the TPS report set (which includes the TPSC250 Host Detected Transaction errors
report,*”! the TPSC254 Harvester Exceptions®” and the TPSC257 POLFS Incomplete
) 808
Summaries Report).*** These reports identify not only manifestations of bugs in Horizon
but also discrepancies caused as a result of user error.
Mr Coyne agreed that sometimes an SPM may report an issue, and therefore a Peak may
be raised, in circumstances where a Peak would automatically have been raised shortly
thereafter:
Q. But sometimes -- and perhaps we can agree on this. I have thought about that as
well and my surmise is that sometimes the SPM gets the report in first and other times
the MSU does, it is just a timing issue.
A. Yes.
Q. And if the SPM has phoned in first then that is the PEAK and there doesn't need
to be a separate PEAK because the MSU puts in a call as well, would you say that's
fair?
A. Sounds reasonable.8*
Thus, where a Peak is recorded as being commenced by an SPM call, it should not be
assumed that the SPM’s call was necessary. For example, all instances of the Callendar
Square and Dalmellington issues would have been automatically picked up and then
resolved regardless of whether or not the SPM reported the issue.
POL00026925
POL00026925
™® See section F8 of these submissions.
80 See para. 2-5
01 See para. 4
3 See para. 6-2.
f Appendix 3.
eg of Appendix 3.
of Appendix 3.
of Appendix 3.
84 (Day17/78:14} to {Day17/78:22}.
AI8/228
639.
640.
641.
642.
643.
644.
POL00026925
POL00026925
F14, Mr Coyne’s View on Bugs with Lasting Impact — A Changing Landscape
In JS2, para 1.15 the parties’ experts agreed that:
“The number of distinct bugs, for which the experts have seen strong evidence of the
bug causing a lasting discrepancy in branch accounts, is between 12 and 29.”8°
Post Office understood this to mean that Mr Coyne was of the view that there were 29
bugs causing lasting discrepancies. It appears that Cs were also of the same view: their
opening submissions stated that:
“between 12 (Dr Worden) and 29 (Mr Coyne) bugs with “strong evidence of the bug
causing a lasting discrepancy in branch accounts” 8°
During his cross examination, Mr Coyne was asked a simple question: how many bugs
have lasting impact? His answer to that question changed four times.
Mr Coyne’s First View: 29 Bugs with Lasting Impact
On his first day of cross examination (Day 14), Mr Coyne confirmed that, in his opinion,
there was strong evidence of lasting impact in the case of 29 bugs:
Q. So what you are saying is for each of the 29 bugs there is strong evidence that the
bugs concerned not only caused discrepancies but they caused discrepancies which
were lasting, which weren't handled by countermeasures. You see? That's your expert
opinion, isn't it?
A. Yes.5”
This response seemed clear. But on Day 17, confusion set in.
Mr Coyne’s Second View: 13 Bugs with Lasting Impact
In response to a question from the Judge just before the short adjournment on Day 17,
Mr Coyne stated that there were only 13 bugs with lasting branch impact.*” This came
as a great surprise to Post Office.
5 (D1/2/29}.
806 {A/1/7}
®7 {Day17/104:20} to {Day17/104:25}.
®8 {Day17/106:6} to {Day17/106:16}.
AI8/229
645.
646.
647.
648.
649.
POL00026925
POL00026925
Mr Coyne’s Third View: 21 Bugs with Lasting Impact
To Post Office’s even greater surprise, after the short adjournment Mr Coyne changed
his opinion again. He stated that in fact he was of the view that there were 21 bugs with
lasting branch impact.
A. Good afiernoon. Sorry, Mr de Garr Robinson, before we start could I please
correct the number I provided just before lunch. In tallying up the numbers in the
column in my report I missed that the last one is actually a heading that includes a
number of bugs, errors and defects, so the actual number is 21. They are in the report,
they are just grouped together for the purposes of the table.°””
As Mr Coyne confirmed, this meant that he took the view that eight of the 29 bugs did
not have lasting branch impact.*!”
Mr Coyne’s Reasons for Changing his Opinion from 13 to 21 Bugs
Mr Coyne tried to explain this change by stating that, in Coyne 2, Bug 22 included a
subset of bugs, bugs 23 to 29, which he had forgotten to include in his calculation before
the short adjournment. In response to a series of questions, he said that Bug 22 was
“actually a group heading, and in the report there's seven bugs, errors and defects
included in that heading.’*"' Later, he confirmed again that Bug 22 covered a host of
different issues.*!*
When pressed on this point Mr Coyne responded:
Q. You are saying that is an aspect of bugs, errors and defects introduced by
previously applied PEAK fixes, are you? Because it is plainly not right, Mr Coyne.
What are you saying?
A. Sorry, I'm drawing a blank now what the cross reference is for these .... Sorry, 1
would need to search for that PEAK reference under the Coyne impact column against
23, bureau de change, just to see where that features.°°
Mr Coyne therefore accounted for this further change, from 13 to 21 bugs which have
lasting financial impact, on the basis that the bugs discussed in Coyne 2 under the
8° {Dayl7/106:24} to {Day17/107:6}.
© {Dayl7/110:2} to {Day17/110:12}.
811 (Day17/107:7} to {Day17/107:25}
812 {Dayl7/118:7} to {Dayl7/118:13}.
813 {Dayl7/119:6} to {Day17/119:13}.
A/8/230
650.
651.
heading “bugs, errors and defects introduced by previously applied PEAK fixes”
encompassed eight distinct bugs, and that these bugs had become Bugs 22 to 29 in the
Bug Table.
This reasoning cannot be right:
650.1 The Peaks referred to in Bugs 23 to 29 are different from the Peaks referred to
under the heading “bugs, errors and defects introduced by previously applied
PEAK fixes”.
650.2 Bugs 23 to 29 are not all bugs/errors/defects introduced by previously applied Peak
fixes.
650.3 On Mr Coyne’s account, Bugs 23 to 29 do not all have lasting branch impact. In
his view, Bug 29 (Network Banking) has a transient effect.
Mr Coyne’s Final View: 22 Bugs with Lasting Branch Impact
Ultimately, Mr Coyne expressed the opinion that there were 22 bugs with lasting branch
impact.*'4 These are:
651.1 Bug 1: Receipts and Payments Mismatch.
651.2 Bug 2: Callendar Square.
651.3 Bug 3: Suspense Account.
651.4 Bug 4: Dalmellington.
651.5 Bug 5: Remming In.
651.6 Bug 6: Remming Out.
651.7 Bug 7: Local Suspense.
651.8 Bug 8: Recovery Issues.
651.9 Bug 9: Reversals.
651.10 Bug 10: Data Tree Build.
651.11 Bug 11: Girobank.
651.12 Bug 12: Counter Replacement.
651.13 Bug 13: Withdrawn Stock Discrepancies.
651.14 Bug 14: Bureau Discrepancies.
POL00026925
POL00026925
84 {Day17/135:2} to {Day17/135:8}.
A/8/231
652.
653.
654.
655.
POL00026925
POL00026925
651.15 Bug 18: Concurrent Logins.
651.16 Bug 22: Bugs/Errors/Defects introduced by previously applied Peak Fixes.
651.17 Bug 23: Bureau de Change. Agreed financial impact on branch accounts.
651.18 Bug 24: Wrong Branch Customer. Experts agree that this bug may have had
financial impact on branch accounts.
651.19 Bug 25: Lyca Top Up.
651.20 Bug 26: TPSC250.
651.21 Bug 27: TPS.
651.22 Bug 28: Drop and Go.
It follows that the other 7 bugs in the Bug Table had only transient impact.
It took over an hour to extract this opinion.*'> This extraordinary exercise meant that Post
Office lost valuable cross examination time which it would otherwise have devoted to
significant bugs.
In the course of this exercise, Mr Coyne relied on a distinction drawn in Coyne 2 between
Horizon Issue 4 and Horizon Issue 1.8'° However, he maintained that Bug 18 is a bug
with lasting financial impact, which is inconsistent with the treatment he gave it in Coyne
2. There, he stated not only that it was relevant to Horizon Issue 4 but also stated that
there was no financial impact at all, lasting or otherwise.*!7
Mr Coyne’s account of his change in opinion from 13 to 21 to 22 bugs with lasting
branch impact is astonishing. It lacks any coherence or cogent explanation. Putting it at
its lowest, it suggests that, until he was asked about the matter in cross examination, Mr
Coyne had given very little, if any, attention to the question whether bugs were or were
not likely to have a lasting effect.
F15. Alleged Bugs which are not bugs at all
It is Post Office’s case that the following alleged bugs are not bugs at all:
815 Mr Coyne was asked to clarify his position first at 12:55 {Day17/101:20} and it took until 15:00 for Mr
Coyne to ultimately explain his change in figures {Day17/140:10}
816 See for example {Day17/123:4} to {Day17/123:18}.
87 ¢D2/4.1/18}.
8
6
AI8/232
656.
657.
658.
659.
660.
661.
655.1 Bug 8: Recovery Issues.
655.2 Bug 13: Withdrawn Stock Discrepancies.
655.3 Bug 15: Phantom Transactions.
655.4 Bug 16: Reconciliation Issues.
655.5 Bug 17: Branch Customer Discrepancies.
655.6 Bug 20: Recovery Failures.
655.7 Bug 29: Network Banking.
655.8 Bug 23: Bureau de Change.
Bug 8: Recovery Issues
This bug is addressed in detail beginning on page 436-
Mr Coyne asserts that it is a bug with lasting financial impact. Post Office submits that
it is not a bug at all. Further, it notes that this heading in fact covers two distinct issues.
There is no way to avoid the risk of a basket of transactions being interrupted during the
course of dealing with a customer. For example, there may be a communications failure
or a power cut. Horizon was designed to deal with these situations, via its recovery
processes. Inevitably, some of these processes have a manual element and involve
Fujitsu. The system ensures that all potential recovery issues are flagged in automatic
reports.
Bug 13: Withdrawn Stock Discrepancies
This bug is addressed in detail beginning on page 465466.
Mr Coyne asserts that it is a bug with lasting financial impact. Post Office submits that
it is not a bug at all. In the Bug Table, Dr Worden makes clear that this issue arose as a
result of human error and not as a result of a bug in Horizon.*!®
Bug 15: Phantom Transactions
This bug is addressed in detail beginning on page 4:
POL00026925
POL00026925
818 ¢D1/2/12}.
AI8/233
662.
663.
664.
665.
666.
667.
668.
669.
670.
671.
Mr Coyne asserts that it is a bug with non-lasting financial impact. Post Office contends
that this is not a bug at all. It further notes that this heading covers three distinct issues.
Manifestations of this alleged bug are either design features of Horizon or user error.
Bug 16: Reconciliation Issues
This bug is addressed in detail beginning on page +8
Mr Coyne asserts that it is a bug with non-lasting financial impact. Post Office contends
that this is not a bug at all. Again, six distinct issues fall under this heading. One was a
Peak that arose in testing, and not the live environment.
Bug 17: Branch Customer Discrepancies
This bug is addressed in detail beginning on page
Mr Coyne asserts that it is a bug with non-lasting financial impact. Post Office contends
that this is not a bug at all. There is no evidence of a bug in Horizon and in any event
instances of any issue were caught by automatic reporting.
Bug 20: Recovery Failures
This bug is addressed in detail beginning on page 4°
Mr Coyne asserts that it is a bug with non-lasting financial impact. Post Office contends
that this is not a bug at all. This is once again three distinct issues under one heading.
There is no evidence of a bug in Horizon and in any event instances of any issue were
caught by automatic reporting.
Bug 23: Bureau de Change
This bug is addressed in detail beginning on page 3&
Mr Coyne asserts that it is a bug with lasting financial impact. As explained above,
although Dr Worden agreed that this may have financial impact on branch accounts, the
evidence suggests that there was in fact no bug in Horizon. Three distinct issues are
raised under this heading and each involves user error as opposed to a bug in Horizon.
Bug 29: Network banking
This bug is addressed in detail beginning on page
POL00026925
POL00026925
AI8/234
672.
673.
674.
675.
676.
677.
Mr Coyne asserts that it is a bug with transient financial impact. Post Office submits that
there is no evidence of a bug in Horizon. The heading encompasses two issues. Neither
of them can properly be described as instances of a “Network Banking Bug”; both stem
from intermittent communications failures emanating from outside Horizon (most likely
from systems operated by BT).
F16. Bugs with no impact on branch accounts
It is Post Office’s case that the following bugs had no branch impact:
673.1 Bug 11: Girobank.
673.2 Bug 21: Transaction Corrections.
673.3 Bug 22: Bugs/Errors Defects introduced by previously applied Peak Fixes.
Bug 11: Girobank
This bug is addressed in detail beginning on page +.
Mr Coyne asserts that it is a bug with lasting financial impact. Post Office submits that
there is no financial impact, let alone a lasting impact. Six issues come under this
heading. None of the Peaks referred to by Mr Coyne demonstrate a direct financial
impact on branches; in most cases this is because the issue affects the print out reports
available whilst the underlying data remains unaffected. The first issue reflects Horizon
working as intended (and is therefore not a bug) and the sixth issue is pure user error.
Bug 21: Transaction Corrections
This bug is addressed in detail beginning on page 5¢
Mr Coyne asserts that it is a bug with lasting financial impact. Post Office contends that
there is no evidence of any financial impact. This heading encompasses three issues. The
first issue was caught in testing and so did not arise in the live environment. The second
issue resulted in the screen freezing when an SPM wished to accept a TC, which resulted
in no discrepancy in branch accounts. The third issue related to TC reporting and resulted
in no discrepancy in branch accounts.
wy
POL00026925
POL00026925
AI8/235
POL00026925
POL00026925
Bug 22: Bugs/Errors Defects introduced by previously applied Peak Fixes
678. These bugs are addressed in detail beginning on page
679. Mr Coyne asserts that they had lasting financial impact. Post Office contends that there
is no evidence of any financial impact. Three distinct issues are raised. A number of the
Peaks relied on arose only in testing. The Peaks that arose in the live environment do not
contain evidence of branch impact.
F17. Bugs with transient impact
680. It is Post Office’s case that the following bugs had transient impact:
680.1 Bug 2: Callendar Square.
680.2 Bug 4: Dalmellington.
680.3 Bug 5: Remming In.
680.4 Bug 6: Remming Out.
680.5 Bug 7: Local Suspense.
680.6 Bug 9: Reversals.
680.7 Bug 12: Counter Replacement.
680.8 Bug 19: Post & Go/TA Discrepancies.
680.9 Bug 25: Lyca Top Up.
Bug 2: Callendar Square
681. This bug is addressed in detail beginning 9 +
sb-parmgnink Noahs
682. Mr Coyne asserts that it is a bug with lasting financial impact. Post Office submits that
this bug resulted in transient impact only. Fujitsu’s automatic reports identified instances
of the Callendar Square bug without the need for each SPM to call in with a report. The
evidence shows that as and when issues arose these were flagged automatically in
Fujitsu’s reports and addressed.
Bug 4: Dalmellington
683. This bug is addressed in detail beginning on page +44
AI8/236
684,
685.
686.
687.
688.
689.
690.
Mr Coyne asserts that it is a bug with lasting financial impact. Post Office contends that
any discrepancies were transient. Instances of this bug were automatically picked up by
Fujitsu. Fujitsu identified 112 potential occurrences of the Dalmellington bug. Of these
112 potential occurrences, 108 items were corrected at the time, either by Post Office
issuing a Transaction Correction or the SPM reversing the duplicate Rem In.8!? Mr
,820
Coyne agreed that these 108 instances were picked up and there was no lasting impac
He stated that “.../ think once everything was detected everything was made good.”**!
With regard to the other four instances, two involve de-minimis discrepancies, namely
£1.00 and £0.01.8? Mr Coyne accepted the chances of these discrepancies being
unresolved — i.c. not made good by Post Office — were “very small”.'?> The two larger
sums were investigated and found not to be instances of the Dalmellington bug and not
to have caused any loss.**+
Bug 5: Remming In
This bug is addressed in detail beginning on page 422424.
Mr Coyne asserts that it is a bug with lasting financial impact. Post Office contends that
any discrepancy would be transient. Remming issues are monitored and automatically
picked up.
Bug 6: Remming Out
This bug is addressed in detail beginning on page 4
Mr Coyne asserts that it is a bug with lasting financial impact. But it raises two separate
issues. Post Office submits that any discrepancy would be transient. Remming issues are
monitored and are automatically picked up.
Bug 7: Local Suspense
This bug is addressed in detail beginning on page 43-
POL00026925
POL00026925
89 (F/1415/3}.
0 (Day15/159:23} to {Day]5/160:12}
©I (Dayl 5/1632}.
82 (F/1415/7} to {F/1415/8}
®3 (Dayl7/161:14}.
84 (F/1427.1/2}.
AI8/237
691.
692.
693.
694.
696.
697.
698.
699.
Mr Coyne asserts that it is a bug with lasting financial impact. Post Office submits that
any discrepancy would be transient. The bug prevented affected branches from rolling
over and was bound to come to Fujitsu's attention very quickly, as indeed it did.
Bug 9: Reversals
This bug is addressed in detail beginning on page +
This bug arose very carly in the life of Horizon Online, during the pilot scheme in which
it was rolled out to a limited number of branches. Mr Coyne asserts that it is a bug with
lasting financial impact. Post Office submits that any discrepancy would not be lasting.
A discrepancy would appear as a remming issue and would be captured by automatic
reporting.
Bug 12: Counter Replacement
9461,
This bug is addressed in detail beginning on page 45:
5. Mr Coyne asserts that it is a bug with lasting financial impact. Post Office submits that
any discrepancy would not be lasting. All instances of the bug would have been flagged
with Fujitsu as a receipts and payments mismatch and with the SPM when rolling
over.
Bug 19: Post & Go/TA Discrepancies
This bug is addressed in detail beginning on page 4
Mr Coyne asserts that it is a bug with lasting financial impact. Post Office submits that
any discrepancy would be transient. Any issues were picked up automatically by
Fujitsu’s automatic reporting.
Bug 25: Lyca top up
This bug is addressed in detail beginning on page $1.65 18.
Mr Coyne asserts that this bug — a reference data bug — had the potential for lasting
financial impact. Post Office contends that any impact was transient. This issue was
identified through Fujitsu's automatic reporting — the NB102 report.**> It was also visible
POL00026925
POL00026925
©5 PC0203215 {F/697.1} and PCO203284 {F/691.2}.
Al8/238
to the SPM in branch as they would have been logged out of the counter and gone through
the recovery process. It would have also been possible to identify whether or not a
transaction had been reversed from the reports available in branch.*”° It was quickly
resolved, with the reference data fix being issued in 7 days of the issue being reported to
Fujitsu.
F18. Bugs with lasting impact
Summary
700. Post Office accepts that the following are bugs with potential lasting impact (although
they were in fact resolved):
700.1 Bug 1: Receipts and Payments Mismatch.
700.2 Bug 3: Suspense Account.
700.3 Bug 10:
700.4 Bug 14:
700.5 Bug 18:
700.6 Bug 24:
700.7 Bug 26:
700.8 Bug 27:
700.9 Bug 28:
Data Tree Build.
Bureau Discrepancies.
Concurrent Logins.
Wrong branch customer.
TPSC250.
TPS.
Drop and Go.
Lasting does not mean unresolved
701. At the outset it should be noted that “lasting” does not mean that SPMs were not made
good or that a fix was not implemented. As set out above, “lasting” means that
manifestations of a bug would not inevitably be caught by countermeasures. Post Office
would of course prefer all bugs to be picked up automatically. However, it is inevitable
that there will be occasions where this does not happen as it is not possible to build a
system that foresees and automatically prevents every anomalous action; this would
require an unachievably perfect system.
POL00026925
POL00026925
©6 PCO203108 {F/694} and PC0202925 {F/692}.
239
AI8/239
702.
703.
704.
705.
706.
707.
708.
709.
710.
7.
These bugs and their impacts were resolved. They were identified, fixes were
implemented and manual reconciliation took place.
Bug 1: Receipts and Payments Mismatch
This bug is addressed in detail beginning on page 486
Fujitsu. monitors for receipts and payments mismatches and investigates such
occurrences. Fujitsu ultimately identified all instances of this bug, which were collected
through their own reports without the need for an SPM to report an issue.
Bug 3: Suspense Account
This bug is addressed in detail beginning on page 44-2
Fujitsu were able to identify all affected branches by reference to historical data. A fix
was implemented which meant that the issue would not reoccur.
Bug 10: Data Tree Build
This bug is addressed in detail beginning on page
This bug arose early in the life of Legacy Horizon. It occurred rarely and although it had
the potential to cause a lasting branch impact this would have required a specific
sequence of events to have occurred in addition to the bug occurring. The issue was
detected and fixed. Only three affected branches have been identified.
Bug 14: Bureau Discrepancies
This bug is addressed in detail beginning on page
There are two distinct issues under this heading. With regards to the first issue, the branch
was made good and a fix was implemented. The second issue was not a bug in Horizon
nor an issue which could have impacted branch accounts; it created what was essentially
a cash flow problem for the branch.
Bug 18: Concurrent Logins
This bug is addressed in detail beginning on page 487489,
POL00026925
POL00026925
A/8/240
712.
713.
714.
715.
716.
717.
718.
POL00026925
POL00026925
There are two distinct issues under this heading. No discrepancies are set out in the Peaks
referred to by the experts. However, if a discrepancy had occurred, that discrepancy
would manifest itself as a receipts/payments mismatch.
Bug 24: Wrong Branch Customer Change
This bug is addressed in detail beginning on page $43
It is a reference data bug. The experts have agreed that, once discovered, reference data
bugs could be quickly fixed (by a change to the reference data).*”’ This was the case with
Bug 24. The issue would have been visible to the SPM as the incorrect quantity would
have displayed on the screen. Fujitsu identified the root cause and developed a fix within
two weeks of the issue being reported by the SPM.
Bug 26: TPSC250
This bug is addressed in detail beginning on page $4952).
The experts have drawn together five distinct issues under the heading of “7PSC250
report’. It is a backend reporting problem and so, if there were any chances of branch
impact at all, those chances would have been very small and indirect (e.g. if they caused
existing branch discrepancies to be missed by Fujitsu or they caused erroneous TCs to
be issued and accepted).*** Of the five issues: the first resulted in incorrectly flagged
exceptions (but no reconciliation was required); the second was limited to the process of
copying/ harvesting transactions to Post Office's back-end systems; the third did not
result in a mismatch between the files sent to Post Office and the branch data; the fourth
was flagged by automatic reporting and the fifth could result in a receipts and payments
mismatch being missed by Fujitsu.
Bug 27: TPS
This bug is addressed in detail beginning on page 5.
It is a backend reporting problem and so, if there were any chances of branch impact at
all, those chances would have been very small and indirect.*” Fujitsu fully investigated
"7 (D1/4/7} para 4.4
8 {1/2/24}.
9 (D1/2/24}.
Al8/241
719.
720.
721.
722.
each of these instances and checked to see whether there had been any impact on the
affected branches. The guidance in KEL ballantj2547K** enables Fujitsu to identify any
further instances of the issue and ensure there is no impact on branch accounts.
Bug 28: Drop and Go
This bug is addressed in detail beginning on page
It is a reference data bug. As set out above, the experts have agreed that, once discovered,
such bugs should be quickly fixed (by a change to the reference data) once the bug is
correctly identified.‘*! This was the case with Bug 28.
F19. Pre-Announcements Interface Defect
During Dr Worden’s cross-examination, his attention was drawn to this issue which was
a defect which came to light following service of both of the experts’ reports. This is a
defect which was automatically detected and had no lasting impact on branch accounts.
On 4 March 2019 an incident was raised regarding manual entry of the value of pouches
of cash that were to be remmed into a branch.**? There were failures in the pre-
announcement interface to Horizon.**? This meant that instead of being able to scan the
barcode, branches had to manually key the value in on receipt.** As a result, information
was not automatically inputted into Horizon. When keying the value in, some SPMs
made manual keying errors and so keyed the wrong value into Horizon.**°
723. It will thus be seen that this bug did not create false discrepancies in any branch accounts.
It created a need for manual keying by branches which would otherwise have been
avoided. What created false discrepancies was some instances of human error in the
manual keying. In passing, it is noted that the process for scanning barcodes is a good
80 (F/633}.
1 (D1/4/7} para 4.4.
82 (F/1848.8.1/1}.
89 (P/1848.8.1/2}.
84 {F/1848.8.1/2}.
85 (P/1848.8.1/2}.
te
B
8
POL00026925
POL00026925
AI8/242
724.
725.
726.
727.
POL00026925
POL00026925
example of how Horizon has been designed to minimise the risk of data entry errors by
removing the need for the manual input of figures.
The issue was resolved quickly, with the first incident being raised on 4 March 2019 and
closed on 7 March 2019.5*°
This bug was picked up automatically. The incident report notes “/m/inimal impact as
incidents were routed directly to the Early Life Support Team.”**"
TCs were required to reconcile branch accounts. However, as noted in the incident report
this exercise was being undertaken.*** On 6 March 2019 a report showing failed pre-
announcements was requested.*? This report was to be checked against branch rem-
ins.“ The report notes that all affected branches will be identified and that TCs will be
issued to affected branches:
. A list of branches that had called to report the issue was passed to the FSC who
advised on standard process and will manage the Transaction Corrections.
° The Horizon team are currently working on a report to identify all missing pre-
announcements which will be used to check against Horizon rem-ins in CFS to
confirm where mistakes were made in manual keying; these will then be passed to
the FSC to manage the necessary Transaction Corrections.
e Any revaluation losses caused by this incident are picked up by POL and not by
the individual branches.**!
A fixed was identified, tested and implemented on 5 March 2019.8” In addition to the
fix being implemented, active monitoring was put in place:
° The number range issue on Horizon has already been fixed. A follow up action will
be taken to put in place a process and controls to ensure that this is proactively
managed in the future.
. An additional error handling step has also been added in the interface middleware
(PI) so that if a similar issue occurs in the future it will be identified immediately °”
86 {F/1848.8.1/1}.
87 {F/1848.8.1/2}.
88 (F/1848.8.1/2}.
89 {F/1848.8.1/5}.
40 {F/1848.8.1/5}.
MI {P/1848.8.1/3}.
2 {F/1848.8.1/5}.
8 {P/1848.8.1/3}.
AI8/243
POL00026925
POL00026925
F20. Conclusion on the 29 Alleged Bugs
728. On the basis of the 29 bugs listed in the Bug Table, Mr Coyne has formed the view that
during the lifetime of Horizon there are likely to have been no more than 40 detected
bugs with branch impact.** However, it should be noted that:
728.1 Of those 29 bugs, Mr Coyne only asserts that 22 have lasting impact.
728.2 For its part, Post Office contends that:
(a) only 21 are bugs at all;
(b) — only 18 have actual or potential branch impact; and
(c) only nine have actual or potential lasting branch impact.
728.3 If one scales up from these figures, the number of bugs with lasting branch impacts
which have arisen during the 20 year life of Horizon is likely to be significantly
lower than 40.
F21. Expert evidence on Remote access
F22. Scope and focus of the remote access issues
729. The Horizon Issues that address remote access are numbered 7, 10, 12 and 13. The key
concept in each of them is the potential for transaction data in branch accounts to be
affected in some way by the use of facilities at Post Office or Fujitsu.
730. Issue 7 appears on its face to be concerned with read-only access to transaction data. It
uses the verb “access”, rather than any of the words that the parties have chosen to
describe the alteration or addition or subtraction of transaction data. The experts treated
Issue 7 on this basis in JS3: see {D1/4/10}.
4 (Day15/124:21} to {Day15/124:25}.
AI8/244
731.
732.
733.
734,
735.
POL00026925
POL00026925
Horizon Issue 10 concerns facilities/abilitics to insert, inject, edit or delete transaction
data or data in branch accounts, to implement fixes that had the potential to affect such
data and the rebuilding of branch transaction data.
Horizon Issue 11 relates to the permission controls and records (logs) in relation to such
facilities/abilities and their use.
Horizon Issues 12 and 13 focus on (a) how often the facilities/abilities referred to in issue
10 were used and (b) to what extent the use of such facilities/abilities had the potential
to affect the reliability of branches’ accounting positions.
It is apparent from the Horizon Issues themselves, therefore, that the main focus of the
enquiry is on the extent to which the abilities of Post Office and/or Fujitsu to remotely
inject, edit or delete transaction data / data in branch accounts (a) were used and (b)
potentially affected branch accounts. That focus is unsurprising given that the parties”
pleaded cases revolve around Cs’ contention that the remote alteration of transaction data
affected branch accounts: see GPOC, para 21.3,**5 238° and 25.8’ The same focus is
seen in the duty pleaded at GPOC, para 64.10°** and the allegedly guilty knowledge
pleaded at GPOC, para 115.8.5”
Questions of terminology — “transaction data” and “remote access”
Given the clear focus on effects on transaction data and branch accounts, it is important
to establish at the outset what is covered by the term “transaction data”. Mr Coyne fairly
accepted that a practical and useful definition of that term would be as follows: “records
of transactions: price, date, quantity, money paid”,**° i.e. the basic and defining details
of the transaction. This can be contrasted to the system / operational data associated with
the transaction, such as the various time stamps and other markers that become
associated with the transaction as it passes through the system.
SS (C3/1/8}.
6 (€3/1/8}.
7 (€3/1/9}
8 (C3/1/36}.
49 (€3/1/56}.
80 {Day16/2:5} to {Day16/2:14}.
AI8/245
736.
737.
738.
The transaction data for a bill payment, for example, would include: (a) product (i.e. the
bill), (b) payer/customer, (c) payee/client, (d) amount of payment, (e) time of payment
and (f) method of payment. Other transactions would involve fewer items of transaction
data, such as the purchase of a packet of stamps. The transaction data is the data that, if
different, would alter the branches’ accounting position.
If the transaction data is correct, this will ensure that the branch accounting position is
correct, irrespective of any changes to other elements of the accounting process. Mr
Coyne fairly accepted the following: **!
737.1 A branch account is formed of an aggregation of transactions from a given starting
point. (A fuller definition would include that the account results from the
comparison between (a) the aggregated figures for cash and stock and (b) the
actual figures for the cash and stock in the branch).
737.2 A branch’s opening position is an aggregation of the transactions up to that point.
As long as the opening position is a genuine aggregation of those transactions, the
account will be correct at that time.
737.3 A change to the accounting period should not affect whether or not there is a
shortfall.
It is also important to clarify that the “remote access” addressed by the Horizon Issues
concerns injecting, editing or deleting transaction data in branch accounts (rather than
other copies of that data that exist in other systems). The pleadings and the Horizon
Issues focus on effects on branch accounts, rather than the processing of data within other
systems and other business operations. It follows that:
738.1 Transaction data recorded in Post Office’s back-end MIS (or anywhere else other
than Horizon) is not the subject of these Horizon Issues.
738.2 Any changes made by the Transaction Repair Tool to data in the TPS is not remote
access within the meaning of these Horizon Issues. It is not a change made to
transaction data in branch accounts.
POL00026925
POL00026925
81 {Day16/2:18} to (Day16/3:21}.
AI8/246
739,
740.
TAL.
742.
POL00026925
POL00026925
Information available to the experts
Mr Coyne fairly accepted that the experts had available to them very substantial amounts
of contemporaneous documentation in relation to remote access. Specifically, Mr Coyne
accepted the following:
739.1 He had seen many tens of thousands of OCPs, OCRs and MSCs**” (i.e. records
that would reveal changes made by Fujitsu).
739.2 These documents were a “good source of information that would indicate what
steps were taken” 5°
This is of course in addition to the evidence in relation to remote access that can be taken
from Peaks, KELs, technical documents and the witness statements. In this regard, Mr
Coyne fairly accepted that, if there is some remote access, that will often be recorded in
the relevant Peak(s).*** He accepted that Fujitsu’s process was to record in the body of
the relevant Peak the detail of any remote access used in resolving the issue addressed
by the Peak.**>
It is difficult to reconcile this frank evidence with some of Mr Coyne’s remarks in his
reports and the Joint Statements: see, for example, the suggestion at JS4, para 11.13,
giving the insertion of messages in Legacy Horizon as an example of a corrective fix that
left “no evidence trail”.*°°
Mr Coyne had a tendency in his written evidence to overstate
the problems with the evidence available to him, but he made appropriate concessions
when asked about these points in cross-examination.
The Court will also recall Mr Coyne’s evidence as to the sophisticated search and review
processes that he was able to use. He sometimes wished to downplay the extent to which
he was able to deploy those processes in relation to remote access, but there is no reason
to think that there was some special impediment here. The only candidate for such an
82 {Dayl6/17:12}.
83 (Day16/17:18}.
84 (Dayl6/17:18} to {Day 6/18:21}.
5 (Dayl6/28:15} to {Day16/28:20}
86 (D1/5/13}.
AI8/247
743.
744,
74S.
impediment is the fact that the MSC logs were disclosed in the form of three
spreadsheets, which Mr Coyne noted.**7
Mr Coyne also fairly accepted that one could take a sample of FAD codes and use
intelligent search techniques to identify the extent of remote access in relation to that
sample of branches. He accepted that to do so would be a suitable and helpful process.***
It was a fair point for Mr Coyne to observe that Post Office had, during relatively early
exchanges in relation to RFIs, stated that it was inappropriate to seek to identify
information “tied to individual cases” **° (which is not quite the same thing). But that is
largely beside the point given that all Mr Coyne had to do was identify a sample of FAD
codes to use. He could have used Claimant FAD codes,*® but he could also, as he
accepted, have taken a random sample of any other FAD codes.°*!
The experts had at their disposal a substantial body of contemporaneous evidence from
which to assess the nature and extent of remote alteration of branch data. Cs cannot
speculate as to what they have been in a position to determine.
F23. Overall scale and importance of remote access
The remote access issues have always had great prominence in these proceedings, for
entirely understandable reasons. There has now been a deep and thorough investigation
of Fujitsu’s abilities to alter branch transaction data and, perhaps more importantly, how
and to what extent Fujitsu in fact exercised those abilities. As set out in detail below, the
outcome of the experts’ combined efforts has been to show that Fujitsu displayed an
appropriate reluctance to do anything that would interfere with branch accounts; that,
when it had to make some change, it did so with great care; and that its practice was to
make sure that the SPM knew what was being done and could confirm that the change
took effect as intended.
POL00026925
POL00026925
857 Mr Green QC invited Mr Coyne in re-examination to complain about the MSC logs having been disclosed
in three spreadsheets, rather than one, which he duly did.
88 {Day16/20:24} to {Dayl6/22:17}.
859 {C5/21/5}. Mr Coyne was unfairly criticised for this, and Post Office apologises for that.
80 Cs themselves of course had their FAD codes, and a full list was provided with Ms Van Den Bogerd’s
second witn
s statement: see {F2/5/41} (para. 179), referring to the document at {F/1837}.
81 {Day16/24:24}.
AI8/248
746. Mr Coyne fairly accepted that he had identified relatively few instances of remote access
being used to affect branch accounts, especially compared to the vast number of branch
accounts over the life of Horizon:
MR DE GARR ROBINSON...I would be right in thinking, wouldn't I, that of the
PEAKs you have seen you found relatively few examples of remote access having been
exercised? Would the answer to my question be right?
A, I don't know exactly what the number will be, but it is tens, twenties
Q. Looking at your report it would be low tens, wouldn't it? You haven't found
hundreds?
A. No, I haven't found evidence of hundreds, no.
Q. So you have found, as I say, a relatively small number; relative to the fact that we
are talking about 3 million branch accounts over the last 20 years, all you have
actually found is a very small number which is less than 20 or 30, let's call it less than
30, would you agree with that?
A. Yes.8
747. Mr Coyne agreed that the evidence he had seen showed that where Fujitsu performed
748.
requirement that every change be witnessed or monitored by another SSC employee.
remote access affecting branch accounts it did so carefully.*® This is consistent with all
the witness evidence, including Mr Roll's oral evidence in relation to the "four eyes"
864
Mr Coyne also accepted that, given that remote access to alter branch accounts was
relatively rare and was done carefully, there was only a small chance of it adversely
affecting branch accounts: “/t would be fair to say that [the chance] would be reasonably
small. I don’t know about saying it was vanishingly small” 5° To similar effect, Mr
Coyne accepted that the chance of a false figure being introduced into an SPM’s account
through remote access was “small”.
POL00026925
POL00026925
82 (Dayl6/28:25} to {Day16/29:15}.
83 {Day16/31:7} to {Day16/31:13}.
8 (Day4/127:15} to {Day4/128:19} and {Day4/156:10} to {Day4/158:9}. This requirement is found in
Fujitsu’s “Access Control Policy” at para. 4.5.5.4 {F/121/40}. See also later versions of the same policy at
{F/970.1/41}, {F/1333.1.1/42} and {F/508.01/41}.
85 {Day16/29:16} to {Day16/30:6}.
86 {Day16/31:14} to {Dayl6/31:18}.
AI8/249
749.
750.
751.
752.
753.
In fact, Mr Coyne was able to refer to only a single instance in which he contended a
change made through remote access had introduced error into an SPM’s account.*®”
Putting to one side that the instance to which he refers in fact shows no such thing,** a
single instance across 3 million branch accounts and 20 years would not indicate a
material risk of remote access introducing a false shortfall into any given branch account.
Mr Coyne fairly pointed out that, under Legacy Horizon, there were other processes
through which Fujitsu might have affected the content of branch accounts (what he refers
to as “rebuilding” branch data in counters). But the picture there is not materially
different — there is no evidence of erroneous transaction data being introduced through
the “rebuilding” process; the purpose of the process was to improve the accuracy of the
branch accounts; and it was done carefully (as the Court will recall Mr Roll readily
accepted in his oral evidence).
Ultimately, the evidence has borne out Dr Worden’s view that remote access is, as a
second order issue, unlikely to materially affect the robustness of the Horizon system
and the accuracy of branch accounts.
F24. Specific forms of remote access
Legacy Horizon
Mr Coyne identified the following abilities / facilities to insert, edit or delete transaction
data or data in branch accounts under Legacy Horizon:
752.1 Rebuilding branch transaction data (in a counter).
752.2 Injection of an additional message in the branch messagestore.
752.3 Deletion of messagestore data.
These are identified in para 10.4 of JS4.8° Mr Coyne confirmed in cross-examination
that he was not aware of any other forms of remote access under Legacy Horizon.*” JS4,
POL00026925
POL00026925
87 {Day16/30:18}.
868 See further at paras 7!
et seq below.
® (D1/5/4}.
7 {Day16/91:24}.
A/8/250
754.
755.
756.
757.
POL00026925
POL00026925
para 10 also refers to the use of the Transaction Repair Tool, but this is addressed
separately below (as it is common to both Legacy Horizon and Horizon Online).
(1) “Rebuilding” a counter’s data
Mr Coyne agreed that, in the ordinary case, the process of “rebuilding” a counter’s
messagestore in fact involved the deletion of that messagestore and its automatic
replication from a “mirror” messagestore, using Riposte.’’”! That process does not
involve any discretionary change to individual messages or transactions in the
messagestore. Mr Coyne accepted that such automatic “rebuilding” was a back-up
process that contributed to robustness and should result in the counter containing the
right data.’ The process is described in Mr Parker’s first and second statements.*”*
There is no evidence of the automatic process ever having generated errors in branch
accounts.
Mr Coyne stated that he had identified “ten or so” instances where some manual process
was required.*”* It is common ground that manual intervention was occasionally required
(where the automatic replication process could not be used for some reason). Mr Parker
identified the two forms of manual intervention in paras 38.2 and 38.3 of his second
witness statement.5”>
In relation to those manual processes, Mr Coyne fairly accepted that, where Fujitsu had
to retrieve the hard disk from a faulty counter, it would require a physical card to be
provided by the SPM, who would therefore be aware of what was happening.*”° He also
agreed with Mr Parker’s evidence that, if Fujitsu were to change anything in that manual
process, it would be to remove the “envelope” surrounding the transaction data: “Yes.
They go through a process of stripping the CRC off and then recreating it afierwards” 5”
® (Day16/32:21} to {Day 6/33:11}.
82 {Day16/33:25} to {Dayl6/34:10}.
® Parker 1, paras $5.3-55.4 {E2/11/15} and Parker 2, para. 37 {E2/12/12}.
® (Day16/33:23}.
8 €E2/12/12}.
876 {Day16/79:13} to {Day16/80:5}.
®7 (Day16/80:11}.
A/8/251
758.
759.
POL00026925
POL00026925
Mr Coyne said that he had seen a Peak that showed Fujitsu manually renumbering
transactions using a “Jext Pad” and then reinserting transactions using “Message
Factory” 57> Mr Coyne undertook to locate the Peak overnight,5” but he did not return
to this point the next day (or at all). It seems likely that the Peak that Mr Coyne had in
mind was {F/377.1} (to which he did refer, albeit on a different point, on day 17). That
Peak contains reference to both “textpad” and “Messagel‘actory”. The Peak is
PC0142763. It dates from January 2007 and in fact shows the following:
758.1 An SPM reported that he had missing transactions following the replacement of
the two base units in his branch: {F/377.1/1}.
758.2 There appeared to have been a failure in replication, due to a fault event in Riposte:
see at {F/377.1/2} timed at 09:13.
758.3 Fujitsu recovered the old counters from which messages had not replicated: see
the fourth entry at {F/377.1/3} timed at 11:02.
758.4 The missing messages were downloaded from the old counters: see the first entry
at {F/377.1/4} and the second to fifth entries at {F/377.1/5}.
758.5 The missing transactions would be imported into the accounts and would be
marked as having been performed on counters 11 and 12 (rather than I and 2) that
they would they could be seen as such: see the fifth entry at {F/377.1/4}.
758.6 The SPM had mistakenly re-performed some of the missing transactions in the
meantime, resulting in duplicates, but this was resolved: see the first entry at
{F/377.1/5}.
758.7 The SPM had a loss on rollover but grudgingly accepted that this was a genuine
loss, rather than being caused by the re-insertion of the missing transactions: see
the sixth entry at {F/377.1/5}.
The references to “textpad” and “MessageFactory” come in the final entry on the Peak,
where David Seddon set out the “audit of actions taken to recovered messages”. The
8 (Day16/80:20} to {Dayl6/81:10}.
879 At one point, Mr Coyne referred to a “handful” of similar documents and agreed to bring these to Court,
but he did not do so: see {Day16/85:3} to {Day16/85:8}.
i
A
ia
AI8/252
760.
761.
762.
763.
Peak provides a good example of the care taken by Fujitsu to record the changes that it
made and to make sure that they could be identified (in this case, by giving the inserted
transactions different and much higher counter numbers)**”. It also shows the extent to
which the SPM was kept informed and involved in the process of correction.
Mr Coyne fairly accepted that the chance of introducing error through this (rare) manual
process would be “/ow”.**! There would of course be some risk, but the process described
by Mr Parker and shown in Peak PC0142763 is a careful and precise one. It would carry
a very low risk of introducing error into branch accounts, not least given that the inserted
transactions were identified as such and the process involved cooperation with the SPM.
Mr Coyne also referred to having seen a Peak describing a situation where the SPM
manually re-performed transactions before they were recovered by Fujitsu and re-
inserted, resulting in duplicates.*** Mr Coyne said that he would find the Peak and
provide it the next day. He did not specifically draw attention to it, but this again appears
to be Peak PC0142763 {F/377.1} (see para 758 above). The duplicates were immediately
discovered and removed.
In short, the single example of manual rebuilding that Mr Coyne was able to identify
shows a careful process, done in collaboration with the SPM and carrying a very small
risk of introducing any error into the branch accounts.
(2) Injection of messages into the branch messagestore
Insertion at the correspondence server (counter number 32)
Mr Parker’s unchallenged evidence is that the standard process for inserting a message
in Legacy Horizon involved the injection of the message at the correspondence server,
resulting in a transaction with a notional counter number of 32.°*? Mr Coyne did not feel
POL00026925
POL00026925
8° The branch at issue did not have 11 or more counters. No agency branch does have that many counters.
881 (Day16/85:21}
882 (Day16/87:19} to {Day16/88:21}.
883 Parker 2, paras 27-28 {E2/12/9}. No branch would have 32 counters.
AI8/253
764.
76S.
766.
767.
able to positively agree that this was the standard process, but he fairly accepted that he
was not in a position to disagree with Mr Parker in this respect.***
One advantage of that standard process was that the counter number associated with the
inserted message (32) would distinguish it clearly from messages created at a branch
counter, given that no branch counter would have a number that high.
For the first time in his oral evidence, Mr Coyne suggested that he had seen “occurrences
where the counter number of 32 has not been used” for an injection at the correspondence
server.**> He undertook to provide documents evidencing this the next day. On the next
day, Mr Coyne proffered one document, namely Peak PC0142763 {F/377.1}. That
document has been considered extensively above. It appears to be the only example of a
counter number less than 32 having been used, and even in that case the transactions
were clearly identified through the use of unrealistically high counter numbers (11 and
12, replacing the original counter numbers of I and 2), and the SPM was kept informed
and involved throughout the process.
Insertion at the counter itself
Mr Coyne also accepted that Mr Parker gave a fair account of PC0175821,°°° which
exemplifies what Mr Parker describes as the non-standard process of injecting a
transaction at the counter. That Peak dates from February 2009 and is at {F/485}. It
shows the SPM (or his staff) was kept informed of the process and that the account
balanced following the insertions. The change was required to correct for corrupted
bureau transactions.
Mr Parker identified 14 instances of this non-standard process, (only one of which
involved the insertion of a transaction).**”
He identified the relevant Peaks. Mr Coyne
had reviewed the Peaks to which Mr Parker referred.*** He did not dispute Mr Parker’s
POL00026925
POL00026925
8! {Day16/93:9}.
*5 {Day16/94:11} to {Day]6/95:3}.
®86 {Day16/93:15} to {Day16/94:4}.
87 {Day16/97:11} to {Day16/98:24}.
888 {Day16/97:10}.
AI8/254
768.
769.
770.
771.
POL00026925
POL00026925
characterisation of the 14 insertions. (The extra Peaks later identified by a technician at
Fujitsu are addressed in the consideration of Mr Parker’s evidence.)
Mr Coyne confirmed that the search carried out by Fujitsu to identify the instances of
insertions into counter was likely to have been effective.**” He added that he had himself
carried out further searches: see the quotation from the transcript at para 331 above. He
accepted that his own searches had identified only a “relatively low” number of
insertions, a number that was “in the tens”.5”° It appears from Mr Coyne’s re-examination
that he first carried out these further searches at some point after receiving the letter dated
20 March 2019 at {H/253}.8"
The next day, Mr Coyne proposed to inform the Court how many such instances he had
identified, but without providing any reference to the relevant Peaks,*”*
making it
impossible to challenge any number that he might offer. Post Office objected to that
course and maintains that objection. Mr Coyne has frequently referred to there being
many instances of a given phenomenon but, when the documents relied on are
interrogated, they prove not to support the point that he makes. It would be unfair for Mr
Coyne to be able to present evidence based on further work that he carried out but kept
to himself, in preference to engaging with Dr Worden and/or notifying Post Office of his
new opinion and the evidential basis for it. The approach that Mr Coyne tried to adopt
would prevent any interrogation of the basis for his evidence in the documents.
Mr Coyne also referred to a Peak that he contended showed an erroneous shortfall
generated by a mistake that Fujitsu made in inserting a transaction. The Peak is
PC0152014.°* It dates from December 2007 and is at {F/432}. It can conveniently be
referred to as “the $1,000 Peak”. Mr Godeseth was taken to it in his cross-examination.
Post Office acknowledges that this Peak appears to show Fujitsu, in breach of its usual
practices, not informing the SPM of a change made by remote access. Mr Coyne fairly
89 {Day16/99:15}.
% {Day16/99:24} to {Day16/100:15}.
1 (Dayl7/181:14} to {Day17/181:25}.
®2 {Day17/8:24} to {Day17/9:20}.
© Tt was referred to at Coyne 2/3.234 {D2/4.1/75}.
AI8/255
772.
773.
774,
775.
776.
777.
POL00026925
POL00026925
confirmed that this is the sole instance of which he is aware of this having happened.*"*
The rationale for not informing the SPM appears to have been that he was unaware of
the problem and that, if it could be resolved before roll-over, he need not be troubled by
it. Post Office considers the decision taken there to have been wrong — the SPM should
have been informed, in accordance with standard process.
Mr Coyne was cross-examined at some length on this Peak and its associated OCP and
OCRs.*”> It is possible, through an admittedly pain-staking process, to establish from
these contemporaneous documents what happened in relation to the $1,000 Peak. On a
careful analysis, it is clear that the changes made by Fujitsu did not cause any shortfall
in the branch’s accounts.
The underlying problem was that a foreign exchange transaction for $1,000 had become
corrupted and lacked a settlement line (i.e. the record of payment, in sterling, connected
to the transaction): see the entry at the bottom of page I of the Peak {F’432/1}.
The chronology was as follows.
The Peak was raised on Friday, 7 December 2007 and as a result of the MSU’s review
of automatically generated reports.°°°
On the same day, OCR 17493 was raised and approved: see the entries timed at 10:35
and 10:41 on the Peak at {F/432/1}. The change was planned for the same day but in
fact took place on 10 December 2007. An unmatched counter message for $1,000 was
removed from the POLFS feed (i.e. the Transaction Repair Tool was used to adjust the
summaries table in the TPS): see the OCR at
and the first full entry on page 2 of the Peak {F/432/2}.
..} under “Comments”
On Monday, 10 December 2007, OCP 17510 was raised: see the second entry on page 2
of the Peak {F/432/2}. The change made under this OCP was to the counter messagestore
(i.e. a transaction insertion) and was the inverse of the original transaction, cancelling it
84 Dayl7/103:20} to {Day17/104:3}
®5 (Day16/104:4} to {Dayl6/132:6}.
86 Mr Coyne accepted this: {Day16/104:4} to {Day16/105:12}.
AI8/256
TE.
779.
780.
out: see under “Extra detail” on the OCP {F/432.2/1}. The inserted message was to
include a comment to show that it had been inserted by Fujitsu to correct the problem.
The OCP records that the change would first be tested on a copy of the messagestore
within the SSC environment, including rolling over the stock unit to confirm that there
were no unexpected consequences: see the third para under “Extra detail”. The change
was delayed for a short time by a problem with the test rig and was not effected until
Tuesday, 11 December 2007: see the entry above “Other details” on page 2 of the OCP
(F/432.2/2}.
On Wednesday, 12 December 2007, a further OCR was raised - OCR 17532: see the
entry timed at 12:03 on page 2 of the Peak {F/432/2}. This further OCR was also to
correct the POLFS feed and also involved the use of the Transaction Repair Tool (rather
than making any change to the branch accounts): see {F/434.1}. By the time of this
correction:
779.1 The $1,000 transaction had already been deleted from the POLFS feed under OCR
17493 (addressed above).
779.2 The effects of the $1,000 transaction at the counter level had been removed by
inserting an inverse transaction under OCP 17510 (addressed above).
This second OCR did not relate to the $1,000. That had been effectively removed from
both the branch accounts and from the POLFS feed. It related to a different but related
problem: the absence of USD bureau transaction totals for the branch in the POLFS feed.
Post Office had to provide this explanation at trial on instructions, but all the relevant
facts can be taken from the OCR itself:
780.1 The product code indicated on the OCR (under “Comments” at {F’434.1/1}) is
5129. That is the same product ID as for the corrupted transaction and the message
inserted to the counter. It is the product ID for USD: see the product ID list at
{F/1292.2/3}.
780.2 The “SaleValue” stated for the product is 1014.73. That is the sterling value of the
insertion being made into the POLFS feed. (Note that the SaleValue for the
POL00026925
POL00026925
AI8/257
781.
782.
783.
POL00026925
POL00026925
corrupted transaction in OCP 17510 was 484, the sterling equivalent of $1,000
(F/432.2/1}).
780.3 The “PQty” stated for the product is 2080. That is the dollar value of the insertion
being made into the POLFS feed. (Note that the PQty for the corrupted transaction
in OCP 17510 was 1000, reflecting that it was a $1,000 transaction {F/432.2/1}).
780.4 These two values must be the sterling and dollar amounts for something other than
the $1,000 transaction. And the insertion of those values was into the POLFS feed,
rather than into the branch accounts (which had already been corrected by the
insertion under OCP 17510).
On the same day, the branch rolled over into the next trading period. The final entry on
page 2 of the Peak says this: “Worth noting that the branch did not have any issues with
the mismatched transactions because this was fixed before they did the roll.” {F/432/2}.
On Friday, 14 December 2007, Anne Chambers provided a comprehensive explanation
of the action that had been taken:
As detailed above, the two POLFS incomplete summaries issues have been resolved.
The counter problem which caused the first issue has been corrected by inserting a
message into the messagestore, for equal but opposite values/quantities, as agreed
with POL (OCP 17510).
As a result of this corrective action, the net effect on POLFS is zero, and POLFS
figures are in line with the branch. POLMIS received both the original message and
the corrective message.
Once the problem was corrected, there should have been no impact on the branch.
{F/432/3}
Ms Chambers went on to explain that the branch nonetheless did have a loss: “However
it has been noted that the stock unit BDC had a loss of $1,000, which was generated
after the correction was made”. Post Office had been informed. She noted that “This
appears to be a genuine loss at the branch, not a consequence of the problem or
correction”: see {F/432/3}. It appears from that entry, which refers to “reconciliation”
Al8/258
784,
785.
786.
787.
and from the fact that the Peak shows a BIMS was issued that the loss may well have
been addressed by Post Office.
That is consistent with the entry from Anne Chambers at the bottom of page 3 of the
Peak,**” where she identifies that there had been a problem with the two pouch remittance
operations either side of the $1,000 transaction that had been corrupted.
It is impossible, at this remove in time and without more evidence, to identify what
precisely happened to cause the $1,000 loss in the branch. But the documentary record
does not suggest that it was caused by any of the corrective steps taken by Fujitsu.
More specifically, the case advanced by Cs in the cross-examination of Mr Godeseth was
clearly wrong: the loss in the branch was not caused by some error in the amount of the
correction to the branch accounts. Mr Godeseth was referred to the values in OCR 17532
in this regard,*** and that OCR did not relate to a change to the message store (i.e. data
forming the branch accounts) but only to the POLFS feed (i.e. data being supplied into
Post Office back-end system). Mr Coyne accepted this point on the OCR.*” Mr Coyne
also accepted that the figures shown in OCR 17352 bear no relation to cither £484 or
$1,000 so would be difficult to understand unless they related to something other than
the corrupted transaction and its correction.°°
Another way of making the same point is to observe that the only correction that could
have resulted in a loss in the branch was the OCP. That was the only change that affected
the branch account, rather than the data feed from the TPS. And review of the OCP shows
no basis on which to conclude that the change involved any error, and every reason to
think that it was made properly and carefully, after testing.
POL00026925
POL00026925
87 £F/432/3}.
8 (F/434.1}.
9 (Day16/122:2}.
9 {Day16/127:2}.
259
Al8/259
POL00026925
POL00026925
(3) Deletion of messagestore data
788. It became apparent in Mr Coyne’s cross-examination that he was not in fact aware of
any examples of the deletion or editing of transaction data other than as part of one of
other of the “rebuilding” processes addressed above. He confirmed the following:
788.1 He had not seen examples of the discretionary manual deletion of transactions in
the message store. He commented that it was “¢ypically wholesale”: deletion of
the message store- i.e. the first step in the automatic “rebuild” process described
above.
788.2 He had not seen any evidence of Fujitsu changing the value of any transaction in
the message store, as opposed to “data around” the transaction.”
This appeared
to be a reference to the removal of the “envelope” around the transaction data, as
discussed above.
788.3 He was not aware of any instance where someone from Fujitsu had remotely
accessed a branch messagestore and change a transaction in that messagestore.””>
Again, he referred to a manual “rebuild” (involving the physical removal of the
branch counter).
789. Mr Coyne then directed the Court to {F/377.1}.°%! That Peak has been addressed
exhaustively above.
790. On analysis, Mr Coyne’s third form of remote access was not additional to his second.
Horizon Online
791. Inhis second report, Mr Coyne expressed the view that Global Users were (or might be)
able to remotely alter branch transaction data. He formed this view on the basis of a
technical document. Mr Godeseth then responded to that point in his third witness
sor
902
903
904
{Day16/132:21} to {Day16/133:8}.
(Day16/134:13}.
{Day16/137:4}.
{Day16/135:10}.
A/8/260
792.
793.
794,
795.
905
statement.” At trial, Mr Coyne fairly accepted that Mr Godeseth’s explanation had
906
removed his concern about Global Users.”’° He observed that, now that he had seen the
explanation, the position made “complete sense”.
Putting that to one side, Mr Coyne identified the following abilities / facilities to insert,
edit or delete transaction data or data in branch accounts under Horizon Online:
792.1 Injection of a transaction (or operational data) into the BRDB using an SQL.
Command Line Editor.
792.2 Use of the Transaction Correction Tool. (There appears to be a typographical error
in the reference in JS4 to the “Balancing Transaction Tool”).?””
These abilities are identified in para 10.2 of JS4.°°S Mr Coyne confirmed in cross-
examination that he was not aware of any other forms of remote access under Horizon
Online.” JS4, para 10.2 also refers to the use of the Transaction Repair Tool, but this is
addressed separately below (as it is common to both Legacy Horizon and Horizon
Online).
Mr Coyne also, in a different para of JS4, addresses the possibility of changes being
made using the APPSUPP role: see para 10.5 {D1/5/5}. He confirmed in cross-
examination that such changes would be made using the SQL Command Line Editor, so
fall within the first category of change identified in JS4, para 10.2.°!°
(1) The SOL Command Line Editor /APPSUP privileges
Mr Coyne frankly accepted that every big system like Horizon would require “super user
access”°"! The experts agree that it is necessary for “some technical users to have
POL00026925
POL00026925
5 £2/14/2} at paras 4 to 9.
%6 {Day16/70:13} to {Dayl6/71:15}.
%7 This is clear from the correct term being used in JS4, para. 10.3 {D1/3/5}.
8 {D1/5/4}.
9 {Day16/91:24}
%® {Day16/144:17} to {Day16/144:24}. This accords with JS4, para. 10.7 {D1/3/6}.
1 (Day16/145:9}.
A/8/261
POL00026925
POL00026925
privileged access to databases, with wide-ranging capabilities, for system maintenance
and problem-solving purposes”: JS4, para 11.2.2"
796. Mr Coyne referred in his second report to uses of the SQL command to delete data: see
Coyne 2/3.226-3.276."'* On analysis, the Peaks on which he relies all relate to one of
two scenarios, neither of which involves the deletion (or editing or insertion) of
transactions in the BRDB. Mr Coyne suggested that he had seen other Peaks but accepted
that the Peaks relied on his report were representative.”!*
Deletion of unresolved recovery sessions
797. Mr Coyne acknowledged in cross-examination that the bulk of the Peaks to which he
referred provided examples of the deletion of sessions containing recovery flags, rather
than the deletion of transaction data.?'> He readily accepted the following points about
that process:
797.1 The purpose of the deletion is to address a problem where the existence of an
unresolved recovery flag is preventing use of the counter and/or rollover.”'®
797.2 This does not involve the deletion of any data in the branch accounts.°'”
797.3 Nor does it involve any change being made to the data in the branch accounts.”!*
798. Mr Coyne’s contention was that the deletion of the session containing the unresolved
recovery flag could result in branch discrepancies, depending on what had in fact
happened in relation to the transactions that were not yet committed to the BRDB (i.e.
the transactions that would or might have been recovered and so completed, had the
recovery been successful).?!?
92 (D1/5/10}.
93 (12/4.1/83}.
°M {Dayl6/156:9} to {Dayl6/15
6 Tid.
917 {Day16/153:1} to {Day16/153:10}.
18 Tid.
919 See, for example, {Day16/153:9}, referring to data that is “in an unconfirmed state that needs dealing
with”,
i
&
8
AI8/262
799.
800.
801.
802.
803.
It is fair to identify the possibility of error, but the likelihood of adverse effects in practice
is very low. The starting point is that Fujitsu has in place detailed processes for resolving
problems in the recovery process: see the various scenarios addressed in KEL acha959T
{F/700.1}. Most instances of failed recoveries are processed in a straightforward way.
In the relatively unusual case where there is a failed recovery process that requires the
deletion of the recovery session data, the SPM will be aware of and involved in the
resolution of the problem. The counter will have become unusable and/or the relevant
stock unit will be unable to rollover.
It is apparent from the Peaks that Fujitsu was reluctant to delete session data, even if
only to clear an unresolved recovery flag: see, for example, Peak PC0234786, in which
Supid Sur comments to the effect that deleting the session data should only be pursued
(and only then with express authorisation) if the problem cannot be resolved at the
counter by an engineer.””° This reflects Fujitsu’s general reluctance to intervene in any
way that might affect branch accounts: see further below at para. 810.
There are, however, unusual circumstances in which there is no alternative but to delete
the session. One such circumstance is where a counter has been removed from the branch
and the last user did not log out correctly before the counter was removed: see, for
instance, PC0239436 (December 2014), in the fourth entry on page 5”! and PC02363716
922
(October 2017), in the second entry on page 7.
In those few Peaks where a deletion was required, there is evidence of Post Office’s
concern to make sure that authorising the deletion would not cause any discrepancy in
the branch accounts. This can be seen, for example, in Peak PC0241528, which
concerned a problem with a Health Lottery transaction (which could not be recovered or
cleared) that formed part of the same session as an otherwise recoverable banking
transaction. There was a clear concern to make sure that the deletion of the session did
not result in any branch discrepancy in respect of either of the transactions:
POL00026925
POL00026925
220 £F/1120/2}
1 {F/1284.1/5}.
222 {F/1703/6}.
AI8/263
POL00026925
POL00026925
803.1 As regards the Health Lottery transaction, an email from Patricia Bursi of Post
Office in the middle of page 6 states as follows:
“...Pat has kindly rung the branch and confirmed the customer isn't impacted,
nothing changed hands for the Health Lottery transaction, and there is no
branch discrepancy.
I'm therefore happy to authorise this session to be deleted so that the kit at the
branch can return to BAU state...” °*
803.2 As regards the banking transaction, the entry starting at the bottom of page 8
identifies the need for a TC to be issued: “The cash withdrawal txn was authorised
and PM should have paid the money out.... However this will leave this office
£296.70 short... POL need to do appropriate reconciliation; transaction
correction’®*’, The entry for 24 March 2015 at 14:10 shows that a BIMS was duly
issued to trigger the TC process.
804. Mr Coyne fairly accepted that the Peaks show that the SPM was consulted in order to
avoid any branch discrepancy arising:
Q. ... There was deletion of data relating to recovery markers which were concerned
with sessions, would you agree with that?
A. Yes, but contained within there is session data that's in an unconfirmed state that
needs dealing with.
Q. Yes, And in each of those cases it was dealt with in consultation with the
subpostmaster to ensure that the accounts were right, to ensure that nothing was done
which produced an inappropriate result for the accounts.
Would you agree with that?
A. Lagree actually with the process, yes.?”*
23 £F/1320/6}.
4 £F/1320/8}.
8 {Day16/153:6} to {Day16/153:16}.
AI8/264
805.
806.
807.
808.
POL00026925
POL00026925
Deletion of stock unit opening positions
Mr Coyne also referred to PCO197592 (April 2010)°° which concerns the deletion of an
erroneous branch opening position, rather than transaction data.°’? As Mr Coyne
accepted in cross-examination:?”*
805.1 It concerned a branch involved in the Horizon Online pilot stage.
805.2 The branch was unable to rollover from trading period 11 (“TP 11”) into TP 12
because, due to a problem during migration,” rogue figures (zeros) had been
wrongly entered as stock unit opening positions for TP 12.
805.3 The SQL command was used to delete the rogue opening positions.
805.4 The result of this was to permit the branch to rollover, in the normal way, into TP
12. Upon rollover, the correct opening positions would be automatically generated,
based on the transactions in the branch database.
Mr Coyne captured the point elegantly when he observed that the result of Fujitsu’s
intervention was that, following rollover, the opening position of the branch in TP 12
would have been “Whatever it should be”° The Peak provides a good example of
Fujitsu being required, in unusual circumstances, to intervene remotely in the branch
accounts, and doing so in a way that creates no adverse effect.
The Peak also shows that SPM was kept informed of the process by the SSC directly:
28
see the penultimate entry on page 2 (Anne Chambers Authorisation was obtained
from Post Office via an OCP (referred to in the header to the Peak).
The possibility of other changes made using SOL
It is important to apply a measure of reality here. The experts have conducted an
exhaustive review of the vast contemporaneous documentation that would disclose any
6 {F/611}.
7 Coyne 2/3.275 {D2/4.1/84}.
28 (Day16/156:24} to {Day16/161:5}.
9 See the fifth para. in the entry that begins at the bottom of page 2 {F/611/2}.
30 Day16/160:8}.
9°31 {F/611/2}.
AI8/265
809.
810.
811.
POL00026925
POL00026925
alteration of branch data. Mr Coyne in particular was concerned to root out any evidence
to support Cs’ case that remote access might account for disputed shortfalls in branch
accounts. In that context, while the experts and the witnesses of fact quite properly avoid
absolute statements to the effect that other changes have certainly never happened,”*? the
Court cannot properly be invited to speculate.
There is a further point of common sense and inference. Fujitsu created the Transaction
Correction Tool (“TCT”, discussed below) for good reason — to have the benefit of
carefully drafted SQL templates that would allow transactions to be inserted with relative
ease and with a low risk of error. The experts agree that the one use of the TCT took
effect through “about 500 lines of SQL, in a pre-defined template” (emphasis added).
It is inherently unlikely that, having seen the need to build, test, improve and re-test the
TCT, Fujitsu would then put the carefully crafted tool to one side and engage in free-
hand SQL drafting. The documents and the witness evidence speak to a culture within
the SSC that was process-driven and emphasised appropriate caution.
Fujitsu was and is reluctant to make changes that would have an effect on branch
accounts. Mr Coyne accepted that the contemporaneous documents show this general
reluctance.°*4 Dr Worden expressed essentially the same view, saying that a privileged
user would be particularly reluctant to make manual changes, rather than using pre-
defined tools.”*> Mr Parker’s evidence is that the SSC was “hugely reluctant to change
financial data as that was not their job and they recognised the seriousness of doing
so”.°*° The Peaks support that unchallenged evidence.
It is therefore right to say that it cannot be entirely excluded that a free-hand SQL.
command might have been used in a way that affected a branch account, but that
possibility should not be equated with probability (or even a material risk).
°82 See, for example, the final sentence of JS4, para. 10.12 {D1/3/8}.
%3 JS4, para. 12.2 {D1/5/14}.
4 Day16/162:9}
9°85 Worden 1/1108-1110 {D3/1/243}.
°6 Parker 2, para. 34 {E2/12/11}.
AI8/266
POL00026925
POL00026925
812. Mr Coyne appeared in his second report to rely on a remark by Anne Chambers in
PC0208119°°7 as suggesting that the APPSUP role was used to make changes to the
BRDB that might affect branch accounts.”*> Ms Chambers said this: “When we go off
piste we use appsup”.°*” As Dr Worden observed, the question is what Ms Chambers
meant by those words, which is not really a matter for the experts.”"° If one looks at the
context in which the words were used, it is highly unlikely that what she meant was that
she and other SSC employees had been engaging in undocumented or otherwise
inappropriate changes to transaction data in the BRDB:
812.1 The Peak involves a frank and considered exchange of views between senior
Fujitsu personnel as to how to resolve a mis-match between the roles assigned to
system tools (referred to as scripts in the Peak) designed by the development team
and the roles in fact granted to SSC staff that were transferred across from Legacy
Horizon.
812.2 The context of that discussion is that scripts had been assigned to the SSC role,
whereas existing the actual SSC employees had been granted the old APPSUP role
during migration to Horizon Online. The new scripts were for “tidyup tasks (like
clear failed recoveries)”: see the second entry on page I {F/768/1}. There is no
mention in the Peak of any of the scripts being used to insert, edit or delete
transaction data.
812.3 It is apparent from Peaks already discussed above that SSC employees, including
Ms Chambers herself, were sometimes required to make changes using SQL
commands (i.e. without the benefit of specific tools): see the free-hand SQL
proposed by Gareth Jenkins to address the problem addressed in PC0197592
{F/611/3} (a Peak in which Ms Chambers was heavily involved). An ad hoc
intervention to resolve a problem that had not been encountered before would quite
7 {F/768}
9°88 In fairness to Mr Coyne, this is implicit only in his report: see Coyne 2/3.277-3.282 {D2/4.1/86}. But it
is a point that Cs have run with.
%9 {F/768/2} in the middle of the page.
% {Day20/86:25} to {Day20/87:10}.
AI8/267
813.
814.
815.
naturally be referred to as going “off piste”. It is notable that the two Peaks are
relatively close in time — April 2010 and February 2011.
812.4 In the body of the Peak itself, it is made clear that any unaudited use of write
access would involve a security breach: see the final entry on page 3 {F/768/3}. It
is clear that nobody involved in the Peak understood Ms Chambers to have
dropped the bombshell that she and others were merrily committing security
breaches.
Cs’ reliance on Ms Chamber’s casually-worded remark is perhaps understandable, but it
cannot bear the weight placed on it. There remains no evidence of manual changes to
transaction data being executed by privileged users. It is a different question whether
Fujitsu addressed the user privileges problem addressed in PC0208119 with appropriate
urgency — see further below at para 476-8
below.
(2) The Transaction Correction Tool
The experts agree, and Mr Coyne confirmed,™"! that the TCT writes to the journal table
identified in JS4, para 12.3.9”
The journal table has been disclosed. The experts agree that it contains only one entry
that shows a transaction being inserted into the BRDB: see JS4, para 12.2.° That usage
of the tool is documented in Peak PC0195561.°* (Mr Coyne noted, correctly, that the
journal also includes entries that relate not to changes to transactions but to changes to
recovery flags.)°"*
POL00026925
POL00026925
%! {Day16/139:14}.
2 (D1/S/14}.
°8 (DL/S/14}.
4 (F/590}.
°45 Day16/143:18}. The full explanation for this is that another tool (which affects recovery flags) writes to
the same journal table — the “Outstanding Recovery Transaction Tool”: see the detail provided in
correspondence at {H/302}.
AI8/268
816.
817.
818.
819.
POL00026925
POL00026925
In his second report, Mr Coyne stated that Peak PC0195962 “suggests that the
modification by Fujitsu support staff to the Horizon Branch Database (BRDB) is not
unusual”; Coyne2/3.223.°4° That Peak is at {F/594}. As Mr Coyne fairly accepted:°4”
816.1 It was opened the day after the one and only use of the TCT to insert a transaction
(12 March 2010).
816.2 It was opened by the SSC employee who had used the TCT, Cheryl Card.
816.3 Ms Card was, in essence, saying that now that the TCT had actually been used,
she could think of some ways to improve it. (See the second entry at {F/594/1}).
816.4 There followed a debate as to how the tool could be improved. The debate lasted
quite a long time. (In fact, the changes were not finalised until January 2011 — see
{F/594/4}.)
The improvements that were made to the TCT templates are summarised in the entry for
8 October 2010 at 16:18: see {F/594/3}. Notably, the Peak records that regression testing
would be used to reduce the risk of any errors in the tool’s templates resulting in adverse
effects on branch accounts and that no such problems had in fact occurred: see the long
entry starting at the bottom of {F/594/1}.
In this light, it is difficult to understand how PC0195962 could ever fairly be said to
suggest that modifications to branch accounts in the BRDB were “not unusual”. In fact,
the document shows that Fujitsu took any modification to accounts very seriously — that
is why, notwithstanding that the TCT had been used only once to insert a transaction,
considerable time and effort was expended in improving the TCT to reduce the risk of
any error arising from its use.
Transaction Repair Tool (not remote access)
Mr Coyne correctly identified the use of the Transaction Repair Tool (“TRT”) as being
relevant in the same way to both Legacy Horizon and Horizon Online. On the face of
6 ¢D)2/4.1/72}.
7 {Day16/140:14} to {Day16/141:22}.
269
AI8/269
JS4, para 10.2, Mr Coyne appeared to suggest that the TRT was used to “insert, inject,
edit or delete transaction data or data in branch accounts”: see {D1/5/4}.
820. Mr Coyne accepted, however, that the TRT was not used to do any of those things.” It
was used to correct data in the TPS. Mr Coyne accepted a change to data in the TPS was
not a change to branch accounts.” He therefore accepted that his second report had been
“misleading” in that evidence of the use of the TRT had been set out under a heading
“Evidence of Insertions/Deletions within Branch Accounts” 2°°
821. Ultimately, Mr Coyne’s contention was that a change to data in the TPS could, indirectly,
result in a change to branch accounts by prompting an erroneous decision to issue a TC.
822. That is a theoretical possibility, but there is no evidence of it ever having happened. And
Mr Coyne fairly accepted the following points, all of which suggest that the risk of it
happening is very low indeed:
822.1 The purpose of the TRT is to ensure coherence between data in the TPS and the
branch accounts and client data.°*! It was ordinarily used to make straightforward
changes to operational data to allow harvesting to take place effectively.
822.2 The TRT could, therefore, only create an inconsistency with the branch accounts
if it was used erroneously so as to alter transaction data.°*? Mr Coyne suggested
that it was “possible” for human error to result in a mistaken change to the
transaction data.?* There is no evidence of this having happened.
822.3 Post Office would then have to prefer the erroneous view of the transaction to the
correct position as shown in the branch accounts on the BRDB and/or in the client
POL00026925
POL00026925
8 {Day16/144:15}.
9 {Day16/48:20} to {Day16/49:24}.
%8 {Day16/66:13} to {Dayl6/68:22}.
%1 (Day16/53:9} to {Day16/54:5}.
92 {Day16/54:10}.
%83 {Day16/54:13}.
A/8/270
823.
824.
825.
826.
data, resulting in a mistaken TC being issued to the branch.?* There is no evidence
of this having happened.
822.4 For the branch accounts to be affected, the branch would have to fail to oppose the
erroneous TC successfully.°** This would be in circumstances where the SPM
would have the benefit of the correct position being identified in the branch
accounts. That is another unlikely step in the chain of causation.
Mr Coyne, with some understatement, accepted that the chance of error being introduced
99956
into branch accounts through the use of the TRT was “/ow”°® and would affect only a
“fraction of a percent” of transactions.”*” He accepted that it would require the
combination of a serious of unfortunate events, none of which was particularly likely.°*
It is important to recall that the use of the TRT is not remote access in the sense of the
Horizon Issues (as set out at the start of this section). It could only be relevant to Horizon
Issues I and 3, where its relevance is very small, for the reasons given above.
Malicious alteration of transaction data
The spectre of malicious changes to transaction data — the “master criminal” theory - has
quietly receded into the shadows. There is no evidence whatsoever of anyone at Fujitsu
or Post Office ever having made changes to transaction data as part of a fraudulent
diversion of money or goods or services. Mr Coyne accepted that he had no reason for
thinking that this had ever happened.°?
Dr Worden’s view has always been that such malicious activity would likely be detected,
not least because it would leave traces for Fujitsu to follow.°® In his cross-examination,
Mr Coyne agreed that malicious use of privileged user rights would necessarily raise
POL00026925
POL00026925
94 {Day16/58:2} to {Day16/59:16}.
°55 {Day16/65:23}.
9°56 {Day16/59:16}.
°57 {Day16/66:12}.
°88 (Day16/65:24} to {Day16/66:5}.
%9 {Day16/76:18}.
%9 See, e.g., IS4, para, 10.14 {D1/5/8}.
A/8/271
827.
828.
829.
830.
831.
alarm bells and trigger an investigation and, accordingly, that it would be professional
suicide for anyone to engage in it.”
Mr Coyne’s reliance on E&Y audits
The audits are addressed in greater detail in paragraph: et seq of these closing
submissions.
Mr Coyne placed reliance on alleged deficiencies highlighted in the 2011 management
letter. Mr Coyne stated that he mostly focused on “/c/hange management and
permissions” in his reports.°°
The 2011 letter specifically states that, with respect to this
area, “/t/he recommendations we have made in this report should be seen as refinements
rather than fundamental control deficiencies in comparison (emphasis added).""°
Although Mr Coyne asserted deficiencies in these areas, by reference solely to the 2011
audit, he failed to comment on the most obvious source for assessing whether changes
were implemented on foot of that audit — subsequent audits. The 2013 management letter
specifically states:
“Focused management action in the past few years has addressed many of the issues
raised in the prior year management letters. Whilst there continue to be challenges
in areas including POL’s IT environment management have taken steps to ensure
these challenges are and continue to be addressed.’
As is apparent from the 2011 letter itself Post Office and Fujitsu responded and acted
upon the points raised by E&Y."
Mr Coyne agreed that no harmful events were noted in the 2011 management letter in
relation to change management and permissions.” Not one instance of irregular or
unauthorised remote access was recorded in the E&Y Post Office audits.
POL00026925
POL00026925
%1 {Day16/72:18} to {Day16/76:3}.
%2 {Day 16/164:5} to {Day16/164:5}.
%83 {F/869/3}.
964 FF/1138/4}.
%S {F/869/29} to {F/869/38} and {F/869/47} to {F/869/49}.
%6 {Day16/165:13} to {Day16/165:17}.
8
8
AI8/272
832.
833.
834.
POL00026925
POL00026925
24>
The purpose and focus of the E&Y service audits are addressed at paragraphs 242 ct
tb of these closing submissions. Reports are available for 2012,°°7 2013, 2014°°
2015,°” 2016”! and 2017.°” These service audits do consider matters relevant to remote
access. Control Objectives 10, 11 and 13 are particularly relevant:
832.1 “Control Objective 10: Controls provide reasonable assurance that access to
system resources, including computing platforms and operating systems, is
restricted to properly authorised individuals.”
832.2 “Control Objective 11: Controls provide reasonable assurance that access to
databases, data files, and programs is restricted to properly authorised
individuals.”°"4
832.3 “Control Objective 13: Controls exist to provide reasonable assurance that remote
access is appropriately restricted to authorised personnel.”?”>
Across the six reports only one deviation was noted across control objectives 10, 11 and
13 — in the 2012 service audit under control objective 10.°” Although certainly not
determinative, these service audits provide a useful resource in evaluating how Horizon
performed against these relevant control objectives at the time.
F25. Permission controls and logs (Horizon Issue 11)
There is a good measure of agreement between the experts on the nature and scope of
the relevant permission controls. It is set out in JS4: see {D1/5/10}. There are several
points that merit more detailed consideration.
987 ¢F/1041}.
968 {F/1176.1}.
989 {F/1305}
70 §F/1562}.
7) {F/1626}.
92 4F/1776.1}.
973 {F/1041/83} to {F/1041/84}.
27" F/1041/85} to {F/1041/86}.
5 {F/104 1/88}.
6 4F/1041/83}.
AI8/273
835.
836.
837.
838.
839.
Privileged user controls
It would be a fair criticism of Fujitsu to observe that, having been made aware of the
problem of SSC users having a more powerful role than was (usually) necessary, it took
a great deal of time for an appropriate resolution to be identified and implemented: see
the long course of the discussion in PC0208119.?”7
It is apparent from that Peak that Fujitsu considered that the unnecessarily powerful role
assigned to SSC users had not resulted in any adverse effects: see the third entry on page
4 —“No actual impact/incidents of problems relating to this issue have been experienced
yet (and not expected)” ?”* It should also be noted that the original fix, proposed in 2011
and implemented for new SSC users in 2012, proved to be inappropriate for existing
users because some of the databases had not been updated to recognise the SSC role
(making that role inadequate for the full spectrum of support activities): see the fourth
and sixth entries on page 7.”
Those two points go some way to explaining the time taken to resolve the problem, but
it nonetheless should have been addressed with a greater sense of urgency. It was
inappropriate for the APPSUP role to be generally available to all SSC users, rather than
having to be granted on request and for a specific purpose.
There is no evidence that the laxity in this regard in fact resulted in any inappropriate
changes being made to the BRDB, let alone that such changes affected branch accounts
and created false shortfalls. This is again an area in which the experts have delved deeply
and not identified any material risk of adverse effects on any given branch account.
Obtaining permission for changes from Post Office
JS4, para 11.1 states as follows: “Evidence from several Peaks indicates that usually
when Fujitsu needed to make any change to data which impacted branch accounts, they
were concerned to seek permission from PO to do so, and to ensure that PO took
POL00026925
POL00026925
77 SF /T68}
O78 ¢F/768/4}.
79 4F/768/7}.
AI8/274
840.
841.
842.
843.
responsibility for the resulting change”. The underlined words were agreed by the
experts during the trial and in replacement of the word “whenever”.
This is an example of the experts’ reluctance to commit to categorical statements. Post
Office respectfully submits that the evidence would in fact support a firmer statement as
to the extent of Fujitsu’s adherence to the permission processes, and Mr Coyne largely
accepted this in cross-examination.
Mr Coyne accepted that he had not seen even a single document providing clear evidence
of Fujitsu adopting a careless attitude to obtaining the appropriate consent from Post
Office.”*° The Peaks addressed above all show Fujitsu seeking and obtaining consent
from Post Office. No reason has been advanced as to why SSC staff would not comply
with the requirement to obtain consent (and the Fujitsu witnesses were not challenged
on their evidence in this regard).
The Court has Post Office’s submissions on Cs’ attempt to adduce late evidence (during
the trial and unsupported by documents) in relation to retrospective consent having been
obtained in some small percentage of OCPs or OCRs. There is nothing to suggest that
there was a sinister reason for Fujitsu to prefer to obtain consent retrospectively (given
that this would generate the same paper trail as prospective consent but would be nothing
like as valuable to Fujitsu in terms of managing its relationship with Post Office). If the
Court were, despite Post Office’s objection, to admit Mr Coyne’s evidence on this point,
it would also wish to have regard to Dr Worden’s evidence at {Day20/147:4} to
{Day20/147:19}.
There is a further short point on what Mr Coyne referred to as “retrospective consent”.
If, as appears to be the case, he simply searched for OCRs and OCPs that are described
on their face as “retrospective”, he will likely have identified many documents that do
not in fact evidence the retrospective grant_of consent but only the retrospective
production of the OCP/OCR document to record that consent. OCP 11903, for example,
is headed “Retrospective OCP for information” but records the grant of consent
(apparently by telephone) in advance of the change being made.**' The purpose of the
POL00026925
POL00026925
98 {Dayl6/161:18} to {Day16/162:23}.
981 {F/296.2}.
Al8/275
844.
845.
846.
847.
POL00026925
POL00026925
OCP in that case appears to have been to formalise the consent that had already been
provided informally.
Fujitsu internal supervision of changes — “four eyes”
The Court will recall Mr Roll’s evidence that Fujitsu required that any change made by
remote access be witnessed by another SCC employee and that, as far as he was aware,
this requirement was always complied with.°*?
In his cross-examination, Mr Coyne raised for the first time the suggestion that some
changes were witnessed by the same person who made the change. There was no proper
explanation for why this was not mentioned in his reports or in the more recent Joint
Statements and had not been raised with Dr Worden.°* Four OCPs”** were put to Dr
Worden as examples of a person apparently witnessing his or her own activity.°*°
As was fairly acknowledged in the course of that cross-examination, there are “/ots” of
OCPs that show clearly that the change was properly monitored. In fact, there are many
thousands. It is now too late for Post Office to be able to obtain evidence from Fujitsu to
explain the apparent breaches of its “four eyes” policy. Taken at its highest, however, the
inexplicably late evidence would only show that there was not perfect compliance with
the policy.
Finally, this section of these submissions has addressed the principal issues which the
experts considered. To the extent that the experts’ views are relevant to other issues,
including evidence given by witnesses, this is considered in the relevant sections of these
submissions and in Post Office's Opening Submissions.
982 {Day4/127:22} to {Day4/128:13}.
°83 {Day16/13:7} and following.
94 F/292.4}, {F/485.2}, {F/540.01} and {F/616.1}.
9°85 {Day20/147:25} to {Day20/151:5}.
AI8/276
POL00026925
POL00026925
POST OFFICE’S OTHER WITNESSES
848.
849.
850.
851.
Mrs Van Den Bogerd
The relevant evidence:
848.1 Mrs Van Den Bogerd’s second witness statement dated 16 November 2018:
(E2/5}
848.2 Corrections made to Mrs Van Den Bogerd’s second witness statement: {E2/16}
848.3 Transcript: {Day5/8:3} to {Day5/197:22} and {Day6/5:5} to {Day6/113:7}.
Mrs Van Den Bogerd’s first witness statement, from the Common Issues trial, was also
added to the trial bundle at Cs’ request on the day Mrs Van Den Bogerd began to give
evidence. It is not clear why. It is at {E4/1}. That evidence has not been tested in this
trial and it would be wrong to make any findings in relation to it.
Mrs Van Den Bogerd is employed as the Business Improvement Director at Post Office.
Her written evidence deals principally with the Claimant-specific witness evidence
adduced by Cs, certain matters addressed in Mr Henderson’s witness statement and
several factual points arising from Mr Coyne’s first expert report. She was, however,
taken in cross examination to many documents which she had not seen before — a
recurring theme of Cs’ cross examination. It is submitted that such an exercise is of very
limited utility.
Mrs Van Den Bogerd’s evidence which is no longer required
As set out in Post Office’s opening submissions:°*°
851.1 Mrs Van Den Bogerd responds to Mr Singh’s evidence in paragraphs 31 to 58 of
her second witness statement. Since Mr Singh was not called by Cs, Mrs Van Den
Bogerd’s response effectively falls away and was not considered at trial.
986 {4/2/27} to {A/2/28}.
AI8/277
852.
853.
851.2 Similarly, in paragraphs 9 to 27 of her witness statement, Mrs Van Den Bogerd
responds to certain points made by Mr Henderson. Given that the Court has
indicated that it will not make findings regarding the opinions that Mr Henderson
summarises or refers to in his evidence,’*’ Mrs Van Den Bogerd’s responsive
evidence need not be considered.
851.3 Finally, in paragraphs 111 to 152 Mrs Van Den Bogerd set out responsive evidence
to some issues raised by some of the Lead Claimants in their witness statements
prepared for the Common Issues Trial. This was because Mr Coyne had made
reference to the evidence of these Claimants in his first report,°** and it was not
known at that time whether Cs proposed to call any of the Lead Claimants to give
evidence again in the Horizon Issues trial. Since these individuals were not called
for the present trial, Mrs Van Den Bogerd’s responsive evidence also does not need
to be considered.
More generally, and perhaps inevitably, Mrs Van Den Bogerd’s evidence on some points
has been effectively overtaken by the more detailed consideration given to those points
in the expert evidence. It was appropriate for her witness evidence to address, for
example, the Helen Rose Report in order to provide early notice of Post Office’s case on
the issues raised by it, but that document has now been the subject of substantial
consideration by the experts.
Mrs Van Den Bogerd devotes a substantial proportion of her witness statement to
responding to the Claimant-specific evidence relied on by Cs (i.c. the witness statements
of Mr Latif, the Patnys, Mr Tank and Mrs Burke): see paras 28-110. °*? That responsive
evidence is addressed in the sections of these submissions that deal with those Claimant-
specific witnesses, and that material is not repeated here. No discourtesy is intended in
observing that Mrs Van Den Bogerd’s responsive evidence, like the Claimant-specific
evidence itself, is of secondary relevance and importance.
°87 Judgment at PTR on 14/2/19, para 10 {C7/41/4}.
988 {2/5/27}; {E2/5/28}: {E2/5/29}; {E2/5/30}; {E2/S/31}; {2/5/31}; {E2/5/32}; (E2/5/33}; (E2/5/34}.
989 ££ 2/5/10}.
POL00026925
POL00026925
AI8/278
854.
855.
856.
Unfortunately, many of the questions and contentions put to Mrs Van Den Bogerd in
cross-examination were on matters outside her direct knowledge and experience. For
example: she was shown Peak PC0065021 and questioned as to a Fujitsu customer call
being closed without permission of the SPM who had raised the problem.” As Cs are
aware, she is neither involved in the generation of Peaks nor aware of the detail of
Fujitsu’s practices and procedures. Mrs Van Den Bogerd was able to observe
(presumably from her experience of working closely with Fujitsu) that the feedback that
says “user error” would not have contributed to the closing of the call.°”' It would have
992
depended on whether other investigations had taken place.’”* But is very questionable
what value this evidence can have — it was really a matter for Mr Parker.
An example of this approach was when Mrs Van Den Bogerd was taken to an audit of
Mr Bates’ branch?’ and to a passage which stated that "A correct assessment of cash
holdings could not be made because the Horizon system intermittently adds the previous
day's cash holdings to the daily declaration.". Mrs Van Den Bogerd explained that this
was nota problem she had seen at any of the branches for which she had responsibility.°"*
The insinuation being made was that this was a problem in Horizon — and Mrs Van Den
Bogerd was understandably not in a position to respond substantively to that, since this
was not a matter covered by her evidence. In fact the process described is a designed
function in Horizon. See:
856.1 The Horizon System User Guide:??°
A daily cash declaration must be produced each day at the close of business or by
19:00 (whichever is earlier) as this information supplies SAP ADS with
information on the total amount of notes and coins held in each stock unit in an
outlet. If a declaration has not been made for a stock unit (for example, if a stock
% {Day5/30:24} to {Day5/32:10}.
1 {Day5/33:9} to {DayS/33:14}.
%2 {Day5/33:9} to {Day5/33:14}.
98 (F/99,1/4}
4 {Day 5/169:5} to {Day 5/169:21}
%§ {F/46,03/8} to {F/46,03/9}.
279
POL00026925
POL00026925
AI8/279
857.
858.
859.
860.
POL00026925
POL00026925
unit is not used on a particular day) the values from the previous declaration will
be carried forward.
856.2 A Horizon manual entitled Balancing with Horizon:°”°
Ifa declaration has not been made for a stock unit on any day, e.g. the stock unit
was not used on that day, the values from the previous declaration will be carried
forward. This is also the case with portions of a shared stock where a declaration
for a particular ID number has not been overwritten with new values or zero’s.
There are, however, several aspects of Mrs Van Den Bogerd’s witness evidence that have
remained important, as they feed into the expert evidence and/or have been given
prominence in Cs’ criticism of her and Post Office’s case.
SPM’s reporting problems and these being investigated.
Para 187 of the second witness statement of Ms Angela Van Den Bogerd states
“Generally when discrepancies are of a value of several hundreds of pounds, I would
expect Subpostmasters to contact NBSC”.2?"7
As set out in these closing submissions, this has been confirmed by several of Cs’
witnesses.°”* It is also readily apparent from a review of Peaks (including some of the
most prominent Peaks in the trial) that problems are often raised by SPMs, and passed
to the SSC for investigation, even where the transaction or shortfall (or gain) at issue is
relatively small — say, in the tens or low hundreds of pounds.
Mr Green QC referred in his cross-examination of Mrs Van Den Bogerd to “UEB” or
“user error bias” as he defined it.” This is not a term referred to by either of the experts.
No source was cited for the statement that “/i/t is where people in IT constantly blame
the user when actually it is not their fault’. Mrs Van Den Bogerd, like Dr Worden, had
never heard of the term.
996 45/59/29}.
7 ¢E2/5/4},
%8 {Day2/125:24} to {Day2/128:6}; {Day2/166:18} to {Day 2/167:3}; paragraphs. 448, 363-and-484.
%° {Day6/51:17} to {Day6/52:3}.
A/8/280
861.
862.
863.
864.
POL00026925
POL00026925
Mr Green QC suggested that blaming the user when the problem was not in fact caused
by user error was a theme at Post Office.!°° Mrs Van Den Bogerd was taken to problems
with recovering transactions and other difficulties encountered by SPMs, and it was
suggested (in essence) that the relevant Post Office employees had been too quick to
blame the user. Mrs Van Den Bogerd answered the suggestion of bias fully and fairly:
861.1 The thrust of her evidence on the various examples given was that the SPMs in
question should, and would ordinarily, have received a better response. She stated,
for example, that it was wrong and atypical for an SPM with a recovery problem
to have been told by the Helpline operator it was for the SPM to resolve the
problem.!°! (The idea that the SPM should resolve the problem him or herself
does not accord with the systems and procedures that are in place for addressing
failed recoveries, as exemplified in Mrs Burke’s case.)
861.2 Mrs Van Den Bogerd nonetheless quite fairly accepted that there were “some
instances” in which Post Office suffered from user error bias.!°
It was suggested to Mrs Van Den Bogerd that her witness evidence dealing with specific
cases points to the likelihood of errors made by SPMs or their assistants (rather than
system errors). Mrs Van Den Bogerd agreed.'°” It was not suggested to Mrs Van Den
Bogerd that she was wrong to have concluded that user error was involved in those cases.
The Claimant-specific evidence is addressed elsewhere in these submissions.
The objective evidence that is now available, from the experts and from the Peaks, KELs
and BIMS reports, shows that SPMs often report small discrepancies that, on their face,
might well be explained by user error but which still find their way to the SSC and are
investigated thoroughly and without any bias (as Mr Roll confirmed).
It is also important to recall in this context that many problems that might indicate the
existence of a bug would be drawn to the SSC’s attention through automatic reporting,
rather than requiring that any SPM spot a problem and raise it through the NSBC.
1000 {Day6/S2:4} to {Day6/52:7}
1001 {Day6/S7:1} to {Day6/58:9}.
1002 {Day6/77:18}.
1003 {Day5/11:22} to {Day5/12:2 }.
A/8/281
865.
866.
Improvements to Horizon - Drop & Go
Mrs Van Den Bogerd gave outline evidence on various improvements to Horizon at paras
156 to 158 of her witness statement.'° This was in response to Mr Coyne having relied
on a Post Office presentation that had observed that relatively small changes could avoid
errors and mistakes in branches. The thrust of the responsive evidence was to point out
that Post Office had in fact made such changes (although it could of course always do
more).
Mrs Van Den Bogerd was cross-examined regarding a small change to the system
processes for Drop & Go transactions. She was referred to an Atos presentation at
{F/1346}. It was suggested, by reference to that presentation that she had painted a
“slightly rosy picture” of Drop & Go in her witness statement.’ There are two
problems with the criticism mounted here:
866.1 First, Mrs Van Den Bogerd did not claim to have provided a summary of any and
all difficulties encountered with Drop & Go transactions. She had merely stated
that there had been two stages of improvements brought in to resolve specific
problems with those transactions;
866.2 Even putting that to one side, and looking at the criticism on its merits, Mrs Van
Den Bogerd’s attention was unfortunately not drawn to those parts of the
presentation that show that the problems to which she was referred were in fact
were investigated and fixed quickly. For example, from “25 Nov, after which
point all issues were well understood and remedial actions invoked”'" and in
terms of the effort put in by the support teams “they /the SC] have been working
later in the evenings to help clear the backlog of locked accounts ”.}°°*
The spectre of big problems in Horizon
POL00026925
POL00026925
1004 {2/5/35}.
1005 ¢F/1346}.
1006 {Day5/117:11} to {Day5/117:13}.
1007 (F/1346/3}.
1008 (F/1346/9}.
wy
Ed
8
AI8/282
867.
868.
869.
Mrs Van Den Bogerd was asked about the planned roll-out of some new reference data
which was to take place in the run-up to a bank holiday. She was taken to an email chain
at {F/1640} in this regard.
She was asked whether she had heard that there was a “risk of a break to the bigger
network” if the roll-out had gone ahead.!°
Mrs Van Den Bogerd was not aware of this,
and it was suggested to her that problems in the background might not find their way to
her level of seniority.'°'° It appeared to be a criticism of Mrs Van Den Bogerd that she
had not been aware of the event. Again, the criticism was wholly unwarranted:
868.1 Mrs Van Den Bogerd is not an expert on IT systems, bugs or testing. It is
unsurprising that she did not come to Court having conducted an extensive review
of all system problems experienced over the years, including those that never went
beyond the test environment.
868.2 In any event, the point was overblown. As Mrs Van Den Bogerd pointed out, even
from just looking at the email chain, it was clear that a decision was taken not to
proceed to roll-out. The model office did its job of identifying potential problems
for the network. That is one of the purposes of the model office — to test new
reference data before it is rolled out. The model office did its job.!°""
A similar issue was raised in relation to the Horizon Online pilot scheme,!°? and the
same points apply. There seemed to be some implied criticism of Mrs Van Den Bogerd
for not having researched the “documentary history” of problems encountered in the pilot
scheme in 2010 before writing her witness statement in 2018.'°'% Mrs Van Den Bogerd
had not even mentioned the pilot scheme in her evidence, let alone purported to address
any problems encountered in it. She was nonetheless taken through an 11-page Peak'*"*
(which she had never seen before) and asked to comment on its various entries.'°!5
100 (Day5/124:7} to {DayS/124:10}.
10 (Day5/124:16} to {Day5/124:20}.
Wl (DayS/124:12} to {Day5/124:15}.
Wl2 (Day5/172:15} to {Day5/176:6}.
113 {Day5/173:6} to {Day3/173:13}.
1014 PCQ195380 {F/588}.
1918 (Day5/173:18} to {Day5/176:6}.
POL00026925
POL00026925
AI8/283
870.
871.
872.
POL00026925
POL00026925
It is questionable how helpful that can be to the Court, given that Mrs Van Den Bogerd
had neither the knowledge nor even the opportunity necessary to provide any meaningful
response in relation to the Peak and the problem to which it related.
If Mrs Van Den Bogerd had been familiar with the Peak, she could fairly have observed
that it indicated that the problems identified by the SPM were addressed urgently, that a
bug was identified within days and that a code fix was tested and then issued to the
network within 2 months. The individual SPM’s experience was of course unpleasant
and regrettable, but the Peak shows the pilot scheme operating well to remove bugs
before the new system went live for the whole network.
Technical points raised in Mrs Van Den Bogerd's written and/or oral evidence
Mrs Van Den Bogerd does not have any technical expertise or experience in relation to
the Horizon system, but she is a fairly sophisticated user of it and has long experience of
addressing problems raised with Post Office. Her written evidence therefore touched on
several matters that can be more appropriately be addressed by Fujitsu witnesses and the
experts. It was fair for these matters to be raised with her, and she gave what answers
she could:
872.1 Mrs Van Den Bogerd stated that although she was aware from Fujitsu that it was
possible to insert a transaction but she had never herself actually seen this
occur.'*'® She had only had known this for about a year (i.e. she had learned it
during the course of these proceedings).
872.2 It was suggested to Mrs Van Den Bogerd that Peak PC0065021 does not show a
great willingness to acknowledge what appear to be independently observed faults
in Horizon.'°!” Mrs Van Den Bogerd replied quite fairly “/n/ot from this, no” \°'S
Mrs Van Den Bogerd’s attention was not drawn to the fact that the Peak indicates
that Fujitsu had spent 200 hours had been spent investigating the problems
identified by the SPM.'°"? It is questionable how much value can be derived from
116 (Day5/50:15} to {Day5/51:24}.
17 (Day5/40:10} to (Day5/40:14}.
18 {Day5/40:10} to [DayS/40:14}.
1019 F/97/10}.
AI8/284
POL00026925
POL00026925
any of her answers based on reading isolated extracts from a document that she
had not read and to which she did not contribute. Phantom transactions are dealt
with elsewhere in these closing submissions.
872.3 It was suggested to Mrs Van Den Bogerd that there is a closure code for Peaks
where a fault has been identified in the Horizon system but Post Office and Fujitsu
agree for the fault to remain in the system'©”° (although no specific closure code
or Peak was provided to her for comment). She replied that she was not aware of
any such closure code.
872.4 Mrs Van Den Bogerd was cross-examined regarding para.98 of her second witness
statement, concerning two TAs received by a branch on 18 January 2018.1! Mrs
Van Den Bogerd’s written evidence was that this was a data entry error rather than
an issue with Horizon. Mrs Van Den Bogerd agreed with the suggestion that the
process for TA was in fact completely automated.'°”* This is mostly correct, and it
was a fair point to put to the witness, but it is not entirely accurate. The TA process
is automated from an early stage, but it does rely on a manual input at the very
start of the process - specific product information has to be supplied in reference
data, for use in generating the TAs. This can be seen from the mistaken TAs issued
to Mr Latif’s branch, where incorrect reference data supplied by Post Office led to
the TAs having the wrong sign. There was human error in the creation of the
reference data used to initiate the automated processes.
872.5 Mrs Van Den Bogerd was asked questions regarding challenging a TA.’ She
stated that an SPM could get in touch with the financial service centre (the
“BSC”),1024
872.6 Mr Green QC also raised mis-keying mistakes.'°> There are of course always
questions as to whether the system (and, in particular, the user interfaces) might
1020 {Day5/83:14} to {Day5/84:21}.
1021 (E2/5/24}.
102 (Day5/67:20} to {DayS/67:23}.
128 (Day5/69:12} to {Day3/70:17}
1024 (Day5/69:22} to {Day5/69:24}.
1025 (Day5/107:5} to {Day5/11:23}.
AI8/285
873.
874,
875.
POL00026925
POL00026925
have better prevented users from making errors. As Dr Worden explained in his
evidence, user interface design is itself a specific (and difficult) area of
expertise!
Cs did not lead any expert evidence on the issue, but there are some
contemporaneous documents that address the point: see, e.g., the Mis-Keyed
Project Feasibility Study at {F/994}.'°’ Mrs Van Den Bogerd did her best to
answer the questions put to her (although she was mostly asked to confirm the
content of documents).
Cost reductions
A recurring theme in Cs’ cross-examination has been cost reduction. It is well-known
that Post Office has generally had to rely on support from government to remain
commercially viable and to sustain a large agency branch network. Controlling costs is
acutely important to the delivery of the services that Post Office makes available to the
public and to Post Office’s ability to offer adequate remuneration to SPMs.
Mrs Van Den Bogerd was taken to {F/555} in relation to the introduction of Horizon
Online. She fairly accepted that there had been a focus on “business equivalence” and
cost-saving. !°** That is a fair summary of the general thrust of the document. However,
Mrs Van Den Bogerd was not taken to those parts of the document that record that there
was consultation with SPMs and that the opportunity was taken to make some
improvements to the system: see, e.g., pp.8-9, 15 and 18.
Mrs Van Den Bogerd was cross-examined in relation to the Branch Support
Programme.'””? Mr Green QC suggested the key performance indicator identified in the
document at $244 was the reduction of operating costs.!°° Mrs Van.
EEL
Den Bogerd stated that “part of’ the KPIs was cost-saving.!°! That much is apparent
from the indicators identified at the top of p.2.
1026 {Day19/93:3} to {Day19/93:15}.
1027 An earlier version is {F/932}. This document is addressed further at para, 379.6514 above.
1028 (Day6/170:3} to {Day6/172:14}.
10 (Day6/5:25} to {Day6/7:18}.
10 (Day6/5:25} to {Day6/6:3}.
14 (Day6/6:10} to {Day6/6:12}.
AI8/286
876.
877.
878.
POL00026925
POL00026925
As Mrs Van Den Bogerd explained, the programme aimed to reduce errors “at the front
end” (i.e. in branches), which would generate costs savings at the “back end” (i.e. in Post
Office).!°? That is to the benefit of both SPMs and Post Office. This project in fact
shows Post Office responding to feedback from SPMs and implementing changes: see
the various changes identified at pp.2-4, including the reference to obtaining feedback
from SPMs (final bullet point on p.2). The pursuit of these cost-savings is not to the
detriment of SPMs; on the contrary, the measures that reduce costs often also confer real
benefits on SPMs.
Call logs
Similarly, Mrs Van Den Bogerd was cross-examined in relation to the provision of
Helpline call logs and whether or not they were routinely provided to SPMs.'°? No
justification was suggested to Mrs Van Den Bogerd as to why these logs should be
offered “routinely”.'°*4 This is not relevant to any Horizon Issue and in any event there
is no suggestion here of a fault in Horizon or even its supporting processes, but simply
an argument as to a business practice and the alleged desirability of providing records
more freely.
Helen Rose Report
Mrs Van Den Bogerd was also cross-examined regarding the Helen Rose report.!°*> She
was asked about the reversal process. Mr Green QC suggested to Mrs Van Den Bogerd
that it was not a reversal initiated by the SPM, but in fact was a system recovery that had
reversed the session.'°*° The user initiated the recovery. The system then generated the
reversal as part of the recovery process. Mrs Van Den Bogerd later clarified that:
“What I meant was that the actual reversal was part of that recovery and it had
actually taken place as it should have taken place, which is what I meant in that. So
it wasn't a failed reversal because it actually had happened as it should have
happened, but I accepted in there that the -- it wasn't obvious to the postmaster at the
182 {Day6/6:17} to {Day6/6:22}.
38 (Day6/14:3} to {Day6/15:6}.
1M (Day6/14:10} to {Day6/14:12}
1035 (F/1082}.
1036 (Day5/82:15} to {Day5/82:19}.
AI8/287
879.
880.
881.
882.
time that what had happened -- that he hadn't -- because it didn't show that he had
actually -- it showed that he had done it and he knew he hadn't done what we referred
to earlier was an explicit reversal. That's what I meant in that.”°*7
Mrs Van Den Bogerd was cross-examined regarding whether or not receipts were
printed. She was asked whether or not receipts were issued for a reversal, and it was
suggested that no receipts were issued.'** This was suggested on the basis of a statement
in the Helen Rose report as follows:
“The fact that there is no indication of such a receipt in the events table suggests the
counter may have been rebooted and so perhaps may have crashed in which case the
clerk may not have been told exactly what to do.”'°*?
Page three, which Mrs Van Den Bogerd was not referred to makes reference to a
receipt.'™° In particular it states:
“The files ... are part of the standard ARQ returned. Rows 141 to 143... clearly show
a reversal. Also row 70 ... shows that session 537803 ... has been recovered and this
event has the same time stamp as the reversal session. Also row 71 of events ... shows
that a receipt was generated from the session 537805 (not explicitly, but it was the
only session at that time)."
In re-examination her attention was drawn to page three of the document.!°*! She stated
that page three indicated that a receipt was produced. The document on its own does not
make entirely clear which receipt or receipts were printed. It certainly cannot support a
suggestion that no receipts were printed.
In any event, Mrs Van Den Bogerd made clear that she had personally seen the printed
receipts.!°? She was taken to the relevant emails in re-examination.'°* These showed
clearly that all appropriate receipts had been printed.
POL00026925
POL00026925
6 (Day6/102:11} to {Day6/103:13}.
038 (Day5/95:19} to {Day5/98:24 }.
1089 (F/1082/2}.
1010 {F/1082/3}. As addressed elsewhere in these submissions, the Report is in fact difficult to follow on this
point.
141 {Day6/92:14} to {Day6/93:15}.
16 {Day5/97:19} to {Day5/98:5}.
118 (Day6/96:3} to {Day6/103:13}.
AI8/288
883.
884.
885.
886.
887.
POL00026925
POL00026925
Conclusion on Mrs Van Den Bogerd’s evidence
Mrs Van Den Bogerd was an open witness and careful witness. She did her best to
respond to questions on the many documents and matters that were not, or not wholly,
within her knowledge or experience: Mrs Van Den Bogerd did her best to assist the Court
on such matters but her commentary on random selected documents is unlikely to
provide a great deal of assistance. She made appropriate concessions and gave fair
answers.
Ms Phillips
The relevant evidence:
884.1 Ms Phillips’ first witness statement dated 28 September 2018 {E2/3}.
884.2 Transcript: {Day6/113:10} to {Day6/147:11}.
Ms Phillips has been working at Post Office in various capacities since 1999. Since 2014
she has worked in the Agent Accounting Team, and since 2016 she has been the Team
Leader for Agent Accounting and Santander Banking. She oversees the process of
recovering the losses that SPMs declare in branches.'°*4 She provided a short witness
statement outlining the balancing process and how discrepancies are disputed.
Disputing discrepancies identified at the end of the Trading Period
Ms Phillips explained in her witness statement that where a discrepancy at the end of a
Trading Period is over £150, the SPM has the option to “settle centrally”. Her
department writes to SPMs to make arrangements for such amounts to be paid back or
deducted from remuneration.!4
Ms Phillips also explained that an SPM is not able to dispute discrepancies on Horizon
or record that they have raised a dispute on Horizon. However, if an SPM calls the team
104 (E2/3/1}, paras 3-4.
1018 (6 2/3/1}, paras 7-8.
289
A/8/289
888.
889.
890.
and explains that they are raising a dispute, a block is placed on the account until the
dispute is resolved.!*
Post Office is concerned at what appears to be a misapprehension on the part of the Court
in relation to this matter. In the Judgment on the Recusal Application, the Court said
this, at para. 299147;
Miss Philips, in her evidence for the Post Office in the Horizon Issues trial, readily
confirmed what had been in dispute for so long during the Common Issues trial,
namely that SPMs had no option but to accept the figures provided to them, even
though amounts may have been “settled centrally”. This is notwithstanding that her
original terminology in her witness statement said that SPMs “chose to accept” TCs.
She accepted that SPMs did not have a choice. It had been necessary for me to make
findings on this very point in the Common Issues trial, and all the Lead Claimants in
that trial (but particularly Mr Abdulla) had been directly challenged on this very point
in their cross-examination. Mr Godeseth also gave evidence in the Horizon Issues
trial that the lack of any ability on the part of a SPM to dispute an item in Horizon
was “by design”. All a SPM could do was to telephone the Helpline. This shows that
the Post Office had been advancing a case, at least for a substantial part of the
Common Issues trial, which was directly contrary to the evidence of its own witnesses
of fact in the Horizon Issues trial.
In fact, Ms Phillips had stated in terms in para. 9 of her witness statement that a SPM.
cannot dispute discrepancies on Horizon. (She does not refer to accepting TCs.) She
explained in cross-examination that when she referred in para 7 of her witness statement
to an SPM choosing to settle centrally, this was intended to refer to the fact that the
choice was between making good and settling centrally.'°* This is obvious on any fair
reading of paras 6 and 7 of her witness statement; it was not a concession made for the
first time in cross-examination. Mr Johnson gives the same evidence in his first witness
statement.'°”
It is important to make clear that it has never been any part of Post Office’s case that
SPMs could raise a dispute through Horizon.
POL00026925
POL00026925
1016 (£2/3/1}, paras 9-10.
1047 (©7/49/76}.
1018 (Day6/114:22} to {Day6/115:9}.
10 (E2/4/18} para, 51.
290
A/8/290
POL00026925
POL00026925
890.1 Para. 43(3) of the Generic Defence and Counterclaim!’ (GDCC”) states as
follows (with added emphasis):
Where the Subpostmaster disputes liability for the shortfall, he or she is required
to raise a dispute by calling the Helpline and in the meantime (if the amount
involved is less than £150) to provide it from his or her own funds pending
resolution of the dispute or (if the amount is £150 or more) to settle it centrally,
thereby bringing the branch accounts into balance. Raising a dispute causes a
block to be placed on the value of the shortfall that has been transferred to the
Subpostmaster's personal account with Post Office. The blocked value is not (and
is not treated as) a debt due to Post Office.
890.2 Para 46(1) of the GDSS states as follows:
It is admitted that there is no "option within Horizon" to dispute a shortfall, in
the sense that the process of raising and resolving a dispute does not take place
through the Horizon system. The process for disputing a shortfall requires the
dispute to be lodged by calling the Helpline.
890.3 Para. 39(6) of the GDCC relates to TCs, which are not addressed in Ms Phillips
evidence, and states as follows (with added emphasis):
If the Subpostmaster wishes to query or dispute the Transaction Correction, he
or she should contact the person identified in the Transaction Correction
notification. This process is identified at page 34 of the Branch Trading Manual.
If, having discussed the matter and reviewed any further information provided by
the person identified, the Subpostmaster wishes _to_dispute the proposed
Transaction Correction, he or she should accept it or settle it centrally and then
lodge a dispute with the Post Office by contacting the Helpline. Where it is settled
centrally, the amount of the Transaction Correction is transferred to the
Subpostmaster's personal account with Post Office and a block is placed of the
amount transferred to the personal account whilst the dispute is resolved.
891. Post Office’s case has been clear and consistent on these points, right from the first
pleading that it produced in this litigation in July 2017. Post Office is concerned that the
Court appears to have taken an adverse view of Post Office based on a misapprehension.
Detail of the dispute process
892. Ms Phillips explained the process which Post Office follows and what letters are sent out
and when. She also explained that her team routinely check the Helpline logs and that
1050 £€3/3/16}.
291
A/8/291
893.
894.
895.
896.
they check at least the larger amounts (generally over £5,000'°*') which have been settled
centrally, but due to the volume of such transactions, they are unable to check every
one.’ The team is provided with a report setting out everything that had been settled
centrally for that month, together with the Helpline logs, although they would call the
branch if there was any issuc.'°* The process is audited by an external firm.'°*
The process described is thorough and fair. There are fairly short deadlines for providing
dispute forms, but it is to be expected that SPMs will provide relevant details promptly
(not least so that time does not allow memories to fade). If an SPM is not satisfied with
a decision made at the end of the process followed by Ms Phillips’ team, support services
are called in and they carry out a further, extremely detailed, investigation.'°°
It would be fair to observe that the processes in this regard have improved over time.
Mrs Mather
The relevant evidence:
895.1 Mrs Mather’s first witness statement dated 16 November 2018: {E2/8}.
895.2 Transcript: {Day6/48:14} to {Day6/175:9}.
Mrs Mather is another very experienced Post Office employee, having worked at Post
Office since 1987. She is a Finance Service Team Leader. She explains in her witness
statement the use made of Credence and that it was used as an information tool, designed
to work alongside and in conjunction with other applications and sources of information
available to Post Office.'°%°
Credence
POL00026925
POL00026925
1051 {Day6/123:24} to {Day6/124:7}.
152 {Day6/121:25} to {Day6/122:11}.
58 (Day6/125:20} to {Day6/126:7}.
4 (Day6/126:8} to {Day6/126:16}.
1055 (Day6/145:1} to {Day6/145:14}.
1056 (£2/8/3} para 12-13.
AI8/292
897.
898.
899.
POL00026925
POL00026925
Mrs Mather clarified in oral evidence that when she said in her witness statement that
Credence recorded every keystroke, she had intended to say that it recorded all
transactional data in terms of sales and non-sales.'°%7
It is thought to be common ground
between the experts that Credence does not in fact contain a record of all keystrokes,
although that information is available to Post Office from Fujitsu.
Mrs Mather was questioned at some length as to the lengths of time for which various
sources of data are retained and the fact that there had been an increase in the retention
period for Credence data. She was understandably not able to assist with all of these
questions, but she confirmed that in relation to cheques (her current area of
responsibility) her team had the information required to investigate.'°°*
Investigations into SPM’s questions and problems
Mrs Mather was clear that her team investigated queries thoroughly:
Q...Now, if you just go over the page at {F/1120/2}, you see at the top of the page: "We
need sight of the actual transaction retrieving from the archives (this is required to
disprove any future claim against Horizon integrity via the clerk)." Was it your experience
at all that there was any defensiveness about the Horizon system when it was challenged
by SPMs?
A. Ifa postmaster challenged in the area that I worked in, we always took them seriously
Q. You did?
A, -- and we always looked at what they were actually saying and then we would go back
to our evidence of issuing the TC.
Q. Okay. Can we look at your paragraph 13 please {E2/8/3}. You can see there, third
line: "For example, a subpostmaster might telephone FSC and/or the helpline and allege
to Post Office that he/she has done a reversal.” Yes?
A. Yes.
Q. That's an SPM claiming to have done something. The way that is expressed suggests
some doubt a little bit, doesn't it, about the truth of what they're saying?
A. I think what I was trying to get across in this particular statement was if a postmaster
had a query, he could always phone us and we would try and help, whatever his query
was. That's what I was trying to explain in this.!°°
187 (Day6/149:12} to {Day6/149:20}.
1058 (Day6/159:16} to {Day6/159:23}.
10 {Day6/160:23} to {Day6/162:1}.
293
A/8/293
900.
901.
902.
903.
904.
Mrs Mather also gave evidence, having spoken to Christopher Knight, Intel & Admin
Team Leader, that there is no evidence of anyone being deterred from making ARQ
requests because of fees that might have to be paid by Post Office.'° Mrs Mather fairly
accepted that this was not within her direct knowledge.
The “till roll” suggestion
Bizarrely, one of the questions put to Mrs Mather implied that, unlike Post Office, the
SPM could only investigate problems in the branch by printing of the transaction log on
the counter printer.!°! That is entirely wrong and flatly contrary to the expert evidence,
which is that the SPM can access over one hundred different reports in the branch,
filtering and presenting transaction data in a variety of ways.'°° The Court will recall
Mr Latif’s oral evidence as to the reports that he could and did use.
Mr Smith
The relevant evidence:
902.1 Mr Smith’s first witness statement dated 16 November 2018: {E2/9}.
902.2 Mr Smith’s second witness statement dated 8 March 2019: {E2/15}.
902.3 Transcript: {Day6/175:12} to {Day6/197:20}.
Mr Smith has worked for Post Office / Royal Mail since 1996. He has worked in a wide
variety of roles and is currently the Operations Support Manager. His responsibilities
include: reviewing data; producing reporting packs detailing current progress and
backlogs against targets; and handling specific product-based problems.'°%
In his second witness statement, Mr Smith corrected information from his first statement
in relation to disputes involving Santander (which had given a figure that appeared to
POL00026925
POL00026925
1060 §£:2/8/4}, para 19. There was some confusion in the cross-examination on this paragraph. Ms Mather had
not spoken to Mr Godeseth; she had spoken to Mr Knight: {Day6/168:17}.
1061 (Day6/168:9} to {Day6/168:16}
1062 (1)2/1/143} and {D3/1/220}.
1063 {£2/9/2} para 10
294
AI8/294
POL00026925
POL00026925
represent disputed TCs but in fact related to disputes between Post Office and
Santander).
TCs
905. Mr Smith outlined the TC process in his first witness statement:
Subpostmasters are able to query or challenge any TC by contacting the person who
issued the TC (whose details are provided with the TC). This can be done within the
Horizon system (by selecting a "seek evidence" option where offered when processing a
TC on a branch terminal) and by calling the person who sent the TC and by calling the
NBSC. Disputes are decided on their merits, having regard to the available evidence. If
the dispute is successful, a compensating TC is issued. '°°*
906. In oral evidence, he explained the interaction between the various departments of Post
Office where a problem may have affected several transactions and, therefore, products:
Q. And you say that: "Although each team have their own responsibilities, there is
interaction between teams to reach a final resolution on any discrepancy."
A. There is.
Q. When would that typically occur?
A. That would occur when there is evidence within the investigation that there could be
a further activity that's needed. As an example, we have already heard about cheques. It
could be that the impact of a cheque discrepancy could also impact on another product,
so if you have a miskey it could be that that miskey transaction has been settled to cheque
with higher than what it should be value, but then there's also going to be the bill payment
or the actual transaction itself that's also been at a higher value. So working together
you can put together all the pieces of the puzzle, get the right answer and also mitigate
or minimise the impact on the branch by making sure that any corrections that are done
are done at the same time to ensure that the financial impact is either mitigated or
reduced."
907. Mr Smith set out in his first witness statement information concerning the level of TCs
in various departments, some of which he obtained from various Team Leaders.'° He
explained why a particular incident described in Coyne 1 which Mr Coyne thought might
have been an error, was in fact properly dealt with. Two TCs were required to remedy a
branch mis-keying error since the method of payment was cheque: one TC was the debit
1064 (£2/9/3} para 15.
1065 (Day6/178:5} to {Day6/178:25}.
1066 (£2/9/3} paras 16-29.
295
AI8/295
908.
909.
910.
O11.
912.
POL00026925
POL00026925
TC to the branch and the other was a credit TC once funds overpaid to the BOI had been
recovered, !°7
Mr Smith also provided details of a new case management system that was being rolled
out to record each challenge to a TC.'° He confirmed in cross-examination, and to
questions from the Managing Judge, that the roll-out process was ongoing at the time of
his oral evidence.
Mr Johnson
The relevant evidence:
909.1 Mr Johnson’s first witness statement dated 28 September 2018: {E2/4}.
909.2 Mr Johnson’s second witness statement dated 16 November 2018: {E2/6}.
909.3 Transcript: {Day7/3:14} to {Day7/27:18}.
Mr Johnson has worked for Post Office since 1984. He is currently a Training & Audit
Advisor and has been in that role since 2012. He started work as a counter clerk and
worked his way up to Branch Manager at Barry Crown Office.'°® He had worked in
Crown Offices throughout his time in branch, although he confirmed that, in terms of
operating the Horizon system, there was no difference between an agency branch and a
Crown office.'°”
In his first witness statement, Mr Johnson describes various features of the current
version of Horizon Online and then briefly describes how the system has changed since
it was first introduced. This evidence was not challenged in any significant way in cross-
examination.
Mr Johnson’s second witness statement addressed points raised by Professor McLachlan
(who was in the event not called by Cs) and Mr Henderson (whose opinions are not relied
1067 {£9/9/6} paras 30-33.
1068 (£2/9/3} para. 16.
100 $£2/4/1} paras 1-3.
197 {Day7/27:10} to {Day7/27:16}.
296
AI8/296
913.
POL00026925
POL00026925
on by Cs). It is doubtful that much of Mr Johnson’s second statement will require
consideration.
User interface design
During cross-examination, it was suggested to Mr Johnson that some of the buttons on
the Horizon screen were too close together. He explained the practical realities during
re-examination:
Q. Thank you. It was suggested to you that -- you were shown one of the screenshots in
your witness statement and it was suggested to you that the buttons were close together.
What you weren't asked about is whether you took the view that the buttons you saw, or
the buttons on the screen -- that their closeness together is a cause of any difficulty in
using the system. Would you like to comment on that question?
A. The fact that buttons are close together can mean that one does occasionally press the
wrong one. However, that is easily rectified.
Q. And when you say it is easily reconciled, why is it easily reconciled?
A. Rectified.
Q. Rectified, I'm so sorry.
A. Ifone would notice that the wrong button had been pressed -- for example, you may
go to sell a first class stamp and you know yourself that the cost of that stamp is 67p. If
you were to hit the first class large stamp, which is next to it, in error, the amount on the
stack would be a different amount, so it would be easily recognisable that you had then
pressed the wrong button.'°”'
Information available to SPMs
914. Mr Johnson’s second witness statement sects out some detail on the data available to
SPMs in the transaction log:
21. The transaction log is a list of all transactions and transfers completed in the
branch, in chronological order.
. The transaction log can be used as a general investigation of all transactions or
it can be filtered by time, value and product. The transaction log records the
transaction that has taken place and also shows how it was settled, for example by
cash, card or cheque. It is therefore possible to see all the transactions where
customers have paid by debit or credit card.
23 Transaction data is only as reliable as what has been manually input into Horizon.
Where an error in recording the transaction has been made by the user, this will be
reflected in the transaction logs. The position was the same before Horizon was
introduced in branches: if a member of staff made a mistake when processing a
transaction that mistake would be reflected in the records of the transaction such as
the paper receipts generated.
10 (Day7/25:1} to {Day7/25:22}.
297
AI8/297
91S.
916.
917.
In response to questions from the Managing Judge, Mr Johnson provided further detail
on the information available to an SPM (even just using the transaction log):
MR JUSTICE FRASER: Why don't you just give me a walk-through of the steps that
you say hypothetically someone in Mr Bates' position should take.
A. So if a postmaster wants to check what transactions have gone through, they would
go to the transaction log. If they wanted to look at a particular -- as I have suggested,
it could be filtered in a number of ways: by date, by product, by value, by stock unit,
by user, there are a number of ways it can be filtered, and obviously the more you can
filter it, the shorter the report will be, and then it would be a question of, as you say,
printing the transaction log out and manually checking each item.\°
MR JUSTICE FRASER: Can I just ask you a question. When you say the filtering
that it can be done, that can be done in branch, can it?
A. Yes.
MR JUSTICE FRASER: Am I right that would only be of any -- that would only be
of any assistance if you knew what you were looking for already, wouldn't it?
A, Not necessarily. A postmaster may have come to the conclusion that a discrepancy
occurred on a particular day by virtue of looking at his cash declaration for the
previous day and the current day, and the previous day had no discrepancy, the
current day does, so he may want to filter then to that particular day when the
discrepancy occurred on his cash declaration.'°"
Armed with extensive practical experience of operating a branch, Mr Johnson was able
to give useful and reliable evidence on points of this kind. His evidence, along with the
experts’ evidence,!°*
is entirely contrary to Cs’ contention that SPMs are left without
effective tools to investigate discrepancies in the branch. The experts both note that a
vast number of reports were available to SPMs in the branch, and Dr Worden notes that
transaction data could be filtered in various ways in the branch.!°”>
The Court will doubtless have noted how many of the problems identified in the
documents were immediately identified by the SPM as relating to a particular transaction
or branch operation (e.g. a problem with a transfer between stock units, a failed recovery,
a duplicated remittance operation, etc). There is an obvious explanation for this. Many
problems will be immediately apparent because the transaction or operation goes wrong
in some visible way. Further, even where a problem is only identified from a discrepancy
POL00026925
POL00026925
172 (Day7/20:18} to {Day7/20:21}.
1078 {Day7/22:3} to {Day7/22:16}.
1074 {1/2/39},
1015 ()3/1/222}.
298
A/8/298
POL00026925
POL00026925
noticed at the end of the day (through the mandatory cash declaration), it can often be
easily linked to a particular transaction because of the coincidence between the value of
the discrepancy and the value of the transaction. Most obviously, very large
discrepancies would often be referable to either a remittance, a stock unit transfer or a
large banking transaction.
H. CS’ OTHER WITNESSES
HI. Claimant-Specific Evidence
918. As noted in Post Office’s Opening Submissions, the Court made it clear that there should
be the “barest” evidence of fact at the Horizon Issues trial. This is why the agreed list of
Horizon issues attached to the Order for Directions dated 23 March 2018 said that there
should be “limited, if any, evidence of fact”,'°"° and why the Fourth CMC Order referred
to below directed that any witness evidence should not be claimant-specific. Thus:
918.1 By para 10 of the Fourth CMC Order dated 5 June 2018,'°"’ the parties were
ordered to file and serve “witness statements of any witness of fact whose generic
evidence (in distinction to Claimant-specific evidence) they wish to rely upon for
the purposes of determining the Horizon Issues”. By the Fifth CMC Order,'°”* the
date for these witness statements was extended to 28 September 2018;
918.2 By para 11 of the Fourth CMC Order, the parties were given permission to file and
serve supplemental witness statements “in response to factual matters that are
referred to and relied upon by either parties’ IT expert in their expert reports”,
such reports being due to be served a few weeks earlier. The idea was there would
no supplemental evidence of a claimant-specific nature, and that this supplemental
evidence would only need to deal with limited points from the
expert reports. By a Consent Order dated 12 November 2018,'°” the date by which
1076 (©7/14/3}.
167 §C7/18/3}
1078 (€7/22/1}.
0 {C7/31/1}.
299
A/8/299
919.
920.
921.
922.
923.
POL00026925
POL00026925
Post Office was to serve supplemental witness evidence was extended to 16
November 2018.
Cs completely disregarded these stipulations.'°*° They have not even attempted to
explain why.
The evidence served by Cs on behalf of Mr Latif, Mr Tank, Messrs Patny and Mrs Burke
is not generic evidence but is “evidence of individual cases” and (except for Mrs Burke)
“Claimant-specific evidence”, i.e. evidence relating to specific examples of individual
experiences with Horizon of the type the parties were directed not to adduce. It is
precisely the sort of evidence that the parties were directed not to give.
With the exception of Mrs Burke’s evidence, much of it is vague as to dates,
circumstances and as to the details of the specific complaints being made. It suffers the
combined vices of being too vague to assist in the determination of the particular C’s
claim (even were that appropriate at this stage) and too specific to assist in the resolution
of any of the Horizon Issues, which are supposed to be generic.
It remains incumbent on Cs to explain the relevance of this Claimant-specific evidence
to the Horizon Issues. To date, no such explanation has been forthcoming. None of these
witnesses give evidence relating to specific bugs (as opposed to an incident they believe
to have been caused by a bug), and their evidence is not relied on by Mr Coyne to explain
his reasoning or to demonstrate a chain of evidence linking a particular SPM’s
experience to an identified KEL or Peak.
Further, this evidence was given without warning and without its proper context being
addressed. Cs appear to be asking the Court to make findings without anything close to
the full story being available:
923.1 None of the claimant witnesses have produced individual Particulars of Claim.
Each of them (i.e. Messrs Latif, Tank and Patny) has simply referred to a few of
the incidents which they say they experienced. They have not set out the entirety
1989 To protect its position, in the limited time available Post Office did its best to say something in response
to Cs’ claimant-specific evidence. It did this under protest. There could be no basis for drawing adverse
inferences against Post Office because that evidence is neither full nor complete. It did not have time to
prepare full evidence and it should not have had to prepare any such evidence in any event.
300
A/8/300
POL00026925
POL00026925
of their claims or addressed their wider experiences as SPMs. For its part Post
Office has responded, as far as it was able to in the time available, to the account
given of those specific incidents. The wider context, which might show for
example that a branch was otherwise well or poorly run, has not been and could
not be explored. This would be important in understanding the relative likelihood
that a particular incident was caused by a bug or user error or some other reason.
923.2 Cs gave no prior notice that they would be calling witnesses of this nature prior to
the witness evidence being served. Essentially from a standing start, Post Office
had only six weeks to assimilate that evidence, investigate the allegations — most
of which were in the vaguest terms, making investigation more difficult — and
produce responsive evidence. The difficulty and effort involved was considerable:
as Mrs Van Den Bogerd explained, it took about three weeks to obtain the relevant
information (ARQ data etc) so that that work could actually begin.'°*!
923.3 Further, the claimant witnesses have not given proper disclosure. In light of the
Court's prohibition of claimant-specific evidence, Post Office had agreed an Order
that Cs need only disclose the documents on which they relied.'°*? Had Post Office
known that it would be facing individual complaints it would have insisted on
proper disclosure from each person giving evidence in the usual way. It is therefore
not known whether the claimant-witnesses have searched the documents in their
possession or control for relevant material. Very little material has in fact been
disclosed by the Cs for this trial (83 new documents) and Cs' solicitors have
refused to give disclosure of documents that have come to light during the course
of the trial. Such documents might be highly relevant but disclosure has been
refused simply on the ground that they fall outside the narrow disclosure Order
that Post Office had agreed without appreciating the evidence that Cs would
subsequently call.'°83
923.4 Post Office did not have an opportunity to properly search its own documents
before giving its witness evidence. The disclosure given on those cases was what
18! (Dayl6/111:17} to (Day16/113:3}.
108 (€7/18/2}.
18 (H/242.6}; {H/255}; {H/280}; {H/284}: {H/297} and {1/303}.
301
A/8/301
924.
925.
POL00026925
POL00026925
could be found in the limited time available. The use of ARQ data at trial was only
possible because Post Office extracted and disclosed that data, all on a voluntary
basis.
923.5 The experts have not analysed the individual cases of the claimant witnesses,
which one would expect to happen before any Court findings in a case about a
technical IT dispute between two parties.
It is to be assumed that Cs consider these witnesses to be their best examples of how
individuals have been affected by Horizon bugs. However, as Ms Van Den Bogerd’s
evidence explains, there are alternative explanations for all of the situations cited by
these witnesses. The Court is not in a position to make findings in relation to which
version of events is to be preferred — and has made it clear that this is not the purpose of
this trial. Post Office submits that the Court cannot properly and should not make
findings as to whether any of these witnesses was in fact affected by a bug in Horizon.
In the circumstances set out above, Post Office also submits that there should also be no
criticism of Post Office for how it has responded to Cs’ claimant-specific evidence:
925.1 There should be no criticism of the nature and completeness of the evidence which
it served in response to Cs’ claimant-specific evidence, given the lack of prior
notice, pleadings, disclosure and the severe time-constraints.
925.2 Post Office’s only aim in cross-examining the claimant witnesses was to establish
that there is a plausible alternative explanation for the matters alleged to have been
experienced by the witness, for example misunderstanding or user error or a failure
properly to follow the procedures required by Post Office.
925.3 Nevertheless, as Cs called the claimant-specific evidence, it was necessary for Post
Office reluctantly to cross examine them and to call its own responsive evidence.
In the following paragraphs, Post Office sets out its submissions on the issues
raised by the claimant-specific witnesses. It does so without prejudice to its
submissions above as to the findings that can properly be made in relation to their
evidence. But one thing the evidence does demonstrate is how important it is to
302
AI8/302
test vague and unspecific claims of the sort made by the claimant witnesses in this
case.
926. Post Office accepts that irrespective of the result of this trial, it is possible that any
individual C, including the claimant witnesses, might succeed in their claim in due
course. Post Office submits, however, that it is not appropriate to make findings about
the existence of bugs affecting the claimant witnesses in this trial. In any event, quite
apart from the considerations discussed above, the sort of clear evidence that would be
needed to justify such findings is not present. On the contrary, if (contrary to Post
Office’s submission) the Court were determined to make any findings as to the existence
of such bugs, the only findings that it could make on the material available would be that
Cs had not shown that they were affected by bugs.
Mr Latif
927. The relevant evidence is as follows:
927.1 Mr Latif’s amended witness statement dated I March 2019 (originally dated 28
September 2018) {E1/1}.
927.2 Mrs Van Den Bogerd’s second witness statement dated 16 November 2018 at paras
85-102 {E2/5/22} to {E2/S/24}.
927.3 Transcript of Mr Latif’s oral evidence {Day2/2:9} to {Day2/98:9}.
927.4 Transcript of Mrs Van Den Bogerd’s oral evidence in relation to Mr Latif’s case:
{Day6/15:7} to {Day6/48:2}.
928. Mr Latif was the SPM for the Caddington branch between October 2001 and September
2018. His witness statement wrongly stated that he was currently the SPM, and he
explained that he had been told not to correct this because it did not matter. !°*
POL00026925
POL00026925
108 See {Day2/93:15} to (Day2/94:6}.
303,
A/8/303
POL00026925
POL00026925
929. Mr Latif’s witness statement raises two matters:'°° (1) an alleged failed transfer of
£2,000 between stock units in or around July 2015 and (2) an alleged problem with
transaction acknowledgments (“TAs”) for Camelot scratch cards in January 2018.10
930. There are four general points to note about Mr Latif’s evidence.
931. First, his witness statement omits basic details. The two matters that he describes in any
detail at all together account for only 9 short paras. The evidence given in the witness
statement is seriously lacking in clarity. Mr Latif added many important details for the
first time in response to cross-examination. He had apparently not been shown Mrs Van
Den Bogerd’s witness statement,'°’’ and he had only very recently asked his assistants
to check the records from the branch (to confirm the date of the TAs in early 2018).'°5*
932. Second, although Mr Latif generally tried to answer the questions put to him in his oral
evidence, he did at times stray away from the question, giving several long narrative
answers that lacked focus. This may be explained in part by the difficulties of giving
evidence by video link, and Post Office does not make serious criticism of this aspect of
his evidence. Nonetheless, combined with the absence of detail in his written evidence,
this made it difficult to identify with precision what factual case Mr Latif in fact wished
to advance.
933. Third, Mr Latif’s account was not supported by any contemporaneous documents,
notwithstanding that these must on his account have existed and, on his own evidence,
were available to him (at least until very recently). Post Office sought disclosure of
further documents during the trial and was told in correspondence that Mr Latif had no
access to documents relating to the branch,'°*’ but that is flatly inconsistent with what
Mr Latif himself said about access to documents in response to cross-examination:
1085 Mr Latif refers in passing to “numerous occasions” of other problems, but he provides no detail in relation
to them, and Post Office was unable to respond substantively to these other points.
1086 Mr Latif’s original witness statement gave a date of March 2018 for the TAs, but he corrected this by
amendment in March 2019: see para 9 {E1/1/2}.
1087 See {Day2/11:8}.
1088 See {Day2/67:15} to {Day2/68:13}.
108 £11/303}.
304
AI8/304
934.
935.
936.
937.
POL00026925
POL00026925
A: Thad a look at the ~ we hold records for the information in the office, so I had my
assistants look at the records, transaction logs and that’s when I confirmed that it
was January rather than March...
Q: When do you say you asked your assistants to check about the date?
A: Yes.
Q: Sorry, when do you say that happened?
A: It was afier I made the initial statement, I was checking.
Q: Roughly when, Mr Latif?
‘A: It would have been a few weeks ago, sir. (emphasis added)'®”
Fourth, as set out in detail below, the records available to Post Office contradict much of
the evidence that Mr Latif has given. On the basis of those records, including most
notably the transaction data, Mr Latif appears to be wrong about both of the two matters
he sets out in his witness statement.
The first complaint: a £2,000 stock unit transfer that “disappeared” in July 2015
Mr Latif’s evidence was that he personally carried out a transfer between stock units AA
and SPI‘! in or around July 2015. He agreed with Mrs Van Den Bogerd’s evidence in
paras 87 to 88 of her witness statement as to how transfers are carried out.!°
Mr Latif confirmed that, when a transfer is performed, a transfer out slip is printed,
signed and retained by the branch, and that the same is true of the corresponding transfer
103 He said that he had the transfer out slip with him when he went to perform the
in slip.
transfer in.!°°* Mr Latif does not refer to these slips in his witness statement, and none
has been disclosed.
The account given by Mr Latif can conveniently be broken down into three sections:
first, the transfer out from stock unit AA; second, the allegedly failed transfer in to stock
1099 {Day2/67:15} to {Day2/68:5}.
109! Mr Latif accepted in cross-examination that the reference in his witness statement to “SJ1” was a mistake:
{Day2/9:21}.
102 {Day2/11:9} to {Day2/12:23}
1093 (Day2/13:13} to {Day2/14:10}.
1094 {Day2/17:6}.
305
A/8/305
938.
939.
940.
941.
unit SP1; and, third, the steps taken by Mr Latif to investigate and report the failed
transfer that he alleges.
First, as regards the transfer out from stock unit AA:
938.1 Mr Latif’s oral evidence was that the transfer out from stock unit AA had been
completed without problem.'°” This coincides with paras 6 and 7 of his witness
statement {E1/1/2}.
938.2 He says that he confirmed this by counting and declaring the cash for stock unit
AA, which showed that the Horizon cash figure and the actual cash figure had both
been reduced by £2,000 (reflecting the transfer and the physical removal of the
cash, respectively).!°° He said for the first time in oral evidence that he also
checked the transaction log for stock unit AA to identify the transfer out.!°°7
As set out in detail below, there is no support for this in the ARQ transaction data. Most
notably, there were no £2,000 transfers out from stock unit AA performed by Mr Latif in
June, July or August 2015.
Second, as regards the allegedly failed transfer in to stock unit SP1:
940.1 Mr Latif’s evidence was that he knew that the transfer in had failed because there
was no transfer in showing on the system for stock unit SP 1.8
940.2 It follows that his evidence is that he did not carry out any transfer in on stock unit
SPI for the £2,000 that he says he transferred out of stock unit AA.
In his oral evidence, Mr Latif said for the first time that he had carried out a reversal of
the transfer out from stock unit AA.'”? He explained the failure to refer to this reversal
in his witness statement as having been related to the passing of his father and his own
wish to keep the statement “‘as simple as possible”. He fairly accepted that he could have
195 {Day2/14:16}
1096 {Day2/16:13} to {Day2/17:2}.
1 (Day2/17:12}.
1098 {Day2/20:15} to {Day2/20:19}.
10% See {Day2/20:19} to {Day2/22:5} and {Day2/37:18} to {Day2/38:1}.
306
POL00026925
POL00026925
A/8/306
942.
943.
POL00026925
POL00026925
added the detail when he amended his witness statement in March 2019, shortly before
the trial.
In relation to the alleged reversal, Mr Latif said as follows:
942.1 He had carried out a reversal of the transfer out from stock unit AA on Horizon
and put the physical cash back in the cash store for that stock unit.!!°°
942.2 Following the reversal, stock unit AA did not balance (showing a £2,000 cash
shortfall).!!°! There was no evidence in the witness statement about the process of
identifying a cash shortfall on stock unit AA, and Mr Latif explained this as being
because the steps he described in oral evidence were “basic logic steps”."? That
explanation is unsatisfactory.
942.3 The Horizon (derived) cash figure for stock unit AA had inexplicably gone up by
£2,000.'!° (Presumably, there must have been an increase of £2,000 in addition to
the £2,000 increase caused by reversing the transfer out).
942.4 He raised this issue with his area manager, Mr Navjot Jando in July 2015.'!°4 Mr
Latif said that this was not mentioned in his witness statement because he did not
think it was relevant.'!> That explanation is unsatisfactory.
Third, as regards the investigation carried out by Mr Latif:
943.1 Mr Latif explained that he checked the CCTV for two reasons: (1) to make sure
that nothing untoward had happened, including that the £2,000 had been placed
back in stock unit AA and not taken from the branch,'!°° and (2) to confirm that
1100 {Day2/39:7} to {Day2/39:15}.
101 {Day2/47:6} to {Day2/47:12}.
102 {Day2/48:23} to {Day2/49:11}.
403 (Day2/51:18} to {Day2/51:23}.
404 (Day2/54:3} to 4
105 (Day2/55:20} to {Day2/55:22}.
106 See {Day2/19:21} and {Day2/23:5;
307
AI8/307
POL00026925
POL00026925
the steps in the process had all been carried out correctly, including pressing the
right buttons on the screen,!!°7
943.2 Mr Latif fairly accepted that the footage would have helped to confirm important
parts of his evidence and would have enabled him to dispute any shortfall.!!°° The
solicitors for Cs have explained in correspondence that the CCTV footage to which
Mr Latif refers was not retained.''”? When asked about the footage, Mr Latif
himself gave a long and tangential answer as to it had not been disclosed (the thrust
of which is difficult to summarise fairly).'"!° Shortly after this, the Managing
un
Judge asked Mr Latif to try to confine his answers to the questions put to him.
It remains unclear why Mr Latif did not take any steps to preserve the footage.
943.3 Mr Latif fairly accepted that Horizon would have provided him with adequate
information to identify a problem with a failed transfer:
Q: Horizon provided you with the information you needed to know that the
transfer out had succeeded, didn’t it?
Q: And you say that you saw immediately on stock unit SP1 that the transfer in
had not succeeded, so Horizon told you what you needed to know there as well,
didn't it?
A: Yes.
Q: And we have already discussed how you could have printed off an
unreconciled transfer report that would have identified any transfer out for
which there was not a transfer in; that’s right, isn’t it?
Q: And you could also carry out reports, or checks on all of the stock units to
see any changes in their Horizon cash figures, couldn't you?
N07 {Day2/22:19} to (Day2/22:25} and {Day2/25:7} to {Day2/25:10}
208 (Day2/26:8}.
1109 £H/303}
10 (Day2/26:13} to {Day2/28:5}.
1M Day2/29:11}.
308
A/8/308
POL00026925
POL00026925
Q: You could also physically check the cash in any relevant stock unit to see
whether it was the same as what the Horizon figure showed, couldn't you?
A: Yes, sir, you can.
Q: So if your evidence were right that the transfer out succeeded but the
transfer in failed, Horizon could tell you everything you needed to know to
confirm that position, couldn't it?
A: I think the issue — yes, sir, but the issue is what’s happened in-between and
I cant [sic] see what’s happened in-between, sir. And that’s - - I think you're
not labouring that comment, that you are saying ~ you are skirting around that,
but that’s the crux of the issue.!""? (emphasis added)
943.4 Ultimately, Mr Latif’s evidence on the information available to him through
Horizon was that there were no limitations on what Horizon could tell him about
the transfers as regards the “front end”, but that it would not tell him about
anything that had gone wrong in the system (a “glitch”) between the transfer out
and the alleged failed transfer in.'!! It is unclear what further information it is Mr
Latif wanted to see and, crucially, how such information would have assisted him:
if the transfer failed, this would have been apparent from the reports available in
the branch; if the transfer somehow duplicated (or did anything else with effects
on the branch accounts), this would also have been apparent from those reports;
all that would not be communicated was technical information about any bug that
resulted in those effects, but the experts agree that “the extent to which any IT
system can alert its users to bugs within the system itself is necessarily limited”!
943.5 Mr Latif said that he phoned the Helpline, but that the Helpline was not very
good.'''> He was, however, taken to call logs showing two examples of his
assistants calling the Helpline to ask how to identify transfers through Horizon,
and he accepted that they had been told how to show transfers on reports that could
be printed in the branch.'"* On the basis of those call logs, it is clear that the
412 (Day2/30:1} to {Day2/31:4}.
413 (Day2/35:15} to {Day2/36:17}.
11M Joint Statement 2, para 21 {D1/2/38}.
415 See {Day2/32:12}. However, Mr Latif’s evidence on whether or not he called the Helpline was not
entirely clear. He said at {Day2/40:14} that he may in fact have called his area manager, rather than the
Helpline, but that it was a long time ago (presumably meaning that his recollection on this point may be
imperfect). He later appeared more confident that he had called the Helpline: {Day2/56:1}.
M16 {Day2/32:24} to {Day2/35:4}.
309
A/8/309
POL00026925
POL00026925
Helpline had proven itself able to provide effective assistance in relation to
problems with transfers. In any event, as identified below, there is no record of the
call that Mr Latif alleges: see paragraph 944.1 below.
944. Post Office makes three main points about Mr Latif’s evidence in relation to alleged
failed transfer:
944.1 The transaction data and other contemporaneous records are inconsistent with Mr
Latif’s account. Not one of the key elements of his account is supported by the
records produced at this trial.
944.2 Mr Latif’s evidence in relation to the alleged transfer is unsatisfactory. He failed
in his witness statement to mention many of key elements of his account as it
emerged at trial, and the account that he gave at trial is unlikely to be accurate.
944.3 Mrs Van Den Bogerd’s evidence on the transfers is consistent with the transaction
data and other records and was not materially undermined in cross-examination.
945. These three points are expanded upon below.
Point 1: The transaction data and other contemporaneous records are inconsistent with
Mr Latif’s account.
946. The transaction data for June 2015 show no transfers of £2,000 between stock units AA
and SP1 performed by Mr Latif (or any of his staff).'"!7
947. The data for July 2015'!'* show the following two relevant transfers of £2,000:
Transfer 1
17 (/1353.1}.
nis
310
A/8/310
948.
947.1 A transfer out of £2,000 from stock unit AA on 21 July 2015 and timed at 13:52:22,
performed by Christine Barnett:!! see row 18352. Column J shows the product
code 6277, which is the code for a transfer out.!!7°
947.2 A corresponding transfer in of £2,000 on stock unit SP1 on 21 July 2015 and timed
at 13:55:01, performed by Christine Fensome:!!”! see row 18359. Column J shows
1122
the product code 6276, which is the product code for a transfer in.
Transfer 2
947.3 A transfer out of £2,000 from stock unit SPI on 29 July 2015 and timed at
11:49:19, performed by Robert Deacock:!!* see row 25988. Column J shows the
product code 6277.
947.4 A corresponding transfer in of £2,000 on stock unit SP1 on 3 August 2015 and
timed at 11:50:30, performed by Mohammed Latif:'!** see row 25990. Column J
shows the product code 6276.
The data for August 2015''?> show four transfers from stock unit AA to stock unit SP1,
each of them having both a transfer out and a corresponding transfer in. Mr Latif did not
wish to see this data, accepting that it would show four such transfers.'!2° For
completeness, the relevant transfers are as follows:
Transfer 1
POL00026925
POL00026925
419 User ID CBAOO1 is Christine Barnett: see the list of assistants at {F/1038.1}.
120 See the Product ID List {F/1292.2/1} (towards the bottom of the table).
121 User ID CFE002 is Christine Fensome: see the list of assistants at {F/1038.1}.
42 See the Product ID List {F/1292.2/1} (towards the bottom of the table).
423 User ID RDE001 is Robert Deacock: see the list of assistants at {F/1038.1}.
424 User ID MLAO01 is Mohammed Latif, Mr Latif’s brother: see {Day2/64:6}. He is not recorded as an
assistant in the log at {F/1038.1}
125 (F/1371.1}.
126 (Day2/66:5}.
A/8/311
948.1 A transfer out of £2,000 from stock unit AA on 1 August 2015 and timed at
11:56:51, performed by Mohammed Latif:!!27 see row 526. Column J shows the
product code 6277.
948.2 A corresponding transfer in of £2,000 on stock unit SP1 on 1 August 2015 and
timed at 11:57:22, performed by Robert Deacock:!!* see row 528. Column J
shows the product code 6276.
Transfer 2
948.3 A transfer out of £2,000 from stock unit AA on 3 August 2015 and timed at
16:38:25, performed by Christine Barnett:!'?° see row 2402. Column J shows the
product code 6277.
948.4 A corresponding transfer in of £2,000 on stock unit AA on 29 July 2015 and timed
1130
at 19:40:11, performed by Muhammed Tabassum: see row 2488. Column J
shows the product code 6276.
Transfer 3
948.5 A transfer out of £2,000 from stock unit AA on 5 August 2015 and timed at
16:25:40, performed by Christine Barnett:''*! see row 4749. Column J shows the
product code 6277.
948.6 A corresponding transfer in of £2,000 on stock unit SP1 on 5 August 2015 and
timed at 16:26:21, performed by Michael Brumwell:!'** see row 4751. Column J
shows the product code 6276.
POL00026925
POL00026925
427 User ID MLAOO1 is Mohammed Latif, Mr Latif’s brother: see {Day2/64:6}.
128 User ID RDE0O1 is Robert Deacock: see the list of assistants at {F/1038.1}.
429 User ID CBAO01 is Christine Barnett: see the list of assistants at {F/1038.1}.
480 User ID MTAOO1 is Muhammed Tabassum: see the lists of assistants at {F/1038.1}.
431 User ID CBAO01 is Christine Barnett: sce the list of assistants at {F/1038.1}.
432 User ID MBROO1 is Michael Brumwell: see the list of assistants at {F/1038.1}.
AI8/312
949,
950.
Transfer 4
948.7 A transfer out of £2,000 from stock unit AA on 26 August 2015 and timed at
14:46:05, performed by Christine Barnett:'!*> see row 23538. Column J shows the
product code 6277.
948.8 A corresponding transfer in of £2,000 on stock unit SP1 on 26 August 2015 and
timed at 19:36:15, performed by Michael Brumwell:'!** see row 23833. Column J
shows the product code 6276.
The August data also show a transfer from stock unit SP1 to stock unit AA (i.e. a transfer
in the other direction), as follows:
949.1 A transfer out of £2,000 from stock unit SP1 on 3 August 2015 and timed at
09:12:32, performed by Robert Deacock:!'* see row 1210. Column J shows the
product code 6277.
949.2 A corresponding transfer in of £2,000 on stock unit AA on 3 August 2015 and
timed at 10:16:30, performed by Mohammed Latif:'"°° see row 1353. Column J
shows the product code 6276.
Two key points emerge from the transaction data for June, July and August 2015:
950.1 There were no transfers of £2,000 out of stock unit AA or into stock unit SP1
performed by Mr Latif. The data contradicts Mr Latif’s evidence that he personally
carried out a transfer out from stock unit AA: see para bove.
950.2 For each £2,000 transfer out of stock unit AA, there is a corresponding transfer in
on stock unit SP1. The data contradict Mr Latif’s evidence that he carried out a
transfer out from stock unit AA and was then unable to accept the corresponding
transfer in on stock unit SP1: see para 940 above.
950.3 There were no reversals by Mr Latif in June 2015.
POL00026925
POL00026925
33 User ID CBAOO1 is Christine Barnett: see the list of assistants at {F/1038.1}.
"4 User ID MBRO01 is Michael Brumwell: sec the list of assistants at {F/1038.1}.
135 User ID RDEO01 is Robert Deacock: see the list of assistants at {F/1038.1}
136 User ID MLA001 is Mohammed Latif, Mr Latif’s brother: see {Day2/64:6}.
313,
AI8/313
951.
952.
953.
954.
When asked about the two transfers in July 2015, Mr Latif said that there may have been
corresponding transfers out and transfers in, but that the transfers may not have been
“successful” in the sense of changing the Horizon (derived) cash positions of the relevant
stock units. He said this could have happened because of a “glitch”.!'57 There are two
fatal problems with that suggestion.
First, it is not what Mr Latif in fact says happened. Mr Latif’s evidence is not that he
accepted the transfer in on stock unit SP1 but it did not take effect so as to increase the
Horizon (derived) cash figure for that stock unit; his evidence is that the transfer in was
not available to accept on stock unit SPI: see para 940 above.
Second, each of the transfers out and transfers in identified above is accompanied in the
transaction data by the appropriate (inverse) accounting entry that adjusts the Horizon
cash figure for the relevant stock unit. These are the entries on the spreadsheets that
appear with the Product ID 1 (cash) in column J and that are in the same amount as the
transfer but which carry the inverse sign of the transfer (adjusting the derived cash
position down for a transfer out or up for a transfer in).'"**
This was addressed briefly in cross-examination:
954.1 The Managing Judge asked about these entries during Mr Latif’s cross-
examination and was told as follows (on instructions but in counsel’s own words):
... They are not transfers. They correspond to the value of the transfer but they
are not actually transfers. They perform a back office reconciliation function
that wouldn't be visible to a Subpostmaster. If Mr Latif had called up records
in his branch he would have seen the two transactions to which I have referred
you, he wouldn’t see the effective inverse transaction{s] that are only visible in
the back office systems.'°
954.2 Mrs Van Den Bogerd was taken in cross-examination to two of the entries and
explained that represented the cash value of the transfer for the relevant stock unit:
POL00026925
POL00026925
1137 (Day2/65:4} to {Day2/65:9}.
1138 gee (F/1371.1} rows 3384-3386 for an illustration of the position where there is a mixture of cash and
stock.
489 (Day2/63:1} to {Day2/63:14}.
314
AI8/314
955.
956.
POL00026925
POL00026925
Q: ...When Mr Latif was being cross-examined about this....it was suggested to
him that the transactions either side of these two entries were back office entries
that he didn’t need to worry about.{''*°I Actually, code I is cash, isn’t it?
A: That’s right.
Q: So that’s an important part of the transaction?
A: Yes.
There is nothing surprising about the fact that the double accounting cash entry is not
shown on branch reports. The effect of the entry is of course visible (in that the Horizon
derived cash figure goes up or goes down), but the entry itself is no more visible than
the equivalent entry for any sale. The SPM would not expect to see the double entry
figures for every sale of stamps (i.e. one entry increasing the derived cash figure and
another separate entry decreasing the derived stock figure): he would expect to see a
record of the sale of the stock, including its cash value where appropriate.''! He would
see the same for a transfer: the detail of the transfer along with its cash value.
The information provided to SPMs about a transfer can be seen on the sample transfers
summary report shown at {F/856/520}.''” For each transfer, the report gives the
following information:
956.1 The session number in which the transfer occurred.
956.2 The source stock unit (“SRC”) for the transfer.
956.3 The destination stock unit (“DEST”) for the transfer.
956.4 The date and time of the transfer.
956.5 The value of the transfer.''**
1140 This is the passage quoted above. Mr Green QC’s summary of the passage is inaccurate.
141 For "value stock items" (such as open postage and currency), you would see the cash value, but for
“volume items" (such as stamps), you would see the volume (i.e. no value).
42 Branches can also produce and print out a transfer reconciliation report: see {F/856/586} (2011) and
{F/1782/158} (2018).
M8 See footnote
[142 above.
315
AI8/315
957.
958.
Mr Latif’s evidence in relation to a call to the Helpline is also contradicted by the
available records. Specifically:
957.1 When Mr Latif first mentioned this call, he was confident that there would be a
record of it in Post Office’s call logs.'!*4
957.2 Mr Latif was shown that there was no such record on the call logs for June to
August 2015.!"45
957.3 When asked why there was no record of such a call in those logs, Mr Latif first
said that he may in fact have called his area sales manager, rather than the
1146 and then, after being taken to the call log, added that the operators
Helpline
sometimes failed to record calls.''*7 If (contrary to Post Office’s submission) it
were appropriate to make findings about this, the more likely explanation would
be that the call, which was not mentioned at all in his witness statement, did not
occur.
In re-examination, Mr Latif was taken back to the call log. The purpose of the re-
examination appeared to be to create the impression that the questions in relation to the
call log had been somehow unfair or misleading. Mr Green said that Mr Latif was told
that the entry at row 89 on the spreadsheet at {F/1829.1} had a “date of 29 June 2015
which is the first entry on this log for June”.!\*8 Mr Latif was then taken on a filtered
spreadsheet to row 95, which he was told was an entry for “22 June”. This was a bad
point, for three reasons.
958.1 If the spreadsheet at {F/1829.1} is filtered by “created date”, the entry at row 95
is for 22 June 2016. The call that Mr Latif alleges he made was in or around July
POL00026925
POL00026925
14 (Day2/32:3} to {Day2/32:15}.
145 See the call log at {F/1829.1} and the cross-examination on it at {Day2/40:15} to {Day2/44:16}.
146 (Day2/40:5} to {Day2/40:14}.
147 (Day2/44:13}.
148 The log had not been filtered to put the entries in date order. The call logs are of course not stored in
spreadsheet format but are downloaded into Excel and have to be ordered through using the “sort and filter”
function.
316
AI8/316
959.
960.
961.
2015. Mr Latif was taken in cross-examination to the call records for June to
August 2015,'!° given that this was the relevant period.
958.2 The calls were not shown on the spreadsheet in chronological order, and the
suggestion from Post Office was that the call on 29 June 2015 was the first shown
on the spreadsheet, rather than the first in time. If the spreadsheet is filtered by
“created date”, it shows for June to August 2015 precisely the same 9 entries as
Mr Latif was shown in cross-examination (appearing at rows 45 to 53).
958.3 In any event, Mr Latif was not taken in re-examination to any entry on the log that
was (or even could be) said to provide evidence of the call to which Mr Latif
referred in cross-examination. There is no such call recorded in June, July or
August 2015.
If the Court were minded to make findings about this, Post Office would submit that, on
the present evidence, it is likely that the call did not happen and that Mr Latif’s
recollection on this point was mistaken. Mr Latif accepted that his recollection might not
be reliable, given the passage of time:
Q: Which do you say you phoned, the area sales manager or the Helpline?
A: I believe it was the Helpline, but it’s a long time ago.''*°
The suggestion that Mr Latif had contacted his area manager (Navjot Jando) was made
for the first time at trial, and Post Office had no evidence from Mr Jando.
Lastly as regards the contemporaneous records, there is the alleged reversal of the
transfer out from stock unit AA. Mr Latif mentioned this for the first time in cross-
examination. The transaction data and event data show no reversal of a transfer out on
stock unit AA carried out by Mr Latif in June, July or August 2015. Mr Latif was unable
to explain this.!!5!
POL00026925
POL00026925
4 (Day2/33:1} to {Day2/33:4}
4199 (Day2/40:12}.
451 (Day2/38:2} to [Day2/38:22}.
317
AI8/317
962.
963.
964.
Point 2: Mr Latif’s evidence in relation to the alleged transfer is unsatisfactory.
There are many important elements of Mr Latif’s evidence that emerged for the first time
in response to cross-examination. Specifically, the following important matters were not
mentioned in his witness statement:!!?
962.1 The alleged call to the Helpline to report the failed transfer of £2,000. As noted a
above, Mr Latif was at times more and less sure as to whether or not he in fact did
call the Helpline (see the passage quoted at para 243.4959 above).
962.2 The alleged call(s) made by Mr Latif to his area manager, Mr Jando.
962.3 The alleged reversal of the transfer out from stock unit AA.
962.4 The alleged cash declaration conducted on stock unit AA after the reversal which
showed a £2,000 shortfall in that stock unit. There is no mention of this in the
witness statement despite reference to another (earlier) cash declaration on that
stock unit: see para 7 of the witness statement {E1/I/2}.
962.5 The allegation that Mr Latif checked the CCTV in the branch to confirm not only
that the right buttons were pressed (which is what would ordinarily be taken from
the third sentence of para 7 of the witness statement {E1/1/2}) but also that nobody
had taken the £2,000 from the branch: see para 943 above.
Post Office had to extract the detail of Mr Latif’s account through cross-examination.
That is unsatisfactory, and it was unfair to Post Office that it did not have proper advance
notice of Mr Latif’s evidence. Despite the vast resources expended by Cs on this
litigation, little attention seems to have been devoted to identifying and explaining the
basic elements of their own claims.
In addition, there are the following problems with the account that Mr Latif gave in his
witness statement and on Day 2 of the trial.
POL00026925
POL00026925
1152 Post Office of course understands that some of the inadequacies of Mr Latif’s evidence may be explained
by the sad loss of his father. Some allowance must be made for this. It cannot fully explain Mr Latif’s failure
to provide a properly detailed account of his complains in his written evidence, and he gave several different
reasons for not having mentioned key parts of his case.
318
AI8/318
965.
966.
967.
968.
969.
First, as noted above, it is contradicted by the transaction data and other
contemporancous records.
Second, Mr Latif’s evidence in relation to the CCTV was very unsatisfactory. His
evidence that he checked the CCTV to confirm that nothing untoward had happened to
the £2,000 is one thing,''* but his evidence that he checked the CCTV to make sure that
nothing untoward (including theft) had happened is quite another. In circumstances
where he had personally carried out the transfer out and the reversal and had personally
retained custody of the physical cash throughout the process, this evidence makes little
sense. This point was put in cross-examination, and Mr Latif could provide no real
answer to it.!'*4
Third, the account that Mr Latif now gives is inconsistent, in important respects, with
what is said (and confirmed with a statement of truth) in his Schedule of Information
(“SOI”). The SOI includes the following complaint:
In approximately July 2015 somehow Horizon lost £2,000. According to the system
this was transferred from one stock unit to another but this did not happen as the
second stock unit was not £2,000 up. This could not be explained and our CCTV was
checked to make sure the money had not physically been removed by staff. In the end
Thad to put the cash in myself?5
The account in the SOI suggests that the shortfall arose on stock unit SP1 (the receiving
stock unit), rather than stock unit AA. It also appears to rely on the transfer having
succeeded, but the physical cash having gone missing. It contains no mention of an
unexplained movement in the Horizon cash figure for stock unit AA, which was at the
core of the account given by Mr Latif in oral evidence (in the sense that, without that
unexplained change, there cannot have been any shortfall).
Fourth, the account given by Mr Latif in oral evidence could only be correct if there were
several different bugs within Horizon, all of which struck in unison to bring about the
factual circumstances that he alleges. As is explained below, there is no evidence
suggesting the existence of any of the bugs having the effects that his account implicitly
POL00026925
POL00026925
13 Tt is also consistent with the content of Mr Latif’s Schedule of Information at para 3.1 {F/1654.2/4} (see
further below).
1184 (Day2/19:5} to {Day2/20:19} and {Day2/22:6} to {Day2/25:10}.
"SS (F/1654.2/4}at para 3.1.
319
AI8/319
970.
971.
POL00026925
POL00026925
requires, despite the experts having had the opportunity to consider his claims (including
by searching for relevant Peaks and KELs and other contemporaneous documents).
The first hypothetical bug is required to have caused the transfer in for stock unit SP1 to
be unavailable to accept. None of the bugs identified by the experts is relevant here (as
to the Callendar Square and Dalmellington bugs, see para 982 below).
The second hypothetical bug is required to explain an otherwise unexplained increase in
the derived cash figure for stock unit AA (without which there could be no shortfall
relating to the allegedly failed transfer — see para 972 below). This can be shown by
tracing the changes that would have occurred, on Mr Latif’s account, to the Horizon
derived cash figure (“Horizon cash”) and the actual cash in the relevant cash store
(“actual cash”) for stock unit AA, assuming that both figures started at £10,000:
971.1 Mr Latif transfers £2, 000 out of stock unit AA and removes the £2,000 in physical
cash. Horizon cash is now £8,000; actual cash is now £8,000. The stock unit
balances. This was confirmed by Mr Latif to be what happened (using the same
hypothetical figures).!1°
971.2 The transfer into stock unit SP1 is not processed. Mr Latif returns the £2,000 in
physical cash to stock unit AA. (Mr Latif confirmed that he physically returned
the cash.'!°’) Horizon cash should still be £8,000; actual cash is back to £10,000.
There should be a £2,000 surplus in the stock unit, and the branch would have been
alerted to a receipts and payments mis-match when it tried to balance. Mr Latif
appeared to confirm that stock unit AA showed a surplus at this point, although in
fairness the suggestion put to him was that the stock unit “would” show a
surplus.! 58
971.3 Mr Latif carries out a reversal on stock unit AA to reverse the transfer out. Horizon
cash should now return to £10,000; actual cash will be unchanged at £10,000. The
stock unit balances.
456 (Day2/16:13} to {Day2/17:2}.
497 (Day2/47:21} and {Day2/51:2}.
458 (Day2/39:18}.
320
A/8/320
972. Mr Latif accepted that, in order for there to result a shortfall in stock unit AA, the Horizon
973.
cash figure would have had to have increased by £2,000 for no valid reason.''*? If that
had happened — a change in stock unit AA’s Horizon cash figure that was not caused by
transactions on that stock unit — Mr Latif would have had ample evidence to demonstrate
this. On his case, he would have had the following:
972.1 The transfer out slip for stock unit AA.
972.2 The record of a reversal of that transfer out.
972.3 The “transfer log” that Mr Latif produced at some point after the failed transfer.!'
972.4 The cash declaration performed on stock unit AA immediately after failed transfer:
see witness statement at para 7 {E1/1/2}."'6!
972.5 The transaction log for stock unit AA that he printed out at some point after the
transfer out.''
972.6 The balance snapshot printed after the transfer to SP1 failed.'
972.7 The cash declaration performed on stock unit AA that showed a £2,000
shortfall.!'°4
972.8 The transaction log for stock unit AA over the whole period covered by these
actions, showing the transactions on that stock unit, allowing the implied
movements in the Horizon cash figure to be tracked.
Relying on these records, Mr Latif would have been able to show the £2,000 change in
the Horizon derived cash figure for stock unit AA and point to the absence on the
transaction log of any transaction to explain that movement in the Horizon cash figure.
Mr Latif was asked why, in light of this, he did not dispute the £2,000 shortfall that he
POL00026925
POL00026925
139
160
{Day2/52:22}
{Day2/48:6}. Mr Latif could have been referring to cither the transaction log (filtered to show only
transfers) or a transfers report.
nel
162
1163
164
Mr Latif would have been able to view the cash declaration on screen and would have had the option to
print a copy too.
{Day2/17:14}.
{DayQi4eis
{Day2/S1:10}.
A/8/321
POL00026925
POL00026925
alleges. At first, he suggested that he had not disputed the shortfall because he considered
that he was liable for losses caused by glitches in the system:
Q: Do you say that you did this without disputing this shortfall, Mr Latif?
A: We are liable. The Post Office’s contract clearly says that we are liable for any
shortfalls.
Q: Is it your understanding that you are liable for a shortfall even if it is a computer
glitch, if that is what you are saying?
A: Yes, sir, we're liable.
974. Mr Latif was pressed on this point, and he returned to his earlier evidence that he had
reported the problem to Post Office. He said that he had failed in the past to persuade
Post Office that there had been software errors in his branch (none of which are identified
in his witness statement) and that Post Office had not accepted that there were such
software errors, blaming user error instead:
Q: .. If you genuinely believed that the derived cash figure on stock unit AA had been
increased by a glitch and that therefore you were in no way responsible for any of
this, what do you say to the suggestion that it's very surprising that you wouldn't raise
that with Post Office and complain about it?
A: Well, Ihave complained to the area manager, sir, so I don't know [why] your saying
I haven't complained. I have complained to the area manager, Mr Navjot Jando, a
number of times, so I don't know why you're saying I haven't complained about it or
raised it.
Q: So do you now say you disputed a £2,000 shortfall in July 2015 by contacting your
area manager and complaining that there was a glitch; is that what you are now
saying?
A: Well, yes, we would have obviously raised questions, but there is a glitch or
something and we don't know what's happened. This is just one instance, sir, but there
are a number of other instances which I haven't given in my statement. It happens all
the time and generally we think it's the operator that's causing the problem and that's
what the Post Office keep telling us, it's operator error, not necessarily it's a software
error, and this is clearly a software error, sir...''°
975. But this is precisely the point — on his account, Mr Latif would have been able to show
very clearly that there had been an unexplained jump in the Horizon cash figure for stock
unit AA. He would be able to point to the absence of any entry on the transaction log to
6 (Day2/53:11} to {Day2/53:18}.
16 {Day2/53:21} to {Day2/54:20}.
AI8/322
976.
977.
978.
POL00026925
POL00026925
explain the change in the figure. He would have been able to show something must have
gone wrong in the system.
It is important to note that this second hypothetical bug would be very different from any
of the bugs identified by the experts. There is no known bug that somehow affects the
Horizon cash figure other than through a transaction that would show in the transaction
data for the affected branch. There is no trace of any unexplained and erroneous £2,000
transaction on stock unit AA.
Further, for Mr Latif’s oral evidence to be correct, there would have to be yet more bugs
in Horizon that came together to affect his branch, as follows:
977.1 A third bug that has somehow removed the record of the reversal that Mr Latif
says he carried out on stock unit AA. The experts have not identified any bugs that
would create this effect.
977.2 A fourth bug that has erased all record of Mr Latif personally having carried out
£2,000 transfers out of stock unit AA and into stock unit SPI in or around July
2015. The experts have not identified any bugs that would create this effect.
977.3 A fifth bug that erased all record of the £2,000 shortfall that Mr Latif alleges (e.g.
in the balance snapshot and cash declarations that he says he carried out or in a
trial balance). The experts have not identified any bugs that would create this
effect.
Point 3: Mrs Van Den Bogerd’s evidence on the transfers is consistent with the
transaction data and was not materially undermined in cross-examination.
Mrs Van Den Bogerd explains that she has, with the help of her team, considered the
transaction and event data for Mr Latif’s branch from June to August 2015. She does not
purport to give a final and certain explanation of events. Her conclusion is suitably
limited: “the records that Post Office has reviewed do not support what Mr Latif has said
and I believe that he may have mis-recollected events from 3 years ago”.
167 ££ 2/5/24} at para 95.
323
AI8/323
979.
980.
981.
Mrs Van Den Bogerd faced criticism for the fact that she had to make corrections to her
witness statement to MORE accurately reflect the transaction data on which she had
relied.!"©* Some criticism was therefore justified, but in this regard the following points
should be noted:
979.1 It is important not to lose sight of the context in which Mrs Van Den Bogerd found
herself having to respond, in short order, to claimant-specific evidence covering
many years and raising many discrete issues. It was not surprising that some small
mistakes were made.
979.2 Mrs Van den Bogerd made clear that members of staff within Post Office had
helped her examine the transaction data!'® and that those members were more
experienced in examining the back-office accounting data.'!”° She explained, for
example, the slightly round-about way in which she personally had used the
transaction data and event data together to identify the relevant transfers in and
transfers out.!!7!
979.3 Ultimately, the question is what the transaction and event data show. The Court
can review the documents on which she relies.
Mrs Van Den Bogerd was not taken to the transaction data for June, July or August. It
was not even suggested to her that her witness evidence (as corrected) was wrong about
what the data show. This was despite her referring to the transaction data in response to
cross-examination and, in effect, inviting challenge on it.!!”? Cs refused that invitation.
Cs put two main points in cross-examination about Mr Latif’s case. Neither of them
withstands analysis or undermines Mrs Van Den Bogerd’s evidence.
POL00026925
POL00026925
1168 (Day6/25:22} to {Day6/27:23} (see {E2/16/3}for the corrections she made to paras 91.1 and 91.2 of her
second witness statement {E2/5/23}). She also faced criticism for having referred to the wrong documents
(using POL disclosure numbers). As she pointed out, however, she had not herself entered the POL document
references: see {Day6/18:10}.
169 : ;
1170 ,
171 (Day6/23:18} to {Day6/24:3}.
,
ALG} 30:2 I, reflecting the witness statement at para 30 {E2/5/11}.
, for example, {Day6/24:6} to {Day6/24:10} and {Day6/31:2} to {Day6/31:5}.
4 (Day2/35:15} to {Day2/35:21}.
324
AI8/324
POL00026925
POL00026925
982. First, it was suggested that Mrs Van Den Bogerd should have considered the Callendar
Square and Dalmellington bugs when addressing Mr Latif’s allegations about the failed
transfer of £2,000.'!” That was a bizarre suggestion, for several reasons:
982.1 The Dalmellington bug related to remming in cash at outreach branches.!!7* Mr
Latif’s branch was not an outreach branch, and his alleged problem did not relate
to remming in. Mrs Van Den Bogerd pointed out the (obvious) irrelevance of the
Dalmellington bug several times when questioned on it.!!75
982.2 The Callendar Square bug was resolved in 2006.'!”° It was self-evidently irrelevant
to any problem with transfers in 2015.
982.3 The symptoms of the two bugs were different from the circumstances that Mr Latif
alleges. The Callendar Square bug allowed SPMs to duplicate transfers in on
different counters (due to a failure of the Riposte system used to communicate
between counters in Legacy Horizon). Mr Latif’s complaint is not that a transfer
in was mistakenly duplicated (whether at one counter or more than one counter)
but simply that the transfer failed, was reversed and somehow caused a loss in the
originating stock unit. That is different from Callendar Square. And as for the
Dalmellington bug, this caused SPMs mistakenly to duplicate remming in
operations. Again, Mr Latif's complaint does not involve the accidental
duplication of a transfer in (let alone a rem in to an outreach branch).
982.4 In any event, there is no record in the transaction data for June, July or August of
any duplicated transfers in for £2,000. The transfers are identified above. None of
them is duplicated. If there had been a duplicate transfer in, the branch would have
suffered a receipts and payments mis-match. and would not have been able to
rollover into the next trading period. Mr Latif does not say that happened. Further,
Mr Latif would have been able quickly to identify the existence of unreconciled
473 (Day6/20:22} to {Day6/21:17} and {Day6/33:7} to {Day6/35:8}
1 See JS2, entry 4 {D1/2/4} and KEL acha621P {F/1426}
175
See, for example, {Day2/35:5}.
16 See JS2, entry 2 {D1/2/3}.
325
AI8/325
POL00026925
POL00026925
transfers using the reporting tools that he accepted were available to him on
Horizon.'!”7
983. Second, Mrs Van Den Bogerd was asked whether it had been drawn to her attention that
(1) there were two receipts printed for one of the transfers in and (2) the existence of
multiple receipts had been a feature of the Dalmellington bug.'!® This was a
misconceived line of questioning, for the following reasons:
983.1 The Dalmellington bug was very different from the problem alleged by Mr Latif:
see para 962 above.
983.2 The fact that two receipts were printed is innocuous. Horizon automatically prints
one receipt/ slip for a transfer out, and it does the same for a transfer in.'!” But it
is clear (and unsurprising) that the user can also press “print” to print an additional
1180
receip' as Mr Latif confirmed this in oral evidence in respect of transfers out
specifically.!!*!
983.3 If it is suggested that because two transfer in receipts were printed, there must have
been two transfers in (as with the duplicated rem in operation in Dalmellington),
that can easily be tested by examining the relevant transaction data. The
transaction data for the relevant transfer in show that it was not duplicated:
(a) The two transfer in slips for stock unit SP1 were printed on 21 July 2015 at
13:55:00 and 13:55:04 (by Christine Fensome): see rows 8534 and 8535 in
the events data at {F/1354}.
. It shows that
(b) The transaction data for July is at + {FB
Christine Fensome carried out a transfer in of £2,000 on stock unit SPI on
21 July 2015 and timed at 13:55:01: see row 18539 (as is mentioned above
47 {Day2/30:8}. See also para 943.3 above. Example transfer reconciliation reports can be seen at
{F/856/587} (2011) and {F/1782/159} (2018)
178 .
1179 Mr Latif accepted this: {Day2/13:13} to {Day2/14:8}.
118 See {F/1594/16} at 3.7 “Duplicate Receipts”.
18! {Day2/13:24}. The technical specifications for the two types of transfer slip are materially identical: see
{F/1594/71} (transfer in slip) and {F/1594/73} (transfer out slip).
24 3:21} to
326
AI8/326
984.
985.
at para $47,262). This corresponds with the printing of the transfer in slips.
There is no duplicated transfer in shown in the surrounding rows of data.
(c) The point boils down to this: the events data show that two receipts were
printed, but the transaction data show that there was only one transfer in that
corresponds to those two receipts. Cs’ cross-examination is premised on the
idea that one should look at the event data and draw a weak inference from
it (i.e. that two receipts means two transactions), rather than carrying out a
check against the transaction data that can confirm or refute the inference.
The transaction data clearly refute the inference.
983.4 As noted above, Mr Latif does not say that any transfer in was duplicated.
In short, therefore, the challenge to Mrs Van Den Bogerd’s evidence did not undermine
the account she gives, and it did not provide any support for Mr Latif’s case on the failed
transfer.
Conclusion
On the evidence available at this trial, it is possible that Mr Latif’s branch suffered a loss
of £2,000 in or around July 2015 (although there is no record of any shortfall in that
amount in the logs), but there is no convincing evidence to indicate that a bug in Horizon
played a role in causing any such loss. There are various possible explanations for how
such a loss may have arisen. Merely by way of illustration, the current evidence would
be consistent with the following series of events:
985.1 In or around July 2015, Mr Latif identified that £2,000 was missing from the
branch.
985.2 Mr Latif asked his staff about the loss and was told that it was related to problems
with a cash transfer between stock units (these being a common occurrence in his
branch). He had not himself personally carried out the transfer in question. I!*
POL00026925
POL00026925
482 The transaction data for June to August 2015 show relatively few transaction carried out by Mr Latif
personally and quite a few days where Mr Latif did not perform any transactions at all. He confirmed in oral
evidence that there were days when he did not work in the branch: {Day2/7:5} to {Day2/7:24} (although it
327
AI8/327
986.
987.
988.
989,
POL00026925
POL00026925
985.3 Mr Latif therefore investigated the loss through looking at CCTV. He could not
identify anything untoward, and it is for this reason that he suspects a bug in
Horizon.
This seems a likely explanation. But the evidence available at this trial is limited. It
would, in Post Office’s submission, be unfair to both Mr Latif and Post Office to seek to
determine definitively what caused any £2,000 loss that he may have sustained in or
around July 2015.
Post Office invites the Court to conclude only that it cannot be satisfied, on the present
evidence, that any £2,000 loss in Mr Latif’s branch in or around July 2015 was caused
by a bug in Horizon as he alleges, although that issue has not been fully explored.
The problem with Camelot TAs in January 2018
This point can be taken more shortly. Mr Latif believes that some difficulties he
encountered with TAs in January 2018 were the result of a bug in Horizon. Ona fuller
analysis, it can be seen that incorrect TAs were issued, but that was an accounting error
(with subsequent correction) by Post Office, not a technical error with Horizon. Mr Latif.
accepted most of the explanation of the problem given by Mrs Van Den Bogerd at paras
98-102 of her second witness statement.''*? The remaining issue seemed to revolve
around a misunderstanding as to what Mrs Van Den Bogerd’s evidence implied in terms
of the branch’s scratch card stock position and, relatedly, why the stock TC issued to Mr
Latif’s branch on 24 January 2018 was for 100 scratch cards (and so carried a value of
£1,000) rather than for 50 scratch cards (with a value of £500).
Mr Latif agreed with almost all of Ms Van Den Bogerd’s evidence as to (1) the role
played by TAs in scratch card sales and (2) the specific problem encountered in his
branch in January 2018. In particular, Mr Latif agreed the following elements of her
evidence:
is unclear how common he says this was — he refers to be it being “very common” but also says it happened
only “occasionally”).
48S (E2/5/24}.
328
AI8/328
POL00026925
POL00026925
989.1 The explanation of how scratch cards are activated in branch and enter into the
accounts through TAs (see paras 25 and 26.6-26.7 of Mrs Van Den Bogerd’s
second witness statement at {E2/5/8}).!!84
989.2 The branch received and processed two TAs on 18 January 2018, and the two TAs
were mistakenly negative in value, reducing the branch’s scratch card stock level
(see para 98 {E2/5/24}).''> Mr Latif specifically confirmed that the TAs were
negative.!'*°
989.3 The effect of this was to cause the branch to have a negative level of scratch cards
(see para 99 {E2/5/24}).1187
989.4 The branch attempted a balance, but this was prevented due to the negative stock
of scratch cards. The branch carried out a sales reversal to bring the stock level to
zero, which automatically decreased the Horizon cash figure, leading to a cash
surplus (see para 99 {E2/5/24}).''*S Mr Latif specifically confirmed the effect of
the reversal.'!*?
989.5 The branch received a TC on 24 January 2018 which cancelled out the incorrect
TAs (recording a positive increase in scratch card stock). Due to the sales reversal
that had been carried out in the meantime, this would have left a shortfall of scratch
card stock. This is because the Horizon stock figure had been adjusted up twice —
once by the reversal and then again by the transaction correction (see para 100
{E2/5/24+).!!°° It should be noted that, although Mr Latif agreed with para 100
when specifically asked about it, other replies that he gave suggested that he
perhaps did not agree all of it.
184 (Day2/78:25} to {Day2/80:6}.
485 (Day2/80:7} to {Day2/80:24}.
M86 {Day2/82:8}.
487 (Day2/81:15}.
188 (Day2/84:17}.
18 (Day2/87:23}.
119 (Day2/86:10}.
329
A/8/329
POL00026925
POL00026925
989.6 Following the acceptance of the TCs, the branch would have been carrying a
scratch card shortfall but also a corresponding cash surplus. A manual stock
adjustment would have resolved this, effectively reversing the earlier sales reversal
(see para 101 {E2/5/24}).'1°! Either way, there would have been no overall branch
shortfall.
990. This account is supported by the transaction and event data, which show the following:
990.1 The two TAs received and processed on 18 January 2018 are at rows 12983 and
12984 of the transaction data {F/1761.1}. They each have product code 35341
(column J), which is the product code for £10 scratch cards.!!°* Each of the two
TAs is for 25 scratch cards (column K). The effect of each on branch derived stock
levels is negative.'!°
990.2 The fact that the stock unit OOH then had a negative stock level is shown by the
failed balance of that stock unit on 18 January 2018 and timed at 09:22: see row
6728 on the events log {F/1761.2}, stating “Balance Check of SU OOH failed.
Negative Stock exists”.
990.3 The sales reversal for 50 £10 scratch cards is at row 13357 of the transaction data
{F/1761.1}, timed at 13:52 on 18 January 2018.'1%4
990.4 The TC sent out on 24 January 2018 is shown at row 115 of {F/1833.1}, where the
content reads as follows:
THE LAUNCH OF In Pounds10 SCRATCHCARD GAME 1100 In Pounds4M
BLACK CREATED INCORRECT TA'S F‘ TES . i .
INCLUSIVE. PLEASE ACCEPT THIS 100 STOCK CREDIT TC INTO THE
SCRATCHCARD STOCK UNIT TO INCREASE In Pounds 10 STOCK. PLEASE
ADJUST In Pounds 10 STOCK TO THE CORRECT LEVEL AND REMOVE ANY
RELATED CASH ENTRY FROM EMERGENCY SUSPENSE TO RESOLVE
THIS ISSUE. ANY FURTHER ASSISTANCE PLEASE CALL NBSC ON 0845
6011022
491 (Day2/86:20}.
42 See F/1292.2/4} (8"* row down).
1% Positive entries in column K indicate a decrease in stock from the branch’s perspective. Contrast the
other TAs in January 2018, each of which carries a negative sign (indicating an increase in stock from the
branch’s perspective).
11 Tt is also shown on the events log as “REVERSAL” on stock unit OOH at row 6875 {F/1761.2}.
330
A/8/330
991.
992.
993.
POL00026925
POL00026925
990.5 The effect of the TC is shown in the transaction data as increasing the stock level
of £10 scratch cards by 100 (column K)!'!*°: see {F/1761.1} at row 18370
(Christine Barnett accepting the TC at 09:01 on 25 January 2018).
Mr Green QC cross-examined Mrs Van Den Bogerd on the mistaken hypothesis that each
TA should have both a negative and a positive entry. The relevant questions and answers
are at {Day6/38:16} to {Day6/39:13}, ending with “So you get a negative entry and then
a corresponding positive one”. On this mistaken hypothesis, it was suggested that one of
the entries for TAs on 18 January 2018 should have been negative:
Q: ...Let’s look at row 12983. We've got two positive 25s there, do you see that?
A: Yes.
Q: And if one of the 25s should have been minus 25 he is going to be actually 50
short, isn’t he? Because instead of recording the amount he actually should have and
cancelling out, he is going to end up with a record which is 50 rather than zero and
therefore he is going to be 50 short, do you agree?!
The basic factual premise of this line of cross-examination was wrong. The earlier entries
in the January 2018 transaction data for £10 scratch cards in fact showed two separate
things. First, there was a scratch card TA, the effect of which was to increase the stock
of £10 scratch cards without making any change to the Horizon cash figure (because a
TA effectively rems in stock). Second, there is a sale of the same number of scratch cards
later in the same day, which shows in the transaction data as a reduction in the Horizon
stock level, accompanied by a corresponding increase in the Horizon cash position.!!°7
This can be seen from the transaction data {F/1761.1}, as follows:
The TA on 4 January 2018
993.1 On 4 January 2018, timed at 06:55, a TA for 20 scratch cards was processed,
increasing the scratch card stock by 20: see row 2237. The TA does not carry a
cash value: see column L.
1195 As noted above, the entry on the spreadsheet has a negative sign, corresponding to an increase in stock
from the branch’s perspective.
1196 {Day6/41:5} to {Day6/41:13}.
1197 Mrs Van Den Bogerd explains this at (Day6/44:12} to (Day6/44:20}.
A/8/331
POL00026925
POL00026925
993.2 On the same day, but timed at 08:54, the branch sold 20 scratch cards, reducing
the stock of scratch cards by 20: see row 2309. As a sale, and unlike the TA, this
had a cash value: see column L (£200). And it resulted in an increase in the Horizon
cash figure: see row 2310 (where the scratch card sale’s value is included in the
1198
cash figure''”* of £810, along with three other transactions in the same basket —
see rows 2306-2308).
The TA on 9 January 2018
993.3 On 9 January 2018, timed at 06:11, a TA for 25 scratch cards was processed,
increasing the scratch card stock by 25: see row 6046.
993.4 On the same day, but timed at 09:15, the branch sold 25 scratch cards, reducing
the stock of scratch cards by 25: see row 6213. As a sale, and unlike the TA, this
had a cash value: see column L (£250). And it resulted in an increase in the Horizon
cash figure: see row 6214 (where the scratch card sale’s value is included in the
cash figure of £430, along with another transaction performed in the same basket,
namely that in row 6212).
The TA on 15 January 2018
993.5 On 15 January 2018, timed at 06:12, a TA for 25 scratch cards was processed,
increasing the scratch card stock by 25: see row 10202.
993.6 On the same day, but timed at 09:12, the branch sold 25 scratch cards, reducing
the stock of scratch cards by 25: see row 10331. Asa sale, and unlike the TA, this
had a cash value: see column L (£250). And it resulted in an increase in the Horizon
cash figure: see row 10332 (where the scratch card sale’s value is included in the
cash figure of £500, along with one other transaction performed in the same basket,
namely that in row 10330).
994. In short, on three occasions in early January 2018, the branch accepted TAs early in the
morning and then, somewhat later in the same day, performed a sale of the whole packet
of scratch cards, making them all available for sale to the public without needing to enter
4198 Column J shows the product ID 1, which is cash {F/1292.2/1} (top entry).
AI8/332
POL00026925
POL00026925
individual sales on Horizon. Mrs Van Den Bogerd explained that branches often account
for scratch cards in this way (effectively, a bulk sale to the branch’s retail operation) and
that this was the pattern seen in Mr Latif’s branch:
A:...Mr Latif’s practice was to — as soon as he activated the scratchcards in his post
office he would then buy them ~ sell them on to his retail. So what he would do straight
away is he would process a sale for £500. That’s what he would do and that was his
practice and that’s quite — a lot of people do that because it is easier to keep the cash
separate.
That would show as a sale on the information...!'°?
995. This point was not challenged. Cs nonetheless continued the cross-examination on the
false hypothesis set out above, contending that the TC for 100 units of £10 scratch cards
was wrong. Mrs Van Den Bogerd acknowledged that the situation was complex, but tried
to explain it in stages in her oral evidence.'?
996. It is clear from a Helpline call from Mr Latif’s branch on 25 January 2018 that he or his
staff were concerned that the TC was for 100, rather than 50, scratch cards. The content
of the call is recorded as follows: “TC received for launch of £10 x 100 scratch card
games...only received 50...if we accept TC, will cause a discrepancy?”: see {F/1834.1}
at row 171.
997. Nonetheless, Mr Latif confirmed that the TC was accepted and would, for the reason
given by Mrs Van Den Bogerd, have corrected the position,'*"' but for the sales reversal
that had been carried out in the meantime. That sales reversal could itself have been (in
effect) reversed by carrying out a manual stock adjustment, leaving the branch in balance
again: see para. 989.6 above.
998. Mr Latif’s written evidence on the TC was difficult to follow: see para 11 {E1/1/2} (the
natural reading of which is that the TC did not work because it did not increase the
physical stock of scratch cards, which is nonsensical).'°? In response to cross-
examination, Mr Latif put forward a more intelligible case, insisting that the TC had not
4% (Day6/44:13} to {[Day6/44:20}.
200 {Day6/46:11}to {Day6/47:18}.
1201 ‘The mathematics is simple, The mistaken TA had reduced the stock level by 50; it had been intended to
increase the stock level by 50. The difference between a reduction of 50 and an increase of 50 is 100. The
value of 100 units of £10 scratch cards is £1,000.
2 This point was explored unsuccessfully in cross-examination. Counsel and Mr Latif were perhaps at
cross-purposes as to what cach of them meant by “worked”: see {Day2/76:20} to {Day2/78:10}.
AI8/333
POL00026925
POL00026925
adjusted the Horizon stock position for scratch cards, suggesting that this remained
negative at the time of the transfer audit in September 2018:
A: Can I just confirm, there was an audit done in September of this year, an audit by
a Post Office trained auditor, and my stockholding was still showing negative. And a
Jane Lawrence is the auditor and she has ~ still could not resolve this matter, so the
problem hasn’t gone away, the problem is still there. And there have been a number
of calls to the Helpline to resolve that negative stock and it hasn't worked. They
haven’t come back with a response.'?°
999, There are several problems with that evidence:
999.1 The Horizon stock figure for scratch cards did not remain negative: it was
increased by 100 by the TC on 24 January 2018. Mr Latif appeared at one point to
accept this (although his evidence was not very clear): see para 989.5 above. In
any invent, the increase in the Horizon stock level is clear from the transaction
data: see para 990.5 above.
999.2 The Horizon stock figure could not have remained negative — the branch would
1204
otherwise have been unable to balance'*™’ and so unable to rollover at the end of
the trading periods between January and September 2018.
999.3 The actual stock figure could of course not be negative. That is nonsensical.
999.4 The transfer audit record is at {F/1829.2}. It does not show any stock
discrepancies.
999.5 There is no record of any follow-up calls after 25 January 2018: see the call log at
{F/1834.1}.
1000. As with the allegedly failed £2,000 transfer, there were various elements of Mr Latif’s
account that had been omitted from his witness statement:
1000.1 Mr Latif failed to mention that (he says) the transfer audit confirmed the
existence of a negative stock level.'2°° He justified not mentioning this in his
1203 (Day2/87:5} to {Day2/87:13}. See also {Day2/88:12} to {Day2/88:20}
204 Mr Latif confirmed this (by agreeing Mrs Van Den Bogerd’s evidence): see para 989.4 above.
1205 (Day2/89:2} to {Day2/89:13}.
334
AI8/334
statement on the basis that Post Office should have known about it anyway. That
is an unsatisfactory explanation.
1000.2 Mr Latif failed to refer expressly to receiving a notice through memo view
(although he did refer to a “notice” in para 10 {E1/1/2}). The memo views for the
relevant period were disclosed in response to Mr Latif’s reference to them in cross-
examination.'2% They provide no support for Mr Latif’s claim that he was sent a
notice about duplicate TAs.
1001. Ultimately, Mr Latif’s account was confused. Despite agreeing Mrs Van Den Bogerd’s
evidence on the problem with the TAs, he nonetheless suggested in oral evidence that
the (real) problem was that there had been a duplicate TA.!””’ That is plainly wrong:
1001.1 As already noted, it is inconsistent with those parts of Mrs Van Den Bogerd’s
evidence that he accepted to be correct. Her evidence explains the problem
(throughout its stages) and accounts for all the relevant movements in stock and
cash. There is no room for duplication in that account: the numbers all line up.
1001.2 It is inconsistent with the call made to the Helpline on 25 January 2018. The
complaint in that call was the branch had activated only 50 (rather than 100)
scratch cards, not that it had activated only 25 and so been sent two TAs for 25 in
error. The caller knew that 50 scratch cards had been activated in the branch and
that the TAs were correct in that respect.
1001.3 If the TAs on 18 January 2018 had been wrongly duplicated, the inescapable
inference is that this would have been raised with Post Office at the time. It would
have been a simple point to raise and require to be investigated.
1001.4 The basis for the duplication argument that was advanced by Cs in cross-
examination of Mrs Van Den Bogerd was demonstrably mistaken. It was based on
POL00026925
POL00026925
1206 (F/1727.1}, {F/1727.2}, {F/1749.1}, {F/1749.2}, {F/1749.3}, {F/1749.4}, {F/1749.5}, {F/1749.6},
{F/1749.7}, {F/1749.8}, {F/1749.9}, {F/1749.10}, {F/1749.11}, {F/1749.12}, {F/1750.1},
{F/1750.2}, {F/1750.3}, (F/1752.1}, {F/1752.2}, {F/1752.3}, {F/1752.4}, {F/1752.5}, {F/1752.6},
{F/1753.1}, {F/1753.2}, (F/1753.3}, {F/1753.4}, {F/1753.5}, {F/1753.6}, {F/1754.1}, {F/1754.2},
{F/1754.3}, {F/1758.1}, {F/1758.2}, {F/1759.1}, {F/1759.2}, {F/1759.3} and {F/1759.4}.
1207 ¢Day2/89:24} to {Day2/90:7}. See also {Day2/69:16} to {Day2/70:10}. This reflects para 9 of the
witness statement {E1/1/2}.
AI8/335
1002.
1003.
1004.
1005.
1006.
POL00026925
POL00026925
a misunderstanding as to how TAs work and a misunderstanding of the transaction
data for Mr Latif’s branch.
Conclusion
Mr Latif’s complaint in relation to the TAs may ultimately prove to be a good example
of a mistake that creates confusion and mis-trust of Horizon but which ultimately does
not disclose (or even relate to) any bug in the system.
Post Office invites the Court to conclude that, on the present evidence, it cannot be
concluded that there was some bug or error in Horizon that caused a duplication of TAs
in 2018 in his branch and resulted in a lasting discrepancy, but that issue has not been
fully explored.
Mr Tank
The relevant evidence is as follows:
1004.1 Mr Tank’s first witness statement dated 28 September 2018 {E1/6}.
1004.2 Mr Tank’s supplemental witness statement dated 27 February 2019 {E1/11}.
1004.3 Mrs Van Den Bogerd’s second witness statement dated 16 November 2018 paras
75-84 {62/5/20} to {E2/5/22}.
1004.4 Transcript: Mr Tank {Day2/99:15} to {Day2/99:15}; {Day2/161:23} to
{Day/61:23); Mrs Van Den Bogerd {Day6/48:19} to {Day6/65:5}.
Like Mr Latif’s evidence, Mr Tank’s evidence is brief, vague and not properly detailed.
Mr Tank raises three issues, each of which he says was caused by a bug in Horizon:
1006.1 A shortfall of £600 said to have been caused by an outage.
1006.2 A shortfall of £195.04 also said to have been caused by an outage.
1006.3 An issue with printing labels.
These will be dealt with in turn.
336
AI8/336
1007.
1008.
1009.
1010.
The shortfall of £600
In Mr Tank’s first witness statement, he says that a power failure “definitely occurred in
or around 2010-2011”! (emphasis added) and that this resulted in a loss to the branch
of £600. Mrs Van Den Bogerd pointed out that she could find no record of Mr Tank
contacting NBSC about any such shortfall around that time, but that there was a record
of contact being made about a shortfall of £195.04.1?°°
Having seen Mrs Van Den Bogerd’s second statement, Mr Tank investigated a forum
group that he had occasionally posted to'?!°
and discovered various things, including that
he had posted about a £600 loss in September 2014. Accordingly he concluded that the
events recounted in his first witness statement concerning this loss happened in
September 2014 and not 2010-2011 as he had originally said.!?!!
Cs apparently invite the Court to accept that his recollection of events is correct and that
the only explanation for the matters he describes is that there was a bug in Horizon.
However, his own evidence is vague and imprecise and was evidently not prepared with
any great care.
Mr Tank’s evidence is that he had not initially looked at the forum posts because he did
not think he would be able to access the forum group and “it did not seem relevant”?!
Given that he looked at the forum every day and posted to it at least some of the time
1213 that is a surprising approach to have adopted. When pressed
when he had a problem,
as to why he had not looked for the forum posts earlier, he gave the impression that the
evidence he had given had been casually prepared:
Q. So you were well aware that this was something which was relevant to the nature
of issues being faced by postmasters?
A, Yes.
POL00026925
POL00026925
1208
1/6/2} para 6.
1209 (2/5/20: para 77; calll log {F/1286.1;} remedy tab, row 120}.
200 (F/1257.1}.
mn eB
RPE
1/11/13} para 11
1/11/2} para 6. {E1/11/3}
218 {Day2/103:21} to {Day2/104:6}.
337
AI8/337
POL00026925
POL00026925
Q. And when you prepared your first witness statement you obviously knew this
resource existed, didn't you?
A. I did.
Q. And it is plain that it is relevant to your evidence, isn't it?
A, It is.
Q. Why didn't you think to look for this material when you were preparing your first
witness statement?
A. Because my first witness statement was I think short, brief, and it was just my way
of -- I didn't really fully research the whole background regarding it, I just put my
statement in.
Q. Can I just press you a little bit on that. You understand that what's being said in
this case, by you amongst many others, is that you carried out certain very specific
actions which you did correctly and that the Post Office is at fault, or the Horizon
system is at fault?
A. Yes.
Q. So it is important, isn't it, to have been precise in the evidence that you give?
A. Yes.
Q. And I'm just wondering why you didn't make some effort to find what was plainly
a relevant document in putting forward what you agree to be the need for precise
evidence?
A. Because I didn't -- I didn't feel that my information in my initial witness statement
was going to be taken any further, so I -- it wasn't as important as it now has
become.!°!4
Q. How did you reach the conclusion that the forum posts would not be relevant?
A. I didn't reach that conclusion.
Q. Well, you say in paragraph 6, Mr Tank: "... [did not think I would be able to access
the forum group and it did not seem relevant." I'm just wondering how you reached
that conclusion?
A. Because when I made my first initial witness statement I wasn't aware of £195.04
loss, that information only came to light after reading Ms van den Bogerd's statement.
Q. But you were putting forward evidence, Mr Tank, about various matters including
a number of matters that you now say you got the date wrong in relation to?
A. I wasn't aware that my first initial witness statement was evidence. I thought it was
just a witness statement. I thought ... yes, that's what I thought. I didn't think it was -
124 (Day2/104:7} to {Day2/105:15}.
AI8/338
1011.
1012.
1013.
POL00026925
POL00026925
- Q. Is your evidence that you didn't take care over the preparation of your first
witness statement?
A. I did.
Q. But not much?
A. It we ry general. My original witness statement -- I was just trying to get the
point across -- because I was referring back to my memory as well, I couldn't realise
the importance of what was important as -- it is only subsequently after finding the
information and having -- being able to go back into -- it was only a few weeks ago
that I managed to get back into the forum. I lefi my post office in 2016, so I stopped
visiting the forum?
The fact that Mr Tank could mis-remember the date of an event by so many years itself
calls into question the accuracy of his evidence.
Mr Tank’s inaccuracies do not end there. He was initially certain that other than the date
of the £600 shortfall, his evidence relating to that loss as set out in his first witness
statement was accurate'?'® but he later accepted that — while maintaining that there had
been two incidents of outage — it was possible that the evidence given in his first witness
statement is relevant to the £195.04 shortfall rather than the £600 one.'?!”
Mr Tank’s evidence on this was confusing. If (contrary to Post Office’s submission) any
findings are to be made in this regard, it is submitted that on the information available
the likelihood is that his evidence in his first witness statement does relate to the £195.04
shortfall and that the £600 shortfall had nothing to do with any outage:
1013.1 There is no documentary evidence that Mr Tank called the NSBC Helpline in
September 2014 and referred to any power outage (although it is right to point out
the Mr Tank’s evidence is that he made a call to the Post Office Banking Team).'?!*
215 (Day2/106:6} to {Day2/107:11}.
1216 (Pay2/108:20} to {Day2/109:19}.
217 (Day2/118:24} to {Day2/119:9}.
28 ¢E
1/6/2} para 10.
339
AI8/339
1014,
1015.
POL00026925
POL00026925
1219
1013.2 In his forum post relating to the £600 shortfall,’*’” which was contemporaneous,
no mention is made of any power outage: Mr Tank’s suggestion that he “just
91220
omitted it seems unlikely.
1013.3 In Mr Tank’s Amended SOI'”' he refers (on p.4) to a shortfall of £600 which he
says was caused by a power failure and a separate incident (on p.5) “in August
2014” when there was a shortfall of £660. In his second witness statement these
events appear to be elided since the letter he refers to in relation to the £600 loss!?”?
is about a shortfall of £660'?*.
1013.4 There is no evidence surrounding the £600 transaction in September 2014 of the
sort Post Office would expect to see which indicates there was an outage!?*4 —
whereas such evidence does exist in relation to the £195.04 shortfall (see below).
1013.5 In short, Mr Tank’s evidence of the outage makes sense if it is evidence in
relation to £195.04 but makes little or no sense if it is evidence which relates to
the £600 shortfall in September 2014.
Turning to the £600 shortfall itself, the shortfall relates to a transaction concerning a Post
Office Card Account (“POCA”). The POCA had a daily cash withdrawal limit of £600
and had limited services associated with it: for example, if a customer wants to transfer
cash from their POCA to another bank, two transactions are required: the withdrawal of
cash from the POCA and the deposit of that cash in the other bank.'??5
The evidence shows that on 16 September 2014 there was a withdrawal of £600 from a
POCA and immediately thereafter there was a cash deposit of the same amount in a
39
1.1276
Lloyds Accoun'
1219 ¢F,
/1257.1/6}
1220 (Day2/112:22} to {Day2/112:22}.
2221 (F/1717.1}.
1222 (EI/11/3} para 14.
2223 (F/1262.1}.
124 (F/1257.4} row 11676.
1225 (Day2/120:14} to {Day2/121:18}.
1226 (F/1257.4} rows 11676-11677.
340
AI8/340
POL00026925
POL00026925
1016. Mr Tank says that £600 was lost due to a power outage. As set out above, there is no
evidence that he made any such claim at the time and there is no evidence of any such
outage.
1017. There is a possibility that a user error was made and that cash was credited to the Lloyds
account at the same time as it was handed over to the customer. Mr Tank accepted that
that was possible but said that he had checked the CCTV evidence.'”” This suggestion
was made for the first time during cross examination. It was not mentioned in Mr Tank’s
witness evidence or even in his Amended Schedule of Information.
1018. Ultimately, even Mr Tank accepted that the situation was unsatisfactory:
Q. Well, I understand that it is a frustrating loss, but there's a perfectly simple
explanation for it which is an understandable user error?
A. I agree.
Q. And you nevertheless come into court and say on oath that you are so confident
that you did nothing wrong, that that didn't happen, because of evidence you have
seen and we haven't.
A, Okay. Fair point.
Q. Ina situation where you couldn't, until reminded by Post Office, recall the date of
the transaction to the tune of four years?
A, But you have to bear in mind I paid that £600 back. I beared that loss. I never
thought that I would actually get it back. My relationship with Post Office ended a
couple of years ago, so that money was written off. I wrote that money off myself. I
paid that back.
Q. I understand that. But if it was, as I'm suggesting to you, or if it's possible that it
was a user error then you would have to pay it back, wouldn't you?
A, Yes,!228
1019, Part of Mr Tank’s complaint was that in order to investigate this discrepancy, he had to
122
print out a full transaction report which was around 15-18 ft long.!””? He accepted that
in fact he could have filtered the information in various ways and that he did not have to
print it at all:
Q. ... But you didn't have to print that, did you, you could have looked at it on the
screen?
A. Yes, but the length of the report and the screen would mean you would probably
have to page down over 100 times.
Q. But you could have filtered it in various ways, couldn't you?
127 (Day2/122:20} to {Day2/123:13}.
2228 (Day2/124:10} to {Day2/125:5}.
122 (E1/6/2} para 9.
AI8/341
1020.
1021.
1022.
1023.
A. I could have filtered it, yes.
Q. The point is that the impression that your evidence gives is that you had to print
out this comically long document and go through it, but you didn't have to do that,
did you?
A. No.!?°°
The shortfall of £195.04
As noted above, this issue was not mentioned by Mr Tank in his first witness statement
but is dealt with in his supplemental witness statement, he having been reminded of the
matter by Mrs Van Den Bogerd.
Mr Tank could not recall the details of what had happened. He accepted that it was
possible that what had happened was that the transaction (which was a POCA
transaction) was in the stack, money had been taken from the POCA and before the stack
was cleared there was an outage.'*! Looking at the event log for 12 December 2011,
there is a message “Session 531865 could not recover 4502497” at the relevant time!”
and no activity for 20 minutes or so thereafter: this is the sort of evidence that Post Office
would expect to see when there is an outage.'?*
This ties in with the call which Mr Tank made to the NBSC on 13 December 2011.'?4
Mr Tank’s evidence in his forum post was that three receipts were produced and that
because they stated that cash was due to the customer, the branch paid out.'?°> There is
confusion as to whether his evidence in his first witness statement about what happened
to the receipts related to the £600 shortfall, the £195.04 shortfall or both, but Mr Tank
accepted in cross examination that all of the receipts relating to the transaction causing
the £195.04 shortfall had been handed over to the customer, and that this was not the
POL00026925
POL00026925
1230 ¢Day2/111:7} to {Day2/111:20}.
2231 {Day2/131:23} to {Day2/132:4}.
1232 (F/897/1} Sheet 1 tab/ row 328.
123 ay2/133:14} to {Day2/133:19}.
1235
2M (F/1286.1} row 120.
AI8/342
proper procedure.!**° He also accepted that the procedure to be followed is properly
described in Mrs Van Den Bogerd’s statement at paragraph 53.1757
1024. In passing, it is noted that Mr Tank said that he had not previously seen the description
138 it does seem
of the procedure to follow which he was taken to in cross examination.
that that version post-dated the relevant events: however, the Court will recall that the
version in place at the time of the incident in question was later handed up'?*? and this
was is in substantially the same terms. Mr Tank accepted that he would have been taken
through the relevant recovery screens on Horizon.'24°
1025. Following Mr Tank’s call to the NBSC a Peak was raised'*4! which records what had
happened. Note the second entry in this Peak dated 13 December 2011: this states that
the Peak concerned a Failed Recoveries Report Client: CAPO produced on 12/12/2011”.
The “Call Logger” entry in the top right hand corner of the Peak is “Raj Fains - MSU
Indt Mgt’. This shows that the call originated from the MSU, the Fujitsu department
which monitors failed recovery situations and reports them to the SSC so that they can
be investigated.'?4
1026. This investigation resulted in a Transaction Correction being raised.!’4* Mr Tank
accepted that he had been refunded the relevant sum. He also accepted that he had failed
to mention this in his witness statement.'**
1027. Mr Tank was doubtless caused some inconvenience by this matter and perhaps also some
disquiet. In an ideal world perhaps there would be no outages and perhaps a Helpdesk
operative could have been a little more emollient. But problems do happen in the real
world and the only questions for present purposes are whether the Horizon system was
designed to deal with this failed recovery in the way that it did and whether Horizon
POL00026925
POL00026925
1236
1237
1238
1239
1240
41
{Day2/140:1} to {Day2/140:13}.
{Day2/133:24} to {Day2/134:17}; Mrs Van Den Bogerd’s statement para 53 {E2/3/16}
{F/1365}.
{Day5/2:1} to {Day5/2:23}.
{Day2/139:8} to {Day2/139:12}.
{F/870}.
12 Mr Coyne agreed with this during his cross-examination see {Day16/104:4} to {Day16/104:16}. Cf an
example of a Peak which shows that the Peak originated with a Customer Call e.g. {F/1045}.
1243
1244
{F/871.1}.
{Day2/142:10} to [Day2/142:14}.
343,
AI8/343
1028.
1029.
handled this particular issue properly. The answer to both questions is yes. It is obvious
that a recovery process is required, and the sequence of events does not suggest a bug in
Horizon. The evidence shows that notwithstanding a failure to follow proper procedure,
Mr Tank was properly compensated. There should be no complaint.
Issues with printing labels
Mr Tank’s final complaint is extremely vague. In his first witness statement he said that
“in or around 2007, [his branch] had recurring issue” with labels'*4°
whereby in certain
circumstances Horizon jumped to the end of a transaction and could not print a label.
Mrs Van Den Bogerd said that given the passage of time there was no record she could
find of complaints being made by Mr Tank in 2007,'“° although she had seen evidence
of the issue being raised by Mr Tank in 2014 and 2015. Mr Tank in his supplemental
witness statement said that he now recalled that the issue “began in or around 201]”247
His explanation for this further significant inaccuracy was that since preparing his first
witness statement he had reviewed “Horizon generated receipts, print-outs, with hand-
written dates and reference numbers on them”.!**8 Such material has been requested by
Post Office but none of it has been disclosed: this is considered in more detail below in
the section on Cs’ attitude to disclosure.
Mrs Van Den Bogerd’s evidence, on which she was not challenged, is that it is possible
to process a completely separate transaction for spoiled labels and that this option is also
available if the printer does not produce a label at all.'”4? Mr Tank pointed out that the
guidance!?° appears to require a label to be on hand!°! although he accepted that he
could follow the procedure described even if the printer had not produced a label at all —
and said that he also thought that this would be contrary to other unspecified
instructions.!?°?
POL00026925
POL00026925
248 ¢E
1/6/3} para 15.
1246 (1)2/5/21} para 79.
1247 9B
1/11/3} para 15.
1288 {Day2/146:16} to {Day2/146:17}.
189 62/5/21} para 82.
1250 £F/1848.6}.
2251 {Day2/147:14} to {Day2/147:19}.
2282 (Day2/147:22} to {Day2/148:2}.
344
AI8/344
1030.
1031.
1032.
1033.
Overall, it is submitted that Mr Tank’s evidence on the label issue is so vague that no
proper findings could be made in relation to it.
Conclusion on Mr Tank's Evidence
Mr Tank’s evidence is unsatisfactory in many respects. Notwithstanding his
supplementary witness statement purporting to provide further detail, his evidence
remains vague and unclear. The fact that he makes extensive revisions to dates in his
supplementary statement is itself reason to approach his evidence with caution.
There is evidence of an outage in December 2011 but Horizon dealt with this
satisfactorily and Mr Tank received an appropriate TC. There is no evidence of any other
relevant outage and no evidence of any bug which led to the loss of £600 in September
2014. The evidence on labels is so vague as to attract no weight.
There is a great deal more to explore with Mr Tank in due course if his case is one which
comes to full trial. Mr Tank (like Cs’ other witnesses) confined himself to a small sub-
set of his complaints, presumably because it was felt that this threw light on the Horizon
issues — and Post Office’s evidence was similarly confined. There are many other aspects
of his claim that will need to be explored in due course.
1033.1 Post Office invites the Court to conclude in relation to the alleged £600 shortfall
that:
(a) It cannot be satisfied, on the present evidence, that there is convincing
evidence that there was a £600 loss in Mr Tank's branch.
(b) Alternatively, if such a shortfall did exist, it cannot be satisfied, on the
present evidence that it was caused by a bug in Horizon as Mr Tank alleges.
1033.2 Post Office invites the Court to conclude in relation to the alleged £195.04
shortfall that:
345
POL00026925
POL00026925
AI8/345
1034.
1035.
1036.
POL00026925
POL00026925
(a) The recovery process, and the supporting practices around the recovery
process, meant that Mr Tank was not left with a lasting loss as a TC was
issued to him.
(b) The Court cannot be satisfied, on the present evidence, that the shortfall of
£195.04 was caused by a bug in Horizon as Mr Tank alleges.
Messrs Anup and Aakash Patny
Both Messrs Patny give evidence relating to the same events and for convenience they
will be dealt with together. Father (Mr Anup Patny) and son (Mr Aakash Patny) will be
referred to respectively as “Mr Patny Snr” and “Mr Patny Jnr” and together as “the
Patnys”.
The relevant evidence is as follows:
1035.1 Mr Patny Snr’s witness statement dated 28 September 2018 {E1/3}.
1035.2 Mr Patny Jnr’s amended witness statement dated 27 February 2019 {E1/2}.
1035.3 Mrs Van Den Bogerd’s second witness statement dated 16 November 2018 paras
59-74 {E2/5/17} to {E2/5/20}.
1035.4 Transcript: Mr Patny Snr: {Day2/162:22} to {Day2/181:13}; Mr Patny Jnr:
{Day3/8:21} to +Day3-4 f 6
to {Day6/77:18}.
; Mrs Van Den Bogerd {Day6/65:6} {Day3/66:10}
Between them, the Patnys raise the following issues:
1036.1 an issue relating to the monthly balance on 11 May 2016;
1036.2 a problem with the stamp declaration on or about 19 May 2016; and
1036.3 a MoneyGram issue on or about 23 February 2016
These will be dealt with in turn.
346
AI8/346
1037.
1038.
1039.
1040.
POL00026925
POL00026925
Monthly balance on 11 May 2016
Mr Patny Snr’s evidence is that a result of a power outage on 9 May 2016, his branch
suffered a shortfall of over £17,000.'2°
There is no dispute that an outage occurred on 9 May 2016: it is clear from the events
spreadsheet!?** which records that there was “No recovery required”. Further, all that
was happening at the branch when the outage took place was that some postage labels
were being printed.'*°> Mr Patny Snr accepted that so far as this data was concerned, the
worst case scenario was that labels might have been handed over and payment taken in
cash (the PIN pad would not work if the Horizon terminal was not working) without the
correct recovery process being followed, which would have resulted in a small surplus
of cash.!?°° Mr Patny Jnr maintained that the £17,000 discrepancy might still have had
something to do with the outage but was unable to suggest why.'?°7
The events data!?>* shows that the cash declaration for 9 May 2016 was £48,021.94 with
a discrepancy of -£1,138.21.!?° It follows that the outage on 9 May 2016 did not cause
a discrepancy of around £17,000.
The discrepancy complained of first appears two days later, on 11 May 2016.'° Mr
Patny Jnr (who actually carried out the declarations'**') called the Helpline and reported
1262 1263
the issue.’* The description of the problem in the call log'*®* suggests that he reported
that the branch had remmed in £16,000 of coins, although he denied that he would have
said that. In any event, he was asked to make sure there was only once cash declaration
1283
{E
1/3/2; para 93; and {Day2/170:18} to {Day2/170:23}.
284 4F/1834.3} row 47.
255 See rows 42 — 45.
1286 {Day2/169:12} to {Day2/169:20}
257 (Day3/12:21} to {Day3/13:4}.
1258 (F/1507.1} row 13904 (remove filter).
1289 ¢F/1507.1} row 13905.
126 ¢F/1507.1} row 14515 (remove filter).
226 (Day3/10:21} to {Day3/11:3}.
128 ¢F/1507.1} row 136.
26 ¢F/1507.1}Column N.
347
AI8/347
1041.
1042.
1043.
POL00026925
POL00026925
and to ensure that the rem was scanned in correctly.!7° Mr Patny Jnr’s evidence was that
the Helpline told him to make various adjustments!?°
and that this appeared to sort
matters out. It is unclear what these steps might have been although it is clear that no
further cash declaration was made that day — and Mr Patny Jnr could not recall if he had
made any such cash declaration.'°°
The suggestion that Mr Patny Jnr was taken through steps which were not recorded on
the Helpline log — and that the Helpline log was inaccurate as a result — is highly
questionable. If (contrary to Post Office’s submissions) findings are to be made on such
matters, it is much more likely that Mr Patny Jnr did not recall these events accurately.
Cash was remmed in at the branch on 11 May 2016 including £16,000 of £10 notes.'267
Post Office’s records indicate that no cash was remmed in on 12 May 2016.'2°° Mr Patny
Snr accepted that that was the case — cash only came in on a Wednesday other than in
exceptional circumstances.'*® It should be noted that the cash management report for
the branch'?”°
shows that the branch was reporting £16,070 worth of £10 notes on 10
May 2016, but only £22,130 worth of £10 notes on 11 May — notwithstanding the
addition of £16,000 worth of £10 notes received in the 11 May remming. Then, on 12
May the figure goes up to £37,650 — an increase of over £15,000 even though no cash
was remmed in on 12 May.
The most natural explanation of this is that there had been an oversight in the counting
of cash on 11 May and that this was sorted out by later cash declarations: it is to be noted
that the cash declaration for 13 May 2016 shows a positive discrepancy of
£17,964.41,27!
164 ef,
2265
/1507.1}Column V.
1/2/2} para 10
1266 {Day3/17:13} to {Day3/17:15}.
267 (F/1834.2}.
1268 (F/1438.1}.
12 (Day2/174:8} to {Day2/175:4}
27 (F/1514.1}.
2211 (F/1507.1} row 15030.
348
AI8/348
POL00026925
POL00026925
1044. Mr Patny Snr was unable to suggest any reason why this alternative analysis was
1273
wrong.'?”? Mr Patny Jnr accepted that this was a possibility, !?”* although he did not recall
finding physical cash.'?4 He provided a good description of how keying errors could
result in mistakes being made:
Q. When you say you re-enter them again, would that involve physically counting
cash again
A. Sometimes.
Q. -- or checking the piles of cash or whatever it is you're doing?
A, Sometimes, but sometimes it could just genuinely be a case of, you know, as you're
rushing through pressing tab keys -- you have to bear in mind we were serving
customers whilst we were doing this as well. It was a fairly busy branch after closing
times -- well, general post office closing times are half past five; because we were one
of the very few in the city that were open after 5.30 so we would get customers coming
from all around, other side of the city coming to us because they knew there was a
post office open, so as you're rushing through sometimes you didn't tab correctly and
you would end up instead of writing 20,000 £20 notes you could add the £10 notes of
say 10,000 £10 notes into that as so you could end up with 40,000 in £20 notes.
Just a minor mistake can cause .
Stamps declaration on or about 19 May 2016
1045. The issue raised regarding the declaration of stamps is complicated. It is clear that there
was a declaration for stamps on 18 May 2016 in the amount of £18,274.99!?” and it is
common ground that that figure is wrong. There are three possible explanations as to
how that figure came to be declared:
1045.1 It was declared erroneously. Mr Patny Jnr accepted that this was entirely possible
but considered that this did not explain how the issue returned.!?77
22 {Day2/177:20} to {Day2/178:4}.
278 {Day3/22:23} to {Day3/23:3}.
22 (Day3/23:21} to {Day3/24:4}.
2275 (Day3/27:1} to {Day3/27:20}.
216 (F/1507.1} row 16156.
22” (Day3/29:11} to {Day3/29:18}.
349
AI8/349
1046,
1047.
1048.
1049.
POL00026925
POL00026925
1045.2 It was declared dishonestly, perhaps in an attempt to disguise a shortfall of cash.
Mr Patny Jnr denied that this was the case and insisted that all the declarations he
made were accurate to the best of his knowledge.'?’8 Post Office put this to be Mr
Patny Jnr after careful consideration. This point was to be raised in this closing as
one of several possible explanations and so it was only fair that he had a chance to
respond to such a serious allegation.
1045.3 There was a bug or bugs in Horizon.
Either of the first two of these explanations is far more likely than the third (although
Post Office does not believe that the Court, in this trial, needs to make any such finding
for present purposes or a finding as to which of the first two may be correct). Post Office
does not wholly reject the possibility of the third explanation but there is insufficient
evidence before the Court at this stage that a bug existed and neither expert has given
evidence of a bug which would explain the Patnys’ alleged experience here.
Moreover, there is a plausible explanation which makes sense both of Mr Patny Jnr’s
evidence and of the various declarations which were made.
To navigate through this explanation, it is important to understand the difference between
declared and derived stock positions:
1048.1 The declared stock position is what the SPM has declared as cash or stock having
physically counted i.e. — which if done correctly should match the real position in
the branch;
1048.2 The derived stock position is what Horizon thinks should be in the branch.
The second limb is where the Court has seen evidence of bugs, but that is not the Patnys'
case in relation to stamps. The Patnys are alleging that there was a bug affecting the
Seco: limb — that Horizon generated an incorrect declaration of stamps.
1278 {Day3/46:24} to {Day3/47:6}.
350
A/8/350
POL00026925
POL00026925
1050. From 10 February 2016 to 11 May 2016, in the Patny's branch the stamps are declared
once per week at around 6-7pm on a Wednesday evening in a range from £1 ,633.96 to
£2,053.98.!” This happens like clockwork: there are no missing or extra declarations.
1051. On Wednesday 18 May 2016 at 18:42 the stamps are declared as £18,274.99:!?°° a
substantial move up from the previous declaration on 11 May of £1,633.96.'8! This
results in the branch accounts showing at 18:55 a stamp surplus of £16,689.21.'?°?
1052. On 19 May 2016 Ms Debra Lambley of Post Office calls the branch about the high stamp
declaration. What Mr Patny Jnr should have done was to re-declare the stamps at the
proper value. Either through Ms Lambley giving the wrong advice or Mr Patny Jnr not
understanding what he was supposed to do, at 13:10 that day the derived stamp figure is
wrongly adjusted by £16,689.21, causing the derived cash figure automatically to go up
by the same amount. !?*
It is noted that there is no other stock adjustment made to the
derived stamp position in the previous two months according to the data, so this did not
appear to be a regularly used mechanism in this branch, unlike making stamp
declarations which were routine.
1053. The stamps are then declared at 19:00 on 19 May, this time at £18,273.66, being slightly
different to the declaration the previous day.'?5*
The movement to the cash position due
to the incorrect stamp adjustment above may be the cause of a cash shortfall of
£16,715.45 declared at 19:03'*S°, but it should be noted that stamp adjustment would
have created a corresponding gain so, all other things being equal, the branch should still
balance overall with offsetting stock gains and cash losses.
1054. The stamps are declared on 20 May at 07:07 for £18,273.66 at 12:44 for £18,273.66,'7°°
127 {F/1507.1} rows 161, 1080, 2051, 2980, 3951, 4987, 6064, 6885, 7943, 9065, 10254, 11376, 12702 and
14502
1280 (F/1507.1} cf. rows 14502 and 16156.
2281 F/1507.1} rows 14502 and 16156.
128 (F/1438.1} row 31271, note figures in the Session Data are inverted, so a minus sign means a surplus.
128 (F/1438.1} row 31466.
28 (F/1507.1} row 16407.
1285 (F/1507.1} row 16414.
1286 ¢F/1507.1} rows 16427 and 16498.
AI8/351
1055. A further adjustment to the derived stamp position happens on 23 May at 15:07 where
the stamps stock is adjusted down by £16,692.92 and there is a corresponding increase
in the derived cash position.'?°7
1056. The stamps are declared on 25 May at 18:56 for £18,243.10, being the third different
figure to be declared during this period.'?5*
1057. Then, 19 minutes later, at 19:15 the stamps are declared at £1,551.25'?* and the accounts
record a minor surplus of stamps of £3.70.17°°
1058. From that point to the end of July, the stamps are declared on 14 more occasions in a
range from £1,479.07 to £1,904.07, save for one declaration of £0 which was corrected
27 minutes later to £1,536.29. 19!
1059. Post Office's explanation of these events is that:
1059.1 Mr Patny Jnr knew how to make stamp declarations and had made them like
clockwork every week without issue for 3 months before May 2016.
1059.2 On 18 May, he made either mistakenly or dishonestly made a stamp declaration
of £18k that generated a surplus of stamps of nearly £17k that, whether by
coincidence or design, broadly set off a cash loss in the branch at around the same
time and of approximately the same amount.'?”
1059.3 He was then contacted by Post Office and in ensuing attempts to correct the
position the accounts got into a mess. Either through incorrect advice or his own
mistake, he made stock adjustments to the derived stamp position rather than re-
declaring the stamps at the correct level. In trying to resolve the problem or for
some other purpose, he made four further stamp declarations at around £18k.
POL00026925
POL00026925
1287
1288
1289
1290
1291
{F/1438.1} row 33311 and 33312.
{F/1507.1} row 18007.
{F/1507.1} row 18026.
{F/1438.1} row 34770.
{F/1507.1} rows 19634, 19642, 19947, 20235, 21450, 23161, 24807, 26349, 26865, 26878, 27936,
29476, 30833 and 32287.
1292
The cash declarations and variances fluctuate significantly during this period but there are a number of
cash discrepancies declared of just under £17k — see for example {F/1507.1} row 16411 (19 May),
17384 (23 May), and 18022 (25 May)
AI8/352
1060.
1061.
1059.4 The position was corrected by 25 May, when the stamps were declared at £1.5k
being in the usual range for his branch.
By comparison, the Patnys’ explanation — that this is evidence of a bug in Horizon — is
implausible.
Mr Patny Jnr says that he did not make these declarations (which is to be distinguished
from saying that he made declarations but the system recorded the wrong figures). On
Mr Patny Jnr's case that means that the supposed bug or bugs would have to have:
1061.1 Not occurred before 18 May 2016;
1061.2 Then generated five stamp declarations at seemingly random times in a seven
day period. There does not appear to have been any preceding trigger event in
common across the declarations. The system has no reason to automatically
generate a stamp declaration given that the necessary step before a declaration is
a physical count of the stamps.
1061.3 Of the five declarations, three different numbers were declared. Computers do
not just randomly generate numbers, they follow logical processes which would
mean that for Mr Patny Jnr's explanation to be correct there were either different
bugs, or the same bug was triggered three times from the same input and twice by
different inputs.
1061.4 Generated two stock adjustments on two different days for two different
amounts. Again, there is no commonality of cause for this.
1061.5 Corrected themselves just in time for the next rollover on 1 June.'??; and
1061.6 Then never occurred again.
POL00026925
POL00026925
1293 (F/1507.1} row 19650.
353
AI8/353
1062.
1063.
1064.
1065.
1066.
1067.
POL00026925
POL00026925
Cs do not suggest that there was anyone else around this time suffering any similar issues
(it would likely be in a Peak somewhere if the problem were widespread and doubtless
they have looked for it).
Accordingly, Post Office invites the Court to conclude that it cannot be satisfied, on the
present evidence, that there was a bug in Mr Patny's branch causing incorrect stamp
declarations.
MoneyGram issue on or about 23 February 2016
The complaint here is that Mr Patny Jnr tried to take payment for a MoneyGram
transaction (for £3,100) twice. but the customer’s card was declined and this resulted in
a shortfall of £6,200 “exacrly”.'° Mr Patny Jnr confirmed that what he was alleging
was that the transaction was put onto the stack on the Horizon sercen, that the customer’s
card was declined by which time there was something on the screen which had to be
dealt with in some way in order that the account is balanced.!?°5
The relevant transaction is clear from the transaction data.'?°° There is no evidence of
any second transaction in the same amount.
Mr Patny Jnr accepted that in fact the situation he described the proper procedure to be
followed — described in the online guide'?”’ — was both cancellation and reversal and that
he had only cancelled and not reversed the transaction.'”°* Mr Patny Jnr rang the Helpline
and they told him that cancellation and reversal were both required.'””
Mr Patny Jnr agreed that he had not reversed the transaction at the time but that he had
reversed it later.!*°° The evidence does not support this: the relevant entries on the call
1301
log'*°! state that the transaction had not been reversed. In due course a TC was received
198
1/2/4} para 20.
1295 (Day3/36:21} to {Day3/37:5}.
1296 ¢F/1436.1} row 1067.
1297 ¢F/1391.4}.
229% (Day3/39:6} to {Day3/40:5}.
12% ¢F/1522.1} row 115.
1300 (Day3/42:15} to {Day3/42:25}.
1301 ¢F/1522.1} rows 116 and 117.
354
AI8/354
1068.
1069.
1070.
for £3,100.'° (In fact if the transaction had been successfully reversed as well then the
branch would have ended up with a gain of £3,100 — since the TC would have not been
required at all).
Mr Patny Jnr was clear that the discrepancy the branch experienced was exactly £3,100.
This is not borne out by the data:
1068.1 The actual cash shortfall on 23 February 2016 was £6,825.95%°, with a cash
declaration of £25,803.87.5
1068.2 The cash declaration for the previous day was £34,405.46'°°> i.e. there was a
difference between 22 and 23 February of £8,601.59.
1068.3 The total decrease in cash between the two declarations was £1,806.71.'°°°
1068.4 So of the movement of £8,601.59, £4,906.71 can be accounted for (£3,100 +
£1,806.71). This leaves £3,694.88 i.e. not exactly £3,100.
Mr Patny Jnr was unable to explain why he still felt able to say that the discrepancy was
exactly £3,100. He referred to various requests he had made to see the Credence data
and to conversations he says he had with his area manager.'*°” None of these matters had
been canvassed in his witness statement, Post Office had not had any opportunity to
address them in its evidence and so it was not possible to explore them in his oral
evidence. It is not possible to place reliance on unexpected evidence which emerges in
such a way.
Mrs Van Den Bogerd was cross examined'*** about a moneygram report dated 18 July
2016'** which reported that duplicate transactions had been created as a result of Post
Office time outs. Mrs Van Den Bogerd had not previously seen the document.'*!° The
POL00026925
POL00026925
1302 (E1/2/4} para 22
1508 (F/1507.1} row 1903.
801 (F/1507.1 }row 1904.
505 Row 1717.
1306 (F/1436.1.1}.
1807 (Day3/56:14} to {Day3/57:22}.
1308 (Day 5/149:1}
1309 (F/1502/29}.
131 Day 5/150:11} to {Day5/150:12}.
355
AI8/355
issue it described does not assist in relation to the events alleged by Mr Patny: first, the
document is describing a problem in the moneygram system not in Horizon; and
secondly Mr Patny does not say there was any Post Office time out, but that the
customer’s card was declined.
1071. Accordingly, Post Office invites the Court to conclude that it cannot be satisfied, on the
present evidence, that there was a bug in Mr Patny's branch that caused a duplicate
moneygram transaction.
Conclusion on the Patnys’ evidence
1072. The Court cannot arrive at concluded views on what was happening in the Patnys’
branch. The picture that emerges is one of considerable disorder in the branch. The fact
that cash declarations were made within minutes of cach other'*!' is an indication of this,
as is the fact that it seems there was considerable laxity about who was using which log
on. Mr Patny Snr:
Q. Okay, thank you. Seven minutes apart for the same sign-on, APAOOI -- is that you
or is that your son?
A. That's me, but it would be my son because that looks like it has been logged on
since the afternoon.
Q. So he was using your log on?
A. He was, yes.
Q. Why would he do that?
A. Well, normally between three of us nobody else would go on apart from the three
of us, so we didn't really mind, whosoever is logged on we just carried on and
continued with it.3”
1073. Mr Patny Snr was an SPM for less than two years.'*!3 As with every C, there is a great
deal more to be investigated about this branch in due course. The Court has heard of only
a small selection of events which the Patnys have chosen to give evidence about. It could
prejudice future breach trials if the Court does any more than conclude that the events
POL00026925
POL00026925
1311 (F/1834.3} sheet I/row 2336 and 2337: 7 minutes apart.
1312 (Day2/178:13} to {Day2/178:23}.
1318 (E1/3/1} para 1.
356
AI8/356
described could plausibly have been caused by user error. There has been no
identification of any particular bug nor any expert support for any other finding.
Mrs Burke
1074. The relevant evidence is as follows:
1074.1 Mrs Burke’s amended witness statement dated 27 February 2019 {E1/4}.'5"4
1074.2 Mrs Van Den Bogerd’s second witness statement dated 16 November 2018 at
paras 103-110 {E2/5/25} to {E2/5/27}.
1074.3 Transcript of Mrs Burke’s evidence oral evidence at {Day2/66:17} to
{Day2/104:22}.
1074.4 Transcript of Mrs Van Den Bogerd’s oral evidence in relation to Mrs Burke’s
case at {Day5/16:11} to {Day5/19:20}.
1075. Mrs Burke’s evidence relates to the failed recovery of a £150 withdrawal on 9 May 2016
in the Newport branch. Most, if not all of the material facts appear to be common ground:
1075.1 Mrs Burke was an experienced SPM and assistant: see her witness statement at
51315
para and her oral evidence that she had worked for around 5 years as an SPM
and around 15 years as an assistant.'*!°
1075.2 The need to recover transactions on 9 May 2016 resulted from a nationwide
system outage that affected Mrs Burke’s branch.'*!7
1075.3 Mrs Burke identified that the £150 withdrawal had not been recovered
successfully and that, if the branch did not receive transaction correction, there
would result a £150 shortfall (because Mrs Burke had paid out the cash to the
customer).'*!
POL00026925
POL00026925
14 The original witness statement was amended to clarify two small points (paras 8 and 13). At trial, Mrs
Burke made an update to her evidence in para 5 (relating to her current employment.
BIS (E1/4/2}.
1316 {Day3/68:7} to {Day3/68:19}
1317 Mrs Burke was told this when she called the Helpline para 20 {E1/4/4}.
1318 See the oral evidence at {Day3/84:14} to {Day3/85:2}.
357
AI8/357
1076.
1077.
1078.
1075.4 Mrs Burke almost immediately telephoned the Helpline. In summary, she was
told that there was a nationwide problem, that it was being addressed and that,
although no guarantee could be given, this could resolve the loss in the branch.
The transcript of the call is at {F/1466}. Post Office accepts that the Helpline
operatives did not give Mrs Burke the comfort that she reasonably wanted.'*!°
1075.5 Mrs Burke then undertook her own investigation to identify the customer who
made the £150 withdrawal, and she obtained records from him and his bank to
show that the withdrawal had been effective.'*?? Mrs Burke (via her husband)
informed Post Office of the outcome of her investigation on 13 May 2016.13!
Mrs Burke was an honest and careful witness. Her witness statement was clear and
relatively detailed. She was fair and frank in her responses to cross-examination. Post
Office challenged only small parts of her written evidence, and she responded to those
challenges with care, making appropriate concessions.
It was suggested to Mrs Van Den Bogerd that Post Office had put to Mrs Burke in cross-
examination that she had caused or contributed to the failed recovery of the £150
withdrawal by serving two customers from a single basket / stack on Horizon.'*”?
That appears to be a misunderstanding — no such suggestion was put to Mrs Burke. Post
Office in fact put the following points about the circumstances in which the failed
recovery arose:
1078.1 Serving two customers from one basket is not the procedure that SPMs are
supposed to follow, and Mrs Burke would not ordinarily do it. Mrs Burke very
readily accepted this. 57>
1078.2 On 9 May 2016, Mrs Burke (exceptionally) did serve two customers from one
basket, and she did so in the circumstances created by the system problems that
POL00026925
POL00026925
319 See the cross-examination on the call at {Day3/88:11} to {Day3/93:18} and, making this point clear,
{Day3/95:2} to {Day3/95:9}.
1520 Witness statement, paras 20-24 {E1/4/4}.
821 (Day3/94:20} to {Day3/95:2}.
182 (Day5/16:22} to {Day3/19:20}.
1823. (Day3/75:1} to {Day3/75:19}.
358
AI8/358
1079,
1080.
1081.
POL00026925
POL00026925
she was encountering. Mrs Burke accepted this too, and no criticism was made of
her in relation to it."
1078.3 Following proper procedure would have reduced the opportunity for a system
outage to occur while transactions were still in the stack (and so not committed to
the accounts). Mrs Burke did not entirely follow the points put to her in relation to
this, '*?5 but no criticism was or is made of her in that regard.
It is understandable that Cs wish to emphasise that Mrs Burke’s acknowledged failure to
follow correct procedure did not cause the failed recovery. But that is accepted. No
contrary suggestion was made in cross-examination, and Ms Van Den Bogerd made a
correction to her witness statement to spell this out clearly: see {E2/16/3} (an
amendment to para 104 {E2/5/25}).
Post Office challenged only three elements of Ms Burke’s written evidence.
First, Post Office challenged Mrs Burke’s statement at para 20 that “I had been watching
my stack carefully and it told me to pay £150 to the customer” (emphasis added)
{E1/4/4}. Mrs Burke responded fairly to the challenge put to here in this regard:
1081.1 She accepted the points put to her about how the stack / basket in fact
operates. 57°
1081.2 She agreed that it is only once the “Enter” key is pressed to close the basket that
the user should pay the customer (or take payment from the customer, as
appropriate).!*2” Her oral evidence was that she would ordinarily prepare the cash
to pay to the customer (counting it out) and then pay it over after, or perhaps at the
same time as, pressing the button to clear the stack.!°** That is good practice. She
understood (correctly) that it is only once “enter” is pressed to close the stack that
the transactions enter into the accounts. Although she would quite sensibly profess
no knowledge of the technical workings of the system “behind the scenes”, she
1824 (Day3/75:20} and {Day3/77:9} to {Day3/77:20}.
185 (Day3/77:21} to {Day3/80:21}.
326 (Day3/70:21} to {Day3/72:21}.
1827 {Day3/73:3} to {Day3/74:21}.
18% (Day3/74:4} to {Day3/74:21}.
359
AI8/359
1082.
1083.
1084.
was aware that it was closing the stack that triggered the transactions being
submitted / entered into the accounts:
Q....What I suggest is that you would have known that it is when you close the
stack that the transactions in the stack go into your accounts?
A: Yes, what should be in the stack, yes, should go into our accounts.
Q: And that happens when you close the stack, because that’s when those
transactions are finally confirmed?
A: When you press the “enter” button and it clears the screen, everything that’s
in the stack should be in your accounts.!°°°
1081.3 With the sensible caveat that she would not know how the back office procedures
work, Mrs Burke agreed with the evidence that Ms Van Den Bogerd’s gives in
relation to bank withdrawals at para 105 of her witness statement {E2/5/25}.!°°
It follows from the points that Mrs Burke accepted that it was inaccurate to state that
Horizon “told” her to pay £150 to the customer before the basket had been closed. The
closest that the system will have come to this is informing the user that the withdrawal
had been authorised by the bank: see the screenshot at {F/1848.06}. The transaction
would then appear in the stack (along with any others): see {F/1848.07}.
Second, Post Office challenged Mrs Burke’s statement at para 19 that “Zhere was
therefore no means through the Horizon system for the discrepancy to be identified or
{for its cause to be established in my situation” {E1/4/4}. Similar phrases appear in other
witness statements produced for the Horizon Trial: see Latif, para 8 {E1/1/2}, Tank, para
14 {E1/6/3} and Singh (not called), para 11 {E1/8/3}.
In her oral evidence, Mrs Burke was characteristically careful and frank in accepting that
it was wrong to say that the Horizon system did not provide a means to identify the
discrepancy:
Q: ... I'm focussing quite closely, if 1 may, on the wording of the last sentence of para
19 of your statement. It says there: “There was therefore no means through the
Horizon system for the discrepancy to be identified...". Stopping there, I suggest to
POL00026925
POL00026925
1329 (Day3/76:23} to {Day3/77:7}.
1890 {Day3/75:22} to {Day3/77:7}.
360
A/8/360
POL00026925
POL00026925
you that’s wrong because you knew exactly what the discrepancy would be, it would
be a £150 shortfall. So that’s not correct, is it?
MR JUSTICE FRASER: Well, Mr Draper - - well, I will wait for the answer and
then...
A: Well, the Horizon system didn’t identify it; I knew it was the £150. Oh, I see what
you mean, I had the two receipts. Yes, I will agree with that.****
1085. Mrs Burke also accepted that the cause of the discrepancy was in fact known to her at
the time:
Q: And focussing on the second half please of the sentence: “or for its cause to be
established in [your] situation”. I suggested that based on what we have just
discussed, you knew what the cause of the £150 shortfall would be, didn’t you?
A: Sorry, could you say that again?
Q: You knew that when a £150 shortfall arose —
Q: You knew what would have caused it, because you had worked out that the £150
withdrawal hadn't made its way to your account?
A: I did after studying all that paperwork, yes. °°
1086. The Court was interested to identify the precise means through which Horizon identified
(or could have identified) the discrepancy to Mrs Burke. As to this:
1086.1 Mrs Burke was in fact able to identify the discrepancy and the cause of the
discrepancy through the process that she describes in her witness statement at para
17, namely reviewing the transaction log!**?
to identify the transactions shown on
the Disconnected Session Receipt'***: see {E1/4/3} and {Day3/83:9} to
{Day3/85:2}. She knew from these documents, and the fact that she had paid out
the £150, that the failed recovery would result in a £150 shortfall.
1086.2 Mrs Burke could also have identified the shortfall through running a balance
1335
snapshot (as she appeared to accept or through conducting a trial balance of
1831 (Day3/86:1} to {Day3/86:14}.
1892 (Day3/86:15} to {Day3/87:3}.
3% The annotated transaction log is at {F/1465}
1 The receipt is at {F/1461}.
335. (Day3/88:4}.
A/8/361
POL00026925
POL00026925
the relevant stock unit or through simply declaring the cash on the stock unit to
identify any shortfall relative to the Horizon derived figure. All of these steps
would involve Horizon identifying the shortfall in an on-screen balance. Mrs
Burke was of course aware that one or more transactions from the disconnected
session had not been recovered,'**° so she would (at worst) have been on the
lookout for any shortfall that matched the amount of one of those transactions
when she ran the cash declarations on the evening of the same day. But this is
purely hypothetical: Mrs Burke very sensibly dealt with the problem as soon as it
arose and while she was best able to identify the shortfall and its cause.
1087. Horizon provided sufficient information to enable Mrs Burke to identify both the
discrepancy (a £150 shortfall) and the cause of the discrepancy (the failed recovery of
the £150 withdrawal transaction). The final sentence of para 19 of Mrs Burke’s witness
statement was therefore inaccurate, as she fairly acknowledged.
1088. Third, Post Office invited Mrs Burke to revisit the view expressed in her witness
statement that the £150 shortfall would not have resolved the shortfall if she had not
carried out her own investigation and collected evidence.!**” Mrs Burke made clear in
her witness statement that this was her view (rather than a fact), and she indicated the
basis on which she had formed that view. Her evidence on the point was clear and
straightforward.
1089. In cross-examination, Post Office took Mrs Burke to documents showing the work done
by Fujitsu and Post Office to identify and resolve any shortfall arising from the failed
recovery.'**8 The documents are as follows:
1089.1 A BIMS Incident Report'**? produced by Fujitsu on 12 May 2016: {F/1470}. It
shows (at the bottom of the page) that Fujitsu had identified a series of transactions
1336 See the failed recovery receipt at {F/1464}.
B37
See para 27 of Mrs Burke’s witness statement: “Based on my initial experience of the Helpline, I do not
think that Post Office would have resolved this if I had not had the clear proof that the £150 transaction had
in fact been authorised and that the money had left the customer’s bank account” {E1/4/5}.
1838 See {Day3/95:11} to {Day3/104:2}.
18° BIMS stands for “Business Incident Management Service”.
AI8/362
1090.
across the network that were affected by the system outage on 9 May 2016 and
required reconciliation action to be taken by Post Office.
1089.2 An email sent by Fujitsu to Post Office on 12 May 2016, attaching a spreadsheet
identifying the transactions that required reconciliation: {F/1470.1}.
1089.3 The spreadsheet that was attached to that email, identifying Mrs Burke’s branch
at row 12 and stating as follows in column H {F/1848.4}:
The £150 cash withdrawal transaction was authorised by the FI and an
AUTHORISED receipt was produced on the counter. However, when the user
attempted to settle the transaction it failed due to the known datacentre issue at
the time so disconnected session receipts were produced and the user was logged
off The user managed to log back in but recovery also failed. As an
AUTHORISED receipt was produced the user should have handed money over
to the customer but we cannot be certain that they actually did so. Assuming
money was handed over the customer account will be correct but the branch will
have a shortage given that the transaction hasn't been recorded on the system.
This will need to be manually reconciled.
1089.4 There is a later version of that spreadsheet in which Post Office had added a
column indicating what it proposed to do {F/1848.5}. The above text appears at
column H. In a new column J, headed “Action taken”, the spreadsheet states:
“Branch contracted. Shortage in branch so TC issued to adjust cash”.
1089.5 There is a record of the TCs for the Newport branch, showing a £150 TC issued
on 16 May 2016 {F/1687.1}. It states, in relevant part, as follows at row 5, column
L: “Zo correct communications failure on 9.5.16 for in pounds 150...so0 credit to
office”.
1089.6 Mrs Burke confirmed that she received and processed the TC on 17 May
016.140
The contemporaneous documents show that Fujitsu had been monitoring failed
recoveries across the network and position and had by 12 May 2016 identified the
unrecovered transaction at the Newport branch and informed Post Office that manual
reconciliation would be required (i.e. issuing a TC to the branch, assuming that the
money for the withdrawal had in fact been paid out to the customer). This confirms Mrs
POL00026925
POL00026925
540 Day3/103:13}.
363,
AI8/363
POL00026925
POL00026925
Van Den Bogerd’s written evidence that “there is a process built into Horizon for
flagging non-recovered transactions which would have prompted an investigation
and...would have led to the same outcome”.'**' The need for a TC was identified before
Mrs Burke told Post Office of the outcome of her own investigation on 13 May 2016.'3
1091. When Mrs Burke was invited to comment on what she could see from the Fujitsu and
Post Office documents, she gave a short and fair response:
Q:...with the benefit of that what you have seen here about what Post Office and
Fujitsu were doing in the background, as it were, unknown to you, with the benefit of
that knowledge would you accept that it looks as though Post Office would have been
able to resolve the problem in your branch even if you hadn't taken the steps that you
did?
A: Possibly, yes.
1092. Post Office respectfully submits that the contemporaneous documents make clear that,
even without Mrs Burke’s diligence, the branch would have been compensated for the
failed recovery through a TC, in the ordinary way. Mrs Burke confirmed in cross-
examination that her usual experience was that discrepancies that could not be resolved
by the branch itself were resolved through TCs.'*43
Conclusion
1093. Given that Mrs Burke is not a claimant and that there is a fairly full evidential record
before the Court (including from Mrs Burke herself), Post Office accepts that unqualified
findings can properly be made.
1094. Mrs Burke was an honest and careful witness. She was a skilled and diligent SPM and
assistant who appears to have encountered very few difficulties in operating Horizon
over the years.'*4
1095. It is regrettable that Mrs Burke was not saved the trouble of conducting her own
investigations. The Helpline operators could have made it clearer to Mrs Burke that there
34 Para 110 {E2/5/26}.
138 Mrs Burke confirmed this date at {Day3/94:20} to {Day3/95:2}
18 (Day3/69:2} to {Day3/70:12}.
1M See {Day3/68:20} to {Day3/70:12}. Note also the very few TCs shown at {F/1687.1}.
364
AI8/364
1096.
1097.
1098.
1099.
1100.
1101.
would be a process of identifying unrecovered transactions and issuing TCs to affected
branches as appropriate, allowing her to simply wait for the TC, as she says she usually
did when faced with a shortfall that she could not herself resolve.'*4°
Nonetheless, the documentary evidence shows that the supporting process around
Horizon worked properly to identify the need for a TC resulting from the failed recovery.
It follows that, even without Mrs Burke’s diligence, the branch would not have suffered
a lasting discrepancy. Mrs Burke’s case provides no evidence of any bugs or errors in
Horizon (beyond the point that the system, like any other, suffers occasionally from
system outages, the consequences of which are addressed by the failed recovery
procedures outlined above).
H2. Mr Henderson’s Evidence
Mr Henderson’s witness statement is at {E1/5}.
His evidence can be taken shortly. The Court made clear at the PTR that it would not
permit the trial to be side-tracked into a consideration of whether or not conclusions in
Second Sight’s various reports were correct: see the Judgment dated 14 February 2019
at paras 10 and 11 {C7/41/4}.
It is respectfully submitted that the Court was right to do this. Mr Henderson had little,
if any, admissible evidence to give in relation to the Horizon Issues and it is difficult to
understand why Cs felt it necessary — or even appropriate — to call him as a witness.
Post Office cross-examined Mr Henderson purely on matters of which he could give
non-opinion evidence. This was explained to Mr Henderson at the outset.'*4°
He gave clear and fair answers to cross-examination. He confirmed the following in
relation to the Mediation Scheme and Second Sight’s role in it:
POL00026925
POL00026925
15 See {Day3/70:4} to {Day3/70:12}. It appears that Mrs Burke had not encountered a failed recovery
before
9 May 2016, despite working in Post Office branches for many years.
1M6 (Day4/166:11}.
365
AI8/365
1101.1 Second Sight’s investigation was substantially broader than the Horizon
Issues.'*47
1101.2 The anticipation at the outset of the Mediation Scheme was that the Working
Group would set relatively demanding timeframes and progress quite quickly.'***
1101.3 Second Sight’s role in the Mediation Scheme was to produce thematic reports
and also case review reports (“CRRs”) on the complaints made by individual
participants in the scheme.'*”
1101.4 Second Sight was, at least from July 2014,!°°° not required to determine every
issue raised by an SPM, but only to carry out a reasonable investigation and offer
an opinion on key issues.!**!
1101.5 Under the Mediation Scheme, Second Sight’s CRR was one of three key
documents to go before the Working Group to enable it to decide whether or not
to recommend mediation, the other two being the applicant’s complaint and Post
Office’s response to it.'*°?
1101.6 The progress of the Mediation Scheme, including the preparation of those
documents, went slower than had been hoped.'**? This is apparent from the
contemporaneous documents at {F/1325/137} (letter from Sir Anthony Hooper in
mid-December 2014) and {F/1325/186} (letter from CEDR in March 2015), the
latter showing that few of the 136 cases in the scheme had made it to the mediation
stage.
POL00026925
POL00026925
1 (May4/167:3} to {Day4/167:23}
1348 (Day4/172:24}.
5” (Day4/169:19} to {Day4/169:22}.
13590
Post Office does not have any earlier agreement of similar detail to the July 2014 agreement at
{FA228.1}
851 (Day4/172:11}.
13592
{Day4/174:14} to {Day4/175:2} (Mr Henderson quite fairly said that his recollection was slightly
different but his answer was largely in agreement).
853 (Day4/175:14}.
366
AI8/366
1102.
POL00026925
POL00026925
1101.7 It was broadly consistent with Mr Henderson’s recollection that by March 2015
there had been only 12 mediations.!°**
Mr Henderson also gave clear and fair oral evidence in relation to the termination of
Second Sight’s engagement in March 2015. The termination letter is at {F/1324.1}. A
much longer letter, sent by Post Office on the same date, is at {F/1324.2}. That longer
letter set out proposed terms for Second Sight to complete its outstanding work. Mr
Henderson explained that he was told by Post Office at the time that “we were at the
point where the mediation process using CEDR could continue without any further input
from the working group and that alternative arrangements would be made for Second
1103.
1104.
Sight to complete the I think it was 20 outstanding reports at that point” °°
There followed exchanges of correspondence'#*° through which new arrangements were
negotiated, ultimately resulting in the agreement at {F/1333.2}. Under that agreement,
which is dated 15 April 2015, Second Sight was given until 31 May 2015 to complete
the outstanding CRRs and otherwise continued to work on terms that were relatively
similar to those that pre-dated the termination.
Mr Henderson confirmed the context in which the new arrangements were put in place
in March-April 2015:
1104.1 By the time of the termination, Version 2 of the Part 2 Briefing Report (the final
thematic report) was “substantially complete”, and Second Sight were happy to
agree to complete it by 10 April under a new agreement. A first draft of the
report was in fact overdue by the date of termination. !**
854 (Dayd/178:12}.
855 (Day4/180:15} to {Day4/180:20}.
BS6 Th
(letter
2015).
key letters and emails are at {F/1327.1} (letter from Second Sight dated 17 March 2015), {F/1328.1}
from Post Office dated 26 March 2015) and {F/1329.1} (email from Second Sight dated 30 March
857 (Day4/180:24} to {Dayd/181:3}.
858 (Day4/181:10}.
367
AI8/367
POL00026925
POL00026925
1104.2 The CRRs that had not been completed by the time of termination were produced
later than had been anticipated by the Working Group, but under the new
arrangements Second Sight was able to produce them in relatively short order.!5°?
1104.3 Post Office and Second Sight engaged in negotiations over the new arrangements
(referred to above), and the main point of difficulty was (as Mr Henderson could
now recall) whether Post Office should pay for Second Sight’s work indirectly
through payments to applicants or through direct payment. Second Sight wanted
to continue with direct payment, and Post Office “readily agreed” to that. '*°°
1105. The short point is that Second Sight was given time to complete the outstanding work.
There is no hint in any of the correspondence following the termination that Post Office
was requiring Second Sight to comply with impossible or even unduly difficult deadlines
for the outstanding CRRs. As noted above, the Version 2 report was already substantially
complete.
1106. In that context, some of the language used by Mr Henderson in his witness statement
(see para 2.6 {E1/5/5}, for example) is unfortunate in that it might tend to suggest to the
reader that the termination brought Second Sight’s work to a premature end. Cs certainly
try to paint that picture when referring to the termination of Second Sight’s engagement.
It is an unfair picture.
1107. As Mr Henderson acknowledged, the closure of the Working Group did not put a stop to
mediations. On the contrary, Post Office offered mediation to all the remaining applicants
in the scheme (save for criminal cases).
1108. Mr Henderson gave surprising evidence in relation to confidentiality restrictions that he
said had prevented him giving the evidence that he otherwise would have given. He said
this at the start of his cross-examination: “/’m a party to an agreement between sort of
Post Office and the claimants that restricts the matters on which I can give evidence, so
that is a further limitation in my evidence today”.'3°!
"89 {Day4/182:15} to (Day4/182:23}. Second Sight was under a reasonable endeavours obligation to comply
with deadlines set by the Working Group: see clause 5.3 of the agreement at {3.41.2 3.1
136 {Day4/181:20} to {Day4/181:20}
Be {Day4/166:1}.
368
AI8/368
1109,
1110.
1111.
1112.
POL00026925
POL00026925
This was the first time it had been suggested to Post Office that Mr Henderson had
evidence that he wished to give but was prevented from giving by a confidentiality
restriction. No such point had been raised with Post Office in correspondence, despite
the parties having corresponded at length in relation to the content of Mr Henderson’s
evidence (specifically, as to its status and admissibility).'*
The only potentially relevant confidentiality restrictions are those contained in the
agreements to which Mr Henderson was taken in cross-examination. However:
1110.1 Those restrictions were very substantially relaxed (at Cs’ request) by a detailed
agreement drafted for the purposes of these proceedings. It is at {F/1679.1} (“the
Protocol”). The only matters that remain “off limits” under the Protocol are
identified at clause 3.1.5 (principally, information subject to legal professional
privilege). The Protocol requires cooperation (clauses 3.1.4 and 3.1.6).
1110.2 The Protocol also contains the following provision at clause 9.1
Any issues, disputes or matters for resolution or determination relating to this
protocol shall be referred to the Managing Judge or the Managing Master
nominated to manage the GLO.
There was therefore a clear mechanism for raising any concerns as to the confidentiality
restrictions. It may not be Mr Henderson’s fault that they were not used, but the fact they
exist makes it very difficult to understand how he can have come to be in the difficult
position to which he described in his oral evidence.'*
Mr Henderson was asked whether confidentiality restrictions had caused him any
inhibition in answering the questions put to him in cross-examination. He said that he
had the issue in the back of his mind and that he had tried to make sure that his answers
did not infringe the protocol.'*! It is understandable that Mr Henderson would wish to
be careful, but the idea that he was restricted in answering Post Office’s questions is
bizarre. He was asked, for the most part, about the conduct and administration of the
mediation scheme ~ matters of public record, and nothing to do with (or even close to)
B See {H/126} and {H/127}.
136 The only hint that confidentiality restrictions might be relevant in the witness statement is at para 1.5,
where
Mr Henderson says that he believes he has complied (without suggesting any concern or difficulty)
{E1/5/3}.
1364 (Day3/187:3}.
369
AI8/369
POL00026925
POL00026925
legal professional privilege or data personal to non-claimants. Nothing that he was asked
can have caused him difficulty.
I. DISCLOSURE
i. Background
1113. There were various complaints about disclosure made by Cs in their Opening
Submissions. These have been repeated and expanded during this trial.
1114. In circumstances where Cs have: (i) not made any formal applications for specific
disclosure; (ii) not suggested that Post Office has breached any Order for disclosure; and
(iii) failed to follow an agreed and Court ordered procedure for resolving any differences
regarding disclosure, it is inappropriate for criticism to be directed at Post Office.
1115. The Court is reminded that because of the way in which the Horizon trial developed, Cs
have never presented a properly pleaded case and that there are no detailed allegations
made by reference to which disclosure was to be given. This has put Post Office into a
difficult position.
1116. The Court directed Cs to provide an outline case.'*° This was served on 17 August
2018.13 It was no substitute for a properly pleaded case, provided virtually no assistance
to Post Office and has hardly been referred to by either party.
1117. Instead, Cs have developed their case through Mr Coyne’s reports which are sprawling,
scattergun documents. His so-called “Supplemental Report” (i.e. Coyne 2) is 273 pages
long. It was served on I February 2019, a little over one month before the start of trial,
presented an entirely new analysis and raised many new issues.
1365 (€7/12/5} para 15
1365 {Coyne HAL ICL/2. (our
370
A/8/370
POL00026925
POL00026925
1118. Cs’ general position appears to be that, notwithstanding this background, Post Office
should have been able to infer all the documents Cs was likely to want without any
proper, still less timely, explanation from Cs as to the categories of documents sought,
and should have volunteered those documents even in the absence of any order or
application.
12. Model C Disclosure
1119. By the time the Horizon trial was ordered, it had already been ordered that there was to
be no standard disclosure; and that Model C disclosure was to apply instead. Paragraph
5 of the Order dated 2/2/18!°°’ provided that:
Save as otherwise specifically provided in this Order, pursuant to CPR .31.5(7)(c),
such disclosure is to be given on an issue by issue basis. Such disclosure shall be
given in accordance with Model C and the draft Practice Direction ‘Disclosure Pilot
for the Business and Property Courts' (the "draft Practice Direction"), which shall be
adopted save as follows:-
(a) The requirements of Paragraph 5 (Basic Disclosure) be dispensed with.
(b) The requirement upon the parties to produce Disclosure Review Documents in
accordance with paragraphs 10 and Appendix 2 of the draft Practice Direction is
dispensed with, as is any further requirement imposed upon the parties relating to
Disclosure Review Documents.
(c) For all relevant purposes, any reference in the draft Practice Direction to
sure Review Documents shall be taken as a reference to the E
Dis losure Questionnaires exchanged by the parties in accordance with paragraph
11 of the First CMC Order.
1120. In relation to Extended Disclosure generally (of which Model C is one type) 51UPD.6
para 6.5 provides that:
“Ht is for the party requesting Extended Disclosure to show that what is sought is
appropriate, reasonable and proportionate (as defined in paragraph 6.4)...”
1121. Model C requires focused and specific requests: i.e. it is “Request-led search-based
disclosure”. 51UPD.8 provides in relation to Model C that:
1367
371
A/8/371
1122.
1123.
1124.
1125.
1126.
(1) The court may order a party to give disclosure of particular documents or narrow
classes of documents relating to a particular Issue for Disclosure, by reference to
requests set out in or to be set out in Section 1B of the Disclosure Review
Document or otherwise defined by the court.
(2) If the parties cannot agree that disclosure should be given, or the disclosure to be
given, pursuant to a request, then the requesting party must raise the request at
the case management conference...
The reality of Cs’ approach over the last year is that what Cs really wanted — and think
they are entitled to — is standard disclosure. Cs have not produced properly focused
requests for particular documents or narrow classes of documents; and Post Office has
done far more than it was obliged to in providing documentation. Cs have also not sought
to raise any complaints with the Court.
The significant challenges started with Cs adopting a parallel track of seeking docs —
some from Freeths, following Model C; and some presented as “Requests for
Information” from Mr Coyne. Freeths did nothing to control this process or to ensure Mr
Coyne’s requests complied with Model C.
Given the emphasis in this trial on disclosure, it is unfortunately necessary to remind the
Court of some of the detail of the relevant background.
By the Court’s Fourth CMC Order'*°* Post Office was ordered to disclose the documents
set out in Schedule I to the Order. Schedule I was largely agreed by the parties (mainly
before the CMC but some of it after). It focused on various documents (in particular
reports and briefings) being provided to various categories of custodians which were
defined in the Schedule itself.
This was an example of Model C disclosure working broadly as it was intended to: Cs
identifying categories of documents which related, or appeared to relate, to the issues
such as they had been identified at that time.
POL00026925
POL00026925
368 (C7/18}.
AI8/372
POL00026925
POL00026925
13. Expert-led Requests for disclosure
1127. By paragraph 13 of the Third CMC Order'*® following a hearing on 22 February 2018
the Court wanted!” the following provision in the Order:
The parties and their IT experts are reminded that experts have a right, pursuant to
CPR 135,14, to file written requests to the Court for directions for the purpose of
assisting them in carrying out their functions.
1128. Pursuant to this, Mr Coyne made his own separate requests for information and
documents and it is this separate stream of requests which has given rise to many of the
complaints in the case. It is unclear to Post Office what if any involvement Freeths had
in this process. It should have taken control of Mr Coyne and ensured that his requests
aligned with the requirements of Model C. Freeths failed to do so, and instead allowed
Mr Coyne to pursue these matters.
1129. Mr Coyne’s first request was sent to the Court on 29 May 2018.!°”! Post Office responded
to this on 4 June 2018'*” i.e. the day before the Fourth CMC. At that CMC, the Court,
by paragraphs 8 and 9 of the Order, ordered the experts to provide an Error Codes List
and jointly to compile a list of information which either or both considered they
required.'57°
1130. This joint report was duly produced on 26 June 2018.15” It is a request for a huge amount
of information, explanation and documentation. Dr Worden did not support the requests
since his view was that his current requirements for information were already being met
and that he preferred to develop his understanding further before deciding to ask “the
right focused questions ”.375
18 {C7/12/4}.
8 {C8,4/4/73} line B.
1871 (05/8).
BP ICS/LL}.
8 {C7/18/3}.
184 (C5/13}.
875 (€3/13/2}.
373
AI8/373
1131.
1132.
1133.
1134.
On 20 July 2018 Mr Coyne sent an email to Freeths and WBD requesting various wide-
ranging requests for documents which he wanted following an inspection of the TfS and
Peak systems.!>”° These documents were generally Fujitsu documents and Post Office
did not understand many of the requests or precisely why they were relevant to issues in
the case.
14. The Fifth CMC Order
Some mechanism was clearly required in order that (i) each side’s position on the various
requests being generated by Mr Coyne was properly set out and could be fully considered
and (ii) the requirements of Model C were met.
The parties agreed a sensible mechanism which the Court approved in the Fifth CMC
Order dated 24 July 2018.'57’ The parties agreed, and the Court ordered, the following:
1. The Defendant shall, by 8 August 2018, provide to both experts the information
requested by Mr Coyne (i) in the "Requests for Information" documents sent 12 July
2018 and dated 26 June 2018 (ii) in relation to the PEAK system, by email on 20 July
2018, save in respect of any requests to which the Defendant serves an objection
(stating reasons) in writing, by the same date.
2. If the Defendant objects to a request, and the Claimants still wish to pursue that
request, the Claimants will within 10 days of the. Defendant's objection explain in
writing (i) the relevance of the request to the determination of the Horizon Issues (ii)
why the request is reasonable and proportionate and (iii) why it falls within CPR 35.9
Ifthe procedure set out in this Order had been followed by Cs, they would have explained
the relevance, reasonableness and proportionality of any disputed requests by the middle
of August 2018. With the benefit of such explanations, the parties would have been in a
position to liaise and cooperate with each other with a view to achieving an agreed
position. If any issues remained these could and should have been brought before the
Court and determined, long before the trial.
POL00026925
POL00026925
118 (C5/16/1}.
87 (C7/22}.
374
AI8/374
1135,
1136.
1137.
1138.
1139.
Many of the challenges relating to disclosure stem from Cs’ failure to comply with the
terms of this Order, and with the requirements of Model C Disclosure. Cs have never
provided any explanation as to why they did not comply with the Fifth CMC Order.
Post Office provided its response, as required by the Fifth CMC Order, on 8 August
2018.75 In many cases, it provided the information requested and additional
documentation. In other cases it did not. Post Office was genuinely unsure of what Cs”
case was in relation to many of the Horizon Issues — neither the Outline Document nor
Mr Coyne’s reports had been served at this point — and many of the requests sought not
categories of documents but lengthy analyses and explanations of matters which Post
Office did not have and which appeared to cut across the carefully negotiated categories
of documentation which the Court had ordered at the Fourth CMC.
Following this impasse, Cs were put to their election. Pursuant to the Fifth CMC Order,
Cs should have served an explanation by 18 August 2018 if they maintained a claim to
any of these requests, together with an explanation of how they related to the Horizon
Issues. This would have enabled Post Office to consider whether the requests were valid
Model C requests relevant to an issue in the case. Cs did not do so: and have never
provided any explanation as to why they feel able to flout the Court’s Order.
On 19 September 2018'5” Freeths sent WBD a document from Mr Coyne marked
“Draft”'5®° which sought to resurrect various of his original requests. WBD responded
on 25 September 2018'**! pointing out that Cs had failed to comply with the Fifth CMC
Order, either as to time or substance and requesting a proper explanation as to why the
requested documents were relevant to the Horizon Issues and reasonable and
proportionate.
Nothing was heard until Mr Coyne’s further RFI dated 14 December 2018'5*? which
repeated some of the original requests and introduced some new ones. WBD responded
POL00026925
POL00026925
378 (C5/21}.
81 (C5/25}.
38 £C5/26}
1881 (C5/27}.
18 (C5/29}.
375
AI8/375
POL00026925
POL00026925
on 21 December 2018!** again pointing out the flagrant breach of the Fifth CMC Order
and asking again for an explanation. Despite not receiving any such explanations, Post
Office had by this time seen Coyne I and WBD did respond to Mr Coyne’s latest RFI
on 17 January 2019.38*
1140. This is a deeply unsatisfactory approach by Cs to disclosure in a difficult, complex trial
where the issues have never been properly pleaded.
15. Burden on Post Office
1141. The burden of disclosure on PO has been immense. In this regard, the disparity between
the respective burdens on the parties has also been extraordinary: Post Office has
disclosed some 517,965 documents and Cs only 1525. The approach of Cs to their own
disclosure obligations has been unsatisfactory, as set out below.
1142. Moreover, Post Office has been heavily dependent on Fujitsu who themselves have had
to search for documents over a 20-year period. This is exceptional. It will only be
necessary on the rarest of occasions for searches to take place over such a lengthy period
of time and on the scale required here. It is inevitable that there will be archiving and
deletions and that the searches will be complex and in some cases unsuccessful.
1143. Post Office’s approach has been as follows:
1143.1 Where a proper Model C request has been made for relevant documents, Post
Office responded co-operatively and has generally given far more documents than
Cs were strictly entitled to;
1143.2 On many occasions, the breadth of docs requested has been enormous.
Documents were often not kept in the way the request pre-supposed e.g. where Cs
asked for documents relating to bugs which affected branches or the like. Where
Post Office or Fujitsu could not provide such documents, Post Office has resisted
188 £€5/30}.
18 £C5/33}.
376
AI8/376
1144.
1145,
1146.
1147,
1148.
the full request — often not refusing outright but reasonably requesting a narrowing
of the category or further explanation of relevance.
Cs’ approach has hampered this process. Freeths did not control Mr Coyne’s requests,
nor did it follow them up as required by the Fifth CMC Order. If Cs had genuine concerns
about the scope and contents of Post Office’s disclosure, they could and should have
made appropriate applications to the Court back in July or August 2018. They did not do
so. Even today, after the issue of disclosure has featured repeatedly, Cs have advanced
no complaint that Post Office is in breach of any Court Order.
16. The Court’s interventions on disclosure
Post Office is concerned that the Court may have lost sight of the nature of the Orders
made and of the approach which the Court ordered. It is striking that Cs have not
themselves made any applications for specific disclosure nor have they advanced any
complaint that particular disclosure orders have not been complied with.
The recent judgment of the Court of Appeal in Serafin v Malkiewicz & ors [2019] EWCA
Civ 852 @ para 118 is relevant:
“We are also highly troubled by the repeated demands and criticisms by the Judge
regarding the Claimant’s disclosure, in circumstances where pre-trial disclosure had
been completed by both sides at a time when both the Claimant and Defendants had
been represented by solicitors and counsel, and no application for further disclosure
had been made by the Defendants...”
KELs
Post Office has never denied that KELs existed. There was debate in the early stages as
to their relevance. However, as early as 22 September 2017 Post Office offered to allow
Cs’ expert to inspect the KEL database.'**°
KELs were agreed to be disclosed once the Horizon Issues Trial was ordered, and were
disclosed on 9 May 2018. Post Office was not initially made aware by Fujitsu that some
POL00026925
POL00026925
185 (H/13}.
377
AI8/377
1149.
1150.
ISL.
1152.
1153.
POL00026925
POL00026925
KELs had been deleted (meaning archived). Once that was discovered the additional
KELs were disclosed on 17 January 2019,158°
In any event, Cs had the KELs in good time. Mr Coyne says in Coyne I that he examined
5,114 of them. There is therefore no legitimate complaint to be made here.
Peaks
The Peak system was included in Post Office’s EDQ filed on 6 December 2017 before
any disclosure ordered. See {C9/1/47}: “If Fujitsu identifies an issue in Horizon or
Horizon Online that requires a programmatic fix then it is logged in its database, the
Peak System, and labelled as a 'Peak'.”
Mr Coyne asked for access to the Peak system in his RFI dated 4 June 2018.'*87 By para
6 of the Fourth CMC Order'*** Post Office made arrangements for the experts to have
joint access to Peak database on 15 June 2018.
Following the Fifth CMC Order, WBD sent a response’*®° to Mr Coyne's RFI'*”° and his
email dated 20 July 2018'*?! saying that Post Office was working with Fujitsu to see if
some mechanism could be created to provide access to 200,000 Peak entries. The Peaks
are stored in a proprietary Fujitsu database — raw data disclosure of which would be
illegible and useless. This exercise was a software project in its own right which took
time and a considerable amount of resources.
Mr Coyne was invited to a second day of Peak inspection but he did not take up
invitation.
1386 {11/169/8}.
187 {C5 /11/4}.
1388 (C7/18/2}.
1389 (5/21).
1390 (C5 /8}.
1991 (€5/22}.
378
AI8/378
1154,
1155.
1156.
1157.
POL00026925
POL00026925
Disclosure of c.218,000 Peaks was given on 27 September 2018. A further 3,885 Peaks
were disclosed on 25 October 2018 after review — these had been responsive to a
privileged search term.
Post Office did all it reasonably could to accommodate Cs’ requests here.
MSCs & OCPs
As part of Mr Coyne's email on 20 July 2018, disclosure was sought of any Master
Service Change (“MSC”), Operational Corrective Requests (“OCR”) or Operation
Control Procedures (“OCP”) “where the data to be changed has had a financial impact
on Post Office or where they relate to fixing a peak”.*°? On I August 2018, Post Office
sought clarification of this request,'*** in particular why the MSCs, OCRs and OCPs
were relevant to the Horizon Issues and what was meant by “financial impact on Post
Office”. No response was received to this request for clarification.
1394 on 8
Pursuant to the mechanism set out in paragraph I of the Fifth CMC Order,
August 2018 Post Office objected to the request for disclosure of the MSCs, OCRs and
OCPs on the basis that it would be necessary for Fujitsu to carry out a retrospective
analysis to attempt to locate the documents which would fall within the disclosure
request since the information had not been pooled or collated as part of ordinary working
practices. If the Cs wished to continue to pursue these requests then they were required
by 18 August 2018 to provide the explanation and particulars required by paragraph 2 of
the Fifth CMC Order.'*°*
592 (C5/16}.
898 5/17}
894 (C7/22/1}.
895 (C7/22/1}.
379
AI8/379
1158. Cs did not reply until 19 September 2018!°°° when Freeths wrote to WBD explaining
that Mr Coyne had considered Post Office's responses to his RFIs and asked Post Office
to reconsider its objection to the disclosure of the MSCs, OCRs and OCPs.
1159. On 25 September 2018, '*”” further information was sought by WBD from Cs as to why
these requests for disclosure were relevant to the determination of the Horizon Issues.
1160. On 14 December 2018, '*°8 almost 3 months after WBD’s letter of 25 September 2018
and almost 4 months later than ordered by the Court in the Fifth CMC Order, Mr Coyne
provided a revised RFI which contained those outstanding requests which he believed
were of the most significance to his preparation of his supplemental report. The revised
RFI requested disclosure of the MSC, OCRs and OCPs, gave a brief response to the
clarification sought on the meaning of financial impact and then merely stated "Request
still valid".
1161. Disclosure of the MSCs was given on 21 December 2018. '°? On 17 January 2017,!4
WBD explained that Fujitsu had developed and tested a solution to extract the OCPs and
disclosure of the OCPs and OCRs was given on 24 January 2019.'4"
1162. On 18 April 2019 WBD wrote to Cs and explained that Post Office had recently been
provided with an additional tranche of OCRs by Fujitsu and provided disclosure in
respect of them.'*” In a letter dated 3 May 2019 WBD set out how these additional OCRs
had come to the attention of Post Office.'4* In short, Fujitsu had found a back-up of pre-
‘August 2006 OCRs which Fujitsu had previously believed no longer existed. Further
detail was provided by WBD in its letter dated 3 June 2019.'** At the Court’s direction,
POL00026925
POL00026925
1396 165/25}.
1397 (C5/27}.
138 (€5/28}; {C5/29}.
1399 11/155}.
M00 45/33/9}.
M01 11/179}.
402 11/263}.
M03 11/273}.
M04 (11/323}.
380
A/8/380
a witness statement was provided which sets out the history of the disclosure of these
further OCRs in April 2019."4°°
Release Notes
1163. Complaint was also made in Cs’ Opening Submissions about release notes. Cs’ account
of this is misleading. An accurate history is set out in WBD’s letter of 5 March 2019,'4°°
1164. Mr Coyne did not originally ask for all Release Notes; he asked for a chronology of
them. No such chronology existed but Post Office with help from Fujitsu created one,
which was provided on 8 August 2018.
1165. On 8 August 2018 Post Office invited Mr Coyne to say if there were any particular
Release Notes he was interested in but that invitation was not taken up.'4”” Nor was any
complaint raised pursuant to the Fifth CMC Order. In the circumstances Post Office
reasonably assumed that the request had been answered.
1166. On 14 December 2018 Mr Coyne raised what appeared to be a repeat of his original
request in which he sought “the dates and release names (or version name) of each
version/release of software that was in use in branches from Horizon inception to present
day”.\*°8 Disclosure of particular release notes were not asked for.
1167. Post Office responded on 17 January 2019'*° saying that it did not have branch by
branch information of the type requested.
1168. On 14 February 2019!" Cs said that what they were seeking were names and documents
such as release notes, of different versions of Horizon software used in branches during
the currency of Horizon. This was the first request for a copy of an actual release note.
Andrew Parsons’ 18" witness statement {C11/23}.
406 {1/233}.
407 (H/233}.
408 £C5/29/6}.
409 (C5/33/11} to {C5/33/12}.
410 FH/204}.
381
POL00026925
POL00026925
A/8/381
1169.
1170.
1171.
1172.
POL00026925
POL00026925
The release notes requested were provided to Frecths by way of letter on 5 March
2019,!411
Royal Mail disclosure
There was a mix-up made by Post Office in relation to the disclosure of certain
documents held by Royal Mail. These documents were not held by Post Office but by a
third party, namely Royal Mail. WBD mistakenly believed that pre-2011 E&Y audit
reports had been requested from Royal Mail whereas in fact the request made had been
narrower. As soon as this was discovered, Freeths and the Court were informed. The
1412
Court ordered a witness statement to be provided and the situation was explaine
The incident was regrettable but should not be the subject of criticism of Post Office.
. Cs’ Own Disclosure
Despite Post Office making appropriate and narrow requests for Model C disclosure of
documents held by those claimants who were witnesses in the Horizon Issues Trial, Cs
have not engaged in discussions concerning the scope of Cs disclosure nor provided
disclosure of the documents sought by Post Office.
By paragraph 5 of the Fourth CMC Order, Cs were ordered to provide disclosure of
documents upon which they intend to rely at Horizon Issues Trial by 17 July 2018:
the reason for such an order was that there was to be no claimant-specific evidence. Cs
were also required to provide disclosure of known adverse documents as an ongoing duty
under Practice Direction 51U, part 3.1(2).'4"4
It should be noted that the Fourth CMC Order provided for witness statements to be
limited to "any witness of fact whose generic evidence (in distinction to Claimant-
M11 77/233}.
42 (CLI/I7}.
MIB (C7/18/2}.
4414 The draft Practice Direction Disclosure Pilot for the Business and Property Courts, which is now known
as CPR 51U, was ordered to be adopted by the parties at paragraph 5 of the Second CMC Order {C7/11/2}.
AI8/382
1173.
1174,
1175.
1176.
1177.
POL00026925
POL00026925
specific evidence) they wish to rely upon for the purposes of determining the Horizon
Issues."4'S
It was therefore not appropriate for Cs to serve claimant-specific evidence from current
or former SPMs and therefore disclosure of documents from such custodians was not
sought by Post Office at the time of the Fourth CMC Order. Given the limit on the scope
of the evidence, orders for such disclosure were not envisaged as being required by either
the Court or Post Office.
On 17 July 2018 Cs provided disclosure of 45 documents. At this time it was not known
to Post Office that Cs would be serving extensive claimant-specific witness statements.
Cs premised their disclosure with the comment that “For the avoidance of doubt, the
intention of paragraph 5 of the Fourth CMC order is not to preclude the Claimants from
relying on any further documents that come to their attention and fall to be disclosed,
pursuant to the their ongoing duty of disclosure. ""'°
Post Office was concerned by these comments and sought confirmation from Cs on 20
July 2018 as to how they had sought to identify disclosable documents.'*!7 On 30 July
2018, the Cs responded stating that “additional relevant documents may come to their
attention, and most likely as a result of the work of the experts.”"">
No mention was made by Cs that they would be serving claimant-specific evidence and
that no disclosure had been provided by any of those claimant witnesses on 17 July 2018.
Cs did however confirm that “However, at present, it is not the Claimants’ intention to
make another round of disclosure in relation to the Horizon Issues trial.”"*"
Between 2 August 2018 and 25 September 2018 Post Office continued to press Freeths
on the scope of disclosure which had been given by Cs.'#°
1415 Paragraph 10; {C7/18/3}.
M16 (H/91}.
M17 11/93}.
M18 (H/97}.
1419 H/97},
1420 See correspondence from WBD to Freeths on 2 August 2018 {1/101}, from Freeths to WBD on 3 August
2018 {H/102}, from WBD to Freeths on 12 September 2018 {H/113/2} and 25 September 2018 {H/117}.
383,
AI8/383
1178.
1179,
1180.
1181.
1182.
Post Office was therefore seeking to work with Cs to ensure that as far as possible all
relevant disclosure for the Horizon Issues Trial had been provided by Cs so as to avoid
a repeat of the late disclosure of documents close to trial which had happened in the
Common Issues Trial. At the same time, in relation the Common Issues Trial, Cs
provided on 6 September 2018 the fourth additional round of Lead Claimant disclosure
which amounted to c.1451 pages and should have been disclosed in February 2018
pursuant to the Court's Order.'4?!
On 28 September 2018, Cs served 9 witness statements, 6 of which were claimant-
specific evidence from current or former SPMs (or, in one case, the son of an SPM).
These included witness statements were served by Mr Latif!? and Mr Tank.'43
These exhibited seventeen documents none of which had not been previously disclosed.
The disclosure list contained the documents exhibited to Cs’ witness statements and a
number of other documents relating to Dalmellington, Newport and Remuneration
overpayments were provided on 2 October 2018.'**4 A total of thirty new documents
were disclosed by Cs.
In light of further disclosure provided by Cs, on I October 2018 Post Office continued
to seek information from Cs on the scope of their disclosure.'*”> Post Office wrote to the
Cs again on 22 October 2018,'#° 7 November 2018,'4?” 30 November 2018'** and 20
December 2018.'*”? No response was received.
Cs responded on 14 January 2019, stating that the reason for disclosure not being ordered
by specific Claimants for the Horizon Issues Trial was: "Given the nature of the Horizon
POL00026925
POL00026925
Issues Trial, there was good reason for the court to seek to limit the ambit and cost of
421 (H/113/1}. The total number of documents disclosed by the Claimants in these four additional tranches
amounted to 433 documents (2,149 pages) meaning 10% was disclosed late by the Cs with no explanation.
M22 EB
1023
1}
1/6}.
M24 (H/122}.
425 (H1/120.1}.
6426 (H/130}.
M7 (H/133/2}.
628 (H/142}.
62 FH/152}.
384
AI8/384
1183.
1184,
1185.
POL00026925
POL00026925
disclosure from the Claimants themselves. Hence, the disclosure order made here; the
obvious good sense of that has not changed.""*°
This reasoning was of course misconceived in light of the fact that Cs had served the
claimant-specific evidence.
Given that Cs had flouted that Order, Post Office sought to understand what searches (if
any) had been conducted by Cs so that it could understand whether further disclosure
would be required in light of Cs evidence. A further request for details by Cs was made
by Post Office on 17 January 2019.'4*!
This lack of response by the Cs should be viewed in light of the approach adopted by
Post Office. One example of the further disclosure that Post Office has agreed to give to
Cs which is outside that ordered by the Court is the requests for disclosure made by Cs
on 18 December 2018 in which the Cs sought:
"... disclosure of the documents that were responsive to searches by, and collated by,
the Defendant in respect of the operation of branches by Angela Burke, Akash Patny,
Anup Patny, Jayesh Tank, Setpal Singh and Adrees Latif. We would expect such
documents to include but not be limited to:
Filtered transactional data
Unfiltered transactional data
Audit Request Query data
Audit Store data
NBSC Call logs
Lists of all transaction corrections sent to the branch.
Internal Post Office communications regarding (or relating to) the issues addressed in
the witness statements of the above-named individuals.
Adverse documents relating to the above-named individuals.""*?
M80 (H/167}.
431 (H/169/6}.
682 (H/149}.
385
AI8/385
1186. Further requests for claimant-specific disclosure were made by Cs of Post Office on 4
February 2019, with Cs raising 16 new clarification / disclosure requests. '*> Post Office
responded to these requests on 11 February 2019'*** and provided disclosure of these
documents on 20 February 2019."*5 The failure by Cs to provide their own claimant-
specific disclosure suggests that they clearly intended disclosure to be a one-sided
exercise.
1187. On 26 February 2019, Cs wrote to Post Office explaining that:
"Mr Tank's review of the responsive witness evidence given by Ms Van Den Bogerd
at paragraphs 75-84 of her second witness statement, including Ms Van Den
Bogerd’s description of a £195.04 shortfall and the related PEAK, has prompted Mr
Tank to search within an archived Subpostmasters “Yahoo Groups” forum that he
used to post in for any relevant posts.
From this search, Mr Tank has identified a post he made on 13 December 2011
relating to the £195.04 shortfall referred to by Ms Van Den Bogerd and also a post
he made on 29 September 2014 relating to the £600 shortfall that he refers to in
paragraphs 6-11 of his witness statement. Mr Tank has also found a letter he
received from Post Office’s Agents Accounting Team in Chesterfield dated 13
October 2014 which relates to this shortfall (and which Mr Tank referred to in his
forum post)."1
1188. Ms Van Den Bogerd’s second witness statement had been served on 16 November 2018
and no explanation was provided as the timing of this disclosure or as to why such a long
period had passed between service of the witness statement and this disclosure being
provided. ‘37
Mr Tank
1189. During cross examination of Mr Tank (Day 2), Mr Tank was asked a number of questions
about the approach which he had taken in relation to disclosure.
Q. And when you prepared your first witness statement you obviously knew this
resource existed, didn't you?
A. I did.
POL00026925
POL00026925
1433 11/186}.
4M (H1/196/6}.
435 (H/213}
6496 (1/224).
437 (E/5}.
386
AI8/386
POL00026925
POL00026925
Q. And it is plain that it is relevant to your evidence, isn't it?
A. Itis.
Q. Why didn't you think to look for this material when you were preparing your first
witness statement?
A, Because my first witness statement was I think short, brief, and it was just my way
of -- I didn't really fully research the whole background regarding it, I just put my
Statement in.
Q. Can I just press you a little bit on that. You understand that what's being said in
this case, by you amongst many others, is that you carried out certain very specific
actions which you did correctly and that the Post Office is at fault, or the Horizon
system is at fault?
A. Yes.
Q. So it is important, isn't it, to have been precise in the evidence that you give?
A. Yes.
Q. And I'm just wondering why you didn't make some effort to find what was plainly
a relevant document in putting forward what you agree to be the need for precise
evidence?
A. Because I didn't -- I didn't feel that my information in my initial witness statement
was going to be taken any further, so I -- it wasn't as important as it now has become.
Q. Were you asked to look for relevant documents?
A. Yes.
Q. And what effort did you make to find them?
A. [kept all my Post Office sort of related paperwork in a box file and that's -- when
1 was asked to look for evidence I went strictly to that box file and that's where I
sourced all my information from.
Q. Can we look at your second witness statement {E1/11/2} in paragraph 6. Your
evidence there is you hadn't looked at this previously: "... as I did not think I would
be able to access the forum group and it did not seem relevant." Can you just explain
how on earth you could conclude that it wouldn't seem relevant?
A. Sorry, can you ask the question again.
Q. How did you reach the conclusion that the forum posts would not be relevant?
A. I didn't reach that conclusion.
387
AI8/387
POL00026925
POL00026925
Q. Well, you say in paragraph 6, Mr Tank: "... I did not think I would be able to
access the forum group and it did not seem relevant." I'm just wondering how you
reached that conclusion?
A. Because when I made my first initial witness statement I wasn't aware of £195.04
loss, that information only came to light after reading Ms van den Bogerd's
statement.°°
1190. Later in cross examination, Mr Tank also explained:
"Q. Now, why were you able to say that it definitely occurred in those dates?
A. Because when I went to the box file that I mentioned earlier there would have been
a handwritten note somewhere in amongst that that would refer to that incident and
that's what led me to believe that that was the particular date.""*°
1191. Mr Tank continued:
"A. I couldn't say for sure. As I tried to explain earlier, in drafting my original
witness statement I went back to my box file and it was filled with various bits of
paper, different sizes, different formats and with handwritten scribbled notes on and
that's where I tried to piece together the information that I provided in this.""4
1192. Further:
"Q. And again can you help with how that error came to be made?
A, Again, my initial witness statement I was just relying on my memory and my
supplemental witness statement is when I was able to research it a bit more.
Q. And what was the evidence that you got for your second witness statement that
helped you on this date, because I'm not sure that I'm aware of any?
A. Again, it was Horizon generated receipts, print-outs, with hand-written dates and
reference numbers on them.
Q. Sorry, where are these documents?
A, With my solicitors.
Q. Oh. I don't think they have made their way over, but I might be wrong.""4#!
438 (Day 2/104:11} to {Day2/106:16}.
M4 (Day2/108:13}.
M40 (Day2/117:22} to {DayWA8S+!
M441 (Day2/146:8} to {Day2/146:21}.
388,
A/8/388
1193,
1194.
1195.
POL00026925
POL00026925
As a result of locating these documents, Mr Tank served a supplemental witness
9.4” Disclosure of these documents should have been
statement on 27 February 201
provided by Cs on 17 July 2018. As explained in WBD's letter of 5 March 2019 no reason
was provided by Cs as to why Mr Tank did not try and access the Yahoo forum to locate
relevant documents which would be relied upon at trial nor why it was not brought to
Post Office's attention that there was a document source which contained relevant (and
potentially adverse documents) which the Cs could not access.'4*
Mr Latif
In relation to Mr Latif's disclosure, Mr Latif served an amended witness statement on 1
March 2019.'*# This amendment is understood to have been made following enquires
with Mr Latif’s former branch manager, who consulted the documents within the branch
to confirm the date on which the transaction corrections had been issued.'“** Mr Latif
did not provide any disclosure of these documents for the Horizon Issues Trial.
During cross examination of Mr Latif on Day 2 of the Horizon Issues Trial, Mr Latif
explained:
"Q. First question about that: by amendment to your statement you now say that this
was in around January 2018 rather than March 2018.
A. Correct.
Q. What caused you to make that correction?
A. Thad a look at the -- we hold the records for the information in the office, so I had
my assistants look at the records, transaction logs and that's when I confirmed that it
was January rather than March. (Inaudible) was logs in March as well. We made
calls effectively every month to Horizon help desk concerning this issue.
Q. When do you say you asked your assistants to check about the date?
A. Yes.
Q. Sorry, when do you say that happened?
A. It was after I made the initial statement, I was checking.
Me ELI}
M488 11/235}
MM ELL}.
M45 (Day2/67:16}.
389
A/8/389
POL00026925
POL00026925
Q. Roughly when, Mr Latif?
A. It would have been a few weeks ago, sir:
Q. So is it right that you didn't check those records from the branch before making
your witness statement?
A. No, I thought I was correct but I double checked and made sure that actually in
fact they were correct, those (inaudible), so I was right but initial incident happened
in January when TA (inaudible) transaction acknowledgement, the TC, the
corrections, they came in March.
Q. Okay. So you say, do you, that the TCs in relation to these transaction
acknowledgements came in March 2018?
A. Yes.
Q. And you say that's something you have checked from your records?
A. Yes, 1446
Requests for Disclosure
1196. In light of the above cross examination of Mr Latif and Mr Tank, Post Office wrote to
Cs on 15 March 2019 requesting disclosure of:
1196.1 Records and transaction logs consulted by Mr Latif for the purposes of drafting
his witness statement;
1196.2 Mr Tank's handwritten notes, documents in the "box file" and the CCTV
recordings which were consulted by Mr Tank for the purposes of drafting his
witness statement;
1196.3 Any other documents consulted by Mr Latifand Mr Tank in the course of drafting
their witness statements; and
M446 (Day2/67:11} to {Day2/68:19}.
390
A/8/390
1197.
1198,
1199.
1200.
1196.4 Any other documents which should have been disclosed by the Claimants
pursuant to paragraph 3 of the Fourth CMC Order or due to being adverse
documents.'447
On Days 6 and 7 of the Horizon Issues Trial, Mr de Garr Robinson QC also brought the
non-disclosure of these documents to the attention of the Court, and noted that a response
from Cs to Post Office’s letter was still awaited. '**
Cs responded to these requests on 27 March 2019 refusing to provide disclosure of the
requested documents on the basis that they were cither outside of Cs’ control, no longer
existed or were not relevant the Cs evidence and not relied upon at trial."
Post Office responded on 14 May 2019 offering to assist with gaining access to the
documents which were held in Mr Latif's former branch (and therefore outside of his
control despite his former branch manager accessing them for the purposes of witness
evidence) and repeating its request for disclosure. Post Office also asked that these
requests be treated as Model C requests for disclosure.'** Post Office chased for a
response on 23 May 2019.'*°! Cs responded on 29 May 2019 refusing to provide
disclosure of the documents sought:'* in relation to Mr Latif, Cs said there were no
documents to disclose and in relation to Mr Tank Cs stated that they had considered the
documents in his “box file” and concluded that they were not “known adverse
documents”.
Cs are showing double standards in their attitudes. In contrast to Post Office’s approach,
Cs have taken a narrow and restrictive view of their disclosure obligations. At the same
time they direct bitter criticism towards Post Office who have adopted a much more co-
operative approach.
M7 119/242.6/3}.
MIS (Day6/4:2} to {Day6/4:23}; {Day7/1:6} to {Day7/2:10}.
M89 117/255}.
6450 £11/280}
MSI 41/297}.
M52 11/303}.
391
POL00026925
POL00026925
A/8/391
POL00026925
POL00026925
MISCELLANEOUS
1201.
1202.
1203.
1204.
1205.
1206.
1207.
“Shadow experts”
In its Opening Submissions, Cs made a number of points about so-called “shadow
experts” which are improper and misleading. Due to time constraints, Post Office did
not deal with these in oral opening but indicated that these points would be covered in
closing.
In dealing with these points, Post Office does not in any way waive privilege.
It is commonplace in litigation of this scale that independent advisers are retained to
assist parties. The mere fact of the retainer should be no criticism
As a preliminary point, the suggestion that the advisers have been used in order to shield
Dr Worden from evidence is an allegation which should not have been made: it is
baseless, improper and should be withdrawn.
In fact, advisers were not retained as expert witnesses but as an investigation team. Their
principal role was to assist with the pleading of the defence and most of their work
occurred before this point.
Indeed, apart from a handful of small follow up questions, their work was concluded
before the Second CMC on 2 February 2018 — i.e. the first time the Horizon Issues Trial
was canvassed. They had no role in scoping disclosure or in reviewing documents to be
disclosed or in making decisions on documents to be disclosed
As part of their investigation they provided a list of documents they had reviewed. That
list is privileged but all documents on that list have been disclosed.
ANTHONY DE GARR ROBINSON QC
SIMON HENDERSON
OWAIN DRAPER
AI8/392
REBECCA KEATING
One Essex Court and 4 Pump Court
27 June 2019
K. APPENDICES
POL00026925
POL00026925
Kl.
APPENDIX 1: POST OFFICE ANALYSIS OF MR COYNE’S
ADDITIONAL “COST/BENEFIT” PEAKS
1. PC00S5804:145
Ll
1.2
1.3
This Peak relates to an issue with APSC2053 "abending" (crashing) resulting in
there being no APS reports for three days in respect of branch FAD 3746429. The
Peak notes that "/t/his problem has been identified as real and still needs fixing".
As a temporary fix for the issue, an OCP (2771) was applied to the live system. At
that point, the Peak cited by Mr Coyne was cloned from PC0055265 for "detailed
analysis at a later date", noting that the "OF P forum has assessed this as having
a low business impact".
A permanent fix was scheduled to be issued in the next release (M1), however,
upon investigation, the problem was found to be "bigger than expected". The Peak
notes that "/t/his fix is needed to make the system clean, but is not urgent for M1
as database contraints [sic] have been relaxed", referring to the temporary fix
applied by OCP 2771.
The Peak notes that the permanent fix "is also going to be tricky to implement and
will take time and care" and so is delayed until after M1. Ultimately, the Peak
concludes that "at best it would not be cost effective and could be counter-
productive to analysecfurther [sic] or attempt to fix this call without
M53 (F/78.1}.
393,
AI8/393
demonstration that it still occurs in the system currently under test or in Live.
Moving to Awaiting KEL in preparation for closure".
The clear inference from the Peak is that the fix being potentially not "cost-
effective" is not the primary reason for not implementing it. It appears that the fix
would be complex and potentially “counter-productive", and may not be necessary
in light ofa lack of evidence that the issue can still occur in a test/live environment.
In any event, this issue would not have had an impact on branch accounts.
2. PC0133933:'454
2.1
2.2
2.3
2.4
This Peak relates to a branch (FAD 101801) whose Harvester Exception Report
(TPSC254) showed an error message reflecting a new harvester exception as a
result of Riposte messages containing null mode attributes. On the day that the
Peak was raised, the specific problem transaction was repaired.
The issue is described as “a well known problem and does not just affect Mails
messages" and it is initially suggested that the call should be passed to Escher to
develop a fix on the basis that the issue may be caused by a bug in its code.
Ultimately, it was concluded that the Peak could be closed on the basis that an
adequate workaround was already in place. Further, without a reproducible
scenario, it was thought unsuitable to pass the investigation of the root cause to
Escher. For these reasons "and the limited life expectancy of the Horizon counter
it was agreed that [attempting to fix the underlying cause] would not be cost
effective and hence this PEAK should be closed". Anne Chambers, who originally
investigated the issue, later comments that "J never really expected the root cause
to be investigated or fixed. The typo which caused the agent circumvention to fail
was fixed a long time ago. Closing call".
Whilst cost-effectiveness is mentioned, the primary reason for not implementing
a fix in relation to this issue therefore appears to be that an adequate workaround
POL00026925
POL00026925
M4 (F/338}.
394
AI8/394
was already in place. In any event, this issue would not have had an impact on
branch accounts.
3. PC0136622:'45
3.1 This Peak relates to an issue raised by System Testing relating to a Geller PC
attempting to start Avery D104 scales upon reboot and raising a "red event" when
it cannot do so (there are no scales connected to a Geller PC).
3.2 The Peak notes that "/¢/he event causes no problems operationally and it has been
KEL'd in KSaunders5754S but it would be nice to have a fix for the next Geller
build if possible" and, later, that "/a]t worst we may just have to live with the
event".
3.3 A fix was applied which addressed the problems caused by the issue, however, the
Peak notes that, "/s/trictly speaking, the root cause has not been fixed but because
of the architecture of this platform, it would probably not be cost effective to do
so. The problems caused by this have however been resolves [sic] so call can be
closed".
3.4 Cost-effectiveness is considered as part of the ultimate decision to not implement
a fix. However, this is in the context of an issue which has no operational impact
and whose associated problems have been resolved. In any event, this issue would
not have had an impact on branch accounts.
4. PC0227613:'4¢
4.1 This Peak relates to an HMS/TMS issue resulting in card payment authorisation
requests being timed out for a 25-minute period. The Peak notes that, as a result,
"{t]ens of thousands of customers unable to pay by card at branches for over 30
minutes. Two known occurrences in 15 months. If the DCS agent handled the
situation better, the duration of the outage may have been significantly reduced
and the issue might have been noticed more quickly".
POL00026925
POL00026925
655. (F/345,1}.
M56 FF/I115.1}.
395,
AI8/395
POL00026925
POL00026925
4.2 The Peak notes that "/t/hrough the changes we deployed we mitigated this issue
to the extent that we will not suffer from it again". Two options were being
considered in order to permanently fix the issue. The decision not to implement a
permanent fix is explained in two entries in the Peak:
(a)
(b)
()
(d)
"Tam guessing we would need a business case and many more inputs than
we have today, the forum would need to be extended initially to involve the
solution owner and then commercial people. Ultimately we would need
enough justification to get this on the road map, not knowing the costs it
would be difficult to say how the account would like to progress option 2 if
indeed it got that far the first step for me would be an internal feasibility
study to identify whether or not this would be worthwhile in the current
landscape as we don’t have all the information to answer that at present"
"We did briefly pursue this with both architecture and senior service mgmt
at the time, I recall a brief discussion with Alex who informed us that the
current x25 circuits are either already paid for until the end of the current
contract or contractual agreements are in place to that effect and therefore
circuit cost reduction savings would not be recognized. This combined with
where we are today with the customer, (Towers Model) and service extension
projects throughout next year to me means we have less reasons today for
pursuing this than we did at the time of the high profile fault when the
decision then was to not pursue migrating this service onto TCP/IP.
I would suggest closing the PEAK down as the issue has been mitigated to
the extent that it will no longer impact service. Whether or not we ever end
up migrating DCS onto an e2e TCP/P service can remain as a potential
option for us at any point in time whilst we are still an active player in this
service, we don?t really need a PEAK for that but it does give us justification
if we ever did go down that path.
Included Alex/Steve just in case they disagree with any of the above, I may
have quoted Alex incorrectly about the X25 circuits as it was a long time
ago but I am pretty sure even if I have the net result is the same, no cost
benefit"
Note that the reference in b) to "cost reduction savings" and “no cost benefit"
appears to be a reference to the fact that the preferred permanent fix was
initially thought to have the collateral benefit of reducing costs. It was later
determined that this was not the case. This particular quote is therefore not
an example of a fix not being implemented because of the associated costs.
The Peak demonstrates that a number of factors influenced the decision not
to implement a permanent fix; primarily, the issue had been "mitigated to
the extent that it will no longer impact service". Whilst the potential costs of
396
AI8/396
a permanent fix are referred to, they are not considered in any detail. In any
event, this was an issue affecting the back office and would not have had an
impact on branch accounts.
5. PC0083308:'4°7
5.1 This Peak relates to an issue in which the ACDB and OCMS audit log switch fails.
This Peak is a clone of PC0081847 (POL-0255113 —not in the trial bundle). The
Peak (which was assigned to Richard Roll) notes that the issue is "not a High
priority and can be targeted at BI3".
5.2. Aworkaround was developed whilst a "full solution" was being considered. At this
point, the Peak referred to by Mr Coyne is cloned from PC0081847 to investigate
the full solution.
5.3. The Peak concludes that "/i/t has been agreed that there is insufficient cost benefit
to justify this development, which affects several platforms. The Maestro
workaround described above has fixed the immediate problem".
5.4 Whilst "cost benefit" is considered, the primary reason for the decision not to
implement the permanent fix is that an adequate workaround is already in place.
In any event, that this issue affected the back office and would not have had an
impact on branch accounts.
6. PCO145617:'5
6.1 This Peak relates to an issue at branch FAD 468519 in which the system freezes
during a transaction. The issue was caused by a combination of intermittent comms.
issues on a dial-on-demand network together with a Riposte timer issue.
6.2 The Peak notes that "/i/nitial investigation has shown that the problem is affecting
10-12 Post Offices, who are getting the symptoms 2-3 times per week".
POL00026925
POL00026925
M7 £F/123.21}.
M88 (F/414.1}.
397
AI8/397
6.3 The Peak was cloned from PC0145212 (POL-0315551 — this is not in the trial
bundle). PC0145617 considers the Riposte timer issue; PC0145212 considers the
comms issue, PCO145212 results in a fix being implemented for the comms issue.
6.4 A workaround for the PC0145617 issue was developed in spite of the associated
cost of using it, since it is concluded that "it would be more cost-effective than the
alternatives". Various factors are considered in the decision to rely on the
workaround without implementing a permanent fix: "this problem is not readily
reproducible and any potential (speculative) fix is likely to be fairly invasive" and
"significant testing would be required". It is thought unlikely that Escher would
develop a fix and even if they did, it was thought very unlikely that it would be
implemented,
6.5 Apossible alternative is described as "brutal", clumsy, lacking in recovery actions,
"more likely to cause problems than to solve them" and having "possible side
effects”.
6.6 The Peak does mention that there will need to be a discussion of the "cost-benefit
and likely timing of making a NB Framework change before committing more
effort to investigating potential fixes". However, the decision not to develop a
permanent fix is primarily based upon the lack of viable options (as set out above)
together with the adequacy of the workaround.
6.7 Inany event, this issue would not have impacted branch accounts.
7. KEL LKiang4936R:'4°°
7.1 This KEL relates to a reporting issue in which the "host figure" was missing a
£36.79 reversal but the counter figures were correct (branch FAD 163925). The
KEL stems from PC0100350 (F/198.1), which provides slightly more detail than
the KEL.
7.2 The Peak notes that no reconciliation was required for the branch in question, since
it was a reporting issue only.
POL00026925
POL00026925
499 (F/198.2}.
398
AI8/398
7.3
Ultimately, it is concluded in the Peak that "/s/ince the problem is now over a year
old and module TPSC268 will be discontinued in approximately 2 to 3 months time
(when TPS migrates to S80) I agree that it will not be cost effective to fix this
problem". This is reflected in the KEL. Therefore, whilst cost-effectiveness is
mentioned, ultimately, the primary reason for not implementing a fix is that the
issue would be resolved upon the migration of TPS anyway. In any event, this
issue affected the back office and would not have had an impact on branch
accounts.
399
POL00026925
POL00026925
A/8/399
POL00026925
POL00026925
K2. APPENDIX 2: THE 29 BUGS IN JS2
The Bug Appendix
1. This Appendix addresses the 29 bugs alleged in the JS2 Bug Table!*® in the order in
which they appear in that Bug Table. It should be read in conjunction with Section F) 2
29 of Post Office’s Closing Submissions.
2. Hundreds of Peaks and KELs are referred to in JS2. It is not possible to address each and
every Peak and KEL. This Appendix considers the main Peaks and KELs relevant to
each alleged bug.
3. Of the 29 bugs, the relevant Peaks and KELs demonstrate that:
3.1 Eight are not bugs at all.'°!
3.2. Three had no branch impact.'4?
4.1463
3.3 Nine had or potentially had only transient impac
3.4 Nine caused or had the potential to cause lasting impact, but were resolved by Post
Office and Fujitsu.!4*
4. The following alleged bugs are not bugs at all:
4.1 Bug 8: Recovery Issues.
4.2 Bug 13: Withdrawn Stock Discrepancies.
4460 (1)1/2/3} to {D1/2/26}.
461 Bug 8: Recovery Issues; Bug 13: Withdrawn Stock Discrepancies; Bug 15: Phantom Transactions; Bug
16: Reconciliation Issues; Bug 17: Branch Customer Discrepancies and Bug 20: Recovery Failures; Bug 23:
Bureau de Change and Bug 29: Network Banking.
M@ Bug 11: Girobank; Bug 21: Transaction Corrections and Bug 22: Bugs/Errors Defects introduced by
previously applied Peak Fixes.
M® Bug 2: Calendar Square; Bug 4: Dalmellington; Bug 5: Remming In; Bug 6: Remming Out; Bug 7:
Local Suspense; Bug 9: Reversals; Bug 12: Counter Replacement; Bug 19: Post & Go/TA Discrepancies and
Bug 25: Lyca top up.
46! Bug 1: Receipts and Payments Mismatch; Bug 3: Suspense Account; Bug 10: Data Tree Build; Bug 14:
Bureau Discrepancies; Bug 18: Concurrent Logins; Bug 24: Wrong branch customer; Bug 26: TPSC250;
Bug 27: TPS and Bug 28: Drop and Go.
400
A/8/400
4.3. Bug 15: Phantom Transactions.
4.4 Bug 16: Reconciliation Issues.
4.5 Bug 17: Branch Customer Discrepancies.
4.6 Bug 20: Recovery Failures.
4.7 Bug 29: Network Banking.
4.8 Bug 23: Bureau de Change.
The following bugs had no branch impact:
5.1 Bug 11: Girobank.
5.2 Bug 21: Transaction Corrections.
5.3. Bug 22: Bugs/Errors Defects introduced by previously applied Peak Fixes.
The following bugs had transient impact:
6.1 Bug 2: Callendar Square.
6.2 Bug 4: Dalmellington.
6.3 Bug 5: Remming In.
6.4 Bug 6: Remming Out.
6.5 Bug 7: Local Suspense.
6.6 Bug 9: Reversals.
6.7 Bug 12: Counter Replacement.
6.8 Bug 19: Post & Go/TA Discrepancies.
6.9 Bug 25: Lyca top up.
The following are bugs with lasting impact (although they were resolved):
7.1 Bug 1: Receipts and Payments Mismatch.
7.2 Bug 3: Suspense Account.
401
POL00026925
POL00026925
A/8/401
7.3
74
75
7.6
77
78
79
Bug 10: Data Tree Build.
Bug 14: Bureau Discrepancies.
Bug 18: Concurrent Logins.
Bug 24: Wrong branch customer.
Bug 26: TPSC250.
Bug 27: TPS.
Bug 28: Drop and Go.
Bug 1: Receipts & Payments Mismatch
1. The key documents:
Ll
1.3
Summary:
Peaks: PC0204765;'4° — PC0204263;' —PC0203864;'4°7
PC0204889;'4 and PC0205076.'4”
Coyne 2, paras 3.27 — 3.33.'47!
Js2.'4?
PC0204537;'4°8
2. Post Office accepts that Bug 1: Receipts and Payments Mismatch had a potentially
lasting financial impact. In the event, however, this bug resulted in transient impact only.
POL00026925
POL00026925
MOS {F/718}.
M466 {F/709}.
M67 {F/705}.
M66 {F/714}.
MO {F/721}.
M19 (F/726}.
471 ()2/4,1/19} to {D2/4.1/22}.
472 ()1/2/3}.
402
Al8/402
POL00026925
POL00026925
Nature of the issue
3. This b
Tt was
“lost”
4. KEL
ug was acknowledged by Post Office in its Letter of Response on 28 July 2016.'4°
investigated by Fujitsu, which produced a note entitled ‘Correcting Accounts for
discrepancies’ (the “Lost Discrepancies Note”).!474
wrightm33145J'4 shows that the issue related to the process of moving
discrepancies into a branch’s local suspense account. It occurred following an unusual
sequence of events, namely:
4.1
4.2
43
44
45
46
47
The branch had an unresolved discrepancy.
The SPM, during the balancing of a stock unit, pressed the Preview or Print button
to produce the Trial Balance Report, causing the counter to return to the rollover
screen.
Having checked the Trial Balance Report, the SPM then pressed the rollover
button.
Due to the unresolved discrepancy, Horizon would then ask the SPM whether they
would like to:
(a) _ transfer the relevant discrepancy to the branch’s local suspense account; or
(b) cancel the rollover.
Instead of transferring the discrepancy to local suspense, the SPM chose to ‘cancel’
the rollover.
This would cause the system to return to the rollover screen, allowing the SPM
various choices, including cancelling the entire the rollover process or the choice
of again pressing the Preview or Print button to produce the Trial Balance Report.
The SPM then proceeded to rollover to a new Trading Period in the same session.
473 (H/2/97}.
414 £F/1000}.
MTS £F/1450}.
403
A/8/403
4.8 Ifall those things happened, a bug in the code was triggered by the ‘cancel’ button
being pressed and this incorrectly caused the discrepancy to be cleared.
Following the above steps, the SPM was able to rollover with an unresolved discrepancy.
Detection and resolution
5.
Peak PC0204765 (25 September 2010) is cited in JS2.!4”° The Peak indicates that
multiple branches had been affected by this issue (the “Payments Mismatch” issue) and
it refers to several earlier Peaks.
The Peak records that the issue was being detected and monitored by Fujitsu and that all
instances were being reported to Post Office’s Duty Manager, together with the relevant
figures and reports. It also records that Post Office was to make contact with the relevant
SPM regarding any required remedial action.'4””
The Lost Discrepancies Note provides details about the processes Fujitsu followed to
identify those branches that had been affected by the bug and ensure that monitoring
continued to enable Fujitsu to identify any future instances,’ including raising Peaks
for each occurrence of the issue and running a monthly check.'*”
Fujitsu prepared three spreadsheets showing those branches that had been affected by
the bug as follows:
8.1 Interim spreadsheet dated 21 October 2010.'48°
8.2 Spreadsheet dated 9 December 2010 confirming there were 62 individual branches
affected between 31.03.2010 and 20.10.2010, with 2 branches being affected twice
—i.e. there were 64 different instances and 62 affected branches. '4*!
8.3. Final spreadsheet in December 2010.'*?
POL00026925
POL00026925
1476 (F/718}.
M77 ¢F/718/7}.
M78 ¢F/1000/2} to {F/1000/3}.
47 ¢F/1000/3}.
M480 F/736.1}.
MSI (F/754.1}.
M82 1F/754.2}.
404
AI8/404
POL00026925
POL00026925
As can be seen from the ‘issue notes’ prepared by Fujitsu,'4** Post Office elected to
resolve any discrepancies with the affected branches via TCs. There was one exception
to this: in the occurrence recorded in Peak PC0204765, Fujitsu injected a transaction into
the branch’s accounts (as explained in Post Office’s Letter of Response). 48+
Mr Coyne suggested that the true extent of the bug is not fully confirmed. There was no
time to cross examine him about this, but it is worth noting that he did not refer to the
steps taken by Fujitsu to identify those branches affected by the bug or to the fact that
PC0204765 confirmed that Post Office would in due course make contact with the
relevant SPM regarding any required remedial action.'4*
As can be seen from PC0204263,'“** Fujitsu implemented a reference data fix to correct
the issue in the code that was causing the discrepancies to become ‘lost’.
Relevant discussions during trial
12.
During Dr Worden’s cross-examination, Mr Green QC referred Dr Worden to the
Payments Mismatch ‘issue notes’,'“*’ specifically the following extract:
“Note the Branch will not get a prompt from the system to say there is Receipts and
Payments mismatch, therefore the branch will believe they have balanced
correctly” 485
Mr Green QC noted that, in relation to the Payments Mismatch issue:
“That is not the system alerting the Subpostmaster or subpostmistress to what’s going
on is i127"
In response, Dr Worden confirmed:
“I agree that is different from other cases where I think receipts and payments
mismatch shows in a trial balance or something that the postmaster sees”**°
48 (F/1001/3}.
M84 {11/2/25}.
M85 (F/718/7},
486 (F/709}.
487 (F/1001/3}.
M88 (Dayl 9/1352)
} to (Day19/136:1}.
48 (Day19/136:6} to {Day19/136:8}.
1490 (Dayl9/136:9} to {Dayl9/136:11}.
405
AI8/405
As indicated above, the Payments Mismatch issue only occurred following an unusual
sequence of events during the process of moving discrepancies into the local suspense
account. It did not generate a receipts and payments mismatch alert when the branch
balanced at the end of its trading period, but Fujitsu was able to identify instances of the
bug (and indeed identified the existence of the bug) and it would have been visible to the
SPM via their Final Balance Report (see KEL wrightm33145J)'*!.
Conclusion
14.
Ordinarily Horizon alerts a SPM to a receipts and payments mismatch when they balance
at the end of a trading period, but that did not happen here for the reasons given above.
However, as can be seen from KEL wrightm33145J, the unresolved discrepancy would
have been visible to the SPM if they checked their Final Balance Report. Instances of
the bug were picked up and carefully monitored thereafter. All previous instances
identified and were made good.
Bug 2: Callendar Square
1. The key documents:
1.1 Peaks: PCO126042'*”; and PCO126376,'"°
1.2 Coyne 2, paras 3.34 — 3.42.14
1.3 Js2.\4°5
Summary
2. I Mr Coyne asserts that Bug 2: Callendar Square is a bug with lasting financial impact and
in JS2, Dr Worden appears to agree that there is strong evidence of this, but the
underlying documents show that the bug was identified early and Fujitsu dealt with
1491 1F/1450}.
1492 1F/297}.
1493 (F/298}.
144 11)2/4.1/22} to {D2/4.1/25}.
495 (1)1/2/3}.
406
POL00026925
POL00026925
AI8/406
relevant instances as and when they arose. They make it clear that the bug resulted in
transient impact only.
Nature of the issue
3.
The Callendar Square bug was acknowledged by Post Office in its Letter of Response
on 28 July 2016.'" It concerned the process in Legacy Horizon of transferring cash
between different stock units (“SUs”) in a branch"? which involved the following two
transactions:
3.1 a ‘Transfer Out’ transaction for one SU, which resulted in a message containing
the relevant details about the transfer being created and replicated to all counters
in the branch;'*°* and
3.2 acorresponding ‘Transfer In’ transaction for the ‘receiving SU’, which used details
from the message to complete the transfer.'4?
There was a bug in the Riposte software (used to replicate data between counters in the
branch) which meant that, when the SPM completed the Transfer In on one counter:
4.1 The message indicating that the SPM had completed the Transfer In was not
replicated to all the counters in the branch.
4.2. The SPM was accordingly able to duplicate the Transfer In on another counter
in the branch.
4.3 Asa result of the above, the counter on which the SPM completed the original
Transfer In would become temporarily unusable until it was restarted.
The Riposte bug had several manifestations. The Callendar Square manifestation only
occurred in unusual circumstances. Dr Worden termed this an “event storm”.'°°°
POL00026925
POL00026925
1496
4497 See Godeseth 2, para 13.2 {E2/7/4} for an explanation of why it would have been necessary to transfer
cash between different stock units in a branch.
M98 (E2/7/4}.
499 (E2/7/4}.
1500 (Dayl9/172:1} to {Dayl7/172:2}.
407
A/8/407
POL00026925
POL00026925
Detection and resolution
6.
Fujitsu was aware of the wider Ripsote issue for quite some time. Its various
manifestations were being identified and resolved long before the Callendar Square Peak
was raised. PC0126042,'*"! the Callendar Square Peak, arose when the SPM at the
Callendar Square branch reported that he could see a transfer on certain counters in the
branch, but not on counter 3, and requested that this be investigated.
As is shown in the BIMS Incident Report BE/0126042'5” for the Callendar Square
branch:
7.1 the “Riposte errors” allowed the SPM to complete three different Transfers In
twice, resulting in a loss of £3,489.69;
7.2 the loss of £3,489.69 would need to be corrected by Post Office via an error notice;
and
7.3 Fujitsu telephoned the SPM to explain what the problem was and advised the SPM
to contact the NBSC regarding any further issues.
Mr Coyne suggests that it is “wnclear how this bug was resolved”.'5° But as is shown in
PC0126376,'* the Riposte bug was fixed as part of the Horizon software update s90
released in 2006.
Mr Coyne notes from the documents that the Callendar Square bug was “operating and
resident in the system for years without any comprehensive linkage being observed by
Fujitsu”. However, this provides only an incomplete picture, as Post Office would have
demonstrated if it had had the time it would have need to cross examine him on this bug
(that time was taken by Mr Coyne’s extraordinary changes in evidence on the afternoon
of Day 17, the final afternoon of his cross examination). Two further points should also
be noted from the documents:
1501 (F/287}.
1502 (F/653.1}
1508 ()2/4.1/24} to (D2/4.1/25}.
1504
2208 S268,
408
A/8/408
10.
POL00026925
POL00026925
9.1 The first relates to those instances where a duplicate ‘Transfer In’ had incorrectly
taken place. These occurrences resulted in a receipts and payments mismatch at
the branch, which were in turn detected by Fujitsu using the following reports:
(a) The Host Detected Cash Account Lines Comparison Report TPSC256,
which is run on a daily basis and shows those branches where the net total
of the transactions in branch, i.e. the value of credits less debits, does not net
to the value of zero, as expected.'*°
(b) IThe TPSC268 Host Cash Account Reconciliation Report which shows the
discrepancies between the values at the Counter and the TPS-Host.'5°°
These reports are discussed in Appendix 3.'%°” As can be seen from Coyne 1,'°°8
Mr Coyne knew about the TPS Report set.!*
9.2 Second, as shown in JBallantyne5245K,'*!° a temporary solution was in place
between 2000 and 2006 whereby the SMC would monitor for relevant events and,
where an event was spotted, they would contact the SPM and advise them to restart
the counter, which would typically resolve the issue.
Finally, once Fujitsu had identified that there was an issue with a particular branch, either
via one of the receipts and payments mismatch reports or as a result of Fujitsu’s
monitoring of events or as a result of a call from an SPM, a Peak would be raised and
the SPM made good in the ordinary course.!*"! It was open to Cs to search for all the
relevant Peaks in the 220,000+ Peaks that have been disclosed, but most of them were
identified by Anne Chambers when she was asked to do so nine years after the event, in
2015. These are set out in the spreadsheet ‘Branches affected by Riposte lock
problem’,'*!? which indicates that she had identified 30 instances of the wider Riposte
1505,
1506
1507
1508
1509
1510
isl
112
{F/896/67}.
{F/100.2/45} and JC2 at {D2/4.1/58}.
See Appendix 3
{D2/1/201}.
{Day15/64:1} to (Day15/64:12} and {Day15/64:13} to {Day15/65:3}.
{F/565/3}.
{F/224.1}; (F/253} and (F/210}.
{F/322.1}.
409
A/8/409
since the launch of Horizon, including 19 instances of the mismatch issue seen at
Callendar Square.
Relevant discussions during trial
i.
12.
Dr Worden described how a receipts and payments mismatch would be “a pretty
prominent red flag to Fujitsu” during cross-examination:
A. Well, we always have this issue of if there is a discrepancy, how obvious is it and
therefore how much can one infer about whether it is likely to have been corrected or
not? And receipts/payment mismatch, which I think this eventually led to, is a pretty
prominent red flag to Fujitsu. And so a lot of receipts/payment mismatches I would
expect -- you see them in PEAKs when they occur and you see the amount involved,
so I would expect on a high proportion of cases generally R/P mismatches would be
sorted out?!
Dr Worden described how a receipts and payments mismatch would be “a pretty
prominent red flag to Fujitsu” during cross-examination:
A. Well, we always have this issue of if there is a discrepancy, how obvious is it and
therefore how much can one infer about whether it is likely to have been corrected or
not? And receipts/payment mismatch, which I think this eventually led to, is a pretty
prominent red flag to Fujitsu. And so a lot of receipts/payment mismatches I would
expect -- you see them in PEAKs when they occur and you see the amount involved,
so I would expect on a high proportion of cases generally R/P mismatches would be
sorted out.!°!*
Dr Worden was taken through the first Callendar Square Peak,'*'*but was not referred to
earlier Peaks which addressed the underlying Riposte issue. In re-examination, Dr
Worden was therefore asked about a number of Peaks which arose prior to the Callendar
Square Peak and which demonstrate that the Riposte issue was not lurking in Horizon
undetected. He was referred to:'*!°
1518 (Dayl9/133:13} to {Day19/133:22}.
SM (Day19/133:13} to {Day19/133:22}.
515 (F/3 12.1}.
1516 (Day20/175:1} to {Day20/179:1}.
410
POL00026925
POL00026925
AI8/410
13.1
PC0109305 (8 October 2004)'*!” which was raised in response to the affected
branch appearing in the following automatic reports: TPSC256 and TPSC268
reports (both explained in Appendix 3).!5!
PC0116670 (24 February 2005)'*!? which was raised in response to the affected
branch appearing in the following automatic reports: TPSC256, TPSC252 and
TPSC268.
PC0103864 (3 June 2004)'*2° which was raised in response to the SPM raising an
issue with the NBSC, however, as per the entry on 8 June 2004 at 14:48, the branch
also appeared in Fujitsu’s TPSC252, TPSC256 and TPSC268 reports.'°7!
14. When asked about these documents, Dr Worden said:
14.1
14.2
14.3
PC0109305'>” showed that some of the instances of the Riposte issue:
.. were being detected by routine monitoring and, for instance I would also
suspect that any manifestation of the Riposte bug which led to a double stock
transfer in and a single stock transfer out, that is a double entry accounting
failure and that would certainly be picked up at some stage. So there are lots
of ways in which the phenomenon would be automatically detected or easily
detected at the back-end.'>*
PCO0116670'**4 showed that
several manifestations of the Riposte problem were easily detectable by Fujtisu
and they would be. They had systems set up to detect just this sort of thing.'°°>
The symptoms that we now refer to as Callendar Square involved a
POL00026925
POL00026925
1517 (F/224,1}.
1SI8 See Appendix 3.
1519 (F/253}.
1520 £F/210}.
1521 (F/210/3}.
1822 (F/224,1}.
828 (Day20/17
1524 (F/253}.
1525. (F/253}.
1} to (Day20/179:1}.
4i1
Al8/411
15.
POL00026925
POL00026925
double stock transfer in, single stock transfer in. That is a double entry
accounting mismatch failure and that will certainly be detected at the back end
when it gets to POLSAP or if not before.'°?°
14.4 PCO103864'°”’ showed that the actual Callendar Square symptoms would be
picked up by Fujitsu using the TPSC252, TPSC256 and TPSC268 reports.'*"* The
symptoms were:
picked up on three reports at the back-end. There is a lot of going back and
forth, but end of the back and forth is the BIMS and Post Office being alerted
and the branch being sorted out®°
1530 confirms that
Dr Worden’s analysis, which is clearly supported by the three Peaks,
Fujitsu’s automatic reporting was robust in terms of alerting Fujitsu to both the wider
Riposte issues and the particular Callendar Square issue and furthermore that Fujitsu
ensured that any exceptions in the relevant reports were investigated, so that any
financial impact on branch accounts would be resolved by Post Office.
Did not persist until 2010
16.
17.
It was suggested to Dr Worden during cross-examination that the Callendar Square issue
may have continued to 2010. This is incorrect. As Dr Worden pointed out, Callendar
Square arose following a number of specific steps occurring:
Q. Did you get a feel in your reading into this problem of how long this problem had
been around for and how many branches it was actually affecting?
A. Well, there is areal problem about the definition of what “this problem” is because
there is the Riposte bug and then there’s certain event storms which causes problems
in replication, which may be short-term problems for a few minutes, and then there
are event storms which make them long-term problems, and then there’s what
happens in event storms and in particular the double transfer in. So the whole of
Callendar Square is those things all happening together, whereas I think some of the
20 PEAKs or 30 PEAKs, whatever, are different from that, different combinations.’°*!
Dr Worden confirmed this again:
1526 (Day20/180:!}.
1527 (F/210}.
1528 (Day20/182:1}.
1S (Day20/18
1590 (F/224.1};
/253}; and {F/210}.
15M (Dayl9/169:15} to {Day19/170:4}.
AI8/412
18.
20.
21.
POL00026925
POL00026925
A. As I said, my view has always been that there is a chain of events and the Riposte
lock problem, which is at the source of the chain, happens much more frequently than
the whole chain.°*?
In response to the suggestion that the bug may have persisted until 2010 Dr Worden
confirmed that “this is on the message correspondence server and it is a different thing
from failure to replicate in the branch.”?>
Dr Worden expanded on these points in re-examination:
Q. You say another symptom of the Riposte problem. Perhaps you could explain a
little bit what you mean by that?
A, Well, I did explain and I will explain it again. That there is a problem in Riposte
which leads to a failure to replicate -- failures to release a lock I think -- and then on
certain occasions that leads to a short-term failure to replicate. On other occasions
there is a so-called event storm during which there is a longer term failure to replicate
and during these failures to replicate all sorts of different things may happen, for
instance they double transfer into a stock (inaudible), a precise Callendar Square
phenomenon, whereas other things such as system freezes, I can’t remember all the
other details, but there are many other symptoms of then underlying Riposte problem
and they have been noted over this whole period.!?>4
Finally, Dr Worden confirmed he was not aware of any later incidents of Callendar
Square.!°*
The only basis for any suggestion that the Callendar Square issue may have persisted
after the 2006 fix was the fact that a relevant KEL (KEL JSimpkins338Q)'**° was
updated in 2010 by the addition of a new Peak, PC0193012 (dated 9 January 2010).'5°7
However, even a brief perusal of that Peak is sufficient to demonstrate that the issue it
records is not the Callendar Square issue.
1592 (Day19/171:18} to {Day19/171:21}.
833. (Day 9/175:
1M (Day20/174:
to {Day19/175:8}.
} to {Day20/175:5}.
1535 (Day20/185:5} to {Day20/185:10}
1536 (F/565}.
137 {F/563}.
413
AI8/413
Bug 3: Suspense Account Bug
1. The key documents:
1.1 Peaks: PC0223870;'**§ PC0224126;'*° PC0198077;'* and Release Peak
PC0228641.'*4!
1.2 Coyne 2, paras 3.43 — 3.45.15
1.3 382,08
Summary
2. Post Office accepts that Bug 3:Suspense Account Bug had a potentially lasting financial
impact. In the event, however, this bug resulted in transient impact only.
Background: Local Suspense account
POL00026925
POL00026925
3. This issue relates to the Local Suspense account. If the SPM declares that there is a
discrepancy between:
3.1 the amount of cash and/ or stock in the branch and the amount of cash; and/ or
3.2 the amount of stock recorded on Horizon
the loss or gain is stored as a temporary accounting record in a separate part of Horizon
called the ‘Discrepancy Account’. This enables the SPM to remove the discrepancies
from the branch’s live Horizon records.'*4
4. Atthe end of the Trading Period, the figures in the Discrepancy Account must be cleared
before the branch can rollover into the next Trading Period. To do this, the SPM must
transfer the net value of all discrepancies recorded in the Discrepancy Account into the
1838 (F/1045}.
1839 F/1048.1}.
1840 (6/6303.
141 E/LL41.1}.
1882 (1)9/4,1/25} to {D2/4.1/26}.
158 (D)1/2/4}.
ISH 1/2/97}.
44
Al8/414
POL00026925
POL00026925
Local Suspense Account. The SPM is then required to resolve any shortfall or surplus
in the Local Suspense Account. Once the discrepancy has been resolved, the Local
Suspense Account will reset to zero and the branch will be able to rollover to the next
Trading Period.’*4°
Nature of this issue
5. This bug was acknowledged by Post Office in its Letter of Response on 28 July 2016.'%4°
The bug has been investigated separately by Fujitsu, who prepared a note for Post Office
titled ‘Local Suspense Problem’ which addressed this issue (the “Suspense Problem
Note”).'*"’ Additionally, Fujitsu identified the 14 affected branches.'*"
6. In terms of the specific issue that arose, as can be seen from Peak PC0223870'* and
the Suspense Problem Note:'°°°
6.1 I Changes were made to the archiving strategy relating to SUs on 3 July 2011. This
change inadvertently caused certain records in the Local Suspense Account that
should have been deleted to be left in the relevant table used for creating the
Branch Trading Statement.'*>!
6.2 As a result, the Local Suspense values that should have been deleted, later
appeared in the affected branches’ Branch Trading Statements when the same
Trading Period was next reached. Taking the Willen Village branch as an example,
this meant:
(a) 8 December 2010: The branch rolled over from Trading Period 9 to Trading
Period 10. The branch experienced a genuine loss for SU 1 of £9,799.88,'5°
1585 H/2/97}.
1846 11/2/97} to {1/2/98}.
1547 €F/1075}.
1548 {F/1075/3}.
19 ¢F/1045/4}.
18 F/1075/3} to {F/1075/4}.
1581 (F/1075/2}.
52 F/1075/4}.
4s
AI8/415
(b) 3 July 2011: The change to the archiving strategy happened (as explained
1.1583
above at para.
(c) 4 January 2012: The branch rolled over from Trading Period 9 to Trading
Period 10. The archived record, being the £9,799.88 loss, incorrectly
reappeared in the Branch Trading Statement. !5**
(d) 2 January 2013: The branch rolled over from Trading Period 9 to Trading
Period 10. The archived record, again being the £9,799.88 loss, incorrectly
reappeared in the Branch Trading Statement.
6.3 Asaresult of the historical entries incorrectly reappearing in the affected branches
Branch Trading Statement, SPMs re-settled entries in order to clear their Local
Suspense Accounts on two further occasions, despite those entries having been
previously settled.'5**
Detection and resolution
7. Asshown in the Release Peak PC0228641,'°%° Fujitsu produced a code fix in 2013. This
was first piloted to 50 branches on 21 October 2013'**” and deployed to all branches on
29 October 2013.15
8. In relation to the affected branches, PC0223870 indicates the lengths that Fujitsu went
to in order to investigate the issue and identify the 14 branches impacted by the bug. 5°?
The Peak records that there was a conference call between Fujitsu and Post Office on 28
February 2013 and that the spreadsheet showing the impact of the problem on the 14
branches was sent to Post Office by Steve Bansal of Fujitsu.'5°
POL00026925
POL00026925
1553 (F/1075/5}.
1554 (F/1075/5}.
S85 ¢F/1045/5}.
1896 F/114 1.1}.
1857 F/1141.1/4}.
S88 E/L4L1/4}
1589 F/1045/4} to {F/1045/5}.
150 ¢F/1045/6}.
416
AI8/416
POL00026925
POL00026925
9. Mr Coyne’s view is that this bug could have affected branches prior to Fujitsu’s
investigations and as a consequence suggests that it is unlikely that Post Office and/ or
Fujitsu have captured its full effects. However, as shown in the Peak:
9.1 Fujitsu identified that the historical data that was incorrectly retained related to
Autumn 2010.*! It was not possible for the bug to have affected branches prior
to this date.
9.2 Fujitsu was able to identify those affected branches from both the Branch Trading
Statements and the Suspense Account Reports.'°°
9.3. Events would be raised with Fujitsu moving forwards on branch rollover if:
(a) the next Trading Period's Opening Figures, generated for stock units,
included Local Suspense products that did not net to zero value; and
(b) the sum of the two Discrepancy Transferred lines on the Branch Trading
Statement, for the Branch Total, do not equal the sum of the two Discrepancy
Resolved lines.'°°*
This meant that Fujitsu would automatically be alerted to any reoccurrences of this
issue.
1561 (F/1045/5}.
15 (F/1045/5}.
1563 (F/1045/10}.
417
AI8/417
POL00026925
POL00026925
Bug 4: Dalmellington
1. The key documents:
1.1 Peaks: PC0246949;' PC0246997;'55 PCO247207;'5°° PC0247250'5°" and
Release Peak PC0248024.'5*
1.2. KEL: acha621P.'°
1.3. Coyne Supplemental Report 3.46 — 3.54.15
1.4 Js2,'87
Summary
2. This is a bug which Mr Coyne states has lasting impact on branch accounts. Post Office
submits that there was no lasting impact on branch accounts.
Background: outreach branches
3. The issue relates to outreach branches. These are usually small part-time branches, for
example, in a village hall or mobile van, and will be connected to a ‘core’ branch. While
Horizon treats outreach branches as being separate from their core branches, all money
is sent to and from the core branch, meaning it is the SPM’s responsibility to transfer
money between their respective core and outreach branches.
1564 {F/1303}
1565 ¢F/1 389.1}.
166 (F/1393}.
167 ¢F/1393.1}.
1568 F/1408.1}.
5 £F/1426}.
1570 (1)2/4.1/26} to {D2/4.1/28}.
1571 (D1/2/4} to (D1/2/5}.
418
Al8/418
Nature of the issue
4. Peak PC0246949'5” relates to a problem that arose when a core branch, being the
Dalmellington branch, attempted to transfer cash to its outreach branches.
‘S73 As shown
5. This problem involved two separate issues relating to two different scripts.
in the Peak'* and KEL,!°” it resulted in the following sequence of events being
possible:
5.1 The SPM logged into Horizon to make a cash declaration. Due to inactivity, the
SPM was logged out of the counter.
5.2 The SPM logged back into the counter and performed a transfer of cash from their
core branch to the outreach branch. After the two receipts were printed, the user
pressed ‘Enter’ which printed the ‘Rem In’ slip.
5.3 Instead of the ‘Remittances and Transfers Home’ screen being displayed, the
‘Pouch Delivery’ screen was still showing, with ‘Enter’ enabled.
5.4. The SPM pressed ‘Enter’ a further 3 times, resulting the total amount of £32,000
cash being ‘Remmed Out’ of the core branch.
6. All occurrences of the Dalmellington issue would have been clearly visible to the
relevant SPM, who would have had sight of both the duplicate Rems In to the outreach
branches and relevant Rem Out from the core branch.
Detection and resolution
7. In response to the Dalmellington issue being brought to Post Office and Fujitsu’s
1576
attention,'°’° steps were taken to identify the full extent of the issue and those branches
affected that may require financial reconciliation:
POL00026925
POL00026925
1S (F/1389}.
1573 (E/1415}.
1S (F/1389/4}
1575 (F/1426/1}.
1516 (F/1389}.
419
Al8/419
7.1 In 2015 Post Office instructed Fujitsu to undertake an audit of the files sent to POL
SAP each day. These files contain a summary of each day’s transactions for each
branch (the “BLE files”).
7.2 As part of this audit and to identify potential occurrences of the Dalmellington
issue, Fujitsu specifically searched for occurrences of Duplicate Pouch IDs being
Remmed In over the course of the previous 5 year period within the BLE files.'°””
7.3 On 10 December 2015, Fujitsu provided Post Office with the Branch Outreach
Issue (Initial Findings) presentation (the “Initial Findings Presentation”)!*”*
which summarised the results of the audit.
The Initial Findings Presentation was revealing: Fujitsu identified 112 potential
occurrences of the Dalmellington issue. Of these 112 potential occurrences, 108 items
were corrected at the time, cither by Post Office issuing a Transaction Correction or the
SPM reversing the duplicate Rem In.'°” This left 4 remaining potential occurrences, for
which Fujitsu had not yet established the outcome.
As shown in the Release Peak,'**° once the full extent of the Dalmellington issue was
understood, Fujitsu produced a code fix without delay and it was rolled out in January
2016.
Relevant discussions from trial
10.
Mr Coyne treats this as a branch affecting bug and he draws attention to the fact that it
operated for five years without detection.
As can be seen from the Initial Findings Presentation, the nature of the issue was such
that it was not reported to Fujitsu until 2015 (because occurrences of it were either
resolved in by the SPM or Post Office issuing a transaction correction). It cannot fairly
be characterised as a bug which was left to linger in the system while its impacts were
unresolved.
POL00026925
POL00026925
157 (E/1415/3}.
S78 FE/L415}.
15 (F/1415/3}.
1580 (F/1389.1/9} to {F/1389.1/10}.
420
A/8/420
13.
POL00026925
POL00026925
During Mr Coyne’s cross-examination, Mr de Garr Robinson QC asked Mr Coyne:
“So would you accept that, in relation to the Dalmellington bug, the overwhelming
likelihood is that that bug caused no lasting deficiencies in branch accounts”'**!
Mr de Garr Robinson QC explained what he meant in relation to the lasting deficiencies
in branch accounts as follows:
“All of those instances were picked up by the system and either the SPM himself
reversed the Rem in some way and made himself good, or it was picked up by Post
Office and TCs were sent, correct”
To which Mr Coyne agreed and confirmed:
“I think once everything was detected everything was made good”'***
Although Mr Coyne gave the impression in his reports that Dalmellington lay
undetected, he agreed in cross-examination that one of the reasons the issue was able to
exist in Horizon without detection was because robust automatic processes were in place
that would ensure instances involving duplicate remittances were resolved.
As put to Mr Coyne by Mr de Garr Robinson, from an outsider’s perspective, you would
not be able to tell the difference between someone remming in twice by accident, due to
human error and someone remming in twice because of an error on screen. While Mr
Coyne did not wish to admit this point, he acknowledged that the fact that something had
been remmed in twice would always be visible in the logs and receipts and information
available to the SPM, and specifically confirmed:
“Yes, if you was to review the detail then typically that information will be in there,
yes 584
88 (Dayl5/162:1}.
'S® (Day 5/162:
158 (Day 5/16
88 (Dayl 5/1441}.
421
Al8/421
POL00026925
POL00026925
15. It is clear that the issue was not resolved until 2015 because it was extremely difficult to
distinguish from human error and SPMs had no difficulty in reviewing and resolving
their duplicate remittances in any event.
The “unknown outcomes”
16. Mr Coyne noted that the impact of the issue for four branches was still to be
confirmed.'**° However:
16.1 As shown in the Initial Findings Presentation, two of these occurrences relate to
de-minimis values, namely £1.00 and £0.01 .158°
16.2 Post Office investigated the other two occurrences, namely £25,000 and £2,500,
by reviewing NBSC call logs, Credence data and branch audit data and making
enquiries of the FSC team in relation to the knowledge of any Transaction
Corrections issued to the relevant branches.!**” These investigations confirmed
that, in relation to these two occurrences:
(a) neither of these branches operates outreach services;
(b) the potentially ‘duplicate’ remittance transactions were completed on
different counters; !**
(c) where a pouch does not scan and needs to be entered manually, there is a
possibility of the same barcode being entered twice. This can only occur if
the transaction is completed in two separate sessions; '**?
(d) although it was widely thought that all pouch barcodes were unique, Fujitsu
had found that there are a small number of duplicate barcodes on the
system;'*?°
1585 ()2/4.1/28}
1586 {F/1415/7} to {F/1415/8}.
1587 ¢F/1427.1/3} and {F/1427.1/5} to {F/1427.1/6}
1588 (F/1427.1/1}.
158 (F/1427.1/1}.
1590 (F/1427.1/8}.
AI8/422
17.
(ec) there was no shortage or transaction correction recorded for these
branches;!**! and
(f) there was no contact from these branches raising issues for the relevant
periods in question, being February and March 2013.'°°
As a result of the above, Post Office concluded that the correct amount of cash
pouches were delivered and Remmed In on Horizon.'*? In cross-examination, Mr
Coyne stated that he was not of the view that Post Office did nothing about these
remming issues. Indeed, he was of the view that “the opposite” was the case. That
is “likely that they do something about it.”'°°*
During Mr Coyne’s cross-examination, Mr de Garr Robinson QC referred Mr Coyne to
the report prepared by Post Office’
1595 on these unknown issues, which summarised the
conclusions reached above.!°°° Mr Coyne confirmed that he:
17.1
17.4
Accepted that the investigation in terms of the conclusions reached in relation to
the £25,000 and £2,500 unknown outcomes.'**”
Had no reason to think that Post Office’s investigation and conclusions reached
were wrong. 5°
Agreed that in the main Fujitsu are quite good at identifying the branches that have
been affected by bugs like Dalmellington.'*””
The chances of the two smaller unknown occurrences being unresolved — i.e. not
made good by Post Office were “very small” ,'°°
POL00026925
POL00026925
1891 (F/1427.1/2}.
15% (F/1427.1/2}.
1593 (F/1427.1/2}.
15 (Day 4/54:17} to (Day14/55:22}.
1595 (F/1427}.
186 Day17/158;!} to {Day17/160:)}.
1897 (Day17/159:)} to {Day1 7/160: 1}.
1598 (Day17/159;)} to {Day17/160:)}.
18 (May14/93:12} to {Day14/93:15}.
1600 (Day17/161:14}.
423
AI8/423
Bug 5: Remming In
1. The key documents:
1.1 PCO203085'*; PCO195380.16
1.2 KEL acha4221Q.'0
1.3 Coyne 2, paras 3.56 — 3.66.64
1.4 J82,10°8
Summary
2. Mr Coyne states that Bug 5: Remming In is a bug with lasting financial impact. Post
Office submits that any discrepancy would be transient as instances of this bug are
caught by automatic reporting.
Nature of this issue
3. Mr Coyne has conflated two distinct issues under the heading of “Remming In”:
3.1 Coyne 2, paras 3.56 — 3.59 relate to PC0203085' (“Issue 1”).
3.2 Coyne 2, para 3.60 relates to PCO195380'° and other Peaks associated with KEL
acha4221Q' (“Issue 2”).
4. Both issues relate to remming in cash and result in a cash pouch being recorded twice
in error, but the sequence of steps taken by the Subpostmasters to trigger them are
significantly different.
5. Subpostmasters rem in pouches of cash sent to the branch from the Post Office cash
centre. Each pouch has a unique barcode that needs to be scanned by the branch. This
POL00026925
POL00026925
1601 (F/695,1}.
1602 (F/588}.
3603 (F/794}.
1604 (1)2/4,1/29} to {D2/4.1/33}.
1605 (1)1/2/5}.
1606 (F/695,1}
1607 (F/588}.
1608 £F/794}..
424
AI8/424
Issue 1
automatically looks up the contents of the pouch so that the branch can confirm that the
physical contents of the pouch match up to the record on Horizon. Horizon then prints
a physical receipt to rem in and the cash is then added to the branch cash holdings.
If there is a difference between what the system says is in a pouch of cash and the
amount of cash actually in the pouch, the Subpostmaster should raise the issue with
NBSC in order to get a Transaction Correction.
A remming error leads to a mismatch between the amounts of cash remmed out to one
place and the amounts remmed in from another. Remming errors are a violation of Data
Entry Accounting and are picked up by Horizon.
During his cross-examination Mr Coyne confirmed that he believed these bugs to have
lasting financial impact.’ This is incorrect. While both caused shortfalls in branch
accounts, those shortfalls were cancelled out by Transaction Corrections.
As set out in Peak PC0203085, the Subpostmaster started remming in a pouch on
Counter I by scanning its barcode. The Subpostmaster did not complete the rem but
stopped doing it halfway through, without cancelling it. She then moved to a different
counter and, whilst remaining logged onto Counter I and using someone else’s log on
details on Counter 2, scanned the pouch barcode again and the rem was completed as
normal. The Subpostmaster then completed the same on Counter 1, which caused the
rem to be recorded twice and two lots of cash to be added to the branch accounts on
Horizon, creating a shortfall because there was actually only one lot of cash.
The duplicate rem would have been clearly recorded and visible to the Subpostmaster
in the transaction log. Rem ins are large, round numbers due to them being cash
deliveries and the resultant shortfall would therefore have been large and noticeable to
the Subpostmaster.
Horizon keeps a list of all remmed in pouches and if a pouch has been remmed in
already, it will reject the pouch when the barcode is scanned in. However, a pouch is
only added to the remmed in list once the rem in process is fully completed. However,
in this case the Subpostmaster did not complete the rem in process on Counter 1 before
POL00026925
POL00026925
1609 {Day17/130:9} to {Day17/130:12}.
425
AI8/425
15.
Issue 2
16.
she scanned the same pouch on Counter 2, which led to Horizon allowing the rem to go
through. Had the Subpostmaster cancelled and restarted the process by scanning the
barcode again, Horizon would not have allowed the second rem on Counter 1, because
it would have been recorded as already completed on Counter 2.
Mr Coyne states at para. 3.58 of his report that it took ten months to fix this issue.'©!°
This is inaccurate and it appears that Mr Coyne has muddled the two issues, counting
from the start of Issue 2 (March 2010) to the end of Issue 1 (January 2011). Issue 1
was raised on 17 August 2010 and a BIMS was issued to Post Office containing
information for Post Office to remedy the discrepancy on 18 August 2010. Fujitsu then
developed a fix to prevent further occurrences.
Mr Coyne states at paras. 3.59 of his report that “/t/his bug was only brought to the
attention of Fujitsu/Post Office following notification from the Subpostmaster”*"
While it is correct that the issue was reported by the Subpostmaster, it would have been
spotted by Post Office in any event because Post Office reconciles all rems on Horizon
with cash leaving their cash centre.
Mr Coyne’s suggestion at paragraphs 3.59 and 3.60 that the two Issues were related to
Dalmellington and “differing manifestation[s] relating from the same core bug” is not
correct.” Dalmellington was a separate issue that only affected outreach branches
(although both issues had similar symptoms).
A fix was implemented on 23 January 2011'°'%, This meant that pouches are
temporarily added to the “remmed in” list once the barcode is scanned, rather than
waiting for the rem to be completed. This was part of the original design of the system,
but it was not properly implemented.
In Peak PC0195380, the Subpostmaster started remming in a pouch by scanning in its
barcode and then pressed “PREV” which moved them back one screen. They then
POL00026925
POL00026925
161 ()2/4,1/30}.
1611 ()2/4.1/30}.
1612 ()2/4,1/30}.
1613 (F/695.1}.
426
AI8/426
18.
19.
20.
POL00026925
POL00026925
scanned the barcode again. Horizon recorded the same pouch twice but only printed
one receipt. This caused a shortfall in the branch accounts.
The duplicate rem would have been evident in the branch’s transaction records.
This issue occurred in the pilot phase of HNG-X. A fix was implemented that disabled
the “PREV” button from the rem in process. Thereby eliminating the risk of
Subpostmasters moving back a screen and duplicating the rem in. Affected branches
would have been corrected by ordinary rem checks.
Mr Coyne has correctly identified at para. 3.62 of his Supplemental Report that an
associated Peak to KEL acha4221Q was raised after Peak PC0195380 was fixed.'°'
This is likely to be because it can take a few days for a fix to percolate through the
entire network.
Regarding the other Peaks associated with KEL acha4221Q referred to by Mr Coyne in
para 3.63 of Coyne 2:15
20.1.1 PCO195511'°!*; PCO196120'°"” and PCO196154'*"* are all duplicates of Peak
PC0195380'*!" and were sent by Fujitsu to Post Office to compensate the
Subpostmaster on 4 March 2010, 23 March 2010 and 18 March 2010
respectively.
20.1.2 PCO196671;' — _PCO197032;'?! — PC0197034;'° — PCO197605;°
PCO197651;'°* = PCO197753'>; PCO197828;'°° ~~ PCO197837;'°7
164 (1 2/4.1/31}.
1618 ¢F/794}..
1616 £F/589}.
1617 4F/595}.
1618 £F/596}..
1619 ¢F/588}.
1620 {F/599}.
1621 (F/603}.
1622 ¢F/604}.
1623 £F/612}.
1624 £F/615}..
1625 £F/620}.
126 £F/622}.
1627 £F/623}..
427
AI8/427
21.
PCO197838;'°% PCO197872;'% PCO197873;'° PCO198115'%*! are all issues
that were detected by Fujitsu. Fujitsu produced a report that would detect all
of them except PC0197605'°* (because that did not involve an error and the
duplicate rem was intentional). The report enables MSU to detect any further
incidents of the issues and raise a BIMS without SSC involvement. BIMS
were sent to Post Office by Fujitsu to compensate the relevant
Subpostmasters on various dates in April and May 2010.
20.1.3 It is unclear why PC0226230'* is on Mr Coyne’s list, as there is no reference
to KEL acha4221Q within it.
20.1.4 PC0246629'* is not a Horizon error, as Mr Coyne correctly identifies. It
appears to be user error.'6*
20.1.5 PC0251952'**° is an operational issue, not a bug. It refers to the branch
having incorrect pouch types.
Mr Coyne’s suggestion at para. 3.64 of Coyne 2 that “/t/he Release PEAK in relation
to the fix for PCO195380 (referenced by Dr Worden and Mr Parker in relation to KEL
acha4221Q) does not document every PEAK that would be impacted by the fix or
reference that the fix specifically applied to KEL acha4221Q. It also does not record
whether it was fully rolled out across the estate as at 19 April 2010 (the date given by
Mr Parker in his witness statement)”'®*” is incorrect: that is not the purpose of a Release
Peak. The purpose of a release Peak is to show which software fixes have been issued.
The original Peak PCO1953805%4-416 stated that the fix was to be rolled out from 19
April 2010 (which is presumably the document from which Mr Parker obtained this
information). The Release Peak says that it was actually rolled out from 22 April 2010.
POL00026925
POL00026925
1628 (F/624}.
1629 (F/626}.
1630 (F/627}.
1651 (F/632}.
1632 (F/6 12}.
633 (F/1081}.
16 £F/1380}.
1695 ()2/4,1/32}.
1636 (F/1 484}.
167 (1)2/4,1/32} to (D2/4.1/33}.
1638 4K
428
AI8/428
Conclusion on Bug 5: Remming In
22. These issues occurred because of unusual steps taken by Subpostmasters.
23. Peak PC0195380 occurred during the pilot phase of Horizon Online and the other Peaks
in Issue 2 led to a report being created by Fujitsu to detect further occurrences and
report them to Post Office so that Post Office could issue Transaction Corrections. The
two branches in Issue I were also remediated with Transaction Corrections.
Bug 6: Remming out
1. The key documents:
1.1 PC0143435;'° PCO120937.'°
1.2 Coyne 2, paras 3.67 - 3.77.10"
1.3 J82,'60
1.4. KEL acha508S;!? GMaxwell3853P.!04
1.5 Service Review Book February 2007.1
Summary
2. Mr Coyne states that Bug 6: Remming Out is a bug with lasting financial impact. It
comprises two separate issues, only one of which was a bug. Any discrepancy caused
by wither issue would be transient as instances of both issues were caught by automatic
reporting.
1639 (F/384}.
16 {F/271}.
16H (1)2/4.1/33} to {D2/4.1/37}.
168 (1)1/2/5} to (D1/2/6}.
1688 ¢F/403}.
164 £F/276}.
1688 1F/409.1}.
429
POL00026925
POL00026925
Al8/429
POL00026925
POL00026925
Nature of this issue
3. Mr Coyne has conflated two unrelated issues under the heading “Remming Out”:
3.1 Para 3.67 —3.72 relate to PC0143435'° and other Peaks that relate to the remming
out of coins (“Issue 1”).
3.2 Paras 3.73 — 3.76 relate to issue 2 (PC0120937)'*" (“Issue 2”).
4. These Issues occurred in Legacy Horizon.
5. Mr Coyne confirmed in his cross-examination that he believed “Remming Out” at 6(1)
and 6(2) of JS2 had lasting financial impact.'6*
6. A remming error leads to a mismatch between the amounts of cash remmed out to one
place and the amounts remmed in from another. Remming errors are a clear violation
of Data Entry Accounting and are picked up by Horizon.
Tssue 1
7. By way of background, Subpostmasters rem out pouches of cash to be returned to the
Post Office Cash Centre. A single bag may contain multiple bags of coins or cash and
each bag can only hold one denomination.
8. There is a limit on how much cash can go in one bag, with two or more bags being used
if this limit is exceeded.
9. A branch has to record on Horizon the amount being placed in each pouch, by number
of bags, value and denomination. Once remmed out, the cash is removed from the
branch’s cash holdings in the accounts and is no longer part of the branch’s cash
position for balancing purposes. It is temporarily recorded on a separate line in the
accounts as “cash in pouches”. This is a holding area in the accounts for the cash, where
it remains until the pouch is physically collected by cash collection van — usually the
1616 (F/384}.
1647 F/271}.
168 (Dayl7/130:13} to {Day17/130:16}.
430
A/8/430
10.
ll.
12.
14.
same day or the next day. On collection, the collection team scan a barcode on the
pouch and the cash is removed from the “cash in pouches” line of the accounts.
When remming cash out, branches should have made one entry for each denomination
and value and, if there were multiple bags for a particular denomination, the quantity
of bags should have been specified in that single entry (e.g. 2 x £500 of £2 coins).
However, some Subpostmasters made multiple entries for each denomination and value
(e.g. one entry for 1 x £500 of £2 coins and then a second entry for 1 x £500 of £2
coins). Horizon only recorded the first bag as having left the branch’s cash holdings,
but all of the bags would show on the “cash in pouches” line. Therefore when the
pouches were collected a shortfall was created in the branch accounts (e.g. only £500
of £2 coins were recorded in the accounts as being remmed out, but in reality two bags
were physically removed from the branch).
If the branch spotted the error and reversed the rem, only one bag would be returned to
the branch. This would mean that the cash position in the branch accounts would be
correct (one bag remmed out; one bag reversed back in), but the “cash in pouches”
position would also only reverse one bag (two bags put in “cash in pouches” and only
one reversed; leaving one bag to be collected). If the bag was collected, this would
create a shortfall.
This issue occurred as a result of Release T30 INC1.'*?
Of the twenty-one Peaks referred to by Mr Coyne, two were erroneous duplicates, six
originated from calls bye Subpostmasters and thirteen originated from an automated
payments mismatch report.
Fujitsu created KEL acha508S to advise branches to correct the problem manually and
ran automated reports to spot any further occurrences. Fujitsu then made changes to
Release T30 INCI and rolled it back out to fix the issue.'°?
POL00026925
POL00026925
169 (F/409.1}.
1650 (F/384/5}.
431
A/8/431
15.
Tssue 2
POL00026925
POL00026925
In these cases the shortfalls were caused by the way in which users remmed out cash.
Fujitsu ran automated reports to spot further occurrences in the period until the fix was
rolled out.
In Peak PC0120937'®*! the Subpostmaster was trying to process a rem out of £100 of
5p coins. The Subpostmaster scanned the barcode on a stock pouch when they were
trying to rem out coins and Horizon generated a message saying “Incorrect Pouch
Type”. This is because the Subpostmaster should have used the pouch designated for
coins. The Subpostmaster was then presented with the option to either cancel the rem
out or retry.
In this instance, the Subpostmaster pressed cancel. Horizon then began the process of
reversing the rem out. During this process, the “Home” button was displayed for a brief
moment and the Subpostmaster could press the icon while it was displayed. The
“Home” button is a commonly used button that returns the Subpostmaster to the home
screen on Horizon. Pressing it caused the remittance of £100 to be transferred to the
suspense account rather than be cancelled.'°*
As the sum was placed in the suspense account with no correlating pouch ID (because
the barcode scan failed), the Subpostmaster could not remove the sum from the
suspense account and so called the helpline.
Mr Coyne states that there was a “/ack of system control preventing the input error in
relation to remming out coins” and that “this issue arises from functionality that should
not be available (but is) when the Horizon system is under load”!®*. As is evident from
the Peak, Fujitsu attempted to recreate this situation but they could not access the
“Home” button. They then reviewed the Horizon software code to find the issue. This
led them to conclude that for a brief moment, following the cancellation of a rem when
Horizon was doing automated actions (no user input required), the Home button might
651 (F/271}.
1652 (F/271/2}.
4653. ()2/4,1/36}.
AI8/432
20.
21.
be displayed'®*4,
It was only likely to be possible for a user to press the home button
ona “very slow or extremely busy”'®* branch terminal, presumably because on a slow
machine the Home button would be displayed for a slightly longer period allowing the
user to press it before it automatically disappeared.
Mr Coyne is correct in his suggestion that there was no fix for this bug. Rather, KEL
GMaxwell3853P was created and a Transaction Correction was issued to fix the single
known incident.'°* As set out in the Peak, a decision was made “/g/iven the frequency
of the problem & the apparent risk involved in introducing a code fix” that “the KEL
should be adequate” ‘7
This incident only affected one branch. Also, it caused no financial impact, which Mr
Coyne has failed to acknowledge: the £100 remittance was recorded in the suspense
account, which was still part of the branch accounts. It would not have left the branch's
because the remittance process was not completed on Horizon, meaning there was no
loss or gain.
Conclusion on Bug 6: Remming Out
22.
23.
24.
Issue I was a bug having an impact on branch accounts, was caused by an unexpected
series of events. In any event, it was picked up automatically.
Issue 2 was not a bug and had no impact on branch accounts. A KEL was put in place
in case a Subpostmaster came across this incident again.
Mr Coyne’s suggestion at para. 3.77 that Issues I and 2 “illustrate how a bug that
manifests in slightly different ways can be analysed and diagnosed differently amongst
the varying technical support members” is incorrect. Issues I and 2 were distinct issues.
POL00026925
POL00026925
1654 (F/271/3} to {F/271/4}.
1655 (F/271/3}.
1656 (F/276}.
1657 £F/271/5}.
AI8/433
Bug 7: Local Suspense Issue
1. The key documents:
1.1 Peaks: PCO197409;'%8 PCO198077;'°° PCO197797;1° and PC0204396.'°!
1.2 KELs: acha5259Q'* and PorterS199P.'°
1.3 Coyne 2, paras 3.78 — 3.83.16
1.4 Js2,16°5
Summary
2. Mr Coyne states that Bug 7: Local Suspense is a bug with lasting financial impact. Post
Office submits that any discrepancy would not be lasting.
Nature of this issue
3. Peak PC0197409'°* relates to an intermittent system issue which temporarily prevented
branches from rolling over into the next Trading Period.
4. This issue only affected those branches that had unresolved discrepancies in their
Discrepancy Accounts. Unresolved discrepancies must be cleared by the SPM before the
branch can rollover.
5. After the SPM had selected a settlement option for the relevant discrepancy, the next step
should have been for a message to appear on Horizon to confirm the Stock Unit had
rolled over. However, this issue caused Horizon to repeatedly ask the SPM how they
wanted to settle the discrepancy and then to display an error message. This meant that
the branch was unable to rollover into its next Trading Period.
POL00026925
POL00026925
1658 (F/609}.
689 (F/630}.
1660 (F/618}.
1661 E/T 11}.
1662 (F/637}.
1665 (F/629}.
1664 (1)2/4.1/37} to (D2/4.1/39}
1665 (1)1/2/6} to {D1/2/7}.
1666 {F/609}.
434
AI8/434
6. There were two consequences for the affected branches:
6.1
6.2
they were temporarily unable to rollover; and
where the SPMs kept selecting to make good the discrepancy, their balance
snapshot would show a nil-discrepancy and an inflated cash figure.
Detection and Resolution
7. — This issue would have been clearly visible to the SPM as they were repeatedly asked to
make good the discrepancy, they were shown an error message and they were unable to
rollover. Peak PC0197409'%’ shows that the issue was reported to the NBSC by a
number of SPMs in April 2010, during the early days of Horizon Online (during the pilot
scheme
8. As can
Fujitsu:
8.1
8.2
8.3
when Horizon Online was rolled out to a limited number of branches).
be seen from Peak PC0197797,'%* in response to an issue reported by SPMs
Investigated the issue and determined that branches would be able to use their
housekeeping functions to rectify the issue and clear the losses and/ or gains from
their Local Suspense Accounts.'°?
Searched for “other branches which were affected by the same problem.”'*
Prepared a spreadsheet identifying the 33 branches which were affected by the
problem and highlighting those branches that “may need action” before the end
of the Trading Period.’ As per the entry in PCO197797 on 28 April 2010 at
17:39,'6” Anne Chambers asked for this spreadsheet to be passed to Post Office,
together with the information relating to the ability for SPMs to use their
housekeeping functions to correct the position in their Local Suspense Accounts.
POL00026925
POL00026925
1657 {F/609}.
1668 (F/6 18}.
160 (F/6 18/4}.
160 §F/6 18/4}.
1671 (F/6 18}.
1672 (F/6 18/4}.
435
AI8/435
POL00026925
POL00026925
8.4 Issued a BIMS Incident Management report on 30 April 2010.'673
8.5 Passed the issue to development for further investigation.'°*
9. Apermanent code fix was released in September 2010, Release Peak PCO198077.'°7°
10. As can be seen from Peak PC0204396, in April 2010.17
10.1 Anne Chambers confirmed that there had been no further occurrences of the
issue since the code fix was implemented in September 2010.'°7
10.2 On 17 September 2010 at 11:18 Anne Chambers stated that she would “make
some checks for branches affected by this problem and send details via
BIMS.”'678
10.3. Anne Chambers prepared a spreadsheet which showed that 60 branches had been
affected by this problem since June, of which she confirmed she had fully
investigated 27 instances where the impact was greater than £20.'°”
10.4 A final BIMS report, together with the spreadsheet containing the 60 branches
that had been affected by this problem since June was sent to Post Office on 24
September 2010,'°°
Conclusion
ll. This was a teething problem that arose in the early days of the Horizon Online pilot
scheme. The problem was immediately identified and Fujitsu implemented checks to
identify the affected branches and investigated the individual instances. Once its
investigations were complete, Fujitsu sent details of the affected branches to Post Office
to ensure that there were no lasting discrepancies in the affected branches.
1673 £F/618/8}.
1674 (F/6 18/8}.
1675 £F/630}.
1676 ¢F/711/5}..
1677 ¢F/TLL/5}..
1678 (F/T11/5}.
1679 F/713.1}.
1680 £F/711/6}.
436
AI8/436
POL00026925
POL00026925
12. Despite Mr Coyne stating during cross-examination that there is evidence that this bug
1681
had lasting financial impac his reports make no reference to any such evidence.
The affected branches would never have suffered lasting discrepancies, because as well
as causing Horizon to repeatedly ask the affected branches how they wanted to settle the
discrepancy and to display an error message, the bug prevented them from rolling over.
The matter was bound to come to Fujitsu's attention very quickly, as indeed it did.
Bug 8: Recovery Issues
The key documents
1. Peaks: PCO197769,'° PCO198352'* and PCO199000'**.
2. Coyne 2, paras 3.84 — 3.87.15
3. -JS2,1686
Summary
4. Mr Coyne states that Bug 89: Recover s is a bug with lasting financial
sed
impact. Post Office submits that if is pot snwatierobbedastiig.
Nature of this issue
5. Sometimes a basket of transactions is interrupted during the course of dealing with a
customer, which could be for several reasons including a power failure, system crash or
communication failure between the branch and data centre. There is no way to eliminate
this risk, which is why a safeguard is needed in the form of a recovery process.
1681 (Day 17/130:17} to {Day17/130:18}.
16 (F/617}.
168 (F/636}.
168 £F/650.1}.
1685 ()2/4,1/39} to {D2/4.1/40}.
4686 ()1/2/7}.
AI8/437
POL00026925
POL00026925
6. Horizon runs various automated reports each day to look for failed recovery events.
Where there has been a failed recovery, this is flagged automatically by Horizon to SSC
at Fujitsu for investigation!®*”,
7. Mr Coyne refers to two distinct recovery issues in his report:
7.1 Para 3.84 relates to issue 1 (PCO197769) (“Issue 1”).
7.2 Para 3.87 relates to issue 2 (KEL acha959T'®*) (“Issue 2”).
Tssue 1
8. Mr Coyne refers to PC0197769 and its associated KEL acha5650L. It is evident from
the Peak that:
8.1 this issue arose during the pilot phase of Horizon Online;
8.2 the Peak was raised on 15 April 2010 and Fujitsu identified the root cause of the
issue on 26 April 2010;
8.3. work ona fix began on 27 April 2010; and
8.4 a report was put in place to detect other affected branches.
9. Release Peak PC0199000'*° shows that a fix was implemented quickly by Fujitsu. It
went to the Model Office on 1 June 2010 and Fujitsu have advised that it would have
been rolled out to the rest of HNG-X pilot by mid-June.
10. The issue is described in KEL achaS650L.®° It involved recovered transactions being
written to a different trading period than the original transaction. In summary:
10.1 ifa transaction failed in one trading period and the recovered transaction went into
the next trading period, there would be a loss in the first trading period 5 and a
corresponding gain in the next trading period such that there would be no overall
loss in the branch; and
2687 SVM/SDM/PRO/0012 {F/1051}.
1688 (F/1700}
168 (F/650.1}.
1690 ¢F/1025}.
438
AI8/438
12.
‘further financial discrepancies would be likely in respect of this same KEL issue.
POL00026925
POL00026925
10.2 if the recovered transaction was written to an earlier trading period, there would
be a loss in the current trading period but no corresponding gain because the
previous trading period would have already been rolled over again. There would
be a net loss in that scenario.
In his first witness statement Mr Parker'®! discussed KEL acha5650L'®? and
commented that there was no financial impact because there were offsetting transactions
at the one branch in question. That analysis is correct for the branch in question.
Mr Coyne refers to Peak PC0198352'°* as an example of this issue happening in another
branch on 29 April 2010 and creating a discrepancy. However, Mr Coyne overlooks the
fact that:
12.1 PCO198352 (2 May 2010)!©* was raised as a result of the automated reports that
Fujitsu had put in place to detect further occurrences of this bug; and
12.2 the Peak shows that a BIMS was issued to POL to give POL the information
needed to correct the issue, presumably by way of a transaction correction.
Mr Coyne’s conclusion in para 3.86: “/t is my opinion that with additional research,
991695
This overlooks the fact that any further branches affected by this bug would have no
lasting impact due to the monitoring put in place by Fujitsu and Fujitsu reporting any
further occurrences of the issue (before the fix was implemented) to Post Office.
Issue 2
14.
Mr Coyne references several KELs and Peaks in support of his assertion that failed
recoveries do happen and that they cause a financial impact. However, the definition of
financial impact appears to be the actual point in issue.
1691 {E2/11/76}.
162 £F/1025}.
1693 (F/636}.
164 (F/636}.
4695 ()2/4,1/40}.
439
AI8/439
18.
POL00026925
POL00026925
KEL acha959T! is the master KEL for when the automatic monitoring for failed
recoveries identifies an issue (a “State 4”). It was created on 28 February 2010 during
the early pilot of Horizon Online. It is therefore frequently referenced in Peaks and in
other KELs, such as KEL carde464Q. At paragraph 3.95 of his report Mr Coyne states
that he has randomly selected a Peak that refers to KEL carde464Q and states at
paragraph 3.96 that it relates to issues “where customer transactions are part processed
but the transactions are not recorded in the BRDB or on the Counter.”"°"
It should be noted that this recovery process only relates to Horizon Online and no
transactions are recorded on the Counter in Horizon Online (that is how Old Horizon
worked). Leaving that point to one side, this entire sequence of KELs and Peaks is
designed to capture and correct exactly the scenario described by Mr Coyne and KELs
and Peaks referred to by Mr Coyne show that the monitoring process is working.
At paragraph 3.89 Mr Coyne notes that the resolution of Peak PC0256502 was not
recorded in the Peak “as this would ultimately be down to Post Office to issue a
Transaction Correction, whether they did or did not has been deemed out of scope by
Post Office.°°*
Mr Coyne does refer to one specific incident (PC0256566'” and PC0256502!™) which
he tries to use as an example of how a failed recovery could impact a branch, but a full
review of the Peaks show that the above monitoring and correction process worked as
designed.
Mr Coyne’s handling of Peaks PC0256566'! and PC0256502'”” is muddled by the fact
that he refers to them in reverse chronological order, which means he appears to have
become confused about the outcome. The SPM suffered a transaction failure due to a
communications issue — this is not a bug but an inherent risk in any networked IT system.
1696 F/1700}.
1697 ()2/4,1/43}.
1698 (2/4.1/41}.
1699 £F/1602}.
1700 £F/1600}.
1701 (F/1602}.
1702 £F/1600}.
440
A/8/440
20.
21.
22.
POL00026925
POL00026925
The recovery process also failed due to a communications issues. There was no bug —
the communications issues were outside of Post Office’s control.
This is a clear example of why Fujitsu has monitoring in place. On 16 January 2017, the
automatic monitoring service spotted the failed recovery and Peak PC0256502'7° was
raised by Fujitsu. A BIMS was issued to Post Office to correct the discrepancy!
On 17 January 2017, the SPM contacted the helpline regarding the same issue. A second
Peak PC0256566 is raised. As quoted by Mr Coyne in his report, this Peak clearly
records that this issue was already identified by Fujitsu and resolved under the earlier
Peak. Hence the late Peak is closed without any action being taken.
At para 3.89 Mr Coyne insinuates that the Peaks were closed and recorded as “no impact”
even though there was a financial impact and that a “manual reconciliation” was required
to correct it. This fails to recognise the failed recoveries monitoring and the BIMS
process was already in motion and correcting the issue.
Conclusion
23.
24,
There is no way to eliminate the risk of a basket of transactions being interrupted during
the course of dealing with a customer, which is why a safeguard is needed in the form of
a recovery process.
Dr Worden addressed why manual recovery processes are necessary:
A. It is a different theory really. Recoverable transactions are a big subject and they
are complicated because the point at which a recoverable transaction can go wrong
is very variable through the sequence, and therefore the number of recovery actions,
the type of recovery actions is complicated. Horizon was designed so that with the
assistance of the postmaster most of these could be recovered, but there are things
called failed recoveries, which again were part of the design of Horizon, and those
were the failed recoveries but particularly the ones where Fujitsu had to get involved.
But failed recoveries means the recovery process had failed, that’s the way Horizon
1703 (F/1600}.
1704 (E/L601/1}.
441
Al8/441
POL00026925
POL00026925
was designed because these things are so complicated you can’t handle them all
automatically. So it is a big subject but there is plenty of useful evidence about it.17°
25. Horizon contains various automated reports which look for failed recovery events and
these are flagged automatically by Horizon to SSC at Fujitsu for investigation. Fujitsu
then passes information to Post Office where appropriate. The KELs and Peaks referred
to by Mr Coyne are examples of this.
Bug 9: Reversals covery-Issues
1. The key documents:
1.1 Peaks: PC0089918;'7°° PC0090109'7" and PC0083954.!7
1.2 KEL PSteed2847N.'™
1.3 Coyne 2, paras 3.99 — 3.104.'7!°
1.4 Second Joint Statement.!7!!
Summary
2. Mr Coyne states that Bug 9: Reversals is a bug with lasting financial impact. Post Office
submits that any discrepancy would not be lasting.
Nature of this issue
3. Peak PC0089918!7" relates to an issue in which attempted reversals of remming in
transactions resulted in the magnitude of the transaction doubling, rather than the
transaction being reversed.
1795 Day20/39:19} to {Day20/40:9}.
1706 (F/149}.
1707 (F/149.1}.
1708 (F/127},
1709 FF /152}.
170 (1)2/4.1/44} to (D2/4.1/45}
1711 (1) 1/2/8} to {D1/2/9}.
172 4F/149},
Al8/442
POL00026925
POL00026925
The Subpostmaster (branch FAD003227) was trading in Stock Unit Y and remmed in
£13,910 of cash. He continued to trade in that stock unit in error, instead of moving to
Stock Unit I. He therefore attempted to reverse all of the transactions he had made in Y
(including the rem in). Instead of Y showing a balance of zero, the rem in had doubled
to show a discrepancy of £27,820. Having spoken to NBSC, the Subpostmaster
attempted further reversals but these failed and produced an error message indicating
that the initial reversal had been completed successfully.
The issue was caused by a software error, which had been introduced as part of
implementing the fix for PC0083954 in Legacy Horizon, and which incorrectly
calculated the reversal value and quantity (essentially, the wrong mathematical sign was
applied when reversing RIAD transactions (+ instead of -)).'" The PC0083954 fix
introduced a code change to ensure that a partial settlement of cash always tried to reduce
the stack value. However, that fix did not work when reversing a rem in and so the
solution was to pass the mode to the function also.
Resolution
6.
Within 3 days of the issue first being identified, KEL PSteed2847N'7"* had been raised.
Within 6 days of the issue first being identified, Fujitsu’s 4" line team had identified the
root cause — a software error was introduced when a fix was implemented in respect of
Peak PC0083954.'7!5 Because of the potential impact on live branches, the development
of a fix was fast-tracked; within 15 days of the issue first being raised, Fujitsu had
implemented a fix to 2,178 branches. It is not known precisely when the fix reached all
branches in the estate, but Fujitsu believe it is likely to have been before 2 June 2003
(PC0089918 was raised on 25 April 2003).!7!°
Branch impact
7.
In ordinary circumstances, a transaction and its reversal are preceded by opposing
mathematical signs (+ or -) on the transaction report. The effect of this issue was that
the transaction and its reversal were preceded by the same mathematical sign on the
1718 (B/127}.
174 (F152).
IS 4F/127}.
1716 (F/149}.
443
AI8/443
transaction report. Therefore, the issue would have been apparent to a Subpostmaster
from his transaction report and, as a result, any financial impact of the issue should have
been transient.
In this particular case, the Subpostmaster noticed and raised the issue. The Peak notes
that MSU and NBSC were going to liaise in order to assist the Subpostmaster with
dealing with the discrepancy and that an error notice would need to be issued to the
Subpostmaster.
Whilst PC0089918 notes that more than one branch was affected by the issue, no other
similar Peaks or references to KEL PSteed2847N have been found.'7!” This will be partly
attributable to the fact that the 1* line help desks are notified of identified issues and
instructed to advise Subpostmasters accordingly.
Conclusion
10.
ll.
The issue referred to in PC0089918 had the potential to impact branch accounts.!7'*
However, the issue would have been apparent to a Subpostmaster from his/her
transaction report and would have been rectified by an error notice, therefore, any
financial impact is likely to have been transient.
Fujitsu fast-tracked the investigation of the issue, based on its potential impact, and were
able to diagnose the issue, and develop and implement a fix within a short period of time.
17 (F/152}.
1718 ¢F/149}.
444
POL00026925
POL00026925
Al8/444
POL00026925
POL00026925
Bug 10: Data tree build issues
The key documents:
1. Issue 1:
1.1 Peaks: PC0033128;!7 —PC00S8161;!7° PC0046811;'7! —PC0055964;!7?
PC0059497;!75 PC0038631;!724 PC0045847'75 and PC0043811.'7°
1.2. Coyne 2, para 3.106.'7”7
13 1728
Second Joint Statement
2. Issue 2:
2.1 Peaks: PCO121925;!”° PCO123319;'7 PCO132133!™*! and PC0144386.!7
2.2 KEL MSCardifield22198.!7*°
2.3. Coyne 2, paras 3.110'7 and 3.11517
4 Second Joint Statement!7*°
1719 4/22},
1720 (F/76}.
121 (F/29}.
1722 (F/66}.
1723 (F/76.1}.
174 (F/26},
1725 (F/27}.
1726 (E/24}.
1 ()2/4,1/45}.
178 (1/2/9}.
1729 (F/275.1}.
1730 (F/288.1}.
11 (F/332}.
1782 (F/410}.
1793 (F/428}.
17 §D)2/4.1/47},
1795. ()2/4,1/48}.
1736 ()1/2/9}.
445
AI8/445
Summary
3.
Bug 10: Data Tree Build is a bug with the potential for lasting financial impact. Mr Coyne
has drawn together two distinct issues under the heading of “data tree build failure
discrepancies.” The first arose early in the life of Horizon, was detected quickly and was
fixed. Only three branches were affected and all three were identified. The second was
identified in testing before the changes made to Horizon in 2005 and was fixed by a
notification procedure before those changes were rolled out. There were some rare
instances where this procedure was switched off and monitoring system was put in place
to counter this.
Horizon’s “data tree”
The data tree was part of the Legacy Horizon counter, which was used to build up a
picture of the accounts based on a search for all transactions within a given period
(the current balancing period). The counter would scan the messagestore for each
transaction between limits specified by marker objects and, for each transaction
found, would assign it to one or more points within the accounting hierarchy (defined
in reference data). This then enabled reports such as office snapshots to be produced
and any discrepancies to be identified.
Nature of this issue
5.
Mr Coyne has drawn together two distinct issues under the heading of “data tree
build failure discrepancies”:
5.1 Para 3.106'77 relates to issue 1 (PC0033128'7*8) (“Issue 1”).
5.2. Paras 3.110'7°° and 3.115!7° relate to issue 2 (PCO132133'7"! and PCO144386!7)
(“Issue 2”).
POL00026925
POL00026925
1787 ()2/4,1/45}.
1738 (F/22},
1789 ()2/4,1/47},
10 ()2/4.1/48}
1741 (F/332}.
12 (F/410}.
446
AI8/446
POL00026925
POL00026925
Issue 1
6.
Peak PC0033128 relates to an issue in November 1999; the Subpostmaster (Dungannon
branch) reported a £43,000 discrepancy after balancing stock units and doing an office
snapshot.!78
An office snapshot is a report that could be run from Horizon that showed the current
accounting position of the branch, including any cash discrepancy. To produce the office
snapshot, Horizon scanned the messagestore for the necessary information (eg. initial
cash and stock levels, all cash and stock transactions, plus other service transactions). It
then compiled that information together in order to produce the office snapshot. This is
the “data tree”.
Horizon would, on occasion, fail to read the necessary transaction data for a number of
reasons and so fail to compile an accurate report. In the case of the Dungannon branch,
this failure was caused by a missing payments node.
Issue 1 caused the incomplete data tree to be built. This meant that the Subpostmaster
would be presented with an incorrect office snapshot with no knowledge that it was
incorrect. That office snapshot might show a shortfall that was not real. The Dungannon
branch’s snapshot showed a false shortfall of £43,000.
In addition to the Dungannon branch, two branches were identified in March 2000 as
having been affected by Issue 1 — Yate Sodbury (FAD 025511) and Appleby-in-
Westmorland (FAD 158410). Yate Sodbury’s snapshot showed a false shortfall of
£52,814.29 and Appleby’s snapshot showed a false shortfall of £9,368.40. MSU were
involved in the investigation of Issue I and it is therefore likely that the issues at Yate
Sodbury and Appleby were resolved via a BIMS report and that the Subpostmasters were
held harmless. However, due to the age of Issue 1, comprehensive records are not
available and therefore Post Office is not in a position to provide detailed commentary.
PC0058161 was also affected by Issue 1, resulting in a £3,236 discrepancy.'™ The
diagnostic fix referred to above was in the process of being rolled out when this
178 (F/22},
14 (F/76}.
447
AI8/447
12.
discrepancy arose; a BIMS report was raised in order for the Subpostmaster to be held
harmless.
Mr Coyne also refers to a number of other Peaks which relate to Issue 1. These are:
PC00468 11,175 PC0055964'"° and PC0059497.'7” PC0038631 relates to a separate
issue but stems from a similar (possibly identical) data server issue root cause as Issue
1.!48 Owing to the age of these Peaks, comprehensive records are no longer available
and Post Office is therefore not in a position to provide detailed commentary on them.
Resolution
13.
In the short term, in the case of the Dungannon branch, the branch manager and Post
Office agreed to adjust the branch’s figures to remove the discrepancy. This occurred 2
days after the Peak was raised.
In the long term, as confirmed in PC0033128, a diagnostic fix had been implemented
across 99% of the estate by 16 May 2000.'” A full fix for Issue 1 was released in the
second half or 2000 (a more precise date is not available). On 9 October 2000, the Peak
notes that no further instances of Issue 1 have been detected since the fix was
implemented. The issues experienced by the branches referred to in PC0045847 and
PC0043811 were addressed by the fix.
Branch impact
POL00026925
POL00026925
15. Of itself, Issue 1 would not affect branch accounts; there was no issue with the
underlying transaction data and, if the office snapshot was re-run, it would very likely
provide the correct information, because the data reading issue was temporary. However,
if the branch ran the office snapshot, got an inaccurate report and then rolled over
(making good any discrepancies in the process), then the shortfall would have an impact
on branch accounts.
16. Further, hardware issues such as that which caused Issue 1 would generate a “systems
event” which is visible to Fujitsu’s SMC (2™ line support team) who monitor all serious
IHS 4F/29},
1746 1F/66}.
1747 §F/76.1}.
148 £F/26}.
1749 4F/29},
448
Al8/448
17.
18.
19.
POL00026925
POL00026925
system events and raise these with SSC. This is a means by which Fujitsu can identify
affected branches.
Given that Dungannon’s balance figures were manually adjusted between the branch
manager and Post Office, there was no lasting impact on branch accounts caused by Issue
1.
PC0033128 is likely to have been created during the initial rollout of Horizon to
branches, with pilots having been run since approximately September 1996. The Peak
notes that “Certainly, with the current system, a missing Payments node now would not
go undetected”. This is because Fujitsu made improvements to counter error handling
based on their experiences — including their experience of issues such as Issue 1 — of the
testing cycle of early Horizon.
The age of Issue I also means that comprehensive records are not available and Post
Office is therefore not in a position to provide a detailed commentary in relation to every
Peak referred to by Mr Coyne.
Tssue 2
20.
22.
As part of the changes to support IMPACT (the move from weekly cash account to
monthly branch trading statements), some optimisations were introduced to the data
server to reduce the number of times that the messagestore was scanned to pick up
transactions during balancing. A Riposte mechanism known as “Notifications” was used
to add new transactions to the existing totals as further transactions were generated
during the balancing process (rather than rebuilding the data server tree of transactions
from scratch).
Peak PC0121925 relates to an issue which was initially raised by Post Office in respect
of one of its test branches.'7*° A test branch is operated by Post Office on a Fujitsu test
rig; it is not a live branch and therefore any issue in a test branch has no impact on live
branches.
The test branch experienced a gain of £45.05 following a cash declaration and rolling
into branch trading. Initially, Escher were unable to replicate this scenario and so no
1780 £F/275,1}.
449
Al8/449
23.
24,
further action could be taken. Subsequently, as demonstrated in PC0123319!75" (a clone
of PC0121925), Fujitsu were able to replicate the scenario and implement a fix on 5
September 2005. The fix was implemented before any branches had switched to use the
new branch trading code, meaning that the issue in PCO121925 could not have impacted
any live branches.
PC0132133'”* relates to an issue in which the notification mechanism referred to in
PC0121925 was accidentally switched off.!7> Mr Coyne notes at para 3.112 that this
Subpostmaster “did not suffer an actual discrepancy” and that “a software bug fix was
subsequently implemented”.‘** Additional diagnostics were also added in order to trap
any other manifestations of failures in the notification mechanism. It is therefore not
disputed that this issue did not impact branch accounts.
PC0144386'7*> is a manifestation of the same issue as in PCO132133.'%° As noted in
the Peak, the Subpostmaster noticed the issue, reported it and the discrepancy was
cleared, so there was no long term impact on branch accounts.
Bug 11: Girobank Discrepancies
The key documents:
(1) Issue 1: KEL mwright531p;'757 PC0044232;!75 PC00S0418;!7 PCO050861;!7
PC0052804;'7°! PC0053975!"” and PC0054846.'7°°
POL00026925
POL00026925
1751 (F/288.1}.
1752 (F/332}.
1783 (F/275,1}.
1784 ()2/4,1/47}.
1755 (F/410}.
1786 (F/332}.
1757 This KEL has been deleted and is irretrievable.
1758 (F/25},
1799 (F/36}.
1700 (F/38}.
1761 (F/55},
1762 (F/60}.
1763 (F/64}.
450
AI8/450
POL00026925
POL00026925
(2) Issue 2: PC0044232,'74
(3) Issue 3: PC0052575!7% and PC0052704.17°°
(4) Issue 4: PC0068633.177
(5) Issue 5: PC0073855'7 and PC0075312.17°
(6) Issue 6: PC0076065.'7”
(7) Coyne 2, paras 3.119 — 3.128.177"
(8) Second Joint Statement.!7”
Summary:
2. Mr Coyne states that Bug 11: Girobank is a bug with lasting financial impact. Post Office
submits that there is no evidence of any financial impact on branch accounts, let alone a
lasting impact.
3. This was another example of an issue Mr Coyne drew attention to which arose early in the
life of Horizon:
Q. So again very early days in Legacy Horizon?
A. Yes.1773
Nature of this issue
4. Mr Coyne has drawn together six distinct issues under the heading “Girobank
discrepancies”. Each issue is summarised below.
1764 (F/25}.
1765 (F/49.1}.
1766 (F/52}.
1767 F/O}.
1768 (F/112}.
1769 SE/LI4},
177 (E/L18}.
177. ()2/4.1/48} to {D2/4.1/51}
17 ()1/2/9} to {D1/2/10}.
178 {May17/39:9} to {Day17/39:10}.
451
Al8/451
POL00026925
POL00026925
Issue 1
“Giros”
5. There are several different types of product referred to as “giros”. In this issue, it refers to
giro payments. This is where a customer presents the branch with a giro (that looks like a
cheque and is sometimes referred to as a docket or voucher) and exchanges it for cash.
Giros could be issued for several reasons but were commonly used for benefit payments in
the early 2000s.
6. In simple terms, the process for giros was that: i) the branch took the giro from the customer
and entered it on Horizon; ii) the branch gave the customer the cash; iii) at the end of each
day, the branch sent the paper giro to Girobank; and iv) Girobank compared the paper giros
for each day to the entries on Horizon.
7. The dispatch of the giros from the branch to Girobank could occur before the end of the
trading day; for example, because the post was collected prior to the closing time of the
branch. This meant that additional giros could be taken after the day’s giros were
dispatched. As a result, branches had to “cut-off” the day at some point before the giros
were dispatched in order to notionally demarcate the start of one day and the beginning of
the next. Therefore, for example, a Wednesday dispatch of giros may contain some giros
from Tuesday evening.
8. When Girobank compared the paper giros with the Horizon entries, Girobank would
inform Post Office if there was a difference. If the relevant paper giros could not be found
in order to address the discrepancy, Girobank would not pay Post Office and that could
result in a transaction correction (formerly an error notice) being issued to the branch.
Peaks referred to by Mr Coyne
9. The following Peaks, which were raised in 2000, are examples of Issue 1. Due to the age
of these Peaks, comprehensive records are not available and Post Office is therefore not in
a position to provide detailed commentary.
10. In PC0044232, being the main Peak referred to by Mr Coyne in relation to Issue 1,
Girobank had noticed that there was a £505.72 discrepancy between a branch’s cash
account and its daily reports:
AI8/452
POL00026925
POL00026925
(1) This was a known issue dealt with by KEL MWright53lp. This KEL is now deleted
and irretrievable, but details about it can be gleaned from its associated Peaks.
(2) The issue arose when a giro transaction was entered and then reversed, with the
reversal being entered after the report cut-off time. This meant that the transaction
was included on that day’s daily report (which was sent to Girobank), but the reversal
was not included until the following day’s report. Girobank would have been
expecting to receive the paper giro for the transaction, whilst the branch would not
have submitted it because they thought they had reversed the transaction.
(3) This led to an error notice being issued on the mistaken basis that the branch had a
discrepancy.
(4) The fact that the reversal, performed after the daily cut-off, did not show on that day’s
report reflects the intended operation of Horizon. Subpostmasters were instructed that,
if a reversal is carried out to giro transactions after cut-off, a manual summary will
need to be produced for Girobank. Issue I is therefore not a “bug”.
(5) Rather, Issue I relates to reporting. The underlying data is correct and the branch’s
accounts would have been correct at the end of the trading period, once the reversal
had been recognised (at the time of this Peak, the trading period was weekly). Mr
Coyne appeared to accept this in cross-examination.'”* In this particular case, the
only possible impact would be if the branch had accepted the error notice received
because of the reporting issue.
(6) This also means that a “fix” would not have been deemed necessary; a KEL was
already in place and the issue is one of reporting rather than any issue with the
underlying data.
(7) Mr Coyne accepted in cross-examination that the detection and investigation of Issue
1 and Issue 2 in PC0044232 demonstrated the good and effective operation of
robustness countermeasures in Horizon (RDS and MID)!” and that PC0044232 was
174 (Dayl7/51:6} to {Day17/51:25}.
15 (Mayl7/52:1} to {Day17/56:1} and {Day17/63:1} to {Day17/64:1}.
453
AI8/453
not evidence of a transaction correction or error notice being issued to the
Subpostmaster in such a way as to subject him to a risk of loss.'””°
11. Mr Coyne admitted that this Peak is in fact an example of the good and effective operation
of countermeasures:
Q. There was a reconciliation problem and it went straight to the SSC, and the SSC
worked out what the cause of the problem was. Do you accept that what this PEAK
shows is not a threat to the robustness of Horizon, actually it shows the operation, the
good and effective operation of countermeasures to possible threats to Horizon, do
you see?
A. Ido, but it is a little obvious that what we are looking at is PEAKs, so we will only
POL00026925
POL00026925
see the times that somebody calls in and it is recorded. We won't have records if
somebody hasn't reported it.!”””
12. The issue in PC005S0418 was thought to be the same issue — the branch’s largest
discrepancy was £323.32. However, due to the length of time it took for the issue to reach
SSC, the branch’s messagestore had been archived — the Subpostmaster raised the call on
29 June and the call was sent to PINICL on 17 July. The Subpostmaster was told that they
would need to provide further information (such as the Transaction ID) to allow Fujitsu to
investigate. The Subpostmaster does not appear to have pursued this. The Peak notes that
Girobank were going to send an error notice, but due to the age of this Peak the relevant
records are not available and Post Office is not in a position to provide detailed
commentary.
13. The issue in PC0050861 was thought to be the same issue — the branch experienced
discrepancies of £59.33 and £40.00. The Peak notes that the branch received an error
notice. The Peak also states that the fact that a reversal does not show on a daily report if
it is performed after cut-off is “how the system is supposed to work”.
14. In PC0052804, the Subpostmaster reported a £55 Girobank discrepancy, based on an issue
which was ultimately diagnosed as the same known mwright531p issue. The branch’s
weekly reports were correct. No error notice would have been needed as the Subpostmaster
had ultimately just misread their report.
1778 {Day 7/65:6} to {Day17/65:23}.
17 {ayl7/53:4} to {Day17/53:14}.
454
AI8/454
POL00026925
POL00026925
15. In PC0053975, the Subpostmaster (FAD 11413) reported a £40 difference between their
daily report and CAP (having been notified of it by Girobank). This was diagnosed as the
known mwright531p issue and the weekly reports were correct.
16. In PC0054846, the Subpostmaster (FAD 212113) reported a £99.13 Girobank discrepancy.
This was diagnosed as the known mwright53 Ip issue and the weekly reports were correct.
Tssue 2
17. Mr Coyne refers to a “secondary problem”! in relation to PC0044232 (this issue was
picked up by Fujitsu in the course of investigating Issue 1). Issue 2 was that an £81 giro
deposit was included on two consecutive daily reports. This is because the transaction was
entered onto Horizon in a precise (and very small) window of time between two system
calls being undertaken, resulting in a duplication. The overall branch position would still
have been correct, but the daily reports to Girobank may have been wrong. If they were
(i.e. if the same transaction was included on two consecutive daily reports), it is expected
that this would have been spotted and a TC would not have been issued to the branch.
18. Fujitsu’s fix for the issue was actioned on 30 May 2000 and so would have been released
between that date and 23 June 2000.
Issue 3
19. Issue 3 applies to two Peaks listed in the table at para 3.123'7”. These Peaks were raised
in 2000 and so comprehensive records are not available.
20. In PC0052575, the Subpostmaster reported £20 and £628.25 discrepancies between the
daily giro report and the office daily report. On 31 August, the issue was diagnosed as
arising out of the use of a shared stock unit. There is a window of time between a user
printing and cutting-off a report. If another user was to perform a transaction during that
window, that transaction may not show on the report. The issue was already due to be fixed
in a future release. Mr Coyne accepted in cross-examination that:
20.1. this issue was not a bug which creates discrepancies in branch accounts!”*°;
1778 ()2/4.1/49}.
179 41)2/4.1/49} to {D2/4.1/50}.
178 {Day17/60:8} to {Day17/60:13}.
455
Al8/455
POL00026925
POL00026925
20.2. in fact, the Peak demonstrates that “the Horizon system and all the support
operations surrounding it and supporting it operated well in identifving if there
were any discrepancies and checking to see if there were any problems created by
those discrepancies”'”*';
20.3. the Peak is a good example of “how these countermeasures increase rather than
detract from the robustness of Horizon”'”*?; and
20.4. the Peak was not evidence of a transaction correction or error notice being issued
to the Subpostmaster in such a way as to subject him to a risk of loss.!”*
21. Mr Coyne admitted that PC0052575 indicated a problem with a report, and not the
transaction log.'”** Mr Coyne also agreed that the issue is not one with the software that
generates branch accounts.'”° Mr Coyne went on to agree that this Peak does not relate to
a discrepancy in branch accounts.'78°
22. In PC0052704, the relief Subpostmaster reported that two transactions were missing from
his 3-week balance (total £116.08). The Subpostmaster attempted to re-enter the two
transactions before cut-off, resulting in the same issue as in PC0052575. Ultimately, when
the branch re-checked the figures there were no outstanding issues. Using the wrong report
(human error) caused the Subpostmaster to re-enter two transactions he believed were
missing and it’s likely that he reversed the later transactions.
23. It was put to Mr Coyne that:
Q. That, I would suggest, is not the result of a bug in Horizon.
A. So if that was right, why would Martin Harvey, when he wrote that at 9.30 on 23rd
August, choose to reference the KEL for that particular defect? It would make no
sense if he didn’t consider that that was part of his consideration.!”"”
1781 (Day17/61:11} to {Day17/61:19}.
178 (Dayl7/62:19} to {Day17/63:9}.
1 (Dayl7/65:6} to {Day17/65:23}.
17 (Dayl7/58:20} to {Day17/59:6}.
"5 (Dayl7/60:2} to {Day} 7/60:7}.
178 {Day17/62:10} to {Day17/62:12}.
18 (Dayl7/74:15} to {Dayl7/74:21}.
456
AI8/456
POL00026925
POL00026925
24. There are many reasons why a KEL may be referred to in the context of a Peak. It may be
to signal a particular defect is that set out in the KEL. It can also be so that others can
identify that this Peak is an example where, although symptoms may look similar, the issue
is not the same. The Peak does not state that this incident is a manifestation of the issues
set out in the KEL.
25. Issue 3 would therefore have no direct financial impact on branches; it affected reporting
only and not data.
Issue 4
26. Issue 4 applies to PC0068633 (referred to at para 3.124'7*). This Peak was raised in 2001
and so comprehensive records are not available.
27. In that Peak, the Subpostmaster reported that his cash account showed two giro deposits of
£1,503 but that his reports showed only one. The Subpostmaster received an error notice
from Girobank which cleared the error, but he raised the issue because he believed that an
error in Horizon was duplicating the transaction.
28. The issue was caused by a cut-off being performed on one counter despite an attempt to
print a transaction failing on another counter. This resulted in the cut-off report including
the transaction that had failed to print.
29. A fix was actioned by Fujitsu on 18 December 2001.
30, Issue 4 only occurred in a very specific set of circumstances and would have had no direct
financial impact on accounts; it merely had the effect that a transaction was missing from
the reports.
Issue 5
31. Issue 5 applies to PC0073855 and PC0075312 (referred to in the table at para
3.127'78), These Peaks were raised in 2002 and so comprehensive records are not
available.
1788 11)9/4.1/50}.
178 ()2/4.1/50} to {D2/4.1/51}.
457
AI8/457
Issue 6
POL00026925
POL00026925
. In PC0073855, a Subpostmaster (FAD 233618) reported that his office snapshot
figures were double the figures on the balance snapshot (£6.76 discrepancy).
Fujitsu were unable to replicate the issue and were therefore unable to issue a
specific fix. However, a new version of the component was released with extra
tracing code so that if the issue re-occurred, Fujitsu would be able to gather more
evidence.
. The issue affected the office snapshot but not the balance report or the cash account
figures. This means that the Subpostmaster was provided with an inaccurate report
but that all the correct data was still available and would not have affected the
branch when balancing.
. In PC0075312, a Subpostmaster (FAD 159546) raised an issue with printing her
giro deposits. The issue was identified as being caused by the same root issue as
PC0073855 which was already with the development team.
. Mr Coyne clarified in cross-examination that the Peaks referred to in the table at
para 3.127!” were not intended to be presented as Peaks which, in his opinion,
constitute bugs within Horizon that impact branch accounts; rather, they are
merely Peaks which refer to KEL AChambers4410R.17"
32. Issue 6 applies to PC0076065 (referred to in the table at para 3.127!””). In that Peak,
the Subpostmaster (FAD 179309) reported that two giro deposits (£11 and £24) made
the previous day were not showing on the previous or that day’s reports. Fujitsu
discovered that the Subpostmaster had produced two reports. The first report was
produced before the transactions were committed (and therefore did not include them).
The second, subsequent report did show the transactions. It was assumed that the
Subpostmaster had been mistakenly looking at the first report. This means that there
was no actual discrepancy and the issue was user error.
179 (1)2/4.1/50} to (D2/4.1/51}
1791 (Dayl7/44:6} to {Day17/45:8}.
17% ()2/4.1/50} to {D2/4.1/51}.
458
Al8/458
we
a
34,
35.
POL00026925
POL00026925
Mr Coyne was initially evasive as to whether Peak PC0076065 was or was not an
example of a discrepancy relating to the Girobank bug. Mr Coyne’s report, upon an
ordinary reading, clearly suggested that the Peaks set out in the table above paragraph
3.128 contained examples of bugs causing financial discrepancies in branch accounts.
Mr Coyne confirmed that the table that contains Peak PC0076065, along with another
table, sets out examples of bugs causing branch discrepancy:
Q. Sorry, if you look at 3.127 below, I'ma bit confused: “Further associated PEAKs
that reference [that KEL] are provided in the table below.” I would like to ask you
about those. It is the PEAK on page 51. It is PC0076065 at {F/118/l}. If we could
look at that please. I should, for completeness, actually read out what you say
immediately below that table. This is in 3.128, we don't need to go back to it on the
transcript -- on the machine: “The above PEAKs related to Girobank discrepancies
are clear examples of bugs within Horizon that affect branch accounts by way of a
financial discrepancy and illustrate, by their interlinking natures, the complexities of
the PEAKs/KELs.” So what you are saying there is that the PEAKs referred to in that
table are clear examples of bugs in Horizon that affect -- that create financial
discrepancies in branch accounts, correct?
A. Yes, I’m referring to the PEAKs within this section, not just that particular table.
If you read up, there is a table there and there is a table before it.”
Despite saying this when asked specifically about Peak PC0076065!74 Mr Coyne was
evasive as to whether or not it was an example of a bug causing a financial discrepancy
in branch accounts. It should have been a simple yes or no response:
Q. What I’m seeking to elicit from you, Mr Coyne, and I think you have confirmed it,
is that it is your contention, your judgment, your expert opinion that {F/118/I} is a
clear example of a bug which has caused a financial discrepancy in a branch
account?
A. The text in my report is: “Giro deposit cut off... Branch unknown.” So a number
of other ones in the table actually list the discrepancy. That one doesn’t list the
discrepancy next to it!”
Mr Coyne was asked again:
173 May17/40:19} to {Day17/41:14}.
1794 FE/LI8}.
175 May17/41:15} to (Day17/41:24}.
459
Al8/459
36.
37,
Q. So you accept then that {F/118/1} isn’t a bug which creates discrepancies in
branch accounts?
A. No, it is linked by way of KEL; the Anne Chambers --!”°°
Yet again Mr Coyne was asked:
Q. We can save some time then. You accept that {F/118/1}, or the PEAK that is
referred to there, isn't a bug at all, don’t you?
A, I have described it here as a cut-off issue, branch unknown. So it likely isn’t a --
1797
1798
Mr Coyne accepted in cross-examination that Issue 6 is not a bug,’”’* much less a bug
which creates discrepancies in branch accounts.!”°
Conclusion
38.
39.
40.
Mr Coyne comments that the “above PEAKs related to Girobank discrepancies are
clear examples of bugs within Horizon that affect branch accounts by way of a
financial discrepancy and illustrate, by their interlinking natures, the complexities of
the PEAKs/KELs.”'8°°
That analysis is incorrect. None of the Peaks referred to by Mr Coyne demonstrate a
direct financial impact on branches; in most cases this is because the issue affects
reporting whilst the underlying data remains unaffected. Issue I reflects Horizon
working as intended (and is therefore not a bug) and Issue 6 is pure user error. Mr
Coyne accepted in cross-examination that a number of the Peaks referred to did not
concern issues which had impact on branch accounts.
Many of the Peaks referred to by Mr Coyne were raised in 2000. As Mr Coyne
accepted in cross-examination,'*”' this demonstrates that the issues experienced in the
POL00026925
POL00026925
1796 (Day17/44:15} to {Day17/44:17}.
177 ‘Day17/45:9} to {Day17/45:13}.
1798 May17/46:25} to {Day]7/47:8}.
1799 (Day17/44:15} to {Day17/44:17}.
41800 ()2/4.1/51}.
1801 {Day17/39:5} to {Day17/39:16}.
460
A/8/460
POL00026925
POL00026925
very early days of Horizon had not manifested themselves in any Peaks or KELs since
then.
Bug 12: Counter Replacement Causing One Sided Transactions
1. The key documents:
1.1. Peaks: PC0058528;!8 PC0052823;'8°° PC0071836;'8 PC0133822'*° and
PCO153851.'8°
1.2. KEL JBallantyneS328R.'8°”
1.3. Coyne 2, paras 3.129 — 3.131.188
1.4. Second Joint Statement.'*”
Summary
2. Mr Coyne states that Bug 12: Counter replacement is a bug with lasting financial
impact. Post Office submits that any discrepancy would not be lasting.
Nature of this issue
3. PC0058528 relates to an issue in November 2000 in which the branch (FAD 253329)
was a single counter branch whose counter’s hard drive had been replaced due to a
hardware issue. That replacement caused two messages relating to an OBCS
transaction to be overwritten, resulting in a receipts and payments mismatch; a
transaction with a value of £167.12 was not added to the cash account.
802 (F/77.1}.
1803. (F/54},
800 (F/107}.
1805 (F/337}.
1806 (F/438}.
1807 (F/42 1}.
1808 (1)2/4,1/51} to (D2/4.1/52}.
1809 (D)1/2/10} to {D1/2/11}.
461
AI8/461
6.
When a counter is replaced, it builds its messagestore by replicating with its
neighbours in “recovery mode”. The neighbours it has depends on the office size and
node number.
For a single counter office, the neighbours are the correspondence server in the
datacentre and the mirror disk (the second hard drive in the same counter).
For node I at a multi-counter office, the neighbours are the correspondence server and
all other nodes at the office.
For any node number higher than I at a multi-counter office, it is all other nodes in the
office (known as slaves).
A replacement counter will come out of recovery mode when it believes it has
successfully replicated all relevant messages from its neighbours. In this case, the
replacement counter came out of recovery mode early, before it had replicated all
messages from its neighbour.
The replacement counter started writing messages from the point at which it believed it
had replicated all relevant messages from its neighbour. This meant that it used
message IDs that had been used for messages that had not been replicated from its
neighbour and this prevented the “missing” messages from being replicated later on
(because that would have created duplicate message IDs). The missing message was
therefore “overwritten” by the replacement counter.
The issue arose in cases of counter replacements where the new counter was not
connected to all of its configured neighbours while rebuilding. This may have been
because the branch infrastructure was not complete (eg not all neighbouring counters
are online, multiple swaps or a counter increase/decrease occurring) or the engineer did
not connect the system properly. Engineering instructions were constantly being
refined to avoid instances of this happening.
Resolution
The branch reported a receipts and payments mismatch of £167.12. This discrepancy
would have been flagged to the Subpostmaster on the cash account when he attempted
to roll over.
POL00026925
POL00026925
AI8/462
14,
POL00026925
POL00026925
The SSC inspected the Riposte mirror messagestore, compared the relevant messages
in the neighbour’s messagestore with those in the node reporting the problem, and
thereby identified and retrieved the specific messages which had been overwritten.
This is detailed in KEL JBallantyne5328R and does not involve inserting a message
into Riposte.
Information of the overwritten messages was passed to MSU who created a BIMS
report for Post Office and an error notice would have been issued to hold the branch
harmless thereafter.
The long-term fix for the issue is detailed in PC0052823. The fix involved enforcing a
minimum number of local neighbours for replication and then to slowly lower this
number over time. A further change was made to stop Riposte writing messages as it
came online.
Other Peaks associated with JBallantyne5328R
15.
Mr Coyne states that he has performed a search of all Peaks disclosed for references to
KEL JBallantyne5328R and that that search returned “approximately 88 further
PEAKs”.!*° Mr Coyne has “randomly selected” three of these Peaks for analysis:
15.1. PC0071836:1s11 this is an example of the same issue as JBallantyneS328R. This
Subpostmaster had a receipts and payments mismatch of £3.27 as a result of three
overwritten messages following a replacement of the branch’s single counter. The
same fix was applied following KEL JBallantyne5328R. Due to the age of this
issue, comprehensive records are no longer available and therefore Post Office is
not in a position to provide detailed commentary.
15.2. PC0133822:'*! this is not the same issue as JBallantyne5328R but is related.
The branch had two counters removed, leaving it as a single counter branch.
However, the counter did not have a mirror disk; the mirror disk was a second
hard disk within a single counter that provided a replication neighbour for the
810 ()2/4.1/51}
1811 (F/107}.
812 (F/337}.
463
AI8/463
POL00026925
POL00026925
main hard disk messagestore. This meant that the branch would have no
replication of data if it was not connected to the datacentre and six messages
on the counter had not been replicated to the data centre. The messages were
extracted and sent to MSU for a BIMS report!*! to be raised. An error notice
would have followed the BIMS report and so there would have been no lasting
impact on branch accounts.
15.3. PC0153851:'*"4 this is not the same issue as JBallantyne5328R but it does involve
areceipts and payments mismatch. Riposte failed to index four messages resulting
in some items being missing from the receipts side of the balance report. The Peak
notes that the branch did not experience a discrepancy as a result because this was
a reporting issue only; indexes are not used when replicating data and so cash/stock
were unaffected. The Subpostmaster was happy with the explanation provided
and the call was closed. No BIMS report was required in respect of this Peak.
Branch impact
16.
18.
In PC0058528,'*!* the Subpostmaster noticed the receipts and payments mismatch and
reported the issue; in any event, the issue would have been flagged on the cash account
when he attempted to roll over. MSU issued a BIMS report to Post Office which
would have resulted in an error notice being issued to the branch; there would be no
lasting impact on the branch’s accounts.
In respect of other branches, the issue would not necessarily create an impact —
transient or otherwise — on branch accounts; the effect of the issue would depend upon
the precise messages that were overwritten and the reason why the data did not
replicate correctly in the first instance.
If all the un-replicated events were overwritten then it would be detected through a
payment/receipts mismatch or a balance issue. If not, all the messages were
overwritten, and an old message for the counter existed elsewhere, then events would
be created by Riposte and detected by the SMC (self-originating message detected).
813 (F/680.1}
1814 1F/438}.
1815 $F/77.1}.
464
AI8/464
19.
This issue was limited to Legacy Horizon; data has not been held on counters since the
introduction of Horizon Online.
Inserting items within the Horizon messagestore
20.
Mr Coyne’s conclusion in relation to this issue is that “since Fujitsu support had the
facility to insert items within the Horizon message store, without process audit...the
effects of one-sided transactions and their applied corrective fixes is clearly larger than
the “one balancing transaction” as suggested by Post Office”.'*!© Mr Coyne has
arrived at that conclusion on the basis of a misunderstanding of the fix for this issue
and the different Horizon systems. To be clear:
(1) _ Fujitsu did not insert messages into branch systems in order to fix this issue.
(2) Fujitsu identifies the source of the issue and passed the information to MSU, who
issued a BIMS report to Post Office.
POL00026925
POL00026925
1816 ()2/4,1/52}.
465
AI8/465
Bug 13 — Withdrawn stock discrepancies
1
The key documents:
121.1 Peaks: PC0207834;'*!7 PCO208918;'*!8 PC0209602'*"” and PCO211932.'*°
121.2 KEL PothapragadaC4913L.'°!
121.3 Coyne 2, paras 3.132 — 3.139.'8??
121.4 Second Joint Statement.'***
Summary
>
Mr Coyne states that Bug 13: Withdrawn Stock Discrepancies is a bug with lasting
financial impact. Post Office submits that this is not a bug at all.
Post Office’s withdrawal of products
POL00026925
POL00026925
3. Post Office add and withdraw products from sale for a variety of reasons. Before a
product is withdrawn, this is communicated to branches by way of Branch Focus which
outlines precisely what branches need to do in relation to the withdrawal. When a
product is withdrawn from sale, any corresponding stock in the branch will either need
to be destroyed or returned to Post Office within a certain period of time.
4, When a product is withdrawn from sale the various corresponding processes within
Horizon will be withdrawn. This is done by an update to reference data and not a
change to the core code in the system.
1817 (F/765}.
1818 (F/783},
1819 (£/789}.
1820 £F/§30.1}.
1821 ££ /678}.
1822 (1)2/4,1/52} to {D2/4.1/54}.
823 1/2/11} to {D1/2/12}.
466
AI8/466
POL00026925
POL00026925
Nature of this issue
6.
Peak PC0207834 relates to Post Office withdrawing a £5 saving stamp from sale.'5*4
Subpostmasters would have been instructed to rem out any excess stock of the stamps
and then return them to Post Office within a certain period of time before the processes
for transacting the stamps were withdrawn on Horizon.
However, the Subpostmaster in this case returned the stamps (with a total value of
£685) to Post Office without first remming them out. This aspect of PC0207834 was
pure user error.'*?
The branch conducted a trading period balance on 17 November 2010 which,
necessarily, would have involved the Subpostmaster undertaking either a stock
declaration or a stock adjustment to reflect that the stock of stamps held was actually
zero. The relevance of the distinction between the two processes is explained below.
The branch declared a £685 shortfall. The lack of a rem out meant that Horizon
thought that the branch was still holding the stamps and therefore when the
Subpostmaster either declared or adjusted the stock of stamps to zero without a
corresponding gain in cash, a shortfall was generated. The Subpostmaster elected to
make good the shortfall and a credit transaction correction was subsequently issued for
£685 to rectify the issue.
However, a bug in Horizon caused the £685 of stamps to be subsequently re-introduced
into the branch’s accounts on two occasions. By this point, Horizon was showing that
the branch was holding £1,370 of the stamps. The Subpostmaster noticed this, adjusted
or declared the stock as zero, and reported the issue. At that point, PC0207834 was
created.'*?6
The Peak shows that the Subpostmaster was also experiencing some other, unrelated
trading issues (such as a gain of £537.49 in December 2010). These issues are
unrelated to the withdrawn stock issue except to the extent that it is possible that they
1824 (F/765}.
1825. (F/765}.
1826 (F/765}.
467
AI8/467
i.
caused him not to notice the first instance of the withdrawn stock being re-introduced
at first.
This issue relates only to HNG-X and not also to Legacy Horizon.
Resolution
12.
14,
This issue was an example of a known issue covered by KEL PothapragadaC4913L.
That KEL concerned foreign currency, but the same issue could occur in other products
(such as the stamps in PC0207834).
PothapragadaC4913L contains a workaround for the issue. The Subpostmaster must
undertake a stock declaration (not a stock adjustment) and declare the stock of the
withdrawn product as zero. This prevents the withdrawn stock from being
subsequently re-introduced. Performing a stock adjustment would not solve the issue.
Stock declarations and stock adjustments are different processes which produce the
same end result (except for the purposes of the workaround). However, stock
adjustments are quicker and easier to perform and Subpostmasters perform them much
more often than stock declarations. In PC0207834, it appears that the Subpostmaster
must have initially performed a stock adjustment, meaning that the stamps issue would
not be resolved, and that it was only upon NBSC talking him through the workaround
(whereby it would be clear that a stock declaration was required) that the issue was
resolved.!*”’ This is not explicit from the documents available but is thought to be the
likely explanation for the Subpostmaster’s issue in resolving the problem.
NBSC assisted the Subpostmaster in implementing the workaround by 2 March 2011.
On I April 2011, a reference data fix was issued to address the root cause (recorded in
PC0208918).'*?5 PCO209602'*”? was subsequently opened by Fujitsu in order to
investigate the possibility of correcting the relevant Horizon code in order to prevent
the issue occurring in relation to other products; that fix was issued to the live estate on
21 September 2011 (PC0211932).'**
POL00026925
POL00026925
1827 (F/765}.
1828 (F/783}.
1829 (F/789}.
890 (F/830.1}.
468
Al8/468
POL00026925
POL00026925
Branch impact
16. The original £685 shortfall was caused by user error by virtue of the Subpostmaster
failing to rem out the stamps and then, most likely, undertaking a stock adjustment
instead of a stock declaration. A branch who rems out the withdrawn stock as
instructed would experience no financial impact. Further, a branch who does not do so,
but who subsequently performs a stock declaration as instructed, would not experience
the issue of the withdrawn stock being re-introduced.
17. The second issue was identified by the Subpostmaster and resolved promptly via a
workaround.
Conclusion
18. For the reasons set out above, this was not a bug at all. The issues set out in the Peaks
arose as a result of human error and not as a result of any bug, error or defect in
Horizon.
Bug 14 — Bureau discrepancies
1. Mr Coyne draws together two distinct issues under the heading “bureau discrepancies”.
The key documents
2. Issue 1:
2.1 Peaks: PC0261541'**! and PCO261710'S*,
2.2 Coyne 2, paras 3.140 — 3.143.'8%
2.3. Second Joint Statement.'**
3. Issue 2:
851 (F/1679}.
882 (F/1681}.
1833. ()2/4,1/54}.
8M 1/2/12} to {D1/2/13}.
469
Al8/469
3.1 Peaks: PC0265443!*"5.
3.2 Coyne 2, paras 3.145 — 3.146.'8*°
3.3. Second Joint Statement.'**”
Summary
4.
Bug 14: Bureau Discrepancies is a bug with the potential for lasting financial impact.
There are two distinct issues which fall under this heading. With regards to the first issue
the branch was made good and a fix was implemented. The second issue was not a bug
in Horizon nor an issue which could have impacted branch accounts; it created what was
essentially a cash flow problem for the branch.
Issue 1
Nature of the issue
PC0261541 relates to an issue involving a branch (FAD 2078430) attempting to pre-
order two currencies.'*** The Subpostmaster tried to pre-order £1,000.07 in Indonesian
rupiah and £204.59 in Singaporean dollars for a customer. The rupiah order was created,
but there was a network timeout at the point when the Subpostmaster tried to add the
dollar order.
When the system re-connected, a warning message suggested that the second order may
not have been placed, but the basket and transaction log were showing both orders. The
Subpostmaster attempted to cancel the whole order, but the cancellation only worked for
the rupiah order, leaving the dollar order of £204.59 in the branch accounts.
This is because, at the time of the Peak, multiple currency orders were processed as
multiple transactions. The rupiah order was added first at the counter and then sent to
the BAL, and this caused the order ID PBX1048411 to be created. The network timeout
occurred at such a time that the dollar order had been added to the counter stack but it
had not yet reached the BAL. This meant that it did not become associated with order
POL00026925
POL00026925
895 (F/1722}.
1836 ()2/4.1/55} to (D2/4.1/56}
897 {1/2/12} to {D1/2/13}.
838 (F/1679}.
470
AI8/470
POL00026925
POL00026925
PBX1048411 and so, when the Subpostmaster attempted to cancel the whole transaction
(PBX 1048411), the dollar order was not cancelled. This would then create a shortfall as
Horizon will have expected the Subpostmaster to have taken a payment from the
customer of £204.59.
This issue could only occur because of the very specific circumstances of PC0261541
including the occurrence of a network timeout at a very precise moment in the
transaction. I**
Resolution
10.
The Peak’s priority was raised by Fujitsu because of the possible financial impact of the
issue.
The Peak notes that the issue was referred to Post Office in order for them to “decide
what reconciliation or transaction correction is required to balance” at the branch. Post
Office have confirmed that a transaction correction was issued to the branch on 2
November 2017 and accepted by the branch on 7 November 2017 — this is not apparent
from the Peak itself.
PC0261541'*4° was cloned and the technical investigation into the root cause continued
under PC0261710.'**! Ultimately, the issue was fixed in November 2017 via a change
to the AP-ADC script which had the effect of simplifying multiple currency orders to be
processed as a single transaction, meaning that the window of time in which the network
timeout occurred in PC0261541 no longer existed.
Issue 2
Nature of the issue
12.
PC0265443 relates to an issue at branch FAD 091912.'** Post Office’s internal cash
management team’s system (POLSAP) indicated that the branch was holding €4,500 and
$1,000 more than the branch was declaring it held. This meant that, from the cash
management team’s perspective, the branch should have been holding sufficient cash to
1889 1F/1679}.
1810 (F/1679}
1841 (F/1681}.
182 (F/1722}.
471
Al8/471
conduct transactions, and so were reluctant to provide more cash to the branch. This
resulted in a cash flow issue for the branch who were sometimes unable to serve
customers as a result.
Resolution
13.
14.
The Peak is long and complex and it demonstrates that the issue was extensively
investigated by Post Office, ATOS, Accenture and Fujitsu, since the issue was thought
to be occurring somewhere between the branch, Horizon and POLSAP. The conclusion
of Fujitsu and Accenture’s investigations (over approximately six months) was that the
issue was not caused by any issue within their respective systems. Further, Post Office
sent a trainer to the branch to verify the cash position and found no discrepancies; the
amount of cash physically held in branch was the same as the amount showing on
Horizon (but not POLSAP).
The root cause of the issue does not appear to have been determined. As it did not appear
to be a case of user error, it appears that Post Office wrote off the discrepancy in POLSAP
at no cost to the Subpostmaster to bring that system’s figures into line with those in
Horizon.
Branch impact
15. This issue was not a bug in Horizon nor an issue which could have impacted branch
accounts; it created what was essentially a cash flow problem for the branch, but the
branch accounts were unaffected. It appears that Post Office took the decision to write
off the discrepancy in POLSAP.
Conclusion
16. For the reasons set out above, these are two distinct issues. A fix was implemented for
the first issue and all SPMs were made good. The second issue was not a bug in Horizon.
Bug 15: Phantom Transactions
The key documents:
POL00026925
POL00026925
AI8/472
1.1 Peaks: PC0052025;'*# PC0062561;'*4 PC0065021'**> and PC0068327.'*4
1.2 Coyne 2, paras 3.148 — 3.153.'57
1.3 Js2.'8*
Summary
2.
Mr Coyne states that Bug 15: Phantom Transactions is a bug with non-lasting financial
impact. Post Office submits that this is not a bug at all. Manifestations of this alleged
bugs are either design features of Horizon or user error.
Nature of this issue
3.
Mr Coyne has drawn together three distinct issues under the heading of “phantom
transactioi
3.1 Paras 3.148 - 3.149!" relate to issue 1 (PC0065021)'** (“Issue 1”).
3.2 Paras 3.150 to 3.151'**! relate to issue 2 (PC0052025)'**? (“Issue 2”).
3.3 Para 3.152'%° relate to issue 3 (PC0052025)'8*4 (“Issue 3”).
Phantom transactions were reported early on in the life of Horizon. Mr Coyne confirmed
during cross-examination that have been no relevant Peaks in 18 years:
Q. So would I be right in thinking that whatever problems there were in 2000 and
2001 regarding phantom transactions, the PEAKs indicate that such transactions
have not raised their ugly head for the last 18 years?
POL00026925
POL00026925
1844
1845 (F/97},
1846 1F/100/1}.
1847 (1)2/4.1/56} to {D2/4.1/57}
1848 (1) 1/2/13}
1809 (1D 2/4.1/56}.
1850 {F/97}.
1851 (1)2/4.1/56}.
182 (F/48}.
1853 (1)2/4.1/57}.
1854 FF /48}.
473
AI8/473
A. Yes,1855
Tssue 1
5.
Peak PC0065021,'*** refers to multiple branches and SPMs (note the different branch
names and changes in pronoun throughout the Peak). The Peak constituted a master Peak
for recording user complaints that there may have been a phantom transaction. A number
of clear conclusions can nevertheless be drawn from this Peak: see below.
The Peak indicates the lengths that Fujitsu went to in order to resolve issues raised by
SPMs. It records that the issues reported by the Old Isleworth SPM were investigated
by Romec engineers and specialist Field Service Managers on site.!**” Indeed, on 19
June 2001, Fujitsu suspected that phantom transactions were being created by input
peripherals (eg. touchscreen, keyboard, etc.) and put in place monitoring software to
identify this.'8°°
Mr Coyne agreed that Fujitsu undertook a very thorough investigation of this incident:
Q. It is fair to say from reading the PEAK as a whole -- we don’t have time to go
through it all -- that the investigations that Fujitsu carried out were very thorough,
weren't they?
A. Yes. I think they ultimately determined they could not decide what the fault was
but the process seemed to be a reasonable process to go through."°”
On the basis of these hardware investigations Mr Coyne appears to have concluded that
the main branch (Old Isleworth, FAD111025) may have been experiencing phantom
transactions as a result of hardware issues. Mr Coyne fails to refer in his analysis at all
to Peaks PC0062561'*® and PC0068327,'**! both of which state that the issues at Old
Isleworth branch were attributable to user error.
Mr Coyne only quotes one extract from Peak PC0065021, which is a record of a Romec
engineer on site seeing a phantom transaction.
POL00026925
POL00026925
855 (Dayl7/16:19} to {Day7/16:23}.
856 (F/97},
857 (F/97/3}.
1858 (F/97/7}.
1889 (Day17/29:24} to {Day17/30:5}.
1860 (F/88,2}.
861 £F/100.1}.
474
AI8/474
POL00026925
POL00026925
10. With regard to ROMEC’s expertise Mr Coyne stated that although familiar with the
hardware, they may not be familiar with the Horizon software:
Q. And they sent engineers from Romec, Do you know what Romec’s familiarity with
the system is?
A, I would suspect that they know the hardware and the communication equipment
very well. I don’t know if they will know how Horizon as a software product would
work,
Q. That is a very fair answer. You saw where I was going and that is a very fair
answer, Mr Coyne. They might very well know how to set the system up, make sure
everything is connected and see that everything is logged on properly, but when it
comes to the internal workings of the system itself they may well not be familiar, yes?
A. Yes, 1802
11. It is not possible from the Peak to know what the ROMEC engineer saw. Indeed there
are a host of possible explanations, some of which were put to Mr Coyne:
Q. Maybe, Mr Coyne, another counter hadn't been used for 59 minutes, it had
uncompleted transactions on it, and it automatically completed them and printed a
receipt. That is possible, isn’t it?
A. That is possible.
Q. And it is possible that an engineer with his familiarity of the hardware would not
know that feature of the operating system of Horizon, so he might be surprised by
that, correct?
A. He might be surprised by that. But I think we have got to read this in context, that
the subpostmaster had already explained that they perceived that there were problems
with what they said was phantom transactions. So Post Office and/or Fujitsu would
have gone through, I would presume, their support process, would have looked at
various logs and things like that before dispatching a hardware engineer to site. So
if they suspected that it was just the simple case of a counter coming to the end of its
59 minutes of suspension and doing something automatically, I think they would have
dealt with that before dispatching an engineer.
Q. We can agree, Mr Coyne, that the SSC had experience of these things and they
had access to information that we don’t have. We can agree on that. But just focusing
on the Romec engineer visit, would you not accept that it is quite possible that the
engineer saw something which was an example of Horizon operating as it should and
was not in fact a phantom transaction at all, and he misinterpreted it because he
82 {Day17/30:6} to {Day17/30:18}.
47S
AI8/475
POL00026925
POL00026925
didn’t have the familiarity with the system that someone at Fujitsu might have? Would
you accept that that was at least possible?
A. Laccept that it was possible if you look at that visit in isolation, yes.°°
As evident from the Peak, Mr Patrick Carroll undertook the investigation for the SSC.
Mr Parker confirmed during cross-examination that Mr Carroll “was one of the senior
technicians within the SSC.”"84 Mr Coyne agreed that Mr Carroll was ina good position,
given the information available to him and his experience, to identify the likely cause of
the issues:
Q. And it is fair to assume, isn’t it, that he is in a good place, with the information
and the experience that he has, to form a judgment as to what the likely cause of these
problems are, yes?
A. Yes,1865
Mr Carroll ultimately determined that there was not a fault in Horizon. Mr Coyne agreed
that he was not in a position to say that Mr Carroll was wrong:
Q. ...My question was do you think you now, with the information you have, are ina
position to say that Mr Carroll was wrong?
A. No.!866
Peak PC0062561 shows that the issues continued at the Old Isleworth branch until 17
August 2001.!8° Although there are references to equipment being swapped out, the 17
August entry states that Fujitsu had “exhaust/ed]” every possible course of action in
trying to solve [the reported phantom transactions]” and suggests that the issues were
1868 As is apparent from that Peak numerous routes had been
caused by user error.
exhausted including: a limited visual observation and attendance at the counter;
collecting information on transactions and observing for possible trends; replacement of
88 (Dayl7/32:6} to {Day17/33:13}.
86! (Day12/65:6} to {Day12/65:7}.
865 (Dayl7/34:8} to {Day17/34:12}.
866 (Dayl7/37:22} to {Dayl7/37:25}.
1867 (F/88.2}.
1868 (F/88.2/2}.
476
AI8/476
all kit and cables; installation of a new screen and sending off screen for testing; and the
offer of a longer observation.'8°
15. Peak PC0065021'8” was closed on 12 November 2001 on the basis that “/i/n all cases
where these have occurred a user error related cause can be attributed to the
phenomenon”.'8™!
16. Mr Coyne gives a misleading impression of what the Peak actually says and ignores the
later content of Peak PC0062561 and indeed the conclusions of the Peak. Mr Coyne fails
to mention at all the conclusion to Peak PC0065021 that all reported cases are
attributable to user error.
Issue 2
17. Peak PC0052025'8” relates to a report by an SPM that a receipt containing three
transactions printed by itself. Mr Coyne fails to note that this is a manifestation of a
designed system process which is to benefit SPMs.
18. Horizon automatically logs off a user after a period of inactivity - a standard security
measure of many IT systems. If the user has recorded, but not completed, some
transactions in the stack, then when the systems automatically logs off, it will complete
these transactions and assume payment was made by cash. This was a PO requirement
for Horizon. A receipt is printed so that the SPM knows what has happened.
19. This reason is clearly set out in the Peak:
“Messages in the messagestore confirm that the ‘phantom’ transactions were due to
them being in a suspended session that was later forcefully committed. Explained this
to PM who was happy with explanation but she says she is sure she never pressed the
suspend icon. Nevertheless she agrees closure for this problem. Can only assume that
she hit the suspend icon by accident.”'*
20. Mr Coyne fails to note in his reports that issue 2 is not a bug at all. It is a manifestation
of a designed system process.
POL00026925
POL00026925
1869 (F/88.2/2}.
1870 (F/97}.
1871 (F/97/9}.
187 (F/48}.
1873 1F/48/2}.
477
AI8/477
21. As set out above, the SPM accidentally touched “suspend” with her hand with the result
that the session was suspended for that period of time. Mr Coyne stated that this was the
system doing something “wrong”:
Q. Well, that’s what you say, Mr Coyne, but isn’t a fair reading of what's in that box
that the postmaster accidentally must have touched “suspend” with her hand or
something with the result that the session was suspended for that period of time?
A. Yes, that’s possible, but it isn’t that that was the reference that I was making to
without user interaction. It is the later event, the automatic commit that would appear
to have been done automatically without a user being involved.
Q. I see. So what you are suggesting is that if the system automatically commits an
uncommitted basket after a period of time that’s evidence of phantom transactions,
that’s evidence of a bug in Horizon that needs to be corrected, is that what you are
suggesting?
A. Well, it is evidence of the system doing something without the user choosing to do
it.
Q. Are you suggesting it is something -- it is evidence of the system doing something
wrong? I think you are, aren’t you?
A. Yes 1874
22. Mr Coyne’s attention was then drawn to Ms Van Den Bogerd’s second witness
statement.'’”> Mr Coyne agreed the following:
Q. That's what the system is designed to do. The system has to decide whether just
to delete the entire session from the system or to commit it. Either way a design choice
has to be made, and either way what’s important is that the subpostmaster knows
what happens, and this system is designed to give the postmaster that information by
printing out a receipt, correct?
A. Yes.!876
23. It was put to Mr Coyne that he knew about this design feature when he referred to this
Peak, which he denied:
Q. Right. And you knew that, didn’t you, when you included this PEAK as a phantom
transaction bug in your second report?
POL00026925
POL00026925
'8™ (Dayl7/19:18} to {Day 7/20:13}.
1875 (E2/5/4} para 14.2.
816 (Dayl7/21:18} to {Dayl7/21:25}.
478
AI8/478
POL00026925
POL00026925
A. No, I don’t believe I did. No.'°””
24. In fact, Mr Coyne referred to Ms Van Den Bogerd’s witness statement when he raised
this Peak in his own report:
Q. Well, let’s have a quick look at that just to see. It is at {D2/4.1/114}. Do you have
the page? Under the heading “Angela Margaret Van Den Bogard [sic] ” it says, 4.69:
“Mrs Van Den Bogard [sic] has provided a witness statement commenting on
individual cases and various disparate factual matters, which I do not attempt to
comment on in detail here. I note the following discrete points.” Then under the
heading “Phantom Transactions”, you say {D2/4.1/115}: “I have seen evidence of
phantom sales recorded in the disclosed documents. PEAKs PC0065021 216 and
PC0052025 ...”. Which is the one we are just looking at?
A, Yes.
Q. “... (documented in further detail at section 3, ‘Phantom Transactions (Horizon
Issue 4)’ above) refer to phantom transactions in branches, the former which was
observed by an engineer on site at the branch and the latter which refers to
discrepancy arising from them.”
A. Yes.
Q. So you did know what Mrs Van Den Bogard [sic] was saying because you
identified this very PEAK when you were responding to that very evidence?
A. Yes.!87*
25. Mr Coyne was evasive in acknowledging that he must have been aware that this was a
design feature of Horizon:
Q. It is completely wrong, isn’t it, Mr Coyne?
A, My understanding is now that it would appear that it is a design feature, that that
happens.
Q. Ifyou don’t mind my saying so, Mr Coyne, you appear to be rather evasive. What
I have just shown you is that Mrs Van Den Bogard [sic] set out quite clearly in a
witness statement how the system was designed to operate, and it was designed to
operate in a way that committed transactions that were left on the machine for a
certain amount of time and a receipt was printed so the postmaster knew what had
happened?
A. Yes.
187 (Dayl7/22:1} to {Day17/22:4}.
878 (Dayl7/22:5} to {Day17/23:3}.
479
AI8/479
POL00026925
POL00026925
Q. And you responded to that by saying, well, I have got some evidence of phantom
transactions and half of the evidence, one of the two PEAKs you rely on for that
purpose, is a PEAK which actually demonstrates the truth of what Mrs Van Den
Bogard [sic] is actually saying, doesn’t it?
A. Certainly one of the two examples it would appear that it is operating in line with
the design, but I can certainly understand how a user would perceive that that would
be a phantom transaction because they didn’t complete it —
Q. Mr Coyne, if we were looking at a table which was a table of areas where users
might get confused, then we wouldn’t be having this conversation.
A. Mm.
Q. We are having this conversation because you have added this piece of evidence as
evidence in support of the proposition, firstly, that there are bugs in Horizon and,
secondly, that they cause losses, lasting losses to postmasters. That is right, isn’t it?
A. Yes,1879
26. Mr Coyne ultimately agreed that this Peak was not an example of a bug in Horizon:
Q. And you agree now, do you, that this PEAK is not evidence either of a bug or of
the causing of a lasting loss to a postmaster?
A. Ido agree.!*"°
Issue 3
27. Peak PC0052025 relates to a report by an SPM that certain icons differed on his two
counters.'**! This was diagnosed as the second counter not having received the latest
Riposte release (which changed the button text) so the two counters were one version
apart. The second counter would have been automatically upgraded as part of a software
rollout and tail management, but Fujitsu explicitly upgraded it early to bring it in line
with counter one. As with Issue 2, Mr Coyne again fails to note that this was not a bug.
28. Although included in Mr Coyne’s analysis of branch affecting bugs, Issue 3 had no
financial or operational impact on a branch accounts.
879 (Dayl7/23:4} to {Day17/24:11}
188 (Day 7/24:12} to {Dayl7/24:15}.
1881 (F/48},
480
A/8/480
POL00026925
POL00026925
Conclusion on Bug 15: Phantom Transactions
29. Mr Coyne has failed to analyse, or even mention, important Peaks and sections of those
Peaks in his analysis. It is misleading for Mr Coyne to draw these three distinct issues
together to support his conclusion at paragraph 3.153 that they “il/ustrate potential for
errors in data recorded within Horizon arising for Hardware failure and accepted design
features” '8°
30. For the reasons set out above, there is no indication that phantom transactions had a cause
other than user error, as indicated in the Peaks. Further Mr Coyne has included Peaks in
his analysis which do not indicate bugs in Horizon on or a financial impact on branch
accounts.
Bug 16: Reconciliation Issues
1. The key documents:
1.1. PC0039832;'8%5 PC0075240;'*** PCO075415'%*5 and PC0077508. '**°
1.2. Coyne 2: 3.154 — 3.173.'887
1.3. 582.188
Summary
2. Mr Coyne now states that this bug does not have lasting financial impact on branch
accounts. Post Office submits that this is not a bug at all.
el82 (D2/4.1/57}.
188 (F/23},
18 (F/113}.
885 (E/I15}.
1886 £F/116}
1887 (1)2/4,1/57} to {D2/4.1/61}.
888 1/2/13} to {D1/2/14}.
481
A/8/481
Nature of this issue
3. Mr Coyne refers to six issues under the heading “Reconciliation Issues”:
3.1. Paras 3.154 — 3.157 (PC0639832)'*® (“Issue 1”).
3.2, Paras 3.158 — 3.162 (PC0075240'%”, PC0075415'%"" and PC0077508)'*”? (“Issue
2).
3.3. Para 3.163 (PC0049578)'S” (“Issue 3”).
3.4. Paras 3.165 — 3.167 (PC0045847)'*" (“Issue 4”).
3.5, Paras 3.170 — 3.171 (PC0236246)'% (“Issue 5”).
3.6. Paras 3.172 — 3.173 (PC0204872)'% (“Issue 6”).
4. While giving evidence Mr Coyne acknowledged that the process of reconciliation was the
process of comparing two sets of data and that it was carried out electronically'**’. While
he suggested that he did not know what process Post Office uses for dealing with
reconciliation issues, Mr Coyne acknowledged that it was likely that Post Office did
something about accountancy discrepancies and that that explains why he felt able to judge
Horizon as relatively robust.'*°*
5. Mr Coyne also acknowledged that a report is generated on a regular basis which identifies
reconciliation exceptions and that is where there is a manual clement involved'*””. He
conceded that the fact that there are large numbers of reconciliation reports “suggest/ed/
rather than confirm[ed]” that there are bugs in Horizon which have an impact on branch
POL00026925
POL00026925
889 (F/23},
890 (F/113}.
891 (F/L15}.
1892 (F/116}.
1893 (/33},
1894 (F/27},
1895 (F/1245}.
896 (F/719}.
87 (Dayl4/76:15} to {Day14/6:16}.
898 (Dayl4/55:5} to {Day14/55:22}.
8 (Dayl4/76:25} to {Dayl4/77:1}.
AI8/482
POL00026925
POL00026925
1900
accounts. He acknowledged that “it is certainly more likely” that a Transaction
Correction will correct an error that’s occurred, rather than create one!””!,
6. On day 17, Mr Coyne agreed that “Reconciliation Issues” were unlikely to have lasting
financial impact.!°°?
Issue 1
7. Peak PC0039832' is an example of an issue affecting a Subpostmaster’s Cash Account
Period (CAP).!% Mr Coyne states that “reconciliation discrepancies have appeared but
did not feature on the expected reconciliation exception reports. 1°
8. By way of background, at the end of each working day, the counter calculated information
for the Cash Account in two independent ways to ensure that they matched and a report
was generated if they did not match.
9. In this case, the reconciliation software detected discrepancies relating to two low value
transactions (£8.06 and £0.08, totalling £8.14). It appears from the Peak that there was a
bug in the reconciliation software, although the Peak is not fully conclusive. If that was
the case, a false reconciliation report would have been generated but there would have been
no financial impact on the branch.
10. There are several misunderstandings in paragraph 3.154 of Coyne 2'°°°. First, Mr Coyne
alleges that the bug “did not feature on the expected reconciliation exception reports ”!°"”.
Mr Coyne has confused (i) the information captured by Horizon with (ii) the information
that was provided to SSC during its investigation. The reconciliation error was clearly
captured, as that is what triggered an investigation. Second, at paragraph 3.156 Mr Coyne
1900 (Day14/79:16} to (Day! 4/79:23}.
1991 (Day14/80:1} to {Day14/80:17}.
1902 (Day 17/124:19} to {Day17/124:21}.
1903 (F/23},
190! 4 CAP is “a period (usually a week) of office accounting as defined by the Post Office” — {F/40/11}. A
Cash Account is a “statement of cash, stock and transactions produced by the Horizon system at the end of
the accounting period” — {F/40/11}. in 2005 the Cash Account system was replaced with the Branch Trading
system. Branches must balance their accounts and produce a Branch Trading Statement at the end of each 4-
5 week Trading Period — {E2/5/38}.
1908 ()2/4.1/57}
1906 ()2/4.1/57}.
1907 ()2/4.1/57}.
483
AI8/483
seems to suggest that this bug had a financial impact on a branch, but this is not evident
from the Peak. Coyne 2, paras seems to assume the word “discrepancy” means something
that was impacting the branch, rather than a reconciliation issue “behind the scenes”.
11. Mr Coyne refers to Peak PC0039832 being fixed “five months after the original PEAK was
raised”. This amount of time is consistent with this being an issue that did not affect branch
accounts. The fix is documented in PC0047955'°"°.
Issue 2
12. Mr Coyne alleges that Peaks PC0075240, PC0075415 and PC0077508'"™ all relate to a
Horizon reconciliation report that shows a discrepancy for the branches documented. The
Peaks relate to an issue where a branch counter total differed from the amount on the TPS
host.
13. Whilst Mr Coyne’s summary of events is accurate, the suggestion that there was a
discrepancy in branch accounts is incorrect. In these cases, the counter calculated a value
of Ip and the calculations carried out at the host gave a value of £0.0099. However as the
first three digits were 000 it took this to be Op and so reported a false discrepancy.
14. This is an example of a check and balance raising a false alert. Fujitsu spent time
investigating a false report of a non-existent discrepancy. The issue was automatically
detected by Fujitsu and a fix was rolled out.
Issue 3
15. Peak PC0049578!"" was raised in testing and fixed before the software went live. This
may not have been apparent to Mr Coyne, as he fails to mention it in his analysis.
16. The Peak describes a problem with producing a report used to confirm that all data has
been passed to TIP. In short, the underlying data transferred to TIP was correct, however,
the number of files transferred did not match.
POL00026925
POL00026925
1908 {F/30}.
19 (F/113}; {F/115} and {F/116}.
B
1910 ¢F/33}
484
AI8/484
17. There is also a report to confirm what files were transferred, and that report was incorrect
in the count of files transmitted. If this had gone to Live then it could have caused
confusion, but there would have been no impact at all on branch accounts. This is because
the issue is isolated to the data centre and is entirely separate to transaction data, meaning
it is not possible for this issue to impact branch accounts.
18. This concerns getting data from Riposte Message store out to TPS so it can be sent to POL
SAP (the TIP system).
19. The report showed a difference between the number of files recorded as being transferred
to TIP and the number of files transferred.
20. The Peak investigations confirm that the root cause arises from the following:
20.1. The report was run and the number of files counted after the sort by org-unit and
trading date.
20.2. When the report was run, it was not accounting for the files already counted, i.e.
it appears that a count was being done in the wrong place.
21. As previously indicated, the issue is isolated to the data centre and is entirely separate to
transaction data, meaning it is not possible for this issue to impact branch accounts,
contrary to Mr Coyne’s suggestion in the Joint Statement.'*"! It also occurred in testing,
not the live environment.
POL00026925
POL00026925
9 ED1/2/14}.
485
Al8/485
Bug 17: Branch Customer Discrepancies
The key documents:
1.1 Peaks: PCO156174'°!? and PC0156246.'93
1.2 KEL: carde226K."""4
1.3 Coyne 2, paras 3.174 — 3.177.195
1.4 Js2.!!¢
Summary
2.
Mr Coyne now states that this bug does not have lasting financial impact on branch
accounts. Post Office submits that this is not a bug at all. There is no evidence of a
bug in Horizon and in any event instances of any issue, not bugs, were caught by
automatic reporting.
Nature of this issue
POL00026925
POL00026925
3. Mr Coyne only references Peak PC0156246'*"” in his report. This is the second of two
Peaks that relate to an issue that affected one branch, the other being PCO156174.!""5
4. These Peaks relate to a discrepancy between Post Office’s records (from Horizon) and a
Financial Institution’s records after a counter crashed while a transaction was being
processed. There was no bug in Horizon.
5. When the counter was restarted, the branch will have been presented with a recovery
process. If there was a discrepancy in the branch’s accounts after this event, that will be
because the branch failed to follow the recovery process correctly.
1912 (F/446.1}.
18 {F446}.
1914 (F/418.01}.
18 ()2/4,1/61} to {D2/4.1/62}.
16 (D1/2/16}.
917 {F446}.
1918
486
Al8/486
POL00026925
POL00026925
Detection and Resolution
6. As shown in Peaks PCO156174'?!° and PCO156246'”":
6.1 The issue was identified by Fujitsu’s automatic reporting using the NB102:
Exception Summary.'*! These are a series of reports that are run daily which
show discrepancies between Financial Institutions’ view of what has happened
and Horizon’s record of the same transaction.!? The first Peak (PC0156174)
was raised for Fujitsu to investigate in response to the NB102: Exception
Summary. This is an example of Fujitsu’s reporting picking up issues that arise
automatically.
6.2 When Fujitsu investigated the issue, they saw that there had been no subsequent
log on to the affected counter and this meant the branch had not been presented
with the recovery process.!??3
6.3. Fujitsu telephoned the branch and asked the SPM log in to the counter and follow
the recovery messages in Horizon to resolve the issue.!°**
6.4 Fujitsu issued a BIMS Incident Report to Post Office!’ which advised that the
SPM had now logged in to the counter and as such Fujitisu were able to see that
the recovery messages were in the branch messagestore.
6.5 However, the branch appeared in the NB102: Exception Summary reports again
and the second Peak (PC0156246!°”°) was raised for Fujitsu to investigate.
6.6 Fujitsu issued a second BIMS Incident Report to Post Office!” which advised
that while the recovery messages were received by the SPM, the SPM had
declined them. This indicates that no money changed hands during the
transaction (i.e. the SPM elected not to recover the transaction because no money
1919 (F/446.1}.
1920 1F/446}.
121 (F/446.1}.
1922 4F/1687/9}.
193 4F/446,1/1}.
1924 F/446,1/1}.
1995 ¢F/444.2}
1926 (F446).
1927 F/446.2}.
487
AI8/487
had changed hands; if money had changed hands, the SPM should have elected
to recover the transaction). As can be seen in the BIMS report, Fujitsu asked
Post Office to:
(a) check with the SPM whether any money changed hands; and
(b) if money had not changed hands, ensure that the discrepancy between Post
Office’s records and the Financial Institution, here Citybank, was resolved.
7. Mr Coyne only references the second Peak (PC0156246'***) in his report, which means
that his analysis and explanation is incomplete. Mr Coyne criticises Fujitsu for not
reaching a clear diagnosis, but the following points can be made in response:
7.1 While there was a possibility that the Financial Institution registered a withdrawal
from the customer’s account (depending on how the Financial Institution’s IT
system is configured), this would have caused a loss for the customer, not the
branch (assuming that the customer received no cash from the branch). If the
customer did receive cash from the branch, the resultant discrepancy would have
been caused by the branch not following the recovery procedure correctly.
7.2 In this case, Fujitsu’s automatic reporting detected the difference between Horizon
and the Financial Institution’s records. In response to this, Fujitsu raised a Peak
to investigate and issued a BIMS Incident Report to Post Office to enable Post
Office to further investigate what happened with the SPM and reconcile any
discrepancies with the Financial Institution to ensure there was no loss to the end
customer.
7.3 During Mr Coyne’s cross examination, he confirmed that he did not consider this
issue to have lasting financial impact.'*”?
Conclusion
8. There was no bug in Horizon.
POL00026925
POL00026925
1928 (F446).
19 (Dayl7/122:12}.
488
Al8/488
9. If there was a discrepancy in the branch’s accounts due to this issue, it will be because
the branch did not follow the recovery procedure correctly.
Bug 18: Concurrent Logins
1. The key documents:
212.1 PC0027581'°*; PCO0S1327'°*!; PC00S0974'°?
212.2 Coyne 2, paras 3.179 — 3.183.193
212.3 382.14
Summary
2. Post Office accepts that Bug 18: Concurrent Logins had a potentially lasting financial
impact. There is no evidence of any discrepancy in the Peaks referred to by the
experts.
Nature of this issue
3. This issue is known as “Concurrent Logins”.
4. Mr Coyne has referred to two issues under the heading of “Concurrent Logins” :
214.1 Para 3.180 relates to issue I (PC0027581)!"* (“Issue 1”).
214.2 Para 3.181 relates to issue 2 (PC0051327)'°* (“Issue 2”).
5. These issues occurred very early on in Horizon (1999/2000).
POL00026925
POL00026925
1930 FF/15},
1931 (F/43},
1932 (F/38.1}.
933 ()2/4,1/62} to {D2/4.1/63}.
19 (D1/2/16}.
1935 (F/15}.
1936 (F/43},
489
A/8/489
Coyne’s change in position
6.
Mr Coyne repeatedly confirmed during cross-examination that where a bug was listed
as Horizon Issue 4 this meant that there was no lasting impact on branch accounts. '°*7
Mr Coyne maintained during cross-examination that Bug 18 is an example of a bug with
lasting financial impact. This is inconsistent with the approach he took in his second
report. He stated not only that it was relevant to Horizon Issue 4 but also stated that there
was no financial impact at all, lasting or otherwise.'°**
When asked about this Mr Coyne stated the following:
Q. 18, do you say there's evidence of lasting impact with 18, concurrent logins?
A. Yes.
Q. I’m interested that you should say that, Mr Coyne, because if we could go back to
your second report at {D2/4.1/18}.
A. Yes.
Q. Look at the table and look at concurrent logins, for “Evidence of Branch Impact”
you say “No” there. Did you change your mind?
A. Afier discussion with Dr Worden, yes.!°°
Mr Coyne failed to identify how his discussions with Dr Worden changed his view so
dramatically. Dr Worden is of the opinion that this bug is not a bug at all. It is therefore
difficult to conceive how discussions with Dr Worden would have resulted in Mr Coyne
departing from his view that there was no financial impact at all of this bug.
Although the experts agreed that there may be impact on branch accounts, Dr Worden’s
comments (both in his reports and JS2) do not indicate any lasting impact.
Mr Coyne failed to identify or justify this change of position. In addition, such a view in
inconsistent not only with Dr Worden’s reasoned analysis of this bug but also Mr
Coyne’s own evidence.
POL00026925
POL00026925
1987 See for example {Day17/123:4} to {Day17/123:18}
1938 ()2/4,1/18}.
989 (Dayl7/132:12} to {Day17/132:22}.
490
A/8/490
POL00026925
POL00026925
Issue 1
12.
16.
Peak PC0027581 refers to an issue where a counter froze whilst printing a final cash
account. This counter was still showing ‘printing report’ when the Subpostmaster was
able to log onto a different counter and print the same report.!°4°
In Legacy Horizon, one counter was the Gateway counter that connected to the data
centre. The other counters were slave counters that connected to the Gateway counter.
In this specific issue, the Subpostmaster was printing a final cash account on counter 2
when the counter froze. The Subpostmaster logged onto the Gateway counter to try and
print out the final cash account on a different counter, and this printed off. The slave
(counter 2) was still showing ‘printing report’ whilst the Subpostmaster was logging onto
the Gateway and printing the same report.
The issue arose because counter 2 had frozen and could not respond to a login request
on the Gateway counter. When logging onto a counter, the counter checks that the user
is not logged into any other counter. It does this by sending a message to the other
counters and waiting for a response. In this case, counter 2 was frozen and therefore
unable to respond to the Gateway counter to say that the Subpostmaster was already
logged in. The Gateway counter therefore assumed that the Subpostmaster was not
logged in and allowed him to log into the Gateway counter.
Mr Coyne has alleged in his Supplemental Report that Concurrent Logins can “cause
transactions to be abandoned and risk discrepancies” '°"'
There was no discrepancy in
this case - this issue related to a frozen transaction that was the printing of a report rather
than the processing of a transaction.
Mr Coyne has suggested that Fujitsu did not follow this issue up properly with Esher.’
Mr Coyne has failed to take into account the entry dated 15 September 2000 timed at
08:07 where Fujitsu, in believing this was a problem with the underlying Riposte
software, passed the issue to Esher and the issue was thought to be “Now formally fixed
1910 (E/15},
1941 ()2/4,1/62}.
92 ()2/4,1/63}.
491
A/8/491
17.
in Build 223 update 19 which was released overnight.” The new release from Esher
did not however, as it was expected to, fix the problem.
Peak PC0027581 was “Closed at management request.... Mr Lui is no longer employed
by the Post Office and has not been for some years. Should the problem reoccur then
s91944
please reopen this call. The call was not re-opened after this which suggests that
few or no other branches were affected.
Tssue 2
18.
20.
PC0051327 refers to an issue whereby the Subpostmaster logged onto two terminals
concurrently, resulting in receipts and payments mismatches over three Cash Accounting
Periods. The problem was caused by the Subpostmaster rolling counter 3 whilst logging
in and processing a transfer out on counter 4.
The net difference of these receipts and payments mismatches amounted to zero,
meaning there was no overall impact on the branch. This would have been clear to Mr
Coyne as the very nature of this issue is not that it causes money to be lost, but money
to be accounted for in the incorrect time period. The Peak shows that there are offsetting
receipts and payments gains and losses and therefore it is possible to ascertain from the
Peak whether the branch suffered a discrepancy.
The issue was fixed through a planned software roll out that changed the code in the area
that caused the bug (release C145) and the call was closed on 30 November 2000.!°4°
Conclusion
21.
For the reasons set out above, there are two distinct issues. No discrepancies are set out
in the Peaks referred to by the experts. However, if a discrepancy had occurred, that
discrepancy would manifest itself as a receipts/payments mismatch.
POL00026925
POL00026925
1983 fF/15/7}.
19M F/15/12}.
1818 F/43/4)
492
Al8/492
POL00026925
POL00026925
Bug 19: Post & Go
The key documents:
1.1 Peaks: PCO218702;'° PCO219432;'947 PCO220393'* and PC0221150.'°?
1.2 Coyne Supplemental Report 3.185 — 3.190.!°%°
1.3 Js2,'°!
Summary
2.
Mr Coyne states that Bug 19: Post & Go/TA Discrepancies is a bug with lasting financial
impact. Post Office submits that any discrepancy would be transient. Any issues were
picked up automatically by Fujitsu’s automatic reporting.
Nature of the issue
3.
Peak PC0220393 relates to an issue that stopped data being transferred from Horizon to
POLSAP for reconciliation purposes.!?>?
The issue relates to “Post & Go” (“P&G”). These are self-service terminals. These
terminals are now only held in Crown and WH Smith main branches. (There were a few
P&G machines piloted in a few main branches but this was for a short period only.) This
issue does not relate to branches that are the focus of this trial. The issue is therefore
irrelevant.
In terms of the specific issue that arose, P&G transactions were being accurately
recorded by the P&G terminal and then transferred to Horizon. Horizon creates a TA of
all the transactions on the P&G machine and, when accepted by the branch, these
transactions are added to the branch accounts. The data is also passed from Horizon to
various other systems, including POLSAP.
1946 (F/945}.
197 (F/962}.
198 (F/974}.
199 (F/996.1}.
1950 (1)2/4.1/64} to (D2/4.1/65}
951 1/2/18}.
1952 (F/O74}.
493
AI8/493
6.
There was no difficulty with the transfer of data from the P&G terminals to Horizon, or
with the production and acceptance of TAs in branch. Further, branch accounts were not
impacted. The issue was at the reconciliation step within POLSAP. Fujitsu transfers the
relevant data to POLSAP via BLE files. When POLSAP was comparing the Horizon
data to the Wincor data, discrepancies were found as data for two P&G terminals in the
branch were not being sent to POLSAP. There were six P&G terminals at the branch.
Data from four terminals only were being transferred to POLSAP because the two other
terminals were not associated with conducting P&G transactions.
Mr Coyne has labelled this section as relevant to Horizon Issue 4 — data processing in
Horizon. However, there was no error of data processing in Horizon itself. The problem
concerned the transfer of data from Horizon to POLSAP.
Detection and resolution
8.
This issue was identified by Fujitsu automatic reporting using the “Swbfiles_on_hold”
report and as such is an example of Fujitsu’s automatic reporting picking up issues that
arise. Mr Coyne suggests that if Post Office had been monitoring the report they would
1953
have noticed the issue faster.'”°* There are two points to note in response:
8.1 The first relates to the assumption that there was a 43 day period of impact.'°** Mr
Coyne alleges that the fault would not have impacted for the length of time, 43
days, had Post Office been monitoring the report. However, this is based on a false
assumption on the length of the issue and/or a mistake on the part of Mr Coyne.
The duration of the problem appears from the Peak to have been from 29 August
2012 to 17 September 2012 — 19 days not 43 days as Mr Coyne suggests. The
figure of 43 days appears to relate to the 43 days of data that was stuck in Horizon
and not transferred to POLSAP.
8.2 Secondly, the final comment from Anne Chambers states “We strongly recommend
that POL monitor the SubfilesOnHold report which is sent to them daily”..° This
does not imply that Post Office should have been monitoring that report for that
POL00026925
POL00026925
53 ()2/4.1/65}.
1954 ()2/4,1/65}.
1955 (F/O 74/5}.
494
AI8/494
purpose beforehand. There is no other reference which suggest that Post Office
should have been monitoring it for this purpose prior to this later comment.
9. The issue was resolved quickly. Despite the Peak not being high priority (as there was
no financial impact on branch accounts) the issue was identified and a fix developed and
deployed in a short period of time. As indicated in the Peak it was opened on 29 August
2012 and closed less than a month later on 17 September 2012.
10. A work-around, MSC 043J0355958, was developed to create a zero value transaction for
the two P&G terminals that were missing one and this forced the branch to create the
stock unit association. Once the associations were created the withheld transactions were
posted to POLSAP successfully. A permanent fix was released, Release Peak
PC0221150, that automatically generated a zero value TA to force the association. !°°
Conclusion on Bug 19: Post & Go/TA discrepancies in POLSAP
11. Despite this bug appearing in Mr Coyne’s analysis of branch affecting bugs, the branches
suffered no discrepancies. The issue was identified through Fujitsu’s automatic reporting
and was quickly resolved.
POL00026925
POL00026925
1956 £F/996.1}.
495
AI8/495
Bug 20: Recovery Failures
1. The key documents:
1.1 PC0220532;°°7 PCO241242'°* and PC0197643'°?
1.2 Coyne 2, paras 3.191 — 3.196!
1.3 Js2.°
Summary
2. Mr Coyne states that Bug 20: Recovery Failures is a bug with non-lasting financial
impact. Post Office submits that this is not a bug at all. There is no evidence of a bug
in Horizon and in any event instances of any issue, not bugs, were caught by automatic
reporting.
Nature of this issue
3. Mr Coyne has referred to three issues under the heading of “recovery failures” :
3.1 Paras 3.191 and 3.192 relates to issue 1 (PC0220532)'°° (“Issue 1”).
3.2 Para 3.193 relates to issue 2 (PC0241242)!°% (“Issue 2”).
3.3 Paras 3.194 and 3.195 relate to issue 3 (PC0197643)'°™ (“Issue 3”).
4. Mr Coyne has accepted during cross-examination that this bug did not have lasting
financial impact and therefore should be removed from table 1 of his Bugs, Errors or
Defects located in the Peaks reviewed.!°%
POL00026925
POL00026925
1957 4F/978}.
1988 (F/1315}
1959 (F/613}.
196 {1)2/4.1/66} to {D2/4.1/67}.
1961 {1D 1/2/19} to {D1/2/20}.
1982 ¢F/978}..
1968 ¢F/1315}.
1964 F/613}.
968 (1)2/4.1/18} and {Day17/126:2}.
496
AI8/496
Issue 1
POL00026925
POL00026925
Peak PC0220532 relates to a cash discrepancy that was only raised by the
Subpostmaster with Post Office and Fujitsu nine months after the issue occurred. Mr
Coyne criticises Fujitsu for not reaching a full resolution in paragraph 3.192 of his
Supplemental Report'®°, however he fails to mention the nine month gap that made
retrospective analysis impossible, nor does he mention the reference in the Peak to this
issue appearing to be an accounting issue, rather than an IT problem, as per the earlier
analysis that they had undertaken (entry dated 6 September 2012 timed at 10:40).!°°7
Mr Coyne does, however, agree in JS2 that this Peak “may not indicate a Horizon
bug/error or defect” '°
Mr Coyne does not explain at paragraphs 3.191 and 3.192 that the memory dump and
base unit swap occurred after the loss arose, nor that nothing in the Peak raises any
recovery issues.
As is evident from the Peak, Fujitsu’s investigation is detailed given the lack of
explanation or information from the Subpostmaster.
The call was closed on 6 September 2002 following advice being given following
Fujitsu’s investigation.°°
Mr Coyne alleges at paragraph 3.192 of his Supplemental Report that “/i/1 is unclear
[...] what the full resolution or conclusion of the issue was since Post Office have not
disclosed in detail full Transaction Correction information for all reported
discrepancies”.'°"° This gives a misleading impression, particularly given this Peak
does not relate to a Horizon issue but an accounting issue and there are circa 100,000
Transaction Corrections each year.
1966 spy resey
1967 {F/978/1} to {F/978/2}.
1968 (D)1/2/19}.
19 (F/978/2}.
1970 ¢1)9/
/4,1/66}.
497
AI8/497
Issue 2
10.
Tssue 3
13.
15.
POL00026925
POL00026925
Peak PC0241242 concerned a Subpostmaster who reportedly was unable to complete a
recovery on node (counter) 4. The root cause of the issue was ultimately the failure of
the ap-ade recovery script (Automated Payment Advanced Data Capture script, which
is written by Atos). The workaround to get the counter through recovery so that it was
operational again was to remove the Recovery data relating to the Lottery Transaction.
The final solution was actually to mark the Recovery Data for the lottery transaction as
complete (update) rather than deleting. This required Privileged User access which was
authorised using MSC:043J0428695 which also contains the SQL (Structured Query
Language) statement used. As the recovery failure on the Health lottery also caused
the banking transaction to fail to recover a Transaction Correction was needed for the
banking transaction because otherwise it left a cash shortage of £70.66.
There was no long term loss caused by this issue as reconciliation was done for the
banking transaction and a Transaction Correction done for the cash shortage.
Peak PC0197643 has been cited by Mr Coyne as a recovery failure with a financial
1971 However, recoveries sometimes fail and Mr Coyne
impact on branch accounts.
accepted during his cross-examination that it is inevitable with any system that there
will be a small proportion of cases where the automatic recovery processes do not work
and that PCO197643 is an example of this.
Recovery failures are detected by Fujitsu, as evidenced by failures are reported to is
supported by the two KELs listed in the Peak, KEL acha959T and KEL dsed2640M.'”
Mr Coyne was asked questions regarding KEL dsed2640M.'°”* Mr Coyne replied on
72
what would be visible to an SPM in branch:
Q. Then it says: “The print problem should be evident by 0607 errors being displayed
to the PM and the same error being recorded in the PostOfficeCounter.log at the time.
There will also be Warnings logged to this log with the words “Received second print
17 D2,
1? FV]
/4.1/66}.
700}; {F/587}.
1978 (F/S87}.
498
Al8/498
16.
request, before completing first print request.” So would it be right to infer from what
that KEL says that this problem, as well as going through to the MSU and being
pushed through to the SSC in the normal way, the problem would also be evident to
the postmaster who was undertaking the transaction at the branch, would you agree
with that?
A. Yes, It is said there will be a 0607 error displayed. So if the screen is working as
a result of whatever this failure is, then they would probably see that. The other two
things there, PostOfficeCounter.log and warnings in the log, they won't be visible to
the subpostmaster.
Q. So one way or another, both the postmaster and Fujitsu independently will know
that there’s a problem when this arises, yes?
A. Yes, and at this stage the counter probably hasn't yet booted back up yet. It is in
a failed state obviously.!°4
Therefore, both the SPM would be aware (given the information visible in branch) and
Fujitisu would be aware give the reports they had available.
Mr Coyne stated during cross-examination that KEL acha959T addressed workarounds
for bugs, errors or defects in Horizon. Mr Coyne’s reasoning appeared to focus on the
following:
Q. Mr Coyne, we will have to agree to differ on that. My suggestion to you is that
this is just an explanatory KEL which explains how the system works. It is actually
describing how the system should work, it is not describing what should happen when
the system fails, but I can see that we are not going to agree about that.
A. It is inconsistent that users should be expected to go through to Fujitsu third line
report to deal with a problem that almost should happen. That can’t be right.’°”°
Mr Coyne has conflated the need for a manual process with the existence of a bug, error
or defect in Horizon. There are many reasons why manual assistance may be required.
Mr Coyne agreed that it is inevitable that any system, however well designed, will
require a manual recovery process too:
Q. I didn’t ask the question properly, and it is my fault not yours. What I meant to
ask was it is inevitable with any system, however well designed, that there will be a
POL00026925
POL00026925
1974 (Dayl7/81:1} to {Day17/81:24}.
975 (Dayl6/151:19} to {Day16/152:3}.
499
A/8/499
POL00026925
POL00026925
small proportion of cases where the automatic recovery processes don’t work. That's
Just inevitable, isn’t it?
A, Yes.197
20. I Mr Coyne went on to add to this:
Q. We have agreed that there will always be cases, a small proportion of cases, where
the recovery process that one would like to operate automatically for one reason or
another doesn't, and in that small proportion of cases some form of manual assistance
is required, isn’
A. Yes.
Q. And the system for example isn’t designed -- the recovery system isn’t required to
require the counter to keep trying to log on and log on perpetually, it is designed to
log on only twice?
A, Yes.
Q. That is a design feature, because if it is perpetual you can’t use the machine?
Q. That is very helpful. So the fact that you have recovery failures is not of itself a
threat to robustness unless the proportion of the recovery failures you have is too
high?
A. Yes.
Q. Here the system that’s operated by Horizon is that where you have a recovery
failure it is always reported both to Fujitsu and, by error warnings, also to the
subpostmaster, yes?
A. Yes.
Q. So both Fujitsu and the postmaster, where there is a failed recovery, know there is
a problem and they know they need to deal with it by communicating with each other,
do you agree with that?
A. Yes.
Q. And that’s the way that the Horizon system is constructed, correct?
A. Yes.
1978 {Day17/89:5} to {Day17/89:11}.
500
A/8/500
Q. One other aspect of the recovery process that’s very important is to know when
money changes hands?
A. Yes.
Q. And let me just expand on why that’s important. You have a transaction, ex
hypothesi it is a transaction involving a financial institution making or receiving a
payment, and let’s say it is a bank deposit. The customer hands in £100 at the branch,
the branch presses the buttons so that the customer's bank account goes up by £100,
and the problem that arises with recoverable transactions is that the bank may have
been told to increase the balance by £100 before the transaction has actually been
committed to the accounts of the branch.
A. Yes,!977
21. Therefore, Mr Coyne agreed that there are reasons why a system needs to be designed
to have a manual recovery element.
22. This Peak is good evidence of countermeasures working as they should. The issue was
detected and Fujitsu sent Post Office a BIMS report to enable Post Office to resolve the
issue.
23. No fix was required because there was no software bug.
Conclusion
24. For the reasons set out above, this is not a bug at all. There are three distinct issues
under this heading. There is no evidence of a bug in Horizon and in any event, instances
of any issue were caught by automatic reporting.
POL00026925
POL00026925
19 {Dayl7/89:25} to {Day17/91:20}.
SOI
A/8/501
POL00026925
POL00026925
Bug 21: Transaction Correction Issue
1. The key documents:
Ll
1.2
1.3
14
1.5
Summary
2. Durin;
Peaks: PC0114154"8; PCO118562"°; —PCO120459'°8°;— PCO121331'*';
PCO129587'"*?; PCO129774'"*; PCO130056'"**; PCO130057'**; PC0204350'°8°
and PC0205567.!°%7
Coyne 2, paras 3.197-3.210.!°%
JS2. 1989
LKiang2837P.'°°
KEL obenge2336R.!"!
cross-examination Mr Coyne clarified that he does not consider Bug 21:
Transaction Correction Issue to be a bug with lasting financial impact.'”? Post Office
submits that there is no evidence of any financial impact on branch accounts.
1978 £F/245}.
1979 FF/261}.
1980 {F/268}.
1981 £F/274}.
982 4F/314}.
1985 (F/318}.
1984 ¢F/321}.
1985 ¢F/322}.
1986 (F/710}.
1987 £F/734.1}
1988 (1D 2/4.1/6
7} to (D2/4.1/70}.
98 {1/2/13}.
199 (F/324,2}
1991 §F/73
2}.
92 (Dayl7/122:6} to {Dayl7/122:8}.
502
AI8/502
POL00026925
POL00026925
Nature of this issue
3. Mr Coyne has drawn together three distinct issues under the heading of “transaction
correction issue”:
3.1 Paras 3.201 - 3.202'°** relate to issue 1 (PCO120459)'; (PCO118562)'"5;
(PCO114154)!°%; and (PCO121331)'%” (“Issue 1”).
3.2 Paras 3.198 — 3.203! relate to issue 2 (PC0129587)'; (PC0130056)°;
(PC0130057)'; (PC0129774); and (KEL LKiang2837P)" (“Issue 2”).
3.3 Paras 3.204 - 3.210°° relate to issue 3 (PC0204350)°; (PC0205567); and
(KEL obenge2336R)” (“Issue 3”).
Issue 1
4. Peak =PC0120459,S Peak PC0118562," Peak PC0114154 and Peak
PC0121331?""' refer to issues with the TC button on screen.
5. Each of these Peaks was logged by Fujitsu’s Solution Validation and Integration team,
known as the ‘SV&I’ team, which is responsible for completing Solution Validation
193 (1)2/4.1/68}.
194 £F/268}.
1995 {F/261}.
1996 ¢F/245}.
1997 {F/274}.
198 (1)2/4.1/67} to {D2/4.1/68}.
1999 (F/3 14}.
2000 {F/321} This was an internal Peak raised by FJ to ensure the fix is carried forward to the next release.
2001 {F/322}.
2002 {F/318}.
2003 1F/324.2}
2004 {1)2/4.1/69} to {D2/4.1/70}
2005 $F/710}.
2006 {F/734.1}.
2007 {F/736.2}.
2008 F/268}.
20 {F/261}.
2010 {F/245}.
2011 {F/274},
503,
A/8/503
POL00026925
POL00026925
Testing.” As such, these Peaks all relate to issues raised by users during testing prior
to the TC functionality going live in September 2005 (when it replaced error notices).
6. Mr Coyne did not explain how these Peaks support the suggestion at paragraph 3.197 of
his report that they provide “an insight in relation to the technical flaws surrounding the
processing of transaction corrections” 203
7. In fact, this issue is an example of Fujitsu’s testing processes picking up issues before
TC functionality went live.
8. Although included in Mr Coyne’s analysis of branch affecting bugs, Issue 1 was not a
bug at all and it had no financial impact on branch accounts.
Issue 2
9. Peak PC0129587"""* and Peak PC0130056*""* refer to the Petersfield branch. Peak
PC0130056""* is a clone of Peak PC0129587.7°"7
10. Peak PC012958770"* relates to a report by a SPM on I December 2005 that cach time
she went to select a TC for a £9,000 debit, which arose from an incorrect entry for
Premium Bonds, her screen froze and she was unable to accept it.
Il. Peak PC0129774"" and Peak PC0130057°” refer to the Bosham branch. These Peaks
relate to a report by a SPM on 6 December 2005 that each time she went to select a TC,
a £22,500 debit for an incorrect entry for Premium Bonds, her screen froze and she was
unable to accept it.
12. Mr Coyne’s analysis proceeds on the basis that this issue was only experienced by these
two branches. Mr Coyne fails to refer to KEL LKiang2837P,””! which demonstrates
that: 50 other branches reported an issue with their screen freezing to Fujitsu during
2012 (F/1211/6} to {F/1211/7}.
2013 (1)2/4.1/67}.
2014 (F/3 14}.
215 (F/321}.
2016 (F/321}.
2017 FR/3 14}.
2018 (F/3 14}.
2019 {F/318}.
2020 (F/322}.
2021 {F/324.2}.
504
AI8/504
13.
December 2005; 48 branches reported that this prevented them selecting an outstanding
Camelot Lottery TC and rolling over into the next trading period; and 4 branches
reported that this prevented them selecting an outstanding Premium Bond Sale TC and
rolling over into the next trading period, including the two branches referred to in the 4
Peaks mentioned by Mr Coyne.
The diagnosis of the issue was that the text drafted into the TCs contained a string of 35
characters without a space. The code to put each TC on a screen was attempting to split
the text at a space (as is normal for a word processor), but the code was unable to process
a string of this length.
The issue was resolved quickly. Having first been reported on 1 December 2005, a work-
around was developed whereby the affected branches were instructed to roll over all
stock units except one (the one relating to the unprocessed TC). This enabled them to
roll over into the new trading period. A software fix?’ was then released on 22
December 2005 which enabled the branches to select and process the outstanding TC,
and roll over the last stock unit. To avoid the issue reoccurring in future, Post Office
confirmed that strings of this length without spaces would no longer be included in the
text of TCs.
Although included in Mr Coyne’s analysis of branch affecting bugs, the impact of Issue
2 was limited to the affected branches being unable to roll over to the new trading period
for a short period of time. It had no financial impact on branch accounts.
Issue 3
16.
Peak PC0204350" refers to a report by an SPM on 14 September 2010 that he had
suffered an £80 cash loss which he believed was due to a “system error”°°* In support
of this he referred the Fujitsu representative on the call to a TC report he had generated
on 10 September 2010, which had allowed him to request a date range of 60 days, but
which did not provide him with data on TCs which were more than 40 days old.
The Peak indicates that on 14 September 2010, Fujitsu requested detail of the specific
transactions relating to the £80 loss from the SPM in order that it could carry out an
POL00026925
POL00026925
222 COUNTER_EPOSS25_9.
2023 4F/710}.
2024 {F/TLO/1}.
S05
AI8/505
18.
20.
investigation into the issue. No information was provided by the SPM. Given the
absence of evidence, the matter was thereafter closed by Fujitsu.
On 20 October 2010 Fujitsu noted that when a TC report was generated, data was not
available for TCs older than 40 days due to the occurrence of an issue following the
introduction of Horizon Next Generation. When generating a TC report however, the
option to select a date range of 60 days was still available to SPMs.
Peak PC0205567*® is a clone of Peak PC0204350°° with additional detail added. This
Peak refers to the fix created by Fujitsu, which ensured that the retention period for TC
data was changed from 40 to 60 days.
Although included in Mr Coyne’s analysis of branch affecting bugs, Issue 3 had no
financial impact on branch accounts.
Conclusion on Bug 21: Transaction Correction Issues
21.
22.
Mr Coyne has failed to analyse, or even mention, important documents such as KEL
LKiang2837P”™”’ in his analysis.
For the reasons set out above, it is misleading for Mr Coyne to draw these three distinct
Issues together to support his conclusion at paragraph 3.197 that the Peaks cited “provide
an insight in relation to the technical flaws surrounding the processing of transaction
corrections”°> Mr Coyne has included Peaks in his analysis of Issue I which do not
indicate bugs in Horizon. Further, despite the bug referenced in Issues 2 and 3 appearing
in Mr Coyne’s analysis of branch affecting bugs, no financial impact was suffered by the
branches, as acknowledged by Mr Coyne during cross-examination.°?
POL00026925
POL00026925
2025 F/734.1}.
2028 (Dy
2029 4
2026 {F/710}.
$42.2+-
2/4,1/67}.
ay17/122:6} to {Dayl7/122:8}.
506
A/8/506
Bug 22: Bugs/Errors/Defects introduced by previous applied Peak fixes
1. The key documents:
1.1 Peaks: PCO0531607°*; PC0098230"*!; PC005S2776"; and PC0049702.70*
1.2 Coyne 2, paras 3.211 — 3.219.
1.3 382.2
1.4 JR320°
Summary
2. I Mr Coyne states that Bug 11: Girobank is a bug with lasting financial impact. Post Office
submits that there is no evidence of any financial impact on branch accounts. A number
of the Peaks arose in testing. The Peaks that arose in the live environment do not indicate
evidence of branch impact.
Nature of this issue
POL00026925
POL00026925
3. Mr Coyne has drawn together three distinct issues under the heading of
“Bugs/Errors/Defects introduced by previous applied PEAK fixes”:
3.1 Para 3.2127” relate to issue 1 (PC00531607""%) (“Issue 1”).
3.2. Paras 3.213 — 3.216°” relate to issue 2 (PC00982307™°) and (PC00970817*!)
(“Issue 2”).
2030 (F/57},
2031 F/B}.
2032 1/53}.
2033 F/35}.
2084 (1)2/4.1/70} to (D2/4.1/72}
2035 (D1 /2/21}
2086 {D1/4}.
2037 {1)2/4.1/70}.
2038 (F/57}.
20 ¢1)2/4.1/70} to {D2/4.1/71}.
2000 fF/184}.
204 4F/164}.
507
AI8/507
4.
3.3 Paras 3.217 — 3.2197” relate to issue 3 (PC0052776""""); (PC00497027*); and
(PC00475187"**) (“Issue 3”).
Mr Coyne states that this bug had lasting impact on branch accounts.
Issue 1
5.
Peak PC00531607" refers to a report by a Fujitsu test team member on 29 August 2000
which concerns an issue with the Training Counter which was freezing when a delegate
mis-heard or miskeyed a keying sequence when completing a transaction log report.
Further testing by Fujitsu demonstrated that the issue could also be present in the live
environment, but that there was no evidence that it was. A software fix was deployed
on 6 September 2000.
On the basis of Peak PC0053160,7°4” Mr Coyne has concluded that the fix implemented
caused regression bugs. Although there are references to regression bugs in Peak PC
PC0053160,?"* further investigations by Fujitsu have concluded that rather than a
regression, this was an error in the way that the test rig was setup; an approved
combination of work packages was not being used.
Despite this bug appearing in Mr Coyne’s analysis of branch affecting bugs, there is no
evidence that Issue I had an operational or financial impact on branch accounts.
Issue 2
8.
Peak PC0098230"™ refers to an issue reported by a SPM on 13 January 2004 relating to
a discrepancy with his cash account. Fujitsu called the SPM to obtain further details and
ascertained that when the SPM was declaring his cheques, the value of cheques declared
as stock doubled.
POL00026925
POL00026925
2 (12/4.1/71} to {D2/4.1/72}
2043 £F/53}.
20 1/35}.
2088 (F/29.1}.
2046 (F/57}.
2047 (F/57}.
2088 (F/57}.
29 (F/184}.
508
A/8/508
POL00026925
POL00026925
9. The issue was diagnosed as a code regression relating to the fix implemented in Peak
PC0097081.2°°° A work-around was proposed two days later to the SPM, on 15 January
2004, that the SPM should follow a different procedure when declaring his cheques. A
software fix was thereafter released for the code regression.
Issue 3
10. Peak PC00497027°! and Peak PC0052776*°» refer to issues discovered by Fujitsu in
its standard development testing.
11. Peak PC0049702”"* refers to an issue raised on 7 July 2000 with a payments discrepancy
that was discovered during Fujitsu testing at the Danby House branch, which was used
as a pilot site. This was a regression issue as it arose following the fix being implemented
in Peak PC0047518.7°4 Peak PC00475187° refers to an issue also raised during Fujitsu
testing on 13 June 2000 that the C/A reconciliation at the counter was not totalling the
payments and receipts fields correctly.
12. Peak PC0052776**° was raised on 21 August 2000 and refers to the same discrepancy
issue reported in Peak PC0049702.”°*” This issue arose during further testing by Fujitsu.
Peak PC00497027"** was closed on 21 August 2000 and Peak PC0052776”"® was closed
on 31 August 2000 following a software fix being implemented.
13. Although included in Mr Coyne’s analysis of branch affecting bugs, it is clear from the
two Peaks that they relate to issues raised during Fujitsu testing. Given this, Issue 3 had
no financial or operational impact on branch accounts.
2080 (F/164}.
2051 (F/35}.
2052 (F/53}.
2053 (F/35}.,
2064 (F/29.1}.
2055 F/29.1}.
2056 (F/53}.
2057 {F/35}.
2058 (F/35}.
2059 {F/53}.
509
A/8/509
Conclusion on Bug 22: Bugs/Errors/Defects introduced by previous applied PEAK fixes
14,
Despite this bug appearing in Mr Coyne’s analysis of branch affecting bugs, a work-
around for the SPM who reported Issue 2 was implemented within 2 days, and a software
fix was implemented thereafter. The financial and operational impact on their branch
accounts was therefore limited. Issues 1 and 3 had no financial or operational impact on
branch accounts.
S10
POL00026925
POL00026925
A/8/510
POL00026925
POL00026925
Bug 23: Bureau de Change
1, The key documents:
1.1 PCO1297677°; — PCO1S17877°!;, — PCO137437%; — PCO2000427%;
PC02000907; — PC02004357"; — PCO201340°°; — PC02092407°%”;
PC02265737°; PC02544477; PC0260834?™,
12 382207
1.3 KEL AChambers2252R”"”; AgnihotriV917N”””*; Agnihotriv245L7°%
Summary
2. I Mr Coyne states that Bug 23: Bureau de Change discrepancies is a bug with lasting
financial impact. There are three issues identified within this heading. There is no
evidence of a bug in Horizon. Each issue is an example of user error in Horizon.
3. _ The experts agree that bugs that involve reference once discovered, they could be
quickly fixed (by a change to the reference data) once the bug is correctly
identified.2075 However, in this case although bureau de change is a particular
product, the Peaks cited by Mr Coyne do not indicate a bug in Horizon but rather user
error.
2060 (F/318.1}.
2061 {F/430.1}.
2062 1F/346.1}.
2063 {F/662.1}.
2064 1/663}.
2065 1F/665.1}.
2066 {F/678.1}.
2067 {F/785.1}.
2068 {F/1092. 1}
2069 (F/1549.1}.
207 (F/1667.1}.
21 41/2/21} to {D1/2/22}.
2072 4F/316}.
2073 1/668}.
204 1F/667}.
2075 (1)1/4/7} para 4.4.
Sil
AI8/511
Nature of this issue
4.
In JS2 Mr Coyne mentions three Peaks under the heading of “Bureau de Change”:
(1) PC0129767°°” (“Issue 1”).
(2) PC0137437°°”7 (“Issue 2”).
(3) PC0151787°°’ (“Issue 3”).
These Peaks relate to reversals and declarations concerning foreign currency. They are
described further below.
Mr Coyne confirmed during his cross-examination that he believes that these issues
had a lasting financial impact.”°” However, as noted below there is no evidence to
support this.
Issue 1
7.
In JS2 Mr Coyne stated that Peak PC0129767 was an example of Horizon allowing the
reversal of the same transaction twice, impacting branch accounts.”’*° While this
incident did have an impact on branch accounts, it was caused by the Subpostmaster
carrying out a reversal incorrectly rather than a bug, error or defect in Horizon.”**!
The Subpostmaster attempted to reverse a transaction for the sale of 1,000 euros.
However, the Subpostmaster entered the transaction ID for the cash settlement entry
rather than the transaction ID for the currency/margin entry, which meant that only the
cash settlement of the transaction was reversed. The Subpostmaster then adjusted their
stock to remove the currency they had failed to reverse and this left the margin for the
transaction as a loss. This represents user error.
POL00026925
POL00026925
2076
2077 (F/346.1}.
2078 ¢F/430.1}.
20 {Day17/133:3} to (Day17/133:4}.
208 {TD 1/2/21}.
2081 {F/318.1/4}.
AI8/512
9. Areversal receipt would have been generated by Horizon and this would have shown
that the Subpostmaster had reversed the incorrect entry. Indeed, the Peak shows that
the Subpostmaster raised the issue with Post Office leading to the Peak being created.
10. There is no evidence to suggest that the discrepancy was not rectified, either by the
Subpostmaster reversing the transaction correctly or by a transaction correction being
issued.
Issue 2
11. In Peak PC0137437 the Subpostmaster reversed the cash settlement part of the original
bureau transaction rather than the currency and margin part of the original bureau
2082
transaction. This is the same thing that happened in Issue 1, although Mr Coyne has
correctly identified that Issue 2 represents user error in JS2.7°%
12. The Peak concludes by stating that the Subpostmaster needed “to reverse the currency
and margin part of the original bureau transaction”. This would have removed the
discrepancy in the branch accounts.
Issue 3
13. In Peak PCO151787 the Subpostmaster reported a discrepancy on his main stock unit
that he believed was related to currency transactions he had undertaken. However, the
Peak shows that Fujitsu investigated the matter and concluded that the Subpostmaster
had been making incorrect cash declarations, declaring his cash at one value and then
performing a rem in or transfer which would effect this value, but subsequently
declaring his cash at the original value.
14. Post Office has subsequently reviewed ARQ data for the branch and there is nothing to
suggest that the branch’s remming in /out of foreign currency caused a loss in the
branch.
15. This is another example of user error.
POL00026925
POL00026925
2082 (F/346.1/3}.
2083 11/2/21}.
S13
AI8/513.
Other Peaks referred to in JS2
16. None of the remaining Peaks referred to in JS2 (PC0200042°**; PC02000907°;
PC02004357*; PCO2013407%’; PCO2092407"**; PCO226573°; PC02544477°";
PC0260834) had an impact on branch accounts.
Conclusion on Bug 23: Bureau de Change
17. These three issues are all examples of user error.
18. Issues 1 and 2 (incorrect reversals) could be rectified by the Subpostmaster performing
the reversal again correctly or via a transaction correction. Because Issue 3 relates to
the Subpostmaster declaring their cash incorrectly, it is not clear whether declaring the
cash correctly would have resolved the matter or if there would have been a
discrepancy caused by another issue or issues in the branch.
POL00026925
POL00026925
2084 £F/662.1}.
2085 (F/663}.
2086 {F/665.1}.
2087 4F/678.1}.
2088 {F/785.1}.
2089 {F/1092.1}.
20% (F/1549.1}.
2091 {F/1667.1}.
514
AI8/514
Bug 24: Wrong Branch Customer Change
The key documents:
1.1 Peaks: PCO1282647°” and PCO12979 179,
1.2 Js2?7,
Summary
2.
Bug 24: Wrong Branch Customer Change is a bug with the potential for lasting financial
impact. This is a reference data bug. The experts have agreed that that while reference
data bugs may be a significant proportion of the bugs with financial impact, once
discovered, they could be quickly fixed (by a change to the reference data) once the bug
is correctly identified.?”° This was the case with Bug 24. The issue would have visible
to the SPM as the incorrect quantity would have displayed on the screen. Fujitsu
identified the root cause and developed a fix within two weeks of the issue being reported
by the SPM.
Nature of this issue
POL00026925
POL00026925
3. This issue relates to quantities of stamps and postage labels (Smartpost Transactions)
not correctly resetting to 1.
4. I When a quantity of greater than 1 was entered for a Smartpost Transaction, the quantity
was not reset to I when the clerk moved on to the settlement screen”””’. This could result
in subsequent items in the session being multiplied by whatever quantity remained and
could affect further items being sold or the amount being tendered towards settlement.
5. Peak PCO128264 was opened on 4 November 2005 as a result of a SPM reporting the
issue on 4 November 2005. The matter was passed to Fujitsu’s development team for a
software fix on around on 10 November 2005 and a fix was implemented on 18
November 2005. The Peak shows that Fujitsu suspected that the problem was introduced
202 4/310}.
2093 4/317},
20 1/2/22}.
2095 (D1/4/7} para 4.4,
296 Symptoms outlined in KEL AChambers4134R {F/319}.
SIS
AI8/515
POL00026925
POL00026925
by changes to the Smartpost Transactions that had been implemented from 24 October
2005.
6. On 6 December 2005, a further instance was reported (Peak PC01297917°"). The root
cause was identified and it was found that this issue related to Peak PC0128264 which
documented the fix that had been put in place. On 7 December 2005, Fujitsu found that
the fix had not been applied to a group of branches and the reference change data fix was
then implemented overnight to the remaining branches.
7. The first fix and change to the reference data was applied to the active group of branches,
group 111111113. It was live and effective on 18 November 2005””*. However, after the
further instance of the issue was reported, that led Fujitsu to investigate further and
realise that there was an active group of branches, group 111111112, that hadn’t received
the fix?’ As explained above, the fix and change to the reference data was applied to
group 111111112 overnight on 7 December and was effective from 8 December 20057).
Impact
8. This issue could have only been active for a small period of time, from when the changes
to the Smartpost Transactions were made on 24 October 2005 to 7 December 2005 when
the second tranche of branches received the fix.
9. The issue would have been visible to the SPM as the incorrect quantity would have
displayed on the screen. If the SPM did not notice the quantity had changed then they
could have:
9.1 Charged the customer for item(s) the customer did not receive which would likely
cause a gain for the branch; or
9.2 Potentially hand too much change to the customer which would likely cause a loss
for the branch.
2097 4F/317},
2098 £F/310/4}.
2099 {F/317/2}.
2000 4F/317/3}.
516
AI8/516
13.
POL00026925
POL00026925
Mr Coyne correctly categorises the bug in JS27!!
as being caused by an issue in the
reference data. His assertion that this would “/ead to the operator” providing the branch
customer with the wrong amount of money is not strictly accurate.
As explained above, the incorrect value displayed on the screen would have been visible
to the user and relatively easily identified by the user because if the incorrect quantity
was carried through as this would affect the basket total and therefore change the total
amount of cash and stock shown to the SPM on the Counter. This therefore means that
the SPM had two opportunities to spot an occurrence of this issue: (1) by spotting the
quantity error; and (2) by spotting that the basket total had changed.
As long as the issue was identified by the user and the correct value/amounts calculated
by the user, there should have been no discrepancy in the branch accounts.
Notwithstanding the above, if the user did not notice the issue, it is possible that this
could lead the SPM to give more change back to the customer than they should have,
resulting in a loss and/or gain to the branch accounts.
Fix applied
14.
Fujitsu identified the root cause and developed a fix within two weeks of the issue being
reported by the SPM. While there was an issue with the fix being rolled out,
PC01297917™ confirms that Fujitsu looked into the problem of the fix not being applied
fully to an active group of branches and how this could be avoided in the future. This
is a good example of Fujitsu acknowledging what had gone wrong and adopting a
lessons-learned approach moving forward regarding the deployment of fixes.
201 ¢D)1/2/22}.
202 4F/317/2} to {F/317/3}.
S17
AI8/517
Bug 25: Lyca top up
The key documents:
1.1 Peaks: PC0202925*°*; PC02031087"*; PCO2031377!°; PCO2032847!%;
PC0202894*'"”; and PC0203215.?!°*
1.2. KEL: ballantj020J.2"
1.3 JS2?!° and JR42!""
Summary
2.
Bug 25: Lyca Top Up is a bug with the potential for lasting financial impact. This is also
reference data bug. As set out above, the experts have agreed that that while reference
data bugs may be a significant proportion of the bugs with financial impact, once
discovered, they could be quickly fixed (by a change to the reference data) once the bug
is correctly identified.2!”? This was the case with Bug 25. This issue was identified
through Fujitsu's automatic reporting — the NB102 report.”""9
Nature of this issue
POL00026925
POL00026925
3. Peak PC02031087!"* related to incorrect reference data, rather than a fault in the Horizon
software. This is a point that the experts agree on.?!!5
4. The issue relates to Lyca top up transactions for mobile phones. These transactions are
entered on the Horizon counter by the SPM and, once the transaction is processed by
2103 (F/692}.
2104 (F694),
205 1F/696.1}.
206 {F/699.1}.
207 (F/691.2}.
208 (F/697.1}
210 (F/698}.
20 (191/2/23}.
2 {D1/4/7}.
2? 191/4/7} para 4.4.
2113 PCO203215 {F/697.1} and PCO203284 {F/691.2}.
21M (F/694}.
218 D1/4/7}.
S18
Al8/518
POL00026925
POL00026925
Horizon and authorised by E-Pay (the financial institution), a receipt is printed which the
SPM should provide to the customer. It is this receipt that contains the customer’s
voucher to apply the top up to their mobile phone.
As shown in the Peak, Lyca top up transactions were being accurately recorded by
Horizon, but the counter was unable to process the authorisation response returned from
E-Pay which resulted in an error message being displayed on the counter and the SPM
2116
being logged off before the Lyca top-up receipt was printed. What happened next
was dependent upon the action taken by the SPM when they logged back into the counter:
5.1 As shown in Peak PC0203108,7!'” if the SPM recovered the transaction and
incorrectly confirmed on Horizon that the top up had been successful, despite no
top up receipt having been printed, the transaction would be recorded in the
branch’s accounts, meaning the branch would likely have experienced a shortfall
to the value of the top-up as no money would have been taken from the
customer.”1'*
5.2 If the SPM logged back into the counter and correctly confirmed that the
transaction had not been successful, a zero value transaction would be recorded
in the branch accounts and a reversal generated for the top up. This should have
resulted in the affected branch accounts being correct, but due to the reference
data issues the reversal being sent to E-Pay caused E-Pay to treat the top-up
incorrectly as a successful transaction. This is shown in Peak PC0203284.7!'°
Detection and resolution
6. The first known occurrence of the issue was recorded in Peak PC0202925 and reported
to Fujitsu on 13 August 2010.7!”
7, The issue would have been visible to the SPM in branch as they would have been logged
out of the counter and gone through the recovery process. It would have also been
2116 (F/694/1},
217 (F/694/2}.
2118 We can reasonably assume no money would have been taken from the customer as no receipt would have
been printed with the top-up voucher.
2119 {F/699.1/2}.
2120 (F/692/1}.
S19
AI8/519
POL00026925
POL00026925
possible to identify whether or not a transaction had been reversed from the reports
available in branch. This is supported by Peaks PC02031087'*! and PC02029257!?
which were both raised in response to SPMs reporting the issue to the NBSC.
8. As shown by Peaks PC02032157'?* and PC0203284,”!4 this issue was also identified by
Fujitsu’s automatic reporting using the NB102 report and as such is an example of
Fujitsu’s automatic reporting picking up issues that arise.
9. In terms of resolving any financial discrepancies with the affected branches:
9.1 As can be shown in Peak PC0203108,7!*5 once the issue had been reported,
Fujitsu would in turn refer the SPM to the NBSC for reconciliation with Post
Office.
9.2 Where the issue was identified as a result of Fujitsu’s automatic reporting,
Fujitsu would issue a BIMS Incident Report to Post Office. As shown by BIMS
Incident Reports BE/0203215?!7° and BE/0203284,”!”’ these reports would
summarise the particular occurrence and explain the potential discrepancy to
enable Post Office to reconcile the position with the branch.
10. A permanent reference data fix was released on 20 August 2010, Release Peak
Pco202925.71*
Conclusion on Bug 25: Lyca top up
11. This issue was identified through Fujitsu’s automatic reporting and by SPMs
reporting it. It was quickly resolved, with the reference data fix being issued in 7
days of the issue being reported to Fujitsu.
221 (F/694}.
2122 (F/692}.
223: (F/697.1}.
24 (F/691.2}.
225 (F/694}.
2126 1F/700.2}
(
{
2128 £F/692/5}.
520
A/8/520
POL00026925
POL00026925
Bug 26: TPSC250 report bug
1. The experts have drawn together five distinct issues under the heading of “7PSC250
report” in JS2:
1.1 KEL: AChambers253L7!”’ and 13 Peaks”? relate to issue 17°! (“Issue 1”).
1.2 Peaks: PC0122630;7!* PCO122631;7'* and PC0122544?'* relate to issue 2
(“Issue 2”).
1.3 PC01226647!%5 and JAnscomb1935Q”"* relate to issue 3 (“Issue 3”).
1.4 PC0122766 relates to issue 4 (“Issue 4”).
1.5 KEL Ballantj2547K7!*’ and PC0156718 relate to issue 5 (“Issue 5”).
Summary
2. This was a backend reporting problem and so the chances of branch impact are small.7!*
Of the five issues: issue I resulted in incorrectly flagged exceptions but no reconciliation
was required; issue 2 resulted in no financial or operational impact on a branch accounts
and was limited solely to the process of copying/ harvesting transactions to Post Office's
back-end systems; issue 3 did not result in a mismatch between the files sent to Post
Office and the branch data; issue 4 was flagged by automatic reporting and issue 5 could
result in a receipts and payments mismatch thus needing reconciliation.
2129 (F/449},
213 PCO115804 {F/251}, PCO118350 {F/259.1}, PCO117659 {F/255.1}, PCO118677 {F/261.1}, PCO119978
{F/262.1}, PCO120063 {F/263.1}, PCO122147 {F/276.1}, PCO122304 {F:278.1}, PC0122354 {F278.2},
PC0123056 {F/287.1}, PCO123058 {F/287.2}, PCO189625 {F’/540.1} and PCO115456 {F/249.1}.
231 {F/97}.
2182 (F/278.4}.
233 (F/278.5}.
2M {F/278.3.1}.
2135: (F/279.1}.
286 (F/279.2}.
2137 1/633}.
2138 ()1/2/24}.
A/8/521
POL00026925
POL00026925
Issue 1
3.
As can be seen from KEL AChambers253L,”"” this was a reconciliation reporting issue
which involved Peaks being automatically raised as a result of Fujtisu’ s reconciliation
reports incorrectly raising exceptions.
Despite the Peak?!“?
not being high priority (as there was no financial impact on branch
accounts), the issue was identified and a fix developed and deployed in a short period of
time. As indicated in the Peak it was opened on 10 February 2005 and, following
Fujitsu’s investigations, the KEL?!"! was raised on 14 February 2005 to address any
future occurrences of the problem and provide Fujitsu with a workaround to check that
the reconciliation reports were incorrectly flagging exceptions.
A permanent fix involving a code change was released to all branches in 2005 as part of
the s80 software upgrade.
Conclusion Issue 1
6. Mr Coyne states that the KEL explains the branch account impact.”!? Mr Coyne fails to
note the following which can be clearly seen from the Peak?!**:
6.1 On this occasion the relevant report, referred to in the Peak as the “check” has
simply not worked, resulting in the exception being incorrectly raised.
6.2 Fujitsu conducted a thorough investigation into the issue and concluded that no
reconciliation was required.
2129 {F/449},
2140 4/251},
24 1/449}.
22 (D1/2/24}.
243 4F/251/1} to {F/251/2}.
AI8/522
POL00026925
POL00026925
Issue 2
7.
10.
The experts reference Peaks PC01226307!* and PC01226317!*° alongside the Peaks
relevant to Issue 1 above.?'"
These Peaks were raised as a result of Fujitsu’s automatic reporting using the TPSC250
report. As can be seen in the EPOSS/TIP Reconciliation Controls — Procedures for Errors
and Exception Handling,”"*7
the TPSC250 Report shows a list of exceptions where the
totals for the transactions sent from the Host BRDB to Post Office’s back-end systems
do not match the Daily Transaction Totals calculated by the counters in branch. The
TPSC250 Report therefore raises alerts during the process of copying/ harvesting
transactions to back-end systems.
The experts do not identify the Master Peak PC0122544,7'4* which references 5 known
instances of the issue and details Fujitsu’s investigations into the issue and the release of
the fix. As can be seen in the Master Peak:
9.1 This issue relates to the ‘Virtual PO’. ‘Virtual PO’ is a term used by Fujitsu for
a collection of messages against the group ID ‘999998’, which contained entries
for all ‘open’ branches in Legacy Horizon. It was therefore not a real branch.
Data held in the Virtual PO was held on the correspondence server and was not
available to individual branches.
9.2 The immediate impact of the issue was that the affected branch counters would
not be harvested to Post Office’s back-end systems.
9.3. There was no suggestion that the issue would be able to impact branch accounts
directly.
In terms of resolving the issue, the Master Peak shows the following:
10.1 Fujitsu investigated the 5 different instances of the problem and identified that
there was an incorrectly formed field in the Virtual PO, specifically there was an
204 4F/278.4}.
245 (F/278.5}.
2146 {1/2/24}.
27 4F/100,2/15}.
248 £F/278.3.1}.
523
AI8/523
POL00026925
POL00026925
additional EOD ‘Date’ field, which caused the exceptions in the TPSC250 Report
as the report was unable to access the relevant EOD messages.
10.2 Fujitsu identified that the program responsible for writing the relevant EOD
messages had been amended as part of the s80 software upgrade which had taken
place the previous weekend.
10.3. A workaround fix was introduced for the 5 affected branches, which involved
the EOD messages that would be sent to Post Office’s back-end systems (not the
data in branch) being rewritten by Fujitsu without the additional EOD Date
field.?"”
(a) A permanent code fix was released to all branches as part of the s80R
software release, which would result in the EOD Date field being populated
correctly, with the date in dd-Mon-yyyy format.?!*°
Conclusion on Issue 2
11. Issue 2 had no financial or operational impact on a branch accounts and was limited
solely to the process of copying/ harvesting transactions to Post Office’s back-end
systems.
12. Despite the Master Peak!*! not being high priority (as there was no financial impact on
branch accounts), the issue was identified by Fujitsu’s automatic reporting and a fix
developed and deployed in a short period of time.
Issue 3
13. This involves an issue in the reconciliation between the counter reported totals in branch
and the TPS host generated totals.
2149 {F/278,3.1/2}
2150
28 fF/251},
524
AI8/524
14.
15.
13.1 Peak PC01226647! was raised as a result of Fujitsu’s automatic reporting
(specifically the TPSC250 report described above) to investigate why the TPS total
was showing as being higher than the counter total in branch.
As can be seen from KEL JAnscomb1935Q:7!*
14.1 The difference in totals in the TPSC250 report was caused by the same problem
identified in Issue 2 above involving the additional EOD Date field.
14.2 The additional EOD Date field resulted in the TPSC250 report adding the totals
for day 0 and day 1 together, meaning that the counter total was correct for day
1, but did not match the TPS total in the TPSC250 report.
14.3 This issue would be resolved as part of the fix identified in Issue 2.
As can be clearly seen from the ent4-ry in the Peak on 5 July 2005 at 06:51,7'%4 Fujitsu’s
investigations confirmed that there was “no impact on the files sent to POL”. This means
that while the TPSC250 report totals were incorrect, the actual files sent to Post Office
matched the data in branch. Consequently, no fix was required for this particular instance
in PC0122664.7'°5
Issue 4
16.
Peak PC0122766"'* relates to a single issue involving Riposte errors that affected a
counter in branch.
As shown in the Peak:
17.1 The Peak was raised on 4 July 2005 in response to the branch appearing in
Fujitsu’s TPSC250 report.
17.2. The route cause was identified as being Riposte errors on counter I in branch.”!57
17.3. By 7 July 2005, counter I had been replaced in the branch.
282 {F/279.1/1}.
2183 {F/279,2}.
254 (F/279.1/2}.
2185 {F/279.1/1}.
2156 {F/281.1}.
257 £F/281.1/2}.
525
POL00026925
POL00026925
AI8/525
POL00026925
POL00026925
17.4 Fujitsu were able to run a check that confirmed, as transactions had only been
performed on counter 2, the number, quantity and value of the counter 2
transactions matched the TPS totals in the TPSC250 report, meaning there was
no need for reconciliation.
18. This is a good example of Fujitsu’s automatic reporting swiftly alerting Fujitsu to an
issue in the branch, here arising from an issue with one of the branch’s counters. Fujitsu
clearly identified the origin of the problem and had rectified the issue within 3 days’ of
the Peak being raised.
Issue 5
19. The experts referenced Peak PC01567187'** alongside the Peaks relevant to Issue I
above.”
20. While this was originally suspected to be an instance of KEL AChambers253L,”"°
Fujitsu later confirmed that this was an instance of KEL Ballantj2547K.2"!
21. As can be seen from the KEL:
21.1 The issue resulted in the branch appearing in the TPSC250 report showing a
mismatch between the TPS ‘Absolute Value’ and the branch counter ‘Absolute
Value’, specifically:
(a) The TPS total and counter total values for the ‘Number’ and ‘Absolute
Quality’ columns were the same; and
(b) There was a difference in the ‘Absolute Value’ at the counter, which was
greater than the TPS value.
21.2 While Fujitsu was not able to reproduce the issue, Fujitsu was able to determine
that where the session nets to zero, this would have no impact on branches and
no reconciliation would be needed.
2158 {F/448.1}.
2199 (D1/2/24}.
21 £F/449},
2161
UES
526
AI8/526
21.3. In the event that the session did not net to zero, the branch would experience a
receipts and payments mismatch in branch and reconciliation would be required.
22. As shown in the Peak: Fujitsu investigated and diagnosed the issue within 6 days of the
branch showing in the TPSC250 report. Fujitsu were able to confirm that no
reconciliation was required because the branch did not appear in the incomplete
summaries report — i.e. the session did net to zero and there was no receipts and payments
mismatch.
Bug 27: TPS Bug
1. The key documents:
1.1 Peaks: = PC0141145?'%; PCO15S73577%; PCO1592737!%; PCO1968937'%;
PC0174587"'*; and PCO156718.7'%
1.2. KEL: ballantj2547K.7!®
1.3 Js22!7
Summary
2. Bug 27: TPS is a bug with the potential for lasting impact. Dr Worden is of the view that
this was a backend reporting problem and so the chances of branch impact are small.?!7!
POL00026925
POL00026925
216 (F/364}.
264 (F/452.1}
2165 (F/454.1}.
266 (F/604.1}.
2167 {F/480.1}.
2168 (F/448.1}.
210 15/633}.
2% (1)1/2/24} to {D1/2/25}.
27 11/2/24}.
527
AI8/527
Nature of this issue
3.
This is a reconciliation reporting issue that affected SmartPost transactions. The
SmartPost application was supplied by Escher and was designed to help users to
calculate the postage required for any item and print labels to attach to the relevant items.
Peak PC01411457!” related to a problem with SmartPost which wrote slightly corrupt
transactions.
In terms of the specific issue that arose:
5.1 The affected SmartPost transactions were either:
(a) Missing a grammar attribute and/ or had a corrupted grammar attribute. The
grammar attribute is used to calculate the value for the reconciliation
reporting, meaning that, as the attribute was either missing or corrupted,
reconciliation reporting errors were produced;”!”* or
(b) I Complete and not corrupted as far as the branch accounting was concerned.
Fujitsu were able to determine this by checking the session nets to zero, as
per the guidance in KEL ballantj2547K.?!*
5.2. The grammar attribute involved was used to calculate the ‘EPOSSDailyRecon’
absolute values, which in turn is what triggers the TPSC250 and TPSC257
exceptions where these is a mismatch between the counter and the TPS host
generated totals.
Detection and resolution
POL00026925
POL00026925
6. The issue was identified through Fujitsu’s automatic reporting using the Incomplete
Summaries Report and TPSC250 report and as such is an example of Fujitsu’s automatic
reporting picking up issues that arise.
7. Mr Coyne states that this issue does impact branch accounts.”!”> Mr Coyne only quotes
one extract from Peaks PC0196893 and PC0174587, which is a note from Fujitsu that
22 (F/364}.
273
F
{
{F/364/4}.
F
{
214 £F/633}.
275 4D) 1/2/24}.
528
AI8/528
POL00026925
POL00026925
this issue “may also have caused a receipts and payments error”. This gives a
misleading impression of both the issue generally and extent of Fujitsu’s investigations
and furthermore ignores the later content of these Peaks. There are two points to note in
response:
7.1 As clearly shown in the relevant Peaks:7!"° Fujtisu investigated each incident
raised. In order to determine whether any reconciliation is required, following
the guidance in KEL ballantj2547K,”!”’ Fujitsu ran a check to confirm that the
session for the affected transaction net to zero, meaning there would be no
receipts and payments mismatch and consequently no impact on the affected
branches’ branch accounts.
7.2. While it is acknowledged that this issue could have resulted in a receipts and
payments error, Fujitsu’s automatic reporting would have picked any such issue
up separately and it would have been dealt with through business as usual
processes.
Conclusion
8. It is clear from all of the Peaks referenced by the experts in JS2 that Fujitsu fully
investigated each of these instances and checked to see whether there had been any
impact on the affected branches.
9. The guidance in KEL ballantj2547K~!”S enables Fujitsu to identify any further instances
of the issue and ensure there is no impact on branch accounts.
2176 ¢F/452.1}, {F/A54.1}, {F/604.1}, (F/480.1} and {F/448.1}
2177 4/633}.
278 4F/633},
529
AI8/529
Bug 28: Drop and Go Bug
The key documents:
1.1 Peaks: PCO2602697!” (initial instance); and PC02732347'® (further instance).
1.2 KELs: carde235Q°'*!
1.3 3822!
Summary
2.
Bug 28: Drop and Go is a bug with the potential for lasting financial impact. This is also
reference data bug. As set out above, the experts have agreed that that while reference
data bugs may be a significant proportion of the bugs with financial impact, once
discovered, they could be quickly fixed (by a change to the reference data) once the bug
2183
is correctly identified.*'** This was the case with Bug 28.
Nature of this issue
3.
Peak PC02602697'* relates to an issue involving a Drop and Go transaction (a £100
mobile phone top up) that timed out on Horizon. The transaction did not appear to
complete from the SPM’s perspective, but it was recorded in the branch accounts. The
branch then processed the transaction again. While the customer was credited with £100,
the branch was debited £200 and this created a discrepancy of £100 in the branch
accounts which was resolved by a Transaction Correction being issued on 29 June 2017.
As explained below, the issue which led to Peak PC0260269 (a bug in the reference data/
script provided by ATOS rather than a fault in the Horizon software) was fixed by ATOS.
A similar issue arose subsequently (PC0273234) and was again fixed by ATOS.
POL00026925
POL00026925
279 {F/1660.1}.
2180 {F/1848.8.2}.
2181 £F/1660}.
2182 (1) 1/2/25}.
218 (D1/4/7} para 4.4,
2184 £F/1660.1}.
530
A/8/530
POL00026925
POL00026925
Detection and resolution
5. On 29 June 0217 the SPM contacted the NBSC and reported that there was a duplicate
transaction in their branch accounts.”!** This led to Peak PC0260269 being raised. The
Peak shows that Fujitsu:
5.1
5.2
5.3
5.4
Investigated the issue;
Identified that the potential root cause could be the relevant script failing on 5
July 2015;
Raised KEL carde235Q”'® on 5 July 2017 to enable any future occurrences to
be identified; and
Passed the issue to ATOS to investigate further as ATOS supply and maintain the
affected script.
6. Once the issue had been passed to ATOS, as per the Test Summary Report,'*’ ATOS
identified issues with two scripts:
6.1
6.2
Balance & Top Up script. This is a transaction script that captures and transmits.
data when a customer wants to Top Up the balance of their Drop and Go account.
As per the Test Summary Report, ATOS identified a bug in the script, referred to
as ‘Issue 1’ that incorrectly allowed the transaction to progress after an
irrecoverable timeout had been produced.?'**
Open Account script. This is a transaction script that captures and transmits data
when a customer wants to open a Drop and Go Account. As per the Test
Summary Report, ATOS referred to this as ‘Issue 2’ and confirmed that there was
the same issue as with the Balance & Top Up Script in that the transaction was
able to progress after an irrecoverable timeout had been produced.”!*°
2185 {F/1659.1}
2186 £F/1660}.
2187 £F/1787}.
2188 {F/1787.1/3}.
208 {F/1787.1/3}.
S31
A/8/531
POL00026925
POL00026925
These script issues caused the Drop and Go transaction to be committed to the branch
accounts, despite the counter having crashed.
As can be seen from the Test Summary Report, these script issues were fixed by 6 April
2018 following the release of updated versions of the script, specifically version 6.12 of
the Balance & Top Up Script and version 6.11 of the Open Account Script.7!”°
A similar incident led to Peak PC0273234*""' being raised on 21 August 2018.
As can be scen from the Peak,”'*? Fujitsu identified that this was a further instance of the
Drop and Go issue and swiftly connected this issue to the KEL*!*?
to ATOS.
and passed the matter
This further instance was investigated by ATOS who identified issues with a different
Drop and Go script, specifically the Count Mails transaction script. This can be seen
from ATOS? internal email chain from August — September 2018 which confirms an
updated script had been produced and that the script fix had been tested.7!°* The
reference data/ script fix was released to live overnight on 25 September 2018, meaning
the change would have been effective in all branches on 26 September 2018.
In terms of the Count Mails script functionality:
11.1 The Count Mails script was intended to provide the functionality for SPMs to
accept a number of Drop and Go transactions at the same time, registering the
number of items received and issuing a receipt to the customer for the number of
Drop and Go items purchased. The primary functionality of the Count Mails
script was therefore to capture the number of Drop and Go items purchased.
11.2 Once these additional items were purchased, the SPM would then perform a
separate transaction on Horizon to process the items individually after the
customer had left the branch. The purpose of this was to enable the customer to
purchase several Drop and Go items at the same time and enable the SPM to
process these on Horizon at a later stage — i.e, at a quiet moment in the branch
219 {F/1787.1/3}.
2191 (F/1848,8.2}.
2192 {F/1848,8.2/3}
213 {F/1660}.
2194 £F/1825.01}.
AI8/532
12.
POL00026925
POL00026925
when no other customers were present. It is only at this second stage that the
SPM is able to identify whether there is a sufficient balance on the customer’s
account to process the items.
11.3. Asa result of ATOS’ investigations into this further issue, they determined that
the Count Mails script has a secondary functionality that enables customers to
top up also. This is a particularly unusual transaction as it would require the
customer to have prior knowledge of their account balancing prior to performing
the transaction.
On this occasion the customer topped up their account at the same time as performing
several Drop and Go items, which would have triggered the Count Mails script. ATOS
were, at the time of the original script fixes in April 2018, unaware that the Count Mails
script had this secondary functionality, meaning it had not been addressed at the same
time as the April 2018 fix and had the same issues in that it allowed the relevant
transaction to complete after the counter had timed out.
Conclusion
13.
14.
15.
These issues were caused by bugs in the reference data/ script provided by ATOS, rather
than a fault in the Horizon software.
In his report Mr Coyne correctly notes that the first issue resulted in the customer being
credited with £100 but the branch being debited with £200, meaning the branch would
have experienced a discrepancy to the value of £100 and during his cross-examination
Mr Coyne opined that there is evidence of this issue having lasting financial impact.7!?°
However, a Transaction Correction was issued for £100 to correct the discrepancy on 29
2196
June 2017, as shown in the Spreadsheet of TCs for this particular branch*'”®. There was
therefore no lasting financial impact to the branch.
During Dr Worden’s cross-examination, Mr Green QC took Dr Worden to this Peak and
suggested that the original fix was insufficient as the same problem had subsequently
emerged.”!”’ As explained above, the similar issue related to a secondary functionality
of a completely separate Drop and Go script. As such, it is incorrect to suggest that the
2195 {Day17/133:19} to {Day17/133:21}.
2196 (F/1645.1}.
2197 {Day20/55:1} to {Day20/56:!}.
AI8/533
original script/ reference data fix simply did not work. Once the similar issue arose, it
was swiftly passed to ATOS to investigate, who identified the issue with the separate
Count Mails script and implemented a reference data/ script fix. There has been no
further known instances of the issue.
Bug 29: Network Banking Bug
1. The key documents:
1.1 Peaks: PCO1090207!°*; PCO109642*!*? and PCO142872°™.
1.2 KEL CHawkes1745L."!
1.3 Coyne 2, para 3184.70?
1.4 Second Joint Statement.”°*
Summary
1. Mr Coyne now states that this bug does not have lasting financial impact on branch
accounts. Post Office submits that there is no evidence of a bug in Horizon. Neither of
the primary Peaks referred to above can properly be described as instances of a “Network
Banking Bug”; both issues stem from intermittent communications failures emanating
from outside Horizon (most likely from systems/kit operated by BT).
2. This bug was under one of the sub-headings that Mr Coyne asserted fell under bug 22 —
”Bugs/Errors Defects introduced by previously applied Peak Fixes”. As set out above,
given that Mr Coyne subsequently included this bug in his list of non-lasting impact bugs
it is difficult to understand how Mr Coyne went from 13 to 22 bugs on the basis of
including this bug as a subcategory of bug 22 in JS2.
2198 {F/223},
2199 (F/223.1}.
2200 {F/379.1}.
2201 {F/272.1}
2202 {1)2/4.1/63}.
2203 {11/2/25} to {D2/1/26}.
534
POL00026925
POL00026925
AI8/534
POL00026925
POL00026925
3. Further, this bug is distinct from bug 22 in JS2 and is not a sub-category as Mr Coyne
suggests.
4. Two Peaks are referred to in the Second Joint Statement under the heading “Network
Banking Bug.” These Peaks represent two instances of intermittent communications
issues and are not indicative of a “bug” in Horizon.
5. Issue 1
6. In PCO109020, a Subpostmaster (branch FAD 286207) reported that her ISDN line was
down and that two customers’ pensions transactions had been declined. The transactions
were for £90 and £50 and the customers returned to the branch the next day saying that
the money had been taken from their accounts, even though the transactions had been
declined.
7. The Peak notes that the Subpostmaster advised that the £90 transaction had been
refunded and that therefore there was no need for it to be investigated. The
Subpostmaster requested that the £50 transaction be investigated; when the customer
complained, the Subpostmaster had refunded the £50 herself.
o> 22
8. This Peak is described as an example of a “Network Banking Bug”.°°* However, the
Peak demonstrates that the issue was ultimately an issue in BT’s communications line
(i.e. the cause of the issue was not part of Horizon). Two possible causes were
investigated in the Peak:
8.1 Various possible causes of the communications issue were considered and
investigated, including possible causes from within and outside of Horizon.
Various tests were undertaken to narrow down the potential cause. Ultimately, an
intermittent fault was discovered with BT’s communications line. This is not an
issue within Horizon and so it was passed to BT for investigation.
8.2 A possible issue with errors being wrongly reported by CNIM was considered.
PC0109020 was cloned to PC0109642 to investigate this potential issue.
PC0109642 concluded that CNIM was working correctly and that, in any event,
since the branch migrated to the ADSL service, there had been no further issues.
The Peak notes that “The rest of the estate is also migrating towards one or other
2204 {11/2/25}.
535
AI8/535
10.
POL00026925
POL00026925
of the “always connected” services, so again this particular issue will have a
reducing (to nothing) impact on the service” .??°
PC0109020 refers to KEL CHawkes1745L, which is a generic KEL relating to online
services failures that covers a wide range of services and possible reasons for failure.
The purpose of the KEL is to ensure that Fujitsu’s 1°‘ and 2™ line teams capture as much
relevant information as possible to aid investigation, and to provide further pointers as
to areas to investigate. The KEL is not specific to the issue in PCO109020 and does not
provide a specific fix or workaround for PC0109020.
The BT communications issue was outside of Horizon and was referred to BT to
investigate; there was therefore no specific fix for Fujitsu to implement. In any event,
as noted in PCO109642, the branch had experienced no further issues after migrating to
ADSL.
A BT communications failure of this kind does not constitute an issue within Horizon.
The communications issues created a brief window (of approximately one day) in which
money had been electronically debited from the customer’s account despite the customer
not physically receiving the money, but before the counter had completed an automatic
reversal of the debit. The delay caused by the communications issue meant that the
customer had time to notice that the money had been debited, complain to the
Subpostmaster and receive a refund, before the reversal took effect. Had the
Subpostmaster not provided the refund at that point, on the basis that there was no
completed transaction and that the system would shortly reverse the debit, then there
would have been no loss incurred in the branch.
Issue 2
12.
13.
In PC0142872 (branch FAD 123025), the Subpostmaster reported intermittent issues
with online services being unavailable. No discrepancies in branch accounts were
reported.
The Peak demonstrates that Fujitsu diligently applied KEL CHawkes1745L in seeking
to obtain as much information as possible from the Subpostmaster in order to diagnose
the issue. After a week, with the branch experiencing no further issues, the Peak is
2205 4F/223.1/6}.
AI8/536
POL00026925
POL00026925
closed. The intermittent communications issues are thought most likely to be as a result
of adverse weather; in any event, they are not found to be caused by anything within
Horizon.
14. The Peak notes that, should the office experience similar issues in the future, the issue
will need to be referred to BT/Post Office in order for the communications line to be
fixed. However, at the time of the Peak, there was no specific fix that Fujitsu could
implement.
15. As noted in the Second Joint Statement”, no financial impact is recorded in
PC0142872; there is no reference in the Peak to any discrepancy arising as a result of
the communications issues.
Conclusion
16. Neither of the primary Peaks referred to above can properly be described as instances of
a “Network Banking Bug”; both issues stem from intermittent communications failures
emanating from outside Horizon (most likely from systems/kit operated by BT).
17. There was demonstrably no impact on branch accounts in PC0142872.
18. The £50 loss experienced by the branch in PCO109020 was caused by user error.
19. In cross-examination, Mr Coyne accepted that the Peaks collectively referred to as the
“Network Banking Bug” provided no evidence of lasting financial impact.?2°7
2206 41) 1/2/26}.
207 {Day17/133:) } to {Day1 7/1341}.
537
AI8/537
POL00026925
POL00026925
APPENDIX 3: AUTOMATIC REPORTS
Introduction
1. Anumber of automatic reports are regularly run in Horizon that enable Post Office and
Fujitsu to identify and investigate potential discrepancies in branch accounts across the
networks. Both parties’ experts acknowledge that these reports are run and improve
Horizon's robustness.
2. The contemporaneous documents show that, in response to exceptions appearing in these
reports, Fujitsu would raise Peaks to investigate, identify and resolve both potential and
actual discrepancies in branch accounts across the network.
3. Both parties’ experts gave evidence about Fujitsu's automatic reporting during their
cross-examinations. Mr Coyne gave evidence during cross-examination that this
automatic reporting consisted of a "suite of reports that is handed over between Fujitsu
and Post Office automatically each morning"2°°> Dr Worden referenced routine
monitoring” and automatic detection involving Post Office's back-end
reconciliation.””!°
4. The reports summarised within this appendix are:
4.1 NB102 Exception Summary;
4.2 The TPS Report Set;
4.3 TPSC256 Host Detected Cash Account Lines Comparison Report/ Program.
2208 Day15/63:5-6}.
20 {Day20/178:19-20}.
2210 (Day20/179: 1-2}.
AI8/538
POL00026925
POL00026925
NB102 Exception Summary
5. This report contains twelve sub-reports which show discrepancies between financial
institutions' view of what has happened and Horizon's record of the same transaction.””!!
6. As can be seen from the End to End Reconciliation Reporting document,”’!” these
reports:
6.1 Provide the details of the transaction, including the transaction type, the receipt
2213
date and type and the type of discrepancy reported;**!* and
6.2 Are produced daily and delivered to Post Office and Fujitsu at 08:00, the day
following the report being run — i.e. the report run and received by Post Office on
Tuesday morning will contain any exceptions that appeared from transactions
performed during the Monday before.??!4
7... Mr Coyne gave evidence in cross-examination that he was aware of the above
reconciliation process between Post Office's information relating to transactions and the
information held by financial institutions. Specifically, Mr Coyne agreed that automatic
reconciliation would take place”?!
and where there was a discrepancy between Post
Office's information and the relevant financial institution's information, an exception
would appear in the appropriate report.”!° (Furthermore, in Mr Coyne's first report he
also makes reference to exceptions or error states appearing in the NB102 report, where
there are incomplete, corrupt or duplicate components within a transaction.”!”)
8. Peak PC01561747?'* shows Fujitsu responding to an exception in the NB102 Exception
Summary report. This can be seen from the second entry in this Peak dated 25 March
2008 at 10:33,2?!° which indicates that the Peak was raised in response to the exception
2211 (F/1678/9 - 16}.
2212 {F/1678}.
2213
{RASTA
214 (F/1678/9}.
2215 Day 4/74:19}.
216 (Nayl4/74:23: i {Davi 4/75:3' 75:34
217 Para 6.35 {D2/1/117}.
218 (F/446.1}.
2219 (F/446.1/1}.
539
AI8/539
raised for this particular branch in the report from the previous day. In terms of Fujitsu's
investigations into the exception, the Peak shows the following sequence of events:
8.1
8.2
8.3
Fujitsu identified that the discrepancy in the NB102 report was as a result of the
counter crashing part way through the recovery of the relevant transaction. As no
one had logged back into the counter since it crashed, the transaction has not yet
been recovered, meaning there was a difference between the branch's records and
the financial institution's records.
Andy Keil of Fujitsu telephoned the branch to explain that they would need to log
back into the counter and follow the recovery messages on screen to rectify the
discrepancy.
The Peak was then closed later that day.
9. Further to the above, a second Peak, PC0156246,””° was raised the following day, which
related to a connected issue with the same branch. As can be seen from the Peak:
91
9.2
9.3
The Peak was raised in response to the branch showing a new exception in the
2221
NB102 Exception Summary Report.’’*
The new exception was connected to the previous day's Peak. Specifically, the
relevant steps taken by Fujitsu the day before were summarised in the second Peak,
including that the branch was contacted and the SPM was told how to recover the
messages by logging back into the counter.
Fujitsu confirmed that, while the relevant transaction messages had been recovered
by the SPM, it appeared that these had been declined in the branch. Fujitsu deduced
that the likely reason for the recovery messages being declined was that no money
changed hands in branch during this transaction. Consequently, this would mean
the financial institution's view of the transaction would be incorrect, as their
POL00026925
POL00026925
2220 £F/446}.
2221 4F/446/1}.
540
A/8/540
reports would show that the transaction was authorised and the end-customer's
account debited in error. 7°”?
9.4 Post Office was asked to contact the SPM to "double check that no money changed
hands for certain". After checking no money had changed hands, the BIMS was
issued to Post Office.
9.5 Fujitsu left the Peak open for a further 24 hours to allow for any further enquires
relating to the exceptions. The Peak was then closed the following day.
10. Mr Coyne only refers to the second Peak - PC0156246””4 - in his report and says that
this Peak “emphasises the need for sufficient process adherence and clarity between Post
Office and the support teams in order to appropriately identify and correct
discrepancies".?**> These Peaks are in fact a very good example of the processes in place
working to enable Fujitsu and Post Office to identify, investigate and resolve
discrepancies.
11. Mr Coyne continues to state that the wording in the Peak 'It is likely the branch balanced’
is not a "clear diagnosis"?”® This is an unfair observation in circumstances where the
Peaks make it clear that Fujitsu asked Post Office to contact the SPM to check what
happened in branch, before taking steps to resolve the financial discrepancy.
The Transaction Processing System ("TPS") Report Set???”
12. The TPS Report Set consists of three reports: TPSC250; TPSC254; and TPSC257. The
TPS Report Set concerns issues that arise during the process of copying/ harvesting
transactions to Post Office's back-end systems, namely POLSAP and POLMIS.””**
POL00026925
POL00026925
222 {F/446/1}.
223 4F/446/2}
2224 4B /446},
225 11)2/4,1/62}
2226 £1)2/4.1/62}
2227 Note that Mr Coyne refers to all 3 of the TPS Report set in JS1 at D2/1/201
228 1F/976.1/37}
S41
AI8/541
13.
Both experts in their first reports discuss the TPS Report and agree that the reports
contained information that would allow Post Office to identify errors and consistencies
which could lead to shortfalls in branch accounts.”””?
TPSC250 Host Detected Transaction errors report
14.
16.
This report shows a list of exceptions where the totals for the transactions sent from the
Host BRDB to Post Office's back-end systems do not match the Daily Transaction Totals
calculated by the Counters in branch.??°°
An example TPSC250 Report is at {F/1687/63}, where it can be seen that there is a
difference between the 'TPS' Total and the 'Counter' Total, generating the discrepancy,
meaning something has gone wrong during the copying process to Post Office's back-
end systems.
Peak PC0109772*! shows Fujitsu responding to an exception in the TPSC250 Report.
This can be seen from the first entry in this Peak dated 18 October 2004 at 14:10, which
indicates that the Peak was raised as a result of this branch appearing in the TPSC250
Report from the previous day. In terms of Fujitsu's investigations into the exception, the
Peak shows the following sequence of events:
16.1 Fujitsu identified that there was a missing attribute for a particular transaction,
here being a £4.78 'mails' transaction.
16.2 Asa result of this missing attribute, Post Office's back-end systems rejected the
transaction, causing the difference in totals between the branch account total and
the TPS total.??
16.3 Fujitsu resubmitted the transaction with the missing attribute to Post Office's back-
end systems. As per the Peak entry on 19 October 2004 at 16:01, Fujitsu raised
POL00026925
POL00026925
222 ()2/1/201} and {D3/1/43}
228 (F/976.1/37}.
2231 (F/228},
2222 4F/228/1}.
AI8/542
17.
POL00026925
POL00026925
an OCR in accordance with its change procedure, evidencing that proper
authorisation was obtained by Fujitsu prior to repairing the transaction.”**
This is a good example of Fujitsu's automatic reporting detecting issues with data that is
copied to Post Office's back-end systems.
TPSC254 Harvester Exceptions
18.
20.
21.
This report shows a list of exceptions generated during the copying process from the
Host BRDB where there has been a failure to process one or more messages to Post
2234
Office's back-end systems.”
An example TPSC254 Report appears at {F/896/66}. The "Exception Detail’ contains the
details of the message and reason for the branch appearing in the report.
KEL GCSimpson5834N~** is associated with the TPSC254 Report. As can be seen from
this KEL, this particular instance:
20.1 Involved an exception that occurred during the branch's usual rolling over, while
. . . : »
revaluing foreign currency, specifically Estonian Kroon.??*°
20.2 Related to an issue where there was a missing attribute in the message, namely the
exchange rate. The exchange rate was no longer available in branch for the foreign
currency. The reason for this was that this currency was ‘unwanted’ and as such
should already have been remmed out by the branch.?257
In response to the branch appearing in the TPSC254 Report, as shown in the KEL, Fujitsu
informed Post Office of the issue to ensure that the SPM took steps to "remove the
unwanted currency at the branch by remming it out, so that they can avoid future
problems each time the SU is rolled over". ***
233: (F/228/2}.
26 1F/396/66}.
5 {F/1707}.
2236 {F/1707/1}.
2287 {F/1707/1}.
2238 {F/1707/2}.
543,
AI8/543
22.
POL00026925
POL00026925
As such, this is a good example of the TPS Report Set alerting Fujitsu to a problem
caused by the SPM failing to correctly Rem Out withdrawn currency.
TPSC257 POLFS Incomplete Summaries Report
23.
24.
25.
26.
27,
This report is run on a daily basis and shows those branches where the net total of the
transactions in branch (i.e. the value of credits less debits) does not net to the value of
zero, as expected.”
An example TPSC257 Report appears at {F/896/67}. This shows those branches where
the net total of the transactions performed that day did not net to zero. During cross-
examination, Mr Coyne agreed that this report picks up receipts and payments
2240
mismatches in branches***° and furthermore confirmed that he had seen Peaks that
discussed Fujitsu's investigations in response to exceptions being raised in the TPSC257
2241
Report.”
This would plainly be an enormously useful way of spotting any bugs that were somehow
causing transactions to be incomplete.
TPSC256 Host Detected Cash Account Lines Comparison Report/ Program
This report generates a list of discrepancies, where receipts and payments do not match
2242
in individual cash account lines in branch.’**? An example TPSC256 Report appears at
{F/31.1/39}. This shows the details of the FAD code, trading date, receipts total and
payments total for the branch on that particular day.
Peak PC0120064°** shows Fujitsu responding to an exception in the TPSC256 Report.
This can be seen from the entry in this Peak dated 22 April 2005 at 10:00.
2229 {F/896/67}.
2240 {Day15/65:2
24 {Day15/65:7), 4
222 (F30.1.1/33}.
3
544
{F/30.1.1/33}
AI8/544
POL00026925
POL00026925
28. In terms of Fujitsu's investigations into the exception, the Peak shows that Anne
Chambers of Fujitsu was able to review the report and investigate the relevant
transactions. Following Anne Chambers' review, she was able to determine that there
had been no loss or gain in branch on this occasion, but raised a KEL in case the issue
re-occurred in future.?*4*
2244 £F/263/2}.
S45,
AI8/545