RLITO000005
RLITO000005
Case No: HO16X01238, HO17X02637 and HO17X04248
Neutral Citation Number: [2019] EWHC 3408 (OB
THE POST OFFICE GROUP LITIGATION
IN _THE HIGH COURT OF JUSTICE
QUEENS BENCH DIVISION
Rolls Building Fetter Lane London, EC4A 1NL
Date: 16 December 2019
Before :
Between :
Alan Bates and Others
Claimants
- and -
Post Office Limited
Patrick Green QC, Kathleen Donnelly, Ognjen Miletic and Reanne MacKenzie for the
Claimants (instructed by Freeths LLP)
Antony de Garr Robinson QC, Simon Henderson, Owain Draper and Rebecca Keating for the
Defendant (instructed by Womble Bond Dickinson LLP)
Hearing dates: 11, 12, 13, 14, 18, 19, 20, 21 and 27 March;
3, 9 and 12 April; 4, 5, 6, 7, 11, 13 and 14 June 2019;
land 2 July 2019
Draft distributed to parties on 28 November 2019
Mr Justice Fraser:
1.This judgment is part of the long-running legal dispute between the claimants, who are all either
former or serving sub-post masters and sub-post mistresses (“SPMs”), and the Post Office. All the
RLITO000005
RLITO000005
SPMs in the litigation had contracted with the Post Office at different times to run branch Post Offices
in different locations across the country. The Post Office introduced a computer system called Horizon
in 2000 across all its branches. That was changed in 2010 to an online version called Horizon Online
or HNG-X, and the former version is now called Legacy Horizon. The claimants’ case is essentially
that both Legacy Horizon and Horizon Online (which used many elements of the existing system) were
or are unreliable, and this led to unexplained shortfalls and discrepancies in their branch accounts.
The Post Office denies this, asserting that the systems were and are robust, and extremely unlikely to
be the cause of such matters. There are numerous different causes of action brought against the Post
Office, and the Post Office counterclaims against the claimants, including seeking damages for fraud.
This is group litigation under CPR Part 19.
2.This judgment concerns the operation and functionality of the Horizon system itself. It follows the
lengthy Common Issues judgment, which is Judgment (No.3). That judgment was extremely long, for
two main reasons. It concerned six different claimants, each of whom had contracted with the Post
Office at different times, and each of those six claimants’ cases concerned very different facts. Very
few of the material facts relating to contract formation were agreed for any of these six, and therefore
factual findings had to be made for each of them in this respect. Approximately 500 paragraphs were
necessary to do this. Although first instance judgments must be reasoned so that parties understand
the outcome (and so that any judgment can be subject to appropriate review), very lengthy judgments
can be difficult to follow. If a judgment becomes too long, or too technical, it can be counterproductive
to wider understanding on the part of those not immersed in the fine detail of every aspect of the
case.
3.1 have therefore included technical detail about the Horizon system, its operation, and some aspects
of the technical evidence, in a Technical Appendix to this judgment. The contents of that appendix
should not be seen as being of subordinate effect to the contents of the judgment itself; this is done
simply for the convenience of readers. It is also intended that readers who do need to immerse
themselves in technical matters to a very fine level of detail may not find it necessary to study the
appendix, and reading the judgment alone may be sufficient. Notwithstanding that approach, however,
this judgment too is extremely long. This is due to the complexity of the Horizon system, which even
for computer systems is extraordinarily complicated for the reasons explained below; the period of
time over which the complaints range (which starts with the introduction of Horizon in 2000 and runs
to date); and due to the way that the litigation has unfolded. There are some matters in this judgment
that go to issues affecting the group litigation going forwards, such as disclosure, which have
increased its length. It is also the case that this litigation is being very strongly contested on both
sides. I have endeavoured to provide a reasonable level of detail to explain my findings on the Horizon
Issues to assist both sides as much as possible. Finally, the only findings that are made in this
judgment are those necessary to come to conclusions on the Horizon Issues. All other matters in all
the claims of the claimants and the counterclaims by the Post Office remain for decision in later
judgments.
4.This judgment is in the following parts:
Paragraph no.
A.Introduction 5
B.The Horizon Issues 18
C.Features of this Group Litigation 57
RLITO000005
RLITO000005
D.Evidence of Fact: The Claimants 76 E. Evidence of Fact: The Post Office 202
F.Documents and Available Information 559
G.The Experts’ Agreements 654
H.The Period 21 March 2019 to 4 June 2019 706
L.The Expert Evidence 731
J.Conclusion on Expert Evidence 870
K.Audit Data 905
L.Overall Conclusions 925
M.Answers to the Horizon Issues 965
Appendices:
1.Technical Appendix
2.Summary of bugs, errors and defects
3.Glossary
A, Introduction
5.These proceedings are being conducted pursuant to a Group Litigation Order (“GLO”) made on 22
March 2017 by Senior Master Fontaine, and approved by the President of the Queen’s Bench Division.
A more comprehensive introduction to the issues generally between the parties is contained at [2] to
[43] of Judgment (No.3) “Common Issues”, which is at [2019] EWHC 606 (QB). Other judgments
concerning procedural rather than substantive issues, are the first Judgment at [2017] EWHC 2844
(QB) and Judgment (No.2) “Strike Out” at [2018] EWHC 2698 (OB). During the Horizon Issues trial
which is the subject of this judgment, the Post Office issued an application that I recuse myself as
Managing Judge in this group litigation, and stop the Horizon Issues trial, so that it could be
recommenced at some later date in the future (before a replacement Managing Judge). That
application was refused and the judgment is at Judgment (No.4) “Recusal Application” at [2019]
EWHC 871 (OB). Permission to appeal was refused by the single Lord Justice on 9 May 2019. There is
also Judgment (No.5) “Common Issues Costs” at [2019] EWHC 1373 (OB) which made various orders
in respect of the costs of the Common Issues trial, a hearing in respect of that having taken place on
23 May 2019.
6.This trial concerns what the parties referred to at the case management stage as the Horizon
Issues. The intention behind the case management of this litigation was that the contractual issues
(which were called the “Common Issues” and affected all the claimants) and the computer issues
relating to the operation, functionality and reliability of the Horizon system (which were called the
“Horizon Issues”, and also affected all the claimants) would be resolved first. The parties needed time
to prepare for these two trials, which took place in late 2018 (for the Common Issues) and into
the spring of 2019 (for this one), in particular to perform disclosure, and prepare and serve evidence.
For the Horizon Issues, this included expert evidence, for which I gave permission, from two IT
experts, one for the claimants (Mr Coyne) and one for the Post Office (Dr Worden).
RLITO000005
RLITO000005
7.The recusal application caused significant delay and disruption to the Horizon Issues trial.
Originally, that trial had been programmed to finish (including delivery of oral closing submissions) by
11 April 2019, a date which became unachievable as a result of that application. Following Judgment
(No.4) the remaining witness of fact for the Post Office to be called in person, Mr Parker, a senior
Fujitsu employee, was called on 11 April 2019.
8.The two IT experts could not be called until 4 June 2019 onwards. The two experts therefore had a
period of some weeks, following on from the evidence of fact, before they were called to give their
oral evidence. There was therefore an interval between 11 April 2019 and 4 June 2019 when the
experts were called. Their evidence collectively took two court weeks.
9.The effect of this interruption upon the trial timetable meant that there was therefore time available
for each of the experts to consider the full scope of the crossexamination of the factual witnesses that
had taken place some weeks before, and reflect (with more time than is usual for experts whilst a trial
is underway) upon whether that impacted upon their views. Experts are supposed to consider the
factual evidence advanced by both sides in litigation neutrally, and if witnesses are crossexamined,
then - at least potentially - other evidence that might emerge ought, if it is important, to be
considered by those experts in arriving at their final opinions given in evidence. In a conventional
trial, expert evidence will usually follow on almost immediately after the evidence of fact has been
heard. This means that, in practical terms, most experts will only have a very short time in which to
consider any potentially important factual developments in any case before they are themselves cross-
examined. Here, there was no such restriction.
10.The Horizon Issues trial timetable was therefore different to a conventional trial in this respect,
because the vast majority of the oral factual evidence, with the single exception of Mr Parker's
evidence, was completed by 21 March 2019. That is 75 days, or almost 11 weeks, before Mr Coyne
was called by the claimants, and even longer before Dr Worden was called by the Post Office. The
approach of the experts to the opportunity presented by that interval further to assist the court with
their evidence is notable, in my judgment. I will deal with this further in Parts H and I of this
judgment. For a more detailed description of the architecture of the Horizon system (both Legacy
Horizon and Horizon Online) reference should be made to the Technical Appendix. For a general
understanding of the issues in this litigation, an overview is sufficient.
11.Horizon is both an accounting system, and also supports a large number of what are called
business applications. Some transactions that a customer might wish to carry out in their local Post
Office branch are what are called retail activities, such as buying a Post Office product. One example
of this is buying a book of stamps. However, the Post Office has a large number of business customers
whose products are offered to the public through branch Post Offices. The Post Office calls these
companies “clients”. For every kind of activity which a customer wishes to transact in their Post Office
branch, Horizon needs to have the functionality to perform it. This functionality means supporting the
counter activity of carrying out the transaction, which is another way of saying Horizon enables the
SPM or their assistant to transact the business over the counter. That business may be a combination
of Post Office retail activity, and purchasing services or products offered by the Post Office on behalf
of its clients. However, in conjunction with the ability of a SPM to serve a customer (what is called
counter or front office), there is the associated “back office activity” of settling with Post Office's
‘client' organisation, who has provided some service to the customer - such as the DVLA, or a bank.
Accounting is a theme or thread that runs through all of these business requirements of Horizon, but
it is only a part of them. The number of services provided by Post Office branches is large and has
increased steadily from 1998 to the present day. The number of clients has almost certainly increased,
RLITO000005
RLITO000005
and the functionality of Horizon has expanded in line with the growth in service, both on the counter
and in the back office.
12.For those activities where the Post Office branch is acting like a retail outlet (such as selling
stamps), both the hardware and software is provided by Horizon to support this activity. This is the
Electronic Point of Sale Software component of Horizon, that is referred to as “EPOSS”. EPOSS allows
the SPM or assistant to record that some goods have been provided to a customer, compute the price
of those goods, and allow the customer to pay the money required for all their purchased goods, using
either cash or a credit/debit card. Often, a customer may wish to carry out two or more different
activities in one visit to the counter - for instance, to settle a utilities bill and to buy some stamps. This
can be done in the same activity and so Horizon has the concept of a customer carrying out a 'basket'
of activities and settling the total amount due for the basket in several ways - by one single credit card
transaction, by a cheque, or by cash.
13.Baskets of Post Office activities and non-Post Office activities are not supported by Horizon. Often,
a local Post Office branch will be a retail outlet too, selling non-Post Office goods such as groceries,
newspapers or even (as with Mr Bates at Craig-yDon) a haberdashery. If a customer wishes to buy a
newspaper and some stamps, the newspaper is not sold by the Post Office; it is simply sold by the
associated, though in accounting terms separate, retail outlet run by the SPM which uses the same
premises. So, the customer has to settle in two parts. In some premises, a customer may queue up to
purchase (say) greetings cards and pens (from the retail side of the branch Post Office). If they then
wish to perform some activities with the Post Office, they may then need to queue up at a separate
counter position to do that; that separate and second activity would be transacted through Horizon. In
this respect, the National Lottery is an exception and spans the two businesses.
14.The Post Office’s clients include high street banks, Camelot, gas and electricity companies (for
paying utility bills), DWP (for paying benefits and pensions) and DVLA (for paying road fund tax).
Because of the different nature of the services provided through the Post Office for all of these many
hundreds of client organisations, the services provided through the Post Office will be different from
the service provided for other clients. There is therefore the need for some unique software
functionality within Horizon. This must be provided both in the branch and the back office to support
the activities for that client. This is a part of what makes Horizon such a large and complex system.
The other reason that Horizon is so complex is because it has evolved over a long period. As will be
seen below, the original concept and design for Horizon was for a joint Post Office/DWP project so that
welfare benefits could be paid to benefit claimants in a certain way. That changed and the DWP
withdrew from the project, but the need to offer other and an increasing number of products has led
to many additions being added on to the system, increasing its complexity over time.
15.As well as the counter activities necessary to run a branch Post Office, Horizon also supports the
periodic process of balancing and rollover for each branch; essential elements in the accounting to the
Post Office performed by the SPM. Every branch operates in Trading Periods, which are either four or
five weeks (according to a timetable published periodically by Post Office). At the start of each
Trading Period the branch is supposed to be ‘in balance’. This means that the physical stock and cash
in the branch agrees with the data regarding the stock and cash held in Horizon. Then, during the
Trading Period, Horizon records all customer transactions made at the branch, so it records the
changes in cash and stock. It is also required to record any replenishments or remittances of cash or
stock in the branch. Thus, Horizon records all changes in cash and stock held at the branch during the
Trading Period, and should be able to compute, from the starting amounts and the changes, the
expected amounts of cash and stock at the end of the period.
RLITO000005
RLITO000005
16.Without reciting the entire background to the group litigation between the Post Office and the
claimants, it is the accounting and functional accuracy of Horizon that is at the heart of the current
disputes, which have run for a great many years. The claimants maintain that the Horizon system in
operation threw up numerous discrepancies and shortfalls in their branch accounts, for which the
Post Office unfairly held them responsible. The Post Office dispute this, and maintain that these
occurrences are explicable by carelessness, fault or criminality on the part of the claimants.
17.Horizon, whether in its first incarnation as Legacy Horizon until 2010, or what is called Horizon
Online since then (originally HNG-X, now since 2017 HNG-A running on a different windows system),
is at the centre of this group litigation. It should also be noted that it is not a system that is operated
by the Post Office. It was originally designed, “rolled out” and operated by ICL, a company that was
partly owned by Fujitsu, and which is now fully absorbed within Fujitsu, and has been for many years.
There is a contract between Fujitsu and the Post Office in respect of Horizon, and perhaps in its
current form it is rather different than the original contractual relations between those two entities.
The contractual relationship between the Post Office and Fujitsu is not of direct relevance to the
Horizon Issues, but arises tangentially only in respect of what is a sub-issue, namely the charging
structure operated by Fujitsu for what are called ARQ requests for audit data and whether this
inhibited the Post Office in this respect. The operation, functionality and accuracy of Horizon is an
extremely thorny issue (or bundle of issues), although given the breadth of allegation and counter-
allegation in the group litigation they are not the only issues. Providing the answers to the Horizon
Issues will not lead to complete resolution of the litigation. It should however resolve one of the
central issues about Horizon’s accuracy and functionality.
B. The Horizon Issues
18. The Horizon Issues are as follows. In the 3rd Case Management Order of 1 March 2018, leading
counsel for the parties were ordered to meet and seek to agree the Horizon Issues to be tried in
March 2019. This was done, the issues were agreed by the parties and approved by the court. I take
the following from the Case Management
Order dated 23 March 2018, which was the next order made, and was a Consent
Order. The paragraph references to the pleadings were included in the list of Horizon Issues, which
were themselves appended to the Order of 23 March 2018 itself. These include at the beginning an
agreed definition of “the Horizon System”.
"AGREED LIST OF HORIZON ISSUES
definitions for the purpose of this list of issues
“the Horizon System” shall for the purposes of this list of issues mean the Horizon computer system
hardware and software, communications equipment in branch and central data centres where records
of transactions made in branch were processed, as defined in the Generic Particulars of Claim (“the
GPOC”), at §16 and as admitted by Post Office in its Defence at §37.
BUGS, ERRORS AND DEFECTS IN HORIZON
Accuracy and integrity of data
(1)To what extent was it possible or likely for bugs, errors or defects of the nature alleged at §§23 and
24 of the GPOC and referred to in §§ 49 to 56 of the Generic Defence to have the potential (a) to cause
apparent or alleged discrepancies or shortfalls relating to Subpostmasters’ branch accounts or
RLITO000005
RLITO000005
transactions, or (b) undermine the reliability of Horizon accurately to process and to record
transactions as alleged at §24.1 GPOC?
(2)Did the Horizon IT system itself alert Subpostmasters of such bugs, errors or defects as described
in (1) above and if so how?
(3)To what extent and in what respects is the Horizon System “robust” and extremely unlikely to be
the cause of shortfalls in branches?
Controls and measures for preventing / fixing bugs and developing the system
(4)To what extent has there been potential for errors in data recorded within Horizon to arise in (a)
data entry, (b) transfer or (c) processing of data in Horizon?
(5)How, if at all, does the Horizon system itself compare transaction data recorded by Horizon against
transaction data from sources outside of Horizon?
(6)To what extent did measures and/or controls that existed on Horizon prevent, detect, identify,
report or reduce an extremely low level of risk of the following:
a.data entry errors;
b.data packet or system level errors (including data processing, effecting, and recording the same);
c.a failure to detect, correct and remedy software coding errors or bugs;
d.errors in the transmission, replication and storage of transaction record data; and
e.the data stored in the central data centre not being an accurate record of transactions entered on
branch terminals?
OPERATION OF HORIZON
Remote Access
(7) Were Post Office and/or Fujitsu able to access transaction data recorded by Horizon remotely (i.e.
not from within a branch)?
Availability of Information and Report Writing
(8)What transaction data and reporting functions were available through Horizon to Post Office for
identifying the occurrence of alleged shortfalls and the causes of alleged shortfalls in branches,
including whether they were caused by bugs, errors and/or defects in the Horizon system?
(9)At all material times, what transaction data and reporting functions (if any) were available through
Horizon to Subpostmasters for:
a.identifying apparent or alleged discrepancies and shortfalls and/or the causes of the same; and
b.accessing and identifying transactions recorded on Horizon?
Access to and/or Editing of Transactions and Branch Accounts
(10) Whether the Defendant and/or Fujitsu have had the ability/facility to (i) insert, inject, edit or
delete transaction data or data in branch accounts; (ii) implement fixes in Horizon that had the
potential to affect transaction data or data in branch accounts; or (iii) rebuild branch transaction data:
RLITO000005
RLITO000005
a.at all;
b.without the knowledge of the Subpostmasters in question; and
c.without the consent of the Subpostmaster in question.
(11)If they did, did the Horizon system have any permission controls upon the use of the above facility,
and did the system maintain a log of such actions and such permission controls?
(12)If the Defendant and/or Fujitsu did have such ability, how often was that used, if at all?
(13)To what extent did use of any such facility have the potential to affect the reliability of the
Branches’ accounting positions?
Branch trading statements, making good and disputing shortfalls
(14) How (if at all) does the Horizon system and its functionality:
a.enable Subpostmasters to compare the stock and cash in branch against the stock and cash
indicated on Horizon?
b.enable or require Subpostmasters to decide how to deal with, dispute, accept or make good alleged
discrepancy by (i) providing his or her own personal funds or (ii) settling centrally?
c.record and reflect the consequence of raising a dispute on an alleged discrepancy, on Horizon
Branch account data and, in particular:
(i)does raising a dispute with the helpline cause a block to be placed on the value of an alleged
shortfall; and
(ii)is that recorded on the Horizon system as a debt due to Post Office?
d.enable Subpostmasters to produce (i) Cash Account before 2005 and (ii) Branch Trading Statement
after 2005?
e.enable or require Subpostmasters to continue to trade if they did not complete a Branch Trading
Statement; and if so, on what basis and with what consequences on the Horizon system?
Transaction Corrections
(15) How did Horizon process and/or record Transaction Corrections?”
19.Given the nature of the proceedings, and the disputes about the Horizon system which were
apparent on the face of the pleadings, I had indicated to the parties at an early stage that the generic
disputes about the operation of the Horizon system, which would need expert evidence, would be
resolved after the contractual issues (which became called the Common Issues, and led to Judgment
(No.3)). The issues were not foisted on the parties by the court. The court approved the wording of the
issues agreed by the parties during the case management stage of the litigation. There was no
difficulty about this at the time.
20.However, the meaning of issues 1 and 3 in particular proved, at the Horizon Issues trial itself, to be
controversial. This is regrettable. Nor was any controversy aired with the court prior to the actual
trial. That too is regrettable. The court is well used to parties who disagree over the answers to
certain issues in litigation generally. It is unsatisfactory when they also disagree about what any
particular issue itself actually
RLITO000005
RLITO000005
means, or the question that is being posed (including being posed to experts) by an issue, particularly
once that issue has already been finalised and ordered.
21.Here, the two main areas of dispute over the meaning of the Horizon Issues were as follows (there
were others, but these are the most important ones). The claimants approached Issue 1 as it was
worded, requiring consideration of whether bugs, errors or defects had the potential to cause
discrepancies or shortfalls in SPMs’ branch accounts or transactions. In other words, this issue was
not limited to consideration of whether bugs, errors or defects had in fact actually caused
discrepancies or shortfalls. The Post Office, on the other hand, approached Issue 1 as requiring
consideration of whether bugs, errors or defects had in fact actually caused discrepancies or
shortfalls, and by reference to the claimants’ accounts specifically. This approach by the Post Office
sought, in my judgment, to narrow the scope of the Horizon Issues. This was not raised at the case
management stage.
22.The second main area of dispute concerned Issue 3. This includes consideration of the concept of
“robustness”. This word has been at the heart of the Post Office’s defence of the Horizon system for
many years. The Post Office has said publicly, and on many occasions, that no computer system is
100% accurate and/or perfect but Horizon is “robust”. This approach by the Post Office pre-dated the
commencement of the group litigation and was at the heart of the Post Office’s response to the
increasing criticisms from different quarters about Horizon. I deal with this important concept below
in more detail at [36] below. What the Post Office effectively means by this (in outline terms only) is
that the Horizon system can properly and safely be relied upon by the Post Office for the purposes for
which it is designed and intended. The parties - and this is clear on the pleadings, as well as in the
terms of Horizon Issue 3 - disagree over whether Horizon is robust. In the Reply, the claimants
challenged this in the following terms:
“It is therefore denied that Horizon ‘is robust and [...] is extremely unlikely to the cause of losses in
branches’ (paragraph 16). In fact, the relatively small chance of errors admitted by the Defendant,
would be likely to produce the very picture reflected in the Claimants' case.”
23.Mr de Garr Robinson for the Post Office was somewhat critical at the trial of the drafting of both
Issues 1 and 3. He was not involved in the drafting of the Horizon Issues, as the Post Office had used
different leading counsel. However, that criticism overlooked that the Post Office had agreed to the
wording of all the Horizon Issues through its other leading counsel, and the wording had also been
approved by the court. Although there have been a total of three leading counsel altogether instructed
for the Post Office thus far at first instance, with a fourth leading counsel used by the Post Office to try
and appeal the Common Issues to the Court of Appeal, simply because different counsel are used does
not mean that the issues ordered to be tried become different. The Horizon Issues themselves were
appended to the Case Management Order itself.
24.Lists of issues, whether in group litigation or generally, are very important. I can do no better than
quote Longmore LJ in Scicluna v Zippy Stitch Ltd [2018] EWCA Civ
1320, who said at [14] “ever since the Woolf reforms, parties in the High Court have been required to
agree lists of issues formulating the points which need to be determined by the judge. That list of
issues then constitutes the road map by which the judge is to navigate his or her way to a just
determination of the case.” In this litigation, the Horizon Issues are not only important, but are vitally
important. The whole of the proceedings concerns the operation and reliability of the Horizon system.
It could quite easily be called the Horizon Post Office Case. Although there are a great number of
different causes of action pleaded in both directions, both by the claimants and by the Post Office,
RLITO000005
RLITO000005
against one another, the central core of the case is about Horizon. I consider that the wording of both
Horizon Issues 1 and 3 - indeed, all the Horizon Issues - to be clear and had they not been, I would
not have approved the agreed wording.
25.However, some explanation is necessary given the disagreement to which I have referred.
26.The phrase “bugs, errors or defects” is sufficiently wide to capture the many different faults or
characteristics by which a computer system might not work correctly. The parties in this litigation,
usually but not always for convenience, would often refer simply to “bugs”, and one part of the 2nd
Experts’ Joint Statement became known as “the Bug Table”. Computer professionals will often refer
simply to “code”, and a software bug can refer to errors within a system’s source code, but “software
bugs” has become more of a general term and is not restricted, in my judgment, to meaning an error
or defect specifically within source code, or even code in an operating system. Source code is not the
only type of software used in a system, particularly in a complex system such as Horizon which uses
numerous applications or programmes too. Reference data is part of the software of most modern
systems, and this can be changed without the underlying code necessarily being changed. Indeed,
that is one of the attractions of reference data. Software bug means something within a system that
causes it to cause an incorrect or unexpected result. During Mr de Garr Robinson’s cross-examination
of Mr Roll, he concentrated on “code” very specifically and carefully. There is more to the criticisms
levelled at Horizon by the claimants than complaints merely about bugs within the Horizon source
code.
27.The wording of Horizon Issue 1, “to what extent was it possible or likely for bugs, errors or
defects.....” is therefore very wide. Although that wording was proposed and agreed by the parties, I
considered at the case management stage when the Horizon Issues were ordered, and still consider
now, that these words clearly cover the whole range of criticisms levelled at Horizon by the claimants,
and the matters that could potentially be wrong with the Horizon system without restriction. Bugs,
errors or defects is not a phrase restricted solely to something contained in the source code, or any
code. It includes, for example, data errors, data packet errors, data corruption, duplication of entries,
errors in reference data and/or the operation of the system, as well as a very wide type of different
problems or defects within the system. “Bugs, errors or defects” is wide enough wording to include a
wide range of alleged problems with the system.
28.1 also consider that the words “possible or likely” are also wide enough to cover both ends of the
spectrum of what the parties would have the court decide. In other words, it is a neutrally worded
issue of wide effect which does not, by its phrasing, indicate any particular starting point or end point.
“Possible” means something potentially could happen; or whether it could happen or not. “Likely”
means that something is, on
the balance of probabilities, more likely to have happened than not. The issue uses both terms, and
therefore poses a wide question so that the answer provided by the court will be of maximum utility in
the group litigation.
29.Finally, the word “potential”, in the clause “have the potential to (a) cause apparent or alleged
discrepancies or shortfalls relating to Subpostmasters’ branch accounts or transactions, or (b)
undermine the reliability of Horizon.....” clearly means, on its express terms, that this issue is dealing
with possible or prospective effects, rather than whether bugs, errors or defects have in fact caused
actual or alleged discrepancies or shortfalls. The way the wording of this issue was dealt with in the
expert evidence will be dealt with below.
RLITO000005
RLITO000005
30.If there were any ambiguity in Issue 1, and I do not consider that there is, the issue specifically
refers to certain passages in the pleadings. It would therefore be of assistance to consider the
pleadings. The relevant passages from the Particulars of Claim as amended are as follows:
“23. However, the Claimants aver that there were a large number of software coding errors, bugs or
defects which required fixes to be developed and implemented. There were also data or data packet
errors. There was a frequent need for Fujitsu to rebuild branch transaction data from backups, giving
rise to the further risk of error being introduced into the branch transaction records. The Claimants
understand that Fujitsu maintained a 'Known Error Log' relating to some or all of these issues which
was provided to the Defendant but which has not been disclosed.
24. Further, the Claimants aver and rely upon the following:
24.1.Insufficient error repellency in the system (including sufficient prevention, detection,
identification and reporting of errors), both at the data entry level and at the data packet or system
level (including data processing, effecting and reconciling transactions, and recording the same);
24.1A bugs and/or errors and/or defects in Horizon and any data or data packet errors had the
potential to produce apparent shortfalls which did not represent a real loss to the Defendant;
24.2.Horizon is imperfect and has the potential for creating errors (as the Defendant has admitted in
pre-action correspondence, in the Letter of Response, dated 28 July 2016, at paragraph 1.3);
24.3.bugs and/or errors have on some occasions produced discrepancies and/or apparent shortfalls
(as the Defendant has admitted in pre-action correspondence, in the Letter of Response, Schedule 6)
and such shortfalls may also have arisen from data or data packet errors; and, further
24.4.the Defendant sought and/or recovered such alleged shortfalls from Subpostmasters (as is
presently understood to be admitted by the Defendant in the Letter of Response, Schedule 6,
paragraphs 4.1 to 4.5).”
31.The passages in the Defence which are identified in the Issue are paragraphs 49 to 56. This is a
lengthy series of paragraphs but as all are referred to expressly within Issue 1, I will reproduce them
all. These state the following:
“Bugs, errors or defects in Horizon
49. As to paragraph 22:
(1)If and to the extent that the Claimants wish to assert that any of the shortfalls for which they were
held responsible were Horizon-generated shortfalls, it is for them to make that distinct allegation and
seek to prove it. Post Office notes that they do not make the allegation in the GPoC. It further notes
that, in paragraph 20 of their solicitors' letter to Post Office's solicitors dated 27 October 2016, the
Claimants make it clear that they do not allege that there is a systematic flaw in Horizon or indeed
any flaw which has caused any Claimant to be wrongly held responsible for any shortfall.
(2)It is denied that Post Office has unreasonably or otherwise failed to provide “obviously relevant
disclosure" in relation to bugs, errors or defects in Horizon. There has been no order or application
for disclosure and, in the premises set out above, there appears to be no basis for providing such
disclosure.
50. Paragraph 23 is embarrassing for its lack of particularity, in that (amongst other things) it does
not identify the errors, bugs or defects on which the Claimants rely or how “large” their number was
RLITO000005
RLITO000005
or the period in which they are said to have occurred and nor does it identify the transaction data that
Fujitsu is alleged to have rebuilt, how "frequent" was the need to rebuild it or the extent of the "risk of
error" which is said to have been introduced. In the premises, Post Office cannot plead to the first
three sentences of this paragraph. However:
(1)All IT systems experience software coding errors or bugs which require fixes to be developed and
implemented. As is noted in paragraph 53 and 54 below, there are robust measures in place in
Horizon for their detection, correction and remediation.
(2)All IT systems involving the transmission of data over the internet experience data or data packet
errors during transmission and such systems routinely have protective measures in place to prevent
such errors creating any difference between the data transmitted and the data received and retained
by the recipient. Horizon has robust controls making it extremely unlikely that transaction data input
in a branch would be corrupted when being transferred to, and stored in, Post Office's data centre in a
manner that would not be detected and remedied.
(3)Like all IT systems, Horizon has backups to guard against any loss of data due to local hardware
failure. Where hardware fails, the data on that hardware is recovered from the backup. Post Office
takes the term "rebuild" to refer to the situation before the introduction of Horizon Online where a
new terminal was introduced to a branch and the data stored on the other branch terminals (or on a
disc where it was a single counter branch) was restored to the new terminal. In this context, Post
Office does not accept that there was a "frequent" need to "rebuild" data from back-ups.
(4)It is admitted that Fujitsu maintain a “Known Error Log". This is not used by Post Office and nor is
it in Post Office's control. To the best of Post Office's information and belief, the Known Error Log is a
knowledge base document used by Fujitsu which explains how to deal with, or work around, minor
issues that can sometimes arise in Horizon for which (often because of their triviality) system-wide
fixes have not been developed and implemented. It is not a record of software coding errors or bugs
for which system-wide fixes have been developed and implemented. To the best of Post Office's
knowledge and belief, there is no issue in the Known Error Log that could affect the accuracy of a
branch's accounts or the secure transmission and storage of transaction data.
51.In paragraph 24, the Claimants again bundle many ambiguous and/ or misleading allegations
together. Post Office separates out and addresses those allegations in paragraphs 52 to 56 below.
52.As paragraph 24.1 does not explain what is meant by “error repellency", what sorts of errors are
referred to, what is meant by "data entry level", what would constitute "sufficient" prevention,
detection, identification or reporting of these errors, or in what respects the error repellency of
Horizon was insufficient, Post Office cannot plead to this paragraph. However, the general thrust of
paragraph 24.1 is denied and the robust controls, procedures and practices pleaded in paragraphs 53
and 54 below are noted.
53.As to paragraph 24.14, it is a truism that errors or bugs in an IT system and data or data packet
errors have the potential to create errors in the data held in that system. However, Horizon has at all
material times included technical control measures to reduce to an extremely low level the risk of an
error in the transmission, replication and storage of the transaction record data. These have varied
from time to time and they currently include the following:
(1)Horizon creates, transmits and stores trans action data in the form of "baskets". A basket is a
complete transactional session between a customer and Post Office and may include one, several or
RLITO000005
RLITO000005
many individual transactions taking place within the same session. Horizon will not accept a basket of
transactions that does not net to zero (i.e. the value of any sales is set off by the value of any payment
made or received). This reduces greatly the risk of any error in the data entered within any given
basket.
(2)If a basket of transactions fails properly to complete its transmission to the central database
(because, for example, of a power loss), the system rejects any partial transmission and requests the
full basket from the branch terminal. This reduces greatly the possibility of baskets of transactions
failing to be recorded.
(3)At the point of a basket being accepted by Horizon, it is assigned a unique sequential number (a
“JSN") that allows it to be identified relative to the other baskets transmitted by that branch. This
reduces greatly the risk of recording duplicate baskets or there being a missing basket.
(4)Each basket is also given a digital signature, i.e. a unique code calculated by using industry
standard cryptography. If the data in the basket were to change after the digital signature was
generated, this would be apparent upon checking the digital signature.
(5)Initial data integrity checks are undertaken when baskets are received at the Post Office data
centre from a branch. Baskets are then copied from the central database to the Audit Store where a
digital seal is then applied (the "Audit Store Seal"). If the baskets and/ or the data within the baskets
were altered after the application of the Audit Store Seal, this would be apparent when the baskets
are extracted from the Audit Store.
(6)Horizon and the above controls are themselves subject to various audits and checks including
audits carried out by third parties.
54. Further as to paragraph 24.1A, in addition to the technical controls referred to above, there are
several operational procedures and practices conducted by Post Office and Subpostmasters that serve
to increase the reliability of the data stored in the central data centre as an accurate record of the
transactions entered on branch terminals. These currently include the following:
(1)For many transaction types, Post Office compares its own transaction record against the
corresponding records held by Post Office clients. If an error in Horizon were to result in the
corruption of transaction data, this should be revealed by the comparison.
(2)There are detailed procedures in place to address the risk of data loss resulting from interrupted
sessions, power outages or telecommunications failures in branches. These are set out in the
“Recovery - Horizon Online Quick Reference Guide" and Horizon guides the system user through the
recovery process (which include completing any transactions that are cut short). These procedures
should prevent any data errors arising from interrupted sessions, power outages and
telecommunications failures.
(3)The display of the transactions being effected on-screen at the branch terminal allows the user of
the system to identify any inconsistency between the information shown on the screen and the
transaction that the user has keyed into the system. If, for example, a hypothetical bug in the terminal
were to cause a key-strike on number S to be recorded as an input of number 6, this would be
detected rapidly by system users, given the large number of system users and the huge number of
transactions effected on Horizon.
RLITO000005
RLITO000005
(4)The accounting and record-keeping obligations placed on Subpostmasters reduce the risk of any
errors going undetected. For example, there is an obligation for each branch to count and declare to
Post Office the cash it holds on a daily basis, which increases the likelihood of promptly detecting any
overstatement or understatement of the cash position on Horizon. If a Subpostmaster detects that an
error has been made at an early stage, its cause is more likely to be identified.
(5)Fujitsu operates industry standard processes for developing and updating Horizon and for
investigating and resolving any identified potential system errors.
55.As to paragraph 24.2, Post Office admits that, like all other IT systems, Horizon is not a perfect
system which has never had any errors or bugs. However, as indicated in paragraphs 53 and 54
above, it has robust systems in place to identify them, fix them and correct their consequences (if
any).
56.As to paragraphs 24.3 and 24.4:
(1)There have been occasions on which bugs or errors in Horizon have resulted in discrepancies and
thus shortfalls or net gains in some branch accounts, as outlined in Schedule 6 of the Letter of
Response. It is denied (if it be alleged) that such bugs or errors have affected any of the Claimants.
(2)On each occasion, both the bugs or errors and the resulting discrepancies in the relevant branch
accounts were corrected. Post Office took steps to ensure that it had identified all branches affected
by the bugs or errors and that no Subpostmaster was ultimately held responsible for any resultant
shortfalls.
(3)Paragraphs 4.1 to 4.5 of Schedule 6 to the Letter of Response relate to the socalled Suspense
Account Bug. Without prejudice to the burden of proof, none of the branches affected by the Suspense
Account Bug are branches for which the Claimants were responsible.
(4)None of the Subpostmasters whose branches were affected by the Suspense Account Bug were
ultimately held responsible for the shortfalls that it generated. The Claimants are therefore wrong to
understand Post Office as having admitted that it "recovered such alleged shortfalls from
Subpostmasters". Where Subpostmasters in the affected branches had made good or settled centrally
shortfalls that were later corrected, those Subpostmasters received a payment or credit in the amount
of the shortfall.”
32.It can be seen that the word “robust” is used throughout these passages of the Defence, and
“robust systems” in paragraph 55. Given “robustness” forms part of Issue 3 there is no need
separately to consider it at this stage when analysing the differences between the parties about what
Issue 1 means.
33.The passages in both parties’ pleadings which I have reproduced above show that it is the
potential, in general terms, of the Horizon system to have the effect complained of that is the subject
matter of Horizon Issue 1. In order to read Issue 1 in the manner contended for by the Post Office, one
would have to delete entirely the phrase “to have the potential” from the issue. One would also have
to substitute the term “Subpostmasters’ branch accounts or transactions” with “the claimants’ branch
accounts or transactions”. That is not what Issue 1 states, it is not what the parties agreed the issue
would be, and it is not the issue that the court approved and ordered. In my judgment, the approach
of the Post Office to what Horizon Issue 1 actually means is too narrow.
RLITO000005
RLITO000005
34.In my judgment, the correct construction of Horizon Issue 1 is that contended for by the claimants.
In other words, it involves a two-stage process. Firstly, consideration of whether there were, or are,
bugs, errors or defects in the Horizon system as alleged by the claimants. Secondly, if the answer is
that there were, or are, such bugs, errors or defects, the second stage is to consider whether these
have (or did have previously) the potential to cause apparent or alleged discrepancies in SPMs’
branch accounts generally. The issue is not whether such bugs, errors or defects did in fact cause
such discrepancies or shortfalls in the claimants’ accounts specifically. That separate or different issue
- the effect upon claimants’ branch accounts - is a more claimant specific one. It will have to be
determined at some stage, for any of the claimants whose individual claims come to trial in the future.
It may require expert forensic accountancy evidence. It was not ordered to be dealt with in the
Horizon Issues trial. The Horizon Issues were intended to be, and in my judgment on their wording
are, generic issues relating to Horizon and its operation. However, if a bug, error or defect is shown to
have had an actual impact on any SPM’s accounts, then by definition it has had the potential to have
such an impact. Actual impact on branch accounts can therefore be of assistance in considering Issue
1 as ordered, namely including potential.
35.The passages in the pleadings quoted above do show that the Post Office had, in
Schedule 6 to its Letter of Response, accepted the existence of two bugs or errors in Horizon, one of
which was called the Suspense Account Bug, but denied that the latter had affected any branch
accounts of any of the claimants. The passages in paragraph 56 of the Defence that explained that no
SPMs had been “ultimately held responsible for the shortfalls that it generated” reads a little
differently with the hindsight provided from evidence in two lengthy trials. This is because the Post
Office’s Defence effectively accepts that “Subpostmasters in the affected branches had made good or
settled centrally” the sums in question - that is the shortfalls in their branch accounts that Horizon
showed - but states that these were later corrected. Both the expressions “making good” and “settled
centrally” means that the Post Office had originally held the SPM responsible for the losses, as the
shortfalls formed part of their branch accounts. These terms mean that the SPMs had either paid the
money (making good) or been given time to pay sums which the Post Office treated as a debt, even if
they were disputed sums (settling centrally), meanings given to these terms that were confirmed by
Ms Van Den Bogerd in her evidence for the Common Issues trial. However, the pleading means that
the Post Office had then, on the Post Office’s case, subsequently corrected this impact on branch
accounts which was why it pleaded no SPMs were “ultimately held responsible”. Therefore there
might prove not to be as much between the parties on this point in reality, as the group litigation
unfolds, as there appears to be on the pleaded case. The Suspense Account Bug undoubtedly had
actual impact upon SPMs’ branch accounts.
The meaning of “robustness”
36.Turning to the disagreement about Issue 3, given the parties disagreed about whether the Horizon
system is (or was) “robust”, it is a fairly elementary step to consider the meaning of that term, and
how it is being used by the parties. Context is important so far as the meaning of the word “robust” is
concerned. If someone is in robust health, it usually is taken to mean that they are healthy, or even
very healthy. A robust exchange of views can be a polite way of referring to an argument. Given the
importance of the concept to the Horizon system, its prominence in the Post Office’s defence of the
system, and its express inclusion (admittedly in inverted commas) in the Horizon Issues, I asked each
side in the litigation during oral closing submissions for a reference from their pleadings or
submissions for the meaning which they ascribed to the word. I referred to this as their benchmark
RLITO000005
RLITO000005
definition. Robustness was referred to by both sides in the litigation in numerous places, but not
always in the same precise terms, and clarity is to be welcomed.
37.The claimants answered this by reference to the remainder of Issue 3, namely “extremely unlikely
to be the cause of shortfalls in branches” and explained that the claimants had found the word robust
“difficult to define” other than by reference to this. This would mean therefore that it had no separate
independent meaning other than as a summary of the longer second part of Issue 3. In other words, a
robust system would be one that is extremely unlikely to be the cause of shortfalls in branches. The
claimants also implicitly, if not expressly, criticised use of the term both by the Post Office in its
pleadings and written submissions as being more aligned to public relations than as a performance
standard.
38.The Post Office asked for some time to provide the reference that I requested. Given the meaning
of “robust” is so central in the Post Office’s defence of the Horizon system, I granted the Post Office
the time that was requested.
39.The Post Office subsequently, after the trial ended, submitted a short document entitled “the Post
Office’s case on the meaning of robustness”. This was not what was intended when I sought a
reference from the Post Office to their definition, and the document submitted went rather further and
made wider ranging submissions. The document did state, so far as the meaning of the word is
concerned, the following: “In Post Office’s submission, the meaning of robustness is a matter for
expert opinion. Robustness is a well-established concept in the IT industry and is the subject of
academic study: see para. 361 of Post Office’s written closing”.
40.I do not consider that the meaning of words is a matter for expert opinion. The two experts in this
case are IT experts, not experts in linguistics or the meaning of language. However, the meaning of
robustness within the field of IT is, arguably, a matter upon which the experts’ opinions should be
considered, not least because they were applying that term to their expert exercise. The Post Office
also relied upon the 1st Experts’ Joint Statement which in respect of Issue 3 stated the following as
agreed:
“There are different dimensions of robustness, such as robustness against hardware failure, software
defects and user error. The robustness of the system also depends on the processes around it.
Robustness does not mean perfection; but that the consequences of imperfection must be managed
appropriately. If the extent of imperfection is too high, this would be very difficult to do which would
imply less robustness.
Horizon has evolved since its inception. Therefore, its robustness may have varied throughout its
lifetime. The level of robustness may have increased or decreased as the system was changed.
The existence of branch shortfalls is agreed. The experts do not agree at this point as to whether this
indicates any lack of robustness.”
41.In the areas of disagreement in this Joint Statement, each expert provided the following. Mr Coyne
stated (inter alia):
“For the purposes of addressing the robustness of Horizon, I have applied the following definition of
robustness:
RLITO000005
RLITO000005
‘The ability to withstand or overcome adverse conditions, namely, the ability of a system to perform
correctly in any scenario, including where invalid inputs are introduced, with effective error
handling.
In consideration of the likelihood of Horizon to be the cause of shortfalls in branches, Horizon is not
determined to be robust in this regard because:
(a)it contained high levels of bugs, errors and defects as set out under Issue 1 above which created
discrepancies in the branch accounts of Subpostmasters;
(b)it suffered failures of internal mechanisms which were intended to ensure integrity of data;
(c)the system did not enable such discrepancies to be detected, accurately identified and/or recorded
either reliably, consistently or at all;
(d)the system did not reliably identify ‘Mis-keying’, which is inevitable in any system with user input,
and did not reliably have in place functionality to restrict users from progressing a mis-key;
(e)it required numerous processes and workarounds to be in place to allow Fujitsu to modify data
already recorded by Horizon, which would not be required in a “robust” system; and/or
(there were weaknesses and risks of errors and other sources of unreliability within Horizon.”
(italics present in original)
42.Dr Worden stated in the same Joint Statement:
“The definition of 'robust' proposed above by Mr Coyne is not adequate, for reasons given below. The
term ‘robust' is not, as implied in para 3.1 of the outline, either illdefined or a piece of IT public
relations. Robustness (which is closely related to resilience) is an engineering objective, and large
parts of project budgets are devoted to achieving it. It receives its meaning in the phrase ‘robust
against... [some risk or threat]', and there are a large number of risks that business IT systems need
to be robust against - such as hardware failures, communications failures, power cuts, disasters, user
errors or fraud. These are the dimensions of robustness.
In all these dimensions, robustness does not mean ‘be perfect’; it means ‘address the risks of being
imperfect’. The extent of robustness is to be interpreted as: in how many dimensions was Horizon
robust? and: in each dimension, how large were the remaining risks?
In my report I shall survey the evidence I have found that Fujitsu paid sufficient attention to the
dimensions of robustness, and that they did so successfully. I shall also address evidence from Mr
Coyne implying that Horizon fell short of its robustness objectives.
In my current preliminary opinion, Horizon is a highly robust system, and this has important
implications for the other Horizon issues, notably issue 1.”
43.It can be seen therefore that Dr Worden in the Joint Statement did not agree Mr Coyne’s definition,
and expressly said it was not adequate. In any event, the meaning of any word - even “robust”, or
“robustness” - ought to be capable of description by the parties themselves. Although on its face it did
not appear that Dr Worden agreed with Mr Coyne’s definition, a footnote in the Post Office first set of
post-hearing submissions suggested that Dr Worden was not disagreeing with the first part of Mr
Coyne’s text, in other words that part of the text that contained his definition of robustness (which
was in italics in the 1st Joint Statement). Obviously if the parties (or their experts) could agree the
RLITO000005
RLITO000005
definition to be applied so far as the Horizon System is concerned, that ought to be identified. I
therefore asked the Post Office via email whether it agreed with the definition adopted by Mr Coyne,
and if not, what its alternative definition was.
44.This led to a further document being received from the Post Office dated 18 July 2019. It referred
to the passage in the 1st Joint Statement (which is quoted at [40] above) as “the agreed definition”.
That rather overlooks that Mr Coyne identified the definition of robustness which he was applying,
and Dr Worden expressly disagreed with this in the same Joint Statement under the heading “Areas of
Disagreement”, and stated “the definition of ‘robust’ proposed above by Mr Coyne is not adequate, for
the reasons given below”. It also overlooks that in the 3rd Joint Statement, paragraph 3.1 had an
agreed entry which stated the following:
“Irrespective of how you define the detail of robustness, in line with most other largescale computer
systems, Horizon's robustness has generally improved.
From our experience of other computer systems, Horizon is relatively robust. We agree that 'robust'
does not mean infallible and therefore Horizon has and will continue to suffer faults. Robustness limits
the impact of those faults and other adverse events.
Thisi . . 1 from Post Office di .
defects in live use and then applying fixes and improving monitoring.”
(emphasis added)
45.Later in the same document of 18 July 2019 the submission was made by the Post Office that “the
robustness of a system is the effectiveness of the system in managing the risks of imperfections
(which are inevitable in any system) and their consequences”. It was also submitted that “As Post
Office understands it, this is what Mr Coyne meant when in his comments in [the 1st Joint Statement]
he defined robustness as “the ability to withstand or overcome adverse conditions, namely, the ability
of a system to perform correctly in any scenario, including where invalid inputs are introduced, with
effective error handling”.
46.This was precisely the definition which Dr Worden, in his areas of disagreement on the 1st Joint
Statement, described as “inadequate”. The end position therefore is as follows.
47.The claimants found “robustness” difficult to define in the abstract and tied it in with the other
wording of Horizon Issue 3; a robust system would be “extremely unlikely to be the cause of shortfalls
in branches”. That however is a consequence of how a robust system would operate, not a definition
of what robustness means.
48.The Post Office defined it as follows: “the robustness of a system is the effectiveness of the system
in managing the risks of imperfections (which are inevitable in any system) and their consequences”.
The Post Office was also prepared to accept Mr Coyne’s italicised definition in the 1st Joint Statement,
namely ‘The ability to withstand or overcome adverse conditions, namely, the ability of a system to
perform correctly in any scenario, including where invalid inputs are introduced, with effective error
handling”.
49.Mr Coyne applied the definition he set out in italics in the 1st Joint Statement, quoted in the
immediately preceding paragraph of this judgment and at [41] above.
50.Dr Worden’s definition was as follows:
RLITO000005
RLITO000005
“Robustness (which is closely related to resilience) is an engineering objective, and large parts of
project budgets are devoted to achieving it. It receives its meaning in the phrase ‘robust against...
[some risk or threat]', and there are a large number of risks that business IT systems need to be
robust against - such as hardware failures, communications failures, power cuts, disasters, user errors
or fraud. These are the dimensions of robustness.
In all these dimensions, robustness does not mean ‘be perfect'; it means ‘address the risks of being
imperfect'. The extent of robustness is to be interpreted as: in how many dimensions was Horizon
robust? and: in each dimension, how large were the remaining risks?”
51.The Post Office also submitted that Mr Coyne’s definition was not “materially different” to that of
Dr Worden.
52.The Post Office made submissions in paragraph 3(b) of the written submissions dated 18 July 2019
on robustness that stated that Mr Coyne cannot have intended to exclude the effect of
countermeasures when he considered the concept of robustness. I shall return to this topic when
dealing with countermeasures. This is because some of the countermeasures considered by Dr
Worden are not parts of the Horizon System at all, such as SPMs noticing adverse entries in their
branch accounts, and the manual issuing of Transaction Corrections (TCs) by the Post Office (which
both parties agree are outside of the Horizon System).
53.1 do however accept the Post Office’s submissions that there is not a great or material difference in
the definitions of robustness adopted by the parties’ experts. I do not accept the claimants’
submission that robustness is difficult to define. Dr Worden defined robustness by using what he
termed as “the dimensions of robustness”. It is rather circular to describe the meaning of robustness
as being “robust against” some particular risk. Although Mr Coyne provided his definition in the 1st
Joint Statement, the statement by Dr Worden that this was “inadequate” may only have been aimed at
the entirety of Mr Coyne’s entry in the areas of disagreement, as effectively accepted by the Post
Office in their most recent written submissions on the subject. Whether that is an explanation of the
lack of agreement in the Joint Statement, I also agree with the Post Office that Mr Coyne’s definition
is not materially different to that used by Dr Worden.
54.Robustness is indeed an engineering concept. It means the ability of any system to withstand or
overcome adverse conditions. A robust system is strong and effective in all or most conditions. The
robustness of a system is the effectiveness of the system in managing the risks of imperfections
(which are inevitable in any system) and their consequences; this is the same meaning as how
robustness was described in the Post Office’s written submissions dated 18 July 19. Robustness does
not mean perfection.
55.The exercise necessary above, to arrive at the definition of robustness in [54] above, is not judicial
pedantry. Given the central importance of robustness to the disputes about the Horizon System, and
the Horizon Issues, it is in my judgment essential. It is mildly surprising, given how central the
assertion of robustness has been to the Post Office’s defence of the Horizon System, that Dr Worden’s
interpretation of the term has been relied upon so heavily by the Post Office, given the term was used
by the Post Office for some years prior to his involvement.
56.However, regardless of that passing observation, I find that both experts correctly understood what
robustness in fact means, and applied the definition at [54] above in considering their expert
evidence. I will return to the expert evidence in some detail later in the judgment, including in the
Technical Appendix.
RLITO000005
RLITO000005
C. Features of this Group Litigation
57.There are many different ways of managing group litigation. The subject matter of such litigation is
different from group to group, and what is appropriate in one set of proceedings will not necessarily
be the best approach in another set. Group litigation may involve many claimants - even tens of
thousands, in some cases - all with what is essentially the same type of claim, governed by a number
of common issues that apply across all or most of the cases. Some group litigation may involve similar
numbers of claimants, all with similar claims that are factually different. In this case, there are 584
claimants, which in the context of some group litigation, is not very many, although it is still in the
many hundreds, and the period of time over which the relevant events are said to have taken place is
about 16 or 17 years. There are different aspects to each individual case, but to deal with the
litigation efficiently, cost-effectively and proportionately it is simply not feasible for the same judge to
try all the claims, one after the other, in full on all of their respective merits. Such a process would
take several years. It is not what group litigation is intended to achieve.
58.The concept of group litigation is that the Managing Judge, whoever he or she may be, with the
assistance of the parties insofar as that may be available, selects the most suitable mechanism for that
particular set of proceedings in order to achieve compliance with the overriding objective. That will -
or should - lead to cost and time saving. That does however require a high degree of co-operation
from the litigants. Here, my intention of holding a substantive trial each judicial term onwards to
resolve as much of this group litigation as possible, as quickly as practicable, became simply
unachievable as a result of the issuing of the recusal application by the Post Office, explained at [5]
above. A further trial in the autumn of 2019 also became undesirable as a result of the parties wishing
to have time to consider mediation, and to find out whether the Court of Appeal would grant the Post
Office permission to appeal on the contractual issues in Judgment (No.3). Over the numerous hearings
and two full substantive trials that I have conducted, I have gained the distinct impression that the
Post Office is less committed to speedy resolution of the entire group litigation than are the claimants,
but it is not possible to state with finality whether that is correct.
59.All litigation is important to those involved in it. In this group litigation, all the claimants are of the
view summarised at [10] in Judgment (No.3) in terms of the behaviour of the Post Office. Recitation
that the claimants are of that view does not mean that any similar views are held by the court, or that
any decisions have yet been made by the court on that point one way or the other. The total sums
claimed by way of liquidated sums (excluding heads of general damages not yet quantified) are
approximately £18.7 million. The Post Office counterclaims for certain sums against the claimants and
also alleges fraud. Reputations are plainly at stake on both sides. That is the case in many types of
proceedings, particularly high-profile ones that attract press interest. However, that sum of money is
not large, in the context of large scale and protracted litigation. The parties’ joint costs are
approximately now £27 million, and because of the notification provisions in one of the earlier Case
Management Orders I made, regular notifications of the total sum of costs are made to the court. The
Post Office alone spent over one million pounds in little more than a month earlier this year; the
notification letter of 13 May 2019 from its solicitors stated that its costs were in excess of
£12,800,000; by the time of its letter of 25 June 2019, the Post Office’s costs were in excess of
£13,900,000. Both this level, and rate, of expenditure is very high, even by the standards of
commercial litigation between very high value blue chip companies.
60.1 have already explained that the Post Office has now used four different leading counsel, and it
also engaged a second commercial firm of solicitors to act for it on its appeal. The claimants are
funded by means of litigation funding, explained in Judgment No.3 so far as it is relevant. The
RLITO000005
RLITO000005
claimants are ordinary individuals who ran branch Post Offices, and the Post Office is publicly funded,
so none of the parties are high value FTSE 100 companies. Not all of the Post Office’s costs will be
recoverable costs regardless of the outcome, not least due to the existence of two Costs Management
Orders which are in smaller amounts than the Post Office’s rapidly growing grand total, but the total
figures still represent real expenditure of actual money.
61.These proceedings also have the additional feature of criminal convictions on the part of some
SPMs. As explained at [10] in Judgment (No.3), there is a Criminal Cases Review Commission
(“CCRC”) review underway in respect of the convictions of
some claimants. I have been told that these are subject to a stay pending judgment upon the Horizon
Issues. I have explained this before in open court, but matters such as criminal convictions are no part
of this group litigation. This court has no jurisdiction over such matters.
62.Although criminal convictions are no part of these proceedings, examples of alleged software bugs,
errors and/or defects affecting branch accounts, which led to certain adverse consequences for SPMs,
did form part of the subject matter of the Horizon Issues trial. Given the impact (actual or potential)
upon branch accounts, with the potentially adverse implications for SPMs generally in the
background, and given the way that the SPM witnesses in this trial were cross-examined (as with the
Common Issues trial, some being accused expressly of criminal offences), criminal convictions were
not part of the trial, but were part of the background. In the claimants’ oral closing submissions, Mr
Green QC for the claimants identified a high level chronology of what he called “doubling up”, by
which he meant bugs, errors and/or defects that led to certain entries in branch accounts being
“doubled” incorrectly. He did so by specific reference to allegations by a particular SPM who had been
convicted of a criminal offence.
63.Mr de Garr Robinson QC for the Post Office objected to this, complaining of what he said was jury
advocacy. In the course of what became an increasingly vigorous exchange between counsel at the
very end of the trial he stated, by reference to what he said were the rules of commercial litigation,
that “one of those rules is that one doesn't say things incautiously that might have an impact on
evaluations being done in another place in relation to different proceedings.”
64.Prior to considering the “rule” to which he was referring - it is not one of the Civil Procedure
Rules, so far as I am aware - it was therefore necessary to identify exactly what “different
proceedings” were being referred to in this disagreement between the parties. There are five
elements to this matter.
(1)Firstly, it had never been raised by any party at any stage during this group litigation that there
were any criminal prosecutions currently underway such that it was necessary to consider any
reporting restrictions for the group litigation. I expressly raised this with the Post Office on the
occasion referred to at [62] and [63] above, who confirmed there were no such prosecutions
underway. The “different proceedings” to which Mr de Garr Robinson referred were the proceedings
already before the CCRC.
(2)Secondly, the possibility of future (as opposed to current) criminal prosecutions, or the potentially
criminal impact upon individual SPMs, did more than hover in the background to the Horizon Issues
trial. Some claimants who gave evidence in this trial were expressly accused by the Post Office of
criminal offences in crossexamination in this trial, something which had also occurred in the Common
Issues trial. It should be clearly understood that any findings I make in respect of any witnesses do
RLITO000005
RLITO000005
not determine with finality any issues to be tried in any particular trial to follow, including the trial(s)
of their individual claims.
(3)Thirdly, the rule to which leading counsel for the Post Office referred was not identified. Journalists
must be alive to the risk of serious prejudice when criminal proceedings are live, as a result of the
Contempt of Court Act 1981, and are thereby restricted in reporting matters generally in that respect.
That statutory provision replaced what had been referred to as the sub judice rule, but this did not
relate to statements made in evidence in open court. The principle of open justice is a most important
one and there were no restrictions imposed on any reporting of the Horizon Issues trial (nor the
Common Issues trial) at all, nor were any sought by the claimants or by the Post Office.
(4)Fourthly, there was no specific restrictions imposed on the parties by the court in terms of the
evidence they could adduce and what could be said by way of submissions (written or oral), due to
potential impact upon any “different proceedings”. I have not heard any argument on the matter, but I
do not consider it would be proper to do so in any event. It was also not argued before me that the
CCRC proceedings, which are subject in any event to a stay pending at least part of the outcome of
the group litigation, are in any way of such a character that reporting of the Horizon Issues trial ought
to be restricted in any way, or such that they could be prejudiced by the group litigation. Indeed, the
presence of a stay of those proceedings pending the outcome of some of the group litigation issues
would suggest directly to the contrary. The objection therefore by the Post Office to the way in which
the claimants sought to make closing submissions on the actual impact upon different SPMs of what
were said to be bugs, errors and defects in the Horizon System was, so far as it could be understood,
not a well-founded one.
(5)Finally, the claimants by their counsel were trying to draw parallels between what had happened to
one particular SPM regarding “doubling up”, and numerous entries in internal Fujitsu and Post Office
documents prior to that of similar “doubling” occurrences being caused by bugs, errors or defects.
Those parallels are obvious on the face of the documents. The degree to which the CCRC find such
parallels of assistance, if at all, is a matter entirely for them. The Post Office submitted most strongly
that these different references were not to one single bug, but were references to a number of
different issues or bugs that had been experienced in Horizon over the years. Whether that makes the
claimants’ point for them or not, it is important to remind all the parties that the issues in this
litigation are not going to be decided with sympathy, or lack of it, coming into account in the analysis
in any respect whatsoever.
65.Mr Green for the claimants submitted that the Post Office was “trying to distract from the stinging
nature of the underlying documents by objecting” to the submissions and that the Post Office may be
finding it “uncomfortable” to have its own documents, in a particular date range, highlighted by
reference to particular criminal cases, prosecutions mounted by the Post Office, and the experiences
of SPMs in those cases. Certainly the Post Office and Fujitsu have been exposed to a degree of
scrutiny in this litigation which does not appear to have occurred before; however, there has not been
litigation of this type on these issues before.
66.The claimants and the Post Office doubtless know this already - certainly their legal
representatives will - but given the high profile nature of this dispute and the fact that this judgment
may be read by those other than the parties themselves, I will make the position very clear:
(1)The Senior Courts of England and Wales (who acquired this name by reason of the Constitutional
Reform Act 2005) comprises the Court of Appeal, the High Court and the Crown Court.
RLITO000005
RLITO000005
(2)The Court of Appeal has two divisions, Criminal and Civil. Appeals from the High Court are to the
Civil Division. Appeals from the Crown Court are to the Court of Appeal Criminal Division, also known
as the CACD.
(3)The High Court is the highest court of first instance in civil cases.
(4)The Crown Court is the highest court of first instance in criminal cases. Some criminal matters are
dealt with by the High Court (by way of Divisional Court) but these are narrow in scope and do not
arise in any respect concerning the group litigation. All appeals against both convictions and sentence
from the Crown Court are dealt with by the CACD.
(5)The Criminal Cases Review Commission, or CCRC, is a statutory body responsible for investigating
alleged miscarriages of justice in England, Wales and Northern Ireland. It was established by section
8 of the Criminal Appeal Act. 1995. It is an independent non-departmental government body, and has
the power to refer cases to the CACD.
(6)This group litigation is concerned only with the issues arising in the civil claims being brought
against the Post Office by the claimants, and the Post Office’s counterclaims. It will result in a series
of judgments on those issues which are public. What, if anything, the CCRC do in any respect
following any of the judgments is entirely a matter for the CCRC and forms absolutely no part
whatsoever of the group litigation.
(7)This court has no jurisdiction in respect of any of the convictions of those SPMs who were
successfully prosecuted by the Post Office. Although the presence of criminal convictions does have
evidential effect in respect of individual claims by individual claimants who have been convicted of
false accounting, these have not arisen in either of the two substantive trials held to date (Common
Issues and Horizon Issues) nor will they arise in either of the next two (the principles governing
Heads of Loss, and then some individual claims).
67.There has been no restriction imposed on any party, or any witness, in this group litigation by the
court in terms of the evidence that could be adduced, or submissions that could be made, with one
exception. Mr Henderson of Second Sight was subject to a restriction upon his evidence by reason of
the terms of an agreement that he had entered into with the Post Office when Second Sight were
engaged in what was originally intended to be consensual resolution of the claims. This was called the
Second Sight Mediation Scheme. This feature of the evidence of Mr Henderson is a matter which is
dealt with in more detail in Part D of this judgment. It was a restriction imposed upon him, or agreed,
by the parties, and the court was not involved in that restriction in any way.
68.Turning to a different matter, I have already made certain criticisms of the Post Office in Judgment
(No.3) in terms of how it had conducted itself in the litigation, and I had
also made criticisms of some of its senior witnesses. One of those witnesses, Ms Van Den Bogerd, was
also called as a witness in the Horizon Issues trial. So far as I am concerned, she came to the court for
the Horizon Issues trial with a clean slate in terms of whether her evidence on the Horizon Issues
would be accepted or not. A different way of putting the same point is that simply because certain
adverse findings had been made concerning the evidence she had given in the Common Issues trial,
this did not mean that I adopted any particular starting position so far as her factual evidence was
concerned for this trial. The court was entirely neutral in terms of starting position. Although it was
certainly not a point that went to her credit that I had already made adverse findings of the accuracy
of her evidence in the Common Issues trial, that did not mean that I started with any fixed view of the
RLITO000005
RLITO000005
likely accuracy of her evidence in the Horizon Issues trial. Her evidence in the Horizon Issues trial is
dealt with at [203] below.
69.Further, before turning to the detail of each side’s evidence, an approach was adopted by the Post
Office on occasion of seeking to adduce what was (or should have been) in reality evidence of fact, but
by way of submission, or points made “on instruction”. Sometimes, depending upon the nature of the
subject matter, such an approach is understandable or unavoidable, and may be unobjectionable. It is
not therefore sensible to state that this should never be done in any conceivable circumstance in any
trial. However, on important points that have been dealt with by a particular witness in their evidence
of fact, it is not a suitable device to adopt. This was particularly done in terms of the cross-
examination of Mr Coyne concerning evidence already given by Mr Godeseth in his cross-examination,
about alteration by Fujitsu of a particular branch account. I deal with that in detail at [376] to [379]
below.
70.This was also done in Appendix 2 of the Post Office’s Closing Submissions, where (sometimes
detailed) factual explanations were given in respect of bugs in the Bug Table. That appendix was
compiled by different teams of solicitors and counsel, something explained by the Post Office when,
some months after it was submitted, they discovered that three pages were missing and sought
permission to serve them rather late. I granted permission for them to be added, as they had been
prepared before the deadline for service and omitted due to an administrative oversight.
71.Submissions should not contain evidence, or positive evidential assertions, that are not present in
the evidence served in the trial. This is a fundamental point. I provide some examples in the Technical
Appendix by reference to specific entries for specific bugs. Blurring (or ignoring) the lines between
submission and evidence is entirely unhelpful. Evidence is something that comes from a witness (lay
or expert) and which the opposing side is entitled to test by way of cross-examination. It is not
appropriate for detailed factual assertions to be made in closing submissions that are not directly
referable to evidence in the case. There is no way such factual assertions can be tested; if they come
in closing submissions, there is no way that the opposing party can deal with those assertions in their
own evidence, or even put relevant points to witnesses for the other party in cross-examination.
72.Further, this is not a case that is being tried in a Specialist List, such as the Technology and
Construction Court - it is a general Queen’s Bench Division case - but it readily could have been tried
in such a list. It contains a great deal of technical subject matter, particularly in this trial dealing with
the Horizon Issues. The two IT
experts have each given evidence in other computer litigation before. Such subject matter, and such
expert evidence, is readily suited to analysis by the parties and precision, which is the usual approach
of courts generally. In my judgment, this is particularly important in technical matters such as these.
Bluster, unfounded assertion in cross-examination, detailed technical explanations of fact given “on
instruction” and submission are rarely helpful.
73.Finally, there has been a vast amount of highly detailed material deployed by both sides, not simply
evidence of fact, and expert evidence, but also reference to a great many documents. The use of an
electronic bundle has made this easier to manage than otherwise, but written submissions alone were
in excess of 1100 pages in total from both sides. It is neither possible nor desirable to recite in this
judgment, or in the Technical Appendix, or resolve, every single disputed item, or every single
disputed fact, no matter how minor. I only make findings in this judgment that are necessary to enable
me to resolve the Horizon Issues themselves. Simply because I do not specifically refer to a particular
submission or piece of evidence, it should not be thought that I have not had regard to it. I have
RLITO000005
RLITO000005
considered all the material, evidence (both factual and expert), submissions, and passages in
contemporaneous documents, multiple times. The experts’ agreements in particular have been of
great assistance, but everything has been considered. This judgment, together with the Technical
Appendix, will be of substantial length, and to recite everything would very probably make it of
unmanageable proportions. It is in any event far longer than is ideal, but that is partly explained by
the fact it deals with the life of a complex computer system that spans some 18 years.
74.In adopting this approach, I have borne very much in mind the overriding objective in the Civil
Procedure Rules, the need for proportionality, but also the obvious need to provide a reasoned
judgment. I have also taken specific account of the dicta of Males
LJ in Simetra Global Assets Ltd and another v [kon Finance Ltd and another
2019] EWCA Civ 1413. That case concerned foreign exchange trading, and claims for dishonest
assistance and damages for deceit and conspiracy against a total of 12 different defendants, both
personal and corporate. Although it concerns those matters, plainly very different to the Horizon
Issues, it does state generally what ought to be included in a judgment. The requirement for a judge
to give adequate reasons in a judgment is analysed at [37] to [46], with the expression “the building
blocks of the reasoned judicial process” (used by Henry LJ in Glicksman v Redbridge Healthcare
NHS Trust [2001] EWCA Civ 1097) approved at [42] by Males LJ. Four points are summarised at [46]
of Simetra:
“Without attempting to be comprehensive or prescriptive, not least because it has been said many
times that what is required will depend on the nature of the case and that no universal template is
possible, I would make four points which appear from the authorities and which are particularly
relevant in this case. First, succinctness is as desirable in a judgment as it is in counsel's submissions,
but short judgments must be careful judgments. Second, it is not necessary to deal expressly with
every point, but a judge must say enough to show that care has been taken and that the evidence as a
whole has been properly considered. Which points need to be dealt with and which can be omitted
itself requires an exercise of judgment. Third, the best way to demonstrate the exercise of the
necessary care is to make use of "the building blocks of the reasoned judicial process" by identifying
the issues which need to be decided, marshalling (however briefly and without needing to recite every
point) the evidence which bears on those issues, and giving reasons why the principally relevant
evidence is either accepted or rejected as unreliable. Fourth, and in particular, fairness requires that a
judge should deal with apparently compelling evidence, where it exists, which is contrary to the
conclusion which he proposes to reach and explain why he does not accept it.”
75. I am acutely conscious that the first of those points, succinctness, is not likely to be achieved in
this judgment. This is not only due to the nature of the subject matter, the Horizon Issues, but also the
fact that both Legacy Horizon and Horizon Online are involved, and the system was brought in some
time ago, namely the year 2000. Due to the nature of the claims brought (and the limitation issues
certain to arise), the functionality of the Horizon system(s), robustness, and the other Horizon Issues
had to be dealt with at an early stage in the group litigation. I will return to the other three of Males
LJ’s four points after my review of the evidence, as well as his comments on the importance of
contemporaneous documents at [48] and [49] of Simetra. I do this at [937] and following below.
D. Evidence of Fact: The Claimants
RLITO000005
RLITO000005
76.The claimants originally served statements from a greater number of witnesses of fact than were in
fact called at the trial. This was because, just before the Pre-Trial Review on 14 February 2019, the
Post Office objected to the evidence of two particular individuals. Very shortly before the Common
Issues trial, the Post Office had issued an application to strike out a significant number of passages in
the six Lead Claimants’ witness statements, and that application had been dismissed in Judgment (No.
2) “Strike Out”. On this occasion, although no application to strike out was issued, the Post Office
sought a ruling at the PTR in respect of the evidence of Mr McLachlan and Mr Henderson.
77.Mr McLachlan had been called as an expert witness in the criminal trial of Mrs Seema Misra, a
SPM at West Byfleet in Surrey who was charged both with theft from her branch, and also false
accounting. He was called for the defence. The sums in question were approximately £74,000. Mrs
Misra pleaded not guilty, and her defence was that the Horizon system was to blame. She was
convicted by a jury after a trial in late 2010 at Guildford Crown Court, and was sentenced to a term of
imprisonment of 15 months. She was pregnant at the time of her conviction and imprisonment. Expert
evidence was called by the Post Office at her trial from Mr Gareth Jenkins from Fujitsu. Mr Jenkins
was not called as a witness by the Post Office in the Horizon Issues trial before me, but a large
amount of the evidence from the Fujitsu witnesses was attributed to information directly given to
them by Mr Jenkins. This was a controversial matter between the parties at the Horizon trial. The Post
Office did not proffer an explanation for Mr Jenkins’ non-appearance as a witness; they were not
obliged to do so. However, in their closing submissions and after all the evidence had been called by
both parties, they did so. The explanation provided was by way of submission and not evidence. This is
a matter with which I deal with further at [509] to [513] below. Prior to the PTR, the Post Office
objected to Mr McLachlan‘s evidence on the basis that it contained opinion evidence and the
claimants did not have permission for it, which is something required under the CPR. This was
resolved by the claimants deciding not to call Mr McLachlan, and explaining at the PTR itself
that they would not call him. On the face of it, the claimants were wise to concede this point as if his
statement had included opinion evidence, the Post Office would have had a valid point of objection. I
have not therefore considered the evidence of Mr McLachlan which forms no part of the case.
78.The other witness whose statement was subject to objection was Mr Henderson. Mr Henderson is a
Director of a company called Second Sight Support Services Limited (“Second Sight”). Second Sight
was appointed by the Post Office to conduct a review into problems with the Horizon system in July
2012. A number of reports were produced by Second Sight, and the Post Office responded to these.
These reports were as follows:
1.Interim Report dated 8 July 2013;
2.Briefing Report Part One dated 25 July 2014;
3.Briefing Report Part Two version 1 dated 21 August 2014;
4 Briefing Report Part Two version 2 dated 9 April 2015.
79.The Post Office submitted at the PTR that I should give a ruling concerning those parts of Mr
Henderson’s witness statement upon which the Post Office was required to cross-examine, and those
parts upon which it was not. This was said to be justified by a concern the Post Office had, namely that
if it were required to cross-examine upon the correctness or accuracy of the contents of the Second
Sight reports themselves, the Horizon Issues evidence would not be capable of being completed
within the time estimate for the trial, which at that stage was 16 days. Mr de Garr Robinson also
RLITO000005
RLITO000005
expressed concern that any failure by the Post Office specifically to cross-examine upon any particular
point in any of the accompanying Second Sight material, rather than the actual witness statement of
Mr Henderson, would lead to the claimants submitting that such a point was not challenged, and that
any such point would therefore be taken by the court as being formally agreed by the Post Office. I
declined to give a ruling directing the Post Office upon those parts of Mr Henderson's statement.
which should be subject to cross-examination. This was for three reasons.
80.Firstly, the answers to the different Horizon Issues would not be determined based on the accuracy
or correctness of the contents of the Second Sight reports themselves. The court was to hear detailed
expert evidence from two IT experts whose evidence would go directly to the Horizon Issues. Whether
the Second Sight reports were, or were not, correct in their conclusions (which were generally critical
of Horizon) would not form part of that process, and would not even qualify as a sideshow. Certainly
their contents would not be determinative of the Horizon Issues in the group litigation. I have had no
regard to the contents of the different Second Sight reports, nor to the Second Sight conclusions, in
arriving at the answers to the Horizon Issues. I am aware that all of the Second Sight conclusions
were challenged by the Post Office.
81.Secondly, it is not for the court in civil litigation to identify in advance to any litigant that it need
not cross-examine upon particular evidence of fact of the opposing party. This is particularly so in a
time-limited trial such as this one, but in my judgment is a general point of principle. The court was
effectively being asked to direct the Post Office as to how it should conduct its case, and also
consequentially, about how much of its time at the trial should be used for particular witnesses. This is
not the function of the court. I was aware that Mr Henderson’s evidence was challenged by the Post
Office. It is for this reason that he was to be cross-examined. How the Post Office chose to do that, the
trial time its advisers chose to allocate to that exercise, and which (if any) parts of the statement were
to be challenged and how, were all matters for the Post Office to decide.
82.Thirdly, it would in any event be wholly unconventional in a time-limited trial dealing with the
subject matter of this group litigation for the court to give any weight to any submissions by the
claimants that particular points of criticism or detail in such detailed documentation as the Second
Sight reports were essentially “accepted” by the Post Office because they had not been specifically
challenged in cross-examination. There is never sufficient time, in any time limited trial, to cross-
examine upon everything. A time limited trial in the 21st century is not conducted in the same manner
as a trial would have been many years ago, particularly in a detailed technical dispute such as this
one. There was no forensic route available to the claimants whereby a failure to cross-examine upon
the Second Sight conclusions or reports, whether due to the length of the trial or otherwise, would or
could lead to a conclusion by the court that some aspect of the Second Sight reports was agreed or
admitted by the Post Office, when it plainly was not. The period in question in the group litigation
spans some 18 years. If attention is to be paid to other court users, the overriding objective and the
Civil Procedure Rules, it is simply not possible (nor desirable) for any trial judge to permit the parties
to have unlimited time to cross examine upon everything. The quid pro quo of that is a party cannot be
expected to cross examine upon everything. There should have been no concern on the part of the
Post Office that the Second Sight report(s) would be taken as not challenged by the Post Office unless
their contents were subject to cross examination. Given there appeared to be such a concern, I
explained the position at the PTR. The claimants also expressly said at the PTR that no such point
would be taken. That latter element was unnecessary, as whether the point was taken by the
claimants or not, it would not have been accepted.
RLITO000005
RLITO000005
83.The position was therefore clarified or explained at the PTR and the Post Office, notwithstanding
that no formal ruling was made on those specific parts of Mr Henderson’s statement that it had to
challenge in cross-examination, appeared to be content. A transcript of this hearing is available. The
contents of the Second Sight reports were not cross-examined upon when Mr Henderson gave
evidence, but in my judgment there was no need for them to be. As it happened, Mr Henderson was
not cross-examined for very long, but that does not matter. I have not taken any account of the
contents of the Second Sight reports in deciding the Horizon Issues.
84.The claimants therefore called the following witnesses of fact. For the reasons that I explain below
in relation to Mr Roll for the claimants, and Mr Godeseth for the Post Office, I consider these two
witnesses in particular to have been of the greatest assistance in resolving the Horizon Issues.
However, all of the factual witnesses contributed to my understanding of the system over the years,
both generally and specifically. The specific experiences of the claimant witnesses who gave evidence
about what occurred in their branches are, if I accept them, specific examples of the working of the
Horizon system at the branch Post Office end in practice. I will deal with each of the witnesses in turn.
Mr Latif
85.Mr Adrees Latif was the SPM at Caddington Post Office in Caddington, Bedfordshire, since 2001
until late September 2018. His appointment with the Post Office therefore ended after the litigation
commenced, and he was subjected to an audit in September 2018. He was cross-examined by video
link from Islamabad, as it had been necessary for him to travel to Pakistan as a result of a family
bereavement. He left the UK on 19 February 2019, and he travelled from Kashmir to Islamabad to
give his evidence by video.
86.He gave evidence about two specific incidents. One occurred in July 2015 and related to the
transfer of £2,000 from the AA stock unit to the stock unit designated SP1. Originally his statement
had said it was the SJ1 stock unit but this was a typographical error which he readily accepted. He
successfully transferred the £2,000 from AA, but when he went to the SP1 unit, the same sum had not
transferred into that unit successfully. There was no explanation for this that he could come up with,
including having checked his own CCTV, and he was sure he had carried out the transaction correctly
- it was not an unusual transaction. He described the sum of £2,000 as having “simply disappeared
from Horizon” and explained that this would lead to a shortfall in the branch account for that sum.
87.The second incident occurred in January 2018 in relation to Camelot and scratch cards. His
heading for this he corrected to “Transaction Acknowledgement Issue”. He explained that certain
information was sent by Camelot to Horizon twice, and the Post Office sent out a notice stating that
due to this mistake by Camelot, a Transaction Correction or TC would be issued. The correction
occurred and he accepted it, but the stock figure for scratch cards in his branch on Horizon remained
unchanged. This therefore showed his branch as having £1,000 more in scratch cards than was
actually present in the branch. This issue was still outstanding as at the date of his amended
statement which was 1 March 2019, and remained outstanding as at the date of his cross-examination
which was 12 March 2019. This is, obviously, a period well in excess of one year.
88.The Post Office’s case on the £2,000 was put methodically to Mr Latif. That regarding the stock
transfer is best summed up in the evidence of Ms Van Den Bogerd, who stated that provided certain
steps or action were carried out correctly, what Mr Latif had said happened simply would not occur.
She said, inter alia “providing these two actions are completed, the stock unit from where the cash is
transferred should not show a discrepancy" (emphasis added). As she put it, her “strong belief is that
Mr Latif has recalled these events incorrectly”. She also said that “the records that Post Office has
RLITO000005
RLITO000005
reviewed do not support what Mr Latif has said and I believe that he may have mis-recollected events
from 3 years ago”. The Post Office’s case was essentially that Mr Latif had not done the steps
correctly, because had he done so, what he said happened could not have happened.
89.Mr Latif’s response to that was both consistent, considered and credible, and was best summarised
in one of his answers:
“A, I'm experienced -- I have been running a post office for 17 years, sir. I have also worked for the
Post Office on training other offices how to run a post office. I was also involved in running and
introducing the new Horizon software changes in 2006 onwards, where I went to several offices on
behalf of the Post Office to give them training. So I'm an experienced, trained subpostmaster and I ran
my business successfully for 17 years. So I may have been a bit brief in the statement but obviously I
can run through those -- exactly those steps that we would take to make sure that there is no operator
error on our behalf.”
90.The conflicting evidence on this particular point is a good illustration of the “poles apart” position
to Horizon by the Post Office and the claimant SPMs in this litigation. Because of the Post Office’s
position on Horizon, almost all and any of the criticisms or accounts of factual events which the
claimants made, or make, about how this system worked in practice are attributed to fault or
carelessness by the SPM or their assistants. Indeed, without fault or carelessness by an SPM, the Post
Office simply cannot explain these occurrences. The Post Office’s position is therefore to challenge the
factual account - which it is entitled to do - because if the factual account by an SPM is accepted as
truthful and accurate, then the Post Office would have to accept that there must be a fault or faults
within Horizon. Therefore, the Post Office cannot accept that the factual account is truthful and/or
accurate. Thus the dispute goes around and around in endless circles. This litigation is aimed at
breaking that deadlock.
91.It is also the case that Mr Latif had the following point positively put to him about why he had
checked the CCTV that he had within his branch. “So you now say you looked at the CCTV because
your colleagues were concerned that you hadn't done the transaction properly?” even though Mr Latif
had said no such thing. Mr Latif was subject to a fairly robust attack, not only on his account, how it
matched up with other records which the Post Office said contradicted it, what he and his assistants
had or had not been doing, and indeed upon the full scope of his evidence and his credibility - as
shown by the question I have reproduced. That question was framed as though even his own
colleagues had concerns about what he had done. It was positively put to him by the Post Office that
he had not even complained to the Post Office, although he provided the name of his Area Manager
Mr Navjot Jando and said he had complained to him many times. The Post Office did not call Mr Jando
to rebut this. One exchange will suffice as an example of the type of attack upon Mr Latif:
“Q. You don't say anywhere in your witness statement that the £2,000 physical cash also somehow
disappeared, but that seems to be what you are now saying, is that right?
A.Well, the system gave a shortfall of £2,000 and that's been my statement all the way through, sir, so
I don't know what you're trying to confuse me, but there's a shortfall of £2,000 in stock unit AA and
there should not be a stock shortfall. The money is physically there.”
92.Mr Latif’s evidence had never been, so far as his witness statement and evidence orally before the
court, that £2,000 in cash had physically disappeared. His evidence was that there was a shortfall of
that amount shown in Horizon as a result of what he had done. There were some aspects of the cross-
examination of Mr Latif which were simply unhelpful. Firstly, extensive spreadsheets were put to Mr
RLITO000005
RLITO000005
Latif which he had not seen before. They are plainly not in chronological order, and had in any event
been what was called “filtered” by the Post Office legal team; they were not agreed by the claimants,
nor was the “filtering” process explained at any point. There is a limitation on the degree of assistance
to be obtained by such an exercise. Further,
there was no agreed exercise by the experts whereby the two of them had gone through all the
records directly relating to this specific instance and agreed what the records did, or did not, show.
The Post Office’s case amounted, literally, to a pure challenge of fact that what Mr Latif said had
occurred, simply did not occur in fact.
“Q. On the basis of that, Mr Latif, Post Office says there was no failed transfer such as that described
in your witness statement and that you are simply wrong about that, it never happened.
A.So you are calling me a liar?
Q. Mr Latif, you may be mistaken or you may be lying. I put the question that it didn't happen.
A. Well, I state that they did.”
93.So far as the other issue experienced by Mr Latif in January 2018 was concerned, the Post Office
accepted, as Ms Van Den Bogerd’s original written evidence (prior to later correction) put it, that its
data showed “that the branch received two TAs (meaning Transaction Acknowledgement) on 18
January 2018. However, due to an error by Post Office, instead of increasing the scratch-card stock,
the TAs decreased the stock. To be clear, this was a data entry error by Post Office and not an issue
with Horizon. Horizon processed the TAs accurately. Lnote that the TAs were accepted by the branch.
which could have heen challenged at that point if the user had noticed that the TAs were not for a
(emphasis added)
94.However, this is notable for the following reasons. The first was that the blame is shifted back to
Mr Latif for not spotting the mistake and challenging the TA. His evidence was that it was not possible
to challenge a TA. They simply have to be accepted. This was accepted by Ms Van Den Bogerd when
she came to give evidence orally. Therefore the notable point made by Ms Van Den Bogerd, a director
of the Post Office, in her written evidence, to shift the blame back onto Mr Latif, is simply wrong in
fact. TAs cannot be challenged, they have to be accepted.
95.The second is that the Post Office evidence entirely omits any reference to the “memo view” - a
type of communication used by the Post Office on Horizon which leads to a “pop up” on the terminal
the next time each user logs on. There was simply no reference to this at all by Ms Van Den Bogerd,
and Mr Latif said that this was sent out by the Post Office “to everyone” saying the error on the
Lottery would be corrected by means of the issuing of a TC. Finally, Ms Van Den Bogerd corrected this
passage in her evidence in chief by way of printed corrections prepared before she was called. She
said that these corrections had been handed to her solicitors prior to the trial. If that is correct, it is
surprising that Mr Latif was cross-examined on the basis of the original evidence in her statement, as
that was not going to be her evidence in chief. However, regardless of that, she readily agreed that
there is no choice regarding the acceptance of TAs - they simply have to be accepted by the SPM. In
those circumstances, there is no explanation as to how the emphasised sentence in [93] above came to
be in her statement at all, and it was plainly incorrect. Also, the call logs (about which Ms Van Den
Bogerd was cross-examined) concerning this incident entirely support Mr Latif’s evidence.
RLITO000005
RLITO000005
96.Finally, although this is a minor point compared to the ones in the preceding paragraph, the tenor
of her witness evidence that Horizon processed the TAs in this respect gave the impression that there
were no problems with the Lottery and TAs so far as Horizon were concerned.
97.Not only did the steps taken by the Post Office - the TAs - not correct the issue that occurred in Mr
Latif’s branch, but in cross-examination Mr Latif explained further:
“A. Can I just confirm, there was an audit done in September of this year, an audit by a Post Office
trained auditor, and my stockholding was still showing negative. And a Jane Lawrence is the auditor
and she has - still could not resolve this matter, so the problem hasn't gone away, the problem is still
there. And there have been a number of calls to the helpline to resolve that negative stock and it
hasn't worked. They haven't come back with a response.”
Mr Latif was very clear about this:
“A. I still state that an independent audit was done by the Post Office, a Mrs Jane Lawrence, in
September 2018 the stockholding was still negative and as the branch was handed over to another
subpostmaster that is going to be investigated, it's going to be investigated. If you are now coming
back with this evidence, I still say that there is a problem somewhere and I don't know what's
happened but we have still got a negative stock figure within our branch.”
He returned to this again, in the light of being pressed yet further on the point that it was human
error at the branch that had caused the discrepancy or shortfall.
“So the fact remains there was an independent audit happened by your Post Office -- by Post Office's
auditor, her recommendation was that they look into it as is (inaudible) the strategy is still there. And
there were a number of calls to the helpline pleading with them to resolve this issue before the audit
and there will be a complete trail of that, sir.”
He also referred to the what had happened as being “a glitch”.
98.I accept the evidence of Mr Latif, who struck me as a reliable and careful person, and who had
personally been the one who had tried to perform the transfer from one stock unit to another. He had
personally experienced what he explained to the court in this respect. I accept his direct evidence on
this in preference to that of the Post Office, which effectively was from people who were not there,
who maintained, more or less, that it simply could not have happened, and who had nothing to
substantiate or corroborate the challenge made to Mr Latif’s primary evidence. I find as a fact that it
did happen as Mr Latif explained. I find that Mr Latif performed the required steps correctly in
respect of the stock transfer between units, as one would expect of someone who had 17 years of
experience, and was sufficiently skilled at his role such that the Post Office had, prior to the litigation,
been sufficiently satisfied of his competence that he was used by the Post Office as a trainer for
training other SPMs. I also accept his evidence, which is direct primary evidence of the state of the
accounts at this branch, that neither he nor the auditor for the Post Office whom he named, have
resolved the issue concerning the Lottery, the incorrect TAs and the effect of the
TC, which remains an unexplained shortfall or discrepancy. Ms Van Den Bogerd’s evidence is a
number of steps removed from the branch, and is little more sophisticated than assertions that there
must have been other matters to blame, alternatively reliance upon records which did not, due either
to their contents or to the way that they were deployed in cross-examination of Mr Latif, demonstrate
the points that the Post Office maintained they demonstrated. I make further findings in respect of Ms
Van Den Bogerd’s evidence in the section below dealing with the Defendant’s Evidence in Part E. Mr
RLITO000005
RLITO000005
Latif’s branch was subject to an audit, and his appointment as a SPM ended then or shortly
afterwards.
99.In due course Mr Latif will have his own individual claim, and the Post Office with have its
individual counterclaim against him, tried. The only findings I am making in respect of his evidence
are those necessary for me to resolve the Horizon Issues. All other issues remain to be tried in those
later proceedings. Those later proceedings may explore in some detail not only the two specific
matters in respect of which he gave evidence in the Horizon Issues trial, but any others which are
relevant to both claim and counterclaim. They will do so by reference to other documents, as
explained in the section of this judgment Part K, Audit Data. The degree to which the findings of fact
that I make affect my conclusions on the expert evidence will be dealt with in Part L, Overall
Conclusions.
Mr Tank
100.Mr Jayesh Tank was the SPM at Fleckney Post Office, in Fleckney Leicestershire, from 4 May
2006 to 15 March 2017 when it was closed as part of the Network Transformation Programme. He
gave evidence regarding certain issues with Horizon that he had experienced, which included the
effects of a complete power failure to the building, which occurred mid-transaction. This occurred
whilst a customer was making a withdrawal in the branch from her Post Office card account. On his
evidence, this led to a shortfall in his accounts of about £600. Mr Tank paid this sum to the Post
Office, and the letter from the Post Office stating how it came to be owed described it as a “branch
discrepancy”; it was in fact £660. He paid it by way of a deduction from his remuneration,
101.The second issue was one concerning mail labels, when Horizon would (as he put it) “jump
ahead” to the end of the transaction and no label (which is what would or should be affixed to the
packet to be posted) would be printed. This would cause a loss in the branch accounts.
102.Ms Van Den Bogerd again gave evidence about these specific instances. Her witness statement
led Mr Tank to correct the date when he remembered the power failure occurring (he accepted it was
2014, not earlier in 2010 or 2011 as he had initially thought), and he corrected this in a supplemental
statement. Her statement had helped him pin down the date. This incident became the subject of a
PEAK at Fujitsu, although Mr Tank did not know this at the time. This supplementary statement also
exhibited to it old “posts” he had made contemporaneously on an internet forum on the website Yahoo
in respect of another loss, £195, which he had suffered in 2011.
103.Mr Tank had been interviewed by the Post Office investigators in 2015 and in the
Horizon Issues trial it became clear that certain matters extremely critical of his conduct were going
to be put to him in cross-examination. He was therefore given the warning against self-incrimination
under the Civil Evidence Act in the same way that the two claimants in the Common Issues trial, who
were accused in cross-examination of criminal offences, were given it. Mr Tank answered all the
questions that were put to him. Mr Tank had, eventually, used the icon on Horizon in his branch for
“Official Postage” incorrectly, and his explanation for this was that this was deliberate and was done
in order to get the attention of the Post Office. He said at his 2015 interview that all his attempts by
way of phoning the helpline and contact with his area manager had simply not resolved the numerous
complaints he had made. He frankly accepted that he had not sent the post by official means which he
had entered on the Horizon system as official postage, doing what he did by way of protest. It was put
to him in cross-examination that he had “helped himself” to official postage, which on the evidence
RLITO000005
RLITO000005
available was a suggestion that did not seem to have any basis in fact, if he had been doing what he
said he was doing. There was certainly no evidence before the court to substantiate this assertion.
104.In his interview on 5 November 2015 by the Post Office investigator Mr Bridges, it had been
accepted by Mr Bridges (who interviewed him) that Mr Tank had reversed the entries for official
postage in any event. This acceptance at the time by the Post Office investigator was directly contrary
to the way that the points were put to Mr Tank in his cross-examination, which were that he had
“helped himself” to postage and “improved his financial position” by acting as he did. Detailed
findings on this will have to wait for the full trial of his claim and counterclaim, but it is notable, in my
judgment, that the attack on the credibility of this witness in the Horizon Issues trial, and positive
allegations to him of criminality, were not consistent with the contemporaneous record of the
acceptance by Mr Bridges for the Post Office in the investigation interview in 2015. Mr Bridge clearly
accepted that the entries had been reversed. Indeed, Mr Bridges introduced this subject in that
interview as follows:
105."Okay thank you. Let's move on to the postage claims for the moment. So in my letter I gave
details of the claims and the reversals that you completed since 25 August. I think my first question
would be that I know that you reversed them but why undertake them in the first place bearing in
mind these are, in effect you are stating the transactions which took, well say the transactions have
taken place which you have used official postage for which in effect did not take place.”
106.The questions in cross-examination to Mr Tank, which must have been put on instruction, did not
take account of what the Post Office had accepted at that time in the interview. Another feature of the
interview is that Mr Tank was told by the interviewer “there are no issues with Horizon”. That
exchange is as follows:
“KB (Mr Bridges) I can confirm that there are no issues with Horizon and as my letter stated there is
no issue with the production of labels.
Jay (Mr Tank) Then obviously you conducted your investigation into my concerns without asking me
for my evidence. I've got the evidence here to show you. I'm happy for you to take copies but I'm
going to hang on to it but how can you say something is fully investigated if you've not looked at all of
the evidence.”
On the face of it, that is a valid point that Mr Tank made. Had there been an investigation, his
evidence would plainly be something that ought to have been taken into account by any investigator.
107.The Post Office relied, again, upon evidence from Ms Van Den Bogerd. Mr Tank was cross-
examined by reference to this and also by reference to a technical explanation concerning the fact, it
was said in cross-examination, that the power outage must have occurred at a particular point in time.
The cross-examination was as follows:
“MR HENDERSON: So what appears to have happened was a transaction from the Post Office card
account was in the middle of being processed and so it was in the stack presumably.
A. Yes.
Q. But had not yet been posted to Horizon. So you hadn't cashed out on that transaction, you hadn't
completed everything to do with that transaction?
A. On the stack - Q. It's on the stack.
RLITO000005
RLITO000005
A. It's on the stack, but the stack has a balance of zero, so to clear the stack you just press "enter" and
it goes straight -
Q. But you hadn't got to the point of clearing the stack?
A. I'm not sure.
Q. Okay. My suggestion is that there was probably an outage at just the point where the money had
been taken from Post Office card account but had not been processed onto Horizon. That's my
suggestion to you. A. Okay.
Mr Justice Fraser: Well, is the witness going to be in a position to agree or disagree?
Mr Henderson: Well, he might be if he recalled.
Mr Justice Fraser: Do you recall that happening when there were outages? A. No. I cannot recall.
Mr Justice Fraser: Were you aware of when outages would occur like that? A. Not all the time.
Mr Justice Fraser: Do you want to put the question again?
Mr Henderson: Yes. What I'm suggesting is that the cause of this problem was that an outage
occurred at a particular point in time. A. Yes.
Q. You were in the process of effecting a transaction from POCA?
A. Yes.
Q. It was in the stack and it had cleared from POCA?
A. Actually you mentioned the word “outage”. I'm not -- was there a power outage?
. iti l i .
A. Ah, okay.
Q. The system went down in some way.
A. So -- yes, because you said that if there's a power outage then there's evidence when you have to
log back in, so did that happen on this occasion?
Q. Okay, I want to come to all this and I'm doing this clumsily, What I'm suggesting is that what may
have happened -- and if you don't recall, you don't recall, but what may have happened is that the
. . ken fi he Post Offi ibef
you cleared the stack there was an outage . A. Possibly.
Q. Okay.”
(emphasis added)
108.1 have reproduced this passage because it demonstrates the following. The Post Office put a
possible explanation, based on technical grounds, to the witness who did not have either at the time,
or at the trial, the technical expertise, or the relevant technical records, or the data, to enable him to
agree or disagree. Even that explanation itself accepted “there may have been a problem with the
system” and “the system went down in some way”. The witness’ answer, which in my judgment is the
only sensible answer that could sensibly be given in these circumstances, was “possibly”.
RLITO000005
RLITO000005
109.This type of evidence obtained on cross-examination is of no assistance in resolving the Horizon
Issues, other than an implicit acceptance by the Post Office (on this occasion) that outages could,
potentially, in technically terms lead to what Mr Tank experienced. In any event, the Post Office’s
explanation accepted that “there may have been a problem with the system” which is equally
consistent with the claimants’ case, as it is with the Post Office’s case, if not more consistent with the
claimants’ case than with that of the Post Office. Power outages do happen, and there is a process to
be followed. Given Mr Tank was using Horizon Online, he was then in his crossexamination taken to
Version 5 of a HOL quick reference guide. The version to which he was taken was not, however, the
version available to him at the time, as that was a single A4 double-sided version in his branch, and
the one put to him at the trial was longer than two pages and available online. The date attributed in
the electronic trial bundle to Version 5, the version put to him in cross-examination, is 30 July 2015.
His forum posts and his supplementary witness statement made it clear that the incident occurred in
September 2014, a later date admittedly than given in his first statement, but still somewhat earlier
than July 2015. This was however the version of the guide that Ms Van Den Bogerd relied upon in
paragraph 78 of her 2nd witness statement to challenge Mr Tank’s account. I find the version that he
would have had available at the time was not Version 5, and was the single A4 version that he had
available to him in his branch in September 2014. However, and regardless of this, Mr Tank accepted
that recovery receipts would or should be generated and set out the procedure. He also accepted that
giving all the receipts to the customer was not the proper procedure.
110.In the PEAK dealing with the shortfall of his accounts of £195, which is PEAK PC0214226 and is
headed “Failed Recovery Transaction(s)” Fujitsu recorded:
“Date:14-Dec-2011 10:43:53 User: Wayne Bragg
Summary:
The banking transaction had completed (A3 rec'd and authorised @ 13:33:37), including the receipt
print (13:33:42), and money should have changed hands.
The basket settlement failed from 13:35 with ‘No response received from data centre’, and the two
retries also failed, and the attempt CANCELLED at 13:37.
The Disconnected Session receipts show "Cash TO CUSTOMER 195.04" so the customer's account
should be correct but the branch will have a shortage (for a withdrawal) because the session hasn't
been recorded.”
(emphasis added)
111.In my judgment this entry clearly supports Mr Tank’s evidence about how Horizon operated on
the occasion to which he referred. The heading of the PEAK was “failed recovery transactions”. The
PEAK clearly states that the customer's account would be correct - they were given the cash of
£195.04 as they expected, but “the branch will have a shortage (for a withdrawal)” as the session was
not recorded. A transaction correction was issued. There were related PEAKs, and also there was an
entry "KEL acha959T may be relevant” on the PEAK, and Anne Chambers was referred to, The PEAK
also recorded “Sending to SSC for investigation.” That KEL was raised by Anne Chambers on 28
February 2010, and the title was “HNGX banking reconciliation - state 4”. This is an important KEL,
and I deal with its further detail in the Technical Appendix.
112.There was a dispute between Mr Tank and counsel for the Post Office about whether this was an
example of “Horizon working as it is supposed to”, which Mr Tank did not accept, and also whether he
RLITO000005
RLITO000005
would have been refunded the sums (which he accepted had occurred) had he not phoned in and
reported the problem. It was put to him that he would have been refunded even had he not called in to
report, although how this would have occurred given the way PEAKs are initiated was not explained
to him. In any event, the PEAK demonstrated that what Mr Tank had said occurred, had indeed
occurred. Transaction Corrections are issued outside the scope of the Horizon system.
113.The position regarding the procedure for spoiled labels was, again, a situation whereby the Post
Office’s explanation was this simply could not have happened. Ms Van Den Bogerd’s evidence was that
there was a procedure available that Mr Tank should have used, yet the document put to Mr Tank that
was said to support this stated expressly in terms that “the label could only be spoiled if the label was
on hand” and Mr Tank’s point was that no label was “on hand”, given the problem he had was that the
label did not print at all. Using the procedure in the document therefore would not be possible, and
indeed would be contrary to the Post Office instructions that it could only be used if the label was on
hand. These instructions also required the SPM to write “spoilt” on the label and keep it with the
Horizon receipt for two years. This could not be done if the label had never printed. The explanation
by Ms Van Den Bogerd, put to Mr Tank, was also directly contrary to the contents of a letter dated 7
September 2015 from the Post Office to Mr Tank which stated:
“Your enquiry has been investigated and I can confirm there are no issues with Horizon online or the
production of postage labels which would cause the situation you described. However if a user either
pressed the yes or return key quickly before screen messages appear this can lead to a user
confirming a postage print has happened when in rare circumstances it may have failed and put the
cost of the failed print into the basket. In this circumstance Subpostmasters should contact Network
Business Support Centre to arrange a credit for the spoiled postage.”
This means that the Post Office in that letter accepted that in some circumstances, depending upon
how quickly a key was pressed, the printing of a postage label could fail and cause an impact to
branch accounts. The SPM must make a telephone call to the helpline, to correct the fact that the
branch account would show a charge for the postage even though no label had been printed and
hence could not be provided to the customer. This letter suggests not that Mr Tank had not followed a
particular procedure for dealing with spoiled (rather than non-existent labels) but that the system
would, depending upon how quickly a user pressed a particular key, not print a label, even though the
cost of the failed print would go in the basket. By going “in the basket” this means the branch
accounts would include the cost of that label as a debit to those accounts. In other words, this letter
supported Mr Tank’s direct evidence.
114.The Post Office in cross-examination put the following to Mr Tank:
“Q: There were procedures built into Horizon to cater for the situation that you explained -- I have to
say in the vaguest of terms, but as I understand what you are saying, there were procedures in place
which ensured you could deal with the situation, weren't there? A. No.
Q. We will have to differ.”
I would express this rather differently. If the speed of pressing a key could lead to the cost of a printed
label being added to the basket (and hence branch accounts) even though that label had not printed,
it is difficult to see that there is a “procedure built into Horizon” as defined to cater for this, On the
contrary, it appears from the Post Office’s own letter in 2015 that there was no such procedure. For
the avoidance of doubt, I find that there was no such procedure built into Horizon. The evidence
RLITO000005
RLITO000005
demonstrates to the contrary. Phoning the helpline and asking for a refund is not “a procedure built
into Horizon”.
115.Further, it is difficult to see how the explanation proffered by Ms Van Den Bogerd regarding how
to deal with a spoiled label could apply to Mr Tank for two reasons. Firstly, I accept that as a matter of
common sense interpretation and language, a label cannot be “on hand” if it was never printed. Mr
Tank did not consider he could properly use the procedure in the guide for the problems he
experienced, and I agree with him. Secondly, the Post Office’s own explanation to him in 2015 was
very different to what he was told in cross-examination in 2019 that he should have been doing. Even
were I to assume (in the Post Office’s favour) that there must have been instructions from the Post
Office somewhere regarding how a SPM should behave when a label failed entirely to print, as print
failures (which would not produce a label “on hand”) were plainly not unheard of, it remains to be
seen what those instructions were, and whether those instructions conflicted with what he was told in
the letter in 2015. No such instructions have been produced by the Post Office in any event.
116.1 consider Mr Tank to have been a credible witness. I find that Mr Tank’s experience in branch
was not an example of Horizon working “as it was supposed to”. I find that the creation both of a
PEAK, and reference to a KEL within that PEAK, to be consistent with my conclusion. The proposition
that Horizon worked as it was supposed to is an obviously flawed one. Indeed, that proposition is also
inconsistent with common sense. PEAKs and KELs are not created for situations where Horizon is
working “as it is supposed to”. They are created to deal with errors. The acronym KEL is for Known
Error Log. Nor are matters referred to the SSC for investigation when Horizon is working correctly.
Whether what occurred afterwards was an example of Horizon working correctly depends on the
categorisation of Transaction Corrections and whether they are part of the Horizon system or not. I
explain later in this judgment that they are not, therefore the issuing of a TC cannot be Horizon
working as it should. Mr Tank also gave evidence, which I accept - and this is made out in the
contemporaneous documents - that he had some difficulty in advancing this matter through the
correct channels. The helpline operator with whom he was dealing initially refused to “pass up” the
matter to a more senior person. Whether this was an isolated instance of unhelpfulness in this single
case, or a more generally obstructive approach across the helpline, will have to wait for future trials
in this litigation, and I recite it for completeness only in respect of Mr Tank’s experience. It does not
form part of my consideration of the Horizon Issues.
117.Mr Tank was accused of criminal offences and it was said that he had “helped himself” to official
postage and had “taken official postage”. These accusations were not put to him at his interview in
2015 and indeed the text of that interview shows that the Post Office interviewer expressly accepted
that he had reversed the transactions, which is not consistent with Mr Tank “helping himself”. There
are 10 pages of detailed closing submissions by the Post Office about Mr Tank’s evidence, and
although he is criticised for being vague, confused, imprecise and for not having prepared his
evidence with care, the allegation of criminality is not raised. However, the point was positively put to
him in open court, and Mr Tank is entitled to a finding on it. In my judgment, it would be quite wrong
to leave such an accusation hanging in the air, unresolved in this way. I find that on the material
before the court in this trial and on the evidence available and put to him, these serious accusations
are not made out. That is not to say I have made binding findings on the full details of his claim which
remains to be tried in due course. The Post Office submitted, both in respect of Mr Tank and other of
the claimants’ witnesses, that they confined their evidence to what was described as “a small sub-set
of his complaints, presumably because it was felt that this threw light on the Horizon issues - and Post
Office’s evidence was similarly confined”.
RLITO000005
RLITO000005
118.However, there are two important points that must be borne in mind about that submission, both
as it is made in respect of Mr Tank, and indeed the other witnesses called for the claimants. This has
not been a trial of their individual claims. There may be other issues about Mr Tank’s branch
accounts, raised both by him and by the Post Office, at the full trial of his claim and the Post Office’s
counterclaim. That does not mean that this specific evidence, about specific incidents experienced on
Horizon, should be discounted. Indeed, this specific evidence is not only directly relevant to the
Horizon Issues, but is, in my judgment, very important evidence going to those issues. The second
point is the parties’ agreed wording of the Horizon Issues does not allow for any wider issues going to
Mr Tank's branch accounts to be considered in this trial. All the other issues between Mr Tank and the
Post Office remain to be tried in the later proceedings. The only point of difference between them on
one of his two issues concerning Horizon was whether what occurred when a transaction correction
was issued could properly be described as “Horizon working as it is supposed to”, and certainly the
KEL referred to in the PEAK appeared during the Horizon Issues trial more than once. The creation of
a PEAK, and the KEL to which reference was made
within it, in my judgment demonstrates that this is not Horizon working as it should, as does the issue
of a TC. The wider consideration of these matters that are dealt with both by the experts and later in
this judgment. I reject the suggestion that Horizon was working as it was supposed to on this
occasion. So far as Mr Tank’s experience with the £195 shortfall is concerned, there was a Horizon
Online failure; three identical receipts were printed, which should not have happened; the receipts all
showed a disconnected session; and there was a loss in the branch accounts that evening for the
amount paid out to the customer. This was even though Mr Tank had settled with the customer for the
amount specified on the receipt. There was no record of the transaction at all on the transaction log
that was produced for the period of the Horizon Online failure. None of these, in my judgment, are
examples of “Horizon working as it was supposed to”.
119.The operation of the helpline is not part of the Horizon Issues, so it is not necessary to consider
and make findings on what Mr Tank said the helpline told him, which he explained in his posts was
“that the loss is mine unless I can sort out with customer directly.”
120.The later proceedings concerning Mr Tank’s claims (and any counterclaim) will probably hear
further evidence which are relevant to both claim and counterclaim. They will do so by reference to
other documents, as explained in the section of this judgment Part K, Audit Data. The degree to which
the evidence of fact affects my conclusions on the expert evidence will be dealt with in Part L, Overall
Conclusions.
Mr Anup Patny
121.Mr Anup Patny became the SPM of Spencefield Post Office in Leicester in October 2014. He is
currently suspended by the Post Office, having been suspended on 17 August 2016. His son, Mr
Aakash Patny, also gave evidence. I shall refer to Mr Anup Patny as Mr Patny Senior for this reason. I
shall also deal with his evidence, and that of his son, together for reasons that will become apparent.
122.There was what he described as a “major system outage” on 9 May 2016 in his branch, which was
a single counter branch run in conjunction with his retail business in the same premises. This was just
a few months before he was suspended. This outage led to the closure of his branch and he believed
that this had affected the whole network. At the end of the next trading period, which was on 11 May
2016, there was a shortfall of almost £17,000, predominantly made up of a shortage of 16,000 coins of
£1 denomination. This was a very large number and he said that he knew he would never have such a
large amount of £1 coins in his branch. He was also the person in the branch who had dealt with the
RLITO000005
RLITO000005
cash delivery that took place on the relevant morning, which is called “remming in” the cash. This
means accepting cash into the branch from the cash delivery that Post Office makes to the branch,
and entering the cash received into Horizon. His son, who also gave evidence, followed this matter up
with the Post Office. On 19 May 2016 the Post Office contacted him about a discrepancy in respect of
stamps, which so far as he was concerned related to the £16,000 regarding coins.
123.The Post Office accepted that there had been an outage but disputed that this either did, or could
have, resulted in the discrepancy which Mr Patny said occurred at his
branch. Ms Van Den Bogerd gave evidence about this, and the Post Office’s position is best
summarised in the explanation put to Mr Patny Senior in his crossexamination:
“Q. Well, what I'm suggesting is that the most natural explanation for this, whatever adjustments were
made, is that at some point on 11 May someone hadn't counted a big pile of £10 notes. They had been
put in a safe and forgotten about, which is understandable, and that they were found the next day, or
located, and there was an accurate cash declaration on 12 May. A. I don't know about that, sir.
Q. It's perfectly plausible, isn't it?
A.l can't say anything to that.”
124.This suggestion by the Post Office is, in my judgment, somewhat fanciful. A “big pile of £10 notes”
is not the sort of thing that ordinarily gets “put in a safe and forgotten about”, and then suddenly
“found”. It was also put to the witness that “things were pretty chaotic in your branch when it comes
to these sorts of things” and “the cash declarations look like they are all over the place”. Mr Patny
Senior did not agree with this characterisation of the way his branch was run. He certainly did not
agree with the “big pile of £10 notes” theory the Post Office put forward. His evidence was also that it
was his son who had contacted the helpline, and who had made the adjustments to Horizon that he
was advised to do by the helpline. Initially the Post Office sought to challenge this account through
cross-examination of Mr Patny Senior only. Counsel for the Post Office did not initially intend to put
questions on this to his son, even though his son was plainly the correct witness to answer the
detailed questions, as he was the person who said he had done certain things, and he was about to be
called as a witness. The reluctance of the Post Office to put questions to the person who was plainly
the relevant person to ask about this was said to be because “the records are the records” and the
Post Office did not intend to repeat cross-examination that had already been done. I was not prepared
to permit this. I required questions about what Mr Patny had, or had not, done, to be put to him
directly, and not have the crossexamination conducted by proxy through his father. Mr Patny Senior’s
son had given a witness statement in respect of these matters, and was plainly the correct witness to
whom the questions should be put. He was the next witness listed on the trial timetable, and was
going to be called next for the claimants. The notion that questions challenging what he had done, or
not done, should more properly be directed to his father is contrary to how cross-examination should
be conducted.
Mr Aakash Patny
125.Mr Aakash Patny is the son of Mr Patny Senior, and worked in the branch and retail premises with
him, as did Mrs Patny, the wife of Mr Patny Senior and mother of Mr Patny. He is not a claimant. He
was given the warning against self-incrimination under the Civil Evidence Act. He did however answer
all the questions put to him.
RLITO000005
RLITO000005
126.Mr Patny assisted his father, particularly with balancing. He would usually arrive at lunch time,
and so was not present when the national outage occurred on the morning of 9 May 2016, which had
been resolved by the time he arrived that date. He gave evidence concerning the same amount of
money, which both he and his father considered a shortfall, and which the Post Office had claimed was
due to it and was a
branch discrepancy for which the branch was responsible. It had been settled centrally but had not
been paid to the Post Office.
127.So far as 11 May 2016 was concerned, when Mr Aakash Patny had arrived at the branch he
became aware of the shortfall of over £17,000, which was a “complete surprise” to him and which he
considered could not be explained. He had contacted the helpline, and had followed the steps he was
advised to take, which he thought had resolved the issue. These steps were things he did on the
Horizon terminal, and sequences of entries he was talked through over the phone. The documents
available supported his evidence that he had made such a call, although some of the entries did not
match what he said he had told them. There was no record of what he was told to do, nor was there
any record put to him showing what key strokes had been inputted at the branch.
128.The Post Office’s case on the shortfall was that the outage on 9 May 2016 did not lead to any
discrepancy. It was not put to Mr Patny that he had not made key strokes as advised over the
telephone, or that he made different keystrokes to those advised. All that was put, somewhat
unclearly, was that “the problem” was resolved, cash had been physically “found” that had been
missed at the time of the previous cash declaration, and also that a figure showed on a cash
declaration of “plus £17,000 odd” had the effect of “cancelling out” the previous discrepancy and
showed that “there was no longer a cash problem”. Mr Patny denied any money had been found that
had been lost; denied that if a discrepancy had been cancelled out the figure would have been plus
£17,000, rather than zero; and said that although he thought following the key strokes he was told to
input had resolved the issue, it had not. He was told to “readjust” the cash stock figure, something he
had never done before, and relied upon the instructions he was given by the helpline. The Post
Office’s cross-examination was done by reference to the cash management report, not the audit data. I
will return to audit data later in this judgment in Part K, but the audit data would have showed what
keys were pressed, in what sequence and when. Such data would have been of great assistance, but
was not produced by the Post Office.
129.On 19 May 2016 the branch was contacted by the Post Office, in respect of an overdeclaration of
stamps. The amount of this was approximately the same value as the amount of the discrepancy of
cash had been a few days earlier. He had done the declaration for stamps. A person called Debra
Lambley phoned the branch. He was struck by the co-incidence in the amount of the stamp
discrepancy, and told Ms Lambley the stamp declaration was impossible due to the number of stamps
this represented. To assist a reader of this judgment on that latter point, but something that was not
explored in the evidence, to obtain a broad idea of how many stamps that would represent, the
following points are relevant. On 28 March 2016 the Royal Mail announced an increase to take effect
that week in the price of stamps. 1st class stamps became 64p each, and 2nd class stamps became
55p each. This shortfall would therefore have represented 25,000 1st class stamps, or over 29,000
2nd class stamps. It goes without saying that this is a great deal of stamps.
130.He was instructed to re-adjust the stamp stock figure, followed her instructions as to how to do
this, and the system then showed the £16,000 shortage. It was by now past 7.00pm, he phoned the
helpline, they contradicted Ms Lambley and he then had to re-
RLITO000005
RLITO000005
declare the following day, being unable to do so that day as it was after the 7.00pm cut off.
131.The Post Office’s case on this, as it was put to Mr Patny, was firstly that he had made a mistake in
declaring the stamps. He denied this. The circularity of the Post Office’s case can be seen from a
lengthy passage of evidence, which I will not reproduce, from page 26 on that day’s transcript to page
35. Mr Patny explained, basically, that the helpline gave instructions to resolve the cash discrepancy;
this resulted in the figure for stamps being wrong by the same amount; Ms Lambley was not prepared
to have this, and following her instructions this resolved the stamps situation, but the cash
discrepancy reappeared. Then the sequence happened a second time. Mr Patny was prepared to
accept he may have made a mistake once, but denied he could have made exactly the same mistake
the same way a second time.
132.It was also then positively put to him that one way of disguising a shortfall in cash would be to
over-declare the number of stamps. A positive allegation of dishonesty was put to him, which he
rejected.
133.The premise behind this suggestion of dishonesty was that it was said by the Post Office that if
the stamps figure went up, the cash figure would automatically go down by the same amount. He did
not agree with this. It was put as a positive statement in the following terms:
“Q. I suggest finally, Mr Patny, it is rather surprising, given that you do all the balancing, or did all the
balancing in the branch, that you don't understand that the effect of increasing the declaration of
stamps is to have a corresponding effect on the declaration of cash?
A.I wasn't aware of that, no.”
134.There was no direct evidence to support the proposition that by increasing a declaration of
stamps, it would have a corresponding effect in the opposite direction upon the declaration of cash
held in the branch. In any case, in the Common Issues trial, there was evidence that making a cash
declaration required specifying how many notes of each denomination was physically held in the
branch. An SPM had to declare both the cash and the stock that was held in the branch. It is difficult
to see how changing the number of stamps held in a branch would, or could, affect the physical
counting and declaration of the amount of cash held in a branch by way of specific denominations.
However, and in any event, the allegation of dishonesty against Mr Patny was a line of cross-
examination that was not supported by evidence from Ms Van Den Bogerd, who was expressly asked
about it, and who accepted that she had not made an allegation of dishonesty in her witness statement
and said she did not offer an opinion on whether Mr Patny was dishonest or not. That means that
there was no evidence from any witness called for the Post Office to support the allegation of
dishonesty.
135.The second problem in relation to which Mr Patny gave evidence was a MoneyGram transaction
in February 2016. This related to a failed payment by card by a customer, who tried to send £3,100 by
MoneyGram but whose card was declined twice. However, the branch accounts still showed two
debits in that amount in the branch accounts that led to a loss showing at the branch of £6,200. A TC
was issued but only for one of these, which led to a £3,100 loss. Mr Patny accepted that he had only
cancelled the transaction at the time, and not reversed it as well (which was required) until later, in
the evening, when advised to do so by the helpline. As he put it, the amount had “doubled up”, in that
the MoneyGram transaction, which was cancelled and reversed later on (which was for £3,100) had
led to a loss showing of £6,200. If that is correct, then it is indeed double the value of the transaction.
Ms Van Den Bogerd said that this was “not supported by the data” and also relied upon the fact that
RLITO000005
RLITO000005
the cash shortfall on the day was greater than £6,200, namely about £500 greater than the cash
shortfall which Mr Patny said had been caused by MoneyGram. She could not proffer an alternative
explanation, however, and in her witness statement said:
“On 23 February, the branch declared cash holdings of £25,803.87, a net downward movement of
£8,601.59. The net value of transactions during this period resulted in a £1,806.71 decrease in cash.
The Moneygram transaction described above would account for a further £3,100, bringing the total
explainable cash movement to a £4,906.71 decrease. However, this leaves £3,694.88 of cash
movement unaccounted for. Lcannot say for certain what caused this additional loss of cash but there
is nothing in the accounts that suggest a problem with Horizon. It appears more likely to me to be a
problem with cash handling in the branch or a user error when making cash declarations.”
(emphasis added)
136.This statement does not answer the evidence of Mr Patny on the point, and the more that counsel
for the Post Office tried to press the point(s) in cross-examination, the more stark it became that this
evidence by the Post Office was no answer. Firstly, it assumes that the MoneyGram transaction was
only responsible for £3,100, not the figure of £6,200 which Mr Patny stated. It assumes that £3,694.88
does not include the lower figure of £3,100, which in my judgment, on the documents deployed, it
plainly does. Secondly, Ms Van Den Bogerd cannot say what caused the additional loss of cash.
Thirdly, it states that that there is “nothing in the accounts that suggest a problem with Horizon”
(emphasis added). That is rather stating the obvious. If all the problems complained of by the
claimants with Horizon could be seen in the branch accounts, this litigation would not be occurring.
Fourthly, she retreats to criticism of the way cash is handled in the branch, or other mistakes - “user
error”. It is a perfect summary of the Post Office’s case, but it does not provide an answer. Other than
vague “user error”, there is no explanation provided. I find that there is nothing in any of the
documents to suggest user error.
137.Additionally, Mr Patny explained that he had fully explained all of this to the area manager Mr
Irwin, who agreed with him that the MoneyGram transaction had appeared to “double up”. Further
disclosure by the Post Office when the trial of Mr Patny Senior’s claim comes to be tried, together
with evidence from Mr Irwin if he is called, will provide further information. Finally, the exercise
undertaken by the Post Office in cross-examination failed to take account of the fact that one had to
consider both the cash figure, and the shortfall, from day to day to make sure that one is comparing
apples with apples. I consider the exercise undertaken in crossexamination to be flawed.
138.1 cannot resolve this purely on the factual material put before the court by the Post Office. The
audit data showing the actual key strokes inputted will be required in order to do that, together with
findings on the expert evidence and my findings on the existence of bugs, defects and errors within
Horizon. However, I do find that Mr Patny came across as a careful person, and as a credible witness,
and I accept that he took the steps he did as explained in his evidence. He accepted that he had not
reversed the transaction until later, rather than immediately as he ought to have done. The resolution
of the movement of the cash position, and what if any explanation is available, simply cannot be done
with finality at this stage on the basis of the factual evidence before the court alone. The Post Office
had, according to Mr Patny, failed to provide the Credence data which would make it clear what had
happened and how. The helpline logs show that Mr Patny had chased this on numerous occasions, and
also the entries suggested that a Credence report was being prepared. The audit data showing what
key strokes he had undertaken and when, was not produced by the Post Office.
RLITO000005
RLITO000005
139.There was no evidence before the court to support the allegation of dishonesty that was expressly
made against him, and it was not supported by Ms Van Den Bogerd. On the material before the court
in this trial, I find the allegation of dishonesty against him not to be made out and I reject it. That is
not to say that the subsequent trial dealing with Mr Patny Senior’s claims will automatically accept all
the evidence given by his son, if he calls him as a witness. All future issues in individual claims will be
dealt with, and resolved, on the evidence led by the parties in those subsequent trials. I am only
deciding matters in this judgment that are necessary to resolve the Horizon Issues. However, in
circumstances where positive allegations of criminality are expressly put to a witness in a trial such as
this, it is only fair to deal with those allegations. They cannot as a matter of fairness be left hanging in
mid-air, particularly where there is no evidence provided in support from the party making the
allegation in any event.
140.The later proceedings concerning Mr Patny Senior’s claims will explore in some detail not only
the specific matters in respect of which he and his son gave evidence in the Horizon Issues trial, but
any others which are relevant to both his claim and counterclaim. They will do so by reference to the
evidence advanced in that later trial and the other documents, as explained in the section of this
judgment Part K, Audit Data. The degree to which the evidence of Mr Patny Senior and his son affects
my conclusions on the expert evidence will be dealt with in Part L, Overall Conclusions.
Mrs Burke
141.Mrs Angela Burke had worked for the Post Office in various roles since she was 16. Most recently,
she worked in the Newport Post Office in Brough where her husband was the SPM. She was a Branch
Assistant and had worked as an assistant for about 15 years, and had also previously been a SPM
herself. She and her husband had an associated greeting cards and stationery business, and the
branch was closed in October 2017 as part of Network Transformation. The card business has also
closed.
142.Her evidence related to events of 9 May 2016 when the national outage occurred, the same date
as the occasion about which both the Mr Patnys had given evidence. She gave evidence about the
impact upon the branch business, the way she was serving customers and how Horizon was being
very slow that day, with a sand timer appearing on the screen for a very long time. She served one
customer, who was making a cash withdrawal, as she obtained the relevant messages and approvals
on the screen. However, after they had left, a receipt printed saying “Recovery failed” and the
withdrawal of £150 was not shown. She then later studied the transaction log and this latter
transaction did not appear.
143.She realised what was happening and decided to close her branch that morning. She sat down
and worked through all the transactions in her branch, using what was available to her. She was given
an explanation by the helpline which she did not accept, as she knew and could remember that the
payment to the customer had been authorised, which is what led to her handing over that sum. She
also was engaged in a contemporaneous exercise whilst it was very fresh in her memory. She said
there was no alert from Horizon, and “there was no means through the Horizon system for the
discrepancy to be identified or for its cause to be established in my situation”. The transcript of this
call was available - she and her husband had obtained it using a subject access request from the Post
Office.
144,Mrs Burke then went to extraordinary lengths. She also proved herself very tenacious, as many
people may well have simply given up on the sum of £150. She identified the customer, and she
tracked him down. She went to his house and explained what had occurred. He happened still to have
RLITO000005
RLITO000005
the receipt from the transaction at her Post Office. It entirely matches her account. She went with the
customer to the customer’s bank, which was the TSB in Goole. She explained with the customer to the
bank cashier what had happened, and the cashier printed out the bank statement and showed that the
sum had been withdrawn from the customer's bank account. The customer permitted Mrs Burke to
have this.
145.She and her husband then pursued this through the helpline and their call was escalated. They
did receive a TC for £150, but actually that referred to an amount in respect of Lloyds Bank and not to
TSB. This was pointed out to the Post Office who claimed there was no code on Horizon for the TSB. If
that is correct, I find it highly surprising that there should be no separate code for the TSB. The bank
currently known as Lloyds used to be known as Lloyds TSB - there are cases on bailii that use that
name in relation to other litigation, for example - but the TSB was split from Lloyds TSB, which then
became simply Lloyds Bank. This split occurred some years before the events of which Mrs Burke
gave evidence. TSB and Lloyds Bank had been separate for some time by May 2016.
146.Mrs Burke’s evidence was explored in very considerable detail in cross-examination, including the
fact that she chose to keep the stack open in order to serve more than one customer at once, how she
had served them and the other details of what had occurred. She was challenged on what was said to
be a “relatively small” point, of whether she would have received a TC from the Post Office without
having done all the work that she had done, in tracking down the customer and going to his bank. I
reject that categorisation of the point by the Post Office. It is not a small point. It is, in the context of
the overall litigation, a major point. It is difficult to see, given the evidence collected by Mrs Burke
which was provided to the Post Office, how they could have failed to issue her with a TC. Whether this
TC would have issued absent the work that she did to demonstrate the fault and its effect is not a
“small point”.
147.There are really two points that are relevant. The first is a broader one: is the process for issuing
TCs part of the Horizon system? The second is a case specific one: would, on the balance of
probabilities, the Post Office have issued a TC in her case without the evidence that Mrs Burke herself
collected and provided to the Post Office?
148.So far as the first point is concerned, the process for issuing TCs is not part of the Horizon
system. That is addressed elsewhere in the judgment. So far as the second point is concerned, the
documents put to Mrs Burke demonstrate that Fujitsu had worked out what had occurred; that the
internal document attached to an internal email of 12 May 2016 did not accept that Newport branch
had in fact paid the money to the customer, even though Mrs Burke had told them this on 9 May; and
that by 16 May 2016 the decision had been taken that the branch would receive a TC anyway. Her
response to the “small point” was “possibly, yes”.
149.The conclusion that I draw from this is that the Post Office required something more from Mrs
Burke than her word that the money had been paid out by her branch to a customer - the internal
documents refer to “not knowing” if the money had been paid, even though she had already told the
Post Office this. I find that the fact that this is a Fujitsu document rather than a Post Office document
does not matter for this purpose. That something more that was required, in her case, was the
evidence she had personally compiled that was of great weight, and was accepted by the Post Office,
as the TC was issued. Whether evidence of lesser weight would have been accepted by the Post Office
is not possible to say, and is entirely hypothetical. Fujitsu had, though, identified this particular
problem experienced at the Newport branch by the national outage. The information of what
occurred, and the problem, was not visible to the SPM or their assistants in the branch.
RLITO000005
RLITO000005
150.Counsel for the Post Office’s “relatively small point” put to Mrs Burke, that the Post Office would
have issued her with a TC anyway, is in fact not entirely consistent with the Post Office’s own evidence
for this trial. Mrs Van Den Bogerd stated in paragraph 110 of her witness statement that “Once Post
Office was presented with evidence that the customer had received the cash and the customer's bank
had recorded the withdrawal, a transaction correction was issued to bring the branch.” She makes it
clear, in my judgment, that it was the work that Mrs Burke did personally that had led to the issuing of
the TC. She also makes it clear that she believes that what happened occurred as a result of what Mrs
Burke did:
“Following Mrs Burke's investigation, Post Office generated a transaction correction for the £150
withdrawal that had not been recovered at the time. If Mrs Burke had not done this herself, there is a
process built into Horizon for flagging non-recovered transactions which would have prompted an
investigation and I'm sure would have led to the same outcome.”
151.She explains that had this not been done, more would have been required from the Post Office -
namely an investigation - before the Post Office would have reached the decision to refund the £150,
although she does say that she is sure the same outcome would have been reached. This is rather
different to the way the case was put to Mrs Burke, which did not include the requirement for any
such further investigation by the
Post Office being required. The impression sought to be given was that on the Fujitsu documents
already available, a TC would have been issued anyway to Mrs Burke a few days later. I do not accept
that. Something more at the Post Office end would have been plainly required, as Ms Van Den Bogerd
explains. Mrs Burke was the person who had in fact provided proof of payment out, and so no
investigation was undertaken.
152.It is in my judgment likely, on the basis of all the evidence in the litigation to date, that the sum of
£150 would have showed up as a shortfall at the end of the next branch trading period, as any further
investigation was not likely to have been completed by the end of that period. However, if Mrs Burke
had not acted as she had, and if an investigation had been done by the Post Office, if that had been
resolved in her favour, and if that had led to a TC being issued, then the shortfall in the branch
accounts would have been corrected in a later branch trading period, namely the period during which
the TC was issued. No loss flowed to Mrs Burke in respect of this incident, as a result of her own
investigation. In any event, this is not a point of difference between the parties that it is necessary to
resolve in order to resolve the Horizon Issues. There was plainly a potential impact to her branch
accounts in any event, and that potential impact was caused by the Horizon system.
Mr Roll
153.Mr Richard Roll had worked at Fujitsu between 2001 and 2004. This was during the days of
Legacy Horizon and he had no experience of Horizon Online. He provided two witness statements, the
first of which was dated 11 July 2016. This was obviously very much in the early days of the group
litigation. The second statement addressed certain factual matters in Dr Worden’s first expert report.
I consider Mr Roll to be an important factual witness in this group litigation. He has no personal
interest in the litigation and is not a claimant. He has never worked for the Post Office, although
whilst at Fujitsu working on Legacy Horizon he obviously had a great amount of involvement in the
Horizon system. However, given the expert evidence and particularly the degree of agreement
between Mr Coyne and Dr Worden on the number of accepted bugs in Horizon, which even on Dr
Worden’s position was 11 different ones, in addition to Mr Roll’s evidence there was ample other
evidence in relation to my findings on the Horizon Issues. Mr Roll’s career has started in the Royal Air
RLITO000005
RLITO000005
Force, which he joined in 1976 and left in 1989. Whilst in the RAF, which he had joined as an avionics
engineer, he worked on mainframe computer systems and was selected for a software development
team working on aircraft control and attack systems. After he left the RAF, he worked in various roles
in development and support, and joined Fujitsu in January 2001. There, he worked in the Software
Support Centre, or SSC, in Fujitsu at Bracknell in 3rd line support.
154.After he left Fujitsu he changed career direction entirely and attended the University of
Southampton, from where he obtained a BSc in Podiatry. He worked in the NHS until 2011 and then
went into private practice, where he remains, now with his own clinic. The claimants described Mr
Roll as a “whistleblower”, effectively making public allegations about how Fujitsu dealt with Legacy
Horizon and the access which was available to Fujitsu to SPMs’ branch accounts. This was access of a
far wider nature than admitted to by the Post Office or Fujitsu publicly.
155.It was Mr Roll’s evidence that was substantially behind the eventual acceptance by Fujitsu
witnesses of the ability to obtain remote access without SPM knowledge or permission, and the
injection of messages into the counter. In particular Mr Parker had, in his first statement, described
that Fujitsu lacked this “power” and had said that Mr Roll’s evidence was inaccurate and misleading.
Mr Parker later corrected his 1st statement, and I deal with his explanation about this change - and
what he said he meant in his 1st statement - in my summary of Mr Parker’s evidence below at [464] to
[498].
156.Mr Roll described himself as an IT specialist, a description with which I agree, and one which is
demonstrably correct given the number of years he performed his role at Fujitsu in the SSC in 3rd line
support. Problems or issues would only reach 3rd line support if 1st or 2nd line support did not
resolve them. His IT experience includes, in my judgment, a considerable degree of expertise in
software. Indeed, although I am critical of Fujitsu elsewhere in this judgment, that criticism does not
extend to any suggestion that they employed personnel within SSC who were not experienced and
sufficiently expert in software matters to perform that role. Had Mr Roll not been sufficiently qualified
to work in 3rd line support, he would not have been there for the period that he was.
157.The Post Office in cross-examination attacked both his recollection, and seemed also to challenge
his expertise. This the Post Office is entitled to do. Given he was 3rd line support, in my judgment he
obviously had a high degree of expertise. There was a debate at one stage about whether he was
“elite” or “super elite”, and he put himself in the former category. The Post Office also concentrated
on a narrower band, or subset, of the issues regarding which he gave evidence, focussing more on
potential problems with code and Mr Roll’s experience of those, rather than the wider issues generally
in respect of Horizon of which he gave evidence. He had made clear very early in his cross-
examination that, so far as he was concerned, data corruption was an issue in Legacy Horizon as well
as software issues which were related to the code.
158.Mr Roll had left Fujitsu in 2004 and there will inevitably be difficulties in recollection by any
witness of detailed points after such a passage of time. He was willing to indicate when he could not
remember something, and would agree with points put to him by the Post Office when relevant. These
were often on the basis of hypotheses which he was asked to accept, often in relation to a spreadsheet
which Mr Parker had prepared showing how relatively few software issues Mr Roll was said to have
worked upon. Mr Roll also, again inevitably, was not submerged in the fine details of the subject
matter of the litigation, in the same way as some of the Fujitsu witnesses such as, for example, Mr
Parker himself, who is the Head of Post Office Application Support and the SSC, and who has worked
RLITO000005
RLITO000005
on the Post Office account at Fujitsu since 1997, before Horizon was even introduced. Given Mr Roll
left Fujitsu in 2004, it is 15 years since he worked on Horizon.
159.His two statements were fairly short (4 and 8 pages respectively) and given at a high level. One
example can be provided from his cross-examination. He was of the view that problems referred to
3rd line support, which by definition would mean that 1st and 2nd line support had not been able to
deal with them, would not include problems caused by mistakes by SPMs. Mr Parker disagreed with
this and this was put to Mr Roll.
“Q: Then the next unnumbered paragraph, Mr Parker says:
“If NBSC were unable to identify the cause of a discrepancy they would often fall back on a default
statement along the lines of ‘this looks like a software issue' so that the SSC would investigate it.
However, Mr Roll's statement that 'if an error was referred to us then it was extremely unlikely to be
due to a mistake made by a postmaster’ is not correct. The vast majority of discrepancies investigated
by the SSC as pseudo ‘software issues' were (and are) not caused by software issues."
I would like to suggest to you, Mr Roll, that that's true, isn't it?
A.The way that I remember it, it was issues to do with the software that were causing the problems.
Whether that was the programme that had been written or data corruption, that's what I remember as
our problems being.
Q. You remember there being problems with data corruption?
A. Yes.
Q. I'm not asking you about data corruption, Mr Roll, I'm asking you about software issues. The claim
I would like to put to you again is the last sentence of that paragraph:
“The vast majority of discrepancies investigated by the SSC as pseudo ‘software issues' were (and are)
not caused by software issues." Are you in a position to agree to that?
A. From my recollection I would disagree with that, but it was a long time ago. Q. I'm grateful.”
160.This passage of cross-examination, and there are a number of other similar examples, shows the
following:
1.Mr Roll’s conclusion was a general one, with which Mr Parker disagreed. Assertion and counter-
assertion are not, of themselves, of assistance to resolving the Horizon Issues. His conclusion, as with
that of Mr Parker, is simply their point of view.
2.If a problem made it to 3rd line support, then by definition it had not been capable of being resolved
by 1st and 2nd line support.
3.Fujitsu would, even at the 3rd line support level, attribute some problems which they could not
understand or resolve to SPM error, and Mr Parker's evidence is consistent with this approach by
Fujitsu. This is made clear in the text of PEAKs put to Mr Parker. There is a very stark example in one
of the PEAKs where the Post Office’s own auditors have specifically ruled out user error, yet Anne
Chambers chose to close the PEAK with the defect cause being user error.
4.The Post Office would try to keep the evidence in cross-examination of Mr Roll within very narrow
bounds. This passage shows that Mr Roll expressly gave data corruption as one of the problems of
RLITO000005
RLITO000005
which he had experience. Horizon Issue 1 included the expression “bugs, errors and defects”. In my
judgment data corruption is included within that phrase. Mr Roll was kept very close in his cross-
examination to evidence concerning code alone. Bugs, errors and defects are far wider than simply
code. The Horizon Issues were not specifically restricted to software issues as narrowly re-defined by
the Post Office, nor are they restricted to code.
5.Mr Roll would sensibly concede that the events of which he gave evidence were a long time ago.
6.As shown by the answer “the way that I remember it, it was issues to do with the software that were
causing the problems”, even at this remove of time, Mr Roll had a specific recollection that there were
“issues with the software” that were “causing the problems”. This is as long ago as 2001 to 2004.
161.Some of the cross-examination was of no assistance in resolving the Horizon Issues, for example
an exercise with a spreadsheet which Mr Parker had been involved in preparing, showing the
percentage of calls to 3rd line support and the broad categories into which they had been placed (by
Mr Parker, as it turned out, and inaccurately at that). Mr Roll was in no position to give any helpful
evidence in respect of this, and the categories were in any event misleading as headline descriptive
terms. A witness who had left Fujitsu’s employment some 15 years ago would not, sensibly, be in a
position to agree or disagree with such detailed points or collation of data in such a detailed
spreadsheet in any way that would provide the court with useful evidence, particularly when it
emerged (as it later did when Mr Parker was cross-examined) how misleading the Fujitsu headings
were.
162.Mr Roll had looked at Mr Parker’s statements, although he said only briefly, and he did not know
how many statements Mr Parker had made. He could not remember some of the details, such as the
detailed descriptions of 1st and 2nd line support at SSC, and was frank about this. He could not recall
the split between 1st line support by Fujitsu and the Post Office’s own 1st line help desk called the
National Business Support Centre (or NBSC). He acknowledged that he worked in 3rd line support,
rather than “third/fourth line support” as he had put it in his witness statement. He did not however
over-state his abilities; for example he said some members of 3rd line support were “super elite”, but
he would not count himself as falling within that description. Part of his cross-examination which I
consider fairly encompasses his approach to giving evidence is the following, where parts of Mr
Parker’s witness statements were being put to him.
“Q. And if we can pick it up at paragraph 28 of Mr Parker's witness statement, you worked in third
line for between -- well:
“Between 1 January 2001 and 31 December 2014 ...” and those are the years that you worked for
Fujitsu "... the SSC received a total of 27,005 calls, meaning that on average 563 calls per month were
dealt with over this 4-year period."
And he refers to a spreadsheet setting that out and he then analyses the data in that spreadsheet.
Would you accept that that is a fair reflection of the amount of calls coming in, the amount of
incidents coming in to SSC when you were there, third line?
A. can't really remember. I know there were periods when it was very busy and periods when it
wasn't so busy. Sometimes we had three or four jobs on the go at once, other times we were given
other work to do from the manager. Q. And paragraph 29, Mr Parker says:
“Transferred calls (ie those not resolved by the SSC) are of interest." He says:
RLITO000005
RLITO000005
“A very small proportion of calls transferred to 4th line support would have concerned software errors
requiring resolution ..." Stopping there, Mr Roll, that's true, isn't it? A. Yes.
Q. So he then says:
".., it would be interesting [therefore] to know the number of calls transferred to fourth line."
Would you agree to that? It would give some indication of the extent to which incidents coming into
the SSC properly, genuinely represented software areas that required fixing? That would be a useful
way of -- a touchstone of trying to work out -?
A. Yes.
Q. Thank you. He then says that, unfortunately:
"... while the SSC have records of the volume of transferred calls, we do not retain records of where
they are transferred to and it is not the case that all of these would have been transferred to 4th line
support. For example incidents would often arrive at SSC from internal teams for routing back to help
desks." Do you remember that?
A, I don't remember that particular ...
Q. But would it be right to say that of the calls coming into third line support, a significant proportion
would go -- of calls that would then be transferred out, would go to places other than fourth line
support? Would that be fair? Does that accord with your recollection?
A. The way I recollect it is that calls would come in and we would work on them, either fix them, in
which case they would go back to the originator, or we would pass them on.
Q. To other people?
A. To other people.
Q. Depending on the nature of the problem?
A. Yes.
Q. So if there was an infrastructure problem you would probably pass the call on to the infrastructure
team, it wouldn't go to fourth line support, would it? A. Probably.
Q. And there are a number of other teams that calls coming into third line would be passed to who
would not be fourth line support?
A. Probably. I can't remember the full details.”
163.This can be seen all to be at a very high level. Mr Roll is not likely to have known the total
number of calls that came into SSC during a 4 year (or indeed any) period; the average number of
calls per month; or of how these were dealt with. He would also not be likely to be able to recall these
details, even had he known them at the time. The analysis which was put to him would have been
“useful to know” - the percentage, or even total number, of calls transferred from 3rd line support to
4th line support - would not have been something he would have known at the time either.
Interestingly, not only would such information have been something that Fujitsu could have readily
recorded at the time, as both 3rd and 4th line support at SSC are both Fujitsu functions, and not only
was this plainly accepted as something that would be of assistance, but Fujitsu “did not retain records
RLITO000005
RLITO000005
of where they are transferred to”. Thus the absence of a useful record in this respect is a result of how
Fujitsu retained its records.
164.Mr Roll was then taken to an analysis of calls coming in, and calls going out, which had been done
by Mr Parker. This related to some 3,764 calls. Mr Roll was never going to be in a position to dispute
that figure; to dispute or agree the percentages given for how the calls were dealt with; or provide
any meaningful evidence about Mr Parker's analysis at all. It was presented to him for the first time in
the witness box. Sometimes unreasonable or unhelpful witnesses take issue with matters of which
they have no knowledge; Mr Roll did not do so. If he did not know, or was not in a position to answer
substantively, he would simply say so.
165.Another passage of his evidence was as follows, when a passage of Mr Parker’s evidence was put
to him that stated "From the SSC, only a tiny proportion of incidents were escalated to the 4th line
support team. It follows that only a tiny fraction of incidents raised actually needed to be looked at by
the only team who might potentially effect changes in software." He was asked if he accepted the
conclusion:
“A. When you take it as an average then yes I suppose so, but the system evolved from -- I started
there in 2000 I think it was and left in August 2004. The nature of the work changed over time and the
ones that stick in my mind are the ones where there was sort of the fire-fighting efforts where it was ~
there were difficult periods where there were software issues, so ...
Q. Yes, Mr Roll. We could by all means go back to the graphs because you will have seen from the live
PEAKs into and out of the SSC it actually gave monthly figures, but what I'm suggesting to you, Mr
Roll, is that ~ although you may well have been busy on all sorts of things, the fact of the matter is
that software problems requiring a software fix represented a tiny fraction of the work that was
handled by the SSC third line support.
A.Yes.”
166.This theme of the way that the Post Office put its case carried on into the evidence of Dr Worden
and his Section 8 statistical exercise, which I address below. The expressions “tiny proportion”, “tiny
fraction” and other terms used sought to demonstrate, basically, that Horizon worked very well most,
if not all, of the time. Given the use to which the Horizon system was put by the Post Office, namely
monthly balances which governed the branch accounts of SPMs, with shortfalls and discrepancies
being “made good” - which means paid - by the SPMs in question, this approach can be seen to have
limited utility. Firstly, these are subjective terms. Secondly, given the PEAKs show that a SPM may
report a single unexplained discrepancy for (say) £25,000, or even £1,000 or £500, the fact that this
may be only a tiny fraction of the number of calls the SSC worked on that month might come as cold
comfort to that specific SPM. The number of calls, and how they were dealt with, is of some relevance
to robustness, but such high level and subjective points were not likely to be of enormous assistance,
one way or the other, in challenging Mr Roll’s evidence.
167.It was put to Mr Roll that his role, which he described as “product specialist”, was “the junior
level of people working” in SSC, which he disputed. He explained that most people in SSC were
“product specialists”. 25 of the 30 or so total number of people were, according to Mr Roll, all product
specialists.
"Q. -- the SSC? Of that 30 people how many people were at your level?
RLITO000005
RLITO000005
AWhen I worked there I -- it was two or three people were senior levels, I think Mr Parker was one, a
couple of others, then there was Mike Peach and I believe the rest of us -- 25 or so.
Q. The figures I would like to ask suggest to you, Mr Roll, are about 25 people had the junior level and
there were about five people who were true specialists who I think you fairly described as specialist
earlier on in your evidence. Would you accept that that might be the case?
A. Junior level seems to -- it doesn't fully explain the complexity of the system or the knowledge of the
system required, but yes, I suppose ...
Q. Well, let's agree on ordinary ~- I'm not seeking to cast any imputations and it is right that you
should - words do have implications. So the ordinary level was your level - A. Base level.
Q. -- and there was this perhaps five or so people above -- the senior people above you and what did
they do? Did they do the more challenging work?”
An interesting post script to this line of questioning is that later in the trial it emerged that all of the
SSC personnel had the very powerful APPSUP privileges at this time. It is highly unlikely that this was
given to those at the “junior level”.
168.The Post Office attempted at the trial to dilute Mr Roll’s experience and cast him as someone who
was not sufficiently skilful to understand, or give evidence about, the matters he explained in his
witness statement. It is clear to me, and this was effectively confirmed by the documents put to Mr
Parker later in the trial, that everyone at 3rd level support in SSC was a specialist. The notion that
these 30 people, in a department separate from the normal Fujitsu work areas, to which access was
security restricted, were all at “the junior level” or “ordinary” with 5 people managing them who were
“the true specialists” is simply misleading. There were a handful of people at the senior level, and the
rest of 3rd line support were product specialists. A Fujitsu internal document, put to Mr Parker by the
claimants (but not to Mr Roll by the Post Office) explained it rather more comprehensively. This is
from a 2011 document but there was no evidence to suggest that there was a wholesale
reorganisation of SSC 3rd and 4th line support between Mr Roll’s time there and this document.
“3rd line support
3rd line support groups within RMGA include:
SSC - 3rd line support for RGMA written application code.
MSS - 3rd line support for software distribution and event management
3rd_line support staff apply analytical skills to the symptoms and evidence gathered by 1st and 2nd
line and undertake in-depth investigation into incidents. They have detailed knowledge of the system
; .
Trai - CoTs z sats .
used within the application. They are also expected to self train by examination of support guides,
design documentation written for the components of the end user application. They will also have
diagnostic code. scripts and support tools.
Itis incumbent upon the 3rd line support unit to produce a work around and on 4th line to produce
the final code solution to any software problem. This does not preclude the production of a
RLITO000005
RLITO000005
workaround by other units or negate the requirement for 4th line to provide assistance in the
generation of a workaround.
The SSC ible f . ion of .
(emphasis added)
169.This gives a more comprehensive picture of the type of expertise required to be in 3rd line
support, which is where Mr Roll worked whilst he was at Fujitsu. This is of more assistance than the
terms deployed, such as “the five true specialists” (for the senior managers) and “the junior level” for
Mr Roll. It shows that 3rd level support have a high level of necessary expertise. They are the only
unit with the necessary authorisation and security controls to be permitted to implement workarounds
that require data changes to the live system. This is not the role of junior personnel.
170.This area of questioning also focused on code, and introduced another concept, the “core
specialist team”.
“Q. I'm really trying to grope towards what the five true specialists, what kind of work they were
doing. They were doing the more challenging kind of work, weren't they?
A. I find it difficult to answer that from what I remember of the way we worked. There were some
areas where some of the senior people where I would perhaps have been more -- had more experience
because of previous work and previous programming that I would have been better off and had more
knowledge than they would have done, but in other areas then they would be far superior to me.
Q. Well, let me suggest one area where they would have been far superior to you, Mr Roll, or at least
it was perceived within the organisation -- I'm not trying to have a debate with you about your own
perceptions of yourself, that would be completely unfair, but the perception at the SSC was that were
there software errors or potential software errors that required large amounts of code to be
examined, the people who would generally do that examination would be those five people, people like
Mr Parker. Would you accept that?
A. Generally I suppose, yes, although -
Q. And it would be relatively rare for someone outside that core specialist team to be doing that kind
of work -- not impossible but relatively rare, yes?
A, Several of us looked at the code on occasions. I was a C programmer in that and other languages
before I moved to Fujitsu so there were areas of knowledge we had from previous areas, but I suppose
most of it would then be passed on to those people.
Q. see. So would you accept from me then that generally it would be them that would look at lots of
code but occasionally there might be occasions when someone else would look at lots of code?
A. From a PinICL perspective yes, but some of us looked at code more often, just out of interest.”
171.This series of questions introduced another subjective variable, “large amounts of code”. Mr Roll,
in his answer above, explained that he had been a C programmer before he moved to Fujitsu. C is a
general purpose programming language which is widely used. Mr Roll, as someone who had been a
programmer both in C and other languages before he went to Fujitsu, had both written code himself
(as a C programmer), and I find would have been very familiar with code in its different guises.
RLITO000005
RLITO000005
Indeed, given the Fujitsu documentation describing the expertise of 3rd line support in [168] above,
which stated 3rd line support would have “detailed knowledge of the system based on documentation
and source code inspection”, it is clear to me, and I find, that Mr Roll was amply experienced to give
evidence to this court on the matters that he did concerning the Horizon system (which was at that
point Legacy Horizon), the problems that were experienced with it, and what occurred at SSC. Source
code is a term used to describe a version of software, or a description, as it is originally written. There
are different definitions of source code in different computer authorities (such as the Linux
Information Project) but the precise definition is not important for the purposes of the Horizon Issues
trial. The Horizon Issues do not concentrate solely on problems either with code, or source code. The
phrase is “bugs, errors and defects” and the Horizon System is defined in the Horizon Issues.
172.What is important, in my judgment, is that Mr Roll used to programme in C programme language
and other languages before he went to Fujitsu; he was required to understand and have “detailed
knowledge of the system based on documentation and source code inspection” by virtue of being a
member of 3rd line support at SSC; all the evidence demonstrates that he did his job effectively and
competently; and he did not leave Fujitsu under any sort of a cloud. It was effectively accepted by the
Post Office that he had a responsible position and a responsible role, and was well trained. The Post
Office also sought to draw the following conclusion from Mr Roll’s evidence in favour of Fujitsu:
namely that “....a clear picture emerged of Fujitsu as an organisation which was thorough,
professional and conscientious and which took considerable care to ensure that matters were properly
investigated and dealt with.” I do not accept that this picture emerged from Mr Roll’s evidence, but
there was a considerable amount of other evidence in respect of Fujitsu in the trial other than that of
Mr Roll. I will return to this claim by the Post Office at the end of my review of all the evidence,
factual and expert, which is in Part L “Overall Conclusions”. That section of the judgment begins at
[925] below.
173.Mr Roll’s evidence was that errors made by SPMs were relatively easy to pick up at Ist and 2nd
line support level; that most errors he dealt with were coding errors or data corruption; that issues
were identified that required software “fixes” to be written by Development; and - importantly in my
judgment - that the type of issues that were routinely encountered at SSC could and did cause
financial discrepancies to branch accounts. He also stated that:
“If we were unable to find the cause of the discrepancy then this was reported up the chain and it was
assumed that the postmaster was to blame.”
174.He said that even if software fixes were developed the problem would sometimes reappear
several weeks later. He also stated that remote access to branch systems was possible; the ability was
extensive; that this was done without the SPMs being aware; that data and transaction information
could be changed by Fujitsu; and that sometimes SSC would log into a branch system whilst it was
switched on but not in use.
175.The final paragraph of his 1st statement said:
“In summary, the issues with coding in the Horizon system were extensive. Furthermore, the coding
issues impacted on transaction data and caused financial discrepancies on the Horizon system at
Branch level. It was those issues that I, and other colleagues at Fujitsu, were routinely working on
daily. Furthermore, remote access to the Horizon system at Branch level was extensive, as was the
ability to change data and change transaction information, even while the postmaster was working,
without the postmaster being aware of this.”
RLITO000005
RLITO000005
176.He had also said that “during the course of resolving the software issues, we would frequently
access a Post Office counter IT system remotely”.
177.His use of “frequently” and “routine” are, in my judgment, subjective, and as explained above in
terms of “tiny fraction”, subjective terms are not entirely helpful. What he meant by this is it was not
unusual for this to occur. It is difficult to judge, at the remove of 15 years from when Mr Roll left
Fujitsu, just how often something that he remembers as frequently or routine in fact occurred. He
sensibly accepted that the more regular or mundane matters in which one is involved tend not to be
remembered, with rather more unusual events sticking in the memory. In terms therefore of how often
this occurred whilst he was there, it is not possible to come to a concluded view. However, I consider
it important that the occasions to which Mr Roll refers were not isolated incidents on his evidence,
nor were they unusual, and that financial discrepancies on the Horizon system at branch level was
something which he recalls working on daily. It is also important that he gave evidence that if Fujitsu
could not track down the cause then it was assumed the SPM was responsible, in other words user
error would be used as a default setting for investigations. This matches the evidence of an enormous
number of PEAKs in the Technical Appendix.
178.He also gave evidence about hardware failures, and gave specific responsive evidence in respect
of Dr Worden’s expert report.
179.1 accept Mr Roll’s evidence, which is supported by the contemporaneous documents. The Post
Office in its closing submissions accepted that he was a careful and precise witness, and also that “he
was at pains to assist the court and to give accurate answers”. It also, however, submitted that his
evidence “was, unsurprisingly, hazy in many respects”.
180.Notwithstanding the limitations on his evidence due to the passage of time, I found Mr Roll to be
a reliable and helpful witness. I do not consider it was hazy in any important respects. I also found his
evidence to be very important. The Post Office set out in cross-examination to demonstrate that he did
not have the expertise that he claimed to have; that he was at too low a level in 3rd line support to
have been involved in the matters which he described, certainly so far as software issues were
concerned; and that Mr Parker's evidence in particular about what he had been doing should be
preferred. On those two former points the Post Office failed, in my judgment. The success or failure of
the latter point also, but not exclusively, depended upon the evidence of Mr Parker. After Mr Parker
was cross-examined, it was clear to me that Mr Parker’s exercises that were put to Mr Roll could not
be relied upon to demonstrate what they sought to demonstrate. I prefer and accept the evidence of
Mr Roll to that of Mr Parker and the other Fujitsu witnesses (with the exception of Mr Godeseth) by
some considerable margin. I find that during the years when Mr Roll worked at Fujitsu, 2001 to
December 2004, SSC were not infrequently involved in attempting to remedy unexplained shortfalls
and discrepancies in branch accounts reported by SPMs. If they were unable to find the cause of the
discrepancy then the assumption would be made that it must be the SPM to blame. This is clearly
shown in my analysis in the Technical Appendix particularly on the different entries in the Bug Table,
and my findings on the number of bugs present in the system.
181.There is one area of Mr Roll’s evidence, again a wholly subjective one, to which I do not attribute
any weight, not that I disbelieve him. This was concerning the pressure of work within the SSC and
the fact that members of the SSC were under pressure to “close calls”, which basically means record
them as having been completed. The relevant passages are as follows:
“15. During my time at Fujitsu I know that there were budget pressures and redundancies which
impacted system development and testing. The test team felt they were under enormous pressure to
RLITO000005
RLITO000005
complete the testing within certain timescales which negatively affected the test regime. Meanwhile,
the development team had to balance time spent on fixes, with time spent on developing new features
for Legacy Horizon and time spent developing a new system which I believe later became Horizon
Online.
16. In my first statement, I refer to the pressure that the SSC team and Fujitsu were under generally
due to an awareness of the financial penalties imposed by the service level agreements between Post
Office and Fujitsu (paragraph 12 of my first statement). I believe that although individual penalties
were quite modest, when applied across multiple counters/post offices the cumulative figures involved
were very high, potentially amounting to tens of millions or more. I disagree with Stephen Parker’s
statement that these potential financial penalties were not a factor for the SSC (paragraph 43 of
Stephen Parker’s witness statement) as we were aware of them and often commented on them, e.g.
“That’s saved Fujitsu another £25 million”.”
182.1 do not attach much weight to this as pressure in the workplace is such a subjective matter.
Some members of SSC may have felt under great pressure; others less so. This is a subjective account
of what it was to work at SSC at that time from the point of view of Mr Roll. The type of statement to
which Mr Roll refers in terms of millions of pounds saved does not advance matters one way or the
other. Given the type of problems that 3rd level support was involved in attempting to resolve, there
would have been an inevitable pressure involved in attempting to resolve such issues speedily. Mr Roll
chose to leave Fujitsu, and therefore he left those pressures behind. Putting it at its mildest, it would
be extraordinarily disappointing if PEAKs were closed, attributing fault to a SPM, simply because that
was the easiest and quickest way for a SSC product specialist to keep on top of their workload.
Without separate cross-examination of each product specialist who chose to close PEAKs where prior
entries suggested user error had been ruled out, it is not possible to determine the degree to which
this was their motivation in any individual case. However, it is not relevant. It is no part of the Horizon
Issues to determine why, or whether, individual personnel at SSC acted in a particular way -
budgetary concerns for their employer; exercises of judgement; even laziness; or any of the other
many possibilities. Mr Parker gave evidence that “the possibility of financial penalties was never a
factor for the SSC”, and Mr Roll said that it was. Mr Parker's evidence on this suffers from a
wholesale attribution of this lack of motive to every member of SSC, as indeed does Mr Roll’s, in the
other direction. Everyone is different, and there will have been a
range of different reasons operating on each member of SSC each day. Any sensible business will, in
any event, always have at least part of its attention on financial performance and this is
understandable. The Horizon Issues trial is not an inquiry into how Fujitsu manages its personnel, or
its business. I do question the allocation of Category A, B or C to some of the PEAKs that were used in
the trial, but this is something that was pursued (though not very far) with Mr Parker and I deal with
it there.
183.The way that Dr Worden was simply not prepared to accept Mr Roll’s evidence at face value, and
set out to disprove his factual evidence, something which in my judgment an independent expert
should never do, is dealt with in the section of this judgment where I deal with Dr Worden’s evidence.
The degree to which the evidence of Mr Roll, and his experience of Legacy Horizon between 2001 and
2004, affects my conclusions on the expert evidence will be dealt with in Part L, Overall Conclusions.
184.The fact that I find Mr Roll reliable does not mean that I automatically accept all his conclusions. I
have given a specific example at [181] about financial pressures. However, Mr Roll’s evidence was
supportive of the claimants’ case and it was necessary for the Post Office to render his evidence
RLITO000005
RLITO000005
unreliable, insofar as they could, in order to damage the claimants’ case and bolster their own. In my
judgment, this the Post Office failed to do. Mr Roll’s evidence was also what directly led to the true
picture emerging of remote access. However, of itself this does not provide an entire answer to the
Horizon Issues.
Mr Henderson
185.As explained in [78] and [79] above, Mr Henderson is a director of a company called Second Sight
which was involved in a mediation scheme, until the Post Office withdrew from it. Initially Second
Sight was engaged to perform an investigation, but after its interim report its scope of involvement
changed. It ran from 2012 to 2015, when Second Sight’s appointment was terminated. The
investigation and subsequent mediation scheme were set up in the light of disquiet about the
complaints of various SPMs, the operation of Horizon, and concerns that were publicly discussed, not
least by Members of Parliament. All of the documents and emails which Second Sight had at that time
were returned to the Post Office. Mr Henderson estimates these as 16,500 emails and 18,500 other
documents, which he said were sensibly organised in 1,700 folders. They were in good order.
186.The documents in that set that were provided by the Post Office to the claimants for this litigation
were not, according to Mr Henderson, provided in the same way. He criticised them for having no
folder structure, having had metadata removed, and lacking the original date and time stamps which
were present on the original documents with which Second Sight had been provided by the Post
Office. Some important documents, such as the Fujitsu XML transaction reports, were described by
the Post Office as “unreadable”. He had however reviewed some, but not all, of the XML data when
inspection was provided of some of the unreadable documents. He did not believe that the documents
provided in the Horizon litigation were the complete set of data provided to Second Sight, which he
had returned. I accept his evidence on this. He was clear in his recollection about how the documents
were organised, and approximately how many there were and their type. I do not know why the
claimants were not given, in disclosure, the same documents, prepared and collated in the same way,
as the Post Office themselves had received them from Second Sight.
187.He described Mr Jenkins, whom he had met, as the Fujitsu lead engineer on the Post Office
contract. He had met him in September 2012. He explained in his witness statement that he had been
given sample XML data by Mr Jenkins, and had used this to see a level of detail that neither SPMs nor,
interestingly, the Post Office, had available to them to view. He had also been told at the same meeting
that remote access was “occasionally used” by Fujitsu. He asked for complete email records for 2008
for the 7 employees known to be working at Bracknell at the time, so he could satisfy himself as to
whether such remote access was done without the SPM’s permission or knowledge. He was not given
these, and the partial records he was given were inconclusive. His reverse engineering of the XML
data he was given gave him grounds for concern but Second Sight’s engagement was terminated
before he could reach any concluded view.
188.He had also identified what he considered to be particular difficulties that Horizon had with
foreign currency - it is (or was) a single currency system and only used pounds sterling - and also the
National Lottery. In view of the evidence of Ms Van Den Bogerd in the Common Issues trial and the
Ping fix, that latter point may not be particularly controversial in any event as this litigation
progresses. Foreign exchange is the subject matter of bugs 14 and 23 in the bug table. Bug 14 is now
accepted by the Post Office in Appendix 2 of its Closing Submissions as one of a number of “bugs with
lasting impact (although they were resolved)”. I find in the Technical Appendix that Bug 23 is a bug.
The contents of the Technical Appendix are therefore consistent with Mr Henderson’s views on this.
RLITO000005
RLITO000005
189.A working group was involved in the Second Sight scheme, and the progress of cases all the way
through the mediation scheme was slower than the working group wanted. Second Sight’s
appointment was terminated before Mr Henderson could reach finalised conclusions on most of the
issues that he said had been uncovered. I have already stated the approach that I adopt in relation to
the technical nature of the Horizon Issues, and how the evidence of the experts in IT who gave
evidence before me effectively takes precedence over observations by other witnesses on technical
issues, which must include Mr Henderson.
190.Mr Henderson explained in his oral evidence that he considered his evidence to the court not to
be entirely unrestricted. He considered it was subject to a restriction that had been imposed upon
him. There was a letter of engagement in this respect, which
was dated 1 July 2014 and accompanied by a document which was signed by someone called Chris
Aujard for the Post Office. The letter of engagement set out several pages of terms governing Second
Sight’s engagement. Clauses 2.1 onwards set out what Second Sight were to do:
“2.1 The Scheme has been set up to try to achieve the mutual and final resolution of a
Subpostmaster's concerns about Horizon and any associated issues.
2.2.Second Sight Support Services Limited ("Second Sight" or "you") has agreed to be a member of
the "Working Group" whose role it is to oversee the Scheme and to assist in investigating individual
Subpostmaster complaints.
2.3.This letter and its schedules will form the basis of the terms of Second Sight's engagement by Post
Office Limited ("Post Office") to provide Services to the Working Group in relation to the Scheme.”
191,Clause 6.2 stated:
“6.2 Second Sight will not, and will ensure that the SS Directors and any SS Personnel will not, act,
directly or indirectly, in any capacity (whether for any former or current Subpostmaster or a
competitor of Post Office or otherwise) against Post Office or any of its officers, directors or employees
save to the extent a) that it is expressly agreed in writing by Post Office that the work proposed to be
undertaken will not have a material adverse effect on Post Office's commercial or financial interests or
reputation, or b) as required by applicable law or by the mandatory rules or requirements of any
regulatory authority, government department or agency to which Second Sight is subject or c) as
required by an order of a court of competent jurisdiction.”
Clause 6.3 imposed this for a period of 15 months.
192.The shorter one page document which Mr Henderson and his co-director were asked to sign and
return contained different terms. The letter of engagement set out the terms upon which Second Sight
would act in the mediation scheme. The shorter document stated the following:
“T will not act, directly or indirectly, in any capacity (whether for any former or current Subpostmaster
or a competitor of Post Office or otherwise) against Post Office or any of its officers, directors or
employees save to the extent a) that it is expressly agreed, in writing by Post Office that the work
proposed to be undertaken will not have a material adverse effect on Post Office's commercial or
financial interests or reputation, or b) as required by applicable law or by the mandatory rules or
requirements of any regulatory authority, government department or agency to which Second Sight is
subject or c) as required by an order of a court of competent jurisdiction.”
RLITO000005
RLITO000005
The period of that was stated to be 15 months, corrected by hand from the period initially typed in of
12 months. This obviously therefore, as signed, represented if not the same then almost identical
terms as those contained in paragraph 6.2 of the letter of engagement.
193.It is a curiosity that the detailed terms in the letter of engagement were produced by the Post
Office - as Mr Henderson explained - about half way through the process. It is also the case that the
services to be provided by Second Sight were to be provided to the Working Group, which also
included members of the Justice for Subpostmasters Alliance or JFSA, although the services were to
be paid for by the Post Office. I have outlined these terms above to make it clear, on a transparent
basis, what restrictions were imposed upon Second Sight. The Post Office’s wish to avoid any
“material adverse effect on Post Office's commercial or financial interests or reputation” is expressed
in very wide terms. Its concern about its reputation is similar to a provision included in its funding
agreement with the National Federation of Subpostmasters which is referred to in Judgment (No.3).
194.The restriction to which Mr Henderson referred in his oral evidence, which he considered limited
his evidence, was an agreement between the Post Office and the claimants about what Mr Henderson
could state in evidence in this litigation, which he referred to as the “protocol agreement”. The Post
Office was therefore in the position that it could, to a certain extent, control the scope and extent of
Mr Henderson’s evidence. He also thought that there was a restriction on the length of the witness
statement he could provide. For clarity, there was no such restriction imposed by the court.
195.The Post Office, in its closing submissions, submitted the following:
“Mr Henderson was asked whether confidentiality restrictions had caused him any inhibition in
answering the questions put to him in cross-examination. He said that he had the issue in the back of
his mind and that he had tried to make sure that his answers did not infringe the protocol. It is
understandable that Mr Henderson would wish to be careful, but the idea that he was restricted in
answering Post Office’s questions is bizarre.”
196.The submissions then went on to deal with the questions put to Mr Henderson, and how there
was no restriction that could have impacted upon his answers.
197.The actual question posed to Mr Henderson was not as set out in the above extract of the
submissions at [195] above, it was rather wider. It was posed by me and it was at the end of his cross-
examination. It was in the following terms:
“I just want to be clear: is it your evidence therefore that because of that protocol agreement your
evidence of fact to this court is narrower in scope than it would be absent the protocol agreement?”
198.His answer was “yes, it is”. He was then asked by counsel for the Post Office whether it had
inhibited him in answering questions, and he said it did not. I do not consider that the closing
submissions are an accurate summary of both questions put to him about this, or the restrictions he
considers had an impact upon his evidence. The restriction was wider than impacting upon the
questions he was asked, and his answer showed that his evidence as a whole had been affected. In any
event, those questions were posed to him by the Post Office, as he was called as a witness for the
claimants.
199.The mediation scheme was brought to an end by way of notice given by the Post Office to Second.
Sight on 10 March 2015. The cross-examination on this was as follows:
RLITO000005
RLITO000005
:.... If we could just look at it very briefly, the termination letter to which you have just referred is at
{F/13/24.1}. That's the termination letter I think to which you at least indirectly referred. A. That's
correct.
Q. If we could look then to {F/1324.2}. This is a much longer letter from Post Office that came to you
on the same day. Do you recall this longer letter? If I maybe tell you broadly what it is to do with. It's a
letter in which Post Office sets out a plan for how Second Sight could finish its outstanding work. Do
you remember that letter? A. Yes, I do.
Q. And the second paragraph makes clear that Second Sight was expected to continue working during
the notice period and that even beyond the notice period there would be a proposed future role for
Second Sight. That's what this document dealt with, isn't it? It's a fairly long document.
A. It is, but there's another document of that date or very close to that date which you haven't
mentioned which was the press release from Post Office announcing the winding up with immediate
effect of the mediation scheme itself. That I understand was the primary decision and our termination
was a consequence of that, not a separate issue.
Q. Well, I think when you say mediation scheme it's fair to say, isn't it, that this winding up process, as
you describe it, did not put an end to mediations; in fact -
A. It put an end to the mediation working group with, as I understand it, no consultation and the
announcement was made I think the day immediately before the next planned meeting of the working
group, so it was a considerable sort of shock to everybody.”
The following points can be made in respect of the evidence from Mr Henderson:
1.Second Sight was engaged to act in a mediation process. Mediation is consensual dispute
resolution, and such processes require a high degree of confidentiality within them if they are to be
effective.
2.The termination of that process unilaterally by the Post Office, with or without consultation, is, so
far as this group litigation is concerned, something that may or may not in the future call for further
consideration in terms of costs. It is not at this stage relevant to the answers to the Horizon Issues.
3.Reasonable and conventional terms of confidentiality were to be expected in any terms of
engagement agreed between the Post Office and Second Sight, given the scheme was a mediation
scheme.
4.Terms seeking to protect the “Post Office's commercial or financial interests or reputation”, which
were included, whether justified or not in all the circumstances of the role being performed by Second
Sight, are not relevant to the answers to the Horizon Issues.
5. In Farm Assist Ltd (in liquidation) v Secretary of State for the Environment,
Food and Rural Affairs [2009) EWHC 2{TCC) a mediator, who had been issued with a witness
summons compelling her attendance to give evidence, failed in her bid to have the summons set aside.
Ramsey J gave a judgment refusing to do so, having considered the position on confidentiality, without
prejudice privilege and other privilege. He stated, in his conclusions at [44]:
“The court will generally uphold that confidentiality [of mediation proceedings] but where it is
necessary in the interests of justice for evidence to be given of confidential matters, the Courts will
RLITO000005
RLITO000005
order or permit that evidence to be given or produced.” The point has not, however, been argued in
this case in any respect, and it may never arise.
6.In this case the restriction has been imposed by the Post Office and agreed by the claimants. It is
regrettable, in my judgment, that any witness of fact feels their evidence to be restricted by any
existing agreement with a party to that litigation. Apart from anything else, it is something of a
contradiction for a witness, who swears or affirms that their evidence is “the truth, the whole truth
and nothing but the truth” to then add that there is such a restriction. This appears to contradict the
requirement to tell the whole truth. However, the court has never been asked to become involved in
resolving any dispute between the parties in this respect.
7.1 do not consider that any such restriction - the scope of which I am in any case unaware - will have
had any effect upon my consideration of the correct answers to the Horizon Issues, or the answers
themselves.
200.Mr Henderson seemed to me to be a careful and honest witness. Although his involvement in the
Second Sight scheme may have led to his having particular personal views about Horizon, he was
measured about how he expressed these. His views about concerns over issues in particular affecting
the National Lottery and foreign currency exchange are, to a certain extent, corroborated both by the
Ping fix and the position of the Post Office on bug 14 in the bug table. However, whether they are
corroborated or not, it is not necessary to come to a concluded view on Mr Henderson's evidence
about these, as it is the two IT experts whose evidence will be of most assistance in this respect.
201.The degree to which the evidence of fact by the claimants affects my conclusions on the expert
evidence will be dealt with in Part L, Overall Conclusions.
E, Evidence of Fact: The Post Office
202.The Post Office called the following witnesses. For the reasons explained further below, I consider
Mrs Van Den Bogerd and Mr Godeseth to be the most important witnesses called by the Post Office.
Mrs Van Den Bogerd
203.Mrs Van Den Bogerd gave evidence in the Common Issues trial, and indeed her witness statement
for the Horizon Issues trial was served during that trial in November 2018, and shortly before she was
called as a witness. She had become the Business Improvement Director of the Post Office shortly
before the Common Issues trial. She is a senior director at the Post Office and is, so far, the most
senior member of Post Office personnel to have given evidence in the group litigation. I had made
certain adverse findings about her evidence in Judgment (No.3) on the Common Issues, which was
sent out to the parties in draft form under the usual embargo shortly before the Horizon Issues trial
started, and was formally handed down at the end of the first week in the that trial, just after the
claimants’ evidence of fact had been completed.
204.1 have already explained that Ms Van Den Bogerd’s evidence was considered completely afresh in
this trial, and that simply because I had made the findings that I had about her previously, did not
mean that I would adopt the same conclusions in this trial. Her written evidence for the Horizon
Issues trial was still substantially of the same tenor in relation to individual SPMs, in terms of
widespread attribution of fault to SPMs as a default setting, but I did detect a change of approach in
Ms Van Den Bogerd in this trial, compared to the Common Issues trial.
RLITO000005
RLITO000005
205.Originally, for example, in her statement Mrs Van Den Bogerd had specifically attributed Mr
Patny’s loss on MoneyGram to user error, based on what she said the data showed. This passage, in
paragraph 72, was corrected by her on a sheet of corrections of a number of the Post Office's
witnesses, and in that correction (there were a number of others) she explained that Mr Patny had
cancelled the transaction but had not reversed it. She therefore accepted that the transaction had
been cancelled.
206.She generally demonstrated in her cross-examination a more realistic approach to the accuracy of
her evidence than she originally demonstrated in the Common Issues trial. By the time she was cross-
examined she would have had the benefit of reading Judgment (No.3), and that might explain her new
approach.
207.She had also amended, in her sheet of corrections, certain statements that had been included in
her Horizon Issues witness statement that were simply not sustainable on the facts. One example of
this was in relation to Mrs Stubbs (a claimant and witness in the Common Issues trial, though not in
the Horizon Issues trial) whom Mrs Van Den Bogerd had said “chose to settle centrally” items that
were, in fact, obviously and plainly disputed by Mrs Stubbs; and in respect of which no SPM had any
real “choice”. Their choice, such as it was, was either paying immediately or settling centrally, which
meant not paying immediately, but seeking time to pay. Mrs Van Den Bogerd’s explanation for this was
that she had not known these sums were disputed by Mrs Stubbs. Quite how that could be, given the
extended saga in relation
to these sums, the involvement of Mrs Stubbs’ MP on her behalf (Sir John Redwood, a former Cabinet
Minister), the Post Office’s promises both to Mrs Stubbs and her MP of an investigation (the results of
which, if one was ever done, have still not emerged, so far as I know), and indeed Mrs Stubbs’ own
evidence in the Common
Issues trial, is entirely unclear.
208.She also explained that in relation to Mrs Burke “I have looked at other evidence in relation to
Mrs Burke and what was very clear to me is that Mrs Burke had done absolutely nothing wrong in
that situation.” This was not at all how Mrs Burke had been cross-examined. Mrs Van Den Bogerd said
that the corrections she had made to her statement had been communicated to the Post Office’s
solicitors before the Horizon Issues trial had started. If that is true, I do not see how counsel for the
Post Office would have cross-examined on the basis of her un-corrected statement.
209. She gave evidence about out of hours transactions and so-called phantom sales, the latter of
which she explained (in her written evidence) as follows. “I am informed by Post Office’s solicitors
that in the course of investigating this matter, Fujitsu have advised that ‘phantom sales’ were reported
in around 2000 which appeared to be caused by hardware issues”. There is a master PEAK in relation
to this from 2001, and even though Mrs Van Den Bogerd was very closely involved in the issues on
Horizon, she had not known about this until some time later. Indeed, she could not remember even the
approximate year when she had become aware of it. She did not even recall, in the witness box,
having seen the master PEAK before.
210.1 am most surprised that Mrs Van Den Bogerd could not remember seeing this PEAK before she
was shown it in cross-examination. It is a very important PEAK. It is PEAK number PC0065021, dated
17 April 2001. The reason it is important is as follows. It relates to multiple branches. It concerns
phantom transactions. It identifies dissatisfaction from more than one SPM as to how phantom
transactions are being investigated and resolved, or more accurately, how they are not being. It shows
RLITO000005
RLITO000005
calls being “closed” - ie brought to an end - without the permission of the SPM, even though that
should not be done. It also shows at least one SPM threatening not to comply with their contractual
obligations due to lack of confidence in the system, and also threats of legal action. Further, in one
branch, where items had been the subject of phantom transactions (according to the SPM) ROMEC,
the Royal Mail’s own engineers, attended that branch to fit suppressors and other equipment in an
effort to rectify this.
211.The PEAK plainly records the involvement of ROMEC, the Royal Mail’s own engineering
personnel, as follows. “ROMEC have been to site and state that they have actually seen the phantom
transactions, so it is not just the PM's word now.” (emphasis added). The significance of this entry is
obvious, and notable. Mrs Van Den Bogerd agreed that this was “independent site visit corroboration
of the problem by Royal Mail’s own engineers at the branch”, and she also agreed that this was
“clearly not user error any more”. I do not understand how the master PEAK containing such
important information could not have been at the forefront of Mrs Van Den Bogerd’s mind. It is, in my
judgment, important corroboration from those with experience of Horizon (the Royal Mail’s own
engineers) who stated they had actually seen the phantom transactions.
212. However, the conclusion reached by Fujitsu and recorded in the PEAK was as follows:
“Phantom transactions have not been proven in circumstances which preclude user error. In all cases
where these have occurred a user error related cause can be attributed to the phenomenon"
The PEAK also concludes “No fault in product".
213.This conclusion by Fujitsu is only not made out on the factual evidence, including the
contemporaneous entries in the PEAK itself, but it is, in my judgment, simply and entirely
unsupportable. It wholly ignores the independent support of the ROMEC engineers, who have
reported that “they have actually seen the phantom transactions” and it arrives at a conclusion that,
in my judgment, entirely contradicts the evidence available to Fujitsu at the time, and indeed
contradicts common sense. Given the entry that “it is not just the PM’s word now”, this conclusion
ignores two entirely different sources of actual evidence. One, what the SPM reported. Two, what the
ROMEC engineers visiting the branch actually saw.
214. Another PEAK Mrs Van Den Bogerd was asked about demonstrated the lack of accuracy in
Fujitsu’s characterisation of the type of problems that made their way to the SSC (and which formed
the basis of the Post Office’s cross-examination of Mr Roll based on the Fujitsu/Parker spreadsheet).
This was PEAK PC0208335 of 14 February 2011, also headed “Phantom Stock Declaration”. This
related to withdrawn stock discrepancies, and was summarised as “Branches will be forced to declare
stock when they don't want to. Apparent reappearance of withdrawn stock may cause spurious
discrepancies." It was recorded in the PEAK that this could affect 10 branches per week over the next
few months. The PEAK showed that a SPM was told to declare the correct stock, which that SPM
disagreed with, and which Mrs Van Den Bogerd agreed would mean the SPM entering an account into
the Horizon system with which the SPM disagreed. Mrs Van Den Bogerd had not seen this PEAK
either, before she was asked about it in the witness box. She agreed that this PEAK appeared to be a
software problem, and also that it was in a sense easier for the SPM because the phantom transaction
related to stock that was not even any longer in use. The SPM in question simply could not have dealt
in that stock.
215.In any event, I found this PEAK of great assistance, not only due to the content I have
summarised above. A fix was developed by Fujitsu but Anne Chambers, a Fujitsu employee who had
RLITO000005
RLITO000005
also given evidence in legal proceedings in court before (in at least one case, the Castleton case) had
entered the following in the PEAK which stated that “this fix and the MSC already applied doesn't
remove all old declarations”. Further, Fujitsu chose to categorise this PEAK as “Administrative
Response”. That term does not begin, in my judgment, properly to describe or summarise the
problems to which the body of the PEAK referred. I find that this PEAK clearly related to a software
problem, regardless of the misleading way that Fujitsu chose to categorise it as “Administrative
Response”. I consider that anyone reading this PEAK at the time could only sensibly conclude that this
was a software problem.
216.Mrs Van Den Bogerd was taken through a number of examples of real-world situations, recorded
in a variety of contemporaneous documents, where a wide variety
of SPMs reported a very wide range of problems. In one an internal Post Office email reported that a
SPM had “found sensitive issue with Horizon when the system put a phantom cheque on the cheque
line in July 2013. Claims to have evidence to support his claim. Although he himself did not suffer a
loss, thinks that Horizon is flawed. Did not ask to be contacted about this. Just wanted to say that he
had this information and threatened to go to MP as a result.”
217.The question was posed internally at the Post Office:
“Given the current media and in particular the BBC's attention on Horizon, do you think it is
worthwhile looking into this ‘alleged flaw' with Horizon that this SPMR has highlighted to pre-empt
any enquiries from his MP?"
218.The ultimate response from Andrew Winn of the Post Office was that the claim could not be
investigated without further details and Fujitsu involvement, that Mr Winn did not understand the
purpose of the call by the SPM, and also stated:
“My instinct is that we have enough on with people asking us to look at things.”
219.Mrs Van Den Bogerd agreed that this was an inadequate response. She said it would have been
very easy for Mr Winn to have contacted the branch and obtained further details, and he should have
done so. In my judgment, the stance taken by the Post Office at the time in 2013 demonstrates the
most dreadful complacency, and total lack of interest in investigating these serious issues, bordering
on fearfulness of what might be found if they were properly investigated. This SPM, whose branch
was known to the Post Office, should obviously have been asked for further details (if further details
were required for an investigation), and the Post Office and/or Fujitsu should plainly have investigated
the matter as a matter of some importance. By 2013 Horizon was an extraordinarily controversial
subject; there can simply be no sensible excuse for the Post Office’s failure to try and understand this
particular subject. This is particularly reprehensible given that an internal Post Office document in
August 2013 showed that Mr Winn’s involvement in this was because his area of responsibility was as
follows: “also responsible for resolving specific branch accounting issues.” It was his specific job to
resolve specific branch accounting issues, yet he decided at the time that “we have enough on”. I
agree with Mrs Van Den Bogerd that this is inadequate - that is putting it at its most favourable for
the Post Office. Somewhat stronger terms are also justified.
220.Mrs Van Den Bogerd had only learned of the ability of Fujitsu in terms of remote access, namely
the insertion of transactions at the counter under Legacy Horizon “within the last year or so”, which
given she is central to the Post Office’s position in the group litigation shows, in my judgment, a
RLITO000005
RLITO000005
remarkable situation. It is not necessary, in order to resolve the Horizon Issues, to go further into why
she has found this out so belatedly, or why that might be.
221.Mrs Van Den Bogerd would accept changes to her evidence in cross-examination where these
were justified. One example was in relation to Mr Latif and corrections issued in respect of the lottery.
Her written statement had said:
“However, due to an error by Post Office, instead of increasing the scratch card stock, the TAs
decreased the stock. To be clear, this was a data entry error by Post Office and not an issue with
Horizon.”
Actually, given the process of issuing TAs for the lottery was entirely automated, and done in
conjunction with the Camelot system and by Horizon, she accepted that the passage should be
corrected. The cross-examination was as follows:
"Q: So would it be fair to correct that part of your statement - A. It would be actually, yes.
Q. - to say “To be clear, this is an issue with Horizon and not a data entry error by Post Office"; is that
a fair correction to make?
A. [have made a mistake in that the way I have worded that, absolutely, yes. So, yes, it is fair.
Q. Would you agree with the formulation I have given, or would you prefer something slightly
different? What would you want the court to note as your evidence?
A. So the automated -- my understanding is the transaction -- the TA is the information that comes
from Camelot to us and then it is passed through into Horizon, so in that respect Horizon just conveys
it, is my understanding, and the information that's come from Camelot in that respect would be
incorrect.
Q. So the point is that either way, it is not a manual data entry by Post Office?
A. No, it's not.
Q. Whatever it is, it's definitely not that, you agree with that?
A. Yes, absolutely not. Agree, yes.
Q. What it could be is some problem with the information somewhere between the terminal in the
branch - A. The lottery terminal, yes.
Q. The lottery terminal in the branch and the matters showing up on the face of the Horizon terminal
in the branch? A. Yes, I agree that.
Q. Somewhere there?
A. Yes.
Q. So it's definitely not a user error, is it?
A. No, that's not user error.
Q. Right. And on the face of it, it at least suggests some doubt as to the robustness and integrity of the
Camelot data coming through in that automated system, doesn't it?
RLITO000005
RLITO000005
A, Twould say yes.”
222.It can therefore be seen that her written evidence - “To be clear, this was a data entry error by
Post Office and not an issue with Horizon” - could more accurately be stated, after she answered
these questions, as “to be clear, this was not a data entry error by Post Office and was an issue with
Horizon”. There is a world of difference between those two statements, which are, in my judgment,
poles apart both in their content and effect. Indeed, the accurate statement is the exact opposite of
how it had been put in her witness statement.
223.She also accepted that whereas her written evidence relied, so far as Mr Latif’s experience was
concerned, upon the fact that TAs had been “accepted by the branch,
which could have been challenged at that point”, in fact a branch had no choice but to accept TAs.
They simply had to be accepted by the SPM, and could not be challenged. She also accepted that
there were problems with TAs not just for the Lottery, but for Post and Go, and also Paystation,
including in Crown Offices (which were not branch Post Offices but which also used Horizon).
224.She had also said that Mr Coyne had relied upon extracts from the Rose Report which were
“taken out of context and mistakenly claim[s] that the relevant reversal was issued in error by Horizon
and not the” SPM. She then went on to explain why this was not correct. The Rose Report is explained
in more detail both below and at [940] onwards where I consider contemporaneous documents.
However, in her crossexamination, she accepted that this part of her statement was not correct. The
Rose Report, in fact and in express terms, stated that the reversal was done by the system, something
Mrs Van Den Bogerd accepted; she said that it was “done by the system, absolutely yes”, that it was
“definitely generated by the system” and there was “no question” it was generated by the system. She
stated she was not “making herself clear” in her witness statement.
225.Mrs Van Den Bogerd was re-examined on this specific point and the question and answer are as
follows:
“Q: The extracts taken from the report by Helen Rose referred to by Mr Coyne are taken out of
context and mistakenly claim that the relevant reversal was issued in error by Horizon not the
subpostmaster."
I just want to give you an opportunity -- Mr Green was pressing you to accept that actually what you
had said in that sentence was wrong. I would like to give you an opportunity to explain to the court
what you actually meant.
A. What I meant was that the actual reversal was part of that recovery and it had actually taken place
as it should have taken place, which is what I meant in that. So it wasn't a failed reversal because it
actually had happened as it should have happened, but I accepted in there that the -- it wasn't obvious
to the postmaster at the time that what had happened -- that he hadn't -- because it didn't show that
he had actually -- it showed that he had done it and he knew he hadn't done what we referred to
earlier was an explicit reversal. That's what I meant in that.”
226.1 do not accept Mrs Van Den Bogerd’s characterisation of the evidence she had originally given in
her witness statement. She had clearly stated that Mr Coyne, having taken passages out of context,
“mistakenly claim[ed] that the relevant reversal was issued in error by Horizon, not the
Subpostmaster”. In fact she accepted that the reversal had been generated by the system, which in
my judgment it plainly had. Not making oneself clear is a curious way of describing that her own
RLITO000005
RLITO000005
statement had said the exact opposite of the factual situation. In my judgment her witness statement
had, on the face of the statement, stated the exact opposite on this part of the Rose Report.
227.In brief terms, the Rose Report dealt with a situation whereby the data being used made it appear
that a reversal had been done by the SPM, when in fact it had been done by the Horizon system. Mrs
Van Den Bogerd was asked about one particular feature of the Rose Report (named after its author,
Helen Rose), which dealt with the fact that no separate code was used such that this could be
identified. This arose as a result of an occurrence at Lepton when the SPM engaged a forensic
accountant as although he paid the shortfall shown on Horizon (about £80) he was adamant that he
had not done the reversal that the system showed he had done. In other words, a reversal could
appear in the Credence data as though it had been done by the SPM, yet in fact it had been the
system (and not the SPM) that had done the reversal. I consider this to be important, given that the
Credence data was used by the Post Office in any investigations. An extract from the report was put to
her in her crossexamination (because she had given the evidence above about Mr Coyne, the Rose
Report) together with a quotation from an email exchange with Mr Jenkins:
"Q:
“Just one question from my part - if the reversal is system created but shows as an existing reversal,
could this not be reflected with a different code ie SR (system reversed) to clear up any initial
challenges. My feelings at the moment are not questioning what Horizon does as I
fully believe that it is working as it should, it is just that I don't think that some of the system based
correction and adjustment transactions are clear to us on either Credence or ARQ logs."
That's what she is saying, yes? A. Yes.
Q. And that was a fair observation, wasn't it, by her?
A. That was a fair observation, yes.
Q. About the shortcomings of Credence and ARQ logs, yes?
A. Yes.
Q. And Gareth Jenkins’ answer:
“L understand your concerns. It would be relatively simple to add an extra column into the existing
ARQ report spreadsheet, that would make it clear whether the reversal basket was generated by
recovery or not. I think this would address your concern. I'm not sure what the formal process is for
changing the report layout. Penny, can you advise as to the process: is this done through a CR?"
Do you know what a CR is? A. Change request.
Q. Change request, okay. Then at the bottom:
" do believe that the system has behaved as it should and I do not see this scenario occurring
. f i
My recommendation is that a change request is submitted so that all system created reversals are
clearly identifiable on both Fujitsu and Credence."
RLITO000005
RLITO000005
Do you know if that change request was acted on after that?
A. I don't believe it has been acted on.
Q. You don't believe it has been?
A. I don't believe so.”
(emphasis added)
228.Another express passage or conclusion in the report was also put to her:
Q. "The reversal was due to recovery (Counter Mode Id= 8 118) so this was not an explicit reversal by
the clerk. This scenario is fairly rare so it is certainly quite easy for the clerk to have made a mistake
and either he or the customer could be in
pocket/out-of-pocket (depending on exactly what happened!)." Then this:
“The system is behaving as it should."
So there were issues, weren't there, where a problem could arise for a subpostmaster by design of the
system; were you aware of that? There was a whole category of PEAK codes for faults which are
agreed between Fujitsu and Post Office to just stay like that as part of the design?
A. Okay.”
These two questions were then put individually:
“Let's take it in stages. You can see there it says "The system is behaving as it should"? A. Yes.
Q. That would be problematic for a subpostmaster, wouldn't it?
A. The way Gareth [Jenkins] describes it here would be, yes, because what he says is that it would
have printed the session receipt but it doesn't seem as if it did, when actually disconnection
transaction receipts were actually printed in this example and a recovery receipt was printed. But
that's not referred to in here.
Q. Well, let's just have a look at that. Let me just ask you my second question and then we will go on to
probe that with more care. The second point is are you aware of a closure code for Fujitsu for PEAKs
which refers to faults which are known in the Horizon system but agreed between Post Office and
Fujitsu to stay there? A. I'm not aware of a closure code.
Q. You didn't know about that?
A. No, I don't know about that.”
229.The following important points arise in respect of this:
1.Mr Jenkins thought that “the system had behaved as it should”. Given Mrs Van Den Bogerd accepted
that this situation could lead either to an SPM, or a customer, either being in pocket or out of pocket, I
disagree that an accounting system should work in that way. This is not only an optimistic description,
it is in my judgment entirely wrong.
2.The conclusion was that the scenario might not occur regularly and create "large losses”. The size
of the potential losses was plainly relevant to Fujitsu. Given how the Post Office used the Horizon
RLITO000005
RLITO000005
system vis a vis its SPMs, sums that were not “large” to Fujitsu does not mean that smaller losses
would not still be appreciable sums to individual SPMs, who would have to pay for them if there were
branch shortfalls. There is also no way of discerning what Mr Jenkins meant by “regularly” in terms of
the risk of re-occurrence. Both regularity, and whether potential losses were large, are subjective. The
Lepton incident showed that the Post Office required the SPM initially to pay the shortfall, which he
did.
3.The Credence data was inadequate to show what had actually happened. Indeed, not only was this
data inadequate, it made it look as though the SPM had done something that he or she had not done,
and which the system had done.
4.It was expressly accepted that more was required in terms of the required accuracy for “giving
evidence and using the same data for prosecutions”. It was concluded that system created reversals
had to be identifiable, which they were not as at the date of the Rose Report and these emails.
5.A change required a formal Change Request, and that was neither initiated nor acted upon.
6.Finally, and in my judgment also importantly, any risks that Fujitsu and/or the Post Office “cannot
clearly see what has happened on the data available to” them and “this in itself may be misinterpreted
when giving evidence and using the same data for prosecutions” is a serious matter. I do not
understand how a report containing such a reference to such a serious matter could be mis-
summarised by Mrs Van Den Bogerd in her witness statement dealing with Mr Coyne’s analysis of
this.
230.A large number of documents, including operational documents, were put to her, many of which it
appeared she had not seen before. For example, she was shown documents in respect of MoneyGram
dealing with a software release in January 2016 which showed, to use the IT speak of the document,
“multiple instances of system latency” - or, to use more normal language, problems in the operation of
the system. These persisted through 2016 until nearly the end of that year. The internal summary of
the problem was:
“For the last several months Post Office has experienced a live operational issue with MoneyGram
transactions across the branch network . In the event of a transaction timing out at the counter, a
system error message is displayed to the user ... and the transaction is aborted. This leaves no record
of the transaction at the counter and the transactions and funds may or may not have been committed
inthe MoneyGram domain, This causes significant issues for Post Office and MoneyGram and for
customers."
(emphasis added)
231.Mrs Van Den Bogerd said that she had not seen the internal document setting all this out until
she gave evidence in the witness box. One of Mr Patny’s issues in his evidence was about MoneyGram.
232.She had also been involved in a meeting on 17 May 2012 with James Arbuthnot MP, Oliver Letwin
MP, and others from the Post Office, including the Chairman and the Chief Executive of the Post
Office. The documents, including the briefing pack, demonstrate that the MPs were told that the Post
Office wished to be “open and transparent”. This was not accurate. The MPs were told that the
Horizon system had undergone an upgrade in 2010 and had the full support of the NFSP. In reality, its
functionality for SPMs had not been upgraded, and the NFSP had privately expressed its concern to
the Post Office about Horizon. There is nothing open and transparent about telling these MPs
RLITO000005
RLITO000005
information to the contrary. James, now Lord, Arbuthnot had been the Minister for Defence
Procurement under the Conservative government of Sir
John Major. Oliver, now Sir Oliver, Letwin was Minister of State for Government Policy under the
Coalition government, which had come to power on 11 May 2010. These gentlemen had become
involved in this matter on behalf of their constituents. They were entitled to expect accurate
information from the Post Office. They did not receive it.
233.A large number of different issues with both Horizon Legacy and Horizon Online were also put to
Mrs Van Den Bogerd, only some of which she knew about. She has been closely involved with the
defence of Horizon for some years. To be fair to Mrs Van Den Bogerd, she has also been responsible
for some changes over time which have made Horizon easier to use, more reliable and (to use the
wording of Horizon Issue 4), more robust. Horizon as at the date the experts were looking at it is a
more robust system than it was, say, 5 or even 3 years ago. A great deal of this is likely to be due to
the efforts of Mrs Van Den Bogerd.
234.Equally, however, the contents of some of the internal Post Office documents that were put to her
were, in my judgment, very damaging to the Post Office’s case on the Horizon Issues. In an “Extracts
from Lessons Learned Log” document of 11 November 2015, which was heavily redacted (including
the redaction of single words), one entry under “issues identified” was as follows regarding the Post
Office:
ilu n_and honest when i rise eg roll out of Horizon, HNGx migration issues/issues
affecting few branches not seemingly publicised."
(emphasis added)
This frank expression in a document authored by Mrs Van Den Bogerd herself as recently as 2015 is
consistent with the general tenor of the claimants’ case and their criticism of the Post Office. This
entry in this contemporaneous internal Post Office document is, in my judgment, inconsistent with the
picture which the Post Office continually seeks to portray, namely that it wishes to be open and
transparent about issues with Horizon, and is as interested in getting to the bottom of any problems
with Horizon as anyone else. It is an internal recognition of a “failure to be open and honest when
issues arise.”
235.There were two periods that she was asked about which clearly showed the problems experienced
by Horizon almost from the outset. These were the introduction of Legacy Horizon, in late 1999 and
2000 (the national roll-out); and the migration to Horizon Online in 2010 (including the pilot scheme
in 2009). Mrs Van Den Bogerd gave some evidence in her statements about both of these, in terms
complimentary of Horizon, although she was far more involved in the latter than the former. At the
time of the introduction of Legacy Horizon she was only responsible for a smaller number of branches
in her then role.
236.In PEAK PC0033128 an entry in November 1999 shows that the branch at Dungannon
experienced a discrepancy of £43,000. The PEAK states:
“PM - Dugannon PO £43k discrepancy
Outlet has a discrepancy of £43,000 after balancing SUs and doing office snapshot.
Phil Turnock POCL BSM has advised outlet on this week’s balance. Steve Warwick development is
investigating why this mis-balance occurred.
RLITO000005
RLITO000005
Immediate impact of this week’s balance has been addressed but POCL are concerned that the cause
is still unknown and this will affect this and other outlets.”
237.After some investigation the following entry appears:
"I have talked with development ref this problem. It is seen as a one-off. No fault can be found and
developments do not expect to be able to find a fault with the evidence available, There is no
additional information available as evidence. I suggest this call be placed on monitor for 1 month."
The approach to this discrepancy mirrors so much of the case. A discrepancy occurs; Fujitsu cannot
find a fault; and say they do not expect to be able be find one “with the evidence available.”
By February 2000 Fujitsu stated “counter 5 looks suspect” and continued:
“Further examination of the event logs for these two counters indicate that counter 5 looks suspect (C
drive nearly full and big gap of no messages).
Calls from PO into HSH for period between 30-Oct and 10-Nov indicate a reboot (counter not
specified, but would tie in with counter 5 event log) on Saturday 30-Oct1999.
The evidence in the message store was that messages continued to be written to the message store
but that all the ‘Payment’ transactions which should have been recorded in the rollover trailer
messages failed to appear (although others did, such as the Rem OUT and Transfer OUT totals).
This indicates that the problem was not one of running out of Disk space but of failing either to
retrieve, or write out, transaction totals for one particular node in the node hierarchy.
Given that there were known problems with corrupted Persistent Object indexes at about this time, it
is possible that an update to an EPOSSNodes object failed to be registered correctly at the outlet,
causing the node accumulation to fail.
It was decided to prove this out by deleting the ‘Payments’ node in the node Hierarchy and then
running the SU balance, to attempt to identify the root cause of the problem. Call passed to testing to
be scheduled.
Update 18th Feb 2000
The test was carried out on 16th February as follows: delete the Payments EPOSSNodes object before
producing a SU balance, on a version of the current live system (CI2_2R).
When trying to print the Payments part of the SU balance, the missing node is detected by the system
and an error tablet with message “A system error has occurred whilst printing. Please ring the
helpdesk. Error at 67640." is generated. So the balance could not be finished.
This type of error trapping error trapping was introduced at the end of last year when resolving AI298
issues and we are investigating if the outlet did not have such error handling when the problem
occurred.
Certainly, with the current system, a missing Payments node now would not go undetected.
The problem is currently back with development for further investigation.”
238.On 17 March 2000 there is another entry that says there has been another incident “of a very
similar nature” at another office. This can therefore be seen to be something that was not, as recorded
RLITO000005
RLITO000005
initially, a “one-off”, even if that were acceptable given the discrepancy of £43,000. Mrs Van Den
Bogerd agreed that this showed that the underlying information was not checked until February 2000,
even though the discrepancy occurred in November 1999. She was asked about a later entry in April
2000 that referred to another occurrence, this time in Appleby for £9,000 approximately. The cause
for this was recorded in July 2000 as “data trees failing to build” and that entry states “there have
been a number of calls relating to this kind of issue. A fix has been put in at C14 which will prevent
this happening”. Mrs Van Den Bogerd accepted that this was an example of an error occurring, and
the system itself failing to spot that the error had occurred.
239.She was shown PEAK PC0027887 which was dated July 1999 and related to “receipts and
payments misbalance”. This showed an escalating misbalance that increased, in my judgment
dramatically, in steps up to over one million pounds:
“On week 9 receipts and payments misbalance of £1337.05, week 10 misbalance of £24000, week 11
misbalance of £12000, week 12 misbalance of £1051111.48 and week 13 misbalance of £17426.05,
she has a difference on week 11 of balance due to post office and balance brought fwd on week 12 of
£1082544.32
Overall these weeks net out a difference of £27343.84. she needs business support (reconciliation) to
look into this.”
240.Mrs Van Den Bogerd agreed that £1.05 million was a large amount for any branch and could not
be a “real amount of cash” that “the SPM has put in her pocket”. A later entry in the same PEAK of 27
July 1999 was:
“Balance brought forward was multiplied twice due the known software error. The initial balance
brought forward for this CAP was £1196622.72. This was multiplied twice to give a total BBF of
£2279189.04.
The discrepancy was therefore £1082540.28. This was due a known software error which has no been
resolved.”
241.She did not know if “no resolved” meant “now resolved” or “not resolved”. That PEAK was not
closed until 31 August 2000, and was categorised as “Administrative Response”. One of the latest
entries in this PEAK, which was not put to Mrs Van Den Bogerd but in my judgment is relevant to
Fujitsu’s approach generally, in particular to categorisation of PEAKs (as this had already stated there
was a “known software error”), was dated 31 August 2000 (one year after the original issue occurred)
and it is most convenient to reproduce here in this judgment. This stated:
“I have now taken over analysing this problem from Steve Warwick who requested additional
information on 13/10/1999 17:49:21. I have looked at the information and I am not convinced it can
possibly be complete.
For example he said he wanted information from 20th May until 26th May. I opened the spreadsheet
mc20may.xls and observed it only had 6 counter 32 transactions in - is this correct?
In any case my team is not used to dealing with data in this form (spreadsheets). We have developed
techniques to look at problems using a full message store plus audit and event logs from the failing
counter. Even if all the relevant spreadsheets could be obtained I do not think it would be worthwhile
me getting to grips with this new method of analysing problems.
I see this is a very old problem (21/07/1999) and there have been many software updates since then.
RLITO000005
RLITO000005
May I suggest we discontinue investigation of this particular problem but that if a similar problem
occurs again you send full message store plus audit and event logs from the failing counter.”
And
“Closing call on basis of insufficient evidence. As this is such an old call I have not contacted the call
originator. I suggest that this call remains closed!
INSUFFICIENT EVIDENCE - ADVICE GIVEN”
(block capitals present in original)
To close such a PEAK as “Administrative Response” is, in my judgment, remarkable.
242.The number of different incidents recorded in the monthly incident review increased in November
and December 2000, which is shortly after (for example) the date when Horizon was installed in Mr
Bates’ branch office at Craig-y-Don, as set out at [105] of Judgment (No.3). By then the Horizon
System had been rolled out nationally. As the ICL Management Summary that was put to Mrs Van Den
Bogerd states:
“The most frequently occurring incidents in November were both types of Receipts and Payments
Incidents (Migration and Post Migration), with 31 incidents per category. The Migration incidents
have remained at the same level, whereby the Post Migration occurrences have increased. This was
followed by 17 Transactions Polled by TIP but not by HAPS, these were due to delayed transactions as
reported on APSS 2133c. These transactions are added back into normal processing.”
243.Mrs Van Den Bogerd was not aware of these problems going on at the time; she was a Regional
Network Manager and did not have a national role then. No background information was provided to
her then about problems that branches other than her own were experiencing, which were being
passed back to ICL at the time. She therefore had no knowledge of the type of problem experienced
by Mr Bates in June 2001, which was recorded in a Post Office audit document of his branch that was
put to her which stated “A correct assessment of cash holdings could not be made because the
Horizon system intermittently adds the previous days cash holdings to the daily declaration.”
244.By 2010 and the introduction of Horizon Online, she did have a national role and broader
responsibility. She agreed that Horizon Online did not upgrade functionality, and documents were put
to her that showed, as Mr Green put it “a litany of issues” with Horizon. The pilot scheme went
sufficiently badly that Fujitsu put it on “red alert” and the high volume pilot was suspended in April
2010. Mrs Van Den Bogerd
had not even mentioned this in her witness statements, and said she did not know this had occurred.
The NFSP are recorded as “raised concerns but remain supportive”. All migrations were suspended as
from 25 March 2010. There were database errors recorded in a Post Office document at F/614 called
“Horizon Online Programme Update” which required an Oracle patch set to be supplied by Oracle,
and on one occasion for a limited period 50% of MoneyGram transactions failed.
245.A Network Functional Report of May 2010 was put to her. Over 5 pages of the 6 pages are
redacted. Mrs Van Den Bogerd could not recall if she had seen this document, and said that given the
extent of redactions, it would be hard for her to recognise it, an observation with which I agree. She
said that she was not aware of there being issues with Horizon over Easter 2010, which the document
recorded.
RLITO000005
RLITO000005
246.Redactions had been incorrectly applied by the Post Office and/or its solicitors to some of the
relevant documents. This can be seen from the following example. I asked leading counsel for the Post.
Office to review certain redactions and having done so, an unredacted version was disclosed of a
document dated 25 June 2014 entitled “Branch Support Programme”. This had been co-authored by
Mrs Van Den Bogerd. The purpose of the paper was to “Update the Post Office Executive Committee
on the progress of the Branch Support Programme.” Each one of the 6 key performance indicators
against which the programme would be tracked had initially been redacted by the Post Office, and
following the review by counsel as requested by the court they were unredacted. They were as
follows:
“* Reduction of operating costs by £3m per annum
*Reduction in net agent debt by £1m
*Reduction in subpostmaster suspensions as a result of audit shortages to a level of 60 per year
*Reduction of calls into NBSC by 25%
*Reduction of audit losses of £10k of over by 50%
*Satisfaction with on-line training models of 95%”
247.Apart from the first bullet point, three of the other five are consistent with an aim by the Post
Office to make improvements to reduce the amount of debt incurred by, and suspensions, of SPMs,
and reduce the audit losses. The other two are related to training. This is not consistent with a view
that the debt/suspensions/audit losses are incurred by carelessness on the part of SPMs or criminal
activity. It is also hard to see how it could be justified that these had been redacted originally.
248.So far as her evidence about Mr Latif was concerned, after Mrs Van Den Bogerd had been taken
through the features of his experience with the failed stock transfer and the similarities with accepted
bugs in Horizon, namely the printing out of two receipts very close to one another (in his case, 4
seconds apart), and asked if this worried her, she said she “would want to go back and have a look at”
it. In my judgment, part of the way through the Horizon Issues trial in 2019 is rather too late for that.
I conclude
that there was no real answer available to Mrs Van Den Bogerd to the points put to her concerning Mr
Latif’s experience. Had there been, I have no doubt she would have provided that answer in evidence.
249.Mrs Van Den Bogerd was in the witness box for in excess of one day, the longest period of any of
the witnesses of fact for either the claimants or the Post Office. Her cross examination led to a far
greater understanding of the Horizon Issues on the part of the court, although her written evidence
was, as originally drafted, extraordinarily one-sided. She minimised any reference to problems or
issues with Horizon, and reverted to potential user error whenever possible as a potential explanation,
an approach which she explained in her written statement as providing “plausible” explanations. Her
witness statement also stated, in terms, the exact opposite of what the reality of the situation was, and
T have given examples at [221], [223] and [226] above. Witness statements are supposed to be
factually accurate, and care must be taken in future rounds of this group litigation that they are
drafted in accordance with the rules. Making statements that are the exact opposite of the facts is
never helpful, to put it at its mildest. It is also the opposite of what witness statements are supposed
to be.
RLITO000005
RLITO000005
250.However, during her cross-examination Mrs Van Den Bogerd readily conceded a number of points
which were put to her. Her accuracy in her oral evidence was also more satisfactory than it had been
during the Common Issues trial. She accepted her written evidence suggesting a causative user error
by Mr Tank was not correct, for example. I do not consider that her written evidence had provided
plausible explanations. It provided explanations that the Post Office wished to advance, as these
explanations would, if accepted, provide a defence to the claimants’ factual case. These explanations
were not based on the facts.
251.There were no evident attempts on this occasion to mislead me in her oral evidence, or avoid
uncomfortable points through a claim of ignorance (with the possible exception of the PEAK referred
to at [210] above) or of not having seen a document before. When she told me that she had not seen a
particular document before, I accept that. However, she also told me that she had been assisted in
preparing her evidence by a team of ten people, and the Post Office had devoted “high level
resources” to her evidence, as one would expect. It is therefore very surprising that she had not seen
many of the important documents that Mr Green put to her in cross-examination. Some, such as the
master PEAK on phantom sales, one of the recurring issues on Horizon, would have been the obvious
place for anyone, still less a team of ten, to start when considering preparation of evidence for a
witness statement. She explained that there was some pressure of time in terms of how long was
available to her to prepare the statement, and this was further explained in supplementary
reexamination, but I do not accept insufficient time as a valid explanation for her lack of knowledge on
such important points. For example, she told me that at the time of preparing her witness statement,
she had not even heard of the Callendar Square bug, one of two bugs that the Post Office accepted
some time ago had been present in the Horizon system. This is an extraordinary gap in her
knowledge. She did not know that there was a KEL dealing with failed recoveries, originally raised by
Anne Chambers in Fujitsu as long ago as 2010, which is called KEL acha959T and which was updated
most recently in 2017. This described failed recoveries, and seemed on its face to accept that these
would recur, and was very close to the experience of both Mr Tank
and also Mrs Burke. I do not see how Mrs Van Den Bogerd (assisted by her team of ten, and with the
benefit of the Post Office’s considerable resources) could seek to give accurate evidence in the
Horizon Issues trial without referring to this KEL, still less without even knowing about it. I am also
somewhat disappointed - putting it at its very best for the Post Office - that a team of ten could have
assisted Mrs Van Den Bogerd in preparing a witness statement that was so inaccurate on such
important points as I have identified above.
252.Although my findings on her evidence in the Common Issues trial cannot be ignored, I am of the
view that her approach in the Horizon Issues trial to answering questions was far more constructive
and aligned to what is expected of any witness giving evidence in court, particularly a senior witness
of an organisation such as the Post Office. I do however consider that this litigation, and indeed her
cross-examination, is a very expensive way for a senior director at the Post Office to become educated
about the myriad issues contained in the documents that were put to her. Either the team of ten
people assisting her with her evidence had the aim of producing entirely one-sided evidence in chief,
or they were unaware of all the documents relied upon by the claimants. Either alternative is highly
regrettable.
253.Finally, and this is a point in Mrs Van Den Bogerd’s favour, there is no doubt that the Post Office is
not as sufficiently close to the detail of what has occurred over the years on Legacy Horizon and
Horizon Online, as Fujitsu. As will be seen from my analysis of the Fujitsu evidence of fact, I have
certain views about the lack of accuracy on the part of Fujitsu witnesses in their evidence. If that lack
RLITO000005
RLITO000005
of accuracy has also been included in reporting to the Post Office by Fujitsu, then that goes some way
to explaining the Post Office’s lack of grasp of so much material that is consistent with the claimants’
case. As at late 2019, the date of this judgment, the Post Office also has the added benefit of the views
of both the IT experts in the litigation, their four joint statements, the agreed number of bugs in the
bug table (12, if one takes the number accepted by Dr Worden), the total number of bugs Mr Coyne
says he has discovered (which so far as Horizon Issue 1 is concerned became 21), and collectively my
findings on the Horizon Issues. These are not restricted solely to the number of bugs which I have
found to exist which relate to Horizon Issue 1, but also those that relate to Horizon Issue 4. There are
multiple bugs, errors and defects in both Legacy Horizon and Horizon Online in its HNG-X form.
254.The degree to which the evidence of Mrs Van Den Bogerd sits with my conclusions on the expert
evidence will be dealt with in Part L, Overall Conclusions.
Ms Phillips
255.Ms Phillips is a Team Leader at the Post Office for Agent Accounting and Santander Banking. She
was originally engaged by the Post Office in 1999 through an employment agency, and has worked in
a number of roles since then. In her current role, she oversees the processes of recovering losses from
SPMs that they have declared in their branches. The department used to have “debt” rather than
“accounting” in the name, but its function remained the same.
256.Ms Philips confirmed what had been in dispute for so long during the Common Issues trial,
namely that SPMs had no option but to accept the figures provided to them, even
though they may have been “settled centrally”. This is notwithstanding that her terminology in her
witness statement was that SPMs “chose to accept”. This terminology was used throughout the
witness statements for the Post Office, and it has already been seen that Mrs Van Den Bogerd
accepted TAs had to be accepted, and there was no choice in this respect, as Mr Latif had explained.
SPMs clearly did not have a choice, and the wording “chose to accept” is not an accurate description.
It obviously states or suggests that SPMs had a choice, when no such choice was available to them.
The only choice they had was whether to pay the amount immediately, or settle centrally which means
obtaining time to pay. It had been necessary for me to make findings on what “settling centrally”
meant in the Common Issues trial, and the Lead Claimants in that trial had been directly challenged
on this very point in their cross-examination.
257.There is no reference in the letters sent to the SPMs in question relating to any ability to dispute
the sums which the Post Office treated as debts. Ms Phillips’ evidence went to the way that the Post
Office seek to collect sums from SPMs who have settled centrally, even though they may have earlier
raised a dispute (or sought to raise a dispute) with the helpline. Sometimes amounts above £5,000
would be checked by reference to the helpline call logs, but the amount was not a set threshold and it
depended upon how many sums had been settled centrally as to whether this happened. This checking
did not therefore always occur, even above that figure.
258.Her evidence demonstrated that there was no dispute function within Horizon. Essentially, her
task was debt collecting, as the Post Office saw amounts “settled centrally” as debts. Her concern was
to obtain payment, or agreements to pay, and if not, then to escalate the matter to a SPM’s contract
adviser. Usually, she said, the first or even second letter would lead to payment by an SPM. Neither of
those standard letters referred to disputes. Mr Godeseth confirmed that the lack of any dispute button
or function on Horizon was, as he put it, “by design”. This shows that the Post Office had been
advancing a case, at least for a substantial part of the Common Issues trial, which was directly
RLITO000005
RLITO000005
contrary to the evidence of its own witnesses of fact in the later Horizon Issues trial. I find this
difficult to understand or explain. However, Ms Phillips’ role was fairly clear and she explained the
process of branch dispute forms which would be used when a SPM raised a dispute directly with her
team. This elementary system seems sensible, but it was only introduced in early 2018, and it would
lead to a seven day period after which (if the form was not returned) the “block” placed on the amount
would be lifted. She estimates that it would be used at least 5 times a month.
259.Ms Phillips gave useful evidence on the system used by her team, and I found her credible. The
evidence did however go more to the Post Office’s accounting functions, and how it pursued sums that
it considered to be due from SPMs. The reforms introduced in 2018 in terms of the branch dispute
form are part of that process, but are not part of the Horizon system.
260.She had also provided some evidence to Mr Smith (who was also called as a Post Office witness)
in terms of the number of Santander TCs that were issued, which had not been described correctly in
his witness statement and which had been misunderstood by both the experts. This was clarified, and
also she explained that of the number of TCs issued in one year in her particular area, 2016-2017,
which was
3,968, the Post Office could not say how many were disputed because that data was not kept. This
would appear to be a fairly elementary piece of information for the Post Office to monitor or record.
The failure to do so is surprising.
261.Ms Phillips’ evidence in her witness statement did not directly impact upon the Horizon Issues to
any appreciable degree. Her evidence went to wider issues in the group litigation. Her figures for
Santander TCs, which had been given in evidence (for some reason) by Mr Smith and not by her, were
considered by the experts.
Mrs Mather
262.Mrs Tracy Mather is a Team Leader of the Finance Service Centre or FSC. She started working
for the Post Office in 1987 and has been in the FSC throughout, becoming a team leader in 1999. She
has managed teams in Cheques, Postal Order, Pay-out and MoneyGram. Her witness statement was
served in response to Mr Coyne’s 1st Expert Report, and explained why the Post Office used Credence
to investigate discrepancies. She explained the difference between Credence, which records the
transactional data, and POLSAP, which is what is called a “back end” accounting system. She
explained the latter is on a higher level. Credence is used by her team to investigate differences
between what a branch says has happened to a transaction through Horizon, and as she put it, what a
different source of information might say. Credence was adopted in 2009.
263.She confirmed that Credence does not record key strokes, it records “transactional data as in
sales and non-sales”. Mrs Van Den Bogerd had stated that Credence records actual key strokes by
stating in her written evidence that Credence “records all key stroke activity performed in that branch
by the user ID, date and time.” This was not correct, and was corrected by Mrs Mather in chief. Her
team does not have access to the ARQ data, which has to be obtained from Fujitsu. This is also called
audit data. Credence is not used for MoneyGram. I accept her evidence on this point and find that
Credence does not record all key stroke activity.
264.Doubts expressed in the Ernst & Young Management Letter for the year ended 27 March 2011
about the integrity of data in Credence were put to her, but she was unaware of any such doubts
having been expressed and did not know about them. She had not been in a team that had requested
RLITO000005
RLITO000005
audit data, and said that would be the fraud or security team who would do this. Passages from
internal documents suggesting the audit data came at a high cost to the Post Office were not
something of which she seemed to be aware. She started using Credence in 2016.
265.She gave written evidence about what was called the Rose Report, although she did not actually
know about it herself, and she also was unaware of anyone being deterred from making ARQ requests
due to the cost to the Post Office (charged by Fujitsu), which was a point Mr Coyne had raised. Given
her team did not make ARQ requests, her evidence on this was unsurprising. She was aware that a
certain amount of requests were allowed, and above that amount a charge was raised by Fujitsu;
however she did not know what the charge was.
266.There was one passage of her witness statement which went plainly beyond any direct evidence
she could usefully give. It was in paragraph 20 and gave vague evidence
about her understanding of some confirmation given by Fujitsu to the Post Office’s solicitors about the
contractual limit on ARQ requests, when or if that had been exceeded, a “new commercial term”
between Fujitsu and the Post Office about what was agreed and the cost of these. Her conclusion was
“any requests above the agreed limit are chargeable but I understand that the terms depend on the
details of the requirement.” She was not cross examined about this, and given her answers in cross
examination about her understanding of ARQ requests (which was minimal) and her experience of
having made such requests (which was never), that is understandable. However, in terms of providing
guidance to the parties on the contents of witness statements, to assist the efficient conduct of these
proceedings going forwards, witnesses should not have paragraphs of this nature in their witness
statements about matters in which they have had no involvement at all. Mrs Mather - and there is no
criticism of her in this - has simply never been involved either in making ARQ requests, or in the
contractual arrangements between Fujitsu and the Post Office, or the “new commercial term”
between those two entities in relation to the extra data required for the litigation. She stated that she
had “no idea” about the charges that would be raised.
267.1 found Mrs Mather to be a credible and helpful witness. Her evidence was very useful regarding
Credence, and otherwise of less than central relevance to the Horizon Issues. The degree to which the
evidence of fact supports my conclusions on the expert evidence will be dealt with in Part L, Overall
Conclusions.
Mr Smith
268.Mr Paul Smith is the Operations Support Manager at the FSC and has been since November
2016, having started at the Post Office in 1996. Originally he started in pensions and then moved to
human resources and payroll, until in 1999 he moved to the Network Business Support Centre
helpline, or NBSC. In 2000 he moved to work as a Problem Manager, dealing with problems that came
through the helpline. He led the team that updated the knowledge base for the helpline, and was
Lottery Manager in 2004. After performing different roles he became a Change Analyst in 2011, and
then the Santander manager. He returned to Lottery within the Post Office in 2013 and worked there
as a Team Leader. His existing role has a number of different activities including handling specific
product based problems. His evidence dealt with volume of TCs and a suggestion made in Mr Coyne’s
report that TCs for a branch in Potters Bar were initially made in error.
269.One passage in his statement is of particular importance in resolving the Horizon
Issues. It states:
RLITO000005
RLITO000005
“TCs are issued by FSC. I understand from Post Office’s solicitors that the processes by which FSC
determines whether a TC is required is outside the scope of the Horizon Issues trial.”
270.The reason that this is important is as follows. Dr Worden relies upon TCs as one of his
“countermeasures” that goes to the robustness of the Horizon system. Mr Coyne does not, and the
claimants submit that the whole process of TCs is entirely outside the scope of the Horizon system.
This evidence from one of the Post Office’s own witnesses makes it clear that the claimants’ point of
view is shared (or at least was
when Mr Smith's statement was prepared) by the Post Office’s solicitors, and the decisions regarding
whether TCs are issued are outside the scope of the Horizon Issues. This evidence by Mr Smith
supports the claimants’ position on this, and also supports the approach of the expert evidence of Mr
Coyne, and is contrary to the position adopted by Dr Worden.
271.A case management system was introduced by the Post Office in September 2018 to record each
individual challenge to a TC. This is called the Dynamics System, and is still in what is called “roll
out”. Individual challenges to TCs were not recorded prior to that. Further information about how this
system was working was obtained in cross-examination, and leads me to the conclusion that the way
that TCs are recorded now is far improved from what it was before the Dynamics System was
introduced. That does not however take one very far. Mr Smith knew of reports that were available
from that system but he had not produced any. Those reports would be the best evidence of the type
of points Mr Smith was seeking to make for the most recent periods at least, but given he had not
used them, it is not possible to speculate on what they might show.
272.Mr Smith had obtained almost all the data contained in his statement from others, and some was
plainly incorrect, and other data so vague as to be wholly unhelpful. For example, for Santander,
originally he had provided a table with a column stating that 2,890 disputes were received from
branches in the 2016/2017 financial year. In fact, that column should have been disputes received by
Santander from the Post Office. This was corrected in his second witness statement, but Mr Smith was
unnecessarily combative, in my judgment, about the need for the correction, maintaining that “the
facts and figures were correct” in the first statement when they plainly were not. The figures were
correct as figures but the descriptions of what the figures were, were plainly wrong.
273.Even after the correction, the various figures extracted by him from information provided by
others was not particularly useful. For example, the figures for the numbers of TCs disputed, and for
compensating TCs, for BOI Retracts and the Lottery, were both estimated at 1,500 and 500, for both
line entries. This estimate was plainly of the most vague type, and even Mr Smith could not comment
on the reliability of either of those estimated figures for either line entry, although he maintained the
figure for issued TCs would be reliable. This evidence from him, was of the most general type, and
was not of great, or indeed any, assistance. For example, in respect of MoneyGram his evidence was
that one or two disputes a month were received but they were rarely if ever accepted. Given he had
no idea at all of the overall total, this evidence does not assist me in resolving the Horizon Issues one
way or the other.
274.His explanation for the TCs at Potters Bar was that a cheque that was deposited for £90,000 was
mis-keyed, and entered for £900,000. Accordingly TCs were required to correct this at the branch,
and the need for two was because the payment was made by cheque.
275.1 found Mr Smith’s evidence, with three exceptions, to be of no assistance whatsoever. The two
exceptions are his evidence I have summarised in [269] which is relevant to the scope of the Horizon
RLITO000005
RLITO000005
Issues and Dr Worden’s countermeasures, and his explanation about Potters Bar and the two TCs. I
accept his explanation about the Potters Bar TCs. Both of those exceptions are relevant to the Horizon
Issues. The third exception is his evidence about the Dynamics System. This goes to the robustness of
Horizon as of 2019 and is therefore relevant. However, the absence of such a system also goes to
robustness prior to its introduction in 2018, and is also therefore relevant. However, his wider
evidence is of limited relevance at best. His knowledge of the figures contained in his statement, and
the vagueness of his evidence generally, were such that the utility of the exercise which he presented
in his statement is de minimis. In any event, the substance of that evidence went to an attempt to
demonstrate that some TCs are challenged by branches, and fewer challenges are upheld than are
made. The figures vary inevitably from year to year, and this case is not about general statistical
trends in any event. In some areas of business TCs have fallen to a very low level, for example the
DVLA, which Mr Smith explained was only 4 in the last year, none of which have been upheld.
However, the records that were put to him show that there were 2,717 in the period 2010/2011. All
this evidence does is show that far fewer TCs are now issued by the Post Office in relation to the DVLA
business, which as Mr Smith explained, has fallen considerably in any event. None of these points are
of particular relevance to resolving the Horizon Issues. The experts are agreed that Horizon as it is in
mid-2019 is more robust than it was in the past. One would expect, therefore, fewer TCs to be issued
now.
Mr Johnson
276.Mr David Johnson is a Training and Audit Advisor at the Post Office. He started his career at the
Post Office in 1984 as a counter clerk in a Crown Branch, and worked his way up until he was the
Branch Manager at the Barry Crown Office. He was in that post when Horizon Online was introduced
in 2010. He moved to his current role in 2012, and his role used to be called Field Support Adviser.
Two thirds of his time is spent training SPMs; the other third is spent auditing them. He had also been
involved in training some of the legal representatives for both sides involved in the group litigation,
and the experts, on the Horizon system. As well as his first witness statement, he had provided a
supplementary one dealing with some points in Mr Henderson's witness statement which was served
for the claimants.
277.He was cross-examined about screen layout and design, the risk of mis-keying (which did not
seem to concern him, although there were internal Post Office documents showing it was a known
problem) and also the reports and functions available to SPMs to investigate shortfalls and
discrepancies, rather than those which the Post Office itself used such as Credence. He agreed that
the ways available to SPMs to investigate in branch were “not the most user-friendly way of
investigating”.
278.He agreed with a statement in a 2016 internal Post Office document entitled “Network
Development Programme Operation Simplification” that stated:
“There are a number of branch operations processes, especially around branch accounting and
reconciliation, which operate using legacy processes. They are unnecessarily complex and detract
Post Office branch resources from serving customers. Stock Unit Management and accountability is
very poorly controlled and is operated on very complex business rules. The lack of accountability and
visibility of cash and stock transfers between Stock Units can lead to errors, rework and provides
(emphasis added)
RLITO000005
RLITO000005
279.He also stated that the following entry was “not completely unreasonable” although he said that
his experience of suspense accounts in branches was that they worked and did what they were
supposed to do:
“Similarly, Suspense Accounting is based upon legacy cash accounting practices, illdefined and out of
date processes. Inefficiencies lead to poor utilisation of resources both in Post Office branches and
Support Services."
280.Mr Johnson is a highly experienced person, as one would expect given that the Post Office chose
him to train the legal teams and experts on Horizon. All his experience, so far as using Horizon is
concerned, is in Crown Offices, which are not the same as branch Post Offices, in that the Post Office
operates Crown Offices itself. This means that a SPM in a branch Post Office is personally responsible
for any losses in that branch; whereas a Crown Office manager is an employee of the Post Office. This
therefore gives him a somewhat different angle of approach to the use of Horizon, but I accept him as
an accurate and helpful witness.
281.His evidence was very useful in terms of how Horizon works. The Horizon Issues are not about
the actual design of the Horizon terminals in branch, and whether (say) this could be improved by
having larger buttons or the screen layout adapted differently. In some of Mr Johnson’s evidence, the
claimants’ counsel strayed into the areas of potential improvements to functionality in branch and
design. These are not part of the Horizon Issues. The degree to which the evidence of fact affects my
conclusions on the expert evidence will be dealt with in Part L, Overall Conclusions.
Mr Dunks
282.Mr Andy Dunks is at IT Security Analyst at Fujitsu Services Ltd, and was the first Fujitsu witness
to be called for the Post Office. He has been employed at Fujitsu on the Post Office account since
2002. He has working knowledge of the Horizon System. His evidence went to audit data extraction
and the integrity of the data during that process. His evidence related to the process as it was when
he made his statement, although given he had been involved for some 17 years his experience also
related to Legacy Horizon and his evidence about data extraction covered that period too. As emerged
in his cross-examination, he had also given a witness statement in the criminal trial of Mrs Seema
Misra.
283.He agreed with the definition of data integrity put to him, which is that data integrity is the
overall completeness, accuracy and consistency of that data which you can measure by comparing
between sources. He said the 12 controls that he had listed in his statement explaining these he had
compiled as a list from his recollection, and that he had not worked off a document to come up with
the list in his statement.
284.Mr Dunks gave every indication, in the first part of his cross-examination, of being helpful and
frank, and explained the process of data extraction and answered questions put to him openly.
However, this encouraging start came to an end when he was asked about two curiously worded
paragraphs in his statement. These stated as follows:
“8. There is no reason to believe that the information in this statement is inaccurate because of the
improper use of the system. To the best of my knowledge and belief at all material times the system
was operating properly, or if not, any respect in which it was not operating properly, or was out of
operation was not such as to effect the information held within it.
RLITO000005
RLITO000005
9. Any records to which I refer in my statement form part of the records relating to the business of
Fujitsu Services Limited. These were compiled during the ordinary course of business from
information supplied by persons who have, or may reasonably be supposed to have, personal
knowledge of the matter dealt with in the information supplied, but are unlikely to have any
recollection of the information or cannot be traced. As part of my duties, I have access to these
records.”
285.Before I come to his evidence about this, it is obvious that the wording of paragraph 8 is almost
that of a legal disclaimer (or a legally worded claim of accuracy, to be more precise), rather than a
witness’ actual evidence. It would be very curious for a witness of fact to decide to put such a formally
(and rather clumsily) worded paragraph in their witness statement. However, he was asked about
these passages. He was asked what he meant by “the system”. The transcript records the following:
“Q: What is the "system" there? Is that the system of the process of extracting audit data, or is it
something else?
(Pause).
A.Good question. There's no -- I'm not sure what I was meaning by that, "There is no reason to believe
286.He was asked if, in relation to the part which states “the system was working properly or, if not,
any respect in which it was not operating properly, or was out of operation” the following:
“Q: Again, pausing there, it is slightly confusing. Are you aware or not aware of any instances where
that system was not operating properly?
A.No, not really, no.”
287.He was asked if that part of the statement had been written for him by someone else, and he said
he could not remember. He was also asked if it was a “Fujitsu party line” to put this in statements
when it comes to extracting data and he said no, and that he was not aware of a Fujitsu party line. He
accepted that gaps and duplicates in audit data were problematic, and he explained that they had to
be investigated by audit support. It was clear to me that Mr Dunks considered audit data that had
gaps and duplicates should not, in that form, be used. He said he would contact audit support if these
occurred.
288,However, some of Mr Dunks’ evidence, in particular what he had to say about paragraph 8 of his
own statement, proved in my judgment to be somewhat misleading.
289.He was shown a witness statement from Mr Gareth Jenkins from the Misra criminal trial in 2010.
He said he had not seen this statement before. The statement put to him is some 8 years older,
approximately, than his own statement. Mr Jenkins’ statement stated at the end:
“There is no reason to believe that the information in this statement is inaccurate because of the
improper use of the computer. To the best of my knowledge and belief at all material times the
computer was operating properly, or if not, any respect in which it was not operating properly, or was
out of operation was not such as to effect the information held on it. I hold a responsible position in
relation to the working of the computer.”
RLITO000005
RLITO000005
290.The similarities between this passage, and the one in paragraph 8 of Mr Dunks’ witness
statement almost 9 years later, are somewhat striking. Given he had not seen the statement before,
there was no obvious explanation. His answer was as follows:
“Q. And you're not sure why it would largely replicate your paragraph 8?
A.No, I mean, we do have a standard witness statement that we produce for ARQs. When we supply
ARQs we are sometimes asked for a witness statement to go through the process and verify as far as
I'm aware that the data I supplied is accurate. Now, we use that quite a lot and it may actually be in
that statement.
Q. I see. So when I asked you earlier whether it was something of a Fujitsu party line you said you
didn't think it was, but actually it looks as though it is on the basis of what you have just told me and
on this document as well.
A. It could be. It may be part of our standard witness statement that we supply.”
291.Mr Jenkins had given evidence in his witness statement from 2010 about duplicate entries in the
audit data that had had to be removed. Mr Dunks was then taken to the witness statement of another
Fujitsu witness in the Misra trial, called Penelope Thomas. She had given evidence about data
extraction and listed a number of controls, which were remarkably similar to the ones that Mr Dunks
gave evidence about (although Ms Thomas had omitted one). This is hardly likely to be a coincidence,
if Mr Dunks really had compiled his list of controls from memory or experience. She also had the
identical passage in her witness statement as Mr Dunks’ paragraph 8.
292.Mr Dunks also said in respect of this, which in my judgment painted a rather different picture to
that portrayed in his witness statement, the following:
“A. The statement again is to verify the integrity of the data once extracted and given. We don't
control what's in the data. Our process is about extracting it and securely passing it over to the Post
Office. It's not our concern of what's in the data. Q. It's not your concern what's in the data?
A.No, it's what we're -- we're process-driven to extract certain types of data for certain requests.
Q. And so whether or not that might match another record or not, or replicate or duplicate or have
gaps, that's not part of your remit, that's not really part of your concern?
A. No, it's not.
Q. I see. So reading that and reading the paragraph 8 as it was in your witness statement, it doesn't
give much comfort to somebody that's then trying to rely on ARQ data as being a gold standard to
compare and investigate anomalies, does it? A. Possibly not.”
293.The reference to “gold standard” was the expression that had been used by Dr Worden in his
report to describe the quality of the audit data that was held by Fujitsu. It should be a very great
concern to anyone tasked with extracting audit data, should that data prove to have gaps and
duplicates in it. This is because gaps and duplicates in the data affects the accuracy of that data. I
consider that such a cavalier approach to whether audit data has gaps and duplicates, as evidenced by
Mr Dunks’ saying that such things were not part of his concern, to be entirely contradictory to a
statement verifying the accuracy of such audit data.
294.My findings in relation to this are as follows:
RLITO000005
RLITO000005
1.Mr Dunks expressly sought to mislead me by stating that there was no “Fujitsu party line” when it
came to the contents of drafting witness statements about audit records for legal proceedings. There
plainly is; it was used in the Fujitsu statements in 2010 and it was used by him in his statement for the
Horizon Issues trial.
2.The passage included in paragraph 8 of his statement is plainly a standard form of words, and reads
as though it is a legally drafted assertion of accuracy. The witness’ assertion that the system was at all
material times working properly, or if it wasn’t, this did not affect accuracy, does not sit consistently
with Mr Dunk’s acceptance of gaps and duplicates in audit data.
3.That Mr Dunks did not really know very much about paragraph 8 of his own witness statement was
confirmed by the way he was puzzled when he was asked about it.
4.However, Mr Dunks’ acceptance of standard wording provided by others (probably within Fujitsu) is
less important than the fact that he originally tried to mislead the court.
5.1 do not accept that he identified the 12 controls in his statement from his recollection. He must
have been working off another document, which he was not prepared to identify. I draw this
conclusion because of the marked similarity with the controls included in Mrs Thomas’ statement
from 2010; the wording of the controls themselves; and the fact he was prepared to mislead me about
the paragraph 8 point.
6.In re-examination he asserted that the statements in his paragraph 8 were true, and that he would
not have signed his statement had this not been the case. I find that he cannot have known whether
those statements were true or not. There is no evidence of any steps he took to check whether or not
they were true or not. I find they were simply standard sentences supplied to him, and I find that he
signed his statement without any independent knowledge of whether they were true or not.
7.1 find that passage in his witness statement simply to be meaningless assertion.
295.He said that he was unaware of Post Office internal concerns that there was “a lack of system
audit trail” in a document of 22 October 2016. I find his lack of knowledge about this surprising, given
his role.
296.1 found Mr Dunks very unsatisfactory as a witness. He was both plainly aware of the Fujitsu
“party line”, or corporate position, regarding the words asserting accuracy of audit data, and he was
very anxious to keep to it, whilst initially denying that there was one. He sought to mislead me about
both his paragraph 8 wording, and the way he had compiled his list of controls. The degree to which
the evidence of fact affects my conclusions on the expert evidence will be dealt with in Part L, Overall
Conclusions.
Mr Godeseth
297.Mr Torstein Godeseth is employed by Fujitsu and is the Chief Architect on the Post Office account.
He had provided three witness statements, as had Mr Parker. After graduating from Oxford with a
degree in physics in 1974, he worked for Rolls Royce as a combustion engineer and then joined the
Royal Navy in 1977 as an Instructor Officer. Whilst in the Royal Navy, he started his career in IT
working in systems programming. It will be remembered that in the 1970s computing and IT was a
relatively young field, and Mr Godeseth has therefore been involved in that industry since its relative
infancy. He left the Royal Navy in 1981 and worked in systems programming and technical support for
RLITO000005
RLITO000005
the IT at Forward Trust Ltd. He joined the Post Office IT department in November 1987. Initially this
was to work on a project to introduce technology into Post Office branches.
298.He was working with the Post Office as a technical advisor when the Post Office and Benefits
Agency procured the Horizon system originally. He remained involved in different roles, including
being outsourced from the Royal Mail IT department to Xansa in 2003, and contracted to the Post
Office to act as the technical adviser interfacing with Fujitsu amongst others. Although an
independent contractor, he worked between 2005 and 2010 with the Post Office as the technical
advisor on IT projects including the change from Legacy Horizon to Horizon Online. He joined Fujitsu
as a full employee in 2010. He described himself in cross-examination as the designer of Horizon.
299.He had a greater knowledge of Horizon Online than of Legacy Horizon, although given his
involvement in Horizon from its early days he had significant knowledge about both. Horizon was
initially, in the late 1990s, called the Pathway project and was a joint effort between the Post Office
and the Benefits Agency, including a way to computerise the payment of benefits to those entitled to
them, through payments to benefits claimants which were to be made through the Post Office by
means of a swipe card method. This project ran from approximately 1996 to 1999 and was to be
provided by ICL (which Fujitsu partly owned and later fully acquired). At some point during the
development of this project the Benefits Agency withdrew, and what had been the Pathway project,
essentially a tri-partite venture (the parties being ICL, the Post Office and the Benefits Agency)
became Horizon, now called Legacy Horizon.
300.Although this litigation is not about the way that the Horizon system was designed, it is about its
functionality and robustness in use, Mr Godeseth said that it was a specific Post Office decision not to
have any dispute button/function for SPMs built into the Horizon system.
301.He said:
“I think the basic argument was that disputes -- we wanted the flow of data through the system as
quickly as possible because that keeps our books tidy and it was an inference that there was always
the -- you had to press a button to take things through, but then you would pick up the phone to NBSC
and say that wasn't right.” (emphasis added)
302.1 accept this factual evidence. It is, however, directly contrary to a major part of the Post Office’s
case in the Common Issues trial, namely that by reason of an SPM accepting items at the end ofa
branch trading period, that gave the branch trading statement the effect in law of being what is called
a “settled account”. I rejected that argument by the Post Office then, unaware when I did so of later
evidence of fact that would emerge of the specific decision taken at the time by the Post Office that
expressly did not include any feature within the Horizon system for a SPM to dispute items with which
they disagreed. My finding in the Common Issues trial is consistent with Mr Godeseth’s evidence on
this point.
303.1 deal with Mr Godeseth’s evidence within this judgment in more detail than any of the other
witnesses of fact from either the claimants or the Post Office. This is because his evidence was
considerably more detailed, and of more direct assistance in resolving the Horizon Issues, than any
other witness of fact. He has been involved in Horizon in both its iterations for 20 years and has a vast
amount of knowledge of its operation. I found the majority of his evidence reliable. I provide my
conclusions regarding him as a witness at the end of this section below at [453] to [463].
RLITO000005
RLITO000005
304.In Legacy Horizon, a messaging system was responsible for storing all data in Post Office
branches and replicating it to data centres. This was called Riposte. Mr Godeseth decribed himself as
being “on the other side of the fence” during this period and he had consulted Mr Jenkins for the
section of his evidence dealing with when Riposte was in use. However, Mr Godeseth did describe
himself as “having a pretty good knowledge of Riposte”.
305.When it was brought in, Horizon Online was aimed, not at improving functionality of Horizon, but
at reducing cost, and re-used (the term put to Mr Godeseth was “recycled”) many application
components of Legacy Horizon. The data was no longer to be held at the branches (on the counters),
it was to be held centrally in the branch database, or BDRB. Accordingly, harvesters were required to
extract transactions from the BDRB. The branch database receives information from different sources.
Data in Horizon Online is obtained from different sources.
306.As an example with the lottery, if a person buys some lottery tickets in their branch Post Office,
the information about a particular transaction starts at the lottery terminal in a branch; goes directly
from there to Camelot; from Camelot it goes to Credence; and then from Credence it goes to the
branch database, which is effectively that information arriving in Horizon. It is then transmitted to the
terminals (in the branch)
and the SPM would see a series of TAs on the terminal in the branch, which that SPM would have to
accept; these TAs would relate to the lottery tickets sold at the beginning of this short history. The
transaction of the purchase of the lottery tickets by this route enters the branch accounts when the
TAs are accepted, and the data is captured by the audit system. This is how lottery transactions work
after what was called the Ping fix. Prior to that it was somewhat less streamlined, although given the
process I have described is the post-Ping fix streamlined version, it is obvious that pre-Ping fix, the
route for the data was even more convoluted.
307.Other data for other transactions for other products goes into the branch accounts from what is
called the basket. That basket is compiled by, as an example, a customer at the counter purchasing
various Post Office and other client products; the SPM or assistant serving that customer and pressing
an icon on the screen (for example for stamps, and other purchases); or for other products, from a
PIN pad, scanning a barcode or weigh scales. The cost of a book of stamps is taken from the reference
data table and the terminal shows the SPM or assistant how much money the total number of
transactions are worth and what the cost to the customer is to be. It is basically the shopping basket
which is put together by the SPM (or assistant) serving the customer and adding, by means of
pressing buttons on the screen, different items to the basket as the customer goes through the items
or products they wish to buy. At the end of this hypothetical transaction, if the cost of the basket is
(say) £21.98, the customer will pay that amount (say by cash) which will be taken by the SPM and put
into the cash amount held in the branch.
308.In Legacy Horizon, the data was held in the branch on the counters in a message store. The
message system called Riposte was how the replicating of the data to data centres occurred. The data
was then stored on the correspondence server message store. This was explained by Mr Godeseth in
the following way.
309.All counter data was held in a bespoke message store (which was part of the Riposte product
supplied by another company called Escher Inc.). This data was replicated within each branch to all
counter positions and from each branch to the data centres where it was held in the correspondence
server message stores. Similarly, any data inserted into the message store at the data centre (for
example reference data or authorisations for banking transactions) would be replicated back to the
RLITO000005
RLITO000005
branch counters. Selected data was then extracted from the correspondence servers to update Post
Office's back end systems.
310.Users with sufficient access permissions could inject additional messages (i.e. data) at the
correspondence server. Any additional messages injected at the correspondence server by users with
sufficient access permissions included information including the identity of the user. That information
would not be visible in the standard audit extracts, but it would be visible in a detailed examination of
the raw audit data. An SPM did not have access to audit extracts.
311.There was some disagreement between Mr Godeseth and internal Post Office documents about
whether the audit data from Riposte was held in Riposte attributable language (his view) or whether
an audit conversion tool was required to convert existing audit data from Riposte to another readable/
searchable format. This ultimately may not matter, and it may be that the author of the internal
document suggesting otherwise did not understand that Riposte attributable language was a
readable/searchable format. This difference in view does not matter for the purpose of resolving the
Horizon Issues. The Riposte software was provided by another entity called Escher. There was no
evidence about the contractual relationship between Fujitsu and Escher, and that does not matter for
the purpose of resolving the Horizon Issues.
312.Mr Godeseth had studied a number of problems in Horizon, and knew there were problems with
Riposte, having seen numerous PEAKs referring to this. He had also been responsible for some testing
(for example potential failures of controls, such as a basket not balancing to zero). He was not aware
of items with the same Journal Sequence Number (or JSN) having been committed to the BRDB, and
agreed that this should not be possible. This is because the whole way the BRDB works (or one of the
principles of it, at least) is that each item committed there should be given a unique JSN. It should not
be possible for two different items to be given the same JSN.
313.Mr Godeseth explained that “the legacy version of Horizon was far more susceptible to
communication glitches” and agreed that there were two improvements made when Horizon Online
was introduced in that respect. These were that the susceptibility to communication glitches was
reduced; and also the quality of the communications infrastructure was improved. It should also be
added that because the storage of data was moved to somewhere not in branch (somewhere which
had not existed in Legacy Horizon, namely the BRDB), and because Horizon Online was an online
system, the communications infrastructure had to change substantially in any event.
314.One internal document from as long ago as 2008 stated: “In addition, in common with many
elderly systems that have been subjected to a succession of major changes, it [ie Legacy Horizon] has
become increasingly difficult to make those changes, and expensive to operate." This was put to Mr
Godeseth:
“Q: Now, that's a fair description of how the system originally was perhaps designed jointly with the
DSS at the beginning and launched and then over the years, between then and 2008, there have been
lots of sort of bolt-ons and additional things that have been changed on the system, haven't there?
A.Yes, I think that's fair. The major one probably in my mind would be banking.”
The reference to the DSS meant the Benefits Agency. In other places this was referred to as DWP (or
Department of Work and Pensions) but all three references - DSS, Benefits Agency, DWP - meant the
department involved in paying welfare benefits. Different documents and references would use
different acronyms to refer to the same entity.
RLITO000005
RLITO000005
315,He explained that a small group of Fujitsu users from the SSC (about 30 users) had the ability to
inject additional transactions into a branch's accounts in Horizon Online, using a designed piece of
functionality called a Balancing Transaction. This was approximately the whole number of personnel
who worked in SSC.
316.He had given information in his first witness statement, namely that in Legacy Horizon, any
transactions injected by SSC would have used the computer server address as the counter position
which would be a number greater than 32, so it would
be clear that a transaction had been injected in this way by someone other than the SPM. This is
important because it would be consistent with the case originally advanced by the Post Office that any
such injections would be entirely visible as having been done externally (ie, not within the branch) due
to the counter number used.
317.However, this important information was simply incorrect, and was corrected both by Mr
Godeseth and Mr Parker in subsequent statements before they were called, and as a direct result of
Mr Roll’s evidence. The information that was incorrect, and therefore had to be corrected, had come
directly from Mr Jenkins. This shows that Mr Jenkins did, in at least one very important respect, give
Mr Godeseth directly incorrect information about the visibility of injected transactions, which not only
could have an effect on branch accounts, and whether this would show (or rather, not show) that the
impact on those accounts had come from injections made outside the branch. Mr Godeseth only found
out the true position when Mr Parker was preparing his subsequent witness statement in the weeks
prior to the commencement of the Horizon Issues trial, in other words in 2019. He had not known that
before. His explanation about this was as follows.
“Q. You were finding out a detail that you didn't know before in quite a controversial area, weren't
you?
A.It was clearly an area that was going to be of interest because of the fact that we were inserting
transactions into Riposte. It was an operational necessity and it was done in a controlled way. I had
believed that the way that transactions were being injected would give them a counter position
greater than 32 because the correspondence servers basically had nodes or addresses which were
above 32, there was a special address for the gateway server, there was a special address for the
extra disc in a single position branch and Lhad basically expected messages to be introduced using a
different counter position and having read a whole number of PEAKs, I can quite clearly see that the
standard practice in Fujitsu was to label something which was being inserted into Riposte so as to
make it as clear as possible that it was not being done -- it was being done as something out of the
ordinary, it was being inserted because of a problem.
So we had techniques for doing that. You could put in an attribute because this wouldn't be visible to
asubpostmaster, I fully understand that, but it would be visible in the audit trail, when you ever come
back to pull out the audit trail, you could put in an attribute to say "This was done under PEAK 75".
si 1 ag - - - L
fully aware that that is the case. It was a better audit than Mr Roll was alluding to when he said that it
was left in a PINICL, because that would have been an audit which is separate from the actual data
that we would be looking at should we ever need to pull stuff out of the audit trail and the intention
The techniques used to make it as visible to the subpostmaster as possible would be to put in
references which referred to a counter that didn't exist in the branch, such as -- I saw a technique
RLITO000005
RLITO000005
described in a number of cases which said put in a -- you know, if you are correcting something for
counter 1, call it counter 11; if you're correcting something for counter 2, call it counter 12. These
things would have been visible to a subpostmaster and the reason that you had to do it that way was
to make sure that these transactions also got picked up and dealt with because these were legitimate
counter numbers.
ignored by systems further down the track. Q. So which were legitimate counter numbers?
A. Up to 32.”
(emphasis added)
318.He also said in relation to Riposte that “Riposte was responsible for actually wrapping the
message that we were looking to insert at the counter, and in doing that, Riposte will tell you the
counter ID, or technically it was a stream, it would pick up the user ID, it would pick up the time, so
this was effectively the envelope which wrapped the payload that we were looking to inject. If there
was no user logged on at the counter then Riposte would introduce a blank user ID and that would be
picked up in later processing.” This therefore meant that if, say, a SPM (or an assistant) was logged on
at the time that the message was inserted at the counter, it would appear next to that SPM’s user ID
(or that of the assistant). It would look as though the SPM had been responsible.
319.This was made crystal clear in different passages of cross-examination:
“Q: You realised for the first time that it was possible to inject or insert a transaction with a counter
position less than 32 when Mr Parker was preparing his second statement?
A.Correct.
Q. And you knew that that was a contentious issue in this litigation, yes? A. Yes.”
320.And further:
“Q. Now, it would be possible, would it not, to use a counter number of 1, or 2, or 3? A. It would.
he SPM. fi A , 1 -
branch. by them or their assistants?
A.Yes.”
(emphasis added)
321.1 consider this to be extremely important evidence, both in resolving the Horizon Issues, and
indeed in the whole group litigation. Its import is obvious. It means that Fujitsu could remotely insert
a transaction into the accounts of a branch using a counter number which was the same as a counter
number actually in use by the SPM (or an assistant). This would appear to the SPM from the records
that they could see (and anyone else looking at those records) as though the inserted transaction had
been performed in the branch itself. This information was only disclosed by Fujitsu (and therefore the
Post Office) in this group litigation in January and February 2019. Even Mr Godeseth, a very senior
person in Fujitsu so far as Horizon is concerned, said that he did not know this before.
RLITO000005
RLITO000005
322.Mr Godeseth was also shown a Fujitsu document which was "Fujitsu restricted" and "Copyright
Fujitsu Limited 2017". The title of the document was "Post Office Account - Customer service problem
management procedure" and its purpose is described as "To describe and document the customer
service problem management process." He was not sure he had seen the document before, and he
was certainly not familiar with its contents. Both experts had seen the document and proceeded on
the basis that what it recorded had in fact been adopted. “The customer” for the purpose of the
document was the Post Office.
323.He sensibly accepted that to have a robust system it was important to make informed
assessments of where problems lie, based on the relevant information that was available, and also
that it was important to capture and track that information in a way that could be readily analysed.
Indeed, Mr Godeseth’s role included awareness that changes being made to the system were
implemented without prejudicing the continued operation of the system. The document itself
expressly defined “problem” as the unknown underlying root cause of one or more incidents. That, to
me, seems a sensible definition. An unknown root cause of even one incident is a problem in a system
such as this one. This is accepted by the document’s definition of “problem”.
324.The document identified the metrics that were to be used within Fujitsu to measure this. Mr
Godeseth agreed that they were professional and sensible metrics. They are stated as follows:
“The following metrics, to be reported monthly, will be used to measure effectiveness of the process
and drive performance of the process and overall service in general:
‘I Number and impacts of incidents occurring before root problem is identified and
resolved
Number of reps«
Number of problem records arising from pro -active actions and trend analysis
Number of changes arising from proactive actions
Percentage of r
Average length
RLITO000005
RLITO000005
Number of inci
I The number of Problem Records arising from Managed Changed (MSC, cp) activities
I The number of Problem Records arising from the implementation of new
services/major releases.”
325.However, when the claimants had sought to obtain reports that would be expected to exist, which
included or demonstrated the reporting against these bullet points which
Fujitsu’s own document stated were required, and which were supposed to be prepared every month,
Fujitsu stated (through the Post Office’s solicitors) that “Fujitsu believes that it does not record
problems in such a way that would allow this to be determined without retrospectively carrying out
detailed analyses” and that it would require “a disproportionate effort and cost” to provide these. Mr
Godeseth also gave written evidence that the reporting system identified in Fujitsu's own document
had not been implemented and that the records did not even exist. The passage in his statement said:
“IT have spoken to my colleague Steve Bansal, Fujitsu's senior service delivery manager, who has
informed me that the Post Office account customer service problem management procedure document
was introduced by Saheed Salawu, Fujitsu's former Horizon lead service delivery manager and that
Saheed Salawu left the Fujitsu Post Office account in around February 2013, before the new
procedure had been implemented. Lunderstand from Steve that Saheed Salawu's replacement did not
wish to implement the changes and therefore the records referred to by Mr Coyne in paragraphs
(emphasis added)
326.Mr Godeseth did not know who Saheed Salawu’s replacement was, although he thought it might
be a Mr Wicks. He did not know what the “previous existing reporting methodology” to which he
referred in his own witness statement in fact was. He could not explain why the report which was said
to have been abandoned, or at least not pursued after February 2013 when Saheed Salawu had left
Fujitsu, was issued for approval in July 2014, over one year later. Indeed, he could not really explain
why the records sought by the claimants did not exist at all. He thought the reporting might be done
at meetings with ATOS, but he could not remember going to one. In fact, his evidence on this was so
very vague as to be somewhat alarming, given his role within Fujitsu. He did not know if there were
any records of the particular type apart from between the years 2014 to 2017, and he only knew that
because Mr Bansal had told him.
327.In another document, he was shown an entry which recorded that First Rate, a joint venture
between the Post Office and the Bank of Ireland dealing with foreign exchange (also called bureau de
change or bureau services), had identified an anomaly over the way that Horizon reversed
transactions that were recorded and polled through to them. This was fixed in something called
counter release 9, but although he said this happened “on his watch” this had not been brought to his
attention.
RLITO000005
RLITO000005
328.In the 2015 Problem Management Problem Review, the Dallmellington bug (which is an
acknowledged and agreed software bug) was referred to, but without identifying how many branches
were affected, for how long they were affected, or the amounts by which their branch accounts were
impacted. He accepted that it was fair to describe the report as “not a particularly rigorous or robust
treatment of recording the problem, its extent and duration and effect.”
329.The following conclusions can be drawn from this evidence by Mr Godeseth. Fujitsu should, had it
been interested in providing a robust system to the Post Office, been collating and reporting
problems, which should obviously have included software
bugs, errors and defects, against the metrics contained in the Customer service problem management
procedure document, or very similar ones. On the Fujitsu evidence before the court, it did not do so.
The discrepancies in the dates of distribution and issue of the document entitled "Post Office Account
- Customer service problem management procedure" and its "Copyright Fujitsu Limited 2017" do not
tally with the account in Mr Godeseth’s witness statement that this process was abandoned or not
pursued when Mr Salawu left Fujitsu in 2013. The dates just do not match up. The document was
issued for approval one year after that gentleman left Fujitsu. Even the evidence that it was not
pursued or implemented after Mr Salawu left was second hand from Mr Bansal, the source of Mr
Godeseth’s knowledge.
330.Had Fujitsu done what its own document, copyrighted in 2017, states was required, the experts in
this litigation could, and probably would, have used those reports. Those reports would have included
comprehensive records of precisely the sort of matters both experts had to investigate in order to
provide their evidence to the court on the Horizon Issues. It is, in my judgment, a stark deficiency in
the robustness both of Legacy Horizon and Horizon Online that such records were not kept by Fujitsu.
This deficiency must inevitably also have an adverse impact upon the quality of information that the
Post Office itself would have had.
331.In the Post Office’s closing submissions, an explanation was given in submission in relation to the
explanation given by Mr Godeseth in his witness statement which I have set out in [325] above about
why the records did not exist, that explanation being that the procedure was not implemented. The
Post Office submitted that it was only section 1.4 of the document that was not implemented, and not
the whole document. This was in paragraph 147 of its written closing submissions, which dealt with
what the Post Office submitted were “errors” made by Mr Godeseth which it submitted (and which the
Post Office accepted) were “significant” and which it also stated had taken the Post Office “by
surprise”. It sought to correct these errors, by way of submission rather than evidence, and one of
those corrections was in relation to the implementation of the document in question. The claimants
objected to this, and said the explanation that was provided was nowhere in the evidence, and the
Post Office should not be permitted to give evidence by way of submissions.
332.Leading counsel for the Post Office was asked about this specific part of the submissions at the
end of oral closing submissions and said that the explanation in paragraph 147.4 of the closing was
“my instructions but, my Lord, it is based upon a previous version of the document and then an
amended version of the document. And I will undertake to give your Lordship the two references.” In
an email of 10 July 2019, which provided some references, further explanation was given which were
said to support the submission that it was only section 1.4 of the document that was not implemented.
The Post Office relied upon the following four points in support of that submission:
1.The passage in Mr Godeseth’s witness statement I have reproduced at [325] above.
RLITO000005
RLITO000005
2.That there were different versions of the document that were disclosed, but only one had been put
in the trial bundle. The first version containing section 1.4 was dated 6 June 2010 and was authored
by Mr Salawu.
3.This indicated therefore that the change was introduced by him. Most of the procedure document
had been in existence for a number of years before that.
4.Earlier versions had been disclosed, none of which contained section 1.4, and none included section
1.4 and none were distributed to or even mentioned Mr Salawu.
333.The following points can be made about this.
1.Witness statements are expected to be factually correct.
2.Each party in this litigation who has submitted witness statements had ample opportunity to ensure
those statements contain accurate evidence before they were served. Further, each witness is asked,
at the beginning of their evidence when they confirm the contents of their statements in their
evidence in chief. Apart from qualifications that were sometimes made, in this case (and in the great
majority) the statements are usually stated to be true to the best of that witness’ knowledge and
belief.
3.If a witness’ written evidence is shown to be incorrect (as Mr Godeseth’s, along with other of the
Fujitsu witnesses, was) then the appropriate way to correct this is either in re-examination or by way
of supplementary evidence. Here, Mr Godeseth was asked about this document and gave evidence in
respect thereof on 20 March 2019. Mr Parker, also a senior witness from Fujitsu, was not called to
give evidence until 11 April 2019. That was, again, ample time for this self-contained point to have
been dealt with by supplementary evidence from Mr Parker, if that was thought by the Post Office to
be important. In any event, there is no reason for it having been incorrect in the first place. Whether a
procedure was, or was not, adopted by Fujitsu internally, is not something that ought to be capable of
misinterpretation in any event, let alone on a subject as important as recording problems in the
Horizon system.
4.1 do not accept a matter such as this is suitable to be corrected by submission “on instruction” in
any event.
5.However, even if it were, I am afraid that my view of information emanating from Fujitsu is, based
on numerous other instances of such information being wrong in the actual evidence, to be somewhat
sceptical of its reliability. It would also be potentially unfair to the claimants to allow this type of
“correction” as they would be deprived of the opportunity of testing the new explanation. Absent some
corroboration of this type of information coming from Fujitsu that supported it, I would not accept it
in this case on this issue.
6.Further, and even if I were to take a contrary view of the points at 1 to 5 above, and accept that the
Post Office should be given the opportunity of correcting Mr Godeseth’s evidence in this way, the
explanation given by way of closing submissions does not assist the Post Office, for the reasons
explained in the following paragraphs, [334] and [335] below.
334.This is for the following reasons. Mr Coyne’s point, to which Mr Godeseth was responding, was
that the Fujitsu document identified metrics and KPIs to measure/control and reduce the risk of
failure to detect, correct and remedy Horizon
RLITO000005
RLITO000005
errors and bugs. He was basing this on section 1.4 of the document. His conclusion was that “From
the above, it is my opinion that Post Office should be aware of all recorded bugs/errors/defects in
addition to those previously acknowledged by them, from the process metrics compiled above.” These
records were sought by the claimants and the Post Office explained that Fujitsu’s position was that
the records did not exist. They were not therefore provided to the experts. Mr Godeseth’s
crossexamination showed that the Fujitsu explanation was wrong (and the Post Office’s closing
submissions accepted it was wrong). The records however were not produced.
335.Finally on this, after receiving the email of 10 July 2019 to which I refer at [332] above, I asked in
an email of 12 July 2019 for hard copies of the documents referred to in the relevant paragraph of that
email (and hence relied upon by the Post Office in this respect) to be delivered to me. This was done,
and a file served on the court containing four versions of the Post Office’s Customer Service Problem
Management Procedure. They are version 0.1 dated 13 November 2007; version 2.0 dated 22 April
2008; version 2.1 dated 6 June 2010; and the version put to Mr Godeseth which was in the trial
bundle, dated 5 September 2017. Given the additional versions produced all pre-date Mr Salawu
leaving Fujitsu in 2013, this means that the points I have identified at [329] and [330] are still valid
ones, and are still in the claimants’ favour.
336.Even on Dr Worden’s evidence and his views on the lower number of bugs in the Bug Table (he
accepts that there are 11, although Mr Coyne’s figure is somewhat higher), there are plainly more
bugs in Horizon than Fujitsu itself was aware of. Over the years of both Legacy Horizon and Horizon
Online, the total number of software bugs, defects and errors in Horizon considered by the experts is
far greater than the number to which Fujitsu have admitted. This is shown in the appendix to this
judgment summarising the number of bugs, errors and defects, and their years of operation. The total
number of software bugs, defects and errors in both versions of Horizon is very important
information. Why Fujitsu chose not to collate and report these in the manner that even a Fujitsu
internal document stated would (or should) be done, is wholly unclear. A comprehensive record,
whether using the metrics identified by Mr Salawu or other similar ones, would surely only assist both
Fujitsu and the Post Office in its administration of Horizon. Given that such a comprehensive record
has not been produced to the experts for their consideration, notwithstanding the express request
made on behalf of Mr Coyne, the court is entitled to conclude that no such comprehensive record
exists.
337.Mr Godeseth was also asked about the BRDB transaction correction tool. This applies to Horizon
Online. The original design document for this stated that it was to allow the SSC to correct
transactions by inserting balancing records to transactional/accounting or stock tables directly into
the BRDB system, and also to audit the changes made. It was not designed to delete or update the
records. However, in cross-examination the claimants’ leading counsel analysed, and put to him, some
careful points on how this tool had been used in practice.
338.The tool works by means of an SQL insert into a variety of tables in the BRDB. SQL is an
abbreviation for Structure Query Language, and is a domain-specific language used in programming
and designed for managing data. It is very widely used. Although Mr Godeseth would refer simply to
“Oracle” as the programme, Oracle Database or Oracle is a database management system which is
produced and marketed
by the US corporation which is also called Oracle. Although that is the name of the company, and SQL
was developed (or invented) before the Oracle Corporation was founded, in this trial the terminology
sometimes became a little shorthand on the part of those IT professionals very experienced in such
RLITO000005
RLITO000005
matters. The Oracle Corporation is very well known, not only in the computing field, but through its
co-founder Mr Larry Ellison who, amongst other achievements, has backed and competed in various
America’s Cup races, including winning the 33rd America’s Cup in 2010 in the trimaran USA 17 with
Oracle BMW Racing. Mr Godeseth, and the experts, are all highly experienced in the field and
shorthand references to well-known languages or applications such as SQL and Oracle are to be
expected. This use of shorthand did not lead to any lack of clarity in the evidence, and I refer to it only
for completeness.
339.Information provided by the Post Office’s solicitors prior to the trial stated that the transaction
correction tool had been used 2,297 times. However, all save one of these usages were said to be
“Type 1” which were for the most part used to unlock stock units and which had no impact on branch
accounts. There was one admitted instance of what was called “Type 2”, which was where a Balancing
Transaction was inserted where it changed transaction data in the main transactional tables. This will
have affected branch accounts. Mr Godeseth had not been involved in drafting that information
supplied by the Post Office’s solicitors, but he knew about it and it matched his view of the intended
use of the transaction correction tool. The 10 database objects, or tables, and sequences into which
the tool was permitted to write are identified in a table at 2.4.1 of the low level design document for
the tool. They are identified in terms of their object names, and were the BRDB operational exceptions
table; the system parameters table; the FAD hash outlet mapping table; the process audit table; the
process audit sequence; the transaction correction tool journal table; the FAD hash current instance
table; the transaction correction tool control table; the branch information table; and finally the
branch operators (or operational) exception sequence.
340.The design document made clear that:
“The following transaction tables have been granted INSERT privileges to OPS$SUPPORTTOOLUSER.
The transaction correction statement is only allowed to insert into these tables.”
These 9 tables were then listed. The process would use a "Transaction file containing an SQL INSERT
statement that creates the required balancing transaction.", which is an Oracle command, and the
SQL INSERT statement would provide a missing half of a transaction, where only one half was
present. As it was put to Mr Godeseth, “the "SQL INSERT" statement effectively goes in and puts in
the missing other side of that transaction?”, a point with which he agreed. The SQL INSERT command
would put that missing other side into the records.
341.The SSC members would log into their own UNIX user, and then change directory and place their
transaction file into the sub-directory. The document then records “They [ie the SSC member using
the tool] will then invoke BRDBX015 manually.
The shell script module will be owned by the UNIX user ‘supporttooluser’.
342.However, the explanation that was provided by the Post Office’s solicitors, of which Mr Godeseth
said he was fully aware, about the use of the tool was as follows:
“Each document is associated with a single SQL statement which made a database correction. There
are two different types of correction shown in the files - the SQL statements for each are of the form:
"1, Update OPS$BRDB.brdb rx recovery transactionsSET_ settlement complete time stamp =..."
And then the "INSERT INTO" command.
RLITO000005
RLITO000005
343.The explanation of Type 1 was “Type 1 reflects the action taken to reset the recovery flag on a
transaction. This will have no effect on branch accounts....”. It was also stated expressly that "Type 2
reflects the action taken to insert a Balancing Transaction where it changes transaction data in the
main transactional tables. This will affect branch accounts."
344.Mr Godeseth expressly accepted, however, that this showed that the command used for what was
being called Type 1 was not an insert command, it was an update command. Mr Godeseth described
himself as not expert on Oracle, but he had a working knowledge of it, and he is plainly more
experienced with using Oracle than most people. I find his experience in Oracle more than sufficient
for him to be able to answer questions on this subject. Mr Godeseth’s evidence shows that Type 1 was
an update - something which the low level design document expressly said the tool would not be used
for. The design document did not contain the necessary database object fields table for performing the
unlocking function, which the Post Office’s solicitors said represented all of the occasions, save one,
when the tool had been used. This was all put carefully and clearly to Mr Godeseth in his cross-
examination, and he agreed with all the points put.
345.This was further confirmed - the mandatory use of insert, and the fact that an update command
was not included in the design intent of the tool - elsewhere in the design document where it stated
under method “The module will read the contents of the input transaction file, which will be in the
form of an SQL insert statement. Only a single insert statement is allowed and (after an optional
introductory comment) it must start with the ‘insert into’ clause." The insert would be in the SQL
script, and Mr Godeseth expressly confirmed this. This is another example of Fujitsu failing to observe
its own design intent for the use of a tool that can have an impact upon branch accounts, and in my
judgment is a factor that goes to the robustness of Horizon as well as the accuracy of data. As was put
to Mr Godeseth, the use of the tool had gone “way beyond” the intent in the design document, and he
agreed. He said that “there is tooling which is based on this” - by which he meant based on the
transaction correction tool - “which has two aspects to it, certainly, so I think I’m agreeing with you”.
It is agreed that there was only one occasion when what was called Type 2 was used. This evidence
goes to the wider point that I have explained, namely Fujitsu failing to observe its own design intent.
346.Mr Godeseth was taken, very carefully, through a specific use of the transaction correction tool in
2010. In PEAK 0195561, a problem was reported to the SSC on 4 March 2010 where a SPM had tried,
on 2 March 2010, to transfer out £4,000 (referred to in the PEAK as 4,000 pds, which means either
pounds (plural) or pounds sterling)
from an individual stock unit into the shared main stock unit when the system crashed. The SPM was
then issued with 2 x £4,000 receipts. These two receipts had the same session number. The PEAK, as
one would expect, records various matters in note form and also uses informal shorthand. However,
the main thrust is that when the SPM did the cash declaration, although the main stock unit (into
which the £4,000 was being transferred) “was fine”, the unit from which the cash was taken “was out
by 4000 pounds (a loss of 4000 pds)”. This is very similar to what Mr Latif said had happened to him,
although the transfer in July 2015 to which he referred was £2,000. The PEAK related to Horizon
Online and was the admitted occasion when the Balancing Transaction tool had been used.
347.Mr Godeseth had obtained the information in his witness statement dealing with this from Mr
Parker. He also said he thought it was puzzling, and he was not sure that the PEAK “would be a totally
accurate reflection of what happened” and was “an interpretation put together by a developer who is
investigating the problems.” He suggested the original log files would be better (by which he meant
better sources of information) and still exist, but they were not before the court. These log files would
RLITO000005
RLITO000005
be held by Fujitsu, so any failure to have them before the court was no fault of the parties in this
litigation, and that can only be laid at the door of Fujitsu. If a senior Fujitsu witness thought they
would be better records, they should have been accessed and evidence about them could have been
given directly. Certainly Fujitsu witnesses could have given evidence that relied upon them, had they
so wished.
348.The PEAK records the following:
“After discussion with Gareth Jenkins, the suggested correction is to negate the duplicate transfer out
by writing 2 lines to the BRDB_RX_REP_SESSION and BRDB_RX_ EPOSS_TRANSACTIONS tables,
with:
1)Product 1, Quantity 1, Amount 4000.00, Counter mode id 7 (TD
2)Product 6276, Quantity -1, Amount -4000.00, Counter mode id 7 (TD)
This should be done using the Transaction Correction tool. An OCP approved by POL will be needed.”
349,This concerns Horizon Online, due to the references to OCP and OSR, and shows extensive efforts
being made at Fujitsu to try to understand why this has occurred, and general doubt that the efforts
to reproduce the fault in testing has worked. Some at Fujitsu wanted the priority to be downgraded;
other entries resisted this, shown by some of the entries such as:
“What is missing from this Peak is an explanation of the events in terms of the requests, how they
were ordered and when any was committed. Only then can we qualify the priority. The assumption is
that we have a fix. The facts are -
1; A settlement request to timed.
2: A retry of request timeout occurred.
3: According to the DB entries both later succeeded.
Now unlike other reconciliation Peaks this stands out.....
We can’t reduce the priority unless we understand what is going on.”
350.The PEAK is lengthy and demonstrates a degree of frustration within Fujitsu at their own failure
to get to the root cause. One entry on 24 March 2010 states inter alia “.....this shouldn’t have
happened”. One might observe that this is stating the obvious. A later one on 25 March 2010 states
“advised on the latest update in the call she states she was going to check if it was the same thing
that happened before. [The Post Office] are want to know why this has happened? - why does it keep
happening? - can you advise on this.”
(emphasis added)
351.One of the reasons or concerns expressed in the PEAK is that JSN entries - which should be
unique and in respect of which no duplicates should be permitted in the journal table - appear to have
occurred. Mr Godeseth confirmed that this should not have happened. Duplicate entries should not be
permitted in the database. The documents showed that this did occur. Mr Godeseth said that the “JSN
is part of the primary key into the message table, I cannot see how an Oracle database would allow
that to happen” and “that would have to be a bug in Oracle, and it was certainly not the bug that I was
looking - it was not the bug that I was looking at for the red alert”. He also said: “If this in fact
RLITO000005
RLITO000005
happened as written down here [ie in the PEAK] then it would have had to have been a bug in Oracle
and I certainly don't remember any such bug.”
352.The PEAK also shows computer script being used, both in an attempt to correct, and understand
and analyse, what occurred. On 23 April 2010 an entry states:
“T think we have done as much as we can on this one. In conclusion although we haven't been able to
totally explain the behaviour, the risk of this type of PEAK occurring again has been minimised in live
due to a change of behaviour in the BAL with respect to transactions.
duplicate of PC0194893”.
‘his may now be marked as a
Rather concerningly - but entirely consistently with what I consider to be Fujitsu’s general approach
to closure codes - the following entry appears that same day in another entry stating that the “Defect
cause” is “updated to 40 - General - User”.
353.However, later the same day the following entry is made:
“I am sending this call back with Response Rejected.
Closing a call as ‘Duplicate Call’ results in a black mark against me. It basically means that I should
not have sent the call over since the same problem has already been sent over in a previous call.
PC0195561 (duplicate transfer of 4000.00 cash) may have been caused by the same underlying fault
as PC0194893 (banking reconciliation), however I could not have been reasonably expected to link the
2 calls and take the decision that it was not necessary to send PC0195561 over for further
investigation.
Please close this call with category ‘Advice After Investigation’
[End of Response]
Response code to call type L as Category 52 -- Pending -- Response Rejected
Response was delivered to Consumer.”
354.Eventually KEL cardc262s was updated with the information in the PEAK on 4 May 2010 and the
call was closed. One of the final entries states that day “Category 60 - Final - S/W Fix Released to Call
Logger”. “S/W fix” means software fix. The call is closed the same day. This is 2 months after the
incident. The PEAK also referred to other PEAKs. Mr Godeseth was asked about those too. PEAK
0195962 records that the transaction tool has “been used in live”. That led to an Operational Change
Process or OCP 25882 and stated:
“Due to a system fault, the branch did a Transfer Out of £4000 and a corresponding Transfer In of
£8000
Justification: Correct a loss of £4000 at the branch due to a system fault
When: Planned for 10/03/2010 16:00 with a duration of 30 minutes
Extra detail: The Transfer In details were incorrectly doubled up when they were written to the BRDB.
This needs to be corrected using the Transaction Correction tool.”
(emphasis added)
355.The actual printed address of the OCP document is as follows:
RLITO000005
RLITO000005
http://deathstar/SSC2/SSC_OCP/viewocp.jsp?0CPRef=25882&SID=f630192836137
356.This may show that the folder in which the document was kept was named by someone with an
interest in a particular series of well-known films, or it may not, and this does not matter. It does,
however, show that it was plainly generated or kept by the SSC.
357.Mr Godeseth agreed that the OCP showed at least one use of the transaction correction tool for
one balancing transaction. Part of Mr Godeseth’s explanation for what had occurred, even though
there had been two duplicate JSN entries, was as follows:
“Tn this situation the symptoms, as I'm reading them, are that because there was a bug in Horizon and
this is in the new system, it was pretty early days of the Horizon Online, because there was a problem,
. . , Li fai
MR JUSTICE FRASER: I understand that. I think what Mr Green is putting to you is that this shouldn't
have got past the journal filter.
A. It's a bug, so certainly the way that the system should have worked, a JSN -- the same JSN coming
up would be just a simple repeat of the message for -- because there was some sort of glitch.”
(emphasis added)
358.His final evidence in cross-examination on this was the following:
“A, Somebody may have been trying to look for a duplicate JSN entry. I can't really comment on what
the guy was doing at the time who was trying to investigate. I am simply asserting that barring an
Oracle bug, which would have been huge, you cannot have two entries with the same JSN.”
359.In my judgment, the Horizon Issues are sufficiently wide and costly to resolve concerning the
Horizon System, both Legacy and Online, without widening them to include some attempt at including
specific bugs within Oracle too. The Oracle Corporation is not even a party, and has had no
opportunity to provide evidence or submissions. I simply recount Mr Godeseth’s evidence as he
attempted to explain the PEAK. He did, as can be seen, refer to potential bugs in Oracle as an
explanation for something that he believed to be puzzling. He accepted it was a bug and/or “a glitch”.
In any event, other than Mr Godeseth’s evidence expanded in re-examination that he was involved in
2010 when establishing himself in Fujitsu (and in this he referred to an Oracle bug, which involved
nodes in the database going down) there is no evidence before me that there were any bugs in Oracle
in respect of this doubling up, and I reject the suggestion that this can be laid at the door of Oracle.
What this PEAK shows, in my judgment, is that Fujitsu itself, with a great deal of time to prepare for
the Horizon Issues trial, simply could not explain then, and cannot explain now, what caused this
event to occur. That is relevant to my consideration of the Horizon Issues, and in my judgment shows
that there was in that respect a bug, error or defect within Horizon. It also shows that there were
those in Fujitsu who wished to close the call with the error put down to the user, as shown by the
“Defect cause” being “updated to 40 - General - User”. There is nothing in any of the material on this
incident to suggest it was in any way caused by the user.
360.He was taken to another PEAK, PC0175821 which was dated 19 February 2009. This is during the
period of Legacy Horizon.
361.PEAK 01320275 was in relation to an incident on 21 December 2005 which showed a SPM, at a
five counter site with seven stock units, had rolled over a particular stock unit called BB. He rolled it
RLITO000005
RLITO000005
over in what was called “an effectively empty state” but having declared the correct amount of cash,
and also adjusting the stock levels to the correct volumes, his branch account showed “a gain of
approximately £18,000”. This was an obvious impact upon his branch account, and it was
investigated. One entry in the PEAK stated:
“We are unable to correct the system figures safely. We can however provide accurate figures for what
should have been in the Final Balance for BB, to enable POL [ie the Post Office] to make the
correction perhaps by using a Transaction Correction.
POL need to make a decision on whether they are able to correct the problem in this way, however we
do not see any other alternative. Corrective action should be taken before 11th January when the
branch is due to roll into TP10.
The cause of the problem is unknown and is under investigation.”
(emphasis added)
362.This accepts in terms that there is a problem, and accepts that its cause is unknown. The
suggestion by Fujitsu later in the PEAK on 3 January 2006 was to generate a negative figure
effectively by changing the relevant data in the messagestore to try and cancel out the £18,000 which
the SPM had showing as a positive figure in his branch accounts. It is obvious to me that this would
only remedy the effect of the problem, rather than identify what had caused the discrepancy in the
branch accounts in the first place. However, the entry is illuminating in any event:
“If we get to the problem before the office is rolled we are able to change objects in the messagestore
to reset the stockunit back to the CAP (TP) rollover trailer. The PM can then rollover. PM should get a
. .
We don't want to be having to do this as making manual changes to the messagestore is open to error
and each time we have to seek authorisation from POL to make the changes.
If we get to the problem after the office is rolled (as in this call) then we are unable to correct the
system figures safely. Its not been decided how we get the PM sorted out.
All in all, we want this fixed asap.”
(emphasis added)
363.Mr Godeseth accepted that generating the artificial negative figure in the message store could be
done, but said he was “not involved interfacing with Horizon” at the time and was not at the
operational level shown in the PEAK. He was not involved in obtaining authority from the Post Office
for such matters. In my judgment, the way that Fujitsu anticipated resolving this, by artificially
seeking to engineer “a large shortage which cancels out the large gain” is not exactly grasping the
nettle. It would have the effect of disguising why the large gain had occurred in the first place, in
other words a bug, error or defect.
364.Another PEAK 0152014 dated 10 December 2007 showed a problem that was described in the
PEAK as “POLFS Incomplete Summaries Report” which related to a foreign currency transaction. At
this time, £484 sterling could buy $1,000 (the PEAK did not specify US or Canadian, but the
associated OCP identified US$ specifically) and the PEAK recorded that:
“This is due to a single SC line written for $1000 (£484) with no settlement in the middle of two RISP
transactions.
RLITO000005
RLITO000005
On call PCO151718 the harvester exception was corrected and now the transaction for the day don't
zero, hence this issue with the incomplete summaries report.
Am currently retrieving the messagestore for this branch, we will then be inserting a new message on
the counter to remove the effects of this. OCP 17510 has been raised.”
365.Again, this was being done to correct the effect of what had occurred, rather than discovering
what had led to the issue in the first place. In two places in the PEAK, the following entry appeared:
“«*Again, this may also have caused a receipts and payments error, can EDSC please confirm whether
this is a gain or loss at the counter and the amount.**”
366.Mr Godeseth had been asked to look at this PEAK in advance (his evidence went over two days
and this was agreed by the parties), and therefore he had been given a chance to familiarise himself
with its contents, and those of an associated KEL. The PEAK stated "Worth noting that the branch did
not have any issues with the mismatched transactions because this was fixed before they did the roll.
The branch is not aware of this and it's best that the branch is not advised." (emphasis added)
367.When it was put to Mr Godeseth that this showed that on not all occasions were SPMs advised of
impacts on their branch accounts of particular problems, he said that was “a fair inference”. An
inference is a common sense conclusion, and I agree with Mr Godeseth that it is indeed fair to draw
that inference. He did however then add “I think I would say that Post Office were well aware of this
and I would argue that it's a Post Office decision whether or not to tell a subpostmaster.” This shows
that Mr Godeseth was not immune from arguing the case or making suggestions not supported by the
evidence. Nothing was produced by either side to show that the SPM in this case was told. Certainly
the person at Fujitsu who wrote “....it’s best that the branch is not advised” cannot have shared Mr
Godeseth’s view at the time. Mr Godeseth did however also say, in my judgment again importantly,
that “in the background to this there was a dialogue with [the] Post Office” and he agreed with
counsel who put to him that the “Post Office would have been aware of what was being done by
Fujitsu”.
368.The associated OCP however, whose subject was “write corrective bureau message for FAD
183227” - bureau meaning foreign currency - gave a great amount more detail. It stated:
“A single SC message 183227-7-1101211 was written in error on 26th November at 12:43:17, selling
1000 US dollars, with no corresponding settlement line. To remove the effects of this message at both
the branch and on POLFS, we will insert a new message to negate the effects of the original message.
Justification: If the change is not made in the counter messagestore (before the stock unit is balanced
on Wednesday), the branch will have an unexpected gain of £484 (or thereabouts - depends on
exchange rate), and a receipts and payments mismatch. This gain would have to be resolved at the
branch. There would also be an inconsistency between the branch and POLFS to be resolved. By
correcting the problem locally, the branch may not be aware of the problem, and there will be no
inconsistency between the branch and POLFS.”
And:
“The message will include a comment to show it has been inserted to resolve this problem (this will
os
This change will first be applied to a copy of the messagestore within the SSC environment, and the
stock unit then rolled over to make sure there are no unexpected consequences.
RLITO000005
RLITO000005
Nei 7 il bei POLFS
Gary Blackburn (POL) is already aware of this issue.”
(emphasis added)
369.This shows that the Post Office did know, as Mr Godeseth had said. It also shows that the branch
would not be able to see what was done. It also showed that the intention was to reverse, by means of
a specific insert into the messagestore by Fujitsu, an entry or data in order to correct a discrepancy in
the branch accounts caused by an error or problem within Horizon Legacy. This was approved by the
Post Office as shown by a later entry. The OCP had been raised by Anne Chambers. The “extra detail”
- effectively what the message needed to contain or achieve - was:
“Extra detail: The original message had ProductNo:5129, Qty:1, SaleValue:484, PQty:1000. The new
message will have Qty:-1, SaleValue:-484, PQty:-1000, with other attributes (including exchange rate)
as before.”
370.The specific message that was inserted into POL FS, as shown in the OCR, was as follows:
“Extra detail: This OCR is being raised so that EDSC is authorised to amend the txn details for branch
183227 and insert these into tps_pol_fs_ summaries incomp table on the host.
Comments
Andy Keil (POA SSC Support) wrote at 12/12/2007 15:07: Updated POLFS feed for branch 183227
product 5129 mode SC with SaleValue=1014.73 and PQty=2080”
Mr Godeseth agreed that this showed “just over $2,000 being inserted in the Post Office system”.
371.However, the PEAK clearly demonstrated that the fix or corrective action that had been applied
had simply made matters worse. This is because it stated for 14
December 2007, two days after the message had been inserted by Andy Keil of Fujitsu, the following:
“The counter problem which caused the first issue has been corrected by inserting a message into the
messagestore, for equal but opposite values/quantities, as agreed with POL (OCP 17510).
As a result of this corrective action, the net effect on POLFS is zero, and POLFS figures are in line
with the branch. POLMIS received both the original message and the corrective message.
Once the problem was corrected, there should have been no impact on the branch. However it has
been noted that the stock unit BDC had a loss of $1000, which was generated after the correction was
made. We have already notified Gary Blackburn at POL (email attached). This appears to be a genuine
(emphasis added)
372. On 17 December Anne Chambers also stated in the PEAK:
“Summary for development:
A single SC line was written for $1000 (£484), with no settlement, in the middle of two RISP
transactions.
RLITO000005
RLITO000005
The line was missing some AdditionalData so it wasn't harvested properly, but the main problem was
the lack of settlement. POL authorised us to insert an equal but opposite message, to prevent a
discrepancy (in theory anyway) and to avoid problems on POLFS. Please note that this is exceptional
and must not be seen as a convenient avoidance in place of a fix.
Subsequent investigations showed that a second packed pouch barcode had been scanned, before the
receipt for the first had finished processing. See the audit log / messages on counter 7 26th Nov at
12:43 / 12:44.
PC0147357 is already with development for what I think is a similar problem - Mode reverts to what it
was prior to RISP.
Please also see update on that call.
Routing to EPOSS-Dev via QFP.
[End of Response]
Response code to call type L as Category 40 -- Pending -- Incident Under
Investigation.”
(emphasis added)
373.The PEAK was finally closed on 17 April 2008, the same day that the “defect cause” was “updated
to 14: Development - Code”.
374.There are two important points to make in relation to the entry in this PEAK which I have quoted
at [371] above. Firstly, the discrepancy in the branch accounts, which
was, after the “corrective action”, in deficit, was broadly in the amount of the difference between the
two corrections that were made, namely the one by the OCP and the OCR. These corrections should,
in my judgment, have been in the same amount, as one relates to “front end” (or the branch
messagestore) and the other “back end” of the accounts so far as Horizon is concerned. The statement
in the PEAK that “this appears to be a genuine loss at the branch, not a consequence of the problem
or correction” is simply insupportable. That is a statement made by Anne Chambers, which in my
judgment flies in the face of the documents. I am supported in my conclusion by two things. Firstly,
Mr Godeseth’s evidence, when it was put to him that this was a fair possibility when he said “having
read this PEAK in more detail overnight then yes, clearly that is what appears to have been the case.”
I accept that evidence. It is supported by the documents and by the text of the two different
corrections. Secondly, plain and obvious common sense.
375.The second important point is that the PEAK makes it clear this, or something very similar, has
happened before. It states “PC0147357 is already with development for what I think is a similar
problem - Mode reverts to what it was prior to RISP.” The reference to “development” is the
department within Fujitsu where software is written to correct issues with the system. This is
supported by the closure of the PEAK and defect cause stating “Development - Code”.
376.There is an associated issue that arises on this PEAK, the number of which I will again recite for
clarity, PEAK PC0152014. I will deal with this here, rather than in the section dealing with Mr Coyne’s
evidence or within the Technical Appendix (although the latter place could be a more suitable home
for it). This is because it goes to the weight to be given to Mr Godeseth’s evidence.
RLITO000005
RLITO000005
377.A detailed explanation was put to Mr Coyne on this very matter, which occurred on the third day
of his evidence on 6 June 2019 and went from page 103 of the transcript, ran until page 115 when the
OCP was then put (which went to page 120) and then the OCR which was put between page 120 and
126. Leading counsel for the Post Office stated “I can tell you on instruction what that figure means”;
“So on instructions I can tell you than the sale value of £1,0114.73 and the quantity of 2,080......” and
“I appreciate that I’m telling you this on instructions because I have to say nobody knew that this
suggestion was going to be made until it was put to Mr Godeseth....” It was made clear that there was
what was being put to Mr Coyne as a detailed explanation of what had occurred, was “on instruction”,
and it was nowhere in the evidence.
378.Mr Godeseth was cross-examined on 20 and 21 March 2019. When Mr Coyne was cross
examined, about 2 2 months later, a great deal was put to him “on instruction”, as I have identified. It
was a detailed factual explanation that was not included in any evidence at all. Firstly, the OCR that
went with it showed that “This OCR is being raised so that EDSC is authorised to amend the txn
details for branch 183227 and insert these into tps_pol_fs_summaries_incomp table on the host.” This
was required precisely because Mr Godeseth was correct, in my judgment, in his understanding of
what had occurred. I accept his evidence on this. Secondly, the explanation put to Mr Coyne “on
instruction” was (broadly) that there were two problems. A careful attempt was made to explain to Mr
Coyne how (basically) Mr Godeseth had misunderstood, and to downplay the effect of what had
occurred.
379.1 do not accept that this subject was capable of being corrected - even if that is what the
documents showed, which in my judgment they plainly did not - in the way adopted by the Post Office
by attempting to give evidence by way of submissions from counsel “on instruction”. This deprived the
claimants of any ability to challenge the explanation, if explanation it was, by cross-examination.
Further and in any event, there was ample time for the Post Office to have provided a short witness
statement from a Fujitsu witness, such as Mr Parker (who was not called until 11 April 2019) dealing
with this, if that was the evidential explanation. If that had been done, permission for that statement
would have been required, but given the matter arose from something Mr Godeseth was asked about,
that application could have been dealt with at the time and if this genuinely was evidence that the
Post Office could not have known in advance would be required, it would have been allowed in. Even
supplementary questions in chief could have been used as a method of obtaining actual evidence on
this point. It simply is not procedurally acceptable, or fair, for evidence of this nature to be given by
way of submission “on instruction”. However, and in any case, I have considered the Post Office’s case
on this notwithstanding these points, and I find that the documents do not substantiate the
explanation given on Day 16 to Mr Coyne when he was being cross-examined.
380.The explanation did not make sense on the face of the documents, and they did not support the
points that were being put “on instruction”. Firstly, as Mr Coyne pointed out, the PEAK identified that
a message was to be inserted into the messagestore. None of the documents put to Mr Coyne “on
instruction” identified what that message was. This point was made by Mr Coyne during the cross-
examination:
"Q. Yes. So what they are talking about here, what this change here is, is a change being made by
using the TIP repair tool into the TPS, correct?
A. Right, so this doesn't relate to the creation of the message then.
Q. It doesn't relate to the branch accounts, Mr Coyne, does it? This is an OCR which involves an
exercise -- well, the use of the TIP repair tool to change data that is in the TPS system, yes?
RLITO000005
RLITO000005
A. Yes, but the PEAK refers to the insertion of a message into the messagestore.”
381.The critical point, that the explanation skipped over, was what the message that was inserted in
fact was, in actual terms. Had that message been produced, it could have been looked at, and
deciphered. Understanding such a message is not an insuperable task beyond mortals. The second
point is that the PEAK refers to a KEL, namely KEL obengce3120K. At the beginning of the trial, I
asked for hard copies of all PEAKs and KELs put to witnesses to be prepared in hard copy too, so that
Thad a working file that I could use during the trial for PEAKs and KELs already referred to (in
addition to the three OPUS trial bundle screens that I had to work from). This was done. That KEL is
at tab 21 of volume 2 of the file entitled “PEAKs and KELs referred to in Days 5-8 and 12”. The KEL
states the following, and I will reproduce the text in full, including the relevant table references.
“KEL obengc3120K
Title:
Not harvested: Could not update database: Updating table TMS_RX_BDC_ TRANSACTIONS,
ORA-02290: check constraint
(OPS$TPS.BT23B_BUREAU_REGION CHK) violated.
Summary:
Harvesterd did not harvest message: Could not update database: Updating table
TMS_RX_BDC_ TRANSACTIONS, ORA-02290: check constraint
(OPS$TPS.BT23B_BUREAU_REGION_ CHK) violated
Symptoms
The Harvester tried to insert into the BdC table TMS_ RX BDC TRANSACTIONS a message with
missing ‘Blackbox’ data. This caused Oracle [ORA-02290] to detect a check constraint
(OPS$TPS.BT23B_BUREAU_REGION CHK) violation, hence the appearance in the
TPSC254.<br><br>Harvester exceptions do not normally cause entries on TPSC250, but this one
seems to. <br><br>TPSC257 displays the contents of the TPS Incomplete Summary table where the
summarisation program detects that the total harvested transactions for the day don’t NET to zero.
Problem
Due to some data in the ‘Blackbox’ or AdditionalData of the messagestore not being populated, the
bureau product was missing key data
including<br>BUREAU_REGION<br>MARGIN<br>MARGIN_PRODUCT<br>EF
FECTIVE_EXCHANGE_RATE<br>and so when the Harvester attempted to insert the message into the
BdC table TMS_RX_BDC_TRANSACTIONS, however Oracle [ORA-02290] detected a check constraint
(OPS$TPS.BT23B_BUREAU_REGION CHK) being violated.
Solution - ATOS
Warning: Check the AMOUNT (or sum of AMOUNTS) for the branch on the TPSC254
report.<br><br>If the total AMOUNT is NOT ZERO, and the branch is NOT on TPSC257 (same day),
<br>or the total AMOUNT IS zero and the branch IS on TPSC257 (same day) then there is probably a
, . sc . -
RLITO000005
RLITO000005
settlement, Use KEL <ahref=kel. view kel.jsp?KELRef=acha3159Q>acha3159Q</
a>.<br><br>Check the messages anyway, just to be on the safe side. If the session was settled
properly....<br><br>MSU must raise an OCR so that SSC can use the TIP Repair Tool to populate the
missing columns, using values from another transaction for the same currency, same branch, same
day if possible.<br><br>NOTE: By repairing the txn, the TPS_POL_ FS SUMMARIES INCOMP will
be corrected automatically after that txn has been successfully harvested. <br><br>*** In the past.
. ine SC . i
been corrupted. However I think it may not be a genuine sale, but is related to an attempted pouch
reversal (this is certainly the case for the more extreme instances where the settlement is missing). If
the SaleValue and the PQty have opposite signs, and there is no reversal (mode ER) for the
transaction, this may have caused a loss or gain at the branch which they can’t resolve themselves.
Let Anne know if there is another occurrence like this.
(emphasis added)
382.This KEL shows that there was a potential impact on branch accounts - it states “this may have
caused a loss or gain at the branch which they can’t resolve themselves”, which is very clear. It shows
that the explanation put to Mr Coyne “on instruction” is, in my judgment, completely wrong. It also
shows that it has happened before, as “in the past we have just repaired these”. I would also note for
completeness that this KEL is not dealt with at all by Mr Parker in the table accompanying his 1st
witness statement. Two KELs from Ms Obeng are, namely CObeng1123 at number 24 in the table at
E2/11/40; and obengc5933K at number 31 in the table at E2/11/43. However, this one is not.
383.1 accept Mr Godeseth’s evidence on this point, which is wholly consistent with the documents, in
particular the PEAK and the KEL. The text of those documents shows that a message was intended to
be written into the messagestore; the actual message used is nowhere available; the effects of all this
clearly being an impact on branch accounts.
384.A further PEAK was PC0175821, which was dated 22 February 2009. This also related to foreign
currency transactions. This PEAK related to two problems, or a problem with two elements to it. It is
at F/485/1 in the trial bundle, and it clearly and expressly refers to the same KEL, obengc3120K.
385.The two aspects were explained by Mr Godeseth as a change using the transaction repair tool to
“get the feed into POLFS correct” and also “a change to the messagestore to get the branch aligned.”
The repair “had to go two ways” because the branch and the POLFS had to be aligned. If they were
not, there would be a discrepancy in branch accounts. This showed the two problems, or two sides to
the same problem. It was stated as follows in the PEAK:
“The first is where all five SC transactions missing core data as described in the above-mentioned
KEL." This means obengc3120K, referred to at the beginning of the PEAK.
And the “Second is absence of equal but opposite 2 (ie settlement) lines. See PC0152014 for a similar
problem and how problem was resolved."
“There are two sides to the problem relating to these txns. The first is where all five SC txns missing
core data as described in the above mentioned KEL. Second is absence of equal but opposite [i.e.
settlement] lines. See PC0152014 for a similar problem and how problem was resolved.
For the first problem, I have used the TRT [the transaction repair tool] to insert the missing data i.e.
Region, Margin, Margin Product and EffectiveExRate.”
RLITO000005
RLITO000005
The reference to “txns” means transactions. PC0152014 is the very PEAK referred to in [376] above.
386.Mr Godeseth agreed with both of those points in the PEAK as they were put to him by the
claimants. Basically, half the data was missing. The transaction repair tool or TRT was used for the
first problem.
387.Numerous other PEAKs and some KELs were put to him with very specific details. Both Gareth
Jenkins and Anne Chambers were involved in raising, or investigating, a great many of them and they
are referred to in the text of both classes of documents. In all of them, Mr Godeseth’s answers to
questions were, in my judgment, consistent with the entries within the documents, and also consistent
with the claimants’ case on the Horizon Issues. In particular, those in the PEAKs dealing with foreign
currency which I have explained above was clear, and settlement lines were missing, which Fujitsu
sought to correct by injecting lines into both the messagestore and into POLFS.
388.Finally, a series of questions were put to him demonstrating what are called APPSUP permissions.
This is a very powerful permission (which really means level of access) and PEAK PC0208119, which
runs from February 2011 onwards into 2012 and beyond (the final entry being in 2015), made the
following relevant comment:
“As per the previous PEAK comments, the role ‘APPSUP' is extremely powerful and should only be
used under extreme circumstances and under MSC supervision. As such the Branch Database design
was that 3rd line support users should be given the
'SSC' role, which is effectively read access, ie. 'select_any table + select_catalogue'. SSC team
members should only have to [access] BRSS for normal support investigations, unless the information
has not replicated in time. SSC should only given the optional role ‘APPSUP’ temporarily (by Security
Ops authorisation/emergency MSC) if required to make emergency data amendments in BRDB Live.
It is a security breach if any user write access is not audited on Branch Database, hence the
emergency MSC for any APPSUP role activity must have session logs attached under the MSC. Host-
Dev previously provided scripts, such as the Transaction Correction Tool, are written to run under the
SSC role and also write to the audit logs.
SSC users created on BRDB should only have the SSC role, and the user creation script should be
amended by Host-Dev to reflect this. A separate script giving/revoking emergency MSC access via
APPSUP can be delivered, logging this to the hostaudit directory. In parallel Host-Dev should
investigate any Host-Dev delivered script to ensure they are all executable by the SSC role. SSC
should investigate any of their own scripts to ensure they have sufficient permissions under the SSC
role, taking into account they should primarily perform their work on BRSS.
Any day to day scripts should not access BRDB directly. Any scripts needing more than the SSC role
should be questioned, except those that would run under MSC APPSUP. Once the investigation is
complete, all BRDB SSC users with APPSUP should have the role removed by ISD (via MSC) and
ensure they do have the SSC role.
If anyone is in disagreement with the above course of action, then I'll set up a meeting with yourselves
and Security when I'm down in BRAO1 next week.”
(emphasis added)
389.This entry was from Andy Beardmore, Senior Software and Solution Design Architect Application
Services. The experts are agreed that the APPSUP role would, effectively, permit anyone who had that
RLITO000005
RLITO000005
permission to do almost anything on Horizon. It was available to 3rd line support at SSC, the level at
which Mr Roll was employed by Fujitsu. This PEAK further substantiates the evidence of Mr Roll and
is consistent with it. APPSUP was described by Mr Parker as “the more technically correct name for a
type of privileged access to the BRDB”. It is a very powerful permission.
390.A later entry in the same PEAK states:
“The Business Impact has been updated:
1.Cost: There is currently no "cost" to this issue. The users affected have more access than is
required.
2.Perceived Impact: The customer is not aware of this problem or change.
3.Scope: No actual impact/incidents of problems relating to this issue have been experienced yet (and
not expected)”.
(emphasis added)
The customer was the Post Office. The “issue” referred to was Fujitsu SSC users having had far
greater access, namely the powerful APPSUP permissions, on more than the intended basis, which
was supposed to be in extreme circumstances only and temporarily. This is shown in an entry for 1
February 2011 by Mark Wright who stated "....SSC users have the APPSUP role” and “When we
created SSC users for BDB/BRS etc. we used “appsup” as that is what SSC have always been and
what they migrated as on Horizon databases.” (emphasis added)
It is clear that all the members SSC had always APPSUP, but were not supposed to have this powerful
role all the time.
391.Anne Chambers said, in an entry on 1 February 2011:
“Unfortunately development write their scripts explicitly to use SSC. So I think we're stuck with it
unless they deliver new scripts (which would not be a popular or quick option).
When we go off piste we use appsup. Can we have both??”
(emphasis added)
392.Mr Godeseth said that by “off piste” he was very confident that she meant “having to fix a
problem that was not catered for by a script that is available to these people” by which he meant SSC.
It was clear that SSC were using APPSUP and wished to continue to do so. It was equally clear that all
the members of SSC were not supposed to have it.
393.The Post Office’s auditors for the year ended 27 March 2011 were Ernst & Young (“E&Y”) a very
well-known global firm. In the Management Letter for that year, E&Y had identified the following:
“The main area we would encourage management focus on in the current year is improving the IT
governance and control environment.
Within the IT environment our audit work has again identified weaknesses mainly relating to the
L envi POL's thi Ir iors.
Our key recommendations can be summarised into the following four areas:
RLITO000005
RLITO000005
Improve gover!
(emphasis added)
394.The “third party IT suppliers” referred to were Fujitsu. Whether it was the work performed by
E&Y that had led to the decision at Fujitsu that the APPSUP permissions were not being sufficiently
restricted or controlled, or something else, does not much matter. A company such as E&Y would not
lightly refer to “again identified weaknesses” in the Management Letter for a particular year unless
that conclusion had been reached after a high degree of professional and in depth investigation, of a
type that even a lengthy adversarial trial such as this one cannot hope to replicate.
395.Indeed, the Post Office’s solicitors confirmed in a letter in this litigation dated 11 February 2019
that Fujitsu SSC users who were identified as PERSON-POA UNIX (also sometimes referred to as POA
Unix users) had a range of privileges and were granted the UNXADM role. This may be a descriptor
for Unix (hence UNX) Administrator (hence ADM) but this does not matter. The UNXADM role
contains, to quote from the letter of 11 February 2019 itself, the “DBA role” which is stated to be:
“This is an Oracle supplied role for use by database administrators (DBAs). Lots of privileges are
. ae . {the B
(emphasis added)
396.POA Unix users also are:
“Granted the ability to execute the following executable functions:
(a) OPS$SUPPORTTOOLUSER.PKG BRDB TXN_CORRECTION —framework to
allow the user to insert fully audited balancing records into a BRDB transaction table (made against
node ID 99).”
397.Both of these are powerful roles. Mr Godeseth was asked specifically about the DBA role
explained in [395] above. His specific answers merit quotation verbatim:
“Q. You would agree that those people have the role which allows them privileges to update, delete, or
insert into branch database tables whether they are using the correction tool or not?
A. Those people could log on to the database and do an awful lot of damage.
Q it of - 2015 ff. . sen't it?
A.C ”
(emphasis added)
398.Mr Coyne had given evidence that he had identified 2,175 occasions when it had been used. Mr
Parker in his 3rd witness statement had said that figure was wrong, but also said:
RLITO000005
RLITO000005
“T have not examined the privileged user logs, but based on my experience my expectation is that
these uses of APPSUP or at least the vast majority, are for support work that does not involve changes
to transaction data. I cannot recall any cases in which it has been used to change transaction data,
but I cannot state unequivocally that there are no circumstances in which it has ever happened.”
399.Mr Godeseth said in his written evidence that “Privileged users can, theoretically, inject, edit or
delete transaction data in Horizon Online. As far as I am aware, this has never happened.” His
evidence above about prior to 2015 makes it clear that he could not work out the number of times this
may have happened in any event. Therefore the fact that he was not aware it had ever happened is
neither here nor there, and does not assist the Post Office.
400.In his re-examination Mr Godeseth was asked about an entry in an OCP in relation to a fix that
stated UNIX user brdb. He said he did not know the detail, it meant logging on to the database, and
the “UNIX user” referred to “would be one of the guys in Ireland”. These were not members of the
SSC. When asked to clarify “who are the ‘guys in Ireland’ exactly?” he said “the people who support
the hardware, so UNIX is an operating system so they work at a pretty low level on the systems” and
by “low level” he meant “they have pretty powerful user rights but they are very much driven by
process as to how they use them”. He also said there was complete and utter control on the process
they go through before they did any of this type of activity, which in that re-examination example was
“Logon to BRDB Node 1 as UNIX user
‘brdb’."
401.He was also cross-examined about some of the bugs, which appear in the Bug Table considered
and partly agreed by the experts. The main admitted bug by the Post Office (in distinction to other
bugs admitted by its expert, Dr Worden) is called the Callendar Square bug. Mr Godeseth gave
evidence in his witness statement about this, partly in response to a statement that had been given by
Mr McLachlan, who was not
in the event called by the claimants. However, Mr Godeseth’s evidence about Callendar Square on this
was still called by the Post Office. It is a Legacy Horizon issue. Mr Godeseth had no first hand
knowledge of this but had reviewed documents and spoken to Mr Jenkins.
402.One of the KELs to which he was taken, with the reference JSimpkins338Q, related to problems
with Riposte and what was sometimes called the Riposte lock or unlock problem. This was when an
unexpected error occurred in Riposte while attempting to insert a message. The KEL shows it ran
from 2002, with other events occurring in 2003, 2004 and 2005. The entry for September 2005
expressly states that “this problem is occurring every week, in one case at the same site on 2
consecutive weeks”. A PEAK was sent to development. Under “solution” in the KEL the following
entry appears:
“SMC: If the event is seen at a multi-counter office during the working hours of the PO, or up to 18:00
on a weekday (in case they are balancing out-of-hours), RING THE OFFICE AND GET THEM TO
REBOOT the eventing counter. If they are in the process of balancing, it is strongly advised that they
reboot before continuing with balancing as they are at risk of producing an incorrect balance. Warn
the PM that if transactions appear to be missing, they should not be re-entered - they will become
visible after the counter has been rebooted.
If a reboot/Cleardesk does not resolve this problem, send the call over for further investigation - SSC
can rebuild the messagestore on the affected counter.”
RLITO000005
RLITO000005
(Block capitals in original; emphasis by underlining added)
403.The risk of a branch producing an incorrect balance as a result of this problem with Riposte is
clearly recorded. An incorrect balance is another way of referring to discrepancies in branch
accounts, a central issue in this litigation.
404.The Callendar Square issue or consequences (now admitted by the Post Office to be a bug) were
discovered in 2005 at the Callendar Square branch (hence the name). It is also sometimes called the
Falkirk bug, but I shall refer to it as Callendar Square. It undoubtedly caused discrepancies in the
accounts, The SPM in question had support from their Area Manager and the documents demonstrate
the following, from the Area Intervention Manager Visit Log:
“This office had severe problems balancing on Wk 25, resulting in a shortage of £6,414.46. After
checking various reports I am satisfied that the error is made up of:
£3,489.69 - Transfers
£2,870 - Giro Deposits
£54.52 - unidentified (however due to all the coming and going with re- keying entries, then this could
come back as an error).
The S . Hori f 09.05 f 5.30 Thi
was picked up when a member of staff noticed that a transaction, which had been taken by another
member of staff, had not been entered onto the system, so therefore she put the transaction through.
She checked at the time with her colleague who said that she thought she had put it through already
however she accepted that she could have made a mistake. Following on from that, it was picked up
that other giro business deposits that had been entered had not come up on the system, so they were
re-keyed.
There was also a problem with transfers from one stock to another, in that they had doubled up. The
Spmr made several telephone calls to the NBSC, telling them about his problems and he was advised
to carry on with balancing and produce his Cash Account. Whilst doing this a warning came up,
however the NBSC told the staff to continue to roll over. The result was that the office balanced
£6.414.46 short.
The Spmr spoke to the Horizon Support Centre (ref E0509150123) who investigated and agreed that
there had been a navigation problem that had now been rectified. They told the Spmr that they would
report to NBSC that they had identified and rectified the problem and that the amount could be held
in the suspense account, However, as part of the shortage relates to transfers, and no error notice will
be issued, then the Suspense Account Team are not prepared to authorise the entry.
I telephoned The Suspense Account Team (Ann Wilde), who told me that checks could be made with
Girobank after next Wednesday, and if that shows that duplications have been made, then they will
authorise the amount to be moved to the suspense account, until the office receives an error notice.
However, Ann stands by what she said about the transfer problems. and that they would not move this
amount to The Suspense Account.
I went back to the Horizon Support Centre and spoke to a supervisor (Ken). He said that the call had
now been closed as the problem had been rectified. I asked what was to happen about the resulting
shortage and he referred me back to NBSC, who they said would do various checks. I then contacted
RLITO000005
RLITO000005
NBSC, spoke to Rob Hughes and told him the story - he said he would put a call through to Service
Support. No follow up was received from Service Support regarding this call.”
(emphasis added)
405.The important points from this are as follows. This problem turned out to be an admitted bug. At
the time, the SPM noticed the problem - and could only identify it as a “problem with Horizon” and
reported it. The branch accounts were plainly affected by a sizeable amount, over £6,400. Horizon
Support Centre, after investigation, agreed that there was a “navigation problem” - this is a problem
with the Horizon system, and was caused by a bug. The Suspense Account Team would not authorise
the shortfall in the branch accounts being moved to the Suspense Account; in other words, they were
not prepared not to hold the SPM responsible. Mr Godeseth also accepted that “the genesis of it
appears to go back to February 2003 and similar lock agent problems back in November 2000”, a
point that he accepted as “that’s fair”. The history of the problem was therefore somewhat longer
than something that just occurred in 2005 for the first time. During that five year period, branch
accounts were usually being produced every four weeks.
406.Mr Godeseth admitted that it was “a horrible position” for the SPM to have been in. He said
“Service Support” was not Fujitsu, it was the Post Office. The story at [404] and [405] read like a mini-
summary of many of the factual issues in the litigation generally, with the exception that this is an
admitted bug by the Post Office.
407.The subsequent documents in this respect show that the matter went on for some considerable
time. In November 2005 the following is stated:
“1/ This problem is the root cause of the reconciliation error closed in PC0126042
2/ Presumably the root cause is deemed to be software not hardware
3/ The Postmaster has a workaround in place which is not to duplicate transactions (e.g. Transfer In)
just because the original attempts were successful but not showing on all nodes
4/ POA CS MSU have a workaround in place which is that if 3/ above is not followed & PC0126042
reoccurs, a BIMS will be issued advising POL to issue a Transaction Correction
5/ There is no SLT for software fixes as they are delivered based upon the priority or severity of the
issue and could remain open until both businesses decide a fix is necessary or the work around is
adequate.”
This shows that the entry in the PEAK identifies this as a software problem; that is effectively now
accepted, but this entry from 2005 did not lead to any public acceptance of a bug in Legacy Horizon
either then, shortly afterwards, or for some years after that.
408.The acronym SLT means Service Level Target. What this entry means is that this problem had
occurred before, and was known to have occurred before; there was something called “a workaround”
in place, but that required the SPM physically deciding not to duplicate transactions if the original
transaction did not show up as successful on all the nodes; if that were not done, TCs would be
required; both Fujitsu and the Post Office had to decide a software fix was required in order for this to
be corrected by fixing the software. No such decision had been taken at this point.
409.At Callendar Square itself, in January 2006 the SPM phoned one of his superiors. The record of
that call in the log states the following:
RLITO000005
RLITO000005
“Issue Raised: PM WANTS TO SPEAK TO SANDRA MCKAY URGENTLY REGARDING SERIOUS.
PROBLEMS WITH HORIZON IT IS REPEAT PERFORMANCE AS TO WHAT WAS HAPPENING
BEFORE AND HSH
FAILING TO PICK IT UP
Response by SPM: Telephoned the office and Allan said that he was having problems again with
transfers. He has contacted the Horizon helpdesk who have subsequently come back to him to say
that there is no system problem and that he should contact
NBSC. He did this and from what I can understand the NBSC have told him that he is
trying to balance on two different terminals. Allan disputes this and is adamant that there is a system
error.”
(Block capitals in original; emphasis by underlining added)
410.The statement that “there is no system problem” which is told to the SPM evidently conflicts with
what the PEAK at [407] had recorded, that “presumably the root cause” was the software.
411.Emails in February 2006 to and from Anne Chambers stated expressly the following:
“.... Inotice that in the early guise of this problem in the call it states the PM as Female:
Wed 12 October 2005 17:39 by UK956078 / HSH1 Saved: Wed 12 October 2005 17:39 Pm was trying
to transfer £2490 from node 7 onto node 2. She states that she has accepted the transfer on node 2
but the system is not showing this. On node 7 it is showing pending transfer but it is not showing on
node 2. It appears on her transfer sheet as completed.
is Male He reports:
You may recall that in September the above office had major problems with their Horizon system
relating to transfers between stock units.
The Spmr has reported that he is again experiencing problems with transfers, (05.01.06) which
resulted in a loss of around £43k which has subsequently rectified itself. [know that the Spmr has
reported this to Horizon Support, who have come back to him stating that they cannot find any
problem,
Clearly the Spmr is concerned as we have just spent a number of months trying to sort out the first
instance and he doesn't want a repeat performance. He is convinced that there is something wrong
with his Horizon kit. I would be grateful if you could investigate this and give him any support that
you can. I'm due to visit the office tomorrow to have a look at his paperwork and discuss the situation
with him.
So apologies for the long windedness but I have been given this by Liz as a problem so:
LL . 2 is it Hori . . ff 2
2.If there is an issue is this S90 release the cure? how confident are you/we it will fix the problem?
3.$90 counter release due week 4th March. Getting Sarah to check if this site is in the pilot 24th or
just part of the general release 4th March.
Appreciate your comments please.”
RLITO000005
RLITO000005
(emphasis added)
412. In the answer to this question posed in February 2006, Anne Chambers stated the following:
“Haven't looked at the recent evidence, but I know in the past this site had hit this Riposte lock
problem 2 or 3 times within a few weeks. This problem has been around for years and affects a
number of sites most weeks, and finally Escher say they have done something about it. I am interested
in whether they really have fixed it which it why I left the call open - to remind me to check over the
whole estate once S90 is live - call me cynical but I do not just accept a 3rd party's word that they
have fixed something!
What I never got to the bottom of, having usually had more pressing things to do, was why this outlet
. Possi fol .
sequence which makes it more likely to happen? This could still be worth investigating, especially if
they have continuing problems, but I don't think it is worthwhile until we know the S90 position.
Please note that KELs tell SMC that they must contact sites and warn them of balancing problems if
they notice the event storms caused by the held lock, and advise them to reboot the affected counter
before continuing with the balance. Unfortunately in practice it seems to take SMC several hours to
notice these storms by which time the damage may have been done.”
(emphasis added)
413.This shows the following important points. At least Anne Chambers in early 2006, and all those
with whom she was corresponding, knew that this problem - now admitted to be a software bug - had
been around “for years”. Horizon Support were telling the SPM, whose branch accounts were affected
by discrepancies, that “they cannot find any problem”. The SMC - the part within Fujitsu responsible
for providing corrective action for the “event storms” - would not always notice these had occurred in
time and by then “the damage may have been done”. I find that by “the damage” this can only mean
impact upon branch accounts.
414.Mr Godeseth gave evidence about S90, the software release intended to remedy this, which was
distributed in March 2006. This was 6 years after Legacy Horizon became in use. His evidence was
that this release was one year before what he called “the relevant period” for Ms Misra’s branch, one
of the SPMs convicted of criminal offences, which he was asked about. He also gave evidence that the
Callendar Square bug affected 30 branches, and had caused mismatches in the branch accounts at 19
(corrected in supplementary evidence in chief from 20) of them. He said in his witness statement that
this information had come from Matthew Lenton, Fujitsu’s Post Office Account Document Manager
and that “Fujitsu has established this for the purposes of this statement using the event logs
described above.” He accepted that this part of his witness statement was not correct. The figures
that he had provided had not come from an exercise specifically for the purposes of his evidence, or
for the Horizon Issues trial. It had actually come from “the spreadsheet [which] is something that was
done at the time, as I understand it by Anne Chambers, to actually investigate what had happened
when we had had these locks.”
415.This was an important correction by Mr Godeseth, for reasons that I will now explain.
416.It is a matter of public record that Anne Chambers gave evidence before the High Court in the
case of Post Office Ltd v Castleton [2007] EWHC 5 (OB) before HHJ Havery QC. Her evidence in
that case is summarised at [23] of that judgment. Callendar Square is referred to, but as a single
branch affected. The reason that I have referred to this case is because it was specifically referred to
RLITO000005
RLITO000005
by Mr Jenkins in an email of 8 March 2010 which was referred to by Mr Godeseth in paragraph 14 of
his witness statement. Mr Godeseth said that he agreed with that email, which stated:
“I've been asked about the issue at Callender Square, Falkirk (I’m not quite sure about the spelling)
that came up at the Castleton Trial.
I thought I’d better keep you in the loop on this.
I’ve now dug back into the archives to provide the following summary:
1.The problem occurred when transferring Cash or Stock between Stock Units. Note that West Byfleet
does operate multiple Stock Units so the issue could have occurred.
2.It manifests itself by the Receiving Stock Unit not being able to “see” the Transfer made by the
“sending” Stock Unit and is compounded by attempting to make a further transfer. Note that such
transactions usually reappear the next day.
3.It is clearly visible to the User as a “Receipts and Payments mismatch’ at the time that one of the
Stock Units is Balanced. This usually results in the Branch raising a call. There are no such calls in
Andy Dunks’ Witness Statement which summarises the calls raised by West Byfleet. Also this can be
checked on any Balance Reports or Branch Trading Statements that are available from the Branch
which should show that Receipts and Payments do match and that the Trading Position is zero.
4.The problem is also visible when looking at system events associated with the Branch. The System
events from 30/06/2005 to 31/12/2009 for West Byfleet have been checked and no such events have
been found.
5.The problem was fixed in the S90 Release which went live in March 2006 and so would not have
been relevant at the time of the detailed Transaction Logs obtained for West Byfleet between
December 2006 and December 2007.
Therefore I can conclude that the problems identified in Calendar Square, Falkirk are not relevant to
West Byfleet.”
417.This email does not provide anywhere near the same degree of information about the Callendar
Square bug as was available at the time of the Horizon Issues trial, or as recorded in the PEAKs
above. There has obviously been further investigation at Fujitsu into this specific issue since then, as
made clear by Mr Godeseth’s evidence about the information coming from Mr Lenton and Ms Anne
Chambers. The email does not use the term “software bug”. It does not refer to the fact that it is
accepted by
Fujitsu, following an investigation, that the admitted bug had affected at least 30 branches. Nor does
it refer to the fact that the Callendar Square bug caused mismatches at 19 of those 30 branches.
418.The spreadsheet to which Mr Godeseth referred as having been “prepared at the time” by Anne
Chambers was also put to him. It included textual notes and a list of call references and branches
affected by approximate date, which were all in 2004 and 2005. The notes were:
“NB Iti jority of
those which came to Peak, having either reported a problem or it caused a reconciliation report entry.
RLITO000005
RLITO000005
From Sept 2005, cash accounts were replaced by branch trading statements and the TPSC256 report
was no longer populated. Lcan't remember how we then knew about receipts and payments
mismatches and if we would have picked up on further issues.”
(emphasis added)
419.These notes make clear that there were other branches potentially affected, and the list of 30
branches compiled in the spreadsheet were “the majority”. It does not state that it is an exhaustive
list and the notes suggest it is not. The summary for each branch listed gives a variety of
consequences or effects. They include:
1.Receipts and Payments Mismatch.
2.Receipts and Payments Error.
3.Host Generated Cash Account Line.
4,Unable to roll Stock Units.
5.Reconciliation Report.
6.Slow running counter.
7.Pension and Allowance Report printouts.
8.Screen freezes.
9.Problems with declarations.
10.Unable to create AA.
11.Loss due to double transfer.
12.Transfers not showing on nodes.
13.CAC Lines not matching.
420.The spreadsheet was disclosed to the claimants on 27 February 2019, or very shortly before the
Horizon Issues trial, and the properties shown on the electronic document identify Anne Chambers as
the author, with a date of 22 December 2015.
421.Mr Godeseth also, entirely accurately in my judgment, accepted that the Callendar Square bug
had been present for a considerable amount of time prior to 2005. He said “I would agree that the
underlying bug had been there for a considerable time, probably since the Horizon went in”. This is
made clear by, amongst other documents, Anne Chambers’ entry in February 2006 stating that the
problem had “been around for years”. In another of the many PEAKs that dealt with this issue, the
entry for 22 September 2005 states “Note - a few of these errors seem to occur every week at
different sites.”
422.Mr Godeseth was taken to the contents of two letters from the Post Office’s solicitors to the
claimants’ solicitors. The first was dated 28 July 2016. This was the Post Office’s solicitors’ substantive
response to the claimants’ letter of claim, is very lengthy (as was the letter of claim itself) and ran to
99 pages. The second letter was dated 11 January 2019. He agreed that certain important elements of
the paragraphs in those letters explaining Callendar Square were incorrect. They were:
RLITO000005
RLITO000005
1.That the bug was only discovered in 2005.
2.That the bug had only affected one branch. The second letter stated in very clear terms that “The
Falkirk/Callendar Square issue was only known to have affected that one branch.”
423.Not only were both those statements in the Post Office’s solicitors’ correspondence completely
wrong, but the numerous documents at [398] to [421] above demonstrate that there were a great
number of references, including in both KELs and PEAKs (documents of a type that had been at one
stage claimed to be wholly irrelevant to the issues in this litigation) to the direct contrary.
424.Mr Godeseth’s witness statement about this, prior to his cross-examination, presented a very
different picture to the one that eventually emerged. Indeed, I would go further, and I find that his
witness statements omitted some very important headline points in respect of the Callendar Square
bug, presented a chronology very different to the real one, and had the effect (whether intended or
accidental) of giving a misleading impression of the Callendar Square bug and its impact.
425.1 find that the headline points omitted by Mr Godeseth are as follows:
1.The Callendar Square bug had existed “probably” (Mr Godeseth’s word in crossexamination) since
the inception of Horizon in 2000. There were numerous incidents of it occurring prior to 2005.
2.By February 2006, Anne Chambers and others at Fujitsu knew that this bug “had been around for
years”.
3.It had an impact “most weeks”. Preparing a software fix was not seen as a priority.
4.The bug had directly impacted the branch accounts of at least 19 branches, and possibly more, as
the Anne Chambers’ spreadsheet listed the 30 branches as “these are the majority”.
5.There were numerous effects of this bug, and at least 13 different ones identified in the spreadsheet.
6.Even in 2006, Horizon Support was telling at least one SPM that they “could not find any problem”
with Horizon when the bug’s impacts were reported to it, and this was recorded in documents linked
by Fujitsu directly with the Callendar Square bug.
7.Fujitsu had not established the total number of branches affected by consulting the events logs for
the purposes of preparing Mr Godeseth’s witness statement, or otherwise. To represent the
spreadsheet prepared by Anne Chambers as being the totality of branches affected by the bug is to
ignore the text of the notes.
426.Finally on this point, the Post Office’s solicitors’ letters were obviously not factually inaccurate.
Nor can the Post Office’s own legal team have known about the Anne Chambers’ spreadsheet until
very early 2019, otherwise disclosure of such an important document would surely not have been
given only on 27 February 2019, only seven working days before the Horizon Issues trial began. I do
not know why the Anne Chambers’ spreadsheet of 22 December 2015 was only disclosed to the
claimants a mere 7 working days before the start of the Horizon Issues trial, and it is not necessary to
speculate.
427.Although the Callendar Square bug in Legacy Horizon could, amongst other effects, cause a
receipts and payments mismatch, there is another bug in Horizon Online that also had the same
effect. This was called the Receipts and Payments Mismatch bug and is the first entry in the Bug
Table, Bug 1.
RLITO000005
RLITO000005
428.Mr Godeseth gave evidence about the Receipts and Payments Mismatch bug. This was a problem
at the cache. His information came both from Mr Jenkins and his own investigations. He did not know
about it at the time in 2010, and he did not in his witness statement refer to what is in my judgment a
very important document concerning this bug, namely the Receipts/Payments Mismatch issue notes
dated in the trial bundle index 17 October 2012, although it is likely to be a 2010 document as an
associated document from Mr Jenkins (trial bundle reference F/1000/1 and F/1001/1) is dated 29
September 2010. I will reproduce certain parts of the issue notes, as I consider them to be highly
relevant to the Horizon Issues. There were 10 attendees, including Mr Jenkins. The document was
marked Commercial in Confidence and is, according to the trial bundle index, a memo of a meeting.
Another document with the same date is headed “Correcting Accounts for “lost” Discrepancies” and
states “any branch encountering the problem will have corrupted accounts”. The issues notes
document stated:
“What is the issue?
Discrepancies showing at the Horizon counter disappear when the branch follows certain process
steps, but will still show within the back end branch account. This is currently impacting circa 40
Branches since migration onto Horizon Online, with an overall cash value of circa £20k loss. This
issue will only occur if a branch cancels the completion of the trading period, but within the same
session continues to roll into a new balance period.
At this time we have not communicated with branches affected and we do not believe they are
exploiting this bug intentionally. (emphasis added)
The problem occurs as part of the process when moving discrepancies on the Horizon System into
Local Suspense.
When Discrepancies are found during Stock Unit rollover into a new Transaction Period, then the
User is asked if the discrepancy should be moved to Local Suspense. If the branch presses cancel at
this point the Discrepancy is zeroed on the Horizon System.
Note at this point nothing into feeds POLSAP and Credence, so in effect the POLSAP and
Credence shows the discrepancy whereas the Horizon system in the branch doesn't. So the
branch will then believe they have balanced.
If at the next screen the rollover is completely cancelled, then no harm is done. However if the
Rollover is re-attempted at this point, the rollover will continue without any discrepancy meaning
Horizon doesn't match POLSAP or Credence.
This has the following consequences:
+There will be a Receipts and Payment mismatch corresponding to the value of Discrepancies that
were "lost"
Note the Branch will not get a prompt from the system to say there is Receipts and Payment
mismatch, therefore the branch will believe they have balanced correctly.
*When the Branch begins the new Branch Trading period the discrepancies will show at Zero,
however the Receipts and Payment mismatch will carry over into the next period.
Impact
+The branch has appeared to have balanced, whereas in fact they could have a loss or a gain.
RLITO000005
RLITO000005
+Our accounting systems will be out of sync with what is recorded at the branch
*If widely known could cause a loss of confident in the Horizon System by branches « Potential impact
upon ongoing legal cases where branches are disputing the integrity of Horizon Data
*It could provide branches ammunition to blame Horizon for future discrepancies.
Identifvi , fi uti
The Receipts and Payment mismatch will result in an error code being generated which will allow
Fujitsu to isolate branches affected this by this problem, although this is not seen by the branches. We
have asked Fujitsu why it has taken so long to react to and escalate an issue which began in May.
They will provide feedback in due course.
Fujitsu are writing a code fix which stop the discrepancy disappearing from Horizon in the future.
They are aiming to deliver this into test week commencing 4th October. With live proving at the model
office week commencing 11th October. With full roll out to the network completed by the 21st of
October. We have explored moving this forward and this is the earliest it can be released into live.
The code fix will on stop the issue occurring in the future, but it will not fix any current mismatch at
branch.
Proposal for affected Branches
There are three potential solutions to apply to the impacted branches, the groups recommendation is
that solution two should be progressed.
SOLUTION ONE - Alter the Horizon Branch figure at the counter to show the discrepancy. Fujitsu
would have to manually write an entry value to the local branch account.
IMPACT - When the branch comes to complete next Trading Period they would have a discrepancy,
which they would have to bring to account.
RISK- This has significant data integrity concerns and could lead to questions of "tampering" with the
branch system and could generate questions around how the discrepancy was caused. This solution
could have moral implications of Post Office changing branch data without informing the branch.
SOLUTION TWO - P&BA will journal values from the discrepancy account into the Customer Account
and recover/refund via normal processes. This will need to be supported by an approved POL
communication. Unlike the branch "POLSAP" remains in balance albeit with an account
(discrepancies) that should be cleared. IMPACT - Post Office will be required to explain the reason for
a debt recovery/ refund even though there is no discrepancy at the branch.
RISK - Could potentially highlight to branches that Horizon can lose data.
SOLUTION THREE - It is decided not to correct the data in the branches (ie Post
Office would prefer to write off the "lost"
IMPACT - Post office must absorb circa £20K loss
RISK - Huge moral implications to the integrity of the business, as there are agents that were
potentially due a cash gain on their system.”
(bold present in original)
RLITO000005
RLITO000005
429.1 find this to be a most disturbing document in the context of this group litigation. It is a 2010
document and issues between the Post Office and many SPMs concerning the accuracy of Horizon
had, for Legacy Horizon, gone on for a decade (2000 to 2010) and these continued under Horizon
Online (introduced in 2010). Under “Impact”, some of the bullet points incorporate a summary of
these issues.
“« The branch has appeared to have balanced, whereas in fact they could have a loss or a gain.
+0 A lie Hori Post Office's] wi £ ‘ A
the branch
*If widely known could cause a loss of confident in the Horizon System by branches * Potential impact
ae . ity of Horizon D
*It could provide branches ammunition to blame Horizon for future discrepancies.” (emphasis added)
430.The attendees at this meeting included at least one member of Post Office (rather than Fujitsu)
personnel, Andrew Winn of POL Finance. There were obviously legal cases going on at the time,
hence the reference in the underlined bullet point to “ongoing legal cases”. If these were criminal
cases, the Post Office would be the prosecuting authority, with certain important duties. If these were
civil cases, the Post Office would be a party with disclosure obligations. An affected branch would
believe it had balanced its accounts correctly; it would not have done so. There is an evident concern
amongst those at the meeting which is recorded in this document that this issue should not become
“widely known” in order to avoid causing “a loss of confidence in the Horizon System”. Fujitsu do not
seem to have been particularly prompt in either identifying the problem or reacting to it.
431.0f the three solutions considered, the one that was adopted was issuing a debt recovery/refund
even though there would be no discrepancy at the branch. Given one of the Common Issues was the
status of a Branch Trading Statement, which the Post Office argued should legally have the status of a
settled account with the SPM as agent, I note the suggestion adopted with more than passing interest.
The solution explains “even though there is no discrepancy at the branch”. This means the Branch
Trading Statement would be correct, or in balance. The discrepancy would not be in the Branch
Trading Statement, it would be in POLSAP or Credence. That this is correct is shown in an associated
document from Mr Jenkins (trial bundle reference F/777/2) where he stated that “the data used for
the BTS will also have a zero value for Discrepancies at the end of the period.” BTS means Branch
Trading Statement. A similar entry is in the document at F/1000/1 which states at /2 “Note that if the
bug was not present, then the Discrepancy would have been transferred to Local Suspense and that
would have been cleared, so there are a number of things wrong with the BTS.” (emphasis added). In
other words, the Post Office itself was not considering the BTS as having the status of a settled
account, based on these entries.
432.Further, an associated document entitled “Receipts and Payments Mismatch’ states:
“1. The purpose of this note is to document a request that we have had from Post Office in terms of
presenting details of what happened as a result of a bug in HNG-X
in September 2010 which caused a Receipts and Payments mismatch and also resulted in
Discrepancies being lost.
RLITO000005
RLITO000005
The background to this is the fact that there was a BBC documentary broadcast on Monday 7 th
February 2011 reporting on postmasters being unhappy about being pursued for losses by
Postmasters on Horizon.
It should be noted that the issues described here relates to HNG-X (Horizon Online) and that the
implementation of the accounting mechanisms in the two systems is totally different (but they do
produce the same reports and support the same business process).”
(emphasis added)
433.The two references - one to “ongoing legal cases”, the second to a BBC documentary - show that
there was a distinct sensitivity within both the Post Office and Fujitsu about keeping this information
to themselves in order to avoid a “loss of confidence” in Horizon and the integrity of its data. A less
complimentary (though accurate) way of putting it would be to enable the Post Office to continue to
assert the integrity of Horizon, and avoid publicly acknowledging the presence of a software bug. The
solution adopted was the issuing of TCs to the relevant branches. There was no publicity given to the
SPMs at the time about the presence of this software bug in Horizon. An entry at F/1000/3 states:
“It should be noted that as Discrepancies are normally Losses, then a Lost Discrepancy would
normally work in the Branches favour and so there is no incentive for the branch to report the
problem. Also if we do amend the data to re-introduce the Discrepancy, this will need to be carefully
(emphasis added)
434.Mr Godeseth was asked about the lack of communication of this.
“Q. Are you surprised that this had not been communicated so people would just be warned about it?
A. There was obviously a fear that subpostmasters may be looking to exploit this because it gave ~
there was a fear that people could see this as a way of defrauding the Post Office.
Q. So concealing it from SPMs who were honest was justified because of the expectation of dishonesty
of subpostmasters in the network, in a nutshell?
A. In my view, this was a decision made by Post Office on how to manage this particular bug. You
could interpret it the way that you have put it. Q. Do you agree with the way that I have put it to you?
A, I think I'm agnostic. I can see the ~ I can see a rationale for not broadcasting this, but equally, if the
objective is to be totally open and honest and take the risk of causing more chaos in the network then
yes, I would have to agree with that.”
435.Indeed, the approach of the Post Office to notification of the existence of bugs to SPMs is an
illuminating one. The Post Office communicates with SPMs generally in a number of ways. Methods
that have been referred to in both of the Common Issues trial and also the Horizon Issues trial include
“Branch News”, a type of newspaper that was used, and also “Memo View”, a pop up message that
appears on Horizon terminals. During the preparation of this judgment, in view of the fact that
judgments are required to be factually accurate, because dates can sometimes become lost in the
midst of huge trial bundles and very detailed submissions, and also because the simplest way to arrive
at a date when something has been done is (sometimes) simply to ask the parties for a date ora
reference, on 7 October 2019 the parties were jointly asked the following:
RLITO000005
RLITO000005
“For each of the Dalmellington and Callendar Square bugs, could the parties please indicate (by
means of an agreed fact but only if that is possible, if not each party’s contended for answer) their
answers to the following two questions:
1.The date when the existence of each of these was communicated by the Post Office to SPMs;
2.A trial bundle reference to a contemporaneous document, email or communication from the Post
Office demonstrating question 1.”
436.The parties could not agree on a date when the existence of these two bugs - Callendar Square
being an acknowledged bug in the Bug Table dealt with below (from the 2nd Joint Statement of the
two experts) and Dalmellington, which is now accepted to be a bug - was communicated to SPMs. The
claimants’ answer to the two questions was the same for each bug, namely that there was “no date in
any contemporaneous document from the Post Office in the trial bundle showing that such
communication in fact took place and identifying the actual content of any such communication.” The
Post Office’s answer was as follows:
437.For Callendar Square: “It has never been Post Office’s case that, once it was discovered, the
existence of the Callendar Square bug was communicated by Post Office to all SPMs or communicated
to SPMs who had been affected. Post Office's case is that it was unnecessary and would have been
inappropriate to do so.” A number of references and further explanation then followed, stating that
when “an SPM was affected by the bug, this was detected and corrected in the ordinary course” and
also that “In some cases, SPMs who were affected by the Callendar Square bug may have been made
aware that there was a problem in Horizon which had caused an error in their accounts which
required correction.”
438.For Dalmellington: “It has never been Post Office’s case that, once it was discovered, the
existence of the Dalmellington bug was communicated by Post Office to all SPMs or communicated to
the SPMs who had been affected. Post Office's position is that it was unnecessary and would have
been inappropriate to do so.” The Post Office also stated that “when an SPM was affected by the bug
this was detected and corrected in the ordinary course” and that the Dalmellington bug “was
prevented, by effective counter-measures, from causing even a single impact on branch accounts”.
439.The Post Office also stated that there were real disadvantages to notifying SPMs of this. The Post
Office relied upon passages in Dr Worden’s report that summarise reasons why users of an IT system
should not be given information about parts of that system which they did not encounter in their daily
work; that there was “no point in trying to educate all the users in details and terminology of the
system which will never concern them”; that the best thing to do is try and fix a bug or defect, rather
than “create some new error message to the users”; that “automated messages from the system are
only of limited help to users”; and other points in similar vein. The Post Office also provided
documentary references to instances where “an SPM might have been made aware”, although these
documents were not deployed in the evidence.
440.1 will reproduce two entries from one of these documents, a PEAK with reference PC0103864.
This shows that on 3 June 2004 an SPM reported “that he had a problem
with some transfers yesterday, he was transferring stock and cash between the aa main stock unit and
the bb shared stock unit and although only one transaction shows for the transfer out the transactions
were transferred into the bb stock unit twice giving the pm a discrepancy.”
RLITO000005
RLITO000005
441.The discrepancy in the cash account was £22,290 in total (shown in the entry for 8 June 2004)
although the reconciliation error reported by the Host was £44,580. This was finally resolved on 5
August 2004 with the issue of an error notice. There is nothing to suggest that the SPM was told that
this had been caused by a software bug.
442.The Post Office’s approach to this, in my judgment, entirely misses the point. In my judgment, the
above passages are simply extraordinary. Two of the documents above are dated 2010, some 9 years
ago. The PEAK is from 2004, 15 years ago. Their contents support the claimants’ case on the Horizon
Issues. Fujitsu knew, to take Callendar Square as an example, that this bug existed in Horizon. They
knew that it had affected branch accounts. It was not, as the Post Office puts it “unnecessary and
inappropriate” to notify SPMs of this. I have listed the points on this bug at [425] above omitted by Mr
Godeseth from his written evidence. Those same points all lead to the same conclusion in my
judgment, namely that the Post Office ought to have notified, at the very least, all those SPMs whose
branch accounts had been impacted by this bug that this had occurred, and that it had occurred as a
result of a software bug. The fact that the integrity of Horizon data was a live issue at this time should
not have influenced the decision to notify SPMs of a software bug. Further, the Post Office’s
explanation in its submissions that SPMs had their accounts “corrected in the ordinary course” is not
a suitable phrase, unless by “ordinary course” one means keeping the cause or reason for the
correction secret and therefore hidden from the other party in the accounting transaction, namely the
SPM. Also, one is not educating users in the details and terminology of the system (as suggested by Dr
Worden) if one informs them that there is a software bug in the system and its symptoms are as
follows. This is relevant to Horizon Issue 2, namely whether the Horizon IT system itself alerted SPMs
of such bugs, errors or defects as described in Issue 1 above and if so how, and also Horizon Issue 9,
which concerns transaction data and reporting functions available to SPMs. It is also relevant to Dr
Worden’s consideration of countermeasures, which he considers includes vigilance by users (which
means SPMs).
443.Mr Godeseth gave express evidence in his witness statement that the Receipts and Payments
mismatch bug occurred in September 2010. That date too was factually incorrect. The issue notes
refer to Fujitsu knowing about it far earlier, and Mr Godeseth accepted he had seen this document
before his cross-examination. That document even records Fujitsu being taken to task by the Post
Office about how long it had taken to react, as in “We have asked Fujitsu why it has taken so long to
react to and escalate an issue which began in May. They will provide feedback in due course.” Not just
the impression, but the express text in Mr Godeseth’s witness statement, was to the effect that the
bug was discovered in September 2010 and almost immediately dealt with. That was far from the
case, and that written evidence was simply wrong.
444.Mr Godeseth gave evidence about another bug, called the Local Suspense Bug, also in Horizon
Online. His witness statement quoted almost word for word from a contemporaneous document
prepared by Mr Jenkins. Almost all his information on
this bug had come from Mr Jenkins. The bug caused entries from local suspense accounts in 2010 to
be reproduced in Horizon in two successive years. What was supposed to be only temporary data was
retained, not deleted as it ought to have been, and the system then used it again. It affected certain
tables in the branch database and the archiving strategy of deleted Stock Units. This affected 15
branches. The effects of the bug were brought to the attention of the Post Office by two SPMs “with
the largest discrepancies” and Fujitsu became aware of it in January 2013. However, the note by Mr
Jenkins of 15 May 2013 summarising it states that the problem was known about far earlier than that:
RLITO000005
RLITO000005
“In April 2011 a problem was found with the archiving strategy related to Stock Units that have been
deleted in a Branch. A consequence of this is that some changes were made to the archiving strategy
on 3rd July 2011. An unintended consequence of this change was that any Branch that deleted a Stock
Unit at the end of 2010 which had local suspense transaction in that Stock Unit before it was deleted
were left in the table used for constructing the BTS. This meant that as Trading Periods cycle around
each year, these BTS records became visible in 2011 when the same Trading Period was reached.
The effect of these old records was that after the BTS was produced an incorrect figure was generated
for the Opening Balance of the Local Suspense Account for the following period. This amount
corresponded to the value of the historical record. These orphaned records were created between
16th November 2010 and 9th December 2010.
When the next Trading Period was balanced, then this incorrect Opening Figure would result in the
total value for Local Suspense being calculated incorrectly and the SPMR being asked to make good
an incorrect amount. It is at this point that transactions would be generated into the audit trail
reflecting the fact that the SPMR had cleared the Local Suspense account for an incorrect amount.
The audit trail operated correctly in the sense that it accurately recorded the transaction on the
system.
This problem was not reported to Fujitsu in 2011/12 and only affected a small number of Branches
and only for a single Trading Period. However the two branches with the largest discrepancies did
report the issue to Post Office Ltd who could see the impact of the problem in their back end system
and wrote off the loss or gain for the branch but did not ask Fujitsu to investigate further.
At the same Trading Period in 2012/13, the problem re-occurred and this time one of the affect
Branches reported the problem to Fujitsu on 25th February 2013 (PEAK 223870) resulting in a
detailed analysis of this issue and finding the orphaned BTS records. The root cause was determined
by 28th February 2013 and a preliminary report was sent to Post Office Ltd. A further update was sent
on 14th March 2013 with a full analysis of the issue and all the affected branches.”
(emphasis added)
445.Another software bug which Mr Godeseth gave evidence about was called the Dalmellington Bug.
This is also known as the Branch Outreach Issue or Bug, as it affected what are called Outreach
branches, which are those that have a core branch
(such as a rural or small branch post office) and an outreach branch, such as a mobile post office van,
or village hall somewhere else (where post office services would be provided, say, one day a week).
This “outreach” element is used, particularly in very rural areas, to provide post office services to
remote communities. A SPM needs to scan or record a pouch from one branch (the core) into the
outreach branch (the van or village hall) to record that the contents of the pouch are not in the core
branch, but are in the outreach branch instead. This is a branch to branch cash remittance, if the
pouch contains cash. Mr Godeseth had no first hand knowledge of this bug. As the claimants put it in
opening submissions, this is a useful way of examining certain features in Horizon as the same person,
the SPM, controls “both ends” of the transactions.
446.The Dalmellington Bug related to Horizon Online and actually included two potentially separate
issues, and the combination of these. One related to what is called Forced Log Out, when the Post Log
On script was not correctly closed down and was left on the stack of incomplete processes. The other
related to the way the Pouch Delivery script operated; because the Post Log On script had not
RLITO000005
RLITO000005
correctly closed down meaning the stack was not empty, the Pouch Delivery script thought it had not
finished and attempted to repeat the last part of the script. This had the effect of recording the
remittance transactions and printing of receipts. This would lead to duplicate pouch IDs. Pouches
have bar codes that are scanned and each is supposed to have a unique ID. The duplicate pouch ID
would have a value attached to it - the sum in the original pouch. It would however be recorded twice
as a result of this.
447.What happened at Dalmellington was the SPM scanned £8,000 from her core branch into her
outreach branch. She was lucky in this sense, in my judgment, in that she “controlled” both ends of
the transaction. The transfer replicated four times. She had Horizon receipts in her outreach branch
of £32,000 (4 times £8,000) and therefore a discrepancy of £24,000, as she had only transferred
£8,000 in cash out of her core branch into it. This was investigated by SSC and she was issued with a
TC a few weeks after it had occurred. This was caused by a software bug, and the bug is now admitted
by the Post Office, although it describes it as having “transient impact”. That simply means it was
corrected.
448.When investigating this Fujitsu found 112 occurrences affecting 88 different branches in the
previous 5 years. They were as follows, taken from an internal Fujitsu presentation dealing with this:
Feb 2010 to Jan 2011 65 incidents
20116 incidents
20129 incidents
20137 incidents
20149 incidents
201516 incidents
449.The same document showed fixes applied in April 2010, January 2011 and January 2016. It also
showed only one call to SSC at Fujitsu in 2015, and none for any of the years 2010 to 2014. The
presentation also showed that some branches that were affected were so-called “mediation branches”
- in other words the SPMs would have been in the Second Sight mediation scheme that was brought
to an end - but that none of the dates when those branches were affected matched the “mediation
dates”, which means the periods for which ARQ data was to be used in the mediations to analyse
exactly what had happened at each branch. There is nothing in the Fujitsu document used by Mr
Godeseth for his evidence, to suggest that this analysis led to each branch being told of these
software bug occurrences outside the mediation date range, and the items were usually corrected by
means of TCs.
450.Mr Godeseth said that “Unfortunately this particular error is not subject to receipts and payments
problems because it could be -- it could be the user doing something twice, it's -- the bug had the
effect of making it look as though a user was simply doing something multiple times.” (emphasis
added) He said that “the issue was not raised with Fujitsu until 2013”, it was put to him it was 2015
(based on the slides) but regardless of which of these dates is right, the Fujitsu slides show only one
call to Fujitsu in 2015, none for the other years, although the reference to fixes in 2010 and 2011
suggests it was reported to Fujitsu earlier. If those fixes were intended to correct the bug they cannot
have worked. Fujitsu maintained that they investigated and found all occurrences, and the Post Office
maintained that they had corrected any financial impact of this bug. Whether this is correct or not in
RLITO000005
RLITO000005
all cases, the evidence demonstrated that the effects of this bug were experienced for the period from
2010 until 2016.
451.In his re-examination, some of which I have already referred to in the context of UNIX user and
privileges above, Mr Godeseth also said that having seen the OCPs and OCRs in cross-examination, he
accepted data had been deleted but did not consider it to be transaction data. Initially he described it
as “operational data” but said that was too wide a term, and:
“There possibly ought to be a different definition for it because equally data in the database which is
telling you which stock unit, you know, what -- which TP [ie trading period] you're in, whether a stock
unit is in use, all of those are operational data which do not go down and impact the transaction data,
so for me, the definition of transaction data -- I should be able to get from one opening -- one set of.
opening figures to the end opening figures by going back to the transaction data in its absolutely raw
form.
452.He said it was what one went through to bring different totals together “to get to your end
position in a trading statement.” Regardless therefore of whether the data which he accepted was
being deleted is, or is not, properly described as “transaction data”, it would feed into the end position
in a branch trading statement. It would therefore, in my judgment, have an impact upon branch
accounts, regardless of the term used to categorise it.
453.By the end of Mr Godeseth’s evidence, and by virtue of the way he answered questions, the
reality of the development and operation of both Legacy Horizon and Horizon Online was far clearer
than it had been before the Horizon Issues trial. I
found Mr Godeseth to be a truthful, credible and helpful witness in his oral evidence. His evidence
was of considerable assistance; he did not always answer questions the first time he was asked, but
the final answer would be reached. Given his role throughout the whole of the life both of Legacy
Horizon and Horizon Online, I found his evidence in cross-examination to be of central relevance to
the Horizon Issues. His evidence in cross-examination was also, in my judgment, broadly supportive of
the claimants’ general case.
454.There are, however, two aspects of Mr Godeseth’s evidence - though not his performance as a
witness in the witness box - that were highly unsatisfactory and cannot pass without comment. The
first aspect relates to his written evidence. The totality of his evidence, after his cross-examination,
bore very little semblance to the picture that had been portrayed in his written witness statements.
On some extremely important factual matters, such as the dates when Fujitsu had become aware of a
particular bug, or the spreadsheet exercise by Anne Chambers prepared in 2015 (and disclosed in
2019), his written evidence was simply directly wrong. On others, such as the headline points omitted
on the Callendar Square Bug that have been set out in [425] above, very important central elements
detrimental to Fujitsu were simply omitted.
455.Legacy Horizon had, as has been explained, started life originally as something rather different to
what it became, having initially been intended as a tri-partite project involving payment of benefits. It
did not unfold in this way and became rather different. Horizon Online also did not have a happy birth.
The pilot for it had to be stopped, and Fujitsu put it on what was called “red alert”. Mr Godeseth
described this as “very serious”. The biggest issue was with Oracle, which was what Mr Godeseth was
working on and hence knew the most about, but he explained that there were other problems going
on at the same time. Some of these problems were put to him - and it must be remembered that this
was a pilot scheme, with some problems to be expected - and they included cash being short on one
RLITO000005
RLITO000005
day by £1,000 because a transaction for £1,000 did not show up on the online report facility; cash
withdrawals being authorised on screen yet the printed receipt being declined (the customer very
honestly brought the cash back next day having noticed the receipt wording); a similar problem with a
cash deposit; and remming in figures all being doubled up. These are all somewhat - and indeed
markedly - similar to some of the problems alleged by the different claimants’ witnesses in this
litigation. These all occurred during the pilot scheme.
456.Not one of these different problems was referred to in Mr Godeseth’s witness statements, of
which there were three. Witness statements are supposed to be accurate, and in a case such as this
one with such centrally important issues, accuracy is clearly important. Quoting only selectively from,
or wholly ignoring, contemporaneous documents prepared by (say) Mr Jenkins, who was the extensive
source of much of the evidence, is not only unhelpful, it presents an entirely misleading evidential
picture. It is not necessary to consider further how many personnel at Fujitsu may have assisted Mr
Godeseth in producing such documents. Their content was wholly misleading in their original written
form. Fortunately the cross-examination of Mr Godeseth led to a far clearer picture in so far as his
evidence is concerned.
457.The second unsatisfactory aspect which arose from his evidence is the approach of Fujitsu as
demonstrated in various documents, including the PEAKs and KELs, but also in particular in the
Receipts/Payments Mismatch issue notes. To see a concern expressed that if a software bug in
Horizon were to become widely known about it might have a potential impact upon “ongoing legal
cases” where the integrity of Horizon Data was a central issue, is a very concerning entry to read ina
contemporaneous document. Whether these were legal cases concerning civil claims, or criminal
cases, there are obligations upon parties in terms of disclosure. So far as criminal cases are
concerned, these concern the liberty of the person, and disclosure duties are rightly high. I do not
understand the motivation in keeping this type of matter, recorded in these documents, hidden from
view; regardless of the motivation, doing so was wholly wrong. There can be no proper explanation for
keeping the existence of a software bug in Horizon secret in these circumstances.
458.The degree to which either, or both of, Fujitsu and/or the Post Office, expressly or constructively,
knew exactly what and when, is for future trials in this litigation, and I make no findings in that
respect in this judgment. They are not necessary in order to resolve the Horizon Issues and I do not
speculate.
459.In my judgment, however, there are sufficient entries in the contemporaneous documents to
demonstrate not only that Fujitsu has been less than forthcoming in identifying the problems that
have been experienced over the years, but rather the opposite. The majority of problems and defects
which counsel put to Mr Godeseth, and which were effectively admitted by him, simply would not have
seen the light of day without this group litigation.
460.I do not know if this is because of concerns about the future of Horizon going forwards; or for
Fujitsu to protect its corporate reputation; or for some other reason. It is unnecessary to speculate,
and I do not do so. An internal Post Office IT risk management document from 2017 stated that “the
HNG-X platform is end of life and is running on unsupported Windows software”, that it needs
replacing, and also that the "Branch counter technology is aged and unreliable, with frequent
hardware failures, resulting in branch disruptions." Mr Godeseth agreed with all of this. The
unsupported platform is Windows NT4. As Mr Godeseth put it, “any technologist would tell you that
was too old”. Windows NT4 is an operating system that was released in 1996, and Microsoft ended
mainstream support for it in 2002 and extended support in 2004. Horizon now runs, not on NT4, but
RLITO000005
RLITO000005
on Windows 10 and it is called HNG-A rather than HNG-X, This change to Windows 10 occurred in
February 2017 and the roll out was administered by Computacentre, with Fujitsu providing the
software. The Horizon system of 2019 is therefore a very different system than both Legacy Horizon
and the earlier version of Horizon Online, HNG-X.
461.The original Horizon Online system was introduced in 2010, and was even then substantially
based on the existing system, Legacy Horizon, itself by then at least 10 years old. That had been
introduced in 2000, and therefore it should not be controversial to describe Horizon Online in its
HNG-X form as old technology. There have of course been upgrades and other improvements since
2010, and other software additions and add-ons, but Horizon will by necessity require replacement at
some point. I detected a degree of sensitivity in the litigation generally about any plans for replacing
Horizon Online, as though the Post Office feared that any hint that it might
be replaced would be seized upon by the claimants (and/or assumed by the court) as accepting that
the criticisms of Horizon Online by the claimants would be corroborated by any Post Office plans for
its replacement. Such fears, if there are any, are groundless, at least from the court’s point of view. It
is inevitable that technology systems, particularly complex systems from as long ago (in technology
terms) as 2010 will need replacing at some point. Technology moves at an increasingly rapid pace.
There is reference in some of the documents in the trial bundle, but not in the evidence, to something
called “Thin Client”, about which Dr Worden said he had no knowledge. He thought it was a cloud
based system. It is unnecessary to speculate about this either. It is sufficient to state that, in my
judgment, any plans there may be for replacement of Horizon Online in the ordinary course of the
Post Office’s wishes to improve its business are not something that I need to consider in order to
resolve the Horizon Issues.
462.The degree to which Mr Godeseth’s evidence affects my conclusions on the expert evidence will
be dealt with in Part L, Overall Conclusions.
463.Very slightly before the end of Mr Godeseth’s cross-examination - about 30 minutes or so - the
Post Office sought to bring the Horizon Issues trial to an end by issuing their recusal application
seeking to remove me as the Managing Judge, and to have the trial re-started at some indeterminate
point in the future before another judge. There was therefore a substantial interruption in the trial,
and a period of 22 days before the Horizon Issues trial resumed and the next factual witness for the
Post Office, Mr Parker, was called.
Mr Parker
464.Mr Stephen Parker is also a Fujitsu employee, and is now the Head of Post Office Application
Support. He is therefore a very senior person. He first started work on what was then called the Royal
Mail Group Account in 1997, which was before the introduction of Horizon. He has continued to
provide support to the Post Office Account in the various roles he has occupied at Fujitsu throughout
the whole of Horizon’s life, by which he meant both Legacy Horizon and Horizon Online.
465.He was originally a support consultant within the SSC, and stated that he was effectively the
deputy manager of the SSC, providing 3rd line support. He was lead designer and part of development
team for the internal website providing the support knowledge database (which is also called KEL),
technical documentation management and operational change control. He assisted the SSC manager
in the provision of the support service and operational management. He became the Manager of the
SSC in March 2010, The SSC later expanded and provided support services to other Fujitsu
customers, but the Post Office Horizon system is still the largest. He has between 25 and 40 staff in
RLITO000005
RLITO000005
the SSC whom he manages. He said that he was deputy manager of the SSC when Mr Roll worked
there, although he did not have that title formally, and that people in the SSC would know this
because he would stand in for the manager in his absence, and make decisions on approving actions
for him.
466.He had provided three different witness statements for the Horizon Issues trial, and his 2nd and
3rd statements were further to, and ultimately corrected, his earlier evidence, which arose in respect
of what Mr Roll said had been done whilst he had been at the SSC, including injecting transactions
during the time of Legacy Horizon. These were done using Riposte, the language then in use. During
the Horizon Issues trial and just before the issuing of the recusal application, the Post Office’s
solicitors wrote a letter to the claimants which stated the following. This letter was dated 20 March
2019, and was sent the night before Mr Parker was originally due to be crossexamined, his evidence
then becoming delayed due to the recusal application:
“We understand from Fujitsu that the SSC has been carrying out further work to identify any PEAKs
that show transactions being injected at the counter in Legacy Horizon in addition to those referred to
in paragraphs 29 and 30 of Mr Parker's second statement. On Monday [ie the 6th day of the Horizon
Issues trial] we learned that an SSC technician has:-
*searched for all KELs that mentioned “RiposteMessageFile", "Ripostelmport" and “RiposteMessage';
*collated the responsive KEL references (AChambers2226M, CObeng1029162824, DSeddon822M,
MYoung5043M, RColeman1250R, acha2340K, ballantj498J, dsed344J, AHolmes3343J, DSeddon1753N,
GMaxwell46141, PCarroll12541, RKing5135L, 81111, pcar847S, wbra716s);
*re-searched the Peak system for any PEAKs which contained those KEL references; and
identified the following PEAKs: PC0105560, PC0106885, PC0063599, PC0063871, PC0065796,
PC0066061 and PC0083998 (which cross refers to PC0076029).
We understand that the PEAKs referred to above relate to either:- * correcting configuration data
after a PinPad change; or
*reference data.
We also learned that during this exercise the SSC technician has also identified three examples of the
marooned transaction scenario described in paragraph 38.2 of Mr Parker's second statement:
PC0068495, PC0099141 and PC0079196.”
467.The search terms identified in the letter were different to the ones that Mr Parker had used in his
2nd witness statement. These were "RiposteMessageFile", "RiposteMessage", "LPO Delete",
"Marooned" and "RiposteObject put". The reason this is important is the command “RiposteImport”,
used by the SSC as a search term during the week commencing 18 March 2019, but not used for the
purposes of Mr Parker’s 2nd witness statement in January 2019, is actually one of the commands that
was in fact used to inject transactions. In other words, the exercise carried out for the purposes of the
witness statement was wholly deficient, in that it did not search for one of the specific terms used to
inject transactions.
468.When asked to explain this in his cross-examination, and the corrections that he made, Mr Parker
stated the following:
RLITO000005
RLITO000005
“A I notified the legal team that Mr Simpkins had done some more work and as a result of that we
changed -- we -- this letter was actually generated.
Q And you didn't think it was important to correct paragraph 29 of your second witness statement
accurately to reflect the situation as it would have been on the day you were due to give evidence, or
indeed today?
A That wasn't a choice I made personally. I was advised that we generated this letter.”
(emphasis added)
He did not say who made the choice to which he referred, namely not to correct his witness
statement.
469.It certainly seems to me highly curious - at best - that the exercise performed for Mr Parker’s 2nd
statement, by personnel at SSC who are expert in such things, should have omitted using the search
term “RiposteImport”. Given the purpose of paragraph 29 and 30 of Mr Parker’s 2nd witness
statement was to identify how many times data had been in fact injected, a search exercise that did
not use “RiposteImport” was never going to give the full picture. That initial search exercise was
plainly inadequate. It is also rather curious that, having done that, the exercise that was described in
the letter of 20 March 2019 from the Post Office’s solicitors that correctly used “RiposteImport” as a
search term should have been done so very closely before Mr Parker was originally due to be called as
a witness, namely 21 March 2019. In re-examination, Mr Parker said that “the thought occurred to
him that he could add some other search terms into the work that he had done previously” and “he
came to me with the data”. Why this thought should have suddenly occurred to the person it occurred
to, who was not a witness, and at the time that it did, could not be pursued. Mr Parker denied that he
was trying to conceal anything, and said he was just trying “to get the right data for the court which
can be difficult sometimes when you are going back fifteen years.” This explanation overlooks that the
decision not to use “RiposteImport” was one taken for his witness statements. His first witness
statement was dated 16 November 2018; his second was dated 29 January 2019, and his third was 28
February 2019. None of these are “going back fifteen years”.
470.Mr Parker had also given evidence in his 1st witness statement in respect of Mr Roll’s written
evidence that Fujitsu could inject and affect branch transaction data without the knowledge of the
SPM. He had originally said the following:
“As I explain below, those suggestions are incorrect and Mr Roll's account of Fujitsu's actions and
powers is inaccurate and misleading".
471.He explained this paragraph as follows in his cross-examination.
“A. In that paragraph I am trying to make the point that the suggestion that we frequently changed
branch transaction data without informing the branches that such actions were being actually taken is
not correct. "Frequently" is a subjective term but I would not have described the rate at which we
were changing branch transaction data as "frequently".”
472.One has only to read the original paragraphs as drafted to see that Mr Parker was not originally
taking issue with frequency, or how often such actions were taken; he was stating in express terms
that this could not be done by Fujitsu at all. For someone who has effectively spent the best part of 20
years in SSC, rising to a very senior post, such evidence must have been - and I find that it was - quite
obviously incorrect. His explanation of this was as follows:
RLITO000005
RLITO000005
“Q Now, given the change of your evidence in your later statements about the ability remotely to
access and inject transaction data, can you explain to the court what you were saying you believed
about that sentence? What should the court now read for those words in the light of the three
statements that it has before it?
A If I take the last sentence in isolation, which is what I think you are asking me to do, then I don't
understand how I apply it, because I am -- I have been simply trying to say there that the frequency
was not high, and that we would always involve the
Subpostmaster wherever possible if that sort of action was actually being taken. That's what I'm
trying to say by that.”
473.This was a very poor attempt by Mr Parker at explaining away quite directly incorrect factual
evidence in a witness statement on a very important area in the litigation, namely Fujitsu's ability to
inject transactions remotely. Mr Parker’s only recourse was to claim he had not been saying what he
plainly had been saying in his 1st witness statement - that Fujitsu did not have the power to do it, and
that Mr Roll’s evidence to the contrary was inaccurate and misleading. The power to do something
cannot be equated with frequency.
474,Mr Parker also accepted that at the time he signed his 1st witness statement he had known that
Fujitsu had the power to insert transactions into individual branch counters by using the
correspondence server, a process Mr Roll described as “to piggy back through the gateway”. It was
put to him that he had been giving his evidence initially from a position of not being very well
informed about what could or could not be done. He denied that; he said “that would be wrong” and
that in general terms he was confident of the information he gave. The obvious conclusion that I draw
from this is that Mr Parker chose specifically to give the impression in his 1st witness statement that
Fujitsu did not have the power (the word Mr Parker expressly chose) to inject transactions into the
counter at branches, even though he knew that it did. This paints him in a very poor light as a credible
witness. There is also no adequate explanation for why Mr Parker's colleague should have decided, in
the middle of the trial, to use the “RiposteImport” search term; equally, there is no adequate
explanation for why this search term was not used for the exercise for Mr Parker’s witness statement.
Injection of messages into the counter has been an extremely central issue in the long running dispute
between the Post Office and its SPMs for a long time, and Mr Parker’s reliance in his re-examination
upon the passage of time and the difficulty of “getting the right data” for the court is, in my judgment,
a poor excuse. In reality, he had no credible explanation.
475.A series of propositions was put to him about the degree of expertise expected of all the members
of SSC 3rd line support. He did not agree with all of the points put, and in particular did not agree
that all those in the 3rd line support role would be expected to have detailed knowledge of the system,
based on both documentation and the inspection of source code; or that they would be trained in the
coding languages used within the application. He was, however, left rather exposed in terms of the
accuracy of this evidence when it was shown that the points that he had been addressing in this part
of the cross-examination - some of which he would not accept - were taken from a Fujitsu internal
document, entitled “End to End Application Support Strategy” marked Fujitsu Restricted and
Commercial in Confidence which he had himself drafted. Indeed, it is page 1 of that document that
specifically identifies Mr Parker himself as the author of the document, and it was approved by the
Head of Application Services at Fujitsu, an obviously very senior role. This refusal by Mr Parker to
accept his own previously drafted points in what I consider to be an important contemporaneous
document also paints him in a very poor light as a credible witness.
RLITO000005
RLITO000005
476.He accepted that Mr Roll was conscientious, skilled and experienced, and that he had given Mr
Roll a reference when he left Fujitsu.
477.He, and others not identified in his statement (although he said who they were when he was
asked, Mr John Simpkins and Mr Mark Wright), had prepared a spreadsheet that had been put to Mr
Roll showing the total number of calls to SSC between 2001 and 2004. Mr Simpkins, for what it is
worth, appears to have been the person behind the exercise referred to in the letter at [466] above, so
he was obviously involved both in preparing evidence and doing searches. Mr Parker agreed that SSC
during Mr Roll’s time needed to address coding issues, even if they did not necessarily have an estate
wide impact, and he also agreed that the list of PEAKs that had been identified as involving Mr Roll
was only those where he had been the person who put the final response on the PEAK, rather than
where he had worked on the PEAK at all.
478.He also accepted that in categorising “software errors” he had excluded from the count those
that were categorised as Avoidance Action, Administrative Response and Solicited Known Error.
479.He was taken to a PEAK in 1999 where an initial balance was multiplied twice “due to a known
software error” in the amount of a discrepancy of over one million pounds - £1,082,544,32 to be exact.
The PEAK ran to 14 pages. 10 people worked on it including those in Development as well as the SSC.
He accepted that this would have involved people looking into code. It was, however, categorised as
“Administrative Response”. That PEAK alone demonstrates to my satisfaction the lack of inaccuracy in
the Fujitsu closure codes, but there were many others. Other PEAKs to which he was taken showed
reconciliation errors, and a note by the developer identifying that what were “up to 10 unnecessary
reconciliation errors each week” and that the PEAK was a regression of another PEAK; the
investigation into it led to the conclusion that a subsequent software release had not caught a fix. It
stated "Risks (of releasing and of not releasing proposed fix): Without this fix, there will be
possibilities of system errors at counter and while doing reversal transaction". This software fix was
“released to live”, which means the fix was actually written and introduced into the system. Mr Parker
had not included this in his software category either, because the final person working on it had
categorised this as Administrative Response.
480.In another PEAK, it was recorded that “the system is still playing up in that the screen is hanging
in the middle of transactions -- PM did transaction ... but left office for 1 hour -- when he came back
the monitor had 141 first-class stamps on screen totalling £38.07". Mr Parker accepted that this was
not how the system was supposed to work, but the same PEAK also recorded the SPM becoming upset
because the SSC were advising it was user error. The SPM wished to make a complaint about the
person at SSC and the PEAK records “we both feel that this PM is complaining unjustly”.
481.The same PEAK recorded that ROMEC engineers had been to the site, and had actually seen the
problem themselves, which were phantom transactions. It recorded
“it’s not just the PM’s word now”. Indeed, this PEAK is particularly concerning, in terms of the
accuracy of PEAK closures by Fujitsu. It records in one of the Fujitsu entries the following:
“T now have pressing evidence to suggest that unwanted peripheral input is occurring, the likely
source being the screen. This has been seen at Old Isleworth ...When the PM has been asked to leave
the screen on overnight I have observed system activity corresponding to screen presses happening
with no corresponding evidence of either routine system activity or human interference. The way
forward now is to correlate this with Microtouch complied monitoring software and to this ends
Wendy is arranging for installation of the kit on Friday..."
RLITO000005
RLITO000005
However, notwithstanding this, the subsequent entries include “Phantom transactions have not been
proven in circumstances which preclude user error" and “In all cases where these have occurred a
user error related cause can be attributed to the phenomenon?”.
482.The use of the phrase “in all cases” in respect of attributing user error wholly ignores that the
ROMEC engineer is said to have seen the problem themselves, and it is not simply the word of the
SPM in this instance. It ignores the activity observed overnight by the Fujitsu employee who made the
entry in the PEAK, when the branch was not even open. Also, there is no reason, given the lengthy
entries and studying the text throughout the PEAK in detail, why user error should be reached as a
conclusion at all. This appears to be been used simply as an excuse to avoid an adverse conclusion to
Fujitsu.
483.Notwithstanding the detailed entries in the PEAK itself that in my judgment plainly point to the
contrary, the PEAK was still categorised as No Fault in Product.
484.In another, dealing with phantom transactions, one branch was described as “a problem office”
that was accordingly being “monitored”. The PEAK recorded that “there have been incidents which
showed a possible correlation between system activity and phantom transactions. These pointed to a
touch screen problem and as a result the screen was replaced with a resistive model. As this produced
no measurable improvement it has to be assumed that the problems were user related". This was also
closed as No Fault in Product.
485.A KEL on phantom transactions stated "There have been several calls over the last few months
where Postmasters have reported phantom sales. Items appear by themselves for which the PM has
not pressed an Icon. These may be individual items or several of the same item. Sometimes when no
one has been near the screen items may appear". These are similar problems being reported by a
number of different SPMs. That alone should have suggested, not as a matter of computer expert
evidence but simply basic common sense, that something was amiss. If a number of different and
unconnected people all report the same symptom, then that could well suggest a common problem to
them all. However, the KEL, which ran from 2000 to 2004, suggested that the cable between the
screen and the base unit was the root cause. The dates on this KEL show that, even on the face of the
document, the problem was manifested for some years.
486.Another PEAK that showed a problem described as “non-zero trading position on office rollover”
was put to Mr Parker. This had been given a priority C status by Fujitsu, a status he accepted meant it
was not taken account in the Service Level Agreement with the Post Office in liquidated damages
thresholds (a type of financial penalty), which was a feature of those software PEAKs given higher
priority status A and B. Status C also was non-critical. The description of the problem was described
as “PM states that he has rolled over but the system is telling him that he hasn't -- PM states that he is
in balance period 7 and he states he is getting the message ‘wrong trading period MSG 31318 office
balancing error" and “Non-zero trading position ... on rollover of branch by user WMC002 to trading
period 8".
487.Mr Parker described himself as “hesitant” to describe this as a payments mismatch issue, and
said it was an “office balancing error”, even though later in the PEAK senior personnel at SSC
recorded “NBSC have ruled out user error” and also that "The problem occurred on 15/09/10 when
stock unit 02 rolled over. This was originally reported, as per KEL, BALLANTJL759Q, in call
PC0204537 ... but for some reason the call was closed without being investigated. There is a known
problem with the use of the Cancel button during the stock unit rollover. This is fully described in KEL
WRIGHTM33145]".
RLITO000005
RLITO000005
(emphasis added)
488.He said he had not recognised the KEL number as being the payments mismatch
KEL. He accepted that in another entry in the PEAK the matter was stated to be “Severity: Critical” -
which meant it could not be status C, defined as non-critical - and also included the entry “the branch
accounts will need to be corrected.” This PEAK was however, on 1 November 2010, closed as category
“Avoidance Action Supplied”. Mr Parker’s spreadsheet did not include “Avoidance Action Supplied” as
constituting a software issue. He claimed that he had used “the definition my experience dictated
rather than that in the document” and that “I forgot that that was exactly what I had done in that
circumstance when I prepared that a few months ago.” Finally, after the question was put a third time,
he accepted that it was a deliberate decision on his part to depart from the definition in Fujitsu’s own
documentation.
489.Another PEAK also identified as category C, non-critical, related to pouches being remmed in
twice, the duplicate pouch being £25,000 in value. The PEAK stated “this might be a software issue”
but also “PM is unsure if this is user error if it is a software fault. NBSC have not confirmed if it was
user error or if they can reverse it.” One entry stated "POL have been informed of the error. Hopefully
they'll issue a TC to correct loss at the branch. The underlying problem caused by using previous
button during or just after scanning pouch barcodes, is still under investigation". This was closed as a
Solicited Known Error.
490.One PEAK which I found most illuminating, in terms of Mr Parker’s response to it when he was
asked questions, is PEAK PC0229446. This is dated 17 November 2013, hence is Horizon Online, not
in the period of the pilot project and sufficiently after the introduction of it, in 2010, for it not to be
put down as the sort of issue that might occur in very early days of a system whilst minor issues were
ironed out. The problem
was described in the PEAK as “PM doing cash declarations every now and again has major loss.”
Entries in the PEAK included "PM has had cash declaration problem throughout the year and is losing
a lot every now and again"; "He phoned up helpline told him can't [have] declared properly. He states
that he loses £2,000 then jumps suddenly to £4,000, one point they lost £8,000 and is always losing
money. PM stated that he has three post offices, only happens on this site"; and then "Done a
declaration this morning and had a £6,000 also. It shows no error message when doing it. No report
prints out only print-out of cash declarations". This passage of his crossexamination followed this:
“Q. Pausing there, if the PM is correctly reporting that, then that would be very serious for the
Postmistress or Postmaster, wouldn't it? A. If it is being correctly reported, yes.
Q And it would not be the system working as it should.
A If we attribute it as a system fault, yes.”
491.This again was categorised by Fujitsu as Category C, non-critical. In a sense, those two questions
and answers sum up a large part of the case. The reason that this plainly, in my judgment, does
demonstrate a system fault is Fujitsu’s own PEAK recorded:
“Voiced NBSC quoted ref H18174172 to see what checks they have done themselves before
transferring call to Horizon.
They stated they had trainers come into the office and ruled out user error.”
RLITO000005
RLITO000005
And
“NBSC states the user error checks were carried out by Auditors at the site and not over the phone.”
(emphasis added)
492.This matter was assigned to Anne Chambers, whose name appears very often in many of the
documents. The final part of her conclusion was (having said that “there were no known issues that
would result in the variance being incorrect”):
“T can't tell why the declared cash doesn't match the expected cash figure, the branch need to make
(emphasis added)
This is simply passing the buck back to a SPM in a branch. How an SPM could “investigate the
anomalies” given they appeared to be generated by the Horizon system is not explained, nor could it
sensibly be.
493.She also closed the response with the category “Final - no fault in product” and closed the call by
defect cause "General - User”. I consider this PEAK to be an ideal illustration of what I consider to be
the most extraordinary situation at SSC, and one which on the face of it is difficult to explain, given
the function of SSC was to investigate faults:
1,Fujitsu routinely assigned non-critical Category C to matters that were really very important in their
own right in any event, but of extreme importance to SPMs whose branch accounts were being
directly affected. Mr Parker accepted that only Category A or B attracted financial penalties. It is not
possible, on the evidence before the court, to conclude one way or the other whether it was this that
affected the categorisation adopted by Fujitsu personnel.
2.Fujitsu would ignore information directly from the Post Office itself that demonstrated that a SPM
was not at fault. ROMEC engineers observing specific matters occurring, or in this case the Post
Office’s own auditors ruling out user error, were simply ignored. This effectively amounted to the
Fujitsu SSC personnel positively over-ruling the Post Office’s own non-SPM personnel, who were
giving them essential information about failures in the Horizon system.
3.Fujitsu would also close PEAKs in a variety of ways that entirely mischaracterised the issue. This
particular one was “no fault in product” and defect cause “general - user”, in other words, it was
caused by the SPM. This had been specifically ruled out by the Post Office’s own auditors who had
visited the branch.
4.On the PEAKs that were used in the Horizon Issues trial, Fujitsu would routinely assign lower
categories of importance to reported matters that were directly impacting SPMs’ branch accounts.
They would also routinely assign user error to reported matters, not because they had uncovered user.
error, but because they could not explain what had occurred. It seems to have been used as a default
setting. Given my findings in the Technical Appendix, this approach by Fujitsu simply cannot be
justified in technical terms.
494,Fujitsu do not, on the face of these documents shown to Mr Parker, appear to me to have properly
and fully investigated these myriad problems, nor did Fujitsu categorise such incidents correctly. They
also seem to have moved away, in their investigations, from concluding that there were any issues
with the software wherever it was possible for them to do so, regardless of evidence to the contrary,
an approach that has been carried into the Fujitsu evidence for the Horizon Issues trial. In re-
RLITO000005
RLITO000005
examination Mr Parker was asked about the way he had done his exercise, and also the point was put
to him that the PEAKs to which he was taken were outwith the period when Mr Roll was employed at
SSC. The implication of this was that Fujitsu may, or would, have been more accurate in the period in
question rather than in later periods. Mr Parker said he could not possibly read all the PEAKs, that he
considered his data to be accurate, and although “you may find a few PEAKs where the response code
does not tally with the document” given the number of PEAKs involved “human beings will make those
errors”.
495.I reject that explanation, such as it is. I do not consider that Mr Parker was interested in accuracy
in any of his evidential exercises, and I do not consider that he was objective in the way he presented
his evidence, although he sought to give the impression that he was. I consider that Mr Parker, and
the team who assisted him, sought to portray the Horizon system - Legacy Horizon and Horizon
Online - in a light as favourable as possible to Fujitsu, regardless of its own internal evidence to the
contrary, and regardless of the facts. Mr Parker’s spreadsheet plainly did not calculate all the
categories that were used by Fujitsu for problems with the system, and sought
to give the impression that Mr Roll had been involved in far fewer incidents of this type than were the
case.
496.My conclusion about Mr Parker’s lack of accuracy, is supported by the changing story of Fujitsu
about remote injection of messages; the need for correcting supplementary statements, including
from Mr Parker himself; his admission that he knew Fujitsu had the power to do this, even though his
1st statement expressly said otherwise; the attempt to shift the blame for this on to Mr Roll; the
assertions in his subsequent statement that Mr Roll had been unclear; and also by his attempt to
portray his original evidence as dealing with frequency, not power or ability. My conclusions are also
supported by his “deliberate decision” to depart from Fujitsu’s definition in its own internal document,
which I have explained at [488] above. I also consider that the fact that someone such as Mr Parker
had been on the Post Office Account for such a very long time, in his case throughout the whole life of
Horizon, means he was hardly best placed to be objective in this type of evidence. But even making
allowance for the natural reaction of an employee to wish to protect his employer’s interests, which
many people may have sub-consciously, I find that Mr Parker’s evidence to the court was inaccurate to
a significant degree.
497.In my judgment, the exercise done by Mr Parker to demonstrate the number of PEAKs that Mr
Roll worked on involving software, included in the spreadsheet of categorisation he and others
prepared, is of no evidential value whatsoever. It relies upon the categorisation of the PEAKs
performed by the Fujitsu personnel themselves, which in the vast majority of documents used in the
trial were simply wrong and/or misleading; and it also used a different definition to the one included
in Fujitsu’s own documents such as the PEAKs above. He accepted that he had made a deliberate
decision to use a definition other than the one in the Fujitsu documents, which he justified by saying
was based on his experience. That deliberate decision alone is remarkable. I find that he was, by
means of the exercise he was involved in, trying to downplay the extent of Mr Roll’s involvement in
software issues, and also trying to downplay the extent of software problems that were experienced
on Legacy Horizon during Mr Roll’s employment at SSC.
498.Although Mr Parker agreed, as it was put to him more than once, that accuracy is important, I do
not consider his evidence in his witness statements to have been remotely accurate, even though he
stoutly maintained that it was. He continued to maintain this in his re-examination, even though by
then he can have been in no doubt that he had departed from Fujitsu’s own definition - which he had
RLITO000005
RLITO000005
said he had forgot about. I found him a very unsatisfactory witness, who presented in his witness
statements a misleading and one-sided sanitised version of actual problems and events that Fujitsu
had experienced throughout his time at SSC. Although he provided tables with one of his witness
statements that provided a detailed commentary on a large number of PEAKs and KELs, that
commentary appearing in a column headed “Fujistu’s Comments” with two columns, “Response to Mr
Coyne” and “Financial Impact on branch accounts” - that latter almost always “no impact”. The
entries in those columns are almost entirely self-serving by Fujitsu and I find them to be of no
evidential value. The text of the actual PEAKs and KELs themselves are far more useful, obviously
entered at a time when the Fujitsu/Post Office position in the litigation was not at the forefront of the
different authors’ minds, and these are what for the most part the experts have considered. Mr
Parker's table is of no assistance.
499.The degree to which the evidence of fact affects my conclusions on the expert evidence will be
dealt with in Part L, Overall Conclusions.
Mr Membury
500.Mr William Membury is another Fujitsu employee, namely a Fujitsu Central Quality Partner and
he is specifically focused on the Post Office account. He was taken ill during the early part of the
Horizon Issues trial, and when (after dismissal of the recusal application) the trial resumed with
outstanding factual evidence on 11 April 2019, he was not well enough to be called in person. His
witness statement was dealt with as hearsay evidence and given the circumstances the claimants
agreed to extend time for the relevant notice to be given under CPR Part 33.2.
501.Mr Membury has worked for Fujitsu since 1998 and became the Quality Risk and Compliance
Manager for the Post Office account in 2011. He has overseen multiple audits of the Horizon system.
Given the timing of this, these would have all been on Horizon Online. In 2014 he became a Central
Quality Partner specifically focusing on the Post Office as well as the Payment Card Industry (PCI)
Security Standards Subject Matter Expert (SME) for Fujitsu UK and Ireland. In 2015 he became the
PCI SME for the Europe, Middle East, India and Africa regions. He returned to the Fujitsu Post Office
Account as Head of Quality and Compliance in October 2018.
502.He described his statement as an overview of the auditing and development regimes in place in
relation to Horizon to the extent that those points were not covered by the disclosed technical
documents. The statement was said to provide a high-level overview of the audits to which Horizon
had been subject since 2000; Fujitsu’s audit methodology and his role in the audits; and to describe
the development of changes to Horizon and how that tied in with or into the audit process.
503.Mr Membury’s statement was very brief; it ran to only 5 pages of text. It also omitted some highly
material matters. For example, he stated that the Post Office and Fujitsu agreed in 2010 that Horizon
Online would be audited against the International Standard on Assurance Engagements (ISAE) No.
3402 assurance standard (Horizon ISAE 3402 Audit), entitled "Assurance Reports on Controls at a
Service
Organization", which was issued in December 2009. This was developed to provide an international
assurance standard for allowing public accountants to issue a report for use by user organisations and
their auditors (user auditors) on the controls at a service organisation that are likely to impact or be a
part of the user organisation's system of internal control over financial reporting. It provides
assurance that the solutions in place to manage financial transactions are appropriate.
RLITO000005
RLITO000005
504.However, he completely omitted any reference to the Ernst & Young Management Letter for the
year ending 27 March 2011 which has been referred to in [264], [393] and [524], which included
concerns about privileges and other critical matters in relation to IT. Indeed, he made only two
references to Ernst & Young. One was to state that:
“Ernst & Young have carried out the Horizon ISAE 3402 Audit since the 2012/2013 financial year
(preparations for the audit began with Post Office in 2011)”
505.The second was a reference to how Ernst & Young performed the ISAE 3402 audit for the period
1 April 2014 to 31 December 2014. His failure to mention the 2011 Management Letter is, in my
judgment, a serious omission.
506.He could not be asked any questions about his evidence; although that was not his fault, this goes
to the weight which I attach to his evidence in any event.
507.1 consider that Mr Membury’s evidence is of limited, if any, assistance in resolving the Horizon
Issues. It does however continue the very one-sided picture presented by all the Fujitsu witness
statements, which was to omit any reference to important contemporaneous documents that criticise
or demonstrate any deficiencies with Horizon.
The absence of Mr Gareth Jenkins
508.It is entirely a decision of the parties which witnesses they choose to call in any proceedings in
respect of any evidence. The position of one person, however, who did not appear in the Horizon
Issues trial, must be considered in more detail than would be usual, as the claimants make
considerable complaint about this. The person in question is Gareth Jenkins, a senior Fujitsu employee
who, although he retired recently, was obviously widely available to the Post Office and the source of a
great amount of information both to the Post Office’s witnesses of fact, and also to Dr Worden
(although he was not separately identified in Dr Worden’s “sources of information” paragraphs in his
1st Report). The fact that he provided information to Dr Worden emerged during the latter’s cross-
examination. Mr Jenkins had previously given expert evidence for the Post Office in some of the
criminal prosecutions of SPMs, in particular that of Ms Misra, to whom I have referred above, who
was convicted of criminal offences in Guildford Crown Court in 2010.
509.When the Post Office served its evidence of fact, there was no witness statement from Mr Jenkins,
although many of their witnesses relied upon him as their source of information, he was referred to
very often, and he obviously knew a great deal about Horizon. The extent and way in which Mr
Jenkins had been closely involved was explained by Mr Godeseth in his cross-examination. Mr
Godeseth had, in respect of the receipts and payments mismatch, originally stated in paragraph 42 of
his 2nd witness statement that 60 branches were affected. He had corrected this to 62, a factual
correction that was specifically made by him. The following passage of evidence is relevant to Mr
Jenkins’ involvement in this.
“A. No, no, I didn't do a calculation to come up with the 60. I was quoting from other people.
Mr Justice Fraser: Someone just gave you the 60, did they?
A. I thought I was quoting from other people. I -- Gareth even said to me that in my statement I had
said “approximately 60", so I was not -- clearly I didn't because the statement here doesn't contain
that word. I had rather hoped it had when this was first brought to my attention, but no, I certainly
did not do any specific calculation to come up with the 60 that I put into my original statement.
RLITO000005
RLITO000005
Mr Green: Did Gareth explain the change to you from 60 to 62? A. No.
Q. But how did he come to tell you what was in your original statement? What was that conversation?
A. I picked up the number -- to be -- my objective in this was to explain to the court the symptoms of
the bug and how -- the technical aspect of it. I did not pay particular attention to getting the detail on
how many branches were affected, correct. Q. Okay. So you have spoken to Gareth since your
statement?
A. I don't think I have spoken to him about this in particular. I was -- as I say, when Gareth had said I
had originally said “approximately 60" I was thinking that was quite neat, but that's not the case.
Q. Well, you said “Gareth even said to me that in my statement I had said ‘approximately 60'", so he
must have said that to you after your statement had been filed?
A. It was a comment in a document that we were exchanging.
Q. But you hadn't spoken to him about remote access since your first statement?
A. No.
Q. Why have you stayed off that topic with him?
A. Oh, sorry, this was just a comment. We have been exchanging documents, we have been
commenting on documents, so it was not a particular conversation. It is merely a case of Gareth had
commented on this when it was pushed back to us that I had originally said 60 and actually the
answer was different.”
510.There is nothing wrong with Mr Godeseth correcting the number of affected branches from 60 to
62. There is also nothing wrong with him reproducing a number given to him by someone else, as long
as it was properly identified that the information came from someone else, who ought to be identified
(which it wasn’t in his statement, but was, in at least outline terms, in his cross-examination). The
reason for reproducing these passages is simply to identify the extent to which Mr Jenkins was so
closely involved in the litigation, a point relied upon by the claimants. There were a great many
references throughout Mr Godeseth’s evidence, written and oral, about information he had obtained
from Mr Jenkins.
511.When the Post Office served their evidence of fact, the claimants had asked the Post Office why
there was no statement from Jenkins, whether Mr Jenkins was available to give evidence, and also
whether he was involved as one of a team of what the claimants referred to as the “shadow experts”.
This description was challenged by the Post Office, and the question of shadow experts is addressed
further at [556] below. No explanation was given for Mr Jenkins’ absence in response to these
requests, or in evidence in the trial, although it was confirmed that Mr Jenkins was not one of the
team of so-called “shadow experts”.
512.There the matter might have rested. However, in the Post Office’s written closing submissions, an
explanation of sorts was for the first time provided. This was in the context of two matters: firstly, by
way of explanation of Mr Godeseth’s evidence, and potentially to downplay its impact; secondly, in
relation to the claimants’ complaints about the second hand nature of some of the Post Office’s factual
evidence because in large part this had emanated from Mr Jenkins. This explanation by the Post Office
included the following passages in its written submissions:
RLITO000005
RLITO000005
“144, [The claimants] understandably complain that Mr Jenkins and the other source of Mr Godeseth’s
information could have given some of this evidence first hand. However:
144,1Taking into account that Mr McLachlan’s evidence specifically addressed things said or done by
Mr Jenkins in relation to the Misra trial, Post Office was concerned that the Horizon Issues trial could
become an investigation of his role in this and other criminal cases.
144.2Moreover, Post Office was conscious that if it only adduced first hand evidence in the trial, it
would end up having to call more witnesses than could be accommodated within the trial timetable.
144.3Furthermore, so far as Post Office was aware, the relevant parts of Godeseth 2 were most
unlikely to be controversial. For example, the Misra trial was a matter of public record, the four bugs
were covered by contemporaneous documentation and Post Office had no reason to doubt Fujitsu’s
account of the documents it held.”
513.In a footnote to paragraph 144.2 of the closing submissions, the Post Office added “.....As noted
above, had its witnesses only given first hand evidence, Post Office estimates that some 34 additional
witnesses would have been required.”
514.The following are relevant in my judgment:
1.0f primary importance is the principle that it is for each party to decide whom to call as a witness,
and what evidence they may seek to obtain from any particular witness.
2.Mr Jenkins is an important and central person so far as the operation, efficacy and robustness of
Horizon is concerned, and also in respect of the number of incidents over the years that have led to
PEAKs, KELs, problems and fixes. That is reinforced by the Post Office’s own closing submissions that
stated, at paragraph 138, that there were two possible candidates for the witness giving the overview
of Horizon, namely Mr Godeseth and Mr Jenkins. It also stated that taking into account Mr Jenkins’
involvement in cases before the CCRC (including the Misra case) the decision was taken to use Mr
Godeseth. For that decision to have been taken, it is implicit that Mr Godeseth must know enough
about the system to give the evidence that he did.
3.The point in the footnote about a potential further 34 witnesses is not relevant to whether Mr
Jenkins was, or was not, called by the Post Office. The Post Office did not at any stage apply to the
court and explain that further time was required for the length of the Horizon Issues trial for the
specific reason that relevant evidence of fact from the Post Office would require a particular minimum
number of witnesses. However, given the primacy of point (1) above, it was entirely for the Post Office
to decide whether to call Mr Jenkins or not, how many witnesses to call, whom those witnesses were
to be and what the content of their evidence should include.
4.The Post Office chose to proffer a reason for Mr Jenkins’ absence in closing submissions. They were
not obliged to explain. However, the reason in paragraph 144.1 quoted above is not a valid reason for
his absence. The claimants would have been entitled, in cross-examination, to put to Mr Jenkins any
previous inconsistent statements he had made on the same subject, but obviously only if there were
any. These could, potentially, have included previous statements he may have made in earlier
proceedings, but in order to be allowed to do that, such statements would have had to be inconsistent
with his evidence in the Horizon Issues trial. Putting a previously inconsistent statement on a
particular fact is permitted as part of crossexamination if a witness’ evidence on that fact is in issue.
However, the Horizon Issues trial would not have become “an investigation of his role in this and
other criminal cases” - the Horizon trial was about the Horizon Issues. Also, this type of cross-
RLITO000005
RLITO000005
examination would only have arisen if any statement(s) in the Horizon Issues trial was or were not
consistent with those he had made in previous cases in which he had been involved.
5.Regardless of the validity of the explanation given by the Post Office, Mr Jenkins was not called as a
witness, and it merits repetition that it is entirely a matter for the Post Office which witnesses it
called, and which it did not. No speculation is permitted as to what evidence he might have given in
the Horizon Issues trial. It ought to be recorded however that no limit on the number of witnesses was
imposed by the court on any party, and the Post Office could (had it wished) have called both Mr
Godeseth and Mr Jenkins. Nobody forced the Post Office to choose one or the other.
6.There is established authority that in certain circumstances the court can draw adverse inferences
from the absence of a witness. The claimants do not invite me to draw any adverse inferences from
the absence of Mr Jenkins, and I do not do so.
7.The fact that the Post Office chose to advance certain evidence in the Horizon trial, that emanated
from Mr Jenkins, by means of another witness saying “Mr Jenkins told me that.
claimants were deprived of the opportunity properly to test that evidence by asking Mr Jenkins about
" means that the
it, It also emerged in cross-examination, although not in his written report, that Dr Worden has also
obtained certain information directly from Mr Jenkins. By dealing with the material in this way, and
having deprived the claimants of the opportunity to cross-examine Mr Jenkins, the weight to be given
to evidence emanating from Mr Jenkins is less than it would be otherwise.
515.There are two consequences of the Post Office choosing to adduce such evidence in this way. The
first is that Mr Jenkins did not have to undergo any cross-examination on such evidence; the second is
that such evidence would not be given the same weight by the court as though Mr Jenkins had given it
himself. The Post Office would have known this in advance, as this is wholly conventional. How these
different factors were weighed by the Post Office and its legal advisers in deciding whether or not to
call him, is solely a matter for the Post Office. The end result is that Mr Jenkins did not appear and
was not cross-examined.
516.However, I reject the suggestion that Mr Godeseth was insufficiently knowledgeable of the
matters upon which he was cross-examined for his evidence to be of significant evidential weight. The
Post Office’s Closing Submissions sought to dilute the effect of Mr Godeseth’s cross-examination by, at
the same time, implicitly suggesting that
Mr Jenkins (or someone else, but not Mr Godeseth) would have been a far better
person to have answered those questions; yet explaining why Mr Jenkins had not been called. These
submissions, though understandable from the Post Office’s point of view in forensic terms only, have
little force. A clear choice was made by the Post Office not to call Mr Jenkins as a witness; that was
their right. A clear choice was also made to call Mr Godeseth. The Post Office was are entitled to call
such evidence as deemed necessary by both the Post Office and its advisers. They cannot, however,
have matters both ways, and try to downplay Mr Godeseth’s evidence because he was called to deal
with certain matters, and Mr Jenkins was not. The Post Office has to abide by the consequences of
their choice of witnesses, in terms of the evidence now before the court to resolve the Horizon Issues
trial. I have already explained my view of Mr Godeseth’s evidence in Part E above, and it need not be
repeated.
Inaccurate Statements by the Post Office
RLITO000005
RLITO000005
517.Litigation in this jurisdiction is adversarial. What that means is each side advances their own
case, and challenges or rebuts the case of the other side, and the court assesses the evidence before it
adduced by the parties, makes necessary findings on the facts and applies the law. Parties in civil
litigation will usually advance their own evidence of fact, and although mechanisms are available for
compelling other witnesses to attend and answer questions, there can be some pitfalls in doing so and
issuing a witness summons (what used to be called a subpoena) is a relatively rare step. The court can
only therefore resolve the case on the evidence before it, although it can draw inferences, that is to
say common sense conclusions, on the evidence it does have.
518.The Post Office has no obligation to assist the claimants in advancing their case against it. The
Post Office has, however, maintained publicly that it was seeking to be “transparent” about Horizon,
and prior to the litigation it made certain public statements in relation to the increasing disquiet on
the part both of SPMs, and others who became involved either on their behalf (such as some Members
of Parliament) or in an investigative way (such as the BBC Panorama programme and other
journalists). These statements by the Post Office routinely and strongly insisted that there was nothing
in the criticisms being levelled at the accuracy of Horizon, and that losses that were shown in SPMs’
branch accounts were caused either by carelessness or dishonesty on the part of the different SPMs
who experienced what they considered to be unexplained discrepancies and losses. In 2015 a
Parliamentary Select Committee held hearings into the mediation scheme that had been set up to
attempt to address the claims by SPMs in respect of Horizon. I have not considered any evidence
submitted to the Select Committee, or any details of the mediation scheme (which was being
conducted by Second Sight) and refer to them here simply as a matter of chronological record.
519.The claimants rely upon a public statement released by the Post Office in 2015 after the BCC
Panorama programme. Part of the statement was headed “The Horizon System” and stated:
“The Horizon Computer System
Horizon is robust and effective in dealing with the six million transactions put through the system
every day by our postmasters and employees at 11,500 Post Office branches. It is independently
audited and meets or exceeds industry accreditations. There have been 500,000 users of the system
since it was introduced.
Nevertheless, rigorous re-investigations were undertaken into claims made by 136 mainly former
postmasters that the system caused losses in their branches.
There is overwhelming evidence that the losses complained of were caused by user actions, including
in some cases deliberate dishonest conduct. The investigations have not identified any transaction
caused by a technical fault in Horizon which resulted in a postmaster wrongly being held responsible
for a loss of money.
Q . . A ing al : ,
to the system, Transactions as they are recorded by branches cannot be edited and the Panorama
programme did not show anything that contradicts this.”
(emphasis added)
520.Another document from 2015 upon which the claimants rely is an internal email chain, which
originated from Paula Vennells, then the Chief Executive of the Post Office, on 30 January 2015. This
was prior to her appearance before the House of Commons Select Committee in February 2015. She
RLITO000005
RLITO000005
posed the following question in an email sent internally to Mark Davies and Lesley Sewell, both of the
Post Office:
“Dear both, your help please in answers and in phrasing those answers, in prep for the SC:
1) “is it possible to access the system remotely? We are told it is."
What is the true answer? Lhope it is that we know this is not possible and that we are able to explain
why that is. Lneed to say no it is not possible and that we are sure of this because of xxx and that we
know this because we have had the system assured.”
(emphasis added)
521.Ms Vennells obviously needed to know whether the answer matched her understanding, which
was - as she put it, both “I hope” and “I need” -- that it was not possible to access the system
remotely. This query was passed on through various people, including at one stage from James
Davidson who has both a Fujitsu and Post Office email address, who answered to Mark Underwood:
“As discussed, can you hook up with Kevin to review what answers have already been provided to
second sight as this should form the Post Office response.”
522.The answer was provided by Mark Underwood on 30 January 2015 in an email which is part of
the same email chain or string:
“Can Post Office or Fujitsu edit transaction data without the knowledge of a Subpostmaster?”
Post Office confirms that neither it nor Fujitsu can edit transaction data without the
knowledge of a Subpostmaster.
There is no functionality in Horizon for either a branch, Post Office or Fujitsu to edit, manipulate or
remove a transaction once it has been recorded in a branch's accounts.
The following safeguards are in place to prevent such occurrences:”
(various matters are then listed in the remainder of the email)
(bold present in original)
523.This then was subject, in the email chain, to a degree of refinement. Kevin Lenihan forwarded the
email onwards to Mark Underwood and others, and stated:
“Mark / Mel,
James has had a look at your answer to Q1. And thinks there’s too much detail for Paula - this was
written for a different type of audience. He has captured the same points but in a more appropriate
format :- He states:-
{remote access) are not understood. I suggest that Paula is briefed along the lines of the following.
1)No transaction data is held locally in any branch. Transactions are completed and stored in a central
database and copies of all data is sent to a secure audit database.
2)Sub-postmasters directly manage user access and password setting locally so system access (to
create transactions) are limited to approved local personnel only who are responsible for setting their
RLITO000005
RLITO000005
own passwords. Users are only created following an approval process which requires authorisation by
the sub-postmaster. All subsequent transactions are recorded against the id used to log on to the
system.
3)Once a transaction has been completed, there is no functionality (by design) for transactions to be
edited or amended. Each transaction is given a unique number and ‘wrapped’ in a digital encryption
seal to protect its integrity. All transactions are then posted to a secure and segregated audit server.
4)On approval, there is the functionality to add additional transactions which will be visible and have
a unique identifier in the audit trail. This is extremely rare and only been used once since go live of
the system in 2010 (March 2010)
5)Support staff have the ability to review event logs and monitor, in real time, the availability of the
system infrastructure as part of standard service management processes.
6)Overall system access is tightly controlled via industry standard ‘role based access’ protocols and
assured independently in annual audits for ISO 27001, Ernst and Young for IAS 3402 and as part of
PCI audits.”
(emphasis added)
524.1 do not see how the statement that “I do not know how the fundamentals around this question
(remote access) are not understood” can sensibly be made. Nor do I know what the expression “the
fundamentals around this question” in fact means, in plain English. The question from Ms Vennells
was very straightforward. It was as follows, using her words, and separating out each clause of the
enquiry:
1."Is it possible to access the system remotely? We are told it is.”
2."What is the true answer?”
525.Both of these are really very simple questions. Question 1. means can it be done, because Ms
Vennells is being told it can be done. Question 2, “what is the true answer?” means she is seeking the
true answer to question 1. This is not complicated, either in linguistic, computing or even business
terms, nor is it difficult to understand. She then expressed her aspirations in terms of what the true
answer might be. “I hope it is that we know this is not possible and that we are able to explain why
that is. I need to say no it is not possible.” (emphasis added)
526.This trial has shown that the true answer to the enquiry she made in early 2015 was “yes, it is
possible.” It has taken some years, and many tens of millions of pounds in costs, to reach that answer.
Kevin Lenihan, who wrote that he did not understand “the fundamentals around this question (remote
access)”, seems to me to have avoided providing a simple answer to a simple question. It is not
necessary to consider why that was, whether from a lack of understanding on his part, or otherwise.
527.1 am unaware of whether there is any other final answer to Ms Vennells internally to her very
straightforward question, or of what she in fact told the Select Committee. None of the people
through whom this email chain passed were called as witnesses in the Horizon Issues trial. However,
the point in bold in the email quoted at [522] above is now known, as a result of this litigation, simply
to be incorrect. Such editing can take place without the knowledge of the SPM. All and any previous
statements by the Post Office stating that this cannot be done are simply factually wrong. The more
detailed points at (1) to (6) of the email quoted at [523] provide only selected information and, in my
judgment, are entirely off the point. They have the effect of obscuring what ought to be a simple
RLITO000005
RLITO000005
answer, with a level of detail which makes the actual answer highly unclear. The answer at (1) applies
only to Horizon Online. For Legacy Horizon, transaction data was held locally in the first instance, on
what was called the counter. This was explained very clearly by the Post Office in its opening,
including in its oral opening submissions. The first sentence of (3) in the list in the email is not
correct. The statement in (4) - which may refer to the Transaction Correction tool - is correct (or at
least, not incorrect) but only so far as it goes, and in my judgment crucially omits the ability to edit
transactions without visibility to the SPM. It also fails to consider the existence of APPSUP.
permissions at all, which every member of SSC had. The experts are agreed that the APPSUP
permission was very powerful and enabled an SSC employee to do pretty much whatever they wanted,
to paraphrase the Post Office’s own expert. It can also be seen, particularly by the evidence that
emerged in the cross-examination of Mr Godeseth, that in a great many cases the PEAKs show that
user error was attributed to problems that were not user error at all, and that others within the Post
Office (for example its own auditors and ROMEC engineers) had either witnessed for themselves and/
or satisfied themselves that user error was not the cause of specific issues, but Fujitsu decided they
were user error in any event.
528.The extent of permissions enjoyed by SSC was the subject of some express concern by Ernst and
Young (the Post Office’s own auditors) in 2011, and although the position may have been regularised
by 2015, certainly prior to 2011 (which covers the
Legacy Horizon years) the situation was very different. The email also states “he has captured the
same points” yet the most important point to which an answer is given, or certainly the answer given
in bold in the answer at [522] above, is not corrected, even though it is plainly wrong. The overall
tenor of what Ms Vennells received, if it reflected the full content of this email chain, would have been
that the Post Office and Fujitsu could not edit transaction data without the knowledge of the SPM. The
evidence in this trial is directly to the contrary, and Fujitsu could do so. It is important to bear in mind
the distinction between the way Legacy Horizon, and Horizon Online, work. Referring to keeping data
on the “counters” is a Legacy Horizon expression. This refers to the way the data was held at the
branch; these were discs (including mirror discs) at the branch which contained the data. The discs
were contained in terminals on the counter. In Horizon Online, the data was held at a central database
or server (the phrase used by the Post Office in its oral opening).
529.This tenor of these public statements made by the Post Office was maintained into, and as part of,
the group litigation. For example, certain statements were made in the Generic Defence in these
proceedings. A party’s Defence is the formal pleading document which sets out a party’s legal position
in the litigation. Since the reforms in civil litigation adopted after what are called the Woolf Reforms,
which led to the introduction of the Civil Procedure Rules in 1998 which govern all civil litigation in
the jurisdiction, such pleadings must be accompanied by a Statement of Truth. These are, for larger
companies, usually signed by directors or very senior personnel, which occurred in this case. That
statement attests to the truth of the matters stated in the pleading.
530.This issue was dealt with in the pleadings in the following way. By this point the Post Office had
accepted that there was some limited ability to do that which it had previously stated was not
possible. In the Generic Statement of Case, the claimants set out the following part of their case:
“25, Further, the Defendant was, by itself and/or via its agent Fujitsu, able to alter branch transaction
data directly and carry out changes to Horizon and/or transaction data which could affect branch
accounts.
26. However, the Defendant has made public statements in the following terms:
RLITO000005
RLITO000005
26.1."Horizon does not have functionality that allows Post Office or Fujitsu to edit or delete the
transactions as recorded by branches" (Defendant's published reply to Second Sight's Briefing Report
Part Two, concerning a review of the Horizon system); and
26.2."Transactions as they are recorded by branches cannot be edited" (Defendant's published reply
to the BBC Panorama documentary in relation to Horizon).
27. These statements were untrue, as the Defendant now admits.”
531.These paragraphs were pleaded to by the Post Office in its Generic Defence in the following
terms:
“57. Paragraph 25 appears to be concerned with the editing or deletion of transaction data input by or
on behalf of a Subpostmaster without his or her consent. Accordingly, Post Office assumes that it is
not concerned with transactions such as Transaction Corrections which are sent to branches but must
be accepted by or on behalf of the Subpostmaster before forming part of his or her branch account. As
to the circumstances in which such transaction data can be edited or deleted without the consent of
the Subpostmaster:
(1) Neither Post Off Euii as H anal
so as to conduct transactions.
(2)A Post Office employee with "global user" authorisation can, when physically present at a branch,
use a terminal within the branch to add a transaction into the branch's accounts. The purpose of
"Global User" authorization is to allow access to the systems for during training and/ or audits. Any
7 ff G U: - G User ID .
identifi
entered at the branch. Their intended purpose is to allow Fujitsu to correct errors or bugs in Horizon
by cancelling the effect of an error or bug on a branch's data. They may be conducted only by a small
number of specialists at Fujitsu and only in accordance with specific authorisation requirements. They
are rarely used. To the best of Post Office's information and belief. only one Balancing Transaction has
tf : . . 1
Claimant. A Balancing Transaction is readily identifiable as such.
(4)There are a small number of Fujitsu specialists who have certain privileged user access rights
which they could in theory use to amend or delete the transaction data for a branch. The intended
purpose of privileged user rights is system support, not the alteration of branch transaction data. To
have abused those rights so as to alter branch transaction data and conceal that this has happened
would be an extraordinarily difficult thing to do, involving complex steps (including the writing of
sophisticated computer programmes and circumvention of sophisticated control measures) which
would require months of planning and an exceptional level of technical expertise. Post Office has
never consented to the use of privileged user rights to alter branch data and, to the best of its
information and belief, these rights have never been used for this purpose.
(5)Post Office cannot conceive of a reason why any Fujitsu personnel would have sought to add, inject,
amend or delete any transactions in any branch accounts so as to create a false shortfall. It would for
all practical purposes be impossible for any of them to generate significant shortfalls without
RLITO000005
RLITO000005
detection and, even if they were able to do so, they would be unable to take the benefit of such
shortfalls for themselves.” (emphasis added)
532.This stance was maintained by the Post Office in the evidence served on its behalf for the Horizon
Issues trial, until service of Mr Roll’s 2nd witness statement. To be fair to the Post Office, its origin
was the witness statements served by Fujitsu employees, rather than Post Office employees. The
position within the Fujitsu witness evidence,
prior to its correction by the later statements from Mr Parker, was that what Mr Roll said was possible
on Legacy Horizon, and what he had himself done, was simply not possible. Indeed, Dr Worden
considered it sufficiently clear that as an IT expert he felt able confidently to assert in his Ist Expert
Report that he, Dr Worden, had “established” that Mr Roll’s evidence of fact in this respect was
wrong. After service of Mr Roll’s 2nd witness statement, Fujitsu finally came clean and confirmed (via
Mr Parker) that what Mr Roll said was correct. Data could be altered by Fujitsu on Horizon as if at the
branch; under Legacy Horizon, transactions could be inserted at the counter in the way Mr Roll
described. This could be done without the SPM knowing about this. Mr Godeseth also confirmed that
it would appear as though the SPM themselves had performed the transaction. This is directly
contrary to what the Post Office had been saying publicly for many years.
533.Therefore some of the earlier public statements made by the Post Office, and the important one
contained in the Post Office’s pleaded Defence to which I have referred above, were factually untrue
in at least one highly important respect. This concerns the ability of Fujitsu to insert transactions into
a branch’s accounts remotely, without the SPM being aware of this, and without the transaction being
identifiable in the transaction data as having been inserted remotely in this way. The term “remotely”
means from a location elsewhere than within the branch itself.
534.Although the phrase “remote access” was used during the Horizon Issues trial, the parties did not
always use the expression in the same way as one another. The Post Office’s definition was explained
in oral opening in the following way:
“When I talk about remote access I'm talking about action taken remotely to either inject new
transactions or to edit existing transactions or to delete existing transactions in a way that could
change the accounting position of the relevant branch.”
For the purposes of this judgment, I too use the term “remote access” in this way. The true picture is
now that, following the Horizon Issues trial, there is evidence of transactions recorded by branches
being altered through remote access by Fujitsu; and transactions as they are recorded by branches
being edited; that this can be done without a SPM’s knowledge or permission; and without being
identifiable as having been done remotely. Dr Worden was asked about the 2015 Post Office statement
in response to Panorama, which he said he had not read, and he explained that it was the audit
records in the audit store that could not be edited. He was not prepared to say the 2015 statement
was factually wrong. The BRDB records could be edited, and he also referred to the joint statement in
which the two experts had agreed that they could not say that anything could not be done. He was
very hesitant, if not reluctant, to state that the 2015 statement was wrong.
535.In my judgment the 2015 Post Office statement plainly is wrong, I find that the 2015 statement
was not true. Again, to be fair to the Post Office, this was accepted in the litigation and at the hearing
before Senior Master Fontaine. The quotation from the transcript of that hearing when this was
addressed by the Post Office in oral submissions is as follows:
RLITO000005
RLITO000005
“Master, first of all, could I just deal with the remote access point? The letter to which my learned
friend took you was, as you might expect, written by people who thought it was correct. The Horizon
system is a very complicated system. It involves lots of departments in ... both in Fujitsu and in the
Post Office. And the people who are responsible for the correspondence didn't know that, in fact,
there were these two other routes. Very few people at Post Office knew that there were these two
other routes. They were ... they were routes that are under ... essentially under the control of Fujitsu
who's the expert independent contractor that is involved in the operation of the system. And it is a
matter of enormous regret that the people who wrote that correspondence and made those
submissions weren't aware of that but, you know, we are where we are; the point is that, the point
having been discovered, the Post Office wasted no time in ... in bringing the truth ... the accurate ...
and accurate set of facts to the knowledge of the claimants.”
(emphasis added)
536.The claimants rely upon the way this sequence unfolded, and the admission by the Post Office
that its previously public statements were untrue, as making it particularly important that the true
and accurate position was then provided by the Post Office. Indeed, the submissions I have quoted at
[535] above make it clear that the court and the claimants were being told that the Post Office had
wasted no time in bringing the truth, “the accurate set of facts”, to the knowledge of the claimants. As
the analysis of the Defence and the evolution of the Fujitsu evidence of fact on remote access for the
Horizon Issues trial demonstrates, the accurate set of facts did not emerge at that time either. The
truth, the accurate set of facts, has only emerged as a result of the final supplementary statements for
the Horizon Issues trial that were served, and only finally (so far as Fujitsu is concerned) after their
witnesses had initially stated clearly in their witness statements that Mr Roll was wrong. Eventually
these witnesses accepted that he was right. The supplementary witness statements that accepted this
were dated 29 January 2019. Prior to that date, the opposite position had been maintained by the Post
Office.
537.The Post Office in its opening submissions stated the following:
“Taking remote access as an example, the need for remote intervention affecting branch accounts will
obviously be rare. On any view, the occasions on which privileged users at Fujitsu have exercised their
ability to remotely inject, edit or delete branch transactions or accounting entries will represent a tiny
So, compared with the volume of business recorded in branch accounts, the number of cases in which
false data will have been remotely introduced will be extremely small (multiplying a small chance by a
small chance). This is a “second order effect” (a small proportion of a small proportion) which is, by
definition, extremely unlikely to have any significant impact on the robustness of Horizon.” (emphasis
added)
538.That submission misses the point, in my judgment. It elides two matters, namely whether
something is technically possible, and the number of times that it has in fact been done. The former,
whether it was possible, had been expressly denied, and that denial is now shown to be wrong. The
latter, the number of times it was done and
with what effect, is a different matter. There is also very little, if any, evidence that is relied upon by
the Post Office to justify the assertion that it was a “tiny percentage” of times. That is a subjective
assertion of very limited weight.
RLITO000005
RLITO000005
539.1 consider the significance of the previously factually untrue statements to be considerable. The
statement was made publicly by the Post Office, turned out not to be factually correct, and the Post
Office gave an explanation and said the full set of facts was now available. The situation was pleaded
to by the Post Office in its Generic Defence, with a statement of truth. That too turned out not to be
correct, and the true position has only emerged in the Horizon Issues stage of the litigation as a result
of the evidence of Mr Roll, which I have dealt with above. It was only following his written evidence
that Mr Parker, and Mr Godeseth - both senior Fujitsu employees - prepared their supplementary
witness statements correcting their first statements. These first statements, as I have explained
above, were simply untrue in that important respect. These witnesses had previously stated that this
was not possible. Mr Parker said Fujitsu did not have the power to do this.
540.1 find as a fact that Fujitsu do have the ability to insert transactions into branch accounts on a
remote basis (in other words, remote access as the Post Office defined it in opening submissions
existed) and this could be done without these being visible to the SPM in that branch in question,
either at the time or subsequently. It also follows that this could be done without an SPM in this
position having knowledge of this, and therefore without consenting to it. Someone who does not
know something is happening cannot give permission for it to happen. Mr Godeseth also gave
important evidence about how such transactions would appear. It is not possible to say, in blanket
terms, that permission was not sought and/or given in some cases. Indeed, Dr Worden accepted that
the APPSUP permissions meant that - as he put it - Fujitsu personnel could do “pretty much
anything”. On Day 18 he said in cross-examination the following:
“Q. Dr Worden, you knew that a central issue, not only a central issue legally but a very high-profile
issue in the case, was the extent to which Post Office had remote access to the counters, didn't you?
You knew that?
A. Yes, and what I'm talking about, what I was talking about was the extent to which this could happen
without the knowledge of the subpostmaster.
Q. And that's the ~
A. And we agreed in the joint statement that more or less Fujitsu or Post Office could do anything.”
541.He was asked about this answer two days later.
“Q. It is Day 18, page 67. I think it is lines 5 and 6 actually. It says: "Answer: And we agreed in the
joint statement that more or less Fujitsu or Post Office could do anything."
A. Yes. We can see precisely what we agreed in a minute, obviously.
Q. In relation to the -
A. We agreed the experts couldn't demonstrate that they couldn't do everything. I mean, that's sloppy
wording by me there. I think the joint expert statement says it better.”
542.The exact wording in the joint statement in respect of repairing transactions is as follows at 10.2
of the 4th Joint Statement:
“Certain facilities and procedures used by Fujitsu to repair the more common issues which arose in
Horizon were standardised, and evidence of them persists. However, to repair less common issues
which arose from time to time, standard tools and procedures might not have been sufficient, and
RLITO000005
RLITO000005
evidence might not persist of what was done at the time. Even when evidence does persist, it may be
extremely difficult for the experts to interpret it today, because of the scale and complexity of Horizon.
Therefore, it is usually difficult for the experts to make categorical negative statements of the form: 'X
or Y never happened’.”
543.This litigation is between the claimants and the Post Office. Fujitsu is not a party. Although one of
the contracts between the Post Office and Fujitsu was uploaded to the electronic trial bundle on the
penultimate day of evidence (by the Post Office’s solicitors) the litigation is not concerned with the
details of that contractual relationship, other than (tangentially only) the cost to the Post Office of
making ARQ requests, as the ARQ data is maintained by Fujitsu and charges are raised to the Post
Office in this respect. However, Fujitsu’s involvement does cast something of a shadow. That contract
has been addressed in terms of whether certain documents were, or were not, in the control of the
Post Office. I have already dealt with the issue of PEAKs and KELs, and how the Post Office
maintained initially it was not obliged to disclose these. Fujitsu personnel who were not witnesses
were involved in providing information to the Post Office’s expert, Dr Worden. It was the Fujitsu
witnesses who originally, and in my judgment unequivocally and directly, flatly contradicted what Mr
Roll said could be done on the subject of remote access. I have also identified that in my judgment, the
majority of the Fujitsu witnesses were more interested in following some sort of Fujitsu party line
than they were in answering questions in cross-examination wholly frankly, although I exempt Mr
Godeseth from that criticism. From what was said by the Post Office’s leading counsel to Master
Fontaine at the very first case management hearing, those within the Post Office had relied upon
Fujitsu in obtaining relevant information for its earlier public statements, and it plainly relied upon
Fujitsu in terms of evidence of fact in the Horizon Issues trial, because it called a number of Fujitsu
witnesses. I am unaware of the degree to which the Post Office has depended upon Fujitsu in the
background to the Horizon Issues trial, and there is no reason why I should be, and I make it clear
that I do not speculate on this. This information must have originated from Fujitsu at some point, but
the Post Office must bear some responsibility for such incorrect statements having been made before,
both publicly and in its pleadings.
544.Nor could it be said, in the Post Office’s and/or Fujitsu’s defence, that remote access was a side
issue. It has been a very important central element of the whole dispute between SPMs and the Post
Office over the Horizon system for many years. Mr Green for the claimants adopted a non-computing
analogy, namely that of “a back door”. The origin of this was a phrase of Mr Roll, who referred to the
back door to the system. The branch accounts can be likened to a locked room, to which the SPM has
the key. This development in the case is comparable, after many years of the Post Office and Fujitsu
maintaining that only the SPM has the key, that only the SPM could perform anything within the
locked room, and if the Post Office and/or Fujitsu
needed access to the room the SPM would know because they would have to borrow the key, to now
being an admission by the Post Office (and Fujitsu) (and a finding of fact by the court) that there is a
back door which they could, and did, use without the SPM even knowing such a door existed. This
analogy is not entirely apt, because software and remote access to edit transactions is not completely
the same as physical access through a door, but it is an attempt to capture the general idea in lay
terms.
545,The truth concerning remote access has now emerged in 2019, in group litigation that started in
2017. I find it notable that the truth did not emerge in the first Fujitsu witness statements that were
originally served for the Horizon Issues trial. Such statements stand as the evidence in chief of
RLITO000005
RLITO000005
witnesses of fact. They are supposed to be accurate. Minor corrections are not unusual and indeed are
almost expected, as a trial approaches, as witnesses either research further or remember (when
preparing for trial) other minor details. This topic, however, did not undergo that type of correction,
and is a subject far more central and important than that. The truth only finally emerged in later
statements, which were required to correct what I find were directly inaccurate statements in the first
witness statements of Mr Godeseth and Mr Parker. There has been no adequate explanation for the
contents of those first statements, which not only omitted this important fact, but contained evidence
directly to the contrary. Those first witness statements were misleading. The statement in the Defence
was misleading too. It ought also to be noted that the truth did not emerge internally within the Post
Office in the email answers provided to internal inquiries in 2015 by senior Post Office personnel,
such as the Chief Executive, who posed the specific question in preparation for providing evidence to
a Select Committee and asked: “What is the true answer?”
546.She also said in the same email “I hope it is that we know this is not possible and that we are able
to explain why that is”. The true answer is that, contrary to her aspiration, it was possible.
547.She also stated “I need to say no it is not possible and that we are sure of this because of xxx and
that we know this because we have had the system assured.” The true answer to that was also “yes, it
is possible”.
548.It is also difficult to believe that the signatory of the Statement of Truth in the Generic Defence
would have signed that statement if she did not believe the contents of that pleading to be true. There
is another relevant feature in this case, which is that the Generic Defence also includes a
counterclaim seeking damages against the SPM claimants, including damages for fraud. Fraud is the
most serious allegation that can be brought in civil litigation and there are special rules in relation to
pleading it, which means that a pleading containing a fraud allegation should be subject to particular
scrutiny before it is served.
549.It may therefore be that the Post Office itself fell into error as a result of information provided to
it by Fujitsu on this important matter. It may be that some within the Post Office were themselves
surprised by these revelations prior to, and during, the Horizon Issues trial. There is no need for me
to speculate on this, and I do not do so. Certainly Mr Godeseth did not appear to have known about
this for very long. Whatever the origin of this, and whether it came from Fujitsu, internally, being less
than frank with the Post Office or not, the effect is that the Post Office has made specific and factually
incorrect statements about what could be done with, or to, branch accounts in terms of remote access
without the knowledge of the SPM. The evidence in this trial has made it clear that such remote
access to branch accounts does exist; such remote access is possible by employees within Fujitsu; it
does exist specifically by design; and it has been used in the past.
550.It follows that the previously stated public position of the Post Office to the contrary, in the
statements to which I have referred above, is specifically wrong in fact.
551.Further, and as part of the Post Office serving supplementary witness statements from both Mr
Godeseth and Mr Parker in respect of Mr Roll’s evidence, the Fujitsu witnesses adopted a position in
the litigation, the rationale of which is somewhat difficult to understand. Responsibility for the
incorrect and potentially misleading evidence of Mr Godeseth and Mr Parker (in their first statements)
was effectively laid at the door of Mr Roll for giving what was said to be vague evidence, or evidence
that, it was said, had not been fully understood by Fujitsu. Infallibility is a rare commodity, and
everyone is capable of making mistakes. However, it is how one reacts to mistakes that is telling. In
this instance, the initial reaction of the Fujitsu witnesses was to seek to shift the blame for their own
RLITO000005
RLITO000005
misleading written evidence upon someone else. In this case, that “someone else” was their former
Fujitsu colleague whose very evidence was responsible for exposing the full picture. This is not
something that could have been arrived at lightly. To be fair to Mr de Garr Robinson, it was not a point
he adopted on the Post Office’s behalf during the Horizon Issues trial. In my judgment, he was entirely
sensible in not doing so. That it was done at all in the written Fujitsu statements speaks volumes, in
my judgment, about Fujitsu’s reaction to being caught out.
552.Mr Parker’s supplementary statement blaming Mr Roll said the following:
“In paragraph 20 of Roll 2, Mr Roll describes a process by which transactions could be inserted via
individual branch counters by using the correspondence server to piggy back through the gateway. He
has not previously made this point clear. Now that he has, following a discussion with colleagues who
performed such actions I can confirm that this was possible. I did not mention it in my first witness
statement because, when faced with a less clear account in Mr Roll's first statement, my recollection
was that if it was necessary for the SSC to inject a transaction data into a branch's accounts, it would
have been injected into the correspondence server (injecting via the server was the default option
which was followed in the vast majority of cases).”
553.The recusal application was issued the day after Mr Godeseth’s cross-examination had made it
clear, not only that this remote access existed, but after he was taken in careful cross-examination
through specific examples of Fujitsu personnel manipulating branch accounts, and leading to
discrepancies in branch accounts. I am aware that criticism of the Post Office and Fujitsu in this
respect may prove to be controversial, as earlier criticism of certain aspects of the Post Office’s case
in Judgment (No.3) was not well received by it. However, if criticism is justified, I consider it would be
detrimental to proper resolution of the group litigation if that criticism were to be withheld simply
because it might lead to a further negative reaction by the Post Office. It is also an inherent part of
the judicial function in any litigation to make findings, which may include criticisms where justified,
that may be contrary to a litigant’s own view of the merits of their case. Some litigants are so
convinced of the righteousness of their own position that they consistently refuse to accept any
possible view of the litigation other than their own. Such a blinkered view is rarely helpful, and would
be particularly unhelpful from a publicly owned institution.
554.1 consider that criticism is justified of the Post Office’s incorrect previous statements - which
included public and high profile statements such as the response to the BBC Panorama programme,
whatever may have been said on the subject to the Select Committee, as well as in its own pleadings -
about remote access without the knowledge of SPMs. The Post Office should have done its best to
discern whether such remote access was possible when this subject first arose, and whether it had
occurred, before these statements were made. Fujitsu should have been frank and unequivocal,
internally, with the Post Office, so that there could be no possibility of incorrect statements on this
important point being made publicly by the Post Office. The Fujitsu witnesses should not, in their first
witness statements, have made the incorrect statements that they did. Had those initial statements
been factually accurate, there would have been no need for the supplementary statements from them
that eventually led to the true position being accepted by Fujitsu, and therefore by the Post Office. It
is highly regrettable that such a situation as this should have developed. I am not making any findings
on any future issues, whether of fact or law, yet to be tried.
555.There is however considerably more to resolving the Horizon Issues than simply making findings
about remote access to branch accounts.
So-called “shadow experts” instructed by the Post Office
RLITO000005
RLITO000005
556.1 return to the issue of so-called “shadow experts.” The parties in this case agreed that costs
management would apply, and the Post Office in its costs budget for the Horizon Issues trial included,
for the costs management hearing on 5 June 2018, an item for incurred expert costs in its costs
budget in the sizeable amount of approximately £800,000. This was in addition to the amount incurred
by that point in terms of the fees of Dr Worden. His costs at that stage were only £58,000. The sum of
£800,000 broke down to about £300,000 paid to Fujitsu, and £500,000 paid to other experts, who
were not being instructed to give expert evidence. Dr Worden was the expert who would be giving
evidence to the court, which meant he would owe the relevant duties of independence to the court
under the CPR. The description of these other experts was given in a skeleton argument for the Post
Office in the following terms:
“The Defendant has spent around £500,000 on investigations by internally appointed experts for the
purposes of determining its litigation strategy. The resulting material - which is privileged - has not
been provided to the Defendant's expert for the purposes of this litigation.”
557.The claimants adopted the term “shadow experts” for these, a term with which the Post Office did
not agree. At the cost management hearings, the claimants pointed out that by “internally instructed”
this meant these experts were not instructed by the Post Office’s solicitors. It was also pointed out
that there was no corresponding entry in the budget for any conference with any of their counsel, and
this meant that this was “an entirely ring-fenced operation”. The Post Office did, however, reserve the
right to recover their costs as costs in the litigation, and therefore this item was included in its costs
budget. All of the material produced by these other experts was said to be privileged.
558.It is a highly unusual situation that entirely separate experts, instructed directly by a party,
without the involvement either of that party’s solicitors or their counsel (all the more so when those
experts are not even identified), are instructed on such a task, whatever that task might actually be. I
recorded an adverse comment in the costs management order of 23 July 2018 stating that the Post
Office’s incurred costs for experts were extraordinarily high, unreasonable and disproportionate; and
that Fujitsu’s costs of assisting with the litigation, and the costs of these internally appointed experts
did not, on the face of it, appear to be properly recoverable sums in the litigation. I should clarify that
this adverse comment should not be taken as applying to Dr Worden’s costs that had been incurred at
that stage, which were modest. It is not necessary to deal with this matter any further to resolve the
Horizon Issues.
F, Documents and Available Information
559.There are certain categories or descriptions of classes of documents that have featured heavily in
the evidence at the Horizon Issues trial. The path to disclosing them has not always been smooth. The
majority, if not all, of the technical documents that relate to how Horizon was actually operating in
fact in IT terms are in the possession of either the Post Office or (more usually) Fujitsu. The two most
important categories, in my judgment, are Known Error Logs (also known as “KELs”) and PEAKs. The
first of these records or logs known errors, which means errors with the Horizon system. The latter is
a browser-based software incident and problem management system used by Fujitsu for the Post
Office account, in other words for incidents and problems associated with Horizon that occur.
560.Disclosure had previously been a troublesome topic between these parties. I had in Judgment (No.
3) on the Common Issues, been somewhat critical of certain aspects of the Post Office’s approach to
the disclosure of some documents. These observations were not well received by the Post Office and
some had been categorised, incorrectly, by Mr Parsons in his 14th witness statement supporting the
recusal application as “critical invective”. Whether the contents of Judgment (No.3) encouraged the
RLITO000005
RLITO000005
claimants to mount, or continue to advance, criticisms of the Post Office’s disclosure in the Horizon
Issues trial (in the sense that they may have felt they had a “following wind”) is not clear. The Post
Office, in the Horizon Issues trial (predominantly in solicitors’ correspondence, but also in some
submissions), also criticised the claimants’ disclosure. For the sake of clarity, I wish to make it clear
that the claimants had no such following wind.
561.The Horizon Issues are decided in this judgment based on all the evidence and the submissions. I
deal with disclosure in the subsequent paragraphs of this judgment in order to assist and guide the
parties concerning the future conduct of the group litigation. Some issues, such as whether certain
Fujitsu documents were within the Post Office’s control, are likely to crop up again in respect of other
types of documents in the future. The court’s interest in disclosure is to promote compliance
with the rules, and also to ensure cost effective progress of the litigation. The answers to the
substantive Horizon Issues are no different because, for example, the experts were not given
particular documents relating to the specific bugs in good time. For the most part, large numbers of
KELs and PEAKs were provided to them, and the experts had an abundance of material. Some 5,000
KELs emerged from Fujitsu well after the trial ended, in October 2019. The experts did not consider
these as they were not given them. Mr Parker had given evidence (at paragraph 61.9 of his 1st
witness statement) that 1,491 KELs had been deleted at the time of writing that statement, but there
was no reference to the existence of this far greater number that emerged after the trial was over.
562.The Post Office’s written closing submissions also included a separate section on disclosure as
follows, which I will quote verbatim:
“I6. The Court's interventions on disclosure
1145.Post Office is concerned that the Court may have lost sight of the nature of the Orders made and
of the approach which the Court ordered. It is striking that Cs have not themselves made any
applications for specific disclosure nor have they advanced any complaint that particular disclosure
orders have not been complied with.
1146.The recent judgment of the Court of Appeal in Serafin v Malkiewicz & ors
(20191 EWCA Civ 852 @ para 118 is relevant:
“We are also highly troubled by the repeated demands and criticisms by the Judge regarding the
Claimant's disclosure, in circumstances where pre-trial disclosure had been completed by both sides
at a time when both the Claimant and Defendants had been represented by solicitors and counsel, and
no application for further disclosure had been made by the Defendants...” ”
563.These submissions identified what were said to have been “the Court’s interventions on
disclosure” in the Horizon Issues trial. The authority identified by the Post Office, and stated to be
relevant to that submission, concerned a defamation case in which the judge had demanded
disclosure of a litigant in person, and made criticisms of him, both during the trial of that litigant’s
libel claim, and in the judgment itself. In some instances the court in that case had, of its own volition,
demanded production of certain documents from the litigant in person, and within a very short
timescale. This had been part of findings made by the Court of Appeal in respect of procedural
irregularity and the successful appeal against the judgment.
564.There were no “interventions on disclosure” by the court during the Horizon Issues trial, nor
were there “demands and criticisms”. When asked during oral closing submissions about this passage
RLITO000005
RLITO000005
in the Post Office’s written submissions citing Serafin, Mr de Garr Robinson did not demonstrate a
great deal, if any, enthusiasm in advancing any claims of procedural irregularity having occurred
during the Horizon Issues trial. He submitted that “this is not that case” - which does raise the
question of why the passages from Serafin were included in the Post Office’s submissions in the first
place.
565.There were, however the following matters that arose in this litigation during the Horizon Issues
trial, which given paragraphs 1145 and 1146 of the Post Office’s closing submissions, should be
recorded:
1.Redactions. In Judgment (No.3) I had identified the issue of unjustified redactions made by the Post
Office of some contemporaneous documents that were used in the Common Issues trial. During the
early part of the Horizon Issues trial some documents were referred to which again had redactions,
which due to their nonredacted parts did not seem to me to be of a type that would necessarily attract
privilege. In view of the previous history of this subject, I asked leading counsel for the Post Office to
perform his own review of the redactions that had at that point been made of documents deployed in
the Horizon Issues trial. This review resulted in some of the redactions in some of the documents
being removed, and unredacted versions of those documents becoming available to the claimants,
some of which were then used in cross-examination. Redactions of 2 documents were maintained on
the grounds of legal professional privilege, which is entirely conventional. He also provided a helpful
two page note explaining the approach that had been adopted. Asking counsel to do a review of this
nature in these circumstances is entirely conventional. There was no short timescale imposed for this
to be done. I asked for a summary of the position on 4 June 2019, which was after the interval which
occurred as a result of the recusal application, and about 2 % months after I had asked him to
perform the review. It was an outstanding matter, of a type sometimes referred to in a trial as
“housekeeping”, which had not been entirely forgotten, but which had been overtaken in the latter
weeks of March when the trial was temporarily halted.
2.A witness statement was ordered from the Post Office to explain the express, and factually incorrect,
submissions made to the court by the Post Office about the Royal Mail’s refusal to produce, at the Post
Office’s request, audit documents. The Royal Mail had, contrary to what the court had expressly been
told by the Post Office, not even been asked by the Post Office for these. The Post Office’s leading
counsel accepted that he had, entirely unwittingly, and on instruction, provided misleading
information to the court and explained and apologised as soon as he discovered this. A witness
statement from the Post Office, which was provided by Mr Parsons, was entirely justified in these
circumstances.
3.A witness statement was also ordered from the Post Office in respect of the production of over 2,000
documents in disclosure by the Post Office in the interval in the trial period between 11 April 2019
(completion of the factual evidence), and the experts commencing their oral evidence on 4 June 2019.
Given the timing and quantity of these, in the circumstances a witness statement (again, being
provided by Mr Parsons) was justified. The first explanation given to the court, again on instruction,
again proved to be wrong. These documents were originally said to have all been of some age (ten
years or so) and discovered at Fujitsu on a “long-forgotten” server. One deployed the day after this
explanation in cross-examination turned out to be dated 21 August 2018. A witness statement was
again called for.
566.None of the above, in my judgment, fall into the category of the court losing sight of the nature of
the orders that had been made concerning disclosure, or of demands or criticisms by the court,
RLITO000005
RLITO000005
unjustified or otherwise. Further, the claimants made numerous complaints of the Post Office about
disclosure. Indeed, the factually incorrect submissions made to the court about the audit documents
and the Royal Mail came at the end of a court day when the claimants, by their leading counsel,
expressly attempted to make a specific disclosure application. I was not prepared to hear that
application at that point. Royal Mail was the correct respondent to such an application, not the Post
Office. Royal Mail was not represented, nor even present in court, nor had any notice been given by
the claimants to the Royal Mail of such an application. Each of these features is a far from promising
start. I did however make directions for the making of a third party disclosure application against the
Royal Mail, but this did not ultimately prove to be required. Once those at the Royal Mail were
actually asked for these documents, the Royal Mail readily produced them. The claimants also made
regular complaints about disclosure and would often point out that documents had been produced
during the hearing. This is because documents were often produced during the hearing. This is
something that had also occurred during the Common Issues trial.
567.The Post Office is, however, correct to point out that there have been no specific disclosure
applications mounted by the claimants, with the exception of the one to which I refer in [566] above in
respect of the Royal Mail. General complaints about disclosure only take a party so far, and that is not
very far, if deficiencies are known about in advance of a trial, and no applications are made.
568.The parties had also voluntarily adopted Model C of the Disclosure Pilot, which I approved. This
was done prior to the coming into force of the Pilot, the full title of which is Disclosure Pilot for the
Business and Property Courts (“BPC”) which became mandatory in all BPC cases where the first CMC
is held after 1 January 2019. This group litigation is not a BPC case and the first CMC was held before
that date in any event. The Pilot is designed to run for a two year period. The Pilot is found at Practice
Direction 51U, which is made under CPR Part 51
The features of the Pilot that are particularly
relevant here are the Principles included in paragraph 2 of the Pilot. These state:
2. Principles, “document”, “adverse” and “known adverse documents”
2.7 Disclosure extends to “adverse” documents. A document is “adverse” if it or any information it
contains contradicts or materially damages the disclosing party’s contention or version of events on
an issue in dispute, or supports the contention or version of events of an opposing party on an issue in
dispute.
2.8 “Known adverse documents” are documents (other than privileged documents) that a party is
actually aware (without undertaking any further search for documents than it has already undertaken
or caused to be undertaken) both
(a) are or were previously within its control and (b) are adverse.”
3.1 A person who knows that it is or may become a party to proceedings that have been commenced
or who knows that it may become a party to proceedings that may be commenced is under the
following duties (“the Disclosure Duties”) to the court—
(1)to take reasonable steps to preserve documents in its control that may be relevant to any issue in
the proceedings;
(2)once proceedings have commenced against it or by it and in accordance with the provisions of this
unless they are privileged;”
RLITO000005
RLITO000005
(emphasis added)
569.The fact that the parties chose Model C does not go to answer the complaints that the claimants
have made about the Post Office’s disclosure. The Post Office had an obligation to disclose known,
adverse documents, as do the claimants as well. “Adverse” as defined plainly includes any documents
that refer to bugs, errors or defects, or the operation of the Horizon system that led to potential
impact upon branch accounts. This obligation is “regardless of any order for disclosure made”, in
other words it exists whether any specific classes of documents are, or are not, already the subject of
any order. The rational for this is obvious. Reliance upon the adoption of Model C is not therefore the
answer to the claimants’ complaints in this respect. However, the fact that the claimants did not make
applications for disclosure is relevant to the criticisms made at the Horizon Issues trial.
570.The Post Office has, essentially, submitted that the breadth of disclosure requests has been too
wide. Particular criticism is that “Freeths’ [the claimants’ solicitors] did not control Mr Coyne’s
requests”. I am not persuaded that it is the correct approach to matters of independent expert
evidence to seek to “control” the documents that an independent expert wishes to see. The claimants
did not issue any opposed applications for disclosure of documents which their expert wished to see,
so a filter of some sort was applied in that sense to the documents Mr Coyne wished to see, and was
not given.
571.Finally, disclosure is a particular feature of litigation in this jurisdiction. However, there was a
clear perception in commercial litigation generally that the then-current disclosure regime had
become inadequate and also that standard disclosure was often excessive in scale, cost and
complexity. This is why the Disclosure Working Group was created in May 2016 at the initiative of Sir
Terence Etherton, then the Chancellor of the High Court and now the Master of the Rolls. This led to
the Disclosure Pilot.
572.It would be highly regrettable if disclosure in this case were to become yet another battleground
between the parties. There are more than enough points in issue between the parties without adding
endless disagreements about disclosure to that list. There are the following areas, however, of
disclosure which should be recorded. In my judgment, the background to the evolution of the
following categories does throw light on the parties’ different approaches to the litigation.
The Known Error Log or KELs
573.KELs are sometimes referred to in the singular - for example, in some places reference is made to
the Known Error Log, as though it were a single Log. In practice, the Log as a whole includes multiple
KELs, as they were referred to during the trial. If an incident or problem occurs, a PEAK is raised, and
if the problem or incident refers to an existing issue which is known to have occurred before,
reference will be made to that KEL entry, or more usually, it is said that the PEAK will refer to the
KEL. Both
PEAKs and KELs have reference numbers; the former are usually all numbers after the prefix PC,
whereas KELs have letters (taken from the identity of the employee who raised the KEL followed by
four digits and a letter). Therefore, when a problem (which I shall refer to generally as an example as
problem x) occurs for the first time, a KEL will be raised which will be given a reference. Subsequent
PEAKs which identify the same problem x again, over time, afterwards will all refer to the same KEL.
Sometimes a PEAK will be identified as including a problem which is similar to problems identified in
more than one KEL, in which case both the KELs will be referred to within that PEAK. Any creation of
RLITO000005
RLITO000005
a new KEL, or updating of an existing KEL, must be authorised by SSC before it can be seen by all
users.
574,The experts agreed the following in the 2nd Joint Statement about both PEAKs and KELs: “KELs
and PEAKs together form a useful source of information about bugs in Horizon but are a limited
window on what happened. It is sometimes necessary to use evidence from both to try to understand,
but even so they are not a comprehensive picture. It is to be expected that both KELs and PEAKs are
incomplete in various respects.” It is also agreed that KELs will often give information about the
impact of a bug or user error and they may also provide information about causes. There are other
documents referred to as OCPs and OCRs. They are Operational Change Proposals and Operational
Change Requests.
575.The subject matter of this litigation spans many years, starting from the first implementation of
the Horizon system (what is now being called Legacy Horizon) in 2000. In the Letter of Claim from the
claimants dated 28 April 2016, the Known Error Log was sought from the Post Office, that letter
stating:
“We understand that Fujitsu maintained a ‘Known Error Log’ for Horizon and that such reports will
have been provided to Post Office. Please see the list of the categories of documents relating to
Fujitsu referred to below, that we request disclosure of.” Item 22 in the list of documents sought was
“The ‘known error log' kept by Fujitsu and provided to Post Office as referred to above, and all
correspondence relating to the same.”
576.The answer in a letter from the Post Office’s solicitors against the specific item 22 was:
“In circumstances where you have not particularised any factual basis on which Horizon is defective,
disclosure of these documents (if they exist) is not relevant, reasonable or proportionate.”
577.The suggestion in that letter that the Known Error Log was not relevant, is simply wrong, and in
my judgment, entirely without any rational basis. The further suggestion, viewed with the hindsight
now available, that the “known error log” may not exist, is disturbing. The claimants’ request used the
precise title - “known error log” - and this clearly did exist. To suggest in an answer “if they exist” is
somewhat misleading.
578.Item 23 in the same list of documents sought was “Internal memoranda from Fujitsu and POL,
referred to by Second Sight as identifying a ‘ Horizon bug' within Horizon Online.”
579.The answer against that item was:
“We do not recognise the document to which you refer. Please provide further details.”
580.In my judgment, the documents sought in that entry must clearly include any PEAKs that
identified a bug within Horizon Online. “Internal memoranda’ is a plural reference, yet it was
interpreted by the Post Office’s solicitors as though it were singular, and the request was for a single
document, or a document with the title “internal memoranda’. This is, in my judgment, obstructive.
581.The claimants were not to be dissuaded, and sought the Known Error Log or KELs again. A reply
from the Post Office’s solicitors on 13 October 2016 is relied upon by the claimants as showing that
the Post Office was denying the relevance of the Known Error Log. This reply stated:
“The claims which you have particularised concern errors with the Core Audit Log. Following a review
of the Known Error Log, Fujitsu have confirmed that there have been no logs in respect of Core Audit
RLITO000005
RLITO000005
Log. The remainder of the Known Error Log does not relate to the claim which you have particularised
(emphasis added)
582.Existence of the Known Error Log was at that stage accepted, but its relevance to the
proceedings was now challenged. The Post Office’s solicitors stated that its contents “did not relate to
the claim” and that “disclosure of this document is not relevant”. Disclosure of it was plainly resisted.
The claimants did not therefore have it when the Generic Particulars of Claim was pleaded on 6 July
2017. In the Generic Defence, which is dated 18 July 2017, the Post Office changed its position, and
now pleaded that the Known Error Log was not in its control. At paragraph 50(4) of the Generic
Defence, the Post Office stated:
“Itis admitted that Fujitsu maintain a “Known Error Log". This is not used by Post Office and nor is it
in Post Office's control. To the best of Post Office's information and belief, the Known Error Log is a
knowledge base document used by Fujitsu which explains how to deal with, or work around, minor
issues that can sometimes arise in Horizon for which (often because of their triviality) system-wide
fixes have not been developed and implemented. It is not a record of software coding errors or bugs
for which system-wide fixes have been developed and implemented. To the best of Post Office's
knowledge and belief, there is no issue in the Known Error Log that could affect the accuracy of a
branch's accounts or the secure transmission and storage of transaction data.”
(emphasis added)
583.This was expressly challenged in paragraphs 3 and 4 of the Generic Reply.
584.The two main points to consider within that passage in the Defence are that the Post Office stated
that it did not have control of the Known Error Log, and that entries within it related to minor/trivial
matters and could not affect the accuracy of a branch’s accounts. This position was maintained,
notwithstanding the terms of the
Post Office’s skeleton argument for the first case management hearing on 19 October 2017. That had
stated:
“Cs’ response to the criticism of their case on Horizon is to argue that they do not have the access to
information and documents that would allow them to plead any properly articulated case as to the
“bugs” that they wish to blame for the shortfalls in their branches. Without prejudice to its position
that the case is not properly pleadable, even at a generic level, Post Office is prepared to take steps to
help Cs investigate Horizon, take an informed view as to whether they really wish to maintain their
claims and, if so, decide how to proceed with them. Post Office wishes to put an end to the speculative
attacks on Horizon and the related allegations as to its own conduct. It puts forward its proposals in
paragraphs 2 to 8 of the Draft Order as a pragmatic compromise between the parties’ competing
interests and concerns.” (italics present in original, emphasis by underlining added)
585.At the first case management hearing itself, upon specific enquiry by the court, this position was
further maintained.
586.Disclosure of the Known Error Log was resisted by the Post Office on the grounds both of control,
and content. So far as control is concerned, Mr Parsons had provided his 4th witness statement for
that hearing, the first before me as the managing judge, which expressly stated at paragraph 35
“Despite Post Office explaining the irrelevance of the Known Error Log and that it was not within Post
RLITO000005
RLITO000005
Office’s control...” and also at paragraph 37 “Due to the large amount of information involved and the
fact that the Known Error Log is not in Post Office's control....”. This position was confirmed by the
Post Office through its leading counsel in the following exchange with the court at that hearing, the
transcript of that showing the following exchange.
“Mr Justice Fraser: Do you still maintain it is not in your control?
A: My Lord, yes. It’s a Fujitsu document.
Mr Justice Fraser: No. Just because it’s a Fujitsu document doesn’t mean it’s not in your control.
A: That’s right. At no point has Post Office ever suggested that it’s in its control so far as I’m aware in
any of the letters that I - Mr Justice Fraser: Okay.
A; (Reading from pleading)
“To the best of Post Office's information and belief the known error log is a knowledge based
document used by Fujitsu which explains how to deal with or workaround minor issues that can
sometimes arise in Horizon for which often because of their triviality system-wide fixes have not been
developed and implemented. It is not a record of software coding errors or bugs for which system-
wide fixes have been developed and implemented. To the best of Post Office's knowledge and belief
there is no issue in the known error log that could affect the accuracy of branch accounts or the
secure transmission and storage of transaction data."
587.This was the relevant passage in the Generic Defence that was read out and Post Office’s leading
counsel then continued:
“Tt will be clear from what I've just read to your Lordship that Post Office thinks that this is a complete
red herring.”
588.The position was then, again, expressly confirmed as follows.
“Mr Justice Fraser: I think it's going to be useful for me, certainly, for you to define all the different
grounds upon which you currently resist disclosure of the known error log. Number (1) is you say it's
not in your control.
A: Control.
Mr Justice Fraser: Number (2) you say the subject matter is not relevant. Is that right? A: My Lord,
yes. An order that Post Office disclose documents relating to bugs and errors causing the branch
account errors, if Post Office had control of the document and it dutifully complied with that order it
would not be disclosing the known error log because, as far as it is aware, the known error log is not
that kind of document.”
589.So far as the content of the Known Error Log is concerned, when asked by the court “does it
contain any errors at all?” the following answer was given by the Post Office’s leading counsel:
“Tt contains things like there's a problem with printers. There's a printer. You have to kick it on the
left-hand side to make the printer work. I mean there's a vast range of hardware problems of that sort
and maybe some software problems .... but not the kind of bugs, errors and defects that the claimants
are wishing to pursue in their particulars of claim so far as Post Office is aware.”
590.This exchange with the court then continued.
RLITO000005
RLITO000005
“Mr Justice Fraser: Well, that's the rider which slightly might concern
A: Well, my Lord, unfortunately, that's all that Post Office can say because it's not Post Office's
document. It's Fujitsu's document. Fujitsu are the experts.”
591.Given there was no witness statement from anyone at Fujitsu available on the subject at that
hearing, that was as far as the matter could be taken on that occasion in terms of making any specific
order relating to the Known Error Log. The Post Office had made clear that any order for disclosure
that did not specify the Known Error Log, but was more general requiring disclosure of documents in
relation to bugs and errors causing branch account errors, would not lead to disclosure of the Known
Error Log in any event. A practical way forward, without requiring a fully argued separate application
which would have included Fujitsu, was adopted whereby the IT experts were permitted to inspect the
Known Error Log. The status of the Known Error Log as a disclosable document was resisted both on
the grounds of control and relevance. It has to be acknowledged that this practical solution was one
suggested by Mr de Garr Robinson.
592.It must therefore have come as a surprise to the Post Office, given the contents of its pleadings
and the express submissions that it had made to the court, that both experts considered the Known
Error Log to be highly relevant. This should also have led to the Post Office beginning to doubt what it
was being told by Fujitsu, given the source of what the court was told about this was what Fujitsu had
told the Post Office, as set out above. The explanation of what the Known Error Log was, what it
contained, and its lack of relevance, was not remotely accurate.
593.Although a certain number of entries in the Known Error Log, which led to a significant number
of different entries which are each called KELs being deployed in the trial and examined closely, were
thereafter available, the issue of control of the Known Error Log and the Post Office’s earlier position
resisting disclosure of this, did not go away. An entire appendix to the claimants’ opening submissions
was devoted to criticism of the Post Office on disclosure, including its shifting position on the Known
Error Log. In closing submissions, at the very end of the final day of the Horizon Issues trial, the
claimants handed up a document entitled “Claimants’ Points in Reply”. Part F of that was a short
document headed “KELs said not to be in the Post Office’s control” together with a variety of
references. It ought also to be recorded here that some 5,000 KELs were later disclosed by the Post
Office in October 2019, well after the trial ended, once the Post Office was told by Fujitsu that
previous entries, which Fujitsu had previously told the Post Office were not retained, were in fact
retained.
594.The subject of whether the Post Office had control of the Known Error Log therefore remained
somewhat live. It simply was not, therefore, going to go away. The Post Office’s legal team had loaded
version 12 of its contract with Fujitsu (“the Fujitsu contract”) onto the electronic bundle database on
the last day of evidence, Day 20, for re-examination of its expert witness, Dr Worden. The contractual
situation between the Post Office and Fujitsu was therefore before the court, which it had not been for
the first case management hearing on 19 October 2017. It is also the case that “control” of documents
(other than the KELs) held by Fujitsu could potentially crop up again in this litigation going forwards.
It is cost effective and efficient to resolve that now, for the assistance of the parties.
595.1 therefore gave both parties the opportunity to lodge further written post-hearing submissions
restricted to whether the Known Error Log was in the Post Office’s control. It was a point relied upon
by the claimants and it has to be resolved. There were two reasons for this. One is that in group
litigation of this type, the parties’ approach to disclosure goes beyond a single document, or type of
document. Guidance as to the court’s general approach will be of assistance in the litigation going
RLITO000005
RLITO000005
forwards. There may well be other documents held by Fujitsu that need to be disclosed later, and the
issue of control needs resolving. Secondly, the court is entitled to expect accurate evidence from
parties on interlocutory matters, and accurate submissions.
596.The Fujitsu contract that was loaded onto the electronic bundle was a recent one. This document
has a version history at the front which shows that it runs, with amendments therein identified, from
31 August 2006 to version 12 on 3 July 2017. Version 12 pre-dates the date of the Generic Defence,
but in any event, clause 25 (which appears below at [602]) does not appear against any of the later
version numbers in the version history of the Fujitsu contract. It is likely therefore that clause 25 in
substantially the same (if not identical) form appeared in all the earlier versions of the Fujitsu
contract prior to Version 12.
597.The claimants made the requested post-hearing written submissions based on the meaning of
‘control’, for the purposes of CPR Part 31.8, which is uncontroversial.
598.That rule provides as follows:
“(1) A party’s duty to disclose documents is limited to documents which are or have been in his
control.
(2) For this purpose a party has or has had a document in his control if - (a) it is or was in his physical
possession;
(b)he has or has had a right to possession of it; or
(c)he has or has had a right to inspect or take copies of it.”
599.Sub-paragraphs (a) to (c) are not exhaustive, but collectively those two subparagraphs mean, in
this case concerning this document or documents, that if the Post Office has, or had, a right to
possession of the Known Error Log, and/or a right to inspect or take or be provided with copies of it, it
is within the Post Office’s control. Control will be established if a party has a contractual right to
inspect or take copies of the document or documents in question. Given the terms of the Fujitsu
contract, the claimants submitted that the Known Error Log is in the Post Office’s control and has
been at all material times.
600.The Post Office’s written submissions on this were also received. These conceded, as it was put in
paragraph 3, that the KEL was in its control. That paragraph stated at sub-paragraph (1):
“As at July 2017, Post Office understood the KEL to be a relatively unimportant internal working
document produced and used by Fujitsu to assist in the performance of some services under the
contract. On that basis, the KEL would n: roperly hav n_charé
the perf of Fujitsu’ Fujitsu C
within Post Office’s control. But, on the facts now known to Post Office, it would not contend the KEL
to be outside its control.”
(emphasis added)
601.The title of the log - the Known Error Log, or known error log - includes rather obviously the two
words “known” and “errors”. The Horizon Issues use the expression “bugs, errors and defects”. The
presence of the word “error” in both might give an early indication of likely relevance of the log.
Further, as shown in [603] below, in the Fujitsu contract the term records includes the phrase “full
and accurate records” (emphasis added). Although records is used in the definition of Records - a
RLITO000005
RLITO000005
point correctly identified by the Post Office, which it says is largely circular - it is clear that “full and
accurate records” are included.
602.Clause 25 in version 12 of the Fujitsu contract states the following:
"25.8 U - . C 25.2 25.3, Fujitsu Servi -
Court Case Support Services to Post Office in relation to prosecutions and other disputes (whether
civil or criminal) with any third party including but not limited to any fraud, theft, breach of contract
or impropriety (the “Court Case Support Services”). The Court Case Support Services shall include
Eujitsu Services to Post Office, its agents or its subcontractors (including Post Office Service
Integrator and any Tower Contractor). Fujitsu Services shall provide the Court Case Support Services
within the timeframes required by Post Office or the relevant court or other authority.
25.9 Without prejudice to Clause 25.3, the Court Case Support Services shall comprise:
25.9.1the provision of copy reports;
25.9.2the provision of data (including transaction data, event logs, helpdesk call logs, non-polled data
nd remuneration di where ch is hel rin th ntrol of Fujitsu Services;
25.9.6live witness evidence at Court if any of the information provided (including without limitation
that provided pursuant to Clauses 25.9.1 to 25.9.5) is challenged to the extent to which Fujitsu
Services provided said information; and
25.9.7the right of access to Records, including but not limited to information, reports and data, held
by_or in the control of Fujitsu Services, and the assistance of Fujitsu Services’ personnel with
appropriate knowledge of the applicable Records (to the extent any such personnel remain employed
or contracted to Fujitsu Services) for any independent experts and/or legal advisors instructed by Post
Office and/or any other claimant(s) or defendant(s) and the Prosecution in any mediation, arbitration
tribunal, court case or dispute in which Post Office is involved in relation to the Horizon and HNG-X or
any other system provided by or on behalf of Fujitsu Services to Post Office.”
(emphasis added)
603.Records is defined in the Fujitsu contract in Schedule 1 “Interpretation”, in the following way:
“Records: means the full and accurate records relating to the performance of the Services.”
I consider that this term plainly includes the Known Error Log, as that relates to the performance of
the Services, and in any event this contains data and hence also falls within clauses 25.9.2 and 3. I
reject the lengthy analysis in the Post Office’s submissions on this point that the KEL is not properly
characterised as being a record. It plainly is.
RLITO000005
RLITO000005
604.In my judgment the Post Office clearly has, and had, a contractual right to be provided with the
Known Error Log by Fujitsu, given these are civil proceedings for (inter alia) damages sought by the
claimants for breach of contract, and also fraud being alleged against the claimants by the Post Office
by way of counterclaim. I do
not accept that the KEL is a type of document covered by the authority cited by the Post Office at
paragraph 11 of its submissions on this point, namely working papers prepared by professionals for
their own assistance in carrying out expert work for their clients; Hanley v JC & A Solicitors [2018]
EWHC 2592 (OB) at [42]. That point is , in my judgment, plainly wrong. Given the terms of the
contract between the Post Office and Fujitsu, it is not necessary to consider that point further.
605.1 consider it verging on entirely unarguable, given the express terms of the Fujitsu contract which
is now available to the court, that the Known Error Log was not in the control of the Post Office. Mr
Parson’s witness statement had merely stated “it was not within Post Office’s control” and I simply
cannot understand the basis for that statement, given the express terms of the contract that the Post
Office had with Fujitsu. It plainly is in the Post Office’s control, given the terms I have reproduced
above, and that point is now conceded.
606.The fact that the Post Office has submitted that in July 2017, on its understanding then, the KEL
was not a “record” that “related to the performance” of the Fujitsu services demonstrates a worrying
lack of knowledge on the part of the Post Office, about both Horizon, and Fujitsu’s record keeping. It
also means, when this is put together with what the Post Office, by its leading counsel, submitted to
the court at the 1st case management conference, that Fujitsu were extraordinarily inaccurate about
the information it provided to the Post Office at that stage of these proceedings. As at 2017, the
Horizon system (both Legacy Horizon and Horizon Online) had been in use for about 17 years. KEL
appears in a table of Abbreviations/Definitions at F/87/3 in a document entitled “CS Support Services
Operations Manual” that is dated 29 January 2001. That document is referred to in footnote 34 of Mr
Coyne’s 1st report, and three passages within it state the following:
“4.5.1 Maintaining the Known Error Log on the SSC intranet site
The SSC generates and maintains a Known Error Log (KEL) system that uses searchable documents in
HTML format. The mechanism for searching is a query entry in an intranet site. The KEL system is
available to first, second, third and fourth line support units as well as SSC staff.
4.5.2 Transferring knowledge between support units
The SSC intranet site has KEL search facilities and other useful diagnostic data, documents and tools.
SSC and SMC staff raise KELs based on customer-observed symptoms. KELs are further maintained
once the fault has been resolved.”
“4.7.1 Known Error Logs (KELs)
The intranet site holds known error details in Microsoft Word format, the contents of which may be
searched for, in full text form. Documents are created to a defined template wherever possible. An
application has been generated which limits the properties of the document to a subset of possible
values, for clarity and ease of search. This application is made available to all support units.
The process for creating KEL entries outside of the SSC has not yet been formulated, but it is
expected that no KEL will be allowed onto the system before it has been authorised by SSC staff.”
RLITO000005
RLITO000005
(bold present in original)
607.The Post Office plainly did not know this when making the submissions that were made in 2017.
The Services Operation Manual had been in existence for 16 years by then, and this ignorance about
the Known Error Log, which is clearly explained within that document, is most surprising.
608.Even without such an express obligation in the Fujitsu contract itself, I find it very difficult to
believe that Fujitsu could mount any sort of coherent or legitimate objection to producing it at the
request of the Post Office. The assertion that the Log was not in the control of the Post Office is, in my
judgment, not a valid one.
609.Similarly unsustainable is the submission that was made to the court in 2017 that its contents
were not relevant to the Horizon Issues, and that it did not contain references to bugs and errors
having an impact on branch accounts. On that latter point, the Post Office’s submissions were based
upon what the Post Office had been told by Fujitsu. Fujitsu were, again, being clearly inaccurate about
what they had told the Post Office was actually contained in the Known Error Log.
610.Nor is the passage quoted above in the Generic Defence an accurate description of the Known
Error Log. The Known Error Log does far more than “explain how to deal with, or work around, minor
issues that can sometimes arise in Horizon for which (often because of their triviality) system-wide
fixes have not been developed and implemented”. The Known Error Log appears to be a
comprehensive record of all the errors and defects of which Fujitsu have become aware over the life
of Horizon. On the basis of the evidence that has been heard in the Horizon Issues trial, there are
many issues in the Known Error Log that could affect the accuracy of a branch's accounts or the
secure transmission and storage of transaction data, contrary to what is pleaded by the Post Office.
Some KELs led to software fixes too. Indeed, Mr Coyne has used it in order to discover the existence
of, and then investigate, the numerous different bugs from what was called “the Bug Table” in the 2nd
Joint Statement, 11 of which Dr Worden agrees in his written evidence. Dr Worden himself has
discovered some of these bugs, from the sample of the entries in the KELs that he had inspected. It is
the KELs that are the origin of these expert conclusions, due to the text in the entries that they
contain.
611.This discovery of these different bugs by both experts has directly flowed from the entries in the
Known Error Log by Fujitsu, together with the associated PEAKs.
612.It is only fair to record that after contested disclosure had been considered at the first hearing
before me on 19 October 2017 (dealt with in detail above), the Post Office’s leading counsel suggested
the “pragmatic solution” that was giving Mr Coyne the opportunity to inspect it. It was because of this
that the matter of control was not resolved at that point, as disclosure was permitted. Mr Coyne
inspected the Known Error Log at Fujitsu’s premises. He found it highly relevant. The claimants
received most of the KELs they had for the trial on 10 May 2018, with further KELs in respect of
particular bugs disclosed on 16 November 2018, one month after the first expert
reports of Mr Coyne and Dr Worden had been served. Further KELs were provided in 2019, and a very
substantial quantity of 5,000, well after the trial in October 2019. There has been no detailed
explanation for that very late post-trial disclosure in any form of any witness statement from anyone
at the Post Office or Fujitsu.
613.The KELs provided to the experts prior to their reports featured centrally not only in Mr Coyne’s
two reports, but those of Dr Worden too. They were also the subject of a great deal of cross-
RLITO000005
RLITO000005
examination by both sides. I consider that disclosure of the KELs to the experts - the pragmatic
solution suggested by leading counsel for the Post Office - has been central in the discovery and
investigation of the bugs, errors and defects that the experts agree were, or are, present in the
Horizon system, and also of the bugs, errors and defects that are not agreed, but upon which both
experts opine and in respect of which I make findings in the Technical Appendix. Dr Worden has also
analysed them extensively, and devoted a separate appendix of his report to them. The best
description of the KELs is taken from Dr Worden, who stated that they were “a rich source of
evidence”. The notion that they were not relevant, or did not contain relevant material, is extremely
difficult to fathom, and I do not understand why, or how, Fujitsu would or could sensibly choose to
inform its own customer of many years, the Post Office, directly to the contrary. If Fujitsu’s description
of the contents of the KELs had been taken at face value and not challenged, then the knowledge now
available to the two experts and to the court about the extent of bugs, errors and defects would not
have been available, with the obvious detrimental impact upon the fair resolution of the Horizon
Issues.
614.The further developments, after the trial, that led to disclosure of another 5,000 KELs by the Post
Office after the trial in October 2019 are dealt with at [627] below and following under the heading
“Further post trial disclosure”.
PEAKs
615.Turning to PEAKs, upon inspecting the KELs at Fujitsu's premises in 2018, Mr Coyne discovered
the existence of a different category of documents called PEAKs, as these were referred to within
different KELs. These documents were requested from the Post Office in July 2018. The claimants
maintain that the Post Office were obstructive over this, but whether that is correct or not, over
218,000 different PEAKs were disclosed by the Post Office’s solicitors on 27 September 2018. This
was two weeks before the date for exchange of experts’ reports. A further 3,866 were disclosed on 25
October 2018 - two weeks after the experts’ first reports. OCP documents (which are mentioned in
PEAKs) were disclosed on 24 January 2019, and a number of PEAKs were identified by Mr Coyne in
his supplementary report which he had identified but which had not been disclosed. These were
provided (or extracts of them) on 18 February 2019, which is only three weeks before the Horizon
Issues trial started.
616.A large quantity of further documentation was then disclosed by the Post Office, on dates that
were not only well after the dates ordered for expert evidence to be served, but on dates that were
even after the intended end date of the Horizon Issues trial itself. 2,590 OCRs were disclosed on 18
April 2019, and other PEAKs were disclosed on 31 May 2019, literally one working day before Mr
Coyne was called to give evidence on 4 June 2019. This highly unsatisfactory situation led to my
seeking an explanation. I consider an explanation was plainly required given the timescales. This is a
large quantity of documents, dealing with central elements of the Horizon Issues, and the stage at
which they had been provided was very late. PEAKs are important documents. I was told in
submissions that Fujitsu had discovered an old database that had been copied more than ten years
previously, and as soon as Fujitsu had told the Post Office this, matters had moved speedily and the
Post Office had provided the contents of this old database to the claimants.
617.However, the day after this explanation was given, and during cross-examination, a PEAK dated
21 August 2018 was put to Dr Worden and the court was told this was one of the documents that had
been disclosed on 31 May 2019. This plainly could not have come from “an old database” in excess of
RLITO000005
RLITO000005
ten years of age which Fujitsu had only recently discovered. A further and more accurate explanation
was plainly called for.
618.I had already ordered another witness statement to be provided by the Post Office’s solicitors,
this one being the 18th witness statement of Mr Parsons, which was served on 19 June 2019. The
contents of that witness statement were highly unsatisfactory. Firstly, it showed that the existence of
the forgotten Fujitsu database came about because of questions raised by the Post Office’s solicitors
“during preparation for the Horizon Issues trial at the beginning of April 2019”. The Horizon Issues
trial in fact had an enforced interlude, from 21 March 2019 when the recusal application was issued,
until that application was dismissed on 9 April 2019. In other words, those questions were asked very
late in the trial process. Further, Mr Parsons’ witness statement stated, so far as the latter disclosure
of PEAKs on 31 May 2019 is concerned, the following:
“20. In the course of investigations into the bugs referred to by Mr Coyne and/or Dr Worden, including
in the Second Joint Statement, on 29 March 2019 Mr Lenton identified PCO273234 to Ms Simmonds
as being a document relating to the Drop and Go Bug. For the next couple of months my firm
continued to investigate these bugs and a number of further documents relating to these bugs came
to light. PCO273234, along with 17 other documents (excluding redundant image files) that had come
to my firm's attention when preparing for the Horizon Issues trial were disclosed in a single batch on
31 May 2019.”
(emphasis added)
619.Mr Lenton is a member of Fujitsu personnel. The 2nd Joint Statement was signed by the two
experts on 25 February 2019. This passage means that one month after that document, Mr Lenton
identified the specific document relating to the Drop and Go Bug. This is one week after Mr Godeseth
had completed his evidence. I find the rather glossed-over reference in the witness statement to
“continuing investigations” that went on “for the next couple of months” extraordinarily opaque, in
the context of a trial that commenced on 11 March 2019, and a document identified on 29 March
2019. I also find the wholesale lack of any explanation of when the “further documents relating to
these bugs came to light” - and indeed how - puzzling, again in a witness statement ordered
specifically to explain what was already a highly unusual situation. Further, the PEAK relating to the
Drop and Go bug was plainly identified by Mr Lenton, a Fujitsu employee, on 29 March 2019. It is in
my judgment verging on the extraordinary, in the context of any trial that was actually underway, that
it
took two months more for this PEAK to be disclosed to the claimants. This is particularly concerning
in the context of this trial between these parties on these issues. It is also obvious from the witness
statement that the PEAK in question dealing with the Drop and Go bug was not disclosed immediately,
until “for the next couple of months my firm [ie the Post Office’s solicitors] continued to investigate
these bugs.” There is no reason that investigations, whether ongoing or not, should delay disclosing a
document of this nature to the other side, particularly in the middle of such a trial as this.
620.The conclusion to be drawn from this is that disclosure was given in a manner that could only
have disrupted and delayed proper investigation of the issues contained in the documents. In this
specific case, that includes the Drop and Go bug. That document should have been provided to the
claimants very rapidly once it came to the attention of the Post Office’s solicitors, not kept “for the
next couple of months”, as Mr Parsons puts it. In subsequent trials, if relevant documents of this
nature are discovered by either side or their legal advisers, it is a matter of some importance that
RLITO000005
RLITO000005
disclosure is given very rapidly. This is particularly important whilst the trial is underway. A failure to
do so risks disruption, delay and increased costs.
621.The experts agreed the following about PEAKs and their content. “PEAKs record a timeline of
activities to fix a bug or a problem. They sometimes contain information not found in KELs about
specific impact on branches or root causes - what needs to be fixed. They are written, by people who
know Horizon very well. They do not contain design detail for any change. They are generally about
development activities and timeline rather than about potential impact. PEAKs typically stop when
development has done its job, so they are not likely to contain information about follow-on activities,
such as compensating branches for any losses.” It is also agreed, and indeed can be seen from the
actual PEAKs themselves, that some of them record observations of financial impact. PEAKs do not
however always use the symbol “£”, which was a search item used by Dr Worden to create a set of
259 KELs that contained that symbol. Even where they referred to a financial discrepancy, PEAKs
sometimes just recorded the figure, or sometimes used “pds”, and did not use the symbol “£”.
Release Notes
622.A spreadsheet listing the Release Notes was produced which shows that there have been a great
number of changes and updates both to Legacy Horizon and Horizon Online over the years. The total
number of Release Notes is 19,842 at final version. Each expert provided an estimate of how many
changes per week there have been to Horizon. Mr Coyne’s estimate is approximately 19 changes per
week. Dr Worden’s estimate is 5 changes per working day. Given the number of working days per
week, these figures are broadly the same. No details of the actual Release Notes were provided. The
spreadsheet was not, in my judgment, complete, nor was it in useful usable form. No Release Notes
detrimental to the Post Office’s case have been produced. The Post Office adopted an unhelpful
approach to Release Notes, requiring the claimants to identify from the very difficult spreadsheet
which Release Notes they would like to see, with an offer to provide those. On 14 February 2019 the
claimants’ solicitors indicated which ones they did want to see. On 20 February 2019, the Post Office’s
solicitors stated “we are currently taking instructions in relation to the requests”. Both the request
from the claimants, and the response, are wholly unhelpful
and far too late. To be fair to the Post Office, a late request will undoubtedly lead to a late response.
Further, the spreadsheet itself is almost wholly unusable.
623.1 have commented upon the approach of the Post Office to disclosure in this litigation before, and
not in favourable terms. At the first CMC in October 2017, I said in a short ruling on disclosure that
the Post Office had been “obstructive” in its pre-action behaviour in this respect, and that its attitude
to disclosure up to that point had been somewhat less than ideal. In Judgment (No.3) I made some
further criticisms, which it is not necessary to repeat. It is regrettable that the Post Office’s approach
to disclosure, admittedly this time in conjunction with Fujitsu, has continued in different respects to
fall short of what is required. However, it does seem to be improving and it is to be hoped that such
improvement will continue. Particularly in litigation such as this, with enormous distrust and suspicion
on both sides, it is simply counterproductive to have that atmosphere added to by an unnecessarily
combative stance on disclosure.
624.The claimants submitted in their opening that the Post Office’s approach to disclosure had the
effect of impeding the claimants from obtaining a full view of the documents and the totality of the
Horizon system. I agree with that submission. However, balanced against this, it does have to be
remembered that the claimants did not issue any applications for disclosure, as I have already
explained.
RLITO000005
RLITO000005
625.The Post Office’s case in this litigation will be subject to exactly the same degree of measured and
independent assessment, and weighed in the same objective way, as the claimants’ case, and as it
would be were these failures not present. However, there must be a change in approach to this group
litigation. Disclosing documents late, and in the middle of trials, in such sizeable numbers, and failing
(as clearly identified from the passage of Mr Parsons’ own witness statement referred to in [618]
above) to act with expedition when clearly relevant documents are unearthed actually during a trial,
has two distinct effects. It is disruptive to the proceedings, and it leads to an increase in the costs and/
or causes delay. It is the antithesis of co-operation, which the Civil Procedure Rules expressly require.
I consider that these passages within this judgment should provide sufficient encouragement to a
more constructive approach to prompt disclosure of obviously relevant documents when (or if) they
are discovered shortly before, or even during the course of, forthcoming trials.
626.A PEAK discovered on 29 March 2019 relating to the Drop and Go bug ought to have been
disclosed to the claimants within a period of about 48 to 72 hours, not two months. With modern
electronic communication, that is a readily achievable timescale.
Further post-trial disclosure
627.There was then yet further post-trial disclosure given by the Post Office during September 2019.
However, the solicitors for the Post Office also notified the court in a letter of 3 October 2019 that
further previous information given to the claimants and to the court in the Post Office’s e-disclosure
questionnaire was, again, wrong. The letter to the court stated inter alia:
“We write further to the email sent by the Claimants' solicitors to the Managing
Judge's clerk on 27 September 2019 at 12:29, which brought to the Managing Judge's attention that
32 documents relating to the Horizon Issues Trial were disclosed by Post Office on 25 September
2019.
During the course of responding to a number of queries that the Claimants' solicitors had raised in
relation to the disclosure of these documents (which can be located at [H/358] of the Horizon Issues
Trial Bundle), it has come to Post Office's attention that a statement within its EDO is incorrect and
Claimants. Please find enclosed a letter which was sent to the Claimants' solicitors today explaining
the background to these documents coming to light and the proposals which have been made in
respect of providing any disclosure which may be sought by the Claimants.”
(emphasis added)
628. The actual statement that was wrong was further explained in the accompanying letter to the
claimants, which stated:
“In respect of your query regarding the fact that intervening changes to KELs are not captured on the
face of the KELs disclosed, we have made further enquires with Fujitsu to confirm our understanding
that previous versions of the KELs are no longer held. We regret to say that these enquiries have
revealed that our understanding was wrong.
As you will be aware, Post Office's EDQ stated that "[t]he KEL only contains the current database
entries and is constantly updated and so the current version will not necessarily reflect the version
that was in place at the relevant time. The previous entries /versions of the current entries are no
1 ilable"
RLITO000005
RLITO000005
This statement was based on info provided by Fujitsu. In response to our recent enquiries, however,
we were informed by Fujitsu on 30 September 2019 that this is incorrect and that previous versions of
KELs are available.
Thi Post Off ise. ink , . Fuii
f fill EDO , Iti is inf ‘
was incorrect.
Regardless of whether these documents may or may not be “adverse", Post Office has taken
immediate steps towards arranging for the previous versions of these KELs to be extracted and has
instructed Fujitsu to begin producing a script to extract the documents into a readable HTML format.
Unless you disagree, once this script has been produced we will ask Fujitsu to extract all of these
versions.
When this has been done, again unless you disagree, we propose to take immediate steps to disclose
previous versions of the KELs which were referred to by either Dr Worden or Mr Coyne in their expert
reports or the joint statements or the bug table, or were for any other reason included in the trial
bundle.
Disclosure of the previous versions of these KELs (where there are previous versions) will be given as
a matter of urgency.
We invite you to tell us whether the Claimants also wish to be provided with disclosure of previous
versions of the KELs which were not referred to by the experts or included in the trial bundle.
Gi < f thi if M ing f thi
immediately and will send him a copy of this letter under cover of an explanatory email.”
(emphasis added)
629.This was correctly characterised by the Post Office’s solicitors as a serious matter. Further, when
these KELs were produced and disclosed, it turned out that there were approximately 5,000 of them.
This is an extraordinary number, and far more than were available to the experts for the trial.
630.There was no explanation of the way that what was obviously now incorrect information - that
previous versions/entries of KELs were not retained by Fujitsu - had been communicated to the Post
Office’s solicitors, or by whom, and there was no explanatory witness statement provided on a
voluntary basis. I did not order one, nor did this disclosure interrupt preparation of the draft
judgment, or lead to any application for further evidence or cross-examination.
631.1 include it therefore for completeness. There are two points that arise from this.
1.It would have been a matter for argument, and an exercise in discretion, as to whether any further
submissions or evidence (including cross-examination) would be permitted, had such an application
been made. The Post Office were pragmatic and sensible, and essentially left this decision to the
claimants. The claimants made no such application. However, had they done so, my preliminary view
(without hearing any argument) is that I would not have permitted the Horizon Issues trial to be
resumed and further evidence adduced. This is for three reasons. Firstly, there has to come a point
when the parties are given a decision on the Horizon Issues. The trial had already been interrupted to
a substantial degree by the recusal application. This trial was supposed to be completed in April 2019.
Resuming it would have led to that date becoming sometime in early 2020. The over-riding objective
requires consideration of other court users in other cases. Secondly, subsequent stages of the group
RLITO000005
RLITO000005
litigation would, once again, become consequentially delayed. It is with a sense of wistfulness that I
recall a statement at [16] in Judgment No.3 that this litigation would be speedily progressed. Thirdly,
the parties have told me that they wish to mediate before Christmas 2019 and would like to know the
outcome of the Horizon Issues trial to assist in that process. This would not be possible if the trial
were resumed.
2.This extensive further disclosure does, however, rather reinforce the criticism which the claimants
level at the Post Office and Fujitsu. The Post Office are the party in the litigation, and they are the
party with the disclosure obligations. They obviously rely upon Fujitsu in this respect. Quite how
Fujitsu, a reputable company with a worldwide reputation who have been involved in Horizon for
nearly 20 years, could be so wrong on yet another important point going to Horizon is not explained,
and will remain unclear. Given the statements made by Fujitsu, relied upon by leading counsel for the
Post Office at an earlier hearing and repeated to the court, that the KELs did not contain any material
relevant to bugs, errors and defects, a pattern has emerged on the part of Fujitsu. This is that its
statements about KELs have been shown to be clearly wrong on more than one occasion, and in
important respects.
Criticisms of the disclosure of the claimants
632.1 turn therefore to the criticism that the Post Office has levelled at the claimants. It should be
borne in mind that the bulk of the disclosure burden fell upon the Post Office, simply because of the
nature of the Horizon Issues. However, that does not mean that the claimants were not themselves
obliged to give relevant disclosure to the Post Office. This disparity is obvious in the number of
documents disclosed by each party, which the Post Office in its Closing Submissions stated was some
517,965 documents by the Post Office, and only 1,525 by the claimants. Obviously that previous
number does not include the 5,000 KELs disclosed by the Post Office in September and October 2019,
after the submissions. The comparison in the figures demonstrates the different burden upon the
claimants and the Post Office. It is also correct to identify that for many categories of very important
documents, the Post Office is reliant upon Fujitsu. The claimants are not.
633.The Post Office submitted that despite it making appropriate and narrow requests for Model C
disclosure of documents held by those claimants who were witnesses in the Horizon Issues Trial, the
claimants have not engaged in discussions concerning the scope of the claimants’ disclosure nor
provided disclosure of the documents sought by the Post Office.
634.The Post Office, in the context of its complaints about the claimants’ disclosure, submitted that “it
was therefore not appropriate for claimants to serve claimantspecific evidence from current or former
SPMs"” because of the nature of the Horizon Issues and the terms of the Order that ordered witness
statements, and that the Post Office sought to avoid making disclosure requests nearer to the trial.
However, once the claimants did so, and served their nine witness statements in evidence in
September 2018, the Post Office’s solicitors wrote to the claimants on 22 October 2018, 7 November
2018, 30 November 2018 and 20 December 2018 and did not receive any response. Even though the
Common Issues trial was underway for part of this time, I do not see why such letters should have
gone unanswered by the claimants’ solicitors. When the response did come, it was to the effect that
there would be no further disclosure as this was not envisaged by existing orders.
635.The Post Office contrast this with the disclosure it was prepared to give in respect of specific
witnesses, and also stated that “given that claimants had flouted that Order, Post Office sought to
understand what searches (if any) had been conducted by claimants so that it could understand
RLITO000005
RLITO000005
whether further disclosure would be required in light of claimants’ evidence. A further request for
details by claimants was made by Post Office on 17 January 2019.”
636.I am invited to contrast this lack of response by the claimants, which it is said should be viewed in
light of the approach adopted by the Post Office. One example of the further disclosure that Post
Office had agreed to give which was outside that ordered by the Court was disclosure made in respect
of the “operation of branches by Angela Burke, Akash Patny, Anup Patny, Jayesh Tank, Setpal Singh
and Adrees Latii
637.The Post Office complains that the claimants “clearly intended disclosure to be a onesided
exercise”, seeking disclosure of this nature but refusing to provide their own claimant-specific
exercise. Mr Tank did search, in February 2019, within an archived Subpostmasters “Yahoo Groups”
forum where he used to post (which means to make comments on the site) and after doing so, Mr
Tank identified a post he made on 13 December 2011 relating to the £195.04 shortfall referred to by
Ms Van Den Bogerd,
and also a post he made on 29 September 2014 relating to the £600 shortfall that he refers to in
paragraphs 6-11 of his witness statement.
638.Mr Tank has also found a letter he received from Post Office’s Agents Accounting Team in
Chesterfield dated 13 October 2014 which related to this shortfall (and which Mr Tank referred to in
his forum post).
639.Mr Tank was cross-examined about this, and explained that at the time of his first witness
statement he “hadn't fully researched the whole background” and he did not realise at that time that
the post he had made previously was as important as it had subsequently become, by which he meant
in the trial. He also said he had only become aware of the £195 loss to which it related once he had
read Mrs Van Den Bogerd’s witness statement. He also explained that he kept a box file and that was
where he found the documents. Complaint is made of the fact that the documents that were disclosed
should have been disclosed on 17 July 2018, the date for disclosure. I accept that, and even if one
makes the assumption in the claimants’ favour that they did not know at that date that Mr Tank was to
be a witness, certainly no later than the date of his first statement (which was 28 September 2018)
disclosure ought to have been given. Mr Tank’s lack of understanding of relevance and disclosure
should not have arisen, and the existence of his box file should have become known to the claimants’
solicitors far earlier than it was.
640.Further disclosure was also necessary in relation to Mr Latif. This came about because he served
an amended witness statement on 1 March 2019, which came about following enquires with Mr Latif's
former branch manager, who consulted the documents within the branch to confirm the date on which
particular transaction corrections had been issued.
641.He explained this in his cross-examination:
“ Q. First question about that: by amendment to your statement you now say that this was in around
January 2018 rather than March 2018. A. Correct.
Q. What caused you to make that correction?
A. Thad a look at the -- we hold the records for the information in the office, so I had my assistants
look at the records, transaction logs and that's when I confirmed that it was January rather than
RLITO000005
RLITO000005
March. (Inaudible) was logs in March as well. We made calls effectively every month to Horizon help
desk concerning this issue.
Q. When do you say you asked your assistants to check about the date?
A. Yes.
Q. Sorry, when do you say that happened?
A. It was after I made the initial statement, I was checking.
Q. Roughly when, Mr Latif?
A. It would have been a few weeks ago, sir.
Q. So is it right that you didn't check those records from the branch before making your witness
statement?
A. No, I thought I was correct but I double checked and made sure that actually in fact they were
correct, those (inaudible), so I was right but initial incident happened in January when TA (inaudible)
transaction acknowledgement, the TC, the corrections, they came in March.
Q. Okay. So you say, do you, that the TCs in relation to these transaction acknowledgements came in
March 2018? A. Yes,
Q. And you say that's something you have checked from your records?
A. Yes."
642.The Post Office made requests for disclosure during the trial, on 15 March 2019 (which was the
end of the first hearing week) requesting disclosure of various documents relating to Mr Latif and Mr
Tank and their answers in cross-examination. This also included requests of CCTV recordings which
Mr Tank said he had looked at. These requests were refused on 27 March 2019, and the claimants
stated that this refusal was on the basis that they were either outside their control, no longer existed
or were not relevant to the claimants’ evidence and were not relied upon at trial.
643.Documents that were held in Mr Latif's former branch were plainly outside of his control, even
though his former branch manager had accessed them previously. However, that does not mean that
the former branch manager could not at least have been asked for them. Neither the claimants did
that, nor indeed did the Post Office (so far as I can tell). I consider the claimants’ attitude in this
respect unhelpful, although it should be noted that for at least the documents in respect of Mr Latif,
the Post Office could have taken its own steps in this regard.
644.There are some unsatisfactory aspects to the Post Office’s submissions about the CCTV footage.
Mr Latif clearly said in his first witness statement at paragraph 7 “I also checked the CCTV to make
sure that I had performed the transaction correctly.” Secondly, the Post Office’s closing submissions
refer to requests for “the CCTV recordings which were consulted by Mr Tank for the purposes of
drafting his witness statement”. He was asked about the CCTV during his evidence and did not say
that he had consulted it for the drafting of his witness statement, he said that he had checked it at the
time. Had the Post Office wished, it could have asked for that footage shortly after receiving that
statement. It is also the case that the event in respect of which he referred to the CCTV occurred in
July 2015. It may be that the CCTV images do not still survive, so long after the event in question, but
this point was not pursued in cross-examination.
RLITO000005
RLITO000005
645.There is, however, one major difference between the claimants and the Post Office in this respect.
The Post Office were entitled to, and did, explore with specific witnesses certain issues in respect of
disclosure by asking the relevant person (in the above example, Mr Latif) questions about it. The
claimants did not have this opportunity for the following reason. The majority of complaints made
about the Post Office’s disclosure arise as a result of Fujitsu. Save for rare circumstances, such as Mr
Parson’s specific witness statement about the Royal Mail issue, all that the claimants and the court
are usually told is “Fujitsu say X” or “Fujitsu told the Post Office” such and such. Specific individuals
are not identified, and they are certainly not called as witnesses to be asked questions about their
approach, or (for example in respect of KELs) how they could have been so factually wrong in their
explanation of the KELs’ contents. I have formed a dim view of the accuracy of information of this
nature coming from Fujitsu, regardless of the person in that organisation responsible for
communicating it.
646.Notwithstanding this, I wish to make one matter completely clear. If a witness refers toa
particular document in their oral evidence, and the other party in this litigation wishes to see a copy
of that document, then (absent issues of privilege) that ought to be disclosed, even more so if a
specific request is made. This is by far the most sensible and proportionate way forward. It is in no
party’s interests to engage in squabbling about relevance of a specific document in those
circumstances. It costs far more to argue back and forth in solicitors’ letters than it does simply to
give disclosure. In any event, the relevance of the document is made clear by the fact that it has been
referred to orally in evidence.
647.It is also correct to say that notwithstanding the different burden upon the different sides in
terms of disclosure in the Horizon Issues trial, which has undoubtedly been far heavier for the Post
Office, this burden will not always be borne in these proportions. The Post Office also has the benefit
of a far higher number of legal advisers. As at the date of this draft judgment, simply in terms of the
total number of leading counsel engaged in the group litigation, the Post Office outnumbers the
claimants by 4:1. In terms of the number of solicitors’ firms, the Post Office currently outnumbers the
claimants by 2:1; although only one firm was acting for the Post Office on the Horizon Issues, and the
second firm acted for the Post Office on its unsuccessful attempt to appeal the Common Issues.
Summary on disclosure
648.The Post Office is, in my judgment, somewhat at risk of downplaying certain unsatisfactory
aspects of its own disclosure that have emerged during the Horizon Issues trial. For example, the
incident concerning the incorrect statements made to the court about the Royal Mail refusing
disclosure without an order of the court was described in the Post Office’s closing submissions as a
“mix up” which was “regrettable but should not be the subject of criticism of Post Office.” I do not
consider telling the court something in express terms which is factually wrong can properly or
correctly be identified as merely a “mix up”. Two points that were expressly explained by its leading
counsel were simply not factually accurate. The Royal Mail was not refusing disclosure, and it did not
require an order of the court before it would provide certain documents. It had not even been asked
for them.
649.Similarly, in its closing submissions the Post Office referred to the interlocutory discussions about
the Known Error Log as a “debate about relevance”. That again downplays what had happened,
including the positive submissions made, that the Log was specifically not relevant and even if an
order were made in respect of documents showing bugs, errors and defects, the Log would not be
disclosed (even if it were in the Post Office’s control, which it was stoutly maintained it was not). The
RLITO000005
RLITO000005
submissions made by the Post Office on the adequacy of its own disclosure have been rather
undermined, since the trial ended, by the production (emanating from Fujitsu, once again) of over
5,000 KELs, something rightly described by the Post Office’s solicitors as “a serious matter”.
650.The claimants have not been entirely co-operative on their own disclosure, however. I consider
that the criticisms of the claimants for serving what was described as “claimant specific” evidence are
unjustified. At least one of the experts, Mr Coyne, adopted a specific approach and investigated
specific issues or occurrences, and there is no reason that both of the experts could not have done so.
Dr Worden did not, but
that does not mean that the claimants were wrong to do so. Some of the criticisms made by the Post
Office regarding the claimants are unjustified, therefore, but not all of them are. Although I do
harbour the suspicion that the Post Office may have decided that, on disclosure at least, attack may be
the best form of defence, some of the criticisms made in the Post Office’s closing submissions are
justified. For example, for the claimants’ solicitors simply to state that “at present, it is not the
Claimants’ intention to make another round of disclosure in relation to the Horizon Issues trial” as
was done in July 2018, is not co-operative. Further, the reason that documents which Mr Tank had in
his box, that were plainly relevant, were not disclosed was because Mr Tank had not been asked for
them. Had he been asked - which is something that the claimants’ solicitors should have done, once it
was known he would be a witness - these could and would have been disclosed. They certainly should
have been.
651,The following points are of general application to both parties in this litigation. At some point a
party who seeks to complain to the court about disclosure ought either to shelve its general disquiet
on this subject, or make an application.
652.It is important to identify, so that the parties in this group litigation are aware, the court’s
approach to disclosure. This is given for the purposes of guidance, and costeffective and efficient
progress of the group litigation going forwards.
1.Care must be taken when explanations or submissions are made to the court. If further time is
needed in order for accuracy to be achieved, that time should be sought. Those giving instructions to
counsel must be astute to ensure that they are giving correct information.
2.All the substantive issues will be resolved on the evidence, the applicable law, and taking account of
submissions. A party does not obtain a preliminary advantage or “head start” on the substantive
issues because of any criticisms about disclosure.
3.Criticisms about the other party’s disclosure should be proportionate. Depending upon the
circumstances and the nature of the document, the fact that disclosure is given late might be relevant,
but simply because continuing disclosure is given by one party does not mean that there is anything
sinister lurking in the background.
4.The court can always be asked to resolve disputes about documents. Mr Coyne sought a large
number of different types of documents, but no applications were made by the claimants even though
these were not provided. Criticisms of the Post Office’s disclosure by the claimants has to be seen in
the light that the claimants did not consider it sufficiently important at the time to make any
applications to the court. This is not to encourage applications. Sensible and well-advised parties who
cooperate will usually deal with almost all disclosure issues consensually.
RLITO000005
RLITO000005
5.The parties are expected to comply with their disclosure obligations. If a document comes to light
that should have been disclosed but has not been, it should be provided promptly. If this is discovered.
actually during a trial, it must be disclosed very rapidly indeed. As happened with the document
referred to at [618] above, a document concerning the Drop and Go bug was discovered actually
during the Horizon Issues trial, then (as Mr Parsons put it in his explanatory witness statement) “for
the next couple of months” kept by the Post Office’s solicitors while “research” was done. It was then
disclosed just before resumption of the trial “in a batch”. This is not working to the time scale either
required, or expected, by the court, and it is not the correct way to deal with disclosure during a trial.
This is a laissez-faire approach to disclosure of an important document during the trial itself, and the
document should have been disclosed very promptly.
6.Both parties will be held to the same standards. The Post Office has, so far, and given the issues that
have been tried, borne the bulk of the disclosure burden. The claimants are expected to observe their
disclosure obligations in the same way, and will be expected to do so in the future when further issues
and the claimants’ different cases are tried in 2020.
7.Witness statements will be ordered from a party when the court requires an explanation. Nothing
else should be read into the ordering of a witness statement, other than it means precisely that -- an
explanation is required.
8.It is obvious that the Post Office has had to rely upon Fujitsu to a large degree. However, given it
was Fujitsu who told the Post Office what the Known Error Log contained - see further [586] above -
Fujitsu has, so far, shown itself not to be entirely reliable in this respect. Fujitsu are also responsible
for the Post Office making a directly incorrect important statement in its EDQ about retention of
KELs, which led to the disclosure of about 5,000 of these some months after the trial closed.
9.Adoption of the Model C procedure does not mean that parties can keep adverse documents to
themselves and not disclose them until a specific request is made. Such adverse documents should be
disclosed promptly.
10.Finally, disclosure is very expensive. The court will be astute to guard against it becoming either
satellite litigation or a weapon in the interlocutory arsenal.
653.1 have exhorted the parties to co-operation many times before in this group litigation. I will
continue to do so. Eventually, it is to be hoped that they will emerge into the broad sunlit uplands of
the over-riding objective on disclosure matters.
G. The Experts’ Agreements
654.There were four experts’ agreements in this case. Experts’ agreements are very important.
Permission to rely upon IT expert evidence was granted in the First CMC Order dated 25 October
2017. The two experts met on a number of occasions and agreed certain points.
655.As will be seen from Part H of this judgment, the experts approached their respective exercises in
very different ways. However, they did reach some areas of agreement that were very helpful. Their
joint statements or experts’ agreements did however also set out a considerable number of areas of
disagreement. As the trial approached, I requested through the parties that the joint statements of the
experts focus more on areas of actual agreement, rather than recitations of position and
disagreements.
RLITO000005
RLITO000005
656.The 1st Joint Statement was dated 4 September 2018 and set out the following, inter alia, at the
very beginning of the statement:
“Each expert's approach to writing his report, and to this joint memorandum which foreshadows their
reports, could broadly be one of three possible approaches:
a)To focus mainly on negative points found in the disclosed documents about where Horizon may have
fallen short.
b)To focus mainly on those aspects of Horizon which were intended to achieve robustness and
reliability, and the evidence implying that they succeeded.
c)To provide the court with a clear foundation for understanding the design and operation of Horizon;
then, building on that foundation, to provide a balanced assessment of the ways in which Horizon
succeeded, whilst addressing any disclosed issues where Horizon may have fallen short.”
657.This was the first area of disagreement. Mr Coyne stated in the section immediately following,
entitled Areas of Disagreement:
“In my opinion the technical issues can be answered without reference to the Claimants high level
allegations document. The issues are about how Horizon and its interactive components operated and
the processes employed by Post Office and Fujitsu in supporting these systems and the data within.
Whilst my report will take a balanced approach, it is the case that many of the issues require a deep
focus on the occurrences of bugs, errors and defects as well as the potential for modification of
transactional data. Whilst context will be provided as to how Horizon should work in typical
circumstances, it is the non-typical operation where focus will be placed.”
658.This approach was not agreed by Dr Worden. He stated:
“T intend to take the balanced approach (c).
To provide a joint statement at this stage which will be useful to the court, under each Horizon issue
the experts need first to agree what is worth agreeing having regard to the case which is being
alleged. For instance, it would not help the court for the experts to list a number of detailed points as
agreed, if one or the other expert thinks that those points are irrelevant or unimportant to the case
alleged; or if collectively the points imply a view which, in the opinion of either expert, is not a
balanced view. Therefore, the areas of agreement may at this stage be limited and more high level.
My approach will also focus on the Horizon system. It will consider Horizon within its wider business
context, but I do not believe that the Horizon issues extend to offering any opinion on the wider
business practices of Post Office.”
659.I consider that the balanced approach, which both experts stated they would be adopting, was the
correct one. However, I do not find the following statement by Dr Worden in this 1st Joint Statement
as helpful:
“under each Horizon issue the experts need first to agree what is worth agreeing having regard to the
case which is being alleged. For instance, it would not help the court for the experts to list a number
of detailed points as agreed, if one or the other expert thinks that those points are irrelevant or
unimportant to the case alleged.”
RLITO000005
RLITO000005
660.It is a matter for the court to consider how helpful a particular point of detail, agreed by the
experts, is or is not to the issues to be decided. Obviously the parties can make submissions about
that, but it is not for an expert to decide that a point of detail, upon which he may be agreed with his
opposite number, should however not be agreed because that point will not be helpful to the court; or
that it should not be included in a joint statement. It is also not for experts to direct the court in any
particular direction. The Horizon Issues were carefully drafted by the parties, approved by the court,
and specifically included in an Order. They were the issues that the experts were to agree, and
detailed points of an IT nature that arose within those Horizon Issues should, where possible, have
been agreed.
661.Because Mr Coyne and Dr Worden approached their tasks so differently, the former inspecting a
number of PEAKs and KELs about 10 times greater in number than the latter, this meant that a great
amount of very detailed matters shown within those documents were not agreed by Dr Worden. He
approached his expert exercise from what he described as “a top down” approach, a methodology that
found its most stark manifestation in his statistical analysis of likelihood in his Section 8.
662.1 will include within this judgment, not the full text of all four of the different Joint Statements,
but some lengthy passages. This is to identify the large degree of agreement between the experts. I
will however omit (with some isolated exceptions) the passages where the experts set out their
separate, not-agreed, views.
The 1 st Joint Statement
663.1 also quote at this stage from the 1st Joint Statement, signed on 4 September 2018, and the
points of agreement and disagreement in respect of Horizon Issue 1. This is because in my judgment
it is an important point of agreement, in respect of an important Horizon Issue, but also it is useful to
set out the approach within the Joint Statements.
"24 Posi
2.1 Issue 1
Issue 1 - To what extent was it possible or likely for bugs, errors or defects of the nature alleged at
§§23 and 24 of the GPOC and referred to in §§ 49 to 56 of the Generic Defence to have the potential to
(a) cause apparent or alleged discrepancies or shortfalls relating to Subpostmasters’ branch accounts
or transactions, or (b) undermine the reliability of Horizon accurately to process and to record
transactions as alleged at §24.1 GPOC?
Areas of Agreement:
E . f . f ati
ib) mi ranch unts/tran: ‘ion:
Each time any IT system (including Horizon) is changed there is the potential to introduce new bugs/
errors/defects.
Once bugs/errors/defects are discovered, they take time to resolve and therefore systems such as
Horizon often continue to operate with bugs/errors/defects with or without workarounds in place.
D F fe in Hori
a . fi lati S ,
RLITO000005
RLITO000005
In the event that any specific bug/error/defect had such an effect, the experts have differing views as
to the ‘extent’ of the impact that such bugs/errors/defects may have had on branch accounts.”
(Emphasis added)
The statement continues:
“Areas of Disagreement:
Jason Coyne: As identified by select Known Error Logs (KELs), many thousands of bugs/errors and
defects occurred and many of these known errors had impacts that recurred in different
circumstances, therefore the reliability of Horizon to accurately process and record transactions is
questionable.
Such facts undermine the reliability of Horizon to accurately process and record transactions.
Examples:
(the Calendar Square/Falkirk bug - which led to Horizon failing to recognise transfers between
different stock units, thereby affecting Branch accounts;
(ii)the Payments Mismatch defect - which affected at least 62 branches and was related to the process
of moving discrepancies into the local suspense account, thereby affecting Branch accounts. This
defect was not capable of being identified by Subpostmasters, who would have believed from Horizon
that their account was balanced when it was not; and
(iii)the Suspense Account bug - which erroneously replicated suspense account items for 14 branches
from 2010 in the same monthly trading periods in 2011 and 2012, thereby affecting Branch accounts.
In my opinion there is possibility of bugs/errors/defects existing in Horizon that have not yet been
discovered. Further, most bugs/errors/defects that exist in Horizon have existed for extended periods
of time before they were discovered.
I understand that Horizon had at least 19,842 ‘releases’ of new software, each one of these could have
introduced new bugs/errors/defects.
Dr Worden:
My current preliminary view is that undetected bugs/errors/defects in Horizon were very unlikely to
be the cause of permanent shortfalls or discrepancies in a branch's accounts.
It is necessary to define measures for the ‘extent' of issue 1. At least two measures are possible: the
number of such bugs and errors, and their net expected financial impact on claimants' branch
accounts, compared to the total shortfall experienced by all claimants, which is of the order of £18
million. I intend to assess both measures. The latter measure may be more useful, because if any set
of bugs has expected financial impact much less than £18M, it can do little to account for the
claimants' shortfalls, either collectively or individually.
I propose to assess these measures from a number of different sources of information:
(a) the defects of Horizon cited by the claimants in para 1.3 of the outline; (b) the KELs identified by
the claimants in para 1.4 of the outline; (c) other KELs; (d) data on their shortfalls provided by the
claimants in their schedules of information; and (e) data from other sources (if available).
RLITO000005
RLITO000005
Tam still scoping the data needed for (e). If it is available, I will provide it to Mr Coyne as soon as I
have it.
My preliminary analyses of the data on (a) - (4) imply that:
«For the majority of KELs, (which record known bugs), it can be shown, from an understanding of the
robustness measures built into Horizon or otherwise, that they would have no permanent impact on
branch accounts.
«The net financial impact on claimants' branch accounts of any other bugs, including undiscovered
bugs, is very small compared to the claimants’ total shortfalls. I am not yet able to quantify this, but I
intend to do so.”
664.In my judgment, this presages Dr Worden’s statistical analysis of likelihood, which I deal with as
part of Section I. The following text used by Dr Worden should be considered:
“_...if any set of bugs has expected financial impact much less than £18M, it can do little to account for
the claimants’ shortfalls, either collectively or individually” The phrase “much less than £18M” could,
potentially, direct expert consideration of Horizon Issue 1 away from what it is in fact asking, towards
a different issue entirely. The experts were not tasked only with investigating bugs, errors and defects
that had a proven financial impact of (say) £18M, £10M, or any financial figure at all, and certainly
not one in the millions of pounds. This was for the following reasons. It was never part of the Post
Office’s position at the case management stage when the Horizon Issues were discussed and
considered, that there should be any sort of threshold for financial impact, above which the experts
should investigate, and below which they should not. The wording of Horizon Issue 1 has already been
addressed above, and this uses the terms “possible or likely”. This notion of imposing a financial
threshold must have come from Dr Worden.
665.Nor is the litigation intended to be a wide-ranging enquiry into all of the Post Office’s accounts
with all of its SPMs over the 15 or 16 year period in question. The Post Office’s position is that
Horizon is sufficiently robust such that it cannot be the cause of the different shortfalls complained of
by the hundreds of different claimants. The counter-position to that by the claimants is not that
particular discrete sets of bugs had, individually, an impact of millions of pounds upon their branch
accounts. If that were the claimants’ case, then Dr Worden’s approach as set out in this joint
statement might be understandable. Rather, the claimants’ case is that there are a large number of
different bugs, errors and defects that each affected each of their branch accounts over time, with the
end result being that each of them was (in accounting terms) identified as owing the Post Office sums
in the tens of thousands, or thousands of pounds (or less) with the consequences which are well
known. Some discrepancies were only a few pounds, but are said to have occurred many times. Some
were said to be tens of thousands of pounds cumulatively. Some occurred many times; others less
often, but with greater individual effect. One bug - of which more later - is said to “double up”, which
would mean an exponential effect as a loss doubles, then quadruples and so on. If one imposes, at the
beginning of the expert investigation process, a filter or threshold that looks for a “set of bugs”
causing losses of “millions of pounds” that will have the effect of restricting the field of investigation.
There is no reason for such a restriction, in my judgment.
666.On Horizon Issue 2, the experts were agreed, so far as that issue was concerned, that the extent
to which any IT system can automatically alert its users to bugs within the system itself is necessarily
limited, and while Horizon has automated checks which would detect certain bugs, there are types of
bugs which would not be detected by such checks. The experts were also agreed, in respect of
RLITO000005
RLITO000005
Horizon Issue 6, that whilst Horizon contains measures and controls for detecting system integrity
concerns, the automatic mechanisms have failed in the past. The experts did not agree as to the
‘extent’ of prevention, detection, identification, reporting or risk reduction that the automatic and
manual control measures delivered; however, the agreement that automatic mechanisms have failed
in the past, which was contained in the 1st Joint Statement, is in my judgment important. It means
that the Post Office cannot rely upon the presence of automatic mechanisms or processes as being an
answer to the consequences of failures in system integrity.
667.It was also agreed by the experts that the causes of some types of apparent or alleged
discrepancies and shortfalls may be identified from reports or transaction data available to the SPMs,
but that other causes of apparent or alleged discrepancies and shortfalls may be more difficult or
impossible to identify from reports or transaction data available to the SPMs in question, because of
their limited knowledge of the complex back-end systems. Identification required the cooperation of
PO staff and the SPMs. Again, I consider this expert agreement to be important. This was, essentially,
the essence of the complaints raised by Mr Bates himself in the very earliest days of Legacy Horizon,
namely that there was or were insufficient information or reports available to him to enable him to
identify the causes of apparent or alleged discrepancies. The experts are agreed that, for some causes
of apparent or alleged discrepancies this is indeed the case, due to the complexity of the back-end
systems (back-end means not at the branch) and the limited knowledge a SPM would have.
668.The experts agreed that Horizon had evolved, and its robustness may have varied throughout its
lifetime. They also agreed that the level of robustness may have increased or decreased as the system
was changed. The existence of branch shortfalls was agreed, but they were not agreed at the stage of
the first joint statement that this indicated any lack of robustness. They were also agreed, as at the
date of their 3rd Joint Statement, that Horizon as it is in 2019 is relatively robust. However, given the
span of the years under consideration is 2000 to 2010 for Legacy Horizon, and 2010 to date for
Horizon Online, and given the agreement to which I have just referred that robustness may have
increased or decreased, this only takes one so far.
669.On Horizon Issue 4, the experts agreed that there are a number of actual reported errors in data
recorded within Horizon arising from (a) data entry, (b) transfer or (c) processing of data. Therefore,
the potential exists. The experts do not agree as to “its “extent”.” Dr Worden saw this issue as a
subset of Horizon Issue 3. He felt that because of the measures “built into Horizon”, errors of data
entry, transfer and processing were very unlikely to affect branch accounts. The experts agreed on
Horizon Issue 6 that the automatic measures and controls for detecting system integrity within
Horizon had failed in the past; they could not agree as to the extent
that the automatic and manual control measures delivered. Dr Worden also made the point that bugs,
when discovered, could not be fixed instantly and what he called “trade offs” were required when
dealing with practical release management.
670.So far as remote access and Horizon Issue 7 was concerned, at the time of the 1st Joint Statement
Dr Worden was unclear about this and the technical document provided to him by Mr Coyne had not
been received in sufficiently timely manner such that he could consider it. By the time the Horizon
Issues trial had started, the Fujitsu witnesses such as Mr Parker had corrected their earlier incorrect
witness statements and accepted that this could be done. However, those statements were not served
until November 2018 so would not have been in existence as at the time of the 1st Joint Statement.
671.0n Horizon Issue 9, the experts agreed that the causes of some types of apparent or alleged
discrepancies and shortfalls may be identified from reports or transaction data available to SPMs.
RLITO000005
RLITO000005
Other causes of apparent or alleged discrepancies and shortfalls may be more difficult or impossible
to identify from reports or transaction data available to SPMs, because of their limited knowledge of
the complex back-end systems. Identification required the cooperation of PO staff and SPMs.
672.On Horizon Issue 10, it was agreed that by the very nature of rolling out fixes in any IT system,
including those implemented by Fujitsu, these had the potential to affect transaction data or data in
branch accounts. This could be seen as stating the obvious, but at least the experts agreed it. The
experts were also agreed that the use of tools and facilities to do this should be auditable, however
the maintenance of logs recording this would be dependent upon retention periods and the size of the
logs in question. Again, this could be seen as stating the obvious. The experts were not agreed on
Issue 12, how many times the ability to do this was used, if at all. Nor was there any agreement on
Issue 13, the extent to which such facilities had the potential to affect the reliability of branch
accounts; Issue 14, a detailed issue with 7 sub-issues concerning functionality; or Issue 15, which
concerned Transaction Corrections or TCs.
673.The experts agreed, so far as the existence and use of permission controls is concerned, which is
the essence of Horizon Issue 11, that the usage of tools and platforms which existed specifically for
the purpose of accessing and modifying transaction data, should be auditable. It was agreed however
that the maintenance of logs would be dependent upon retention periods and size. Such tools which
were identified initially in the 1st Joint Statement under Horizon Issue 10 areas of disagreement by
Mr Coyne, were at that stage split into the following:
1.Global Branches which would enable the input of transactions within Horizon as though it had come
from an actual Branch;
2.the Branch Transaction Correctional tool;
3.the Transaction Information Processing repair tool.
674.The best description of “Global Branch” or “Global User” is contained in a Post Office document
headed “Global User Process Management” which said that it allowed people such as auditors “to be
able to go out to any branch and use the same log on to access any Horizon kit in Branch without
having to request a single use password at each branch. The individual requiring a Global Use
account has to be approved by a Manager showing on the Global User requester and approver list”.
However, the document also identified that PWC had discovered that “the current Global User list
inherited from Fujitsu contained employees who had left the Business. This raised concerns over the
security of access to Horizon. Although this is a low risk in terms of Horizon access (the individual
would have to gain access into a Branch first) it is still a risk and needed to be mitigated.” There were
32 such former employees. It was in respect of the permission control procedures concerning these
tools that the documented authorisation process was identified, using documents called the “Master
Service Change” or MSC documents and “Operational Control Procedures” or OCPs. These documents
were requested by Mr Coyne but no actual disclosure application was issued. The document to which I
have referred explaining Global User was in the trial bundle.
The 2 nd Joint Statement
675.The 2nd Joint Statement is dated 25 February 2019 and identified and collated the different bugs,
most of which were given actual names, which had either been identified and agreed, or which Mr
Coyne maintained were bugs. This statement was in landscape form and lengthy, and included what
became to be referred to as “the bug table”. I adopt the terms used by the experts for the descriptions
RLITO000005
RLITO000005
of the different bugs, or alleged bugs. The number of different bugs that were agreed by Dr Worden,
by the time the trial ended, was 11. The description of this in the joint statement itself was
“The structure of the document captures i) A table of bugs/errors/defects containing evidence of
financial impact upon branch accounts that both experts agree (or indicate if they do not), ii) all
expert agreements grouped by Horizon issue, and iii) additional comments and observations input by
the respective expert.”
676.Prior to the commencement of this litigation, the Post Office had acknowledged the existence of
only two bugs in Horizon. Mr Coyne’s evidence was that there were 29 relevant to Horizon Issues 1
and 4, and he clarified in his cross-examination that the number that he considered had lasting impact
upon branch accounts was 21. I deal with that cross-examination at [794] to [798] below.
677.The 2nd Joint Statement also said in its Introduction that:
“Because of time pressures and the complexity of the issues, we have not been able to address all the
Horizon issues in this joint statement. We will issue a subsequent joint statement addressing those
issues we have not yet addressed. For those issues we have addressed in this statement (issues 1, 2, 9,
14 and 15), the layout and references are not as polished as we would have wished.”
678.It is extremely useful for the court to have Joint Statements from experts, and I am grateful to the
experts for producing these statements, in particular what was called “the bug table” in the 2nd Joint
Statement, in what is a very complex case, and when they were under considerable time pressure.
The bug table was very widely used in both cross-examination by the parties and indeed in their
submissions.
The so-called “Bug Table”
679.The bug table, and my findings upon the bugs within it, are in my judgment central to the Horizon
Issues. Evidence concerning bugs in the Bug Table formed a large part of the expert evidence that
was tested in cross-examination. The different bugs, or alleged bugs, identified in the Bug Table were
as follows. I have added to the following the summary of the position of the Post Office in Appendix 2
of its Closing Submissions, which in some cases accepted there was such a bug, and qualified that
acceptance in most instances by references to what was said to be the impact of the bug. I deal with
detailed findings on those bugs that are in issue in the Technical Appendix to this judgment. For the
detailed submissions by the Post Office for each bug, the paragraph numbering in Appendix 2 started
again at 1 for each bug. This did not cause particular difficulty, and because the Post Office explained
that Appendix 2 had been put together by different parts having been drafted by different counsel/
solicitor teams, this is understandable. I refer to it here, because otherwise a reader of the passages
below might become confused as to why there seem to be so many paragraphs all (for example)
numbered 2 in Appendix 2. The following summary of the Bug Table is intended to be a useful
summary for this judgment, and is not restricted solely to the entries within the 2nd Joint Statement.
1, Receipts and Payments Mis-match bug.
This was agreed in the bug table as an “acknowledged bug” which had an impact on branch accounts.
Dr Worden added in his comments that “Therefore, the extent of this bug is well established, in the GJ
analysis.” GJ is Gareth Jenkins. The identified year of its effect was 2010. It is accepted by the Post
Office in paragraph 7 of Appendix 2 as one of a number of “bugs with lasting impact (although they
were resolved)”. In the summary in Appendix 2, the Post Office stated that “In the event, however, this
bug resulted in transient impact only.”
RLITO000005
RLITO000005
2. Callendar Square/Falkirk bug.
This too was agreed in the bug table as an “acknowledged bug” which had an impact on branch
accounts. Its identified years of effect were 2000 - 2006. It is accepted as a bug by the Post Office in
paragraph 6 of Appendix 2, but is said to have had “transient impact”. Dr Worden added different
comments, including that “the bug arose from a fault in the underlying Riposte software, so it is not
surprising that it took Fujitsu some time to understand it, or that they had to rely on the suppliers to
fix it. It does not show poor system design or support by Fujitsu”. That latter sentence is a little
surprising, and seemingly very defensive on Fujitsu’s behalf. This is because the Horizon Issues are all
about the operability and functionality of Horizon, not who is to blame for the presence of any bugs,
errors or defects. Riposte software was part of Legacy Horizon. The fact that it is a product designed
and supplied by Escher, and not Fujitsu, is not relevant to the dispute between the Post Office and the
claimants. In any event, and as accepted and recited in paragraph 2 of the Summary of Appendix 2
dealing with this bug, “Mr Coyne asserts that Bug 2: Callendar Square is a bug with lasting financial
impact and in JS2, Dr Worden appears to agree that there is strong evidence of this...”
3. Suspense Account bug.
This was agreed in the bug table as an “acknowledged bug” and its identified years are 2010 to 2013.
It is accepted by the Post Office in paragraph 7 of Appendix 2 as one of a number of “bugs with lasting
impact (although they were resolved)”. Dr Worden’s comments do not include that it had an agreed
impact upon branch accounts. This is because of a concept he introduced (expanded in his reports) of
“transient effect”. What this meant was that if there was an impact on branch accounts by something
within Horizon, but that was then corrected by a TC, he concluded it had a “transient effect” upon
branch accounts and dealt with it differently. I address this concept below. Other comments of his in
relation to this acknowledged bug are:
“Tt was a transient effect arising not from a fault in the software, but from a change in database
archiving policy in 2010. The delay in correcting it arose from a failure of communication between PO
and Fujitsu. Because the bug would only manifest itself annually for any affected branch, the effects of
this delay were not widespread.
Peak PC0223870 shows that Fujitsu were able to identify the branches affected, even when
Subpostmasters did not report it. There is evidence that the branches were compensated, as I would
expect from the normal error correction processes.”
4. Dalmellington bug/Branch Outreach Issue.
This was not acknowledged as a bug in the bug table, but is effectively accepted by the Post Office as
a bug. It is accepted as a bug by the Post Office in paragraph 6 of Appendix 2, but is said to have had
“transient impact”. Indeed, for some time during the trial the Post Office would refer to this as “an
issue”. I find that it is a bug - that much does not seem to be controversial, as paragraph 2 of the
detailed part of Appendix 2 relating to this bug states the following:
“This is a bug which Mr Coyne states has lasting impact on branch accounts. Post
Office submits that there was no lasting impact on branch accounts.”
The years of its effect were 2010 to 2015. It is named in respect of the branch where it was
“discovered” in 2015, although the investigation undertaken by Fujitsu in that year identified previous
occurrences. There were over 110 occurrences of it over this 5 year period (the actual number is
RLITO000005
RLITO000005
probably 116, but at least 114) with 88 branches affected. Some were impacted multiple times. It is
what is called a “cash remming error” and only happened in certain circumstances. It was explained
further by Mr Godeseth in his evidence at [445] onwards above. It would, in my judgment, have an
impact upon branch accounts - it is the extent of that impact that is in issue, whether it was lasting or
transient. Dr Worden relied upon what he described as “a well-tested process of reconciliation and
TCs to detect and correct errors in cash remming (used 20,000 times per year)” and said it was
“straightforward for Horizon to detect any discrepancy between a “rem out” and the corresponding
“rem in” (a mismatch arising either from a miscount, or a multiple count of a pouch) and then a TC
can be issued.” He also added that “this process catches and corrects remming errors, whatever their
cause - including if they arise from, or are provoked by, software faults.” He therefore implicitly
accepted that the Dalmellington bug was a software fault, although he did not say so in terms. I find
that it was plainly was. He was also reliant upon the process of TCs to correct it. Further, the bug was
there for 5 years and was not discovered, although its effects doubtless were. I find that this was a
software bug that impacted upon branch accounts and I will come to the nature of its effect in my
findings on the bugs that are not agreed.
5, Remming In bug.
This was not acknowledged as a bug in the bug table. However, Dr Worden accepted it was a bug or
defect in his cross examination and it is accepted as a bug by the Post Office in paragraph 6 of
Appendix 2, but is again said to have had transient impact. The years it was said to have been present
in terms of effect were “March - August 2010 and recorded as fixed approx. 2011”, Mr Coyne said
there were 14 branches affected. Dr Worden’s entry in the Joint Statement relied upon TCs, and
stated “As for the Dalmellington bug, above - PO had a robust process for detecting and correcting
remming errors, whatever their origin. So, there were no lasting effects on branch accounts.” In
paragraph 2 of the detailed part of Appendix 2 relating to this bug, the Post Office submitted that “any
discrepancy would be transient as instances of this bug are caught by automatic reporting.”
(emphasis added)
The answer to this is therefore dependent upon my answer to the point concerning lasting and
transient impact.
6. Remming Out bug.
This was split into two in the bug table, 6(i) which was identified in KEL acha508S and 6(ii) which was
identified in KEL GMaxwell3853P. The former was identified in February/April 2007, recorded as fixed
approx. 2007, with Mr Coyne identified 57 branches affected; and the latter was in May 2005 with one
branch being affected. They were both remming out issues, hence the grouping by the experts in the
bug table. It was not acknowledged as a bug in the bug table. However, it is now accepted as a bug by
the Post Office in paragraph 6 of Appendix 2, but is said to have had transient impact. Again, for both
of these iterations, 6(i) and 6(ii), Dr Worden used the same wording as he had in the entry for bug 5,
the Remming In bug, namely “As for the Dalmellington bug, above - PO had a robust process for
detecting and correcting remming errors, whatever their origin. So, there were no lasting effects on
branch accounts” (emphasis added). In paragraph 2 of the detailed part of Appendix 2 dealing with
this bug, the Post Office stated that this “comprises two separate issues, only one of which was a bug.
Any discrepancy caused by either issue would be transient as instances of both issues were caught by
automatic reporting.” It was also said that Mr Coyne had conflated two unrelated issues under the
heading “Remming Out”. Given this was an entry in the 2nd Joint Statement agreed by the experts, it
RLITO000005
RLITO000005
is a little unfair to state that Mr Coyne had done this, as the entry was obviously agreed by Dr
Worden. However, it was split into 6(i) and 6(ii) and I will consider each separately.
7.Local Suspense Account issue, not the same as 3. Suspense Account bug. This was reported in 2010
and recorded as fixed in September 2010. It was not acknowledged in the bug table, but is now
accepted as a bug by the Post Office in paragraph 6 of Appendix 2, although it is one of those bugs
said to have had transient impact. Mr Parker had identified 33 branches affected. Mr Coyne recorded
what he said were four associated KELs, namely acha5259Q (for which there were 6 PEAKs)(this KEL.
is mistakenly recorded twice in column 3, with acha5838T only mentioned in the text in that column);
cardc2043L (10 PEAKs); PorterS199P (3 PEAKs); and acha5838T (which states there are “two
different but similar problems” and appears in the text, but not in the list of KELs at the end of the
text by Mr Coyne). Dr Worden’s comments were that
“The KEL acha5259Q implies that PO and Fujitsu were able to identify all occurrences of the problem,
without being notified by any Subpostmaster. I would therefore expect them to have corrected any
impact on branch accounts as part of normal error correction processes.
I would not expect evidence of all corrections to accounts to have survived to the present day. PEAKs
and KELs are not used to record corrections of financial impact.”
He also relied upon a statement by Mr Parker that there was ‘Temporary financial impact which would
have been cancelled out in the following period by a corresponding discrepancy’.
This introduces another concept, similar if not identical to Dr Worden’s “transient impact”, and that is
“temporary financial impact”. Dr Worden is correct that PEAKs and KELs do not record corrections of
financial impact; they do however record financial impact, because when an SPM reaches SSC (which
raises the PEAK) they often start by recording “SPM says he/she has a problem in that......” and
financial impact is often then recorded. The Post Office in paragraph 2 of the detailed part of
Appendix 2 dealing with this bug states that “any discrepancy would not be lasting”. This does
therefore accept, albeit implicitly, that it would have an impact upon branch accounts. It is also
submitted that it was a “teething problem” from the early days of the Horizon Online pilot scheme.
8. Recovery Issues.
The Post Office does not accept these are bugs at all. These are not agreed as bugs by the experts,
and there are four different types included in the table, with years of effect from 2010 to 2018. Mr
Coyne’s entry stated that “The text within the PEAKs and KELs suggests that in each case a branch
account discrepancy would be evident and would require correction by the Post Office.” Dr Worden
stated that:
“The KELs and PEAKs cited by Mr Coyne are not indicative of errors in Horizon. They provide
guidance on how to correct discrepancies caused by human errors or other errors in transaction
recovery (‘recoverable transactions’)
Because there were many such errors, there were many Calls to the help desk and many PEAKs and
KELs. Normally, correction of errors involved back office reconciliation and issuing TCs. This was
accurate and effective; I have derived an upper limit of £2 per branch per month on the mean impact
of erroneous TCs.
One important KEL acha959T was guidance to the back office MSU, not for Subpostmasters”
RLITO000005
RLITO000005
There was a minor typographic error in the submissions in paragraph 4 of the detailed part of
Appendix 2 which referred to bug 9. However, the four different types of recovery issues are
addressed in the subsequent paragraphs and the conclusion paragraphs deal with bug 8. This was
later corrected in a sheet of corrections, which clarified that paragraph 4 of the detailed part of
Appendix 2 should have stated "Mr Coyne states that Bug 8: Recovery Issues is a bug with lasting
financial impact. Post Office submits that it is not a bug at all". The relevant findings on these issues
are dealt with in the Technical Appendix.
9. Reversals.
This was not acknowledged as a bug in the bug table but is now accepted as a bug by the Post Office
in paragraph 6 of Appendix 2, but is said to have had transient impact. Indeed, Dr Worden’s entry in
the 2nd Joint Statement in relation to this stated “Transaction reversals are a complex area which,
like recoverable transactions, are less familiar to Subpostmasters and are more prone to human error.
They lead to many calls to the help line and to many KELs and PEAKs - not necessarily related to any
fault in Horizon.” This can be seen as yet another example of attributing fault to the SPMs where
possible, but given it is now accepted as a bug it is not necessary to consider that any further. Mr
Coyne’s entry in the 2nd Joint Statement stated “In April 2003 due to a failure in regression testing,
Horizon version S30 was released by Fujitsu and this introduced a bug where the value of
transactions reversed by Subpostmasters was shown twice in the amount of the reversal in branch
accounts.”
10. Data Tree Build Failure discrepancies.
This is accepted by the Post Office in paragraph 7 of Appendix 2 as one of a number of “bugs with
lasting impact (although they were resolved)”. The PEAK dealing with this reads “Data trees have
been failing to build fully, and the system has not been detecting this.” Data trees were part of Legacy
Horizon, and is used to build a summary (or picture, the word used by the Post Office in its Appendix
2) of the accounts. The building of data trees is a software function, and given “the system” is Horizon
and is supposed to detect failures of this nature, it is difficult to see how it could ever have been in
issue that this was a bug. Mr Coyne’s entry in the bug table recites that “Dugannon branch suffered a
£43,000 discrepancy but the cause was not immediately known. £52,814.29 at the Yate Sodbury
Branch. £9,368.40 at the Appleby Westmoreland branch.” These are sizeable sums and arose in the
branch accounts. That this is a bug was de facto accepted by Dr Worden in the bug table due to the
text of his entry which states:
“There was a bug which has potential impact on branch accounts, early in the lifetime of Horizon.
Soon after it arose, the error was trapped and detected by DEP and was then soon fixed.
The fault was easily noticeable at branches before the error trapping which was soon introduced and
would be even more noticeable after that. Only three branches appear to have been affected, as
described by Mr Coyne.
Because it was so noticeable at the branch, and the Peak i: mcerned with ware error rather
than any other cause, I would expect any discrepancies in branch accounts to have been corrected.”
(emphasis added)
This is therefore undoubtedly a bug, but findings in respect of this therefore depend upon the nature
of its impact on branch accounts, which are made in the Technical Appendix.
RLITO000005
RLITO000005
11. Girobank discrepancies.
Mr Coyne considered this was a bug. Dr Worden did not and stated that “the first fault concerns
reports. A fault in a report is not a discrepancy in branch accounts, and only causes one if it causes a
person to make a mistake.” This is now essentially accepted as a bug by the Post Office, but it is
submitted it had no branch impact. It is included in paragraph 5 of Appendix 2 under the heading “the
following bugs had no branch impact.” The detailed part of Appendix 2 dealing with this submits that
there is no evidence of any financial impact upon branch accounts, let alone a lasting impact. This was
in the early days of Legacy Horizon. There are said to be six distinct issues arising under this heading,
and further detail on this is included within the Technical Appendix. I do not know if what are called
“giros” are still in use in 2019.
12. Counter-replacement issues.
There were two KELs associated with this dealt within the bug table. It is now accepted as a bug by
the Post Office in paragraph 6 of Appendix 2, but is said to have had transient impact. Mr Coyne
considered that when replacing a counter within a branch, the process could result in “the total loss
of a transaction”. Dr Worden stated that the cause, recorded in the first KEL (which was created in
December 2000 and last updated in July 2007) was that Riposte was coming online from the Recovery
mode too early, and causing messages to be overwritten. The nature of the correction was stated as
being “to find the overwritten transactions for reconciliation we need to look at the Ripostemirror
messagestore' followed by detailed instructions”. Riposte, as has been explained, was part of Legacy
Horizon and was a product provided by another company, Escher, but it is plainly part of Horizon. Dr
Worden also stated that “the incident arose from a hardware replacement (probably from a hardware
fault) not from a fault in Horizon. It is a different kind of recovery issue.” In my judgment the
hardware is part of Horizon. Findings in respect of this are made in the Technical Appendix.
13. Withdrawn stock discrepancies.
The Post Office does not accept these as a bug. Mr Coyne maintains that it is, and in the bug table he
extracted part of a PEAK that stated these “Can cause confusion and unexpected (though hopefully
temporary) discrepancies at branches by allowing them to declare stock which has already been
withdrawn.” Dr Worden stated that “some impact on branch accounts cannot be ruled out, although it
is small”. The Post Office detailed submissions in Appendix 2 of its Closing Submissions concentrated
on the fact that withdrawn stock - which of course is part of the way that the Post Office manages its
business, adding and removing types of products from time to time - this removal of products is done
by means of an update to reference data, “and not a change to the core code in the system.” This
again is concentrating on the code, rather than the way the system operates. Findings on this are
made in the Technical Appendix.
14. Bureau discrepancies.
These relate to foreign exchange, hence the name. It must be differentiated from bug 23 (also foreign
currency, but entitled Bureau de Change). This one arose in 2017. Mr Coyne considered it to be a bug,
and Dr Worden effectively agreed in that his entry stated “This appears to be a system error with
impact on branch accounts. Although it is possible that a subsequent discrepancy between branch
accounting and POLSAP would reveal the problem, leading to a correction (e.g. see Peak PC0265443,
and Mr Coyne's para 3.146), I cannot be certain of this.” It is now accepted by the Post Office in
paragraph 7 of Appendix 2 as one of a number of “bugs with lasting impact
RLITO000005
RLITO000005
(although they were resolved)”. The detailed part of Appendix 2 states that Mr Coyne has drawn
together two distinct issues. Paragraph 4 of the detailed part also states the following:
“Bug 14: Bureau Discrepancies is a bug with the potential for lasting financial impact. There are two
distinct issues which fall under this heading. With regards to the first issue the branch was made good
and a fix was implemented. The second issue was not a bug in Horizon nor an issue which could have
impacted branch accounts; it created what was essentially a cash flow problem for the branch.”
Findings on this are made in the Technical Appendix, but so far as the first issue, given a software fix
was required, it is undoubtedly a bug, and it is correct to record (as the Post Office do) that there was
the potential for lasting financial impact, as without that, there would be no need for the “branch to
be made good” because that means the Post Office corrected the branch account discrepancy caused
by the bug.
15. Phantom Transactions.
There are three different issues grouped under this heading. The Post Office does not accept these as
a bug. Mr Coyne in his cross-examination stated that this was referable to Horizon Issue 4. He also
accepted that some of the issues were not bugs in Horizon. Dr Worden relied upon the fact that what
he called “the master PEAK” was closed as “no fault in product”; however, the evidence of fact on this
is dealt with above at [209] to [213] when the PEAK was put to Mrs Van Den Bogerd. Dr Worden
stated that “There is no evidence for bugs in Horizon with impact on branch accounts.” That entry
was obviously before the cross-examination, which in my judgment provided greater factual
information. I do not consider reliance can be placed upon the Fujitsu conclusion in the PEAK.
Findings on this are dealt with in the Technical Appendix.
16. Reconciliation issues.
The Post Office does not accept these as a bug. There are a number of different issues grouped
together in this heading. The issue was that the SPM was shown a discrepancy on his or her screen.
Mr Coyne accepted that the discrepancy would not be shown in the branch accounts; Dr Worden
stated that “as it concerns an issue in reporting, the software fault (which was fixed after 5 months)
had no direct impact on branch accounts. The only effect of an error in this report would be to mislead
or confuse the Subpostmaster - probably leading him to check his figures more carefully and costing
him some time.” That accepts a software fault; the fault was in fact a miscounting of the number of
files by the system. However, there are six different issues grouped under this heading and the
process of reconciliation, which is effectively the comparison by Horizon of two different sets of data,
is part of its design function. Mr Coyne in his cross-examination also stated that this was referable to
Horizon Issue 4, rather than Horizon Issue 1. The recovery messages were held in the branch
messagestore. Findings on this are dealt with in the Technical Appendix.
17. Branch Customer discrepancies.
This was an entry in respect of Horizon Issue 4 and recorded as such in the bug table. The Post Office
does not accept these as a bug. Mr Coyne accepted in his crossexamination that the entry in the PEAK
did not suggest any impact upon branch accounts. The entry in the bug table originated in a single
PEAK, although there were two PEAKs that relate to the same issue, which was a discrepancy
between the financial records held by the Post Office (from Horizon) and a bank’s records, after a
counter crashed whilst a transaction was being processed. Findings on this are dealt with in the
Technical Appendix.
RLITO000005
RLITO000005
18. Concurrent logins.
It is accepted by the Post Office in paragraph 7 of Appendix 2 as one of a number of “bugs with lasting
impact (although they were resolved)”. In paragraph 2 of the detailed part of Appendix 2 dealing with
this bug, the following is stated by the Post Office: “Post Office accepts that Bug 18: Concurrent
Logins had a potentially lasting financial impact. There is no evidence of any discrepancy in the
PEAKs referred to by the experts.” The problem was in the early days of Legacy Horizon when it was
possible for users to log in to two terminals at once. Dr Worden in the bug table made the following
useful summary statement:
“discrepancies could occur - manifesting themselves as a receipts/payments mismatch. This had the
potential to affect branch accounts. The mismatch would bring it to the attention of the
Subpostmaster, who would require it to be investigated, except possibly in the case of small
mismatches, which he might pass off as an error in the branch (e.g. of counting stock).
“.,...Fujitsu believed it was a problem with the underlying Riposte software, and passed it to Escher. In
September 2000, the problem was 'Now formally fixed in Build 223 update 19 which was released
overnight.’ However, the new release from Escher did not, as it was expected to, fix the problem.
Escher denied that is was a bug in Riposte, but Fujitsu believed in July 2001 that ‘This is clearly a bug
in the Supplier's code’.
Dr Worden considered that “these faults had to the potential to produce discrepancies in branch
accounts, of small amounts, for a short period of time”. How the bug is therefore characterised
depends upon my findings in relation to lasting and transient impact, which is dealt with below. There
is no doubt, in my judgment, that this was a bug, and Fujitsu clearly recognised and recorded that in
2001. The fact that the bug was in the code supplied by Escher, as part of Riposte, and not that of
Fujitsu is wholly irrelevant for the purposes of the Horizon Issues.
19. Post & Go/TA discrepancies in POLSAP.
It is accepted as a bug by the Post Office in paragraph 6 of Appendix 2, but is said to have had
transient impact. The entry relates to Horizon Issue 4 rather than 1. “Post & Go” are self-service
terminals, that are no longer available in branch Post Offices, and are only available in Crown and WH
Smith main branches. They are basically selfservice kiosks, designed to avoid queuing, and a
customer can weigh a letter or parcel, the terminal will print the relevant stamps/labels, and the item
is then posted. The Post Office submitted that this issue was irrelevant to the Horizon Issues trial as
“this issue does not relate to branches that are the focus of this trial”. Findings on this are dealt with
in the Technical Appendix.
20. Recovery Failures.
The entry in the bug table identifies this as relating to Horizon Issue 4. Post Office does not accept
these as a bug and states that there is no evidence of a bug in Horizon. There are three different
PEAKs. Mr Coyne accepted in his cross-examination that this (or these) should be removed from the
table that he used as the originator for his list of bugs (which was Mr Coyne’s Table 1). Dr Worden.
stated in the bug table that “there was some implication of hardware faults, with a replacement of a
base unit, but the PEAK has no evidence of software faults in Horizon.” In one of the PEAKs, he
concluded that there was no evidence of any fault in Horizon. Findings on this are dealt with in the
Technical Appendix.
21. Transaction Correction Issues.
RLITO000005
RLITO000005
This is de facto accepted as a bug by the Post Office, but it is submitted it had no branch impact. Mr
Coyne’s entry was that “Transaction Correction bugs/errors and defects do not cause discrepancies
with branch accounts but” and he then listed various consequences. In his report he referred to
“technical flaws”. Dr Worden, in respect of some PEAKs, accepted they were a bug, such as PEAK
PC0129587 where he stated “In my opinion this bug would result in an inconvenience to the
Subpostmaster (inability to rollover to the next TP) but would not result in inaccurate processing of
any TC, or any impact on branch accounts”. The bug is included in the list of paragraph 5 of Appendix
2 under the heading “the following bugs had no branch impact.”
These are submitted by the Post Office as there being “no evidence of any financial impact upon
branch accounts”. Findings on this are dealt with in the Technical Appendix.
22. Bugs/errors/defects introduced by previously applied PEAK fixes.
This is de facto accepted as a bug by the Post Office, but it is submitted it had no branch impact. It is
included in paragraph 5 of Appendix 2 under the heading “the following bugs had no branch impact”.
It is also submitted by the Post Office that “a number of PEAKs arose in testing. The PEAKs that arose
in the live environment do not indicate evidence of branch impact.” It is listed in the bug table as
concerning Horizon Issue 4. Mr Coyne stated in the Joint Statement that “Branch accounts would be
affected by this bug which would cause a discrepancy when handling cheques where the value of the
cheque would be doubled” although he accepted that the SPM in question was processing a cheque in
a different manner to that recommended. Dr Worden used the term “a fault” in his entries, maintained
that there was no impact on branch accounts if the SPM followed correct procedures, and also in
respect of another branch relied upon the fact that the effect was an error of 2p only. Findings on this
are dealt with in the Technical Appendix.
23. Bureau de change.
These arose in 2005, 2006 and 2010, and should be differentiated from bug 14, which arose in 2017,
was during Horizon Online and is referred to as Bureau Discrepancies. The Post Office does not
accept this as a bug. The Post Office also states that Mr Coyne has identified three issues with the
same heading, and submits that they all relate to user error. Dr Worden’s entry in the bug table in
relation to one of them states “Analysing the second KEL (2010) I noted: ‘Impact small until bug fixed
- rounding errors 10-5 in exchange rates.’ The Post Office submitting that this is not a bug is a
somewhat bold submission, given the KEL to which Dr Worden refers expressly states that the
problem is a different exchange rate appearing on the HNG-X rateboard and the Horizon rateboard. It
states “Problem - there is a bug in the code” (emphasis added). Findings on this are dealt with in the
Technical Appendix.
24. Wrong branch customer change displayed.
Mr Coyne concluded this was a bug and part of his entry in the 2nd Joint Statement was “the KEL
explains that “the cash amount entered is multiplied by the Qty and hence the new stack total is
wrong”, this Horizon bug was due to incorrect reference data and led to an incorrect amount of
change being displayed on the branch screen leading to the operator to provide the branch customer
with the wrong amount of money thereby leaving a discrepancy in Branch Accounts. It is possible that
the amount of change shown on screen is more than the actual money tendered by the customer.” Dr
Worden’s entry was “When analysing this KEL I noted ‘Sounds like a genuine problem which may
have led to giving the customer the wrong amount - i.e. not recoverable.’ It is now accepted by the
Post Office in paragraph 7 of Appendix 2 as one of a number of “bugs with lasting impact (although
RLITO000005
RLITO000005
they were resolved)”. It is also said that it is a reference data bug, and that therefore once discovered
could be quickly fixed by changing the relevant reference data. Findings on this are dealt with in the
Technical Appendix.
25. Lyca top up.
It is accepted as a bug by the Post Office in paragraph 6 of Appendix 2, but is said to have had
transient impact. It is also a reference data bug, and paragraph 2 of the detailed part of the
submissions on this in Appendix 2 states “Lyca Top Up is a bug with the potential for lasting financial
impact. This is also [a] reference data bug. As set out above, the experts have agreed that that while
reference data bugs may be a significant proportion of the bugs with financial impact, once
discovered, they could be quickly fixed (by a change to the reference data) once the bug is correctly
identified.” It is also submitted that it was identified through Fujitsu’s automatic reporting. Findings
on this are dealt with in the Technical Appendix.
26. TPSC250 Report.
It is accepted by the Post Office in paragraph 7 of Appendix 2 as one of a number of “bugs with lasting
impact (although they were resolved)”. The origin of the experts discovering this bug is a KEL raised
by Anne Chambers in February 2005, and last updated in April 2008. Findings on this are dealt with in
the Technical Appendix. I do however find it notable that although Dr Worden says the amounts
involved are what he calls “small”, he accepts it is as a “back end reporting problem” - which is
nothing to do with the SPM, by definition - and also the KEL states that “the accounting tree has not
handled this properly when calculating the daily recon figures and it has resulted in a mismatch...”
The mismatch in that case was 66p. The accounting tree is part of Horizon. The KEL also appears to
be incomplete because only the first page is present. The section that follows the heading “Evidence”
is entirely missing. There is also nothing in terms of the text available that demonstrates what
occurred in 2008, even though it is plain for the page that is available that the KEL was updated then.
The Post Office submits that there are five separate issues drawn together within this bug. Findings
on this are dealt with in the Technical Appendix.
27. TPS.
It is accepted by the Post Office in paragraph 7 of Appendix 2 as one of a number of “bugs with lasting
impact (although they were resolved)”. One of the Fujitsu documents referenced in the 2nd Joint
Statement states that the Transaction Repair Tool or TRT is being used “to repair 1 harvester
exceptions for” a particular branch and “There is no correction to be performed and hence no call for
confirmtiprepair - this is just an oddity performed by that very flaky mails code.” (emphasis added)
There were 40 associated PEAKs, and Mr Coyne observed that both the credit and debit sides of a
transaction were doubled, so the net impact of the bug was zero, although he drew attention to the
entry in two PEAKs that suggested that SSC requested confirmation of any gain or loss at the counter.
Dr Worden believed it was a back-end reporting problem, although the chances of impact upon branch
accounts were small. Findings on this are dealt with in the Technical Appendix.
28. Drop and Go.
It is accepted by the Post Office in paragraph 7 of Appendix 2 as one of a number of “bugs with lasting
impact (although they were resolved)”. This occurred in July 2017 and related to a duplicate “Drop
and Go” transaction for £100. This was performed twice; the branch was debited with £200, but the
customer credited with only £100 (which had been the amount “topped up” by the customer). Mr
RLITO000005
RLITO000005
Coyne considered it a bug with impact upon branch accounts; Dr Worden stated in his entry “My
analysis of this KEL was ‘Possible financial impact. Seems very visible on the counter. Script =
reference data - therefore fixed easily’. Paragraph 2 of the detailed part of Appendix 2 on this bug
states “Peak PC0260269 relates to an issue involving a Drop and Go transaction (a £100 mobile phone
top up) that timed out on Horizon” (emphasis added). This does not appear to be correct, and I do not
believe that the transaction relates to a mobile phone. However, I deal with this further in the findings
in the Technical Appendix.
29. Network Banking Bug.
This related to a KEL which was raised in 2004 and updated in 2005, with 12 associated PEAKs in the
range 2004 to 2010. The Post Office does not accept this as a bug. Mr Coyne stated that “Horizon
appears to mis-handle communications, leading to errors within network banking and in turn causing
the potential for branch account discrepancies.” One of these was pension transactions being declined
yet the customer’s bank account being debited; when the customer complained, the SPM refunded the
£50 in question. Dr Worden considered it was “mainly about a communication problem from BT,
outside Horizon” but also wanted to investigate further due to an entry that referred to a “CNIM own
goal”. The Post Office maintained there were two separate issues and the £50 was paid out due to
“user error”. Findings on this are dealt with in the Technical Appendix.
680.The names given to the different bugs above came, sometimes, from the names applied to the
bugs by the Fujitsu personnel who knew about these bugs and worked upon them over the years.
These names would sometimes appear in the PEAKs themselves that identified the bugs or which
represented the views reached by different personnel within Fujitsu who investigated the background,
sometimes identifying that previous incidents had similar, or sometimes even identical, characteristics
to earlier occurrences of the bug, and who worked to provide fixes or solutions to some of them, or
came to conclusions that there was nothing wrong with Horizon and attributed blame, fault or
responsibility to the SPM. Some of the names are self-explanatory. Some, such as Callendar Square or
Dalmellington, are the names of places where that bug was first identified by Fujitsu.
681.Certain terms were used, dependent upon whether a particular bug was agreed as being a bug
(such as the Callendar Square bug), or something else, such as “issues” or “discrepancies.” However,
as at the end of the trial, it can be seen from the above entries that the experts’ agreements, the
clarification provided by each during crossexamination, and the other developments in the trial such
as (one assumes) some reflection by the Post Office on the actual text of PEAKs and KELs, lead to the
following situation. Of the 29 different entries in the bug table, eight were alleged by the Post Office
not to be bugs at all. They are those numbered 8, 13, 15, 16, 17, 20, 23 and 29. This is set out in
paragraph 4 on the first page of Appendix 2 to the Post Office’s Closing Submissions. This careful
terminology therefore became a little clearer. The remaining number can properly therefore be
described as bugs, even on the Post Office’s own case, although their effect was challenged.
682.This means that the remainder of what is, in my judgment, a lengthy list, were accepted to be
bugs. In my judgment, that number of accepted bugs, even on the Post Office’s own case about their
effect, and the non-acceptance of the eight others, is a sizeable number. It is approximately a ten-fold
increase in the number admitted by the Post Office prior to the Horizon Issues. It is correct that the
Post Office did not accept lasting impact on a number - a point I deal with below under “Lasting and
transient impact” - and maintained that only 9 “had the potential to cause lasting impact, but were
resolved by the Post Office and Fujitsu” but, in my judgment, that is rather to miss the point. The
wording of Horizon Issue 1, agreed by the parties, and (one assumes) a major issue that both parties
RLITO000005
RLITO000005
wished to have resolved in order sensibly to progress this group litigation (and omitting the
references to the pleadings) is:
“To what extent was it possible or likely for bugs, errors or defects of the nature alleged... to have the
potential to (a) cause apparent or alleged discrepancies or shortfalls relating to Subpostmasters’
branch accounts or transactions, or (b) undermine the reliability of Horizon accurately to process and
to record transactions.”
683.The use of the words “possible” and “potential” in this issue makes it clear, in my judgment, that
the number of “bugs, errors or defects” that fall to be considered is, even on the Post Office’s own
submissions, 21 different bugs. I find that to be a significant total.
684.Part only of Appendix 2 to the Post Office’s Closing Submissions dealing with bug 16,
Reconciliation Issues, was, by administrative error, not submitted by the date for closing submissions,
or until some time after oral submissions were delivered. The Post Office’s solicitors realised this in
September 2019. There were three missing pages. I am satisfied that these were drafted prior to the
date for service, and I allowed these to be submitted late, together with a clarification document, and
short responsive written submissions by the claimants (as the claimants had been deprived of the
opportunity to address these three missing pages orally in July). I consider that I am permitted to do
this under the court’s general case management powers, and that to do this was to adopt a reasonable
and proportionate course. The Post Office’s Closing Submissions was a very substantial document,
and it was in my judgment sensible to admit the 3 pages only that had been omitted from one of the
appendices.
685,Other elements of the 2nd Joint Statement were also agreed. These included that Horizon had
produced over 3 million sets of monthly branch accounts (a point relevant to Dr Worden’s Section 8
statistical exercise), and also that the experts had different views on “branch impact”. The relevant
passages in the Joint Statement were that:
“Mr Coyne refers to any discrepancy that caused a loss (or gain) within branch accounts that needed
corrective action as an “impact to branch accounts”. Dr Worden only considers an effect or impact on
branch accounts where a discrepancy loss (or gain) was not rectified by a correction such as a
Transaction Correction.”
686.This in particular demonstrates an important difference in the approach of the experts. Given, as I
have explained above, that the Horizon Issues do not refer to Transaction Corrections, which even the
Post Office’s own witnesses explained had not been addressed because the Post Office’s solicitors said
these were outside the scope of the Horizon Issues trial, I consider Mr Coyne’s approach to “impact
on branch accounts” to be the correct one. Dr Worden’s approach minimises or avoids impact on
branch accounts that have been caused by bugs, errors or defects, even where those that are (or have
been) present in the Horizon system have in fact affected the branch accounts of SPMs. It is a feature
that I take into account when I consider which of the two experts’ approaches and evidence I prefer. I
also explain this further in “Lasting and Transient Impact” below.
687.The 3rd Joint Statement continued the process of setting out the experts’ agreed position by
working through the remaining Horizon Issues. This statement dealt with Horizon Issues 3 to 8 and
was dated 1 March 2019. The experts agreed the following points in respect of each. The numbering
is mine. One entry below relates to one passage of Dr Worden’s separate and not agreed view under
Horizon Issue 3. It was not the only such entry - in different places the experts would record their
own separate views.
RLITO000005
RLITO000005
The 3 rd Joint Statement
Horizon Issue 3
“1. Irrespective of how you define the detail of robustness, in line with most other large-scale
computer systems, Horizon's robustness has generally improved. From our experience of other
computer systems, Horizon is relatively robust. We agree that 'robust' does not mean infallible and
therefore Horizon has and will continue to suffer faults. Robustness limits the impact of those faults
and other adverse events. This increase in robustness has, in part, developed from Post Office
discovering bugs/errors and defects in live use and then applying fixes and improving monitoring.
2.Computer systems are considered more robust if access to the back-end databases is restricted
tightly.
3.It was possible for some of the Horizon support staff working at Fujitsu to modify the Horizon back-
end branch database. In 2012, Post Office's auditors observed that there were inappropriate system
privileges assigned to the APPSUP role (which allowed amendments to the BRDB).
4.Post Office does not consult the full audit data (unfiltered ARQ Data) before deciding how to handle
discrepancies and issuing Transaction Corrections.
5.PEAKs show that some defects have lain undetected in Horizon for extended periods without being
diagnosed and fixed.
6.During the life of Horizon there have been 19,842 changes made to it via the Fujitsu/Post Office
release mechanism.
7.It is common modern IT development practice to make frequent incremental builds and releases of
software.
8.Specific release note detail has not been provided. Of the 19,842 changes, we would expect that
many were minor changes. It is likely that others contained changes to improve the system or to fix
bugs and defects.
9.The effectiveness of various countermeasures changed throughout the life of Horizon.
10.Countermeasures are basic elements of practical IT system design
11.Countermeasures work by limiting the impact of Horizon bugs/error and defects on branch
accounts. Countermeasures do not always eliminate the effects of adverse events (they are not
perfect) but they are often effective in the area where they are deployed; that is why they have
become basic elements of practical IT systems design.
12.It is difficult to measure the extent of the robustness of Horizon, apart from how it might limit the
extent of impact on branch accounts, as in Issue 1.
13.There are indications that in its first year of operation, and in the first year after the introduction of
Horizon Online, the system suffered from more problems than in other years. One might expect a
higher level of problems in these early periods. The extent to which these problems were serious, or
evaded countermeasures, or caused discrepancies in branch accounts, is not agreed.”
14.Dr Worden stated the following, which was not agreed, following the entry immediately above.
“The main way in which the experts have assessed the extent of robustness of Horizon is to ask to
RLITO000005
RLITO000005
what extent failures of its robustness impacted branch accounts. I have addressed this in Horizon
Issue 1. In answering this question, fluctuations over the years are of less importance than the sum
over all years, if the sum over all years is small. I have found the sum of impacts on branch accounts
over all years to be very small”. (emphasis added). This approach is considered further by me when I
address Dr Worden’s Section 8 statistical analysis.
15.The agreed passages continued. “The users of any IT system play a role in assuring its robustness.
The designers of a system should not make unrealistic assumptions about the users of the system.
Unrealistic assumptions would lead to inappropriate design, making the system less usable.
16.As Horizon has changed throughout its lifetime, the existence and effectiveness of any
countermeasures has too. To have considered the time dependence of all robustness countermeasures
over 20 years, would have made the expert reports impossibly lengthy. There was not the time to do
so.
17. Many software bugs can have the same effects as a user error (as illustrated, for instance, by the
D li . “ ”
(emphasis added)
Horizon Issue 4
“1. Bugs, errors and defects identified in relation to Horizon Issue 1 are often relevant to Issue 4 in
that they are ultimately errors arising from the processing of data in Horizon.
2.There is evidence within the PEAKs and KELs of bugs/errors/defects within Horizon arising from
parts (a), (b) and (c) of this issue that occurred without causing financial discrepancies as well as
some that occurred causing financial discrepancies.
woos
(emphasis added)
4.0f the bugs which in the experts’ opinion had the potential to produce discrepancies in branch
accounts there may be some involvement of reference data in Bureau Discrepancies, Bureau de
Change, Wrong branch customer change displayed, Lyca top-up, and Drop and Go. (Rows 14, 23, 24,
25, and 28 of the bugs table in the second expert joint statement). It is notable that these bugs all
concerned specific products (arising from the reference data defining those products).
So, while reference data bugs may be a significant proportion of the bugs with financial impact, once
discovered, they could be quickly fixed (by a change to the reference data) once the bug is correctly
identified.
5.Reconciliation between transactions recorded on Horizon and transactions recorded by Post Office's
clients is largely automated. Detected discrepancies were subject to manual corrective fixes and/or
the issue of Transaction Corrections/Error Notices to the Subpostmasters.
6.The adequacy of Post Office back office processes to prevent discrepancies in branch accounts can
be measured by the quality of the TC process. This quality includes:
+The processes of consideration of available data
RLITO000005
RLITO000005
+The level of errors observed in the process
+The level of complaints or disputes raised following a TC
+The level of upheld complaints following a TC
+The level of financial impact of erroneous TCs
7.Exxors in third-party data have led to discrepancies in branch accounts. through erroneous TCs
being issued on Subpostmasters.
8.PO Lof oe thirds . oa .
to errors in TCs), or the delays in their processes (which may lead to delays in TCs).
(emphasis added)
9.PO can and should ensure, by careful investigation of disputed TCs, that only a small proportion of
errors by PO clients lead to losses for Subpostmasters, provided that the Subpostmasters are in good
control of their branches and have the required information available.”
Horizon Issue 6
“1. It is agreed that there are many measures and controls within Horizon that existed to prevent,
detect, identify report or reduce the risk of varying errors.”
Horizon Issue 7
“1, Fujitsu could access all transaction data recorded by Horizon.
2. Both Post Office and Fujitsu can read data remotely, and FJ needs remote access for support
purposes.”
Horizon Issue 8
“1. Post Office had access to data which would not have been available to Subpostmasters.
2.The descriptions of facilities for PO in the two expert reports are consistent and can be taken
together as a description of those facilities.
3.Post Offi - Fujitsu for di isof £ f
caused by bugs/errors or defects within the Horizon system.” (emphasis added)
688.It can therefore be seen from the passages of agreement reached by the two IT experts in the 3rd
Joint Statement, that a great amount of relevant material directly relevant to the Horizon Issues was
not in issue. It was agreed that errors in reference data have led to discrepancies in branch accounts,
and also that errors in third-party data have led to discrepancies in branch accounts too. Further, the
Post Office has no control over the level of errors made by the third-party clients of the Post Office,
and this is agreed. The point also ought to be made that SPMs have no control over that either. Both
errors in reference data, and also errors in third party data, being agreed to have led to discrepancies
in branch accounts, are points consistent with the claimants’ case in respect of the Horizon system
generally.
The 4 th Joint Statement
RLITO000005
RLITO000005
689.This dealt with Horizon Issues 10 to 12, and was the last joint statement agreed by the experts. It
was dated 4 March 2019 and was agreed just before the trial commenced. Again, the numbering is
mine, and I also reproduce one passage from each expert that
was not agreed. One entry below in relation to Horizon Issue 11 was agreed with the word
“whenever”. Both experts agreed that this should be worded “usually when” and this latter agreement
on that wording was reached during the expert evidence phase of the trial.
Horizon Issue 10
1. Dr Worden stated the following, which was not agreed, in relation to privileged user access, and as
one of a number of passages at the beginning of the Joint Statement that were not agreed concerning
remote and privileged access. “The Privileged User Access logs are not a useful source of evidence
about remote access, including Balancing Transactions.”
The following were agreed:
“2. Within Horizon Online, a Balancing Transaction can be created using the BRDB Transaction
Correction Tool (called BRDBX015).
3.In the one acknowledged Balancing Transaction which has been disclosed, there were SQL
INSERTS into only two tables of the BRDB. These tables were ops$brdb.brdbb_rx_eposs_transactions
and
ops$brdb.brdb_rx_rep_session_data”
4.Mr Coyne stated the following, as one of a number of passages that were not agreed. “Outside of the
Transaction Correction Tool other methods were available in Horizon Online.
The APPSUP role used by Fujitsu has elevated privilege and could be used “for unenvisaged ad-hoc
live amendment not covered by BRDBX015” as outlined in the referenced document.
Fujitsu also has the ability to perform balancing transactions via direct SQL operations (using a
command line interface) to perform corrective transactions in Horizon Online (including deletions of
operational data).
Such modifications may or may not be visible to the Subpostmaster and they could be done without
their consent.”
The following were agreed:
5.""Fixes’ implemented by Fujitsu had the potential to affect transaction data for branch accounts, for
transactions which occurred after the fix was deployed.
6.We have not been provided with logs or audits from the Transaction Repair Tool (TRT).
7.Certain facilities and procedures used by Fujitsu to repair the more common issues which arose in
Horizon were standardised, and evidence of them persists. However, to repair less common issues
which arose from time to time, standard tools and procedures might not have been sufficient, and
evidence might not persist of what was done at the time. Even when evidence does persist, it may be
extremely difficult for the experts to interpret it today, because of the scale and complexity of Horizon.
Therefore, it is usually difficult for the experts to make categorical negative statements of the form: ‘'X
or Y never happened’”.
RLITO000005
RLITO000005
Horizon Issue 11
All of the following are agreed entries:
“1, Evidence from several PEAKs indicates that usually when [corrected from whenever] Fujitsu
needed to make any change to data which impacted branch accounts, they were concerned to seek
permission from PO to do so, and to ensure that PO took responsibility for the resulting change.
2.For any large commercial IT system, it is necessary for some technical users to have privileged
access to databases, with wide-ranging capabilities, for system maintenance and problem-solving
purposes rather than application-related purposes. It is important to keep the number of these users
audited.” (emphasis added)
“3. The logging of Privileged User Access (in PAU logs) commenced October 2009.
Between 2009 and 2015 these logs only displayed the fact that a Privileged User had logged on or off
but not what actions they had taken whilst the Privileged User was logged in.
The use of the Transaction Correction Tool cannot be seen in these logs.
4.At all times, any privileged user access log only shows what tables of BRDB were accessed for a very
small minority of accesses.
There are no privileged user access logs which show access to the two tables of BRDB used to insert
the one acknowledged balancing transaction.
5.The authorisation for the use of the Transaction Correction Tool for the acknowledged Balancing
Transaction can be seen in one OCP within the OCP disclosure of 25th January 2019.
The record of the uses of the Transaction Correction Tool, including the one acknowledged Balancing
Transaction, can be seen in the .aud files, disclosed 22nd February 2019.”
Horizon Issue 12
The following are all agreed entries:
“1, Post Office report that there have been 2,297 uses of the Transaction Correction Tool. For each of
these, the usage was logged and the SQL used to alter BRDB has been disclosed.
Only one of these uses (POL-0512131) changed transaction data in two transaction tables, in March
2010.
This Balancing Transaction consists of about 500 lines of SQL, in a pre-defined template.
None of the other disclosed uses of the Transaction Correction tool contain the relevant SQL INSERT
commands such as: ‘INSERT INTO ops$brdb.brdb rx_rep_session_data ...’
2. Usage of the Transaction Correction Tool is recorded in the BRDB_TXN_CORR_ TOOL JOURNAL
table within the BRDB.
The tools Low Level Design sets out that it has the capability to insert entries into the following
tables:
BRDB_RX_BUREAU_TRANSACTIONS
RLITO000005
RLITO000005
BRDB_RX_EPOSS TRANSACTIONS
BRDB_RX_APS TRANSACTIONS
BRDB_RX_EPOSS_EVENTS
BRDB_RX_NWB_ TRANSACTIONS
BRDB_RX_REP_SESSION_DATA
BRDB_RX_DCS_TRANSACTIONS
BRDB_RX_REP_ EVENT DATA
BRDB_RX_CUT_OFF SUMMARIES"
690.Of the agreed entries in relation to Horizon Issue 11, I find the following of very great relevance.
The experts are agreed that for any large commercial IT system, which Horizon obviously is, it is
necessary for some technical users to have privileged access to databases with wide-ranging
capabilities. This however, is not for application related purposes, but is for system maintenance and
problem-solving purposes. It is also agreed by both IT experts that it is important that the number of
users with these abilities is kept down to the minimum possible, AND for each action that they take
whilst logged in to be recorded and audited (my use of block capitals).
691.1 consider that to be entirely conventional, obviously agreed by both the experts, and also to be
technically justified, particularly in a system such as Horizon which deals with accounting. It is also
not requiring an unrealistic level of perfection; it is simply a matter of having properly controlled
access to a system such as this one.
692.However, when one then considers the subsequent passages of the 4th experts statement, it can
be seen how far from this joint agreed (and technically justified) position the Horizon system was. The
logging of Privileged User Access (in PUA logs) commenced in October 2009. For the period 2009 to
2015 - obviously a 6 year period - these logs only displayed the fact that a Privileged User had logged
on or off, “but not what actions they had taken whilst the Privileged User was logged in”. Therefore
the actions they were taking when logged in were being neither recorded nor audited. All that could
be seen is they were logged in. Further, it has already been seen that the number of users with the
relevant privileges was not, in my judgment, restricted to a minimum. Further, the use of the
Transaction Correction Tool cannot be seen in these logs. Yet further, the experts are agreed that at
all times, any privileged user access log only shows what tables of BRDB were accessed for a very
small minority of accesses.
693. Yet further, it is agreed by the experts that there are no privileged user access logs which show
access to the two tables of BRDB used to insert the one acknowledged balancing transaction. Indeed,
one of Dr Worden’s entries in the experts’ joint statements was
“The Privileged User Access logs are not a useful source of evidence about remote access, including
Balancing Transactions.”
694.What this amounts to, in my judgment, is a serious deficiency both in the required level of
controls in Horizon, in the recording of what privileged users were actually doing (other than that
they were simply logged on) and also a corresponding absence of recording and auditing of those
activities.
RLITO000005
RLITO000005
695.In my judgment, and based upon the experts’ agreement in this respect, this serious deficiency
lasted up to 2015. It also means, because of the extent (or rather the power) of the APPSUP privilege,
the Transaction Correction Tool is but a small subset of the way that records in the BRDB can be
affected or changed by Fujitsu. It also means that the tool’s Low Level Design document that was
available in the trial bundle, and considered by the experts, has been overtaken by use of the
Transaction Correction Tool, because the SQL INSERT command is not present in 2,296
acknowledged uses of the tool. Both the number of uses, and the non-use of the SQL INSERT
command, are in agreed passages by the experts in the 4th Joint Statement. This is also a point that
was put to Mr Godeseth in his cross-examination (and accepted) and is dealt with at [339] to [344]
above.
Lasting and transient impact
696.Of the total number of bugs accepted by the Post Office, the Post Office submits that nine of them
had, or potentially had, only transient impact. These are the following:
Bug 2: Callendar Square;
Bug 4: Dalmellington;
Bug 5: Remming In;
Bug 6: Remming Out;
Bug 7: Local Suspense (not to be confused with Bug 3. Suspense Account bug)
Bug 9: Reversals;
Bug 12: Counter Replacement;
Bug 19: Post & Go/TA Discrepancies in POLSAP; and
Bug 25: Lyca top up.
697.Transient impact is a concept introduced by Dr Worden. It does not form any part of the Horizon
Issues as formulated, agreed and ordered. Dr Worden agreed, in his cross examination, that he had
added in the word “lasting” to Horizon Issue 1. He also said: “...for better or worse I took my role to
be concerned with lasting effects on branch accounts rather than transient ones.”
698.This is a plain and obvious departure from the wording of Horizon Issue 1, and indeed in my
judgment represents a narrowing of it. I consider this departure unjustified. It is not the issue that
was drafted and agreed by the parties, approved by the court and ordered to be tried.
699.Dr Worden also said in the 2nd Joint Statement (in one of the passages that was not agreed) that
“transient inaccuracies in branch accounts, which needed some form of correction, have arisen so
frequently and from so many causes that to list them is not useful; and that evidence of each
correction being carried out is unlikely to persist to this day.”
700.He stated that “not rectified” means lasting, whereas “rectified” means transient. He also stated
that if a bug caused something that required correcting with a Transaction Correction or TC, then that
would mean (if the TC were issued) that it were a transient effect; but if the TC was never issued, that
would be a lasting effect. The passages were:
RLITO000005
RLITO000005
“Q. So if the TC process takes a long time, then you might say something was lasting, even
notwithstanding years later it might be corrected?
A. That is not how I understood lasting ~
Q. How did you understand lasting?
A. There were delays in the TC process which might be due to client organisations or might be due to
all sorts of things, and they could be at the outside, I believe, several months. My definition of lasting
did not depend on TCs coming in within a certain timeframe. If TC never came in, that would be
lasting, but if TC took several months to come in that is not what I would call a lasting effect. A lasting
701.The use of the phrase “lost money forever” demonstrates the artificiality, in my judgment, of this
distinction. Whether this was the intended purpose or not, applying such a concept is to minimise as
much as possible the impact on branch accounts of bugs, errors and defects that otherwise exist. He
also stated that “the branch accounts would look wrong” for the period prior a TC being issued. That
is undoubtedly correct; the accounts would not only “look wrong”, they would be wrong. This makes it
even more clear that Dr Worden’s approach is at odds with the wording of the Horizon Issues, as well
as with the way which the Horizon System was operated.
702.His approach is, in my judgment, wholly artificial, for at least five reasons.
1.It ignores the express wording of Horizon Issue 1 itself.
2.It ignores the very function of Horizon, which was used as the branch accounting mechanism
between SPMs and the Post Office. A SPM would “roll over” their branch each trading period, usually
a 4 week period (sometimes a 5 week period). At the end of each trading period, the SPM would
produce a Branch Trading Statement (or “BTS”) using Horizon, which was the accounting statement
of the branch for that period. Any shortfall or discrepancy in the accounts as demonstrated by the BTS
would have to be made good by the SPM (which means the amount of the loss would have to be paid)
or “settled centrally” (which means the SPM would seek time to pay). Therefore, any occurrence of
any bug, error or defect within that trading period would have an impact on the branch accounts (by
reason of the regular production of the BTS) at the end of that particular 4 week trading period.
3.The issuing of TCs is done by the Post Office outside of Horizon in any event.
4.[It pays no attention to whether a TC is issued within a branch trading period (and hence before the
creation of the Branch Trading Statement for that period) or afterwards, in the next or any
subsequent trading period. The fact that (for example) a TC might be issued some months later does
not mean that the branch accounts have not been impacted. The branch accounts specifically have
been impacted by the occurrence of the bug, error or defect.
5.The correct characterisation of a bug, error or defect within Horizon, and its impact, cannot sensibly
depend upon an act (the issuing of a TC) that is done outside Horizon. If a SPM were to find
themselves reporting, for the first time, an incident to SSC in relation to a hypothetical bug, bug H,
and upon investigation SSC eventually were to accept that there was a bug and a software fix was
required, then the nature of bug H should not depend upon how the PO chooses to correct it. If, say, a
SPM had retired, closed their branch, been suspended or in any other way ceased to be a SPM, thena
TC could not sensibly be used to them. One assumes that in this hypothetical scenario, the PO would
RLITO000005
RLITO000005
refund the SPM (or their estate, if they had died) but that step should not affect the characterisation
of bug H.
703.The point at (2) above is consistent with the way that branch accounting works. This was the
subject of a considerable amount of evidence at the Common Issues trial which is dealt with in
Judgment (No.3). I do not know if Dr Worden had read this judgment, and he was not asked about it.
However, I do not consider that any aspect of the outline accounting process operated by the SPM
using Horizon at the end of each Branch Trading Period is controversial.
704.In my judgment, if any bug, error or defect within Horizon could have an impact upon branch
accounts in any particular trading period, then unless a TC were issued within the same trading
period, it would have had an actual impact upon branch accounts. Similarly, if any bug, error or defect
within Horizon could have an impact upon branch accounts in any particular trading period, then -
regardless of the issuing of any TC in respect of that occurrence - it would have the potential to
impact upon branch accounts.
705.This can be tested by considering the Callendar Square bug, number 2 in the bug table list. The
evidence I have identified at [401] to [414] above makes it clear that this bug was discovered in 2005,
and Mr Godeseth accepted that it had probably been present in Horizon since its inception in 2000. It
affected at least 30 branches, and was recorded in one of the contemporaneous documents as “this
problem has been around for years and affects a number of sites most weeks.” Yet on Dr Worden’s
somewhat artificial constraining of impact on branch accounts, it would be categorised as only having
a transient impact. The dictionary meaning of transient is fleeting, passing, brief, temporary,
momentary, transitory, short-lived or ephemeral, and plainly does not apply to the effect of this
admitted bug. In my judgment the use of this word by Dr Worden and the Post Office is unjustified.
The use of it by Dr Worden was unnecessarily to restrict his analysis.
H. The Period 21 March 2019 to 4 June 2019
706.During this period two matters of note occurred in the group litigation (in addition to hearing the
recusal application and handing down of Judgment (No.4), which it is not necessary to address in this
judgment).
707.The first was the further disclosure of documentation by the Post Office, which is explained
further at [616] above.
708.The second was service of a further supplementary expert's report by Dr Worden directly upon
the court. This occurred on 22 May 2019. It came under cover of an email from Dr Worden, copied
only to the Post Office’s solicitors. It is extremely unusual, if not verging upon unheard of, for an
expert witness to communicate directly with the trial judge, and directly serve material upon him or
her. It is also the case that no party (whether by itself or its solicitors) in ongoing litigation should
communicate with the court without copying in the other parties to the case. The only very narrow
exception to this is ex parte proceedings, in respect of which there are special rules, and which is not
the case here.
709.In the interests of transparency, I reproduce the whole email from Dr Worden here. It was
accompanied by some attachments: “Third Report Open”; and three appendices. The email stated:
“Dear Judge,
RLITO000005
RLITO000005
As was explained by Mr de Garr Robinson in court on 11 April, I have made some further analyses of
Horizon Issues 1, 12, and 13. This work was done at my own instigation, and was not prompted by the
Post Office or its lawyers. In my opinion this work has led to a material change to my opinions, and I
am therefore obliged to inform the court of those changes in a report. With this email I am sending
you the report and its Appendices, in completion of the action. I am doing this on the basis that the
trial will re-convene on 4th June, and so the court may need to be aware of it then. The report is 13
pages long.
As the court is aware from my first expert report, I have interpreted ‘extent’ in Horizon issue 1 as
requiring a number - the maximum proportion of the claimed losses which is attributable to bugs in
Horizon. My first estimate of that number. in my first report and revised in the second expert joint
is 0.4%. TI . . fe i
may better rely on it.
That is why I started this analysis, and why I am now sending it to the court.
Similarly, for Horizon Issues 12 and 13, the analysis in the attached report is in my opinion simple,
easy to check, and leads to results which I was not able to derive before. Therefore I believe it can be
of assistance to the court.
A draft version of this report was first provided to Mr Coyne on 25th April, and a final version was
sent to him on 16 May. I have not yet been able to have a substantive discussion of this report with
him, other than my explaining points in the report to him.
I understand that when cross examined on the Horizon issues, this report is now a part of my opinion
and I may need to refer to it. It is also my understanding that Post Office does not intend to make any
application to the court in relation to this report.
Yours Sincerely,
Robert Worden”
(emphasis added)
710.The reference to what had occurred on 11 April 2019 is as follows. Following dismissal of the
recusal application on 9 April 2019, the trial resumed on 11 April 2019 with the resumption of the
Post Office’s final evidence of fact. This involved the Post Office calling Mr Parker to be cross-
examined, and explaining that Mr Membury (another of the Fujitsu witnesses) was unable to be
called, and the relevant Civil Evidence Act notice being given under CPR Part 33.2. The position of the
claimants on that latter point was explained as their consenting to the necessary extension of time
(rather than a waiver by them under section 2(3) of the Act, a difference that is immaterial in the
circumstances). Mr Membury’s witness statement was therefore not cross-examined upon, but Mr
Parker was called and was cross-examined.
711.Upon conclusion of Mr Parker’s evidence, Mr de Garr Robinson, leading counsel for the Post
Office, explained that, as he put it, “a matter of some awkwardness” had arisen. He submitted the
following:
RLITO000005
RLITO000005
“Dr Worden has recently realised that there is a new way of looking at the evidence in this case which,
in his view, could greatly assist your Lordship, assist the court, in deciding Horizon issues 1, 12 and
13.
This approach involves focusing on those PEAKs, OCRs, OCPs and MSCs which actually mention the
FAD codes of one or more of the claimant branches. Just to explain, when Fujitsu did any authorised
remote handling of data, to put it neutrally, which might affect branch accounts, they raised an OCP,
OCR or MSC whose text was likely to include the six digit FAD code of the relevant branch. So it's
therefore possible to search all the OCPs, OCRs and MSCs with a view to finding all of those which
mention the claimant branches during the relevant claimant's period of tenure. This search yields a
limited number of OCPs, OCRs and MSCs, and it's therefore possible to assess expert issues 12 and
13, which is how often was remote access facility exercised and what effect did it have. It is possible
to assess those questions as they affect the claimants by examining that much smaller document set.
My Lord, that is the first exercise that he would like to undertake, and indeed he has embarked work
on -- I think this week he has embarked work on that.
Second, if a detected bug affected the accounts of any branch the Peak relating to that bug was likely
to mention that branch's FAD code. Typically, it will also -- it may also mention a sum of money. It's
therefore possible to search all the PEAKs in the same way that I have just outlined, looking for PEAKs
which mention any claimant's FAD code during the relevant claimant's period of tenure, and again,
this document could shed some light on Horizon Issue 1 to which extent is it likely that bugs have
affected the relevant branches.
Now, Dr Worden has specifically asked me to offer his apologies to the parties and to the court that he
didn't think of this before. In fact, frankly, he is kicking himself that he didn't do so. He believes that
he and Mr Coyne would only need a short time to consider the relevant documents and to consider
how it affects their views on those issues. He wishes to discuss the documents with Mr Coyne with a
view to agreeing what they do or do not show.
My Lord, in the days since the recusal application was issued he started to consider how the new
approach affects his views. He believes on Issue 1 it allows the parties to make a much simpler
analysis of the point, and he takes a similar view in relation to the remote access issues. It makes, in
his view, it possible for the experts to form a view as to how often remote access was exercised and
what its likely effect was. It is Dr Worden's view that it is his duty under CPR Part 35 to inform the
court of this change of view and to allow the court to consider whether or not it wishes to see it
considered. That belief is based, as your Lordship will be aware, on CPR35.3 which imposes a duty on
experts to help the court on the matters within their expertise, whether or not they are instructed so
to do. My Lord, it's also based on CPR35 -- I should say the practice direction CPR35, paragraph 2.5,
which requires experts to inform the court of any change of views.
I should emphasise this -- none of this comes at the request or instigation of my client. This has come
from Dr Worden. This is his idea. My Lord, he wishes to discuss it with Mr Coyne in a further meeting
between the experts, but of course it's -~- it's only right that your Lordship should be aware of that. I'm
not making any application for permission to put in supplemental expert reports -
Mr Justice Fraser: I don't think you have any supplemental experts' reports to apply for permission
for, do you?
Mr de Garr Robinson: I'm not making any kind of application, I'm simply sharing with your Lordship
the view that has been expressed to me by Dr Worden.”
RLITO000005
RLITO000005
712.The Post Office, in continuing this explanation of what was happening, referred to another case
which had concerned late expert evidence, the judgment in which had been referred to in the opening
submissions for the Horizon Issues trial. That case is Imperial Chemical Industries Ltd v Merit
Merrell Technology Ltd (No.2 Quantum)
[2018] EWHC 1577 (TCC).
713.The exchange with the court continued:
“Mr Justice Fraser: ...In order to apply for permission to adduce extra expert evidence you would have
to have a draft of the report for which you would be seeking permission, wouldn't you?
Mr de Garr Robinson: My Lord not necessarily.
Mr Justice Fraser: You don't think so?
Mr de Garr Robinson: My Lord, I would submit not. It would depend on the circumstances. Often one
would have such a report. I'm conscious that in the ICI case your Lordship cited as a reason for not
giving the relevant party permission to put in a report which they had prepared, that the experts
hadn't gone through that collaborative process and I'm quite anxious to ensure that my expert doesn't
fall into the same trap, if I can put it that way.
Mr Justice Fraser: Well, depending on whichever point one reaches in terms of your actually make an
application to put in a supplementary expert's report, that application will be dealt with as and when
it's made, so I'm not dealing with that either positively or negatively at the moment.
Mr de Garr Robinson: Thank you.”
714.It is not necessary to dwell upon the point, but any party seeking to obtain permission to rely
upon a very late additional expert’s report should ordinarily expect to have at
least a draft of such a report available. It is difficult to see how permission could be given in general
terms for late expert evidence, the content of which was not available. However, no such application
was made by the Post Office in any event. The trial had commenced on 11 March 2019. Dr Worden’s
first expert report was dated 7 December 2018 and his supplementary report was dated 1 February
2019. This notification to the court on 11 April 2019 of Dr Worden having “recently realised that there
is a new way of looking at the evidence in this case” came over four months after the date of his first
report, over two months after the date of his supplementary report, and exactly one month after the
Horizon Issues trial itself had actually started. I requested a witness statement from the solicitors for
the Post Office setting out the chronology of this realisation by Dr Worden.
715.The case to which reference was made on behalf of the Post Office, ICI v Merit Merrell,
concerned a supplementary report by a quantum expert, instructed on behalf of ICI, which was
produced on the first day of trial without warning. This was based on an analysis of documents the
expert had only recently been provided with by ICI, which it turned out had been in ICI’s possession
for over three years, but which the expert had not previously been given. The application for
permission to adduce that supplementary report in that case was refused. Points relevant to the
application in that different case on those facts were set out in [158] of the judgment, which stated:
“Two other points relevant to the application should also be identified. Firstly, Mr Kitt embarked upon
this exercise with no notice at all to his opposite number, Mr Linnett. This is not co-operative
behaviour. Mr Linnett should have been told that this was being done, and the documents could have
RLITO000005
RLITO000005
even been studied together by both experts so that an agreed position could be produced by them on
what the documents did or did not show, and the use to which they could or could not be put. I
conclude that the only reason for keeping the fact that this exercise was underway, until after it was
ready to be served after the trial had actually started, was to cause maximum disruption to MMT. This
is a much discredited approach to civil litigation. I conclude that this can be the only explanation for
giving Mr Kitt the documents in a more usable (and searchable) format than Mr Mills, MMT's
solicitors.”
716.Giving permission for late expert evidence is a matter of discretion, and each case will be
different. At [237] in ICI v Merit Merrell I had stated the following, as one of a number of principles
that relate to expert evidence:
“5, Where late material emerges close to a trial, and if any expert considers that is going to lead to
further analysis, consideration or testing, notice of this should be given to that expert's opposite
number as soon as possible. Save in exceptional circumstances where it is unavoidable, no expert
should produce a further report actually during a trial that takes the opposing party completely by
surprise.”
717.Here, the following distinct matters fall to be considered. Firstly, just because in the ICI case that
expert performed a new exercise without telling his opposite number he was doing so (which led to
criticism), does not mean that all an expert has to do is to inform his opposite number during a trial
that he has decided to undertake a new exercise, or produce a new report, and that by giving such
advance notice this means the new report will be admitted. The whole ethos of exchange of expert
evidence well in advance of trial is so that each party (and their respective experts) have advance
notice of the opposing point of view, so that evidence can be properly addressed and surprise avoided.
That is undermined by service of very late reports containing new analyses. Sometimes this is
unavoidable and late material emerges. It is not possible to be entirely proscriptive, as each case is
different. That is why permission of the court is required. Secondly, the stage of the trial at which this
3rd Report was produced by Dr Worden was very late indeed. It was weeks after the trial had started.
Thirdly, the extent to which the court needs to consider all the circumstances depends upon whether
the 3rd Report was sought to be relied upon by the Post Office. Finally, and of wider application, is the
way in which the Third Report was dealt with, namely by service by the expert himself directly upon
the court.
718.Dealing with those matters in turn, the first two are effectively bound up with one another. I do
not know the actual date when Dr Worden realised that this new way of looking at things appeared to
him to be useful. However, the court was told he had started thinking about this on 11 April 2019, and
the claimants’ solicitors had already been told of his intention by the Post Office’s solicitors the
previous day on 10 April 2019 (one day after the recusal application was dismissed, and the day
before resumption of the trial). Thereafter, a draft report was provided on a without prejudice basis to
Mr Coyne on 25 April 2019 by Dr Worden, his “current workings” having been provided on the same
basis on 11 April 2019 and the appendices, again on a without prejudice basis on 25 April 2019. There
was then a degree of to-ing and froing between the solicitors, which led to the actual report itself
being sent on an open basis to Mr Coyne on 16 May 2019 (after another experts’ meeting on 15 May
2019).
719.Regardless of the exact date when Dr Worden realised there was a different way of looking at the
evidence in the case, this 3rd Report emerged at an extraordinarily late stage in the proceedings. As
RLITO000005
RLITO000005
can be seen in [721] below, it was in any case based on material which Dr Worden accepted he had
had for some time. It was not based on material he had only received late, for whatever reason.
720.Turning to the third of the matters identified in [717] above, this can be readily dealt with. The
Post Office made it clear, both on 11 April 2019 in the passages which I have quoted above, and
subsequently, that it did not seek permission to adduce the contents of Dr Worden’s 3rd Report in
evidence. It issued no application in this respect. Therefore, it was not necessary for the court to
consider the balancing exercise inherent in resolving any application in this respect (on the basis that
such an application was resisted by the claimants, which was likely). It is inherent within this decision
by the Post Office not to issue an application to adduce it therefore, that there was no explanation of
when or how Dr Worden had experienced this “new way of looking at the evidence”, other than what
has been identified above. It also meant that this report was not adduced as any part of his evidence-
in-chief, and he was not crossexamined about it. He referred to its existence only once, very much in
passing, and given that the contents of it had not been adduced in evidence, that was as far as it went.
721.Had I been faced with a contested application by the Post Office to be permitted to rely upon the
contents of the 3rd Report in evidence, the matter would have been considered in accordance with the
principles in the CPR generally, CPR Part 35, the Practice Direction that goes with it, as well as the
Guidance for the Instruction of Experts to Give Evidence in Civil Claims 2014 (“Guidance of Experts in
Civil Claims”) which is in the White Book with the Practice Direction. Doubtless such an application,
had it been made, would have been supported by a detailed explanation of all the relevant matters,
together with argument on both sides. I do not have the benefit of either a detailed explanation or of
any argument. On the material of which I am aware, but without hearing any submissions on it for
obvious reasons, a substantial hurdle to the success of such an application would have been its
lateness. This is because of the enormous disruption and yet further delay that would have been the
inevitable consequence of permitting reliance upon such a late expert’s report. Severe disruption and
delay had already been caused to the Horizon Issues trial by the time this report emerged, and by late
May, the trial was shortly about to resume after an enforced interval, in very unusual circumstances,
of over two months. Had this new report been admitted into the evidence, Mr Coyne would have
required time to study it, and then potentially produce a further expert's report of his own, and none
of that could have been accommodated between late May (whenever such an application could have
been listed) and 4 June 2019. Calling him would either have been delayed, or he would have had to be
recalled later to deal with it. Further delay to the Horizon Issues trial would have been undoubtedly
caused. Delay is not always fatal to such applications, but the later an application of this kind is made,
the stronger the explanation for that delay has to be.
722.The claimants had also made the point on 23 May 2019, the day after Dr Worden had served his
3rd Report upon the court, that it was based upon an analysis of documents, namely OCPs, OCRs and
MSCs, that Mr Coyne himself had expressly requested in July 2018 and which the Post Office’s
solicitors had then said were irrelevant. In any event, when I asked him at the end of his cross-
examination, Dr Worden confirmed that he had had the documents upon which he had based this 3rd
Report “for some months” and that he had had the full set of PEAKs since, as he put it, “way back in
2018”. As the Post Office’s own counsel explained on 11 April 2019, Dr Worden was “frankly, kicking
himself” that he “didn’t think of this before”. The lateness seems therefore to have been the
consequence of Dr Worden failing to consider this approach until very late in the day, rather than only
very recently being provided with the material. That is not promising ground for the making of such
an application. It is also the case, in my judgment, that there were no exceptional circumstances
which made it unavoidable that this new report was produced during the actual trial. Indeed, rather to
RLITO000005
RLITO000005
the contrary. This was entirely avoidable. Finally, the covering email from Dr Worden made it clear
that his estimate, included in his first report and revised in the 2nd Joint Statement of 0.4%, was “still
sound”. This new exercise simply sought to bolster his conclusions. It was not an explanation of a
change of opinion by him.
723.1 turn therefore to the fourth matter, the issue of direct communication with, and service upon,
the trial judge of an expert's report by the actual expert rather than through solicitors.
724.When the email was received to which I have referred at [708] above, my clerk immediately sent
a copy of the email and attachments to all the legal advisers for both parties in the litigation, stating
that witnesses should not communicate directly with the court, and also requesting that the claimants
should not read any of the attachments until the matter was addressed in court the next day (when
the parties were attending in any event to argue costs orders and other consequential matters
following Judgment (No.3)).
725.The legal advisers for the Post Office did not seem concerned at the hearing of 12 April 2019 that
this direct service upon the court by the expert himself had occurred. An analogy was drawn with an
expert seeking directions under CPR 35.14, and it was dealt with by the Post Office as though the
direct service upon the court was simply the actions of an expert seeking in the normal way to fulfil
his duties to the court, and was entirely usual. It is, however, not usual; it is highly unusual. Further,
the covering email from Dr Worden makes it clear that the new report was not a change of opinion; in
particular, the sentence that makes this entirely clear is “in my opinion, that estimate is still sound”
concerning the calculation in his earlier reports. His exercise was simply a different way of reaching
the same conclusion. Therefore, the provisions of paragraphs 64 to 66 in the Guidance of Experts in
Civil Claims do not apply.
726.1 do not consider that any witness, lay or expert, should communicate directly with the trial judge
about the content of their evidence, or indeed about any subject at all. There is a substantial risk that
the efficient management of proceedings will be compromised if this is done. Communications with
the court should always be done by solicitors, if they are on the record. Further, communications
between an expert and those instructing him, or her, are privileged. If, as here, an expert takes it
upon himself to inform the court of applications that his client may, or may not, be intending to make,
there is a real danger that privileged information might be divulged, both to the court and also to the
other side in the litigation. This is far less likely if communications are performed by solicitors, who
have specific qualifications in legal matters and will be experienced in litigation. Finally, there is
specific provision in the Guidance of Experts in Civil Claims dealing with the situation where an
expert wishes to exercise their right to ask the court for directions. This is contained in paragraphs 28
and 29. These proposed directions have to be sent to the party instructing the expert at least seven
days before the request is made, and to the other parties in the litigation at least four days before.
They are to be made by letter in the form prescribed at paragraph 29(a) to (e). That gives the other
parties the ability, firstly to have advance notice of what is proposed, as well as providing advance
notice that the court will be asked for directions and what those directions are, and why; and also give
the expert’s instructing solicitors the ability to consider the communication in advance so that
privileged information is not included.
727.In this case, that process was not followed. Mr Parson’s 17th witness statement explained the
outline chronology, and made it clear that Dr Worden’s opposite number Mr Coyne had a copy of the
3rd Report, initially on a without prejudice basis, and then on an open basis, and that Dr Worden told
RLITO000005
RLITO000005
Mr Coyne on 16 May 2019 (when he sent it on an open basis) that he would be sending the report to
the court but not before 22 May 2019.
728.Finally on this subject, when I asked Dr Worden at the end of his evidence whether he had ever
preformed this type of direct service himself on the court before, he told me he had not. He also said
that he had sent the email direct to my clerk as this was what he had been advised to do by the “Post
Office’s lawyers”. He said that he had not given his instructing solicitors the opportunity to check or
approve the covering email, but “had been given some advice as to what it should cover”. I assume
that advice did not include telling Dr Worden that he should inform the court of the Post Office’s
potential intentions on what applications it may, or may not, be minded to make in the litigation. This
is a highly unusual situation. In closing submissions the Post Office invited me to provide guidance in
this judgment as to what should be done for future cases, as though the position was unclear. I do not
know why it should be thought necessary to tell witnesses - lay or expert - that they should not
unilaterally communicate with the court. However, if it is necessary, no witnesses should communicate
with the court directly in this way. Any communications from legal advisers should always be copied to
the other parties in litigation. No expert should inform the court directly what applications his or her
solicitors might be contemplating, or not contemplating, in the future conduct of the litigation.
729.The points that I draw from this episode are as follows:
1.Experts should not serve their own material, particularly new material during a trial, and whether
or not accompanied by detailed covering emails, directly upon the court themselves. Service should
always be done by a party’s solicitors if solicitors are on the record.
2.A party’s solicitors will know that all communications to the court must be copied to the other
parties in the litigation.
3.Experts should not notify the court directly of the intentions of their instructing solicitors and/or
ultimate client in the litigation in which they are instructed. Such matters are privileged.
4.It is difficult to be critical of Dr Worden in this respect, given he told the court that he was advised
by the Post Office’s legal team to send his report directly to the court. Advice of that type should not
be given to experts.
5.Dr Worden must have considered that the documents upon which he relied were, at least, relevant.
These were documents of a type sought by Mr Coyne in July 2018, whose relevance was disputed at
that stage by the Post Office. Mr Coyne’s request ought to have led to these documents being
provided to both the experts in the middle of last year, a reasonable period of time after Mr Coyne
specifically asked for them.
730.1 will now turn to each of the expert’s evidence, their reports and their approach to the Horizon
Issues.
I. The Expert Evidence
731.The subject matter of this group litigation spans a long period. Mr Bates was a SPM before
Horizon was introduced in 2000, and it is from Horizon’s introduction that year that, on the claimants’
case, the problems and difficulties arose, and arose as a result of the functionality of the computing
system introduced by the Post Office, which in this judgment is referred to as Legacy Horizon. As the
Post Office pointed out regularly during these proceedings, there are thousands of branches and
RLITO000005
RLITO000005
millions of transactions each year in all of those branches, which means that over such a lengthy
period, there is a vast amount of computing activity that had to be considered by the experts.
732.The expert evidence was therefore required to be substantial. The aspects of the litigation that
require expert evidence are not, however, of unheard of proportions, so far as the High Court is
concerned. The High Court often deals with matters that span a lengthy period; it also often deals
with matters of great complexity that require expert evidence. By comparison with many other cases,
one expert discipline alone, with only one IT expert per party, is not a particularly expert-heavy case.
The experts’ reports, though lengthy, were not of an unusual size. There were two reports from each
expert that were adduced in evidence, and each of them was of a type that would be expected in a
trial of medium complexity, which I consider the Horizon Issues trial to have been. Nor was it
necessary for the experts to examine every claimant SPM’s separate experience; the whole purpose of
having generic Horizon Issues was to avoid precisely that. The Post Office would often concentrate on
how vast the panorama was before the court, but this may have been done simply to emphasise how
few, in relative terms, the claimants are; which also emphasised at the same time how well (so far as
the Post Office generally is concerned) the Post Office business operates day by day, week by week,
across so many branches. The experts agreed that Horizon has produced over three million sets of
monthly branch accounts.
733.At times, however, this focus on scale did seem to come close to an assertion that the IT experts
had an impossibly vast task; or that the court faced a significant challenge even in understanding all
the expert evidence; or that the parties could not practically deal with the other side’s expert evidence
in the time allotted.
734.As has been set out in Part G of this judgment, the experts reached a great deal of agreement in
their different statements. Notwithstanding that, there remained large areas of disagreement. Given
the modern approach to trying cases generally, it was neither possible, proportionate nor desirable to
give the Post Office as much time as their advisers maintained was necessary to conduct the Horizon
Issues trial. As an interesting comparison, in the case of The Ikarian Reefer dealt with further
below, which was before the implementation of the CPR and concerned the US$3 million loss of a
vessel, that trial took no fewer than 87 trial days. Such a lengthy trial, post the introduction of the
CPR, would be highly unusual in a case of the financial scale of this one.
Time limited trials and the approach to detailed evidence
735.Prior to the introduction of the Civil Procedure Rules, fewer trials were time-limited and some
trials dealing with a large number of complex issues could run for a great many months, if not far
longer. Since the introduction of the overriding objective, which is contained in CPR Part 1.1(1), this
states “These Rules are a new procedural code with the overriding objective of enabling the court to
deal with cases justly and at proportionate cost”. That is explained further in Part 1.1(2), which states
that:
“Dealing with a case justly and at proportionate cost includes, so far as is practicable: (a) ensuring
that the parties are on an equal footing;
(b)saving expense;
(c)dealing with the case in ways which are proportionate -
(i)to the amount of money involved;
RLITO000005
RLITO000005
(ii)to the importance of the case;
(iii)to the complexity of the issues; and
(iv)to the financial position of each party;
(d)ensuring that it is dealt with expeditiously and fairly;
(e)allotting to it an appropriate share of the court’s resources, while taking into account the need to
allot resources to other cases; and
(enforcing compliance with rules, practice directions and orders.”
736.Since the Civil Procedure Rules, trials are usually set down for a fixed time estimate and are
expected to be time-tabled to finish within that period. The court system does not have the luxury (if
luxury it is) of giving endless periods of time to litigants to argue their cases to the nth degree. This
case, as with all litigation, is very important to those involved in it - to all of the claimants, but also to
the Post Office, as an institution generally, and also to its other directors, employees, SPMs who are
not claimants and its clients too. The issues and different causes of action in both claims and
counterclaims are of considerable importance, and the amount of money at stake is not small.
However, the amount of money is not large in terms of specialist litigation, for example. It is not
measured in the billions, or even many hundreds of millions. Ignoring claims for general damages, the
amount of damages that are specified in liquidated sums are said to be about £18 million (although
that is only an approximate figure currently). There are, of course, also reputational issues at stake as
well as money.
737. Judgment (No.5) [2019] EWHC 1373 (QB) made it clear at [78] that the total of the parties’
reported costs as at the date of that judgment was over £25 million (£12.6 million for the claimants,
and £12.8 million for the Post Office). Since then, on 25 June 2019 the Post Office notified the court
that its costs had exceeded £13.9 million.
738.The reason for explaining this in detail is to put into context the following points about the scope
of expert evidence, and the number of trial days available for the experts to be cross-examined. At the
Pre-Trial Review, the Post Office submitted that at least four days was needed for the cross-
examination of Mr Coyne. It was realistically accepted that longer than that was not likely to be
permitted, although leading counsel for the Post Office did submit that he would prefer longer. The
claimants submitted that they also wanted four days, but had trimmed that period to three days, for
the cross-examination of Dr Worden. Given there were further experts’ meetings to come, and given
my view of the scope of the expert material and considering the overriding objective, I set a trial
timetable that permitted each side two and a half days to cross-examine the other side’s expert. That
would have given a total number of five days of cross-examination for all the IT expert evidence. At
that time, I considered that would be sufficient, and also proportionate, taking account of the
overriding objective.
739.However, the subsequent Joint Statements following the PTR did not reach as much agreement as
T had (perhaps naively) anticipated. I therefore adjusted the trial timetable and set a new one, with
the Post Office being given four days to crossexamine Mr Coyne, and the claimants being given three
days to cross-examine Dr Worden. That trial timetable also allowed an interval of one non-sitting
week, to follow the evidence of fact, for each side properly to prepare their time-limited expert cross-
examination in detail.
RLITO000005
RLITO000005
740.In my judgment, the period of time I had allotted at the PTR of 2% days each was sufficient in
itself; but the increased period of time allowed was undoubtedly sufficient for each party properly to
cross-examine and test the expert evidence adduced by the other side in the litigation. Seven court
days for the total crossexamination of one expert discipline in a case such as this is, given the Horizon
Issues in this trial, generous. Counsel might not have had time to put each and every single individual
point to the expert on the other side, but they had more than enough time to put the main ones (and a
considerable number of subsidiary ones too, for that matter). Each side had leading counsel, well-
versed in all of the issues, who have been involved in the case from at least the making of the GLO
itself in early 2017, as well as a team of other counsel assisting too. Considerable pre-reading had
been done by the court. As it turned out, the issuing of the recusal application on 21 March 2019 led
to an even longer period of the trial not sitting, with only a half day (for Mr Parker) of the trial taking
place between 21 March 2019 and 4 June 2019, a period of 75 days, or over ten weeks. The Post
Office had a different counsel team acting on the recusal application than the team acting on the
Horizon Issues trial. That 75 day period was a substantial period of time for each side to prepare
detailed cross-examination on all, and any, points that they considered important, prior to the
resumption of the trial and the commencement of the expert evidence on 4 June 2019.
741.The amount of the costs burden explained in [737] above has been incurred on the basis of the
number of trial days made available. Obviously, further time for crossexamination would have led to
even higher cost figures. There was a sufficient and proportionate amount of time at trial made
available for the expert evidence to be properly challenged and tested, and there was more than
sufficient time available for preparation of the time-limited cross-examination. I have set this out in
detail in order to make it clear that the time limited nature of the Horizon trial, and the number of
days allotted for cross-examination, has in my judgment had no effect on my findings on the Horizon
Issues, or restricted either party’s opportunity to put its own case and test the case of the other party.
742.The other feature of time-limited cross-examination, given the depth of experience of counsel, is
that I permitted considerable latitude to the cross-examiners. If counsel wanted to put the same point
to an expert again and again; or spend an hour on one specific sentence or detail, when the same
point could have been accomplished in five minutes; or devote significant time to high-level issues and
avoid descending to specifics; then that was a judgment call made by that counsel. With more junior
counsel, in another case, the court might discourage excessive time on less important or agreed
points, to encourage progress; but here both sides had highly experienced leading counsel. I therefore
intervened rarely. The amount of time made available in the trial timetable for each side to cross-
examine the other’s expert was set down in advance, and if one side wished to devote (say) half a day
to having an expert state, and then restate, an agreed position on a particular point, then that was
very much a matter for them. Such an approach would lead to that period not being available to
allocate to other subjects, but in my judgment this case in particular benefitted from allowing the
parties to conduct their cross-examination, more or less, in the direction they wished.
743.1 did, from time to time, seek either to have clarified what particular Horizon Issue was the
subject of the cross-examination, or to reference it to particular other subjects.
However, this did not occur very often, and the parties were permitted to use their allocated time as
they wished.
744.Each side, both in their cross-examination and closing submissions, criticised the other side’s
expert for not being sufficiently independent. This was done in different terms and in different ways,
RLITO000005
RLITO000005
but the upshot of it was broadly the same. It was that their expert’s evidence should be preferred to
that of the opposing side, for reasons of methodology and approach, as well as for reasons of detail.
745.The court, when faced with opposing experts, might dismiss the views of one expert entirely on
the basis that they had not complied with their duties to the court. However, it is not necessary to do
that, in order to prefer the views of one expert over another. There may, depending upon the subject
matter, be room for opposing expert opinion, and it is simply a case that the court accepts one opinion
for a variety of reasons. Also, in some cases, it might be that one expert's view is accepted on some
points, and the opposing expert’s views on others.
746.It is also not necessary to resolve every single point of detail between experts to arrive at the
answers to the Horizon Issues. However, given these criticisms of the experts have been made, it is
necessary to address them. This is because it is only fair both to the experts, and to the parties, to
decide whether each expert has been sufficiently impartial in the exercise upon which they have been
engaged such that the court can reliably base its answers to the Horizon Issues upon the conclusions
reached by the experts. Impartiality and independence of experts is also of considerable, if not
central, importance to the administration of justice generally.
747.The case that is most often stated in the context of the importance of independence on the part of
experts is National Justice Compania Naviera SA v Prudential
Assurance Co Ltd [1993] 2 Lloyd’s Rep 68 (Comm Ct), but it is almost always referred to by the
name of the ship involved in the case, The Ikarian Reefer, which ran aground off Sierra Leone and
then caught fire, becoming a total loss. Lord Woolf MR, the author of the Civil Procedure Rules, made
clear in Stevens v Gullis [2000] 1 All ER 527, 533 that CPR Part 35.1 and its accompanying practice
direction “did no more than reflect the position as it had been well enunciated in the authorities prior
to the CPR coming into force”. The Ikarian Reefer remains central to the way that experts should
perform their role, including the separate duty owed to the court.
748. In Imperial Chemicals Ltd v Merit Merrell Technology Ltd [2018] EWHC 1577
(TCC), a more recent case in which a number of experts all called for one party were found to be
partisan, at [237] the following was stated:
“The principles that govern expert evidence must be carefully adhered to, both by the experts
themselves, and the legal advisers who instruct them. If experts are unaware of these principles, they
must have them explained to them by their instructing solicitors. This applies regardless of the
amounts at stake in any particular case, and is a foundation stone of expert evidence. There is a
lengthy practice direction to CPR Part 35, Practice Direction 35. Every expert should read it. In order
to emphasise this point to experts in future cases, the following points ought to be borne in mind.
These do not dilute, or change, the approach in The Ikarian Reefer. They are examples of the
application of those principles in practice.
1.Experts of like discipline should have access to the same material. No party should provide its own
independent expert with material which is not made available to his or her opposite number.
2.Where there is an issue, or are issues, of fact which are relevant to the opinion of an independent
expert on any particular matter upon which they will be giving their opinion, it is not the place of an
independent expert to identify which version of the facts they prefer. That is a matter for the court.
RLITO000005
RLITO000005
5.Where late material emerges close to a trial, and if any expert considers that is going to lead to
further analysis, consideration or testing, notice of this should be given to that expert's opposite
number as soon as possible. Save in exceptional circumstances where it is unavoidable, no expert
should produce a further report actually during a trial that takes the opposing party completely by
surprise.
6.No expert should allow the necessary adherence to the principles in The Ikarian Reefer to be
loosened.
It is to be hoped that expert evidence such as that called by ICI in this case, and also in Bank of
Ireland v Watts Group plc, does not become part of a worrying trend in this respect. There are
some jurisdictions where partisan expert evidence is the norm. For the avoidance of any doubt, this
jurisdiction is not one of them. Not only experts, but the legal advisers who instruct them, should take
very careful note of the principles which govern expert evidence.”
749.The Ikarian Reefer, and the requirements of CPR Part 35, are the touchstones of expert
evidence in this jurisdiction. It is of the utmost importance that they are adhered to both by experts
themselves, and also by those who instruct such experts.
Mr Coyne
750.Mr Coyne is a Partner of IT Group UK Ltd, and has 30 years of experience in IT. He has designed,
programmed and supported Electronic Point of Sale (or EPOS) systems for use in payment handling,
stock control and distribution in addition to full business accounting systems. He has also assisted
public sector councils with revenue and benefits processing systems, investigated failures within stock
market trading systems, European gambling and gaming networks and dealt with fraud in retail
banking, point of sale, cash in transit and electronic funds transfer systems.
751.He is a member of the Academy of Experts, the British Computer Society and the Society of
Computers and Law. He has an Expert Witness Diploma from Cardiff University Law School, achieving
this in 2008. He has been instructed as an expert witness in both civil and criminal cases over the last
19 years, including by both UK and International firms of solicitors, in connection with technology
project fault analysis, software development and digital forensic investigations. He has appeared
before different tribunals both in the UK as well as overseas. He served two detailed reports with
numerous appendices. He also signed four Joint Statements with Dr Worden.
752.Both the experts agreed in the 3rd Joint Statement that Horizon as it is now is relatively robust.
The passages in that Joint Statement are that:
“in line with most other large-scale computer systems, Horizon's robustness has generally improved.
From our experience of other computer systems, Horizon is relatively robust. We agree that 'robust'
does not mean infallible and therefore Horizon has and will continue to suffer faults. Robustness limits
the impact of those faults and other adverse events. This increase in robustness has, in part,
developed from Post Office discovering bugs/errors and defects in live use and then applying fixes and
improving monitoring.”
753.1 consider that this is an important passage of the experts’ agreement, and indeed robustness is
an important aspect of the Horizon Issues. It makes clear that, in part, it is the discovery by the Post
Office of bugs, errors and defects, the application of fixes (which means changes to the software by
means of a software fix) and the improvement of monitoring that has led to the increase in robustness
that has been achieved in the system as it is at 2019. This inevitably means that in earlier times,
RLITO000005
RLITO000005
Horizon was less robust, or not as robust as it is now. This applies both to Legacy Horizon and also to
Horizon Online. There are undoubtedly more modern or recent bugs in Horizon, as can be seen from
the Technical Appendix. Drop and Go, for example, was not even a product until about 2013, and the
relevant bug in the Bug Table which is bug 28 occurred in July 2017. That caused an impact to the
branch account in question of £100; modest or tiny in comparison to the total Drop and Go business
for that year, or even that month, but still £100 which appeared as a deficit in the branch accounts,
and still £100 which the SPM had to deal with. Until it was discovered the bug could not have been
fixed. It would not have been present prior to 2013 (because Drop and Go did not exist) and any fix
cannot have occurred prior to July 2017, if not a month or two later. The agreement that Horizon is
relatively robust as at 2018 and 2019 is something that is doubtless of great importance to the Post
Office, as it affects how the Horizon system as it is at the date of this judgment is both used by the
Post Office, and perceived by the SPMs who use it every day. The Horizon Issues however deal with
Horizon from the year 2000 onwards.
754.The other aspect of this part of the experts’ agreements that bears consideration is that the Post
Office have to know about bugs, defects and errors in order that they can agree with Fujitsu that fixes
have to be written and applied. This makes it particularly important, in my judgment, that Fujitsu
were open and transparent with the Post Office about the presence of software bugs, errors and
defects. Given Horizon was being used by the Post Office as the accounting procedure between it and
its SPMs, there is every reason that Fujitsu would (or should) have to make the Post Office aware of
bugs, errors and defects.
755.A point that was put to Mr Coyne (it was taken from the Post Office’s pleadings, but it is a theme
that runs through both its submissions and also the evidence of Dr Worden) is that there are both
what are called technical controls, and also operational procedures and practices conducted both by
the Post Office and by SPMs, to increase the reliability of data. The passage of the Defence that was
put was as follows:
"Q. So robustness -- we can move on to paragraph 54 on page {C3/3/23}:
",., in addition to the technical controls referred to above, there are several operational procedures
and practices conducted by Post Office and subpostmasters that serve to increase the reliability of the
data stored in the central data centre as an accurate record of the transactions entered on branch
terminals.
Then there is a list of those. So you will see that what's being said is that the robustness of the system
is said to be affected both by technical measures and by human measures. Do you see that?
A.Yes, I can see that. That is what is said, yes.”
756.In my judgment it is correct to describe the second part of these, as the Post Office did in its
pleading, as “human measures”. They rely upon humans actually performing or being aware of certain
occurrences. Two of these rely upon SPMs, namely those at paragraph 54(3) and 55(4) of the Defence.
The second of these is that “the accounting and record-keeping obligations placed on Subpostmasters
reduce the risk of any errors going undetected.” In my judgment, for SPMs to be an effective - or
indeed any - part of a “human measure” and to assist or contribute to the process of the Post Office
discovering bugs, errors and defects in live use (so that, where necessary, software fixes can be
initiated, developed and applied) the Post Office ought sensibly to be transparent with SPMs too.
RLITO000005
RLITO000005
757.It does however need to be recognised that human measures are not part of the Horizon system.
They are not included in the definition of the Horizon system that forms part of the Horizon Issues
themselves.
758.It was also, in substance, put to Mr Coyne that Horizon Issue 1 should have been addressed in the
way that Dr Worden had approached it, by effectively looking at actual (rather than potential) impact,
although it was not put in exactly those terms.
“Q. Given it was admitted that there have been occasions when these things have happened, to
answer the question: is it possible or likely that these things have happened by saying "yes" isn't a
very helpful answer, is it?
A.If the answer was just simply "yes" and with no further context then I agree that that would ...
Q. The critical issue that's raised by the four issues that I have read to you is the word "extent" and
the word "likelihood" and the word "risk", isn't it?
A. They certainly are critical elements that would need to be reviewed, yes.
which it wi ind is likely thi shortfalls in any particular set of accounts?
A. Well. across the accounts of subpostmasters, yes.
Q. The extent to which various measures in place reduce the risk faced by any given subpostmaster
when doing a transaction or when producing a particular set of accounts, yes? A. Yes.
Q. That's the critical question on which the court needs your assistance, would you agree?
A. It is one of the critical questions, yes.
Q. Could you identify a question which you say is more critical than that in the context of these
proceedings?
A. I haven't taken a view on -- I have taken each of the Horizon Issues that's put to me with equal
priority. I haven't set out to decide a prioritisation for them.” (emphasis added)
759.This approach by the Post Office was, in my judgment, an attempt effectively to redraft Horizon
Issue 1 in terms more favourable to its case. I have already identified above that the Horizon issues
were drafted and agreed by the parties and approved by the court. Horizon Issue 1 plainly states “to
what extent was it possible or likely for bugs, errors or defects of the nature alleged....to have the
potential to (a) cause apparent or alleged discrepancies or shortfalls relating to” SPMs’ accounts. The
approach taken by the Post Office (and also Dr Worden) was entirely to ignore the word “potential”.
760.1 also consider that it is a matter for the court to decide which issues are more important than
others. An expert might decide a particular issue is important in that it represents a starting point for
their opinion, which will then inform or guide their views on other issues. But it is not for an expert to
decide one issue in a list is more important to the court than others, and concentrate upon that to the
potential exclusion of consideration of some or any of the others. Mr Coyne was therefore correct to
treat all of the Horizon Issues with equal priority.
761.Mr Coyne had not performed a similar statistical analysis to Dr Worden seeking to demonstrate
either the likelihood, or unlikelihood, of a bug impacting the claimants’ branch accounts. In my
RLITO000005
RLITO000005
judgment such an exercise was not necessary in order properly to give expert evidence to the court on
the Horizon Issues. Dr Worden’s evidence, and his alternative approach, was properly put to Mr
Coyne. Mr Coyne accepted that he could have approached his evidence in that way, but explained why
he had not. He described the approach adopted by Dr Worden as “a fundamentally flawed approach”.
“Q: You could, using your skills as an IT risk analyst, you could have, by reference to the KELs and the
PEAKs that you had identified, formed an assessment as to the likely financial impact in each of the
instances that you had identified?
AYes, I could. If I could please answer that by way of illustration to the problem with that approach.
Q. Please do.
A. We will often see a bug, error or defect with a very wide range of impacts and the impact is
typically whatever the counter was doing at that point in time. So if the counter was doing a foreign
currency transaction for just £50 and something goes wrong, the discrepancy may be £50.
By knowing that there is a bug, error or defect in the Horizon system that leads to problems with
foreign currencies, you can't then say it is only a £50 defect because that isn't incorrect. If another
person was to be subject to that defect and they were doing a £10,000 foreign currency transaction,
they would likely have that same level of defect.
Now that's an illustration to say you can't value bugs and their potential impact by looking at what has
happened historically. You can't value it in that way.
Q. But what you can do is form an estimate having regard to the totality of the PEAKs that you have
seen, can't you?
A, But it is a fundamentally flawed approach. If you have seen three branches that have had an impact
because they were doing three foreign currency transactions, a £20, a £50 and a £100, but then there
is a fourth person that believes that they have been subjected to that but that isn't recorded, you can't
simply look at the three where it has been recorded and say the fourth couldn't possibly have
occurred because we know there is a defect there.
Q. I'm not suggesting that you could arrive at a certain conclusion of an absolute cast iron number,
but I repeat my question. Can't you form an estimate having regard to the totality of the PEAKs that
you have seen?
A. Yes, but your estimate would have to be based on the three people where it has been recorded to
have occurred, so you would say £20, £30 and £50, and the best you could possibly do is come up with
an average of that, and you would say that the value of that defect is whatever that is.
Q. There is another way that you could do it, isn't there, which is that you could look at the three
bugs, the receipts and payments mismatch, the suspense account bug and Callendar Square, the ones
that have been thoroughly investigated, and you could form inferences from the scale of those bugs,
yes? Would that be reasonable? A. For those types of bugs, potentially yes.
Q. Those are quite large bugs, aren't they? They are not small bugs in the scheme of things. That's
why they were identified in the letter, because these were major bugs of which even Post Office was
aware?
RLITO000005
RLITO000005
A. Yes. And carrying on from the one we were not told about until later, the Dalmellington one, there
were some which were only pounds, just a few pounds, and I think there was at least one that was
£25,000.
Q. But just fixing our attention for the moment on those three bugs, would you accept that the
evidence indicates that altogether the total financial impact of those three bugs is no greater than
£100,000. Do you agree with that?
A. No, I haven't done that so I would not know whether that was right or close to right”.
762.The documentary evidence showed that Dalmellington had over 100 branches with some being
impacted more than once. Using the Post Office’s definition of “lasting impact”, which means the
branch accounts were given TCs to correct the effect that the bug had upon them, 114 of these
impacts were made good. Of the other four, two turned out not be the Dalmellington bug and the
other two had only very small financial discrepancies.
763.However, this bug, which had taken 5 years to be discovered, was put to Mr Coyne as a good
example of how countermeasures had to be considered.
“Q. In fact, Dalmellington is quite a good example of how countermeasures need to be brought into
account and how it is important to look at financial impact to make an assessment of the extent
question that's raised in Horizon Issue 1, isn't it? Because it is no good saying 118 branches were
affected and some of them were affected by large amounts, when in fact we all know that even before
the bug was detected the branches involved were in fact made whole anyway, yes?
Q. Because, as you have already agreed, to an outside observer it looks exactly like a human error
because it was a human error. The bug wasn't causing losses, it was causing errors to be made and
those errors were picked up, would you agree with that?
A. No, it wasn't a human error, it was a defect of the system. It made it look like a human error.”
(emphasis added)
764.The Dalmellington bug was not accepted as a bug in the Bug Table at the time of the 3rd Joint
Statement where it is Bug 4, but is now accepted as a bug by the Post Office but said to have
“transient impact”. I find that it is a bug, and it cannot properly be described - as the Post Office did
during the above cross examination- as human error. As Mr Coyne pointed out in his answers, it was a
defect of the system. It also did take 5 years for it to be discovered. One of the branches affected by it
had its branch accounts affected to the sum of approximately £25,000. The fact that a TC was issued
does not mean that it did not have an impact upon branch accounts. That impact may have been later
corrected by means of the issuing of a TC, but that cannot mask the fact that the branch account was
affected by that amount for the period of time until that correction was made.
765.Mr Coyne accepted that Dr Worden’s arithmetic was correct, and also accepted that the sum Dr
Worden had calculated in respect of Dalmellington (£100,000) was “quite small compared to the £19
million that’s claimed by the claimants in this case. It is less than 1%”. As Mr Coyne accepted, on the
“pure numbers”, that was correct.
RLITO000005
RLITO000005
766.Mr Coyne did not accept Dr Worden’s basic logic as a starting point, nor did he accept the logic
that there was “no evidence that” the claimants’ branches were “more or less likely to be hit by bugs
than your average Post Office branch”. I deal with that point in further detail below at [821] and [822]
below, but this amounts to an assumption by Dr Worden that a group of SPMs who specifically allege
they have experienced the effects of bugs are to be treated, in statistical terms, as though they are a
random group of SPMs of the same sample size drawn from the wider population of all SPMs. They
plainly are not a randomly drawn sample of nearly 600 SPMs. They are a very specific group (or
sample) of those who say their branch accounts have been impacted by, or have experienced, such
incidents. In statistical terms, the correct term for the group is that they have a bias - they all allege
that they have experienced the effects of bugs, errors and defects.
767.An illuminating series of questions and answers highlighted the entirely different approach of the
two experts.
"Q. Are you aware of some special factor applying to the claimant branches which marks them out as
very different from the rest of the branch network making their susceptibility to bugs very different
from the wide range of branches in the rest of the network?
A.I'm not aware of it, no, but it would require -- in order to look at that it would require a detailed
understanding of the particular processes of the branches. And I think that's illustrated by what we
know of Dalmellington, that impacted -- I think one branch was impacted five times, where there was
only a handful of branches ~ I
think, was it 88 that was impacted in total?
Q. Mr Coyne, it is quite important not to get confused between particular instances of things
happening and the law of averages. What I'm asking you about is what's likely to happen over a large
number of cases, do you understand the difference? So of course if people are all walking down the
streets, some of them are going to get hit by lightning. Everybody won't get hit by lightning in the
same way. It will be a very small number of people who will be hit by lightning and they'll usually be in
particular situations when it happens. But one can make a generalisation about the likelihood of
people being hit by lightning, do you understand, because of the law of averages?
A. I do understand, but in that illustration it would be very unusual for one person to be hit by
lightning five times.”
768.This then continued with some discussion of instances where some people may have been hit by
lightning more than once.
769.There are some important points that arise out of this approach. Firstly, the terms of the contracts
between the Post Office and an SPM does not invoke the law of averages, either as it was explained in
this cross-examination, or at all. The terms of the contracts, whether SPMC or NTC, have a very
precise accounting process which is run by Horizon. Secondly, the existence of a bug such as
Dalmellington, which was present in the Legacy Horizon system for 5 years prior to discovery, is of
direct relevance to the Horizon Issues. It does not support the proposition that a SPM with an
outreach branch in the 5 years before it was discovered was more likely (or equally likely) to be hit by
lightning as having their branch accounts impacted by the Dalmellington bug. Thirdly, even if the “law
of averages” approach were the correct one to adopt in terms of individual SPM complaints that their
branch accounts had been impacted by a bug, error or defect in Horizon (and to be clear, I find that it
is not) it is an exercise that could only sensibly even be attempted by firstly using the total number of
RLITO000005
RLITO000005
bugs present in the system (which would require at least this judgment, if not more investigation too
following the judgment), a definitive total for the number of instances of each bug occurring (again,
which would require another judgment), a definitive total for the full amount of losses caused by each
incident of each bug (information again not yet available, if ever) and some sort of probability
calculation taking account of all those matters. That calculation would only, in any event, lead to a
figure for the probability chances of any SPM in the period having their branch accounts impacted by
a bug. So many of the essential ingredients or pieces of data for such a calculation are not known, that
to try to attempt it in the way put by the Post Office, is to start from the end and work backwards, and
work backwards in a way wholly lacking in mathematical precision. The “law of averages” approach
cannot be used, in my judgment, in the way contended for by the Post Office.
770.So far as Dr Worden’s “scaling factor” is concerned, and his approach to the claimants’ branches,
Mr Coyne was entirely reasonable, in my judgment, in how he would consider hypothetical situations
put to him and also explain why they were not valid.
“Q. At this point in the equation what Dr Worden is trying to do is to work out how likely it is branches
are going to be affected? A. Yes.
Q. He reasons that branches with more transactions, doing more transactions, are statistically over a
large number of occasions more likely to be hit by a bug than a branch doing fewer transactions and
would you agree the principle underlying that observation?
A. think that's probably a reasonable principle. If we look at the defects we found though, there is
probably other factors that could be brought into that.
Q. But you are not aware of any, you are not in a position to suggest a single factor which you have
people that are impacted by the defect and I don't know whether any of the -- because I haven't
studied the claimants, I don't know whether any of the claimants have that or not.
Q. What you are doing is you are speculating that some of the claimants might have outreach
branches?
A. I'm not. I'm illustrating the potential problems with the approach that you are taking just simply
using the unit of number of transactions.”
(emphasis added)
771.Similar points can be made, in my judgment, to each of the bugs in the Bug Table, and not simply
Dalmellington and outreach branches. Some affect foreign currency transactions. Bug 9 in the Bug
Table, Reversals, only affected the value of transactions that were sought to be reversed by a SPM,
and were impacted by the bug actually introduced into Horizon by version S30 which was released by
Fujitsu. That this occurred at all was due to a failure in regression testing by Fujitsu prior to release
of S30. This would, if the bug hit that branch, show the value of a reversal twice in branch accounts.
The impact on the branch accounts would be the value of the reversal; if it were a large figure, the
RLITO000005
RLITO000005
impact would be large, and vice versa. Something of that nature cannot sensibly be dealt with by the
law of averages. Concurrent log ins, in the early days of Legacy Horizon, and numbered bug 18 in the
Bug Table, would only occur if a user was logged in at the same time to two terminals. Again, that
cannot sensibly be dealt with by use of the law of averages.
772.As Mr Coyne put it when asked about Dalmellington above and reminded that there are not many
outreach branches in the Post Office network and how insignificant this was, “well, it wouldn't be
insignificant to the people that are impacted by the defect”. This answer, which in my judgment is an
entirely accurate one in the context of this group litigation, is one that could apply to each of the
instances or incidents of each of the bugs, errors and defects impacting branch accounts that have
now come to light in the Horizon Issues trial. Individually and when considered across the whole of
the 3 million sets of branch accounts in the period from 2000 onwards, the amounts are individually
small amounts (although sometimes they are several thousand pounds). The same point can be
reached by simply taking the sum at stake in the group litigation in terms of accounting losses for all
the claimants (as opposed to other heads of claim not quantified) which is £18.7 million, and
comparing it to the total amount of business transacted in the Post Office branch network as a whole
across the entire UK from the year 2000 to date. No one has calculated that latter figure so far as I
know (and if they have, they have not brought it to my attention) but it will be an enormous sum. The
fact that £18.7 million is but a small proportion of that enormous
figure for total transacted business over 18 years is of no assistance whatsoever, and neither is the
law of averages approach.
773.Finally on this, in one of the answers to Mr Coyne’s requests for information (on another point,
discrepancies from external data sources), the solicitors acting for the Post Office pointed out in the
document dated 4 June 2018, the number of transactions that Horizon deals with each day. This was
done in order to give a sense of scale to the request posed by Mr Coyne. The exact answer provided
was “To give this a sense of scale, it is noted that Horizon processes around 6,000,000 transactions
per day.” The reason for identifying this is that this figure for transactions per day, together with the
agreed figure for the number of monthly branch accounts over the period agreed by the experts,
namely 3 million, gives a sense of scale of the Post Office’s business conducted by Horizon and the
total number of transactions over the period. Given there are fewer than 600 claimants, the number of
impacts on branch accounts complained of by the claimants is a very small fraction of this (whatever
its precise total). The law of averages approach does not assist the Post Office.
774.So far as countermeasures are concerned, Mr Coyne stated that “typically if you were going to
design a countermeasure you would want to have it finding the defect straightaway. If it was only
going to find it sometime later it would be of limited value.” Not only do I agree with that in general IT
terms, but also in terms of the Horizon system and its accounting function between the Post Office
and each SPM’s branch trading period specifically. A “countermeasure” in this instance would be
something that would correct an impact to the branch accounts. If that is only going to occur, as Mr
Coyne put it “sometime later” then it would only be of limited utility, if any, depending upon the period
of time this takes. Further, the Post Office used the term “countermeasure” as including “human
measures and controls”, as made clear to Mr Coyne during his cross-examination:
“I'm using it as a term to refer to measures and controls in existence, both technical and human
measures and controls, which are designed or have the effect of identifying bugs in Horizon or
identifying the consequences of bugs in Horizon so that they can be looked at and fixed.”
RLITO000005
RLITO000005
775.This means therefore that the Post Office was relying upon matters outside the definition of the
Horizon System at the beginning of the Horizon Issues, which stated “ “The Horizon System” shall for
the purposes of this list of issues mean the Horizon computer system hardware and software,
communications equipment in branch and central data centres where records of transactions made in
branch were processed, as defined in GPOC, at §16 and as admitted by Post Office in its Defence at
§37.”
776.As Mr Coyne explained, countermeasures in the system are addressed at the design stage:
“Typically countermeasures are a design aspiration. When you sit down to design a computer system
and what controls are in place, then you would consider what controls are required, where to position
them. Should it be the case that the controls fail do you have a human aspect doing a secondary
check? It could go all the way to employing auditors to re-check figures. You could go on ad infinitum
with -
Q. So you accept, then, that these controls or countermeasures, whatever you would like to call them,
some of them have the effect of stopping the bug in its track. Well, some of them have the effect of
stopping a bug arising in the first place?
A.The best way of dealing with defects is to ensure that they don't arise in the first place ........so what
you would do is throughout your testing regimes you would flush out all of the defects at that point in
time, and you would put in controls to ensure that if a defect does trigger, there is a way that you are
alerted to it and it is fixed as soon as possible.
Q. Yes. So one of the objectives is to have a testing regime which picks up as many bugs as possible?
A. Indeed.
Q. But I think you accept that it is impossible to have a testing regime that picks up all of them?
A. It is impossible to have a testing regime that picks up all of them and that is why you have a
process of categorisation and you would allow a product to go live if -- one of the terms that's used --
all severity 1 and severity 2 defects have been dealt with. Severity 3 might be largely cosmetic and
you would allow a system to go live with cosmetic defects.
Q. That's not quite what I'm asking you. You accept there are going to be bugs which arise in the
operation of any IT system which would simply evade any test that any human being can devise?
A. There's always going to be bugs that are unknown to be within the system at the point that you go
live and they are only discovered sometimes weeks afterwards, sometimes years afterwards.”
777.Both defensive programming and impact management are part of, or are descriptive terms
concerning, countermeasures. In my judgment they are both more to be described as “technical
controls” (to use the language of the Post Office’s pleading) rather than “human measures”. These
matters were all effectively part of the 3rd Joint Statement. So far as countermeasures within the
Horizon System itself are concerned, these plainly affect the robustness of the system. The 3rd Joint
Statement agreed that (using different numbering):
1.The effectiveness of various countermeasures changed throughout the life of Horizon;
2.Countermeasures are basic elements of practical IT system design;
3.Countermeasures work by limiting the impact of Horizon bugs/error and defects on branch
accounts. Countermeasures do not always eliminate the effects of adverse events (they are not
RLITO000005
RLITO000005
perfect) but they are often effective in the area where they are deployed; that is why they have
. f ical IT can.
4,It is difficult to measure the extent of the robustness of Horizon, apart from how it might limit the
extent of impact on branch accounts, as in Issue 1.
(emphasis added)
This shows that IT countermeasures are designed into systems. That is entirely to be expected.
However, it also means that the robustness of a system cannot be measured by other measures (such
as TCs, or a human complaints department as another example) taken to correct the effect of the IT
system. A ready way of illustrating this is as follows. Consider a hypothetical IT accounting system
that has 10 countermeasures designed into it, as a matter of IT design. It also has a human operated
complaints department. All 10 of the countermeasures fail so a user has to contact the complaints
department, which upholds the complaint and issues a correction, that undoes the impact within the
accounts the IT system has allowed to occur. The efficiency of that department, and the accuracy with
which it deals with such matters, cannot be used to assess in IT terms the robustness of the IT system
itself. It is separate.
778.The Post Office also has an inconsistent approach to TCs. On the one hand they are relied upon
(through Dr Worden) as being an important countermeasure - even though they are not part of the
Horizon system as defined - yet on the other hand questions about them were not dealt with as
though they were relevant to the Horizon Issues. For example, one of the requests raised by Mr Coyne
in respect of a particular PEAK was as follows:
“Regarding POL-0032919.pdf, the GoldenGate replication between Oracle 10g and 11g being aborted
and resulting in a number of branches reporting cash declaration and stock reporting discrepancies,
were any transaction corrections sent to the 247 affected branches as a result of the discrepancies
and which branches were affected by the incident?”
779.The answer that was given by the Post Office’s solicitors was as follows:
“The relevance of this request is not understood. Transaction Corrections are issued by Post Office to
correct transient discrepancies in branch accounts in order to restore the correct position.”
780.This answer to a request by the claimants’ IT expert therefore - correctly in my judgment -
identifies that TCs are not part of the Horizon system. That is entirely at odds with them being part of
a countermeasure within the Horizon system. This is effectively accepted in the Post Office’s opening
submissions which expressly state at paragraph 114 that the “Post Office’s reconciliation processes
are outside the scope of the Horizon Issues”. TCs are part of the Post Office’s reconciliation processes,
and I find that they are outside the scope of the Horizon Issues. Dr Worden therefore cannot sensibly
take them into account as a Horizon countermeasure. The Post Office cannot have it both ways.
781.One of the matters that Mr Coyne relied upon was what he described as “the sheer volume” of
KELs and reconciliation reports (a statement, it should be noted, made prior to discovery of the 5,000
KELs disclosed by the Post Office in October 2019). His evidence was that this confirmed “the wide-
ranging extent of the impact of such bugs/errors/defects. This evidence demonstrates that such bugs/
errors/defects would undermine the reliability of the Horizon system to accurately process and record
transactions”. He accepted that there was the need for reconciliation reports, and that this was part
of the way that a system would check whether (say) the banking records at the Post Office
transactions matched the corresponding records at (say) the Bank of Ireland for the same
RLITO000005
RLITO000005
transactions. In my judgment KELs are indeed part of the records (or contemporaneous records) that
can be considered in order to arrive at an overall conclusion for the existence and potential impact of
bugs errors and defects, but the
PEAKs need to be consulted too. This is because the PEAKs are the documents that often have greater
information within them, added to which one single KEL could relate to a couple of short PEAKs, or
alternatively to many a great many far lengthier PEAKs, and it is only from the PEAKs themselves that
one can gain a better view of the impact of that particular bug. Indeed, some of the KELs simply have
a start and end date, with very little information on the number of impacts during the intervening
period. It is clear to me that if there is more than one PEAK referring to the same KEL, one would only
realise that there had been a re-occurrence of the same defect if one considered the PEAKs; the KEL
would not show this. Simply looking at the KEL could give the erroneous impression that such a defect
had only occurred once. It is also telling, in my judgment, that some PEAKs that are clearly related to
errors, for example the PEAK associated with the issue that the subject that “SSC Database users do
not have correct permissions”, namely PC0208119 dated 10 March 2012, expressly states in one of
the 25 October 2011 entries the following “Have relevant KELs been created or updated? No. None
believed to be required”. This clearly demonstrates that Fujitsu would not always even create a KEL
for known errors (as SSC users having the incorrect permissions, all having APPSUP when they were
not supposed to have this, is clearly a known error). SSC users were (as set out in the PEAK) only
supposed to be given “the optional role ‘APPSUP' temporarily (by Security Ops authorisation) if
required to make emergency amendments in BRDB Live”. I find that the KELs alone do not constitute
a record of all bugs, errors and defects.
782.Reconciliation reports are only generated when the information the Post Office has about a
transaction or series of transactions is different from the information held by the client. These are
called exceptions. Mr Coyne was asked why he considered the volume of these in the history of
Horizon was confirmation of his conclusion relating to the wide ranging extent of bugs, errors and
defects having impact on branch accounts.
“Q. But I don't understand, Mr Coyne, why you think that the existence of reconciliation reports
confirms the wide-ranging extent of the impact of bugs in Horizon which are having an impact on
branch accounts?
A.Lhave worked and designed banking systems. stock broking systems. I have never seen the need for
f f wi - ion. TI
something is going wrong. It is working outside of process on a larger scale than I would have
expected.”
(emphasis added)
783.The origin of the number of reconciliation reports is as follows. Mr Coyne had asked, in requests
for information prior to the trial, what was the purpose of setting an NB102 exception to F99 by
Fujitsu. The Post Office in the answer explained that this explanation would have to come from
Fujitsu, and the explanation was given:
“When there is an incident involving a reconciliation exception in Network Banking which has been
fully processed, then the transaction needs to be set to F99 to indicate that processing is complete.
Therefore, this is done for any transaction that appears in a reconciliation report, once the resolution
is complete.
RLITO000005
RLITO000005
The reconciliation process document is SVM/SDM/PRO/0012 RECONCILIATION AND INCIDENT
MANAGEMENT PROCESS and describes how reconciliation is carried out.
SVM/SDM/SD/0020 END TO END RECONCILIATION REPORTING is the specifications of the reports
that are produced for the reconciliation service.
The CCD that describes the contracted service:
SVM/SDM/SD/0015 RECONCILIATION SERVICE: SERVICE DESCRIPTION.”
784.The next request was “How often has [setting an NB102 exception to F99] occurred?”
785.Initially the Post Office would not answer this, stating inter alia that it was very unlikely that the
information would have been pooled or collated. When Mr Coyne persisted, and pointed out it would
be very different if the answer was “it was an isolated incident in 2003” or “it was 10,000 transactions
each day for the last ten years”, an answer was provided by the Post Office through its solicitors. I
shall reproduce it in full:
“Fujitsu currently “F99" 10,000+ transactions per week across all NB102 associated reports (DCP and
NBS)”.
786.This was correctly understood by Mr de Garr Robinson for the Post Office who (understandably)
corrected Mr Coyne when he initially said he thought it was hundreds of thousands per week.
“.... asked for the number of reconciliation problems that required manual intervention, and the
answer I was given was that it was hundreds of thousands per week, I think was the -
Q. 10,000 a week that have to be F99ed, do you remember that?
A. Yes.”
787.So far as the next request, which was associated with this, is concerned, that paints the same
picture of the Horizon system in my judgment. The request was:
“What is the cause of an ‘Uncleared Transaction Corruptions’ and how often do
[{Uncleared Transaction Corruptions] occur?”
The answer was, again from Fujitsu:
“The DCP report NB102 section 3 (Uncleared Corruptions) usually contains 100s if not 1000s of
transactions per week (Kiosk transactions only).
This is where there is a discrepancy on banking and payment card transactions. To determine the
exact cause of any individual Uncleared Transaction Corruption, that particular occurrence would
have to be analysed. However, the most usual cause is for a banking request or authorisation failing to
reach its destination, which is something that can have a variety of causes. An example may be that a
bank may authorise a transaction, but that confirmation is not delivered to the counter so the SPM
believes that the transaction in fact failed due to some issue.”
788.This is a great number of transactions per week that require manual intervention - over 10,000,
on the numbers provided by Fujitsu. I consider that Mr Coyne is correct when he says “this suggests
that something is going wrong”. I do not accept that on a properly functioning and robust system
RLITO000005
RLITO000005
there should be such a high number as that every week. I accept Mr Coyne’s evidence in his answer at
[782] above.
789.Mr Coyne also considered, in respect of the control environment, that the Ernst & Young
management letter of 2011 had recommended changes to strengthen the change management
process. E&Y had noted that the Post Office was not usually involved in testing fixes or maintenance
changes to the in-scope applications (which includes Horizon Online) and they were unable to identify
an internal control with the third-party service provider to authorise fixes and maintenance changes
prior to development for applications. The risks were outlined by E&Y in the following terms:
“There is an increased risk that unauthorised and inappropriate changes are deployed if they are not
adequately authorised, tested and approved prior to migration to the production environment”
Mr Coyne also drew attention to a PEAK some years earlier than this (in 2003) that provided evidence
that Fujitsu did not even always follow the correct processes that were in place in any event. In this
instance a branch terminal was replaced, which resulted in a number of bills which branch customers
had paid (in terms of making payment at the branch Post Office) not in fact being paid (to the ultimate
payee, the Post Office clients that had issued the bills). The PEAK stated that “something went wrong
and the marooned transaction did not find their way back to the office”, something which had
occurred because the mirror disc in the branch terminal was replaced (when it should not have been).
The PEAK also stated:
“I believe that we shall have to confess to POCL that we have not followed the correct procedure and
we should advise that POCL make manual payments.....
POCL means the Post Office, under its earlier name of Post Office Counters Ltd.
790.Mr Coyne was of the opinion that the specific recommendations in the 2011 E&Y letter directly
impacted upon Dr Worden’s view of countermeasures. He also drew attention to the fact that the non-
acceptance of these recommendations by the Post Office was an opportunity to impose or improve
countermeasures in Horizon which the Post Office chose not to take. The four areas, and the
countermeasures they would impact, were:
“1. Improvements to outsourcing application management (Quality and Change
Control; managing non-functional requirements; testing good practice);
2.Improvement to segregation of duties within the “manage change” process (Quality and Change
Control, Security);
3.Strengthen the change management process (Quality and Change Control);
4,Strengthen the review of privileged access (Security).”
791.Mr Coyne also relied upon the fact that although both Legacy Horizon and Horizon Online did
contain a number of measures and controls, these mechanisms had been shown to have failed, and
this was agreed by the experts. The minutes of the Post Office’s own Risk Compliance Committee
meeting of 18 September 2013 showed that not all of the risks identified in the then-recent E&Y audit
had been addressed. All four pages of these minutes were heavily redacted, but one surviving entry
shows that risk not addressed was “relating to the communication by Fujitsu of changes made to the
Horizon system, was still outstanding. It was identified that it would cost over a £1m to implement the
mitigation being suggested by the audit and that this was not proportionate to the risk being
RLITO000005
RLITO000005
managed.” Mr Coyne accepted in his crossexamination that this did not relate to a bug being fixed,
and I accept that evidence. In my judgment it does go to the robustness of Horizon.
792.There was one area which was concentrated upon towards the end of his evidence, and this was
an entry in the 2nd Joint Statement. This was dated 25 February 2019, and under Horizon Issue 1,
under the topic “the extent bugs were possible or likely” the experts had agreed the following entries
relevant to this:
“1.1 We agree that bugs were possible, and the table above displays a number which in the opinion of
either or both experts appeared to have impacted branch accounts.
1.2 Referring to the table of bugs above, the experts agree that the bugs in rows 1, 2, 3, 10, 13, 14,
18, 23, 24, 25, 27 and 28 may have had financial impact on branch accounts. Other rows, the impact
is not agreed between the experts.”
And later on in the 2nd Joint Statement:
“1.15 The number of distinct bugs, for which the experts have seen strong evidence of the bug
causing a lasting discrepancy in branch accounts, is between 12 and 29.”
793 Just before the short adjournment on the final day of his cross-examination, Mr Coyne was asked
about the figures in that range. He was also asked to “forget Horizon Issue 4”. Given that Horizon
Issue 4 asked “to what extent has there been potential for errors in data recorded within Horizon to
arise in (a) data entry, (b) transfer or (c) processing of data in Horizon?”, being asked to ignore it
somewhat narrowed down the utility of the exercise, but it is without doubt the case that Mr Coyne,
when asked for a number for the upper range in the Joint Statement, stated it should not be 29, it
should be a different figure. He initially stated that it should be 13. The court then rose for the short
adjournment. As soon as court resumed one hour later, he immediately sought to correct the number
13 that he had given. He said the number should be 21. A complication in this cross-examination
exercise was that initially Mr Coyne was using what he called his bug table in his 2nd report (which
had been dated 1 February 2019, and amended on 6 March 2019) and the cross-examination was
being done by reference to something also referred to throughout the trial as “the Bug Table”, which
was the one in the 2nd Joint Statement. The former in the report had a column headed “evidence of
branch impact” with the entry “yes” or “no”; the entry in
the Joint Statement about which Mr Coyne was being asked was item 1.15 and is as set out above. The
questioning was done solely by reference to Horizon Issue 1. Table 1 in Mr Coyne’s 2nd report made it
clear that Phantom Transactions, Reconciliation Issues, Concurrent log-ins, Post & Go/TA
discrepancies in POLSAP, Recovery Failures and Transaction Correction Issues were all listed under
Horizon Issue 4, and not Horizon Issue 1. An associated difficulty was the distinction that was being
made by the Post Office between lasting and transient effect.
794.The correct figure being supported by Mr Coyne in his evidence was explored extensively in
cross-examination. Mr Coyne did explain this as follows (in part only):
“Q. Are you suggesting that you have only ever suggested 21 and no one was suggesting anything
above 21?
A. No. There are 29 bugs, errors and defects in the list and in the table starting at number 1, but some
of those go to other Horizon Issues apart from Horizon Issue 1, and they do say that clearly in the
heading.”
RLITO000005
RLITO000005
795.There is no doubt that Mr Coyne made matters a little hard for himself during this passage of
questioning, and the Post Office seized upon his change of number (which was, after all, an agreed
number in an agreed statement by both experts) and made much of it:
“Q. Mr Coyne, everybody on this side of the court, and I mean everybody on this side of the court,
believed that your view was that the evidence showed that there were 29 bugs with lasting financial
impact. You are now saying that we made a mistake, are you?
A. Yes.
Q. Could I ask you to explain where you made it clear what you were actually saying?
A. Well, as set out in my second report, there is a table there that quite clearly says whether there is
evidence of branch impact or not. And there is also a list of which Horizon Issue it specifically relates
to. The only addition to that is when you come to the very last column, there's other bugs, errors and
defects that are re-introduced and they are only covered in the joint statements.”
796.Mr Coyne also said:
“A, Throughout the report I have set out next to every single defect whether it is relating to the test of
Horizon Issue 1 or a later Horizon Issue, and I have also set out whether I believe there was impact on
branch accounts all the way through report number 2.
Q. Let's not talk about report number 2 because the statement -- the most up to date statement of the
position, I think you agreed with me at the beginning of today, the culmination of all your work,
including work you did after your second report, is the bug table in joint statement 2?
A. Yes, but when you put that to me I said that that can't be read in isolation, it must be read with the
second report.”
797.So far as the number is concerned, Mr Coyne settled on 22 different bugs, his explanation of his
earlier different numbers was as follows:
“Q. And I would like to ask you why is it you thought there were 21 bugs at 2 o'clock, whereas you
thought -- was it 14 bugs at 1 o'clock? 13 bugs at 1 o'clock.
A. Because I was just counting down the table at number 1 in my report. In the pressure of the
situation I just tallied that column down, I didn't have enough time to consider it fully.
Q. To go back to a question I asked you earlier, where do you make this clear in the joint statement?
A. In the heading next to the -- in the bug, error or defect.”
798.1 accept that explanation. Although Mr Coyne was criticised for certain entries in the 2nd Joint
Statement, including the headings, and it was put that the “joint statement was seriously misleading”
I do not consider this a valid criticism of him. It was a statement the text of which had been agreed by
both experts, and it appeared in an experts’ agreed joint statement. There is no doubt that the upper
figure of 29 would most likely have come from Mr Coyne, with the lower figure of 12 having come
from Dr Worden (and the claimants’ legal team interpreted the figure of 29 in the same way as the
Post Office, as it appeared in their written opening) but there is a limit to how much can be put on this
point by the Post Office. Joint Statements are (or should be) reached by the parties’ independent
experts between themselves.
RLITO000005
RLITO000005
799.It is the experts’ responsibility to agree the content of their Joint Statements, and this is part of
their duty to the court as independent experts. It is the basis on which the court gives permission for
expert evidence. No authority is required for these propositions as this is wholly conventional, but to
the extent that authority may assist,
Males J (as he then was) fully explained this in [3] of Mayr and others v CMS Cameron McKenna
Nabarro Olswang LLP [2018] EWHC 3669 (Comm). The agreed text of the agreed parts of a joint
statement therefore cannot be approached as though it were a unilateral passage in only one of the
expert's reports. It is an agreed passage. I do not consider that Mr Coyne’s changes of the number he
was asked about demonstrates any unreliability on his part, or discredits him or his evidence.
800.1 consider Mr Coyne to have been a helpful and constructive witness, and I find the suggestions
made to him that he was biased to the claimants and not independent are criticisms that are not
justified. He, and his small number of assistants, had done a great amount of investigation into the
very numerous PEAKs, and the smaller number of KELs, and he had embarked upon a careful and
sensible exercise necessary for him to reach conclusions on the Horizon Issues, as drafted and agreed
by the parties and as approved by the court.
801.There were some fundamental methodological differences between him and Dr Worden, but the
agreements that the two of them reached were detailed notwithstanding this. I found the contents of
all of those agreements of great assistance.
Dr Worden
802.Dr Worden was educated at the Université de Grenoble, Cambridge University and the California
Institute of Technology, better known as Cal Tech. He has a Ph.D in Theoretical Particle Physics. He is
currently a consultant with a practice called Charteris Associates, having started there in 1997, but he
has been a consultant since August 2010. Starting with the first entry on his CV in terms of his work
experience, between 1975 and 1986 he worked on the development of knowledge-based systems and
developed Logica’s proprietary relational database management system, and between 1986 and 1990
he was the Managing Director of Logical Cambridge. This is an advanced technology subsidiary of
Logica. Between 1991 and 1995 he acted as an independent reviewer of many of Logica’s largest
projects, the key projects that constituted a large part of Logica’s commercial risks, and also a large
part of its commercial reputation. Between 1995 and 1997 he was a Business and Technical
Consultant in Logica, moving to Charteris in July 1997. He has given evidence in the High Court
before on several occasions, including in what was at that time the biggest IT dispute tried, namely
BSkyB Ltd v HP Enterprise Services UK Ltd (formerly EDS
Ltd) [2010] EWHC 86 (TCC). This case is notable for a range of reasons, not least that the judgment
(excluding quantum) runs to some 2350 paragraphs; and also that the educational qualifications
considered in the judgment included those of one main factual witness who turned out to have
fabricated the establishment from where he claimed to have been awarded a MBA. This was a point
demonstrated by counsel enrolling his own dog (“Lulu”) in the same establishment, Lulu gaining an
MBA with higher marks than the witness. The hearing ran for 109 hearing days and Dr Worden was
cross-examined in that case for more than 35 hours, which is a very considerable period of time for
anyone to be in the witness box. In his CV he identifies that the majority of his opinions referred to in
the judgment of Ramsey J were accepted by him. At [277] in that judgment he was described as “more
of a generalist in the field of IT with some experience dealing with CRM systems but with good
general knowledge of the estimating, systems integration and the types of problems which can arise
on projects such as” the project in that case.
RLITO000005
RLITO000005
803.He is a highly qualified person, both in IT generally and specifically, and in statistics. He has
earlier experience with relational database management systems when he was at Logica in the late
1970s and early 1980s, and from the dates it is possible to tell that he has been involved in the IT field
for many decades, and (in comparison with many IT personnel today) from its infancy. He has also,
whilst at Charteris, developed and deployed software for data integration and migration. He described
himself as having done databases “since the year dot” at one point. He describes himself in his CV as
having strong practical experience of information technology and its business applications. He has
several years’ line management experience, has successfully managed large IT projects, and acts as a
trouble-shooter of major integration projects and consultant in information strategy. He has particular
expertise in XML, in largescale information architectures and tools, data integration and data
migration, scientific data processing, and in system performance. He explained that as he is trained as
a scientist and engineer, he has used mathematics all his life and part of that mathematics is
probability theory and statistics. Given the subject of his Ph.D, it will be no surprise that he uses
mathematics extensively. I doubt that there are any theoretical particle physicists who are not
extremely skilled and expert at mathematics at a very high level indeed. He therefore has great
experience in statistics, and at one point in his cross-examination was described as a statistician,
something with which he did not disagree. He does not however, as he put it, have any formal
qualification in statistics.
804.He is also a highly experienced expert witness, as demonstrated not simply from the BSkyB case
I have referred to above but many others, and he has given expert evidence in different cases in the
Technology and Construction Court, Chancery Division and Queen’s Bench Division too. He served
two detailed reports with numerous appendices (adding schedules of corrections in the usual way in
supplementary evidence in chief), and he also took part in the joint experts’ exercises and signed the
four Joint Statements with Mr Coyne. He also served a third report directly upon the court, but no
application to adduce this as part of his evidence in chief was made by the Post Office and it did not
form part of his evidence.
805.In the instant case, he was cross-examined for three days. One of his corrections in chief was in
respect of paragraph 761 of his 1st report where he had stated the following in respect of his
summary of what was called a statistical analysis of the likelihood of the bugs that had been found by
the experts being the cause of the claimants’ shortfalls and discrepancies.
“759. In section 8.5, ‘Scaling of Financial L { Bugs’. Lthi i
f . . bugs in Hori ; ; f the Clai :
shortfalls. I summarise it here.
760.Because Post Office has had an average of 13,560 branches over the lifetime of Horizon, the total
number of monthly branch accounts has been about 3 million.
761.Therefore, if a bug like the Suspense Account bug has occurred 16 times in the lifetime of
Horizon, the chance of it having occurred in any given branch in any given month is about 16 in 3
million. Because the Claimants tended to have smaller branches than the average, doing fewer
monthly transactions (by a factor 0.37), the chances of the bug occurring in a Claimant's branch
would be about 2 in 10 million.
762.1 have considered a bug similar to the suspense account bug, which occurred about 10 times, and
had a mean financial impact of about £1000 per occurrence. How many similar bugs would be needed,
to give a one in ten chance of one such bug occurring, with an impact of £1000, on a particular
Claimant's branch in a particular month?
RLITO000005
RLITO000005
763.The answer, given by elementary arithmetic which I describe in section 8.5, is that there would
need to be 50,000 of these distinct bugs. If there were fewer than 50,000 similar bugs, if any Claimant
were to assert that in a given month a shortfall of £1000 in his accounts was caused by bugs in
Horizon, then the chances of his assertion being correct are less than one in ten.
764.So the Claimants cannot credibly assert that their shortfalls were caused by bugs in Horizon,
unless there were something of the order of 50,000 such bugs.
765.Only three such bugs have been found. My own search of KELs has found only 8 other possible
bugs. Mr Coyne's examination of over 5000 KELs has found no other bugs which definitely caused
shortfalls.
766.Thus the Claimants' case requires 50,000 bugs in Horizon - but only a handful have been found by
the experts. Neither expert can quantitatively support the Claimants’ case.”
(emphasis added)
806.His correction was that in respect of paragraph 761 the sentence that reads "the chances of the
bug occurring in a Claimant's branch would be about 2 in 10 million" should be “the chances of the
bug occurring in a Claimant's branch would be about 2 in a million". He said that this was a
typographical error. When he was asked if he had spotted any other errors, he said:
“A. No, I didn't because there were quite a few statements of this "in a million" nature in my report. I
made careful checks of the main thread that led to my key result in section 8.7, and that was my
focus. And similarly on the transaction corrections I had made checks to that spreadsheet and, you
know, I included the spreadsheets with the reports so that you can see how the arithmetic is done
from the spreadsheets.
Q. And some of it you point out is basic arithmetic?
A. I think all of it is basic arithmetic.”
807.He said, in relation to a question as to whether Mr Emery, his assistant, had “checked the maths”
that “No, he didn't actually. The maths is there in the spreadsheets for anybody to check. It is not
advanced statistics, it is basic multiplication and division, and it is laid out in the spreadsheets.”
(emphasis added)
808.His methodology in terms of his statistical analysis was explained by him in section
8.5 of his 1st report. This stated:
“617. I shall assess the financial impact of bugs in Horizon over three different scopes:
617.1.(a) Across all Post Office branches, during the lifetime of Horizon.
617.2.(b) Across all Claimant branches, while they held them.
617.3.(c) On a single Claimant branch in a single month.
618.It is possible to relate the financial impacts on these scopes, by numerical scaling factors. I
calculate those scaling factors in this sub-section.
619.Over the period 2000 - 2018, (i.e. 19 years) the Post Office network has consisted of more than
11,000 branches. The mean number of branches in all years over the period has been about 13,560.
RLITO000005
RLITO000005
This figure is derived from the spreadsheet referred to at paragraph 178 of Ms Van Den Bogerd's
Second Witness Statement, assuming that the spreadsheet is accepted. If this evidence is accepted,
the number of ‘branch months' (a single branch, trading for a single month) has been 13,560 * 12 * 19
= 3,091,680. This is the number of monthly branch accounts that have been produced.
620.This means that for a typical Post Office branch, the scaling factor between scope (a) and scope
(c) for the impact of bugs in Horizon is a factor of 3 million.
621.For Claimants' branches, rather than typical branches, the scaling factor of approximately 3
million may need to be adjusted for two possible effects:
621.1. It might be asserted that Claimants’ branches are more or less likely than other branches to be
hit by bugs in Horizon, because of some special property of Claimants, 621.2. Claimants' branches
may, on average, be smaller or larger than typical branches across Post Office network. If they are
smaller, they handle fewer transactions in a month, and so are less prone to Horizon bugs in those
transactions.
622.IL seems implausible to me that there is some special factor about Claimants' branches. which
makes them much more prone to bugs in Horizon - bugs which one would expect to strike any branch
atrandom. Nevertheless, I have considered the possibility carefully in Appendix F. I have shown there
that there is no significant difference between Claimants' branches and other branches, in proneness
to bugs in Horizon.
623.It appears, from the spreadsheet attached at paragraph 179 of Ms Van Den Bogerd's Second
Witness Statement, that the Claimants’ branches are, in terms of customer transactions carried out
per day, smaller than the average across the whole Post Office branch network.
624.If this spreadsheet is accepted, it implies the following about Claimants’ branches:
624.1.From summing rows of the spreadsheet, the 561 Claimants' branches carried out 558,000
customer transactions per week in 2007.
624.2.This is 558,000/6 = 93,000 transactions per day, assuming a Post Office branch is open for 6
days a week.
624,3.Across 561 Claimant branches, this is an average of 93,000/561 = 165 customer transactions
per branch per day.
625.For comparison with this figure, I need to estimate the average size of branches across Post
Office Network. I have done this using two pieces of evidence. Ms Van Den Bogerd's Second Witness
Statement says that, across the whole Post Office network, there are approximately 48 million
customer transactions per week, or 8 million per day in 2017 (assuming again that branches are open
for 6 days in the week).
[626. I have omitted the table of EPOSS Transaction Rates, which are then summarised in 627]
627.The best estimate I can make of the average daily volume in 2003 from this table is to divide the
peak month by 26 working days. This gives an estimate of approximately 4 million transactions per
day in 2003, compared with 8 million transaction per day in 2017.
628.1 therefore estimate that the average volume of transactions over the period 20002018 has been
approximately 6 million transactions per day, mid-way between the 2003 figure and the 2017 figure
RLITO000005
RLITO000005
(assuming Mrs Van Den Bogerd's evidence is accepted. I note that Mr Coyne quotes the same figure
from her).
629.To estimate how much smaller the typical Claimant's branch is than the average Post Office
branch:
629.1.The average transaction rate for all Post Office branches is 6,000,000/13,650 = 439 customer
transactions per branch per day.
629.2.So, in terms of customer transactions per day, the typical Claimant branch was smaller than the
average Post Office branch by a factor 165/439 = 0.37.
630.So, Claimants' branches, being generally smaller than Post Office average, have fewer
transactions per month and so are less likely to be hit by a Horizon bug in a given month. (The
calculation of the factor 0.37 contains some minor approximations that can be improved with further
effort. I intend to do so in my supplemental report.) The factor 0.37 increases the scaling factor above,
between scopes (a) (see paragraph 617.1) and (c) (617.3) from about 3 million to about 8 million.
631.1 illustrate what the factor of 8 million means using a hypothetical example of a bug which has
occurred 16 times over the lifetime of Horizon, with mean financial impact on these occasions of
£1000. Call this Bug A. The financial impact of Bug A is similar to that of the Suspense Account bug.
632.If I selected a Claimant's branch and a month at random, then the chances of Bug A occurring at
that branch in that month are only 16 in 8 million, or 2 in a million - an extremely small probability.
633.Different types of bug occur independently of one another, so their probabilities of occurring are
additive. If there were a second bug similar to the hypothetical bug above - call it Bug B - then the
chances of either Bug A or Bug B happening to one branch on one month are twice the previous figure
- 32 parts in 8 million.
634.If there were 100 similar bugs - called Bug A, Bug B, Bug C,... Bug Zz - the chances of any one of
them happening to one branch on one month are still only 100 * 16 parts in 8 million, or one part in
5,000. This is still a very small probability.
635.It then follows that in order for one occurrence of a bug, of similar financial impact to the
Suspense Account bug, to have even a one-in-ten chance of occurring to one branch on one month,
there would need to be 50,000 such distinct bugs - because 50,000 * 16 / 8,000,000 =
800,000/8,000,000 = 1/10. There would have to be a Bug A, Bug B, Bug C, and so on, in a list with
50,000 distinct bugs.
636.Even if there were some ‘super-bug' - with financial impact ten times larger than the suspense
account bug - there would have to be approximately 5,000 such superbugs to give a one in ten chance
of affecting a Claimant's branch accounts in a given month. There is no evidence for even one such
super-bug - let alone for 5,000 different ones.”
[637 and 638 then provided a further table and calculation that I omit]
639.The Claimants have never asserted that there are as many as 50,000 distinct bugs in Horizon,
with each bug on the same scale of financial impact as the Suspense Account bug. Their case rests on
two or three known bugs of this scale, and on the unproven assertion that there may be others.
640.If any Claimant were to assert that, for instance, a deficit of £1000 had occurred in his branch in
a particular month, caused by a bug similar to the Suspense Account bug, the chances of that
RLITO000005
RLITO000005
assertion being correct are extremely small, because Horizon bugs strike so rarely - unless Horizon
contained of the order of 50,000 distinct bugs of that kind (and even then, the chances are only one in
ten).
641.Mr Coyne has examined more than 5,000 KELs, and not found definite evidence for even one bug
with impact similar to the Suspense Account bug - let alone 50,000 of them.
642.The implication of this result - the very small probability of any error in one months' accounts
from a bug in Horizon - is that the accounts for any branch on any month are overwhelmingly likely to
be correct (apart from effects such as delayed TCs, which are corrected after a variable delay).
643.In my experience, no commercial IT system could ever go live with as many as 50,000 serious
bugs - and certainly could not have the good in-service record over 18 years that Horizon has had.”
(emphasis added)
809.1 have reproduced extensive passages of Dr Worden’s explanation of his own methodology in
order to allow his explanation to be available to any reader of this judgment. I consider this a wholly
flawed methodology, and obviously so, and I reject it.
810.There is another aspect to this which needs addressing at this point. It needs some context in
order properly to be understood. Sums recovered from SPMs for discrepancies in branch accounts
would be put into something called the suspense account. The claimants pleaded in paragraph 38 of
the Generic Particulars of Claim the following:
“38. The Defendant operated one or more suspense accounts in which it held unattributed surpluses
including those generated from branch accounts. After a period of 3 years, such unattributed
surpluses were credited to the Defendant's profits and reflected in its profit and loss accounts.
39. The Defendant thereby stood to benefit and/or did benefit from apparent shortfalls wrongly
attributed to the Claimants which did not represent real losses to the Defendant.”
811.This was pleaded to by the Post Office in the Defence as follows:
“73, As to paragraph 38:
(1)Post Office does not have a clear understanding of what the Claimants mean by the term “suspense
account" or by the phrase “unattributed surpluses including those generated from branch accounts".
(2)In relation to its dealing with its clients, Post Office is almost always able to reconcile its figures to
those of its clients. However, as is inevitable given the nature and scale of those dealings, occasionally
Post Office is not able to do so. For example:
(a)A client, such as a bank, makes a payment to Post Office in respect of a particular transaction
undertaken in a branch but the payment exceeds the amount that Post Office considers to be due.
(b)Post Office seeks but does not reach agreement with the client as to the amount due and the client
does not accept repayment of what Post Office considers to be the overpayment. In parallel, Post
Office also seeks to determine whether the overpayment should be credited to a branch or to any
other relevant part of its business.
RLITO000005
RLITO000005
(c)Where the proper beneficiary of a credit cannot be determined, Post Office temporarily records the
overpayment in its accounts. If the overpayment is not resolved within 3 years, the overpayment may
be credited to Post Office's profit and loss account.
(3) The existence of processes for recording and resolving credits of the type described above are an
ordinary business practice.”
812.This was then pleaded to by the claimants in their Reply and Defence to Counterclaim:
“29. The Defendant's professed lack of clear understanding of the term ‘suspense account' at
paragraph 73(1), is noted at paragraph 2.1 above, and its response to the Claimant's RFI is similarly
evasive.
30. The effect of the Defendant's example pleaded at paragraph 73(2), is that the Defendant admits
the possibility that:
30.1.a sum of money may be received which the Defendant believes to be an ‘overpayment’ from a
third party client (e.g. a bank) in respect of a transaction undertaken in a branch: paragraph 73(a);
30.2.the Defendant would seek to determine whether the overpayment should be credited to a branch,
or to any other relevant part of its business: paragraph 73(b);
30.3 where the Defendant does not identify ‘the proper beneficiary’ of the credit, Post Office
‘temporarily’ records the overpayment in its accounts and credits that to its profits, if not resolved
within 3 years.
31. This logic would apply where the Defendant's internal records were wrong due to a Horizon error
or in the case of any failure to identify the relevant branch or Subpostmaster”.
813.1 find that there is no sensible basis for the professed lack of understanding by the Post Office in
its Defence of the term “suspense account”. That term is widely used by witnesses for both sides in
the evidence, and I will give two simple examples showing its use. One of the Post Office’s own
witnesses, Mr Johnson used it in his evidence, in a passage quoted at [279] above. Further, in an
internal document which was prepared after the Callendar Square bug was discovered, it was widely
used. An extract from that document appears at [404] above. Not only is the term used in four
separate places in that document, there is even reference to the Suspense Account Team. The notion
that the Post Office did not understand the term used in the Particulars of Claim is wholly fanciful.
Further, there is an agreed bug (number 3 in the Bug Table) which the experts call the Suspense
Account bug.
814.Putting to one side the fact that the Post Office professed not to know what the term suspense
account meant, notwithstanding its use in internal documents, the Defence makes it clear that
unattributed sums within it, the correct destination for which the Post Office could not work out, were
credited to its own profit and loss account. I accept the logic explained in the Reply in respect of the
effect of this.
815.Dr Worden had wholly failed to consider this issue. The point was put to him, not by reference to
the pleadings, but by reference to another document which he admitted he had read, namely a report
by Second Sight. That report had stated what is basically the same point but in slightly different
terms, namely as follows:
RLITO000005
RLITO000005
“2.11 We note that Post Office's control and reconciliation procedures rely on correct information
being supplied by third party clients. It follows that, if incorrect information is provided by any client
company, this can give rise to a loss being charged to a branch. We also note that, for most of the past
five years, substantial credits have been made to Post Office’s Profit and Loss Account as a result of
unreconciled balances held by Post Office in its Suspense Account.”
816.The question and answer are as follows, having read out the passage:
“Now I'm not asking you about that as evidence of itself, I'm just asking were you aware that that was
a concern that Second Sight had raised about substantial credits being made to Post Office's profit
and loss account as a result of unreconciled balances held by Post Office in its suspense accounts?
A. I read the Second Sight reports and the Post Office's reply.
Q. But did you spot that?
A. I didn't spot that, no.”
817.This point was put in the context of evidence Dr Worden had given, namely the effect of small
bugs or what he called “micro bugs” being unlikely to have an effect.
“Q. Okay. Well, what about -- because elsewhere in your report, I don't want to stray too much at the
moment, but you also mention micro bugs, small bugs, being unlikely
because you would expect to see large sums of money somewhere -- A. Money that disappears
somewhere has to pop up --
Q. It's got to come back somewhere else, it's got to pop up somewhere else.”
818.The above passage was then put to him. Dr Worden’s failure to spot what is, in my judgment, a
point directly relevant to the likelihood of Horizon bugs, errors and defects having a potential impact
upon branch accounts is a notable one. Whether the point is taken from the pleadings, or from the
Second Sight report (which Dr Worden accepted he had read, together with the Post Office’s reply to
that document) does not much matter, in my judgment. I consider that this one omission alone - which
I consider to be a serious omission on his part - is sufficiently fundamental as to undermine his
evidence on this issue on its own.
819.The following points are relevant in respect of Dr Worden’s statistical exercise:
1.Statistics is the branch of mathematics that concerns the collection, presentation and analysis of
data. There was no permission granted by the court for expert statistical evidence in this case. The
permission that was granted was for expert IT evidence.
2.Had either party sought permission to rely upon evidence from an expert statistician, I would not
have granted it. Statistical analysis does not assist in resolution of the Horizon Issues, which required
(in my judgment) detailed technical analysis and consideration by IT experts of the detailed
occurrences of what could, potentially, be bugs, errors and defects. This was so that their expert
opinion could be brought to bear, firstly, upon the important issue of whether there were any, and also
if there were, their extent and the degree to which they potentially could impact branch accounts.
This both experts did in any event, and that evidence - based on contemporaneous records and
evidence about their occurrence - is of far greater assistance than this wholly theoretical argument
about why Horizon bugs cannot have affected the claimants’ branch accounts.
RLITO000005
RLITO000005
3.Dr Worden uses statistics, as he uses mathematics, as part of his professional life. Any particle
physicist would, and does, use mathematics every day. Dr Worden accepted that this was not
advanced statistics, and he was right to do so. It is a combination of simple assumption together with
basic multiplication and division. The assumptions are not, in my judgment, valid.
4.None of the Horizon Issues required the experts to perform a mathematical or statistical analysis of
the likelihood of the claimants’ alleged shortfalls and discrepancies being caused by bugs, errors and
defects in Horizon. Horizon Issue 3 - using as it does the term “extremely unlikely to be the cause of
shortfalls in branches” - does make this exercise by Dr Worden relevant if that was how he chose to
address that issue. That does not however make it either correct, or of any assistance.
5.The accepted correction to the figure of “2 in 10 million” - which he accepted before his cross-
examination started should be “2 in a million” - shows, in my judgment, how very broad brush this
exercise was. As he put it in his oral evidence, “...there were quite a few statements of this "in a
million" nature in my report”.
6.It is plain that Dr Worden’s own evidence accepts that there are two fundamental assumptions that
are central to this exercise. The first is that any bugs in Horizon strike branches randomly (which is
spelt out in his report). The second is that the claimants’ branches are a random sample. This can be
seen, implicitly, from his statement at paragraph 621.1 that “it might be asserted that Claimants’
branches are more or less likely than other branches to be hit by bugs in Horizon, because of some
special property of Claimants” and also at paragraph 632 “if I selected a Claimant's branch and a
month at random....” Neither of these assumptions are valid, and the reasons that they are not valid
are explained in this judgment.
7.His paragraph 636 makes it clear how fundamentally unsound this exercise it. He stated “Even if
there were some ‘super-bug' - with financial impact ten times larger than the suspense account bug -
there would have to be approximately 5,000 such super-bugs to give a one in ten chance of affecting a
Claimant's branch accounts in a given month.” However, the evidence shows, quite specifically, that
even for bugs such as Dalmellington and Callendar Square, SPMs’ branch accounts were specifically
affected, and some by quite significant amounts. This therefore demonstrates that some SPMs as
individuals could, and did, have their branch accounts affected by bugs. If the evidence does not
match a theory (which here it plainly does not) then that must mean that the theory is flawed.
8.0f his four different ways of addressing financial impact, set out in his paragraph 614, the second
assesses the net impact of all bugs referred to in KELs. However, KELs do not invariably recite known
financial impact. In some, known financial impact is recited, but a conclusion in many KELs is that
user error cannot be ruled out and therefore a conclusion is reached that a bug is not responsible. I
consider that Fujitsu would regularly record user error simply as a “catch all” and, effectively, require
someone to rule it out to Fujitsu’s satisfaction, and unless this was done Fujitsu would remain wedded
to that as an explanation. Given the technical investigations were being done by Fujitsu itself, Fujitsu
rarely (if ever) ruled our user error by the SPM. This was wholly in Fujitsu’s interests.
9.Dr Worden’s reasoning is entirely circular. By layering assumption upon assumption (and in places
routinely misdescribing these assumptions as estimates) he felt able to conclude that there would
have to be as many as 50,000 serious bugs for the claimants to be correct (as set out in his paragraph
643) and that “no commercial IT system could ever go live with as many as 50,000 serious bugs - and
certainly could not have the good in-service record over 18 years that Horizon has had”. The whole
rationale of this litigation is to work out, more or less, exactly what the “inservice record” of Horizon
in fact is. I do not consider reliance on an assertion that it has had “a good in-service record over 18
RLITO000005
RLITO000005
years” can be used to underpin an exercise, the whole purpose of which is to consider this very issue.
I reject his exercise that led to his figure of 50,000 bugs - it is wholly flawed. I also reject the
circularity of his reasoning, which lacks logic.
10.Dr Worden also implicitly assumes that all known bugs have been both reported and discovered, as
he uses this (the total number of occurrences of the Suspense Account bug) in his table at paragraph
637. It is his label F and the source is “evidence on the Suspense Account bug”. Mr Coyne estimated
that there may be as many as 40, based on an extrapolation of the number of KELs and PEAKs
inspected.
11.Dr Worden also misidentified the approach of the claimants, or certainly the approach of Mr
Coyne. He stated “Their [ie the claimants’] case rests on two or three known bugs of this scale, and on
the unproven assertion that there may be others.” That is not correct. The approach of Mr Coyne has
been to consider PEAKs and KELs; see what is reported contemporaneously and what the conclusions
were (or appeared to be) within Fujitsu; and consider what that contemporaneous account shows. Mr
Coyne then uses that to draw conclusions. A great many of the bugs in the Bug Table use
contemporaneous entries by Fujitsu which records bugs, errors and defects, as part of the
consideration of whether there were such bugs, errors and defects. This is not mere assertion.
12.However and in any event, by the end of the Horizon Issues trial, the Post Office admitted the
existence of far more bugs than the “two or three known bugs” that were referred to by Dr Worden in
the preceding sub-paragraph.
13.Finally, I consider that Dr Worden’s statistical exercise is nothing more than a superficial short cut
which avoids a detailed technical analysis of the history of the Horizon system in the way undertaken
by Mr Coyne. In this case, on these issues, with all the detailed evidence available, it can be shown to
be an unjustified way to resolve any of the Horizon Issues.
820.Dr Worden also assumed that bugs in Horizon strike branches randomly. This too is not borne out
by the evidence. For example, the Dalmellington bug (bug 4 in the Bug Table) appears to have
affected only outreach branches. Relatively few of the Post Office’s branches are outreach branches.
Bug 14 in the Bug Table, which Mr Coyne said was a bug but which Dr Worden described as “appears
to be a system error with impact upon branch accounts” is entitled in the bug table “bureau
discrepancies” and relates to foreign exchange transactions. Dr Worden’s exercise takes no account of
what is, effectively, an admitted bug, and one that has an effect only upon foreign exchange
transactions. If a branch conducted very few, if any, of these, it would not be affected by this bug. The
chances of a branch being impacted by the bureau discrepancies bug depends upon whether it carries
out foreign exchange transactions, and if so how many. That is not a bug striking all branches
randomly. Indeed, rather tellingly for Dr Worden’s exercise, he assumes that any such bug would
strike randomly across the whole branch network and have an equivalent random effect upon the
branch accounts of branches that had performed foreign exchange transactions, and those that had
not. This is plainly invalid.
821.Dr Worden also assumed that the claimants are a random sample of SPMs. That this is one of his
assumptions can be identified in the following way. His analysis of likelihood of any bug impacting any
SPM’s branch accounts would apply equally to any group of SPMs chosen at random from those SPMs
with smaller branches. The only adjustment or account taken of the claimant SPMs’ particular
characteristics is that they are, generally, from smaller branches, and for this Dr Worden applies what
he calls a scaling factor. However, the claimants are not a random sample of SPMs, nor are they even
a random sample of SPMs from smaller branches. As a sample, they
RLITO000005
RLITO000005
have already been filtered or selected in that these particular SPMs already complain of bugs, defects
and errors in Horizon having affected their branch accounts. This means that they are not a random
sample. The way this would be expressed in statistical terms is that the claimant SPMs do not
accurately represent the population of SPMs as a whole (or even the population of SPMs who had
smaller branches). The claimants are essentially self-selected, from those who believe they have
experienced shortfalls and discrepancies in their accounts from the impact of bugs, errors and
defects, and who have been prepared to join the litigation. The group has a bias, in statistical terms.
They plainly cannot be treated, in statistical terms, as though they are a random group of 587 SPMs.
822.The adjustment made by him (or scaling factor, which was refined from that in his 1st report) to
take account of the particular characteristics of the claimants does not validate what is, in my
judgment, an entirely invalid base assumption. He attempts to deal with this by considering whether
there is any “special property” of the claimants’ branches that makes them more susceptible to bugs,
and concluded (at paragraph 622 of his report) that there is not. This is to approach the matter
somewhat backwards, in my judgment. The experts are agreed that there were 3 million sets of
branch accounts in the life of Horizon. The number of sets of branch accounts of which complaint is
made in this litigation is a small (even perhaps tiny) proportion of that 3 million. However, that basic
arithmetical fact does not mean that bugs, errors and defects cannot on the balance of probabilities
be responsible for shortfalls and discrepancies.
823.The exercise also wholly ignores what branch accounts are, how they are supposed to be created
from the full list of transactions carried out in that particular branch in that particular period, and
how precise they are required to be. The analysis of the “lasting and transient impact” approach taken
by Dr Worden has already identified that this ignores the effect on branch accounts of the impact of a
bug if no transaction correction has been issued before the end of the next trading period.
824.There is also another fundamental problem with Dr Worden’s exercise, which is that it entirely
ignores the specific purpose of Horizon, at least so far as the contractual relationship between the
Post Office and SPMs is concerned. The Horizon system is, whether under Legacy Horizon or Horizon
Online, the method by which the Branch Trading Statement or BTS is produced for an individual
SPM’s branch Post Office. The BTS is the accounting mechanism under either the SPMC or NTC (the
two different types of contracts used by the Post Office for SPMs during the period from 2000
onwards, explained in detail in Judgment (No.3) on the Common Issues). It was the common intention
of the parties that the Horizon system would be used for the creation and recording of accurate
information regarding all transactions performed in the branch which would be used for the
production of the Branch Trading Statement for that branch. This means that at the end of each
branch trading period, the BTS will (or should) provide a statement of the branch activity, which
includes whether the accounts are in balance (which is where the cash and stock in the branch
matches the Horizon system’s records of what should be there) or whether there is a discrepancy. If
there is a discrepancy and it is a shortfall, the contractual terms of the SPMC and the NTC require the
SPM to “make good” the shortfall, which means pay the money to the Post Office. The other
alternative for an SPM is to “settle centrally”, which means seek time to pay. If the Post Office agrees,
the shortfall can be paid over 12 months by deduction from the SPM’s remuneration. This purpose of
the Horizon
system is entirely ignored in Dr Worden’s statistical exercise across the whole branch network.
825.Consider a hypothetical bug, bug X. Also consider that bug X impacts upon branch accounts in a
single branch upon a single occasion leading to a shortfall in the branch for that branch trading
RLITO000005
RLITO000005
period. That shortfall would be treated as a debt by the Post Office and the SPM either has to pay,
seek time to pay, or may even end up in a dispute with the Post Office and/or be suspended as a SPM
and not pay. Analysis and resolution of the correct and true situation of the branch accounts between
the Post Office and the SPM for the trading period in question does not depend upon whether, in all
the other millions of branch accounts, there was no such incidence of bug X. The correct analytical
approach in my judgment is to consider the branch activity for that branch for that period; consider
the evidence both for and against (1) the existence of bug X and (2) the likely cause of the
discrepancy, bearing in mind both the burden and standard of proof; make findings on the cause of the
discrepancy; and then apply those findings. Expert IT evidence of most assistance in that exercise
would be whether or not bug X exists or existed, and what were its effects. It is of no assistance to
have an exercise that in effect says the statistical likelihood of any bug having an impact upon the
branch accounts of that branch in that period is very low.
826.The section 8 analysis is, in my judgment, so riddled with plainly insupportable assumptions as to
make it of no evidential value. It is the mathematical or arithmetic equivalent of stating that, given
there are 3 million sets of branch accounts, and given there are so many sets of branch accounts of
which no complaint is made, the Horizon system is mostly right, most of the time. It is a little more
sophisticated than that, but not by very much.
827.The way that this can be further tested against the evidence is as follows. Mr Godeseth accepted
that the Dalmellington bug had probably been present in the system from 2010 until it was discovered
in 2015. Dr Worden’s analysis would lead to the conclusion that it was overwhelmingly unlikely that it
had affected branch accounts during that period. But all the evidence is that this bug did affect
branch accounts. It is correct that the Post Office relies upon the correction of known and identified
discrepancies by means of the later issuing of TCs - a human measure which in my judgment is
outside the definition of Horizon - but even putting that to one side, during this 5 year period branch
accounts were actually affected. The existence of the bug was not known about for 5 years before it
was even discovered. This round of the group litigation is to make findings on the Horizon system and
whether there are any, or many, or no, other such bugs, errors and defects in the Horizon system
similar to this one, or with similar detrimental effects. The conclusion of Dr Worden’s section 8
exercise is that it was overwhelmingly unlikely that any of the claimants’ branch accounts were
affected by any Horizon bugs. This must include a conclusion that the Dalmellington bug cannot have
affected any of the claimants’ accounts. This means that he must assume that Fujitsu were 100%
successful in tracking down all incidents of the Dalmellington bug, and that any claimant SPMs who
report symptoms the same as the Dalmellington bug and who had an outreach branch are either
wrong, or making it up. These assumptions are ones that cannot sensibly be made, in my judgment.
828.The whole section 8 exercise effectively is entirely theoretical, and is made in a vacuum that must
ignore real world evidence to the contrary from one of the Post Office’s own witnesses, Mr Godeseth,
as set out at [445] to [450] above. Even if I am wrong about all of the above, and even if a statistical
analysis of this nature is valid, then in my judgment this would not assist the Post Office in its case on
the Horizon Issues in any event. There are agreed to have been over 3 million sets of branch accounts
over the life of Horizon, and approximately 6 million transactions every day across the whole Post
Office branch network. There are fewer than 600 individual claimants, with the total value of
accounting discrepancies over a 17 year period, amounting to only £18 million. As a proportion of the
value of the Post Office network business over that period, and as a proportion of the total number of
transactions, the number of transactions which underpin the claimants’ case is exceedingly small in
percentage terms.
RLITO000005
RLITO000005
829.Dr Worden’s probability calculation regarding two bugs, Bug A and Bug B, striking independently,
was expressed as follows:
“632. If I selected a Claimant's branch and a month at random, then the chances of Bug A occurring at
that branch in that month are only 16 in 8 million, or 2 in a million - an extremely small probability.
633. Diff f . f - ilities of z
additive. If there were a second bug similar to the hypothetical bug above - call it Bug B - then the
chances of either Bug A or Bug B happening to one branch on one month are twice the previous figure
- 32 parts in 8 million.” (emphasis added)
830.He described the probability of these two unconnected events both occurring, each with a
probability of 16 in 8 million (or 2:106), as being additive and he added 16 in 8 million, to 16 in 8
million, to arrive at their sum, 32 in 8 million. In terms of basic probability theory, the chances of two
independent events occurring is usually seen as the product of the two probabilities, not their
addition. This was recognised, as a SPM would not need both bugs to strike in order to experience an
impact on branch accounts, only one would be sufficient. The principle of additivity is a fundamental
principle that if the event whose probability is sought can be represented as the union of several other
events that have no outcomes in common, then the probability of the union is the sum of the
probabilities of the individual events making up the union. However, regardless of this, the point was
neither put nor explored in any detail, and the outcome in any event depends upon how the events,
and the union, are defined. The situation is further complicated by the fact that depending upon the
nature of how “the bug” was defined, to impact upon branch accounts, this could be either Bug A or
Bug B (in Dr Worden’s example) independently, or in conjunction. The PEAKs show that sometimes
branch accounts would be impacted by a combination of rare circumstances. Such bugs may, but
would not necessarily, have to occur at the same time. If independently able to impact branch
accounts, they would simply have to occur at any time during the 4 week branch trading period. I
refer to this matter for three reasons. Firstly, to demonstrate that probability theory only takes one so
far in this case, and that is not very far; secondly, to explain that depending upon how one defines
“event” and the intersection (or joint probability) of both or either occurrence, one can arrive at
different results; and thirdly, to demonstrate that there were numerous flaws and/or difficulties with
Dr Worden’s approach, not all of which were put, but which in my judgment undermine the validity of
his exercise. This point about the probability calculation of two bugs striking a branch is not, however,
not the main reason (or indeed, any reason) why I have rejected his analysis. The probability
calculation for two bugs occurring in one month is not a central element of the statistical exercise he
performed; only one bug would be needed in any one branch trading period for it to affect those
monthly branch accounts. I have explained this in order to illustrate that it depends upon how one
defines the impact of a bug, and chooses to assess its probability, that one can end up with an answer
that rather suits the way the question is initially posed.
831.Dr Worden accepted so far as his evidence generally was concerned, that he had assumed that
the Fujitsu evidence was true, and in particular that of Mr Jenkins (whom he had met, although Mr
Jenkins was not listed as a source of information in his 1st expert report, and from whom no witness
statement was available) and that “generally, if findings -- if things happen in oral evidence which go
against witness statements then I would need to come back and say what's the impact.”
832.This therefore means that his evidence is of limited assistance in any event. I have found that the
evidence of Mr Godeseth was somewhat different, at the end of his cross-examination, than how it had
been presented in his three witness statements. Litigation cannot go on endlessly. For the Horizon
RLITO000005
RLITO000005
Issues trial, evidence of fact and expert evidence were both ordered, and findings can only be made by
the court in its judgments that resolve the disputes of fact. Disputes of fact in the evidence for the
Horizon Issues will be resolved in the Horizon Issues judgment, namely this one. This is entirely
conventional. If an expert chooses to prepare his or her evidence on the basis that one side’s evidence
is true - which Dr Worden did in his reports, in respect of the party instructing him, namely the Post
Office - they are not entitled to serve further expert evidence and wholly amend their opinions if that
evidence is not accepted by the court. Dr Worden must have known this. There was going to be no
opportunity for him “to come back and say what’s the impact”.
833.This was put to him in his cross-examination.
“Q. The content of this report in this respect, we will come to other examples later, focuses on what
the consequences would be if the defendant's factual evidence is accepted?
A. Generally, if findings -- if things happen in oral evidence which go against witness statements then I
would need to come back and say what's the impact.
Q. Yes. You understand, don't you, that the moment at which everyone finds out whether factual
evidence is accepted or not is when the judgment is handed down? A. Absolutely, yes.
Q. So you are proposing to revise your opinions after that, are you?
A. No. That's a good point. What I was saying there is if it becomes evident, all I can do is make
assumptions. The court will find findings and all I can do is make assumptions and drive them through
my opinions and try and assist the court that way. So all of this i mptions I have mad n
the evidence I have seen, and the court may find differently.
Q. Can I pause there. You were assuming there that Mr Godeseth's version was true?
A. I was assuming mainly Mr Jenkins' written analysis which Mr Godeseth's evidence confirmed.”
(emphasis added)
834.The “evidence I have seen” to which Dr Worden referred was the Post Office’s evidence, and what
Mr Jenkins had said (which was not evidence in any event). He did not take any account of the
claimants’ evidence. This was notwithstanding that in his expert’s declaration he had expressly stated
“I have not assumed that any particular version of events is true.....” when, indeed, he plainly had. It
was also based on Mr Godeseth’s written evidence, not on his oral evidence, which was markedly
different.
835.He also said further on this same subject:
“A. Well, my analysis of the claimants’ evidence is mainly contained in my supplemental report, and I
explained there that individual claimants' evidence, particularly individual subpostmaster evidence, I
did not feel able to make strong use of that and I gave the reasons in my supplemental report. So my
opinions have little dependence on that. And the core of my opinions, the numerical estimates I make,
those estimates have been designed -- or the process and the method has been designed so that if - so
that to make my assumptions evident where those assumptions come in, and if the court finds
something different from my assumptions the court can go to the spreadsheets and re-do the method
for itself.”
(emphasis added)
RLITO000005
RLITO000005
836.This can therefore clearly be seen as his accepting the Post Office/Fujitsu evidence as true,
particularly that of Mr Jenkins (not even a witness, and in respect of whom no witness statement was
served); and also failing to base his opinions upon, or take account of, any of the individual evidence
of claimant SPMs served by the claimants.
837.Dr Worden is also under a misapprehension if he considers that the court’s function is to “re-do
the method for itself” if his assumptions were not made out in the factual evidence. Firstly, he could
have provided alternative expert evidence and opinion, were the written evidence of Mr Godeseth and
the information coming from Mr Jenkins to be rejected. Secondly, he should have taken the claimants’
evidence into account, because there is no basis for an assumption that it was not relevant to the
Horizon Issues which his expert evidence was the subject of. It is entirely normal for experts to
provide alternatives of their opinion, depending upon which party’s evidence is accepted. Thirdly, the
“method”, such as it is, demonstrated in the spreadsheets simply cannot be “re-done” to take account
of findings that the assumptions were incorrect. The invalid and unjustified assumptions lead, in my
judgment, to the inescapable conclusion that the whole methodology is flawed. No amount of “re-
doing” of the calculations by the court can remedy that.
838.Dr Worden also accepted that although he had missed some days of the trial, he was “pretty sure”
that he had heard all of the Post Office’s evidence; he was then asked the following:
“Q. Have you had any changes of heart about anything in your report having heard it?
A. Changes of heart?
Q. Anything you want to change about what you said?
A. Well, had there been some fundamental change I would have felt obliged to communicate it to the
court.
Q. Of course, because you take your duty very seriously?
A. Yes, I do. I can't think of any major change of heart, but there may well be things that you bring me
to and I say there's an adjustment here. But to come back to the point about precision, I am conscious
of how precise my numerical results have to be in order to be of assistance to the court.”
839.This effectively must mean that Dr Worden did not consider that any of the changes, in the Fujitsu
evidence in particular from the cross-examination, would impact upon his expert opinion. This is
notwithstanding that he had assumed that the Post Office’s evidence would all be accepted, and
anyone present for Mr Godeseth’s crossexamination can only have concluded that the factual position
presented by Fujitsu in the written witness statements was very far from an accurate one. Indeed, the
extent to which the cross-examination did not have any effect - either appreciable or otherwise - on
Dr Worden’s expert opinions shows how very theoretical the whole exercise was.
840.Finally on this point, due to the disruption caused by the recusal application, and the unavoidable
re-arrangement of the trial timetable, the experts came to give their evidence in June 2019, and all of
the factual witnesses who were cross-examined (with the sole exception of Mr Parker) had completed
their evidence by 21 March 2019, many weeks earlier. Even Mr Parker had completed his evidence by
11 April 2019. Dr Worden was called on 11 June 2019, two months after this. It is unusual for there to
be such an interval in a trial, but circumstances in this case led to extra time being available. There
was more than ample time for Dr Worden to have identified how, if at all, the Post Office’s evidence of
RLITO000005
RLITO000005
fact that emerged earlier in the trial (in particular from Mr Godeseth, but also from Mr Parker too)
impacted upon his opinions. It plainly did not.
841.1 will provide only one example of Dr Worden’s unsatisfactory approach to the evidence, although
there were many. In terms of whether transactions could be inserted at the counter in Legacy
Horizon, what is called “remote access”, Dr Worden had in his first report concluded that Mr Roll (a
witness of fact) was factually wrong. He expressly stated that he, Dr Worden, had “established” that
Mr Roll was wrong. He accepted that use of the word was wrong but what he had done in his 2nd
report was equally telling, in my judgment. This was after those witnesses who were to be called for
the Post Office, still employed at Fujtisu, had corrected (or partially corrected) their erroneous first
witness statements, and accepted that remote access without the knowledge of the SPM was possible.
He said (at paragraph 83) that he required:
“further factual information before I can comment on this evidence. Which ‘specific person’? Under
what circumstances? How frequently? Until I have that information, it remains possible in my view
that any transaction which ‘would appear to the subpostmaster as though it had been carried out
through the counter in branch' might only be a transaction that he had given his consent for, as the
‘specific person’ - and which had in effect been made on his behalf.”
842.Dr Worden also then stated:
“84, Therefore, Mr Roll's new evidence does not cause me to alter the opinion expressed at paragraph
1119 of my first report, when commenting on Mr Roll's first witness statement, that he could not alter
branch accounts without the Subpostmaster knowing.”
843.This approach by Dr Worden to the evidence for the claimants on the one hand (Mr Roll) and the
Post Office on the other (Mr Parker and Mr Godeseth in their statements) was explored in cross-
examination:
“Q. Yes, the one that we agreed was incorrect in approach and conclusion.
A. We agree now, but at the time I wrote the second report, on the basis of Mr Roll's evidence and Mr
Parker's evidence, I didn't see reason to change that opinion.
Q. I would suggest to you that you were inexplicably reticent to accept something that was contrary to
Post Office's interests.
A. No, I was reticent. Not inexplicable, L was reticent because [ had not seen sufficient evidence to
. . owledge of the subpostmaster.
Q. But Dr Worden, would you accept that the approach you have taken there contrasts very, very
strikingly in how you approach Mr Roll's evidence with the approach you took at 1119 in your first
report when you accepted, effectively, Mr Godeseth's evidence?
A. Ihave accepted that my approach at 1119, that use of the word "established" was wrong and my
approach was wrong, and we have established that -- you know we have done that before the interval
Q. My question is: do you accept the contrast is very striking?
A. I think the court will have to -- I accept that my attempt to make my position clear that I'm not
trying to find findings of fact, 'm not trying to find one witness or the other. I accept that on this
occasion I fell short of that.
RLITO000005
RLITO000005
Q. Do you mean only in your first report or also in your second report?
A, Not in the second report. I believe the second report was -- you know, I believe that the evidence
from Mr Parker and Mr Roll, and if we look again at Mr Roll's paragraph 20, I believe that the
question that I asked in paragraph 83 that Lwant further evidence, I believe that was a valid
approach.
Q. When you say in paragraph 84 -- because we have dealt with what you accept about your first
report at paragraph 1119 several times, haven't we? A. Yes.
Q. So in your second report, you say, notwithstanding what you have now accepted about 1119, in
your second report that you were right not to change that view. A. Lhadn't seen sufficient evidence to
change that view.
Q. But do you say you were right not to change that view when you revisited the entire piece in your
second -
A. At the time I believe it was right not to change that view because I hadn't seen evidence that
A 1 : f That'
where Lam.
Q. Dr Worden, do you accept that that betrays a complete failure to appreciate the need to consider
the situation both on the basis of whether the claimants' evidence is right as well as on the basis that
the defendant's evidence is right?
A. No, I don't accept that.”
(emphasis added)
844.Dr Worden had a further witness statement from Mr Roll saying effectively that Fujitsu could
perform, and had performed, remote access. Dr Worden wanted “further evidence” and also referred
to needing “sufficient evidence”. He said he “hadn’t seen evidence that convinced” him. He accepted
he was reticent to accept the claimants’ evidence. All of this plainly demonstrates that he was taking a
partisan view on evidence of fact. I find that this approach to disputed evidence of fact is simply not
the correct approach for any expert to adopt. By the time of Dr Worden’s 2nd report he had a
responsive witness statement from Mr Roll saying that what Dr Worden had “established” could not
be done (a conclusion that was in any event directly contrary to Mr Roll’s earlier witness statement),
very specifically, could be done. Even then Dr Worden would not accept it. As he stated in his cross-
examination, “I hadn’t seen evidence that convinced me” that this could be done. The evidence was,
very plainly, the witness statement saying that it could be done, and he chose to disregard it. This is
just a more sophisticated way of stating the witness statement was not true. He described this as his
being “balanced and fair”. I disagree with that description: it was not balanced, and it was not fair. He
also said that there were many, many occasions when he had “tried not only to be fair to the claimants
but to be biased in my numerical estimates in favour of the claimants”. I disagree with that too.
Firstly, it is no answer to a criticism that an expert has taken a partisan view of evidence of fact and
ignored one side’s factual account, to say that this has been effectively cancelled out by numerical
assumptions elsewhere in different parts of the report. Secondly, I do not accept that Dr Worden was
performing numerical calculations assuming in the claimants’ favour in any event, when one assesses
the Section 8 exercise as a whole. The assumptions he made were very much not in the claimants’
favour at all. Minor adjustments to numerical figures in an exercise based on assumptions specifically
unfavourable to one group does not either make the result “biased” in favour of that group, or cancel
RLITO000005
RLITO000005
out the unfavourable assumptions. Further, the court requires an independent approach from an
expert at all stages of their analysis, not a raft of slanted analyses, either one way and/or the other,
with the hope (or here, an assertion) that they will eventually cancel each other out. For the avoidance
of doubt, here they do not.
845.Another area of difference between Dr Worden and Mr Coyne is the issue of what Dr Worden
called “countermeasures”. Mr Coyne described these as “self-defined countermeasures” and stated
that many of them - Dr Worden had identified 18 different ones - were basic elements of practical
system design, and that such design aspirations did not, in themselves, demonstrate that Horizon was
particularly robust.
846.The 18 different ones are as follows, together with the acronyms adopted by Dr Worden:
1.Reliable and redundant hardware - RHW;
2.Robust data communication and replication - ROC;
3.Transactional Integrity and database recovery - TIN;
4.Defensive programming - DEP;
5.Generic, data driven software - DDS;
6.Secure kernel hardware and software - SEK;
7.Redundant data storage and computing, with cross-checks - RDS;
8.Double entry accounting - DEA;
9.Early detection of user errors - DUE;
10.Later correction of user errors - UEC;
11.Manual workarounds - WOR;
12.Testing good practice - TGP;
13.Manual Inspection of Data - MID;
14.Bug finding and correction - BFC;
15.Large scale IT architecture - ARC;
16.Quality and change control - QCC;
17.Managing non-functional requirements - NFR;
18.Security - SEC.
847.These acronyms are, for the most part, not used in the IT industry, and were adopted by Dr
Worden in this case. Some of them are, such as NFR and SEC. I use them in this judgment for
consistency, but their use should not be taken to be a suggestion or acceptance that the acronyms are
industry norms; the majority are not. The actual term countermeasures suggests that these consist of
measures present in the Horizon system which have the effect of countering something else (bugs,
errors or defects, issues that would have an adverse impact on robustness) and thereby improving the
robustness of the Horizon system as defined. The Horizon system had an agreed definition for the
RLITO000005
RLITO000005
purpose of the list of Horizon Issues which was as follows: “the Horizon System” shall for the
purposes of this list of issues mean the Horizon computer system hardware and software,
communications equipment in branch and central data centres where records of transactions made in
branch were processed”.
848.However, it can be seen simply from the descriptions above, which are Dr Worden’s own
descriptions, that not all of the 18 are even part of the Horizon system. Some are simply manual
checks. Some genuinely are countermeasures, but Dr Worden groups both types together.
849.This can be seen from the following. Transactional Integrity and database recovery or TIN is
undoubtedly a countermeasure within the Horizon system. The explanation given by Dr Worden in his
summary table for this is “database management systems provide many facilities so that numerous
kinds of failure cannot leave the data in an inconsistent, unusable state, or lose any data that have
been previously stored.” This means if there is a failure, which may happen in a variety of different
ways, the data should not be left in an inconsistent or unstable state, or be lost. The database
management system accomplishes this in a variety of ways. Indeed, Dr Worden himself, with his
extensive experience of database management, has particular familiarity with this. He designed and
developed one of the first relational database management systems; it is an essential requirement in
such a system that data should not be capable of being left in an inconsistent or unstable state. The
reliability of any database relies upon reliability of the data within that database.
850.Transactional integrity is a fundamental facility built into all database management software and
this is made clear in, amongst others, published material such as An Introduction to Database
Systems 8th edition by CJ Pearson published in 2003 which was one of Dr Worden’s appendices. As
stated at 15.1 of that edition:
“Recovery in a database system means, primarily, recovering the database itself: that is, restoring the
database to a correct state after some failure has rendered the current state incorrect, or at least
suspect. (We will elaborate on what we mean by "a correct state of the database" in the next section.)
And the underlying principle on which such recovery is based is quite simple, and can be summed up
in one word: redundancy. (Redundancy, that is, at the physical level; we do not want such redundancy
to show through to the logical level, naturally, for reasons discussed in detail elsewhere in this book.)
In other words, the way to make sure the database is recoverable is to make sure that any piece of
information it contains can be reconstructed from some other information stored, redundantly,
somewhere else in the system.
Before going any further, we should make it clear that the ideas of recovery (indeed, the ideas of
transaction processing in general) are largely independent of whether the underlying system is
relational or otherwise—though it is true that most of the theoretical work on transaction processing
has historically been done, and continues to be done, in a relational context specifically.”
851.This is a countermeasure. Transactional integrity is intended to give protection against a wide
variety of conditions, such as hardware errors or communication failures, software errors (which can
lead to failures and cancellations) and operations being cancelled when they have not been
completed. Such cancellations can be made by what Dr Worden calls the operator (the SPM or their
assistant) but cancellations can - and the factual evidence shows this occurred - be caused by
outages, power cuts and other incidents. A lay person might say that a system has “crashed", or one of
the terms used in the factual evidence in this trial was “system outage”.
RLITO000005
RLITO000005
852.Other measures are not strictly speaking countermeasures, they are sensible design intentions
(Mr Coyne used the term design aspirations). An example of this is defensive programming or DEP.
Since programming is at the heart of any software - indeed, programming creates the software itself -
the way that programs are originally written (what could be called the programming of the system)
constitutes the very system itself. DEP was described by Dr Worden in the following way in his table
summary: “software is divided into small self-contained modules, which do not assume that other
modules are correct, but defend themselves by checking their inputs and raising alerts early.” A better
description was given elsewhere in his report where he stated that DEP was “where small parts of a
program are written to assume that other parts of the program may be in error and are written to
always check their inputs for the presence of errors.” In other words, part of a program does not start
from the point of assuming that all inputs are without error; they check for the presence of errors first
(in a variety of ways) before moving to other aspects of the program.
853.This could be described as a countermeasure, but a more accurate description of it is a design
feature that ensures subsequent steps are taken without repeating or carrying through errors. It is a
function of design.
854.However, Dr Worden included within his 18 separate countermeasures matters that are not, in my
judgment, countermeasures at all and include remedies. For instance, early detection of user errors,
later correction of user errors, manual workarounds,
manual inspection of data and bug finding and correction are not all countermeasures built into the
system. Just to use that latter item as an example, the many hundreds of PEAKs and associated KELs
show that 3rd level of support in the SSC at Fujitsu would investigate certain instances when SPMs
reported certain matters. Often this would take many months. Depending upon the outcome of the
investigations, matters would sometimes be reported onwards to Development for a software fix. The
investigation by the personnel at 3rd level support (of whom, almost invariably, there would be more
than one) and also the software writer in the Development section, were done by personnel who are
plainly not part of the Horizon system as designed. They are human beings who are undertaking
certain tasks. In fact, a software fix designed to prevent a problem re-occurring would change the
Horizon system when that fix was introduced in live, a term meaning added to the software within the
system. It is, in my judgment, illogical to treat such a process in the same way as either defensive
programming (DEP) or transactional integrity (TIN).
855.This approach is at its most extreme in Dr Worden’s view of transaction corrections, which he
considered to be a Horizon countermeasure. The issuing of transaction corrections are plainly a
remedy, or something done by the Post Office outside of Horizon to correct issues that arise because
of Horizon. As Dr Worden accepted, this is not even done in a consistent way, depending upon the
client in question. He stated:
“There are many differences of detail in how reconciliation is carried out for different client
organisations, or where it is carried out; sometimes the client organisation does it from a file sent to it
by Post Office, and sometimes Post Office does it. However it is done, and wherever it is done, the
result of reconciliation is always in principle the same.”
856,.The way that Dr Worden explained the process was as follows:
“285. When reconciliation finds a transaction for which Post Office's record and the client record do
not match, it is passed to a department in Post Office accounts which handles reconciliation
discrepancies. Each such discrepancy is, until it has been dealt with, an error in the accounts - and so
RLITO000005
RLITO000005
it must be dealt with. The task of this department is to determine how each discrepancy arose, which
therefore determines how it needs to be handled.
286. Wi ; in Post Offi ibility £ , .
with the branch, a request for a TC is issued and is passed from POL FS to the BRDB in Horizon
Online. At this point, there is no impact on branch accounts. The request is passed on from the BRDB
to the branch Horizon system, so that it will show on the Subpostmaster's screen when he starts the
Horizon system the next morning. At this point, the Subpostmaster may either accept the correction
or may question it and ask for further investigation.”
(emphasis added)
857.This description makes it clear that there is a substantial link, or series of links, in that chain that
are entirely outside of Horizon. The discrepancy, which Dr Worden correctly identifies as “an error in
the accounts”, is passed to a Post Office department. That department has to consider it, and
determine how it arose. Then that department has to decide responsibility, potentially issue a request
for a TC, and that
is then passed to the BRDB (which is part of Horizon) and the request goes to the branch. A
substantial element of the whole process does not involve Horizon at all. To treat the issuing of TCs as
a countermeasure in the way that Dr Worden does is not logical. Even the Post Office’s own witnesses
of fact did not deal with the process of issuing transaction corrections, because the Post Office’s
solicitor correctly took the view that transaction corrections were not part of the Horizon Issues and
were outside the scope of the Horizon Issues trial.
858.Further, the scope of the experts’ agreements in this respect under Horizon Issue 5 in the 3rd
Joint Statement is relevant. That issue was “how, if at all, does the Horizon system itself compare
transaction data recorded by Horizon against transaction data from sources outside of
Horizon?” (emphasis added). Agreed entries include the following:
1. Under sub-topic Reconciliation:
“5.1 Reconciliation between transactions recorded on Horizon and transactions recorded by Post
Office's clients is largely automated.
Detected discrepancies were subject to manual corrective fixes and/or the issue of Transaction
Corrections/Error Notices to the Subpostmasters.” and:
"5.3 The adequacy of Post Office back office processes to prevent discrepancies in branch accounts
can be measured by the quality of the TC process. This quality includes:
+The processes of consideration of available data
+The level of errors observed in the process
+The level of complaints or disputes raised following a TC
+The level of upheld complaints following a TC * The level of financial impact of erroneous TCs.”
3. Under sub-topic Third Party Data:
“5.4 Errors in third-party data have led to discrepancies in branch accounts, through erroneous TCs
being issued on Subpostmasters.
RLITO000005
RLITO000005
5.5a PO does not control the level of errors made by its third-party client organisations (which may
lead to errors in TCs), or the delays in their processes (which may lead to delays in TCs).
5.5b PO can and should ensure, by careful investigation of disputed TCs, that only a small proportion
of errors by PO clients lead to losses for Subpostmasters, provided that the Subpostmasters are in
good control of their branches and have the required information available.”
859.This shows that not only do the experts treat TCs as being outside Horizon in the Joint Statement,
and also agree that the SPMs must have available the required information, but also agree that the
level of upheld complaints in TCs (and the value of erroneously issued ones) is an ingredient required
in order to measure the quality of the process. I find that the level of upheld complaints in TCs, and
the value of erroneously issued TCs, is necessary in order to assess the quality of the Post Office back
office processes to deal with discrepancies. That data was simply not kept, according to the Post
Office witnesses. Further, Mr Coyne’s analysis showed that 77% of disputed Santander TCS were
upheld. Entry 5.6 in this Joint Statement, not agreed by Mr Coyne, was an entry by Dr Worden which
stated:
“The figure quoted by Mr Coyne (77% of disputed Santander TCs upheld) illustrates that this process
worked well.”
860.This means that Dr Worden’s analysis of “working well” was 77% of TCs issued in respect of
Santander transactions which were challenged by SPMs was upheld. It is obvious that if 77% upheld,
this means that 23% of TCs were not upheld, in other words were TCs that should not have been
issued. I do not consider that a ratio of approximately 1 in 4 TCs being incorrectly issued can be
properly described as the process “working well”, particularly as each TC would have a direct impact
upon branch accounts. In my judgment, for the process of issuing TCs to be “working well”, the level
of accuracy required would be far higher than 77%, and in excess of 90% (if not in the high 90s as a
percentage) at the very least. Nor do I consider that to be unrealistically or unachievably high, given
TCs impact upon branch accounts, the accuracy of which is required to be very high, and expected by
the Post Office to be such that shortfalls in branch accounts have to be paid by the SPMs.
861.As put to Dr Worden in his cross-examination, if his approach to interventions by human beings
were to be extended to its logical conclusion, the group litigation itself would be categorised as a
Horizon countermeasure. In my judgment, that would be the logical consequence of his approach.
Plainly this litigation cannot properly be seen as a countermeasure. His approach to countermeasures
is not a sensible one; he mixed up both genuine countermeasures such as TIN; design functions such
as DEP; and manual corrections as well, and treated them all in the same way. He was wrong to do so.
862.It was also somewhat unhelpful for some of his countermeasures, such as later correction of user
errors or UEC, to be so widely defined as to include both manual checks - including by the SPM
themselves - and the Post Office and Fujitsu’s both automated and manual processes. Another
criticism of Dr Worden’s approach to countermeasures relates to one which Dr Worden described in
his paragraph 463 as being “one of the most important countermeasures in Horizon”, and that is
manual inspection of data. I do not see how this can be a countermeasure in Horizon as defined at all.
Dr Worden’s summary states that “Any large business IT system is used by many people, who view its
outputs and check them against each other for consistency, and against their own knowledge of the
business. Subpostmasters, watching their branch accounts, were a key component of this.” (emphasis
added)
RLITO000005
RLITO000005
863.The inherent illogicality in this is, in my judgment, obvious. SPMs, viewing their own branch
accounts, which are created or are part of the Horizon system, cannot in my judgment be included as
a countermeasure in the Horizon system (even if one ignores entirely the definition of the Horizon
system in the Horizon Issues). The fact that a SPM can or may notice that (say) their accounts are (as
the SPM sees it) wrong, or that there is discrepancy of some thousands of pounds (or more, or less) in
relation to a transaction that they may not recognise, cannot sensibly be described as a
countermeasure in Horizon in the way that Dr Worden does. As Mr Coyne explained, this “assumes
that the person scrutinising the data is able to identify the correct source of data to be relied upon, in
order to rule out what may or may not be erroneous in the first instance. Additionally, its utility as a
countermeasure is heavily reliant upon the person scrutinising the data already knowing there is an
issue that requires data inspection.” It also obviously relies upon a human element, which necessarily
depends upon the skills or knowledge of the person. When this is put with the lack of notification to
SPMs generally by Fujitsu and/or the Post Office of existing bugs (and what the effect of those bugs
was), which was a feature of the evidence, it can be seen that there is no basis for including this in
any list of such measures.
864.The effect of some, but not all, of the bugs might be susceptible to being noticed in this way, for
example if there were a “doubled up” transaction in the same financial figure at the same time - the
SPM might remember that a customer performed a single transaction in the way that Mr Patny did
with MoneyGram as set out at [135] above. But that would depend both upon the type of bug, the type
of transaction, the memory of the SPM and also crucially the amount. Another bug, Dalmellington,
had some noticeable effects as the evidence in this case shows. Other impacts of other bugs would not
be noticed in the same way. This shows the random nature of what Dr Worden describes as a
countermeasure. In my judgment, Dr Worden was quite wrong to include this as a countermeasure. I
find that something that is a “manual corrective fix” cannot sensibly be described as a Horizon
countermeasure.
865.One of Dr Worden’s other countermeasures was “testing good practice”, what he termed TGP. The
purpose of testing any system is not to prove that it is correct; rather it is to prove (or attempt to
prove) that it is not, a point explained by Dr Worden. Examples of this are regression testing, user
testing and testing of edge cases. Dr Worden relied upon what Mr Membury in particular had given
evidence about in his statement. I have summarised Mr Membury’s evidence at [500] to [507] above,
including his serious omissions regarding E&Y and their concerns. Dr Worden included what was
essentially a caveat in his first report, namely at paragraph 315 where he said in his section dealing
with development and testing of Horizon, specifically in relation to Mr Membury:
“His witness statement describes several different audit processes, designed to ensure by
independent external review that quality is managed effectively in Horizon (Witness Statement of Mr
William Membery, 28 September 2018). If these audits have taken place as described in his witness
statement, and if the results are broadly as he describes them, then that would increase my
confidence that the quality of Horizon has been effectively managed.”
866.He also stated that “a large system such as Horizon can only be robust if sufficient attention has
been paid to issues of quality in its development, testing and support.” I have already identified that
the experts are agreed that Horizon as it is now is relatively robust. The documents also show that the
approach of Fujitsu seems to be
an increase in focus upon such issues since about 2016 onwards. Testing and evaluation is in three
stages (taking this from another of Dr Worden’s appendices):
RLITO000005
RLITO000005
1.Unit (or component) testing takes place once the product is developed or procured and ensures that
the product conforms to its requirements.
2.Link (or integration) testing verifies that the product interworks with other major components. This
level of testing is generally performed on a complete release.
3.Acceptance testing proves to the Post Office's client that the development meets its functional
requirements and to the Post Office that it may be brought into use without impacting on the existing
solution or other applications.
867.This is then followed by what is called release management, which may be either major or
incrementally using interdependent components. This approach to testing is entirely standard in the
IT industry. No sensible programmer - or company involved in a system such as Horizon - would
introduce changes without testing their impact. However, there is a paucity of documentation that
demonstrates the degree to which Fujitsu’s testing regime was successful. Dr Worden, in both his
report and appendices, identifies in general terms what was done and why it was important; there is
no document available to the experts that demonstrates (for example) Fujitsu’s testpass percentages,
or how specific failures were managed and re-tested.
868.1 do not consider testing to be a countermeasure. I consider testing to be an essential part of the
foundation of any system. I would go further and say that comprehensive testing is standard in the IT
industry.
869.Finally, so far as security or SEC is concerned, Dr Worden explained that “any system that could
be easily subverted would not be robust. Horizon is secured mainly through ‘separation of duties’,
user authentication, access control and audit.” The evidence of fact made it clear that the degree of
access available to the whole of 3rd line support at SSC was far wider than intended, and was subject
to criticism by E&Y. Further, the extent of the powerful APPSUP privileges was extraordinarily wide.
J. Conclusion on Expert Evidence
870.1 prefer the expert evidence and approach of Mr Coyne to that of Dr Worden. This is for the
following reasons.
871.Dr Worden took a partisan view of the evidence of fact, preferring that of the Post Office’s
witnesses (by which I mean in this instance predominantly Fujitsu) to that adduced by the claimants.
This was most obvious in respect of his treatment of Mr Roll, but also occurred in respect of the
claimants’ evidence of individual experiences of events caused by Horizon. No independent expert
should do this. One example of this was the important issue of whether Fujitsu could inject
transactions under Legacy Horizon without the SPM in question being aware of this. Dr Worden
stated that he had “established” that these would be visible to SPMs. Mr Roll’s evidence of fact was
directly to the contrary. Dr Worden had dealt with this by stating “So, in my opinion, Mr Roll could not
have made these changes to branch accounts “without the Subpostmaster knowing”. In other words,
in this expert’s opinion, the evidence of a witness of fact (and a very important one at that) was simply
not correct in fact.
872.However, in this case, even the Fujitsu witnesses eventually accepted (but prior to the trial, in
their supplementary witness statements) that this had been possible, and that Mr Roll had been
correct. Dr Worden’ supplementary report dealt with this by stating that there was a lack of
understanding or clarity on this point. He stated that “it seems to me that I require further factual
information before I can comment on this evidence” and also that “Mr Rolls new evidence does not
RLITO000005
RLITO000005
cause me to alter the opinion expressed at paragraph 1119 of my first report, when commenting on
Mr Roll’s first witness statement, that he could not alter branch accounts without the Subpostmaster
knowing”. His first opinion in paragraph 1119 was wrong, and in my judgment obviously wrong, and
this should have been unequivocally corrected both in his 2nd report and also in his cross-
examination. The extract of this in [843] above makes it clear that he did not accept that.
873.The sequence is therefore as follows. Mr Roll gave written evidence he had done something, and
Fujitsu gave written evidence this was not possible. Dr Worden “established” that Mr Roll could not
have done what he said he had done, and plainly preferred the Fujitsu evidence. He therefore adopted
a position of which party’s evidence of fact was to be preferred. More factual evidence was then
served and Fujitsu witnesses accepted this could be done, in contradiction to their earlier evidence.
Dr Worden declined to “comment on this evidence” and remained of the view that Mr Roll could not
have done that which he said he had done - in other words, his factual evidence was incorrect.
874.No independent expert giving evidence to the court should do this. It is an obvious preferring of
the evidence of fact of the party instructing him, added, in this case, to a refusal or failure to accept
further evidence of fact to the contrary which subsequently emerged. This is even though the issue of
remote access was accepted by the Fujitsu witnesses in supplementary statements.
875.Mr Coyne did not do this. Indeed, he was fair and balanced in identifying where there were such
differences, and he was also clear that he did not prefer one side’s evidence of fact to the other side.
This is the correct approach for an expert to adopt.
876.1 find Dr Worden’s approach to the factual evidence I have identified concerning Mr Roll a
particularly egregious failure to maintain the necessary standard of impartiality. Injection of
transactions remotely into Legacy Horizon was an important factual point in this litigation, and indeed
in the whole history of the ongoing lengthy dispute between the Post Office and the claimant SPMs.
Further, in his expert declaration, Dr Worden went wider than the standard wording and specifically
referred to an earlier decision in another case. He stated at paragraph 1194 the following:
“T understand that my duty in providing this report and giving evidence is to help the Court on
matters within my expertise, and this duty overrides any obligation to the party by whom I am
engaged or the person who has paid or is liable to pay me. I confirm that I have complied and will
continue to comply with this duty. I have not assumed that any particular version of events is true and
I have had regard to the case of Imperial Chemicals Ltd v Merit Merrell Technology Ltd [2017]
EWHC 1763
(TCC) in producing my report”.
877.That particular case (of which there are a number of judgments, on liability and quantum)
concerned an expert and his preference for the evidence of fact of the party who had instructed him. I
had stated in that judgment at [74] that “An expert's role is not to decide issues of fact themselves,
and choose what facts to believe and what not to believe”. Specific reference was made by Dr Worden
to this judgment in his expert declaration. It might therefore be thought that the requirement upon
him as an independent expert not to take a view on which factual evidence he preferred was, or
should have been, at the forefront of his consideration. In any event, such an approach by the court to
how experts should approach evidence of fact is not particularly ground-breaking. Independence
requires an objective consideration by an expert of both sides of the factual evidence. It is the court’s
role, not that of an expert, to decide which evidence of fact is accepted and which is not.
Notwithstanding this, Dr Worden plainly accepted the initial factual evidence of Fujitsu and rejected
RLITO000005
RLITO000005
the factual evidence of Mr Roll. Then when Mr Parker accepted Mr Roll’s account, Dr Worden still
would not do so. He was quite wrong to do this on both occasions.
878.Further guidance on the approach of an expert, if guidance be needed, is also provided in [237] of
a later judgment in the same ICI v Merit series, namely [2018] EWHC 1577 (TCC) in which I stated:
“1. Experts of like discipline should have access to the same material. No party should provide its own
independent expert with material which is not made available to his or her opposite number.
2. Where there is an issue, or are issues, of fact which are relevant to the opinion of an independent
expert on any particular matter upon which they will be giving their opinion, it is not the place of an
independent expert to identify which version of the facts they prefer.”
879.Dr Worden went so far as to state, in respect of his consideration of the variations in robustness
of Horizon over time, that “in my opinion, as expressed in section 8, the Claimants’ shortfalls are not
caused by bugs in Horizon, or any lack of robustness in Horizon.” The question of whether the
claimants’ shortfalls was caused by bugs was not one of the Horizon Issues as I have explained above,
but in any event, Dr Worden’s conclusions that the specific losses were not caused by bugs/lack of
robustness was only reached by means of his statistical approach in section 8 of his first report. This
was so heavily based on assumptions that were either not valid, or conceptually flawed, that his
conclusions were not valid either. I would go further - the whole process he adopted so far as his
statistical exercise was concerned was both invalid and of no evidential value.
880.He also relied - in my judgment heavily - upon information from Mr Jenkins. Mr Jenkins was not
even identified as one of his sources of information in section 1.3 of his first report, headed “Sources
of information”. This means that Dr Worden was given access to information that was not made
available to his opposite number, contrary to the point at (1) above in [878] above. Although there
were some references throughout the text of the report to Mr Jenkins, Dr Worden did not routinely
identify where he had relied upon Mr Jenkins. He also provided a great deal more information about
this contact with Mr Jenkins in his oral evidence than he did in his written reports. In his cross-
examination, he identified, when asked about a passage, that he had obtained that information from
Mr Jenkins, which plainly took place before the first report was served as he accepted on his first day
in the witness box it was “a year ago”. This was not clear in the report itself. One example of this was
paragraph 654.2, where the report was dealing with the effect of the Receipts/Payments Mismatch
bug, and Dr Worden stated “Because the operation involved was apparently not a double-entry
operation on the BRDB, the countermeasure of checking the double-entry constraint DEA did not
catch it”. This information came from Mr Jenkins, but until Dr Worden was asked this in
crossexamination, no reader would be able to tell this. The involvement of Mr Jenkins in this
explanation in his report was simply hidden. Nowhere was there a note or summary of all the
information that had been given to Dr Worden by Mr Jenkins. In this litigation in particular, and given
the involvement of and information provided by Mr Jenkins, who knew so much about the Horizon
system, such a note or summary was, in my judgment, essential. This was particularly important given
there was no witness statement from Mr Jenkins. Dr Worden had been provided with, and had used,
information from Mr Jenkins in addition to the witness statements served by the parties.
881.Dr Worden also introduced the concept of “lasting” and “transient” effect of bugs or defects on
branch accounts. There was no such distinction in the Horizon Issues themselves. A further difficulty
with this approach is that Dr Worden confirmed that even an impact on branch accounts that was
corrected several months later by a TC would, so far as he was concerned, be of “transient” impact. A
different way of stating the same concept is that, so far as he was concerned, “transient” could be of
RLITO000005
RLITO000005
quite long effect. Given TCs are corrections generated outwith the Horizon system and are effectively
corrective actions issued by the Post Office itself, this moved his consideration away from the generic
Horizon Issues into the realm of the efficacy of the Post Office business systems as a whole, not only
including Horizon but their human processes too. That is not what the Horizon Issues were concerned
with, and he frankly admitted in his cross-examination that had he not done this, his opinion evidence
would have been rather different. Mr Coyne did not use lasting and transient effect, and correctly
addressed the actual Horizon Issues themselves, as drafted and agreed by the parties and as
approved by the court.
882.Dr Worden also adopted this different approach to impact upon branch accounts. Dr Worden only
considered an effect or impact on branch accounts where a discrepancy loss (or gain) was not
rectified by a correction such as a Transaction Correction, as set out in the entry in the 2nd Joint
Statement about the experts’ different approaches. This ignored the word “potential” in the Horizon
Issues themselves. In my judgment, that was a wholly unhelpful approach, and also had the effect of
minimising or reducing the number of bugs that have an impact upon branch accounts. This can be
demonstrated by considering the following hypothetical example.
883.Consider a situation where there is a software bug, which in this instance I shall call H. The
effects of H are such that they cause the branch accounts of a particular branch Post Office to suffer
an unexplained loss of £999.99. This occurs, and the SPM of that branch contacts the helpline and
goes through to SSC, who create a PEAK and identify this and after a few weeks of investigation, work
out that this is an unexplained software bug which they have not come across before. A KEL is
therefore also created identifying H. Future incidents of H occurring would (or
should) lead to new PEAKs which will also (or should) link to the same KEL, and to the same bug.
Fujitsu report to the Post Office about the existence of H, which they have now discovered, and
understand is a software bug. The Post Office accept that the sum of £999.99 has arisen in the branch
accounts as a result of H and issue a TC to the branch in that sum, to correct the unexplained loss in
that sum in the branch accounts. If one were to use Dr Worden’s interpretation of impact upon branch
accounts, by reason of the issue of the TC, H would not be a bug, error or defect with an impact upon
branch accounts. Such an approach is, in my judgment, wholly artificial and incorrect. It also defies
the proper meaning of the words.
884.Mr Coyne also considered substantially more PEAKs and KELs than Dr Worden, by a factor of
almost ten. Mr Coyne stated that he and his team had considered a broad total of about 7,500. Dr
Worden had only considered a far smaller sample. Mr Coyne’s far more vigorous and thorough
exercise relied far less on the process of extrapolation inherent in Dr Worden’s approach, who was
inevitably basing his opinion on a far smaller sample than Mr Coyne. Given the breadth of the
resources being deployed by the Post Office in conducting this case, I find it difficult to understand
why Dr Worden restricted his consideration of PEAKs and KELs in this way. However, I consider that
the conclusions of Mr Coyne, based on the far more substantial consideration of these important
documents, are more reliable than those of Dr Worden.
885.To be fair to Dr Worden, he entirely accepted that he and Mr Coyne had approached their
evidence differently. He stated the following when explaining why he had not supported Mr Coyne’s
information requests.
“A. Well, basically I think it became evident, and it is evident now, that two experts took very different
approaches, and my approach was top down, understand the architecture and work down through
things like KEL. So as far as I was concerned, I had plenty of information to go on, And Mr Coyne's
RLITO000005
RLITO000005
information requests didn't strike me as things that, yes, I have got to really see that because I was --
had a different priority. I was trying to do a top down understanding of the architecture and top down
look at robustness, and so on and so forth. And so I had plenty of documents to look at, basically. So I
think it is the different approaches taken by the two experts that led to lack of overlap.”
886.This continued:
“Q. Sorry, not having supported the very many requests he had to make?
A. Yes, I felt his interests were different from mine and I really had plenty to do.
Q. And you initially only looked at KELs, didn't you, you didn't actually look at PEAKs?
A. No, I looked at KELs and I looked at PEAKs where they were relevant. I felt that KELs were a more
distilled form of information. I felt they were sufficient in many ways, especially when you go and look
at the PEAKs.
Q. You looked at one or two PEAKs referred to in the KELs you looked at?
A. Yes. Look at some of them where you feel the KEL doesn't tell you enough.
Q. And Mr Coyne was looking at PEAKs and KELs?
A. I think in the first report he was looking mainly at KELs, like me, and in his second report he turned
on PEAKs.”
887.The reason that PEAKs are of great assistance is that there is a great amount of further
information contained in them than in the KELs, and they also give a far more complete picture of the
number of different occurrences of a bug. Both types of documents are however useful. I do not know
why Dr Worden should have seen Mr Coyne as having “different interests” to him.
888.Dr Worden also entirely seemed either to misunderstand (or restrict his consideration of) Horizon
Issue 1, which concerned bugs, errors and defects having the potential to cause apparent/alleged
discrepancies or shortfalls in branch accounts or transactions, or undermine the reliability of Horizon
accurately to process and to record transactions. When challenged about his approach to this,
including the absence of the word “cause” and the use of the word “potential” in the issue itself, he
stated the following:
“A. What's the distinction between having the potential and actually doing? I mean, if it doesn't
actually do, then in some sense it didn't have the potential.”
889.This is not the correct, or conventional, use of the word “potential”. It also looks at the matter in
reverse. In the context of the Horizon system, whether bugs, errors or defects in that system have, or
had, the potential to impact branch accounts or transactions, should focus one’s attention upon the
possible effect of those bugs, errors or defects. If, of course, a particular bug actually has had the
effect of in fact impacting branch accounts, then by definition it must also have had the potential to do
so. Actual effect is a sub-set of potential effect. It means impact upon branch accounts must have
actually happened. However, potential effect does not require that impact to have actually occurred.
The two terms are not synonymous. Dr Worden therefore moved away from the actual issues before
the court, and approached a great deal of his evidence from the point of rather different issues, which
he seemed to have narrowed down himself. I consider such narrowing of the issues to be
unwarranted.
RLITO000005
RLITO000005
890.In their opening submissions, the claimants summarised this different approach as follows:
“We respectfully will invite the court to adopt an approach more consistent with Mr Coyne's approach
which is to look at what actually happened with particular examples and trace them through to a
reasoned and careful conclusion and then from the ground up, as it were, draw inferences upwards
rather than from an overarching hypothesis downwards.”
891.The “overarching hypothesis” was that of Dr Worden. Because he had taken such an overall view,
and an overall view on the most recent performance and functionality of the Horizon System (which is
now HNG-A as of February 2017, and is no longer HNG-X), the degree of assistance his evidence
provided to the Horizon Issues across the whole life of the system, both Legacy Horizon and Horizon
Online, was more limited than that of Mr Coyne.
892.In this group litigation, when the first individual claims are being tried out, further analysis will
be required of the actual branch accounts of individual claimants. That stage of the litigation will deal
with actual branch losses, if any, that are said by the claimants to be due to the actual impact of the
bugs, errors and defects which exist. It is at that stage that findings will be made about actual impact
upon the branch accounts of the different claimants. Some of the bugs and defects are agreed by Dr
Worden and Mr Coyne, and those that are not agreed are resolved by me in the Technical Appendix. It
is against the background of those findings that individual claims will be resolved. The purpose of the
Horizon Issues trial was to resolve the generic issues, as the parties were poles apart about even the
existence of such bugs, errors and defects, and whether they had the potential to act upon branch
accounts and transactions in the way alleged by the claimants. Dr Worden moved away from
consideration of those specifically drafted and agreed Horizon Issues and answered rather different
ones,
893.1 also consider that Dr Worden, in his approach to countermeasures, insufficiently differentiated
between measures which were part of the Horizon system and those that were not. He also included
items in his countermeasures that plainly ought not to have been included as such, for example those
the result of human intervention or involvement, and manual inspection of data by SPMs.
894.In my judgment, Mr Coyne emerged more or less unscathed from his four days of cross-
examination by leading counsel for the Post Office. That is not to criticise the cross-examination itself,
which was very thorough and careful. It is just that what the Post Office set out to demonstrate,
including lack of impartiality on Mr Coyne’s part (if not positive bias), lack of reliability in his
conclusions, and the other detailed criticisms of him, simply was not achieved. He was described as
“evasive” in some of the Post Office’s closing submissions, a term which I find is somewhat wide of the
mark in describing how he gave evidence in the case before me, and how he answered questions. He
was also criticised in the same submissions as adopting “an approach [that] was simply to look for
bugs without giving a proper sense of scale, context or even balance”.
895.In the Post Office’s closing submissions, there is a section headed “Judicial criticisms of Mr
Coyne’s approach” where reliance is placed on a judgment in Scotland, called CGI IT UK Ltd v
Agilisys [2018] CSOH 112 by Lord Bannatyne. In that case, Mr Coyne had appeared as an expert and
he was criticised by the judge for an unbalanced view, amongst other things. The judgment was
handed down on 4 December 2018. The fact that he had been judicially criticised in that case was not
specifically put to Mr Coyne. However, the substantive criticisms of him, that he gave unbalanced and
selective evidence, and the other criticisms, were put to him, and he did not accept them. The point
that he had been previously criticised in a judgment ought specifically to have been put by reference
to that case, so that he could answer this point. However, in any event I have read (as invited in the
RLITO000005
RLITO000005
Post Office’s submissions) the entire Scottish judgment, in which the evidence of Dr Hunt, Mr Coyne’s
opposite number, was preferred to his evidence. I have done this with particular care, and I have also
considered all the criticisms made of him in that judgment, as well as the ones put to him in his cross-
examination before me. The Scottish judgment provides a full explanation of the shortcomings of his
approach in that case. I have also taken this prior judicial criticism into account in weighing up his
expert evidence in this trial.
896.The Post Office, amongst other things stated that “it is submitted that the examples set out above
[ie in the Lord Bannatyne judgment] show the same tendency to reverse
and switch in response to difficult points.” In other words, it is submitted that Mr Coyne adopted the
same approach in the Horizon Issues trial before me, as he had in the earlier case in 2018. I have
considered the criticisms of Mr Coyne with particular care in the light of this previous judicial
criticism, even though I do not have the benefit of Mr Coyne’s explanation to the specific points
arising out of that case. I do not find his evidence in the Horizon Issues trial suffers from the same
deficiencies as Lord Bannatyne found in that earlier case. In particular, one of those criticisms in CGI,
that he had failed to consider material documents and matters, simply does not remotely apply here.
He and his team had done an extraordinarily thorough job of examining many thousands of PEAKs and
KELs, and many more than Dr Worden did. I do not find that Mr Coyne reversed his evidence, or
switched, or that any of the other criticisms of him are made out. It may be that he has changed his
approach as a result of his experience in that case or judgment, or there may be other reasons for it.
However, I do not find the criticisms made of him to be made out in this case.
897.In my judgment, given the Horizon Issues, which expressly include “bugs, errors or defects”, the
approach of an IT expert to start by looking in detail at the large volume of the Fujitsu PEAKs and
KELs for bugs is not only entirely sensible, it is verging on essential. I reject the criticism that Mr
Coyne’s evidence was given without having any sense of scale, context or balance. I conclude that Mr
Coyne was impartial in his evidence in the Horizon Issues trial, and was giving the court evidence that
was entirely uninfluenced by the exigencies of litigation or the interests of the party who called him. I
find that he was independent, and that his conclusions in this case on the Horizon Issues are reliable.
898.There were some subjects upon which he became a little flustered in his crossexamination, for
example on the final afternoon when he was taken to task for the entry in the 2nd Joint Experts’
Statement on an agreed passage where he and Dr Worden had agreed that “the number of distinct
bugs, for which the experts have seen strong evidence of the bug causing a lasting discrepancy in
branch accounts, is between 12 and 29”. That latter number had come from Mr Coyne and he wished
to reduce the number, initially to 13 but then upon reflection to 21. Mr de Garr Robinson made a great
deal of this, as one would expect of a skilled cross-examiner, but in reality it is a simple point, as I
have explained above. It did show a change of position because the number of bugs changed as I have
explained. However, it is easy to overestimate the degree to which making an error on figures in such
circumstances does, or does not, represent a lack of reliability on the part of an expert who has
performed a substantial body of work. It must be remembered that this was an agreed entry in an
agreed Joint Statement that was 44 pages in length (the experts agreed four separate statements,
running to 100 pages in total including appendix). I deal with the detailed analysis of each alleged bug
in more detail in the Technical Appendix, but the fact that the number did not remain an immutable
“29” in all circumstances is not, in my judgment, a point against Mr Coyne. His attempts to ensure the
final number was correct was not as smooth as he might have liked, and it gave Mr de Garr Robinson
an opportunity to deploy his forensic skill, but in reality it did not impact upon Mr Coyne’s reliability.
RLITO000005
RLITO000005
899.Mr Coyne was widely attacked for a lack of partiality, and indeed, for positive bias in favour of the
claimants. I do not find those complaints or criticisms to be well founded. They are not based on any
real substantive instances that can be shown in
any specific respect to represent Mr Coyne demonstrating any lack of impartiality. They are not made
out. The points made are, in my judgment, insufficient grounds to dent his credibility as an
independent expert whose evidence can safely be relied upon in resolving the Horizon Issues. He had
not been previously instructed to undertake work for either party, and he had no previous involvement
in the history of the wider dispute between the Post Office and SPMs concerning the Horizon system
and its reliability. I found his evidence before me to be neutral and independent, and not influenced by
the fact that he was instructed for the claimants.
900.Dr Worden’s decision, very late in the day, to prepare his 3rd Report, appears - by reason of the
dates - to have coincided with the dismissal of the recusal application. This matter was not explored at
all in cross-examination and so it is not necessary to consider the timing of Dr Worden’s realisation
that he could perform a new exercise. Given that his oral evidence to me was that he was advised to
serve that 3rd Report directly upon the court, and effectively was left to draft the covering email to
the court himself, I am reluctant to criticise him for this unusual approach unduly. This is therefore
not something that I weigh in the scales to any appreciable degree in deciding whether his opinion
evidence and conclusions are more, or less, reliable than those of Mr Coyne.
901.1 do not wish it to be considered that I am being universally critical of Dr Worden in all respects.
He had done a substantial amount of detailed analysis in his two reports, and the overall knowledge
that the court had, at the conclusion of the crossexamination of both experts, was very much greater
than it had been prior to the commencement of the Horizon Issues trial. Dr Worden discovered some
bugs himself, bugs that neither the Post Office and/or Fujitsu either knew about, or had admitted the
existence of, prior to Dr Worden’s involvement in the litigation. He also took a sensible and considered
view of some elements of the documentation, the importance or relevance of which had not been
admitted by the Post Office earlier in the litigation, such as the KELs. Some of the matters that were
agreed by Dr Worden with Mr Coyne in the 1st to 4th Experts’ Joint Statements saved a considerable
amount of court time. His entries in particular in respect of the Bug Table, where he considered the
detailed contents of a number of PEAKs and KELs, were useful as well.
902.However, Dr Worden had become closely involved with the Post Office’s case in the litigation. In
my judgment he took a partisan view of the evidence of fact of Mr Roll, which he should not have
done, and he also seemed to be acutely aware of the Fujitsu position. For example, if there was a
problem in Horizon that originated with Riposte, rather than part of the system written by Fujitsu, he
was always enthusiastic to make clear this differentiation in Fujitsu’s favour. It was not relevant to the
Horizon Issues whether bugs, errors and defects in Horizon were caused by Fujitsu or Riposte
(provided by Escher, a separate company), or any other part of the system. These are all points that
lead to the same conclusion, namely that in my judgment his expert evidence was not entirely
independent of the Post Office’s and/or Fujitsu’s interests, and his conclusions were substantially less
reliable than those of Mr Coyne. I do not know whether he had found himself subconsciously
influenced by his discussion with Mr Jenkins or not, but regardless of the reason, his conclusions were
not reliable and I reject them in favour of those of Mr Coyne.
903.In all the major respects therefore, and taking account of all the points made both in cross-
examination and submissions by both parties concerning their own, and the opposing side’s, IT expert,
I prefer the evidence and conclusions of Mr Coyne to those of Dr Worden.
RLITO000005
RLITO000005
The number of established bugs, errors and defects in Horizon
904.These are included in summary form in Appendix 2, together with whether they were present in
Legacy Horizon or Horizon Online (or both) and their approximate years of operation. For a detailed
analysis, the Technical Appendix to this judgment (which is Appendix 1) should be consulted.
K Audit Data
905.The experts are agreed, in item 4 of the points of agreement referred to above in respect of
Horizon Issue 3 as part of the 3rd Joint Statement, that the Post Office does not consult the audit data.
The actual text of that entry is the “Post Office does not consult the full audit data (unfiltered ARQ
Data) before deciding how to handle discrepancies and issuing Transaction Corrections.” Regardless
of this agreement, it is obvious on the factual evidence that the Post Office does not do so. Indeed,
throughout both the Common Issues trial and the Horizon Issues trial, it has been increasingly
obvious that the Post Office uses sources other than the audit data when it is challenging SPMs about
what they have actually done in their branch Post Offices. Various terms are used for the audit data.
The 3rd Joint Statement uses “full audit data” and “unfiltered ARQ Data”. In places the term “core
audit data” and “core audit store” was used, which refers to the same data and store. I shall refer to it
as audit data.
906.The issue of audit data arises in the following way. Audit data is a complete and accurate record
of everything that has occurred, which in the context of Horizon means including a full record of
keystrokes used by a SPM (or assistant) in the branch. This accurate record is kept in what is called
the audit store. This is a secure place for the keeping of such data. It is vital to the proper operation of
a system such as Horizon that such accurate audit data is kept.
907.In his evidence Mr Coyne referred to the concept of “Write Once Read Many”, which has its own
acronym, WORM. Dr Worden accepted that this was “pretty common” in the industry. It means,
effectively, that audit data is only written once - indeed, it must be only written once. That is the
whole point of audit data. However, it is written (or in other words, recorded) so that it can
subsequently be read, and the second part of this concept, “Read Many”, means that it can be read
many times. The audit data is there, in my judgment, so that it can be consulted when required. The
experts’ agreement makes it clear that the Post Office does not consult it before deciding how to
handle discrepancies and issue TCs.
908.The Post Office did not accept that it should access the audit store as suggested by the claimants,
or consult the audit data. The claimants’ case is that the audit data should be used in circumstances
where there was a dispute between the Post Office and the SPMs about what had occurred in Horizon
in a particular branch. Mr Coyne for the claimants criticised the Post Office in his 2nd expert report
for not using audit or ARQ data, and relying instead on what was called management data, which was
said (by the
Post Office) to be sufficient to demonstrate what had occurred. I will reproduce part of Mr Coyne’s
cross-examination on this.
“Q. But did you see any documents of any sort indicating or referring to the stream of data flowing on
a continual basis out of the audit store into Post Office's management systems?
A. No, but that's not how things would work. If Post Office wanted to get access to the data in the
audit store they would go to a screen or go to an application on their computer and they would run
the request for that data.
RLITO000005
RLITO000005
Q. Mr Coyne, I would like to suggest to you that it is completely unrealistic to think that a separate
sealed core audit store of the sort we're talking about should be cracked open hundreds of times a day
in preference to using management information systems which are designed for that precise purpose?
A. I think the word "sealed" is misleading and the concept of cracking something open to get access to
it Lthink is misleading as well.
Things in an audit store are only - can be written to and only written to once, and the term that's
often used is write once read many, WORM. So the process is written to once, but people can read
from that store on many occasions.
Q. But just to be absolutely clear, you had not and indeed you have not seen any documents
suggesting that Post Office had the ability to gain access to the audit store on its own systems, had
you? There was no design facility, there was no -- there were no lines of communication between the
audit store and Post Office in any document you had ever seen, correct?
A. No, it looks as if the majority of the references to audit database access was from Fujitsu
personnel.
Q. And one final thing. Would I be right in thinking that now that you understand how the audit store
actually works and the costs and delays associated with extracting data on a large basis from the
audit store, would you accept that it would be dispr« ‘ion: using
for making decisions on transaction corrections in every single case?
A. Yes, it would seem that it would be very expensive and very slow to access the audit store, and
effectively for the number of transaction corrections you couldn't do that, and therefore you accept
that you make decisions on the management information systems rather than the audit store.”
(emphasis added)
909.The Post Office therefore relied upon the fact that the audit data was effectively kept by Fujitsu;
that accessing it was akin to “cracking open” something that was a “separate sealed” store; and that
it would be disproportionate to use it to make decisions on transaction corrections in every case.
910.The first and third of those points are accepted; the second one Mr Coyne described as
misleading. I agree with that response to all three of those points. I also find that the concept of
“cracking open” the “sealed store” is rather off the point of the very purpose of audit data, and the
uses to which it both can and should be put.
911.The following points are, in my judgment, important on this issue.
1.Audit data is necessary so that there is an accurate record of what has occurred on the system,
including what a SPM has done in branch, in terms of the keystrokes actually used. The use of the
word “audit” makes this clear, and also this is not a controversial point.
2.The need to consult such an accurate record does not arise in every case where a transaction
correction is considered or issued. The management data is sufficient for a large number of
transaction corrections.
3.However, where there is a dispute between the Post Office and a SPM about branch accounts, with
the arguments on both sides about what or who is to blame, the audit data should be consulted. That
is one of the purposes of having it in the first place. I can think of no sensible reason not to consult the
audit data in such a scenario.
RLITO000005
RLITO000005
4.Doing so is not equivalent to “cracking open” a “sealed store”. The audit store is sealed in the sense
that data, once in the store, should not be capable of being changed, or written to again. However,
reading that data is not “cracking open” the store. It is consulting or reading the audit data, one of the
main (if not the only) purposes of having such audit data in the first place.
5.The evidence in both this trial, and the Common Issues trial, where the Post Office cross-examined a
number of SPMs on events in their branch accounts by using a variety of management information,
other than the audit data, makes clear to me just how important it is to use the audit data, rather than
other sources including management information. The management information is confusing,
contradictory, has been shown to be wrong and requires numerous assumptions or a “take it from me”
type of approach on the part of a questioner. It is rarely agreed what that management data shows.
The audit data, by its very nature, will be far superior and the best evidence available of what has
occurred on Horizon. It should be consulted in circumstances where there is a dispute between the
Post Office and a SPM.
6.Finally, even the Post Office’s own witnesses were not agreed on what the management data would,
or would not, show. At one point Mrs Van Den Bogerd said that Credence (which is management data)
would record keystrokes in branch. This was then contradicted by another Post Office witness who
said Credence did not do this. Whether the lack of understanding on the part of Mrs Van Den Bogerd
was widespread or not within the Post Office is unclear to me. It is however clear to me that Credence
data does not in all circumstances record all keystrokes performed in branch by a SPM. The audit
data does.
912.Dr Worden explained audit data as being a central principle of the whole system in the following
terms:
“Tt is a central principle of Horizon that the Core Audit Database acts as a secure ‘gold standard’ for
branch accounts (countermeasure SEK) and that the audit record can only contain events which
originated at the counter - either in customer transactions or monthly balancing.”
913.That is an accurate description in my judgment. I can see no sensible or justifiable reason for the
Post Office’s reluctance to consult the audit data in cases of serious
dispute with SPMs, in particular the types of dispute which form the subject of this case, and without
doubt, any dispute that involves criminal proceedings against SPMs. I also consider the same point
applies in relation to internal Post Office proceedings that lead to the suspension or termination of
SPMs. Acceptance of that point does not mean that that the audit data has to be consulted for every
transaction correction issued by the Post Office. As can be seen from the passage of cross-examination
at [908] above, the Post Office effectively jumped to the issue of whether audit data should be
consulted before issuing every single TC. Mr Coyne accepted it was not necessary to do that. That is
not the same as accepting it should not be consulted in circumstances where there was a dispute
between the Post Office and SPMs.
914.Mr Coyne had identified issues with using Credence data. There was a one-hour difference in the
time stamps used between Fujitsu and Credence, which can hardly have helped sensible
investigations when SPMs raised queries, but there is more to this than that. The E&Y review in
March 2011 identified various issues with Credence, including weak change controls within the back-
end of the systems which allowed Logica developers (the third-party provider) to move their own
uncontrolled changes into the production environment, which included both Credence software code
and the data within Credence used for what was called “audit evidence” but which has to be
RLITO000005
RLITO000005
differentiated from what I am referring to audit records in the audit store. There was a lack of further
documentation to approve fixes and patches applied to Credence outside of the release process, which
meant that linking changes to issue tickets to record the original request for the bug fix was not
possible.
915.A concern was also noted by E&Y in respect of front end change process weakness. The passage
states:
“During our walkthrough of user administration of the front end of Credence we noted several users
with administrator rights, including some generic users (this is noted below as a separate point).
These users have the access rights to create and amend reports, including those which may be relied
upon for audit evidence. These users can change report design, and processing without documented
request, test or approval. When users have the rights to change reports that are used by the business
for reconciliation, exception reporting or other processing, there is the risk that the reports are
manipulated either intentionally or accidentally.”
916.There is also at least one specific occasion, considered in the evidence during the trial, which
shows that the Credence data does not show the correct position. This was put to Ms Van Den Bogerd.
This was an occurrence at Lepton in October 2012. It led to something that was referred to
throughout the trial as the Helen Rose Report or Rose Report, as the author of the report into it was
called Helen Rose. I have dealt with it to a degree at [227] above. That report records that:
“A transaction took place at Lepton SPSO 191320 on the 04/10/2012 at 10:42 for a British Telecom bill
payment for £76.09; this was paid for by a Lloyds TSB cash withdrawal for £80.00 and change give for
£3.91. At 10:37 on the same day the British Telecom bill payment was reversed out to cash settlement.
The branch was issued with a Transaction Correction for £76.09, which they duly settled; however the
postmaster denied reversing this transaction and involved a Forensic Accountant as he believed his
reputation was in doubt.” (“duly settled” means the SPM paid the Post Office that sum).
The Credence data showed that the SPM had reversed the transaction. By consulting the audit data,
Mr Jenkins discovered that he had not. This was expressly confirmed, both in the Rose Report and
also by Ms Van Den Bogerd in her cross-examination.
917.This shows that the management data is not entirely reliable. Had the SPM at Lepton simply paid
the approximately £80 and not engaged the forensic accountant - in other words, had he not been
prepared to pursue this matter - the Post Office would simply have received that sum from the SPM to
which it was not entitled, and the Rose Report would not have emerged.
918.The issue of remote access also further complicates (or rather, weakens) the reliability of
management data. All of these concerns reinforce the view that I have in any event, that audit data (as
in, that stored in the audit store) is what should be used in the circumstances that I have identified.
919.One point that occupied the parties a great deal in the Horizon Issues trial, upon which there was
some evidence, and upon which Mr Coyne formed a view based on his analysis of some
contemporaneous documents, is that Fujitsu raised a charge to the Post Office for audit data requests
(also called “ARQ requests”) above a certain number per year. The numbers that were discussed were
in the amount of hundreds of pounds per request. The parties could not agree on what these charges
were, although it was accepted that there were a number each year that were either free, or the costs
of which were included in the contract sum (depending upon which way one looks at this matter). The
RLITO000005
RLITO000005
charging structure will, in terms of the amount per request above a particular number, have changed
over the years in any event.
920.That charges are raised by Fujitsu to the Post Office is not an adequate answer, in my judgment,
to the Post Office’s failure to consult or provide the audit data in cases such as those in [912] above.
There are some contemporaneous references within Post Office documents suggesting this may have
been a disincentive in some cases to raising ARQ requests of Fujitsu. Further, there are numerous
references within Post Office documents (at a high level) of the very great cost to the Post Office of
the Horizon system generally, that cost being paid to Fujitsu. Fujitsu were said in one document to see
the contract with the Post Office as a “cash cow”. This entry came in a document (parts of which were
redacted, which I record simply for completeness) dated 17 January 2017 which were the minutes of
the Post Office Group Executive Meeting of that date. Ms Van Den Bogerd was “in attendance” for two
items only, numbered 6 and 7, but not in the list of those “present” by which I assume she was not
there for the whole meeting. The Post Office Chief Executive and Group Legal counsel were there,
together with a number of others.
921.There is only one agenda item which is not redacted, namely item 5, IT Strategy. Parts of the
minutes relevant to the Horizon Issues are as follows:
“RISKS
+Need to be clear for the Board what operational risks we are facing and how we are mitigating those
risks, are there any choices we can make to accelerate the solution to get inside the risk appetite.
+Already making these changes to the areas of risk, but will make this clearer. Need a strategic
discussion at GE on choices in the IT strategy
«Horizon - cannot accelerate work on the data centre refresh, getting under control. FJ running this
well, risk review & declaration every month. RH will kick again but doing as much as we can.
«Credence - Support and maintenance has moved to Accenture but on old hardware. Single platform if
goes, goes completely. Driving pressure into Accenture to get more capability. Accenture picked up
difficult pass and the additional costs at the moment. Starting to see light at the end of the tunnel. RH
having difficult conversations, flag up any issues, assumptions and risks to GE. KPMG engaged to
undertake a GAP analysis to produce an IT control framework.
+ What are the risks of the ‘hop’ to thin client not working? RH - need to prove thin client works and
the ‘hop' is a safe way to test we could revert to HNGA if the ‘hop’ fails. All nervous as we are dealing
with 1 ms. Moré maf le that we are not changing Back & Front office at the same
time. but still using Horizon. SSKs have proven that thin client can work, it is the journey to
implement that is vital.
because of th rement risk. Only choice if they will not work with us i rt buildin
alternative. Come back to
GE with an update on the FJ negotiations plan”
(emphasis added)
RLITO000005
RLITO000005
922.This document shows that Fujitsu, at least so far as the Post Office are concerned, are seen by the
Post Office as believing that the contract with the Post Office is “too good” and a “cash cow”, such
that Fujitsu may not even “work with us” [ie the Post Office] in terms of improvement. Part of that
very attractive commercial arrangement (attractive for Fujitsu) may be charges that Fujitsu is entitled
contractually to raise for ARQ requests. Part of an attempt to reduce the expense of the contract by
the Post Office may be a reluctance to raise such requests. Dr Worden described this document as
“chiming” and said that “I have always had the impression from the governance structure and the
documentation and so on that Fujitsu were not short of budget, really”. By that he meant Fujitsu were
well-remunerated, and this remuneration was paid by the Post Office.
923.Such private commercial arrangements between the Post Office and Fujitsu are a matter for the
Post Office and Fujitsu. They do not, in any way, justify any failure to seek the audit data - the best
evidence - in cases where SPMs are being suspended and/or having their appointments terminated,
particularly in circumstances where there are so many bugs acknowledged as existing, and also at a
time (much earlier
than this judgment in 2019) when Fujitsu knew there were bugs in Horizon such as Dalmellington and
Callendar Square, and also given the Credence data has been shown to have been wrong on occasion.
There is quite enough in issue in this litigation as it is, without widening it to include the way the Post
Office deals with its commercial arrangements with Fujitsu.
924.Finally on this point, I am surprised that the desirability - if not the actual and basic need - to
consult the audit data is a controversial point. In my judgment it is not only good practice to consult
the audit data, given the very purpose of audit data within a complex IT system such as this one, but it
is also obvious common sense. There is little point in having audit data if it is not consulted in the
circumstances that I have identified above. I appreciate that the audit data itself will not be
immediately comprehensible to some SPMs. There may also be charges raised by Fujitsu in respect of
such requests. However, neither of these are reasons for the failure to consult it. I also make no
findings on whether Fujitsu are, or are not, entitled to raise charges and if so how much. Those latter
two points were not fully argued before me, and Fujitsu were not represented.
L. Overall Conclusions
925.My judgment concerning the two experts, which of their evidence I prefer and why, has been
reached on free-standing grounds. Because of the different way that the experts went about their
tasks, it was not possible to consider in detail and track in Dr Worden’s reports any analysis and
possible explanation of single incidents, and attribute that either to a specific bug, error or defect, or
something else which was none of these things. The approach Dr Worden had adopted for his written
evidence did not permit that, as he was (in a sense) working from the other end (overall reliability)
back towards the beginning (individual experiences), in contrast to the approach of Mr Coyne. Dr
Worden himself described his approach as “top down”. Mr Coyne, on the other hand, using PEAKs and
KELs in particular, identified a great many specific incidents that Fujitsu had recorded, which plainly
demonstrated the existence of numerous bugs. Indeed, some of the terms used by Fujitsu made it
crystal clear that these were known to be bugs. One notable example is the expression used by Anne
Chambers - “this bug has been around for years” - in February 2006. Dr Worden did engage with the
detail of PEAKs both in the joint expert meetings and also in his oral evidence, and the final result is
that by the end of the trial, the summary at the beginning of Appendix 2 of the Post Office closing
submissions, was forced to accept the existence of a great many bugs in Horizon. The extent and
nature of their impact was not agreed, but the Post Office could no longer deny their existence.
RLITO000005
RLITO000005
926.Those claimants who gave examples of specific incidents specified them as well as they could, and
in my judgment truthfully and accurately. It was usually Ms Van Den Bogerd who, for the Pot Office,
attempted to demonstrate that this was not what had occurred, often by means of assertions that if
the SPMs had followed procedures properly, what they said happened, simply could not have
happened. The Fujitsu witnesses gave some limited evidence of fact in relation to specific instances
and some specific PEAKs and KELs. The Post Office’s counsel did their best to challenge the factual
accounts that were given by the claimants. I have provided the relevant findings in respect of those
matters above. I indicated at the Case Management Conference on 22 February 2018 that the Horizon
Issues should be capable of resolution with limited, if any, evidence of fact. Both parties chose to
submit factual witness statements and called factual evidence. The factual evidence submitted by both
the parties was, in my judgment, limited as expected and required. With the exception of the matter to
which I have already referred concerning the evidence of Mr McLachlan and Mr Henderson at the
PTR, the extent of each party’s evidence of fact was not subject to challenge prior to commencement
of the Horizon Issues trial.
927.The factual evidence of specific instances was of assistance in coming to conclusions on the
Horizon Issues. Indeed, I found some of the factual evidence to be of great assistance. That of Mr Roll
and Mr Godeseth was extremely useful. The latter, one of the Post Office’s main witnesses and the
Chief Architect of Horizon, was sufficiently damaging to the Post Office’s case on the Horizon Issues
that they were, essentially, forced almost to disavow him, and the Post Office’s closing submissions
were highly critical of the accuracy of his evidence. As a further example only, Mr Latif gave evidence
that he had performed certain basic routine steps to transfer the sum of £2,000 between terminals,
and (in accounting terms) that sum had - to use his word - “disappeared” from the Horizon system.
The Post Office evidence challenging this was from Mrs Van Den Bogerd, and essentially amounted to
an assertion that if certain steps had been followed by him, this should not, or could not, have
happened. That would be all well and good in an ideal theoretical world that did not pay any attention
to reality, but it entirely ignores Mr Latif’s primary evidence of fact. I accept that primary evidence
from the claimants’ witnesses in this trial as I have explained in Part D. Mr Latif had been a trainer
trusted by the Post Office to train other SPMs, as explained at [98] above. Other references in the
documents show both Romec engineers, and even Post Office auditors, being recorded in PEAKs as
witnessing events supporting the type of occurrences that underpin the claimants’ case. These were
ignored by those at Fujitsu tasked with completing these PEAKs, and conclusions are shown in those
PEAKs being drawn by Fujitsu that flew in the face of what had occurred. Mr Latif’s experience is
extraordinarily similar to the Dalmellington bug, which is numbered 4 in the Bug Table. Mr Latif did
not operate core and outreach branches, therefore his experience is slightly different to what
happened at Dalmellington. However, if one is to take an objective and sensible view of the evidence
of a previously trusted SPM, whom the Post Office itself has used to train other SPMs for some years,
who stated that he did something correctly and X occurred, then meeting that with a bare assertion
that X simply cannot have happened is not particularly sensible, nor is it persuasive.
928.The approach by the Post Office to the evidence of someone such as Mr Latif demonstrates a
simple institutional obstinacy or refusal to consider any possible alternatives to their view of Horizon,
which was maintained regardless of the weight of factual evidence to the contrary. That approach by
the Post Office was continued, even though now there is also considerable expert evidence to the
contrary as well (and much of it agreed expert evidence on the existence of numerous bugs).
929.This approach by the Post Office has amounted, in reality, to bare assertions and denials that
ignore what has actually occurred, at least so far as the witnesses called before me in the Horizon
RLITO000005
RLITO000005
Issues trial are concerned. It amounts to the 21st century equivalent of maintaining that the earth is
flat.
930.When real world examples such as Mr Latif’s are put together with the expert evidence that I
have accepted - or even with Dr Worden’s lower figure for accepted bugs of 11 different ones - it can
be seen that this institutional obstinacy by the Post Office amounts to little more than repeated
assertions that the Horizon system (both Legacy and Online) cannot be to blame for the claimants’
experiences, coupled with (for some) challenges to the claimants’ witnesses because the Post Office
simply cannot accept their factual accounts. The findings that I have made, on the evidence in the
Horizon Issues trial, show that the reality is rather different, and the existence of the bugs, errors and
defects that I have found to exist do have the effect explained by Mr Coyne.
931.The approach that I have adopted to the evidence can be considered as follows. Firstly, to
consider the factual evidence, the relevant contemporaneous documents, and applying the burden and
standard of proof, reach evidential findings. I have considered the challenges to that evidence, and if
the specific instances of factual witnesses explaining what they experienced could be explained (as
the Post Office sought to do with some of the SPMs) by them simply not being reliable, remembering
incorrectly, having been careless or even (for some) being criminal. I have also considered all of the
Post Office’s factual evidence in this trial, the explanations of the Post Office’s witnesses, and reached
my factual conclusions. I have considered the expert evidence on its merits, and then tested my
conclusions against the evidence of fact that was adduced by both parties, and considered whether my
judgment on the evidence of fact matched and supported, or contradicted, or in any way affected or
influenced my conclusions on the expert evidence. In a sense this was to consider the factual
evidence, and the expert evidence that I accepted and preferred, separately, and see if they both
reached the same conclusion. Secondly, the evidence can also be approached in a different sequence.
This is by firstly considering the specific factual evidence, and then considering the expert evidence,
and only in the light of the conclusions on the factual evidence. Alternatively, one could simply decide
the Horizon Issues on the expert evidence alone, although that would be far less preferable and would
ignore entirely the important factual evidence of the Fujitsu witnesses, in particular Mr Godeseth.
However, even if that third way were to be adopted, the same answers to the Horizon Issues would, in
my judgment, be reached.
932.Whichever way is adopted, it must be remembered that any claimant can only ever have personal
experience of one small element of the Horizon system on a number of occasions. The overview and
historical performance of Horizon, both Legacy Horizon and Horizon Online, has to come from Fujitsu
witnesses and the two experts. Dr Worden and Mr Coyne had far more of an overview of the life of the
system, as did Mr Godeseth and Mr Parker. Mr Roll had experience of it in its early years, but his
evidence was what led to the correct picture on remote access finally emerging.
933.1 have done all of these exercises. Regardless of whichever approach is adopted, the same
conclusions are reached. In this respect, all roads lead to Rome. In my judgment, the evidence of the
factual witnesses accepted by me entirely supports and corroborates the conclusions reached by Mr
Coyne. Indeed, I would go further than this, and state that Mr Godeseth’s evidence alone is enough to
support and corroborate Mr Coyne’s conclusions. When that is put together with the evidence of Mr
Roll, and the concessions that were obtained from the Fujitsu witnesses (in the circumstances of their
performance as witnesses, to which I have already referred) it is clear to me that
the correct conclusions to be drawn on the Horizon Issues are those drawn by Mr Coyne, save and to
the extent that I have modified them in any specific respect.
RLITO000005
RLITO000005
934.I consider that the evidence of the Fujitsu witnesses in particular, both former and current, has
been of considerable assistance to me in resolving the Horizon Issues. Mr Roll, Mr Godeseth, and
even in his own way Mr Parker (though not in the way that Mr Parker himself would have intended)
have all provided clear evidence of the problems with the Horizon system, the bugs, errors and
defects within both Legacy Horizon and Horizon Online in its HNG-X form, the way that these
problems were (or were not) dealt with, and the way that Fujitsu had powers which, until shortly
before the trial started, Fujitsu sought to keep from the court, and may not even have fully disclosed
to the Post Office. Because the extent of these powers was kept secret in this way, the Post Office finds
itself now having made misleading public statements previously. If one looks back to an earlier case
management hearing and reconsiders how Fujitsu, through the Post Office, sought to portray the
contents and lack of importance and relevance of PEAKs and KELs, then it can be seen that there has
been a pattern of considerable defensiveness over the Horizon System. There has certainly been a
lack of transparency, and a lack of accuracy in description.
935.1 have explained above at [172] how the Post Office sought to draw a particularly glowing
reference in Fujitsu’s favour from the evidence of Mr Roll in their closing submissions. The Post Office
stated that from his evidence “....a clear picture emerged of Fujitsu as an organisation which was
thorough, professional and conscientious and which took considerable care to ensure that matters
were properly investigated and dealt with.” I have already stated that this is not a conclusion that can
be drawn from the evidence of Mr Roll. Nor is it a conclusion that, in my judgment, can sensibly be
drawn from the totality of the evidence taken together, both factual and expert, in the Horizon Issues
trial. As an example, on too many occasions are there PEAKs that conclude at one point that user
error cannot be to blame, only for a misleading closure code to be used for such a PEAK. Fujitsu
personnel routinely refer in such documents to the known existence of bugs, without this (so far as
the documents deployed in the trial are concerned) being communicated to the SPM in question in
these terms. In places there is even debate at Fujitsu shown in the documents about whether the Post
Office and/or SPMs should be told. I do not see how a thorough, professional and conscientious
organisation can have produced for disclosure in this litigation so many thousands of KELs during
2019 itself, both during and even after the trial. I reject that description; it is an inaccurate
description of Fujitsu and/or its investigative motivation.
936.1 consider, as explained in the Technical Appendix, that Legacy Horizon was not robust, and that
although Horizon Online in its HNG-X form was better than Legacy Horizon (not least, I consider,
because Riposte was no longer part of Horizon) its robustness was questionable, and did not justify
the confidence placed in it by the Post Office in terms of its accuracy. HNG-A is a different matter, and
the experts are agreed that it is far more robust than Horizon in earlier times. On the face of the
relevant KEL, it is not possible to say whether the Drop and Go bug, number 28 in the Bug Table,
which occurred in June 2017, was HNG-X or HNG-A. However, if the latter, it is one of only two such
items in the Bug Table that relate to the existing version of the Horizon system.
937.1 turn therefore to the points identified by Males LJ in Simetra Global Assets Ltd and another
v Ikon Finance Ltd and another [2019] EWCA Civ 1413 to which I have referred at [74] above. I
have already dealt with the first point, succinctness (obviously lacking in a judgment of this length).
The second point is a requirement to consider all the evidence. I have done this, both for and against
each of the cases advanced by both the claimants and the Post Office. I have not recited every single
disputed fact as to do so would lead to an even longer judgment, and substantially delay production of
this judgment well into 2020, and which would be contrary to the overriding objective. The third point
is to identify the issues, consider the evidence in respect of the issues, and give reasons. For this trial,
RLITO000005
RLITO000005
the Horizon Issues were agreed by the parties and approved by the court; the review of the evidence
has been longer than is ideal, but I have considered it all, whether recited in this judgment, or the
Technical Appendix, or not. The fourth point is somewhat salutary, namely that fairness requires that a
judge should deal with apparently compelling evidence, where it exists, which is contrary to the
conclusion which he proposes to reach.
938.In my judgment, there is no such evidence advanced by the Post Office. It is pointless to speculate
on how such compelling evidence might have been presented, if there had been any. The evidence that
has been advanced for the individual occurrences, for example by Mrs Van Den Bogerd, can be seen
once tested in cross-examination to amount to unfounded assertion and, sometimes, an expressed
desire, even in 2019, to go away and do more research. This is not apparently compelling evidence to
the contrary to my conclusions. Additionally, the time for further research by the Post Office is long
past. The Post Office’s approach to evidence, even despite their considerable resources which are
being liberally deployed at considerable cost, amounts to attack and disparagement of the claimants
individually and collectively, together with the wholly unsatisfactory evidence of Fujitsu personnel
such as Mr Parker. The Post Office evidence also includes a very high level overview of Horizon by its
expert which amounts to little more than a claim that it has worked quite or very well, most of the
time. This is effectively what Dr Worden’s section 8 exercise consists of. Counsel for the Post Office at
one point put to Mr Coyne that there was more likelihood of an SPM being hit by lightning than of
having a bug, error or defect in Horizon cause a branch account loss in their branch. Mr Coyne did
not agree with that, and neither do I. Indeed, such a point wholly ignores the enormous number of
PEAKs that show that branch accounts were potentially impacted on a great many occasions.
939.Also relevant are the other points which Males LJ went on to consider, namely the importance of
contemporaneous documents. He stated the following on this subject at [48] and [49] of Simetra:
a
means of getting at the truth, not only of what was going on, but also as to the motivation and state of
mind of those concerned. That applies to documents passing between the parties, but with even
greater force to a party's internal documents including emails and instant messaging. Those tend to
1 . . . i 1 I
has become a commonplace of judgments in commercial cases where there is often extensive
disclosure to emphasise the importance of the contemporary documents. Although this cannot be
regarded as a rule of law, those documents are generally regarded as far more reliable than the oral
evidence of witnesses, still less their demeanour while giving evidence.....
other compelling considerations. It is, however, striking that the judgment in this case contains
virtually no analysis of the contemporary documents many of which appear to shed considerable light
on the nature and purpose of the critical confirmations and the way in which they were understood.”
(emphasis added)
940.1 have already explained that the subject matter of the Simetra case is very different to this one,
and I emphasise here that my analysis of the contemporaneous documents is in respect of the Horizon
Issues in the context of this case, not the Simetra case which concerned very different allegations.
Here, the categories of documents that are most illuminating in terms of specific incidents with
Horizon over the years are the very numerous PEAKs and KELs. These emanate from, and are created
RLITO000005
RLITO000005
within, Fujitsu. They are, in my judgment, a very good means of getting at the truth in this case. They
show what was going on and the type of unexplained problems that numerous SPMs were
experiencing in practice over the years, as they were reported to the SSC. They contain statements
made when Fujitsu personnel’s “guard is down and their true thoughts are plain to see”. Some of
them also record that Romec engineers, or the Post Office’s own auditors, have seen what has
occurred and ruled out user error. Notwithstanding this, Fujitsu attribute user error to what has
occurred.
941.They also contain some very telling expressions, not in the context of a Fujitsu employee giving
evidence as a Post Office witness in this trial, but at the time, with such unguarded comments as Anne
Chambers in February 2006 stating that “this problem has been around for years and affects a
number of sites most weeks” and “this appears to be a genuine loss” on another occasion (as
examples only). These entries are at odds to the publicly stated position by Fujitsu both then and
indeed later. In my judgment, the PEAKs and KELs deployed in the Horizon Issues trial, and there
were very many, are consistent with my conclusions on the evidence and the answers that I have
arrived at to the Horizon Issues. Obviously no account was taken in any of the evidence, and no
account is taken by me in this judgment, of the 5,000 KELs that Fujitsu only provided and disclosed in
September 2019, months after the trial had ended.
942.The Post Office’s internal Horizon Online Induction Training slides and accompanying notes dated
7 December 2009 form a useful snapshot of how Legacy Horizon was viewed. The purpose of HNG-X
was “to deliver a significant reduction in the total annual cost of ownership of Horizon, whilst
ensuring the system remains fit for purpose in the 21st Century.” These materials make it clear that
this programme had been identified in 2005 and was "to deliver a significant reduction in the total
annual cost of ownership of Horizon", Legacy Horizon was seen within the Post Office as being very
expensive, although its overall cost is not relevant to the Horizon Issues. The notes to these materials
state:
“However, we need to continue to manage firmly any over-expectations of the frontline that Horizon
Online will deliver improved functionality - they may see this as a missed opportunity so will not cure
all the issues and problems that users have with Horizon although where practical, and at no extra
cost, we have smoothed away a number of "rough edges".
(emphasis added)
There was no significant upgrade or improvement in functionality intended as part of HNG-X.
943.Another type of document that is illuminating in terms of the Horizon Issues are internal Fujitsu
briefing documents, and to a lesser extent also some internal Post Office emails, that show the same
approach to, or view of, Horizon within those organisations as are drawn by my conclusions. One
example relates to the so-called Ping fix. After this was brought in, the number and value of TCs fell
dramatically so far as Camelot/the lottery was concerned. This appears at [427] to [428] of the
judgment on the Common Issues, Judgment (No.3). In 2007 there had been the following value of TCs
in respect of the lottery: approximately (to 1 decimal point, which represents £100 thousand) £22.8
million; in 2008 there were £12.5 million; in 2009 there were £12.0 million; in 2010 this became £11.3
million; and in 2011 £4.5 million. In 2012, the year the Ping fix was introduced, these fell below £1
million for that year.
944.The Introduction to a Post Office document of 2011 entitled “Transaction Acknowledgements -
End to End Incident Management and Data Flow” explains the following purpose to the Ping fix:
RLITO000005
RLITO000005
“Introduction
The PING Project was part of the Back Office Efficiency Programme within Post Office Ltd.
The objective of this project was to automate a process that converts Client transactional data files
(e.g. Paystation), into a data feed to the branch, i.e. a Transaction Acknowledgement file.
The branch will then process the data electronically on Horizon Online and thus avoid non compliance:
issues and the resulting cost to the business to process queries as a result.
Please note - The term PING is an internal Post Office Ltd term and branches and helplines will know
this new functionality as ‘Transaction Acknowledgements'.” (emphasis added)
945.The compliance issues were caused by the fact that the lottery terminals were on one computer
system, namely that operated by Camelot, whereas Horizon was the system operated by the branch
that was selling the Camelot products, and there was no electronic transfer of data in to Horizon in
respect of Camelot sales. The “objective” in the Post Office document makes it clear that the Ping fix
is a required improvement to the functionality, to overcome issues caused by having two different
systems which
meant that there was (prior to the Ping fix) no automated process to convert the client transactional
data files into the data feed necessary for Horizon for the branch accounts. This has nothing to do
with correcting excessive carelessness or fault on the part of SPMs. It is, in my judgment, about
remedying a deficiency in the functionality of Horizon. That document also made it clear, because
there are express entries to this effect, that outages might mean that the system would not deal with
the matters sufficiently or accurately. A template message for SPMs is even included in the document
for when this occurs, although that template message would not inform SPMs that outages would lead
to inaccuracies. It is plain that outages would indeed lead to this, as this is mentioned in many places
in the document, and acknowledged by its authors. This information was not however, based on the
templates, to be given to SPMs.
946.A theme contained within some of the internal documents is an extreme sensitivity (seeming to
verge, on occasion, to institutional paranoia) concerning any information that may throw doubt on the
reputation of Horizon, or expose it to further scrutiny. One entry in a document that makes it clear
that the Post Office itself had already recognised this is contained in a document authored by Mrs Van
Den Bogerd, entitled “Extracts from Lessons Learned Log” and dated 11 November 2015. One entry
under “issues identified” was as follows in respect of the Post Office’s behaviour up to that date:
" Failure to be open and honest when issues arise eg roll out of Horizon, HNGx migration issues/issues
affecting few branches not seemingly publicised."
(emphasis added)
947.The Post Office’s guard may have been down when that document was written; it predates issue
of the first claim form by about 6 months. I consider that it is the type of document that Males LJ had
in mind when he spoke of a witness’ guard being down, when their true thoughts are plain to see. It
plainly records Mrs Van Den Bogerd’s view of the Post Office’s approach to Horizon when it was
written in November 2015.
948.In July 2016, following the disclosure of the Dalmellington bug, the following emails went to and
from very senior people in the Post Office. On 1 July 2016 the Chief Executive, Paula Vennells stated to
RLITO000005
RLITO000005
Alisdair Cameron and Rob Houghton, in the context of an article that had appeared on the internet
about problems with Horizon:
“Subject: The Dalmellington Error in Horizon / problemswithpol
Dear both,
This needs looking into please. https://problemswithpol.wordpress.com/2015/11/10/the-error-in-
horizon/?iframe=true&theme_preview=true
[This is a reference to an internet post about a conviction of an SPM]
Can I have a report that takes the points in order and explains them.
Tim McCormack is campaigning against PO and Horizon. I had another note from him this am which
Tom will forward, so you are both in the loop. We must take him seriously and professionally.
This particular blog is independent of Sparrow but clearly related in that it appears to present similar
challenges that were raised in the course of the scheme.
I'm most concerned that we/our suppliers appear to be very lax at handling £24k. And want to know
we've rectified all the issues raised, if they happened as Tim explains. Thanks.
Paula”
949.Mr Houghton of the Post Office sent it on to someone else on the same day, copied to Mrs Van
Den Bogerd. This stated:
“I need an urgent review and mini <taskforce> on this one. It probably needs to link up heavily with
Angela’s work as FSC are mentioned extensively - Angela cfi. I don't know how we respond to this but
can we section a few inside people to get all over it and give me/Al/ Paula evidence and
understanding.”
950.However, the same recipients also received the following message from Mr Houghton later on the
very same day:
“Can you stand down on this please? [A redacted section then follows] Any specific actions and I will
revert.
My apologies.”
951.The email chain is heavily redacted and therefore the reasoning behind this volte face within the
Post Office is not shown, nor was it explained by any Post Office witness.
952.Thus the entirely understandable initial reaction of the Chief Executive of the Post Office, that a
point by point investigation was required in respect of specific criticisms of Horizon in respect of one
particular SPM who blamed Horizon for shortfalls and discrepancies, appears to have been very
swiftly shut down and not pursued due to a decision at what must have been very high levels within
the Post Office. I can think of no justifiable reason why the Post Office, institutionally, would not want
to address the Chief Executive’s points and investigate as she initially intended, and find out for itself
the true situation of what had occurred.
953.Both the recipients of the Chief Executive’s email of 1 July 2016 above are specifically named in
the Technology Strategy Update Decision Paper of 30 January 2017 in which the following is stated
RLITO000005
RLITO000005
specifically. Mr Houghton is the Author of the paper, and Mr Al Cameron is “the Sponsor”. Part of that
document records:
“This document forms an update to the IT Strategy approved in July 2016 by the PO Board. In July we
outlined that IT was not fit for purpose , expensive and difficult to change.”
The same document also states that "We need to quickly rationalise and resolve misaligned contracts
enacted to support legacy IT. obsolescence, a lack of PO technical competence, particular focus on
Fujitsu and Accenture” .
(emphasis added)
954.This therefore means that the decision taken not to investigate further in July 2016 and to “stand
down’, in itself a surprising decision for a reputable institution to take
given all the circumstances, was taken at broadly the same time as a conclusion was reached that the
IT “was not fit for purpose”. The reference to the "lack of PO technical competence” is a concern,
given [75] of the Technical Appendix which makes it clear that currently the Post Office can introduce
new clients into Horizon without requiring Fujitsu input. The level of competence may of course have
changed since 2016. The IT Strategy, which is referred to in this Update Decision Paper, and outlined
that “the IT was not fit for purpose” and which was approved in July 2016 by the Post Office Board is,
in my judgment, entirely consistent with my conclusions on the evidence. It is not, however, consistent
with the Post Office’s response to the claimants’ pre-action letter, or the Defence in the group
litigation, or the evidence adduced by the Post Office in the Horizon Issues trial. The Post Office’s
solicitors’ response to the pre-action letter is dated 28 July 2016, very lengthy, and states that “the
investigations to date have consistently pointed towards human error or dishonest conduct in
branches as the most likely cause of shortfalls.”
955.Nor is it consistent with the way the Post Office explained, in its oral opening submissions, that
the Post Office had been investigating incidents earlier in terms of bugs. So far as Dalmellington is
concerned, the oral submissions were as follows:
“....even the documents that your Lordship saw this morning with the Dalmellington bug, your
Lordship will see the rigour that's applied. There's concern that postmasters aren't given advice that
might be incorrect. The rigour associated with that process and the determination of Fujitsu and the
other people involved, the other stakeholders, to get to the bottom of what happened is quite striking
in my submission.”
956.Dalmellington was investigated in 2015, and the outcome of that investigation is available to this
court. The outcome of that investigation is far from favourable to the Post Office’s position on bugs,
errors and defects, and the robustness of Horizon. The oral submission that the Dalmellington bug
was investigated rigorously (which it was) does not easily sit with the documents showing that one
year later the Post Office specifically decided not to investigate other incidents at all, and to “stand
down” those tasked with investigating it. This is notwithstanding that the Chief Executive’s initial (and
in my judgment entirely proper, if not the only proper) reaction was that those allegations needed to
be looked at “seriously and professionally”, that an investigation was needed, and that an urgent
review and mini task force was required. Nor does it sit easily with the way that other bugs that were
present in both Legacy Horizon and Horizon Online were investigated by Fujitsu.
957.Further, a conclusion in terms that the IT is “not fit for purpose” is not something that would have
been reached lightly in an IT strategy document, or approved lightly by the Post Office Board. The
RLITO000005
RLITO000005
Post Office Board is a serious level within the organisation. The Board are not likely to be involved in,
nor to have brought to their attention, matters that are anything other than serious, considered and
fully researched. This conclusion in this internal contemporaneous document has not driven my
conclusions on the evidence in the slightest. Rather, it is the other way around. I have reached my
conclusions on the evidence independent of contemporaneous references such as this one, but then
tested my evidential conclusions against these to see if there is a contradiction. However, there is
none, and they are entirely consistent.
958.There are only very isolated Post Office documents that are inconsistent with my conclusions. I
shall deal with only one. On 17 August 2015 the BBC Panorama programme broadcast a programme
which contained allegations about Horizon. The contemporaneous response to that by the Post Office
in its statement to SPMs started with the following passages, which I shall reproduce. The bold
emphasis was present in the original:
“The Post Office wholly rejects extremely serious allegations repeated in BBC’s Panorama programme
of 17 August 2015. The allegations are based on partial, selective and misleading information.
The Post Office does not prosecute people for making innocent mistakes and never has There is no
evidence that faults with the computer system caused money to go missing at these Post Office
branches There is evidence that user actions, including dishonest conduct, were responsible for
missing money”
959.The first of those bullets point is for another day, and is not for this court in any event, for the
reasons explained at [60] to [66] above. This judgment deals with the detailed background and
technical evidence behind the second bullet point. The confidence expressed by the Post Office in
2015, in that second bullet point, in the computer system and its accuracy is not consistent with the
evidence I have accepted, the expert evidence on the number of bugs in the Horizon system (both
Legacy Horizon and HNG-X) and my findings. This Post Office document was however drafted for
public consumption, and this must explain the inconsistency. To the extent that the second bullet point
conflicts with my findings in this judgment, that public statement in 2015 by the Post Office is
factually incorrect. This is not to make any findings on any individual claim, or on the cause of money
said to be missing in the accounts of any particular branch Post Office or individual claimant SPM.
960.In my judgment, a number of both the Post Office’s own internal documents, and Fujitsu ones too,
namely those that were not drafted for public consumption, plainly support my conclusions on the
evidence. Further, certain matters that have emerged in the Horizon Issues trial - such as discussions
within Fujitsu itself as to whether the Post Office should be told certain detrimental information about
the Horizon system, and the Post Office’s own decision at the highest level not to investigate certain
matters as recently as 2016 - are of great concern. The Post Office has gone to great lengths over the
years, and spent a great deal of time and a huge amount of public money, in defending the
performance of Horizon. It is also the case that the Post Office must have been reliant on Fujitsu to a
certain degree in terms of being provided with accurate information of a technical nature. That is not
only obvious from the evidence, but has also been agreed by the experts in the 3rd Joint Statement.
That accuracy from Fujitsu has not always been available, as demonstrated by this judgment.
961.However, regardless of that - and this judgment does not deal with who, if anyone, at the Post
Office knew precisely what about Horizon, and when, as these are not part of the Horizon Issues - in
my judgment a number of the specific internal contemporaneous documents, rather than being
contrary to my conclusions on the evidence, are entirely consistent with those conclusions. I do not
know if the Post
RLITO000005
RLITO000005
Office Board, who were told in July 2016 that its own IT was not fit for purpose, will be particularly
surprised at the findings on the expert evidence in the Bug Table, and the number of admitted bugs. It
does not much matter, for the purposes of resolving the Horizon Issues, whether they are surprised or
not.
962.The experts’ agreements in particular have been of great assistance, but everything has been
considered. This judgment, together with the Technical Appendix, is of substantial length, and to
recite everything that has been deployed by both sides, in terms of evidence, submission and
reference to all contemporaneous documents, is neither necessary nor desirable.
963.It must also be remembered that Horizon as it is today, or at least in the last couple of years since
it became HNG-A, is a very different system to earlier times. Legacy Horizon ceased to be used in
2010. Modern Horizon Online as it is today is not the same as the online system that was introduced
in 2010; it is different and more robust, on the agreed expert evidence, to the system as it was even
when the litigation started with the issue of the first claim form on 11 April 2016. It became HNG-A in
2017, rather than HNG-X, and runs on a completely different Windows platform. The dates of the bugs
in the Bug Table are (between them collectively) from 2000 to 2018, although there are far fewer in
recent years. Having said that, some - such as Bug 8, Recovery Issues, and Bug 28, Drop and Go, have
been experienced up to 2018 and 2017 respectively. The latter is a reference data bug and hence
easily fixed. The
Technical Appendix at [421] provides a summary of the different bugs from Legacy Horizon and
Horizon Online. It can be seen that there were more in Legacy Horizon than HNG-X in any event, and
far fewer in HNG-A. In any case, the cut-off date for claims to be issued in the group litigation is 24
November 2017, as extended by paragraph 15 of the 1st Case Management Order sealed on 27
October 2017.
964.The Post Office has been very concerned as to the outcome of this litigation on its wider business.
It described the group litigation as long ago as its Opening Submissions in November 2018 for the
Common Issues trial as representing “an existential threat” to that business. I do not consider that
these answers to the Horizon Issues represent a significant threat to the Post Office’s entire business.
Findings in this judgment as to the performance and robustness of Legacy Horizon from 2000 to 2010,
and then of Horizon Online (in both its forms, HNG-X and HNG-A) from 2010 to 2018 are not findings
on the Horizon system as it exists at the date of this judgment. These findings cannot be routinely
applied to the way that HNG-A operates as at December 2019. It is agreed by the experts that the
Horizon System in its HNGA form is now relatively robust. This judgment is a historical analysis of the
Horizon System as it relates to the period in question in the group litigation, not a judgment upon
Horizon HNG-A as it is today.
M. Answers to the Horizon Issues
965.This section is intended as a very brief summary of what has gone before, both in this judgment
and also the Technical Appendix. It does not replace the previous 964 paragraphs or that appendix.
The detailed passages themselves should be consulted for a more detailed consideration of the issues.
966.1 therefore turn to the answers to the Horizon Issues themselves. For convenience, I shall
reproduce the specific issue above each answer, in order to avoid the reader having constantly to turn
back some hundreds of pages to remind themselves. I shall also retain the headings and sub-headings.
The wording of the Horizon Issues was agreed by the parties and approved by the court.
RLITO000005
RLITO000005
BUGS, ERRORS AND DEFECTS IN HORIZON
Accuracy and integrity of data
967.Issue (1): To what extent was it possible or likely for bugs, errors or defects of the nature alleged
at §§23 and 24 of the GPOC and referred to in §§49 to 56 of the Generic Defence to have the potential
(a) to cause apparent or alleged discrepancies or shortfalls relating to Subpostmasters’ branch
accounts or transactions, or (b) undermine the reliability of Horizon accurately to process and to
record transactions as alleged at §24.1 GPOC?
968.Answer: It was possible for bugs, errors or defects of the nature alleged by the claimants to have
the potential both (a) to cause apparent or alleged discrepancies or shortfalls relating to
Subpostmasters’ branch accounts or transactions, and also (b) to undermine the reliability of Horizon
accurately to process and to record transactions as alleged by the claimants.
969.Further, all the evidence in the Horizon Issues trial shows not only was there the potential for this
to occur, but it actually has happened, and on numerous occasions. This applies both to Legacy
Horizon and also Horizon Online. It has happened under both the HNG-X and HNG-A iterations of the
Online system, but far less frequently under the latter than the former. Indeed, there are only isolated
instances of it happening in respect of HNG-A, which the experts agree is a better system than either
of the other two iterations of Horizon.
970.I accept the claimants’ submissions that, in terms of likelihood, there was a significant and
material risk on occasion of branch accounts being affected in the way alleged by the claimants by
bugs, errors and defects. This is amply demonstrated by Dr Worden’s evidence. He accepted that
there was strong evidence of at least 12 bugs causing a lasting discrepancy in branch accounts. This
conclusion was reached even within the context of his unjustified creation of both “transient” and
lasting impact, which limited his consideration of this, and is an approach which I have rejected. This
number of bugs was very much smaller than the number contended for by Mr Coyne, and the number
which I have found in Part E of the Technical Appendix, but even on that smaller number, was itself
significant. Dr Worden’s reliance upon TCs as a countermeasure led to a failure by him to follow the
agreed wording of the Horizon Issues, as TCs are not part of the Horizon system. Given TCs are part
of the way that the Post Office corrects the branch accounts (and branch accounting statements)
produced by Horizon, the only way that a fully precise answer could be given for the number of times
such impacts have occurred would be if the figures were available for the years of operation of
Horizon, demonstrating how many TCs each year had to be issued in respect of the effects on branch
accounts of such discrepancies. Figures
would also be required of the number of TCs that were upheld when disputed by SPMs.
971.However, those figures are simply not available for the majority of the life of both Legacy Horizon
and Horizon Online. Nor is there any data centrally recorded by Fujitsu of the number of bugs, errors
and defects which were discovered by dissatisfied SPMs reporting incidents to Fujitsu, with
percentage outcomes of what the investigations by the SSC led to, in terms of result. Such data as
there is, namely Mr Parker’s summation of closure codes of PEAKs, is wholly unreliable and I have
rejected it. Nor are the closure codes that were used by Fujitsu accurate in any event. The evidence of
fact was that full and accurate records of this nature are simply not kept by Fujitsu. Had they been,
both experts could (and probably would) have used such data as a starting point for their
investigations. The sheer scale of the number of TCs issued by the Post Office each year - which is
over one hundred thousand for many of the years that are the subject of the group litigation -
RLITO000005
RLITO000005
supports my conclusion that there was a significant and material risk of inaccuracy in branch
accounts as a result of bugs, errors and defects in the Horizon System (both Legacy Horizon and
HNG-X).
972.Issue (2): Did the Horizon IT system itself alert Subpostmasters of such bugs, errors or defects as
described in (1) above and if so how?
973.Answer: Although the experts were agreed that the extent to which any IT system can
automatically alert its users to bugs within the system itself is necessarily limited, and although
Horizon has automated checks which would detect certain bugs, they were also agreed that there are
types of bugs which would not be detected by such checks. Indeed, the evidence showed that some
bugs lay undiscovered in the Horizon system for years. This issue is very easy, therefore, to answer.
The correct answer is very short. The answer to Issue 12 is “No, the Horizon system did not alert
SPMs”. The second part of the issue does not therefore arise.
974.Issue (3): To what extent and in what respects is the Horizon System “robust” and extremely
unlikely to be the cause of shortfalls in branches?
975.Answer: This issue or question is in two parts. I shall deal with them sequentially. The first is the
extent and in what respects the Horizon system is “robust”. The experts are agreed that HNG-A is
relatively robust, and I have found that the system as it is in 2019 is far more robust than it was prior
to 2017. The Technical Appendix in general, and [444] of that in particular, should be consulted for a
more complete answer on robustness. However, the Horizon Issues considered the whole life of the
system from 2000 to date. In summary terms only, Legacy Horizon was not remotely robust. The
number, extent and type of impact of the numerous bugs, errors and defects that I have found in
Legacy Horizon makes this clear. Further, Dr Worden considered that TCs were an essential
countermeasure. If TCs are excluded from consideration of the Horizon system, which they must be,
then the answer becomes even more stark, even on the evidence of the Post Office’s own expert.
976.HNG-X, the first iteration of Horizon Online, was slightly more robust than Legacy Horizon, but
still had a significant number of bugs, errors and defects. This was particularly so in the years 2010 to
2015, but also after that. This is clearly demonstrated from the findings in the Technical Appendix.
The functionality of
Legacy Horizon was not therefore materially improved when the system moved to the online version,
although one of the major differences between Legacy Horizon and HNG-X was the former’s use of
Riposte (which was not used at all in HNG-X). I find that the robustness of HNG-X was questionable
and did not justify the confidence routinely stated by the Post Office (prior to February 2017) in its
accuracy. HNG-A is far more robust than either of the previous two iterations of the system. I have
taken into account, in reaching these conclusions, not only the evidence of bugs, errors and defects,
but also the lack of records or logs at Fujitsu over time over the use and controls of the powerful
access roles permitted to Fujitsu personnel. This deficiency in reporting and controls seems to have
been corrected in about 2017. This adds to my conclusion (and that of the experts) that HNG-A is
more robust than HNG-X. As the experts agreed, this increase in robustness has, in part, developed
from Post Office discovering bugs, errors and defects in live use and then applying fixes and
improving monitoring. This is important, as it means that the robustness today of HNG-A cannot be
relied upon by the Post Office in justifying, or being equated to, robustness in HNG-X. The same point
applies to HNG-X and Legacy Horizon.
RLITO000005
RLITO000005
977.1 accept Mr Coyne’s evidence that there are far more TCs than expected even now, compared
with comparable systems in the banking and finance sectors. However, it may be that this is the only
way in which it can now sensibly be operated. TCs are outside the scope of the definition of the
Horizon system in any event. The experts are agreed that HNG-A is robust, and far better than it was
in either its Legacy Horizon or HNG-X form.
978.Turning to the second part of the issue, namely to what extent and in what respects is the
Horizon system extremely unlikely to be the cause of shortfalls in branches. I accept that this
presupposes a shortfall in a branch, and then requires consideration of whether Horizon is extremely
unlikely to be cause of that shortfall. By shortfall, this issue means an unexplained discrepancy of the
type explained by the claimants’ witnesses, and also recorded in numerous PEAKs. In my judgment,
there is a material risk that such a shortfall in a branch’s accounts was caused by the Horizon system
during the years when both Legacy Horizon and HNG-X were in use, which is 2000 to 2010 and 2010
to 2017 respectively. For each of those iterations of the system, the Horizon system was therefore not
“extremely unlikely” to be the cause of shortfalls in branches, as contended for by the Post Office. I
realise that is a double negative, but I consider wording the answers as closely as possible to the
agreed issues is likely to be of greater assistance to the parties.
Controls and measures for preventing / fixing bugs and developing the system
979.Issue (4): To what extent has there been potential for errors in data recorded within Horizon to
arise in (a) data entry, (b) transfer or (c) processing of data in Horizon?
980.Answer: The experts were agreed that there are a number of actual reported errors in data
recorded within Horizon arising from each and all of (a) data entry, (b) transfer or (c) processing of
data. Therefore, not only does the potential for these errors exist, but this has in fact happened, and
this has led to financial discrepancies in branch accounts. The experts were also agreed that bugs,
errors and defects identified in relation to Horizon Issue 1 were often relevant to Horizon Issue 4 in
that they are ultimately errors arising from the processing of data in Horizon.
981.The “extent” to which this has occurred has not been fully assessed by the experts. However, Dr
Worden considered this issue to be a subset of Horizon Issue 3, in respect of which I have effectively
or substantially accepted the claimants’ case, at least so far as Legacy Horizon and HNG-X are
concerned.
982.The experts do not agree as to its ‘extent’ in terms of actual effect, although that is not what the
issue asks. It was agreed by the experts that there was evidence within the PEAKs and KELs of bugs/
errors/defects within Horizon arising from parts (a), (b) and (c) of this issue that occurred that caused
financial discrepancies, as well as some that occurred without causing financial discrepancies. It was
also agreed that reference data is critical to the operation of Horizon. I have explained in the
Technical Appendix what reference data is, and the important part it plays both in the software and
operation of the system. It is agreed by the experts that errors in reference data have led to
discrepancies in branch accounts. It has also been agreed by the experts that errors in third-party
data have led to discrepancies in branch accounts. The Post Office does not have control over errors
in third-party data, but given these have an impact upon branch accounts, unless they are corrected,
then those errors will also have an adverse effect upon a SPM’s branch accounts.
983.I do not consider that the extent of the actual effect can be fully answered in precise
mathematical terms on the evidence as it is. This is not least because in order to do so, the experts
would have to consider the full data set of KELs, and several thousand KELs were produced by Fujitsu
RLITO000005
RLITO000005
during, and well after, the trial. However, a precise mathematical answer is not required. The answer
to the issue as posed is that there is a material risk for errors in data recorded within Horizon to arise
in (a) data entry, (b) transfer or (c) processing of data in Horizon in both the Legacy Horizon and
HNG-X forms.
984.Issue (5): How, if at all, does the Horizon system itself compare transaction data recorded by
Horizon against transaction data from sources outside of Horizon?
985.Answer: This is largely automated. It has not however been automated throughout the whole life
of Horizon from 2000, as shown by the Ping fix in respect of the Lottery/Camelot. The experts’
agreement on reconciliation states inter alia that “Reconciliation between transactions recorded on
Horizon and transactions recorded by Post Office's clients is largely automated. Detected
discrepancies were subject to manual corrective fixes and/or the issue of Transaction Corrections/
Error Notices to the” SPMs. What this means is that if the data in Horizon was wrong (due to bugs,
errors or defects within Horizon) and did not reconcile with third-party data, then “manual corrective
fixes” and the issuing of TCs would be undertaken, which would not be automatic.
986.Some comparison of transaction data from some clients of the Post Office is done within the
Horizon system and other client data is compared outside the Horizon system. Again, the evidence did
not deal in a comprehensive way with each such method for every one of the Post Office’s many
hundreds of clients. Nor do I consider that such evidence would be required properly to answer this
issue.
987.Issue (6): To what extent did measures and/or controls that existed in Horizon prevent, detect,
identify, report or reduce to an extremely low level of risk the following:
a.data entry errors;
b.data packet or system level errors (including data processing, effecting, and recording the same);
c.a failure to detect, correct and remedy software coding errors or bugs;
d.errors in the transmission, replication and storage of transaction record data; and
e.the data stored in the central data centre not being an accurate record of transactions entered on
branch terminals?
988.Answer: The experts were agreed that there were many measures and controls within Horizon
that existed to prevent, detect, identify report or reduce the risk of varying errors. However, the
experts were also agreed that whilst Horizon contains measures and controls for detecting system
integrity concerns, these automatic mechanisms have been shown to have failed in the past. The
experts did not agree as to the ‘extent’ of prevention, detection, identification, reporting or risk
reduction that the automatic and manual control measures delivered. There were numerous specific
matters explored in some detail in the trial that clearly demonstrated such failures, such as the
Callendar Square bug, the Receipts and Payments Mismatch bug, gaps and duplicates in entries in the
audit store, keying and mis-keying errors (which were referred to in different PEAKs) and fixes which
introduced other issues. The doubling of the foreign currency discrepancy that was put to Mr
Godeseth was a specifically stark example, even though Fujitsu - inexplicably in my judgment - chose
to attribute this to user error.
989.1 do not consider that the measures and/or controls that existed in Legacy Horizon, or HNG-X
prevented, detected, identified, reported or reduced to an extremely low level of risk any of the sub-
RLITO000005
RLITO000005
items at (a) to (e). Indeed, I reject the Post Office’s case that there was such an extremely low level of
risk. That is, in my judgment, a wholly complacent and unjustified position that has existed for many
years, based on my findings in the Technical Appendix.
OPERATION OF HORIZON
Remote Access
990.Issue (7): Were Post Office and/or Fujitsu able to access transaction data recorded by Horizon
remotely (i.e. not from within a branch)?
991.Answer: Yes. This answer is substantially agreed. Both the Post Office and Fujitsu could read data
remotely. Fujitsu could access all transaction data recorded by
Horizon. The experts agreed that Fujitsu needed remote access for support purposes. Fujistu has
direct access to the branch database. In Legacy Horizon, this was the messagestores held on the
correspondence servers; in Horizon Online, it is the BRDB. In Horizon Online, Fujitsu can also
download logs stored on the counters. The Post Office has access to transaction data which is copied
to its Management Information System or MIS. In Horizon Online, the Post Office has read-only access
to branch data on the BRDB which includes the levels of cash at a particular branch.
Availability of Information and Report Writing
992.Issue (8): What transaction data and reporting functions were available through Horizon to Post
Office for identifying the occurrence of alleged shortfalls and the causes of alleged shortfalls in
branches, including whether they were caused by bugs, errors and/or defects in the Horizon system?
993.Answer: This issue is substantially agreed. The Post Office had access to data and systems that
were not available to SPMs. The role of the Transaction Processing System (or TPS) is to gather the
transactions taking place in the branches, and to pass them on both to Horizon’s back-end systems
such as APS and DRS. These would also be passed on to other IT systems in the Post Office, such as
Credence and a succession of systems based on SAP, which have finally culminated in what is called
POLSAP. The Post Office therefore has a range of Horizon reports and its own management
information systems (MIS). Dr Worden stated that he expected the MIS to include analytical facilities
which were not required by, and so were not available to, Subpostmasters. None of this was
controversial by the end of the trial, or in issue between the experts.
994.The Post Office would also be able to use standard database reporting tools to retrieve and
analyse information about branch transactions. It could also obtain audit data as explained in Part K
of this judgment. A request had to be made to Fujitsu for this audit data, however. The Post Office
could not simply access such audit data itself without recourse to Fujitsu, who had a charging
structure in place in respect of such requests.
995.Importantly in my judgment, the only source of actual key stroke information (in the sense of
what buttons had been pressed in the branch) was to be found within the audit data. Credence does
not provide this. I also consider that although the Post Office did have access to “the causes of alleged
shortfalls in branches, including whether they were caused by bugs, errors and/or defects in the
Horizon system”, this had to be obtained through Fujitsu. This should be recorded for completeness.
The evidence does not show the Post Office IT department either being capable of investigating
directly itself, or if it was capable of doing so, actually undertaking this. All investigations recorded in
PEAKs and KELs show that Fujitsu did this. The terms of the experts’ agreement in this respect was
RLITO000005
RLITO000005
that the Post Office was “reliant upon Fujitsu for diagnosis of whether the occurrence of shortfalls was
caused by bugs/errors or defects within the Horizon system.” I find that the Post Office plainly was
reliant upon Fujitsu in this way.
996. Issue (9): At all material times, what transaction data and reporting functions (if any) were
available through Horizon to Subpostmasters for:
a.identifying apparent or alleged discrepancies and shortfalls and/or the causes of the same; and
b.accessing and identifying transactions recorded on Horizon?
997.Answer: The answer to this Issue is also substantially agreed. The experts agreed that the causes
of some types of apparent or alleged discrepancies and shortfalls may be identified from reports or
transaction data available to SPMs. Other causes of apparent or alleged discrepancies and shortfalls
may be more difficult, or impossible, to identify from reports or transaction data available to the
SPMs, because of their
limited knowledge of the complex back-end systems. I would add that the subject matter of every
PEAK referred to in the Bug Table, and the sort of problems that initiated the creation of each of those
PEAKs, plainly could not be investigated by the SPM or SPMs in question in each case on the material
that Horizon made available to those SPMs.
998.In Judgment No.3 I had identified certain features or characteristics of both the SPMC and the
NTC, the contracts under which SPMs were engaged (dependent upon the dates of their engagement).
The experts were not aware of these contractual findings when they reached any of their expert
agreements, as that judgment was not provided even in draft until just before the Horizon Issues trial
started. It was handed down formally at the end of the first week of factual evidence. It is of interest, I
consider, that the two experts considered at least the essence of one of the implied terms (although
not in contractual terms) in their agreement that “identification of the causes of apparent or alleged
discrepancies required the co-operation of PO staff and SPMs”. Co-operation is an important part of
any investigation, as it is an important part of the contractual relationship between the Post Office and
SPMs.
999.SPMs did have access to many reports relating to the different aspects of running a branch (such
as currency exchange, stock and so on). They also had access to transaction logs that would show
individual transactions, and event logs, but these were only available for a few weeks (initially this
was for 6 weeks or 42 days, then this changed to 60 days). It was possible for a SPM to print these out
on paper till rolls. There was evidence in the Common Issues trial from both claimant SPMs and Post
Office witnesses about how unwieldy and non-user friendly this was. Indeed, in that earlier trial one of
the Post Office witnesses used the size of the Super Court in the Rolls Building (where both the
Common Issues trial and the Horizon Issues trial took place) graphically to demonstrate how long
such a till roll would be even just for a day in a busy branch.
1000.Because the reports and data available to SPMs were so limited, their ability to investigate was
itself similarly limited. The expert agreement to which I refer at [998] above makes it clear in IT terms
(based on the transaction data and reporting functions available to SPMs) that SPMs simply could not
identify apparent or alleged discrepancies and shortfalls, their causes, nor access or properly identify
transactions recorded on Horizon, themselves. They required the co-operation of the Post Office.
Access to and/or Editing of Transactions and Branch Accounts
RLITO000005
RLITO000005
1001.Issue (10): Whether the Defendant and/or Fujitsu have had the ability/facility to (i) insert, inject,
edit or delete transaction data or data in branch accounts; (ii) implement fixes in Horizon that had the
potential to affect transaction data or data in branch accounts; or (iii) rebuild branch transaction data:
a.at all;
b.without the knowledge of the Subpostmaster in question; and
c.without the consent of the Subpostmaster in question.
1002.Answer: I shall answer this by reference firstly to Fujitsu, and then to the Post Office. 1003.
Fujitsu: The answer is “yes” to each of Issue 10(i), 10(ii) and 10(iii), and to each of (a), (b) and (c) too.
There are nine different questions posed (each of (a), (b) and (c), posed after the answer to each of (i),
(ii) and (iii) if that first answer is “yes”) and the answer is “yes” regarding Fujitsu to all nine
permutations. The expressions used in the evidence that permitted them to do this included those in
the SSC role utilising various tools; those with the APPSUP role; those exercising Privileged User
rights; and those exercising Database Administration rights. Mr Godeseth also referred to the “guys in
Ireland” who had access to a very “low level”, by which he meant very powerful access (as in, they
could go to a very deep or low level within the system). This arose in re-examination and hence was
not pursued. There is no way of knowing if these Fujitsu personnel, who had the role that is called
“UNIX user”, were included within the descriptions I have already given. If not, then UNIX user
should be added to that list.
1004.The injecting of transactions into branch accounts by Fujitsu, and the visibility of these (as in
whether the SPM could tell these had been done in this way, due to the use of a counter position
greater than 32) was very controversial prior to the trial. By the time the Fujitsu witnesses were
called, they had accepted that what Mr Roll had said was correct, and Mr Godeseth confirmed this
would look as though the SPM had done it.
1005.The Post Office: this could only be done by the Post Office utilising its Global User role. This
permitted the Post Office to inject transactions without the knowledge or consent of the SPM.
However, it could only be done by someone who was physically present in the branch, as that is the
only way Global User could be used by the Post Office. The answer therefore for the Post Office is
“yes” only in respect of that part (relating to injecting transactions) of (i)(a) (b) and (c). With that
exception, the answer for the Post Office is “no” to the other eight permutations. The qualifying
statement has to be made that whoever at the Post Office was exercising Global User rights would
have to be physically present in the branch. This would therefore be likely to mean that a SPM or their
assistant would at least know someone was in their branch doing something in terms of access to
their branch accounts. A description of “Global User” is provided at [674] above. The Post Office’s
contended for answer to this Issue was “No” to all nine permutations so far as the Post Office (rather
than Fujitsu) is concerned. This is wrong, and ignores the existence of the Global User role.
1006.The Balancing Transaction Tool, and how many times it was used, occupied a certain amount of
time at the trial, as its existence had arisen when the experts were preparing their reports and had
led to inter-solicitor correspondence. The majority of the factual evidence on this is dealt with in the
section of this judgment dealing with Mr Godeseth’s evidence. This tool enabled Fujitsu specifically to
insert a transaction into branch accounts. The one documented occasion of its use was done with the
consent of the SPM in question. However, in my judgment, this issue was a sideshow. This is because
the design intent in the Low Level Design document in relation to this tool had plainly been overtaken,
over time, as the numerous other times it was used showed, based on the messages in fact inserted.
RLITO000005
RLITO000005
Secondly, and more importantly, the very powerful roles identified at [1003] above enable those with
such roles to do far more than someone simply using the Tool.
1007.Issue (11): If they did, did the Horizon system have any permission controls upon the use of the
above facility, and did the system maintain a log of such actions and such permission controls?
1008.Answer: Given the answers to Issue 10 above, this issue does arise and requires answering.
There were permission controls, but the roles were very wide and in my judgment they were not
controlled. This included but is not limited to the lack of any proper logs. Although at an earlier stage
in the trial, the Balancing Transaction tool was focused upon, it emerged that there were other roles
and tools available to Fujitsu in addition to the Balancing Transaction tool which enabled Fujitsu to
achieve the same result, namely injection and alteration of transaction data. Indeed, not only were
these roles (such as APPSUP and the other roles identified in the answer to Issue 10 above) more
powerful, but they were widely available and in my judgment effectively unaudited. The whole of SSC
had the APPSUP role for many years, and internal Fujitsu documents recorded that they were not
supposed to have that role. There were a large number of personnel within SSC. Whilst some
Privileged User Logs do exist, these logs did not exist prior to 2009 at all. Prior to July 2015, they only
recorded when a user logged on and logged off, with no recording of what they were doing. This is a
plainly inadequate record. Even after July 2015, the experts agree that such logs are not a useful
source of evidence about remote access, due to their content (more accurately, this should be their
lack of content). These deficiencies in the control environment were identified as long ago as before
the audit report by E&Y in 2011, such that the deficiencies were recorded as being in existence in
2011. Even so, this was not promptly corrected.
1009.0CPs do evidence documentation of remote alteration of transactional and other branch data.
However, on the evidence available at the trial, these OCPs constitute what could be described as a
piecemeal and ad hoc process, and a not insubstantial percentage were issued retrospectively.
Further, in many cases there is evidence that the person raising the OCP was the person who was to
monitor it. In my judgment, and given the powerful nature of the roles, this method of control is
inadequate.
1010.The view of Dr Worden, the Post Office’s own IT expert, who as I have identified is a man of
considerable experience in the IT world stretching over decades, is that the number of Privileged
Users and SSC users who can create a Balancing Transaction seems high. The lack of control of these
roles, the recording of access, the deficiencies in the logs, and the concerns expressed by E&Y,
demonstrate in my judgment an inadequacy of controls. Roles were not properly controlled even after
concerns were raised by E&Y in the 2011 Audit, particularly in respect of the APPSUP role.
1011.The answer therefore is that there were permission controls; they were however inadequate.
Logs were maintained but only after 2009. From 2009 to July 2015 they did not record actions, only
whether a user had logged on and logged off. Even after July 2015, the experts agree that such logs
are not a useful source of evidence about remote access, due to their content (more accurately, this
should be their lack of content). In my judgment, this amounts to a deficiency in controls.
1012.Issue (12): If the Defendant and/or Fujitsu did have such ability, how often was that used, if at
all?
1013.Answer: Due to the nature of Fujitsu’s record keeping, the experts could not provide any
confident evidence on this subject of frequency. Mr Coyne established that the APPSUP role was used
2,175 times between 2009 and 2019, Dr Worden confirmed that this was consistent with his review of
RLITO000005
RLITO000005
the Privileged User logs and he equated this to about one person using this role each working day
during that period. The Fujitsu witnesses stated this would be in a “support role” but very little detail
is available about what this was, or why such regular access of someone with such a powerful role
was required. In any event, the Fujitsu witnesses of fact had not reviewed the Privilege User Logs in
any detail.
1014.Indeed, the lack of evidence provided in respect of this issue supports and corroborates my
conclusions on Horizon Issue 11 which precedes it. The inability of the Post Office, and/or the experts
for both parties, to be able to provide a clear and precise answer to what is, in essence, a very simple
question (even though the actual answer might be a large number) directly arises from Fujitsu's
plainly inadequate records. On the evidence from the two experts, the answer should be in
approximate terms only and would be about once a day over the whole life of the system, to the extent
that any records are available.
1015.Issue (13): To what extent did use of any such facility have the potential to affect the reliability
of the Branches’ accounting positions?
1016.Answer: The design abilities of the roles to which I have referred was very wide - as a single
example only, the experts agreed that anyone with the APPSUP role could pretty much do whatever
they wanted. The answer to this issue is therefore that these facilities had the potential to affect the
reliability of a SPM’s branch accounts to a material extent. Further, the evidence shows clearly that
there were instances when this in fact occurred, which goes wider than the issue posed (which asks
about potential) but which I find is also relevant to the Horizon Issues as a whole.
Branch trading statements, making good and disputing shortfalls
1017.Before I turn to Horizon Issues 14 and 15, there is a point that is relevant to the group litigation
as a whole that can conveniently be made at this stage. Firstly, the answers to both of the remaining
two Horizon Issues are largely agreed by the experts. This means, almost by definition, that the
subject matter of those issues ought not to be controversial.
1018.However, the fact that a substantial part of Issue 14, in particular, was so controversial for so
long is, in my judgment, notable. The Post Office ought to have known how its own system works. It is
agreed by the experts that an SPM cannot record a dispute on Horizon at all. Indeed, this point was so
very clearly conceded by Mr Godeseth when he was asked about it in cross examination that it was
remarkable. He explained that there was no “dispute” button on Horizon, and that this was something
expressly and deliberately done by design. Yet there were two substantial points in issue in the
Common Issues trial on this very same subject. One was the ability of a SPM to dispute the figures
shown on Horizon. The second was whether, in law, a branch trading statement had the effect of what.
is called a settled account between agent and principal (which would mean that certain legal
consequences would flow if it were). Judgment No.3 found in favour of the claimants, and against the
Post Office, on both those points. It is therefore more than a considerable surprise to see that, as a
matter of evidence in the later Horizon Issues trial, those points should never have been put in issue
by the Post Office at all.
1019.In complex litigation, it is important that points that are not, in reality, contentious, are not put
in issue if there is no proper factual basis for doing so. If, as a matter of reality, any particular point is
not controversial, putting it in issue will simply increase time and costs. A SPM cannot be criticised
for “accepting” an entry in their accounts if they have no choice but to accept that entry. Nor can a
branch trading statement be given the status of a freely agreed (and therefore settled) account if it is
RLITO000005
RLITO000005
no such thing. I wish to remind the parties to this litigation that points that should not be in issue
ought to be conceded at a sufficiently early stage, such that time and legal costs are not spent in
argument on an erroneous factual basis.
1020.Issue (14): How (if at all) does the Horizon system and its functionality:
a.enable Subpostmasters to compare the stock and cash in branch against the stock and cash
indicated on Horizon?
b.enable or require Subpostmasters to decide how to deal with, dispute, accept or make good alleged
discrepancy by (i) providing his or her own personal funds or (ii) settling centrally?
c.record and reflect the consequence of raising a dispute on an alleged discrepancy, on Horizon
Branch account data and, in particular:
(i)does raising a dispute with the helpline cause a block to be placed on the value of an alleged
shortfall; and
(ii)is that recorded on the Horizon system as a debt due to Post Office?
d.enable Subpostmasters to produce (i) Cash Account before 2005 and (ii) Branch Trading Statement
after 2005?
e.enable or require Subpostmasters to continue to trade if they did not complete a Branch Trading
Statement; and if so, on what basis and with what consequences on the Horizon system?
1021.Answer: This is substantially agreed by the experts. The Horizon system can compare cash and
stock figures, but it has no facility to record a dispute. The comparison is done by the system
comparing its electronic records of what cash and stock is held in a branch, with the corresponding
figures inputted by the SPM at the end of a trading period.
1022.The functionality of the system is as follows. The SPM inputs, at the end of each trading period,
the relevant figures for the cash and stock that are physically present. The Horizon system identifies
any discrepancy and the system invites the SPM to transfer the amount of that discrepancy into the
local suspense account and continue “to roll over”, or to discontinue this operation. Rolling over is
required before the next trading period can commence. For discrepancies of less than £150, the SPM
has to “make good” that amount, by adding money to the till (if it is negative) or removing it (if
positive). For negative sums in excess of £150, the SPM has the choice of making good (by adding the
money) or “settling centrally” which basically just means seeking
time from the Post Office to pay. This is extensively explained in the Common Issues judgment,
Judgment No.3. This is not done as part of the Horizon system. A slightly different process was
available prior to 2005 which was called cash accounting, but the process was broadly the same,
although “settling centrally” was not available under cash accounting. Making good would cause the
actual cash position to increase by the amount put into the till from the SPM’s own resources. If, say,
there was a negative discrepancy of £100 between the actual cash position (the cash physically
present in the till), and the derived cash position (shown on Horizon) then making good would lead to
the figures being the same. The meaning of “settling centrally” simply means paying the amount to
the Post Office over time (or having it deducted by the Post Office from the remuneration paid to the
SPM).
RLITO000005
RLITO000005
1023.A SPM had to do this in order to “roll over” to the next accounting period. This means that a
branch trading statement had to be completed if the SPM wished to open the branch for the next
trading period. The branch trading statement had to be done on the figures for cash and stock shown
on the Horizon system. There was extensive evidence of fact in both the Common Issues trial and the
Horizon Issues trial that this had to be done. Unless it was, the branch could not trade in the next
trading period.
1024.So far as disputed amounts are concerned, an SPM cannot dispute a discrepancy or any figure
on Horizon, or record on Horizon that they have raised a dispute. Horizon records figures even when
they are not correct, and there is evidence of figures on Horizon being known to be incorrect by the
SPM and the Post Office, or both. A dispute is normally raised by a SPM through contacting the
helpline, which is a telephone service which was widely referred to in the Horizon Issues trial. There
is a finding in Judgment No.3 that by contacting the helpline an SPM was registering a dispute with
the Post Office. The helpline and its function are however outside the scope of the Horizon Issues, and
the helpline does not form part of the Horizon system.
Transaction Corrections
1025.Issue (15): How did Horizon process and/or record Transaction Corrections?
1026.Answer: This too is substantially agreed in terms of technical content, and I have explained in
the body of the accompanying judgment, that TCs are outside the definition of the Horizon system.
1027.The creating of TCs is a manual process. Dr Worden relied upon TCs as underpinning his
conclusion on robustness of the Horizon system. The creation of TCs is no part of the Horizon system.
Without TCs, which are very numerous, the Post Office system of accounting with its SPMs would be
even more problematic that it has been. There have been over 100,000 TCs issued each year since
2006. This is more than 2,000 per week.
1028.The Post Office has a central accounting function that comes to a decision that an adjustment
has to be made to a branch’s accounts, as the branch transaction data (which comes from Horizon) is
not consistent with the data the Post Office has received from its clients or suppliers. This adjustment
is the result of a human process and is made by way of the issuing of TCs to a particular branch. This
is done without
the involvement of SPMs and the decision to do so is taken outside the Horizon system. In the early
days of Horizon they were called Error Notices.
1029.A TC is then sent by the Post Office to the SPM and the SPM processes such TCs by accepting
them. By that acceptance, the correction that is the subject of the TC will enter the branch accounts
on Horizon. Disputes to TCs are done by the SPM contacting the helpline and again this is outside the
Horizon system, which does not record that dispute. If the dispute is upheld (as in, the original TC is
accepted by the Post Office as having been wrongly issued) then another TC will be issued to correct
it. That subsequent TC will correct the effect of the first TC in the branch accounts when it is accepted
by the SPM. The issuing of the subsequent TC is also done outside the Horizon system.
1030.The Post Office does not have comprehensive records of how many TCs that have been
challenged by SPMs have been upheld. I have provided at [859] and [860] above an explanation of
part of the 3rd experts’ joint statement in respect of TCs, and the percentage of TCs relating to
Santander that the Post Office said were upheld. 77% of these were upheld; which plainly means that
23%, or 1 in 4, of those disputed were wrongly issued. I will also record here for completeness
RLITO000005
RLITO000005
(although they are not TCs) that Transaction Acknowledgments or TAs are also used for making
corrections to branch accounts. To use the Lottery and Camelot as an example, the issuing of TAs is
now automated following the so-called Ping fix. However, a SPM has no option whether to accept TAs
or not; they have to be accepted in branch. This was agreed by a number of the Post Office’s factual
witnesses.
THE POST OFFICE GROUP LITIGATION
IN THE HIGH RT OF TICE
QUEENS BENCH DIVISION
Rolls Building Fetter Lane London, EC4A 1NL
Date: 16 December 2019
Before :
Between :
Alan Bates and Others Claimants
- and -
Post Office Limited
Technical Appendix to
Judgment (No.6) “Horizon Issues”
1. This is the Technical Appendix to Judgment (No.6) which deals with the Horizon Issues. The purpose
in preparing this is to avoid what is already a long judgment being vastly increased in length by
including technical IT details within it, which are not necessary for the vast majority of readers to
understand.
2.This appendix has the following structure:
A.The History of the Horizon System
B,Structure of Horizon generally
C.Legacy Horizon and Horizon Online
D.Submissions and Evidence
RLITO000005
RLITO000005
E.Bugs, errors and defects
FConclusions on Technical Issues
3. The contents of this appendix are predominantly drawn from the two expert reports, as well as the
factual evidence, but also the parties’ extensive submissions and some of the contemporaneous
documents deployed in the Horizon Issues trial. Given the size of the system, and its evolution into
Horizon Online from the days of Legacy Horizon, it would have been entirely possible to draft an
appendix that dwarfed the length of Judgment (No.6). In order to avoid that, as such a lengthy
document would not be proportionate or reasonable, I have summarised certain aspects of the
technical evidence and documentation where necessary. There may therefore be some passages in
this appendix which, for example, to someone vastly experienced in writing Oracle code, may not read
as being 100% technically accurate or which seem to be a superficial summary. However, in my
judgment, such lack of full technical accuracy in description or terminology does not affect either the
consideration of the evidence, or the conclusions, or my findings on the Horizon Issues.
A. The History of the Horizon System
4.There was a great amount of agreement between the experts on both the history of Horizon and its
architecture, both Legacy Horizon and Horizon Online.
5.Originally, in the 1990s, there was an intended tri-partite project between the Post Office, the
government department responsible for welfare benefits, and ICL. During evidence the second of
those two parties was often referred to as the Department of Work and Pensions, or the Benefits
Agency. In fact, prior to 2001 the relevant department was the Department of Social Security or DSS
(which had been created in 1988 from its predecessor, the Department of Health and Social Security
or DHSS). The Department of Work and Pensions was not created until 2001. The different
terminology does not matter, other than for the sake of accuracy. At least part of the reason for the
project was so that benefits claimants could be paid by using a card, rather than what used to be the
case with paper books which a branch Post Office would stamp, remove and exchange for cash.
6.The tri-partite project was called Pathway. ICL Pathway (the department within ICL formed for the
purpose) was awarded a contract for an electronic way of paying benefits, the Benefits Payment Card,
in May 1996, and there was a Horizon Pilot that was introduced in a small number of branches in
1996. This tri-partite scheme was abandoned in 1999. Pathway cited “greater than expected
complexity” and “...major implications for the degree of difficulty of the project”, quotations included
in Mr Coyne’s report, as ultimately leading to the failure of the project. In July 1999 Post Office
Counters Ltd (or POCL, the previous name of what is now the Post Office Ltd) and Pathway agreed to
utilise the project to automate branch Post Offices. This is what is now called the Horizon System or
Legacy Horizon, and it was introduced (the technical term is “rolled out”) from late 1999 onwards.
7.Anyone with a modern smart phone will know that modern IT technology is marching ahead at an
increasingly ferocious pace, and particularly those adept at its use now (including the younger
generations) may be surprised that so much of what is now dealt with electronically, used to be done
using paper. Prior to the introduction of Horizon, the branch Post Office system used a system of
paper-based accounts. A paper-based system was used by SPMs to account to the Post Office for the
branch activity, although some SPMs used software packages to assist them in this. Although Horizon
is used as an accounting system - it is one of its main uses - there is more to it than that. However,
computerised accounting systems have been in use since the 1950s, and Dr Worden explained that
accounting was one of the “most mature applications” for computers in business. This means that as a
RLITO000005
RLITO000005
computing application, accounting has been such an application for a long time, even in the late
1990s.
8.Dr Worden explained the origin of double-entry bookkeeping and the principles that underlie it, and
explained the process of manual bookkeeping, even (in footnote form) as far back as the clay tablets of
Ur, a Sumerian city-state in ancient Mesopotamia which dates from 3800BC. The cuneiform tablets of
Ur, as he pointed out, were not double-entry bookkeeping, which was invented in the 14th century.
They were singleentry; an entry would be recorded once. Double-entry bookkeeping means that
discrepancies can be discovered much more speedily, and easily, than otherwise. I consider Dr
Worden’s explanation of this to be sufficiently concise and helpful that I will reproduce parts of it from
one of his appendices:
“12. Because any accounting system is intended to track external reality, and to give the most
accurate possible picture of that reality, it is essential from time to time to check the picture of reality,
held in the accounting system, against reality itself - the physical assets of the business, its money in
cash or banks, its obligations and debts.
13.However, checking against external reality is (or was) an expensive process. You cannot simply
look at the books - you have to get up from your desk, go out into the warehouse and count stock,
check your bank balance and count your cash, consult other people, and so on. Because checking
against reality was an expensive process, it did not get done very frequently. If the check is only made
occasionally, and mistakes are found, the interval of time in which one or more mistakes might have
occurred is a long one. The error is more likely to have arisen from multiple mistakes. Looking for
several mistakes together is much harder than looking for one mistake; there is no ‘telltale number’ to
look for. The process of ‘drilling down’ to find the origin of a mistake is difficult and unguided, with
few clues.
14.Double entry bookkeeping changed this. If a bookkeeping mistake is made, that mistake will lead
to a discrepancy against external reality, which will eventually be found in the external reality check.
B A A . A , ffi .
mistakes on the other columns. So, the mistake will immediately destroy the trial balance. Checking
a < . 1 fi
15.So double entry bookkeeping immediately reduces the cost of keeping accurate and trustworthy
accounts. It was an early, and highly effective, form of error repellency - in a time when manual errors
of bookkeeping were likely to be frequent. The error repellency guarded against a single point of
failure (i.e. a mistake in a single column of figures) by making any such mistake rapidly and obviously
visible, in the next trial balance.”
(emphasis added)
9.A very simple example was given by Dr Worden in his Appendix B, which again I will reproduce. For
those with no experience of the pre-decimal system the units of money might be a little vague, but the
principle applies whether one expresses the money in pounds or shillings:
RLITO000005
RLITO000005
“8. Double entry bookkeeping can be understood by starting with the simple case of a trader ona
farmer's market, who has some money, and who has some sheep. He can keep a list of his money, and
a separate list of his sheep - complete with their names if he wants to. He can track changes to those
lists, with entries like ‘23rd July: sold one sheep - Dolly’, From the changes he can work out his
current position, in money or in sheep: ‘now I have 17 sheep left’. This is single-entry accounting.
9. Double-entry bookkeeping starts when his list of sheep contains two types of information - the
sheep he has, and the price he paid for them; and he also keeps a separate list of his money. He tracks
the changes to these lists in a series of dated transactions: '25th July: bought one sheep for 5
shillings'. With that transaction, the sum of his money list goes down by 5 shillings, and the money
total of his sheep list goes up by 5 shillings. So, he makes two entries - in those two lists - with money
value -5 shillings and +5 shillings, and the total money value of the two lists does not change. Because
of the self-consistency of mathematics, however he chooses to do those two sums, the sum of the two
sums should not change from day to day. This is his double entry ‘trial balance’. If the number does
change from any day to the next, he knows he has made a mistake - and he can start to track it down.”
10.This simple example can be applied to any stock held in any business, in this litigation branch Post
Offices, and the sum of money which is used to purchase stock in a branch by a customer. To return to
the well-worn example of the book of 1st class stamps used earlier in the litigation, if a SPM sells one
book of these to a customer for cash, the cash position of the branch improves by the cost of that book
of stamps, paid to the branch by the customer (as of today, £8.40 for a book of 12 1st class stamps)
and the stock of books of stamps held in the branch reduces by one. This is a simple example because
the stamps are part of the branch’s stock. There are more complex transactions in a branch that do
not involve stock actually held within the branch; this
will usually be because the branch offers services and products on behalf of the Post Office’s
numerous clients, that are not stock in the traditional sense physically held within that branch.
11.Dr Worden also explained the advantages of double-entry accounting, which include reduction of
fraud and theft. Notwithstanding the differences to which accounting systems are put (or to put it
another way, how the information held within such systems is used), such as management accounting
(for those responsible for making management decisions about the business in question) or financial
accounting (for example, for departments who have to declare profits and account for taxation) both
sets of information are subject to audit, which is a word which means review, examination, inspection
and appraisal. In lay terms, it is the checking of the information held within the accounts.
12.The word “audit” is used in different ways, and for different things, in this group litigation. The
Post Office employs a number of people called auditors to check on the accounts of branch Post
Offices. When Mr Bates, for example, started as a SPM, the Closing Day Audit was conducted on the
branch (run by his predecessor) as part of what occurred on what was called Branch Handover Day.
This is done so that a SPM on their first day starts with particular stock and cash, identified by the
Closing Day Audit, and their branch activity uses that as a starting point. Some SPMs may be
suspended, or have their engagements suspended, as a result of what the Post Office (through its
auditors) uncovers at other audits, which may be performed at any time (which is entirely
understandable, and is included in the terms of the SPMs’ contracts). These audits are done to check
the state of the branch accounts, and the cash and stock held in the branch.
13.The Post Office itself is also audited by its auditors (such as Ernst & Young or E&Y), and the
Horizon System is itself audited in the sense of checks being performed that it complies with various
RLITO000005
RLITO000005
standards. Audit is also used as part of the expression “audit data”, further explained in Part K of
Judgment (No.6) itself.
14.The financial accounting functionality of the Post Office was provided by something called SAP,
which is an industry leading resource planning and accounting system. From 2004 onwards this was
by something called POLFS, and in 2010, with the introduction of Horizon Online, this became
POLSAP. POLSAP was created by merging two other systems, POLFS and SAP ADS, the ADS in that
latter acronym standing for Administrative Data Services. Horizon provided mainly two things;
management accounting functionality, as well as Electronic Point of Sale functions for the branches
(also called EPOS). There is, in Horizon Online, overlap between Horizon and POLSAP. Dr Worden
stated:
“I understand that the financial accounting functionality for the Post Office was provided by SAP over
most of the disputed period (from 2004 by POLFS, a SAP application, which in 2010 merged with SAP
ADS to form POLSAP), whereas Horizon provided mainly management accounting functionality, as.
well as Point of Sale functions for the branches. This dual nature required close integration between
Hori SAP ffi . we
times mutually consistent, There is a large overlap between the information stored in the two systems,
which will be described below.”
(emphasis added)
15.Some computers provide a highly complex computation function. Dr Worden stated, and I agree
with him, that the functions of an accounting system include the input, secure storage and output of
many types of information, in large volumes (many transactions per day), with rather little
computation. By far the most important type of computation that occurs in financial accounting is
straightforward summation of numerical quantities such as money and stock, subject to the rules of
double entry bookkeeping. For management accounting, some other kinds of computation are needed,
such as for forecasting purposes. However, these are not usually computationally demanding or
complex. This is not disparaging of those who designed the system, but given the function of Horizon
so far as branch Post Offices are concerned, Horizon does not possess (nor does it need to possess) a
highly complex computational function in the sense that very difficult and demanding complex
mathematical equations have to be processed at high speed, for example. The computers at NASA in
the United States that are responsible for space exploration, and control of satellites, will be dealing
with far more complex computation in terms of mathematical function; however, rarely would there be
12,000 moon shots underway at once. Horizon deals with a huge number of transactions per day
(about 6 million) across many thousands of branches, but the nature of the calculations is essentially
arithmetical. This is not to understate the complexity of Horizon both in terms of the scale of the
system, the number of products or the number of different clients with whom the Post Office (and
hence SPMs in branches, on behalf of the Post Office) deal. Fujitsu, in a document of 14 January 2004,
described Horizon as ‘Europe's largest non-military IT contract’, so Horizon is at the high end of
complexity amongst IT systems. Dr Worden stated that it represents “many thousands of man-years
effort in development and testing, and its documentation alone more than 100,000 documents”. It can
therefore clearly be seen to be undoubtedly highly complex.
16.However, Dr Worden also said this:
“On the other hand, compared with the IT estates of various large organisations I have worked for
(such as the NHS, Barclays Bank, UBS or RBS), the Horizon system is probably no more complex, and
in some ways less complex. The banks' IT estates, like Horizon, have a complex corporate back-end
RLITO000005
RLITO000005
and an extensive branch office network. They were developed over a longer time period (30-40 years)
using development team sizes similar to or larger than Horizon, often merging together or integrating
the IT systems of previously independent organisations. This gave them a degree of legacy complexity,
and design compromise, and corporate amnesia, not found in Horizon. There are parts of these IT
estates which ‘nobody dares touch’.”
17.This means that simply because Horizon evolved as it has is not an answer to the more
controversial of the Horizon Issues. Its origins as a system to accomplish something else, the payment
of benefits, its incremental addition of products and clients, and its age, are not of themselves
indicative of any particular level of performance.
B. Structure of Horizon Generally
18.Although computerised accounting systems had been used widely for some decades since about
the 1960s, in the 1980s there was what was called a “major step change in their architecture”, when
relational databases were invented/developed and used. A relational database is a digital database
based on the relational model of data, and was first proposed in 1970 by EF Codd. Mr Codd was an
English computer scientist and invented the relational model for database management. This model
uses a structure and language consistent with what is called first-order logic, although it is also called
other things (such as quantificational logic) and this is used both in mathematics, computer science
and also linguistics. Inter alia it allows the use of sentences that contain variables. In the relationship
model, all data is represented in terms of tuples, grouped into relations. In relational database theory,
a relation is a set of tuples (d1, d2, ..., dn), where each element dj is a member of Dj, which is called a
data domain. Each element is termed an attribute value, and an attribute is a name paired with a
domain (which is also called a data type, or simply a type). In mathematics a tuple is a finite ordered
list or sequence of elements.
19.A database organised in terms of the relational model is called a relational database. A software
system used to manage relational databases is called a relational database management system or
RDBMS. Many such systems use Structured Query Language or SQL to query and maintain the
database. A famous and widely used RDBMS is Oracle RDBMS, which is sometimes called Oracle
Database or even simply Oracle. This is a proprietary multi-model database management system
produced and marketed by the Oracle Corporation, a US technology corporation based in California.
It was first released in 1979.
20.The system software therefore used for a relational database is built and supported not by the
accounting system developer, but by the supplier of system software, which will be a major company
such as Microsoft or Oracle. The accounting software makes calls to the RDBMS to store and retrieve
the information. These work (in outline terms) by storing structured information. Unstructured
information can be stored in computer systems generally, but this requires very advanced computer
programs to understand it. More simple programs, such as accounting systems, store and use
structured information. Structured information is (for example) something contained in a spreadsheet,
made up of information in different cells in rows and columns. Adding up the value of a list of
transactions can be accomplished simply by adding up the figures in a particular column that relates
to this information for a number of transactions.
21.The main use of a relational database is securely to store large volumes of structured information.
The way it does so can be understood as having large numbers (tens, hundreds or even more) of
different spreadsheets (which are called tables) and which are linked to one another. Two different
RLITO000005
RLITO000005
tables in a database are linked to one another (in a 'relation') when they both have one or more
columns with the same meaning and share values in those columns.
22.Dr Worden explained:
“44, A typical relational database may have tens or hundreds of separate tables; each table may have
tens or occasionally hundreds of columns; and a table can have any number of rows, up to millions if
necessary.
45.One relational database can act as a 'server' to one or more application programs, which are
performing actions directly visible to their users. The application programs make calls (requests) to
the relational database management system, which alters or retrieves the data to fulfil the requests.
The core service provided by the relational database is to securely store and retrieve this information,
for the application programs, or for others who retrieve information from the database more directly.
Both the phrases ‘securely store’ and ‘retrieve’ have a lot packed into them.
46.The phrase ‘securely store’ implies several guarantees: that once an item of information has been
stored in the database, it will not be lost or unintentionally changed; and that it is a part of a
consistent collection of information, whose integrity will not be compromised in any way.”
23.The integrity of information is central to relational databases. Indeed, I consider that the integrity
of information could be said, in a way, to be central to the concept of the operation of Horizon, if not
accounting altogether.
24.Dr Worden explained the concept of integrity, and integrity constraints, in the following way:
“47. The idea of integrity of information is central to relational databases. When the structure (tables
and columns) of a relational database is first defined (in a relational schema) that schema defines
various types of integrity constraint on the data, for instance that:
47.1.certain columns in tables are mandatory, and will always be given values.
47.2.There can never be a record in some table (call it table A) unless there is a corresponding record
in some other table B. For instance, there can never be a record of an invoice to a customer, unless a
record exists for the same customer. The invoice table and the customer table both have a column
‘customer id'; for every invoice record, there must be a customer record with the same customer id.
48. The database management system guarantees that these integrity constraints will be true for all
time. If any application tries to make a change to the database which would violate an integrity
constraint, that change is rejected by the DBMS, and no change is made at all. One change to the
database may involve changes to several tables at once, so that after all the changes are made, the
integrity constraints are still true; but part way through the changes, the constraints are temporarily
untrue. One such package of changes, involving changes to one or more tables, is called a database
‘transaction’. The DBMS guarantees that:
48.1.After any completed transaction, the database will still obey all its integrity constraints.
48.2.After any completed transaction, the changes made to the database can never be lost - even in
the event of hardware failures; there are robust ways to recover the information.
48.3.If a transaction would violate an integrity constraint, it is not allowed by the DBMS; no changes
will be made to any table, so the database will still be in a consistent state, as if the change had never
been requested.
RLITO000005
RLITO000005
48.4.When one application is making multiple changes to the database in a transaction, so that the
database is temporarily in an inconsistent state (when some but not all of the changes have been
made), those changes are never visible to other applications or users before they have all been
completed. Other applications and users can only ever see a consistent state of the database (either
before all the changes, or after all changes), in which all the integrity constraints are true.
49. This set of guarantees, given by the DBMS, is called ‘transactional integrity’. It has been built into
the fabric of all relational databases and has been relied upon by thousands of applications which use
relational databases, since the 1980s. Builders of applications can have a very high degree of
confidence that their DBMS will maintain transactional integrity. Builders of accounting systems have
relied on that guarantee.”
25.Relational databases support the SQL language, and small numbers of records can be retrieved
from within application programs from a limited number of linked tables, and the records are filtered
based on data values. Dr Worden considered that the possibility of there being a bug in Oracle to be
“extremely remote”, and he ignored it. As he put it:
“51. SQL can be used directly by end users, without the use of any application program, to selectively
retrieve and display records. However, it is more common for the users to use a general report writing
tool, which not only retrieves the required records, but formats the information in easy-to-read
reports - with appropriate column headers, formatting of fields, grouping of records, computations of
sums of groups of records, and so on. Nearly all the reports needed from an accounting system are
created using these report writing products, which can be rapidly configured to produce any new kind
of report, either one-off or regularly as required.
52. All these capabilities of a relational database have been present in relational database products
such as Oracle since the 1980s and have been tested by possibly millions of different applications
which use those capabilities and rely on them. So, when I am discussing the issue of bugs in an
application such as Horizon, the possibility that these bugs arise from bugs in the underlying DBMS -
particularly when it is the world market leader Oracle - is extremely remote, and I shall ignore it.”
26.This is a similar approach to that adopted by Mr Godeseth when he was giving his evidence about
PEAK 0195561, when a problem was reported to the SSC on 4 March 2010. This was where a SPM
had tried, on 2 March 2010, to transfer out £4,000 (referred to in the PEAK as 4,000 pds, which
means either pounds (plural) or pounds sterling) from an individual stock unit into the shared main
stock unit when the system crashed. Mr Godeseth had been looking for a bug at the time of a “red
alert” during the early days of Horizon Online, and his evidence about this is included at
[341] to [349] of Judgment (No.6). He referred to one possibility being a bug within
Oracle, which is dealt with at [350] of that judgment, which he said “would be huge” and which he
also said “was not the bug that I was looking at for the red alert”.
27.For the avoidance of doubt, when the experts reach agreements, or where I make any findings,
about the presence of bugs, errors and defects in the Horizon System (either Legacy or Online) either
in this appendix, or in Judgment No.6, these are not made concerning exactly where within the
different programs or applications within the Horizon System such bugs, errors or defects are
present, or by apportioning responsibility to any particular software provider. The Horizon System
includes Oracle as the RDBMS, but there are a considerable number of other companies involved.
Those companies are not represented and that is not what the Horizon Issues trial is about. The
RLITO000005
RLITO000005
experts would sometimes refer to what were called “sub-contracting parties” but it was not clear to
me whether they were sub-contracted either to Fujitsu or to the Post Office. Nor is it relevant to the
Horizon Issues whether the fault is within a particular layer within the system. The relevant issue is
whether there are bugs, errors or defects within Horizon.
28.The software in Horizon is what is called data-driven. If the accounting application software was
written in a way that depended directly on the accounting needs of a client (here, the Post Office,
what Dr Worden called a “chart of accounts”), then each company would need different accounting
software. This would be very uneconomical (in writing and testing all that software), and this is not
the way in which software is developed. The chart of accounts is not 'hard coded’ into the application
software. The software is written to work with any valid chart of accounts; and the chart of accounts
itself is treated as data and stored in the database. In this way, changes in the chart of accounts
(which occur from time to time) do not require changes in the accounting software.
“59. This is one example of data-driven software - in which any requirement which might change
frequently is encoded as data, rather than software code. The code is written and tested to work with
all allowed values of the data, to meet a wide range of requirements by changing the data rather than
the code. The modern practice of software engineering is to use data-driven software as far as
possible. This is not always as far as one might like; some differences between companies and their
requirements are best expressed as differences in code, rather than data”.
29.Accordingly, the accounting application or applications are built as layered software architecture.
This means that the accounting application is built “on top of” the relational database. There will
usually be a minimum of three layers in a typical system of this type. Layers are also sometimes called
tiers. These are the user interface layer, which presents information to users, and accepts their inputs.
This is the layer that a SPM would both observe, and which would react to the actions of the SPM (or
their assistants) in the branch such as pressing buttons at the terminal for various transactions. The
second layer is the business logic layer, responsible for carrying out business processes, and
supporting users in doing so. The third layer is the data layer, responsible for storing and retrieving
data. More complex architecture has more than these three layers, and there may even be layers
within layers. In the case of Horizon, the user interface also includes a Point of Sale interface which is
used in the branches. This interface can be thought of as part of the Horizon Point of Sale application,
rather than the Horizon accounting application.
30.Almost all complex IT applications are designed in this way, in order to isolate different kinds of
complexity within different layers, and to reduce the possibility of unwanted interactions between
functions in different layers. A typical ‘client server' layering structure would include at the very least
the three layers identified above (a user interface layer, a business logic layer, and a data layer) but
the layering in Legacy Horizon is more complex than this.
31.The main purpose of defining an architecture in layers is to separate the functionality into parts in
the different layers, with well-defined and simple interfaces between the layers. This not only makes
each layer easier to design, build and test; but also, if there are errors that are not found in testing, it
should be easier to understand and isolate the cause of the errors by inspecting the exchanges
between layers. Dr Worden considered layered architecture to be an important countermeasure. It
could also be seen as an important part of designing such a system. It is part of modern practice.
Legacy Horizon
RLITO000005
RLITO000005
32.The architecture of Legacy Horizon up to 2002 is described in a lengthy document called Technical
Environment Description dated 22 October 2002. This document states: 'The system architecture
adopted to meet these requirements is not based on conventional client-server models. Nor does it
conform to traditional central-system models. It adopts an entirely original and highly innovative four-
tier model that effectively merges the qualities of central systems and client server systems'. The
diagram that accompanies this in the document actually has five layers rather than four. If two boxes
in the diagram which are entitled 'Agents' and 'Correspondence' are counted as one ‘Agents’ layer,
which is what Dr Worden did, there are four layers:
1.Counter.
2.Agent.
3.Host.
4.External Interface.
His
Dairies ien
Comaer
33.The counter layer consists of all hardware and software in the branch. It includes all hardware and
software required to support the counter activities required for all products and customer services
offered in the branch. This was largely built based on a commercial product, Riposte from Escher
Group, a separate company from Fujitsu.
34.Riposte provided much of the Graphical User Interface (which is the basis of all user input and
output at the counter) and provided a mechanism for secure distribution of messages between the
branches and the two back-office data centres, which are also referred to in some places (including Dr
Worden’s report) as campuses. These were located at Bootle and Wigan. The message distribution
passed through the Wide Area Network in the diagram.
35.The Correspondence Servers handled communication over the network.
RLITO000005
RLITO000005
36.The function of the Agent layer was to provide two-way translation of data between the formats
used in the counter layer and the network (these formats were described by Attribute Grammars) and.
the formats used in the Host layer. The agent layer is also responsible for extracting the Audit of all
data passing through the Correspondence Servers.
37.An Attribute Grammar is a description of a message structure which is set out in the same pattern
as a tree. Dr Worden used the phrase “a tree-like message structure in terms of its parts and their
sub-parts”. In more recent IT systems, tree-like messages are usually sent in XML (Extensible
Message Language), with their structure defined in a notation called XML Schema. This is used in
parts of Horizon Online. An XML document is a string of what are called characters. These characters
are divided into markup and content. What is called the processor analyses the markup and passes
structured information to an application. Generally, strings that constitute markup
begin with the character < and end with a >, or they begin with the character & and end with a ; (ie a
semi-colon). Strings of characters that are not markup are content.
38.Because Legacy Horizon was developed before the use of XML became widespread, Attribute
Grammars fulfilled this function in Legacy Horizon. This is probably the way that Riposte worked. In
Legacy Horizon, Riposte provided the facility for reliable replication of data between the branches
and the back-office or data centres/campuses. This means that if certain types of data were created at
the branches, Riposte should have ensured (the term used by Dr Worden was “guaranteed”, but I
consider that overstates it) that the same data would be available at the data centres/campuses. Dr
Worden accepted that “if the underlying network was unreliable, it might take some time for Riposte
to deliver this guarantee”. He did not say how long that might be and he was not asked about it.
39.One of the bugs in the bug table was also called the “Riposte Lock” problem. This will be further
addressed in Part E of this appendix. It was a problem with Riposte that was known about at Fujitsu.
40.The bulk of the back-office functionality was provided in the Host layer. Host applications were
typically batch systems, processing data in large batches on a daily basis. A complex daily batch
schedule was used to control the sequence and timing of these batch processes, using the Maestro
scheduling product (not to be confused with the Maestro software, made freely available by NASA to
allow users to look at photographs and the daily progress of the Mars exploration rovers). It was the
Host layer (and for most purposes, only the Host layer) which communicated with the IT systems of
Post Office client organisations, through the External Interface Gateways. Dr Worden described the
three layers of architecture which resided at Wigan and Bootle as “the back-end architecture”, and
these were the Agent layer (which included the Correspondence layer), the Host layer, and the
External Interface layer.
RLITO000005
RLITO000005
41.The following are shown in the above diagram as elements, and some description of them is
required.
42.RDMS is the Reference Data Management System. Reference Data is described further at [52]
below. This management system is a dedicated IT application which is needed to manage the
reference data, and to distribute it appropriately to branches.
43.The Data Warehouse consists of one or more databases, whose structure is designed to support
flexible and open-ended querying and reporting by the Post Office. Different kinds of information
which pass through the host systems are siphoned off into the data warehouse and stored there. This
storage is within data structures designed for reporting (which is done by means of querying those
structures). Functionality which depends on a data warehouse includes the Management Information
System or MIS and other applications such as those which look for unanticipated trends and
correlations in data.
44.MIS is the component built on the data warehouse to provide the Post Office with relevant
information. The data warehouse and the MIS are important parts of the Horizon System. Dr Worden
stated that “in cases of human error in business processes, operational errors in managing Post Office
business on Horizon, or software errors in Horizon, some resulting discrepancy or aberration will be
rapidly visible through the MIS. MIS facilities were also used by Fujitsu staff. Many pairs of eyes are
inspecting the outputs of the MIS, in hundreds of different reports or spreadsheets. One purpose of
this is to ensure the rapid detection and correction of many types of errors. These include software
errors”. By “pairs of eyes” Dr Worden was referring to human intervention or observation.
45.The Transaction Processing System or TPS has as its purpose the ‘harvesting’ of all types of
transaction taking place in the branches, and to pass them on to other IT systems in the Post Office -
initially to TIP and later to POLFS when the system was changed.
46.Transaction Information Processing or TIP was, as at 2003 (the date of the diagram above) the
gateway to all other Post Office data processing, including accounting. After 2004, Post Office
accounts were held on an SAP system, which was called POLFS; so TPS passed data to POLFS, rather
than to TIP.
47.The black circles in the diagram denote points, as Dr Worden expresses it, ‘at which ownership of
data conceptually changes and hence at which audit information is generated’.
RLITO000005
RLITO000005
48.Dr Worden quotes the 2002 Technical Environment Description document, what he refers to as the
“the 2003 design document” to explain the Host at paragraph 165 of his report. ‘One essential task
that can only be carried out at the Host layer is reconciliation. The Host is the only system component
that can detect discrepancies between the transactions carried out at the Counter (and hence
reported back to Post Office Ltd via TPS), and those that were authorised or expected. It should be in
a position to send reconciliation reports back to its Client. These enable the discrepancy with the TPS
records to be identified and resolved.’.
49.He considered that “this reconciliation, carried out in the Host layer, is an essential element within
[Legacy] Horizon for detecting and correcting errors made at the counter (robustness measure UEC)”.
50.Reconciliation and TCs have the effect of detecting and correcting the effects of many possible
software errors, as well as human errors. Dr Worden was of the view that if there were any such
software errors, these would probably occur with such high frequency, and occur uniformly across all
branches, as giving rise to so many TCs, that the Post Office would soon suspect a software error (for
instance, seeing the effect repeatedly in some MIS report) and require Fujitsu to correct it. However,
this is supposition in my judgment. This is because it depends upon a software error occurring very
frequently, and being noticed rapidly. Some accepted errors (for example Callendar Square) lay in the
system for 5 years. The evidence of fact advanced by the Post Office itself, which includes that in
respect of the admitted or acknowledged bugs, makes it obvious that this supposition is not made out.
Further, the detail of the numerous PEAKs show that it is a combination of factors, including the
speed at which a SPM performs some tasks, that sometimes led to unexpected events occurring. If
something happened at a particular point - one point put by the Post Office to Mr Tank was that an
outage occurred at a very precise moment - then an adverse impact would or may occur. That precise
combination of circumstances may not occur with the frequency that Dr Worden stated. His
supposition was that a bug would have a wide impact across many branches with a high frequency,
which would mean that the Post Office would suddenly realise its existence. That is not a valid
supposition or assumption given the evidence in this trial.
51.The Horizon System has certain core components. They include both Reference Data and
Transaction Data. The difference between these is as follows.
52.Reference Data is effectively data about products and operational elements and is a critical
element of the Horizon system. Reference Data interfaces with a wide range of Horizon components.
Without Reference Data, Horizon would not fully function, nor could Subpostmasters operate their
branches. It informs the operation of the Point of Sale system at the counter. This is now managed and
operated by ATOS. Many of the business applications in the branches are driven by reference data,
and this approach has advantages over hard-coding the different business applications, as that would
require changes to the code if the applications were to be changed. It is much more flexible, to
manage changes over time and across branches to use the reference data approach. This is because
when changes are made, the reference data is changed, but not the code.
53.The integrity of Reference Data is critical for the correct operation of a variety of systems within
the Horizon architecture. The Post Office’s Reference Data Management Centre (RDMC) supports the
loading, storage and release of Reference Data within the Horizon system. The Reference Data
Distribution Service (RDDS) distributes Reference Data to Post Office branches and other data centre
systems. The Post Office Reference Data Team deals with the delivery of Reference Data and
verification of operational business change through Reference Data. Horizon counter Reference Data
RLITO000005
RLITO000005
was distributed by the Riposte messaging facility in Legacy Horizon, although this is not used in
Horizon Online.
54.Mr Coyne considers that there were insufficient appropriate change control processes prior to
2017, based on a document he has seen which is dated July 2017 which stated “... we have now
aligned that all Reference Data changes go through the appropriate change process”. Mr Coyne takes
this to mean that prior to this date, all Reference
Data was not going through the appropriate change process, or that change process
did not exist. In my judgment that is probably reading too much into that particular entry; however
and in any event, change control processes require two things in order to work effectively. Firstly, they
must exist. Secondly, they must be applied.
55.Some aspects of how business applications were built on Riposte were explained in the expert
evidence, without going into enormous layers of explanation of technical complexity that were
unnecessary. These included the following: Zero-sum baskets for customers meant that the overall or
net impact of all the services provided for one customer was a sum of money which the customer was
required to settle. It was required that the cash or other money produced by the customer should
exactly match the cost of services provided; therefore, the whole basket of services and customer
settlement had to be what is called “zero-sum”, before the basket could be recorded in the branch
(and then, through Riposte replication, later recorded in the back-office systems). The impact of every
business application would be fed into the relevant systems at the Post Office, typically overnight.
These systems were operated by double entry bookkeeping. The only way to put postings into the
accounting system was by double entry, which in turn could only be done for zero-sum baskets. Dr
Worden considered this a robustness countermeasure.
56.Any other actions performed in the branches which had an impact upon the Post Office’s accounts
(which would include stock management, cash drawer management, balancing and reconciliation)
also had to be zero-sum. These could only be carried out in the branch in packages of updates which
were zero-sum, when summed across different Post Office account codes. This was another instance of
a robustness countermeasure.
57.All branch applications had to have what was called transactional integrity. This means that all
customer business applications, balancing and reconciliation, cash management and stock
management were built so that any zero-sum package of updates from those applications would either
succeed completely, or would fail completely. In the latter case, they were supposed to have no impact
on branch accounts. This transactional integrity was given the acronym TIN, and was supposed to be
enforced by the Riposte infrastructure. It should have been impossible in the event of a failure (such
as hardware failure) for a part-completed set of updates to be recorded in the branch and then
replicated to the back-office systems. This was necessary to prevent the accounting system from being
subjected to non-zero sum updates, which would violate its double entry basis and cause later failures
of its trial balances.
58.The only exception to this principle was that of so-called ‘recoverable transactions’ - where some
irreversible interaction with a Post Office client system took place part way through a transaction - so
it could not be undone in the case of a later failure. In these cases, the user on the counter would be
guided through a short set of recovery steps, to produce a consistent zero-sum result which reflected
what had happened. As a comment made here by me for convenience, it is the case that some of the
evidence of fact from SPMs which I have accepted about transactions that gave rise to difficulty in
RLITO000005
RLITO000005
branches occurred when there were outages. It will be seen from consideration of the bugs in the bug
table that these were not the only instances when bugs had some impact on branch accounts.
However, whether there was a hardware failure or outage, the system was built so that partly
completed transactions were not supposed to be capable of entering the accounting system. Dr
Worden opined that it
was, as he put it, “of course, possible for the user to make some mistake in these steps, which may
have been unfamiliar. In these cases, the mistake would often be detected later by a reconciliation
process, which would typically lead to a TC. This robustness measure was a correction of user errors
(UEC).” However, that statement in my judgment jumps straight to a conclusion that user error would
be to blame for failures in recoverable transactions, and this would “typically lead” to the Post Office
issuing a TC. In my judgment, the correct approach is to consider bugs, errors or defects within the
Horizon System and whether these would lead to failures in recoverable transactions. This is the
correct approach to answering the Horizon Issues.
59.Many applications were not coded individually but were coded as generic applications which could
be configured to run different specific applications by altering reference data, as has already been
explained. This is called “data-driven software”. These applications were referred to in Legacy
Horizon as ‘soft-centred’ applications. They have the attraction of adaptability in that the data can be
changed without new code having to be written. There were however still some hard-coded
applications in Legacy Horizon in any event. Errors could often be corrected more quickly if
applications are driven by reference data in this way. Reference data is much more concise and
understandable than code, so it is much easier to create it or detect errors in it. Dr Worden
considered that errors in the underlying generic code would affect a set of specific applications, and
so should be easy to detect. However, this can be seen not to be a statement of universal application.
The Callendar Square bug was present for 5 years before it was properly discovered (although its
effects were experienced, to use Anne Chambers terms, “most weeks”).
60.Reference data could be as simple as lists of available products and their prices (which clearly
might change frequently), or might be more complex - for instance, to describe the sequence of steps
needed to handle some business transaction.
61.The other important component that can be usefully addressed following Reference Data is
Transaction Data. There are four sources of transactions that make up transaction data:
1.Those manually entered by a user (which will be a SPM or their assistant) in branch at the counter;
2.Transaction Corrections (TCs) which are produced by the Post Office to be accepted by a user in
branch to correct discrepancies in the accounts;
3.Transaction Acknowledgements (TAs) which are non-counter transactions and typically initiate from
somewhere else. This is from another area outside the Horizon System so far as the Lottery is
concerned, because the information comes from Camelot relating to the lottery sales a branch has
performed. These transactions are typically relayed to Post Office/Fujitsu and need “accepting” into
Horizon before forming part of the branch’s transaction data. This is done by means of TAs sent to
each branch. The SPM does not have the option to reject them. TAs did not exist prior to the Ping fix;
and
RLITO000005
RLITO000005
4.Fujitsu inserted transactions. These are injected into branch accounts by Fujitsu. They may be
performed in order to ‘balance’ a discrepancy. These do NOT require acceptance by SPMs in the same
manner as TCs and TAs.
Horizon Counter
62.The user in a Post Office branch, the SPM or assistant, will deal with customers and for each
transaction press a number of different icons in order to compile that customer's “basket”. As has
been seen, different products from different clients can be provided and sold to the customer in the
same basket. These may be paying bills, prepayment for services, acquisition of licences (such as a TV
licence), and money and insurance management.
63.The SPM is not concerned that (for example) some of the money collected from a particular
customer may need to go to DVLA (for car tax) or to a utility company (for a gas bill). It is the Post
Office that accounts to these clients, and it does so centrally and by means of a back-office activity.
64.As well as these counter activities, Horizon also needs to support the periodic process of balancing
and rollover for each branch. Every branch operates in Trading Periods, which are either four or five
weeks (according to a timetable published periodically by the Post Office). At the start of each Trading
Period the branch is supposed to be ‘in balance’. This means that the physical stock and cash in the
branch agrees with the data on stock and cash held in Horizon. Then, during each Trading Period,
Horizon records all customer transactions made at the branch, so it records the changes in cash and
stock. It also records any replenishments or remittances of cash or stock in the branch. Horizon will
(or should) record all changes in cash and stock held at the branch during the Trading Period, and can
compute, from the starting amounts and the changes, the expected amounts of cash and stock at the
end of the period. This leads to something called the Branch Trading Statement.
65.At the end of each Trading Period, the SPM counts the physical cash and stock in the branch and
compares it with Horizon's data for the same items. This is called 'balancing'. If the numbers are all
equal, the branch is in balance and is permitted to 'roll over' to the next period. If the two sets of
numbers are not equal, there is a discrepancy between what Horizon considers should be present in
the branch (for example the actual cash) and what is physically present (the actual cash held at the
branch). This also applies to some stock such as stamps. These discrepancies are at the heart of this
group litigation, as the reason for the discrepancies are so controversial. As part of “rolling over”, a
SPM with a shortfall or discrepancy has to “make good” or “settle centrally” at the time, as part of
bringing that Trading Period to an end. Unless they are prepared to do that, the next branch Trading
Period cannot commence. Each of those two acts, making good or settling centrally, involved payment
by the SPM to the Post Office; the only difference is the former means payment immediately, and the
latter means seeking time to pay.
66.Horizon also supports the activities of replenishing stock such as stamps, and of replenishing or
remitting cash. Accepting a delivery of cash from the Post Office into the branch is called, by factual
witnesses on both sides in the litigation, “remming in” that cash. This is probably shorthand for
remitting or remittance, but all involved in the case call it “remming”.
67.Dr Worden used DVLA as an example of how the back-office works. He stated:
“Across the UK in any day, Post Office accepts a large amount of money from customers paying their
road fund tax. All this money needs to be paid to DVLA. Therefore, Post Office has a back-office
activity - carried out centrally -of summing all these amounts of money and paying DVLA. DVLA.
RLITO000005
RLITO000005
knows how much money it expects to receive in this way and checks the amount it expects against the
amount calculated by Post Office. This cross-check is an example of reconciliation and supporting it
and reflecting its outcomes are central to Horizon.”
68.The counter included both hardware and software. Dr Worden described the hardware in the
branches as “not always reliable” but considered there were sufficient measures built into Legacy
Horizon to ensure that hardware failures and communication failures could not adversely affect
branch accounts.
69.Because of the difference in how data was and is stored in Legacy Horizon and Horizon Online
respectively, availability of the network was required for the latter but not the former. In Legacy
Horizon, sufficient data was held persistently in the branches, such that a branch could continue to
trade, and could support most business applications, even if the wide-area network was unavailable.
Whenever the network became available again, Riposte data replication should have ensured that the
required data became available to the back-office systems. The only applications which could not run
in this way were those that required some immediate validation from a client organisation - for
instance, withdrawing cash from a bank account. Horizon Online this was no longer the case as the
persistent data was all stored remotely in the BRDB. This meant that without a working network, a
branch could no longer trade. The network infrastructure (which is another way of saying the
reliability of the connections to the branch) by 2010 had made this viable.
70.The role of the Agent layer was to manage communications and translate data between the
representation used in the branches and the network on Riposte, and the representations used in the
Host layer. It is the Host layer that takes data from external clients. This is explained in the main
design document for Legacy Horizon:
‘The systems at the Host Layer can provide permanent storage for information if required by the
application’s business rules. The Host systems can accept data from external Clients, and translate a
file-based view of this information into discrete transactions or “messages”. These are then passed to
the Counters via the Agent and Correspondence Layers. Similarly, messages received from the
Counters are translated back into a file-based view for transmission to the external Clients.’
71.Each of the different business applications in Legacy Horizon was typically tied only to the
different Post Office client in respect of whose business it was concerned. For example, Camelot does
not need to communicate with the DVLA. Transactions in respect of them may be transacted in the
same basket, but this facility is separate to the applications that deal with Post Office/Camelot
business and Post Office/DVLA. Accordingly, each different business application in Legacy Horizon
can, in Dr Worden’s words “be regarded as a vertical 'slice' though the diagram and is largely
independent of the other slices”. He categorised this as “another example of robustness through
architecture” but again, it could be seen as a design element.
72.Visual Basic, C and C++ programming languages were used in the Legacy Horizon system along
with Oracle development tools. Riposte, the product developed and marketed by a company called
Escher (and which featured in many PEAKs) was used in Legacy Horizon too.
73.The Horizon Online counter and BAL (Branch Access Layer) were developed in Java, while the
agents and host modules remained in C and C++. C is a general purpose code, an imperative
procedural language which was invented by the late Dennis Ritchie. He was also one of the co-
creators of UNIX with his colleague Ken Thompson. Neither of these gentlemen are household names
in society generally in the same way as, for example, Bill Gates of Microsoft or Steve Jobs of Apple,
RLITO000005
RLITO000005
but are very well known in the computer software world. Both C and UNIX are extremely important
software technologies which are the heart of a great many, if not almost all, software products used
today. Together Mr Ritchie and Mr Thompson won the Turing Award in 1983, a prize with similar
status to a Nobel Prize, which is awarded by the Association for Computing Machinery. All of Java, C
and C++ are well known languages that are in widespread use.
74.Horizon Online was not a completely new system; it built upon, and continued to use, certain
elements of Legacy Horizon. The move away from Riposte and Visual Basic to Java brought in more
modern development tools and enabled techniques such as object orientation to be used. Object
orientation means that modules must be selfcontained, and will communicate via pre-defined and
documented interfaces. These interfaces are not dependent on the application's physical
implementation. Third party and commodity products (such as operating systems and device drivers)
are simply bought in by Fujitsu. Mr Coyne stated that “Horizon is therefore ultimately a composition
of many sub-contracted components”, which I consider to be a fair description.
75.Applications are developed by suppliers (who supply generic software or products), third parties
who supply applications that meet Horizon requirements, Fujitsu developers who develop software to
meet specific goals, or the Post Office. If the latter, this is done by configuring standard components.
As at the date of the expert reports for the Horizon Issues trial, the architecture is data driven such
that the Post Office could introduce new clients without reference to Fujitsu. All of the different
applications must however integrate with the other existing applications, as well as any other new
applications that are added. The complexity of the system will therefore have increased, year on year,
as further products and applications are added. Once an application has been developed, it must be
tested and evaluated, and then the process of integration takes place. It will then be released (which
means introduced into live) and maintained thereafter.
76.The Horizon system includes an audit database. This is an accurate record of any activity which
can affect the branch accounts. These records are available to be consulted in the event of any
discrepancy arising anywhere in Horizon (for instance, due to a bug in some other Horizon
application, or some operational error in running a batch process, or a dispute about what data was
entered at the counter) but the parties could not agree on whether and when this should be done. The
audit database is a robustness measure of a secure kernel (SEK) which also involves redundant data
storage (RDS). In Legacy Horizon, all data travelled from the counter through the software application
at the branch, through Riposte data replication to the two campuses, then through the Audit Agent to
the Audit Store. In Horizon Online, all data travelled from the counter through the software
application at the branch, through communications hardware and software to the Branch Access
Layer (BAL), into the BRDB and then nightly to the audit store.
77.The Post Office currently has several hundred client organisations, which shows the diversity of
services and products available in a branch. This also makes it obvious that, for most of these clients,
the service provided through Post Office will be different in nature from the service provided by the
Post Office to its other clients, so some unique software functionality must be provided within Horizon
both in the branch (for the counter) and also the back office to support the activities for each
particular client. This is a part of what makes Horizon such a large and complex system.
78.Checks are built into accounting systems. These can be by way of checks on data entry, double
entry bookkeeping checks in two or adjacent layers, consistency checks and defensive programming.
RLITO000005
RLITO000005
79.As I have already identified in Judgment (No.6), Dr Worden grouped countermeasures including
within the Horizon System with other characteristics that are not properly described as that, together
with those (such as manual checking by SPMs) that are not properly described in that way.
80.In a Post Office internal presentation document dated 7 December 2009, the following passages
appears in relation to “Horizon - Current State”. I consider it an apt description of Legacy Horizon:
“CURRENT STATE
Before discussing the future development of Horizon, lets remind ourselves of the 'system' we have
today
OVERVIEW
Horizon is 13 years old and the design criteria/technology was different and the business needs were
different - the demand for on-line working was low, the telecoms were expensive for on-line. Horizon
, ffi .
Very high security (e.g .. user ID/ access/ screens encryption) this was a requirement from Gov
because Horizon was built to support benefits payments,
Horizon is the 2nd most secure system in Europe. The MOD being #1 !
As time has passed and more product have been developed Horizon has evolved - from a technical
perspective essentially we have bolted things on the side - we undo the barbed wire stick a bit in then
wrap everything back up.
Design was optimised at the time to minimise costs (esp. network) - offline working.” (emphasis
added)
81.The following headline points emerge from thi:
1. Horizon was originally designed to support benefits payments to welfare claimants.
2.Horizon was already old in 2009. Although there were some major changes made when it became
Horizon Online from Legacy Horizon, Horizon Online still used major components of the existing
system. It is now somewhat older than it was in 2009.
3.It was originally built for off-line trading.
4.It has a relational database at its heart, but very numerous other applications that have been bolted
on to it over the years as the Post Office’s business has evolved.
5.There are a very wide number of other computing companies involved in the evolution of this
system, not only in terms of software. Oracle, Escher Group, ICL/Fujitsu, ATOS, Computacenter and
many more. It is a bespoke system that uses different encryption to other systems, such as Link.
6.The complexity of the different interfaces, as a result, is very high.
RLITO000005
RLITO000005
82.There have also been a total of some 19,842 release notes (in relation to software changes) in the
life of Horizon. This is consistent with each of these notes being a change to the Horizon system.
83.Fujitsu is the entity that was responsible for Legacy Horizon (putting to one side any contractual
nuances or differences between the Post Office and Fujitsu that may have existed, which are no part
of this litigation) and it remains the main partner with the Post Office for Horizon Online. However,
other suppliers are also now involved. Atos provides first line support via the Service Desk;
Computacenter supplies the hardware, and Verizon is responsible for networking (what might also be
called the connections between the different parts of the system). The main element within Fujitsu
that was referred to in the evidence was SSC, in particular 3rd line support, and “Development”. SSC
also serves other Fujitsu customers.
84.The Post Office Account - headed by Mr Parker - is responsible for developing and maintaining
Horizon, deploying it to branches, managing the operation of the system and supporting users. The
organisation of the Post Office Account is shown by the following graphic:
Post Office Account - Organisation
Meg tous I parte enn I Pegi tmation I Some
ed iiesodaal AnNSigcits Sao Business
Support
Functions
1
85.After the project to introduce the Payments Benefit Card in 1999 (the tri-partite project to which I
have already referred) was abandoned, the Post Office and ICL moved to automate post offices and
this was the introduction of what is now called Legacy Horizon. In August 2000 there was the Release
of what is called the Horizon Core System, or Core System Release. This introduced Automated
Payment Smart cards and APS/TPS reconciliation into branches. Not all branches received installation
of Horizon in the same month; that would, I imagine, be impossible in practical terms given the
number of branches in the Post Office network. Some dates are included in Judgment No.3 from the
Common Issues trial, in relation to those claimants called as witnesses in that other trial. Horizon was
installed in Mr Bates’ branch in October 2000; Mrs Stubbs said it was installed in her branch “in the
course of 2001” but it must have been earlier than that, as letters in relation to her shortfalls both to,
and from, Mr Manning are dated 1 and 2 November 2000. The precise date of introduction of Horizon
at each of the branches in the Post Office network is not relevant.
86.An upgrade of Horizon was performed in February 2001, which is called Maintenance Release M1.
The prime purposes of the upgrade were the enhancement of the CSR+ Applications (APS, LFS,
EPOSS, EPS, OBCS), enabling of the AP client variable day file transmission, enhancement to
RLITO000005
RLITO000005
Reference Data products and minor changes to TIPAIS Pathway generated CPs to improve operability
of the system.
87.A summary of some of the Releases appears in the chronology milestones for Legacy Horizon
below at [96] in the list of chronology milestones.
88.Horizon included what is called an Electronic Point of Sale System or EPOSS, which refers to the
part of Horizon within the branch. The parties referred to this as activity at the counter, or simply
“counter”. This term comes from the counter in the branch where a customer is served. In addition to
this part of the system, there was another element which the parties (for shorthand purposes in a
sense) referred to as the “back
office” function. This refers to the way in which the Post Office reconciles its transactions with its own
clients.
89.EPOSS must allow the counter staff to record that some goods have been provided to a customer,
compute the price of those goods, and allow the customer to pay the money required for all their
purchased goods, for instance by cash or a credit card. If a customer wants to carry out two or more
different activities in one visit to the counter - for instance, to settle a bill and to buy some stamps -
Horizon does not require the customer to settle the amount in two separate pieces. Horizon has the
concept of a customer carrying out a 'basket' of activities and settling the total amount due for the
basket in several ways - by one credit card transaction, by a cheque, by cash, or by a mixture of these.
90.At the stage when Horizon was introduced to branches - and indeed, this was a fundamental
distinguishing feature of Legacy Horizon, compared to Horizon Online - the data was held in branches
on terminals. Escher Group, a separate company, provided the Riposte Message Server (which was
responsible for storing all the data in the Post Office branches and replicating it to the Data Centres).
Terminals had two discs within them, one of which was termed a “mirror disc”, which retained all the
data of what occurred on that terminal. The branch would have a hard disc, which would store the
data for the branch and this would also be replicated on other counter positions (if a multi-counter
branch) or, in the instance of a single counter branch, stored to additional external storage on the
counter. It would then be passed on from the counter to the Horizon data centre where it is stored in
the CS messagestore.
91.In the summer of 2000, a ‘proof of concept’ was undertaken to investigate the integration of
internet technologies within the then-current Horizon System (which was being/about to be rolled out)
to support the delivery of banking transactions. This was called the Network Banking Solution. A
primary facet of the Network Banking Solution was the delivery of the banking transactions within the
already established Escher WebRiposte environment. The full installation and integration of this was
the task of Fujitsu (at the time ICL Pathway). WebRiposte was a message passing technology from
Escher Group that extended the functionality of Riposte Message.
92.IBM were selected for the supply of the Network Banking Engine which was designed to handle
the interface between the Horizon System and the agreed Financial Institutions; these are also called
Post Office’s clients, although in some documents they are referred to as ‘External Clients’. This was
described by Mr Coyne in the following way: “This communication was initially based upon Escher’s
Riposte messaging system. This was later supported by the WebRiposte system to accommodate
Network Banking changes and the Network Banking Engine supplied by IBM to deal with the increase
and change in type of business transacted in the branches. This was subsequently migrated to
Horizon Online. Horizon is therefore ultimately a composition of many sub-contracted components.”
RLITO000005
RLITO000005
93.It can therefore be seen that Horizon Online, although it includes some major differences (for
example where the data is stored; this is no longer in the branches as with Legacy Horizon) built upon
and adapted the existing functionality of Legacy Horizon. It was not an entirely new system.
94.Mr Coyne opines that the introduction of network banking (as well as developments in the EPOSS
function) brought with it a more complex enhanced architecture within which further systems to
ensure transaction integrity and reconciliation could be imposed. However, he also considered that
Horizon “originally stemmed from an inherited system and architecture with an initial, fundamentally
different design requirement” (namely the Pathway Benefits Payment Card project). Regardless of
that, there is no doubt that Horizon is a highly complex and multi-faceted system, because it has so
many different elements and interacts with so many other systems. It has been added to multiple
times over the years, as clients are added, new products are introduced, and indeed other sub-
contracted parties and software products have become involved. I do not consider that resolution of
the Horizon Issues requires an analysis of the reason or reasons why, in historical terms, Horizon may
have developed (or had inherent within it) any bugs, errors or defects.
95.The Horizon System that was initially installed in the branches in 2000 onwards is Legacy Horizon.
ICL Pathway was (at least partly) owned by Fujitsu and I use the term Fujitsu to refer to the computer
specialists with whom Post Office had contractual relations in respect of Horizon. I therefore make no
distinction between the years when ICL Pathway or Fujitsu was or were contractually responsible for
Legacy Horizon, or indeed with any of the other computer specialist companies such as Escher Group
who provided Riposte. Legacy Horizon involved the installation and use of Horizon terminals within
branches, which were used by the staff there (including the SPM) to transact all of the branch’s Post
Office business with the branch’s customers. If a customer in a branch wanted to purchase something
from the associated retail outlet, they would transact that business at a separate till, not at the
Horizon terminal. The National Lottery is an exception to this, as certainly now, it spans both parts of
the business of a branch.
96.Mr Coyne recited some “chronology milestones” for Legacy Horizon, which he explained were not
exhaustive but which he considered were milestone dates. Dr Worden also drew attention to changes
during the period 2000 to 2010 which he described as significant changes.
97.This chronology included the following:
1.Pathway (later Fujitsu Services) awarded contract for Benefits Payment Card May 1996
2.Horizon Pilot 1996
3.Pathway cited “greater than expected complexity” and “...major implications for the degree of
difficulty of the project” which ultimately lead to failure of the project.
4.Post Office Counters Ltd and Pathway sign agreement to utilise the project to automate Post Offices
July 1999
5.Horizon rollout 1999 - 2002
6.Core System Release - This included the introduction of Automated Payment Smart cards and APS/
TPS reconciliation. August 2000
7.Maintenance Release M1 - Prime purposes of the upgrade were the enhancement of the CSR+
Applications (APS, LFS, EPOSS, EPS, OBCS), enabling of the AP client variable day file transmission,
RLITO000005
RLITO000005
enhancement to Reference Data products and minor changes to TIPAIS Pathway generated CPs to
improve operability of the system February 2001
8.S04 Release Additional functionality on the Horizon Pilot outlets to permit the printing of forms
approx. July 2001
9.S06 Release Day D rectification measures - this included a new automatically generated broadcast
message to detail when counters at an outlet were offline. This was to be implemented in a staged
manner and included a receipts and payments fix June 2001
10.S10 Release Data centre and counter upgrade introduced unattended reboot facility at the counter
September 2001
11.B11 Release of the first network banking release, changed the version of Tivoli used by the whole
estate in approx. December 2001
12.$11 Release January 2002
13.B12 Release June 2002
14.$20 Release September 2002
15.B13 Release in approx. September 2002
16.In 2003 there were different changes introduced. Network Banking was introduced at this time, as
were the Data Reconciliation Services (DRS) and debit card processing. There were also the S30 and
$50 releases which immediately follow this entry.
17.S30 Mails Application /Escher Mails 3.3 package (1 Feb 2003)
19.S50 Release October 2003
20.860 Release Approx. February 2004
21.S52 Release March 2004
22.S70 Release October 2004
23.S75 Release (containing changes to support the changeover to use of NBX banking agents in
approx. October 2004
24,.IMPACT Programme
25.POLFS (a SAP-IS Retail System) was implemented in 2004
26.S80 Release Jan 2004 - Aug 2005
27.BI3 S80 T&T Harvester Agent accommodation was introduced. Mr Coyne thought that this may
have been approximately Nov 2004.
28.POLSAP rationalisation (rationalisation of disparate systems SAPADS and POLFS) between
2007-2009
28.Pension & Allowance Order Books were replaced by the Post Office Card Account in 2005. The Post
Office had always had business relationships with banks such as Girobank.
RLITO000005
RLITO000005
29.890 ‘Bureau Plastic’ accommodation introduced in January 2006
30.S92 Release March 2006
31.T10 Release August 2006
32.T40 Release January 2007
32. AP/ADC was introduced in around 2007/2008.
In around 2010, POLFS and SAP ADS were merged to make POLSAP.
98.Horizon was migrated to Horizon Online in 2010. Horizon Online is also called HNGX, but now it is
on a different platform since its move to Windows 10 and it is now called HNG-A. The letters “NG”
stand for “New Generation”. In 2010 there was no sudden change in the range of business
applications supported by Old Horizon in the branches. This range of applications has increased
continually over the lifetime of Legacy Horizon and Horizon Online. The main reason or rationale for
the change to online was to exploit advances in the underlying communication technology, and
improvements in its reliability. This meant that it had become possible to store all persistent data at a
centre rather than in the branch (on the counters), with the consequence that a branch could only
operate when communications were available, but the risk of failed communications was significantly
lower in 2010 than it had been 10 to 12 years before. Dr Worden considered that this change mirrored
the wider changes across the IT industry, where increased reliability of communications infrastructure
meant that applications could be dealt with in this way.
99.The centralised storage of transaction data allowed the architecture to become simpler in some
respects, with simpler management of the branches in the event of hardware failures or replacements
and other events. This is because in those cases branch data would not be lost. There was also no
dependence on Riposte data replication, which meant that Riposte could be removed entirely. Riposte
is therefore not used in Horizon Online. One feature of Horizon which emerges from all the evidence
provided is that it had a difficult interface with the systems of the Post Office’s clients, such as
Camelot. It is not necessary to go into the technicalities of why that was, and was recognised by the
Post Office (in this instance by Ms Van Den Bogerd herself) so far as Camelot was concerned by the
introduction of the Ping fix in 2012.
Horizon Online
100.A document entitled Counter Business Architecture document of 4 August 2017 stated that “The
objective of the HNG-X programme is to develop a system with structural and operational
characteristics that substantially reduce ongoing support and maintenance costs with respect to the
current Horizon system” and also “The overall requirement is that the business capabilities offered by
the current system (Horizon) are preserved in the new system (HNG-X). However, a limited number of
business capabilities will be revised based on a joint optimisation of business requirements and
system properties.”
101.The fundamental change was that in Horizon Online, no transaction data was held in any
persistent form in the branches. The same document explained that storage of transactional data
within counters in the branches led to the need for security mechanisms that affected “both the
structural complexity and operational performance of the Counter Business Application.”
102.In Horizon Online, before completion of a basket of customer services, that basket was
transmitted and had to be acknowledged by the BRDB and the basket could not complete successfully
RLITO000005
RLITO000005
at the counter until that had happened. This is why Horizon Online could not operate in the branches
without working communications. The main difference at the “back-end” was the existence of the
BRDB, which was the main persistent store of all transactions for all branches. Many business
applications in the back-end were unchanged (and were referred to as ‘legacy'), except for the need
for them to interface with the BRDB rather than with the previous Agent layer. Other copies of
transaction data continued to be stored in those applications.
103.The previous branch architecture had been based on Riposte, which provided functionality on
different levels. These included user interfaces, some business applications, and message storage and
replication. Because in Horizon Online, Riposte was completely removed, all elements of the branch
software were replaced. A great deal of the Legacy Horizon branch code had been written in Visual
Basic, for Horizon Online nearly all the branch software was written in Java. Java is a newer language,
and supported more modern programs (or what Dr Worden called modern programming paradigms).
This allowed a more modern software architecture in the branch, which did not have to be fitted
around the existing architecture of Riposte.
104.The new architecture is shown in the following diagram:
Physical Architecture
Presentation Penpherais
viruaxieation
Business Process Objects Business Dats Objects
(Rg. AADC scree) 10.9 Reterence ae
Loca Serwces:
ren * _ pigs = papas
QOQOOO0O
105.The top ‘Presentation’ layer is the one responsible for displaying information to the user and for
accepting the inputs. The next ‘Interaction’ layer provides the foundation or building blocks for this
interaction, such as menus. The effect of these two layers was to provide a user interface similar in
style to that which had been provided by Riposte in Old Horizon. This therefore meant that the
experience of the person, either SPM or assistant, using Horizon would be similar to what it had been
in Legacy Horizon, but the system would be using Java technology, rather than Riposte as before.
RLITO000005
RLITO000005
106.The ‘Business’ layer provides the functionality of the different business applications, which is
done in what is called an object-oriented fashion. This means that there are several general-purpose
software objects (i.e. modular blocks of software) with names such as customer, basket and payment,
which represent the behaviour that is required of those in the real world, but is done in a way that can
be easily reused in many different business applications. There are therefore certain core design
elements used for many different applications.
107.Business process objects and business data objects are more specialised to support the many
business applications. Business process objects support the sequence of steps which make up a
business process, and the business data objects hold the necessary data, which is presented at the
counter or stored. As has already been noted, this is generally not done by writing completely
different software for each business application, and a great many applications are driven by
reference data. This data defines the sequence of steps in completing each type of service for a
customer. This reference data-driven style of software is common modern practice. The theory is that
new applications can be supported just by adding new reference data, rather than by writing new
software. There are different capabilities in the business layer which it is not necessary to list.
108.The Services layer provides a set of software objects which provide services in support of many
business applications. Most of these services deal with organising information and sending it for
storage in the BRDB. The facilities for ensuring that each basket is zero-sum before it is sent, and
transactional integrity are provided generically in the services layer. These functions do not therefore
need to be coded individually in the business objects, and do not have to be built individually into any
new business application. The facilities for zero-sum baskets, which is one of the ways of complying
with the requirement for double entry bookkeeping, and transactional integrity are provided
generically in the services layer.
109.The Process Engine is a key component of the Services layer, and provides a simplified way for
the counter to provide services which involve a sequence of steps. The sequences of steps need not be
defined in Java code but are defined in a specialised Process Definition Language (PDL), which is
executed by the Process Engine. PDL was developed for Horizon Online by Fujitsu. The use of PDL
means that complex sequences of steps are simpler to define and test. This is an example of generic
data-driven software.
110.Some data are stored persistently on the branch hardware (as shown by the discshaped boxes in
the diagram) however, these data do not include customer transaction information. These include
business process definitions (definitions of sequences of steps in a process), other reference data, data
defining reports that can be output in a branch, and other information required to support operations.
The reference data is refreshed daily from the data centre. There are services which provide these
data to the other layers in forms that are convenient for them to use.
111.The Services layer of the branch architecture is the only layer which communicates with the data
centre, through the communications subsystem. The purpose of this layered approach is to provide
each kind of functionality (such as reliable and robust communication with the data centre) in one
layer only, and not have to reinvent it for many different business applications. In effect, the services
layer in Horizon Online now provides many of the services which were formerly provided by Riposte.
112.The Services layer also provides interfaces for online services, where it is necessary to contact
another non-Post Office IT system in order to provide the service to a branch customer. These online
services include banking, the use of credit/debit cards, mobile phone top-ups and other services.
RLITO000005
RLITO000005
113.Both the Branch Access Layer (BAL) and the BRDB are completely new elements of the Horizon
Online back-end. The principal function of the BAL is to exchange messages with the counter software
in the branches. It also has a role in checking that the information is conformant, logging of all
exchanges, and recovery from many kinds of error conditions. Because it has to handle more than 25
million transactions per day, the BAL has many design features to ensure high performance
(principally by distributing the load in parallel across many machines), as well as robustness - for
instance, through reliable and redundant hardware.
114.The branch database or BRDB is a large, high-performance Oracle database whose main function
is to store all customer transactions which originate in any branch. It also has features to ensure high
performance and robustness. These are, for example, through the use of transactional integrity and
recovery.
115.There are other notable milestones in my judgment of the Horizon System which are not included
in the list above at [96] include the following:
1.Migration to Horizon Online in 2010.
2.The discovery of what has become called the Callendar Square bug. This was experienced and
reported in that branch in 2005. Mr Godeseth’s evidence was that it had probably been in the Horizon
System since 2000. Ms Chambers’ entry in one particular PEAK in February 2006 was that “this
problem has been around for years and affects a number of sites most weeks.” The correct place to
put it in any chronology, on the basis of the evidence currently before the court, is 2000.
3.The introduction of what was called the Ping fix, which was in relation to the accounting with
Camelot for the National Lottery, which was initiated in 2011 and occurred in 2012. Much evidence
was given by Ms Van Den Bogerd, both in the Common Issues trial and the Horizon Issues trial, about
the improvement in accuracy of accounting regarding Lottery sales since the Ping fix was introduced.
4.In June 2014 ATOS started to provide 1st line support. The management and operations with regard
to Reference Data has been outsourced from within Post Office control to ATOS since the same date.
5.The move from HNG-X to HNG-A (which uses Windows 10 rather than Windows NT4). This change
to Windows 10 occurred in February 2017 and the roll out was administered by Computacentre.
116.Having provided what is essentially an overview of the Horizon System, both Legacy and Online,
it is convenient to turn to the main battleground between the parties concerning the subject matter of
the Horizon Issues trial. This is the presence of bugs, errors and defects within both Legacy Horizon,
and also Horizon Online. This can be done most conveniently by retaining the numbering of the
different alleged bugs from what was called “The Bug Table” in the 2nd Experts’ Joint Statement.
D. Submissions and Evidence
117.There were two features of the Horizon Issues trial which were somewhat unusual. This was a
lack of detailed evidence adduced by the Post Office in relation to the specific factual matters
recorded in PEAKs and KELs in particular, and the way that this situation was sought to be remedied.
This second element was by way of points explained or put “on instruction”, and also in submissions.
This is referred to in [69] to [71] of the judgment that accompanies this appendix. It is an obvious
point that submissions are not evidence, and vice versa.
118.The first of these, the putting of points “on instruction”, in this case related predominantly to the
way that certain matters were put to Mr Coyne in crossexamination to correct evidence of fact that
RLITO000005
RLITO000005
had emerged during the crossexamination of Mr Godeseth. The second arose in the context of the Post
Office’s Appendix 2 to its written Closing Submissions. The need for both, or either, arose in my
judgment because of the way Dr Worden approached his task in comparison with Mr Coyne. Because
he had done what he himself described as a “high level” analysis of Horizon, his written reports did
not deal with the detailed contents of so many PEAKs and KELs in the same way that Mr Coyne did.
Therefore, when the Post Office came to challenge Mr Coyne’s conclusions in closing submissions,
they could not simply meet his evidence with opposing expert evidence of the same type.
119.Lengthy written opening and closing submissions were provided by both parties, as is usually the
case in modern and complicated cases. The Post Office’s Closing Submissions in particular were very
lengthy, and included appendices, one of which, Appendix 2 at Section K2, itself ran to some 137
pages. This document attempted to collate material relevant to the Bug Table, in a logical form,
dealing with each bug in numerical order. It identified key documents, expert evidence (usually from
Mr Coyne), entries in joint statements by the experts, and other sections including “relevant
discussions during trial.”
120.During the oral closing submissions, the claimants identified certain instances in which positive
evidential assertions were made in Appendix 2 that were not, it was said, contained in the evidence.
Particularly given Appendix 2 was so large, it was simply not possible to resolve these objections
during that oral hearing. I therefore directed that the claimants serve a schedule identifying all such
passages to which objection was taken, so that a response could be provided by the Post Office. This
led to a “clarification” schedule served by the Post Office, which stated in respect of each challenged
passage the following:
Whether the passage was based on evidence in the trial, and if so the reference; any relevant
documentary evidence; if documentary, whether that document was deployed in the trial and if so
when; and/or whether the Post Office relied upon what Mr De Garr Robinson submitted would be a
suitable response in some cases, namely common sense.
121.That clarification document was itself rather lengthy. It contained a further 36 pages, with 175
different entries. Of those entries, 115 were to documents, and in only 28 cases had those documents
been referred to at the trial. Further, some of the submissions went rather further that the documents
supporting them justified. For example, on 8 Recovery Issues in the Bug Table (which was not
accepted as a bug) the Post Office has submitted the following in relation to a necessary software fix:
“It went to the Model Office on 1 June 2010 and Fujitsu have advised that it would have been rolled
out to the rest of the HNG-X pilot by mid-June.”
The supporting reference was to a document, PEAK PC0199000. However, the submission “Fujitsu
have advised that it would have been rolled out...” reads as though Fujitsu have provided evidence,
behind the scenes, and not called at the trial, about what would have or did happen. Further, the
supporting entry in the PEAK relied on for the entries for 1 June 2010 and throughout that month are
as follows:
“Date:01-Jun-2010 14:05:40 User:John Budworth
Pre-Requisite Reference Data and Data Centre BAL upgrades have been applied to live.
Please one-shot to the distribute and commit to Model Office branch 699010 by 15:30 today.
Please distribute tonight to the 50 branches in Pilot 50 RNT9566.xls attached as evidence.
RLITO000005
RLITO000005
Date:04-Jun-2010 17:20:16 User:John Budworth
Committed to Pilot 50 RNT9566.xls by MSS on Thursday June 3rd as verbally requested by RM.
Date:14-Jun-2010 11:57:50 User:John Budworth
Please distribute and commit to the balance of the HNGX estate
Date:14-Jun-2010 11:58:11 User:John Budworth
Evidence File Updated - RNT9566.”
122.This entry reinforces my view that the submission goes a little wider than the document's
contents suggests. However, this is a rare example, and in any event I am entitled to draw an
inference (that is, a common sense conclusion) that if the PEAK requests distribution to the entire
estate, that did occur. For the most part this type of example is rare, and it is not usually the case that
the submissions go further than the underlying documents suggest. I would like to record that I found
Appendix 2 very useful; it must have taken an enormous time to produce. There are some omissions
within it; for example, Dr Worden’s cross-examination is not always referred to. However, the task of
producing this appendix to the judgment has been rather easier as a result of it.
123.In order to use it however, it was not necessary to make a ruling on each and every one of the 115
different passages challenged by the claimants and the responses to those challenges by the Post
Office. I have only referred to the actual material necessary in order for me to come to findings on the
Bug Table, although I have considered the full content of Appendix 2 and its accompanying
clarification document. I have not given documents that were not referred to at all in the trial the
same weight as those that were, and I have not given those that were, the same weight as the actual
evidence that was deployed in the trial. I have however considered them all.
E. Bugs, errors and defects and their symptoms
124.The first three of the Horizon Issues have the heading “accuracy and integrity of data”. Given the
purpose of the Horizon System, both Legacy and Online, and the fact that it was the accounting
mechanism by which branch accounts were produced in terms of the Branch Trading Statement, the
presence of bugs, errors and defects with the potential to cause apparent or alleged discrepancies or
shortfalls in branch accounts was an extremely important element of the Horizon Issues trial. This is
plainly included within Horizon Issue 1.
125.The degree to which the Branch Trading Statement accounts did or did not, in law, have the effect
of settled accounts between principal and agent was addressed in
Judgment (No.3) on the Common Issues. The majority of the Horizon Issues - and certainly the first
three - were all aimed at the same principle, namely (in summary) that of computing and accounting
accuracy of the Horizon System in both Legacy and Online form. It was doubtless for this reason that
so much of the cross-examination of the two experts dealt with the presence of bugs, errors and
defects.
126.In my judgment, the most convenient approach is to consider each of the alleged bugs, errors and
defects in the Experts’ so-called Bug Table in order to come to a judgment on the existence, or
otherwise, of each. Horizon Issue 1 requires this, and in addition, Horizon Issue 3 should be answered
in the context of whatever the answers are to the different component elements of Horizon Issue 1.
This also means Horizon Issue 3 can be considered in the knowledge of how many, if any, of the
RLITO000005
RLITO000005
alleged bugs errors and defects in fact existed in Horizon over time, and what their potential effect
was. By the end of the Horizon Issues trial, in reality there was less in dispute between the parties on
the different bugs in the Bug Table than appeared at the beginning of that process. This is not to say
that there were many concessions, certainly in terms of the existence of bugs, although there were
some. However, the expert evidence of both Dr Worden and Mr Coyne, although they differed in their
conclusions, was not so far apart on the factual existence of certain bugs, errors and defects. Given
the use during the trial of so many PEAKs and KELs, which contained contemporaneous records at
Fujitsu of the effect upon SPMs of different features within Horizon, this is not so surprising. Some
overarching themes remained, for example the difference between transient and lasting impact of
bugs. These are addressed in the body of the judgment itself.
127.Turning to the 29 different entries in the Bug Table in turn, and in summary terms only (as to
reproduce references to all of the documentation, evidence and submissions on each would lead to a
very lengthy analysis of each such bug, which in my judgment is disproportionate and not necessary),
my findings on these are as follows. Because I am using the same numbering as in the bug table, the
following bugs are not dealt with in this appendix in chronological order, but in the order chosen by
the experts in their joint statement.
1, Receipts and Payments Mis-match bug.
128.This bug occurred in 2010. It is a bug which appeared in Horizon Online. It was agreed in the Bug
Table as an “acknowledged bug” which had an impact on branch accounts. Dr Worden added in his
comments that “Therefore, the extent of this bug is well established, in the GJ analysis.” GJ is Gareth
Jenkins. It is accepted by the Post Office in paragraph 7 of Appendix 2 as one of a number of “bugs
with lasting impact (although they were resolved)”. In the summary in Appendix 2, the Post Office
stated that “In the event, however, this bug resulted in transient impact only.” I have dealt with the
concept of transient impact in the main judgment on the Horizon Issues.
129.The effect of this bug was identified upon the accounts of approximately 60 branches. Dr Worden
and Mr Coyne, in the agreed entry in the 2nd Joint Statement, had stated that “the number of distinct
bugs, for which the experts have seen strong evidence of the bug causing a lasting discrepancy in
branch accounts, is between 12 and 29.” (emphasis added)
130.Dr Worden had therefore seen “strong evidence of lasting impact” in respect of this bug. This was
one of his 12. The same joint statement said it had affected “approximately 60 branches”. Mr Coyne in
his report used the more precise number of 62. Of these 62, two branches were affected twice.
131.The bug related to the process of moving discrepancies into the local suspense account. The
majority of incidents are recorded as occurring between August and October 2010. The bug was
documented in a report from Mr Gareth Jenkins dated 29 September 2010 where it was stated:
“This has the following consequences: There will be a receipts and payment mismatch corresponding
to the value of discrepancies that were lost. Note that if the user doesn't check their final balance
report carefully they m: naware of the i ince there is no explicit message when a receipts
and payment mismatch is found on the final balance (the user is only prompted when one is just
detected during a trial balance)” (emphasis added)
132.This issue is reported as causing discrepancies that disappeared at the counter or terminal when
the branches followed certain steps, but which persisted or remained within the back-end branch
account. It is therefore something which is contrary to the principle of double entry bookkeeping, and
RLITO000005
RLITO000005
should plainly not have occurred. The issue occurred when a branch cancelled the completion of the
trading period and then, within the same session, rolled into a new balance or trading period. Because
the discrepancy disappeared at the counter, the SPM would not know that the discrepancy existed.
133.The lack of an explicit message to the user, in my judgment, simply compounds the problem. This
bug was acknowledged by the Post Office in its letter of response to the letter of claim sent in relation
to the litigation. That letter of response was dated 28 July 2016. I am unaware of any acceptance by
the Post Office of this bug prior to that date, but in any event that is not relevant to the substantive
issue of answering the Horizon Issues.
134.The Post Office in its Closing Submissions stated that this bug only occurred if there was “an
unusual sequence of events”, namely as follows. If a branch had an unresolved discrepancy, when
balancing the stock unit, and if the SPM pressed the Preview or Print button to produce the Trial
Balance Report, this would cause the counter to return to the rollover screen. Having checked the
Trial Balance Report, if the SPM then pressed the rollover button, Horizon would then ask the SPM
whether they would like to transfer the discrepancy to the local suspense account or cancel the
rollover.
135.If they chose to cancel the rollover, this would cause the system to return to the rollover screen,
allowing the SPM various choices, including cancelling the rollover process or the choice of again
pressing the Preview or Print button to produce the Trial Balance Report. If the SPM then proceeded
to rollover to a new Trading Period in the same session the fault would occur. This was a bug in the
code that was triggered by the ‘cancel’ button being pressed and this incorrectly caused the
discrepancy to be cleared so far as the user in the branch was concerned.
136.1 do not accept that this was a particularly unusual sequence. If there was a discrepancy, I do not
see why it is suggested it would be unusual for a SPM to decide to cancel the rollover. This is, in my
judgment, something which could be expected to
happen sometimes, and plainly the absence of any message to the user could potentially contribute to
this.
137.The Post Office also, in the main body of its submissions, stated that Fujitsu monitors for receipts
and payment mismatches, and investigates such occurrences. However, that does not mean that this
bug does not exist; rather to the contrary. The fact that Fujitsu was supposed to watch for it, and then
investigate it, does not affect its existence. Further, the submissions also relied upon the fact that
instances of this bug “were collected through [Fujitsu] reports without the need for an SPM to report
an issue.” In my judgment this overstates the factual position in the Post Office’s favour. PEAK
PC0204263 dated 16 September 2010 shows that this PEAK was raised by a “customer call” on 13
September 2010, which means a SPM, who had a discrepancy of £109. This was given call priority B
by Fujitsu. The SPM did report the discrepancy; they could not report a payments and receipts
mismatch because they would not have known either about the existence of the bug, or its effects.
138.That this was a problem that required a code fix is shown by the following entry in the PEAK
dated 24 September 2010.
“HNGX CODE FIX
FIX DESCRIPTION
Described Above
PROPOSED BRANCH
TBD
COUNTER JAVA FILES CHANGED
None.
COUNTER PDL FILES CHANGED
RolloverStockUnitBLO.pdl updated COUNTER REFDATA FILES CHANGED
None.
SHARED CODE FILES CHANGED
None.
BAL JAVA CODE FILES CHANGED
None.
SQL FILES CHANGED
None.
OTHER FILES CHANGED
None.
APPROPRIATE CODE COMMENTS
YES
DEPENDENCIES.
None.
RELATED PROBLEMS
None
UNIT TESTING EVIDENCE screenshots attached.
REGRESSION TEST CLASS NA.
BACKWARDS COMPATIBILITY
its a reference data change hence backward compatibility test is required on LSt
DEVELOPMENT DOCUMENTATION
None.
REQUIREMENTS DOCUMENTATION
None.
RLITO000005
RLITO000005
RLITO000005
RLITO000005
HELP
None.”
139.The Counter PDL files were changed, and as this was a reference data change, certain checks
were needed which is called a backwards compatibility test. This is to ensure that the fix itself does
not cause other problems. This led, eventually on 5 October 2010, to the entry in the PEAK of
“Product Error Fixed” after a patch had been released. A patch is a change, or set of changes, to a
programme or its supporting data, and it is something that changes the operation of the software. The
entry of 14 October 2010 states:
“Ths Reference Data hot fix for this PEAK has completed LST testing and has been passed to the
Reference Data Team for live deployment.
Note this hot fix will only work when a counter receives the 02.12 counter upgrade that is currently in
live pilot. (Overnight package COUNTER _X0212 56_1 or immediate COUNTER APP 56_1).
[End of Response]
Response code to call type L as Category 71 -- Final -- Fix Released to Call Logger Routing to Call
Logger following Final Progress update.
Service Response was delivered to Consumer.”
140.1 find that this is a bug with potential lasting impact, and indeed it did cause actual impact. The
fact that the Post Office, once it was discovered that it was a bug, corrected the accounts affected by
issuing TCs (and in one instance Fujitsu injected the necessary correction into the branch accounts)
does not reduce its importance. A reference data fix was necessary to correct it. As explained above,
reference data was part of the architecture in Horizon, and in my judgment is part of the software.
The fact that aspects of the system such as the SQL files or the BAL Java code was not changed does
not mean that this was not a software defect. Even though the underlying code was not re-written or
corrected does not minimise the impact of this
bug which required a change to the reference data as shown in the PEAK above, and other of the
contemporaneous Fujitsu documents.
2. Callendar Square/Falkirk bug.
141.It is agreed that this bug occurred between the years of 2000 and 2006, although there is an
issue about when it stopped which I deal with at [149] below. It is a bug which was present in Legacy
Horizon. It too was agreed by the Post Office in the letter of response in the litigation of 28 July 2016.
It was also agreed in the Bug Table as an “acknowledged bug” which had an impact on branch
accounts. It is accepted as a bug by the Post Office in paragraph 6 of Appendix 2, but is said to have
had “transient impact”. Dr Worden added different comments, including that “the bug arose from a
fault in the underlying Riposte software, so it is not surprising that it took Fujitsu some time to
understand it, or that they had to rely on the suppliers to fix it. It does not show poor system design or
support by Fujitsu”. That latter sentence is in my judgment a little surprising. It is not relevant to the
Horizon Issues whether fault or defects can be laid at the door of Fujitsu, Escher Group (who supplied
Riposte) or any of the other specialist companies who provided software for different parts of Horizon.
Dr Worden seemed to me to be very defensive on Fujitsu’s behalf. The Horizon Issues are about the
operability and functionality of Horizon, and there are no findings made in the judgment or this
RLITO000005
RLITO000005
Technical Appendix regarding which specialist company is to blame for the presence of any bugs,
errors or defects.
142.Riposte software was part of Legacy Horizon, although it is not used in Horizon Online, as one of
the main changes between the two different types of system (Legacy, and Online) was specifically the
way data was stored and messages sent. Legacy Horizon used Riposte and it was a central feature of
that system. Riposte is not used in Horizon Online. The fact that Riposte is a product designed and
supplied by Escher, and not Fujitsu, is not relevant to the dispute between the Post Office and the
claimants. In any event, and as accepted and recited in paragraph 2 of the Post Office’s Appendix 2 to
its closing submissions dealing with this bug, “Mr Coyne asserts that Bug 2: Callendar Square is a bug
with lasting financial impact and in JS2, Dr Worden appears to agree that there is strong evidence of
this....”
143.In my judgment Dr Worden does more than “appear to agree”. Given the entry in the experts’
joint statement, he does in fact agree. This bug occurred when a SPM was transferring cash between
different “stock units”, which means positions.
144.One of the Post Office’s submissions on this accepted bug merits quotation in full. The Post Office
in its Appendix 2 stated:
“Fujitsu was aware of the wider Ripsote issue for quite some time. Its various manifestations were
being identified and resolved long before the Callendar Square PEAK was raised.”
145.This is undoubtedly correct; Fujitsu was aware of the bug “for quite some time”. I do not consider
that this assists the Post Office’s position on the Horizon Issues. The knowledge Fujitsu had about
issues with Riposte was not shared with SPMs, so far as I can tell. This is also the bug that a Fujitsu
employee, Anne Chambers, stated in an email in February 2006:
“Haven't looked at the recent evidence, but I know in the past this site had hit this Riposte lock
problem 2 or 3 times within a few weeks. This problem has been around for years and affects a
number of sites most weeks, and finally Escher say they have done something about it.”
146.The Post Office stated in its submissions that “Fujitsu’s automatic reports identified instances of
the Callendar Square bug without the need for each SPM to call in with a report”. That submission
ignores the numerous examples in the numerous PEAKs where SPMs did in fact call in with a report
of a discrepancy. The Post Office also, entirely incorrectly in my judgment, complains in its Closing
Submissions that it did not have the time to cross-examine Mr Coyne fully on this bug. It submitted
the following:
“Mr Coyne notes from the documents that the Calendar Square bug was “operating and resident in
the system for years without any comprehensive linkage being observed by Fujitsu”. However, this
provides only an incomplete picture, as Post Office would have demonstrated if it had had the time it
would have needed to cross examine him on this bug (that time was taken by Mr Coyne’s
extraordinary changes in evidence on the afternoon of Day 17, the final afternoon of his cross
examination).”
147.1 reject this submission for two reasons. Firstly, it was a matter for the Post Office’s legal team
how long it spent on each subject with Mr Coyne, who was crossexamined for four days. It might have
been sensible to have started with the number of bugs for which Mr Coyne was contending, and then
cross-examined on each or some in detail after that. The Post Office chose to leave the total number of
bugs until the last day; there is nothing wrong with that, but it is not correct to state that there was
RLITO000005
RLITO000005
insufficient time to cross-examine on this bug (or any bug) and try to blame Mr Coyne for it. Secondly,
the submission entirely ignores the evidence of Mr Godeseth, called by the Post Office, who expressly
accepted that the bug had probably been present since 2000.
148.The Post Office also submitted that “a Peak would be raised and the SPM made good in the
ordinary course” if this bug affected branch accounts. There are two points that this submission fails
to address. Firstly, the Horizon Issues concern bugs, errors and defects with the potential to affect
branch accounts. The Callendar Square bug is clearly one of these. Secondly, the “ordinary course” in
this instance seems to be that not one of the SPMs across the Post Office branch network were told
about the existence or effect of the Riposte lock problem, which Fujitsu recorded (by Anne Chambers
in February 2006) had “been around for years”. In my judgment, the ordinary course ought to have
been providing SPMs with notification of the existence of this bug and its effects. SPMs and the Post
Office have agreements in place (at the time of this bug the SPMC) that govern their legal
relationship. Judgment No.3 makes clear what the scope of the parties’ legal obligations are.
149.One specific factual point of disagreement between the parties was the period of time when the
effects of this bug were no longer experienced. The Post Office maintains this came to an end in 2006,
and the claimants submitted that it ran until Riposte was no longer used, namely 2010 when Horizon
Online came into being. One passage of re-examination of Dr Worden is pertinent in this respect:
“Q. You say another symptom of the Riposte problem. Perhaps you could explain a little bit what you
mean by that?
A. Well I did explain and I will explain again. That there is a problem in Riposte which leads to a
failure to replicate -- failures to release a lock I think -- and then on certain occasions that leads to a
short-term failure to replicate. On other occasions there is a so-called event storm during which there
is a longer term failure to replicate and during these failures to replicate all sorts of different things
may happen, for instance they double transfer into a stock (inaudible), a precise Callendar Square
phenomenon; whereas other things such as system freezes, I can’t remember all the other details, but
(emphasis added)
150.By “this whole period” Dr Worden was, in my judgment, clearly referring to the period 2000 to
2010. Whether the short-term failure to replicate, and the “event storm” with its longer term failure to
replicate, are seen as one bug in Riposte (which is the way the experts approached it) or two sub-
elements of one bug, which it could be (given one is a short term failure, and the other a long term
failure, both being failures to replicate) does not for these purposes matter. In my judgment, the
period when the effects of this occurred are 2000 to 2010. Dr Worden’s evidence does not support the
Post Office’s position that this ended in 2006, and is indeed to the contrary.
151.1 find that this is a bug with potential lasting impact, and indeed it did cause actual impact.
3. Suspense Account bug.
152.This is a bug in Horizon Online and its identified years of effect are 2010 to 2013. It was also
agreed in the Bug Table as an “acknowledged bug”. This bug was the third of the existing bugs
acknowledged by the Post Office in its Letter of Response on 28 July 2016.
RLITO000005
RLITO000005
153.It is accepted by the Post Office in paragraph 7 of Appendix 2 as one of a number of “bugs with
lasting impact (although they were resolved)”. Dr Worden’s comments do not include that it had an
agreed impact upon branch accounts. This is because of the concept he introduced (expanded in his
reports) of “transient effect”. What this meant was that if there was an impact on branch accounts by
something within Horizon, but that was then corrected by a TC, he concluded it had a “transient
effect” upon branch accounts and dealt with it differently. I address this concept in the substantive
judgment under “lasting and transient effect”. Other comments of his in relation to this acknowledged
bug are:
“Tt was a transient effect arising not from a fault in the software, but from a change in database
archiving policy in 2010. The delay in correcting it arose from a failure of communication between PO
and Fujitsu. Because the bug would only manifest itself annually for any affected branch, the effects of
this delay were not widespread.”
Peak PC0223870 shows that Fujitsu were able to identify the branches affected, even when
Subpostmasters did not report it. There is evidence that the branches were compensated, as I would
expect from the normal error correction processes.”
154.The Post Office accepts in its Closing Submissions that Fujitsu identified affected branches by use
of “historical data”, in other words, it had to research it as an investigation was required. Mr Gareth
Jenkins in 2013 prepared a note entitled “Local Suspense Problem” which identified that, at that
stage, 14 branches had been affected. He stated:
“The root cause of the problem was that under some specific, rare circumstances some temporary
data used in calculating the Local Suspense was not deleted when it should have been, and so was
erroneously re-used a year later. When the SPMR was asked to clear Local Suspense the actual (ie
incorrect) amount was recorded in the Audit Trail. This means that there was no corruption in the
audit trail and it accurately reflects the transactions that occurred in the Branch.
If the BTS from the previous period was taken to provide a set of Opening Balances and all
transactions that were logged to the audit trail during the period were taken as adjustments, then this
would show the correct value that should be in the Local Suspense account.”
155.He also stated, in a passage that is of interest in terms of the dispute between the parties about
the use of Audit Data or management data such as Credence (the Post Office maintains that the latter
are sufficient, the claimants insist the former are the relevant accurate records) the following:
“As well as passing these Local Suspense transactions to the normal accounting tables that are used
to update POL SAP and Credence, they are also written to a table in the Branch Database that is used
to support the printing of the Branch Trading Statement (BTS) after that Branch has been fully
Balanced.”
156.This demonstrates, in my judgment, that in this instance Credence would have been wrong.
Further detail of the problem explained by Mr Jenkins in the same report was:
“In April 2011 a problem was found with the archiving strategy related to Stock Units that have been
deleted in a Branch. A consequence of this is that some changes were made to the archiving strategy
on 3rd July 2011. An unintended consequence of this change was that any Branch that deleted a Stock
Unit at the end of 2010 which had local suspense transaction in that Stock Unit before it was deleted
were left in the table used for constructing the BTS. This meant that as Trading Periods cycle around
each year, these BTS records became visible in 2011 when the same Trading Period was reached.
RLITO000005
RLITO000005
The effect of these old records was that after the BTS was produced an incorrect figure was generated
for the Opening Balance of the Local Suspense Account for the following period. This amount
corresponded to the value of the historical record.
These orphaned records were created between 16th November 2010 and 9th December 2010.”
157.This meant that the figures for the preceding year were effectively recycled as correct figures for
the subsequent period. The Note prepared by Mr Jenkins makes it clear that, despite how the Post
Office sought to present this in their submissions, this problem persisted.
158.The document states: “This problem was not reported to Fujitsu in 2011/12 and only affected a
small number of Branches and only for a single Trading Period. However the two branches with the
largest discrepancies did report the issue to Post Office Ltd who could see the impact of the problem
. i ff inf f i Fuii
investigate further. At the same Trading Period in 2012/13, the problem re-occurred and this time one
of the affected Branches reported the problem to Fujitsu on 25th February 2013 (Peak 223870)
resulting in a detailed analysis of this issue and finding the orphaned BTS records. The root cause was
determined by 28th February 2013 and a preliminary report was sent to Post Office Ltd. A further
update was sent on 14th March 2013 with a full analysis of the issue and all the affected branches.”
(emphasis added)
159.It is somewhat disingenuous, in my judgment, for Mr Jenkins in this note effectively to blame
others (which appear to include SPMs and the Post Office), rather than Fujitsu, for this problem not
becoming known about until some years later. The trial did not explore communications between the
Post Office and Fujitsu on this subject in any great detail, but SPMs are not left with a choice of
whether to report something to the Post Office, or whether to report it to Fujitsu. One branch had a
loss of approximately £9,800 (the exact sum was £9,799.88); some were £161 or less, and another had
a gain of £3,100.
160.Mr Coyne’s view is that this bug could have affected branches prior to Fujitsu’s investigations. He
also suggests that it is unlikely that Post Office and/or Fujitsu have captured its full effects. My
findings on that are as follows. Fujitsu appear to have conducted a thorough investigation in 2014,
and there are spreadsheets showing the branches identified as having been affected. A code fix was
produced in 2013, and this was first piloted to 50 branches on 21 October 2013 and deployed to all
branches on 29 October 2013. The bug could not have arisen prior to 2010 in any event as that was
the date that Horizon Online was initiated.
161.1 do not know if any of the claimants in this litigation have losses which, when their claims are
fully tried, they claim have arisen in the relevant period and as a result of this type of incident, but
whose branches do not appear in the Fujitsu list. If so, that will have to be addressed at future trials.
However, there is no evidence before the court currently of branches other than those included in the
spreadsheet having been affected in the relevant period.
162.1 find that this is a bug with potential lasting impact, and indeed it did cause actual impact.
4, Dalmellington bug/Branch Outreach Issue.
163.This is a Horizon Online bug. It was not expressly acknowledged as a bug in the Bug Table, but is
effectively accepted by the Post Office as a bug, as it appears under the heading in its closing
submissions as one of “the following bugs had transient impact.” It is therefore accepted as a bug by
RLITO000005
RLITO000005
the Post Office in paragraph 6 of Appendix 2, but this is subject to the point regarding “transient
impact”. For some time during the trial the Post Office would refer to this as “an issue”, and avoid use
of the word “bug”. I find that it is a bug - that much does not seem to be controversial, as paragraph 2
of the detailed part of Appendix 2 relating to this bug states the following:
“This is a bug which Mr Coyne states has lasting impact on branch accounts. Post Office submits that
there was no lasting impact on branch accounts.”
164.It is therefore the nature of its impact on branch accounts that is now in issue. Its effects were
experienced between 2010 and 2015, although it is named after the branch where its occurrence led
to its discovery as a bug in 2015. Fujitsu performed an investigation as a result of the impact of this
bug in 2015 at the Dalmellington branch. The investigation shows the following:
Feb 2010 to Jan 2011 65 incidents
20116 incidents
20129 incidents
20137 incidents
20149 incidents
201516 incidents
165.The same document showed fixes applied in April 2010, January 2011 and January 2016. It also
showed only one call to SSC at Fujitsu in 2015, and none for any of the years 2010 to 2014 when the
figures above show there were 93 occurrences of this bug.
166.It is what is called a “cash remming error” and happened in certain circumstances. It was
explained both by Mr Coyne, and further by Mr Godeseth in his evidence at [428] onwards, identified
in the judgment itself. Basically, places where a full time branch Post Office is not justified, such as
remote areas, may have such services on (say) half a day per week, in a mobile van or a village hall.
These are called “outreach” branches, and they are connected to (or part of) a core branch, run by the
same SPM. This means that the SPM is in charge of both the core branch and the outreach branch.
The Dalmellington bug affected such branches.
167.A PEAK was created on 13 October 2015 in respect of Dalmellington, namely PEAK PC0246949.
The SPM in that branch had attempted to transfer cash to its outreach branches. The SPM sought to
transfer £8,000 and ended up with a £32,000 transaction. This problem involved issues relating to two
different scripts, the Post Log On script and the Forced Log Out script. The latter did not close down
the former correctly (or in software terms, did not lead to the former script correctly closing down the
operation) which meant that the script was left on the stack of incomplete processes. If the SPM then
logged back into the counter to perform a transfer of cash
from their core branch to the outreach branch, the “pouch delivery” screen would still show, with
“enter” being enabled. If the SPM were then to press “enter”, then the value of the transaction would
repeat. As expressed in a confidential briefing given to the Post Office by Fujitsu:
“It relies on a mechanism which checks the stack of incomplete processes to see if it is complete. Due
to the fact that the stack is not empty (following the first problem) it thinks it has not finished and as a
result attempts to repeat the last part of the script, which in this case is to record the remittance
transactions and print the receipts”
RLITO000005
RLITO000005
168.This would result in the amount of cash (in the case of Dalmellington, £32,000 as enter was
pressed a further three times) being ‘Remmed Out’ of the core branch. Because each “pouch” which
holds cash has its own ID, Fujitsu researched the number of times that duplicate pouch IDs had
occurred in the BLE files.
169.There are two points upon which the Post Office relies in respect of this bug. The first point is
that the Fujitsu investigation in December 2015 showed that 112 potential occurrences of the
Dalmellington issue had been identified in the previous 5 years. Of these 112 potential occurrences,
108 items were corrected at the time, either by Post Office issuing a Transaction Correction or the
SPM reversing the duplicate Rem In. This left 4 remaining potential occurrences, for which Fujitsu
had not yet established the outcome. However, the claimants rely upon two points of their own in this
respect. One is that this showed that even though the bug had occurred multiple times and affected
branch accounts, neither Fujitsu or the Post Office even knew it was a bug prior to late 2015. In my
judgment, that is a valid point for the claimants to make. The second feature is that TCs are, as has
been identified in the Horizon Issues judgment itself, outside of the Horizon System.
170.A third feature is that the Post Office did not disclose the existence of this bug more widely.
171.The Post Office also submits that “once the full extent of the Dalmellington issue was understood,
Fujitsu produced a code fix without delay and it was rolled out in January 2016.” However, it cannot
seriously be suggested that correction of the bug was “without delay” when the full history of the
PEAKs are considered, with incidents going back to 2010. By the time this bug was remedied in early
2016, the mediation scheme had been brought to an end some time before (this occurred in March
2015) but some of the branches affected by the Dalmellington bug were within the scheme, according
to the Fujitsu investigation. However, none of the periods when the bug hit matched what were
recorded in the investigation as “Mediation dates”. Not only that, but for the period up to the
commencement of the mediation scheme, the Post Office was meeting a case brought by the claimants
that there were widespread bugs within Horizon which impacted the branch accounts. Dalmellington
shows that there was such a bug, which lay undiscovered (although its effects were apparent) until
2015.
172.The Post Office also submitted the following:
“As put to Mr Coyne by Mr de Garr Robinson, from an outsider’s perspective, you would not be able to
tell the difference between someone remming in twice by accident, due to human error and someone
remming in twice because of an error on screen.”
173.1 am not sure why this submission seeks, as it does, to deploy that submission as a point in the
Post Office’s favour on the Horizon Issues. It is rather a point in the claimants’ favour, as it means that
there was no visible sign that the bug had done what it had done. It could be misinterpreted as human
error; its effects looked exactly the same, and Mr Godeseth in his evidence accepted this. It is also not
a point in the Post Office’s favour that any known instances of this bug were corrected by means of
TCs, or “by the SPM himself reversing the REM in some way”, which is how it was put in cross-
examination.
174,Two of the four unknown outcomes were for £25,000, and £2,500. The Post Office, in its closing
submissions, explained why neither of these were in fact incidents of the Dalmellington bug because
they were not outreach services, and also although bar codes for cash pouches were supposed to be
unique, there was a small batch of them that were not. The documents from which these submissions
were drawn show that the two branches with these discrepancies were those with FAD codes 209311
RLITO000005
RLITO000005
and 157242. However, the documents also report that both these branches had duplicate IDs for
remming in of cash, that there had been “no contact from the branches raising issues for the period in
question (February and March 2013)” and come to the following conclusion (although I consider it is
worth noting that the SPMs in each of these branches do not appear to have been asked, nor does the
ARQ data appear to have been used):
“Post Office concludes the issues at the branches have arisen as a result of remittances pouches
received at the branch entered manually which had the same barcode id. Thus creating duplicate
entries which Fujitsu highlighted as part of the BLE files checks. However, in these instances from the
available evidence Post Office concludes that the correct amount of pouches were delivered, accepted
and entered on Horizon. This is supported by the fact that there has been no negative impact in the
branch accounts and no record of an issue raised by the branches with Post Office.”
And “Moreover, these two branches have resolved in branch the issue encountered without referral to
Post Office via NBSC and FSC or Fujitsu.”
175.1 find that this bug was present in Horizon Online from the date that Online was brought in,
namely the year 2010. In my judgment, this is absolutely clear, and does not appear to be in dispute
given the dates in the Fujitsu investigation.
176.This bug would, in my judgment, have had an impact upon branch accounts - it is the extent of
that impact that is in issue, whether it was lasting or transient. Dr Worden relied upon what he
described as “a well-tested process of reconciliation and TCs to detect and correct errors in cash
remming (used 20,000 times per year)” and said it was “straightforward for Horizon to detect any
discrepancy between a “rem out” and the corresponding “rem in” (a mismatch arising either from a
miscount, or a multiple count of a pouch) and then a TC can be issued.” He also added that “this
process catches and corrects remming errors, whatever their cause - including if they arise from, or
are provoked by, software faults.” He therefore implicitly accepted that the Dalmellington bug was a
software fault, although he did not say so in terms. I find that it was plainly was. I also reject his
evidence that it was “straightforward for Horizon to detect any discrepancy”; the evidence about this
bug shows that Horizon did not detect it in these many cases.
177.Dr Worden was also reliant upon the process of TCs to correct it. Further, the bug was there for 5
years and was not discovered, although its effects doubtless were experienced during that period.
178.It is correct to record for completeness that all known impacts of this bug appear, on the face of
the evidence before the court at the Horizon Issues trial, to have been corrected by the Post Office,
with the exception of the two branches identified which were not outreach branches. The documents
show that branch 209311 had no rem issue on, or about, the date of the duplicate ID incident of
£2,500 (which was 1 March 2013) and the SPM had not raised any issue with the Relationship
Manager Mr Andy Winn. There was no record of any TC having been issued for £2,500 and an audit in
August 2013 had shown a shortfall at that branch of £350.55.
179.The other branch, 157242, was operated by a SPM who transferred to a “Main agreement” in
May 2014. Such an agreement did not feature in the Common Issues trial but I assume it means it is
now a Main Post Office and not a branch. Cash rem in issues were shown at the branch in February
2013 but none of the same type as the £25,000 duplicate rem in, and none on the exact date in
February which was 18 February 2013. Although two of these were for very substantial amounts, none
were for £25,000 and Mr Winn again said that no issue had been raised. I recite the findings in
respect of both these branches for completeness.
RLITO000005
RLITO000005
180.1 find that this was a software bug that impacted upon branch accounts. In my judgment, it had
lasting impact upon branch accounts. The fact that the occurrences of it (over 100 in a 5 year period)
were corrected by means of TCs does not mean that it ought not properly to be characterised as a bug
causing lasting impact to branch accounts. However, there were no accounting shortfalls ultimately
experienced by SPMs whose branches suffered from the effects of the Dalmellington bug as these
were corrected by means of TCs. SPMs whose branches suffered the effect of this bug were not
therefore required to pay the sums the subject of the shortfalls to the Post Office on the evidence
before the court.
5. Remming In bug.
181.This is a Horizon Online bug. It was present for about five months in 2010 between March and
August. It was not expressly acknowledged as a bug in the Bug Table. However, Dr Worden accepted
it was a bug or defect in his cross examination on Day 20 and it is accepted as a bug by the Post Office
in paragraph 6 of Appendix 2, but is again said to be one that had only transient impact. The years it
was said to have been present in terms of effect was “March - August 2010 and recorded as fixed
approx. 2011”. Mr Coyne said there were 14 branches affected. Dr Worden’s entry in the Joint
Statement relied upon TCs, and stated “As for the Dalmellington bug, above - PO had a robust process
for detecting and correcting remming errors, whatever their origin. So, there were no lasting effects
on branch accounts.” In paragraph 2 of the detailed part of Appendix 2 relating to this bug, the Post
Office submitted that “any discrepancy would be transient as instances of this bug are caught by
automatic reporting.”
(emphasis added)
182.The Post Office relies upon automatic reporting to catch incidents of this bug, and also maintains
that Mr Coyne has conflated two issues, one relating to PEAK PC0203085 (which it terms “Issue 1”)
and the other related to PEAK PC0195380 and other PEAKs associated with KEL acha4221Q (which it
refers to “Issue 2”). Both of these relate to the remming in of cash, and both result in a cash pouch
being recorded twice in error. However, they are said to be different issues because the sequence of.
steps taken by SPMs to trigger them are significantly different.
183.The process of “remming in” works as follows. SPMs receive pouches of cash which are sent to
the branch from the Post Office cash centre. A certain amount of cash is held in each branch for the
purposes of transacting Post Office business. Each pouch has a unique barcode that needs to be
scanned by the branch when it is received. This automatically looks up the contents of the pouch so
that the branch can confirm that the physical contents of the pouch match up to the record on
Horizon. It is for this reason that each code is supposed to be unique.
184.The way the system is supposed to work in theory is best taken from the Post Office’s closing
submissions: “If there is a difference between what the system says is in a pouch of cash and the
amount of cash actually in the pouch, the Subpostmaster should raise the issue with NBSC in order to
get a Transaction Correction. A remming error leads to a mismatch between the amounts of cash
remmed out to one place and the amounts remmed in from another. Remming errors are a violation of
Data Entry Accounting and are picked up by Horizon.”
185.That latter sentence was said by the claimants, after the written submissions were received, not
to be present in any of the evidence used in the Horizon Issues trial. This point is dealt with in general
terms at [120] above. In response to that challenge, the Post Office clarified its submissions and relied
upon Dr Worden’s report, which was in general terms in respect of the general principle of double
RLITO000005
RLITO000005
entry accounting; the HNG-X architecture document, which again demonstrated the theory or
principle of the system; a PEAK document; and Mr Coyne’s cross-examination. It is necessary to
consider those therefore, in order to see if the submission is substantiated.
186.The principles of what is supposed to happen within Horizon, and the architecture, is not in
dispute. Mr Coyne’s cross-examination passage that is relied upon does not support the general
submission made to the PEAKs the subject of this bug, as it was as follows. I will quote the passage
relied upon and the passages either side for context:
“Q. So in forming a judgment on robustness you have first of all to see -- let's take a bug. A bug
happens and the first question you ask is: did that bug or could that bug have had an impact on
branch accounts? A. Yes.
Q. Then you form a judgment on whether and to what extent the countermeasures in place in the
Horizon system would have enabled that impact to be identified and fixed, yes?
A. No, I think you would look at whether it did -- whether any countermeasure or control did prevent
it from having a consequential impact, not whether it should have. Q. Well, whether it would have?
A. Or whether it would have.
Q. You don't consider whether it would have, you consider whether it did?
A. Well, if it did there would be evidence that it did. It would be documented.
Q. But in some cases it will be blindingly obvious, won't it, Mr Coyne? Let me give you an example,
remming. A Post Office example. Remming in and remming out. Money is sent from Chesterfield to a
branch. Chesterfield has a record of the money it sends over. The branch receives the money and
types in -- or usually it is a barcode actually, but it records on the Horizon system how much money
the branch has received. You are aware, aren't you, that there is an automatic system that checks
Chesterfield's figures against the branch's figures? A. Yes.
Q. Are you suggesting that every time over the last 20 years there has been a discrepancy between
Chesterfield's figures and the branch's figures, are you suggesting that it is necessary for you to see
the evidence of what happened next, of whether a transaction correction was sent and what the
evidence was in relation to that branch? Are you really suggesting that's necessary?
A. No, because in a typical scenario where the systems operate as they should, as they are designed,
then the positioning of the countermeasures that you have put in place would pick up on that so that
would work absolutely fine. In the scenario where the system doesn't operate as expected there is a
bug, error or defect or communication fault, then a different set of scenarios will likely be
encountered and it is understanding then what happens that is important.
Q. Let me get this straight. You are suggesting that there are cases when you can take it as read, the
situation is such that you can take it as read that a problem will be identified and it will be fixed. But
you are suggesting there are other situations where you can't take it as read where specific evidence
is needed, is that right? A. Yes.
Q. Okay. But nonetheless, although you say that is necessary, it is your considered opinion that the
Horizon system that you now see is robust, yes?
A. Relatively robust, yes.”
RLITO000005
RLITO000005
187.This evidence does not support the submission that remming errors are picked up by Horizon. It
is necessary therefore to look at the actual PEAKs, to see what they show. The one associated with
what the Post Office called Issue 1, PC0203085, is dated 22 August 2010 and is headed “pouch
remmed in on two counters at same time”. The first entry under impact statement is:
“The same pouch can be remmed in to the system more than once, resulting in a shortage at the
branch which POL have to rectify by issuing a Transaction Correction.
1.call has to be processed, and corrective action taken, by SSC, MSU and POL
2.visible to POL and the branch when it happens
3.very rare”
188.In my judgment, that entry alone is evidence of a bug. It shows a pouch can be remmed in more
than once - admittedly rarely - and that a TC is necessary to correct this. However, in order to be
clear, the PEAK goes on for some pages - 13 in total - and includes contradictory entries within it
from Fujitsu such as the following two. One is on 17 August 2010 from Anne Chambers:
“A cash pouch was remmed in twice at branch 126109:
Pouch barcode 399347067204
2p coin £60
50p coin £250
5p coin £100
Session 1-350379 16/09/2010 10:08
Session 2-195226 16/09/2010 10:08
The PM cannot reverse the transaction since rem reversal isn't allowed.
This is NOT another example of the duplicate rem problem that we have seen in the past, where use
of the Prev key accepted the same pouch twice. In this case the pouch was processed on both
counters...
09:05 c2 get pouch status, retrieve pouch details
09:06 cl get pouch status, retrieve pouch details
09:08 c2 settle pouch delivery
09:08 cl settle pouch delivery
There were some printer problems on counter 2 which probably explain why this was done.”
189.The other contradictory entry within the same PEAK is the next day, 18 August 2010 and states,
again from Anne Chambers
“I checked whether there were any exceptions in the BAL OSR logs for any of the messages, there was
nothing.
RLITO000005
RLITO000005
Gareth Jenkins thinks that it should not be possible to complete the rem in on both counters. Please
investigate.”
190,Under Root cause and solution, the following entry is made on 19 August 2010:
“When an auto remin pouch id is settled successfully, the system updates the
COUNTER READ TIMESTAMP in LFS_RDC_HEADER to a not null value for that pouch id.
The race condition for auto remin pouch delivery is handled at
SettlePouchDeliveryServiceSettlementProcessor.processPouch().
This method checks during settlement whether the
COUNTER READ_TIMESTAMP in LFS_RDC_HEADER is null or not null value.
If null, the pouch id is good and settlement completes successfully.
If not null, the pouch id is already processed and error is thrown.
The query that gets the COUNTER_READ_TIMESTAMP from
LFS_RDC_HEADER is 'SettlePouchDeliveryPreCheck’.
In this query, the input parameter for pouch id is defined incorrectly.
It is given "pouchBarcodefString]", but in dyno the pouch id is "pouchId".
This is the root cause why the query always returns null although the COUNTER READ TIMESTAMP
is not null.
The solution is to define the pouch id input param to “pouchid[String}" in
SettlePouchDeliveryPreCheck’.
191.A fix is then identified, stated to be a low risk and low complex issue, which also states:
“LIST OF LIKELY DELIVERABLES:
Updated sql of SettlePouchDeliveryPreCheck”.
192.In my judgment, this PEAK is evidence of a bug and a fix is required to remedy it. It also shows
that remming in errors are not always picked up by Horizon.
193.The second issue is related to what are different PEAKs, namely PC0195380 and other PEAKs
associated with KEL acha4221Q. I will not deal with them all, but these relate to what the Post Office
terms Issue 2. This KEL was raised by Anne Chambers on 2 March 2010, updated on 3 May 2011, and
the keywords are remittance, remittence, remin, pouch and delivery. It can be seen that the following
is included in the KEL:
“Symptoms
The clerk went into the Delivery menu and scanned two pouches (one of currency and one of coins).
The second pouch was recorded twice on the system, resulting in a loss of £80.
Two Rem In slips relating to the second pouch were output, both identical, as well as one for the first
barcode.
RLITO000005
RLITO000005
In the most recent instance, the same pouch was remmed in on two different counters at about the
same time.
Problem
This was caused by using the Prev key during / just after the pouch barcode scans. Now fixed - details
in PC0195380.
In PC0203085, the same pouch was processed on both counters...
09:05 c2 get pouch status, retrieve pouch details
09:06 c1 get pouch status, retrieve pouch details
09:08 c2 settle pouch delivery
09:08 c1 settle pouch delivery
PC0203085 - fix applied Jan 2011.
Solution - ATOS
A transaction log search will prove that the Rem In has been duplicated.
Send call to SSC, quoting this KEL.
SSC:
Known problems have been fixed, so needs fresh investigation if it happens again.
POL may need to issue a TC to undo the effects of the extra rem in (in the meantime the branch will
report a shortage), so MSU need to inform POL via BIMS.”
194.Given that the PEAK on Issue 1 is dated August 2010, and the KEL itself was raised in March
2010, the KEL must have been updated with the entry relating to that PEAK and that the fix was
applied in 2011. Those entries cannot have been included in the KEL originally, as they post date its
creation in March 2010. This means therefore that someone at Fujitsu obviously concluded that the
later PEAK in time (on Issue 1) was relevant to the same KEL (on Issue 2) as it was added to the KEL.
I agree with whoever at Fujitsu linked these PEAKs to the same KEL. I consider that the KEL must
refer to broadly the same issue, namely the remming in of pouches that were counted twice by the
Horizon System. A fix was in any event issued for the issue identified in the KEL. However, it is
correct and I accept the Post Office’s submissions that upon closer analysis it can be seen that there
are two separate, but very similar in effect and related, issues here, both of which have the same end
result, namely the duplication of remming in of cash pouches.
195.Finally, in the Post Office’s clarification or response to the challenge brought by the claimants to
the submissions on this bug, (which complained that there was no evidence to support the submission
that remming in issues would be automatically “picked up” by Horizon), the Post Office referenced
another document at {F/624/1}. It stated that the document was PC0197838, but in fact the reference
is BE/0197828, as it is a BIMS Incident Report and not a PEAK, and is dated 16 April 2010. The
reference number is the same but the initials are BE and not PC because it is a different type of
document. However and in any event, the document relied upon by the Post Office identifies an
exception type as “Receipts and Payments do not balance (post migration)” but it does deal with the
RLITO000005
RLITO000005
same issue as this one, as it states within the body of the document that “The PM has used the
Previous key whilst scanning in pouches, which has caused a duplicated rem” and also:
“The two identical transactions each comprise of:
Prod ID = 6287 (Rem In Cash from AD); Qty = 1; Amount = £1680 The Post office Counter log shows
this being made up of:
£100 of 20p coins
£80 of 2p coins
£500 of 50p coins
£100 of 5p coins
£500 of £1 coins
Action for Post Office Ltd:
Post Office Ltd will need to contact the PM to confirm if the PM has two identical Rem In slips for
Session 489165, both showing Pouch ID 399345656721.
If thi Rem I . - sags .
possibly by using a Transaction Correction.” (emphasis added)
196.I accept that the issues on this bug can be split into Issue 1 and Issue 2 in the way that the Post
Office suggest. If 1am wrong about that, then the PEAKs I have identified above, including the BIMS
Incident Report, are evidence of and relate to the same bug. Given the finding of two issues, then
Issue 1 and Issue 2 could either relate to different iterations of one bug, or to two different bugs.
These bugs or this bug (depending upon which semantic analysis of Issue 1/Issue 2 is correct) both
impact branch accounts. Issue 2 related to the pilot phase of Horizon Online. However, neither of
these issues are caught by automatic reporting. The Post Office accepts that Issue 1 was reported to it
by the SPM affected. The fact that the BIMS report requires the Post Office to “contact the PM to
confirm if the PM has two identical Rem in slips” for the session identified makes it clear that this also
applies to Issue 2 as well. There is therefore, in my judgment, no practical difference whether one
approaches this as two different issues with the same effect, or one.
197.On either analysis, it is a software bug and I find that these or this impacted upon branch
accounts. In my judgment, it had lasting impact upon branch accounts. The fact that the occurrences
of it were corrected by means of TCs does not mean that it ought not properly to be characterised as a
bug causing lasting impact.
6. Remming Out bug.
198.This arises under Legacy Horizon. The Post Office’s submissions effectively accept that there was
a bug under this heading, because in the detailed entry in its Appendix 2 it recites the following:
“Mr Coyne states that Bug 6: Remming Out is a bug with lasting financial impact. It comprises two
separate issues, only one of which was a bug. Any discrepancy caused by either issue would be
transient as instances of both issues were caught by automatic reporting.”
(emphasis added)
RLITO000005
RLITO000005
199.It was split into two in the Bug Table, 6(i) which was identified as KEL acha508S and 6(ii) which
was identified as KEL GMaxwell3853P. The former was identified as February/April 2007, recorded as
fixed approx. 2007, with Mr Coyne identifying 57 branches affected; and the latter was in May 2005
with one branch being affected. They were both remming out issues, hence the grouping by the
experts in the bug table. It was not acknowledged as a bug in the bug table. However, it is now
accepted as a bug by the Post Office in paragraph 6 of Appendix 2, but is said to have had transient
impact.
200.Again, for both of these iterations, 6(i) and 6(ii), Dr Worden used the same wording as he had in
the entry for bug 5, the Remming In bug, namely “As for the Dalmellington bug, above - PO hada
robust process for detecting and correcting remming errors, whatever their origin. So, there were no
lasting effects on branch accounts” (emphasis added). It was also said by the Post Office in its
submissions that Mr Coyne had conflated two unrelated issues under the heading “Remming Out”.
Given this was an entry in the 2nd Joint Statement agreed by the experts, it is a little unfair to state
that Mr Coyne had done this, as the entry was obviously agreed by Dr Worden. However, it was split
into 6(i) and 6(ii) and I will consider each separately.
201.Issue 6(i) arises as follows. What is called “a remming error” leads to a mismatch between the
amounts of cash remmed out to one place and the amounts remmed in from another. The Post Office
has submitted that remming errors are a clear violation of Data Entry Accounting and are picked up
by Horizon. The two different issues are as follows.
202.As the obverse of the coin of remming in, SPMs rem out pouches of cash to be returned to the
Post Office Cash Centre. A single pouch may contain multiple bags of coins or cash and each bag can
only hold one denomination, and there is a limit on how much cash can be placed into a pouch. The
cash can be remmed out before it is physically collected. When remmed out the cash appears in a
different line in the branch accounts. On collection, the collection team scan a barcode on the pouch
and the cash is removed from the “cash in pouches” line of the accounts.
203.When remming cash out, branches should have made one entry for each denomination and value
and, if there were multiple bags for a particular denomination, the quantity of bags should have been.
specified in that single entry (e.g. 2 x £500 of £2 coins). However, if the SPMs had made multiple
entries for each denomination and value (e.g. one entry for 1 x £500 of £2 coins and then a second
entry for 1 x £500 of £2 coins), Horizon would only record the first bag as having left the branch’s
cash holdings, but all of the bags would show on the “cash in pouches” line. This would create a
discrepancy in the branch accounts, because all of the cash would have been collected.
204.This issue occurred as a result of Release T30 INC1 (which is referred to in the Fujitsu Service
Review Book of February 2007 covering that month) and the Post Office’s submissions accept this. A
further entry for Monday 12 February 2007 in the same document states:
“The issue was found to be when Post Masters would REM out 2 identical items instead of using the
quantity button. As this was an error in the code, the counter 36_6 s/w was regress overnight. After
the regression of the s/w overnight, it was then identified that 49 branches had an issue with the REM
out process causing a cash imbalance at the counter.”
205.I consider this to be a bug. Fujitsu created KEL acha508S to advise branches to correct the
problem manually and ran automated reports to spot any further occurrences. Fujitsu then made
changes to Release T30 INC1 and rolled it back out to fix the issue. However, that the bug was fixed
does not, in my judgment, mean it ought to be ignored. Paragraph 22 of the Appendix 2 submissions
RLITO000005
RLITO000005
on this by the Post Office submits “in any event, it was picked up automatically”. This is not correct.
An entry in the February Service Review Book on page 11 of 33 of that document shows that there
were 526 calls to the Horizon Service Desk as a result of this specific issue, of what were said to be a
high number of software calls in that month of 3792. That does not support a submission that it was
picked up automatically.
206.So far as Issue 6(ii) is concerned, the following evidence from Dr Worden - which is entirely
ignored in the Post Office’s submissions in Appendix 2 - is relevant.
“Q. So we are looking at 6(ii).
A. Yes. So again my standard wording comes in.
Q. That's it. You will see there:
“Remming out' (ii) Bug (not acknowledged)."
This isn't one that you had in either of your tables in your appendix originally. A. No.
Q. Did you have a look at the KEL in that case?
A. I believe I did, yes.
Q. Can we look at it now, please. This is at {F/276/1}.
A. Rem out ... Yes.
Q. This is the GMaxwell3853P KEL. If we look at the bottom where it says "Solution - Atos".
A. Bug in the code, yes, right.
Q. "Development have identified a possible bug in the counter code. However, due to the frequency of
the problem & the risks involved in making the necessary changes it
has been decided that a code change will not be made." A. Yes.
Q. You hadn't noticed that, had you?
A, I believe I had, but again the point about permanent effects, lasting effects, on branch accounts is
not altered by that.
Q. And the point about this is there was a lasting bug?
A. There may have been a lasting bug, but they took a decision based on its frequency and the
difficulty of making the change, perhaps the risk of making the change, that it was not worth
addressing. That actually the remming process would pick it up and its frequency -- they took a
decision, I don't know what the details were of their trade off. They took a decision that fixing the
code was not worthwhile. Q. I'm just asking you: it is a lasting bug?
A. It isa lasting bug, yes.
Q. Let's go to number 7 now, if we may, please.”
207.There was then an interjection by counsel for the Post Office who drew attention to possible
misunderstanding in terminology given the use of the phrase “lasting bug”. Dr Worden then added the
RLITO000005
RLITO000005
following, when asked by the claimants’ counsel whether he had understood what was meant by the
question:
“A. I did. And I'll clarify what I mean: it's a bug that was not fixed, we cannot see was fixed, but it was
a bug without lasting effects in my opinion.”
208.In my judgment, it was clearly accepted by Dr Worden as a bug, and the entry in the KEL itself.
also says there was “a possible bug in the counter code” but a decision was taken not to change the
code because there was only one instance of this occurring.
209.1 find that 6(ii) also relates to a bug and reject the Post Office’s submission that “it wasn’t a bug”.
210.The claimants’ case on the Remming Out bug in the Bug Table at item 6 is therefore made out.
Both of these - issue 6(i) and 6(ii) - are, in my judgment, software bugs. I find that these had potential
lasting impact upon branch accounts.
7, Local Suspense Account issue, not the same as 3. Suspense Account bug.
211.This was reported in 2010 and recorded as fixed in September 2010. It arose in the early days of
Horizon Online, during the pilot scheme. It was not acknowledged in the
Bug Table, but is now accepted as a bug by the Post Office in paragraph 6 of Appendix 2, although it is
one of those bugs said to have had transient impact. Mr Parker had identified 33 branches affected.
Mr Coyne recorded what he said were four associated KELs, namely acha5259Q (for which there were
6 PEAKs) (this KEL is mistakenly recorded twice in column 3 of the Bug Table, with acha5838T only
mentioned in the text in that column); cardc2043L (10 PEAKs); PorterS199P (3 PEAKs); and
acha5838T (which states there are “two different but similar problems” and appears in the text, but
not in the list of KELs at the end of the text by Mr Coyne). Dr Worden’s comments in the table were
that:
“The KEL acha5259Q implies that PO and Fujitsu were able to identify all occurrences of the problem,
without being notified by any Subpostmaster. I would therefore expect them to have corrected any
impact on branch accounts as part of normal error correction processes.
I would not expect evidence of all corrections to accounts to have survived to the present day. PEAKS
and KELs are not used to record corrections of financial impact.”
212.He also relied upon a statement by Mr Parker that there was “Temporary financial impact which
would have been cancelled out in the following period by a corresponding discrepancy”. To use the
Post Office’s own submissions it was “an intermittent system issue which temporarily prevented
branches from rolling over into the next Trading Period”.
213.The statement of Mr Parker introduces another concept, similar if not identical to Dr Worden’s
“transient impact”, and that is “temporary financial impact”. Dr Worden is correct that PEAKs and
KELs do not record corrections of financial impact; they do however record financial impact, because
when an SPM reaches SSC (which raises the PEAK) they often start by recording “SPM says he/she
has a problem in that...... ” and financial impact is often then recorded. The Post Office in paragraph 2
of the detailed part of Appendix 2 dealing with this bug states that “any discrepancy would not be
lasting”. This does therefore accept, albeit implicitly, that it would have an impact upon branch.
accounts. It is also submitted that it was a “teething problem” from the early days of the Horizon
Online pilot scheme.
RLITO000005
RLITO000005
214,The cross-examination of Dr Worden on this bug was as follows, and I find to be highly relevant.
“Q. Then you say:
“T would not expect evidence of all corrections to accounts to have survived to the present day. PEAKs
and KELs are not used to record corrections of financial impact." A. Yes.
Q. Fujitsu, in fairness, analysed the KEL, you say here, and said:
“... "Temporary financial impact which would have been cancelled out in the following period by a
corresponding discrepancy." Do you see that? A. Yes.
Q. Now, did you look at the KEL?
A. Well, I infer from that Fujitsu did it, but I had done it previously.
Q. Let's go, please, to {F/637/1} which is the KEL acha5259Q.
A. Right. Sorry, can I remind myself of the ..."... local suspense cleared ... balance report ... cash ..."
Q. Do you see if we look at the problem --
A. Yes.
Q. -- the summary was that the PM was forced to clear local suspense several times resulting in the
cash figure on the balance report changing and possibly a discrepancy in the new trading period. A.
Yes.
Q. So that certainly had the potential to cause an impact on branch accounts at the time, subject to
later correction? A. Yes.
Q. And probably did, yes?
A. I think it did, yes.”
215.It is in my judgment self-evident, that if the branch accounts had to be corrected later, in a new
trading period, that the accounts for the period when the effects of the bug were experienced were
wrong. This was a bug. It was a bug that had a lasting impact upon branch accounts as I have defined
that phrase. The fact that the discrepancy was later corrected, in the words of Mr Parker a
discrepancy “which would have been cancelled out in the following period by a corresponding
discrepancy” makes this entirely clear.
8. Recovery Issues.
216.These are Horizon Online issues. The Post Office does not accept these are bugs at all. These are
not agreed as bugs by the experts, and there are four different types included in the table, with years
of effect from 2010 to 2018. Mr Coyne’s entry stated that “The text within the PEAKS and KELs
suggests that in each case a branch account discrepancy would be evident and would require
correction by the Post Office.” Dr Worden stated that:
“The KELs and PEAKs cited by Mr Coyne are not indicative of errors in Horizon. They provide
guidance on how to correct discrepancies caused by human errors or other errors in transaction
recovery (‘recoverable transactions’)
RLITO000005
RLITO000005
Because there were many such errors, there were many calls to the help desk and many PEAKs and
KELs. Normally, correction of errors involved back office reconciliation and issuing TCs. This was
accurate and effective; I have derived an upper limit of £2 per branch per month on the mean impact
of erroneous TCs.
One important KEL acha959T was guidance to the back office MSU, not for Subpostmasters”.
217.There was a minor typographic error in the submissions in paragraph 4 of the detailed part of
Appendix 2 which referred to bug 9. However, the four different types of recovery issues are
addressed in the subsequent paragraphs and the conclusion paragraphs deal with bug 8. This was
later corrected in a sheet of corrections, which clarified that paragraph 4 of the detailed part of
Appendix 2 should have stated "Mr Coyne states that Bug 8: Recovery Issues is a bug with lasting
financial impact. Post Office submits that it is not a bug at all".
218.Recovery deals with the situation where there has been an interruption, which means when a
basket of transactions is interrupted during the course of dealing with a customer. This could be for
several reasons including a power failure, system crash or communication failure between the branch
and data centre. This is a risk which has to be dealt with as it cannot sensibly be eliminated in a
system such as Horizon. Horizon runs various automated reports each day to look for failed recovery
events. Mr Coyne dealt with two different documents in his report, PC0197769 and KEL acha959T.
The former was created on 15 April 2010 during the pilot phase of Horizon Online. The root cause of
the issue was identified on 26 April 2010, work on a fix was started and this was released in early June
as shown in Release PEAK PC0199000 (and then estate-wide to pilot branches) by mid-June 2010. It
did however undoubtedly evidence a bug. The reference to the “959T” KEL below is the KEL whose
full title is acha959T which was raised by Anne Chambers.
219.KELacha959T is also the very same KEL referred to in PEAK PC0214226 dated 14 December
2011, headed “Failed Recovery Transaction(s) 12 Dec 11” which I have considered at [110] and [111]
of the substantive judgment, as it related to Mr Tank's specific experience at his branch regarding the
£195 transaction. I have already dealt with one entry in that PEAK in that part of the judgment. I have
accepted Mr Tank’s evidence on this incident. That PEAK was closed with the categorisation “Final -
Advice after Investigation” and “Root Cause - General in Procedure”. The entry at 10:40:51 on 14
December 2011, also from Mr Bragg, shows the transaction details which plainly identify an
authorised transaction, a failure of the basket settlement, the failure of repeated retries by the system
and the failed recovery. It was this analysis that preceded Mr Bragg’s conclusion that “the customer’s
account should be correct but the branch will have a shortage (for a withdrawal) because the session
hasn't been recorded”. This is plainly a fault in Horizon, and is directly contrary both to the Post
Office’s case on failed recoveries in the Bug Table, and also its case on Mr Tank’s evidence.
220.Dr Worden’s cross-examination on the issue of recoveries, by reference to entry 8 in the Bug
Table, was also highly relevant:
“Q. Let's move if we may now to number 8 and we find that -- just to do it the same way again, if we go
back to the joint statement 2 table and we look at {D1/2/7}. A. Row 8.
Q. It is row 8. The recovery issues are identified there.
A. Mm.
Q. Mr Coyne has made references to the text in certain PEAKs.
RLITO000005
RLITO000005
A. Yes.
Q. And you have given your opinion there, you say they are not indicative of errors in Horizon, they
provide guidance on how to correct discrepancies caused by human errors or other errors. A. Yes.
Q. You say: "Because there were many such errors, there were many calls to the help desk and many
PEAKs and KELs." Then it is basically your same theory, that: “Normally, correction of errors involved
back office reconciliation and issuing TCs." And that's accurate.
A. It is a different theory really. Recoverable transactions are a big subject and they are complicated
because the point at which a recoverable transaction can go wrong is very variable through the
sequence, and therefore the number of recovery actions, the type of recovery actions is complicated.
Horizon was designed so that with the assistance of the postmaster most of these could be recovered,
, . led failed . . A A ign of Hori 1
those fai . , Fuji . But fail
Tecoveries means the recovery process had failed, that's the way Horizon was designed because these
things are so complicated you can't handle them all automatically. So it is a big subject but there is
plenty of useful evidence about it. Q. You say in your table at {D3/2/124}.
A. This is the 959T KEL, is it?
Q. Yes. You say:
“This is another complex KEL with strong overlap with previous KEL." A. Yes. This KEL is cited in a
large number of PEAKs.
Q. And in fact when we go back to the table at {D1/2/7}, Mr Coyne has actually quoted from one of
the PEAKs we see there, the first one he quotes from. Do you see "PC0198352"? A. Yes.
Q. And he has quoted: "This problem caused a loss at the branch for which they should not be liable."
Pausing there. This did have the potential to cause the type of discrepancy which you have given
evidence about being later corrected, didn't it? A. The word "problem" doesn't imply a bug in Horizon.
Q. You didn't read these PEAKs yourself, did you?
A. [have read a fair sample of recovery PEAKs. Maybe not these ones, but I have read a fair sample.
Q. If we go over the page {D1/2/8}, do you see Mr Coyne points out there that this
KEL is referred to by 2,473 PEAKs --
A. Yes, that's --
Q. -- from 2010 to 2018.
MR JUSTICE FRASER: Where are we looking?
MR GREEN: At the top of that page under the "Coyne Opinion as to branch account impact", my Lord.
MR JUSTICE FRASER: Yes.
A. That is correct.
RLITO000005
RLITO000005
Q. Just pausing there, Dr Worden. If something is handled by NBSC and corrected straightaway, it
doesn't make it necessarily up the line to SSC, does it?
A. No. I mean there are various levels of recoverable transactions. There is a recoverable transaction
that the subpostmaster can follow the instructions and can fix it himself, there is the ones where he
needs help from NBSC, there's ones where he needs help from further down, and there are failed
recoveries where PEAKs are involved. All of those things happen.
Q. In the case of a failed recovery, it can be sorted by NBSC in some way or by a transaction
correction without referring to SSC?
A. That's a good question. I think the failed recovery port usually involves not just NBSC, it involves
MSU, and there is quite a complicated process of guiding the transaction through various states until
it gets to the F99 state.
Q. Can I just invite you to look at what Mr Coyne said --
A. Yes.
Q. ~ having actually tried to identify through the available disclosure how many PEAKs were
referenced and set them out. He says it is referred to by 2,473 other PEAKs from over that range:
"... each of these may (but may not) indicate a similar issue with the Horizon recovery process and
potentially creating a discrepancy within branch accounts." If we accept his definition of discrepancy
and not yours, yes?
A. This is all about temporary discrepancies in branch accounts.
Q. Park the temporary discrepancy point about which we have heard a lot.
A. Yes.
Q. What he has said there is correct in the number of PEAKs as far as you know?
A. It is correct, yes.
Q. And it is scrupulously fair in how he has described it?
A. Well, “issue with the Horizon recovery process", I believe these issues are how Horizon was
designed.
Q. But what he's said there, he has carefully put "issue", leaving open the availability of an argument
between experts about what the nature of the issue is, but what he has scrupulously done there is
perfectly fair, isn't it?
A. I think so. I mean the difference between the experts is that, to summarise something Mr Coyne
said in his oral evidence, he believes that these recovery processes can be automatic. Now, I don't
believe that. I believe recovery of recoverable transactions is very complicated and needs human
intervention and things like failed recoveries are inevitable.”
(emphasis added)
221.1 prefer the evidence of Mr Coyne that recovery processes can be automatic in systems, and I also
accept that “failed recoveries” as they were defined above are inevitable. The issue here is whether
the Horizon System process for dealing with the recovery processes when they had in fact failed was
RLITO000005
RLITO000005
working as it should, or defectively. The fact that there are so many PEAKs identified in the KEL, and
the contents of those PEAKs clearly demonstrates, in my judgment clearly, that it was not. I reject Dr
Worden’s evidence that human intervention should be required or expected for some, most or all
failed recoveries. I also reject his evidence about the word in the PEAK “problem”. He said “The word
"problem" doesn't imply a bug in Horizon.” In my judgment, on this subject, it clearly does.
222.The Post Office’s own submissions set out how trading periods were affected. I shall reproduce
them:
“10. The issue is described in KEL acha5650L. It involved recovered transactions being written to a
different trading period than the original transaction. In summary:
iif a transaction failed in one trading period and the recovered transaction went into the next trading
period, there would be a loss in the first trading period 5 and a corresponding gain in the next trading
period such that there would be no overall loss in the branch; and
ii)if the recovered transaction was written to an earlier trading period, there would be a loss in the
current trading period but no corresponding gain because the previous trading period would have
already been rolled over again. There would be a net loss in that scenario.”
(emphasis added)
223.That demonstrates that branch accounts would be affected. In the first possibility, the accounts
would be wrong for one trading period (until the so-called “corresponding gain” occurred in the next
period) and in the second, it is accepted there would be a net loss.
224.In my judgment, the issuing of a later TC is neither here nor there in this scenario. Where there
has been a failed recovery, this is now flagged automatically by Horizon to SSC at Fujitsu for
investigation. So far as the modern or more recent way that the system deals with failed recoveries, it
is plainly the case that Horizon now automatically flags these. The fact that it did not do so previously,
in my judgment, and the existence of all these PEAKs on this subject, clearly demonstrate a bug, error
or defect, and that conclusion is not avoided by the fact that a later TC was issued.
225.On 16 January 2017, the automatic monitoring service spotted a failed recovery and PEAK
PC0256502 was raised by Fujitsu. A BIMS was issued to Post Office to correct the discrepancy. On 17
January 2017, the SPM in question contacted the helpline regarding the same issue. A second PEAK
PC0256566 was raised. As quoted by Mr Coyne in his report, this PEAK clearly records that this issue
was already identified by Fujitsu and resolved under the earlier PEAK. Hence the later PEAK in time
was closed. However, although this example required corrective action by the Post Office (not by
Horizon) it is not evidence of a bug. The former PEAK namely PC0197769 did evidence a bug. I also
find that PEAK PC0214226 is evidence of a bug.
226.Mr Parker’s evidence also makes it clear that this bug had an impact upon branch accounts. Mr
Tank’s evidence demonstrates the same, albeit for a different date and time. This impact would persist
until it was later corrected.
9. Reversals.
227.This is a Legacy Horizon bug and occurred in 2003. It was not acknowledged as a bug in the Bug
Table but is now accepted as a bug by the Post Office in paragraph 6 of Appendix 2. It is however said
only to have had transient impact.
RLITO000005
RLITO000005
228.Dr Worden’s entry in the 2nd Joint Statement in relation to this stated “Transaction reversals are
a complex area which, like recoverable transactions, are less familiar to Subpostmasters and are more
prone to human error. They lead to many calls to the help line and to many KELs and PEAKs - not
necessarily related to any fault in Horizon.” This can be seen as an example of Dr Worden attributing
fault to the SPMs rather than accepting the presence of a bug, but given it is now accepted as a bug it
is not necessary to consider that any further. Mr Coyne’s entry in the 2nd Joint Statement stated “In
April 2003 due to a failure in regression testing, Horizon version $30 was released by Fujitsu and this
introduced a bug where the value of transactions reversed by Subpostmasters was shown twice in the
amount of the reversal in branch accounts.”
229.This was clearly a bug. A Fujitsu document, PEAK PC0089918, identifies an issue in which
attempted reversals of remming in transactions resulted in the magnitude of the transaction doubling,
rather than the transaction being reversed. Initially, this is
blamed upon the SPM in question and the Post Office in its Appendix 2 continues this, submitting that
“The Subpostmaster (branch FAD003227) was trading in Stock Unit Y and remmed in £13,910 of cash.
He continued to trade in that stock unit in error, instead of moving to Stock Unit I. He therefore
attempted to reverse all of the transactions he had made in Y (including the rem in). Instead of Y
showing a balance of zero, the rem in had doubled to show a discrepancy of £27,820. Having spoken
to NBSC, the Subpostmaster attempted further reversals but these failed and produced an error
message indicating that the initial reversal had been completed successfully.”
230, However, it was clearly caused by a bug. This is effectively accepted later in the submissions
which state, correctly in my judgment, that “the issue was caused by a software error, which had been
introduced as part of implementing the fix for PC0083954 in Legacy Horizon, and which incorrectly
calculated the reversal value and quantity (essentially, the wrong mathematical sign was applied when
reversing RIAD transactions (+ instead of -))”.
231.If a minus arithmetical function is required, namely subtraction, but by software fault this actually
occurs as an addition, then this is clearly a bug. It is certainly an error or defect in the system. This
was then compounded by PC0083954, a fix which introduced a code change to ensure that a partial
settlement of cash always tried to reduce the stack value. However, that fix did not work when
reversing a rem in and so the solution was to pass the mode to the function also.
232.A fix was required, and because of the potential impact on live branches, the development of a fix
is said to have been “fast-tracked”. The Post Office submissions stated that “within 15 days of the
issue first being raised, Fujitsu had implemented a fix to 2,178 branches. It is not known precisely
when the fix reached all branches in the estate but Fujitsu believe it is likely to have been before 2
June 2003....” That again comes perilously close to introducing evidence by the back door through
submissions.
233.This was clearly a bug with potential lasting impact to branch accounts. The fact that the impacts
of its occurrence were corrected by means of TCs does not affect its existence. It occurred for a short
period in 2003. There is no evidence before the court of branches being affected by this outside that
period, or of branch accounts affected by the bug that were not later corrected.
10. Data Tree Build Failure discrepancies.
234.This is a Legacy Horizon bug. Its identified effect was during 1999 and 2000, so the very early
days of Horizon. However, KELs relating to the PEAKs in those dates were created in July 2005 and
RLITO000005
RLITO000005
updated in November 2007. A data tree is an accounting node hierarchy, or a cash account node
hierarchy.
235.It is accepted by the Post Office in paragraph 7 of Appendix 2 as one of a number of “bugs with
lasting impact (although they were resolved)”. The PEAK dealing with this reads “Data trees have
been failing to build fully, and the system has not been detecting this.” Data trees were part of Legacy
Horizon, and were used to build a summary (or picture, the word used by the Post Office in its
Appendix 2) of the accounts. The building of data trees is a software function, and given “the system”
is
Horizon and is supposed to detect failures of this nature, it is difficult to see how it
could ever have been in issue that this was a bug. Mr Coyne’s entry in the Bug Table recites that
“Dugannon branch suffered a £43,000 discrepancy but the cause was not immediately known.
£52,814.29 at the Yate Sodbury Branch. £9,368.40 at the Appleby Westmoreland branch.” These are
sizeable sums and arose in the branch accounts. That this is a bug was de facto accepted by Dr
Worden in the Bug Table due to the text of his entry which states:
“There wi which hi ntial im: mn_ branch nts, early in the lifetime of Horizon.
Soon after it arose, the error was trapped and detected by DEP and was then soon fixed.
The fault was easily noticeable at branches before the error trapping which was soon introduced and
would be even more noticeable after that. Only three branches appear to have been affected, as
described by Mr Coyne.
Because it was so noticeable at the branch, and the Peak is concerned with a software error rather
than any other cause, I would expect any discrepancies in branch accounts to have been corrected.”
(emphasis added)
236.This is therefore undoubtedly a bug, but findings in respect of this therefore depend upon the
nature of its impact on branch accounts. Dr Worden in his cross-examination changed the text of one
of his entries in one of the appendices to his report, where he had said “This is further evidence of the
failed recovery report doing its job - alerting [Fujitsu] to failed recoveries, so they can investigate
them and make any necessary corrections to accounts.” He said this later part should be changed:
“Perhaps I should have said have “guided the transactions to the correct state". He said it was caused
by a complex grey communications failure, by which he meant an intermittent one, although he was
not really sure where he had obtained the word “grey” from, he thought perhaps from a PEAK or KEL.
237.The first PEAK PC0033128 related to an issue in November 1999 at Dungannon. The SPM
reported a £43,000 discrepancy after balancing stock units and doing an office snapshot. An office
snapshot is a Horizon report that showed the current accounting position of the branch, including any
cash discrepancy. To produce the office snapshot, Horizon scanned the messagestore for the
necessary information (eg. initial cash and stock levels, all cash and stock transactions, plus other
service transactions). It then compiled that information together in order to produce the office
snapshot. This is the “data tree”. On occasion Horizon would fail to read the necessary transaction
data for various reasons so the report would be incomplete. This failure at Dungannon was caused by
a missing payments node.
238.Two other branches were affected by the same issue, Yate Sodbury and Appleby-inWestmorland.
The snapshot at the former was incorrect showing a shortfall of £52,814.29 and Appleby’s snapshot
RLITO000005
RLITO000005
showed a shortfall of £9,368.40. MSU were involved in the investigation of this and the Post Office
submits that “it is therefore likely that the issues at Yate Sodbury and Appleby were resolved via a
BIMS report and that the Subpostmasters were held harmless. However, due to the age of [this issue],
comprehensive records are not available and therefore Post Office is not in a position to provide
detailed commentary.” Two points need to be made. The shortfall was in a snapshot and not ina
Branch Trading Statement; and secondly, there is no evidence of how it was corrected if at all. There
are references in the PEAK in respect of Dungannon that shows the SPM and the Post Office agreed to
remove the discrepancy from the account. For present purposes, although all these shortfalls are
substantial, there is no evidence that the Post Office required payment of these sums from SPMs.
239.In my judgment, the Post Office correctly identifies branch impact in the following passage in its
submissions:
240."Of itself, Issue 1 [ie the issue above] would not affect branch accounts; there was no issue with
the underlying transaction data and, if the office snapshot was re-run, it would very likely provide the
correct information, because the data reading issue was temporary. However, if the branch ran the
office snapshot, got an inaccurate report and then rolled over (making good any discrepancies in the
process), then the shortfall would have an impact on branch accounts.”
241.There is an associated issue in respect of data trees, namely PEAK PC0121925. As part of the
changes made to move branch accounting from weekly cash accounts to monthly branch trading
statements (a programme called IMPACT), changes to the data server were made to reduce the
number of times that the messagestore was scanned to pick up transactions during balancing. A
Riposte mechanism known as “Notifications” was used to add new transactions to the existing totals
as further transactions were generated during the balancing process (rather than rebuilding the data
server tree of transactions from scratch). One of the test branches experienced an issue, where the
test branch had a gain of £45.05 following a cash declaration and rolling into branch trading. Initially,
Escher were unable to replicate this scenario and so no further action could be taken, Subsequently,
as demonstrated in a later PEAK in 2005, PC0123319 Fujitsu was able to replicate the scenario and
implement a fix on 5 September 2005. The fix was implemented before any branches had switched to
use the new branch trading code, meaning that the issue in PCO121925 could not have impacted any
live branches. This is, however, a bug, error or defect, but not in the Horizon System as it is used in
actual branches.
242.Entry 10 in the Bug Table therefore does relate to a bug in Legacy Horizon which had an impact
upon branch accounts. As the Post Office submits, in this instance the impact of that bug was
corrected.
243.Two PEAKs relate to a second element of this, one in February 2006, PC0132133 and another in
March 2007, PC0144386. The Post Office accepts these PEAKs are essentially the same (one is said by
the Post Office to be a manifestation of the other) and the former is said to “relate to an issue in which
the notification mechanism referred to in PC0121925 was accidentally switched off”. There is no
explanation of how or why this switching off occurred. The KEL that goes with these states in its title:
“Title: Multiple cash declarations may cause incorrect figures in Discrepancy, Variance and Balance
Reports
Summary: Intermittent misleading figures in Discrepancy, Variance and Balance Reports”
RLITO000005
RLITO000005
244.In the text in the KEL, certain passages make it clear that this is a software issue that persisted
into 2007.
“Solution - ATOS
<b>Helpdesk:</b><br/>The Declare Cash problem clears itself overnight. If the PM logs a call on
the day he is having problems, ask him to try the following workaround:<br/><br/>1. The clerk
should log out of the affected counter.<br/>2. Another clerk attached to a different (individual, not
shared) stock unit should log into the <b>same</b> counter, declare cash for his own stock unit,
then logout.<br/>3. The first clerk can now login to the same counter and declare cash again. The
variance should be correctly recalculated.
Alternatively log on to a different counter and do the cash declaration there. If the workaround is not
successful or the problem does not clear itself overnight, send a call to SSC, otherwise no call is
needed. November 2007: a fix is currently being piloted and is likely to be sent to the whole estate in
January
(COUNTER EPOSS 39 3 or later). If this problem is reported after
COUNTER EPOSS 39 3 has been applied, send call to SSC.”
(emphasis added)
245.A fix is only required if there is a bug, error or defect in the software.
246.The Post Office submitted that there was “no long term impact upon branch accounts”. This was
therefore a slightly different approach to “lasting impact” upon branch accounts. “Long term” is not
defined. If the impact of this bug occurred prior to a BTS then correction by means of human
intervention and TC would be required.
247.1 therefore conclude that this is a bug with the potential to cause lasting impact to branch
accounts. The fact that the impacts of its occurrence were corrected by means of TCs does not affect
its existence. The November 2007 entry in the KEL shows that a fix was required in late 2007
intended to be released in January (which must mean January 2008). Accordingly, it cannot be
considered to have been limited in its effect only to the early days of Legacy Horizon and obviously
persisted for some years.
11. Girobank discrepancies.
248.This was a Legacy Horizon issue that occurred between May and September 2000. Mr Coyne
considered this was a bug. Dr Worden did not and stated that “the first fault concerns reports. A fault
in a report is not a discrepancy in branch accounts, and only causes one if it causes a person to make
a mistake.” This is now essentially accepted as a bug by the Post Office, but it is submitted it had no
branch impact. It is included in paragraph 5 of Appendix 2 under the heading “the following bugs had
no branch impact.” The detailed part of Appendix 2 dealing with this submits that there is no evidence
of any financial impact upon branch accounts, let alone a lasting impact. This was in the early days of
Legacy Horizon. There are said by the Post Office to be six distinct issues arising under this heading.
249.1 do not know if what are called “giros” are still in use at all in 2019, but they look similar to
cheques and used to be widely used, for benefits payments amongst other things. In basic terms, the
giro (which is paper, and was sometimes included in a book of giros) would be exchanged for cash. A
customer would present a giro for (say) £50
RLITO000005
RLITO000005
to the branch Post Office. The SPM or their assistant would take the giro, giving the customer the £50
in cash from the till, entering that transaction on Horizon. This would reduce the cash holding in the
branch by £50. The giro would be sent to Girobank at the end of the day and that would be considered
by Girobank together with the entries that would have come via Horizon. Girobank would then pay the
Post Office the sum of £50 in respect of that particular transaction.
250.If there was a difference between the value of the giros entered on Horizon, and the physical
giros received by Girobank, then Girobank would not pay the Post Office the full amount. In other
words, there could be a mismatch between Girobank’s view of the amount of money paid out in
exchange for giros (based on the physical giros it received), and the Post Office’s view (and hence the
amount of money paid out of the branch by the SPM to customers presenting giros).
251.The summaries of the different issues taken from the PEAKs show the following problems:
1.Girobank taking the view that there was a discrepancy between giros received from (say) Branch X,
and cash paid out by Branch X for those giros. The explanation for why this occurred was the “cut off”
time for post being collected from Branch X (which would include the giros to go to Girobank) being
earlier than the end of the trading day. Branch X might pay out (say) £85 to a customer at 4.45pm, but
the post (including all giro vouchers then available at the branch) might have left at 3.15pm or
4.30pm. This would lead to the difference explained at [250] above. It would also lead to an error
notice being issued by Girobank. The Post Office’s submissions state that “in this particular case, the
only possible impact would be if the branch had accepted the error notice received because of the
reporting issue”. In other words, the only way the system would work correctly would be if the SPM
did not accept the error notice. The Post Office have to rely upon a SPM not accepting an error notice
issued to him or her to avoid any impact on branch accounts. That is not a software bug, it is true; it is
however an error or defect in the system. The error notices are issued through or as part of the
Horizon System. This type of incident is referred to in PC0044232, which concerned a £505.72
discrepancy. This was a known issue dealt with by KEL MWright531p. This KEL is now said by Fujitsu
to have been deleted and irretrievable due to its age.
2.A second issue was identified by Fujitsu in the course of investigation of the incident in PC0044232.
This was that the same £81 giro deposit had been included on two consecutive daily reports. Part of
the Post Office’s submissions on this are as follows: “This is because the transaction was entered onto
Horizon in a precise (and very small) window of time between two system calls being undertaken,
resulting in a duplication. The overall branch position would still have been correct, but the daily
reports to Girobank may have been wrong. If they were (i.e. if the same transaction was included on
two consecutive daily reports), it is expected that this would have been spotted and a TC would not
have been issued to the branch.” This is, in my judgment, evidence of a bug, error or defect in the
system, as the duplication referred to should not have occurred. There is no evidential basis for any
“expectation” that if the reports to Girobank had been wrong, it would “have been spotted” and led to
the issuing of a TC. No such TC has been produced.
252.Some other issues relating to Girobank were identified by the Post Office separately, as issues 3,
4, 5 and 6 under the same heading of entry 8 in the Bug Table. Issue 3 applied to two PEAKs, both
from 2000. They were PC0052575 (in which the SPM reported discrepancies of £20 and £628.25) and
the issue was diagnosed as arising out of the use of a shared stock unit. The Post Office submissions
were “There is a window of time between a user printing and cutting-off a report. If another user was
to perform a transaction during that window, that transaction may not show on the report. The issue
was already due to be fixed in a future release”. Mr Coyne accepted in cross-examination that these
RLITO000005
RLITO000005
were indications of a discrepancy being identified, but these were not of themselves evidence of a bug
in Horizon. His overall evidence on this was contained in his final answer which was “It created a
financial discrepancy within the Horizon system which could then ultimately have an impact on
branch accounts.” I accept that this issue caused a financial discrepancy. Given the issue was “due to
be fixed” in what is described as a “future release” the issue arose from the operation of the system
and is therefore, in my judgment, correctly described as a defect.
253.Issue 4 was identified as follows in the Post Office’s detailed submissions on Entry 8 in Appendix
2:
“26. Issue 4 applies to PC0068633 (referred to at para 3.124). This Peak was raised in 2001 and so
comprehensive records are not available.
27.In that Peak, the Subpostmaster reported that his cash account showed two giro deposits of £1,503
but that his reports showed only one. The Subpostmaster received an error notice from Girobank
which cleared the error, but he raised the issue because he believed that an error in Horizon was
duplicating the transaction.
28.The issue was caused by a cut-off being performed on one counter despite an attempt to print a
transaction failing on another counter. This resulted in the cut-off report including the transaction
that had failed to print.
29.A fix was actioned by Fujitsu on 18 December 2001.
30.Issue 4 only occurred in a very specific set of circumstances and would have had no direct financial
impact on accounts; it merely had the effect that a transaction was missing from the reports.”
(emphasis added)
254.In my judgment, the content of those submissions clearly demonstrate that this was a bug, error
or defect which had an effect upon the report available to the SPM. This is made very clear in the
PEAK itself, which states “Lhave duplicated this bug. In fact it occurs in all reports that use
dataserver (i.e. the majority). I shall now check to see whether or not the problem still occurs at $10.”
(emphasis added) The author of that PEAK was a Fujitsu employee and expressly referred to a bug. It
did not have an effect upon the branch accounts of the SPM in question but it is undoubtedly evidence
of a bug.
255.Issue 5 applies to PC0073855 and PC0075312 from 2002. Again, there was not a great amount of
material available due to the age of these PEAKs. A summary is as follows.
256.In PC0073855, a SPM reported that the office snapshot figures were double the figures on the
balance snapshot (with a £6.76 discrepancy). Fujitsu were unable to replicate the issue and were
therefore unable to issue a specific fix. However, a new version of the component was released with
extra tracing code so that if the issue reoccurred, Fujitsu could gather more evidence. The SPM would
have had an inaccurate report but the correct data was still available and this would not have affected
the branch when balancing. In PC0075312, another SPM raised an issue with printing her giro
deposits. The issue was identified as being caused by the same root issue as PC0073855 which was
already with the development team. These did not impact branch accounts. However - and I consider
this to be important - they constitute evidence of a bug. The KEL with which they are associated is
KEL AChambers4410R, the same KEL as the PEAK in Issue 4. The only reason the Fujitsu
RLITO000005
RLITO000005
“development team” would be involved would be to develop a fix. A fix is only required if changes are
required to the software, which means there is a bug.
257.Finally, Issue 6 applies to PCO076065, which was that two giro deposits were reported by a SPM
not to be showing on the previous or that day’s reports. The SPM identified the amounts of the
deposits (£11 and £24) which were said to be missing, but Fujitsu discovered that SPM had produced
two reports, and on this occasion it was the SPM who was in error. One report did not have the
deposits on it, but the next day’s report had both. I find that this is not a bug and is an example of
user error (that is, and to be clear, SPM error).
258.There is an associated issue with giros which is identified in another PEAK, which is not included
in the six issues. I shall reproduce the Post Office’s submissions on this
PEAK:
“The issue in PC0050418 was thought to be the same issue [as in [251](1) above] - the branch’s
largest discrepancy was £323.32. However, due to the length of time it took for the issue to reach
SSC, the branch’s messagestore had been archived - the Subpostmaster raised the call on 29 June and
the call was sent to PINICL on 17 July. Th mi or Wi id that they would n OVI
further information (such as the Transaction ID) to allow Fujitsu to investigate. The Subpostmaster
does not appear to have pursued this. The Peak notes that Girobank were going to send an error
notice, but due to the age of this Peak the relevant records are not available and Post Office is not ina
position to provide detailed commentary.” (emphasis added)
259.In my judgment, that is prima facie evidence of a shortfall in the sum of £323.32 being caused to
that branch's accounts as a result of this defect or error, notwithstanding that the SPM reported it.
The SPM was told further evidence was needed in order to investigate. It is a self-contained example
that in my judgment supports the claimants’ case on the existence of a bug, error or defect affecting
Girobank issues. The fact that the SPM for whatever reason did not pursue it does not affect that.
260.Although the very numerous PEAKs identified in respect of this bug in the Bug Table share some
common features, as can be seen above there are different issues in respect of many of the PEAKs.
This was counted as a single bug in the Bug Table, even though my detailed findings above show that
there were more than one bug, error and defect that related to Girobank. I will however for
consistency count it as a
single bug in the overall total, as that is how the experts treated the total number of bugs in the Bug
Table.
12. Counter-replacement issues.
261.This also is a Legacy Horizon issue. The counter is part of the hardware of Horizon.
262.There were two KELs associated with this dealt within the bug table. The first was created in
2000 and last updated in July 2007, and refers to 88 PEAKs. The second was created in 2002 and
notes occurrences running up to 2009. It is now accepted as a bug by the Post Office in paragraph 6
of Appendix 2, but is said to have had transient impact. Mr Coyne considered that when replacing a
counter within a branch, the process could result in “the total loss of a transaction”. Dr Worden stated
that the cause, recorded in the first KEL (which was created in December 2000 and last updated in
July 2007) was that Riposte was coming online from the Recovery mode too early, and causing
messages to be overwritten.
RLITO000005
RLITO000005
263.In theory, when a counter was replaced, it builds its messagestore by replicating with its
neighbours in “recovery mode”. The neighbours it has depends on the office size (which would affect
the number of other counters) and node number. For a single counter office, the neighbours are the
correspondence server in the datacentre and the mirror disk (the second hard drive in the same
counter). For a multi-counter office, the neighbours are the correspondence server and all other nodes
at the office, or all the other nodes in the office (known as slaves) depending upon the node number of
the counter being replaced.
264.A replacement counter is supposed to come out of recovery mode when it believes it has
successfully replicated all relevant messages from its neighbours. The Post Office submissions state
that “In this case, the replacement counter came out of recovery mode early, before it had replicated
all messages from its neighbour. The replacement counter started writing messages from the point at
which it believed it had replicated all relevant messages from its neighbour. This meant that it used
message IDs that had been used for messages that had not been replicated from its neighbour and
this prevented the “missing” messages from being replicated later on (because that would have
created duplicate message IDs). The missing message was therefore “overwritten” by the replacement
counter.”
265.The Post Office also stated that “the issue arose in cases of counter replacements where the new
counter was not connected to all of its configured neighbours while rebuilding. This may have been
because the branch infrastructure was not complete (eg not all neighbouring counters are online,
multiple swaps or a counter increase/decrease occurring) or the engineer did not connect the system
properly.” Regardless of the reason, the failure to replicate messages is a failure in the internal
processes of Horizon to cope with the hardware replacement of the counter, and in my judgment is a
bug, error or defect.
266.PEAK PC0052823 gives a technical explanation for this, part of which is:
“Tt would appear that after the recovery from the squirrel completed, the message processor came out
of recovery mode after synchronising up to message number 660 for node id 1. Suspect this was by
replication with the Correspondence server.
Counter then wrote a Riposte on-line message as 661 for node 1 before 1 second later attempting to
synchronise to the F: drive mirror message store. At this point a red event regarding ‘self originated
message’ was generated, the server switched back to recovery mode and the remaining messages
from the F: drive mirror message store above 661 were synchronised.”
It also notes that “Gareth Jenkins viewed this error on rig with Mike Berrisford.”
267.The nature of the correction in the KEL was stated as being “to find the overwritten transactions
for reconciliation we need to look at the Ripostemirror messagestore' followed by detailed
instructions”. Riposte, as has been explained, was part of Legacy
Horizon and was a product provided by Escher, but it is plainly part of Horizon. Dr Worden also stated
that “the incident arose from a hardware replacement (probably from a hardware fault) not from a
fault in Horizon. It is a different kind of recovery issue.” In my judgment the hardware is part of
Horizon.
268.There was a short term and long term fix. The former dealt with the actual branch by the SSC
inspecting the Riposte mirror messagestore, and retrieving the specific messages which had been
overwritten. Information of the overwritten messages was passed to MSU who created a BIMS report
RLITO000005
RLITO000005
for the Post Office. The Post Office submits that “an error notice would have been issued to hold the
branch harmless thereafter”; no such error notice has been produced but I consider that the KEL and
PEAK together suggest that such a notice was produced in order to correct the impact upon branch
accounts.
269.The long-term fix for the issue was an actual fix, which is detailed in PC0052823. This involved
enforcing a minimum number of local neighbours for replication and then to slowly lower this number
over time. A further change was also made to stop Riposte writing messages as it came online. This
further change was a change to how Riposte operated. This therefore means that the fix changed the
software of the system.
270.The dates of other PEAKs shows that this persisted.
271.PC0071836 is accepted by the Post Office in Appendix 2 as an example of the same issue as KEL
JBallantyne5328R; this is dated November 2001. This SPM had a receipts and payments mismatch of
£3.27 as a result of three overwritten messages
following a replacement of the branch’s single counter. The same fix was applied following KEL
JBallantyne5328R. PC0133822 is not the same issue as JBallantyne5328R but is accepted as being
related. This PEAK is dated 24 March 2006. The branch had two counters removed, leaving it as a
single counter branch. However, the counter did not have a mirror disk; the mirror disk was a second
hard disk within a single counter that provided a replication neighbour for the main hard disk
messagestore. This meant that the branch would have no replication of data if it was not connected to
the datacentre and six messages on the counter had not been replicated to the data centre. The
messages were extracted and sent to MSU for a BIMS report to be raised. The Post Office submits
that “An error notice would have followed the BIMS report and so there would have been no lasting
impact on branch accounts”. However, that effectively accepts that absent the error notice, the branch
accounts would have been affected. PC0153851 is not the same issue as JBallantyne5328R but it does
involve a receipts and payments mismatch; it is dated 7 February 2008. Riposte failed to index four
messages resulting in some items being missing from the receipts side of the balance report. The
PEAK itself notes that the branch did not experience a discrepancy as a result because this was a
reporting issue only; indexes are not used when replicating data and so cash/stock were unaffected.
This does however show that the problem with Riposte persisted.
272.1 find that this was a bug, error or defect with potential lasting impact to branch accounts. The
fact that it originated with the necessity to replace a counter (which is plainly hardware, as Dr
Worden notes) does not matter because the hardware is part of the Horizon System. Nor does the fact
that the impacts of its occurrence were corrected by means of error notices/TCs affect the existence
of the fault. A change, or changes, had to be made to Riposte to stop this occurring, and there were
still later manifestations of it occurring.
13. Withdrawn stock discrepancies.
273.This is a Horizon Online issue. The Post Office does not accept these as a bug. Mr Coyne
maintains that it is, and in the bug table he extracted part of a PEAK that stated these “Can cause
confusion and unexpected (though hopefully temporary) discrepancies at branches by allowing them
to declare stock which has already been withdrawn.” Dr Worden stated that “some impact on branch
accounts cannot be ruled out, although it is small”. The Post Office’s detailed submissions in Appendix
2 of its Closing Submissions concentrated on the fact that when the Post Office withdrew stock -
which of course is part of the way that the Post Office manages its business, adding and removing
RLITO000005
RLITO000005
types of products from time to time - this removal of products is done by means of an update to
reference data, “and not a change to the core code in the system.” This again is concentrating on the
code, rather than the way the system operates.
274.There were two steps that led to this happening. The Post Office submissions on this rather gloss
over the entries in the PEAK, so I will reproduce the actual entries of 19 January 2011 in PEAK
PC0207834. Although that is the date of PEAK, the text makes clear that the event occurred in
November 2010. The context is that the Post Office withdrew the £5 saving stamp from its business,
but obviously some branches retained (or had at the time they were withdrawn) such stamps in their
stock:
“This office physically held 137 £5 PO saving stamps and did not rem them out before the date the
rem out icon disappeared. The office physically returned the stamps to Transaction Processing as
advised and the office then did a Trading Period balance on 17/11/2010 and showed this value as a
loss. The spmr put the cash into the till at this point to make good the loss. Transaction Processing
then issued a transaction correction to the branch on 19/11/2010 for the value of £685 to make good
the loss. The office brought this to account and when they did the next Trading Period balance on
15/12/2010 it showed a gain of £537.49 which was settled centrally. The spmr says that they did have
a loss showing again in the balance for the value of the PO Saving stamps, £685, again, even though
this stock line was not showing in either adjust or declare stock. After this Trading Period balance the
spmr took the £685 back out of the till and when they balanced again on 12/01/2011 it again showed a
loss of 137 £5 PO Saving stamps and the nett discrepancy in the branch was £1370, which again was
settled centrally. The office declares stock when they do the balance but insist there is no line showing
in either adjust stock or declare stock for the PO Saving stamps. The spmr is convinced the £1370
loss, which is twice the £685, is due to the fact these stamps are still on the system somehow and
keep giving the office the losses. The office has faxed a copy of their stock holdings for the branch at.
the moment and it clearly shows “Saving Stamps £5 137”, This stock line should have been removed
when the stock item was made obsolete and removed from Horizon,
The two user names who do the balance are SKI001 and PCA001.
I have spoken to Phil Herrett in Transaction Processing who has confirmed he is aware of about 8
offices with similar issues with the stamps still showing on the stock. He gave me FAD 0640123 as an
example.
Can this be investigated to see why the stock is still showing in the office, and if this keeps giving the
office a loss of £685 every time they do a Trading Period balance.” (emphasis added)
275.The Post Office submissions state that “However, the Subpostmaster in this case returned the
stamps (with a total value of £685) to Post Office without first remming them out. This aspect of
PC0207834 was pure user error.” However, that is not what the first entry in the PEAK states. It states
that the SPM could not rem them out because the rem out icon had disappeared for this stock. That is
undoubtedly an error by the SPM. However, the fact that the withdrawn stock remained on the stock
holdings, which are part of the branch accounts, even after the stock has been returned and a TC
issued to correct the discrepancy for their value, cannot be laid at the door of the SPM. Further, other
offices (“about 8” of them) were also experiencing the same issue. This is an error or defect with
Horizon in my judgment. Part of the submissions by the Post Office states as follows:
“8. The branch declared a £685 shortfall. The lack of a rem out meant that Horizon thought that the
branch was still holding the stamps and therefore when the Subpostmaster either declared or
RLITO000005
RLITO000005
adjusted the stock of stamps to zero without a corresponding gain in cash, a shortfall was generated.
The Subpostmaster elected to make good the shortfall and a credit transaction correction was
subsequently issued for £685 to rectify the issue.
9. However. a bug in Horizon caused the £685 of stamps to be subsequently reintroduced into the
branch’s accounts on two occasions. By this point, Horizon was showing that the branch was holding
£1,370 of the stamps. The Subpostmaster noticed this, adjusted or declared the stock as zero, and
reported the issue.” (emphasis added)
276.The Post Office therefore accept in those submissions that the subsequent effect upon the
accounts was a bug in Horizon. I do not consider it much matters whether this is characterised as a
bug, or an error, or a defect in Horizon. It plainly was a bug that had the potential to cause lasting
impact to branch accounts (and in this case actually did so prior to the issue of the TC).
14. Bureau discrepancies.
277.These relate to foreign exchange, hence the name. It must be differentiated from alleged bug at
entry 23 in the Bug Table (also concerning foreign currency, but entitled Bureau de Change). This is a
Horizon Online issue and arose in 2017.
278.Mr Coyne considered it to be a bug, and Dr Worden effectively agreed in that his entry stated
“This appears to be a system error with impact on branch accounts. Although it is possible that a
subsequent discrepancy between branch accounting and POLSAP would reveal the problem, leading
to a correction (e.g. see Peak PC0265443, and Mr Coyne's para 3.146), I cannot be certain of this.” It
is now accepted by the Post Office in paragraph 7 of Appendix 2 as one of a number of “bugs with
lasting impact (although they were resolved)”. The detailed part of Appendix 2 states that Mr Coyne
has drawn together two distinct issues. Paragraph 4 of the detailed submissions also states the
following:
“Bug 14: Bureau Discrepancies is a bug with the potential for lasting financial impact. There are two
distinct issues which fall under this heading. With regards to the first issue the branch was made good
and a fix was implemented. The second issue was not a bug in Horizon nor an issue which could have
impacted branch accounts; it created what was essentially a cash flow problem for the branch.”
279.So far as the first issue of the two identified by the Post Office is concerned, given a software fix
was required, it is undoubtedly a bug, and it is correct to record (as the Post Office do) that there was
the potential for lasting financial impact, as without that, there would be no need for the “branch to
be made good” because that means the Post Office had to correct the branch account discrepancy
caused by the bug. Ina sense that is stating the obvious. Given those submissions, I will deal with the
first issue very briefly.
280.This occurred as follows. In August 2017 an SPM tried to pre-order £1,000.07 in Indonesian
rupiah and £204.59 in Singaporean dollars for a single customer. The rupiah order was created, but
there was a network timeout at the point when the SPM tried to perform the dollar order. When the
system re-connected, a warning message suggested that the second order may not have been placed,
but the basket and transaction log were showing both orders. The SPM attempted to cancel the whole
order, but the cancellation only worked for the rupiah order, leaving the dollar order of £204.59 in the
branch accounts. This is because as recently as August 2017 multiple currency orders were processed
as multiple transactions. The rupiah order was added first at the counter and then sent to the BAL,
and this caused the order ID PBX1048411 to be created. The network timeout occurred at this point -
RLITO000005
RLITO000005
such a timeout has nothing to do with the SPM. At that time that the dollar order had been added to
the counter stack but it had not yet reached the BAL. This meant that it did not become associated
with order PBX1048411 and so, when the SPM attempted to cancel the whole transaction
(PBX1048411), the dollar order was not cancelled. This then created a shortfall as Horizon will have
expected the SPM to have taken a payment from the customer of £204.59.
281.This issue could only occur because of the very specific circumstances of PC0261541 including
the occurrence of a network timeout at a very precise moment in the transaction. The Post Office’s
detailed submissions state the following:
“The PEAK notes that the issue was referred to Post Office in order for them to “decide what
reconciliation or transaction correction is required to balance” at the branch. Post Office have
confirmed that a transaction correction was issued to the branch on 2 November 2017 and accepted
by the branch on 7 November 2017 - this is not apparent from the PEAK itself.”
282.This means that the accounts of that branch were incorrect, and in the Post Office’s favour, for
the period mid-August to mid-November 2017. A change to the software was made in November 2017
via a change to the AP-ADC script which had the effect of simplifying multiple currency orders to be
processed as a single transaction.
283.The second issue is even more marked, in my judgment, in terms of arriving at the correct answer
to Horizon Issue 1. PC0265443 is dated 29 December 2017 but relates to a call on 17 December 2017.
The Post Office’s internal cash management team’s system (which is referred to as POLSAP) indicated
that the branch in Burgess Hill was holding €4,500 and US$1,000 more than the branch was declaring
it held. This meant that, from the cash management team’s perspective, the branch held sufficient
cash to conduct transactions, and so were reluctant to provide more cash to the branch. This resulted
in a cash flow issue for the branch who were sometimes unable to serve customers as a result. This
PEAK is 57 pages long; this is many pages longer than most PEAKs, which are usually 10 pages or
less. It evidences a very long investigation.
284.This investigation is best summarised by quoting the Post Office’s own submissions under the
heading “Resolution”:
“13. The Peak is long and complex and it demonstrates that the issue was extensively investigated by
Post Office, ATOS, Accenture and Fujitsu, since the issue was thought to be occurring somewhere
between the branch, Horizon and POLSAP. The conclusion of Fujitsu and Accenture’s investigations
(over approximately six months) was that the issue was not caused by any issue within their
respective systems. Further, Post Office sent a trainer to the branch to verify the cash position and
found no discrepancies; the amount of cash physically held in branch was the same as the amount
14. The root cause of the issue does not appear to have been determined. As it did not appear to be a
case of user error, it appears that Post Office wrote off the discrepancy in POLSAP at no cost to the
Subpostmaster to bring that system’s figures into line with those in Horizon.”
(emphasis added)
285.This shows that POLSAP was not in line with Horizon; that no problem could be found by Fujitsu
with its system, nor by Accenture; but that the amount of cash (in terms of foreign currency) in the
branch was being correctly declared by the branch.
RLITO000005
RLITO000005
286.Not only do I find that this is evidence that supports the existence of this bug in the Bug Table,
which potentially causes lasting impact to branch accounts, but it is also consistent with my finding on
a separate issue, namely the need to use ARQ data in circumstances where there are serious disputes
between the Post Office and SPMs, rather than Post Office management data.
15, Phantom Transactions.
287.These arise under Legacy Horizon in terms of the dates. There are three different issues grouped
under this heading. The Post Office does not accept these as a bug. Mr Coyne in his cross-examination
stated that this was referable to Horizon Issue 4, rather than Horizon Issue 1. He also accepted that
some of the issues were not bugs in Horizon. Dr Worden relied upon the fact that what he called “the
master PEAK” was closed as “no fault in product”; however, the evidence of fact on this is dealt with
in the substantive judgment where I consider the evidence of Mrs Van Den Bogerd when the PEAK
was put to her.
288.Dr Worden stated that “There is no evidence for bugs in Horizon with impact on branch
accounts.” That entry was obviously before the cross-examination of Mrs Van Den Bogerd, which in
my judgment provided greater factual information. I do not consider reliance can be placed upon the
Fujitsu conclusion in the PEAK.
289. PEAK 0065021 is dated 17 April 2001. I consider it to be an important PEAK. It relates to
multiple branches. It concerns phantom transactions. It identifies dissatisfaction from more than one
SPM as to how phantom transactions are being investigated and resolved, or more accurately, how
they are not being. It shows calls being “closed” - ie brought to an end, without the permission of the
SPM, even though that should not be done. It also shows at least one SPM threatening not to comply
with their contractual obligations due to lack of confidence in the system, and also threats of legal
action. Further, in one branch, where items were the subject of phantom transactions (according to
the SPM) ROMEC, the Royal Mail’s own engineers, attended that branch to fit suppressors and other
equipment in an effort to rectify this.
290. The PEAK plainly records the involvement of ROMEC, the Royal Mail’s own engineering
personnel, as follows. “ ROMEC have been to site and state that they have actually seen the phantom
transactions , so it is not just the PM's word now.” (emphasis added). I consider that this is significant.
Mrs Van Den Bogerd agreed that this was “independent site visit corroboration of the problem by
Royal Mail’s own engineers at the branch”, and she also agreed that this was “clearly not user error
any more”. This entry in the PEAK is, in my judgment, important corroboration from those with
experience of Horizon (the Royal Mail’s own engineers) who state they have seen the phantom
transactions. The conclusion reached by Fujitsu and recorded in the PEAK was as follows:
“Phantom transactions have not been proven in circumstances which preclude user error. In all cases
where these have occurred a user error related cause can be attributed to the phenomenon." The
PEAK also concludes “No fault in product".
291. I reject both of those conclusions. In my judgment, they are both self-serving and are
contradicted by the factual matters reported within the PEAK by Fujitsu, which include that these
have been witnessed by ROMEC engineers.
292. A Fujitsu entry from 19 June 2001 in PEAK PC0065021 states:
source being the screen. This has been seen at Old Iselworth (OI) and Wawne (W) with OI being the
RLITO000005
RLITO000005
best site; when the PM has been asked to leave the screen on overnight Lhave observed system
system activity or human interference, the way forward now is to correlate this with the Microtouch
supplied monitoring software and to this ends Wendy is arranging for installation of Kit at OI on
Friday, we can then, provided the PM agrees, leave screens on over the weekend and record what
happens.
Once these results have been analysed I feel sure that we will be in a position to move forwards at OI.
All other cases should be considered on their individual merits but you must appreciate that this is a
fairly intensive analytical activity and I cannot hope to provide answers on all cases in the short term.”
293.1 consider this to be evidence of a bug in Horizon. Mr Coyne notes that there is no specific branch
account discrepancies noted in this respect; however, given Fujitsu concluded in November 2001 the
following “Phantom Txns have not been proven in circumstances which preclude user error. In all
cases where these have occurred a user error related cause can be attributed to the phenomenon. I
am therefore closing this call as no fault in product” then the lack of noted discrepancies for this is
not surprising. Nor, in my judgment, is it necessary for there to be specific identified discrepancies
given the word “potential” in Horizon Issue 1.
294.Both the ROMEC engineers who had observed the phantom transactions, and indeed Mr Carroll
in his entry of 19 June 2001, considered based on the evidence they had at that time that there were
so-called phantom transactions occurring. Mr Carroll’s entry in particular that states “when the PM
has been asked to leave the screen on overnight I have observed system activity corresponding to
screen presses happening with no corresponding evidence of either routine system activity or human
interference” can only sensibly be explained by a bug. The Post Office submitted that “For the reasons
set out above [in its submissions], there is no indication that phantom transactions had a cause other
than user error, as indicated in the PEAKs.” However, that ignores the entries in what was referred to
as the master PEAK that indicate precisely the opposite.
295.1 consider that an inference, that is to say a common sense conclusion, can be drawn from all the
evidence on these matters that there was such a bug in 2001 that caused phantom transactions. This
had the potential to cause impact to branch accounts.
16. Reconciliation issues.
296.This heading in the Bug Table grouped both Legacy Horizon and Horizon Online issues with
reconciliation together. The Post Office does not accept these as a bug. There are a number of
different issues grouped together in this heading. The issue was that the SPM was shown a
discrepancy on his or her screen. Mr Coyne accepted that the discrepancy would not be shown in the
branch accounts; Dr Worden stated that “as it concerns an issue in reporting, the software fault
(which was fixed after 5 months) had no direct impact on branch accounts. The only effect of an error
in this report would be to mislead or confuse the Subpostmaster - probably leading him to check his
figures more carefully and costing him some time.” (emphasis added)
297.That entry by Dr Worden plainly accepts a software fault; the fault was in fact a miscounting of
the number of files by the system. However, there are six different PEAKs grouped under this heading
and the process of reconciliation, which is effectively the comparison by Horizon of two different sets
of data, is part of its design function. The recovery messages were held in the branch messagestore.
Mr Coyne in his cross-examination also stated that this was referable to Horizon Issue 4, rather than
Horizon Issue 1. Three of the six different issues were missed off the copy of the submissions lodged
RLITO000005
RLITO000005
for oral submissions in July 2019, but this was made clear by the omission of “Issue 4” to “Issue 6”
and following submission of Mr Parsons’ 19th witness statement these were added.
298.This is one area where the cross-examination of Mr Coyne in particular does lead to a conclusion
that this is not a bug that causes potential lasting impact to branch accounts. The six matters
identified by the Post Office in the detailed part of Appendix 2 all deal with different PEAKs.
299.Peak PC0039832 is dated 3 March 2000 and is an example of an issue affecting a SPM’s Cash
Account Period. Reconciliation discrepancies appeared but did not feature on the expected reports.
The counter calculated information each day for the Cash Account in two independent ways to ensure
that they matched and a report was generated if they did not match. In this case, the reconciliation
software detected discrepancies relating to two low value transactions (£8.06 and £0.08, totalling
£8.14). The PEAK suggests that “there was a bug in the reconciliation software, although the Peak is
not fully conclusive” as the Post Office’s submissions recognise in this sentence that I have quoted.
This would have generated a false reconciliation report but there would have been no financial impact
on the branch. The reconciliation error was captured, as that is what triggered the investigation in the
PEAK. The fix is documented in PC0047955 of 19 June 2000. It is evidence of a bug but given the
nature of it, it would not have impacted upon branch accounts.
300.PEAKs PC0075240, PC0075415 and PC0077508 are all from April 2002. These all relate to the
same (or at least a markedly similar) issue where a branch counter total differed from the amount on
the TPS host. There was no discrepancy in branch accounts. In all these cases, the counter calculated
a value of 1p and the calculations carried out at the host gave a value of £0.0099. As the first three
digits were 000 the system took this to be Op and so reported a false discrepancy. There was however
no financial discrepancy. A fix was however required, which demonstrates it was a bug, error or
defect, but it was not one with the potential to affect branch accounts.
301.PEAK PC0049578 is from July 2000. It was raised in testing and fixed before the software went
live. It was a bug, error or defect but the purpose of testing is to discover such things, and as such
this had no potential to impact branch accounts. The problem was producing a report used to confirm
that all data has been passed to TIP. The underlying data transferred to TIP was correct, however, the
number of files transferred did not match. There was also a report to confirm what files were
transferred, and that report was also incorrect in the count of files transmitted. If this had gone to live
then the answer to this might be different, however regardless of that, it was fixed before release.
302.The fourth issue relates to a PEAK dated May 2000. The submissions stated that “the
Peak describes an automatic detection of a receipts and payments mismatch of £4,464.46. It was
automatically reported to Fujitsu and was investigated. The issue related to a hardware failure. It
therefore did not relate to reconciliation issues arising as a result of a software fault.” It is however a
defect or error with the Horizon System, which comprises both hardware and software.
303.PEAK PC0236246 is dated 12 September 2014, It relates to a discrepancy of £110,706. The
Fujitsu entry in the PEAK is somewhat illuminating:
“This problem is likely to occur whenever products are introduced with non-midnight times, a
common occurrence when data is created (and subsequently corrected) in MDM. Based on current
data, such products may be seen roughly once a month.
RLITO000005
RLITO000005
When it does occur, any transactions against the products are omitted from the CTS report, resulting
in discrepancies in trading totals, and operations are required to investigate, usually requiring the
assistance of development.
The CTS Report is used by POL to settle payments with their Clients. Incorrect values on the CTS
POL's lack of confidence in our IT Solution will not help our bid for the renewal of the Front Office
contract.”
(emphasis added)
304.CTS stands for Client Transaction Summary. In my judgment, the error shown in this PEAK is
plainly a software fault that affects the accuracy of accounting at the Post Office. However, in this
case, it has an effect upon the CTS Report, used to account between the Post Office and its clients,
and not branch accounts. There was therefore, under this heading, a software fault, however it had no
potential to impact upon branch accounts for the reasons I have identified above.
305.The final of the six different issues also relates to the CTS Report, but one of the two PEAKs is
that which I have already reproduced at [303] already. The other is PEAK PC0204872 and is dated 4
October 2010. The Post Office submissions state that “the issue relates to a difference between the
CTS Report and the branch figure. The root cause identified was an issue with POLSAP, and not with
Horizon”. The PEAK records that there was a problem with the CTS report for a branch concerning
the Alliance & Leicester:
“The CTS report is received daily and is compared with the vendor (in this case A&L) reports. The
figures for each day should match.
If the CTS report is larger than the vendor figure, the vendor account will be credited. The credit
usually shows a couple of days later as a positive discrepancy.
The CTS report was showing as being larger than the vendor figures on the following dates, although
there does not appear to have been any counter credit showing on the vendor figures following on
from this:
7th May 2010 - CTS was greater than vendor figures by £84.86. POL have suggested that this may
have been related to an event from 27th February for FAD 490519, although we can find no BIMS
record of this from a Reconciliation perspective.
25th July 2010 - CTS was greater than the vendor figures by £3,260.00. No additional information is
available.
27th August 2010 - CTS was greater than the vendor figures by £846.00. No additional information is
available.”
306.The next entry is:
“Following this additional information this would appear to be an issue with POLSAP.
The SSC do not support POLSAP, so we can take no action here.”
RLITO000005
RLITO000005
307.That submission and the entry in the PEAK is of assistance in dealing with the ARQ data/
management data dispute, but does not necessarily evidence a bug in Horizon potentially impacting
branch accounts. It does not, however, for obvious reasons, take the matter very much further
forwards.
308.1 find that these PEAKs, although demonstrating software faults, do not show that there is a bug
with the potential to cause impact to branch accounts
17. Branch Customer discrepancies.
309.This was a Legacy Horizon issue and is an entry only in respect of Horizon Issue 4 and recorded
as such in the Bug Table. The Post Office does not accept these as a
bug. Mr Coyne accepted in his cross-examination that the entry in the PEAK did not suggest any
impact upon branch accounts. The entry in the bug table originated in a single PEAK, although there
were two PEAKs that relate to the same issue, which was a discrepancy between the financial records
held by the Post Office (from Horizon) and a bank’s records, after a counter crashed whilst a
transaction was being processed. The amount in question was £165.26. The date of the PEAK is 29
March 2008 and the incident occurred between 22 March 2008 and 26 March 2008 (the date of the
card account).
310.The SPM did not take the sum in question from the branch customer, but that person had their
account debited. The first PEAK was raised by Fujitsu who picked up on this as part of automatic
reporting (an NB102: Exception Summary report) and the SPM was advised how to go through
recovery. Fujitsu had issued a BIMS Incident Report to the Post Office, and had confirmed that the
recovery messages were in the branch messagestore (even though they do not appear to have been
shown by Horizon to the SPM in the first place). However, the branch appeared in the NB102 report
again, and upon investigation this was found to be because the SPM (quite properly, in my judgment)
had not followed the recovery messages because they had not taken the sums from the customer.
311.This is effectively accepted by the Post Office in its submissions which state: “Fujitsu issued a
second BIMS Incident Report to Post Office which advised that while the recovery messages were
received by the SPM, the SPM had declined them. This indicates that no money changed hands during
the transaction (i.e. the SPM elected not to recover the transaction because no money had changed
hands; if money had changed hands, the SPM should have elected to recover the transaction).”
312.The Post Office also submits the following:
“While there was a possibility that the Financial Institution registered a withdrawal from the
customer’s account (depending on how the Financial Institution’s IT system is configured), this would
have caused a loss for the customer, not the branch (assuming that the customer received no cash
from the branch). If the customer did receive cash from the branch, the resultant discrepancy would
have been caused by the branch not following the recovery procedure correctly.”
313.1 find that a very surprising submission. The counter - part of Horizon - crashed during a
transaction. That is not the fault of the customer, and it is not the fault of the SPM, nor is it the fault of
the customer’s bank. I do not know if solace is taken by the Post Office that any loss caused to the
branch customer can, apparently, be laid at the door of the customer's branch, but I doubt solace can
be properly taken. Nor do I see how the SPM can be criticised for “not following the recovery
procedure correctly” as it is accepted that this should have occurred if the SPM had taken money
RLITO000005
RLITO000005
from the customer, but if they had not, it should not have been followed. On either analysis, this is a
fault with Horizon.
314.However, it does not appear to have occurred more than this single occasion. I agree with Mr
Coyne, and this is relevant to Horizon Issue 4 only. I consider that it is an error or defect in Horizon,
but does not constitute one that had the potential to cause impact to branch accounts.
18. Concurrent logins.
315.This is a Legacy Horizon bug. It is accepted by the Post Office in paragraph 7 of Appendix 2 as
one of a number of “bugs with lasting impact (although they were resolved)”. In paragraph 2 of the
detailed part of Appendix 2 dealing with this bug, the following is stated by the Post Office: “Post
Office accepts that Bug 18: Concurrent Logins had a potentially lasting financial impact. There is no
evidence of any discrepancy in the PEAKs referred to by the experts.” The problem was in the early
days of Legacy Horizon when it was possible for users to log in to two terminals at once. Dr Worden in
the bug table made the following useful summary statement:
“discrepancies could occur - manifesting themselves as a receipts/payments mismatch. This had the
potential to affect branch accounts. The mismatch would bring it to the attention of the
Subpostmaster, who would require it to be investigated, except possibly in the case of small
mismatches, which he might pass off as an error in the branch (e.g. of counting stock).”
316.He also stated that “....Fujitsu believed it was a problem with the underlying Riposte software,
and passed it to Escher. In September 2000, the problem was ‘Now formally fixed in Build 223 update
19 which was released overnight.' However, the new release from Escher did not, as it was expected
to, fix the problem. Escher denied that it was a bug in Riposte, but Fujitsu believed in July 2001 that
“This is clearly a bug in the Supplier's code',”
(emphasis added)
317.Paragraph 2 of the detailed submissions by the Post Office in its Appendix 2 state:
“[{The] Post Office accepts that Bug 18: Concurrent Logins had a potentially lasting financial impact.
There is no evidence of any discrepancy in the PEAKs referred to by the experts.”
318.Dr Worden considered that “these faults had the potential to produce discrepancies in branch
accounts, of small amounts, for a short period of time”. How the bug is therefore characterised
depends upon my findings in relation to lasting and transient impact, which is dealt with in the
substantive judgment to which this is an appendix. I find that this was a bug, with the potential to
cause lasting impact to branch accounts. Dr Worden attempts to play this down and so, in my
judgment, does the Post Office. Fujitsu clearly recognised and recorded that in 2001. The fact that the
bug was in the code supplied by Escher, as part of Riposte, and not that of Fujitsu is wholly irrelevant
for the purposes of the Horizon Issues.
319.One of the PEAKs related to the printing of a report whilst a counter crashed. The Gateway
counter was then used, and because the first counter had crashed (what is called a “slave” counter)
this was permitted because it could not communicate to the Gateway counter who was logged in to it,
or what it was doing. This PEAK is PC0027581 and relates to an incident in July 1999. That PEAK runs
to February 2002. It is notable, and continuing the theme of Fujitsu’s closure codes, that it is closed
with the code “Administrative Response” even though the entry for 13 July 2001 states “This is clearly
RLITO000005
RLITO000005
a bug in the Supplier’s Code and as you say, management pressure must be brought to bear as
necessary to make the Supplier accept and respond to that fact.”
320.The second PEAK is PC0051327 dated 2 August 2000. The entry which opened the PEAK is:
CALL PC0051327:Priority B:CallType L - Target 02/08/00 12:32:13
28/07/00 12:22 office 182432 reports for CAP17 a receipts total of £412224.58 and a payments total
of £430724.58. the difference is £18500.00. this office earlier raised a query because a transfer for an
amount of £9250.00 seemed to have gone missing. the amount of the transfer is exactly half the
amount of the difference between the receipts & payments. a call raised earlier PC0050974 was
closed in error.
28/07/00 12:27 GB082158
Information: passing to EDSC as requested.”
(emphasis added)
321.A further entry is:
“PM advises that the problem started when she transferred from counter No8 (by mistake - should
have been from counter No1) £5590 to counter No3 and £3660 to counter No4 - while No8 was in the
process of rolling over! Consequently, the cash left counter No8 in CAP 18 and arrived in counters
Nos3 and 4 in CAP 17. She eventually rolled the office, accepting the £9250 discrepancy and then
transferred
£9250 from counter Nol to counter No8. How was she able to transfer out of counter No8 while it
was rolling over?”
322.Part of the Post Office submissions are:
“19. The net difference of these receipts and payments mismatches amounted to zero, meaning there
was no overall impact on the branch. This would have been clear to Mr Coyne as the very nature of
this issue is not that it causes money to be lost, but money to be accounted for in the incorrect time
period. The PEAK shows that there are offsetting receipts and payments gains and losses and
therefore it is possible to ascertain from the Peak whether the branch suffered a discrepancy.
20. The issue was fixed through a planned software roll out that changed the code in the area that
caused the bug (release C145) and the call was closed on 30 November 2000.”
(emphasis added)
That applies to the second PEAK but not to the one headed “Simultaneous Logon” which is PEAK
PC0027581, which was closed in February 2002.
323.1 find that this was a bug in Horizon which plainly had the potential to cause impact to branch
accounts. The fact that in this specific case the discrepancy was later corrected does not, in my
judgment, alter that characterisation. Indeed, on the Post Office’s own case, the accounts were
affected (albeit corrected). Further, the way that PEAK PC0027581 unfolded - a lengthy PEAK which
runs from July 1999 to February 2002 - shows the unsatisfactory way some of these incidents were
dealt with at Fujitsu. The entry of 9 November 1999 states “....at first glance it could have a fair bit of
business impact”. Even as late at August 2001 and January 2002 this PEAK was still given the code
RLITO000005
RLITO000005
“Incident under Investigation”. One wonders quite how long Fujitsu would need properly to
investigate something identified as “a bug in the Supplier’s Code” in July 2001. The final entry
contains the sentence “Mr Lui [who is the SPM] is no longer employed by the Post Office and has not
been for some years.”
19. Post & Go/TA discrepancies in POLSAP.
324.This is during Horizon Online and occurred in 2012. It is accepted as a bug by the Post Office in
paragraph 6 of Appendix 2, but is said to have had “transient” impact. The entry by Mr Coyne in the
Bug Table relates to Horizon Issue 4 rather than 1. “Post & Go” are self-service terminals, that are no
longer available in branch Post Offices, and are only available in Crown and WH Smith main branches.
They are basically self-service kiosks, designed to avoid queuing. A customer can weigh a letter or
parcel, the terminal will print the relevant stamps/labels, the customer deals with that themselves,
and the item is then posted without the need for queuing. The Post Office submitted that this issue
was irrelevant to the Horizon Issues trial as “this issue does not relate to branches that are the focus
of this trial”.
325.1 do not consider that this is irrelevant as the Horizon Issues require consideration of bugs, errors
and/or defects in the Horizon System.
326,One of the PEAKs shows that Post and Go transactions were being accurately recorded by the
relevant terminal and then transferred to Horizon. Horizon creates a
TA (or transaction acknowledgement) of all the transactions on the P&G machine and, when accepted.
by the branch, the value of these transactions are added to the branch accounts. The data is also
passed from Horizon to various other systems, including POLSAP. This incident was caused due to an
issue at the reconciliation step within POLSAP. Fujitsu transfers the relevant data to POLSAP via BLE
files. When POLSAP was comparing the Horizon data to the Wincor data, discrepancies were found as
data for two (of the total of six) terminals in the branch was not being sent to POLSAP. The Post Office
submissions state “There were six P&G terminals at the branch. Data from four terminals only were
being transferred to POLSAP because the two other terminals were not associated with conducting
P&G transactions”.
327.This is not an accurate summary of the PEAK. This states:
“An example the customer has provided shows amounts of 115.05, 46.88, 52.13 & 75.23 totalling
289.29 received on the file from Wincor and into POLSAP via BLE.
The same (contra) amounts are also showing as being received from the branch when the TA has been
accepted and are closed items in the account (netted off to 0.00).
However, there is another amount of 289.29 which just has the date in the assignment field.”
328.The Fujitsu response is:
“Postings on the TfS call refer to a similar previous incident (A1040049 => Peak PC021943293),
which was resolved between POL and Wincor Nixdorf; no details of this resolution are available to us.
This incident is a week old, but only came to SSC late last night... The trading-date in this call,
2012-08-09, is three weeks ago which too old for us to be able to see the incoming file from Wincor
Nixdorf... There is no evidence of a fault in HNG-X, and without the incoming file from Wincor Nixdorf
there is nothing further for us to investigate.
RLITO000005
RLITO000005
We can only suggest that POL do the same as they did with A1040049, and refer the matter to Wincor
Nixdorf.”
329.Anne Chambers records that:
“Branch 020511 has many entries in the Subfiles_on_hold report. This report should be monitored (by
?) to make sure problems are followed up - this should be resolved before closing this call.
Horizon is receiving PG data for 6 separate PG tills at the branch, but only 4 of them have associated
stock units. This causes the entire subfile for the branch to be Held, and the transaction data is not
being sent to POLSAP. However the TA data for the 4 tills which are properly associated IS being sent
through, and I think this is probably the cause of the POLSAP anomalies.
The two unassociated tills are not doing any cash transactions - this is a known problem (see
PC021870294), and means the PM isn't prompted to create an association. This may need fixing via
MSC.”
330.It was not that two of the six terminals were not doing Post & Go transactions, which is how the
Post Office submissions seek to have the PEAK interpreted. It was that they were not going cash
transactions. There are types of transaction associated with Post & Go that are not cash transactions.
They plainly are all six of them Post & Go terminals, as the PEAK records. If they were not doing Post
& Go transactions at all, there would be no problem because there would be no subfiles on hold, or
held.
331.As Mr Coyne said in his supplementary report “a bug fix to the Horizon system was identified by
Fujitsu, scheduled for implementation 13 September 2012 after 1800hrs and the Branches stock was
to be corrected at 1700hrs that same day.” On 17 September 2012 Anne Chambers herself reported in
the PEAK that:
“Following a change made centrally to facilitate this, the stock unit associations for the two new Post
and Go terminals have been created by the branch and all the held external data (43 different days)
has now been processed and passed through to POLSAP... We strongly recommend that POL monitor
the SubfilesOnHold report which is sent to them daily, so that any other external terminals with
problems can be investigated quickly in case a similar correction is needed”.
332.The “change made centrally” can only have been to the software. It is also the case that the
underling bug, error or defect was having an effect for 43 days, which is in excess of one trading
period which are four weeks long (and occasionally five weeks) but both those periods are shorter
than 43 days. It is also the case that POLSAP was wrong for that period, which is a factor which
impacts upon the dispute between the parties about the use of ARQ data/management data. The Post
Office submits that it was “quickly resolved” but I reject that. 43 days, or over 7 weeks, is not
resolving this bug (or its effects) quickly.
333.1 find that this is a bug with potential to cause impact to branch accounts. I do not know the date
when the Post & Go service was withdrawn from branch post offices, or how many branches had it
during the period when it was so available. The statements made in respect of this in the Post Office
submissions are said, in the clarification table, to be based on “instructions from [the] Post Office”.
The information is vague. I accept that this service is only now available in WH Smith and Crown
branches, but other details cannot be specified because there are none available from any source.
20. Recovery Failures.
RLITO000005
RLITO000005
334.The entry in the Bug Table identifies this as relating to Horizon Issue 4. These arise under
Horizon Online and not Legacy Horizon. The Post Office does not accept these as a bug and states
that there is no evidence of a bug in Horizon; it refers to the subject matter of the different PEAKs as
“issues”. There are three different PEAKs relevant to this. Mr Coyne accepted in his cross-
examination that this (or these) should be removed from the table that he used as the originator for
his list of bugs (which was Mr Coyne’s Table 1). Dr Worden stated in the bug table that “there was
some implication of hardware faults, with a replacement of a base unit, but the PEAK has no evidence
of software faults in Horizon.” In one of the PEAKs, he concluded that there was no evidence of any
fault in Horizon.
335.The situation in terms of PEAK PC0220532 is unsatisfactory. The Post Office submit that this
“relates to a cash discrepancy that was only raised by the Subpostmaster with Post Office and Fujitsu
nine months after the issue occurred.”
336.It is therefore clearly submitted that this took nine months for the SPM to report at all. This is
based on the date of the PEAK (15 September 2012) and the date given in the very first entry that this
happened on 6 January of the same year. However, that is simply incorrect on the face of the
document. The date of the PEAK does not mean that the SPM took that period to report the matter to
the Helpline or their manager, for example. Reporting in January by the SPM is in fact recorded in the
PEAK itself, but not in the first entry. Further, that submission ignores these entries later in the PEAK
for 6 September 2012 that refer to a TFS call being raised on 7 January 2012:
“However I have checked our call logging system which has provided some useful information.
According to our archived TFS call login system, the Node: 5 did have ?Physical memory Dump? on
6/1/12 and a TFS call (5043049) was raised. The node was swapped on 7/1/12.”
On 13/1/12, PM raised another TFS call (5060273). PM reported that she had a loss of £300, she
wanted to know if this was related to the Base unit fault. PM was referred to NBSC to investigate the
discrepancy.
On 13/1/12 NBSC raised a TFS call (5060420) and asked for the discrepancy to be investigated.
On 13/1/12 @16:32pm, the frontline helpdesk contacted the PM and the following information was
logged on the call.
?Spoke to the PM who states that she had the loss before the base unit was swapped out the loss was
for 190.00 PM states that she rolled over her TP on the Wednesday which had no loss or gains. PM
states that she then did a transaction log on the Sth Jan this shows a loss of 190.00 The base unit got
replaced on Saturday 7th Jan.
PM states that she had a different error message on the system every day but did not record the error
message PM states that she had physical memory dump message come up on the Friday 6th Jan and
the base unit was then swapped on the 7th Jan?” (emphasis added)
337. Later entries in the PEAK state:
“Assuming the cash declaration on 04/01/12 was correct, NBSC will need to go through the
transactions that were done in BB during this period (possibly by use of Transaction Log), to try to
determine where the discrepancy came from. The fact that on 04/01/12 there was a negative cash
discrepancy balanced out by a similar but positive stamps discrepancy, may be related to this issue.?
RLITO000005
RLITO000005
It is not clear what investigation/s were carried out by NBSC and no further TFS call was raised with
regarding to this discrepancy.
If further investigation by Fujitsu is required, Post Office will have to request that the branch
transaction data is retrieved from the audit server. If there is any possibility that this is required for
litigation, it must come through the Security (ARQ) route.
Otherwise queries of this nature should be sent via Mark Wardle at POL, and should be routed to the
reconciliation team in the first instance. Such requests may be chargeable.”
338.This trial is not to determine individual discrepancies, although they are identified where relevant
to the Horizon Issues. On the information before the court on this item, it is not possible to resolve
this. This means that there is insufficient evidence to make a finding of a specific bug in this case, or
on this PEAK.
339.The second issue under this heading related to PEAK PC0241242 on 23 February 2015. This
caused a discrepancy and the Post Office submits that “a Transaction Correction was done for the
shortage”. The shortage was £70.66, and the claimants challenge the Post Office on the origin of this
positive statement that this was done. In clarification, the Post Office relied upon Fujitsu stating in the
PEAK that the Post Office needed to do this. The PEAK states on 5 March 2012 at 1546 hrs:
“This office was doing the following cash withdrawal txn for £70.66 on 23/2/15 @10:46.
The session also contained a non financial Health Lottery txn.
The settlement failed due to poor communication with the data centre. The disconnected session
receipts were printed which advised PM to pay the money out. Spoke to PM who confirmed that the
money exchanged hands.
When PM tried to log back the recovery kicked-in. However the Health Lottery APADC recovery script
failed and left counter unusable.
Yesterday POL Branch Support team authorised us to remove/update the session.
Today I have carried out and completed the task. PM is now able to use the node again.
Reconciliation needed for the banking transaction:
The cash withdrawal txn was authorised and PM said they paid the money out.
This will leave this office £70.66 short (cash shortage) as the session not completed fully. POL need to
do appropriate reconciliation; transaction correction.”
340.However, a later correction in the PEAK for 5 March 2015 at 16.06 states “MSU please note that
it was a CAPO cash withdrawal for £140 and NOT £70.66.” This is repeated at 16.12. The entry for 9
March 2015 states “I have also advised MSU to do the necessary reconciliation and raise BIMS with
POL for the Cash Withdrawal txn for £140.”
341.It is clear therefore that the root cause of the issue was the failure of the AP-ADC recovery script
(Automated Payment Advanced Data Capture script, which is written by Atos). This therefore directly
shows that there was an error or defect in Horizon, and this led to a potential impact upon branch
accounts.
RLITO000005
RLITO000005
342.For the purposes of the Horizon Issues trial, this is relevant to Horizon Issue 4, as explained by
Mr Coyne.
343.The third issue shown in another PEAK is PC0197643 dated 14 April 2010. This is a failure of
automatic recovery. This will inevitably happen in any system, as accepted by Mr Coyne.
344.The Post Office submitted in respect of all three of these PEAKs the following:
“For the reasons set out above, this is not a bug at all. There are three distinct issues under this
heading. There is no evidence of a bug in Horizon and in any event, instances of any issue were
caught by automatic reporting.”
345.That is correct as far as it goes - the PEAKs are not evidence of bugs. The first two are however
evidence of errors and defects in Horizon. They do however go to Horizon Issue 4, not Horizon Issue
1.
21. Transaction Correction Issues.
346.This arises under Legacy Horizon. This is de facto accepted as a bug by the Post Office, but it is.
submitted it had no branch impact. Mr Coyne’s entry in the Bug Table was that “Transaction
Correction bugs/errors and defects do not cause discrepancies with branch accounts but” and he then
listed various consequences. In his report he referred to “technical flaws”. He also in his cross-
examination accepted that he did not consider this to be a bug with lasting financial impact. Dr
Worden, in respect of some PEAKs under this heading, accepted they were evidence of a bug, such as
PEAK PC0129587 where he stated “In my opinion this bug would result in an inconvenience to the
Subpostmaster (inability to rollover to the next TP) but would not result in inaccurate processing of
any TC, or any impact on branch accounts” (emphasis added). The bug is included by the Post Office
in the list of paragraph 5 of Appendix 2 under the heading “the following bugs had no branch impact.”
347.The submissions by the Post Office is that there is “no evidence of any financial impact upon
branch accounts”. It is therefore effectively accepted that it is, or there were, bugs responsible for
these issues, but it is also effectively agreed by the parties that there would have been no financial
impact upon branch accounts.
348.Transaction Corrections were introduced in September 2005, replacing the previous method
which were called Error Notices.
349.Some of the PEAKs, such as PC0120459 dated 4 May 2005 and others from around that time,
refer to issues with the TC button on the screen at that terminal in a branch not working as it should.
The so-called “pick list” shown on the screen would freeze. These issues were logged by the team at
Fujitsu called the Solution Validation and Integration team. This was part of testing. None of the
documents referred to by the Post Office in its submissions to justify the submission “As such, these
PEAKs all relate to issues raised by users during testing prior to the TC functionality going live in
September 2005 (when it replaced error notices)” were referred to in the trial. I do however accept
that they show that the TC process was tested prior to release to the whole branch network, and other
PEAKs also show this. The Post Office also submits that this was not “a bug at all”. If that is right,
then it must have been an error or
defect, but given that the documents show that it was fixed during the testing process, the precise
terminology probably does not matter.
RLITO000005
RLITO000005
350.Other PEAKs however relate to the period after September 2005 when the testing process was
over. PEAKs PC0129587 and Peak PC0130056 refer to the Petersfield branch, and are dated 4
December 2005 and 14 December 2005. They both relate to the same issue and the latter is what is
called a clone of the former. The first PEAK relates to a TC for a £9,000 debit and part of that PEAK
states:
“01/12/05 16:01 Caller states that each time they try to do a transaction correction the counter is
freezing up and going to system busy, has attempted on 3 different counters and the G/W was on
system busy for about 2 hours, pm has had to reboot 3 times attempting to do this.
01/12/05 16:07 uk955556
Advice: Asked pm to provide details regarding transaction and fault.
01/12/05 16:08 uk955556
Information: pm has attempted to run this correction on 3 counters now, 1, 5 and 2 and each time the
counter is freezing up and going to system busy, will not come back from this.
01/12/05 16:09 uk955556
Information: Transaction was for a premium bond on the AP system.
PM was working on stock unit - E at the time.
01/12/05 16:10 uk955556
Information: Transaction was carried out on 29th October at 12.23.
01/12/05 16:10 uk955556
Advice: Advised pm that i will have to look into this and get back to her.
01/12/05 16:10 uk955556
Information: pm is fine with this but states if it is not resolved by 14th December she will be unable to
rollover.
01/12/05 16:32 uk955556
KEL Ref No.: checked for kel's but all kel's found on transaction corrections refer to specefic events
which are not present in this case.”
The lack of previous KELs on this subject at that point means that this was a new issue which had not
arisen before. It cannot therefore be put down to testing which was completed by then.
351.A later entry on 6 December 2005 is:
“Several interesting points to note:
(a)111111111 subscription group on core server contains 35 more msgs than on the counters: 3280
compared with 3315 msgs. The extra msgs are just Bureau: CurrencyRates and CSvcMargins msgs.
(b)A couple of times after getting tired of waiting for the TC to process, the Clerk did a session
transfer to another counter which, according to EP/SPG/001, should not be possible.
RLITO000005
RLITO000005
(c)I have reproduced the problem by importing the mstore onto the SSC counters and setting a cut off
date of 01/12/05 11:27:10 which was before the PM?s first attempt.
(d)Peak PC0120459, raised on $80 E2E XI, reported the same symptoms and the root cause was found
to be missing/incorrect ref data. This call was investigated by Mike Coon.”
(emphasis added) (question mark present in original as “PM?s”)
352."Ref data” means reference data, which is part of the software of Horizon. In my judgment this
shows clearly that this was a bug. This was therefore known about by Fujitsu in 2005.
353.1 shall quote some paragraphs of the Post Office’s detailed submissions on the next PEAK:
“10. Peak PC0129587 relates to a report by a SPM on 1 December 2005 that each time she went to
select a TC for a £9,000 debit, which arose from an incorrect entry for Premium Bonds, her screen
froze and she was unable to accept it.
11.Peak PC0129774 and Peak PC0130057 refer to the Bosham branch. These PEAKs relate to a report
by a SPM on 6 December 2005 that each time she went to select a TC, a £22,500 debit for an
incorrect entry for Premium Bonds, her screen froze and she was unable to accept it,
12.Mr Coyne’s analysis proceeds on the basis that this issue was only experienced by these two
branches. Mr Coyne fails to refer to KEL LKiang2837P [dated 5 January 2006] which demonstrates
that: 50 other branches reported an issue with their screen freezing to Fujitsu during December 2005;
48 branches reported that this prevented them selecting an outstanding Camelot Lottery TC and
rolling over into the next trading period; and 4 branches reported that this prevented them selecting
an outstanding Premium Bond Sale TC and rolling over into the next trading period, including the two
branches referred to in the 4 PEAKs mentioned by Mr Coyne.
13.The diagnosis of the issue was that the text drafted into the TCs contained a string of 35
characters without a space. The code to put each TC on a screen was attempting to split the text at a
space (as is normal for a word processor), but the code was unable to process a string of this length.
14.The issue was resolved quickly. Having first been reported on 1 December 2005, a work-around
was developed whereby the affected branches were instructed to roll over all stock units except one
(the one relating to the unprocessed TC). This enabled them to roll over into the new trading period. A
software fix was then released on 22 December 2005 which enabled the branches to select and
process the outstanding TC, and roll over the last stock unit. To avoid the issue reoccurring in future,
Post Office confirmed that strings of this length without spaces would no longer be included in the
text of TCs.
15.Although included in Mr Coyne’s analysis of branch affecting bugs, the impact of Issue 2 [ie this
issue in this PEAK] was limited to the affected branches being unable to roll over to the new trading
period for a short period of time. It had no financial impact on branch accounts.”
(emphasis added)
354.The code explanation above was not contained in any evidence and was taken by the Post Office
in its submissions from documents that were not even deployed in the trial. However, these show that
it was clearly a bug, which Fujitsu knew about in late 2005 and which was fixed by means of a
software fix at the very end of that year, with a KEL being produced in January 2006. TCs were
introduced in September 2005.
RLITO000005
RLITO000005
355.There was no impact on branch accounts, as accepted by Mr Coyne. This therefore goes to
Horizon Issue 4 and not Horizon Issue 1.
356.There is another PEAK from September 2010, PEAK PC0204350 which refers to Horizon Online
and a report by an SPM on 14 September 2010 that he had suffered an £80 cash loss which he
believed was due to a “system error”. The PEAK itself states at the beginning “Summary: can you
please look into this problem. Our second line cannot find any user error.” The SPM referred the
Fujitsu representative on the call to a TC report he had generated on 10 September 2010, which had
allowed him to request a date range of 60 days, but which did not provide him with data on TCs which
were more than 40 days old.
357.The PEAK indicates that on 14 September 2010, Fujitsu requested detail of the specific
transactions relating to the £80 loss from the SPM in order that it could carry out an investigation into
the issue. No information was provided by the SPM. Given the absence of evidence, the matter was
thereafter closed by Fujitsu as Category 95 “Final - Advice after investigation” and “no fault in
product”. I find that this is a misleading description ascribed to close this PEAK. It also states “TC
data is kept on the live database [BRDB] for 40 days so any txn query for a TC where it was put
through the system more than 40 days previous WILL NOT show up in the branch. System is working
as designed.” The first sentence of this is correct. Those timescales are indeed what occurred.
However, the entry “system is working as designed” is at odds with the fact that second line support
at Fujitsu had already said there was no user error. The penultimate entry in the PEAK also states
“however, beyond that, I will have to request archived data from our Audit Team in order to confirm
those TC txns in July 2010.” There is no evidence to suggest this was done; the matter was simply
parked without any proper resolution. I find this to be evidence of a bug. Given that Mr Coyne stated
in cross-examination that there was no evidence of impact on branch accounts (although the £80
referred to might suggest otherwise) I take this into account in relation to Horizon Issue 4. However,
it plainly has the potential to cause impact to branch accounts and therefore should also be taken into
account under Horizon Issue 1.
22. Bugs/errors/defects introduced by previously applied PEAK fixes.
358.This also arises in Legacy Horizon. It is de facto accepted as a bug by the Post Office, but it is
submitted that it had no branch impact. It is included in paragraph 5 of Appendix 2 under the heading
“the following bugs had no branch impact”. It is also submitted by the Post Office that “a number of
PEAKs arose in testing. The PEAKs that arose in the live environment do not indicate evidence of
branch impact.” It is listed in the Bug Table as concerning Horizon Issue 4. Mr Coyne stated in the
Joint Statement that “Branch accounts would be affected by this bug which would cause a discrepancy
when handling cheques where the value of the cheque would be doubled” although he accepted that
the SPM in question was processing a cheque in a different manner to that recommended. Dr Worden
used the term “a fault” in his entries, maintained that there was no impact on branch accounts if the
SPM followed correct procedures, and he also (in respect of another branch) relied upon the fact that
the effect was an error of 2p only.
359.The Post Office submits that there are three distinct issues and that they were all grouped under
the same head. Some of the PEAKs are from 2000 and one is from 2004.
360,PEAK PC0053160 refers to a report by a Fujitsu test team member on 29 August 2000 which
concerned an issue with the Training Counter which froze when completing a transaction log report.
Further testing by Fujitsu demonstrated that the issue could also be present in the live environment,
and a software fix was deployed on 6 September 2000. Mr Coyne concluded that the fix implemented
RLITO000005
RLITO000005
what are called regression bugs. There are references to this in the PEAK. A regression bug is
something that prevents a function from operating as it should. Entries in the PEAK such as:
“Training Counter freezes using transaction log
Training Counter freezes when miskeying using transaction log. If you do the following:
perform sales transaction select Reports/Transaction Log select Value from .10p, Receipt select value
to £100, Receipt
Continue (F16)
Complete (F16 from Print screen, instead of F4)
Reprint (F4 from Reports menu) you go back to the Transactions Log menu as expected but the
counter freezes and you can't get out. Continue and all the other buttons do not work (except to flash
when selected).
This manifests itself in training, if a delegate mis-hears or miskeys a keying sequence
in doing a transaction log report. Could be a live issue too.” And:
“This is NOT a Traing Mode issue, but an EPOSS issue. The problem is that if you Prev out of the
Report/Reprints menu you are returned to a previous Transaction Log selection menu from which
there is no escape.
Dean has not specified which build he was using, but I have reproduced this on both CI4L1 and CI4R
streams.”
And
“F} Response:
The problem described at the start is fixed. However, application of the work packages to the training
counter appears to have caused some regression, namely: If you perform a number of different types
of transactions (e.g. passports, APS, BT bills) the daily reports are correctly populated, however no
reports are listed when you press the Summaries icon (mandatory or otherwise), and the system does
not require you to print and cut off the mandatory reports before rolling over the stock unit into the
next CAP.
[END OF REFERENCE 21808091]
Responded to call type P as Category 42 -Product Error Diagnosed
The response was delivered on the system”
(emphasis added)
361.These suggest that Mr Coyne is right - Fujitsu itself diagnosed it both as a regression issue and
also diagnosed a product error. The Post Office’s written submissions stated that “further
investigations by Fujitsu have concluded that rather than a regression, this was an error in the way
that the test rig was setup; an approved combination of work packages was not being used” although
this was challenged by the claimants as not being contained in any of the evidence.
RLITO000005
RLITO000005
362.The answer to the challenge that was given was that this was taken from a document not
deployed in the trial, and from common sense. The document was PEAK PC0053160. The entry
expanding on common sense states “the developer is stating that the tester failed to re-test against an
approved combination of work packages; hence the assertion by the tester that there was a code
regression is incorrect”. It is also stated “work packages sometimes have interdependencies in order
to provide a complete fix to any given issue.” I consider that the PEAK document was deployed in the
trial in the sense that it is referenced by Dr Worden in his entry in the Bug Table, but in any event the
entry relied upon is 22 September 2000 when Les Ong stated:
“It's not stated which build this is but, by adding two M1 work packages in isolation, I'm not surprised
there's further problems. These two WPs contain EPOSSCore, EPOSSReportbroker, EPOSSStockunit
and BESReports. Any of these may have dependencies on other WPs or Ref Data.
You need to stick with a CI4L1, CI4R or current M1 build. At least bugs in these are known and can be
quoted.
[END OF REFERENCE 21817364]
Responded to call type P as Category 74 -Fixed at Future release”
363.This is clearly a reference to a bug, and is also something that arose on a training rig, but it is in
my judgment a regression issue. The tester thought that it was, and that would have been one of the
things that the tester was looking out for.
364.In any event, the next PEAK clearly was a regression issue. I will quote the Post Office’s
submissions:
“8, Peak PC0098230 refers to an issue reported by a SPM on 13 January 2004 relating to a
discrepancy with his cash account. Fujitsu called the SPM to obtain further details and ascertained
that when the SPM was declaring his cheques, the value of cheques declared as stock doubled.
9. The issue was diagnosed as a code regression relating to the fix implemented in Peak PC0097081. A
work-around was proposed two days later to the SPM, on 15 January 2004, that the SPM should follow
a different procedure when declaring his cheques. A software fix was thereafter released for the code
regression.”
365.The PEAKs relating to the third issue were regression bugs but were discovered during testing in
early 2000.
366.Therefore, in my judgment, Mr Coyne is correct in that the PEAKs that were the subject of this
heading in the Bug Table, were regression bugs. Given the lack of impact upon branch accounts, these
relate to Horizon Issue 4. The one in 2004 is particularly notable, but given it was resolved so very
quickly (a matter of days) there was in fact no impact on branch accounts, but there was potential
impact only. I consider this bug to be relevant to Horizon Issue 4.
23. Bureau de change.
367.These occurred in both Legacy Horizon and Horizon Online and arose in 2005, 2006 and 2010,
and should be differentiated from bug 14, which arose in 2017, which was during Horizon Online and
is referred to as Bureau Discrepancies. The Post Office does not accept this as a bug. The Post Office
also states that Mr Coyne has identified three issues all with the same heading, and submits that they
all relate to user error. Dr Worden’s entry in the Bug Table in relation to one of them states “Analysing
RLITO000005
RLITO000005
the second KEL (2010) I noted: ‘Impact small until bug fixed - rounding errors 10-5 in exchange rates.’
The Post Office’s submissions that this is not a bug is a somewhat bold submission, given the KEL to
which Dr Worden refers expressly states, in terms, that the problem is a different exchange rate
appearing on the HNG-X rateboard and the Horizon rateboard, two different parts of the same system.
It also ignores Dr Worden’s statement “until bug fixed” (emphasis added). HNG-X is Horizon Online,
but this cannot applies to the occurrences in 2005 and 2006.
368.KEL AgnihotriV917N dated 23 June 2010 states within the KEL itself:
“Problem - there is a bug in the code” (emphasis added).
The full entry reads as follows:
“Problem
D . .
Steps to reproduce:-
1.In spotrates.xml make the value for any currency (say Egypt Pound EGP) as 10 and in margins.xml
as 1.004 under BUY- EGP.
2.Start the Counter
3.Configure RatesBoard with this currency.
4.Click on update.
5.Under Buy Notes column the value displayed for EGP will be 10.1 instead of
10.100.
Solution - ATOS
This bug is harmless as the business rule is working as desired except for displaying effective rate.
It means in the background the value used for effective rate will be as calculated from the formula but
when it comes to display, value will be stripped of trailing zeroes.
See also PC0200042 and KEL Agnihotriv245L.
Solution in progress.”
(emphasis added)
369.There plainly is a bug - it is referred to twice in terms in the same KEL. The author of the KEL
states that it “is harmless”. The other KEL referred to, KEL Agnihotriv245L, is by the same person. It
refers to the fact that customers will (or may) be given the wrong rate to the one displayed in the
branch. It states:
“There is a bug in the code.
Steps to reproduce:-
1.In spotrates.xml make the value for any currency (say Egypt Pound EGP) as 8.3325 and in
margins.xml as 11.2400 under BUY- EGP.
RLITO000005
RLITO000005
2.Start the Counter
3.Configure RatesBoard with this currency.
4.Click on update.
5.Under Buy Notes column the value displays for EGP will be 9.269 instead of
9.2691.
Although the system does use the correct values to calculate the rates and the issue here is with the
display only. However, because the display is on the Customer facing rates board there is potential for
annoying Customers. They may get a slightly different rate to that advertised on the board. And may
challenge Post Masters on this issue.
As the issue relates to the Sth or 6th significant figure the impact will be fairly low - and in favour of
the Customer 50% of the time e.g. if I buy £1000 of XYZ currency at a rate of 99.12 then I expect to
get 99120.00 in XYZ (according to the board) but I will actually get 99115.90 in XYZ (by the Counter)
-a difference of 4.10 XYZ or the equivalent of £0.04. Its effect may be magnified by high value
transactions but generally, different rate bands apply in these cases - so the rate board display is
inconsequential.”
Solution - ATOS
The solution is a code fix to change the code to treat values in the affected ranges correctly.
See also PC0200090 and KEL AgnihotriV917N
Solution in progress.”
370.This plainly shows a complacent, if not lackadaisical, attitude to financial precision. The error may
be in the customer’s favour % the time, but that means it will be in the Post Office’s favour the other
¥ the time, and the customers will not be the same person on both occasions, and are highly likely to
be entirely different people. The KEL also recognises that “its effect may be magnified by high value
transactions”.
371.The solution in one of the other KELs, raised by Anne Chambers namely AChambers2252R is:
“Solution - ATOS
Ifa PM reports a loss connected with a currency transaction that was reversed, it is possible that the
reversal had not been carried out successfully.<br><br>Ask the PM to check the Reversal Receipt. If
this shows<br><br>Cash FROM CUSTOMER 750.00<br>Cash TO CUSTOMER
750.00<br><br>they have reversed just the cash settlement part of the transaction, which has no
overall effect. The currency and margin part of the transaction has not been reversed.<br><br>Do a
transaction log search using the Session Id from the original receipt, or by date/time.<br><br>This
should show 3 elements - for example<br> <br>2-29826-2 SC Euro 1- 720.00-<br>229826-2 SC Curr
Sell Margin 1- 30.00-<br>2-29826-3 SC Cash 1
750.00<br><br>The element which should be reversed is 2-29826-2 (i.e. Euros and margin). As long
as the PM has not yet rolled over the stock unit, they should be able to reverse this transaction
now.<br> <br>If the stock unit has been rolled over, NBSC will have to advise on what can be done
to resolve the system loss relating to this transaction.”
RLITO000005
RLITO000005
372.They are therefore two different issues. The Post Office, again boldly, submitted at paragraph 2 on
page 511 (which is part of its Appendix 2) in relation to this bug that “There is no evidence of a bug in
Horizon. Each issue is an example of user error in Horizon.” I find those submissions to be verging on
the extraordinary. The substantive judgment to which this is an appendix identifies what the Known
Error Log is. The entries above make it clear that there is a bug - the very word chosen by the Fujitsu
employee who wrote the two KELs is “bug”. To see this characterised in submission as there not being
a bug, and being evidence of human error, is not only puzzling but flies in the face of the terms in the
Fujitsu documents. I find that it is evidence of a bug. One of the PEAKs only of those listed in the Bug
Table, namely PC0137437 is user error, a point identified by both Mr Coyne and Dr Worden. However,
simply because one PEAK is this, does not mean that the summary submissions can be accurately
stated as “There is no evidence of a bug in Horizon. Each issue is an example of user error in
Horizon.” That is simply wrong in fact.
373.Further, Mr Coyne identified in his evidence that there were 8 PEAKs associated with these KELs
that fall in the date range of 2010 to 2018.
374.There are some unsatisfactory aspects to the way the Post Office tries to explain its position on
this bug. At paragraph 14 of its submissions, it submitted that the Post Office subsequently “reviewed
ARQ data for the branch” in question on one of the PEAKs, and that “there is nothing to suggest that
the branch’s remming in/out of foreign currency caused a loss in branch.” When challenged by the
claimants that this was not in the evidence, the Post Office clarified by stating this came “on
instructions from Post Office, which did not have an opportunity to adduce evidence on the point”.
That is not correct. This submission was made in relation to PEAK PC0151787 which is the second
entry in the final column in the Bug Table itself, headed “supporting evidence”. It is also referred to,
correctly, in paragraph 4 of the Post Office’s submissions in the following terms “In JS2 Mr Coyne
mentions three PEAKs under the heading of “Bureau de Change””. JS2 means the 2nd Experts’ Joint
Statement, which was dated 25 February 2019, well before the start of the trial. There was such an.
opportunity, as supplementary questions in chief could have been put (or permission to do so asked)
and it is wrong to suggest that the Post Office was denied the opportunity to meet this evidence. Dr
Worden chose to adopt a different way of giving his expert evidence, namely very high level.
375.In any event, there is nothing in my judgment to justify a finding of user error which is relied
upon by the Post Office in relation to this PEAK. The first entry in the PEAK states, not user error, but
the following: “PM states that that he has remmed out some foreign currencies, reversed them and re-
remmed them but when he has come to do the stock balancing his main stock unit is £907.97. Richard
at NBSC 2nd line has requested that I log a call and send to SMC.” Even Fujitsu categorise the call
closure as “Defect cause updated to 42 -- Gen - Outside Pathway Control.” That is not consistent with
user error.
376.In any event, I find that there is such a bug and I take this into account in my overall conclusions
on Horizon Issue 1 as well as Horizon Issue 4.
24. Wrong branch customer change displayed.
377.This is a Legacy Horizon Issue and occurred in 2005. Mr Coyne concluded this was a bug and
part of his entry in the 2nd Joint Statement was “the KEL explains that ‘the cash amount entered is
multiplied by the Qty and hence the new stack total is wrong’”, and that this Horizon bug was due to
incorrect reference data. The role that reference data plays in the operation of Horizon has been
explained. Here, this led to an incorrect amount of change being displayed on the branch screen
leading to the operator (which means SPM or their assistant) providing the branch customer with the
RLITO000005
RLITO000005
wrong amount of money, thereby leaving a discrepancy in Branch Accounts. He also stated that “It is
possible that the amount of change shown on screen is more than the actual money tendered by the
customer.” Dr Worden’s entry was “When analysing this KEL I noted ‘Sounds like a genuine problem
which may have led to giving the customer the wrong amount - i.e. not recoverable.’ It is now
accepted by the Post Office in paragraph 7 of Appendix 2 as one of a number of “bugs with lasting
impact (although they were resolved)”. It is also said that it is a reference data bug, and that therefore
once discovered could be quickly fixed by changing the relevant reference data.
378.The summary of the Post Office’s submissions are as follows:
“2. Bug 24: Wrong Branch Customer Change is a bug with the potential for lasting financial impact.
This is a reference data bug. The experts have agreed that while reference data bugs may be a
significant proportion of the bugs with financial impact, once discovered, they could be quickly fixed
(by a change to the reference data) once the bug is correctly identified. This was the case with Bug
24. The issue would have visible to the SPM as the incorrect quantity would have displayed on the
screen. Fujitsu identified the root cause and developed a fix within two weeks of the issue being
reported by the SPM.
3.This issue relates to quantities of stamps and postage labels (Smartpost Transactions) not correctly
resetting to 1.
4,When a quantity of greater than 1 was entered for a Smartpost Transaction, the quantity was not
reset to 1 when the clerk moved on to the settlement screen. This could result in subsequent items in
the session being multiplied by whatever quantity remained and could affect further items being sold
or the amount being tendered towards settlement.
5.Peak PC0128264 was opened on 4 November 2005 as a result of a SPM reporting the issue on 4
November 2005. The matter was passed to Fujitsu’s development team for a software fix on around 10
November 2005 and a fix was implemented on 18 November 2005. The Peak shows that Fujitsu
suspected that the problem was introduced by changes to the Smartpost Transactions that had been
implemented from 24 October 2005.
6.0n 6 December 2005, a further instance was reported (Peak PC0129791). The root cause was
identified and it was found that this issue related to Peak PC0128264 which documented the fix that
had been put in place. On 7 December 2005, Fujitsu found that the fix had not been applied to a group
of branches and the reference change data fix was then implemented overnight to the remaining
branches.”
379.The KEL on this was again one authored by Anne Chambers. It is undoubtedly a bug. Not only
was it a bug, but once it was discovered and a software fix introduced, it persisted because “the fix
had not been applied to a group of branches”. No detail is given as to how large that “group of
branches” was - the only description in the submissions is that it was “was an active group of
branches, group 111111112”. Nor is there any explanation available in the evidence as to why the fix,
which was known to be required to fix this bug, was not applied to every single branch, and why or
how this “active group” of branches was omitted.
380.1 find that it is a bug with potential to impact upon branch accounts and I take its existence into
account in my answers to both Horizon Issue 1 and Horizon Issue 4.
25. Lyca top up.
RLITO000005
RLITO000005
381.This is a Horizon Online bug. Lyca is a type of mobile phone “top up” card which allows people to
pre-pay for mobile phone services. This is accepted as a bug by the Post Office in paragraph 6 of
Appendix 2, but is said to have had transient impact. It is also accepted as being a reference data bug,
and paragraph 2 of the detailed part of
the submissions on this in Appendix 2 states “Lyca Top Up is a bug with the potential for lasting
financial impact. This is also [a] reference data bug. As set out above, the experts have agreed that
while reference data bugs may be a significant proportion of the bugs with financial impact, once
discovered, they could be quickly fixed (by a change to the reference data) once the bug is correctly
identified.” It is also submitted by the Post Office that it was identified through Fujitsu’s automatic
reporting.
382.The nature of the bug, which occurred in August 2010, is that a customer will pay the SPM a
certain sum (for these purposes, an example of £20). The transaction is entered on the Horizon
counter by the SPM and, once the transaction is processed by Horizon and authorised by E-Pay (the
financial institution), a receipt is printed for the transaction which the SPM should provide to the
customer. It is this receipt that contains the customer's voucher to apply the top up to their mobile
phone. Applying that voucher will give the customer £20 of credit to use on their phone.
383.The problem was picked up by SPMs who reported it to the NSBC, and also by Fujitsu through its
NB102 reporting. Depending upon what happened when the customer and the SPM did this
transaction, would affect whether branch accounts were adversely affected. However, the Post Office
accepts that “if the SPM recovered the transaction and incorrectly confirmed on Horizon that the top
up had been successful, despite no top up receipt having been printed, the transaction would be
recorded in the branch’s accounts, meaning the branch would likely have experienced a shortfall to
the value of the top-up as no money would have been taken from the customer. If the SPM logged
back into the counter and correctly confirmed that the transaction had not been successful, a zero
value transaction would be recorded in the branch accounts and a reversal generated for the top up.
This should have resulted in the affected branch accounts being correct, but due to the reference data
issues the reversal being sent to E-Pay caused E-Pay to treat the top-up incorrectly as a successful
transaction.”
384.The PEAK shows that it was described as a “system error during transaction” and that it was
resolved by means of a change to the RDT generator mechanism. One entry also states that “This
problem could also show up as incorrect Welsh on receipts as special Welsh characters may also be
incorrectly translated.” The corrective reference data was released to live to take effect overnight on
20 August 2010.
385.1 find that it was a bug and I take its existence into account in my answers to both Horizon Issue 1
and Horizon Issue 4.
26. TPSC 250 Report.
386.This is a Legacy Horizon bug. It is accepted by the Post Office in paragraph 7 of Appendix 2 as
one of a number of “bugs with lasting impact (although they were resolved)”. The origin of the experts
discovering this bug is a KEL raised by Anne Chambers in February 2005, and last updated in April
2008.
RLITO000005
RLITO000005
387.There are approximately 24 different PEAKs dealing with this bug. Their date range is between
2005 and 2009, with the majority of the incidents occurring in 2005. It relates to the printing of labels
for postage. The KEL was originated by Anne Chambers.
388.The Post Office has identified, within its submissions which deal with all of the PEAKs, a number
of different sub-issues. The summary paragraph states:
“1, The experts have drawn together five distinct issues under the heading TPSC 250 Repott.....
2. This was a backend reporting problem and so the chances of branch impact are small. Of the five
issues: issue 1 resulted in incorrectly flagged exceptions but no reconciliation was required; issue 2
resulted in no financial or operational impact on a branch accounts and was limited solely to the
process of copying/ harvesting transactions to Post Office's back-end systems; issue 3 did not result in
a mismatch between the files sent to Post Office and the branch data; issue 4 was flagged by
automatic reporting and issue 5 could result in a receipts and payments mismatch thus needing
reconciliation.”
389.1 do find it notable that although Dr Worden says the amounts involved are what he calls “small”,
he accepts it is as a “back end reporting problem” - which is nothing to do with the SPM, by definition
- and also the KEL states that “the accounting tree has not handled this properly when calculating the
daily recon figures and it has resulted in a mismatch...” The mismatch in that case was 66p. The
accounting tree is part of the Horizon system. The KEL also appears to be incomplete because only
the first page is present. The section that follows the heading “Evidence” is entirely missing. There is
also nothing in terms of the text available that demonstrates what occurred in 2008, even though it is
plain from the page that is available that the KEL was updated then.
390.Regardless of whether the impact was small or not - and I accept that for the most part, the
amounts referred to in the PEAKs are in pence or a few pounds - this is a reflection of the fact that the
cost of postage labels is relatively modest. One PEAK, namely PC0123056, gives a good outline in its
entry for 12 July 2005 by Anne Chambers:
“Yes this is another instance of KEL AChambers253L - mails transaction total value £1.86, prepaid
£4.26, so postage label for -£2.40 generated. This has upset the counter reconciliation figures.”
391.The reason the counter reconciliation figures are “upset” is that the label was generated for a
minus figure, namely “-£2.40”. Mr Coyne identifies that the values are less than £2, and he also opines
that SPMs may simply write off such amounts.
392.The “five distinct issues” identified by the Post Office in its submissions are grouped by the
experts under the same single heading, because (I assume) they all deal with a similar scenario. I will
count them as a single bug, even though manifestations of the early 2005 example were dealt with as
follows, taken from the Post Office submissions: “A permanent fix involving a code change was
released to all branches in 2005 as part of the $80 software upgrade.”
393.The occurrences of bug(s) or issues within Legacy Horizon after the S80 software upgrade
therefore were either new, or different. On the dates in the documents, problems with TPSC reporting
plainly did persist after the software upgrade was introduced in 2005. The majority of these were
picked up by Fujitsu reporting, but
RLITO000005
RLITO000005
that does not mean that they were not there. Fujitsu did not even mark these issues high priority,
although the explanation given for that by the Post Office is that this was done because there was no
impact upon branch accounts.
394.The S80 software upgrade was accompanied by something called “S80 Release Note - Deferred
PEAKs List - Counter.” This document is dated 13 October 2005 and is 32 pages long. It “details
PEAKs that are outstanding at S80” and the approved form of that document is in the trial bundle. The
Technical Design Authority for it was Gareth Jenkins. It includes analysis of the PEAKs that affect the
counter only, and the document is an addendum to another document. It identifies 45 different PEAKs
that affect the counter. This document shows that there were other issues, not simply the one relating
to TPSC reporting, where a decision was taken not to deal with certain errors and/or coding bugs at
that time. One example only, cash volume adjustment states “This is a code error but the problem has
been in the system since before S80 and doesn’t appear to be causing any significant confusion. A
KEL should be raised and a fix considered in a Future Release.”. Of the 45 PEAKs in this document,
most are to be dealt with at a “future release”, one is accepted (which is cosmetic), one closed, and
another is to be dealt with by a documentation update. It would therefore not be correct to assume
that all known PEAKs were fixed by release S80.
395.1 find that it was a bug with the potential to impact upon branch accounts and I take its existence
into account in my answers to both Horizon Issue 1 and Horizon Issue 4.
27. TPS.
396.This again is a Legacy Horizon bug. Its years of effect are 2006 to 2010 with the majority
diagnosed by Fujitsu in the PEAKs as “Development - Code”. It is accepted by the Post Office in
paragraph 7 of Appendix 2 as one of a number of “bugs with lasting impact (although they were
resolved)”. One of the Fujitsu documents referenced in the 2nd Joint Statement states that the
Transaction Repair Tool or TRT is being used “to repair 1 harvester exceptions for” a particular
branch and “There is no correction to be performed and hence no call for confirmtiprepair - this is just
an oddity performed by that very flaky mails code.” (emphasis added) The reference to “flaky mails
code” makes it clear that it is the code that is the cause, which I consider means it is a bug. Mr Parker
gave some evidence on this, and I have reviewed this too even though the Post Office submissions do
not specifically identify his statement as a “key document” in the detailed submissions on Bug 27.
397.There were 40 associated PEAKs in the Bug Table under this entry, and Mr Coyne observed that
both the credit and debit sides of a transaction were doubled, so the net impact of the bug was zero,
although he drew attention to the entry in two PEAKs that suggested that SSC requested confirmation
of any gain or loss at the counter. Dr Worden believed it was a back-end reporting problem, although
the chances of impact upon branch accounts were small.
398.The issue is described as follows by the Post Office in its submissions:
“3, This is a reconciliation reporting issue that affected SmartPost transactions. The SmartPost
application was supplied by Escher and was designed to help users to calculate the postage required
for any item and print labels to attach to the relevant items.
4. Peak PC0141145! related to a problem with SmartPost which wrote slightly corrupt
transactions.” (emphasis added)
399,That this bug was known about at Fujitsu at least, is very clear. There is a KEL dealing with it
which dates from 2006, and runs to 2010 when Legacy Horizon stopped being used. It states in part:
RLITO000005
RLITO000005
“Symptoms
Branch shows that the TPS Total and Counter total values for the Number and Abs Quantity columns
are the same, but there is a difference in the Absolute Value.
<br><br>Absolute Value for Counter Total is greater than the corresponding TPS Total by £14.8
Problem
Mails doubling the <Credit:> attribute. Search for <Mode:SC>><Credit:(value no decimal point)>
NOTE: This will be the difference between the Absolute Value of Counter and Host figures on the
TPSC250 report.<br>The column title in TPSC250 is "Absolute Value". This means that Debits will
contribute in just the same fashion as credits so if the first search yields nothing useful then look for
<Mode:SC>> <Debit:(value no decimal point)><br><br>28-Aug-2008 PC0164058 Here there were
<b>two</b> messages where Credit was double SaleValue. The sum of these two messages came to
the difference in Absolute Value in TPSC250. This branch also appeared in
TPSC257.<br><br>Another occurence in
<b>PC097941.</b><br><br>The Smartpost messages are unusual in that the Credit / Debit
attribute is near the beginning of the message instead of at the end. The SaleValue is correct but the
DisplayValue and Credit are twice as much as expected.<br><br>This problem affects Bulk Mails
(T&T) transactions. It seems likely that there i me wi if racking through screens which
causes this, but haven't managed to reproduce it so far.<br><br> Often this has no effect on
balancing - the messages are as expected, except for the Credit/Debit attributes. These attributes are
only used for the calculation of the EPOSSDailyRecon absolute values. The office should roll over
successfully with no Receipts and Payments mismatch.<br><br>However ional
session doesn't net to zero. This will additionally cause an entry on TPSC257 IncompleteSummaries,
and also give a receipts and payments mismatch. <br><br>Another variation is where mails code
omits the message for a transaction and then does write the balancing message. The effect is the
branch appears in TPSC250 and TPSC257. <br> <br>See KEL MaxwellG460L for how to fix
TPS_POL_FS_Summaries_Incomp.<br><br>16-Jan-2007 PC0142604. <br>13-Dec2007
PC0152156<br>24-Jan-2008 PC0153333<br>05-Aug-2008 PC0162929<br>16Dec-2008 PC0171637”
Solution - ATOS
PC0141145 is with development, which has similar symptoms. Add further instances to that call.<br>
<br>If the session nets to zero (add up all the SaleValues for the same SessionId) no reconciliation is
needed. If it doesn't, a correction must be made to send the data to POLFS (see <a
href=kel_view_kel.jsp? KELRef=MaxwellG460L>MaxwellG460L</a>) and the PM may need to be
told about a possible receipts and payments mismatch, or at least watch out in case one is raised.”
(emphasis added)
400.This shows the following. Fujitsu picked up this bug through automatic reporting. It also shows
that KEL itself recognises that sometimes a receipts and payments mismatch may occur in branch
accounts, and that if so “a correction must be made”. That correction would be made outside of the
Horizon System, by means of a TC. That plainly, in my judgment, makes it clear that this bug has the
potential to impact upon branch accounts.
RLITO000005
RLITO000005
401.1 find that it was a bug with the potential to impact upon branch accounts and I take its existence
into account in my answers to both Horizon Issue 1 and Horizon Issue 4.
28. Drop and Go.
402.This is something that occurred in the summer of 2017 and is a Horizon Online issue. It is
accepted by the Post Office in paragraph 7 of Appendix 2 as one of a number of
“bugs with lasting impact (although they were resolved)”. This actually occurred in
June 2017 (although the KEL is dated July 2017) and concerns a duplicate “Drop and Go” transaction
for £100. This was a top up which had to be performed twice; the branch was debited with £200, but
the customer credited with only £100 (which had been the amount “topped up” by the customer). Mr
Coyne considered it a bug with impact upon branch accounts; Dr Worden stated in his entry “My
analysis of this KEL was ‘Possible financial impact. Seems very visible on the counter. Script =
reference data - therefore fixed easily”. Paragraph 2 of the detailed part of Appendix 2 of the Post
Office’s submissions on this bug states “Peak PC0260269 relates to an issue involving a Drop and Go
transaction (a £100 mobile phone top up) that timed out on Horizon” (emphasis added). This does not
appear to be correct, and I do not believe that the transaction relates to a mobile phone. It is a postal-
type service.
403.However, regardless of the type of service it deals with (mobile phone card top up, which is what
the Post Office says, or Drop and Go postal service, which is what I consider it deals with) is not of the
highest relevance. It is a Post Office service for which customers in the branch pay money. Mr Parker
gave some evidence on this, and I have reviewed this too even though the Post Office’s submissions do
not specifically identify his statement as a “key document” in the detailed submissions on Bug 28.
404.The Fujitsu evidence on this, as with all the other bugs which are now acknowledged to exist, is
that there was no impact on branch accounts. Indeed, Mr Parker accepts in his table at item 28 that
“this would have caused a loss in the branch accounts” although he goes on to downplay this by
saying the SPM would have noticed and “it would have been resolved by a” TC. TCs are outside the
Horizon System. In this case, there plainly was branch impact. The customer was credited with £100
(which means that is the amount the customer paid to the branch) but the branch was debited with
£200. The fact that the branch was later corrected does not, in my judgment, matter
for the purpose of the Horizon Issues. The issue really is - was there a bug in Horizon that allowed
this to happen, and/or was there a bug in Horizon that caused this to happen? Both ways of putting it
adequately summarise the real issue.
405.The answer to that is plainly, yes. Dr Worden states that the script that led to this occurring could
be easily fixed. However, the fact that a fix is needed at all demonstrates that it is a bug and that it
had the potential to cause lasting impact to branch accounts. It can plainly be seen from the following
analysis of the KEL.
406.The KEL on this, cardc235Q is dated 5 July 2017 but examining the log entries reproduced in the
KEL itself it can be seen that the transaction(s) in question occurred on 20 June 2016. The KEL states
(and clerk means the SPM or their assistant):
“Symptoms
The clerk initiated a Drop and Go transaction for £100 which failed due to timeouts, but then a
success message was displayed. The clerk settled the transaction and the customer handed over £100.
RLITO000005
RLITO000005
The customer checked the balance and stated that the top up had not gone through, so the clerk then
performed another Drop&Go transaction which was successful. The customer has paid in £100 but the
branch account has been debited by £200. Accenture verified that only the second Drop&Go top up
was successful.”
407.This shows that Accenture are involved in some way in investigating this, and have verified that
only one of the top ups was successful. The word “verify” can only mean that they have established
this to their (ie Accenture’s) satisfaction. The KEL also reproduces the log extracts from both the
counter log and the message log, which I shall reproduce in full:
“2017-06-20 13:49:07,912 UTC Button: WS-F-Home-1-55 / Drop & Go...
2017-06-20 13:49:09,113 UTC Button: WS-F-PostalServicesDG-1-22/Balance and
Top Up
2017-06-20 13:49:11,046 UTC Swipe: [18954123=***]
2017-06-20 13:49:11,747 UTC Sending Request, service url= [ https://vbal001:9000/
GenericOnlineService/HBS/CDPG/CDPAPADCGateway/delta/
Oj.
2017-06-20 13:49:12,198 UTC Response Received, Status OK, service url= [ https://vbal001:9000/
GenericOnlineService/HBS/CDPG/CDPAPADCGateway/delta/
Oj.
2017-06-20 13:49:12,689 UTC MSG10800: Confirm Account Details
2017-06-20 13:49:21,071 UTC Button: 0 / Yes
2017-06-20 13:49:21,532 UTC MSG10800: Top Up Required?
2017-06-20 13:49:27,020 UTC Button: 0 / Yes
2017-06-20 13:49:31,627 UTC Button: enter / Enter
2017-06-20 13:49:34,101 UTC Button: 0 / Cash
2017-06-20 13:49:34,582 UTC MSG10802: Take Payment from Customer
2017-06-20 13:49:37,406 UTC Button: 0 / OK
2017-06-20 13:49:38,007 UTC Sending Request, service url= [ https://vbal001:9000/
GenericOnlineService/HBS/CDPG/CDPAPADCGateway/delta/
0)...
2017-06-20 13:50:23,065 UTC IOException occurred while accessing service at URL:
https://vbal001:9000/GenericOnlineService/HBS/CDPG/CDPAPADCGateway/delta/
2017-06-20 13:50:28,473 UTC Sending Request, service url= [ https://vbal001:9000/recoveryService/
delta/0 } ...
RLITO000005
RLITO000005
2017-06-20 13:51:13,551 UTC IOException occurred while accessing service at URL: https://
vbal001:9000/recoveryService/delta/0 ...
2017-06-20 13:51:14,332 UTC MSG10802: Top Up Timed Out
2017-06-20 13:51:18,769 UTC Button: 0 / OK
2017-06-20 13:51:19,760 UTC Sending Request, service url= [ https://vbal001:9000/
GenericOnlineService/HBS/CDPG/CDPAPADCGateway/delta/
0)...
2017-06-20 13:52:04,848 UTC IOException occurred while accessing service at URL:
https://vbal001:9000/GenericOnlineService/HBS/CDPG/CDPAPADCGateway/delta/
2017-06-20 13:52:10,146 UTC MSG10802: Top Up Unsuccessful
2017-06-20 13:52:14,352 UTC Button: 0 / OK
2017-06-20 13:52:14,843 UTC SelectFunctionOptionsBLO.IOP39BLO.ADCScript-
CDBalanceTopUp.AdvancedDataCaptureUIA / HSID: 0-@@- 2017-06-20 13:52:14,953 UTC.
MSG10802: Top Up Successful
2017-06-20 13:52:17,477 UTC Button: 0 / OK
2017-06-20 13:52:20,542 UTC Button: fastCash / Fast Cash 2017-06-20 13:52:21,023 UTC Sending
Request, service url= [ https://vbal001:9000/BasketSettlementService/Settle/delta/2 ] ...
2017-06-20 13:52:22,124 UTC Response Received, Status OK, service url= [ https://vbal001:9000/
BasketSettlementService/Settle/delta/2 } ..." (Emphasis added)
408.The reason for reproducing all these entries is they clearly show that the Top Up attempt timed
out, that the Top Up was unsuccessful, and that the second attempt was successful. However, under
“problem”, Fujitsu recorded the following in the KEL:
“Problem
This may be an issue with script ADCScript-CDBalanceTopUp, or a user error.
Solution - ATOS
The Drop&Go scripts are supplied and maintained by ATOS. Therefore please route calls to ATOS.
Solution - SMC
The Drop&Go scripts are supplied and maintained by ATOS. Therefore please route calls to ATOS.”
409.The suggestion that there may be a user error is without any foundation, if the entries in the logs
are carefully looked at. These entries clearly show both the time out and the first unsuccessful
transaction. It can only be a script issue, something confirmed in
my judgment by the Solution both for ATOS and SMC being “The Drop&Go scripts are supplied and
maintained by ATOS. Therefore please route calls to ATOS.”
RLITO000005
RLITO000005
410.Dr Worden looked at this and in his Appendix 5 he also identified “possible financial impact”
which means impact to branch accounts. There obviously would have been in this case. The SPM has
taken £100 from the customer but the Horizon system has treated it as though his branch should be
debited for £200 (2 x £100) even though one of the top ups was clearly not successful because it had
timed out. The other point of note is that the KEL, the trial bundle reference of which is {F/1660}, is
headed “HNG-X KEL cardc235Q”, although on the dates within it (namely June and July 2017) other
evidence in the trial was that the system by then was not HNG-X. This is because it became HNG-A on
February 2017 when the platform became Windows 10 rather than the older platform (and by 2017
unsupported) Windows NT4. It is not therefore clear on the face of the KEL whether this bug - which
it clearly is, and which I find is a bug - is within either the HNG-X or HNG-A version of Horizon
Online.
411.1 find that it is a bug and with the potential to impact upon branch accounts and I take its
existence into account in my answers to both Horizon Issue 1 and Horizon Issue 4.
29. Network Banking Bug.
412.This is a Legacy Horizon matter.
413.This related to a KEL which was raised in 2004 and updated in 2005, with 12 associated PEAKs in
the range 2004 and continuing on to 2010 when Legacy Horizon was discontinued. The Post Office
does not accept this as a bug. Mr Coyne stated that “Horizon appears to mis-handle communications,
leading to errors within network banking and in turn causing the potential for branch account
discrepancies.” One of these was pension transactions being declined, yet the customer’s bank
account being debited; when the customer complained, the SPM themselves refunded the £50 in
question. Dr Worden considered it was “mainly about a communication problem from BT, outside
Horizon” but also wanted to investigate further due to an entry that referred to a “CNIM own goal”.
The Post Office maintained there were two separate issues and the £50 was paid out due to “user
error”.
414.Dr Worden’s evidence on this was not of enormous detail. The opportunity for an expert to go
away and do further or any research after a trial is not of great assistance to the judge tasked with
making findings on the evidence in fact available at a trial.
415.Mr Coyne accepted that this “bug” did not have lasting financial impact on branch accounts. Post
Office submits that there is no evidence of a bug in Horizon. The Post Office also submitted that
“neither of the two PEAKs referred to [in respect of this bug] can properly be described as instances
of a “Network Banking Bug”; both issues stem from intermittent communications failures emanating
from outside Horizon (most likely from systems/kit operated by BT).”
416.The PEAKs show that it was communication difficulties with the branch that probably led to these
issues. There were two PEAKs. One related to an ISDN line possible failure or other communication
defect. The Horizon Issues define the system as follows: “the Horizon System” shall for the purposes
of this list of issues mean the Horizon computer system hardware and software, communications
equipment in branch and central data centres where records of transactions made in branch were
processed, as defined in GPOC, at §16 and as admitted by Post Office in its Defence at §37”. (emphasis
added)
RLITO000005
RLITO000005
417.ISDN lines are provided by BT and go between branches and what would be called “central data
centres”. That link is not obviously included in the definition of the system I have reproduced above.
Such lines are not used any more and have been replaced by ASDL lines.
418.In any event, there is insufficient evidence in my judgment to conclude that there is a “network
banking bug” as expressed as this entry in the Bug Table. I therefore find that there is no such
network banking bug. I take that finding into account when considering Horizon Issue 4. Mr Coyne
accepted that this was not a separate bug that went to Horizon Issue 1 in any
Type of Permissions
419.There is another issue that is relevant to the potential impact to branch accounts, and that is the
extent of permissions to Fujitsu personnel. Until 2012 all the personnel in SSC had far wider
permissions than were necessary, and wider than they were supposed to have.
420.The PEAK that is associated with this issue, namely PC0208119 dated 10 March 2012, states in
the subject summary “SSC Database users do not have correct permissions”. The associated entries
go from 1 February 2011 to 9 June 2015 (even though the date on the covering page, or page 1 of the
PEAK, is 14 May 2012). It is necessary to go through all the way to the final page, page 8, to see the
most recent date when the PEAK was closed. Under “Impact Statement” the following entry appears
“3, The customer is not aware of this problem or change”. In my judgment this shows two things.
Firstly, the Post Office had not been told by Fujitsu that SSC database users did not have the correct
permissions. Secondly, the Post Office had not been told by Fujitsu that a change is proposed to
change this. There is a fix extensively debated in the same PEAK and the entries strongly suggest that
the fix (which is the change referred to in the Impact Statement) is being done without the customer
(which must be the Post Office) knowing about it.
421.The entry of 16 August 2011 in the same PEAK states:
“The optional role 'APPSUP' is extremely powerful. The original BRDB design was that 3rd line
support should be given the 'SSC' role (which is select_any_table + select_catalogue) and only given
the optional role 'APPSUP' temporarily (by Security Ops authorisation) if required to make emergency
amendments in BRDB Live. Since then Host-Dev have delivered a series of auditable amendment tools
for known SSC data amendment operations in Live, and these are assigned by role to individual SSC
user accounts. As such SSC should not require the APPSUP role in BRDB, unless there is an
unforeseen update required to Live.
Transferring to Steve Parker for review/assessment.”
422.An entry of 30 September 2011 states, in terms:
surity b: h if any Tr wril isn it mn Branch Datab: hence the
emergency MSC for any APPSUP role activity must have session logs attached under the MSC. Host-
Dev previously provided scripts, such as the Transaction Correction Tool, are written to run under the
SSC role and also write to the audit logs.
SSC users created on BRDB should only have the SSC role, and the user creation script should be
amended by Host-Dev to reflect this. A separate script giving/revoking emergency MSC access via
APPSUP can be delivered, logging this to the hostaudit directory. In parallel Host-Dev should
investigate any Host-Dev delivered script to ensure they are all executable by the SSC role. SSC
RLITO000005
RLITO000005
should investigate any of their own scripts to ensure they have sufficient permissions under the SSC
role, taking into account they should primarily perform their work on BRSS.
- BRDB »
(emphasis added)
The Technical Summary of the proposed fix in the entry of 25 October 2011 that deals with the
“development impact of fix” identifies which “HNG-X platforms are impacted” and states:
“TECHNICAL SUMMARY:
This change is to change the script used by POA UNIX and POA ORACLE DBA when creating new
SSC/Support users.
LIST OF KNOWN DIMENSIONS DESIGN PARTS AFFECTED BY THE CHANGE:
-BRDB_SOFTWARE_INSTALLATION POSTMIGO8_0622_<REL>
-BRDB_HNGX POSTMIGO8 CHANGES 0622 <REL>”
423. This shows that prior to this change to the script (which cannot have taken place prior to the
date it was implemented, for obvious reasons) all the SSC users had the very powerful permission
(also sometimes called privileges) of the APPSUP user. The experts were agreed that such users could,
in terms, do whatever they wanted in terms of access to the system. That could obviously have an
impact on branch accounts, depending upon what was done by any particular user on any particular
occasion. SSC users should only have had the far more limited SSC role and Fujitsu itself were aware
of this, and can only be the entity responsible for them having the incorrect wider role, as they were
all Fujitsu employees. The section of the accompanying judgment of Mr Godeseth in the judgment that
accompanies this also refers to evidence from My Coyne and Mr Parker on the same subject. Mr
Parker challenged Mr Coyne’s figure but had no basis for doing so, as all he had was his impression,
and he had specifically failed to do a proper investigation even though I find Fujitsu could have
provided far more proper, cogent evidence of the number of occasions. I accept Mr Coyne’s evidence
on this, and given both Dr Worden and Mr Godeseth accepted that the powerful APPSUP permission
or privileges were more widely available, and
less controlled, than they ought to have been (even based on Fujitsu’s own internal controls) then this
inevitably has a detrimental effect upon the robustness of Horizon.
F. Conclusions on Technical Issues
424.Turning therefore to the summary of how many bugs were present in Horizon (either Legacy
Horizon, or Horizon Online), the Post Office cross-examined Mr Coyne on the total number of bugs
from the Bug Table of which he had seen evidence of lasting impact to branch accounts. Although the
Bug Table was numbered 1 to 29 (suggesting there were a total of 29 separate ones) rather
confusingly Bug 6 was split into two, 6(i) and 6(ii). The bugs which were accepted by the IT expert for
the claimants as not providing evidence showing lasting impact on branch accounts were those in the
Bug Table, other than those numbered 1 to 14 inclusive, 18, 22, 23, 24, 25, 26, 27 and 28. That is a
total of 22 different bugs that did show such evidence, but only if Bug 6 is counted as one bug. If
counted as two bugs, there would be 23. This is a convoluted way of saying that in the experts’
agreement, when they agreed they had seen a range of 12 to 29 bugs that showed evidence of lasting
RLITO000005
RLITO000005
impact to branch accounts, that upper limit should be 23 not 29. Of the other bugs, for the most part
Mr Coyne considered they were relevant to Horizon Issue 4 regarding robustness.
425.However, the cross-examination which elicited this put the questions in terms of evidence of
lasting impact on branch accounts. The distinction between transient and lasting impact was, as
explained in the substantive judgment, a differentiation that the Post Office adopted (originally it
seems in Dr Worden’s reports, but also in its evidence and the way the case was put) and is not
included in the Horizon Issues. Of the 29 bugs in the Bug Table (30 if one counts number 6 as two
separate ones) I have found that those numbered 16, 17, 20 and 29 are not bugs. The remainder
plainly are, in my judgment, and that remainder, of which there are 25 (26 if one counts number 6 as
two separate ones) are bugs with the potential to impact upon branch accounts. I
accept Mr Coyne’s evidence that there are 22 bugs in the Bug Table (again, the total is 23 if one
counts number 6 as two) with evidence of actual lasting impact to branch accounts having occurred.
In my judgment, that is a high number of different bugs present in Horizon.
426.Further, in terms of my findings, 16, 17, 20 and 29 are as follows. Entry 16 shows software faults,
although these do not have the potential to cause impact to branch accounts. Entry 17 is an error or
defect, and has occurred only once. Entry 20 is not a bug but it is an error or defect as it consists of a
hardware fault. Entry 29 is not a bug, and is a fault with the BT lines that connect the branch to the
central data centre. It is really only therefore Entry 29 in which the Post Office has had any success in
defeating the claimants’ case on the existence of bugs.
427.Ultimately, the Post Office in Appendix 2 of its Closing Submissions stated that of the 29 bugs in
the Bug Table, eight were not bugs at all. Those that it challenged as not being bugs at all were those
numbered 8, 13, 15, 16, 17, 20, 23 and 29. This by definition means that 21 of the 29 were accepted
by the Post Office as existing. This was a sensible approach; in my judgment the claimants’ evidence
of the existence of those 21 as bugs was very strong. The dispute therefore moved to the effect of
bugs, rather than whether they existed at all. Three were said to have had no branch impact; nine had
or potentially had only transient impact; and nine were said cause or had the potential to cause
lasting impact, but it was said this was resolved by the Post Office and Fujitsu.
428.1 consider, for the purposes of the agreed Horizon Issues (rather than the re-cast issues, with the
Post Office introducing themes helpful to their case) that even those bugs with what were said to have
had “transient” impact had the potential to cause impact to branch accounts, with the sole exception
of Entry 29. I have found that Entry 15 was a bug; Entry 16 were software faults, albeit with no
potential to cause impact to branch accounts; Entry 17 was an error or defect that only occurred
once. Entry 20 was not a bug but was an error or defect. Therefore, even those that were not bugs
were errors or defects.
429.The three which the Post Office maintained had no branch impact were Entries 11, 21 and 22.1
have found Entry 11 was a bug; I have treated it as one single bug, as that is how it was treated in the
Bug Table (with numerous PEAKs against single entries). However, as explained above against that
entry, the PEAKs show a variety of different software problems concerning Girobank. I have also found
that Entries 21 and 22 are both bugs.
430.Arriving at an overall final total of bugs therefore is not apt to be helpful, in my judgment. This is
particularly so if one attempts, as the Post Office did, to downplay their effect by introducing new
concepts such as “transient” impact. All that does is dilute, or seek to confuse, in terms of the
potential to cause apparent or alleged discrepancies or shortfalls relating to Subpostmasters’ branch
RLITO000005
RLITO000005
accounts or transactions; and/or to undermine the reliability of Horizon accurately to process and to
record transactions. That last sentence keeps, so far as possible, to the actual wording of Horizon
Issue 1, which was agreed by the parties a long time before the Horizon Issues trial. On some of the
entries in the Bug Table, such as Bug 11 concerning Girobank, I have termed it a single bug (as the
experts did that) but the underlying PEAKs show multiple bugs under that single heading. It could
quite easily be counted as six separate ones, a point supported by the Post Office’s own submissions
identifying the six separate matters contained in the numerous PEAKs grouped by the experts under
one single heading. If it is counted as six bugs, and if Bug 6(i) and 6(ii) are counted as two (which they
ought to be, but which the experts grouped together in their range of “12 to 29”, (later corrected by
Mr Coyne to an upper number of 23) then the total number of bugs of which I have found specific
evidence of having the potential to cause impact to branch accounts is about 30.
431.There is no specific number of bugs in the Bug Table which acts as a threshold below which
Horizon suddenly becomes reliable, or above which it suddenly becomes unreliable. The whole of the
expert evidence, including the analysis and agreements within the Bug Table, and the effect of the
very numerous bugs, errors and defects which I have found to exist, must all be taken into account, as
well as the ultimate total. That I have done in reaching my conclusions on the Horizon Issues.
432.Finally on this subject, during the claimants’ oral closing submissions, there was an outline
chronology narrative undertaken of a series of Post Office and Fujitsu documents running from May
2000 onwards, as the claimants’ leading counsel explained it, to “identify the problem that SPMs had
with doubles”. I have referred to this in part at [62] to [64] of the substantive judgment. This
“doubling” problem or issue was that sometimes there would be a discrepancy - in the example given
in
PEAK 0043811 of May 2000, a sum which was expressed in the PEAK as being a “discrepancy in the
cash account”, in that instance the sum being £6,343,07 - which then inexplicably doubled to
£12,686.14. The chronology then went through different years, 2003, 2004, 2006 and into 2010.
Similar documents showed a variety of issues all related to such doubling. The majority of the
documents had all been put to different witnesses in the trial, and were referred to in the Bug Table.
Finally the chronology ended with a heavily redacted internal Post Office document of 23 May 2018,
called “Agenda - Operations Board” one entry of which stated (bureau means bureau de change, or
foreign currency):
“SAP GUI Bureau Value Issues
There have been 3 reported instances this week where the sterling value of remittances of bureau to
branches has doubled when the delivery has been booked into the branch, For example, Aylesbury
GPO were sent a bureau rem with a total value of £6.219,81, however when they booked it into
branch it populated as £12,439.62”
(emphasis added)
433. This narrative exercise drew some criticism from leading counsel for the Post Office. He said that
the exercise in which the claimants’ counsel was engaged was contrary to the rules of commercial
litigation, and the objection to this was put as follows:
“Another one of those rules is not to conflate a whole series of issues, all of which are very different,
bureau rems in, we have got other kinds of rems, we have got transfers between stock units, all sorts
of different issues which my learned friend picks up randomly over a period of 20 years and seeks to
RLITO000005
RLITO000005
promote them to your Lordship as a single problem which Post Office has known about for all that
time.
Your Lordship will have seen from the PEAKs that that's not the case. There were individual instances
of individual problems, as far as I can tell, some of these documents I have not seen before of course,
which itself is extraordinary given the nature of this trial and that we are at the end of it.
But my Lord what's happened is my learned friend is seeking to jumble things up and then create an
impression.”
He invited the claimants’ counsel to stop. The claimants’ counsel countered this objection by
explaining that he was demonstrating that “we have found a wide range of strands of causes of
doubling issues in the accounts of SPMs. We don't want to conflate them. We want to identify that, not
only are there cases where a SPM -- for example, the regional line manager puts it into suspense
[account] and it doubles...”
434.The reason for reproducing this is as follows. The Post Office seemed to draw comfort from the
fact that there was no single bug or cause of what was, in my judgment, on the face of the PEAKs,
quite inexplicable doubling of branch discrepancies, but rather the 18 year history of these arose from
a series of different problems within Horizon. The Post Office also insisted, and seemed to wish to
continue to insist, that these were “individual instances of individual problems”. That approach has
gone on for many years, and one would have thought that by now the
Post Office would have realised that something of a pattern has emerged in terms of Horizon’s
performance. The internal Post Office and Fujitsu documents (running through the whole life of
Horizon in this example) all demonstrated, to my satisfaction, unreliability of certain branch account
entries due to this doubling up of discrepancies. Even the 2018 Operations Board document showed
that these were still occurring in mid-2018 - on three separate occasions in that week alone. It does
not matter, in my judgment, to the claimants’ case, whether there was one single cause for these
specific discrepancies, or a number of different ones, all of which had a similar effect. The claimants’
case was, in any event, that there were a number of different bugs, errors and defects in Horizon that
all had this, or a similar, effect. Nor does it much matter for the purposes of the Horizon Issues
whether there was one single cause, or a number of different ones over a number of years. My
findings in this appendix show that there were a large number of bugs, errors and defects. There was
a far larger number than were admitted by the Post Office at the beginning of the Horizon Issues trial,
and a far larger number than ought to have been present in the system if Legacy Horizon and HNG-X
were to be considered sufficiently robust such that they were extremely unlikely to be the cause of
shortfalls in branches.
435.The issue of the state of knowledge of the Post Office is for another subsequent trial. This trial
deals with the answers to the Horizon Issues only, and all the other matters in the litigation remain for
trial on future occasions.
436.Going through the evidence, PEAKs and KELs, it can be seen that there is a slight majority of the
bugs, errors and defects that I have found to exist present in Legacy Horizon, as compared to the
number in Horizon Online. However, there are still a significant number in Horizon Online, although
not quite as many as in Legacy Horizon. The former regarding Horizon Online are those numbered 1,
3, 4, 5, 7 (although in the pilot), 8, 13, 14, 19, 23, 25 and 28. Those in Legacy Horizon are 2, 6, 9, 10,
11, 12, 15, 17 (an error or defect, not a bug), 18, 20, 21, 22, 24, 26 and 27. It is also notable, in my
judgment, that those in Horizon Online were in its earlier years. There is only one that dates from
RLITO000005
RLITO000005
2017 onwards, namely bug 14, Bureau Discrepancies, with some instances of bug 8, Recovery Issues
running into 2018. It must however be remembered that this analysis is on the KELs made available to
the experts. A very significant number of KELs were produced by Fujitsu towards the end of the trial
which the experts did not have time to consider, and an even more significant number were produced
in September 2019, well after the trial had ended. Neither expert has studied these.
437.1 do not know if what appears to be a marked improvement in the performance of Horizon since
about 2016/2017 is because, since this litigation commenced, matters have changed at Fujitsu in
relation to the Horizon System, including reporting and controls. There is no need to speculate. The
experts are agreed that Horizon, as it is in 2019, is relatively robust.
438.However, the Horizon Issues are not concerned with the Horizon system as it is at 2019. They are
concerned with the whole life of the system, from 2000 onwards. Mr Coyne also gave evidence that,
given he had not been able to consider all of the PEAKs and KELs that were disclosed prior to the
trial, as a matter of extrapolation (on the number of KELs of which the claimants were aware at that
time, which did not include the many thousands disclosed later in 2019), he would expect there to be
a maximum of about 40 bugs in the Horizon System.
439.1 accept that Mr Coyne was doing his best to assist the court, but this evidence is rather broad
brush in its approach. Given there have been, effectively, three iterations of the system, the number of
bugs, errors and defects have to be considered against the iteration of the system in which they
occurred. An obvious example of this is Legacy Horizon and Riposte. Riposte is no longer used and
has not been since 2010. A simple extrapolation across all 19 years of the life of Horizon (ie, 2000 to
2019) would not lead to a sound conclusion. In any event, the number of bugs, errors and defects that
have been found by me based on the Bug Table, without extrapolation but by the above analysis,
shows that there is ample material, without the need for any extrapolation, to come to measured
conclusions on robustness in any event.
440.The Post Office contended, in its closing submissions, for answers (this was most stark in the
issues that arose regarding robustness) that treated Horizon simply as a single system. As an
example, the answer for which it contended on robustness was summarised as “the Horizon system is
and was very robust (being more robust than most comparable systems).”
441.1 do not consider that it is justified to have one single or simplistic answer to the issue of
robustness that governs the whole life of Horizon over its 19 years of operation. Given the expert
evidence and agreements alone, it is not correct that any finding of the robustness of Horizon as at
2018 and 2019 equally applies to the system, say, in 2001 and 2002. I reject this as the correct
approach.
442.1 consider that the correct way to approach this issue is to consider the following three iterations
of the system:
1.Legacy Horizon, 2000 to 2010;
2.Horizon Online in its HNG-X form, which ran from the introduction of Horizon Online in 2010 to
February 2017;
3.Horizon Online in its HNG-A form, when it was changed to run on a different platform, namely
Windows 10.
RLITO000005
RLITO000005
443.It is correct to record that the experts did not approach the matter with a routine marked
distinction between the two versions of Horizon Online, HNG-X and HNG-A. However, that distinction
undoubtedly exists and Mr Godeseth explained the switch from NT4 to Windows 10 in his evidence.
The internal Post Office IT risk management document from 2017 stated that “the HNG-X platform is
end of life and is running on unsupported Windows software”, that it needed replacing, and also that
the "Branch counter technology is aged and unreliable, with frequent hardware failures, resulting in
branch disruptions." This plainly recognised that replacing the platform, which is what occurred when
the system became HNG-A, would result in some improvements, and this appears to have happened,
although the counter technology in the branch would not necessarily also be changed by a change
from one Windows platform (NT4) to another. The risk management document envisaged the required
replacement of both. Also, it has to be taken into account that the Horizon Online bugs analysed in
Part E of this appendix are, for the most part, those in HNGX. I do not see why the HNG-A form,
which both experts agree is relatively robust, should be considered in exactly the same light of the
PEAKs that arose (and there are a great many) during the HNG-X years. Dr Worden really only
concentrated on the HNG-A iteration of the system in any event, as he considered the operation and
functionality of the system as at 2018 and 2019, and this means HNG-A. He readily accepted that his
reports concentrated on Horizon as it was at the time of his investigation.
444.In my judgment, the following conclusions on robustness can be drawn regarding the three
different iterations of the Horizon system from 2000 to date:
1.Legacy Horizon: This was not remotely robust. Indeed, the issue about its robustness (or more
accurately, its lack of robustness) became increasingly obvious during the Horizon Issues trial. The
fact that the Post Office’s final submissions were forced to concede the existence of so many bugs,
with the battleground moving to the type of effect they had, rather than their existence, clearly
demonstrates in my judgment that all the weight of evidence, both of fact and expert, was heavily
against the proposition that Legacy Horizon was robust. It clearly was not.
2.Horizon Online in its HNG-X form: This appears to be slightly more robust than Legacy Horizon, but
still had a significant number of bugs, particularly in the years 2010 to 2015, but also after that and
leading up to 2017. Its robustness was therefore, in my judgment, questionable and did not justify the
confidence routinely stated by the Post Office (prior to February 2017) in its accuracy.
3.Horizon Online in its HNG-A form: This is far more robust than either of the previous two iterations
of the system. I accept Mr Coyne’s evidence that there are still far more TCs than expected arising
during its use, compared with comparable systems in the banking and finance sectors. However, it
may be that this is the only way in which it can sensibly still be operated. TCs are outside the scope of
the definition of the Horizon system in any event. It is also an unavoidable conclusion that,
notwithstanding the change of platform from Windows NT4 to Windows 10, it is still a Horizon system,
which given my conclusions at (1) and (2) is hardly a glowing endorsement of its antecedents.
However, the experts are agreed that HNG-A is robust, and better than it was in either its Legacy
Horizon or HNG-X forms. Horizon is not free from bugs from any particular date. As shown by the
Drop and Go bug, number 28 in the Bug Table, this occurred in June 2017 (the KEL is headed HNG-X
as explained in [410] which is after the date HNG-A was introduced, so it may be a bug in HNG-A).
Drop and Go appears to be a product introduced by the Post Office relatively recently, but I am
unaware of the precise date. The experts should seek to agree whether Bug 28 is a HNG-X or HNG-A
bug.
RLITO000005
RLITO000005
445,It is also relevant to record that the Horizon system as defined in the Horizon Issues, and as
pleaded in the Generic Particulars of Claim and Defence referred to in that definition, does not
provide any particular end date. Therefore HNG-A is included within that definition. It is also,
however, relevant to record that the effect of any bugs, errors and defects in HNG-A is likely to be
limited in this group litigation, as there is a cut-off date for claimants to have joined the group
litigation. This cut-off date is 24 November 2017, as extended in paragraph 15 of the 1st Case
Management Order sealed on 27 October 2017. The final date by which claims had to have been
entered on to the Group Register was 8 December 2017. November 2017 is only about nine months
after HNG-A came into being; therefore the vast majority of the claims are likely to arise under
Legacy Horizon and HNG-X, rather than HNG-A.
446.However, that does not mean that HNG-A is free from faults. Paragraph 3.1 of the 3rd Joint
Statement recorded that the experts agreed that “'robust' does not mean infallible and therefore
Horizon has and will continue to suffer faults. Robustness limits the impact of those faults and other
adverse events.” Also, Mr Latif’s experience with the Lottery and the scratch card discrepancy
occurred in January 2018. That means it arose under HNG-A. I have accepted his evidence on this,
which means that HNG-A must still contain bugs, errors and defects that have the potential to impact
branch accounts.
447.1 have also identified in the substantive judgment that the Post Office’s plans, if there are any, in
terms of the evolution of its IT systems and replacement of HNG-A, are not of themselves to be taken
as supporting the claimants’ case. I assume any plans for what was referred to as Thin Client are
either developed or developing, but there is no need to speculate.
448.1 return therefore to the passage at [8] above which is reproduced from Dr Worden’s report
where he was dealing with the double entry accounting principle. He stated that “any mistake is most
likely to have occurred in one column of figures, without any balancing mistakes on the other
columns. So, the mistake will immediately destroy the trial balance.” The consequence of the many
bugs that I have found that exist in Horizon, in both its Legacy and Online form, have the effect of
making the detection of errors so much harder than in conventional double entry bookkeeping. This is
demonstrated by so many of the Fujitsu documents where a great amount of investigation was
required by Fujitsu simply to work out what had actually happened, and why. On many investigations
this took even Fujitsu an extraordinary length of time, in excess of a year, and sometimes even two
years. Members of 3rd line support were involved, on multiple occasions, in this process. The experts
are also agreed that SPMs only had a relatively limited amount of reporting available to them on
Horizon, and that proper investigation required co-operation between the Post Office and the SPMs. It
must also be understood that the trading periods for each Post Office branch were usually 4 weeks,
and sometimes 5 weeks. At the end of each trading period a SPM had to complete a Branch Trading
Statement and then “roll over” for the next period to start. If a bug in Horizon had an impact on
branch accounts, then the Branch Trading Statement for that period would include the effect of that
impact.
449.One of the other functions of an accounting IT system was described by Dr Worden as the secure
storage and output of many types of information. That secure storage (what was called in this
litigation the audit store) is not effectively used if it is never, or rarely, accessed, particularly in
circumstances where there is a dispute between both parties to the account. By “both parties” I refer
to a SPM and the Post Office. This point is addressed in more detail in Part K of the substantive
judgment.
RLITO000005
RLITO000005
450.There is reference in some of the more recent documents to the involvement of Accenture. The
analysis of the Drop and Go bug, number 28 in the Bug Table, shows that even when a conclusion was
drawn by Accenture (in this instance relating to the failure of one of two £100 “top up” transactions),
this did not prevent Fujitsu from identifying in the KEL the following: “Problem: this may be an issue
with script ADCScript-CDBalanceTopUP or a user error”, even though there was plainly no user error.
User error means an error by the SPM. I do not know if this would be recorded
at Fujitsu in the same way if an incident similar to this were to occur today. However, it shows that for
an occurrence even as relatively recently as the summer of 2017, Fujitsu failed to give sufficient - or
indeed any - weight to the views of Accenture, who had “verified” that one of the top-ups had failed.
Fujitsu still recorded potential user error in the KEL, a point which is plainly contradictory to the
outcome of Accenture’s investigation in that instance.
451.The expression put to Ms Van Den Bogerd in cross-examination was “user error bias”, an
expression which she said she had not heard before. This term was explained as a tendency or bias
regularly to blame the user of an IT system for something, and the point put to her was that this was
“a common theme that runs through the Post Office’s approach” in this dispute. The exact passages of
her evidence are as follows: “Q. UEB, have you ever heard of that? A. No.
Q. User error bias?
A. No, I can't say --
Q. It is where people in IT constantly blame the user when actually it is not their fault?
A, Okay, I haven't heard that before.
Q. You haven't heard that?
A. No.
Q. Well, let's look at this because I'm suggesting to you this is a common theme that runs through Post
Office's approach when these issues are raised. Is that a fair suggestion?
A. I think what I have tried to explain is what the approach is, in 99 out of 100 that would be the case.
In this situation it is slightly different.”
452.1 do not know from where Ms Van Den Bogerd obtained her 99% statistic. Given there are 6
million transactions per day, it may not be particularly helpful to the Post Office in this group litigation
even if it were accurate. If it was meant as a 99% statement of certainty on her part of the strength of
the Post Office’s case on the Horizon Issues, it is plainly very wrong, as demonstrated by my findings
both in this appendix and the substantive judgment. As can be seen from the answers to the Horizon
Issues in Part M of the judgment, the claimants have been largely successful in their case on the
Horizon Issues. In my judgment - and regardless of whether Fujitsu and/or the Post Office was
applying user error bias, or not - it is clear to me that Fujitsu was far too ready, even after
investigations that clearly included the express discovery of bugs in code, to ascribe possible user
error to the effect of bugs, errors and defects that caused impact to branch accounts. It is to be hoped
that this is no longer the situation.
RLITO000005
RLITO000005
THE POST OFFICE GROUP LITIGATION
IN_THE HIGH COURT OF JUSTICE
QUEENS BENCH DIVISION
Rolls Building Fetter Lane London, EC4A 1NL
Date: 16 December 2019
Before :
Between :
Alan Bates and Others Claimant
- and -
Post Office Limited Defendant
Appendix 2
Summary of Bugs, Errors, Defects
Judgment (No.6) “Horizon Issues”
Summary of Bugs, Errors and Defects (Legacy Horizon or Online) and Years Experienced
1.Receipts and Payments Mis-match bug. This is a bug present in Horizon Online This bug occurred in
2010.
The majority of incidents are recorded as occurring between August and October 2010.
2. Callendar Square/Falkirk bug.
This is a bug present in Legacy Horizon.
It is agreed that this bug occurred between the years of 2000 and 2006, although there is an issue
about when it stopped. In my judgment, the period when the effects of this occurred are 2000 to 2010.
3, Suspense Account bug.
This is a bug present in Horizon Online.
Its identified years of effect are 2010 to 2013.
4.Dalmellington bug/Branch Outreach Issue. This is a bug present in Horizon Online.
Its effects were experienced between 2010 and 2015.
5, Remming In bug.
RLITO000005
RLITO000005
This is a bug present in Horizon Online.
It was present for about five months in 2010 between March and August.
6. Remming Out bug
This is a bug present in Legacy Horizon.
It was split into two in the Bug Table, 6(i) and 6(ii).
Bug 6(i) was identified as February/April 2007, recorded as fixed approx. 2007. Bug 6(ii) was in May
2005.
7.Local Suspense Account issue, not the same as 3. Suspense Account bug This is a bug present in
Horizon Online.
This was reported in 2010 and recorded as fixed in September 2010.
8. Recovery Issues.
These are bugs, errors and defects present in Horizon Online. With years of effect from 2010 to 2018.
9. Reversals.
This is a bug present in Legacy Horizon.
Occurred for a short period in 2003.
10. Data Tree Build Failure discrepancies.
This is a bug present in Legacy Horizon.
Its identified effect was during 1999 and 2000, running up to 2007.
11. Girobank discrepancies.
This is a bug present in Legacy Horizon.
This is treated as a single bug by the experts in the Bug Table. The Technical Appendix finds there was
more than one bug, error or defect under this heading. It occurred between May and September
2000.
12. Counter-replacement issues.
This is a bug, error, or defect present in Legacy Horizon. It also included hardware issues. There were
two KELs associated with this. The first was created in 2000 and last updated in July 2007. The
second was created in 2002 and notes occurrences running up to 2009.
13. Withdrawn stock discrepancies.
This is a bug present in Horizon Online.
Mentioned in 2010 and 2011.
14. Bureau discrepancies.
This is a bug present in Horizon Online.
RLITO000005
RLITO000005
It arose in 2017.
15. Phantom Transactions.
This is a bug present in Legacy Horizon. This is referable to Horizon issue 4 and not Horizon issue 1.
It arose in 2001.
16. Reconciliation issues.
This is a bug present in both Legacy Horizon and Horizon Online.
This is not a bug with a potential to cause impact on branch accounts. The underlying PEAKs do
however demonstrate software faults. Mentioned in 2000, 2002, 2010 and 2014.
17. Branch Customer discrepancies.
This arises in Legacy Horizon. This is referable to Horizon issue 4 and not Horizon issue 1.
This is an error, or defect and does not have the potential to cause impact to branch accounts.
Occurred in March 2008.
18. Concurrent logins.
This is a bug present in Legacy Horizon.
This occurred in 1999 and 2000, and the PEAKS run to 2002.
19. Post & Go/TA discrepancies in POLSAP.
This is a bug present in Horizon Online.
Occurred in 2012
20. Recovery Failures.
This arises under Horizon Online. This is referable to Horizon issue 4 and not Horizon issue 1.
Two of the PEAKs show errors and defects, they do not show bugs. Mentioned in 2010, 2012 and
2015.
21. Transaction Correction Issues.
This arises in Legacy Horizon. This is referable to Horizon issue 4 and not Horizon issue 1. Although
the Technical Appendix finds it is evidence of a bug, there is no impact on branch accounts.
Occurred in 2005 to 2010.
22. Bugs/errors/defects introduced by previously applied PEAK fixes.
This is a bug present in Legacy Horizon, There is no impact on branch accounts and this is therefore
taken into account under Horizon issue 4 and Horizon issue 1. Some of the PEAKS are from 2000 and
one is from 2004.
23. Bureau de change.
RLITO000005
RLITO000005
This is a bug present in both Legacy Horizon and Horizon Online. Arose in 2005, 2006 and 2010
onwards.
24. Wrong branch customer change displayed.
This is a bug present in Legacy Horizon.
Occurred in 2005.
25. Lyca top up.
This is a bug present in Horizon Online.
Occurred in August 2010.
26. TPSC 250 Report.
This is a bug present in Legacy Horizon.
The date range is between 2005 and 2009, with the majority of the incidents occurring in 2005
27. TPS.
This is a bug present in Legacy Horizon.
Its years of effect are 2006 to 2010
28. Drop and Go.
This is a bug present in Horizon Online.
Occurred in July 2017. The date of the KEL suggests HNG-A, but the KEL is expressly headed HNG-X.
29. Network Banking Bug.
This arises under Legacy Horizon.
This did not have financial impact on branch accounts. It is an error, or defect but arises from
communication failures from outside Horizon, probably from equipment or lines provided by BT.
KEL raised in 2004 and updated in 2005, with 12 associated PEAKs in the range 2004 and continuing
on to 2010 when Legacy Horizon was discontinued.
Case No: HO16X01238, HOQ17X02637 and HO17X04248
: ZW)
THE POST OFFICE GROUP LITIGATION
IN _THE HIGH COURT OF JUSTICE
QUEENS BENCH DIVISION
Rolls Building Fetter Lane London, EC4A 1NL
Date: 16 December 2019
Before :
Between :
Alan Bates and Others
- and -
Post Office Limited
Appendix 3
Glossary
Judgment (No.6) “Horizon Issues”
Term
AP
APOP
APS
BAL
BMS
BP
BRDB
BTS
CA
CAP
CMMI
DBA
DBMS
DEA
DRS.
DVLA
DWH
DWP
EPOSS.
ERP
HLD
HNG
RLITO000005
RLITO000005
Meaning
Automated Payment or
Automated Debt Collection
Automated Payment Out-pay
Database
Automated Payment Service
Branch Access Layer
Business Management System
Balancing Period
Branch Database
Branch Trading Statement
Cash Account
Cash Account Period
Capability Maturity Model
Integration
Database administrator
Database Management System
Double entry accounting
Data Reconciliation Service
Driver and Vehicle Licencing
Agency
Data Warehouse
Department of Word & Pensions
Electronic Point of Sale Software
Enterprise Resource Planning
High Level Design
Horizon New Generation (or
Online) implemented in 2010
HNG -A
HNG -X
HSD
Is
JSN
KEL
Legacy
Horizon
LFS
MD5
MID
MIS
MSC
NBSC
Pathway
PCI
PDL
POA
POL FS
POLSAP
POS
PRINCE2
remming
Riposte
SAP
SAP ADS
SEC
SEK
SLA
SMC
SPM
ssc
su
TA
TC
RLITO000005
RLITO000005
Horizon Online since 2017
Horizon Online 2010 - 2017
Horizon Service Desk
Infrastructure Services
Journal Sequence Number
Known Error Log
Horizon 2000 - 2010
Logistics Feeder Service
Technique for creating a hash code for some
piece of data, which would change if the data
was altered in any way
Manual Inspection of data
Management Information System
Managed Service Change
Network Business Support Centre
ICL division created for Horizon, which later became part of Fujitsu
Payment Card Industry
Process Definition Language (an example of reference data)
Post Office Account
Earlier name for POLSAP, from 2004 - 2010
PO’s central accounting system since 2010
Point of Sale
Project Management Methodology
Pre-Trial Review
Real Application Custer
Reference Data Delivery Service
Reference Data Management Centre
Reference Data Management System
Redundant data storage and computing
Remitting cash or stock
Software product from Escher, used to build software for Post Offices
Industry leading resource planning and accounting system
SAP applied to administrative data services
Security
Secure kernel hardware and software
Service Level Agreement
System Management Centre
Subpostmaster
Software Support Centre
Stock Unit
Transaction Acknowledgement
Transaction Correction
TES
TIN
TIP
TP
TPS
VPN
WSPOS
XML
Transaction Enquiry Service
Transactional integrity and recovery
Transaction Information Processing
Trading Period
Transaction Processing Service
Virtual Private Network
Web Services POS
Extensible Message Language
RLITO000005
RLITO000005
1 {7/364}.