POL00022841
POL00022841
Case No: HQ16X01238, HQ17X02637 and HQ17X04248
Neutral Citation Number: [2019] EWHC 3408 (QB)
THE POST OFFICE GROUP LITIGATION
IN THE HIGH COURT OF JUSTICE
QUEENS BENCH DIVISION
Before :
Rolls Building
Fetter Lane
London, EC4A INL
Date: 16 December 2019
THE HONOURABLE MR JUSTICE FRASER
Between :
Alan Bates and Others
- and -
Post Office Limited
Technical Appendix to
Judgment (No.6) “Horizon Issues”
Claimants
Defendant
POL-0019320
N
=mBOOD>
we
POL00022841
POL00022841
This is the Technical Appendix to Judgment (No.6) which deals with the Horizon
Issues. The purpose in preparing this is to avoid what is already a long judgment
being vastly increased in length by including technical IT details within it, which are
not necessary for the vast majority of readers to understand.
This appendix has the following structure:
The History of the Horizon System
Structure of Horizon generally
Legacy Horizon and Horizon Online
Submissions and Evidence
Bugs, errors and defects
Conclusions on Technical Issues
The contents of this appendix are predominantly drawn from the two expert reports,
as well as the factual evidence, but also the parties’ extensive submissions and some
of the contemporaneous documents deployed in the Horizon Issues trial. Given the
size of the system, and its evolution into Horizon Online from the days of Legacy
Horizon, it would have been entirely possible to draft an appendix that dwarfed the
length of Judgment (No.6). In order to avoid that, as such a lengthy document would
not be proportionate or reasonable, I have summarised certain aspects of the technical
evidence and documentation where necessary. There may therefore be some passages
in this appendix which, for example, to someone vastly experienced in writing Oracle
code, may not read as being 100% technically accurate or which seem to be a
superficial summary. However, in my judgment, such lack of full technical accuracy
in description or terminology does not affect either the consideration of the evidence,
or the conclusions, or my findings on the Horizon Issues.
The History of the Horizon System
There was a great amount of agreement between the experts on both the history of
Horizon and its architecture, both Legacy Horizon and Horizon Online.
Originally, in the 1990s, there was an intended tri-partite project between the Post
Office, the government department responsible for welfare benefits, and ICL. During
evidence the second of those two parties was often referred to as the Department of
Work and Pensions, or the Benefits Agency. In fact, prior to 2001 the relevant
department was the Department of Social Security or DSS (which had been created in
1988 from its predecessor, the Department of Health and Social Security or DHSS).
The Department of Work and Pensions was not created until 2001. The different
terminology does not matter, other than for the sake of accuracy. At least part of the
reason for the project was so that benefits claimants could be paid by using a card,
rather than what used to be the case with paper books which a branch Post Office
would stamp, remove and exchange for cash.
The tri-partite project was called Pathway. ICL Pathway (the department within ICL
formed for the purpose) was awarded a contract for an electronic way of paying
benefits, the Benefits Payment Card, in May 1996, and there was a Horizon Pilot that
was introduced in a small number of branches in 1996. This tri-partite scheme was
POL-0019320
POL00022841
POL00022841
abandoned in 1999. Pathway cited “greater than expected complexity” and “...major
implications for the degree of difficulty of the project”, quotations included in Mr
Coyne’s report, as ultimately leading to the failure of the project. In July 1999 Post
Office Counters Ltd (or POCL, the previous name of what is now the Post Office Ltd)
and Pathway agreed to utilise the project to automate branch Post Offices. This is
what is now called the Horizon System or Legacy Horizon, and it was introduced (the
technical term is “rolled out”) from late 1999 onwards.
Anyone with a modern smart phone will know that modem IT technology is marching
ahead at an increasingly ferocious pace, and particularly those adept at its use now
(including the younger generations) may be surprised that so much of what is now
dealt with electronically, used to be done using paper. Prior to the introduction of
Horizon, the branch Post Office system used a system of paper-based accounts. A
paper-based system was used by SPMs to account to the Post Office for the branch
activity, although some SPMs used software packages to assist them in this. Although
Horizon is used as an accounting system — it is one of its main uses — there is more to
it than that. However, computerised accounting systems have been in use since the
1950s, and Dr Worden explained that accounting was one of the “most mature
applications” for computers in business. This means that as a computing application,
accounting has been such an application for a long time, even in the late 1990s.
Dr Worden explained the origin of double-entry bookkeeping and the principles that
underlie it, and explained the process of manual bookkeeping, even (in footnote form)
as far back as the clay tablets of Ur, a Sumerian city-state in ancient Mesopotamia
which dates from 3800BC. The cuneiform tablets of Ur, as he pointed out, were not
double-entry bookkeeping, which was invented in the 14" century. They were single-
entry; an entry would be recorded once. Double-entry bookkeeping means that
discrepancies can be discovered much more speedily, and easily, than otherwise. I
consider Dr Worden’s explanation of this to be sufficiently concise and helpful that I
will reproduce parts of it from one of his appendices:
“12. Because any accounting system is intended to track external reality, and to give
the most accurate possible picture of that reality, it is essential from time to time to
check the picture of reality, held in the accounting system, against reality itself - the
physical assets of the business, its money in cash or banks, its obligations and debts.
13. However, checking against external reality is (or was) an expensive process. You
cannot simply look at the books - you have to get up from your desk, go out into the
warehouse and count stock, check your bank balance and count your cash, consult
other people, and so on. Because checking against reality was an expensive process, it
did not get done very frequently. If the check is only made occasionally, and mistakes
are found, the interval of time in which one or more mistakes might have occurred is a
long one. The error is more likely to have arisen from multiple mistakes. Looking for
several mistakes together is much harder than looking for one mistake; there is no
‘telltale number’ to look for. The process of ‘drilling down' to find the origin of a
mistake is difficult and unguided, with few clues.
14. Double entry bookkeeping changed this. If a bookkeeping mistake is made, that
mistake will lead to a discrepancy against external reality, which will eventually be
found in the external reality check. But any mistake is most likely to have occurred in
POL-0019320
POL00022841
POL00022841
one column of figures, without any balancing mistakes on the other columns. So, the
mistake will immediately destroy the trial balance. Checking the trial balance is much
easier and cheaper than checking against external reality. It can all be done by sitting
down at a desk with the books and an abacus or calculator - so it can be done much
more frequently. When a discrepancy is found, it is now much easier to drill down
and find its cause. For instance, if each entry in the books is dated (as it will be), by
just inspecting the books you can find the exact date and nature of the transaction
which was not recorded as zero-sum, and which destroyed the balance.
15. So double entry bookkeeping immediately reduces the cost of keeping accurate
and trustworthy accounts. It was an early, and highly effective, form of error
repellency - in a time when manual errors of bookkeeping were likely to be frequent.
The error repellency guarded against a single point of failure (i.e. a mistake in a single
column of figures) by making any such mistake rapidly and obviously visible, in the
next trial balance.”
(emphasis added)
A very simple example was given by Dr Worden in his Appendix B, which again I
will reproduce. For those with no experience of the pre-decimal system the units of
money might be a little vague, but the principle applies whether one expresses the
money in pounds or shillings:
“8. Double entry bookkeeping can be understood by starting with the simple case of a
trader on a farmer's market, who has some money, and who has some sheep. He can
keep a list of his money, and a separate list of his sheep - complete with their names if
he wants to. He can track changes to those lists, with entries like '23rd July: sold one
sheep - Dolly’. From the changes he can work out his current position, in money or in
sheep: 'now I have 17 sheep left’. This is single-entry accounting.
9. Double-entry bookkeeping starts when his list of sheep contains two types of
information - the sheep he has, and the price he paid for them; and he also keeps a
separate list of his money. He tracks the changes to these lists in a series of dated
transactions: '25th July: bought one sheep for 5 shillings’. With that transaction, the
sum of his money list goes down by 5 shillings, and the money total of his sheep list
goes up by 5 shillings. So, he makes two entries - in those two lists - with money
value -5 shillings and +5 shillings, and the total money value of the two lists does not
change. Because of the self-consistency of mathematics, however he chooses to do
those two sums, the sum of the two sums should not change from day to day. This is
his double entry ‘trial balance’. If the number does change from any day to the next, he
knows he has made a mistake - and he can start to track it down.”
This simple example can be applied to any stock held in any business, in this litigation
branch Post Offices, and the sum of money which is used to purchase stock in a
branch by a customer. To return to the well-worn example of the book of 1“ class
stamps used earlier in the litigation, if a SPM sells one book of these to a customer for
cash, the cash position of the branch improves by the cost of that book of stamps, paid
to the branch by the customer (as of today, £8.40 for a book of 12 1* class stamps)
and the stock of books of stamps held in the branch reduces by one. This is a simple
example because the stamps are part of the branch’s stock. There are more complex
transactions in a branch that do not involve stock actually held within the branch; this
POL-0019320
13.
14.
POL00022841
POL00022841
will usually be because the branch offers services and products on behalf of the Post
Office’s numerous clients, that are not stock in the traditional sense physically held
within that branch.
Dr Worden also explained the advantages of double-entry accounting, which include
reduction of fraud and theft. Notwithstanding the differences to which accounting
systems are put (or to put it another way, how the information held within such
systems is used), such as management accounting (for those responsible for making
management decisions about the business in question) or financial accounting (for
example, for departments who have to declare profits and account for taxation) both
sets of information are subject to audit, which is a word which means review,
examination, inspection and appraisal. In lay terms, it is the checking of the
information held within the accounts.
The word “audit” is used in different ways, and for different things, in this group
litigation. The Post Office employs a number of people called auditors to check on the
accounts of branch Post Offices. When Mr Bates, for example, started as a SPM, the
Closing Day Audit was conducted on the branch (run by his predecessor) as part of
what occurred on what was called Branch Handover Day. This is done so that a SPM
on their first day starts with particular stock and cash, identified by the Closing Day
Audit, and their branch activity uses that as a starting point. Some SPMs may be
suspended, or have their engagements suspended, as a result of what the Post Office
(through its auditors) uncovers at other audits, which may be performed at any time
(which is entirely understandable, and is included in the terms of the SPMs’
contracts). These audits are done to check the state of the branch accounts, and the
cash and stock held in the branch.
The Post Office itself is also audited by its auditors (such as Ernst & Young or E&Y),
and the Horizon System is itself audited in the sense of checks being performed that it
complies with various standards. Audit is also used as part of the expression “audit
data”, further explained in Part K of Judgment (No.6) itself.
The financial accounting functionality of the Post Office was provided by something
called SAP, which is an industry leading resource planning and accounting system.
From 2004 onwards this was by something called POLFS, and in 2010, with the
introduction of Horizon Online, this became POLSAP. POLSAP was created by
merging two other systems, POLFS and SAP ADS, the ADS in that latter acronym
standing for Administrative Data Services. Horizon provided mainly two things;
management accounting functionality, as well as Electronic Point of Sale functions
for the branches (also called EPOS). There is, in Horizon Online, overlap between
Horizon and POLSAP. Dr Worden stated:
“I understand that the financial accounting functionality for the Post Office was
provided by SAP over most of the disputed period (from 2004 by POLFS, a SAP
application, which in 2010 merged with SAP ADS to form POLSAP), whereas
Horizon provided mainly management accounting functionality, as well as Point of
Sale functions for the branches. This dual nature required close integration between
Horizon and the SAP systems, so that pictures of financial reality given by the two
systems were at all times mutually consistent. There is a large overlap between the
information stored in the two systems, which will be described below.”
POL-0019320
POL00022841
POL00022841
(emphasis added)
Some computers provide a highly complex computation function. Dr Worden stated,
and I agree with him, that the functions of an accounting system include the input,
secure storage and output of many types of information, in large volumes (many
transactions per day), with rather little computation. By far the most important type of
computation that occurs in financial accounting is straightforward summation of
numerical quantities such as money and stock, subject to the rules of double entry
bookkeeping. For management accounting, some other kinds of computation are
needed, such as for forecasting purposes. However, these are not usually
computationally demanding or complex. This is not disparaging of those who
designed the system, but given the function of Horizon so far as branch Post Offices
are concerned, Horizon does not possess (nor does it need to possess) a highly
complex computational function in the sense that very difficult and demanding
complex mathematical equations have to be processed at high speed, for example. The
computers at NASA in the United States that are responsible for space exploration,
and control of satellites, will be dealing with far more complex computation in terms
of mathematical function; however, rarely would there be 12,000 moon shots
underway at once. Horizon deals with a huge number of transactions per day (about 6
million) across many thousands of branches, but the nature of the calculations is
essentially arithmetical. This is not to understate the complexity of Horizon both in
terms of the scale of the system, the number of products or the number of different
clients with whom the Post Office (and hence SPMs in branches, on behalf of the Post
Office) deal. Fujitsu, in a document of 14 January 2004, described Horizon as
‘Europe's largest non-military IT contract’, so Horizon is at the high end of
complexity amongst IT systems. Dr Worden stated that it represents “many thousands
of man-years effort in development and testing, and its documentation alone more
than 100,000 documents”. It can therefore clearly be seen to be undoubtedly highly
complex.
However, Dr Worden also said this:
“On the other hand, compared with the IT estates of various large organisations I have
worked for (such as the NHS, Barclays Bank, UBS or RBS), the Horizon system is
probably no more complex, and in some ways less complex. The banks' IT estates,
like Horizon, have a complex corporate back-end and an extensive branch office
network. They were developed over a longer time period (30-40 years) using
development team sizes similar to or larger than Horizon, often merging together or
integrating the IT systems of previously independent organisations. This gave them a
degree of legacy complexity, and design compromise, and corporate amnesia, not
found in Horizon. There are parts of these IT estates which 'nobody dares touch'.”
This means that simply because Horizon evolved as it has is not an answer to the
more controversial of the Horizon Issues. Its origins as a system to accomplish
something else, the payment of benefits, its incremental addition of products and
clients, and its age, are not of themselves indicative of any particular level of
performance.
POL-0019320
19.
20.
21.
22.
POL00022841
POL00022841
Structure of Horizon Generally
Although computerised accounting systems had been used widely for some decades
since about the 1960s, in the 1980s there was what was called a “major step change in
their architecture”, when relational databases were invented/developed and used. A
relational database is a digital database based on the relational model of data, and was
first proposed in 1970 by EF Codd. Mr Codd was an English computer scientist and
invented the relational model for database management. This model uses a structure
and language consistent with what is called first-order logic, although it is also called
other things (such as quantificational logic) and this is used both in mathematics,
computer science and also linguistics. Inter alia it allows the use of sentences that
contain variables. In the relationship model, all data is represented in terms of tuples,
grouped into relations. In relational database theory, a relation is a set of tuples
(di, do, ..., dn), where each element d; is a member of Dj, which is called a data
domain. Each element is termed an attribute value, and an attribute is a name paired
with a domain (which is also called a data type, or simply a type). In mathematics a
tuple is a finite ordered list or sequence of elements.
A database organised in terms of the relational model is called a relational database. A
software system used to manage relational databases is called a relational database
management system or RDBMS. Many such systems use Structured Query Language
or SQL to query and maintain the database. A famous and widely used RDBMS is
Oracle RDBMS, which is sometimes called Oracle Database or even simply Oracle.
This is a proprietary multi-model database management system produced and
marketed by the Oracle Corporation, a US technology corporation based in California.
It was first released in 1979.
The system software therefore used for a relational database is built and supported not
by the accounting system developer, but by the supplier of system software, which
will be a major company such as Microsoft or Oracle. The accounting software makes
calls to the RDBMS to store and retrieve the information. These work (in outline
terms) by storing structured information. Unstructured information can be stored in
computer systems generally, but this requires very advanced computer programs to
understand it. More simple programs, such as accounting systems, store and use
structured information. Structured information is (for example) something contained
in a spreadsheet, made up of information in different cells in rows and columns.
Adding up the value of a list of transactions can be accomplished simply by adding up
the figures in a particular column that relates to this information for a number of
transactions.
The main use of a relational database is securely to store large volumes of structured
information. The way it does so can be understood as having large numbers (tens,
hundreds or even more) of different spreadsheets (which are called tables) and which
are linked to one another. Two different tables in a database are linked to one another
(in a 'relation') when they both have one or more columns with the same meaning and
share values in those columns.
Dr Worden explained:
POL-0019320
23.
24.
POL00022841
POL00022841
“44. A typical relational database may have tens or hundreds of separate tables; each
table may have tens or occasionally hundreds of columns; and a table can have any
number of rows, up to millions if necessary.
45. One relational database can act as a 'server' to one or more application programs,
which are performing actions directly visible to their users. The application programs
make calls (requests) to the relational database management system, which alters or
retrieves the data to fulfil the requests. The core service provided by the relational
database is to securely store and retrieve this information, for the application
programs, or for others who retrieve information from the database more directly.
Both the phrases 'securely store’ and 'retrieve' have a lot packed into them.
46. The phrase ‘securely store’ implies several guarantees: that once an item of
information has been stored in the database, it will not be lost or unintentionally
changed; and that it is a part of a consistent collection of information, whose integrity
will not be compromised in any way.”
The integrity of information is central to relational databases. Indeed, I consider that
the integrity of information could be said, in a way, to be central to the concept of the
operation of Horizon, if not accounting altogether.
Dr Worden explained the concept of integrity, and integrity constraints, in the
following way:
“47. The idea of integrity of information is central to relational databases. When the
structure (tables and columns) of a relational database is first defined (in a relational
schema) that schema defines various types of integrity constraint on the data, for
instance that:
47.1. certain columns in tables are mandatory, and will always be given values.
47.2. There can never be a record in some table (call it table A) unless there is a
corresponding record in some other table B. For instance, there can never be a record
of an invoice to a customer, unless a record exists for the same customer. The invoice
table and the customer table both have a column ‘customer id’; for every invoice
record, there must be a customer record with the same customer id.
48. The database management system guarantees that these integrity constraints will
be true for all time. If any application tries to make a change to the database which
would violate an integrity constraint, that change is rejected by the DBMS, and no
change is made at all. One change to the database may involve changes to several
tables at once, so that after all the changes are made, the integrity constraints are still
true; but part way through the changes, the constraints are temporarily untrue. One
such package of changes, involving changes to one or more tables, is called a database
‘transaction’. The DBMS guarantees that:
48.1. After any completed transaction, the database will still obey all its integrity
constraints.
POL-0019320
i
”
26.
POL00022841
POL00022841
48.2. After any completed transaction, the changes made to the database can never be
lost - even in the event of hardware failures; there are robust ways to recover the
information.
48.3. If a transaction would violate an integrity constraint, it is not allowed by the
DBMS; no changes will be made to any table, so the database will still be in a
consistent state, as if the change had never been requested.
48.4. When one application is making multiple changes to the database in a
transaction, so that the database is temporarily in an inconsistent state (when some but
not all of the changes have been made), those changes are never visible to other
applications or users before they have all been completed. Other applications and
users can only ever see a consistent state of the database (either before all the changes,
or after all changes), in which all the integrity constraints are true.
49. This set of guarantees, given by the DBMS, is called ‘transactional integrity’. It
has been built into the fabric of all relational databases and has been relied upon by
thousands of applications which use relational databases, since the 1980s. Builders of
applications can have a very high degree of confidence that their DBMS will maintain
transactional integrity. Builders of accounting systems have relied on that guarantee.”
Relational databases support the SQL language, and small numbers of records can be
retrieved from within application programs from a limited number of linked tables,
and the records are filtered based on data values. Dr Worden considered that the
possibility of there being a bug in Oracle to be “extremely remote”, and he ignored it.
As he put it:
“51. SQL can be used directly by end users, without the use of any application
program, to selectively retrieve and display records. However, it is more common for
the users to use a general report writing tool, which not only retrieves the required
records, but formats the information in easy-to-read reports - with appropriate column
headers, formatting of fields, grouping of records, computations of sums of groups of
records, and so on. Nearly all the reports needed from an accounting system are
created using these report writing products, which can be rapidly configured to
produce any new kind of report, either one-off or regularly as required.
52. All these capabilities of a relational database have been present in relational
database products such as Oracle since the 1980s and have been tested by possibly
millions of different applications which use those capabilities and rely on them. So,
when I am discussing the issue of bugs in an application such as Horizon, the
possibility that these bugs arise from bugs in the underlying DBMS - particularly
when it is the world market leader Oracle - is extremely remote, and I shall ignore it.”
This is a similar approach to that adopted by Mr Godeseth when he was giving his
evidence about PEAK 0195561, when a problem was reported to the SSC on 4 March
2010. This was where a SPM had tried, on 2 March 2010, to transfer out £4,000
(referred to in the PEAK as 4,000 pds, which means either pounds (plural) or pounds
sterling) from an individual stock unit into the shared main stock unit when the
system crashed. Mr Godeseth had been looking for a bug at the time of a “red alert”
during the early days of Horizon Online, and his evidence about this is included at
[341] to [349] of Judgment (No.6). He referred to one possibility being a bug within
POL-0019320
27.
28.
29.
POL00022841
POL00022841
Oracle, which is dealt with at [350] of that judgment, which he said “would be huge”
and which he also said “was not the bug that I was looking at for the red alert”.
For the avoidance of doubt, when the experts reach agreements, or where I make any
findings, about the presence of bugs, errors and defects in the Horizon System (either
Legacy or Online) either in this appendix, or in Judgment No.6, these are not made
concerning exactly where within the different programs or applications within the
Horizon System such bugs, errors or defects are present, or by apportioning
responsibility to any particular software provider. The Horizon System includes
Oracle as the RDBMS, but there are a considerable number of other companies
involved. Those companies are not represented and that is not what the Horizon Issues
trial is about. The experts would sometimes refer to what were called “sub-contracting
parties” but it was not clear to me whether they were sub-contracted either to Fujitsu
or to the Post Office. Nor is it relevant to the Horizon Issues whether the fault is
within a particular layer within the system. The relevant issue is whether there are
bugs, errors or defects within Horizon.
The software in Horizon is what is called data-driven. If the accounting application
software was written in a way that depended directly on the accounting needs of a
client (here, the Post Office, what Dr Worden called a “chart of accounts”), then each
company would need different accounting software. This would be very
uneconomical (in writing and testing all that software), and this is not the way in
which software is developed. The chart of accounts is not ‘hard coded' into the
application software. The software is written to work with any valid chart of accounts;
and the chart of accounts itself is treated as data and stored in the database. In this
way, changes in the chart of accounts (which occur from time to time) do not require
changes in the accounting software.
“59. This is one example of data-driven software - in which any requirement which
might change frequently is encoded as data, rather than software code. The code is
written and tested to work with all allowed values of the data, to meet a wide range of
requirements by changing the data rather than the code. The modern practice of
software engineering is to use data-driven software as far as possible. This is not
always as far as one might like; some differences between companies and their
requirements are best expressed as differences in code, rather than data”.
Accordingly, the accounting application or applications are built as layered software
architecture. This means that the accounting application is built “on top of” the
relational database. There will usually be a minimum of three layers in a typical
system of this type. Layers are also sometimes called tiers. These are the user
interface layer, which presents information to users, and accepts their inputs. This is
the layer that a SPM would both observe, and which would react to the actions of the
SPM (or their assistants) in the branch such as pressing buttons at the terminal for
various transactions. The second layer is the business logic layer, responsible for
carrying out business processes, and supporting users in doing so. The third layer is
the data layer, responsible for storing and retrieving data. More complex architecture
has more than these three layers, and there may even be layers within layers. In the
case of Horizon, the user interface also includes a Point of Sale interface which is
used in the branches. This interface can be thought of as part of the Horizon Point of
Sale application, rather than the Horizon accounting application.
POL-0019320
30.
POL00022841
POL00022841
Almost all complex IT applications are designed in this way, in order to isolate
different kinds of complexity within different layers, and to reduce the possibility of
unwanted interactions between functions in different layers. A typical ‘client server’
layering structure would include at the very least the three layers identified above (a
user interface layer, a business logic layer, and a data layer) but the layering in
Legacy Horizon is more complex than this.
The main purpose of defining an architecture in layers is to separate the functionality
into parts in the different layers, with well-defined and simple interfaces between the
layers. This not only makes each layer easier to design, build and test; but also, if
there are errors that are not found in testing, it should be easier to understand and
isolate the cause of the errors by inspecting the exchanges between layers. Dr Worden
considered layered architecture to be an important countermeasure. It could also be
seen as an important part of designing such a system. It is part of modern practice.
Legacy Horizon
32.
The architecture of Legacy Horizon up to 2002 is described in a lengthy document
called Technical Environment Description dated 22 October 2002. This document
states: 'The system architecture adopted to meet these requirements is not based on
conventional client-server models. Nor does it conform to traditional central-system
models. It adopts an entirely original and highly innovative four-tier model that
effectively merges the qualities of central systems and client server systems’. The
diagram that accompanies this in the document actually has five layers rather than
four. If two boxes in the diagram which are entitled 'Agents' and ‘Correspondence’ are
counted as one 'Agents' layer, which is what Dr Worden did, there are four layers:
1. Counter.
2. Agent.
3. Host.
4. External Interface.
POL-0019320
33.
36.
37.
POL00022841
POL00022841
Cioms TP OCS: SAPADS = -NBE
ORS
Hoss
Disrribuion
=
&
Conanccr APS I EPOSSI ORCS; I LFS NaS Courer
Parpherals
The counter layer consists of all hardware and software in the branch. It includes all
hardware and software required to support the counter activities required for all
products and customer services offered in the branch. This was largely built based on
a commercial product, Riposte from Escher Group, a separate company from Fujitsu.
Riposte provided much of the Graphical User Interface (which is the basis of all user
input and output at the counter) and provided a mechanism for secure distribution of
messages between the branches and the two back-office data centres, which are also
referred to in some places (including Dr Worden’s report) as campuses. These were
located at Bootle and Wigan. The message distribution passed through the Wide Area
Network in the diagram.
The Correspondence Servers handled communication over the network.
The function of the Agent layer was to provide two-way translation of data between
the formats used in the counter layer and the network (these formats were described
by Attribute Grammars) and the formats used in the Host layer. The agent layer is also
tesponsible for extracting the Audit of all data passing through the Correspondence
Servers.
An Attribute Grammar is a description of a message structure which is set out in the
same pattern as a tree. Dr Worden used the phrase “a tree-like message structure in
terms of its parts and their sub-parts”. In more recent IT systems, tree-like messages
are usually sent in XML (Extensible Message Language), with their structure defined
in a notation called XML Schema. This is used in parts of Horizon Online. An XML
document is a string of what are called characters. These characters are divided into
markup and content. What is called the processor analyses the markup and passes
structured information to an application. Generally, strings that constitute markup
POL-0019320
38.
39.
40.
41.
POL00022841
POL00022841
begin with the character < and end with a >, or they begin with the character « and end
with a ; (ie a semi-colon). Strings of characters that are not markup are content.
Because Legacy Horizon was developed before the use of XML became widespread,
Attribute Grammars fulfilled this function in Legacy Horizon. This is probably the
way that Riposte worked. In Legacy Horizon, Riposte provided the facility for reliable
teplication of data between the branches and the back-office or data centres/campuses.
This means that if certain types of data were created at the branches, Riposte should
have ensured (the term used by Dr Worden was “guaranteed”, but I consider that
overstates it) that the same data would be available at the data centres/campuses. Dr
Worden accepted that “if the underlying network was unreliable, it might take some
time for Riposte to deliver this guarantee”. He did not say how long that might be and
he was not asked about it.
One of the bugs in the bug table was also called the “Riposte Lock” problem. This
will be further addressed in Part E of this appendix. It was a problem with Riposte
that was known about at Fujitsu.
The bulk of the back-office functionality was provided in the Host layer. Host
applications were typically batch systems, processing data in large batches on a daily
basis. A complex daily batch schedule was used to control the sequence and timing of
these batch processes, using the Maestro scheduling product (not to be confused with
the Maestro software, made freely available by NASA to allow users to look at
photographs and the daily progress of the Mars exploration rovers). It was the Host
layer (and for most purposes, only the Host layer) which communicated with the IT
systems of Post Office client organisations, through the External Interface Gateways.
Dr Worden described the three layers of architecture which resided at Wigan and
Bootle as “the back-end architecture”, and these were the Agent layer (which included
the Correspondence layer), the Host layer, and the External Interface layer.
TIP MIS: External Interfaces
$ e °
TPS Data [4 Host RDMS
Warehouse ° s
hd ,
Agent
Counter
The following are shown in the above diagram as elements, and some description of
them is required.
POL-0019320
42.
43.
44.
45,
46.
47.
48.
49.
POL00022841
POL00022841
RDMS is the Reference Data Management System. Reference Data is described
further at [52] below. This management system is a dedicated IT application which is
needed to manage the reference data, and to distribute it appropriately to branches.
The Data Warehouse consists of one or more databases, whose structure is designed
to support flexible and open-ended querying and reporting by the Post Office.
Different kinds of information which pass through the host systems are siphoned off
into the data warehouse and stored there. This storage is within data structures
designed for reporting (which is done by means of querying those structures).
Functionality which depends on a data warehouse includes the Management
Information System or MIS and other applications such as those which look for
unanticipated trends and correlations in data.
MIS is the component built on the data warehouse to provide the Post Office with
relevant information. The data warehouse and the MIS are important parts of the
Horizon System. Dr Worden stated that “in cases of human error in business
processes, operational errors in managing Post Office business on Horizon, or
software errors in Horizon, some resulting discrepancy or aberration will be rapidly
visible through the MIS. MIS facilities were also used by Fujitsu staff. Many pairs of
eyes are inspecting the outputs of the MIS, in hundreds of different reports or
spreadsheets. One purpose of this is to ensure the rapid detection and correction of
many types of errors. These include software errors”. By “pairs of eyes” Dr Worden
was referring to human intervention or observation.
The Transaction Processing System or TPS has as its purpose the ‘harvesting’ of all
types of transaction taking place in the branches, and to pass them on to other IT
systems in the Post Office - initially to TIP, and later to POLFS when the system was
changed.
Transaction Information Processing or TIP was, as at 2003 (the date of the diagram
above) the gateway to all other Post Office data processing, including accounting.
After 2004, Post Office accounts were held on an SAP system, which was called
POLFS; so TPS passed data to POLFS, rather than to TIP.
The black circles in the diagram denote points, as Dr Worden expresses it, ‘at which
ownership of data conceptually changes and hence at which audit information is
generated’.
Dr Worden quotes the 2002 Technical Environment Description document, what he
refers to as the “the 2003 design document” to explain the Host at paragraph 165 of
his report. ‘One essential task that can only be carried out at the Host layer is
reconciliation. The Host is the only system component that can detect discrepancies
between the transactions carried out at the Counter (and hence reported back to Post
Office Ltd via TPS), and those that were authorised or expected. It should be in a
position to send reconciliation reports back to its Client. These enable the discrepancy
with the TPS records to be identified and resolved.’.
He considered that “this reconciliation, carried out in the Host layer, is an essential
element within [Legacy] Horizon for detecting and correcting errors made at the
counter (robustness measure UEC)”.
POL-0019320
50.
51.
52.
53.
54.
POL00022841
POL00022841
Reconciliation and TCs have the effect of detecting and correcting the effects of many
possible software errors, as well as human errors. Dr Worden was of the view that if
there were any such software errors, these would probably occur with such high
frequency, and occur uniformly across all branches, as giving rise to so many TCs,
that the Post Office would soon suspect a software error (for instance, seeing the
effect repeatedly in some MIS report) and require Fujitsu to correct it. However, this
is supposition in my judgment. This is because it depends upon a software error
occurring very frequently, and being noticed rapidly. Some accepted errors (for
example Callendar Square) lay in the system for 5 years. The evidence of fact
advanced by the Post Office itself, which includes that in respect of the admitted or
acknowledged bugs, makes it obvious that this supposition is not made out. Further,
the detail of the numerous PEAKs show that it is a combination of factors, including
the speed at which a SPM performs some tasks, that sometimes led to unexpected
events occurring. If something happened at a particular point — one point put by the
Post Office to Mr Tank was that an outage occurred at a very precise moment — then
an adverse impact would or may occur. That precise combination of circumstances
may not occur with the frequency that Dr Worden stated. His supposition was that a
bug would have a wide impact across many branches with a high frequency, which
would mean that the Post Office would suddenly realise its existence. That is not a
valid supposition or assumption given the evidence in this trial
The Horizon System has certain core components. They include both Reference Data
and Transaction Data. The difference between these is as follows.
Reference Data is effectively data about products and operational elements and is a
critical element of the Horizon system. Reference Data interfaces with a wide range of
Horizon components. Without Reference Data, Horizon would not fully function, nor
could Subpostmasters operate their branches. It informs the operation of the Point of
Sale system at the counter. This is now managed and operated by ATOS. Many of the
business applications in the branches are driven by reference data, and this approach
has advantages over hard-coding the different business applications, as that would
require changes to the code if the applications were to be changed. It is much more
flexible, to manage changes over time and across branches to use the reference data
approach. This is because when changes are made, the reference data is changed, but
not the code.
The integrity of Reference Data is critical for the correct operation of a variety of
systems within the Horizon architecture. The Post Office’s Reference Data
Management Centre (RDMC) supports the loading, storage and release of Reference
Data within the Horizon system. The Reference Data Distribution Service (RDDS)
distributes Reference Data to Post Office branches and other data centre systems. The
Post Office Reference Data Team deals with the delivery of Reference Data and
verification of operational business change through Reference Data. Horizon counter
Reference Data was distributed by the Riposte messaging facility in Legacy Horizon,
although this is not used in Horizon Online.
Mr Coyne considers that there were insufficient appropriate change control processes
prior to 2017, based on a document he has seen which is dated July 2017 which stated
“*. we have now aligned that all Reference Data changes go through the appropriate
change process”. Mr Coyne takes this to mean that prior to this date, all Reference
Data was not going through the appropriate change process, or that change process
POL-0019320
55.
57.
58.
POL00022841
POL00022841
did not exist. In my judgment that is probably reading too much into that particular
entry; however and in any event, change control processes require two things in order
to work effectively. Firstly, they must exist. Secondly, they must be applied.
Some aspects of how business applications were built on Riposte were explained in
the expert evidence, without going into enormous layers of explanation of technical
complexity that were unnecessary. These included the following: Zero-sum baskets
for customers meant that the overall or net impact of all the services provided for one
customer was a sum of money which the customer was required to settle. It was
required that the cash or other money produced by the customer should exactly match
the cost of services provided; therefore, the whole basket of services and customer
settlement had to be what is called “zero-sum”, before the basket could be recorded in
the branch (and then, through Riposte replication, later recorded in the back-office
systems). The impact of every business application would be fed into the relevant
systems at the Post Office, typically overnight. These systems were operated by
double entry bookkeeping. The only way to put postings into the accounting system
was by double entry, which in turn could only be done for zero-sum baskets. Dr
Worden considered this a robustness countermeasure.
Any other actions performed in the branches which had an impact upon the Post
Office’s accounts (which would include stock management, cash drawer
management, balancing and reconciliation) also had to be zero-sum. These could only
be carried out in the branch in packages of updates which were zero-sum, when
summed across different Post Office account codes. This was another instance of a
robustness countermeasure.
All branch applications had to have what was called transactional integrity. This
means that all customer business applications, balancing and reconciliation, cash
management and stock management were built so that any zero-sum package of
updates from those applications would either succeed completely, or would fail
completely. In the latter case, they were supposed to have no impact on branch
accounts. This transactional integrity was given the acronym TIN, and was supposed
to be enforced by the Riposte infrastructure. It should have been impossible in the
event of a failure (such as hardware failure) for a part-completed set of updates to be
recorded in the branch and then replicated to the back-office systems. This was
necessary to prevent the accounting system from being subjected to non-zero sum
updates, which would violate its double entry basis and cause later failures of its trial
balances.
The only exception to this principle was that of so-called ‘recoverable transactions’ -
where some irreversible interaction with a Post Office client system took place part
way through a transaction - so it could not be undone in the case of a later failure. In
these cases, the user on the counter would be guided through a short set of recovery
steps, to produce a consistent zero-sum result which reflected what had happened. As
a comment made here by me for convenience, it is the case that some of the evidence
of fact from SPMs which I have accepted about transactions that gave rise to
difficulty in branches occurred when there were outages. It will be seen from
consideration of the bugs in the bug table that these were not the only instances when
bugs had some impact on branch accounts. However, whether there was a hardware
failure or outage, the system was built so that partly completed transactions were not
supposed to be capable of entering the accounting system. Dr Worden opined that it
POL-0019320
60.
61.
POL00022841
POL00022841
was, as he put it, “of course, possible for the user to make some mistake in these
steps, which may have been unfamiliar. In these cases, the mistake would often be
detected later by a reconciliation process, which would typically lead to a TC. This
robustness measure was a correction of user errors (UEC).” However, that statement
in my judgment jumps straight to a conclusion that user error would be to blame for
failures in recoverable transactions, and this would “typically lead” to the Post Office
issuing a TC. In my judgment, the correct approach is to consider bugs, errors or
defects within the Horizon System and whether these would lead to failures in
recoverable transactions. This is the correct approach to answering the Horizon
Issues.
Many applications were not coded individually but were coded as generic applications
which could be configured to run different specific applications by altering reference
data, as has already been explained. This is called “data-driven software”. These
applications were referred to in Legacy Horizon as 'soft-centred' applications. They
have the attraction of adaptability in that the data can be changed without new code
having to be written. There were however still some hard-coded applications in
Legacy Horizon in any event. Errors could often be corrected more quickly if
applications are driven by reference data in this way. Reference data is much more
concise and understandable than code, so it is much easier to create it or detect errors
in it. Dr Worden considered that errors in the underlying generic code would affect a
set of specific applications, and so should be easy to detect. However, this can be seen
not to be a statement of universal application. The Callendar Square bug was present
for 5 years before it was properly discovered (although its effects were experienced,
to use Anne Chambers terms, “most weeks”).
Reference data could be as simple as lists of available products and their prices
(which clearly might change frequently), or might be more complex - for instance, to
describe the sequence of steps needed to handle some business transaction.
The other important component that can be usefully addressed following Reference
Data is Transaction Data. There are four sources of transactions that make up
transaction data:
1. Those manually entered by a user (which will be a SPM or their assistant) in branch
at the counter;
2. Transaction Corrections (TCs) which are produced by the Post Office to be
accepted by a user in branch to correct discrepancies in the accounts;
3. Transaction Acknowledgements (TAs) which are non-counter transactions and
typically initiate from somewhere else. This is from another area outside the Horizon
System so far as the Lottery is concerned, because the information comes from
Camelot relating to the lottery sales a branch has performed. These transactions are
typically relayed to Post Office/Fujitsu and need “accepting” into Horizon before
forming part of the branch’s transaction data. This is done by means of TAs sent to
each branch. The SPM does not have the option to reject them. TAs did not exist prior
to the Ping fix; and
POL-0019320
POL00022841
POL00022841
4. Fujitsu inserted transactions. These are injected into branch accounts by Fujitsu.
They may be performed in order to ‘balance’ a discrepancy. These do NOT require
acceptance by SPMs in the same manner as TCs and TAs.
Horizon Counter
62. The user in a Post Office branch, the SPM or assistant, will deal with customers and
for each transaction press a number of different icons in order to compile that
customer’s “basket”. As has been seen, different products from different clients can
be provided and sold to the customer in the same basket. These may be paying bills,
prepayment for services, acquisition of licences (such as a TV licence), and money
and insurance management.
63. I The SPM is not concerned that (for example) some of the money collected from a
particular customer may need to go to DVLA (for car tax) or to a utility company (for
a gas bill). It is the Post Office that accounts to these clients, and it does so centrally
and by means of a back-office activity.
64. As well as these counter activities, Horizon also needs to support the periodic process
of balancing and rollover for each branch. Every branch operates in Trading Periods,
which are either four or five weeks (according to a timetable published periodically
by the Post Office). At the start of each Trading Period the branch is supposed to be
‘in balance’. This means that the physical stock and cash in the branch agrees with the
data on stock and cash held in Horizon. Then, during each Trading Period, Horizon
records all customer transactions made at the branch, so it records the changes in cash
and stock. It also records any replenishments or remittances of cash or stock in the
branch. Horizon will (or should) record all changes in cash and stock held at the
branch during the Trading Period, and can compute, from the starting amounts and the
changes, the expected amounts of cash and stock at the end of the period. This leads
to something called the Branch Trading Statement.
65 At the end of each Trading Period, the SPM counts the physical cash and stock in the
branch and compares it with Horizon's data for the same items. This is called
‘balancing’. If the numbers are all equal, the branch is in balance and is permitted to
‘roll over' to the next period. If the two sets of numbers are not equal, there is a
discrepancy between what Horizon considers should be present in the branch (for
example the actual cash) and what is physically present (the actual cash held at the
branch). This also applies to some stock such as stamps. These discrepancies are at
the heart of this group litigation, as the reason for the discrepancies are so
controversial. As part of “rolling over”, a SPM with a shortfall or discrepancy has to
“make good” or “settle centrally” at the time, as part of bringing that Trading Period
to an end. Unless they are prepared to do that, the next branch Trading Period cannot
commence. Each of those two acts, making good or settling centrally, involved
payment by the SPM to the Post Office; the only difference is the former means
payment immediately, and the latter means seeking time to pay.
66. Horizon also supports the activities of replenishing stock such as stamps, and of
replenishing or remitting cash. Accepting a delivery of cash from the Post Office into
the branch is called, by factual witnesses on both sides in the litigation, “remming in”
that cash. This is probably shorthand for remitting or remittance, but all involved in
the case call it “remming”.
POL-0019320
67.
68.
69.
70.
71.
POL00022841
POL00022841
Dr Worden used DVLA as an example of how the back-office works. He stated:
“Across the UK in any day, Post Office accepts a large amount of money from
customers paying their road fund tax. All this money needs to be paid to DVLA.
Therefore, Post Office has a back-office activity - carried out centrally —of summing
all these amounts of money and paying DVLA. DVLA knows how much money it
expects to receive in this way and checks the amount it expects against the amount
calculated by Post Office. This cross-check is an example of reconciliation and
supporting it and reflecting its outcomes are central to Horizon.”
The counter included both hardware and software. Dr Worden described the hardware
in the branches as “not always reliable” but considered there were sufficient measures
built into Legacy Horizon to ensure that hardware failures and communication
failures could not adversely affect branch accounts.
Because of the difference in how data was and is stored in Legacy Horizon and.
Horizon Online respectively, availability of the network was required for the latter but
not the former. In Legacy Horizon, sufficient data was held persistently in the
branches, such that a branch could continue to trade, and could support most business
applications, even if the wide-area network was unavailable. Whenever the network
became available again, Riposte data replication should have ensured that the required
data became available to the back-office systems. The only applications which could
not run in this way were those that required some immediate validation from a client
organisation - for instance, withdrawing cash from a bank account. Horizon Online
this was no longer the case as the persistent data was all stored remotely in the BRDB.
This meant that without a working network, a branch could no longer trade. The
network infrastructure (which is another way of saying the reliability of the
connections to the branch) by 2010 had made this viable.
The role of the Agent layer was to manage communications and translate data
between the representation used in the branches and the network on Riposte, and the
tepresentations used in the Host layer. It is the Host layer that takes data from external
clients. This is explained in the main design document for Legacy Horizon:
‘The systems at the Host Layer can provide permanent storage for information if
required by the application’s business rules. The Host systems can accept data from
external Clients, and translate a file-based view of this information into discrete
transactions or “messages”. These are then passed to the Counters via the Agent and
Correspondence Layers. Similarly, messages received from the Counters are
translated back into a file-based view for transmission to the external Clients.’
Each of the different business applications in Legacy Horizon was typically tied only
to the different Post Office client in respect of whose business it was concerned. For
example, Camelot does not need to communicate with the DVLA. Transactions in
respect of them may be transacted in the same basket, but this facility is separate to
the applications that deal with Post Office/Camelot business and Post Office/DVLA.
Accordingly, each different business application in Legacy Horizon can, in Dr
Worden’s words “be regarded as a vertical ‘slice’ though the diagram and is largely
independent of the other slices”. He categorised this as “another example of
robustness through architecture” but again, it could be seen as a design element.
POL-0019320
72.
73.
74,
7S.
76.
POL00022841
POL00022841
Visual Basic, C and C++ programming languages were used in the Legacy Horizon
system along with Oracle development tools. Riposte, the product developed and
marketed by a company called Escher (and which featured in many PEAKs) was used
in Legacy Horizon too.
The Horizon Online counter and BAL (Branch Access Layer) were developed in Java,
while the agents and host modules remained in C and C++. C is a general purpose
code, an imperative procedural language which was invented by the late Dennis
Ritchie. He was also one of the co-creators of UNIX with his colleague Ken
Thompson. Neither of these gentlemen are household names in society generally in
the same way as, for example, Bill Gates of Microsoft or Steve Jobs of Apple, but are
very well known in the computer software world. Both C and UNIX are extremely
important software technologies which are the heart of a great many, if not almost all,
software products used today. Together Mr Ritchie and Mr Thompson won the Turing
Award in 1983, a prize with similar status to a Nobel Prize, which is awarded by the
Association for Computing Machinery. All of Java, C and C++ are well known
languages that are in widespread use.
Horizon Online was not a completely new system; it built upon, and continued to use,
certain elements of Legacy Horizon. The move away from Riposte and Visual Basic
to Java brought in more modern development tools and enabled techniques such as
object orientation to be used. Object orientation means that modules must be self-
contained, and will communicate via pre-defined and documented interfaces. These
interfaces are not dependent on the application’s physical implementation. Third party
and commodity products (such as operating systems and device drivers) are simply
bought in by Fujitsu. Mr Coyne stated that “Horizon is therefore ultimately a
composition of many sub-contracted components”, which I consider to be a fair
description.
Applications are developed by suppliers (who supply generic software or products),
third parties who supply applications that meet Horizon requirements, Fujitsu
developers who develop software to meet specific goals, or the Post Office. If the
latter, this is done by configuring standard components. As at the date of the expert
reports for the Horizon Issues trial, the architecture is data driven such that the Post
Office could introduce new clients without reference to Fujitsu. All of the different
applications must however integrate with the other existing applications, as well as
any other new applications that are added. The complexity of the system will
therefore have increased, year on year, as further products and applications are added.
Once an application has been developed, it must be tested and evaluated, and then the
process of integration takes place. It will then be released (which means introduced.
into live) and maintained thereafter.
The Horizon system includes an audit database. This is an accurate record of any
activity which can affect the branch accounts. These records are available to be
consulted in the event of any discrepancy arising anywhere in Horizon (for instance,
due to a bug in some other Horizon application, or some operational error in running a
batch process, or a dispute about what data was entered at the counter) but the parties
could not agree on whether and when this should be done. The audit database is a
robustness measure of a secure kernel (SEK) which also involves redundant data
storage (RDS). In Legacy Horizon, all data travelled from the counter through the
software application at the branch, through Riposte data replication to the two
POL-0019320
77.
78.
79.
80.
81.
POL00022841
POL00022841
campuses, then through the Audit Agent to the Audit Store. In Horizon Online, all
data travelled from the counter through the software application at the branch, through
communications hardware and software to the Branch Access Layer (BAL), into the
BRDB and then nightly to the audit store.
The Post Office currently has several hundred client organisations, which shows the
diversity of services and products available in a branch. This also makes it obvious
that, for most of these clients, the service provided through Post Office will be
different in nature from the service provided by the Post Office to its other clients, so
some unique software functionality must be provided within Horizon both in the
branch (for the counter) and also the back office to support the activities for each
particular client. This is a part of what makes Horizon such a large and complex
system.
Checks are built into accounting systems. These can be by way of checks on data
entry, double entry bookkeeping checks in two or adjacent layers, consistency checks
and defensive programming.
As I have already identified in Judgment (No.6), Dr Worden grouped
countermeasures including within the Horizon System with other characteristics that
are not properly described as that, together with those (such as manual checking by
SPMs) that are not properly described in that way.
In a Post Office internal presentation document dated 7 December 2009, the following
passages appears in relation to “Horizon - Current State”. I consider it an apt
description of Legacy Horizon:
“CURRENT STATE
Before discussing the future development of Horizon, lets remind ourselves of the
‘system’ we have today
OVERVIEW
Horizon is 13 years old and the design criteria/technology was different and the
business needs were different - the demand for on-line working was low, the telecoms
were expensive for on-line. Horizon was built to support off-line trading.
Very high security (e.g .. user ID/ access/ screens encryption) this was a requirement
from Gov because Horizon was built to support benefits payments.
Horizon is the 2nd most secure system in Europe. The MOD being #1 !
Horizon is a costly system. For example Horizon is a bespoke system that uses a
different encryption. Link for instance is unable to decrypt the encryption we have, so
we have to decrypt before sending!
Horizon is also a system that is wrapped up in ‘barbed wire' - making changes difficult
and costly - test everything!
As time has passed and more product have been developed Horizon has evolved -
from a technical perspective essentially we have bolted things on the side - we undo
the barbed wire stick a bit in then wrap everything back up.
Design was optimised at the time to minimise costs (esp. network) - offline working.”
(emphasis added)
The following headline points emerge from this:
1. Horizon was originally designed to support benefits payments to welfare claimants.
POL-0019320
82.
83.
84.
POL00022841
POL00022841
2. Horizon was already old in 2009. Although there were some major changes made
when it became Horizon Online from Legacy Horizon, Horizon Online still used
major components of the existing system. It is now somewhat older than it was in
2009.
3. It was originally built for off-line trading.
4. It has a relational database at its heart, but very numerous other applications that
have been bolted on to it over the years as the Post Office’s business has evolved.
5. There are a very wide number of other computing companies involved in the
evolution of this system, not only in terms of software. Oracle, Escher Group,
ICL/Fujitsu, ATOS, Computacenter and many more. It is a bespoke system that uses
different encryption to other systems, such as Link.
6. The complexity of the different interfaces, as a result, is very high.
There have also been a total of some 19,842 release notes (in relation to software
changes) in the life of Horizon. This is consistent with each of these notes being a
change to the Horizon system.
Fujitsu is the entity that was responsible for Legacy Horizon (putting to one side any
contractual nuances or differences between the Post Office and Fujitsu that may have
existed, which are no part of this litigation) and it remains the main partner with the
Post Office for Horizon Online. However, other suppliers are also now involved. Atos
provides first line support via the Service Desk; Computacenter supplies the
hardware, and Verizon is responsible for networking (what might also be called the
connections between the different parts of the system). The main element within
Fujitsu that was referred to in the evidence was SSC, in particular 3“ line support, and
“Development”. SSC also serves other Fujitsu customers.
The Post Office Account — headed by Mr Parker — is responsible for developing and
maintaining Horizon, deploying it to branches, managing the operation of the system
and supporting users. The organisation of the Post Office Account is shown by the
following graphic:
POL-0019320
85.
86.
87.
88.
POL00022841
POL00022841
Past Office Account - Organisation
Business
Support
Fonctons
After the project to introduce the Payments Benefit Card in 1999 (the tri-partite
project to which I have already referred) was abandoned, the Post Office and ICL
moved to automate post offices and this was the introduction of what is now called
Legacy Horizon. In August 2000 there was the Release of what is called the Horizon
Core System, or Core System Release. This introduced Automated Payment Smart
cards and APS/TPS reconciliation into branches. Not all branches received
installation of Horizon in the same month; that would, I imagine, be impossible in
practical terms given the number of branches in the Post Office network. Some dates
are included in Judgment No.3 from the Common Issues trial, in relation to those
claimants called as witnesses in that other trial. Horizon was installed in Mr Bates’
branch in October 2000; Mrs Stubbs said it was installed in her branch “in the course
of 2001” but it must have been earlier than that, as letters in relation to her shortfalls
both to, and from, Mr Manning are dated 1 and 2 November 2000. The precise date of
introduction of Horizon at each of the branches in the Post Office network is not
relevant.
An upgrade of Horizon was performed in February 2001, which is called Maintenance
Release M1. The prime purposes of the upgrade were the enhancement of the CSR+
Applications (APS, LFS, EPOSS, EPS, OBCS), enabling of the AP client variable day
file transmission, enhancement to Reference Data products and minor changes to
TIPAIS Pathway generated CPs to improve operability of the system.
A summary of some of the Releases appears in the chronology milestones for Legacy
Horizon below at [96] in the list of chronology milestones.
Horizon included what is called an Electronic Point of Sale System or EPOSS, which
refers to the part of Horizon within the branch. The parties referred to this as activity
at the counter, or simply “counter”. This term comes from the counter in the branch
where a customer is served. In addition to this part of the system, there was another
element which the parties (for shorthand purposes in a sense) referred to as the “back
POL-0019320
89.
90.
91.
92.
93.
POL00022841
POL00022841
office” function. This refers to the way in which the Post Office reconciles its
transactions with its own clients.
EPOSS must allow the counter staff to record that some goods have been provided to
a customer, compute the price of those goods, and allow the customer to pay the
money required for all their purchased goods, for instance by cash or a credit card. Ifa
customer wants to carry out two or more different activities in one visit to the counter
- for instance, to settle a bill and to buy some stamps - Horizon does not require the
customer to settle the amount in two separate pieces. Horizon has the concept of a
customer carrying out a ‘basket! of activities and settling the total amount due for the
basket in several ways - by one credit card transaction, by a cheque, by cash, or by a
mixture of these.
At the stage when Horizon was introduced to branches — and indeed, this was a
fundamental distinguishing feature of Legacy Horizon, compared to Horizon Online —
the data was held in branches on terminals. Escher Group, a separate company,
provided the Riposte Message Server (which was responsible for storing all the data
in the Post Office branches and replicating it to the Data Centres). Terminals had two
discs within them, one of which was termed a “mirror disc”, which retained all the
data of what occurred on that terminal. The branch would have a hard disc, which
would store the data for the branch and this would also be replicated on other counter
positions (if a multi-counter branch) or, in the instance of a single counter branch,
stored to additional external storage on the counter. It would then be passed on from
the counter to the Horizon data centre where it is stored in the CS messagestore.
In the summer of 2000, a ‘proof of concept’ was undertaken to investigate the
integration of internet technologies within the then-current Horizon System (which
was being/about to be rolled out) to support the delivery of banking transactions. This
was called the Network Banking Solution. A primary facet of the Network Banking
Solution was the delivery of the banking transactions within the already established
Escher WebRiposte environment. The full installation and integration of this was the
task of Fujitsu (at the time ICL Pathway). WebRiposte was a message passing
technology from Escher Group that extended the functionality of Riposte Message.
IBM were selected for the supply of the Network Banking Engine which was
designed to handle the interface between the Horizon System and the agreed Financial
Institutions; these are also called Post Office’s clients, although in some documents
they are referred to as ‘External Clients’. This was described by Mr Coyne in the
following way: “This communication was initially based upon Escher’s Riposte
messaging system. This was later supported by the WebRiposte system to
accommodate Network Banking changes and the Network Banking Engine supplied
by IBM to deal with the increase and change in type of business transacted in the
branches. This was subsequently migrated to Horizon Online. Horizon is therefore
ultimately a composition of many sub-contracted components.”
It can therefore be seen that Horizon Online, although it includes some major
differences (for example where the data is stored; this is no longer in the branches as
with Legacy Horizon) built upon and adapted the existing functionality of Legacy
Horizon. It was not an entirely new system.
POL-0019320
94,
95.
96.
97.
POL00022841
POL00022841
Mr Coyne opines that the introduction of network banking (as well as developments
in the EPOSS function) brought with it a more complex enhanced architecture within
which further systems to ensure transaction integrity and reconciliation could be
imposed. However, he also considered that Horizon “originally stemmed from an
inherited system and architecture with an initial, fundamentally different design
requirement” (namely the Pathway Benefits Payment Card project). Regardless of
that, there is no doubt that Horizon is a highly complex and multi-faceted system,
because it has so many different elements and interacts with so many other systems. It
has been added to multiple times over the years, as clients are added, new products
are introduced, and indeed other sub-contracted parties and software products have
become involved. I do not consider that resolution of the Horizon Issues requires an
analysis of the reason or reasons why, in historical terms, Horizon may have
developed (or had inherent within it) any bugs, errors or defects.
The Horizon System that was initially installed in the branches in 2000 onwards is
Legacy Horizon. ICL Pathway was (at least partly) owned by Fujitsu and I use the
term Fujitsu to refer to the computer specialists with whom Post Office had
contractual relations in respect of Horizon. I therefore make no distinction between
the years when ICL Pathway or Fujitsu was or were contractually responsible for
Legacy Horizon, or indeed with any of the other computer specialist companies such
as Escher Group who provided Riposte. Legacy Horizon involved the installation and
use of Horizon terminals within branches, which were used by the staff there
(including the SPM) to transact all of the branch’s Post Office business with the
branch’s customers. If a customer in a branch wanted to purchase something from the
associated retail outlet, they would transact that business at a separate till, not at the
Horizon terminal. The National Lottery is an exception to this, as certainly now, it
spans both parts of the business of a branch.
Mr Coyne recited some “chronology milestones” for Legacy Horizon, which he
explained were not exhaustive but which he considered were milestone dates. Dr
Worden also drew attention to changes during the period 2000 to 2010 which he
described as significant changes.
This chronology included the following:
1. Pathway (later Fujitsu Services) awarded contract for Benefits Payment Card May
1996
2. Horizon Pilot 1996
3. Pathway cited “greater than expected complexity” and “...major implications for
the degree of difficulty of the project” which ultimately lead to failure of the project.
4. Post Office Counters Ltd and Pathway sign agreement to utilise the project to
automate Post Offices July 1999
5. Horizon rollout 1999 — 2002
6. Core System Release - This included the introduction of Automated Payment Smart
cards and APS/TPS reconciliation. August 2000
POL-0019320
POL00022841
POL00022841
7. Maintenance Release M1 - Prime purposes of the upgrade were the enhancement of
the CSR+ Applications (APS, LFS, EPOSS, EPS, OBCS), enabling of the AP client
variable day file transmission, enhancement to Reference Data products and minor
changes to TIPAIS Pathway generated CPs to improve operability of the system
February 2001
8. S04 Release Additional functionality on the Horizon Pilot outlets to permit the
printing of forms approx. July 2001
9. $06 Release Day D rectification measures - this included a new automatically
generated broadcast message to detail when counters at an outlet were offline. This
was to be implemented in a staged manner and included a receipts and payments fix
June 2001
10. S10 Release Data centre and counter upgrade introduced unattended reboot
facility at the counter September 2001
11. B11 Release of the first network banking release, changed the version of Tivoli
used by the whole estate in approx. December 2001
12. $11 Release January 2002
13. B12 Release June 2002
14. $20 Release September 2002
15. B13 Release in approx. September 2002
16. In 2003 there were different changes introduced. Network Banking was
introduced at this time, as were the Data Reconciliation Services (DRS) and debit card
processing. There were also the $30 and S50 releases which immediately follow this
entry.
17. S30 Mails Application /Escher Mails 3.3 package (1 Feb 2003)
19. $50 Release October 2003
20. S60 Release Approx. February 2004
21. S52 Release March 2004
22. S70 Release October 2004
23. S75 Release (containing changes to support the changeover to use of NBX
banking agents in approx. October 2004
24. IMPACT Programme
25. POLFS (a SAP-IS Retail System) was implemented in 2004
26. S80 Release Jan 2004 - Aug 2005
POL-0019320
98.
99.
POL00022841
POL00022841
27. BI3 S80 T&T Harvester Agent accommodation was introduced. Mr Coyne
thought that this may have been approximately Nov 2004.
28. POLSAP rationalisation (rationalisation of disparate systems SAPADS and
POLFS) between 2007-2009
28. Pension & Allowance Order Books were replaced by the Post Office Card
Account in 2005. The Post Office had always had business relationships with banks
such as Girobank.
29. $90 ‘Bureau Plastic’ accommodation introduced in January 2006
30. $92 Release March 2006
31. T10 Release August 2006
32. T40 Release January 2007
32. AP/ADC was introduced in around 2007/2008.
In around 2010, POLFS and SAP ADS were merged to make POLSAP.
Horizon was migrated to Horizon Online in 2010. Horizon Online is also called HNG-
X, but now it is on a different platform since its move to Windows 10 and it is now
called HNG-A. The letters “NG” stand for “New Generation”. In 2010 there was no
sudden change in the range of business applications supported by Old Horizon in the
branches. This range of applications has increased continually over the lifetime of
Legacy Horizon and Horizon Online. The main reason or rationale for the change to
online was to exploit advances in the underlying communication technology, and
improvements in its reliability. This meant that it had become possible to store all
persistent data at a centre rather than in the branch (on the counters), with the
consequence that a branch could only operate when communications were available,
but the risk of failed communications was significantly lower in 2010 than it had been
10 to 12 years before. Dr Worden considered that this change mirrored the wider
changes across the IT industry, where increased reliability of communications
infrastructure meant that applications could be dealt with in this way.
The centralised storage of transaction data allowed the architecture to become simpler
in some respects, with simpler management of the branches in the event of hardware
failures or replacements and other events. This is because in those cases branch data
would not be lost. There was also no dependence on Riposte data replication, which
meant that Riposte could be removed entirely. Riposte is therefore not used in
Horizon Online. One feature of Horizon which emerges from all the evidence
provided is that it had a difficult interface with the systems of the Post Office’s
clients, such as Camelot. It is not necessary to go into the technicalities of why that
was, and was recognised by the Post Office (in this instance by Ms Van Den Bogerd
herself) so far as Camelot was concerned by the introduction of the Ping fix in 2012.
Horizon Online
100.
A document entitled Counter Business Architecture document of 4 August 2017
stated that “The objective of the HNG-X programme is to develop a system with
POL-0019320
101.
102.
103.
104.
POL00022841
POL00022841
structural and operational characteristics that substantially reduce ongoing support
and maintenance costs with respect to the current Horizon system” and also “The
overall requirement is that the business capabilities offered by the current system
(Horizon) are preserved in the new system (HNG-X). However, a limited number of
business capabilities will be revised based on a joint optimisation of business
requirements and system properties.”
The fundamental change was that in Horizon Online, no transaction data was held in
any persistent form in the branches. The same document explained that storage of
transactional data within counters in the branches led to the need for security
mechanisms that affected “both the structural complexity and operational
performance of the Counter Business Application.”
In Horizon Online, before completion of a basket of customer services, that basket
was transmitted and had to be acknowledged by the BRDB and the basket could not
complete successfully at the counter until that had happened. This is why Horizon
Online could not operate in the branches without working communications. The main
difference at the “back-end” was the existence of the BRDB, which was the main
persistent store of all transactions for all branches. Many business applications in the
back-end were unchanged (and were referred to as legacy’), except for the need for
them to interface with the BRDB rather than with the previous Agent layer. Other
copies of transaction data continued to be stored in those applications.
The previous branch architecture had been based on Riposte, which provided
functionality on different levels. These included user interfaces, some business
applications, and message storage and replication. Because in Horizon Online,
Riposte was completely removed, all elements of the branch software were replaced.
A great deal of the Legacy Horizon branch code had been written in Visual Basic, for
Horizon Online nearly all the branch software was written in Java. Java is a newer
language, and supported more modern programs (or what Dr Worden called modern
programming paradigms). This allowed a more modern software architecture in the
branch, which did not have to be fitted around the existing architecture of Riposte.
The new architecture is shown in the following diagram:
POL-0019320
106.
107.
POL00022841
POL00022841
Physical Architecture
Counter Data Centre
Presentation Peripherals
virtuatteation
UE Moget
leg Wire Hirwery. Sone Fw, Kayooare, Penta
Counter Domain Objects 7
tg. Cees, cuttoreae nace, cayment, nenort oduct Tarcatton, ernie)
Business Process Ovjects Business Data Onjects.
eg APADS serge) eg Retererce Sey
Remote Sersoes
toca tartene Gp DAA PAD.. Toure
se mB hashiet ong te ee surtove, Resets enaecor}
= OOOO < Q
NANA O>
. 1
1 \
1
I
I
i
The top 'Presentation' layer is the one responsible for displaying information to the
user and for accepting the inputs. The next 'Interaction' layer provides the foundation
or building blocks for this interaction, such as menus. The effect of these two layers
was to provide a user interface similar in style to that which had been provided by
Riposte in Old Horizon. This therefore meant that the experience of the person, either
SPM or assistant, using Horizon would be similar to what it had been in Legacy
Horizon, but the system would be using Java technology, rather than Riposte as
before.
The ‘Business’ layer provides the functionality of the different business applications,
which is done in what is called an object-oriented fashion. This means that there are
several general-purpose software objects (i.e. modular blocks of software) with names
such as customer, basket and payment, which represent the behaviour that is required
of those in the real world, but is done in a way that can be easily reused in many
different business applications. There are therefore certain core design elements used
for many different applications.
Business process objects and business data objects are more specialised to support the
many business applications. Business process objects support the sequence of steps
which make up a business process, and the business data objects hold the necessary
data, which is presented at the counter or stored. As has already been noted, this is
generally not done by writing completely different software for each business
application, and a great many applications are driven by reference data. This data
defines the sequence of steps in completing each type of service for a customer. This
reference data-driven style of software is common modern practice. The theory is that
POL-0019320
108.
109.
110.
1.
112.
113.
POL00022841
POL00022841
new applications can be supported just by adding new reference data, rather than by
writing new software. There are different capabilities in the business layer which it is
not necessary to list.
The Services layer provides a set of software objects which provide services in
support of many business applications. Most of these services deal with organising
information and sending it for storage in the BRDB. The facilities for ensuring that
each basket is zero-sum before it is sent, and transactional integrity are provided
generically in the services layer. These functions do not therefore need to be coded
individually in the business objects, and do not have to be built individually into any
new business application. The facilities for zero-sum baskets, which is one of the
ways of complying with the requirement for double entry bookkeeping, and
transactional integrity are provided generically in the services layer.
The Process Engine is a key component of the Services layer, and provides a
simplified way for the counter to provide services which involve a sequence of steps
The sequences of steps need not be defined in Java code but are defined in a
specialised Process Definition Language (PDL), which is executed by the Process
Engine. PDL was developed for Horizon Online by Fujitsu. The use of PDL means
that complex sequences of steps are simpler to define and test. This is an example of
generic data-driven software.
Some data are stored persistently on the branch hardware (as shown by the disc-
shaped boxes in the diagram) however, these data do not include customer transaction
information. These include business process definitions (definitions of sequences of
steps in a process), other reference data, data defining reports that can be output in a
branch, and other information required to support operations. The reference data is
refreshed daily from the data centre. There are services which provide these data to
the other layers in forms that are convenient for them to use.
The Services layer of the branch architecture is the only layer which communicates
with the data centre, through the communications subsystem. The purpose of this
layered approach is to provide each kind of functionality (such as reliable and robust
communication with the data centre) in one layer only, and not have to reinvent it for
many different business applications. In effect, the services layer in Horizon Online
now provides many of the services which were formerly provided by Riposte.
The Services layer also provides interfaces for online services, where it is necessary to
contact another non-Post Office IT system in order to provide the service to a branch
customer. These online services include banking, the use of credit/debit cards, mobile
phone top-ups and other services.
Both the Branch Access Layer (BAL) and the BRDB are completely new elements of
the Horizon Online back-end. The principal function of the BAL is to exchange
messages with the counter software in the branches. It also has a role in checking that
the information is conformant, logging of all exchanges, and recovery from many
kinds of error conditions. Because it has to handle more than 25 million transactions
per day, the BAL has many design features to ensure high performance (principally
by distributing the load in parallel across many machines), as well as robustness - for
instance, through reliable and redundant hardware.
POL-0019320
114.
115.
116.
117.
118.
POL00022841
POL00022841
The branch database or BRDB is a large, high-performance Oracle database whose
main function is to store all customer transactions which originate in any branch. It
also has features to ensure high performance and robustness. These are, for example,
through the use of transactional integrity and recovery.
There are other notable milestones in my judgment of the Horizon System which are
not included in the list above at [96] include the following:
1. Migration to Horizon Online in 2010.
2. The discovery of what has become called the Callendar Square bug. This was
experienced and reported in that branch in 2005. Mr Godeseth’s evidence was that it
had probably been in the Horizon System since 2000. Ms Chambers’ entry in one
particular PEAK in February 2006 was that “this problem has been around for years
and affects a number of sites most weeks.” The correct place to put it in any
chronology, on the basis of the evidence currently before the court, is 2000.
3. The introduction of what was called the Ping fix, which was in relation to the
accounting with Camelot for the National Lottery, which was initiated in 2011 and
occurred in 2012. Much evidence was given by Ms Van Den Bogerd, both in the
Common Issues trial and the Horizon Issues trial, about the improvement in accuracy
of accounting regarding Lottery sales since the Ping fix was introduced.
4. In June 2014 ATOS started to provide 1“ line support. The management and
operations with regard to Reference Data has been outsourced from within Post Office
control to ATOS since the same date.
5. The move from HNG-X to HNG-A (which uses Windows 10 rather than Windows
NT4). This change to Windows 10 occurred in February 2017 and the roll out was
administered by Computacentre.
Having provided what is essentially an overview of the Horizon System, both Legacy
and Online, it is convenient to turn to the main battleground between the parties
concerning the subject matter of the Horizon Issues trial. This is the presence of bugs,
errors and defects within both Legacy Horizon, and also Horizon Online. This can be
done most conveniently by retaining the numbering of the different alleged bugs from
what was called “The Bug Table” in the 2° Experts’ Joint Statement.
Submissions and Evidence
There were two features of the Horizon Issues trial which were somewhat unusual.
This was a lack of detailed evidence adduced by the Post Office in relation to the
specific factual matters recorded in PEAKs and KELs in particular, and the way that
this situation was sought to be remedied. This second element was by way of points
explained or put “on instruction”, and also in submissions. This is referred to in [69]
to [71] of the judgment that accompanies this appendix. It is an obvious point that
submissions are not evidence, and vice versa.
The first of these, the putting of points “on instruction”, in this case related
predominantly to the way that certain matters were put to Mr Coyne in cross-
examination to correct evidence of fact that had emerged during the cross-
POL-0019320
119.
120.
121.
POL00022841
POL00022841
examination of Mr Godeseth. The second arose in the context of the Post Office’s
Appendix 2 to its written Closing Submissions. The need for both, or either, arose in
my judgment because of the way Dr Worden approached his task in comparison with
Mr Coyne. Because he had done what he himself described as a “high level” analysis
of Horizon, his written reports did not deal with the detailed contents of so many
PEAKs and KELs in the same way that Mr Coyne did. Therefore, when the Post
Office came to challenge Mr Coyne’s conclusions in closing submissions, they could
not simply meet his evidence with opposing expert evidence of the same type.
Lengthy written opening and closing submissions were provided by both parties, as is
usually the case in modern and complicated cases. The Post Office’s Closing
Submissions in particular were very lengthy, and included appendices, one of which,
Appendix 2 at Section K2, itself ran to some 137 pages. This document attempted to
collate material relevant to the Bug Table, in a logical form, dealing with each bug in
numerical order. It identified key documents, expert evidence (usually from Mr
Coyne), entries in joint statements by the experts, and other sections including
“relevant discussions during trial.”
During the oral closing submissions, the claimants identified certain instances in
which positive evidential assertions were made in Appendix 2 that were not, it was
said, contained in the evidence. Particularly given Appendix 2 was so large, it was
simply not possible to resolve these objections during that oral hearing. I therefore
directed that the claimants serve a schedule identifying all such passages to which
objection was taken, so that a response could be provided by the Post Office. This led
to a “clarification” schedule served by the Post Office, which stated in respect of each
challenged passage the following:
Whether the passage was based on evidence in the trial, and if so the reference; any
televant documentary evidence; if documentary, whether that document was deployed
in the trial and if so when; and/or whether the Post Office relied upon what Mr De
Garr Robinson submitted would be a suitable response in some cases, namely
common sense.
That clarification document was itself rather lengthy. It contained a further 36 pages,
with 175 different entries. Of those entries, 115 were to documents, and in only 28
cases had those documents been referred to at the trial. Further, some of the
submissions went rather further that the documents supporting them justified. For
example, on 8 Recovery Issues in the Bug Table (which was not accepted as a bug)
the Post Office has submitted the following in relation to a necessary software fix:
“Tt went to the Model Office on 1 June 2010 and Fujitsu have advised that it would
have been rolled out to the rest of the HNG-X pilot by mid-June.”
The supporting reference was to a document, PEAK PC0199000. However, the
submission “Fujitsu have advised that it would have been rolled out....” reads as
though Fujitsu have provided evidence, behind the scenes, and not called at the trial,
about what would have or did happen. Further, the supporting entry in the PEAK
relied on for the entries for 1 June 2010 and throughout that month are as follows:
“Date:01-Jun-2010 14:05:40 User:John Budworth
POL-0019320
122.
123.
124.
POL00022841
POL00022841
Pre-Requisite Reference Data and Data Centre BAL upgrades have been applied to
live.
Please one-shot to the distribute and commit to Model Office branch 699010 by 15:30
today.
Please distribute tonight to the 50 branches in Pilot 50 RNT9566.xls attached as
evidence.
Date:04-Jun-2010 17:20:16 User:John Budworth
Committed to Pilot 50 RNT9566.xls by MSS on Thursday June 3rd as verbally
requested by RM.
Date: 14-Jun-2010 11:57:50 User:John Budworth
Please distribute and commit to the balance of the HNGX estate
Date: 14-Jun-2010 11:58:11 User:John Budworth
Evidence File Updated - RNT9566.”
This entry reinforces my view that the submission goes a little wider than the
document’s contents suggests. However, this is a rare example, and in any event I am
entitled to draw an inference (that is, a common sense conclusion) that if the PEAK
requests distribution to the entire estate, that did occur. For the most part this type of
example is rare, and it is not usually the case that the submissions go further than the
underlying documents suggest. I would like to record that I found Appendix 2 very
useful; it must have taken an enormous time to produce. There are some omissions
within it; for example, Dr Worden’s cross-examination is not always referred to.
However, the task of producing this appendix to the judgment has been rather easier
as a result of it.
In order to use it however, it was not necessary to make a ruling on each and every
one of the 115 different passages challenged by the claimants and the responses to
those challenges by the Post Office. I have only referred to the actual material
necessary in order for me to come to findings on the Bug Table, although I have
considered the full content of Appendix 2 and its accompanying clarification
document. I have not given documents that were not referred to at all in the trial the
same weight as those that were, and I have not given those that were, the same weight
as the actual evidence that was deployed in the trial. I have however considered them
all.
Bugs, errors and defects and their symptoms
The first three of the Horizon Issues have the heading “accuracy and integrity of
data”. Given the purpose of the Horizon System, both Legacy and Online, and the fact
that it was the accounting mechanism by which branch accounts were produced in
terms of the Branch Trading Statement, the presence of bugs, errors and defects with
the potential to cause apparent or alleged discrepancies or shortfalls in branch
accounts was an extremely important element of the Horizon Issues trial. This is
plainly included within Horizon Issue 1.
The degree to which the Branch Trading Statement accounts did or did not, in law,
have the effect of settled accounts between principal and agent was addressed in
Judgment (No.3) on the Common Issues. The majority of the Horizon Issues — and
POL-0019320
126.
127.
POL00022841
POL00022841
certainly the first three - were all aimed at the same principle, namely (in summary)
that of computing and accounting accuracy of the Horizon System in both Legacy and
Online form. It was doubtless for this reason that so much of the cross-examination of
the two experts dealt with the presence of bugs, errors and defects.
In my judgment, the most convenient approach is to consider each of the alleged bugs,
errors and defects in the Experts’ so-called Bug Table in order to come to a judgment
on the existence, or otherwise, of each. Horizon Issue I requires this, and in addition,
Horizon Issue 3 should be answered in the context of whatever the answers are to the
different component elements of Horizon Issue 1. This also means Horizon Issue 3
can be considered in the knowledge of how many, if any, of the alleged bugs errors
and defects in fact existed in Horizon over time, and what their potential effect was.
By the end of the Horizon Issues trial, in reality there was less in dispute between the
parties on the different bugs in the Bug Table than appeared at the beginning of that
process. This is not to say that there were many concessions, certainly in terms of the
existence of bugs, although there were some. However, the expert evidence of both Dr
Worden and Mr Coyne, although they differed in their conclusions, was not so far
apart on the factual existence of certain bugs, errors and defects. Given the use during
the trial of so many PEAKs and KELs, which contained contemporaneous records at
Fujitsu of the effect upon SPMs of different features within Horizon, this is not so
surprising. Some overarching themes remained, for example the difference between
transient and lasting impact of bugs. These are addressed in the body of the judgment
itself.
Turning to the 29 different entries in the Bug Table in turn, and in summary terms
only (as to reproduce references to all of the documentation, evidence and
submissions on each would lead to a very lengthy analysis of each such bug, which in
my judgment is disproportionate and not necessary), my findings on these are as
follows. Because I am using the same numbering as in the bug table, the following
bugs are not dealt with in this appendix in chronological order, but in the order chosen
by the experts in their joint statement.
1. Receipts and Payments Mis-match bug.
128.
129.
130.
This bug occurred in 2010. It is a bug which appeared in Horizon Online. It was
agreed in the Bug Table as an “acknowledged bug” which had an impact on branch
accounts. Dr Worden added in his comments that “Therefore, the extent of this bug is
well established, in the GJ analysis.” GJ is Gareth Jenkins. It is accepted by the Post
Office in paragraph 7 of Appendix 2 as one of a number of “bugs with lasting impact
(although they were resolved)”. In the summary in Appendix 2, the Post Office stated
that “In the event, however, this bug resulted in transient impact only.” I have dealt
with the concept of transient impact in the main judgment on the Horizon Issues.
The effect of this bug was identified upon the accounts of approximately 60 branches.
Dr Worden and Mr Coyne, in the agreed entry in the 2"4 Joint Statement, had stated
that “the number of distinct bugs, for which the experts have seen strong evidence of
the bug causing a lasting discrepancy in branch accounts, is between 12 and 29.”
(emphasis added)
Dr Worden had therefore seen “strong evidence of lasting impact” in respect of this
bug. This was one of his 12. The same joint statement said it had affected
POL-0019320
POL00022841
POL00022841
“approximately 60 branches”. Mr Coyne in his report used the more precise number
of 62. Of these 62, two branches were affected twice.
131. The bug related to the process of moving discrepancies into the local suspense
account. The majority of incidents are recorded as occurring between August and
October 2010. The bug was documented in a report from Mr Gareth Jenkins dated 29
September 2010 where it was stated:
“This has the following consequences: There will be a receipts and payment mismatch
corresponding to the value of discrepancies that were lost. Note that if the user doesn't
check their final balance report carefully they may be unaware of the issue since there
is no explicit message when a receipts and payment mismatch is found on the final
balance (the user is only prompted when one is just detected during a trial balance)”
(emphasis added)
132. This issue is reported as causing discrepancies that disappeared at the counter or
terminal when the branches followed certain steps, but which persisted or remained
within the back-end branch account. It is therefore something which is contrary to the
principle of double entry bookkeeping, and should plainly not have occurred. The
issue occurred when a branch cancelled the completion of the trading period and then,
within the same session, rolled into a new balance or trading period. Because the
discrepancy disappeared at the counter, the SPM would not know that the discrepancy
existed.
133. The lack of an explicit message to the user, in my judgment, simply compounds the
problem. This bug was acknowledged by the Post Office in its letter of response to the
letter of claim sent in relation to the litigation. That letter of response was dated 28
July 2016. I am unaware of any acceptance by the Post Office of this bug prior to that
date, but in any event that is not relevant to the substantive issue of answering the
Horizon Issues.
134. The Post Office in its Closing Submissions stated that this bug only occurred if there
was “an unusual sequence of events”, namely as follows. If a branch had an
unresolved discrepancy, when balancing the stock unit, and if the SPM pressed the
Preview or Print button to produce the Trial Balance Report, this would cause the
counter to return to the rollover screen. Having checked the Trial Balance Report, if
the SPM then pressed the rollover button, Horizon would then ask the SPM whether
they would like to transfer the discrepancy to the local suspense account or cancel the
rollover.
135. If they chose to cancel the rollover, this would cause the system to return to the
rollover screen, allowing the SPM various choices, including cancelling the rollover
process or the choice of again pressing the Preview or Print button to produce the
Trial Balance Report. If the SPM then proceeded to rollover to a new Trading Period
in the same session the fault would occur. This was a bug in the code that was
triggered by the ‘cancel’ button being pressed and this incorrectly caused the
discrepancy to be cleared so far as the user in the branch was concerned.
136. I do not accept that this was a particularly unusual sequence. If there was a
discrepancy, I do not see why it is suggested it would be unusual for a SPM to decide
to cancel the rollover. This is, in my judgment, something which could be expected to
POL-0019320
137.
138.
POL00022841
POL00022841
happen sometimes, and plainly the absence of any message to the user could
potentially contribute to this.
The Post Office also, in the main body of its submissions, stated that Fujitsu monitors
for receipts and payment mismatches, and investigates such occurrences. However,
that does not mean that this bug does not exist; rather to the contrary. The fact that
Fujitsu was supposed to watch for it, and then investigate it, does not affect its
existence. Further, the submissions also relied upon the fact that instances of this bug
“were collected through [Fujitsu] reports without the need for an SPM to report an
issue.” In my judgment this overstates the factual position in the Post Office’s favour.
PEAK PC0204263 dated 16 September 2010 shows that this PEAK was raised by a
“customer call” on 13 September 2010, which means a SPM, who had a discrepancy
of £109. This was given call priority B by Fujitsu. The SPM did report the
discrepancy; they could not report a payments and receipts mismatch because they
would not have known either about the existence of the bug, or its effects.
That this was a problem that required a code fix is shown by the following entry in the
PEAK dated 24 September 2010.
“HNGX CODE FIX
FIX DESCRIPTION
Described Above
PROPOSED BRANCH
TBD
COUNTER JAVA FILES CHANGED
None.
COUNTER PDL FILES CHANGED
RolloverStockUnitBLO.pdl updated
COUNTER REFDATA FILES CHANGED
None.
SHARED CODE FILES CHANGED
None.
BAL JAVA CODE FILES CHANGED
None.
SQL FILES CHANGED
None.
OTHER FILES CHANGED
None.
APPROPRIATE CODE COMMENTS
YES
DEPENDENCIES
None.
POL-0019320
139.
140.
POL00022841
POL00022841
RELATED PROBLEMS
None
UNIT TESTING EVIDENCE
screenshots attached.
REGRESSION TEST CLASS,
NA.
BACKWARDS COMPATIBILITY
its a reference data change hence backward compatibility test is required on LSt
DEVELOPMENT DOCUMENTATION
None.
REQUIREMENTS DOCUMENTATION
None.
HELP
None.”
The Counter PDL files were changed, and as this was a reference data change, certain
checks were needed which is called a backwards compatibility test. This is to ensure
that the fix itself does not cause other problems. This led, eventually on 5 October
2010, to the entry in the PEAK of “Product Error Fixed” after a patch had been
released. A patch is a change, or set of changes, to a programme or its supporting data,
and it is something that changes the operation of the software. The entry of 14
October 2010 states:
“Ths Reference Data hot fix for this PEAK has completed LST testing and has been
passed to the Reference Data Team for live deployment.
Note this hot fix will only work when a counter receives the 02.12 counter upgrade
that is currently in live pilot. (Overnight package COUNTER_X0212 56_1 or
immediate COUNTER_APP 56_1).
[End of Response]
Response code to call type L as Category 71 -- Final -- Fix Released to Call Logger
Routing to Call Logger following Final Progress update.
Service Response was delivered to Consumer.”
I find that this is a bug with potential lasting impact, and indeed it did cause actual
impact. The fact that the Post Office, once it was discovered that it was a bug,
corrected the accounts affected by issuing TCs (and in one instance Fujitsu injected
the necessary correction into the branch accounts) does not reduce its importance. A
reference data fix was necessary to correct it. As explained above, reference data was
part of the architecture in Horizon, and in my judgment is part of the software. The
fact that aspects of the system such as the SQL files or the BAL Java code was not
changed does not mean that this was not a software defect. Even though the
underlying code was not re-written or corrected does not minimise the impact of this
POL-0019320
POL00022841
POL00022841
bug which required a change to the reference data as shown in the PEAK above, and
other of the contemporaneous Fujitsu documents.
2. Callendar Square/Falkirk bug.
141,
142.
143.
144.
145,
It is agreed that this bug occurred between the years of 2000 and 2006, although there
is an issue about when it stopped which I deal with at [149] below. It is a bug which
was present in Legacy Horizon. It too was agreed by the Post Office in the letter of
response in the litigation of 28 July 2016. It was also agreed in the Bug Table as an
“acknowledged bug” which had an impact on branch accounts. It is accepted as a bug
by the Post Office in paragraph 6 of Appendix 2, but is said to have had “transient
impact”. Dr Worden added different comments, including that “the bug arose from a
fault in the underlying Riposte software, so it is not surprising that it took Fujitsu
some time to understand it, or that they had to rely on the suppliers to fix it. It does
not show poor system design or support by Fujitsu”. That latter sentence is in my
judgment a little surprising. It is not relevant to the Horizon Issues whether fault or
defects can be laid at the door of Fujitsu, Escher Group (who supplied Riposte) or any
of the other specialist companies who provided software for different parts of
Horizon. Dr Worden seemed to me to be very defensive on Fujitsu’s behalf. The
Horizon Issues are about the operability and functionality of Horizon, and there are no
findings made in the judgment or this Technical Appendix regarding which specialist
company is to blame for the presence of any bugs, errors or defects.
Riposte software was part of Legacy Horizon, although it is not used in Horizon
Online, as one of the main changes between the two different types of system
(Legacy, and Online) was specifically the way data was stored and messages sent.
Legacy Horizon used Riposte and it was a central feature of that system. Riposte is
not used in Horizon Online. The fact that Riposte is a product designed and supplied
by Escher, and not Fujitsu, is not relevant to the dispute between the Post Office and
the claimants. In any event, and as accepted and recited in paragraph 2 of the Post
Office’s Appendix 2 to its closing submissions dealing with this bug, “Mr Coyne
asserts that Bug 2: Callendar Square is a bug with lasting financial impact and in JS2,
Dr Worden appears to agree that there is strong evidence of this...”
In my judgment Dr Worden does more than “appear to agree”. Given the entry in the
experts’ joint statement, he does in fact agree. This bug occurred when a SPM was
transferring cash between different “stock units”, which means positions.
One of the Post Office’s submissions on this accepted bug merits quotation in full.
The Post Office in its Appendix 2 stated:
“Fujitsu was aware of the wider Ripsote issue for quite some time. Its various
manifestations were being identified and resolved long before the Callendar Square
PEAK was raised.”
This is undoubtedly correct; Fujitsu was aware of the bug “for quite some time”. I do
not consider that this assists the Post Office’s position on the Horizon Issues. The
knowledge Fujitsu had about issues with Riposte was not shared with SPMs, so far as
I can tell. This is also the bug that a Fujitsu employee, Anne Chambers, stated in an
email in February 2006:
POL-0019320
146.
147.
148.
149.
POL00022841
POL00022841
“Haven't looked at the recent evidence, but I know in the past this site had hit this
Riposte lock problem 2 or 3 times within a few weeks. This problem has been around
for years and affects a number of sites most weeks, and finally Escher say they have
done something about it.”
The Post Office stated in its submissions that “Fujitsu’s automatic reports identified
instances of the Callendar Square bug without the need for each SPM to call in with a
report”. That submission ignores the numerous examples in the numerous PEAKs
where SPMs did in fact call in with a report of a discrepancy. The Post Office also,
entirely incorrectly in my judgment, complains in its Closing Submissions that it did
not have the time to cross-examine Mr Coyne fully on this bug. It submitted the
following:
“Mr Coyne notes from the documents that the Callendar Square bug was “operating
and resident in the system for years without any comprehensive linkage being
observed by Fujitsu”. However, this provides only an incomplete picture, as Post
Office would have demonstrated if it had had the time it would have needed to cross
examine him on this bug (that time was taken by Mr Coyne’s extraordinary changes in
evidence on the afternoon of Day 17, the final afternoon of his cross examination).”
I reject this submission for two reasons. Firstly, it was a matter for the Post Office’s
legal team how long it spent on each subject with Mr Coyne, who was cross-
examined for four days. It might have been sensible to have started with the number
of bugs for which Mr Coyne was contending, and then cross-examined on each or
some in detail after that. The Post Office chose to leave the total number of bugs until
the last day; there is nothing wrong with that, but it is not correct to state that there
was insufficient time to cross-examine on this bug (or any bug) and try to blame Mr
Coyne for it. Secondly, the submission entirely ignores the evidence of Mr Godeseth,
called by the Post Office, who expressly accepted that the bug had probably been
present since 2000.
The Post Office also submitted that “a Peak would be raised and the SPM made good
in the ordinary course” if this bug affected branch accounts. There are two points that
this submission fails to address. Firstly, the Horizon Issues concern bugs, errors and
defects with the potential to affect branch accounts. The Callendar Square bug is
clearly one of these. Secondly, the “ordinary course” in this instance seems to be that
not one of the SPMs across the Post Office branch network were told about the
existence or effect of the Riposte lock problem, which Fujitsu recorded (by Anne
Chambers in February 2006) had “been around for years”. In my judgment, the
ordinary course ought to have been providing SPMs with notification of the existence
of this bug and its effects. SPMs and the Post Office have agreements in place (at the
time of this bug the SPMC) that govern their legal relationship. Judgment No.3 makes
clear what the scope of the parties’ legal obligations are.
One specific factual point of disagreement between the parties was the period of time
when the effects of this bug were no longer experienced. The Post Office maintains
this came to an end in 2006, and the claimants submitted that it ran until Riposte was
no longer used, namely 2010 when Horizon Online came into being. One passage of
re-examination of Dr Worden is pertinent in this respect:
POL-0019320
150.
151.
POL00022841
POL00022841
“Q. You say another symptom of the Riposte problem. Perhaps you could explain a
little bit what you mean by that?
A. Well I did explain and I will explain again. That there is a problem in Riposte
which leads to a failure to replicate -- failures to release a lock I think -- and then on
certain occasions that leads to a short-term failure to replicate. On other occasions
there is a so-called event storm during which there is a longer term failure to replicate
and during these failures to replicate all sorts of different things may happen, for
instance they double transfer into a stock (inaudible), a precise Callendar Square
phenomenon; whereas other things such as system freezes, I can’t remember all the
other details, but there are many other symptoms of then underlying Riposte problem
and they have been noted over this whole period.”
(emphasis added)
By “this whole period” Dr Worden was, in my judgment, clearly referring to the
period 2000 to 2010. Whether the short-term failure to replicate, and the “event
storm” with its longer term failure to replicate, are seen as one bug in Riposte (which
is the way the experts approached it) or two sub-elements of one bug, which it could
be (given one is a short term failure, and the other a long term failure, both being
failures to replicate) does not for these purposes matter. In my judgment, the period
when the effects of this occurred are 2000 to 2010. Dr Worden’s evidence does not
support the Post Office’s position that this ended in 2006, and is indeed to the
contrary.
I find that this is a bug with potential lasting impact, and indeed it did cause actual
impact.
3. Suspense Account bug.
152.
153.
This is a bug in Horizon Online and its identified years of effect are 2010 to 2013. It
was also agreed in the Bug Table as an “acknowledged bug”. This bug was the third
of the existing bugs acknowledged by the Post Office in its Letter of Response on 28
July 2016.
It is accepted by the Post Office in paragraph 7 of Appendix 2 as one of a number of
“bugs with lasting impact (although they were resolved)”. Dr Worden’s comments do
not include that it had an agreed impact upon branch accounts. This is because of the
concept he introduced (expanded in his reports) of “transient effect”. What this meant
was that if there was an impact on branch accounts by something within Horizon, but
that was then corrected by a TC, he concluded it had a “transient effect” upon branch
accounts and dealt with it differently. 1 address this concept in the substantive
judgment under “lasting and transient effect”. Other comments of his in relation to
this acknowledged bug are:
“Tt was a transient effect arising not from a fault in the software, but from a change in
database archiving policy in 2010. The delay in correcting it arose from a failure of
communication between PO and Fujitsu. Because the bug would only manifest itself
annually for any affected branch, the effects of this delay were not widespread.”
POL-0019320
154.
15S.
156.
POL00022841
POL00022841
Peak PC0223870 shows that Fujitsu were able to identify the branches affected, even
when Subpostmasters did not report it. There is evidence that the branches were
compensated, as I would expect from the normal error correction processes.”
The Post Office accepts in its Closing Submissions that Fujitsu identified affected
branches by use of “historical data”, in other words, it had to research it as an
investigation was required. Mr Gareth Jenkins in 2013 prepared a note entitled “Local
Suspense Problem” which identified that, at that stage, 14 branches had been affected.
He stated:
“The root cause of the problem was that under some specific, rare circumstances some
temporary data used in calculating the Local Suspense was not deleted when it should
have been, and so was erroneously re-used a year later. When the SPMR was asked to
clear Local Suspense the actual (ie incorrect) amount was recorded in the Audit Trail.
This means that there was no corruption in the audit trail and it accurately reflects the
transactions that occurred in the Branch.
If the BTS from the previous period was taken to provide a set of Opening Balances
and all transactions that were logged to the audit trail during the period were taken as
adjustments, then this would show the correct value that should be in the Local
Suspense account.”
He also stated, in a passage that is of interest in terms of the dispute between the
parties about the use of Audit Data or management data such as Credence (the Post
Office maintains that the latter are sufficient, the claimants insist the former are the
relevant accurate records) the following:
“As well as passing these Local Suspense transactions to the normal accounting tables
that are used to update POL SAP and Credence, they are also written to a table in the
Branch Database that is used to support the printing of the Branch Trading Statement
(BTS) after that Branch has been fully Balanced.”
This demonstrates, in my judgment, that in this instance Credence would have been
wrong. Further detail of the problem explained by Mr Jenkins in the same report was:
“Tn April 2011 a problem was found with the archiving strategy related to Stock Units
that have been deleted in a Branch. A consequence of this is that some changes were
made to the archiving strategy on 3“ July 2011. An unintended consequence of this
change was that any Branch that deleted a Stock Unit at the end of 2010 which had
local suspense transaction in that Stock Unit before it was deleted were left in the
table used for constructing the BTS. This meant that as Trading Periods cycle around
each year, these BTS records became visible in 2011 when the same Trading Period
was reached.
The effect of these old records was that after the BTS was produced an incorrect
figure was generated for the Opening Balance of the Local Suspense Account for the
following period. This amount corresponded to the value of the historical record.
These orphaned records were created between 16" November 2010 and 9"" December
2010.”
POL-0019320
157.
158.
160.
161.
162.
POL00022841
POL00022841
This meant that the figures for the preceding year were effectively recycled as correct
figures for the subsequent period. The Note prepared by Mr Jenkins makes it clear
that, despite how the Post Office sought to present this in their submissions, this
problem persisted.
The document states: “This problem was not reported to Fujitsu in 2011/12 and only
affected a small number of Branches and only for a single Trading Period. However
the two branches with the largest discrepancies did report the issue to Post Office Ltd
who could see the impact of the problem in their back end system and wrote off the
loss or gain for the branch but did not ask Fujitsu to investigate further. At the same
Trading Period in 2012/13, the problem re-occurred and this time one of the affected
Branches reported the problem to Fujitsu on 25" February 2013 (Peak 223870)
resulting in a detailed analysis of this issue and finding the orphaned BTS records.
The root cause was determined by 28" February 2013 and a preliminary report was
sent to Post Office Ltd. A further update was sent on 14" March 2013 with a full
analysis of the issue and all the affected branches.”
(emphasis added)
It is somewhat disingenuous, in my judgment, for Mr Jenkins in this note effectively
to blame others (which appear to include SPMs and the Post Office), rather than
Fujitsu, for this problem not becoming known about until some years later. The trial
did not explore communications between the Post Office and Fujitsu on this subject in
any great detail, but SPMs are not left with a choice of whether to report something to
the Post Office, or whether to report it to Fujitsu. One branch had a loss of
approximately £9,800 (the exact sum was £9,799.88); some were £161 or less, and
another had a gain of £3,100.
Mr Coyne’s view is that this bug could have affected branches prior to Fujitsu’s
investigations. He also suggests that it is unlikely that Post Office and/or Fujitsu have
captured its full effects. My findings on that are as follows. Fujitsu appear to have
conducted a thorough investigation in 2014, and there are spreadsheets showing the
branches identified as having been affected. A code fix was produced in 2013, and
this was first piloted to 50 branches on 21 October 2013 and deployed to all branches
on 29 October 2013. The bug could not have arisen prior to 2010 in any event as that
was the date that Horizon Online was initiated.
I do not know if any of the claimants in this litigation have losses which, when their
claims are fully tried, they claim have arisen in the relevant period and as a result of
this type of incident, but whose branches do not appear in the Fujitsu list. If so, that
will have to be addressed at future trials. However, there is no evidence before the
court currently of branches other than those included in the spreadsheet having been
affected in the relevant period.
I find that this is a bug with potential lasting impact, and indeed it did cause actual
impact.
4. Dalmellington bug/Branch Outreach Issue.
163.
This is a Horizon Online bug. It was not expressly acknowledged as a bug in the Bug
Table, but is effectively accepted by the Post Office as a bug, as it appears under the
heading in its closing submissions as one of “the following bugs had transient
POL-0019320
POL00022841
POL00022841
impact.” It is therefore accepted as a bug by the Post Office in paragraph 6 of
Appendix 2, but this is subject to the point regarding “transient impact”. For some
time during the trial the Post Office would refer to this as “an issue”, and avoid use of
the word “bug”. I find that it is a bug — that much does not seem to be controversial,
as paragraph 2 of the detailed part of Appendix 2 relating to this bug states the
following:
“This is a bug which Mr Coyne states has lasting impact on branch accounts. Post
Office submits that there was no lasting impact on branch accounts.”
164. It is therefore the nature of its impact on branch accounts that is now in issue. Its
effects were experienced between 2010 and 2015, although it is named after the
branch where its occurrence led to its discovery as a bug in 2015. Fujitsu performed
an investigation as a result of the impact of this bug in 2015 at the Dalmellington
branch. The investigation shows the following:
Feb 2010 to Jan 2011 65 incidents
2011 6 incidents
2012 9 incidents
2013 7 incidents
2014 9 incidents
2015 16 incidents
165. The same document showed fixes applied in April 2010, January 2011 and January
2016. It also showed only one call to SSC at Fujitsu in 2015, and none for any of the
years 2010 to 2014 when the figures above show there were 93 occurrences of this
bug.
166. It is what is called a “cash remming error” and happened in certain circumstances. It
was explained both by Mr Coyne, and further by Mr Godeseth in his evidence at [428]
onwards, identified in the judgment itself. Basically, places where a full time branch
Post Office is not justified, such as remote areas, may have such services on (say) half
a day per week, in a mobile van or a village hall. These are called “outreach”
branches, and they are connected to (or part of) a core branch, run by the same SPM.
This means that the SPM is in charge of both the core branch and the outreach branch.
The Dalmellington bug affected such branches.
167. A PEAK was created on 13 October 2015 in respect of Dalmellington, namely PEAK
PC0246949. The SPM in that branch had attempted to transfer cash to its outreach
branches. The SPM sought to transfer £8,000 and ended up with a £32,000
transaction. This problem involved issues relating to two different scripts, the Post
Log On script and the Forced Log Out script. The latter did not close down the former
correctly (or in software terms, did not lead to the former script correctly closing
down the operation) which meant that the script was left on the stack of incomplete
processes. If the SPM then logged back into the counter to perform a transfer of cash
POL-0019320
168.
169.
170.
171.
172.
POL00022841
POL00022841
from their core branch to the outreach branch, the “pouch delivery” screen would still
show, with “enter” being enabled. If the SPM were then to press “enter”, then the
value of the transaction would repeat. As expressed in a confidential briefing given to
the Post Office by Fujitsu:
“Tt relies on a mechanism which checks the stack of incomplete processes to see if it
is complete. Due to the fact that the stack is not empty (following the first problem) it
thinks it has not finished and as a result attempts to repeat the last part of the script,
which in this case is to record the remittance transactions and print the receipts”
This would result in the amount of cash (in the case of Dalmellington, £32,000 as
enter was pressed a further three times) being ‘Remmed Out’ of the core branch.
Because each “pouch” which holds cash has its own ID, Fujitsu researched the
number of times that duplicate pouch IDs had occurred in the BLE files.
There are two points upon which the Post Office relies in respect of this bug. The first
point is that the Fujitsu investigation in December 2015 showed that 112 potential
occurrences of the Dalmellington issue had been identified in the previous 5 years.
Of these 112 potential occurrences, 108 items were corrected at the time, either by
Post Office issuing a Transaction Correction or the SPM reversing the duplicate Rem
In. This left 4 remaining potential occurrences, for which Fujitsu had not yet
established the outcome. However, the claimants rely upon two points of their own in
this respect. One is that this showed that even though the bug had occurred multiple
times and affected branch accounts, neither Fujitsu or the Post Office even knew it
was a bug prior to late 2015. In my judgment, that is a valid point for the claimants to
make. The second feature is that TCs are, as has been identified in the Horizon Issues
judgment itself, outside of the Horizon System.
A third feature is that the Post Office did not disclose the existence of this bug more
widely.
The Post Office also submits that “once the full extent of the Dalmellington issue was
understood, Fujitsu produced a code fix without delay and it was rolled out in January
2016.” However, it cannot seriously be suggested that correction of the bug was
“without delay” when the full history of the PEAKs are considered, with incidents
going back to 2010. By the time this bug was remedied in early 2016, the mediation
scheme had been brought to an end some time before (this occurred in March 2015)
but some of the branches affected by the Dalmellington bug were within the scheme,
according to the Fujitsu investigation. However, none of the periods when the bug hit
matched what were recorded in the investigation as “Mediation dates”. Not only that,
but for the period up to the commencement of the mediation scheme, the Post Office
was meeting a case brought by the claimants that there were widespread bugs within
Horizon which impacted the branch accounts. Dalmellington shows that there was
such a bug, which lay undiscovered (although its effects were apparent) until 2015.
The Post Office also submitted the following:
“As put to Mr Coyne by Mr de Garr Robinson, from an outsider’s perspective, you
would not be able to tell the difference between someone remming in twice by
accident, due to human error and someone remming in twice because of an error on
screen.”
POL-0019320
173.
174.
176.
POL00022841
POL00022841
I am not sure why this submission seeks, as it does, to deploy that submission as a
point in the Post Office’s favour on the Horizon Issues. It is rather a point in the
claimants’ favour, as it means that there was no visible sign that the bug had done
what it had done. It could be misinterpreted as human error; its effects looked exactly
the same, and Mr Godeseth in his evidence accepted this. It is also not a point in the
Post Office’s favour that any known instances of this bug were corrected by means of
TCs, or “by the SPM himself reversing the REM in some way”, which is how it was
put in cross-examination.
Two of the four unknown outcomes were for £25,000, and £2,500. The Post Office, in
its closing submissions, explained why neither of these were in fact incidents of the
Dalmellington bug because they were not outreach services, and also although bar
codes for cash pouches were supposed to be unique, there was a small batch of them
that were not. The documents from which these submissions were drawn show that
the two branches with these discrepancies were those with FAD codes 209311 and
157242. However, the documents also report that both these branches had duplicate
IDs for remming in of cash, that there had been “no contact from the branches raising
issues for the period in question (February and March 2013)” and come to the
following conclusion (although I consider it is worth noting that the SPMs in each of
these branches do not appear to have been asked, nor does the ARQ data appear to
have been used):
“Post Office concludes the issues at the branches have arisen as a result of remittances
pouches received at the branch entered manually which had the same barcode id. Thus
creating duplicate entries which Fujitsu highlighted as part of the BLE files checks.
However, in these instances from the available evidence Post Office concludes that
the correct amount of pouches were delivered, accepted and entered on Horizon. This
is supported by the fact that there has been no negative impact in the branch accounts
and no record of an issue raised by the branches with Post Office.”
And “Moreover, these two branches have resolved in branch the issue encountered
without referral to Post Office via NBSC and FSC or Fujitsu.”
I find that this bug was present in Horizon Online from the date that Online was
brought in, namely the year 2010. In my judgment, this is absolutely clear, and does
not appear to be in dispute given the dates in the Fujitsu investigation.
This bug would, in my judgment, have had an impact upon branch accounts — it is the
extent of that impact that is in issue, whether it was lasting or transient. Dr Worden
relied upon what he described as “‘a well-tested process of reconciliation and TCs to
detect and correct errors in cash remming (used 20,000 times per year)” and said it
was “straightforward for Horizon to detect any discrepancy between a “rem out” and
the corresponding “rem in” (a mismatch arising either from a miscount, or a multiple
count of a pouch) and then a TC can be issued.” He also added that “this process
catches and corrects remming errors, whatever their cause - including if they arise
from, or are provoked by, software faults.” He therefore implicitly accepted that the
Dalmellington bug was a software fault, although he did not say so in terms. I find
that it was plainly was. I also reject his evidence that it was “straightforward for
Horizon to detect any discrepancy”; the evidence about this bug shows that Horizon
did not detect it in these many cases.
POL-0019320
177.
178.
179.
180.
POL00022841
POL00022841
Dr Worden was also reliant upon the process of TCs to correct it. Further, the bug was
there for 5 years and was not discovered, although its effects doubtless were
experienced during that period.
It is correct to record for completeness that all known impacts of this bug appear, on
the face of the evidence before the court at the Horizon Issues trial, to have been
corrected by the Post Office, with the exception of the two branches identified which
were not outreach branches. The documents show that branch 209311 had no rem
issue on, or about, the date of the duplicate ID incident of £2,500 (which was 1 March
2013) and the SPM had not raised any issue with the Relationship Manager Mr Andy
Winn. There was no record of any TC having been issued for £2,500 and an audit in
August 2013 had shown a shortfall at that branch of £350.55.
The other branch, 157242, was operated by a SPM who transferred to a “Main
agreement” in May 2014. Such an agreement did not feature in the Common Issues
trial but I assume it means it is now a Main Post Office and not a branch. Cash rem in
issues were shown at the branch in February 2013 but none of the same type as the
£25,000 duplicate rem in, and none on the exact date in February which was 18
February 2013. Although two of these were for very substantial amounts, none were
for £25,000 and Mr Winn again said that no issue had been raised. I recite the findings
in respect of both these branches for completeness.
I find that this was a software bug that impacted upon branch accounts. In my
judgment, it had lasting impact upon branch accounts. The fact that the occurrences of
it (over 100 in a 5 year period) were corrected by means of TCs does not mean that it
ought not properly to be characterised as a bug causing lasting impact to branch
accounts. However, there were no accounting shortfalls ultimately experienced by
SPMs whose branches suffered from the effects of the Dalmellington bug as these
were corrected by means of TCs. SPMs whose branches suffered the effect of this bug
were not therefore required to pay the sums the subject of the shortfalls to the Post
Office on the evidence before the court.
5. Remming In bug.
181.
182.
This is a Horizon Online bug. It was present for about five months in 2010 between
March and August. It was not expressly acknowledged as a bug in the Bug Table.
However, Dr Worden accepted it was a bug or defect in his cross examination on Day
20 and it is accepted as a bug by the Post Office in paragraph 6 of Appendix 2, but is
again said to be one that had only transient impact. The years it was said to have been
present in terms of effect was “March ~ August 2010 and recorded as fixed approx.
2011”. Mr Coyne said there were 14 branches affected. Dr Worden’s entry in the Joint
Statement relied upon TCs, and stated “As for the Dalmellington bug, above — PO had
a robust process for detecting and correcting remming errors, whatever their origin.
So, there were no lasting effects on branch accounts.” In paragraph 2 of the detailed
part of Appendix 2 relating to this bug, the Post Office submitted that “any
discrepancy would be transient as instances of this bug are caught by automatic
reporting.”
(emphasis added)
The Post Office relies upon automatic reporting to catch incidents of this bug, and
also maintains that Mr Coyne has conflated two issues, one relating to PEAK
POL-0019320
184
185.
186.
POL00022841
POL00022841
PC0203085 (which it terms “Issue 1”) and the other related to PEAK PC0195380 and
other PEAKs associated with KEL acha4221Q (which it refers to “Issue 2”). Both of
these relate to the remming in of cash, and both result in a cash pouch being recorded
twice in error. However, they are said to be different issues because the sequence of
steps taken by SPMs to trigger them are significantly different.
The process of “remming in” works as follows. SPMs receive pouches of cash which
are sent to the branch from the Post Office cash centre. A certain amount of cash is
held in each branch for the purposes of transacting Post Office business. Each pouch
has a unique barcode that needs to be scanned by the branch when it is received. This
automatically looks up the contents of the pouch so that the branch can confirm that
the physical contents of the pouch match up to the record on Horizon. It is for this
reason that each code is supposed to be unique.
The way the system is supposed to work in theory is best taken from the Post Office’s
closing submissions: “If there is a difference between what the system says is in a
pouch of cash and the amount of cash actually in the pouch, the Subpostmaster should
raise the issue with NBSC in order to get a Transaction Correction. A remming error
leads to a mismatch between the amounts of cash remmed out to one place and the
amounts remmed in from another. Remming errors are a violation of Data Entry
Accounting and are picked up by Horizon.”
That latter sentence was said by the claimants, after the written submissions were
received, not to be present in any of the evidence used in the Horizon Issues trial. This
point is dealt with in general terms at [120] above. In response to that challenge, the
Post Office clarified its submissions and relied upon Dr Worden’s report, which was
in general terms in respect of the general principle of double entry accounting; the
HNG-X architecture document, which again demonstrated the theory or principle of
the system; a PEAK document; and Mr Coyne’s cross-examination. It is necessary to
consider those therefore, in order to see if the submission is substantiated.
The principles of what is supposed to happen within Horizon, and the architecture, is
not in dispute. Mr Coyne’s cross-examination passage that is relied upon does not
support the general submission made to the PEAKs the subject of this bug, as it was
as follows. I will quote the passage relied upon and the passages either side for
context:
“Q. So in forming a judgment on robustness you have first of all to see -- let's take a
bug. A bug happens and the first question you ask is: did that bug or could that bug
have had an impact on branch accounts?
A. Yes.
Q. Then you form a judgment on whether and to what extent the countermeasures in
place in the Horizon system would have enabled that impact to be identified and
fixed, yes?
A. No, I think you would look at whether it did -- whether any countermeasure or
control did prevent it from having a consequential impact, not whether it should have.
Q. Well, whether it would have?
POL-0019320
187.
POL00022841
POL00022841
A. Or whether it would have.
Q. You don't consider whether it would have, you consider whether it did?
A. Well, if it did there would be evidence that it did. It would be documented.
Q. But in some cases it will be blindingly obvious, won't it, Mr Coyne? Let me give
you an example, remming. A Post Office example. Remming in and remming out.
Money is sent from Chesterfield to a branch. Chesterfield has a record of the money it
sends over. The branch receives the money and types in -- or usually it is a barcode
actually, but it records on the Horizon system how much money the branch has
received. You are aware, aren't you, that there is an automatic system that checks
Chesterfield's figures against the branch's figures?
A. Yes.
Q. Are you suggesting that every time over the last 20 years there has been a
discrepancy between Chesterfield's figures and the branch's figures, are you
suggesting that it is necessary for you to see the evidence of what happened next, of
whether a transaction correction was sent and what the evidence was in relation to that
branch? Are you really suggesting that's necessary?
A. No, because in a typical scenario where the systems operate as they should, as
they are designed, then the positioning of the countermeasures that you have put in
place would pick up on that so that would work absolutely fine. In the scenario where
the system doesn't operate as expected there is a bug, error or defect or
communication fault, then a different set of scenarios will likely be encountered and it
is understanding then what happens that is important.
Q. Let me get this straight. You are suggesting that there are cases when you can
take it as read, the situation is such that you can take it as read that a problem will be
identified and it will be fixed. But you are suggesting there are other situations where
you can't take it as read where specific evidence is needed, is that right?
A. Yes.
Q. Okay. But nonetheless, although you say that is necessary, it is your considered
opinion that the Horizon system that you now see is robust, yes?
A. Relatively robust, yes.”
This evidence does not support the submission that remming errors are picked up by
Horizon. It is necessary therefore to look at the actual PEAKs, to see what they show
The one associated with what the Post Office called Issue 1, PC0203085, is dated 22
August 2010 and is headed “pouch remmed in on two counters at same time”. The
first entry under impact statement is:
“The same pouch can be remmed in to the system more than once, resulting in a
shortage at the branch which POL have to rectify by issuing a Transaction Correction.
1. call has to be processed, and corrective action taken, by SSC, MSU and POL
POL-0019320
188.
189.
190.
POL00022841
POL00022841
2. visible to POL and the branch when it happens
3. very rare”
In my judgment, that entry alone is evidence of a bug. It shows a pouch can be
remmed in more than once — admittedly rarely — and that a TC is necessary to correct
this. However, in order to be clear, the PEAK goes on for some pages ~ 13 in total —
and includes contradictory entries within it from Fujitsu such as the following two.
One is on 17 August 2010 from Anne Chambers:
“A cash pouch was remmed in twice at branch 126109:
Pouch barcode 399347067204
2p coin £60
50p coin £250
5p coin £100
Session 1-350379 16/09/2010 10:08
Session 2-195226 16/09/2010 10:08
The PM cannot reverse the transaction since rem reversal isn't allowed.
This is NOT another example of the duplicate rem problem that we have seen in the
past, where use of the Prev key accepted the same pouch twice. In this case the pouch
was processed on both counters...
09:05 ¢2 get pouch status, retrieve pouch details
09:06 cl get pouch status, retrieve pouch details
09:08 c2 settle pouch delivery
09:08 cl settle pouch delivery
There were some printer problems on counter 2 which probably explain why this was
done.”
The other contradictory entry within the same PEAK is the next day, 18 August 2010
and states, again from Anne Chambers
“T checked whether there were any exceptions in the BAL OSR logs for any of the
messages, there was nothing.
Gareth Jenkins thinks that it should not be possible to complete the rem in on both
counters. Please investigate.”
Under Root cause and solution, the following entry is made on 19 August 2010:
“When an auto remin pouch id is settled successfully, the system updates the
COUNTER_READ_TIMESTAMP in LFS_RDC_HEADER to a not null value for
that pouch id.
The race condition for auto remin pouch delivery is handled at
SettlePouchDeliveryServiceSettlementProcessor.processPouch().
This method. checks during settlement whether the
COUNTER_READ_TIMESTAMP in LFS_RDC_HEADER is null or not null value.
If null, the pouch id is good and settlement completes successfully.
If not null, the pouch id is already processed and error is thrown.
The query that gets the © COUNTER READ TIMESTAMP _ from
LFS_RDC_HEADER is 'SettlePouchDeliveryPreCheck'.
In this query, the input parameter for pouch id is defined incorrectly.
POL-0019320
191,
192.
193.
POL00022841
POL00022841
It is given "pouchBarcode[String]", but in dyno the pouch id is "pouchId".
This is the root cause why the query always returns null although the
COUNTER_READ_TIMESTAMP is not null.
The solution is to define the pouch id input param to "pouchId[String]" in
'SettlePouchDeliveryPreCheck'.
A fix is then identified, stated to be a low risk and low complex issue, which also
states:
“LIST OF LIKELY DELIVERABLES:
Updated sql of SettlePouchDeliveryPreCheck”.
In my judgment, this PEAK is evidence of a bug and a fix is required to remedy it. It
also shows that remming in errors are not always picked up by Horizon.
The second issue is related to what are different PEAKs, namely PC0195380 and
other PEAKs associated with KEL acha4221Q. I will not deal with them all, but these
relate to what the Post Office terms Issue 2. This KEL was raised by Anne Chambers
on 2 March 2010, updated on 3 May 2011, and the keywords are remittance,
remittence, remin, pouch and delivery. It can be seen that the following is included in
the KEL:
“Symptoms
The clerk went into the Delivery menu and scanned two pouches (one of currency and
one of coins). The second pouch was recorded twice on the system, resulting in a loss
of £80.
Two Rem In slips relating to the second pouch were output, both identical, as well as
one for the first barcode.
In the most recent instance, the same pouch was remmed in on two different counters
at about the same time.
Problem
This was caused by using the Prev key during / just after the pouch barcode scans.
Now fixed - details in PCO195380.
In PC0203085, the same pouch was processed on both counters...
09:05 c2 get pouch status, retrieve pouch details
09:06 cl get pouch status, retrieve pouch details
09:08 c2 settle pouch delivery
09:08 cl settle pouch delivery
PC0203085 - fix applied Jan 2011.
Solution - ATOS
A transaction log search will prove that the Rem In has been duplicated.
POL-0019320
194.
195.
POL00022841
POL00022841
Send call to SSC, quoting this KEL.
SSC:
Known problems have been fixed, so needs fresh investigation if it happens again.
POL may need to issue a TC to undo the effects of the extra rem in (in the meantime
the branch will report a shortage), so MSU need to inform POL via BIMS.”
Given that the PEAK on Issue 1 is dated August 2010, and the KEL itself was raised
in March 2010, the KEL must have been updated with the entry relating to that PEAK
and that the fix was applied in 2011. Those entries cannot have been included in the
KEL originally, as they post date its creation in March 2010. This means therefore
that someone at Fujitsu obviously concluded that the later PEAK in time (on Issue 1)
was relevant to the same KEL (on Issue 2) as it was added to the KEL. I agree with
whoever at Fujitsu linked these PEAKs to the same KEL. I consider that the KEL
must refer to broadly the same issue, namely the remming in of pouches that were
counted twice by the Horizon System. A fix was in any event issued for the issue
identified in the KEL. However, it is correct and I accept the Post Office’s
submissions that upon closer analysis it can be seen that there are two separate, but
very similar in effect and related, issues here, both of which have the same end result,
namely the duplication of remming in of cash pouches.
Finally, in the Post Office’s clarification or response to the challenge brought by the
claimants to the submissions on this bug, (which complained that there was no
evidence to support the submission that remming in issues would be automatically
“picked up” by Horizon), the Post Office referenced another document at {F/624/1}.
It stated that the document was PC0197838, but in fact the reference is BE/0197828,
as it is a BIMS Incident Report and not a PEAK, and is dated 16 April 2010. The
reference number is the same but the initials are BE and not PC because it is a
different type of document. However and in any event, the document relied upon by
the Post Office identifies an exception type as “Receipts and Payments do not balance
(post migration)” but it does deal with the same issue as this one, as it states within
the body of the document that “The PM has used the Previous key whilst scanning in
pouches, which has caused a duplicated rem” and also:
“The two identical transactions each comprise of:
Prod ID = 6287 (Rem In Cash from AD); Qty = 1; Amount = £1680
The Post office Counter log shows this being made up of:
£100 of 20p coins
£80 of 2p coins
£500 of 50p coins
£100 of Sp coins
£500 of £1 coins
Action for Post Office Ltd:
Post Office Ltd will need to contact the PM to confirm if the PM has two identical
Rem In slips for Session 489165, both showing Pouch ID 399345656721.
If this is the case, then the Rem In has been duplicated and this duplication needs to
be reconciled, possibly by using a Transaction Correction.”
(emphasis added)
POL-0019320
196.
197.
POL00022841
POL00022841
I accept that the issues on this bug can be split into Issue 1 and Issue 2 in the way that
the Post Office suggest. If I am wrong about that, then the PEAKs I have identified
above, including the BIMS Incident Report, are evidence of and relate to the same
bug. Given the finding of two issues, then Issue 1 and Issue 2 could either relate to
different iterations of one bug, or to two different bugs. These bugs or this bug
(depending upon which semantic analysis of Issue 1/Issue 2 is correct) both impact
branch accounts. Issue 2 related to the pilot phase of Horizon Online. However,
neither of these issues are caught by automatic reporting. The Post Office accepts that
Issue 1 was reported to it by the SPM affected. The fact that the BIMS report requires
the Post Office to “contact the PM to confirm if the PM has two identical Rem in
slips” for the session identified makes it clear that this also applies to Issue 2 as well.
There is therefore, in my judgment, no practical difference whether one approaches
this as two different issues with the same effect, or one.
On either analysis, it is a software bug and I find that these or this impacted upon
branch accounts. In my judgment, it had lasting impact upon branch accounts. The
fact that the occurrences of it were corrected by means of TCs does not mean that it
ought not properly to be characterised as a bug causing lasting impact.
6. Remming Out bug.
198.
199.
200.
This arises under Legacy Horizon. The Post Office’s submissions effectively accept
that there was a bug under this heading, because in the detailed entry in its Appendix
2 it recites the following:
“Mr Coyne states that Bug 6: Remming Out is a bug with lasting financial impact. It
comprises two separate issues, only one of which was a bug. Any discrepancy caused
by either issue would be transient as instances of both issues were caught by
automatic reporting.”
(emphasis added)
It was split into two in the Bug Table, 6(i) which was identified as KEL acha508S and
6(ii) which was identified as KEL GMaxwell3853P. The former was identified as
February/April 2007, recorded as fixed approx. 2007, with Mr Coyne identifying 57
branches affected; and the latter was in May 2005 with one branch being affected.
They were both remming out issues, hence the grouping by the experts in the bug
table. It was not acknowledged as a bug in the bug table. However, it is now accepted
as a bug by the Post Office in paragraph 6 of Appendix 2, but is said to have had
transient impact.
Again, for both of these iterations, 6(i) and 6(ii), Dr Worden used the same wording
as he had in the entry for bug 5, the Remming In bug, namely “As for the
Dalmellington bug, above — PO had a robust process for detecting and correcting
remming errors, whatever their origin. So, there were no lasting effects on branch
accounts” (emphasis added). It was also said by the Post Office in its submissions that
Mr Coyne had conflated two unrelated issues under the heading “Remming Out”. Given
this was an entry in the 2 Joint Statement agreed by the experts, it is a little unfair to
state that Mr Coyne had done this, as the entry was obviously agreed by Dr Worden.
However, it was split into 6(i) and 6(ii) and I will consider each separately.
POL-0019320
201.
202.
203.
204.
205.
206.
POL00022841
POL00022841
Issue 6(i) arises as follows. What is called “a remming error” leads to a mismatch
between the amounts of cash remmed out to one place and the amounts remmed in
from another. The Post Office has submitted that remming errors are a clear violation
of Data Entry Accounting and are picked up by Horizon. The two different issues are as
follows.
As the obverse of the coin of remming in, SPMs rem out pouches of cash to be
returned to the Post Office Cash Centre. A single pouch may contain multiple bags of
coins or cash and each bag can only hold one denomination, and there is a limit on
how much cash can be placed into a pouch. The cash can be remmed out before it is
physically collected. When remmed out the cash appears in a different line in the
branch accounts. On collection, the collection team scan a barcode on the pouch and
the cash is removed from the “cash in pouches” line of the accounts.
When remming cash out, branches should have made one entry for each denomination
and value and, if there were multiple bags for a particular denomination, the quantity
of bags should have been specified in that single entry (e.g. 2 x £500 of £2 coins).
However, if the SPMs had made multiple entries for each denomination and value
(e.g. one entry for 1 x £500 of £2 coins and then a second entry for 1 x £500 of £2
coins), Horizon would only record the first bag as having left the branch’s cash
holdings, but all of the bags would show on the “cash in pouches” line. This would
create a discrepancy in the branch accounts, because all of the cash would have been
collected.
This issue occurred as a result of Release T30 INCI (which is referred to in the
Fujitsu Service Review Book of February 2007 covering that month) and the Post
Office’s submissions accept this. A further entry for Monday 12 February 2007 in the
same document states:
“The issue was found to be when Post Masters would REM out 2 identical items
instead of using the quantity button. As this was an error in the code, the counter 36_6
s/w was regress overnight. After the regression of the s/w overnight, it was then
identified that 49 branches had an issue with the REM out process causing a cash
imbalance at the counter.”
I consider this to be a bug. Fujitsu created KEL achaS08S to advise branches to
correct the problem manually and ran automated reports to spot any further
occurrences. Fujitsu then made changes to Release T30 INC] and rolled it back out
to fix the issue. However, that the bug was fixed does not, in my judgment, mean it
ought to be ignored. Paragraph 22 of the Appendix 2 submissions on this by the Post
Office submits “in any event, it was picked up automatically”. This is not correct. An
entry in the February Service Review Book on page 11 of 33 of that document shows
that there were 526 calls to the Horizon Service Desk as a result of this specific issue,
of what were said to be a high number of software calls in that month of 3792. That
does not support a submission that it was picked up automatically.
So far as Issue 6(ii) is concerned, the following evidence from Dr Worden — which is
entirely ignored in the Post Office’s submissions in Appendix 2 — is relevant.
“Q. So we are looking at 6(ii).
A. Yes. So again my standard wording comes in.
POL-0019320
207.
208
209.
210.
POL00022841
POL00022841
Q. That's it. You will see there:
“Remming out' (ii) Bug (not acknowledged)."
This isn't one that you had in either of your tables in your appendix originally.
No.
. Did you have a look at the KEL in that case?
. I believe I did, yes.
. Can we look at it now, please. This is at {F/276/1}.
. Rem out ... Yes.
This is the GMaxwell3853P KEL. If we look at the bottom where it says
"Solution - Atos”.
A. Bug in the code, yes, right.
Q. "Development have identified a possible bug in the counter code. However, due to
the frequency of the problem & the risks involved in making the necessary changes it
has been decided that a code change will not be made."
A. Yes.
Q. You hadn't noticed that, had you?
A. I believe I had, but again the point about permanent effects, lasting effects, on
branch accounts is not altered by that.
Q. And the point about this is there was a lasting bug?
A. There may have been a lasting bug, but they took a decision based on its
frequency and the difficulty of making the change, perhaps the risk of making the
change, that it was not worth addressing. That actually the remming process would
pick it up and its frequency -- they took a decision, I don't know what the details were
of their trade off. They took a decision that fixing the code was not worthwhile.
Q. I'm just asking you: it is a lasting bug?
A. Itis a lasting bug, yes.
Q. Let's go to number 7 now, if we may, please.”
OPOPO>
There was then an interjection by counsel for the Post Office who drew attention to
possible misunderstanding in terminology given the use of the phrase “lasting bug”.
Dr Worden then added the following, when asked by the claimants’ counsel whether
he had understood what was meant by the question:
“A. I did. And I'll clarify what I mean: it's a bug that was not fixed, we cannot see
was fixed, but it was a bug without lasting effects in my opinion.”
In my judgment, it was clearly accepted by Dr Worden as a bug, and the entry in the
KEL itself also says there was “a possible bug in the counter code” but a decision was
taken not to change the code because there was only one instance of this occurring.
I find that 6(ii) also relates to a bug and reject the Post Office’s submission that “it
wasn’t a bug”.
The claimants’ case on the Remming Out bug in the Bug Table at item 6 is therefore
made out. Both of these — issue 6(i) and 6(ii) — are, in my judgment, software bugs. I
find that these had potential lasting impact upon branch accounts.
7. Local Suspense Account issue, not the same as 3. Suspense Account bug.
211
This was reported in 2010 and recorded as fixed in September 2010. It arose in the
early days of Horizon Online, during the pilot scheme. It was not acknowledged in the
POL-0019320
212.
213.
214.
POL00022841
POL00022841
Bug Table, but is now accepted as a bug by the Post Office in paragraph 6 of
Appendix 2, although it is one of those bugs said to have had transient impact. Mr
Parker had identified 33 branches affected. Mr Coyne recorded what he said were four
associated KELs, namely acha5259Q (for which there were 6 PEAKs) (this KEL is
mistakenly recorded twice in column 3 of the Bug Table, with achaS838T only
mentioned in the text in that column); cardc2043L (10 PEAKs); PorterS199P (3
PEAKs); and acha5838T (which states there are “two different but similar problems”
and appears in the text, but not in the list of KELs at the end of the text by Mr Coyne).
Dr Worden’s comments in the table were that:
“The KEL acha5259Q implies that PO and Fujitsu were able to identify all
occurrences of the problem, without being notified by any Subpostmaster. I would
therefore expect them to have corrected any impact on branch accounts as part of
normal error correction processes.
I would not expect evidence of all corrections to accounts to have survived to the
present day. PEAKS and KELs are not used to record corrections of financial
impact.”
He also relied upon a statement by Mr Parker that there was “Temporary financial
impact which would have been cancelled out in the following period by a
corresponding discrepancy”. To use the Post Office’s own submissions it was “an
intermittent system issue which temporarily prevented branches from rolling over into
the next Trading Period”.
The statement of Mr Parker introduces another concept, similar if not identical to Dr
Worden’s “transient impact”, and that is “temporary financial impact”. Dr Worden is
correct that PEAKs and KELs do not record corrections of financial impact; they do
however record financial impact, because when an SPM reaches SSC (which raises
the PEAK) they often start by recording “SPM says he/she has a problem in that...... ”
and financial impact is often then recorded. The Post Office in paragraph 2 of the
detailed part of Appendix 2 dealing with this bug states that “any discrepancy would
not be lasting”. This does therefore accept, albeit implicitly, that it would have an
impact upon branch accounts. It is also submitted that it was a “teething problem”
from the early days of the Horizon Online pilot scheme.
The cross-examination of Dr Worden on this bug was as follows, and I find to be
highly relevant.
“Q. Then you say:
“T would not expect evidence of all corrections to accounts to have survived to the
present day. PEAKs and KELs are not used to record corrections of financial impact."
A. Yes.
Q. Fujitsu, in fairness, analysed the KEL, you say here, and said:
".. 'Temporary financial impact which would have been cancelled out in the
following period by a corresponding discrepancy.”
Do you see that?
A. Yes.
Q. Now, did you look at the KEL?
A. Well, l infer from that Fujitsu did it, but I had done it previously.
Q. Let's go, please, to {F/637/1} which is the KEL acha5259Q.
POL-0019320
215.
POL00022841
POL00022841
A. Right. Sorry, can I remind myself of the .
teport ... cash ..."
Q. Do you see if we look at the problem --
A. Yes.
Q. -- the summary was that the PM was forced to clear local suspense several times
resulting in the cash figure on the balance report changing and possibly a discrepancy
in the new trading period.
A. Yes.
Q. So that certainly had the potential to cause an impact on branch accounts at the
time, subject to later correction?
A. Yes.
Q. And probably did, yes?
A. I think it did, yes.”
local suspense cleared ... balance
It is in my judgment self-evident, that if the branch accounts had to be corrected later,
in a new trading period, that the accounts for the period when the effects of the bug
were experienced were wrong. This was a bug. It was a bug that had a lasting impact
upon branch accounts as I have defined that phrase. The fact that the discrepancy was
later corrected, in the words of Mr Parker a discrepancy “which would have been
cancelled out in the following period by a corresponding discrepancy” makes this
entirely clear.
8. Recovery Issues.
216.
217.
These are Horizon Online issues. The Post Office does not accept these are bugs at
all. These are not agreed as bugs by the experts, and there are four different types
included in the table, with years of effect from 2010 to 2018. Mr Coyne’s entry stated
that “The text within the PEAKS and KELs suggests that in each case a branch
account discrepancy would be evident and would require correction by the Post
Office.” Dr Worden stated that:
“The KELs and PEAKs cited by Mr Coyne are not indicative of errors in Horizon.
They provide guidance on how to correct discrepancies caused by human errors or
other errors in transaction recovery (‘recoverable transactions’)
Because there were many such errors, there were many calls to the help desk and
many PEAKs and KELs. Normally, correction of errors involved back office
reconciliation and issuing TCs. This was accurate and effective; I have derived an
upper limit of £2 per branch per month on the mean impact of erroneous TCs.
One important KEL acha959T was guidance to the back office MSU, not for
Subpostmasters”.
There was a minor typographic error in the submissions in paragraph 4 of the detailed
part of Appendix 2 which referred to bug 9. However, the four different types of
recovery issues are addressed in the subsequent paragraphs and the conclusion
paragraphs deal with bug 8. This was later corrected in a sheet of corrections, which
clarified that paragraph 4 of the detailed part of Appendix 2 should have stated "Mr
Coyne states that Bug 8: Recovery Issues is a bug with lasting financial impact. Post
Office submits that it is not a bug at all”
POL-0019320
218.
219.
220.
POL00022841
POL00022841
Recovery deals with the situation where there has been an interruption, which means
when a basket of transactions is interrupted during the course of dealing with a
customer. This could be for several reasons including a power failure, system crash or
communication failure between the branch and data centre. This is a risk which has to
be dealt with as it cannot sensibly be eliminated in a system such as Horizon. Horizon
runs various automated reports each day to look for failed recovery events. Mr Coyne
dealt with two different documents in his report, PC0197769 and KEL acha959T. The
former was created on 15 April 2010 during the pilot phase of Horizon Online. The
root cause of the issue was identified on 26 April 2010, work on a fix was started and
this was released in early June as shown in Release PEAK PC0199000 (and then
estate-wide to pilot branches) by mid-June 2010. It did however undoubtedly
evidence a bug. The reference to the “959T” KEL below is the KEL whose full title is
acha959T which was raised by Anne Chambers.
KELacha959T is also the very same KEL referred to in PEAK PC0214226 dated 14
December 2011, headed “Failed Recovery Transaction(s) 12 Dec 11” which I have
considered at [110] and [111] of the substantive judgment, as it related to Mr Tank’s
specific experience at his branch regarding the £195 transaction. I have already dealt
with one entry in that PEAK in that part of the judgment. I have accepted Mr Tank’s
evidence on this incident. That PEAK was closed with the categorisation “Final —
Advice after Investigation” and “Root Cause — General in Procedure”. The entry at
10:40:51 on 14 December 2011, also from Mr Bragg, shows the transaction details
which plainly identify an authorised transaction, a failure of the basket settlement, the
failure of repeated retries by the system and the failed recovery. It was this analysis
that preceded Mr Bragg’s conclusion that “the customer’s account should be correct
but the branch will have a shortage (for a withdrawal) because the session hasn’t been
tecorded”. This is plainly a fault in Horizon, and is directly contrary both to the Post
Office’s case on failed recoveries in the Bug Table, and also its case on Mr Tank’s
evidence.
Dr Worden’s cross-examination on the issue of recoveries, by reference to entry 8 in
the Bug Table, was also highly relevant:
“Q. Let's move if we may now to number 8 and we find that -- just to do it the same
way again, if we go back to the joint statement 2 table and we look at {D1/2/7}.
Row 8.
It is row 8. The recovery issues are identified there.
Mm.
Mr Coyne has made references to the text in certain PEAKs.
Yes.
. And you have given your opinion there, you say they are not indicative of errors
in Horizon, they provide guidance on how to correct discrepancies caused by human
errors or other errors.
A. Yes.
Q. You say: "Because there were many such errors, there were many calls to the help
desk and many PEAKs and KELs." Then it is basically your same theory, that:
“Normally, correction of errors involved back office reconciliation and issuing TCs."
And that's accurate.
A. It is a different theory really. Recoverable transactions are a big subject and they
are complicated because the point at which a recoverable transaction can go wrong is
OPO>O>
POL-0019320
POL00022841
POL00022841
very variable through the sequence, and therefore the number of recovery actions, the
type of recovery actions is complicated. Horizon was designed so that with the
assistance of the postmaster most of these could be recovered, but there are things
called failed recoveries, which again were part of the design of Horizon, and those
were the failed recoveries but particularly the ones where Fujitsu had to get involved.
But failed recoveries means the recovery process had failed, that's the way Horizon
was designed because these things are so complicated you can't handle them all
automatically. So it is a big subject but there is plenty of useful evidence about it.
Q. You say in your table at {D3/2/124}.
A. This is the 959T KEL, is it?
Q. Yes. You say:
"This is another complex KEL with strong overlap with previous KEL.”
A. Yes. This KEL is cited in a large number of PEAKs.
Q. And in fact when we go back to the table at {D1/2/7}, Mr Coyne has actually
quoted from one of the PEAKs we see there, the first one he quotes from. Do you see
"PCO198352"?
A. Yes.
Q. And he has quoted: "This problem caused a loss at the branch for which they
should not be liable."
Pausing there. This did have the potential to cause the type of discrepancy which you
have given evidence about being later corrected, didn't it?
A. The word "problem" doesn't imply a bug in Horizon.
Q. You didn't read these PEAKs yourself, did you?
A. I have read a fair sample of recovery PEAKs. Maybe not these ones, but I have
read a fair sample.
Q. If we go over the page {D1/2/8}, do you see Mr Coyne points out there that this
KEL is referred to by 2,473 PEAKs --
A. Yes, that's --
Q. -- from 2010 to 2018.
MR JUSTICE FRASER: Where are we looking?
MR GREEN: At the top of that page under the "Coyne Opinion as to branch account
impact", my Lord.
MR JUSTICE FRASER: Yes.
A. That is correct.
Q. Just pausing there, Dr Worden. If something is handled by NBSC and corrected
straightaway, it doesn't make it necessarily up the line to SSC, does it?
A. No. I mean there are various levels of recoverable transactions. There is a
recoverable transaction that the subpostmaster can follow the instructions and can fix
it himself, there is the ones where he needs help from NBSC, there's ones where he
needs help from further down, and there are failed recoveries where PEAKs are
involved. All of those things happen.
Q. In the case of a failed recovery, it can be sorted by NBSC in some way or by a
transaction correction without referring to SSC?
A. That's a good question. I think the failed recovery port usually involves not just
NBSC, it involves MSU, and there is quite a complicated process of guiding the
transaction through various states until it gets to the F99 state.
Q. Can I just invite you to look at what Mr Coyne said --
A. Yes.
POL-0019320
221.
222.
POL00022841
POL00022841
Q. -- having actually tried to identify through the available disclosure how many
PEAKs were referenced and set them out. He says it is referred to by 2,473 other
PEAKs from over that range:
" ... each of these may (but may not) indicate a similar issue with the Horizon
recovery process and potentially creating a discrepancy within branch accounts."
If we accept his definition of discrepancy and not yours, yes?
. This is all about temporary discrepancies in branch accounts.
. Park the temporary discrepancy point about which we have heard a lot.
Yes.
. What he has said there is correct in the number of PEAKs as far as you know?
. It is correct, yes.
. And it is scrupulously fair in how he has described it?
Well, "issue with the Horizon recovery process", I believe these issues are how
Horizon was designed.
Q. But what he's said there, he has carefully put "issue", leaving open the availability
of an argument between experts about what the nature of the issue is, but what he has
scrupulously done there is perfectly fair, isn't it?
A. I think so. I mean the difference between the experts is that, to summarise
something Mr Coyne said in his oral evidence, he believes that these recovery
processes can be automatic. Now, I don't believe that. I believe recovery of
recoverable transactions is very complicated and needs human intervention and things
like failed recoveries are inevitable.”
(emphasis added)
POPOPO>
I prefer the evidence of Mr Coyne that recovery processes can be automatic in
systems, and I also accept that “failed recoveries” as they were defined above are
inevitable. The issue here is whether the Horizon System process for dealing with the
recovery processes when they had in fact failed was working as it should, or
defectively. The fact that there are so many PEAKs identified in the KEL, and the
contents of those PEAKs clearly demonstrates, in my judgment clearly, that it was
not. I reject Dr Worden’s evidence that human intervention should be required or
expected for some, most or all failed recoveries. I also reject his evidence about the
word in the PEAK “problem”. He said “The word "problem" doesn't imply a bug in
Horizon.” In my judgment, on this subject, it clearly does.
The Post Office’s own submissions set out how trading periods were affected. I shall
reproduce them:
“10. The issue is described in KEL achaS6S0L. It involved recovered transactions
being written to a different trading period than the original transaction. In summary:
i) if a transaction failed in one trading period and the recovered transaction went
into the next trading period, there would be a loss in the first trading period 5
and a corresponding gain in the next trading period such that there would be
no overall loss in the branch; and
ii) if the recovered transaction was written to an earlier trading period, there
would be a loss in the current trading period but no corresponding gain
because the previous trading period would have already been rolled over
again. There would be a net loss in that scenario.”
POL-0019320
POL00022841
POL00022841
(emphasis added)
223. That demonstrates that branch accounts would be affected. In the first possibility, the
accounts would be wrong for one trading period (until the so-called “corresponding
gain” occurred in the next period) and in the second, it is accepted there would be a
net loss.
224. In my judgment, the issuing of a later TC is neither here nor there in this scenario.
Where there has been a failed recovery, this is now flagged automatically by Horizon
to SSC at Fujitsu for investigation. So far as the modern or more recent way that the
system deals with failed recoveries, it is plainly the case that Horizon now
automatically flags these. The fact that it did not do so previously, in my judgment,
and the existence of all these PEAKs on this subject, clearly demonstrate a bug, error
or defect, and that conclusion is not avoided by the fact that a later TC was issued.
225. On 16 January 2017, the automatic monitoring service spotted a failed recovery and
PEAK PC0256502 was raised by Fujitsu. A BIMS was issued to Post Office to
correct the discrepancy. On 17 January 2017, the SPM in question contacted the
helpline regarding the same issue. A second PEAK PC0256566 was raised. As
quoted by Mr Coyne in his report, this PEAK clearly records that this issue was
already identified by Fujitsu and resolved under the earlier PEAK. Hence the later
PEAK in time was closed. However, although this example required corrective action
by the Post Office (not by Horizon) it is not evidence of a bug. The former PEAK
namely PC0197769 did evidence a bug. I also find that PEAK PC0214226 is evidence
of a bug.
226. Mr Parker’s evidence also makes it clear that this bug had an impact upon branch
accounts. Mr Tank’s evidence demonstrates the same, albeit for a different date and
time. This impact would persist until it was later corrected.
9. Reversals.
227. This is a Legacy Horizon bug and occurred in 2003. It was not acknowledged as a bug
in the Bug Table but is now accepted as a bug by the Post Office in paragraph 6 of
Appendix 2. It is however said only to have had transient impact.
228. Dr Worden’s entry in the 2"¢ Joint Statement in relation to this stated “Transaction
reversals are a complex area which, like recoverable transactions, are less familiar to
Subpostmasters and are more prone to human error. They lead to many calls to the
help line and to many KELs and PEAKs - not necessarily related to any fault in
Horizon.” This can be seen as an example of Dr Worden attributing fault to the SPMs
rather than accepting the presence of a bug, but given it is now accepted as a bug it is
not necessary to consider that any further. Mr Coyne’s entry in the 2" Joint Statement
stated “In April 2003 due to a failure in regression testing, Horizon version S30 was
released by Fujitsu and this introduced a bug where the value of transactions reversed
by Subpostmasters was shown twice in the amount of the reversal in branch
accounts.”
229. This was clearly a bug. A Fujitsu document, PEAK PC0089918, identifies an issue in
which attempted reversals of remming in transactions resulted in the magnitude of the
transaction doubling, rather than the transaction being reversed. Initially, this is
POL-0019320
230.
231.
232.
233.
POL00022841
POL00022841
blamed upon the SPM in question and the Post Office in its Appendix 2 continues
this, submitting that “The Subpostmaster (branch FAD003227) was trading in Stock
Unit Y and remmed in £13,910 of cash. He continued to trade in that stock unit in
error, instead of moving to Stock Unit I. He therefore attempted to reverse all of the
transactions he had made in Y (including the rem in). Instead of Y showing a balance
of zero, the rem in had doubled to show a discrepancy of £27,820. Having spoken to
NBSC, the Subpostmaster attempted further reversals but these failed and produced
an error message indicating that the initial reversal had been completed successfully.”
However, it was clearly caused by a bug. This is effectively accepted later in the
submissions which state, correctly in my judgment, that “the issue was caused by a
software error, which had been introduced as part of implementing the fix for
PC0083954 in Legacy Horizon, and which incorrectly calculated the reversal value
and quantity (essentially, the wrong mathematical sign was applied when reversing
RIAD transactions (+ instead of -))”.
If a minus arithmetical function is required, namely subtraction, but by software fault
this actually occurs as an addition, then this is clearly a bug. It is certainly an error or
defect in the system. This was then compounded by PC0083954, a fix which
introduced a code change to ensure that a partial settlement of cash always tried to
reduce the stack value. However, that fix did not work when reversing a rem in and
so the solution was to pass the mode to the function also.
A fix was required, and because of the potential impact on live branches, the
development of a fix is said to have been “fast-tracked”. The Post Office submissions
stated that “within 15 days of the issue first being raised, Fujitsu had implemented a
fix to 2,178 branches. It is not known precisely when the fix reached all branches in
the estate but Fujitsu believe it is likely to have been before 2 June 2003....” That
again comes perilously close to introducing evidence by the back door through
submissions.
This was clearly a bug with potential lasting impact to branch accounts. The fact that
the impacts of its occurrence were corrected by means of TCs does not affect its
existence. It occurred for a short period in 2003. There is no evidence before the court
of branches being affected by this outside that period, or of branch accounts affected
by the bug that were not later corrected.
10. Data Tree Build Failure discrepancies.
234.
235.
This is a Legacy Horizon bug. Its identified effect was during 1999 and 2000, so the
very early days of Horizon. However, KELs relating to the PEAKs in those dates
were created in July 2005 and updated in November 2007. A data tree is an
accounting node hierarchy, or a cash account node hierarchy.
It is accepted by the Post Office in paragraph 7 of Appendix 2 as one of a number of
“bugs with lasting impact (although they were resolved)”. The PEAK dealing with
this reads “Data trees have been failing to build fully, and the system has not been
detecting this.” Data trees were part of Legacy Horizon, and were used to build a
summary (or picture, the word used by the Post Office in its Appendix 2) of the
accounts. The building of data trees is a software function, and given “the system” is
Horizon and is supposed to detect failures of this nature, it is difficult to see how it
POL-0019320
236.
237.
238.
POL00022841
POL00022841
could ever have been in issue that this was a bug. Mr Coyne’s entry in the Bug Table
recites that “Dugannon branch suffered a £43,000 discrepancy but the cause was not
immediately known. £52,814.29 at the Yate Sodbury Branch. £9,368.40 at the
Appleby Westmoreland branch.” These are sizeable sums and arose in the branch
accounts. That this is a bug was de facto accepted by Dr Worden in the Bug Table due
to the text of his entry which states:
“There was a bug which has potential impact on branch accounts, early in the lifetime
of Horizon. Soon after it arose, the error was trapped and detected by DEP and was
then soon fixed.
The fault was easily noticeable at branches before the error trapping which was soon
introduced and would be even more noticeable after that. Only three branches appear
to have been affected, as described by Mr Coyne.
Because it was so noticeable at the branch, and the Peak is concerned with a software
error rather than any other cause, I would expect any discrepancies in branch accounts
to have been corrected.”
(emphasis added)
This is therefore undoubtedly a bug, but findings in respect of this therefore depend
upon the nature of its impact on branch accounts. Dr Worden in his cross-examination
changed the text of one of his entries in one of the appendices to his report, where he
had said “This is further evidence of the failed recovery report doing its job - alerting
[Fujitsu] to failed recoveries, so they can investigate them and make any necessary
corrections to accounts.” He said this later part should be changed: “Perhaps I should
have said have "guided the transactions to the correct state". He said it was caused by
a complex grey communications failure, by which he meant an intermittent one,
although he was not really sure where he had obtained the word “grey” from, he
thought perhaps from a PEAK or KEL.
The first PEAK PC0033128 related to an issue in November 1999 at Dungannon. The
SPM reported a £43,000 discrepancy after balancing stock units and doing an office
snapshot. An office snapshot is a Horizon report that showed the current accounting
position of the branch, including any cash discrepancy. To produce the office
snapshot, Horizon scanned the messagestore for the necessary information (eg. initial
cash and stock levels, all cash and stock transactions, plus other service transactions).
It then compiled that information together in order to produce the office snapshot.
This is the “data tree”. On occasion Horizon would fail to read the necessary
transaction data for various reasons so the report would be incomplete. This failure at
Dungannon was caused by a missing payments node.
Two other branches were affected by the same issue, Yate Sodbury and Appleby-in-
Westmorland. The snapshot at the former was incorrect showing a shortfall of
£52,814.29 and Appleby’s snapshot showed a shortfall of £9,368.40. MSU were
involved in the investigation of this and the Post Office submits that “it is therefore
likely that the issues at Yate Sodbury and Appleby were resolved via a BIMS report
and that the Subpostmasters were held harmless. However, due to the age of [this
issue], comprehensive records are not available and therefore Post Office is not in a
position to provide detailed commentary.” Two points need to be made. The shortfall
was in a snapshot and not in a Branch Trading Statement; and secondly, there is no
POL-0019320
239.
240.
241.
242.
243.
244.
POL00022841
POL00022841
evidence of how it was corrected if at all. There are references in the PEAK in respect
of Dungannon that shows the SPM and the Post Office agreed to remove the
discrepancy from the account. For present purposes, although all these shortfalls are
substantial, there is no evidence that the Post Office required payment of these sums
from SPMs.
In my judgment, the Post Office correctly identifies branch impact in the following
passage in its submissions:
“Of itself, Issue 1 [ie the issue above] would not affect branch accounts; there was no
issue with the underlying transaction data and, if the office snapshot was re-run, it
would very likely provide the correct information, because the data reading issue was
temporary. However, if the branch ran the office snapshot, got an inaccurate report
and then rolled over (making good any discrepancies in the process), then the shortfall
would have an impact on branch accounts.”
There is an associated issue in respect of data trees, namely PEAK PC0121925. As
part of the changes made to move branch accounting from weekly cash accounts to
monthly branch trading statements (a programme called IMPACT), changes to the
data server were made to reduce the number of times that the messagestore was
scanned to pick up transactions during balancing. A Riposte mechanism known as
“Notifications” was used to add new transactions to the existing totals as further
transactions were generated during the balancing process (rather than rebuilding the
data server tree of transactions from scratch). One of the test branches experienced an
issue, where the test branch had a gain of £45.05 following a cash declaration and
rolling into branch trading. Initially, Escher were unable to replicate this scenario and
so no further action could be taken. Subsequently, as demonstrated in a later PEAK.
in 2005, PC0123319 Fujitsu was able to replicate the scenario and implement a fix on
5 September 2005. The fix was implemented before any branches had switched to
use the new branch trading code, meaning that the issue in PC0121925 could not have
impacted any live branches. This is, however, a bug, error or defect, but not in the
Horizon System as it is used in actual branches.
Entry 10 in the Bug Table therefore does relate to a bug in Legacy Horizon which had
an impact upon branch accounts. As the Post Office submits, in this instance the
impact of that bug was corrected.
Two PEAKs relate to a second element of this, one in February 2006, PC0132133 and
another in March 2007, PC0144386. The Post Office accepts these PEAKs are
essentially the same (one is said by the Post Office to be a manifestation of the other)
and the former is said to “relate to an issue in which the notification mechanism
referred to in PCO121925 was accidentally switched off’. There is no explanation of
how or why this switching off occurred. The KEL that goes with these states in its
title:
“Title: Multiple cash declarations may cause incorrect figures in Discrepancy,
Variance and Balance Reports
Summary: Intermittent misleading figures in Discrepancy, Variance and Balance
Reports”
In the text in the KEL, certain passages make it clear that this is a software issue that
persisted into 2007.
POL-0019320
246.
247.
POL00022841
POL00022841
“Solution - ATOS
<b>Helpdesk:</b><br/>The Declare Cash problem clears itself overnight. If the PM
logs a call on the day he is having problems, ask him to try the following
workaround:<br/><br/>1. The clerk should log out of the affected counter.<br/>2.
Another clerk attached to a different (individual, not shared) stock unit should log into
the <b>same</b> counter, declare cash for his own stock unit, then logout.<br/>3.
The first clerk can now login to the same counter and declare cash again. The variance
should be correctly recalculated.
Alternatively log on to a different counter and do the cash declaration there. If the
workaround is not successful or the problem does not clear itself overnight, send a
call to SSC, otherwise no call is needed. November 2007: a fix is currently being
piloted and is likely to be sent to the whole estate in January
(COUNTER_EPOSS 39.3 or later). If this problem is reported after
COUNTER_EPOSS 39_3 has been applied, send call to SSC.”
(emphasis added)
A fix is only required if there is a bug, error or defect in the software.
The Post Office submitted that there was “no long term impact upon branch
accounts”. This was therefore a slightly different approach to “lasting impact” upon
branch accounts. “Long term” is not defined. If the impact of this bug occurred prior
to a BTS then correction by means of human intervention and TC would be required.
I therefore conclude that this is a bug with the potential to cause lasting impact to
branch accounts. The fact that the impacts of its occurrence were corrected by means
of TCs does not affect its existence. The November 2007 entry in the KEL shows that
a fix was required in late 2007 intended to be released in January (which must mean
January 2008). Accordingly, it cannot be considered to have been limited in its effect
only to the early days of Legacy Horizon and obviously persisted for some years.
11. Girobank discrepancies.
248.
249.
This was a Legacy Horizon issue that occurred between May and September 2000. Mr
Coyne considered this was a bug. Dr Worden did not and stated that “the first fault
concems reports. A fault in a report is not a discrepancy in branch accounts, and only
causes one if it causes a person to make a mistake.” This is now essentially accepted
as a bug by the Post Office, but it is submitted it had no branch impact. It is included
in paragraph 5 of Appendix 2 under the heading “the following bugs had no branch
impact.” The detailed part of Appendix 2 dealing with this submits that there is no
evidence of any financial impact upon branch accounts, let alone a lasting impact.
This was in the early days of Legacy Horizon. There are said by the Post Office to be
six distinct issues arising under this heading.
I do not know if what are called “giros” are still in use at all in 2019, but they look
similar to cheques and used to be widely used, for benefits payments amongst other
things. In basic terms, the giro (which is paper, and was sometimes included in a book
of giros) would be exchanged for cash. A customer would present a giro for (say) £50
POL-0019320
250.
251
POL00022841
POL00022841
to the branch Post Office. The SPM or their assistant would take the giro, giving the
customer the £50 in cash from the till, entering that transaction on Horizon. This
would reduce the cash holding in the branch by £50. The giro would be sent to
Girobank at the end of the day and that would be considered by Girobank together
with the entries that would have come via Horizon. Girobank would then pay the Post
Office the sum of £50 in respect of that particular transaction.
If there was a difference between the value of the giros entered on Horizon, and the
physical giros received by Girobank, then Girobank would not pay the Post Office the
full amount. In other words, there could be a mismatch between Girobank’s view of
the amount of money paid out in exchange for giros (based on the physical giros it
received), and the Post Office’s view (and hence the amount of money paid out of the
branch by the SPM to customers presenting giros).
The summaries of the different issues taken from the PEAKs show the following
problems:
1. Girobank taking the view that there was a discrepancy between giros received from
(say) Branch X, and cash paid out by Branch X for those giros. The explanation for
why this occurred was the “cut off’ time for post being collected from Branch X
(which would include the giros to go to Girobank) being earlier than the end of the
trading day. Branch X might pay out (say) £85 to a customer at 4.45pm, but the post
(including all giro vouchers then available at the branch) might have left at 3.15pm or
4.30pm. This would lead to the difference explained at [250] above. It would also lead
to an error notice being issued by Girobank. The Post Office’s submissions state that
“in this particular case, the only possible impact would be if the branch had accepted
the error notice received because of the reporting issue”. In other words, the only way
the system would work correctly would be if the SPM did not accept the error notice.
The Post Office have to rely upon a SPM not accepting an error notice issued to him
or her to avoid any impact on branch accounts. That is not a software bug, it is true; it
is however an error or defect in the system. The error notices are issued through or as
part of the Horizon System. This type of incident is referred to in PC0044232, which
concerned a £505.72 discrepancy. This was a known issue dealt with by KEL
MWrightS3lp. This KEL is now said by Fujitsu to have been deleted and
irretrievable due to its age.
2. A second issue was identified by Fujitsu in the course of investigation of the
incident in PC0044232. This was that the same £81 giro deposit had been included on
two consecutive daily reports. Part of the Post Office’s submissions on this are as
follows: “This is because the transaction was entered onto Horizon in a precise (and
very small) window of time between two system calls being undertaken, resulting in a
duplication. The overall branch position would still have been correct, but the daily
reports to Girobank may have been wrong. If they were (i.e. if the same transaction
was included on two consecutive daily reports), it is expected that this would have
been spotted and a TC would not have been issued to the branch.” This is, in my
judgment, evidence of a bug, error or defect in the system, as the duplication referred
to should not have occurred. There is no evidential basis for any “expectation” that if
the reports to Girobank had been wrong, it would “have been spotted” and led to the
issuing of a TC. No such TC has been produced.
POL-0019320
252.
253.
254.
255.
POL00022841
POL00022841
Some other issues relating to Girobank were identified by the Post Office separately,
as issues 3, 4, 5 and 6 under the same heading of entry 8 in the Bug Table. Issue 3
applied to two PEAKs, both from 2000. They were PC0052575 (in which the SPM
reported discrepancies of £20 and £628.25) and the issue was diagnosed as arising out
of the use of a shared stock unit. The Post Office submissions were “There is a
window of time between a user printing and cutting-off a report. If another user was
to perform a transaction during that window, that transaction may not show on the
report. The issue was already due to be fixed in a future release”. Mr Coyne accepted
in cross-examination that these were indications of a discrepancy being identified, but
these were not of themselves evidence of a bug in Horizon. His overall evidence on
this was contained in his final answer which was “It created a financial discrepancy
within the Horizon system which could then ultimately have an impact on branch
accounts.” I accept that this issue caused a financial discrepancy. Given the issue was
“due to be fixed” in what is described as a “future release” the issue arose from the
operation of the system and is therefore, in my judgment, correctly described as a
defect.
Issue 4 was identified as follows in the Post Office’s detailed submissions on Entry 8
in Appendix 2:
“26. Issue 4 applies to PC0068633 (referred to at para 3.124). This Peak was raised in
2001 and so comprehensive records are not available.
27. In that Peak, the Subpostmaster reported that his cash account showed two giro
deposits of £1,503 but that his reports showed only one. The Subpostmaster received
an error notice from Girobank which cleared the error, but he raised the issue because
he believed that an error in Horizon was duplicating the transaction.
28. The issue was caused by a cut-off being performed on one counter despite an
attempt to print a transaction failing on another counter. This resulted in the cut-off
teport including the transaction that had failed to print.
29. A fix was actioned by Fujitsu on 18 December 2001.
30. Issue 4 only occurred in a very specific set of circumstances and would have had
no direct financial impact on accounts; it merely had the effect that a transaction was
missing from the reports.”
(emphasis added)
In my judgment, the content of those submissions clearly demonstrate that this was a
bug, error or defect which had an effect upon the report available to the SPM. This is
made very clear in the PEAK itself, which states “Ihave duplicated this bug. In fact it
occurs in all reports that use dataserver (i.e. the majority). I shall now check to see
whether or not the problem still occurs at $10.” (emphasis added) The author of that
PEAK was a Fujitsu employee and expressly referred to a bug. It did not have an
effect upon the branch accounts of the SPM in question but it is undoubtedly evidence
of a bug.
Issue 5 applies to PC0073855 and PC0075312 from 2002. Again, there was not a
great amount of material available due to the age of these PEAKs. A summary is as
follows.
POL-0019320
256.
257.
258.
259.
260.
POL00022841
POL00022841
In PC0073855, a SPM reported that the office snapshot figures were double the
figures on the balance snapshot (with a £6.76 discrepancy). Fujitsu were unable to
replicate the issue and were therefore unable to issue a specific fix. However, a new
version of the component was released with extra tracing code so that if the issue re-
occurred, Fujitsu could gather more evidence. The SPM would have had an
inaccurate report but the correct data was still available and this would not have
affected the branch when balancing. In PC0075312, another SPM raised an issue with
printing her giro deposits. The issue was identified as being caused by the same root
issue as PC0073855 which was already with the development team. These did not
impact branch accounts. However — and I consider this to be important — they
constitute evidence of a bug. The KEL with which they are associated is KEL
AChambers4410R, the same KEL as the PEAK in Issue 4. The only reason the Fujitsu
“development team” would be involved would be to develop a fix. A fix is only
required if changes are required to the software, which means there is a bug.
Finally, Issue 6 applies to PC0076065, which was that two giro deposits were
reported by a SPM not to be showing on the previous or that day’s reports. The SPM
identified the amounts of the deposits (£11 and £24) which were said to be missing,
but Fujitsu discovered that SPM had produced two reports, and on this occasion it was
the SPM who was in error. One report did not have the deposits on it, but the next
day’s report had both. I find that this is not a bug and is an example of user error (that
is, and to be clear, SPM error).
There is an associated issue with giros which is identified in another PEAK, which is
not included in the six issues. I shall reproduce the Post Office’s submissions on this
PEAK:
“The issue in PC0050418 was thought to be the same issue [as in [251](1) above] —
the branch’s largest discrepancy was £323.32. However, due to the length of time it
took for the issue to reach SSC, the branch’s messagestore had been archived — the
Subpostmaster raised the call on 29 June and the call was sent to PINICL on 17 July.
The Subpostmaster was told that they would need to provide further information (such
as the Transaction ID) to allow Fujitsu to investigate. The Subpostmaster does not
appear to have pursued this. The Peak notes that Girobank were going to send an
error notice, but due to the age of this Peak the relevant records are not available and
Post Office is not in a position to provide detailed commentary.”
(emphasis added)
In my judgment, that is prima facie evidence of a shortfall in the sum of £323.32
being caused to that branch’s accounts as a result of this defect or error,
notwithstanding that the SPM reported it. The SPM was told further evidence was
needed in order to investigate. It is a self-contained example that in my judgment
supports the claimants’ case on the existence of a bug, error or defect affecting
Girobank issues. The fact that the SPM for whatever reason did not pursue it does not
affect that.
Although the very numerous PEAKs identified in respect of this bug in the Bug Table
share some common features, as can be seen above there are different issues in
respect of many of the PEAKs. This was counted as a single bug in the Bug Table,
even though my detailed findings above show that there were more than one bug,
error and defect that related to Girobank. I will however for consistency count it as a
POL-0019320
POL00022841
POL00022841
single bug in the overall total, as that is how the experts treated the total number of
bugs in the Bug Table.
12. Counter-replacement issues.
261. This also is a Legacy Horizon issue. The counter is part of the hardware of Horizon.
262. There were two KELs associated with this dealt within the bug table. The first was
created in 2000 and last updated in July 2007, and refers to 88 PEAKs. The second
was created in 2002 and notes occurrences running up to 2009. It is now accepted as a
bug by the Post Office in paragraph 6 of Appendix 2, but is said to have had transient
impact. Mr Coyne considered that when replacing a counter within a branch, the
process could result in “the total loss of a transaction”. Dr Worden stated that the
cause, recorded in the first KEL (which was created in December 2000 and last
updated in July 2007) was that Riposte was coming online from the Recovery mode
too early, and causing messages to be overwritten.
263. In theory, when a counter was replaced, it builds its messagestore by replicating with
its neighbours in “recovery mode”. The neighbours it has depends on the office size
(which would affect the number of other counters) and node number. For a single
counter office, the neighbours are the correspondence server in the datacentre and the
mirror disk (the second hard drive in the same counter). For a multi-counter office,
the neighbours are the correspondence server and all other nodes at the office, or all
the other nodes in the office (known as slaves) depending upon the node number of
the counter being replaced.
264. A replacement counter is supposed to come out of recovery mode when it believes it
has successfully replicated all relevant messages from its neighbours. The Post Office
submissions state that “In this case, the replacement counter came out of recovery
mode early, before it had replicated all messages from its neighbour. The
replacement counter started writing messages from the point at which it believed it
had replicated all relevant messages from its neighbour. This meant that it used
message IDs that had been used for messages that had not been replicated from its
neighbour and this prevented the “missing” messages from being replicated later on
(because that would have created duplicate message IDs). The missing message was
therefore “overwritten” by the replacement counter.”
265. The Post Office also stated that “the issue arose in cases of counter replacements
where the new counter was not connected to all of its configured neighbours while
rebuilding. This may have been because the branch infrastructure was not complete
(eg not all neighbouring counters are online, multiple swaps or a counter
increase/decrease occurring) or the engineer did not connect the system properly.”
Regardless of the reason, the failure to replicate messages is a failure in the internal
processes of Horizon to cope with the hardware replacement of the counter, and in my
judgment is a bug, error or defect.
266. PEAK PC0052823 gives a technical explanation for this, part of which is:
“It would appear that after the recovery from the squirrel completed, the message
processor came out of recovery mode after synchronising up to message number 660
for node id 1. Suspect this was by replication with the Correspondence server.
POL-0019320
267.
268.
269.
270.
271
POL00022841
POL00022841
Counter then wrote a Riposte on-line message as 661 for node I before I second later
attempting to synchronise to the F: drive mirror message store. At this point a red
event regarding 'self originated message’ was generated, the server switched back to
recovery mode and the remaining messages from the F: drive mirror message store
above 661 were synchronised.”
It also notes that “Gareth Jenkins viewed this error on rig with Mike Berrisford.”
The nature of the correction in the KEL was stated as being “to find the overwritten
transactions for reconciliation we need to look at the Ripostemirror messagestore’
followed by detailed instructions”. Riposte, as has been explained, was part of Legacy
Horizon and was a product provided by Escher, but it is plainly part of Horizon. Dr
Worden also stated that “the incident arose from a hardware replacement (probably
from a hardware fault) not from a fault in Horizon. It is a different kind of recovery
issue.” In my judgment the hardware is part of Horizon.
There was a short term and long term fix. The former dealt with the actual branch by
the SSC inspecting the Riposte mirror messagestore, and retrieving the specific
messages which had been overwritten. Information of the overwritten messages was
passed to MSU who created a BIMS report for the Post Office. The Post Office
submits that “an error notice would have been issued to hold the branch harmless
thereafter”; no such error notice has been produced but I consider that the KEL and
PEAK together suggest that such a notice was produced in order to correct the impact
upon branch accounts.
The long-term fix for the issue was an actual fix, which is detailed in PC0052823.
This involved enforcing a minimum number of local neighbours for replication and
then to slowly lower this number over time. A further change was also made to stop
Riposte writing messages as it came online. This further change was a change to how
Riposte operated. This therefore means that the fix changed the software of the
system.
The dates of other PEAKs shows that this persisted.
PC0071836 is accepted by the Post Office in Appendix 2 as an example of the same
issue as KEL JBallantyneS328R; this is dated November 2001. This SPM had a
receipts and payments mismatch of £3.27 as a result of three overwritten messages
following a replacement of the branch’s single counter. The same fix was applied
following KEL JBallantyneS328R. PC0133822 is not the same issue as
JBallantyneS328R but is accepted as being related. This PEAK is dated 24 March
2006. The branch had two counters removed, leaving it as a single counter branch.
However, the counter did not have a mirror disk; the mirror disk was a second hard
disk within a single counter that provided a replication neighbour for the main hard
disk messagestore. This meant that the branch would have no replication of data if it
was not connected to the datacentre and six messages on the counter had not been
replicated to the data centre. The messages were extracted and sent to MSU for a
BIMS report to be raised. The Post Office submits that “An error notice would have
followed the BIMS report and so there would have been no lasting impact on branch
accounts”. However, that effectively accepts that absent the error notice, the branch
accounts would have been affected. PC0153851 is not the same issue as
JBallantyneS328R but it does involve a receipts and payments mismatch; it is dated 7
POL-0019320
272.
POL00022841
POL00022841
February 2008. Riposte failed to index four messages resulting in some items being
missing from the receipts side of the balance report. The PEAK itself notes that the
branch did not experience a discrepancy as a result because this was a reporting issue
only; indexes are not used when replicating data and so cash/stock were unaffected.
This does however show that the problem with Riposte persisted.
I find that this was a bug, error or defect with potential lasting impact to branch
accounts. The fact that it originated with the necessity to replace a counter (which is
plainly hardware, as Dr Worden notes) does not matter because the hardware is part of
the Horizon System. Nor does the fact that the impacts of its occurrence were
corrected by means of error notices/TCs affect the existence of the fault. A change, or
changes, had to be made to Riposte to stop this occurring, and there were still later
manifestations of it occurring.
13. Withdrawn stock discrepancies.
273.
274.
This is a Horizon Online issue. The Post Office does not accept these as a bug. Mr
Coyne maintains that it is, and in the bug table he extracted part of a PEAK that stated
these “Can cause confusion and unexpected (though hopefully temporary)
discrepancies at branches by allowing them to declare stock which has already been
withdrawn.” Dr Worden stated that “some impact on branch accounts cannot be ruled
out, although it is small”. The Post Office’s detailed submissions in Appendix 2 of its
Closing Submissions concentrated on the fact that when the Post Office withdrew
stock — which of course is part of the way that the Post Office manages its business,
adding and removing types of products from time to time — this removal of products is
done by means of an update to reference data, “and not a change to the core code in
the system.” This again is concentrating on the code, rather than the way the system
operates.
There were two steps that led to this happening. The Post Office submissions on this
tather gloss over the entries in the PEAK, so I will reproduce the actual entries of 19
January 2011 in PEAK PC0207834. Although that is the date of PEAK, the text
makes clear that the event occurred in November 2010. The context is that the Post
Office withdrew the £5 saving stamp from its business, but obviously some branches
retained (or had at the time they were withdrawn) such stamps in their stock:
“This office physically held 137 £5 PO saving stamps and did not rem them out
before the date the rem out icon disappeared. The office physically returned the
stamps to Transaction Processing as advised and the office then did a Trading Period
balance on 17/11/2010 and showed this value as a loss. The spmr put the cash into the
till_at_this point to make good the loss. Transaction Processing then issued a
transaction correction to the branch on 19/11/2010 for the value of £685 to make good
the loss. The office brought this to account and when they did the next Trading Period
balance on 15/12/2010 it showed a gain of £537.49 which was settled centrally. The
spmr says that they did have a loss showing again in the balance for the value of the
PO Saving stamps, £685, again, even though this stock line was not showing in either
adjust or declare stock. After this Trading Period balance the spmr took the £685 back
out of the till and when they balanced again on 12/01/2011 it again showed a loss of
137 £5 PO Saving stamps and the nett discrepancy in the branch was £1370, which
again was settled centrally. The office declares stock when they do the balance but
insist there is no line showing in either adjust stock or declare stock for the PO Saving
POL-0019320
276.
POL00022841
POL00022841
stamps. The spmr is convinced the £1370 loss, which is twice the £685, is due to the
fact these stamps are still on the system somehow and keep giving the office the
losses. The office has faxed_a copy of their stock holdings for the branch at the
moment and it clearly shows “Saving Stamps £5 137”. This stock line should have
been removed when the stock item was made obsolete and removed from Horizon.
The two user names who do the balance are SKI001 and PCAO01
I have spoken to Phil Herrett in Transaction Processing who has confirmed he is
aware of about 8 offices with similar issues with the stamps still showing on the stock.
He gave me FAD 0640123 as an example.
Can this be investigated to see why the stock is still showing in the office, and if this
keeps giving the office a loss of £685 every time they do a Trading Period balance.”
(emphasis added)
The Post Office submissions state that “However, the Subpostmaster in this case
returned the stamps (with a total value of £685) to Post Office without first remming
them out. This aspect of PC0207834 was pure user error.” However, that is not what
the first entry in the PEAK states. It states that the SPM could not rem them out
because the rem out icon had disappeared for this stock. That is undoubtedly an error
by the SPM. However, the fact that the withdrawn stock remained on the stock
holdings, which are part of the branch accounts, even after the stock has been returned
and a TC issued to correct the discrepancy for their value, cannot be laid at the door of
the SPM. Further, other offices (“about 8” of them) were also experiencing the same
issue. This is an error or defect with Horizon in my judgment. Part of the submissions
by the Post Office states as follows:
“8. The branch declared a £685 shortfall. The lack of a rem out meant that Horizon
thought that the branch was still holding the stamps and therefore when the
Subpostmaster either declared or adjusted the stock of stamps to zero without a
corresponding gain in cash, a shortfall was generated. The Subpostmaster elected to
make good the shortfall and a credit transaction correction was subsequently issued
for £685 to rectify the issue.
9. However, a bug in Horizon caused the £685 of stamps to be subsequently re-
introduced into the branch’s accounts on two occasions. By this point, Horizon was
showing that the branch was holding £1,370 of the stamps. The Subpostmaster
noticed this, adjusted or declared the stock as zero, and reported the issue.”
(emphasis added)
The Post Office therefore accept in those submissions that the subsequent effect upon
the accounts was a bug in Horizon. I do not consider it much matters whether this is
characterised as a bug, or an error, or a defect in Horizon. It plainly was a bug that had
the potential to cause lasting impact to branch accounts (and in this case actually did
so prior to the issue of the TC).
14. Bureau discrepancies.
POL-0019320
277.
278.
279.
280.
281.
POL00022841
POL00022841
These relate to foreign exchange, hence the name. It must be differentiated from
alleged bug at entry 23 in the Bug Table (also concerning foreign currency, but
entitled Bureau de Change). This is a Horizon Online issue and arose in 2017.
Mr Coyne considered it to be a bug, and Dr Worden effectively agreed in that his
entry stated “This appears to be a system error with impact on branch accounts.
Although it is possible that a subsequent discrepancy between branch accounting and
POLSAP would reveal the problem, leading to a correction (e.g. see Peak
PC0265443, and Mr Coyne's para 3.146), I cannot be certain of this.” It is now
accepted by the Post Office in paragraph 7 of Appendix 2 as one of a number of “bugs
with lasting impact (although they were resolved)”. The detailed part of Appendix 2
states that Mr Coyne has drawn together two distinct issues. Paragraph 4 of the
detailed submissions also states the following:
“Bug 14: Bureau Discrepancies is a bug with the potential for lasting financial impact.
There are two distinct issues which fall under this heading. With regards to the first
issue the branch was made good and a fix was implemented. The second issue was not
a bug in Horizon nor an issue which could have impacted branch accounts; it created
what was essentially a cash flow problem for the branch.”
So far as the first issue of the two identified by the Post Office is concerned, given a
software fix was required, it is undoubtedly a bug, and it is correct to record (as the
Post Office do) that there was the potential for lasting financial impact, as without
that, there would be no need for the “branch to be made good” because that means the
Post Office had to correct the branch account discrepancy caused by the bug. In a
sense that is stating the obvious. Given those submissions, I will deal with the first
issue very briefly.
This occurred as follows. In August 2017 an SPM tried to pre-order £1,000.07 in
Indonesian rupiah and £204.59 in Singaporean dollars for a single customer. The
tupiah order was created, but there was a network timeout at the point when the SPM
tried to perform the dollar order. When the system re-connected, a warning message
suggested that the second order may not have been placed, but the basket and
transaction log were showing both orders. The SPM attempted to cancel the whole
order, but the cancellation only worked for the rupiah order, leaving the dollar order
of £204.59 in the branch accounts. This is because as recently as August 2017
multiple currency orders were processed as multiple transactions. The rupiah order
was added first at the counter and then sent to the BAL, and this caused the order ID
PBX1048411 to be created. The network timeout occurred at this point — such a
timeout has nothing to do with the SPM. At that time that the dollar order had been
added to the counter stack but it had not yet reached the BAL. This meant that it did
not become associated with order PBX1048411 and so, when the SPM attempted to
cancel the whole transaction (PBX1048411), the dollar order was not cancelled. This
then created a shortfall as Horizon will have expected the SPM to have taken a
payment from the customer of £204.59.
This issue could only occur because of the very specific circumstances of PC0261541
including the occurrence of a network timeout at a very precise moment in the
transaction. The Post Office’s detailed submissions state the following:
POL-0019320
282.
283.
284.
285
286.
POL00022841
POL00022841
“The PEAK notes that the issue was referred to Post Office in order for them to
“decide what reconciliation or transaction correction is required to balance” at the
branch. Post Office have confirmed that a transaction correction was issued to the
branch on 2 November 2017 and accepted by the branch on 7 November 2017 — this
is not apparent from the PEAK itself.”
This means that the accounts of that branch were incorrect, and in the Post Office’s
favour, for the period mid-August to mid-November 2017. A change to the software
was made in November 2017 via a change to the AP-ADC script which had the effect
of simplifying multiple currency orders to be processed as a single transaction.
The second issue is even more marked, in my judgment, in terms of arriving at the
correct answer to Horizon Issue 1. PC0265443 is dated 29 December 2017 but relates
to a call on 17 December 2017. The Post Office’s internal cash management team’s
system (which is referred to as POLSAP) indicated that the branch in Burgess Hill
was holding €4,500 and US$1,000 more than the branch was declaring it held. This
meant that, from the cash management team’s perspective, the branch held sufficient
cash to conduct transactions, and so were reluctant to provide more cash to the
branch. This resulted in a cash flow issue for the branch who were sometimes unable
to serve customers as a result. This PEAK is 57 pages long; this is many pages longer
than most PEAKs, which are usually 10 pages or less. It evidences a very long
investigation.
This investigation is best summarised by quoting the Post Office’s own submissions
under the heading “Resolution”:
“13. The Peak is long and complex and it demonstrates that the issue was extensively
investigated by Post Office, ATOS, Accenture and Fujitsu, since the issue was
thought to be occurring somewhere between the branch, Horizon and POLSAP. The
conclusion of Fujitsu and Accenture’s investigations (over approximately six months)
was that the issue was not caused by any issue within their respective systems.
Further, Post Office sent a trainer to the branch to verify the cash position and found
no discrepancies; the amount of cash physically held in branch was the same as the
amount showing on Horizon (but not POLSAP).
14. The root cause of the issue does not appear to have been determined. As it did not
appear to be a case of user error, it appears that Post Office wrote off the discrepancy
in POLSAP at no cost to the Subpostmaster to bring that system’s figures into line
with those in Horizon.”
(emphasis added)
This shows that POLSAP was not in line with Horizon; that no problem could be
found by Fujitsu with its system, nor by Accenture; but that the amount of cash (in
terms of foreign currency) in the branch was being correctly declared by the branch.
Not only do I find that this is evidence that supports the existence of this bug in the
Bug Table, which potentially causes lasting impact to branch accounts, but it is also
consistent with my finding on a separate issue, namely the need to use ARQ data in
circumstances where there are serious disputes between the Post Office and SPMs,
rather than Post Office management data
POL-0019320
POL00022841
POL00022841
15. Phantom Transactions.
287.
288.
289.
290.
291.
292.
These arise under Legacy Horizon in terms of the dates. There are three different
issues grouped under this heading. The Post Office does not accept these as a bug. Mr
Coyne in his cross-examination stated that this was referable to Horizon Issue 4,
rather than Horizon Issue 1. He also accepted that some of the issues were not bugs in
Horizon. Dr Worden relied upon the fact that what he called “the master PEAK” was
closed as “no fault in product”; however, the evidence of fact on this is dealt with in
the substantive judgment where I consider the evidence of Mrs Van Den Bogerd when
the PEAK was put to her.
Dr Worden stated that “There is no evidence for bugs in Horizon with impact on
branch accounts.” That entry was obviously before the cross-examination of Mrs Van
Den Bogerd, which in my judgment provided greater factual information. I do not
consider reliance can be placed upon the Fujitsu conclusion in the PEAK.
PEAK 0065021 is dated 17 April 2001. I consider it to be an important PEAK. It
relates to multiple branches. It concerns phantom transactions. It identifies
dissatisfaction from more than one SPM as to how phantom transactions are being
investigated and resolved, or more accurately, how they are not being. It shows calls
being “closed” — ie brought to an end, without the permission of the SPM, even
though that should not be done. It also shows at least one SPM threatening not to
comply with their contractual obligations due to lack of confidence in the system, and
also threats of legal action. Further, in one branch, where items were the subject of
phantom transactions (according to the SPM) ROMEC, the Royal Mail’s own
engineers, attended that branch to fit suppressors and other equipment in an effort to
rectify this.
The PEAK plainly records the involvement of ROMEC, the Royal Mail’s own
engineering personnel, as follows. “ROMEC have been to site and state that they have
actually seen the phantom transactions, so it is not just the PM's word now.”
(emphasis added). I consider that this is significant. Mrs Van Den Bogerd agreed that
this was “independent site visit corroboration of the problem by Royal Mail’s own
engineers at the branch”, and she also agreed that this was “clearly not user error any
more”. This entry in the PEAK is, in my judgment, important corroboration from
those with experience of Horizon (the Royal Mail’s own engineers) who state they
have seen the phantom transactions. The conclusion reached by Fujitsu and recorded
in the PEAK was as follows:
"Phantom transactions have not been proven in circumstances which preclude user
error. In all cases where these have occurred a user error related cause can be
attributed to the phenomenon." The PEAK also concludes “No fault in product".
I reject both of those conclusions. In my judgment, they are both self-serving and are
contradicted by the factual matters reported within the PEAK by Fujitsu, which
include that these have been witnessed by ROMEC engineers.
A Fujitsu entry from 19 June 2001 in PEAK PC0065021 states:
“T now have pressing evidence to suggest that unwanted peripheral input is occurring.
the likely source being the screen. This has been seen at Old Iselworth (OI) and
POL-0019320
293.
294.
295.
POL00022841
POL00022841
Wawne (W) with OI being the best site; when the PM has been asked to leave the
screen on overnight I have observed system activity corresponding to screen presses
happening with no corresponding evidence of either routine system activity or human
interference, the way forward now is to correlate this with the Microtouch supplied
monitoring software and to this ends Wendy is arranging for installation of Kit at Ol
on Friday, we can then, provided the PM agrees, leave screens on over the weekend
and record what happens.
Once these results have been analysed I feel sure that we will be in a position to move
forwards at OI. All other cases should be considered on their individual merits but
you must appreciate that this is a fairly intensive analytical activity and I cannot hope
to provide answers on all cases in the short term.”
I consider this to be evidence of a bug in Horizon. Mr Coyne notes that there is no
specific branch account discrepancies noted in this respect; however, given Fujitsu
concluded in November 2001 the following “Phantom Txns have not been proven in
circumstances which preclude user error. In all cases where these have occurred a user
error related cause can be attributed to the phenomenon. I am therefore closing this
call as no fault in product” then the lack of noted discrepancies for this is not
surprising. Nor, in my judgment, is it necessary for there to be specific identified
discrepancies given the word “potential” in Horizon Issue 1.
Both the ROMEC engineers who had observed the phantom transactions, and indeed
Mr Carroll in his entry of 19 June 2001, considered based on the evidence they had at
that time that there were so-called phantom transactions occurring. Mr Carroll’s entry
in particular that states “when the PM has been asked to leave the screen on overnight
I have observed system activity corresponding to screen presses happening with no
corresponding evidence of either routine system activity or human interference” can
only sensibly be explained by a bug. The Post Office submitted that “For the reasons
set out above [in its submissions], there is no indication that phantom transactions had
a cause other than user error, as indicated in the PEAKs.” However, that ignores the
entries in what was referred to as the master PEAK that indicate precisely the
opposite.
I consider that an inference, that is to say a common sense conclusion, can be drawn
from all the evidence on these matters that there was such a bug in 2001 that caused
phantom transactions. This had the potential to cause impact to branch accounts.
16. Reconciliation issues.
296.
This heading in the Bug Table grouped both Legacy Horizon and Horizon Online
issues with reconciliation together. The Post Office does not accept these as a bug.
There are a number of different issues grouped together in this heading. The issue was
that the SPM was shown a discrepancy on his or her screen. Mr Coyne accepted that
the discrepancy would not be shown in the branch accounts; Dr Worden stated that
“as it concerns an issue in reporting, the software fault (which was fixed after 5
months) had no direct impact on branch accounts. The only effect of an error in this
report would be to mislead or confuse the Subpostmaster - probably leading him to
check his figures more carefully and costing him some time.” (emphasis added)
POL-0019320
297.
298.
299.
300.
301
302.
POL00022841
POL00022841
That entry by Dr Worden plainly accepts a software fault; the fault was in fact a
miscounting of the number of files by the system. However, there are six different
PEAKs grouped under this heading and the process of reconciliation, which is
effectively the comparison by Horizon of two different sets of data, is part of its
design function. The recovery messages were held in the branch messagestore. Mr
Coyne in his cross-examination also stated that this was referable to Horizon Issue 4,
rather than Horizon Issue 1. Three of the six different issues were missed off the copy
of the submissions lodged for oral submissions in July 2019, but this was made clear
by the omission of “Issue 4” to “Issue 6” and following submission of Mr Parsons’
19" witness statement these were added.
This is one area where the cross-examination of Mr Coyne in particular does lead to a
conclusion that this is not a bug that causes potential lasting impact to branch
accounts. The six matters identified by the Post Office in the detailed part of
Appendix 2 all deal with different PEAKs.
Peak PC0039832 is dated 3 March 2000 and is an example of an issue affecting a
SPM’s Cash Account Period. Reconciliation discrepancies appeared but did not
feature on the expected reports. The counter calculated information each day for the
Cash Account in two independent ways to ensure that they matched and a report was
generated if they did not match. In this case, the reconciliation software detected
discrepancies relating to two low value transactions (£8.06 and £0.08, totalling £8.14)
The PEAK suggests that “there was a bug in the reconciliation software, although the
Peak is not fully conclusive” as the Post Office’s submissions recognise in this
sentence that I have quoted. This would have generated a false reconciliation report
but there would have been no financial impact on the branch. The reconciliation error
was captured, as that is what triggered the investigation in the PEAK. The fix is
documented in PC0047955 of 19 June 2000. It is evidence of a bug but given the
nature of it, it would not have impacted upon branch accounts.
PEAKs PC0075240, PC0075415 and PC0077508 are all from April 2002. These all
relate to the same (or at least a markedly similar) issue where a branch counter total
differed from the amount on the TPS host. There was no discrepancy in branch
accounts. In all these cases, the counter calculated a value of Ip and the calculations
carried out at the host gave a value of £0.0099. As the first three digits were 000 the
system took this to be Op and so reported a false discrepancy. There was however no
financial discrepancy. A fix was however required, which demonstrates it was a bug,
error or defect, but it was not one with the potential to affect branch accounts.
PEAK PC0049578 is from July 2000. It was raised in testing and fixed before the
software went live. It was a bug, error or defect but the purpose of testing is to
discover such things, and as such this had no potential to impact branch accounts. The
problem was producing a report used to confirm that all data has been passed to TIP.
The underlying data transferred to TIP was correct, however, the number of files
transferred did not match. There was also a report to confirm what files were
transferred, and that report was also incorrect in the count of files transmitted. If this
had gone to live then the answer to this might be different, however regardless of that,
it was fixed before release.
The fourth issue relates to a PEAK dated May 2000. The submissions stated that “the
Peak describes an automatic detection of a receipts and payments mismatch of
POL-0019320
POL00022841
POL00022841
£4,464.46. It was automatically reported to Fujitsu and was investigated. The issue
related to a hardware failure. It therefore did not relate to reconciliation issues arising
as a result of a software fault.” It is however a defect or error with the Horizon
System, which comprises both hardware and software.
303. PEAK PC0236246 is dated 12 September 2014. It relates to a discrepancy of
£110,706. The Fujitsu entry in the PEAK is somewhat illuminating:
“This problem is likely to occur whenever products are introduced with non-midnight
times, a common occurrence when data is created (and subsequently corrected) in
MDM. Based on current data, such products may be seen roughly once a month.
When it does occur, any transactions against the products are omitted from the CTS
report, resulting in discrepancies in trading totals, and operations are required to
investigate, usually requiring the assistance of development.
The CTS Report is used by POL to settle payments with their Clients. Incorrect values
on the CTS report cause Post Office Clients to lose confidence in POL Accounting
resulting in loss of face to our Customer. POL also therefore have loss of confidence
in the precision of our IT Systems and this is an especially sensitive issue whilst POL
are being investigated for accounting accuracy.
POL"'s lack of confidence in our IT Solution will not help our bid for the renewal of
the Front Office contract.”
(emphasis added)
304. CTS stands for Client Transaction Summary. In my judgment, the error shown in this
PEAK is plainly a software fault that affects the accuracy of accounting at the Post
Office. However, in this case, it has an effect upon the CTS Report, used to account
between the Post Office and its clients, and not branch accounts. There was therefore,
under this heading, a software fault, however it had no potential to impact upon
branch accounts for the reasons I have identified above.
305. The final of the six different issues also relates to the CTS Report, but one of the two
PEAKs is that which I have already reproduced at [303] already. The other is PEAK
PC0204872 and is dated 4 October 2010. The Post Office submissions state that “the
issue relates to a difference between the CTS Report and the branch figure. The root
cause identified was an issue with POLSAP, and not with Horizon”. The PEAK
records that there was a problem with the CTS report for a branch concerning the
Alliance & Leicester:
“The CTS report is received daily and is compared with the vendor (in this case A&L)
reports. The figures for each day should match.
If the CTS report is larger than the vendor figure, the vendor account will be credited.
The credit usually shows a couple of days later as a positive discrepancy.
The CTS report was showing as being larger than the vendor figures on the following
dates, although there does not appear to have been any counter credit showing on the
vendor figures following on from this:
POL-0019320
306.
307.
308,
POL00022841
POL00022841
7th May 2010 - CTS was greater than vendor figures by £84.86. POL have suggested
that this may have been related to an event from 27th February for FAD 490519,
although we can find no BIMS record of this from a Reconciliation perspective.
25th July 2010 - CTS was greater than the vendor figures by £3,260.00. No additional
information is available.
27th August 2010 - CTS was greater than the vendor figures by £846.00. No
additional information is available.”
The next entry is:
“Following this additional information this would appear to be an issue with
POLSAP.
The SSC do not support POLSAP, so we can take no action here.”
That submission and the entry in the PEAK is of assistance in dealing with the ARQ
data/management data dispute, but does not necessarily evidence a bug in Horizon
potentially impacting branch accounts. It does not, however, for obvious reasons, take
the matter very much further forwards.
I find that these PEAKs, although demonstrating software faults, do not show that
there is a bug with the potential to cause impact to branch accounts
17. Branch Customer discrepancies.
309.
310.
311.
This was a Legacy Horizon issue and is an entry only in respect of Horizon Issue 4
and recorded as such in the Bug Table. The Post Office does not accept these as a
bug. Mr Coyne accepted in his cross-examination that the entry in the PEAK did not
suggest any impact upon branch accounts. The entry in the bug table originated in a
single PEAK, although there were two PEAKs that relate to the same issue, which
was a discrepancy between the financial records held by the Post Office (from
Horizon) and a bank’s records, after a counter crashed whilst a transaction was being
processed. The amount in question was £165.26. The date of the PEAK is 29 March
2008 and the incident occurred between 22 March 2008 and 26 March 2008 (the date
of the card account).
The SPM did not take the sum in question from the branch customer, but that person
had their account debited. The first PEAK was raised by Fujitsu who picked up on
this as part of automatic reporting (an NB102: Exception Summary report) and the
SPM was advised how to go through recovery. Fujitsu had issued a BIMS Incident
Report to the Post Office, and had confirmed that the recovery messages were in the
branch messagestore (even though they do not appear to have been shown by Horizon
to the SPM in the first place). However, the branch appeared in the NB102 report
again, and upon investigation this was found to be because the SPM (quite properly,
in my judgment) had not followed the recovery messages because they had not taken
the sums from the customer.
This is effectively accepted by the Post Office in its submissions which state: “Fujitsu
issued a second BIMS Incident Report to Post Office which advised that while the
recovery messages were received by the SPM, the SPM had declined them. This
POL-0019320
312.
313.
314.
POL00022841
POL00022841
indicates that no money changed hands during the transaction (i.e. the SPM elected
not to recover the transaction because no money had changed hands; if money had
changed hands, the SPM should have elected to recover the transaction).”
The Post Office also submits the following:
“While there was a possibility that the Financial Institution registered a withdrawal
from the customer’s account (depending on how the Financial Institution’s IT system
is configured), this would have caused a loss for the customer, not the branch
(assuming that the customer received no cash from the branch). If the customer did
receive cash from the branch, the resultant discrepancy would have been caused by
the branch not following the recovery procedure correctly.”
I find that a very surprising submission. The counter — part of Horizon — crashed
during a transaction. That is not the fault of the customer, and it is not the fault of the
SPM, nor is it the fault of the customer’s bank. I do not know if solace is taken by the
Post Office that any loss caused to the branch customer can, apparently, be laid at the
door of the customer’s branch, but I doubt solace can be properly taken. Nor do I see
how the SPM can be criticised for “not following the recovery procedure correctly” as
it is accepted that this should have occurred if the SPM had taken money from the
customer, but if they had not, it should not have been followed. On either analysis,
this is a fault with Horizon.
However, it does not appear to have occurred more than this single occasion. I agree
with Mr Coyne, and this is relevant to Horizon Issue 4 only. I consider that it is an
error or defect in Horizon, but does not constitute one that had the potential to cause
impact to branch accounts.
18. Concurrent logins.
315.
316.
This is a Legacy Horizon bug. It is accepted by the Post Office in paragraph 7 of
Appendix 2 as one of a number of “bugs with lasting impact (although they were
resolved)”. In paragraph 2 of the detailed part of Appendix 2 dealing with this bug,
the following is stated by the Post Office: “Post Office accepts that Bug 18:
Concurrent Logins had a potentially lasting financial impact. There is no evidence of
any discrepancy in the PEAKs referred to by the experts.” The problem was in the
early days of Legacy Horizon when it was possible for users to log in to two terminals
at once. Dr Worden in the bug table made the following useful summary statement:
“discrepancies could occur - manifesting themselves as a_receipts/payments
mismatch. This had the potential to affect branch accounts. The mismatch would
bring it to the attention of the Subpostmaster, who would require it to be investigated,
except possibly in the case of small mismatches, which he might pass off as an error
in the branch (e.g. of counting stock).”
He also stated that “....Fujitsu believed it was a problem with the underlying Riposte
software, and passed it to Escher. In September 2000, the problem was ‘Now formally
fixed in Build 223 update 19 which was released overnight.’ However, the new release
from Escher did not, as it was expected to, fix the problem. Escher denied that it was a
bug in Riposte, but Fujitsu believed in July 2001 that 'This is clearly a bug in the
Supplier's code’.
POL-0019320
317.
318.
319.
320.
321.
POL00022841
POL00022841
(emphasis added)
Paragraph 2 of the detailed submissions by the Post Office in its Appendix 2 state:
“[The] Post Office accepts that Bug 18: Concurrent Logins had a potentially lasting
financial impact. There is no evidence of any discrepancy in the PEAKs referred to by
the experts.”
Dr Worden considered that “these faults had the potential to produce discrepancies in
branch accounts, of small amounts, for a short period of time”. How the bug is
therefore characterised depends upon my findings in relation to lasting and transient
impact, which is dealt with in the substantive judgment to which this is an appendix. I
find that this was a bug, with the potential to cause lasting impact to branch accounts.
Dr Worden attempts to play this down and so, in my judgment, does the Post Office.
Fujitsu clearly recognised and recorded that in 2001. The fact that the bug was in the
code supplied by Escher, as part of Riposte, and not that of Fujitsu is wholly
irrelevant for the purposes of the Horizon Issues.
One of the PEAKs related to the printing of a report whilst a counter crashed. The
Gateway counter was then used, and because the first counter had crashed (what is
called a “slave” counter) this was permitted because it could not communicate to the
Gateway counter who was logged in to it, or what it was doing. This PEAK is
PC0027581 and relates to an incident in July 1999. That PEAK runs to February
2002. It is notable, and continuing the theme of Fujitsu’s closure codes, that it is
closed with the code “Administrative Response” even though the entry for 13 July
2001 states “This is clearly a bug in the Supplier’s Code and as you say, management
pressure must be brought to bear as necessary to make the Supplier accept and
respond to that fact.”
The second PEAK is PC0051327 dated 2 August 2000. The entry which opened the
PEAK is:
CALL PC0051327:Priority B:CallType L - Target 02/08/00 12:32:13
28/07/00 12:22 office 182432 reports for CAP17 a receipts total of £412224.58 and a
payments total of £430724.58. the difference is £18500.00. this office earlier raised a
query because a transfer for an amount of £9250.00 seemed to have gone missing. the
amount of the transfer _is exactly half the amount of the difference between the
receipts & payments. a call raised earlier PC0050974 was closed in error.
28/07/00 12:27 GB082158
Information: passing to EDSC as requested.”
(emphasis added)
A further entry is:
“PM advises that the problem started when she transferred from counter No8 (by
mistake - should have been from counter Nol) £5590 to counter No3 and £3660 to
counter No4 - while No8 was in the process of rolling over! Consequently, the cash
left counter No8 in CAP 18 and arrived in counters Nos3 and 4 in CAP 17. She
POL-0019320
322.
323.
POL00022841
POL00022841
eventually rolled the office, accepting the £9250 discrepancy and then transferred
£9250 from counter Nol to counter No8. How was she able to transfer out of counter
No8 while it was rolling over?”
Part of the Post Office submissions are:
“19. The net difference of these receipts and payments mismatches amounted to zero,
meaning there was no overall impact on the branch. This would have been clear to
Mr Coyne as the very nature of this issue is not that it causes money to be lost, but
money to be accounted for in the incorrect time period. The PEAK shows that there
are offsetting receipts and payments gains and losses and therefore it is possible to
ascertain from the Peak whether the branch suffered a discrepancy.
20. The issue was fixed through a planned software roll out that changed the code in
the area that caused the bug (release C145) and the call was closed on 30 November
2000.”
(emphasis added)
That applies to the second PEAK but not to the one headed “Simultaneous Logon”
which is PEAK PC0027581, which was closed in February 2002.
I find that this was a bug in Horizon which plainly had the potential to cause impact to
branch accounts. The fact that in this specific case the discrepancy was later corrected
does not, in my judgment, alter that characterisation. Indeed, on the Post Office’s own
case, the accounts were affected (albeit corrected). Further, the way that PEAK
PC0027581 unfolded — a lengthy PEAK which runs from July 1999 to February 2002
— shows the unsatisfactory way some of these incidents were dealt with at Fujitsu. The
entry of 9 November 1999 states “....at first glance it could have a fair bit of business
impact”. Even as late at August 2001 and January 2002 this PEAK was still given the
code “Incident under Investigation”. One wonders quite how long Fujitsu would need
properly to investigate something identified as “a bug in the Supplier’s Code” in July
2001. The final entry contains the sentence “Mr Lui [who is the SPM] is no longer
employed by the Post Office and has not been for some years.”
19. Post & Go/TA discrepancies in POLSAP.
324.
325.
326.
This is during Horizon Online and occurred in 2012. It is accepted as a bug by the
Post Office in paragraph 6 of Appendix 2, but is said to have had “transient” impact.
The entry by Mr Coyne in the Bug Table relates to Horizon Issue 4 rather than 1.
“Post & Go” are self-service terminals, that are no longer available in branch Post
Offices, and are only available in Crown and WH Smith main branches. They are
basically self-service kiosks, designed to avoid queuing. A customer can weigh a
letter or parcel, the terminal will print the relevant stamps/labels, the customer deals
with that themselves, and the item is then posted without the need for queuing. The
Post Office submitted that this issue was irrelevant to the Horizon Issues trial as “this
issue does not relate to branches that are the focus of this trial”.
I do not consider that this is irrelevant as the Horizon Issues require consideration of
bugs, errors and/or defects in the Horizon System.
One of the PEAKs shows that Post and Go transactions were being accurately
tecorded by the relevant terminal and then transferred to Horizon. Horizon creates a
POL-0019320
POL00022841
POL00022841
TA (or transaction acknowledgement) of all the transactions on the P&G machine
and, when accepted by the branch, the value of these transactions are added to the
branch accounts. The data is also passed from Horizon to various other systems,
including POLSAP. This incident was caused due to an issue at the reconciliation step
within POLSAP. Fujitsu transfers the relevant data to POLSAP via BLE files. When
POLSAP was comparing the Horizon data to the Wincor data, discrepancies were
found as data for two (of the total of six) terminals in the branch was not being sent to
POLSAP. The Post Office submissions state “There were six P&G terminals at the
branch. Data from four terminals only were being transferred to POLSAP because the
two other terminals were not associated with conducting P&G transactions”.
327. This is not an accurate summary of the PEAK. This states:
“An example the customer has provided shows amounts of 115.05, 46.88, 52.13 &
75.23 totalling 289.29 received on the file from Wincor and into POLSAP via BLE.
The same (contra) amounts are also showing as being received from the branch when
the TA has been accepted and are closed items in the account (netted off to 0.00).
However, there is another amount of 289.29 which just has the date in the assignment
field.”
328. The Fujitsu response is:
“Postings on the TfS call refer to a similar previous incident (A1040049 => Peak
PC021943293), which was resolved between POL and Wincor Nixdorf; no details of
this resolution are available to us. This incident is a week old, but only came to SSC
late last night... The trading-date in this call, 2012-08-09, is three weeks ago which
too old for us to be able to see the incoming file from Wincor Nixdorf... There is no
evidence of a fault in HNG-X, and without the incoming file from Wincor Nixdorf
there is nothing further for us to investigate.
We can only suggest that POL do the same as they did with A1040049, and refer the
matter to Wincor Nixdorf.”
329. Anne Chambers records that:
“Branch 020511 has many entries in the Subfiles_on_hold report. This report should
be monitored (by ?) to make sure problems are followed up - this should be resolved
before closing this call.
Horizon is receiving PG data for 6 separate PG tills at the branch, but only 4 of them
have associated stock units. This causes the entire subfile for the branch to be Held,
and the transaction data is not being sent to POLSAP. However the TA data for the 4
tills which are properly associated IS being sent through, and I think this is probably
the cause of the POLSAP anomalies.
The two unassociated tills are not doing any cash transactions - this is a known
problem (see PC021870294), and means the PM isn't prompted to create an
association. This may need fixing via MSC.”
POL-0019320
330.
331.
332.
333.
POL00022841
POL00022841
It was not that two of the six terminals were not doing Post & Go transactions, which
is how the Post Office submissions seek to have the PEAK interpreted. It was that
they were not going cash transactions. There are types of transaction associated with
Post & Go that are not cash transactions. They plainly are all six of them Post & Go
terminals, as the PEAK records. If they were not doing Post & Go transactions at all,
there would be no problem because there would be no subfiles on hold, or held.
As Mr Coyne said in his supplementary report “a bug fix to the Horizon system was
identified by Fujitsu, scheduled for implementation 13 September 2012 after 1800hrs
and the Branches stock was to be corrected at 1700hrs that same day.” On 17
September 2012 Anne Chambers herself reported in the PEAK that:
“Following a change made centrally to facilitate this, the stock unit associations for
the two new Post and Go terminals have been created by the branch and all the held
external data (43 different days) has now been processed and passed through to
POLSAP... We strongly recommend that POL monitor the SubfilesOnHold report
which is sent to them daily, so that any other external terminals with problems can be
investigated quickly in case a similar correction is needed”.
The “change made centrally” can only have been to the software. It is also the case
that the underling bug, error or defect was having an effect for 43 days, which is in
excess of one trading period which are four weeks long (and occasionally five weeks)
but both those periods are shorter than 43 days. It is also the case that POLSAP was
wrong for that period, which is a factor which impacts upon the dispute between the
parties about the use of ARQ data/management data. The Post Office submits that it
was “quickly resolved” but I reject that. 43 days, or over 7 weeks, is not resolving this
bug (or its effects) quickly.
I find that this is a bug with potential to cause impact to branch accounts. I do not
know the date when the Post & Go service was withdrawn from branch post offices,
or how many branches had it during the period when it was so available. The
statements made in respect of this in the Post Office submissions are said, in the
clarification table, to be based on “instructions from [the] Post Office”. The
information is vague. I accept that this service is only now available in WH Smith and
Crown branches, but other details cannot be specified because there are none
available from any source.
20. Recovery Failures.
334.
The entry in the Bug Table identifies this as relating to Horizon Issue 4. These arise
under Horizon Online and not Legacy Horizon. The Post Office does not accept these
as a bug and states that there is no evidence of a bug in Horizon; it refers to the
subject matter of the different PEAKs as “issues”. There are three different PEAKs
relevant to this. Mr Coyne accepted in his cross-examination that this (or these)
should be removed from the table that he used as the originator for his list of bugs
(which was Mr Coyne’s Table 1). Dr Worden stated in the bug table that “there was
some implication of hardware faults, with a replacement of a base unit, but the PEAK
has no evidence of software faults in Horizon.” In one of the PEAKs, he concluded
that there was no evidence of any fault in Horizon.
POL-0019320
335.
336.
337.
POL00022841
POL00022841
The situation in terms of PEAK PC0220532 is unsatisfactory. The Post Office submit
that this “relates to a cash discrepancy that was only raised by the Subpostmaster with
Post Office and Fujitsu nine months after the issue occurred.”
It is therefore clearly submitted that this took nine months for the SPM to report at all.
This is based on the date of the PEAK (15 September 2012) and the date given in the
very first entry that this happened on 6 January of the same year. However, that is
simply incorrect on the face of the document. The date of the PEAK does not mean
that the SPM took that period to report the matter to the Helpline or their manager, for
example. Reporting in January by the SPM is in fact recorded in the PEAK itself, but
not in the first entry. Further, that submission ignores these entries later in the PEAK
for 6 September 2012 that refer to a TFS call being raised on 7 January 2012:
“However I have checked our call logging system which has provided some useful
information. According to our archived TFS call login system, the Node: 5 did have
?Physical memory Dump? on 6/1/12 and a TFS call (5043049) was raised. The node
was swapped on 7/1/12.”
On 13/1/12, PM raised another TFS call (5060273). PM reported that she had a loss of
£300, she wanted to know if this was related to the Base unit fault. PM was referred to
NBSC to investigate the discrepancy.
On 13/1/12 NBSC raised a TFS call (5060420) and asked for the discrepancy to be
investigated.
On 13/1/12 @16:32pm, the frontline helpdesk contacted the PM and the following
information was logged on the call.
2Spoke to the PM who states that she had the loss before the base unit was swapped
out the loss was for 190.00 PM states that she rolled over her TP on the Wednesday
which had no loss or gains. PM states that she then did a transaction log on the Sth Jan
this shows a loss of 190.00 The base unit got replaced on Saturday 7th Jan.
PM states that she had a different error message on the system every day but did not
record the error message PM states that she had physical memory dump message
come up on the Friday 6th Jan and the base unit was then swapped on the 7th Jan?”
(emphasis added)
Later entries in the PEAK state:
“Assuming the cash declaration on 04/01/12 was correct, NBSC will need to go
through the transactions that were done in BB during this period (possibly by use of
Transaction Log), to try to determine where the discrepancy came from. The fact that
on 04/01/12 there was a negative cash discrepancy balanced out by a similar but
positive stamps discrepancy, may be related to this issue.?
It is not clear what investigation/s were carried out by NBSC and no further TFS call
was raised with regarding to this discrepancy.
If further investigation by Fujitsu is required, Post Office will have to request that the
branch transaction data is retrieved from the audit server. If there is any possibility
that this is required for litigation, it must come through the Security (ARQ) route.
POL-0019320
338.
339.
340.
341.
POL00022841
POL00022841
Otherwise queries of this nature should be sent via Mark Wardle at POL, and should
be routed to the reconciliation team in the first instance. Such requests may be
chargeable.”
This trial is not to determine individual discrepancies, although they are identified
where relevant to the Horizon Issues. On the information before the court on this item,
it is not possible to resolve this. This means that there is insufficient evidence to make
a finding of a specific bug in this case, or on this PEAK.
The second issue under this heading related to PEAK PC0241242 on 23 February
2015. This caused a discrepancy and the Post Office submits that “a Transaction
Correction was done for the shortage”. The shortage was £70.66, and the claimants
challenge the Post Office on the origin of this positive statement that this was done. In
clarification, the Post Office relied upon Fujitsu stating in the PEAK that the Post
Office needed to do this. The PEAK states on 5 March 2012 at 1546 hrs:
“This office was doing the following cash withdrawal txn for £70.66 on 23/2/15
@10:46.
The session also contained a non financial Health Lottery txn.
The settlement failed due to poor communication with the data centre. The
disconnected session receipts were printed which advised PM to pay the money out.
Spoke to PM who confirmed that the money exchanged hands.
When PM tried to log back the recovery kicked-in. However the Health Lottery
APADC recovery script failed and left counter unusable.
Yesterday POL Branch Support team authorised us to remove/update the session.
Today I have carried out and completed the task. PM is now able to use the node
again.
Reconciliation needed for the banking transaction:
The cash withdrawal txn was authorised and PM said they paid the money out.
This will leave this office £70.66 short (cash shortage) as the session not completed
fully. POL need to do appropriate reconciliation; transaction correction.”
However, a later correction in the PEAK for 5 March 2015 at 16.06 states “MSU
please note that it was a CAPO cash withdrawal for £140 and NOT £70.66.” This is
repeated at 16.12. The entry for 9 March 2015 states “I have also advised MSU to do
the necessary reconciliation and raise BIMS with POL for the Cash Withdrawal txn
for £140.”
It is clear therefore that the root cause of the issue was the failure of the AP-ADC
recovery script (Automated Payment Advanced Data Capture script, which is written
by Atos). This therefore directly shows that there was an error or defect in Horizon,
and this led to a potential impact upon branch accounts.
POL-0019320
342.
343.
344.
345.
POL00022841
POL00022841
For the purposes of the Horizon Issues trial, this is relevant to Horizon Issue 4, as
explained by Mr Coyne.
The third issue shown in another PEAK is PC0197643 dated 14 April 2010. This is a
failure of automatic recovery. This will inevitably happen in any system, as accepted
by Mr Coyne.
The Post Office submitted in respect of all three of these PEAKs the following:
“For the reasons set out above, this is not a bug at all. There are three distinct issues
under this heading. There is no evidence of a bug in Horizon and in any event,
instances of any issue were caught by automatic reporting.”
That is correct as far as it goes — the PEAKs are not evidence of bugs. The first two
are however evidence of errors and defects in Horizon. They do however go to
Horizon Issue 4, not Horizon Issue 1.
21. Transaction Correction Issues.
346.
347.
348,
349.
This arises under Legacy Horizon. This is de facto accepted as a bug by the Post
Office, but it is submitted it had no branch impact. Mr Coyne’s entry in the Bug Table
was that “Transaction Correction bugs/errors and defects do not cause discrepancies
with branch accounts but” and he then listed various consequences. In his report he
referred to “technical flaws”. He also in his cross-examination accepted that he did
not consider this to be a bug with lasting financial impact. Dr Worden, in respect of
some PEAKs under this heading, accepted they were evidence of a bug, such as
PEAK PC0129587 where he stated “In my opinion this bug would result in an
inconvenience to the Subpostmaster (inability to rollover to the next TP) but would
not result in inaccurate processing of any TC, or any impact on branch accounts”
(emphasis added). The bug is included by the Post Office in the list of paragraph 5 of
Appendix 2 under the heading “the following bugs had no branch impact.”
The submissions by the Post Office is that there is “no evidence of any financial
impact upon branch accounts”. It is therefore effectively accepted that it is, or there
were, bugs responsible for these issues, but it is also effectively agreed by the parties
that there would have been no financial impact upon branch accounts.
Transaction Corrections were introduced in September 2005, replacing the previous
method which were called Error Notices.
Some of the PEAKs, such as PC0120459 dated 4 May 2005 and others from around
that time, refer to issues with the TC button on the screen at that terminal in a branch
not working as it should. The so-called “pick list” shown on the screen would freeze.
These issues were logged by the team at Fujitsu called the Solution Validation and
Integration team. This was part of testing. None of the documents referred to by the
Post Office in its submissions to justify the submission “As such, these PEAKs all
relate to issues raised by users during testing prior to the TC functionality going live
in September 2005 (when it replaced error notices)” were referred to in the trial. I do
however accept that they show that the TC process was tested prior to release to the
whole branch network, and other PEAKs also show this. The Post Office also submits
that this was not “a bug at all”. If that is right, then it must have been an error or
POL-0019320
351.
POL00022841
POL00022841
defect, but given that the documents show that it was fixed during the testing process,
the precise terminology probably does not matter.
Other PEAKs however relate to the period afier September 2005 when the testing
process was over. PEAKs PC0129587 and Peak PC0130056 refer to the Petersfield
branch, and are dated 4 December 2005 and 14 December 2005. They both relate to
the same issue and the latter is what is called a clone of the former. The first PEAK.
relates to a TC for a £9,000 debit and part of that PEAK states:
“01/12/05 16:01 Caller states that each time they try to do a transaction correction the
counter is freezing up and going to system busy, has attempted on 3 different counters
and the G/W was on system busy for about 2 hours, pm has had to reboot 3 times
attempting to do this.
01/12/05 16:07 uk955556
Advice: Asked pm to provide details regarding transaction and fault.
01/12/05 16:08 uk955556
Information: pm has attempted to run this correction on 3 counters now, 1, 5 and 2
and each time the counter is freezing up and going to system busy, will not come back
from this.
01/12/05 16:09 uk955556
Information: Transaction was for a premium bond on the AP system.
PM was working on stock unit - E at the time.
01/12/05 16:10 uk955556
Information: Transaction was carried out on 29th October at 12.23.
01/12/05 16:10 uk955556
Advice: Advised pm that i will have to look into this and get back to her.
01/12/05 16:10 uk955556
Information: pm is fine with this but states if it is not resolved by 14th December she
will be unable to rollover.
01/12/05 16:32 uk955556
KEL Ref No.: checked for kel's but all kel's found on transaction corrections refer to
specefic events which are not present in this case.”
The lack of previous KELs on this subject at that point means that this was a new
issue which had not arisen before. It cannot therefore be put down to testing which
was completed by then.
A later entry on 6 December 2005 is:
POL-0019320
352.
353.
POL00022841
POL00022841
“Several interesting points to note:
(a) 111111111 subscription group on core server contains 35 more msgs than on the
counters: 3280 compared with 3315 msgs. The extra msgs are just Bureau:
CurrencyRates and CSvcMargins msgs.
(b) A couple of times after getting tired of waiting for the TC to process, the Clerk did
a session transfer to another counter which, according to EP/SPG/001, should not be
possible.
(c) I have reproduced the problem by importing the mstore onto the SSC counters and
setting a cut off date of 01/12/05 11:27:10 which was before the PM?s first attempt.
(d) Peak PC0120459, raised on S80 E2E XI, reported the same symptoms and the root
cause was found to be missing/incorrect ref data. This call was investigated by Mike
Coon.”
(emphasis added) (question mark present in original as “PM?s”)
“Ref data” means reference data, which is part of the software of Horizon. In my
judgment this shows clearly that this was a bug. This was therefore known about by
Fujitsu in 2005.
I shall quote some paragraphs of the Post Office’s detailed submissions on the next
PEAK:
“10. Peak PC0129587 relates to a report by a SPM on 1 December 2005 that each
time she went to select a TC for a £9,000 debit, which arose from an incorrect entry
for Premium Bonds, her screen froze and she was unable to accept it.
11. Peak PC0129774 and Peak PC0130057 refer to the Bosham branch. These
PEAKs relate to a report by a SPM on 6 December 2005 that each time she went to
select a TC, a £22,500 debit for an incorrect entry for Premium Bonds, her screen
froze and she was unable to accept it.
12. Mr Coyne’s analysis proceeds on the basis that this issue was only experienced by
these two branches. Mr Coyne fails to refer to KEL LKiang2837P [dated 5 January
2006] which demonstrates that: 50 other branches reported an issue with their screen
freezing to Fujitsu during December 2005; 48 branches reported that this prevented
them selecting an outstanding Camelot Lottery TC and rolling over into the next
trading period; and 4 branches reported that this prevented them selecting an
outstanding Premium Bond Sale TC and rolling over into the next trading period,
including the two branches referred to in the 4 PEAKs mentioned by Mr Coyne.
13. The diagnosis of the issue was that the text drafted into the TCs contained a string
of 35 characters without a space. The code to put each TC on a screen was attempting
to split the text at a space (as is normal for a word processor), but the code was unable
to process a string of this length.
14. The issue was resolved quickly. Having first been reported on 1 December 2005,
a work-around was developed whereby the affected branches were instructed to roll
over all stock units except one (the one relating to the unprocessed TC). This enabled
POL-0019320
354.
355
POL00022841
POL00022841
them to roll over into the new trading period. A software fix was then released on 22
December 2005 which enabled the branches to select and process the outstanding TC,
and roll over the last stock unit. To avoid the issue reoccurring in future, Post Office
confirmed that strings of this length without spaces would no longer be included in
the text of TCs.
15. Although included in Mr Coyne’s analysis of branch affecting bugs, the impact of
Issue 2 [ie this issue in this PEAK] was limited to the affected branches being unable
to roll over to the new trading period for a short period of time. It had no financial
impact on branch accounts.”
(emphasis added)
The code explanation above was not contained in any evidence and was taken by the
Post Office in its submissions from documents that were not even deployed in the
trial. However, these show that it was clearly a bug, which Fujitsu knew about in late
2005 and which was fixed by means of a software fix at the very end of that year, with
a KEL being produced in January 2006. TCs were introduced in September 2005.
There was no impact on branch accounts, as accepted by Mr Coyne. This therefore
goes to Horizon Issue 4 and not Horizon Issue 1.
There is another PEAK from September 2010, PEAK PC0204350 which refers to
Horizon Online and a report by an SPM on 14 September 2010 that he had suffered an
£80 cash loss which he believed was due to a “system error”. The PEAK itself states
at the beginning “Summary: can you please look into this problem. Our second line
cannot find any user error.” The SPM referred the Fujitsu representative on the call to
a TC report he had generated on 10 September 2010, which had allowed him to
request a date range of 60 days, but which did not provide him with data on TCs
which were more than 40 days old.
The PEAK indicates that on 14 September 2010, Fujitsu requested detail of the
specific transactions relating to the £80 loss from the SPM in order that it could carry
out an investigation into the issue. No information was provided by the SPM. Given
the absence of evidence, the matter was thereafter closed by Fujitsu as Category 95
“Final — Advice after investigation” and “no fault in product”. I find that this is a
misleading description ascribed to close this PEAK. It also states “TC data is kept on
the live database [BRDB] for 40 days so any txn query for a TC where it was put
through the system more than 40 days previous WILL NOT show up in the branch.
System is working as designed.” The first sentence of this is correct. Those timescales
are indeed what occurred. However, the entry “system is working as designed” is at
odds with the fact that second line support at Fujitsu had already said there was no
user error. The penultimate entry in the PEAK also states “however, beyond that, I
will have to request archived data from our Audit Team in order to confirm those TC
txns in July 2010.” There is no evidence to suggest this was done; the matter was
simply parked without any proper resolution. I find this to be evidence of a bug.
Given that Mr Coyne stated in cross-examination that there was no evidence of
impact on branch accounts (although the £80 referred to might suggest otherwise) I
take this into account in relation to Horizon Issue 4. However, it plainly has the
potential to cause impact to branch accounts and therefore should also be taken into
account under Horizon Issue 1.
POL-0019320
POL00022841
POL00022841
22. Bugs/errors/defects introduced by previously applied PEAK fixes.
358.
359.
360.
This also arises in Legacy Horizon. It is de facto accepted as a bug by the Post Office,
but it is submitted that it had no branch impact. It is included in paragraph 5 of
Appendix 2 under the heading “the following bugs had no branch impact”. It is also
submitted by the Post Office that “a number of PEAKs arose in testing. The PEAKs
that arose in the live environment do not indicate evidence of branch impact.” It is
listed in the Bug Table as concerning Horizon Issue 4. Mr Coyne stated in the Joint
Statement that “Branch accounts would be affected by this bug which would cause a
discrepancy when handling cheques where the value of the cheque would be doubled”
although he accepted that the SPM in question was processing a cheque in a different
manner to that recommended. Dr Worden used the term “a fault” in his entries,
maintained that there was no impact on branch accounts if the SPM followed correct
procedures, and he also (in respect of another branch) relied upon the fact that the
effect was an error of 2p only.
The Post Office submits that there are three distinct issues and that they were all
grouped under the same head. Some of the PEAKs are from 2000 and one is from
2004.
PEAK PC0053160 refers to a report by a Fujitsu test team member on 29 August
2000 which concerned an issue with the Training Counter which froze when
completing a transaction log report. Further testing by Fujitsu demonstrated that the
issue could also be present in the live environment, and a software fix was deployed
on 6 September 2000. Mr Coyne concluded that the fix implemented what are called
regression bugs. There are references to this in the PEAK. A regression bug is
something that prevents a function from operating as it should. Entries in the PEAK
such as:
“Training Counter freezes using transaction log
Training Counter freezes when miskeying using transaction log. If you do the
following:
perform sales transaction
select Reports/Transaction Log
select Value from .10p, Receipt
select value to £100, Receipt
Continue (F16)
Complete (F16 from Print screen, instead of F4)
Reprint (F4 from Reports menu)
you go back to the Transactions Log menu as expected but the counter freezes and
you can't get out. Continue and all the other buttons do not work (except to flash when
selected).
This manifests itself in training, if a delegate mis-hears or miskeys a keying sequence
in doing a transaction log report. Could be a live issue too.”
And:
“This is NOT a Traing Mode issue, but an EPOSS issue. The problem is that if you
Prev out of the Report/Reprints menu you are returned to a previous
Transaction Log selection menu from which there is no escape.
Dean has not specified which build he was using, but I have reproduced this
on both CI4L1 and CIA4R streams.”
And
POL-0019320
361.
362.
363.
364
POL00022841
POL00022841
“F} Response:
The problem described at the start is fixed. However, application of the work
packages to the training counter appears to have caused some regression, namely:
If you perform a number of different types of transactions (e.g. passports, APS, BT
bills) the daily reports are correctly populated, however no reports are listed when you
press the Summaries icon (mandatory or otherwise), and the system does not require
you to print and cut off the mandatory reports before rolling over the stock unit into
the next CAP.
[END OF REFERENCE 21808091]
The response was delivered on the system”
(emphasis added)
These suggest that Mr Coyne is right — Fujitsu itself diagnosed it both as a regression
issue and also diagnosed a product error. The Post Office’s written submissions stated
that “further investigations by Fujitsu have concluded that rather than a regression,
this was an error in the way that the test rig was setup; an approved combination of
work packages was not being used” although this was challenged by the claimants as
not being contained in any of the evidence.
The answer to the challenge that was given was that this was taken from a document
not deployed in the trial, and from common sense. The document was PEAK
PC0053160. The entry expanding on common sense states “the developer is stating
that the tester failed to re-test against an approved combination of work packages;
hence the assertion by the tester that there was a code regression is incorrect”. It is
also stated “work packages sometimes have interdependencies in order to provide a
complete fix to any given issue.” I consider that the PEAK document was deployed in
the trial in the sense that it is referenced by Dr Worden in his entry in the Bug Table,
but in any event the entry relied upon is 22 September 2000 when Les Ong stated:
“It's not stated which build this is but, by adding two M1 work packages in isolation,
I'm not surprised there's further problems. These two WPs contain EPOSSCore,
EPOSSReportbroker, EPOSSStockunit and BESReports. Any of these may have
dependencies on other WPs or Ref Data.
You need to stick with a CI4L1, CI4R or current M1 build. At least bugs in these are
known and can be quoted.
[END OF REFERENCE 21817364]
Responded to call type P as Category 74 -Fixed at Future release”
This is clearly a reference to a bug, and is also something that arose on a training rig,
but it is in my judgment a regression issue. The tester thought that it was, and that
would have been one of the things that the tester was looking out for.
In any event, the next PEAK clearly was a regression issue. I will quote the Post
Office’s submissions:
“8. Peak PC0098230 refers to an issue reported by a SPM on 13 January 2004 relating
to a discrepancy with his cash account. Fujitsu called the SPM to obtain further
POL-0019320
365.
366.
POL00022841
POL00022841
details and ascertained that when the SPM was declaring his cheques, the value of
cheques declared as stock doubled.
9. The issue was diagnosed as a code regression relating to the fix implemented in
Peak PC0097081. A work-around was proposed two days later to the SPM, on 15
January 2004, that the SPM should follow a different procedure when declaring his
cheques. A software fix was thereafter released for the code regression.”
The PEAKs relating to the third issue were regression bugs but were discovered
during testing in early 2000.
Therefore, in my judgment, Mr Coyne is correct in that the PEAKs that were the
subject of this heading in the Bug Table, were regression bugs. Given the lack of
impact upon branch accounts, these relate to Horizon Issue 4. The one in 2004 is
particularly notable, but given it was resolved so very quickly (a matter of days) there
was in fact no impact on branch accounts, but there was potential impact only. I
consider this bug to be relevant to Horizon Issue 4.
23. Bureau de change.
367.
368,
These occurred in both Legacy Horizon and Horizon Online and arose in 2005, 2006
and 2010, and should be differentiated from bug 14, which arose in 2017, which was
during Horizon Online and is referred to as Bureau Discrepancies. The Post Office
does not accept this as a bug. The Post Office also states that Mr Coyne has identified
three issues all with the same heading, and submits that they all relate to user error. Dr
Worden’s entry in the Bug Table in relation to one of them states “Analysing the
second KEL (2010) I noted: ‘Impact small until bug fixed - rounding errors 10° in
exchange rates.’ The Post Office’s submissions that this is not a bug is a somewhat
bold submission, given the KEL to which Dr Worden refers expressly states, in terms,
that the problem is a different exchange rate appearing on the HNG-X rateboard and
the Horizon rateboard, two different parts of the same system. It also ignores Dr
Worden’s statement “until bug fixed” (emphasis added). HNG-X is Horizon Online,
but this cannot applies to the occurrences in 2005 and 2006.
KEL AgnihotriV917N dated 23 June 2010 states within the KEL itself:
“Problem — there is a bug in the code” (emphasis added).
The full entry reads as follows:
“Problem
There is a bug in the code.
Steps to reproduce:-
1. In spotrates.xml make the value for any currency (say Egypt Pound EGP) as 10 and
in margins.xml as 1.004 under BUY- EGP.
2. Start the Counter
3. Configure RatesBoard with this currency.
POL-0019320
369.
POL00022841
POL00022841
4. Click on update.
5. Under Buy Notes column the value displayed for EGP will be 10.1 instead of
10.100.
Solution - ATOS
This bug is harmless as the business rule is working as desired except for displaying
effective rate.
It means in the background the value used for effective rate will be as calculated from
the formula but when it comes to display, value will be stripped of trailing zeroes.
See also PC0200042 and KEL Agnihotriv245L.
Solution in progress.”
(emphasis added)
There plainly is a bug — it is referred to twice in terms in the same KEL. The author of
the KEL states that it “is harmless”. The other KEL referred to, KEL Agnihotriv245L,
is by the same person. It refers to the fact that customers will (or may) be given the
wrong rate to the one displayed in the branch. It states:
“There is a bug in the code.
Steps to reproduce:-
1. In spotrates.xml make the value for any currency (say Egypt Pound EGP) as 8.3325
and in margins.xml as 11.2400 under BUY- EGP.
2. Start the Counter
3. Configure RatesBoard with this currency.
4. Click on update.
5. Under Buy Notes column the value displays for EGP will be 9.269 instead of
9.2691.
Although the system does use the correct values to calculate the rates and the issue
here is with the display only. However, because the display is on the Customer facing
tates board there is potential for annoying Customers. They may get a slightly
different rate to that advertised on the board. And may challenge Post Masters on this
issue.
As the issue relates to the 5th or 6th significant figure the impact will be fairly low -
and in favour of the Customer 50% of the time e.g. if I buy £1000 of XYZ currency at
a rate of 99.12 then I expect to get 99120.00 in XYZ (according to the board) but I
will actually get 99115.90 in XYZ (by the Counter) - a difference of 4.10 XYZ or the
equivalent of £0.04. Its effect may be magnified by high value transactions but
generally, different rate bands apply in these cases - so the rate board display is
inconsequential.”
POL-0019320
370.
371.
372.
POL00022841
POL00022841
Solution - ATOS
The solution is a code fix to change the code to treat values in the affected ranges
correctly.
See also PC0200090 and KEL AgnihotriV917N
Solution in progress.”
This plainly shows a complacent, if not lackadaisical, attitude to financial precision.
The error may be in the customer’s favour 4 the time, but that means it will be in the
Post Office’s favour the other 4 the time, and the customers will not be the same
person on both occasions, and are highly likely to be entirely different people. The
KEL also recognises that “its effect may be magnified by high value transactions”.
The solution in one of the other KELs, raised by Anne Chambers namely
AChambers2252R is:
“Solution - ATOS
If a PM reports a loss connected with a currency transaction that was reversed, it is
possible that the reversal had not been carried out successfully.<br><br>Ask the PM
to check the Reversal Receipt. If this shows<br><br>Cash FROM CUSTOMER
750.00<br>Cash TO CUSTOMER 750.00<br><br>they have reversed just the cash
settlement part of the transaction, which has no overall effect. The currency and
margin part of the transaction has not been reversed.<br><br>Do a transaction log
search using the Session Id from the original receipt, or by date/time.<br><br>This
should show 3 elements - for example<br> <br>2-29826-2 SC Euro 1- 720.00-<br>2-
29826-2 SC Curr Sell Margin 1- 30.00-<br>2-29826-3 SC Cash 1
750.00<br><br>The element which should be reversed is 2-29826-2 (i.e. Euros and.
margin). As long as the PM has not yet rolled over the stock unit, they should be able
to reverse this transaction now.<br> <br>If the stock unit has been rolled over, NBSC
will have to advise on what can be done to resolve the system loss relating to this
transaction.”
They are therefore two different issues. The Post Office, again boldly, submitted at
paragraph 2 on page 511 (which is part of its Appendix 2) in relation to this bug that
“There is no evidence of a bug in Horizon. Each issue is an example of user error in
Horizon.” I find those submissions to be verging on the extraordinary. The substantive
judgment to which this is an appendix identifies what the Known Error Log is. The
entries above make it clear that there is a bug — the very word chosen by the Fujitsu
employee who wrote the two KELs is “bug”. To see this characterised in submission
as there not being a bug, and being evidence of human error, is not only puzzling but
flies in the face of the terms in the Fujitsu documents. I find that it is evidence of a
bug. One of the PEAKs only of those listed in the Bug Table, namely PC0137437 is
user error, a point identified by both Mr Coyne and Dr Worden. However, simply
because one PEAK is this, does not mean that the summary submissions can be
accurately stated as “There is no evidence of a bug in Horizon. Each issue is an
example of user error in Horizon.” That is simply wrong in fact.
POL-0019320
373.
374.
375
376.
POL00022841
POL00022841
Further, Mr Coyne identified in his evidence that there were 8 PEAKs associated with
these KELs that fall in the date range of 2010 to 2018.
There are some unsatisfactory aspects to the way the Post Office tries to explain its
position on this bug. At paragraph 14 of its submissions, it submitted that the Post
Office subsequently “reviewed ARQ data for the branch” in question on one of the
PEAKs, and that “there is nothing to suggest that the branch’s remming in/out of
foreign currency caused a loss in branch.” When challenged by the claimants that this
was not in the evidence, the Post Office clarified by stating this came “on instructions
from Post Office, which did not have an opportunity to adduce evidence on the point”.
That is not correct. This submission was made in relation to PEAK PC0151787 which
is the second entry in the final column in the Bug Table itself, headed “supporting
evidence”. It is also referred to, correctly, in paragraph 4 of the Post Office’s
submissions in the following terms “In JS2 Mr Coyne mentions three PEAKs under
the heading of “Bureau de Change”. JS2 means the 2‘ Experts’ Joint Statement,
which was dated 25 February 2019, well before the start of the trial. There was such
an opportunity, as supplementary questions in chief could have been put (or
permission to do so asked) and it is wrong to suggest that the Post Office was denied
the opportunity to meet this evidence. Dr Worden chose to adopt a different way of
giving his expert evidence, namely very high level.
In any event, there is nothing in my judgment to justify a finding of user error which
is relied upon by the Post Office in relation to this PEAK. The first entry in the PEAK
states, not user error, but the following: “PM states that that he has remmed out some
foreign currencies, reversed them and re-remmed them but when he has come to do
the stock balancing his main stock unit is £907.97. Richard at NBSC 2nd line has
requested that I log a call and send to SMC.” Even Fujitsu categorise the call closure
as “Defect cause updated to 42 -- Gen - Outside Pathway Control.” That is not
consistent with user error.
In any event, I find that there is such a bug and I take this into account in my overall
conclusions on Horizon Issue 1 as well as Horizon Issue 4.
24. Wrong branch customer change displayed.
377.
This is a Legacy Horizon Issue and occurred in 2005. Mr Coyne concluded this was a
bug and part of his entry in the 2" Joint Statement was “the KEL explains that ‘the
cash amount entered is multiplied by the Qty and hence the new stack total is
wrong’”, and that this Horizon bug was due to incorrect reference data. The role that
reference data plays in the operation of Horizon has been explained. Here, this led to
an incorrect amount of change being displayed on the branch screen leading to the
operator (which means SPM or their assistant) providing the branch customer with the
wrong amount of money, thereby leaving a discrepancy in Branch Accounts. He also
stated that “It is possible that the amount of change shown on screen is more than the
actual money tendered by the customer.” Dr Worden’s entry was “When analysing
this KEL I noted ‘Sounds like a genuine problem which may have led to giving the
customer the wrong amount - i.e. not recoverable.’ It is now accepted by the Post
Office in paragraph 7 of Appendix 2 as one of a number of “bugs with lasting impact
(although they were resolved)”. It is also said that it is a reference data bug, and that
therefore once discovered could be quickly fixed by changing the relevant reference
data.
POL-0019320
378.
379.
380.
POL00022841
POL00022841
The summary of the Post Office’s submissions are as follows:
“2. Bug 24: Wrong Branch Customer Change is a bug with the potential for lasting
financial impact. This is a reference data bug. The experts have agreed that while
reference data bugs may be a significant proportion of the bugs with financial impact,
once discovered, they could be quickly fixed (by a change to the reference data) once
the bug is correctly identified. This was the case with Bug 24. The issue would have
visible to the SPM as the incorrect quantity would have displayed on the screen.
Fujitsu identified the root cause and developed a fix within two weeks of the issue
being reported by the SPM.
3. This issue relates to quantities of stamps and postage labels (Smartpost
Transactions) not correctly resetting to 1.
4. When a quantity of greater than 1 was entered for a Smartpost Transaction, the
quantity was not reset to 1 when the clerk moved on to the settlement screen. This
could result in subsequent items in the session being multiplied by whatever quantity
remained and could affect further items being sold or the amount being tendered
towards settlement.
5. Peak PC0128264 was opened on 4 November 2005 as a result of a SPM reporting
the issue on 4 November 2005. The matter was passed to Fujitsu’s development team
for a software fix on around 10 November 2005 and a fix was implemented on 18
November 2005. The Peak shows that Fujitsu suspected that the problem was
introduced by changes to the Smartpost Transactions that had been implemented from
24 October 2005.
6. On 6 December 2005, a further instance was reported (Peak PC0129791). The root
cause was identified and it was found that this issue related to Peak PC0128264 which
documented the fix that had been put in place. On 7 December 2005, Fujitsu found
that the fix had not been applied to a group of branches and the reference change data
fix was then implemented overnight to the remaining branches.”
The KEL on this was again one authored by Anne Chambers. It is undoubtedly a bug.
Not only was it a bug, but once it was discovered and a software fix introduced, it
persisted because “the fix had not been applied to a group of branches”. No detail is
given as to how large that “group of branches” was — the only description in the
submissions is that it was “was an active group of branches, group 111111112”. Nor is
there any explanation available in the evidence as to why the fix, which was known to
be required to fix this bug, was not applied to every single branch, and why or how
this “active group” of branches was omitted.
I find that it is a bug with potential to impact upon branch accounts and I take its
existence into account in my answers to both Horizon Issue I and Horizon Issue 4.
25. Lyca top up.
381.
This is a Horizon Online bug. Lyca is a type of mobile phone “top up” card which
allows people to pre-pay for mobile phone services. This is accepted as a bug by the
Post Office in paragraph 6 of Appendix 2, but is said to have had transient impact. It
is also accepted as being a reference data bug, and paragraph 2 of the detailed part of
POL-0019320
382.
384.
385,
POL00022841
POL00022841
the submissions on this in Appendix 2 states “Lyca Top Up is a bug with the potential
for lasting financial impact. This is also [a] reference data bug. As set out above, the
experts have agreed that while reference data bugs may be a significant proportion of the
bugs with financial impact, once discovered, they could be quickly fixed (by a change to
the reference data) once the bug is correctly identified.” It is also submitted by the Post
Office that it was identified through Fujitsu’s automatic reporting.
The nature of the bug, which occurred in August 2010, is that a customer will pay the
SPM a certain sum (for these purposes, an example of £20). The transaction is entered
on the Horizon counter by the SPM and, once the transaction is processed by Horizon
and authorised by E-Pay (the financial institution), a receipt is printed for the
transaction which the SPM should provide to the customer. It is this receipt that
contains the customer’s voucher to apply the top up to their mobile phone. Applying
that voucher will give the customer £20 of credit to use on their phone.
The problem was picked up by SPMs who reported it to the NSBC, and also by
Fujitsu through its NB102 reporting. Depending upon what happened when the
customer and the SPM did this transaction, would affect whether branch accounts
were adversely affected. However, the Post Office accepts that “if the SPM recovered
the transaction and incorrectly confirmed on Horizon that the top up had been
successful, despite no top up receipt having been printed, the transaction would be
recorded in the branch’s accounts, meaning the branch would likely have experienced
a shortfall to the value of the top-up as no money would have been taken from the
customer. If the SPM logged back into the counter and correctly confirmed that the
transaction had not been successful, a zero value transaction would be recorded in the
branch accounts and a reversal generated for the top up. This should have resulted in
the affected branch accounts being correct, but due to the reference data issues the
reversal being sent to E-Pay caused E-Pay to treat the top-up incorrectly as a
successful transaction.”
The PEAK shows that it was described as a “system error during transaction” and that
it was resolved by means of a change to the RDT generator mechanism. One entry
also states that “This problem could also show up as incorrect Welsh on receipts as
special Welsh characters may also be incorrectly translated.” The corrective reference
data was released to live to take effect overnight on 20 August 2010.
I find that it was a bug and I take its existence into account in my answers to both
Horizon Issue 1 and Horizon Issue 4.
26. TPSC 250 Report.
386.
387.
This is a Legacy Horizon bug. It is accepted by the Post Office in paragraph 7 of
Appendix 2 as one of a number of “bugs with lasting impact (although they were
resolved)”. The origin of the experts discovering this bug is a KEL raised by Anne
Chambers in February 2005, and last updated in April 2008.
There are approximately 24 different PEAKs dealing with this bug. Their date range
is between 2005 and 2009, with the majority of the incidents occurring in 2005. It
relates to the printing of labels for postage. The KEL was originated by Anne
Chambers.
POL-0019320
POL00022841
POL00022841
388. The Post Office has identified, within its submissions which deal with all of the
PEAKs, a number of different sub-issues. The summary paragraph states:
“1. The experts have drawn together five distinct issues under the heading TPSC 250
2. This was a backend reporting problem and so the chances of branch impact are
small. Of the five issues: issue I resulted in incorrectly flagged exceptions but no
reconciliation was required; issue 2 resulted in no financial or operational impact on a
branch accounts and was limited solely to the process of copying/ harvesting
transactions to Post Office's back-end systems; issue 3 did not result in a mismatch
between the files sent to Post Office and the branch data; issue 4 was flagged by
automatic reporting and issue 5 could result in a receipts and payments mismatch thus
needing reconciliation.”
389. I do find it notable that although Dr Worden says the amounts involved are what he
calls “small”, he accepts it is as a “back end reporting problem” — which is nothing to
do with the SPM, by definition — and also the KEL states that “the accounting tree has
not handled this properly when calculating the daily recon figures and it has resulted
in a mismatch...” The mismatch in that case was 66p. The accounting tree is part of
the Horizon system. The KEL also appears to be incomplete because only the first
page is present. The section that follows the heading “Evidence” is entirely missing.
There is also nothing in terms of the text available that demonstrates what occurred in
2008, even though it is plain from the page that is available that the KEL was updated
then.
390. Regardless of whether the impact was small or not — and I accept that for the most
part, the amounts referred to in the PEAKs are in pence or a few pounds — this is a
reflection of the fact that the cost of postage labels is relatively modest. One PEAK,
namely PC0123056, gives a good outline in its entry for 12 July 2005 by Anne
Chambers:
“Yes this is another instance of KEL AChambers253L - mails transaction total value
£1.86, prepaid £4.26, so postage label for -£2.40 generated. This has upset the counter
reconciliation figures.”
391. The reason the counter reconciliation figures are “upset” is that the label was
generated for a minus figure, namely “-£2.40”. Mr Coyne identifies that the values are
less than £2, and he also opines that SPMs may simply write off such amounts.
392. The “five distinct issues” identified by the Post Office in its submissions are grouped
by the experts under the same single heading, because (I assume) they all deal with a
similar scenario. I will count them as a single bug, even though manifestations of the
early 2005 example were dealt with as follows, taken from the Post Office
submissions: “A permanent fix involving a code change was released to all branches
in 2005 as part of the S80 software upgrade.”
393. The occurrences of bug(s) or issues within Legacy Horizon after the S80 software
upgrade therefore were either new, or different. On the dates in the documents,
problems with TPSC reporting plainly did persist after the software upgrade was
introduced in 2005. The majority of these were picked up by Fujitsu reporting, but
POL-0019320
394.
395.
POL00022841
POL00022841
that does not mean that they were not there. Fujitsu did not even mark these issues
high priority, although the explanation given for that by the Post Office is that this
was done because there was no impact upon branch accounts.
The S80 software upgrade was accompanied by something called “S80 Release Note
— Deferred PEAKs List — Counter.” This document is dated 13 October 2005 and is
32 pages long. It “details PEAKs that are outstanding at S80” and the approved form
of that document is in the trial bundle. The Technical Design Authority for it was
Gareth Jenkins. It includes analysis of the PEAKs that affect the counter only, and the
document is an addendum to another document. It identifies 45 different PEAKs that
affect the counter. This document shows that there were other issues, not simply the
one relating to TPSC reporting, where a decision was taken not to deal with certain
errors and/or coding bugs at that time. One example only, cash volume adjustment
states “This is a code error but the problem has been in the system since before S80
and doesn’t appear to be causing any significant confusion. A KEL should be raised
and a fix considered in a Future Release.”. Of the 45 PEAKs in this document, most
are to be dealt with at a “future release”, one is accepted (which is cosmetic), one
closed, and another is to be dealt with by a documentation update. It would therefore
not be correct to assume that all known PEAKs were fixed by release S80.
I find that it was a bug with the potential to impact upon branch accounts and I take its
existence into account in my answers to both Horizon Issue I and Horizon Issue 4.
27. IPS.
396.
397.
398.
This again is a Legacy Horizon bug. Its years of effect are 2006 to 2010 with the
majority diagnosed by Fujitsu in the PEAKs as “Development — Code”. It is accepted
by the Post Office in paragraph 7 of Appendix 2 as one of a number of “bugs with
lasting impact (although they were resolved)”. One of the Fujitsu documents
referenced in the 2" Joint Statement states that the Transaction Repair Tool or TRT is
being used “to repair I harvester exceptions for” a particular branch and “There is no
correction to be performed and hence no call for confirmtiprepair - this is just an
oddity performed by that very flaky mails code.” (emphasis added) The reference to
“flaky mails code” makes it clear that it is the code that is the cause, which I consider
means it is a bug. Mr Parker gave some evidence on this, and I have reviewed this too
even though the Post Office submissions do not specifically identify his statement as a
“key document” in the detailed submissions on Bug 27.
There were 40 associated PEAKs in the Bug Table under this entry, and Mr Coyne
observed that both the credit and debit sides of a transaction were doubled, so the net
impact of the bug was zero, although he drew attention to the entry in two PEAKs that
suggested that SSC requested confirmation of any gain or loss at the counter. Dr
Worden believed it was a back-end reporting problem, although the chances of impact
upon branch accounts were small.
The issue is described as follows by the Post Office in its submissions:
“3. This is a reconciliation reporting issue that affected SmartPost transactions. The
SmartPost application was supplied by Escher and was designed to help users to
calculate the postage required for any item and print labels to attach to the relevant
items.
POL-0019320
399.
POL00022841
POL00022841
4, Peak PC0141145! related to a problem with SmartPost which wrote slightly corrupt
transactions.” (emphasis added)
That this bug was known about at Fujitsu at least, is very clear. There is a KEL
dealing with it which dates from 2006, and runs to 2010 when Legacy Horizon
stopped being used. It states in part:
“Symptoms
Branch shows that the TPS Total and Counter total values for the Number and Abs
Quantity columns are the same, but there is a difference in the Absolute Value.
<br><br>Absolute Value for Counter Total is greater than the corresponding TPS
Total by £14.8
Problem
Mails doubling the <Credit:> attribute. Search for <Mode:SC>><Credit:(value no
decimal point)> NOTE: This will be the difference between the Absolute Value of
Counter and Host figures on the TPSC250 report.<br>The column title in TPSC250 is
"Absolute Value". This means that Debits will contribute in just the same fashion as
credits so if the first search yields nothing useful then look for <Mode:SC>>
<Debit:(value no decimal point)><br><br>28-Aug-2008 PC0164058 Here there were
<b>two</b> messages where Credit was double SaleValue. The sum of these two
messages came to the difference in Absolute Value in TPSC250. This branch also
appeared in TPSC257.<br><br>Another occurence in
<b>PC097941.</b><br><br>The Smartpost messages are unusual in that the Credit /
Debit attribute is near the beginning of the message instead of at the end. The
SaleValue is correct but the DisplayValue and Credit are twice as much as
expected.<br><br>This problem affects Bulk Mails (T&T) transactions. It seems
likely that there is some way of backtracking through screens which causes this, but
haven't managed to reproduce it so far.<br><br> Often this has no effect on balancing
- the messages are as expected, except for the Credit/Debit attributes. These attributes
are only used for the calculation of the EPOSSDailyRecon absolute values. The office
should roll over successfully with no Receipts and Payments
mismatch.<br><br>However_ occasionally there _is_a very similar_problem with
incorrect Credit / debit attributes in bulk mails smartpost messages where the session
doesn't net_to zero. This will additionally cause an entry on TPSC257
IncompleteSummaries, and_also_give a receipts and payments mismatch.
<br><br>Another variation is where mails code omits the message for a transaction
and then does write the balancing message. The effect is the branch appears in
TPSC250 and TPSC257. <br> <br>See KEL MaxwellG460L for how to fix
TPS_POL_FS_Summaries_Incomp.<br><br> 16-Jan-2007 PC0142604. <br>13-Dec-
2007 PC0152156<br>24-Jan-2008 PC0153333<br>05-Aug-2008 PC0162929<br>16-
Dec-2008 PC0171637”
Solution - ATOS
F/364}.
POL-0019320
400.
401.
POL00022841
POL00022841
PC0141145 is with development, which has similar symptoms. Add further instances
to that call<br> <br>If the session nets to zero (add up all the SaleValues for the
same SessionId) no reconciliation is needed. If it doesn't, a correction must be made
to___send__the _data__to POLFS (see <a __ href =kel_view_kel.jsp?
KELRef=MaxwellG460L>MaxwellG460L</a>) and_the PM may need to be told
about a possible receipts and payments mismatch, or at least watch out in case one is
raised.”
(emphasis added)
This shows the following. Fujitsu picked up this bug through automatic reporting. It
also shows that KEL itself recognises that sometimes a receipts and payments
mismatch may occur in branch accounts, and that if so “a correction must be made”.
That correction would be made outside of the Horizon System, by means of a TC.
That plainly, in my judgment, makes it clear that this bug has the potential to impact
upon branch accounts.
I find that it was a bug with the potential to impact upon branch accounts and I take its
existence into account in my answers to both Horizon Issue 1 and Horizon Issue 4.
28. Drop and Go.
402.
403.
404.
This is something that occurred in the summer of 2017 and is a Horizon Online issue.
It is accepted by the Post Office in paragraph 7 of Appendix 2 as one of a number of
“bugs with lasting impact (although they were resolved)”. This actually occurred in
June 2017 (although the KEL is dated July 2017) and concerns a duplicate “Drop and
Go” transaction for £100. This was a top up which had to be performed twice; the
branch was debited with £200, but the customer credited with only £100 (which had
been the amount “topped up” by the customer). Mr Coyne considered it a bug with
impact upon branch accounts; Dr Worden stated in his entry “My analysis of this KEL
was ‘Possible financial impact. Seems very visible on the counter. Script = reference
data - therefore fixed easily”. Paragraph 2 of the detailed part of Appendix 2 of the
Post Office’s submissions on this bug states “Peak PC0260269 relates to an issue
involving a Drop and Go transaction (a £100 mobile phone top up) that timed out on
Horizon” (emphasis added). This does not appear to be correct, and I do not believe
that the transaction relates to a mobile phone. It is a postal-type service.
However, regardless of the type of service it deals with (mobile phone card top up,
which is what the Post Office says, or Drop and Go postal service, which is what I
consider it deals with) is not of the highest relevance. It is a Post Office service for
which customers in the branch pay money. Mr Parker gave some evidence on this, and
T have reviewed this too even though the Post Office’s submissions do not specifically
identify his statement as a “key document” in the detailed submissions on Bug 28.
The Fujitsu evidence on this, as with all the other bugs which are now acknowledged
to exist, is that there was no impact on branch accounts. Indeed, Mr Parker accepts in
his table at item 28 that “this would have caused a loss in the branch accounts”
although he goes on to downplay this by saying the SPM would have noticed and “it
would have been resolved by a” TC. TCs are outside the Horizon System. In this case,
there plainly was branch impact. The customer was credited with £100 (which means
that is the amount the customer paid to the branch) but the branch was debited with
£200. The fact that the branch was later corrected does not, in my judgment, matter
POL-0019320
405.
406.
407.
POL00022841
POL00022841
for the purpose of the Horizon Issues. The issue really is ~ was there a bug in Horizon
that allowed this to happen, and/or was there a bug in Horizon that caused this to
happen? Both ways of putting it adequately summarise the real issue.
The answer to that is plainly, yes. Dr Worden states that the script that led to this
occurring could be easily fixed. However, the fact that a fix is needed at all
demonstrates that it is a bug and that it had the potential to cause lasting impact to
branch accounts. It can plainly be seen from the following analysis of the KEL.
The KEL on this, carde235Q is dated 5 July 2017 but examining the log entries
reproduced in the KEL itself it can be seen that the transaction(s) in question occurred
on 20 June 2016. The KEL states (and clerk means the SPM or their assistant):
“Symptoms
The clerk initiated a Drop and Go transaction for £100 which failed due to timeouts,
but then a success message was displayed. The clerk settled the transaction and the
customer handed over £100. The customer checked the balance and stated that the top
up had not gone through, so the clerk then performed another Drop&Go transaction
which was successful. The customer has paid in £100 but the branch account has been
debited by £200. Accenture verified that only the second Drop&Go top up was
successful.”
This shows that Accenture are involved in some way in investigating this, and have
verified that only one of the top ups was successful. The word “verify” can only mean
that they have established this to their (ie Accenture’s) satisfaction. The KEL also
reproduces the log extracts from both the counter log and the message log, which I
shall reproduce in full:
“2017-06-20 13:49:07,912 UTC Button: WS-F-Home-1-55 / Drop & Go...
2017-06-20 13:49:09,113 UTC Button:WS-F-PostalServicesDG-1-22/Balance and
Top Up
2017-06-20 13:49:11,046 UTC Swipe: [18954123=***]
2017-06-20 13:49:11,747 UTC Sending Request, service url= [
https://vbal001:9000/GenericOnlineService/HBS/CDPG/CDPAPADCGateway/delta/
0}...
2017-06-20 13:49:12,198 UTC Response Received, Status OK, service url= [
https://vbal001:9000/GenericOnlineService/HBS/CDPG/CDPAPADCGateway/delta/
0)...
2017-06-20 13:49:12,689 UTC MSG10800: Confirm Account Details
2017-06-20 13: 1,071 UTC Button: 0/ Yes
2017-06-20 13:49:21,532 UTC MSG10800: Top Up Required?
2017-06-20 13:49:27,020 UTC Button: 0/ Yes
2017-06-20 13:49:31,627 UTC Button: enter / Enter
2017-06-20 13:49:34,101 UTC Button: 0 / Cash
2017-06-20 13:49:34,582 UTC MSG10802: Take Payment from Customer
2017-06-20 13:49:37,406 UTC Button: 0 / OK
2017-06-20 13:49:38,007 UTC Sending Request, service url= [
POL-0019320
408.
409.
POL00022841
POL00022841
https://vbal00 1 :9000/GenericOnlineService/HBS/CDPG/CDPAPADCGateway/delta/
0]...
2017-06-20 13:50:23,065 UTC IOException occurred while accessing service at
URL:
https://vbal001:9000/GenericOnlineService/HBS/CDPG/CDPAPADCGateway/delta/
2017-06-20 13:50:28,473 UTC Sending Request, service url= [
https://vbal001:9000/recoveryService/delta/0 ] ...
2017-06-20 13:51:13,551 UTC IOException occurred while accessing service at
URL: https://vbal001:9000/recoveryService/delta/0 ...
2017-06-20 13:51:14,332 UTC MSG10802: Top Up Timed Out
2017-06-20 13:51:18,769 UTC Button: 0 / OK
2017-06-20 13:51:19,760 UTC Sending Request, service url= [
https://vbal001:9000/GenericOnlineService/HBS/CDPG/CDPA PA DCGateway/delta/
0)...
2017-06-20 13:52:04,848 UTC IOException occurred while accessing service at
URL:
https://vbal00 1 :9000/GenericOnlineService/HBS/CDPG/CDPAPADCGateway/delta/
2017-06-20 13 0,146 UTC MSG10802: Top Up Unsuccessful
2017-06-20 1 4,352 UTC Button: 0 / OK
2017-06-20 13:52:14,843 UTC SelectFunctionOptionsBLO.IOP39BLO.ADCScript-
CDBalanceTopUp.AdvancedDataCaptureUIA / HSID: 0-@@-
2017-06-20 13:52:14,953 UTC MSG10802: Top Up Successful
2017-06-20 13:52:17,477 UTC Button: 0 /OK
2017-06-20 1 0,542 UTC Button: fastCash / Fast Cash
2017-06-20 13:52:21,023 UTC Sending Request, service url= [
https://vbal001:9000/BasketSettlementService/Settle/delta/2 ] ...
2017-06-20 13:52:22,124 UTC Response Received, Status OK, service url= [
https://vbal001:9000/BasketSettlementService/Settle/delta/2 ] ...”
(Emphasis added)
The reason for reproducing all these entries is they clearly show that the Top Up
attempt timed out, that the Top Up was unsuccessful, and that the second attempt was
successful. However, under “Problem”, Fujitsu recorded the following in the KEL:
“Problem
This may be an issue with script ADCScript-CDBalanceTopUp, or a user error.
Solution - ATOS
The Drop&Go scripts are supplied and maintained by ATOS. Therefore please route
calls to ATOS.
Solution - SMC
The Drop&Go scripts are supplied and maintained by ATOS. Therefore please route
calls to ATOS.”
The suggestion that there may be a user error is without any foundation, if the entries
in the logs are carefully looked at. These entries clearly show both the time out and
the first unsuccessful transaction. It can only be a script issue, something confirmed in
POL-0019320
410.
411.
POL00022841
POL00022841
my judgment by the Solution both for ATOS and SMC being “The Drop&Go scripts
are supplied and maintained by ATOS. Therefore please route calls to ATOS.”
Dr Worden looked at this and in his Appendix 5 he also identified “possible financial
impact” which means impact to branch accounts. There obviously would have been in
this case. The SPM has taken £100 from the customer but the Horizon system has
treated it as though his branch should be debited for £200 (2 x £100) even though one
of the top ups was clearly not successful because it had timed out. The other point of
note is that the KEL, the trial bundle reference of which is {F/1660}, is headed
“HNG-X KEL cardce235Q”, although on the dates within it (namely June and July
2017) other evidence in the trial was that the system by then was not HNG-X. This is
because it became HNG-A on February 2017 when the platform became Windows 10
rather than the older platform (and by 2017 unsupported) Windows NT4. It is not
therefore clear on the face of the KEL whether this bug — which it clearly is, and
which I find is a bug — is within either the HNG-X or HNG-A version of Horizon
Online.
I find that it is a bug and with the potential to impact upon branch accounts and I take
its existence into account in my answers to both Horizon Issue I and Horizon Issue 4.
29. Network Banking Bug.
412.
413.
414.
415.
416.
This is a Legacy Horizon matter.
This related to a KEL which was raised in 2004 and updated in 2005, with 12
associated PEAKs in the range 2004 and continuing on to 2010 when Legacy Horizon
was discontinued. The Post Office does not accept this as a bug. Mr Coyne stated that
“Horizon appears to mis-handle communications, leading to errors within network
banking and in turn causing the potential for branch account discrepancies.” One of
these was pension transactions being declined, yet the customer’s bank account being
debited; when the customer complained, the SPM themselves refunded the £50 in
question. Dr Worden considered it was “mainly about a communication problem from
BT, outside Horizon” but also wanted to investigate further due to an entry that
referred to a “CNIM own goal”. The Post Office maintained there were two separate
issues and the £50 was paid out due to “user error”.
Dr Worden’s evidence on this was not of enormous detail. The opportunity for an
expert to go away and do further or any research after a trial is not of great assistance
to the judge tasked with making findings on the evidence in fact available at a trial.
Mr Coyne accepted that this “bug” did not have lasting financial impact on branch
accounts. Post Office submits that there is no evidence of a bug in Horizon. The Post
Office also submitted that “neither of the two PEAKs referred to [in respect of this
bug] can properly be described as instances of a “Network Banking Bug”; both issues
stem from intermittent communications failures emanating from outside Horizon
(most likely from systems/kit operated by BT).”
The PEAKs show that it was communication difficulties with the branch that probably
led to these issues. There were two PEAKs. One related to an ISDN line possible
failure or other communication defect. The Horizon Issues define the system as
follows: “the Horizon System” shall for the purposes of this list of issues mean the
POL-0019320
417.
418.
POL00022841
POL00022841
Horizon computer system hardware and software, communications equipment_in
branch and central data centres where records of transactions made in branch were
processed, as defined in GPOC, at §16 and as admitted by Post Office in its Defence
at §37”. (emphasis added)
ISDN lines are provided by BT and go between branches and what would be called
“central data centres”. That link is not obviously included in the definition of the
system I have reproduced above. Such lines are not used any more and have been
teplaced by ASDL lines.
In any event, there is insufficient evidence in my judgment to conclude that there is a
“network banking bug” as expressed as this entry in the Bug Table. I therefore find
that there is no such network banking bug. I take that finding into account when
considering Horizon Issue 4. Mr Coyne accepted that this was not a separate bug that
went to Horizon Issue I in any
Type of Permissions
419.
420.
421.
422.
There is another issue that is relevant to the potential impact to branch accounts, and
that is the extent of permissions to Fujitsu personnel. Until 2012 all the personnel in
SSC had far wider permissions than were necessary, and wider than they were
supposed to have.
The PEAK that is associated with this issue, namely PC0208119 dated 10 March
2012, states in the subject summary “SSC Database users do not have correct
permissions”. The associated entries go from 1 February 2011 to 9 June 2015 (even
though the date on the covering page, or page I of the PEAK, is 14 May 2012). It is
necessary to go through all the way to the final page, page 8, to see the most recent
date when the PEAK was closed. Under “Impact Statement” the following entry
appears “3. The customer is not aware of this problem or change”. In my judgment
this shows two things. Firstly, the Post Office had not been told by Fujitsu that SSC
database users did not have the correct permissions. Secondly, the Post Office had not
been told by Fujitsu that a change is proposed to change this. There is a fix
extensively debated in the same PEAK and the entries strongly suggest that the fix
(which is the change referred to in the Impact Statement) is being done without the
customer (which must be the Post Office) knowing about it.
The entry of 16 August 2011 in the same PEAK states:
“The optional role 'APPSUP' is extremely powerful. The original BRDB design was
that 3rd line support should be given the 'SSC' role (which is select_any_table +
select_catalogue) and only given the optional role 'APPSUP' temporarily (by Security
Ops authorisation) if required to make emergency amendments in BRDB Live. Since
then Host-Dev have delivered a series of auditable amendment tools for known SSC
data amendment operations in Live, and these are assigned by role to individual SSC
user accounts. As such SSC should not require the APPSUP role in BRDB, unless
there is an unforeseen update required to Live.
Transferring to Steve Parker for review/assessment.”
An entry of 30 September 2011 states, in terms:
POL-0019320
423
POL00022841
POL00022841
“Tt_is a security breach if any user write access is not audited on Branch Database.
hence the emergency MSC for any APPSUP role activity must have session logs
attached under the MSC. Host-Dev previously provided scripts, such as the
Transaction Correction Tool, are written to run under the SSC role and also write to
the audit logs.
SSC users created on BRDB should only have the SSC role, and the user creation
script should be amended by Host-Dev to reflect this. A separate script
giving/revoking emergency MSC access via APPSUP can be delivered, logging this
to the hostaudit directory. In parallel Host-Dev should investigate any Host-Dev
delivered script to ensure they are all executable by the SSC role. SSC should
investigate any of their own scripts to ensure they have sufficient permissions under
the SSC role, taking into account they should primarily perform their work on BRSS.
Any day to day scripts should not access BRDB directly.”
(emphasis added)
The Technical Summary of the proposed fix in the entry of 25 October 2011 that
deals with the “development impact of fix” identifies which “HNG-X platforms are
impacted” and states:
“TECHNICAL SUMMARY:
This change is to change the script used by POA UNIX and POA ORACLE DBA
when creating new SSC/Support users.
LIST OF KNOWN DIMENSIONS DESIGN PARTS AFFECTED BY THE
CHANGE:
- BRDB_SOFTWARE_INSTALLATION_POSTMIG08_0622_<REL>
- BRDB_HNGX_POSTMIG08_CHANGES_0622_<REL>”
This shows that prior to this change to the script (which cannot have taken place prior
to the date it was implemented, for obvious reasons) all the SSC users had the very
powerful permission (also sometimes called privileges) of the APPSUP user. The
experts were agreed that such users could, in terms, do whatever they wanted in terms
of access to the system. That could obviously have an impact on branch accounts,
depending upon what was done by any particular user on any particular occasion. SSC
users should only have had the far more limited SSC role and Fujitsu itself were
aware of this, and can only be the entity responsible for them having the incorrect
wider role, as they were all Fujitsu employees. The section of the accompanying
judgment of Mr Godeseth in the judgment that accompanies this also refers to
evidence from My Coyne and Mr Parker on the same subject. Mr Parker challenged
Mr Coyne’s figure but had no basis for doing so, as all he had was his impression, and
he had specifically failed to do a proper investigation even though I find Fujitsu could
have provided far more proper, cogent evidence of the number of occasions. I accept
Mr Coyne’s evidence on this, and given both Dr Worden and Mr Godeseth accepted
that the powerful APPSUP permission or privileges were more widely available, and
POL-0019320
POL00022841
POL00022841
less controlled, than they ought to have been (even based on Fujitsu’s own internal
controls) then this inevitably has a detrimental effect upon the robustness of Horizon.
F. Conclusions on Technical Issues
424. Turning therefore to the summary of how many bugs were present in Horizon (either
Legacy Horizon, or Horizon Online), the Post Office cross-examined Mr Coyne on
the total number of bugs from the Bug Table of which he had seen evidence of lasting
impact to branch accounts. Although the Bug Table was numbered 1 to 29 (suggesting
there were a total of 29 separate ones) rather confusingly Bug 6 was split into two,
6(i) and 6(ii). The bugs which were accepted by the IT expert for the claimants as not
providing evidence showing lasting impact on branch accounts were those in the Bug
Table, other than those numbered I to 14 inclusive, 18, 22, 23, 24, 25, 26, 27 and 28.
That is a total of 22 different bugs that did show such evidence, but only if Bug 6 is
counted as one bug. If counted as two bugs, there would be 23. This is a convoluted
way of saying that in the experts’ agreement, when they agreed they had seen a range
of 12 to 29 bugs that showed evidence of lasting impact to branch accounts, that upper
limit should be 23 not 29. Of the other bugs, for the most part Mr Coyne considered
they were relevant to Horizon Issue 4 regarding robustness.
425. However, the cross-examination which elicited this put the questions in terms of
evidence of /asting impact on branch accounts. The distinction between transient and
lasting impact was, as explained in the substantive judgment, a differentiation that the
Post Office adopted (originally it seems in Dr Worden’s reports, but also in its
evidence and the way the case was put) and is not included in the Horizon Issues. Of
the 29 bugs in the Bug Table (30 if one counts number 6 as two separate ones) I have
found that those numbered 16, 17, 20 and 29 are not bugs. The remainder plainly are,
in my judgment, and that remainder, of which there are 25 (26 if one counts number 6
as two separate ones) are bugs with the potential to impact upon branch accounts. I
accept Mr Coyne’s evidence that there are 22 bugs in the Bug Table (again, the total
is 23 if one counts number 6 as two) with evidence of actual lasting impact to branch
accounts having occurred. In my judgment, that is a high number of different bugs
present in Horizon.
426. Further, in terms of my findings, 16, 17, 20 and 29 are as follows. Entry 16 shows
software faults, although these do not have the potential to cause impact to branch
accounts. Entry 17 is an error or defect, and has occurred only once. Entry 20 is not a
bug but it is an error or defect as it consists of a hardware fault. Entry 29 is not a bug,
and is a fault with the BT lines that connect the branch to the central data centre. It is
really only therefore Entry 29 in which the Post Office has had any success in
defeating the claimants’ case on the existence of bugs.
427. Ultimately, the Post Office in Appendix 2 of its Closing Submissions stated that of the
29 bugs in the Bug Table, eight were not bugs at all. Those that it challenged as not
being bugs at all were those numbered 8, 13, 15, 16, 17, 20, 23 and 29. This by
definition means that 21 of the 29 were accepted by the Post Office as existing. This
was a sensible approach; in my judgment the claimants’ evidence of the existence of
those 21 as bugs was very strong. The dispute therefore moved to the effect of bugs,
rather than whether they existed at all. Three were said to have had no branch impact;
nine had or potentially had only transient impact; and nine were said cause or had the
POL-0019320
428.
429.
430.
431.
432.
POL00022841
POL00022841
potential to cause lasting impact, but it was said this was resolved by the Post Office
and Fujitsu.
I consider, for the purposes of the agreed Horizon Issues (rather than the re-cast
issues, with the Post Office introducing themes helpful to their case) that even those
bugs with what were said to have had “transient” impact had the potential to cause
impact to branch accounts, with the sole exception of Entry 29. I have found that
Entry 15 was a bug; Entry 16 were software faults, albeit with no potential to cause
impact to branch accounts; Entry 17 was an error or defect that only occurred once.
Entry 20 was not a bug but was an error or defect. Therefore, even those that were not
bugs were errors or defects.
The three which the Post Office maintained had no branch impact were Entries 11, 21
and 22. I have found Entry 11 was a bug; I have treated it as one single bug, as that is
how it was treated in the Bug Table (with numerous PEAKs against single entries).
However, as explained above against that entry, the PEAKs show a variety of
different software problems concerning Girobank. I have also found that Entries 21
and 22 are both bugs.
Arriving at an overall final total of bugs therefore is not apt to be helpful, in my
judgment. This is particularly so if one attempts, as the Post Office did, to downplay
their effect by introducing new concepts such as “transient” impact. All that does is
dilute, or seek to confuse, in terms of the potential to cause apparent or alleged
discrepancies or shortfalls relating to Subpostmasters’ branch accounts or
transactions; and/or to undermine the reliability of Horizon accurately to process and
to record transactions. That last sentence keeps, so far as possible, to the actual
wording of Horizon Issue 1, which was agreed by the parties a long time before the
Horizon Issues trial. On some of the entries in the Bug Table, such as Bug 11
concerning Girobank, I have termed it a single bug (as the experts did that) but the
underlying PEAKs show multiple bugs under that single heading. It could quite easily
be counted as six separate ones, a point supported by the Post Office’s own
submissions identifying the six separate matters contained in the numerous PEAKs
grouped by the experts under one single heading. If it is counted as six bugs, and if
Bug 6(i) and 6(ii) are counted as two (which they ought to be, but which the experts
grouped together in their range of “12 to 29”, (later corrected by Mr Coyne to an
upper number of 23) then the total number of bugs of which I have found specific
evidence of having the potential to cause impact to branch accounts is about 30.
There is no specific number of bugs in the Bug Table which acts as a threshold below
which Horizon suddenly becomes reliable, or above which it suddenly becomes
unreliable. The whole of the expert evidence, including the analysis and agreements
within the Bug Table, and the effect of the very numerous bugs, errors and defects
which I have found to exist, must all be taken into account, as well as the ultimate
total. That I have done in reaching my conclusions on the Horizon Issues.
Finally on this subject, during the claimants’ oral closing submissions, there was an
outline chronology narrative undertaken of a series of Post Office and Fujitsu
documents running from May 2000 onwards, as the claimants’ leading counsel
explained it, to “identify the problem that SPMs had with doubles”. I have referred to
this in part at [62] to [64] of the substantive judgment. This “doubling” problem or
issue was that sometimes there would be a discrepancy — in the example given in
POL-0019320
433.
434.
POL00022841
POL00022841
PEAK 0043811 of May 2000, a sum which was expressed in the PEAK as being a
“discrepancy in the cash account”, in that instance the sum being £6,343,07 — which
then inexplicably doubled to £12,686.14. The chronology then went through different
years, 2003, 2004, 2006 and into 2010. Similar documents showed a variety of issues
all related to such doubling. The majority of the documents had all been put to
different witnesses in the trial, and were referred to in the Bug Table. Finally the
chronology ended with a heavily redacted internal Post Office document of 23 May
2018, called “Agenda — Operations Board” one entry of which stated (bureau means
bureau de change, or foreign currency):
“SAP GUI Bureau Value Issues
There_have been 3 reported instances this week where the sterling value of
remittances of bureau to branches has doubled when the delivery has been booked
into the branch, For example, Aylesbury GPO were sent a bureau rem with a total
value of £6.219,81, however when they booked it into branch it populated as
£12,439.62”
(emphasis added)
This narrative exercise drew some criticism from leading counsel for the Post Office.
He said that the exercise in which the claimants’ counsel was engaged was contrary to
the rules of commercial litigation, and the objection to this was put as follows:
“Another one of those rules is not to conflate a whole series of issues, all of which are
very different, bureau rems in, we have got other kinds of rems, we have got transfers
between stock units, all sorts of different issues which my learned friend picks up
randomly over a period of 20 years and seeks to promote them to your Lordship as a
single problem which Post Office has known about for all that time.
Your Lordship will have seen from the PEAKs that that's not the case. There were
individual instances of individual problems, as far as I can tell, some of these
documents I have not seen before of course, which itself is extraordinary given the
nature of this trial and that we are at the end of it.
But my Lord what's happened is my learned friend is seeking to jumble things up and
then create an impression.”
He invited the claimants’ counsel to stop. The claimants’ counsel countered this
objection by explaining that he was demonstrating that “we have found a wide range
of strands of causes of doubling issues in the accounts of SPMs. We don't want to
conflate them. We want to identify that, not only are there cases where a SPM -- for
example, the regional line manager puts it into suspense [account] and it doubles...”
The reason for reproducing this is as follows. The Post Office seemed to draw
comfort from the fact that there was no single bug or cause of what was, in my
judgment, on the face of the PEAKs, quite inexplicable doubling of branch
discrepancies, but rather the 18 year history of these arose from a series of different
problems within Horizon. The Post Office also insisted, and seemed to wish to
continue to insist, that these were “individual instances of individual problems”. That
approach has gone on for many years, and one would have thought that by now the
POL-0019320
POL00022841
POL00022841
Post Office would have realised that something of a pattern has emerged in terms of
Horizon’s performance. The internal Post Office and Fujitsu documents (running
through the whole life of Horizon in this example) all demonstrated, to my
satisfaction, unreliability of certain branch account entries due to this doubling up of
discrepancies. Even the 2018 Operations Board document showed that these were still
occurring in mid-2018 — on three separate occasions in that week alone. It does not
matter, in my judgment, to the claimants’ case, whether there was one single cause for
these specific discrepancies, or a number of different ones, all of which had a similar
effect. The claimants’ case was, in any event, that there were a number of different
bugs, errors and defects in Horizon that all had this, or a similar, effect. Nor does it
much matter for the purposes of the Horizon Issues whether there was one single
cause, or a number of different ones over a number of years. My findings in this
appendix show that there were a large number of bugs, errors and defects. There was
a far larger number than were admitted by the Post Office at the beginning of the
Horizon Issues trial, and a far larger number than ought to have been present in the
system if Legacy Horizon and HNG-X were to be considered sufficiently robust such
that they were extremely unlikely to be the cause of shortfalls in branches.
435. The issue of the state of knowledge of the Post Office is for another subsequent trial.
This trial deals with the answers to the Horizon Issues only, and all the other matters
in the litigation remain for trial on future occasions.
436. Going through the evidence, PEAKs and KELs, it can be seen that there is a slight
majority of the bugs, errors and defects that I have found to exist present in Legacy
Horizon, as compared to the number in Horizon Online. However, there are still a
significant number in Horizon Online, although not quite as many as in Legacy
Horizon. The former regarding Horizon Online are those numbered 1, 3, 4, 5, 7
(although in the pilot), 8, 13, 14, 19, 23, 25 and 28. Those in Legacy Horizon are 2, 6,
9, 10, 11, 12, 15, 17 (an error or defect, not a bug), 18, 20, 21, 22, 24, 26 and 27. It is
also notable, in my judgment, that those in Horizon Online were in its earlier years.
There is only one that dates from 2017 onwards, namely bug 14, Bureau
Discrepancies, with some instances of bug 8, Recovery Issues running into 2018. It
must however be remembered that this analysis is on the KELs made available to the
experts. A very significant number of KELs were produced by Fujitsu towards the end
of the trial which the experts did not have time to consider, and an even more
significant number were produced in September 2019, well after the trial had ended.
Neither expert has studied these.
437. Ido not know if what appears to be a marked improvement in the performance of
Horizon since about 2016/2017 is because, since this litigation commenced, matters
have changed at Fujitsu in relation to the Horizon System, including reporting and
controls. There is no need to speculate. The experts are agreed that Horizon, as it is in
2019, is relatively robust.
438. However, the Horizon Issues are not concerned with the Horizon system as it is at
2019. They are concerned with the whole life of the system, from 2000 onwards. Mr
Coyne also gave evidence that, given he had not been able to consider all of the
PEAKs and KELs that were disclosed prior to the trial, as a matter of extrapolation
(on the number of KELs of which the claimants were aware at that time, which did
not include the many thousands disclosed later in 2019), he would expect there to be a
maximum of about 40 bugs in the Horizon System.
POL-0019320
439.
440.
441.
442.
443.
POL00022841
POL00022841
I accept that Mr Coyne was doing his best to assist the court, but this evidence is
rather broad brush in its approach. Given there have been, effectively, three iterations
of the system, the number of bugs, errors and defects have to be considered against
the iteration of the system in which they occurred. An obvious example of this is
Legacy Horizon and Riposte. Riposte is no longer used and has not been since 2010.
A simple extrapolation across all 19 years of the life of Horizon (ie, 2000 to 2019)
would not lead to a sound conclusion. In any event, the number of bugs, errors and
defects that have been found by me based on the Bug Table, without extrapolation but
by the above analysis, shows that there is ample material, without the need for any
extrapolation, to come to measured conclusions on robustness in any event.
The Post Office contended, in its closing submissions, for answers (this was most
stark in the issues that arose regarding robustness) that treated Horizon simply as a
single system. As an example, the answer for which it contended on robustness was
summarised as “the Horizon system is and was very robust (being more robust than
most comparable systems).”
I do not consider that it is justified to have one single or simplistic answer to the issue
of robustness that governs the whole life of Horizon over its 19 years of operation.
Given the expert evidence and agreements alone, it is not correct that any finding of
the robustness of Horizon as at 2018 and 2019 equally applies to the system, say, in
2001 and 2002. I reject this as the correct approach.
I consider that the correct way to approach this issue is to consider the following three
iterations of the system:
1. Legacy Horizon, 2000 to 2010;
2. Horizon Online in its HNG-X form, which ran from the introduction of Horizon
Online in 2010 to February 2017;
3. Horizon Online in its HNG-A form, when it was changed to run on a different
platform, namely Windows 10.
It is correct to record that the experts did not approach the matter with a routine
marked distinction between the two versions of Horizon Online, HNG-X and HNG-A.
However, that distinction undoubtedly exists and Mr Godeseth explained the switch
from NT4 to Windows 10 in his evidence. The internal Post Office IT risk
management document from 2017 stated that “the HNG-X platform is end of life and
is running on unsupported Windows software”, that it needed replacing, and also that
the "Branch counter technology is aged and unreliable, with frequent hardware
failures, resulting in branch disruptions." This plainly recognised that replacing the
platform, which is what occurred when the system became HNG-A, would result in
some improvements, and this appears to have happened, although the counter
technology in the branch would not necessarily also be changed by a change from one
Windows platform (NT4) to another. The risk management document envisaged the
required replacement of both. Also, it has to be taken into account that the Horizon
Online bugs analysed in Part E of this appendix are, for the most part, those in HNG-
X. I do not see why the HNG-A form, which both experts agree is relatively robust,
should be considered in exactly the same light of the PEAKs that arose (and there are
a great many) during the HNG-X years. Dr Worden really only concentrated on the
POL-0019320
444.
445.
POL00022841
POL00022841
HNG-A iteration of the system in any event, as he considered the operation and
functionality of the system as at 2018 and 2019, and this means HNG-A. He readily
accepted that his reports concentrated on Horizon as it was at the time of his
investigation.
In my judgment, the following conclusions on robustness can be drawn regarding the
three different iterations of the Horizon system from 2000 to date:
1. Legacy Horizon: This was not remotely robust. Indeed, the issue about its
robustness (or more accurately, its lack of robustness) became increasingly obvious
during the Horizon Issues trial. The fact that the Post Office’s final submissions were
forced to concede the existence of so many bugs, with the battleground moving to the
type of effect they had, rather than their existence, clearly demonstrates in my
judgment that all the weight of evidence, both of fact and expert, was heavily against
the proposition that Legacy Horizon was robust. It clearly was not.
2. Horizon Online in its HNG-X form: This appears to be slightly more robust than
Legacy Horizon, but still had a significant number of bugs, particularly in the years
2010 to 2015, but also after that and leading up to 2017. Its robustness was therefore,
in my judgment, questionable and did not justify the confidence routinely stated by
the Post Office (prior to February 2017) in its accuracy.
3. Horizon Online in its HNG-A form: This is far more robust than either of the
previous two iterations of the system. I accept Mr Coyne’s evidence that there are still
far more TCs than expected arising during its use, compared with comparable systems
in the banking and finance sectors. However, it may be that this is the only way in
which it can sensibly still be operated. TCs are outside the scope of the definition of
the Horizon system in any event. It is also an unavoidable conclusion that,
notwithstanding the change of platform from Windows NT4 to Windows 10, it is still
a Horizon system, which given my conclusions at (1) and (2) is hardly a glowing
endorsement of its antecedents. However, the experts are agreed that HNG-A is
robust, and better than it was in either its Legacy Horizon or HNG-X forms. Horizon
is not free from bugs from any particular date. As shown by the Drop and Go bug,
number 28 in the Bug Table, this occurred in June 2017 (the KEL is headed HNG-X
as explained in [410] which is after the date HNG-A was introduced, so it may be a
bug in HNG-A). Drop and Go appears to be a product introduced by the Post Office
relatively recently, but I am unaware of the precise date. The experts should seek to
agree whether Bug 28 is a HNG-X or HNG-A bug.
It is also relevant to record that the Horizon system as defined in the Horizon Issues,
and as pleaded in the Generic Particulars of Claim and Defence referred to in that
definition, does not provide any particular end date. Therefore HNG-A is included
within that definition. It is also, however, relevant to record that the effect of any
bugs, errors and defects in HNG-A is likely to be limited in this group litigation, as
there is a cut-off date for claimants to have joined the group litigation. This cut-off
date is 24 November 2017, as extended in paragraph 15 of the 1% Case Management
Order sealed on 27 October 2017. The final date by which claims had to have been
entered on to the Group Register was 8 December 2017. November 2017 is only
about nine months after HNG-A came into being; therefore the vast majority of the
claims are likely to arise under Legacy Horizon and HNG-X, rather than HNG-A.
POL-0019320
POL00022841
POL00022841
446. However, that does not mean that HNG-A is free from faults. Paragraph 3.1 of the 3
Joint Statement recorded that the experts agreed that “robust' does not mean infallible
and therefore Horizon has and will continue to suffer faults. Robustness limits the
impact of those faults and other adverse events.” Also, Mr Latif’s experience with the
Lottery and the scratch card discrepancy occurred in January 2018. That means it
arose under HNG-A. I have accepted his evidence on this, which means that HNG-A
must still contain bugs, errors and defects that have the potential to impact branch
accounts.
447. Ihave also identified in the substantive judgment that the Post Office’s plans, if there
are any, in terms of the evolution of its IT systems and replacement of HNG-A, are
not of themselves to be taken as supporting the claimants’ case. I assume any plans
for what was referred to as Thin Client are either developed or developing, but there is
no need to speculate.
448. I return therefore to the passage at [8] above which is reproduced from Dr Worden’s
teport where he was dealing with the double entry accounting principle. He stated that
“any mistake is most likely to have occurred in one column of figures, without any
balancing mistakes on the other columns. So, the mistake will immediately destroy
the trial balance.” The consequence of the many bugs that I have found that exist in
Horizon, in both its Legacy and Online form, have the effect of making the detection
of errors so much harder than in conventional double entry bookkeeping. This is
demonstrated by so many of the Fujitsu documents where a great amount of
investigation was required by Fujitsu simply to work out what had actually happened,
and why. On many investigations this took even Fujitsu an extraordinary length of
time, in excess of a year, and sometimes even two years. Members of 3“ line support
were involved, on multiple occasions, in this process. The experts are also agreed that
SPMs only had a relatively limited amount of reporting available to them on Horizon,
and that proper investigation required co-operation between the Post Office and the
SPMs. It must also be understood that the trading periods for each Post Office branch
were usually 4 weeks, and sometimes 5 weeks. At the end of each trading period a
SPM had to complete a Branch Trading Statement and then “roll over” for the next
period to start. If a bug in Horizon had an impact on branch accounts, then the Branch
Trading Statement for that period would include the effect of that impact.
449. One of the other functions of an accounting IT system was described by Dr Worden as
the secure storage and output of many types of information. That secure storage (what
was called in this litigation the audit store) is not effectively used if it is never, or
rarely, accessed, particularly in circumstances where there is a dispute between both
parties to the account. By “both parties” I refer to a SPM and the Post Office. This
point is addressed in more detail in Part K of the substantive judgment.
450. There is reference in some of the more recent documents to the involvement of
Accenture. The analysis of the Drop and Go bug, number 28 in the Bug Table, shows
that even when a conclusion was drawn by Accenture (in this instance relating to the
failure of one of two £100 “top up” transactions), this did not prevent Fujitsu from
identifying in the KEL the following: “Problem: this may be an issue with script
ADCScript-CDBalanceTopUP, or a user error”, even though there was plainly no user
error. User error means an error by the SPM. I do not know if this would be recorded
POL-0019320
451.
452.
POL00022841
POL00022841
at Fujitsu in the same way if an incident similar to this were to occur today. However,
it shows that for an occurrence even as relatively recently as the summer of 2017,
Fujitsu failed to give sufficient — or indeed any - weight to the views of Accenture,
who had “verified” that one of the top-ups had failed. Fujitsu still recorded potential
user error in the KEL, a point which is plainly contradictory to the outcome of
Accenture’s investigation in that instance.
The expression put to Ms Van Den Bogerd in cross-examination was “user error
bias”, an expression which she said she had not heard before. This term was explained
as a tendency or bias regularly to blame the user of an IT system for something, and
the point put to her was that this was “a common theme that runs through the Post
Office’s approach” in this dispute. The exact passages of her evidence are as follows:
“Q. UEB, have you ever heard of that?
No.
User error bias?
. No, I can't say --
. It is where people in IT constantly blame the user when actually it is not their fault?
. Okay, I haven't heard that before.
. You haven't heard that?
No.
Well, let's look at this because I'm suggesting to you this is a common theme that
runs through Post Office's approach when these issues are raised. Is that a fair
suggestion?
A. I think what I have tried to explain is what the approach is, in 99 out of 100 that
would be the case. In this situation it is slightly different.”
OPO>POPO>
I do not know from where Ms Van Den Bogerd obtained her 99% statistic. Given
there are 6 million transactions per day, it may not be particularly helpful to the Post
Office in this group litigation even if it were accurate. If it was meant as a 99%
statement of certainty on her part of the strength of the Post Office’s case on the
Horizon Issues, it is plainly very wrong, as demonstrated by my findings both in this
appendix and the substantive judgment. As can be seen from the answers to the
Horizon Issues in Part M of the judgment, the claimants have been largely successful
in their case on the Horizon Issues. In my judgment — and regardless of whether
Fujitsu and/or the Post Office was applying user error bias, or not — it is clear to me
that Fujitsu was far too ready, even after investigations that clearly included the
express discovery of bugs in code, to ascribe possible user error to the effect of bugs,
errors and defects that caused impact to branch accounts. It is to be hoped that this is
no longer the situation.
POL-0019320