FUJ00159313
FUJ00159313
THE POST OFFICE GROUP LITIGATION
Claim Nos HQ16X01238, HQ17X02637 and HQ17X04248
IN THE HIGH COURT OF JUSTICE
QUEEN'S BENCH DIVISION
BEFORE THE HONOURABLE MR. JUSTICE FRASER
BETWEEN:
ALAN BATES & OTHERS
Claimants
POST OFFICE LIMITED
Defendant
CLAIMANTS’ PROVISIONAL/OUTLINE DOCUMENT IN RELATION
TO THE HORIZON ISSUES
This provisional outline document is served in compliance with paragraph 15 of the Third
CMC Order and has been prepared without the benefit of full disclosure and the exchange of
evidence. It should be read with the Claimants’ Generic Particulars of Claim and Reply.
FUJ00159313
FUJ00159313
HORIZON ISSUES
BUGS, ERRORS AND DEFECTS IN HORIZON
Accuracy and integrity of data
rep)
To what extent was it possible or likely for bugs, errors or defects of the nature alleged
at_§§23 and 24 of the GPOC and referred to in §§ 49 to 56 of the Generic Defence to
have the potential to (a) cause apparent or alleged discrepancies or shortfalls relating to
Subpostmasters’ branch accounts or transactions, or (b) undermine the reliability of
Horizon accurately to process and to record transactions as alleged at §24.1 GPOC?
The Claimants’ outline case is that it is clear that bugs, errors and defects have affected
both Subpostmasters’ branch accounts and transactions and have undermined Horizon’s
reliability. The fact that these bugs, errors or defects could arise, and that they have in
fact arisen, is consistent with, if not probative of, the possibility and likelihood that other
similar bugs, defects and errors have arisen which would similarly affect
Subpostmasters’ branch accounts and/or transactions. The Defendant admits that certain
bugs or errors in Horizon have occurred and that some of them resulted in discrepancies
and thus shortfalls in branch accounts (paragraph 56(1) of the Generic Defence and
Counterclaim).
It is common ground that the Horizon system is not perfect and it is not asserted by the
Defendant that it has never had any errors or bugs. It is also agreed that the volume of
transactions effected through Horizon is very large. The admitted imperfections in
Horizon (which the Defendant maintains result in a low probability of errors arising) are
entirely consistent with the levels of errors asserted by the Claimants, when regard is had
to the millions of transactions processed by Horizon.
The bugs, errors or defects admitted by the Defendant, include:
(i) the Calendar Square/Falkirk bug — which led to Horizon failing to recognise
transfers between different stock units, thereby affecting Branch accounts;
FUJ00159313
FUJ00159313
(ii) the Payments Mismatch defect — which affected at least 62 branches and was
related to the process of moving discrepancies into the local suspense account,
thereby affecting Branch accounts. This defect was not capable of being
identified by Subpostmasters, who would have believed from Horizon that their
account was balanced when it was not; and
(iti) the Suspense Account bug — which erroncously replicated suspense account
items for 14 branches from 2010 in the same monthly trading periods in 2011 and
2012, thereby affecting Branch accounts.
It is alleged by paragraph 56 of the Generic Defence and Counterclaim that no
Subpostmasters were “ultimately” held responsible for the resultant shortfalls caused by
these bugs, errors or defects. (The evidence shows, for example, that the Payments
Mismatch document shows that for a significant period of time, Subpostmasters were
not informed of a known defect affecting their accounts.) Even taking the Defendant’s
case at its highest, it is therefore self-evident that bugs, errors or defects in Horizon did
have the potential to (a) cause apparent or alleged discrepancies or shortfalls relating to
Subpostmasters’ branch accounts or transactions, or (b) undermine the reliability of
Horizon accurately to process and to record transactions.
Other bugs, errors and defects which have been identified thus far and which caused
discrepancies affecting Branch accounts, and therefore had the potential to do so,
included:
(i) Known Error Log achal233J — causing a discrepancy between branch cash
declarations and the amount received by the Cash Management system;
(ii) I Known Error Log achal717T — caused by an “unknown system problem”
causing a difference between the cash declared by the branch and the cash which
the system considered that the branch should have had;
(iii) I Known Error Log acha621P — caused the wrong screen to be presented leading
to duplication and/or incorrect entries of Remittance In transactions (see also
paragraph 4.1 below);
(iv) I Known Error Log acha 194L — resulting in cash declarations from kiosk not
being automatically passed into Horizon;
(2)
2.1
G)
FUJ00159313
FUJ00159313
(v) I Known Error Log LKiang3014S — after making multiple cash declarations and
then running a trial balance report, the calculation for the trial balance was
incorrect;
(vi) Known Error Log MScardifield2219S — cached data was not updated via
Riposte, resulting in incorrect data being presented in discrepancy, variance and
balance reports;
(vii) Known Error Log wbra5353J — a customer was charged twice for the same
transaction, as a likely side effect of bad reference data within Horizon; and
(viii) Known Error Log jsimS530K — reference data controls would not allow the
reversal of cheque rem-out mode transactions.
Did the Horizon IT system itself alert Subpostmasters of such bugs, errors or defects as
described in (1) above and if so how.
The Claimants are not aware of any automated system within the Horizon system to
inform Subpostmasters of bugs, errors or defects, either in relation to matters set out in
paragraph I above or at all. The Claimants infer and invite the Court to infer that there
is no such automated system. The way in which the Defendant dealt with the Payments
Mismatch defect (paragraph 1.3(ii) above) demonstrates that errors known to the
Defendant were not automatically communicated to Subpostmasters by Horizon and that
the Defendant (a) did not do so for a significant period and (b) then considered whether
and how to do so. Given that the Defendant has been unable to identify any
communication by which Subpostmasters were informed of the Payments Mismatch
defect, the Claimants will invite the Court to draw the inference that they were not so
informed, unless the Defendant can demonstrate otherwise.
To_ what extent_and in what respects is the Horizon System “robust” and_ extremely
unlikely to be the cause of shortfalls in branches?
[GPOC §23 and 24; Defence §§49 to 56]
3.1
3.2
FUJ00159313
FUJ00159313
There is no single, authoritative definition of “robust” in the context of a system such as
Horizon, which appears to be more commonly used in public relations than as an
objective performance standard. Pending any clarification of its objective meaning, it
appears to relate to the ability of a system to perform correctly in any scenario, including
where invalid inputs are introduced, namely, to have in place effective error repellency.
It is clear that, on a sensible construction of the term “robust”, Horizon did not meet this
standard because:
(a)
(b)
(d)
(ce)
(6)
it contained bugs, errors and defects as set out in paragraph I above which
created discrepancies in the branch accounts of Subpostmasters;
it suffered failures of internal mechanisms which were intended to ensure
integrity of data. Examples include Known Error Log dsed4733R (causing
multiple failed recoveries as a result of a wrongly named recovery script) and
Known Error Log obenge5933K (transaction only recovered partially following
a loss of communications);
the system did not enable such discrepancies to be detected, accurately identified
and/or recorded either reliably, consistently or at all;
the system did not reliably identify ‘Mis-keying’, which is inevitable in any
system with user input, and did not reliably have in place functionality to restrict
users from progressing a mis-key;
it required numerous processes and workarounds to be in place to allow Fujitsu
to modify and correct data already recorded by Horizon, which would not be
required in a “robust” system; and/or
there were weaknesses and risks of errors and other sources of unreliability
within Horizon, of which the Defendant was well aware from internal or third
party reports, examples of which the Claimants have only recently identified in
Stage 3 Disclosure, as follows:
3.3
(i)
(ii)
(iii)
(iv)
FU,
POL-219218 — Ernst & Young report to the Defendant on, inter alia, weaknesses
relating to the control environment operated by the Defendant’s third party IT
suppliers for year ended 27 March 2011;
POL-217341 — Defendant’s report (Review of Key System Controls in
POLSAP) dated November 2012 highlighting the potential for errors not being
identified or resolved leading to discrepancies in client balances and the risk of
inappropriate access being assigned to users, leading to inappropriate activities
being undertaken within the system;
POL-217378 — Minutes (redacted) of a meeting on 18 September 2013 of the
Defendant’s Risk and Compliance Committee recording a decision not to
mitigate risks identified by Ernst & Young relating to the communication by
Fujitsu of changes made to the Horizon system; and
POL-0216106 — Report (draft — 1 October 2013) by Detica (BAE Systems)
‘Fraud and Non-conformance in the Post Office’; Challenges and
Recommendations’ concluding, inter alia, (a) ineffective process, policy and
working practice across the various Post Office central teams to gather
information, prioritise and act in a coordinated manner; (b) technology
available to central teams are not fit for purpose; and (c) Post Office systems
are not fit for purpose in a modern retail and financial environment.
However, as noted above, whether Horizon is ‘robust’ plainly depends upon the
definition given to that word. Even the small chance of errors, bugs and defects
admitted by the Defendant and/or supplemented by those underlying the reports above,
would be likely to produce the result alleged by the Claimants. Therefore, as noted
above, even if the overall probability of a bugs or errors affecting Branch accounts is
small, the sheer number of transactions undertaken by the Horizon system is consistent
with the level of discrepancies arising from bugs and errors in issue in these proceedings.
FUJ00159313
1J00159313
FUJ00159313
FUJ00159313
Controls and measures for preventing / fixing bugs and developing the system
(4)
4l
42
43
To what extent has there been potential for errors in data recorded within Horizon to
arise in (a) data entry, (b) transfer or (c) processing of data in Horizon?
There were inadequate controls incorporated as part of Horizon to prevent and/or
remedy data entry errors, as evidenced by the ability to select and enter different
methods of payment (Known Error Log allend1645p) and the non-appearance of the
correct screen to successfully process a cash pouch, which caused a clerk to
inadvertently double up the amount of cash recorded (Known Error Log acha621P).
There was the potential for errors with the transfer of data in Horizon, as evidenced by:
(a) a transaction authorised by a bank but cancelled by pin pad, which was not reversed
(Known Error Log cardc219R);
(b) a successfully recorded transaction failing to appear in the PODG file and
consequently not being transferred to the relevant third party (the Environment
Agency) (Known Error Log jharr1323L); and
(c) the erroneous recording of the same transaction twice after a session transfer to a
different counter, which was recognised as a possibility for both NS&I and
Network Banking transactions (Known Error Log Marris34331).
There was the potential for errors in the processing of data, as evidenced by:
(a) the system bug which led to both the old value and the amended value being
stored and used in a transaction when a counter level correction was made via
the “Previous Key” (Known Error Log CharltonJ2752T);
(b) a declined network banking transaction that still resulted in money being taken
from the customer’s account (Known Error Log SSur343P);
(c) E-pay transactions crediting the customer account twice, although only one
payment had been taken (Known Error Log LKiang3526R and Known Error
Log SSur5310P);
FUJ00159313
FUJ00159313
(d) POLSAP outage — 25 & 26 January 2016 causing £90m of inward remittances
from branches failing to be processed (see document POL-216412); and
(e) “Accuracy of the FAT’ where figures recorded on the R&B and Tier 2 FAT
have not been accurate and have not agreed with the figures on Horizon (POL-
0216223).
(5) How, if at all, does the Horizon system itself compare transaction data recorded by
Horizon against transaction data from sources outside of Horizon?
5.1 It is presently understood that the process by which the Horizon system compares
transaction data recorded by it against transaction data derived from sources outside of
it is known as Reconciliation. This comprises a large number of complex, electronic
systems to monitor transactions but there is also the ability for manual intervention to
establish and correct erroneous data. Those systems presently identified by the
Claimants are listed at Appendix 1. The scope for error in Reconciliation depends, in
part, upon the interface between these systems. The Claimants believe that manual
interventions could be undertaken by at least the POL Finance Department, the Fujitsu
Management Support
Unit and the Fujitsu Third Line Support. How Horizon’s functionality allowed these to
be made is still being considered, but errors occurring in the electronic interface between
systems and the exchange of data (e.g. erroneous or duplicate data) appears to be a
significant risk area for generating errors which may affect Branch accounts.
(6) To what extent did measures and/or controls that existed in Horizon prevent, detect.
identify, report or reduce to an extremely low level the risk of the following:
a. data entry errors;
b. data packet or system level errors (including data processing, effecting, and
recording the same);
c. a failure to detect, correct and remedy software coding errors or bugs:
d. errors in the transmission, replication and storage of transaction record data;
and
FUJ00159313
FUJ00159313
e. the data stored in the central data centre not being an accurate record of
transactions entered on branch terminals?
[GPOC §§5, 14-15, 24.1, 24.1A, 94A, 95;
Defence §§35(2), 36, 38(1), 50(1), 52-54; Reply §41]
6.1 The Claimants refer to the matters set out above under issue (4) and _ find it difficult to
identify any such measures and controls, let alone measures and controls which were
effective to prevent, detect, identify, report or reduce to an extremely low level the risk
of the errors referred to above. On the basis of the Claimants’ investigations to date,
such measures and controls as were incorporated were insufficient and inadequate for
the purposes referred to in the question, as evidenced by:
(a) the inadequate controls incorporated as part of Horizon to prevent and/or
remedy data entry errors, as referred to in paragraph 4.1 above;
(b) the issuing of a Transaction Correction rendering the Subpostmaster liable
following a reversal initiated by Horizon (Horizon data Lepton SPSO 191320);
(c) the absence of a formal reconciliation produced between the POLSAP system
and the Credence transaction scheme; and
(d) the absence of an instantaneous or prompt fix for bugs, errors and defects
detected or identified. Such issues once discovered were often deferred and
dealt with only on a cost/benefit basis.
OPERATION OF HORIZON
Remote Access
(7) Were Post Office and/or Fujitsu_able to access transaction data recorded by Horizon
remotely (i.e. not from within a branch)?
[Defence §7; Reply §9]
7A
72
73
FUJ00159313
FUJ00159313
The Defendant has publicly stated that it was unable to access transaction data and edit
or delete transactions recorded by branches (paragraph 26 of the Generic Particulars of
Claim).
The Defence now admits that this was incorrect (paragraphs 56-58 of the Generic
Defence and Counterclaim): such remote access was possible.
The Claimants are still investigating the scope of the functionality by which this was
possible.
Availability of Information and Report Writing
(8)
8.1
(a)
(b)
(c)
(d)
(ec)
a)
8.2
What transaction data_and reporting functions were available through Horizon to Post
Office for identifying the occurrence of alleged shortfalls and the causes of alleged
shortfalls in branches, including whether they were caused by bugs, errors and/or defects
in the Horizon system?
[Defence §7; Reply §9]
It is believed that the following data and reporting functions were available to the
Defendant:
the TPS Report Set;
the APS Report Set;
the DRS Report Set;
an ARQ report (which would provide substantial information on a branch account but
which would only be made available to the Defendant on request to Fujitsu on
commercial terms and/or at a cost not presently known to the Claimants);
reports and data obtained by means of the Business Incident Management process;
and
reports and data obtained by means of the Problem Management Procedure.
The Claimants are still investigating what other data was available to the Defendant.
FUJ00159313
FUJ00159313
(9) At_all_ material times, what transaction _data_and reporting functions (if any) were
available through Horizon to Subpostmasters for:
a. identifying apparent or alleged discrepancies and shortfalls and/or the causes of
the same; and
b. accessing and identifying transactions recorded on Horizon?
[GPOC §§14.2-14.3, 17 and 19.3; Defence
§§38(2)(b), 38(3), 46(2); Reply §15.2-15.3]
91 Subpostmasters were able to access limited reports and receipts produced by the
Counter (i.e. the terminal in Branch). These reports provided basic information of the
individual transaction as it appeared on Horizon but would not in themselves enable
apparent or alleged discrepancies and shortfalls and/or the cause for such faults to be
identified. Further, transactions listed on Trading Statements were presented as totals
so that any individual errors could not be identified. In addition, transactions involving
third parties (eg., the National Lottery) could be aggregated for periods that did not
align with the trading periods used by the Defendant. If and insofar as information was
available, this would not enable Subpostmasters to identify any error which occurred
beyond their Branch and even errors made in the Branch were difficult to trace or
identify.
Access to and/or Editing of Transactions and Branch Accounts
(10) Whether the Defendant and/or Fujitsu have had the ability/facility to: (i) insert, inject.
edit or delete transaction data or data in branch accounts; (ii) implement fixes in Horizon
that had the potential to affect transaction data or data in branch accounts: or (iii) rebuild
branch transaction data:
a. atall;
b. without the knowledge of the Subpostmaster in question; and
c. without the consent of the Subpostmaster in question.
10.1 The Claimant refers to issue (7) above. The Defendant has admitted that it and/or
Fujitsu had the ability and facility to do what is set out in sub-paragraphs (10) (i) — (iii).
FUJ00159313
FUJ00159313
10.2 Investigations undertaken by the Claimants to date suggest that this was possible by
means of the use of:
(i) “global branches’ (with branch codes such as 999998 and 999999), which
would enable the input of transactions within Horizon as though it had come
from an actual Branch;
(ii) the Branch transaction correctional tool; and/or
(v) the Transaction Information Processing repair tool.
10.2 The Claimants have found no functionality by which use of these facilities would be
visible to, known by or communicated to Subpostmasters either prior to or after their
use. On this basis, the Claimants infer that these facilities could be used without the
knowledge or consent of Subpostmasters and will invite the Court so to find.
(11) Ifthey did, did the Horizon system have any permission controls upon the use of the
above facility, and did the system maintain a log of such actions and such permission
controls?
[GPOC §§21.3, 23, 25; Defence §§48(3), 50, 57]
11.1 The Claimants do not yet have this information. However, the system for the insertion
of corrective Transactions by the Reconciliation Service required authorisation from the
Defendant before such a change was made. The Claimants are unable to confirm
whether such actions were logged and/or whether such permissions were obtained in
practice and/or whether the requirement for authorisation was applied generally. It
would be normal for any such exceptional changes made in an IT system to be logged or
some form of audit trail kept of them. No such log or audit trail has yet been identified,
but the Claimants will continue to investigate.
(12) Ifthe Defendant and/or Fujitsu did have such ability, how often was that used, if at all?
12.1 The Claimants refer to issue (11) above. In the circumstances, the answer to this
question is not yet known to the Claimants, but the Claimants will continue to
investigate.
FUJ00159313
FUJ00159313
(13) To what extent did use of any such facility have the potential to affect the reliability of
Branches’ accounting positions?
[GPOC §§21.3, 23, 25; Defence §§48(3)(c), 57]
13.1 The very purpose of such facilities was to be able to affect Branch accounts and/or
transaction data upon which such accounts depend. Their use therefore had this
potential.
The Claimants are continuing their investigation.
Branch trading statements, making good and disputing shortfalls
(14) How (if at all) does the Horizon system and its functionality:
a.
enable Subpostmasters to compare the stock and cash in a branch against the
stock and cash indicated on Horizon?
enable or require Subpostmasters to decide how to deal with, dispute, accept
or make good an alleged discrepancy by (i) providing his or her own personal
funds or (ii) settling centrally?
record _and_reflect_the consequence of raising a dispute on _an_alleged
discrepancy, on Horizon Branch account data and, in particular:
i. does raising a dispute with the Helpline cause a block to be placed on
the value of an alleged shortfall; and
ii, is that recorded on the Horizon system as a debt due to Post Office?
enable Subpostmasters to produce (i) Cash Account before 2005 and _(ii)
Branch Trading Statement after 2005?
enable or require Subpostmasters to continue to trade if they did not complete
a Branch Trading Statement; and, if so, on what basis _and with what
consequences on the Horizon system?
[Defence §§42-46; Reply §§17.1-17.2, 21]
14.1
14.2
14.3
FUJ00159313
FUJ00159313
When a Subpostmaster performs a Daily Cash Declaration, this would show any
discrepancy between cash held by the branch and the amount of cash which Horizon
stated should be in the branch. When a Subpostmaster undertakes a mandatory Monthly
Trading Period Rollover, this requires all discrepancies (including any discrepancies
which had been moved into a suspense account in the intervening period) to be resolved.
A Branch Trading statement is produced at the conclusion of this process
Prior to 2005, branches were required to undertake a weekly balance and produced a
Horizon generated ‘Cash Account’. Discrepancies identified were placed in the
Branch’s cash account (in the “suspense section”). The discrepancy would either result
in the issue of an error notice or remedial action by the Subpostmaster to balance the
account (eg. by adding money into the Branch).
After 2005, when a discrepancy was identified, a Transaction Correction was sent to the
relevant Branch through the Horizon system, and appeared to Subpostmasters as if it
was generated by the system. The range of options available to a Subpostmaster for
processing the Transaction Correction in Horizon was: (1) Write off to P&L; (2) Assign
to Nominee; (3) Seek Evidence; (4) Stock Write Off; (5) Cancel; (6) Accept Now. The
Horizon system therefore did not acknowledge the possibility of error arising other than
by the fault of the Subpostmaster and there was no ability to challenge a Transaction
Correction using the Horizon system itself. If the option to ‘Seek Evidence’ was
selected, additional evidence would be provided only once and entirely at the discretion
of the Defendant. A new Transaction Correction would be then issued, but to which
there would
be no option for the Subpostmaster to “Seek Evidence” again, even if the evidence sent
by the Defendant had not provided the information sought by the Subpostmaster. The
Claimants understand, from an internal email sent by Mark Dinsdale to Nigel Allen dated
24 June 2010 (at 17:37 hours), that it was not the Defendant’s practice to supply ARQ
data to Subpostmasters in response either to the “Seek Evidence” option or at all
(although there may have been very occasional exceptions to the Defendant’s practice in
this regard).
14.4
14.6
14.7
FUJ00159313
FUJ00159313
Transaction corrections were communicated to Subpostmasters via Horizon and were
presented to Subpostmasters as being part of the Horizon system. It has transpired,
during the course of the present proceedings, that Transaction Corrections were
generated by the manual entry of data by employees or agents of the Defendant.
Discrepancies (shortages and surpluses) were, with authorisation from the Defendant,
placed in the Branch "suspense section" of their cash account. This discrepancy was
held until enquiries into the discrepancy were concluded (whether or not the cause of the
discrepancy was identified) and was then removed by the issuing of an error notice (now
known as a Transaction Correction) by the Defendant and / or by the Subpostmaster
putting the money into the branch (or settling centrally) to cover the loss or removing
the value of the gain from the branch to balance the account.
After 2005, a discrepancy had to be resolved prior to the required date for the Monthly
Trading Period Rollover (unless varied) as Subpostmasters are not permitted to continue
trading until the Branch Trading Statement process is complete.
A loss is recorded as a debt to the Defendant in the event that the discrepancy is not
resolved in favour of the Subpostmaster.
Transaction Corrections
(15)
15.1
How did Horizon process and/or record Transaction Corrections?
[Defence §§12, 39-40, 45-46; Reply §21]
The Claimants refer to issue (14) above. Whilst it is now understood that Transaction
Corrections were generated by manual entries, the Claimants are not aware of any error
repellency measures which were in place or adopted in order to safeguard the accuracy
of this process, whether as to errors in the data from which the manual entries were
made or in the course of making the manual entries themselves.
Transaction Corrections, if successfully processed, would be the subject of a
confirmatory message. The total volume of Transaction Corrections processed during a
month would appear on the Branch Trading Statement.
15
FUJ00159313
FUJ00159313
15.3 In summary, the Horizon system made it difficult for Subpostmasters to challenge
discrepancies the cause of which was unknown to them.
Dated 17 August 2018
Served by Freeths LLP, Floor 3 West One, 100 Wellington Street, Leeds, LS1 4LT
Appendix 1
a. Banks (LINK A&L, CAPO) for online authorisation of banking transactions (DCS
for debit and credit card authorisations) and transaction data used for reconciliation
b. Online Clients (e-pay, Streamline, DVLA) for online authorisation of transactions
and
(for e-pay and Streamline) data used for reconciliation
c SAP ADS - A Post Office system that handles cash and Foreign Currency logistics.
Data includes cash on hand statements from cach branch, planned orders, replenishment
deliveries and delivery/collection data
d. HR SAP - A SAP system that handles remuneration to the branch franchises and
‘multiples’ such as Tesco.
e. POL MIS - An Oracle based system to provide MI reporting to Post Office.
f. First Rate - Provides bureau rate information. It is also passed all bureau transactions to
allow First Rate to undertake MI.
g. Siemens Metering - Provides Rates and Customer data for Quantum gas pre-payment
card.
h Camelot — Provides National Lottery services.