POL00190856
POL00190856
Confidential
V18a - Interim Report into alleged problems with the Horizon system
1. Introduction and Scope
1.1. Following discussions with Post Office Limited ('POL') Senior Management in June and July
2012 and with the Rt Hon James Arbuthnot MP and Alan Bates and Kay Linnell representing
the Justice for Subpostmasters Alliance (‘the JFSA’), Second Sight was appointed to carry out
a Review into alleged problems with POL's 'Horizon' System.
1.2. The remit of the Investigation/Inquiry was later defined as:
"to consider and to advise on whether there are any systemic issues and/or concerns
with the "Horizon" system, including training and support processes, giving evidence
and reasons for the conclusions reached".
1.3. It was also agreed that Second Sight's report would:
“report on the remit and if necessary will contain recommendations and/or
alternative recommendations to Post Office Limited relating to the issues and
concerns investigated during the Inquiry. The report and recommendations are to
be the expert and reasoned opinion of Second Sight in the light of the evidence seen
during the Inquiry.".
1.4. It became necessary to ensure that references to “the Horizon System" were understood and
agreed by all stakeholders. Was Second Sight to look only for defects in the software code
of Horizon? Or, was it to take a broader view and also examine:
a) the surrounding Operational Processes, both at branch level and in POL's central
processing centres;
b) the interfaces between the Horizon system and other systems that are the
responsibility of organisations other than POL such as Camelot, the Bank of Ireland,
the Co-Op, various Energy Companies and the 'LINK' system for processing Credit
and Debit Card payments and withdrawals; [Scope creep UNLESS raised ina
particular Spot Review] [Comms to other partners?]
Pagel
POL00190856
POL00190856
c) the power supply and telecommunications equipment that connects every Horizon
terminal to POL's centralised data centres;
d
the training available to Sub-Postmasters (‘SPMRs’) and their staff and whether it
was commensurate with the demands of the day-to-day job at the counter;
e) the actions need to ‘balance’ at the end of each Trading Period (‘TP’) and the
investigation work needed in dealing with errors and Transaction Corrections ('TCs');
f) the level of support available to SPMRs and their staff from POL's Helpdesk system;
g) the effectiveness of POL's audit and investigative processes, both in assisting SPMRs
who called for help in determining the underlying root causes of mysterious [E]
shortfalls and in providing evidence for other action by POL such as in Civil and
Criminal Proceedings.
1.5. In answering the question as to whether Second Sight was to only examine the narrowly-
defined Horizon software, or the far more broadly-defined Horizon system, POL's own
definition of ‘Horizon’ provided much of the answer.
1.6.In May 2011, POL's Head of Information Compliance defined "Horizon" as follows:
"I can advise that the name Horizon relates to the entire application. This
encompasses the software, both bespoke and software packages, the computer
hardware and communications equipment installed in branch and the central data
centres. It includes the software used to control and monitor the systems. In
addition, I can advise you that testing and training systems are also referred to as
Horizon".
sn't POLs Head of but Royal Mail - and why u
e this definition when we had agreed a ToR.
1.7. This POL definition does not include the above-mentioned "audit and investigative
processes", but it quickly became clear that POL's audit and investigation methods have had
a profound impact on the SPMRs involved in almost all of the cases examined. [When did
“audit” come within scope? Has this come from a specific Spot Review or have they
investigated this?]
1.8. Second Sight's Investigation has consequently addressed matters well beyond the narrow
definition of the core software component of Horizon in order to ensure that we have
adequately address the totality of the concerns raised by SPMRs.
1.9. Before describing the approach adopted to this Investigation, it is necessary to put the scale
of the Investigation in context.
Page 2
POL00190856
POL00190856
1.10. Second Sight has been asked to investigate 47 cases submitted to either the JFSA or
to the office of the Rt Hon James Arbuthnot MP. All of these submissions are highly critical
of POL's Horizon system and in many cases, the way that POL has dealt with the matter
reported.
1.11, The entire Horizon system involves approximately 68,000 users [Enough context? 47
cases out of 68,000] and processes over 6 million transactions every day. The entire
population of over 11,800 branches was notified about the proposed investigation by
Second Sight and this resulted in a only a small number of additional cases [how many?]
being reported. Whilst in no way minimizing the potential importance of the cases
reported, this level of response suggests that the vast majority of SPMRs and branches are
at least reasonably happy [E] with the Horizon system.
How many additional branches when they make I
. Approach adopted
2.1.Second Sight has examined cases submitted from two sources. The first selection of cases
were those submitted by SPMRs, with the endorsement of their constituency MP, through
the office of the Rt Hon James Arbuthnot MP. There were 29 such cases.
2.2. The second source of cases was through the JFSA. These cases were submitted in
accordance with an Agreement dated December 2012 between POL, Second Sight and the
JFSA (see Appendix 1). That Agreement set a cut-off date of 28th February 2013 for the
submission of suitable cases to the JFSA or directly to Second Sight.
2.3.In the event, over 60 SPMRs contacted the JFSA and 18 cases were considered by the JFSA to
be suitable for submission to Second Sight. These 18 JFSA-sourced cases were generally
simpler, more recent and better documented than the cases submitted via MPs.
2.4.In dealing with each case, Second Sight first requested copies of all the documents in POL's
Case File. The initial plan was to interview each SPMR after all the POL-sourced
documentation had been examined. This has proved to be much more difficult than was
expected. Delays in producing case documentation to Second Sight have added materially
to the cost [FACT - to Post Office] of the investigation and to the time taken to complete it.
The main problem here seems to be that POL does not maintain one central file for each
case. Rather, documents have to be gathered from multiple internal sources.
2.5.Where MP sponsored cases have been subject to either Civil Recovery or Criminal
Prosecution, POL's centralised Legal Department was able to supply many documents.
However, we found that a significant number of cases had not progressed this far and that
documentation was held in many locations within POL, including the National Business
Support Centre (‘the NBSC'), the Helpdesk, the Branch Support Team, the Security Team,
the Former Agent Accounting Department, and Legal Services.
How many cases - what do they mean by significant.
Page 3
POL00190856
POL00190856
2.6. In several instances, POL's seven-year Document Retention Policy [whilst industry practice]
meant that little or no documentation was available for Second Sight to examine. The same
retention policy applies to the underlying Horizon computer data. Relying on the SPMR to
supply copies of POL-produced documents [in the [4] Spot Reviews], as well as their own,
was both time-consuming and unsatisfactory [E]. In a number of cases we were provided
with POL created documents by SPMRs, where POL had been unable to supply the same
document. Document completeness and integrity [in respect of the 4 Spot Reviews] was
compromised as a result.
There is a context around retention as 7 years tends t
could be a lot less,
industry stan
2.7. After examining all of the available documents and in some cases [Why not all cases?] the
Horizon computer data relating to each case, Second Sight has been making contact with
each SPMR in order to obtain, through telephone calls and face-to-face interviews, the
SPMR’s version of events. Second Sight then summarised the SPMR's assertions into one or
more 'Spot Reviews’. To date, 29 Spot Reviews have been created by Second Sight and
other Spot Review reports are planned. Nine Spot Reviews have been sent to POL and a
formal response received. Twenty Spot Reviews are currently ‘work in progress’.
3. The concept of a ‘Spot Review’
3.1. It became clear at an early stage in the investigation that it would not be efficient or cost
effective for Second Sight to examine all of the issues raised by SPMRs or covered in POL’s
Case Files.
3.2. Accordingly, and with the consent and approval of both the JFSA and individual SPMRs,
Second Sight conducted a ‘fast track’ review of the available information in each case and
identified the key issues that were relevant to the remit of the Investigation. Each key issue
was then dealt with as a ‘Spot Review’. A case with multiple issues would give rise to
multiple Spot Reviews, each of which would be dealt with on an individual basis.
3.3. It was agreed by POL, Second Sight, the JFSA and the Rt Hon James Arbuthnot MP that any
report issued by Second Sight would maintain anonymity with regard to the identity of
individual SPMR cases. Accordingly, this Report does not reveal the identity of any of the
cases being considered. In all instances where POL was asked to respond to specific issues,
the SPMR's identity was revealed to POL, but only after the SPMR's permission had been
obtained.
3.4. This approach to Spot Reviews was intended to be a self-contained, easy to understand
procedure, free from unexplained acronyms and backed up by SPMR supplied evidential
material. Each Spot Review was then submitted to POL for a formal response. The POL
response was then discussed with both the SPMR and the JFSA and an attempt made to
Page 4
POL00190856
POL00190856
reach agreement and closure between POL and the SPMR by asking the SPMR to agree with
the POL version, as to the issues dealt with in each Spot Review.
3.5. Regrettably [E], in all instances so far, no such agreement and closure has been achieved. In
the face of assertions, by both the SPMR and by POL, supported in many cases by only
partial or conflicting evidence, Second Sight has attempted to find out what really
happened. [In most [how many?] of the Spot Reviews investigated so far, we have been
able to find new information not previously known to POL or disclosed to the SPMR.]
{UNDER REVIEW]
How can SS make
‘not completed the reviews?
3.6. This Interim Report covers 4 Spot Reviews where we have been able to reach a preliminary
conclusion or at least make substantial progress in matters being reviewed.
3.7. As Spot Reviews were prepared, discussed and responded to by POL, Second Sight was able
to see a number of ‘thematic issues’ that were of concern to many SPMRs. These issues,
some of which are described in Section 7 of this Interim Report, and which will be addressed
in more detail in the Final Report, are those that have featured in many of the 47 cases
reported to us.
4. Involvement of the JFSA:
4.1. At the request of the MPs representing their SPMR constituents and with agreement from
POL, Second Sight has worked closely with Mr Alan Bates of the JFSA and with the JFSA's
appointed Forensic Accountant Kay Linnell. This developed into a sound working
relationship and Second Sight wishes to put on record its thanks to both Mr Bates and Ms
Linnell for their help and professional conduct throughout the investigation.
5. Spot Reviews and Responses from POL:
5.1. This Interim Report deals with just 4 of the 29 Spot Reviews so far prepared by Second Sight.
These 4 Spot Reviews deal with events that are typical of the matters reported to Second
Sight by many SPMRs. These 4 Spot Reviews also relate to matters that appeared, both at
the time they were issued to POL and when the selection was made for inclusion in this
Interim Report, to be particularly relevant to the remit of the Investigation. [These were to
be for JA the “Worst cases”]
Page S
POL00190856
POL00190856
5.2. Second Sight has urged [E] POL to deliver Spot Review responses that would prove as easy to
understand as the Spot Reviews themselves; that addressed the spirit, as well as the letter,
of the SPMRs' complaints; and that were backed up by evidence.
5.3.Whilst the Spot Review responses received from POL can be seen to be thorough, they are
long and highly technical documents that are likely to prove hard for most SPMRs to
understand. [Also, in some cases [which ones?], they present counter-assertions, based on
Standard Operating Procedures and Controls, rather than tangible evidence of what actually
happened.] [POL believe they have always provided tangible evidence] [Accordingly, it has
been necessary to summarise and simplify the responses received. — Rewriting the evidence
and not then shared it with POL]
5.4. [Our experience shows that even apparently robust controls sometimes fail to work, or can
be circumvented by a determined and skilful person.] [E and unsubstantiated] Second Sight
is therefore seeking further evidence in support of POL's responses to some of the issues
covered by the Spot Reviews dealt with in this Interim Report.
5.5.A consequence of this is, that whereas we had expected that some form of closure would be
reached between POL and the SPMR associated with each Spot Review, this has so far not
been achieved.
5.6. It is of course hard for POL to prove the negative (i.e. that controls have not been
circumvented) but it is only fair to say that POL now finds itself in the same situation that
has faced all of the SPMRs who have submitted cases. They too, were unable to prove that
the mysterious [E] shortages or transactions that they reported to POL, and in respect of
which they sought POL's help, were not the result of their own (or their employees’) errors
or criminal activity. [In every case we have looked at, there have been shortcomings in the
evidence and investigative help provided to SPMRs by POL.] [False Hope PPI risk]
The whole paragraph is totally unprofessional. fo comment on this and 5.7
5.7. Unfortunately, so far, POL has only acknowledged minor failings in the implementation of its
procedures and processes, or in other relevant areas. It has agreed in principle to a number
of process improvements relating to the matters under investigation by Second Sight, and
some of these have been implemented already. Unfortunately, many of the SPMRs we have
dealt with remain aggrieved and dissatisfied with what they see as POL's overly technical
and apparently unsympathetic response. [Despite describing the relationship between POL
and SPMRs as a ‘partnership’, it is clearly not a partnership of equals.] [E]
Page 6
POL00190856
POLO0190856
6. Did defects in Horizon cause some of the losses for which SPMRs or their
staff were blamed?
6.1. [Even on the 4 Spot Reviews] There is still much work to be done [on the cases Second Sight
has been asked to investigate]. We have concluded in one of the four Spot Reviews covered
by this Interim Report (Spot Review SRO1) that although the Horizon system operated as
designed, [the lack of timely, [accurate] and [complete] information presented to a SPMR
was a significant factor in the SPMR failing to follow the correct procedure.] [Substantiate]
6.2. In that incident, shortcomings in the branch's primary and fall-back telecommunications
equipment [meant the process ] led to a poor counter-level experience both for the SPMR
and his customer.
6.3. We also note in Spot Review SR22, that POL made a very substantial change, a few days
after the SPMR was suspended, to the underlying technical processes relating to the events
reported. [It is possible, that if this change had been implemented earlier, many of the
problems would not have occurred.] [aka “If you had found a solution before a problem
occurred then no problem would ever exist” = not a logical statement]
I
Page 7
POL00190856
POLO0190856
I aera
6.4. In the course of our extensive discussions with POL over the last 12 months, POL has
disclosed to Second Sight that, in 2011 and 2012, it had discovered "defects" in Horizon
online that had impacted 77 branches.
6.5. The first defect, referred to as the "Receipts and Payments Mismatch Problem", impacted 63
branches. It was discovered in September 2010 as a result of Fujitsu's monitoring of system
events (although there were subsequent calls from branches). The aggregate of the
discrepancies arising from this system defect was £9,029, the largest shortfall being £777
and the largest surplus £7,044. [It is not yet clear whether the SPMRs with the surpluses
profited from them, or whether those showing shortages had to make them good.] [Simon
Baker to check]
6.6. The second defect, referred to as the "Local Suspense Account Problem", affected 14
branches, and generated discrepancies totalling £4,486, including a shortfall of £9,800 at
one branch and a surplus of £3,200 at another (the remaining 12 branches were all
impacted by amounts of less than £161).
6.7. POL was unaware of this second defect until, when a year after its first occurrence in
December 2011, it re-occurred and an unexplained shortfall was reported by an SPMR.
{POL to clarify — standard procedure is to wait a month to see if issue re-occurs: this issue
turned out to be an annual event only.]
6.8. POL's initial investigations in 2012 failed to reveal the system defect and, because the cause
could not be identified, the amount was written off. Fujitsu looked into the matter early in
2013 and discovered, and then corrected, the system defect.
Page 8
POL00190856
POLO0190856
6.9. It seems however, that the shortfalls (and surpluses) that occurred at the first occurrence (in
2011) resulted in the SPMRs being asked to make good incorrect amounts.
6.10. [There may be other, currently unknown, system defects that come to light as
Second Sight’s investigation continues.] [E]
6.11. POL has informed us that it has disclosed in Witness Statements to English Courts,
information about subsequently-corrected defects (‘bugs') in the Horizon software.
Page 9
POL00190856
POL00190856
7. Thematic or Systemic Issues
7.1. It has become clear that whereas the Horizon system appears to achieve its intended
purpose almost all of the time and operates smoothly for most SPMRs and their staff, some
combinations of events can trigger situations where problems occur.
7.2. The following issues have been reported to us by multiple SPMRs [how many?] as being of
particular concern about the Horizon system:
a) A multi-product system that is far more complex and demanding than, for example,
that found in a typical high street bank;
b) Multiple transactional interfaces (hand-offs') to systems outside of Horizon such as
Lottery Scratch Card and Bank of Ireland ATMs that cause repeated and possibly
large shortfalls that take undue amounts of time to investigate and resolve; [It’s
our business model]
c) Unreliable hardware leading to printer failures, screen misalignment (pressing one
icon sometimes results in the system selecting an incorrect icon) and failed
communications links;
d) The complexity of end of Trading Period ('TP') processes and the lack of a ‘suspense
account’ option which would allow disputed transactions to be dealt with in a
neutral manner;
e) Inexperienced trainers and gaps in training coverage;
f) The lack of some form of on-site Supervision and Quality Control similar to that
made available to staff employed in POL Crown Offices;
g) The receipt of centrally input, overnight ‘corrections’ and other changes allegedly
not input by the SPMR or their staff;
h) Inadequate Helpdesk support, with responses that are often ‘script-based’ and
sometimes cause further or, greater problems;
i) POLinvestigation and audit teams that have an asset-recovery or prosecution bias
and fail to seek the ‘root cause’ of reported problems;
j) Acontract between SPMRs and POL that transfers almost all of the commercial risk
to the SPMRs, but with decreasing support being provided. In its risk reward
decision making, POL benefits from any savings, while SPMRs may suffer increased
risk.
7.3.We have read all of the examples of problems reported to us by the SPMRs we have
contacted. We can’t help concluding that had POL investigated more of the "mysterious [E]
shortages" and problems reported to it, with the thoroughness that it has investigated
Page 10
POL00190856
POL00190856
those reported to it by Second Sight, POL would have been in a much better position to
resolve the matters raised, but also would have benefited from process improvements.
7.4. It may be that a significant limitation in the way that POL responds to matters reported to it
are the terms of reference for the POL Investigations Division. The standard contract
between POL and SPMRs states:
"The Investigation Division does NOT enquire into matters where crime is not
suspected.".
7.5. This appears to suggest that POL does not provide any investigation support to SPMRs,
except where criminality is suspected. The cases we have examined show that POL does
sometimes provide limited investigative support to SPMRs reporting problems, but clearly,
POL’s ability to do this is constrained.
7.6. It is also unfortunate, in our view, that when POL does investigate cases, there is often a
focus on ‘asset recovery solutions’ without first establishing the underlying root cause of the
problem. This is also an example of a missed opportunity to be in a much better position to
resolve problems and to benefit from process improvements.
7.7. [Another issue raised by approximately 45% of the SPMRs we have had contact with, is the
allegation that the only time they were provided with the full contract between POL and
SPMRs, was when POL commenced litigation or recovery actions against them. However ,
approximately 20% of SPMRs did report that they received a copy of the full contract at
about the time they were appointed.] [E]
The following 4 sections now cover the 4 Spot Reviews where we have been able to reach
preliminary conclusions.
Page 11