POL00120479
POL00120479
rage 101
Mike Granville
From: Rod Ismay
Sent: 29 July 2010 19:53
To: Mark Burley; lan Trundell; Dave Pardoe; Mandy Talbot; Rob G Wilson; Keith Woollard; Lynn
Hobbs; Sue Lowther; Michele Graves; Sue Huggins; Mike Granville
Subject: FW: Horizon Challenges - draft report
Attachments: Horizon integrity 290710.doc
Dear all - Latest version incorporating, as best I can manage, all the input you have made. Thanks for
your time and comments in the last couple of days.
This is a complex area and I would value any further comments you have, but realistically those
comments would have to be by lunch tomorrow as I expect to have to submit this to Dave tomorrow. I
shall agree that in a meeting tomorrow morning with Mike Moores.
Regardless of how this document is finalised, there are a number of improvement points which we will
need to work on together and I would propose that we convene a group during next week to kick that off.
The priority should probably be to provide any input considered appropriate for closing down the issues
that cause Channel 4 to consider this a news item. Also to ensure we are prepared for the next court
cases.
Thanks, Rod
From: Rod Ismay
Sent: 29 July 2010 19:42
To: Mike Moores
Subject: Horizon Challenges - draft report
Mike — Val has booked a call for us at 9.30 tomorrow.
One of my fillings has fallen out this evening in the process of finalising the report and so I may have to
call you en route to another session at the dentist. Apologies for this.
This paper is longer than proposed and I have some obvious appendices and pages which I would
remove for the version for Dave. But I include them for our discussion tomorrow as I think you need to be
aware of the broad points arising from them.
The whole document is currently 17 pages. I would propose at least the following reductions for
finalisation tomorrow.
Appendix 2 — 1 page - list of live cases which may include challenges about Horizon
Appendix 3 — 3 pages — detailed commentary from Fujitsu
Appendix 4 — 1 page — additional detail on the screen freezes. A short summary is in the main body
Section 4 - 2 pages —I will try to halve this to one page
That gets it to 11 pages.
Other sections which are relevant to the issue, but could be overkill for Dave are
Section 3 - 1 page — other IT issues which branches complain about, but are not in the legal cases. I
include these for completeness of conversations with you and Mike
Section 5-1 page ~ branch accounting issues and stats on transaction corrections and most frequent
product problems
Appendix 1 — 1 page — improvement areas. I need to test and take the key ones forward with colleagues
in IT, Network etc. But they may not be necessary for answering Dave's question
I hope the 8 page residue would then be OK for Dave. It is still longer than we would like but this is a
complex area and I think the residue is relevant.
Look forward to speaking tomorrow.
30/07/2010
POL00120479
POLO00120479
Page 2 of 2
lam also going to share this version with colleagues who have contributed as there are final validations I
need to do with them on some content.
Thanks, Rod
\ccounting - Finance - Post Office Ltd
ie field, S:
30/07/2010
CONFIDENTIAL
To From
Dave Smith, Managing Director Rod Ismay, Head of Product & Branch Accounting
Mike Moores, Finance Director 29 July 2010
Mike Young, Chief Technical & Services Officer
Cc
Mark Burley, Head of Projects (IT) Andy McLean, Head of Service Delivery
Lynn Hobbs, GM Network Support Rob Wilson, Head of Criminal Law (RM Group)
John Scott, Head of Security Mandy Talbot, Principal Lawyer (Civil)
Lesley Sewell, Head of IT Keith Woollard, Head of Compliance
Horizon — Response to Challenges Regarding Systems Integrity
Post Office Ltd has, over the years, had to dismiss and prosecute a number of subpostmasters and
Crown staff, following financial losses in branches. A small number of these have made counter
claims that they were not guilty of the charges made but that the Horizon system was faulty.
Various lobby groups have been set up by former subpostmasters and these have at times
received national media coverage and in some cases been taken up by local MPs. Most recently,
Channel 4 has proposed a news article about this area.
This paper has been compiled as an objective, internal review of POL’s processes and controls
around branch accounting. It includes an overview of
—POL's control environment and POL’s response to accounting errors
IT systems — Horizon versus Horizon Online and resolution of known issues
© Third party perspectives — Case law, media and audit
© — Statistics on branch accounting issues, suspensions and prosecutions
Executive Summary
POL has extensive controls spanning systems, processes, training and support.
Horizon is robust, but like any system depends on the quality of entries by the users. Horizon
Online brings benefits to running costs and change management. It is not being done because of
any doubt about the integrity of Horizon.
Accounting errors do happen through user mistakes, but these can be explained and resolved case
by case, Systems issues have also arisen but again POL has been able to explain them and rectify
them. Whilst they have affected the availability and functionality of the system, they do not bring
the integrity of the system into question.
When POL takes a subpostmaster to court we have strong processes for the compilation of
evidence, compassionate factors are borne in mind and we have a high success rate.
We remain satisfied that the allegations about Horizon are a desperation move and this has been
reflected in court cases so far, but it is hard to stop people claiming that there “could” be issues.
This is particularly so in an environment of internet lobby groups, “no win no fee” lawyers and
media with an appetite to criticise public sector IT systems. It creates an unfair situation whereby
POL is considered guilty until proven innocent.
Nevertheless, POL has been successful to date in ensuring that the courts focus on the facts of
transaction logs and not on speculation about the “what ifs’.
There are several improvement opportunities for POL and these are set out in Appendix 1. They
do not undermine POL’s assertion regarding the integrity of Horizon, but they would tackle some
of the other noise which complainants feed on. They may also help when POL does take action.
Page 1 of 17
POL00120479
POLO00120479
POL00120479
POLO00120479
CONFIDENTIAL
1, POL’s Controls
POL has extensive controls within Horizon and in the communication of branch accounting records
to the central finance systems (primarily the POLFS system). POL also has clear processes for the
recruitment and training of staff and for access to support services such as Helplines.
1 (a) Systems
The independent IT consultancy Gartner has described the Horizon Online architecture as first rate
and the existing Horizon system has had positive press. Horizon and Horizon Online were both
designed with the principle that only authorised branch users can create or accept transactions in
the system. Both systems are also recognised for leading edge security.
Failures in systems and file transfer do happen, but POL has controls to detect these. The
frequency of file transfer failures has been unacceptably high recently andis a top priority for IT
The system depends on the quality and accuracy of data entry by users. POL has enabled controls
ata branch level for the prevention and detection of errors and these are supplemented in P&BA.
Hardware and software development, installation, enhancement and maintenance
® Change management — formal processes and decision points with authority levels
e Extensive testing including user acceptance tests
© Trained engineers on site and with requirements for proof of ID when working
Physical Access, Systems Users and log-ons
Physical security processes for branches to prevent unauthorised staff on site
Leaver / joiner controls for allowing and removing access to Horizon
All users require log on IDs and passwords
The person in charge in the branch can set the appropriate levels of access for different users
All transactions and events are tagged and traceable to the user
‘Systems is ISO 27001 compliant (governs system controls and security processes
Transaction automation — minimising manual entry and choice
e Use of cards, tokens, barcodes and system reference data to minimise manual data entry
® Cash centre automation and CCTV in the event of branches disputing cash rems
Business continuity and systems recovery
© Alerts for failed transactions in branch and centrally eg. for power cut or severed lines
Transaction recording, back up and proof against tampering
Sequential referencing of transactions and associated user IDs. Prevents gaps, duplication or
anonymous transactions
Transaction back ups — back up server updated as each transaction is completed
Audit file and “read only’ control — transaction records securely sealed
Events such as branch balancing are also securely recorded and are traceable
System drives double entry accounting and ensures it remains in balance
Interfaces to central finance systems
Automated interface checklists, batch schedules and exception reports
Batch controls (“BLE and “BIMS") between branch, middleware and finance systems
e Non polled reports if no data is received from a certain branch
e SAP system and middleware — daily checks for stuck data and for exceptions (iDocs)
e Reconciliation and alignment of different data feeds — eg. between data to clients and to POL
Page 2 of 17
CONFIDENTIAL
1 (b) Processes
Accounting and control in branch
POL00120479
POL00120479
“Accurate accounting is not difficult for well run branches.” The systems and support are in place
for branches to maintain comprehensive, accurate and timely accounts.
Branch accounting is included in induction training and support is available in service.
Horizon operates on a “double entry” accounting basis and maintains a balanced trading position.
Staff are then responsible for ensuring that they record transactions and methods of payment
promptly and accurately. They have the tools, and are encouraged, to do physical checks.
Mistakes do occur (intentionally and unintentionally) and this is where supervisory checks are
important in the branch and where detection processes from central teams would come into play.
Balancing routines
Daily cash declarations — an event where the branch does a partially “blind” declaration of
physical cash holdings and Horizon then flags up any difference compared to the system
© Cut off routines (daily and weekly) — for onward submission to P&BA / Clients / Suppliers
© Weekly Balance Period Rollovers — a full check and declaration of all branch assets and
comparison to trading records. Discrepancies can be ringfenced for resolution in suspense
Monthly Trading Period Rollover — as above but requires the discrepancies to be resolved either
by putting cash in or “settling centrally” to a debt ledger managed from P&BA
Branch Trading Statement, physical signature and retention — a monthly requirement
Other control reports in branch eg. Balance Snapshots, Transaction Logs, Suspense Reports,
Daily Sales, Reversals etc. These help identify errors and internal fraud
Transaction capture
Automated products — cards, tokens and barcodes force a transaction into the system
because the clerk has to scan or swipe something
Matching routines for stand alone kit — branches may forget or make mistakes in recording
transactions at kit such as the ATM, Lottery terminal or PayStation. P&BA get a feed from the
terminal to compare to Horizon and alert the branch to errors / omissions
© Balancing routines to detect omission of stock sales — physical stock checks spot these
© Cut off routines to detect omissions or misclassifications of vouchers encashed and methods.
of payment accepted — the control is the physical match against the system
e System reference data — eg. postage and travel insurance have embedded price tables
IT systems interventions
Incomplete transactions — systems could fail or lines could be disconnected during online
banking transactions. This could mean that customer money has changed hands without the
system being updated or vice versa. IT controls would detect these outages and raise recovery
alerts to the branch such that the branch can check and update the accounts if needed
Independent checks in branches
¢ Around [2,000] branch audits are conducted each year, driven by risk model
e All branches are receiving a high level cash count during migration to Horizon Online
@ Area managers may from time to time enquire about branch accounts during visits
Page 3 of 17
POL00120479
POLO00120479
CONFIDENTIAL
Accounting and control in P&BA.
Central controls over accounting are primarily in the Product & Branch Accounting team (P&BA)
but also include activity in cash centres and alerts from our method of payment processors.
P&BA has initiated a turnaround time commitment with Network for prompt notification of errors
to branches. P&BA has also worked closely with the NFSP ET on tone of voice and constructive
service as well as robust intervention in the accounting arena.
P&BA tracks its delivery and its accounts accuracy through interface alerts, reconciliation of data
streams, a focus on prompt resolution and various checks on individual account balances.
Complete, timely and accurate data feeds
© Joint working and alerts with IT teams in respect of expected interfaces, batch routines and
any incidents of data getting stuck in transmission
© Reference data and account mappings — these are secured in the system and there is
segregation of duties around changes
Data matching
© —P&BA data matching — comparison of stand alone branch terminal data against entries in
Horizon (for ATM, Camelot, PayStation, Sodexho Asylum Seekers, Post & Go)
—A&L paper matching — comparison by A&L of paper vouchers sent in by branches against
Horizon daily and weekly summaries
= Cash and bureau in transit matching — checks of what branches recorded as sent or received
compared to what cash centres recorded
@ =~ Cheques in transit — comparison of what branches reported as sent against what the cheque
processor physically receives
Matching routines — automated routines with defined parameters and rules
Error resolution
e = Thresholds and policy — P&BA operates a “maintained error” policy whereby differences below
defined values are written off and not investigated further on certain products
@ Client enquiries — issues of completeness or timeliness of data files may results in
amendments to data in POL systems or client systems
@ Customer enquiries — response to ad hoc queries received at P&BA and Customer Care
Branch ownership — branches are advised of issues and referred to Operations Manuals
@ judgmental write offs and authority levels — P&BA has defined authority levels for write offs
and requirements for rationale which would not set adverse precedents in branches
© Transaction corrections — “double entry” adjustments sent to branches for their acceptance.
There is no “back door” to force them into branch accounts. Visibility is key to ensure
ownership of the issue and prevent any risk of arguments about “back doors”
© Dispute process — this has been communicated to branches whereby transaction corrections
and discrepancies may be challenged by any branch
Account reconciliation, probity and audit
Individual accounts are subject to reconciliation with checks against expectations and
supporting evidence
Reconciliations are also subject to external audit
Internal audit reviews are conducted each year on “Critical Business Processes” and these have
covered some processes within P&BA
Page 4 of 17
CONFIDENTIAL
The interaction of P&BA, Network, Audit, Security and Legal
POL00120479
POL00120479
P&BA is in the front line of detecting suspicious activity. This fits well as an aspect of managing
and assuring client, branch and bank balances.
P&BA works closely with colleagues in other directorates and has clear escalation channels for
alerting about any systems or process issues, for proposing themes for fraud risk models and for
highlighting individual branches were a surprise visit by auditors or investigators may be needed.
P&BA also works closely with the civil and criminal law teams in the support of court actions.
Conformance and Fraud Risk monitoring and intervention
Accounts checks based on past experience of poor conformance and fraud routes
Prompt dialogue with branches enforcing the awareness that P&BA are watching
Joint working groups with Security etc — eg. Fraud Forum and Losses Group
Joint work with Cash Services on unusual cash trends and postings in branches
Cash delivery and collection optimisation to minimise the scope for theft in branch
Debt Recovery
e Visibility of issues in branch — anomalies in the accounts or in physical stocks are visible from
standard reports and basic stock counts. This creates a foundation for ownership
® Acceptance of debt by the branch — branches do actively choose options in Horizon
© Contractual responsibility for losses — this is clear in the subpostmaster contract
Enquiries and transaction corrections — branches are informed of issues to enable to them
prevent future recurrence of the same themes
© Statements and reminders — there are clear processes and milestones for reminders
© Repayment schedules are agreed with Network management and with sensitivity to financial
hardship of the payee
Suspension and follow on
e — Auditors have defined processes for their tests in branch and defined escalation routes
© Decisions not to suspend an agent require authority on a tiered basis within the Network
Legal action
. When discrepancies are identified, the case goes to Investigations to decide whether there is a
criminal case to be pursued
e Decisions on taking civil or criminal action involve qualified legal experts
[ Qualified decisions are made as to pursuing a specific debt under “theft” in the criminal court
or pursuing “false accounting” where the value awarded can involve the judge ]
Court proceeding are driven by the Legal Team
Decisions not to continue legal action, to resolve out of court or to write off are taken with
informed authority and consciously with a view to not setting a precedent
e Decisions do take a variety of circumstances into account including compassionate factors as
well financial cost / benefit. We consider POL errs more to the compassionate side than the
draconian side in its legal actions
e [Criminal prosecutions are not based on cost / benefit. As a “Prosecuting Authority” Royal
Mail has to ensure a consistently robust treatment of crime not based purely on value }
Standard approach to statements for court including what should have happened and what
actually happened
Page 5 of 17
CONFIDENTIAL
Recruitment and Training
POL00120479
POLO00120479
Agents are recruited based on business cases, credit checks and behavioural interviews. Agents are
then responsible for the recruitment of their own staff.
It is acknowledged that the induction controls and subsequent monitoring of agents in service
need strengthening and this is being taken forward as a legacy of the Network Efficiency
Programme.
Training is provided to Subpostmasters at induction. They are then responsible for onward
training of their own staff. Subsequent training or communications happen when significant new
products or processes are deployed.
Training and targetry have increasingly focussed on sales. Whilst regulatory compliance has
tightly had more airtime there may be a need to reinforce awareness of the key accounting
routines in branch and to facilitate more of a “self help” model with greater accountability.
Support
Support is available to branches through 3 main channels - the Network Business Support Centre
(NBS), operational instructions and Network line management.
Over time, however, branches have accumulated contact points in many parts of POL and in
suppliers and clients. This has therefore undermined the consistency of support. A project is under
way within Service Delivery to clarify “Single Point of Contact” and this will be a helpful step.
Network Business Support Centre (NBSC)
e Atiered helpline which can refer callers to more specialist support lines. Emergency support
such as for Security may be available 24/7 whereas accounting support is more likely to be 9-6
and with possible referral directly to P&BA
A specific referral from NBSC can be to the Horizon Help Line
The NBSC has a knowledgebase of standard guidance to advise branches with
Suppliers
e Branches have contact numbers for various suppliers. These may be for engineering matters
but can stray into accounting eg. with Wincor Nixdorf on ATMs
Reference material
Counter Operations Manual — paper based file with monthly updates for insertion
Horizon Online Help — the incoming replacement for the paper based operations manual
Operations Focus — weekly updates and guidance on transactions and services
Memoview — messages direct to the Horizon terminal
Various magazines which may include “training content’ eg. SubSpace
Page 6 of 17
POL00120479
POL00120479
CONFIDENTIAL
2. IT Systems
Horizon
Horizon was developed as an electronic point of sale to replace archaic paper based accounting
processes. It utilised hardware in branches which had strong security features and interfaces to a
central data centre with high security. Data was replicated between units in the branch and once it
reached the central data centres it was replicated between those.
This gave three strengths which are important in terms of the allegations being made:
1. Horizon infrastructure was robust from a security and access perspective
2. It was resilient in terms of being able to continue customer service and to hold data in a queue
in the event of incidents, and
3. It had strong back up and integrity features with data backed up in branch and centrally
This provided a strong audit trail. Further narrative from Fujitsu is included at Appendix 3.
Horizon v Horizon Online (HNG)
Horizon Online has been designed to give the Functional Equivalence of Horizon. It has been
driven to reduce costs and to improve change management by having the system on a central
data store rather than being managed by software drops to individual branches. Horizon Online is
made up of a combination of new and migrating components.
Counter
© No data stored locally, all data is stored centrally on Branch Database
® System in now completely online so it is not possible to transact without connectivity to
the data centre
© Solution uses existing equipment in Branch (base unit, keyboard, screen pin pad, scales
etc)
Router
© Improved backup connections using the mobile network
@ New Branch Router to allow discrete counter connection to data centre without reliance
on Gateway PC
Network
Secure VPN’s used to connect to data centres
Data Centre
© Infrastructure now located in state of the art datacentres in Belfast
© =Bladeframe infrastructure provides faster, more reliable infrastructure
© Full resilience - solution is n+1 (ie additional capacity to allow for failures)
.
Storage and backups are faster and more resilient spanning data across the two data
centres
Security
© No data held locally in Branch
e = Vulnerability scanning, reporting and patching systems. Intrusion Detection/Prevention
systems in key areas.
© Segmented architecture with firewalls and routers protecting traffic in and out of sensitive
domains including cardholder environment
Page 7 of 17
POL00120479
POLO00120479
CONFIDENTIAL
3. Known IT Issues and Their Non Applicability to the Allegations Made
(a) Screen Freeze / Recovery
Many Horizon Online branches have been frustrated by system outages. The root causes
have been clarified and resolved and this is evidenced in the accelerated rollout.
Branches were frustrated when they could not serve at all, but were also concerned about
accounts integrity when transactions were cut of in mid-flow.
However, transaction alerts were in place and revised operational instructions have now been
issued to enable branches to complete or to cancel the accounting entries dependent on
whether they physically completed the cash transaction with the customer. Further detail is
included at Appendix 3.
(b) Barcode Sticking
Asmall number of branches have experienced a situation where a customer transaction (eg. a
bill payment) sticks on the details of the preceding transaction. In some cases the branch has
spotted it immediately but in some cases it has only come to light when the customer
complains that they are being chased for an “unpaid” bill. Incidents go back to 2005.
Up to now it had been understood that it related to a version of scanners where POL did not
know which other branches had the same version. It was not therefore possible to isolate
other branches. It was also a rare event on the scanners themselves. It was not a systematic
failure with every transaction on the particular scanner.
Fujitsu have investigated this and now advised that it is not hardware related. The issue only
impacts Horizon as it relates to Escher code. It will therefore cease to be a problem with HNG.
This issue does not appear to have arisen in any of the legal cases in question and the
incidents have been resolved at all the branches where it has been noted. This is not
considered a systematic integrity issue for Horizon and it should be resolvable from the facts
of records in the Horizon transaction logs.
(© Non Polling
Small numbers of branches fail to poll due to issues such as telephone lines being dug up. The
system holds the data in a queue so this does not undermine integrity. It just affects the
timeliness of data, and this is catered for in decision making by P&BA. The issue goes away
with Horizon Online as the data store is online not in branch.
(d) File delivery failures
These do not affect branch records but have meant P&BA and clients may not have received a
days transactions. This is a priority fix for IT, but carries a PR risk.
(e) Horizon / POLFS differences
In 2005, P&BA moved onto a SAP system (POLFS). This was an exceedingly complex IT
migration and there were some issues in management of the cut off which meant P&BA was
out of synch with some branches in terms of opening balances for cash and bureau. This did
not affect the integrity of Horizon and has been catered for in error resolution with branches,
but it has affected service to some branches ie. Where decision making on cash supply was
based on wrong data centrally. Some issues have continued to come to light recently but this
is now under control. It is not relevant to the allegations.
Page 8 of 17
POL00120479
POL00120479
CONFIDENTIAL
4. Third Party Comment — Case Law, Media and Audit
Case Law
There have been cases, when taken to court by POL, where the defence has claimed that
the accounting syste Horizon was at fault and that there were incidents such as “ghost
transactions’ or “electrical supply issues’ which have corrupted the Horizon records.
With 2 notable exceptions, POL has been able to rebut these assertions by ensuring a
focus on the facts of the Horizon transaction logs and a request for the defence to be
specific about which transactions they consider to be “ghost” and why
Since 2005, which was the start of the existing case management system, there have
been 382 [ criminal law ] cases forwarded for legal advice of which 230 proceeded to
court. Of those 169 have been found guilty and 18 defendants cautioned. Of the
remaining 43, 1 was found not guilty but this was nothing to do with any Horizon
challenge and 42 cases were not carried forward. There is no suggestion in these 42 that
POL had any concerns itself about Horizon — rather, there was new evidence which came
to light which caused POL to withdraw.
There are three “landmark” cases which feature in the arena of challenges to Horizon.
1. Clevelleys — subpostmistress dismissed in 2001 soon after Horizon was introduced.
The defence produced a report which showed how Horizon “could” have caused an
error and POL did not have the audit transaction logs to refute the claim. POL settled
out of court for £187k, but subsequently improved the retention of audit transaction
logs. [ It would not now be possible for this situation to arise again? ]
2. Castleton —Lee Castleton claimed that Horizon was faulty and found other
subpostmasters to back him. However, POL presented the audit transaction log to his
solicitor who promptly advised Castleton there was no basis to his case. Castleton
sacked him, lost the case, was found liable for £300k and went bankrupt. The judge
decided there was “no flaw’ in the Horizon system and said “the logic of the system is
correct...the conclusion is inescapable that the Horizon system was working properly
in all material aspects’. This case appeared to have put a stop to allegations, however,
Castleton has continued to promote lobby groups, assisted others to find no win no
fee laywers and remains vocal in the press. Various publications have in turn quoted
him and although POL could argue that this is [ libellous ? ] we have made a strategic
decision across the group so far so quietly respond to each individual rather than
mount a pro-active PR campaign.
3. Alderley Edge (2010) — The subpostmaster, Mr Darlington, did plead guilty in
Macclesfield Magistrates Court, but POL had initially pursued him for theft and had to
reduce the charge to false accounting due to printer issues with the legibility of
branch trading reports. Press comment has focussed on certain comments by the
judge. The judge said he had issues with the proof of size of the loss and also said
“there are issues relating to the Post Office computer system which I do not feel able
to judge”. Critics of Horizon therefore focus on these comments rather than the fact
that Mr Darlington pleaded guilty.
In summary for POL, the record of prosecutions does support the assertion that the
subpostmasters have been guilty rather than that Horizon is faulty. However, this does
not stop speculation about the system. It is not possible to stop people saying “what if.”
Page 9 of 17
Media
POL00120479
POLO00120479
CONFIDENTIAL
Media and public discussion about Post Office systems has included:
1. TV—S4C's “Tara Naw’ programme in September 2009. This included similar
individuals to those involved in the current internet groups. Welsh MP David Jones
was proposing a Commons debate about Post Office IT systems on the back of it.
Comments were made by NFSP ET which were supportive of POL but these were not
included
TV—Channel 4 continue their investigations and had proposed a main news slot
The Grocer magazine — various articles linking back to Mr Castleton
Computer Weekly —links to articles including “MP seeks answers over Post Office IT
systems’, “Post Office slammed by MPs over Horizon system” and “Bankuptcy,
prosecution and disrupted livelihoods — Postmasters tell their story”. This in turn led
to discussions with MPs to rebut the suggestion that POL was delivering Horizon
Online as a cover up to tackle errors in the basic Horizon system
5. Accountancy Age — similar comments and suggestions from “IT experts” that POL
should get an independent review commissioned
6. Internet — “Justice For Subpostmasters”
PWN
To date, POL has ridden the wave of press comment but chosen not to pro-actively mount
our own media campaign.
Independent Review and Audit Angles
POL has actively considered the merits of an independent review. This has been purely
from the perspective that we believe in Horizon but that a review could help give others
the same confidence that we have.
Our decision between IT, Legal, P&BA, Security and Press Office has continued to be that
no matter what opinions we obtain, people will still ask “what if” and the defence will
always ask questions that require answers beyond the report.
Ernst & Young and Deloittes are both aware of the issue from the media and we have
discussed the pros and cons of reports with them. Both would propose significant caveats
and would have limits on their ability to stand in court, therefore we have not pursued this
further.
The external audit that E&Y perform does include tests of POL’s IT and finance control
environment but the audit scope and materiality mean that E&Y would not give a specific
opinion on the systems from this.
It is also important to be crystal clear about any review if one were commissioned — any
investigation would need to be disclosed in court. Although we would be doing the review
to comfort others, any perception that POL doubts its own systems would mean that all
criminal prosecutions would have to be stayed. It would also beg a question for the Court
of Appeal over past prosecutions and imprisonments.
Page 10 of 17
CONFIDENTIAL
5. Branch Accounting Issues — Most Common Issues and Work to Tackle Them
In 2009/10, P&BA issued 153,000 transaction corrections (TCs) to branches (down from 165,000
year on year). Several of these were in turn a consolidation for a month of a sequence of daily
accounting errors by a branch.
The most common themes for TCs were:
Type % of total Volume Proposed solution
Caused by branch
Camelot 31 48,283 “Ping” automated data feed
Cash rems from branches 11 19,994 Branch conformance reminders
A&L banking and Giro cheques 8 11,863 Branch conformance reminders
Cheques sent to EDS 6 8,585 Branch conformance reminders.
Aon 4 5,694 Pricing automation rolled out
Bureau 4 5,410 Branch conformance reminders
PayStation 3 4,423 “Ping” automated data feed
Other 18 27,331
85 131,583
Not caused by branch
ATM retracts 5 7,864 Branch awareness of retracts
Cash rems to branch 4 5,426 Cash centre conformance
Stock rems from stock centre 2 3,833 Stock centre conformance
Other 4 4,957
100 153,663
Branch errors included under and over statement of transactions and the omission of activities of
both a deposit and a withdrawal nature.
The average value of a TC to credit cash balances was £546 and the average for a credit was £527.
Many individual TCs were small, but some such as banking deposits where the transaction was
miskeyed with too many zero’s could be in the hundreds of thousands of pounds.
The decision to issue small value TCs is based on giving the branch the comprehensive view of all
their accounting issues and that it is more cost effective for P&BA to automate the whole run
rather than separate out low value items. The NFSP have been supportive of this approach.
The underlying volume of customer transactions is of a much higher order. POL serves around 21
million customers a week.
On a LEAN Sigma basis, the accuracy of recording some product transactions is in the six sigma
domain. However, others are poor but are a reflection of bad branch conformance.
P&BA is working with Marketing and Operations to design out the scope for error where possible,
but this is not a short journey.
TCs are not an indicator of systems integrity issues. They are a reflection of poor adherence to
defined processes. Nevertheless the underlying errors create noise in the system and an
environment where it is easier to make the broader allegations.
Page 11 of 17
POL00120479
POLO00120479
POL00120479
POLO00120479
CONFIDENTIAL
Appendix 1 — Improvement Areas
Branch awareness and ownership / accountability
PWN>
u
Branch training — to raise awareness of key Horizon reports and scope for self help
Absentee managers — to identify and address absentee subpostmasters and managers
Ownership and accountability in branch — to improve the standard of ownership
Password security —local commitment to not sharing passwords and to locking keyboards
when away from the counter
Prioritisation of conformance and accounting in the supervision and targetry of branches
Cash remittances — better conformance by branches in making up contents of rems
Single points of contact and speed of turnaround
1
2.
Single point of contact — to rationalise and communicate contact points for branches
Incident response — to improve response times and clarity of ownership
Readiness for court, focus on facts and strategy for dealing with public allegations
1,
w
No
Court cases — to ensure consistent use of Horizon transaction logs and explanations. To
include the resourcing point between criminal and civil prosecutions
Consider contract change with Fujitsu to increase access to “ARQ transaction logs’ and
whether POL uses its contractual allowance for the right priorities
[Learnings from Alderley Edge..]
Case management system — to give single view of records between P&BA, Network, Legal
and Security
Press comment — to consider whether case law gives a precedent for court orders or libel
action
Pro-active briefings — ensuring key stakeholders do understand the facts
Review strategy for response to allegations and speculation
Plain English instructions
1.
Processes and instructions need to be clearer and punchier
Systems
1
5.
Software code changes and proof of authorisation — to improve the sign off process and
documentary evidence of sign off. This has been a criticism in the 2009/10 external audit.
It did not apply to Horizon, but was a theme from reviews of Credence change control
Manual transactions — continued automation and greater use of user entry checks
Cash remittances — automation of Scottish centres to same level as England
Stock remittances and accounting — to improve tracking between branch and stores and
to make branch recording easier
“Sticking transactions” — to resolve the recent increase in apparent barcode reader issues
Process and control documentation — improving accountability and version control
Balance of firmness and compassion
1
2.
Benchmark POL suspension and prosecution rates against other retailers
Continue acknowledging hardship but could fairly charge interest on debt
Minimising opportunity — keeping cash down and closing enquiries quickly
Page 12 of 17
POL00120479
POL00120479
CONFIDENTIAL
Appendix 2 — Current Cases With Possible Challenges to Horizon
As of April there were 12 live cases where it was thought the defence team were challenging the
integrity of Horizon. Case notes are below. In several cases these do beg a larger question of
“where is the cash” rather than “what could be wrong in the data’. It will be important that POL and
Group marshall our resources to ensure a robust and consistent approach to responding to these
criminal cases as well as to the other population of civil cases.
Post Office Date Toss Ek Other pertinent information(i.e. magistrates comments or external audit done)
fw Uriale comune tet ii Caste umuiny ngures were Hl Ute region OF EHUUUU. re
then stated that due to a number of Horizon technical problems he had not been paying money]
out to customers and had therefore generated a build up of cash. Forensic Accountants used
Newsome 2005 to 2008 169 by Defence.
Ms Hodgson explained that she had seen a planned order on Horizon in October 2008 asking
for £25,000 to £30,000 to be returned. She stated that she did not have this amount to return}
0 started altering the figures at this time to agree with the Horizon figures. Throughout Ms
Carmichael insisted that she believed this to be an administration error and had not stolen anyI
IArrochar 2007 to 2009 32 of the money.
When asked why the office had been over £11,000 short on the day of the audit, again Ms
Hodgson stated that this was due to mistakes. She daimed that she didn’t always count the
cash & stock properly and instead relied on the Horizon data.
Kirkoswaid 2008 to 2009 18
‘When asked to explain how the shortage had occurred Mr Dawson stated that he had made an
error during the summer of 2009, consisting of incorrectly entering a cash deposit into HorizonI
of around £31,800 rather than £3,180. He then stated that when he balanced that night
Horizon showed that he was £40,000 to £45,000 short rather than the expected £28,000 to
£29,000. Mr Dawson claimed that the Horizon system was andent and was due for renewal in
January 2010. He also stated that he was aware of other offices that had experienced
‘electrical glitches causing unexplained losses. When asked to give examples of these Mr
Dawson stated that he would need to ask their permission before relaying any details. He has
litiochry 2009 16 not provided any further information, (electrical fauit data) despite requests.
Audit conducted, shortage identified. Majority of loss was in the cash, Sub-postmaster bellevesI
problem with ATM. Sub-postmaster also runs Conway Road SPSO (440614) which was audited
the following day, small shortage identified. Letter received from Solicitor stating suspect has
been ill, but can be interviewed the week commencing 1 March to 5 March 2010 . Allegation
that the Horizon data is not correct("ATM data haywire.") Unknown whether this defence will
JOld Colwyn Up to 2008 ‘44 be presented in court.
Interviewed SPMR in June 2009, admitted to false accounting, but denied theft. Solicitor now
alleging that the Horizon data is incorrect.
Rinkfield 2008 to 2009 25
‘Accused changed plea to not guilty. Fresh court process underway.
Lazy Hit 2007 to 2009 3
Pleaded guilty in Macclesfield magistrates court. Sentenced to 3 months imprisonment
‘suspended. Awaiting court transcripts in order to verify if the magistrate did make comments,
lAlderley Edge 2008 to 2009 45 as per press releases.
‘Accused suggesting Horizon data to blame, although no specific transactions identified. Trial
IWest Byfleet 2006 to 2009 75 date set 15 March 2010
Told auditors on the day of the audit that he would be short. Disputing the amount, (60K) and
‘opposed to auditors finding of 74K, and "technical faults.” Did not raise Horizon at all during
lwillesden 2008 to 2009 75 the PACE interview.
Porters Avenue 72 Offender instructed a forensic accountant to analyse the data/transactions
Matfield 41 No specific transactions given relating to the Horizon database
Page 13 of 17
POL00120479
POLO00120479
CONFIDENTIAL
Appendix 3 — Fujitsu Report on Horizon Data Integrity
© Copyright Fujitsu Services Ltd 2009 Ref: ARC/GEN/REP/0004
COMMERCIAL IN CONFIDENCE AND WITHOUT PREJUDICE Version: 1.0
Uncontrolled if Printed or Distributed Date: 02/10/2009
Document Title: Horizon Data Integrity
Document Reference: ARC/GEN/REP/0004
Document Type: Report (REP)
Release: N/A
Abstract: This document describes the measures that are built into Horizon to
ensure data integrity. Note that it only covers Horizon and not HNG-X (Horizon Online).
Document Status: Final Draft
Author & Dept: Gareth I Jenkins
1 Purpose
This document is submitted to Post Office for information purposes only and without prejudice. In
the event that Post Office requires information in support of a legal case Fujitsu will issue a formal
statement.
This document is a technical description of the measures that are built into Horizon to ensure data
integrity, including a description of several failure scenarios, and descriptions as to how those
measures apply in each case.
Note that this document only covers Horizon. It does not cover HNG-X (Horizon Online).
2 Horizon Data Integrity
The Horizon system is designed to store all data locally on the counter’s hard disk. Once the data
has been successfully stored there it is then replicated (copied) to the hard disks of any other
counters in the branch (and in the case of a single counter branch to the additional external
storage on the single counter). Data is also passed on from the gateway counter to the Horizon
data centre using similar mechanisms.
The replication process is designed such that should the data fail to be copied immediately (for
example due to a failure on the local IT network within the branch or another counter being
switched off or the branch being disconnected from the data centre), then further attempts are
made to replicate the data at regular intervals until it is finally copied successfully. Once the data
reaches the Data Centre a further copy is taken and added into the audit trail where it is available
for retrieval for up to 7 years. Data in the audit trail is “sealed” with a secure checksum that is held
separately to ensure that it has not been tampered with or corrupted.
Every record that is written to the transaction log has a unique incrementing sequence number.
This means it is possible to detect if any transitions records have been lost.
While a customer session is in progress, details of the transactions for that customer Session are
normally held in the computer's memory until the customer session (often known as the “stack’) is
settled. At that point all details of the transactions (including any methods of payment used) are
written to the local hard disk and replicated (as described above). It should be noted that double
entry bookkeeping is used when recording all financial transactions, ie for every sale of goods or
services, there is a corresponding entry to cover the method of payment that has been used. When
a “stack’ is secured it is written in such a way that either all the data is written to the local hard disk
or none of it is written. This concept of “atomic writes’ is also taken into account when data is
replicated to other systems (ie other counters, external storage or the data centre).
The data for a stack will have been successfully secured to the local hard disk before the screen is
updated indicating that a new customer session can be started. Note that although an attempt will
Page 14 of 17
CONFIDENTIAL
have been made to replicate the data to an external system at this time, there is no guarantee at
this point that such replication will have been successful. For example if there is a Network Failure
followed by a Terminal failure there is a slight risk that transactions in the intervening period could
be lost.
All data that is written includes a “checksum” value (known as a CRC) which is checked whenever
the data is read to ensure that it has not been corrupted. Any such corruptions detected on reading
will result in failures being recorded in the event logs which are held on the local hard disk for a few
days for immediate diagnosis and also immediately sent through to the data centre where they
are held for 7 years.
Any failures to write to a hard disk (after appropriate retries) will result in the counter failing and
needing to be restarted and so will be immediately visible to the user.
Whenever data is retrieved for audit enquiries a number of checks are carried out:
1. The audit files have not been tampered with (ie the Seals on the audit files are correct)
2. The individual transactions have their CRCs checked to ensure that they have not been
corrupted.
3. A check is made that no records are missing. Each record generated by a counter has an
incremental sequence number and a check is made that there are no gaps in the sequencing.
3 Scenarios
It should be noted that these scenarios are all to do with equipment failures and these will always
be visible to Fujitsu through the event logs which are retained.
3.1 A counter fails
When a counter fails, there are two possible scenarios:
OO lt can be successfully restarted
O0it cannot be successfully restarted, so needs to be physically replaced
In each case the Data Integrity considerations are different and so are described separately below.
Once the counter has been restarted (regardless of whether or not it has been replaced) recovery
may be carried out if recoverable transactions are detected on the counter. This is also discussed
below.
3.1.1 The Counter is Successfully Restarted
In this case all the data that had been secured prior to the failure is still present on the counter and
so is available for use. If the User is in any doubt as to whether a transaction had been completed
or not prior to the failure they can use the transaction logs to confirm one way or the other.
3.1.2 The Counter is Physically Replaced
In this case there is no data on the local hard disk of the replacement counter. However, since the
data should have been replicated to other counters in the branch (or in the case of a single counter
branch to the external storage — which should have been physically moved to the replacement
counter), then the data should be retrieved and copied to the new counter. If for some reason the
data were not available locally in the branch, then it will be copied back from the data centre. This
all happens automatically as part of the counter replacement procedure.
Note that the hard disks are encrypted so there is no danger of data protection issues once the old
counter has been removed (or if it is stolen).
When a counter is physically replaced, there is a possibility that not all data has been successfully
replicated to another system prior to the failure. In this scenario it is essential that the user
confirms what the last successful transaction on that counter was, again by using the transaction
logs.
Page 15 of 17
POL00120479
POL00120479
POL00120479
POLO00120479
CONFIDENTIAL
3.1.3 Transaction Recovery
Some classes of transaction generate recovery data as they go along, so as to ensure that in the
event of a failure between the transaction starting and the basket being secured, there is sufficient
information available to enable the transaction to be recovered. On Horizon there are two separate
mechanisms to cover different classes of transaction:
OO Banking Recovery
ODAP Recovery
Both these mechanisms are automatically invoked during Log On, should the system detect that.
there has been a possible failure. These are described below.
3.1.3.1 Banking Recovery
This covers credit card and debit card transactions and e-Top-Up transactions as well as online
banking transactions.
A check is carried out to see if any incomplete banking style transactions (i.e. network banking,
credit / debit card or e-Top-Up) exist in the transaction logs for that counter. An incomplete
transaction is one where an authorisation request has been sent to the financial institution, and
there is no corresponding completion message which is normally secured as part of settlement at
the end of the Customer session.
In most cases, recovery information stored in the transaction log can be used to ascertain the
outcome of the transaction being recovered and a suitable completion record is then recorded at
the time of recovery.
In some cases the user is prompted to confirm whether or not the transaction has completed
successfully and the response from that prompt is used to generate the completion record.
3.1.3.2 AP Recovery
In the case of Automated Payments (AP), the user is asked if they wish to carry out AP recovery
and they have the option of doing so immediately or leaving it until later.
If the user carries out recovery they will be asked about the last successful AP transaction (which
can be seen from the branch copies of the AP receipts that are printed) and the system will then
check to see if it has been completed in the system. If it has not been completed in the system,
then the system will use the AP Recovery data stored in the transaction logs to ensure that all
incomplete AP transactions on the counter up until the one specified by the user are completed at
recovery time. To assist with this process, each AP transaction has a unique, incrementing
sequence number which is printed on the receipt.
Fujitsu understand that these processes are defined in Post Office's Horizon User Guides.
3.2 A counter has a "Blue Screen of Death”
This is just a special case of a counter failure, so please see section 3.1 above.
3.3 There are package collisions on networks
The replication protocols used to copy details of transactions between counters and also between
the gateway counter and the data centre ensure that the data is copied successfully.
Should packets collide on the network (or should there be any other network issues such as the IT
communications link failing) then the replication protocols will ensure that the data is re-sent. Such
retries will continue until the data is finally successfully transmitted.
Page 16 of 17
POL00120479
POL00120479
CONFIDENTIAL
Appendix 4 - Screen Freeze / Recovery
The HNG-X architecture allows transactions to be built up on the counter in a basket.
Simplistically, the settlement of the basket is the point at which the clerk is informed that
the customer transactions have been committed to the Data Centre and thus the signal
to the clerk to complete the exchange of goods and money with the customer.
However, some customer transactions have a more immediate effect as they happen; for
instance they generate valuable artefacts or involve interactions with external systems
before the point of settlement. Examples of such customer transactions are:
© On-line Banking
* Settlement with Credit / Debit Card
¢ E-Top ups
e DVLA Transactions
e APOP Generic Online (including Postal Orders and Moneygram)
Should such customer transactions have been included in a basket that failed to complete
(eg due to system freeze, loss of connection) then some action is required to recover the
customer transaction and return the overall system into a consistent state. In order to
support this, recovery data is stored at the Data Centre regarding recoverable
transactions to assist with the recovery process.
Page 17 of 17
POL00120479
POLO00120479