POL00103263
POL00103263
Message _
From: Rodric Williams!
on behalf of — Rodric Williams;
Sent:
To: ;
Subject: PRIVATE & CONFIDENTIAL - Post Office Limited
Attachments: Thematic Issues - Mediation Cases V95.xIsx
Dear Amanda,
I refer to our recent emails and discussions, and respond on the various issues raised below:
MacDonald case
Zubes Patel is happy to discuss with the CCRC his experience at the Broughton branch, where he supplied temporary
postmaster services following the suspension of Mrs MacDonald’s contract with Post Office.
I will separately send an email introducing Mr Patel to Anona from your team. Copies of the documents requested from
Post Office Limited’s "Electronic Filing Cabinet" will also follow shortly.
Documents
1. attach the final copy of the thematic spreadsheet created by Second Sight as requested.
2. You also asked for a June 2014 report prepared by Mr Warmington on POL's investigations department and
prosecution process. After making internal enquiries, we were unable to locate such a document. You helpfully
provided further information from Mr Warmington about this report in your email to me sent on 24 October
2016 which we are now using to help us in our search. However, because the employees to whom Mr
Warmington’s report was apparently sent both left the business quite some time ago, we are having to search
our email archives. This will take time, and may not produce a positive hit without more precise information
around the time of sending.
It may therefore be quicker and easier for Mr Warmington to provide the report. Are you aware of any reason
why he would not be able to do this?
3. You asked whether we could locate the transaction logs for the Misra case. We requested the relevant data
from Fujitsu and have now received transaction data for the following periods: February 2006 — October 2006;
May 2007—July 2007; August 2007—January 2008. Please note that data relating to the following periods was no
longer available: June 2005-January 2006; and November 2006-February 2007. We will arrange for the available
data to be uploaded onto the data room and let you know when it is ready for your review.
Questions
1. The issue reported in Computer Weekly in November 2015 is indeed also referred to as the "Dalmellington
Error". This issue was fixed in January 2016. It affected remote ("outreach") branches and as the name suggests
was first identified at the Dalmellington branch.
I set out below an explanation of the issue, the problem identification, problem correction and wider context as
provided to Post Office by Fujitsu:
"The Dalmellington Post Mistress who runs a Core and Outreach branch remitted £8,000 out of the Core branch and then
attempted to remit the cash into the Outreach Branch. However when the cash was remitted into the Outreach Branch
the system repeated the inward remittance transaction 3 times, thus remitting in a total value of £32,000. This was
POL-0102846
POL00103263
POL00103263
£24,000 more than the actual cash being remitted and so resulted in the system at the Outreach Branch showing a deficit
of £24,000. All these transaction show up in the Branch Reports.
Problem Identification
There were actually 2 separate issues which if they occurred in combination lead to the scenario described above. These
were the pouch script itself and the Force Log Off, which if they occurred together caused the issue. The scenario was:
(i) Clerk starts to log on, entering username and password
(ii) Clerk steps through some of the “post login checks” (e.g. Cash Declaration)
(iii) Clerk abandons “post login checks” — walks away
(iv) Counter “times out” and eventually Force Logs Off the Clerk due to Inactivity
(a) An omission in the Counter code left this in the “incorrect state”
(b) The state in question tracks the progress through the use-case
(v) Clerk later logs on again — this time going all the way through to the Front Office menu
(vi) Clerk tries to perform a Remittance
(a) This Remittance use-case “inherits” the “incorrect state” from earlier.
(vii) At the end of the Remittance the transaction gets “stuck” due to the earlier bad state which prevents
Remittances from finishing properly.
Each repeat press of the enter key caused a new remittance to be recorded but each one appears in the branch reports
and transaction records.
The Dalmellington case was the first time this had been identified to Fujitsu as an issue (even though there had been
present in HNGX from day 1).
Problem Correction
This error was corrected at Counter release 12.88, CTR_APP_X1288_V646 following approval by POL by modifying the
logic for these transactions so that the state tracking the progress through the current PDL transaction is reset so that it
is not “inherited” by the next PDL use case. The milestones of solution implementation were:
° 4th January 2016 Model office testing.
° 7th January 2016 Pilot phase.
° WC 11th January 2016 Live deployment.
Prior to the fix, following the issue being raised, active monitoring was put in place. No incidents were seen.
Wider Context
As well as performing the fix Fujitsu ran a process of checking for duplicate pouch details from the transaction archive
and transaction logs. In total 112 instances were found. These were all individually investigated and had either at the
time been ‘corrected’ by the subpostmaster themselves or the Post Office support desks when a call from a sub
postmaster had raised the occurrence. All cases explained and closed.
Since the fix has been live no further incidents have been seen and the active monitoring for this has now been stopped."
I hope this explanation is sufficient for your needs.
POL-0102846
POL00103263
POL00103263
2. It seems that there may be some confusion over the terminology being given to the different issues frequently
cited about Horizon. The "Calendar Square/Falkirk issue" is not the same as the "Receipts and Payments
Mismatch Problem", so in order that we may use the same terminology going forward I summarise the more
frequently cited issues below:
CALENDAR SQUARE / FALKIRK
This defect, which was discovered in 2005 and fixed in March 2006, involved Horizon failing to recognise transfers
between different stock units (i.e. individual terminals/serving positions).
For context, sometimes it is necessary to transfer cash between different stock units in a branch. For example, one stock
unit may be running short on cash so the postmaster or branch staff may transfer some cash from stock unit A to stock
unit B. Alternatively, it may be that an amount of cash has been remitted into the branch entirely on one stock unit
creating a need to distribute that cash among the other stock units so they are all equipped to perform transactions.
The Falkirk anomaly came to the attention of Fujitsu when a postmaster in the Calendar Square branch in Falkirk
highlighted a receipts and payments mismatch when balancing one of their stock units. This meant that, when the
postmaster came to balance the branch’s stock units, while the total for the amount of receipts into the system and the
total for the amount of payments out of the system should have matched (owing to the double entry book-keeping
principle), in this case they did not match.
The problem was that information recorded on one terminal (the terminal from which cash was being sent) was not
being passed properly to the other terminal (the terminal receiving the cash). This meant that while cash was being
transferred from one terminal to another, the transfer was only visible on the terminal it was being transferred out of,
not on the terminal it was being transferred into. Effectively, the terminal receiving the cash could not “see” the
transfer.
The problem produced a visible trace of “events” in the event logs*, in particular an event called “time out waiting for
lock” which means that information which Horizon was trying to communicate to the terminal was effectively locked out
from the terminal. In the Calendar Square branch, this problem was visible on the event logs multiple times.
After the problem was identified by Fujitsu, it was solved by putting a software fix into the system. This fix was
distributed to the entire network — not just the affected branch — in March 2006. When the problem was diagnosed,
advice was passed to the Calendar Square branch as to what to do if the problem happened again. If the branch was to
simply re-start the terminal, the issue would correct itself the following day. The same guidance was made available on
the help desk for any other branches experiencing the issue.
This issue was the subject of expert evidence in the criminal prosecution of Seema Misra in 2010. The expert for the
prosecution, Gareth Jenkins, then of Fujitsu, explained that the Falkirk bug could not have caused the losses in Mrs
Misra’s branch because (a) as explained above, the manifestation of this bug was clearly visible in branch records and
there was no sign of its occurrence in Mrs Misra’s branch records; and (b) the bug had been resolved more than a year
before the relevant period in Mrs Misra’s branch.
The Falkirk bug was also raised as part of a defence in a civil action by Post Office against a former postmaster, Lee
Castleton in December 2006 /January 2007. The Court accepted the evidence from Fujitsu’s witness, Anne Chambers,
and found “no evidence” of the Falkirk bug in Mr Castleton’s branch.
(* The event logs reflect the “back office” procedures such as users logging on and off Horizon, cash declarations and
balancing, as opposed to “front office” transactional data.)
PAYMENTS MISMATCH
POL-0102846
POL00103263
POL00103263
The payments mismatch bug affected 62 branches (13 crown; 12 multiples; 37 postmasters). It related to the process of
moving discrepancies into the local suspense account and the majority of incidents occurred between August and
October 2010.
The identification of this bug came about through Horizon’s own in-built checks and balances which are designed to flag
up such issues.
When discrepancies come to light during the rolling over of a stock unit onto a new transaction period, the user is asked
if that discrepancy should be moved into the local suspense account. If the branch pressed the “cancel” icon at this
stage, the discrepancy was “zeroed” on Horizon.
The effect of this was that the back end branch account (Post Office’s central accounting system) showed the
discrepancy while Horizon, in the branch, did not. The branch may have thought they had balanced when they had not.
In the affected branches, this created a “receipts and payment mismatch” equal to the value of the lost discrepancies.
When the new Trading Period began, the opening figures for discrepancies in the new period was zero rather than the
actual value of the discrepancy.
The first remedial step was for Fujitsu to ascertain which branches were affected. The mismatch generated an error
code which allowed Fujitsu to identify the relevant braches. Fujitsu were then able to carry out analysis on each affected
branch to gather relevant information. For example, they needed to ascertain when the receipts/payments mismatch
occurred, the value of the lost discrepancy and whether it was a gain or a loss.
There were 17 postmasters who had a loss attributed to their branch. They were notified of this in March 2011 and,
where appropriate, they were reimbursed. Postmasters who made a gain through the anomaly were not asked to refund
this amount to Post Office.
SUSPENSE ACCOUNT BUG
The suspense account bug caused a small number of entries in the suspense accounts* of 14 branches (4 crown and 10
postmasters) in 2010 to be erroneously reproduced in those branches' suspense accounts for the same monthly trading
period in 2011 and 2012.
By way of context, if a postmaster declares on Horizon that there is a discrepancy between the amount of cash and/or
stock in the branch and the amount of cash and/or stock recorded on Horizon (say following an ad hoc cash/stock
count), the discrepancies can be removed from the branch's live Horizon records, so that the branch accounts will reflect
the cash and stock actually in the branch at that point. However, the loss or gain in cash and/or stock is stored as a
temporary accounting record in a separate part of Horizon called the "Discrepancy Account".
At the end of each trading period, the figures in the Discrepancy Account must be cleared before the branch can move
on to trade during the next trading period (called "rolling over"). To do this, postmasters transfer the net value of all
discrepancies recorded in Discrepancy Account during that trading period into a branch "Suspense Account". The
postmaster can then settle any shortfall or surplus in the Suspense Account by making good the discrepancy or settling.
After settling any shortfall or surplus, the Suspense Account resets to zero and the branch rolls over.
The suspense account bug was discovered when two postmasters who suffered discrepancies raised the matter with
Post Office in January 2012. This error caused postmasters to re-settle the incorrect entries in order to clear their
Suspense Accounts in 2011 and 2012 despite those entries already having been settled in 2010. In effect, some branches
accidentally benefited from the same gain three times and some branches suffered the same loss three times.
Post Office began an investigation and identified the bug as being the cause of the issue in January 2013. Post Office
suspended any attempts to recover known losses from affected postmasters whilst the issue was resolved.
POL-0102846
POL00103263
POL00103263
3. You asked for a description of the “fault log” maintained by Fujitsu, which I believe is known at Fujitsu as the
Known Error Log or “KEL”. I understand that “Peak incident reports” and “pinnacle” are different parts of
Fujitsu's service management processes and that they dovetail into the KEL. I would need to ask for more details
from Fujitsu to understand their interaction fully but, for the current purposes, I believe that the KELs are the
most important aspect.
The KEL is Fujitsu's log of known issues with Horizon. This captures the range of issues from very minor to more
major issues. There are thousands of entries on the KEL, most of them unconnected to branch accounting. We
have asked Fujitsu for some random examples of these entries, and will provide these to you in due course.
Fujitsu has confirmed there are no recorded issues in the KEL which directly affect the normal operation of the
Core Audit Log. The Core Audit Log is the technical process within Horizon that forms the master record of the
data entered on Horizon in branches and the transactions performed on terminals across the Post Office
network.
If you would like to look into this line of enquiry further, it would first be useful to understand the type of
information that you are looking for. Fujitsu may then be able to narrow the search to avoid having to trawl
randomly the voluminous entries in the KEL.
4. lam seeking responses to the enquiries you have made about the April 2008 transaction log from the
MacDonald case, and will forward these to you as soon as possible.
I hope this is helpful. Please let me know if you require anything further.
Kind regards, Rodric
« Rodric Williams
Solicitor, Corporate Services
Post Office Ltd
POL-0102846