WITNO0170100
WITNO0170100
Witness Name: Anne Olivia Chambers
Statement No: WITNO017_01
Exhibits: None
Dated: 15 November 2022
POST OFFICE HORIZON IT INQUIRY
FIRST WITNESS STATEMENT OF ANNE OLIVIA CHAMBERS.
I, Anne Olivia Chambers, will say as follows
1. This witness statement is made to assist the Post Office Horizon IT Inquiry with
the matters set out in the Rule 9 request provided to me dated 6 October 2022.
It relates to matters that occurred between 2000 and 2016, at which point I
retired. As I have indicated throughout the witness statement, I have tried to
assist to the best of my ability despite the difficulties of recalling all details of
events across that period at this remove and given my limited access to
contemporaneous documentation.
2. I graduated from University College of Wales, Aberystwyth in 1978 with a 2:1
degree in Statistics with Pure Mathematics, and started working for Dataskil, the
software house of International Computers Limited (ICL). I coded and supported
various packages, mostly statistical or database related. From 1986 I was home-
based, working part time for ICL as a Software Diagnostician, investigating
errors reported by users in various software products and producing code fixes.
Page 1 of 63
WITNO0170100
WITNO0170100
During the 1990s I increased my working hours but remained home-based,
continuing support work but also checking various external systems for possible
Year 2000 issues.
From around 1997 I also did some coding and support in respect of part of the
new Pathway system for the Post Office concerning the transfer and validation
of files from the DSS / Benefits Agency. This work was subcontracted to my
department. Most of this part of the system was abandoned in 1999.
I have been asked to explain my role in relation to the Horizon IT Project. In
October 2000 I joined the System Support Centre, also referred to as EDSC or
3" line support (“SSC”) as a System Support Specialist, working full-time in the
office, providing 3% line support (technical, software not hardware, but not
responsible for code fixes) for the Post Office Horizon systems which were just
being installed at branches across the country.
Initially it was intended that my role would be in supporting the parts of the
system which extracted data from all the branch transactions and passed it to
3" parties, for example to TV licensing, but I gravitated towards support of the
branch counter systems, whilst learning much about the whole system and the
many changes that were introduced over the years.
In around 2002 ICL, which had the original contract with the Post Office for
Horizon, was taken over and rebranded by Fujitsu.
I remained a System Support Specialist but was often treated as a technical
lead, both within SSC and the Fujitsu Post Office team generally.
Page 2 of 63
WITNO0170100
WITNO0170100
8. From around 2012, in addition to providing general support, I was responsible
for producing reports checking that the system had sufficient capacity for the
transactions being done.
9. I retired in January 2016 having reduced my working hours over the preceding
couple of years.
KEL system
10. I have been asked to explain the purpose and operation of the KEL database.
KEL stands for Known Error Log. The KEL database was a set of documents
describing the symptoms and solutions for problems and bugs. It was intended
for use by Fujitsu support at all levels. The Horizon support desk (“HSD”), which
received calls from branches, the System Maintenance Centre (“SMC”), which
monitored Events, and the Management support unit (“MSU”), which monitored
Reconciliation Reports, on receiving a report of a problem, were each expected
to check for relevant KELs for a solution before passing service tickets to SSC.
11. KELs usually included a hot link to the associated PEAK(s) (see paragraph 18
below) and often to other related KELs. Sometimes a master KEL would be
created with guidance on distinguishing between several problems with similar
symptoms.
12. When a service ticket was passed to SSC, a person in the role of Pre-Scanner
(an SSC Administrator, replaced later in my time there by members of SSC staff
on a rota) would check whether HSD or SMC had found an appropriate KEL,
and possibly add to or amend the KEL before allocating the service ticket to a
particular SSC diagnostician.
Page 3 of 63
13.
14.
15.
16.
17.
18.
WITNO0170100
WITNO0170100
Within SSC, we were all meant to able to handle any type of service ticket, but
in practice most of us specialised in particular aspects of the system and would
be familiar with many of the known errors of a particular category. The vast
majority of KELs were written by SSC staff, and we would edit KELs to improve
the instructions as to what evidence to look for and what action to take. We were
users of the KEL system too, but primarily we were the authors.
Some KELs were written by the Development team, to record the existence of
a few known errors which were not going to be fixed before the revised counter
software system, HNG-X, went live.
The KEL system was redeveloped by SSC at some point, but I cannot remember
when. The purpose of the system remained the same.
I am asked whether I consider that the KEL system was adequate for its
purpose. Overall, I think the KEL system worked well although there were some
problems. For example, many KELs documented similar symptoms, and
service tickets could be passed to SSC with the wrong KEL quoted.
Further, SSC and Development did not always know how many branches had
reported a particular problem because once it was being investigated and had
a KEL raised, generally further service tickets were not meant to be sent to SSC
for the same problem. This did not apply to MSU-raised reconciliation calls
which were always looked at, and many KELs specified that any new
occurrences should be passed over to us.
I am asked to describe the process a person would follow to search the KEL
database to determine if there was a pre-existing KEL for a problem. I cannot
remember exactly what the ‘Search KEL’ interface looked like, but it was a web-
Page 4 of 63
WITNO0170100
WITNO0170100
based service and all the KEL text was searchable. I usually had a good idea of
what I was looking for and so could find it.
PinICL and PEAK systems
19. I am asked to explain how the PinICL and PEAK systems worked. The service
ticket system used initially for Horizon was PinICL, which had been used for
other ICL projects. At some point it was rewritten and renamed as PEAK. I
cannot remember much about the former, or the differences between the two
systems, but the user interface was similar, so I shall just refer to PEAK here.
20. I am asked whether I considered that the system was adequate to manage
active service tickets. I did not consider that the system impeded our work but
I am asked specifically as to whether I think there were potential changes which
would have improved the systems. The problem I would identify was that the
system assumed only one person or team needed to be actively working on the
PEAK ticket at any one time whereas this was not always the case. If the
Postmaster who raised the service ticket telephoned for an update, HSD might
put that request on the Powerhelp call system which they operated and that
might not get copied to the PEAK, depending on which option was chosen,
which might not be noticed by the developer or tester who owned the call at that
moment, who would not expect to give an update to a postmaster.
21. — Similarly, if a problem had a financial impact, then it was necessary to inform
Post Office (often via the MSU team), whilst separately, it was necessary to look
for the root cause(s), which might involve passing the problem to the
Development team for consideration of a code fix, and also to check whether
Page 5 of 63
22.
23.
24.
WITNO0170100
WITNO0170100
any other branches had been or might be affected by the same problem. That
might result in two or more people trying to work on the same PEAK.
It was possible to clone PEAKs so then one could go to MSU and a second to
Development, which is something I used to do to try to manage things better
and that practice could have been more widespread. Looking back I can see
that basing the whole process around a single PEAK (arising from an incident)
per problem was not good and it is likely there is a methodology in existence
somewhere for handling this type of situation better.
I am asked how service tickets were assigned. HSD and SMC would raise
service tickets on their own system which was called Powerhelp (Later
Powerhelp was superseded by Triole for Service (TfS). For simplicity I will refer
to both as Powerhelp in this statement). When they decided to transfer the ticket
to SSC, they pressed a button which automatically created a PEAK call and
copied in the information from the Powerhelp ticket. This interface was called
the OTI. The Powerhelp ticket remained open until the underlying PEAK was
closed.
The PEAK call would be on the SSC stack (for historical reasons called EDSC),
and the SSC Pre-Scanner would review the information provided and then either
return it to the originator if there was insufficient information on it or allocate it to
an SSC technician — so it moved onto their individual stack. The Pre-Scanner
knew, as we all did, who specialised in which areas of the system. It was not
unusual for someone to tell the pre-scanner that they would like a particular call
to be allocated to them.
Page 6 of 63
25.
26.
27.
28.
29.
WITNO0170100
WITNO0170100
The PEAK would be updated as the investigation progressed. It might be routed
to one of the Development teams; if they did a code fix it would then go to
Release Management for scheduling, then possibly through the test teams and
back through Release Management. The PEAK would always come back
through SSC to be closed (which would copy the response back onto the
Powerhelp call). This could be many months after the service ticket was first
logged.
SSC, developers and testers could also raise PEAKs directly.
I am asked how service tickets were prioritised. These were set by the original
service ticket logger and were normally prioritised from A to C, with A being the
most urgent. We could alter the original priority level in the course of our work
as the situation developed.
‘A’ priority was normally for a system-wide problem where many branches had
lost functionality, for example problems with debit card payments, or where a
branch was completely unable to trade, or for some reconciliation issues with a
likely financial impact. Anything affecting or likely to affect a large number of
branches would probably be looked at more quickly than something affecting a
single branch, especially if it was still able to trade normally.
Any major problem affecting the entire estate would probably be taken on by
several more senior people, regardless of whether it was allocated to them or
what else was going on. Something that looked potentially serious to members
of the SSC, even if not given a high priority by the original service ticket logger,
would be investigated quickly.
Page 7 of 63
WITNO0170100
WITNO0170100
30. I am asked about the ways in which information was obtained, stored and
accessed. Evidence came from various sources, depending on the type of
problem and whether it was during the period of operation of Legacy Horizon or
HNG-x. In relation to counter issues for Legacy Horizon, the primary sources of
evidence would be:
e Riposte messagestore for the branch, extracted from the data centre
messagestore;
e Application event log from the counter;
¢ psstandard.log from the counter;
and for HNG-X:
e Data for the branch from various tables in the Branch Support Database;
e UNIX application events from the counter;
¢ Log files from the BAL.
31. For HSD raised service tickets, i.e. those instigated by a call from a sub-
postmaster, sometimes we would call the branch to get a better first-hand
description of the problem or some clarification.
32. Over the years various people in SSC developed tools to help extract and
analyse evidence from these and other sources. Some of these were available
to HSD too, so they could check for known problems more effectively. This
evidence could be attached to the PEAK and would be if it was passed on to
Development. Details of SSC investigation and analysis would also be added,
either as a Response section or sometimes as an attached spreadsheet or Word
doc. Otherwise, evidence was kept in our area within the server.
Page 8 of 63
WITNO0170100
WITNO0170100
33. A service ticket was designated as closed on PEAK when an SSC investigation
was complete. This could be due to a number of factors:
e System working as designed; user error or misunderstanding;
e Something wrong but not software, for example a printer problem;
e Something wrong but already being progressed on another service
ticket;
e Investigation could find no indication of any problem with the system;
e Software or reference data bug identified and fixed;
e Problem acknowledged but not fixed (root cause not found or found but
too risky to fix, and very limited impact and/or frequency).
34. 1am asked whether I was aware of any contractual requirements or contractual
penalties that that Fujitsu was subject to in respect of its processing of service
tickets. I believe there were various requirements and penalties but I cannot
remember the details. It was not something that drove the way we worked,
although, subject to what I have said above, ‘A’ priority calls did take priority.
System Support Centre (SSC
35. As set out above, I joined SSC in October 2000 and left in January 2016, working
reduced hours for a year or so prior to that.
36. I am asked questions as to how the SSC operated in practice. When I joined
SSC in late 2000, Horizon had been installed at some but not all branches. There
were about 25 diagnosticians and an administrator, all reporting to the manager
Mik Peach. Different people specialised in different parts of the system (for
Page 9 of 63
37.
38.
39.
40.
4.
WITNO0170100
WITNO0170100
example, on counters, or on several different central databases), though we were
all expected to handle any type of incident if necessary.
There were always five or six of us, including me, who would be most likely to
handle any service tickets to do with counter balancing.
Everybody in SSC could see all the PEAKs allocated to team members and we
often cooperated or shared knowledge, for example ifa problem appeared to be
a side-effect of something we had already been working on or another instance
of a known error.
Through the 2000s, some staff left and fewer arrived, but it was a relatively stable
team with a lot of knowledge of the system and of each other’s strengths and
weaknesses. In the year or so before the introduction of HNG-X in 2010 there
were a few extra joiners. Mik Peach left around this time, and I think was
replaced by Tony Little for a few months, before Steve Parker, who had been
Mik’s unofficial deputy, took over. Not long after that, four or so team leaders
were appointed. My team leader was Mark Wright.
After a very busy period of about six to seven months when HNG-X was
introduced in 2010, the workload decreased somewhat. Over the next few years
some additional, non-Post Office work was taken on, SSC took responsibility for
the production of capacity reports for the system and various other tasks. The FJ
reference data team and MSU team were merged into SSC. By the time I left in
2016 the team was probably around 12 to15 people.
SSC, also referred to as EDSC or 3" line support, was the only team with full
access to live systems. The team was responsible for investigating system
problems that could not be resolved by HSD/SMC/MSU, maintaining KELs for
Page 10 of 63
42.
WITNO0170100
WITNO0170100
use by all the support teams, gathering evidence sets to be passed to 4" line
support for code-level investigation where there was sufficient evidence of a
software problem. From around 2007 a real-time monitoring system was
developed (by SSC) to alert us to system-wide problems, for example if there
were large numbers of failing debit card transactions; this system was tweaked
and extended over the years. We also sometimes handled ad hoc requests for
data extracts from system architects and Post Office. SSC provided 24-hour,
seven day a week support but I was never on the out-of-hours rota.
SSC interacted directly with all the following teams except the Network Business
Support Centre:
(i) NBSC: This was the Post Office’s business support team and was outside
Fujitsu. I understood NBSC was responsible for business-related queries
from postmasters, including discrepancies, and questions as to how to use
the system. NBSC had no direct interface with SSC, and the two teams had
no access to each other’s call logging systems.
(ii) HSD (Horizon Support Desk) (IMT (Incident Management Team) was a
subset of HSD): This was a Fujitsu-run helpdesk for branch-raised system
problems, hardware problems, and any suspected software problems. For
the latter, postmasters would often have talked to NBSC first then been told
to telephone HSD if the issue could not be resolved, for example because
there was an inexplicable loss. Late on in the period, I believe, HSD was
renamed HSH and run offshore by ATOS. All incidents were logged by HSD
on their system which, certainly initially I recall, was called the Powerhelp
system. HSD were meant to get a clear description of the problem then
Page 11 of 63
WITNO0170100
WITNO0170100
search the KELs and, if they found a match, follow the advice there. If the
KEL instructed that the matter be referred to SSC, or if HSD could not find
an appropriate KEL, the Powerhelp call would be routed to SSC via the OTI,
creating a PEAK.
(iii) SMC (System Maintenance Centre): This team was responsible for
monitoring system events and messages generated by the live system (from
numerous backend servers as well as counters), to pick up on anything that
was not running as expected. This service was later outsourced to a team
in India, but I cannot remember when that happened. Incidents were logged
on Powerhelp, checked against KEL database and if appropriate, passed to
SSC through OT! automatically raising a PEAK.
(iv) I MSU (Management Support Unit): MSU monitored Reconciliation Reports
generated by the system each day to check the integrity of data, for example
the TPSC250 report (there were a suite of such reports) and banking/debit
card reports which showed any anomalies in such transactions (cross-
checked with data from the banks). MSU raised PEAKs so SSC could
investigate cause and impact of every entry on the Reconciliation Reports.
My understanding is that MSU informed a team in Post Office of any such
errors which potentially had a financial impact on a branch, via a BIM report
and that it was then up to Post Office to notify the branch and make any
necessary correction.
(v) 4" line support: handled by the Development teams. These support teams
varied in size and number as the system evolved. Most of the Legacy
Horizon Counter software was maintained by the EPOSS (electronic point
Page 12 of 63
WITNO0170100
WITNO0170100
of sale system) team. Much of HNG-X was developed by GDC in India who
then provided 4" line support too, with some technical expertise still retained
in the UK.
(vi) Fujitsu Problem Management Team - liaised with Post Office when
problems affecting a major part of the system occurred.
(vii) I Team based in Belfast looking after the central servers and databases. For
Legacy Horizon these were at data centres in Wigan and Bootle. I cannot
remember where the data centre for HNG-X was physically located.
(viii) Fujitsu and Post Office reference data teams. Much of the counter
information (for example, price of products, which branches could sell what,
what buttons appeared where and on which screen, and much more) was
controlled by reference data rather than code, so sometimes a bug would
be identified which was caused by incorrect reference data.
(ix) I MSC -1am asked about MSC but have no clear recollection of there being
a team of that name. There was a process called MSC which involved the
application of changes to the live system. SSC (or another team) would
raise a request, for example, to get a server restarted. This had to be
authorised variously, sometimes including authorisation directly from Post
Office, before being actioned.
I am asked as to the minimum level of skills and expertise required of SSC staff.
I was never involved in recruitment so cannot answer this question. In terms of
the required training, I can say the following. In 2000, I and some other new
joiners attended the same counter training course as was provided for
postmasters. I think all SSC members would have done this course if they joined
Page 13 of 63
44,
45.
WITNO0170100
WITNO0170100
in the first few years, and the course notes were given to all who joined during
the life of Legacy Horizon. Some of us also attended a Riposte training course.
Each new joiner would be assigned a mentor and they would work together for
several weeks. When new functionality was introduced to the system, the
Development team or designers would give training sessions which we were all
expected to attend.
I am asked how members of the SSC were briefed or updated on potential
problems, bugs, errors or defects within the Horizon IT System and on how to
rectify those. We did this by talking to each other, perhaps also via email to
make sure the whole team was alerted to a new problem and knew which KEL
to use if there were likely to be many service tickets for the same bug.
I am asked whether I considered that the SSC had sufficient resources and / or
expertise to meet its purposes and / or provide an adequate service reliably.
Specifically, I am asked this with regard to the level of demand. My response is
that generally SSC had sufficient resources to handle the volume of calls
received and to provide cover 24 hours a day seven days a week, albeit there
were only a couple of people on call for the central systems overnight and at
weekends. We were stretched when HNG-X was being piloted in early to mid
2010. Subsequently, at times we were over-resourced, though the need for 24/7
support meant the team could not be reduced too much. As a result, the team
took on some additional work streams. Most of the team were flexible; if there
was an urgent ongoing problem then people would work at it for as long as
required.
Page 14 of 63
46.
47.
48.
49.
WITNO0170100
WITNO0170100
I am asked whether I considered that the Horizon Support Desk, NBSC and / or
SMC referred an appropriate proportion of issues to the SSC for investigation.
NBSC did not refer issues directly to SSC, though they may occasionally have
made a request to HSD that SSC investigate. It is impossible for me to comment
as to the proportion of matters that were never brought to the attention of SSC.
My sense was that generally matters were correctly routed.
I am asked whether, if I consider there was sufficient resources or expertise,
and if not, what effect that had on the service provided by the SSC. As I have
stated, I do not consider it can be said that we were under-resourced or lacked
appropriate expertise.
I am asked whether I ever felt under pressure to avoid finding bugs, errors and
defects in the Horizon IT software. I never did; that was my job.
I am asked about FUJ00086462 which is a short email chain which concerns
the performance (i.e. processing speed) of the Horizon system. I cannot
remember exactly when it happened, but at some point, weekly Cash
Accounting Periods were replaced by four-weekly Trading Periods, optionally
split into weekly Balance Periods. This was supposed to have the benefit that
branches need only produce Balance Reports every four weeks, but the design
and implementation resulted in a process that could take hours to complete on
Horizon, in addition to the time taken to do the associated paperwork. Many
sub-postmasters complained about this, particularly busy branches with several
counters operating a single stock unit. As I recall, at the point in time at which
these emails were sent, the Development team had been looking at this issue
for some time, initially saying the system was working as designed and as
Page 15 of 63
50.
51.
52.
WITNO0170100
WITNO0170100
agreed with Post Office. I had gathered data from several branches to show the
time being taken and was aware of the frustration being caused, so when invited
by Graham Welsh (FJ service management) to comment to John Burton
(counter software development manager, I think) I made my point forcefully. 1
am asked whether there was guidance provided on performance issues and I
do not recall there being any.
I am asked to set out any further recollections of my time working on the SSC
which I think are relevant to the Inquiry, including details of any positive or
negative aspects about the service provided by the SSC. In general, I would
say that we provided a good service, keeping a very complex system up and
running. We were proud of what we did, and skilled at finding every bit of
evidence we could squeeze out of the system to help our investigations.
However, when it came to branch problems, we could only work on the basis of
what was recorded on the system and any extra information which the
postmaster or clerk could provide. We could not see what had physically
occurred in the branch; whether for example, there was an inputting error that
was not evident from the pattern of Riposte messages. Sometimes that meant
we could not provide an answer to explain a discrepancy because none of the
forms of evidence available to us gave any indication and there were no
inconsistencies in the figures recorded and calculated by Horizon. In those
circumstances we would have to record that we could not find any evidence of
a systems error.
lam asked to explain the process Fujitsu would follow when a bug was identified
or suspected. SSC would make sure a KEL was in place to document the
Page 16 of 63
53.
54.
WITNO0170100
WITNO0170100
symptoms and the action that was required if there were any further
occurrences. SSC might also try to reproduce the problem on a test system. 4"
line / Development would look for the root cause with a view to producing a fix.
SSC and other teams might work out what if anything needed to be done to sort
out the consequences of the bug, for example, on branch accounts or the
backend systems. The Problem Management team might be involved and might
liaise with Post Office. If the bug was in newly released software, the software
might be regressed.
I am asked whether Fujitsu took pro-active steps to identify bugs and / or
discrepancies in branch accounts caused by the same. The automatic cross-
checks made and reported on the TPS, APS and banking reconciliation reports
highlighted inconsistencies which might indicate a bug. These were always
investigated, and MSU informed Post Office via a BIM report if the bug had
affected the branch accounts, or accounts with other third parties. If a bug was
found to be affecting branch accounts which had not caused a reconciliation
report entry, we would do our best to identify all branches affected, as we did for
Bug 3 (see PC0223870). However, I cannot say that this was done consistently
for all bugs ever found, especially in the early days of the project.
I am asked whether there were any written or unwritten practices, policies or
procedures to restrict what information about a bug or potential bug could or
would be shared with others, either for limited periods or indefinitely. I was not
aware of any such. If! spoke to a postmaster about a problem and I identified
it had been caused by a system error, I would say so.
Page 17 of 63
55.
56.
57.
58.
WITNO0170100
WITNO0170100
1am asked how a fix would be implemented if it involved changes to the Horizon
system’s code. The code change would be made by the relevant Development
team, tested by them, tested by LST (Live System Test) and then released to
the live estate. Anything very complex or having little impact would probably be
kept for the next scheduled release (e.g. S70, T10).
I am asked in what circumstances a work-around would be offered for an
identified bug, rather than a change to the code. Changing code to fix a bug
was potentially risky; in fixing one thing you might unwittingly break something
else, despite extensive testing. So occasionally a change would not be made
even if a code bug had been identified. In other cases, there might be insufficient
evidence to find the root cause, especially if it was not possible to reproduce the
problem. Thirdly, there could be a delay of several months between identification
of a bug and a fix being applied. In such circumstances a work-around would
be offered if available, else SSC or other teams would have to continue to deal
with the consequences of further instances as they arose.
1am asked to look at FUJ00080215 which is a document entitled Reconciliation
and Incident Management Joint Working Document. My name appears on the
internal distribution list within this document because I had helped its author,
Penny Thomas, and her predecessors understand the flow of data through the
system, in particular for Network Banking and the DRS, which checked for each
banking / card payment transaction to see that the branch outcome matched the
bank outcome and reported any transactions where this was not the case.
My attention is also drawn to FUJ00080213 which describes the HNG-X
Reconciliation Report Suite. There were similar documents for Legacy Horizon.
Page 18 of 63
59.
60.
61.
WITNO0170100
WITNO0170100
I am asked to explain my involvement with the task of reconciliation and/or with
the Reconciliation Service. I had no formal involvement with the Reconciliation
Service, except as a member of SSC investigating PEAKs raised for report
entries. Informally I gave support and advice to various MSU / Reconciliation
Service staff members over the years.
I am asked to assess the efficacy of Reconciliation at identifying discrepancies
and/or bug, errors or defects. There is an explanation of Reconciliation in
section 1.1 at page 9 of FUJ00080215. A Reconciliation Report entry was
generated whenever there was an inconsistency between data recorded in the
branch accounts and data recorded elsewhere in the system, i.e. in:
e the central messagestore;
¢ customer's bank account;
e automated payment client such as BT;
¢ data sent to Post Office’s financial databases;
¢ adifference in Receipts and Payments; or
e branch transactions not netting to zero.
I would say the Reconciliation process was an effective means of identifying
bugs, errors and defects. These inconsistencies were all investigated, and
sometimes were caused by a bug, and would sometimes have caused a
discrepancy at the branch accounts. The Reconciliation process has nothing to
do with the investigation of discrepancies between the physical stock and cash
declared in branch and the system-calculated cash/stock value.
I am asked to speak to the BIM system: when was it introduced; how did it
operate and how effective it was. I had no visibility of the BIM system. At some
Page 19 of 63
WITNO0170100
WITNO0170100
point I became aware that MSU usually just cut-and-pasted SSC’s PEAK
response into the BIM report, so I and others tried to make sure we wrote our
responses in a way that would be clear to Post Office staff.
Bugs etc identified in Appendix 2 of Horizon Issues
62. I have been asked to comment on most of the items identified in Appendix 2. I
do so with the following caveats. I retired in January 2016. I have no access to
any contemporaneous information other than what has been provided to me by
the Inquiry, and a few PEAKs and KELs disclosed by the police. My explanation
is based purely upon my memory, aided by this information and by reading Mr
Justice Fraser’s Horizon Issues judgment (No. 6) and Appendix 1. I am not able
to provide accurate dates nor certainty as to who knew what when. I cannot
meet in full the request set out in paragraph 20 of the Inquiry’s request. I am
asked about further bugs I recall. There were other bugs, which were reflected
in PEAKs, but it is difficult for me to speak to them now without the
documentation.
Receipts and payments mismatch (bug 1 in Horizon Issues)
63. A Receipts and Payments mismatch is a symptom of an underlying system
problem. Over the years there were various underlying problems which caused
this symptom. Every Final Balance Report included Receipts and Payments
totals. Receipts (money coming in, and stock/cash held at the start of the period)
should be equal to Payments (money paid out, and stock/cash held at the end
of the period). The postmaster would move any Discrepancy (a difference
between the cash the system thought should be held, based on transactions
Page 20 of 63
64.
65.
66.
WITNO0170100
WITNO0170100
recorded, and what the postmaster actually held) to Local Suspense when the
Final Balance Report was produced.
As a very simple hypothetical example, sale of a single £1 stamp: there would
be a pair of transactions £1 stamp out and £1 cash in. Each set of transactions
must always net to zero. But if only the stamp transaction was stored, with no
cash element, you would have a Receipts and Payments mismatch when you
produced a Balance Report. If you had taken the cash, you would also have a
discrepancy (£1 gain) but even after accepting the gain, you would still have an
R&P mismatch. If Receipts and Payments were not equal on the report, this
might be noticed by the postmaster. It would in any event be flagged centrally
by Legacy Horizon on one of the TPSC reconciliation reports (monitored by
MSU) and by HNG-X as a critical system event (monitored by SMC).
In respect of the specific problem identified in Appendix 1, a Receipts and
Payments mismatch occurred where the balancing process was cancelled by
the user at a particular point and then completed apparently successfully (not
the only or most likely point for the balance to be cancelled, but perfectly valid).
The amount of the mismatch was the discrepancy being moved to Local
Suspense (this is not a simple case like my example above but a much more
complicated instance of the R&P mismatch problem).
This would be evidently a bug in HNG-X which needed to be fixed, and in the
interim monitored to make sure that all instances were trapped. Post Office
would have been informed of each instance, I am not sure whether this was via
a BIM report or some other route. Fujitsu would not have contacted branches
directly unless the branch had raised a call in the first place.
Page 21 of 63
67.
68.
69.
70.
WITNO0170100
WITNO0170100
FUJ00081062 is an email exchange between Gareth Jenkins and me of 6 May
2010 in which Gareth Jenkins asked me whether there was a KEL for Receipts
and Payments mismatch events. I replied to Gareth that there was a KEL
created by a colleague which required SMC to raise a call if they saw this event,
but I had seen no such calls from SMC. Gareth had access to the SMC event
system and had noticed two occurrences of the Receipts and Payments
mismatch event. POL00029425 is KEL ballantj1759Q, which SMC should have
found and followed when confronted by this type of incident.
lam asked what would have been my initial thoughts on the cause and / or likely
symptoms of the reported events. I would have concluded that we had a system
problem and would have expected SMC to have raised a service ticket. I think
SSC did subsequently make further regular checks for these events, by looking
at historic records and on an ongoing basis, and that could well be because
SMC had missed these at the time.
Looking at the documents now, I am not sure that these two events noted by
Gareth were caused by the specific bug 1 discussed in paragraphs 128-140 of
Appendix 1, which was apparently investigated in September 2010. I cannot
remember who looked at them nor any details. KEL ballantj1759Q lists three
possible causes of HNG-X R&P mismatches that were found.
I am asked specifically to expand on the use of the phrase “PM-raised call from
a few weeks back”. I do not remember it but infer that I must have had a PEAK
assigned to me in Spring 2010, where a sub-postmaster had noticed a difference
in the Receipts and Payments totals on his balance report, which I had not yet
Page 22 of 63
WITNO0170100
WITNO0170100
investigated. It certainly should not have been left for weeks, but the HNG-X
pilot was over that period and it was an exceptionally busy time for us all.
71. lam asked to expand on the use of the phrase “The branch accounts may need
to be corrected” in KEL ballantj1759Q. I am asked to identify who would correct
the branch accounts and by what method. Without seeing KEL wrightm33145J
and the associated PEAK, I cannot be certain what had to be done to correct
the accounts.
Callendar Square / Falkirk Bug (bug 2 in Horizon Issues)
72. This issue became known as the Callendar Square bug because the sub-
postmaster at the Callendar Square branch encountered a problem with
Transfer of cash several times in 2005 and 2006, had a loss because of the
underlying problem, was understandably very unhappy with the way it was dealt
with, and shared his concerns with other sub-postmasters.
73. Within SSC we referred to the underlying problem as the Riposte Lock problem.
Normally Riposte messages were automatically replicated between counters so
each counter held an identical set of all transaction and reference data relating
to that branch. But occasionally one counter would fail to accept any messages
from any other counters. This usually seemed to be triggered by something
early in the declaration or balancing process. Repeated application events were
generated which were not visible to the user. The event storm and failure to
replicate would persist until the counter was rebooted or Cleardesk was run.
Cleardesk was an operation which consisted of an automatic shutdown and
restart of all counter processes in the early hours of every morning.
Page 23 of 63
74.
75.
76.
77.
WITNO0170100
WITNO0170100
The counter would still be able to serve customers and would appear to be
working normally, but anything done on other counters after the events started
would not be visible. Reports printed on the counter would not include
transactions done on other counters so those transactions might be re-entered.
Incorrect discrepancies could be reported if the money was in the till but the
transactions weren't included in the balance. Transfers between stock units
might be accepted in twice, causing a discrepancy and areceipts and payments
mismatch. Single counter branches could not have this problem.
In many, probably most, cases there was no impact on the accounts because
balancing was done on a single counter after the end of trading. Sometimes the
sub-postmaster gave up in frustration and found it was all resolved in the
morning. If they did complete the balance, any financial irregularity might show
on one of the Fujitsu reconciliation reports, or the branch could have an equal
but opposite discrepancy the following period, which would include the unseen
transactions.
The root cause of the problem appeared to be within the Riposte software,
which was a product supported by a company called Escher. If we had been
able to reproduce the problem, or work out what triggered it, it might have been
possible for Fujitsu code to deal with it, but as it was, we were reliant upon
Escher to provide the ultimate solution.
I have been shown Jez Murray's entry in FUJO0083663 on 10 November 2005
and I am asked how effective was the work-around suggested by Jez. It was
not effective. It relied on the postmaster and other staff being aware that
something was wrong with the system. I can clarify that Point 4 in that entry is
Page 24 of 63
78.
79.
80.
WITNO0170100
WITNO0170100
not a workaround, it is concerned with trying to sort out the financial
consequences after the event.
I am asked about the phrase, within POL00028984, “this problem has been
around for years and affects a number of sites most weeks”. I had known about
this Riposte lock problem since soon after I arrived at SSC in 2000. I and others
in the SSC understood the cause of the problem to be a problem in the Riposte
software which we thought was being investigated by Escher.
I can see now that there was a substantial delay in the matter being addressed
as between the Fujitsu Development team and Escher. That appears to have
been because Development were waiting to see further instances of the
problem. A misunderstanding occurred because there were two KELs for this
problem only one of which was updated to explain that Development were
waiting for further instances, whereas the other KEL, which was the one referred
to in practice, had not been updated with that information and indicated that
Development and Escher had the problem in hand.
I am asked but cannot recall how many sites were affected each week. I stress
that the transfer problem seen at Callendar Square was just one possible
outcome of the underlying Riposte bug. This outcome was not happening at
several sites per week. I am asked what steps were taken to mitigate the
problem. SMC would raise service tickets in respect of event storms from
counters and advise the sub-postmaster to reboot. HSD would advise branches
reporting that transactions done on one counter were not showing on another to
reboot before continuing with balancing.
Page 25 of 63
WITNO0170100
WITNO0170100
81. I am asked whether Post Office or sub-postmasters were told about the
problem. It was not raised as a wider problem with Post Office; each instance
was treated individually. In some cases a reconciliation report entry was
generated (for example the double transfer-in at Callendar Square which caused
a Receipts and Payments mismatch and a large loss); a BIM report was sent to
Post Office for these.
82. _Iamasked why no fix was trialled or implemented prior to release S90. I suspect
it was not raised with Escher until 29 September 2005 (see PC0126376) for the
reasons I have set out. They supplied a fix in their next delivery. As Riposte
was at the heart of the system, any changes to it had to go through a full test
cycle.
83. I am asked to explain how the SSC would rebuild the messagestore, which is
mentioned in KEL JSimpkins338Q FUJ00083720. I cannot remember the
process exactly, but you could delete the counter messagestore, recreate an
empty messagestore and let the messages replicate in from another counter.
Each counters messagestore held all the messages (transactions and many
other records) for that branch, plus reference data etc from the central servers.
84. lam asked about the spreadsheet with reference POL00029308. I have a vague
memory of being asked, I think by Steve Parker (SSC manager), to try to
produce a list of PEAKs relating to the Riposte Lock problem where there may
have been a financial impact. This was just before I left Fujitsu so would have
been end of December 2015 or early January 2016. I assume this is that list
although I am surprised to see the dates in American format and do not know
whether the spreadsheet was changed after I produced it. I think I must have
Page 26 of 63
85.
86.
87.
WITNO0170100
WITNO0170100
searched the PEAK system for any mention of Riposte Lock events, and
relevant KELs or similar, making several passes using different search terms. I
would then have looked at the individual PEAKs to assess what the likely impact
had been on the branch. I may have provided more background in an
accompanying email or a separate worksheet, quite possibly with a caveat that
it was not a definitive list, but I do not have a copy.
Looking at that document now I have a couple of reservations. I am surprised
there is nothing pre-2003 and wonder whether all PEAKs / PinICLs were still in
existence, or whether some of the earlier ones been lost when we switched to
PEAK. I am also not sure the statement to the effect that TPSC256 is no longer
populated is correct.
I have been asked to consider FUJ00083712. This is an email chain regarding
some ARQ requests from Post Office. The archived events for these branches
were checked to see if there were any Riposte Lock events which might have
impacted the branch accounts. This relates to the Callendar Square bug.
Counter 2 at Branch 107026 was generating repeated Riposte Lock events
between 00:04 and 02:27 on 5th January 2006. I am asked to comment on the
premise that this incident was after the S90 counter release but that is incorrect
as it was not (scheduled 04/03/06 to 14/04/06 according to email in
FUJ00083722) which explains how it was that the problem had not then been
fixed.
I am asked whether this problem had the potential to cause discrepancies in
branch accounts or otherwise to affect the integrity of the Horizon IT System. It
had the potential to do so but would leave evidence that something had gone
Page 27 of 63
WITNO0170100
WITNO0170100
wrong which would have been obvious to me and normally to the user as well.
It might encourage the user to put items through again but that would usually be
picked up at the back end. It might cause the user to accept a transfer in for a
second time but there would then be a receipts and payments mismatch on the
backend report in addition to the discrepancy. It might result in opposite
discrepancies across periods. It did not have the potential to cause unexplained
discrepancies across multiple weeks. The Riposte Lock events would be visible
in the counter application event log (one of the files always checked by SSC.
when investigating problems) for several weeks, and subsequently could be
looked for in the archived event files, even if not noticed by SMC when they
initially occurred. Accordingly, when, for example, I investigated the problem at
the branch at Marine Drive where Lee Castleton was the sub-postmaster, if the
Callendar Square Bug had been involved that would have been apparent to me
and in the event I was able to eliminate it as a possible factor.
88. I am asked specifically why I sent Gareth Jenkins the email dated 8 February
2010 at 14:17 (FUJ00083722). I assume I forwarded the email chain to Gareth
because he had asked me for some information in relation to his investigations
at a different branch.
Suspense account bug (bug 3 in Horizon Issues) and Local Suspense Account Issue
(bug 7 in Horizon Issues)
89. These are two separate bugs, both to do with the HNG-X Local Suspense
Account. When stock units were balanced at the end of a Balance Period
(usually 1 or 4 weeks) the clerk would transfer any discrepancy for each stock
unit (loss or gain) into a ‘pot’ on the system called Local Suspense. At the end
Page 28 of 63
90.
91.
92.
WITNO0170100
WITNO0170100
of the Trading Period (4 weeks), when the final stock unit was rolled over into
the new Trading Period, the postmaster would have to clear the balance in Local
Suspense, for example by making good any loss.
Bug 7 was the first in time, occurring in April 2010. The relevant documents
(FUJ00081867, POL00029380, FUJ00084125, FUJ00081896, FUJ00081868)
show that a postmaster was forced to clear Local Suspense several times,
getting stuck in a loop after printing the Final Balance Report when rolling into
new Trading Period. FUJ00084125 KEL acha5259Q has a clear description of
the symptoms. The underlying cause was a corrupt message sent from the
central server which the counter could not handle, causing the counter to
present the wrong screen to the user although a record to clear Local Suspense
had already been written in the branch database. There was no impact on the
branch accounts for the Trading Period just completed, but there would be an
impact in the following period.
I recall that restarting central servers stopped the corrupt messages in the short
term. The underlying cause of these particular corrupt messages was fixed in
May 2010 (FUJ00081867 PC0197409 and POL00029380 KEL PorterS199P).
An improvement to counter error handling was released in September 2010
(FUJ00081896 PC0198077).
Post Office staff in NBSC were aware of the branch problem from the outset,
raising a service ticket with HSD on 15 April 2010. Information on all affected
branches and the suggested workaround were sent to Post Office via MSU
(FUJ00081868 PC0197797, 28/04/2010 17:39).
Page 29 of 63
93.
94.
95.
96.
WITNO0170100
WITNO0170100
I am asked about the entry of 22 April 2010 in PC0198077 which states “The
solution we thought we had for Hucclecote...has not resolved the problem, but
has actually doubled the discrepancy”. This was an update from Ibrahim at
NBSC, who had tried to sort out the problem at Hucclecote before contacting
Fujitsu. Fujitsu did not talk to the branch directly, but my update at 27/04/2010
17:35 was relayed to Ibrahim.
I am asked about the entry “PC0198077 is B priority, whereas PCO197261 is C
priority. Hence PC0198077 has a better chance of being delivered even though
[it's] the same issue” in PC0198077. It appears that two PEAKs had been sent
to Development which required changes in the same area of counter error
handling. Rather than using the earlier, lower-priority PEAK for delivery,
Development wanted to use the later higher-priority PEAK, so it was more likely
to be put forward for integrated system testing and release to the live estate.
I am asked about the phrase “wider issue” in PC0198077. The wider issue
referred to was that there could still be discrepancies and other counter
problems if a corrupt message was received from the central server and the
counter did not take an appropriate error path. This was why it was important
that the counter error handling fix was implemented.
I am asked about the phrase “If this starts happening again, check whether
CastClassExceptions are occurring again” used in KEL acha5259Q. ATOS /
HSD could use a tool provided by SSC, as detailed in the KEL, to check for
these particular events, which would indicate that the central server was
generating corrupt messages which might cause branch accounting problems
or freezes. Restarting the appropriate central server as soon as possible would
Page 30 of 63
97.
98.
99.
WITNO0170100
WITNO0170100
help prevent similar problems at other branches (there were multiple servers
processing counter messages so one could be stopped and restarted without
impacting the live service).
I am asked to expand on my comment on 17 September 2010 in PC0198077
that this “problem has happened intermittently since April but never at the levels
seen then. Being investigated on PC0204396. The good news is [that no]
instances have been seen since this fix was applied”. I assume I checked back
at that point, or had been monitoring regularly since April, for these exceptions.
Once the counter error handling fix was implemented, corrupt messages were
handled sensibly and the counter would not freeze nor go down the wrong path.
Bug 3 affected a small number of branches in Jan/Feb 2013. The relevant
documents are FUJ00081875, FUJ00085079, FUJ00084857 and
FUJ00084827. When they rolled over the last stock unit into a new Trading
Period and cleared Local Suspense of the accumulated losses/gains, the
amount that was cleared was incorrect. It transpired that these branches had
had the same problem a year earlier, but it had not been reported to Fujitsu then
as far as I am aware.
I have tried to refresh my memory from the various PEAKs, KELs and emails
but this was an extremely complex technical problem which I will struggle to
make clear. In the simplest terms, a record related to Local Suspense,
belonging to a subsequently deleted stock unit, had been left lying around and
had been picked up when the same numbered Trading Period was reached the
following years. NBSC initially investigated one branch reporting the problem,
then raised a service ticket with HSD on 25 February 2013, who immediately
Page 31 of 63
WITNO0170100
WITNO0170100
passed it to SSC. It was soon obvious to me that it was a serious problem
affecting the branch accounts and I would have notified the SSC manager and
Problem Manager. There was a conference call with Post Office on 28 February
2013, by which time I had identified 14 branches affected. Fujitsu did not contact
any of the affected branches and Post Office undertook to resolve the
consequences at the branches and back-end systems, I having provided data
about the errors at each branch.
100. I made various checks regarding some other undeleted records in the same
table (not for Local Suspense) but concluded they had no impact on the
accounts. I also identified how to recognise the problem from the Branch
Trading Statements; my checks would not have found any affected branches
which had subsequently closed. Steps were taken to clear the old records from
the database (PC0224126 relates but has not been supplied to me).
101. There is a reference in PC0223870 to “couple of new checks to the balancing
process, to alert us if anything similar happens again” (see entry on 24 October
2013 in PC0223870). This relates to what was a code change to try to spot
anomalies in branch trading statements. The new checks were set out in KEL
acha2230K (FUJ00085079): 1. check that the opening figures generated for the
new Trading Period net to zero; 2. check that what is cleared from Local
Suspense equals what was putin. These are the two checks that would identify
the presence of the problem.
102. If these extra checks had been in place from the beginning of HNG-X we would
have known about Bug 3 when it initially impacted branches, rather than
depending on the sub-postmaster and Post Office reporting the problem to
Page 32 of 63
WITNO0170100
WITNO0170100
Fujitsu. It would not have prevented that problem from happening. The change
displayed a message to the postmaster and logged a system alert to be picked
up by SMC.
103. I am asked to explain in non-technical language what “teething problems with
archiving” were, as referred to in the entry on 28 February 2013 in POC0223870.
Records in some of the central Branch Database tables did not include a Year
field, just a Trading Period (TP). So these records needed to be removed, by
the automated archiving process, before the same TP was reached the next
year. The strategy was to remove the records once the branch was 3 TPs ahead.
However, the process did not work fully for records created by stock units which
were subsequently deleted; this was noticed in April 2011 when an old record
(not related to Local Suspense) caused a problem, though it had no impact on
branches. Changes made to the archiving strategy in July 2011 were expected
to remove any other old records belonging to deleted stock units but missed a
small number of such records.
104. I am asked for my recollection of a conference call with Post Office on 27
February 2013. As set out above, PC223870 gives the date for the call as the
28th with a later call in March. I have no substantive recollection of the call
beyond a vague memory that they were very unhappy that we had identified a
bug.
105. I am asked to explain what steps Fujitsu and/or Post Office took to identify all
branches affected by the bug referred to in this PEAK and KEL. For my part I
looked at the database for all branches to see if there were any further instances
Page 33 of 63
WITNO0170100
WITNO0170100
of historic and persistent data of a similar type which would be indicative of the
bug.
106. I am asked about FUJ00084857. The purpose of this email was to provide
Andrew Winn at Post Office with a technical explanation of the bug and
consequences. It seems likely that this came out of the second conference call
with Post Office and my email to Andrew Winn was in response to a specific
request because I would not have contacted Andrew Winn directly in other
circumstances. I identified the common features of the subset of branches
affected and the archiving strategies that were at the root of the problem. I see
that I gave an example, with a diagram (not now included) to try to make the
explanation clearer.
107. Iam asked about the email at FUJ00084742. I have no recollection of this email,
but I must have been involved in testing something on the RDDT test system.
This was not within my role, so I do not now understand why I was involved. I
do not think this email was to do with Bug 3.
108. I am asked about FUJ00084827. The extra checks referred to are those
described in my paragraph 101 above. I am asked but cannot recall how I
checked Andy Winn’s conclusions following his analysis of the financial
consequences of the affected branches. I think the phrase “Politically, it may
run and run’ likely related to Post Office’s unhappiness that there was a bug.
Dalmellington bug / Branch Outreach Issue (bug 4 in Horizon Issues)
109. An outreach branch (Dalmellington) scanned in a pouch of £8000 cash received
from its core branch. Two delivery receipts were printed as expected, then the
Rem In screen was displayed; the user pressed Enter to print the Rem In slip
Page 34 of 63
110.
111.
112.
WITNO0170100
WITNO0170100
and automatically add the cash to the system cash total. This should have been
the end of the process, but the user was shown the Rem In screen again. If
they pressed Enter again, another Rem In slip was printed and the system cash
increased again. This happened four times at Dalmellington before they
cancelled out of the process, meaning they would show a loss of £24000 when
they balanced.
The problem happened when a user had logged on but Horizon timed out due
to inactivity or lack of connectivity before the logon process was fully completed.
When they logged back on, Horizon was in an inconsistent state which caused
the Rem In screen to be redisplayed. This only affected manual Rem Ins which
were used mostly for movement of cash between core and outreach branches.
While this was a problem which affected a number of branches, my view was
that it would be obvious to the user, or anyone doing the balance and finding a
large discrepancy, that it was caused by the duplicated rems. I remain surprised
that it was not reported sooner. Sub-postmasters could and very occasionally
did scan the same pouch twice so perhaps if it was reported to NBSC it was
assumed that they had made an error. POL FSC could issue a TC to correct the
branch accounts, and when I checked previous instances, I found this had
happened in some cases. A code fix was produced which went live January
2016.
lam asked about an entry on PC0246949. There is a reference to the fact that
when investigating, initially I could only check back two months because the
relevant database table only held data for two months. I think potentially the
bug may have caused discrepancies from the introduction of HNG-X in 2010.
Page 35 of 63
WITNO0170100
WITNO0170100
113. 1am asked about an update to POLO0029902. This update was not added by
me but by Tony Wicks and I cannot speak to it save to say that I would assume
his use of the phrase “high visibility” reflects the fact, as is apparent from the
email chain referred to below, that Post Office and ATOS were aware of the
problem.
114. lam asked about the email at 09:53 on 20 October 2015 (FUJ00085864) and
the advice given to Post Office in that instance. I had closed PC0246949
(FUJ00085831) setting out that there was a possible avoidance action that the
branch could take, which I could see had been done at other branches (not at
Fujitsu’s suggestion). This was to Rem Out the excess remmed in (while not
actually removing any cash from the outreach branch), which would correct the
system cash figure. I knew that the Horizon system did not match up pouches
remmed out of core branches with those remmed into the outreach branches,
so at the time it seemed reasonable, to avoid them rolling over with a large
discrepancy. However, whether to deploy that approach was a business
decision for NBSC/Post Office and I knew I must not suggest it to the branch.
The underlying problem was _ still being investigated (PC0246997
POL00029902).
115. I am asked about my email at 11:11 on 26 October 2015 (FUJ00085864).
Specifically I am asked to expand on the phrase “/ don’t think it is realistic to
expect postmasters to avoid the problem by not pressing Enter multiple times”
and to set out what I felt was required to rectify this issue. It needed a code fix
so postmasters were not presented with the wrong screen. I am asked why I
could not advise on how to deal with discrepancy the answer to which is that
SSC were not allowed to advise sub-postmasters on business issues. The
Page 36 of 63
WITNO0170100
WITNO0170100
other alternative to be considered was for Post Office to issue a Transaction
Correction.
116. I am asked about an email of 15:49 on 9 November 2015 said to be within
FUJ00085864 but cannot see an email of that description in the material
provided to me.
117. lam asked about my email of 11:31 on 9 November 2015 (FUJ00085864). I
was responding to Steve Parker's request to ‘put the issue into perspective’. I
do not know to whom it was to be passed and my answer was not tailored for
any particular recipient.
118. Iam asked about my work as referenced in an email at 11:22 on 3 December
2015 (FUJ00085894). Initially I checked in the Branch Support Database
transaction table for duplicate pouches being remmed in. This found four other
affected branches in the previous two months. I realised the BLE files, going
back several months, included pouch ids so could be scanned for duplicates.
Initially I had only checked for duplicate cash rem ins but I rechecked for
currency too. FFinally, all BLE files from the start of HNG-X (2010) were
extracted from archive and checked. Checks were also made going forward
until the code fix was applied.
119. FUJ00085894 is an update to my manager of where I had got to by 3 December
2015. FUJ00085895 is a spreadsheet showing progress extracting and
checking the files. FUJ00085865 is a sample of pouch info from the BLE data
being examined. FUJ00085896 is the results of the analysis (pages 3 and 4
belong on the end of 1 and 2). This may not be the final version and I would
probably have written a covering note
Page 37 of 63
WITNO0170100
WITNO0170100
120. lam asked about my email of 11 December 2015 (FUJ00085864). I wrote there
“Not sure who needs to know this” because I had thought of a way to check
further back than two months and wanted to inform my management. In the
event we scanned for all cash and currency pouches back to 2010.
121. I am asked what information from these of my investigations was shared with
Post Office and/or sub-postmasters. Post Office had my findings and followed
up on them, on which point see the FUJ00085922 emails. I do not know what
Post Office shared with sub-postmasters.
122. Iam asked to explain HORICE (FUK00085886). HORIce was a tool that the
Helpdesk could use to run a pre-defined query on the Branch Support Database.
The purpose of the checks requested was to look for any duplicate pouches
newly remmed in.
123. The fix for this bug was a counter code change to ensure the counter was reset
to a tidy state if there was a system logout or inactivity logout before the post-
logon checks had been completed. I can see from PC0246997 POL00029902
that rollout of that fix to live counters commenced 12 January 2016 and there
were 400 counters outstanding by 14 January 2016.
Remming in bug(s) (bug 5 in Horizon Issues)
124. This relates to two separate bugs which had the same effect of accepting the
same pouch twice. The bug referenced in PC00195511 (FUJ00083494) and
PC00195380 (FUJ00081865) happened during the HNG-X pilot. It was possible
for the user to use the PREV key to backtrack while remming in a pouch, and
this would result in the Rem In being completed twice, causing a discrepancy.
Page 38 of 63
WITNO0170100
WITNO0170100
125. PC00203085 (FUJ00081870) relates to a situation where the user started the
process on one counter but could not complete it, probably due to a printer
problem, so the user remmed in on another counter but the process did
complete on the first counter as well, causing a discrepancy. There was a check
in the code to prevent this happening on two counters at once, but it had not
been implemented properly. This bug too was present from the introduction of
HNG-X in 2010.
126. I wrote that the “problem can cause losses which are hard for the branch to
identify”. \n fact, the branch could see from the transaction log or remittance
report that the same pouch had been remmed in twice and I can see I was
somewhat overemphasising the impact to encourage Development to deal with
it as a priority.
127. The counter code fix for the first problem was applied around 19 April 2010. The
second problem was solved by a central (BAL) fix applied on 23 January 2011.
128. Iam asked to expand on “in the meantime the branch will report any shortage”
in KEL acha4221Q. The KEL actually says “will report a shortage” meaning they
will appear to have a loss, caused by the duplicate rem in.
129. I am asked to set out any steps Fujitsu took to identify whether this/these bug(s)
had caused discrepancies in branch accounts that had not been reported by
sub-postmasters or Post Office. I do not remember, but investigation of
duplicate pouches in 2015 picked up those in the first set, and I wrote then (see
FUJ00085894) “The first set, highlighted in blue, was documented in KEL
acha42210/ PC0195380, fixed around 19 April 2010. A check was run at the
Page 39 of 63
WITNO0170100
WITNO0170100
time looking for duplicate pouches so POL FSC could be informed via BIMS
(and there should be Peaks covering these).”
130.1 am asked to set out any steps Fujitsu took to assist sub-postmasters in
identifying when there had been a remming in error. However, I would expect
NBSC to do that as part of their responsibility for business issues.
Remming out (Bug 6 in Horizon Issues)
131. In February 2007 there was a remming out bug in T30 software update. There
is a good description of it in the Appendix at paragraphs 201 to 203. The
software was regressed as soon as the problem was identified (which would
have been within a day or so of it occurring) so no others would hit the same
problem. The software was changed to fix the bug then tested and released.
Steps were taken to identify all affected branches. I believe there are relevant
KELs which are not now available to me.
132. Ihave no recollection of the May 2005 bug under this heading.
Recovery Issues (Bug 8 in Horizon Issues)
133. FUJ00081976 is PEAK PC0197769 which documents that I was assigned this
PEAK, investigated it and referred it for a fix which was successfully applied and
I then closed the PEAK.
134. Recovery was the process invoked if transactions had been started but the
customer session was not completed (for example because of loss of comms
with the data centre, or because a user logged on at a different counter without
settling the session first). The next person logging back on to the counter would
be asked whether the transaction had in fact been completed (e.g. had a cash
Page 40 of 63
WITNO0170100
WITNO0170100
withdrawal been handed to the customer). It was a complicated area for the
users to negotiate.
135. This was not an instance of a failed recovery, it was successful, but the
recovered transaction was assigned the wrong Trading Period. This was a bug,
which occurred in April 2010 during the HNG-X pilot, which would cause a
discrepancy. A fix was rolled out to the counters around 14 June 2010.
136. I do not remember specifically monitoring for these errors, nor whether Post
Office were aware, nor what was done about the branch accounts (in this case
they had a loss in one period but a corresponding gain in the next) but according
to PC0197769 I talked to Gareth Jenkins about it and we intended to check for
other instances. My final update on 23 June 2010, “We've already seen that
this fix is being effective”, suggests we had been monitoring.
137. Iam asked what steps Fujitsu took to monitor failed recoveries. A daily report
was produced. I cannot remember if this was there from the beginning of HNG-
X or was introduced a little later. Failed recoveries were flagged as such in the
Branch Database recovery table and this was scanned to produce the report. I
cannot remember exactly what caused a failed recovery, rather than a
successful one.
138. I am asked but cannot recall the process that would be followed once a failed
recovery was identified. The Appendix suggests [at 218] the answer is
documented in KEL acha959T which I do not have. SSC staff would look at
each instance, so many PEAKs were raised for failed recoveries.
Page 41 of 63
WITNO0170100
WITNO0170100
Reversals (Bug 9 Horizon Issues)
139. I have no recollection of this bug, but I can see from the information in Appendix
1 that it affected reversals of Rem Ins for a short period in 2003. This would have
been a relatively unusual operation, and it would have been obvious to the user
and to anyone else checking the branch transactions that the reversal had not
worked as it should.
Data Tree Build Failure Discrepancies (Bug 10 Horizon Issues)
140. According to Appendix 1, data tree build failures occurred in 1999-2000 and
caused sizeable discrepancies in office snapshots at some branches. This was
before I joined SSC. I have no recollection of any incidents involving data tree
build failures until 2005-2006.
141. FUJ00086490 is PEAK PC0146170 which documents that in 2007 I was
assigned this PEAK, investigated it and referred it for a fix which was
successfully applied later that year after which I reduced the priority from A to B
and subsequently closed the PEAK.
142. PC0146170 FUJ00086490 was raised in May 2007. SPM Declared cash /
currency etc, then transferred out cash/currency or did some transactions.
When they declared again (for the correct adjusted amount), the system
reported a variance equal to the value of the transfer / transactions.
143. I’m not aware that this was ever reported to Post Office as a problem. If SSC
had investigated an instance and found it had caused a discrepancy at a branch
when they balanced, it should have been notified to Post Office via a BIM report,
but I don’t know if that ever arose. Although the variance, and potentially
discrepancies, were calculated incorrectly, it was often obvious to the user that
Page 42 of 63
WITNO0170100
WITNO0170100
the system was wrong. If they did complete a balance with some transactions
missing, those transactions should have been picked up in the next period.
144. 1am asked whether I considered the workaround in KEL MScardifield2219S to
be adequate. I did not. It was slightly better than having to reboot the counter.
Initially the workaround was for the user to logout and backon, but that was not
always sufficient to clear the problem, so this more complex method was
suggested. I am asked but do not know how often the problem was reported.
145. I am asked to explain why a fix for this problem was not sent for development
prior to May 2007. A similar problem had been seen during Post Office
acceptance testing of the S80 release, in June 2005. PC0121925
POL00028867. As it occurred only once and could not be replicated, it was
decided to document it (KEL MScardifield2219S FUJ00086474). It was then
seen again in testing, in slightly different circumstances (PC0121925
04/07/2005). The fault was logged with Escher (the root cause appeared to be
in their code, not Fujitsu’s) but closed a year later with no indication of any
investigation.
146. In the meantime, similar problems were reported by SPMs, and the helpdesk or
SSC followed KEL MScardifield2219S with the avoidance action. But with
increasing numbers of calls (from some time in 20067), and after talking to the
manager at branch 080940 which was frequently hitting the problem, I sent
PC0146170 to 4" line support.
147. did not record what I told the sub-postmaster was the cause of the problem but
I would probably have said that it was a system error and we were going to
investigate further.
Page 43 of 63
WITNO0170100
WITNO0170100
Girobank discrepancies (Bug 11 Horizon Issues)
148. I have no independent recollection of this bug. From Appendix 1 my
understanding is that the issue was found to have occurred only between May
and September 2000 and I understand that it was a Post Office business issue
which did not cause discrepancies at branches.
Counter replacement issues (Bug 12 Horizon Issues)
149. I remember dealing with instances where counter replacement resulted in a new
RiposteOnline message being written with the same message number as one
created earlier, thus effectively overwriting the older message, and a ‘self-
originated message’ event being generated. If it was a financial message, this
would cause a discrepancy, and almost certainly an entry on one or more of the
reconciliation reports. I cannot remember exactly what action was taken. Post
Office would have been sent a BIMS report if there was a lasting financial impact
for a particular branch but I do not know whether it was ever flagged as an
ongoing problem. I remember steps were taken to stop it happening but I cannot
remember when it stopped.
150. The bug referred to in PC0153851 from February 2008 (as described in
paragraph 271 of the Appendix) is a completely different problem to do with a
Riposte index, not message replication. Again, I remember seeing a very small
number of problems caused by incorrect Riposte indexes during the lifetime of
Legacy Horizon. SSC could rebuild the indexes (and I think, but am not certain,
that they were rebuilt nightly anyway). As this was seen as a transient problem
it may not have been investigated by Escher or FJ development.
Page 44 of 63
WITNO0170100
WITNO0170100
Withdrawn Stock discrepancies (Bug 13 Horizon Issues)
151. Branches were given plenty of notice by Post Office (both online memos and in
weekly paper publication) that they should rem out and return stock that was
being withdrawn, but Horizon Online did not handle properly any remaining stock
that no longer had any valid transaction modes. I think that in some cases Post
Office reactivated withdrawn products at affected branches (via reference data)
so they could rem out the stock.
Bureau discrepancies, Phantom transactions and Concurrent log ins (Bugs 14, 15 &
18 Horizon Issues)
152. I have no independent recollection of Bugs 14, 15 or 18. Looking at the
information about them now, I note for example, that Bug 18 could only be in
play in circumstances where there were concurrent logins by the same user at
multiple counters and that would be evident.
Post & Go/TA discrepancies in POLSAP (Bug 19 Horizon Issues)
153. FUJ00085483 is PEAK PC0220393 which I note from the document was
assigned to me on 31 August 2012. After the solution was found I made a note
that we strongly recommended that POL monitor for the problem at other
branches so that the same correction could be applied.
154. I cannot remember sufficiently the interactions between the Post and Go
terminals and the Horizon system. The problem here was that the SPM was
only prompted (and able) to associate a PG terminal with a stock unit when a
cash Transaction Acknowledgement (TA) was received for that terminal. These
TAs came via central systems, there was no direct feed from the terminals into
Page 45 of 63
WITNO0170100
WITNO0170100
Horizon at the branch. So PG terminals not doing cash transactions (1 assume
they were card only) would not have an associated stock unit.
155. It seems unlikely to me now that this would have any direct impact on the branch
accounts, as there was no cash element, unless the PG stock was managed on
Horizon. I do not recall any branch-raised calls about this. It did however impact
POLSAP as it held up all the data for affected branches.
156. I cannot remember how many branches there were with non-cash PG terminals
but it was a small number (tens rather than hundreds).
Bureau de change (Bug 23 Horizon Issues)
157. POL00001264 is PEAK PC0129767 which shows that this matter came to me
on 6 December 2005 and that I considered that the system was working as
intended but the user had made an error. I noted that the system could be
simplified or better explained and I addressed how that might be progressed
recognising that it was a decision for Post Office and not for me.
158. KEL AChambers2252R (POL00001272) gave advice on reversing currency
transactions and was raised because of PC0129767 (POL00001264). An SPM
had reversed a currency sale but when they balanced, found they had a surplus
of Euros and a shortage of cash. So they accepted the surplus (which put the
holding up to the correct level on the system, and reduced the cash shortage),
but they were still short by the margin on the transaction.
159. I am asked to explain why I considered there to be user error. The session in
which the currency was sold would have consisted of three transactions,
something like:
Page 46 of 63
WITNO0170100
WITNO0170100
Euros -£100; Margin -£10; Cash £110
160. Toreverse the sale, the SPM had to enter the transaction id. I cannot remember
exactly how they found this but possibly via the session id (printed on the
receipt) and then they would be shown the transactions in the session? The
first two transactions had the same id (ending -2), because they are linked, and
the cash settlement was different (ending -3).
161. In this case the SPM entered the id ending -3. This wrote a pair of reversal
transactions Cash -£110; Cash £110 and printed a receipt showing this. If they
had selected -2, both the Euros and Margin would have been reversed, and
settled to cash. This was the same for all reversals, not just currency.
162. It seemed appropriate therefore to categorise this as “User Knowledge”.
Wrong Branch Customer Change Displayed (Bug 24 Horizon Issues)
163. POL00001253 is PEAK PC0129791 which shows this matter was assigned to
me on 7 December 2005 and that I recognised that the problem should have
been cured by a reference data fix that had been rolled out but had subsequently
been removed. It was reapplied on 8 December 2005. It is recorded that I
explained the situation to the postmaster. This is also reflected in the associated
KEL which is at POL00001210.
164. Iam asked to consider POL00001210 and POL00001253. If the clerk used the
Quantity button in Smartpost (for example to print multiple mails labels), it was
not reset to 1. Sales of other items might be multiplied unintentionally, and/or if
the customer handed over £10 say, and the clerk entered this on the system
and let the system calculate the change, it would be treated as £20.
Page 47 of 63
WITNO0170100
WITNO0170100
165. It seems highly likely that this would have happened and not been noticed at
some branches, and they would have made a loss with too much change given
to customers. Or customers may have been overcharged and the branch made
again.
166. 1! do not know whether Post Office were told of this problem (originally reported
on PC0128264 04/11/2005).
167. Iam asked how many branches were affected by not being included within the
fix on 28 November 2005. My answer is that all branches were affected. Group
111111112 was not a subset of branches, it was a reference data group. A
complete set of reference data for Smartpost existed in two sets, one of which
was active at any one time (so changes could be prepared and delivered to all
counters, then all switched to the updated version at the same time). The fix
applied to the active ref data group 111111113 on 18" Nov was lost when group
111111112 became active on 28" Nov.
168. I am asked what I told the sub-postmaster (my entry of 7 December 2005 in
POL00001253) and that was probably that the system was wrongly using the
Smartpost quantity for subsequent transactions within the same session, and
they should reset Quantity to 1 before continuing.
Lyca top up Bug 25 Horizon Issues)
169. I have no recollection of this bug.
TPSC250 (Bug 26 Horizon Issues,
170. TPSC250 was one of the suite of daily reconciliation reports, and any entries on
the report would have a PEAK raised for them by MSU and then SSC would
Page 48 of 63
WITNO0170100
WITNO0170100
investigate. There were several different causes for these entries over the life of
Legacy Horizon, some of them bugs. If investigations found that the problem
had caused an incorrect discrepancy at a branch, Post Office were notified via
a BIMS report.
171. Mr Justice Fraser made reference to the specific postage labels problem which
was the underlying cause of many of the entries. Having considered his
conclusions, I would clarify that the report entries per se did not cause
discrepancies, but the separate bug which caused a negative label to be printed
did cause Post Office a loss, with either the branch or the customer making a
gain.
172. In rare circumstances where a parcel already had stamps or prepayment
attached, and postal service options (e.g. additional insurance) were selected
and deselected to first exceed and then be less than the prepayment, a negative
value postage label was printed. When the customer session was completed, it
would show that the excess cash should be handed back to the customer. If the
clerk did this, the branch accounts were all correct and consistent. If they did
not, thinking that once the stamps were stuck on the customer had committed
to spending them, then the branch made a gain of the amount; the branch
accounts showed what should have happened.
173. The report entry occurred because at the end of the day, all the transaction
absolute values (i.e. ignoring sign) were added together on the branch computer
and compared against the total on the data centre database. This was done for
all branches, every single day, to look for any anomalies. These were the
counter reconciliation figures and they were only used in this cross-checking,
Page 49 of 63
WITNO0170100
WITNO0170100
not in the branch balancing process. The reconciliation totalling done at the
branch did not handle the negative-value postage labels correctly and so the
total did not match that at the data centre, which was correct.
174. The balancing process did handle the negative values correctly and so this
particular bug had no impact whatsoever on branch accounts, save to the extent
as described above.
175. There are certainly other PEAKs relating to TPSC250 entries (but insufficient
information in Appendix 1 for me to expand) and in some cases there will have
been a genuine problem which impacted the branch. The whole point of the suite
of reconciliation reports was to make sure anomalies were investigated.
TPS Bug 27 (Horizon Issues)
176. I.am not sure why this is described as TPS (which was the name of one of the
central databases). The bugs discussed under this heading in Appendix 1
appear to relate to corrupt messages written by Smartpost code, which caused
entries on one or more of the TPSC Reconciliation Reports and so were all
investigated by SSC. In a subset of instances, this bug could impact branch
accounts. I cannot remember whether the root cause was ever found or whether
there was any attempt to fix it.
Drop and Go (Bug 28 Horizon Issues;
177. I have no knowledge of what is referred to as bug 28 which I see is said to have
occurred in 2017 which was after I had left Fujitsu.
Page 50 of 63
WITNO0170100
WITNO0170100
Further Questions
PEAK PC0229446 (FUJ00083493)
178. I am asked to explain the nature of the problem reported in this PEAK. The sub-
postmaster reported that he had a cash declaration problem and was losing a
lot of money. The only specific example reported via the helpdesk was a loss
of £6000 at 12:34 on 12 November 2013.
179. I extracted the cash transactions stored in the branch support database for this
branch / stock unit since the start of the balance period on 7 November 2013,
and put them into an Excel spreadsheet, also the cash opening figure of
£60,125, and added a column to calculate the system cash position after each
transaction. I then compared the system cash position with the amount they
Declared they were holding (normally done at the end of each day). The
difference was called a Variance. I think there was an option for them to view
the Variance each time they Declared cash.
180. I could see that the Variance reported each time by the system had been
correctly calculated, based on the cash transactions recorded and the
declarations made. In my response I gave the specific figures for 7 November
2013, after the rollover, showing that at 15:29 they apparently had £5,528.60
less than the system expected, with just three (or three types of) transactions
carried out in the new period.
181. I would also have checked for any system events and reconciliation report
entries for the branch and looked for any obvious duplication of transactions.
182. I documented what had happened on the simplest day as an example. At the
time it did not appear necessary to document the detail of what I found on the
Page 51 of 63
WITNO0170100
WITNO0170100
subsequent days but I could have made it clearer that I had checked all the
Variances for the week and found that the system had calculated them correctly,
based on what had been input at the branch.
183. I am asked to expand on my entry ‘I can’t tell why the declared cash doesn't
match the expected cash figure, the branch need to make sure that what they
have recorded on the system is correct, and investigate the anomalies” and to
explain how the branch could have investigated further. The system cash figures
were consistent. If they were wrong, then either the Declaration amount
recorded on the system (and printed on the Declaration report) did not match
what the sub-postmaster thought they had declared, or the transactions were
not an accurate record of what had been done at the branch. This could be user
error or potentially system error but could only be identified by the branch. So
looking now at the situation as it was on 7 November, the branch should have
been able to determine whether they had actually remmed in £22,300 and
transferred out £6,500 and had one cash receipt of £11,183.60. I was not able
to see whether they had £81,580 cash in the drawer, only the branch could
confirm that.
184. I have noticed that I made a transcription error in the figures: I said the system
cash total was £87038.60, should have been £87108.60. That error does not
change the conclusion or the other figures.
185. Iam asked to explain why I updated the root cause to “General — User”. There
was no evidence to suggest there was any kind of bug or system error. As I
have explained above, in such circumstances I had no further resource through
which to progress an investigation. In those circumstances I would have
Page 52 of 63
WITNO0170100
WITNO0170100
considered that user error was the most probable cause. I would accept I might
alternatively have expressed this as the being an inference from the absence of
evidence of system error.
PEAK PC0208335
186. I am asked about FUJ00086720 and specifically asked to explain what was the
“known problem with declarations containing withdrawn products”. The answer
is Bug 13 in Horizon Issues, as discussed above. I am asked why it was
anticipated that this problem could affect 10 branches per week “over the next
few months”. I could see the declarations, coming up to a year old, in one of the
database tables, so could estimate which branches might hit the problem and
when. I am asked with reference to my entry at 14:42 on 14 February 2011,
how was the “old declaration...removed from branch 169217”. NBSC advised
the branch on how to make a zero stock declaration, which would remove the
old data. This did not actually affect their stock levels in any way, it just meant
that when they balanced they would not be forced to Declare Stock (often they
used Adjust Stock instead, if they did want to Declare Stock then they would do
anew declaration with the correct figures, as normal).
187. I am asked whether I consider this PEAK evidenced a software problem and, if
so, how it fixed. I do think there was a software problem in that the archiving
being done for the table did not clear out old declarations as it should. A fix was
made to correct the archiving criteria, but in the short term the old declaration
records were deleted from the database. I am asked but do not know why the
PEAK was closed as “Administrative Response”. It was not my entry.
Page 53 of 63
WITNO0170100
WITNO0170100
PC0055072 and AChambers232K (FUJ00059003 and FUJ00077691)
188. I am asked to explain the nature of the problem described in this PEAK and this
KEL. I have almost no recollection of this, it must have been one of the first
service tickets I investigated. My response on PC0055072 FUJ00077691 on 5
October 2000 explains that AssetManager (part of Riposte, but not used by
Horizon) was causing a critical system event when trying to start each night. As
AssetManager was not used, this had no impact on the counter operation,
except possibly on performance. PC0055072 says the problem was seen on
only two counters (out of approximately 35000), I do not know if this remained
the case.
189. I am asked why it was not investigated further. Quite a lot of investigation was
done within Fujitsu and it was sent to Escher, to investigate why AssetManager
was trying to start on this counter when the DisableAssetManager flag was set,
but eventually it was closed without a fix in January 2002.
JBallantyne5245K, PC0083101 (FUJ00083631 and FUJ00059049)
190. I am asked to consider FUJ00059049 and FUJ00083631. FUJ00059049 KEL
JBallantyne5245K is the same document as FUJ00083878 and relates to the
Callendar Square bug which I have discussed above. FUJ00083631
PC0083101 reports a single instance of the Riposte Lock event while one of the
overnight processes was running, not when anyone was logged on and
balancing. The process completed successfully after Riposte was restarted
automatically a couple of hours later.
Page 54 of 63
WITNO0170100
WITNO0170100
PC0074630 (FUJ000821
191. I am asked to comment on FUJ00082021. This was a fairly common user error,
which I understand is documented in KEL AChambers037R (with which I have
not been provided). At the end of each day, all the cash in each stock unit had
to be counted and entered (by denomination) using the Declare ONCH
(overnight cash holding) function. The clerk also had to enter a two-digit
Declaration Id. This allowed branches with shared stock units to count and
report the cash in each separate till, by using different Ids. The system totalled
the declarations for each stock unit and sent the information overnight to
SAPADS. On Wednesdays, as part of the balancing process, Declare Cash
was used as well, or instead. This was also included in the SAPADS totals. If
the same cash was included in two declarations with different Ids, it was counted
twice. In PC0074630 FUJ00082021, on Wednesday the cash was included in
ONCH Id 01 (made by the clerk), then the SPM Declared Cash for balancing
using Id 03. So the amount notified to SAPADS was twice what it should have
been. This did not have any impact on the branch accounts (the ONCH
declarations were not used for balancing) but branches might be told by Post
Office that they were holding too much cash. The system was working as
designed and agreed by Post Office.
192. I am asked about but have no recollection of my conversation with Deirdre
Conniss regarding this incident. Nor can I assist as to how many branches were
affected but I do recall it continued to be reported as a problem from time to
time. I am asked about my comment “May require changes to PO
documentation”. I thought perhaps the Post Office’s training and/or Operations
Manual might need clarification to help SPMs avoid this mistake.
Page 55 of 63
WITNO0170100
WITNO0170100
Escher / Riposte
193. Iam asked how SSC investigated problems that were suspected to relate to third-
party software such as Riposte. I have explained this above in respect of
Callendar Square. Escher supplied Riposte. SSC investigated problems and
passed them on to 4" line, who would decide whether the fault lay in Fujitsu
code or in Riposte. I had no direct contact with Escher and cannot speak to the
relationship between Fujitsu and Escher.
Remote Access
194. Iam asked about the privileges I and other members of the SSC had with regard
to the viewing and/or modifying of live branch data. All members of SSC were
able to view live branch data all the time I worked in SSC, and we could not have
done our job otherwise. All members of SSC had access to tools/functions
which could be used to insert extra transactions, and at HNG-X (2010 until
unknown end date) to delete or modify them. We did not have access to the
Desktop view of Horizon that the branch users had, except on completely
separate test systems.
195. I am asked about vetting for this purpose. I was vetted before joining SSC but
cannot remember the details of that; I assume that was standard practice but do
not know.
196. I am asked to explain the extent of those access rights. For Legacy Horizon,
SSC could retrieve the messagestore for any branch, from the copies held at
the data centre or from the counter itself, and examine the transactions and
messages written when stock units were balanced and when cash accounts /
Page 56 of 63
WITNO0170100
WITNO0170100
trading statements were produced. For HNG-X, we could extract transaction
data from the Branch Support Database (BRSS) or Branch Database (BRDB).
For both systems, we developed various tools to facilitate this, for use both
within SSC and by other support teams.
197. With Legacy Horizon, it was possible for any SSC technician to append
messages to the messagestore for any branch counter. It was not possible to
alter any entry that had already been written, nor todelete anything. Messages
(written in something similar to XML) could be hundreds of characters long and
had to be correctly constructed or would cause harvester errors subsequently.
We would usually try on a test system (we could load a full branch messagestore
onto a test counter) before applying to the live service, to ensure it had the
desired effect.
198. At HNG-X, initially SSC staff (though possibly not all) had access to the APPSUP
role on the new BRDB and BRSS databases, though we would not have used it
when viewing data (normally on BRSS). If used, this gave full
modify/delete/insert access to the tables holding branch transactions. We also
had access to the correction tools written by Development, which SSC could
use in exceptional circumstances to add to the main transaction table (which
would affect branch accounts), and to update entries in the failed recovery table
(with no impact on accounts).
199. Iam asked whether a person using a branch terminal would be expressly notified
by Horizon if changes were made using these access rights. It is hard to
remember now what was done so long ago. If the branch had raised the service
Page 57 of 63
WITNO0170100
WITNO0170100
incident in the first place, via HSD, then I would probably have told them if I was
going to make a change to their system to correct the problem. If they had not
raised it, but instead we knew there was a problem because of a Reconciliation
Report entry and the branch had not yet done their balance, I probably would
not have contacted the branch, but would have informed my manager or the
Fujitsu Problem Manager. If the problem was notified to the Post Office and
they had authorised the change, I would expect Post Office to decide whether
or not to notify the branch.
200. I am asked about the audit data that would be kept to record any changes made
using these access rights. In Legacy Horizon, the messages themselves were
visible in the messagestore and would be included in the audited data archive
for the branch. We would usually create the message using a non-existent
counter number, e.g. 99, and the username SSC, and include a comment field
with the PEAK reference. This was my normal practice but it was not enforced.
In HNG-X audit files were created when the correction tools were used. The
HNG-X audit files were held in filestore on one of the servers. I am asked
whether they could be modified manually. I assume, but don’t know, that they
could not subsequently be deleted or modified.
201. I am asked whether any procedures and/or checks were in place at different
times to control remote access. An operational change request would have to
be raised and approved first. The approvers would include the SSC manager,
probably Post Office, a Fujitsu Problem Management and possibly others —
again this may have varied with the different systems over the years and I cannot
be more specific. Scripts and before / after data would be collected and saved
Page 58 of 63
WITNO0170100
WITNO0170100
(on the associated PEAK I think, or possibly on the change request document).
A second SSC technician would have to witness the change being made. I am
asked who was responsible for ensuring compliance with these procedures and
I do not now know but it was probably the SSC manager.
202. I am asked whether I consider that the use of remote access could affect the
reliability of branch accounts. Viewing the data for support purposes had no
impact on the accounts. On the occasions that I added in extra messages that
affected the accounts, or was aware of anyone else doing so, this was done to
put the accounts into the state they would have been in if the system had worked
as it should have in the first place. For example, very rarely a branch was able
to Transfer In a single transfer (of cash from another stock unit) twice, resulting
in a loss and Receipts and Payments mismatch when they balanced. If they
had not already completed the balance, SSC might write a pair of equivalent but
opposite messages to ‘undo’ the second Transfer In.
203. APPSUP was an Oracle database role which allowed users to update, insert and
delete records in database tables. I am asked for which period of time SSC
users had access to APPSUP. At Legacy Horizon, the set of central databases
didn’t contain branch accounting data, but extracts which were sent on to Post
Office and APS clients such as BT or banking clients. SSC could look at data
in these databases using the default, read-only role, but very occasionally it
might be necessary to make some change (possibly to correct an error
introduced by a bug or incorrect Post Office reference data). The user would
switch to role APPSUP to make the change. This would not affect the branch
accounts.
Page 59 of 63
WITNO0170100
WITNO0170100
204. At HNG-X, initially SSC staff also had access to the APPSUP role on the new
BRDB and BRSS databases, though we would not have used it when viewing
data (normally on BRSS). I don’t remember all the details of PC0208119
FUJ00089756, nor when APPSUP access to the BRDB was removed, but we
should only have been using the SSC role which would allow us to use the
correction tools written by Development.
205. I am asked about FUJ00089756 and my phrase “When we go off piste we use
appsupp”. I am asked about the circumstances to which I was referring.
Potentially there might be a need to make a change to other BRDB tables (not
holding branch transactions) where no tool had been provided, possibly quickly
if, for example, a branch could not trade at all until the change was made. I
cannot remember now whether this ever happened in practice but I think not; in
the Peaks that I have seen recently where records did need to be removed, for
example old declarations or Local Suspense information not removed properly
by archiving, Development provided a script to be run.
206. I am asked what security and/or audit procedures were in place when APPSUP
was used. An MSC would be raised and approved prior to any change, and a
log kept of what was done (stored on the MSC or on the associated Peak - I
cannot remember which). As I have said, it was the practice that a colleague
would witness any alteration. There may well have been more data which
recorded SSC actions which I no longer recall.
Page 60 of 63
WITNO0170100
WITNO0170100
General
207. Iam asked questions as to the overall robustness of the Horizon system. I have
dealt with many of the specifics above. I repeat the point that the pilot and
introduction of HNG-X was difficult and various problems were identified in that
period of early to mid 2010, and I would not say the system was fully robust
during that period. I cannot speak reliably to the first months when Legacy
Horizon was being rolled out when I was new to SSC and still learning about the
system. Otherwise, I considered that during my time in SSC, the system was
relatively robust, which is not to suggest, as will be clear from my discussion
above, that there were no bugs: it was our job in SSC and our expectation that
we would investigate and discover bugs that required fixes.
208. I am asked whether I believe that sub-postmasters had access to adequate
advice and assistance in how to use the Horizon IT System. I do not know
whether they did. I did not have much direct contact with sub-postmasters
though I would try to talk to them to explain a user error (if it was one) or that
there was a system problem, rather than just passing the call back to the
helpdesk, and they often seemed grateful for this contact.
209. I am asked what I think was/were the cause(s) of the problems experienced in
using the Horizon IT System. I do not know and I wish I did. Throughout my
time in SSC, I spent a long time looking for and hypothesising possible causes
of problems. Some of the bugs discussed here, and others, almost certainly
caused losses or gains at some branches on some occasions. But I never found
a bug within Horizon which would cause continuing losses, week after week, at
the same branch while leaving no sign of the problem in the branch accounts or
Page 61 of 63
WITNO0170100
WITNO0170100
the Horizon reconciliation reports. I do not know whether discrepancies reported
to Post Office via BIM reports were always resolved with branches, nor whether
SSC/MSU always reported these in a way that made it clear that there had been
an impact on a branch.
210. Finally, I think it was extremely hard for sub-postmasters to check that the data
entered on the system always fully matched what had actually taken place at
the branch and it seems likely to me that some of the problems were down to
transposed digits and other inadvertent errors, which could then not be traced
at the end of the week or month.
211. I am asked whether with hindsight there are changes to the support service
provided by Fujitsu would have improved the assistance provided to sub-
postmasters. I think that there was a problem with the way in which problems
were investigated as separate incidents without greater oversight of the wider
position. As I have said, when I was given something to investigate my scope
and the range of evidence available to me was limited. There was a division
between the technical operation of the Horizon System and the operation of the
business of running branches which meant that it was not apparent to me that
anyone had the oversight and control to investigate across that. An alternative
approach might have been to create a further role for people knowledgeable
about both the Horizon system and Post Office business who could go to
branches to help investigate any problem, whether suspected to be a fault in the
system or with business processes.
Page 62 of 63
WITNO0170100
WITNO0170100
212. A point of frustration with the system, was that the users, namely the sub-
postmasters, were not our clients and there was a practical limit as to the extent
to which we could work together with them to investigate problems.
Statement of Truth
Page 63 of 63