FUJ00083835
FUJ00083835
Filed on behalf of the: Defendant
Witness: Steven Paul Parker
Statement No.: First
Exhibits: SPP1
Date Made: 16 November 2018
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
IN THE HIGH COURT OF JUSTICE
QUEEN'S BENCH DIVISION
ROYAL COURTS OF JUSTICE
BETWEEN:
ALAN BATES & OTHERS
Claimant
AND
POST OFFICE LIMITED
Defendant
WITNESS STATEMENT OF STEPHEN PAUL PARKER
I, STEPHEN PAUL PARKER o}
SAY as follows:
1. I am employed by Fujitsu Services Limited (Fujitsu) as Head of Post Office
Application Support.
2. I am authorised to make this statement on behalf of Post Office Limited (Post
Office), the Defendant in these proceedings, in relation to the Horizon Issues trial
listed for March 2019.
3. The facts set out in this statement are within my own knowledge, or if they are
outside my knowledge, I have explained the source of my information or belief.
4. In this statement I refer to copy documents attached and marked Exhibit SPP1.
BACKGROUND
5. I started working on the Royal Mail Group Account, later called the Post Office
Account, in July 1997, before the introduction of Horizon. I have continued to
provide support to the Post Office Account in my various roles at Fujitsu
throughout the whole of Horizon's life.
6. Prior to my work on the Post Office account, I held a numberof roles within the IT
industry, including an in-house operations and support role for critical online
systems, support consultancy services and design and development role in an
AC_152871086_1 1
FUJ00083835
FUJ00083835
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
End User Technology group in relation to Microsoft tools for internal and external
customers.
7. I began working in July 1997 as a support consultant and deputy manager within
the Software Support Centre (SSC) for the Royal Mail Group Account, providing
third line application support for the Horizon application. Within this role I was the
lead designer and part of development team for the internal website providing
support knowledge (KEL) database, technical documentation management ad
operational change control. I also assisted the SSC manager in the provision of
the support service and operational management Although I did not have the
formal title, I acted as the deputy manager to the SSC as a whole.
7A Between December 2009 and March 2010 I was a full time Problem Manager /
Operational Manager of the SSC, responsible for the management of incidents
through the whole support process.
7.2 In March 2010 I became the Manager of the SSC and was responsible for the
provision of third line application support to the Post Office Account, including the
management of the staff working on the account. The SSC subsequently
expanded into a shared service, providing support services to a number of Fujitsu
customers, the largest of which is still the Post Office Horizon system. As head
of this unit I am currently responsible for strategic support, managing 25 — 40
staff.
RICHARD ROLL'S STATEMENT
8. I have been asked to comment on the witness statement of Mr Roll dated 11 July
2016 put forward by the Claimants in relation to the Horizon Issues trial in March
2019. I worked with Mr Roll while he was employed by Fujitsu in the SSC (I
understand from Fujitsu's HR department that this was from 5 March 2001 until 17
September 2004). As explained above, although I did not have the formal title, I
acted as deputy manager while Mr Roll was there.
9. The Horizon system was rolled out across the Post Office network between 1999
and 2000. In 2010 there was a migration from the system commonly referred to
as “Legacy Horizon" to an online version ("HNG-X" or “Horizon Online").
Therefore, when Mr Roll refers to the Horizon system he is referring to Legacy
Horizon and where I respond to an assertion made by Mr Roll I am also referring
to Legacy Horizon unless otherwise stated. Much of my statement also applies to
Horizon Online, however.
10. I found Mr Roll to be a conscientious worker and provided him with a personal
reference for a position that he applied for after leaving Fujitsu (it was a personal
AC_152871086_1 2
FUJ00083835
FUJ00083835
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
reference because Fujitsu does not allow staff to provide references on behalf of
the company).
11. In his statement Mr Roll suggests that there were frequent instances of software
problems in Horizon that had an impact on branch transaction data and which
Fujitsu resolved "remotely" (i.e. not in a branch), not merely by changing the
software but also by frequently changing branch transaction data (by injecting
new transaction data and by editing or deleting existing transaction data), without
informing branches that such actions were being taken. As I explain below, those
suggestions are incorrect and Mr Roll's account of Fujitsu's actions and powers is
inaccurate and misleading.
12. In order to explain why, it is necessary to have a proper understanding of what
Fujitsu was able to do, what the effects of its actions were and the extent to which
such actions had any effect on transaction data in branches. This is described
further below, but I would like to make some general responses immediately.
13. It is correct that the SSC had (and has) the ability to view data in branches and
other sources such as data centres remotely (in read-only mode): that is to be
expected given their support role which is described in paragraph 26 below.
14. Post Office did (and does) not have the same ability— what it could (and can) do
is view copies of transaction data as replicated from Horizon to a variety of Post
Office systems. In Horizon Online, facilities have been added which allow Post
Office to remotely examine branch data held in the Branch Database @RDB) in
read-only mode for a variety of business reasons, such as monitoring levels of
cash that are in branches based on the data entered onto Horizon by branch staff.
15. It is also correct that issues sometimes arose which necessitated changes to the
Horizon software. However, this was not something that Mr Roll played any
significant part in, as I describe in paragraphs 34 and 35 below.
16. It was (and is) theoretically possible for there to be a software problem which
could cause a financial impact in branch accounts, but this was (and is) extremely
rare and Horizon's countermeasures were (and are) very likely to pick such
matters up. In my experience, these problems have always represented a very
small proportion of the issues which led to software changes and a very small
proportion of the overall issues dealt with by the SSC.
17. On the very rare occasion that a software problem which could cause a financial
impact in branch accounts arose, it would be investigated and resolved and
Fujitsu would determine its impact on the Horizon estate and inform Post Office of
any financial impact on branches so that they could be resolved.
AC_152871086_1 3
18.
19.
20.
20.1
20.2
20.3
21.
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
In Legacy Horizon it was possible for the data in a particular counter in a branch
to become inconsistent with replicated copies, and Mr Roll appears to be
describing this in paragraph 17 of his statement. In that situation there could be
remote management by Fujitsu to correct the problem, but branch transaction
data was not changed in any way. As explained in paragraph 55 below, the work-
round involved replicating the correct data from another counter in the affected
branch or from the data centre copy.
The suggestion that Fujitsu edited or deleted transaction data is not correct. In
Legacy Horizon it was not possible to delete or edit messages that had been
committed to the message store. See the explanation of my colleague Torstein
Godeseth in paragraph 37 of his witness statement dated 27 September 2018
which reflects my understanding and experience of Horizon’s functionality.
In para 18 of his statement, Mr Roll also suggests that in Legacy Horizon he and
others in the SSC could "insert transactions and transfer money without the sub-
postmaster knowing". However:
No Fujitsu personnel have ever had the ability to "transfer money’ out of Horizon
into, for example, an individual's account
Some members of the SSC were (and some remain) able to insert transaction
data. SSC access privilege gave the ability to inject transactions, but appropriate
change controls were in place and no such insertion would have happened
without complying with those controls.
I should make it clear that, in this witness statement, I am concentrating on what
the support process is designed and able to do and not any malicious misuse of
these facilities. Malicious misuse makes most things possible, as with any other
IT system, however, Horizon had a number of checks and security settings in
place that made it very difficult to carry out malicious misuse. In any event such
misuse would have been discovered by consistency checks or colleagues (all
access was controlled and audited) and would have resulted in instant dismissal.
But even a malicious user would not have been able to “transfer money”.
As there is no way in which money could be taken from a branch and moved
anywhere else (for example into the employee's bank account), it follows that
there was no motive for anyone to do this. It is not clear to me why anyone at
Fujitsu would insert transaction data into a branch's accounts without there being
a legitimate reason for doing so. Furthermore:
Any transaction that was inserted would immediately cause a discrepancy to
arise in the branch's accounts. For example, if a transaction were to be inserted
AC_152871086_1 4
FUJ00083835
FUJ00083835
22.
23.
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
which stated that £1,000 of stamps had been boughtby a customer who paid
cash, that would immediately cause a reduction in stock levels of stamps in that
branch and the branch would have £1,000 less in cash than Horizon expected it
to have.
In other words, although a transaction could be inserted, it would immediately
become apparent that this had been done and ultimately it would not benefitany
member of staff to behave in this way.
It is correct that the "remote access" described above could have been carried out
without the permission of a Subpostmaster. However, any additional transactions
inserted remotely would be identifiable as such from the transaction logs that are
available to Subpostmasters from Horizon.
I will now describe the structure of the Horizon support teams during the period
when Mr Roll was employed by Fujitsu before responding to the specific
suggestions that Mr Roll makes.
The Horizon Support Teams
24.
25.
26.
26.1
There were four lines of support for Horizon while Mr Roll was employed by
Fujitsu and they are described in paragraph 26 below. There are still four lines of
support for Horizon today, albeit that some names have changed and some
responsibilities have moved around teams.
It is common within the industry to have a multi-level support model. Generally,
as you move up through the levels of support the cost of the staff providing the
service increases because they are more qualified. Having said that, there is
often overlap of skills between adjacent lines of support and while a team may be
responsible for a particular level of support, staff within that team can have skills
which allow them to perform a role that is more usually performed by the next
level of support.
The four lines of support for Horizon while Mr Roll was employed were as
follows:-
1* line: The 1“ line involved several different elements:
26.1.1 the Horizon Service Desk (HsbD)' was a helpdesk operated by Fujitsu
that branches could contact with issues relating to the Horizon
application or the hardware provided in branch by Fujitsu to run the
' The HSD has also been known as the Horizon System Helpdesk and the Horizon
Incident Team.
AC_152871086_1 5
FUJ00083835
FUJ00083835
FUJ00083835
FUJ00083835
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
Horizon application. I estimate that there were around 80 members of
the HSD while Mr Roll was employed, who:-
(a) dealt with straightforward enquiries such as password issues and
scheduling engineers to attend branches to investigate reports of
hardware issues;
(b) monitored the live estate and took corrective actions defined in
knowledge documents (this role was fulfilled by two units: the
System Management Centre (SMC) for data centre events; and
the Counter Eventing Team for Post Office counter events. The
corrective actions did not involve any changes to any branch
transaction data); and
(c) referred other issues to the 2 line support function.
26.1.2 there was also a 1°‘ line Communications Management Team operated
by Fujitsu which specifically focused on communication incidents; and
26.1.3 Post Office also operated a 1“ line helpdesk for operational issues
called the National Business Support Centre (NBSC).
If a branch required assistance to attempt to determine the cause of a
discrepancy they would contact NBSC in the first instance. Discrepancies are not
unusual in a retail system. They indicate a difference between the operator's
declaration of cash and stock on hand and the systems calculation and as such
are a business operation issue. However, it was not always possible for NBSC to
identify the cause of a discrepancy. For example, a user may enter a deposit of
£100 into a customer's bank account on Horizon but rather than taking £100 from
the customer, they may make a mistake and give the customer £100 as if it had
been a withdrawal. In that scenario, NBSC would not have been be able to
identify the cause of a discrepancy. Clearly, NBSC is also unable to assist when
losses have been caused by theft.
If NBSC were unable to identify the cause of a discrepancy they would often fall
back on a default statement along the lines of “this looks like a software issue" so
that the SSC would investigate it. However, Mr Roll's statement that "[i/f an error
was referred to us then it was extremely unlikely to be due to a mistake made by
a postmaster" is not correct. The vast majority of discrepancies investigated by
the SSC as pseudo “software issues” were (and are) not caused by software
issues.
AC_152871086_1 6
26.2
26.3
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
2" Iine: 2" line support was provided by senior members of the HSD and SMC
and junior members of the SSC (as explained below, the SSC also fulfilled a 30
line support function). The 2" line support function mainly involved staff
searching knowledge articles based on the descriptions of issues reported by
branches, gathering evidence and applying simple, well-defined work-arounds
(often on the phone). An example of this would be resetting passwords.
3" line: the SSC also provided 3” line support. The staff that provided 3" line
support had a detailed knowledge of the Horizon application based on
documentation and some inspection of source code. They:-
26.3.1 designed, tested and documented work rounds for the 4! and 2™ lines
of support;
26.3.2 applied analytical skills to the symptoms and evidence gathered by the
14°! and 2™ line functions and undertook in-depth investigation into
incidents (incidents are the basic unit of work for the support team and
come from helpdesk calls and other Horizon support teams);
26.3.3. undertook complex configuration (configuration items can be used to
alter the behaviour of the application) and data fixes which might have
required the generation of special tooling;
26.3.4 designed, wrote and documented new support tools;
26.3.5 undertook source code examination, complex diagnosis and
documentation (including methods to recreate faults) of new application
problems before sending them to the 4" line support group for root
cause software fix; and
26.3.6 provided technical support to other internal Fujitsu teams working on
Horizon.”
The 3“ line support function used a system called Peak (until 2003 it was called
PINICL) to log and manage incidents passed to them which were suspected to be
faults. It also maintained a Known Error Log (KEL) which describes the
symptoms of problems with some analysis of causes, (potential) solutions to the
problems and workarounds that might be needed before a permanent solution
2 An example of this which applies to Horizon Online is support to the Management
Support Unit (MSU) who are responsible for the Reconciliation Process documented in
SVM/SDM/PRO/0020 (Reconciliation and Incident Management Joint Working
Document). The reconciliation process also applied to Legacy Horizon.
AC_152871086_1 7
FUJ00083835
FUJ00083835
26.4
27.
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
can be implemented. The Peak and KEL systems are still in use today and are
described further in paragraph 62 onwards below.
4" line: 4" line support staff had an intimate knowledge of narrow areas of the
system and were (and are) ultimately responsible for the production of permanent
fixes to repair the root cause of an incident or problem in the live application.
They had knowledge of computer languages which they used to amend source
code to fix problem in the live application code. There was often overlap between
4" line and developers, who added new features into the application.
The structure of the support for Horizon is broadly the same today. One of the
main changes is that the HSD function is no longer operated by Fujitsu (it has
been operated by Atos since June 2014).
Further context — call volumes and transfers
28.
29.
30.
34
Between 1 January 2001 and 31 December 2004 (the years Mr Roll worked for
Fujitsu), the SSC received a total of 27,005 calls, meaning that on average 563
calls per month were dealt with over this 4 year period. This is shown by a
spreadsheet prepared by a team in the SSC which appears at Exhibit SPP1 (Tab
"RRP_Live_Peaks_Into_SSC"). Where (as here) I analyse data in this statement,
the analysis is mine.
Transferred calls (i.e. those not resolved by the SSC) are of interest. A very small
proportion of calls transferred to 4” line support would have concerned software
errors requiring resolution, so it would be interesting to know the number of calls
transferred to them. However, while the SSC have records of the volume of
transferred calls, we do not retain records of where they are transferred to and it
is not the case that all of these would have been transferred to 4" line support.
For example, incidents would often arrive at SSC from internal teams for routing
back to helpdesks.
As evidenced by the data in Tab "RRP Live Peaks Out of SSC" of Exhibit SPP1,
an average of 78 calls per month (14%) were transferred to teams outside SSC,
for example, to 2" and 4" line support. This indicates that only a small volume of
calls received and escalated to the SSC related to software errors requiring
resolution.
In terms of calls to the HSD, Fujitsu only holds records of the volume of calls
made to HSD from December 2008 onwards. I understand from my colleague
Sandie Bothick (Fujitsu’s Service Delivery Manager) that the HSD received
13,225 calls in December 2008 and 13,005 in January 2009. The witness
statement of Angele Van Den Bogerd provides that Post Office’s NBSC call
AC_152871086_1 8
FUJ00083835
FUJ00083835
32.
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
volume data shows an average of 1,096 calls per day were received in the period
2015 to 2018. While these figures relate to different periods, they indicate that
only a tiny proportion of issues reported to 4" ine support helplines were
escalated to SSC.
From the SSC, only a tiny proportion of incidents were escalated to the 4” fine
support team. It follows that only a tiny fraction of incidents raised actually needed
to be looked at by the only team who might potentially effect changes in software.
The SSC and Mr Roll's role within it
33.
34.
35.
36.
I agree with Mr Roll's recollection that there were around 30 individuals working
on the 6" floor in Bracknell (i.e. in the SSC) at any one time while he was
employed.
As noted above, the SSC team provided both 2° and 3” line support. As with
any mix of people, there are (and were) various levels of talent within SSC. Mr
Roll was primarily used in Operational Business Change (OBC), which involved
supporting the engineers who were opening and closing branches and increasing
and decreasing the number of counters in branches. Mr Roll would also have
been regularly correcting the application environment after engineers had
replaced failed counter hardware and clearing temporary files to increase disk
space. This could fairly be described as 2" line work and it was done by the SSC
because it required a higher level of access to the system than other support
teams had.
Some members of the 3° line support group identified the need for software fixes
via source code examination and would pass this on to the 4" line team for a
code fix to be written. Mr Roll did not play any significant part in this and was not
involved in any extensive source code examination. An application code fix would
not be written by anyone in the 3° line team and he was not involved in the
provision of 4" line support
I disagree with Mr Roll's suggestion that much of the work being carried out by the
SSC while he was employed could be described "as "fire fighting" coding
problems in the Horizon system." There were times when the SSC was very
busy, for example, networking problems causing application issues across the
whole estate and data centre outages. But there were only rare circumstances
where a coding issue had an estate wide impactand, in those instances, Mr Roll
would have been involved executing avoidance actions to mitigate impact to the
estate (i.e. following established work-arounds) rather than working on the root
cause.
AC_152871086_1 9
FUJ00083835
FUJ00083835
37.
38.
39.
40.
40.1
40.2
40.3
40.4
40.5
40.6
41.
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
The SSC had (and has) access to view, but not amend, source code. Senior
members of the team would have looked at it from time to time to confirm exactly
how the Horizon application would process a given input and what the outputs
would be when investigating specific issues or general education on how the
system works. However, the access was rarely used. Moreover, Mr Roll was not
working at a level where he would be required to review code. His role in the
SSC was predominantly following work around processes designed by other
people and making configuration changes.
In support of the statements above relating to SSC workload I have undertaken
some analysis of the work carried out by SSC between 1 January 2001 and 31
December 2004 (as stated above Mr Roll was employed from 5 March 2001 until
17 September 2004). My colleague John Simpkins provided the summary data
from the Peak system, which I analysed.
When an incident is resolved, the SSC team member (or technician as they are
sometimes called) types a summary of the incident (known as a Final Response)
and allocates a response code to the incident in order to classify it. While
guidance is provided on when to use each response code (see paragraph 64.5
below), allocation is the subjective view of the technician closing the incident and
there is no re-examination of the response codes later to ensure consistency.
With that in mind, the final response codes that were allocated to incidents (i.e.
Peaks) reported to SSC between 1 January 2010 and 31 December 2004 were
as follows:-
known issue / work around - 35.3%;
admin - 27.5%
reconciliation - 15.7%;
potential user error - 10.9%;
potential software error - 8.3%; and
hardware error - 1.2%.
It should be noted that:
a major part of 1“ line's raison d'étre is to deal with user error and therefore the
percentage of issues attributable to user error would be much higher at 1" line;
AC_152871086_1 10
FUJ00083835
FUJ00083835
41.2
41.3
414
42.
42.1
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
very few hardware incidents reached the SSC because they were the preserve of
the HSD (i.e. they were relatively easy to spot and therefore filtered out by 4 ine
support);
8.3% of calls to the SSC (2252) are attributed to potential software errors over
these four years. This includes duplicates and does not provide any clarity on the
significance of the error corrected. Many software errors, particularly in a new
product, were insignificant, such as correcting capitalisation in printed output. I
cannot be more precise without examining each Peak and even then it might not
be possible to determine how a Peak should have been properly classified
(Peaks are essentially notes made by Fujitsu personnel to chart the progress
made in resolving an issue and these notes can vary in fullness and clarity); and
Classifying an incident as a "potential" software error does not necessarily mean
that there was a software error and, even if there was, it does not mean that the
error was one that could have caused a financial impact in a branch's accounts (a
large proportion of these would be errors in numerous data centre resident
systems that the Subpostmaster never sees — errors were often as trivial as the
use of “Kg” instead of “kg” on receipts). As stated in paragraph 16 above, such
errors were extremely rare. They were all resolved (resolutions include a source
code fix, a configuration fix or a workaround).
Mr Roll's suggestion in paragraph 10 that software issues in Horizon "routinely"
caused discrepancies in branch accounts is misleading. In the vast majority of
cases such an occurrence would cause a receipts and payments (R&P) mismatch
that would be flagged by the branch system as part of the balancing process (the
Horizon system carries out selfconsistency checks which generate alerts in the
event of a receipts and payments mismatch that are picked up by SMC and
incidents raised for the SSC) and appear on MSU reporting. These would then
be investigated and resolved by the SSC.
Since the introduction of Horizon in 1999 there have been 7335 live incidents
which refer to “Payments and Receipt mismatch’ (i.e. incidents recorded against
components of the system providing Horizon service to Post office rather than
incidents raised against test systems). This figure has been obtained using a
textual search across all incidents where the title or one of the incident updates
contains all of the words “receipts”, “payments”, “mismatch”. It should be noted
that this is not 735 unique incidents; there will be a lot of duplicates with the same
root cause. The only way to determine how many unique incidents there were
would be to manually review all of the incidents. All of them were resolved.
AC_152871086_1 11
FUJ00083835
FUJ00083835
42.2
42.3
43,
44,
45.
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
These incidents are reported as a result of self-consistency checks carried out by
Horizon. It should be noted that a R&P mismatch is not only caused by a
software error. It can also be caused by incorrect product reference
(configuration) data
Receipts and payments mismatches happened more often during the early life of
Horizon (see Tab “All RnP by RCode and Date” of Exhibit SPP1). My analysis of
that data shows that there were around 8.6 such incidents per month on average
between 1 January 2001 and 31 December 2004 (417 out of a total of 27,005
incidents into SSC or 1.5% of SSC incidents during that period).
Mr Roll refers to a "perception...that the Service Level Agreements between Post
Office Ltd and Fujitsu involved financial penalties payable by Fujitsu to Post
Office" (paragraph 12). I am aware that there were Service Level Agreements for
issues such as stuck transactions (Fujitsu had 10 days to retrieve transactions
that had not replicated from a counter). It is quite normal for contracts such as
the one between Fujitsu and Post Office relating to Horizon to have such
agreements. The same level of diligence was (and is) applied to all incidents,
whether an SLA was relevant of not. The possibility of financial penalties was
never a factor for the SSC.
I do not understand what Mr Roll means when he says that “any discrepancy in
the post office accounts had to be resolved speedily" (paragraph 12). There was
(and is) a process run by the Management Support Unit (MSU) which involves
examination of various system reporting and may result in Business Incident
Management Service (BIMS) entries going to Post Office. An incident may also
be raised by MSU with the SSC to provide support to the MSU in resolution of the
BIMS. These are subject to Service Level Agreements and Mr Roll may be
referring to this process. However, if Mr Roll is suggesting that Fujitsu routinely
rushed out fixes or workarounds due to SLA time pressure, that is not the case.
Fixes would be expedited based on service impact. It would be quite wrong to
suggest that they were not done properly because of any SLAs.
It is correct that there are a limited number of opportunities to release software
updates and that these releases could take 6 weeks or more to be released to the
live service (Mr Roll’s statement, paragraph 13). These longer timescales would
be employed for non-urgent updates wrapped up into a consistent set for
deployment. On those rare occasions when a problem has an impact on the
financial integrity of the system a “hot fix" would be deployed which involved
much shorter timescales. I would expect a timescale measured in days not weeks
to deploy a hot-fix. Mr Roll also states that a bug could reappear several weeks
AC_152871086_1 12
FUJ00083835
FUJ00083835
46.
47.
47.1
47.2
47.3
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
after a coding fix had been released due to software issues. I am aware of only
one or two cases where a fix regressed in my time at Fujitsu.
In paragraph 14 Mr Roll states, "/ would reiterate that the main recurring issues
were software issues." It is a symptom of working within a software support team
that the majority of issues that come in have been attributed to a software issue
by, for example, a lower line of support. This can lead to a mind set of “look at all
these Horizon errors”, but what this indicates to me is that the previous levels of
support are functioning correctly, removing the majority of other causes (user /
hardware problems). It does not indicate that the majority of Horizon errors could
be attributed to software.
Mr Roll states (paragraph 7) that "[sjoftware programs were written by us to strip-
out irrelevant data, to enable us to more easily locate the error." The support
tools are used to filter information and present information to technicians in ways
that make the support process easier. I am aware of two support tools (also
known as software programs) that were written while Mr Roll was employed by
Fujitsu:-
the Smiley support tool written by my colleague John Simpkins, which
amalgamates information from various sources (e.g. databases) into a single
view pertinent to a particular support task and provides a unified interface to run
various tools to achieve a single support outcome; and
another tool (I cannot remember its name) written by my colleague Richard
Coleman whose function was to retrieve messages (i.e. data) from the
correspondence server to local text files for examination and which was
eventually subsumed into the Smiley support tool.
Neither of these tools changed the underlying data in a branch's accounts.
The work carried out by Mr Roll
48.
48.1
48.2
Mr Roll states that he would investigate financial discrepancies that had arisen in
branches by "work[ing] sequentially through all transactions over the relevant
period, and also work[ing] through thousands of lines of computer coding’
(paragraph 7).
It is not the role of the SSC to routinely investigate discrepancies.
In very rare circumstances a discrepancy could be caused by a software issue
and in those circumstances it might be necessary for the 4" line support function
to work through thousands of lines of computer coding to determine the root
AC_152871086_1 13
FUJ00083835
FUJ00083835
49.
50.
51.
52.
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
cause. However, Mr Roll would not have worked through thousands of lines of
computer coding to investigate a discrepancy in a branch: he did not work in the
4" line.
Mr Roll further states that if SSC was "unable to find the cause of the discrepancy
then this was reported up the chain and it was assumed that the postmaster was
to blame" (paragraph 10). That is not my experience: it is a simple truth of support
that the majority of issues reported in a system are attributable to user action or
user misunderstanding of system functionality. Hence, someone working in a
support environment analysing a new issue would examine the possibilities of
user error as a first hypothesis but any final conclusion is only generated based
on the evidence. Where the evidence does not support aconclusion that there is
a problem with Horizon, the SSC feeds the existent factual data back to Post
Office and might say something along the lines of "all indications are that the
branch has made a mistake", but Fujitsu neither attributes "blame" or agrees the
final conclusion with the Postmaster
When an incident is resolved, the SSC team member (or technician as they are
sometimes called) types a summary of the incident (known as a Final Response)
and allocates a response code to the incident in order to classify it. While
guidance provided on when to use each response code, allocation is the
subjective view of the technician closing the incident and there is no re-
examination of the response codes later to ensure consistency.
The Peaks that Mr Roll works on while employed by Fujitsu indicate that he dealt
with 915 incidents (see Tab "Final Responses" of Exhibit SPP1). To give some
clarity on these incidents I have analysed their final response codes that were
allocated to them by Mr Roll. He classified them as follows:-
Response code % allocated
by Mr Roll
Known issue / work around 61.9
Reconciliation 14.5
Admin 9.3
Potential software error 3.2
This supports my recollection that Mr Roll mainly followed work-arounds devised
by other people (61.9%) and that he was rarely involved in the detailed
examination of potential software errors (3.2%). As explained in paragraph 41.4
above ‘potential’ software errors do not necessarily mean software errors, let
alone software errors resulting in discrepancies to branch accounts.
AC_152871086_1 14
FUJ00083835
FUJ00083835
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
Remote access
53.
53.1
53.2
53.3
54.
55.
55.1
55.2
55.3
I understand from Post Office's solicitors that the term "remote access" has been
used to describe Fujitsu and Post Office carrying out the following actions when
not physically present in a branch:-
accessing branch data in read-only mode;
inserting new transaction data; and
editing or deleting existing transaction data.
As noted above, it is correct that Fujitsu had (and has) the ability to view data in
branches remotely given their support role.
Mr Roll claims that "[djuring the course of resolving the software issues, we would
frequently access a Post Office counter IT system remotely" (paragraph 15). Mr
Roll gives the example of ("when a binary bit would "flip", thus a "1" became a
"O") =
This probably relates to a condition known as a CRC (Cyclic Redundancy Check)
error which would happen when a hard drive became faulty at a branch counter.
Although this is a hardware problem, remedial action was needed by the SSC to
resolve. To clarify, this process did not involve changing any transaction data.
As explained by Mr Godeseth in paragraph 35 of his statement, in Legacy
Horizon:-
55.2.1 all counter data was held in a bespoke message store in each branch?
and the data was replicated to all counter positions in the branch and
from the branch to the data centres where it was held in
correspondence server message stores;
55.2.2 any data inserted into the message store at the data centre (for
example reference data or authorisations for banking transactions)
would be replicated back to the branch counters; and
55.2.3. selected data was extracted from the correspondence servers to
update Post Office's back end systems.
If one of the sets of data on a branch counter became corrupted it would
generate an event that would be picked up by the SMC and/or reported to HSD
3A message means data and transaction data is asubset of the data in the message
store.
AC_152871086_1 15
FUJ00083835
FUJ00083835
55.4
55.5
55.6
56.
56.1
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
by the branch (an incident reporting a “CRC Error’). There were a total of 629
CRC errors over the life of Legacy Horizon (see Tab "All CRC by Date" of Exhibit
SPP1).
The issue would be reported to the SSC, who would delete the entire set of data
on that counter and replace it with a copy of the data from one of the other
sources that had not been corrupted. While this process involves deleting and
replacing a set of data, no new data is produced; all that happens is that the
replicated data is used to replace the data that has become corrupted from
another counter in the branch. It would have been necessary forthe SSC to
inform a branch before carrying out this task because it is likely that any attempt
to use that counter would fail while the process was being carried out. In Mr Rolls
capacity as an OBC specialist, he would have been involved in this type of
activity.
Alternatively, Mr Roll's reference to a binary code flipping may relate to a
configuration item (an item that can change the behaviour of the application)
which can become locked in the wrong binary setting (1, 0) One example of
which would be a stock unit lock which, in the wrong state, would prevent
updates to stock units within a branch. This issue was corrected by a member of
the SSC accessing Horizon remotely, but it did not involve accessing or editing
transaction data in any way or re-creating databases.
I cannot think of any other examples of incidents that Mr Roll may be referring to
in paragraph 15.
Mr Roll claims that “some errors were corrected remotely without the sub-
postmaster being aware" (paragraph 16).
It may be that Mr Roll is referring to issues relating to the end of day concept in
Legacy Horizon. Essentially there was a cut-off point for transactions every day
at 7:00pm and each counter had to write an end of day message to the branch's
master counter to enable the master counter to write a branch end of day
message, which would then trigger the data centre to harvest messages
(including details of transactions) to Post Office's back end systems.
Occasionally a counter in a branch would fail to write an end of day message and
there was a process for correcting this. The issue would be reported to SSC by
way of an incident (either as a result of a call to HSD or sometimes Fujitsu could
spot issues via system events).
AC_152871086_1 16
FUJ00083835
FUJ00083835
56.2
57.
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
In lay terms, SSC would force the counter to generate a report based on the data
already in the counter. A message injected in this way would go into the audit
trail. This would not alter the branch's transaction data.
Mr Roll also states that, there were some errors where it was necessary to
“download and correct the data and prepare it for uploading back on to the post
office computer, then call the postmaster to inform him that there was problem
and that we needed two or three minutes to correct it' (paragraph 17).
57.1 _ It is not clear what errors Mr Roll is referring to or how he says they
were corrected. The issue referred to could be another instance
where the work round of re-creating transaction data from a
replicated copy was required as described in paragraphs 55.3 and
55.4 above.
SUMMARY
58.
In summary, the suggestion that Fujitsu would manipulate a branch's transaction
data in a way which was detrimental to a particular Postmaster and undetectable
is wrong.
58.1 All support action taken by Fujitsu is directed to ensuring that legitimate
transactions entered by Subpostmasters are correctly processed by the Horizon
application and correctly passed to other POL systems as appropriate.
58.2 Any financial corrections required are communicated by Fujitsu to Post Office for
execution or approval.
58.3 Any support intervention that requires the insertion of a transaction is identifiable
in the audit trail.
58.4 There is no financial incentive for a support technician to circumvent the controls
within the system.
58.5 There are strong controls in place to prevent intervention by support staff with
malicious intent or misguided attempts at financial gain
59. The statement that issues with coding in the Horizon system were extensive and
impacted branch finances is incorrect for the reasons stated above.
KELS AND PEAKS
60. I have been asked by Post Office's solicitors to provide some more information on
the two toolsets used to support the Horizon system, the KEL knowledge base
AC_152871086_1 17
FUJ00083835
FUJ00083835
FUJ00083835
FUJ00083835
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
(AKA KEL) and the incident management system (PINICL / Peak). This
information is applicable to both Legacy Horizon and Horizon Online except
where explicitly stated otherwise.
61. Fujitsu use a custom solution, developed and administrated by the SSC, which
allows us to record support knowledge into a structure known as a KEL (Known
Error Log). KELs record support knowledge which is intended to assist staff in
the support and understanding of the Horizon system. KELs do not contain the
history of an incident (see my analysis of the Peak system below). KELs are
generated for a number of reasons, for example: to explain system behaviour or
messages that originate from central and counter systems, to record symptoms
and outcomes from incidents referred by help desks or identified as a result of
Fujitsu's reconciliation processes; and to record information on issues seen in test
environments (resolved before the feature is passed on to users).
61.1. The acronym KEL is a misnomer inherited from a previous system. KEL entries
are support knowledge entries and do not have a one to one relationship with
errors on the system. There are a lot more general supporting knowledge KELs
than KELs relating to specific errors.
61.2 Guidelines for the generation and use of KELs are documented in
SVM/SDM/PRO/0875 (End to End Support Strategy) section 11 Knowledge Base
Maintenance. Although some aspects of this document need revision due to
changes in the support structure for Horizon, Section 11 is still fundamentally
correct in relation to Horizon.
61.3 KELs reflect community sourced knowledge to assist staff involved in the support
of the Horizon solution. There are no mandated rules for when a KEL should be
created other than a desire to make resolving a problem easier for all concerned.
Guidelines exist in SVM/SDM/PRO/0875. Any KELs created or updated are
referred to the SSC for approval to provide a basic check that the information at
the time of the change is valid.
61.4 KELs can be created by the senior people in HSD (2% line) to supplement their
own knowledge base (rather than taking an active role in the KEL process) and
the SMC monitoring team. The 1“ line helpdesk function do not create KELs
(HSD 1“ line used their own knowledge base).
61.5 AKEL is a living document that reflects support knowledge at a given point in
time. KELs are not designed to provide a history of a particular symptom or
support process and a particular KEL cannot be considered the definitive source
containing all possible information regarding the problem it addresses. It is up to
AC_152871086_1 18
FUJ00083835
FUJ00083835
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
the technician to check other potentially relevant KELs and information sources
(e.g. support guides) and use their analytical skills to determine the right course
of action to take in a given situation.
61.6 There have been historic requests to remove large numbers of KELs based on
date updated to reduce the number of search results that are returned to help
desks when naive search terms were used. The dates given in the KELs, while
an indicator of potentially irrelevant support information, are not precise. This led
to the concept of “deactivated” KELs (deactivated KELs are removed from the
default searches support people use although the user interface allows the user
to explicitly request a search to include deactivated KELs). At the time of writing,
there are 113 deactivated Legacy Horizon KELs and 1024 deactivated Horizon
Online KELs.
61.7 For most of its lifetime, there has been no fixed routine for the review of KELs. If
a technician recognises an inaccurate KEL as they analyse information they are
expected to update in order to improve the knowledge base. Approximately 2
years ago the KEL review forum was introduced. This forum meets weekly to
review KELs and update or deactivate as appropriate.
61.8 Before creating a KEL there is an expectation that the creator will search the
existing information to ensure that it is not duplicated. Because people express
information using different words, it is not possible for the system to perform such
acheck. Human fallibility and unique expression mean that duplicated
information is present in the KEL system.
61.9 Archiving: There is no requirement to keep historic support information. Once an
item is no longer relevant to current systems it can be removed without any
implications to the support of the system. KELs are deleted when they have no
value to the support of the current systems. This can happen at any time and is
carried out by the SSC who are the final arbiters of what information is currently
relevant. That does not mean that all current KELs are still relevant; it may be
that irrelevant KELs have not been deleted yet. At the time of writing there are
1,491 deleted KELs.
61.10 KEL entries have a single field to record an incident (Peak) reference. This is not
a record of all incidents the KEL is relevant to. Generally, it is used to record the
Peak that was being investigated when the KEL was created or updated. This is
a manual entry and is not checked by the system because a KEL may not have
an associated incident. There is no requirement to update the KEL when
information in it is re-used to provide guidance on a different incident.
AC_152871086_1 19
FUJ00083835
FUJ00083835
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
62. Peak (not an acronym) is browser-based software incident and problem
management system used by Post Office Account for all development, test and
support teams except the 1" line help desk. It enables details of the incident and
diagnostic progress to be captured in a searchable format and allows the tracking
of problems from detection through to resolution. Peak was developed in-house
by the SSC from the PinICL system it replaced in 2003. The system has been
customised and enhanced over its 15 years of operation andstill continues to be
developed to Post Office Account requirements today.
62.1 When Peak was implemented, data from PINICL was migrated to Peak.
62.2 One source of Peak incidents are the 1° line support teams (including the HSD
the NBSC helplines). Peak is also used to process incidents generated by other
support units, monitoring and testing teams.
62.3 The structure of a Peak enforces a workflow (it gives a process structure leading
to a defined outcome). As a result, the Peak system has also been used to
record and progress other items loosely associated with incident management.
For example, Release management process, reference data delivery process,
Post Office Data Gateway route definition process. So the Peak system contains
incidents which do not directly impact the Horizon service provided to the Post
Office counters (AKA “Live service”). These additional types of “incident” are
differentiated by the Peak type: L = Live service, R = Release management etc
62.4 For most of its life cycle a Peak is assigned to a particular support team anda
person within that team who is responsible for the next action om the incident. As
the incident is progressed by various members of the support community, they
add textual comments and supporting evidence to the Peak to document the
progress of the incident. These updates are date / time stamped and forma
record of the diagnostic and resolution process. Progress updates cannot be
deleted / amended by users once committed to Peak. A Peak may also be
transferred between teams and people as it progresses to final resolution.
62.5 _ A final response code (numeric) is applied to an incident when it has reached its
conclusion along with text that supports the response code. Response codes are
the subjective opinion of the person closing the incident and are not subject to
review but they remain the best way to classify the final outcome of an incident.
Response codes and their expected usage are documented in
SVM/SDM/PRO/0875 (End to End Support Strategy) in Section 9 Incident closure
categories. Although some aspects of this document need revision due to
changes in the support structure for Horizon, Section 9 is still fundamentally
correct and is relevant to Horizon (Legacy and Online).
AC_152871086_1 20
62.6
62.7
62.8
62.9
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
The Peak system has no archiving policy. Effective incident management
requires that you can track the current issues and those from the recentpast.
Data retention on Peak has been impacted by a lack of disk space, the primary
cause being large evidence files attached to Peaks. I am told by my colleague
who maintains the Peak system that-
62.6.1 encrypted evidence files are removed one month after the incident is
closed; and
62.6.2 due to disk space issues in the past it has been necessary to remove
evidence attached to older Peaks.
KEL references can be added to a Peak entry. There is no system check to
ensure a KEL has been added since KELs are not relevantto all incidents being
processed.
Incident priorities and appropriate usage are described in SVM/SDM/PRO/0875
(End to End Support Strategy) in Section 7 and SVM/SDM/PRO/0018 (Incident
Management Process). Incidents with a financial impacton branches are treated
as high priority.
Target times to resolve software incidents are described in SVM/SDM/PRO/0875
(End to End Support Strategy) in Section 8.
KEL ANALYSIS
MR COYNE'S KELS.
63.
64.
65.
I have been shown paragraph 5.114 of Mr Coyne’s report, in which he says that
he has analysed 5,114 KELs to determine the scope and impact of potential
Peaks. Mr Coyne explains that out of these 5,114 KELs, he believes he has found
163 that could be of "significant interest" and that he has referred to 76 of these in
his report.
Post Office's solicitors have reviewed Mr Coyne's report and have provided me
with a list of KELs that they have identified as being referred to in the report
(Coyne KELs). I do not know why there appears to be a difference between this
number and the number of 76 quoted by Mr Coyne.
It is not clear what Mr Coyne means by “significant interest’. \t may be that he
means that a KEL presents evidence of a bug, error or defect in Horizon that has
AC_152871086_1 21
FUJ00083835
FUJ00083835
FUJ00083835
FUJ00083835
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
caused a financial discrepancy in branch accounts. I refer to this as "financial
impact" as shorthand in this statement.
66. KELs are written by and for members of Fujitsu’s support groups (i.e. all of the
teams who support the Horizon solution for Fujitsu) who have a deep knowledge
of the design and operation of Horizon, and they are often expressed in a
shorthand way. I believe that it would be helpful to expiain the significance and
implications of the Coyne KELs. Annexed to this statement is a table which
contains the initial explanations that have been produced by a team from SSC at
my request in the time available.
DR WORDEN'S KELS
67. Dr Worden has selected a sample of 48 KELs. A list of these KELs was passed
to Fujitsu by Post Office's solicitors (the Worden KELs).
68. A table which explains the Worden KELs is annexed to this statement. As with
the Coyne KELs, given the limited time available to prepare this statement, the
initial explanations have been produced by the team referred to in paragraph 66 I
above.
STATEMENT OF TRUTH
I believe that the
statement are true.
Signed:
Date:
AC_152871086_1 22
Appendix 4
FUJ00083835
FUJ00083835
Filed on behalf of the: Defendant
Witness: Steven Paul Parker
Statement No.: First
Exhibits: SPP1
Date Made: 16 November 2018
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
KEL Mr Coyne's Report Fujitsu's Comments
ID Short-name I Paragraph Mr Coyne's Summary Response to Mr Coyne Financial impact on branch
(if applicable) No. (paraphrased) accounts
T.[__wrightm Payments 56-511 I This issue is reported as causing I This issue, referred to as the ‘Payment Mismatch’ by Mr Coyne, is dealt I The issue caused a receipts and
33145J Mismatch discrepancies showing at the I with substantively in the second witness statement of Torstein Olav
Horizon counter which disappeared
when branches followed certain
process steps. While a workaround
was established by KEL wright
331454, it is not clear how many
corrections were required to fix all
instances of this or event that all
instances were fixed.
Godeseth.
payments mismatch in the accounts
of affected branches, which was
detected by the monitoring of system
events by Fujitsu. Post Office were
informed and corrected the relevant
branch accounts.
AC_152871086_1
FUJ00083835
FUJ00083835
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
KEL Mr Coyne's Report Fujitsu's Comments
ID Short-name I Paragraph Mr Coyne's Summary Response to Mr Coyne Financial impact on branch
(if applicable) No. (paraphrased) accounts
2. I achat233J Unexplained 521-523 I Discrepancies between the branch I The cash planning system calculates the branch's cash levels figures from I No impact.
Discrepancies cash declarations and the amount I one day to the next. This was a timing issue due to figures from a previous
(cash received by SAP (the cash I day being used in association with other figures from the current day
declarations) management system). The KEL
states that this is not a user error or I This issue affected the figures being used by the back end cash planning
anything that can be corrected at I system and did not affect any branch accounts. The KEL exists to deal
branch level. This is therefore I with further complaints.
consistent with the problem being
due to the existence of a software I In the event that there is a discrepancy be tween the amount received by
bug. the Post Office cash centre in a cash pouch returned by a branch and the
amount the branch entered on Horizon as having been put in, this would
be identified by Post Office and a Transaction Correction for the
appropriate amount would be issued
AC_152871086_1 2
FUJ00083835
FUJ00083835
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
KEL
Mr Coyne's Report
Fujitsu's Comments
ID
Short-name
(if applicable)
Paragraph
No.
Mr Coyne's Summary
(paraphrased)
Response to Mr Coyne
Financial impact on branch
accounts
I
achat717T
Unexplained
Discrepancies
(cash
declarations)
5.22
There is an acknowledgment that
cash declaration discrepancies
could be due to an "Unknown
System Problem"
By way of context, this is a generic KEL relating to the handling of calls
concerning discrepancies. There are a number of possible causes for
unexplained discrepancies between the cash declaration and figures on
SAP, including
‘+ Subpostmaster has made an incorrect declaration
* Transactions as recorded on the system do not match what actually
happened at the branch
‘+ Outstanding recovery
© Withdrawn products
+ Known system problem (these should be monitored for or be easy to
spot from events etc.)
‘+ Unknown system problem
This term “unknown system problem” is a term that the KEL creator would
have used to indicate that the issue may have been caused by a new
(previously undetected) defect, that would follow the normal diagnosis
process and will then become a known system problem once fixed.
The relevant Peak (PC.0202239) relates to an investigation surrounding
the possible explanation of a £240 discrepancy. The investigation
concluded that the discrepancy was likely to have been caused by human
error and it appears that the Postmaster accepted this conclusion.
None.
AC_152871086_1
FUJ00083835
FUJ00083835
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
KEL Mr Coyne's Report Fujitsu's Comments
ID Short-name I Paragraph Mr Coyne's Summary Response to Mr Coyne Financial impact on branch
(if applicable) No. (paraphrased) accounts
4 I achaé2iP Unexplained 5.23 Evidence of cash declaration This issue, also referred to as the ‘Dalmellington’ issue is dealt with This issue caused a discrepancy in
Discrepancies discrepancies arising from clerks substantively in the second witness statement of Torstein Olav Godeseth. I the Subpostmaster's outreach branch
(duplicate Rem duplicating Rem In transactions as__I Fujitsu identified the branches affected by this issue and gave the which was easy to identify from the
In) a result of wrong messages being _I information to Post Office to resolve the issue with affected branches. transaction logs available through
presented on the Horizon counter Horizon and the fact that separate
screen. receipts were printed for each
transaction. Post Office issued
Transaction Corrections or advised
Subpostmasters how to take
corrective action to remove the
discrepancies.
AC_152871086_1 4
FUJ00083835
FUJ00083835
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
KEL Mr Coyne's Report Fujitsu's Comments
ID Short-name I Paragraph Mr Coyne's Summary Response to Mr Coyne Financial impact on branch
(if applicable) No. (paraphrased) accounts
5. I LKiang3014S I” Unexplained 5.24 Tssue following multiple cash This KEL relates to the next KEL (MScardifield2219S) as follows: This issue may have hada
Discrepancies
(muttiple cash
declarations)
declarations and trial balance report
being inaccurate. On this occasion
the support department was unable
to identify the root cause of the
discrepancy although it was
reported that a correction could be
made at the Post Office counter
level by redoing the cash
declaration using the same amount
already declared. Within KEL
MScardifield 2219S Fujitsu identify
the underlying software bug as
being caused by ‘cached data’ not
being updated via Riposte.
* LKiang3014S describes the symptoms of the issue; and
* MScardifield describes the cause of the issue.
This was a software/ environmental issue involving the Horizon system
struggling to find cash declarations, which would tend to happen ifa
‘Subpostmaster was undertaking multiple declarations in a stock unit.
This particular instance involved Riposte failing to find one of the Cash
Declarations and thus generating a temporary discrepancy. If left
unresolved this would result in a loss to the Postmaster f or this period
However, when the Subpostmaster subsequently declared the cash held
in branch accurately, this would generate an equal and opposite
discrepancy that would cancel out the earlier discrepancy.
This issue was not resolved, but further diagno stics were added to enable
further investigation should the problem happen again
temporary financial impact, but it
would have been resolved when the
branch next declared the cash held in
the branch accurately
AC_152871086_1
FUJ00083835
FUJ00083835
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
KEL Mr Coyne's Report Fujitsu's Comments
ID Short-name I Paragraph Mr Coyne's Summary Response to Mr Coyne Financial impact on branch
(if applicable) No. (paraphrased) accounts
6 I MScardifield2 I~ Cached Data 5.24 Delay in ‘cached data’ being This KEL relates to the previous KEL (LKiang3014S) as follows: As above:
2198 Delay updated via Riposte. This resulted
in incorrect data being presented in + LKiang3014S describes the symptoms of the issue; and
any discrepancy, variance and + MScardifield describes the cause of the issue.
balance reports.
This is not a ‘Live’ issue, but something found on a Test rig. Experiments
Mr Coyne comments that this is, did find that this could occur around once in 100,000 balance calls when
likely to be confusing for the the rig was heavily loaded
Postmaster and could lead them to
making unnecessary modifications if I 13 Live Peaks reference this KEL, each Peak was treated separately and
they are unaware that the problem the issue was discussed with the Subpostmasters. The work around of
should clear itself overnight. subsequently correctly declaring cash declarations was identified and
communicated to Subpostmasters.
AC_152871086_1 6
FUJ00083835
FUJ00083835
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
KEL Mr Coyne's Report Fujitsu's Comments
ID Short-name I Paragraph Mr Coyne's Summary Response to Mr Coyne Financial impact on branch
(if applicable) No. (paraphrased) accounts
7. I DSeddonS42_I~ Cash Pouch 5.25 A ailure in pouch delivery resulted I The cause of this was that the system was in “Price shopping mode” This had a financial impact, but when
6eP Delivery in a cash gain when the branch which can be used to buy, for example, £15 of 1st class stamps. Price the branch investigated the
declaration was carried out shopping is not supported for products like cash, so when the branch discrepancy the failed remittance
attempted to remit the cash deliver in while in Price shopping mode the would have been identifiable in the
cash remittance failed. logs that are available to users of
Horizon.
In this particular KEL, the remittance was repeated later and was
successful. A fix to ignore “Price shopping mode" for Rems was applied to. I In addition:
Live in April 2007. * — acritical event was written
that will have been picked
up by Fujitsu's support
teams and Fujitsu will have
advised Post Office to
contact the branch; and
* — Post Office's own
reconciliation procedures
would have identified that
the remittance in was not
completed successfully
and contacted the branch
and resolved the matter
with a Transaction
Correction
AC_152871086_1 7
FUJ00083835
FUJ00083835
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
KEL Mr Coyne's Report Fujitsu's Comments
ID Short-name I Paragraph Mr Coyne's Summary Response to Mr Coyne Financial impact on branch
(if applicable) No. (paraphrased) accounts
8 I achat94L ‘Automated 5.26 Problem affecting around 15% of ‘Automated cash declarations from self service kiosks failed at random. No impact on branch accounts.
Cash kiosk branches that prevented these I Analysis shows that this was due to rounding errors on the arithmetic
Declarations branches from being able to carried out due to use of incorrect data types. The business impact of the error is
automatically make cash on cash management and the
declarations. The lack of cash declarations have no impact on Branch accounts, but will I delivery of cash meaning the cash
result in cash planning potentially having incorrect or missing figures and I needed at a branch may be
thus failing to calculate accurate amounts of cash to send to the branch. incorrectly estimated, leading to late
or insufficient cash deliveries.
This was a coding error and was fixed in December 2015,
The vast majority of self service
kiosks are located in Crown branches
and the others are located in large
mains branches.
9. I DSeddon314 I Reference Data 5.30 Examples have been found in KEU I The particular KEL related to the incorrect maximum values being setup I No impact - it was not possible fo
Q Peak records that indicate in Reference Data and the counter not handling this error correctly. This rem in the coin and therefore it could
Reference Data has animpact upon I was caused by a human error by Post Office in the Reference Data not be sold by branches
daily counter activities. Specifically, it was not possible to rem in a particular commemorative coin
because it did not have the requisite Reference Data
The Reference Data was fixed on 14/3/06 and a code fix to handle the
scenario better was applied in June 2006. This did affect a number of
branches during the day that the problem was live, but a message was
sent out to all branches advising them of the issue and how to correct it.
AC_152871086_1 8
FUJ00083835
FUJ00083835
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
KEL Mr Coyne's Report Fujitsu's Comments
ID Short-name I Paragraph Mr Coyne's Summary Response to Mr Coyne Financial impact on branch
(if applicable) No. (paraphrased) accounts
10 johnbascoG5 531 ‘An Automated Payment transaction I For some unknown reason, the counter was associating an Automated No impact.
222H was reported as having failed due to I Payment transaction with an inval id client code. There was no fault in the
“Unknown Agent Code 3046". Client I Reference Data. The fault was impossible to reproduce, and so not
account code 3046 was found to not I understood.
exist in Reference Data and the
fault was not reproducible when the I Failure to complete a transaction would not produce an error in accounts -
problem was analysed and tested. It I a double entry transaction would either all succeed or all fail
was acknowledged that, due to the
business impact, a fix would be
provided to check and validate the
client account code exists in
Reference Data before the
transaction is committed.
Ti{ achaTOL I Reference Data 5.32 KEL documents how branches were I Incorrect Reference Data was issued, which had the effect that payments _I No impact
Errors unable to accept cards for rent and I of Council Tax to Vale of Glamorgan Council were rejected. The
council tax payments due to Reference Data was fixed overnight and it all worked the next day. Failure
incorrect Reference Data, to complete a transaction would not produce an error in accounts - a
double entry transaction would either all succeed o r all fal.
AC_152871086_1 9
FUJ00083835
FUJ00083835
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
KEL Mr Coyne's Report Fujitsu's Comments
ID Short-name I Paragraph Mr Coyne's Summary Response to Mr Coyne Financial impact on branch
(if applicable) No. (paraphrased) accounts
72 MWrighti458 [~ Withdrawn 5.33 Issue involving withdrawn products I When products are withdrawn, there is a withdrawal process which Temporary financial impact caused
Q Products impacting a branch accounting requires users to have Remmed Out the relevant products (which involved I by the Subpostmaster’s failure to
position because the Postmaster the stock being returned to the stock centres) or the stock would ne edto I follow the correct procedure in
will have products that cannot be be adjusted. The grace period for the Subpostmaster either Remming Out I relation to the Remming Out of
accounted for as there is no or adjusting the stock was 6 weeks. After the Reference Data was obsolete stock
remaining Reference Data for them I withdrawn, any unused stock would result in a discrepancy (a change was
to later declare that stock item in the I introduced by Impact in 205 which meant that this could no longer occur). I Fujitsu was able to resolve the issue
accounts. The assumption was that the branch had actually sold the missing stock by adding a compensating
and failed to record it on Horizon so the stock was removed from the transaction.
system and a cash discrepancy would be generated for its value.
In this particular case, the branch failed to Rem Out its unused stock of
obsolete products (stamps) within the grace period despite being asked to
do so on multiple occasions. This left the Subpostmaster with a problem
she could not solve herself, but needed Post Office s upport to do so. To
resolve the issue, corrective transactions were added to the Message
store in consultation with Post Office at the data centre to reflect the value
of the obsolete stock which enabled the branch to roll over correctly.
AC_152871086_1 10
FUJ00083835
FUJ00083835
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
KEL Mr Coyne's Report Fujitsu's Comments
ID Short-name I Paragraph Mr Coyne's Summary Response to Mr Coyne Financial impact on branch
(if applicable) No. (paraphrased) accounts
T3[ wbraS363J_I Reference Data 5.34 The customer was charged twice for I This was a fault involving a self-service Kiosk, which resulted in a I No impact
Errors the same transaction which was customer being debited three times, after which the session was
reported to be a side effect of errors} cancelled.
within Reference Data
This was an issue with how the MGR kiosk interfaced to Horizon, which
attempted multiple Debit Card transactions with the same ID. NOR
diagnosed this as being due to some invalid Reference Data being sent to
the kiosk.
The fault appears to result from two causes:
faulty Reference Data (the result of human error), which was
easily corrected
ii, a fault in the kiosk software, which came from an external
supplier
AC_152871086_1 1
FUJ00083835
FUJ00083835
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
KEL Mr Coyne's Report Fujitsu's Comments
ID Short-name I Paragraph Mr Coyne's Summary Response to Mr Coyne Financial impact on branch
(if applicable) No. (paraphrased) accounts
T4[ GMaxwell365 Duplicate 5.38 Failures or interruptions in service I This was an issue where approximately 635 transactions were sent to. I No impact
1K Payment with the harvesting process can Streamline (Post Office's previous payment services provider) on 2
Transactions cause duplicate payment successive days due to an operational issue in Horizon back end
transactions to be processed. processing. These were identified by Streamline as duplicates and not
processed so customers were not charged twice. This also resulted in
errors picked up by Fujitsu's Reconciliation service so it was trapped by
both ends of the interface. The DRS (Data Reconciliation service) takes
inputs from multiple sources (counters, TPS database and Fl reports) and
compares them and produces reports detailing transactions where they
disagree
15[_ surs357P 5.38 Failures or interruptions in service I Same type of incident as previous KEL at row 14 - same analysis. No impact.
with the harvesting process can
cause duplicate payment This KEL additionally states: as no customer accounts have been debited
transactions to be processed. twice no further reconciliation is needed.
By way of further explanation, the reason for there being 2 KELs is that
this issue occurred in both December 2004 and again in March 2009.
However, in both cases both Streamline and Fujitsu picked up the
problem.
12
AC_152871086_1
FUJ00083835
FUJ00083835
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
KEL Mr Coyne's Report Fujitsu's Comments
ID Short-name I Paragraph Mr Coyne's Summary Response to Mr Coyne Financial impact on branch
(if applicable) No. (paraphrased) accounts
16I _ jharr8328 Recoverable B41 ‘Acknowledgment that the recovery _I This KEL relates to a case where the system was behaving as designed, I There would be no impact if the user
Transactions process (in relation to recoverable _I but the user in branch failed to answer the questions correctly and did not I followed the recovery process
transactions) is a complex area. follow the recovery process properly. presented by Horizon. if the user
failed to do this there could be an
impact due to the user's error.
T7I__carde464Q Failed 542 Particular difficulty in processing a I This KEL does not indicate any software error. The specific scenario ‘As Mr Coyne acknowledges, in this
Recoveries recoverable transactions. In this described by this KEL would always appear as a failed transac tion at a
instance the settlement of the
transaction had not been written
into the Branch database so the
recovery failure would have had no
impact on branch or customer
accounts.
Branch, so it is highly unlikely that any cash was exchanged and the
branch or customer was out of pocket.
All failed recoveries are monitored by Fujitsu and result in the exact
circumstances being checked out and a transaction correction issued in
the event of a discrepancy
All failed recoveries are automatically identified in a daily report that the
security operations team receives (formerly MSU). They review this and
raise Peaks for any that require further investigation by the SSC team
case there was no impact on the
branch account.
AC_152871086_1
13
FUJ00083835
FUJ00083835
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
KEL
Mr Coyne's Report
Fujitsu's Comments
ID
Short-name
(if applicable)
Paragraph
No.
Mr Coyne's Summary
(paraphrased)
Response to Mr Coyne
Financial impact on branch
accounts
18] seng2037L
and
acha959T
5.43
These KELs describe how various
transaction states may also indicate
a failed recovery
These are both generic KELs describing how support should process
specific scenarios found on reconciliation reports. These reports are
generated daily for banking/debit card/e top up transactions where the
counters, agent and financial information (Fl) feeds differ in some way (for
example, the counter timed out a transaction but the Fi authorised it and
ring fenced the funds). They are sent daily to the security opera tions
team. Each transaction is checked and sometimes further investigation is
required from the SSC team via a Peak.
These KELs describe legitimate states that can occur during reconciliation
following a failed transaction which is usually recovered from correctly
There are 23 legitimate (normal process) states and 39 error states that a
transaction can enter. Each state has a specific meaning depending upon
the responses from the Counter, TPS and Fl. A legitimate state would be
that the TPS and Counter both know about a transaction but the Fl may
never send information about it (state 6); this is a legitimate state but it
would warrant investigation.
There would be no impact if the user
followed the recovery process
presented by Horizon
failed to do this there could be an
If the user
impact due to the user's error.
AC_152871086_1
14
FUJ00083835
FUJ00083835
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
KEL Mr Coyne's Report Fujitsu's Comments
ID Short-name I Paragraph Mr Coyne's Summary Response to Mr Coyne Financial impact on branch
(if applicable) No. (paraphrased) accounts
19[ dsed4733R 544 Example where transaction This KEL refers to a failed recovery report (report of failed recoveries) and I This was a zero value transaction so
(no 20) recovery has failed due to a wrongly I in particular some unexpected items in it. there was no impact on branch
named recovery script. Jason
Coyne refers to a Horizon system
error arising because of incorrect
Reference Data
The existence of the failed recovery report is evidence of routine
robustness countermeasures in place to deal with failed recoveries. In
this case, the unexpected behaviour seems to have arisen from faulty
Reference Data, which was corrected within a couple of days.
This KEL refers to problems specifically with an ADCSoript -HPBB_REC1
recovery script. HPBB stands for Home Phone Broadband and it appears
that the transaction would have been a customer application for a Post
Office Home Phone Broadband service where information is collected
from the customer but no payment is made
accounts,
If the transactions had a financial
value then the issue would have had
an effect on the branch. However, it
would have been raised on Fujitsu's
failed recovery reports, investigated
and reported to the POL via a BIMs
report.
AC_152871086_1
15
FUJ00083835
FUJ00083835
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
KEL Mr Coyne's Report Fujitsu's Comments
ID Short-name I Paragraph Mr Coyne's Summary Response to Mr Coyne Financial impact on branch
(if applicable) No. (paraphrased) accounts
20] PSteed2847N Incorrect 551 Software issues resulting in Horizon I Auserhad Remmed In some cash to the wrong stock unit. When they Temporary financial impact which
Mathematical applying the wrong mathematical realised this they carried out a reversal of the Rem In but, due to a was obvious to the Postmaster (who
Sign sign when reversing transactions software error the value of the Rem In was doubled instead. The reversal _I reported the issue) and corrected by
(ie. a plus (+) instead of a minus (- I receipt would have informed the user that something had gone wr ong and_I Post Office issuing an error notice.
») in this case they stated that their Rem In amount had doubled when
reporting the issue. Presumably they looked at a remittance report or
balance report to see this.
‘A Rem In reversal is not a particularly common transaction and it was
prohibited as part of Impact changes in 2004. The user has attempted it
for a specific reason so if after performing the reversal it hasn't had the
desired effect you would expect the user to clearly notice this and raise a
call with the helpdesk to query it. Upon confirming the error the NBSC
could then issue an ‘error notice’ to correct any anomaly.
It took just 13 days from reporting to active a fast track fix.
AC_152871086_1 16
FUJ00083835
FUJ00083835
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
KEL Mr Coyne's Report Fujitsu's Comments
ID Short-name I Paragraph Mr Coyne's Summary Response to Mr Coyne Financial impact on branch
(if applicable) No. (paraphrased) accounts
21I cardo5756N Pouch Rem 552 This is an example where the This concemed the reversal of a Rem Out of a pouch, a rarely used Temporary financial impact which
Out Reversal system failed to reverse all items in I process, by which when a banch has prepared a pouch returning cash to I would be picked up by Post Office's
‘a multi-line pouch, meaning only the I the cash centre and then realised that they either made a mistake (as cash centre reconciliation process
first item was reversed happened in this case) or need the money after all. It was not possible to I and corrected by a transaction
rule out the possibility that this was caused by a software issue, but it was I correction.
not possible to replicate it so this could not be investigated further.
The Rem Out reversal appears to have gone wrong in this case and only
part of the pouch contents was reversed, thus leaving some of the value
still in suspense.
22 GCSimpsont Foreign 5.54 All currencies in branch doubled up I The discrepancy (between the Horizon record and physical cash) was I Temporary financial impact which
o49L Currency following successful balancing eight I picked up in branch and a call raised. It appears that this particular I was resolved when the branch
Discrepancies days previously. incident was resolved by the branch upon monthly balancing. carried out the next monthly balance
(when it declared the actual value of
Due to the delay in providing the information to the development team, the I currencies held in branch).
lack of any record of this incident having happened previously and there
being no further reports of similar problems, we are unable to confirm the
root cause
AC_152871086_1 17
FUJ00083835
FUJ00083835
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
KEL Mr Coyne's Report Fujitsu's Comments
ID Short-name I Paragraph Mr Coyne's Summary Response to Mr Coyne Financial impact on branch
(if applicable) No. (paraphrased) accounts
23] MHarvey3527 I Insufficient 5.55 Evidences that in certain The KEL says No impact.
1 Diagnostic Data investigations there was insufficient
diagnostic data to be able to fully ‘As this is, at the moment a one off event and clearly no further progress
diagnose an issue. can be made at this stage, I have therefore closed PC113202 as
“insufficient evidence". However, any further occurrences should be sent
to APS Counter Dev for investigation.’
This is an issue in the transferring of copies of transactional data from
Horizon to Post Office's back-end systems. The specific data mentioned
here is not financial. The underlying issue was that there was
unnecessary validation of copies of Reference data being passed back to
Post Office and this validation was removed as part of Impact (a joint
working body to introduce improvements to the system and processes) in
2005,
241 CObengi123 I Stock Gains 5.56 Unexplained discrepancies (gains) I This was a complicated memory loss issue in branch. Extensive searches I Not known due to the age of the
Q for different stock unit types (Cash __I were made for memory loss issues in test at the time and only one was matter.
and Stamps) was reported. The found and explained (which did not relate to this issue). It therefore
incident remained unexplained and _I appears that this was a one off incident. Due to the passing of time, we're
no record of advice having been unable to identify the cause, but if the counter software was running short
provided to Postmasters. of memory we would expect the counter to display a warning to the user
which would have been seen.
25] DRowe1625K 587 This KEL is not available.
AC_152871086_1
4
8
FUJ00083835
FUJ00083835
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
KEL Mr Coyne's Report Fujitsu's Comments
ID Short-name I Paragraph Mr Coyne's Summary Response to Mr Coyne Financial impact on branch
(if applicable) No. (paraphrased) accounts
26 dsed525Q PIN pad 5.68 There was a failure in the This was a faulty PIN pad. It prevented the Postmaster from carrying out __I No impact.
Failures Postmasters being able to transact _I some transactions but this would not affect branch accounts.
various types of transactions
including payment transactions
using a PIN pad. An error message
and code were generated, and a
new PIN pad was the
recommended solution.
27[~ surs3941P PIN pad 5.68 There was a failure in the This appears to have been an issue with a corrupt customer card. Itwas I No impact.
Failures Postmasters being able to transact I set up with no CVM (CVM is something on a card that indicates whether a
various types of transactions PIN or a Signature is to be used to authorise a transaction). SSC suggest
including payment transactions that this was an attempt to do a Balance Enquiry on a Credit Card (which
using a PIN pad. An error message _I isn't allowed). Reference Data should have prevented that being
and code were generated, anda attempted and it's not clear why it didn't. It could have been a corrupt card
new PIN pad was the that the customer had.
recommended solution.
AC_152871086_1 19
FUJ00083835
FUJ00083835
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
KEL Mr Coyne's Report Fujitsu's Comments
ID Short-name I Paragraph Mr Coyne's Summary Response to Mr Coyne Financial impact on branch
(if applicable) No. (paraphrased) accounts
28] BrailsfordS22 PIN pad 5.68 There was a failure in the This was a faulty PIN pad. It prevented the Postmaster from carrying out __I No impact.
39K Failures Postmasters being able to transact _I some transactions but this would not affect branch acco unts.
various types of transactions
including payment transactions
using a PIN pad. An error message
and code were generated, and a
new PIN pad was the
recommended solution.
29I carde219R PIN pad 5.69, 5.135 I This KEL indicated that PIN pad The KEL relates to a specific (one -off) hardware issue with the old No impact.
Failures related issues would usually result I Hypercom PIN pad where the transaction was authorised but not
in the recommendation of a new confirmed due to the hardware fault. The process is request, authorisation
PIN pad, regardless of the error. InI and confirmation, however, the final stage of confirmation did not take
this case a transaction had been place. As per the KEL, this was identified as part of the reconciliation
declined by the PIN pad but did not I process (ie. the same DRS reconciliation process referred to above) and
get reversed passed back to Post Office and a Transaction Correction issued
automatically. There was no impact on branch accounts, however, the
customer was charged twice and they should have been reimbursed by
Post Office's back end processes.
30] dsed4733R 5.92 Identifies multiple failed recoveries _I See analysis at line 19 above. See analysis at line 19 above.
‘occurring because of a wrongly
named recovery script
AC_152871086_1 20
FUJ00083835
FUJ00083835
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
KEL Mr Coyne's Report Fujitsu's Comments
ID Short-name I Paragraph Mr Coyne's Summary Response to Mr Coyne Financial impact on branch
(if applicable) No. (paraphrased) accounts
31] obenge5933K 5.93 There was a loss of This is further evidence of the failed recovery report doing its job and This may have caused a temporary
communications following network I alerting Fujitsu to failed recoveries to enable them to investigate them and _I financial discrepancy
banking transactions and the make any necessary corrections to accounts by sending a BIMs to Post
printing of the customer receipts Office. If this had not happened, a transaction correction would have
resulting in a message to the data_I been issued as a result of Post Office's own reconciliation processes.
centre timing out. Consequently, the
Postmaster was asked to follow The incident was caused by a complex ‘grey’ communications failure (ie
recovery but the transaction was the network kept switching between good and bad; not solid good or solid
only able to recover partially. bad), which the development team could not reproduce.
The KEL gives no reason to suppose that, even if this condition had
persisted, the backstop of reconciliation and Transac tion Corrections
would not have corrected any resulting errors in accounts.
As per KEL, the failed recovery will be centrally reported and investigated
via the DRS reconciliation process.
32] wrightm33145 I Payments 5.116, 5.118 I Non-zero trading position This is the same issue as line 1 above. See analysis at item 1 above. See analysis at item 1 above
J Mismatch
AC_152871086_1
21
FUJ00083835
FUJ00083835
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
KEL Mr Coyne's Report Fujitsu's Comments
ID Short-name I Paragraph Mr Coyne's Summary Response to Mr Coyne Financial impact on branch
(if applicable) No. (paraphrased) accounts
33] AmoldA2153 Withdrawn S117 This generates the same Receipts _I This issue was found during internal Horizon Online testing and fixed There may have been an issue if the
Pp Stock and Payments Mismatch error before Horizon Online went live branch had continued to rollover
message (to KELs in lines 1 and 31) before the products were reinstated,
but in fact relates to a mismatch In 2016, some products were withdrawn during a trading period when they I however, in the 2 cases recorded,
during the balancing of a stock unit I were still being traded. This is contrary to Reference Data procedures the branches were advised to delay
that contains withdrawn product. and caused an issue in branches that had traded those products. the rollover until the products were
re-instated so no impact on branch
A fix was put in place involving the products being re-instated and the I accounts.
branches affected rolled over successfully.
34] ballaniji759Q Payment 5118 Details 5 conditions that may cause I This is a generic KEL ensuring that the event monitoring team raise a call _I N/A (this is a generic KEL).
Mismatch a receipts/ payments mismatch that I every time a Receipts and Payment mismatch event is seen. It references
may impact on branch accounts. other KELs that are known issues for specific cases. This is an example
of how our monitoring is made to work effectively. All of these call s get
investigated and may need manual correction to avoid errors in branch
accounts. From a Horizon perspective, none of the calls raised should
therefore be left without investigation / resolution.
AC_152871086_1 22
FUJ00083835
FUJ00083835
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
KEL Mr Coyne's Report Fujitsu's Comments
ID Short-name I Paragraph Mr Coyne's Summary Response to Mr Coyne Financial impact on branch
(if applicable) No. (paraphrased) accounts
35] achat357Q Incorrect 5.120 Ttis possible for discrepancies to The issue related to declarations from the same trading period a year ago _I There may have been some financial
Declarations have been accepted by the becoming visible again and thus causing confusion. It only affecte d impact, but there were reports
Postmaster based upon incorrect branches that had done a Stock Declaration the previous year but available in branch that could have
declarations. The problem could normally didn't do them. The fix was to change the archiving strategy so been used to identify incorrect
arise due to old stock declarations _I that all declarations that had not been updated for 6 months were declared amounts.
not being automatically removed removed. A check was made at the time for any bran ches that had old
from the system. These discrepancies that might become current again in the next 2 months (to Further, a corresponding gain / loss
could only be removed by making —_I allow for the archiving fix to be made) and these were removed manually I would occur in a subsequent trading
zero-value declarations or deleting I by MSC. A period so that would resolve the
the stock unit then waiting overnight issue.
before balancing, The initial call was raised on 11th Feb 2011. A central workaround to
avoid further issues was implemented under MSC and the official fix was
released in June 2011
AC_152871086_1 23
FUJ00083835
FUJ00083835
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
KEL Mr Coyne's Report Fujitsu's Comments
ID Short-name I Paragraph Mr Coyne's Summary Response to Mr Coyne Financial impact on branch
(if applicable) No. (paraphrased) accounts
36] acha3145Q 5.121 Provision of full support solution __I This KEL is cited in the previous KEL dealt with at line 35 and relates toa I There may have been some financial
for incorrect stock declarations and I stock balancing problem caused by the user doing some uncommon impact is is branch process at this
discrepancies. sequence — i.e. not caused by withdrawn products. stage, it is effectively as if the SPM
had wrongly declared the cash or
stock and the system will warn them
that this does not match the
calculated value. In this case it was
an old declaration that got included.
Redeclaring the cash/stock will fix the
issue. If they do not they will roll with
a loss this BP/TP but have a gain
next TP/BP.
37[_allendt645p Horizon 5.129 Provides an example of Horizon’s __ I Horizon allowed the clerk to select ‘Debit card’ as a method of payment This is a case of the Postmaster
Interface weak interface controls and lack of I_I and later switch to ‘fast cash’ at the end of the customer session. This was _I being responsible for errors made by
Controls data entry validation. In a single a subsequent user error which involved the user at the branch failing to staff. This would have shown as a
sales transaction the user was able I take payment of £500 for the 40 euros. discrepancy caused by user error in
to select and enter different not taking the payment that was due
methods of payment (Debit Card from the customer and the
and Fast Cash). Horizon allowed Postmaster would be liable for this.
the transaction to be settled via Fast
Cash when the Debit Card payment
method had already been selected,
AC_152871086_1 24
FUJ00083835
FUJ00083835
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
KEL Mr Coyne's Report Fujitsu's Comments
ID Short-name I Paragraph Mr Coyne's Summary Response to Mr Coyne Financial impact on branch
(if applicable) No. (paraphrased) accounts
38] acha62iP 5.130 The correct screen to successfully I This issue, also referred to as the ‘Dalmellington’ issue is dealt with
process a cash pouch did not
appear resulting in the clerk in an
outreach branch inadvertently
doubling up the amount of cash
recorded, The issue appears to
have been caused because of an
earlier system logout or inactivity
which in turn resulted in incomplete
checks being conducted by Horizon
post logon
substantively in the second witness statement of Torstein Olav Godeseth.
Temporary financial impact which
was rectified by a Transaction
Correction being issued.
AC_152871086_1
25
FUJ00083835
FUJ00083835
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
KEL Mr Coyne's Report Fujitsu's Comments
ID Short-name I Paragraph Mr Coyne's Summary Response to Mr Coyne Financial impact on branch
(if applicable) No. (paraphrased) accounts
39] EJohnson393 Horizon 5.182 This enabled Postmasters to carry _I Whilst Remming In currency it is possible to create a transaction with a No impact.
7R Interface out “Rem In” transactions without a I positive quantity and a zero value
Controls value being entered
This issue was caused by the Reference Data test team (not following the
correct process). The functionality was changed the following year so that
amounts did not need to be entered.
Auto Rems (introduced around 2004) meant that the content of cash and
currency pouches was sent to the branch electronically so when a pouch
was delivered, the system automatically told the Subpostmaster what the
content was and used that value for the Rem in rather than asking the
‘Subpostmaster to key it in.
AC_152871086_1 26
FUJ00083835
FUJ00083835
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
KEL Mr Coyne's Report Fujitsu's Comments
ID Short-name I Paragraph Mr Coyne's Summary Response to Mr Coyne Financial impact on branch
(if applicable) No. (paraphrased) accounts
40I PSteed145J_I Phantom Sales 5.133 Involved ‘phantom’ sales appearing I These "phantom" sales were caused by hardware problems and fixed by I No impact If the transaction related
on the Horizon counter screen but _I replacing hardware. to stock, when the branch declared
which had not been selected by the their stock and cash the discrepancy
user. would cancel out (e.g. a sale of
stamps would reduce the stock of
stamps and increase the cash figure
by a corresponding amount; when
balancing the correct number of
stamps should be declared and this
will cancel out the effect of the
phantom sale of stamps).
41] pearrolli235R I Screen Freezes 5.133 Instructions on how to deal with These instructions (which relate to issues that do not concern bugs or No impact on branch accounts.
environmental issues and hardware I errors in Horizon) were distributed to those who called for help via this
are contained in this KEL. Jason KEL.
Coyne says “it is not known how
widely these were distributed to
SPMs", implying that they should
have been.
AC_152871086_1 27
FUJ00083835
FUJ00083835
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
KEL Mr Coyne's Report Fujitsu's Comments
ID Short-name I Paragraph Mr Coyne's Summary Response to Mr Coyne Financial impact on branch
(if applicable) No. (paraphrased) accounts
42) jharri323aL 5.1368 Example of a successfully recorded I This relates to a fishing rod licence request not sent fo the Environment No impact.
transaction initiated in a Post Office I Agency and was only detected at a later date. In this particular instance,
branch (where a customer receipt —_I the transaction had been reversed by a user; this is not a software issue
was generated) which failed to Once a transaction is reversed, all relevant data is discarded and not sent
appear in the Post Office Data to the AP Client as its effectively as though the transaction never took
Gateway. place (Fujitsu only keep Post Office Data Gateway records readily
available for 30 days) but it is committed to the audit store and therefo re
any additional investigation would have needed to be undertaken by
audit)
It is not known why the reversal took place. This could be due to
fraudulent activity or could be that the customer sought a refund and the
money was refunded.
AC_152871086_1 28
FUJ00083835
FUJ00083835
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
KEL Mr Coyne's Report Fujitsu's Comments
ID Short-name I Paragraph Mr Coyne's Summary Response to Mr Coyne Financial impact on branch
(if applicable) No. (paraphrased) accounts
3I" MAris34331 5.137 The ability of Horizon to erroneously I This was a software bug which allowed a transaction to be recorded twice _I No impact (fixed in testing)
(no 46) record the same transaction twice _I after a session transfer and was a fairly rare circ umstance. This issue was
after a session transfer to a different
counter. This happened with both
NS&l (National Savings &
Investments) and Network Banking
(NWB) transactions. The KEL was
passed to a development team to
provide a bug fix as part of the S60
rollout but it is unknown if this was
ever resolved
picked up during internal testing and despite numerous attempts
development were unable to recreate the scenario. There are no Peaks
which refer to this KEL.
AC_152871086_1
29
FUJ00083835
FUJ00083835
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
KEL Mr Coyne's Report Fujitsu's Comments
ID Short-name I Paragraph Mr Coyne's Summary Response to Mr Coyne Financial impact on branch
(if applicable) No. (paraphrased) accounts
44] CharitonJ275 I~ Previous Key 5.139 Counter level corrections made via_I A software error in a PDL file meant that when a user used the "Previous" I No impact
2T Software Error the ‘Previous’ key led to both the key for a transaction that used an ADC Script, old and amended values
old value and amended value being I were stored and used. This resulted in an incorrect transaction for the sum
stored and used in error in the amount of both the old and amended values being added to the sales
transaction. A fix was released in basket
the Live environment eight days
after the issue was first raised. As the user has used the "Previous" key to go back and amend a value, it
should have been obvious to the user if then a completely different value
item is added to the sales basket. If, for whatever reason, this was not
noticed then the the customer will end up being overcharged as the
system will ask the user to take a larger payment. Assuming the user does
what the system says there will be no impact to the branch accounts.
This issue affected 3 products and only occu rred when "Previous" key
was used to correct the amount entered. It was fixed 8 days after it was
spotted. A search was made for all branches that had used those
products twice within a session and the results were sent to Post Office.
AC_152871086_1 30
FUJ00083835
FUJ00083835
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
KEL
Mr Coyne's Report
Fujitsu's Comments
ID
Short-name
(if applicable)
Paragraph
No.
Mr Coyne's Summary
(paraphrased)
Response to Mr Coyne
Financial impact on branch
accounts
45] SSurs43P
5.141
Example of a declined network
banking
transaction that resulted in money
being taken from the customer's
account.
‘An error in network banking caused the customer's account to be debited
although the transaction failed at the branch and th e Postmaster was told
at the counter that the transaction failed. Banking transactions with a
response code of 26 (Fl Unavailable, Try again later) would be recorded
as zero value transactions at the branch, a DECLINED receipt would have
been produced and so no money should have been handed over to the
customer. Therefore no impact on branch accounts. On rare occasions
the financial institution may have debited or credited the customer bank
account despite us not receiving the authorisation. in these instan ces, if
the automatic reversals fail to resolve matters then the issue would be
picked up as part of the DRS reconciliation process meaning Fujitsu would
inform Post Office of what has happened at the counter so that they can
liaise with the financial institution to ensure their systems match so
customer is not out of pocket. The root cause was between the NBE and
the Fi, which is outside of Horizon
No impact.
AC_152871086_1
31
FUJ00083835
FUJ00083835
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
KEL Mr Coyne's Report Fujitsu's Comments
ID Short-name I Paragraph Mr Coyne's Summary Response to Mr Coyne Financial impact on branch
(if applicable) No. (paraphrased) accounts
46] LKiang3526R 5.142 Examples of E-Pay transactions This was caused by two authorisation agents being active at once, when __I No impact
crediting the customer account only one should have been, resulting in the phone being credited with £10
twice although only one payment twice and ePay charging Post Office or the phone provider (depending on
has been taken. the arrangement between the two) being charged twice, meaning the
customer would have got two sets of top up for the price of one
This is a back end system problem which would be picked up by cou nter
measures, causing a BIMs to be raised.
47I_ SSurS310P 5.142 Examples of E-Pay transactions ‘Similar to the KEL above at line 46 No impact.
crediting the customer account This is a back end system problem which would have had no impact on
twice although only one payment the branch accounts. Despite a phone being topped up twice at the
has been taken. branch only a single top up would have been recorded in branch along
with the required payment for it.
Again, this is a back end system problem and would need to be resolved
by Post Office centrally and would not impact branch accounts.
AC_152871086_1 32
FUJ00083835
FUJ00083835
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
KEL Mr Coyne's Report Fujitsu's Comments
ID Short-name I Paragraph Mr Coyne's Summary Response to Mr Coyne Financial impact on branch
(if applicable) No. (paraphrased) accounts
48] pothapragada 5.165 Jason Coyne says “itis The issue was the ability to declare stock that could not then be This may have had a financial impact
c4359R acknowledged that simple fixes transacted (due to Reference Data rules). but if so it would be due to human
ought and were implemented to error (ie. declaring that it held an
either fix bugs or provide additional I The only impact on a branch's account were if the branch were to actually _I item of stock that it couldn't transact).
data validation checks". declare that it held an item of such stock. This is unlikely as the item had I This discrepancy would be removed
been withdrawn and should be returned to the stock centre. It should also I if the branch accurately declared that
be noted that most branches do not undertake stock declarations as stock _I it had no such stock.
is normally manged using stock adjustments (which didn't have this
issue). Should the stock be declared by mistake by user -error, then a
further declaration of the correct (ie. zero) holdings would resolve the
issue
49] Marris4123N 5.165 Jason Coyne says “itis This was a problem observed during test in Disaster Recovery for DVLA __I No impact.
acknowledged that simple fixes
ought and were implemented to
either fix bugs or provide additional
data validation checks"
transactions - a very rare circumstance, which should be handled
correctly, but would have no impact on bran ch accounts in the unlikely
case that a branch encountered the issue
When recovering a failed counter, the user is asked to input data from a
receipt. To handle poor typing there is a check sum which should ensure
that it has not been altered. This bug re lates to the fact that not all the
data entered is controlled by the check sum. Therefore, when a tester
deliberately input incorrect data, the system did not detect it. NB this did
not include financial data.
AC_152871086_1
33
FUJ00083835
FUJ00083835
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
KEL Mr Coyne's Report Fujitsu's Comments
ID Short-name I Paragraph Mr Coyne's Summary Response to Mr Coyne Financial impact on branch
(if applicable) No. (paraphrased) accounts
30] acha2230K ‘Additional 5.186 Jason Coyne highlights a problem I This issue, also referred to as the Local Suspense Account issue is dealt. I This issue had a financial impact
Checks with additional checks which were I with substantively in the second witness statement of Torstein Olav which was resolved by Post Office
implemented to identify system Godeseth. writing off discrepancies.
errors/ inconsistencies when
balancing
during branch and references a
note within the KEL as follows:
“This should never happen -
something has gone horribly wrong.
Or possibly the checks haven't been
implemented as intended.”
AC_152871086_1 34
FUJ00083835
FUJ00083835
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
KEL Mr Coyne's Report Fujitsu's Comments
ID Short-name I Paragraph Mr Coyne's Summary Response to Mr Coyne Financial impact on branch
(if applicable) No. (paraphrased) accounts
St] dsed2049S 5.187 Jason Coyne suggests that this KEL I This KEL relates to withdrawn products that were converted to cash on No impact if the correct process is
highlights the lack of system rollover but the losses are carried forward into the next period instead of _I followed for returning the withdrawn
‘communication and/or support being dealt with there and then products to Post Office. In the event
‘communication in respect of certain that the process is not properly
system features which could Withdrawn products should be sent back by the Postmaster to Post Office I followed, a Transaction Correction
subsequently result in errors. so the branch is not holding stock that cannot be sold can be issued to correct any impact
on branch accounts.
If this process is not followed, the branch will be left with a loss at the next
trading period and could be corrected by a Transaction Correction. The
stock will also be converted to cash if the Postmaster has purchased it
personally, for example. The fix was made to make this clear to the
Postmasters.
Jason Coyne implies this was a bug which took 6 months to fix. There
was a minor bug in that the cash value of the withdrawn stock was not
included in the current rollover, but delayed until the next rollover.
However there was no specific loss to the branch (other than the value of
the withdrawn stock which they were responsible for as described)
AC_152871086_1 35
FUJ00083835
FUJ00083835
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
KEL Mr Coyne's Report Fujitsu's Comments
ID Short-name I Paragraph Mr Coyne's Summary Response to Mr Coyne Financial impact on branch
(if applicable) No. (paraphrased) accounts
$2] acha3250R 5.189 The reconciliation process used by _I This was an issue with Post Office's back end reports, caused by timing No impact.
Post Office to assist with identifying I issues (APS transactions arriving a day late) which were outside of Po st
any accounting differences is not Office or Fujitsu's control.
able to easily identify genuine
differences and/or differences This caused discrepancies in certain back-end reports, which could
resulting from external APS however be understood by cross checking other reports. This is a “false
transactions from old trading dates. I error" being reported by reconciliation relating to transactions occurring
around the end of day, being 7pm.
There is nothing wrong in the actual transactions ~ just an error in the way
that reconciliation totals are calculated.
33] achatoaiL 76 During the recovery process (when I The KEL says: As there is no reconciliation needed,
‘some transactions recover but there is no impact on branch
others fail to recover) itis only the I "This is not really a problem, it is just confusing when investigating a state I accounts
recovered transactions printed on _I 4 call. The Disconnected Session receipts will show all the transactions in
the receipt. The disconnected the session. The successfully recovered transaction needs no
session receipt should also identify I reconciliation."
those transactions not recovered.
These are printed for the This shows that any incomplete information went not to the branch, but to
Postmaster to retain someone in Post Office or Fujitsu investigating a state 4 call.
AC_152871086_1 36
FUJ00083835
FUJ00083835
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
KEL Mr Coyne's Report Fujitsu's Comments
ID Short-name I Paragraph Mr Coyne's Summary Response to Mr Coyne Financial impact on branch
(if applicable) No. (paraphrased) accounts
Ba] surs1147Q Failed TT This relates to the solution toa While the advice to the user in the KEL is somewhat counter -intuitive and _I No impact.
Recovery failed recovery requesting the user _I indicates that nothing should be done and the user should wait for the
to log onto the relevant counter and I counter to time out, leaving the system error, there is nothing to suggest
start the recovery process, but the recovery was not resolved. The root cause of this issue was an error
leave the counter displaying the in Post Office's script relating to the Dangerous Goods products, that
system error message. resulted in recovery failing. We're unable to tell whether or not the soript
was corrected.
This would have no direct impact on branch accounts, but clearly would
be very inconvenient as the counter is out of action.
85] wrightm33145 Payment 79, 7At Cited by Coyne. This issue, also referred to as the ‘Payment Mismatch’ issue is dealt with I See item 1 above
j Mismatch substantively in the second witness statement of Torstein Olav Godeseth.
56] boismaisons1 I Disc Space 912 This KEL describes the running This is an issue with hard disks. Disc space sizes have nothing to do with _ I No impact.
328M Sizes commands on counters to assess _—_I branch transaction data which in any case at the time (2012 ) was not
disk space sizes. stored in the branch
AC_152871086_1 37
FUJ00083835
FUJ00083835
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
KEL Mr Coyne's Report Fujitsu's Comments
ID Short-name I Paragraph Mr Coyne's Summary Response to Mr Coyne Financial impact on branch
(if applicable) No. (paraphrased) accounts
87I SeemungalG I~ Transaction 9.49 Records an instance where This was an issue when posting transactions from old Horizon to BRDB No impact.
5190 Amendments transaction prior to a branch migrating from old horizon to Horizon Online. In some
amendments carried out are cases, such transactions fell foul of validation in TPS and needed to be
causing exceptions. amended before being sent to Post Office’ s backend systems. Such
amended transactions were also posted to BRDB. Any amendments were
related to the trading period they related to and not any financial values.
They were only in BRDB to be used if the branch migrated during the
current trading period from Old Horizon to Horizon Online.
Tip Repair is a back end process to make sure all required transactions
get sent to the relevant external systems. There is no effect on the branch
accounts.
36] MHarvey2255 I Corrective 950 Records the manual addition of This is a back-end balancing issue caused by missing or invalid No impact.
P (no 64) Balancing corrective Reference Data. The insertion of correction records is done into the TPS
Transactions balancing transactions inserted by _I database to allow the branch data to be forwarded to POLSAP. There is
SSC affecting the TPS system no effect on the Branch Database therefore no effect on the Branch
Accounts. It is not understood why this is marked as remote access as the
work is done within the Data Centre.
AC_152871086_1 38
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
FUJ00083835
FUJ00083835
APPENDIX 2
KEL Fujitsu's analysis
ID Nickname Summary of the KEL Analysis Impact on branch accounts
1 achad2aK Cash Button I The Cash button on the settlement This is not a bug, rather itis a feature of how the system No impact unless the system is misused by
screen can be used for either a receipt inI operates. branch staff.
ora payment out of cash. Horizon
decides on the context depending on
whether the stack total button says TAKE
or PAY,
2. achadBES Targe This involved both transactions being I Although the clerk took the money from the customer, the I No impact
Transactions I completed in a single session. session wasn't settled because £1m is too large for FastCash.
The settlement should instead be entered as two £500,000 cash
payments. This is an issue with hitting system limits with very
large transactions over £1M. This would be very noticeable and
there is an avoidance action to take two smaller payments.
Looking at the Peak, the Transactions were actually recovered
automatically by the system (AP Txns were recoverable in old
Horizon) so no impact on branch accounts once recovery was
carried out.
3. achas0eS Postmaster reports problems with I This was a bug in the handling of multiple bags of coins when I Temporary impact that would have been
remming out, in particular differences I being Remmed Out. It was fixed by a code fix issued in April I identifiable from the reports available to
between the two receipts which are I 2007 and fully rolled out by June 2007. Subpostmasters. The KEL description states
AC_152871086_1
39
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
FUJ00083835
FUJ00083835
printed after the pouch barcode is
scanned
The original problem was found on 12" Feb 2007 and was
presumably due to a software update rolled out at that time.
Investigations were carried out and a list of affected branches
was generated (this is no longer available) and provided to Post
Office. NBSC was informed about a workaround to the issue.
that “differences between the two receipts
which are printed after the pouch barcode is
scanned... The second receipt (Office Copy)
only shows one bag of each”
It would have been rectified by a transaction
correction.
4. acha522T
Cash
Withdrawal
This involved two separate cash
transactions most likely by two separate
customers as follows:
* customer 1: given £200 in cash
. customer 2: given £320.90 in cash
(being the total value of the stack)
The loss was caused by the user not
This is not a bug, rather the incident appears to be caused by
human error of the user not reading the screen carefully when
doing a withdraw limit CAPO transaction after failing to settle an
earlier customer basket.
Impact caused by human error.
settling stack in between banking
transactions.
3.I acha2i40S I £im Cheques I If the branch holds cheques to a value of I This will be rare and if encountered it will be highly visible by the I No impact.
more than £1M, then the value of I Postmaster.
cheques cannot be adjusted using the
normal Stock Adjustment mechanisms. _I The best thing to do would be to rem out the Cheque for £1M
and then any further adjustments can be handled as normal.
6. I achass47Q Tfa stock unit carried forward from This was an issue for a branch following migration from old I No impact.
Horizon is deleted before the first trading
period rollover on Horizon Online, the
Horizon to Horizon Online and only affected a branch if it had
deleted a Stock Unit after migration and before the first rollover.
AC_152871086_1
40
FUJ00083835
FUJ00083835
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
check for Tast stock unit’ may not be
applied properly, and all the stock units
can roll over without Local Suspense
being cleared
Local Suspense for a particular trading
period has to be zero before the branch
can be rolled over. Once a stock unit is in
the new trading period, it can put
gainsilosses into Local Suspense and
clear them, but this has no effect on
Local Suspense for the old trading period
- think of them as two separate
containers.
The issue would be clearly visible as the Branch would be
unable to rollover without following a complex work around.
a]
acha3610P_
‘Advantageous
Exchange
Rates
There are advantageous exchange rates
for transactions over certain limits
(shown on Foreign Currency report as
DDE for Euros, DDU for US dollars). In
this case the limit was £500.
The user pressed 'Buy Euros’ and
entered 600. The exchange rate shown
was the standard exchange rate, not the
rate for transactions over the 500 limit,
the reason being the £500 limit had not
been reached.
This appears to be a misunderstanding of what exchange rates
touse. The “over 500" rate applies to when the sterling value is,
over £500 (not the Euro value)
No impact
I
acha4221Q
The clerk went into the ‘Delivery’ menu
and scanned two pouches (one of
This was a bug in the early days of Horizon Online following an
unusual (but valid) sequence of events. It was fixed on 19"
Financial impact would have been clear to the
branch because a duplicate receipt was
AC_152871086_1
41
FUJ00083835
FUJ00083835
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
‘currency and one of coins). The second
pouch was recorded twice on the
system, resulting in a loss of £80
Two Remittance In slips relating to the
second pouch were output, both
identical, as well as one for the first
barcode.
Apri 2070.
printed. It would have also been identifiable
from the reports available from Horizon.
It was resolved by a transaction correction
being issued.
9. acha4353P_
The counter froze while cashing a Postal
Order and now recovery won't complete
so the counter could not be used The
counter was rebooted, but when they
logged back in they got a Postal Order
encashment transaction _— recovery
message for a negative amount, follow ed
by the Invalid Value message again
This was due to an invalid Postal Order with a negative value
being set up by an external system which resulted in a counter
being frozen.
A fix was applied to check the value of Postal Orders coming
from external systems as being positive and this was applied on
15/8/2011. A system fix was made to allow recovery to be
bypassed on this counter to make it useable again
No impact.
TO] acha5226J
When @ branch puts through a bureau
transaction in excess of £5,000 a
message should appear on the screen to
remind the user to ask the customer to
take two forms of ID from the customer
to conform with Anti Money Laundering
regulations. This reminder prompt is not
appearing on the Horizon Online
counters.
This issue was fixed in October 2010
No impact,
achaS259Q
The Postmaster wrote a discrepancy of
£167.17 to Local Suspense, then this
was cleared from Local Suspense as
Tt appeared to only affect branches balancing in April 2070 and
33 branches were identified as being impacted. Details of these
branches were passed to Post Office.
Temporary financial impact which would have
been cancelled out in the following period by a
corresponding discrepancy
AC_152871086_1
42
FUJ00083835
FUJ00083835
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
normal and the Postmaster selected to
make good the losses. At this point the
system printed out a final balance report
for the trading period with the cash
figure amended and nil discrepancy
Normally the system would then come up
with a message to confirm rollover but
instead went back to the screen asking
how the discrepancy was going to be
made good,
This was an issue found in the early days of Horizon Online and
was resolved in July 2010.
72I AChambersié3 I Reportissues I A transaction for £6.67 was done at I A timing issue to do with printing reports at a counter. There I No impact
3 17:23 and was settled at 17:25. The daily I was no impact on the branch accounts ~ just confusion due to a
APS transaction report was done on a I transaction being missed from a report and the report not being
different counter at 17:25 and did not I re-printed when it appeared.
include the APS transaction.
T3I_ AChambers225 Postmaster sold some foreign currency I This was not a bug, rather an issue in how a Currency I Impact, but guidance on how to correctly
2R (1000 euros, sale value: £750). The I transaction was incorrectly reversed on old Horizon. perform an existing reversal was all that was
Postmaster realised the transaction had
been settled to the wrong product in this
case being cash instead of debit card
Existing Reversal was used to reverse
the transaction, and then re-run correctly
When it came to balance at the end of
the trading period, the currency stock
holding on the system was too high by
1000 Euros. When corrected, this gave a
gain on currency of £720 and a cash loss
Foreign currency transactions consist of two parts: the currency
itself, which has a value based on the exchange rate, and
margin, which is added to cover the cost of the transaction.
When the transaction was reversed, the Postmaster entered the
transaction for the cash settlement part of the transaction. While
the Postmaster believed the whole transaction had been
reversed, it had not as the margin had not been reversed. When
the stock unit was balanced, the wrong number of Euros
became apparent. The stock holding was corrected by the
needed to rectify it.
AC_152871086_1
43
FUJ00083835
FUJ00083835
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
‘Of £750, being a net loss to the branch of I declaration of the actual number held. Again, this did not correct
£30 the margin, which is generated as part of the currency sale and
is not directly linked to the stock.
The way Reversals are handled on Horizon Online means that
this sort of issue can no longer happen.
14] AChambers413, Multiple quantity for stamps/postage I When a quantity greater than 1 is entered for a Smartpost I This may have had an impact but a user should
4aR label affects cash settlement or I transaction, the Quantity is not reset to 1 when the user moves I have been able to spot it and the sums involved
subsequent transactions on to the settlement screen. are likely to be small due to the issue affecting
mails products.
If the transaction is settled to Fast Cash / Fast Cheque or Debit
Card, this doesn't matter, but some users habitually use the
Cash (F2) button to enter the cash presented by the customer,
then give the customer change as indicated by the new stack
total. If this is done, the cash amount entered is multiplied by
the Quantity and hence the new stack total is wrong,
This was fixed in December 2005 (Reference Data fixed).
15I AChambers441 I Receipts and I This appears to have been an issue with I This was picked up by Reconciliation Reports(looking for I Temporary financial impact which would have
3a Payments I doing a Transfer Out which was not I receipts and payments mismatches) and investigated been cancelled out in the following period by a
Mismatch I picked up correctly when balancing. corresponding discrepancy
16I AChambers671 Postmaster balancing on counter 1, then I This occurred due to a counter failure around the time EOD I No impact
1K completing on counter 2 due to counter 1. I activities were happening and thus resulted in a mismatch in
timing out, causing a discrepancy I Reconciliation reports.
between reports
T7I_ Agnihotriv245c Horizon I The last digit of the exchange rate is I No Financial impact. It was fixed in September 2010. No impact,
Online occasionally being displayed incorrectly
AC_152871086_1
44
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
FUJ00083835
FUJ00083835
Exchange
Rate
‘on the rates board
The system does use the correct values
to caloulate the rates -the issue here is
with the display only. However, because
the display is on the Customer facing
rates board there is potential for
annoying Customers as they may get a
slightly different rate to th at advertised on
the board
18] AOConnor1581
Transaction
Limits
Failed to Harvest a quantity of 11743997
pennies as being too large.
This is another problem with limits. The branch tried to declare a
cash holding of over £100,000 in 1p and this hit a limit in
Reference Data. The fix was to police the system limit on
declarations.
No impact,
19] AOConnorS2571
The Postmaster remmed in a cheque for
£3,200, however, it did not show up on
his balance snapshot or adjust stock, but
itis showing in his Suspense Account.
This is a user error in how Post Office cheques were handled.
Following advice and Guidance the problem was resolved.
Temporary impact caused by user error.
20) arnolda229R
‘An Open Value Encashment for £5.00
was performed, the transaction was
authorised and added to the Basket but
the counter crashed before the Basket
could be settled. On Login Recovery was
invoked and a ‘Recovery Failure’ receipt
was printed for £0.00.
There was a typographical error in the script causing the issue.
This issue was found during testing of Horizon Online and fixed
before the first counter went live.
No impact (resolved before Horizon Online
went live).
21) ArnoldA2341L.
Currency
All currency codes, in terms of “iMoney”
This was an issue identified by developers regarding the way
No impact
AC_152871086_1
45
FUJ00083835
FUJ00083835
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
Code
Validation
‘Objects should be [SO-compliant, or
verified by £-sign However, we do not
Validate this at all anymore, in order to
‘support future currency codes. This
means that junk codes are accepted.
currency was handled within the counter code. It had no impact
in the real world it was closed,
22] ballantj020J
Postmaster states that she has sold a
Lyca top up for £10, the message appear
unable to connect to the data centre then
logs Postmasters out.
The transaction request has been
authorised and the reversal may not be
effective which will cause an 'E21"
reconciliation error.
There were 2 issues here:
1.There was an issue with how Reference Data was generated
which resulted in some counter scripts failing. This was fixed on
20/08/2010 (2 days after this problem occurred);
2. The Ref data issue caused an e-pay transaction to fail and
the Postmaster didn’t handle the recovery correctly
The failure to handle recovery correctly may have resulted in a
loss of £10.
This may have resulted in a small impact due to
a failure to follow the correct procedure.
23{ ballantj2547K
The ‘Transaction Processing System
Total and Counter total values for the
Number and Absolute Quantity columns
are the same but the Absolute Value for
Counter Total is greater than the
corresponding Transaction Processing
System Total by £14.80.
If the session nets to zero (add up all the
SaleValues for the same Sessionld) no
reconciliation is needed. If it doesn't, a
correction must be made to send the
data to POLFS (see <a
This is a problem with Smart Post which seemed to write slightly
corrupt transactions in that there were missing attributes
required by back end systems and reconciliation systems, but
are complete as far as the branch accounting is concerned.
This therefore has no impact on branch accounts, but does
result in reconciliation errors which are fixed by amending the
transaction copies in the backend systems. The fixes are
always to dates and not to values.
This KEL refers to amount mismatches, but the amounts used
by the reconciliation are different from those used by
accounting. They should be identical, but in this case are not.
No impact,
AC_152871086_1
46
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
FUJ00083835
FUJ00083835
href=kel_view_Kel jsp?KELRef=Maxwell
G460L>MaxwellG460L</a>) and the PM
may need to be told about a possible
receipts and payments mismatch, or at
least watch out in case one is raised
The accounting values are the correct amounts.
24] ballantj3342L
Reconciliation picked up a scenario
associated with a failed banking
transaction. Specifically, the request
from the counter never reached the
Branch Access Layer.
The application event log shows
‘AdminCfg receiving data and causing a
VPNKeyChange
In this case the request never reached
the Authorising Agent and therefore no
money was requested, the C0 did reach
the Authorising Agent but was
unexpected and has caused this
reconciliation incident,
Instruct MSU that no reconciliation
required
As the transaction failed there was no impact on the branch
accounts.
This did identify some issues in the way that failed transactions
were handled and why they resulted in reconciliation errors and
these were fixed in April 2010
No impact.
25I bambers355aL Currency
Specification
symbol
‘A MoneyGram Send transaction was
initiated. There are 3 options to define
the amount being sent:
i. including fee
ii. € not including fee
This was an issue found during testing of Horizon Online and
relates to a “E" sign being displayed when the user is being
asked to input an amount in another currency to a MoneyGram
transaction
It was agreed not to fix this until a later time (not clear if it ever
No impact.
AC_152871086_1
47
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
FUJ00083835
FUJ00083835
ii, Receive Amount excluding
fee.
The 3rd option allows a customer to
specify an amount to be sent in local
currency, such as $300, in which case
the receiver will get $300 and the send
amount is calculated in sterling
However, at the input of the receive
amount, the prompt appears (corre ctly)
as ‘Amount in USD’ but the input box
(which is a currency datatype) has a 'S"
‘symbol present. This leads the user to
think that they have selected the wrong
option and would lead to incorrect
amounts being entered here. Please see
the attached screen shot for evidence.
was fixed, however, the MoneyGram product was re-engineered
in 2015 so it would have behaved differently after that time
anyway).
This would have no impact on branch accounts, but may have
caused some mild confusion to the Subpostmaster.
26I bambers4236K
Electronic Top
Ups ("ETU")
ETU E-voucher for £10.00 is erroneously
declined as a New Reversal
The basic problem is that, to support
ETU Reversals, we rely on the
Authorising Agent remembering details
of the original ETU transaction. Ina CTO
we only have an Agent Simulator and it
is not configured to handle ETU
Reversals. Given the simplicity of the
Simulator it would be very difficult to
support ETU Reversals
This is an issue in the Counter Training service in that it doesn't
support reversals of E Top ups.
It was discovered during testing and was agreed that this would
be a restriction on the functionality supported for Counter
Training
As this only impacted counter training then there is no impact on
any Live Branch accounts,
No impact,
AC_152871086_1
48
FUJ00083835
FUJ00083835
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
This as a restriction for CTOs but is not
documented in DES/GEN/REP/0006 nor
REQICUS/STG/0004
27 BluerP5546R I Upside Down I For PING transactions (Transaction I This was a cosmetic issue when processing Transaction I No impact.
Pound Sign I Acknowledgments) £ signs will appear I Acknowledgments on a Training Counter in that the “E" sign was
as upside down question marks in the I displayed incorrectly. This has no Financial Impact as it was
training counters. only affecting training counters.
This issue was spotted in internal testing and we had no reports.
of this issue from Live Counter Training Offices. A fix went live
in January 2011
28[ _ carde235Q DropandGo IThe user initiated a Drop and Go I This was a problem in handling errors correctly in a Drop and I This would have caused a loss in the branch
transaction for £100 which failed due to I Go script provided by Post Office. accounts, although the issue was identified by
timeouts, Following the failure, a success the Subpostmaster and it would have been
message was displayed. The user I This was passed to Post Office to fix the scripts. resolved by a transaction correction.
settled the transaction and the customer
handed over £100. The customer
checked the balance and stated that the
top up had not gone through so the clerk
performed another Drop and Go
transaction which was successful. The
customer has paid in £100 but the
branch account has been debited by
£200. Accenture verified that only the
second Drop and Go top up was
successful
20{ _ carde330P Receipt I A Transfer Out of £511 cash was done I The system is behaving correctly and there would be no I No impact - the Transfer Reports and the
AC_152871086_1
49
FUJ00083835
FUJ00083835
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
Printing
‘on counter 14, session id 14-2966714-1
A corresponding Transfer in was done
immediately afterwards on counter 2,
session id 2-2046304-1. Before the
Transfer In had completed, a receipt with
the wrong session id (14-2966714-1)
was printed. After the Transfer In had
completed, the correct receipt was
printed
The extra receipt, which is almost
identical to the actual Transfer In receipt,
is quite confusing. It may lead the
Postmaster to suspect that the transfer
has been carried out twice, when in fact
it has not. Advise the Postmaster to use
the ‘Preview’ rather than the Print button
if they wish to view the individual
products being transferred
financial impact on the branch account, but the Postmaster may
be confused as to what exactly has happened
Transaction log will show exactly what has
actually happened.
30] carde427T
The transaction appeared as a state E26
on the NB102 Section 2 Link report,
This issue has also resulted in a
transaction being reported as both a
State 4 and State 6 - see PCPC0244934
PC0197368 was fixed and released as
part of release
This was an issue with failed banking transactions and then
recovery from a subsequent counter failure using the same
identifiers. Some of these appeared in reconciliation reports
when they shouldn’t be and so causing additional work to
Support teams.
However as these are all failed transactions they are all for zero
amounts and so have no impact on branch accounts.
No impact. failed banking
transaction the next banking transaction may
reuse the same unique reference. This will
cause that transaction to fail also as the
Following a
authorisation software knows the transaction
previously failed and will not pass it on to the
FI."
AC_152871086_1
50
FUJ00083835
FUJ00083835
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
CTRO1_22_01_00_RELEASE (June
2010).
Unfortunately the change was regressed
as part of a BAL change in February
2015 - see peaks PCPC0243030 and
PCPC0241771, Another fix is in
progress.
31{ carde2326R
End-of-session
sales prompts -
usability issues
The user will usually count out the cash
to be paid before pressing Fast Cash,
because once they have pressed Fast
Cash then the amount to be paid out is
reset to zero and the stack disappears.
However, if the transaction results in a
sales prompt then the stack is not
cleared and the amount payable remains
on the screen. If the user selects the
sales prompt and transacts another
product, this is added to the stack and
the ‘Total Due To Customer’ is updated
by the relevant amount.
If the user is distracted or busy then they
potentially pay out the new amount in
addition to the original amount. The
system is working as designed and
Postmasters should be referred to the
The system is operating as designed and no change was
requested by Post Office.
There may be confusion in relation to the way Sales Prompts
are handled at the end of a session.
if the correct procedures were followed there
would be no impact.
AC_152871086_1
51
FUJ00083835
FUJ00083835
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
NBSC.
32[_carde3335R
Vodafone Text
Pack Vouchers
being declined
K call was raised with e-pay to determine
why requests were being declined. This
is e-pay's response: The reason for the
Vodafone £10, £15 and £20 Text Pack
vouchers being declined by e-pay is
because they were deactivated on
request by Vodafone. We sent out a
Product Configuration document in May
detailing this change. This document was
sent to Dave Cooke at Fujitsu as well as
Clare Tetley and lain Gilbert at Royal
Mail. The deactivation was rolled out on
June 1st
Not a bug. Certain E Top Up products had been withdrawn but
the Reference Data had not been updated to remove them from
the counter. This meant that e-pay declined the requests.
Following the investigation, then the Reference Data was
updated to remove the products from the counter the following
weekend.
No impact,
33{ carde3415N
‘When a branch migrated from Horizon to
Horizon Online, differences were
reported between the ‘Pre Quantity
Move' and ‘Post Quantity Move' figures.
This was a problem in the migration of a branch from old
Horizon to Horizon Online.
The report produced pre-migration on the old Horizon didn't
take into account a Transaction Correction carried out in the last
trading period so adjust stock levels. The report produced post
migration was correct,
No impact - the figures post migration were
correct and the issue down to an inaccurate
pre-migration report.
34[ cardo5946P
Halifax 7 Bank of Scotland bank cards
declined with response 05 - ‘do not
honour’.
Branches with an “&” in their name were resulting in Banking
transactions being declined by Halifax / Bank of Scotland
No information was given by Halifax / Bank of Scotland
regarding why they are declining the cards. This was fixed by
May 2011
No financial impact on branch accounts.
AC_152871086_1
52
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
FUJ00083835
FUJ00083835
35] CCard1223Q
Counter hangs when attempting to clear
local suspense during stock unit rollover.
Tt would appear that an Invalid option was presented on the
menu of options available when settling local suspense. This
seems to have been fixed shortly afterwards,
No impact,
36] CCard4658N
The stock unit balance report only
includes figures for Add or Remove Cash
transactions done in the current
balancing period. It should however
‘show the cumulative total since the start
of the trading period.
Buttons were introduced to record when cash was added and
removed due to variances being spotted. However the
behaviour of these was not carried forward from one balancing
period to the next and so caused confusion
When the problem was identified, the buttons were “padlocked”
and a fix was issued in March 2006,
No impact
37] CharltonJ222L
Tog on event timestamp can be after the
log off and other associated events when
looking at rep events for the HBS Kiosks
The Tog on event appears to be recorded at the time the log on
is processed by the HBS server, but the other rep events are
recorded against the "DateTime" field in the incoming message
No impact,
38] CharltonJ2752T
See item 24 of Appendix 1
‘See item 24 of Appendix 7
See item 24 of Appendix 4
39{ CObeng1123Q
See item 44 of Appendix 7
See item 44 of Appendix 7
See item 44 of Appendix 1.
40] acha4349K
Reconciliation reports relating to declined
e-pay transactions are not clearing down
correctly. The affected transactions are
zero value, and have been declined by e-
pay
This has no impact on the branch but caused unnecessary work
for the Fujitsu and Post Office's reconciliation teams. Issue was
fixed on 11/10/2010.
Zero value transactions have no financial
impact on branch accounts.
4i[_acha4745R
This was an issue relating to back end
reconciliation where there was a £20
difference between 2 totals relating to
millions of pound worth of LINK
transactions.
KEL suggests issue was with reconciliation reports not being
(manually) processed correctly. Peak PC0219762 applies. This
is due to a recovery performed at the branch on the following
day which reversed the transaction. IT caused confusion in the
reports.
This was raised by POL's reporting systems.
Not known.
AC_152871086_1
53
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
FUJ00083835
FUJ00083835
42] achaSé50L
‘Kuser logged into a counter where there
was an unsettled banking transaction
requiring recovery, which had been done
in stock unit AA.
Stock unit AA was still in TP 12 BP 4.
Stock unit BB was already in TP1 BP 1
Recovery completed successfully,
correctly writing the transaction and its
settlement into stock unit AA, but for TP
18P 1
The stock unit was short by the amount
of the banking transaction in TP 12 BP 4,
but then had a matching gain in the
following period
Ifthe TP/BP is incorrect, but the stock
unit will roll into that period in the future,
then this problem will cause a
discrepancy in the current period, but it
will be balanced by an equal and
opposite discrepancy in the future
Advise the PM of this.
Bug in the recovery process that could post transactions to the
wrong TP / BP. This would result in 2 discrepancies in 2
separate periods which would cancel out, so no long term
impact on Branches.
Fix issued in June 2010, but in theory could impact counter that
migrated and hit this problem on the day of migration to Horizon
Online until September 2010 (when migration completed)
No impact,
43] acha633R
The Settle Gain/Loss Centrally products
have a minimum transaction value of
£150 and should not be available if the
This is an issue identified by Post Office rather than a branch in
that branches were being allowed to Settle Centrally small
losses (limit should be £150 or more). This was identified on
No impact
AC_152871086_1
54
FUJ00083835
FUJ00083835
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
discrepancy to be cleared from Local
Suspense is less than £150. Horizon
Online does not appear to check the
minimum value when building the pick
list and so branches can choose settle
Centrally for any value.
75” May 2010 and the fix rolled out to all branches by 5” July
2010
‘44, AChambers253,
L
This was an issue picked up by the
reconciliation checks due to smart Post
not correctly checking that the pre-paid
amount for postage was less than the
actual amount and then attempting to
generate a postage label for a negative
amount
Unclear what branches would have done at the time, however
the impact is likely to have been very small (pence rather than
pounds). This was fixed in 2005
‘Any impact would have been very minimal,
pence rather than pounds.
45] AllenD2519J
POLSAP report that a particular TC or
TAs missing from the expected BLE file.
The TC / TA entry is actually present in
the BLE file, but it lacks the TC / TA
Reference value which allows POLSAP
to identify the item.
Back end problem, involving data sent to POL from a PO client.
This is an issue caused by incorrect Reference Data with
passing data to POL’s Back end systems (POL SAP) and may
delay the processing of a TA/ TC
No impact,
46] = AllenD429U
‘One of the central systems failed during
the evening and when it restarted it
picked up the wrong time, which meant
that when it was trying to decide whether
a Txn happened before or after 7pm (the
EOD cut off) if got the answer wrong and
this meant that some data was
associated with the wrong day in back
end systems,
Looks like a back end problem, with no impact on branches
This is a problem in the data Centre which delays passing data
to Post Office's back end systems. However, it has no impact
‘on Branch Accounts.
No impact
AC_152871086_1
55
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
FUJ00083835
FUJ00083835
47] carde4027Q
This was a problem in incorrectly
handling a transaction which had been
rejected by the PIN PAD resulting in a
spurious reconciliation error on a report.
The issue first occurred in 2011 and another similar issue
occurred in 2013. It appears to be related to the old Hypercom
PIN Pads which were replaced after the second occurrence of
this issue so no further action was taken
As the transaction had been clearly declined, then there would
be no financial impact at the Branch.
No impact,
48] carde5444K
The Postmaster received a Planned
Order as follows:
Based on the last declared Cash on
Hand figure of
£97,875,00 notes + £9,156,29 coins
on 13.07.2010 you will need to remit to
the Cash Centre
£85,000.00 in notes on your next
scheduled collection day
The PM had declared cash for all his
stock units on that day, however his
actual Cash Declaration value was much
lower than the Planned Order cash
declaration value. The Planned Order
asked him to remit too much cash.
The PM normally used Till Id 1 for his
shared stock unit, but had accidentally
done a declaration with Till Id 16 during
the previous week. He was unaware that
the Till 16 declaration was being added
In a Shared SU, then it is possible to make multiple Cash
Declarations from different tills which are added together.
In this case a Cash Declaration had been made accidently for
Till 16 (instead of Till 1) which resulted in the value of cash
being doubled
This should have been spotted when the branch balanced,
However, before then the Cash Planning identified that the
Branch had too much cash and asked for some to be returned.
This is a case of branch user error and there was no actual
impact on the accounts, once the spurious Cash Declaration
was identified and removed
Temporary impact due fo human error.
AC_152871086_1
56
Claim No: HQ16X01238, HQ17X02637 & HQ17X04248
FUJ00083835
FUJ00083835
to every overnight cash declaration sent
to the cash centre.
AC_152871086_1
57